It seems that the equation is missing.

The notation here is not technically precise. The left-hand side here should be

\(\dfrac\partial{\partial \mathbf{u}}(\text{expression inside ``$[\cdots]$" in the HJB})\)

instead of

\(\dfrac\partial{\partial \mathbf{u}}.\)

Why do we expect a solution to the differential Riccati equation to maintain positive definiteness in the first place?

The differential Riccati equation is a Final Value Problem (FVP) since only a terminal condition is given. This motivates writing the differential equation with "\(-\dot{\mathbf S}(t)\)" instead of "\(\dot{\mathbf S}(t)\)", since the dynamics move backward in time.

In particular, we guess the form of the solution, where \(\mathbf{S}_{xx}\) is a symmetric positive-definite matrix-valued function and \(\mathbf{s}_x\) is a vector-valued function, and \(s_0\) is a scalar valued function that remain to be determined.

Once we have the functions \(\mathbf S\) and \(\mathbf s\), we can plug them into \(\mathbf{u}^*(\mathbf{x}, t)=\mathbf{u}d(t)-\mathbf{R}^{-1} \mathbf{B}^T\left[\mathbf{S}{x x}(t) \mathbf{x}+\mathbf{s}_x(t)\right]\) to produce a (time-varying) feedback law.

The optimal feedback law \(\mathbf{u}^\) depends on \(\mathbf{x}\) so its arguments should be \(\mathbf{u}^(\mathbf x, t).\)

Do we get the same differential equations if \(\mathbf{A}\) and \(\mathbf B\) are time-varying?

The second equation comes from the HJB equation by the following process: 1. Substitute \(u^\) for \(u\), and the calculated expressions for \(\partial J^/\partial \mathbf{x}\) and \(\partial J^*/\partial t.\) 2. Split \(2x^\top S(x) Ax\) into \(x^\top S(x) Ax + x^\top A^\top S(x)x,\) which in turn is equal to \(x^\top (S(x) A + A^\top S(x))x,\) (This is advantageous because the matrix \(S(x) A + A^\top S(x)\) is symmetric.) 3. Collect like-terms (namely \(\mathbf S(t)\mathbf B\mathbf R^{-1}\mathbf B^\top\mathbf S(t) - 2\mathbf S(t)\mathbf B\mathbf R^{-1}\mathbf B^\top\mathbf S(t)\)) and pulling the \(\mathbf x^\top\) and \(\mathbf x\) to the outside. 4. Solve for \(-\dot{\mathbf S}.\) (Why don't we leave the negative sign on the other side?)

Where \(S\) is a matrix-valued function of time that remains to be determined.

In this tracking problem, the functions \(\mathbf{x}_d\) and \(\mathbf{u}_d\) are given reference trajectories that we want to track.

This section considers the case where we impose a terminal constraint on the system's trajectory. Namely, we want \(x(t_f) = 0.\) This is equivalent to making the the terminal cost infinite for \(x(t_j) \neq 0\) or \(\mathbf{Q}_f\) infinite.

In the second equation, it should be \(\frac{\partial J^}{\partial t}\) instead of \(\frac{\partial J}{\partial t}.\)