Toronto Math Forum

The critical points would be when $x'=0$ and $y'=0$:
$$\begin{align}0=x^2+xy-x=x(x+y-1)\end{align}$$
$$\begin{align}0=2y^2+xy-3y=y(2y+x-3)\end{align}$$
From $(1)$ and $(2)$ the critical points would be $(0,0)$, $(0,\frac{3}{2})$, $(1,0)$ and $(-1,2)$.
The Jacobian matrix would be:
$$\begin{align}J=\begin{pmatrix}F_{x}&F_{y}\\G_{x}&G_{y}\end{pmatrix}=\begin{pmatrix}-2x-y+1&-x\\-y&-4y-x+3\end{pmatrix}\end{align}$$
At $(0,0)$:
$$\begin{align}J=\begin{pmatrix}1&0\\0&3\end{pmatrix}\end{align}$$
Since it is a triangular matrix, the eigenvalues would just be the diagonal entries so $r_{1}=1$ and $r_{2}=3$. This means that $0<r_{1}<r_{2}$ so the nonlinear system would be an unstable node.
At $(0,\frac{3}{2})$:
$$\begin{align}J=\begin{pmatrix}\frac{-1}{2}&0\\\frac{-3}{2}&-3\end{pmatrix}\end{align}$$
Since it is a triangular matrix, the eigenvalues would just be the diagonal entries so $r_{1}=\frac{-1}{2}$ and $r_{2}=-3$. This means that $0>r_{1}>r_{2}$ so the nonlinear system would be an asymptotically stable node.
At $(1,0)$:
$$\begin{align}J=\begin{pmatrix}-1&-1\\0&2\end{pmatrix}\end{align}$$
Since it is a triangular matrix, the eigenvalues would just be the diagonal entries so $r_{1}=-1$ and $r_{2}=2$. This means that $r_{1}<0<r_{2}$ so the nonlinear system would be an unstable saddle point.
At $(-1,2)$:
$$\begin{align}J=\begin{pmatrix}1&1\\-2&-4\end{pmatrix}\end{align}$$
Solving gives eigenvalues $r_{1}=\frac{-3+\sqrt17}{2}$ and $r_{2}=\frac{-3-\sqrt17}{2}$. Since $r_{2}<0<r_{1}$ the nonlinear system would be an unstable saddle point.

The critical points would be when $x'=0$ and $y'=0$:
$$\begin{align}0=(1+x)\sin(y)\end{align}$$
$$\begin{align}0=1-x-\cos(y)\end{align}$$
From $(1)$, $\sin(y)$ would be $0$ when $y=n\pi$ where $n=0, 1, 2, 3,...$. Since there is a $\cos(y)$ in $(2)$, the previous equation can be split into two cases: $y=2n\pi$ where $n=0, 1, 2, 3,....$ and $y=n\pi$ where $n=1, 3, 5,....$. Then in case 1 $\cos(y)=1$ and in case 2 $cos(y)=-1$. Plugging both of these into $(2)$ gives the critical points:
$$\begin{align}(0, 2n\pi) \quad where \quad n=0, 1, 2, 3,...\end{align}$$
$$\begin{align}(2, n\pi) \quad where \quad n=1, 3, 5,...\end{align}$$
The Jacobian matrix would be:
$$\begin{align}J=\begin{pmatrix}F_{x}&F_{y}\\G_{x}&G_{y}\end{pmatrix}=\begin{pmatrix}\sin(y)&-\cos(y)(1+x)\\-1&\sin(y)\end{pmatrix}\end{align}$$
At $(0, 2n\pi)$:
$$\begin{align}J=\begin{pmatrix}0&1\\-1&0\end{pmatrix}\end{align}$$
This gives eigenvalues $r_{1}=i$ and $r_{2}=-i$. Since $r_{1}, r_{2}=\lambda\pm i\mu$ where $\lambda=0$ the nonlinear system would be an indeterminate center or spiral point.
At $(2, n\pi)$:
$$\begin{align}J=\begin{pmatrix}0&-3\\-1&0\end{pmatrix}\end{align}$$
The eigenvalues would be $r_{1}=\sqrt{3}$ and $r_{2}=-\sqrt{3}$. Since $r_{2}<0<r_{1}$ the nonlinear system would be an unstable saddle point.

I got something different for part (b). I got that $u_{2}'$ should be:
$$\begin{align}u_{2}'=\frac{-6e^{3t}}{e^t+1}\end{align}$$
Therefore $u_{2}$ would be:
$$\begin{align}u_{2}(t) = -3e^{2t}+6e^t-6\ln(e^t+1)+c_{2}\end{align}$$
Then the general solution would be:
$$\begin{align}\textbf{x}(t)=(18\ln(e^t+1)+c_{1})e^{-t}\begin{pmatrix}1\\0\end{pmatrix}+(-3e^{2t}+6e^t-6\ln(e^t+1)+c_{2})e^{-3t}\begin{pmatrix}1\\-2\end{pmatrix}\end{align}$$
Subbing in the initial condition $\textbf{x}(0)=\begin{pmatrix}1-3\ln2\\-3-3\ln2\end{pmatrix}$:
$$\begin{align}1-3\ln2=c_{1}+3+c_{2}\end{align}$$
$$\begin{align}-3-3\ln2=-6-2c_{2}\end{align}$$
Solving these equations for the constants gives:
$$\begin{align}c_{1}=\frac{-19-9\ln2}{2}\end{align}$$
$$\begin{align}c_{2}=\frac{-3+3\ln2}{2}\end{align}$$
Therefore the general solution for the IVP is:
$$\begin{align}\textbf{x}(t)=\frac{-19-9\ln2}{2}e^{-t}\begin{pmatrix}1\\0\end{pmatrix}+\frac{-3+3\ln2}{2}e^{-3t}\begin{pmatrix}1\\-2\end{pmatrix}+18\ln(e^t+1)e^{-t}\begin{pmatrix}1\\0\end{pmatrix}-3e^{-t}\begin{pmatrix}1\\-2\end{pmatrix}+6e^{-2t}\begin{pmatrix}1\\-2\end{pmatrix}-6\ln(e^t+1)e^{-3t}\begin{pmatrix}1\\-2\end{pmatrix}
\end{align}$$

I got a different answer for part (b). Aside from a missing constant, I got everything the same up until $(6)$ so I will start from there. Rewriting $(6)$ I got:
$$\textbf{x}(t)=(ln(e^t+1)+c_{1})e^t\begin{pmatrix}1\\1\end{pmatrix}+c_{2}e^{-2t}\begin{pmatrix}1\\-2\end{pmatrix}=ln(e^t+1)e^t\begin{pmatrix}1\\1\end{pmatrix}+c_{1}e^t\begin{pmatrix}1\\1\end{pmatrix}+c_{2}e^{-2t}\begin{pmatrix}1\\-2\end{pmatrix}$$
Plugging in the initial condition:
$$\textbf{x}(0)=\begin{pmatrix}3\\0\end{pmatrix}$$
This gives:
$$3=ln(2)+c_{1}+c_{2} \implies c_{1}=3-ln(2)-c_{2}\qquad(7)$$
and:
$$0=ln(2)+c_{1}-2c_{2}\qquad(8 )$$
Plugging $(7)$ into $(8 )$ gives $c_{2}=1$. Plugging this back into (7) gives $c_{1}=2-ln(2)$. Therefore the constants are:
$$c_{1}=2-ln(2)\quad and \quad c_{2}=1$$
Therefore the general solution would be:
$$\textbf{x}(t)=ln(e^t+1)e^t\begin{pmatrix}1\\1\end{pmatrix}+(2-ln(2))e^t\begin{pmatrix}1\\1\end{pmatrix}+e^{-2t}\begin{pmatrix}1\\-2\end{pmatrix}$$