The critical points would be when $x'=0$ and $y'=0$:
$$\begin{align}0=x+x^2+y^2\end{align}$$
$$\begin{align}0=y-xy\end{align}$$
From $(1)$ and $(2)$ the critical points would be $(0,0)$ and $(-1,0)$.
The Jacobian matrix would be:
$$\begin{align}J=\begin{pmatrix}F_{x}&F_{y}\\G_{x}&G_{y}\end{pmatrix}=\begin{pmatrix}2x+1&2y\\-y&1-x\end{pmatrix}\end{align}$$
At $(0,0)$:
$$\begin{align}J=\begin{pmatrix}1&0\\0&1\end{pmatrix}\end{align}$$
Since it is a triangular matrix, the eigenvalues would just be the diagonal entries so $r_{1}=r_{2}=1$. This means that $r_{1}=r_{2}>0$ so the nonlinear system would be an unstable proper node or spiral point.
At $(-1,0)$:
$$\begin{align}J=\begin{pmatrix}-1&0\\0&2\end{pmatrix}\end{align}$$
Since it is a triangular matrix, the eigenvalues would just be the diagonal entries so $r_{1}=-1$ and $r_{2}=2$. Since $r_{1}<0<r_{2}$ the nonlinear system would be an unstable saddle point.
