The critical points would be when $x'=0$ and $y'=0$:
$$\begin{align}0=x^2+xy-x=x(x+y-1)\end{align}$$
$$\begin{align}0=2y^2+xy-3y=y(2y+x-3)\end{align}$$
From $(1)$ and $(2)$ the critical points would be $(0,0)$, $(0,\frac{3}{2})$, $(1,0)$ and $(-1,2)$.
The Jacobian matrix would be:
$$\begin{align}J=\begin{pmatrix}F_{x}&F_{y}\\G_{x}&G_{y}\end{pmatrix}=\begin{pmatrix}-2x-y+1&-x\\-y&-4y-x+3\end{pmatrix}\end{align}$$
At $(0,0)$:
$$\begin{align}J=\begin{pmatrix}1&0\\0&3\end{pmatrix}\end{align}$$
Since it is a triangular matrix, the eigenvalues would just be the diagonal entries so $r_{1}=1$ and $r_{2}=3$. This means that $0<r_{1}<r_{2}$ so the nonlinear system would be an unstable node.
At $(0,\frac{3}{2})$:
$$\begin{align}J=\begin{pmatrix}\frac{-1}{2}&0\\\frac{-3}{2}&-3\end{pmatrix}\end{align}$$
Since it is a triangular matrix, the eigenvalues would just be the diagonal entries so $r_{1}=\frac{-1}{2}$ and $r_{2}=-3$. This means that $0>r_{1}>r_{2}$ so the nonlinear system would be an asymptotically stable node.
At $(1,0)$:
$$\begin{align}J=\begin{pmatrix}-1&-1\\0&2\end{pmatrix}\end{align}$$
Since it is a triangular matrix, the eigenvalues would just be the diagonal entries so $r_{1}=-1$ and $r_{2}=2$. This means that $r_{1}<0<r_{2}$ so the nonlinear system would be an unstable saddle point.
At $(-1,2)$:
$$\begin{align}J=\begin{pmatrix}1&1\\-2&-4\end{pmatrix}\end{align}$$
Solving gives eigenvalues $r_{1}=\frac{-3+\sqrt17}{2}$ and $r_{2}=\frac{-3-\sqrt17}{2}$. Since $r_{2}<0<r_{1}$ the nonlinear system would be an unstable saddle point.
