For the system of ODEs
\begin{equation*}
\left\{\begin{aligned}
&x'_t =4x^2y-2x^2-4xy+2y, \\
&y'_t =-4xy^2+2y^2+4xy-2x
\end{aligned}\right.
\end{equation*}
(a) linearize the system at $\ x_0 = 1\, , \, y_0 = 1\ $ and sketch the phase portrait of this linear system,
(b) find the equation of the form $\ H(x,y) = C\ $ satisfied by the trajectories of the nonlinear system,
(c) describe the type of the critical point $\ x_0 = 12 \, , \, y_0 = 10\ $ of the nonlinear system.
Solution
(a) Let $f= 4x^2y -2x^2-4xy +2y$, $g= -4xy^2 + 2y^2+4xy -2x$; then $f_x(1,1)=0$, $f_y(1,1)= 2$, $g_x(1,1)=-2$, $f_y(1,1)= 0$ and the linearized system is
\begin{equation*}\left\{\begin{aligned}
&X'_t =2Y, \\
&Y'_t = -2X
\end{aligned}\right.
\end{equation*}
with phase portrait consisting of clock-wise circles.
(b) Rewriting system as $fdx-gdy=0$ we get
$$
(4xy^2-2y^2 -4xy+2x)\, dx+ (4x^2 y-2x^2 -4xy+2y)\,dy =0
$$
which is exact; then
$$
H_x= 4xy^2-2y^2 -4xy+2x , \quad H_y= 4x^2 y-2x^2 -4xy+2y)$$
and the first equation implies that
$$
H= 2x^2 y^2 -2xy^2 -2x^2y +x ^2 + \phi(y)$$
and the second equation implies that $\phi'=2y$ and $y=y^2$ and then
$$
H=2x^2 y^2 -2xy^2 -2x^2y +x ^2 + y^2 = x^2(y-1)^2 + y^2(x-1)^2.
$$
(c) Since linearized system has a center and original system has a solution $H(x,y)=C$ the type of the stationary point is a center.
Remark
In fact the system has also critical point $(0,0)$ of the type center, and critical point $(\frac{1}{2},\frac{1}{2})$ of the type saddle
