Since the given DE is not exact, we need to find a function $\mu (x,y)$ such that the equation $\mu (3x^{2}y+2xy+y^{3}) + \mu (x^{2}+y^{2})y’=0$ is exact.
This means that
$$\frac{∂}{∂y}[\mu (3x^{2}y+2xy+y^{3})]=\frac{∂}{∂x}[\mu (x^{2}+y^{2})]$$
$$\frac{∂\mu}{∂y}(3x^{2}y+2xy+y^{3})+\mu (3x^{2}+2x+3y^{2}) = \frac{∂\mu}{∂x}(x^{2}+y^{2}) + \mu (2x)$$
$$\frac{∂\mu}{∂y}(3x^{2}y+2xy+y^{3})+\mu (3x^{2}+3y^{2}) = \frac{∂\mu}{∂x}(x^{2}+y^{2})$$
Suppose $\mu$ is a function of $x$ only. (i.e. $\mu=\mu (x)$)
Then $\frac{∂\mu}{∂y}=0$ and $\frac{∂\mu}{∂x}=\frac{d\mu}{dx}$
Then $$\mu (3x^{2}+3y^{2}) = \frac{d\mu}{dx}(x^{2}+y^{2})$$
Divide both sides by $(x^{2}+y^{2})$ we have $$3\mu = \frac{d\mu}{dx}$$ which is a separable equation.
$$\int{\frac{1}{\mu}d\mu} = \int{3 dx}$$
$$ln(\mu)=3x$$
$$\mu = e^{3x}$$
Thus, $\mu = e^{3x}$ is an integration factor for the given DE.
Multiply the given DE by $\mu = e^{3x}$, we get
$$e^{3x} (3x^{2}y+2xy+y^{3}) + e^{3x} (x^{2}+y^{2})y’=0$$ which is exact.
Then by theorem, we know there exist a function $\phi(x,y)=C$ which satisfies the given DE
We also know that
$\frac{∂\phi}{∂x} = e^{3x} (3x^{2}y+2xy+y^{3})$ and $\frac{∂\phi}{∂y} =e^{3x} (x^{2}+y^{2})$
Integrate $\frac{∂\phi}{∂y} =e^{3x} (x^{2}+y^{2})$ with respect to $y$ we have
$$\phi(x,y)= e^{3x} (x^{2}y+\frac{1}{3}y^{3}) +g(x)$$
Then, take derivative on both sides with respect to $x$, we have
$$\frac{∂\phi}{∂x} = 3e^{3x} (x^{2}y+\frac{1}{3}y^{3}) + e^{3x} (2xy) + g’(x)$$
Simplify a little bit we have
$$\frac{∂\phi}{∂x} = e^{3x} (3x^{2}y + y^{3} + 2xy) + g’(x)$$
Since we know that
$\frac{∂\phi}{∂x} = e^{3x} (3x^{2}y+2xy+y^{3})$
This means that $g’(x)=0$. So $g(x)=C$
Altogether, we have $$\phi(x,y)= e^{3x} (x^{2}y+\frac{1}{3}y^{3})=C$$
Which means the general solution to the given DE is $$e^{3x} (x^{2}y+\frac{1}{3}y^{3})=C$$
