We start by taking the $ u = X(x)Y(y) $, then plugging in gives us:
$$ X'' Y + Y'' X = -\lambda XY $$
$$ X(0)Y(y) = X'(a)Y(y) = 0 \ \ and \ \ X(x)Y(0) = X(x)Y(b)' = 0 $$
Dividing both of these expressions by $XY$ gives us
$$\frac{X''}{X} + \frac{Y''}{Y} = -\lambda $$
$$ X(0) = X'(a) = 0 \ \ and \ \ Y(0) = Y(b)' = 0 $$
Now we know that both $\frac{Y''}{Y} $ and $\frac{X''}{X} $ are independent of each other, i.e. they should be equivalent to some constant. Introduce constants $\lambda_1, \lambda_2$ such that $ \lambda = \lambda_1 + \lambda_2$, thus $\frac{Y''}{Y} = -\lambda_{1} $ and $\frac{X''}{X} = -\lambda_{2} $
Then, we can examine the different cases of $\lambda_{1,2}$.
i) If both $\lambda_{1} = 0 = \lambda_{2} $:
We get a simplified eigenvalue problem of:
$$ X'' = 0 , Y'' = 0 $$
Meaning that:
$$ X = A_0 x + B_0 , Y = C_0y + D_0 $$
Running it through the boundary conditions, we can easily show that: $ B_0 = 0 , D_0 = 0 , A_0 = 0 , C_0 = 0 $
I.e. this leads to a trivial solution of the eigenvalue problem.
For $\lambda_{1}, \lambda_{2} >0 $
We will get eigenvalue problem of $$X'' + \lambda_2 X = 0 , Y'' + \lambda_1 Y = 0$$
This results in:
$$X(x) = Acos\sqrt{\lambda_2}x + Bsin\sqrt{\lambda_2}x$$
$$Y(y) = Ccos\sqrt{\lambda_1}y + Dsin\sqrt{\lambda_1}y$$
Apply the boundary conditions, and we get:
$ A = 0 , C= 0, 0 = \sqrt{\lambda_2}Bcos\sqrt{\lambda_2}a, 0 = \sqrt{\lambda_1}Dcos\sqrt{\lambda_1}b$
We're in search of nontrivial solutions, which can be attained if $\sqrt{\lambda_{1,2}}b,a = \frac{\pi(2n+1)}{2} $, thus we have eigenvalues and eigenfunctions of:
$$ \lambda_1 = (\frac{\pi(2m+1)}{2b})^2 , Y_{m} = sin(\frac{\pi(2m+1)}{2b})y , \lambda_2 = (\frac{\pi(2n+1)}{2a})^2, X_{n} = sin(\frac{\pi(2n+1)}{2a}x) $$
For the case of $\lambda_1 , \lambda_2 < 0 $ we solve the eigenvalue problem of:
$$X'' - \lambda_2 X = 0 , Y'' - \lambda_1 Y = 0 $$, which gives us, in turn:
$$X(x) = Ae^{\sqrt{\lambda_2} x} + Be^{-\sqrt{\lambda_2} x} , Y(y) = Ce^{\sqrt{\lambda_1} y} + De^{-\sqrt{\lambda_1} y} $$
Apply the boundary conditions to get: $ A+ B = 0, C+D = 0$ , $ 0 = \sqrt{\lambda_2} A (e^{\sqrt{\lambda_2}a} +e^{-\sqrt{\lambda_2}a}) = 2A\sqrt{\lambda_2}\cosh(\sqrt{\lambda_2}a) $
and $ 0 = \sqrt{\lambda_1}C(e^{\sqrt{\lambda_1}b} +e^{-\sqrt{\lambda_1}b}) = 2C\sqrt{\lambda_1}\cosh(\sqrt{\lambda_1}b) $
But since the $\cosh$ function never reaches 0, we can't have a nontrivial solution. Therefore there only exists a trivial solution in this case.
Note: updated solution to reflect feedback
