For part (a), first we separate variables, let $u(x,t) = X(x)T(t)$
Then, $$\frac{T''(t)}{T(t)} = \frac{X''(x)}{X(x)} = -\lambda$$
Take $\lambda = \omega^2$, where $\omega > 0$
Then, we have $$X(x) = A\cos(\omega x) + B\sin(\omega x)\\ T(t) = C\cos(\omega x) + D\sin(\omega x)$$
Plug in Boundary Conditions, $ u(0,t) = u(a,t) = u(x,0) = u(x,b) = 0$
$$X(0) = A\cos(0) + B\sin(0) = A = 0\\X(x) = B\sin(\omega x)\\X(x)=Bsin(\omega a) = 0$$
To get non-trivial solutions, we must have $sin(\omega a) = 0$, hence $\omega a = n\pi, a=\frac{n\pi}{\omega}$, where $n\in Z$
Similarly, we get $$T(t) = D\sin(\omega b) = 0$$
To get non-trivial solutions, we must have $sin(\omega b) = 0$, hence $\omega b = n'\pi, b=\frac{n'\pi}{\omega}$, where $n'\in Z$
Hence, $\frac{a}{b}\in Q$. If $\frac{a}{b}\notin Q$, we can only get trivial solution such that $B=D=0$.
For part (b), if $\frac{a}{b}\in Q$, take $p=n\pi, q=n'\pi$, then $\omega = \frac{p}{a} = \frac{q}{b} $
Then $$X(x) = B\sin(\frac{px}{a})\\T(t) = D\sin(\frac{qt}{b})$$
Take $B = D = 1$, then there exists a nontrivial solution such that $$u(x,t)=\sin(\frac{px}{a})\sin(\frac{qt}{b})$$
