(a):
We solve homo firstly:
$$
det(A-{\lambda}I)=0\\
\begin{vmatrix}
1-\lambda & 1 \\
-2 & 4-\lambda
\end{vmatrix}=-5\lambda+{\lambda}^2+6=0\\
(\lambda-2)(\lambda-3)=0\\
\lambda_1=2,\lambda_2=3
$$
Then:
$$
(A-{\lambda}I)x=0
$$
$$
When \lambda=2
$$
$$
\begin{pmatrix}
-1 & 1 \\
-2 & 2
\end{pmatrix}
\quad= \begin{pmatrix}
0 \\
0
\end{pmatrix}
\quad
$$
RREF:
$$
\begin{pmatrix}
-1 & 1 \\
0 & 0
\end{pmatrix}
\quad= \begin{pmatrix}
0 \\
0
\end{pmatrix}
\quad
$$
$$
\text{Let x_2=t, so }x_1=x_2=t\\
t*\begin{pmatrix}
1 \\
1
\end{pmatrix}
\quad
$$
$$
When \lambda=3
$$
$$
\begin{pmatrix}
-2 & 1 \\
-2 & 1
\end{pmatrix}
\quad= \begin{pmatrix}
0 \\
0
\end{pmatrix}
\quad
$$
RREF:
$$
\begin{pmatrix}
2 & -1 \\
0 & 0
\end{pmatrix}
\quad= \begin{pmatrix}
0 \\
0
\end{pmatrix}
\quad
$$
$$
\text{Let x_2=t, so }x_1=0.5t, x_2=t\\
t*\begin{pmatrix}
1 \\
2
\end{pmatrix}
\quad
$$
So, the general solution is :
$$
y= c_1e^{2t}\begin{pmatrix}
1 \\
1
\end{pmatrix}
\quad +c_2e^{3t}\begin{pmatrix}
1 \\
2
\end{pmatrix}
\quad
$$
(b):
$$
\phi = \begin{pmatrix}
e^{2t} & e^{3t} \\
e^{2t} & 2e^{3t}
\end{pmatrix}
\quad
$$
$$
\phi * u' = g(t)
$$
$$
\begin{pmatrix}
e^{2t} & e^{3t} \\
e^{2t} & 2e^{3t}
\end{pmatrix}
\quad *{\begin{pmatrix}
u_1' \\
u_2'
\end{pmatrix}
\quad}=\begin{pmatrix}
\frac{e^{4t}}{e^{2t} + 1 } \\
0
\end{pmatrix}
\quad
$$
Simplify we can get:
$$
u_1'=-\frac{-e^t}{e^{2t} + 1}\\
u_2'=\frac{2e^{2t}}{e^{2t}+1}
$$
Therefore:
$$
u_1=ln(e^{2t}+1)+c_1\\
u_2=-arctane^t+c_2
$$
Finally:
$$
x=\phi * u=(ln(e^{2t}+1)+c_1)*{\begin{pmatrix}
e^{2t} \\
e^{2t}
\end{pmatrix}
\quad} +(-arctane^t+c_2)*\begin{pmatrix}
e^{3t} \\
2e^{3t}
\end{pmatrix}
\quad
$$
OK, except LaTeX sucks:
1) * IS NOT a sign of multiplication
2) "operators" should be escaped: \cos, \sin, \tan, \ln
