First we find the solution for the homogeneous system
\begin{equation}
y^{(3)}-2y^{(2)}+4y^{(1)}-8y=0
\end{equation}
The corresponding characteristic equation is
\begin{equation}
r^3-2r^2+4r-8=0
\end{equation}
Three roots are
\begin{equation}
r_1=2
\end{equation}
\begin{equation}
r_2=2i
\end{equation}
\begin{equation}
r_3=-2i
\end{equation}
Then the solution for the homogeneous system is
\begin{equation}
y_c(t)=c_1e^{2t}+c_2\cos(2t)+c_3\sin(2t)
\end{equation}
Then we follow to find a particular solution $Y(t)$
We should have
\begin{equation}
Y(t)=Y_1(t)+Y_2(t)
\end{equation}
where
\begin{equation}
Y_1(t)=Ate^{2t}
\end{equation}
and
\begin{equation}
Y_2(t)=M\sin(t)+N\cos(t)
\end{equation}
By plugging in to the equation, we can find that $A=2$, $M=2$ and $N=-4$
In this way we get the required general solution
\begin{equation}
y(t)=c_1e^{2t}+c_2\cos(2t)+c_3\sin(2t)+2te^{2t}+2\sin(t)-4\cos(t)
\end{equation}
