It looks like the system is
[tex]x' = \begin{bmatrix} -1 & 5 \\ -1 & 1 \end{bmatrix} x + \begin{bmatrix} \sin(t) \\ -2 \cos(t) \end{bmatrix}[/tex]
Compute the eigenvalues of the coefficient matrix.
[tex]\begin{vmatrix} -1 - \lambda & 5 \\ -1 & 1 - \lambda \end{vmatrix} = \lambda^2 + 4 = 0 \implies \lambda = \pm2i[/tex]
For [tex]\lambda = 2i[/tex], the corresponding eigenvector is [tex]\eta=\begin{bmatrix}\eta_1&\eta_2\end{bmatrix}^\top[/tex] such that
[tex]\begin{bmatrix} -1 - 2i & 5 \\ -1 & 1 - 2i \end{bmatrix} \begin{bmatrix} \eta_1 \\ \eta_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}[/tex]
Notice that the first row is 1 + 2i times the second row, so
[tex](1+2i) \eta_1 - 5\eta_2 = 0[/tex]
Let [tex]\eta_1 = 1-2i[/tex]; then [tex]\eta_2=1[/tex], so that
[tex]\begin{bmatrix} -1 & 5 \\ -1 & 1 \end{bmatrix} \begin{bmatrix} 1 - 2i \\ 1 \end{bmatrix} = 2i \begin{bmatrix} 1 - 2i \\ 1 \end{bmatrix}[/tex]
The eigenvector corresponding to [tex]\lambda=-2i[/tex] is the complex conjugate of [tex]\eta[/tex].
So, the characteristic solution to the homogeneous system is
[tex]x = C_1 e^{2it} \begin{bmatrix} 1 - 2i \\ 1 \end{bmatrix} + C_2 e^{-2it} \begin{bmatrix} 1 + 2i \\ 1 \end{bmatrix}[/tex]
The characteristic solution contains [tex]\cos(2t)[/tex] and [tex]\sin(2t)[/tex], both of which are linearly independent to [tex]\cos(t)[/tex] and [tex]\sin(t)[/tex]. So for the nonhomogeneous part, we consider the ansatz particular solution
[tex]x = \cos(t) \begin{bmatrix} a \\ b \end{bmatrix} + \sin(t) \begin{bmatrix} c \\ d \end{bmatrix}[/tex]
Differentiating this and substituting into the ODE system gives
[tex]-\sin(t) \begin{bmatrix} a \\ b \end{bmatrix} + \cos(t) \begin{bmatrix} c \\ d \end{bmatrix} = \begin{bmatrix} -1 & 5 \\ -1 & 1 \end{bmatrix} \left(\cos(t) \begin{bmatrix} a \\ b \end{bmatrix} + \sin(t) \begin{bmatrix} c \\ d \end{bmatrix}\right) + \begin{bmatrix} \sin(t) \\ -2 \cos(t) \end{bmatrix}[/tex]
[tex]\implies \begin{cases}a - 5c + d = 1 \\ b - c + d = 0 \\ 5a - b + c = 0 \\ a - b + d = -2 \end{cases} \implies a=\dfrac{11}{41}, b=\dfrac{38}{41}, c=-\dfrac{17}{41}, d=-\dfrac{55}{41}[/tex]
Then the general solution to the system is
[tex]x = C_1 e^{2it} \begin{bmatrix} 1 - 2i \\ 1 \end{bmatrix} + C_2 e^{-2it} \begin{bmatrix} 1 + 2i \\ 1 \end{bmatrix} + \dfrac1{41} \cos(t) \begin{bmatrix} 11 \\ 38 \end{bmatrix} - \dfrac1{41} \sin(t) \begin{bmatrix} 17 \\ 55 \end{bmatrix}[/tex]
