Find where the two curves meet.
[tex]x^3 - x^2 + x + 8 = 5x^2 - 7x + 8 \\\\ \implies x^3 - 6x^2 + 8x = 0 \\\\ \implies x (x-2) (x - 4)= 0 \implies x=0, x=2, x=4[/tex]
The area between the curves is
[tex]\displaystyle \int_0^4 \left|\left(x^3-x^2+x+8\right) - \left(5x^2 - 7x + 8\right)\right| \, dx = \int_0^4 \left|x(x-2)(x-4)\right| \, dx[/tex]
When [tex]x[/tex] is between 0 and 2, [tex]x(x-2)(x-4)[/tex] is positive; when [tex]x[/tex] is between 2 and 4, [tex]x(x-2)(x-4)[/tex] is negative. So we split the integral at [tex]x=2[/tex] to get
[tex]\displaystyle \int_0^2 x(x-2)(x-4) \, dx - \int_2^4 x(x-2)(x-4)\,dx[/tex]
In the second integral, substitute [tex]y=x-2[/tex] to get
[tex]\displaystyle \int_0^2 x(x-2)(x-4) \, dx - \int_0^2 (y+2)y(y-2)\,dy[/tex]
[tex]\displaystyle \int_0^2 x(x-2) \bigg((x-4) - (x+2)\bigg) \, dx[/tex]
[tex]\displaystyle -6 \int_0^2 x(x-2) \, dx[/tex]
[tex]\displaystyle 6 \int_0^2 \left(2x - x^2\right) \, dx[/tex]
[tex]\displaystyle 6 \left(x^2 - \frac13x^3\right)\bigg|_0^2 = \boxed{8}[/tex]
