Perhaps you know that
[tex]S_2 = \displaystyle\sum_{k=1}^n k^2 = \frac{n(n+1)(2n+1)}6[/tex]
and
[tex]S_3 = \displaystyle\sum_{k=1}^n k^3 = \frac{n^2(n+1)^2}4[/tex]
Then the problem is trivial, since
[tex]\displaystyle\sum_{k=1}^n k^2(k+1) = S_2 + S_3 \\\\ = \frac{2n(n+1)(2n+1)+3n^2(n+1)^2}{12} \\\\ = \frac{n(n+1)\big((2(2n+1)+3n(n+1)\big)}{12} \\\\ = \frac{n(n+1)\big(4n+2+3n^2+3n\big)}{12} \\\\ = \frac{n(n+1)(3n^2+7n+2)}{12} \\\\ = \frac{n(n+1)(3n+1)(n+2)}{12}[/tex]
Then
[tex]12\bigg(1^2\cdot2+2^2\cdot3+3^2\cdot4+\cdots+n^2(n+1)\bigg) = n(n+1)(n+2)(3n+1)[/tex]
so that a = 3 and b = 1.
