Answer: Choice B
[tex](f * g)(x) = \frac{x^2+6x+8}{x^2+2x-15}, \ \text{ for } x \ne -5 \text{ and } x \ne 3\\\\[/tex]
=================================================
Work Shown:
[tex]h(x) = (f * g)(x)\\\\h(x) = f(x) * g(x)\\\\h(x) = \frac{x^2-16}{x^2+3x-10}*\frac{x^2-4}{x^2-7x+12}\\\\h(x) = \frac{(x-4)(x+4)}{(x+5)(x-2)}*\frac{(x-2)(x+2)}{(x-3)(x-4)}\\\\h(x) = \frac{x+4}{(x+5)(x-2)}*\frac{(x-2)(x+2)}{x-3} \ \ \text{ ... see note 1}\\\\h(x) = \frac{x+4}{x+5}*\frac{x+2}{x-3} \ \ \text{ ... see note 2}\\\\h(x) = \frac{(x+4)(x+2)}{(x+5)(x-3)}\\\\h(x) = \frac{x^2+6x+8}{x^2+2x-15}\\\\[/tex]
note 1: A pair of (x-4) terms canceled
note 2: A pair of (x-2) terms canceled
------------------------------
Extra info (optional section):
The fact that [tex]x \ne -5 \text{ and } x \ne 3[/tex] is to avoid a division by zero error in the simplified version of h(x).
I would argue that [tex]x \ne 2 \text{ and } x \ne 4[/tex] should be thrown in as well simply so that the domains match up perfectly with the original f(x) and g(x) functions.
So I think the full domain should be that x is any real number but
[tex]x \ne -5 \text{ and } x \ne 2\\x \ne 3 \text{ and } x \ne 4[/tex]
Put another way: if x = 2 is allowed in h(x), then that clashes with the fact that it's not allowed in f(x). The same idea happens with x = 4 but with g(x) this time. It's possible your teacher glossed this fact over, or ran out of room.
