Let [tex]x[/tex] and [tex]y[/tex] be the length and height, respectively, of the rectangle.
Let the bottom left vertex be the origin (0, 0), and position the triangle so that the bottom leg lies on the horizontal axis in the [tex]x,y[/tex]-plane. The hypotenuse of the triangle then lies on the line through (0, 0) and (6, 8), with slope 8/6 = 4/3. Then for some [tex]x[/tex] between 0 and 6, we have [tex]y = \frac43x[/tex].
This means for some fixed distance [tex]x[/tex] from the origin, the rectangle has length [tex]6-x[/tex] and height [tex]\frac43x[/tex]. Thus the area of the rectangle can be expressed completely in terms of [tex]x[/tex] as
[tex]A(x) = (6-x) \times \dfrac43x = 8x - \dfrac43 x^2[/tex]
Non-calculus method:
Complete the square to get
[tex]8x - \dfrac43 x^2 = -\dfrac43 (-6x + x^2) = 12 - \dfrac43 (9 - 6x + x^2) = 12 - \dfrac43 (x-3)^2[/tex]
which is maximized when the quadratic term vanishes at [tex]x=3[/tex], giving a maximum volume of [tex]\boxed{12\,\mathrm{in}^2}[/tex].
Calculus method:
Differentiate [tex]A(x)[/tex] and find the critical points.
[tex]A'(x) = 8 - \dfrac83 x = 0 \implies x = 3[/tex]
Differentiate again to check the sign of the second derivative at the critical point.
[tex]A''(x) = -\dfrac83 \implies A''(3) = -\dfrac83 < 0[/tex]
which indicates a local maximum at [tex]x=3[/tex]. Hence the maximum area is [tex]A(3) = 12[/tex], as before.
