Electrical – MOSFET getting over hot

While I was writing this answer, @Connor posted an answer covering most of it. In any case...

There are a few things that need to be addressed in the presented circuit.

1. Eliminate C2 entirely: The MOSFET is being used in a switching topology, not for linear amplification, C2 completely subverts the sharp switching desired for minimal power loss. The IRFZ44N needs to be switched as rapidly as possible between fully conducting and fully blocking states, for least power wastage i.e. heat.
2. The maximum current available to charge the gate at the high-going edge of the gate input (from PWM signal) is limited by R3 = 4.7k ==> Ig < 5.1 mA. This current charges up the substantial gate capacitance at each rising edge for V_gs to rise, and is way too low. This will cause R_ds to rise very slowly, and while in this rising part of the graph, the MOSFET will waste a lot of power as heat.
   • Reduce R3 as far as the optocoupler's collector current rating will allow, or better yet:
   • Use the optocoupler to drive a BJT or smaller FET with very low gate capacitance as a switch to allow far higher gate charging current to the power MOSFET.
3. Apply the same rationale to the discharging of the gate capacitance on each falling edge of gate input. For the very high PWM frequencies indicated, a push-pull gate driver, either an integrated device or made of discrete components, is typically used instead of a passive gate drive such as shown.
4. If the rather high PWM frequency mentioned is not really needed, consider moving to a far lower PWM frequency: 500+ Hertz is often good enough, but 20-30 KHz is typical, so as to be beyond human hearing and therefore PWM noise from the motor. The higher the frequency, the greater the percentage of time the power MOSFET will be in its intermediate transition stage, rather than on or off. Therefore, more heat.
   Edit: 244 Hz as updated by OP is much more realistic.
5. The higher temperature at low duty cycles is again due to capacitor C2: It is being unable to charge up to the gate's switching voltage during the too-brief high pulses of the PWM signal. The V_gs to aim for is not the V_gs(th) of 2 to 4 Volts, but 6+ Volts, where the curve begins to flatten out in Figure 3 of the datasheet. With higher duty cycles, the capacitor does manage to breach the desired V_gs most of the time.

It seems to me that what you need is a voltage snubber across the MOSFET. An easy way to do that is to simply connect a series capacitor + resistor across the MOSFET. I'd guesstimate that a value of about 2.7 nF (about 3x capacitance of the MOSFET) and resistor of 100 \$\Omega\$ would be about right.

This ancient application note describes the various kinds of snubber circuits, including when and how to use them. You might find some inspiration there.