Electronic – Boost SMPS getting very hot. Why

While I was writing this answer, @Connor posted an answer covering most of it. In any case...

There are a few things that need to be addressed in the presented circuit.

1. Eliminate C2 entirely: The MOSFET is being used in a switching topology, not for linear amplification, C2 completely subverts the sharp switching desired for minimal power loss. The IRFZ44N needs to be switched as rapidly as possible between fully conducting and fully blocking states, for least power wastage i.e. heat.
2. The maximum current available to charge the gate at the high-going edge of the gate input (from PWM signal) is limited by R3 = 4.7k ==> Ig < 5.1 mA. This current charges up the substantial gate capacitance at each rising edge for V_gs to rise, and is way too low. This will cause R_ds to rise very slowly, and while in this rising part of the graph, the MOSFET will waste a lot of power as heat.
   • Reduce R3 as far as the optocoupler's collector current rating will allow, or better yet:
   • Use the optocoupler to drive a BJT or smaller FET with very low gate capacitance as a switch to allow far higher gate charging current to the power MOSFET.
3. Apply the same rationale to the discharging of the gate capacitance on each falling edge of gate input. For the very high PWM frequencies indicated, a push-pull gate driver, either an integrated device or made of discrete components, is typically used instead of a passive gate drive such as shown.
4. If the rather high PWM frequency mentioned is not really needed, consider moving to a far lower PWM frequency: 500+ Hertz is often good enough, but 20-30 KHz is typical, so as to be beyond human hearing and therefore PWM noise from the motor. The higher the frequency, the greater the percentage of time the power MOSFET will be in its intermediate transition stage, rather than on or off. Therefore, more heat.
   Edit: 244 Hz as updated by OP is much more realistic.
5. The higher temperature at low duty cycles is again due to capacitor C2: It is being unable to charge up to the gate's switching voltage during the too-brief high pulses of the PWM signal. The V_gs to aim for is not the V_gs(th) of 2 to 4 Volts, but 6+ Volts, where the curve begins to flatten out in Figure 3 of the datasheet. With higher duty cycles, the capacitor does manage to breach the desired V_gs most of the time.

Remove the N-channel, you have no use for it. They're describing digital logic circuits where you are driving cascades of logic elements and the input impedance/resistance of the next stage is very high. In your case, you have a relatively low impedance of the next stage (tail light) probably being in the 100s of ohms tops. So your output will drop to zero automatically and rapidly without the NMOS at all.

Another potential issue with your circuit is the 3 Mohm. This is going to cause a very slow drop in the gate voltage causing very slow turn-on time for the PMOS. With the redundant NMOS also turning off really slow at the same time, you have a good chance that you've burned both your PMOS and NMOS and that's why you're seeing the 120 ohms across your pulldown rather than the 3 Mohm. Your NMOS is partially shorted now so you definitely want to chuck that part.

Your circuit as it stands would probably work if you had a much lower (say 1-10k pulldown) which would prevent the PMOS and NMOS from turning on too long at the same time. You allowed them to turn on "forever" with the 3 Mohm which causes a direct short from the top of the PMOS to the bottom of the NMOS as they transition from one state to the next.

Summary:
1. Remove NMOS from circuit.
2. Decrease your pull down resistance to 1-10k ohm.