Electronic – What material should I use to insulate the MOSFET from a heatsink

First you need to determine how much power is dissipated in the transistor. The second factor is the maximum ambient temperature that you expect the transistor to experience. Finally, decide what maximum temperature you want at the transistor chip itself...maybe 75°C is a good target. Subtract the ambient temperature from the maximum transistor die temperature and you have the maximum temperature increase from the transistor chip to the outer surface of the heatsink. Divide the temperature difference by the power dissipated and you have a value for thermal resistance in °C/W (degrees C per watt). This is the maximum thermal resistance you can allow from the transistor die to the ambient air. Find out the thermal resistance of the transistor package itself. For the transistor you list above, this is given in the data sheet as "Thermal Resistance, Junction to Case" and it has a value of 6.25°C/W. Now subtract the thermal resistance of the transistor package from the total allowable thermal resistance you calculated earlier. Now go to the catalogs and look for a heatsink that fits the TO-220 package and has a thermal resistance, in still air, equal to or less than this value.

Going by the picture (if that's the amplifier under question), that's a pretty large heatsink for 4 MOSFETs dissipating 5W each. It looks like there's other heat sources attached to the heatsink, however, and although you don't say what these are dissipating, or what the thermal impedences from other heat sources to MOSFETs-in-question are, I'm tempted to say the they (in combination with the heatsink) are as likely to heat up your MOSFETs as cool them!

Another thing missing in the calculation of temp at the temp sensor is the thermal resistance from furthest transistor to temp sensor (and assuming no other thermal sources are at work here). Heat sinks are good, but not perfect, heat conductors, and at least some of the transistor heat will be flowing past the temp sensor to other parts of the heat sink. However, this in unlikely to alter your conclusions much.

As was pointed out by others, 5W is not a great deal of power to be dissipating in a transistor, and in this case the transistor may well not be the operative thermal limit. You might consider the poor user, who certainly would not want to handle an amplifier with a 150˚C heatsink! Health and safety concerns may well dictate a much lower thermal cutoff than 150˚C, and I would suggest that at any thermal cutoff higher than 100˚C, you are "playing with fire", health and safety-wise.

Other than the above-mentioned 2 unaccounted thermal considerations, I think you're calculations are OK. You are right that you don't want to be cooking the other PCBs with a 150˚C temp, and most of your other components are likely rated at something like a max temp of 75˚C. Given that, and the H&S consideration, you might even consider setting 75˚C as your thermal cutout temp.