There are many losses associated with switching, but it sounds like you are most concerned about the additional thermal load introduced into the MOSFETs in the period transitioning between on and off. I thought it would be easy to find some application notes on this, but surprisingly it wasn't. The best I found was AN-6005 Synchronous buck MOSFET loss calculations with Excel model from Fairchild, the relevant parts of which I'll summarize here.
During the switching transition, the voltage and current in the MOSFET will look approximately like this:
The switching losses we are going to calculate are those in periods \$t2\$ and \$t3\$ due to the voltage and current in the MOSFET. The way to approach this is to calculate the energy of each transition, then convert this into an average power according to your switching frequency.
If you look at just \$t2\$, \$V\$ is nearly constant, and \$I\$ increases approximately linearly, forming a triangle. Thus, the power also increases linearly, and the total energy is the time integral of power. So the energy is just the area of that triangle:
$$ E_{t2} = t_2 \left( \frac{V_{in} I_{out}}{2} \right) $$
\$t3\$ also forms a triangle. In this case, the voltage is changing instead of the current, but still the power makes a triangle, and the calculation of energy is the same.
Since the calculation is the same for \$t2\$ and \$t3\$, then it's not really important how much time is spent in \$t2\$ vs \$t3\$; all that really matters is the total time spent switching. The energy losses from one switch are thus:
$$ E_{switch} = (t_2 + t_3) \left( \frac{V_{in} I_{out}}{2} \right) $$
And, your switching frequency is how many times per second you incur this energy loss, so multiplying the two together gets you the average power loss due to switching:
$$ P_{switch} = f (t_2 + t_3) \left( \frac{V_{in} I_{out}}{2} \right) $$
So, taking your calculation of the switching period being \$150ns\$, and the maximum current being \$330A\$, and the voltage \$12V\$, and the switching frequency \$30kHz\$, the power losses from switching are:
$$ 30kHz \cdot 150ns \left( \frac{12V \cdot 330A}{2} \right) = 8.91W $$
That's \$8.91W\$, ideally, shared between three transistors, so only about \$3W\$ each, which is pretty insignificant compared to your other losses.
This number can be checked for sanity with a simpler model: if you spent \$150ns\$ switching, and you do it \$30000\$ times per second, then you can calculate the fraction of the time you spend switching, and make the most pessimistic assumption of the full power of \$12V\cdot330A\$ being lost in the transistors:
$$ \require{cancel}
\frac{150 \cdot 10^{-9} \cancel{s}}{\cancel{switch}}
\frac{30 \cdot 10^3 \cancel{switches}}{\cancel{s}}
\cdot 12V
\cdot 330A = 17.82W $$
Of course, over the switching period, the average current and voltage is only half that of the maximum, so the switching losses are half this, which is what we just calculated.
However, I bet in practice, your switching times will be slower. A "\$2A\$ gate driver" isn't a constant current source as these calculations assume. The real picture is rather more complicated than this simple model. Additionally, the current will be limited by the resistance, and usually more significantly, the inductance of the transistor packages and the traces leading to them.
Let's just say the inductance of the gate driver, transistor package, and traces to it is \$1\mu H\$. If your gate drive voltage is \$12V\$, then \$di/dt\$ is limited to \$12V/1\mu H = (1.2\cdot 10^7)A/s\$. This may seem like a lot, but on the time scale of \$150ns\$, it's not. Keeping the inductance low will take some very careful layout.
So, I would say that these calculations show that your switching losses may be manageable, though you won't know for sure until you've made the layout and tested it. Even if you can't reach the ideal of a \$150ns\$ switching time, the losses are low enough relative to your other problems that you have some margin to do worse and still function.
Your bigger problem is probably getting the three MOSFETs to switch at the same time. Otherwise, one of them will get a disproportionate share of the total current, and thus heat, leading to premature failure.
Going by the picture (if that's the amplifier under question), that's a pretty large heatsink for 4 MOSFETs dissipating 5W each. It looks like there's other heat sources attached to the heatsink, however, and although you don't say what these are dissipating, or what the thermal impedences from other heat sources to MOSFETs-in-question are, I'm tempted to say the they (in combination with the heatsink) are as likely to heat up your MOSFETs as cool them!
Another thing missing in the calculation of temp at the temp sensor is the thermal resistance from furthest transistor to temp sensor (and assuming no other thermal sources are at work here). Heat sinks are good, but not perfect, heat conductors, and at least some of the transistor heat will be flowing past the temp sensor to other parts of the heat sink. However, this in unlikely to alter your conclusions much.
As was pointed out by others, 5W is not a great deal of power to be dissipating in a transistor, and in this case the transistor may well not be the operative thermal limit. You might consider the poor user, who certainly would not want to handle an amplifier with a 150˚C heatsink! Health and safety concerns may well dictate a much lower thermal cutoff than 150˚C, and I would suggest that at any thermal cutoff higher than 100˚C, you are "playing with fire", health and safety-wise.
Other than the above-mentioned 2 unaccounted thermal considerations, I think you're calculations are OK. You are right that you don't want to be cooking the other PCBs with a 150˚C temp, and most of your other components are likely rated at something like a max temp of 75˚C. Given that, and the H&S consideration, you might even consider setting 75˚C as your thermal cutout temp.
Best Answer
From the data sheet Fig. 2, you can see a curve at different VGS values. Using the lowest curve (3.0 V), you can see it crosses 2 A at 0.2 V (it's a log-log scale). Therefore the on-resistance is 0.2/2 = 0.1 ohm. This is at +150 C, which is also the maximum allowed junction temperature (this is quite hot and would burn your finger if you touched it). Note that this is actually a typical plot, not a guaranteed worst case value, but in practice FETs are within 20 % of this value. Therefore let's assume 0.12 ohm.
If you don't have a real heatsink, it might be reasonable to assume a thermal resistance of 100 deg/W (with it mounted on a large sheet of metal, it might be 30 deg/W or less). Assuming an ambient of 60 deg (also very warm), and allowing up to 150 C junction, you can have a temperature rise of 150-60 = 90 deg. At 100 deg/W, this allows 90/100 = 0.9 W dissipation.
Using P = I^2*R, and 0.12 ohm gives a current of sqrt(0.9/0.12), or just over 2 A.
The 12 V LED strip value doesn't directly matter (as long as the FET's max VDS rating exceeds this which it does).
In cases like this, it's always better to be on the safe side and round calculations to the 'safe' side.