There are many losses associated with switching, but it sounds like you are most concerned about the additional thermal load introduced into the MOSFETs in the period transitioning between on and off. I thought it would be easy to find some application notes on this, but surprisingly it wasn't. The best I found was AN-6005 Synchronous buck MOSFET loss calculations with Excel model from Fairchild, the relevant parts of which I'll summarize here.
During the switching transition, the voltage and current in the MOSFET will look approximately like this:
The switching losses we are going to calculate are those in periods \$t2\$ and \$t3\$ due to the voltage and current in the MOSFET. The way to approach this is to calculate the energy of each transition, then convert this into an average power according to your switching frequency.
If you look at just \$t2\$, \$V\$ is nearly constant, and \$I\$ increases approximately linearly, forming a triangle. Thus, the power also increases linearly, and the total energy is the time integral of power. So the energy is just the area of that triangle:
$$ E_{t2} = t_2 \left( \frac{V_{in} I_{out}}{2} \right) $$
\$t3\$ also forms a triangle. In this case, the voltage is changing instead of the current, but still the power makes a triangle, and the calculation of energy is the same.
Since the calculation is the same for \$t2\$ and \$t3\$, then it's not really important how much time is spent in \$t2\$ vs \$t3\$; all that really matters is the total time spent switching. The energy losses from one switch are thus:
$$ E_{switch} = (t_2 + t_3) \left( \frac{V_{in} I_{out}}{2} \right) $$
And, your switching frequency is how many times per second you incur this energy loss, so multiplying the two together gets you the average power loss due to switching:
$$ P_{switch} = f (t_2 + t_3) \left( \frac{V_{in} I_{out}}{2} \right) $$
So, taking your calculation of the switching period being \$150ns\$, and the maximum current being \$330A\$, and the voltage \$12V\$, and the switching frequency \$30kHz\$, the power losses from switching are:
$$ 30kHz \cdot 150ns \left( \frac{12V \cdot 330A}{2} \right) = 8.91W $$
That's \$8.91W\$, ideally, shared between three transistors, so only about \$3W\$ each, which is pretty insignificant compared to your other losses.
This number can be checked for sanity with a simpler model: if you spent \$150ns\$ switching, and you do it \$30000\$ times per second, then you can calculate the fraction of the time you spend switching, and make the most pessimistic assumption of the full power of \$12V\cdot330A\$ being lost in the transistors:
$$ \require{cancel}
\frac{150 \cdot 10^{-9} \cancel{s}}{\cancel{switch}}
\frac{30 \cdot 10^3 \cancel{switches}}{\cancel{s}}
\cdot 12V
\cdot 330A = 17.82W $$
Of course, over the switching period, the average current and voltage is only half that of the maximum, so the switching losses are half this, which is what we just calculated.
However, I bet in practice, your switching times will be slower. A "\$2A\$ gate driver" isn't a constant current source as these calculations assume. The real picture is rather more complicated than this simple model. Additionally, the current will be limited by the resistance, and usually more significantly, the inductance of the transistor packages and the traces leading to them.
Let's just say the inductance of the gate driver, transistor package, and traces to it is \$1\mu H\$. If your gate drive voltage is \$12V\$, then \$di/dt\$ is limited to \$12V/1\mu H = (1.2\cdot 10^7)A/s\$. This may seem like a lot, but on the time scale of \$150ns\$, it's not. Keeping the inductance low will take some very careful layout.
So, I would say that these calculations show that your switching losses may be manageable, though you won't know for sure until you've made the layout and tested it. Even if you can't reach the ideal of a \$150ns\$ switching time, the losses are low enough relative to your other problems that you have some margin to do worse and still function.
Your bigger problem is probably getting the three MOSFETs to switch at the same time. Otherwise, one of them will get a disproportionate share of the total current, and thus heat, leading to premature failure.
Yup, that's the way MOSFET datasheets work. The maximum current rating really means "This is the maximum current you can ever possibly get thru this thing, if you were to somehow not violate other specs in the process, although we have no idea how to do that. We put this here because we think it's cool, and maybe someone is dumb enough to buy a truckload of them before realizing they can't actually run the part at this value for any set of real world conditions".
Basically, each of the limits of the device are specified separately. You have to look at what you're doing and carefully check each one. The real limit on current is usually die temperature. To check that, look at the max Rdson for your gate drive level, compute the dissipation due to your current, multiply that by the die to ambient thermal resistance, add that to your ambient temperature, and compare the result to the maximum die operating temperature. When you figure all this backwards to find the maximum current the device can take before overheating, you'll usually find that's well below the absolute maximum current spec.
Best Answer
Going by the picture (if that's the amplifier under question), that's a pretty large heatsink for 4 MOSFETs dissipating 5W each. It looks like there's other heat sources attached to the heatsink, however, and although you don't say what these are dissipating, or what the thermal impedences from other heat sources to MOSFETs-in-question are, I'm tempted to say the they (in combination with the heatsink) are as likely to heat up your MOSFETs as cool them!
Another thing missing in the calculation of temp at the temp sensor is the thermal resistance from furthest transistor to temp sensor (and assuming no other thermal sources are at work here). Heat sinks are good, but not perfect, heat conductors, and at least some of the transistor heat will be flowing past the temp sensor to other parts of the heat sink. However, this in unlikely to alter your conclusions much.
As was pointed out by others, 5W is not a great deal of power to be dissipating in a transistor, and in this case the transistor may well not be the operative thermal limit. You might consider the poor user, who certainly would not want to handle an amplifier with a 150˚C heatsink! Health and safety concerns may well dictate a much lower thermal cutoff than 150˚C, and I would suggest that at any thermal cutoff higher than 100˚C, you are "playing with fire", health and safety-wise.
Other than the above-mentioned 2 unaccounted thermal considerations, I think you're calculations are OK. You are right that you don't want to be cooking the other PCBs with a 150˚C temp, and most of your other components are likely rated at something like a max temp of 75˚C. Given that, and the H&S consideration, you might even consider setting 75˚C as your thermal cutout temp.