Electronic – MOSFET package with lower thermal resistance than D2PAK

To directly answer your first question: No, not even close. A little more copper around a surface mount part isn't going to get rid of 11W of heat. No way.

One answer may be to parallel multiple FETs. Not only does that cut the total dissipation by the number of parts, but the dissipation on each FET is reduced by the square of the number of parts. So if one FET dissipates 10W, then two parallel FETs would dissipate 5W total, and each FET would dissipate only 2.5W.

That's in theory. In practise they won't share the load exactly equally, so you have to design for a bit worse than that per FET. The good thing about paralleling FETs is that they have a positive temperature coefficient. The Rdson goes up with temperature. This helps them ballance somewhat and prevents runaway of a single part, like could be the case with bipolar transistors.

Ultimately you have to decide what you really want. Switching 40A is going to make some heat. One way or the other you're going to have to deal with that. You can tell us you're space contrained all you want, but ultimately the physics will dictate a certain amount of space, surface area, forced cooling, or whatever. It may not be possible to meet all the constraints. Not all combinations of small size, high current, and low cost are possible.

Just a fan blowing over the PCB won't be sufficient; a TO-263 doesn't have good thermal contact with the air surrounding it. I would solder this

heatsink over the FET. Like I said in another answer earlier today an SMD loses much more heat through conduction than through convection, but the heatsink has better convection loss thanks to its larger surface. The heatsink mounts over the SMD, not on it.

Heat transfer will occur via the PCB's copper, so increase the TO-263's pad so that the heatsink shares it.