Use a 11.1V LiPo directly on 12V electronics

If your flight controller already have its own DC/DC converter you should go for it, because it is probably designed for this particular use.

Most speed controllers with BEC are based on linear regulators. Basically these regulators turn excess voltage into heat, the more current you will draw the hotter they will be. If, for some reason, they become too hot they will go into protection mode, and stop to work.

To disable a speed controller BEC just disconnect the red wire from the from servo plug, unless your flight controller board already have a system allowing to choose the power source.

You only have two practical solutions

a) A mechanical switch

b) MOSFETs

Thyristors and SSRs are out due to their minimum 'on' voltage which would create too much heat to dissipate in your required volume.

... and if it's for system isolation, only (a) will do.

You'll note that (b) is plural. For super cheap (relatively), I expect you would be using multiple TO-220 type packages in parallel. Fortunately, MOSFETs share nicely when in parallel and fully on. With sufficiently low RDSon, you could manage the heat produced by 180A within your volume.

The industry is producing many excellent new FETs with VDS in the 70-100v region for 42v car electrics, these would be a perfect fit.

To hold them on, FETs require no power, which would allow you to design the drive circuit fairly economically. If you switch the negative rail, so have the positive available for the gate supply, you may need little more than a potential divider to stay within the max gate voltage.

Your drive may need to switch them quickly, to avoid excess dissipation if switching on or off under load.

^{simulate this circuit – Schematic created using CircuitLab}

The above might give food for thought. R1 and R2 pot the battery voltage down to 50%. The gate voltage range must be correct for the FET, usually >10v for best RDSon, <20v for gate breakdown, though check the specific FETs you end up using. LiPos should keep the terminal voltage within a 2:1 range. If you are fussy, you might reduce the value of R1 and parallel a zener with R2. Put a LED in series with your zener, or use several LEDs in series as your zener is an alternative.

Q1 and Q2 speed up charging/discharging the gate for on-load switching, omit one or both if this will never happen. I've shown a mechanical on/off switch, though electronic control is easy enough to arrange.

Switching the negative rail like this means that if you routinely ground the negative terminal of things, the battery pack and the powered electronics cannot share the same ground. There is the scope to get things wrong.

A high side switch would require some sort of voltage booster to operate the gates.