Electrical resistance converted to wire temperature

Over 18 inches you would drop about 0.675 volts with 18 gauge wire. 12 gauge would drop about 0.1 volts. You would need to use 6 or 7 gauge to handle 30 A over an extended period.

If you are not using continuous current, but pulses, you could get away with using a smaller gauge wire. However an undersized wire would heat up significantly in just a few seconds. Unless you are sending pulses of less then 1 sec, you will need a large diameter wire.

AS for the insulation, it depends on how the wire is being used and how it is routed. 30 A is a lot of current. If some one can come in contact with it, you need heavy insulation for safety. Especially if it flexes. If it is total enclosed in a non conductive housing, you could use thinner insulation.

I think you have a borderline but significant core saturation problem. Using your 3rd example, the current into the primary rises at a rate of V/L where V is about 311 volts (rectified AC and smoothed) and L is 1.6 mH. So, in 5 us I would expect to see the current rise to about 1 amp.

This is based on the basic inductor formula of V = L.di/dt

5 us is the on time for a 50:50 duty cycle at a switching frequency of 100 kHz

The primary magneto motive force (MMF) is ampere turns or 1 x 95 At. But, to calculate the H field, we need the effective length of the core (57.5 mm in the data sheet linked in the question) so H = 1652 At/m.

An ungapped core would certainly be saturating but yours is gapped and has an effective permeability of around 170 compared to a permeability of around 1520 ungapped (again these were numbers I calculated from the data sheet you linked). The effect of gapping can be seen as reducing the H field so, your H field reduces to an equivalent value of around 185 At/m for an ungapped core. This allows us then to look at the published BH curve.

Now, if you look at the BH curve for N27 you will see this: -

On the two diagrams I've taken the liberty of drawing a red line that shows where the equivalent ungapped H field peak value is sitting (185 At/m). As can be seen on the left diagram (ambient of 25 degC) 185 At/m is starting to significantly saturate your core.

It's quite critical that a flyback transformer does not saturate very much.

So, as the core saturates the inductance tends to fall and instead of a linear rise in current per micro second you get a seemingly out of control rise like this: -

This may lead to a very significant rise in the peak H field and the core starts to get quite hot. But, you may say: -

So what, the controller will limit the current to that needed to store only the energy needed to pass to the secondary load

However, as the core saturates the inductance falls so what was sufficient current (for a given value of inductance and therefore the correct amount of energy based on E = \$I^2L/2\$), now needs to be more current.

Do you see the problem and this isn't even considering what happens when the core gets warm (see the graph on the right in the picture above). At 100 degC there is even more core saturation.

I think you are running into saturation problems.