Over 18 inches you would drop about 0.675 volts with 18 gauge wire. 12 gauge would drop about 0.1 volts. You would need to use 6 or 7 gauge to handle 30 A over an extended period.
If you are not using continuous current, but pulses, you could get away with using a smaller gauge wire. However an undersized wire would heat up significantly in just a few seconds. Unless you are sending pulses of less then 1 sec, you will need a large diameter wire.
AS for the insulation, it depends on how the wire is being used and how it is routed. 30 A is a lot of current. If some one can come in contact with it, you need heavy insulation for safety. Especially if it flexes. If it is total enclosed in a non conductive housing, you could use thinner insulation.
I think you have a borderline but significant core saturation problem. Using your 3rd example, the current into the primary rises at a rate of V/L where V is about 311 volts (rectified AC and smoothed) and L is 1.6 mH. So, in 5 us I would expect to see the current rise to about 1 amp.
This is based on the basic inductor formula of V = L.di/dt
5 us is the on time for a 50:50 duty cycle at a switching frequency of 100 kHz
The primary magneto motive force (MMF) is ampere turns or 1 x 95 At. But, to calculate the H field, we need the effective length of the core (57.5 mm in the data sheet linked in the question) so H = 1652 At/m.
An ungapped core would certainly be saturating but yours is gapped and has an effective permeability of around 170 compared to a permeability of around 1520 ungapped (again these were numbers I calculated from the data sheet you linked). The effect of gapping can be seen as reducing the H field so, your H field reduces to an equivalent value of around 185 At/m for an ungapped core. This allows us then to look at the published BH curve.
Now, if you look at the BH curve for N27 you will see this: -
On the two diagrams I've taken the liberty of drawing a red line that shows where the equivalent ungapped H field peak value is sitting (185 At/m). As can be seen on the left diagram (ambient of 25 degC) 185 At/m is starting to significantly saturate your core.
It's quite critical that a flyback transformer does not saturate very much.
So, as the core saturates the inductance tends to fall and instead of a linear rise in current per micro second you get a seemingly out of control rise like this: -
This may lead to a very significant rise in the peak H field and the core starts to get quite hot. But, you may say: -
So what, the controller will limit the current to that needed to store
only the energy needed to pass to the secondary load
However, as the core saturates the inductance falls so what was sufficient current (for a given value of inductance and therefore the correct amount of energy based on E = \$I^2L/2\$), now needs to be more current.
Do you see the problem and this isn't even considering what happens when the core gets warm (see the graph on the right in the picture above). At 100 degC there is even more core saturation.
I think you are running into saturation problems.
The resistance of the wire will be 12/10 = 1.2 ohms
R = rho * length / Area (where rho is the resistivity of the material chosen, remembering to convert length etc. to appropriate unit)
then Area = rho * length /1.2 this will give you the cross sectional area. Look up the X sectional area to give a wire gauge.
All this will do is give you a wire of the correct length and resistance for a given material.
The temperature the wire will 'get up to' depends on how quickly energy is lost to the environment. (Power out = power in @ equilibrium temperature).
There is no set formula for the final temperature of the wire because there are so many variables e.g. how well insulated the wire is, how it is wound (large or small coils), in air or vacuum and so on and so on ...
You have two basic options.
Experiment: Get your wire (as calculated above), feed it with 12V and see what happens temperature wise.
Devise a circuit to limit how hot the wire gets (within your desired range) by controlling the amount of power dissipated in the wire. (e.g. - fit a thermostat)