I am designing a discontinous conduction mode flyback converter, 100kHz using UC3844 IC with the spec. 220V,50Hz to 5V 5A. The issue is that after testing it for an hour, the transformer starts heating up (tried with different cores.)
Ambient temperature: 27degC
Transformer specifications:

1st try: EE20/10/6 core, 0.5mm gap, Al=100nH/turns2 (EE20/10/6 Datasheet)

Turns ratio: 21 to keep Vds(max) low

Primary: 1.7mH, 30 SWG, 130 turns, 1 strand

Secondary: 56uH, 6 turns, 30 SWG, 7 strands (I know, the diameter that I used is very low but could fit only this in the winding area and the strands have to be increased, for testing purpose designed this)

Auxiliary: 52uH, 14 turns,30SWG, 1 strand.
With this configuration, the transformer was heating up to 80deg C in just 30 min.
From the calculations for the core loss, it was coming to 170mW, but copper losses were huge
Primary: Irms=0.4A, Resistance_dc=(total length)Res. per length, Rp=(7.64130)0.221 mOhm=873mOhm, P=0.40.40.873*1.5=0.2W.
Secondary: Irms=9A, R_dc=(7.646/5)0.221=8mOhm, P=99*.008*1.5=0.972W(This is the prob)
Seeing this with the thermal resistivity of 50K/W, the temperature would rise to almost 80 degC. So, it confirms the problem.
I could not fit more diameter winding on this bobbin, I had to increase the core size.
2nd try: EE25/13/7 0.2mm gap Al=290nH/turns2 EE25/13/7
 Same inductance parameters.
 Primary: 77turns, 30 SWG, 1 strand
 Secondary: 22SWG, 5 strand, 4 turns.
But even then it is still rising to 6570degC.
Currently working at 78% efficiency.
 What is happening in this case?
 Are my previous calculations correct?
 Is it because AC resistance has increased due to skin effect since operating at 100 Khz because of 22SWG wire, or something else?
Edit:
3rd Try:
EE25/13/7 core: 0.4mm gap, 187nH/turns2 [earlier(340nh/turns2)]
 Primary: 30SWG, 1 strand, 95 turns, 1.6mH
 Secondary: 22SWG, 5 strand, 5 turns, 5.2mH
 Auxiliary: 35 SWG, 1 strand, 16turns.
Although the temp decreased but still hovering around 65degC.
Circuit diagram:
Best Answer
I think you have a borderline but significant core saturation problem. Using your 3rd example, the current into the primary rises at a rate of V/L where V is about 311 volts (rectified AC and smoothed) and L is 1.6 mH. So, in 5 us I would expect to see the current rise to about 1 amp.
This is based on the basic inductor formula of V = L.di/dt
5 us is the on time for a 50:50 duty cycle at a switching frequency of 100 kHz
The primary magneto motive force (MMF) is ampere turns or 1 x 95 At. But, to calculate the H field, we need the effective length of the core (57.5 mm in the data sheet linked in the question) so H = 1652 At/m.
An ungapped core would certainly be saturating but yours is gapped and has an effective permeability of around 170 compared to a permeability of around 1520 ungapped (again these were numbers I calculated from the data sheet you linked). The effect of gapping can be seen as reducing the H field so, your H field reduces to an equivalent value of around 185 At/m for an ungapped core. This allows us then to look at the published BH curve.
Now, if you look at the BH curve for N27 you will see this: 
On the two diagrams I've taken the liberty of drawing a red line that shows where the equivalent ungapped H field peak value is sitting (185 At/m). As can be seen on the left diagram (ambient of 25 degC) 185 At/m is starting to significantly saturate your core.
It's quite critical that a flyback transformer does not saturate very much.
So, as the core saturates the inductance tends to fall and instead of a linear rise in current per micro second you get a seemingly out of control rise like this: 
This may lead to a very significant rise in the peak H field and the core starts to get quite hot. But, you may say: 
However, as the core saturates the inductance falls so what was sufficient current (for a given value of inductance and therefore the correct amount of energy based on E = \$I^2L/2\$), now needs to be more current.
Do you see the problem and this isn't even considering what happens when the core gets warm (see the graph on the right in the picture above). At 100 degC there is even more core saturation.
I think you are running into saturation problems.