Seems to me you are driving the transformer with a current pulse that should turn one output transistor on and the other off, via the opposing secondaries. Then there is an 'off' period on the primary side, to give you the duty cycle control, and then you reverse the primary current to switch the output transistors in the opposite sense, then off again, and repeat ?
Q1 and Q2 need volts of drive to switch them, and that means D2 and D3 will be short-circuiting the transformer on each negative - going output pulse. If you want to protect the FETs from voltage spikes, or limit the negative gate charge to speed up switching, use 2 x 10V zener diodes back-to-back, for each of D2, D3.
That should make a dramatic improvement, but if still unsatisfactory, do you have a part number for the core, or have you measured its characteristics? The problem may be the core you have chosen. Toroids in power supplies have widely differing characteristics, depending on the frequency and circuit they are designed for. Older switching PSU may operate at 50-80kHz, and the manufacturer isn't going to fit a core that is characterised for low loss at 200kHz if it isn't needed. Worse still, if the toroid was part of a filter circuit it may be intentionally lossy at high frequencies, to dissipate energy that would otherwise be radiated or conducted out.
Toroids may also have a built-in distributed "air gap" - the equivalent of a gapped E-core, - made by binding ferromagnetic particles in a non-magnetic matrix, to prevent saturation caused by a DC component of the winding current. Such cores have a much lower permeability than a solid ferrite or iron dust type.
Finally, regarding waveform distortion, once the shunt diodes are removed the transformer load is 1K and 1300pF. That will resonate with winding inductances, so you may have ringing and voltage spikes to contend with. Core and winding design influence that, too. To limit them you may need a zener clamp or RC snubber on the primary of the transformer, but that will introduce additional losses.
The power limit isn't referring to losses or efficiency. It is referring to the point at which the core saturates. Under load, there is still a point where the core saturates, and so no more power can be transferred using the magnetic field.
This power handling capacity is proportional to the amount of iron in the core, and so the power rating is roughly proportional to its weight.
Best Answer
Bmax is based on the residual current that flows in the primary irrespective of load. So, it's not based on power-throughput but it is based on current (but not load-current).
Imagine you stripped away the secondary and drove the primary winding. You have primary inductance and when you apply a voltage to that inductance, current rises linearly (hopefully). That rising current is determined by: -
$$\dfrac{di}{dt} = \dfrac{V}{L}$$
That rising current produces a rising flux.
Then when you activate the other transistor, an equal value of inductance is connected that linearly ramps down the flux to a negative value and the process repeats: -
The average level of flux is zero and it peaks at negative and positive values. That is what you ideally want. But a magnetic core isn't perfect and it does saturate and that starts to produce anomalies in the current so you try and keep these anomalies at bay but you do have to accept some saturation.
Take Ferroxcube 3F3 material (just as an example): -
Picture source.
The material hard saturates at about 400 mT (at 25 °C) and it will get hot if driven at this level (see red line in picture). As it warms the saturation level drops to about 340 mT (at 100 °C) and things just carry on getting hotter. This must be avoided.
So, instead of driving it at 400 mT consider driving it at 200 mT (green line). Now it is fairly temperature stable should temperature rise due to (say) copper loss. You can also say that maybe 300 mT is a good level but I initially tend to stick at 200 mT for this type of material.
Getting back to the root problem; the primary inductance is driven with a voltage and will produce a triangle wave of current (and flux) and, the peak of that current is the point at which flux is highest (and hence flux density is highest).
Note that in the BH diagram above, the H value is about 23 ampere turns per metre to deliver 200 mT.
And this is now getting towards solving the issue of how many turns are needed. If you chose a core type that has an effective length of 150 mm and an inductance factor (\$A_L\$) of 2000 nH/t and initially considered 18 uH to be an adequate inductance then you will need three turns because one turn gives 2 uH and inductance is proportional to turns squared hence 3 turns gives 18 uH.
With 18 uH and (say) 12 volts applied over half the switching period (dt), current (di) is 3.333 amps. That value is the peak current both positively and negatively. So the peak ampere-turns is 3.333 x 3 = 10 At.
To convert to H you divide ampere-turns by the length of the core so now H is 10/0.15 = 66.7 At/m and too big for the flux density limit of 200 mT. You need to change inductance to a higher value so maybe you go for 6 turns (an inductance of 72 uH).
I should say that I'm explaining a process rather than giving a solution that is suitable for the OP....
The peak current will be 0.8333 amps and the ampere-turns will be 5 At. The H field will be 5/0.15 = 33 At/m and will probably result in a peak flux density of about 240 mT and this will likely be OK.
So, in summary: -
For one turn it tells you how much inductance you get and you multiply that by turns squared to give you total inductance.
If you can fit 6 turns of thick copper wire (and don't underestimate how thick this needs to be to deliver nearly 100 amps of load current AKA 1000 VA from a 12 volt source) AND you can fit the secondary winding then this will work providing the copper losses are not so excessive to cause problems.
And, of course, with the split primary design you have, you'll need 6 + 6 primary turns because only one half is active at any time. I have done a similar design for about 200 watts and to keep efficiency as high as I could, I hard switched the primary as per your design but the centre tap I fed from a 95% efficient synchronous buck converter to do the regulation.
I'm not saying that this is the best route to go for you but I am saying that for my design this suited what I needed to achieve AND I ran at an operating frequency much higher than 100 kHz i.e. 600 kHz. Running higher than 100 kHz is probably what you'll need to do so maybe consider 250 kHz and choose a slightly more exotic ferrite material such as 3F3.
Choosing a core can be very difficult and a bit of try-this, try-that theoretically is par for the course.