Electronic – How to wire up this transformer

Since transformers by their nature are bi-directional, the selection of the primary side totally depends on your input voltage and desired output voltage.

The transformer you describe likely has multiple taps on the "primary" side, may have multiple windings on the "primary" side and likely has multiple windings on the secondary side. Start with a low range DMM, and check for continuity between different leads on each side of the transformer. Once you have mapped continuity, check resistance between the same leads. You should be prepared for the transformer to be as complex as this:

The "secondary" side may be a single coil with multiple taps, or it may have multiple outputs more like the above example.

Once you've reverse-engineered the coil arrangement, you'll need to determine the turns ratio between each set of coils. I would NOT recommend your 120VAC test for this. Start with a much lower (and safer) voltage. Find a small "wall-wart" type power supply that you can sacrifice. The lower the output voltage the better. You want it just for its transformer, not the rectification and regulation components, so if you can find an AC-output wall-wart, you can use it's output as-is. What you want is a low voltage AC source that you can use to test individual windings. Note that applying a low voltage AC source to the "secondary" may result in lethal voltages on the "primary", so be careful!

Find one set of windings to apply your AC input to, and measure the resulting output on each set of coils and on each tap. Transformers are ratiometric, so the relative voltages will be the same using your low voltage AC test vs. when you identify the intended primary winding and apply 115VAC to it.

Doing this, you should have a good sense as to what windings are present that the relative turns ratio between each. Good luck!

Flyback topology is one of the common schemes for SMPS of less than a few hundred watts because of the simplicity. These equations give the first order calculation of some parameters you ask for: $$ I_{peak} = \frac{V_{in} t_{on}}{L} $$ Energy stored by the transformer per period: $$ E_{L} = \frac{1}{2}LI_{peak}^{2}$$ Power transferred: $$ P = \frac{E_{L}}{t_{period}} $$
t_period would 10us for 100KHz. t_on is the switch on time, assuming 50% duty as a starting point, it would be 5us.

1. The peak current must be less than the saturation current while accounting for input voltage variation and tolerances plus safety margin. The peak current does not translate directly to output current through the turn ratio.
2. When wired in parallel, the output windings should be capable of giving 9A.
3. When wired in series, the output windings should be capable of 3 times the voltage @ 3A.
4. For SMPS and flyback SMPS in particular, the turn ratio does not translate to actual input voltage vs output voltage.

I don't know how to say much more within a page or two. Possible resources would be some manufacturers' sites such as TI's. There are usually loads of application information. Pick SMPS ICs that seem to meet your requirements and some of them would have online calculators of recommended circuit arrangements.