I'm understanding this question to mean that you're trying to run a three-phase motor off a single-phase line. If you're trying to run the motor directly off the AC line, the phase angles involved will make it difficult to get the motor started, which is part of the reason three-phase exists in the first place. Single-phase motors usually have motor start caps for just that reason. That sounds like what you're describing.
The simple answer to your question is that to get three-phase AC from single-phase AC, you need to rectify the single-phase AC line into DC, then run the DC back through an inverter to get controlled three-phase AC. There are other electronic approaches, but they're less common in my (limited) experience. There are also mechanical approaches, which may be more convenient if you have the parts.
I'd suggest using a drive to operate your three-phase motor. Typical variable-frequency three-phase drives are exactly what I described above: a rectifier, followed by an inverter. I can't speak as to what's on the market in a given power class, but larger three-phase drives typically have terminals for the three-phase AC line input, the DC bus, and the three-phase motor output. If you have those terminals, you have two options.
One is to run single-phase AC through the three-phase input of the drive. If the voltages are correct, the drive should operate fine. The caveat is that you'll have to derate the drive somewhat. The input diodes are spec'd assuming that the drive's constant-power load will be distributed among three legs of the rectifier. If you distribute that same load over just two legs, those diodes will get hotter. The internal bus capacitors will also get hotter, because they'll see more ripple current without the third phase. Check with the drive manufacturer for the derating info.
If your drive has DC bus terminals, your other option is to skip it's internal rectifier and use an external one. Rectify the single-phase AC, then use that DC as the input to the drive. This will let you avoid derating the drive. My company makes something exactly for that purpose, though its power range may be larger than is cost-effective for your application. You'd have to price both options out to find out for sure. Read this for more details.
Tricky solution:
Measure the winding resistances. Auxiliary windings are often thinner and therefore have a higher resistance (even if it's just a little bit higher).
Once you know which terminals are used for each of the two windings, you
can connect them to the power supply and the capacitor as shown in the following picture:
The picture was taken from the book "R. Fischer: Elektrische Maschinen, 15.Auflage, Hanser Verlag München"
Cleaner solution
Apply a (small) DC voltage step to a test circuit, consisting of a resistor and one of the windings in serial combination. Capture the current with an oscilloscope. Measure the time constant of the system and calculate the inductance L of the winding. Repeat this process for the second winding. The winding with the higher inductance value should be the main winding.
Best Answer
I believe you can use a standard wiring diagram from something like a Commander SK which anticipates what you want to do. I can't remember the circuit off the top of my head. I use them for three phase motors.
You can change the phase angle by changing the capacitor. I believe the torque will be very low.
I had to recently turn a single phase motor fan into a variable fan and we used a single phase chopper like is used to slow down a (series wound) drill. It worked fine and rattled a lot at low speed. That's all. The fan has a very low start up load and at low speed the magnetic field just slips a lot - no problem.
It works by "trying" to be synchronous but the voltage is not high enough so it slips and therefore slows down. You are far better off using the correct frequency so it tries to keep up, but can't because of low power, than to try to be synchronous and go at a low frequency/speed using a motor that is inherently unable to do that.
The only issue you will have with the former case is you may want to manually switch out the starter windings if it is running below the cut out speed. It might not be necessary. Just see if it starts to overheat.