I assume this is a 2-stroke engine.
So you want to use the stepper to deliver power, and hence maintain a relatively constant angular velocity as the power stroke finishes, and continue through compression until the IC engine fires and generates power again.
That should be two fixed points on each rotation, though it might take a bit of care finding where those points are (hence, partly, my comment lower-down about using a high-resolution sensor)
You could sense those positions with two Hall switches and magnets attached to the shaft. That is how some motor vehicle engines sense shaft position.
Hall effect sensors should be good for more than 1000 rps, e.g. 60,000 rpm.
Most reasonable microcontrollers could track 4,000rpm with much better than 0.1% accuracy.
However, driving the stepper, with only 12 steps might be tricky to set up, and drive. 12 steps is 30 degrees per step, which is quite a lot compared to the motor's cycle. This sounds more like a BLDC motor than a stepper motor.
Even with 8 step micro-stepping, the angle is quite big. AND, 8hp is about 6kW+, which is quite a lot of power to switch and control.
Further, to maintain near-maximum torque, the movement of the magnetic field needs to track the motor's rotation reasonably accurately. I'd be tempted to go for 'overkill' and use a high resolution rotation sensor. That might be Hall Effect, like something from AustriaMicroSystems (AMS), or something optical.
Edit:
Texas instruments (TI) have some useful documentation and videos on 'Feld Oriented Control' (FOC) for BDC motors which may help. A web search will find this stuff.
TI have some affordable (sub $100) development boards for low-power (10W?) control too, as does ST Micro, and I'm sure, others. There are 'fast/easy start development kits' for motor control. I haven't used them, but they claim to have control software 'ready to go'.
Summary:
Sensing the shaft position for the IC engine might be a relatively easy part of the project.
It all comes down to inductance. But first lets recap on motors.
You can think of a stepper motor as essentially two independent L-R circuits - one for each phase. The inductor represents the windings (windings = coil = inductor), and a resistance which represents a mixture of resistive losses (winding resistance, etc.) and mechanical power. For maximum torque you want to get as much mechanical power out of the motor as possible, which means you want the "resistor" in the L-R circuit to dissipate as much power as possible.
However you have an inductor in the circuit, what effect does that have? Recall that the voltage across an inductor is given by:
$$v_L = L \frac{\mathrm{d}i}{\mathrm{d}t}$$
In other words, the faster you try to change the current, the higher the voltage drop of an inductor.
In our simple motor model, the supply voltage is shared between the inductance and the resistance. The higher the voltage drop of the inductor, the lower the voltage drop of the resistance, and hence the lower the power output of the motor (and by extension torque). So if you try to change the current faster and faster, more and more voltage is dropped across the inductance, and so there is less power output from the motor.
We can see this in the diagram below which shows the voltage across the inductor and current through the resistor vs. time - notice how it takes time for the peak current to be reached. You can read more here.
Image Source
Now recall what happens when the motor is moving. Unlike a DC motor, with a stepper we are sequencing the windings - turning them on and off and/or reversing the polarity. Clearly each time we 'step', the current will change in winding, which as we have seen above reduces the output power.
If the speed is low (or stopped), the current has time to rise to its peak before the next step. However if the speed is high, the step time shorter than \$5\tau\$ in the above diagram, the current doesn't have time to reach its peak and hence the peak power output drops. And this is where your problem lies.
With such a high gear ratio, the motor has to spin very fast compared to the output, and so you lose a lot of torque in the process - an at some point you lose more torque due to the switching speed than you gain from the gearbox.
So how do we deal with the problem? Well, modern stepper motor drivers use a much higher voltage than the motor is rated for, but they ensure that the current is limited to within the motors ratings. It does this using PWM (a chopper circuit). This allows the voltage drop across the "resistance" to reach its peak much faster as there is more voltage shared between the inductor and resistor. As a result, you can run the motor faster while still maintaining torque.
The current limiting is done by measuring the current through the coil and turning the supply off just before it goes too high for the motor. The current will then slowly start decaying again, at which point the supply is turned on again, and so on - essentially PWM, with the duty cycle being adjusted to ensure the current is limited to safe levels.
Drivers such as the DRV8825 and A4988 which are common amongst hobbyists use this technique to allow for higher top speeds.
Best Answer
You need a variable-reluctance step motor. They are available from 24 to 200 steps/revolution. With zero excitation, they spin freely. There is no permanent magnet.