Forget the 7812. If you're going to use a switcher to create a higher voltage first and then use the 7812 to go to 12V, you might as well use the switcher to go to 12V directly.
If the output voltage is close to the input voltage (sometimes below, sometimes above), you want a buck-boost regulator, like National's LM5118. The Webench Designer creates a design to your parameters.
You have chosen an excellent MOSFET, but your circuit is incorrect as shown.
If this is how it is actually connected it will not work.
A better datasheet for your IPP096N03L MOSFET is here
A MOSFET requires a control voltage to be applied between gate and source.
In the case of an N Channel MOSFET the gate must be more +ve than the source. In this case +4V to +5V n the gate will work well.
While your circuit is incorrect it does not explain what you are seeing. I suspect you may have drain and source reversed (or worse :-) ).
Place FET on a table, label side up, pins towards you.
Left side pin = gate.
Right side pin = source.
Middle pin (if present) and tab = Drain.
BUT - When the motor is on the source will be at 12V so the gate needs to be at 12+4 = 16V.
SO - The motor should be in the DRAIN of the MOSFET and not in the source.
The gate level also MUST be controlled at all times.
This is the original incorrect circuit with required changes shown:
Correct circuit.
Ensure pinout is correct as above.
Then:
- Remove motor from source
- Connect source to ground
- Connect motor from 12V to drain.
- Connect diode across motor as shown now.
- Consider connecting an eg 10k resistor gate to ground to ensure gate is grounded if drive is ever disconnected.
Drive with 0v/5V.
If this does not work the MOSFET is dead.
Your circuit will look more like this.
ZD1 is optional but useful for inductive loads (such as a motor). zener voltage should be higher than max drive voltage. Say a 12V zener. Optional.
R1 is not strictly necessary when playing. Say 10 ohms.
TTL gate shown here is replaced by a microcontroller in your case.
Note that this circuit works well for slow switching (maybe 10's to hundreds of Hz) but for higher switching speeds yu will need a gate driver. Simple and cheap to do but necessary at say 1 kHz up.
Best Answer
In fact, the output is not held at exactly 12 volts. If the 7812 output is at 12 volts, then depending on the current levels at the load the output voltage will be less than 12 volts, and possibly a good deal less.
EDIT - As Russell McMahon pointed out, I was ignoring the diode in the linked circuit. I was also ignoring the fact that 7812s have a nominal output in the range of 11.4 to 12.6 volts. Combining the two suggests that the base voltage will be in the range of 12.1 to 13.3 volts. Apply this as appropriate to the following. In effect, the added diode is intended to compensate for the transistor base-emitter voltage drop. And it does, but not very well. END EDIT
With the base held at 12 volts, the voltage drop of the base-emitter junction will determine the exact output voltage. Since this is a silicon transistor, the usual number is a voltage drop of ~0.7 volts. However, that number really only applies at currents around 1 to 10 mA. For 3 amps you should expect a voltage drop in the range of 1.5 to 1.6 volts. The base current supplied by the 7812 will be multiplied by the gain ($h_fe$ or $\Beta$) of the 3055, so the 7812 does not have to provide an enormous amount of current, although at maximum current the gain will drop, so you might expect a gain of 20 or so, for a base current requirement from the 7812 of 150 mA.
For an open circuit (no load), you can expect about 12 volts from this circuit. At 3 amps, something in the vicinity of 10.5 volts, or maybe a bit lower. This is not a very good circuit (but it is simple and cheap).
The other thing you need to watch out for is power dissipation. At full output and a 24 volt supply, The 2N3055 will be dropping about 13-14 volts, and will be carrying most of the current (at least 95%). So the power dissipated will be about 3 amps times 14 volts, or roughly 40 watts. If you build this, you must provide a fairly beefy heat sink for the transistor. Also note that, due to the low gain at high currents, the 7812 will dissipate significant power. To provide 150 mA of base current at a supply voltage will dissipate .15 amps times 12 volts, or 1.8 watts. This is more than a bare 7812 can handle without going into thermal shutdown, so you will also need a heatsink here as well, although not nearly as big as on the 3055.