Look at the spec for the device. There are two types; one switches at 1.6MHz and the other switches at 600kHz. Let's say, just to make life easier on folk reading this answer that it switches at 1MHz.
How much energy can it charge the inductor with - again the spec has the answer - maximum duty cycle is (on average between the two devices) about 90%.
For the sake of mathematical convenience lets call it 1\$\mu s\$ (90% of 1MHz period is still about 1\$\mu s\$).
\$E = L\frac{di}{dt}\$ and this means \$ di = dt\frac{E}{L}\$
di is how much current the inductor is taking when the maximum on-time is reached (1us). If E = 9V and L = 33\$\mu\$H, then
di = \$ 1\mu s \times \frac{9}{33e^{-6}} = 273mA\$.
Is this current going to supply the modem when it is taking 1.2A? No
What if the inductor was lowered to (say) 4.7uH? Current would be 9/4.7 which is approximately 2A however, the internal FET is only rated at 1.8A so it looks like you need to find a part that has more muscles.
EDIT assuming better switcher and 1.7\$\mu H\$ inductor (revised due to error)
The power output requirement is about 13W and if the switcher switches at a 1MHz rate this means an energy transfer per \$\mu\$s of 13\$\mu\$J. Knowing that this energy comes from the inductor means we can calculate peak current in inductor and its duty cycle.
Energy in inductor is = \$\frac{LI^2}{2} \therefore\$ peak current is \$\sqrt{\frac{2 \times 13e^{-6}}{1.7e^{-6}}}\$ which equals just about 4A. But, the topology of this type of switcher means that the inductor is only needed to transfer enough energy to raise the output level above the input voltage level. In other words the first 6V are a given.
The power needed by the load (above the 6V level) is \$1.2A \times (10.5-6)V = 5.4W\$ and this means the inductor "charge" current is 2.52A.
How long will the inductor be "charging" for?
V = \$L\frac{di}{dt}\$ - we know V (6V minimum), L (1.7uH) and di (2.52A) therefore dt is \$\frac{1.7e^{-6}\times 2.52}{6}\$ = 0.714us or a duty of 71.4% and this seems reasonable.
So, the input voltage is 5V and you're trying to get 5V output?
If that is the case, a boost isn't going to work out for you. A boost can only output a higher (or equal) voltage than the input. In fact it can't supply anything less than the input voltage (minus the diode drop), since current from the input will pass through L1 and D1 to Vout. This is true always. If the switch is switched at some duty cycle, then higher voltage will show up on the output. If Vin ranges both higher and lower than Vout, you will have to use a buck-boost or sepic converter.
You are correct that with no (or very light load) a very high voltage can show up on the output. Look at the circuit from pin 5 of the AD1613 to Vout. There is just a diode and a capacitor. A peak detector. So, without load, even a whisper thin PWM to the switch will put energy into the inductor that will transfer to Cout with nowhere to go until a leakage path is found. This is why it is common to put a zener diode, rated a little higher than Vout, across Cout.
Best Answer
Solderless breadboards are useless for prototyping circuits over a few hundred kHz. For high frequencies, you need to construct your prototype circuits "deadbug" style over a copper clad ground plane, like so: