Yes, a square wave can be fine. When you mix, you get the sum and the difference of the frequency components from each input. The square wave consists of the fundamental and all the odd harmonics, so you need to do some filtering somewhere, of course.
The Softrock Ensamble RXTX is a simple direct conversion HF transceiver which makes a good example. There's a schematic on that page (a pretty bad one, but hey, you get what you pay for). U3 generates a square wave of a programmable frequency. U5 splits this into two quadrature clocks, QSD CLK 1 and 2. These find their way to U10, which is the mixer. It's not much more than a couple of analog switches. The output of the mixer goes to a couple of simple amplifiers and then to the audio input of a computer.
It's notable that this design wouldn't work with anything but square waves for the LO. U10 is an analog switch, with digital inputs to determine the state of the switches. It's not an ideal multiplier. If you fed it with a sine wave, the gain of the input transistors in U10 would make it a square wave anyway. This isn't true of all mixers, but here it is.
The filtering on this radio happens between the antenna and the mixer. The filters strip out all the harmonics above the band the kit was built for. Were this not done, then the LO frequency, plus all of its odd harmonics, would be aliased down to baseband. However, all those odd harmonics don't exist in the signal coming from the antenna after the filter, so there's no problem1.
You can also use this to your advantage. There's another kit in the same family, the Softrock RX Lite II, which when built for 20m or 30m, samples at a lower harmonic. That is, the LO is actually 1/3rd of what it would otherwise be, and it's the harmonic of that square wave input to the LO that actually mixes the signal down. Again, filtering removes out-of-band signals between the antenna and the mixer.
Given your difficulty in finding a suitable crystal, maybe this is good for you. If you can find one that is at 1/3rd of the desired frequency, you could make that work, with appropriate filtering.
1: provided, of course, that you don't have some really strong signal near one of these harmonics that can find its way in despite the filtering. It could be an issue if you lived right next to a broadcast station or you had to deploy this radio in a high-RF environment. It's an inexpensive hobbyist kit, not professional equipment.
Why are you using a VCO (variable) if you only need a fixed frequency?
In any case, according to the datasheet you linked, pin 1 should never be connected to VDD. In the VCTCXO variation, this is the control voltage, which must remain between 0.5 and 2.5 V. In the plain TCXO variation, it should either be left open or grounded.
Best Answer
One approach would be to use a simple single-chip transmitter designed for the 902-928 MHz ISM band. Here is a typical circuit:
This chip uses a PLL and internal divider to lock to a reference crystal. Power output is +23dBm (200mW) and it does not require a uC to set it up.
There are a number of app notes and a somewhat pricey eval board available from the supplier.
In answer to your question about a minimum comparison frequency for a PLL, there is no minimum, but the loop filter frequency and the lock-in time will increase proportionally to the inverse of the comparison frequency (all other things being equal), so if you compare at 100Hz vs. 10MHz it will take 100,000 times as long to lock in and stabilize.