Yes, a square wave can be fine. When you mix, you get the sum and the difference of the frequency components from each input. The square wave consists of the fundamental and all the odd harmonics, so you need to do some filtering somewhere, of course.
The Softrock Ensamble RXTX is a simple direct conversion HF transceiver which makes a good example. There's a schematic on that page (a pretty bad one, but hey, you get what you pay for). U3 generates a square wave of a programmable frequency. U5 splits this into two quadrature clocks, QSD CLK 1
and 2. These find their way to U10, which is the mixer. It's not much more than a couple of analog switches. The output of the mixer goes to a couple of simple amplifiers and then to the audio input of a computer.
It's notable that this design wouldn't work with anything but square waves for the LO. U10 is an analog switch, with digital inputs to determine the state of the switches. It's not an ideal multiplier. If you fed it with a sine wave, the gain of the input transistors in U10 would make it a square wave anyway. This isn't true of all mixers, but here it is.
The filtering on this radio happens between the antenna and the mixer. The filters strip out all the harmonics above the band the kit was built for. Were this not done, then the LO frequency, plus all of its odd harmonics, would be aliased down to baseband. However, all those odd harmonics don't exist in the signal coming from the antenna after the filter, so there's no problem1.
You can also use this to your advantage. There's another kit in the same family, the Softrock RX Lite II, which when built for 20m or 30m, samples at a lower harmonic. That is, the LO is actually 1/3rd of what it would otherwise be, and it's the harmonic of that square wave input to the LO that actually mixes the signal down. Again, filtering removes out-of-band signals between the antenna and the mixer.
Given your difficulty in finding a suitable crystal, maybe this is good for you. If you can find one that is at 1/3rd of the desired frequency, you could make that work, with appropriate filtering.
1: provided, of course, that you don't have some really strong signal near one of these harmonics that can find its way in despite the filtering. It could be an issue if you lived right next to a broadcast station or you had to deploy this radio in a high-RF environment. It's an inexpensive hobbyist kit, not professional equipment.
Dual gate MOSFETs (tetrodes) have characteristics that make them very suitable for RF mixers. Typically RF is applied to G1 the first gate, and local oscillator is applied to G2. This maintains good isolation between oscillator and RF circuits, and low third order intermodulation (by which strong signals can wipe out nearby weak signals)
While just about any triode transistor (bipolar or FET) can be pressed into service as some sort of mixer, getting decent performance out of them is quite an art.
Better to build the circuit as designed if possible. Perhaps use a more readily available BF981 (more associated with VHF). perhaps two JFETs in a "cascode" configuration, which operates a bit like the tetrode MOSFET. (Figure1 on the Wiki : RF in to Vin, local oscillator to the grid (cough, gate) of the upper FET, though it is shown grounded in the article.
Or substitute a different mixer; such as a balanced mixer using schottky diodes.
Best Answer
No, you do not have to use that circuit at the 1dB compression point (or any other compression point) for it to work as a mixer. If you were feeding both signals into the base (or emitter), then yes, you would -- but you're not.
Done properly, that circuit should multiply input at the base by a function of the signal injected at the emitter. In the extreme of "done right" you'd inject a square wave at the desired frequency into the emitter, and the result would be a chopped and amplified version of the input.
Note that I'm not advocating for this circuit, unless you're building something that absolutely needs to be extremely inexpensive, low power drain, or small. Note also that you'd be challenged to make this work at 900MHz with the given component values -- it really wants to be designed around lower impedances.