I've just recently started studying RF circuit design, thereby my help might be of limited use. What we can infer from your problem description is essentially that connecting the mixer alters your gain; this makes sense as the latter is given by the gm times load product of the circuit. In your case the load is modified by the introduction of the common base stage of the mixer. This part essentially behaves as current follower (linearized by R19, but let's ignore that for the analysis), thereby your new gain is to expected to be somewhere around the gm of Q1 times the load of the mixer! As for the introduction of the undesired frequencies: I have to give that some more thought, without a rigorous mathematical approach I believe pinpointing the exact reason will be difficult.
A true multiplicative mixer produces components at both the sum and difference of the frequencies (real mixers will usually produce components at other frequencies too but we can ignore that for now)). Generally filters will be arranged so that only one of those components passes through to the next stage.
Frequencies (in this context) are always positive, so difference here means absoloute value of the difference.
In this case the filters are not shown, but from the frequencies labeled on the diagram they are clearly always taking the difference and filtering out the sum.
So writing this out as equations.
$$f_\mathrm{if1} = |f_\mathrm{sig}-f_\mathrm{lo1}|$$
$$f_\mathrm{if2} = |f_\mathrm{if1}-f_\mathrm{lo2}|$$
$$f_\mathrm{if3} = |f_\mathrm{if2}-f_\mathrm{lo3}|$$
If we combined these equations together and tried to solve them we would end up with 8 possible soloutions for \$f_\mathrm{sig}\$ in terms of \$f_\mathrm{if3}\$. Which of those 8 soloutions is correct? well we need to go back to our diagram.
$$f_\mathrm{sig} < f_\mathrm{lo1} \rightarrow f_\mathrm{if1} = f_\mathrm{lo1} - f_\mathrm{sig}$$
$$f_\mathrm{if1} > f_\mathrm{lo2} \rightarrow f_\mathrm{if2} = f_\mathrm{if1} - f_\mathrm{lo2}$$
$$f_\mathrm{if2} > F_\mathrm{lo3} \rightarrow f_\mathrm{if3} = f_\mathrm{if2} - f_\mathrm{lo3}$$
Combing these equations.
$$ f_\mathrm{if3} = ((f_\mathrm{lo1} - f_\mathrm{sig}) - f_\mathrm{lo2})- f_\mathrm{lo3} $$
Getting rid of the brackets.
$$ f_\mathrm{if3} = f_\mathrm{lo1} - f_\mathrm{sig} - f_\mathrm{lo2}- f_\mathrm{lo3} $$
Rearrange to make \$f_\mathrm{sig}\$ the subject.
$$ f_\mathrm{sig} = f_\mathrm{lo1} - f_\mathrm{lo2}- f_\mathrm{lo3} - f_\mathrm{if3}$$
Which is equvilent to
$$ f_\mathrm{sig} = f_\mathrm{lo1} - (f_\mathrm{lo2} + f_\mathrm{lo3} + f_\mathrm{if3})$$
P.S curiously the diagram labels the first Lo as 5.1 to 8.7 GHz but to get the frequencies shown in the rest of the diagram would require a first LO of 8.7225 GHz.
Best Answer
A RF mixer doesn't actually multiply two signals, at least not directly. Instead, the signals are first added and then passed through a nonlinear circuit, which ideally has a quadratic relationship between input and output.
We can therefore represent the mixer by a function:
And putting the sum of two signals into that function gives us:
This clearly contains the original input signals. Only the product of the two input signals is desired, the rest has to be filtered out.
Most mixers produce even more frequencies than those because they're not purely quadratic but also have summands with an even higher exponent. Third-order is especially nasty to filter out.
A typical mixer uses a diode as its nonlinear element, which has (roughly) the following function, ignoring some constant factors:
Computing its taylor series, we get:
This clearly has b != 0, so we get some of the original input signal at the mixer's output if we don't filter it out.