With FM you get much more than two sideband frequencies. The sidebands you describe sound more like AM (amplitude modulation) sidebands. You may want to supply a link where you get this information from or re-check your information.
Next, in your final paragraph you might be getting confused. At any one moment in time the FM signal you describe will have a frequency that is somewhere between two limits. The limits are called the deviation meaning that's how much the carrier deviates +/- from its nominal centre frequency.
That deviation is nothing to do with the modulating signal's frequency but has everything to do with the modulating signal's amplitude. The bigger the peak amplitude the bigger the deviation from the nominal centre frequency.
If the PLL is designed correctly, and its low-pass filter permits the VCO to track the carrier within the deviation limits, the low-pass filtered signal (that feeds the VCO) will represent the signal that caused the original modulation i.e. the PLL is an FM demodulator.
You also mention "keeps re-locking" - this is not something that should happen in this type of circuit - the PLL will remain locked to the modulated carrier. There will be a slight error in the instantaneous lock because you need an error to drive the mechanism that tries to maintain lock but, this error will be small and this error is not regarded as the PLL losing-lock.
You might also be getting confused with what happens when the spectral content of FM is analyzed. Yes it has several components of frequency but these do not occur together at any one instant - the spectral content is a time averaged evaluation of what the carrier is doing - moving about following the amplitude of the input modulating signal.
Have you looked at the datasheet of the 4046 PLL ? The 4046 contains both types of PD.
The type-1 PD implemented as an XOR outputs 0 when both it's input signals are equal and outputs a 1 when they are not. It cannot distinguish between both signals so it cannot detect if Fvco is too high or too low. It can only detect that it is "not the same" as Fin.
At phase = π the signal inverts so at phase = π - delta the PD's output signal is the same as what it is at phase = π + delta. This explains the positive slope changing to a negative slope at phase = π. The input signal inverts but the XOR treats it the same way, it cannot do any better !
"Why would a PLL with this type of PD not lock if the phase difference input to the PD is greater than 2π?"
Your assumption is wrong, it does lock.
Let me explain:
I give you two signals and a 4 channel oscilloscope.
At t = 0 I provide you with 3 signals:
signal A is a 1 kHz sinewave starting at phase = 0
signal B is a 1 kHz sinewave starting at phase = π
signal C is a 1 kHz sinewave starting at phase = 10 π
Now tell me which signal is which !
Think about it before reading any further !
The answer is that you can only tell me which is signal B.
You cannot distinguish signals A and C because a sinewave repeats
itself every 2π of the phase.
Like you, a type-2 PD also cannot distinguish signals which are shifted by 2π
so it will treat a phase of delta the same as a phase of delta + 2π
or delta + 4π. That is why the graph only shows 0 to 2π, the graph repeats itself every 2π just like a sinewave.
It can however distinguish a phase of π - delta from a phase of π + delta !
That is it's advantage over a type-1 PD.
For a type-2 PD it is not the absolute phase that is locked, it is the modulo(2π) of that phase and that is OK as the signal repeats.
Best Answer
If input frequency and feedback frequency (after dividing) are the same then the PLL is potentially going to fall into a state of in-lock because the phase detector doesn't care about phase differences that are multiples of 2\$\pi\$: -
As the "wandering" clock leaves perfect phase alignment with the static clock (left side), the EXOR output starts to produce thin pulses that become wider as the the wandering clock leads the static clock by greater amounts. At perfect anti-phase between the two clocks the EXOR output is a constant "1" and as the leading extends even more, the EXOR output repeats itself as the phase difference between wandering clock and static clock is 2\$\pi\$ (right side).