First: Do you have a 20 MHz crystal or crystal oscillator? Those are two different things. A crystal oscillator will all on its own generate a 20 MHz clock signal for the PIC and you use the external oscillator option with it.
On the other hand, the quartz crystal is an external part of the internal oscillator and internal components together with the crystal and load capacitors make a complete oscillator. In such configuration, you use various crystal modes. Also take a look at figure 2.2 on page 27 of the datasheet.
Now to set up this part correctly, you need to understand a few things, so I'll quote the datasheet:
When the PIC18F4550 is used for USB connectivity, it must have either
a 6 MHz or 48 MHz clock for USB operation, depending on whether
Low-Speed or Full-Speed mode is being used.
You need to combine things so that the USB clock is 48 MHz or 6 MHz and then you need to set-up the microcontroller operating frequency so that it works at suitable speed. Those two clocks may be different.
On page 26 of the datasheet, you have a nice diagram which you should take time to analyze. The USB PLL input expects 4 MHz frequency which it will use to generate the 96 MHz from which it will derive the operating frequency for USB and the microcontroller.
In your screenshot, the 20 MHz are divided by 5 to get the 4 MHz needed for USB PLL which then raises that to 96 MHz, as seen in the PLL prescaler section.
Then you have the system clock postscaler section. It is currently set to use the 96 MHz created by USB PLL and divided by 2 as the main system clock. You also have other options to set the main systme clock. I can't remember exactly what they are and I've just formatted my HDD, so mikroC isn't installed yet. They should offer you to derive the system clock from an internal oscillator or directly from the clock used to generate the 4 MHz for the USB PLL or as it is shown in the screenshot from the 96 MHz generated by the USB PLL.
The point here is that you can independently select the main clock and the USB clock. For example, if you have a 20 MHz oscillator, you could run the PIC main clock at those 20 MHz and at the same time run the USB clock at needed 48 MHz.
Next you have the oscillator selection part. For real crystal oscillators, you should use EC options and connect the output of the oscillator to the OSC1/CKLI pin (in your case pin 9). You can then use the 20 MHz oscillator to drive the PIC.
In case you're using a crystal, you need to use the crystal options. They are XT, for low frequency crystals, up to 4 MHz, and HS for high frequency crystals up to 20 MHz, if I remember correctly.
As for which crystal is better, well that depends on a lot of things such as which exact crystal you're using, its characteristics, characteristics of the PLL used in the PIC and so on.
Usually low frequency crystals drift less over time and produce cleaner signal while high frequency crystals often give as their output a harmonic of some lower frequency and the signal is usually weaker. I myself would use the 4 MHz crystal here.
Also I forgot the last part of your question: In the "Oscillator frequency" field, you should enter the effective operating frequency of the PIC, that is to say the frequency the "primary clock" on figure 2.1 on page 25 of the datasheet sees. In your particular case, that would be 48 MHz.
So to sum this up: In the 20 MHz crystal case, you should first set the "oscillator selection" to HSPLL. That will give 20 MHz at the input of "primary oscillator" in the above-mentioned figure 2.1. Next, you should set the PLL prescaler to divide by 5, so you get 4 MHz which are multiplied by 24 to get the 96 MHz for USB. Next set the "USB clock selection" to 96 MHz divided by 2 and set the "System clock postscaler selection" to 96 divided by two. Finally, set the Oscillator frequency to 48 MHz and you're done with this part.
For the 4 MHz crystal, you should first set HSPLL. Set the PLL prescaler to divide by 1 and then set the "USB clock selection" to 96 MHz divided by 2 and set the "System clock postscaler selection" to 96 divided by two and set the Oscillator frequency to 48 MHz and that's it.
I don't mean to rain on your wireless parade. You've ran into a tough but unexpected requirement. Something like that warrants re-evaluation of the whole system design.
1st thing that comes to mind is to clock both units off one oscillator. You have Bluetooth communication, which hints that the range is on the order of 10m. You could connect your units with RG174 coax cable or an optical fiber, which would carry the clock.
2nd, there are precision oscillators. In order of increasing precision and cost.
- TCXO (temperature compensated crystal oscillator). 1 to 3 ppm drift, typically.
- OCXO (oven controlled crystal oscillator). Drift on the order of 0.02ppm. Some OCXO have drift down to 0.0001 ppm.
- Atomic clock (Rubidium standard, for example). I'm mentioning atomic clock mostly to give a frame of reference. More on that here.
3rd, precision oscillator trained with GPS. Every GPS satellite has several atomic clocks on board. Usually, there are plenty of GPS satellites in view. GPS is used for precision timing a lot (less known usage compared to sat nav). Most GPS receivers have a 1PPS output (one pulse per second), which provides timing accurate to 50ns.
To have a 0.5μs drift over 600s (10min), your clock (the 12MHz clock in your present design) should have drift less than 0.0008ppm. But if you can correct the timing error every so often from an low drift external source, the requirement for the drift in the clock can be more relaxed. If you can correct every second, then your clock could have a 0.5ppm drift.
Best Answer
A Phase-Locked Loop will do this.
https://en.wikipedia.org/wiki/Phase-locked_loop
The simplest phase comparator is an XOR gate. This simplest loop filter is an RC filter. The VCO should be a Voltage Controlled Crystal Oscillator (VCXO).
This circuit will cleanly transition between the clocks, no risk of glitches.
The VCXO may break your budget.
You can find PLL design equations online. Surprisingly, the circuit is easy to make stable.