The MCU maker is likely at fault. There is absolutely no excuse for not designing a modern MCU RTC oscillator to reliably function with any typical commercially available 32kHz crystal.
Unfortunately, the opposite is much more common, as you have already discovered - in your case the MCU data sheet fails to mention that 6pF load capacitance does not work.
The root issue is that you are dealing with a system of two components, made by two different manufacturers. One of them speaks silicon and the other speaks quartz, and they have never properly agreed how to tell designers how their products reliably work together.
So, as you have found out, the crystal oscillator can be a trap for the unweary. I have seen a major automotive production line grind to a standstill because of crystal oscillator startup issues!
Anyways, to get to your question of WHY, there are four important parameters at stake:
Output impedance of the MCU oscillator. This varies over frequency and is often complicatated by configuration bits such as "drive level" or "power level". I have never seen these values specified/guaranteed by any MCU maker.
Input impedance of the external capacitor-crystal-capacitor "pi" network. This is primarily determined by the capacitor on the input side, which in turn is determined by the load capacitance specified by the crystal maker.
Voltage gain of the (loaded) MCU oscillator at resonance. The oscillator gain has to make up for the ESR-induced loss of the external Pi network. This gain changes significantly over temperature, supply voltage and manufacturing batch. I have never seen this gain (and driver output impedance) properly specified /tested/guaranteed by any CPU maker. Some makers specify transconductance \$G_m\$ instead of voltage gain.
Voltage gain (actually loss) of the external Cap-Xtal-Cap "Pi" circuit at resonance. This is primarily determined by the internal equivalent series resistance (ESR) of the crystal. The crystal you mentioned specifies ESR=50k. The resistance also increases over age (as moisture/impurties leak into the crystal case) and is also affected by soldering temperature/time. (Impurities in the crystal case evaporating & settling on the quartz) ESR can also vary significantly between manufacturing batches. 50k is a fairly typical ESR for a 32kHz crystal - the lowest I have seen specified at 32kHz for small form factor crystals is 30k.
For any oscillator to work, the total voltage gain, which is the the product of (3) and (4) must >1. In addition, the phase of the gain (yes, gain is a complex number) must be 360 degrees. About half of the phase, 180 degrees, is provided by the inverting amplifier, and the "second inversion" is provided by the cap-xtal-cap network.
Here's a simple online simulation that can help you get a feeling for how gain, output impedance and the capacitor values interact and affect startup. Right-click any component to change its value. (Note - this simulation uses 1mV residual capacitor voltage to fake startup, but in real life noise in the amplifier is the source of startup, like in this one)
So what happened in your case? Most likely, the MCU oscillator designer designed his output stage to reliably function with 12.5pF loaded crystals, and it turned out that at 6pF loading, either the voltage gain or phase requirements were simply not met. Since nothing about the design assumptions is stated in the data sheet, voila, problem for you - and many others.
Wow, what should an embedded designer do?
First, always be aware that a marginal crystal oscillator can cost your business a lot of money.
Second, in light of the above, especially if you lack experience or if your MCU vendor doesn't specify crystal parameters in the datasheet, your best investment could be an external low power 32kHz oscillator.
Third, ensure you use a crystal with ESR and capacitance specified by your MCU maker. If you don't see any in the data sheet, ask your supplier for a list of recommened crystal part numbers, or choose an MCU that does.
Fourth, test, test, test! Over all voltages and temperatures. Note how long startup takes by timing it in firmware using an RC clock if possible, and if production units exceed the norm by, say 2x, let your test firmware set a flag so it can be noticed in production testing. That way, production units can't get out the door with marginal oscillators without alarm bells ringing.
What do experienced production verification engineers do?
They work around the general lack of proper information by requiring a 10x safety margin between "what works" and "what works reliably" - they measure the actual ESR, then add an extra 10x additional "handicap resistance" in series with the crystal into the cap-xtal-cap network. If the "handicapped ESR" system works over all voltage and temperature combinations, then it's assumed that the 10x safety margin is sufficient to cover the unknown variabilities in both ESR and MCU gain. This is partially explained in figure 3 of this application note.
What should YOU do?
If you can't perform the above test for any reason and want to sell a product in thousands, you are certainly better off investing the extra pennies for an off-the-shelf 32kHz oscillator from an oscillator vendor who has done all that testing for you, or by switching to an MCU that specifies a specific crystal (or crystal requirements) in the device data sheet.
While you may "fix" the situation by selecting a crystal with lower internal resistance and/or by playing with different/asymetrical capacitor values, your solution could still be marginal, for the reasons explained above.
TL;DR:
Crystal oscillators can cost your business a lot of time and money. Use an external oscillator if you can, or peform the "handicapped ESR" test as described above over all voltage and temperature ranges.
Finally, be sure to use NPO capacitors for temperature stability.
Best Answer
Edit: After coming back to this question a week or so after the original post, I'm pretty sure that my answer here is incorrect. Please see the comments for the discussion. While most of the discussion is correct, my presumption in this answer about how frequency is specified by manufacturers seems to be incorrect. In particular, frequency is specified at the exact specified value of Cload, not at the mean frequency that can result from the range of tolerable C_load.
A crystal unlike an oscillator depends on user-provided load capacitance to determine its oscillating frequency:
The capacitance tolerance range of the user-provided capacitance is centered around the capacitance's nominal value. The relationship between load capacitance and frequency, however, is not linear:
(For background info see the graph on page 4 and discussion on page 3)
In figure 2,
f
is the nominal crystal frequency (in green),C_load
is the nominal load capacitance (in green), andmin
andmax
(in red) denote the minimum and maximums of tolerance ranges.The result of the non-linearity is that the frequency at the mean of the tolerance range of the capacitance (the purple line) does not correspond with the mean of the frequency tolerance range
f
.It is conventional for manufacturers to define nominal values of components as being the mean of their highest and lowest tolerable values given a list of conditions. The given conditions for the crystal in this case are
C_load
+/- tolerance.Relative the frequency at the nominal load capacitance (the purple line), the lowest tolerable load capacitance
C_load_min
results in a higher frequency shift upward (tof_max
) than does the highest tolerable load capacitanceC_load_max
cause a shift downward (tof_min
). This means that the nominal value of the crystal frequencyf
--which is defined by convention to be the mean of the highest and lowest frequencies -- will be slightly higher than the frequency that results if the load capacitance is exactly the nominal value of the load capacitance (the purple line).That slightly higher mean frequency is where the numbers after the decimal point come from in the nominal frequency of 12.000393 MHz.