Electronic – why do rectified voltage boosts after adding a capacitor

The rectified AC waveform catches the peaks. The input 9VAC is RMS (Root-Mean-Square average) equivalent -- the actual amplitude of the sinewave is about 40% higher than the RMS average (square root of 2 is 1.414). So on your picture the 9V equivalent is about 70% of the way between 0V and the peaks.

The numbers don't work out exactly to the ideal square-root-of-two crest factor, because there is some voltage drop across the two diodes that are on, and also because there is some variation in the line voltage.

The reason RMS is used to describe AC voltages, is because the amount of power (heat) delivered to the resistive load is the same as it would be for a 9V DC 1A source.

Edit: explaining the observed difference in load voltage measurement for different load capacitor condition, explaining why DMM gives wrong measurement for full-wave rectified waveform...

Voltage is not actually being boosted in this circuit. When the capacitor is removed, the full-wave rectified signal doesn't sustain the peak voltages. As Ignacio Vazquez-Abrams mentions, the DMM may not be measuring the waveform correctly, especially in the case where there was no capacitor -- assuming you measured with the DMM's DC Voltage setting, without the capacitor the full-wave rectified waveform would be confusing the measurement. The 9V DC measurement reported by the DMM matches the rated 9V AC RMS equivalent, so maybe the DMM was somehow measuring the RMS value. Then when you added a capacitor, the waveform peaks were sustained long enough for the DMM to start measuring accurately. Sadly, it is possible for measurement equipment to "lie" to us under some conditions. Happens to the best of us sometimes.

The DMM is just a electronic machine, it's not a magic box that always gives the right voltage measurement. Most DMM's use a measurement technique called dual-slope integration, where a capacitor is first rapidly charged up to the voltage being sampled, and then the sampling capacitor is discharged through a constant-current source. The DMM counts how long it takes to discharge the capacitor back to zero. The value of that counter is what the DMM displays. Calibration depends on the current source, the comparator offset voltage, and the quality of the sampling capacitor. This technique is cheap to implement and it works great, as long as the input signal doesn't change very quickly. But when connected to that full-wave rectified signal, the sampling capacitor doesn't stay at the peak voltage. It's indeterminate where the sample interval begins and ends, so it's hard to know exactly how many counts the DMM might report. So if the C is omitted, then it's not really a DC circuit, so the DMM DC measurement isn't valid.

You also asked about using different value of C. The bridge rectifier is not regulated, its output voltage can vary with different load impedance. Changing the load capacitance C affects the capacitor's reactance Xc, which also affects load impedance.

$$ Xc = \frac{1}{2 * pi * frequency * C} $$

A lower load impedance, just like load resistance, will draw more current at a given voltage. But unlike resistance, the current and voltage waveforms may be out of phase. So it's possible to have voltage across a capacitor even with zero current, and it's possible to have current through an inductor even with zero voltage (under some conditions).

Putting a reactance in parallel with a resistance is a bit more complicated than putting resistors in parallel, because the voltage and current waveforms are in-phase for the resistor but are 90 degrees out-of-phase for the capacitor. In AC circuit analysis, we use complex numbers and phasor notation (yes, that really is a thing) to model these AC circuit elements. If you think of impedance as a vector, with the length of the vector acting similar to resistance in Ohm's law, and reactance acting at right angles to resistance, then putting the resistor and capacitor in parallel gives the total load impedance Z. Although it's possible to go deeper into the maths, there's another important point worth mentioning:

This circuit isn't regulated. If you want to get 12V DC output, you can't just select a capacitor value and expect this to always give 12V DC output regardless of how much load current is drawn. This circuit is a good building block to start with, but the full-wave rectified output voltage will vary with changes in the utility line-in voltage as well as the load current. If you really want it to be regulated, add a regulator circuit such as 78M05 (or 78M12 if you really need 12V). In that case you'll need the full-wave bridge to provide a bit more then 12V so that the regulator has some headroom to work with (but not too much, because the linear regulator works by wasting the unwanted energy.)

AC circuits theory can be kind of mind-bending at first, because there are all these surprising mathematical things like imaginary numbers and Euler's law, that turn out to actually work in real life. The comment about how the capacitor "evens out the peaks" is... kind of true... but it's a major oversimplification. As you've discovered, a qualitative statement like that doesn't help you determine how much capacitance you need to achieve your design goal of making a 12V DC power supply.

I'm not going to be able to fully explain AC circuits theory here, but here are at least some interesting breadcrumbs:

See https://en.wikipedia.org/wiki/Electrical_reactance

See https://en.wikipedia.org/wiki/Electrical_impedance

See https://en.wikipedia.org/wiki/Phasor