Relying on a capacitor to fail in an overvoltage condition in order to protect some other equipment is a bad design practice. Capacitors may fail open, short, or somewhere in between; unpredictability makes for poor protection. And I've personally seen overvoltaged caps emit flames.
You should spec all your components such that its voltage rating is at least the highest voltage you expect them to ever see, plus some safety margin. If its unexpected voltage spikes you're concerned about, you can use more capacitance on a DC link to limit the voltage rise, or MOVs on an AC line connection.
It's like the "paradox" of the immovable object meeting the irresistible force. In reality, neither can exist.
A real power source will have some impedance. A real capacitor will have some impedance. Real wires connecting them have resistance and inductance.
So in reality when you slap a fairly 'stiff' power source across a fairly good real capacitor there's a spark and the capacitor charges through those series resistances with some ringing and stuff due to the inductances. Ignoring the inductances, the voltage difference would simply divide in ratio to the internal resistance of the capacitor and the internal resistance of the power supply and the wire resistance.
To answer your specific question:
If the capacitor and voltage source and wires are ideal, you have a mathematical problem, like division by zero. It's of no consequence in the real world- it just illustrates that the ideal models of the power source and the capacitor and wires are insufficiently accurate to describe their real-world behavior. Modelling any one of those as a real part with real resistance (and inductance) will make the mathematical problem go away, but it won't likely give you an accurate indication of what is actually happening.
For example, if the wire (or the capacitor or the power supply) had 10m\$\Omega\$ resistance, and there is a 10V difference, you could predict you'd see 1000A (which is very high, but not infinity) and the capacitor would charge very quickly. In reality that isn't likely going to happen because of other non-ideal factors. If the 10m\$\Omega\$ was modeled as in the power supply, the power supply voltage would drop. If the 10m\$\Omega\$ was modeled as in the capacitor, the voltage would suddenly appear across the capacitor terminals. If the 10m\$\Omega\$ was modeled as in the wire, the voltage would appear across the wire. But none of those is very realistic.
If you modeled as a circuit with no resistance at all and a tiny bit of inductance (even superconducting wires of any length have inductance) then a simple mathematical model would predict ringing that would persist forever, energy sloshing back and forth between the inductance and capacitance at an angular frequency of \$\omega_0 = {1 \over \sqrt{LC}}\$.
Best Answer
While not a perfect analogy, think of the voltage on the capacitor similar to the liter capacity of a tank. It will hold "35 V" but you needn't fill it completely. But like @JustJeff said, you'd be wise to ensure the container can hold more than necessary to prevent spills (and in an electrolytic capacitor's case, the electrolyte can expand and quite literally "spill" out).
Note that a better analogy to capacity would be the farad unit, since that's a measure of a capacitor's charge capacity, so don't get that confused with voltage, which is the potential to do work.