Summary:
That circuit uses "overkill" with that application but serves as an OK example.
Here is a typical LM7805 datasheet
It can be seen on page 22 that having two capacitors at Vin abd two at Vout is not necessarily a standard arrangement, and that the capacitor values in the supplied circuit are relatively large.
Below is fig22 from the datasheet.
Your circuit:
A large capacitor like the 2200 uF act as a "reservoir" to store energy from the rough DC out of the bridge rectifier. The larger the capacitor the less ripple and the more constant the DC. When large current peaks are drawn the capacitor supplied surge energy helps the regulator not sag in output.
The white and black bars on the capacitor symbol show that it is a "polar " capacitor - it only works with + and - on the selected ends.
Such capacitors are usually "electrolytic capacitors". These have good ability to filter out low frequency ripple and to respond to reasonably fast load changes.
By itself it is not enough to do the whole job as it is not good at filtering higher frequency noise because electrolytics tend to have large internal inductance + large (relatively) internal series resistance (ESR).
The small input capacitor (here shown as u1 = 0.1 uF) will be non polarized and will usually nowadays be a multilayer ceramic capacitor with low ESR and low inductance giving it excellent high frequency response and noise filtering capabilities. By itself it is not enough to do the whole job as it cannot store enough energy to deal with the energy needed to filter out ripple changes and large load transients.
The same applies in general terms to the output capacitors.
C4 = 10 uF helps to supply any gross load changes thus taking some load off the regulator. It is not usually deemed necessary to have more than a very small capacitor here. Some modern regulators need a largish capacitor here for stability reasons but the LM78xx does not.
Here the second output capacitor is 0.1 uF and it is there to deal with high frequency noise.
Note that having a large capacitor on the output can cause problems. If the input was shorted so that power was removed C4 would discharge back through the regulator.
Depending on voltage and capacitor size this can cause damage.
One method of dealing with this is to provide a usually reverse-biased diode from regulator output to regulator input. If the regulator input is shorted to ground the output capacitor will discharge through the now forward biased diode.
Added: Nils noted:
A very large reservoir capacitor may lead to increased noise. The on-time of the diodes would get shorter yet the same amount of power is transferred. This causes current spikes in the transformer which start to radiate out a noisy magnetic field. Bigger is not always better here. It's unlikely to cause problems in circuits that uses the 78xx series regulators though, they just don't move enough power usually.
Good point. Adding a small series resistor between transformer and 1st capacitor serves to "spread" the conduction angle, reduce current peak, reduce noise and make life easier for the diodes. Working out the diode current can be somewhat mind-taxing I seem to recall (having done it as an exercise long ago). Nowadays a simulation is easy enough to make calculation unusual.
The first rule of a control system is to never get information late. Unfortunately, this is exactly what low pass filters in the feedback path do. In particular, R13, C8, R12, C3, and C6 are going to cause stability problems in current limit mode. Also the fact the IC3B is being run open loop means the current shut off signal will slam back and forth. You are basically feeding a digital signal into a linear control loop. That is fine for something like a hard shutdown, but is not going to work for any kind of regulation.
The above is assuming the mess in the lower right corner of your schematic is a current sense so that its OUT pin is ultimately a ground-referenced voltage proportional to the current thru R_Shunt. The rest of your schematic is mostly reasonable, but this mess needs to be fixed. There isn't even a component designator for that chip, let alone any indication as to what it is!
If you want to tweak the controller for good response, you need to do careful analysis. This is usually done in the S domain (Laplace transforms) looking at poles and zeros. I won't get into that here. If you want to keep it simple and are willing to give up some performance, then use the easy rule that says to make sure the controller has lowest bandwidth of anything in the system.
In voltage mode, your controller is IC3A, the thing being controlled are the FETs, and the feedback is R10 and R9. That's a straight forward system that looks reasonable except for C7. I would put C7 directly between the negative input and the output of IC3A. That effectively slows the controller. More capacitance slows it more. Eventually you reach a point where the system is stable enough. AC-wise, Vregulated and Voltage_Ctrl are close enough so that C7 still largely works where it is, but it would be better where I described. To allow for smaller values of C7, make R10 and R9 larger, like 10 kΩ for example.
On a separate note, what is D1 for? Unless you think the load can back drive this power supply, that is just a waste of a diode voltage drop.
Added:
Now that it has been clarified that this circuit is for charging a "12 V" lead-acid battery, more specific recommendations are possible.
We know the load won't change suddenly, so transient response is not a important consideration. Therefore compensating IC3A with a cap from its output to its negative input is good enough. Find the cap value that seems to make it stable and double it. The response will be slow, but again, that matters little for charging a battery.
To do current limiting, I think it would be easiest to use a separate controller with the FETs being driven by the lower of the two controller output voltages. This could be as simple as diode ORing between the two, although that would require a pullup for driving the FETs high, which would make the high and low drive assymetric. One possibility is to do diode ORing of the voltage and current mode control opamp outputs, then buffer that with a third opamp which drives the FET gates.
The trickiest part will be to make sure there isn't instability when the supply crosses over between current and voltage regulation mode. I think if each of the two controller opamps are sufficiently slowed down with a cap between their output and negative input this should be OK. Some experimentation will be required. Fortunately you are in a position to err on the side of slow response in favor of stability, so this should all be doable.
Best Answer
Caps are located close to each digital IC, or small set of such ICs, to act as local reservoirs to smooth out the rapidly fluctuating current demands of such ICs. This prevents those rapidly fluctuating currents from causing fluctuating voltages on longer supply wires (PCB traces) and possibly disrupting other chips connected to those supply wires.
In some instances you will also see a large cap parallel with a small cap right next to it. The large cap provides a large reservoir, but has a significant internal resistance, so doesn't respond as quickly as a small cap can. So together the two caps can respond quickly and provide a large reservoir.
Real capacitors have both some internal resistance and inductance in series with their "ideal" capacitance. The effects are larger with larger-value capacitors, and vary with capacitor material and construction. For the current discussion, both these non-ideal characteristics act to slow the speed with which the capacitor can respond.
A good discussion can be found here: http://www.analog.com/library/analogdialogue/anniversary/21.html
An additional article on board layout for high-speed digital: http://www.ti.com/lit/an/scaa082/scaa082.pdf