Electronic – Current regulation – oscillation problems

For a linear supply i would recommend using a foldback current limiting scheme, especially on something that has a tendency to be shorted/over driven as often as a bench supply. Theres lots of ways to do this and how you do it will be based on the linear regulator design. That being said googling for 'foldback current limiting' especially on google books should give you all the information you need.

For current measurement, you can go high side or low side, whatever you want. The reason to avoid low side is generally to avoid increasing resistance to ground. The small sense resistor you use probably wouldn't matter much in a bench supply. The easiest way to implement this is to use a custom made IC. Such as the MAX4173 there are lots of manufacturers for these though, look around for the one that best fits your needs/cost/package. There are also some that output PWM which can be more accurate to work with than the ADC on most cheap microcontrollers. You just measure the pulse width using a timer and usually either a pin with external interrupt or a comparator input.

The first rule of a control system is to never get information late. Unfortunately, this is exactly what low pass filters in the feedback path do. In particular, R13, C8, R12, C3, and C6 are going to cause stability problems in current limit mode. Also the fact the IC3B is being run open loop means the current shut off signal will slam back and forth. You are basically feeding a digital signal into a linear control loop. That is fine for something like a hard shutdown, but is not going to work for any kind of regulation.

The above is assuming the mess in the lower right corner of your schematic is a current sense so that its OUT pin is ultimately a ground-referenced voltage proportional to the current thru R_Shunt. The rest of your schematic is mostly reasonable, but this mess needs to be fixed. There isn't even a component designator for that chip, let alone any indication as to what it is!

If you want to tweak the controller for good response, you need to do careful analysis. This is usually done in the S domain (Laplace transforms) looking at poles and zeros. I won't get into that here. If you want to keep it simple and are willing to give up some performance, then use the easy rule that says to make sure the controller has lowest bandwidth of anything in the system.

In voltage mode, your controller is IC3A, the thing being controlled are the FETs, and the feedback is R10 and R9. That's a straight forward system that looks reasonable except for C7. I would put C7 directly between the negative input and the output of IC3A. That effectively slows the controller. More capacitance slows it more. Eventually you reach a point where the system is stable enough. AC-wise, Vregulated and Voltage_Ctrl are close enough so that C7 still largely works where it is, but it would be better where I described. To allow for smaller values of C7, make R10 and R9 larger, like 10 kΩ for example.

On a separate note, what is D1 for? Unless you think the load can back drive this power supply, that is just a waste of a diode voltage drop.

Added:

Now that it has been clarified that this circuit is for charging a "12 V" lead-acid battery, more specific recommendations are possible.

We know the load won't change suddenly, so transient response is not a important consideration. Therefore compensating IC3A with a cap from its output to its negative input is good enough. Find the cap value that seems to make it stable and double it. The response will be slow, but again, that matters little for charging a battery.

To do current limiting, I think it would be easiest to use a separate controller with the FETs being driven by the lower of the two controller output voltages. This could be as simple as diode ORing between the two, although that would require a pullup for driving the FETs high, which would make the high and low drive assymetric. One possibility is to do diode ORing of the voltage and current mode control opamp outputs, then buffer that with a third opamp which drives the FET gates.

The trickiest part will be to make sure there isn't instability when the supply crosses over between current and voltage regulation mode. I think if each of the two controller opamps are sufficiently slowed down with a cap between their output and negative input this should be OK. Some experimentation will be required. Fortunately you are in a position to err on the side of slow response in favor of stability, so this should all be doable.