Electronic – Not getting the expected output from the switching regulator

I could see this being on/off control (a switcher) if there was a inductor in series with the FET. However since there isn't, this won't work as a switcher.

This topology looks like it is intended for linear operation. However, the capacitor slows down the system response so that the controller could easily oscillate under some conditions. This is not a good idea in this case. The way to fix it is to slow down the controller. This can be done by "compensating" the opamp if it has such control. If not, the right size cap between the opamp output and its - input will slow it down. For that to work, there will have to be a resistor between Vout and the - input. You can start with a 1 kΩ to 10 kΩ resistor and then use the capacitor value that is just a little above the minimum where the system doesn't oscillate at all operating points you care about.

There are more sophisticated ways to control power supplies, but what I described above is a quick fix that only adds two parts to the existing topology. The tradeoff is that is will slow the transient response. If you care about performance, then you need to start from the beginning and design a power supply the right way.

It'll work with 1Mohm and you could use a 10k or 100k with out problems BUT consider this scenario: -

The battery gets low and switches off the load. The dumping of the load will inevitably cause the battery voltage to rise again because it no longer is under a load condition. This rise then re-enables the load FET and you are into a possible problematic situation. You really should be "latching" the battery low condition with a flip-flop BUT now we're into trying to work out how this situation remedies itself - what mechanism is acceptable for resetting the power?

The low-battery comparator does have hysterisis but it's only 10mV - it maybe enough but i wouldn't count on it. I'm sure the battery could rise (under load to no-load conditions more than 5% and this would be 25mV).

Another option is to have a resistor from the LB output to the LBI - this will increase hysterisis but it's a suck it and see approach. Having said that it'll probably work. You'll need more like a 10k resistor on the LBO and possible no smaller that 100k as feedback to LBI. Your normal LBI resistors will need to be about the 10k area to prevent too much hysterisis too.

EDIT following disclosure of circuit Q1 won't work how you imagine it to - if the source was connected to 0V and the drain to the negative end of the load I believe this would work. The 1Mohm pull-up would keep the load connected via the FET to 0V. As you now have it this won't work. If you are intent on a high-side switch (maybe because the load HAS to be always connected to ground) then there is a way but it involves two fets somewhat along these lines: -

BUT.....

Replace the BC547 with an n-channel FET and get rid of the 10kohm. Ignore the 5.5V - read it as 5V - it's getting late over here and i didn't have the perseverence to fully correct the diagram in paint!!! The MCU line is your LBO. The high-side FET is P-channel - choose one with very low Rds(on) and one that has a "logic-level" gate drive voltage. The n-channel is less stringent - it can't be a bioplar given the high impedance pull-up on the LBO pin BUT, given that you may use a 10kohm (previouse pre-edit comments) it could be a bipolar (just about). Stick with an n-channel logic gate driven device to be sure!