A very nicely presented 1st question (or 100th or ...).
Lots of detail to assimilate but it all seems relevant and useful if a good answer is to be found. I cannot spend the time needed now on this but will throw in a few comments and see what others have said later.
I spent about 15 minutes just going to and fro over the circuits and layouts and doing some basic sanity checking. I'm sure your rule checking would have eliminated basic errors.
I have NOT tried to work out what your fault may be caused by specifically - and suspect that it may be a hard fault or misthought rather than the design areas touched on below. BUT any of the following may relate.
Have you tried placing the whole PCB on a PCB ground plane? Can help heaps with single sided. May not.
The two unrouted nets shown presumably have wire links added by hand. (If not that would be an easy fix :-) )
A single side board MAY be doable but with such a complex beast with two switchers and the ability for feedback between them you'd need real care, a scope glued to your right hand and some luck. Even a double sided board (which is about as cheap and quick from many board houses) costs much the same.
A problem is (which may have led to a problem that you get) that the IC seems to have pinouts which assume you can route across the IC with ease so that critical current loops have little area. Because you are on 1 layer this is not true and you have several such loops that more or less overlap and seem to invite disaster.
The obvious ones to minimise to start are the two inductor loops p7-L1-p15 and p16&p17-L2-p14.The L1 loop involves an added jumper and how you route this may have an effect.
Noise getting into the feedback dividers can be bad news indeed. I see you have used c5 across R4 as per their circuit but have no cap across R8 - shown as Copt on one of their circuits and not on another. Simplistically this passes fast load transients or noise that affects output into the feedback pin at a greater rate and level than you get from the divider. Presence or absence in SOME designs is life or death.
Draw lines on printouts of the layout with different coloured markers as to where the loops seem likely to be that are used by different processes (Inductor currents, feedback dividers, ...). (Draw on a screen if that works for you - I find paper and markers more powerful). You can then see likely interactions and any loops that have large open front doors for noise / cross coupling to rush in and out of.
More later maybe.
Best Answer
At the higher current point use a scope to check if the circuit is oscillating or ringing on the input or output pins.
Be sure the feedback line is far enough away from the toroid so as not to pick up noise from it, (per the data sheet). While a good commercially made toroid should not radiate much magnetic noise a hand made part may not be so perfect.