A very nicely presented 1st question (or 100th or ...).
Lots of detail to assimilate but it all seems relevant and useful if a good answer is to be found. I cannot spend the time needed now on this but will throw in a few comments and see what others have said later.
I spent about 15 minutes just going to and fro over the circuits and layouts and doing some basic sanity checking. I'm sure your rule checking would have eliminated basic errors.
I have NOT tried to work out what your fault may be caused by specifically - and suspect that it may be a hard fault or misthought rather than the design areas touched on below. BUT any of the following may relate.
Have you tried placing the whole PCB on a PCB ground plane? Can help heaps with single sided. May not.
The two unrouted nets shown presumably have wire links added by hand. (If not that would be an easy fix :-) )
A single side board MAY be doable but with such a complex beast with two switchers and the ability for feedback between them you'd need real care, a scope glued to your right hand and some luck. Even a double sided board (which is about as cheap and quick from many board houses) costs much the same.
A problem is (which may have led to a problem that you get) that the IC seems to have pinouts which assume you can route across the IC with ease so that critical current loops have little area. Because you are on 1 layer this is not true and you have several such loops that more or less overlap and seem to invite disaster.
The obvious ones to minimise to start are the two inductor loops p7-L1-p15 and p16&p17-L2-p14.The L1 loop involves an added jumper and how you route this may have an effect.
Noise getting into the feedback dividers can be bad news indeed. I see you have used c5 across R4 as per their circuit but have no cap across R8 - shown as Copt on one of their circuits and not on another. Simplistically this passes fast load transients or noise that affects output into the feedback pin at a greater rate and level than you get from the divider. Presence or absence in SOME designs is life or death.
Draw lines on printouts of the layout with different coloured markers as to where the loops seem likely to be that are used by different processes (Inductor currents, feedback dividers, ...). (Draw on a screen if that works for you - I find paper and markers more powerful). You can then see likely interactions and any loops that have large open front doors for noise / cross coupling to rush in and out of.
More later maybe.
Best Answer
They mean like this:
simulate this circuit – Schematic created using CircuitLab
Because there's a lot of current circulating in that loop between the chip Vdd or chip GND and L1/Rtrace/Ltrace/Count you don't want to take the feedback point before the capacitor- rather a kind of Kelvin connection so any voltage drop in that loop does not unduly contaminate the feedback signal. You also should keep the loop area for those two loops (switch on/switch off loops) as small as practical to reduce EMI and avoid other problems.