Parallelling panels:
This can be done to increase the maximum producible current, but generally good practise to put a diode in series with each panel's output to deal with the situation where the panels are not equally illuminated. One panel that is in bright sunlight, and the other that is in shade, the panel in sunlight will be driving current back down into the panel in shade and potentially could damage it, so put some diodes in to ensure current can't flow back into the panels. Some panels come with built in diodes for this, and some don't.
Chose a diode with a low forward voltage drop (you don't want to waste too much energy as the panels are naturally inefficient anyway).
DC-DC converters are not necessarily the device you want to use. You need to understand the difference between DC-DC converters, voltage regulators.
Often what you really want is a low drop out (LDO) voltage regulator to keep the output at a nice steady level, regardless of what voltage is coming out of the PV panel. Of course, if the sunlight intensity drops too much, then the output voltage of the panel will drop so low that the voltage falls below the minimum input voltage for the regulator and the regulator will not be able to maintain its output at the specified regulated output voltage.
It's difficult to maximise the power output from a solar panel because the input voltage can vary so much, you might want a 5 volt steady output but find the panel voltage can vary from say 3V to 12V. When the sun is bright enough to produce 6 or 7 volts or more from the panel, then you'll get a nice steady 5V on the output of the regulator, but when the sunlight intensity drops and the panel only produces 5.0V volts or less, then the regulator can't function and the output drops, but the panel is still producing power but you can't use it. This is where you need to start thinking about Maximum Power Point Tracking (MPPT) solar charge controllers. (I haven't yet seen a decent accurate explanation of how they work on the internet).
To answer your specific question, why can't you remove the DC-DC converters? As I indicated earlier, I'm not convinced they are really the type of device you want to use. If you remove them (or remove a voltage regulator), you may find your panel output voltage goes too high and causes damage to the device it is powering.
I think i am missing something, but here are my two cents:
The buck/boost will operate on the input voltage given by the solar panel. The internal switch control will determine if it works as buck or as boost (obviously, if the solar voltage is lower than 5V it is a boost, if it is higher it is a buck).
Depending on the size of the load, the solar (input) voltage may drop. If the voltage gets too low for the boost to handle, the output voltage will also drop.
Solar panels shouldn't be excessive loaded, because their efficiency drops in that case. Because of that there are special buck/boost ICs which provide point of load control for the solar panel (BQ24650 for example). In essence they lower their output power to not overly load the solar panel.
Best Answer
The BQ25504 datasheet says the maximum input voltage at VBAT might be 5.5V. Make sure you don't exceed that and just connect your charger circuitry to the battery in parallel. Battery chargers will have a high output impedance when not powered externally (because they'd otherwise drain the battery), so this will not be a problem.
It might look comfortable to put the 5V supply on the input, but don't: The peak power the BQ25504 will sustain is 400mW, and it's not recommend to charge with more than 300mW flowing through the device (*recommended operation conditions in the datasheet). So, your battery will charge really slow. When you attach 5V, you'll probably just want the battery charged as fast as possible; so invest a couple of cents in one of the hundreds of LiIon Charger ICs and attach that to your battery.