I assume you mean "open circuit or source of power dissipation" - if it was short circuit it would cause substantial power dissipation.
Summary: It is usual to fit blocking diodes and good practice but on small to medium panels at voltages under say 100V it is probably not strictly necessary.
Detail:
It is usual to fit blocking diodes to prevent reverse current when Vbattery is > Vpanel.
I have an experimental setup here with a 30Vmp 250 Wmp panel so Imp is about I=W/V ~= 8.3A. Those are the values you would get in full sun (insolation about 1 sun = 1000 W/m^2). It's 10:45pm here at present and the panel is in near complete darkness. I just applied 25VDC to the panel and it drew 40 mA or about 0.04/8.3 ~= 0.5% of full current. That is not a trivial amount but may be acceptable. In a typical day with say 4 sunshine hours the % current lost at that rate is in the order of say 12 hours x 40 mA / (4 hours x 8.3A) =~ 1.5% of the day's insolation wasted. The current lost will be higher at the dark to day transitions but still probably modest - see below.
In practice the ability of the panel to pass current relates to the current that it would provide if optimally loaded. Consider maximum power = Wmp to occur at 1 sun of insolation = 1000 W/m^2. For much of the light level range from about 5% x 1 sun or less up to 1 sun the panel is close to a current source with current close to being proportional to light level and Vout being about 70% of max output at low light levels and close to Vmp at most light levels above 10%. Vpanel only falls substantially at say 1% x 1 sun or less.
The implications of the above is that when Vpanel is well below battery voltage the current it will support is minimal and in the dark it is about zero. While a blocking diode will prevent this loss it also has potential losses of its own. A schottky diode will drop say 0.4V to 0.6v in smaller systems depending on load and more than that in large systems. On a say 12V battery system Vmp is typically 18V (which is probably higher than needed for most situations). A fully charged floated lead acid battery requires about 13.7 V and you "never" need over 15V so a diode drop is not too problematic except in VERY low light conditions.
At very high voltages (hundreds of volts) you can get cell breakdown in some cases and diodes seem like a good idea. Also eg 40 mA x 1000V = 40W - not to be sneezed at.
First off, great job so far. There are certainly chips and modules that could do this for you and at efficiencies higher than one could realistically obtain using discrete circuitry, but using them would be of little educational value. Just playing around and reinventing the wheel from stuff on hand is a great way to learn, and it looks like you're doing the 'playing around' part very well. And an oscilloscope is an oscilloscope. The only difference between a 'professional' one and an arduino sampling as hard as it can is what you can measure. If the thing you are measuring can be measured accurately enough with an 'arduinoscope', then there is no difference between it and a 'professional' oscilloscope. Just be sure you can trust your code and it's a perfectly adequate tool for what you're doing.
Now, to actually answer your question!
There is nothing wrong with your comparator circuit. In fact, it's behaving exactly as it should. Which unfortunately is not how you expected or intended it to behave (electrons just don't care about our intentions, they do what they want!).
This is a common problem I've seen when someone at home with digital electronics starts getting into analog circuitry and thinks about it as if it were digital. It isn't. Things aren't high or low, on or off. And any closed loop control circuit (closed loop meaning the output can effect the input - in this case, the comparator can effect the voltage it sees at its inverting input) is going to settle on a specific operating point, or just oscillate uselessly (because it is unstable due to taking too long to react - or too out of phase with it's feedback).
Comparators are not digital. They behave very non-linearly if they are open loop, and are the most 'digitally' in that usage case. Open loop means their output will not effect their input. This is, of course, not the case in your circuit. And here is the dirty secret: comparators are just high gain differential amplifiers. In other words, they're op amps with internal resistors arranged in a differential amplifier topology for the sake of convenience. They have very high gain, but it is not infinite. If you have a closed negative feedback loop, exactly like you do, it is going to to behave like an op amp. It is not a digital logic gate, it's an analog component that is designed to interface with digital circuitry, but you're not using it like that.
Also, a MOSFET is not digital either. They are not switches. They are transimpedance amplifiers. In fact, BJTs behave much more like switches than MOSFETs do. FETs can be modeled as a voltage controlled resistor, and a very linear one at 'intermediate' voltages. What you call intermediate is known as the linear region - and MOSFETs have a very wide such region. Much larger than other semiconductors. A FET is about the least-digital switching element you can find.
As the gate voltage gets higher, it loses its linear voltage to Rds behavior, but to get to the point where it is 'on', it must cross that linear region. Your comparator is not going to turn on the MOSFET sharply because it will rapidly close in on the intermediate voltage, just like the op amp it secretly is, and do whatever it needs to to keep the voltage at its inputs the same. It's oscillating near the ideal control point due to being configured to be a comparator, but it WANTS to turn the MOSFET partly on, it wants to make it a resistor, and it is doing a decent job despite being configured for a very different purpose. The true nature and op amp heart of a comparator reveals itself.
Why does it want the FET to be a resistor? Because that is what it needs to do. If the voltage goes above its noninverting terminal, it will turn on the FET until the voltage drops below the inverting terminal, and it will back off. If it could, it would settle on a specific voltage that keeps the FET only turned on enough to keep the voltage at its terminals equal. It can't do that, but it is trying and it is still doing a half-way respectable job of achieving that, even if it is oscillating around the voltage instead of settling on it. It will never turn the FET on quickly, or even all the way, as that would make the panel voltage drop too much. The ATtiny is unable to react fast enough, and so the voltage over and undershoots all over the place, but the comparator is fast and reacts continuously.
And it's working perfectly. It's tracking the power point you've set for the panel. You've given it a resistor it can control, so it is going vary the resistance of the MOSFET as needed to keep the panel voltage near the set point. What you've built is a somewhat awkward constant power dummy load. It's getting hot because the panel can't deliver the power drawn by the 100Ω resistor, so the MOSFET is being used to dynamically add extra series resistance until a power point is tracked. But it will only add just enough resistance to track that power point, and lower or raise it to constantly consume that amount of power. So it should get hot. Not because anything is wrong, however. The circuit is working, or trying to. If you used a proper op amp, it wouldn't oscillate, and instead keep it at whatever resistance is needed to consume a fixed wattage from the panel.
This is why MPPT is hard. You can't track a power point by turning the load on and off, you just get massive voltage swings up and down like you see with your ATTiny. You can't deliver all the power to the load without switching something on and off, because anything else means burning up the excess power. This is fundamentally the same reason linear regulators operate as they do. This is, really just a linear regulator. It's regulating a voltage. It doesn't matter that it is the voltage from a solar panel. It's still linear, and it is still trying to keep that voltage at a set point. And the only way it will do that is by burning power off as heat. Which is exactly what it is doing.
There is no software problem here. No amount of software can overcome this problem, unfortunately. If you want efficiency, you cannot use linear power point tracking. You will need to make it switch (which I know was your original intention), and you will need an energy storage device that will be alternatively charged and discharged by the switch. What I just described, and what you are envisioning (even if you didn't realize it) is a switching regulator. I recommend an inductor as your energy storage device. A PWM control chip would work in place of the comparator. A TL494 is a classic.
There is no way to do this with any amount of efficiency in the way you've set up, no matter software or anything else. A MPPT controller is usually an input-tracking buck-boost regulator for that reason, and one that charges a battery or super capacitor through an inductor. Building such a circuit is beyond the scope of this answer, but it is certainly worth giving 'TL494 MPPT' a quick google. Image search brings up a lot of examples. You could even try to do it using the attiny85 controlling the MOSFET, but you'll need to add an inductor, diode, capacitors, some other components. Unfortunately, the problem is physics, and it has not really shifted to software as it is, though it might seem like it. No matter what you do, you'll not get around burning that power without first adding the necessary components in a switching topology.
Anyway, building that would certainly be a terrific way to learn about electronics - and the scary parts like inductance and magnetics. It is also not for the feint of heart. Either way, good luck!
Best Answer
C2 seems way to small for your consumption and the circuit will be very unstable. I would try to use at least a capacitor rated for 25V 10000uF. Maybe even several of these in parallel.