First off, great job so far. There are certainly chips and modules that could do this for you and at efficiencies higher than one could realistically obtain using discrete circuitry, but using them would be of little educational value. Just playing around and reinventing the wheel from stuff on hand is a great way to learn, and it looks like you're doing the 'playing around' part very well. And an oscilloscope is an oscilloscope. The only difference between a 'professional' one and an arduino sampling as hard as it can is what you can measure. If the thing you are measuring can be measured accurately enough with an 'arduinoscope', then there is no difference between it and a 'professional' oscilloscope. Just be sure you can trust your code and it's a perfectly adequate tool for what you're doing.
Now, to actually answer your question!
There is nothing wrong with your comparator circuit. In fact, it's behaving exactly as it should. Which unfortunately is not how you expected or intended it to behave (electrons just don't care about our intentions, they do what they want!).
This is a common problem I've seen when someone at home with digital electronics starts getting into analog circuitry and thinks about it as if it were digital. It isn't. Things aren't high or low, on or off. And any closed loop control circuit (closed loop meaning the output can effect the input - in this case, the comparator can effect the voltage it sees at its inverting input) is going to settle on a specific operating point, or just oscillate uselessly (because it is unstable due to taking too long to react - or too out of phase with it's feedback).
Comparators are not digital. They behave very non-linearly if they are open loop, and are the most 'digitally' in that usage case. Open loop means their output will not effect their input. This is, of course, not the case in your circuit. And here is the dirty secret: comparators are just high gain differential amplifiers. In other words, they're op amps with internal resistors arranged in a differential amplifier topology for the sake of convenience. They have very high gain, but it is not infinite. If you have a closed negative feedback loop, exactly like you do, it is going to to behave like an op amp. It is not a digital logic gate, it's an analog component that is designed to interface with digital circuitry, but you're not using it like that.
Also, a MOSFET is not digital either. They are not switches. They are transimpedance amplifiers. In fact, BJTs behave much more like switches than MOSFETs do. FETs can be modeled as a voltage controlled resistor, and a very linear one at 'intermediate' voltages. What you call intermediate is known as the linear region - and MOSFETs have a very wide such region. Much larger than other semiconductors. A FET is about the least-digital switching element you can find.
As the gate voltage gets higher, it loses its linear voltage to Rds behavior, but to get to the point where it is 'on', it must cross that linear region. Your comparator is not going to turn on the MOSFET sharply because it will rapidly close in on the intermediate voltage, just like the op amp it secretly is, and do whatever it needs to to keep the voltage at its inputs the same. It's oscillating near the ideal control point due to being configured to be a comparator, but it WANTS to turn the MOSFET partly on, it wants to make it a resistor, and it is doing a decent job despite being configured for a very different purpose. The true nature and op amp heart of a comparator reveals itself.
Why does it want the FET to be a resistor? Because that is what it needs to do. If the voltage goes above its noninverting terminal, it will turn on the FET until the voltage drops below the inverting terminal, and it will back off. If it could, it would settle on a specific voltage that keeps the FET only turned on enough to keep the voltage at its terminals equal. It can't do that, but it is trying and it is still doing a half-way respectable job of achieving that, even if it is oscillating around the voltage instead of settling on it. It will never turn the FET on quickly, or even all the way, as that would make the panel voltage drop too much. The ATtiny is unable to react fast enough, and so the voltage over and undershoots all over the place, but the comparator is fast and reacts continuously.
And it's working perfectly. It's tracking the power point you've set for the panel. You've given it a resistor it can control, so it is going vary the resistance of the MOSFET as needed to keep the panel voltage near the set point. What you've built is a somewhat awkward constant power dummy load. It's getting hot because the panel can't deliver the power drawn by the 100Ω resistor, so the MOSFET is being used to dynamically add extra series resistance until a power point is tracked. But it will only add just enough resistance to track that power point, and lower or raise it to constantly consume that amount of power. So it should get hot. Not because anything is wrong, however. The circuit is working, or trying to. If you used a proper op amp, it wouldn't oscillate, and instead keep it at whatever resistance is needed to consume a fixed wattage from the panel.
This is why MPPT is hard. You can't track a power point by turning the load on and off, you just get massive voltage swings up and down like you see with your ATTiny. You can't deliver all the power to the load without switching something on and off, because anything else means burning up the excess power. This is fundamentally the same reason linear regulators operate as they do. This is, really just a linear regulator. It's regulating a voltage. It doesn't matter that it is the voltage from a solar panel. It's still linear, and it is still trying to keep that voltage at a set point. And the only way it will do that is by burning power off as heat. Which is exactly what it is doing.
There is no software problem here. No amount of software can overcome this problem, unfortunately. If you want efficiency, you cannot use linear power point tracking. You will need to make it switch (which I know was your original intention), and you will need an energy storage device that will be alternatively charged and discharged by the switch. What I just described, and what you are envisioning (even if you didn't realize it) is a switching regulator. I recommend an inductor as your energy storage device. A PWM control chip would work in place of the comparator. A TL494 is a classic.
There is no way to do this with any amount of efficiency in the way you've set up, no matter software or anything else. A MPPT controller is usually an input-tracking buck-boost regulator for that reason, and one that charges a battery or super capacitor through an inductor. Building such a circuit is beyond the scope of this answer, but it is certainly worth giving 'TL494 MPPT' a quick google. Image search brings up a lot of examples. You could even try to do it using the attiny85 controlling the MOSFET, but you'll need to add an inductor, diode, capacitors, some other components. Unfortunately, the problem is physics, and it has not really shifted to software as it is, though it might seem like it. No matter what you do, you'll not get around burning that power without first adding the necessary components in a switching topology.
Anyway, building that would certainly be a terrific way to learn about electronics - and the scary parts like inductance and magnetics. It is also not for the feint of heart. Either way, good luck!
Best Answer
The situation is far worse than you imagine.
The maximum available current from a series string of cells is the current available from the lowest current cell. A lightly shaded cell will reduce panel output. A heavily shaded cell will almost stop panel output - and as this happens reverse potential from all the unshaded cells develops across the shaded cell so that it contributes nothing. If bypass diodes are fitted current then flows in an appropriate diode around the shaded cell, and also nearby cells if the diode spans several cells.
Accordingly, panels and cells within a panel which are connected in series need to be matched as closely as possible both in performance and in received isolation (light) levels. A 1000 W/m2 : 200 W/m2 imbalance such as you suggest can be fatally bad and should be avoided by design to the maximum extent possible. Bypass diodes should be fitted to prevent large reverse voltages occurring across shaded cells as this can destroy whole installations, or worse.
Bypass diodes are connected in reverse polarity to a cell's output when it is acting as a current source. If the cell current is lower than the string current the voltage acting to drive current through the cell causes it to produce less output and then to reverse bias it completely. A bypass diode across the cell conducts when the cells reverse polarity exceeds the forward polarity of the diode. So none of a partially shaded cells output contributes to string current when the bypass diode is conducting and instead the diode reduces output by a diode drop, plus whatever voltage the shaded cell(s) no longer produce.
While this description may seem contrary to what you expect, you will find it well described in many online articles.
Many examples can be found here and here
The effect can be easily visualised using the diagram below. For the diode to conduct the usually -ve terminal of the PCV-cell must be positive relative to it's usual +ve output. No current is produced.
Diagram from Dale Marshall here
Note that in the absence of bypass diodes the full remaining string voltage will build up across the shaded PV-cell. In high voltage systems this may be in the 400-1000 V range. Cells with defects my have low isolation voltages and break down. In such cases fires can (& do) occur which destroy the cell concerned, often damage others, will usually write that panel off and may destroy the whole installation or even an the building it is mounted on.