Electronic – Using capacitor as a rechargeable battery

As part of my learning of electronics, I'm currently trying to develop my own network of sensors.

So far, I've managed to make myself a satisfying small setup :
A few nodes, built with an atmega328p, a set of sensors and a RF emitter, capture data and send it through RF to a central server, which takes care of that data.

This works well, and I'm now trying to get rid of the battery part.
My current solution works fine with either a 9v battery or a 5V USB charger, and I'd like to try to get other kind of energy inputs (like solar panels, windmills, etc…).

My circuit requirements are simple : 5v / 65mA.
However, these requirements are only necessary for 1 sec every once in a while, since i don't need the circuit to run constantly. a few times a day is largely enough, and it can sleep / power-off the rest of the time.

I got myself what should have been a 0.5W solar panel (5v/100mA). However, when I try to measure the current I get from it, fully facing sunlight, I only get 5v/5mA.
This is not enough for my circuit, so I figured I would go and try to store that energy.

So I was thinking that maybe I could use capacitors :
After reading a bit on it, I've learned how to charge and discharge it.
However, to my knowledge, this always requires some changes on the circuit in order to switch from charging to discharging : Either removing the input battery like Capacitors as a Storage for Electricity, or doing something like this https://www.arduino.cc/en/Tutorial/CapacitanceMeter . I'm not sure if the link could be suitable in my case, because this requires the Microcontroller to be powered, and at this point, it is not.

The other option was to use rechargeable batteries, but I couldn't find really small ones and I would have the same issue as above. I don't want to buy LR44/CR2032 stuff because I would have to change them quite often.

My questions are :

1. Should I simply replace the solar panel ? Is it normal that the current I read is so low even though the voltage I get is correct ?
2. Is there a way to make a circuit that charges a capacitor, and when it's reached a sufficient charge, automatically use the energy stored in it in order to power some other part of the circuit until it's emptied, and all over again ?
3. Same question with a rechargeable battery.
4. In the case where this would not be possible, which I'm starting to believe, what is the common approach for this ? If possible, I'd like to avoid using classical "AA" or "AAA" batteries, but rather use "in-circuit" batteries (I don't know their name though)

Note : Sorry if anything is unclear or seems stupid. I'm still begining in electronics. Feel free to make any suggestion relative to this project.

Thanks in advance