Electronic – How do simple low power devices store setup when switched off

What you need is a technique called wear leveling. It doesn't write your data every time at the same location in the EEPROM, but uses some algorithm to use different locations. I've read about complex wear leveling algorithms, but I wouldn't know why the following simple method wouldn't work.

Add to your data a 24-bit counter, so that your data block is for instance 8 bytes long. Pages on a 24AA64 are 32 bytes long, so a 64kb EEPROM holds 256 pages. From the datasheet:

"When doing a write of less than 32 bytes the data in the rest of the page is refreshed along with the data bytes being written. This will force the entire page to endure a write cycle, for this reason endurance is specified per page."

so it doesn't make sense to use data blocks smaller than a 32 bytes page.

Look at the counter of the first page. If it's zero you used the maximum number of write cycles for that page, so you move on to the next page and check that counter. Repeat until you find a counter > zero. That's the page you're currently using. Microchip's EEPROMs have a 1 million cycles endurance, which you can increase to 256 million with the given example of maximum 32 bytes per block in a 64kb EEPROM. That should be enough to outlast your product: 40 years if you write once every 5 seconds(!).

You'll want to initialize your EEPROM on first use. How do you know when that is. Use the last page to write a unique signature upon initialization. Check at each power-up if the signature is there. If it isn't the device has to be initialized. You can preset the counter in each page with 0xF4240 (for 1 million) or clear everything to 0xFF and write the 0xF4240 when you first use the page.
Initializing an EEPROM is needed because sometimes a certain pattern is written to it in the production/test process.

edit
The wear leveling should solve your problems, but I still want to comment on the capacitor solution. You say the board is rather power-hungry, but maybe you can isolate the microcontroller/EEPROM's power from the rest of the board with a diode. So you'll probably need only a few mA when main power is gone. The 24AA64 writes a page in less than 5ms, then at 10mA and an allowable voltage drop of 100mV you'll need

\$ C = \dfrac{I \cdot t}{\Delta V} = \dfrac{10mA \cdot 5ms}{100mV} = 500\mu F \$

Easy with a small supercap.

further reading
datasheet 24AA64
EEPROM Endurance Tutorial

The first circuit on the page you linked to is already pretty efficient in that just about all supply current is used to light the LEDs. So the issue isn't lower power as much as to be able to run from a lower voltage.

Typical coin cells, like the common CR2032, put out about 3V. This mean you have to use LEDs that light at somewhat lower voltage. Fortunately common green LEDs typcially light at around 2.1V. Switching to green LEDs allows for using a 3V supply.

To cut down power, you have to cut down the current thru the LEDs. This directly effects brightness. If this device is for indoor use, even 1mA may be good enough to noticeably light the LEDs. Getting decent LEDs with decent efficiency definitely helps here. Let's say you want to draw only 2 mA from the coin cell. The voltage drop accross the resistor in series with the LED will be roughly 800 mV. 800mV / 2mA = 400Ω. That's a starting point. Then you can trade off battery life with brightness from there. Lower resistors will give higher brightness and shorter battery life.