Is it possible for an out-of-control (e.g. due to stack corruption) user application to inadvertently invoke the bootloader sector code in an AVR (e.g. ATmega1284p)? Said differently, is it possible for self programming instructions in the bootloader sector to execute at any other time but on a reset?
What I'm worried about is runaway code inspiring the bootloader to corrupt the application image. If that's plausible, what if anything can be done to prevent it or at least minimize the impact of it happening.
Best Answer
It can happen yes. Imagine you somehow execute a jump instruction that happens to land in the bootloader code. It will execute quite happily - say it hit an SPM instruction.
Part of the solution to this is to have a way of detecting that whether the bootloader was entered as a result of a power on reset - information in one of the status registers of an AVR. If it wasn't then jump back to the reset vector. This works well to minimise the risk of runaway execution if the program counter keeps counting through everything in the flash and gets to the bootloader code - the PORF flag won't be set, so it will get sent back to the reset vector.
However this does not protect from inadvertently jumping in to part of the bootloader routines with a branch or IJMP type instruction. In this case what are the options to stop what would ensue.
Fortunately the SPM instruction requires a timed sequence of register writes (write to SPMCSR then within 4 cycles call SPM) to get it to execute, so if you hit just an SPM instruction it wouldn't do anything - your run away would have to run through the full programming sequence. But this just means there is slightly less chance of an accident - though it doesn't prevent it.
One option you could have in your bootloader is your own check during the timed sequence to see if the reset flags are set. The bootloader is the first thing that runs, and it can clear these reset source flags once it exits. That way if you ever accidentally jump into the SPM sequence it could not complete the sequence as it was executed from something that wasn't the bootloader - because the reset flags would have been cleared by the bootloader already.