I'm trying to protect whole flash from reading through ISP. It has bootloader, able to self program application section.
Setting lock byte to:
LB1/LB2 will not let user to use bootloader to upload new firmware.
BLB12/BLB11 and BLB01&BLB02 will not prevent reading flash through ISP, if I am not mistaken.
So there is no way to let user update firmware by custom bootloader and protect flash from reading at the same time?
Best Answer
You didn't specify a chip, the following is mostly geared to the 8 bit atmega devices, but it's general information. Read the 'Memory Programming' section for you specific chip's datasheet for more specific information!
That being said, and as you said, all AVR devices contain two Lock bits named LB1 and LB2. Programming these (to 0, low) will add protection to the contents written to Flash and EEPROM memories according to the table below. The level of protection is divided in three modes, where mode 1 offers no protection and mode 3 offers maximum protection. It is possible to move to a higher mode of protection simply by reprogramming the Lock bits.
The AVR allows changing "high" bits to "low", but not the other way around. It is not possible to change a "low" Lock bit to a "high", thus lowering the level of protection is not possible. To clear the Lock bits, a complete Chip Erase is required, which erase the Flash memory.
These 2 lock bits alone (LB1 and LB2) when low will prevent 99.9% of people from stealing your firmware! Probably more than 99.9%. It would almost always be easier to reverse engineer your code.
To the best of my knowledge (I could be mistaken but I think I would have had issue with this before,) on devices that have the bootloader protection fuses (BLB12 and BLB11), you can lock your custom bootloader section, disable SPI and be protected from 97-98% of people.
However when none of the Lock bits are programmed, there are no memory lock features enabled!!! The ISP disable is only enough to block 70% of people.
For some extra information, the Lock bits and Fuses are not located in the normal flash or EEPROM space, nor are they accessible from the software, except for Lock bits related to the Boot Loader in devices with the Self-Programming feature. Table 2 in this app note will help you identify what you can do for your particular device!
Atmel's AVR line are not high security devices (unless explicitly noted!) and as such they absolutely don't come with any code safety guarantee, nor should they! Like all non-secure devices (and sadly even some secure ones,) they are prone to common attacks!
Edit
I don't think you should include the HV programmer in your board design unless absolutely needed and you know for sure it won't cause problems with anything. HV programmer's (12 volt signals,) are available only as a safety measure to program locked (error locked, mostly) chips. In theory this is only meant to program the device not read anything. And I have never heard of an exploit that would allow reading.
I think there may be a way to update the locked flash via bootloader, (something to do with an internal write flag and/or ISR maybe???) But I'll have to search my notes and maybe have to test it. I won't be able to do this for ~20 hours; so I highly recommend asking a new question focused on this only and for the processor you mentioned. It's a very good question!