We've 4 embedded firmware at hand. Two of them are released, are in maintenance phase. Other two are going to be released. The released product uses OKI 411 micro, where as yet to be released products are on OKI 431 micro.
So far we've been compiling the code using OKI's CCU8 3.08 compiler. It supports both 411 and 431 hardware. Recently they've updated the compiler to 3.10 and we started using it for all the products.
Suddenly we've found that 3.10 is producing unexpected behaviour in one 431 based project. It is related to compiler optimization. So the release build exhibited a bug and two whole days were wasted to figure it out that the firmware works fine if compiled with 3.08 version of the compiler. Note that the other 431 based product didn't show any issue.
So we've decided to abandon the latest compiler for all projects and use the previous one.
What steps should be taken when this kind of situation occurs? Also when upgrading compiler, what should be done to ensure that it will not break any build?
Best Answer
Often, "upgrading the compiler introduced a bug" really means "upgrading the compiler revealed an existing bug".
This is especially so for C and C++, where the language spec has lots of areas where the behaviour of such-and-such is not defined. Aggressive optimization can causes changes in behaviour for code that strays outside of the envelope of what is defined.
(OTOH, it could be a real compiler / optimizer bug. However, you need some hard evidence before you start pointing fingers.)
There are some ideas:
Recompile with compiler warnings turned up to the max, and go though and fix your code so that the warnings go away.
Run the code through Lint or an equivalent bug checker, and fix every reported problem.
Turn down / off optimization. The down side is that your code will run slower.
Revert to the old compiler. The downside is that you risk getting stuck.
Check out the release notes / forums / issues database for the development suite to see if there are clues to what might have changes.
If you are willing to do some difficult detective work, try and isolate the location of the problem to a particular file, then compare the machine code of the working / non-working versions of the code.
There's nothing you can really do. You just have to try it and see what happens. Of course, that means that you need to be able to revert top the old tool-chain (at least as a short-term fix) if something does go wrong.
The other thing is to get out of the mindset that a broken build is a disaster ... or being embarrassed about. It is only a major problem if the broken build has real consequences for production systems ... or your customers (loosely defined).