Electronic – Best way to diagnose embedded computer in field problems

debuggingembedded

I'll soon be deploying a new embedded system. I have a separate serial port that can connect to a diagnostic program. I also have two LEDs, one I flash to indicate the OS is running, the other that the application is running. I fear that it's going to be hard to work out what happened if the embedded system experiences a total crash however. It does have a watchdog so would reboot, but I'd like to be able to work out why the crash occurs, if it does.

The only way I've done this in the past is to have a parallel system back in the lab, give it the same type of inputs, try and trigger the problem, then analyze through debug prints or outputs to digital I/Os to figure out what could be causing it. Often it is very hard to replicate the problem however.

Does anyone have any advice on good methods they have for debugging in field problems?

Thank you,
Fred

Best Answer

Recently, I deployed a few units of a new embedded product for a friendly trial. I was trying to discover how well the IP stack performed out in the real world, so using the on-board ethernet for debug wasn't an option.

I bought several serial dataloggers from Sparkfun (many similar devices are available) and attached one to each unit over TTL serial. When I got the units back, I retrieved the logs from the SD cards.

In the case of a system crash, dumping registers out to the serial port will let you find the offending code via the map file later.