Troubleshooting Code – How to Test Without Hardware Setup

hardwareprogramming practicestest-automationtesting

I work at a mid-sized company (150ish employees, ~10 size engineering team), and most of my projects involve interfacing with lab equipment (oscilloscopes, optical spectrum analyzers, etc) for the purpose of semi-automated test applications. I have run into a few different scenarios where I am unable to efficiently troubleshoot or test new code because I no longer or never had the hardware setup available to me.

Example 1: A setup where 10-20 "burn-in" processes are run independently using a bench top type sensor – I was able to obtain one such sensor for testing and could occasionally steal a second for simulating all of the facets of interfacing to multiple devices (searching, connecting, streaming, etc).

Eventually a bug showed up (and ultimately ended up being in the device firmware & drivers) that was very difficult to reproduce accurately with only one unit, but hit near "show stopper" levels when 10-20 of these devices were in use simultaneously. This is still unsolved and is ongoing.

Example 2: A test requiring an expensive optical spectrum analyzer as its core component. The device is pretty old, legacy according to the manufacturer who was acquired by a larger company and basically dissolved, and its only documentation was a long winded (and uninformative) document that seems poorly translated. During initial development I was able to keep the device at my desk, but now its tied up, both physically and in schedule during its 24/7 multi-week tests.

When bugs show up related or unrelated to the device, I often need to go through the trouble of testing code external to the application and fitting it in, or writing code blindly and attempting to squeeze in some testing time in between runs, as much of the program logic requires the OSA and the rest of the test hardware to be in place.

I guess my question is how should I approach this? I could potentially spend time developing device simulators, but figuring that into the development estimate will balloon it more than most would probably appreciate. It may not accurately reproduce all issues either, and it's pretty rare to see the same equipment used twice around here. I could get better at unit testing…etc…I could also be loud about the issue and make others understand that temporary delays will be required, not much more than a headache for Research and Development but usually a perceived as a joke when pitched to manufacturing.

Best Answer

Management understands it will take longer to develop and maintain software when you don't have full access to test hardware. You need to take this into account when doing your estimates. Part of the acceptance criteria for putting your software into production should be that you have a way to maintain the software under most circumstances without stopping manufacturing. If you're practicing TDD, this should happen pretty much naturally.

I used to write software for $60 million aircraft. Obviously, there's a high degree of reliability required, and they are reluctant to give every developer one for their desk. We basically had 5 levels of test environments, with more of the real hardware at each level, up to a full aircraft. I estimate 95% of our software could be developed and debugged only with emulators and unit tests. 95% of the remaining features could be worked on the next level up, and so on.

Try to set up similar levels of test environments for yourself. You can't expect to never need access to the real hardware, but if you've set it up so you can't work on your software's GUI without the hardware available, you're wasting valuable time on an expensive resource (not to mention you have some coupling issues with your architecture). Consider that other developers likely have the same issues as you. I would ask the hardware vendor if they already have emulators or other test resources available.

You also need to change your mindset somewhat if you only have limited access to hardware. Rather than trying to debug your application in the normal serial manner, you often need to write code specifically for the purpose of gathering information as quickly as possible.

For example, perhaps you have a bug and you can think of 10 possible causes. If the only time you can get on a machine is the 15 minutes while the operator is on break, write a Short, Self Contained, Correct (Compilable), Example that triggers the bug and write 10 automated tests using that SSCCE to test your theories and log a bunch of data. Afterward back at your desk you can take as long as you need to sift through the data for your next attempt. The idea is to maximize the utility of your limited time with the hardware.

Related Topic