I work for a software product company. We have large enterprise customers who implement our product and we provide support to them. For example, if there is a defect, we provide patches, etc. In other words, It is a fairly typical setup.
Recently, a ticket was issued and assigned to me regarding an exception found by a customer in a log file that has to do with concurrent database access in a clustered implementation of our product. So this customer's specific configuration may well be critical in the occurrence of this bug. All we got from the customer was their log file.
The approach I proposed to my team was to attempt to reproduce the bug in a configuration setup similar to that of the customer and get a comparable log. However, they disagree with my approach saying that I don't need to reproduce the bug as it's overly time-consuming and will require simulating a server cluster on VMs. My team suggests I simply "follow the code" to see where the thread- and/or transaction-unsafe code is and put in the change working off of a simple local development, which is not a cluster implementation like the environment from which the occurrence of the bug originates.
To me, working out of an abstract blueprint (program code) rather than a tangible, visible manifestation (runtime reproduction) seems difficult, so I wanted to ask a general question:
Is it reasonable to insist on reproducing every defect and debug it before diagnosing and fixing it?
Or:
If I am a senior developer, should I be able to read multithreaded code and create a mental picture of what it does in all use case scenarios rather than require to run the application, test different use case scenarios hands-on, and step through the code line by line? Or am I a poor developer for demanding that kind of work environment?
Is debugging for sissies?
In my opinion, any fix submitted in response to an incident ticket should be tested in an environment simulated to be as close to the original environment as possible. How else can you know that it will really remedy the issue? It is like releasing a new model of a vehicle without crash testing it with a dummy to demonstrate that the air bags indeed work.
Last but not least, if you agree with me:
How should I talk with my team to convince them that my approach is reasonable, conservative and more bulletproof?
Best Answer
You should give it your best effort. I know that sometimes there are conditions and environments that are so complex they can't be reproduced exactly, but you should certainly try if you can.
If you never reproduced the bug and saw it for yourself, how can you be 100% certain that you really fixed it? Maybe your proposed fix introduces some other subtle bug that won't manifest unless you actually try to reproduce the original defect.
I would not trust someone who runs the code "in their head", if that's their only approach. It's a good place to start. Reproducing the bug, and fixing it and then demonstrating that the solution prevents the bug from reoccurring - that is where it should end.
Because if they never reproduced the bug, they can't know for certain that it is fixed. And if the customer comes back and complains that the bug is still there, that is not a Good Thing. After all, they are paying you big $$$ (I assume) to deal with this problem.
If you fail to fix the problem properly, you've broken faith with the customer (to some degree) and if there are competitors in your market, they may not remain your customer.