TDD – Do We Need Logging When Doing TDD?

I feel there are two issues here. The first is that you didn't realize in advance that your original design may not be the best approach. Had you known this in advance, you may have chosen to develop a quick throw-away prototype or two, to explore the possible design options and to assess which is the most promising way to follow. In prototyping, you need not write production quality code and need not unit test every nook and cranny (or at all), as your sole focus is on learning, not on polishing the code.

Now, realizing that you need prototyping and experiments rather than starting the development of production code right away, is not always easy, and not even always possible. Armed with the knowledge just gained, you may be able to recognize the need for prototyping next time. Or may not. But at least you know now that this option is to be considered. And this in itself is important knowledge.

The other issue is IMHO with your perception. We all make mistakes, and it is so easy to see in retrospect what we should have done differently. This is just the way we learn. Write down your investment into unit tests as the price of learning that prototyping may be important, and get over it. Just strive not to make the same mistake twice :-)

As an extension of your rule of logging where in the application the log statement came from, you may want to add per module level logging flags. Instead of logging everything, all the time, this allows you to selectively target sections of your application instead. There is overhead in this, and you need to create a facility that allows you to enable / disable that logging. Ideally, you would be able to enable / disable on-the-fly as the application is running.

I'm used to seeing a layer below debug which I term "Trace", but that's not necessarily a universal term. "Trace" level logging tracks as much as you can possibly stand, including module entry / exit, timestamps with entry / exit, and bonus points for capturing passed values. Obviously, that generates a LOT of data and it's not something you turn on willy-nilly. But it has advantages with respect to debugging when you can't attach to the process or you don't have a core dump of the errant application.

I like to see file / module references and timestamps with my log information. It can be handy when trying to hunt down race conditions between threads as well as coordinating the activities of multiple areas of the application. To be fair, I know of some folk who think these details clutter the log file. Adding in the timestamp is something to discuss with the team. (Apologies if log4j already does that.)

If the logging isn't being taken care of by it's own thread / process, that's something to consider as well. Instead of making the application thread wait for the logging to process, the log message gets passed off to the log handler and the application thread goes on its merry way. Alternatively, creating some sort of buffer mechanism to handle the log messages is another way to speed up application responsiveness.

Having controls on the size and history of log files is another feature to consider. You don't want the app blowing out all the disk space on the host system, nor do you necessarily want to keep all log files for all eternity.