Why Does the TRACE Level Exist and When to Use It Over DEBUG?

As an extension of your rule of logging where in the application the log statement came from, you may want to add per module level logging flags. Instead of logging everything, all the time, this allows you to selectively target sections of your application instead. There is overhead in this, and you need to create a facility that allows you to enable / disable that logging. Ideally, you would be able to enable / disable on-the-fly as the application is running.

I'm used to seeing a layer below debug which I term "Trace", but that's not necessarily a universal term. "Trace" level logging tracks as much as you can possibly stand, including module entry / exit, timestamps with entry / exit, and bonus points for capturing passed values. Obviously, that generates a LOT of data and it's not something you turn on willy-nilly. But it has advantages with respect to debugging when you can't attach to the process or you don't have a core dump of the errant application.

I like to see file / module references and timestamps with my log information. It can be handy when trying to hunt down race conditions between threads as well as coordinating the activities of multiple areas of the application. To be fair, I know of some folk who think these details clutter the log file. Adding in the timestamp is something to discuss with the team. (Apologies if log4j already does that.)

If the logging isn't being taken care of by it's own thread / process, that's something to consider as well. Instead of making the application thread wait for the logging to process, the log message gets passed off to the log handler and the application thread goes on its merry way. Alternatively, creating some sort of buffer mechanism to handle the log messages is another way to speed up application responsiveness.

Having controls on the size and history of log files is another feature to consider. You don't want the app blowing out all the disk space on the host system, nor do you necessarily want to keep all log files for all eternity.

You're right in that this question is very much answered by, "what are your requirements", but this is my opinion.

Be sure that any event initiated by a user is traceable back to the user. Each logged event should state who is executing the event, or at least contains a transaction ID that can be traced back to the user creating the transaction. This will allow you to fulfill non-repudiation requirements where a user would not be able to deny creating the transaction.

I wouldn't recommend creating a new log file for each session. A better approach would be to create new log files once they reach a certain size (10 MB for instance). Or, just creating a new log file each day or week (depending on the size of the files). If there a lot of events being logged, a database may be better as it can handle large volumes of data easier and you will be able to query the results as needed.

For libraries, there are plenty out there for Java EE. Log 4J is fine. I've also used Apache's Commons Logging, but I prefer the combination of SLF4J and Logback personally - they are easy to use and integrate well with other logging frameworks.

You may also want to look into Aspect Oriented Programming to create the common logging events that will appear in a lot of methods. Specifically, AOP can help create debug logging for method enter / exit / throw. I've done AOP with Spring and can say it is very helpful keeping the code clean.

Speaking of logging levels, it will be very beneficial to make good use of each level as appropriate and create a secure way to change the level at runtime if you are experiencing issues.

Log only errors that would be beneficial to have logged. Tautological, maybe, but logging can often get out of hand. Having too many records to sift through means you will not be able to identify outliers or actual problems. Logging too little means you may miss an issue (if you didn't log it, it never happened), so, make good use of logging levels for these reasons as well. There may be no reason to log a user validation error (e.g., creating a new password less than the allowable number of chacters), but system errors should always be logged (e.g., unable to contact the database).

Likewise, only notify the user of what they need to know. They will need to know if the password they entered failed validation and why, but they don't need to know if the database is down (they only need to know that the site is experiencing issues and what they can do in the mean time - i.e., call a support number). Chattiness to the user should be kept to a minimum as constant error notifications that they can't personally resolve would be a poor user experience.

I suggest you take a look at OWASP's Logging Cheat Sheet for some more information on logging best practices.