What are the best practices for managing Test results

It is useful to add hooks, as suggested, to recreate exact states. Also instrument the system so that it can dump its "seeds" (in your case, including the PRNG seed as well as the kernel ring buffer, and any other sources of nondeterministic input.)

Then run your tests both with true random input, and regression-style with any previously-discovered interesting cases.

In the particular case of your access to the kernel, I'd recommend making a mock in any case. Use the mock to force equivalence classes that are less likely to show up in practice, in the spirit of "empty" and "full" for containers, or "0, 1, 2^n, 2^n+1, many" for countable things. Then you can test with the mock and with the real thing, knowing that you have handled and tested the cases you've thought of so far.

Basically, what I'm suggesting amounts to a mix of deterministic and nondeterministic inputs, with the deterministic ones being a mix of those you can think of and those you were surprised by.

Functional tests are very important. Yes, they take time to write but if you are writing the right functional tests, they will be more than worth it.

There are a few good reasons to do automated functional tests on an application.

• When a new feature is added to your web site, it let's you know right away if changes made for that new feature break any other functionality on your site.
• It's documented knowledge of how the application runs and works together to achieve the business requirements.
• When it's time to update a 3rd party library, you can update it and run your functional test suite to see if anything breaks. Instead of having to go through every page yourself, you can have a computer do it for you and give you a list of all the tests that broke.
• Load testing! You can simulate thousands of simultaneous users all hitting your site at once and you can see where your site slows down and buckles under the pressure. You can see how your web site behaves long before you get a late night call that the site has crashed.
• Functional testing takes time to do manually. Yes, it takes long to write the cases, but if you had to sit down with a binder with 500 pages of tests that you had to complete before you could ship the product you'd wish you had the automated tests!
• Testing documents get out of date fast. When a new feature is added, you have to make sure to update the master testing document. If someone skips some tests you all of a sudden get bugs creeping into pages that are "done and tested". I currently work in an environment like that, and I can assure you, it's a nightmare.

In the end, yes it takes time to write these cases, but you should take pride in writing them. It's your way of proving, beyond a shadow of a doubt that your code works and it works with all the other features out there. When QA comes to you and says there is a bug, you fix it, and then add it to your test suite to show that it's fixed and make sure it never happens again.

It is your safety net. When someone goes in and hijacks a stored proc and makes a small change so it'll work with their code, you'll catch that it has broken 3 other features in the process. You'll catch it that night and not the night before the deadline!

As for writing functional tests for only system critical functions. That won't give you the whole picture and it'll allow bugs to sneak through. All it take is for one little feature to be added that isn't system critical, but interacts indirectly with a system critical function and you have the potential to have a bug introduced.