Diffing two filesystems

Is there any sensible way to diff two filesystems?

On occassion, you have a box that is working, and, say, perform an update and it no longer works. You have dozens of other machines that are (theoretically) identical, save for user data [and information tied to the specific hardware, and, say, ssh keys and so on]. Is there any way to determine what is different between the failed machine and one of the good machines?

I've tried mounting volumes and running ls -l or md5deep and recording the results and diffing that. Heck, I've even tried mounting two filesystems and just plain diffing them (with a graphical tool, mind you). The results have not been as useful as I'd expect.

Part of me wonders about having a database of files on each machine and what their md5sums are, with a cron job to update it on occasion, but I'm not even sure that would be as useful in practice as it sounds in theory.

So, is there any sensible way to diff two filesystems (or even, diff one filesystem against itself at a different time)?