Documentation – As-A-Manual vs. As-A-Checklist for Disaster Recovery

disaster-recoverydocumentationprocess

I've had discussions in the past with other people in my department about documentation, specifically, level-of-detail and requirements. In their view, documentation is a simple checklist of Y things to do when X things go wrong.

I disagree. I think that this presumes that all issues in IT can easily be boiled down to simple checklists of recovery procedures. I think that it completely ignores the complexity of the situation, and as the other people in the department don't always have a depth of understanding about the issue (which is why I'm writing the document – so they have something to refer to) that the documentation should include some basic background material, such as:

  • Purpose of the (sub)system in question
  • Why it is configured in that manner
  • Expectations of events to occur when the settings/procedures are implemented
  • Potential issues that can cause procedures to fail

However, I'm rather outvoted on this, so my documentation is required to be re-written into a form that says "Steps A-B-C applied in order will resolve problem X". I often hear the lament that it needs to fit onto a single page of paper. Try explaining the configuration of Squid ACLs to someone in this manner, including troubleshooting, through a single-page document. That's just one of a half-dozen documents that are "waiting to be written" as recovery checklists.

Is the method I'm advocating really going overboard? Or are they right, and I should just mind my business here and just write them a simple checklist? My concern is that, no matter how well you write a procedure checklist, it really doesn't solve an issue that requires a SysAdmin to think things through. If you're spending time doing a checklist of recovery procedures that ends up not resolving the issue (because there are additional factors that aren't a part of the document, due to the narrow focus of the document), and the purpose of the document was to avoid re-reading man pages and wikis and websites all over again, then why am I going through the motions? Am I just worrying too much, or is this a real issue?

EDIT:

There currently is no helpdesk position in the department. The audience for the documentation would be for the other admins or for the department head.

Best Answer

When writing mine I've always devolved into writing two three sets. The get-er-done checklist, with a MUCH LONGER appendix about the architecture of the system including why things are done the way they are, probable sticking points when coming online, and abstract design assumptions. followed by a list of probable problems and their resolutions, followed by a longer section with information about how a system works, why it does it that way, and other information useful for pointing people in the right direction should something unique happen.

At my last job we were required to write doc so that even level-1 helpdesk people could bring things back up. This required checklists, which generally became out of date within 3 months of the writing. We were strongly urged to write troubleshooting guides whenever possible, but when the contingency tree gets more than three branches in it, you just can't write that doc without going abstract.

When leaving my last job, I turned in a 100 page 'how to do my job' manual before I left. It had the abstract stuff in it, design philosophy, as well as integration points. Since I was presumably writing for another sysadmin who was going to replace me, I aimed it at someone who could take abstract notions and turn them into concrete actions.

Five years have passed and I find my opinion on this has shifted somewhat. Both Document as Manual and Document as Checklist have very valuable places in the hierarchy of documentation and both need to be produced. They target very different audiences, though.

Document as Checklist

The target market for this kind of documentation are coworkers who want to how how to do a thing. They come in two types:

  • Coworkers who just want to know how to do a thing and don't have time to thumb through a fifteen page manual and figure out the steps for themselves.
  • Procedures that are fairly complex in steps, but only need to be run once in a while.

Impatience is the driver for the first kind. Maybe your coworker doesn't actually want to know why the output has to be piped through a 90 character perl regex, just that it has to be in order to close the ticket. Definitely include a statement like, "For a detailed explanation for why this workflow looks like this, follow this link," in the checklist for those that do want to know why.

The second point is for procedures that aren't run often but contain pitfalls. The checklist acts as a map to avoid the Certain Doom of just winging it. If the checklist is kept in a documentation repo, it saves having to search email for the time the old admin sent out a HOWTO.

In my opinion good checklist-documentation also includes sections on possible failure points, and responses to those failures. This can make the document rather large and trigger TL;DR responses in coworkers, so I find that making the failure-modes and their responses a link from the checklist rather than on the page itself makes for an unscary checklist. Embrace hypertextuality.

Document as Manual

The target market for this kind of documentation are people who want to learn more about how a system works. The how-to-do-a-thing style documentation should be able to be derived from this documentation, but more commonly I see it as a supplement to checklist-style documentation to back up the decisions made in the workflow.

This is the documentation where we include such chewy pieces like:

  • Explaining why it's configured this way.
    • This section may include such non-technical issues like the politics surrounding how the whole thing was purchased and installed.
  • Explaining common failure modes and their responses.
  • Explaining any service-level-agreements, both written and de facto.
    • De facto: "if this fails during finals week it's a drop-everything problem. If during summer break, go back to sleep and deal with it in the morning."
  • Setting out upgrade and refactoring goals.
    • The politics may be different later, why don't we fix some of the bad ideas that introduced in the beginning?

Which are all very useful for obtaining a comprehensive understanding of the whole system. You don't need a comprehensive understanding to run simple human-automation tasks, you need it to figure out why something broke the way it did and have an idea where to make it not do that again.

You also mentioned Disaster Recovery documentation that has to be a checklist.

I understand, you have my sympathies.

Yes, DR documentation does need to be as checklist-like as possible.
Yes, DR documentation is the most resistant to checklisting due to how many ways things can break.

If your DR checklist looks like:

  1. Call Dustin or Karen.
  2. Explain the problem.
  3. Stand back.

You have a problem. That is not a checklist, that is an admission that the recovery of this system is so complex it takes an architect to figure out. Sometimes that's all you can do, but try to avoid it if at all possible.

Ideally DR documentation contains procedure checklists for a few different things:

  • Triage procedures to figure out what went wrong, which will help identify...
  • Recovery procedures for certain failure-cases. Which is supported by...
  • Recovery scripts written well beforehand to help minimize human error during recovery.
  • Manual-style documentation about the failure cases, why they occur and what they mean.

Triage procedures are sometimes all the DR documentation you can make for some systems. But having it means the 4am call-out will be more intelligible and the senior engineer doing the recovery will be able to get at the actual problem faster.

Some failure cases have straight-forward recovery procedures. Document them. While documenting them you may find cases where lists of commands are being entered in a specific order, which is a great use-case for scripting; it can turn a 96 point recovery procedure into a 20 point one. You'll never figure out if you can script something until you map the recovery procedure action by action.

Manual-style documentation for failure cases is the last ditch backstop to be used when there ARE no recovery procedures or the recovery procedures failed. It provides the google-hints needed to maybe find someone else who had that problem and what they did to fix it.

Related Topic