Runbook Documentation: Writing Runbooks People Actually Use

Here is the uncomfortable truth about runbooks: they are written by a person who knows the system, on a calm afternoon, with coffee — and executed by a person who may not, at 3 a.m., with a service down and a phone buzzing. The gap between those two situations is where runbooks fail.

The author writes "restart the service" because to them it is obvious which service, on which host, with which command. The 3 a.m. reader has three services it could plausibly mean and no one awake to ask. The author skips the verification step because they can tell at a glance whether it worked. The reader cannot, so they either declare victory too early or sit there refreshing a dashboard, unsure.

So the standard for a runbook is not "is it accurate" — it is: can a competent engineer who did not write it, and does not know the system, execute it under pressure without asking anyone a question? Every choice below follows from that test.

A runbook that holds up has the same five parts, and the order is part of the design — it mirrors the order in which the reader needs each answer.

1. Scope.One or two sentences: what this runbook accomplishes, and — just as important — what it does not cover. "Restores the database from the most recent nightly backup. Does not cover point-in-time recovery or replica rebuilds." The reader decides in ten seconds whether they are holding the right document. A wrong runbook confidently executed is worse than no runbook.
2. Preconditions. What must be true before step one: required access, services that must be stopped, approvals needed, the amount of disk space the restore will consume. Preconditions discovered at step six become disasters; preconditions checked up front are a thirty-second checklist.
3. Steps. Numbered, exact, one action each. The commands themselves — not descriptions of commands — with expected output noted wherever the reader might wonder if it worked.
4. Verification.How to prove the procedure succeeded: the specific checks, the specific expected results. "It should be fine now" is not verification; "the health endpoint returns 200 and the queue depth is falling" is.
5. Rollback. How to get back to the starting state if the procedure fails or makes things worse. Some procedures have no clean rollback — say so explicitly, because knowing a step is irreversible changes how carefully the reader approaches it.

If your current runbooks are missing sections, it is almost always the last two — and the last two are the ones an incident actually tests.

The steps section is where good intentions meet bad habits. A few rules, each learned the hard way by someone:

• Exact commands, real names. Write the command the reader will run, with the actual hostname, path, and flags. Where a value varies, use an unmistakable placeholder and say where to find the real value.
• One action per step."Stop the service, clear the cache, and restart" is three steps wearing one number. When step four fails, you want to know exactly which action failed.
• Show expected output. After any command whose result is not obvious, note what success looks like. This is the single cheapest improvement you can make to an existing runbook.
• Flag the dangerous steps. If step seven is irreversible or disruptive, mark it before the step, not after. The reader should feel the speed bump before they cross it.
• Cut the theory. Background on why the system works this way belongs in the knowledge base, linked from the runbook. Under pressure, every paragraph between the reader and the next command is friction — and friction at 3 a.m. means skipped reading.

Then run the only test that counts: hand the draft to a teammate who has never done the procedure and watch them execute it in a safe environment. Every question they ask out loud is a bug in the runbook. Fix the document, not the reader.

You cannot runbook everything, and you should not try — a library of two hundred stale runbooks is worse than a library of fifteen trusted ones. Prioritize by pain and by rarity:

• Anything that pages people. Service recovery, failover, restore from backup — the procedures executed under the most pressure by whoever happens to be on call.
• Rare-but-scheduled work. Certificate renewals, quarterly failover tests, the yearly license true-up — done just seldom enough that everyone half-forgets the details.
• Procedures with a blast radius. Anything where a mis-step takes down production or loses data earns exact steps, even if it is performed by experts.

What does not need a runbook: routine work the whole team does weekly from muscle memory, and one-off tasks that will never recur. Writing those anyway is how libraries bloat and trust erodes.

A runbook is a claim about how a system behaves, and systems change. Without maintenance, every runbook drifts from asset to hazard — the confident document that walks the 3 a.m. engineer into commands that no longer do what the page says they do. A stale runbook is more dangerous than a missing one, because a missing one at least announces itself.

Three habits, none of them heroic, keep the library true:

• Fix on use. Whoever executes a runbook and hits a discrepancy fixes it immediately — same hour, while the difference between the page and reality is in front of them. This is the highest-value documentation work that exists, and it costs minutes.
• Update on change. A change to a system is not done until its runbooks reflect it. Put it in the ticket template; make it part of the definition of done, not a favor someone remembers later.
• Date everything. Every runbook carries a last-verified date. Readers calibrate trust instantly, and a quick periodic sweep for old dates tells you exactly which runbooks need a test run — or retirement.

Runbooks are one piece of a larger discipline — infrastructure records, disaster recovery plans, and a knowledge base round out the set, and they reinforce each other. For how the pieces fit together, start with our complete guide to IT documentation. But if you do only one thing after reading this: pick the procedure that most recently paged someone, and write its runbook this week — scope, preconditions, steps, verification, rollback. The next 3 a.m. belongs to whoever is on call. Write for them.