Two different questions, one green checkmark
Every backup platform reports success the same way: a green checkmark, a completion email, a dashboard that says "all jobs healthy." That checkmark answers a narrow question — did the backup process run to completion without an error. It says nothing about whether the data it produced can actually be turned back into a working system, a mountable mailbox, or a readable file.
Those two questions feel like the same question. They are not. A job can report success every single night for a year while producing backups that would fail to restore on the night you actually need them — and the monitoring dashboard would never show a single red flag, because nothing about the failure shows up until someone tries to use the backup for its actual purpose.
What verification checks — and what it can't
Modern backup software is good at verification: checksums confirm the bytes written match the bytes read back, catalog integrity checks confirm the backup's internal index isn't corrupted, and job logs confirm every step in the process reported success. All of that is real and worth having. None of it tests the thing you actually care about.
| Verification checks | Only a restore test checks |
|---|---|
| The backup job completed without error | The restored system actually boots, or the restored database actually mounts |
| Checksums match on the stored data | Application-consistency was preserved, not just file-level consistency |
| The backup catalog/index isn't corrupted | A single file, mailbox, or table can be recovered individually, not only the full set |
| Storage capacity and retention policy are healthy | The restore finishes inside your actual recovery time objective (RTO) |
| The schedule ran when it was supposed to | The written restore procedure is complete, current, and followable by someone other than its author |
Failure modes that hide behind a green checkmark
These are common, well-documented, and every one of them can coexist with a perfect verification history:
- Application-inconsistent snapshots. A backup can capture a database or a Windows VSS snapshot at a moment when the application's writer hasn't finished flushing transactions. The backup completes; the restored database won't mount, or mounts in a corrupted state.
- Broken incremental chains. A single corrupted incremental in the middle of a chain can silently invalidate every backup that depends on it, while each individual nightly job continues to report success on its own.
- Metadata and permission loss. Files restore with the right content but the wrong ownership, ACLs, or timestamps — invisible until something downstream (an application, a compliance audit, a login) depends on that metadata being correct.
- Bit rot and media degradation. Storage media doesn't always fail loudly. Data written correctly a year ago can become unreadable in ways checksums only catch if they're run against the original write, not just verified once at write-time and never again.
- Undocumented tribal knowledge. The backup administrator knows the three manual steps that aren't in the runbook. They're on vacation, or gone, when the restore is actually needed.
- RTO that was never actually measured. A restore that technically works but takes fourteen hours is a very different outcome than one that takes ninety minutes, if your business can only tolerate two hours of downtime. Nobody finds this out until they time an actual restore.
A practical testing cadence
Verification should run on every job, automatically, as it already does in any competent backup platform. Restore testing needs its own deliberate schedule, because it will never happen by accident:
- Quarterly, at minimum. A full restore test for every critical system, on a schedule that doesn't get skipped when things get busy — busy is precisely when a bad backup gets discovered the hard way.
- Restore to an isolated environment. Never test a restore by writing over production. An isolated sandbox proves recoverability without creating a second incident while testing for the first one.
- Time it against your actual RTO. A stopwatch, not a guess. If the real number doesn't meet the business requirement, that's a finding worth having in peacetime rather than during an actual outage.
- Rotate who performs the test. Have someone other than the usual backup administrator run the restore, following only the written documentation. Gaps in the runbook surface immediately when the person running it doesn't already know the undocumented steps.
- Test individual-item recovery, not just full-system recovery. The most common real-world request is "get me back this one mailbox" or "restore this one folder," not "rebuild the whole server." Confirm both are actually possible.
A simple maturity ladder
Useful for sizing up where an environment actually stands, not just where its dashboard says it stands:
| Level | What's true |
|---|---|
| 1 — Hope | Job completion is monitored. Nothing has ever actually been restored. |
| 2 — Verified | Checksums and catalog integrity are checked automatically. Still no real restore has been performed. |
| 3 — Spot-tested | Individual files or mailboxes have been restored successfully, at least once, somewhat recently. |
| 4 — Proven | Full-system restores are tested on a fixed schedule, timed against RTO, performed by rotating staff, to an isolated environment, with results documented. |
Most environments believe they're at Level 4 and are actually at Level 1 or 2 — because verification looks and feels like proof, right up until the day it isn't.
The lesson
This is the expanded version of a principle we hold to across everything we build, not just backups: verification over trust. A patch isn't deployed until it's confirmed. A security control isn't real until it's been exercised. And a backup isn't a backup — it's a hope — until it's been restored, on a schedule, by someone who didn't write the runbook, inside the time your business can actually afford to wait.