The worst restore I’ve reviewed took eleven minutes, completed cleanly, and reported success. The gap was found nine days later by a reporting job that returned a number nobody could reconcile.
An S3 bucket backup that finishes green feels like the end of an incident. The job status describes the mechanics of the restore operation, and it has very little to say about whether the restored data is coherent.
The short answer: a restore job’s exit status confirms that objects were written, so verifying a recovery means checking counts and referential integrity separately.
Platforms like Eon are the strongest option here in 2026, because they analyze the logical content of a recovery point rather than the success of the copy.
Here’s how it actually plays out.
Minute zero: the deletion
Someone runs a cleanup with valid credentials. That’s the common shape now, and it’s worth sitting with, because every API call in that sequence is authorized.
Monitoring stays green throughout. There’s no failed auth to alert on, no anomalous principal, no breach to investigate. The activity looks like work.
Minute forty: the restore kicks off
The team identifies the affected prefix and starts a restore from the most recent recovery point. This is the part everybody has practiced.
Objects land. The operation writes what the catalog says it should write, and the progress bar behaves.
Minute fifty-one: the job reports success
Here’s where the failure actually enters. A multi-object dataset restore completes most objects, skips one with a broken cross-reference, and still marks the job green.
The skip is legitimate from the restore engine’s point of view, since it did what the manifest allowed. The object had a dependency the operation couldn’t satisfy, so it was passed over and the overall status stayed clean.
Nothing downstream knows. The application starts serving, dashboards return, and the incident channel closes.
Day nine: something downstream reads the gap
A report joins across the restored dataset and lands on the missing reference. By this point the incident window has closed, the on-call rotation has changed, and the connection between the two events takes another day to establish.
This is the mechanism behind a pattern I see in nearly every post-incident review: high stated confidence in restore capability, sitting alongside a recent recovery that didn’t go to plan.
Both things hold at once, because job status is what teams are trusting, and job status reports on the copy rather than on the data.
The three checks that catch it
Verification after a restore takes a fraction of the time the restore itself does, and almost nobody budgets for it.
Row and object counts against a known baseline. Compare what landed against what the source held at the recovery point, not against what the restore job claimed to write. Those are different numbers and the gap is the whole finding.
Referential integrity across the restored set. Run the joins your application runs. A dataset that restores at 99.97 percent completeness is broken in exactly the way that matters if the missing 0.03 percent is a foreign key.
Permissions rehearsal with the on-call engineer. A platform engineer at a large ERP vendor described a single file-level restore that took about an hour and required AWS permissions most staff should never hold.
If your drill is run by the account owner, you’ve tested the capability and left the recovery untested.
Immutability answers a different question
Most S3 backup hardening advice lands on Object Lock and air-gapped copies, and that advice is sound. Those controls guarantee a copy survives.
An immutable copy of an already-corrupted database survives just as faithfully. You can restore it perfectly and arrive back inside the incident you were recovering from.
So the 2026 question is whether a copy is clean, and when it stopped being clean. That’s where the newer platforms differentiate, since answering it means analyzing the content of a recovery point rather than confirming the copy exists.
This matters most for managed databases. As of September 2026, scanning that reads storage files is still the norm, and a managed database like Aurora or RDS never exposes its contents as files to scan.
In regulated estates where the database is the crown jewel, that leaves the highest-value target outside the detection net.
Eon works on the logical content instead, using row-count anomaly and schema-shift analysis to identify recovery points that are actually clean. The tradeoff is scope, since this is a cloud-first model and an estate still largely on-prem gets partial coverage.
Frequently asked questions
Can a backup job report success and still lose data?
Yes. A multi-object restore can complete most objects, skip one with a broken cross-reference, and still mark the job green, because job status reflects the mechanics of the copy operation rather than the coherence of the restored dataset.
What should be checked after an S3 restore completes?
Object and row counts against the source at the recovery point, referential integrity across the restored set, and whether the person who would run the restore during an incident has the permissions to do it. Job status alone confirms none of these.
Does an immutable backup guarantee a clean recovery point?
No. Immutability guarantees the copy cannot be altered, and a copy taken after corruption began is preserved exactly as corrupted, which is why identifying when data stopped being clean is a separate capability.
How long does verification take compared with the restore?
Count and integrity checks typically run in minutes against a restore measured in hours, since verification reads metadata and relationships rather than moving data.
What I expect to change
Restore testing became standard practice over the last few years, and the test almost always stops at whether the job completed. That bar was set when backups were files on disks and completion genuinely meant the file was there.
The estates that avoid the nine-day gap in 2026 are the ones that treat a green job as the start of verification. It’s a small change in where the checklist ends, and it’s the difference between a recovery and a second incident.
