Restore Testing Gives Database Teams Proof Their Recovery Plan Works

Database teams monitor backup jobs the way they monitor any scheduled task: the job ran, the file landed in storage, and the dashboard is green. However, these signals say nothing about what happens when someone has to bring the database back, how long that takes, or whether the application on top of it starts cleanly. Enterprises running production data on that assumption often learn the difference during an incident.

Mariyan Clement is a Database Architect at IBM who has spent two decades in database architecture across banking, telecom, manufacturing, and energy. Recent engagements cover PostgreSQL performance tuning on AWS and a Sybase to PostgreSQL migration for an Australian bank. He also led a Siebel to PostgreSQL migration that cut a batch process from 48 hours to 45 minutes. To date, Clement has trained more than 1,000 engineers in database technologies.

"A backup isn't a recovery strategy until you've actually restored from it. Most teams check whether the backup job succeeded and whether the backup is sitting safely in storage. The real questions are: Can you restore it? How long does that take? How much data could you lose? And does the application come up cleanly afterward?" says Clement. He treats the restore as the only evidence that counts. A monitoring stack can report success on every backup job in an estate and still leave a team without an answer to any of those questions.

Proving the restore works

Clement tests his own recovery before telling other teams to test theirs. Object storage sits at the end of most backup chains, and pulling a database back out of it is the step teams write into a runbook and skip in practice. "I built a small lab around exactly this," says Clement. "It is a PostgreSQL and S3 setup where I deliberately dropped the database and rebuilt it from object storage, just to go through the full recovery process myself."

Two of the questions he asks map to numbers most teams already carry on paper. Recovery time objective and recovery point objective appear in continuity plans as targets, agreed with the business and signed off. A completed restore turns them into measured figures, and many teams have never compared the two. "A backup gives you a file," he notes. "A restore test gives you confidence."

Learning in the pool

Clement argues for a standing environment where teams can run scenarios, separate from production and available before anyone needs it. Recovery procedures get written once and run rarely, so the first attempt is usually also the first time anyone has read them closely. "Teams should have a safe environment to practice things like accidental deletion, corruption, point in time recovery, failover, and other recovery scenarios before they happen for real," says Clement. "I think of it as learning in the pool before you're in the ocean."

Clement prefers each drill to end with figures attached to it. The scenario has to run far enough to produce them, past the point where a tabletop exercise usually stops. "DR drills shouldn't just live in a document," Clement explains. "Run the scenario, measure how long recovery actually takes and how much data you'd really lose, see what breaks, and document the gaps."

A team that has worked through a failover twice knows which steps take longest and where the written procedure and the live system disagree. Both of those take more than one run to find. "Each drill should make the next real recovery better," he adds. "You don't want your first rehearsal to be the day production is down."

New workloads, same fundamentals

AI workloads land on the same platforms and change what a database team is asked to guarantee. Clement sees the job widening past uptime and latency to the condition of the data itself. "AI workloads bring new data types, vector search, embeddings, higher concurrency, and new data access patterns," says Clement. "Database professionals increasingly need to ask not just 'Is the database up?' but also 'Is this data reliable, governed, secure, performant, and fit for what's consuming it?'"

Clement argues older work carries more weight now, and the controls teams already built are the ones AI systems inherit. "The fundamentals don't go away. Transactions, indexing, query optimization, backup and recovery, security, data quality, and governance remain important," he says. "If anything, those fundamentals matter more, because AI just raises the cost of getting them wrong."

When asked what a company should fix first, Clement returns to the restore. Monitoring reports on each piece separately, and an incident is the first time they all run at once. "You can have backup jobs succeeding, replicas running, and monitoring dashboards looking great, and still find out during a real incident that recovery doesn't work the way you assumed," Clement concludes. "Don't wait for a production incident to find out your recovery plan only worked on paper."