A Backup Is Not Recovery Evidence

A green backup job proves that a job completed.

It may prove that the backup server accepted bytes, created metadata, and returned success. It does not prove that the archive contains the required data, that the encryption key is available, that the application can start, that dependencies can be reconstructed, or that the restored system will not collide with production.

I stopped giving full recovery credit to backup completion after working through several isolated restore tests. Some backups were technically valid but incomplete for the service. Others restored a virtual machine that carried the original system's identity. One scheduled job tried to include the backup server itself. Another guest referenced installation media that no longer existed. Those are not theoretical edge cases. They are ordinary configuration defects that only become visible when recovery is exercised.

The prior post, Retiring a Security Platform Without Bringing It Back by Accident, treated retained backups as isolated archives. This post explains what has to happen before I call any backup recoverable.

Backup, verification, restore, and recovery are different gates

I separate four claims:

Repository verification is valuable. Proxmox Backup Server can schedule verification jobs and track snapshots that need reverification. Restic provides repository consistency checking and can read repository data during deeper checks. Borg's check command verifies repository and archive consistency, with options that include data verification.

None of those checks proves that an operator can rebuild a service when the primary environment is unavailable.

A database file can be internally readable while application migrations are missing. A virtual-machine image can boot while DNS points to the wrong place. A file archive can restore while ownership, ACLs, xattrs, capabilities, or encryption material are absent. A secrets platform snapshot can be intact while the unseal shares are trapped inside that same platform.

Recovery is a system property.

Define the recovery objective before choosing the backup

RPO and RTO are engineering inputs, not labels added after deployment.

Recovery point objective answers how much data loss is acceptable. Recovery time objective answers how long the service can remain unavailable. Both should come from service impact and dependency analysis.

A service with a one-hour RPO needs capture, replication, retention, and verification capable of producing an accepted recovery point at that interval. A four-hour RTO must include detection, decision, operator access, infrastructure provisioning, data transfer, restore, application startup, dependency repair, validation, and traffic cutover.

The clock does not start when the restore command runs. It starts when the disruption begins or when the continuity plan defines it.

I record the objective and measured result together:

service: secrets-platform
recovery_tier: critical-internal
objective:
  rpo_seconds: 21600
  rto_seconds: 14400
exercise:
  scenario: loss-of-primary-application-host
  selected_recovery_point: 2026-09-06T00:00:00Z
  recovery_point_age_seconds: 11800
  started_at: 2026-09-06T09:00:00Z
  service_accepted_at: 2026-09-06T10:47:00Z
  measured_recovery_seconds: 6420
result:
  rpo: pass
  rto: pass

Do not stop the timer at "files restored." Stop it at the accepted service boundary.

NIST SP 800-34 Revision 1 describes contingency planning as a coordinated process that includes business impact analysis, recovery strategies, testing, training, exercises, and plan maintenance. That broader model is useful because storage is only one dependency in recovery.

Capture application state deliberately

Virtual-machine snapshots are convenient. Their consistency properties depend on what the workload was doing when capture occurred.

Crash-consistent capture approximates abrupt power loss. Filesystems and applications must recover from whatever was in memory or in flight. Application-consistent capture coordinates the workload so its stored state is recoverable under the application's rules.

Neither term should be accepted without evidence. A guest-agent freeze may improve filesystem consistency, but it does not automatically create a valid distributed database backup. An application hook may dump a database correctly while omitting configuration, keys, or attachments.

For each service, identify:

• authoritative databases and transaction logs
• application files and uploaded content
• configuration and secrets
• PKI keys and certificate state
• local identities and trust stores
• package or container versions
• external object storage
• scheduler and queue state
• required operating-system metadata
• dependencies that must be rebuilt rather than restored

Then choose the capture method.

PostgreSQL may use pgBackRest or pg_basebackup with WAL archiving. MariaDB can use MariaBackup or logical dumps depending on scale and objective. Kubernetes applications may use Velero for cluster resources and persistent-volume workflows, but application hooks and storage behavior still determine consistency. Virtual machines may use Proxmox Backup Server, with application-native backups inside the guest for critical databases.

Layering is often appropriate. A VM backup recovers the operating environment. An application-native backup provides a more precise database recovery path.

Keep recovery keys outside the failure domain

Encrypted backups are only recoverable when the decryption material and procedure survive the same incident.

A secrets system cannot be its own only recovery key store. A backup repository should not be the only place that stores its own encryption password. A domain backup should not require the failed domain for operator authentication.

I use a recovery inventory:

recovery_material:
  backup_data:
    location: off-host-repository
    encrypted: true
  decryption_identity:
    location: separate-controlled-vault
    tested: true
  operator_access:
    primary: named-recovery-account
    break_glass: offline-controlled-record
  platform_configuration:
    location: version-controlled-repository
  internal_ca_trust:
    location: recovery-bundle
  runbook:
    location: accessible-without-primary-service

Separation creates operational cost. Recovery material needs ownership, access review, rotation, and testing. That cost is smaller than discovering during an outage that the password is stored in the unavailable system.

Never print recovery keys, unseal shares, private keys, or restored secret values while proving the process. Record successful use and object identity without copying the secret into evidence.

Restore into isolation first

A full-system restore can be dangerous because it contains identity and automation, not just data.

The default recovery environment should prevent production communication:

restore_isolation:
  virtual_nic:
    link_state: down
    production_bridge: prohibited
  startup:
    onboot: false
    ha_managed: false
  storage:
    target: explicitly-approved-recovery-storage
  network_when_required:
    segment: isolated-recovery
    default_egress: deny
    production_control_planes: deny
  identity:
    hostname_change_before_connect: true
    agent_reenrollment_required: true
    cloned_certificates_review_required: true

For a virtual machine, read back every restored disk and NIC before first boot. Confirm that the restore used the intended storage. Confirm that no interface can reach production. Confirm that automatic startup is disabled.

After boot, inspect cloned identity before opening network access:

• machine ID
• hostname and IP configuration
• SSH host keys
• endpoint-security registration
• monitoring and backup agents
• TLS certificates and private keys
• directory-service machine account
• cluster node ID
• message-consumer identity
• scheduled jobs and webhooks

A restore can be functionally correct and still unsafe to connect.

Test negative controls

Recovery testing usually focuses on what should work. I also test what must not work.

The isolated system should not:

• reach production databases or control planes
• register as the original endpoint
• claim a production VIP
• send queued notifications or SOAR actions
• run scheduled maintenance against live targets
• write to the original backup repository
• obtain production secrets automatically
• join the production cluster
• answer the production DNS name

Negative tests turn isolation from a diagram into evidence.

A practical test matrix might be:

Only open a required dependency path after identifying the exact source, destination, service, purpose, and duration. Close it after the test.

Verify the application, not only the process

A running service is not necessarily a recovered service.

Acceptance should test the application's real responsibilities. For a log platform, ingest a canary, search it, and verify retention and index health. For an identity provider, test authentication, authorization, disabled-user denial, MFA, token claims, and signing keys. For a secrets platform, test seal state, policies, audit, a non-sensitive read and write, lease behavior, and PKI operations where applicable. For a backup portal, verify inventory, repository access, job definitions, and an independent restore.

A generic acceptance record can require:

acceptance:
  infrastructure:
    boot: pass
    expected_disks: pass
    isolation: pass
  data:
    integrity_check: pass
    selected_objects_present: pass
    permissions_and_metadata: pass
  application:
    health: pass
    read_path: pass
    write_path: pass
    authentication: pass
    authorization_denial: pass
  security:
    production_identity_collision: pass
    audit_delivery: pass
    secret_exposure_review: pass
  operations:
    measured_recovery_time: 6420
    manual_steps_recorded: true
    unresolved_gaps: []

Use known test objects that do not expose sensitive values. Record counts, hashes, IDs, and query results where they prove integrity without publishing protected content.

Verification scope must be honest

Deep repository verification can consume significant I/O and time. That does not justify presenting a metadata-only check as complete data verification.

Restic supports check and options for reading repository data. Borg distinguishes repository consistency checks from archive and data verification. Proxmox Backup Server verification reads and verifies chunks associated with snapshots. Configure a schedule that balances full coverage, resource impact, and detection time.

Record:

• which repository and namespace were checked
• whether metadata only or data was read
• sample or full scope
• oldest snapshot covered
• completion time
• errors and repaired state
• next required verification date

A rotating subset can be defensible when the cycle guarantees complete coverage within the accepted period. Random sampling without a coverage record can leave the same damaged data unread indefinitely.

Retention, pruning, and garbage collection are different

Retention policy selects which recovery points should remain.

Pruning removes snapshot references according to that policy.

Garbage collection reclaims unreferenced storage when the backup architecture supports it.

These operations should be scheduled and monitored separately. A prune job that succeeds does not prove storage was reclaimed. Garbage collection that succeeds does not prove retention matches business requirements. Aggressive pruning can eliminate the last known recovery point before a new backup has passed restore acceptance.

A safer policy keeps at least one accepted recovery point outside the normal pruning race. For example:

retention:
  keep_daily: 7
  keep_weekly: 4
  keep_monthly: 6
  keep_yearly: 1
  protected_recovery_points:
    - latest-isolated-restore-pass
verification:
  schedule: weekly
  maximum_age_days: 30
restore_exercise:
  schedule: quarterly

The numbers are examples, not universal recommendations. Data-change rate, RPO, legal retention, repository capacity, ransomware risk, and recovery duration should drive them.

Open-source recovery options

Restic, BorgBackup, and Kopia are excellent general-purpose tools when their repository and key models fit the environment. Proxmox Backup Server is strong for Proxmox workloads and deduplicated VM or container backup. Velero addresses Kubernetes resources and volumes, but it is not a universal database-consistency mechanism. Database-native tools usually provide the best recovery semantics for databases.

Tool diversity can reduce one failure mode and increase complexity. Two backup products are not two independent copies if they write to the same storage, use the same administrator identity, and depend on the same network path.

Record the gaps found by recovery

A restore exercise is valuable even when it fails, provided the result changes the system.

Failures should become tracked engineering work:

finding_id: REC-2026-004
stage: isolated-restore
condition: restored-guest-referenced-missing-installation-media
impact: automated-restore-blocked-before-boot
cause: stale-removable-media-reference
corrective_action: detach-stale-media-and-add-pre-backup-validation
owner: platform-operations
verification: pending

Useful pre-backup checks include missing disks or media, locked guests, unhealthy filesystems, failed database backup hooks, unavailable repositories, stale credentials, insufficient capacity, and accidental inclusion of the backup server itself.

The regression test should run before the next scheduled backup, not wait for the next quarterly exercise.

Recovery evidence has a shelf life

A restore that passed six months ago may not prove today's system.

Application versions change. Backup formats change. TLS certificates expire. Dependencies move. Recovery keys rotate. Data grows. Operators leave. Infrastructure APIs change. A previously isolated network may gain a route.

Define evidence freshness from change rate and service importance. Repeat the exercise after material changes such as:

• database or application major-version upgrade
• backup-tool upgrade
• encryption or key-management change
• storage migration
• identity-provider change
• network or DNS redesign
• cluster membership change
• major data growth
• runbook or ownership change

Do not call a backup recoverable forever because one restore once worked.

The claim I am willing to make

After a backup completes, I can say the job reported success.

After verification, I can say the repository passed a defined integrity check.

After an isolated restore, I can say the selected recovery point reconstructed data in a controlled environment.

Only after application acceptance, negative isolation tests, identity review, timing measurement, and cleanup can I say the service demonstrated recovery.

That wording may sound strict. It prevents a green job icon from carrying more meaning than the evidence supports.

Read more in the series by putting deployments, retirements, recovery paths, and control gaps back into the architecture risk model: Threat Modeling the System That Actually Exists.