Software and Versions Used
This explanation applies to:
- Concepts: Cross-distribution automation principles
- Built-in tools: sha256sum, df, ln, mv, standard shell utilities
- Context: Self-hosting infrastructures, digital sovereignty
Manual backups fail because humans are unreliable. A forgotten backup before a critical system update can cost hours of recovery time. Weekly backup routines are skipped during vacations or illness. Automated backup systems eliminate these sources of error through consistent, scheduled execution without human intervention.
The 3-2-1 rule from Article 1 requires three backup copies on two types of storage media with one off-site copy. These requirements multiply automation complexity: Three separate backup jobs with different destinations, coordinated rotation for storage management, and verification of every copy for integrity. Robust automation makes this complexity manageable.
Time-Based Execution for Consistent Backups
Why Time-Based Automation Is Critical
Backup systems work only through absolute reliability. A forgotten weekly backup means there is no recovery option for data that is 7–14 days old. Time-based schedulers execute backup scripts at defined times without relying on human memory.
All modern operating systems provide time-based job schedulers as a fundamental feature. These schedulers support daily, hourly, weekly, and monthly execution times. The exact configuration differs between systems, but the core concept remains identical: script execution at predefined times.
Coordinating Execution Times
Parallel backup jobs compete for database locks, disk throughput, and network bandwidth. Sequential backup execution avoids these resource conflicts through staggered scheduling. The local backup runs at 03:00, the NAS transfer at 04:00, and the external upload not until 05:00—each job receives a dedicated time window without overlap.
Execution coordination:
03:00 Create database dump on local SSD
03:30 Create filesystem backup on local SSD
04:00 Copy local backups to the NAS
05:00 Synchronize NAS backups to the external server
06:00 Upload external backups to cloud storage
This schedule guarantees that the NAS transfer begins only after the local backups have completed. Cloud uploads do not start until the NAS copy has been verified. Failures in earlier backup stages do not propagate to later stages.
Missed Executions Due to System Downtime
Laptops and desktop systems do not run continuously—shutting them down overnight skips the scheduled 03:00 backups. Servers that are shut down for maintenance also miss scheduled backups. Modern schedulers provide mechanisms to catch up on missed jobs when the system starts again.
The catch-up mechanism checks during system startup whether scheduled jobs were missed while the system was offline. Those jobs are executed automatically once the system is running again. A daily 03:00 backup missed because the system shut down at 22:00 runs after the next boot—typically 15–30 minutes after startup to allow the boot process to stabilize.
Backup Rotation for Retention Policies
Generation-Based Rotation
The daily-weekly-monthly scheme from Article 1 requires automatic rotation: Daily backups are deleted after 7 days unless the Sunday backup is promoted to the weekly set. Weekly backups are deleted after 4 weeks unless the backup created at the beginning of the month is promoted to the monthly set. Monthly backups are retained for 12 months.
Rotation logic:
Create daily backup
If Sunday: Move copy to weekly/
If first day of the month: Move copy to monthly/
Delete files older than 7 days from daily/
Delete files older than 28 days from weekly/
Delete files older than 365 days from monthly/
This mechanism automatically maintains three generations with different retention periods. A failed daily backup does not prevent promotion to the weekly set—the Sunday job copies the most recent successful daily backup.
Hard Link-Based Rotation for Storage Efficiency
Hard links allow multiple directory entries to reference the same file data. The ln command creates hard links on all Unix-like systems. Backup systems use hard links to save storage space: Unchanged files are stored as hard links to the previous generation instead of being copied again.
A 100 GB database with 5% daily changes requires only 5 GB of additional storage per day instead of another 100 GB full backup. Seven daily backups consume 130 GB (100 GB + 6×5 GB) instead of 700 GB. Every backup directory appears to be a complete copy while sharing unchanged blocks with other generations.
Hard link rotation concept:
Write new backup to backup.new/
Create unchanged files as hard links to backup.0/ (ln)
After completion: Delete backup.6/
Rename backup.5/ to backup.6/ (mv)
Rename backup.4/ to backup.5/
Rename backup.3/ to backup.4/
Rename backup.2/ to backup.3/
Rename backup.1/ to backup.2/
Rename backup.0/ to backup.1/
Rename backup.new/ to backup.0/
This rotation maintains seven independent backup directories that can each be deleted individually without affecting other generations. Deleting backup.6/ frees only the storage occupied exclusively by that generation.
Backup Verification Detects Corruption
Checksum-Based Integrity Verification
Cryptographic checksums generate unique fingerprints for files. sha256sum calculates 256-bit hash values from file contents—identical checksums guarantee bit-for-bit identical files, while different checksums indicate corruption or tampering. The sha256sum command is available on all modern Unix systems as a standard built-in utility.
Verification workflow:
Create backup: backup-2025-08-10.sql.gz
Calculate checksum: sha256sum backup-2025-08-10.sql.gz > backup-2025-08-10.sql.gz.sha256
Before restore: sha256sum --check backup-2025-08-10.sql.gz.sha256
On mismatch: Backup is corrupt, try the next available backup
Checksum files are stored alongside backups for later verification. Automatic verification runs after every backup transfer—network corruption or incomplete transfers are detected immediately. A corrupt backup is identified before it overwrites older working backups.
Test Restores Validate Recovery Capability
Checksums verify file integrity but not recoverability. A perfectly intact backup may still be impossible to restore because of missing dependencies, incorrect permissions, or incompatible database versions. Regular test restores validate the complete recovery workflow from retrieving the backup to a functioning database.
Test restore concept:
Prepare separate test system (different hardware or VM)
Retrieve backup from production
Verify checksum
Perform restore
Test basic functionality (database queries, application tests)
On success: Mark backup as verified
On failure: Identify the problem, correct the backup process
Minimum test frequency: One complete test restore per quarter. These tests identify gaps in backup coverage before real disasters occur. The documented duration of the test restore provides realistic expectations for production recovery times.
Partial Restore Tests for Specific Scenarios
Complete database restores test recovery after a total system failure. Many real-world problems, however, require partial restores: an individual deleted table, accidentally updated records, or a single corrupted file. Partial restore tests validate that the backup format supports these selective recovery operations.
SQL dumps allow simple table extraction through grep and text processing. Binary backup formats require specialized tools for partial restores. The choice of backup format affects partial restore flexibility—text-based formats provide maximum flexibility, while binary formats provide speed and compression.
Error Handling for Robust Backup Systems
Exit Codes Signal Job Status
When Unix programs terminate, they return numeric exit codes. A value of 0 signals success, while all other values indicate an error. Backup scripts must follow this convention strictly for proper error detection. A script that returns exit code 0 despite a backup failure prevents automatic problem detection.
The shell option set -e terminates scripts immediately on the first error. This option guarantees that failed commands are not ignored. Combined with set -u (undefined variables are errors) and set -o pipefail (propagate pipeline failures), it provides robust error handling.
Error handling pattern:
#!/bin/sh
set -e Exit script immediately on error
set -u Treat undefined variables as errors
set -o pipefail Do not suppress pipeline failures
Run database dump
Calculate checksum
Copy to destination
Verify checksum at destination
If any step fails: Exit with non-zero exit code
If all steps succeed: Exit with exit code 0
Scheduler systems use exit codes for automatic error handling. Non-zero exit codes trigger retry mechanisms, logging, or notifications depending on the scheduler configuration.
Retry Mechanisms with Exponential Backoff
Temporary network problems or short-lived database locks do not justify immediate escalation. Automatic retries with increasing wait times resolve such transient failures without manual intervention. The first retry occurs after 1 minute, the second after 2 minutes, and the third after 4 minutes. After three unsuccessful attempts, the backup is considered to have failed.
Retry logic concept:
retry_count=0
max_retries=3
retry_delay=60
While retry_count is less than max_retries:
Attempt backup operation
On success: Exit script with exit code 0
On failure:
Wait retry_delay seconds (sleep)
Double retry_delay for the next attempt
Increase retry_count by 1
After max_retries without success: Exit script with non-zero exit code
Exponential backoff prevents retry storms from placing additional load on already overloaded systems. An overloaded database server can recover between retry attempts instead of being further degraded by continuous backup requests. The waiting periods allow temporary problems to resolve themselves.
Atomic Operations Prevent Partial Updates
Failed backups must never overwrite working previous backups. Atomic moves guarantee that destination directories contain only valid backups. The mv command is atomic at the filesystem level—either it succeeds completely or nothing changes.
Atomic backup pattern:
Write backup to temporary file: backup.sql.gz.tmp
Verify checksum of the temporary file
On success:
Rename old backup: mv backup.sql.gz backup.sql.gz.old
Activate new backup: mv backup.sql.gz.tmp backup.sql.gz
Delete old backup according to retention policy
On failure:
Delete temporary file: rm backup.sql.gz.tmp
Existing backup.sql.gz remains completely untouched
This mechanism guarantees that failed backups never destroy working backups. The system automatically falls back to the last valid backup without manual intervention. Between the mv operations, there are brief periods during which both files exist—there is no risk of data loss.
Storage Space Management Before Starting a Backup
Backup systems fail when destination partitions run out of space. Pre-flight checks performed before a backup starts compare the available storage space with the expected backup size. The df command reports available space on all Unix systems. The du command measures directory sizes for capacity estimation.
Storage space check concept:
df /backup Determine available space
du -s /var/lib/database Measure source database size
If available space is less than (database size * 1.5):
Delete oldest backup from daily/
Check available space again
If space is still insufficient:
Abort backup attempt
Return non-zero exit code
The factor of 1.5 accounts for compression and temporary files created during backup generation. Backup rotation must free storage space before new backups are created. FIFO rotation (First In, First Out) automatically deletes the oldest generations when storage limits are reached.
Best Practices for Production Backup Automation
Separate Backup Users with Minimal Privileges
Backup scripts run under a dedicated user account with read-only access to the data being backed up. Compromised backup scripts cannot modify or delete data. The database backup user receives only read permissions for the required databases—no write, update, or delete privileges.
Permission principle:
Backup user can:
Create database dumps (read access)
Read files in directories being backed up
Write to the dedicated /backup directory
Backup user CANNOT:
Modify database tables
Change system configurations
Write to other users' directories
Use elevated privileges (sudo, root)
This isolation limits the damage caused by a compromised backup system. Ransomware that takes over backup processes cannot encrypt production databases. The separation between read access (creating backups) and write access (production data) forms a critical security barrier.
Documentation and Runbooks for Emergency Recovery
Backup automation works reliably until the day disaster strikes. Under stress, critical recovery steps are easily forgotten or important details overlooked. Documented runbooks with step-by-step procedures enable fast and correct action during emergencies.
Runbook contents:
Backup storage locations and access information
Step-by-step recovery procedure
Expected recovery time for each backup location
Verification checks after recovery
Contact information for escalation
Post-recovery checklist for completeness verification
Regular runbook reviews keep outdated information current. Quarterly disaster recovery exercises test runbook completeness and identify missing details. These exercises simulate real emergency conditions—server failures, data loss, and time pressure.
Monitoring and Logging for Error Detection
Automatic backups run without human supervision—failed backups remain unnoticed without active monitoring. Exit codes indicate job status but require a monitoring system that evaluates these codes and raises alerts when problems occur.
Monitoring principles:
On success: Update timestamp of the last successful backup
On failure: Return non-zero exit code to the scheduler
Logging: Log all backup operations with timestamps
Alerting: Notify if the last backup is older than X hours
The monitoring system regularly checks backup timestamps and raises an alert if the last successful backup is too old. This mechanism detects both failed backups and completely failed backup jobs that are no longer running at all. Daily backups should be no more than 36 hours old to ensure timely problem detection.
