curriculum-project-hub/.scratch/saas-production-readiness/assets/backup-migration-primary-sources.md

Backup, migration, and disaster-recovery primary-source research

This note is the primary-source evidence pack for the Wayfinder ticket “Audit backup, migration, and disaster recovery safety.” External claims use only PostgreSQL, Prisma, systemd, and rsync first-party documentation. Project recommendations are explicitly labelled as derived implications rather than upstream guarantees.

PostgreSQL links below use docs/current, which was PostgreSQL 18 when this research was performed on 2026-07-10. A production implementation must pin the deployed PostgreSQL major and verify the same pages for that major. The Hub currently depends on Prisma ^6.19.3, so Prisma 6-specific command examples remain relevant even where the live documentation also describes Prisma 7.

Executive conclusions

1. pg_dump can take a consistent single-database snapshot while ordinary reads and writes continue, but it does not include the Hub's workspace tree, cluster-global roles/tablespaces, or a WAL chain. PostgreSQL now explicitly says it is generally not the right regular production-backup mechanism outside simple cases. It is a useful logical/export layer, not proof of whole-service disaster recovery. (pg_dump description, continuous archiving overview)
2. Database PITR requires a physical base backup plus a continuous, unbroken WAL sequence from the backup start through the recovery target. A successful base-backup command alone does not prove that chain exists. (continuous archiving overview, making a base backup)
3. pg_verifybackup is necessary but not sufficient: PostgreSQL explicitly requires test restores followed by checks that the restored database works and contains the expected data. (pg_verifybackup)
4. Prisma migrate deploy applies pending migrations, but it does not detect schema drift and, on PostgreSQL, Prisma does not wrap a migration in a transaction by default. A failed migration can therefore leave partial schema changes requiring explicit operator recovery. (Prisma production workflow, Prisma transactional behaviour, failed-migration recovery)
5. systemd supplies process lifecycle orchestration, not an application-level consistency barrier. It sends the configured termination signal, waits up to TimeoutStopSec=, then can forcibly kill remaining processes; a successful systemctl stop does not prove that an ExecStop= command or application drain succeeded. (TimeoutStopSec=, systemctl stop)
6. The Hub persists one logical service state across two stores: PostgreSQL records Project.workspaceDir, runs, messages, locks, and file-change audit rows, while workspaces contain project files. A workspace transcript helper exists but is not wired into the production runner. PostgreSQL's snapshot/ PITR guarantees cover only PostgreSQL. A whole-service recovery point therefore requires an application-defined database/workspace checkpoint or a journaled/versioned reconciliation protocol; neither PostgreSQL nor systemd creates one automatically. (Hub schema, workspace transcript helper, pg_dump scope)

Current repository facts that make the boundaries material

• The systemd unit runs prisma migrate deploy as ExecStartPre= before every service start and uses Restart=on-failure. (cph-hub.service)
• The deploy script updates the live Hub directory using rsync -az --delete, builds in that same directory, restarts the unit, and accepts a constant liveness endpoint as deployment success. (deploy_platform.sh)
• Project.workspaceDir is database state, but the referenced directory holds independently mutable tenant content. A workspace JSONL transcript helper exists but is not wired into the production runner; file contents still live in the workspace while the database separately carries run/message/file-change facts. (Hub schema, workspace transcript helper)

These are project facts, not upstream guarantees. The sections below establish what the upstream tools do and, critically, what they do not do.

PostgreSQL logical dump and restore

Snapshot and concurrent-write semantics

• pg_dump makes a consistent export even while the database is being used; it does not block ordinary readers or writers. It dumps only one database. Cluster-global objects such as roles and tablespaces require pg_dumpall or pg_dumpall --globals-only. (pg_dump description, whole-cluster logical dumps)
• pg_dumpall invokes pg_dump once per database. Each database is internally consistent, but the snapshots are not synchronized across databases. (whole-cluster logical dumps)
• A parallel dump uses directory format, opens njobs + 1 connections, and uses synchronized snapshots so every worker sees the same data set. The connection count and additional server load are part of its capacity cost. (pg_dump --jobs)
• Concurrent DDL can still interfere. The parallel-dump leader holds ACCESS SHARE locks; if a later exclusive-lock request queues ahead of a worker, that worker uses NOWAIT and aborts the dump rather than deadlocking. --lock-wait-timeout can bound the initial lock wait. (pg_dump --jobs and lock behaviour)
• The ordinary snapshot is suitable for disaster-recovery dump semantics, but it need not correspond to a serial execution of all transactions that later commit. --serializable-deferrable waits for an anomaly-free snapshot; PostgreSQL says that option is not beneficial for a dump intended only for disaster recovery, but it can matter for a continuously updated reporting copy. (--serializable-deferrable)
• The default waits for dump files to be written safely to disk. Production backups must not use --no-sync, because an immediate OS crash can otherwise leave a corrupt dump. (pg_dump --no-sync)

Derived Hub implications:

• An online database dump may be internally valid while referring to a workspaceDir whose files are from another instant. “pg_dump succeeded” is therefore not a whole-Hub consistency claim.
• Backup scheduling must either serialize against schema migration/DDL or treat any lock wait, abort, non-zero exit, and stderr warning as a failed backup. A partially written archive must never be published as restorable.
• If the deployed cluster contains more than the Hub database, a single Hub dump is not a cluster backup. Even pg_dumpall does not create a single cross-database snapshot.

Restore error and atomicity semantics

• pg_restore defaults to continuing after SQL errors and reports an error count at the end. --exit-on-error is required for fail-fast behaviour. --single-transaction wraps all emitted commands in BEGIN/COMMIT, makes the restore all-or-nothing, and implies --exit-on-error. (pg_restore error and transaction options)
• Parallel restore uses separate connections for data loading, index creation, and constraint creation. It supports only custom/directory archives and cannot be combined with --single-transaction; it is not a whole-restore atomic operation. (pg_restore --jobs)
• A single giant restore transaction can be impractical because it locks every restored object and may exhaust the lock table. PostgreSQL 18 offers --transaction-size=N as a bounded intermediate option; it implies --exit-on-error but is not all-or-nothing. (pg_restore --transaction-size)
• Plain SQL restored by psql also continues after errors unless ON_ERROR_STOP is enabled; a single transaction is separately required for all-or-nothing behaviour. (restoring an SQL dump)
• PostgreSQL recommends restoring into a truly empty database based on template0; otherwise local additions in template1 can cause duplicate definition errors. Required roles must exist before owner/ACL restoration. (restoring an SQL dump)
• A dump can execute arbitrary code selected by superusers of the source database. Partial dump/restore does not remove that trust boundary. (pg_dump security warning)
• A pg_dump binary can dump older servers but refuses a server newer than its own major version. Output is intended to load into newer PostgreSQL versions, but loading into an older major is not guaranteed. (pg_dump cross-version notes)

Derived Hub implications:

• Restore should target a new isolated database/cluster. If a non-atomic or parallel restore reports any error, discard that target and start again; never promote a partially restored database.
• The runbook must record the restore atomicity strategy and exact PostgreSQL client/server versions. A green process exit without explicit fail-fast options is insufficient evidence.
• A restored database must remain disconnected from real Feishu/provider side-effect endpoints until application-level validation completes.

Physical backup, WAL, and PITR

Base backups

• pg_basebackup backs up a running PostgreSQL cluster without blocking other clients. A full backup is an exact copy of the entire cluster and can seed PITR or a standby; it cannot select one database or object. (pg_basebackup description)
• It uses the replication protocol. The account needs REPLICATION or superuser privilege, pg_hba.conf must permit it, and max_wal_senders needs capacity for the backup and, when streaming WAL, another connection. (pg_basebackup requirements)
• Default -X stream obtains WAL over a second connection while copying. With -X fetch, WAL is collected only at the end; if required WAL was recycled, the backup fails and is unusable. (pg_basebackup --wal-method)
• The default waits for backup files to be durable. --no-sync is explicitly unsuitable for production because an OS crash can corrupt the backup. (pg_basebackup --no-sync)
• A manifest is generated by default, with per-file checksums. A SHA manifest can detect malicious modification only if the manifest itself is held in a separate trusted location or otherwise authenticated. (pg_basebackup manifest options)

Derived minimum backup record:

• source cluster/system identifier and PostgreSQL version;
• backup tool version, start/end time, start/end LSN and WAL range;
• manifest digest and its independently protected location;
• destination object/version, byte count, exit status, and verification state;
• an explicit VALID, INVALID, or QUARANTINED state so a failed leftover directory cannot enter the retention set merely because files exist.

WAL archival and recovery

• PITR combines a physical backup with WAL replay. It can stop at a chosen time and reach a consistent database state. Logical dumps do not contain enough information for WAL replay. (continuous archiving overview)
• Successful recovery requires a continuous sequence of archived WAL extending back at least to the base-backup start. PostgreSQL tells operators to set up and test WAL archiving before taking the first base backup. (continuous archiving overview)
• The archive command must return zero if and only if durable archival succeeded. Non-zero causes retry. Archive implementations should refuse to overwrite an existing file; a repeated name may report success only when the existing durable content is identical, and differing content must fail. (WAL archiving)
• Repeated archive failure accumulates WAL in pg_wal; filling that filesystem causes PostgreSQL to panic and remain offline until space is freed. PostgreSQL explicitly advises monitoring archival progress. (WAL archiving failure behaviour)
• WAL is archived after a segment is completed, so low traffic can make the last committed change wait a long time for archival. archive_timeout can bound that age by forcing segment switches, at the cost of archive growth. (WAL archiving latency)
• WAL archival does not restore manually edited PostgreSQL configuration files (postgresql.conf, pg_hba.conf, pg_ident.conf); those need a separate file backup. (WAL archiving caveat)
• PostgreSQL's recovery procedure keeps clients out, restores the base backup, configures restore_command and recovery.signal, replays WAL, and requires inspection of database contents before restoring ordinary access. (PITR recovery procedure)
• Recovery can target a timestamp, LSN, transaction ID, named restore point, or the earliest consistent point. Only one target may be set. Timeline history is required to navigate recovery branches, and a configured target not reached before WAL exhaustion is a fatal recovery failure. (recovery target settings, timelines)

Derived Hub implications:

• The database RPO is the age of the newest durably archived, continuous WAL position, not the age of the newest base backup and not merely the name in last_archived_wal.
• Archive keys need a collision-proof cluster/system-identifier namespace and create-if-absent semantics; unrelated clusters must never overwrite a shared WAL name.
• Monitoring must cover archive failures, time since durable success, archive lag, WAL continuity, pg_wal growth, free space, and base-backup age.
• PITR restores PostgreSQL only. It does not rewind workspace files. A database recovery target has no whole-Hub meaning until paired with a workspace recovery point and reconciliation rule.

Verification is a ladder, not a checksum

• pg_verifybackup validates the backup manifest and system identifier, detects missing/extra/changed files, checks file checksums, and parses the required WAL with pg_waldump. WAL verification is currently supported only for plain-format base backups. (pg_verifybackup checks)
• PostgreSQL explicitly says these checks cannot cover everything a running server will encounter and still requires test restores plus checks that the database works and contains correct data. (pg_verifybackup limitation)
• pg_amcheck can check supported heap/index structures, but currently only ordinary/TOAST tables, materialized views, sequences, and B-tree indexes are supported; other relation types are silently skipped. Stronger B-tree checks can block concurrent modifications. (pg_amcheck)

Derived recovery-evidence ladder:

1. Artifact verification: backup command succeeded; manifest and all required WAL validate; any failure marks the artifact unusable.
2. Isolated technical restore: restore to a clean non-production target, reach the intended LSN/time/timeline, and prove PostgreSQL starts without exposing it to users.
3. Database validation: verify migration history/schema identity, tenant counts and relationships, critical constraints/sequences, and appropriate corruption checks.
4. Workspace validation: restore the paired snapshot/version, validate file hashes/ownership/containment, and reconcile database file-change/run state against actual files.
5. Application validation: run read-only tenant-isolation and core workflow smoke tests with external side effects disabled.
6. Recovery report: persist the exact backup, WAL range, workspace point, release/schema versions, checks, observed RPO/RTO, operator, and failures. Only a fully passing report may authorize promotion.

Prisma Migrate production semantics

What migrate deploy proves and does not prove

• In production, migrate deploy compares applied migration checksums with the migration directory, warns when an applied migration was modified, and applies pending migrations. (Prisma 6 production workflow)
• It does not warn when an already-applied migration is missing, detect production schema drift, reset the database, generate Prisma Client, or use a shadow database. (Prisma 6 production workflow)
• The migration directory is the deployable history and must be committed in full, including migration_lock.toml. Prisma warns against editing or deleting migrations already applied in production; migrate deploy executes migration SQL rather than deriving a new migration from the Prisma schema. (Prisma migration histories)
• Prisma uses a database advisory lock for production migration commands so multiple migration commands do not run simultaneously. The documented lock timeout is ten seconds and not configurable; after timeout, the caller must retry. The lock can be disabled by environment variable. (Prisma 6 advisory locking)

Derived Hub implication: migrate deploy exit zero proves that this invocation applied or found no pending migration. It does not prove the live schema equals the expected migration end state, that the new and previous application builds are both compatible with it, or that the migration is operationally safe under production data volume and traffic.

The advisory lock is a migration-command concurrency guard, not a substitute for the repository's deployment lock or application/schema compatibility. As a derived safety rule, production preflight should reject PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK unless an explicit reviewed mechanism provides the same serialization guarantee.

Transactions and failed migrations

• Prisma's first-party migration guidance states that PostgreSQL migrations are not wrapped in a transaction by default. Operators may opt in by adding BEGIN;/COMMIT; to a migration, but large transactional migrations can hold locks longer and consume more database resources. (Prisma transactional behaviour)
• A failed migration records its error in _prisma_migrations.logs; further migrations cannot deploy until the failed state is explicitly recovered. Recovery is either (a) revert/repair partial steps, mark the migration rolled back, and redeploy, or (b) manually complete the exact steps and mark it applied. (Prisma failed-migration recovery)
• Prisma error codes P3009 and P3018 explicitly report that failed migrations block new migration application until recovery. (Prisma error reference)
• migrate resolve --rolled-back changes migration-history state so Prisma may apply the migration again; it does not itself undo SQL already executed. --applied records success without executing the migration SQL. (failed-migration recovery, migrate resolve)
• Prisma can generate a down SQL script, but its documentation limits the simple rollback workflow to a failed up migration. To reverse a successful migration, the supported history is a new forward migration generated from a reverted Prisma schema. A schema down script also cannot restore lost data, application changes, or manually authored SQL automatically. (generating down migrations)

Derived Hub implications:

• Every migration needs a declared atomicity choice. If it is intentionally non-transactional, the migration must be resumable/idempotent or ship an exact, tested partial-failure procedure. Blindly rerunning a partially applied script is unsafe.
• A database backup checkpoint before a destructive or difficult-to-reverse migration is necessary, but restoring that checkpoint is a whole-database recovery event—not a substitute for a compatible rollout design.
• Service restart cannot be treated as application rollback after a successful schema migration. The previous binary may no longer understand the new schema, and a down migration cannot recreate deleted data.

Compatibility and rollout ordering

• Prisma documents expand-and-contract as the way to avoid downtime for fields in active use: add the new representation, deploy code that writes both while reading the old, migrate and verify data, switch reads, stop old writes, then remove the old representation. (Prisma expand-and-contract workflow)
• Prisma also warns that generated “renames” may be expressed as add/drop and lose data unless the SQL is reviewed and customized before application. (customizing migrations)

Derived deployment contract:

1. Migration SQL is immutable after production application and reviewed for destructive operations, lock duration, data volume, and transaction choice.
2. It is rehearsed against a recent, anonymized production-scale restore and its runtime/locks/disk growth are recorded.
3. Each release declares a schema compatibility window. Expand releases remain compatible with both old and new application versions; contract/destructive releases occur only after old binaries can no longer run.
4. Migration status/drift is checked before changing the live release. A failed migration blocks deployment and enters an operator runbook; it must never be hidden by repeated restart attempts.
5. Rollback means switching to a binary proven compatible with the current schema or deploying a new forward migration. Restoring a backup is the last resort and must also restore/reconcile the workspace recovery point.

systemd stop, restart, and dependency semantics

• When ExecStartPre=, ExecStart=, or ExecStartPost= fails or times out, service startup does not continue to the normal running process. ExecStop= is skipped because startup never succeeded; ExecStopPost= is the hook that still runs after a failed startup. (systemd start/stop command semantics, ExecStopPost=, permanent upstream man-page source)
• Restart=on-failure covers non-zero exits and failures/timeouts in service control processes, including ExecStartPre=. An explicit systemd stop does not trigger automatic restart. Restarts are subject to unit start-rate limiting. (Restart=, permanent upstream source)
• With no ExecStop=, systemd sends KillSignal= (default SIGTERM). TimeoutStopSec= bounds the wait; after it expires, remaining processes can be forcibly killed with SIGKILL/FinalKillSignal=. (ExecStop=, TimeoutStopSec=)
• Default KillMode=control-group applies termination to all remaining processes in the unit cgroup. systemd explicitly discourages process or none because children can escape the service lifecycle and remain running after the service is considered stopped. (KillMode=)
• systemctl stop waits for the job unless --no-block is used, but the command does not fail merely because an ExecStop= command failed: the manager still forcibly terminates the unit. (systemctl stop, permanent upstream source)
• systemctl restart is a stop followed by a start; ExecStop= and ExecStopPost= participate. It does not necessarily flush every unit resource in the same way as separate completed stop then start operations. (restart semantics, systemctl restart)
• After= is only an ordering relationship; it does not pull in another unit. Requirement and ordering dependencies are independent. During shutdown the start order is reversed, so a Hub ordered After=postgresql.service stops before PostgreSQL when both are in the shutdown transaction. (Requires= and ordering, Before=/After=)
• systemctl freeze only uses the cgroup freezer to suspend process execution; its documented contract contains no filesystem flush or stable-image guarantee. It is not a substitute for an application drain or a filesystem snapshot barrier. (permanent systemctl freeze man-page source)

Derived assessment of the current unit:

• A Prisma migration failure in the current ExecStartPre= prevents the Hub from starting, then Restart=on-failure can retry startup until rate-limited. That is fail-closed, but it is not controlled migration recovery and can repeatedly hit a database already left in Prisma's failed state.
• KillSignal=SIGTERM plus TimeoutStopSec=30 is only a deadline. Until the Hub installs a real termination handler that makes readiness false, stops ingress, drains/checkpoints active work, closes Feishu and database clients, and reports success, the unit cannot claim a graceful consistency barrier.
• A backup runbook must issue a controlled drain, verify durable work reached a reconciled terminal/checkpoint state, stop the Hub, verify the unit and its cgroup are inactive, and only then snapshot mutable workspace state. A zero systemctl stop exit by itself is insufficient evidence.
• Recovery should not start the normal unit until the restored database and workspace have been validated. The current unit would run pending migrations immediately on first start, mutating the restored database before validation and possibly destroying the ability to inspect the original recovered state.

Filesystem and rsync boundaries

• PostgreSQL says a plain filesystem copy of a live data directory is unusable: the database must be shut down, because tools such as tar do not create an atomic snapshot and server buffering remains. A correctly implemented frozen snapshot can work while PostgreSQL runs, but all volumes containing data, WAL, and tablespaces must be captured simultaneously; otherwise shut down or use the continuous-archive base-backup protocol. (PostgreSQL filesystem backups)
• For a PostgreSQL rsync backup, the documented safe pattern is a first pass while running, followed by shutdown and a second rsync --checksum; the second pass is what makes the result consistent. (PostgreSQL filesystem backups)
• At the filesystem layer, fsfreeze is documented to let in-progress filesystem operations complete, halt new modifications, and flush dirty data, metadata, and logs before returning. That is still only a filesystem-volume guarantee; it does not replace PostgreSQL's backup protocol and does not make separately frozen volumes simultaneous. (util-linux fsfreeze manual source)
• rsync's --delete defaults to deletion during transfer on modern peers, so the receiver tree changes incrementally. --delay-updates merely holds each updated file and renames files into place rapidly at the end; its own manual says this only attempts to make updates “a little more atomic” and points to a separate parallel-tree/switch approach for stronger atomicity. (rsync --delete, rsync --delay-updates)

Derived Hub implications:

• The current in-place rsync --delete deployment is neither an immutable release switch nor a backup snapshot. A failed/interrupted transfer can leave a mixed live tree; service stop/restart cannot reconstruct the deleted prior artifact.
• The workspace snapshot mechanism remains an explicit design choice. Whichever storage technology is selected must prove atomicity over all workspace volumes or use a quiesced/no-writer interval, then publish an immutable version/checksum manifest. Merely copying a changing directory is not enough.

Derived minimum whole-Hub recovery contract

The primary sources support the following minimum contract; exact RPO, RTO, retention, storage vendor, and numeric schedules remain product/operations decisions and must not be invented by implementation.

Backup path

1. Maintain verified PostgreSQL base backups and a monitored continuous WAL archive, or obtain and test an explicitly equivalent managed-PostgreSQL guarantee. Keep logical dumps as a separate portability/selective-restore layer, not the sole PITR mechanism.
2. Keep PostgreSQL configuration, deployment/release manifests, migration history, and workspace snapshots in the backup inventory; WAL alone does not cover them.
3. Create a whole-Hub checkpoint only after ingress is drained and active workspace writers are stopped/reconciled, unless the eventual storage layer supplies a proven atomic snapshot/journal spanning the necessary stores.
4. Persist one signed/immutable checkpoint manifest joining PostgreSQL backup/WAL position, workspace snapshot/version, application revision, migration digest, tool versions, checksums, timestamps, and validation state.
5. Treat database/WAL/workspace backups as cross-tenant sensitive data. Apply least-privilege access, encryption, audit, deletion/retention, and an independently protected integrity manifest; WAL contains effectively all database changes and PostgreSQL advises protecting it from unauthorized readers. (WAL archive confidentiality)

Migration path

1. Serialize deploys and backup checkpoints; run a drift/status gate before migration, not just migrate deploy.
2. Rehearse each migration on a recent production-scale restore and record runtime, locks, disk/WAL growth, transaction choice, and failure recovery.
3. Require expand/contract compatibility for online changes. A release must declare which old/new binaries can run against each schema phase.
4. Separate migration orchestration from ordinary service restart so a failed migration produces one durable incident state and an operator decision—not an opaque restart loop.
5. Verify a restorable checkpoint before destructive change. Do not describe backup restore as a normal “rollback”; it is whole-service disaster recovery.

Restore and promotion path

1. Restore into an isolated environment with real external side effects disabled.
2. Restore the selected PostgreSQL base backup/WAL target and the checkpoint's exact workspace version; do not run pending Prisma migrations implicitly.
3. Prove artifact integrity, database startup, recovery target/timeline, migration/schema identity, tenant invariants, workspace hashes/containment, and read-only application flows.
4. Record observed recovery point and recovery time. A restore that starts but misses its target, loses a tenant workspace, has partial migration state, or fails any invariant is not promotable.
5. Promote explicitly, then enable ingress and external integrations. Preserve the pre-recovery cluster/WAL and recovery report until the incident is closed.

Decisions and evidence still required

These values are not fixed by the upstream tools and remain OPEN for the SaaS production contract:

• whole-Hub RPO and RTO, separately for ordinary deletion, database corruption, host loss, region/provider loss, and migration failure;
• base-backup, logical-dump, workspace-snapshot, WAL, and restore-drill cadence;
• immutable/off-host storage, encryption/key ownership, retention and verified deletion policy, including org-deletion and legal-retention interactions;
• exact workspace checkpoint/journal technology and database/workspace reconciliation semantics for in-flight or partially written runs;
• whether restore promotion is whole-platform only or supports a safe organization/project export-import path (PostgreSQL physical PITR is cluster-wide);
• migration transaction policy, maximum lock/downtime budget, compatibility window, and who can authorize contract/destructive phases;
• evidence from at least one production-like full restore and one PITR drill, including deliberate missing-WAL, corrupted-backup, failed-migration, interrupted-workspace-write, and restart-during-recovery probes.