curriculum-project-hub/.scratch/saas-production-readiness/assets/backup-migration-primary-sources.md

# Backup, migration, and disaster-recovery primary-source research

This note is the primary-source evidence pack for the Wayfinder ticket
“Audit backup, migration, and disaster recovery safety.” External claims use
only PostgreSQL, Prisma, systemd, and rsync first-party documentation. Project
recommendations are explicitly labelled as derived implications rather than
upstream guarantees.

PostgreSQL links below use `docs/current`, which was PostgreSQL 18 when this
research was performed on 2026-07-10. A production implementation must pin the
deployed PostgreSQL major and verify the same pages for that major. The Hub
currently depends on Prisma `^6.19.3`, so Prisma 6-specific command examples
remain relevant even where the live documentation also describes Prisma 7.

## Executive conclusions

1. `pg_dump` can take a consistent **single-database** snapshot while ordinary
   reads and writes continue, but it does not include the Hub's workspace tree,
   cluster-global roles/tablespaces, or a WAL chain. PostgreSQL now explicitly
   says it is generally not the right regular production-backup mechanism
   outside simple cases. It is a useful logical/export layer, not proof of
   whole-service disaster recovery.
   ([`pg_dump` description](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION),
   [continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING))
2. Database PITR requires a physical base backup plus a continuous, unbroken
   WAL sequence from the backup start through the recovery target. A successful
   base-backup command alone does not prove that chain exists.
   ([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING),
   [making a base backup](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-BASE-BACKUP))
3. `pg_verifybackup` is necessary but not sufficient: PostgreSQL explicitly
   requires test restores followed by checks that the restored database works
   and contains the expected data.
   ([`pg_verifybackup`](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP))
4. Prisma `migrate deploy` applies pending migrations, but it does not detect
   schema drift and, on PostgreSQL, Prisma does not wrap a migration in a
   transaction by default. A failed migration can therefore leave partial
   schema changes requiring explicit operator recovery.
   ([Prisma production workflow](https://www.prisma.io/docs/orm/prisma-migrate/workflows/development-and-production#production-and-testing-environments),
   [Prisma transactional behaviour](https://www.prisma.io/blog/prisma-migrate-dx-primitives#what-if-schema-migrations-were-atomic),
   [failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration))
5. systemd supplies process lifecycle orchestration, not an application-level
   consistency barrier. It sends the configured termination signal, waits up
   to `TimeoutStopSec=`, then can forcibly kill remaining processes; a
   successful `systemctl stop` does not prove that an `ExecStop=` command or
   application drain succeeded.
   ([`TimeoutStopSec=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutStopSec=),
   [`systemctl stop`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#stop%20PATTERN%E2%80%A6))
6. The Hub persists one logical service state across two stores: PostgreSQL
   records `Project.workspaceDir`, runs, messages, locks, and file-change audit
   rows, while workspaces contain project files. A workspace transcript helper
   exists but is not wired into the production runner. PostgreSQL's snapshot/
   PITR guarantees cover only PostgreSQL. A whole-service recovery point
   therefore requires an application-defined database/workspace checkpoint or
   a journaled/versioned reconciliation protocol; neither PostgreSQL nor
   systemd creates one automatically.
   ([Hub schema](../../../hub/prisma/schema.prisma),
   [workspace transcript helper](../../../hub/src/agent/transcript.ts),
   [`pg_dump` scope](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION))

## Current repository facts that make the boundaries material

- The systemd unit runs `prisma migrate deploy` as `ExecStartPre=` before every
  service start and uses `Restart=on-failure`.
  ([`cph-hub.service`](../../../hub/deploy/cph-hub.service))
- The deploy script updates the live Hub directory using
  `rsync -az --delete`, builds in that same directory, restarts the unit, and
  accepts a constant liveness endpoint as deployment success.
  ([`deploy_platform.sh`](../../../hub/deploy/deploy_platform.sh))
- `Project.workspaceDir` is database state, but the referenced directory holds
  independently mutable tenant content. A workspace JSONL transcript helper
  exists but is not wired into the production runner; file contents still live
  in the workspace while the database separately carries run/message/file-change
  facts.
  ([Hub schema](../../../hub/prisma/schema.prisma),
  [workspace transcript helper](../../../hub/src/agent/transcript.ts))

These are project facts, not upstream guarantees. The sections below establish
what the upstream tools do and, critically, what they do not do.

## PostgreSQL logical dump and restore

### Snapshot and concurrent-write semantics

- `pg_dump` makes a consistent export even while the database is being used;
  it does not block ordinary readers or writers. It dumps only one database.
  Cluster-global objects such as roles and tablespaces require `pg_dumpall` or
  `pg_dumpall --globals-only`.
  ([`pg_dump` description](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION),
  [whole-cluster logical dumps](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-ALL))
- `pg_dumpall` invokes `pg_dump` once per database. Each database is internally
  consistent, but the snapshots are not synchronized across databases.
  ([whole-cluster logical dumps](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-ALL))
- A parallel dump uses directory format, opens `njobs + 1` connections, and
  uses synchronized snapshots so every worker sees the same data set. The
  connection count and additional server load are part of its capacity cost.
  ([`pg_dump --jobs`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
- Concurrent DDL can still interfere. The parallel-dump leader holds
  `ACCESS SHARE` locks; if a later exclusive-lock request queues ahead of a
  worker, that worker uses `NOWAIT` and aborts the dump rather than deadlocking.
  `--lock-wait-timeout` can bound the initial lock wait.
  ([`pg_dump --jobs` and lock behaviour](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
- The ordinary snapshot is suitable for disaster-recovery dump semantics, but
  it need not correspond to a serial execution of all transactions that later
  commit. `--serializable-deferrable` waits for an anomaly-free snapshot;
  PostgreSQL says that option is not beneficial for a dump intended only for
  disaster recovery, but it can matter for a continuously updated reporting
  copy.
  ([`--serializable-deferrable`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
- The default waits for dump files to be written safely to disk. Production
  backups must not use `--no-sync`, because an immediate OS crash can otherwise
  leave a corrupt dump.
  ([`pg_dump --no-sync`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))

Derived Hub implications:

- An online database dump may be internally valid while referring to a
  `workspaceDir` whose files are from another instant. “`pg_dump` succeeded” is
  therefore not a whole-Hub consistency claim.
- Backup scheduling must either serialize against schema migration/DDL or
  treat any lock wait, abort, non-zero exit, and stderr warning as a failed
  backup. A partially written archive must never be published as restorable.
- If the deployed cluster contains more than the Hub database, a single Hub
  dump is not a cluster backup. Even `pg_dumpall` does not create a single
  cross-database snapshot.

### Restore error and atomicity semantics

- `pg_restore` defaults to continuing after SQL errors and reports an error
  count at the end. `--exit-on-error` is required for fail-fast behaviour.
  `--single-transaction` wraps all emitted commands in `BEGIN`/`COMMIT`, makes
  the restore all-or-nothing, and implies `--exit-on-error`.
  ([`pg_restore` error and transaction options](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS))
- Parallel restore uses separate connections for data loading, index creation,
  and constraint creation. It supports only custom/directory archives and
  cannot be combined with `--single-transaction`; it is not a whole-restore
  atomic operation.
  ([`pg_restore --jobs`](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS))
- A single giant restore transaction can be impractical because it locks every
  restored object and may exhaust the lock table. PostgreSQL 18 offers
  `--transaction-size=N` as a bounded intermediate option; it implies
  `--exit-on-error` but is not all-or-nothing.
  ([`pg_restore --transaction-size`](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS))
- Plain SQL restored by `psql` also continues after errors unless
  `ON_ERROR_STOP` is enabled; a single transaction is separately required for
  all-or-nothing behaviour.
  ([restoring an SQL dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE))
- PostgreSQL recommends restoring into a truly empty database based on
  `template0`; otherwise local additions in `template1` can cause duplicate
  definition errors. Required roles must exist before owner/ACL restoration.
  ([restoring an SQL dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE))
- A dump can execute arbitrary code selected by superusers of the source
  database. Partial dump/restore does not remove that trust boundary.
  ([`pg_dump` security warning](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION))
- A `pg_dump` binary can dump older servers but refuses a server newer than its
  own major version. Output is intended to load into newer PostgreSQL versions,
  but loading into an older major is not guaranteed.
  ([`pg_dump` cross-version notes](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-NOTES))

Derived Hub implications:

- Restore should target a new isolated database/cluster. If a non-atomic or
  parallel restore reports any error, discard that target and start again;
  never promote a partially restored database.
- The runbook must record the restore atomicity strategy and exact PostgreSQL
  client/server versions. A green process exit without explicit fail-fast
  options is insufficient evidence.
- A restored database must remain disconnected from real Feishu/provider
  side-effect endpoints until application-level validation completes.

## Physical backup, WAL, and PITR

### Base backups

- `pg_basebackup` backs up a running PostgreSQL cluster without blocking other
  clients. A full backup is an exact copy of the entire cluster and can seed
  PITR or a standby; it cannot select one database or object.
  ([`pg_basebackup` description](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
- It uses the replication protocol. The account needs `REPLICATION` or
  superuser privilege, `pg_hba.conf` must permit it, and `max_wal_senders`
  needs capacity for the backup and, when streaming WAL, another connection.
  ([`pg_basebackup` requirements](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
- Default `-X stream` obtains WAL over a second connection while copying. With
  `-X fetch`, WAL is collected only at the end; if required WAL was recycled,
  the backup fails and is unusable.
  ([`pg_basebackup --wal-method`](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
- The default waits for backup files to be durable. `--no-sync` is explicitly
  unsuitable for production because an OS crash can corrupt the backup.
  ([`pg_basebackup --no-sync`](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
- A manifest is generated by default, with per-file checksums. A SHA manifest
  can detect malicious modification only if the manifest itself is held in a
  separate trusted location or otherwise authenticated.
  ([`pg_basebackup` manifest options](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))

Derived minimum backup record:

- source cluster/system identifier and PostgreSQL version;
- backup tool version, start/end time, start/end LSN and WAL range;
- manifest digest and its independently protected location;
- destination object/version, byte count, exit status, and verification state;
- an explicit `VALID`, `INVALID`, or `QUARANTINED` state so a failed leftover
  directory cannot enter the retention set merely because files exist.

### WAL archival and recovery

- PITR combines a physical backup with WAL replay. It can stop at a chosen time
  and reach a consistent database state. Logical dumps do not contain enough
  information for WAL replay.
  ([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING))
- Successful recovery requires a continuous sequence of archived WAL extending
  back at least to the base-backup start. PostgreSQL tells operators to set up
  and test WAL archiving **before** taking the first base backup.
  ([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING))
- The archive command must return zero if and only if durable archival
  succeeded. Non-zero causes retry. Archive implementations should refuse to
  overwrite an existing file; a repeated name may report success only when the
  existing durable content is identical, and differing content must fail.
  ([WAL archiving](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
- Repeated archive failure accumulates WAL in `pg_wal`; filling that filesystem
  causes PostgreSQL to panic and remain offline until space is freed. PostgreSQL
  explicitly advises monitoring archival progress.
  ([WAL archiving failure behaviour](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
- WAL is archived after a segment is completed, so low traffic can make the
  last committed change wait a long time for archival. `archive_timeout` can
  bound that age by forcing segment switches, at the cost of archive growth.
  ([WAL archiving latency](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
- WAL archival does not restore manually edited PostgreSQL configuration files
  (`postgresql.conf`, `pg_hba.conf`, `pg_ident.conf`); those need a separate
  file backup.
  ([WAL archiving caveat](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
- PostgreSQL's recovery procedure keeps clients out, restores the base backup,
  configures `restore_command` and `recovery.signal`, replays WAL, and requires
  inspection of database contents before restoring ordinary access.
  ([PITR recovery procedure](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-PITR-RECOVERY))
- Recovery can target a timestamp, LSN, transaction ID, named restore point, or
  the earliest consistent point. Only one target may be set. Timeline history
  is required to navigate recovery branches, and a configured target not
  reached before WAL exhaustion is a fatal recovery failure.
  ([recovery target settings](https://www.postgresql.org/docs/current/runtime-config-wal.html#RUNTIME-CONFIG-WAL-RECOVERY-TARGET),
  [timelines](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-TIMELINES))

Derived Hub implications:

- The database RPO is the age of the newest **durably archived, continuous**
  WAL position, not the age of the newest base backup and not merely the name
  in `last_archived_wal`.
- Archive keys need a collision-proof cluster/system-identifier namespace and
  create-if-absent semantics; unrelated clusters must never overwrite a shared
  WAL name.
- Monitoring must cover archive failures, time since durable success, archive
  lag, WAL continuity, `pg_wal` growth, free space, and base-backup age.
- PITR restores PostgreSQL only. It does not rewind workspace files. A database
  recovery target has no whole-Hub meaning until paired with a workspace
  recovery point and reconciliation rule.

## Verification is a ladder, not a checksum

- `pg_verifybackup` validates the backup manifest and system identifier,
  detects missing/extra/changed files, checks file checksums, and parses the
  required WAL with `pg_waldump`. WAL verification is currently supported only
  for plain-format base backups.
  ([`pg_verifybackup` checks](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP))
- PostgreSQL explicitly says these checks cannot cover everything a running
  server will encounter and still requires test restores plus checks that the
  database works and contains correct data.
  ([`pg_verifybackup` limitation](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP))
- `pg_amcheck` can check supported heap/index structures, but currently only
  ordinary/TOAST tables, materialized views, sequences, and B-tree indexes are
  supported; other relation types are silently skipped. Stronger B-tree checks
  can block concurrent modifications.
  ([`pg_amcheck`](https://www.postgresql.org/docs/current/app-pgamcheck.html#APP-PGAMCHECK))

Derived recovery-evidence ladder:

1. **Artifact verification:** backup command succeeded; manifest and all
   required WAL validate; any failure marks the artifact unusable.
2. **Isolated technical restore:** restore to a clean non-production target,
   reach the intended LSN/time/timeline, and prove PostgreSQL starts without
   exposing it to users.
3. **Database validation:** verify migration history/schema identity, tenant
   counts and relationships, critical constraints/sequences, and appropriate
   corruption checks.
4. **Workspace validation:** restore the paired snapshot/version, validate file
   hashes/ownership/containment, and reconcile database file-change/run state
   against actual files.
5. **Application validation:** run read-only tenant-isolation and core workflow
   smoke tests with external side effects disabled.
6. **Recovery report:** persist the exact backup, WAL range, workspace point,
   release/schema versions, checks, observed RPO/RTO, operator, and failures.
   Only a fully passing report may authorize promotion.

## Prisma Migrate production semantics

### What `migrate deploy` proves and does not prove

- In production, `migrate deploy` compares applied migration checksums with the
  migration directory, warns when an applied migration was modified, and
  applies pending migrations.
  ([Prisma 6 production workflow](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#production-and-testing-environments))
- It does **not** warn when an already-applied migration is missing, detect
  production schema drift, reset the database, generate Prisma Client, or use a
  shadow database.
  ([Prisma 6 production workflow](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#production-and-testing-environments))
- The migration directory is the deployable history and must be committed in
  full, including `migration_lock.toml`. Prisma warns against editing or
  deleting migrations already applied in production; `migrate deploy` executes
  migration SQL rather than deriving a new migration from the Prisma schema.
  ([Prisma migration histories](https://www.prisma.io/docs/orm/prisma-migrate/understanding-prisma-migrate/migration-histories))
- Prisma uses a database advisory lock for production migration commands so
  multiple migration commands do not run simultaneously. The documented lock
  timeout is ten seconds and not configurable; after timeout, the caller must
  retry. The lock can be disabled by environment variable.
  ([Prisma 6 advisory locking](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#advisory-locking))

Derived Hub implication: `migrate deploy` exit zero proves that this invocation
applied or found no pending migration. It does not prove the live schema equals
the expected migration end state, that the new and previous application builds
are both compatible with it, or that the migration is operationally safe under
production data volume and traffic.

The advisory lock is a migration-command concurrency guard, not a substitute
for the repository's deployment lock or application/schema compatibility. As a
derived safety rule, production preflight should reject
`PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK` unless an explicit reviewed mechanism
provides the same serialization guarantee.

### Transactions and failed migrations

- Prisma's first-party migration guidance states that PostgreSQL migrations
  are not wrapped in a transaction by default. Operators may opt in by adding
  `BEGIN;`/`COMMIT;` to a migration, but large transactional migrations can hold
  locks longer and consume more database resources.
  ([Prisma transactional behaviour](https://www.prisma.io/blog/prisma-migrate-dx-primitives#what-if-schema-migrations-were-atomic))
- A failed migration records its error in `_prisma_migrations.logs`; further
  migrations cannot deploy until the failed state is explicitly recovered.
  Recovery is either (a) revert/repair partial steps, mark the migration rolled
  back, and redeploy, or (b) manually complete the exact steps and mark it
  applied.
  ([Prisma failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration))
- Prisma error codes `P3009` and `P3018` explicitly report that failed
  migrations block new migration application until recovery.
  ([Prisma error reference](https://www.prisma.io/docs/orm/reference/error-reference#p3009))
- `migrate resolve --rolled-back` changes migration-history state so Prisma may
  apply the migration again; it does not itself undo SQL already executed.
  `--applied` records success without executing the migration SQL.
  ([failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration),
  [`migrate resolve`](https://www.prisma.io/docs/cli/migrate/resolve))
- Prisma can generate a down SQL script, but its documentation limits the
  simple rollback workflow to a **failed** up migration. To reverse a successful
  migration, the supported history is a new forward migration generated from a
  reverted Prisma schema. A schema down script also cannot restore lost data,
  application changes, or manually authored SQL automatically.
  ([generating down migrations](https://www.prisma.io/docs/orm/prisma-migrate/workflows/generating-down-migrations#considerations-when-generating-down-migrations))

Derived Hub implications:

- Every migration needs a declared atomicity choice. If it is intentionally
  non-transactional, the migration must be resumable/idempotent or ship an
  exact, tested partial-failure procedure. Blindly rerunning a partially applied
  script is unsafe.
- A database backup checkpoint before a destructive or difficult-to-reverse
  migration is necessary, but restoring that checkpoint is a whole-database
  recovery event—not a substitute for a compatible rollout design.
- Service restart cannot be treated as application rollback after a successful
  schema migration. The previous binary may no longer understand the new
  schema, and a down migration cannot recreate deleted data.

### Compatibility and rollout ordering

- Prisma documents expand-and-contract as the way to avoid downtime for fields
  in active use: add the new representation, deploy code that writes both while
  reading the old, migrate and verify data, switch reads, stop old writes, then
  remove the old representation.
  ([Prisma expand-and-contract workflow](https://www.prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations#example-use-the-expand-and-contract-pattern-to-evolve-the-schema-without-downtime))
- Prisma also warns that generated “renames” may be expressed as add/drop and
  lose data unless the SQL is reviewed and customized before application.
  ([customizing migrations](https://www.prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations#example-rename-a-field))

Derived deployment contract:

1. Migration SQL is immutable after production application and reviewed for
   destructive operations, lock duration, data volume, and transaction choice.
2. It is rehearsed against a recent, anonymized production-scale restore and
   its runtime/locks/disk growth are recorded.
3. Each release declares a schema compatibility window. Expand releases remain
   compatible with both old and new application versions; contract/destructive
   releases occur only after old binaries can no longer run.
4. Migration status/drift is checked before changing the live release. A
   failed migration blocks deployment and enters an operator runbook; it must
   never be hidden by repeated restart attempts.
5. Rollback means switching to a binary proven compatible with the current
   schema or deploying a new forward migration. Restoring a backup is the last
   resort and must also restore/reconcile the workspace recovery point.

## systemd stop, restart, and dependency semantics

- When `ExecStartPre=`, `ExecStart=`, or `ExecStartPost=` fails or times out,
  service startup does not continue to the normal running process. `ExecStop=`
  is skipped because startup never succeeded; `ExecStopPost=` is the hook that
  still runs after a failed startup.
  ([systemd start/stop command semantics](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStartPre=),
  [`ExecStopPost=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStopPost=),
  [permanent upstream man-page source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemd.service.xml#L525-L587))
- `Restart=on-failure` covers non-zero exits and failures/timeouts in service
  control processes, including `ExecStartPre=`. An explicit systemd stop does
  not trigger automatic restart. Restarts are subject to unit start-rate
  limiting.
  ([`Restart=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Restart=),
  [permanent upstream source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemd.service.xml#L856-L995))
- With no `ExecStop=`, systemd sends `KillSignal=` (default `SIGTERM`).
  `TimeoutStopSec=` bounds the wait; after it expires, remaining processes can
  be forcibly killed with `SIGKILL`/`FinalKillSignal=`.
  ([`ExecStop=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStop=),
  [`TimeoutStopSec=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutStopSec=))
- Default `KillMode=control-group` applies termination to all remaining
  processes in the unit cgroup. systemd explicitly discourages `process` or
  `none` because children can escape the service lifecycle and remain running
  after the service is considered stopped.
  ([`KillMode=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html#KillMode=))
- `systemctl stop` waits for the job unless `--no-block` is used, but the command
  does not fail merely because an `ExecStop=` command failed: the manager still
  forcibly terminates the unit.
  ([`systemctl stop`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#stop%20PATTERN%E2%80%A6),
  [permanent upstream source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemctl.xml#L444-L459))
- `systemctl restart` is a stop followed by a start; `ExecStop=` and
  `ExecStopPost=` participate. It does not necessarily flush every unit resource
  in the same way as separate completed `stop` then `start` operations.
  ([restart semantics](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStop=),
  [`systemctl restart`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#restart%20PATTERN%E2%80%A6))
- `After=` is only an ordering relationship; it does not pull in another unit.
  Requirement and ordering dependencies are independent. During shutdown the
  start order is reversed, so a Hub ordered `After=postgresql.service` stops
  before PostgreSQL when both are in the shutdown transaction.
  ([`Requires=` and ordering](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Requires=),
  [`Before=`/`After=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Before=))
- `systemctl freeze` only uses the cgroup freezer to suspend process execution;
  its documented contract contains no filesystem flush or stable-image
  guarantee. It is not a substitute for an application drain or a filesystem
  snapshot barrier.
  ([permanent `systemctl freeze` man-page source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemctl.xml#L653-L675))

Derived assessment of the current unit:

- A Prisma migration failure in the current `ExecStartPre=` prevents the Hub
  from starting, then `Restart=on-failure` can retry startup until rate-limited.
  That is fail-closed, but it is not controlled migration recovery and can
  repeatedly hit a database already left in Prisma's failed state.
- `KillSignal=SIGTERM` plus `TimeoutStopSec=30` is only a deadline. Until the Hub
  installs a real termination handler that makes readiness false, stops ingress,
  drains/checkpoints active work, closes Feishu and database clients, and reports
  success, the unit cannot claim a graceful consistency barrier.
- A backup runbook must issue a controlled drain, verify durable work reached a
  reconciled terminal/checkpoint state, stop the Hub, verify the unit and its
  cgroup are inactive, and only then snapshot mutable workspace state. A zero
  `systemctl stop` exit by itself is insufficient evidence.
- Recovery should not start the normal unit until the restored database and
  workspace have been validated. The current unit would run pending migrations
  immediately on first start, mutating the restored database before validation
  and possibly destroying the ability to inspect the original recovered state.

## Filesystem and rsync boundaries

- PostgreSQL says a plain filesystem copy of a live data directory is unusable:
  the database must be shut down, because tools such as `tar` do not create an
  atomic snapshot and server buffering remains. A correctly implemented frozen
  snapshot can work while PostgreSQL runs, but all volumes containing data,
  WAL, and tablespaces must be captured simultaneously; otherwise shut down or
  use the continuous-archive base-backup protocol.
  ([PostgreSQL filesystem backups](https://www.postgresql.org/docs/current/backup-file.html))
- For a PostgreSQL rsync backup, the documented safe pattern is a first pass
  while running, followed by shutdown and a second `rsync --checksum`; the
  second pass is what makes the result consistent.
  ([PostgreSQL filesystem backups](https://www.postgresql.org/docs/current/backup-file.html))
- At the filesystem layer, `fsfreeze` is documented to let in-progress
  filesystem operations complete, halt new modifications, and flush dirty data,
  metadata, and logs before returning. That is still only a filesystem-volume
  guarantee; it does not replace PostgreSQL's backup protocol and does not make
  separately frozen volumes simultaneous.
  ([util-linux `fsfreeze` manual source](https://github.com/util-linux/util-linux/blob/master/sys-utils/fsfreeze.8.adoc))
- rsync's `--delete` defaults to deletion during transfer on modern peers, so
  the receiver tree changes incrementally. `--delay-updates` merely holds each
  updated file and renames files into place rapidly at the end; its own manual
  says this only attempts to make updates “a little more atomic” and points to
  a separate parallel-tree/switch approach for stronger atomicity.
  ([rsync `--delete`](https://rsync.samba.org/ftp/rsync/rsync.1.html#opt--delete),
  [rsync `--delay-updates`](https://rsync.samba.org/ftp/rsync/rsync.1.html#opt--delay-updates))

Derived Hub implications:

- The current in-place `rsync --delete` deployment is neither an immutable
  release switch nor a backup snapshot. A failed/interrupted transfer can leave
  a mixed live tree; service stop/restart cannot reconstruct the deleted prior
  artifact.
- The workspace snapshot mechanism remains an explicit design choice. Whichever
  storage technology is selected must prove atomicity over all workspace
  volumes or use a quiesced/no-writer interval, then publish an immutable
  version/checksum manifest. Merely copying a changing directory is not enough.

## Derived minimum whole-Hub recovery contract

The primary sources support the following minimum contract; exact RPO, RTO,
retention, storage vendor, and numeric schedules remain product/operations
decisions and must not be invented by implementation.

### Backup path

1. Maintain verified PostgreSQL base backups and a monitored continuous WAL
   archive, or obtain and test an explicitly equivalent managed-PostgreSQL
   guarantee. Keep logical dumps as a separate portability/selective-restore
   layer, not the sole PITR mechanism.
2. Keep PostgreSQL configuration, deployment/release manifests, migration
   history, and workspace snapshots in the backup inventory; WAL alone does not
   cover them.
3. Create a whole-Hub checkpoint only after ingress is drained and active
   workspace writers are stopped/reconciled, unless the eventual storage layer
   supplies a proven atomic snapshot/journal spanning the necessary stores.
4. Persist one signed/immutable checkpoint manifest joining PostgreSQL
   backup/WAL position, workspace snapshot/version, application revision,
   migration digest, tool versions, checksums, timestamps, and validation state.
5. Treat database/WAL/workspace backups as cross-tenant sensitive data. Apply
   least-privilege access, encryption, audit, deletion/retention, and an
   independently protected integrity manifest; WAL contains effectively all
   database changes and PostgreSQL advises protecting it from unauthorized
   readers.
   ([WAL archive confidentiality](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))

### Migration path

1. Serialize deploys and backup checkpoints; run a drift/status gate before
   migration, not just `migrate deploy`.
2. Rehearse each migration on a recent production-scale restore and record
   runtime, locks, disk/WAL growth, transaction choice, and failure recovery.
3. Require expand/contract compatibility for online changes. A release must
   declare which old/new binaries can run against each schema phase.
4. Separate migration orchestration from ordinary service restart so a failed
   migration produces one durable incident state and an operator decision—not
   an opaque restart loop.
5. Verify a restorable checkpoint before destructive change. Do not describe
   backup restore as a normal “rollback”; it is whole-service disaster recovery.

### Restore and promotion path

1. Restore into an isolated environment with real external side effects
   disabled.
2. Restore the selected PostgreSQL base backup/WAL target and the checkpoint's
   exact workspace version; do not run pending Prisma migrations implicitly.
3. Prove artifact integrity, database startup, recovery target/timeline,
   migration/schema identity, tenant invariants, workspace hashes/containment,
   and read-only application flows.
4. Record observed recovery point and recovery time. A restore that starts but
   misses its target, loses a tenant workspace, has partial migration state, or
   fails any invariant is not promotable.
5. Promote explicitly, then enable ingress and external integrations. Preserve
   the pre-recovery cluster/WAL and recovery report until the incident is closed.

## Decisions and evidence still required

These values are not fixed by the upstream tools and remain `OPEN` for the SaaS
production contract:

- whole-Hub RPO and RTO, separately for ordinary deletion, database corruption,
  host loss, region/provider loss, and migration failure;
- base-backup, logical-dump, workspace-snapshot, WAL, and restore-drill cadence;
- immutable/off-host storage, encryption/key ownership, retention and verified
  deletion policy, including org-deletion and legal-retention interactions;
- exact workspace checkpoint/journal technology and database/workspace
  reconciliation semantics for in-flight or partially written runs;
- whether restore promotion is whole-platform only or supports a safe
  organization/project export-import path (PostgreSQL physical PITR is
  cluster-wide);
- migration transaction policy, maximum lock/downtime budget, compatibility
  window, and who can authorize contract/destructive phases;
- evidence from at least one production-like full restore and one PITR drill,
  including deliberate missing-WAL, corrupted-backup, failed-migration,
  interrupted-workspace-write, and restart-during-recovery probes.