forked from EduCraft/curriculum-project-hub
docs: resolve backup recovery audit
This commit is contained in:
@@ -0,0 +1,574 @@
|
||||
# Backup, migration, and disaster-recovery primary-source research
|
||||
|
||||
This note is the primary-source evidence pack for the Wayfinder ticket
|
||||
“Audit backup, migration, and disaster recovery safety.” External claims use
|
||||
only PostgreSQL, Prisma, systemd, and rsync first-party documentation. Project
|
||||
recommendations are explicitly labelled as derived implications rather than
|
||||
upstream guarantees.
|
||||
|
||||
PostgreSQL links below use `docs/current`, which was PostgreSQL 18 when this
|
||||
research was performed on 2026-07-10. A production implementation must pin the
|
||||
deployed PostgreSQL major and verify the same pages for that major. The Hub
|
||||
currently depends on Prisma `^6.19.3`, so Prisma 6-specific command examples
|
||||
remain relevant even where the live documentation also describes Prisma 7.
|
||||
|
||||
## Executive conclusions
|
||||
|
||||
1. `pg_dump` can take a consistent **single-database** snapshot while ordinary
|
||||
reads and writes continue, but it does not include the Hub's workspace tree,
|
||||
cluster-global roles/tablespaces, or a WAL chain. PostgreSQL now explicitly
|
||||
says it is generally not the right regular production-backup mechanism
|
||||
outside simple cases. It is a useful logical/export layer, not proof of
|
||||
whole-service disaster recovery.
|
||||
([`pg_dump` description](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION),
|
||||
[continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING))
|
||||
2. Database PITR requires a physical base backup plus a continuous, unbroken
|
||||
WAL sequence from the backup start through the recovery target. A successful
|
||||
base-backup command alone does not prove that chain exists.
|
||||
([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING),
|
||||
[making a base backup](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-BASE-BACKUP))
|
||||
3. `pg_verifybackup` is necessary but not sufficient: PostgreSQL explicitly
|
||||
requires test restores followed by checks that the restored database works
|
||||
and contains the expected data.
|
||||
([`pg_verifybackup`](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP))
|
||||
4. Prisma `migrate deploy` applies pending migrations, but it does not detect
|
||||
schema drift and, on PostgreSQL, Prisma does not wrap a migration in a
|
||||
transaction by default. A failed migration can therefore leave partial
|
||||
schema changes requiring explicit operator recovery.
|
||||
([Prisma production workflow](https://www.prisma.io/docs/orm/prisma-migrate/workflows/development-and-production#production-and-testing-environments),
|
||||
[Prisma transactional behaviour](https://www.prisma.io/blog/prisma-migrate-dx-primitives#what-if-schema-migrations-were-atomic),
|
||||
[failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration))
|
||||
5. systemd supplies process lifecycle orchestration, not an application-level
|
||||
consistency barrier. It sends the configured termination signal, waits up
|
||||
to `TimeoutStopSec=`, then can forcibly kill remaining processes; a
|
||||
successful `systemctl stop` does not prove that an `ExecStop=` command or
|
||||
application drain succeeded.
|
||||
([`TimeoutStopSec=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutStopSec=),
|
||||
[`systemctl stop`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#stop%20PATTERN%E2%80%A6))
|
||||
6. The Hub persists one logical service state across two stores: PostgreSQL
|
||||
records `Project.workspaceDir`, runs, messages, locks, and file-change audit
|
||||
rows, while workspaces contain project files. A workspace transcript helper
|
||||
exists but is not wired into the production runner. PostgreSQL's snapshot/
|
||||
PITR guarantees cover only PostgreSQL. A whole-service recovery point
|
||||
therefore requires an application-defined database/workspace checkpoint or
|
||||
a journaled/versioned reconciliation protocol; neither PostgreSQL nor
|
||||
systemd creates one automatically.
|
||||
([Hub schema](../../../hub/prisma/schema.prisma),
|
||||
[workspace transcript helper](../../../hub/src/agent/transcript.ts),
|
||||
[`pg_dump` scope](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION))
|
||||
|
||||
## Current repository facts that make the boundaries material
|
||||
|
||||
- The systemd unit runs `prisma migrate deploy` as `ExecStartPre=` before every
|
||||
service start and uses `Restart=on-failure`.
|
||||
([`cph-hub.service`](../../../hub/deploy/cph-hub.service))
|
||||
- The deploy script updates the live Hub directory using
|
||||
`rsync -az --delete`, builds in that same directory, restarts the unit, and
|
||||
accepts a constant liveness endpoint as deployment success.
|
||||
([`deploy_platform.sh`](../../../hub/deploy/deploy_platform.sh))
|
||||
- `Project.workspaceDir` is database state, but the referenced directory holds
|
||||
independently mutable tenant content. A workspace JSONL transcript helper
|
||||
exists but is not wired into the production runner; file contents still live
|
||||
in the workspace while the database separately carries run/message/file-change
|
||||
facts.
|
||||
([Hub schema](../../../hub/prisma/schema.prisma),
|
||||
[workspace transcript helper](../../../hub/src/agent/transcript.ts))
|
||||
|
||||
These are project facts, not upstream guarantees. The sections below establish
|
||||
what the upstream tools do and, critically, what they do not do.
|
||||
|
||||
## PostgreSQL logical dump and restore
|
||||
|
||||
### Snapshot and concurrent-write semantics
|
||||
|
||||
- `pg_dump` makes a consistent export even while the database is being used;
|
||||
it does not block ordinary readers or writers. It dumps only one database.
|
||||
Cluster-global objects such as roles and tablespaces require `pg_dumpall` or
|
||||
`pg_dumpall --globals-only`.
|
||||
([`pg_dump` description](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION),
|
||||
[whole-cluster logical dumps](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-ALL))
|
||||
- `pg_dumpall` invokes `pg_dump` once per database. Each database is internally
|
||||
consistent, but the snapshots are not synchronized across databases.
|
||||
([whole-cluster logical dumps](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-ALL))
|
||||
- A parallel dump uses directory format, opens `njobs + 1` connections, and
|
||||
uses synchronized snapshots so every worker sees the same data set. The
|
||||
connection count and additional server load are part of its capacity cost.
|
||||
([`pg_dump --jobs`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
|
||||
- Concurrent DDL can still interfere. The parallel-dump leader holds
|
||||
`ACCESS SHARE` locks; if a later exclusive-lock request queues ahead of a
|
||||
worker, that worker uses `NOWAIT` and aborts the dump rather than deadlocking.
|
||||
`--lock-wait-timeout` can bound the initial lock wait.
|
||||
([`pg_dump --jobs` and lock behaviour](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
|
||||
- The ordinary snapshot is suitable for disaster-recovery dump semantics, but
|
||||
it need not correspond to a serial execution of all transactions that later
|
||||
commit. `--serializable-deferrable` waits for an anomaly-free snapshot;
|
||||
PostgreSQL says that option is not beneficial for a dump intended only for
|
||||
disaster recovery, but it can matter for a continuously updated reporting
|
||||
copy.
|
||||
([`--serializable-deferrable`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
|
||||
- The default waits for dump files to be written safely to disk. Production
|
||||
backups must not use `--no-sync`, because an immediate OS crash can otherwise
|
||||
leave a corrupt dump.
|
||||
([`pg_dump --no-sync`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- An online database dump may be internally valid while referring to a
|
||||
`workspaceDir` whose files are from another instant. “`pg_dump` succeeded” is
|
||||
therefore not a whole-Hub consistency claim.
|
||||
- Backup scheduling must either serialize against schema migration/DDL or
|
||||
treat any lock wait, abort, non-zero exit, and stderr warning as a failed
|
||||
backup. A partially written archive must never be published as restorable.
|
||||
- If the deployed cluster contains more than the Hub database, a single Hub
|
||||
dump is not a cluster backup. Even `pg_dumpall` does not create a single
|
||||
cross-database snapshot.
|
||||
|
||||
### Restore error and atomicity semantics
|
||||
|
||||
- `pg_restore` defaults to continuing after SQL errors and reports an error
|
||||
count at the end. `--exit-on-error` is required for fail-fast behaviour.
|
||||
`--single-transaction` wraps all emitted commands in `BEGIN`/`COMMIT`, makes
|
||||
the restore all-or-nothing, and implies `--exit-on-error`.
|
||||
([`pg_restore` error and transaction options](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS))
|
||||
- Parallel restore uses separate connections for data loading, index creation,
|
||||
and constraint creation. It supports only custom/directory archives and
|
||||
cannot be combined with `--single-transaction`; it is not a whole-restore
|
||||
atomic operation.
|
||||
([`pg_restore --jobs`](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS))
|
||||
- A single giant restore transaction can be impractical because it locks every
|
||||
restored object and may exhaust the lock table. PostgreSQL 18 offers
|
||||
`--transaction-size=N` as a bounded intermediate option; it implies
|
||||
`--exit-on-error` but is not all-or-nothing.
|
||||
([`pg_restore --transaction-size`](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS))
|
||||
- Plain SQL restored by `psql` also continues after errors unless
|
||||
`ON_ERROR_STOP` is enabled; a single transaction is separately required for
|
||||
all-or-nothing behaviour.
|
||||
([restoring an SQL dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE))
|
||||
- PostgreSQL recommends restoring into a truly empty database based on
|
||||
`template0`; otherwise local additions in `template1` can cause duplicate
|
||||
definition errors. Required roles must exist before owner/ACL restoration.
|
||||
([restoring an SQL dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE))
|
||||
- A dump can execute arbitrary code selected by superusers of the source
|
||||
database. Partial dump/restore does not remove that trust boundary.
|
||||
([`pg_dump` security warning](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION))
|
||||
- A `pg_dump` binary can dump older servers but refuses a server newer than its
|
||||
own major version. Output is intended to load into newer PostgreSQL versions,
|
||||
but loading into an older major is not guaranteed.
|
||||
([`pg_dump` cross-version notes](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-NOTES))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- Restore should target a new isolated database/cluster. If a non-atomic or
|
||||
parallel restore reports any error, discard that target and start again;
|
||||
never promote a partially restored database.
|
||||
- The runbook must record the restore atomicity strategy and exact PostgreSQL
|
||||
client/server versions. A green process exit without explicit fail-fast
|
||||
options is insufficient evidence.
|
||||
- A restored database must remain disconnected from real Feishu/provider
|
||||
side-effect endpoints until application-level validation completes.
|
||||
|
||||
## Physical backup, WAL, and PITR
|
||||
|
||||
### Base backups
|
||||
|
||||
- `pg_basebackup` backs up a running PostgreSQL cluster without blocking other
|
||||
clients. A full backup is an exact copy of the entire cluster and can seed
|
||||
PITR or a standby; it cannot select one database or object.
|
||||
([`pg_basebackup` description](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
- It uses the replication protocol. The account needs `REPLICATION` or
|
||||
superuser privilege, `pg_hba.conf` must permit it, and `max_wal_senders`
|
||||
needs capacity for the backup and, when streaming WAL, another connection.
|
||||
([`pg_basebackup` requirements](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
- Default `-X stream` obtains WAL over a second connection while copying. With
|
||||
`-X fetch`, WAL is collected only at the end; if required WAL was recycled,
|
||||
the backup fails and is unusable.
|
||||
([`pg_basebackup --wal-method`](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
- The default waits for backup files to be durable. `--no-sync` is explicitly
|
||||
unsuitable for production because an OS crash can corrupt the backup.
|
||||
([`pg_basebackup --no-sync`](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
- A manifest is generated by default, with per-file checksums. A SHA manifest
|
||||
can detect malicious modification only if the manifest itself is held in a
|
||||
separate trusted location or otherwise authenticated.
|
||||
([`pg_basebackup` manifest options](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
|
||||
Derived minimum backup record:
|
||||
|
||||
- source cluster/system identifier and PostgreSQL version;
|
||||
- backup tool version, start/end time, start/end LSN and WAL range;
|
||||
- manifest digest and its independently protected location;
|
||||
- destination object/version, byte count, exit status, and verification state;
|
||||
- an explicit `VALID`, `INVALID`, or `QUARANTINED` state so a failed leftover
|
||||
directory cannot enter the retention set merely because files exist.
|
||||
|
||||
### WAL archival and recovery
|
||||
|
||||
- PITR combines a physical backup with WAL replay. It can stop at a chosen time
|
||||
and reach a consistent database state. Logical dumps do not contain enough
|
||||
information for WAL replay.
|
||||
([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING))
|
||||
- Successful recovery requires a continuous sequence of archived WAL extending
|
||||
back at least to the base-backup start. PostgreSQL tells operators to set up
|
||||
and test WAL archiving **before** taking the first base backup.
|
||||
([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING))
|
||||
- The archive command must return zero if and only if durable archival
|
||||
succeeded. Non-zero causes retry. Archive implementations should refuse to
|
||||
overwrite an existing file; a repeated name may report success only when the
|
||||
existing durable content is identical, and differing content must fail.
|
||||
([WAL archiving](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
- Repeated archive failure accumulates WAL in `pg_wal`; filling that filesystem
|
||||
causes PostgreSQL to panic and remain offline until space is freed. PostgreSQL
|
||||
explicitly advises monitoring archival progress.
|
||||
([WAL archiving failure behaviour](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
- WAL is archived after a segment is completed, so low traffic can make the
|
||||
last committed change wait a long time for archival. `archive_timeout` can
|
||||
bound that age by forcing segment switches, at the cost of archive growth.
|
||||
([WAL archiving latency](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
- WAL archival does not restore manually edited PostgreSQL configuration files
|
||||
(`postgresql.conf`, `pg_hba.conf`, `pg_ident.conf`); those need a separate
|
||||
file backup.
|
||||
([WAL archiving caveat](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
- PostgreSQL's recovery procedure keeps clients out, restores the base backup,
|
||||
configures `restore_command` and `recovery.signal`, replays WAL, and requires
|
||||
inspection of database contents before restoring ordinary access.
|
||||
([PITR recovery procedure](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-PITR-RECOVERY))
|
||||
- Recovery can target a timestamp, LSN, transaction ID, named restore point, or
|
||||
the earliest consistent point. Only one target may be set. Timeline history
|
||||
is required to navigate recovery branches, and a configured target not
|
||||
reached before WAL exhaustion is a fatal recovery failure.
|
||||
([recovery target settings](https://www.postgresql.org/docs/current/runtime-config-wal.html#RUNTIME-CONFIG-WAL-RECOVERY-TARGET),
|
||||
[timelines](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-TIMELINES))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- The database RPO is the age of the newest **durably archived, continuous**
|
||||
WAL position, not the age of the newest base backup and not merely the name
|
||||
in `last_archived_wal`.
|
||||
- Archive keys need a collision-proof cluster/system-identifier namespace and
|
||||
create-if-absent semantics; unrelated clusters must never overwrite a shared
|
||||
WAL name.
|
||||
- Monitoring must cover archive failures, time since durable success, archive
|
||||
lag, WAL continuity, `pg_wal` growth, free space, and base-backup age.
|
||||
- PITR restores PostgreSQL only. It does not rewind workspace files. A database
|
||||
recovery target has no whole-Hub meaning until paired with a workspace
|
||||
recovery point and reconciliation rule.
|
||||
|
||||
## Verification is a ladder, not a checksum
|
||||
|
||||
- `pg_verifybackup` validates the backup manifest and system identifier,
|
||||
detects missing/extra/changed files, checks file checksums, and parses the
|
||||
required WAL with `pg_waldump`. WAL verification is currently supported only
|
||||
for plain-format base backups.
|
||||
([`pg_verifybackup` checks](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP))
|
||||
- PostgreSQL explicitly says these checks cannot cover everything a running
|
||||
server will encounter and still requires test restores plus checks that the
|
||||
database works and contains correct data.
|
||||
([`pg_verifybackup` limitation](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP))
|
||||
- `pg_amcheck` can check supported heap/index structures, but currently only
|
||||
ordinary/TOAST tables, materialized views, sequences, and B-tree indexes are
|
||||
supported; other relation types are silently skipped. Stronger B-tree checks
|
||||
can block concurrent modifications.
|
||||
([`pg_amcheck`](https://www.postgresql.org/docs/current/app-pgamcheck.html#APP-PGAMCHECK))
|
||||
|
||||
Derived recovery-evidence ladder:
|
||||
|
||||
1. **Artifact verification:** backup command succeeded; manifest and all
|
||||
required WAL validate; any failure marks the artifact unusable.
|
||||
2. **Isolated technical restore:** restore to a clean non-production target,
|
||||
reach the intended LSN/time/timeline, and prove PostgreSQL starts without
|
||||
exposing it to users.
|
||||
3. **Database validation:** verify migration history/schema identity, tenant
|
||||
counts and relationships, critical constraints/sequences, and appropriate
|
||||
corruption checks.
|
||||
4. **Workspace validation:** restore the paired snapshot/version, validate file
|
||||
hashes/ownership/containment, and reconcile database file-change/run state
|
||||
against actual files.
|
||||
5. **Application validation:** run read-only tenant-isolation and core workflow
|
||||
smoke tests with external side effects disabled.
|
||||
6. **Recovery report:** persist the exact backup, WAL range, workspace point,
|
||||
release/schema versions, checks, observed RPO/RTO, operator, and failures.
|
||||
Only a fully passing report may authorize promotion.
|
||||
|
||||
## Prisma Migrate production semantics
|
||||
|
||||
### What `migrate deploy` proves and does not prove
|
||||
|
||||
- In production, `migrate deploy` compares applied migration checksums with the
|
||||
migration directory, warns when an applied migration was modified, and
|
||||
applies pending migrations.
|
||||
([Prisma 6 production workflow](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#production-and-testing-environments))
|
||||
- It does **not** warn when an already-applied migration is missing, detect
|
||||
production schema drift, reset the database, generate Prisma Client, or use a
|
||||
shadow database.
|
||||
([Prisma 6 production workflow](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#production-and-testing-environments))
|
||||
- The migration directory is the deployable history and must be committed in
|
||||
full, including `migration_lock.toml`. Prisma warns against editing or
|
||||
deleting migrations already applied in production; `migrate deploy` executes
|
||||
migration SQL rather than deriving a new migration from the Prisma schema.
|
||||
([Prisma migration histories](https://www.prisma.io/docs/orm/prisma-migrate/understanding-prisma-migrate/migration-histories))
|
||||
- Prisma uses a database advisory lock for production migration commands so
|
||||
multiple migration commands do not run simultaneously. The documented lock
|
||||
timeout is ten seconds and not configurable; after timeout, the caller must
|
||||
retry. The lock can be disabled by environment variable.
|
||||
([Prisma 6 advisory locking](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#advisory-locking))
|
||||
|
||||
Derived Hub implication: `migrate deploy` exit zero proves that this invocation
|
||||
applied or found no pending migration. It does not prove the live schema equals
|
||||
the expected migration end state, that the new and previous application builds
|
||||
are both compatible with it, or that the migration is operationally safe under
|
||||
production data volume and traffic.
|
||||
|
||||
The advisory lock is a migration-command concurrency guard, not a substitute
|
||||
for the repository's deployment lock or application/schema compatibility. As a
|
||||
derived safety rule, production preflight should reject
|
||||
`PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK` unless an explicit reviewed mechanism
|
||||
provides the same serialization guarantee.
|
||||
|
||||
### Transactions and failed migrations
|
||||
|
||||
- Prisma's first-party migration guidance states that PostgreSQL migrations
|
||||
are not wrapped in a transaction by default. Operators may opt in by adding
|
||||
`BEGIN;`/`COMMIT;` to a migration, but large transactional migrations can hold
|
||||
locks longer and consume more database resources.
|
||||
([Prisma transactional behaviour](https://www.prisma.io/blog/prisma-migrate-dx-primitives#what-if-schema-migrations-were-atomic))
|
||||
- A failed migration records its error in `_prisma_migrations.logs`; further
|
||||
migrations cannot deploy until the failed state is explicitly recovered.
|
||||
Recovery is either (a) revert/repair partial steps, mark the migration rolled
|
||||
back, and redeploy, or (b) manually complete the exact steps and mark it
|
||||
applied.
|
||||
([Prisma failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration))
|
||||
- Prisma error codes `P3009` and `P3018` explicitly report that failed
|
||||
migrations block new migration application until recovery.
|
||||
([Prisma error reference](https://www.prisma.io/docs/orm/reference/error-reference#p3009))
|
||||
- `migrate resolve --rolled-back` changes migration-history state so Prisma may
|
||||
apply the migration again; it does not itself undo SQL already executed.
|
||||
`--applied` records success without executing the migration SQL.
|
||||
([failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration),
|
||||
[`migrate resolve`](https://www.prisma.io/docs/cli/migrate/resolve))
|
||||
- Prisma can generate a down SQL script, but its documentation limits the
|
||||
simple rollback workflow to a **failed** up migration. To reverse a successful
|
||||
migration, the supported history is a new forward migration generated from a
|
||||
reverted Prisma schema. A schema down script also cannot restore lost data,
|
||||
application changes, or manually authored SQL automatically.
|
||||
([generating down migrations](https://www.prisma.io/docs/orm/prisma-migrate/workflows/generating-down-migrations#considerations-when-generating-down-migrations))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- Every migration needs a declared atomicity choice. If it is intentionally
|
||||
non-transactional, the migration must be resumable/idempotent or ship an
|
||||
exact, tested partial-failure procedure. Blindly rerunning a partially applied
|
||||
script is unsafe.
|
||||
- A database backup checkpoint before a destructive or difficult-to-reverse
|
||||
migration is necessary, but restoring that checkpoint is a whole-database
|
||||
recovery event—not a substitute for a compatible rollout design.
|
||||
- Service restart cannot be treated as application rollback after a successful
|
||||
schema migration. The previous binary may no longer understand the new
|
||||
schema, and a down migration cannot recreate deleted data.
|
||||
|
||||
### Compatibility and rollout ordering
|
||||
|
||||
- Prisma documents expand-and-contract as the way to avoid downtime for fields
|
||||
in active use: add the new representation, deploy code that writes both while
|
||||
reading the old, migrate and verify data, switch reads, stop old writes, then
|
||||
remove the old representation.
|
||||
([Prisma expand-and-contract workflow](https://www.prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations#example-use-the-expand-and-contract-pattern-to-evolve-the-schema-without-downtime))
|
||||
- Prisma also warns that generated “renames” may be expressed as add/drop and
|
||||
lose data unless the SQL is reviewed and customized before application.
|
||||
([customizing migrations](https://www.prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations#example-rename-a-field))
|
||||
|
||||
Derived deployment contract:
|
||||
|
||||
1. Migration SQL is immutable after production application and reviewed for
|
||||
destructive operations, lock duration, data volume, and transaction choice.
|
||||
2. It is rehearsed against a recent, anonymized production-scale restore and
|
||||
its runtime/locks/disk growth are recorded.
|
||||
3. Each release declares a schema compatibility window. Expand releases remain
|
||||
compatible with both old and new application versions; contract/destructive
|
||||
releases occur only after old binaries can no longer run.
|
||||
4. Migration status/drift is checked before changing the live release. A
|
||||
failed migration blocks deployment and enters an operator runbook; it must
|
||||
never be hidden by repeated restart attempts.
|
||||
5. Rollback means switching to a binary proven compatible with the current
|
||||
schema or deploying a new forward migration. Restoring a backup is the last
|
||||
resort and must also restore/reconcile the workspace recovery point.
|
||||
|
||||
## systemd stop, restart, and dependency semantics
|
||||
|
||||
- When `ExecStartPre=`, `ExecStart=`, or `ExecStartPost=` fails or times out,
|
||||
service startup does not continue to the normal running process. `ExecStop=`
|
||||
is skipped because startup never succeeded; `ExecStopPost=` is the hook that
|
||||
still runs after a failed startup.
|
||||
([systemd start/stop command semantics](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStartPre=),
|
||||
[`ExecStopPost=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStopPost=),
|
||||
[permanent upstream man-page source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemd.service.xml#L525-L587))
|
||||
- `Restart=on-failure` covers non-zero exits and failures/timeouts in service
|
||||
control processes, including `ExecStartPre=`. An explicit systemd stop does
|
||||
not trigger automatic restart. Restarts are subject to unit start-rate
|
||||
limiting.
|
||||
([`Restart=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Restart=),
|
||||
[permanent upstream source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemd.service.xml#L856-L995))
|
||||
- With no `ExecStop=`, systemd sends `KillSignal=` (default `SIGTERM`).
|
||||
`TimeoutStopSec=` bounds the wait; after it expires, remaining processes can
|
||||
be forcibly killed with `SIGKILL`/`FinalKillSignal=`.
|
||||
([`ExecStop=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStop=),
|
||||
[`TimeoutStopSec=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutStopSec=))
|
||||
- Default `KillMode=control-group` applies termination to all remaining
|
||||
processes in the unit cgroup. systemd explicitly discourages `process` or
|
||||
`none` because children can escape the service lifecycle and remain running
|
||||
after the service is considered stopped.
|
||||
([`KillMode=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html#KillMode=))
|
||||
- `systemctl stop` waits for the job unless `--no-block` is used, but the command
|
||||
does not fail merely because an `ExecStop=` command failed: the manager still
|
||||
forcibly terminates the unit.
|
||||
([`systemctl stop`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#stop%20PATTERN%E2%80%A6),
|
||||
[permanent upstream source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemctl.xml#L444-L459))
|
||||
- `systemctl restart` is a stop followed by a start; `ExecStop=` and
|
||||
`ExecStopPost=` participate. It does not necessarily flush every unit resource
|
||||
in the same way as separate completed `stop` then `start` operations.
|
||||
([restart semantics](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStop=),
|
||||
[`systemctl restart`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#restart%20PATTERN%E2%80%A6))
|
||||
- `After=` is only an ordering relationship; it does not pull in another unit.
|
||||
Requirement and ordering dependencies are independent. During shutdown the
|
||||
start order is reversed, so a Hub ordered `After=postgresql.service` stops
|
||||
before PostgreSQL when both are in the shutdown transaction.
|
||||
([`Requires=` and ordering](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Requires=),
|
||||
[`Before=`/`After=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Before=))
|
||||
- `systemctl freeze` only uses the cgroup freezer to suspend process execution;
|
||||
its documented contract contains no filesystem flush or stable-image
|
||||
guarantee. It is not a substitute for an application drain or a filesystem
|
||||
snapshot barrier.
|
||||
([permanent `systemctl freeze` man-page source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemctl.xml#L653-L675))
|
||||
|
||||
Derived assessment of the current unit:
|
||||
|
||||
- A Prisma migration failure in the current `ExecStartPre=` prevents the Hub
|
||||
from starting, then `Restart=on-failure` can retry startup until rate-limited.
|
||||
That is fail-closed, but it is not controlled migration recovery and can
|
||||
repeatedly hit a database already left in Prisma's failed state.
|
||||
- `KillSignal=SIGTERM` plus `TimeoutStopSec=30` is only a deadline. Until the Hub
|
||||
installs a real termination handler that makes readiness false, stops ingress,
|
||||
drains/checkpoints active work, closes Feishu and database clients, and reports
|
||||
success, the unit cannot claim a graceful consistency barrier.
|
||||
- A backup runbook must issue a controlled drain, verify durable work reached a
|
||||
reconciled terminal/checkpoint state, stop the Hub, verify the unit and its
|
||||
cgroup are inactive, and only then snapshot mutable workspace state. A zero
|
||||
`systemctl stop` exit by itself is insufficient evidence.
|
||||
- Recovery should not start the normal unit until the restored database and
|
||||
workspace have been validated. The current unit would run pending migrations
|
||||
immediately on first start, mutating the restored database before validation
|
||||
and possibly destroying the ability to inspect the original recovered state.
|
||||
|
||||
## Filesystem and rsync boundaries
|
||||
|
||||
- PostgreSQL says a plain filesystem copy of a live data directory is unusable:
|
||||
the database must be shut down, because tools such as `tar` do not create an
|
||||
atomic snapshot and server buffering remains. A correctly implemented frozen
|
||||
snapshot can work while PostgreSQL runs, but all volumes containing data,
|
||||
WAL, and tablespaces must be captured simultaneously; otherwise shut down or
|
||||
use the continuous-archive base-backup protocol.
|
||||
([PostgreSQL filesystem backups](https://www.postgresql.org/docs/current/backup-file.html))
|
||||
- For a PostgreSQL rsync backup, the documented safe pattern is a first pass
|
||||
while running, followed by shutdown and a second `rsync --checksum`; the
|
||||
second pass is what makes the result consistent.
|
||||
([PostgreSQL filesystem backups](https://www.postgresql.org/docs/current/backup-file.html))
|
||||
- At the filesystem layer, `fsfreeze` is documented to let in-progress
|
||||
filesystem operations complete, halt new modifications, and flush dirty data,
|
||||
metadata, and logs before returning. That is still only a filesystem-volume
|
||||
guarantee; it does not replace PostgreSQL's backup protocol and does not make
|
||||
separately frozen volumes simultaneous.
|
||||
([util-linux `fsfreeze` manual source](https://github.com/util-linux/util-linux/blob/master/sys-utils/fsfreeze.8.adoc))
|
||||
- rsync's `--delete` defaults to deletion during transfer on modern peers, so
|
||||
the receiver tree changes incrementally. `--delay-updates` merely holds each
|
||||
updated file and renames files into place rapidly at the end; its own manual
|
||||
says this only attempts to make updates “a little more atomic” and points to
|
||||
a separate parallel-tree/switch approach for stronger atomicity.
|
||||
([rsync `--delete`](https://rsync.samba.org/ftp/rsync/rsync.1.html#opt--delete),
|
||||
[rsync `--delay-updates`](https://rsync.samba.org/ftp/rsync/rsync.1.html#opt--delay-updates))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- The current in-place `rsync --delete` deployment is neither an immutable
|
||||
release switch nor a backup snapshot. A failed/interrupted transfer can leave
|
||||
a mixed live tree; service stop/restart cannot reconstruct the deleted prior
|
||||
artifact.
|
||||
- The workspace snapshot mechanism remains an explicit design choice. Whichever
|
||||
storage technology is selected must prove atomicity over all workspace
|
||||
volumes or use a quiesced/no-writer interval, then publish an immutable
|
||||
version/checksum manifest. Merely copying a changing directory is not enough.
|
||||
|
||||
## Derived minimum whole-Hub recovery contract
|
||||
|
||||
The primary sources support the following minimum contract; exact RPO, RTO,
|
||||
retention, storage vendor, and numeric schedules remain product/operations
|
||||
decisions and must not be invented by implementation.
|
||||
|
||||
### Backup path
|
||||
|
||||
1. Maintain verified PostgreSQL base backups and a monitored continuous WAL
|
||||
archive, or obtain and test an explicitly equivalent managed-PostgreSQL
|
||||
guarantee. Keep logical dumps as a separate portability/selective-restore
|
||||
layer, not the sole PITR mechanism.
|
||||
2. Keep PostgreSQL configuration, deployment/release manifests, migration
|
||||
history, and workspace snapshots in the backup inventory; WAL alone does not
|
||||
cover them.
|
||||
3. Create a whole-Hub checkpoint only after ingress is drained and active
|
||||
workspace writers are stopped/reconciled, unless the eventual storage layer
|
||||
supplies a proven atomic snapshot/journal spanning the necessary stores.
|
||||
4. Persist one signed/immutable checkpoint manifest joining PostgreSQL
|
||||
backup/WAL position, workspace snapshot/version, application revision,
|
||||
migration digest, tool versions, checksums, timestamps, and validation state.
|
||||
5. Treat database/WAL/workspace backups as cross-tenant sensitive data. Apply
|
||||
least-privilege access, encryption, audit, deletion/retention, and an
|
||||
independently protected integrity manifest; WAL contains effectively all
|
||||
database changes and PostgreSQL advises protecting it from unauthorized
|
||||
readers.
|
||||
([WAL archive confidentiality](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
|
||||
### Migration path
|
||||
|
||||
1. Serialize deploys and backup checkpoints; run a drift/status gate before
|
||||
migration, not just `migrate deploy`.
|
||||
2. Rehearse each migration on a recent production-scale restore and record
|
||||
runtime, locks, disk/WAL growth, transaction choice, and failure recovery.
|
||||
3. Require expand/contract compatibility for online changes. A release must
|
||||
declare which old/new binaries can run against each schema phase.
|
||||
4. Separate migration orchestration from ordinary service restart so a failed
|
||||
migration produces one durable incident state and an operator decision—not
|
||||
an opaque restart loop.
|
||||
5. Verify a restorable checkpoint before destructive change. Do not describe
|
||||
backup restore as a normal “rollback”; it is whole-service disaster recovery.
|
||||
|
||||
### Restore and promotion path
|
||||
|
||||
1. Restore into an isolated environment with real external side effects
|
||||
disabled.
|
||||
2. Restore the selected PostgreSQL base backup/WAL target and the checkpoint's
|
||||
exact workspace version; do not run pending Prisma migrations implicitly.
|
||||
3. Prove artifact integrity, database startup, recovery target/timeline,
|
||||
migration/schema identity, tenant invariants, workspace hashes/containment,
|
||||
and read-only application flows.
|
||||
4. Record observed recovery point and recovery time. A restore that starts but
|
||||
misses its target, loses a tenant workspace, has partial migration state, or
|
||||
fails any invariant is not promotable.
|
||||
5. Promote explicitly, then enable ingress and external integrations. Preserve
|
||||
the pre-recovery cluster/WAL and recovery report until the incident is closed.
|
||||
|
||||
## Decisions and evidence still required
|
||||
|
||||
These values are not fixed by the upstream tools and remain `OPEN` for the SaaS
|
||||
production contract:
|
||||
|
||||
- whole-Hub RPO and RTO, separately for ordinary deletion, database corruption,
|
||||
host loss, region/provider loss, and migration failure;
|
||||
- base-backup, logical-dump, workspace-snapshot, WAL, and restore-drill cadence;
|
||||
- immutable/off-host storage, encryption/key ownership, retention and verified
|
||||
deletion policy, including org-deletion and legal-retention interactions;
|
||||
- exact workspace checkpoint/journal technology and database/workspace
|
||||
reconciliation semantics for in-flight or partially written runs;
|
||||
- whether restore promotion is whole-platform only or supports a safe
|
||||
organization/project export-import path (PostgreSQL physical PITR is
|
||||
cluster-wide);
|
||||
- migration transaction policy, maximum lock/downtime budget, compatibility
|
||||
window, and who can authorize contract/destructive phases;
|
||||
- evidence from at least one production-like full restore and one PITR drill,
|
||||
including deliberate missing-WAL, corrupted-backup, failed-migration,
|
||||
interrupted-workspace-write, and restart-during-recovery probes.
|
||||
Reference in New Issue
Block a user