forked from bai/curriculum-project-hub
docs: resolve backup recovery audit
This commit is contained in:
+325
@@ -0,0 +1,325 @@
|
||||
# Backup, migration, and disaster-recovery audit
|
||||
|
||||
## Verdict
|
||||
|
||||
The current Hub has no production backup or restore control plane. PostgreSQL
|
||||
can be dumped and restored manually, but the service's authoritative state is
|
||||
split between PostgreSQL and mutable Project workspaces, with no shared
|
||||
consistency point, backup catalog, off-host retention, restore workflow,
|
||||
recovery objectives, or recovery drill.
|
||||
|
||||
The current deployment path can also delete customer workspaces when the
|
||||
documented relative workspace root is used: `deploy_platform.sh` applies
|
||||
`rsync --delete` to the live Hub directory while `.env.example` places
|
||||
workspaces below `./data/project-workspaces`. Project creation has the same
|
||||
cross-store weakness at a smaller scale: it creates the workspace before the
|
||||
database transaction and leaves an orphan directory when that transaction
|
||||
fails.
|
||||
|
||||
Migration safety is incomplete even though the current PostgreSQL migration
|
||||
engine rolled back a deliberately failing migration's DDL in the executed
|
||||
probe. The failed migration record blocks later migrations, migrations execute
|
||||
inside every service start, no compatibility declaration or production-data
|
||||
rehearsal exists, and current history contains destructive changes that make an
|
||||
old application query fail after schema upgrade. Application rollback is
|
||||
therefore not database rollback.
|
||||
|
||||
## Contract and persistent-state baseline
|
||||
|
||||
The audit preserves the accepted contracts rather than inventing missing
|
||||
product policy:
|
||||
|
||||
- `Organization` is the tenant root, but `User` is global and can belong to
|
||||
multiple Organizations. A selective Organization restore is not equivalent
|
||||
to restoring a self-contained database schema.
|
||||
- Project workspaces are authoritative curriculum-engineering files bounded by
|
||||
ADR-0018. PostgreSQL stores their absolute `Project.workspaceDir` locations;
|
||||
it does not store their file content.
|
||||
- Run, provider cursor, ingress, delivery, audit, and workspace commit/recovery
|
||||
semantics remain split across the unresolved lifecycle tickets. A backup
|
||||
design cannot silently choose their crash behavior.
|
||||
- ADR-0022 requires storage ceilings and emergency `DRAIN`/`STOP_NOW`, but does
|
||||
not decide backup frequency, retention, recovery objectives, or tenant-level
|
||||
restore scope.
|
||||
- Audit retention is still `OPEN`; regulatory backup retention and deletion
|
||||
obligations require product/legal input already recorded in the map's Fog.
|
||||
|
||||
The recovery set has at least four classes:
|
||||
|
||||
| State | Current location | Recovery significance |
|
||||
| --- | --- | --- |
|
||||
| Tenant and workflow state | PostgreSQL | Organizations, identities, membership, permissions, Feishu bindings, sessions, runs, locks, receipts, usage facts, and audit records |
|
||||
| Customer artifacts | `HUB_PROJECT_WORKSPACE_ROOT` | Project source files, generated artifacts, inbound attachments, and any workspace-local session/history files |
|
||||
| Credentials and configuration | `/srv/.../.secrets/platform.env` today; future org-scoped encrypted connections plus external key material | Database/Feishu/provider access and ability to decrypt restored Organization credentials |
|
||||
| Release identity | application release, Prisma migrations, `cph`, runtime/toolchain manifest | Required to start a schema-compatible application and reproduce outputs after restore |
|
||||
|
||||
Caches, build output, and reinstallable dependencies should be classified as
|
||||
regenerable rather than silently mixed into the protected recovery set.
|
||||
|
||||
## Executed evidence
|
||||
|
||||
All probes used isolated temporary paths or databases. No production or shared
|
||||
test data was modified.
|
||||
|
||||
### 1. The deployment rsync pattern deletes an in-tree workspace
|
||||
|
||||
A temporary live Hub directory was given the documented workspace shape and a
|
||||
sentinel customer file, then the exact deployment rsync include/exclude pattern
|
||||
was applied:
|
||||
|
||||
```text
|
||||
rsync -a --delete \
|
||||
--exclude node_modules --exclude dist --exclude .env \
|
||||
hub/ <temporary-live-hub>/
|
||||
|
||||
workspace_survived=false
|
||||
reproduced: deploy rsync pattern deletes an in-tree workspace root
|
||||
```
|
||||
|
||||
This is deterministic: `data/` is absent from the release source and is not an
|
||||
exclude, so `--delete` removes it from the destination. The root cause is that
|
||||
the documented persistent path is nested inside the mutable release tree and
|
||||
the deploy contract does not distinguish release artifacts from state.
|
||||
|
||||
### 2. A rejected Project transaction leaves an orphan workspace
|
||||
|
||||
A minimal Node harness called the real built `createProjectFromOrgAdmin` with a
|
||||
valid Organization admin and a missing Folder. The fake Prisma boundary rejected
|
||||
inside the same transaction seam used by production. The harness asserted that
|
||||
the Organization workspace directory must remain empty and was run twice:
|
||||
|
||||
```text
|
||||
node .scratch/saas-production-readiness/probes/orphan-workspace.mjs
|
||||
|
||||
orphan_workspace_entries=project_4aa0b6792c4547059d2d92b505b9299e
|
||||
exit_code=1
|
||||
|
||||
orphan_workspace_entries=project_190e21afb8854244872b1241a600aa58
|
||||
exit_code=1
|
||||
```
|
||||
|
||||
`createManagedProject` calls `mkdir` before `prisma.$transaction`; the failure
|
||||
path has no compensating removal, staged rename, or orphan reconciler. The
|
||||
throwaway harness was removed after reproduction.
|
||||
|
||||
### 3. PostgreSQL rolled back failed migration DDL but retained a blocking failure
|
||||
|
||||
An isolated database and throwaway Prisma project applied one migration that
|
||||
created a table, divided by zero, then would have created another table. The
|
||||
real repository Prisma CLI returned `P3018`:
|
||||
|
||||
```text
|
||||
migration_exit=1
|
||||
before_failure=missing
|
||||
after_failure=missing
|
||||
20260710000000_partial_failure|finished=false|rolled_back=false|steps=0|logs=true
|
||||
```
|
||||
|
||||
This specific PostgreSQL/Prisma 6.19.3 path did **not** leave partial DDL, so the
|
||||
audit does not claim that it did. It did leave an unfinished
|
||||
`_prisma_migrations` row, and Prisma's error explicitly refused new migrations
|
||||
until operator recovery. Because systemd runs `migrate deploy` as
|
||||
`ExecStartPre`, the application cannot start past that state. The throwaway
|
||||
database and migration were removed. This one multi-statement result is not a
|
||||
general atomicity guarantee: Prisma's first-party guidance says PostgreSQL
|
||||
migrations are not transaction-wrapped by default, so each migration must make
|
||||
its transaction and partial-failure behavior explicit.
|
||||
|
||||
### 4. The current schema is not backward-compatible with an older application
|
||||
|
||||
The application immediately before the typed-principal migration queried
|
||||
`PermissionGrant.principal`. That migration drops the column in favor of
|
||||
`principalType` plus `principalId`. Running the old read against the current
|
||||
test database produced:
|
||||
|
||||
```text
|
||||
ERROR: column "principal" does not exist
|
||||
HINT: Perhaps you meant to reference the column "PermissionGrant.principalId".
|
||||
old_query_exit=1
|
||||
```
|
||||
|
||||
This is direct evidence that switching the application symlink back after that
|
||||
migration cannot restore service. The migration history also deletes duplicate
|
||||
grant rows and later drops `ExternalPrincipalMembership.source`; there are no
|
||||
downgrade migrations.
|
||||
|
||||
### 5. A database-only custom dump restores cleanly
|
||||
|
||||
As a positive baseline, `pg_dump --format=custom` captured the current
|
||||
`cph_hub_test` database and `pg_restore --exit-on-error --no-owner` restored it
|
||||
into a new isolated database:
|
||||
|
||||
```text
|
||||
restored_tables=24
|
||||
finished_migrations=9
|
||||
failed_migrations=0
|
||||
dump_bytes=76113
|
||||
```
|
||||
|
||||
All nine finished migration checksums in the source database also matched the
|
||||
checked-in migration files. This proves a manual logical database round trip on
|
||||
the current PostgreSQL 14.20 workstation. It does not prove workspace,
|
||||
credentials, global database roles, production volume, cross-version restore,
|
||||
application readiness, or an operational RPO/RTO.
|
||||
|
||||
## Critical findings
|
||||
|
||||
### 1. No backup, retention, or restore mechanism exists
|
||||
|
||||
The repository has no backup/restore command, systemd timer, off-host target,
|
||||
encryption/immutability policy, retention policy, backup manifest/checksum,
|
||||
success/failure alert, restore command, or scheduled recovery drill. The
|
||||
installer assumes PostgreSQL and its role/database already exist. A database
|
||||
dump alone does not reproduce those global prerequisites or the filesystem and
|
||||
secret state above.
|
||||
|
||||
There is also no decided RPO, RTO, backup frequency, retention duration,
|
||||
geographic copy rule, deletion-in-backups rule, or ownership/escalation path.
|
||||
Those values are product and operations decisions, not safe constants to infer
|
||||
from the single-host topology.
|
||||
|
||||
### 2. Persistent workspaces are inside the documented release tree
|
||||
|
||||
`.env.example` recommends `./data/project-workspaces`; systemd's working
|
||||
directory is the live Hub directory; deployment mutates that same directory
|
||||
with `rsync --delete`. No preflight forbids an in-release workspace root and no
|
||||
exclude preserves it. This can turn a routine deploy into irreversible customer
|
||||
data loss before build, migration, restart, or readiness runs.
|
||||
|
||||
The persistent workspace root must live outside every release directory, have
|
||||
explicit ownership/mode and capacity, and be impossible for release retention
|
||||
or `rsync --delete` to traverse. Immutable release work already exists as a
|
||||
separate ticket, but its acceptance test must include this sentinel case.
|
||||
|
||||
### 3. PostgreSQL and workspaces have no consistency boundary
|
||||
|
||||
PostgreSQL can provide a transactionally consistent database snapshot, but the
|
||||
Hub mutates workspace files independently and has no generation, journal,
|
||||
commit marker, snapshot ID, or shared checkpoint that binds filesystem content
|
||||
to the corresponding database state. A database dump and filesystem copy taken
|
||||
at different instants can disagree about Project existence, active runs,
|
||||
provider cursors, delivered messages, and file effects.
|
||||
|
||||
Project creation already demonstrates the same missing compensation boundary.
|
||||
A replacement host has an additional problem: `Project.workspaceDir` is an
|
||||
absolute path and runtime paths from the database are used directly; restore
|
||||
has no same-mount validation, safe rebasing operation, or missing/orphan scan.
|
||||
|
||||
For the initial single-host topology, a quiesced backup checkpoint is the
|
||||
smallest credible route: stop new admission, drain/cancel according to the
|
||||
accepted run contract, reach durable zero in-flight work, capture database and
|
||||
workspace artifacts under one manifest, then resume. An online scheme would
|
||||
need a newly designed journal/generation protocol; it must not be implied by
|
||||
two independent copy commands.
|
||||
|
||||
### 4. Schema mutation is coupled to service start
|
||||
|
||||
Both `npm start` and the systemd unit execute `prisma migrate deploy`; every
|
||||
ordinary restart can therefore become a schema rollout. The release path has no
|
||||
separate migration lock/gate, `migrate status` check, production-data rehearsal,
|
||||
duration/lock budget, pre-migration recovery checkpoint, or compatibility
|
||||
manifest. CI only applies the full migration history to a fresh empty
|
||||
PostgreSQL database, so it does not exercise real upgrade volume, legacy values,
|
||||
old/new application overlap, failure recovery, or rollback compatibility.
|
||||
|
||||
The current failed-migration behavior is fail-closed, which is better than
|
||||
starting on an unknown schema, but there is no operator procedure for inspecting
|
||||
the failure, deciding roll-forward versus mark-rolled-back, restoring the
|
||||
checkpoint, or preventing a restart loop. Existing destructive migrations prove
|
||||
that application rollback cannot be assumed safe.
|
||||
|
||||
### 5. Restore has no isolated verification or cutover gate
|
||||
|
||||
There is no procedure that restores into a non-production environment with
|
||||
outbound Feishu/provider traffic disabled, verifies backup checksums and
|
||||
migration state, validates workspace paths/content/permissions, checks tenant
|
||||
relationships, decrypts a sample org connection, reconciles non-terminal work,
|
||||
runs application readiness and representative journeys, and measures recovery
|
||||
time/point before controlled cutover.
|
||||
|
||||
Starting the current server is not a neutral verifier: startup deletes all
|
||||
Project locks and independently marks only `ACTIVE` runs failed. Restored data
|
||||
must first pass the durable lifecycle reconciliation designed by the run-state
|
||||
tickets; otherwise validation itself mutates the evidence and can leave
|
||||
`WAITING_FOR_USER` inconsistent.
|
||||
|
||||
### 6. Full-service and Organization-selective recovery are different products
|
||||
|
||||
A full-service restore can recreate the shared database and all workspaces.
|
||||
Restoring one Organization is much harder: users are global and may belong to
|
||||
multiple Organizations, permission/audit/history rows span many related tables,
|
||||
workspace paths are external, and Feishu/provider connections have external
|
||||
state. The repository has no export/import identity mapping or conflict policy.
|
||||
|
||||
The initial production gate must at least prove full-service disaster recovery.
|
||||
Whether Organization-selective restore/export is promised, and how retained
|
||||
backups interact with customer deletion, remains an explicit product/legal
|
||||
decision rather than an accidental property of `pg_restore` filters.
|
||||
|
||||
## Required recovery contract
|
||||
|
||||
Production readiness requires the following guarantees, with exact numerical
|
||||
objectives supplied by the pending policy decision:
|
||||
|
||||
1. **Declared objectives and ownership.** RPO/RTO, backup frequency, retention,
|
||||
copy locations, encryption/key custody, immutability, deletion handling,
|
||||
full-service versus Organization restore scope, operator ownership, and
|
||||
escalation are documented and testable.
|
||||
2. **Complete backup set.** One cataloged manifest identifies release and
|
||||
migration checksums, PostgreSQL artifact and global prerequisites, workspace
|
||||
snapshot, configuration inventory, encrypted org-secret data, external key
|
||||
recovery material, timestamps, sizes, and cryptographic checksums. Backup
|
||||
data and decryption material are not kept only on the protected host or in
|
||||
the same failure domain.
|
||||
3. **One consistency point.** A backup is taken only after an observable
|
||||
quiesce/checkpoint protocol, or by a proven online generation protocol that
|
||||
binds database state and workspace effects. Accepted input, active runs,
|
||||
queues, deliveries, locks, and provider cursors have explicit recovery
|
||||
semantics.
|
||||
4. **Safe migration rollout.** Migration is a release phase, not an incidental
|
||||
service start. The gate checks status/checksums, rehearses on representative
|
||||
production-shaped data, records old/new application compatibility, uses
|
||||
expand-and-contract for destructive changes, requires a recent verified
|
||||
recovery checkpoint, and provides a tested failed-migration procedure.
|
||||
5. **Isolated restore verification.** Every backup class is periodically
|
||||
restored to a clean isolated host/database with outbound integrations off.
|
||||
Verification checks integrity, schema/migration state, global database
|
||||
prerequisites, workspace completeness/path ownership, secret decryption,
|
||||
tenant isolation, lifecycle reconciliation, application readiness, and
|
||||
representative admin/Feishu/agent journeys.
|
||||
6. **Observable operation.** Backup age, duration, artifact/checksum status,
|
||||
off-host replication, retention deletion, WAL/archive continuity if used,
|
||||
restore-drill result, measured RPO/RTO, and failures are logged/metricized
|
||||
and alerted with bounded cardinality.
|
||||
7. **Controlled cutover and fallback.** Disaster recovery has an explicit
|
||||
authority, traffic/workload brake, DNS/proxy/Feishu reconnect sequence,
|
||||
duplicate/gap checks, rollback/fallback point, verification checklist, and
|
||||
incident evidence record.
|
||||
|
||||
## Required release evidence
|
||||
|
||||
- A deploy test proves an in-tree or release-overlapping workspace root is
|
||||
rejected before any rsync/switch, while persistent sentinel files survive
|
||||
build failure, successful release, rollback, and release pruning.
|
||||
- Project creation, archive/deletion, failed transaction, crash, and restore
|
||||
tests prove database/workspace allocation has no silent missing or orphan
|
||||
state; the reconciler reports rather than hides inconsistencies.
|
||||
- Every checked-in migration is rehearsed from the previous production-shaped
|
||||
snapshot, with old/new compatibility and lock/runtime evidence. A deliberately
|
||||
failed migration exercises the documented recovery route.
|
||||
- The backup implementation proves completeness, checksum verification,
|
||||
encryption, off-host copy, retention, monitoring, and fail-closed partial
|
||||
artifact cleanup.
|
||||
- A clean isolated host restores a representative database, workspaces,
|
||||
secrets, and compatible release; lifecycle reconciliation and critical tenant
|
||||
journeys pass with outbound integrations controlled.
|
||||
- Scheduled drills publish achieved recovery point and recovery time against the
|
||||
accepted objectives, and alert when no recent successful drill exists.
|
||||
|
||||
## Primary-source interpretation
|
||||
|
||||
The official PostgreSQL, Prisma, and systemd behavior supporting this audit is
|
||||
collected with claim-level links in
|
||||
[Backup and migration primary sources](./backup-migration-primary-sources.md).
|
||||
That note distinguishes what a PostgreSQL-consistent dump proves from the
|
||||
cross-store guarantees the Hub must build itself.
|
||||
@@ -0,0 +1,574 @@
|
||||
# Backup, migration, and disaster-recovery primary-source research
|
||||
|
||||
This note is the primary-source evidence pack for the Wayfinder ticket
|
||||
“Audit backup, migration, and disaster recovery safety.” External claims use
|
||||
only PostgreSQL, Prisma, systemd, and rsync first-party documentation. Project
|
||||
recommendations are explicitly labelled as derived implications rather than
|
||||
upstream guarantees.
|
||||
|
||||
PostgreSQL links below use `docs/current`, which was PostgreSQL 18 when this
|
||||
research was performed on 2026-07-10. A production implementation must pin the
|
||||
deployed PostgreSQL major and verify the same pages for that major. The Hub
|
||||
currently depends on Prisma `^6.19.3`, so Prisma 6-specific command examples
|
||||
remain relevant even where the live documentation also describes Prisma 7.
|
||||
|
||||
## Executive conclusions
|
||||
|
||||
1. `pg_dump` can take a consistent **single-database** snapshot while ordinary
|
||||
reads and writes continue, but it does not include the Hub's workspace tree,
|
||||
cluster-global roles/tablespaces, or a WAL chain. PostgreSQL now explicitly
|
||||
says it is generally not the right regular production-backup mechanism
|
||||
outside simple cases. It is a useful logical/export layer, not proof of
|
||||
whole-service disaster recovery.
|
||||
([`pg_dump` description](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION),
|
||||
[continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING))
|
||||
2. Database PITR requires a physical base backup plus a continuous, unbroken
|
||||
WAL sequence from the backup start through the recovery target. A successful
|
||||
base-backup command alone does not prove that chain exists.
|
||||
([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING),
|
||||
[making a base backup](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-BASE-BACKUP))
|
||||
3. `pg_verifybackup` is necessary but not sufficient: PostgreSQL explicitly
|
||||
requires test restores followed by checks that the restored database works
|
||||
and contains the expected data.
|
||||
([`pg_verifybackup`](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP))
|
||||
4. Prisma `migrate deploy` applies pending migrations, but it does not detect
|
||||
schema drift and, on PostgreSQL, Prisma does not wrap a migration in a
|
||||
transaction by default. A failed migration can therefore leave partial
|
||||
schema changes requiring explicit operator recovery.
|
||||
([Prisma production workflow](https://www.prisma.io/docs/orm/prisma-migrate/workflows/development-and-production#production-and-testing-environments),
|
||||
[Prisma transactional behaviour](https://www.prisma.io/blog/prisma-migrate-dx-primitives#what-if-schema-migrations-were-atomic),
|
||||
[failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration))
|
||||
5. systemd supplies process lifecycle orchestration, not an application-level
|
||||
consistency barrier. It sends the configured termination signal, waits up
|
||||
to `TimeoutStopSec=`, then can forcibly kill remaining processes; a
|
||||
successful `systemctl stop` does not prove that an `ExecStop=` command or
|
||||
application drain succeeded.
|
||||
([`TimeoutStopSec=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutStopSec=),
|
||||
[`systemctl stop`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#stop%20PATTERN%E2%80%A6))
|
||||
6. The Hub persists one logical service state across two stores: PostgreSQL
|
||||
records `Project.workspaceDir`, runs, messages, locks, and file-change audit
|
||||
rows, while workspaces contain project files. A workspace transcript helper
|
||||
exists but is not wired into the production runner. PostgreSQL's snapshot/
|
||||
PITR guarantees cover only PostgreSQL. A whole-service recovery point
|
||||
therefore requires an application-defined database/workspace checkpoint or
|
||||
a journaled/versioned reconciliation protocol; neither PostgreSQL nor
|
||||
systemd creates one automatically.
|
||||
([Hub schema](../../../hub/prisma/schema.prisma),
|
||||
[workspace transcript helper](../../../hub/src/agent/transcript.ts),
|
||||
[`pg_dump` scope](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION))
|
||||
|
||||
## Current repository facts that make the boundaries material
|
||||
|
||||
- The systemd unit runs `prisma migrate deploy` as `ExecStartPre=` before every
|
||||
service start and uses `Restart=on-failure`.
|
||||
([`cph-hub.service`](../../../hub/deploy/cph-hub.service))
|
||||
- The deploy script updates the live Hub directory using
|
||||
`rsync -az --delete`, builds in that same directory, restarts the unit, and
|
||||
accepts a constant liveness endpoint as deployment success.
|
||||
([`deploy_platform.sh`](../../../hub/deploy/deploy_platform.sh))
|
||||
- `Project.workspaceDir` is database state, but the referenced directory holds
|
||||
independently mutable tenant content. A workspace JSONL transcript helper
|
||||
exists but is not wired into the production runner; file contents still live
|
||||
in the workspace while the database separately carries run/message/file-change
|
||||
facts.
|
||||
([Hub schema](../../../hub/prisma/schema.prisma),
|
||||
[workspace transcript helper](../../../hub/src/agent/transcript.ts))
|
||||
|
||||
These are project facts, not upstream guarantees. The sections below establish
|
||||
what the upstream tools do and, critically, what they do not do.
|
||||
|
||||
## PostgreSQL logical dump and restore
|
||||
|
||||
### Snapshot and concurrent-write semantics
|
||||
|
||||
- `pg_dump` makes a consistent export even while the database is being used;
|
||||
it does not block ordinary readers or writers. It dumps only one database.
|
||||
Cluster-global objects such as roles and tablespaces require `pg_dumpall` or
|
||||
`pg_dumpall --globals-only`.
|
||||
([`pg_dump` description](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION),
|
||||
[whole-cluster logical dumps](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-ALL))
|
||||
- `pg_dumpall` invokes `pg_dump` once per database. Each database is internally
|
||||
consistent, but the snapshots are not synchronized across databases.
|
||||
([whole-cluster logical dumps](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-ALL))
|
||||
- A parallel dump uses directory format, opens `njobs + 1` connections, and
|
||||
uses synchronized snapshots so every worker sees the same data set. The
|
||||
connection count and additional server load are part of its capacity cost.
|
||||
([`pg_dump --jobs`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
|
||||
- Concurrent DDL can still interfere. The parallel-dump leader holds
|
||||
`ACCESS SHARE` locks; if a later exclusive-lock request queues ahead of a
|
||||
worker, that worker uses `NOWAIT` and aborts the dump rather than deadlocking.
|
||||
`--lock-wait-timeout` can bound the initial lock wait.
|
||||
([`pg_dump --jobs` and lock behaviour](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
|
||||
- The ordinary snapshot is suitable for disaster-recovery dump semantics, but
|
||||
it need not correspond to a serial execution of all transactions that later
|
||||
commit. `--serializable-deferrable` waits for an anomaly-free snapshot;
|
||||
PostgreSQL says that option is not beneficial for a dump intended only for
|
||||
disaster recovery, but it can matter for a continuously updated reporting
|
||||
copy.
|
||||
([`--serializable-deferrable`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
|
||||
- The default waits for dump files to be written safely to disk. Production
|
||||
backups must not use `--no-sync`, because an immediate OS crash can otherwise
|
||||
leave a corrupt dump.
|
||||
([`pg_dump --no-sync`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- An online database dump may be internally valid while referring to a
|
||||
`workspaceDir` whose files are from another instant. “`pg_dump` succeeded” is
|
||||
therefore not a whole-Hub consistency claim.
|
||||
- Backup scheduling must either serialize against schema migration/DDL or
|
||||
treat any lock wait, abort, non-zero exit, and stderr warning as a failed
|
||||
backup. A partially written archive must never be published as restorable.
|
||||
- If the deployed cluster contains more than the Hub database, a single Hub
|
||||
dump is not a cluster backup. Even `pg_dumpall` does not create a single
|
||||
cross-database snapshot.
|
||||
|
||||
### Restore error and atomicity semantics
|
||||
|
||||
- `pg_restore` defaults to continuing after SQL errors and reports an error
|
||||
count at the end. `--exit-on-error` is required for fail-fast behaviour.
|
||||
`--single-transaction` wraps all emitted commands in `BEGIN`/`COMMIT`, makes
|
||||
the restore all-or-nothing, and implies `--exit-on-error`.
|
||||
([`pg_restore` error and transaction options](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS))
|
||||
- Parallel restore uses separate connections for data loading, index creation,
|
||||
and constraint creation. It supports only custom/directory archives and
|
||||
cannot be combined with `--single-transaction`; it is not a whole-restore
|
||||
atomic operation.
|
||||
([`pg_restore --jobs`](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS))
|
||||
- A single giant restore transaction can be impractical because it locks every
|
||||
restored object and may exhaust the lock table. PostgreSQL 18 offers
|
||||
`--transaction-size=N` as a bounded intermediate option; it implies
|
||||
`--exit-on-error` but is not all-or-nothing.
|
||||
([`pg_restore --transaction-size`](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS))
|
||||
- Plain SQL restored by `psql` also continues after errors unless
|
||||
`ON_ERROR_STOP` is enabled; a single transaction is separately required for
|
||||
all-or-nothing behaviour.
|
||||
([restoring an SQL dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE))
|
||||
- PostgreSQL recommends restoring into a truly empty database based on
|
||||
`template0`; otherwise local additions in `template1` can cause duplicate
|
||||
definition errors. Required roles must exist before owner/ACL restoration.
|
||||
([restoring an SQL dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE))
|
||||
- A dump can execute arbitrary code selected by superusers of the source
|
||||
database. Partial dump/restore does not remove that trust boundary.
|
||||
([`pg_dump` security warning](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION))
|
||||
- A `pg_dump` binary can dump older servers but refuses a server newer than its
|
||||
own major version. Output is intended to load into newer PostgreSQL versions,
|
||||
but loading into an older major is not guaranteed.
|
||||
([`pg_dump` cross-version notes](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-NOTES))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- Restore should target a new isolated database/cluster. If a non-atomic or
|
||||
parallel restore reports any error, discard that target and start again;
|
||||
never promote a partially restored database.
|
||||
- The runbook must record the restore atomicity strategy and exact PostgreSQL
|
||||
client/server versions. A green process exit without explicit fail-fast
|
||||
options is insufficient evidence.
|
||||
- A restored database must remain disconnected from real Feishu/provider
|
||||
side-effect endpoints until application-level validation completes.
|
||||
|
||||
## Physical backup, WAL, and PITR
|
||||
|
||||
### Base backups
|
||||
|
||||
- `pg_basebackup` backs up a running PostgreSQL cluster without blocking other
|
||||
clients. A full backup is an exact copy of the entire cluster and can seed
|
||||
PITR or a standby; it cannot select one database or object.
|
||||
([`pg_basebackup` description](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
- It uses the replication protocol. The account needs `REPLICATION` or
|
||||
superuser privilege, `pg_hba.conf` must permit it, and `max_wal_senders`
|
||||
needs capacity for the backup and, when streaming WAL, another connection.
|
||||
([`pg_basebackup` requirements](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
- Default `-X stream` obtains WAL over a second connection while copying. With
|
||||
`-X fetch`, WAL is collected only at the end; if required WAL was recycled,
|
||||
the backup fails and is unusable.
|
||||
([`pg_basebackup --wal-method`](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
- The default waits for backup files to be durable. `--no-sync` is explicitly
|
||||
unsuitable for production because an OS crash can corrupt the backup.
|
||||
([`pg_basebackup --no-sync`](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
- A manifest is generated by default, with per-file checksums. A SHA manifest
|
||||
can detect malicious modification only if the manifest itself is held in a
|
||||
separate trusted location or otherwise authenticated.
|
||||
([`pg_basebackup` manifest options](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP))
|
||||
|
||||
Derived minimum backup record:
|
||||
|
||||
- source cluster/system identifier and PostgreSQL version;
|
||||
- backup tool version, start/end time, start/end LSN and WAL range;
|
||||
- manifest digest and its independently protected location;
|
||||
- destination object/version, byte count, exit status, and verification state;
|
||||
- an explicit `VALID`, `INVALID`, or `QUARANTINED` state so a failed leftover
|
||||
directory cannot enter the retention set merely because files exist.
|
||||
|
||||
### WAL archival and recovery
|
||||
|
||||
- PITR combines a physical backup with WAL replay. It can stop at a chosen time
|
||||
and reach a consistent database state. Logical dumps do not contain enough
|
||||
information for WAL replay.
|
||||
([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING))
|
||||
- Successful recovery requires a continuous sequence of archived WAL extending
|
||||
back at least to the base-backup start. PostgreSQL tells operators to set up
|
||||
and test WAL archiving **before** taking the first base backup.
|
||||
([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING))
|
||||
- The archive command must return zero if and only if durable archival
|
||||
succeeded. Non-zero causes retry. Archive implementations should refuse to
|
||||
overwrite an existing file; a repeated name may report success only when the
|
||||
existing durable content is identical, and differing content must fail.
|
||||
([WAL archiving](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
- Repeated archive failure accumulates WAL in `pg_wal`; filling that filesystem
|
||||
causes PostgreSQL to panic and remain offline until space is freed. PostgreSQL
|
||||
explicitly advises monitoring archival progress.
|
||||
([WAL archiving failure behaviour](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
- WAL is archived after a segment is completed, so low traffic can make the
|
||||
last committed change wait a long time for archival. `archive_timeout` can
|
||||
bound that age by forcing segment switches, at the cost of archive growth.
|
||||
([WAL archiving latency](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
- WAL archival does not restore manually edited PostgreSQL configuration files
|
||||
(`postgresql.conf`, `pg_hba.conf`, `pg_ident.conf`); those need a separate
|
||||
file backup.
|
||||
([WAL archiving caveat](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
- PostgreSQL's recovery procedure keeps clients out, restores the base backup,
|
||||
configures `restore_command` and `recovery.signal`, replays WAL, and requires
|
||||
inspection of database contents before restoring ordinary access.
|
||||
([PITR recovery procedure](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-PITR-RECOVERY))
|
||||
- Recovery can target a timestamp, LSN, transaction ID, named restore point, or
|
||||
the earliest consistent point. Only one target may be set. Timeline history
|
||||
is required to navigate recovery branches, and a configured target not
|
||||
reached before WAL exhaustion is a fatal recovery failure.
|
||||
([recovery target settings](https://www.postgresql.org/docs/current/runtime-config-wal.html#RUNTIME-CONFIG-WAL-RECOVERY-TARGET),
|
||||
[timelines](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-TIMELINES))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- The database RPO is the age of the newest **durably archived, continuous**
|
||||
WAL position, not the age of the newest base backup and not merely the name
|
||||
in `last_archived_wal`.
|
||||
- Archive keys need a collision-proof cluster/system-identifier namespace and
|
||||
create-if-absent semantics; unrelated clusters must never overwrite a shared
|
||||
WAL name.
|
||||
- Monitoring must cover archive failures, time since durable success, archive
|
||||
lag, WAL continuity, `pg_wal` growth, free space, and base-backup age.
|
||||
- PITR restores PostgreSQL only. It does not rewind workspace files. A database
|
||||
recovery target has no whole-Hub meaning until paired with a workspace
|
||||
recovery point and reconciliation rule.
|
||||
|
||||
## Verification is a ladder, not a checksum
|
||||
|
||||
- `pg_verifybackup` validates the backup manifest and system identifier,
|
||||
detects missing/extra/changed files, checks file checksums, and parses the
|
||||
required WAL with `pg_waldump`. WAL verification is currently supported only
|
||||
for plain-format base backups.
|
||||
([`pg_verifybackup` checks](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP))
|
||||
- PostgreSQL explicitly says these checks cannot cover everything a running
|
||||
server will encounter and still requires test restores plus checks that the
|
||||
database works and contains correct data.
|
||||
([`pg_verifybackup` limitation](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP))
|
||||
- `pg_amcheck` can check supported heap/index structures, but currently only
|
||||
ordinary/TOAST tables, materialized views, sequences, and B-tree indexes are
|
||||
supported; other relation types are silently skipped. Stronger B-tree checks
|
||||
can block concurrent modifications.
|
||||
([`pg_amcheck`](https://www.postgresql.org/docs/current/app-pgamcheck.html#APP-PGAMCHECK))
|
||||
|
||||
Derived recovery-evidence ladder:
|
||||
|
||||
1. **Artifact verification:** backup command succeeded; manifest and all
|
||||
required WAL validate; any failure marks the artifact unusable.
|
||||
2. **Isolated technical restore:** restore to a clean non-production target,
|
||||
reach the intended LSN/time/timeline, and prove PostgreSQL starts without
|
||||
exposing it to users.
|
||||
3. **Database validation:** verify migration history/schema identity, tenant
|
||||
counts and relationships, critical constraints/sequences, and appropriate
|
||||
corruption checks.
|
||||
4. **Workspace validation:** restore the paired snapshot/version, validate file
|
||||
hashes/ownership/containment, and reconcile database file-change/run state
|
||||
against actual files.
|
||||
5. **Application validation:** run read-only tenant-isolation and core workflow
|
||||
smoke tests with external side effects disabled.
|
||||
6. **Recovery report:** persist the exact backup, WAL range, workspace point,
|
||||
release/schema versions, checks, observed RPO/RTO, operator, and failures.
|
||||
Only a fully passing report may authorize promotion.
|
||||
|
||||
## Prisma Migrate production semantics
|
||||
|
||||
### What `migrate deploy` proves and does not prove
|
||||
|
||||
- In production, `migrate deploy` compares applied migration checksums with the
|
||||
migration directory, warns when an applied migration was modified, and
|
||||
applies pending migrations.
|
||||
([Prisma 6 production workflow](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#production-and-testing-environments))
|
||||
- It does **not** warn when an already-applied migration is missing, detect
|
||||
production schema drift, reset the database, generate Prisma Client, or use a
|
||||
shadow database.
|
||||
([Prisma 6 production workflow](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#production-and-testing-environments))
|
||||
- The migration directory is the deployable history and must be committed in
|
||||
full, including `migration_lock.toml`. Prisma warns against editing or
|
||||
deleting migrations already applied in production; `migrate deploy` executes
|
||||
migration SQL rather than deriving a new migration from the Prisma schema.
|
||||
([Prisma migration histories](https://www.prisma.io/docs/orm/prisma-migrate/understanding-prisma-migrate/migration-histories))
|
||||
- Prisma uses a database advisory lock for production migration commands so
|
||||
multiple migration commands do not run simultaneously. The documented lock
|
||||
timeout is ten seconds and not configurable; after timeout, the caller must
|
||||
retry. The lock can be disabled by environment variable.
|
||||
([Prisma 6 advisory locking](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#advisory-locking))
|
||||
|
||||
Derived Hub implication: `migrate deploy` exit zero proves that this invocation
|
||||
applied or found no pending migration. It does not prove the live schema equals
|
||||
the expected migration end state, that the new and previous application builds
|
||||
are both compatible with it, or that the migration is operationally safe under
|
||||
production data volume and traffic.
|
||||
|
||||
The advisory lock is a migration-command concurrency guard, not a substitute
|
||||
for the repository's deployment lock or application/schema compatibility. As a
|
||||
derived safety rule, production preflight should reject
|
||||
`PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK` unless an explicit reviewed mechanism
|
||||
provides the same serialization guarantee.
|
||||
|
||||
### Transactions and failed migrations
|
||||
|
||||
- Prisma's first-party migration guidance states that PostgreSQL migrations
|
||||
are not wrapped in a transaction by default. Operators may opt in by adding
|
||||
`BEGIN;`/`COMMIT;` to a migration, but large transactional migrations can hold
|
||||
locks longer and consume more database resources.
|
||||
([Prisma transactional behaviour](https://www.prisma.io/blog/prisma-migrate-dx-primitives#what-if-schema-migrations-were-atomic))
|
||||
- A failed migration records its error in `_prisma_migrations.logs`; further
|
||||
migrations cannot deploy until the failed state is explicitly recovered.
|
||||
Recovery is either (a) revert/repair partial steps, mark the migration rolled
|
||||
back, and redeploy, or (b) manually complete the exact steps and mark it
|
||||
applied.
|
||||
([Prisma failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration))
|
||||
- Prisma error codes `P3009` and `P3018` explicitly report that failed
|
||||
migrations block new migration application until recovery.
|
||||
([Prisma error reference](https://www.prisma.io/docs/orm/reference/error-reference#p3009))
|
||||
- `migrate resolve --rolled-back` changes migration-history state so Prisma may
|
||||
apply the migration again; it does not itself undo SQL already executed.
|
||||
`--applied` records success without executing the migration SQL.
|
||||
([failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration),
|
||||
[`migrate resolve`](https://www.prisma.io/docs/cli/migrate/resolve))
|
||||
- Prisma can generate a down SQL script, but its documentation limits the
|
||||
simple rollback workflow to a **failed** up migration. To reverse a successful
|
||||
migration, the supported history is a new forward migration generated from a
|
||||
reverted Prisma schema. A schema down script also cannot restore lost data,
|
||||
application changes, or manually authored SQL automatically.
|
||||
([generating down migrations](https://www.prisma.io/docs/orm/prisma-migrate/workflows/generating-down-migrations#considerations-when-generating-down-migrations))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- Every migration needs a declared atomicity choice. If it is intentionally
|
||||
non-transactional, the migration must be resumable/idempotent or ship an
|
||||
exact, tested partial-failure procedure. Blindly rerunning a partially applied
|
||||
script is unsafe.
|
||||
- A database backup checkpoint before a destructive or difficult-to-reverse
|
||||
migration is necessary, but restoring that checkpoint is a whole-database
|
||||
recovery event—not a substitute for a compatible rollout design.
|
||||
- Service restart cannot be treated as application rollback after a successful
|
||||
schema migration. The previous binary may no longer understand the new
|
||||
schema, and a down migration cannot recreate deleted data.
|
||||
|
||||
### Compatibility and rollout ordering
|
||||
|
||||
- Prisma documents expand-and-contract as the way to avoid downtime for fields
|
||||
in active use: add the new representation, deploy code that writes both while
|
||||
reading the old, migrate and verify data, switch reads, stop old writes, then
|
||||
remove the old representation.
|
||||
([Prisma expand-and-contract workflow](https://www.prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations#example-use-the-expand-and-contract-pattern-to-evolve-the-schema-without-downtime))
|
||||
- Prisma also warns that generated “renames” may be expressed as add/drop and
|
||||
lose data unless the SQL is reviewed and customized before application.
|
||||
([customizing migrations](https://www.prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations#example-rename-a-field))
|
||||
|
||||
Derived deployment contract:
|
||||
|
||||
1. Migration SQL is immutable after production application and reviewed for
|
||||
destructive operations, lock duration, data volume, and transaction choice.
|
||||
2. It is rehearsed against a recent, anonymized production-scale restore and
|
||||
its runtime/locks/disk growth are recorded.
|
||||
3. Each release declares a schema compatibility window. Expand releases remain
|
||||
compatible with both old and new application versions; contract/destructive
|
||||
releases occur only after old binaries can no longer run.
|
||||
4. Migration status/drift is checked before changing the live release. A
|
||||
failed migration blocks deployment and enters an operator runbook; it must
|
||||
never be hidden by repeated restart attempts.
|
||||
5. Rollback means switching to a binary proven compatible with the current
|
||||
schema or deploying a new forward migration. Restoring a backup is the last
|
||||
resort and must also restore/reconcile the workspace recovery point.
|
||||
|
||||
## systemd stop, restart, and dependency semantics
|
||||
|
||||
- When `ExecStartPre=`, `ExecStart=`, or `ExecStartPost=` fails or times out,
|
||||
service startup does not continue to the normal running process. `ExecStop=`
|
||||
is skipped because startup never succeeded; `ExecStopPost=` is the hook that
|
||||
still runs after a failed startup.
|
||||
([systemd start/stop command semantics](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStartPre=),
|
||||
[`ExecStopPost=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStopPost=),
|
||||
[permanent upstream man-page source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemd.service.xml#L525-L587))
|
||||
- `Restart=on-failure` covers non-zero exits and failures/timeouts in service
|
||||
control processes, including `ExecStartPre=`. An explicit systemd stop does
|
||||
not trigger automatic restart. Restarts are subject to unit start-rate
|
||||
limiting.
|
||||
([`Restart=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Restart=),
|
||||
[permanent upstream source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemd.service.xml#L856-L995))
|
||||
- With no `ExecStop=`, systemd sends `KillSignal=` (default `SIGTERM`).
|
||||
`TimeoutStopSec=` bounds the wait; after it expires, remaining processes can
|
||||
be forcibly killed with `SIGKILL`/`FinalKillSignal=`.
|
||||
([`ExecStop=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStop=),
|
||||
[`TimeoutStopSec=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutStopSec=))
|
||||
- Default `KillMode=control-group` applies termination to all remaining
|
||||
processes in the unit cgroup. systemd explicitly discourages `process` or
|
||||
`none` because children can escape the service lifecycle and remain running
|
||||
after the service is considered stopped.
|
||||
([`KillMode=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html#KillMode=))
|
||||
- `systemctl stop` waits for the job unless `--no-block` is used, but the command
|
||||
does not fail merely because an `ExecStop=` command failed: the manager still
|
||||
forcibly terminates the unit.
|
||||
([`systemctl stop`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#stop%20PATTERN%E2%80%A6),
|
||||
[permanent upstream source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemctl.xml#L444-L459))
|
||||
- `systemctl restart` is a stop followed by a start; `ExecStop=` and
|
||||
`ExecStopPost=` participate. It does not necessarily flush every unit resource
|
||||
in the same way as separate completed `stop` then `start` operations.
|
||||
([restart semantics](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStop=),
|
||||
[`systemctl restart`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#restart%20PATTERN%E2%80%A6))
|
||||
- `After=` is only an ordering relationship; it does not pull in another unit.
|
||||
Requirement and ordering dependencies are independent. During shutdown the
|
||||
start order is reversed, so a Hub ordered `After=postgresql.service` stops
|
||||
before PostgreSQL when both are in the shutdown transaction.
|
||||
([`Requires=` and ordering](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Requires=),
|
||||
[`Before=`/`After=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Before=))
|
||||
- `systemctl freeze` only uses the cgroup freezer to suspend process execution;
|
||||
its documented contract contains no filesystem flush or stable-image
|
||||
guarantee. It is not a substitute for an application drain or a filesystem
|
||||
snapshot barrier.
|
||||
([permanent `systemctl freeze` man-page source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemctl.xml#L653-L675))
|
||||
|
||||
Derived assessment of the current unit:
|
||||
|
||||
- A Prisma migration failure in the current `ExecStartPre=` prevents the Hub
|
||||
from starting, then `Restart=on-failure` can retry startup until rate-limited.
|
||||
That is fail-closed, but it is not controlled migration recovery and can
|
||||
repeatedly hit a database already left in Prisma's failed state.
|
||||
- `KillSignal=SIGTERM` plus `TimeoutStopSec=30` is only a deadline. Until the Hub
|
||||
installs a real termination handler that makes readiness false, stops ingress,
|
||||
drains/checkpoints active work, closes Feishu and database clients, and reports
|
||||
success, the unit cannot claim a graceful consistency barrier.
|
||||
- A backup runbook must issue a controlled drain, verify durable work reached a
|
||||
reconciled terminal/checkpoint state, stop the Hub, verify the unit and its
|
||||
cgroup are inactive, and only then snapshot mutable workspace state. A zero
|
||||
`systemctl stop` exit by itself is insufficient evidence.
|
||||
- Recovery should not start the normal unit until the restored database and
|
||||
workspace have been validated. The current unit would run pending migrations
|
||||
immediately on first start, mutating the restored database before validation
|
||||
and possibly destroying the ability to inspect the original recovered state.
|
||||
|
||||
## Filesystem and rsync boundaries
|
||||
|
||||
- PostgreSQL says a plain filesystem copy of a live data directory is unusable:
|
||||
the database must be shut down, because tools such as `tar` do not create an
|
||||
atomic snapshot and server buffering remains. A correctly implemented frozen
|
||||
snapshot can work while PostgreSQL runs, but all volumes containing data,
|
||||
WAL, and tablespaces must be captured simultaneously; otherwise shut down or
|
||||
use the continuous-archive base-backup protocol.
|
||||
([PostgreSQL filesystem backups](https://www.postgresql.org/docs/current/backup-file.html))
|
||||
- For a PostgreSQL rsync backup, the documented safe pattern is a first pass
|
||||
while running, followed by shutdown and a second `rsync --checksum`; the
|
||||
second pass is what makes the result consistent.
|
||||
([PostgreSQL filesystem backups](https://www.postgresql.org/docs/current/backup-file.html))
|
||||
- At the filesystem layer, `fsfreeze` is documented to let in-progress
|
||||
filesystem operations complete, halt new modifications, and flush dirty data,
|
||||
metadata, and logs before returning. That is still only a filesystem-volume
|
||||
guarantee; it does not replace PostgreSQL's backup protocol and does not make
|
||||
separately frozen volumes simultaneous.
|
||||
([util-linux `fsfreeze` manual source](https://github.com/util-linux/util-linux/blob/master/sys-utils/fsfreeze.8.adoc))
|
||||
- rsync's `--delete` defaults to deletion during transfer on modern peers, so
|
||||
the receiver tree changes incrementally. `--delay-updates` merely holds each
|
||||
updated file and renames files into place rapidly at the end; its own manual
|
||||
says this only attempts to make updates “a little more atomic” and points to
|
||||
a separate parallel-tree/switch approach for stronger atomicity.
|
||||
([rsync `--delete`](https://rsync.samba.org/ftp/rsync/rsync.1.html#opt--delete),
|
||||
[rsync `--delay-updates`](https://rsync.samba.org/ftp/rsync/rsync.1.html#opt--delay-updates))
|
||||
|
||||
Derived Hub implications:
|
||||
|
||||
- The current in-place `rsync --delete` deployment is neither an immutable
|
||||
release switch nor a backup snapshot. A failed/interrupted transfer can leave
|
||||
a mixed live tree; service stop/restart cannot reconstruct the deleted prior
|
||||
artifact.
|
||||
- The workspace snapshot mechanism remains an explicit design choice. Whichever
|
||||
storage technology is selected must prove atomicity over all workspace
|
||||
volumes or use a quiesced/no-writer interval, then publish an immutable
|
||||
version/checksum manifest. Merely copying a changing directory is not enough.
|
||||
|
||||
## Derived minimum whole-Hub recovery contract
|
||||
|
||||
The primary sources support the following minimum contract; exact RPO, RTO,
|
||||
retention, storage vendor, and numeric schedules remain product/operations
|
||||
decisions and must not be invented by implementation.
|
||||
|
||||
### Backup path
|
||||
|
||||
1. Maintain verified PostgreSQL base backups and a monitored continuous WAL
|
||||
archive, or obtain and test an explicitly equivalent managed-PostgreSQL
|
||||
guarantee. Keep logical dumps as a separate portability/selective-restore
|
||||
layer, not the sole PITR mechanism.
|
||||
2. Keep PostgreSQL configuration, deployment/release manifests, migration
|
||||
history, and workspace snapshots in the backup inventory; WAL alone does not
|
||||
cover them.
|
||||
3. Create a whole-Hub checkpoint only after ingress is drained and active
|
||||
workspace writers are stopped/reconciled, unless the eventual storage layer
|
||||
supplies a proven atomic snapshot/journal spanning the necessary stores.
|
||||
4. Persist one signed/immutable checkpoint manifest joining PostgreSQL
|
||||
backup/WAL position, workspace snapshot/version, application revision,
|
||||
migration digest, tool versions, checksums, timestamps, and validation state.
|
||||
5. Treat database/WAL/workspace backups as cross-tenant sensitive data. Apply
|
||||
least-privilege access, encryption, audit, deletion/retention, and an
|
||||
independently protected integrity manifest; WAL contains effectively all
|
||||
database changes and PostgreSQL advises protecting it from unauthorized
|
||||
readers.
|
||||
([WAL archive confidentiality](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL))
|
||||
|
||||
### Migration path
|
||||
|
||||
1. Serialize deploys and backup checkpoints; run a drift/status gate before
|
||||
migration, not just `migrate deploy`.
|
||||
2. Rehearse each migration on a recent production-scale restore and record
|
||||
runtime, locks, disk/WAL growth, transaction choice, and failure recovery.
|
||||
3. Require expand/contract compatibility for online changes. A release must
|
||||
declare which old/new binaries can run against each schema phase.
|
||||
4. Separate migration orchestration from ordinary service restart so a failed
|
||||
migration produces one durable incident state and an operator decision—not
|
||||
an opaque restart loop.
|
||||
5. Verify a restorable checkpoint before destructive change. Do not describe
|
||||
backup restore as a normal “rollback”; it is whole-service disaster recovery.
|
||||
|
||||
### Restore and promotion path
|
||||
|
||||
1. Restore into an isolated environment with real external side effects
|
||||
disabled.
|
||||
2. Restore the selected PostgreSQL base backup/WAL target and the checkpoint's
|
||||
exact workspace version; do not run pending Prisma migrations implicitly.
|
||||
3. Prove artifact integrity, database startup, recovery target/timeline,
|
||||
migration/schema identity, tenant invariants, workspace hashes/containment,
|
||||
and read-only application flows.
|
||||
4. Record observed recovery point and recovery time. A restore that starts but
|
||||
misses its target, loses a tenant workspace, has partial migration state, or
|
||||
fails any invariant is not promotable.
|
||||
5. Promote explicitly, then enable ingress and external integrations. Preserve
|
||||
the pre-recovery cluster/WAL and recovery report until the incident is closed.
|
||||
|
||||
## Decisions and evidence still required
|
||||
|
||||
These values are not fixed by the upstream tools and remain `OPEN` for the SaaS
|
||||
production contract:
|
||||
|
||||
- whole-Hub RPO and RTO, separately for ordinary deletion, database corruption,
|
||||
host loss, region/provider loss, and migration failure;
|
||||
- base-backup, logical-dump, workspace-snapshot, WAL, and restore-drill cadence;
|
||||
- immutable/off-host storage, encryption/key ownership, retention and verified
|
||||
deletion policy, including org-deletion and legal-retention interactions;
|
||||
- exact workspace checkpoint/journal technology and database/workspace
|
||||
reconciliation semantics for in-flight or partially written runs;
|
||||
- whether restore promotion is whole-platform only or supports a safe
|
||||
organization/project export-import path (PostgreSQL physical PITR is
|
||||
cluster-wide);
|
||||
- migration transaction policy, maximum lock/downtime budget, compatibility
|
||||
window, and who can authorize contract/destructive phases;
|
||||
- evidence from at least one production-like full restore and one PITR drill,
|
||||
including deliberate missing-WAL, corrupted-backup, failed-migration,
|
||||
interrupted-workspace-write, and restart-during-recovery probes.
|
||||
@@ -1,7 +1,7 @@
|
||||
# Audit backup, migration, and disaster recovery safety
|
||||
|
||||
Type: research
|
||||
Status: open
|
||||
Status: resolved
|
||||
|
||||
## Question
|
||||
|
||||
@@ -9,3 +9,40 @@ What backup, restore, workspace/database consistency, migration compatibility,
|
||||
and disaster-recovery guarantees are needed to deploy current persistent data
|
||||
safely, and what evidence or tooling is missing to prove them?
|
||||
|
||||
## Answer
|
||||
|
||||
The Hub has no production backup/restore control plane. PostgreSQL can be
|
||||
logically dumped and restored, but authoritative state is split across the
|
||||
database, mutable Project workspaces, credentials/key material, and a
|
||||
schema-compatible release with no shared checkpoint or recovery manifest.
|
||||
RPO/RTO, retention, off-host/immutable storage, encryption/key custody,
|
||||
full-service versus Organization-selective restore, monitoring, and recovery
|
||||
drills are all undefined.
|
||||
|
||||
Executed probes proved that the current deploy `rsync --delete` removes the
|
||||
documented in-tree workspace root, a failed Project transaction leaves an
|
||||
orphan workspace, and an older application query fails after a destructive
|
||||
migration. A deliberately failed Prisma migration rolled back that probe's DDL
|
||||
but left a blocking failed migration record; the official contract does not
|
||||
make PostgreSQL migrations transactional by default. A positive custom-format
|
||||
round trip restored 24 tables and all nine current migration records, proving
|
||||
only database-level portability on the local PostgreSQL 14.20 setup.
|
||||
|
||||
The complete state map, probe commands/results, root causes, required recovery
|
||||
contract, and release evidence are in the
|
||||
[backup, migration, and disaster-recovery audit](../assets/backup-migration-disaster-recovery-audit.md).
|
||||
Official PostgreSQL, Prisma, systemd, and rsync semantics with claim-level links
|
||||
are in [backup and migration primary sources](../assets/backup-migration-primary-sources.md).
|
||||
|
||||
The newly-clear decision and implementation frontier is:
|
||||
|
||||
- [Decide recovery objectives, backup retention, and restore scope](39-decide-recovery-objectives.md)
|
||||
- [Implement coherent backup sets and verified restore](40-verified-backup-restore.md)
|
||||
- [Make the Project workspace lifecycle recoverable](41-recoverable-project-workspace-lifecycle.md)
|
||||
- [Establish migration compatibility and failed-migration recovery](14-migration-compatibility-and-recovery.md)
|
||||
|
||||
Persistent-path provisioning and release-tree isolation remain owned by
|
||||
[Provision a runnable non-root Hub service](10-provision-nonroot-service.md)
|
||||
and [Make application releases atomic and rollbackable](12-atomic-releases-and-rollback.md).
|
||||
Full recovery/cutover procedures and drill evidence remain owned by
|
||||
[Write and drill production incident recovery runbooks](31-write-and-drill-incident-runbooks.md).
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Type: task
|
||||
Status: open
|
||||
Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38
|
||||
Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41
|
||||
|
||||
## Question
|
||||
|
||||
|
||||
@@ -8,4 +8,6 @@ Status: open
|
||||
Make clean-host installation create or validate the dedicated service identity,
|
||||
home/cache/state/workspace paths and ownership, generate a complete production
|
||||
configuration, honor the configured bind host, and fail preflight before unit
|
||||
installation when any required value or prerequisite is invalid.
|
||||
installation when any required value or prerequisite is invalid. The persistent
|
||||
workspace root must be outside all release trees, and preflight must reject any
|
||||
path that overlaps rsync, release switching, rollback, or release pruning.
|
||||
|
||||
@@ -9,4 +9,5 @@ Blocked by: 10, 11
|
||||
Replace in-place rsync/build with deployment locking, immutable staged release
|
||||
directories, a revision manifest, atomic current/previous switching, bounded
|
||||
retention, and an explicit application rollback that preserves the live release
|
||||
on build, preflight, switch, or readiness failure.
|
||||
on build, preflight, switch, or readiness failure. Prove deploy, rollback, and
|
||||
release pruning cannot traverse or delete the persistent workspace root.
|
||||
|
||||
+5
-3
@@ -8,6 +8,8 @@ Blocked by: 06
|
||||
|
||||
Define and enforce the application/schema compatibility rule for each release,
|
||||
require expand-and-contract sequencing for destructive changes, add migration
|
||||
status and production-like rehearsal gates, and provide an operator procedure
|
||||
for recovering a partially failed Prisma migration without pretending that an
|
||||
application rollback reverses the database.
|
||||
status/checksum and production-like rehearsal gates, and move schema mutation
|
||||
out of ordinary service start into an explicit release phase. Require a recent
|
||||
verified recovery checkpoint, bounded lock/runtime evidence, and an operator
|
||||
procedure for resolving or restoring a failed Prisma migration without
|
||||
pretending that an application rollback reverses the database.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Type: task
|
||||
Status: open
|
||||
Blocked by: 06, 12, 13, 14, 23, 24, 25, 26, 27, 28, 29, 30
|
||||
Blocked by: 06, 12, 13, 14, 23, 24, 25, 26, 27, 28, 29, 30, 39, 40, 41
|
||||
|
||||
## Question
|
||||
|
||||
@@ -13,4 +13,6 @@ deploy/rollback, host restart during work, and suspected tenant exposure. Give
|
||||
each alert an owner and safe read-first commands, require confirmed/audited
|
||||
scope for destructive or replay actions, execute every runbook against
|
||||
production-shaped staging, and archive evidence that detection, mitigation,
|
||||
reconciliation, verification, escalation, and alert clearing all work.
|
||||
reconciliation, verification, escalation, and alert clearing all work. Include
|
||||
database/workspace backup failure, failed migration, full-host restore and
|
||||
cutover, duplicate/gap review, fallback, and achieved RPO/RTO evidence.
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Decide recovery objectives, backup retention, and restore scope
|
||||
|
||||
Type: grilling
|
||||
Status: open
|
||||
|
||||
## Question
|
||||
|
||||
For the initial SaaS service, decide measurable RPO/RTO targets, backup and
|
||||
restore-drill frequency, retention and deletion-in-backups policy, off-host
|
||||
copy/failure-domain requirements, logical export versus physical base-backup
|
||||
and continuous-WAL/PITR coverage, encryption and key custody, operator ownership,
|
||||
and whether the supported promise is full-service disaster recovery only or
|
||||
also Organization-selective export/restore. Record compliance-dependent items
|
||||
as explicit OPEN inputs rather than inventing defaults.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Implement coherent backup sets and verified restore
|
||||
|
||||
Type: task
|
||||
Status: open
|
||||
Blocked by: 10, 12, 14, 19, 23, 24, 25, 26, 29, 39, 41
|
||||
|
||||
## Question
|
||||
|
||||
Implement the decided recovery policy as cataloged, checksummed, encrypted,
|
||||
off-host backup sets covering PostgreSQL logical exports plus physical base
|
||||
backup/continuous WAL or a proven managed equivalent, Project workspaces,
|
||||
compatible release/migration identity, configuration inventory, org-secret
|
||||
ciphertext, and separately recoverable key material. Establish a proven
|
||||
cross-store checkpoint, retention and alerting, partial-artifact cleanup,
|
||||
isolated restore with outbound integrations disabled, lifecycle reconciliation,
|
||||
critical SaaS journey verification, and measured RPO/RTO drills.
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
# Make the Project workspace lifecycle recoverable
|
||||
|
||||
Type: task
|
||||
Status: open
|
||||
Blocked by: 10, 26, 34
|
||||
|
||||
## Question
|
||||
|
||||
Move persistent workspaces outside every mutable release tree and make Project
|
||||
workspace allocation, database commit, archive/deletion, relocation, restore,
|
||||
and orphan/missing-path reconciliation explicit and crash-safe. Prove failed
|
||||
Project creation leaves no silent orphan, deployment/rollback/pruning cannot
|
||||
delete customer files, restored absolute paths are validated or safely rebased,
|
||||
and every inconsistency is observable rather than auto-hidden.
|
||||
@@ -40,6 +40,7 @@ rollback and recovery, and no known critical security or data-integrity gaps.
|
||||
- [Audit run lifecycle and data integrity under failure](issues/03-audit-run-lifecycle-integrity.md) — retain the database-enforced project-lock uniqueness and normal happy path, but replace receipt-before-success, process-local queues, independent run/session/lock effects, silent delivery/audit loss, and partial startup reset with durable, recoverable state machines; explicitly decide failed-run workspace semantics.
|
||||
- [Define production observability and incident recovery](issues/04-define-observability-recovery.md) — keep the existing Fastify/Prisma/systemd signal fragments, but production requires dependency-aware readiness, typed and correlated errors, durable-work telemetry, an explicit SLO/retention/alerting control plane, and recovery runbooks proven by fault drills.
|
||||
- [Define initial abuse and capacity controls](issues/05-define-abuse-capacity-controls.md) — require layered fail-closed limits, durable Organization-fair admission, explicit backpressure, bounded files/storage/runs, soft usage alerts, and audited emergency workload brakes; calibrate numerical ceilings under production-like load.
|
||||
- [Audit backup, migration, and disaster recovery safety](issues/06-audit-backup-migration-recovery.md) — separate persistent state from releases, define a database/workspace checkpoint and recovery objectives, make migrations compatibility-gated, and prove complete off-host backup sets through isolated restore and cutover drills.
|
||||
|
||||
## Fog
|
||||
|
||||
|
||||
Reference in New Issue
Block a user