docs: resolve backup recovery audit

2026-07-10 12:30:19 +08:00
parent 1094ebd651
commit 599a2f6e66
12 changed files with 997 additions and 9 deletions
@@ -0,0 +1,325 @@
+# Backup, migration, and disaster-recovery audit
+
+## Verdict
+
+The current Hub has no production backup or restore control plane. PostgreSQL
+can be dumped and restored manually, but the service's authoritative state is
+split between PostgreSQL and mutable Project workspaces, with no shared
+consistency point, backup catalog, off-host retention, restore workflow,
+recovery objectives, or recovery drill.
+
+The current deployment path can also delete customer workspaces when the
+documented relative workspace root is used: `deploy_platform.sh` applies
+`rsync --delete` to the live Hub directory while `.env.example` places
+workspaces below `./data/project-workspaces`. Project creation has the same
+cross-store weakness at a smaller scale: it creates the workspace before the
+database transaction and leaves an orphan directory when that transaction
+fails.
+
+Migration safety is incomplete even though the current PostgreSQL migration
+engine rolled back a deliberately failing migration's DDL in the executed
+probe. The failed migration record blocks later migrations, migrations execute
+inside every service start, no compatibility declaration or production-data
+rehearsal exists, and current history contains destructive changes that make an
+old application query fail after schema upgrade. Application rollback is
+therefore not database rollback.
+
+## Contract and persistent-state baseline
+
+The audit preserves the accepted contracts rather than inventing missing
+product policy:
+
+- `Organization` is the tenant root, but `User` is global and can belong to
+  multiple Organizations. A selective Organization restore is not equivalent
+  to restoring a self-contained database schema.
+- Project workspaces are authoritative curriculum-engineering files bounded by
+  ADR-0018. PostgreSQL stores their absolute `Project.workspaceDir` locations;
+  it does not store their file content.
+- Run, provider cursor, ingress, delivery, audit, and workspace commit/recovery
+  semantics remain split across the unresolved lifecycle tickets. A backup
+  design cannot silently choose their crash behavior.
+- ADR-0022 requires storage ceilings and emergency `DRAIN`/`STOP_NOW`, but does
+  not decide backup frequency, retention, recovery objectives, or tenant-level
+  restore scope.
+- Audit retention is still `OPEN`; regulatory backup retention and deletion
+  obligations require product/legal input already recorded in the map's Fog.
+
+The recovery set has at least four classes:
+
+| State | Current location | Recovery significance |
+| --- | --- | --- |
+| Tenant and workflow state | PostgreSQL | Organizations, identities, membership, permissions, Feishu bindings, sessions, runs, locks, receipts, usage facts, and audit records |
+| Customer artifacts | `HUB_PROJECT_WORKSPACE_ROOT` | Project source files, generated artifacts, inbound attachments, and any workspace-local session/history files |
+| Credentials and configuration | `/srv/.../.secrets/platform.env` today; future org-scoped encrypted connections plus external key material | Database/Feishu/provider access and ability to decrypt restored Organization credentials |
+| Release identity | application release, Prisma migrations, `cph`, runtime/toolchain manifest | Required to start a schema-compatible application and reproduce outputs after restore |
+
+Caches, build output, and reinstallable dependencies should be classified as
+regenerable rather than silently mixed into the protected recovery set.
+
+## Executed evidence
+
+All probes used isolated temporary paths or databases. No production or shared
+test data was modified.
+
+### 1. The deployment rsync pattern deletes an in-tree workspace
+
+A temporary live Hub directory was given the documented workspace shape and a
+sentinel customer file, then the exact deployment rsync include/exclude pattern
+was applied:
+
+```text
+rsync -a --delete \
+  --exclude node_modules --exclude dist --exclude .env \
+  hub/ <temporary-live-hub>/
+
+workspace_survived=false
+reproduced: deploy rsync pattern deletes an in-tree workspace root
+```
+
+This is deterministic: `data/` is absent from the release source and is not an
+exclude, so `--delete` removes it from the destination. The root cause is that
+the documented persistent path is nested inside the mutable release tree and
+the deploy contract does not distinguish release artifacts from state.
+
+### 2. A rejected Project transaction leaves an orphan workspace
+
+A minimal Node harness called the real built `createProjectFromOrgAdmin` with a
+valid Organization admin and a missing Folder. The fake Prisma boundary rejected
+inside the same transaction seam used by production. The harness asserted that
+the Organization workspace directory must remain empty and was run twice:
+
+```text
+node .scratch/saas-production-readiness/probes/orphan-workspace.mjs
+
+orphan_workspace_entries=project_4aa0b6792c4547059d2d92b505b9299e
+exit_code=1
+
+orphan_workspace_entries=project_190e21afb8854244872b1241a600aa58
+exit_code=1
+```
+
+`createManagedProject` calls `mkdir` before `prisma.$transaction`; the failure
+path has no compensating removal, staged rename, or orphan reconciler. The
+throwaway harness was removed after reproduction.
+
+### 3. PostgreSQL rolled back failed migration DDL but retained a blocking failure
+
+An isolated database and throwaway Prisma project applied one migration that
+created a table, divided by zero, then would have created another table. The
+real repository Prisma CLI returned `P3018`:
+
+```text
+migration_exit=1
+before_failure=missing
+after_failure=missing
+20260710000000_partial_failure|finished=false|rolled_back=false|steps=0|logs=true
+```
+
+This specific PostgreSQL/Prisma 6.19.3 path did **not** leave partial DDL, so the
+audit does not claim that it did. It did leave an unfinished
+`_prisma_migrations` row, and Prisma's error explicitly refused new migrations
+until operator recovery. Because systemd runs `migrate deploy` as
+`ExecStartPre`, the application cannot start past that state. The throwaway
+database and migration were removed. This one multi-statement result is not a
+general atomicity guarantee: Prisma's first-party guidance says PostgreSQL
+migrations are not transaction-wrapped by default, so each migration must make
+its transaction and partial-failure behavior explicit.
+
+### 4. The current schema is not backward-compatible with an older application
+
+The application immediately before the typed-principal migration queried
+`PermissionGrant.principal`. That migration drops the column in favor of
+`principalType` plus `principalId`. Running the old read against the current
+test database produced:
+
+```text
+ERROR: column "principal" does not exist
+HINT: Perhaps you meant to reference the column "PermissionGrant.principalId".
+old_query_exit=1
+```
+
+This is direct evidence that switching the application symlink back after that
+migration cannot restore service. The migration history also deletes duplicate
+grant rows and later drops `ExternalPrincipalMembership.source`; there are no
+downgrade migrations.
+
+### 5. A database-only custom dump restores cleanly
+
+As a positive baseline, `pg_dump --format=custom` captured the current
+`cph_hub_test` database and `pg_restore --exit-on-error --no-owner` restored it
+into a new isolated database:
+
+```text
+restored_tables=24
+finished_migrations=9
+failed_migrations=0
+dump_bytes=76113
+```
+
+All nine finished migration checksums in the source database also matched the
+checked-in migration files. This proves a manual logical database round trip on
+the current PostgreSQL 14.20 workstation. It does not prove workspace,
+credentials, global database roles, production volume, cross-version restore,
+application readiness, or an operational RPO/RTO.
+
+## Critical findings
+
+### 1. No backup, retention, or restore mechanism exists
+
+The repository has no backup/restore command, systemd timer, off-host target,
+encryption/immutability policy, retention policy, backup manifest/checksum,
+success/failure alert, restore command, or scheduled recovery drill. The
+installer assumes PostgreSQL and its role/database already exist. A database
+dump alone does not reproduce those global prerequisites or the filesystem and
+secret state above.
+
+There is also no decided RPO, RTO, backup frequency, retention duration,
+geographic copy rule, deletion-in-backups rule, or ownership/escalation path.
+Those values are product and operations decisions, not safe constants to infer
+from the single-host topology.
+
+### 2. Persistent workspaces are inside the documented release tree
+
+`.env.example` recommends `./data/project-workspaces`; systemd's working
+directory is the live Hub directory; deployment mutates that same directory
+with `rsync --delete`. No preflight forbids an in-release workspace root and no
+exclude preserves it. This can turn a routine deploy into irreversible customer
+data loss before build, migration, restart, or readiness runs.
+
+The persistent workspace root must live outside every release directory, have
+explicit ownership/mode and capacity, and be impossible for release retention
+or `rsync --delete` to traverse. Immutable release work already exists as a
+separate ticket, but its acceptance test must include this sentinel case.
+
+### 3. PostgreSQL and workspaces have no consistency boundary
+
+PostgreSQL can provide a transactionally consistent database snapshot, but the
+Hub mutates workspace files independently and has no generation, journal,
+commit marker, snapshot ID, or shared checkpoint that binds filesystem content
+to the corresponding database state. A database dump and filesystem copy taken
+at different instants can disagree about Project existence, active runs,
+provider cursors, delivered messages, and file effects.
+
+Project creation already demonstrates the same missing compensation boundary.
+A replacement host has an additional problem: `Project.workspaceDir` is an
+absolute path and runtime paths from the database are used directly; restore
+has no same-mount validation, safe rebasing operation, or missing/orphan scan.
+
+For the initial single-host topology, a quiesced backup checkpoint is the
+smallest credible route: stop new admission, drain/cancel according to the
+accepted run contract, reach durable zero in-flight work, capture database and
+workspace artifacts under one manifest, then resume. An online scheme would
+need a newly designed journal/generation protocol; it must not be implied by
+two independent copy commands.
+
+### 4. Schema mutation is coupled to service start
+
+Both `npm start` and the systemd unit execute `prisma migrate deploy`; every
+ordinary restart can therefore become a schema rollout. The release path has no
+separate migration lock/gate, `migrate status` check, production-data rehearsal,
+duration/lock budget, pre-migration recovery checkpoint, or compatibility
+manifest. CI only applies the full migration history to a fresh empty
+PostgreSQL database, so it does not exercise real upgrade volume, legacy values,
+old/new application overlap, failure recovery, or rollback compatibility.
+
+The current failed-migration behavior is fail-closed, which is better than
+starting on an unknown schema, but there is no operator procedure for inspecting
+the failure, deciding roll-forward versus mark-rolled-back, restoring the
+checkpoint, or preventing a restart loop. Existing destructive migrations prove
+that application rollback cannot be assumed safe.
+
+### 5. Restore has no isolated verification or cutover gate
+
+There is no procedure that restores into a non-production environment with
+outbound Feishu/provider traffic disabled, verifies backup checksums and
+migration state, validates workspace paths/content/permissions, checks tenant
+relationships, decrypts a sample org connection, reconciles non-terminal work,
+runs application readiness and representative journeys, and measures recovery
+time/point before controlled cutover.
+
+Starting the current server is not a neutral verifier: startup deletes all
+Project locks and independently marks only `ACTIVE` runs failed. Restored data
+must first pass the durable lifecycle reconciliation designed by the run-state
+tickets; otherwise validation itself mutates the evidence and can leave
+`WAITING_FOR_USER` inconsistent.
+
+### 6. Full-service and Organization-selective recovery are different products
+
+A full-service restore can recreate the shared database and all workspaces.
+Restoring one Organization is much harder: users are global and may belong to
+multiple Organizations, permission/audit/history rows span many related tables,
+workspace paths are external, and Feishu/provider connections have external
+state. The repository has no export/import identity mapping or conflict policy.
+
+The initial production gate must at least prove full-service disaster recovery.
+Whether Organization-selective restore/export is promised, and how retained
+backups interact with customer deletion, remains an explicit product/legal
+decision rather than an accidental property of `pg_restore` filters.
+
+## Required recovery contract
+
+Production readiness requires the following guarantees, with exact numerical
+objectives supplied by the pending policy decision:
+
+1. **Declared objectives and ownership.** RPO/RTO, backup frequency, retention,
+   copy locations, encryption/key custody, immutability, deletion handling,
+   full-service versus Organization restore scope, operator ownership, and
+   escalation are documented and testable.
+2. **Complete backup set.** One cataloged manifest identifies release and
+   migration checksums, PostgreSQL artifact and global prerequisites, workspace
+   snapshot, configuration inventory, encrypted org-secret data, external key
+   recovery material, timestamps, sizes, and cryptographic checksums. Backup
+   data and decryption material are not kept only on the protected host or in
+   the same failure domain.
+3. **One consistency point.** A backup is taken only after an observable
+   quiesce/checkpoint protocol, or by a proven online generation protocol that
+   binds database state and workspace effects. Accepted input, active runs,
+   queues, deliveries, locks, and provider cursors have explicit recovery
+   semantics.
+4. **Safe migration rollout.** Migration is a release phase, not an incidental
+   service start. The gate checks status/checksums, rehearses on representative
+   production-shaped data, records old/new application compatibility, uses
+   expand-and-contract for destructive changes, requires a recent verified
+   recovery checkpoint, and provides a tested failed-migration procedure.
+5. **Isolated restore verification.** Every backup class is periodically
+   restored to a clean isolated host/database with outbound integrations off.
+   Verification checks integrity, schema/migration state, global database
+   prerequisites, workspace completeness/path ownership, secret decryption,
+   tenant isolation, lifecycle reconciliation, application readiness, and
+   representative admin/Feishu/agent journeys.
+6. **Observable operation.** Backup age, duration, artifact/checksum status,
+   off-host replication, retention deletion, WAL/archive continuity if used,
+   restore-drill result, measured RPO/RTO, and failures are logged/metricized
+   and alerted with bounded cardinality.
+7. **Controlled cutover and fallback.** Disaster recovery has an explicit
+   authority, traffic/workload brake, DNS/proxy/Feishu reconnect sequence,
+   duplicate/gap checks, rollback/fallback point, verification checklist, and
+   incident evidence record.
+
+## Required release evidence
+
+- A deploy test proves an in-tree or release-overlapping workspace root is
+  rejected before any rsync/switch, while persistent sentinel files survive
+  build failure, successful release, rollback, and release pruning.
+- Project creation, archive/deletion, failed transaction, crash, and restore
+  tests prove database/workspace allocation has no silent missing or orphan
+  state; the reconciler reports rather than hides inconsistencies.
+- Every checked-in migration is rehearsed from the previous production-shaped
+  snapshot, with old/new compatibility and lock/runtime evidence. A deliberately
+  failed migration exercises the documented recovery route.
+- The backup implementation proves completeness, checksum verification,
+  encryption, off-host copy, retention, monitoring, and fail-closed partial
+  artifact cleanup.
+- A clean isolated host restores a representative database, workspaces,
+  secrets, and compatible release; lifecycle reconciliation and critical tenant
+  journeys pass with outbound integrations controlled.
+- Scheduled drills publish achieved recovery point and recovery time against the
+  accepted objectives, and alert when no recent successful drill exists.
+
+## Primary-source interpretation
+
+The official PostgreSQL, Prisma, and systemd behavior supporting this audit is
+collected with claim-level links in
+[Backup and migration primary sources](./backup-migration-primary-sources.md).
+That note distinguishes what a PostgreSQL-consistent dump proves from the
+cross-store guarantees the Hub must build itself.