curriculum-project-hub/.scratch/saas-production-readiness/assets/backup-migration-disaster-recovery-audit.md

Backup, migration, and disaster-recovery audit

Verdict

The current Hub has no production backup or restore control plane. PostgreSQL can be dumped and restored manually, but the service's authoritative state is split between PostgreSQL and mutable Project workspaces, with no shared consistency point, backup catalog, off-host retention, restore workflow, recovery objectives, or recovery drill.

The current deployment path can also delete customer workspaces when the documented relative workspace root is used: deploy_platform.sh applies rsync --delete to the live Hub directory while .env.example places workspaces below ./data/project-workspaces. Project creation has the same cross-store weakness at a smaller scale: it creates the workspace before the database transaction and leaves an orphan directory when that transaction fails.

Migration safety is incomplete even though the current PostgreSQL migration engine rolled back a deliberately failing migration's DDL in the executed probe. The failed migration record blocks later migrations, migrations execute inside every service start, no compatibility declaration or production-data rehearsal exists, and current history contains destructive changes that make an old application query fail after schema upgrade. Application rollback is therefore not database rollback.

Contract and persistent-state baseline

The audit preserves the accepted contracts rather than inventing missing product policy:

• Organization is the tenant root, but User is global and can belong to multiple Organizations. A selective Organization restore is not equivalent to restoring a self-contained database schema.
• Project workspaces are authoritative curriculum-engineering files bounded by ADR-0018. PostgreSQL stores their absolute Project.workspaceDir locations; it does not store their file content.
• Run, provider cursor, ingress, delivery, audit, and workspace commit/recovery semantics remain split across the unresolved lifecycle tickets. A backup design cannot silently choose their crash behavior.
• ADR-0022 requires storage ceilings and emergency DRAIN/STOP_NOW, but does not decide backup frequency, retention, recovery objectives, or tenant-level restore scope.
• Audit retention is still OPEN; regulatory backup retention and deletion obligations require product/legal input already recorded in the map's Fog.

The recovery set has at least four classes:

State	Current location	Recovery significance
Tenant and workflow state	PostgreSQL	Organizations, identities, membership, permissions, Feishu bindings, sessions, runs, locks, receipts, usage facts, and audit records
Customer artifacts	HUB_PROJECT_WORKSPACE_ROOT	Project source files, generated artifacts, inbound attachments, and any workspace-local session/history files
Credentials and configuration	/srv/.../.secrets/platform.env today; future org-scoped encrypted connections plus external key material	Database/Feishu/provider access and ability to decrypt restored Organization credentials
Release identity	application release, Prisma migrations, cph, runtime/toolchain manifest	Required to start a schema-compatible application and reproduce outputs after restore

Caches, build output, and reinstallable dependencies should be classified as regenerable rather than silently mixed into the protected recovery set.

Executed evidence

All probes used isolated temporary paths or databases. No production or shared test data was modified.

1. The deployment rsync pattern deletes an in-tree workspace

A temporary live Hub directory was given the documented workspace shape and a sentinel customer file, then the exact deployment rsync include/exclude pattern was applied:

rsync -a --delete \
  --exclude node_modules --exclude dist --exclude .env \
  hub/ <temporary-live-hub>/

workspace_survived=false
reproduced: deploy rsync pattern deletes an in-tree workspace root

This is deterministic: data/ is absent from the release source and is not an exclude, so --delete removes it from the destination. The root cause is that the documented persistent path is nested inside the mutable release tree and the deploy contract does not distinguish release artifacts from state.

2. A rejected Project transaction leaves an orphan workspace

A minimal Node harness called the real built createProjectFromOrgAdmin with a valid Organization admin and a missing Folder. The fake Prisma boundary rejected inside the same transaction seam used by production. The harness asserted that the Organization workspace directory must remain empty and was run twice:

node .scratch/saas-production-readiness/probes/orphan-workspace.mjs

orphan_workspace_entries=project_4aa0b6792c4547059d2d92b505b9299e
exit_code=1

orphan_workspace_entries=project_190e21afb8854244872b1241a600aa58
exit_code=1

createManagedProject calls mkdir before prisma.$transaction; the failure path has no compensating removal, staged rename, or orphan reconciler. The throwaway harness was removed after reproduction.

3. PostgreSQL rolled back failed migration DDL but retained a blocking failure

An isolated database and throwaway Prisma project applied one migration that created a table, divided by zero, then would have created another table. The real repository Prisma CLI returned P3018:

migration_exit=1
before_failure=missing
after_failure=missing
20260710000000_partial_failure|finished=false|rolled_back=false|steps=0|logs=true

This specific PostgreSQL/Prisma 6.19.3 path did not leave partial DDL, so the audit does not claim that it did. It did leave an unfinished _prisma_migrations row, and Prisma's error explicitly refused new migrations until operator recovery. Because systemd runs migrate deploy as ExecStartPre, the application cannot start past that state. The throwaway database and migration were removed. This one multi-statement result is not a general atomicity guarantee: Prisma's first-party guidance says PostgreSQL migrations are not transaction-wrapped by default, so each migration must make its transaction and partial-failure behavior explicit.

4. The current schema is not backward-compatible with an older application

The application immediately before the typed-principal migration queried PermissionGrant.principal. That migration drops the column in favor of principalType plus principalId. Running the old read against the current test database produced:

ERROR: column "principal" does not exist
HINT: Perhaps you meant to reference the column "PermissionGrant.principalId".
old_query_exit=1

This is direct evidence that switching the application symlink back after that migration cannot restore service. The migration history also deletes duplicate grant rows and later drops ExternalPrincipalMembership.source; there are no downgrade migrations.

5. A database-only custom dump restores cleanly

As a positive baseline, pg_dump --format=custom captured the current cph_hub_test database and pg_restore --exit-on-error --no-owner restored it into a new isolated database:

restored_tables=24
finished_migrations=9
failed_migrations=0
dump_bytes=76113

All nine finished migration checksums in the source database also matched the checked-in migration files. This proves a manual logical database round trip on the current PostgreSQL 14.20 workstation. It does not prove workspace, credentials, global database roles, production volume, cross-version restore, application readiness, or an operational RPO/RTO.

Critical findings

1. No backup, retention, or restore mechanism exists

The repository has no backup/restore command, systemd timer, off-host target, encryption/immutability policy, retention policy, backup manifest/checksum, success/failure alert, restore command, or scheduled recovery drill. The installer assumes PostgreSQL and its role/database already exist. A database dump alone does not reproduce those global prerequisites or the filesystem and secret state above.

There is also no decided RPO, RTO, backup frequency, retention duration, geographic copy rule, deletion-in-backups rule, or ownership/escalation path. Those values are product and operations decisions, not safe constants to infer from the single-host topology.

2. Persistent workspaces are inside the documented release tree

.env.example recommends ./data/project-workspaces; systemd's working directory is the live Hub directory; deployment mutates that same directory with rsync --delete. No preflight forbids an in-release workspace root and no exclude preserves it. This can turn a routine deploy into irreversible customer data loss before build, migration, restart, or readiness runs.

The persistent workspace root must live outside every release directory, have explicit ownership/mode and capacity, and be impossible for release retention or rsync --delete to traverse. Immutable release work already exists as a separate ticket, but its acceptance test must include this sentinel case.

3. PostgreSQL and workspaces have no consistency boundary

PostgreSQL can provide a transactionally consistent database snapshot, but the Hub mutates workspace files independently and has no generation, journal, commit marker, snapshot ID, or shared checkpoint that binds filesystem content to the corresponding database state. A database dump and filesystem copy taken at different instants can disagree about Project existence, active runs, provider cursors, delivered messages, and file effects.

Project creation already demonstrates the same missing compensation boundary. A replacement host has an additional problem: Project.workspaceDir is an absolute path and runtime paths from the database are used directly; restore has no same-mount validation, safe rebasing operation, or missing/orphan scan.

For the initial single-host topology, a quiesced backup checkpoint is the smallest credible route: stop new admission, drain/cancel according to the accepted run contract, reach durable zero in-flight work, capture database and workspace artifacts under one manifest, then resume. An online scheme would need a newly designed journal/generation protocol; it must not be implied by two independent copy commands.

4. Schema mutation is coupled to service start

Both npm start and the systemd unit execute prisma migrate deploy; every ordinary restart can therefore become a schema rollout. The release path has no separate migration lock/gate, migrate status check, production-data rehearsal, duration/lock budget, pre-migration recovery checkpoint, or compatibility manifest. CI only applies the full migration history to a fresh empty PostgreSQL database, so it does not exercise real upgrade volume, legacy values, old/new application overlap, failure recovery, or rollback compatibility.

The current failed-migration behavior is fail-closed, which is better than starting on an unknown schema, but there is no operator procedure for inspecting the failure, deciding roll-forward versus mark-rolled-back, restoring the checkpoint, or preventing a restart loop. Existing destructive migrations prove that application rollback cannot be assumed safe.

5. Restore has no isolated verification or cutover gate

There is no procedure that restores into a non-production environment with outbound Feishu/provider traffic disabled, verifies backup checksums and migration state, validates workspace paths/content/permissions, checks tenant relationships, decrypts a sample org connection, reconciles non-terminal work, runs application readiness and representative journeys, and measures recovery time/point before controlled cutover.

Starting the current server is not a neutral verifier: startup deletes all Project locks and independently marks only ACTIVE runs failed. Restored data must first pass the durable lifecycle reconciliation designed by the run-state tickets; otherwise validation itself mutates the evidence and can leave WAITING_FOR_USER inconsistent.

6. Full-service and Organization-selective recovery are different products

A full-service restore can recreate the shared database and all workspaces. Restoring one Organization is much harder: users are global and may belong to multiple Organizations, permission/audit/history rows span many related tables, workspace paths are external, and Feishu/provider connections have external state. The repository has no export/import identity mapping or conflict policy.

The initial production gate must at least prove full-service disaster recovery. Whether Organization-selective restore/export is promised, and how retained backups interact with customer deletion, remains an explicit product/legal decision rather than an accidental property of pg_restore filters.

Required recovery contract

Production readiness requires the following guarantees, with exact numerical objectives supplied by the pending policy decision:

1. Declared objectives and ownership. RPO/RTO, backup frequency, retention, copy locations, encryption/key custody, immutability, deletion handling, full-service versus Organization restore scope, operator ownership, and escalation are documented and testable.
2. Complete backup set. One cataloged manifest identifies release and migration checksums, PostgreSQL artifact and global prerequisites, workspace snapshot, configuration inventory, encrypted org-secret data, external key recovery material, timestamps, sizes, and cryptographic checksums. Backup data and decryption material are not kept only on the protected host or in the same failure domain.
3. One consistency point. A backup is taken only after an observable quiesce/checkpoint protocol, or by a proven online generation protocol that binds database state and workspace effects. Accepted input, active runs, queues, deliveries, locks, and provider cursors have explicit recovery semantics.
4. Safe migration rollout. Migration is a release phase, not an incidental service start. The gate checks status/checksums, rehearses on representative production-shaped data, records old/new application compatibility, uses expand-and-contract for destructive changes, requires a recent verified recovery checkpoint, and provides a tested failed-migration procedure.
5. Isolated restore verification. Every backup class is periodically restored to a clean isolated host/database with outbound integrations off. Verification checks integrity, schema/migration state, global database prerequisites, workspace completeness/path ownership, secret decryption, tenant isolation, lifecycle reconciliation, application readiness, and representative admin/Feishu/agent journeys.
6. Observable operation. Backup age, duration, artifact/checksum status, off-host replication, retention deletion, WAL/archive continuity if used, restore-drill result, measured RPO/RTO, and failures are logged/metricized and alerted with bounded cardinality.
7. Controlled cutover and fallback. Disaster recovery has an explicit authority, traffic/workload brake, DNS/proxy/Feishu reconnect sequence, duplicate/gap checks, rollback/fallback point, verification checklist, and incident evidence record.

Required release evidence

• A deploy test proves an in-tree or release-overlapping workspace root is rejected before any rsync/switch, while persistent sentinel files survive build failure, successful release, rollback, and release pruning.
• Project creation, archive/deletion, failed transaction, crash, and restore tests prove database/workspace allocation has no silent missing or orphan state; the reconciler reports rather than hides inconsistencies.
• Every checked-in migration is rehearsed from the previous production-shaped snapshot, with old/new compatibility and lock/runtime evidence. A deliberately failed migration exercises the documented recovery route.
• The backup implementation proves completeness, checksum verification, encryption, off-host copy, retention, monitoring, and fail-closed partial artifact cleanup.
• A clean isolated host restores a representative database, workspaces, secrets, and compatible release; lifecycle reconciliation and critical tenant journeys pass with outbound integrations controlled.
• Scheduled drills publish achieved recovery point and recovery time against the accepted objectives, and alert when no recent successful drill exists.

Primary-source interpretation

The official PostgreSQL, Prisma, and systemd behavior supporting this audit is collected with claim-level links in Backup and migration primary sources. That note distinguishes what a PostgreSQL-consistent dump proves from the cross-store guarantees the Hub must build itself.