curriculum-project-hub/.scratch/saas-production-readiness/assets/backup-migration-disaster-recovery-audit.md

# Backup, migration, and disaster-recovery audit

## Verdict

The current Hub has no production backup or restore control plane. PostgreSQL
can be dumped and restored manually, but the service's authoritative state is
split between PostgreSQL and mutable Project workspaces, with no shared
consistency point, backup catalog, off-host retention, restore workflow,
recovery objectives, or recovery drill.

The current deployment path can also delete customer workspaces when the
documented relative workspace root is used: `deploy_platform.sh` applies
`rsync --delete` to the live Hub directory while `.env.example` places
workspaces below `./data/project-workspaces`. Project creation has the same
cross-store weakness at a smaller scale: it creates the workspace before the
database transaction and leaves an orphan directory when that transaction
fails.

Migration safety is incomplete even though the current PostgreSQL migration
engine rolled back a deliberately failing migration's DDL in the executed
probe. The failed migration record blocks later migrations, migrations execute
inside every service start, no compatibility declaration or production-data
rehearsal exists, and current history contains destructive changes that make an
old application query fail after schema upgrade. Application rollback is
therefore not database rollback.

## Contract and persistent-state baseline

The audit preserves the accepted contracts rather than inventing missing
product policy:

- `Organization` is the tenant root, but `User` is global and can belong to
  multiple Organizations. A selective Organization restore is not equivalent
  to restoring a self-contained database schema.
- Project workspaces are authoritative curriculum-engineering files bounded by
  ADR-0018. PostgreSQL stores their absolute `Project.workspaceDir` locations;
  it does not store their file content.
- Run, provider cursor, ingress, delivery, audit, and workspace commit/recovery
  semantics remain split across the unresolved lifecycle tickets. A backup
  design cannot silently choose their crash behavior.
- ADR-0022 requires storage ceilings and emergency `DRAIN`/`STOP_NOW`, but does
  not decide backup frequency, retention, recovery objectives, or tenant-level
  restore scope.
- Audit retention is still `OPEN`; regulatory backup retention and deletion
  obligations require product/legal input already recorded in the map's Fog.

The recovery set has at least four classes:

| State | Current location | Recovery significance |
| --- | --- | --- |
| Tenant and workflow state | PostgreSQL | Organizations, identities, membership, permissions, Feishu bindings, sessions, runs, locks, receipts, usage facts, and audit records |
| Customer artifacts | `HUB_PROJECT_WORKSPACE_ROOT` | Project source files, generated artifacts, inbound attachments, and any workspace-local session/history files |
| Credentials and configuration | `/srv/.../.secrets/platform.env` today; future org-scoped encrypted connections plus external key material | Database/Feishu/provider access and ability to decrypt restored Organization credentials |
| Release identity | application release, Prisma migrations, `cph`, runtime/toolchain manifest | Required to start a schema-compatible application and reproduce outputs after restore |

Caches, build output, and reinstallable dependencies should be classified as
regenerable rather than silently mixed into the protected recovery set.

## Executed evidence

All probes used isolated temporary paths or databases. No production or shared
test data was modified.

### 1. The deployment rsync pattern deletes an in-tree workspace

A temporary live Hub directory was given the documented workspace shape and a
sentinel customer file, then the exact deployment rsync include/exclude pattern
was applied:

```text
rsync -a --delete \
  --exclude node_modules --exclude dist --exclude .env \
  hub/ <temporary-live-hub>/

workspace_survived=false
reproduced: deploy rsync pattern deletes an in-tree workspace root
```

This is deterministic: `data/` is absent from the release source and is not an
exclude, so `--delete` removes it from the destination. The root cause is that
the documented persistent path is nested inside the mutable release tree and
the deploy contract does not distinguish release artifacts from state.

### 2. A rejected Project transaction leaves an orphan workspace

A minimal Node harness called the real built `createProjectFromOrgAdmin` with a
valid Organization admin and a missing Folder. The fake Prisma boundary rejected
inside the same transaction seam used by production. The harness asserted that
the Organization workspace directory must remain empty and was run twice:

```text
node .scratch/saas-production-readiness/probes/orphan-workspace.mjs

orphan_workspace_entries=project_4aa0b6792c4547059d2d92b505b9299e
exit_code=1

orphan_workspace_entries=project_190e21afb8854244872b1241a600aa58
exit_code=1
```

`createManagedProject` calls `mkdir` before `prisma.$transaction`; the failure
path has no compensating removal, staged rename, or orphan reconciler. The
throwaway harness was removed after reproduction.

### 3. PostgreSQL rolled back failed migration DDL but retained a blocking failure

An isolated database and throwaway Prisma project applied one migration that
created a table, divided by zero, then would have created another table. The
real repository Prisma CLI returned `P3018`:

```text
migration_exit=1
before_failure=missing
after_failure=missing
20260710000000_partial_failure|finished=false|rolled_back=false|steps=0|logs=true
```

This specific PostgreSQL/Prisma 6.19.3 path did **not** leave partial DDL, so the
audit does not claim that it did. It did leave an unfinished
`_prisma_migrations` row, and Prisma's error explicitly refused new migrations
until operator recovery. Because systemd runs `migrate deploy` as
`ExecStartPre`, the application cannot start past that state. The throwaway
database and migration were removed. This one multi-statement result is not a
general atomicity guarantee: Prisma's first-party guidance says PostgreSQL
migrations are not transaction-wrapped by default, so each migration must make
its transaction and partial-failure behavior explicit.

### 4. The current schema is not backward-compatible with an older application

The application immediately before the typed-principal migration queried
`PermissionGrant.principal`. That migration drops the column in favor of
`principalType` plus `principalId`. Running the old read against the current
test database produced:

```text
ERROR: column "principal" does not exist
HINT: Perhaps you meant to reference the column "PermissionGrant.principalId".
old_query_exit=1
```

This is direct evidence that switching the application symlink back after that
migration cannot restore service. The migration history also deletes duplicate
grant rows and later drops `ExternalPrincipalMembership.source`; there are no
downgrade migrations.

### 5. A database-only custom dump restores cleanly

As a positive baseline, `pg_dump --format=custom` captured the current
`cph_hub_test` database and `pg_restore --exit-on-error --no-owner` restored it
into a new isolated database:

```text
restored_tables=24
finished_migrations=9
failed_migrations=0
dump_bytes=76113
```

All nine finished migration checksums in the source database also matched the
checked-in migration files. This proves a manual logical database round trip on
the current PostgreSQL 14.20 workstation. It does not prove workspace,
credentials, global database roles, production volume, cross-version restore,
application readiness, or an operational RPO/RTO.

## Critical findings

### 1. No backup, retention, or restore mechanism exists

The repository has no backup/restore command, systemd timer, off-host target,
encryption/immutability policy, retention policy, backup manifest/checksum,
success/failure alert, restore command, or scheduled recovery drill. The
installer assumes PostgreSQL and its role/database already exist. A database
dump alone does not reproduce those global prerequisites or the filesystem and
secret state above.

There is also no decided RPO, RTO, backup frequency, retention duration,
geographic copy rule, deletion-in-backups rule, or ownership/escalation path.
Those values are product and operations decisions, not safe constants to infer
from the single-host topology.

### 2. Persistent workspaces are inside the documented release tree

`.env.example` recommends `./data/project-workspaces`; systemd's working
directory is the live Hub directory; deployment mutates that same directory
with `rsync --delete`. No preflight forbids an in-release workspace root and no
exclude preserves it. This can turn a routine deploy into irreversible customer
data loss before build, migration, restart, or readiness runs.

The persistent workspace root must live outside every release directory, have
explicit ownership/mode and capacity, and be impossible for release retention
or `rsync --delete` to traverse. Immutable release work already exists as a
separate ticket, but its acceptance test must include this sentinel case.

### 3. PostgreSQL and workspaces have no consistency boundary

PostgreSQL can provide a transactionally consistent database snapshot, but the
Hub mutates workspace files independently and has no generation, journal,
commit marker, snapshot ID, or shared checkpoint that binds filesystem content
to the corresponding database state. A database dump and filesystem copy taken
at different instants can disagree about Project existence, active runs,
provider cursors, delivered messages, and file effects.

Project creation already demonstrates the same missing compensation boundary.
A replacement host has an additional problem: `Project.workspaceDir` is an
absolute path and runtime paths from the database are used directly; restore
has no same-mount validation, safe rebasing operation, or missing/orphan scan.

For the initial single-host topology, a quiesced backup checkpoint is the
smallest credible route: stop new admission, drain/cancel according to the
accepted run contract, reach durable zero in-flight work, capture database and
workspace artifacts under one manifest, then resume. An online scheme would
need a newly designed journal/generation protocol; it must not be implied by
two independent copy commands.

### 4. Schema mutation is coupled to service start

Both `npm start` and the systemd unit execute `prisma migrate deploy`; every
ordinary restart can therefore become a schema rollout. The release path has no
separate migration lock/gate, `migrate status` check, production-data rehearsal,
duration/lock budget, pre-migration recovery checkpoint, or compatibility
manifest. CI only applies the full migration history to a fresh empty
PostgreSQL database, so it does not exercise real upgrade volume, legacy values,
old/new application overlap, failure recovery, or rollback compatibility.

The current failed-migration behavior is fail-closed, which is better than
starting on an unknown schema, but there is no operator procedure for inspecting
the failure, deciding roll-forward versus mark-rolled-back, restoring the
checkpoint, or preventing a restart loop. Existing destructive migrations prove
that application rollback cannot be assumed safe.

### 5. Restore has no isolated verification or cutover gate

There is no procedure that restores into a non-production environment with
outbound Feishu/provider traffic disabled, verifies backup checksums and
migration state, validates workspace paths/content/permissions, checks tenant
relationships, decrypts a sample org connection, reconciles non-terminal work,
runs application readiness and representative journeys, and measures recovery
time/point before controlled cutover.

Starting the current server is not a neutral verifier: startup deletes all
Project locks and independently marks only `ACTIVE` runs failed. Restored data
must first pass the durable lifecycle reconciliation designed by the run-state
tickets; otherwise validation itself mutates the evidence and can leave
`WAITING_FOR_USER` inconsistent.

### 6. Full-service and Organization-selective recovery are different products

A full-service restore can recreate the shared database and all workspaces.
Restoring one Organization is much harder: users are global and may belong to
multiple Organizations, permission/audit/history rows span many related tables,
workspace paths are external, and Feishu/provider connections have external
state. The repository has no export/import identity mapping or conflict policy.

The initial production gate must at least prove full-service disaster recovery.
Whether Organization-selective restore/export is promised, and how retained
backups interact with customer deletion, remains an explicit product/legal
decision rather than an accidental property of `pg_restore` filters.

## Required recovery contract

Production readiness requires the following guarantees, with exact numerical
objectives supplied by the pending policy decision:

1. **Declared objectives and ownership.** RPO/RTO, backup frequency, retention,
   copy locations, encryption/key custody, immutability, deletion handling,
   full-service versus Organization restore scope, operator ownership, and
   escalation are documented and testable.
2. **Complete backup set.** One cataloged manifest identifies release and
   migration checksums, PostgreSQL artifact and global prerequisites, workspace
   snapshot, configuration inventory, encrypted org-secret data, external key
   recovery material, timestamps, sizes, and cryptographic checksums. Backup
   data and decryption material are not kept only on the protected host or in
   the same failure domain.
3. **One consistency point.** A backup is taken only after an observable
   quiesce/checkpoint protocol, or by a proven online generation protocol that
   binds database state and workspace effects. Accepted input, active runs,
   queues, deliveries, locks, and provider cursors have explicit recovery
   semantics.
4. **Safe migration rollout.** Migration is a release phase, not an incidental
   service start. The gate checks status/checksums, rehearses on representative
   production-shaped data, records old/new application compatibility, uses
   expand-and-contract for destructive changes, requires a recent verified
   recovery checkpoint, and provides a tested failed-migration procedure.
5. **Isolated restore verification.** Every backup class is periodically
   restored to a clean isolated host/database with outbound integrations off.
   Verification checks integrity, schema/migration state, global database
   prerequisites, workspace completeness/path ownership, secret decryption,
   tenant isolation, lifecycle reconciliation, application readiness, and
   representative admin/Feishu/agent journeys.
6. **Observable operation.** Backup age, duration, artifact/checksum status,
   off-host replication, retention deletion, WAL/archive continuity if used,
   restore-drill result, measured RPO/RTO, and failures are logged/metricized
   and alerted with bounded cardinality.
7. **Controlled cutover and fallback.** Disaster recovery has an explicit
   authority, traffic/workload brake, DNS/proxy/Feishu reconnect sequence,
   duplicate/gap checks, rollback/fallback point, verification checklist, and
   incident evidence record.

## Required release evidence

- A deploy test proves an in-tree or release-overlapping workspace root is
  rejected before any rsync/switch, while persistent sentinel files survive
  build failure, successful release, rollback, and release pruning.
- Project creation, archive/deletion, failed transaction, crash, and restore
  tests prove database/workspace allocation has no silent missing or orphan
  state; the reconciler reports rather than hides inconsistencies.
- Every checked-in migration is rehearsed from the previous production-shaped
  snapshot, with old/new compatibility and lock/runtime evidence. A deliberately
  failed migration exercises the documented recovery route.
- The backup implementation proves completeness, checksum verification,
  encryption, off-host copy, retention, monitoring, and fail-closed partial
  artifact cleanup.
- A clean isolated host restores a representative database, workspaces,
  secrets, and compatible release; lifecycle reconciliation and critical tenant
  journeys pass with outbound integrations controlled.
- Scheduled drills publish achieved recovery point and recovery time against the
  accepted objectives, and alert when no recent successful drill exists.

## Primary-source interpretation

The official PostgreSQL, Prisma, and systemd behavior supporting this audit is
collected with claim-level links in
[Backup and migration primary sources](./backup-migration-primary-sources.md).
That note distinguishes what a PostgreSQL-consistent dump proves from the
cross-store guarantees the Hub must build itself.