diff --git a/.scratch/saas-production-readiness/assets/backup-migration-disaster-recovery-audit.md b/.scratch/saas-production-readiness/assets/backup-migration-disaster-recovery-audit.md new file mode 100644 index 0000000..a49ef4c --- /dev/null +++ b/.scratch/saas-production-readiness/assets/backup-migration-disaster-recovery-audit.md @@ -0,0 +1,325 @@ +# Backup, migration, and disaster-recovery audit + +## Verdict + +The current Hub has no production backup or restore control plane. PostgreSQL +can be dumped and restored manually, but the service's authoritative state is +split between PostgreSQL and mutable Project workspaces, with no shared +consistency point, backup catalog, off-host retention, restore workflow, +recovery objectives, or recovery drill. + +The current deployment path can also delete customer workspaces when the +documented relative workspace root is used: `deploy_platform.sh` applies +`rsync --delete` to the live Hub directory while `.env.example` places +workspaces below `./data/project-workspaces`. Project creation has the same +cross-store weakness at a smaller scale: it creates the workspace before the +database transaction and leaves an orphan directory when that transaction +fails. + +Migration safety is incomplete even though the current PostgreSQL migration +engine rolled back a deliberately failing migration's DDL in the executed +probe. The failed migration record blocks later migrations, migrations execute +inside every service start, no compatibility declaration or production-data +rehearsal exists, and current history contains destructive changes that make an +old application query fail after schema upgrade. Application rollback is +therefore not database rollback. + +## Contract and persistent-state baseline + +The audit preserves the accepted contracts rather than inventing missing +product policy: + +- `Organization` is the tenant root, but `User` is global and can belong to + multiple Organizations. A selective Organization restore is not equivalent + to restoring a self-contained database schema. +- Project workspaces are authoritative curriculum-engineering files bounded by + ADR-0018. PostgreSQL stores their absolute `Project.workspaceDir` locations; + it does not store their file content. +- Run, provider cursor, ingress, delivery, audit, and workspace commit/recovery + semantics remain split across the unresolved lifecycle tickets. A backup + design cannot silently choose their crash behavior. +- ADR-0022 requires storage ceilings and emergency `DRAIN`/`STOP_NOW`, but does + not decide backup frequency, retention, recovery objectives, or tenant-level + restore scope. +- Audit retention is still `OPEN`; regulatory backup retention and deletion + obligations require product/legal input already recorded in the map's Fog. + +The recovery set has at least four classes: + +| State | Current location | Recovery significance | +| --- | --- | --- | +| Tenant and workflow state | PostgreSQL | Organizations, identities, membership, permissions, Feishu bindings, sessions, runs, locks, receipts, usage facts, and audit records | +| Customer artifacts | `HUB_PROJECT_WORKSPACE_ROOT` | Project source files, generated artifacts, inbound attachments, and any workspace-local session/history files | +| Credentials and configuration | `/srv/.../.secrets/platform.env` today; future org-scoped encrypted connections plus external key material | Database/Feishu/provider access and ability to decrypt restored Organization credentials | +| Release identity | application release, Prisma migrations, `cph`, runtime/toolchain manifest | Required to start a schema-compatible application and reproduce outputs after restore | + +Caches, build output, and reinstallable dependencies should be classified as +regenerable rather than silently mixed into the protected recovery set. + +## Executed evidence + +All probes used isolated temporary paths or databases. No production or shared +test data was modified. + +### 1. The deployment rsync pattern deletes an in-tree workspace + +A temporary live Hub directory was given the documented workspace shape and a +sentinel customer file, then the exact deployment rsync include/exclude pattern +was applied: + +```text +rsync -a --delete \ + --exclude node_modules --exclude dist --exclude .env \ + hub/ / + +workspace_survived=false +reproduced: deploy rsync pattern deletes an in-tree workspace root +``` + +This is deterministic: `data/` is absent from the release source and is not an +exclude, so `--delete` removes it from the destination. The root cause is that +the documented persistent path is nested inside the mutable release tree and +the deploy contract does not distinguish release artifacts from state. + +### 2. A rejected Project transaction leaves an orphan workspace + +A minimal Node harness called the real built `createProjectFromOrgAdmin` with a +valid Organization admin and a missing Folder. The fake Prisma boundary rejected +inside the same transaction seam used by production. The harness asserted that +the Organization workspace directory must remain empty and was run twice: + +```text +node .scratch/saas-production-readiness/probes/orphan-workspace.mjs + +orphan_workspace_entries=project_4aa0b6792c4547059d2d92b505b9299e +exit_code=1 + +orphan_workspace_entries=project_190e21afb8854244872b1241a600aa58 +exit_code=1 +``` + +`createManagedProject` calls `mkdir` before `prisma.$transaction`; the failure +path has no compensating removal, staged rename, or orphan reconciler. The +throwaway harness was removed after reproduction. + +### 3. PostgreSQL rolled back failed migration DDL but retained a blocking failure + +An isolated database and throwaway Prisma project applied one migration that +created a table, divided by zero, then would have created another table. The +real repository Prisma CLI returned `P3018`: + +```text +migration_exit=1 +before_failure=missing +after_failure=missing +20260710000000_partial_failure|finished=false|rolled_back=false|steps=0|logs=true +``` + +This specific PostgreSQL/Prisma 6.19.3 path did **not** leave partial DDL, so the +audit does not claim that it did. It did leave an unfinished +`_prisma_migrations` row, and Prisma's error explicitly refused new migrations +until operator recovery. Because systemd runs `migrate deploy` as +`ExecStartPre`, the application cannot start past that state. The throwaway +database and migration were removed. This one multi-statement result is not a +general atomicity guarantee: Prisma's first-party guidance says PostgreSQL +migrations are not transaction-wrapped by default, so each migration must make +its transaction and partial-failure behavior explicit. + +### 4. The current schema is not backward-compatible with an older application + +The application immediately before the typed-principal migration queried +`PermissionGrant.principal`. That migration drops the column in favor of +`principalType` plus `principalId`. Running the old read against the current +test database produced: + +```text +ERROR: column "principal" does not exist +HINT: Perhaps you meant to reference the column "PermissionGrant.principalId". +old_query_exit=1 +``` + +This is direct evidence that switching the application symlink back after that +migration cannot restore service. The migration history also deletes duplicate +grant rows and later drops `ExternalPrincipalMembership.source`; there are no +downgrade migrations. + +### 5. A database-only custom dump restores cleanly + +As a positive baseline, `pg_dump --format=custom` captured the current +`cph_hub_test` database and `pg_restore --exit-on-error --no-owner` restored it +into a new isolated database: + +```text +restored_tables=24 +finished_migrations=9 +failed_migrations=0 +dump_bytes=76113 +``` + +All nine finished migration checksums in the source database also matched the +checked-in migration files. This proves a manual logical database round trip on +the current PostgreSQL 14.20 workstation. It does not prove workspace, +credentials, global database roles, production volume, cross-version restore, +application readiness, or an operational RPO/RTO. + +## Critical findings + +### 1. No backup, retention, or restore mechanism exists + +The repository has no backup/restore command, systemd timer, off-host target, +encryption/immutability policy, retention policy, backup manifest/checksum, +success/failure alert, restore command, or scheduled recovery drill. The +installer assumes PostgreSQL and its role/database already exist. A database +dump alone does not reproduce those global prerequisites or the filesystem and +secret state above. + +There is also no decided RPO, RTO, backup frequency, retention duration, +geographic copy rule, deletion-in-backups rule, or ownership/escalation path. +Those values are product and operations decisions, not safe constants to infer +from the single-host topology. + +### 2. Persistent workspaces are inside the documented release tree + +`.env.example` recommends `./data/project-workspaces`; systemd's working +directory is the live Hub directory; deployment mutates that same directory +with `rsync --delete`. No preflight forbids an in-release workspace root and no +exclude preserves it. This can turn a routine deploy into irreversible customer +data loss before build, migration, restart, or readiness runs. + +The persistent workspace root must live outside every release directory, have +explicit ownership/mode and capacity, and be impossible for release retention +or `rsync --delete` to traverse. Immutable release work already exists as a +separate ticket, but its acceptance test must include this sentinel case. + +### 3. PostgreSQL and workspaces have no consistency boundary + +PostgreSQL can provide a transactionally consistent database snapshot, but the +Hub mutates workspace files independently and has no generation, journal, +commit marker, snapshot ID, or shared checkpoint that binds filesystem content +to the corresponding database state. A database dump and filesystem copy taken +at different instants can disagree about Project existence, active runs, +provider cursors, delivered messages, and file effects. + +Project creation already demonstrates the same missing compensation boundary. +A replacement host has an additional problem: `Project.workspaceDir` is an +absolute path and runtime paths from the database are used directly; restore +has no same-mount validation, safe rebasing operation, or missing/orphan scan. + +For the initial single-host topology, a quiesced backup checkpoint is the +smallest credible route: stop new admission, drain/cancel according to the +accepted run contract, reach durable zero in-flight work, capture database and +workspace artifacts under one manifest, then resume. An online scheme would +need a newly designed journal/generation protocol; it must not be implied by +two independent copy commands. + +### 4. Schema mutation is coupled to service start + +Both `npm start` and the systemd unit execute `prisma migrate deploy`; every +ordinary restart can therefore become a schema rollout. The release path has no +separate migration lock/gate, `migrate status` check, production-data rehearsal, +duration/lock budget, pre-migration recovery checkpoint, or compatibility +manifest. CI only applies the full migration history to a fresh empty +PostgreSQL database, so it does not exercise real upgrade volume, legacy values, +old/new application overlap, failure recovery, or rollback compatibility. + +The current failed-migration behavior is fail-closed, which is better than +starting on an unknown schema, but there is no operator procedure for inspecting +the failure, deciding roll-forward versus mark-rolled-back, restoring the +checkpoint, or preventing a restart loop. Existing destructive migrations prove +that application rollback cannot be assumed safe. + +### 5. Restore has no isolated verification or cutover gate + +There is no procedure that restores into a non-production environment with +outbound Feishu/provider traffic disabled, verifies backup checksums and +migration state, validates workspace paths/content/permissions, checks tenant +relationships, decrypts a sample org connection, reconciles non-terminal work, +runs application readiness and representative journeys, and measures recovery +time/point before controlled cutover. + +Starting the current server is not a neutral verifier: startup deletes all +Project locks and independently marks only `ACTIVE` runs failed. Restored data +must first pass the durable lifecycle reconciliation designed by the run-state +tickets; otherwise validation itself mutates the evidence and can leave +`WAITING_FOR_USER` inconsistent. + +### 6. Full-service and Organization-selective recovery are different products + +A full-service restore can recreate the shared database and all workspaces. +Restoring one Organization is much harder: users are global and may belong to +multiple Organizations, permission/audit/history rows span many related tables, +workspace paths are external, and Feishu/provider connections have external +state. The repository has no export/import identity mapping or conflict policy. + +The initial production gate must at least prove full-service disaster recovery. +Whether Organization-selective restore/export is promised, and how retained +backups interact with customer deletion, remains an explicit product/legal +decision rather than an accidental property of `pg_restore` filters. + +## Required recovery contract + +Production readiness requires the following guarantees, with exact numerical +objectives supplied by the pending policy decision: + +1. **Declared objectives and ownership.** RPO/RTO, backup frequency, retention, + copy locations, encryption/key custody, immutability, deletion handling, + full-service versus Organization restore scope, operator ownership, and + escalation are documented and testable. +2. **Complete backup set.** One cataloged manifest identifies release and + migration checksums, PostgreSQL artifact and global prerequisites, workspace + snapshot, configuration inventory, encrypted org-secret data, external key + recovery material, timestamps, sizes, and cryptographic checksums. Backup + data and decryption material are not kept only on the protected host or in + the same failure domain. +3. **One consistency point.** A backup is taken only after an observable + quiesce/checkpoint protocol, or by a proven online generation protocol that + binds database state and workspace effects. Accepted input, active runs, + queues, deliveries, locks, and provider cursors have explicit recovery + semantics. +4. **Safe migration rollout.** Migration is a release phase, not an incidental + service start. The gate checks status/checksums, rehearses on representative + production-shaped data, records old/new application compatibility, uses + expand-and-contract for destructive changes, requires a recent verified + recovery checkpoint, and provides a tested failed-migration procedure. +5. **Isolated restore verification.** Every backup class is periodically + restored to a clean isolated host/database with outbound integrations off. + Verification checks integrity, schema/migration state, global database + prerequisites, workspace completeness/path ownership, secret decryption, + tenant isolation, lifecycle reconciliation, application readiness, and + representative admin/Feishu/agent journeys. +6. **Observable operation.** Backup age, duration, artifact/checksum status, + off-host replication, retention deletion, WAL/archive continuity if used, + restore-drill result, measured RPO/RTO, and failures are logged/metricized + and alerted with bounded cardinality. +7. **Controlled cutover and fallback.** Disaster recovery has an explicit + authority, traffic/workload brake, DNS/proxy/Feishu reconnect sequence, + duplicate/gap checks, rollback/fallback point, verification checklist, and + incident evidence record. + +## Required release evidence + +- A deploy test proves an in-tree or release-overlapping workspace root is + rejected before any rsync/switch, while persistent sentinel files survive + build failure, successful release, rollback, and release pruning. +- Project creation, archive/deletion, failed transaction, crash, and restore + tests prove database/workspace allocation has no silent missing or orphan + state; the reconciler reports rather than hides inconsistencies. +- Every checked-in migration is rehearsed from the previous production-shaped + snapshot, with old/new compatibility and lock/runtime evidence. A deliberately + failed migration exercises the documented recovery route. +- The backup implementation proves completeness, checksum verification, + encryption, off-host copy, retention, monitoring, and fail-closed partial + artifact cleanup. +- A clean isolated host restores a representative database, workspaces, + secrets, and compatible release; lifecycle reconciliation and critical tenant + journeys pass with outbound integrations controlled. +- Scheduled drills publish achieved recovery point and recovery time against the + accepted objectives, and alert when no recent successful drill exists. + +## Primary-source interpretation + +The official PostgreSQL, Prisma, and systemd behavior supporting this audit is +collected with claim-level links in +[Backup and migration primary sources](./backup-migration-primary-sources.md). +That note distinguishes what a PostgreSQL-consistent dump proves from the +cross-store guarantees the Hub must build itself. diff --git a/.scratch/saas-production-readiness/assets/backup-migration-primary-sources.md b/.scratch/saas-production-readiness/assets/backup-migration-primary-sources.md new file mode 100644 index 0000000..67ee80b --- /dev/null +++ b/.scratch/saas-production-readiness/assets/backup-migration-primary-sources.md @@ -0,0 +1,574 @@ +# Backup, migration, and disaster-recovery primary-source research + +This note is the primary-source evidence pack for the Wayfinder ticket +“Audit backup, migration, and disaster recovery safety.” External claims use +only PostgreSQL, Prisma, systemd, and rsync first-party documentation. Project +recommendations are explicitly labelled as derived implications rather than +upstream guarantees. + +PostgreSQL links below use `docs/current`, which was PostgreSQL 18 when this +research was performed on 2026-07-10. A production implementation must pin the +deployed PostgreSQL major and verify the same pages for that major. The Hub +currently depends on Prisma `^6.19.3`, so Prisma 6-specific command examples +remain relevant even where the live documentation also describes Prisma 7. + +## Executive conclusions + +1. `pg_dump` can take a consistent **single-database** snapshot while ordinary + reads and writes continue, but it does not include the Hub's workspace tree, + cluster-global roles/tablespaces, or a WAL chain. PostgreSQL now explicitly + says it is generally not the right regular production-backup mechanism + outside simple cases. It is a useful logical/export layer, not proof of + whole-service disaster recovery. + ([`pg_dump` description](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION), + [continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING)) +2. Database PITR requires a physical base backup plus a continuous, unbroken + WAL sequence from the backup start through the recovery target. A successful + base-backup command alone does not prove that chain exists. + ([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING), + [making a base backup](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-BASE-BACKUP)) +3. `pg_verifybackup` is necessary but not sufficient: PostgreSQL explicitly + requires test restores followed by checks that the restored database works + and contains the expected data. + ([`pg_verifybackup`](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP)) +4. Prisma `migrate deploy` applies pending migrations, but it does not detect + schema drift and, on PostgreSQL, Prisma does not wrap a migration in a + transaction by default. A failed migration can therefore leave partial + schema changes requiring explicit operator recovery. + ([Prisma production workflow](https://www.prisma.io/docs/orm/prisma-migrate/workflows/development-and-production#production-and-testing-environments), + [Prisma transactional behaviour](https://www.prisma.io/blog/prisma-migrate-dx-primitives#what-if-schema-migrations-were-atomic), + [failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration)) +5. systemd supplies process lifecycle orchestration, not an application-level + consistency barrier. It sends the configured termination signal, waits up + to `TimeoutStopSec=`, then can forcibly kill remaining processes; a + successful `systemctl stop` does not prove that an `ExecStop=` command or + application drain succeeded. + ([`TimeoutStopSec=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutStopSec=), + [`systemctl stop`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#stop%20PATTERN%E2%80%A6)) +6. The Hub persists one logical service state across two stores: PostgreSQL + records `Project.workspaceDir`, runs, messages, locks, and file-change audit + rows, while workspaces contain project files. A workspace transcript helper + exists but is not wired into the production runner. PostgreSQL's snapshot/ + PITR guarantees cover only PostgreSQL. A whole-service recovery point + therefore requires an application-defined database/workspace checkpoint or + a journaled/versioned reconciliation protocol; neither PostgreSQL nor + systemd creates one automatically. + ([Hub schema](../../../hub/prisma/schema.prisma), + [workspace transcript helper](../../../hub/src/agent/transcript.ts), + [`pg_dump` scope](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION)) + +## Current repository facts that make the boundaries material + +- The systemd unit runs `prisma migrate deploy` as `ExecStartPre=` before every + service start and uses `Restart=on-failure`. + ([`cph-hub.service`](../../../hub/deploy/cph-hub.service)) +- The deploy script updates the live Hub directory using + `rsync -az --delete`, builds in that same directory, restarts the unit, and + accepts a constant liveness endpoint as deployment success. + ([`deploy_platform.sh`](../../../hub/deploy/deploy_platform.sh)) +- `Project.workspaceDir` is database state, but the referenced directory holds + independently mutable tenant content. A workspace JSONL transcript helper + exists but is not wired into the production runner; file contents still live + in the workspace while the database separately carries run/message/file-change + facts. + ([Hub schema](../../../hub/prisma/schema.prisma), + [workspace transcript helper](../../../hub/src/agent/transcript.ts)) + +These are project facts, not upstream guarantees. The sections below establish +what the upstream tools do and, critically, what they do not do. + +## PostgreSQL logical dump and restore + +### Snapshot and concurrent-write semantics + +- `pg_dump` makes a consistent export even while the database is being used; + it does not block ordinary readers or writers. It dumps only one database. + Cluster-global objects such as roles and tablespaces require `pg_dumpall` or + `pg_dumpall --globals-only`. + ([`pg_dump` description](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION), + [whole-cluster logical dumps](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-ALL)) +- `pg_dumpall` invokes `pg_dump` once per database. Each database is internally + consistent, but the snapshots are not synchronized across databases. + ([whole-cluster logical dumps](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-ALL)) +- A parallel dump uses directory format, opens `njobs + 1` connections, and + uses synchronized snapshots so every worker sees the same data set. The + connection count and additional server load are part of its capacity cost. + ([`pg_dump --jobs`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS)) +- Concurrent DDL can still interfere. The parallel-dump leader holds + `ACCESS SHARE` locks; if a later exclusive-lock request queues ahead of a + worker, that worker uses `NOWAIT` and aborts the dump rather than deadlocking. + `--lock-wait-timeout` can bound the initial lock wait. + ([`pg_dump --jobs` and lock behaviour](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS)) +- The ordinary snapshot is suitable for disaster-recovery dump semantics, but + it need not correspond to a serial execution of all transactions that later + commit. `--serializable-deferrable` waits for an anomaly-free snapshot; + PostgreSQL says that option is not beneficial for a dump intended only for + disaster recovery, but it can matter for a continuously updated reporting + copy. + ([`--serializable-deferrable`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS)) +- The default waits for dump files to be written safely to disk. Production + backups must not use `--no-sync`, because an immediate OS crash can otherwise + leave a corrupt dump. + ([`pg_dump --no-sync`](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-OPTIONS)) + +Derived Hub implications: + +- An online database dump may be internally valid while referring to a + `workspaceDir` whose files are from another instant. “`pg_dump` succeeded” is + therefore not a whole-Hub consistency claim. +- Backup scheduling must either serialize against schema migration/DDL or + treat any lock wait, abort, non-zero exit, and stderr warning as a failed + backup. A partially written archive must never be published as restorable. +- If the deployed cluster contains more than the Hub database, a single Hub + dump is not a cluster backup. Even `pg_dumpall` does not create a single + cross-database snapshot. + +### Restore error and atomicity semantics + +- `pg_restore` defaults to continuing after SQL errors and reports an error + count at the end. `--exit-on-error` is required for fail-fast behaviour. + `--single-transaction` wraps all emitted commands in `BEGIN`/`COMMIT`, makes + the restore all-or-nothing, and implies `--exit-on-error`. + ([`pg_restore` error and transaction options](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS)) +- Parallel restore uses separate connections for data loading, index creation, + and constraint creation. It supports only custom/directory archives and + cannot be combined with `--single-transaction`; it is not a whole-restore + atomic operation. + ([`pg_restore --jobs`](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS)) +- A single giant restore transaction can be impractical because it locks every + restored object and may exhaust the lock table. PostgreSQL 18 offers + `--transaction-size=N` as a bounded intermediate option; it implies + `--exit-on-error` but is not all-or-nothing. + ([`pg_restore --transaction-size`](https://www.postgresql.org/docs/current/app-pgrestore.html#APP-PGRESTORE-OPTIONS)) +- Plain SQL restored by `psql` also continues after errors unless + `ON_ERROR_STOP` is enabled; a single transaction is separately required for + all-or-nothing behaviour. + ([restoring an SQL dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE)) +- PostgreSQL recommends restoring into a truly empty database based on + `template0`; otherwise local additions in `template1` can cause duplicate + definition errors. Required roles must exist before owner/ACL restoration. + ([restoring an SQL dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE)) +- A dump can execute arbitrary code selected by superusers of the source + database. Partial dump/restore does not remove that trust boundary. + ([`pg_dump` security warning](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-DESCRIPTION)) +- A `pg_dump` binary can dump older servers but refuses a server newer than its + own major version. Output is intended to load into newer PostgreSQL versions, + but loading into an older major is not guaranteed. + ([`pg_dump` cross-version notes](https://www.postgresql.org/docs/current/app-pgdump.html#PG-DUMP-NOTES)) + +Derived Hub implications: + +- Restore should target a new isolated database/cluster. If a non-atomic or + parallel restore reports any error, discard that target and start again; + never promote a partially restored database. +- The runbook must record the restore atomicity strategy and exact PostgreSQL + client/server versions. A green process exit without explicit fail-fast + options is insufficient evidence. +- A restored database must remain disconnected from real Feishu/provider + side-effect endpoints until application-level validation completes. + +## Physical backup, WAL, and PITR + +### Base backups + +- `pg_basebackup` backs up a running PostgreSQL cluster without blocking other + clients. A full backup is an exact copy of the entire cluster and can seed + PITR or a standby; it cannot select one database or object. + ([`pg_basebackup` description](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP)) +- It uses the replication protocol. The account needs `REPLICATION` or + superuser privilege, `pg_hba.conf` must permit it, and `max_wal_senders` + needs capacity for the backup and, when streaming WAL, another connection. + ([`pg_basebackup` requirements](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP)) +- Default `-X stream` obtains WAL over a second connection while copying. With + `-X fetch`, WAL is collected only at the end; if required WAL was recycled, + the backup fails and is unusable. + ([`pg_basebackup --wal-method`](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP)) +- The default waits for backup files to be durable. `--no-sync` is explicitly + unsuitable for production because an OS crash can corrupt the backup. + ([`pg_basebackup --no-sync`](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP)) +- A manifest is generated by default, with per-file checksums. A SHA manifest + can detect malicious modification only if the manifest itself is held in a + separate trusted location or otherwise authenticated. + ([`pg_basebackup` manifest options](https://www.postgresql.org/docs/current/app-pgbasebackup.html#APP-PGBASEBACKUP)) + +Derived minimum backup record: + +- source cluster/system identifier and PostgreSQL version; +- backup tool version, start/end time, start/end LSN and WAL range; +- manifest digest and its independently protected location; +- destination object/version, byte count, exit status, and verification state; +- an explicit `VALID`, `INVALID`, or `QUARANTINED` state so a failed leftover + directory cannot enter the retention set merely because files exist. + +### WAL archival and recovery + +- PITR combines a physical backup with WAL replay. It can stop at a chosen time + and reach a consistent database state. Logical dumps do not contain enough + information for WAL replay. + ([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING)) +- Successful recovery requires a continuous sequence of archived WAL extending + back at least to the base-backup start. PostgreSQL tells operators to set up + and test WAL archiving **before** taking the first base backup. + ([continuous archiving overview](https://www.postgresql.org/docs/current/continuous-archiving.html#CONTINUOUS-ARCHIVING)) +- The archive command must return zero if and only if durable archival + succeeded. Non-zero causes retry. Archive implementations should refuse to + overwrite an existing file; a repeated name may report success only when the + existing durable content is identical, and differing content must fail. + ([WAL archiving](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL)) +- Repeated archive failure accumulates WAL in `pg_wal`; filling that filesystem + causes PostgreSQL to panic and remain offline until space is freed. PostgreSQL + explicitly advises monitoring archival progress. + ([WAL archiving failure behaviour](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL)) +- WAL is archived after a segment is completed, so low traffic can make the + last committed change wait a long time for archival. `archive_timeout` can + bound that age by forcing segment switches, at the cost of archive growth. + ([WAL archiving latency](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL)) +- WAL archival does not restore manually edited PostgreSQL configuration files + (`postgresql.conf`, `pg_hba.conf`, `pg_ident.conf`); those need a separate + file backup. + ([WAL archiving caveat](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL)) +- PostgreSQL's recovery procedure keeps clients out, restores the base backup, + configures `restore_command` and `recovery.signal`, replays WAL, and requires + inspection of database contents before restoring ordinary access. + ([PITR recovery procedure](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-PITR-RECOVERY)) +- Recovery can target a timestamp, LSN, transaction ID, named restore point, or + the earliest consistent point. Only one target may be set. Timeline history + is required to navigate recovery branches, and a configured target not + reached before WAL exhaustion is a fatal recovery failure. + ([recovery target settings](https://www.postgresql.org/docs/current/runtime-config-wal.html#RUNTIME-CONFIG-WAL-RECOVERY-TARGET), + [timelines](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-TIMELINES)) + +Derived Hub implications: + +- The database RPO is the age of the newest **durably archived, continuous** + WAL position, not the age of the newest base backup and not merely the name + in `last_archived_wal`. +- Archive keys need a collision-proof cluster/system-identifier namespace and + create-if-absent semantics; unrelated clusters must never overwrite a shared + WAL name. +- Monitoring must cover archive failures, time since durable success, archive + lag, WAL continuity, `pg_wal` growth, free space, and base-backup age. +- PITR restores PostgreSQL only. It does not rewind workspace files. A database + recovery target has no whole-Hub meaning until paired with a workspace + recovery point and reconciliation rule. + +## Verification is a ladder, not a checksum + +- `pg_verifybackup` validates the backup manifest and system identifier, + detects missing/extra/changed files, checks file checksums, and parses the + required WAL with `pg_waldump`. WAL verification is currently supported only + for plain-format base backups. + ([`pg_verifybackup` checks](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP)) +- PostgreSQL explicitly says these checks cannot cover everything a running + server will encounter and still requires test restores plus checks that the + database works and contains correct data. + ([`pg_verifybackup` limitation](https://www.postgresql.org/docs/current/app-pgverifybackup.html#APP-PGVERIFYBACKUP)) +- `pg_amcheck` can check supported heap/index structures, but currently only + ordinary/TOAST tables, materialized views, sequences, and B-tree indexes are + supported; other relation types are silently skipped. Stronger B-tree checks + can block concurrent modifications. + ([`pg_amcheck`](https://www.postgresql.org/docs/current/app-pgamcheck.html#APP-PGAMCHECK)) + +Derived recovery-evidence ladder: + +1. **Artifact verification:** backup command succeeded; manifest and all + required WAL validate; any failure marks the artifact unusable. +2. **Isolated technical restore:** restore to a clean non-production target, + reach the intended LSN/time/timeline, and prove PostgreSQL starts without + exposing it to users. +3. **Database validation:** verify migration history/schema identity, tenant + counts and relationships, critical constraints/sequences, and appropriate + corruption checks. +4. **Workspace validation:** restore the paired snapshot/version, validate file + hashes/ownership/containment, and reconcile database file-change/run state + against actual files. +5. **Application validation:** run read-only tenant-isolation and core workflow + smoke tests with external side effects disabled. +6. **Recovery report:** persist the exact backup, WAL range, workspace point, + release/schema versions, checks, observed RPO/RTO, operator, and failures. + Only a fully passing report may authorize promotion. + +## Prisma Migrate production semantics + +### What `migrate deploy` proves and does not prove + +- In production, `migrate deploy` compares applied migration checksums with the + migration directory, warns when an applied migration was modified, and + applies pending migrations. + ([Prisma 6 production workflow](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#production-and-testing-environments)) +- It does **not** warn when an already-applied migration is missing, detect + production schema drift, reset the database, generate Prisma Client, or use a + shadow database. + ([Prisma 6 production workflow](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#production-and-testing-environments)) +- The migration directory is the deployable history and must be committed in + full, including `migration_lock.toml`. Prisma warns against editing or + deleting migrations already applied in production; `migrate deploy` executes + migration SQL rather than deriving a new migration from the Prisma schema. + ([Prisma migration histories](https://www.prisma.io/docs/orm/prisma-migrate/understanding-prisma-migrate/migration-histories)) +- Prisma uses a database advisory lock for production migration commands so + multiple migration commands do not run simultaneously. The documented lock + timeout is ten seconds and not configurable; after timeout, the caller must + retry. The lock can be disabled by environment variable. + ([Prisma 6 advisory locking](https://www.prisma.io/docs/orm/v6/prisma-migrate/workflows/development-and-production#advisory-locking)) + +Derived Hub implication: `migrate deploy` exit zero proves that this invocation +applied or found no pending migration. It does not prove the live schema equals +the expected migration end state, that the new and previous application builds +are both compatible with it, or that the migration is operationally safe under +production data volume and traffic. + +The advisory lock is a migration-command concurrency guard, not a substitute +for the repository's deployment lock or application/schema compatibility. As a +derived safety rule, production preflight should reject +`PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK` unless an explicit reviewed mechanism +provides the same serialization guarantee. + +### Transactions and failed migrations + +- Prisma's first-party migration guidance states that PostgreSQL migrations + are not wrapped in a transaction by default. Operators may opt in by adding + `BEGIN;`/`COMMIT;` to a migration, but large transactional migrations can hold + locks longer and consume more database resources. + ([Prisma transactional behaviour](https://www.prisma.io/blog/prisma-migrate-dx-primitives#what-if-schema-migrations-were-atomic)) +- A failed migration records its error in `_prisma_migrations.logs`; further + migrations cannot deploy until the failed state is explicitly recovered. + Recovery is either (a) revert/repair partial steps, mark the migration rolled + back, and redeploy, or (b) manually complete the exact steps and mark it + applied. + ([Prisma failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration)) +- Prisma error codes `P3009` and `P3018` explicitly report that failed + migrations block new migration application until recovery. + ([Prisma error reference](https://www.prisma.io/docs/orm/reference/error-reference#p3009)) +- `migrate resolve --rolled-back` changes migration-history state so Prisma may + apply the migration again; it does not itself undo SQL already executed. + `--applied` records success without executing the migration SQL. + ([failed-migration recovery](https://www.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing#failed-migration), + [`migrate resolve`](https://www.prisma.io/docs/cli/migrate/resolve)) +- Prisma can generate a down SQL script, but its documentation limits the + simple rollback workflow to a **failed** up migration. To reverse a successful + migration, the supported history is a new forward migration generated from a + reverted Prisma schema. A schema down script also cannot restore lost data, + application changes, or manually authored SQL automatically. + ([generating down migrations](https://www.prisma.io/docs/orm/prisma-migrate/workflows/generating-down-migrations#considerations-when-generating-down-migrations)) + +Derived Hub implications: + +- Every migration needs a declared atomicity choice. If it is intentionally + non-transactional, the migration must be resumable/idempotent or ship an + exact, tested partial-failure procedure. Blindly rerunning a partially applied + script is unsafe. +- A database backup checkpoint before a destructive or difficult-to-reverse + migration is necessary, but restoring that checkpoint is a whole-database + recovery event—not a substitute for a compatible rollout design. +- Service restart cannot be treated as application rollback after a successful + schema migration. The previous binary may no longer understand the new + schema, and a down migration cannot recreate deleted data. + +### Compatibility and rollout ordering + +- Prisma documents expand-and-contract as the way to avoid downtime for fields + in active use: add the new representation, deploy code that writes both while + reading the old, migrate and verify data, switch reads, stop old writes, then + remove the old representation. + ([Prisma expand-and-contract workflow](https://www.prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations#example-use-the-expand-and-contract-pattern-to-evolve-the-schema-without-downtime)) +- Prisma also warns that generated “renames” may be expressed as add/drop and + lose data unless the SQL is reviewed and customized before application. + ([customizing migrations](https://www.prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations#example-rename-a-field)) + +Derived deployment contract: + +1. Migration SQL is immutable after production application and reviewed for + destructive operations, lock duration, data volume, and transaction choice. +2. It is rehearsed against a recent, anonymized production-scale restore and + its runtime/locks/disk growth are recorded. +3. Each release declares a schema compatibility window. Expand releases remain + compatible with both old and new application versions; contract/destructive + releases occur only after old binaries can no longer run. +4. Migration status/drift is checked before changing the live release. A + failed migration blocks deployment and enters an operator runbook; it must + never be hidden by repeated restart attempts. +5. Rollback means switching to a binary proven compatible with the current + schema or deploying a new forward migration. Restoring a backup is the last + resort and must also restore/reconcile the workspace recovery point. + +## systemd stop, restart, and dependency semantics + +- When `ExecStartPre=`, `ExecStart=`, or `ExecStartPost=` fails or times out, + service startup does not continue to the normal running process. `ExecStop=` + is skipped because startup never succeeded; `ExecStopPost=` is the hook that + still runs after a failed startup. + ([systemd start/stop command semantics](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStartPre=), + [`ExecStopPost=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStopPost=), + [permanent upstream man-page source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemd.service.xml#L525-L587)) +- `Restart=on-failure` covers non-zero exits and failures/timeouts in service + control processes, including `ExecStartPre=`. An explicit systemd stop does + not trigger automatic restart. Restarts are subject to unit start-rate + limiting. + ([`Restart=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Restart=), + [permanent upstream source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemd.service.xml#L856-L995)) +- With no `ExecStop=`, systemd sends `KillSignal=` (default `SIGTERM`). + `TimeoutStopSec=` bounds the wait; after it expires, remaining processes can + be forcibly killed with `SIGKILL`/`FinalKillSignal=`. + ([`ExecStop=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStop=), + [`TimeoutStopSec=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutStopSec=)) +- Default `KillMode=control-group` applies termination to all remaining + processes in the unit cgroup. systemd explicitly discourages `process` or + `none` because children can escape the service lifecycle and remain running + after the service is considered stopped. + ([`KillMode=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html#KillMode=)) +- `systemctl stop` waits for the job unless `--no-block` is used, but the command + does not fail merely because an `ExecStop=` command failed: the manager still + forcibly terminates the unit. + ([`systemctl stop`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#stop%20PATTERN%E2%80%A6), + [permanent upstream source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemctl.xml#L444-L459)) +- `systemctl restart` is a stop followed by a start; `ExecStop=` and + `ExecStopPost=` participate. It does not necessarily flush every unit resource + in the same way as separate completed `stop` then `start` operations. + ([restart semantics](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStop=), + [`systemctl restart`](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#restart%20PATTERN%E2%80%A6)) +- `After=` is only an ordering relationship; it does not pull in another unit. + Requirement and ordering dependencies are independent. During shutdown the + start order is reversed, so a Hub ordered `After=postgresql.service` stops + before PostgreSQL when both are in the shutdown transaction. + ([`Requires=` and ordering](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Requires=), + [`Before=`/`After=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Before=)) +- `systemctl freeze` only uses the cgroup freezer to suspend process execution; + its documented contract contains no filesystem flush or stable-image + guarantee. It is not a substitute for an application drain or a filesystem + snapshot barrier. + ([permanent `systemctl freeze` man-page source](https://github.com/systemd/systemd/blob/924552cfbacd51be0229ac6ce95fe3b69c7b7923/man/systemctl.xml#L653-L675)) + +Derived assessment of the current unit: + +- A Prisma migration failure in the current `ExecStartPre=` prevents the Hub + from starting, then `Restart=on-failure` can retry startup until rate-limited. + That is fail-closed, but it is not controlled migration recovery and can + repeatedly hit a database already left in Prisma's failed state. +- `KillSignal=SIGTERM` plus `TimeoutStopSec=30` is only a deadline. Until the Hub + installs a real termination handler that makes readiness false, stops ingress, + drains/checkpoints active work, closes Feishu and database clients, and reports + success, the unit cannot claim a graceful consistency barrier. +- A backup runbook must issue a controlled drain, verify durable work reached a + reconciled terminal/checkpoint state, stop the Hub, verify the unit and its + cgroup are inactive, and only then snapshot mutable workspace state. A zero + `systemctl stop` exit by itself is insufficient evidence. +- Recovery should not start the normal unit until the restored database and + workspace have been validated. The current unit would run pending migrations + immediately on first start, mutating the restored database before validation + and possibly destroying the ability to inspect the original recovered state. + +## Filesystem and rsync boundaries + +- PostgreSQL says a plain filesystem copy of a live data directory is unusable: + the database must be shut down, because tools such as `tar` do not create an + atomic snapshot and server buffering remains. A correctly implemented frozen + snapshot can work while PostgreSQL runs, but all volumes containing data, + WAL, and tablespaces must be captured simultaneously; otherwise shut down or + use the continuous-archive base-backup protocol. + ([PostgreSQL filesystem backups](https://www.postgresql.org/docs/current/backup-file.html)) +- For a PostgreSQL rsync backup, the documented safe pattern is a first pass + while running, followed by shutdown and a second `rsync --checksum`; the + second pass is what makes the result consistent. + ([PostgreSQL filesystem backups](https://www.postgresql.org/docs/current/backup-file.html)) +- At the filesystem layer, `fsfreeze` is documented to let in-progress + filesystem operations complete, halt new modifications, and flush dirty data, + metadata, and logs before returning. That is still only a filesystem-volume + guarantee; it does not replace PostgreSQL's backup protocol and does not make + separately frozen volumes simultaneous. + ([util-linux `fsfreeze` manual source](https://github.com/util-linux/util-linux/blob/master/sys-utils/fsfreeze.8.adoc)) +- rsync's `--delete` defaults to deletion during transfer on modern peers, so + the receiver tree changes incrementally. `--delay-updates` merely holds each + updated file and renames files into place rapidly at the end; its own manual + says this only attempts to make updates “a little more atomic” and points to + a separate parallel-tree/switch approach for stronger atomicity. + ([rsync `--delete`](https://rsync.samba.org/ftp/rsync/rsync.1.html#opt--delete), + [rsync `--delay-updates`](https://rsync.samba.org/ftp/rsync/rsync.1.html#opt--delay-updates)) + +Derived Hub implications: + +- The current in-place `rsync --delete` deployment is neither an immutable + release switch nor a backup snapshot. A failed/interrupted transfer can leave + a mixed live tree; service stop/restart cannot reconstruct the deleted prior + artifact. +- The workspace snapshot mechanism remains an explicit design choice. Whichever + storage technology is selected must prove atomicity over all workspace + volumes or use a quiesced/no-writer interval, then publish an immutable + version/checksum manifest. Merely copying a changing directory is not enough. + +## Derived minimum whole-Hub recovery contract + +The primary sources support the following minimum contract; exact RPO, RTO, +retention, storage vendor, and numeric schedules remain product/operations +decisions and must not be invented by implementation. + +### Backup path + +1. Maintain verified PostgreSQL base backups and a monitored continuous WAL + archive, or obtain and test an explicitly equivalent managed-PostgreSQL + guarantee. Keep logical dumps as a separate portability/selective-restore + layer, not the sole PITR mechanism. +2. Keep PostgreSQL configuration, deployment/release manifests, migration + history, and workspace snapshots in the backup inventory; WAL alone does not + cover them. +3. Create a whole-Hub checkpoint only after ingress is drained and active + workspace writers are stopped/reconciled, unless the eventual storage layer + supplies a proven atomic snapshot/journal spanning the necessary stores. +4. Persist one signed/immutable checkpoint manifest joining PostgreSQL + backup/WAL position, workspace snapshot/version, application revision, + migration digest, tool versions, checksums, timestamps, and validation state. +5. Treat database/WAL/workspace backups as cross-tenant sensitive data. Apply + least-privilege access, encryption, audit, deletion/retention, and an + independently protected integrity manifest; WAL contains effectively all + database changes and PostgreSQL advises protecting it from unauthorized + readers. + ([WAL archive confidentiality](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-ARCHIVING-WAL)) + +### Migration path + +1. Serialize deploys and backup checkpoints; run a drift/status gate before + migration, not just `migrate deploy`. +2. Rehearse each migration on a recent production-scale restore and record + runtime, locks, disk/WAL growth, transaction choice, and failure recovery. +3. Require expand/contract compatibility for online changes. A release must + declare which old/new binaries can run against each schema phase. +4. Separate migration orchestration from ordinary service restart so a failed + migration produces one durable incident state and an operator decision—not + an opaque restart loop. +5. Verify a restorable checkpoint before destructive change. Do not describe + backup restore as a normal “rollback”; it is whole-service disaster recovery. + +### Restore and promotion path + +1. Restore into an isolated environment with real external side effects + disabled. +2. Restore the selected PostgreSQL base backup/WAL target and the checkpoint's + exact workspace version; do not run pending Prisma migrations implicitly. +3. Prove artifact integrity, database startup, recovery target/timeline, + migration/schema identity, tenant invariants, workspace hashes/containment, + and read-only application flows. +4. Record observed recovery point and recovery time. A restore that starts but + misses its target, loses a tenant workspace, has partial migration state, or + fails any invariant is not promotable. +5. Promote explicitly, then enable ingress and external integrations. Preserve + the pre-recovery cluster/WAL and recovery report until the incident is closed. + +## Decisions and evidence still required + +These values are not fixed by the upstream tools and remain `OPEN` for the SaaS +production contract: + +- whole-Hub RPO and RTO, separately for ordinary deletion, database corruption, + host loss, region/provider loss, and migration failure; +- base-backup, logical-dump, workspace-snapshot, WAL, and restore-drill cadence; +- immutable/off-host storage, encryption/key ownership, retention and verified + deletion policy, including org-deletion and legal-retention interactions; +- exact workspace checkpoint/journal technology and database/workspace + reconciliation semantics for in-flight or partially written runs; +- whether restore promotion is whole-platform only or supports a safe + organization/project export-import path (PostgreSQL physical PITR is + cluster-wide); +- migration transaction policy, maximum lock/downtime budget, compatibility + window, and who can authorize contract/destructive phases; +- evidence from at least one production-like full restore and one PITR drill, + including deliberate missing-WAL, corrupted-backup, failed-migration, + interrupted-workspace-write, and restart-during-recovery probes. diff --git a/.scratch/saas-production-readiness/issues/06-audit-backup-migration-recovery.md b/.scratch/saas-production-readiness/issues/06-audit-backup-migration-recovery.md index 4c1e62b..0532993 100644 --- a/.scratch/saas-production-readiness/issues/06-audit-backup-migration-recovery.md +++ b/.scratch/saas-production-readiness/issues/06-audit-backup-migration-recovery.md @@ -1,7 +1,7 @@ # Audit backup, migration, and disaster recovery safety Type: research -Status: open +Status: resolved ## Question @@ -9,3 +9,40 @@ What backup, restore, workspace/database consistency, migration compatibility, and disaster-recovery guarantees are needed to deploy current persistent data safely, and what evidence or tooling is missing to prove them? +## Answer + +The Hub has no production backup/restore control plane. PostgreSQL can be +logically dumped and restored, but authoritative state is split across the +database, mutable Project workspaces, credentials/key material, and a +schema-compatible release with no shared checkpoint or recovery manifest. +RPO/RTO, retention, off-host/immutable storage, encryption/key custody, +full-service versus Organization-selective restore, monitoring, and recovery +drills are all undefined. + +Executed probes proved that the current deploy `rsync --delete` removes the +documented in-tree workspace root, a failed Project transaction leaves an +orphan workspace, and an older application query fails after a destructive +migration. A deliberately failed Prisma migration rolled back that probe's DDL +but left a blocking failed migration record; the official contract does not +make PostgreSQL migrations transactional by default. A positive custom-format +round trip restored 24 tables and all nine current migration records, proving +only database-level portability on the local PostgreSQL 14.20 setup. + +The complete state map, probe commands/results, root causes, required recovery +contract, and release evidence are in the +[backup, migration, and disaster-recovery audit](../assets/backup-migration-disaster-recovery-audit.md). +Official PostgreSQL, Prisma, systemd, and rsync semantics with claim-level links +are in [backup and migration primary sources](../assets/backup-migration-primary-sources.md). + +The newly-clear decision and implementation frontier is: + +- [Decide recovery objectives, backup retention, and restore scope](39-decide-recovery-objectives.md) +- [Implement coherent backup sets and verified restore](40-verified-backup-restore.md) +- [Make the Project workspace lifecycle recoverable](41-recoverable-project-workspace-lifecycle.md) +- [Establish migration compatibility and failed-migration recovery](14-migration-compatibility-and-recovery.md) + +Persistent-path provisioning and release-tree isolation remain owned by +[Provision a runnable non-root Hub service](10-provision-nonroot-service.md) +and [Make application releases atomic and rollbackable](12-atomic-releases-and-rollback.md). +Full recovery/cutover procedures and drill evidence remain owned by +[Write and drill production incident recovery runbooks](31-write-and-drill-incident-runbooks.md). diff --git a/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md b/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md index 85d08e3..75592dc 100644 --- a/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md +++ b/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md @@ -2,7 +2,7 @@ Type: task Status: open -Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38 +Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41 ## Question diff --git a/.scratch/saas-production-readiness/issues/10-provision-nonroot-service.md b/.scratch/saas-production-readiness/issues/10-provision-nonroot-service.md index c5de681..3044206 100644 --- a/.scratch/saas-production-readiness/issues/10-provision-nonroot-service.md +++ b/.scratch/saas-production-readiness/issues/10-provision-nonroot-service.md @@ -8,4 +8,6 @@ Status: open Make clean-host installation create or validate the dedicated service identity, home/cache/state/workspace paths and ownership, generate a complete production configuration, honor the configured bind host, and fail preflight before unit -installation when any required value or prerequisite is invalid. +installation when any required value or prerequisite is invalid. The persistent +workspace root must be outside all release trees, and preflight must reject any +path that overlaps rsync, release switching, rollback, or release pruning. diff --git a/.scratch/saas-production-readiness/issues/12-atomic-releases-and-rollback.md b/.scratch/saas-production-readiness/issues/12-atomic-releases-and-rollback.md index b3323db..8a5e284 100644 --- a/.scratch/saas-production-readiness/issues/12-atomic-releases-and-rollback.md +++ b/.scratch/saas-production-readiness/issues/12-atomic-releases-and-rollback.md @@ -9,4 +9,5 @@ Blocked by: 10, 11 Replace in-place rsync/build with deployment locking, immutable staged release directories, a revision manifest, atomic current/previous switching, bounded retention, and an explicit application rollback that preserves the live release -on build, preflight, switch, or readiness failure. +on build, preflight, switch, or readiness failure. Prove deploy, rollback, and +release pruning cannot traverse or delete the persistent workspace root. diff --git a/.scratch/saas-production-readiness/issues/14-migration-compatibility-and-recovery.md b/.scratch/saas-production-readiness/issues/14-migration-compatibility-and-recovery.md index ba427fe..5a64bbd 100644 --- a/.scratch/saas-production-readiness/issues/14-migration-compatibility-and-recovery.md +++ b/.scratch/saas-production-readiness/issues/14-migration-compatibility-and-recovery.md @@ -8,6 +8,8 @@ Blocked by: 06 Define and enforce the application/schema compatibility rule for each release, require expand-and-contract sequencing for destructive changes, add migration -status and production-like rehearsal gates, and provide an operator procedure -for recovering a partially failed Prisma migration without pretending that an -application rollback reverses the database. +status/checksum and production-like rehearsal gates, and move schema mutation +out of ordinary service start into an explicit release phase. Require a recent +verified recovery checkpoint, bounded lock/runtime evidence, and an operator +procedure for resolving or restoring a failed Prisma migration without +pretending that an application rollback reverses the database. diff --git a/.scratch/saas-production-readiness/issues/31-write-and-drill-incident-runbooks.md b/.scratch/saas-production-readiness/issues/31-write-and-drill-incident-runbooks.md index 49b4ba2..2a38c1d 100644 --- a/.scratch/saas-production-readiness/issues/31-write-and-drill-incident-runbooks.md +++ b/.scratch/saas-production-readiness/issues/31-write-and-drill-incident-runbooks.md @@ -2,7 +2,7 @@ Type: task Status: open -Blocked by: 06, 12, 13, 14, 23, 24, 25, 26, 27, 28, 29, 30 +Blocked by: 06, 12, 13, 14, 23, 24, 25, 26, 27, 28, 29, 30, 39, 40, 41 ## Question @@ -13,4 +13,6 @@ deploy/rollback, host restart during work, and suspected tenant exposure. Give each alert an owner and safe read-first commands, require confirmed/audited scope for destructive or replay actions, execute every runbook against production-shaped staging, and archive evidence that detection, mitigation, -reconciliation, verification, escalation, and alert clearing all work. +reconciliation, verification, escalation, and alert clearing all work. Include +database/workspace backup failure, failed migration, full-host restore and +cutover, duplicate/gap review, fallback, and achieved RPO/RTO evidence. diff --git a/.scratch/saas-production-readiness/issues/39-decide-recovery-objectives.md b/.scratch/saas-production-readiness/issues/39-decide-recovery-objectives.md new file mode 100644 index 0000000..595368a --- /dev/null +++ b/.scratch/saas-production-readiness/issues/39-decide-recovery-objectives.md @@ -0,0 +1,14 @@ +# Decide recovery objectives, backup retention, and restore scope + +Type: grilling +Status: open + +## Question + +For the initial SaaS service, decide measurable RPO/RTO targets, backup and +restore-drill frequency, retention and deletion-in-backups policy, off-host +copy/failure-domain requirements, logical export versus physical base-backup +and continuous-WAL/PITR coverage, encryption and key custody, operator ownership, +and whether the supported promise is full-service disaster recovery only or +also Organization-selective export/restore. Record compliance-dependent items +as explicit OPEN inputs rather than inventing defaults. diff --git a/.scratch/saas-production-readiness/issues/40-verified-backup-restore.md b/.scratch/saas-production-readiness/issues/40-verified-backup-restore.md new file mode 100644 index 0000000..68cb217 --- /dev/null +++ b/.scratch/saas-production-readiness/issues/40-verified-backup-restore.md @@ -0,0 +1,16 @@ +# Implement coherent backup sets and verified restore + +Type: task +Status: open +Blocked by: 10, 12, 14, 19, 23, 24, 25, 26, 29, 39, 41 + +## Question + +Implement the decided recovery policy as cataloged, checksummed, encrypted, +off-host backup sets covering PostgreSQL logical exports plus physical base +backup/continuous WAL or a proven managed equivalent, Project workspaces, +compatible release/migration identity, configuration inventory, org-secret +ciphertext, and separately recoverable key material. Establish a proven +cross-store checkpoint, retention and alerting, partial-artifact cleanup, +isolated restore with outbound integrations disabled, lifecycle reconciliation, +critical SaaS journey verification, and measured RPO/RTO drills. diff --git a/.scratch/saas-production-readiness/issues/41-recoverable-project-workspace-lifecycle.md b/.scratch/saas-production-readiness/issues/41-recoverable-project-workspace-lifecycle.md new file mode 100644 index 0000000..4088fb5 --- /dev/null +++ b/.scratch/saas-production-readiness/issues/41-recoverable-project-workspace-lifecycle.md @@ -0,0 +1,14 @@ +# Make the Project workspace lifecycle recoverable + +Type: task +Status: open +Blocked by: 10, 26, 34 + +## Question + +Move persistent workspaces outside every mutable release tree and make Project +workspace allocation, database commit, archive/deletion, relocation, restore, +and orphan/missing-path reconciliation explicit and crash-safe. Prove failed +Project creation leaves no silent orphan, deployment/rollback/pruning cannot +delete customer files, restored absolute paths are validated or safely rebased, +and every inconsistency is observable rather than auto-hidden. diff --git a/.scratch/saas-production-readiness/map.md b/.scratch/saas-production-readiness/map.md index 1ab5dcd..b2bb132 100644 --- a/.scratch/saas-production-readiness/map.md +++ b/.scratch/saas-production-readiness/map.md @@ -40,6 +40,7 @@ rollback and recovery, and no known critical security or data-integrity gaps. - [Audit run lifecycle and data integrity under failure](issues/03-audit-run-lifecycle-integrity.md) — retain the database-enforced project-lock uniqueness and normal happy path, but replace receipt-before-success, process-local queues, independent run/session/lock effects, silent delivery/audit loss, and partial startup reset with durable, recoverable state machines; explicitly decide failed-run workspace semantics. - [Define production observability and incident recovery](issues/04-define-observability-recovery.md) — keep the existing Fastify/Prisma/systemd signal fragments, but production requires dependency-aware readiness, typed and correlated errors, durable-work telemetry, an explicit SLO/retention/alerting control plane, and recovery runbooks proven by fault drills. - [Define initial abuse and capacity controls](issues/05-define-abuse-capacity-controls.md) — require layered fail-closed limits, durable Organization-fair admission, explicit backpressure, bounded files/storage/runs, soft usage alerts, and audited emergency workload brakes; calibrate numerical ceilings under production-like load. +- [Audit backup, migration, and disaster recovery safety](issues/06-audit-backup-migration-recovery.md) — separate persistent state from releases, define a database/workspace checkpoint and recovery objectives, make migrations compatibility-gated, and prove complete off-host backup sets through isolated restore and cutover drills. ## Fog