docs: resolve backup recovery audit

2026-07-10 12:30:19 +08:00
parent 1094ebd651
commit 599a2f6e66
12 changed files with 997 additions and 9 deletions
@@ -1,7 +1,7 @@
 # Audit backup, migration, and disaster recovery safety

 Type: research
-Status: open
+Status: resolved

 ## Question

@@ -9,3 +9,40 @@ What backup, restore, workspace/database consistency, migration compatibility,
 and disaster-recovery guarantees are needed to deploy current persistent data
 safely, and what evidence or tooling is missing to prove them?

+## Answer
+
+The Hub has no production backup/restore control plane. PostgreSQL can be
+logically dumped and restored, but authoritative state is split across the
+database, mutable Project workspaces, credentials/key material, and a
+schema-compatible release with no shared checkpoint or recovery manifest.
+RPO/RTO, retention, off-host/immutable storage, encryption/key custody,
+full-service versus Organization-selective restore, monitoring, and recovery
+drills are all undefined.
+
+Executed probes proved that the current deploy `rsync --delete` removes the
+documented in-tree workspace root, a failed Project transaction leaves an
+orphan workspace, and an older application query fails after a destructive
+migration. A deliberately failed Prisma migration rolled back that probe's DDL
+but left a blocking failed migration record; the official contract does not
+make PostgreSQL migrations transactional by default. A positive custom-format
+round trip restored 24 tables and all nine current migration records, proving
+only database-level portability on the local PostgreSQL 14.20 setup.
+
+The complete state map, probe commands/results, root causes, required recovery
+contract, and release evidence are in the
+[backup, migration, and disaster-recovery audit](../assets/backup-migration-disaster-recovery-audit.md).
+Official PostgreSQL, Prisma, systemd, and rsync semantics with claim-level links
+are in [backup and migration primary sources](../assets/backup-migration-primary-sources.md).
+
+The newly-clear decision and implementation frontier is:
+
+- [Decide recovery objectives, backup retention, and restore scope](39-decide-recovery-objectives.md)
+- [Implement coherent backup sets and verified restore](40-verified-backup-restore.md)
+- [Make the Project workspace lifecycle recoverable](41-recoverable-project-workspace-lifecycle.md)
+- [Establish migration compatibility and failed-migration recovery](14-migration-compatibility-and-recovery.md)
+
+Persistent-path provisioning and release-tree isolation remain owned by
+[Provision a runnable non-root Hub service](10-provision-nonroot-service.md)
+and [Make application releases atomic and rollbackable](12-atomic-releases-and-rollback.md).
+Full recovery/cutover procedures and drill evidence remain owned by
+[Write and drill production incident recovery runbooks](31-write-and-drill-incident-runbooks.md).
@@ -2,7 +2,7 @@

 Type: task
 Status: open
-Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38
+Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41

 ## Question

@@ -8,4 +8,6 @@ Status: open
 Make clean-host installation create or validate the dedicated service identity,
 home/cache/state/workspace paths and ownership, generate a complete production
 configuration, honor the configured bind host, and fail preflight before unit
-installation when any required value or prerequisite is invalid.
+installation when any required value or prerequisite is invalid. The persistent
+workspace root must be outside all release trees, and preflight must reject any
+path that overlaps rsync, release switching, rollback, or release pruning.
@@ -9,4 +9,5 @@ Blocked by: 10, 11
 Replace in-place rsync/build with deployment locking, immutable staged release
 directories, a revision manifest, atomic current/previous switching, bounded
 retention, and an explicit application rollback that preserves the live release
-on build, preflight, switch, or readiness failure.
+on build, preflight, switch, or readiness failure. Prove deploy, rollback, and
+release pruning cannot traverse or delete the persistent workspace root.
@@ -8,6 +8,8 @@ Blocked by: 06

 Define and enforce the application/schema compatibility rule for each release,
 require expand-and-contract sequencing for destructive changes, add migration
-status and production-like rehearsal gates, and provide an operator procedure
-for recovering a partially failed Prisma migration without pretending that an
-application rollback reverses the database.
+status/checksum and production-like rehearsal gates, and move schema mutation
+out of ordinary service start into an explicit release phase. Require a recent
+verified recovery checkpoint, bounded lock/runtime evidence, and an operator
+procedure for resolving or restoring a failed Prisma migration without
+pretending that an application rollback reverses the database.
@@ -2,7 +2,7 @@

 Type: task
 Status: open
-Blocked by: 06, 12, 13, 14, 23, 24, 25, 26, 27, 28, 29, 30
+Blocked by: 06, 12, 13, 14, 23, 24, 25, 26, 27, 28, 29, 30, 39, 40, 41

 ## Question

@@ -13,4 +13,6 @@ deploy/rollback, host restart during work, and suspected tenant exposure. Give
 each alert an owner and safe read-first commands, require confirmed/audited
 scope for destructive or replay actions, execute every runbook against
 production-shaped staging, and archive evidence that detection, mitigation,
-reconciliation, verification, escalation, and alert clearing all work.
+reconciliation, verification, escalation, and alert clearing all work. Include
+database/workspace backup failure, failed migration, full-host restore and
+cutover, duplicate/gap review, fallback, and achieved RPO/RTO evidence.
@@ -0,0 +1,14 @@
+# Decide recovery objectives, backup retention, and restore scope
+
+Type: grilling
+Status: open
+
+## Question
+
+For the initial SaaS service, decide measurable RPO/RTO targets, backup and
+restore-drill frequency, retention and deletion-in-backups policy, off-host
+copy/failure-domain requirements, logical export versus physical base-backup
+and continuous-WAL/PITR coverage, encryption and key custody, operator ownership,
+and whether the supported promise is full-service disaster recovery only or
+also Organization-selective export/restore. Record compliance-dependent items
+as explicit OPEN inputs rather than inventing defaults.
@@ -0,0 +1,16 @@
+# Implement coherent backup sets and verified restore
+
+Type: task
+Status: open
+Blocked by: 10, 12, 14, 19, 23, 24, 25, 26, 29, 39, 41
+
+## Question
+
+Implement the decided recovery policy as cataloged, checksummed, encrypted,
+off-host backup sets covering PostgreSQL logical exports plus physical base
+backup/continuous WAL or a proven managed equivalent, Project workspaces,
+compatible release/migration identity, configuration inventory, org-secret
+ciphertext, and separately recoverable key material. Establish a proven
+cross-store checkpoint, retention and alerting, partial-artifact cleanup,
+isolated restore with outbound integrations disabled, lifecycle reconciliation,
+critical SaaS journey verification, and measured RPO/RTO drills.
@@ -0,0 +1,14 @@
+# Make the Project workspace lifecycle recoverable
+
+Type: task
+Status: open
+Blocked by: 10, 26, 34
+
+## Question
+
+Move persistent workspaces outside every mutable release tree and make Project
+workspace allocation, database commit, archive/deletion, relocation, restore,
+and orphan/missing-path reconciliation explicit and crash-safe. Prove failed
+Project creation leaves no silent orphan, deployment/rollback/pruning cannot
+delete customer files, restored absolute paths are validated or safely rebased,
+and every inconsistency is observable rather than auto-hidden.