docs: resolve clean-host deployment audit

.scratch/saas-production-readiness/assets/clean-host-deployment-audit.md

@@ -0,0 +1,200 @@
+# Clean-host deployment and rollback audit
+
+## Verdict
+
+The current `hub/deploy` path is **not a production deployment mechanism**.
+It can update a manually-prepared host in the happy path, but a clean supported
+host cannot become a runnable Hub from the documented inputs, and an upgrade
+failure can leave the live application tree partially replaced with no
+application rollback path.
+
+The accepted initial topology remains viable: one Linux/systemd host,
+PostgreSQL, Feishu WebSocket ingress, and a non-root Hub service. The blocker is
+the release contract and its verification, not a need to replace that topology.
+
+## Executed evidence
+
+### Static gates
+
+- `bash -n hub/deploy/install_service.sh` and
+  `bash -n hub/deploy/deploy_platform.sh` pass.
+- ShellCheck reports only informational SC2029 findings for intentional
+  client-side expansion of remote paths. It does not find a shell syntax defect.
+- A source-to-installer comparison finds six `requireEnv` settings in
+  `hub/src/server.ts`, but the generated production environment omits
+  `HUB_PROJECT_WORKSPACE_ROOT` and `HUB_SESSION_SECRET`.
+
+### Seeded configuration cannot boot the built server
+
+The installer was run against a temporary `BASE`, with the real built Hub as
+`HUB_DIR`. It exited successfully after creating `platform.env`. The generated
+file was then used as the production configuration while supplying the model and
+Feishu values that its message asks the operator to fill. The built server
+exited with status 1:
+
+```text
+[hub] missing required env: HUB_PROJECT_WORKSPACE_ROOT
+```
+
+After that is supplied, `HUB_SESSION_SECRET` is the next required omission.
+`HUB_PUBLIC_BASE_URL` is also absent; its code fallback is localhost, which is
+not a valid public OAuth callback for a production deployment.
+
+### The service identity is referenced but not provisioned
+
+The installer was executed with `SERVICE_USER=definitely-missing-cph-user` and
+mocked `install`/`systemctl` boundaries. It returned success and rendered:
+
+```text
+User=definitely-missing-cph-user
+WorkingDirectory=.../hub
+EnvironmentFile=.../.secrets/platform.env
+```
+
+There is no `getent`, `id`, `useradd`, or equivalent validation/provisioning
+path. It also creates no service home, cache, state, or project-workspace
+directory. This contradicts ADR-0018's implemented deployment invariant that
+the Hub runs as a dedicated unprivileged service user. A genuinely clean host
+will fail at systemd user resolution; a manually-created user can still fail
+later when the agent SDK, `cph` cache, or project workspace needs a writable
+path.
+
+### A failed build mutates the live tree before failure
+
+`deploy_platform.sh` was run with mocked SSH and rsync boundaries. The first
+remote build was forced to exit 42. The observed order was:
+
+```text
+ssh:mkdir -p '/srv/curriculum-project-hub/hub'
+rsync:live-tree-mutated
+ssh:cd '/srv/curriculum-project-hub/hub' && npm ci && npm run build
+```
+
+The deploy command correctly propagated status 42, but it had already applied
+`rsync --delete` to the one live directory. There is no staged release,
+immutable release identity, current/previous pointer, deployment lock, or
+rollback command. `npm ci` also replaces that live directory's dependency tree
+before the new build has passed.
+
+## Blocking gaps
+
+### 1. Incomplete clean-host bootstrap
+
+`install_service.sh` writes a unit containing `User=cph-hub` without creating or
+validating that account. It does not create owned state/cache/workspace paths,
+and its generated environment omits required settings. On its first invocation
+it deliberately exits after seeding the environment, after which
+`deploy_platform.sh` proceeds to restart a unit that has not yet been installed.
+The documented workflow is therefore neither one-shot nor complete.
+
+### 2. The runtime product is not fully delivered
+
+The deploy uploads only `hub/`. The target prerequisite list does not include a
+preinstalled `cph`, and no release step builds or installs `crates/cph-cli`.
+The Hub starts without checking it: the first agent `cph check` or `cph build`
+returns exit 127 while `/api/healthz` remains green. Bubblewrap is asserted by
+the unit, but Node/npm, curl, PostgreSQL connectivity, service-user namespace
+support, writable storage, and `cph --version` are not checked as one preflight.
+
+### 3. Releases are in-place and have no application rollback
+
+Source, dependencies, build output, and the running service all share one
+directory. A build failure, interrupted rsync, or unsuccessful restart can
+leave a mixed tree. No release manifest records the git revision, Node version,
+dependency lock digest, migration set, or `cph` version that was deployed.
+Concurrent deploys are not serialized.
+
+### 4. The deployment health check is only liveness
+
+`/api/healthz` returns a constant success response after the HTTP server starts.
+It does not verify database access, migration state, workspace writability,
+`cph`, the agent sandbox, or Feishu listener startup. The listener's start
+promise is not awaited or reflected in readiness. A release can therefore be
+declared successful while its defining workflows are unavailable.
+
+The service unit also claims graceful shutdown, but `server.ts` installs no
+`SIGTERM`/`SIGINT` handler and never calls `app.close()` or closes the Feishu
+listener. Node's default SIGTERM handling terminates the process; Fastify's
+documented graceful path begins only when `fastify.close()` is invoked. The
+full data-integrity consequence belongs to the run-lifecycle audit, but the
+current deployment contract must not claim draining behavior it does not have.
+
+References:
+
+- [Fastify shutdown hooks](https://fastify.dev/docs/latest/Reference/Hooks/)
+- [Node.js signal events](https://nodejs.org/api/process.html#signal-events)
+
+### 5. Application rollback and schema migration are not coordinated
+
+The unit runs `prisma migrate deploy` immediately before every start. Current
+migration history includes column drops, index replacement, and data
+deduplication; there is no declaration that the previous application remains
+compatible after each migration. Switching application code back after a
+successful migration is therefore not generally safe.
+
+Prisma documents that `migrate deploy` applies pending migrations but does not
+detect schema drift, and a partially failed migration requires explicit
+operator recovery. Its recommended zero-downtime direction is an
+expand-and-contract sequence. The repository has no pre-deploy migration
+status gate, production-data rehearsal, backup checkpoint, expand/contract
+rule, or failed-migration runbook.
+
+References:
+
+- [Prisma production migration behavior](https://www.prisma.io/docs/orm/prisma-migrate/workflows/development-and-production)
+- [Prisma failed-migration recovery](https://docs.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing)
+- [Prisma expand-and-contract migrations](https://docs.prisma.io/docs/guides/database/data-migration)
+
+### 6. Network binding configuration is misleading
+
+The installer writes `HOST=127.0.0.1`, but `server.ts` ignores it and binds
+`0.0.0.0`. An operator following the generated configuration can unintentionally
+expose the Hub directly instead of binding it behind the intended TLS reverse
+proxy. Public URL validation and proxy topology are not part of deployment
+preflight.
+
+## Required release contract
+
+A production-capable implementation on the accepted topology should establish
+these invariants:
+
+1. **Provisioned identity and storage.** Installation creates or validates a
+   non-login `cph-hub` user and explicit persistent state, cache, home, and
+   workspace paths with tested ownership.
+2. **Complete validated configuration.** A preflight validates every required
+   secret/value, the public HTTPS URL, bind address, PostgreSQL connectivity,
+   writable storage, `bwrap`, and the exact compatible `cph` binary before the
+   live service is touched.
+3. **Immutable staged releases.** Each revision is built in a new release
+   directory, identified by revision and manifest. The live service points to a
+   `current` symlink only after build and preflight pass. Deploys are locked.
+4. **Complete product artifact.** The release carries both the Hub and the
+   matching `cph` executable; readiness checks the binary/version rather than
+   deferring failure to an agent run.
+5. **Real readiness.** Liveness remains cheap, while readiness proves the
+   dependencies needed to accept work. Deployment success waits for readiness,
+   not merely an open HTTP port.
+6. **Application rollback.** A failed switch returns `current` to the previous
+   release and verifies readiness. Database changes are never silently reversed.
+7. **Migration compatibility.** Every release states whether its migration set
+   is backward-compatible with the previous application. Future destructive
+   changes use expand/contract releases; failed migrations follow an explicit
+   recovery runbook tied to backup evidence.
+8. **Observable lifecycle.** Release revision, migration status, readiness
+   failures, restart/drain outcomes, and rollback results are present in command
+   output and service logs.
+
+## Verification route
+
+The implementation is not complete until an automated Linux test drives these
+scenarios with the real scripts and a production-like PostgreSQL instance:
+
+- first install on a host without the service user or directories;
+- incomplete configuration fails before unit installation/restart;
+- staged build failure leaves the current release byte-for-byte unchanged;
+- successful deploy reports revision and passes readiness;
+- failed new release switches back to the previous application;
+- missing/wrong `cph`, unwritable workspace, unavailable database, and failed
+  migration each produce a distinct non-zero deployment failure;
+- repeated installation/deployment is idempotent;
+- concurrent deployments serialize or one fails clearly.

.scratch/saas-production-readiness/issues/01-audit-clean-host-deployment.md

@@ -1,7 +1,7 @@
 # Audit the clean-host deployment and rollback contract
 
 Type: research
-Status: open
+Status: resolved
 
 ## Question
 
@@ -9,3 +9,25 @@ What concrete defects prevent a clean, supported Linux host from installing,
 starting, health-checking, upgrading, and rolling back the current Hub without
 manual repository knowledge or unsafe partial deployment?
 
+## Answer
+
+The current path is a happy-path updater for a manually prepared host, not a
+production deployment mechanism. The executed audit found an unrunnable seeded
+configuration, an unprovisioned service identity and storage, a missing `cph`
+artifact, in-place mutation before build success, liveness-only verification,
+no coordinated application/schema rollback, and a configured bind host that the
+server ignores.
+
+The complete evidence, operational consequences, required release invariants,
+and verification scenarios are recorded in the
+[clean-host deployment and rollback audit](../assets/clean-host-deployment-audit.md).
+
+The accepted single-host topology remains suitable. The route forward is now
+explicit in these implementation tickets:
+
+- [Provision a runnable non-root Hub service](10-provision-nonroot-service.md)
+- [Package and verify cph with every Hub release](11-package-cph-with-release.md)
+- [Make application releases atomic and rollbackable](12-atomic-releases-and-rollback.md)
+- [Add real readiness and an accurate service lifecycle](13-real-readiness-and-lifecycle.md)
+- [Establish migration compatibility and failed-migration recovery](14-migration-compatibility-and-recovery.md)
+- [Test clean-host deployment and rollback on Linux](15-test-clean-host-deployments-on-linux.md)

.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md

@@ -2,7 +2,7 @@
 
 Type: task
 Status: open
-Blocked by: 01, 02, 03, 04, 05, 06, 07, 09
+Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15
 
 ## Question
 

.scratch/saas-production-readiness/issues/10-provision-nonroot-service.md

@@ -0,0 +1,11 @@
+# Provision a runnable non-root Hub service
+
+Type: task
+Status: open
+
+## Question
+
+Make clean-host installation create or validate the dedicated service identity,
+home/cache/state/workspace paths and ownership, generate a complete production
+configuration, honor the configured bind host, and fail preflight before unit
+installation when any required value or prerequisite is invalid.

.scratch/saas-production-readiness/issues/11-package-cph-with-release.md

@@ -0,0 +1,10 @@
+# Package and verify cph with every Hub release
+
+Type: task
+Status: open
+
+## Question
+
+Deliver the compatible `cph` executable as part of the same immutable release
+as the Hub, configure its exact path, and make deployment preflight reject a
+missing, non-executable, or version-incompatible binary before restart.

.scratch/saas-production-readiness/issues/12-atomic-releases-and-rollback.md

@@ -0,0 +1,12 @@
+# Make application releases atomic and rollbackable
+
+Type: task
+Status: open
+Blocked by: 10, 11
+
+## Question
+
+Replace in-place rsync/build with deployment locking, immutable staged release
+directories, a revision manifest, atomic current/previous switching, bounded
+retention, and an explicit application rollback that preserves the live release
+on build, preflight, switch, or readiness failure.

.scratch/saas-production-readiness/issues/13-real-readiness-and-lifecycle.md

@@ -0,0 +1,12 @@
+# Add real readiness and an accurate service lifecycle
+
+Type: task
+Status: open
+Blocked by: 03, 04, 10, 11, 12
+
+## Question
+
+Separate liveness from readiness, verify every dependency needed to accept Hub
+work, make deployment wait on readiness, and align systemd shutdown/restart
+claims with an implemented and observable Fastify, Feishu, Prisma, and in-flight
+run lifecycle.

.scratch/saas-production-readiness/issues/14-migration-compatibility-and-recovery.md

@@ -0,0 +1,13 @@
+# Establish migration compatibility and failed-migration recovery
+
+Type: task
+Status: open
+Blocked by: 06
+
+## Question
+
+Define and enforce the application/schema compatibility rule for each release,
+require expand-and-contract sequencing for destructive changes, add migration
+status and production-like rehearsal gates, and provide an operator procedure
+for recovering a partially failed Prisma migration without pretending that an
+application rollback reverses the database.

.scratch/saas-production-readiness/issues/15-test-clean-host-deployments-on-linux.md

@@ -0,0 +1,12 @@
+# Test clean-host deployment and rollback on Linux
+
+Type: task
+Status: open
+Blocked by: 10, 11, 12, 13, 14
+
+## Question
+
+Add an automated Linux production-like harness that executes the real install,
+deploy and rollback paths against PostgreSQL and proves first install,
+idempotence, failure atomicity, readiness failures, application rollback, and
+deployment serialization.

.scratch/saas-production-readiness/map.md

@@ -28,6 +28,8 @@ recovery procedures, and no known critical security or data-integrity gaps.
 
 ## Decisions so far
 
+- [Audit the clean-host deployment and rollback contract](issues/01-audit-clean-host-deployment.md) — keep the accepted single-host topology, but replace the incomplete in-place updater with a provisioned, immutable, readiness-gated and rollbackable release contract.
+
 ## Fog
 
 - The production topology after the initial single-host pilot (HA, horizontal
@@ -38,4 +40,3 @@ recovery procedures, and no known critical security or data-integrity gaps.
   initial safety and operability route is clear.
 - Regulatory retention, residency, and customer-specific compliance controls
   require product/legal inputs that the repository does not currently contain.
-