diff --git a/.scratch/saas-production-readiness/assets/clean-host-deployment-audit.md b/.scratch/saas-production-readiness/assets/clean-host-deployment-audit.md new file mode 100644 index 0000000..1ebed18 --- /dev/null +++ b/.scratch/saas-production-readiness/assets/clean-host-deployment-audit.md @@ -0,0 +1,200 @@ +# Clean-host deployment and rollback audit + +## Verdict + +The current `hub/deploy` path is **not a production deployment mechanism**. +It can update a manually-prepared host in the happy path, but a clean supported +host cannot become a runnable Hub from the documented inputs, and an upgrade +failure can leave the live application tree partially replaced with no +application rollback path. + +The accepted initial topology remains viable: one Linux/systemd host, +PostgreSQL, Feishu WebSocket ingress, and a non-root Hub service. The blocker is +the release contract and its verification, not a need to replace that topology. + +## Executed evidence + +### Static gates + +- `bash -n hub/deploy/install_service.sh` and + `bash -n hub/deploy/deploy_platform.sh` pass. +- ShellCheck reports only informational SC2029 findings for intentional + client-side expansion of remote paths. It does not find a shell syntax defect. +- A source-to-installer comparison finds six `requireEnv` settings in + `hub/src/server.ts`, but the generated production environment omits + `HUB_PROJECT_WORKSPACE_ROOT` and `HUB_SESSION_SECRET`. + +### Seeded configuration cannot boot the built server + +The installer was run against a temporary `BASE`, with the real built Hub as +`HUB_DIR`. It exited successfully after creating `platform.env`. The generated +file was then used as the production configuration while supplying the model and +Feishu values that its message asks the operator to fill. The built server +exited with status 1: + +```text +[hub] missing required env: HUB_PROJECT_WORKSPACE_ROOT +``` + +After that is supplied, `HUB_SESSION_SECRET` is the next required omission. +`HUB_PUBLIC_BASE_URL` is also absent; its code fallback is localhost, which is +not a valid public OAuth callback for a production deployment. + +### The service identity is referenced but not provisioned + +The installer was executed with `SERVICE_USER=definitely-missing-cph-user` and +mocked `install`/`systemctl` boundaries. It returned success and rendered: + +```text +User=definitely-missing-cph-user +WorkingDirectory=.../hub +EnvironmentFile=.../.secrets/platform.env +``` + +There is no `getent`, `id`, `useradd`, or equivalent validation/provisioning +path. It also creates no service home, cache, state, or project-workspace +directory. This contradicts ADR-0018's implemented deployment invariant that +the Hub runs as a dedicated unprivileged service user. A genuinely clean host +will fail at systemd user resolution; a manually-created user can still fail +later when the agent SDK, `cph` cache, or project workspace needs a writable +path. + +### A failed build mutates the live tree before failure + +`deploy_platform.sh` was run with mocked SSH and rsync boundaries. The first +remote build was forced to exit 42. The observed order was: + +```text +ssh:mkdir -p '/srv/curriculum-project-hub/hub' +rsync:live-tree-mutated +ssh:cd '/srv/curriculum-project-hub/hub' && npm ci && npm run build +``` + +The deploy command correctly propagated status 42, but it had already applied +`rsync --delete` to the one live directory. There is no staged release, +immutable release identity, current/previous pointer, deployment lock, or +rollback command. `npm ci` also replaces that live directory's dependency tree +before the new build has passed. + +## Blocking gaps + +### 1. Incomplete clean-host bootstrap + +`install_service.sh` writes a unit containing `User=cph-hub` without creating or +validating that account. It does not create owned state/cache/workspace paths, +and its generated environment omits required settings. On its first invocation +it deliberately exits after seeding the environment, after which +`deploy_platform.sh` proceeds to restart a unit that has not yet been installed. +The documented workflow is therefore neither one-shot nor complete. + +### 2. The runtime product is not fully delivered + +The deploy uploads only `hub/`. The target prerequisite list does not include a +preinstalled `cph`, and no release step builds or installs `crates/cph-cli`. +The Hub starts without checking it: the first agent `cph check` or `cph build` +returns exit 127 while `/api/healthz` remains green. Bubblewrap is asserted by +the unit, but Node/npm, curl, PostgreSQL connectivity, service-user namespace +support, writable storage, and `cph --version` are not checked as one preflight. + +### 3. Releases are in-place and have no application rollback + +Source, dependencies, build output, and the running service all share one +directory. A build failure, interrupted rsync, or unsuccessful restart can +leave a mixed tree. No release manifest records the git revision, Node version, +dependency lock digest, migration set, or `cph` version that was deployed. +Concurrent deploys are not serialized. + +### 4. The deployment health check is only liveness + +`/api/healthz` returns a constant success response after the HTTP server starts. +It does not verify database access, migration state, workspace writability, +`cph`, the agent sandbox, or Feishu listener startup. The listener's start +promise is not awaited or reflected in readiness. A release can therefore be +declared successful while its defining workflows are unavailable. + +The service unit also claims graceful shutdown, but `server.ts` installs no +`SIGTERM`/`SIGINT` handler and never calls `app.close()` or closes the Feishu +listener. Node's default SIGTERM handling terminates the process; Fastify's +documented graceful path begins only when `fastify.close()` is invoked. The +full data-integrity consequence belongs to the run-lifecycle audit, but the +current deployment contract must not claim draining behavior it does not have. + +References: + +- [Fastify shutdown hooks](https://fastify.dev/docs/latest/Reference/Hooks/) +- [Node.js signal events](https://nodejs.org/api/process.html#signal-events) + +### 5. Application rollback and schema migration are not coordinated + +The unit runs `prisma migrate deploy` immediately before every start. Current +migration history includes column drops, index replacement, and data +deduplication; there is no declaration that the previous application remains +compatible after each migration. Switching application code back after a +successful migration is therefore not generally safe. + +Prisma documents that `migrate deploy` applies pending migrations but does not +detect schema drift, and a partially failed migration requires explicit +operator recovery. Its recommended zero-downtime direction is an +expand-and-contract sequence. The repository has no pre-deploy migration +status gate, production-data rehearsal, backup checkpoint, expand/contract +rule, or failed-migration runbook. + +References: + +- [Prisma production migration behavior](https://www.prisma.io/docs/orm/prisma-migrate/workflows/development-and-production) +- [Prisma failed-migration recovery](https://docs.prisma.io/docs/orm/prisma-migrate/workflows/patching-and-hotfixing) +- [Prisma expand-and-contract migrations](https://docs.prisma.io/docs/guides/database/data-migration) + +### 6. Network binding configuration is misleading + +The installer writes `HOST=127.0.0.1`, but `server.ts` ignores it and binds +`0.0.0.0`. An operator following the generated configuration can unintentionally +expose the Hub directly instead of binding it behind the intended TLS reverse +proxy. Public URL validation and proxy topology are not part of deployment +preflight. + +## Required release contract + +A production-capable implementation on the accepted topology should establish +these invariants: + +1. **Provisioned identity and storage.** Installation creates or validates a + non-login `cph-hub` user and explicit persistent state, cache, home, and + workspace paths with tested ownership. +2. **Complete validated configuration.** A preflight validates every required + secret/value, the public HTTPS URL, bind address, PostgreSQL connectivity, + writable storage, `bwrap`, and the exact compatible `cph` binary before the + live service is touched. +3. **Immutable staged releases.** Each revision is built in a new release + directory, identified by revision and manifest. The live service points to a + `current` symlink only after build and preflight pass. Deploys are locked. +4. **Complete product artifact.** The release carries both the Hub and the + matching `cph` executable; readiness checks the binary/version rather than + deferring failure to an agent run. +5. **Real readiness.** Liveness remains cheap, while readiness proves the + dependencies needed to accept work. Deployment success waits for readiness, + not merely an open HTTP port. +6. **Application rollback.** A failed switch returns `current` to the previous + release and verifies readiness. Database changes are never silently reversed. +7. **Migration compatibility.** Every release states whether its migration set + is backward-compatible with the previous application. Future destructive + changes use expand/contract releases; failed migrations follow an explicit + recovery runbook tied to backup evidence. +8. **Observable lifecycle.** Release revision, migration status, readiness + failures, restart/drain outcomes, and rollback results are present in command + output and service logs. + +## Verification route + +The implementation is not complete until an automated Linux test drives these +scenarios with the real scripts and a production-like PostgreSQL instance: + +- first install on a host without the service user or directories; +- incomplete configuration fails before unit installation/restart; +- staged build failure leaves the current release byte-for-byte unchanged; +- successful deploy reports revision and passes readiness; +- failed new release switches back to the previous application; +- missing/wrong `cph`, unwritable workspace, unavailable database, and failed + migration each produce a distinct non-zero deployment failure; +- repeated installation/deployment is idempotent; +- concurrent deployments serialize or one fails clearly. diff --git a/.scratch/saas-production-readiness/issues/01-audit-clean-host-deployment.md b/.scratch/saas-production-readiness/issues/01-audit-clean-host-deployment.md index 2d75919..4b744f1 100644 --- a/.scratch/saas-production-readiness/issues/01-audit-clean-host-deployment.md +++ b/.scratch/saas-production-readiness/issues/01-audit-clean-host-deployment.md @@ -1,7 +1,7 @@ # Audit the clean-host deployment and rollback contract Type: research -Status: open +Status: resolved ## Question @@ -9,3 +9,25 @@ What concrete defects prevent a clean, supported Linux host from installing, starting, health-checking, upgrading, and rolling back the current Hub without manual repository knowledge or unsafe partial deployment? +## Answer + +The current path is a happy-path updater for a manually prepared host, not a +production deployment mechanism. The executed audit found an unrunnable seeded +configuration, an unprovisioned service identity and storage, a missing `cph` +artifact, in-place mutation before build success, liveness-only verification, +no coordinated application/schema rollback, and a configured bind host that the +server ignores. + +The complete evidence, operational consequences, required release invariants, +and verification scenarios are recorded in the +[clean-host deployment and rollback audit](../assets/clean-host-deployment-audit.md). + +The accepted single-host topology remains suitable. The route forward is now +explicit in these implementation tickets: + +- [Provision a runnable non-root Hub service](10-provision-nonroot-service.md) +- [Package and verify cph with every Hub release](11-package-cph-with-release.md) +- [Make application releases atomic and rollbackable](12-atomic-releases-and-rollback.md) +- [Add real readiness and an accurate service lifecycle](13-real-readiness-and-lifecycle.md) +- [Establish migration compatibility and failed-migration recovery](14-migration-compatibility-and-recovery.md) +- [Test clean-host deployment and rollback on Linux](15-test-clean-host-deployments-on-linux.md) diff --git a/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md b/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md index 259940d..4dfe3cc 100644 --- a/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md +++ b/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md @@ -2,7 +2,7 @@ Type: task Status: open -Blocked by: 01, 02, 03, 04, 05, 06, 07, 09 +Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15 ## Question diff --git a/.scratch/saas-production-readiness/issues/10-provision-nonroot-service.md b/.scratch/saas-production-readiness/issues/10-provision-nonroot-service.md new file mode 100644 index 0000000..c5de681 --- /dev/null +++ b/.scratch/saas-production-readiness/issues/10-provision-nonroot-service.md @@ -0,0 +1,11 @@ +# Provision a runnable non-root Hub service + +Type: task +Status: open + +## Question + +Make clean-host installation create or validate the dedicated service identity, +home/cache/state/workspace paths and ownership, generate a complete production +configuration, honor the configured bind host, and fail preflight before unit +installation when any required value or prerequisite is invalid. diff --git a/.scratch/saas-production-readiness/issues/11-package-cph-with-release.md b/.scratch/saas-production-readiness/issues/11-package-cph-with-release.md new file mode 100644 index 0000000..4e57e67 --- /dev/null +++ b/.scratch/saas-production-readiness/issues/11-package-cph-with-release.md @@ -0,0 +1,10 @@ +# Package and verify cph with every Hub release + +Type: task +Status: open + +## Question + +Deliver the compatible `cph` executable as part of the same immutable release +as the Hub, configure its exact path, and make deployment preflight reject a +missing, non-executable, or version-incompatible binary before restart. diff --git a/.scratch/saas-production-readiness/issues/12-atomic-releases-and-rollback.md b/.scratch/saas-production-readiness/issues/12-atomic-releases-and-rollback.md new file mode 100644 index 0000000..b3323db --- /dev/null +++ b/.scratch/saas-production-readiness/issues/12-atomic-releases-and-rollback.md @@ -0,0 +1,12 @@ +# Make application releases atomic and rollbackable + +Type: task +Status: open +Blocked by: 10, 11 + +## Question + +Replace in-place rsync/build with deployment locking, immutable staged release +directories, a revision manifest, atomic current/previous switching, bounded +retention, and an explicit application rollback that preserves the live release +on build, preflight, switch, or readiness failure. diff --git a/.scratch/saas-production-readiness/issues/13-real-readiness-and-lifecycle.md b/.scratch/saas-production-readiness/issues/13-real-readiness-and-lifecycle.md new file mode 100644 index 0000000..86ec284 --- /dev/null +++ b/.scratch/saas-production-readiness/issues/13-real-readiness-and-lifecycle.md @@ -0,0 +1,12 @@ +# Add real readiness and an accurate service lifecycle + +Type: task +Status: open +Blocked by: 03, 04, 10, 11, 12 + +## Question + +Separate liveness from readiness, verify every dependency needed to accept Hub +work, make deployment wait on readiness, and align systemd shutdown/restart +claims with an implemented and observable Fastify, Feishu, Prisma, and in-flight +run lifecycle. diff --git a/.scratch/saas-production-readiness/issues/14-migration-compatibility-and-recovery.md b/.scratch/saas-production-readiness/issues/14-migration-compatibility-and-recovery.md new file mode 100644 index 0000000..ba427fe --- /dev/null +++ b/.scratch/saas-production-readiness/issues/14-migration-compatibility-and-recovery.md @@ -0,0 +1,13 @@ +# Establish migration compatibility and failed-migration recovery + +Type: task +Status: open +Blocked by: 06 + +## Question + +Define and enforce the application/schema compatibility rule for each release, +require expand-and-contract sequencing for destructive changes, add migration +status and production-like rehearsal gates, and provide an operator procedure +for recovering a partially failed Prisma migration without pretending that an +application rollback reverses the database. diff --git a/.scratch/saas-production-readiness/issues/15-test-clean-host-deployments-on-linux.md b/.scratch/saas-production-readiness/issues/15-test-clean-host-deployments-on-linux.md new file mode 100644 index 0000000..122c5ff --- /dev/null +++ b/.scratch/saas-production-readiness/issues/15-test-clean-host-deployments-on-linux.md @@ -0,0 +1,12 @@ +# Test clean-host deployment and rollback on Linux + +Type: task +Status: open +Blocked by: 10, 11, 12, 13, 14 + +## Question + +Add an automated Linux production-like harness that executes the real install, +deploy and rollback paths against PostgreSQL and proves first install, +idempotence, failure atomicity, readiness failures, application rollback, and +deployment serialization. diff --git a/.scratch/saas-production-readiness/map.md b/.scratch/saas-production-readiness/map.md index 3793d03..06a0bf1 100644 --- a/.scratch/saas-production-readiness/map.md +++ b/.scratch/saas-production-readiness/map.md @@ -28,6 +28,8 @@ recovery procedures, and no known critical security or data-integrity gaps. ## Decisions so far +- [Audit the clean-host deployment and rollback contract](issues/01-audit-clean-host-deployment.md) — keep the accepted single-host topology, but replace the incomplete in-place updater with a provisioned, immutable, readiness-gated and rollbackable release contract. + ## Fog - The production topology after the initial single-host pilot (HA, horizontal @@ -38,4 +40,3 @@ recovery procedures, and no known critical security or data-integrity gaps. initial safety and operability route is clear. - Regulatory retention, residency, and customer-specific compliance controls require product/legal inputs that the repository does not currently contain. -