diff --git a/.scratch/saas-production-readiness/assets/production-observability-recovery.md b/.scratch/saas-production-readiness/assets/production-observability-recovery.md
new file mode 100644
index 0000000..7e8c922
--- /dev/null
+++ b/.scratch/saas-production-readiness/assets/production-observability-recovery.md
@@ -0,0 +1,245 @@
+# Production observability and incident-recovery audit
+
+Date: 2026-07-10
+
+Scope: the `hub` HTTP service, Feishu WebSocket ingress and outbound calls, PostgreSQL/Prisma, agent execution, the `cph` subprocess, project workspaces, and the packaged deployment path. This is a release-gate audit, not a proposal to change product semantics. Where the Lean contract is open, this report asks for an operational decision rather than inventing one.
+
+## Verdict
+
+**No-go for unattended SaaS production deployment.** The service exposes a liveness-shaped endpoint, emits some structured component logs, and persists useful run/audit fields, but it cannot currently answer the minimum production questions:
+
+1. Can this instance accept a real request *now* (database reachable, Feishu listener authenticated, required runtime executable and workspace usable)?
+2. Which tenant, request, Feishu event, run, subprocess, and outbound message participated in one failed user action?
+3. How many requests/runs/messages are failing, stuck, delayed, retried, or duplicated?
+4. Which condition should page an operator, and which documented recovery action is safe?
+5. After a restart, deploy, rollback, or dependency outage, what evidence proves that work was neither lost nor silently acknowledged?
+
+Three fault injections demonstrate that the current `/api/healthz` and logging surface can be green while the service is already unable to serve its core paths:
+
+- After startup, cutting PostgreSQL sockets left two consecutive `/api/healthz` requests at HTTP 200.
+- An invalid Feishu app ID was rejected by the installed Feishu SDK while `/api/healthz` remained HTTP 200.
+- With PostgreSQL cut, `/api/me` returned HTTP 400 `bad_request`, exposed a raw Prisma invocation in the response, produced no request-scoped application error log, and only emitted two unscoped Prisma error lines.
+
+Together with the already-confirmed lifecycle and deployment failures, this means green process state, green health checks, and even a visible user error do not prove that the service or its durable workflows are healthy.
+
+## Evidence and positive signals already present
+
+The starting point is not empty. These are useful foundations to retain:
+
+- Fastify's built-in HTTP logging records request/response facts with an internal request ID through the server logger ([`hub/src/server.ts`](../../../hub/src/server.ts)).
+- The same Fastify logger is threaded into the Feishu trigger/runtime, while Prisma emits its configured error/warn events (and optional query events) ([`hub/src/server.ts`](../../../hub/src/server.ts), [`hub/src/db.ts`](../../../hub/src/db.ts)).
+- `AgentRun` persists status, timestamps, error/result fields, token/cost fields, and links to project/session; `ProjectAgentLock` persists a heartbeat; `AuditEntry` and `AgentMessage` provide potential durable trace anchors ([`hub/prisma/schema.prisma`](../../../hub/prisma/schema.prisma)).
+- Agent execution emits intermediate activity to the streaming Feishu card, and lock heartbeat activity is attempted during execution ([`hub/src/agent/runner.ts`](../../../hub/src/agent/runner.ts), [`hub/src/feishu/trigger.ts`](../../../hub/src/feishu/trigger.ts)).
+- Deployment is managed by systemd, so restart policy, exit status, and journal collection have an existing operational host surface ([`hub/deploy/cph-hub.service`](../../../hub/deploy/cph-hub.service)).
+- Database migrations are explicit Prisma artifacts rather than ad-hoc startup DDL ([`hub/prisma/migrations`](../../../hub/prisma/migrations)).
+- Existing tests cover many individual authorization, trigger, lock, queue, runner, and API behaviors. They are valuable regression evidence, although they are not yet a deploy/recovery acceptance suite.
+
+These signals are fragmented rather than joined into an operational contract. Several writes are best-effort or swallowed, identifiers are not propagated end-to-end, there is no metrics/alert surface, and the health endpoint does not test dependencies.
+
+## Failure-to-signal matrix
+
+| Surface | Failure or degraded condition | Current externally visible signal | What is missing / why the signal is unsafe | Minimum production signal and recovery evidence |
+| --- | --- | --- | --- | --- |
+| HTTP | PostgreSQL becomes unreachable after startup | Affected request fails; `/api/healthz` stays 200 | Health is process liveness only. `/api/me` misclassifies the dependency failure as client `400 bad_request`, leaks raw Prisma call text, and creates no request-scoped error record. An LB/operator cannot distinguish bad input from platform outage. | Separate liveness and readiness. Readiness performs a bounded DB probe and returns non-2xx on failure. Unexpected exceptions map to sanitized 5xx with a stable error code and `request_id`; structured log contains the same ID and normalized Prisma category. Alert on readiness failure and 5xx/error-code rate. |
+| HTTP | Handler latency, event-loop stall, exhausted connections, overload | Access log only after a response is completed | No in-flight count, latency histogram, timeout counter, saturation signal, or request correlation. A hanging request may emit no timely evidence. | Request count, duration, in-flight requests, timeout/cancellation count, response class, and event-loop/runtime/process saturation. Bound request duration and log start/end/error with one `request_id`. |
+| Feishu ingress | Invalid/revoked app credentials; WebSocket connect/auth failure | SDK rejects/logs asynchronously; process and `/api/healthz` remain green | Startup success is reported before listener readiness. No listener state, last-connected/last-event timestamp, reconnect counter, or readiness dependency. The installed SDK already exposes `onReady`, `onError`, `onReconnecting`, `onReconnected`, `getConnectionStatus`, and `close`, but the Hub does not retain or wire the `WSClient`. systemd can perpetually supervise a process that cannot receive work. | Listener state machine (`starting/connected/degraded/stopped`), last successful connection/event, reconnect/auth-failure counters, bounded startup readiness gate, and alert on auth failure or prolonged disconnect. Credential contents must never be logged. |
+| Feishu ingress | Duplicate delivery, handler failure after receipt, process restart with queued/batched work | Event receipt/dedup row and selected warning logs | The run-lifecycle audit proved receipt-before-success and in-memory batching/queue loss. There is no durable processing state/lease, retry age, dead-letter view, or alert for accepted-but-unstarted work. | Durable ingress states and timestamps, attempts, last normalized error, event/chat/project correlation, queue depth/oldest age, dead-letter count. Recovery drill proves restart resumes or deliberately redrives exactly the intended work. See [run lifecycle audit](./run-lifecycle-integrity-audit.md). |
+| Feishu outbound | API timeout/ambiguous success, rate limit, invalid credentials, permanent card patch/send failure | Some helpers log a warning; several message helpers catch failure and return `null` | No idempotent delivery record, delivery state, attempt count, provider request/message ID, retry delay, terminal failure metric, or user-visible reconciliation path. Fault injection proved a run can be marked completed although nothing was delivered. | Durable outbox/delivery state; idempotency key; provider message/request ID; attempts, latency and normalized provider error; queue depth/oldest age/dead-letter signals; recovery command to retry or reconcile safely. |
+| PostgreSQL | Startup unavailable | Process startup rejects | This is a useful fail-fast signal, but it is not joined to a deploy gate with bounded logs and rollback criteria. | Installer/supervisor waits for explicit readiness, captures failure reason, and aborts/rolls back without replacing last-known-good service. |
+| PostgreSQL | Runtime outage, pool saturation, slow queries, migration incompatibility | Raw Prisma errors; request symptoms | No readiness failure, pool/saturation metric, query latency/timeout signal, migration compatibility assertion, or tenant/request/run correlation. Raw client internals can cross the HTTP boundary. | DB probe latency/result, Prisma error code/category counters, bounded query timing for critical operations, connection/pool saturation if available, schema migration identifier, sanitized API mapping, and DB outage/runbook drill. Prisma documents structured client error types/codes and configurable event-based logging; use those instead of returning raw exception strings ([Prisma error reference](https://www.prisma.io/docs/orm/reference/error-reference), [Prisma logging](https://www.prisma.io/docs/orm/prisma-client/observability-and-logging/logging)). |
+| Agent run | SDK/model error, credentials invalid, provider throttling, malformed stream | Run may become `FAILED`; the card may show an error and trigger finalization may warn | The installed SDK emits API retry, rate-limit, tool-progress, runtime-init/MCP-status, provider request ID, duration, and error-category messages, but `runner.ts` handles only stream/assistant/user/result messages and drops the other lifecycle signals. Provider/model/attempt/latency/normalized error are not consistently available as metrics or correlated logs. Some persistence and audit operations are best-effort. | Run start/finish transition event with `run_id`, `session_id`, tenant/project, provider/model, status, duration, turns, tokens/cost, normalized error and retryability; capture the SDK's bounded retry/rate-limit/tool/runtime signals; aggregate run outcome/latency/stuck gauges without high-cardinality IDs as metric labels. |
+| Agent run | No result/hang, missed heartbeats, process death, terminal DB write failure | Lock heartbeat may stop; restart sweep mutates some state | No wall-clock deadline/watchdog, stale-lock/stale-active-run metric, readiness impact, page, or documented reconciliation. Lifecycle fault injections proved active run without lock, terminal run with lock, and timeout misclassification are possible. | Deadline and durable lease semantics, gauges for active/waiting/stale runs and locks, oldest age, transition-failure counter, reconciliation command and drill. Startup recovery must be transactional and evidence-producing. See [run lifecycle audit](./run-lifecycle-integrity-audit.md). |
+| `cph` subprocess | Missing binary, permission failure, timeout, non-zero exit, malformed output | In production, `cph_check`/`cph_build` grant the Claude SDK's Bash tool; tool results are forwarded only to the in-memory streaming card. The direct `agent/cph.ts` wrapper captures exit/stdout/stderr but is used only by its integration tests, not by the production runner. | There is no production-owned `cph` invocation boundary, startup/readiness validation of the exact deployed artifact, version/build identity, persisted operation result, duration metric, bounded/redacted stderr contract, or reliable distinction between spawn failure and domain diagnostics. | Readiness checks executable path/version and required filesystem permissions without mutating domain state. Introduce an observable production invocation boundary or capture the SDK Bash/tool lifecycle with operation, run/project correlation, duration, exit category and bounded redacted diagnostic; metrics count timeout/spawn/non-zero/protocol errors. Deployment evidence pins and smoke-tests the artifact. |
+| Workspace | Missing/unwritable root, wrong ownership, disk/inode exhaustion, symlink/root escape, partial agent writes | Error appears only when a command/file operation happens | No readiness check, free-space/inode monitoring, workspace mutation journal/commit marker, or recovery procedure. Security audit already proved containment failures; lifecycle audit proved a run can fail after workspace side effects. | Bounded non-destructive readiness check for configured root, owner/mode, read/write/rename/fsync as required, free space and inode signals, per-project mutation correlation, containment validation, backup/restore and partial-write reconciliation drill. See [tenant/auth security audit](./tenant-auth-security-audit.md) and [run lifecycle audit](./run-lifecycle-integrity-audit.md). |
+| Deploy | Bad env/service user/artifact, failed migration/build, unhealthy new process | Shell/systemd exit in some cases; process may restart | Clean-host audit found missing required env/service identity/artifact and in-place, non-atomic release replacement. Health is liveness-only and there is no last-known-good rollback unit. | Immutable versioned release, preflight, migration compatibility gate, explicit readiness deadline, release/build/schema identity endpoint/log, automated rollback criteria, last-known-good artifact and tested rollback. See [clean-host deployment audit](./clean-host-deployment-audit.md). |
+| Deploy | SIGTERM/restart during active work | systemd stops/restarts the process | No graceful drain/stop readiness, bounded shutdown, durable queue resume, or post-restart reconciliation proof. In-memory queues are lost. | On termination: readiness false, stop ingress, drain or checkpoint durable work, enforce timeout, emit shutdown summary. Recovery drill sends work across restart and verifies a single durable outcome and delivery. systemd supports explicit notification/readiness/watchdog protocols, but the current unit does not implement one ([systemd service types](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html), [`sd_notify`](https://www.freedesktop.org/software/systemd/man/latest/sd_notify.html)). |
+
+## Logs and end-to-end traceability
+
+### Current gaps
+
+- Fastify generates an internal request ID for its access logs, but it is not returned to the client as a support handle or propagated into application errors, Prisma diagnostics, Feishu work, runs, or subprocess calls.
+- Feishu `event_id`, provider request/message IDs, `chat_id`, `project_id`, `organization_id`, `run_id`, `session_id`, and subprocess invocation are not propagated as one causally connected context.
+- The DB-cut experiment produced raw Prisma diagnostics without an application error record that identified route, request, response class, or tenant context.
+- Several critical failure paths catch and ignore persistence errors or only warn, including audit/message persistence and some card/lock finalization paths. This creates absence of evidence precisely when durable state is least trustworthy.
+- The production runner discards SDK `api_retry`, `rate_limit_event`, `tool_progress`, runtime initialization/MCP status, and related system messages even though the installed SDK exposes the fields needed to diagnose provider and tool degradation.
+- Exception strings can cross the API boundary. Besides leaking implementation details, this makes the client-visible error vocabulary unstable and impossible to alert on reliably.
+- Logs have no documented retention, rotation, ingestion, access-control, redaction, or tenant-support lookup contract. Journald availability alone is not a log strategy.
+
+### Minimum structured event envelope
+
+Every boundary and durable state transition should emit one structured event with a stable `event_name`, `timestamp`, `severity`, `service`, `environment`, `release`, and the applicable correlation fields:
+
+`request_id`, `feishu_event_id`, `organization_id`, `project_id`, `chat_id`, `actor_user_id`, `run_id`, `session_id`, `delivery_id`, `provider_request_id`, and `cph_invocation_id`.
+
+Raw prompt/message/file contents, credentials, cookies, authorization headers, provider tokens, full Prisma invocation text, and unrestricted stdout/stderr must not be logged. Tenant/project/user identifiers are operationally necessary but access-controlled personal/customer metadata; metric labels must not use high-cardinality IDs. OpenTelemetry's HTTP semantic conventions specify the stable `http.server.request.duration` instrument, an opt-in `http.server.active_requests` instrument, and low-cardinality route/error guidance ([HTTP metrics semantic conventions](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/)).
+
+At minimum, events are required for:
+
+- request accepted/completed/failed/canceled;
+- Feishu listener connected/disconnected/auth-failed/reconnecting;
+- ingress received/deduplicated/queued/claimed/started/retried/dead-lettered;
+- run and lock state transitions, transition-write failures, heartbeat stale detection, and recovery decisions;
+- provider/agent start/result/error with normalized category;
+- `cph` spawn/result/timeout/protocol failure;
+- workspace preflight and mutation/commit/reconciliation decisions;
+- outbound delivery queued/attempted/confirmed/retried/dead-lettered;
+- deploy, schema migration, readiness transition, drain, shutdown, startup recovery, and rollback.
+
+Errors must be allowed to fail the affected operation rather than disappear. If a non-critical telemetry sink is unavailable, that telemetry failure itself needs a bounded local signal/counter; durable business audit records are not optional telemetry and must follow their explicit transactional contract.
+
+## Health, readiness, and startup contract
+
+The service needs three distinct concepts:
+
+1. **Liveness**: the process/event loop is responsive. This endpoint must be cheap and must not depend on remote services. The present `/api/healthz` can serve this purpose if it is named/documented as liveness only.
+2. **Readiness**: the instance can accept the traffic assigned to it. A bounded probe must include at least PostgreSQL connectivity/schema compatibility, Feishu listener authenticated/connected state for ingress-serving instances, required runtime configuration, exact `cph` artifact availability/version, and workspace root access/capacity. A failed dependency makes readiness non-2xx without killing the process and losing diagnostics.
+3. **Startup/deploy acceptance**: migrations/preflight complete, listener reaches ready state within a deadline, one authenticated HTTP/database smoke path and one controlled runtime/workspace smoke path pass, and the deployed release/schema identity matches the intended version. Only then is traffic enabled and the previous release eligible for cleanup.
+
+Readiness responses should expose only component names, stable state/error codes, and timestamps—not secrets or raw exceptions. Full diagnostic context belongs in correlated restricted logs. The deployment audit must decide whether Feishu and HTTP run in one instance role; if roles are split later, readiness dependencies become role-specific rather than globally optional.
+
+## Minimum metrics and alerts
+
+Exact SLO values are product/operations decisions and are currently unspecified. The release nevertheless needs the following bounded-cardinality instruments and a reviewed initial threshold for each page/ticket alert.
+
+### Traffic and failures
+
+- HTTP request count, duration, active requests, status/error-code class, timeouts/cancellations.
+- Feishu listener connected state, reconnect/auth failures, time since last successful connection/event.
+- Ingress queue/receipt counts by durable state, processing attempts, oldest pending age, dead-letter count.
+- Outbound delivery attempts/results/latency, retry/dead-letter count, oldest pending age, provider rate-limit responses.
+- Prisma operation failures by normalized code/category and bounded critical-query duration; DB readiness result/latency.
+
+### Workflow integrity and capacity
+
+- Runs started and terminal outcomes, run duration, active/waiting/stale count and oldest age.
+- Locks held/stale/orphaned and transition/reconciliation failures.
+- Agent/provider failures by provider/model/error category; turns/tokens/cost as operational/business counters with tenant detail kept out of metric labels.
+- `cph` invocation count/duration/result category/timeouts.
+- Workspace free bytes/inodes, preflight failures, write/commit/reconciliation failures.
+- Process uptime/restarts, event-loop delay, memory, CPU, file descriptors, unhandled rejection/uncaught exception, graceful shutdown outcome.
+- Release/build/schema identity as an info metric or startup event, never as an unbounded label set.
+
+### Required release-time alerts
+
+Before production traffic, alerts and routing must exist for:
+
+- instance readiness false beyond the rollout/startup grace period;
+- Feishu auth failure or prolonged listener disconnect;
+- sustained server-side HTTP errors or request timeouts;
+- database readiness failure / critical Prisma failure spike;
+- oldest ingress/outbound item, active run, or lock exceeds its decided deadline;
+- dead-letter count non-zero or durable state-transition/reconciliation failure;
+- agent/provider or `cph` failure/timeout spike;
+- workspace disk/inodes below headroom threshold or workspace preflight fails;
+- restart loop, failed deployment, failed migration, or rollback;
+- telemetry pipeline blind (expected heartbeat/scrape/log ingestion absent).
+
+Each alert needs an owner, severity, user impact statement, dashboard/log query, first safe diagnostic command, mitigation, escalation criterion, and recovery verification. A page without a runbook is not a production control.
+
+## Incident recovery runbooks required before release
+
+At minimum, checked-in runbooks must cover these scenarios. Commands must default to read-only inspection; destructive/redrive/force-release actions require explicit confirmation, identify tenant/project/run scope, emit an audit record, and be idempotent or document why they are not.
+
+1. **PostgreSQL unavailable/degraded** — identify connection vs credential vs migration vs capacity failure; stop accepting work through readiness; restore service; reconcile active runs, locks, ingress receipts, and outbox; verify no raw errors escape HTTP.
+2. **Feishu listener disconnected/credentials revoked** — distinguish auth from transient network/reconnect; restore credentials safely; verify last event and controlled inbound round trip; inspect ingress gap and redrive rules.
+3. **Ingress backlog/dead letter** — list oldest/attempts/error by tenant-safe scope; decide retry vs reject; prove dedup/idempotency; verify one resulting run.
+4. **Stuck/orphaned run or lock** — inspect run/session/lock/provider/workspace/audit/outbox together; use lease/reconciliation operation rather than direct row deletion; prove terminal state and next queued work behavior.
+5. **Outbound delivery failure** — reconcile provider message ID and durable outbox; retry with idempotency key; confirm user-visible delivery without duplicating a card/message.
+6. **Agent/provider or `cph` outage** — identify provider credential/rate-limit/runtime/protocol failure; halt or queue new runs; test a canary; resume and reconcile affected work.
+7. **Workspace capacity/corruption/containment event** — quarantine affected project, snapshot evidence, restore/rollback documented unit, verify ownership/containment and repository state before resuming.
+8. **Failed migration/deployment** — preserve old release, collect preflight/migration/readiness diagnostics, follow compatibility-aware application/database rollback, and verify release/schema identity plus canaries.
+9. **Process/host restart during work** — drain when planned; after unplanned restart run durable reconciliation; verify receipts, runs, locks, workspace effects and deliveries converge exactly once at the business level.
+10. **Suspected tenant data exposure** — disable affected path/tenant access without deleting evidence, preserve correlated audit/log records, rotate exposed credentials, determine affected org/project/users, and follow an explicitly owned incident-notification decision process.
+
+## Minimum release evidence
+
+Production readiness is not proved by a green unit suite alone. A release candidate needs an archived, attributable evidence bundle containing all of the following:
+
+### Build and supply identity
+
+- immutable application and `cph` artifact digests, source commit, build timestamp/toolchain, dependency lockfiles and vulnerability policy result;
+- deployed configuration manifest listing required keys and instance role, with secret values redacted;
+- migration set/schema identity and a compatibility statement for both forward deploy and rollback;
+- systemd unit and installer/preflight version used on a clean host.
+
+### Automated gates
+
+- typecheck, complete unit/integration suite, migration-from-supported-version tests, and Lean `lake build` when `spec/` changes;
+- clean-host install test with the real packaged artifacts and declared service user/directories;
+- negative startup tests for missing/invalid env, DB unavailable, invalid Feishu credentials, missing/wrong `cph`, and unwritable/full workspace;
+- readiness tests proving each dependency failure independently makes readiness fail while liveness remains diagnostically available;
+- API error-mapping tests proving unexpected dependency exceptions return sanitized 5xx/stable codes and create a correlated error log;
+- lifecycle fault tests from the run-lifecycle audit, including process kill/restart, DB transition failures, duplicate ingress, ambiguous outbound completion, timeout, and concurrent trigger/session creation;
+- tenant isolation/security regression suite, symlink/root containment tests, and dependency audit policy gate.
+
+### Staging drills against production-shaped dependencies
+
+- authenticated Feishu inbound → durable receipt → one run → workspace effect → durable terminal state → one outbound confirmation, searchable end-to-end by a correlation ID;
+- PostgreSQL socket cut/restoration while serving: readiness changes, alert fires, requests return sanitized server errors, state reconciles, alert clears;
+- Feishu credential rejection/disconnect/reconnect: readiness/alert and controlled recovery are visible;
+- SIGTERM and forced process kill during queued, active, and outbound-pending work: restart reconciliation proves no silent loss and no unintended duplicate user effect;
+- `cph` timeout/non-zero/malformed response, provider throttle/error, workspace disk/permission failure, and outbound rate-limit/permanent failure drills;
+- failed deploy/readiness and rollback drill on a clean host, followed by migration/data compatibility and canary verification;
+- backup restore of representative database and workspace data, with measured recovery point/time and tenant isolation checks.
+
+### Operations acceptance
+
+- dashboards expose all release metrics above and show release/schema identity;
+- every required alert is injected once in staging, reaches the named owner, links to its checked-in runbook, and clears only after verified recovery;
+- log search can pivot from request/event to org/project/run/session/delivery without message content or secrets; retention and access controls are documented and tested;
+- on-call ownership, escalation, incident severity, maintenance/rollback authority, backup ownership, and recovery objectives are explicitly decided;
+- a signed release checklist records who reviewed each piece of evidence and where immutable test/drill outputs are stored.
+
+## Prioritized remediation / release gates
+
+### P0 — before any external production traffic
+
+1. Implement distinct bounded liveness, readiness, and startup/deploy acceptance contracts; make deployment traffic activation depend on readiness.
+2. Install centralized unexpected-error handling that returns sanitized stable 5xx responses and emits request-correlated structured errors. Never return raw Prisma/SDK/process exception text.
+3. Make Feishu connection/auth state observable and readiness-relevant; make listener startup rejection fail the startup gate.
+4. Implement the durable ingress, transactional run/lock/session, durable outbox, workspace commit/reconciliation, and durable audit/history work already opened by the lifecycle audit. Observability cannot compensate for acknowledged work that has no recoverable state.
+5. Add end-to-end correlation and stable structured transition events; remove silent catches from critical paths.
+6. Add the minimum metrics, alerts, dashboards, and checked-in recovery runbooks, then execute the fault/restart/rollback drills.
+7. Replace in-place deployment with the clean-host audit's immutable release/readiness/rollback contract and prove it on a clean host.
+8. Close the critical tenant/agent/workspace/Feishu authorization and dependency findings from the security audit before declaring readiness.
+
+### P1 — required for routine operations immediately after the P0 architecture is in place
+
+- Define initial SLOs/error budgets and tune alert thresholds using staging/load evidence.
+- Add capacity/load/soak tests for DB connections, event-loop/process resources, ingress/outbound backlog, agent concurrency, provider limits, workspace disk/inodes, and log/metric pipelines.
+- Exercise backup/restore, credential rotation, tenant-support trace lookup, and incident evidence preservation.
+- Define data retention/deletion/access-control policy for logs, metrics, traces, audits, messages, receipts, deliveries, runs, and workspaces.
+
+## Decision points that remain OPEN
+
+The code/spec does not currently authorize choosing these product/operations semantics implicitly:
+
+- readiness dependency roles (combined HTTP+Feishu process versus independently deployable roles);
+- maximum acceptable ingress, run, lock, and outbound ages; retry budgets; terminal/dead-letter semantics;
+- workspace mutation commit/rollback unit and what a user should observe after partial agent work;
+- audit durability/availability trade-off and exact event taxonomy;
+- SLOs, recovery point objective, recovery time objective, retention, regional/data-residency constraints, and incident-notification ownership;
+- whether deploy migration compatibility must support application rollback across one or more releases.
+
+These decisions should be recorded in the upstream contract/ADR layer before implementation when they affect domain-visible behavior. Operational mechanisms that only expose already-defined behavior can be implemented and tested without inventing domain semantics.
+
+## Source notes
+
+Primary local evidence:
+
+- [`hub/src/server.ts`](../../../hub/src/server.ts), [`hub/src/admin/errors.ts`](../../../hub/src/admin/errors.ts) — process startup, health/listener/restart behavior, and HTTP exception-to-response mapping.
+- [`hub/src/db.ts`](../../../hub/src/db.ts) — current Prisma event/log configuration.
+- [`hub/src/feishu/client.ts`](../../../hub/src/feishu/client.ts), [`hub/src/feishu/trigger.ts`](../../../hub/src/feishu/trigger.ts), [`hub/src/feishu/messageBatcher.ts`](../../../hub/src/feishu/messageBatcher.ts), [`hub/src/feishu/triggerQueue.ts`](../../../hub/src/feishu/triggerQueue.ts) — ingress/listener, batching, dedup, queue, run and outbound behavior.
+- [`hub/src/agent/runner.ts`](../../../hub/src/agent/runner.ts), [`hub/src/agent/roleTools.ts`](../../../hub/src/agent/roleTools.ts), [`hub/src/agent/cph.ts`](../../../hub/src/agent/cph.ts), [`hub/src/lock.ts`](../../../hub/src/lock.ts), [`hub/src/audit.ts`](../../../hub/src/audit.ts) — production SDK/Bash tool routing, the legacy direct cph wrapper, persistence, heartbeat/lock, and audit behavior.
+- [`hub/prisma/schema.prisma`](../../../hub/prisma/schema.prisma) — durable state and available correlation fields.
+- [`hub/deploy`](../../../hub/deploy) — installer/systemd/deploy behavior.
+- [Clean-host deployment audit](./clean-host-deployment-audit.md), [tenant/auth security audit](./tenant-auth-security-audit.md), and [run-lifecycle integrity audit](./run-lifecycle-integrity-audit.md) — prior deterministic audits and their recorded commands/results.
+
+Official references used only for interface semantics and implementation guidance:
+
+- [Prisma Client error reference](https://www.prisma.io/docs/orm/reference/error-reference) and [Prisma logging](https://www.prisma.io/docs/orm/prisma-client/observability-and-logging/logging).
+- [OpenTelemetry HTTP metric semantic conventions](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/).
+- [systemd service types/readiness](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html) and [`sd_notify`](https://www.freedesktop.org/software/systemd/man/latest/sd_notify.html).
+
+No secondary production-readiness checklist was treated as authority. Recommendations are derived from the local failure evidence and the primary interfaces the service actually uses.
diff --git a/.scratch/saas-production-readiness/issues/04-define-observability-recovery.md b/.scratch/saas-production-readiness/issues/04-define-observability-recovery.md
index 28efe03..323ed10 100644
--- a/.scratch/saas-production-readiness/issues/04-define-observability-recovery.md
+++ b/.scratch/saas-production-readiness/issues/04-define-observability-recovery.md
@@ -1,7 +1,7 @@
 # Define production observability and incident recovery
 
 Type: research
-Status: open
+Status: resolved
 Blocked by: 01, 02, 03
 
 ## Question
@@ -10,3 +10,38 @@ Which logs, metrics, readiness signals, audit correlations, alerts, and operator
 recovery actions are missing for a production operator to detect and diagnose
 failures across HTTP, Feishu, PostgreSQL, agent execution, cph builds, and
 workspace storage?
+
+## Answer
+
+The Hub has useful fragments—Fastify access logs, Prisma error/warn output,
+durable AgentRun facts, selected Feishu warnings, explicit migrations, and a
+systemd journal—but no joined operational contract. It has no metrics or alert
+surface, no end-to-end request/event/run/delivery/release correlation, no
+checked-in recovery runbooks, and many critical writes or sends are silently
+best-effort.
+
+Three process-level fault probes confirmed the release gate is actively
+misleading: cutting every PostgreSQL socket left `/api/healthz` at 200 twice;
+an SDK-rejected Feishu app ID also left health at 200; and a runtime database
+outage made `/api/me` return a raw Prisma invocation as `400 bad_request` with
+no request-scoped exception log. The current deploy check can therefore accept
+an instance unable to serve either database or Feishu work, while dependency
+failures evade normal 5xx alerting.
+
+The full failure-to-signal matrix, liveness/readiness/startup contract,
+correlation envelope, metric and alert inventory, required runbooks, executed
+probe evidence, primary sources, and minimum release evidence are in the
+[production observability and incident-recovery audit](../assets/production-observability-recovery.md).
+
+The newly-clear implementation and decision frontier is:
+
+- [Make HTTP dependency failures typed and observable](28-typed-observable-http-errors.md)
+- [Decide the telemetry, SLO, retention, and alerting control plane](29-decide-telemetry-slos-alert-routing.md)
+- [Instrument correlated production runtime telemetry](30-instrument-correlated-runtime-telemetry.md)
+- [Write and drill production incident recovery runbooks](31-write-and-drill-incident-runbooks.md)
+
+[Add real readiness and an accurate service lifecycle](13-real-readiness-and-lifecycle.md)
+owns component probes, Feishu connection state and deploy gating. Durable
+ingress/run/outbound recovery stays with tickets 23–25, audit semantics with
+ticket 27, and backup/restore guarantees with ticket 06; this audit does not
+duplicate those implementations.
diff --git a/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md b/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md
index 238a959..db5ec8e 100644
--- a/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md
+++ b/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md
@@ -2,7 +2,7 @@
 
 Type: task
 Status: open
-Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
+Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31
 
 ## Question
 
diff --git a/.scratch/saas-production-readiness/issues/28-typed-observable-http-errors.md b/.scratch/saas-production-readiness/issues/28-typed-observable-http-errors.md
new file mode 100644
index 0000000..f55a510
--- /dev/null
+++ b/.scratch/saas-production-readiness/issues/28-typed-observable-http-errors.md
@@ -0,0 +1,13 @@
+# Make HTTP dependency failures typed and observable
+
+Type: task
+Status: open
+
+## Question
+
+Replace message-substring HTTP error classification with typed validation,
+domain, authorization, Prisma/dependency, and unexpected-error boundaries.
+Return only sanitized stable client codes, preserve true 5xx/availability
+semantics, emit one request-scoped structured error with a support-visible
+request ID, redact secrets and raw internals, and add a real database-outage
+regression proving Prisma failures can never become client 4xx responses.
diff --git a/.scratch/saas-production-readiness/issues/29-decide-telemetry-slos-alert-routing.md b/.scratch/saas-production-readiness/issues/29-decide-telemetry-slos-alert-routing.md
new file mode 100644
index 0000000..414e554
--- /dev/null
+++ b/.scratch/saas-production-readiness/issues/29-decide-telemetry-slos-alert-routing.md
@@ -0,0 +1,14 @@
+# Decide the telemetry, SLO, retention, and alerting control plane
+
+Type: grilling
+Status: open
+
+## Question
+
+For the accepted initial single-host service, decide the log/metric/trace
+collection and storage path, instance roles, release and correlation identity,
+redaction/access/retention policy, initial service-level indicators and
+objectives, error budgets, alert thresholds, on-call destinations, escalation
+ownership, and telemetry-blind detection. Prefer the smallest operable control
+plane, but record every domain-visible availability or retry assumption in the
+appropriate ADR/spec rather than hiding it in dashboard configuration.
diff --git a/.scratch/saas-production-readiness/issues/30-instrument-correlated-runtime-telemetry.md b/.scratch/saas-production-readiness/issues/30-instrument-correlated-runtime-telemetry.md
new file mode 100644
index 0000000..467c87b
--- /dev/null
+++ b/.scratch/saas-production-readiness/issues/30-instrument-correlated-runtime-telemetry.md
@@ -0,0 +1,15 @@
+# Instrument correlated production runtime telemetry
+
+Type: task
+Status: open
+Blocked by: 13, 23, 24, 25, 27, 28, 29
+
+## Question
+
+Implement the decided telemetry control plane with one redacted structured
+context from HTTP/Feishu ingress through durable work, run/session/lock,
+provider and SDK retry/rate-limit/tool events, cph/workspace operations,
+outbound delivery, audit, startup/drain/recovery, and release/schema identity.
+Expose bounded-cardinality metrics, dashboards and alert rules; eliminate
+silent critical-path failures; and prove logs can pivot end-to-end while metric
+labels contain no tenant/user/run cardinality or secret/message content.
diff --git a/.scratch/saas-production-readiness/issues/31-write-and-drill-incident-runbooks.md b/.scratch/saas-production-readiness/issues/31-write-and-drill-incident-runbooks.md
new file mode 100644
index 0000000..49b4ba2
--- /dev/null
+++ b/.scratch/saas-production-readiness/issues/31-write-and-drill-incident-runbooks.md
@@ -0,0 +1,16 @@
+# Write and drill production incident recovery runbooks
+
+Type: task
+Status: open
+Blocked by: 06, 12, 13, 14, 23, 24, 25, 26, 27, 28, 29, 30
+
+## Question
+
+Check in operator runbooks for PostgreSQL outage/migration failure, Feishu
+disconnect and credential rejection, ingress/outbound backlog or dead letter,
+stuck run/lock, provider or cph failure, workspace capacity/corruption,
+deploy/rollback, host restart during work, and suspected tenant exposure. Give
+each alert an owner and safe read-first commands, require confirmed/audited
+scope for destructive or replay actions, execute every runbook against
+production-shaped staging, and archive evidence that detection, mitigation,
+reconciliation, verification, escalation, and alert clearing all work.
diff --git a/.scratch/saas-production-readiness/map.md b/.scratch/saas-production-readiness/map.md
index e3955c7..bd6e651 100644
--- a/.scratch/saas-production-readiness/map.md
+++ b/.scratch/saas-production-readiness/map.md
@@ -31,6 +31,7 @@ recovery procedures, and no known critical security or data-integrity gaps.
 - [Audit the clean-host deployment and rollback contract](issues/01-audit-clean-host-deployment.md) — keep the accepted single-host topology, but replace the incomplete in-place updater with a provisioned, immutable, readiness-gated and rollbackable release contract.
 - [Audit tenant, authentication, and request security boundaries](issues/02-audit-tenant-auth-security.md) — retain the working org-admin/application authorization core, but production is blocked on agent/MCP isolation, execution-time object binding, org-scoped encrypted credentials, tenant status, browser sessions, and clean dependency audits.
 - [Audit run lifecycle and data integrity under failure](issues/03-audit-run-lifecycle-integrity.md) — retain the database-enforced project-lock uniqueness and normal happy path, but replace receipt-before-success, process-local queues, independent run/session/lock effects, silent delivery/audit loss, and partial startup reset with durable, recoverable state machines; explicitly decide failed-run workspace semantics.
+- [Define production observability and incident recovery](issues/04-define-observability-recovery.md) — keep the existing Fastify/Prisma/systemd signal fragments, but production requires dependency-aware readiness, typed and correlated errors, durable-work telemetry, an explicit SLO/retention/alerting control plane, and recovery runbooks proven by fault drills.
 
 ## Fog