# Define production observability and incident recovery

Type: research
Status: resolved
Blocked by: 01, 02, 03

## Question

Which logs, metrics, readiness signals, audit correlations, alerts, and operator
recovery actions are missing for a production operator to detect and diagnose
failures across HTTP, Feishu, PostgreSQL, agent execution, cph builds, and
workspace storage?

## Answer

The Hub has useful fragments—Fastify access logs, Prisma error/warn output,
durable AgentRun facts, selected Feishu warnings, explicit migrations, and a
systemd journal—but no joined operational contract. It has no metrics or alert
surface, no end-to-end request/event/run/delivery/release correlation, no
checked-in recovery runbooks, and many critical writes or sends are silently
best-effort.

Three process-level fault probes confirmed the release gate is actively
misleading: cutting every PostgreSQL socket left `/api/healthz` at 200 twice;
an SDK-rejected Feishu app ID also left health at 200; and a runtime database
outage made `/api/me` return a raw Prisma invocation as `400 bad_request` with
no request-scoped exception log. The current deploy check can therefore accept
an instance unable to serve either database or Feishu work, while dependency
failures evade normal 5xx alerting.

The full failure-to-signal matrix, liveness/readiness/startup contract,
correlation envelope, metric and alert inventory, required runbooks, executed
probe evidence, primary sources, and minimum release evidence are in the
[production observability and incident-recovery audit](../assets/production-observability-recovery.md).

The newly-clear implementation and decision frontier is:

- [Make HTTP dependency failures typed and observable](28-typed-observable-http-errors.md)
- [Decide the telemetry, SLO, retention, and alerting control plane](29-decide-telemetry-slos-alert-routing.md)
- [Instrument correlated production runtime telemetry](30-instrument-correlated-runtime-telemetry.md)
- [Write and drill production incident recovery runbooks](31-write-and-drill-incident-runbooks.md)

[Add real readiness and an accurate service lifecycle](13-real-readiness-and-lifecycle.md)
owns component probes, Feishu connection state and deploy gating. Durable
ingress/run/outbound recovery stays with tickets 23–25, audit semantics with
ticket 27, and backup/restore guarantees with ticket 06; this audit does not
duplicate those implementations.
