2.4 KiB
Define production observability and incident recovery
Type: research Status: resolved Blocked by: 01, 02, 03
Question
Which logs, metrics, readiness signals, audit correlations, alerts, and operator recovery actions are missing for a production operator to detect and diagnose failures across HTTP, Feishu, PostgreSQL, agent execution, cph builds, and workspace storage?
Answer
The Hub has useful fragments—Fastify access logs, Prisma error/warn output, durable AgentRun facts, selected Feishu warnings, explicit migrations, and a systemd journal—but no joined operational contract. It has no metrics or alert surface, no end-to-end request/event/run/delivery/release correlation, no checked-in recovery runbooks, and many critical writes or sends are silently best-effort.
Three process-level fault probes confirmed the release gate is actively
misleading: cutting every PostgreSQL socket left /api/healthz at 200 twice;
an SDK-rejected Feishu app ID also left health at 200; and a runtime database
outage made /api/me return a raw Prisma invocation as 400 bad_request with
no request-scoped exception log. The current deploy check can therefore accept
an instance unable to serve either database or Feishu work, while dependency
failures evade normal 5xx alerting.
The full failure-to-signal matrix, liveness/readiness/startup contract, correlation envelope, metric and alert inventory, required runbooks, executed probe evidence, primary sources, and minimum release evidence are in the production observability and incident-recovery audit.
The newly-clear implementation and decision frontier is:
- Make HTTP dependency failures typed and observable
- Decide the telemetry, SLO, retention, and alerting control plane
- Instrument correlated production runtime telemetry
- Write and drill production incident recovery runbooks
Add real readiness and an accurate service lifecycle owns component probes, Feishu connection state and deploy gating. Durable ingress/run/outbound recovery stays with tickets 23–25, audit semantics with ticket 27, and backup/restore guarantees with ticket 06; this audit does not duplicate those implementations.