docs: resolve observability recovery audit

2026-07-10 11:10:32 +08:00
parent ec630b9151
commit 16329df357
8 changed files with 341 additions and 2 deletions
@@ -1,7 +1,7 @@
 # Define production observability and incident recovery

 Type: research
-Status: open
+Status: resolved
 Blocked by: 01, 02, 03

 ## Question
@@ -10,3 +10,38 @@ Which logs, metrics, readiness signals, audit correlations, alerts, and operator
 recovery actions are missing for a production operator to detect and diagnose
 failures across HTTP, Feishu, PostgreSQL, agent execution, cph builds, and
 workspace storage?
+
+## Answer
+
+The Hub has useful fragments—Fastify access logs, Prisma error/warn output,
+durable AgentRun facts, selected Feishu warnings, explicit migrations, and a
+systemd journal—but no joined operational contract. It has no metrics or alert
+surface, no end-to-end request/event/run/delivery/release correlation, no
+checked-in recovery runbooks, and many critical writes or sends are silently
+best-effort.
+
+Three process-level fault probes confirmed the release gate is actively
+misleading: cutting every PostgreSQL socket left `/api/healthz` at 200 twice;
+an SDK-rejected Feishu app ID also left health at 200; and a runtime database
+outage made `/api/me` return a raw Prisma invocation as `400 bad_request` with
+no request-scoped exception log. The current deploy check can therefore accept
+an instance unable to serve either database or Feishu work, while dependency
+failures evade normal 5xx alerting.
+
+The full failure-to-signal matrix, liveness/readiness/startup contract,
+correlation envelope, metric and alert inventory, required runbooks, executed
+probe evidence, primary sources, and minimum release evidence are in the
+[production observability and incident-recovery audit](../assets/production-observability-recovery.md).
+
+The newly-clear implementation and decision frontier is:
+
+- [Make HTTP dependency failures typed and observable](28-typed-observable-http-errors.md)
+- [Decide the telemetry, SLO, retention, and alerting control plane](29-decide-telemetry-slos-alert-routing.md)
+- [Instrument correlated production runtime telemetry](30-instrument-correlated-runtime-telemetry.md)
+- [Write and drill production incident recovery runbooks](31-write-and-drill-incident-runbooks.md)
+
+[Add real readiness and an accurate service lifecycle](13-real-readiness-and-lifecycle.md)
+owns component probes, Feishu connection state and deploy gating. Durable
+ingress/run/outbound recovery stays with tickets 23–25, audit semantics with
+ticket 27, and backup/restore guarantees with ticket 06; this audit does not
+duplicate those implementations.
@@ -2,7 +2,7 @@

 Type: task
 Status: open
-Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
+Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31

 ## Question

@@ -0,0 +1,13 @@
+# Make HTTP dependency failures typed and observable
+
+Type: task
+Status: open
+
+## Question
+
+Replace message-substring HTTP error classification with typed validation,
+domain, authorization, Prisma/dependency, and unexpected-error boundaries.
+Return only sanitized stable client codes, preserve true 5xx/availability
+semantics, emit one request-scoped structured error with a support-visible
+request ID, redact secrets and raw internals, and add a real database-outage
+regression proving Prisma failures can never become client 4xx responses.
@@ -0,0 +1,14 @@
+# Decide the telemetry, SLO, retention, and alerting control plane
+
+Type: grilling
+Status: open
+
+## Question
+
+For the accepted initial single-host service, decide the log/metric/trace
+collection and storage path, instance roles, release and correlation identity,
+redaction/access/retention policy, initial service-level indicators and
+objectives, error budgets, alert thresholds, on-call destinations, escalation
+ownership, and telemetry-blind detection. Prefer the smallest operable control
+plane, but record every domain-visible availability or retry assumption in the
+appropriate ADR/spec rather than hiding it in dashboard configuration.
@@ -0,0 +1,15 @@
+# Instrument correlated production runtime telemetry
+
+Type: task
+Status: open
+Blocked by: 13, 23, 24, 25, 27, 28, 29
+
+## Question
+
+Implement the decided telemetry control plane with one redacted structured
+context from HTTP/Feishu ingress through durable work, run/session/lock,
+provider and SDK retry/rate-limit/tool events, cph/workspace operations,
+outbound delivery, audit, startup/drain/recovery, and release/schema identity.
+Expose bounded-cardinality metrics, dashboards and alert rules; eliminate
+silent critical-path failures; and prove logs can pivot end-to-end while metric
+labels contain no tenant/user/run cardinality or secret/message content.
@@ -0,0 +1,16 @@
+# Write and drill production incident recovery runbooks
+
+Type: task
+Status: open
+Blocked by: 06, 12, 13, 14, 23, 24, 25, 26, 27, 28, 29, 30
+
+## Question
+
+Check in operator runbooks for PostgreSQL outage/migration failure, Feishu
+disconnect and credential rejection, ingress/outbound backlog or dead letter,
+stuck run/lock, provider or cph failure, workspace capacity/corruption,
+deploy/rollback, host restart during work, and suspected tenant exposure. Give
+each alert an owner and safe read-first commands, require confirmed/audited
+scope for destructive or replay actions, execute every runbook against
+production-shaped staging, and archive evidence that detection, mitigation,
+reconciliation, verification, escalation, and alert clearing all work.