forked from EduCraft/curriculum-project-hub
48 lines
2.4 KiB
Markdown
48 lines
2.4 KiB
Markdown
# Define production observability and incident recovery
|
||
|
||
Type: research
|
||
Status: resolved
|
||
Blocked by: 01, 02, 03
|
||
|
||
## Question
|
||
|
||
Which logs, metrics, readiness signals, audit correlations, alerts, and operator
|
||
recovery actions are missing for a production operator to detect and diagnose
|
||
failures across HTTP, Feishu, PostgreSQL, agent execution, cph builds, and
|
||
workspace storage?
|
||
|
||
## Answer
|
||
|
||
The Hub has useful fragments—Fastify access logs, Prisma error/warn output,
|
||
durable AgentRun facts, selected Feishu warnings, explicit migrations, and a
|
||
systemd journal—but no joined operational contract. It has no metrics or alert
|
||
surface, no end-to-end request/event/run/delivery/release correlation, no
|
||
checked-in recovery runbooks, and many critical writes or sends are silently
|
||
best-effort.
|
||
|
||
Three process-level fault probes confirmed the release gate is actively
|
||
misleading: cutting every PostgreSQL socket left `/api/healthz` at 200 twice;
|
||
an SDK-rejected Feishu app ID also left health at 200; and a runtime database
|
||
outage made `/api/me` return a raw Prisma invocation as `400 bad_request` with
|
||
no request-scoped exception log. The current deploy check can therefore accept
|
||
an instance unable to serve either database or Feishu work, while dependency
|
||
failures evade normal 5xx alerting.
|
||
|
||
The full failure-to-signal matrix, liveness/readiness/startup contract,
|
||
correlation envelope, metric and alert inventory, required runbooks, executed
|
||
probe evidence, primary sources, and minimum release evidence are in the
|
||
[production observability and incident-recovery audit](../assets/production-observability-recovery.md).
|
||
|
||
The newly-clear implementation and decision frontier is:
|
||
|
||
- [Make HTTP dependency failures typed and observable](28-typed-observable-http-errors.md)
|
||
- [Decide the telemetry, SLO, retention, and alerting control plane](29-decide-telemetry-slos-alert-routing.md)
|
||
- [Instrument correlated production runtime telemetry](30-instrument-correlated-runtime-telemetry.md)
|
||
- [Write and drill production incident recovery runbooks](31-write-and-drill-incident-runbooks.md)
|
||
|
||
[Add real readiness and an accurate service lifecycle](13-real-readiness-and-lifecycle.md)
|
||
owns component probes, Feishu connection state and deploy gating. Durable
|
||
ingress/run/outbound recovery stays with tickets 23–25, audit semantics with
|
||
ticket 27, and backup/restore guarantees with ticket 06; this audit does not
|
||
duplicate those implementations.
|