feat(hub): external capability registry for PDF/ASR transforms (ADR-0027) (#5)

Co-authored-by: Hong Jiarong <me@jrhim.com>
Co-committed-by: Hong Jiarong <me@jrhim.com>

docs/adr/0027-external-capability-registry.md

@@ -0,0 +1,152 @@
+# ADR 0027: External Capability Registry
+
+## Status
+
+Accepted.
+
+## Context
+
+ADR-0026 introduced the `UsageFact` ledger with a `kind = external_capability`
+fact and a `capabilityId` field, but deferred the capability registry itself.
+Two concrete needs now force the issue:
+
+- **PDF→Markdown bundle** conversion (and, imminently, audio/video→text)
+  must run as a side effect of an `AgentRun`, bill in non-token units
+  (pages, seconds), use a different provider than the model loop, and report
+  cost through a different channel. It is not a sub-run (ADR-0026 rejected
+  that) and not a model-provider call (it does not speak the Anthropic/
+  OpenRouter protocol).
+- The Agent already has a Bash tool. Without a first-class capability seam,
+  the path of least resistance is for the agent to shell out to ad-hoc
+  scripts that embed API keys, write to arbitrary paths, and report nothing
+  to the ledger. That is exactly the unattributed, uncontained external
+  consumption ADR-0022/0026 exist to prevent.
+
+The model-provider connection (`OrganizationProviderConnection`, ADR-0024)
+is the wrong seam for these services:
+
+- Its payload schema (`baseUrl` + `authToken` + `anthropicApiKey`) and
+  readiness probe (`/v1/models?supported_parameters=tools`) are specific to
+  OpenRouter/Anthropic. MinerU, Whisper, and future OCR/ASR services have
+  different auth shapes (an API token, optionally a project id) and no
+  `/v1/models` endpoint.
+- Its uniqueness key is `(organizationId, providerId)` where `providerId`
+  is an OpenRouter-style model-routing id. A capability provider id
+  (`mineru`) names a *service*, not a model.
+- Coupling capability credentials into the model-provider table would force
+  every capability's auth shape through `ProviderSecretPayloadV1` and every
+  readiness probe through `probeOpenRouterCredential`.
+
+The Feishu Application Connection (`OrganizationFeishuApplicationConnection`)
+is the right structural precedent: it reuses the ADR-0024 envelope (KEK →
+DEK → AES-256-GCM, AAD-bound to purpose/org/connection/version) but has its
+own connection table, its own payload schema, its own readiness probe, and
+its own per-org uniqueness. Capability connections follow the same pattern.
+
+## Decision
+
+### External Capability
+
+An **External Capability** is a platform-registered, org-enabled document or
+media transform service invoked as a side effect of an `AgentRun`. It is
+identified by a stable `capabilityId` (e.g. `pdf_to_md_bundle`,
+`audio_video_to_text`). A capability:
+
+- Has an **input kind** (PDF, image, audio, video, …) and an **output
+  contract** (markdown bundle with extracted images, transcript text, …).
+- Bills in **non-token units** (pages, audio-seconds) recorded on a
+  `UsageFact` with `kind = external_capability`, or in tokens when the
+  backing service reports them.
+- Writes its output **into the invoking run's workspace** (ADR-0018
+  `AgentSurface` — no escapes).
+- Is invoked through a **capability adapter** in Hub, never by the Agent
+  shelling out with embedded credentials.
+
+### Capability Connection
+
+Credentials for a capability live in an **`OrganizationCapabilityConnection`**,
+structurally identical to the Feishu Application Connection:
+
+- Belongs to exactly one Organization.
+- Unique by `(organizationId, capabilityId)`.
+- `DRAFT` / `ACTIVE` / `DISABLED`; resolution accepts only `ACTIVE` with a
+  valid active secret version.
+- Secret material is an immutable, AAD-bound, KEK-wrapped envelope version
+  (`CapabilityCredentialVersion`), reusing the ADR-0024 encryption
+  machinery with `purpose = "capability"`.
+- Its payload schema is capability-specific (`CapabilitySecretPayloadV1`:
+  `baseUrl`, `apiToken`, optional `projectId`). New capability types extend
+  the payload, not the connection table.
+- A capability-specific **readiness probe** validates the credential before
+  activation (e.g. MinerU: a trivial authenticated GET). The probe is
+  injectable, matching the Feishu/provider pattern, so tests never hit the
+  network.
+
+### Capability Adapter
+
+The adapter is the seam between the Agent and the external service. It:
+
+- Resolves the org's active capability connection (fail-closed, no
+  process-global fallback — ADR-0024).
+- Accepts a workspace-relative input path and an output directory.
+- Calls the backing service (MinerU, Whisper, …) via an injectable
+  `Client` interface so the real HTTP client is swappable and mockable.
+- Writes the produced markdown + image assets into the run's workspace.
+- Writes one `UsageFact` (or more, if the service reports per-stage
+  consumption) with `kind = external_capability`, `capabilityId`,
+  `provider` (the service id), `quantity + unit` (pages / seconds), and
+  `costUsd + costSource = provider_reported` when the service reports cost.
+
+### Registry
+
+The platform maintains a **registry** of known capabilities: their id,
+input kind, output contract, metering unit, and adapter. This is
+code-level registration (like `ToolRegistry`), not a database table — a
+capability is available to an Organization only when (a) the platform
+knows the adapter and (b) the Organization has an `ACTIVE` connection for
+it. Both gates are required.
+
+### What is NOT in this ADR
+
+- The capability invocation is **not** a first-class persisted record
+  (`CapabilityInvocation` table) in this ADR. The `UsageFact` row with
+  `capabilityId` + `correlationId` is the durable trace. If we later need
+  a richer invocation log (retries, partial output, multi-stage status),
+  that is a follow-up; for now the fact is enough.
+- **Pricebook** remains deferred (ADR-0026). Capability facts use
+  `provider_reported` when the service returns cost; otherwise `unknown`.
+- **Org-scoped enable/disable policy** beyond connection status is
+  deferred. An org with an `ACTIVE` connection has the capability; one
+  without does not. A finer "enabled but no credential" toggle is not
+  needed yet.
+- **Agent-facing tool exposure** (how the Agent discovers and calls the
+  capability — MCP tool, Bash wrapper, or built-in) is an implementation
+  detail of the adapter wiring, not a contract concern. The contract pins
+  that the Agent never receives the capability credential.
+
+## Consequences
+
+- Adding a new external capability (e.g. `image_ocr`) is: register an
+  adapter, add a `capabilityId` constant, optionally extend the secret
+  payload — no schema change to `UsageFact` or `AgentRun`.
+- The model-provider connection table stays focused on model routing;
+  capability credentials do not pollute its payload or readiness probe.
+- Three connection types now share the ADR-0024 envelope: model-provider,
+  Feishu application, and capability. Each has its own table, payload
+  schema, and probe, but the same encryption, rotation, and resolver
+  boundary.
+- The Agent's Bash tool remains available, but the intended path for
+  document/media transforms is the capability adapter. Whether to narrow
+  Bash for capability-shaped tasks is an operational policy decision,
+  not a contract one.
+- Tests prove: workspace containment of capability output, fail-closed
+  credential resolution, `UsageFact` attribution with non-token metering,
+  and that the Agent process never receives the capability credential.
+
+## Deferred
+
+- `CapabilityInvocation` as a first-class durable record (status, retries,
+  partial output) — currently the `UsageFact` row is the only trace.
+- Pricebook derivation for capability costs (ADR-0026 deferred).
+- Org-scoped capability enable/disable policy finer than connection status.
+- Agent-facing tool discovery (MCP vs built-in) for capabilities.