forked from EduCraft/curriculum-project-hub
feat(hub): usage fact ledger for run-scoped cost attribution (ADR-0026)
Replace the single-scalar cost model on AgentRun with an append-only UsageFact ledger. One AgentRun owns zero or more UsageFact rows; each records one billable consumption event (model completion, external capability, or tool proxy) with its own provider/model/tokens/quantity/ cost. AgentRun.costUsd/inputTokens/outputTokens become a derived rollup cache. This unblocks external capabilities (PDF->MD bundle, audio/video->text) that bill in non-token units (pages, seconds) through a different provider than the main agent loop, without per-capability schema changes or nested AgentRuns (which would pollute lock/admission/session semantics). Contract: - spec/Spec/System/Agent/Usage.lean pins UsageFact, UsageFactKind, CostSource and three invariants: append-only; belongs to one run, never holds a lock; missing cost != zero (ADR-0022). - ADR-0026 records the decision, the rejected nested-Run alternative, the rollup cache strategy, and the deferred capability registry / pricebook / post-hoc correction flows. Schema: - UsageFact model with indexes on (runId, occurredAt), (runId, kind), (provider, model, occurredAt), (capabilityId, occurredAt). - Migration backfills one synthetic model_completion fact per existing run with recorded cost/tokens (correlationId = runId marks backfill); truly unrecorded runs stay runsWithoutCost per ADR-0022. Write path (trigger finish): - Write the UsageFact first, then mirror it onto AgentRun as two separate statements (not one transaction). The fact is the truth so it goes first; the cache is derived so it goes second. A crash between them leaves the cache stale but the usage service re-reads facts directly, so this is recoverable; the reverse order would lose the truth. Separate statements also avoid an AgentRun row lock held across the insert's FK ShareLock, which deadlocked concurrent workspace teardown under the Organization->Project->AgentRun->UsageFact cascade. Read paths: - org/usage.ts aggregates from UsageFact, ignoring the AgentRun cache. - slash /usage buckets by (fact.provider, fact.model); a run with a main loop + an external call lands in two buckets. - session detail exposes usageFacts[] + costSource for future per-run cost-breakdown UI. Tests: - usage.test.ts: 6 integration tests pin fact aggregation, missing-cost- !=-zero, multi-fact-per-run, empty-run, project-level, empty-org. - trigger.test.ts: existing /usage assertion ($0.0023, openrouter / mock-model) passes on the fact path. - feishu-reactions mock prisma gains usageFact.create.
This commit is contained in:
@@ -0,0 +1,146 @@
|
||||
# ADR 0026: Usage Fact Ledger
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0022 pins the cost-attribution contract for the platform: token,
|
||||
provider-reported cost, run count, and duration are attributed by Organization,
|
||||
Project, Run, model, and Provider Connection; "missing provider cost remains
|
||||
unknown rather than zero." ADR-0021 scopes usage accounting as operational
|
||||
reporting, not payment collection (commercial billing stays deferred).
|
||||
|
||||
The implementation today stores this as **one scalar per `AgentRun`**:
|
||||
`inputTokens`, `outputTokens`, `costUsd?`, `costSource?`, written once from the
|
||||
Claude Agent SDK's `result.total_cost_usd` when the run finishes. This is
|
||||
adequate for a single provider-reported model loop, but it cannot represent:
|
||||
|
||||
- **External capability consumption** inside a run. PDF→Markdown bundle
|
||||
conversion, audio/video transcription, OCR and similar media transforms are
|
||||
not the main agent loop. They run as side effects of a run, may use a
|
||||
different provider/model, may bill in non-token units (pages, seconds,
|
||||
invocations), and may report cost through a different channel than the
|
||||
OpenRouter gateway. Today there is no row to write that cost to — it would
|
||||
either disappear or silently corrupt the run's single scalar.
|
||||
- **Multiple model calls within one run** (e.g. a sub-model invoked by a
|
||||
tool, a gateway-side reroute). The scalar collapses them into one number.
|
||||
- **Pricebook derivation** after the fact. With only a final USD figure and no
|
||||
`(provider, model, occurredAt, tokens)` fact, an operator cannot re-derive
|
||||
cost from a price table when the provider did not report it.
|
||||
|
||||
Treating each external call as a nested `AgentRun` was considered and
|
||||
rejected: `AgentRun` carries lock ownership (ADR-0002), session/provider/role
|
||||
binding (ADR-0017), admission/capacity semantics (ADR-0022), and the
|
||||
user-visible task boundary. External calls hold none of those. Making them
|
||||
`AgentRun`s would pollute run counts, admission, lock semantics, and session
|
||||
continuity, and would still not solve non-token metering.
|
||||
|
||||
## Decision
|
||||
|
||||
Introduce **`UsageFact`** as the single source of truth for billable
|
||||
consumption inside an `AgentRun`. An `AgentRun` owns zero or more
|
||||
append-only `UsageFact` rows; each row records one billable consumption event:
|
||||
|
||||
- `kind` — `model_completion` (the main agent loop) | `external_capability`
|
||||
| `tool_proxy`. The kind set is `OPEN`; new kinds must be surfaced, not
|
||||
silently folded into an existing one.
|
||||
- `provider` — e.g. `openrouter`, `mineru`, `openai_whisper`.
|
||||
- `model?`, `inputTokens?`, `outputTokens?` — token metering, optional
|
||||
because non-token capabilities have none.
|
||||
- `quantity?` + `unit?` — non-token metering (pages, audio_seconds,
|
||||
invocations), coexisting with tokens rather than replacing them.
|
||||
- `costUsd?` + `costSource` — `provider_reported` | `pricebook_derived` |
|
||||
`unknown`. `costUsd = null` means **unknown, not zero** (ADR-0022). When the
|
||||
provider reported a cost, `costSource = provider_reported` and that value
|
||||
wins. When only tokens are known, a later pricebook pass may derive
|
||||
`costUsd` with `costSource = pricebook_derived`. When neither is possible,
|
||||
`costSource = unknown` and `costUsd` stays null.
|
||||
- `occurredAt` — when the consumption happened; the pricebook derivation
|
||||
depends on this, not on `AgentRun.finishedAt`, because an external
|
||||
capability may complete before the run finishes.
|
||||
- `capabilityId?` — for `external_capability` facts, the registered capability
|
||||
id (e.g. `pdf_to_md_bundle`, `audio_video_to_text`).
|
||||
- `correlationId?` — external request id for reconciliation / idempotency;
|
||||
not part of the aggregation key.
|
||||
|
||||
### Invariants
|
||||
|
||||
1. **Append-only.** A `UsageFact` row is never updated or deleted. A cost
|
||||
correction is a new row; the old row stays. `onDelete: Cascade` exists only
|
||||
so a hard run delete (itself not a normal path) cleans up its facts.
|
||||
2. **Belongs to exactly one Run; never holds a lock.** A fact is a side-effect
|
||||
ledger of one run, not a sub-run. External capability calls obey the same
|
||||
boundary: their identity is `capabilityId + correlationId`, not `RunId`.
|
||||
3. **Missing cost ≠ zero.** Aggregation MUST NOT sum `null` `costUsd` as 0.
|
||||
A run whose facts all have `costUsd = null` is "cost unknown" — reported as
|
||||
`runsWithoutCost`, exactly as the pre-migration `costUsd = null` runs are
|
||||
today. A run with at least one `costUsd`-bearing fact contributes its sum.
|
||||
|
||||
### Rollup cache
|
||||
|
||||
`AgentRun.costUsd / inputTokens / outputTokens / costSource` columns are kept
|
||||
as a **derived rollup cache**, not dropped:
|
||||
|
||||
- Existing non-`/usage` readers (slash `/usage`, session detail, integration
|
||||
tests asserting `run.costUsd`) continue to work without code changes for
|
||||
historical runs.
|
||||
- On run finish, the writer writes the `UsageFact` row first and then mirrors
|
||||
it onto `AgentRun` as two separate statements, not one transaction. The
|
||||
fact is the truth, so it is written first; the cache is derived, so it is
|
||||
written second. A crash between the two leaves the cache stale, but the
|
||||
usage service re-reads `UsageFact` directly, so staleness is recoverable
|
||||
(and the reverse order would lose the truth, which is not). Keeping the
|
||||
fact insert and the run update in separate statements also avoids holding
|
||||
the `AgentRun` row lock across the FK ShareLock taken by the insert, which
|
||||
deadlocks against concurrent workspace teardown under the cascade path
|
||||
`Organization → Project → AgentRun → UsageFact`.
|
||||
- The org/project usage service and the slash `/usage` command read from
|
||||
`UsageFact` directly — they are the canonical aggregation path.
|
||||
|
||||
### Migration
|
||||
|
||||
A new `UsageFact` table is added. For every existing `AgentRun` with a
|
||||
non-null `costUsd` or non-null `inputTokens`/`outputTokens`, the migration
|
||||
inserts **one synthetic `model_completion` fact** carrying the run's
|
||||
provider, model, tokens, cost, and `costSource`. Its `correlationId` is the
|
||||
run id, so the row is identifiable as a backfill artefact. This keeps
|
||||
historical reporting correct under the new aggregation path. Pre-migration
|
||||
runs with no recorded cost remain `runsWithoutCost` by design, matching
|
||||
ADR-0022's "missing cost ≠ zero" rule and the existing migration
|
||||
`20260709143000_agent_run_cost_tracking`'s "no backfill" stance for the
|
||||
truly unrecorded.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The billing model now supports external capabilities (PDF→MD, ASR, OCR, …)
|
||||
without per-capability schema changes: a new capability is one new
|
||||
`capabilityId` value and one or more `external_capability` facts.
|
||||
- `AgentRun.costUsd` is no longer the truth; it is a convenience cache.
|
||||
Readers that need the truth (cost breakdown, per-capability attribution)
|
||||
must read `UsageFact`. The cache MUST be kept consistent on the write path.
|
||||
- The capability registry, pricebook, and any commercial settlement remain
|
||||
`OPEN` and out of pilot scope (ADR-0021). This ADR only pins the ledger
|
||||
shape and invariants.
|
||||
- `usage.ts` and `/usage` now perform a join against `UsageFact` rather than a
|
||||
single-table scan of `AgentRun`; the index on `(runId, occurredAt)` and
|
||||
`(provider, model, occurredAt)` keeps the existing org/project/report
|
||||
queries bounded.
|
||||
- Cost corrections (e.g. a provider rebills a run) produce a new fact row; the
|
||||
rollup cache must be recomputed. The initial release writes facts once at
|
||||
run finish and does not support post-hoc correction flows — that remains
|
||||
`OPEN`.
|
||||
|
||||
## Deferred
|
||||
|
||||
- **Capability registry** as a first-class spec entity (`Spec.System.Capability`)
|
||||
with org-scoped enable/disable, input/output contracts, metering schemas,
|
||||
and org-exclusive credentials (ADR-0024 alignment). This ADR only reserves
|
||||
the `capabilityId` field and the `external_capability` fact kind.
|
||||
- **Pricebook** — a versioned price table keyed by `(provider, model, unit)`
|
||||
with time validity. Required to actually produce `pricebook_derived` costs;
|
||||
until then, facts without provider-reported cost stay `unknown`.
|
||||
- **Post-hoc cost correction flow** — append-only today; correction UI and
|
||||
rollup recompute are `OPEN`.
|
||||
- **Commercial billing, invoicing, settlement** — still deferred per ADR-0021.
|
||||
Reference in New Issue
Block a user