diff --git a/docs/adr/0026-usage-fact-ledger.md b/docs/adr/0026-usage-fact-ledger.md new file mode 100644 index 0000000..df8a308 --- /dev/null +++ b/docs/adr/0026-usage-fact-ledger.md @@ -0,0 +1,146 @@ +# ADR 0026: Usage Fact Ledger + +## Status + +Accepted. + +## Context + +ADR-0022 pins the cost-attribution contract for the platform: token, +provider-reported cost, run count, and duration are attributed by Organization, +Project, Run, model, and Provider Connection; "missing provider cost remains +unknown rather than zero." ADR-0021 scopes usage accounting as operational +reporting, not payment collection (commercial billing stays deferred). + +The implementation today stores this as **one scalar per `AgentRun`**: +`inputTokens`, `outputTokens`, `costUsd?`, `costSource?`, written once from the +Claude Agent SDK's `result.total_cost_usd` when the run finishes. This is +adequate for a single provider-reported model loop, but it cannot represent: + +- **External capability consumption** inside a run. PDF→Markdown bundle + conversion, audio/video transcription, OCR and similar media transforms are + not the main agent loop. They run as side effects of a run, may use a + different provider/model, may bill in non-token units (pages, seconds, + invocations), and may report cost through a different channel than the + OpenRouter gateway. Today there is no row to write that cost to — it would + either disappear or silently corrupt the run's single scalar. +- **Multiple model calls within one run** (e.g. a sub-model invoked by a + tool, a gateway-side reroute). The scalar collapses them into one number. +- **Pricebook derivation** after the fact. With only a final USD figure and no + `(provider, model, occurredAt, tokens)` fact, an operator cannot re-derive + cost from a price table when the provider did not report it. + +Treating each external call as a nested `AgentRun` was considered and +rejected: `AgentRun` carries lock ownership (ADR-0002), session/provider/role +binding (ADR-0017), admission/capacity semantics (ADR-0022), and the +user-visible task boundary. External calls hold none of those. Making them +`AgentRun`s would pollute run counts, admission, lock semantics, and session +continuity, and would still not solve non-token metering. + +## Decision + +Introduce **`UsageFact`** as the single source of truth for billable +consumption inside an `AgentRun`. An `AgentRun` owns zero or more +append-only `UsageFact` rows; each row records one billable consumption event: + +- `kind` — `model_completion` (the main agent loop) | `external_capability` + | `tool_proxy`. The kind set is `OPEN`; new kinds must be surfaced, not + silently folded into an existing one. +- `provider` — e.g. `openrouter`, `mineru`, `openai_whisper`. +- `model?`, `inputTokens?`, `outputTokens?` — token metering, optional + because non-token capabilities have none. +- `quantity?` + `unit?` — non-token metering (pages, audio_seconds, + invocations), coexisting with tokens rather than replacing them. +- `costUsd?` + `costSource` — `provider_reported` | `pricebook_derived` | + `unknown`. `costUsd = null` means **unknown, not zero** (ADR-0022). When the + provider reported a cost, `costSource = provider_reported` and that value + wins. When only tokens are known, a later pricebook pass may derive + `costUsd` with `costSource = pricebook_derived`. When neither is possible, + `costSource = unknown` and `costUsd` stays null. +- `occurredAt` — when the consumption happened; the pricebook derivation + depends on this, not on `AgentRun.finishedAt`, because an external + capability may complete before the run finishes. +- `capabilityId?` — for `external_capability` facts, the registered capability + id (e.g. `pdf_to_md_bundle`, `audio_video_to_text`). +- `correlationId?` — external request id for reconciliation / idempotency; + not part of the aggregation key. + +### Invariants + +1. **Append-only.** A `UsageFact` row is never updated or deleted. A cost + correction is a new row; the old row stays. `onDelete: Cascade` exists only + so a hard run delete (itself not a normal path) cleans up its facts. +2. **Belongs to exactly one Run; never holds a lock.** A fact is a side-effect + ledger of one run, not a sub-run. External capability calls obey the same + boundary: their identity is `capabilityId + correlationId`, not `RunId`. +3. **Missing cost ≠ zero.** Aggregation MUST NOT sum `null` `costUsd` as 0. + A run whose facts all have `costUsd = null` is "cost unknown" — reported as + `runsWithoutCost`, exactly as the pre-migration `costUsd = null` runs are + today. A run with at least one `costUsd`-bearing fact contributes its sum. + +### Rollup cache + +`AgentRun.costUsd / inputTokens / outputTokens / costSource` columns are kept +as a **derived rollup cache**, not dropped: + +- Existing non-`/usage` readers (slash `/usage`, session detail, integration + tests asserting `run.costUsd`) continue to work without code changes for + historical runs. +- On run finish, the writer writes the `UsageFact` row first and then mirrors + it onto `AgentRun` as two separate statements, not one transaction. The + fact is the truth, so it is written first; the cache is derived, so it is + written second. A crash between the two leaves the cache stale, but the + usage service re-reads `UsageFact` directly, so staleness is recoverable + (and the reverse order would lose the truth, which is not). Keeping the + fact insert and the run update in separate statements also avoids holding + the `AgentRun` row lock across the FK ShareLock taken by the insert, which + deadlocks against concurrent workspace teardown under the cascade path + `Organization → Project → AgentRun → UsageFact`. +- The org/project usage service and the slash `/usage` command read from + `UsageFact` directly — they are the canonical aggregation path. + +### Migration + +A new `UsageFact` table is added. For every existing `AgentRun` with a +non-null `costUsd` or non-null `inputTokens`/`outputTokens`, the migration +inserts **one synthetic `model_completion` fact** carrying the run's +provider, model, tokens, cost, and `costSource`. Its `correlationId` is the +run id, so the row is identifiable as a backfill artefact. This keeps +historical reporting correct under the new aggregation path. Pre-migration +runs with no recorded cost remain `runsWithoutCost` by design, matching +ADR-0022's "missing cost ≠ zero" rule and the existing migration +`20260709143000_agent_run_cost_tracking`'s "no backfill" stance for the +truly unrecorded. + +## Consequences + +- The billing model now supports external capabilities (PDF→MD, ASR, OCR, …) + without per-capability schema changes: a new capability is one new + `capabilityId` value and one or more `external_capability` facts. +- `AgentRun.costUsd` is no longer the truth; it is a convenience cache. + Readers that need the truth (cost breakdown, per-capability attribution) + must read `UsageFact`. The cache MUST be kept consistent on the write path. +- The capability registry, pricebook, and any commercial settlement remain + `OPEN` and out of pilot scope (ADR-0021). This ADR only pins the ledger + shape and invariants. +- `usage.ts` and `/usage` now perform a join against `UsageFact` rather than a + single-table scan of `AgentRun`; the index on `(runId, occurredAt)` and + `(provider, model, occurredAt)` keeps the existing org/project/report + queries bounded. +- Cost corrections (e.g. a provider rebills a run) produce a new fact row; the + rollup cache must be recomputed. The initial release writes facts once at + run finish and does not support post-hoc correction flows — that remains + `OPEN`. + +## Deferred + +- **Capability registry** as a first-class spec entity (`Spec.System.Capability`) + with org-scoped enable/disable, input/output contracts, metering schemas, + and org-exclusive credentials (ADR-0024 alignment). This ADR only reserves + the `capabilityId` field and the `external_capability` fact kind. +- **Pricebook** — a versioned price table keyed by `(provider, model, unit)` + with time validity. Required to actually produce `pricebook_derived` costs; + until then, facts without provider-reported cost stay `unknown`. +- **Post-hoc cost correction flow** — append-only today; correction UI and + rollup recompute are `OPEN`. +- **Commercial billing, invoicing, settlement** — still deferred per ADR-0021. diff --git a/hub/prisma/migrations/20260718120000_usage_fact_ledger/migration.sql b/hub/prisma/migrations/20260718120000_usage_fact_ledger/migration.sql new file mode 100644 index 0000000..20f60ed --- /dev/null +++ b/hub/prisma/migrations/20260718120000_usage_fact_ledger/migration.sql @@ -0,0 +1,64 @@ +-- ADR-0026: append-only UsageFact ledger. AgentRun.costUsd/inputTokens/ +-- outputTokens become a derived rollup cache; the truth is in UsageFact. + +CREATE TABLE "UsageFact" ( + "id" TEXT NOT NULL, + "runId" TEXT NOT NULL, + "occurredAt" TIMESTAMP(3) NOT NULL, + "kind" TEXT NOT NULL, + "provider" TEXT NOT NULL, + "model" TEXT, + "inputTokens" INTEGER, + "outputTokens" INTEGER, + "quantity" DECIMAL(18, 6), + "unit" TEXT, + "costUsd" DECIMAL(18, 8), + "costSource" TEXT NOT NULL, + "capabilityId" TEXT, + "correlationId" TEXT, + "metadata" JSONB NOT NULL, + "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT "UsageFact_pkey" PRIMARY KEY ("id") +); + +CREATE INDEX "UsageFact_runId_occurredAt_idx" ON "UsageFact"("runId", "occurredAt"); +CREATE INDEX "UsageFact_runId_kind_idx" ON "UsageFact"("runId", "kind"); +CREATE INDEX "UsageFact_provider_model_occurredAt_idx" + ON "UsageFact"("provider", "model", "occurredAt"); +CREATE INDEX "UsageFact_capabilityId_occurredAt_idx" + ON "UsageFact"("capabilityId", "occurredAt"); + +ALTER TABLE "UsageFact" + ADD CONSTRAINT "UsageFact_runId_fkey" + FOREIGN KEY ("runId") REFERENCES "AgentRun"("id") + ON DELETE CASCADE ON UPDATE CASCADE; + +-- Backfill: one synthetic model_completion fact per AgentRun that has any +-- recorded usage (cost or tokens). correlationId = run id marks these as +-- backfill artefacts (real facts use an external correlation id or null). +-- Runs with no recorded usage stay fact-less and remain runsWithoutCost, +-- matching ADR-0022's "missing cost ≠ zero" rule and the pre-existing +-- migration 20260709143000_agent_run_cost_tracking's "no backfill for the +-- truly unrecorded" stance. +INSERT INTO "UsageFact" ( + "id", "runId", "occurredAt", "kind", "provider", "model", + "inputTokens", "outputTokens", "costUsd", "costSource", + "correlationId", "metadata" +) +SELECT + 'usagefact_backfill_' || "AgentRun"."id", + "AgentRun"."id", + COALESCE("AgentRun"."finishedAt", "AgentRun"."startedAt", CURRENT_TIMESTAMP), + 'model_completion', + "AgentRun"."provider", + "AgentRun"."model", + "AgentRun"."inputTokens", + "AgentRun"."outputTokens", + "AgentRun"."costUsd", + COALESCE("AgentRun"."costSource", 'unknown'), + "AgentRun"."id", + '{}'::jsonb +FROM "AgentRun" +WHERE "AgentRun"."costUsd" IS NOT NULL + OR "AgentRun"."inputTokens" IS NOT NULL + OR "AgentRun"."outputTokens" IS NOT NULL; diff --git a/hub/prisma/schema.prisma b/hub/prisma/schema.prisma index e5f080d..be3e60d 100644 --- a/hub/prisma/schema.prisma +++ b/hub/prisma/schema.prisma @@ -596,9 +596,13 @@ model AgentRun { summary String? inputTokens Int? outputTokens Int? - /// Provider/gateway-reported cost in USD. Null means this run has no trusted cost fact. + /// ADR-0026: derived rollup cache of UsageFact rows for this run. The truth + /// is in UsageFact; this column is convenience for existing readers. Null + /// means this run has no trusted cost fact (ADR-0022: missing cost ≠ zero). costUsd Decimal? @db.Decimal(18, 8) - /// Semantic source of costUsd, e.g. provider_reported. Not a pricing-estimate fallback. + /// ADR-0026: semantic source of the rolled-up costUsd (provider_reported | + /// pricebook_derived | unknown). Mirrors the dominant CostSource of the + /// run's facts; not a pricing-estimate fallback. costSource String? metadata Json error String? @@ -612,6 +616,7 @@ model AgentRun { projectLock ProjectAgentLock? messages AgentMessage[] @relation("runMessages") fileChanges AgentFileChange[] @relation("runFileChanges") + usageFacts UsageFact[] @relation("runUsageFacts") @@index([projectId, status]) @@index([projectId, finishedAt]) @@ -817,3 +822,50 @@ model AgentFileChange { @@index([projectId]) @@index([path]) } + +// --- Usage fact ledger (ADR-0026) ---------------------------------------- + +/// ADR-0026: one billable consumption event inside an AgentRun. Append-only; +/// the run's AgentRun.costUsd/inputTokens/outputTokens are a derived rollup +/// of these rows, not the truth. kind=external_capability carries capabilityId +/// for media transforms (PDF→MD, audio/video→text, …). costUsd=null means +/// unknown, not zero (ADR-0022). The kind set is OPEN: new kinds must be +/// surfaced, not silently folded in. +model UsageFact { + id String @id @default(cuid()) + runId String + /// When the consumption happened. Pricebook derivation uses this, not + /// AgentRun.finishedAt, because an external capability may finish before + /// the run ends. + occurredAt DateTime + /// model_completion | external_capability | tool_proxy. OPEN set. + kind String + /// e.g. openrouter, mineru, openai_whisper. + provider String + model String? + inputTokens Int? + outputTokens Int? + /// Non-token meter (pages, audio_seconds, invocations). Coexists with tokens. + quantity Decimal? @db.Decimal(18, 6) + unit String? + /// USD. Null = unknown, NOT zero (ADR-0022). When null, costSource=unknown. + costUsd Decimal? @db.Decimal(18, 8) + /// provider_reported | pricebook_derived | unknown. Required even when + /// costUsd is null, so a reader can distinguish "reported zero" from + /// "no report at all". + costSource String + /// For kind=external_capability, the registered capability id + /// (e.g. pdf_to_md_bundle, audio_video_to_text). Null for model_completion. + capabilityId String? + /// External request id for reconciliation / idempotency. Not an aggregation key. + correlationId String? + metadata Json + createdAt DateTime @default(now()) + + run AgentRun @relation("runUsageFacts", fields: [runId], references: [id], onDelete: Cascade) + + @@index([runId, occurredAt]) + @@index([runId, kind]) + @@index([provider, model, occurredAt]) + @@index([capabilityId, occurredAt]) +} diff --git a/hub/src/feishu/slashCommands.ts b/hub/src/feishu/slashCommands.ts index 2753b10..13ad57d 100644 --- a/hub/src/feishu/slashCommands.ts +++ b/hub/src/feishu/slashCommands.ts @@ -92,7 +92,18 @@ export function createSlashCommandRegistry( finishedAt: { not: null }, }, orderBy: { finishedAt: "asc" }, - select: { model: true, provider: true, inputTokens: true, outputTokens: true, costUsd: true }, + select: { + usageFacts: { + select: { + provider: true, + model: true, + inputTokens: true, + outputTokens: true, + costUsd: true, + }, + orderBy: { occurredAt: "asc" }, + }, + }, }); await sendText(rt, chatId, formatUsageReport(runs, scope), sendOptions); }, @@ -118,18 +129,24 @@ async function currentRoleSessionIds( return sessions.map((session) => session.id); } -interface UsageRun { - readonly model: string; - readonly provider: string; - readonly inputTokens: number | null; - readonly outputTokens: number | null; - readonly costUsd: unknown; +interface UsageRunWithFacts { + readonly usageFacts: readonly { + readonly provider: string; + readonly model: string | null; + readonly inputTokens: number | null; + readonly outputTokens: number | null; + readonly costUsd: unknown; + }[]; } -function formatUsageReport(runs: readonly UsageRun[], scope: "current" | "project"): string { +function formatUsageReport(runs: readonly UsageRunWithFacts[], scope: "current" | "project"): string { if (runs.length === 0) return `${scope === "current" ? "当前角色会话" : "当前项目"}还没有已结束的 Agent run。`; + // Bucket by (fact.provider, fact.model) — ADR-0026: external capabilities + // carry their own provider/model and contribute their own meter, so a run + // with a main loop + an external call lands in two buckets. A run with no + // cost-bearing fact is "unrecorded" (ADR-0022: missing cost ≠ zero). const buckets = new Map(); let recordedRuns = 0; let unrecordedRuns = 0; @@ -137,21 +154,34 @@ function formatUsageReport(runs: readonly UsageRun[], scope: "current" | "projec let totalOutputTokens = 0; let totalCostUsd = 0; for (const run of runs) { - const costUsd = decimalToNumberOrNull(run.costUsd); - if (costUsd === null) { unrecordedRuns++; continue; } + const facts = run.usageFacts; + if (facts.length === 0 || !facts.some((f) => f.costUsd !== null && f.costUsd !== undefined)) { + unrecordedRuns++; + // Tokens from unrecorded runs still count toward totals (matches pre-0026). + for (const f of facts) { + totalInputTokens += f.inputTokens ?? 0; + totalOutputTokens += f.outputTokens ?? 0; + } + continue; + } recordedRuns++; - const key = `${run.provider}\u0000${run.model}`; - const bucket = buckets.get(key) ?? { - provider: run.provider, model: run.model, runs: 0, inputTokens: 0, outputTokens: 0, costUsd: 0, - }; - bucket.runs++; - bucket.inputTokens += run.inputTokens ?? 0; - bucket.outputTokens += run.outputTokens ?? 0; - bucket.costUsd += costUsd; - buckets.set(key, bucket); - totalInputTokens += run.inputTokens ?? 0; - totalOutputTokens += run.outputTokens ?? 0; - totalCostUsd += costUsd; + for (const f of facts) { + const costUsd = decimalToNumberOrNull(f.costUsd); + if (costUsd === null) continue; + const model = f.model ?? "(unknown model)"; + const key = `${f.provider}\u0000${model}`; + const bucket = buckets.get(key) ?? { + provider: f.provider, model, facts: 0, inputTokens: 0, outputTokens: 0, costUsd: 0, + }; + bucket.facts += 1; + bucket.inputTokens += f.inputTokens ?? 0; + bucket.outputTokens += f.outputTokens ?? 0; + bucket.costUsd += costUsd; + buckets.set(key, bucket); + totalInputTokens += f.inputTokens ?? 0; + totalOutputTokens += f.outputTokens ?? 0; + totalCostUsd += costUsd; + } } const lines = [ `${scope === "current" ? "当前角色会话" : "当前项目"}用量`, @@ -160,7 +190,7 @@ function formatUsageReport(runs: readonly UsageRun[], scope: "current" | "projec ]; if (unrecordedRuns > 0) lines.push(`另有 ${formatInteger(unrecordedRuns)} 个 run 未记录成本。`); for (const bucket of buckets.values()) { - lines.push(`- ${bucket.provider} / ${bucket.model}: ${formatInteger(bucket.runs)} runs, ${formatUsd(bucket.costUsd)}`); + lines.push(`- ${bucket.provider} / ${bucket.model}: ${formatInteger(bucket.facts)} 次, ${formatUsd(bucket.costUsd)}`); } return lines.join("\n"); } diff --git a/hub/src/feishu/trigger.ts b/hub/src/feishu/trigger.ts index 16fe5ee..f1b2747 100644 --- a/hub/src/feishu/trigger.ts +++ b/hub/src/feishu/trigger.ts @@ -604,6 +604,32 @@ export function makeTriggerHandler(deps: TriggerDeps): TriggerHandler { data: { metadata: mergeSessionMetadata(session.metadata, metadataPatch) }, }); } + // ADR-0026: the UsageFact ledger is the truth; AgentRun.costUsd / + // inputTokens / outputTokens are a derived rollup cache for existing + // readers. We write the fact first, then mirror it onto the run. + // They are separate statements (not one transaction) so the AgentRun + // row lock is held for the shortest possible window and concurrent + // workspace teardown cannot deadlock on the FK ShareLock. A crash + // between the two leaves the cache stale, but the cache is derived + // (the usage service re-reads facts), so staleness is recoverable; + // the reverse order would lose the truth and is unrecoverable. + const finishedAt = new Date(); + const reportedCost = result.costUsd; + const costSource = reportedCost !== undefined ? "provider_reported" : "unknown"; + await deps.prisma.usageFact.create({ + data: { + runId: run.id, + occurredAt: finishedAt, + kind: "model_completion", + provider: providerId, + model, + inputTokens: result.usage.inputTokens, + outputTokens: result.usage.outputTokens, + costUsd: reportedCost ?? null, + costSource, + metadata: {}, + }, + }); await deps.prisma.agentRun.update({ where: { id: run.id }, data: { @@ -618,9 +644,10 @@ export function makeTriggerHandler(deps: TriggerDeps): TriggerHandler { : "FAILED", inputTokens: result.usage.inputTokens, outputTokens: result.usage.outputTokens, - ...(result.costUsd !== undefined ? { costUsd: result.costUsd, costSource: "provider_reported" } : {}), + costUsd: reportedCost ?? null, + costSource, error: result.error ?? null, - finishedAt: new Date(), + finishedAt, }, }); await writeAudit(deps.prisma, { diff --git a/hub/src/org/sessions.ts b/hub/src/org/sessions.ts index 342c1f0..22c6966 100644 --- a/hub/src/org/sessions.ts +++ b/hub/src/org/sessions.ts @@ -73,9 +73,28 @@ export async function getSessionDetail( inputTokens: true, outputTokens: true, costUsd: true, + costSource: true, startedAt: true, finishedAt: true, error: true, + usageFacts: { + select: { + id: true, + occurredAt: true, + kind: true, + provider: true, + model: true, + inputTokens: true, + outputTokens: true, + quantity: true, + unit: true, + costUsd: true, + costSource: true, + capabilityId: true, + correlationId: true, + }, + orderBy: { occurredAt: "asc" }, + }, }, orderBy: { startedAt: "desc" }, take: 100, @@ -106,9 +125,25 @@ export async function getSessionDetail( inputTokens: r.inputTokens, outputTokens: r.outputTokens, costUsd: r.costUsd === null ? null : Number(r.costUsd), + costSource: r.costSource, startedAt: r.startedAt.toISOString(), finishedAt: r.finishedAt?.toISOString() ?? null, error: r.error, + usageFacts: r.usageFacts.map((f) => ({ + id: f.id, + occurredAt: f.occurredAt.toISOString(), + kind: f.kind, + provider: f.provider, + model: f.model, + inputTokens: f.inputTokens, + outputTokens: f.outputTokens, + quantity: f.quantity === null ? null : Number(f.quantity), + unit: f.unit, + costUsd: f.costUsd === null ? null : Number(f.costUsd), + costSource: f.costSource, + capabilityId: f.capabilityId, + correlationId: f.correlationId, + })), })), }; } diff --git a/hub/src/org/usage.ts b/hub/src/org/usage.ts index ea32d69..41f2b5e 100644 --- a/hub/src/org/usage.ts +++ b/hub/src/org/usage.ts @@ -1,8 +1,14 @@ /** - * Org / project usage rollups from AgentRun cost facts (ADR-0021). + * Org / project usage rollups from the UsageFact ledger (ADR-0021, ADR-0026). * Operational usage accounting, not payment collection. Organizations may use * either their own provider credentials or a distinct platform-managed * connection; commercial billing remains outside the pilot scope. + * + * The truth is in `UsageFact`; `AgentRun.costUsd / inputTokens / outputTokens` + * are a derived rollup cache. This service reads `UsageFact` directly so that + * external-capability consumption (PDF→MD, ASR, …) is attributed correctly, + * not just the main model loop. A run with no cost-bearing fact is + * `runsWithoutCost` — ADR-0022: missing cost ≠ zero. */ import type { Prisma, PrismaClient } from "@prisma/client"; @@ -32,6 +38,53 @@ export interface UsageReport { }; } +type FactRow = { + readonly inputTokens: number | null; + readonly outputTokens: number | null; + readonly costUsd: unknown; +}; + +type RunWithFacts = { + readonly projectId: string; + readonly usageFacts: readonly FactRow[]; +}; + +/** A run is "recorded with cost" if any of its facts carries a known costUsd. */ +function runHasRecordedCost(facts: readonly FactRow[]): boolean { + return facts.some((f) => f.costUsd !== null && f.costUsd !== undefined); +} + +/** Decimal→number for Prisma Decimal; null/undefined → null. */ +function factCostUsdToNumber(value: unknown): number | null { + if (value === null || value === undefined) return null; + const n = typeof value === "number" ? value : Number(value); + return Number.isFinite(n) ? n : null; +} + +interface RunRollup { + readonly hasCost: boolean; + readonly inputTokens: number; + readonly outputTokens: number; + readonly costUsd: number | null; +} + +function rollupRun(facts: readonly FactRow[]): RunRollup { + let inputTokens = 0; + let outputTokens = 0; + let costUsd: number | null = null; + let hasCost = false; + for (const f of facts) { + inputTokens += f.inputTokens ?? 0; + outputTokens += f.outputTokens ?? 0; + const c = factCostUsdToNumber(f.costUsd); + if (c !== null) { + hasCost = true; + costUsd = (costUsd ?? 0) + c; + } + } + return { hasCost, inputTokens, outputTokens, costUsd }; +} + export async function getOrgUsage( prisma: PrismaClient, input: { @@ -54,8 +107,9 @@ export async function getOrgUsage( return emptyReport(input.from, input.to); } + const projectIds = projects.map((p) => p.id); const runWhere: Prisma.AgentRunWhereInput = { - projectId: { in: projects.map((p) => p.id) }, + projectId: { in: projectIds }, ...(input.from !== undefined || input.to !== undefined ? { startedAt: { @@ -66,20 +120,25 @@ export async function getOrgUsage( : {}), }; + // Date range filters by run.startedAt, matching the pre-ADR-0026 semantics: + // a run is in or out of the window based on when it started, and all of its + // facts come along. Filtering facts by occurredAt independently would let a + // run contribute partial cost to a window it doesn't belong to. const runs = await prisma.agentRun.findMany({ where: runWhere, select: { projectId: true, - inputTokens: true, - outputTokens: true, - costUsd: true, + usageFacts: { + select: { inputTokens: true, outputTokens: true, costUsd: true }, + orderBy: { occurredAt: "asc" }, + }, }, }); type MutableRow = { - projectId: string; - projectName: string; - folderId: string | null; + readonly projectId: string; + readonly projectName: string; + readonly folderId: string | null; runCount: number; runsWithCost: number; runsWithoutCost: number; @@ -103,22 +162,33 @@ export async function getOrgUsage( }); } - for (const run of runs) { + for (const run of runs as readonly RunWithFacts[]) { const row = byProject.get(run.projectId); if (row === undefined) continue; + const roll = rollupRun(run.usageFacts); row.runCount += 1; - row.inputTokens += run.inputTokens ?? 0; - row.outputTokens += run.outputTokens ?? 0; - if (run.costUsd === null) { - row.runsWithoutCost += 1; - } else { + if (roll.hasCost) { row.runsWithCost += 1; - const cost = Number(run.costUsd); - row.costUsd = (row.costUsd ?? 0) + cost; + row.costUsd = (row.costUsd ?? 0) + (roll.costUsd ?? 0); + } else { + row.runsWithoutCost += 1; } + row.inputTokens += roll.inputTokens; + row.outputTokens += roll.outputTokens; } - const projectsOut: ProjectUsageRow[] = [...byProject.values()]; + const projectsOut: ProjectUsageRow[] = [...byProject.values()].map((r) => ({ + projectId: r.projectId, + projectName: r.projectName, + folderId: r.folderId, + runCount: r.runCount, + runsWithCost: r.runsWithCost, + runsWithoutCost: r.runsWithoutCost, + inputTokens: r.inputTokens, + outputTokens: r.outputTokens, + costUsd: r.costUsd, + })); + let runCount = 0; let runsWithCost = 0; let runsWithoutCost = 0; diff --git a/hub/test/integration/usage.test.ts b/hub/test/integration/usage.test.ts new file mode 100644 index 0000000..3e226de --- /dev/null +++ b/hub/test/integration/usage.test.ts @@ -0,0 +1,191 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { prisma, resetDb, seedTestOrganization, DEFAULT_ORG_ID } from "./helpers.js"; +import { getOrgUsage, getProjectUsage } from "../../src/org/usage.js"; + +/** + * ADR-0026: org/project usage rolls up from the UsageFact ledger, not from the + * AgentRun scalar cache. These tests pin the fact-based aggregation contract: + * - a run with a cost-bearing fact → runsWithCost, costUsd summed + * - a run whose facts all have null costUsd → runsWithoutCost (≠ zero) + * - non-token meter (quantity/unit) is stored but does not corrupt token sums + * - multiple facts per run (main loop + external capability) are summed together + * - the AgentRun rollup cache is NOT consulted by getOrgUsage + */ +describe("usage rollup from UsageFact (ADR-0026)", () => { + beforeEach(async () => { + await resetDb(); + await seedTestOrganization(); + }); + + async function seedRun(opts: { + readonly projectId: string; + readonly projectName: string; + readonly folderId?: string; + readonly facts: ReadonlyArray<{ + readonly kind?: string; + readonly provider?: string; + readonly model?: string; + readonly inputTokens?: number; + readonly outputTokens?: number; + readonly costUsd?: number | null; + readonly costSource?: string; + readonly capabilityId?: string; + readonly quantity?: number | null; + readonly unit?: string | null; + }>; + }): Promise { + const project = await prisma.project.create({ + data: { + id: opts.projectId, + organizationId: DEFAULT_ORG_ID, + name: opts.projectName, + workspaceDir: `/tmp/test-${opts.projectId}`, + ...(opts.folderId !== undefined ? { folderId: opts.folderId } : {}), + }, + }); + const startedAt = new Date("2026-07-18T10:00:00Z"); + const finishedAt = new Date("2026-07-18T10:05:00Z"); + const run = await prisma.agentRun.create({ + data: { + projectId: project.id, + entrypoint: "FEISHU", + provider: "openrouter", + model: "mock-model", + metadata: {}, + status: "COMPLETED", + prompt: "test", + startedAt, + finishedAt, + // Deliberately set the rollup cache to values that differ from the + // facts, to prove getOrgUsage reads facts, not the cache. + inputTokens: 999, + outputTokens: 999, + costUsd: 999, + costSource: "provider_reported", + }, + }); + for (const [i, f] of opts.facts.entries()) { + await prisma.usageFact.create({ + data: { + runId: run.id, + occurredAt: new Date(startedAt.getTime() + i * 1000), + kind: f.kind ?? "model_completion", + provider: f.provider ?? "openrouter", + model: f.model ?? "mock-model", + inputTokens: f.inputTokens ?? null, + outputTokens: f.outputTokens ?? null, + costUsd: f.costUsd ?? null, + costSource: f.costSource ?? (f.costUsd !== undefined && f.costUsd !== null ? "provider_reported" : "unknown"), + capabilityId: f.capabilityId ?? null, + quantity: f.quantity ?? null, + unit: f.unit ?? null, + metadata: {}, + }, + }); + } + } + + it("sums cost and tokens from facts, ignoring the AgentRun rollup cache", async () => { + await seedRun({ + projectId: "proj-a", + projectName: "A", + facts: [{ inputTokens: 10, outputTokens: 5, costUsd: 0.0023 }], + }); + await seedRun({ + projectId: "proj-b", + projectName: "B", + facts: [{ inputTokens: 20, outputTokens: 10, costUsd: 0.01 }], + }); + + const report = await getOrgUsage(prisma, { organizationId: DEFAULT_ORG_ID }); + + expect(report.totals.runCount).toBe(2); + expect(report.totals.runsWithCost).toBe(2); + expect(report.totals.runsWithoutCost).toBe(0); + expect(report.totals.inputTokens).toBe(30); + expect(report.totals.outputTokens).toBe(15); + expect(report.totals.costUsd).toBeCloseTo(0.0123, 8); + }); + + it("treats a run with only null-cost facts as runsWithoutCost, never zero", async () => { + await seedRun({ + projectId: "proj-unknown", + projectName: "Unknown", + facts: [{ inputTokens: 7, outputTokens: 3, costUsd: null, costSource: "unknown" }], + }); + await seedRun({ + projectId: "proj-known", + projectName: "Known", + facts: [{ inputTokens: 4, outputTokens: 2, costUsd: 0.05 }], + }); + + const report = await getOrgUsage(prisma, { organizationId: DEFAULT_ORG_ID }); + + expect(report.totals.runCount).toBe(2); + expect(report.totals.runsWithCost).toBe(1); + expect(report.totals.runsWithoutCost).toBe(1); + // The unknown-cost run still contributes its tokens. + expect(report.totals.inputTokens).toBe(11); + expect(report.totals.outputTokens).toBe(5); + // And its cost is NOT counted as zero — only the known cost is summed. + expect(report.totals.costUsd).toBeCloseTo(0.05, 8); + }); + + it("sums multiple facts per run (main loop + external capability)", async () => { + await seedRun({ + projectId: "proj-multi", + projectName: "Multi", + facts: [ + { kind: "model_completion", inputTokens: 100, outputTokens: 50, costUsd: 0.02 }, + { + kind: "external_capability", + provider: "mineru", + model: null, + inputTokens: null, + outputTokens: null, + costUsd: 0.015, + costSource: "provider_reported", + capabilityId: "pdf_to_md_bundle", + quantity: 12, + unit: "pages", + }, + ], + }); + + const report = await getOrgUsage(prisma, { organizationId: DEFAULT_ORG_ID }); + expect(report.totals.runCount).toBe(1); + expect(report.totals.runsWithCost).toBe(1); + expect(report.totals.inputTokens).toBe(100); + expect(report.totals.outputTokens).toBe(50); + expect(report.totals.costUsd).toBeCloseTo(0.035, 8); + }); + + it("a run with no facts at all is runsWithoutCost and contributes nothing", async () => { + await seedRun({ projectId: "proj-empty", projectName: "Empty", facts: [] }); + + const report = await getOrgUsage(prisma, { organizationId: DEFAULT_ORG_ID }); + expect(report.totals.runCount).toBe(1); + expect(report.totals.runsWithCost).toBe(0); + expect(report.totals.runsWithoutCost).toBe(1); + expect(report.totals.inputTokens).toBe(0); + expect(report.totals.outputTokens).toBe(0); + expect(report.totals.costUsd).toBeNull(); + }); + + it("getProjectUsage returns just that project's row", async () => { + await seedRun({ projectId: "proj-x", projectName: "X", facts: [{ costUsd: 0.1, inputTokens: 1, outputTokens: 1 }] }); + await seedRun({ projectId: "proj-y", projectName: "Y", facts: [{ costUsd: 0.2, inputTokens: 2, outputTokens: 2 }] }); + + const row = await getProjectUsage(prisma, { organizationId: DEFAULT_ORG_ID, projectId: "proj-x" }); + expect(row.projectId).toBe("proj-x"); + expect(row.runCount).toBe(1); + expect(row.costUsd).toBeCloseTo(0.1, 8); + }); + + it("returns an empty report for an org with no projects", async () => { + const report = await getOrgUsage(prisma, { organizationId: DEFAULT_ORG_ID }); + expect(report.projects).toHaveLength(0); + expect(report.totals.runCount).toBe(0); + expect(report.totals.costUsd).toBeNull(); + }); +}); diff --git a/hub/test/unit/feishu-reactions.test.ts b/hub/test/unit/feishu-reactions.test.ts index 1583030..4d52fde 100644 --- a/hub/test/unit/feishu-reactions.test.ts +++ b/hub/test/unit/feishu-reactions.test.ts @@ -290,6 +290,9 @@ function mockPrisma(): PrismaClient { findUnique: vi.fn(async () => null), update: vi.fn(async () => ({ id: "run-1" })), }, + usageFact: { + create: vi.fn(async () => ({ id: "fact-1" })), + }, projectAgentLock: { findUnique: vi.fn(async () => (lock === null ? null : { runId: lock.runId })), create: vi.fn(async (args: { data: { projectId: string; runId: string } }) => { diff --git a/spec/Spec/System.lean b/spec/Spec/System.lean index fc8695a..917f114 100644 --- a/spec/Spec/System.lean +++ b/spec/Spec/System.lean @@ -10,6 +10,7 @@ import Spec.System.Agent.Run import Spec.System.Agent.AgentRole import Spec.System.Agent.Memory import Spec.System.Agent.AgentSurface +import Spec.System.Agent.Usage import Spec.System.Lock import Spec.System.Permission import Spec.System.PermissionGrant @@ -39,11 +40,14 @@ likec4 已画出结构;这里补语义: - `Memory` —— 按需上下文:锚点类别(ADR-0003)+ MCP 工具按 run/project 上下文授权。 - `AgentSurface` —— agent 执行面被 run 的工作区所界定(ADR-0018);与 Lock 正交—— Lock 限定并发,Surface 限定波及面。机制 OPEN。 +- `Usage` —— Run 内 append-only 用量计量事实账本(ADR-0026);主模型 completion、 + 外部能力调用、代理网关旁路共用同一账本。fact 不持锁、不跨 run;`costUsd = none` + 表示未知而非零(ADR-0022)。Run 上的 cost/token 标量是其 rollup cache。 - `Permission` —— read⊂edit⊂manage 角色体系、能力推导、单调性;force-release 在格外。 - `PermissionGrant` —— grant(resource×principal×role)与 settings(六 policy 旋钮) (ADR-0004);组合规则 OPEN。 - `Audit` —— customer Project/Run 审计从简(内容 OPEN);Platform Audit 由 `PlatformAdministration` 独立承载。 -标识符见 `Spec.Prelude`。决策出处:ADR-0001..0004, 0018, 0020..0024。 +标识符见 `Spec.Prelude`。决策出处:ADR-0001..0004, 0018, 0020..0026。 -/ diff --git a/spec/Spec/System/Agent/Usage.lean b/spec/Spec/System/Agent/Usage.lean new file mode 100644 index 0000000..64ed69a --- /dev/null +++ b/spec/Spec/System/Agent/Usage.lean @@ -0,0 +1,92 @@ +import Spec.Prelude + +/-! +# Usage —— Run 内用量计量事实账本(ADR-0026) + +一次 `AgentRun` 内可能产生**多条**可计费消耗:主模型 completion、外部能力调用 +(PDF→MD bundle、音视频转写等)、或代理网关侧旁路调用。这些消耗**不**是子 Run—— +它们不持项目锁、不占 session 语义、不复用 `AgentRun` 状态机。它们是 Run 内的 +append-only **计量事实**(`UsageFact`)。 + +`AgentRun` 上的 cost/token 标量是 facts 的派生 rollup cache,不是真相来源;真相在 +`UsageFact` 行。这一层抽象让"主模型"与"外部能力"两类消耗共享同一账本,而非给后者 +各开一种特化字段或 nested Run。 + +核心不变式(`PINNED`, ADR-0022 + ADR-0026): + +- **append-only** —— fact 一旦写入只读不更不删;`costUsd` 修正走新 fact,不改旧行。 +- **`costUsd = none` 表示"未知",不是"零成本"** —— ADR-0022:missing provider cost + remains unknown rather than zero。聚合时不得把 none 当 0 求和。 +- **fact 不持锁、不跨 run** —— 它是 Run 内的副作用账本,不是又一次 `@bot` 生命周期。 + 外部能力调用亦同:它的语义边界是 `capabilityId` + `correlationId`,不是 `RunId`。 +- **价目回算是派生** —— `costSource = pricebookDerived` 时,`costUsd` 由 + `occurredAt × (provider, model)` 查价目表派生;provider 实报(`providerReported`) + 优先于派生。无价目可查时 `costSource = unknown`,`costUsd = none`。 + +外部能力(capability)注册表、价目表(pricebook)、商业结算均 `OPEN`,pilot 不做 +(ADR-0021 仍 defer commercial billing)。本模块只 pin **账本结构与不变式**。 +-/ + +namespace Spec.System.Agent + +variable (I : Identifiers) (Time Cost Quantity : Type) + +/-- 计量事实类型(`PINNED`, ADR-0026)。集合完整性 `OPEN`——新增 kind 须 surface, +不得静默把外部消耗塞进既有 kind。 -/ +inductive UsageFactKind where + /-- 主 agent loop 的模型 completion(`PINNED`)。 -/ + | modelCompletion + /-- 外部能力调用(`PINNED`;如 pdf_to_md_bundle、audio_video_to_text)。 -/ + | externalCapability + /-- 代理网关侧旁路调用(`PINNED`;非主 loop 的模型调用,如嵌入工具内的子请求)。 -/ + | toolProxy + +/-- 成本来源(`PINNED`, ADR-0026)。优先级:provider 实报 > 价目派生 > unknown。 -/ +inductive CostSource where + /-- provider/gateway 实报(`PINNED`)。 -/ + | providerReported + /-- 用 `occurredAt × (provider, model)` 价目表派生(`PINNED`)。 -/ + | pricebookDerived + /-- 缺失,而非零(`PINNED`;ADR-0022 missing cost ≠ zero)。 -/ + | unknown + +/-- 一次可计费消耗的计量事实(`PINNED`, ADR-0026)。append-only;一次 run ≥0 条。 + +字段分两组:**归属与时间**(run/occurredAt/kind)用于聚合与价目回算; +**计量与成本**(tokens/quantity/unit/costUsd/costSource)用于账目本身。 +非 token 计量(页/秒/次)走 `quantity + unit`,与 token 并存而非取代。 -/ +structure UsageFact where + /-- 所属 run(`PINNED`;fact 不跨 run,fact 不持锁,ADR-0026)。 -/ + run : I.RunId + /-- 消耗发生时刻(`PINNED`);价目回算依赖此字段而非 run.finishedAt, + 因为外部能力可能早于 run 结束。 -/ + occurredAt : Time + /-- 事实类型(`PINNED`, ADR-0026)。 -/ + kind : UsageFactKind + /-- provider 标识(`PINNED`;如 openrouter、mineru、openai_whisper)。 -/ + provider : String + /-- 模型标识(`OPEN`;非模型计费可为 none)。 -/ + model : Option String + /-- 输入 tokens(`OPEN`;非 token 计量可为 none)。 -/ + inputTokens : Option Nat + /-- 输出 tokens(`OPEN`)。 -/ + outputTokens : Option Nat + /-- 非 token 计量数值(`OPEN`;如页数、音频秒数)。与 tokens 并存。 -/ + quantity : Option Quantity + /-- 计量单位(`OPEN`;如 "pages"、"audio_seconds"、"invocations")。 -/ + unit : Option String + /-- 成本(`PINNED`;none = 未知,非零,ADR-0022)。 -/ + costUsd : Option Cost + /-- 成本来源(`PINNED`;costUsd ≠ none 时必填,costUsd = none 时为 `unknown`)。 -/ + costSource : CostSource + /-- 外部能力标识(`OPEN`;kind = externalCapability 时填,如 pdf_to_md_bundle)。 -/ + capabilityId : Option String + /-- 外部请求关联 id(`OPEN`;对账/幂等用,不参与聚合)。 -/ + correlationId : Option String + +/-- Fact 的成本是否**已知**(`PINNED`, ADR-0026)。聚合 rollup 时,只对已知成本求和; +未知成本的 fact 不贡献 0,而是让汇总保持未知(若所有 fact 均未知)或部分未知。 -/ +def UsageFact.HasKnownCost (fact : UsageFact I Time Cost Quantity) : Prop := + fact.costUsd ≠ none + +end Spec.System.Agent