diff --git a/.scratch/saas-production-readiness/assets/initial-abuse-capacity-controls.md b/.scratch/saas-production-readiness/assets/initial-abuse-capacity-controls.md new file mode 100644 index 0000000..34f160e --- /dev/null +++ b/.scratch/saas-production-readiness/assets/initial-abuse-capacity-controls.md @@ -0,0 +1,125 @@ +# Initial abuse and capacity controls + +## Verdict + +The initial production boundary requires hard safety controls before customer +traffic. The current project lock and small in-memory trigger queue are useful +local protections, but they do not provide multi-tenant admission, durable +backpressure, storage safety, request isolation, bounded execution, or an +operator stop mechanism. + +The accepted contract is recorded in +[ADR-0022](../../../docs/adr/0022-layered-capacity-admission.md) and +[`Spec.System.Capacity`](../../../spec/Spec/System/Capacity.lean). Exact values +remain deliberately open until production-like calibration; the required +dimensions and failure behaviors do not. + +## Current implementation inventory + +| Surface | Existing protection | Production gap | +| --- | --- | --- | +| HTTP | Fastify's default 1 MiB body limit | No route-class, user, Org, IP, or administrative-write rate limits; request/connection/handler timeouts are unset | +| Project execution | Database lock permits one active run per Project | No platform or Org concurrency ceiling and no fair scheduler | +| Busy Project | Process-local FIFO, default five items and five-minute wait | Restart loses accepted work; expiry is logged and dropped without a durable state or user notification | +| Agent execution | Global max-turn default of 25 | No wall-clock, tool-call, output, memory, CPU, process, or Org-specific budget | +| Feishu messages | Message batching limits text count/characters | File/post attachment count and size are unbounded; downloads may buffer whole files | +| Storage | Workspace path boundary | No Project or Org byte quota, archive-expansion limit, or entity-count ceiling | +| Usage | Provider-reported token/cost fields and Org aggregation | Missing cost can disappear from totals; no soft-threshold notification or provider-mode attribution contract | +| Host | systemd lifecycle | No service CPU, memory, or task ceiling and no audited workload brake | + +Most Organization member/team/project lists are also unpaginated, and usage +aggregation can load all matching runs. Those query shapes must be bounded as +part of enforcement rather than hidden behind larger process memory. + +## Accepted contract + +### Layered limits + +- Every mandatory dimension has a versioned, non-overridable platform ceiling. +- An Organization OWNER/ADMIN may configure a lower policy limit. +- Effective limit is `min(platform, organization)`; absent Organization policy + means the platform ceiling, never unlimited. +- Missing or invalid mandatory production ceilings fail startup. + +### Admission and fairness + +- Project exclusivity remains one active run. +- Platform and Organization concurrency limits apply before starting work. +- Available slots are scheduled fairly across Organizations, FIFO within an + Organization, so one tenant cannot monopolize a global FIFO. +- Accepted work is durable and queryable. Queue length and wait are bounded. +- Overflow is explicit `capacity_exhausted` (`429` plus `Retry-After` for HTTP; + a clear busy response for Feishu). Nothing is silently dropped. +- Wait timeout produces terminal `EXPIRED`, not a later automatic start. +- The initiator may withdraw their queued request. Project `MANAGE` or an + Organization admin may cancel queued requests in their scope. Actor, reason, + and notification are recorded. + +### Retry and idempotency + +- Feishu `event_id` and API idempotency identify one admission across retries. +- Once a run starts, the whole run is never automatically replayed because tool + side effects may already exist. +- Only bounded transport retries before model output or tool side effects are + automatic. Later failure/timeouts require an explicit new request. + +### Request, file, storage, and entity safety + +- Authenticated Web requests are limited by user, Organization, and endpoint + class; signed Feishu events by Organization app, chat, and sender; + administrative writes use stricter independent buckets. +- Admin surfaces require both a valid user session and the corresponding + platform/Organization administrator authorization. OAuth callbacks and + signed Feishu ingress remain necessary non-session entry points. +- Hard limits cover request body, single-file bytes, attachment count, archive + expanded bytes/file count/depth, Project/Organization storage, and per-type + Organization entity counts. +- Transfer is streamed and counted; partial files are removed after failure. +- Crossing a ceiling blocks new growth without auto-deleting or hiding existing + resources. Read, export, delete, and administrator remediation stay usable. + +### Run budget + +Every run is bounded by wall time, model turns, tool-call count, per-tool time, +output/log/event bytes, and process memory/CPU/child-process count. A breached +budget ends explicitly as `TIMED_OUT` or `LIMIT_EXCEEDED`, records the dimension +and observed usage, releases capacity, and never masquerades as success. + +### Usage and cost + +- Token usage, provider-reported cost, run count, and duration are attributed by + Organization, Project, Run, model, and Provider Connection. +- Missing provider cost is `unknown`, never zero. +- Thresholds are soft alerts in the pilot; they do not automatically stop runs + and are not payment settlement. +- Org admins receive Organization alerts. Platform admins additionally receive + anomalous-cost alerts for platform-managed connections; BYOK does not trigger + platform financial alerts. + +### Emergency control + +An audited Platform Administrator control supports Organization or global +`DRAIN` (reject new work, pause queue starts, finish active runs) and `STOP_NOW` +(also cancel active runs). Both require an actor, reason, timestamp, explicit +recovery, and user-visible outcomes. This is workload control, not Organization +deletion or account suspension. + +## Required production evidence + +1. Production configuration validation fails closed for every missing or + invalid platform ceiling. +2. Concurrent noisy-neighbor tests prove platform/Org/Project concurrency and + Organization-fair scheduling without starvation. +3. Restart and crash tests prove accepted queue entries, idempotency, expiry, + cancellation, and notifications survive process loss. +4. HTTP and Feishu load tests prove every rate-limit identity and typed overload + response, including retry guidance. +5. File/archive/storage probes prove streaming enforcement, cleanup, and + continued read/delete access while over limit. +6. Hung provider/tool and resource-exhaustion probes prove every run/process + budget terminates, records cause, and releases locks/capacity. +7. Usage tests prove provider-mode attribution, unknown-cost handling, alert + routing, and absence of hard monetary blocking. +8. `DRAIN` and `STOP_NOW` drills prove scoped, audited, reversible behavior. +9. Production-like calibration publishes the tested numerical defaults and + capacity headroom used by the release gate. diff --git a/.scratch/saas-production-readiness/assets/tenant-auth-security-audit.md b/.scratch/saas-production-readiness/assets/tenant-auth-security-audit.md index b5e8a94..c4a7733 100644 --- a/.scratch/saas-production-readiness/assets/tenant-auth-security-audit.md +++ b/.scratch/saas-production-readiness/assets/tenant-auth-security-audit.md @@ -29,8 +29,10 @@ The audit used these upstream requirements: run's project workspace. - ADR-0019: authorization is evaluated against principals resolved at request time so membership changes take effect immediately. -- ADR-0021: customer Feishu and model-provider credentials are org-scoped - secrets, encrypted at rest, and accessed through a resolver. +- ADR-0021: Feishu and model-provider credentials are org-scoped secrets, + encrypted at rest, and accessed through a resolver. Provider ownership may be + Organization-managed BYOK or a distinct platform-managed connection for that + Organization; neither mode permits a process-global shared key. ## Confirmed protections @@ -130,10 +132,10 @@ encrypted credentials "deferred", although ADR-0021 says they **must** be org-scoped, encrypted at rest, and accessed through a resolver. `EnvRuntimeSettings.provider` similarly discards its `scope` argument and uses -one global OpenRouter token for every project. This both leaks tenant ownership -of credentials and makes the accepted "customers supply their own provider -keys" accounting model false. The schema has no encrypted credential store, -key version, rotation state, or resolver seam. +one global OpenRouter token for every project. This erases the accepted +per-Organization provider mode and makes both BYOK and platform-managed cost +attribution false. The schema has no encrypted credential store, key version, +rotation state, or resolver seam. This is also a product reachability blocker: one customer-owned Feishu app cannot receive bot/OAuth traffic for unrelated customer tenants under the @@ -248,7 +250,9 @@ Production readiness requires automated proof of all of the following: organization, project, run, and pending action consistently. 4. ACTIVE Organization status is enforced in the shared authorization seam. 5. Feishu and provider credentials resolve by Organization, are encrypted at - rest with versioned keys, never enter logs/agent tools, and can rotate. + rest with versioned keys, never enter logs/agent tools, and can rotate; + provider resolution honors either Organization-managed BYOK or a distinct + platform-managed connection for that Organization. 6. Cross-org HTTP and Prisma service tests remain green, while new negative tests cover every finding above. 7. Production Node and Rust dependency audits pass with no ignored diff --git a/.scratch/saas-production-readiness/issues/05-define-abuse-capacity-controls.md b/.scratch/saas-production-readiness/issues/05-define-abuse-capacity-controls.md index e237ae8..a30b6da 100644 --- a/.scratch/saas-production-readiness/issues/05-define-abuse-capacity-controls.md +++ b/.scratch/saas-production-readiness/issues/05-define-abuse-capacity-controls.md @@ -1,7 +1,7 @@ # Define initial abuse and capacity controls Type: grilling -Status: open +Status: resolved Blocked by: 02, 03 ## Question @@ -9,3 +9,35 @@ Blocked by: 02, 03 Which request limits, file limits, concurrency limits, agent budgets, tenant quotas, and backpressure policies are mandatory for the accepted initial production boundary, and which values or behaviors remain product decisions? + +## Answer + +The initial production service requires layered, fail-closed capacity controls: +every mandatory dimension has a versioned non-overridable platform ceiling, +while an Organization may configure only a lower policy limit. Agent work uses +a durable bounded queue, fair scheduling across Organizations, FIFO within an +Organization, explicit overload/expiry/cancellation states, and no automatic +whole-run replay after execution starts. + +Hard controls cover request rate/body size, concurrency and queueing, files and +archive expansion, Project/Organization storage and entity counts, run/tool +budgets, and process resources. Reaching a limit blocks new growth without +deleting or hiding existing data. Token and provider cost remain attributed +soft-alert signals in the pilot, not hard spend controls; unknown cost is not +zero. Platform administrators also receive audited `DRAIN` and `STOP_NOW` +workload brakes. + +The full current-state inventory, accepted behavior, and release evidence are +recorded in [Initial abuse and capacity controls](../assets/initial-abuse-capacity-controls.md), +with the durable decision in ADR-0022 and `Spec.System.Capacity`. Numerical +ceilings remain open until production-like calibration. + +The implementation frontier is: + +- [Implement layered limits and multi-dimensional request rate limiting](32-layered-limits-rate-limiting.md) +- [Build the durable Organization-fair admission scheduler](33-durable-fair-admission-scheduler.md) +- [Enforce file, archive, storage, and entity budgets](34-file-storage-budgets.md) +- [Enforce Agent Run and process resource budgets](35-run-runtime-budgets.md) +- [Implement provider-mode usage attribution and soft alerts](36-usage-soft-alerts.md) +- [Add audited emergency workload brakes](37-emergency-workload-brakes.md) +- [Calibrate and prove production capacity ceilings](38-calibrate-production-capacity.md) diff --git a/.scratch/saas-production-readiness/issues/07-audit-product-surface-completeness.md b/.scratch/saas-production-readiness/issues/07-audit-product-surface-completeness.md index e3587dc..ebcf490 100644 --- a/.scratch/saas-production-readiness/issues/07-audit-product-surface-completeness.md +++ b/.scratch/saas-production-readiness/issues/07-audit-product-surface-completeness.md @@ -6,7 +6,8 @@ Status: open ## Question Against ADR-0020 and ADR-0021, which required platform-admin, org-admin, -project-onboarding, team/access, session, usage, and Feishu flows are absent, +project-onboarding, team/access, session, usage, provider-mode, Organization +policy-limit, emergency workload-control, and Feishu flows are absent, unreachable, or only test-facing rather than usable in the initial production -service? - +service? Treat ADR-0022's capacity control surfaces as part of the accepted +product boundary. diff --git a/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md b/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md index db5ec8e..85d08e3 100644 --- a/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md +++ b/.scratch/saas-production-readiness/issues/08-prove-production-release-gate.md @@ -2,7 +2,7 @@ Type: task Status: open -Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31 +Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38 ## Question diff --git a/.scratch/saas-production-readiness/issues/19-design-org-scoped-secret-control-plane.md b/.scratch/saas-production-readiness/issues/19-design-org-scoped-secret-control-plane.md index a71ac2d..798f8b7 100644 --- a/.scratch/saas-production-readiness/issues/19-design-org-scoped-secret-control-plane.md +++ b/.scratch/saas-production-readiness/issues/19-design-org-scoped-secret-control-plane.md @@ -7,6 +7,7 @@ Status: open Which key-management, envelope-encryption, rotation, connection identity, and runtime resolver design will implement ADR-0021's org-scoped Feishu and model -provider credentials, support one customer-owned Feishu app per tenant, and -ensure plaintext credentials never reach business records, logs, or agent -tools? +provider credentials; support one customer-owned Feishu app per Organization; +support Organization-managed BYOK and a distinct platform-managed provider +connection per Organization; and ensure plaintext credentials never reach +business records, logs, or agent tools? diff --git a/.scratch/saas-production-readiness/issues/23-durable-feishu-inbox-and-queue.md b/.scratch/saas-production-readiness/issues/23-durable-feishu-inbox-and-queue.md index a642f84..df4e3de 100644 --- a/.scratch/saas-production-readiness/issues/23-durable-feishu-inbox-and-queue.md +++ b/.scratch/saas-production-readiness/issues/23-durable-feishu-inbox-and-queue.md @@ -2,6 +2,7 @@ Type: task Status: open +Blocked by: 18 ## Question @@ -10,4 +11,6 @@ durable inbound work lifecycle that atomically claims duplicate events, records processing/completion/failure, survives restart, retries recoverable failures, and proves concurrent redelivery cannot duplicate a run or silently lose an accepted prompt. Preserve execution-time authorization from ticket 18 when -deferred work is finally started. +deferred work is finally started, and expose the durable identity/state needed +by ADR-0022's Organization-fair admission scheduler rather than recreating a +second queue lifecycle. diff --git a/.scratch/saas-production-readiness/issues/29-decide-telemetry-slos-alert-routing.md b/.scratch/saas-production-readiness/issues/29-decide-telemetry-slos-alert-routing.md index 414e554..489532e 100644 --- a/.scratch/saas-production-readiness/issues/29-decide-telemetry-slos-alert-routing.md +++ b/.scratch/saas-production-readiness/issues/29-decide-telemetry-slos-alert-routing.md @@ -11,4 +11,6 @@ redaction/access/retention policy, initial service-level indicators and objectives, error budgets, alert thresholds, on-call destinations, escalation ownership, and telemetry-blind detection. Prefer the smallest operable control plane, but record every domain-visible availability or retry assumption in the -appropriate ADR/spec rather than hiding it in dashboard configuration. +appropriate ADR/spec rather than hiding it in dashboard configuration. Consume +ADR-0022's already-decided usage/cost soft-alert ownership as input; do not +reopen its hard-limit or provider-mode financial-alert semantics here. diff --git a/.scratch/saas-production-readiness/issues/32-layered-limits-rate-limiting.md b/.scratch/saas-production-readiness/issues/32-layered-limits-rate-limiting.md new file mode 100644 index 0000000..974ee37 --- /dev/null +++ b/.scratch/saas-production-readiness/issues/32-layered-limits-rate-limiting.md @@ -0,0 +1,13 @@ +# Implement layered limits and multi-dimensional request rate limiting + +Type: task +Status: open +Blocked by: 20, 22 + +## Question + +Implement ADR-0022's fail-closed platform ceilings, Organization-lowerable +policy limits, bounded list/query behavior, and typed multi-dimensional rate +limits for authenticated Web, signed Feishu, OAuth, and administrative writes. +Prove effective limits never exceed platform ceilings and every overload result +is observable, scoped, and carries retry guidance where applicable. diff --git a/.scratch/saas-production-readiness/issues/33-durable-fair-admission-scheduler.md b/.scratch/saas-production-readiness/issues/33-durable-fair-admission-scheduler.md new file mode 100644 index 0000000..9a3fd3d --- /dev/null +++ b/.scratch/saas-production-readiness/issues/33-durable-fair-admission-scheduler.md @@ -0,0 +1,13 @@ +# Build the durable Organization-fair admission scheduler + +Type: task +Status: open +Blocked by: 18, 23, 24, 32 + +## Question + +Extend the durable inbound lifecycle into ADR-0022's bounded Agent Admission +Queue with platform/Organization/Project concurrency, fair scheduling across +Organizations, FIFO within each Organization, durable position/state, +expiration notification, scoped cancellation, idempotent admission, explicit +capacity rejection, and no automatic replay after a run has started. diff --git a/.scratch/saas-production-readiness/issues/34-file-storage-budgets.md b/.scratch/saas-production-readiness/issues/34-file-storage-budgets.md new file mode 100644 index 0000000..fc972c8 --- /dev/null +++ b/.scratch/saas-production-readiness/issues/34-file-storage-budgets.md @@ -0,0 +1,13 @@ +# Enforce file, archive, storage, and entity budgets + +Type: task +Status: open +Blocked by: 17, 23, 32 + +## Question + +Enforce ADR-0022's request/file/attachment/archive-expansion, +Project/Organization storage, and Organization entity-count limits with +streaming accounting, atomic reservation, cleanup of incomplete artifacts, +bounded queries, explicit errors, and continued read/export/delete/admin +remediation while an Organization is at or above a ceiling. diff --git a/.scratch/saas-production-readiness/issues/35-run-runtime-budgets.md b/.scratch/saas-production-readiness/issues/35-run-runtime-budgets.md new file mode 100644 index 0000000..44fda65 --- /dev/null +++ b/.scratch/saas-production-readiness/issues/35-run-runtime-budgets.md @@ -0,0 +1,13 @@ +# Enforce Agent Run and process resource budgets + +Type: task +Status: open +Blocked by: 16, 24, 32 + +## Question + +Implement ADR-0022's wall-time, turn, tool-count, per-tool-time, +output/log/event-size, memory, CPU, and child-process budgets. Termination must +be fail-fast and observable, persist `TIMED_OUT` or `LIMIT_EXCEEDED` with the +triggered dimension and actual usage, cancel the provider/process safely, and +release run locks and admission capacity under every injected failure. diff --git a/.scratch/saas-production-readiness/issues/36-usage-soft-alerts.md b/.scratch/saas-production-readiness/issues/36-usage-soft-alerts.md new file mode 100644 index 0000000..ffda2d1 --- /dev/null +++ b/.scratch/saas-production-readiness/issues/36-usage-soft-alerts.md @@ -0,0 +1,13 @@ +# Implement provider-mode usage attribution and soft alerts + +Type: task +Status: open +Blocked by: 19, 24, 29, 32 + +## Question + +Attribute token usage, provider-reported cost, run count, and duration by +Organization, Project, Run, model, and Provider Connection; preserve missing +cost as unknown; expose Organization-configurable soft thresholds; and route +Organization alerts plus platform-managed anomalous-cost alerts without hard +spend blocking, BYOK financial alerts, or payment-settlement semantics. diff --git a/.scratch/saas-production-readiness/issues/37-emergency-workload-brakes.md b/.scratch/saas-production-readiness/issues/37-emergency-workload-brakes.md new file mode 100644 index 0000000..776f70c --- /dev/null +++ b/.scratch/saas-production-readiness/issues/37-emergency-workload-brakes.md @@ -0,0 +1,13 @@ +# Add audited emergency workload brakes + +Type: task +Status: open +Blocked by: 22, 23, 24, 27, 30, 32, 33 + +## Question + +Implement reversible, reason-bearing, audited `DRAIN` and `STOP_NOW` controls at +Organization and platform scope. Prove new admissions, queued work, active runs, +locks, user notifications, authorization, telemetry, restart persistence, and +explicit recovery follow ADR-0022 without deleting or silently suspending +Organization data. diff --git a/.scratch/saas-production-readiness/issues/38-calibrate-production-capacity.md b/.scratch/saas-production-readiness/issues/38-calibrate-production-capacity.md new file mode 100644 index 0000000..a9df31d --- /dev/null +++ b/.scratch/saas-production-readiness/issues/38-calibrate-production-capacity.md @@ -0,0 +1,14 @@ +# Calibrate and prove production capacity ceilings + +Type: research +Status: open +Blocked by: 30, 32, 33, 34, 35, 36, 37 + +## Question + +On the accepted production-like host and provider path, measure safe numerical +defaults and headroom for every ADR-0022 platform ceiling. Exercise normal and +noisy-neighbor traffic, provider stalls and rate limits, queue saturation, +large files/archives, storage pressure, process budgets, and both workload +brakes; publish reproducible evidence and the checked-in configuration values +required by the production release gate. diff --git a/.scratch/saas-production-readiness/map.md b/.scratch/saas-production-readiness/map.md index bd6e651..1ab5dcd 100644 --- a/.scratch/saas-production-readiness/map.md +++ b/.scratch/saas-production-readiness/map.md @@ -5,18 +5,25 @@ Label: wayfinder:map ## Destination The repository can be deployed from a clean checkout as the initial production -multi-tenant Curriculum Project Hub, operated safely on the ADR-0018/0020/0021 -single-host Linux topology, with repeatable release evidence, rollback and -recovery procedures, and no known critical security or data-integrity gaps. +multi-tenant Curriculum Project Hub: platform staff manage Organizations; every +Organization authenticates through and operates its own customer-owned Feishu +app/bot, and selects either an Organization-managed BYOK provider connection or +a distinct platform-managed provider connection; Organization owners/admins +manage their own members, projects and settings. The ADR-0018/0020/0021 +single-host Linux topology is operated with repeatable release evidence, +rollback and recovery, and no known critical security or data-integrity gaps. ## Notes - `spec/` remains the semantic source of truth; implementation drift must be surfaced rather than silently resolved. - The initial product boundary follows the accepted ADRs: Organization is the - tenant root, customer onboarding may be manual, and full billing, limits, - tenant suspension, and self-service platform administration remain deferred - unless a readiness investigation proves they are required for safe operation. + tenant root and customer onboarding may be manual. Full billing, tenant + suspension, and self-service platform administration remain deferred; + ADR-0022's safety limits and workload brakes are required for production. +- Manual guided setup may provision each Organization's customer-owned Feishu + app during the pilot; one process-global Feishu/OAuth credential shared by + unrelated Organizations is not an acceptable destination state. - Use the `diagnosing-bugs` loop for every reproduced defect. Prefer explicit failures and observable critical paths over fallback behavior. - Work on `saas-production-readiness`; make small conventional commits after @@ -32,6 +39,7 @@ recovery procedures, and no known critical security or data-integrity gaps. - [Audit tenant, authentication, and request security boundaries](issues/02-audit-tenant-auth-security.md) — retain the working org-admin/application authorization core, but production is blocked on agent/MCP isolation, execution-time object binding, org-scoped encrypted credentials, tenant status, browser sessions, and clean dependency audits. - [Audit run lifecycle and data integrity under failure](issues/03-audit-run-lifecycle-integrity.md) — retain the database-enforced project-lock uniqueness and normal happy path, but replace receipt-before-success, process-local queues, independent run/session/lock effects, silent delivery/audit loss, and partial startup reset with durable, recoverable state machines; explicitly decide failed-run workspace semantics. - [Define production observability and incident recovery](issues/04-define-observability-recovery.md) — keep the existing Fastify/Prisma/systemd signal fragments, but production requires dependency-aware readiness, typed and correlated errors, durable-work telemetry, an explicit SLO/retention/alerting control plane, and recovery runbooks proven by fault drills. +- [Define initial abuse and capacity controls](issues/05-define-abuse-capacity-controls.md) — require layered fail-closed limits, durable Organization-fair admission, explicit backpressure, bounded files/storage/runs, soft usage alerts, and audited emergency workload brakes; calibrate numerical ceilings under production-like load. ## Fog diff --git a/AGENTS.md b/AGENTS.md index 9222db5..6f1cd10 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,6 +11,12 @@ - org 后台 project explorer 里 `Folder` 是透明组织节点,不是权限资源;project 仍是权限边界。 普通老师可在飞书群自助建 project 但受 org policy 控制(见 ADR-0021 / `Spec.System.ProjectWorkspace`)。 +- 每个 org 自选 BYOK 或平台托管 model provider connection;平台托管也必须是该 org + 独享的 key/base URL,不得让无关 org 共用 process-global provider key(见 ADR-0021 / + `Spec.System.Organization`)。 +- 生产容量按不可突破的 platform ceiling 与 org 可下调 policy 分层;有效限制取两者较低值。 + Agent admission 必须持久、有界、跨 org 公平且显式背压(见 ADR-0022 / + `Spec.System.Capacity`)。 ## 纪律 diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..1312ff7 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,77 @@ +# Curriculum Project Hub + +The shared language for the multi-tenant service that lets customer organizations manage curriculum projects and operate agents through their own Feishu connection. + +## Language + +**Organization**: +The customer tenant root that owns its memberships, projects, teams, external connections, and settings. +_Avoid_: Account, workspace + +**Platform Administrator**: +An internal operator who creates and operates Organizations without becoming a member of them. +_Avoid_: Organization admin, super member + +**Organization Administrator**: +An Organization OWNER or ADMIN who manages only that Organization's members, projects, connections, and settings. +_Avoid_: Platform admin + +**Admin Surface**: +The private web control plane available only to an authenticated user with the corresponding Platform Administrator or Organization Administrator authority. +_Avoid_: Public management page, member portal + +**Customer-owned Feishu Application**: +The Organization-scoped Feishu application used for that Organization's login, bot messages, and directory integration; unrelated Organizations never share it. +_Avoid_: Global Feishu app, platform bot + +**Provider Credential Mode**: +The Organization's choice between a BYOK Provider Connection and a Platform-managed Provider Connection. +_Avoid_: Global provider setting + +**BYOK Provider Connection**: +An Organization-scoped model-provider connection whose key and base URL are supplied and managed by an Organization Administrator. +_Avoid_: Platform key + +**Platform-managed Provider Connection**: +An Organization-scoped model-provider connection whose distinct key and base URL are supplied and managed by a Platform Administrator for that Organization. +_Avoid_: Shared platform key, BYOK + +**Platform Capacity Ceiling**: +A non-overridable upper bound imposed by the platform to protect shared service capacity. +_Avoid_: Organization quota + +**Organization Policy Limit**: +An optional lower bound configured by an Organization Administrator to govern that Organization's own usage. +_Avoid_: Platform limit + +**Effective Limit**: +The lower of the Platform Capacity Ceiling and the Organization Policy Limit; when no Organization Policy Limit exists, the Platform Capacity Ceiling applies. +_Avoid_: Unlimited default + +**Organization-fair Scheduling**: +The policy that allocates available agent execution capacity fairly across Organizations, while preserving first-in-first-out order within each Organization. +_Avoid_: Global FIFO + +**Agent Admission Queue**: +The durable, bounded set of accepted agent run requests waiting for execution capacity. +_Avoid_: In-memory trigger queue, unlimited backlog + +**Expired Run Request**: +An accepted agent run request that reached its maximum queue wait before execution; it is terminal and will never execute automatically. +_Avoid_: Delayed request, automatic retry + +**Run Request Initiator**: +The Organization member whose action created an agent run request. +_Avoid_: Queue owner, Project owner + +**Explicit Run Retry**: +A new agent run request deliberately created after an earlier run failed or timed out. +_Avoid_: Automatic replay, implicit rerun + +**Run Budget**: +The deterministic resource boundary for one agent run, independent of monetary or token usage alerts. +_Avoid_: Cost budget, unlimited run + +**Emergency Workload Brake**: +An audited Platform Administrator control that prevents new agent work for one Organization or the whole platform and may explicitly stop active work during an incident. +_Avoid_: Organization deletion, service restart diff --git a/docs/adr/0018-agent-execution-surface-bounded-by-workspace.md b/docs/adr/0018-agent-execution-surface-bounded-by-workspace.md index 322f013..01b27fb 100644 --- a/docs/adr/0018-agent-execution-surface-bounded-by-workspace.md +++ b/docs/adr/0018-agent-execution-surface-bounded-by-workspace.md @@ -158,9 +158,12 @@ as it upholds `AgentFileOp.Authorized`. *effects* are then confined, is a policy choice left to implementation. The invariant governs effects either way; current choice is arbitrary commands (sandbox confines effects). -- **Per-user rate limiting and token budgets.** Orthogonal to the surface - boundary (this ADR is about *what* the agent may touch, not *how often* it - may run). Deferred to a separate concern. -- **Inbound file size / MIME limits.** The inbox lives inside the boundary, - but unbounded file size is a DoS vector distinct from the boundary - invariant. Deferred. + +## Related Decisions + +- **Rate limiting and token usage.** ADR-0022 requires multi-dimensional hard + request limits while keeping token thresholds as attributed soft alerts in + the initial release. +- **Inbound file limits.** ADR-0022 requires hard file, attachment, archive, + Project-storage, and Organization-storage limits with explicit failure and + partial-file cleanup. diff --git a/docs/adr/0020-organization-is-saas-tenant-root.md b/docs/adr/0020-organization-is-saas-tenant-root.md index 68b0df6..e7e8358 100644 --- a/docs/adr/0020-organization-is-saas-tenant-root.md +++ b/docs/adr/0020-organization-is-saas-tenant-root.md @@ -83,8 +83,10 @@ separate control plane and must not be modeled as project `MANAGE`. ## Open Questions / Deferred -- Full platform admin UI, billing, limits, and tenant suspension workflows are - deferred. +- Full platform admin UI, billing, and tenant suspension workflows are + deferred. Production safety limits and workload brakes are no longer + deferred; ADR-0022 defines them without treating a workload brake as tenant + suspension. - Whether direct `USER` project grants should require active `OrganizationMembership` is deferred; explicit direct grants remain allowed for now. diff --git a/docs/adr/0021-org-admin-project-onboarding.md b/docs/adr/0021-org-admin-project-onboarding.md index 5fc0db1..3752798 100644 --- a/docs/adr/0021-org-admin-project-onboarding.md +++ b/docs/adr/0021-org-admin-project-onboarding.md @@ -11,8 +11,8 @@ surface is not only the Feishu bot trigger path, but also the control planes around it: - platform staff need a private platform admin area to create and operate orgs; -- customer org owners/admins need an org admin area for teams, roles, model - provider configuration, projects, folders, sessions, and usage accounting; +- customer org owners/admins need an org admin area for teams, roles, provider + mode and BYOK configuration, projects, folders, sessions, and usage accounting; - ordinary teachers should be able to start work from the natural Feishu group flow without entering the web backend. @@ -46,6 +46,19 @@ OAuth login, bot messages, and directory sync. App secrets are org-scoped secrets and must be encrypted at rest; business code should access them through a secret/connection resolver, not raw plaintext columns. +Each Organization chooses one model-provider credential mode: + +- `BYOK`: an Organization OWNER/ADMIN supplies and manages that Organization's + provider key and base URL in the org admin area; +- `PLATFORM_MANAGED`: a platform admin supplies and manages one distinct key + and base URL for that Organization in the platform admin area. Organization + users cannot read the plaintext credential. + +Provider connections remain Organization-scoped in both modes. Unrelated +Organizations never share a process-global provider key. Provider secrets must +be encrypted at rest and resolved at runtime through the same connection/secret +boundary as Feishu credentials. + Project management has two creation paths: - org owner/admin creates projects in the org web backend; @@ -77,9 +90,10 @@ aggregation: Project permissions remain on `PROJECT` resources. Moving a project between folders does not change grants. -Usage accounting is org-wise and project/folder aggregatable. It is not payment -collection in the pilot because customers supply their own model provider API -keys and base URLs. +Usage accounting is org-wise and project/folder aggregatable for both provider +modes. It is not payment collection in the pilot: BYOK spend belongs to the +customer's provider account, while platform-managed spend is attributed to the +Organization for operational reporting; commercial billing remains deferred. ## Consequences @@ -89,8 +103,8 @@ keys and base URLs. explorer without creating a second ACL system. - Service code should expose project creation and chat binding as reusable backend operations so Feishu cards and future web APIs call the same rules. -- Org role/model/provider/billing panels can be added on top of the org tenant - root without changing project authorization. +- Org role/model/provider-mode panels and the platform-managed connection panel + can be added on top of the org tenant root without changing project authorization. ## Open Questions / Deferred diff --git a/docs/adr/0022-layered-capacity-admission.md b/docs/adr/0022-layered-capacity-admission.md new file mode 100644 index 0000000..5c102f0 --- /dev/null +++ b/docs/adr/0022-layered-capacity-admission.md @@ -0,0 +1,88 @@ +# ADR 0022: Layered Capacity Admission And Workload Safety + +## Status + +Accepted. + +## Context + +The initial SaaS deployment shares one production host across Organizations. +The current Hub only has a one-run-per-project lock, a process-local five-item +trigger queue, a five-minute queue wait, a global max-turn setting, and +Fastify's default request-body limit. It has no global or Organization agent +capacity, durable backpressure, fair scheduling, storage quotas, run wall-clock +budget, multi-dimensional rate limiting, or emergency workload control. + +Leaving these limits implicit lets one Organization monopolize the host, lets +accepted work disappear on restart, and makes overload look like success. A +single global FIFO would preserve arrival order while still permitting one +tenant to occupy every execution slot. Unlimited queues or automatic whole-run +replay would hide overload and can duplicate tool side effects. + +## Decision + +Every mandatory capacity dimension has a non-optional **Platform Capacity +Ceiling**. An Organization OWNER/ADMIN may configure a lower **Organization +Policy Limit**; the **Effective Limit** is the lower value, or the platform +ceiling when the Organization value is absent. There is no unlimited fallback. +Production refuses to start when a mandatory platform ceiling is absent or +invalid. Platform ceilings are changed through versioned deployment +configuration in the initial release; live platform-admin editing is deferred +until audited approval and rollback exist. + +Agent work uses a durable, bounded admission queue. Scheduling is fair across +Organizations and FIFO within an Organization, while the existing one-active- +run-per-Project rule remains. Capacity saturation queues only within effective +length and wait limits. Queue overflow is an explicit `capacity_exhausted` +response with retry guidance; an accepted request that waits too long becomes +`EXPIRED`, is never started later, and notifies its initiator. Initiators may +withdraw their own queued requests; Project `MANAGE` and Organization admins +may cancel queued requests in their scope. Every transition is auditable. + +The platform never automatically replays a whole Agent Run after execution has +started. Limited transport retries are allowed only before model output or tool +side effects. Otherwise the run ends explicitly and a retry is a new user- +initiated request. Feishu event identity and API idempotency prevent delivery +retries from creating duplicate admissions. + +The first production gate requires hard limits for request rate/body size, +agent concurrency, queue length/wait, files and attachment counts, archive +expansion, Project and Organization storage, Organization entity counts, run +wall time/turns/tool calls/output, individual tool duration, and execution +process memory/CPU/process count. Reaching an entity or storage ceiling blocks +new growth but does not delete, hide, or lock existing data; read, export, +delete, and administrator remediation remain available. + +Admin surfaces require an authenticated user plus the corresponding platform +or Organization administrator authorization. Authenticated Web traffic is +limited by user, Organization, and endpoint class; signed Feishu traffic is +limited by Organization application, chat, and sender; administrative writes +use separate stricter buckets. Rejections and truncation are never silent. + +Token, provider-reported cost, run count, and duration are attributed by +Organization, Project, Run, model, and Provider Connection. They are statistics +and soft alerts in the initial release, not hard spend controls or settlement. +Missing provider cost remains unknown rather than zero. Organization admins +receive their alerts; platform admins additionally receive anomalous-cost +alerts for platform-managed Provider Connections, not BYOK financial alerts. + +Platform administrators have audited, reason-bearing, reversible workload +brakes at Organization and platform scope. `DRAIN` rejects new work and pauses +queue starts while allowing active runs to finish. `STOP_NOW` also cancels +active runs. These controls do not delete or suspend the Organization. + +Exact numerical ceilings remain open until production-like capacity tests +measure the supported host and provider behavior. Defaults must be checked in, +validated, observable, and proven at the release gate rather than guessed as +domain constants. + +## Consequences + +- The current in-memory `TriggerQueue` and global-only agent settings are + implementation divergences, not acceptable production fallbacks. +- Durable ingress, run/lock recovery, authorization, telemetry, and capacity + enforcement must share one Organization-scoped admission identity. +- HTTP overload uses typed `429` responses with `Retry-After`; Feishu reports a + clear busy/expired/canceled outcome to the initiating user. +- Capacity calibration is a release prerequisite and must include noisy-neighbor, + restart, queue expiry, file/archive, provider-stall, and workload-brake tests. diff --git a/hub/src/org/usage.ts b/hub/src/org/usage.ts index 4050e51..ea32d69 100644 --- a/hub/src/org/usage.ts +++ b/hub/src/org/usage.ts @@ -1,6 +1,8 @@ /** * Org / project usage rollups from AgentRun cost facts (ADR-0021). - * Not payment collection — customers may supply their own provider keys. + * Operational usage accounting, not payment collection. Organizations may use + * either their own provider credentials or a distinct platform-managed + * connection; commercial billing remains outside the pilot scope. */ import type { Prisma, PrismaClient } from "@prisma/client"; diff --git a/spec/Spec/System.lean b/spec/Spec/System.lean index 9e3a899..ca55f45 100644 --- a/spec/Spec/System.lean +++ b/spec/Spec/System.lean @@ -1,6 +1,7 @@ import Spec.System.ProjectGroup import Spec.System.Organization import Spec.System.ProjectWorkspace +import Spec.System.Capacity import Spec.System.Run import Spec.System.Lock import Spec.System.Memory @@ -19,6 +20,8 @@ import Spec.System.Audit - `Organization` —— SaaS tenant root(ADR-0020);project/team 单归属,TEAM grant 不跨 org。 - `ProjectWorkspace` —— org 后台 project explorer:folder 是透明组织节点,project 仍是权限边界 (ADR-0021)。 +- `Capacity` —— platform ceiling 与 org policy 的分层限制、持久 admission request 状态和 + 平台紧急工作负载制动(ADR-0022)。 - `Run` —— AgentRun 状态与终止判定(状态集合完整性 OPEN)。 - `Lock` —— 锁 owner=run(ADR-0002),及"持锁者必为非终止 run"的核心不变式。 - `Memory` —— 按需上下文:锚点类别(ADR-0003)+ MCP 工具按 run/project 上下文授权的不变式。 @@ -29,5 +32,5 @@ import Spec.System.Audit (ADR-0004);role-capability 与 settings-policy 的组合规则 OPEN。 - `Audit` —— 有意从简(内容多为 plumbing,OPEN)。 -标识符见 `Spec.Prelude`。决策出处:ADR-0001..0004, 0018, 0020, 0021。 +标识符见 `Spec.Prelude`。决策出处:ADR-0001..0004, 0018, 0020..0022。 -/ diff --git a/spec/Spec/System/Capacity.lean b/spec/Spec/System/Capacity.lean new file mode 100644 index 0000000..065cd5d --- /dev/null +++ b/spec/Spec/System/Capacity.lean @@ -0,0 +1,89 @@ +import Spec.Prelude + +/-! +# Capacity —— SaaS capacity admission and abuse controls (ADR-0022) + +初始生产服务共享有限的单机资源,但不能让一个 Organization 垄断容量或让无界输入拖垮 +其他租户。ADR-0022 钉死分层限制、持久 admission、显式背压和紧急制动的领域语义; +具体数值必须由生产式容量测试校准,因此保持 `OPEN`,不得把未经验证的数字冒充契约。 +-/ + +namespace Spec.System + +/-- 首个生产版本必须有硬边界的容量维度(`PINNED`, ADR-0022)。金额与 token 用量是 +统计和软告警,不在本硬限制集合中;各维度的具体数值为 `OPEN`,须由容量测试决定。 -/ +inductive CapacityDimension where + | requestRate + | requestBodySize + | agentConcurrency + | admissionQueueLength + | admissionQueueWait + | fileSize + | attachmentCount + | archiveExpansion + | projectStorage + | organizationStorage + | memberCount + | projectCount + | teamCount + | folderCount + | sessionCount + | runWallTime + | runTurns + | runToolCalls + | toolWallTime + | runOutputSize + | processMemory + | processCpu + | processCount + +/-- 一个容量维度的分层限制(`PINNED`, ADR-0022):platform ceiling 永远存在且不可被 +Organization 突破;Organization policy 可以缺省,也只能配置更低的限制。 -/ +structure LayeredLimit where + /-- 由版本化生产部署配置提供的不可突破上限(`PINNED`, ADR-0022)。 -/ + platformCeiling : Nat + /-- Organization 管理员可选的更低策略限制(`PINNED`, ADR-0022)。 -/ + organizationLimit : Option Nat + +/-- 分层限制是良构的 iff Organization policy 未设置或不高于 platform ceiling +(`PINNED`, ADR-0022)。不允许用 `none` 表示 platform unlimited。 -/ +def LayeredLimit.Valid (limit : LayeredLimit) : Prop := + match limit.organizationLimit with + | none => True + | some organizationLimit => organizationLimit ≤ limit.platformCeiling + +/-- 有效限制(`PINNED`, ADR-0022)取 platform ceiling 与 Organization policy 中较低者; +Organization policy 缺省时直接采用 platform ceiling,永不退化为 unlimited。 -/ +def LayeredLimit.effective (limit : LayeredLimit) : Nat := + match limit.organizationLimit with + | none => limit.platformCeiling + | some organizationLimit => min limit.platformCeiling organizationLimit + +/-- 已被服务接受的 agent run request 状态(`PINNED`, ADR-0022)。`expired` 和 +`canceled` 都是可审计终态且不得自动执行;`started` 表示已从 admission queue 移交给 +一个 AgentRun。被容量规则拒绝的输入从未被接受,因此不伪装成 queued request。 -/ +inductive RunRequestState where + | queued + | started + | expired + | canceled + +/-- 平台紧急工作负载制动模式(`PINNED`, ADR-0022)。`drain` 停止新 admission 与队列 +启动但允许当前 run 完成;`stopNow` 还会取消当前 run。它们不等同于删除或封禁 org。 -/ +inductive WorkloadBrakeMode where + | open + | drain + | stopNow + +/-- 紧急制动是否允许接收新的 agent 工作(`PINNED`, ADR-0022):只有 `open` 允许。 -/ +def WorkloadBrakeMode.AcceptsNew : WorkloadBrakeMode → Prop + | .open => True + | .drain | .stopNow => False + +/-- 紧急制动是否允许当前 agent run 继续(`PINNED`, ADR-0022):`drain` 允许收尾, +`stopNow` 要求取消。 -/ +def WorkloadBrakeMode.AllowsActive : WorkloadBrakeMode → Prop + | .open | .drain => True + | .stopNow => False + +end Spec.System diff --git a/spec/Spec/System/Organization.lean b/spec/Spec/System/Organization.lean index 3343dd6..499fd38 100644 --- a/spec/Spec/System/Organization.lean +++ b/spec/Spec/System/Organization.lean @@ -9,7 +9,8 @@ ADR-0020:Hub 长期按 SaaS 形态演进,`Organization` 是客户侧 tenant root 不复用 project 的 `read/edit/manage` 角色格,而是另一个控制面。 本模块只钉死会造成实现分歧的不变量:tenant root 存在、project/team 单归属、team grant -不得跨 org。用户身份、外部目录 provider 细节、平台控制面角色集合仍为 `OPEN`。 +不得跨 org,以及 model provider connection 的凭据归属模式。用户身份、外部目录 +provider 细节、平台控制面角色集合仍为 `OPEN`。 -/ namespace Spec.System @@ -48,4 +49,20 @@ def TeamProjectGrantScope.WellScoped (teamOrg : I.TeamId → Option I.OrganizationId) : Prop := ∃ o, projectOrg grant.project = some o ∧ teamOrg grant.team = some o +/-- Organization 的 model provider 凭据归属模式(`PINNED`, ADR-0021):BYOK 由 org +管理员提供和管理;platform-managed 由平台管理员为该 org 单独提供和管理。两种模式 +都不允许无关 org 共用 process-global provider key。模式切换过程仍为 `OPEN`。 -/ +inductive ProviderCredentialMode where + | byok + | platformManaged + +/-- Organization 的 model provider connection(`PINNED`, ADR-0021):connection 必须 +归属一个且仅一个 organization,并明确采用哪一种凭据模式。key/base URL 的加密表示、 +轮换与 resolver 机制由后续 secret-control-plane 决策规定。 -/ +structure OrganizationProviderConnection where + /-- connection 所属 organization(`PINNED`, ADR-0021)。 -/ + organization : I.OrganizationId + /-- connection 的凭据归属模式(`PINNED`, ADR-0021)。 -/ + mode : ProviderCredentialMode + end Spec.System diff --git a/spec/Spec/System/Run.lean b/spec/Spec/System/Run.lean index 7662200..4ecbd7d 100644 --- a/spec/Spec/System/Run.lean +++ b/spec/Spec/System/Run.lean @@ -8,21 +8,23 @@ namespace Spec.System -/-- AgentRun 运行状态(状态名 `PINNED`, ADR-0001..0003 + likec4;**完整性 `OPEN`** -——散文从未声明"状态恰好这些";实现若需新状态(如 pending)须 surface,不得默认 -本枚举已穷尽)。终止态见 `RunState.Terminal`。 -/ +/-- AgentRun 运行状态(状态名 `PINNED`, ADR-0001..0003, ADR-0022 + likec4; +**完整性 `OPEN`**——散文从未声明"状态恰好这些";实现若需新状态(如 pending)须 +surface,不得默认本枚举已穷尽)。终止态见 `RunState.Terminal`。 -/ inductive RunState where | active | waitingForUser | completed | failed | timedOut + | limitExceeded | canceled -/-- run 处于**终止态**(`PINNED`, ADR-0002:锁在 completes/fails/timesOut/canceled -时释放)。`active`/`waitingForUser` 非终止——后者仍占用项目(锁未释放)。 -/ +/-- run 处于**终止态**(`PINNED`, ADR-0002, ADR-0022:锁在 completes/fails/ +timesOut/limitExceeded/canceled 时释放)。`active`/`waitingForUser` 非终止——后者仍 +占用项目(锁未释放)。 -/ def RunState.Terminal : RunState → Prop - | .completed | .failed | .timedOut | .canceled => True + | .completed | .failed | .timedOut | .limitExceeded | .canceled => True | .active | .waitingForUser => False end Spec.System