docs: define SaaS capacity controls

2026-07-10 12:07:08 +08:00
parent 16329df357
commit 4ca4afb141
27 changed files with 618 additions and 47 deletions
@@ -0,0 +1,125 @@
+# Initial abuse and capacity controls
+
+## Verdict
+
+The initial production boundary requires hard safety controls before customer
+traffic. The current project lock and small in-memory trigger queue are useful
+local protections, but they do not provide multi-tenant admission, durable
+backpressure, storage safety, request isolation, bounded execution, or an
+operator stop mechanism.
+
+The accepted contract is recorded in
+[ADR-0022](../../../docs/adr/0022-layered-capacity-admission.md) and
+[`Spec.System.Capacity`](../../../spec/Spec/System/Capacity.lean). Exact values
+remain deliberately open until production-like calibration; the required
+dimensions and failure behaviors do not.
+
+## Current implementation inventory
+
+| Surface | Existing protection | Production gap |
+| --- | --- | --- |
+| HTTP | Fastify's default 1 MiB body limit | No route-class, user, Org, IP, or administrative-write rate limits; request/connection/handler timeouts are unset |
+| Project execution | Database lock permits one active run per Project | No platform or Org concurrency ceiling and no fair scheduler |
+| Busy Project | Process-local FIFO, default five items and five-minute wait | Restart loses accepted work; expiry is logged and dropped without a durable state or user notification |
+| Agent execution | Global max-turn default of 25 | No wall-clock, tool-call, output, memory, CPU, process, or Org-specific budget |
+| Feishu messages | Message batching limits text count/characters | File/post attachment count and size are unbounded; downloads may buffer whole files |
+| Storage | Workspace path boundary | No Project or Org byte quota, archive-expansion limit, or entity-count ceiling |
+| Usage | Provider-reported token/cost fields and Org aggregation | Missing cost can disappear from totals; no soft-threshold notification or provider-mode attribution contract |
+| Host | systemd lifecycle | No service CPU, memory, or task ceiling and no audited workload brake |
+
+Most Organization member/team/project lists are also unpaginated, and usage
+aggregation can load all matching runs. Those query shapes must be bounded as
+part of enforcement rather than hidden behind larger process memory.
+
+## Accepted contract
+
+### Layered limits
+
+- Every mandatory dimension has a versioned, non-overridable platform ceiling.
+- An Organization OWNER/ADMIN may configure a lower policy limit.
+- Effective limit is `min(platform, organization)`; absent Organization policy
+  means the platform ceiling, never unlimited.
+- Missing or invalid mandatory production ceilings fail startup.
+
+### Admission and fairness
+
+- Project exclusivity remains one active run.
+- Platform and Organization concurrency limits apply before starting work.
+- Available slots are scheduled fairly across Organizations, FIFO within an
+  Organization, so one tenant cannot monopolize a global FIFO.
+- Accepted work is durable and queryable. Queue length and wait are bounded.
+- Overflow is explicit `capacity_exhausted` (`429` plus `Retry-After` for HTTP;
+  a clear busy response for Feishu). Nothing is silently dropped.
+- Wait timeout produces terminal `EXPIRED`, not a later automatic start.
+- The initiator may withdraw their queued request. Project `MANAGE` or an
+  Organization admin may cancel queued requests in their scope. Actor, reason,
+  and notification are recorded.
+
+### Retry and idempotency
+
+- Feishu `event_id` and API idempotency identify one admission across retries.
+- Once a run starts, the whole run is never automatically replayed because tool
+  side effects may already exist.
+- Only bounded transport retries before model output or tool side effects are
+  automatic. Later failure/timeouts require an explicit new request.
+
+### Request, file, storage, and entity safety
+
+- Authenticated Web requests are limited by user, Organization, and endpoint
+  class; signed Feishu events by Organization app, chat, and sender;
+  administrative writes use stricter independent buckets.
+- Admin surfaces require both a valid user session and the corresponding
+  platform/Organization administrator authorization. OAuth callbacks and
+  signed Feishu ingress remain necessary non-session entry points.
+- Hard limits cover request body, single-file bytes, attachment count, archive
+  expanded bytes/file count/depth, Project/Organization storage, and per-type
+  Organization entity counts.
+- Transfer is streamed and counted; partial files are removed after failure.
+- Crossing a ceiling blocks new growth without auto-deleting or hiding existing
+  resources. Read, export, delete, and administrator remediation stay usable.
+
+### Run budget
+
+Every run is bounded by wall time, model turns, tool-call count, per-tool time,
+output/log/event bytes, and process memory/CPU/child-process count. A breached
+budget ends explicitly as `TIMED_OUT` or `LIMIT_EXCEEDED`, records the dimension
+and observed usage, releases capacity, and never masquerades as success.
+
+### Usage and cost
+
+- Token usage, provider-reported cost, run count, and duration are attributed by
+  Organization, Project, Run, model, and Provider Connection.
+- Missing provider cost is `unknown`, never zero.
+- Thresholds are soft alerts in the pilot; they do not automatically stop runs
+  and are not payment settlement.
+- Org admins receive Organization alerts. Platform admins additionally receive
+  anomalous-cost alerts for platform-managed connections; BYOK does not trigger
+  platform financial alerts.
+
+### Emergency control
+
+An audited Platform Administrator control supports Organization or global
+`DRAIN` (reject new work, pause queue starts, finish active runs) and `STOP_NOW`
+(also cancel active runs). Both require an actor, reason, timestamp, explicit
+recovery, and user-visible outcomes. This is workload control, not Organization
+deletion or account suspension.
+
+## Required production evidence
+
+1. Production configuration validation fails closed for every missing or
+   invalid platform ceiling.
+2. Concurrent noisy-neighbor tests prove platform/Org/Project concurrency and
+   Organization-fair scheduling without starvation.
+3. Restart and crash tests prove accepted queue entries, idempotency, expiry,
+   cancellation, and notifications survive process loss.
+4. HTTP and Feishu load tests prove every rate-limit identity and typed overload
+   response, including retry guidance.
+5. File/archive/storage probes prove streaming enforcement, cleanup, and
+   continued read/delete access while over limit.
+6. Hung provider/tool and resource-exhaustion probes prove every run/process
+   budget terminates, records cause, and releases locks/capacity.
+7. Usage tests prove provider-mode attribution, unknown-cost handling, alert
+   routing, and absence of hard monetary blocking.
+8. `DRAIN` and `STOP_NOW` drills prove scoped, audited, reversible behavior.
+9. Production-like calibration publishes the tested numerical defaults and
+   capacity headroom used by the release gate.
@@ -29,8 +29,10 @@ The audit used these upstream requirements:
  run's project workspace.
 - ADR-0019: authorization is evaluated against principals resolved at request
  time so membership changes take effect immediately.
- ADR-0021: customer Feishu and model-provider credentials are org-scoped
-  secrets, encrypted at rest, and accessed through a resolver.
+- ADR-0021: Feishu and model-provider credentials are org-scoped secrets,
+  encrypted at rest, and accessed through a resolver. Provider ownership may be
+  Organization-managed BYOK or a distinct platform-managed connection for that
+  Organization; neither mode permits a process-global shared key.

 ## Confirmed protections

@@ -130,10 +132,10 @@ encrypted credentials "deferred", although ADR-0021 says they **must** be
 org-scoped, encrypted at rest, and accessed through a resolver.

 `EnvRuntimeSettings.provider` similarly discards its `scope` argument and uses
-one global OpenRouter token for every project. This both leaks tenant ownership
-of credentials and makes the accepted "customers supply their own provider
-keys" accounting model false. The schema has no encrypted credential store,
-key version, rotation state, or resolver seam.
+one global OpenRouter token for every project. This erases the accepted
+per-Organization provider mode and makes both BYOK and platform-managed cost
+attribution false. The schema has no encrypted credential store, key version,
+rotation state, or resolver seam.

 This is also a product reachability blocker: one customer-owned Feishu app
 cannot receive bot/OAuth traffic for unrelated customer tenants under the
@@ -248,7 +250,9 @@ Production readiness requires automated proof of all of the following:
   organization, project, run, and pending action consistently.
 4. ACTIVE Organization status is enforced in the shared authorization seam.
 5. Feishu and provider credentials resolve by Organization, are encrypted at
-   rest with versioned keys, never enter logs/agent tools, and can rotate.
+   rest with versioned keys, never enter logs/agent tools, and can rotate;
+   provider resolution honors either Organization-managed BYOK or a distinct
+   platform-managed connection for that Organization.
 6. Cross-org HTTP and Prisma service tests remain green, while new negative
   tests cover every finding above.
 7. Production Node and Rust dependency audits pass with no ignored
@@ -1,7 +1,7 @@
 # Define initial abuse and capacity controls

 Type: grilling
-Status: open
+Status: resolved
 Blocked by: 02, 03

 ## Question
@@ -9,3 +9,35 @@ Blocked by: 02, 03
 Which request limits, file limits, concurrency limits, agent budgets, tenant
 quotas, and backpressure policies are mandatory for the accepted initial
 production boundary, and which values or behaviors remain product decisions?
+
+## Answer
+
+The initial production service requires layered, fail-closed capacity controls:
+every mandatory dimension has a versioned non-overridable platform ceiling,
+while an Organization may configure only a lower policy limit. Agent work uses
+a durable bounded queue, fair scheduling across Organizations, FIFO within an
+Organization, explicit overload/expiry/cancellation states, and no automatic
+whole-run replay after execution starts.
+
+Hard controls cover request rate/body size, concurrency and queueing, files and
+archive expansion, Project/Organization storage and entity counts, run/tool
+budgets, and process resources. Reaching a limit blocks new growth without
+deleting or hiding existing data. Token and provider cost remain attributed
+soft-alert signals in the pilot, not hard spend controls; unknown cost is not
+zero. Platform administrators also receive audited `DRAIN` and `STOP_NOW`
+workload brakes.
+
+The full current-state inventory, accepted behavior, and release evidence are
+recorded in [Initial abuse and capacity controls](../assets/initial-abuse-capacity-controls.md),
+with the durable decision in ADR-0022 and `Spec.System.Capacity`. Numerical
+ceilings remain open until production-like calibration.
+
+The implementation frontier is:
+
+- [Implement layered limits and multi-dimensional request rate limiting](32-layered-limits-rate-limiting.md)
+- [Build the durable Organization-fair admission scheduler](33-durable-fair-admission-scheduler.md)
+- [Enforce file, archive, storage, and entity budgets](34-file-storage-budgets.md)
+- [Enforce Agent Run and process resource budgets](35-run-runtime-budgets.md)
+- [Implement provider-mode usage attribution and soft alerts](36-usage-soft-alerts.md)
+- [Add audited emergency workload brakes](37-emergency-workload-brakes.md)
+- [Calibrate and prove production capacity ceilings](38-calibrate-production-capacity.md)
@@ -6,7 +6,8 @@ Status: open
 ## Question

 Against ADR-0020 and ADR-0021, which required platform-admin, org-admin,
-project-onboarding, team/access, session, usage, and Feishu flows are absent,
+project-onboarding, team/access, session, usage, provider-mode, Organization
+policy-limit, emergency workload-control, and Feishu flows are absent,
 unreachable, or only test-facing rather than usable in the initial production
-service?
-
+service? Treat ADR-0022's capacity control surfaces as part of the accepted
+product boundary.
@@ -2,7 +2,7 @@

 Type: task
 Status: open
-Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31
+Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38

 ## Question

@@ -7,6 +7,7 @@ Status: open

 Which key-management, envelope-encryption, rotation, connection identity, and
 runtime resolver design will implement ADR-0021's org-scoped Feishu and model
-provider credentials, support one customer-owned Feishu app per tenant, and
-ensure plaintext credentials never reach business records, logs, or agent
-tools?
+provider credentials; support one customer-owned Feishu app per Organization;
+support Organization-managed BYOK and a distinct platform-managed provider
+connection per Organization; and ensure plaintext credentials never reach
+business records, logs, or agent tools?
@@ -2,6 +2,7 @@

 Type: task
 Status: open
+Blocked by: 18

 ## Question

@@ -10,4 +11,6 @@ durable inbound work lifecycle that atomically claims duplicate events, records
 processing/completion/failure, survives restart, retries recoverable failures,
 and proves concurrent redelivery cannot duplicate a run or silently lose an
 accepted prompt. Preserve execution-time authorization from ticket 18 when
-deferred work is finally started.
+deferred work is finally started, and expose the durable identity/state needed
+by ADR-0022's Organization-fair admission scheduler rather than recreating a
+second queue lifecycle.
@@ -11,4 +11,6 @@ redaction/access/retention policy, initial service-level indicators and
 objectives, error budgets, alert thresholds, on-call destinations, escalation
 ownership, and telemetry-blind detection. Prefer the smallest operable control
 plane, but record every domain-visible availability or retry assumption in the
-appropriate ADR/spec rather than hiding it in dashboard configuration.
+appropriate ADR/spec rather than hiding it in dashboard configuration. Consume
+ADR-0022's already-decided usage/cost soft-alert ownership as input; do not
+reopen its hard-limit or provider-mode financial-alert semantics here.
@@ -0,0 +1,13 @@
+# Implement layered limits and multi-dimensional request rate limiting
+
+Type: task
+Status: open
+Blocked by: 20, 22
+
+## Question
+
+Implement ADR-0022's fail-closed platform ceilings, Organization-lowerable
+policy limits, bounded list/query behavior, and typed multi-dimensional rate
+limits for authenticated Web, signed Feishu, OAuth, and administrative writes.
+Prove effective limits never exceed platform ceilings and every overload result
+is observable, scoped, and carries retry guidance where applicable.
@@ -0,0 +1,13 @@
+# Build the durable Organization-fair admission scheduler
+
+Type: task
+Status: open
+Blocked by: 18, 23, 24, 32
+
+## Question
+
+Extend the durable inbound lifecycle into ADR-0022's bounded Agent Admission
+Queue with platform/Organization/Project concurrency, fair scheduling across
+Organizations, FIFO within each Organization, durable position/state,
+expiration notification, scoped cancellation, idempotent admission, explicit
+capacity rejection, and no automatic replay after a run has started.
@@ -0,0 +1,13 @@
+# Enforce file, archive, storage, and entity budgets
+
+Type: task
+Status: open
+Blocked by: 17, 23, 32
+
+## Question
+
+Enforce ADR-0022's request/file/attachment/archive-expansion,
+Project/Organization storage, and Organization entity-count limits with
+streaming accounting, atomic reservation, cleanup of incomplete artifacts,
+bounded queries, explicit errors, and continued read/export/delete/admin
+remediation while an Organization is at or above a ceiling.
@@ -0,0 +1,13 @@
+# Enforce Agent Run and process resource budgets
+
+Type: task
+Status: open
+Blocked by: 16, 24, 32
+
+## Question
+
+Implement ADR-0022's wall-time, turn, tool-count, per-tool-time,
+output/log/event-size, memory, CPU, and child-process budgets. Termination must
+be fail-fast and observable, persist `TIMED_OUT` or `LIMIT_EXCEEDED` with the
+triggered dimension and actual usage, cancel the provider/process safely, and
+release run locks and admission capacity under every injected failure.
@@ -0,0 +1,13 @@
+# Implement provider-mode usage attribution and soft alerts
+
+Type: task
+Status: open
+Blocked by: 19, 24, 29, 32
+
+## Question
+
+Attribute token usage, provider-reported cost, run count, and duration by
+Organization, Project, Run, model, and Provider Connection; preserve missing
+cost as unknown; expose Organization-configurable soft thresholds; and route
+Organization alerts plus platform-managed anomalous-cost alerts without hard
+spend blocking, BYOK financial alerts, or payment-settlement semantics.
@@ -0,0 +1,13 @@
+# Add audited emergency workload brakes
+
+Type: task
+Status: open
+Blocked by: 22, 23, 24, 27, 30, 32, 33
+
+## Question
+
+Implement reversible, reason-bearing, audited `DRAIN` and `STOP_NOW` controls at
+Organization and platform scope. Prove new admissions, queued work, active runs,
+locks, user notifications, authorization, telemetry, restart persistence, and
+explicit recovery follow ADR-0022 without deleting or silently suspending
+Organization data.
@@ -0,0 +1,14 @@
+# Calibrate and prove production capacity ceilings
+
+Type: research
+Status: open
+Blocked by: 30, 32, 33, 34, 35, 36, 37
+
+## Question
+
+On the accepted production-like host and provider path, measure safe numerical
+defaults and headroom for every ADR-0022 platform ceiling. Exercise normal and
+noisy-neighbor traffic, provider stalls and rate limits, queue saturation,
+large files/archives, storage pressure, process budgets, and both workload
+brakes; publish reproducible evidence and the checked-in configuration values
+required by the production release gate.
@@ -5,18 +5,25 @@ Label: wayfinder:map
 ## Destination

 The repository can be deployed from a clean checkout as the initial production
-multi-tenant Curriculum Project Hub, operated safely on the ADR-0018/0020/0021
-single-host Linux topology, with repeatable release evidence, rollback and
-recovery procedures, and no known critical security or data-integrity gaps.
+multi-tenant Curriculum Project Hub: platform staff manage Organizations; every
+Organization authenticates through and operates its own customer-owned Feishu
+app/bot, and selects either an Organization-managed BYOK provider connection or
+a distinct platform-managed provider connection; Organization owners/admins
+manage their own members, projects and settings. The ADR-0018/0020/0021
+single-host Linux topology is operated with repeatable release evidence,
+rollback and recovery, and no known critical security or data-integrity gaps.

 ## Notes

 - `spec/` remains the semantic source of truth; implementation drift must be
  surfaced rather than silently resolved.
 - The initial product boundary follows the accepted ADRs: Organization is the
-  tenant root, customer onboarding may be manual, and full billing, limits,
-  tenant suspension, and self-service platform administration remain deferred
-  unless a readiness investigation proves they are required for safe operation.
+  tenant root and customer onboarding may be manual. Full billing, tenant
+  suspension, and self-service platform administration remain deferred;
+  ADR-0022's safety limits and workload brakes are required for production.
+- Manual guided setup may provision each Organization's customer-owned Feishu
+  app during the pilot; one process-global Feishu/OAuth credential shared by
+  unrelated Organizations is not an acceptable destination state.
 - Use the `diagnosing-bugs` loop for every reproduced defect. Prefer explicit
  failures and observable critical paths over fallback behavior.
 - Work on `saas-production-readiness`; make small conventional commits after
@@ -32,6 +39,7 @@ recovery procedures, and no known critical security or data-integrity gaps.
 - [Audit tenant, authentication, and request security boundaries](issues/02-audit-tenant-auth-security.md) — retain the working org-admin/application authorization core, but production is blocked on agent/MCP isolation, execution-time object binding, org-scoped encrypted credentials, tenant status, browser sessions, and clean dependency audits.
 - [Audit run lifecycle and data integrity under failure](issues/03-audit-run-lifecycle-integrity.md) — retain the database-enforced project-lock uniqueness and normal happy path, but replace receipt-before-success, process-local queues, independent run/session/lock effects, silent delivery/audit loss, and partial startup reset with durable, recoverable state machines; explicitly decide failed-run workspace semantics.
 - [Define production observability and incident recovery](issues/04-define-observability-recovery.md) — keep the existing Fastify/Prisma/systemd signal fragments, but production requires dependency-aware readiness, typed and correlated errors, durable-work telemetry, an explicit SLO/retention/alerting control plane, and recovery runbooks proven by fault drills.
+- [Define initial abuse and capacity controls](issues/05-define-abuse-capacity-controls.md) — require layered fail-closed limits, durable Organization-fair admission, explicit backpressure, bounded files/storage/runs, soft usage alerts, and audited emergency workload brakes; calibrate numerical ceilings under production-like load.

 ## Fog