docs: define SaaS capacity controls

2026-07-10 12:07:08 +08:00
parent 16329df357
commit 4ca4afb141
27 changed files with 618 additions and 47 deletions
@@ -1,7 +1,7 @@
 # Define initial abuse and capacity controls

 Type: grilling
-Status: open
+Status: resolved
 Blocked by: 02, 03

 ## Question
@@ -9,3 +9,35 @@ Blocked by: 02, 03
 Which request limits, file limits, concurrency limits, agent budgets, tenant
 quotas, and backpressure policies are mandatory for the accepted initial
 production boundary, and which values or behaviors remain product decisions?
+
+## Answer
+
+The initial production service requires layered, fail-closed capacity controls:
+every mandatory dimension has a versioned non-overridable platform ceiling,
+while an Organization may configure only a lower policy limit. Agent work uses
+a durable bounded queue, fair scheduling across Organizations, FIFO within an
+Organization, explicit overload/expiry/cancellation states, and no automatic
+whole-run replay after execution starts.
+
+Hard controls cover request rate/body size, concurrency and queueing, files and
+archive expansion, Project/Organization storage and entity counts, run/tool
+budgets, and process resources. Reaching a limit blocks new growth without
+deleting or hiding existing data. Token and provider cost remain attributed
+soft-alert signals in the pilot, not hard spend controls; unknown cost is not
+zero. Platform administrators also receive audited `DRAIN` and `STOP_NOW`
+workload brakes.
+
+The full current-state inventory, accepted behavior, and release evidence are
+recorded in [Initial abuse and capacity controls](../assets/initial-abuse-capacity-controls.md),
+with the durable decision in ADR-0022 and `Spec.System.Capacity`. Numerical
+ceilings remain open until production-like calibration.
+
+The implementation frontier is:
+
+- [Implement layered limits and multi-dimensional request rate limiting](32-layered-limits-rate-limiting.md)
+- [Build the durable Organization-fair admission scheduler](33-durable-fair-admission-scheduler.md)
+- [Enforce file, archive, storage, and entity budgets](34-file-storage-budgets.md)
+- [Enforce Agent Run and process resource budgets](35-run-runtime-budgets.md)
+- [Implement provider-mode usage attribution and soft alerts](36-usage-soft-alerts.md)
+- [Add audited emergency workload brakes](37-emergency-workload-brakes.md)
+- [Calibrate and prove production capacity ceilings](38-calibrate-production-capacity.md)
@@ -6,7 +6,8 @@ Status: open
 ## Question

 Against ADR-0020 and ADR-0021, which required platform-admin, org-admin,
-project-onboarding, team/access, session, usage, and Feishu flows are absent,
+project-onboarding, team/access, session, usage, provider-mode, Organization
+policy-limit, emergency workload-control, and Feishu flows are absent,
 unreachable, or only test-facing rather than usable in the initial production
-service?
-
+service? Treat ADR-0022's capacity control surfaces as part of the accepted
+product boundary.
@@ -2,7 +2,7 @@

 Type: task
 Status: open
-Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31
+Blocked by: 01, 02, 03, 04, 05, 06, 07, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38

 ## Question

@@ -7,6 +7,7 @@ Status: open

 Which key-management, envelope-encryption, rotation, connection identity, and
 runtime resolver design will implement ADR-0021's org-scoped Feishu and model
-provider credentials, support one customer-owned Feishu app per tenant, and
-ensure plaintext credentials never reach business records, logs, or agent
-tools?
+provider credentials; support one customer-owned Feishu app per Organization;
+support Organization-managed BYOK and a distinct platform-managed provider
+connection per Organization; and ensure plaintext credentials never reach
+business records, logs, or agent tools?
@@ -2,6 +2,7 @@

 Type: task
 Status: open
+Blocked by: 18

 ## Question

@@ -10,4 +11,6 @@ durable inbound work lifecycle that atomically claims duplicate events, records
 processing/completion/failure, survives restart, retries recoverable failures,
 and proves concurrent redelivery cannot duplicate a run or silently lose an
 accepted prompt. Preserve execution-time authorization from ticket 18 when
-deferred work is finally started.
+deferred work is finally started, and expose the durable identity/state needed
+by ADR-0022's Organization-fair admission scheduler rather than recreating a
+second queue lifecycle.
@@ -11,4 +11,6 @@ redaction/access/retention policy, initial service-level indicators and
 objectives, error budgets, alert thresholds, on-call destinations, escalation
 ownership, and telemetry-blind detection. Prefer the smallest operable control
 plane, but record every domain-visible availability or retry assumption in the
-appropriate ADR/spec rather than hiding it in dashboard configuration.
+appropriate ADR/spec rather than hiding it in dashboard configuration. Consume
+ADR-0022's already-decided usage/cost soft-alert ownership as input; do not
+reopen its hard-limit or provider-mode financial-alert semantics here.
@@ -0,0 +1,13 @@
+# Implement layered limits and multi-dimensional request rate limiting
+
+Type: task
+Status: open
+Blocked by: 20, 22
+
+## Question
+
+Implement ADR-0022's fail-closed platform ceilings, Organization-lowerable
+policy limits, bounded list/query behavior, and typed multi-dimensional rate
+limits for authenticated Web, signed Feishu, OAuth, and administrative writes.
+Prove effective limits never exceed platform ceilings and every overload result
+is observable, scoped, and carries retry guidance where applicable.
@@ -0,0 +1,13 @@
+# Build the durable Organization-fair admission scheduler
+
+Type: task
+Status: open
+Blocked by: 18, 23, 24, 32
+
+## Question
+
+Extend the durable inbound lifecycle into ADR-0022's bounded Agent Admission
+Queue with platform/Organization/Project concurrency, fair scheduling across
+Organizations, FIFO within each Organization, durable position/state,
+expiration notification, scoped cancellation, idempotent admission, explicit
+capacity rejection, and no automatic replay after a run has started.
@@ -0,0 +1,13 @@
+# Enforce file, archive, storage, and entity budgets
+
+Type: task
+Status: open
+Blocked by: 17, 23, 32
+
+## Question
+
+Enforce ADR-0022's request/file/attachment/archive-expansion,
+Project/Organization storage, and Organization entity-count limits with
+streaming accounting, atomic reservation, cleanup of incomplete artifacts,
+bounded queries, explicit errors, and continued read/export/delete/admin
+remediation while an Organization is at or above a ceiling.
@@ -0,0 +1,13 @@
+# Enforce Agent Run and process resource budgets
+
+Type: task
+Status: open
+Blocked by: 16, 24, 32
+
+## Question
+
+Implement ADR-0022's wall-time, turn, tool-count, per-tool-time,
+output/log/event-size, memory, CPU, and child-process budgets. Termination must
+be fail-fast and observable, persist `TIMED_OUT` or `LIMIT_EXCEEDED` with the
+triggered dimension and actual usage, cancel the provider/process safely, and
+release run locks and admission capacity under every injected failure.
@@ -0,0 +1,13 @@
+# Implement provider-mode usage attribution and soft alerts
+
+Type: task
+Status: open
+Blocked by: 19, 24, 29, 32
+
+## Question
+
+Attribute token usage, provider-reported cost, run count, and duration by
+Organization, Project, Run, model, and Provider Connection; preserve missing
+cost as unknown; expose Organization-configurable soft thresholds; and route
+Organization alerts plus platform-managed anomalous-cost alerts without hard
+spend blocking, BYOK financial alerts, or payment-settlement semantics.
@@ -0,0 +1,13 @@
+# Add audited emergency workload brakes
+
+Type: task
+Status: open
+Blocked by: 22, 23, 24, 27, 30, 32, 33
+
+## Question
+
+Implement reversible, reason-bearing, audited `DRAIN` and `STOP_NOW` controls at
+Organization and platform scope. Prove new admissions, queued work, active runs,
+locks, user notifications, authorization, telemetry, restart persistence, and
+explicit recovery follow ADR-0022 without deleting or silently suspending
+Organization data.
@@ -0,0 +1,14 @@
+# Calibrate and prove production capacity ceilings
+
+Type: research
+Status: open
+Blocked by: 30, 32, 33, 34, 35, 36, 37
+
+## Question
+
+On the accepted production-like host and provider path, measure safe numerical
+defaults and headroom for every ADR-0022 platform ceiling. Exercise normal and
+noisy-neighbor traffic, provider stalls and rate limits, queue saturation,
+large files/archives, storage pressure, process budgets, and both workload
+brakes; publish reproducible evidence and the checked-in configuration values
+required by the production release gate.