docs: define SaaS capacity controls

This commit is contained in:
2026-07-10 12:07:08 +08:00
parent 16329df357
commit 4ca4afb141
27 changed files with 618 additions and 47 deletions
@@ -158,9 +158,12 @@ as it upholds `AgentFileOp.Authorized`.
*effects* are then confined, is a policy choice left to implementation. The
invariant governs effects either way; current choice is arbitrary commands
(sandbox confines effects).
- **Per-user rate limiting and token budgets.** Orthogonal to the surface
boundary (this ADR is about *what* the agent may touch, not *how often* it
may run). Deferred to a separate concern.
- **Inbound file size / MIME limits.** The inbox lives inside the boundary,
but unbounded file size is a DoS vector distinct from the boundary
invariant. Deferred.
## Related Decisions
- **Rate limiting and token usage.** ADR-0022 requires multi-dimensional hard
request limits while keeping token thresholds as attributed soft alerts in
the initial release.
- **Inbound file limits.** ADR-0022 requires hard file, attachment, archive,
Project-storage, and Organization-storage limits with explicit failure and
partial-file cleanup.
@@ -83,8 +83,10 @@ separate control plane and must not be modeled as project `MANAGE`.
## Open Questions / Deferred
- Full platform admin UI, billing, limits, and tenant suspension workflows are
deferred.
- Full platform admin UI, billing, and tenant suspension workflows are
deferred. Production safety limits and workload brakes are no longer
deferred; ADR-0022 defines them without treating a workload brake as tenant
suspension.
- Whether direct `USER` project grants should require active
`OrganizationMembership` is deferred; explicit direct grants remain allowed
for now.
+21 -7
View File
@@ -11,8 +11,8 @@ surface is not only the Feishu bot trigger path, but also the control planes
around it:
- platform staff need a private platform admin area to create and operate orgs;
- customer org owners/admins need an org admin area for teams, roles, model
provider configuration, projects, folders, sessions, and usage accounting;
- customer org owners/admins need an org admin area for teams, roles, provider
mode and BYOK configuration, projects, folders, sessions, and usage accounting;
- ordinary teachers should be able to start work from the natural Feishu group
flow without entering the web backend.
@@ -46,6 +46,19 @@ OAuth login, bot messages, and directory sync. App secrets are org-scoped
secrets and must be encrypted at rest; business code should access them through
a secret/connection resolver, not raw plaintext columns.
Each Organization chooses one model-provider credential mode:
- `BYOK`: an Organization OWNER/ADMIN supplies and manages that Organization's
provider key and base URL in the org admin area;
- `PLATFORM_MANAGED`: a platform admin supplies and manages one distinct key
and base URL for that Organization in the platform admin area. Organization
users cannot read the plaintext credential.
Provider connections remain Organization-scoped in both modes. Unrelated
Organizations never share a process-global provider key. Provider secrets must
be encrypted at rest and resolved at runtime through the same connection/secret
boundary as Feishu credentials.
Project management has two creation paths:
- org owner/admin creates projects in the org web backend;
@@ -77,9 +90,10 @@ aggregation:
Project permissions remain on `PROJECT` resources. Moving a project between
folders does not change grants.
Usage accounting is org-wise and project/folder aggregatable. It is not payment
collection in the pilot because customers supply their own model provider API
keys and base URLs.
Usage accounting is org-wise and project/folder aggregatable for both provider
modes. It is not payment collection in the pilot: BYOK spend belongs to the
customer's provider account, while platform-managed spend is attributed to the
Organization for operational reporting; commercial billing remains deferred.
## Consequences
@@ -89,8 +103,8 @@ keys and base URLs.
explorer without creating a second ACL system.
- Service code should expose project creation and chat binding as reusable
backend operations so Feishu cards and future web APIs call the same rules.
- Org role/model/provider/billing panels can be added on top of the org tenant
root without changing project authorization.
- Org role/model/provider-mode panels and the platform-managed connection panel
can be added on top of the org tenant root without changing project authorization.
## Open Questions / Deferred
@@ -0,0 +1,88 @@
# ADR 0022: Layered Capacity Admission And Workload Safety
## Status
Accepted.
## Context
The initial SaaS deployment shares one production host across Organizations.
The current Hub only has a one-run-per-project lock, a process-local five-item
trigger queue, a five-minute queue wait, a global max-turn setting, and
Fastify's default request-body limit. It has no global or Organization agent
capacity, durable backpressure, fair scheduling, storage quotas, run wall-clock
budget, multi-dimensional rate limiting, or emergency workload control.
Leaving these limits implicit lets one Organization monopolize the host, lets
accepted work disappear on restart, and makes overload look like success. A
single global FIFO would preserve arrival order while still permitting one
tenant to occupy every execution slot. Unlimited queues or automatic whole-run
replay would hide overload and can duplicate tool side effects.
## Decision
Every mandatory capacity dimension has a non-optional **Platform Capacity
Ceiling**. An Organization OWNER/ADMIN may configure a lower **Organization
Policy Limit**; the **Effective Limit** is the lower value, or the platform
ceiling when the Organization value is absent. There is no unlimited fallback.
Production refuses to start when a mandatory platform ceiling is absent or
invalid. Platform ceilings are changed through versioned deployment
configuration in the initial release; live platform-admin editing is deferred
until audited approval and rollback exist.
Agent work uses a durable, bounded admission queue. Scheduling is fair across
Organizations and FIFO within an Organization, while the existing one-active-
run-per-Project rule remains. Capacity saturation queues only within effective
length and wait limits. Queue overflow is an explicit `capacity_exhausted`
response with retry guidance; an accepted request that waits too long becomes
`EXPIRED`, is never started later, and notifies its initiator. Initiators may
withdraw their own queued requests; Project `MANAGE` and Organization admins
may cancel queued requests in their scope. Every transition is auditable.
The platform never automatically replays a whole Agent Run after execution has
started. Limited transport retries are allowed only before model output or tool
side effects. Otherwise the run ends explicitly and a retry is a new user-
initiated request. Feishu event identity and API idempotency prevent delivery
retries from creating duplicate admissions.
The first production gate requires hard limits for request rate/body size,
agent concurrency, queue length/wait, files and attachment counts, archive
expansion, Project and Organization storage, Organization entity counts, run
wall time/turns/tool calls/output, individual tool duration, and execution
process memory/CPU/process count. Reaching an entity or storage ceiling blocks
new growth but does not delete, hide, or lock existing data; read, export,
delete, and administrator remediation remain available.
Admin surfaces require an authenticated user plus the corresponding platform
or Organization administrator authorization. Authenticated Web traffic is
limited by user, Organization, and endpoint class; signed Feishu traffic is
limited by Organization application, chat, and sender; administrative writes
use separate stricter buckets. Rejections and truncation are never silent.
Token, provider-reported cost, run count, and duration are attributed by
Organization, Project, Run, model, and Provider Connection. They are statistics
and soft alerts in the initial release, not hard spend controls or settlement.
Missing provider cost remains unknown rather than zero. Organization admins
receive their alerts; platform admins additionally receive anomalous-cost
alerts for platform-managed Provider Connections, not BYOK financial alerts.
Platform administrators have audited, reason-bearing, reversible workload
brakes at Organization and platform scope. `DRAIN` rejects new work and pauses
queue starts while allowing active runs to finish. `STOP_NOW` also cancels
active runs. These controls do not delete or suspend the Organization.
Exact numerical ceilings remain open until production-like capacity tests
measure the supported host and provider behavior. Defaults must be checked in,
validated, observable, and proven at the release gate rather than guessed as
domain constants.
## Consequences
- The current in-memory `TriggerQueue` and global-only agent settings are
implementation divergences, not acceptable production fallbacks.
- Durable ingress, run/lock recovery, authorization, telemetry, and capacity
enforcement must share one Organization-scoped admission identity.
- HTTP overload uses typed `429` responses with `Retry-After`; Feishu reports a
clear busy/expired/canceled outcome to the initiating user.
- Capacity calibration is a release prerequisite and must include noisy-neighbor,
restart, queue expiry, file/archive, provider-stall, and workload-brake tests.