forked from bai/curriculum-project-hub
89 lines
4.9 KiB
Markdown
89 lines
4.9 KiB
Markdown
# ADR 0022: Layered Capacity Admission And Workload Safety
|
|
|
|
## Status
|
|
|
|
Accepted.
|
|
|
|
## Context
|
|
|
|
The initial SaaS deployment shares one production host across Organizations.
|
|
The current Hub only has a one-run-per-project lock, a process-local five-item
|
|
trigger queue, a five-minute queue wait, a global max-turn setting, and
|
|
Fastify's default request-body limit. It has no global or Organization agent
|
|
capacity, durable backpressure, fair scheduling, storage quotas, run wall-clock
|
|
budget, multi-dimensional rate limiting, or emergency workload control.
|
|
|
|
Leaving these limits implicit lets one Organization monopolize the host, lets
|
|
accepted work disappear on restart, and makes overload look like success. A
|
|
single global FIFO would preserve arrival order while still permitting one
|
|
tenant to occupy every execution slot. Unlimited queues or automatic whole-run
|
|
replay would hide overload and can duplicate tool side effects.
|
|
|
|
## Decision
|
|
|
|
Every mandatory capacity dimension has a non-optional **Platform Capacity
|
|
Ceiling**. An Organization OWNER/ADMIN may configure a lower **Organization
|
|
Policy Limit**; the **Effective Limit** is the lower value, or the platform
|
|
ceiling when the Organization value is absent. There is no unlimited fallback.
|
|
Production refuses to start when a mandatory platform ceiling is absent or
|
|
invalid. Platform ceilings are changed through versioned deployment
|
|
configuration in the initial release; live platform-admin editing is deferred
|
|
until audited approval and rollback exist.
|
|
|
|
Agent work uses a durable, bounded admission queue. Scheduling is fair across
|
|
Organizations and FIFO within an Organization, while the existing one-active-
|
|
run-per-Project rule remains. Capacity saturation queues only within effective
|
|
length and wait limits. Queue overflow is an explicit `capacity_exhausted`
|
|
response with retry guidance; an accepted request that waits too long becomes
|
|
`EXPIRED`, is never started later, and notifies its initiator. Initiators may
|
|
withdraw their own queued requests; Project `MANAGE` and Organization admins
|
|
may cancel queued requests in their scope. Every transition is auditable.
|
|
|
|
The platform never automatically replays a whole Agent Run after execution has
|
|
started. Limited transport retries are allowed only before model output or tool
|
|
side effects. Otherwise the run ends explicitly and a retry is a new user-
|
|
initiated request. Feishu event identity and API idempotency prevent delivery
|
|
retries from creating duplicate admissions.
|
|
|
|
The first production gate requires hard limits for request rate/body size,
|
|
agent concurrency, queue length/wait, files and attachment counts, archive
|
|
expansion, Project and Organization storage, Organization entity counts, run
|
|
wall time/turns/tool calls/output, individual tool duration, and execution
|
|
process memory/CPU/process count. Reaching an entity or storage ceiling blocks
|
|
new growth but does not delete, hide, or lock existing data; read, export,
|
|
delete, and administrator remediation remain available.
|
|
|
|
Admin surfaces require an authenticated user plus the corresponding platform
|
|
or Organization administrator authorization. Authenticated Web traffic is
|
|
limited by user, Organization, and endpoint class; signed Feishu traffic is
|
|
limited by Organization application, chat, and sender; administrative writes
|
|
use separate stricter buckets. Rejections and truncation are never silent.
|
|
|
|
Token, provider-reported cost, run count, and duration are attributed by
|
|
Organization, Project, Run, model, and Provider Connection. They are statistics
|
|
and soft alerts in the initial release, not hard spend controls or settlement.
|
|
Missing provider cost remains unknown rather than zero. Organization admins
|
|
receive their alerts; platform admins additionally receive anomalous-cost
|
|
alerts for platform-managed Provider Connections, not BYOK financial alerts.
|
|
|
|
Platform administrators have audited, reason-bearing, reversible workload
|
|
brakes at Organization and platform scope. `DRAIN` rejects new work and pauses
|
|
queue starts while allowing active runs to finish. `STOP_NOW` also cancels
|
|
active runs. These controls do not delete or suspend the Organization.
|
|
|
|
Exact numerical ceilings remain open until production-like capacity tests
|
|
measure the supported host and provider behavior. Defaults must be checked in,
|
|
validated, observable, and proven at the release gate rather than guessed as
|
|
domain constants.
|
|
|
|
## Consequences
|
|
|
|
- The current in-memory `TriggerQueue` and global-only agent settings are
|
|
implementation divergences, not acceptable production fallbacks.
|
|
- Durable ingress, run/lock recovery, authorization, telemetry, and capacity
|
|
enforcement must share one Organization-scoped admission identity.
|
|
- HTTP overload uses typed `429` responses with `Retry-After`; Feishu reports a
|
|
clear busy/expired/canceled outcome to the initiating user.
|
|
- Capacity calibration is a release prerequisite and must include noisy-neighbor,
|
|
restart, queue expiry, file/archive, provider-stall, and workload-brake tests.
|