curriculum-project-hub/docs/adr/0022-layered-capacity-admission.md

ADR 0022: Layered Capacity Admission And Workload Safety

Status

Accepted.

Context

The initial SaaS deployment shares one production host across Organizations. The current Hub only has a one-run-per-project lock, a process-local five-item trigger queue, a five-minute queue wait, a global max-turn setting, and Fastify's default request-body limit. It has no global or Organization agent capacity, durable backpressure, fair scheduling, storage quotas, run wall-clock budget, multi-dimensional rate limiting, or emergency workload control.

Leaving these limits implicit lets one Organization monopolize the host, lets accepted work disappear on restart, and makes overload look like success. A single global FIFO would preserve arrival order while still permitting one tenant to occupy every execution slot. Unlimited queues or automatic whole-run replay would hide overload and can duplicate tool side effects.

Decision

Every mandatory capacity dimension has a non-optional Platform Capacity Ceiling. An Organization OWNER/ADMIN may configure a lower Organization Policy Limit; the Effective Limit is the lower value, or the platform ceiling when the Organization value is absent. There is no unlimited fallback. Production refuses to start when a mandatory platform ceiling is absent or invalid. Platform ceilings are changed through versioned deployment configuration in the initial release; live platform-admin editing is deferred until audited approval and rollback exist.

Agent work uses a durable, bounded admission queue. Scheduling is fair across Organizations and FIFO within an Organization, while the existing one-active- run-per-Project rule remains. Capacity saturation queues only within effective length and wait limits. Queue overflow is an explicit capacity_exhausted response with retry guidance; an accepted request that waits too long becomes EXPIRED, is never started later, and notifies its initiator. Initiators may withdraw their own queued requests; Project MANAGE and Organization admins may cancel queued requests in their scope. Every transition is auditable.

The platform never automatically replays a whole Agent Run after execution has started. Limited transport retries are allowed only before model output or tool side effects. Otherwise the run ends explicitly and a retry is a new user- initiated request. Feishu event identity and API idempotency prevent delivery retries from creating duplicate admissions.

The first production gate requires hard limits for request rate/body size, agent concurrency, queue length/wait, files and attachment counts, archive expansion, Project and Organization storage, Organization entity counts, run wall time/turns/tool calls/output, individual tool duration, and execution process memory/CPU/process count. Reaching an entity or storage ceiling blocks new growth but does not delete, hide, or lock existing data; read, export, delete, and administrator remediation remain available.

Admin surfaces require an authenticated user plus the corresponding platform or Organization administrator authorization. Authenticated Web traffic is limited by user, Organization, and endpoint class; signed Feishu traffic is limited by Organization application, chat, and sender; administrative writes use separate stricter buckets. Rejections and truncation are never silent.

Token, provider-reported cost, run count, and duration are attributed by Organization, Project, Run, model, and Provider Connection. They are statistics and soft alerts in the initial release, not hard spend controls or settlement. Missing provider cost remains unknown rather than zero. Organization admins receive their alerts; platform admins additionally receive anomalous-cost alerts for platform-managed Provider Connections, not BYOK financial alerts.

Platform administrators have audited, reason-bearing, reversible workload brakes at Organization and platform scope. DRAIN rejects new work and pauses queue starts while allowing active runs to finish. STOP_NOW also cancels active runs. These controls do not delete or suspend the Organization.

Exact numerical ceilings remain open until production-like capacity tests measure the supported host and provider behavior. Defaults must be checked in, validated, observable, and proven at the release gate rather than guessed as domain constants.

Consequences

• The current in-memory TriggerQueue and global-only agent settings are implementation divergences, not acceptable production fallbacks.
• Durable ingress, run/lock recovery, authorization, telemetry, and capacity enforcement must share one Organization-scoped admission identity.
• HTTP overload uses typed 429 responses with Retry-After; Feishu reports a clear busy/expired/canceled outcome to the initiating user.
• Capacity calibration is a release prerequisite and must include noisy-neighbor, restart, queue expiry, file/archive, provider-stall, and workload-brake tests.