forked from EduCraft/curriculum-project-hub
docs: define SaaS capacity controls
This commit is contained in:
@@ -0,0 +1,125 @@
|
||||
# Initial abuse and capacity controls
|
||||
|
||||
## Verdict
|
||||
|
||||
The initial production boundary requires hard safety controls before customer
|
||||
traffic. The current project lock and small in-memory trigger queue are useful
|
||||
local protections, but they do not provide multi-tenant admission, durable
|
||||
backpressure, storage safety, request isolation, bounded execution, or an
|
||||
operator stop mechanism.
|
||||
|
||||
The accepted contract is recorded in
|
||||
[ADR-0022](../../../docs/adr/0022-layered-capacity-admission.md) and
|
||||
[`Spec.System.Capacity`](../../../spec/Spec/System/Capacity.lean). Exact values
|
||||
remain deliberately open until production-like calibration; the required
|
||||
dimensions and failure behaviors do not.
|
||||
|
||||
## Current implementation inventory
|
||||
|
||||
| Surface | Existing protection | Production gap |
|
||||
| --- | --- | --- |
|
||||
| HTTP | Fastify's default 1 MiB body limit | No route-class, user, Org, IP, or administrative-write rate limits; request/connection/handler timeouts are unset |
|
||||
| Project execution | Database lock permits one active run per Project | No platform or Org concurrency ceiling and no fair scheduler |
|
||||
| Busy Project | Process-local FIFO, default five items and five-minute wait | Restart loses accepted work; expiry is logged and dropped without a durable state or user notification |
|
||||
| Agent execution | Global max-turn default of 25 | No wall-clock, tool-call, output, memory, CPU, process, or Org-specific budget |
|
||||
| Feishu messages | Message batching limits text count/characters | File/post attachment count and size are unbounded; downloads may buffer whole files |
|
||||
| Storage | Workspace path boundary | No Project or Org byte quota, archive-expansion limit, or entity-count ceiling |
|
||||
| Usage | Provider-reported token/cost fields and Org aggregation | Missing cost can disappear from totals; no soft-threshold notification or provider-mode attribution contract |
|
||||
| Host | systemd lifecycle | No service CPU, memory, or task ceiling and no audited workload brake |
|
||||
|
||||
Most Organization member/team/project lists are also unpaginated, and usage
|
||||
aggregation can load all matching runs. Those query shapes must be bounded as
|
||||
part of enforcement rather than hidden behind larger process memory.
|
||||
|
||||
## Accepted contract
|
||||
|
||||
### Layered limits
|
||||
|
||||
- Every mandatory dimension has a versioned, non-overridable platform ceiling.
|
||||
- An Organization OWNER/ADMIN may configure a lower policy limit.
|
||||
- Effective limit is `min(platform, organization)`; absent Organization policy
|
||||
means the platform ceiling, never unlimited.
|
||||
- Missing or invalid mandatory production ceilings fail startup.
|
||||
|
||||
### Admission and fairness
|
||||
|
||||
- Project exclusivity remains one active run.
|
||||
- Platform and Organization concurrency limits apply before starting work.
|
||||
- Available slots are scheduled fairly across Organizations, FIFO within an
|
||||
Organization, so one tenant cannot monopolize a global FIFO.
|
||||
- Accepted work is durable and queryable. Queue length and wait are bounded.
|
||||
- Overflow is explicit `capacity_exhausted` (`429` plus `Retry-After` for HTTP;
|
||||
a clear busy response for Feishu). Nothing is silently dropped.
|
||||
- Wait timeout produces terminal `EXPIRED`, not a later automatic start.
|
||||
- The initiator may withdraw their queued request. Project `MANAGE` or an
|
||||
Organization admin may cancel queued requests in their scope. Actor, reason,
|
||||
and notification are recorded.
|
||||
|
||||
### Retry and idempotency
|
||||
|
||||
- Feishu `event_id` and API idempotency identify one admission across retries.
|
||||
- Once a run starts, the whole run is never automatically replayed because tool
|
||||
side effects may already exist.
|
||||
- Only bounded transport retries before model output or tool side effects are
|
||||
automatic. Later failure/timeouts require an explicit new request.
|
||||
|
||||
### Request, file, storage, and entity safety
|
||||
|
||||
- Authenticated Web requests are limited by user, Organization, and endpoint
|
||||
class; signed Feishu events by Organization app, chat, and sender;
|
||||
administrative writes use stricter independent buckets.
|
||||
- Admin surfaces require both a valid user session and the corresponding
|
||||
platform/Organization administrator authorization. OAuth callbacks and
|
||||
signed Feishu ingress remain necessary non-session entry points.
|
||||
- Hard limits cover request body, single-file bytes, attachment count, archive
|
||||
expanded bytes/file count/depth, Project/Organization storage, and per-type
|
||||
Organization entity counts.
|
||||
- Transfer is streamed and counted; partial files are removed after failure.
|
||||
- Crossing a ceiling blocks new growth without auto-deleting or hiding existing
|
||||
resources. Read, export, delete, and administrator remediation stay usable.
|
||||
|
||||
### Run budget
|
||||
|
||||
Every run is bounded by wall time, model turns, tool-call count, per-tool time,
|
||||
output/log/event bytes, and process memory/CPU/child-process count. A breached
|
||||
budget ends explicitly as `TIMED_OUT` or `LIMIT_EXCEEDED`, records the dimension
|
||||
and observed usage, releases capacity, and never masquerades as success.
|
||||
|
||||
### Usage and cost
|
||||
|
||||
- Token usage, provider-reported cost, run count, and duration are attributed by
|
||||
Organization, Project, Run, model, and Provider Connection.
|
||||
- Missing provider cost is `unknown`, never zero.
|
||||
- Thresholds are soft alerts in the pilot; they do not automatically stop runs
|
||||
and are not payment settlement.
|
||||
- Org admins receive Organization alerts. Platform admins additionally receive
|
||||
anomalous-cost alerts for platform-managed connections; BYOK does not trigger
|
||||
platform financial alerts.
|
||||
|
||||
### Emergency control
|
||||
|
||||
An audited Platform Administrator control supports Organization or global
|
||||
`DRAIN` (reject new work, pause queue starts, finish active runs) and `STOP_NOW`
|
||||
(also cancel active runs). Both require an actor, reason, timestamp, explicit
|
||||
recovery, and user-visible outcomes. This is workload control, not Organization
|
||||
deletion or account suspension.
|
||||
|
||||
## Required production evidence
|
||||
|
||||
1. Production configuration validation fails closed for every missing or
|
||||
invalid platform ceiling.
|
||||
2. Concurrent noisy-neighbor tests prove platform/Org/Project concurrency and
|
||||
Organization-fair scheduling without starvation.
|
||||
3. Restart and crash tests prove accepted queue entries, idempotency, expiry,
|
||||
cancellation, and notifications survive process loss.
|
||||
4. HTTP and Feishu load tests prove every rate-limit identity and typed overload
|
||||
response, including retry guidance.
|
||||
5. File/archive/storage probes prove streaming enforcement, cleanup, and
|
||||
continued read/delete access while over limit.
|
||||
6. Hung provider/tool and resource-exhaustion probes prove every run/process
|
||||
budget terminates, records cause, and releases locks/capacity.
|
||||
7. Usage tests prove provider-mode attribution, unknown-cost handling, alert
|
||||
routing, and absence of hard monetary blocking.
|
||||
8. `DRAIN` and `STOP_NOW` drills prove scoped, audited, reversible behavior.
|
||||
9. Production-like calibration publishes the tested numerical defaults and
|
||||
capacity headroom used by the release gate.
|
||||
Reference in New Issue
Block a user