forked from bai/curriculum-project-hub
126 lines
6.8 KiB
Markdown
126 lines
6.8 KiB
Markdown
# Initial abuse and capacity controls
|
|
|
|
## Verdict
|
|
|
|
The initial production boundary requires hard safety controls before customer
|
|
traffic. The current project lock and small in-memory trigger queue are useful
|
|
local protections, but they do not provide multi-tenant admission, durable
|
|
backpressure, storage safety, request isolation, bounded execution, or an
|
|
operator stop mechanism.
|
|
|
|
The accepted contract is recorded in
|
|
[ADR-0022](../../../docs/adr/0022-layered-capacity-admission.md) and
|
|
[`Spec.System.Capacity`](../../../spec/Spec/System/Capacity.lean). Exact values
|
|
remain deliberately open until production-like calibration; the required
|
|
dimensions and failure behaviors do not.
|
|
|
|
## Current implementation inventory
|
|
|
|
| Surface | Existing protection | Production gap |
|
|
| --- | --- | --- |
|
|
| HTTP | Fastify's default 1 MiB body limit | No route-class, user, Org, IP, or administrative-write rate limits; request/connection/handler timeouts are unset |
|
|
| Project execution | Database lock permits one active run per Project | No platform or Org concurrency ceiling and no fair scheduler |
|
|
| Busy Project | Process-local FIFO, default five items and five-minute wait | Restart loses accepted work; expiry is logged and dropped without a durable state or user notification |
|
|
| Agent execution | Global max-turn default of 25 | No wall-clock, tool-call, output, memory, CPU, process, or Org-specific budget |
|
|
| Feishu messages | Message batching limits text count/characters | File/post attachment count and size are unbounded; downloads may buffer whole files |
|
|
| Storage | Workspace path boundary | No Project or Org byte quota, archive-expansion limit, or entity-count ceiling |
|
|
| Usage | Provider-reported token/cost fields and Org aggregation | Missing cost can disappear from totals; no soft-threshold notification or provider-mode attribution contract |
|
|
| Host | systemd lifecycle | No service CPU, memory, or task ceiling and no audited workload brake |
|
|
|
|
Most Organization member/team/project lists are also unpaginated, and usage
|
|
aggregation can load all matching runs. Those query shapes must be bounded as
|
|
part of enforcement rather than hidden behind larger process memory.
|
|
|
|
## Accepted contract
|
|
|
|
### Layered limits
|
|
|
|
- Every mandatory dimension has a versioned, non-overridable platform ceiling.
|
|
- An Organization OWNER/ADMIN may configure a lower policy limit.
|
|
- Effective limit is `min(platform, organization)`; absent Organization policy
|
|
means the platform ceiling, never unlimited.
|
|
- Missing or invalid mandatory production ceilings fail startup.
|
|
|
|
### Admission and fairness
|
|
|
|
- Project exclusivity remains one active run.
|
|
- Platform and Organization concurrency limits apply before starting work.
|
|
- Available slots are scheduled fairly across Organizations, FIFO within an
|
|
Organization, so one tenant cannot monopolize a global FIFO.
|
|
- Accepted work is durable and queryable. Queue length and wait are bounded.
|
|
- Overflow is explicit `capacity_exhausted` (`429` plus `Retry-After` for HTTP;
|
|
a clear busy response for Feishu). Nothing is silently dropped.
|
|
- Wait timeout produces terminal `EXPIRED`, not a later automatic start.
|
|
- The initiator may withdraw their queued request. Project `MANAGE` or an
|
|
Organization admin may cancel queued requests in their scope. Actor, reason,
|
|
and notification are recorded.
|
|
|
|
### Retry and idempotency
|
|
|
|
- Feishu `event_id` and API idempotency identify one admission across retries.
|
|
- Once a run starts, the whole run is never automatically replayed because tool
|
|
side effects may already exist.
|
|
- Only bounded transport retries before model output or tool side effects are
|
|
automatic. Later failure/timeouts require an explicit new request.
|
|
|
|
### Request, file, storage, and entity safety
|
|
|
|
- Authenticated Web requests are limited by user, Organization, and endpoint
|
|
class; signed Feishu events by Organization app, chat, and sender;
|
|
administrative writes use stricter independent buckets.
|
|
- Admin surfaces require both a valid user session and the corresponding
|
|
platform/Organization administrator authorization. OAuth callbacks and
|
|
signed Feishu ingress remain necessary non-session entry points.
|
|
- Hard limits cover request body, single-file bytes, attachment count, archive
|
|
expanded bytes/file count/depth, Project/Organization storage, and per-type
|
|
Organization entity counts.
|
|
- Transfer is streamed and counted; partial files are removed after failure.
|
|
- Crossing a ceiling blocks new growth without auto-deleting or hiding existing
|
|
resources. Read, export, delete, and administrator remediation stay usable.
|
|
|
|
### Run budget
|
|
|
|
Every run is bounded by wall time, model turns, tool-call count, per-tool time,
|
|
output/log/event bytes, and process memory/CPU/child-process count. A breached
|
|
budget ends explicitly as `TIMED_OUT` or `LIMIT_EXCEEDED`, records the dimension
|
|
and observed usage, releases capacity, and never masquerades as success.
|
|
|
|
### Usage and cost
|
|
|
|
- Token usage, provider-reported cost, run count, and duration are attributed by
|
|
Organization, Project, Run, model, and Provider Connection.
|
|
- Missing provider cost is `unknown`, never zero.
|
|
- Thresholds are soft alerts in the pilot; they do not automatically stop runs
|
|
and are not payment settlement.
|
|
- Org admins receive Organization alerts. Platform admins additionally receive
|
|
anomalous-cost alerts for platform-managed connections; BYOK does not trigger
|
|
platform financial alerts.
|
|
|
|
### Emergency control
|
|
|
|
An audited Platform Administrator control supports Organization or global
|
|
`DRAIN` (reject new work, pause queue starts, finish active runs) and `STOP_NOW`
|
|
(also cancel active runs). Both require an actor, reason, timestamp, explicit
|
|
recovery, and user-visible outcomes. This is workload control, not Organization
|
|
deletion or account suspension.
|
|
|
|
## Required production evidence
|
|
|
|
1. Production configuration validation fails closed for every missing or
|
|
invalid platform ceiling.
|
|
2. Concurrent noisy-neighbor tests prove platform/Org/Project concurrency and
|
|
Organization-fair scheduling without starvation.
|
|
3. Restart and crash tests prove accepted queue entries, idempotency, expiry,
|
|
cancellation, and notifications survive process loss.
|
|
4. HTTP and Feishu load tests prove every rate-limit identity and typed overload
|
|
response, including retry guidance.
|
|
5. File/archive/storage probes prove streaming enforcement, cleanup, and
|
|
continued read/delete access while over limit.
|
|
6. Hung provider/tool and resource-exhaustion probes prove every run/process
|
|
budget terminates, records cause, and releases locks/capacity.
|
|
7. Usage tests prove provider-mode attribution, unknown-cost handling, alert
|
|
routing, and absence of hard monetary blocking.
|
|
8. `DRAIN` and `STOP_NOW` drills prove scoped, audited, reversible behavior.
|
|
9. Production-like calibration publishes the tested numerical defaults and
|
|
capacity headroom used by the release gate.
|