# Initial abuse and capacity controls

## Verdict

The initial production boundary requires hard safety controls before customer
traffic. The current project lock and small in-memory trigger queue are useful
local protections, but they do not provide multi-tenant admission, durable
backpressure, storage safety, request isolation, bounded execution, or an
operator stop mechanism.

The accepted contract is recorded in
[ADR-0022](../../../docs/adr/0022-layered-capacity-admission.md) and
[`Spec.System.Capacity`](../../../spec/Spec/System/Capacity.lean). Exact values
remain deliberately open until production-like calibration; the required
dimensions and failure behaviors do not.

## Current implementation inventory

| Surface | Existing protection | Production gap |
| --- | --- | --- |
| HTTP | Fastify's default 1 MiB body limit | No route-class, user, Org, IP, or administrative-write rate limits; request/connection/handler timeouts are unset |
| Project execution | Database lock permits one active run per Project | No platform or Org concurrency ceiling and no fair scheduler |
| Busy Project | Process-local FIFO, default five items and five-minute wait | Restart loses accepted work; expiry is logged and dropped without a durable state or user notification |
| Agent execution | Global max-turn default of 25 | No wall-clock, tool-call, output, memory, CPU, process, or Org-specific budget |
| Feishu messages | Message batching limits text count/characters | File/post attachment count and size are unbounded; downloads may buffer whole files |
| Storage | Workspace path boundary | No Project or Org byte quota, archive-expansion limit, or entity-count ceiling |
| Usage | Provider-reported token/cost fields and Org aggregation | Missing cost can disappear from totals; no soft-threshold notification or provider-mode attribution contract |
| Host | systemd lifecycle | No service CPU, memory, or task ceiling and no audited workload brake |

Most Organization member/team/project lists are also unpaginated, and usage
aggregation can load all matching runs. Those query shapes must be bounded as
part of enforcement rather than hidden behind larger process memory.

## Accepted contract

### Layered limits

- Every mandatory dimension has a versioned, non-overridable platform ceiling.
- An Organization OWNER/ADMIN may configure a lower policy limit.
- Effective limit is `min(platform, organization)`; absent Organization policy
  means the platform ceiling, never unlimited.
- Missing or invalid mandatory production ceilings fail startup.

### Admission and fairness

- Project exclusivity remains one active run.
- Platform and Organization concurrency limits apply before starting work.
- Available slots are scheduled fairly across Organizations, FIFO within an
  Organization, so one tenant cannot monopolize a global FIFO.
- Accepted work is durable and queryable. Queue length and wait are bounded.
- Overflow is explicit `capacity_exhausted` (`429` plus `Retry-After` for HTTP;
  a clear busy response for Feishu). Nothing is silently dropped.
- Wait timeout produces terminal `EXPIRED`, not a later automatic start.
- The initiator may withdraw their queued request. Project `MANAGE` or an
  Organization admin may cancel queued requests in their scope. Actor, reason,
  and notification are recorded.

### Retry and idempotency

- Feishu `event_id` and API idempotency identify one admission across retries.
- Once a run starts, the whole run is never automatically replayed because tool
  side effects may already exist.
- Only bounded transport retries before model output or tool side effects are
  automatic. Later failure/timeouts require an explicit new request.

### Request, file, storage, and entity safety

- Authenticated Web requests are limited by user, Organization, and endpoint
  class; signed Feishu events by Organization app, chat, and sender;
  administrative writes use stricter independent buckets.
- Admin surfaces require both a valid user session and the corresponding
  platform/Organization administrator authorization. OAuth callbacks and
  signed Feishu ingress remain necessary non-session entry points.
- Hard limits cover request body, single-file bytes, attachment count, archive
  expanded bytes/file count/depth, Project/Organization storage, and per-type
  Organization entity counts.
- Transfer is streamed and counted; partial files are removed after failure.
- Crossing a ceiling blocks new growth without auto-deleting or hiding existing
  resources. Read, export, delete, and administrator remediation stay usable.

### Run budget

Every run is bounded by wall time, model turns, tool-call count, per-tool time,
output/log/event bytes, and process memory/CPU/child-process count. A breached
budget ends explicitly as `TIMED_OUT` or `LIMIT_EXCEEDED`, records the dimension
and observed usage, releases capacity, and never masquerades as success.

### Usage and cost

- Token usage, provider-reported cost, run count, and duration are attributed by
  Organization, Project, Run, model, and Provider Connection.
- Missing provider cost is `unknown`, never zero.
- Thresholds are soft alerts in the pilot; they do not automatically stop runs
  and are not payment settlement.
- Org admins receive Organization alerts. Platform admins additionally receive
  anomalous-cost alerts for platform-managed connections; BYOK does not trigger
  platform financial alerts.

### Emergency control

An audited Platform Administrator control supports Organization or global
`DRAIN` (reject new work, pause queue starts, finish active runs) and `STOP_NOW`
(also cancel active runs). Both require an actor, reason, timestamp, explicit
recovery, and user-visible outcomes. This is workload control, not Organization
deletion or account suspension.

## Required production evidence

1. Production configuration validation fails closed for every missing or
   invalid platform ceiling.
2. Concurrent noisy-neighbor tests prove platform/Org/Project concurrency and
   Organization-fair scheduling without starvation.
3. Restart and crash tests prove accepted queue entries, idempotency, expiry,
   cancellation, and notifications survive process loss.
4. HTTP and Feishu load tests prove every rate-limit identity and typed overload
   response, including retry guidance.
5. File/archive/storage probes prove streaming enforcement, cleanup, and
   continued read/delete access while over limit.
6. Hung provider/tool and resource-exhaustion probes prove every run/process
   budget terminates, records cause, and releases locks/capacity.
7. Usage tests prove provider-mode attribution, unknown-cost handling, alert
   routing, and absence of hard monetary blocking.
8. `DRAIN` and `STOP_NOW` drills prove scoped, audited, reversible behavior.
9. Production-like calibration publishes the tested numerical defaults and
   capacity headroom used by the release gate.