docs: define SaaS capacity controls

.scratch/saas-production-readiness/assets/initial-abuse-capacity-controls.md

@@ -0,0 +1,125 @@
+# Initial abuse and capacity controls
+
+## Verdict
+
+The initial production boundary requires hard safety controls before customer
+traffic. The current project lock and small in-memory trigger queue are useful
+local protections, but they do not provide multi-tenant admission, durable
+backpressure, storage safety, request isolation, bounded execution, or an
+operator stop mechanism.
+
+The accepted contract is recorded in
+[ADR-0022](../../../docs/adr/0022-layered-capacity-admission.md) and
+[`Spec.System.Capacity`](../../../spec/Spec/System/Capacity.lean). Exact values
+remain deliberately open until production-like calibration; the required
+dimensions and failure behaviors do not.
+
+## Current implementation inventory
+
+| Surface | Existing protection | Production gap |
+| --- | --- | --- |
+| HTTP | Fastify's default 1 MiB body limit | No route-class, user, Org, IP, or administrative-write rate limits; request/connection/handler timeouts are unset |
+| Project execution | Database lock permits one active run per Project | No platform or Org concurrency ceiling and no fair scheduler |
+| Busy Project | Process-local FIFO, default five items and five-minute wait | Restart loses accepted work; expiry is logged and dropped without a durable state or user notification |
+| Agent execution | Global max-turn default of 25 | No wall-clock, tool-call, output, memory, CPU, process, or Org-specific budget |
+| Feishu messages | Message batching limits text count/characters | File/post attachment count and size are unbounded; downloads may buffer whole files |
+| Storage | Workspace path boundary | No Project or Org byte quota, archive-expansion limit, or entity-count ceiling |
+| Usage | Provider-reported token/cost fields and Org aggregation | Missing cost can disappear from totals; no soft-threshold notification or provider-mode attribution contract |
+| Host | systemd lifecycle | No service CPU, memory, or task ceiling and no audited workload brake |
+
+Most Organization member/team/project lists are also unpaginated, and usage
+aggregation can load all matching runs. Those query shapes must be bounded as
+part of enforcement rather than hidden behind larger process memory.
+
+## Accepted contract
+
+### Layered limits
+
+- Every mandatory dimension has a versioned, non-overridable platform ceiling.
+- An Organization OWNER/ADMIN may configure a lower policy limit.
+- Effective limit is `min(platform, organization)`; absent Organization policy
+  means the platform ceiling, never unlimited.
+- Missing or invalid mandatory production ceilings fail startup.
+
+### Admission and fairness
+
+- Project exclusivity remains one active run.
+- Platform and Organization concurrency limits apply before starting work.
+- Available slots are scheduled fairly across Organizations, FIFO within an
+  Organization, so one tenant cannot monopolize a global FIFO.
+- Accepted work is durable and queryable. Queue length and wait are bounded.
+- Overflow is explicit `capacity_exhausted` (`429` plus `Retry-After` for HTTP;
+  a clear busy response for Feishu). Nothing is silently dropped.
+- Wait timeout produces terminal `EXPIRED`, not a later automatic start.
+- The initiator may withdraw their queued request. Project `MANAGE` or an
+  Organization admin may cancel queued requests in their scope. Actor, reason,
+  and notification are recorded.
+
+### Retry and idempotency
+
+- Feishu `event_id` and API idempotency identify one admission across retries.
+- Once a run starts, the whole run is never automatically replayed because tool
+  side effects may already exist.
+- Only bounded transport retries before model output or tool side effects are
+  automatic. Later failure/timeouts require an explicit new request.
+
+### Request, file, storage, and entity safety
+
+- Authenticated Web requests are limited by user, Organization, and endpoint
+  class; signed Feishu events by Organization app, chat, and sender;
+  administrative writes use stricter independent buckets.
+- Admin surfaces require both a valid user session and the corresponding
+  platform/Organization administrator authorization. OAuth callbacks and
+  signed Feishu ingress remain necessary non-session entry points.
+- Hard limits cover request body, single-file bytes, attachment count, archive
+  expanded bytes/file count/depth, Project/Organization storage, and per-type
+  Organization entity counts.
+- Transfer is streamed and counted; partial files are removed after failure.
+- Crossing a ceiling blocks new growth without auto-deleting or hiding existing
+  resources. Read, export, delete, and administrator remediation stay usable.
+
+### Run budget
+
+Every run is bounded by wall time, model turns, tool-call count, per-tool time,
+output/log/event bytes, and process memory/CPU/child-process count. A breached
+budget ends explicitly as `TIMED_OUT` or `LIMIT_EXCEEDED`, records the dimension
+and observed usage, releases capacity, and never masquerades as success.
+
+### Usage and cost
+
+- Token usage, provider-reported cost, run count, and duration are attributed by
+  Organization, Project, Run, model, and Provider Connection.
+- Missing provider cost is `unknown`, never zero.
+- Thresholds are soft alerts in the pilot; they do not automatically stop runs
+  and are not payment settlement.
+- Org admins receive Organization alerts. Platform admins additionally receive
+  anomalous-cost alerts for platform-managed connections; BYOK does not trigger
+  platform financial alerts.
+
+### Emergency control
+
+An audited Platform Administrator control supports Organization or global
+`DRAIN` (reject new work, pause queue starts, finish active runs) and `STOP_NOW`
+(also cancel active runs). Both require an actor, reason, timestamp, explicit
+recovery, and user-visible outcomes. This is workload control, not Organization
+deletion or account suspension.
+
+## Required production evidence
+
+1. Production configuration validation fails closed for every missing or
+   invalid platform ceiling.
+2. Concurrent noisy-neighbor tests prove platform/Org/Project concurrency and
+   Organization-fair scheduling without starvation.
+3. Restart and crash tests prove accepted queue entries, idempotency, expiry,
+   cancellation, and notifications survive process loss.
+4. HTTP and Feishu load tests prove every rate-limit identity and typed overload
+   response, including retry guidance.
+5. File/archive/storage probes prove streaming enforcement, cleanup, and
+   continued read/delete access while over limit.
+6. Hung provider/tool and resource-exhaustion probes prove every run/process
+   budget terminates, records cause, and releases locks/capacity.
+7. Usage tests prove provider-mode attribution, unknown-cost handling, alert
+   routing, and absence of hard monetary blocking.
+8. `DRAIN` and `STOP_NOW` drills prove scoped, audited, reversible behavior.
+9. Production-like calibration publishes the tested numerical defaults and
+   capacity headroom used by the release gate.

.scratch/saas-production-readiness/assets/tenant-auth-security-audit.md

@@ -29,8 +29,10 @@ The audit used these upstream requirements:
   run's project workspace.
 - ADR-0019: authorization is evaluated against principals resolved at request
   time so membership changes take effect immediately.
-- ADR-0021: customer Feishu and model-provider credentials are org-scoped
-  secrets, encrypted at rest, and accessed through a resolver.
+- ADR-0021: Feishu and model-provider credentials are org-scoped secrets,
+  encrypted at rest, and accessed through a resolver. Provider ownership may be
+  Organization-managed BYOK or a distinct platform-managed connection for that
+  Organization; neither mode permits a process-global shared key.
 
 ## Confirmed protections
 
@@ -130,10 +132,10 @@ encrypted credentials "deferred", although ADR-0021 says they **must** be
 org-scoped, encrypted at rest, and accessed through a resolver.
 
 `EnvRuntimeSettings.provider` similarly discards its `scope` argument and uses
-one global OpenRouter token for every project. This both leaks tenant ownership
-of credentials and makes the accepted "customers supply their own provider
-keys" accounting model false. The schema has no encrypted credential store,
-key version, rotation state, or resolver seam.
+one global OpenRouter token for every project. This erases the accepted
+per-Organization provider mode and makes both BYOK and platform-managed cost
+attribution false. The schema has no encrypted credential store, key version,
+rotation state, or resolver seam.
 
 This is also a product reachability blocker: one customer-owned Feishu app
 cannot receive bot/OAuth traffic for unrelated customer tenants under the
@@ -248,7 +250,9 @@ Production readiness requires automated proof of all of the following:
    organization, project, run, and pending action consistently.
 4. ACTIVE Organization status is enforced in the shared authorization seam.
 5. Feishu and provider credentials resolve by Organization, are encrypted at
-   rest with versioned keys, never enter logs/agent tools, and can rotate.
+   rest with versioned keys, never enter logs/agent tools, and can rotate;
+   provider resolution honors either Organization-managed BYOK or a distinct
+   platform-managed connection for that Organization.
 6. Cross-org HTTP and Prisma service tests remain green, while new negative
    tests cover every finding above.
 7. Production Node and Rust dependency audits pass with no ignored