forked from EduCraft/curriculum-project-hub
docs: define SaaS capacity controls
This commit is contained in:
@@ -158,9 +158,12 @@ as it upholds `AgentFileOp.Authorized`.
|
||||
*effects* are then confined, is a policy choice left to implementation. The
|
||||
invariant governs effects either way; current choice is arbitrary commands
|
||||
(sandbox confines effects).
|
||||
- **Per-user rate limiting and token budgets.** Orthogonal to the surface
|
||||
boundary (this ADR is about *what* the agent may touch, not *how often* it
|
||||
may run). Deferred to a separate concern.
|
||||
- **Inbound file size / MIME limits.** The inbox lives inside the boundary,
|
||||
but unbounded file size is a DoS vector distinct from the boundary
|
||||
invariant. Deferred.
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- **Rate limiting and token usage.** ADR-0022 requires multi-dimensional hard
|
||||
request limits while keeping token thresholds as attributed soft alerts in
|
||||
the initial release.
|
||||
- **Inbound file limits.** ADR-0022 requires hard file, attachment, archive,
|
||||
Project-storage, and Organization-storage limits with explicit failure and
|
||||
partial-file cleanup.
|
||||
|
||||
@@ -83,8 +83,10 @@ separate control plane and must not be modeled as project `MANAGE`.
|
||||
|
||||
## Open Questions / Deferred
|
||||
|
||||
- Full platform admin UI, billing, limits, and tenant suspension workflows are
|
||||
deferred.
|
||||
- Full platform admin UI, billing, and tenant suspension workflows are
|
||||
deferred. Production safety limits and workload brakes are no longer
|
||||
deferred; ADR-0022 defines them without treating a workload brake as tenant
|
||||
suspension.
|
||||
- Whether direct `USER` project grants should require active
|
||||
`OrganizationMembership` is deferred; explicit direct grants remain allowed
|
||||
for now.
|
||||
|
||||
@@ -11,8 +11,8 @@ surface is not only the Feishu bot trigger path, but also the control planes
|
||||
around it:
|
||||
|
||||
- platform staff need a private platform admin area to create and operate orgs;
|
||||
- customer org owners/admins need an org admin area for teams, roles, model
|
||||
provider configuration, projects, folders, sessions, and usage accounting;
|
||||
- customer org owners/admins need an org admin area for teams, roles, provider
|
||||
mode and BYOK configuration, projects, folders, sessions, and usage accounting;
|
||||
- ordinary teachers should be able to start work from the natural Feishu group
|
||||
flow without entering the web backend.
|
||||
|
||||
@@ -46,6 +46,19 @@ OAuth login, bot messages, and directory sync. App secrets are org-scoped
|
||||
secrets and must be encrypted at rest; business code should access them through
|
||||
a secret/connection resolver, not raw plaintext columns.
|
||||
|
||||
Each Organization chooses one model-provider credential mode:
|
||||
|
||||
- `BYOK`: an Organization OWNER/ADMIN supplies and manages that Organization's
|
||||
provider key and base URL in the org admin area;
|
||||
- `PLATFORM_MANAGED`: a platform admin supplies and manages one distinct key
|
||||
and base URL for that Organization in the platform admin area. Organization
|
||||
users cannot read the plaintext credential.
|
||||
|
||||
Provider connections remain Organization-scoped in both modes. Unrelated
|
||||
Organizations never share a process-global provider key. Provider secrets must
|
||||
be encrypted at rest and resolved at runtime through the same connection/secret
|
||||
boundary as Feishu credentials.
|
||||
|
||||
Project management has two creation paths:
|
||||
|
||||
- org owner/admin creates projects in the org web backend;
|
||||
@@ -77,9 +90,10 @@ aggregation:
|
||||
Project permissions remain on `PROJECT` resources. Moving a project between
|
||||
folders does not change grants.
|
||||
|
||||
Usage accounting is org-wise and project/folder aggregatable. It is not payment
|
||||
collection in the pilot because customers supply their own model provider API
|
||||
keys and base URLs.
|
||||
Usage accounting is org-wise and project/folder aggregatable for both provider
|
||||
modes. It is not payment collection in the pilot: BYOK spend belongs to the
|
||||
customer's provider account, while platform-managed spend is attributed to the
|
||||
Organization for operational reporting; commercial billing remains deferred.
|
||||
|
||||
## Consequences
|
||||
|
||||
@@ -89,8 +103,8 @@ keys and base URLs.
|
||||
explorer without creating a second ACL system.
|
||||
- Service code should expose project creation and chat binding as reusable
|
||||
backend operations so Feishu cards and future web APIs call the same rules.
|
||||
- Org role/model/provider/billing panels can be added on top of the org tenant
|
||||
root without changing project authorization.
|
||||
- Org role/model/provider-mode panels and the platform-managed connection panel
|
||||
can be added on top of the org tenant root without changing project authorization.
|
||||
|
||||
## Open Questions / Deferred
|
||||
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# ADR 0022: Layered Capacity Admission And Workload Safety
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
## Context
|
||||
|
||||
The initial SaaS deployment shares one production host across Organizations.
|
||||
The current Hub only has a one-run-per-project lock, a process-local five-item
|
||||
trigger queue, a five-minute queue wait, a global max-turn setting, and
|
||||
Fastify's default request-body limit. It has no global or Organization agent
|
||||
capacity, durable backpressure, fair scheduling, storage quotas, run wall-clock
|
||||
budget, multi-dimensional rate limiting, or emergency workload control.
|
||||
|
||||
Leaving these limits implicit lets one Organization monopolize the host, lets
|
||||
accepted work disappear on restart, and makes overload look like success. A
|
||||
single global FIFO would preserve arrival order while still permitting one
|
||||
tenant to occupy every execution slot. Unlimited queues or automatic whole-run
|
||||
replay would hide overload and can duplicate tool side effects.
|
||||
|
||||
## Decision
|
||||
|
||||
Every mandatory capacity dimension has a non-optional **Platform Capacity
|
||||
Ceiling**. An Organization OWNER/ADMIN may configure a lower **Organization
|
||||
Policy Limit**; the **Effective Limit** is the lower value, or the platform
|
||||
ceiling when the Organization value is absent. There is no unlimited fallback.
|
||||
Production refuses to start when a mandatory platform ceiling is absent or
|
||||
invalid. Platform ceilings are changed through versioned deployment
|
||||
configuration in the initial release; live platform-admin editing is deferred
|
||||
until audited approval and rollback exist.
|
||||
|
||||
Agent work uses a durable, bounded admission queue. Scheduling is fair across
|
||||
Organizations and FIFO within an Organization, while the existing one-active-
|
||||
run-per-Project rule remains. Capacity saturation queues only within effective
|
||||
length and wait limits. Queue overflow is an explicit `capacity_exhausted`
|
||||
response with retry guidance; an accepted request that waits too long becomes
|
||||
`EXPIRED`, is never started later, and notifies its initiator. Initiators may
|
||||
withdraw their own queued requests; Project `MANAGE` and Organization admins
|
||||
may cancel queued requests in their scope. Every transition is auditable.
|
||||
|
||||
The platform never automatically replays a whole Agent Run after execution has
|
||||
started. Limited transport retries are allowed only before model output or tool
|
||||
side effects. Otherwise the run ends explicitly and a retry is a new user-
|
||||
initiated request. Feishu event identity and API idempotency prevent delivery
|
||||
retries from creating duplicate admissions.
|
||||
|
||||
The first production gate requires hard limits for request rate/body size,
|
||||
agent concurrency, queue length/wait, files and attachment counts, archive
|
||||
expansion, Project and Organization storage, Organization entity counts, run
|
||||
wall time/turns/tool calls/output, individual tool duration, and execution
|
||||
process memory/CPU/process count. Reaching an entity or storage ceiling blocks
|
||||
new growth but does not delete, hide, or lock existing data; read, export,
|
||||
delete, and administrator remediation remain available.
|
||||
|
||||
Admin surfaces require an authenticated user plus the corresponding platform
|
||||
or Organization administrator authorization. Authenticated Web traffic is
|
||||
limited by user, Organization, and endpoint class; signed Feishu traffic is
|
||||
limited by Organization application, chat, and sender; administrative writes
|
||||
use separate stricter buckets. Rejections and truncation are never silent.
|
||||
|
||||
Token, provider-reported cost, run count, and duration are attributed by
|
||||
Organization, Project, Run, model, and Provider Connection. They are statistics
|
||||
and soft alerts in the initial release, not hard spend controls or settlement.
|
||||
Missing provider cost remains unknown rather than zero. Organization admins
|
||||
receive their alerts; platform admins additionally receive anomalous-cost
|
||||
alerts for platform-managed Provider Connections, not BYOK financial alerts.
|
||||
|
||||
Platform administrators have audited, reason-bearing, reversible workload
|
||||
brakes at Organization and platform scope. `DRAIN` rejects new work and pauses
|
||||
queue starts while allowing active runs to finish. `STOP_NOW` also cancels
|
||||
active runs. These controls do not delete or suspend the Organization.
|
||||
|
||||
Exact numerical ceilings remain open until production-like capacity tests
|
||||
measure the supported host and provider behavior. Defaults must be checked in,
|
||||
validated, observable, and proven at the release gate rather than guessed as
|
||||
domain constants.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The current in-memory `TriggerQueue` and global-only agent settings are
|
||||
implementation divergences, not acceptable production fallbacks.
|
||||
- Durable ingress, run/lock recovery, authorization, telemetry, and capacity
|
||||
enforcement must share one Organization-scoped admission identity.
|
||||
- HTTP overload uses typed `429` responses with `Retry-After`; Feishu reports a
|
||||
clear busy/expired/canceled outcome to the initiating user.
|
||||
- Capacity calibration is a release prerequisite and must include noisy-neighbor,
|
||||
restart, queue expiry, file/archive, provider-stall, and workload-brake tests.
|
||||
Reference in New Issue
Block a user