curriculum-project-hub/docs/adr/0024-local-secret-envelope-and-connection-resolution.md

# ADR 0024: Local Secret Envelope And Connection Resolution

## Status

Accepted.

## Context

ADR-0021 requires every customer Feishu application and model provider
credential to belong to one Organization. It forbids a process-global provider
credential and requires business code to obtain plaintext only through a
resolver. Before the pilot can store real customer credentials, the deployment
also needs a concrete encryption, rotation, recovery, writer-authority, and
connection-identity contract.

The initial production shape is one Hub service and one PostgreSQL database on
a controlled host. An external KMS would add another remote availability and
bootstrap dependency before the pilot has operational support for it. Keeping
plaintext credentials in environment variables, configuration files readable
by the service account, database columns, logs, or Agent environments is not an
acceptable substitute.

## Decision

### Local master-key keyring

The pilot uses a versioned **local master-key keyring**. Each key is a random
256-bit key-encryption key (KEK) with a stable, non-secret key identifier. The
keyring names exactly one active KEK and may retain previous KEKs while rotation
is in progress. It is not stored in PostgreSQL, a release directory, an
environment variable, or the ordinary database/workspace backup set.

In production, the source keyring is owned by root, mode `0600`, and delivered
to the unprivileged Hub service by systemd `LoadCredential`. Hub reads the
read-only file under `CREDENTIALS_DIRECTORY`; production startup rejects a
direct arbitrary file path. Development and tests may use an explicit local
keyring file. Startup fails when the keyring is absent, malformed, contains an
invalid key, has no active key, or names an active key that is not present.

### Envelope format and binding

Every immutable secret version receives a new random 256-bit data-encryption
key (DEK). AES-256-GCM encrypts the secret payload with that DEK. AES-256-GCM
also wraps the DEK with the keyring's active KEK. Nonces are random and never
reused with the same key.

The versioned envelope persists only algorithm/version metadata, the KEK
identifier, nonces, authentication tags, the wrapped DEK, and ciphertext.
Authenticated additional data binds both layers to the secret purpose,
Organization identifier, Connection identifier, immutable secret-version
identifier, and envelope version. Copying ciphertext between rows,
Organizations, connections, purposes, or versions therefore fails
authentication. Unknown algorithms or versions, missing KEKs, malformed
envelopes, and failed authentication are hard errors; there is no fallback to
another credential source.

The encrypted payload is a versioned object specific to its connection type.
For a Provider Connection, provider base URL and authentication material are
inside the payload. For a Feishu Application Connection, application ID,
application secret, verification/encryption material, and bot identity are
inside the payload. HTTP responses, business records, logs, metrics, traces,
audit facts, exceptions, and Agent process environments receive only redacted
metadata, never those plaintext fields or a decrypted DEK.

### Connection identity, lifecycle, and writer authority

A connection has a stable opaque identifier and belongs to exactly one
Organization. Secret versions are immutable; the connection points to one
active version. Rotation creates and validates a new version, atomically moves
the active pointer, and retains the previous encrypted version for rollback and
audit until retention policy removes it. A connection is `DRAFT`, `ACTIVE`, or
`DISABLED`; runtime resolution accepts only an active connection with a valid
active secret version.

There is one customer-owned Feishu Application Connection per Organization.
Its provider-local user, chat, event, and callback identifiers are meaningful
only together with that connection identity. Organization OWNER/ADMIN may
create and rotate that customer-owned connection. Platform repair remains a
separate stepped-up, reason-bearing, fail-closed-audited operation under
ADR-0023, not an Organization API bypass.

A model Provider Connection is unique by Organization and provider identity.
In `BYOK` mode an Organization OWNER/ADMIN creates and rotates it. In
`PLATFORM_MANAGED` mode only a stepped-up Platform Administrator may create or
rotate it, with the platform mutation and redacted Platform Audit Entry in one
transaction. Switching modes creates or activates the correctly governed
connection version; it never re-labels plaintext or grants the other control
plane write authority.

### Resolver boundary

Business code asks one connection resolver for an active connection using an
explicit Organization, or using a Project that is first resolved to its active
Organization. Provider resolution also requires the provider identity; Feishu
resolution requires the intended Feishu Connection identity whenever external
traffic enters the system. Resolution fails for missing, draft, disabled,
cross-Organization, corrupt, or undecryptable records.

Decrypted values exist only in the resolver result for the duration of the
outbound provider/Feishu operation. They are not cached across connections,
returned through administrative reads, passed to Agent tools, or written back
to business rows. Production has no process-global Feishu or model-provider
credential fallback.

The Claude Agent child never receives an Organization's Provider credential.
For each run, Hub opens a loopback-only proxy with a random, short-lived run
capability. The child receives only that local endpoint and capability; Hub
validates the capability, replaces it with the Organization credential at the
upstream hop, refuses automatic redirects, and destroys the listener when the
run ends. Agent sandbox credential rules hide even the run capability from
tool subprocesses.

### KEK rotation, backup, and recovery

KEK rotation is staged: install a new KEK alongside the old keys, mark it
active, rewrap each stored DEK under the new KEK without decrypting its payload,
verify every envelope, then retire the old KEK only after the separately
protected recovery copy and restore drill are current. Secret rotation and KEK
rotation are distinct operations.

A recoverable deployment requires both the ordinary database/workspace backup
and a separately protected backup of the local keyring. Neither copy alone can
recover plaintext credentials. Restore preflight verifies key identifiers and
authenticates sample/all envelopes before traffic is enabled. A missing or
incorrect keyring leaves the service unavailable and produces redacted
diagnostics; operators must restore the correct keyring rather than reset or
silently discard credentials.

## Consequences

- The database may be restored or copied without exposing plaintext secrets,
  but losing the separately held keyring makes encrypted credentials
  unrecoverable.
- The service unit, installer, preflight, backup, restore, and rotation tooling
  must treat the keyring as an explicit production dependency.
- Administrative writes must accept secret material as write-only input and
  return only connection state, version, timestamps, and redacted fingerprints.
- Tests must prove cross-Organization/row substitution failure, wrong/missing
  key failure, version rotation, absence of global fallback, and absence of
  plaintext from persistence, responses, logs, and Agent environments.
- External KMS/HSM adoption is deferred. It may replace the local KEK provider
  without changing connection identity, immutable secret versions, AAD
  binding, writer authority, or resolver behavior.