feat: secure organization provider credentials

docs/adr/0024-local-secret-envelope-and-connection-resolution.md

@@ -0,0 +1,143 @@
+# ADR 0024: Local Secret Envelope And Connection Resolution
+
+## Status
+
+Accepted.
+
+## Context
+
+ADR-0021 requires every customer Feishu application and model provider
+credential to belong to one Organization. It forbids a process-global provider
+credential and requires business code to obtain plaintext only through a
+resolver. Before the pilot can store real customer credentials, the deployment
+also needs a concrete encryption, rotation, recovery, writer-authority, and
+connection-identity contract.
+
+The initial production shape is one Hub service and one PostgreSQL database on
+a controlled host. An external KMS would add another remote availability and
+bootstrap dependency before the pilot has operational support for it. Keeping
+plaintext credentials in environment variables, configuration files readable
+by the service account, database columns, logs, or Agent environments is not an
+acceptable substitute.
+
+## Decision
+
+### Local master-key keyring
+
+The pilot uses a versioned **local master-key keyring**. Each key is a random
+256-bit key-encryption key (KEK) with a stable, non-secret key identifier. The
+keyring names exactly one active KEK and may retain previous KEKs while rotation
+is in progress. It is not stored in PostgreSQL, a release directory, an
+environment variable, or the ordinary database/workspace backup set.
+
+In production, the source keyring is owned by root, mode `0600`, and delivered
+to the unprivileged Hub service by systemd `LoadCredential`. Hub reads the
+read-only file under `CREDENTIALS_DIRECTORY`; production startup rejects a
+direct arbitrary file path. Development and tests may use an explicit local
+keyring file. Startup fails when the keyring is absent, malformed, contains an
+invalid key, has no active key, or names an active key that is not present.
+
+### Envelope format and binding
+
+Every immutable secret version receives a new random 256-bit data-encryption
+key (DEK). AES-256-GCM encrypts the secret payload with that DEK. AES-256-GCM
+also wraps the DEK with the keyring's active KEK. Nonces are random and never
+reused with the same key.
+
+The versioned envelope persists only algorithm/version metadata, the KEK
+identifier, nonces, authentication tags, the wrapped DEK, and ciphertext.
+Authenticated additional data binds both layers to the secret purpose,
+Organization identifier, Connection identifier, immutable secret-version
+identifier, and envelope version. Copying ciphertext between rows,
+Organizations, connections, purposes, or versions therefore fails
+authentication. Unknown algorithms or versions, missing KEKs, malformed
+envelopes, and failed authentication are hard errors; there is no fallback to
+another credential source.
+
+The encrypted payload is a versioned object specific to its connection type.
+For a Provider Connection, provider base URL and authentication material are
+inside the payload. For a Feishu Application Connection, application ID,
+application secret, verification/encryption material, and bot identity are
+inside the payload. HTTP responses, business records, logs, metrics, traces,
+audit facts, exceptions, and Agent process environments receive only redacted
+metadata, never those plaintext fields or a decrypted DEK.
+
+### Connection identity, lifecycle, and writer authority
+
+A connection has a stable opaque identifier and belongs to exactly one
+Organization. Secret versions are immutable; the connection points to one
+active version. Rotation creates and validates a new version, atomically moves
+the active pointer, and retains the previous encrypted version for rollback and
+audit until retention policy removes it. A connection is `DRAFT`, `ACTIVE`, or
+`DISABLED`; runtime resolution accepts only an active connection with a valid
+active secret version.
+
+There is one customer-owned Feishu Application Connection per Organization.
+Its provider-local user, chat, event, and callback identifiers are meaningful
+only together with that connection identity. Organization OWNER/ADMIN may
+create and rotate that customer-owned connection. Platform repair remains a
+separate stepped-up, reason-bearing, fail-closed-audited operation under
+ADR-0023, not an Organization API bypass.
+
+A model Provider Connection is unique by Organization and provider identity.
+In `BYOK` mode an Organization OWNER/ADMIN creates and rotates it. In
+`PLATFORM_MANAGED` mode only a stepped-up Platform Administrator may create or
+rotate it, with the platform mutation and redacted Platform Audit Entry in one
+transaction. Switching modes creates or activates the correctly governed
+connection version; it never re-labels plaintext or grants the other control
+plane write authority.
+
+### Resolver boundary
+
+Business code asks one connection resolver for an active connection using an
+explicit Organization, or using a Project that is first resolved to its active
+Organization. Provider resolution also requires the provider identity; Feishu
+resolution requires the intended Feishu Connection identity whenever external
+traffic enters the system. Resolution fails for missing, draft, disabled,
+cross-Organization, corrupt, or undecryptable records.
+
+Decrypted values exist only in the resolver result for the duration of the
+outbound provider/Feishu operation. They are not cached across connections,
+returned through administrative reads, passed to Agent tools, or written back
+to business rows. Production has no process-global Feishu or model-provider
+credential fallback.
+
+The Claude Agent child never receives an Organization's Provider credential.
+For each run, Hub opens a loopback-only proxy with a random, short-lived run
+capability. The child receives only that local endpoint and capability; Hub
+validates the capability, replaces it with the Organization credential at the
+upstream hop, refuses automatic redirects, and destroys the listener when the
+run ends. Agent sandbox credential rules hide even the run capability from
+tool subprocesses.
+
+### KEK rotation, backup, and recovery
+
+KEK rotation is staged: install a new KEK alongside the old keys, mark it
+active, rewrap each stored DEK under the new KEK without decrypting its payload,
+verify every envelope, then retire the old KEK only after the separately
+protected recovery copy and restore drill are current. Secret rotation and KEK
+rotation are distinct operations.
+
+A recoverable deployment requires both the ordinary database/workspace backup
+and a separately protected backup of the local keyring. Neither copy alone can
+recover plaintext credentials. Restore preflight verifies key identifiers and
+authenticates sample/all envelopes before traffic is enabled. A missing or
+incorrect keyring leaves the service unavailable and produces redacted
+diagnostics; operators must restore the correct keyring rather than reset or
+silently discard credentials.
+
+## Consequences
+
+- The database may be restored or copied without exposing plaintext secrets,
+  but losing the separately held keyring makes encrypted credentials
+  unrecoverable.
+- The service unit, installer, preflight, backup, restore, and rotation tooling
+  must treat the keyring as an explicit production dependency.
+- Administrative writes must accept secret material as write-only input and
+  return only connection state, version, timestamps, and redacted fingerprints.
+- Tests must prove cross-Organization/row substitution failure, wrong/missing
+  key failure, version rotation, absence of global fallback, and absence of
+  plaintext from persistence, responses, logs, and Agent environments.
+- External KMS/HSM adoption is deferred. It may replace the local KEK provider
+  without changing connection identity, immutable secret versions, AAD
+  binding, writer authority, or resolver behavior.