curriculum-project-hub/.scratch/saas-production-readiness/assets/product-surface-completeness-audit.md

Accepted SaaS product-surface completeness audit

Audit date: 2026-07-10

Verdict

The current Hub is not a usable or production-complete SaaS product surface for the accepted destination. It has a worthwhile Organization-scoped backend foundation and a well-tested single-Feishu-application Agent Run slice, but the running product still has one process-global Feishu app/bot and one global model provider credential. It has no platform-administrator control plane, no Organization connection/secret plane, and no Organization-scoped Provider Credential Mode. Its only HTML product page is a login form; successful OAuth redirects an administrator to an unmounted /admin/org/:orgSlug route.

ADR-0022's required production safety surface is also almost wholly absent: there is no durable Organization-fair admission, layered platform/Organization limits, hard file/storage/run/process budgets, usage-alert control plane, or audited workload brake. These are release blockers, not optional polish.

The strongest code should be retained rather than rewritten: all current /api/org/:orgSlug/* routes reauthorize an active OWNER/ADMIN membership; Folder/Project, member, Team, Team-to-Project access, session, and usage services are Organization-scoped; and the shared project-onboarding services correctly create Project/chat/grant relationships for the single-app case. The shortest route is to add explicit Organization connection identity and the two private admin control planes around those seams.

The claim-by-claim source inventory, including route, schema and test locations, is in accepted product-surface code inventory. Official Feishu identifier, OAuth, event and long-connection semantics are in Feishu product-surface primary sources.

Accepted boundary

The audit treats these as required because they are already accepted by the repository's CONTEXT.md, ADR-0020, ADR-0021, ADR-0022, and Lean contract:

• a Platform Administrator is separate from Organization membership and uses a private /admin/platform area to create and operate Organizations;
• an Organization OWNER/ADMIN uses a private /admin/org/:orgSlug area to manage members, projects, Teams, connections, settings, sessions and usage;
• every Organization authenticates and exchanges bot/directory traffic through its own customer-owned Feishu Application;
• every Organization chooses BYOK or an Organization-distinct platform-managed Provider Connection, with different administrators owning the respective plaintext inputs;
• ordinary members can create/bind a Project from an unbound Feishu group only under the Organization policy, while all Project actions retain Project-level authorization;
• production has mandatory platform ceilings, optional lower Organization limits, durable fair admission, bounded work and audited emergency brakes.

The audit does not add deferred self-service signup, payment collection, folder ACL inheritance, commercial billing, or a live editor for platform capacity ceilings. Guided Feishu app setup and manually initiated Organization creation by an authenticated Platform Administrator remain valid for the pilot.

Journey result

Accepted journey	Current product entry	Result	Owning frontier
Platform Administrator signs in	No platform login/session/guard; only a stored PlatformRoleAssignment type	Missing	tickets 42, 45
Platform Administrator creates and operates Organizations	Direct Prisma calls in test fixtures only; no /api/platform/* or UI	Test/database only	ticket 45
Platform Administrator configures each customer Feishu app	One env FEISHU_APP_ID/SECRET/BOT_OPEN_ID; no connection or secret model	Missing and contract-divergent	tickets 43–45
Platform Administrator supplies a distinct platform-managed provider key/base URL	One process-global ANTHROPIC_AUTH_TOKEN/BASE_URL; resolver ignores Project scope	Missing and contract-divergent	tickets 43, 45
Platform Administrator applies and reverses DRAIN/STOP_NOW	No persistence, guard, API or UI	Missing	tickets 37, 45
Organization user logs in through that Organization's Feishu app	OAuth exists, but is built once from the global app and has no Organization/application issuer	Single-app only	tickets 43, 44
Logged-in Organization OWNER/ADMIN opens the management panel	OAuth redirects to /admin/org/:slug; no handler/static application serves it	Stable 404	ticket 46
Organization OWNER/ADMIN manages members, Teams, Projects/Folders and Team access	Guarded Organization JSON APIs and useful integration coverage exist	Backend present; browser product absent	ticket 46
Organization OWNER/ADMIN manages permission/model/role/run policy	Only membersCanCreateProjects is exposed; other stored/runtime settings have no control plane	Partial	ticket 46
Organization selects Provider Credential Mode and manages BYOK	No mode/connection model, API or UI	Missing	tickets 43, 46
Organization views sessions and usage	Guarded JSON APIs exist, but have no direct product-boundary tests or UI and cannot attribute Provider Connection	Partial	tickets 36, 46
Ordinary member creates or binds a Project in Feishu	Card/service path is present and well tested	Works for the one-global-app case	ticket 44 for SaaS routing
Authorized member runs the agent through its Organization bot	Full single-app trigger path exists; provider/app resolution is global	Works only as a single-app deployment	tickets 43, 44 plus lifecycle tickets
Accepted work queues fairly and durably across Organizations	Five-item process-memory Project queue	Contract-divergent	tickets 23, 32–35
Organization sets lower policy limits and receives soft alerts	No policy/alert persistence or product surface	Missing	tickets 32, 34–36, 46

Executed defect evidence

The probes used the real Fastify plugin and test PostgreSQL database through a temporary Vitest integration file. The file was removed after capture and was run twice to separate deterministic behavior from a transient test failure.

1. The successful-login destination is not served

The probe seeded an active Organization ADMIN, minted the normal signed session cookie, and requested the exact destination selected by resolvePostLoginRedirect:

GET /admin/org/test-default
expected_status=200
actual_status=404

Both runs returned 404. The root cause is structural: registerAdminPlugin mounts cookie, auth and Organization API routes only; its own comment says the static SPA is “later”. No other repository HTML/TSX/Vue/Svelte/CSS product application or deployment-level static handler exists. Existing OAuth tests assert only the redirect Location, so they stop immediately before the broken journey.

This is not an authorization defect in the JSON APIs. Those APIs correctly return 401 without a session and 403 to a MEMBER. It is a reachability defect: there is no management panel for either an authorized or unauthorized request to reach. Ticket 46 must guard both the page shell and its data, so an anonymous user may reach only the login entry and never management content.

2. Different Organizations start OAuth with the same application

The probe seeded two Organizations and requested the current OAuth entry with two different Organization return paths:

GET /auth/feishu?returnTo=/admin/org/test-default
client_id=cli_process_global

GET /auth/feishu?returnTo=/admin/org/second
client_id=cli_process_global

Both runs produced the same result. The root cause is that registerAuthRoutes constructs one immutable OAuth configuration from plugin startup arguments. The returnTo path is signed only as a destination; it does not resolve or bind an Organization Feishu Application. server.ts obtains the same app credentials from one environment tuple and also uses them for the only WebSocket listener.

Changing only the redirect page cannot fix this. The composition root and every identifier-bearing record/action need an explicit Organization connection identity, backed by the encrypted resolver in ticket 43.

Feishu identifier and transport consequence

Feishu's official identifier contract makes the current global identifier shape unsafe for customer-owned applications:

• open_id identifies a user inside one application and differs across apps;
• user_id is stable inside one tenant but differs across tenants;
• union_id can link apps from the same application service provider, but the service cannot assume independently customer-owned apps share that provider;
• OAuth authorization identifies the app with client_id; the callback brings code and state, so state and token exchange must remain bound to the selected Organization/application;
• v2 HTTP event/card headers expose application and tenant identity, while an encrypted outer request must be routed to the correct connection before its verification/decryption key is known; a long connection is itself created by one app credential pair, so the local client/listener identity is the issuer;
• Feishu rejects cross-tenant chat/operator combinations and does not provide a contract that permits chat_id to be treated as a global SaaS key.

The current User.feishuOpenId @unique, session {userId, feishuOpenId}, ProjectGroupBinding.chatId, FeishuEventReceipt.eventId, principal IDs, sender caches, and batching keys omit this issuer namespace. An open_id collision across customer apps could merge unrelated external identities; different open_id values for one person across apps cannot be resolved as one global identity automatically. A globally looked-up chatId or event ID has the analogous ambiguity.

The implementation must therefore persist provider-local external identities under a Feishu Application Connection, for example conceptually (connectionId, externalIdType, externalId), and carry that connection through OAuth state/session, ingress, deduplication, chat binding, authorization, onboarding/card actions, rate-limit keys, directory sync, outbound delivery and audit. Whether two external identities represent the same global User is a separate explicit linking decision; it must not be inferred from a provider ID outside its documented scope.

Root causes rather than symptoms

1. Tenant data was added without tenant-scoping external connections

Organization ownership is correctly present on Projects, Teams, Folders and directory-connection metadata, but the process composition still comes from the former single-deployment shape. The app/bot/provider credentials and the identifiers they issue never crossed an Organization connection boundary. RuntimeScope.projectId exists but is discarded by the env resolver, making the missing seam especially explicit.

2. The control-plane work stopped at guarded Organization APIs

The Organization routes provide a credible backend slice, but there is no web application, platform route family, platform guard, Organization provisioning service, connection configuration, or browser journey test. The comment “until admin SPA lands” and redirect-only OAuth assertion document the precise unfinished handoff. A small catch-all page would only hide the gap; the fix must deliver the authenticated, authorized workflows and their safe states.

3. Production capacity semantics exist only in the contract

ADR-0022 intentionally declares the in-memory queue and global-only settings to be implementation divergences. There is no common admission identity that can join Organization, Feishu connection, Project, user and request, so fair scheduling, workload brakes, rate limits, alerts and control-plane operations cannot currently compose. The previously created capacity tickets own this implementation; the admin tickets must expose their safe operator/customer surfaces rather than invent parallel controls.

What should remain

• Keep requireOrgRole's per-request session, active Organization, active membership and OWNER/ADMIN checks; extend the same principle to page data and add a separate requirePlatformAdmin rather than an Organization override.
• Keep the Organization-scoped explorer/member/Team/access/session/usage APIs as the initial backend for the org panel, while filling their missing tests, pagination/limits, audit and typed-error contracts through existing tickets.
• Keep shared Project creation and chat-binding services so web and Feishu entries do not drift into two rule sets.
• Keep the single-app Feishu handler's authorization, onboarding, run/session and dedup behavior, but call it from a connection-bound listener context and move accepted work/delivery into the durable lifecycle tickets.
• Keep missing provider cost unknown rather than fabricating zero; add Provider Connection attribution before presenting SaaS usage claims.

Newly clear execution frontier

The product audit creates four implementation/decision slices after the existing secret and browser-boundary design work:

1. Decide the platform-administrator identity and audit boundary.
2. Implement the Organization connection and secret plane.
3. Route Feishu identity and traffic by Organization connection.
4. Build the private platform-administrator control plane.
5. Build the private Organization-administrator control plane.

The Feishu routing slice precedes durable inbound/action/outbound and signed traffic limiting so those mechanisms use the right issuer namespace from their first durable schema. The platform panel precedes the emergency-brake product entry so ticket 37 can reuse one authenticated/audited operator boundary. Existing tickets 32–38 continue to own capacity enforcement, attribution, alerts, brakes and calibration; the new UI tickets do not duplicate them.

Required production journey evidence

The final release gate must run against at least two configured customer-owned Feishu applications and two Organizations, using deliberately colliding fake provider-local IDs where the test harness permits. It must prove:

1. an unauthenticated request and an authenticated wrong-role user cannot load either management page data or perform mutations;
2. a Platform Administrator can create an Organization, bootstrap its OWNER, configure/test its Feishu app and distinct platform-managed Provider Connection, with redacted responses and complete audit history;
3. an Organization OWNER/ADMIN logs in through that Organization's app, reaches the real org panel, manages the accepted settings/resources, configures BYOK when selected, and cannot read another Organization or platform secret;
4. each bot receives, binds, authorizes, queues and replies only inside its connection's Organization even when open_id, chat_id, event or callback values collide; SUSPENDED/ARCHIVED Organizations fail closed;
5. provider resolution for every run returns that Organization's selected connection and never a process-global fallback;
6. current member onboarding, Project authorization and Agent Run behavior remain intact after connection namespacing;
7. layered limits, fair admission, usage alerts and DRAIN/STOP_NOW are visible only to the corresponding authorized control plane and behave as proven by their dedicated fault/capacity suites;
8. the complete browser + Feishu journey survives the production-shaped deploy, restart, backup/restore and rollback evidence owned by ticket 08.