diff --git a/developer-docs/README.md b/developer-docs/README.md index 1958c5c9..6908da2a 100644 --- a/developer-docs/README.md +++ b/developer-docs/README.md @@ -10,6 +10,7 @@ see the root [README](../README.md#documentation) for that split. - [Getting started](getting-started.md) — development setup, environment variables, running locally - [Local development](local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed) - [Architecture](architecture.md) — system overview, data flow, and design decisions +- [MCP Apps](mcp-apps.md) — run-report View: host render flow, the three app-only tools, the shared data source, `mcp-apps:build`, and the gitignored HTML artifact - [Testing](testing.md) — per-package test frameworks, the shared test-server fixture, S3 mocking - [Watcher](watcher.md) — CLI commands, configuration, run detection, upload modes - [Lambda](lambda.md) — processing pipeline, supported instruments, adding new instruments diff --git a/developer-docs/architecture.md b/developer-docs/architecture.md index 87b42ff4..6a4baf8b 100644 --- a/developer-docs/architecture.md +++ b/developer-docs/architecture.md @@ -60,5 +60,6 @@ Steps 1–3 are the same, but the watcher does not upload immediately. Instead: - **Public page metadata, gated bodies.** Routes commonly shared into Slack/Notion (dashboard, instruments, instrument and run detail, settings) are reachable without a session so link unfurlers can read `` metadata; the page or layout renders a `SignInRequired` CTA in place of the real body when there's no session. `/watchers/*` stays redirected to `/login` by `web/proxy.ts`. Three independent layers prevent search indexing: a `robots` field on the root layout, an `app/robots.ts` that disallows generic crawlers from the product (with `/docs`, the homepage, and icon files allowed so Google can show a favicon, plus an allow-list for Slack/Discord/LinkedIn unfurl bots), and an `X-Robots-Tag: noindex, nofollow` header on responses other than those icon files. - **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so they stay consistent across Lambda and the watcher without duplicating code. - **Wildcard PAT scope for the legacy backfill.** Migration `0022_pat_scopes` backfilled every pre-existing token with `["*"]` so deployed watchers and the Lambda kept working once scope enforcement shipped. `POST /api/v1/tokens` rejects `*` from API callers, so every token minted since carries explicit least-privilege scopes; see [Security and permissions](https://datahub.arcadiascience.com/docs/security#token-scopes) for the scope vocabulary. +- **MCP Apps run report.** `/mcp/v1` also serves a `ui://data-hub/run-report` HTML View (built by `npm run mcp-apps:build` into a gitignored `web/mcp-apps/dist/run-report.html`) plus three app-only tools the host uses to fetch items, CSV pages, and JSON artifacts. See [MCP Apps](mcp-apps.md). - **MCP for AI access (OAuth).** Streamable-HTTP MCP at `/mcp/v1` exposes tools, resources, and prompts. Clients use Better Auth as the OAuth AS (issuer `{BETTER_AUTH_URL}/api/auth`); discovery via `/.well-known/oauth-protected-resource` (incl. `/mcp/v1`), `/.well-known/oauth-authorization-server`, and `/.well-known/openid-configuration`, with consent at `/consent` (shows registered redirect hosts from the DB). Coarse scopes: transport requires `read` (WWW-Authenticate still advertises `read write` so clients request both); mutating tools need `write`. Access tokens must be JWTs with `aud` = `{origin}/mcp/v1` (clients pass RFC 8707 `resource` at token exchange); verified offline via JWKS (~1h), so revoking consent does not kill them immediately. `MCP_ALLOW_PAT_AUTH=true` optionally accepts PATs outside production (hard-disabled on Vercel production and on self-hosted production with a non-loopback `BETTER_AUTH_URL`; PAT `write` requires the `*` wildcard). See [Local development](local-development.md#connecting-an-mcp-client) and the [MCP overview](https://datahub.arcadiascience.com/docs/mcp). - **Integration tests against a real server.** The shared `testing.py` module spins up a real Next.js server backed by a Postgres database, so Lambda and watcher integration tests exercise the actual API surface. diff --git a/developer-docs/first-time-deployment.md b/developer-docs/first-time-deployment.md index 0ff112c2..c76cf493 100644 --- a/developer-docs/first-time-deployment.md +++ b/developer-docs/first-time-deployment.md @@ -196,7 +196,7 @@ The AWS stack you just deployed exposes the outputs the web app needs to reach i | `AWS_ROLE_ARN` | `WebAppRoleArn` stack output | Lets the web app generate presigned S3 URLs and SigV4-sign Lambda Function URL invocations via OIDC federation. | | `LAMBDA_FUNCTION_URL` | `DataHubFunctionUrl` stack output | The Lambda Function URL for manual reprocessing and archive builds. | -The S3 bucket names default to `arcadia-data-hub-raw-` and `arcadia-data-hub-archives-`; override `S3_RAW_DATA_BUCKET` and `S3_ARCHIVES_BUCKET` in Vercel only if your stack uses different names. Redeploy the web app (or push a commit) so it picks up the new variables. +The S3 bucket names default to `arcadia-data-hub-raw-` and `arcadia-data-hub-archives-`; override `S3_RAW_DATA_BUCKET` and `S3_ARCHIVES_BUCKET` in Vercel only if your stack uses different names. Also set `S3_PROCESSED_BUCKET` to `arcadia-data-hub-processed-`: presigned download URLs come from each file row's own bucket, so nothing else needs it, but the MCP Apps run report has to name every origin in its content security policy and processed artifacts are most of what it renders. Redeploy the web app (or push a commit) so it picks up the new variables. > **Note:** Slack channel notifications are configured after deploy, not via an environment variable. A workspace admin pastes the incoming webhook URL under Settings > Notifications > Slack channel (stored in the `slack_channel_config` table). Personal Slack DMs use the optional `SLACK_BOT_TOKEN` / `SLACK_CLIENT_ID` / `SLACK_CLIENT_SECRET` variables instead — see [Environment variables](getting-started.md#environment-variables). diff --git a/developer-docs/getting-started.md b/developer-docs/getting-started.md index 196e08cb..951b8c14 100644 --- a/developer-docs/getting-started.md +++ b/developer-docs/getting-started.md @@ -54,6 +54,7 @@ vercel env pull | `AWS_REGION` | No | AWS region for S3 presigned URLs and Lambda Function URL SigV4 signing (defaults to `us-west-1`) | | `AWS_ROLE_ARN` | No | IAM role ARN for Vercel OIDC federation. Used to presign S3 URLs and SigV4-sign Lambda Function URL invocations (only needed on Vercel) | | `S3_RAW_DATA_BUCKET` | No | S3 bucket for raw data uploads (defaults to `arcadia-data-hub-raw-staging`) | +| `S3_PROCESSED_BUCKET` | No | S3 bucket the Lambda writes processed artifacts to. Only read to build the MCP Apps content security policy — download URLs come from `files.s3_bucket`. Leave it unset and the run report View cannot load processed images, video, or CSVs. See [MCP Apps](mcp-apps.md#content-security-policy) | | `LAMBDA_FUNCTION_URL` | No | Lambda Function URL. Required for file reprocessing and run-archive downloads. | | `STALLED_PROCESSING_AFTER_MINUTES` | No | Minutes a file may stay in `processing` before it is treated as stalled and becomes reprocessable (defaults to 20). Empty, zero, negative, or non-numeric values keep the default. | | `CRON_SECRET` | No | Shared secret for Vercel Cron jobs. The upload-queue sweep (`web/vercel.json`) rejects invocations without it | @@ -110,6 +111,8 @@ Other database commands: ## Running locally +A production `npm run build` in `web/` runs `mcp-apps:build` first so the MCP Apps run-report View is in the serverless bundle. See [MCP Apps](mcp-apps.md). `make dev` does not rebuild that View; re-run `npm run mcp-apps:build` after changing `web/mcp-apps/`. + ```sh # Start the web app dev server (Turbopack). make dev diff --git a/developer-docs/mcp-apps.md b/developer-docs/mcp-apps.md new file mode 100644 index 00000000..5ee48671 --- /dev/null +++ b/developer-docs/mcp-apps.md @@ -0,0 +1,186 @@ +# MCP Apps (run report View) + +Data Hub's MCP server serves the run report as an interactive HTML page, so a chat client that supports the MCP Apps extension renders the same report the website renders instead of rebuilding it from JSON. + +MCP Apps ([SEP-1865](https://modelcontextprotocol.io/specification/2026-01-26/apps)) is an extension to the Model Context Protocol. The server publishes an HTML page at a `ui://` address, the client fetches that page and runs it inside a locked-down iframe, and the page asks the client to make tool calls on its behalf. Throughout this page, "host" means the chat client doing that work, and "View" means the HTML page. + +The same React components render both surfaces. The web app feeds them over REST and the View feeds them over MCP tool calls. + +## What the server publishes + +- `ui://data-hub/run-report` is a resource whose body is one self-contained HTML file, served with the media type `text/html;profile=mcp-app`. +- `get_run_report` carries a pointer to that resource, so a host knows to render the page when the model calls the tool. +- `report_view_items` and `report_view_file_url` are two read-only tools the View calls to locate its own data. Both return short-lived S3 URLs; neither returns file contents. + +Adding those two took the tool count from 32 to 34. Two tests pin that number so the change shows up in review: `web/tests/mcp/mcp-catalog.test.ts` and `web/tests/integration/mcp.test.ts`. + +### Tool metadata + +`runReportToolUiMeta` in `web/lib/mcp/ui-apps.ts` writes the resource pointer twice: + +```ts +{ + ui: { resourceUri: "ui://data-hub/run-report", visibility: ["app"] }, + "ui/resourceUri": "ui://data-hub/run-report", +} +``` + +The nested `ui.resourceUri` is what the current spec asks for. The flat `ui/resourceUri` key is deprecated and is kept because some hosts still read it. `visibility` is `["model", "app"]` on `get_run_report` and `["app"]` on the two view tools. + +## How a host renders a run report + +1. The model calls `get_run_report` for an instrument and a run. +2. The host reads `_meta.ui.resourceUri` on that tool and fetches the resource at `ui://data-hub/run-report`. +3. The server returns the HTML plus a content security policy naming the S3 origins the page may load bytes from. +4. The host renders the HTML in a sandboxed iframe on an origin separate from its own. +5. The page sends `ui/initialize` to the host over `window.postMessage`, and the host replies with the current theme, display mode, container size, and its own capabilities. +6. The page sends `ui/notifications/initialized`, and the host pushes the `get_run_report` result into the page. +7. The page reads the instrument type, instrument ID, and run ID out of that result and renders the matching report. +8. Every later data fetch is a `tools/call` that the page asks the host to forward to the server. + +Steps 5 and 6 happen inside `useApp` from `@modelcontextprotocol/ext-apps`. `web/mcp-apps/run-report/app.tsx` only supplies the callbacks (`ontoolresult`, `ontoolcancelled`, `onerror`). + +## How the View fetches data + +No file bytes pass through the server. Every tool returns a presigned S3 URL and the View reads the object itself, the same way the web app has always read them through the `/api/v1/files/:id/download` redirect. The server only ever runs the permission check and signs. + +### Report items — images, PDFs, spectra, and videos + +1. The View calls `report_view_items` with the instrument, run, item kind, and a window offset. +2. The server looks up the run by its instrument ID and run ID, then asks `getReportItemsPage` for one window of matching files. +3. The server signs a download URL for every file in that window and returns those URLs alongside the file IDs and filenames. +4. The View stores each URL in an in-memory cache keyed by file ID. +5. The carousel points an ``, `