Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions developer-docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ see the root [README](../README.md#documentation) for that split.
- [Getting started](getting-started.md) — development setup, environment variables, running locally
- [Local development](local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed)
- [Architecture](architecture.md) — system overview, data flow, and design decisions
- [MCP Apps](mcp-apps.md) — run-report View: host render flow, the three app-only tools, the shared data source, `mcp-apps:build`, and the gitignored HTML artifact
- [Testing](testing.md) — per-package test frameworks, the shared test-server fixture, S3 mocking
- [Watcher](watcher.md) — CLI commands, configuration, run detection, upload modes
- [Lambda](lambda.md) — processing pipeline, supported instruments, adding new instruments
Expand Down
1 change: 1 addition & 0 deletions developer-docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,5 +60,6 @@ Steps 1–3 are the same, but the watcher does not upload immediately. Instead:
- **Public page metadata, gated bodies.** Routes commonly shared into Slack/Notion (dashboard, instruments, instrument and run detail, settings) are reachable without a session so link unfurlers can read `<head>` metadata; the page or layout renders a `SignInRequired` CTA in place of the real body when there's no session. `/watchers/*` stays redirected to `/login` by `web/proxy.ts`. Three independent layers prevent search indexing: a `robots` field on the root layout, an `app/robots.ts` that disallows generic crawlers from the product (with `/docs`, the homepage, and icon files allowed so Google can show a favicon, plus an allow-list for Slack/Discord/LinkedIn unfurl bots), and an `X-Robots-Tag: noindex, nofollow` header on responses other than those icon files.
- **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so they stay consistent across Lambda and the watcher without duplicating code.
- **Wildcard PAT scope for the legacy backfill.** Migration `0022_pat_scopes` backfilled every pre-existing token with `["*"]` so deployed watchers and the Lambda kept working once scope enforcement shipped. `POST /api/v1/tokens` rejects `*` from API callers, so every token minted since carries explicit least-privilege scopes; see [Security and permissions](https://datahub.arcadiascience.com/docs/security#token-scopes) for the scope vocabulary.
- **MCP Apps run report.** `/mcp/v1` also serves a `ui://data-hub/run-report` HTML View (built by `npm run mcp-apps:build` into a gitignored `web/mcp-apps/dist/run-report.html`) plus three app-only tools the host uses to fetch items, CSV pages, and JSON artifacts. See [MCP Apps](mcp-apps.md).
- **MCP for AI access (OAuth).** Streamable-HTTP MCP at `/mcp/v1` exposes tools, resources, and prompts. Clients use Better Auth as the OAuth AS (issuer `{BETTER_AUTH_URL}/api/auth`); discovery via `/.well-known/oauth-protected-resource` (incl. `/mcp/v1`), `/.well-known/oauth-authorization-server`, and `/.well-known/openid-configuration`, with consent at `/consent` (shows registered redirect hosts from the DB). Coarse scopes: transport requires `read` (WWW-Authenticate still advertises `read write` so clients request both); mutating tools need `write`. Access tokens must be JWTs with `aud` = `{origin}/mcp/v1` (clients pass RFC 8707 `resource` at token exchange); verified offline via JWKS (~1h), so revoking consent does not kill them immediately. `MCP_ALLOW_PAT_AUTH=true` optionally accepts PATs outside production (hard-disabled on Vercel production and on self-hosted production with a non-loopback `BETTER_AUTH_URL`; PAT `write` requires the `*` wildcard). See [Local development](local-development.md#connecting-an-mcp-client) and the [MCP overview](https://datahub.arcadiascience.com/docs/mcp).
- **Integration tests against a real server.** The shared `testing.py` module spins up a real Next.js server backed by a Postgres database, so Lambda and watcher integration tests exercise the actual API surface.
2 changes: 1 addition & 1 deletion developer-docs/first-time-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ The AWS stack you just deployed exposes the outputs the web app needs to reach i
| `AWS_ROLE_ARN` | `WebAppRoleArn` stack output | Lets the web app generate presigned S3 URLs and SigV4-sign Lambda Function URL invocations via OIDC federation. |
| `LAMBDA_FUNCTION_URL` | `DataHubFunctionUrl` stack output | The Lambda Function URL for manual reprocessing and archive builds. |

The S3 bucket names default to `arcadia-data-hub-raw-<env>` and `arcadia-data-hub-archives-<env>`; override `S3_RAW_DATA_BUCKET` and `S3_ARCHIVES_BUCKET` in Vercel only if your stack uses different names. Redeploy the web app (or push a commit) so it picks up the new variables.
The S3 bucket names default to `arcadia-data-hub-raw-<env>` and `arcadia-data-hub-archives-<env>`; override `S3_RAW_DATA_BUCKET` and `S3_ARCHIVES_BUCKET` in Vercel only if your stack uses different names. Also set `S3_PROCESSED_BUCKET` to `arcadia-data-hub-processed-<env>`: presigned download URLs come from each file row's own bucket, so nothing else needs it, but the MCP Apps run report has to name every origin in its content security policy and processed artifacts are most of what it renders. Redeploy the web app (or push a commit) so it picks up the new variables.

> **Note:** Slack channel notifications are configured after deploy, not via an environment variable. A workspace admin pastes the incoming webhook URL under Settings > Notifications > Slack channel (stored in the `slack_channel_config` table). Personal Slack DMs use the optional `SLACK_BOT_TOKEN` / `SLACK_CLIENT_ID` / `SLACK_CLIENT_SECRET` variables instead — see [Environment variables](getting-started.md#environment-variables).

Expand Down
3 changes: 3 additions & 0 deletions developer-docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ vercel env pull
| `AWS_REGION` | No | AWS region for S3 presigned URLs and Lambda Function URL SigV4 signing (defaults to `us-west-1`) |
| `AWS_ROLE_ARN` | No | IAM role ARN for Vercel OIDC federation. Used to presign S3 URLs and SigV4-sign Lambda Function URL invocations (only needed on Vercel) |
| `S3_RAW_DATA_BUCKET` | No | S3 bucket for raw data uploads (defaults to `arcadia-data-hub-raw-staging`) |
| `S3_PROCESSED_BUCKET` | No | S3 bucket the Lambda writes processed artifacts to. Only read to build the MCP Apps content security policy — download URLs come from `files.s3_bucket`. Leave it unset and the run report View cannot load processed images, video, or CSVs. See [MCP Apps](mcp-apps.md#content-security-policy) |
| `LAMBDA_FUNCTION_URL` | No | Lambda Function URL. Required for file reprocessing and run-archive downloads. |
| `STALLED_PROCESSING_AFTER_MINUTES` | No | Minutes a file may stay in `processing` before it is treated as stalled and becomes reprocessable (defaults to 20). Empty, zero, negative, or non-numeric values keep the default. |
| `CRON_SECRET` | No | Shared secret for Vercel Cron jobs. The upload-queue sweep (`web/vercel.json`) rejects invocations without it |
Expand Down Expand Up @@ -110,6 +111,8 @@ Other database commands:

## Running locally

A production `npm run build` in `web/` runs `mcp-apps:build` first so the MCP Apps run-report View is in the serverless bundle. See [MCP Apps](mcp-apps.md). `make dev` does not rebuild that View; re-run `npm run mcp-apps:build` after changing `web/mcp-apps/`.

```sh
# Start the web app dev server (Turbopack).
make dev
Expand Down
Loading