Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 14 additions & 9 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,30 @@
# RawTree API base URL. Optional: defaults to https://api.rawtree.com.
RAWTREE_API_URL=https://api.rawtree.com

# Database that holds the `events` and `recordings` tables.
# Database that holds the `events` and `recordings` tables. Required for the collector.
RAWTREE_DATABASE=web_analytics

# Insert-only key for event and recording ingestion (rtree key create --permission write_only).
# Tables are created on first insert. A key bound to database_roles with only
# `GRANT INSERT ON <database>.*` is tighter, but then create the tables first.
# Required for the collector in both modes. Tables are created on first insert. A key bound
# to database_roles with only `GRANT INSERT ON <database>.*` is tighter, but then create the
# tables first.
RAWTREE_INGEST_KEY=rt_...

# Read-only key for the dashboard and replay (rtree key create --permission read_only),
# or a key bound to database_roles with only `GRANT SELECT ON <database>.*`.
# Mode: with RAWTREE_QUERY_KEY and RAWTREE_INGEST_KEY both set, the dashboard needs no sign-in
# (env mode). Leave RAWTREE_QUERY_KEY empty for sign-in mode: visitors use their own RawTree
# API key or, when RAWTREE_CONNECTOR is set, Connect with RawTree through Vercel Connect.
# The collector keeps using RAWTREE_INGEST_KEY and RAWTREE_DATABASE in both modes.
RAWTREE_QUERY_KEY=rt_...

# Mode: with RAWTREE_QUERY_KEY and RAWTREE_INGEST_KEY set, the dashboard needs no sign-in
# (env mode). Leave both empty for sign-in mode: visitors use their own RawTree API key or,
# when RAWTREE_CONNECTOR is set, Connect with RawTree through Vercel Connect.
# Optional, sign-in mode only. Vercel Connect connector UID, e.g. rawtree/web-analytics
# (vercel connect create rawtree --target api --name web-analytics; run `vercel env pull` locally).
# (vercel connect create rawtree --target api --name web-analytics). Locally, also set
# VERCEL_OIDC_TOKEN: run `vercel env pull .env.vercel` and copy it (plain `vercel env pull`
# overwrites this file).
RAWTREE_CONNECTOR=

# Collector (app/api/collect)
# Collector (app/api/collect). Required: without it the collector answers 500.
# Comma-separated origins (scheme://host[:port]) of the sites that send events to /api/collect,
# or "*" to accept any origin. Origin checks reject unwanted browsers; they are not authentication.
# The dashboard's own Console doesn't need an entry here.
Expand All @@ -33,5 +37,6 @@ ANALYTICS_ALLOWED_ORIGINS=https://example.com
# Leave empty to disable server events. Generate with: openssl rand -hex 32
ANALYTICS_SERVER_TOKEN=

# Optional. Prefix for table names, used for disposable test tables (e.g. e2e_). Empty in production.
# Optional. Prefix for table names, for manual runs against disposable tables (e.g. e2e_).
# Empty in production. The automated tests never reach RawTree.
RAWTREE_TABLE_PREFIX=
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

## Test plan

- [ ] `npm run typecheck`, `npm test`, and `npm run build` pass
- [ ] `npm run typecheck`, `npm test`, and `npm run build` pass (see CONTRIBUTING.md)
- [ ] Verified locally with `npm run dev` (dashboard, recordings, or console, as relevant)
- [ ] New or changed dashboard queries count each event once (`uniqExact` or `LIMIT 1 BY`)

Expand Down
34 changes: 12 additions & 22 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# rawtree-web-analytics

Read `README.md` for what the project does and how it's set up, and `CONTRIBUTING.md` for the checks every change must pass.
Read `README.md` for what the project does and how it's set up. Follow `CONTRIBUTING.md`: its guidelines are the project's rules, and every change must pass its three checks (`npm run typecheck`, `npm test`, `npm run build`; install with `npm ci`).

## Scope

Expand All @@ -9,30 +9,20 @@ Keep it simple. The project is only: an SDK to send events, examples, and a dash
## Layout

- `packages/analytics/`: the publishable SDK (`@rawtree/analytics`). `src/protocol.ts` is the event contract; its types and validator are the source of truth for the SDK and the collector.
- `app/api/collect/route.ts` + `lib/collect.ts`, `lib/rawtree.ts`: the Next.js collector.
- `lib/queries.ts`: every dashboard SQL query (one documented template per widget; aliases are the fields `app/(dashboard)/page.tsx` reads). `lib/dashboard.ts` runs them; pages import only `lib/dashboard.ts`. Bot detection lives in `lib/crawlers.ts`, date ranges in `lib/range.ts`.
- `app/(dashboard)/console/` + `components/console/`: the test console page (`/console`). It sends through the workspace SDK to `app/api/console/collect` (same `handleCollect`, with the visitor's `access.ingest`); `app/api/console/server-event` runs the server SDK in-process and `app/api/console/stored` is the storage check (`access.query`).
- `app/api/collect/route.ts` + `lib/collect.ts`, `lib/rawtree.ts`: the Next.js collector. It always writes with the env ingest key. `lib/access.ts` decides env mode vs sign-in mode for the dashboard and console.
- `lib/queries.ts`: every dashboard SQL query (one documented template per widget; aliases are the fields the sections in `app/(dashboard)/_sections/` read). `lib/dashboard.ts` runs them; pages import only `lib/dashboard.ts`. Bot detection lives in `lib/crawlers.ts`, date ranges in `lib/range.ts`, branding in `lib/site.ts`.
- `app/(dashboard)/page.tsx` loads the data and renders one component per section from `app/(dashboard)/_sections/`. One component per file, no barrel files: `components/ui/` holds the generic primitives (`styles.ts` has `cn` and the shared class names), `components/dashboard/` the cards, charts, and layout the dashboard builds on.
- `app/(dashboard)/console/` + `components/console/`: the test console page (`/console`). It sends through the workspace SDK to `app/api/console/collect` (same `handleCollect`, with the visitor's `access.ingest`); `app/api/console/server-event` runs the server SDK in-process and `app/api/console/stored` is the storage check (`access.query`). Both accept same-origin requests only and cap bodies at 64 KB.
- `examples/recipes/`: tracking recipes (`RECIPES.md` + `.ts` modules), typechecked by `npm run typecheck`.
- `test/`: collector tests.
- `test/`: collector, access, dashboard query, and recording reassembly tests. Every test mocks `fetch`, so none reaches RawTree.

## Commands
## Agent notes

```sh
npm install
npm test # builds the SDK, runs SDK and collector tests
npm run typecheck
npm run build # SDK + Next.js
```

Node ≥ 24 runs `.ts` directly: use erasable syntax only (no enums, namespaces, or parameter properties) and `.ts` extensions in relative imports.

## Invariants

- The browser SDK must stay free of rrweb (`./recorder` is a separate entry) and the server entry free of browser globals. `packages/analytics/test/entries.test.ts` and `server.test.ts` check it (part of `npm test`).
- The collector acknowledges only after RawTree accepted every row. `/api/collect` takes credentials and table destinations only from configuration, never from requests. In sign-in mode the dashboard and `/api/console/*` use the visitor's key or Connect grant (`lib/access.ts`), but SQL and table names still come only from the server.
- Delivery is at least once: keep IDs stable across retries and deduplicate in every query (`uniqExact(event_id)` or `LIMIT 1 BY`).
- RawTree reads lag writes by about 0.1–0.4 s and replicas can disagree briefly. Poll in tests; never assume read-after-write.
- Write synthetic data to the real `events`/`recordings` tables only when the owner asks (since 2026-10-08 they hold example test data at the owner's request). For automated tests use temporary prefixed tables (`RAWTREE_TABLE_PREFIX`) and drop them afterwards. Ask the owner before any other write to shared RawTree resources.
- Never run SQL or table names that come from a request. Credentials come from configuration or `lib/access.ts`.
- Delivery is at least once: deduplicate every count (`uniqExact(event_id)` or `LIMIT 1 BY`).
- Node ≥ 24 runs `.ts` directly: erasable syntax only and `.ts` extensions in relative imports.
- RawTree reads lag writes by about 0.1–0.4 s and replicas can disagree briefly. Poll, and never assume read-after-write.
- Write synthetic data to the real `events`/`recordings` tables only when the owner asks (since 2026-10-08 they hold example test data at the owner's request). For live or automated runs against RawTree use temporary prefixed tables (`RAWTREE_TABLE_PREFIX`) and drop them afterwards. Ask the owner before any other write to shared RawTree resources.
- Secrets live only in `.env.local` (ignored). Keep `.env.example` in sync when configuration changes.

<!-- BEGIN:nextjs-agent-rules -->
Expand Down
17 changes: 9 additions & 8 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ npm run dev # http://localhost:3000
## Before opening a pull request

```sh
npm run typecheck
npm test
npm run typecheck # also typechecks the recipes in examples/recipes/
npm test # offline: RawTree is mocked
npm run build
```

Expand All @@ -28,16 +28,17 @@ CI runs the same three checks on every pull request. Every pull request also get

- **Keep pull requests small and focused**, with a test plan and screenshots for UI changes.
- **The event contract is `packages/analytics/src/protocol.ts`.** Keep it backward compatible: existing events must keep validating and storing the same way.
- **Dashboard SQL lives in `lib/queries.ts`.** Count each event once (`uniqExact(toString(event_id))` or `LIMIT 1 BY`), never `count()` over raw rows. Add new counting queries to `EVENT_QUERIES` so the dedup test covers them.
- **Never accept SQL or table names from the browser.** Credentials and destinations come from server configuration or the signed-in viewer's access (`lib/access.ts`).
- **Tests never write to real tables.** Use temporary tables with `RAWTREE_TABLE_PREFIX` and drop them afterwards.
- **The browser SDK stays free of rrweb** (`./recorder` is a separate entry) and the server entry free of browser globals. `packages/analytics/test/entries.test.ts` checks this.
- **Dashboard SQL lives in `lib/queries.ts`** and follows the [query rules](README.md#-ask-your-ai-agent): count each event once (`uniqExact(toString(event_id))` or `LIMIT 1 BY`), never `count()` over raw rows. Add new counting queries to `EVENT_QUERIES` so the dedup test covers them. The README's [add a widget](README.md#-customize-it) steps cover the rest.
- **The collector acknowledges only after RawTree accepted every row.** Keep event and chunk IDs stable across retries.
- **Never accept SQL or table names from the browser.** Credentials and destinations come from server configuration or the signed-in visitor's access (`lib/access.ts`).
- **Tests stay offline.** Mock `fetch` instead of calling RawTree. For a manual run against real RawTree, set `RAWTREE_TABLE_PREFIX` (e.g. `e2e_`) to use disposable tables, and drop them afterwards.
- **RawTree reads lag writes** by a fraction of a second, and replicas can briefly disagree. Poll, and never assume read-after-write.
- **The browser SDK stays free of rrweb** (`./recorder` is a separate entry): `packages/analytics/test/entries.test.ts` checks it. **The server entry stays free of browser globals:** `server.test.ts` checks it.
- **Node runs the `.ts` sources directly:** erasable TypeScript only (no enums, namespaces, or parameter properties) and `.ts` extensions in relative imports.
- **Next.js 16 differs from older versions.** Check `node_modules/next/dist/docs/` before using its APIs.
- **Configuration changes** go in `.env.example` too. Secrets live only in `.env.local`.
- **No secrets, keys, or private event data** in code, issues, or pull requests.

[AGENTS.md](AGENTS.md) has the same rules in more detail, for people and coding agents.

## Releasing the SDK (maintainers)

`@rawtree/analytics` is published to npm by the [Release SDK](.github/workflows/release-sdk.yml) workflow, with provenance.
Expand Down
Loading
Loading