# RFC: Integration and End-to-End Testing Framework for graph-explorer

[mjuarros]

Abstract / Problem Statement

graph-explorer currently validates behavior exclusively through Vitest unit tests (docs/agents/testing.md), which cover hooks, atoms, and query-template generation in isolation. There is no layer that exercises the composed application routing, Jotai state wiring, React Query cache behavior, and the DOM as a user actually experiences it.

This gap has concrete cost. The schema-sync empty-state flash investigated in #1556/#1557 was a timing defect in SchemaDiscoveryBoundary that unit tests could not have caught: the bug only manifests as a sequence of renders across a real fetch lifecycle, not as a single hook's output. A component-level or DOM-level integration test that asserts "the empty state is never painted while isFetching is true" would have caught the regression at PR time instead of requiring a multi-day trace investigation post-release.

The proposal: add a hermetic, deterministic integration/E2E layer using Playwright, with two tiers network-mocked integration tests (majority) and a small number of true end-to-end tests against an ephemeral, containerized Gremlin server for the highest-risk flows (schema discovery, connection setup, graph rendering).

Scope: packages/graph-explorer vs. packages/graph-explorer-proxy-server

This Playwright layer covers packages/graph-explorer it drives a real browser against the built frontend. packages/graph-explorer-proxy-server is not a gap this RFC needs to fill with a second browser-test framework: app.test.ts already runs a genuine request-level integration suite against the real createApp() Express instance via supertest routing for all five endpoints, CORS, IAM SigV4 signing, header validation, base-path preservation, allowed-origin enforcement, and fetch error handling. That is already integration testing in every sense that matters for that package; it just runs over in-process HTTP via Vitest instead of a browser, and doesn't need to be redone in Playwright.

What neither existing suite proves is the full real topology: browser → proxy-server (routing, IAM signing) → a real Gremlin/SPARQL server → back. The Tier 2 smoke tier is scoped to close exactly that gap see the webServer command below, which boots the proxy server rather than a bare frontend preview.

Architectural Goals

Why Playwright

• Zero infrastructure overhead. Playwright ships its own browser binaries and a test runner; nothing beyond pnpm install and a one-time playwright install --with-deps is required. No Selenium grid, no separate driver management.
• Deterministic by construction. Auto-waiting on actionability (visible, enabled, stable) eliminates the arbitrary sleep/waitFor patterns that make Cypress/Puppeteer suites flaky. This directly targets the failure class in Empty state message flashes briefly during schema sync in schema view #1557 asserting on transient render states requires a tool that can assert "never happened during this window," which Playwright's expect(locator).not.toBeVisible() combined with its trace viewer supports natively.
• First-class network interception. page.route() gives full control over Gremlin/openCypher/SPARQL response timing without a real database for most tests this is what lets integration tests stay hermetic and fast.
• Already the incumbent pattern in AWS frontend OSS. AWS Cloudscape Design System and AWS Amplify UI both use Playwright for their integration/E2E suites. Adopting it here keeps graph-explorer aligned with the tooling contributors moving between AWS-adjacent frontend repos already know, rather than introducing Cypress as a second, unrelated test runner alongside Vitest.
• Single-vendor test story. Vitest (unit) and Playwright (integration/E2E) are both maintained with active TypeScript-first APIs and share enough conceptual surface (expect, describe/test) that contributors don't context-switch mental models, only DOM-vs-hook scope.

Alternatives considered and rejected:

Tool	Why not
Cypress	Weaker multi-tab/multi-origin support, no first-class network condition simulation for slow/flaky fetch sequencing, heavier CI runtime
Puppeteer + Testing Library	No batteries-included test runner, retry/wait logic, or trace viewer would require rebuilding what Playwright ships
WebdriverIO	Selenium-protocol overhead conflicts with the "zero infrastructure" goal

Why this pattern (mocked integration + minimal real-backend E2E)

Best-in-class practices converges on: most confidence should come from tests that render the real component tree with mocked I/O boundaries, not from either isolated unit tests or a large suite of slow, real-backend E2E tests.

• Tier 1 Integration tests (majority): Full app or feature-shell rendered in a real browser, network calls intercepted via page.route() returning canned Gremlin/openCypher/SPARQL responses (reusing the existing graphsonHelpers.ts/ocHelpers.ts/sparqlHelpers.ts response builders from @/utils/testing no new fixture format). Fast, deterministic, no external dependency. This tier directly targets timing/sequencing bugs like Empty state message flashes briefly during schema sync in schema view #1557 by controlling response latency per-request.
• Tier 2 E2E smoke tests (minimal, high-value): A handful of tests against real, ephemeral database backends, driving the browser through the actual proxy server rather than a bare frontend build validating the full topology (browser → proxy-server routing/IAM-signing → database) and the real serialization/wire-format contract end to end. graph-explorer already ships exactly the hermetic fixture needed for the Gremlin case: samples/air_routes/docker-compose.yaml's app service is the published public.ecr.aws/neptune/graph-explorer image, which runs the proxy server (USING_PROXY_SERVER=true) serving the UI against the database service (tinkerpop/gremlin-server:3.8, pre-seeded with the deterministic air-routes dataset) no AWS credentials, no Neptune cluster, no network dependency on any AWS account. This satisfies the "graph database may be needed" constraint without inventing new infrastructure, and it's the one place this RFC exercises graph-explorer-proxy-server's code path rather than relying solely on its existing supertest suite.

This two-tier split is why the constraint about needing a graph database resolves cleanly: only Tier 2 needs one, it's disposable per-run (docker compose up --wait / down -v), and every backend it uses is a public OSS image rather than an AWS-managed service.

Real-backend coverage across supported database flavors

docs/agents/product.md lists the databases graph-explorer targets: Amazon Neptune, Amazon Neptune Analytics, Apache TinkerPop Gremlin Server, and JanusGraph for property graphs, plus SPARQL 1.1 for RDF. Not all of those can be part of a hermetic Tier 2 the split:

• In scope (OSS, runs locally, no AWS account):
  • Apache TinkerPop Gremlin Server already covered above.
  • Blazegraph and Apache Jena Fuseki (a generic SPARQL 1.1 store), run as a paired smoke test against the same spec. This directly targets a documented, real hazard: docs/agents/connectors.md states the query builder must never emit Blazegraph-only hint: triples (e.g. hint:joinOrder), because they silently return zero rows on other SPARQL 1.1 endpoints Graph Explorer supports. A single SPARQL backend can't prove that invariant only running the identical query against both a Blazegraph-flavored endpoint and a vanilla SPARQL 1.1 endpoint can. Exact image/tag selection (e.g. a maintained Blazegraph and Fuseki image) needs to be confirmed during the Phase 2 prototype spike below, not asserted here.
• Named future work, not committed in this RFC:
  • JanusGraph. It speaks Gremlin and has a public image, but there's no documented, specific regression class distinct from the TinkerPop reference server that justifies a third real-backend container today. Add it if and when a JanusGraph-specific bug is actually found not preemptively.
  • openCypher against a real backend. Neptune's openCypher support is a proprietary Neptune extension; no non-Neptune OSS server is known to implement it. A hermetic Tier 2 openCypher smoke test isn't currently achievable without an AWS account, which would break this RFC's "zero infrastructure" goal. openCypher stays covered by Tier 1 (mocked) only, and this is a stated limitation, not an oversight.
• Out of scope, by design: Amazon Neptune and Neptune Analytics themselves are managed AWS services with no local hermetic equivalent. Their proxy-specific behavior the service-type header default that distinguishes them, and IAM SigV4 signing is already covered by graph-explorer-proxy-server's existing supertest suite against mocked upstream responses. That is the correct layer for Neptune-specific behavior; Tier 2 should not try to re-prove it against a real Neptune cluster.

Proposed Directory Layout

Split across two locations not because of package politics, but because of a real tsconfig boundary: packages/graph-explorer/tsconfig.json defines the @/* path alias ("@/*": ["./src/*"]) scoped to that package only. Tier 1 needs that alias (it reuses @/utils/testing's response builders) and never touches the proxy server, so it belongs inside the package whose alias it depends on. Tier 2 needs neither alias it drives the composed app container black-box, over HTTP and is the one piece that genuinely crosses package boundaries, so it stays at the root alongside samples/ and .github/workflows/, which are cross-package for the same reason.

graph-explorer/
├── packages/graph-explorer/
│   └── e2e/                            # Tier 1 mocked, package-scoped
│       ├── playwright.config.ts
│       ├── fixtures/
│       │   └── mockGraphExplorer.ts    # boots app against page.route() mocks
│       ├── mocks/
│       │   ├── gremlinResponses.ts     # thin adapters over @/utils/testing/graphsonHelpers
│       │   └── sparqlResponses.ts      # thin adapters over @/utils/testing/sparqlHelpers
│       └── integration/
│           ├── schema-discovery/
│           │   ├── schema-sync-no-flash.spec.ts     # regression test for #1557
│           │   └── schema-discovery-error.spec.ts
│           ├── connection-setup/
│           │   └── create-connection.spec.ts
│           └── graph-view/
│               └── expand-node.spec.ts
├── e2e/                                # Tier 2 real backends, cross-package, root-scoped
│   ├── playwright.config.ts
│   ├── playwright.shared.config.ts     # use/reporter settings shared with Tier 1's config
│   ├── fixtures/
│   │   └── rdf-backends/
│   │       └── docker-compose.yaml     # Blazegraph + Fuseki, test-only not a user-facing sample
│   └── smoke/                          # gated by connector path, see e2e.yml
│       ├── gremlin/
│       │   └── connect-and-explore.spec.ts
│       └── sparql/
│           └── blazegraph-hint-not-leaked.spec.ts  # same spec run against both backends
└── .github/workflows/
    └── e2e.yml

integration/ and smoke/ remain separate Playwright projects/configs (see below) so CI, and any contributor running locally, can select tier independently and, per the placement above, so integration never needs Docker to run at all.

Reference Implementation

e2e/playwright.shared.config.ts settings common to both tiers

import type { PlaywrightTestConfig } from "@playwright/test";

export const sharedConfig: PlaywrightTestConfig = {
  forbidOnly: !!process.env.CI,
  reporter: process.env.CI
    ? [["github"], ["html", { open: "never" }]]
    : [["list"]],
  use: {
    trace: "retain-on-failure",
    screenshot: "only-on-failure",
    video: "retain-on-failure",
  },
};

packages/graph-explorer/e2e/playwright.config.ts Tier 1 (mocked, no Docker required)

import { defineConfig, devices } from "@playwright/test";
import { sharedConfig } from "../../../e2e/playwright.shared.config";

const PORT = 4173;

export default defineConfig({
  ...sharedConfig,
  testDir: "./integration",
  fullyParallel: true,
  // No --max-failures here, deliberately: these tests are cheap (no real
  // network, no container boot), so a full run gives complete failure
  // visibility across every spec in one CI pass instead of one-fix-per-push.
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 4 : undefined,
  use: {
    ...sharedConfig.use,
    baseURL: `http://localhost:${PORT}`,
  },
  webServer: {
    command: "pnpm preview --port " + PORT,
    url: `http://localhost:${PORT}`,
    reuseExistingServer: !process.env.CI,
    timeout: 60_000,
  },
});

e2e/playwright.config.ts Tier 2 (real backends, Docker required)

import { defineConfig, devices } from "@playwright/test";
import { sharedConfig } from "./playwright.shared.config";

export default defineConfig({
  ...sharedConfig,
  fullyParallel: true,
  // No shared `webServer` block: each project below points at a container
  // the CI step (or the contributor, locally) already started via
  // `docker compose`. Playwright never manages these servers' lifecycle.
  projects: [
    {
      // Run locally with:
      //   docker compose -f samples/air_routes/docker-compose.yaml up app database --wait
      name: "smoke-gremlin",
      testDir: "./smoke/gremlin",
      use: { ...devices["Desktop Chrome"], baseURL: "http://localhost:8080" },
    },
    {
      // Same spec file, run against the app pointed at Blazegraph.
      // Run locally with:
      //   docker compose -f e2e/fixtures/rdf-backends/docker-compose.yaml up --wait
      name: "smoke-sparql-blazegraph",
      testDir: "./smoke/sparql",
      use: { ...devices["Desktop Chrome"], baseURL: "http://localhost:8081" },
    },
    {
      // Same spec file again, run against the app pointed at Fuseki this
      // pairing is what proves Blazegraph-only syntax doesn't leak.
      name: "smoke-sparql-fuseki",
      testDir: "./smoke/sparql",
      use: { ...devices["Desktop Chrome"], baseURL: "http://localhost:8082" },
    },
  ],
});

Network-mocked integration test regression coverage for #1557

// packages/graph-explorer/e2e/integration/schema-discovery/schema-sync-no-flash.spec.ts
import { test, expect } from "@playwright/test";
import { mockSchemaSyncRequest } from "../../mocks/gremlinResponses";
import { openSchemaExplorerWithConnection } from "../../fixtures/mockGraphExplorer";

test("schema explorer never paints the empty-state while sync is in flight", async ({
  page,
}) => {
  // Force a slow, controllable fetch so the intermediate render is observable.
  const schemaRequest = mockSchemaSyncRequest(page, {
    delayMs: 400,
    vertexLabels: ["airport", "country", "continent"],
  });

  await openSchemaExplorerWithConnection(page);
  await page.getByRole("button", { name: "Refresh schema" }).click();

  const emptyState = page.getByText("No Object Properties Available");
  const syncingIndicator = page.getByText("Synchronizing schema");

  // The empty state must never be visible while data is fetching, at any
  // point in the render sequence not just at the start and the end.
  for (let i = 0; i < 5; i++) {
    await expect(emptyState).not.toBeVisible();
    await page.waitForTimeout(50);
  }

  await schemaRequest.resolve();
  await expect(syncingIndicator).not.toBeVisible();
  await expect(page.getByText("airport")).toBeVisible();
});

// packages/graph-explorer/e2e/mocks/gremlinResponses.ts
import type { Page } from "@playwright/test";
import { createGraphSONVertexLabelsResponse } from "@/utils/testing/graphsonHelpers";

export function mockSchemaSyncRequest(
  page: Page,
  opts: { delayMs: number; vertexLabels: string[] },
) {
  let resolveFn: () => void;
  const gate = new Promise<void>((resolve) => (resolveFn = resolve));

  page.route("**/gremlin", async (route) => {
    await gate;
    await new Promise((r) => setTimeout(r, opts.delayMs));
    await route.fulfill({
      status: 200,
      contentType: "application/json",
      body: JSON.stringify(
        createGraphSONVertexLabelsResponse(opts.vertexLabels),
      ),
    });
  });

  return { resolve: () => resolveFn() };
}

Note the deliberate reuse of createGraphSONVertexLabelsResponse from the existing @/utils/testing/graphsonHelpers the same response builders the unit suite already uses. No parallel fixture format is introduced.

Tier 2 fixture e2e/fixtures/rdf-backends/docker-compose.yaml

services:
  blazegraph:
    image: lyrasis/blazegraph:2.1.5 # confirm image/tag during the Phase 2 spike
    ports:
      - "9999:9999"

  app-blazegraph:
    image: public.ecr.aws/neptune/graph-explorer:latest
    ports:
      - "8081:80"
    environment:
      - GRAPH_TYPE=sparql
      - USING_PROXY_SERVER=true
      - GRAPH_CONNECTION_URL=http://blazegraph:9999/blazegraph/namespace/kb/sparql

  fuseki:
    image: stain/jena-fuseki # confirm image/tag during the Phase 2 spike
    environment:
      - ADMIN_PASSWORD=admin
    ports:
      - "3030:3030"

  app-fuseki:
    image: public.ecr.aws/neptune/graph-explorer:latest
    ports:
      - "8082:80"
    environment:
      - GRAPH_TYPE=sparql
      - USING_PROXY_SERVER=true
      - GRAPH_CONNECTION_URL=http://fuseki:3030/ds/sparql

Same app image the existing samples/air_routes/docker-compose.yaml uses, pointed at a different backend per instance no new deployment shape, just a second connection target. Both RDF stores need a fixed seed dataset loaded on container start (a handful of triples, matching the deterministic-dataset pattern air-routes already follows); the loading mechanism (init script vs. pre-baked image) is a detail to settle during the Phase 2 prototype, not this RFC.

Tier 2 smoke test against the real proxy server + Gremlin backend

// e2e/smoke/gremlin/connect-and-explore.spec.ts
import { test, expect } from "@playwright/test";

test("connects through the proxy to a live Gremlin database and renders the graph", async ({
  page,
}) => {
  // baseURL (see playwright.config.ts) points at the docker-compose `app`
  // service the real proxy server, not a bare frontend build so this
  // request exercises proxy-server routing and IAM-signing code, not just
  // the frontend.
  await page.goto("/");
  await page.getByRole("button", { name: "Add connection" }).click();
  await page.getByLabel("Graph type").selectOption("gremlin");
  await page.getByLabel("Connection URL").fill("http://localhost:8182");
  await page.getByRole("button", { name: "Connect" }).click();

  await expect(page.getByText("air-routes")).toBeVisible({ timeout: 15_000 });
  await page.getByRole("button", { name: "Sync schema" }).click();
  await expect(page.getByText("airport")).toBeVisible({ timeout: 15_000 });
});

CI/CD & Automation Integration Plan

New workflow: .github/workflows/e2e.yml

name: E2E and Integration Tests

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "0 6 * * *" # nightly backstop full smoke run regardless of changed paths
  workflow_dispatch: {}

jobs:
  integration:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm --filter graph-explorer exec playwright install --with-deps chromium
      # Config lives inside the package (see "Proposed Directory Layout") —
      # no --config flag needed, cwd is already packages/graph-explorer.
      # Deliberately no --max-failures: full-suite visibility beats saving
      # a few seconds on a tier this cheap.
      - run: pnpm --filter graph-explorer exec playwright test --project=integration
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-integration-report
          path: packages/graph-explorer/e2e/playwright-report/
          retention-days: 7

  changes:
    # Path filters on `on:` gate the whole workflow, not one job, so
    # detecting "which connector did this PR touch" needs its own step.
    # Split by connector so a SPARQL-only PR never pays for a Gremlin
    # container boot, and vice versa.
    runs-on: ubuntu-latest
    outputs:
      gremlin-changed: ${{ steps.filter.outputs.gremlin }}
      sparql-changed: ${{ steps.filter.outputs.sparql }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            gremlin:
              - 'packages/graph-explorer/src/connector/gremlin/**'
              - 'packages/graph-explorer/src/connector/openCypher/**'
            sparql:
              - 'packages/graph-explorer/src/connector/sparql/**'

  smoke-gremlin:
    needs: changes
    # Real-backend tier runs on the PR path, but only when the change
    # actually touches Gremlin/openCypher query-serialization code the
    # class of defect this tier exists to catch. Every other PR skips it.
    # The nightly/manual run is a backstop for anything the path filter
    # misses (e.g. a Playwright fixture change with no connector diff).
    if: needs.changes.outputs.gremlin-changed == 'true' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      # @playwright/test is a root devDependency for Tier 2 the config
      # lives at repo root, not inside any package (see directory layout).
      - run: pnpm exec playwright install --with-deps chromium
      - name: Start ephemeral proxy server + Gremlin backend
        run: docker compose -f samples/air_routes/docker-compose.yaml up app database --wait
      # --max-failures=1: unlike `integration`, each spec here pays real
      # container-boot and network cost. Once one real-backend assertion
      # fails, the environment is presumptively suspect running the rest
      # buys little and burns runner minutes. Bail, fix, re-push.
      - run: pnpm exec playwright test --config e2e/playwright.config.ts --project=smoke-gremlin --max-failures=1
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-smoke-gremlin-report
          path: e2e/playwright-report/
          retention-days: 7
      - if: always()
        run: docker compose -f samples/air_routes/docker-compose.yaml down -v

  smoke-sparql:
    needs: changes
    # Same rationale as smoke-gremlin, gated on the SPARQL connector path
    # instead. Runs the identical spec against Blazegraph and Fuseki to
    # prove Blazegraph-only syntax never leaks to a vanilla SPARQL 1.1
    # endpoint (docs/agents/connectors.md, the `hint:` exclusion rule).
    if: needs.changes.outputs.sparql-changed == 'true' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm exec playwright install --with-deps chromium
      - name: Start ephemeral Blazegraph and Fuseki backends
        run: docker compose -f e2e/fixtures/rdf-backends/docker-compose.yaml up --wait
      # Both projects run in this one invocation, sharing Playwright's worker
      # pool genuinely concurrent, not sequential. --max-failures=1 stops
      # both on the first failure: since they run the identical spec, a
      # failure is either a shared assertion bug (no value in continuing) or
      # a single-backend regression (already pinpointed by that backend's
      # trace/video artifact; the other project's pass/fail is corroborating
      # signal, not the primary one).
      - run: pnpm exec playwright test --config e2e/playwright.config.ts --project=smoke-sparql-blazegraph --project=smoke-sparql-fuseki --max-failures=1
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-smoke-sparql-report
          path: e2e/playwright-report/
          retention-days: 7
      - if: always()
        run: docker compose -f e2e/fixtures/rdf-backends/docker-compose.yaml down -v

Design choices matching AWS OSS CI conventions already present in this repo (unit.yml, test_build_docker.yml):

• Four independent jobs, not one a flaky container pull in either smoke job never blocks the fast, hermetic integration job from reporting, and a Gremlin-only or SPARQL-only PR never pays for the other backend's container.
• docker compose ... down -v in an always() step guarantees no dangling container state between runs on shared runners.
• Trace/video/screenshot artifacts upload only on: failure(), keeping the common case cheap and giving full observability (Playwright trace viewer) exactly when it's needed this is the "high observability on failure" requirement.
• No AWS credentials, no Neptune endpoint, no account dependency anywhere in any job. Every external dependency is a public OSS image (tinkerpop/gremlin-server, Blazegraph, Fuseki) or mocked in-process.

Execution strategy: does one failure stop everything, and what runs in parallel?

Short answer: no single failure stops everything, at two levels, by design and each tier gets a different fail-fast policy on purpose.

Job-level (does Gremlin failing block SPARQL, or vice versa?). integration, smoke-gremlin, and smoke-sparql are three independent GitHub Actions jobs. smoke-gremlin and smoke-sparql both depend only on changes, not on each other, so GitHub Actions runs them concurrently on separate runners whenever both are triggered. A Gremlin regression failing smoke-gremlin has no effect on smoke-sparql's outcome they are separate failure domains, matching how a Gremlin-only PR shouldn't pay for the SPARQL container and vice versa (see the path-filter split above).

Test-level within a job (does one spec failing stop the rest?). This is where the tiers deliberately diverge:

	integration (Tier 1)	smoke-gremlin / smoke-sparql-* (Tier 2)
Cost per spec	Cheap mocked network, no container boot	Expensive real container boot, real network round-trips
--max-failures	Not set runs to completion	1 stops on first failure
Why	Full failure visibility is worth more than the few seconds saved; a PR author should see every failing assertion in one CI pass, not fix one and wait for the next run to surface the next	Once a real-backend assertion fails, the environment is presumptively suspect; running the remaining specs against it buys little and burns runner minutes for low marginal information

Parallelism within smoke-sparql. Its single playwright test --project=smoke-sparql-blazegraph --project=smoke-sparql-fuseki invocation runs both projects concurrently across Playwright's shared worker pool on that one runner not sequentially so Blazegraph and Fuseki are exercised in parallel, at the cost of one shared --max-failures=1 across both (reasonable here specifically because both projects run the identical spec: a failure is either a shared assertion bug, where continuing adds nothing, or a single-backend regression, already pinpointed by that backend's own trace/video artifact).

Keeping it easy to run without it becoming slow. Three levers, already reflected above:

• Path-gating means most PRs never run either smoke tier at all the majority-case cost stays at integration's runtime alone.
• --max-failures=1 on Tier 2 bounds worst-case runner time to roughly one container boot plus one failing spec, not the full suite.
• Locally, a contributor never needs Docker to be productive: pnpm --filter graph-explorer exec playwright test --project=integration (Tier 1, package-scoped, no external command) is the default "run the e2e tests" experience. Running a smoke tier is an explicit, documented opt-in the docker compose ... up --wait command called out in each project's comment above never something pnpm test triggers by default.

Decision: when do the smoke-* tiers run?

Two options were weighed:

	Nightly / scheduled only	Path-scoped, on the pull request (selected)
Detection latency	A real wire-format regression can sit on main for up to 24h before anyone notices	Caught at the exact commit that introduced it
Fix ownership	The contributor who introduced it is out of the loop by the time it surfaces; a maintainer has to trace it back and file a revert/fix	Stays the contributor's own responsibility, as part of getting their PR merged
Infra-caused merge blocking	None	Bounded to PRs that touch the relevant connector path not every PR, only the ones where that specific real-backend check is relevant

Nightly-only was the initial default in this proposal, on the reasoning that it kept the non-hermetic dependencies (container pulls, server boots) off the required-check path entirely. On review, that reasoning underweighted a more important cost: the defect class these tiers exist to catch connector/query-serialization regressions is exactly the kind of bug a contributor's own PR is most likely to introduce, and deferring detection to nightly moves both discovery and the fix off the person who caused it. Path-scoped per-PR execution is the better default, and splitting the filter by connector (Gremlin/openCypher vs. SPARQL see the changes job) bounds each tier's infra-flake exposure to only the PRs where that specific check is relevant. The nightly/workflow_dispatch run stays as a backstop for anything either path filter misses.

Rollout sequencing

1. Land the Playwright scaffold and the integration project only, as a non-required check, alongside the single Empty state message flashes briefly during schema sync in schema view #1557 regression test this is the smallest slice that proves the tool choice and directly closes the gap that motivated this RFC.
2. Add 3-5 more integration specs for the highest-traffic flows (connection setup, node expansion, filtering) over subsequent PRs, each independently reviewable.
3. Promote integration to a required check once flake rate is proven at zero across ~2 weeks of PR traffic.
4. Add smoke-gremlin next, once integration is stable it carries the only real infrastructure dependency in that path and should not block the majority-case win. Gate it to PRs touching the Gremlin/openCypher connector paths (see "Decision" above), with a nightly/workflow_dispatch run as backstop.
5. Add smoke-sparql-blazegraph/smoke-sparql-fuseki after smoke-gremlin is proven stable same gating pattern, scoped to the SPARQL connector path. This is the tier that proves the documented Blazegraph-hint:-leak invariant against real servers; land it deliberately after Gremlin, not alongside it, so any new infra flakiness is diagnosed against one backend family at a time.
6. Once integration specs pass roughly five files, extract a page-object layer (e2e/pages/*.page.ts) per feature area to remove locator duplication across specs. Not worth introducing at prototype size one spec has nothing to deduplicate but plan for it before the suite grows past that point.
7. JanusGraph and a real-backend openCypher check remain named future work (see "Real-backend coverage across supported database flavors" above) revisit only if a concrete regression class specific to either is actually found.

Prototype step

Before committing to this RFC as written, land step 1 above as a standalone draft PR: the Playwright scaffold plus the single #1557 regression spec, run against CI as a non-required check for one week. This validates in practice that the mocked-network pattern reproduces the actual timing defect (not just a plausible-looking assertion), that CI runtime stays acceptable, and that the artifact-on-failure flow is actually useful for triage before any further specs or either smoke tier is built on top of it.

The prototype is done when, and only when:

• It runs on a contributor's machine with no setup beyond pnpm install and one playwright install.
• It passes reliably on GitHub-hosted runners, with no flakes across at least 10 consecutive CI runs.
• No external infrastructure (AWS account, Neptune, live database) is required to run it.
• A forced-failing run produces a usable trace/video/screenshot artifact without local reproduction.
• Total runtime for the integration project stays under a few minutes.

---

Important

If you are interested in working on this issue, please leave a comment.

Tip

Please use a 👍 reaction to provide a +1/vote. This helps the community and maintainers prioritize this request.

[mykola-pereyma]

Reviewed this closely and pressure-tested the reference implementation against main. The strategy is sound — the two-tier hermetic-mock-majority + minimal real-backend split, the per-connector path gating, the tier-specific --max-failures policy, always() teardown, and artifact-on-failure are all well judged, and reusing @/utils/testing response builders instead of a parallel fixture format is the right call. The motivation checks out too (#1557 is real and open; samples/air_routes/docker-compose.yaml already ships the hermetic Gremlin fixture as described).

A few items are worth resolving before this becomes the template contributors copy. Since this is an issue rather than a PR, I've anchored each to the specific snippet inline.

---

1. The graph view renders on a <canvas>, so text locators can't assert graph content.

await expect(page.getByText("air-routes")).toBeVisible({ timeout: 15_000 });
await page.getByRole("button", { name: "Sync schema" }).click();
await expect(page.getByText("airport")).toBeVisible({ timeout: 15_000 });

packages/graph-explorer renders graphs with Cytoscape.js (cytoscape + cytoscape-canvas, and no cytoscape-node-html-label or equivalent), so node/edge labels are painted on a canvas and are invisible to getByText/getByRole. "Graph rendering" is named as one of the three highest-risk flows, but there's no viable assertion path for it as written. Please specify a canvas strategy — e.g. introspect the Cytoscape instance via page.evaluate(() => cy.nodes().length) / a test-only hook, or a visual snapshot — for expand-node.spec.ts and the "renders the graph" smoke assertion. (Sidebar/schema-panel text like the connection name or the schema label list is real DOM and will work; this is specifically about graph-canvas content.)

2. The #1557 regression assertion can pass even when the flash occurs.

for (let i = 0; i < 5; i++) {
  await expect(emptyState).not.toBeVisible();
  await page.waitForTimeout(50);
}

Playwright's web-first matchers auto-retry until the condition holds: .not.toBeVisible() waits for the element to become not-visible and passes as soon as it is — it does not fail on a transient visible state. Combined with 50 ms polling, a brief empty-state flash can be waited away (or fall between samples) and the test still passes, which is the opposite of "never painted while isFetching." Since #1557 is a render-sequence bug, a component-level test (Vitest + Testing Library with a controlled fetch, asserting the empty-state branch never renders while fetching) or a MutationObserver/trace that captures every intermediate state would prove the invariant far more reliably. The prototype exit criteria should require reproducing the RED state first.

3. Tier 1 as written won't boot the app in CI.

webServer: { command: "pnpm preview --port " + PORT, ... }

- run: pnpm --filter graph-explorer exec playwright test --project=integration

Two blockers: (a) packages/graph-explorer has no preview script (only dev, build, vite-build, typecheck, clean), and vite preview also requires a prior pnpm build that the integration job never runs — so the webServer never comes up; and (b) the Tier 1 config defines no projects array and no project named integration (only testDir: "./integration"), so --project=integration matches nothing and errors. Please add a build step and serve the built output (or run against dev), and either define an integration project or drop the --project flag. Minor convention nit while here: the repo pins the Node version via node-version-file: "package.json" (engines.node: ">=24.16.0") and pnpm/action-setup@v5.0.0; e2e.yml hardcodes node-version: 24 / action-setup@v4 — worth aligning.

4. The SPARQL hint:-leak invariant is detected by the vanilla endpoint; Blazegraph is a positive control.

blazegraph-hint-not-leaked.spec.ts — same spec run against both backends

A leaked hint: triple manifests as zero rows on a vanilla SPARQL 1.1 store (Fuseki) — that's the discriminating check. Blazegraph honors hint:, so its pass proves the query returns data (a useful positive control) but says nothing about leakage. Worth stating that explicitly so the pairing's roles are clear. Also, stain/jena-fuseki is effectively unmaintained — prefer the official apache/jena-fuseki — and the seed-data loading mechanism (init script vs. pre-baked image) is load-bearing for determinism and shouldn't be fully deferred.

5. "Zero flake" gate conflicts with retries: 1.

retries: process.env.CI ? 1 : 0,

Promoting to a required check on "flake rate proven at zero across ~2 weeks" is hard to reconcile with a CI retry, which masks exactly the nondeterminism the gate claims is absent. For a truly hermetic Tier 1, consider retries: 0 and treat any flake as a failure; otherwise state a realistic flake budget rather than zero.

6. Reproducibility: pin the Tier 2 images.

image: public.ecr.aws/neptune/graph-explorer:latest

A floating :latest (and unpinned backend tags) undercuts "deterministic by construction" for the real-backend tier — a base-image change can move results out from under a green PR. Pin by digest (or a fixed release tag) for the E2E images.

7. Sequencing: #1557 is still open, so the step-1 "regression test for #1557" would be red until the fix lands. Either fix #1557 in the same slice or land the test as an explicitly-expected failure so the scaffold PR can still go green.

None of this is a reason not to proceed — the direction and the prototype-first rollout are the right shape. These are about making the reference implementation actually runnable and making the flagship test prove what it claims before it's copied across the suite.