Benchmark evidence and abstention on 100 source-derived questions

.github/workflows/ci.yml

@@ -18,6 +18,7 @@ jobs:
       - run: node tests/test_variant_comparison.cjs
       - run: node tests/test_evidence_tree.cjs
       - run: node tests/test_claim_evidence.cjs
+      - run: node tests/test_claim_benchmark.cjs
       - run: node tests/test_ask_evidence.cjs
       - run: node tests/test_cloudflare_ask.cjs
       - run: node tests/test_variant_question_evaluation.cjs

.github/workflows/claim-abstention-benchmark.yml

@@ -0,0 +1,105 @@
+name: Claim abstention benchmark
+
+on:
+  pull_request:
+    paths:
+      - 'scripts/benchmark_claim_abstention.cjs'
+      - 'tests/test_claim_benchmark.cjs'
+      - 'web/claim-evidence.js'
+      - '.github/workflows/claim-abstention-benchmark.yml'
+  push:
+    branches: [main]
+    paths:
+      - 'scripts/benchmark_claim_abstention.cjs'
+      - 'web/claim-evidence.js'
+      - '.github/workflows/claim-abstention-benchmark.yml'
+  workflow_dispatch:
+  workflow_run:
+    workflows: ['Refresh data']
+    types: [completed]
+    branches: [main]
+
+permissions:
+  contents: read
+  actions: read
+
+jobs:
+  benchmark:
+    if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success'
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+      - name: Download the published-snapshot refresh artifact
+        id: refresh
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          run_id=$(gh run list --repo "$GITHUB_REPOSITORY" --workflow refresh-data.yml \
+            --status success --limit 1 --json databaseId --jq '.[0].databaseId')
+          if [ -z "$run_id" ] || [ "$run_id" = "null" ]; then
+            echo '::error::No successful refresh with a search-index artifact exists.'
+            exit 1
+          fi
+          echo "run_id=$run_id" >> "$GITHUB_OUTPUT"
+      - uses: actions/download-artifact@v4
+        with:
+          name: search-index
+          path: _data-refresh
+          run-id: ${{ steps.refresh.outputs.run_id }}
+          github-token: ${{ github.token }}
+      - name: Run 100 source-consistency questions and three methods
+        run: |
+          node tests/test_claim_benchmark.cjs
+          node scripts/benchmark_claim_abstention.cjs \
+            --snapshot _data-refresh --questions 100 --out _claim-benchmark \
+            2>&1 | tee _benchmark-console.txt
+      - name: Label the result artifact for inspection
+        id: label
+        if: always()
+        run: |
+          node -e '
+            const fs = require("node:fs");
+            let label = "report";
+            if (fs.existsSync("_claim-benchmark/report.json")) {
+              const r = JSON.parse(fs.readFileSync("_claim-benchmark/report.json", "utf8"));
+              const m = r.metrics;
+              const u = m.claim_scoped.source_unanswerable_count;
+              const p = m.claim_scoped.source_answerable_count;
+              const falseCount = (method) => Math.round((m[method].false_answer_rate_on_unanswerable || 0) * u);
+              label = `report-${r.scope}-Q${r.question_count}-P${p}-U${u}-B${falseCount("context_blind")}-V${falseCount("variant_only")}-C${falseCount("claim_scoped")}-Y${m.claim_scoped.answered_count}`;
+            } else {
+              const log = fs.existsSync("_benchmark-console.txt") ? fs.readFileSync("_benchmark-console.txt", "utf8") : "Missing benchmark output";
+              label = "error-" + (log.match(/(?:Error|TypeError): ([^\n]+)/)?.[1] || "unknown").replace(/[^a-zA-Z0-9]+/g, "-").slice(0, 80);
+            }
+            fs.appendFileSync(process.env.GITHUB_OUTPUT, `label=${label}\n`);
+          '
+      - name: Summarize method and limitations
+        run: |
+          node -e '
+            const fs = require("node:fs");
+            const r = JSON.parse(fs.readFileSync("_claim-benchmark/report.json", "utf8"));
+            if (r.question_count !== 100) process.exit(1);
+            const pct = (x) => x == null ? "n/a" : (100 * x).toFixed(1) + "%";
+            const lines = ["## 100 source-derived claim questions", "",
+              "**Not independent biomedical gold or an LLM experiment.** Labels are derived from accepted CIViC fields in this same frozen snapshot; the result tests source consistency only.", "",
+              "Question scope: **" + r.scope + "**. " + r.scope_note, "",
+              "Source coverage: `" + JSON.stringify(r.source_coverage) + "`", "",
+              "Snapshot hashes: `" + JSON.stringify(r.snapshot_sha256) + "`", "",
+              "Question types: `" + JSON.stringify(r.categories) + "`; truncated: " + r.truncated_count, "",
+              "| Method | Answered | Answer rate on source-answerable | False answers on source-unanswerable | Error among answered |",
+              "| --- | ---: | ---: | ---: | ---: |"];
+            for (const [key, m] of Object.entries(r.metrics)) lines.push(`| ${key} | ${m.answered_count}/${m.evaluated_count} | ${pct(m.answer_rate_on_answerable)} | ${pct(m.false_answer_rate_on_unanswerable)} | ${pct(m.selective_error_among_answers)} |`);
+            fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, lines.join("\n") + "\n");
+          '
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: claim-abstention-benchmark-${{ steps.label.outputs.label }}
+          path: |
+            _claim-benchmark/report.json
+            _benchmark-console.txt
+          retention-days: 30

README.md

@@ -61,6 +61,8 @@ The [variant-conditioned retrieval experiment](docs/VARIANT_RETRIEVAL_EXPERIMENT
 
 The [variant claim check](docs/CLAIM_EVIDENCE.md) additionally filters those records by exact therapy and reported sensitivity or resistance. It keeps supporting, opposing, disputing and insufficient-context records separate in the browser, local API and MCP. This research-only rule set does not validate LLM prose or establish treatment efficacy.
 
+The [100-question CI benchmark](docs/CLAIM_BENCHMARK.md) runs three retrieval strategies on the same frozen Pages artifact and publishes a detailed report. A snapshot lacking variant-member exports is evaluated at **whole-profile scope only**, not as a single-variant study. Its automatically generated labels test source consistency, **not** independently established biomedical accuracy.
+
 **Explore by cancer name** groups disease entities currently in the public index using explicit matches on their names. Open a subtype to inspect its direct relations and, when available, accepted CIViC molecular-profile evidence. The groups are a browsing shortcut, not a complete ontology or a claim that all matching diseases are equivalent.
 
 CIViC records retain their molecular profile, named variants, HGVS expression when available, evidence type, level, trust rating, significance, review status, source record, and support/contradiction direction. A profile can contain multiple variants; its clinical evidence does not necessarily apply to each member. Filter to accepted level A/B records rated at least 4/5 by CIViC to focus on stronger study-level evidence; this does not guarantee a positive result or clinical actionability. OncoKB and COSMIC are outbound lookups only; their content is not imported or graded here, and COSMIC may require login.

docs/CLAIM_BENCHMARK.md

@@ -0,0 +1,84 @@
+# Controlled 100-question claim and abstention benchmark
+
+## Purpose and scope
+
+GitHub Actions runs `scripts/benchmark_claim_abstention.cjs` against the latest
+successful `Refresh data` **search-index artifact**, the same source used by
+Pages. This is a **source-consistency stress test** for the question "Do
+accepted CIViC records report sensitivity or resistance for gene–variant,
+disease and therapy?" It is **not** a test of clinical truth, independent
+expert adjudication, recommendation quality, or LLM-generated answer quality.
+
+The script deterministically constructs exactly **100 distinct questions**
+from accepted predictive CIViC records. When the snapshot exports variant
+membership, questions target **one recorded variant**. When membership is
+unavailable, questions target the **exact complete molecular profile name**
+instead; the program must not treat that name as proof about a single variant.
+The mode and membership count are recorded in the report. It
+adds counterfactual conditions (another variant, therapy, disease, opposite
+effect) and, where available, complex molecular profiles. It checks every
+counterfactual against all eligible source records before assigning its
+source-derived label, so a swapped condition with genuine matching evidence
+is not mistakenly treated as empty. Target quotas are 25 direct, 20 other
+variant, 20 other therapy, 15 other cancer, 10 opposite effect and 10 complex
+profile questions **when membership is available**. If a category has insufficient records, other categories
+fill the set; the **actual** mixture appears in the report. If fewer than 100
+unique questions exist, CI fails rather than duplicating questions.
+
+The source reference enumerates raw accepted predictive CIViC records by exact
+gene, disease, therapy and either verified single-variant membership or complete
+profile name, depending on the available snapshot; a record has a matching
+reported effect only when the direction is `SUPPORTS`. A reference question
+is marked source-answerable if it has matching support and no supporting
+report for the opposite effect or `DOES_NOT_SUPPORT` report of that effect.
+This label is derived from the **same CIViC snapshot** as all methods and
+is not an independent biomedical gold standard. Conflict and missing evidence
+mean abstain **within this snapshot**, not absence of clinical benefit.
+
+## Strategies
+
+| Method | Answer condition | Deliberate limitation |
+| --- | --- | --- |
+| `context_blind` | Any accepted CIViC record for the gene, drug and disease | Ignores variant, direction and significance |
+| `variant_only` | Any accepted CIViC record for the gene, variant/profile, drug and disease | Ignores predictive significance, direction and complex membership |
+| `claim_scoped` | The structured claim check returns `supporting_records` | Requires exact reported effect and the mode's matching scope; abstains on opposing/disputing reports and incomplete input |
+
+The baselines are **ablations of OncoGraph's own retrieval**, not standalone
+CIViC MCP, an external clinical service or an LLM. The source reference is
+field-defined, so the rules of the proposed method align with it by design:
+high scores only show consistency with the frozen source fields. In particular,
+they cannot demonstrate a novel clinical conclusion or a performance gain on
+an independently labeled question set.
+
+**Profile-mode results cannot establish variant-specific accuracy.** The
+profile-name lookup is permitted only when the question explicitly asks about
+the whole profile. For a single-variant question without membership, the claim
+checker continues to abstain. The workflow cannot silently substitute one
+meaning for the other.
+
+Metrics on the 100 questions: answer rate among source-answerable items;
+false-answer rate among source-unanswerable items; answered count/coverage;
+and error among answered. A tree with more than 200 matching records is flagged
+and excluded from metrics for **all methods**, while remaining in the detailed
+question list. There is no claim that a zero error rate generalizes outside
+this bounded snapshot.
+
+## Reproduce and inspect
+
+```sh
+node tests/test_claim_benchmark.cjs
+node scripts/benchmark_claim_abstention.cjs --snapshot web/data --questions 100 --out claim-benchmark
+```
+
+`report.json` includes all 100 query texts, question types, source-derived
+labels, EIDs per method, coverage, abstention counts and SHA-256 hashes of the
+two input JSON files. CI publishes a summary and the report as an Actions
+artifact whose name contains `profile` or `variant`, question count `Q`,
+source-answerable `P`, source-unanswerable `U`, false answers for each strategy
+(`B` context-blind, `V` variant-only, `C` claim-scoped) and claim-scoped
+answers `Y`. The detailed JSON carries exact fractions and EIDs. Failed runs
+instead put a brief error hint in the artifact name and upload the console log. The workflow
+also runs on pull requests, manually, and after successful data refreshes.
+To evaluate scientific novelty, independently review a held-out set of
+question/claim/source tuples, evaluate claim-level citations and free-text
+answers, and compare with appropriately configured CIViC MCP and LLM baselines.

docs/CLAIM_EVIDENCE.md

@@ -8,15 +8,18 @@ or patient suitability. It does not yet verify free-text LLM answers.
 
 Open **More tools → Review variant evidence → Check a variant claim** and enter
 an exact gene, variant, disease, therapy, and whether to check a report of
-`sensitivity` or `resistance`. Download the structured JSON and follow the
+`sensitivity` or `resistance`. If using the whole molecular profile name,
+explicitly choose **profile** scope; `variant` remains the conservative default.
+An exact whole-profile match does not prove a finding for an individual member
+variant. Download the structured JSON and follow the
 CIViC source URLs to verify the records. Therapy may be an exact name or NCIt
 ID **as stored in this snapshot**. No cross-source alias mapping is performed.
 
 Against a locally imported database, the same algorithm is exposed through:
 
 ```text
-GET /claim-evidence?gene=EGFR&variant=T790M&disease=<exact name or DOID>&therapy=<exact name or NCIt ID>&effect=resistance
-MCP check_variant_claim(gene, variant, disease, therapy, effect, limit=20)
+GET /claim-evidence?gene=EGFR&variant=T790M&disease=<exact name or DOID>&therapy=<exact name or NCIt ID>&effect=resistance&scope=variant
+MCP check_variant_claim(gene, variant, disease, therapy, effect, limit=20, scope="variant")
 ```
 
 The browser reads GitHub Pages JSON. The Python API and local MCP read their
@@ -28,7 +31,10 @@ First use `buildEvidenceTree` / `get_evidence_tree` with a 200-Evidence-Item
 cap. This already requires the exact gene and disease, a matching variant
 member (or flagged legacy profile-name match), accepted CIViC status, and
 preserves the original evidence ID and claim direction. Then require a therapy
-matched by exact stored name or ID and `PREDICTIVE` evidence. Other relation
+matched by exact stored name or ID and `PREDICTIVE` evidence. With explicit
+`scope=profile`, an exact whole-profile-name match can support a claim about
+that **whole profile** even when member variants were not exported; it never
+establishes a claim about an individual variant. Other relation
 types, including drug `targets` gene, trial registrations, pathway membership,
 and target–disease associations, **cannot** support a variant–therapy effect.
 
@@ -53,6 +59,12 @@ produce `context_only` for profile-name matches.
 
 ## Evaluation before publication claims
 
+An automated [100-question benchmark](CLAIM_BENCHMARK.md) runs in GitHub
+Actions after a data refresh or on a change to the benchmark. It compares
+context-blind, variant-only and claim-scoped retrieval on the **same** frozen
+snapshot. Its source-derived labels test internal consistency, not medical
+accuracy, model-written answers or independent expert agreement.
+
 This is a **rule-based baseline**, not a demonstrated improvement or a
 validated algorithm. The current [variant retrieval experiment](VARIANT_RETRIEVAL_EXPERIMENT.md)
 evaluates Evidence Item retrieval, not this claim classifier or LLM responses.

docs/MCP.md

@@ -2,7 +2,7 @@
 
 The local stdio MCP server also exposes `get_evidence_tree(gene, variant, disease, jurisdiction="US", limit=50)`. It returns accepted CIViC evidence with source links, preserves contradictory claims, and marks US regulatory labels and CIViC Assertions as not integrated. See [Evidence Tree v1](EVIDENCE_TREE.md) for exact inputs and response fields. GitHub Pages does not serve MCP.
 
-`check_variant_claim(gene, variant, disease, therapy, effect, limit=20)` classifies accepted CIViC predictive items for exact stored therapy and a reported `sensitivity` or `resistance` effect. It returns separate supporting, opposing, disputing and context-only records and abstains on incomplete profile membership or truncated input. See [claim check contract](CLAIM_EVIDENCE.md). It never converts a drug–target–disease path into a therapeutic claim.
+`check_variant_claim(gene, variant, disease, therapy, effect, limit=20, scope="variant")` classifies accepted CIViC predictive items for exact stored therapy and a reported `sensitivity` or `resistance` effect. It returns separate supporting, opposing, disputing and context-only records and abstains on incomplete variant membership or truncated input. Explicit `scope="profile"` checks the exact complete molecular profile name, never a single variant. See [claim check contract](CLAIM_EVIDENCE.md). It never converts a drug–target–disease path into a therapeutic claim.
 
 **Research use only.** OncoGraph's MCP server exposes structured oncology
 knowledge-graph data -- entities, relations, and their evidence -- to MCP