Skip to content

Hamilton metric provenance, deterministic DAG lineage, seeded groundtruthing testbed - #25

Merged
nyashkn merged 7 commits into
mainfrom
claude/session-03914w
Jul 22, 2026
Merged

nyashkn merged 7 commits into
mainfrom
claude/session-03914w

Conversation

@nyashkn

@nyashkn nyashkn commented Jul 22, 2026

Copy link
Copy Markdown
Owner

What this does

Hardens the Hamilton integration into a full typed-provenance path and adds deterministic DAG lineage plus a seeded, groundtruthed demo fixture.

Metric provenance (ik_emit="metric")

  • A tagged Hamilton node emits a gate claim whose registered input is the node's own input rows (V22) — a value with no live rows lands as payload provenance and cannot hold published (the P1 "verdict-from-proxy" guard, replayed as a test).
  • Claims are backed by a real skill_use extraction chain (and optional research record), not just a fingerprint; generated claim ids satisfy the gate id grammar.
  • Genericity proven with a second metric (MER) using the same adapter unchanged.

Deterministic DAG lineage (item 7)

  • InsightKitHook captures the compiled HamiltonGraph dependency map and stamps every metric claim with its node's transitive upstream closure (direct_upstream, upstream_closure, external_inputs, overridden).
  • Execute-time overrides truncate the closure and are declared — a claim fed caller-supplied values never asserts bypassed sources.
  • New gate module lineage.py (no hamilton import): lineage_of() / trace_to_rows() answer "where did this number come from" from the sealed run bundle alone; absent lineage raises LineageNotRecordedError rather than guessing.

Reconciliation critic

  • check_ratio_identity() generalizes the additive identity to ratio metrics (CAC, MER); emit_reconciliation_critique() turns a verdict into a critic-tier claim with refutes/supports edges.

Seeded groundtruthing testbed (examples/growth_demo)

  • Deterministic generator (Meta+Google spend, orders with a pre-window returning-customer pool, CRM contacts/deals, email events) whose ground truth is computed by construction from the generated rows; write_parquet() scales fixtures on demand — no data committed.
  • Multi-layer demo DAG with a true blended_cac, demo_mer, and a deliberately naive naive_cac (the documented CAC-drift class) kept as a live critic target; tests show the critic refuting it while lineage localizes the drift to the skipped filter.

Adapter hardening (from adversarial review, all repro-confirmed)

  • Scalar-only metric nodes cannot register input (P1 bypass closed); ik_statement lands on the claim; duplicate execute() skips cleanly without orphaned records; optional emit_timestamp gives cross-run reproducible record ids; datagen validates days>=1.

CI

  • Test job installs the hamilton extra — this also fixes the ModuleNotFoundError: No module named 'hamilton' failures that have kept main's CI red (test_input_provenance.py).

Testing

  • 865 tests passing locally (+~30 in this branch), ruff clean.
  • Four layers: unit, integration, golden-replay of documented grind failures (P1/P5), and an agentic E2E where an LLM subagent drove the library end-to-end from a goal, traced a metric to exact input rows via lineage_of, and identified the planted trap by lineage diff before confirming quantitatively.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv


Generated by Claude Code

claude added 7 commits July 22, 2026 16:00
…play tests

Reference implementation for the Hamilton-substrate + insight-kit-gate stack:
wire one real metric (CAC) end to end and validate it by replaying documented
PPC-grind failures against the gate.

- examples/cac_metric.py: CAC as a Hamilton @feature module. CAC = Meta spend /
  new customers, where "new" (customer_order_index==1) is declared as an explicit
  gate selection (V15) rather than left implicit — the exact filter behind the
  documented CAC drift.
- integrations/hamilton/adapter.py: new ik_emit="metric" path on InsightKitHook.
  A tagged metric node emits a typed claim whose value is carried as a field and
  whose registered input is the node's own input rows (V22 registered_input), so
  the fingerprint is over values, not source text. Generates gate-valid claim ids
  (the legacy _gen_claim_id produced ids the gate rejected). Legacy claim_tier
  statement path is untouched.
- tests/integrations/test_hamilton_metric.py: CAC emits a registered_input claim
  with a valid id + recorded selection; the P1 "verdict-from-proxy" replay proves
  a hardcoded CAC (no live rows) is downgraded from published to draft while a
  row-derived CAC is registered_input. The P1 replay needs only the gate + pyarrow
  so it runs in CI; driver tests skip without the hamilton extra.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
The test job synced only dev+polars, so every Hamilton driver/adapter test
(the pre-existing test_hamilton.py and the new metric tests) skipped in CI and
only ran locally. Add --extra hamilton to the test job's install step; the full
suite now runs 839 tests with zero skips there. Leaves pi-seam and eval jobs
unchanged — neither runs the Hamilton path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
…ciliation critic

Completes item 1 of the enhancement plan and adds a second-metric genericity
proof plus the first reconciliation critic (item 3b).

Item 1 — provenance chain: the ik_emit="metric" path now emits a skill_use
record (snapshot = the node's input rows, tool="hamilton", source = upstream
node names) and the claim cites it, so a metric claim is backed by a real
knowledge chain, not just a fingerprint. Research is optional and tag-driven
(ik_research_source) — a pure compute node has no research, so none is faked.
A node with no live input rows still emits a claim-only record (payload
provenance), the correct P1 outcome.

Genericity proof: examples/mer_metric.py computes MER = revenue/ad_spend (a
ratio) through the SAME adapter with no engine change, and omits ik_claim_id so
the id generator runs end to end. test_metric_path_is_generic_mer proves the
metric path is not CAC-specific.

Item 3b — reconciliation critic: check_ratio_identity generalizes
check_annual_equals_monthly_sum from the additive identity to the ratio identity
(CAC==spend/new_customers, MER==revenue/ad_spend). emit_reconciliation_critique
turns the verdict into a critic-tier claim — a refutes edge on failure (the
CAC-drift class), a supports edge on success. Deductive check: a violation is a
proof of a bug, so it refutes rather than warns.

Tests: 847 passed (+8). Reconciliation tests need only the gate so they run in
CI; hamilton driver tests skip without the extra (P1 replay still runs in CI).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
… testbed

Lineage (item 7):
- InsightKitHook now also implements GraphExecutionHook: captures the
  compiled HamiltonGraph's dependency map before execution and stamps every
  ik_emit="metric" claim with its node's transitive upstream closure
  (lineage field: node, direct_upstream, upstream_closure, external_inputs).
- The extraction skill_use snapshot pairs the captured input rows with the
  same trace ({input_rows, lineage}); the data fingerprint stays over rows
  only (V22).
- New gate module lineage.py (no hamilton import): lineage_of() /
  trace_to_rows() answer "where did this number come from" from the sealed
  run bundle alone; a claim without a stamped trace raises
  LineageNotRecordedError — lineage is never guessed.

Seeded dockblocks testbed (groundtruthing fixture):
- examples/dockblocks_demo/datagen.py: deterministic seeded generator for
  Meta+Google ad spend, orders with a pre-window returning-customer pool,
  CRM contacts/deals, email lifecycle events; ground truth computed by
  construction from the generated rows; write_parquet() for on-demand
  large fixtures (nothing committed as data).
- examples/dockblocks_demo/dag.py: multi-layer DAG with blended_cac (true),
  demo_mer, and naive_cac — a deliberate replay of the documented CAC drift
  (distinct customers vs first orders) kept as a live critic target.

Tests (+12): generator determinism/ground-truth/trap/parquet; upstream
closure on the claim; lineage_of trace to exact rows; DAG-vs-ground-truth
equality; naive_cac refuted (and blended_cac supported) by the
reconciliation critic, with lineage localizing the drift to the skipped
new_customer_orders filter.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
…ral demo naming

Scrub (public-repo hygiene): examples/dockblocks_demo -> examples/growth_demo,
example claim namespace DOCK -> DEMO throughout code/tests/CLI help; no
client-related names remain in the source tree.

Adapter hardening (adversarial-review findings, all repro-confirmed):
- execute-time overrides now truncate the stamped lineage closure and are
  declared in an "overridden" list — a claim fed caller-supplied values
  never asserts the bypassed sources (was: forged override tables got
  provenance pointing at the real registered inputs).
- scalar-only metric nodes no longer register input — bare literals land as
  payload provenance and cannot publish (P1 guard held).
- ik_statement tag now lands on the claim as a statement field.
- re-executing one driver/RunState skips cleanly with a warning instead of
  emitting orphaned provenance for a rejected duplicate claim.
- optional emit_timestamp on the hook/build_driver makes record ids
  reproducible across identical runs (golden replays).
- LineageTrace gains overridden + is_overridden; datagen validates days>=1.

Tests (+6): override truncation, scalar payload downgrade, statement field,
duplicate-execute skip, reproducible ids, days=0 rejection.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
…r prior rename)

Content half of the scrub+hardening change (previous commit carried the
file renames only):
- prose/namespace scrub: growth_demo naming, DEMO claim namespace in
  code, tests, and CLI help.
- overrides truncate the stamped lineage closure and are declared in an
  "overridden" list; LineageTrace gains overridden/is_overridden.
- scalar-only metric nodes land as payload provenance (P1 guard held).
- ik_statement lands on the claim; duplicate execute() skips cleanly;
  optional emit_timestamp for reproducible record ids; days>=1 validated.
- regression tests for each confirmed review finding (+6).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
@nyashkn
nyashkn merged commit e45ce3d into main Jul 22, 2026
6 checks passed
@nyashkn
nyashkn deleted the claude/session-03914w branch July 22, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants