Repository navigation
Hamilton metric provenance, deterministic DAG lineage, seeded groundtruthing testbed - #25
Merged
Merged
Conversation
…play tests Reference implementation for the Hamilton-substrate + insight-kit-gate stack: wire one real metric (CAC) end to end and validate it by replaying documented PPC-grind failures against the gate. - examples/cac_metric.py: CAC as a Hamilton @feature module. CAC = Meta spend / new customers, where "new" (customer_order_index==1) is declared as an explicit gate selection (V15) rather than left implicit — the exact filter behind the documented CAC drift. - integrations/hamilton/adapter.py: new ik_emit="metric" path on InsightKitHook. A tagged metric node emits a typed claim whose value is carried as a field and whose registered input is the node's own input rows (V22 registered_input), so the fingerprint is over values, not source text. Generates gate-valid claim ids (the legacy _gen_claim_id produced ids the gate rejected). Legacy claim_tier statement path is untouched. - tests/integrations/test_hamilton_metric.py: CAC emits a registered_input claim with a valid id + recorded selection; the P1 "verdict-from-proxy" replay proves a hardcoded CAC (no live rows) is downgraded from published to draft while a row-derived CAC is registered_input. The P1 replay needs only the gate + pyarrow so it runs in CI; driver tests skip without the hamilton extra. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
The test job synced only dev+polars, so every Hamilton driver/adapter test (the pre-existing test_hamilton.py and the new metric tests) skipped in CI and only ran locally. Add --extra hamilton to the test job's install step; the full suite now runs 839 tests with zero skips there. Leaves pi-seam and eval jobs unchanged — neither runs the Hamilton path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
…ciliation critic Completes item 1 of the enhancement plan and adds a second-metric genericity proof plus the first reconciliation critic (item 3b). Item 1 — provenance chain: the ik_emit="metric" path now emits a skill_use record (snapshot = the node's input rows, tool="hamilton", source = upstream node names) and the claim cites it, so a metric claim is backed by a real knowledge chain, not just a fingerprint. Research is optional and tag-driven (ik_research_source) — a pure compute node has no research, so none is faked. A node with no live input rows still emits a claim-only record (payload provenance), the correct P1 outcome. Genericity proof: examples/mer_metric.py computes MER = revenue/ad_spend (a ratio) through the SAME adapter with no engine change, and omits ik_claim_id so the id generator runs end to end. test_metric_path_is_generic_mer proves the metric path is not CAC-specific. Item 3b — reconciliation critic: check_ratio_identity generalizes check_annual_equals_monthly_sum from the additive identity to the ratio identity (CAC==spend/new_customers, MER==revenue/ad_spend). emit_reconciliation_critique turns the verdict into a critic-tier claim — a refutes edge on failure (the CAC-drift class), a supports edge on success. Deductive check: a violation is a proof of a bug, so it refutes rather than warns. Tests: 847 passed (+8). Reconciliation tests need only the gate so they run in CI; hamilton driver tests skip without the extra (P1 replay still runs in CI). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
… testbed
Lineage (item 7):
- InsightKitHook now also implements GraphExecutionHook: captures the
compiled HamiltonGraph's dependency map before execution and stamps every
ik_emit="metric" claim with its node's transitive upstream closure
(lineage field: node, direct_upstream, upstream_closure, external_inputs).
- The extraction skill_use snapshot pairs the captured input rows with the
same trace ({input_rows, lineage}); the data fingerprint stays over rows
only (V22).
- New gate module lineage.py (no hamilton import): lineage_of() /
trace_to_rows() answer "where did this number come from" from the sealed
run bundle alone; a claim without a stamped trace raises
LineageNotRecordedError — lineage is never guessed.
Seeded dockblocks testbed (groundtruthing fixture):
- examples/dockblocks_demo/datagen.py: deterministic seeded generator for
Meta+Google ad spend, orders with a pre-window returning-customer pool,
CRM contacts/deals, email lifecycle events; ground truth computed by
construction from the generated rows; write_parquet() for on-demand
large fixtures (nothing committed as data).
- examples/dockblocks_demo/dag.py: multi-layer DAG with blended_cac (true),
demo_mer, and naive_cac — a deliberate replay of the documented CAC drift
(distinct customers vs first orders) kept as a live critic target.
Tests (+12): generator determinism/ground-truth/trap/parquet; upstream
closure on the claim; lineage_of trace to exact rows; DAG-vs-ground-truth
equality; naive_cac refuted (and blended_cac supported) by the
reconciliation critic, with lineage localizing the drift to the skipped
new_customer_orders filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
…ral demo naming Scrub (public-repo hygiene): examples/dockblocks_demo -> examples/growth_demo, example claim namespace DOCK -> DEMO throughout code/tests/CLI help; no client-related names remain in the source tree. Adapter hardening (adversarial-review findings, all repro-confirmed): - execute-time overrides now truncate the stamped lineage closure and are declared in an "overridden" list — a claim fed caller-supplied values never asserts the bypassed sources (was: forged override tables got provenance pointing at the real registered inputs). - scalar-only metric nodes no longer register input — bare literals land as payload provenance and cannot publish (P1 guard held). - ik_statement tag now lands on the claim as a statement field. - re-executing one driver/RunState skips cleanly with a warning instead of emitting orphaned provenance for a rejected duplicate claim. - optional emit_timestamp on the hook/build_driver makes record ids reproducible across identical runs (golden replays). - LineageTrace gains overridden + is_overridden; datagen validates days>=1. Tests (+6): override truncation, scalar payload downgrade, statement field, duplicate-execute skip, reproducible ids, days=0 rejection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
…r prior rename) Content half of the scrub+hardening change (previous commit carried the file renames only): - prose/namespace scrub: growth_demo naming, DEMO claim namespace in code, tests, and CLI help. - overrides truncate the stamped lineage closure and are declared in an "overridden" list; LineageTrace gains overridden/is_overridden. - scalar-only metric nodes land as payload provenance (P1 guard held). - ik_statement lands on the claim; duplicate execute() skips cleanly; optional emit_timestamp for reproducible record ids; days>=1 validated. - regression tests for each confirmed review finding (+6). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Hardens the Hamilton integration into a full typed-provenance path and adds deterministic DAG lineage plus a seeded, groundtruthed demo fixture.
Metric provenance (
ik_emit="metric")published(the P1 "verdict-from-proxy" guard, replayed as a test).skill_useextraction chain (and optionalresearchrecord), not just a fingerprint; generated claim ids satisfy the gate id grammar.Deterministic DAG lineage (item 7)
InsightKitHookcaptures the compiledHamiltonGraphdependency map and stamps every metric claim with its node's transitive upstream closure (direct_upstream,upstream_closure,external_inputs,overridden).overridestruncate the closure and are declared — a claim fed caller-supplied values never asserts bypassed sources.lineage.py(no hamilton import):lineage_of()/trace_to_rows()answer "where did this number come from" from the sealed run bundle alone; absent lineage raisesLineageNotRecordedErrorrather than guessing.Reconciliation critic
check_ratio_identity()generalizes the additive identity to ratio metrics (CAC, MER);emit_reconciliation_critique()turns a verdict into a critic-tier claim withrefutes/supportsedges.Seeded groundtruthing testbed (
examples/growth_demo)write_parquet()scales fixtures on demand — no data committed.blended_cac,demo_mer, and a deliberately naivenaive_cac(the documented CAC-drift class) kept as a live critic target; tests show the critic refuting it while lineage localizes the drift to the skipped filter.Adapter hardening (from adversarial review, all repro-confirmed)
ik_statementlands on the claim; duplicateexecute()skips cleanly without orphaned records; optionalemit_timestampgives cross-run reproducible record ids;datagenvalidatesdays>=1.CI
hamiltonextra — this also fixes theModuleNotFoundError: No module named 'hamilton'failures that have keptmain's CI red (test_input_provenance.py).Testing
lineage_of, and identified the planted trap by lineage diff before confirming quantitatively.🤖 Generated with Claude Code
https://claude.ai/code/session_01Joez4MiYjd6owPcRQNcKrv
Generated by Claude Code