Repository navigation
Require source-grounded positive identity evidence before automatically merging namesakes #1193
Description
Current status (2026-10-26): decision-path measurement ready; production evidence policy remains open
#1251 is merged and covers the resolution-side passage-over-profile defect. The remaining adjudication-side path in #1136 is measured by #1377, now ready for review after syncing current dev. Its 48 scripted probes check actual A/B evidence delivery, investigation/source reads and graph/review outcomes; green CI does not mean identity gold errors are fixed. The original eight inputs/gold and named-failure rules remain intact.
The current measurement and next-policy question proposes a limited existing-field source-input/second-look experiment for automatic exact-name same, including ordinary cache and governance paths. Whether that experiment and its possible added human/call cost are acceptable is awaiting maintainer confirmation. No production gate is implemented by #1377, and no accuracy gain is claimed from the earlier F1 deltas.
#1199 stays closed on hold. Its uniqueness gate, kept behavior and old model results below are historical review material, not the current agreed implementation contract. Any additional persisted/public contract would be discussed and recorded before implementation.
Historical first slice (held)
Implemented in #1199, independently of #1192, following the maintainer's scope agreement.
- Only automatic pairs with byte-identical canonical names.
- Current direct facts on both records must establish an existing, explicitly eternal uniqueness relation; each non-empty quote must occur in its current matching source passage. Name, employer/title/topic overlap and noncontradiction alone do not authorize merging.
- Missing checkable identity evidence closes the pair as kept with a reason, without another model request or a new human review. Cached same verdicts recheck current sources.
- Existing fields only; no schema/API/ontology-field changes. Human decisions and existing execution/rollback safeguards remain authoritative. The kept reason preserves an unresolved-left role when the original human namesake review carried it; Resolve ambiguous namesake mentions from source clues before profile scores #1192 consumes that existing-field marker so later mentions cannot reopen human work or grow another bucket.
Completed checks and evaluation
Candidate 883e440, dev baseline da2c45e: DB/PDF guards enabled, workspace 1501 passed / 0 failed / 6 existing ignored; focused suite 7 passed. fmt, clippy, frozen-lockfile web build and diff checks passed. Controlled true-identity fixtures preserve two qualified merges and intentionally lose one employer-only true merge, with zero human reviews added.
Independent official identity bench: qwen3.7-flash-2026-07-15, thinking disabled, pinned local BGE-M3; two rounds, eight fresh KBs per side, all 16 complete, no gateway errors. Mean F1 0.5246 → 0.5285; pending human reviews 0.500 → 0.125.
The gate withholds 3 observed true same-entity merge pairs established by singleton official gold groups on both current records. The PR lists those pairs and all per-KB F1/entity/human/false-merge/false-split/missing counts. False-merge and false-split means increased in this small experiment; this is a narrow evidence gate, not a demonstrated end-to-end quality gain. These measurements and withheld pairs are descriptive, not a causal estimate from F1.
Combined with #1192, a deterministic DB regression verifies 20 further ambiguous mentions after machine-kept reviews: one unresolved entity and zero reopened human reviews. The two live-model experiments are independent and do not measure the combined change.
Original proposal and historical DeepSeek investigation, preserved for context
What I am trying to do
Follow-up to #1136, separate from the mention-resolution change in #1192. I would like namesake adjudication to merge only when there is inspectable evidence connecting the two records, while preserving genuine aliases and full-name/short-name matches.
The original #1136 report includes a bare-name merge at 0.85 justified by different achievements not contradicting one another. Absence of contradiction does not establish identity. The current second-look mechanism can inspect facts and passages, but a model's same verdict and confidence are not themselves a checkable identity link.
This is a proposal to discuss the smallest adjudication change, not a claim that every namesake merge is unsafe or that an implementation has already been evaluated.
Evidence and limits
We measured e9057ac and the resolution-only candidate d8e3cf8 in the same session on 2026-10-11: two rounds, eight fresh identity-bench bases per side, DeepSeek deepseek-flash with thinking enabled/high, and pinned local BGE-M3 on both sides. All document jobs completed.
Four candidate reverse-order bases still contain mixed finance/engineering facts. These should not be attributed to bad adjudication merely because a merge exists. In three, the second look correctly connects the d04 budget mention to the finance director using its explicit title. Those merges (0.86 / 0.90 / 0.90) happen before d03, the engineering document, is ingested; subsequent resolution attaches engineering facts to the survivor. The fourth has no namesake merge. This ordering has been checked, and #1192 documents the remaining resolution limitation.
Separately, byte-identical adjudication requests yielded a changed i/verdict signature in one of nine repeated request groups, and changed confidence-threshold bands in three. Those groups are a variance diagnostic, not proof of a namesake-specific failure or of this proposal's effectiveness. Per-base results and the scope of the resolution experiment are in #1192.
What would help
A small gate before automatic namesake merges:
- Require the decision to identify positive evidence connecting these particular records, using the existing facts/passages/decision trace where possible.
- Check that cited evidence exists, is still applicable, and belongs to the current pair. Same name, topic overlap, a shared employer/project, or merely non-conflicting facts should not satisfy the gate by themselves.
- If a link is missing or cannot be checked, retain the separation or use the existing human-review path, with an explicit reason.
- Apply the requirement to both batch/second-look results and cached
sameresults, so a cache hit cannot bypass the gate.
The evidence categories need agreement first. An explicit source statement of co-reference or a stable unique identifier may be strong evidence; matching department/job title alone can still fit more than one person. Checking provenance and references also does not prove the model's semantic interpretation is correct.
I would start by reusing the existing execution safeguards and decision trace rather than adding another general-purpose model verifier. This should be a separate PR from #1192, not a recall redesign, threshold increase, or rewrite of the identity architecture in #725. If additional persisted fields, ontology/public contracts are necessary, I would discuss the contract and ADR before writing that part.
Acceptance and evaluation
- A high-confidence
samebased only on identical names and absence of contradiction cannot automatically merge. - A supported true same-entity example still merges; missing, stale or unrelated evidence cannot silently authorize a merge.
- Cached decisions obey the same requirement.
- Existing contradiction, human-decision and rollback safeguards remain effective.
- Add minimal regressions that fail without the change and run in the existing CI suite.
- Evaluate before/after against a same-session baseline, several fresh bases and at least two rounds per side, as requested in Two people who share a name end as one entity through the bare-name entity #1136. Report missing/type-unbound anchors, false merges, false splits, human-review load and model usage alongside F1; do not treat a single day's count as a causal effect.
What I do today instead
The resolution guard in #1192 prevents profile scores or fallbacks from guessing between exact-name candidates when passage corroboration cannot distinguish them. That does not establish an evidence contract for adjudication.
I would be happy to take the smallest follow-up once the evidence boundary and scope are agreed. Would this be a useful first adjudication step, or is there an existing identity-evidence cut that this should join?
Activity
WaylandYang commentedon Oct 11, 2026
Thank you for separating this from #1192 and for setting out its limits so carefully.
Yes, we would welcome this as a first step, kept narrow:
sameverdicts pass the same check.Please keep it to existing fields, and tell us here first if one more turns out to be needed. One number beside the ones you list would help: how many true same-entity merges the gate loses.
Thank you again for taking this on.
virtue039 commentedon Oct 11, 2026
I’ll keep this separate from #1192 and use existing fields. The first slice will accept a link only when current facts on both records point to the same value/entity through an already-declared inverse-functional relation (or share the subject of a functional incoming relation), with each fact supported by a non-empty quote that can be checked against its current source passage. Name facts, shared employer/title without that uniqueness declaration, invalidated facts, and missing or unrelated evidence will not authorize a merge.
The check will run on every automatic exact-name
same, including cached verdicts. A failed check will keep the pair apart with a recorded reason, without adding human reviews or another model call. This intentionally does not infer arbitrary prose co-reference. I’ll report supported true same-entity merges lost as well as F1, false merges/splits, anchor coverage, entities and pending human reviews, using a fresh same-session Qwen Flash baseline and at least two rounds.virtue039 commentedon Oct 11, 2026
The agreed first slice is ready in #1199 (
883e440), separate from #1192. It uses only existing fields, rechecks cached same verdicts, records missing evidence as kept, and adds no model request or human review.The fresh Qwen Flash evaluation is complete: two rounds, eight KBs per side. There are 3 observed withheld true same-entity merge pairs; the PR identifies each, alongside all 16 per-KB results. Controlled fixtures separately preserve two qualified true merges and lose one employer-only true duplicate. I have reported the recall cost and the absence of a demonstrated overall quality gain.
Local workspace, DB-backed regressions, fmt, clippy and web build are green. The unresolved-left marker also passes the combined 20-subsequent-mention lifecycle regression with #1192, without claiming a combined live-model result.
virtue039 commentedon Oct 11, 2026
Following the hold requested on #1199, the next step is the test-only measurement in #1202, linked to #1136. This issue's previous gate proposal and Qwen results remain historical evidence, not a design agreed to land.
The new cases cover both identity failure paths and an open-graph true-merge control, with gold labels independent of the scripted verdict. Current
devhas four gold errors in eight ordered cases; strict mode fails and CI reports the errors under an explicit proposed no-regression ceiling. No additional production gate or real-model evaluation is being submitted with this measurement.WaylandYang commentedon Oct 22, 2026
Where this stands: the measurement you proposed is on dev (#1202), #1199 was closed on hold, and #1251 is a draft on the resolution side rather than the adjudication gate described here. Is there a next step you want tracked under this issue, or should it close and let #1251 carry its own discussion?
virtue039 commentedon Oct 22, 2026
Please keep #1193 for the remaining adjudication-side path in #1136: an exact-name batch
sameat 0.8 can merge a bare-name pair without a second look. #1251 handles the resolution-side path independently and has been updated for re-review.#1202 and #1220 already establish the request-content checks. The next deliverable here would focus on the remaining decision path: a deterministic reproduction recording whether investigation runs, what source evidence it actually reads, and the resulting graph/review state. It would cover ordinary/governance and fresh/cached verdicts, both creation orders, and an open-graph true-merge control with source-based gold. The existing eight inputs/gold and known-failure rules would stay intact.
The recorded examples include a wrong merge after investigation and a true pair left for human review, so forcing a second look alone is not yet a demonstrated fix. The measurement would distinguish missing investigation, missing source inspection and an incorrect final judgement before proposing a production change. It would reuse existing fields/tools and would not revive #1199's uniqueness gate or permanent kept decision.
Any candidate implementation would be a separate PR from current dev, with same-session multi-KB identity-bench results, including true merges lost, coverage, human work and model cost. The source evidence sufficient to authorize identity remains an open design question to resolve from those concrete cases before adding another production rule.
virtue039 commentedon Oct 26, 2026
The decision-path measurement is now ready for review in #1377, synced with current dev. It retains the original eight inputs/gold and the existing named-failure rules, with no production policy change.
The 48 scripted probes separate three problems: no investigation, investigation without source inspection, and a final judgement that remains wrong after both source passages were delivered. Ordinary batch/cache
same(0.8)can apply directly; governance uses 0.85 without supporting human precedents, so its 0.8 investigates while its 0.9 can apply directly. Governance does not consume the ordinary verdict cache. A finaldifferent(0.5)leaves a true pair pending human review, not formally decided different. The deliberately scripted conclusions expose these paths; their error totals are not a measured model error rate.For the next production candidate, I propose measuring one limited trigger policy first:
samepairs. Cover ordinary fresh/cached results and governance; retain human decisions, reverts, type restrictions and the current identity-entity execution safeguard.differentor permanentkeptdecision. This may add human work and calls; both costs must be reported.Before writing that candidate, could you confirm whether this limited policy is worth evaluating, and whether use of the existing human path for missing evidence is acceptable with its added queue/call cost reported? The old #1199 uniqueness gate and permanent kept behavior would remain closed. True duplicates in an open graph are required controls; false merges/splits, true merges lost, coverage/missing anchors, entities, human work and model cost would be reported in a frozen same-session comparison with several fresh bases per side. No accuracy gain is claimed from the earlier F1 deltas.