Panel is_real vote is binary — conflates hallucination filter with stochastic-hazard likelihood

[Jodre11]

Summary (needs brainstorming — not a settled fix)

The panel's is_real vote is a hard binary ("true issue or false positive — purely epistemic",
includes/panel-concern-brief.md:8-10). That is the right shape for its original job — killing
hallucinations (a finding describing code that isn't there, or misreading what is). It is the
wrong shape for a conditional / stochastic hazard: a defect whose mechanism is 100% present
in the code but whose manifestation depends on a future or external state. The binary gives such a
finding nowhere honest to land, so a conscientious panellist routes it to not_real, and the true
signal is destroyed at the vote.

This is the per-vote end of the same pipe as #61 / #62 (which concern the aggregation /
synthesiser end). Track together.

Concrete evidence (n=1, retrospective)

Scoped quality A/B: 5 code-review-suite specialists were run standalone against commit cf9bc9d of
HavenEngineering/finance-erp-apps PR #158 (the exact commit an external reviewer, "Marlon", reviewed
with his own tool), and scored against his 5 findings.

Relevant finding: ZB61 IndexOfOptional silent-failure in MarginReportReader.cs. The A&L
sub-department column is read optionally (missing → "") rather than required (throw). If ZB61 is
ever absent from A&L output (report-layout drift, path edited in MarginReportFilter but not here —
the path is duplicated as a private const in both), every A&L row's sub-department silently blanks
to "", which reads as the legitimate value 000 = None — wrong data shown to finance during the
parallel-run sense-check, with no error.

• The mechanism is unconditionally present in the code.
• Whether it bites is conditional (depends on future report/layout state).
• Our correctness specialist stood on the exact line and did not raise this. It raised a
  different, speculative concern ("verify the IsDescription guard is present") that is provably
  false — the guard is at MarginReportReader.cs:82. Its own prose hedged: "I cannot see the full
  body… this may already be handled." — i.e. its true internal state was uncertain, but with only
  a binary to land on it laundered that into a confident-sounding Important that was actually a
  coin-flip.

So even before the panel: a genuinely uncertain internal state was quantised into a false certainty.

The two-stage quantisation problem

1. Per-vote: each panellist rounds an internal probability to {real, not_real}.
2. Aggregation: three binaries collapse to a consensus fraction that can only be {0, 33, 67, 100}.

A hazard all three independently read as "plausible, ~40%" rounds unanimously to not_real → 0/3 →
dropped. The "live but uncertain" signal is destroyed at the vote, not merely diluted at
aggregation.

Why the existing severity machinery does not save it

The brief clearly tried to handle stochasticity — but on the severity axis, which fires too
late:

• panel-concern-brief.md:20-30 — "Severity rates the impact if the issue manifested… decided
  before any mitigation… a coarse control reduces likelihood, not impact, and does not lower
  severity."

That rule only engages after a panellist has already decided to call the finding is_real: true.
The lossy step is upstream of that, at is_real itself, where "true issue vs false positive"
invites routing "doesn't manifest in any state I can see today" straight to not_real. The
likelihood-doesn't-lower-severity rule never gets to fire, because the finding was already killed at
the epistemic gate.

The genuine counter-tension (why the binary exists)

The whole disposition block (panel-concern-brief.md:34-48: "be pedantic, default to raise, never
hedge to a note") is engineered to stop reviewers wishy-washing everything into a mushy middle. A
naive graded-confidence slider invites exactly that drift — everything settles at 50 and nothing
clears a bar. So this is not "binary bad, slider good."

The real diagnosis: the binary conflates two genuinely different questions:

• Does the described mechanism exist in the code? — a true epistemic gate; keep it binary and
  decisive (this is the hallucination filter, and it works).
• How likely is it to bite? — a probability that currently has no home and gets smuggled into
  the is_real vote, corrupting it.

Candidate direction (for brainstorming, not decided)

Split the gate. Make is_real strictly "the described mechanism exists in the code" (pure
hallucination filter, stays binary). Give conditional hazards a separate likelihood / trigger
dimension that feeds the rubric alongside impact — so a latent-but-unlikely defect survives as
"real mechanism · conditional trigger · high impact" instead of vanishing at 0/3. This preserves the
pedantic "default to raise" spine while giving stochastic findings an honest home.

Open questions for brainstorming:

• Does a likelihood axis reintroduce the exact mushy-middle drift the binary was defending against?
• How does the rubric (includes/verdict-rubric.md) combine a likelihood dimension with severity +
  tractability without exploding the tier logic?
• Is this better solved upstream (specialist prompts learning to raise the conditional finding
  crisply) so the panel never has to adjudicate a mis-stated one? Note the comment-truth findings in
  the same A/B were an origination failure, not an adjudication one — the panel only ever grades
  what it's handed (plus what it independently re-derives).
• Interaction with Defect 2 (residual): pro-cyclical APPROVE filter hides thin output #61 / Defect 3: synthesiser ignores other reviewers' prior findings (prior-ingestion) #62 at the aggregation end — should be co-designed.

Related

• Defect 2 (residual): pro-cyclical APPROVE filter hides thin output #61 — Defect 2 (residual): pro-cyclical APPROVE filter hides thin output
• Defect 3: synthesiser ignores other reviewers' prior findings (prior-ingestion) #62 — Defect 3: synthesiser ignores other reviewers' prior findings
• Both are the aggregation-end manifestations of "the pipeline is lossy about low-consensus /
  uncertain findings"; this issue is the per-vote-end manifestation.

Provenance

Retrospective analysis only; nothing posted to PR #158 (merged). n=1 — one PR, one commit. The split
is interpretable and consistent, not statistically established. Needs a proper brainstorming pass
before any prompt/rubric change.

[Jodre11]

Refinement: the real axis is deterministic vs stochastic — and the current framing is correct for one of them

Follow-up thinking. Still unproven — feels right, but how it shapes actual results is an empirical
question (see "How to validate" below), not something to reason our way to confidence on.

Why one framing can't serve both classes

• Deterministic finding — manifests every time its path runs (wrong key selector, always-null
  deref, a test that cannot fail). Likelihood ≈ 1. For these, the current instruction — "rate
  impact-if-manifested, ignore likelihood, don't discount for small/local/tedious"
  (panel-concern-brief.md:20-30, 34-48) — is honest and working as intended. It stops the
  reviewer shrugging a real defect down to a nicety.

• Stochastic finding — manifests only in some future/external state (the ZB61 blank, a race, a
  config/layout drift). Likelihood << 1. For these, "impact-if-manifested, ignore likelihood" asks the
  reviewer to rate a ~5%-likely hazard as if certain. The reviewer knows that overstates expected
  harm, feels the instruction is asking them to overclaim, and discharges the dissonance the only way
  the schema allows: degrade to Suggestion, or vote not_real.

Same instruction, opposite effect. The single blunt cure that prevents degradation for
deterministic findings is the very thing that causes it for stochastic ones. That is why the panel
"tends to degrade everything to a Suggestion" without the right framing — the framing is tuned for the
deterministic class and silently mis-serves the stochastic one.

What each class actually needs

• Deterministic: "don't discount for how small/local/tedious it is" — keep exactly this.
• Stochastic: the opposite permission — "it's fine that this is unlikely; say so on a likelihood
  axis and keep severity high. Low probability is not grounds to downgrade severity or to vote it
  away." Right now likelihood has no home, so it leaks into severity (→ Suggestion) or into is_real
  (→ not_real), and the finding decays either way.

Candidate shape (unchanged intent, sharper): two orthogonal ratings, no self-labelling

Do not ask the panellist to classify a finding as deterministic/stochastic — that just moves the
mushy middle to a new gate and is itself a judgement call. Instead give every finding two
orthogonal ratings and let the class fall out of the numbers:

• Impact (unchanged: impact-if-manifested).
• Trigger / likelihood — coarse ordinal, e.g. always-on / conditional / rare.

The rubric then treats high-impact + conditional as a first-class survivable state, rather than
something a panellist has to protect by lying on the severity axis or smuggling past is_real. The
deterministic-vs-stochastic distinction emerges from (impact × trigger) instead of being a label
anyone has to name.

This composes with the earlier "split is_real" idea: is_real stays the pure hallucination filter
(does the mechanism exist — binary, decisive); the trigger axis carries "how likely to fire."

Open risk (the reason this is not settled)

Does a trigger/likelihood axis just reintroduce the mushy middle the binary was defending against —
everything drifting to conditional and nothing clearing a bar? Plausible. Unknown until tested.

How to validate (empirical, not armchair)

Replay a fixed finding set through both framings (current vs two-axis) and compare outcomes:

• Stochastic exemplars: the ZB61 silent-blank (this A/B), plus a race / config-drift case.
• Deterministic exemplars: a wrong-selector / always-null / tautological-test batch.
• Success = stochastic findings stop decaying to Suggestion / not_real without the deterministic
  ones going soft (no new drift-to-conditional on genuinely always-on defects).

Needs a proper brainstorming pass + this replay before any prompt/rubric edit. Parking here as the
clean starting point.