Skip to content

evidence(indexer): consolidate negative accuracy and cue-study results for #255 - #265

Merged
Dong1017 merged 1 commit into
mainfrom
issue255-consolidated-evidence
Oct 30, 2026
Merged

Dong1017 merged 1 commit into
mainfrom
issue255-consolidated-evidence

Conversation

@Dong1017

Copy link
Copy Markdown
Collaborator

Scope

Evidence-only follow-up to #255, with a consolidated efficiency/accuracy table and the three previously unpublished studies. No production source, scorer, corpus or default behavior changes; no new experiments were run for this packaging. Previously published #256/#259 results remain unchanged. This is evidence delivery, not accepted retrieval improvement; #255 remains open.

Study Preserved result
Structured-string token v3 FAIL: Hit@5 4/6→5/6, MRR7/12→8/15; development gate failed, heldout not evaluated
Blind raw-task/cue preparation INCONCLUSIVE: only3/6 source-supported pairs, below frozen4-case minimum; no retrieval execution
Source-conditioned covered cue Neutral FAIL: both arms Hit6/6 and MRR29/36; strictMRR-gain gate failed

The unified table also links historical timingv1FAIL, separate timingconfirmation with wide uncertainty, and accuracyv2FAIL. It does not pool or relabel those outcomes. Source-conditioned cases and reused trainingDEV are explicitly distinguished from independent validation. The sealed unused heldout content is absent.

Artifacts and provenance

  • Frozen protocols, inputs, exact raw gzip outputs, runner/source receipts and independent audits.
  • Rejected v3 runtime retained only as a patch/Git bundle, not installed into production.
  • Two blind-preparation local metadata fields replaced by labelled placeholders; original/public hashes and unchanged task text documented.
  • Historical source snapshots retain exact bytes with documented replay filename mapping.
  • Three classified evidence bundles add79 hashes; inventory totals311 hashes/14bundles.
  • Main table: consolidated results.

Verification

  • Bounded evidence inventory tests:17passed.
  • Ruff on affected test/validator files:passed; git diff --check:passed.
    -79newhashes,28gzip JSON captures, consolidated report/protocol references and navigation links verified.
  • Independent packaging audit passed at1468e628; finalc3aa9540 changes only replay-path clarification plus its hash, with affected tests passing again. HostedCI not run yet.

No timing/model/service/hardware experiments, spending, runtime promotion, formal approval or Issue acceptance is implied.

…s for #255

Preserve v3 FAIL, blind-cue preparation INCONCLUSIVE and covered-cue neutral FAIL with consolidated table. Evidence only; no runtime promotion or tracker closure.

Exact reviewed local tree from c3aa9540a21eb3be37925bf35ef0990e51b33c49; connector transport preserves tree contents, not local commit metadata.

Signed-off-by: Codex <codex@openai.com>
@Dong1017
Dong1017 marked this pull request as ready for review October 30, 2026 05:15
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T05:16:22.799828Z 00d6f0e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped LGTM for evidence delivery at 00d6f0e (base 6e6ee24). The remote tree matches the independently audited package: 79 added manifest hashes, 28 gzip JSON captures, preserved v3 FAIL / blind-preparation INCONCLUSIVE / covered-cue neutral FAIL, and separate historical timing outcomes. No production source changes or sealed heldout contents. The sole packaging suggestion (consistent restored PREP_ROOT instructions) was adopted and rechecked; current native Codex review 6515560061 reports no unresolved findings, and no other review threads remain.

Verification: 17 bounded evidence tests plus integrity/reference checks passed; Ready CI run 41007369198 passed full-check and installed-package jobs on Python 3.10/3.11, and knowledge-model-policy passed. Earlier cancelled draft CI is superseded. The PR-body hosted-CI statement describes submission time; these are the subsequent results.

Limits: this accepts archival/packaging correctness, not retrieval improvement, generalization, agent efficacy or campaign acceptance. No new experiments were run during packaging. #255 remains open; efficiency and accuracy goals remain unfulfilled. COMMENT only, not a formal APPROVE.

@Dong1017
Dong1017 merged commit c97f25e into main Oct 30, 2026
8 of 12 checks passed
@Dong1017
Dong1017 deleted the issue255-consolidated-evidence branch October 30, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant