Repository navigation
feat(skill-eval): add manual local artifact evaluation for #253 - #258
Conversation
Separate frozen inputs from the writable parser artifact. Preserve author-run provenance, independently replay retained source in a fail-closed namespace, and keep artifact outcomes distinct from skill efficacy and PR acceptance. Agent-assisted-by: Codex (implementation and artifact verifier) Signed-off-by: Codex <codex@openai.com> Reviewed-local-source-commit: c43e76f56ec0f62753e7977a0bd0e8ce389d8fd4 Published via GitHub Git Data API with identical tree bytes; connector commit metadata may differ.
Skill evaluationAutomatic execution / cumulative PR budget: Advisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
PLANNED: execution is not authorized by knowing this hash. Selected comparisons: 0 · Estimated task calls: 0. Applicable only while these revisions and inputs match the current PR. Plan bindings, budget, frozen exam and advisory detailsUnchanged-correct is valid neutral evidence, not incremental benefit. Per-case outcomes (discovery excluded; regressions and gaps first): []Showing at most 20 of 0 pairs; the evidence link contains all outcomes, including omitted regressions/gaps. Plan SHA-256: Model/settings:
Provider/settings: Uncovered / unsupported paths (103)
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 28f3fd6efb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| try: | ||
| return json.loads(raw) | ||
| except (ValueError, UnicodeDecodeError): | ||
| return None |
There was a problem hiding this comment.
On supported controller runtimes such as Python 3.12, an artifact that prints deeply nested JSON and exits successfully can raise RecursionError here despite staying within the 64 KiB output bound. For example, 10,000 nested arrays require only 20,001 bytes. This exception escapes both verify() and capture's SandboxUnavailable handler, aborting capture after evidence files have been written but before the result and receipt exist; subsequent capture attempts then refuse to overwrite the partial attempt. Treat excessive JSON nesting as an invalid artifact response so it produces a failed check rather than crashing the evaluation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in f122b09. _invoke now catches RecursionError while decoding artifact JSON and treats it as an invalid response, so verification records a failed check rather than aborting capture. Added a mocked 20,001-byte nested-array regression. Independent scoped verification: 48 passed, 2 actual-sandbox tests deliberately deselected; published tree 3e20bb204124fe9a746e273f700a9947c72e757e matches the tested tree. New-head hosted CI is pending. This software regression test does not establish actual sandbox qualification or agent efficacy.
Treat decoder recursion exhaustion as an invalid artifact response after the trusted worker boundary, preserving failed checks instead of interrupting evidence capture. Agent-assisted-by: Codex (review follow-up) Signed-off-by: Codex <codex@openai.com> Reviewed-local-source-commit: 3ade25aba495e8896e21bf6edce3281a22c4d67d Published via GitHub Git Data API with identical tree bytes; connector commit metadata may differ.
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Scoped LGTM for the software delivery at f122b09, not completion of #253 or an efficacy acceptance.
Existing findings reconciled: the earlier contract coverage, capture-path safety, replay portability and sandbox-readiness findings were corrected in the reviewed initial tree. The new excessive-JSON-nesting finding (r4419377730) is fixed by catching RecursionError and a 20,001-byte mocked-output regression; independent delta verification passed 48 tests with 2 actual-sandbox tests explicitly deselected. No remaining blocking software finding in this bounded slice; current-head Codex review also reports completed/no findings.
Integration with main 3731b73 was tested in combined tree d4a4b9db3a8246e9a3ec02ce10ede6edaeb3eb2b: 303 passed, 2 skipped, 2 actual-sandbox tests deselected, plus 52 subtests. Current-head Ready CI https://github.com/InferMatrixLab/InferRSI/actions/runs/40981928237 passes both Python versions and installed-package checks.
Disposition of the SkillEval advisory: its zero comparisons/planned-only output supplies no efficacy evidence. This PR delivers the local staging/capture/reporting software and fail-closed behavior only. Actual supported-host sandbox qualification and a genuine baseline/candidate agent comparison remain prerequisites for the corresponding #253 acceptance items. No model/device experiment or paid execution was performed; no unsafe fallback is accepted. #253 remains open. COMMENT only, no formal APPROVE.
Scope
Refs #253.
prepare → stage → author execution → capture → reportflow.Verification
f99ac5f35e11be83c13cda5dd7eeae1de5686412(local reviewed commitc43e76f56ec0f62753e7977a0bd0e8ce389d8fd4, published head28f3fd6efb94dad9d9cfa44702088247021afb01). Current-main integration checks remain separate.Draft / remaining limits
Workflow, diagrams, frozen task and explicit remaining checklist dispositions are documented in
docs/local-skill-evaluation.mdandtests/fixtures/skill_eval_engineering/contract.md.