Portable eval artifacts: mapping the sub_steps/cmp.py grading onto an open TestCase/Grader/Result schema (EvalPort)

[adhabnr-ux]

Hi maintainers,

I've been mapping code-eval harnesses onto EvalPort, an open interchange spec for LLM eval artifacts (TestCase/Grader/EvalSuite in, Result/GraderResult/ResultSet out — Python SDK on PyPI as evalport-sdk, importable as openeval). SciCode's harness looks like a clean fit and I wanted to check interest before building anything.

What I read (eval/scripts/gencode.py, eval/scripts/test_generated_code.py, src/scicode/parse/parse.py, src/scicode/compare/cmp.py):

• Each problem's sub_steps[i] (with step_description_prompt, step_background, function_header, return_line, test_cases) is effectively a TestCase: id f"{problem_id}.{step_number}", input = the step prompt (+ background when --with-background), metadata = {function_header, return_line, required_dependencies}.
• The grading step (test_generated_code.py + process_hdf5_to_tuple) runs each step's generated code against numeric targets pulled from the private test_data.h5, and are_dicts_close / cmp_tuple_or_list in cmp.py do the actual comparison (np.allclose with atol=1e-8, rtol=1e-5, plus special-casing for sympy symbols and scipy sparse matrices). That's naturally a Grader of type: "code" — the tolerance and special-case handling become params, and since the numeric targets are gated, the grader function itself has to keep running inside SciCode rather than travel in the suite JSON.
• The per-step pass/fail/timeout written to logs/*.txt, plus the correct_dict JSON aggregating pass rate per problem, map onto Result (test_case_id, passed, error: {"type": "timeout"} for the timeout case) rolled up into a ResultSet.summary — which already mirrors the two-column "Main Problem Resolve Rate" / "Subproblem" split in your leaderboard table.

Sketch, following the to_openeval()/from_openeval() shape used by adapters/autogen-openeval-adapter (I read that adapter's code end-to-end before citing it):

def to_openeval_test_case(prob_data: dict, step: dict) -> dict:
    return {
        "id": f"{prob_data['problem_id']}.{step['step_number']}",
        "input": step["step_description_prompt"],
        "graders": ["gr_scicode_numeric"],
        "metadata": {
            "function_header": step["function_header"],
            "return_line": step["return_line"],
            "required_dependencies": prob_data["required_dependencies"],
        },
    }

GR_SCICODE_NUMERIC = {
    "id": "gr_scicode_numeric",
    "type": "code",
    "params": {"atol": 1e-8, "rtol": 1e-5},
    "description": (
        "np.allclose against hidden test_data.h5 targets; "
        "are_dicts_close/cmp_tuple_or_list for dict/sparse/tuple cases"
    ),
}

with a matching to_openeval_result() reading the logs/*.txt + correct_dict output of test_generated_code.py into Result / GraderResult / ResultSet.

Why it might be worth it: SciCode ResultSets become diffable against results from other benchmarks converted the same way (the spec's benchmarks/ hub already covers things like GSM8K, MMLU, and HumanEval), and anyone running SciCode through an EvalPort-aware harness (your own eval/inspect_ai/scicode.py is right there) wouldn't need bespoke glue to compare runs.

Happy to build scicode-openeval-adapter as a standalone package (same pattern as the autogen one — doesn't touch this repo's code at all) if that's useful, or drop it if it's not a direction you want. No pressure either way.

— Sahi, independent contributor (not affiliated with this project)