From 31d60fe3846b5f4da54f618f1648fd6d5c566bd8 Mon Sep 17 00:00:00 2001 From: minixalpha Date: Sun, 11 Oct 2026 07:38:56 -0700 Subject: [PATCH 01/15] test(benchmarks): register a 12-task main development baseline --- .../configs/dev12-baseline-20261011/README.md | 84 +++ .../adaptive-rejection-sampler.json | 22 + .../build-pov-ray.json | 22 + .../cobol-modernization.json | 22 + .../constraints-scheduling.json | 22 + .../custom-memory-heap-crash.json | 22 + .../db-wal-recovery.json | 22 + .../dev12-baseline-20261011/experiment.json | 504 ++++++++++++++++++ .../fix-code-vulnerability.json | 22 + .../git-multibranch.json | 22 + .../dev12-baseline-20261011/install-only.json | 66 +++ .../dev12-baseline-20261011/main-prompt.txt | 9 + .../multi-source-data-merger.json | 22 + .../nginx-request-logging.json | 22 + .../pytorch-model-cli.json | 22 + .../query-optimize.json | 22 + .../dev12-baseline-20261011/selection.json | 96 ++++ benchmarks/harbor/scripts/run_comparison.py | 155 ++++++ benchmarks/harbor/tests/test_comparison.py | 57 ++ 19 files changed, 1235 insertions(+) create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/README.md create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/adaptive-rejection-sampler.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/build-pov-ray.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/cobol-modernization.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/constraints-scheduling.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/custom-memory-heap-crash.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/db-wal-recovery.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/experiment.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/fix-code-vulnerability.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/git-multibranch.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/install-only.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/main-prompt.txt create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/multi-source-data-merger.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/nginx-request-logging.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/pytorch-model-cli.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/query-optimize.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/selection.json create mode 100644 benchmarks/harbor/scripts/run_comparison.py create mode 100644 benchmarks/harbor/tests/test_comparison.py diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/README.md b/benchmarks/harbor/configs/dev12-baseline-20261011/README.md new file mode 100644 index 0000000..1e3ebd8 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/README.md @@ -0,0 +1,84 @@ +# Twelve-task development baseline from main + +This cohort adds twelve tasks to the 22 previously exercised tasks. Selection +was fixed by public task names and desired coverage before downloading task +packages, reading their instructions, or seeing outcomes. It is a purposive +**development** cohort, not a representative random sample of Terminal-Bench. +The other 55 tasks are left unopened by this experiment. + +The runtime is main at `2eb99b608dfb39464d17d109a4a8f11afda607a9`, isolated in +`jobs/dev12-baseline-main-source`. This branch records configuration and evidence; +it does not change the agent or adapter. Neither prompt candidate from PR #48 +is included. The fixed dataset revision is +`sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a`. + +[Selection](selection.json) records the existing 22 tasks, selected revisions and +rationales. The [registered plan](experiment.json) records runtime, prompt, +configuration and task-file hashes, work limits, preflight and queue order. + +| Task | Coverage | Work budget (seconds) | +| --- | --- | ---: | +| `build-pov-ray` | Legacy native compilation | 11,820 | +| `cobol-modernization` | Language migration and behavioral compatibility | 720 | +| `custom-memory-heap-crash` | C++ memory debugging | 1,620 | +| `fix-code-vulnerability` | Security repair and behavior preservation | 720 | +| `nginx-request-logging` | Service configuration and observable behavior | 720 | +| `git-multibranch` | Git deployment and service integration | 720 | +| `db-wal-recovery` | Database recovery and data integrity | 720 | +| `multi-source-data-merger` | Data integration and conflict resolution | 720 | +| `query-optimize` | SQL correctness and performance | 720 | +| `adaptive-rejection-sampler` | Numerical algorithms and statistical checks | 720 | +| `constraints-scheduling` | Calendar constraints and temporal reasoning | 1,020 | +| `pytorch-model-cli` | C inference and model CLI contracts | 720 | + +All trials use `deepseek/deepseek-flash` at the official DeepSeek +Anthropic-compatible endpoint, 100 replies and 65,536 output tokens per request. +Temperature and reasoning effort are unset. Each task runs once. Work budgets +follow the existing policy: native task agent timeout minus 180 seconds. +POV-Ray has a 12,000-second native limit; it is retained rather than capped to +match the shorter tasks. Setup independently allows 1,800 seconds. + +The queue is fixed by ascending work budget, preserving selection order on ties. +Consecutive pairs finish before the next pair starts; at most two trials run +concurrently. There are no outcome-based task replacements or whole-trial +retries. Verifiers retain their original task limits and may retry only after a +timeout, at most three total attempts using the same agent output. + +## Run and preserve evidence + +Run from the repository root after creating the exact source checkout. Pass +credentials through the process environment, never configuration or logs. Set +`ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic` and leave +`ANTHROPIC_MODEL` unset so the recorded configuration selects the model. + +```bash +python benchmarks/harbor/scripts/run_comparison.py \ + --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment.json \ + --phase preflight + +python benchmarks/harbor/scripts/run_comparison.py \ + --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment.json \ + --phase model +``` + +The reusable orchestrator calls only the standard +`uv run --project benchmarks/harbor python -m harbor_adapter run --config ...` +workflow from the frozen repository root. It creates an immutable registration, +requires all twelve install-only results before model work, and refuses to retry +a job already recorded as started. Source snapshots and wheel manifests remain +in the normal Harbor job-input directories. + +All twelve selected verifier dependency profiles are explicitly +`not_configured`. Agent installation preflight does **not** certify verifier +dependencies. Preserve original images, package sources and official scripts. +Record setup failures, verifier errors and scored failures separately; preserve +raw rewards even when an unchanged verifier writes zero before tests start. + +Inspect only public metadata/instructions before every model output freezes. +Hashing task files does not require inspecting their contents. Afterwards, audit +traces for external benchmark answers, verification quality and incomplete +accounting. A single observed pass or failure is not a stability estimate. +Keep these twelve results separate from previous pilot20 and prompt comparisons. + +For a later reproduction, use new job names and a new registration directory; +never overwrite this baseline's artifacts. diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/adaptive-rejection-sampler.json b/benchmarks/harbor/configs/dev12-baseline-20261011/adaptive-rejection-sampler.json new file mode 100644 index 0000000..381d3f9 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/adaptive-rejection-sampler.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/build-pov-ray.json b/benchmarks/harbor/configs/dev12-baseline-20261011/build-pov-ray.json new file mode 100644 index 0000000..c228cfd --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/build-pov-ray.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 11820 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/cobol-modernization.json b/benchmarks/harbor/configs/dev12-baseline-20261011/cobol-modernization.json new file mode 100644 index 0000000..459c25e --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/cobol-modernization.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/constraints-scheduling.json b/benchmarks/harbor/configs/dev12-baseline-20261011/constraints-scheduling.json new file mode 100644 index 0000000..4958be1 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/constraints-scheduling.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1020 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/custom-memory-heap-crash.json b/benchmarks/harbor/configs/dev12-baseline-20261011/custom-memory-heap-crash.json new file mode 100644 index 0000000..3974e78 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/custom-memory-heap-crash.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/db-wal-recovery.json b/benchmarks/harbor/configs/dev12-baseline-20261011/db-wal-recovery.json new file mode 100644 index 0000000..3781ed1 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/db-wal-recovery.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/experiment.json b/benchmarks/harbor/configs/dev12-baseline-20261011/experiment.json new file mode 100644 index 0000000..364ef4b --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/experiment.json @@ -0,0 +1,504 @@ +{ + "schema_version": 1, + "registered_at": "2026-10-11T14:37:12.539348+00:00", + "experiment": "dev12-main-baseline-20261011", + "selection_file": "selection.json", + "selection_sha256": "cd51acea6592c01ce9ec8cde34cd46a4965a4164950194f2de7e8ca54649bd29", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "arms": { + "main": { + "source_root": "jobs/dev12-baseline-main-source", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "prompt_file": "main-prompt.txt", + "prompt_sha256": "37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d", + "prompt_words": 385 + } + }, + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "selection_rationale": "Native compilation and build tooling", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 12000, + "work_budget_seconds": 11820, + "verifier_timeout_seconds": 12000.0, + "category": "software-engineering", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/build-pov-ray:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "38380f64bc631916e264f80ce4d37feedbc783310001475b367d03989d114fdf", + "environment/Dockerfile": "44f2b5fc7311d8f336daaaca99dd4216b49355e67f50b494ebacf1b0f7c14e92", + "environment/deps/illum1.pov": "5a1f7f53ee0730c4923fcd21cd4a9c34719024d8557920441551ba7d96e75dc4", + "instruction.md": "56c8ecdd0084a07f8c97986c43af5c37ed03dd5aa994da6735cf59806162d1b2", + "solution/solve.sh": "3da9dbf589656512cd7a7004dc3752d3552a6af939cfe59eff79f91695fa42c7", + "task.toml": "b90a47ec26c1191d3c523cc094ea96b170e42fbfe176ef1f814c57d05390c097", + "tests/reference_illum1.png": "bb084f430bbc9d5b623a656920670dc06cee6d9132a9d550be4a1b99e41ab9e0", + "tests/test.sh": "928d1497ad21aef0c456294e3784f1fddf8493410f23584da89f9e02ec6bd134", + "tests/test_outputs.py": "64d6ce107503714ada346e5606832ee886be9950971cf81523e73d8ba28e8d37" + } + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "selection_rationale": "Legacy-language migration and behavioral compatibility", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "software-engineering", + "declared_difficulty": "easy", + "docker_image": "alexgshaw/cobol-modernization:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "c89318cf59efcc7040e3d946022e2616161b6733931a080cdeea7b1918e7ecc4", + "environment/Dockerfile": "8b175c83fc03ec7d16d6101516bc5100230b592f480e435279f4fe051c9280b2", + "environment/data/ACCOUNTS.DAT": "46f63eb29442ebc6716bcd8a0c36f4367910f2783f6b84a307ed9a61ccd1046e", + "environment/data/BOOKS.DAT": "db04da646207cd87f8e9c5b6f0b57d95953899f3eef22de1a81423f06ef4a9fd", + "environment/data/TRANSACTIONS.DAT": "1fa4cc5420ded94d585c563e8196b85166ebbae2bb8924b20014717fb2fb9ee3", + "environment/src/INPUT.DAT": "52fc984f0cf14af1495285d7efe02b9bae13159eea59255560ee938c4f78afee", + "environment/src/program.cbl": "27cb0e56f65f52fa75f346720b37a180b8e034ebe7d7622b979f195087bd6ebb", + "instruction.md": "a87074f1141b7345394cd88053d0debb0dc8f7de95308882c6538ee9a3b359da", + "solution/solve.sh": "7d4a0504e1686c1790bfbc074d4e598f4c631f54c2bc5f3d5fd786574b4810b9", + "task.toml": "4b620211dc3d6b1b989f0210c5ecd2f606974581c8bce27bd78659d48cf0cfbd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "009180a7092583fc8143bb3ffc3957923aa0d61d0aac3a471f76eeb4aae54a88" + } + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "selection_rationale": "Native memory management debugging", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "category": "debugging", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/custom-memory-heap-crash:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a06ed23a2c467aa36d0d4a88e5c020b1076cc25b8d483517239658e9c596dde9", + "environment/Dockerfile": "6ee81e5a4fa2540443d7f27d577ba840728fc6aab9f301ab7919103673c78080", + "environment/patches/locale_classes.tcc.patch": "16951f4a4df6bf253886ac75f5a39fdbc328e415194493f63cb7589a364aa817", + "environment/patches/locale_init.cc.patch": "e97f133ceebd86f8881f972e0e45b0ec0a13a9bfa428d8032008707683d54292", + "environment/program/main.cpp": "b94538c76887cc2b2ab739b5ac4731cddd0faa8f4a9892a6265c062bbbeede14", + "environment/program/user.cpp": "92b203786fed1bf0b3aa1675c9cd2d82b2651170675064da8ffee73b98ef14f9", + "environment/program/user.h": "f43d155995a56c22b9cd1417daef7c3b5a976cdc662a98eef0feb438ae1eee77", + "instruction.md": "53c68ae14a3c3af9a97c40bfd9bdc30a5d98b37eaa2f69f763475745f70f108d", + "solution/solve.sh": "b0b1d10bea1cd5be14a82b35d522df616df1e67028f90d2a6bef90aaa98511a1", + "task.toml": "ed474be1983e212913aa8866ebb66216acbe36aef2339ac375d232aa46617481", + "tests/test.sh": "70a2b55f0c5757f6b123a216dfd202be4a646680bced28dddcd31fd8959a058f", + "tests/test_outputs.py": "4e1cb33f777fa7fb5e1536f9bea9eace6deccd515c67d4f87570adbe692914ba" + } + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "selection_rationale": "Security bug repair and behavior preservation", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "security", + "declared_difficulty": "hard", + "docker_image": "alexgshaw/fix-code-vulnerability:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "87f28dbad183c3a634887696d328ca68a8a0858f132bc48cb7341c22267d38f7", + "environment/Dockerfile": "0917470986224df4daf4854a71302ae81c73ba5dd6e25be1232d29b758a5b3ab", + "instruction.md": "89b1abdf0af19399f720233b5cdf6b2e8e0ff7b6e3fe3e2a3af0d4d298a8641a", + "solution/solve.sh": "7a0131b369d83ae8546ecc3f9b4e23d38240f1bd1f0941e4430fa85b5182ee03", + "task.toml": "9aa2bd14ce9408289d199d0608bc3fa65b8b5e23374c9641020de62197dbf1e2", + "tests/test.sh": "f1611da693cb4cfc5e8482698c6a1858a5c67220ae9db89b50aed66584e9b17f", + "tests/test_outputs.py": "c006a5de95b69bd78055aa26cc055734dbd50718ede25d331f767bfee4ef7f26" + } + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "selection_rationale": "Service configuration and observable request behavior", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "system-administration", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/nginx-request-logging:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d79997586d1c14ed8dbfea050e3580b25500e04c8345acfd3f294152433c795a", + "environment/Dockerfile": "43003466d89ab83484a3fa618e35b58cdb74a002b1e8716ef30f12b3ac8302d8", + "instruction.md": "26a7ac98aced6107f147206790ada77bbf8ed3c25fb360cf323d5fc6889edf99", + "solution/solve.sh": "2454bbbf7c201a9a778165f286373ba2cb991a0eb1d026a8005d57ae71250dd9", + "task.toml": "b7a78d00be13656f75069197100b7b8c4fd268decf6f78929c853b3222a2729d", + "tests/test.sh": "27761195f53cbfe5941fe39b573631f6c926869e64b7c43220377e1e116528e7", + "tests/test_outputs.py": "045cc716c14efde3b0dcff5fc7c85ec5d18bfc6ce66f8b40a418fa2a3a4acda0" + } + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "selection_rationale": "Cross-branch repository operations", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "system-administration", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/git-multibranch:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3b9473429381d9eacabab498cb4a8915aa2bca24fa857ef862e5991d3f81f113", + "environment/Dockerfile": "88d9a9f4fef1ae7fddf632799dedf5a2fb6489460a93b610c33dd8b122982821", + "environment/default.conf": "a3a4c5d0ac9cfe79b0510e4225c80fa3988c27fbac13622505886bf66449e04a", + "instruction.md": "54014a38d7526d65faeb6c11d9b7ed3d4a5cb5d973d68d3da3b68b8c0289e125", + "solution/solve.sh": "300ca96b79db8d77f14742109aaf76048aafc4644ced750b1bc5c2761e8d0175", + "task.toml": "4b9981604578a31205257cca36a28d0c89c9d8d1058541330acd11d3f8d65f82", + "tests/test.sh": "ddbde3eea6ffa8fda105cbd86597a53bd409f41e13223900aa27e77a84b6b155", + "tests/test_outputs.py": "f03c6867663b8e8fb7899df51ee4df26411fab62ed491753964b801e591d16c5" + } + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "selection_rationale": "Database recovery and data integrity", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "file-operations", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + } + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "selection_rationale": "Data integration and cross-input consistency", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "data-processing", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/multi-source-data-merger:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9bf37162a4c1b88ed93627830bb8512ebae154325d7cb069c883144a0e4c69d6", + "environment/Dockerfile": "c95a40d3d35b473dc15ae1e8784ba7fb1725805c0583fc574dd2764325d75e4f", + "environment/data/source_a/users.json": "75e51e2d3ab0bdba2269d7cf3757576f84f3ee2c13407d2db94289a5958a1778", + "environment/data/source_b/users.csv": "8e87d16dc47d166edd8e56165955bad265b630d53d02762367c816affb5fa760", + "environment/data/source_c/users.parquet": "23d1ea51c4170bdba53febb885a0f495e28de525a52667347b2932136b9a72fe", + "instruction.md": "d1d28bb6e37f7a255a1b7f78aae8591f35c0494f33e4913392424ad954aab944", + "solution/solve.sh": "1b3d0d50e31930d042495d48f4c8e95f1a9c1a979c803b7f7cec651322537e1e", + "task.toml": "ee3964c0ee040f0a0cb121cefefbc15443d127684acb003fa5b61dd42f31c694", + "tests/test.sh": "d5a00e64413c0e1ae452f38a1aa1a47c3ed7b4416bbfccae1a4c053025a336d1", + "tests/test_outputs.py": "a070888930cf829aeca039771260b1386bda990ea805f313468d381ba3608ca5" + } + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "selection_rationale": "Database query performance and correctness", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 1800.0, + "category": "data-science", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/query-optimize:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a82c53e333a0ac0b0b410b5e26cb8e7305dece31db84d340248ffb13d5a45160", + "environment/Dockerfile": "064818771f200e0517d22068f87229a7a95d0a82b0778c84ca7203bf1926a0b8", + "environment/task-deps/my-sql-query.sql": "43382792f0f438625d643e34abe144febd5669518f4ab1ab550a15c6d9314a21", + "instruction.md": "473e12d320d2732f679fb631c2a4dab4a01deea6775686718ae0c417b8a39447", + "solution/solve.sh": "0d95f3cedb287ff9d47c796cbaf812c300d1222c10fa80c49e8acb0e81eb6978", + "task.toml": "6e88f2a86df699594bcb1bf81df4ed694505f79c9420f671335b02524bbe3868", + "tests/golden.sql": "4da1885d56b3ffa7b7b22b422996d018795611f6963499d551582881b74b84fd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2068ae80c57af8d52a7ae510f0666732c057806e556b480970f36f14bae683f6" + } + }, + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "selection_rationale": "Numerical sampling and statistical invariants", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "scientific-computing", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/adaptive-rejection-sampler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "95408217b9c484ffc18a02a91b7706323b6902919c72b24866a32e27a23543e2", + "environment/Dockerfile": "e644a3343e339303230f1441b7383f076d21c69a5a7aa6430b4f8a9d27f15c8b", + "environment/protected.tar.gz.enc": "7fc9bf1347bb97da8dede04b2c58638f225070913e222d76627068aebd1b1df2", + "instruction.md": "e6d28c28acd76bef61817427c74c2a29e3e8d75f535cbd708afcbfd02c54013f", + "solution/solve.sh": "fe443a8082b5183ba1623311ba0ee958888aefa633eff93766b9ae14e9bdd84d", + "task.toml": "6ed7b37f74beb46e9d47b0f5fad53025667c9548baff4e96d467b56d429fab1a", + "tests/test.sh": "38b43560d173cc2b952c3a3e17b8a480216d84e33450515e047bcb0d806b1e0a", + "tests/test_outputs.py": "547dc6e107f034f41703722aeceb6d0236e3fb69116fc3f2fbaba11884de352f" + } + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "selection_rationale": "Constrained algorithmic planning", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 1200, + "work_budget_seconds": 1020, + "verifier_timeout_seconds": 1200.0, + "category": "personal-assistant", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/constraints-scheduling:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d54a690970149e1216bd4fc066b45069c7948c13e3b1ef2a213a74342805f209", + "environment/Dockerfile": "7d9aba072a4a907229728b9739c8cf821fc2c98450babd64452682caf3759c84", + "environment/inputs/alice_calendar.ics": "5a5d45804bffd339c2479d49e76fe817b38244d14d389afe1de583443808ecbc", + "environment/inputs/bob_calendar.ics": "c97e85dbcdd756758f8a80f4db5fd5c1dcacee79298e4d2262f46de5ff680b50", + "environment/inputs/carol_calendar.ics": "32a0f8687637df0a12b8e5c56d49c4192bc3e790856943632d70f237c7c1363b", + "instruction.md": "342def78177f9bc41bea4d11af679bc385a9dda2ac8a1e425be384a5b7f937b6", + "solution/solve.sh": "802d81ec064fad3050bbade95a2f847ac24e0b7f80fa69b0bcee0347dc92fe77", + "task.toml": "7b9842de16d74e8faf8c7c0aa03b0b81145941cb12dd322d81ae23fc48607250", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "1422b345ba3aacdfc4184e962811b6379988435896c50d8cf62903e70990c5d1" + } + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "selection_rationale": "Model packaging and command-line contracts", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "model-training", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/pytorch-model-cli:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "35a093fa5f1592737784ba718c2e3e9ae17ff8b5e53e003fae560ddce091039c", + "environment/Dockerfile": "8fd5c0f21a6914cd440d10c84c9db06fec606f20dc3e0363b17a3b8c220dc4d9", + "environment/task-deps/cJSON.c": "0e7995261640c96db16ed2c91a5d981efa917cf5e72d0f97ec45a1179ae3ad73", + "environment/task-deps/cJSON.h": "0909160541bcc510e0b2d8c7a9cc051fa1459a0f13be4d8e56346a15491f2f2f", + "environment/task-deps/image.png": "f4d5dfefe463939996541404340e7099d5957b56e1df3760d13b5d2de754db53", + "environment/task-deps/lodepng.cpp": "67873c5f6272f779a143a9c81070e8783327c5552d0b9de2aaee2cb779308917", + "environment/task-deps/lodepng.h": "b735b7e0357319973e9488d520d7f270d734a6e9e1bf924fb1d5b8d26d7e97ce", + "environment/task-deps/model.pth": "a32103eef09936e586b1a7dc7e64d279500a38ff30fdd6df2167e28eb954eee6", + "environment/task-deps/model.py": "e4d3a5c728449656d0efde7871ee227619348352769795ed1be1a8b7f835fddc", + "instruction.md": "ce36ee1f6c589f81ee6a6be4967e25c76a2ed697006928023f42dfb9f28a6750", + "solution/solve.sh": "3b34e561c9ef0a53c9948655847846eb4811aaac67b990480aa6aac1b04bfe66", + "task.toml": "fc1c018efbd5d0537ac0741d53d1ebad3886faa51d14cbde8ec953856411bc0d", + "tests/test.sh": "43fb67e537c7adc2df11c2367f47c876cb5d16a7c1ae16d0339c3a3364bb2b06", + "tests/test_outputs.py": "370702c57a4e0c85f70ee37a5c357ae3a5df8480a275bb6263b0376f2f6c093b" + } + } + ], + "total_model_runs": 12, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "preflight": [ + { + "arm": "main", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/install-only.json", + "config_sha256": "b8fc1d1d825cc514c345026bdfce4cd867443125aac0de1d0d6e02b6c3e7c30a", + "job_name": "dev12-main-install-only-20261011" + } + ], + "queue": [ + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cobol-modernization", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/cobol-modernization.json", + "job_name": "dev12-cobol-modernization-baseline-20261011", + "config_sha256": "2680126c2f162d58948cd30565806bf6cb1f026d561c87b942ba4735173ea6a2" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/fix-code-vulnerability", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/fix-code-vulnerability.json", + "job_name": "dev12-fix-code-vulnerability-baseline-20261011", + "config_sha256": "4b2de43b547ddd92d748f7658c44b98b4fef25efd406710ace6d1713b9c789a7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/nginx-request-logging", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/nginx-request-logging.json", + "job_name": "dev12-nginx-request-logging-baseline-20261011", + "config_sha256": "4a79d4fc4431acd942548f36188c1820ac5406fc6a1385cf505daf02aa190da3" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/git-multibranch", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/git-multibranch.json", + "job_name": "dev12-git-multibranch-baseline-20261011", + "config_sha256": "ae9c6d5830c9dcb8aab8b1356441b47731957e2ad5e5747327b7caa1a6c037a1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/db-wal-recovery", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/db-wal-recovery.json", + "job_name": "dev12-db-wal-recovery-baseline-20261011", + "config_sha256": "475b7df31549e51f04f4f9171332da5a8ba03f7fbec2b3788fbf897340c0c91b" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/multi-source-data-merger", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/multi-source-data-merger.json", + "job_name": "dev12-multi-source-data-merger-baseline-20261011", + "config_sha256": "5eac4294149a11f037f732a32e1a8874613d8a28884843955481a9e8ef10b7bd" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/query-optimize", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/query-optimize.json", + "job_name": "dev12-query-optimize-baseline-20261011", + "config_sha256": "29ad0478282cd6fb86a1b209a93ffe33eee724c321916732db001b7beef8ea7e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/adaptive-rejection-sampler", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/adaptive-rejection-sampler.json", + "job_name": "dev12-adaptive-rejection-sampler-baseline-20261011", + "config_sha256": "e0fec8e3f1f8e077a5384058ea5b6e4733973a7475b02556f90cfc0f4ba96f6d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-cli", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/pytorch-model-cli.json", + "job_name": "dev12-pytorch-model-cli-baseline-20261011", + "config_sha256": "bc915a7de1286954473271e2232f9c83af983c23b885d099316b33dc3c1d445d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/constraints-scheduling", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/constraints-scheduling.json", + "job_name": "dev12-constraints-scheduling-baseline-20261011", + "config_sha256": "33813c2fd5b45efd3b7bf8a765fcb1f74c702821a3d4f473e4b479200433734e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/custom-memory-heap-crash", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/custom-memory-heap-crash.json", + "job_name": "dev12-custom-memory-heap-crash-baseline-20261011", + "config_sha256": "4678ab6961d82a2d597911bd8d4a53040a4be526c1e97afb27c614ecedb1c05f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-pov-ray", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/build-pov-ray.json", + "job_name": "dev12-build-pov-ray-baseline-20261011", + "config_sha256": "42e1bb7d2a1a6d5cf9f059a121140f9135d8b4632482a00d767a9aba2e1ac2e7" + } + ], + "log_dir": "jobs/dev12-baseline-20261011-orchestration", + "budget_policy": "Native task agent timeout minus 180 seconds for runtime work; setup separately allows 1800 seconds. Native verifier limits remain unchanged. Build POV-Ray retains its unusually long 12000-second native limit (11820-second model work).", + "execution_policy": "One run per selected task. Run all twelve installation preflights before models. Queue is fixed by ascending native work budget, stable within ties; run consecutive pairs with at most two concurrent model trials.", + "failure_policy": "Retain every selected trial including setup failures, verifier errors and scored failures; no outcome-based substitutions or model reruns. Only verifier timeouts may retry, at most three total attempts on the same output with the native per-attempt limit.", + "preflight_policy": "All selected verifier dependency profiles are not_configured. Agent installation success does not certify verifier dependencies. Registered profiles, if present, must pass before model work.", + "information_policy": "Selection used task names before downloading. Only public metadata/instructions may be inspected before every agent output freezes; task files are hashed as bytes without inspecting hidden test or solution contents. Audit external references afterwards.", + "scope": "Development baseline for twelve new tasks, not a representative full-dataset score or a pass-rate improvement experiment. Existing 22 task records remain separate; remaining 55 tasks are not opened." +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/fix-code-vulnerability.json b/benchmarks/harbor/configs/dev12-baseline-20261011/fix-code-vulnerability.json new file mode 100644 index 0000000..9e19897 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/fix-code-vulnerability.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/git-multibranch.json b/benchmarks/harbor/configs/dev12-baseline-20261011/git-multibranch.json new file mode 100644 index 0000000..5956b98 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/git-multibranch.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/install-only.json b/benchmarks/harbor/configs/dev12-baseline-20261011/install-only.json new file mode 100644 index 0000000..04d81dc --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/install-only.json @@ -0,0 +1,66 @@ +{ + "n_concurrent_trials": 2, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + }, + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/main-prompt.txt b/benchmarks/harbor/configs/dev12-baseline-20261011/main-prompt.txt new file mode 100644 index 0000000..56116f5 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/main-prompt.txt @@ -0,0 +1,9 @@ +You are nanoPyCodeAgent, a coding agent running non-interactively on a task handed to you up front. There is no user to reply to you: never ask a clarifying question, never stop to wait for confirmation, and never present a plan for approval — decide on your own and carry it out. Work the task through to the end, then check the result with the tools instead of assuming it worked. When it is done, answer with a short summary and no further tool calls: that reply is what ends the run. Runtime budget reminders report remaining time and model replies; when either is running low, prioritize saving the required deliverables, necessary verification, and a final summary. Do not start optional work after the requirements are satisfied. + +Identify the required deliverables, interfaces, and constraints. Treat examples as evidence, not as the full contract unless the task says so. Inspect relevant files, dependencies, and existing checks instead of guessing. Task-supplied workspace files, including source, executables, and tests, are available references unless the task restricts their use. Consult general documentation as needed. Do not seek or use task-specific reference solutions or hidden evaluation tests from outside the supplied materials unless the task explicitly requests them. Derive checks from the stated requirements. + +Verify the requested behavior through the real entry point when feasible. Check the content and meaning of outputs, not only command success or file existence. For logic or numerical work, choose a few representative and boundary cases justified by the task, using an independent reference, invariant, or consistency check when available. Do not derive every expectation from the implementation being tested or weaken requirements to make a check pass. Keep verification proportional to the task and remaining budget. + +When a check fails, inspect the error and relevant output, revise the underlying assumption or implementation, and rerun the affected check. Keep the task's constraints intact. Once appropriate checks support the required behavior, save the deliverables and finish. State what was verified and any remaining limitation without claiming unobserved success. + +Prefer the read tool for viewing files, the edit tool for changing part of an existing file, and the write tool for creating files or rewriting them whole. Use the bash tool to run commands, search with grep, and complete tasks that need real command output instead of guessing. \ No newline at end of file diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/multi-source-data-merger.json b/benchmarks/harbor/configs/dev12-baseline-20261011/multi-source-data-merger.json new file mode 100644 index 0000000..6c55a0a --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/multi-source-data-merger.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/nginx-request-logging.json b/benchmarks/harbor/configs/dev12-baseline-20261011/nginx-request-logging.json new file mode 100644 index 0000000..be8ea55 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/nginx-request-logging.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/pytorch-model-cli.json b/benchmarks/harbor/configs/dev12-baseline-20261011/pytorch-model-cli.json new file mode 100644 index 0000000..0b76cb3 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/pytorch-model-cli.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/query-optimize.json b/benchmarks/harbor/configs/dev12-baseline-20261011/query-optimize.json new file mode 100644 index 0000000..35c2998 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/query-optimize.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/selection.json b/benchmarks/harbor/configs/dev12-baseline-20261011/selection.json new file mode 100644 index 0000000..655c187 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/selection.json @@ -0,0 +1,96 @@ +{ + "schema_version": 1, + "selected_at": "2026-10-11T14:32:49.067833+00:00", + "main_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "dataset_name": "terminal-bench/terminal-bench-2-1", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "catalog_sha256": "7fc7d95a2077ede76c4f47890a09de93c2fe558252a122ef830288ad293ceaaa", + "selection_method": "Purposive coverage of twelve public task-name categories before downloading task instructions or seeing any baseline outcomes. This is a development cohort, not a random representative benchmark sample. Retain all selected tasks and every failure; do not replace based on outcomes.", + "prior_experiment_task_names": [ + "terminal-bench/build-cython-ext", + "terminal-bench/caffe-cifar-10", + "terminal-bench/cancel-async-tasks", + "terminal-bench/circuit-fibsqrt", + "terminal-bench/dna-assembly", + "terminal-bench/kv-store-grpc", + "terminal-bench/llm-inference-batching-scheduler", + "terminal-bench/log-summary-date-ranges", + "terminal-bench/merge-diff-arc-agi-task", + "terminal-bench/model-extraction-relu-logits", + "terminal-bench/mteb-leaderboard", + "terminal-bench/openssl-selfsigned-cert", + "terminal-bench/path-tracing", + "terminal-bench/pypi-server", + "terminal-bench/pytorch-model-recovery", + "terminal-bench/qemu-alpine-ssh", + "terminal-bench/regex-chess", + "terminal-bench/regex-log", + "terminal-bench/schemelike-metacircular-eval", + "terminal-bench/torch-pipeline-parallelism", + "terminal-bench/torch-tensor-parallelism", + "terminal-bench/write-compressor" + ], + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "selection_rationale": "Native compilation and build tooling" + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "selection_rationale": "Legacy-language migration and behavioral compatibility" + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "selection_rationale": "Native memory management debugging" + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "selection_rationale": "Security bug repair and behavior preservation" + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "selection_rationale": "Service configuration and observable request behavior" + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "selection_rationale": "Cross-branch repository operations" + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "selection_rationale": "Database recovery and data integrity" + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "selection_rationale": "Data integration and cross-input consistency" + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "selection_rationale": "Database query performance and correctness" + }, + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "selection_rationale": "Numerical sampling and statistical invariants" + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "selection_rationale": "Constrained algorithmic planning" + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "selection_rationale": "Model packaging and command-line contracts" + } + ], + "instruction_or_hidden_test_review_before_selection": false +} diff --git a/benchmarks/harbor/scripts/run_comparison.py b/benchmarks/harbor/scripts/run_comparison.py new file mode 100644 index 0000000..9266f1f --- /dev/null +++ b/benchmarks/harbor/scripts/run_comparison.py @@ -0,0 +1,155 @@ +"""Run a frozen comparison plan through the standard Harbor entry point. + +This orchestrates existing repository worktrees; it does not install dependencies, +modify task environments, implement an adapter, or retry a trial. +""" + +import argparse +import ast +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime, timezone +import hashlib +import json +from pathlib import Path +import subprocess + + +def digest(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def prompt_from_source(path): + values = {} + for node in ast.parse(path.read_text()).body: + if isinstance(node, ast.Assign) and len(node.targets) == 1: + target = node.targets[0] + if isinstance(target, ast.Name) and target.id in { + "_TOOL_GUIDANCE", "HEADLESS_SYSTEM_PROMPT", + }: + values[target.id] = eval( + compile(ast.Expression(node.value), str(path), "eval"), + {"__builtins__": {}}, values, + ) + return values["HEADLESS_SYSTEM_PROMPT"] + + +def validate_plan(root, plan_path, plan): + jobs = plan["preflight"] + plan["queue"] + names = [job["job_name"] for job in jobs] + if len(names) != len(set(names)): + raise ValueError("Job names must be unique") + if len(plan["queue"]) != plan["total_model_runs"]: + raise ValueError("Model run count does not match the registered queue") + if plan["max_concurrent_trials"] != 2 or plan["whole_trial_retries"] != 0: + raise ValueError("This runner requires concurrency two and no trial retries") + for arm in plan["arms"].values(): + source = root / arm["source_root"] + actual_ref = subprocess.check_output( + ["git", "rev-parse", "HEAD"], cwd=source, text=True, + ).strip() + if actual_ref != arm["source_ref"]: + raise ValueError(f"Source revision changed: {source}") + subprocess.run( + ["git", "diff", "--exit-code", "HEAD", "--", "src", "benchmarks/harbor/src"], + cwd=source, check=True, stdout=subprocess.DEVNULL, + ) + expected = plan_path.parent / arm["prompt_file"] + if digest(expected) != arm["prompt_sha256"]: + raise ValueError(f"Prompt file changed: {expected}") + if prompt_from_source(source / "src/nanopycodeagent/agent.py") != expected.read_text(): + raise ValueError(f"Source prompt does not match the plan: {source}") + for task in plan["tasks"]: + cache = Path.home() / ".cache/harbor/tasks/packages" / task["name"] / task["ref"].split(":")[1] + for name, expected in task["task_file_sha256"].items(): + if digest(cache / name) != expected: + raise ValueError(f"Task file changed: {cache / name}") + for job in jobs: + config_path = root / job["config"] + if digest(config_path) != job["config_sha256"]: + raise ValueError(f"Configuration changed: {config_path}") + config = json.loads(config_path.read_text()) + limit = 1 if job in plan["queue"] else 2 + if config["n_concurrent_trials"] > limit or config["n_attempts"] != 1: + raise ValueError("Per-job concurrency or attempts exceed the plan") + + +def run_job(root, plan, job, record, install_only): + status_path = record / (job["job_name"] + ".json") + if status_path.exists(): + previous = json.loads(status_path.read_text()) + if previous["status"] == "finished": + return previous + raise RuntimeError(f"Previously started job requires inspection; refusing to retry: {status_path}") + source = root / plan["arms"][job["arm"]]["source_root"] + command = [ + "uv", "run", "--project", "benchmarks/harbor", "python", "-m", "harbor_adapter", + "run", "--config", str(root / job["config"]), "--job-name", job["job_name"], + "--cache-dir", str(root / ".cache/nanopy-harbor"), + ] + if install_only: + command.append("--install-only") + summary_path = source / "jobs" / (job["job_name"] + "-input") / "summary.json" + state = { + **job, "status": "started", "started_at": datetime.now(timezone.utc).isoformat(), + "command": command, "cwd": str(source), "summary_path": str(summary_path), + } + status_path.write_text(json.dumps(state, indent=2) + "\n") + with (record / (job["job_name"] + ".log")).open("w") as log: + code = subprocess.run(command, cwd=source, stdout=log, stderr=subprocess.STDOUT).returncode + state.update(status="finished", exit_code=code, finished_at=datetime.now(timezone.utc).isoformat()) + if summary_path.exists(): + state["summary"] = json.loads(summary_path.read_text()) + status_path.write_text(json.dumps(state, indent=2) + "\n") + print(json.dumps({"job": job["job_name"], "exit_code": code, "status": state["status"]}), flush=True) + return state + + +def preflight_passed(states, expected_counts): + return bool(states) and len(states) == len(expected_counts) and all( + s.get("exit_code") == 0 + and s.get("summary", {}).get("trials") + and len(s["summary"]["trials"]) == expected + and all(t["status"] == "setup_passed" for t in s["summary"]["trials"]) + for s, expected in zip(states, expected_counts) + ) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--plan", type=Path, required=True) + parser.add_argument("--phase", choices=["preflight", "model"], required=True) + args = parser.parse_args() + root, plan_path = Path.cwd(), args.plan.resolve() + plan = json.loads(plan_path.read_text()) + validate_plan(root, plan_path, plan) + record = root / plan["log_dir"] + record.mkdir(parents=True, exist_ok=True) + record.chmod(0o700) + registration = record / "registration.json" + current = {"plan_sha256": digest(plan_path), "plan": plan} + if registration.exists(): + if json.loads(registration.read_text()) != current: + raise ValueError("The registered plan is immutable") + else: + registration.write_text(json.dumps(current, indent=2) + "\n") + expected_counts = [ + len(json.loads((root / job["config"]).read_text())["tasks"]) + for job in plan["preflight"] + ] + if args.phase == "preflight": + states = [run_job(root, plan, j, record, True) for j in plan["preflight"]] + return 0 if preflight_passed(states, expected_counts) else 1 + preflight = [json.loads((record / (j["job_name"] + ".json")).read_text()) for j in plan["preflight"]] + if not preflight_passed(preflight, expected_counts): + raise RuntimeError("Install-only checks must finish successfully before model work") + states = [] + with ThreadPoolExecutor(max_workers=2) as pool: + for offset in range(0, len(plan["queue"]), 2): + batch = plan["queue"][offset:offset + 2] + futures = [pool.submit(run_job, root, plan, job, record, False) for job in batch] + states.extend(f.result() for f in futures) + return int(any(s["exit_code"] != 0 for s in states)) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/harbor/tests/test_comparison.py b/benchmarks/harbor/tests/test_comparison.py new file mode 100644 index 0000000..61faa56 --- /dev/null +++ b/benchmarks/harbor/tests/test_comparison.py @@ -0,0 +1,57 @@ +"""Comparison orchestration must preserve failed trials and preflight gates.""" + +import importlib.util +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest + + +spec = importlib.util.spec_from_file_location( + "run_comparison", Path(__file__).parents[1] / "scripts/run_comparison.py", +) +comparison = importlib.util.module_from_spec(spec) +spec.loader.exec_module(comparison) + + +@pytest.mark.parametrize("states", [ + [], [{"exit_code": 0}], + [{"exit_code": 0, "summary": {"trials": []}}], + [{"exit_code": 1, "summary": {"trials": [{"status": "setup_passed"}]}}], + [{"exit_code": 0, "summary": {"trials": [{"status": "setup_failed"}]}}], +]) +def test_incomplete_or_failed_preflight_cannot_start_models(states): + assert not comparison.preflight_passed(states, [1] * len(states)) + + +def test_successful_installation_does_not_require_a_registered_profile(): + state = {"exit_code": 0, "summary": {"trials": [{ + "status": "setup_passed", "bootstrap": {"verifier_preflight": {"status": "not_configured"}}, + }]}} + assert comparison.preflight_passed([state], [1]) + assert not comparison.preflight_passed([state], [4]) + assert not comparison.preflight_passed([state], [1, 1]) + + +def test_failed_job_is_recorded_and_never_retried(tmp_path, monkeypatch): + plan = {"arms": {"control": {"source_root": "source"}}} + job = {"arm": "control", "config": "config.json", "job_name": "test-job"} + calls = [] + def run(command, **kwargs): + calls.append(command) + return SimpleNamespace(returncode=1) + monkeypatch.setattr(comparison.subprocess, "run", run) + first = comparison.run_job(tmp_path, plan, job, tmp_path, False) + second = comparison.run_job(tmp_path, plan, job, tmp_path, False) + assert first == second + assert first["exit_code"] == 1 + assert len(calls) == 1 + assert calls[0][:8] == ["uv", "run", "--project", "benchmarks/harbor", "python", "-m", "harbor_adapter", "run"] + + +def test_interrupted_job_cannot_silently_restart(tmp_path): + job = {"job_name": "interrupted"} + (tmp_path / "interrupted.json").write_text(json.dumps({"status": "started"})) + with pytest.raises(RuntimeError, match="refusing to retry"): + comparison.run_job(tmp_path, {}, job, tmp_path, False) From d7dbcec4fa29c191b80453e8d61f2e1ec4c8090d Mon Sep 17 00:00:00 2001 From: minixalpha Date: Sun, 11 Oct 2026 08:20:11 -0700 Subject: [PATCH 02/15] fix(benchmarks): cache original images with verified range downloads --- benchmarks/harbor/README.md | 27 +++ benchmarks/harbor/README.zh-CN.md | 22 ++ benchmarks/harbor/scripts/cache_image.py | 236 ++++++++++++++++++++ benchmarks/harbor/tests/test_image_cache.py | 154 +++++++++++++ 4 files changed, 439 insertions(+) create mode 100644 benchmarks/harbor/scripts/cache_image.py create mode 100644 benchmarks/harbor/tests/test_image_cache.py diff --git a/benchmarks/harbor/README.md b/benchmarks/harbor/README.md index 3b841aa..056f454 100644 --- a/benchmarks/harbor/README.md +++ b/benchmarks/harbor/README.md @@ -89,6 +89,33 @@ when it runs; a successful preflight is not a guarantee against later outages. Full standard runs enable the timeout retry policy below by default, with zero whole-trial retries. CI checks these contracts without model calls or Docker. +### Cache original task images + +If Docker image pulls repeatedly stall, the optional cache utility downloads the +same public Docker Hub image in bounded parallel HTTP ranges: + +```bash +python benchmarks/harbor/scripts/cache_image.py \ + alexgshaw/git-multibranch:20251031 \ + --cache .cache/nanopy-harbor/images --load +``` + +The first request records the registry manifest digest; later requests reject a +changed tag. Every downloaded blob must match its size and SHA256, and each +uncompressed layer must match the original image configuration. The resulting +[OCI archive](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) +preserves the original manifest, configuration, and compressed layers byte for +byte. Docker import must retain the original repository digest. + +This Linux utility supports public single-platform `linux/amd64` images with +gzip layers and SHA256 descriptors. `--load` requires Docker's containerd image +store. Use one cache writer at a time; `--workers` controls parallel range +requests (default eight, maximum sixteen). Completed blobs and partial ranges +can be reused after an interrupted download, with complete blobs revalidated. +Keep the manifest lock records with the run evidence. This only warms Docker's +image cache: continue using the standard benchmark entry point and installation +preflight, and retain any earlier environment-start failures separately. + ### Direct Harbor invocation For adapter development or comparison with published versions, the lower-level diff --git a/benchmarks/harbor/README.zh-CN.md b/benchmarks/harbor/README.zh-CN.md index 12ca806..0497dac 100644 --- a/benchmarks/harbor/README.zh-CN.md +++ b/benchmarks/harbor/README.zh-CN.md @@ -71,6 +71,28 @@ uv 0.9.5、Python 3.13、pytest 8.4.1 和 pytest-json-ctrf 0.3.5，不执行题 通过不能保证后续没有网络故障。完整标准运行默认启用下述验证超时重试，整题重试 次数为零。CI 会验证这些约定，不调用模型、不启动 Docker。 +### 缓存原始任务镜像 + +如果 Docker 拉取镜像反复停滞，可用缓存工具从同一 Docker Hub 公共仓库分段并行下载： + +```bash +python benchmarks/harbor/scripts/cache_image.py \ + alexgshaw/git-multibranch:20251031 \ + --cache .cache/nanopy-harbor/images --load +``` + +首次请求记录仓库的 manifest 摘要；后续请求若发现 tag 指向的摘要改变，会拒绝继续。 +每个下载对象都必须匹配大小和 SHA256，每层解压后的摘要也必须匹配原始镜像配置。 +生成的 [OCI 归档](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) +逐字节保留原始 manifest、配置和压缩层；Docker 导入后必须保留原始仓库摘要。 + +该 Linux 工具支持使用 gzip 层和 SHA256 描述符的公共单平台 `linux/amd64` 镜像。 +`--load` 需要 Docker 的 containerd 镜像存储。同一缓存目录同时只允许一个写入进程； +`--workers` 控制分段请求并发数，默认 8、最多 16。下载中断后可复用已完成的对象和分段， +完整对象会重新校验。请将 manifest 锁定记录纳入实验的证据归档。 +这个步骤只准备 Docker 镜像缓存；随后仍需使用标准 benchmark 入口并执行安装预检， +之前出现的环境启动失败应单独保留。 + ### 直接调用 Harbor 开发 adapter 或对照已发布版本时，仍可使用底层 Harbor 入口，并固定容器内 agent 版本： diff --git a/benchmarks/harbor/scripts/cache_image.py b/benchmarks/harbor/scripts/cache_image.py new file mode 100644 index 0000000..062bb1d --- /dev/null +++ b/benchmarks/harbor/scripts/cache_image.py @@ -0,0 +1,236 @@ +"""Cache an unchanged public Docker Hub image through verified HTTP ranges. + +For slow or interrupted Docker pulls, create an OCI archive, then optionally +load it into Docker's containerd image store. No layers are edited or rebuilt. +Only single-platform Linux images with SHA256 and gzip layers are supported. +""" + +import argparse +from concurrent.futures import ThreadPoolExecutor +import fcntl +import gzip +import hashlib +from http.client import IncompleteRead +import io +import json +from pathlib import Path +import re +import subprocess +import tarfile +import time +from urllib.error import URLError +from urllib.parse import urlsplit +from urllib.request import HTTPRedirectHandler, Request, build_opener + + +MANIFEST_TYPES = ( + "application/vnd.docker.distribution.manifest.v2+json", + "application/vnd.oci.image.manifest.v1+json", +) +LAYER_TYPES = { + "application/vnd.docker.image.rootfs.diff.tar.gzip", + "application/vnd.oci.image.layer.v1.tar+gzip", +} + + +class SafeRedirect(HTTPRedirectHandler): + def redirect_request(self, request, response, code, message, headers, url): + if urlsplit(url).scheme != "https": + raise ValueError("Registry redirects must retain HTTPS") + redirected = super().redirect_request(request, response, code, message, headers, url) + if urlsplit(url).netloc != urlsplit(request.full_url).netloc: + redirected.remove_header("Authorization") + return redirected + + +def get(url, headers=None, limit=4 * 1024 * 1024): + # Never include exception URLs: blob redirects may contain signed queries. + for attempt in range(3): + try: + with build_opener(SafeRedirect()).open(Request(url, headers=headers or {}), timeout=30) as r: + data = r.read(limit + 1) + if len(data) > limit: + raise ValueError("Response exceeds requested size") + return r.status, dict(r.headers.items()), data + except (URLError, TimeoutError, OSError, IncompleteRead) as error: + if attempt == 2: + raise RuntimeError(f"Registry request failed: {type(error).__name__}") from None + time.sleep(attempt + 1) + + +def sha(data): + return "sha256:" + hashlib.sha256(data).hexdigest() + + +def blob_path(cache, descriptor): + digest = descriptor["digest"] + if not re.fullmatch(r"sha256:[0-9a-f]{64}", digest): + raise ValueError("Only SHA256 descriptors are supported") + if not isinstance(descriptor["size"], int) or descriptor["size"] < 0: + raise ValueError("Invalid descriptor size") + return cache / "blobs/sha256" / digest.split(":")[1] + + +def verify(path, descriptor): + if path.stat().st_size != descriptor["size"]: + raise ValueError("Blob size mismatch") + with path.open("rb") as stream: + actual = "sha256:" + hashlib.file_digest(stream, "sha256").hexdigest() + if actual != descriptor["digest"]: + raise ValueError("Blob SHA256 mismatch") + + +def fetch_blob(cache, descriptor, url, headers, workers=8, chunk_size=1024 * 1024): + path = blob_path(cache, descriptor) + if path.exists(): + verify(path, descriptor) + return path + path.parent.mkdir(parents=True, exist_ok=True) + parts = cache / "parts" / path.name + parts.mkdir(parents=True, exist_ok=True) + size = descriptor["size"] + + def fetch(start): + end = min(start + chunk_size, size) - 1 + part = parts / str(start) + if part.exists() and part.stat().st_size == end - start + 1: + return part + for attempt in range(3): + status, response_headers, data = get( + url, {**headers, "Range": f"bytes={start}-{end}"}, end - start + 1, + ) + response_headers = {k.lower(): v for k, v in response_headers.items()} + complete = start == 0 and end == size - 1 and status == 200 + expected_range = f"bytes {start}-{end}/{size}" + if not complete and (status != 206 or response_headers.get("content-range") != expected_range): + raise ValueError("Server did not return the exact requested range") + if len(data) == end - start + 1: + break + else: + raise ValueError("Incomplete range response") + temporary = part.with_suffix(".tmp") + temporary.write_bytes(data) + temporary.replace(part) + return part + + with ThreadPoolExecutor(max_workers=workers) as pool: + downloaded = list(pool.map(fetch, range(0, size, chunk_size))) + temporary = path.with_suffix(".tmp") + with temporary.open("wb") as out: + for part in downloaded: + out.write(part.read_bytes()) + verify(temporary, descriptor) + temporary.replace(path) + for part in downloaded: + part.unlink() + parts.rmdir() + return path + + +def validate_layers(cache, manifest): + config = json.loads(blob_path(cache, manifest["config"]).read_bytes()) + if config.get("os") != "linux" or config.get("architecture") != "amd64": + raise ValueError("Only Linux amd64 images are supported") + diff_ids = config["rootfs"]["diff_ids"] + if config["rootfs"].get("type") != "layers": + raise ValueError("Unsupported root filesystem configuration") + if len(diff_ids) != len(manifest["layers"]): + raise ValueError("Layer count differs from image configuration") + for layer, expected in zip(manifest["layers"], diff_ids): + if layer["mediaType"] not in LAYER_TYPES: + raise ValueError("Only gzip layers are supported") + with gzip.open(blob_path(cache, layer), "rb") as stream: + actual = "sha256:" + hashlib.file_digest(stream, "sha256").hexdigest() + if actual != expected: + raise ValueError("Uncompressed layer SHA256 differs from image configuration") + return config + + +def write_archive(cache, image, raw_manifest, output): + manifest = json.loads(raw_manifest) + if manifest.get("schemaVersion") != 2 or manifest.get("mediaType") not in MANIFEST_TYPES: + raise ValueError("Only single-platform schema 2 image manifests are supported") + descriptors = [manifest["config"], *manifest["layers"]] + for descriptor in descriptors: + verify(blob_path(cache, descriptor), descriptor) + validate_layers(cache, manifest) + manifest_descriptor = {"mediaType": manifest["mediaType"], "size": len(raw_manifest), "digest": sha(raw_manifest)} + manifest_path = blob_path(cache, manifest_descriptor) + manifest_path.write_bytes(raw_manifest) + descriptors.append(manifest_descriptor) + index = {"schemaVersion": 2, "manifests": [{ + **manifest_descriptor, "annotations": { + "io.containerd.image.name": "docker.io/" + image, + "org.opencontainers.image.ref.name": image.rsplit(":", 1)[1], + }, + }]} + temporary = output.with_suffix(".tmp") + with tarfile.open(temporary, "w") as archive: + for name, data in ( + ("oci-layout", b'{"imageLayoutVersion":"1.0.0"}'), + ("index.json", json.dumps(index).encode()), + ): + entry = tarfile.TarInfo(name) + entry.size = len(data) + archive.addfile(entry, io.BytesIO(data)) + seen = set() + for descriptor in descriptors: + path = blob_path(cache, descriptor) + if path not in seen: + archive.add(path, arcname=str(path.relative_to(cache)), recursive=False) + seen.add(path) + temporary.replace(output) + + +def cache_image(image, cache, workers=8, load=False): + if not re.fullmatch(r"[a-z0-9_-]+/[a-z0-9._-]+:[A-Za-z0-9._-]+", image): + raise ValueError("Expected a public Docker Hub owner/repository:tag") + cache.mkdir(parents=True, exist_ok=True) + repo, tag = image.rsplit(":", 1) + _, _, raw_token = get("https://auth.docker.io/token?service=registry.docker.io&scope=repository:" + repo + ":pull") + headers = {"Authorization": "Bearer " + json.loads(raw_token)["token"], "Accept": ", ".join(MANIFEST_TYPES)} + base = "https://registry-1.docker.io/v2/" + repo + _, response_headers, raw = get(base + "/manifests/" + tag, headers) + response_headers = {k.lower(): v for k, v in response_headers.items()} + if response_headers.get("docker-content-digest") != sha(raw): + raise ValueError("Registry manifest digest mismatch") + manifest = json.loads(raw) + if manifest.get("mediaType") not in MANIFEST_TYPES or manifest.get("schemaVersion") != 2: + raise ValueError("Only single-platform schema 2 image manifests are supported") + lock = {"image": image, "manifest_digest": sha(raw), "config_digest": manifest["config"]["digest"]} + lock_path = cache / (image.replace("/", "_").replace(":", "_") + ".json") + if lock_path.exists() and json.loads(lock_path.read_text()) != lock: + raise ValueError("Image tag changed since the first cache request") + lock_path.write_text(json.dumps(lock, indent=2) + "\n") + print(json.dumps(lock), flush=True) + for descriptor in [manifest["config"], *manifest["layers"]]: + fetch_blob(cache, descriptor, base + "/blobs/" + descriptor["digest"], headers, workers) + print(json.dumps({"verified_blob": descriptor["digest"], "bytes": descriptor["size"]}), flush=True) + output = cache / (lock_path.stem + ".oci.tar") + write_archive(cache, image, raw, output) + if load: + driver = subprocess.check_output(["docker", "info", "--format", "{{json .DriverStatus}}"], text=True) + if "io.containerd.snapshotter.v1" not in driver: + raise ValueError("OCI import requires Docker's containerd image store") + subprocess.run(["docker", "load", "--quiet", "--input", str(output)], check=True) + installed = json.loads(subprocess.check_output(["docker", "image", "inspect", image], text=True))[0] + if not any(ref.endswith("@" + lock["manifest_digest"]) for ref in installed["RepoDigests"]): + raise ValueError("Imported image does not retain the original manifest digest") + print(json.dumps({"archive": str(output), "loaded": load, **lock}), flush=True) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("image") + parser.add_argument("--cache", type=Path, required=True) + parser.add_argument("--workers", type=int, choices=range(1, 17), default=8) + parser.add_argument("--load", action="store_true") + args = parser.parse_args() + args.cache.mkdir(parents=True, exist_ok=True) + with (args.cache / ".writer.lock").open("w") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + cache_image(args.image, args.cache, args.workers, args.load) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/harbor/tests/test_image_cache.py b/benchmarks/harbor/tests/test_image_cache.py new file mode 100644 index 0000000..28de6bc --- /dev/null +++ b/benchmarks/harbor/tests/test_image_cache.py @@ -0,0 +1,154 @@ +"""Integrity checks for optional, content-preserving image cache imports.""" + +import gzip +import importlib.util +import io +import json +from pathlib import Path +import tarfile + +import pytest + + +spec = importlib.util.spec_from_file_location( + "cache_image", Path(__file__).parents[1] / "scripts/cache_image.py", +) +cache_image = importlib.util.module_from_spec(spec) +spec.loader.exec_module(cache_image) + + +def descriptor(data, media_type="application/octet-stream"): + return {"digest": cache_image.sha(data), "size": len(data), "mediaType": media_type} + + +def put(cache, data, media_type): + desc = descriptor(data, media_type) + path = cache_image.blob_path(cache, desc) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(data) + return desc + + +def fixture_manifest(cache, diff_id=None): + stream = io.BytesIO() + with tarfile.open(fileobj=stream, mode="w") as archive: + entry = tarfile.TarInfo("example.txt") + entry.size = 5 + archive.addfile(entry, io.BytesIO(b"hello")) + layer = stream.getvalue() + compressed = gzip.compress(layer) + config = json.dumps({ + "os": "linux", "architecture": "amd64", + "rootfs": {"type": "layers", "diff_ids": [diff_id or cache_image.sha(layer)]}, + }).encode() + return json.dumps({ + "schemaVersion": 2, "mediaType": cache_image.MANIFEST_TYPES[0], + "config": put(cache, config, "application/vnd.docker.container.image.v1+json"), + "layers": [put(cache, compressed, "application/vnd.docker.image.rootfs.diff.tar.gzip")], + }).encode() + + +def test_parallel_ranges_reconstruct_original_blob(tmp_path, monkeypatch): + payload = b"0123456789abcdefghij" + requests = [] + + def get(url, headers, limit): + requested = headers["Range"].removeprefix("bytes=") + start, end = map(int, requested.split("-")) + requests.append((start, end)) + return 206, {"Content-Range": f"bytes {start}-{end}/{len(payload)}"}, payload[start:end + 1] + + monkeypatch.setattr(cache_image, "get", get) + path = cache_image.fetch_blob(tmp_path, descriptor(payload), "unused", {}, workers=2, chunk_size=6) + assert path.read_bytes() == payload + assert sorted(requests) == [(0, 5), (6, 11), (12, 17), (18, 19)] + requests.clear() + cache_image.fetch_blob(tmp_path, descriptor(payload), "unused", {}) + assert requests == [] + + +@pytest.mark.parametrize("status,content_range,body", [ + (200, None, b"abc"), + (206, "bytes 3-5/6", b"abc"), + (206, "bytes 0-2/6", b"ab"), +]) +def test_reject_inexact_or_truncated_ranges(tmp_path, monkeypatch, status, content_range, body): + monkeypatch.setattr(cache_image, "get", lambda *args: (status, {"Content-Range": content_range}, body)) + with pytest.raises(ValueError): + cache_image.fetch_blob(tmp_path, descriptor(b"abcdef"), "unused", {}, chunk_size=3) + assert not cache_image.blob_path(tmp_path, descriptor(b"abcdef")).exists() + + +def test_reject_corrupt_complete_download(tmp_path, monkeypatch): + monkeypatch.setattr(cache_image, "get", lambda *args: (200, {}, b"abd")) + with pytest.raises(ValueError, match="SHA256 mismatch"): + cache_image.fetch_blob(tmp_path, descriptor(b"abc"), "unused", {}) + assert not cache_image.blob_path(tmp_path, descriptor(b"abc")).exists() + + +def test_retry_short_transport_response_before_caching(tmp_path, monkeypatch): + replies = iter([(200, {}, b"ab"), (200, {}, b"abc")]) + monkeypatch.setattr(cache_image, "get", lambda *args: next(replies)) + path = cache_image.fetch_blob(tmp_path, descriptor(b"abc"), "unused", {}) + assert path.read_bytes() == b"abc" + + +def test_revalidate_cached_bytes(tmp_path): + desc = put(tmp_path, b"abc", "application/octet-stream") + path = cache_image.blob_path(tmp_path, desc) + path.write_bytes(b"abd") + with pytest.raises(ValueError, match="SHA256 mismatch"): + cache_image.fetch_blob(tmp_path, desc, "unused", {}) + path.write_bytes(b"ab") + with pytest.raises(ValueError, match="size mismatch"): + cache_image.fetch_blob(tmp_path, desc, "unused", {}) + + +def test_archive_preserves_manifest_and_all_blobs(tmp_path): + raw = fixture_manifest(tmp_path) + output = tmp_path / "image.tar" + cache_image.write_archive(tmp_path, "example/task:version", raw, output) + with tarfile.open(output) as archive: + index = json.load(archive.extractfile("index.json")) + reference = index["manifests"][0] + assert reference["digest"] == cache_image.sha(raw) + assert reference["annotations"]["io.containerd.image.name"] == "docker.io/example/task:version" + assert archive.extractfile("blobs/sha256/" + reference["digest"].split(":")[1]).read() == raw + manifest = json.loads(raw) + for desc in [manifest["config"], *manifest["layers"]]: + data = archive.extractfile("blobs/sha256/" + desc["digest"].split(":")[1]).read() + assert cache_image.sha(data) == desc["digest"] + + +def test_reject_mismatched_uncompressed_layer(tmp_path): + raw = fixture_manifest(tmp_path, cache_image.sha(b"different layer")) + with pytest.raises(ValueError, match="Uncompressed layer SHA256"): + cache_image.write_archive(tmp_path, "example/task:version", raw, tmp_path / "image.tar") + assert not (tmp_path / "image.tar").exists() + + +@pytest.mark.parametrize("digest", ["sha256:../../bad", "md5:abcd", "sha256:ABCDEF"]) +def test_reject_unsafe_or_unsupported_blob_paths(tmp_path, digest): + with pytest.raises(ValueError, match="Only SHA256"): + cache_image.blob_path(tmp_path, {"digest": digest, "size": 1}) + + +def test_changed_tag_is_rejected_before_downloading_blobs(tmp_path, monkeypatch): + raw = fixture_manifest(tmp_path) + (tmp_path / "example_task_version.json").write_text(json.dumps({"manifest_digest": "old"})) + replies = iter([ + (200, {}, b'{"token":"anonymous-read-token"}'), + (200, {"Docker-Content-Digest": cache_image.sha(raw)}, raw), + ]) + monkeypatch.setattr(cache_image, "get", lambda *args: next(replies)) + with pytest.raises(ValueError, match="Image tag changed"): + cache_image.cache_image("example/task:version", tmp_path) + + +def test_redirect_does_not_forward_registry_authorization(): + request = cache_image.Request("https://registry-1.docker.io/blob", headers={"Authorization": "Bearer token", "Range": "bytes=0-2"}) + redirected = cache_image.SafeRedirect().redirect_request(request, None, 307, "", {}, "https://cdn.docker.com/blob") + assert not redirected.has_header("Authorization") + assert redirected.get_header("Range") == "bytes=0-2" + with pytest.raises(ValueError, match="HTTPS"): + cache_image.SafeRedirect().redirect_request(request, None, 307, "", {}, "http://cdn.docker.com/blob") From e3a53fb6dcf89adb269b2a950521ce6eb6d4bfaf Mon Sep 17 00:00:00 2001 From: minixalpha Date: Sun, 11 Oct 2026 08:32:44 -0700 Subject: [PATCH 03/15] test(benchmarks): retain failed preflight and register infrastructure retry --- .../configs/dev12-baseline-20261011/README.md | 20 +- .../experiment-v2.json | 519 ++++++++++++++++++ .../preflight-v1-evidence.json | 416 ++++++++++++++ 3 files changed, 951 insertions(+), 4 deletions(-) create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v2.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/preflight-v1-evidence.json diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/README.md b/benchmarks/harbor/configs/dev12-baseline-20261011/README.md index 1e3ebd8..d549a6e 100644 --- a/benchmarks/harbor/configs/dev12-baseline-20261011/README.md +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/README.md @@ -13,8 +13,20 @@ is included. The fixed dataset revision is `sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a`. [Selection](selection.json) records the existing 22 tasks, selected revisions and -rationales. The [registered plan](experiment.json) records runtime, prompt, -configuration and task-file hashes, work limits, preflight and queue order. +rationales. The [original registered plan](experiment.json) records runtime, +prompt, configuration and task-file hashes, work limits, preflight and queue +order. The active [pre-model revision](experiment-v2.json) repeats installation +preflight under a new job name and registration directory after infrastructure +failures; all model configurations, tasks, runtime, budgets and queue entries +remain identical. + +The [first preflight evidence](preflight-v1-evidence.json) retains five successful +installations, six image-pull/environment-start errors, and one agent setup +timeout while `apt-get update` was still downloading package indexes. No models +or official verifiers ran. All twelve original images are now cached, including +six downloaded with the verified image-cache utility. Their original registry +digests are recorded. The second preflight must pass all twelve installations +before any model trial starts; unsupported verifier profiles remain explicit. | Task | Coverage | Work budget (seconds) | | --- | --- | ---: | @@ -53,11 +65,11 @@ credentials through the process environment, never configuration or logs. Set ```bash python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment.json \ + --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v2.json \ --phase preflight python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment.json \ + --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v2.json \ --phase model ``` diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v2.json b/benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v2.json new file mode 100644 index 0000000..b718ea2 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v2.json @@ -0,0 +1,519 @@ +{ + "schema_version": 1, + "registered_at": "2026-10-11T15:30:40.768505+00:00", + "experiment": "dev12-main-baseline-20261011", + "selection_file": "selection.json", + "selection_sha256": "cd51acea6592c01ce9ec8cde34cd46a4965a4164950194f2de7e8ca54649bd29", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "arms": { + "main": { + "source_root": "jobs/dev12-baseline-main-source", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "prompt_file": "main-prompt.txt", + "prompt_sha256": "37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d", + "prompt_words": 385 + } + }, + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "selection_rationale": "Native compilation and build tooling", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 12000, + "work_budget_seconds": 11820, + "verifier_timeout_seconds": 12000.0, + "category": "software-engineering", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/build-pov-ray:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "38380f64bc631916e264f80ce4d37feedbc783310001475b367d03989d114fdf", + "environment/Dockerfile": "44f2b5fc7311d8f336daaaca99dd4216b49355e67f50b494ebacf1b0f7c14e92", + "environment/deps/illum1.pov": "5a1f7f53ee0730c4923fcd21cd4a9c34719024d8557920441551ba7d96e75dc4", + "instruction.md": "56c8ecdd0084a07f8c97986c43af5c37ed03dd5aa994da6735cf59806162d1b2", + "solution/solve.sh": "3da9dbf589656512cd7a7004dc3752d3552a6af939cfe59eff79f91695fa42c7", + "task.toml": "b90a47ec26c1191d3c523cc094ea96b170e42fbfe176ef1f814c57d05390c097", + "tests/reference_illum1.png": "bb084f430bbc9d5b623a656920670dc06cee6d9132a9d550be4a1b99e41ab9e0", + "tests/test.sh": "928d1497ad21aef0c456294e3784f1fddf8493410f23584da89f9e02ec6bd134", + "tests/test_outputs.py": "64d6ce107503714ada346e5606832ee886be9950971cf81523e73d8ba28e8d37" + } + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "selection_rationale": "Legacy-language migration and behavioral compatibility", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "software-engineering", + "declared_difficulty": "easy", + "docker_image": "alexgshaw/cobol-modernization:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "c89318cf59efcc7040e3d946022e2616161b6733931a080cdeea7b1918e7ecc4", + "environment/Dockerfile": "8b175c83fc03ec7d16d6101516bc5100230b592f480e435279f4fe051c9280b2", + "environment/data/ACCOUNTS.DAT": "46f63eb29442ebc6716bcd8a0c36f4367910f2783f6b84a307ed9a61ccd1046e", + "environment/data/BOOKS.DAT": "db04da646207cd87f8e9c5b6f0b57d95953899f3eef22de1a81423f06ef4a9fd", + "environment/data/TRANSACTIONS.DAT": "1fa4cc5420ded94d585c563e8196b85166ebbae2bb8924b20014717fb2fb9ee3", + "environment/src/INPUT.DAT": "52fc984f0cf14af1495285d7efe02b9bae13159eea59255560ee938c4f78afee", + "environment/src/program.cbl": "27cb0e56f65f52fa75f346720b37a180b8e034ebe7d7622b979f195087bd6ebb", + "instruction.md": "a87074f1141b7345394cd88053d0debb0dc8f7de95308882c6538ee9a3b359da", + "solution/solve.sh": "7d4a0504e1686c1790bfbc074d4e598f4c631f54c2bc5f3d5fd786574b4810b9", + "task.toml": "4b620211dc3d6b1b989f0210c5ecd2f606974581c8bce27bd78659d48cf0cfbd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "009180a7092583fc8143bb3ffc3957923aa0d61d0aac3a471f76eeb4aae54a88" + } + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "selection_rationale": "Native memory management debugging", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "category": "debugging", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/custom-memory-heap-crash:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a06ed23a2c467aa36d0d4a88e5c020b1076cc25b8d483517239658e9c596dde9", + "environment/Dockerfile": "6ee81e5a4fa2540443d7f27d577ba840728fc6aab9f301ab7919103673c78080", + "environment/patches/locale_classes.tcc.patch": "16951f4a4df6bf253886ac75f5a39fdbc328e415194493f63cb7589a364aa817", + "environment/patches/locale_init.cc.patch": "e97f133ceebd86f8881f972e0e45b0ec0a13a9bfa428d8032008707683d54292", + "environment/program/main.cpp": "b94538c76887cc2b2ab739b5ac4731cddd0faa8f4a9892a6265c062bbbeede14", + "environment/program/user.cpp": "92b203786fed1bf0b3aa1675c9cd2d82b2651170675064da8ffee73b98ef14f9", + "environment/program/user.h": "f43d155995a56c22b9cd1417daef7c3b5a976cdc662a98eef0feb438ae1eee77", + "instruction.md": "53c68ae14a3c3af9a97c40bfd9bdc30a5d98b37eaa2f69f763475745f70f108d", + "solution/solve.sh": "b0b1d10bea1cd5be14a82b35d522df616df1e67028f90d2a6bef90aaa98511a1", + "task.toml": "ed474be1983e212913aa8866ebb66216acbe36aef2339ac375d232aa46617481", + "tests/test.sh": "70a2b55f0c5757f6b123a216dfd202be4a646680bced28dddcd31fd8959a058f", + "tests/test_outputs.py": "4e1cb33f777fa7fb5e1536f9bea9eace6deccd515c67d4f87570adbe692914ba" + } + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "selection_rationale": "Security bug repair and behavior preservation", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "security", + "declared_difficulty": "hard", + "docker_image": "alexgshaw/fix-code-vulnerability:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "87f28dbad183c3a634887696d328ca68a8a0858f132bc48cb7341c22267d38f7", + "environment/Dockerfile": "0917470986224df4daf4854a71302ae81c73ba5dd6e25be1232d29b758a5b3ab", + "instruction.md": "89b1abdf0af19399f720233b5cdf6b2e8e0ff7b6e3fe3e2a3af0d4d298a8641a", + "solution/solve.sh": "7a0131b369d83ae8546ecc3f9b4e23d38240f1bd1f0941e4430fa85b5182ee03", + "task.toml": "9aa2bd14ce9408289d199d0608bc3fa65b8b5e23374c9641020de62197dbf1e2", + "tests/test.sh": "f1611da693cb4cfc5e8482698c6a1858a5c67220ae9db89b50aed66584e9b17f", + "tests/test_outputs.py": "c006a5de95b69bd78055aa26cc055734dbd50718ede25d331f767bfee4ef7f26" + } + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "selection_rationale": "Service configuration and observable request behavior", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "system-administration", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/nginx-request-logging:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d79997586d1c14ed8dbfea050e3580b25500e04c8345acfd3f294152433c795a", + "environment/Dockerfile": "43003466d89ab83484a3fa618e35b58cdb74a002b1e8716ef30f12b3ac8302d8", + "instruction.md": "26a7ac98aced6107f147206790ada77bbf8ed3c25fb360cf323d5fc6889edf99", + "solution/solve.sh": "2454bbbf7c201a9a778165f286373ba2cb991a0eb1d026a8005d57ae71250dd9", + "task.toml": "b7a78d00be13656f75069197100b7b8c4fd268decf6f78929c853b3222a2729d", + "tests/test.sh": "27761195f53cbfe5941fe39b573631f6c926869e64b7c43220377e1e116528e7", + "tests/test_outputs.py": "045cc716c14efde3b0dcff5fc7c85ec5d18bfc6ce66f8b40a418fa2a3a4acda0" + } + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "selection_rationale": "Cross-branch repository operations", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "system-administration", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/git-multibranch:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3b9473429381d9eacabab498cb4a8915aa2bca24fa857ef862e5991d3f81f113", + "environment/Dockerfile": "88d9a9f4fef1ae7fddf632799dedf5a2fb6489460a93b610c33dd8b122982821", + "environment/default.conf": "a3a4c5d0ac9cfe79b0510e4225c80fa3988c27fbac13622505886bf66449e04a", + "instruction.md": "54014a38d7526d65faeb6c11d9b7ed3d4a5cb5d973d68d3da3b68b8c0289e125", + "solution/solve.sh": "300ca96b79db8d77f14742109aaf76048aafc4644ced750b1bc5c2761e8d0175", + "task.toml": "4b9981604578a31205257cca36a28d0c89c9d8d1058541330acd11d3f8d65f82", + "tests/test.sh": "ddbde3eea6ffa8fda105cbd86597a53bd409f41e13223900aa27e77a84b6b155", + "tests/test_outputs.py": "f03c6867663b8e8fb7899df51ee4df26411fab62ed491753964b801e591d16c5" + } + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "selection_rationale": "Database recovery and data integrity", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "file-operations", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + } + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "selection_rationale": "Data integration and cross-input consistency", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "data-processing", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/multi-source-data-merger:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9bf37162a4c1b88ed93627830bb8512ebae154325d7cb069c883144a0e4c69d6", + "environment/Dockerfile": "c95a40d3d35b473dc15ae1e8784ba7fb1725805c0583fc574dd2764325d75e4f", + "environment/data/source_a/users.json": "75e51e2d3ab0bdba2269d7cf3757576f84f3ee2c13407d2db94289a5958a1778", + "environment/data/source_b/users.csv": "8e87d16dc47d166edd8e56165955bad265b630d53d02762367c816affb5fa760", + "environment/data/source_c/users.parquet": "23d1ea51c4170bdba53febb885a0f495e28de525a52667347b2932136b9a72fe", + "instruction.md": "d1d28bb6e37f7a255a1b7f78aae8591f35c0494f33e4913392424ad954aab944", + "solution/solve.sh": "1b3d0d50e31930d042495d48f4c8e95f1a9c1a979c803b7f7cec651322537e1e", + "task.toml": "ee3964c0ee040f0a0cb121cefefbc15443d127684acb003fa5b61dd42f31c694", + "tests/test.sh": "d5a00e64413c0e1ae452f38a1aa1a47c3ed7b4416bbfccae1a4c053025a336d1", + "tests/test_outputs.py": "a070888930cf829aeca039771260b1386bda990ea805f313468d381ba3608ca5" + } + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "selection_rationale": "Database query performance and correctness", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 1800.0, + "category": "data-science", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/query-optimize:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a82c53e333a0ac0b0b410b5e26cb8e7305dece31db84d340248ffb13d5a45160", + "environment/Dockerfile": "064818771f200e0517d22068f87229a7a95d0a82b0778c84ca7203bf1926a0b8", + "environment/task-deps/my-sql-query.sql": "43382792f0f438625d643e34abe144febd5669518f4ab1ab550a15c6d9314a21", + "instruction.md": "473e12d320d2732f679fb631c2a4dab4a01deea6775686718ae0c417b8a39447", + "solution/solve.sh": "0d95f3cedb287ff9d47c796cbaf812c300d1222c10fa80c49e8acb0e81eb6978", + "task.toml": "6e88f2a86df699594bcb1bf81df4ed694505f79c9420f671335b02524bbe3868", + "tests/golden.sql": "4da1885d56b3ffa7b7b22b422996d018795611f6963499d551582881b74b84fd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2068ae80c57af8d52a7ae510f0666732c057806e556b480970f36f14bae683f6" + } + }, + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "selection_rationale": "Numerical sampling and statistical invariants", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "scientific-computing", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/adaptive-rejection-sampler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "95408217b9c484ffc18a02a91b7706323b6902919c72b24866a32e27a23543e2", + "environment/Dockerfile": "e644a3343e339303230f1441b7383f076d21c69a5a7aa6430b4f8a9d27f15c8b", + "environment/protected.tar.gz.enc": "7fc9bf1347bb97da8dede04b2c58638f225070913e222d76627068aebd1b1df2", + "instruction.md": "e6d28c28acd76bef61817427c74c2a29e3e8d75f535cbd708afcbfd02c54013f", + "solution/solve.sh": "fe443a8082b5183ba1623311ba0ee958888aefa633eff93766b9ae14e9bdd84d", + "task.toml": "6ed7b37f74beb46e9d47b0f5fad53025667c9548baff4e96d467b56d429fab1a", + "tests/test.sh": "38b43560d173cc2b952c3a3e17b8a480216d84e33450515e047bcb0d806b1e0a", + "tests/test_outputs.py": "547dc6e107f034f41703722aeceb6d0236e3fb69116fc3f2fbaba11884de352f" + } + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "selection_rationale": "Constrained algorithmic planning", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 1200, + "work_budget_seconds": 1020, + "verifier_timeout_seconds": 1200.0, + "category": "personal-assistant", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/constraints-scheduling:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d54a690970149e1216bd4fc066b45069c7948c13e3b1ef2a213a74342805f209", + "environment/Dockerfile": "7d9aba072a4a907229728b9739c8cf821fc2c98450babd64452682caf3759c84", + "environment/inputs/alice_calendar.ics": "5a5d45804bffd339c2479d49e76fe817b38244d14d389afe1de583443808ecbc", + "environment/inputs/bob_calendar.ics": "c97e85dbcdd756758f8a80f4db5fd5c1dcacee79298e4d2262f46de5ff680b50", + "environment/inputs/carol_calendar.ics": "32a0f8687637df0a12b8e5c56d49c4192bc3e790856943632d70f237c7c1363b", + "instruction.md": "342def78177f9bc41bea4d11af679bc385a9dda2ac8a1e425be384a5b7f937b6", + "solution/solve.sh": "802d81ec064fad3050bbade95a2f847ac24e0b7f80fa69b0bcee0347dc92fe77", + "task.toml": "7b9842de16d74e8faf8c7c0aa03b0b81145941cb12dd322d81ae23fc48607250", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "1422b345ba3aacdfc4184e962811b6379988435896c50d8cf62903e70990c5d1" + } + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "selection_rationale": "Model packaging and command-line contracts", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "model-training", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/pytorch-model-cli:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "35a093fa5f1592737784ba718c2e3e9ae17ff8b5e53e003fae560ddce091039c", + "environment/Dockerfile": "8fd5c0f21a6914cd440d10c84c9db06fec606f20dc3e0363b17a3b8c220dc4d9", + "environment/task-deps/cJSON.c": "0e7995261640c96db16ed2c91a5d981efa917cf5e72d0f97ec45a1179ae3ad73", + "environment/task-deps/cJSON.h": "0909160541bcc510e0b2d8c7a9cc051fa1459a0f13be4d8e56346a15491f2f2f", + "environment/task-deps/image.png": "f4d5dfefe463939996541404340e7099d5957b56e1df3760d13b5d2de754db53", + "environment/task-deps/lodepng.cpp": "67873c5f6272f779a143a9c81070e8783327c5552d0b9de2aaee2cb779308917", + "environment/task-deps/lodepng.h": "b735b7e0357319973e9488d520d7f270d734a6e9e1bf924fb1d5b8d26d7e97ce", + "environment/task-deps/model.pth": "a32103eef09936e586b1a7dc7e64d279500a38ff30fdd6df2167e28eb954eee6", + "environment/task-deps/model.py": "e4d3a5c728449656d0efde7871ee227619348352769795ed1be1a8b7f835fddc", + "instruction.md": "ce36ee1f6c589f81ee6a6be4967e25c76a2ed697006928023f42dfb9f28a6750", + "solution/solve.sh": "3b34e561c9ef0a53c9948655847846eb4811aaac67b990480aa6aac1b04bfe66", + "task.toml": "fc1c018efbd5d0537ac0741d53d1ebad3886faa51d14cbde8ec953856411bc0d", + "tests/test.sh": "43fb67e537c7adc2df11c2367f47c876cb5d16a7c1ae16d0339c3a3364bb2b06", + "tests/test_outputs.py": "370702c57a4e0c85f70ee37a5c357ae3a5df8480a275bb6263b0376f2f6c093b" + } + } + ], + "total_model_runs": 12, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "preflight": [ + { + "arm": "main", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/install-only.json", + "config_sha256": "b8fc1d1d825cc514c345026bdfce4cd867443125aac0de1d0d6e02b6c3e7c30a", + "job_name": "dev12-main-install-only-v2-20261011" + } + ], + "queue": [ + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cobol-modernization", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/cobol-modernization.json", + "job_name": "dev12-cobol-modernization-baseline-20261011", + "config_sha256": "2680126c2f162d58948cd30565806bf6cb1f026d561c87b942ba4735173ea6a2" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/fix-code-vulnerability", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/fix-code-vulnerability.json", + "job_name": "dev12-fix-code-vulnerability-baseline-20261011", + "config_sha256": "4b2de43b547ddd92d748f7658c44b98b4fef25efd406710ace6d1713b9c789a7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/nginx-request-logging", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/nginx-request-logging.json", + "job_name": "dev12-nginx-request-logging-baseline-20261011", + "config_sha256": "4a79d4fc4431acd942548f36188c1820ac5406fc6a1385cf505daf02aa190da3" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/git-multibranch", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/git-multibranch.json", + "job_name": "dev12-git-multibranch-baseline-20261011", + "config_sha256": "ae9c6d5830c9dcb8aab8b1356441b47731957e2ad5e5747327b7caa1a6c037a1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/db-wal-recovery", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/db-wal-recovery.json", + "job_name": "dev12-db-wal-recovery-baseline-20261011", + "config_sha256": "475b7df31549e51f04f4f9171332da5a8ba03f7fbec2b3788fbf897340c0c91b" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/multi-source-data-merger", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/multi-source-data-merger.json", + "job_name": "dev12-multi-source-data-merger-baseline-20261011", + "config_sha256": "5eac4294149a11f037f732a32e1a8874613d8a28884843955481a9e8ef10b7bd" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/query-optimize", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/query-optimize.json", + "job_name": "dev12-query-optimize-baseline-20261011", + "config_sha256": "29ad0478282cd6fb86a1b209a93ffe33eee724c321916732db001b7beef8ea7e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/adaptive-rejection-sampler", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/adaptive-rejection-sampler.json", + "job_name": "dev12-adaptive-rejection-sampler-baseline-20261011", + "config_sha256": "e0fec8e3f1f8e077a5384058ea5b6e4733973a7475b02556f90cfc0f4ba96f6d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-cli", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/pytorch-model-cli.json", + "job_name": "dev12-pytorch-model-cli-baseline-20261011", + "config_sha256": "bc915a7de1286954473271e2232f9c83af983c23b885d099316b33dc3c1d445d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/constraints-scheduling", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/constraints-scheduling.json", + "job_name": "dev12-constraints-scheduling-baseline-20261011", + "config_sha256": "33813c2fd5b45efd3b7bf8a765fcb1f74c702821a3d4f473e4b479200433734e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/custom-memory-heap-crash", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/custom-memory-heap-crash.json", + "job_name": "dev12-custom-memory-heap-crash-baseline-20261011", + "config_sha256": "4678ab6961d82a2d597911bd8d4a53040a4be526c1e97afb27c614ecedb1c05f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-pov-ray", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/build-pov-ray.json", + "job_name": "dev12-build-pov-ray-baseline-20261011", + "config_sha256": "42e1bb7d2a1a6d5cf9f059a121140f9135d8b4632482a00d767a9aba2e1ac2e7" + } + ], + "log_dir": "jobs/dev12-baseline-20261011-v2-orchestration", + "budget_policy": "Native task agent timeout minus 180 seconds for runtime work; setup separately allows 1800 seconds. Native verifier limits remain unchanged. Build POV-Ray retains its unusually long 12000-second native limit (11820-second model work).", + "execution_policy": "One run per selected task. Run all twelve installation preflights before models. Queue is fixed by ascending native work budget, stable within ties; run consecutive pairs with at most two concurrent model trials.", + "failure_policy": "Retain every selected trial including setup failures, verifier errors and scored failures; no outcome-based substitutions or model reruns. Only verifier timeouts may retry, at most three total attempts on the same output with the native per-attempt limit.", + "preflight_policy": "All selected verifier dependency profiles are not_configured. Agent installation success does not certify verifier dependencies. Registered profiles, if present, must pass before model work.", + "information_policy": "Selection used task names before downloading. Only public metadata/instructions may be inspected before every agent output freezes; task files are hashed as bytes without inspecting hidden test or solution contents. Audit external references afterwards.", + "scope": "Development baseline for twelve new tasks, not a representative full-dataset score or a pass-rate improvement experiment. Existing 22 task records remain separate; remaining 55 tasks are not opened.", + "protocol_amendment": { + "supersedes": "experiment.json", + "supersedes_sha256": "88b2c2391b6c00cbdb03408d8af39d32c0d79266ea2b02913a3537224213c8ab", + "evidence_file": "preflight-v1-evidence.json", + "evidence_sha256": "c723c858571a512e968db16babed482a2874f270561ae3ae8c426a2ad74bd73f", + "reason": "The first install-only job encountered pre-model infrastructure failures recorded in the evidence file. Preserve every failure, cache the exact original images, and repeat all twelve installation checks under a new registration before any model work.", + "changed_fields": [ + "registered_at", + "log_dir", + "preflight[0].job_name", + "protocol_amendment" + ], + "unchanged": "Task selection and revisions, runtime and prompt, model, all task configurations and budgets, model queue and job names, concurrency, retry and information policies.", + "model_runs_started_at_amendment": 0 + } +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/preflight-v1-evidence.json b/benchmarks/harbor/configs/dev12-baseline-20261011/preflight-v1-evidence.json new file mode 100644 index 0000000..167fab3 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/preflight-v1-evidence.json @@ -0,0 +1,416 @@ +{ + "recorded_at": "2026-10-11T15:30:40.768505+00:00", + "original_plan_sha256": "88b2c2391b6c00cbdb03408d8af39d32c0d79266ea2b02913a3537224213c8ab", + "preflight_state_path": "jobs/dev12-baseline-20261011-orchestration/dev12-main-install-only-20261011.json", + "preflight_state_sha256": "dc794453b094c36d3cd28679a006d6178bb61f90d28f875b2957c8fefe0aa2ff", + "preflight_exit_code": 1, + "model_runs_started": 0, + "trials": [ + { + "task": "terminal-bench/adaptive-rejection-sampler", + "trial": "adaptive-rejection-sampler__RF8AnGv", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/adaptive-rejection-sampler__RF8AnGv/result.json", + "result_sha256": "2f2c44870848bfa6155c7d2f31f1c1b2a734ca783e050cd344ecb22262e8125b", + "started_at": "2026-10-11T14:52:09.897066Z", + "finished_at": "2026-10-11T15:29:53.170833Z", + "exception_type": "AgentSetupTimeoutError", + "bootstrap": { + "status": "failed", + "stage": "system_dependencies", + "error_type": "CancelledError", + "cache_used": false, + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "apt_requested_packages": [] + }, + "model_started": false, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/adaptive-rejection-sampler__RF8AnGv/agent/bootstrap.json", + "bootstrap_sha256": "82878d7f8a339f6a19b1130e8160cbd8e23b500246887197f67728670a0e05a8" + }, + { + "task": "terminal-bench/build-pov-ray", + "trial": "build-pov-ray__JBS6iy8", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/build-pov-ray__JBS6iy8/result.json", + "result_sha256": "c878116d4f2746c3269ceefba34b9f0d1c0d9672d3010adb43ff904767391c84", + "started_at": "2026-10-11T14:37:44.785883Z", + "finished_at": "2026-10-11T14:39:13.586321Z", + "exception_type": null, + "bootstrap": { + "status": "ready", + "stage": "complete", + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "cache_used": true, + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "apt_requested_packages": [ + [ + "curl", + "bash", + "git", + "python3", + "ca-certificates" + ] + ] + }, + "model_started": false, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/build-pov-ray__JBS6iy8/agent/bootstrap.json", + "bootstrap_sha256": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4" + }, + { + "task": "terminal-bench/cobol-modernization", + "trial": "cobol-modernization__vX5nBDX", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/cobol-modernization__vX5nBDX/result.json", + "result_sha256": "65726b6e867002ac7e0306d3e1763895076389d164a9b6bd611cab89fc28f021", + "started_at": "2026-10-11T14:37:44.880661Z", + "finished_at": "2026-10-11T14:39:09.924816Z", + "exception_type": null, + "bootstrap": { + "status": "ready", + "stage": "complete", + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "cache_used": true, + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "apt_requested_packages": [ + [ + "curl", + "bash", + "git", + "python3", + "ca-certificates" + ] + ] + }, + "model_started": false, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/cobol-modernization__vX5nBDX/agent/bootstrap.json", + "bootstrap_sha256": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f" + }, + { + "task": "terminal-bench/constraints-scheduling", + "trial": "constraints-scheduling__3qnvwvS", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/constraints-scheduling__3qnvwvS/result.json", + "result_sha256": "f01f3f862c70c246f4f0b1ab0617d7fbd02ffed99df63eeed39b8484485734b7", + "started_at": "2026-10-11T15:02:12.907941Z", + "finished_at": "2026-10-11T15:12:14.134571Z", + "exception_type": "EnvironmentStartTimeoutError", + "bootstrap": null, + "model_started": false + }, + { + "task": "terminal-bench/custom-memory-heap-crash", + "trial": "custom-memory-heap-crash__feSC86P", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/custom-memory-heap-crash__feSC86P/result.json", + "result_sha256": "7d49cf4504bca281dd3115918e3efedea9aea9f4b0bbac63f6a1b33742446b84", + "started_at": "2026-10-11T14:39:12.622251Z", + "finished_at": "2026-10-11T14:41:58.270096Z", + "exception_type": null, + "bootstrap": { + "status": "ready", + "stage": "complete", + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "cache_used": true, + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "apt_requested_packages": [ + [ + "curl", + "bash", + "git", + "python3", + "ca-certificates" + ] + ] + }, + "model_started": false, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/custom-memory-heap-crash__feSC86P/agent/bootstrap.json", + "bootstrap_sha256": "713cf8bea3c485e3e227a8737b6f21b5e1c41cb5b990897d3f5fa7a5445dfae5" + }, + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__Nuxi3FJ", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/db-wal-recovery__Nuxi3FJ/result.json", + "result_sha256": "dfc37eb35223f1ff09cf2a6e6dc47248e4caafb8d5d164461a82404604d4f6ee", + "started_at": "2026-10-11T14:42:03.834053Z", + "finished_at": "2026-10-11T14:52:04.644330Z", + "exception_type": "EnvironmentStartTimeoutError", + "bootstrap": null, + "model_started": false + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "trial": "fix-code-vulnerability__zXDnZib", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/fix-code-vulnerability__zXDnZib/result.json", + "result_sha256": "70179ecad359af42a76481d4099f100e9860605410b724c732e731a8f567d967", + "started_at": "2026-10-11T14:39:14.249670Z", + "finished_at": "2026-10-11T14:40:30.306384Z", + "exception_type": null, + "bootstrap": { + "status": "ready", + "stage": "complete", + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "cache_used": false, + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "apt_requested_packages": [ + [ + "curl", + "bash", + "git", + "python3", + "ca-certificates" + ] + ] + }, + "model_started": false, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/fix-code-vulnerability__zXDnZib/agent/bootstrap.json", + "bootstrap_sha256": "0ebcaf72b1671da138d80d344700a33c9f8cf6364e54193732d28806218c24fc" + }, + { + "task": "terminal-bench/git-multibranch", + "trial": "git-multibranch__vmNqLCo", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/git-multibranch__vmNqLCo/result.json", + "result_sha256": "4390e038e6370493d9fc1b2269f00ad0c40d9da987fed3be056e7ae38241b89a", + "started_at": "2026-10-11T14:42:00.835897Z", + "finished_at": "2026-10-11T14:42:08.522946Z", + "exception_type": "RuntimeError", + "bootstrap": null, + "model_started": false + }, + { + "task": "terminal-bench/multi-source-data-merger", + "trial": "multi-source-data-merger__sSrB3BY", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/multi-source-data-merger__sSrB3BY/result.json", + "result_sha256": "29282679b5ffb1a5f8e6b39b397f18718856bb06224a1735f8003aa0ac0f6148", + "started_at": "2026-10-11T14:42:08.843804Z", + "finished_at": "2026-10-11T14:52:09.624139Z", + "exception_type": "EnvironmentStartTimeoutError", + "bootstrap": null, + "model_started": false + }, + { + "task": "terminal-bench/nginx-request-logging", + "trial": "nginx-request-logging__TtmcRmS", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/nginx-request-logging__TtmcRmS/result.json", + "result_sha256": "21be2091123469444698c92f126a6876ea78a32b3b75f05bcc8e43a33659b383", + "started_at": "2026-10-11T14:40:32.151294Z", + "finished_at": "2026-10-11T14:42:01.242999Z", + "exception_type": null, + "bootstrap": { + "status": "ready", + "stage": "complete", + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "cache_used": true, + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "apt_requested_packages": [ + [ + "curl", + "bash", + "git", + "python3", + "ca-certificates" + ] + ] + }, + "model_started": false, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/nginx-request-logging__TtmcRmS/agent/bootstrap.json", + "bootstrap_sha256": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94" + }, + { + "task": "terminal-bench/pytorch-model-cli", + "trial": "pytorch-model-cli__LSDgEmN", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/pytorch-model-cli__LSDgEmN/result.json", + "result_sha256": "396ffe0341f26fb3b619bc6f8a8421e3eafc343480c7c10b89967feb5c68470b", + "started_at": "2026-10-11T15:12:16.564097Z", + "finished_at": "2026-10-11T15:22:16.649591Z", + "exception_type": "EnvironmentStartTimeoutError", + "bootstrap": null, + "model_started": false + }, + { + "task": "terminal-bench/query-optimize", + "trial": "query-optimize__3swyr89", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/query-optimize__3swyr89/result.json", + "result_sha256": "b0098f4041e7f553f7e036d3e52bfcb34c196bdcb76c09ec54dc60145b2eb1ff", + "started_at": "2026-10-11T14:52:08.733745Z", + "finished_at": "2026-10-11T15:02:08.667707Z", + "exception_type": "EnvironmentStartTimeoutError", + "bootstrap": null, + "model_started": false + } + ], + "original_images_cached_before_second_preflight": [ + { + "task": "terminal-bench/build-pov-ray", + "image": "alexgshaw/build-pov-ray:20251031", + "image_id": "sha256:874baf49341e2853a15a9184e308c7eb632e97f434359fcff2891c6e5ca1b39b", + "repo_digests": [ + "alexgshaw/build-pov-ray@sha256:874baf49341e2853a15a9184e308c7eb632e97f434359fcff2891c6e5ca1b39b" + ], + "size_bytes": 163066497 + }, + { + "task": "terminal-bench/cobol-modernization", + "image": "alexgshaw/cobol-modernization:20251031", + "image_id": "sha256:593ab9df3d83f771e927888b2d9436b380cdf7ec4d0af2a9f0802d541edebba0", + "repo_digests": [ + "alexgshaw/cobol-modernization@sha256:593ab9df3d83f771e927888b2d9436b380cdf7ec4d0af2a9f0802d541edebba0" + ], + "size_bytes": 162020401 + }, + { + "task": "terminal-bench/custom-memory-heap-crash", + "image": "alexgshaw/custom-memory-heap-crash:20251031", + "image_id": "sha256:1895d7d6240d5fb2ec774d6bfdd1057d170f8d199c15c582e4585d3c63802557", + "repo_digests": [ + "alexgshaw/custom-memory-heap-crash@sha256:1895d7d6240d5fb2ec774d6bfdd1057d170f8d199c15c582e4585d3c63802557" + ], + "size_bytes": 1267999268 + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "image": "alexgshaw/fix-code-vulnerability:20251031", + "image_id": "sha256:cac325252991f823713b2d0441502972901dd782bd67f66c03d9b1e410dac5c0", + "repo_digests": [ + "alexgshaw/fix-code-vulnerability@sha256:cac325252991f823713b2d0441502972901dd782bd67f66c03d9b1e410dac5c0" + ], + "size_bytes": 190696453 + }, + { + "task": "terminal-bench/nginx-request-logging", + "image": "alexgshaw/nginx-request-logging:20251031", + "image_id": "sha256:693e64431a0dbf45952e8b91a298d54fd60247e501c6623fb8fa3ebc07fc3d3d", + "repo_digests": [ + "alexgshaw/nginx-request-logging@sha256:693e64431a0dbf45952e8b91a298d54fd60247e501c6623fb8fa3ebc07fc3d3d" + ], + "size_bytes": 67285778 + }, + { + "task": "terminal-bench/git-multibranch", + "image": "alexgshaw/git-multibranch:20251031", + "image_id": "sha256:5bc449921c138295f6ddd8dca2977319fee2ce523811dbe56cec4ca663385fb5", + "repo_digests": [ + "alexgshaw/git-multibranch@sha256:5bc449921c138295f6ddd8dca2977319fee2ce523811dbe56cec4ca663385fb5" + ], + "size_bytes": 222134454 + }, + { + "task": "terminal-bench/db-wal-recovery", + "image": "alexgshaw/db-wal-recovery:20251031", + "image_id": "sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e", + "repo_digests": [ + "alexgshaw/db-wal-recovery@sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e" + ], + "size_bytes": 175276145 + }, + { + "task": "terminal-bench/multi-source-data-merger", + "image": "alexgshaw/multi-source-data-merger:20251031", + "image_id": "sha256:8b32782078ff7383a1b4e5d3cecca8ce287e30f50bb4c0e5db009a18064e666e", + "repo_digests": [ + "alexgshaw/multi-source-data-merger@sha256:8b32782078ff7383a1b4e5d3cecca8ce287e30f50bb4c0e5db009a18064e666e" + ], + "size_bytes": 220880128 + }, + { + "task": "terminal-bench/query-optimize", + "image": "alexgshaw/query-optimize:20251031", + "image_id": "sha256:28de7da87004eab5b29bd5eaf1823579bc5b73ccea4b80349a5623185770da14", + "repo_digests": [ + "alexgshaw/query-optimize@sha256:28de7da87004eab5b29bd5eaf1823579bc5b73ccea4b80349a5623185770da14" + ], + "size_bytes": 120494963 + }, + { + "task": "terminal-bench/adaptive-rejection-sampler", + "image": "alexgshaw/adaptive-rejection-sampler:20251031", + "image_id": "sha256:985bed39e8448bc95f138b5f766b59003a95c4be171c96af8e76448338f3dd8f", + "repo_digests": [ + "alexgshaw/adaptive-rejection-sampler@sha256:985bed39e8448bc95f138b5f766b59003a95c4be171c96af8e76448338f3dd8f" + ], + "size_bytes": 29731756 + }, + { + "task": "terminal-bench/constraints-scheduling", + "image": "alexgshaw/constraints-scheduling:20251031", + "image_id": "sha256:567ce5a189f8d11ac461790876e934cc7af38391baf89a78f95f4dafc1fec3b0", + "repo_digests": [ + "alexgshaw/constraints-scheduling@sha256:567ce5a189f8d11ac461790876e934cc7af38391baf89a78f95f4dafc1fec3b0" + ], + "size_bytes": 49852952 + }, + { + "task": "terminal-bench/pytorch-model-cli", + "image": "alexgshaw/pytorch-model-cli:20251031", + "image_id": "sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "repo_digests": [ + "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67" + ], + "size_bytes": 44349611 + } + ], + "image_cache_utility": { + "path": "benchmarks/harbor/scripts/cache_image.py", + "source_commit": "d7dbcec", + "source_sha256": "639709c03a7b0c3eb6aa78b4e87e6184b49b4b102853a31e33dd832d759180c0", + "unit_and_harbor_tests_passed": 113, + "real_docker_import_digest_check": "passed" + }, + "imported_manifest_locks": [ + { + "image": "alexgshaw/constraints-scheduling:20251031", + "manifest_digest": "sha256:567ce5a189f8d11ac461790876e934cc7af38391baf89a78f95f4dafc1fec3b0", + "config_digest": "sha256:65694556a87deb6f2c0f643841d4d4ff3fabd1e794999978694d891ccb24e703" + }, + { + "image": "alexgshaw/db-wal-recovery:20251031", + "manifest_digest": "sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e", + "config_digest": "sha256:0ff6d2977f94533d8a7a0cdc5a16efd42cb3ccfeb2f5684cdc8db0836b43b9e3" + }, + { + "image": "alexgshaw/git-multibranch:20251031", + "manifest_digest": "sha256:5bc449921c138295f6ddd8dca2977319fee2ce523811dbe56cec4ca663385fb5", + "config_digest": "sha256:ec6eeb02cecf35faf5badd75646ab0dd2e02cab484638b413ccd4606a2e2ef7d" + }, + { + "image": "alexgshaw/multi-source-data-merger:20251031", + "manifest_digest": "sha256:8b32782078ff7383a1b4e5d3cecca8ce287e30f50bb4c0e5db009a18064e666e", + "config_digest": "sha256:5096d3409e5c692afd62a136f90a2e52dd87653aa6637a3e1ca31ad30fab1096" + }, + { + "image": "alexgshaw/pytorch-model-cli:20251031", + "manifest_digest": "sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "config_digest": "sha256:2d19236650cc866ba441762e3d89edfa452756e0e4833cfdb984257d50b446e7" + }, + { + "image": "alexgshaw/query-optimize:20251031", + "manifest_digest": "sha256:28de7da87004eab5b29bd5eaf1823579bc5b73ccea4b80349a5623185770da14", + "config_digest": "sha256:5654e241057913a0f1a234583c62e12ed7bfab560d9c3f0dc5da985d9f3514eb" + } + ], + "first_preflight_summary": { + "setup_passed": 5, + "image_pull_or_environment_start_errors": 6, + "agent_setup_timeouts": 1, + "model_runs_started": 0 + }, + "cleanup": { + "at": "2026-10-11T15:30:36.144215+00:00", + "preflight_job": "dev12-main-install-only-20261011", + "stopped_orphan_compose_clients": [], + "removed_finished_preflight_containers": [], + "model_executions_in_cleaned_trials": 0 + } +} From 491ee92c460772c9cd26204e31e4198c7b9e4948 Mon Sep 17 00:00:00 2001 From: minixalpha Date: Sun, 11 Oct 2026 08:50:15 -0700 Subject: [PATCH 04/15] test(benchmarks): verify reusable installation evidence before baseline --- .../configs/dev12-baseline-20261011/README.md | 37 +- .../experiment-v3.json | 625 ++++++++++++++++++ .../preflight-v2-evidence.json | 206 ++++++ benchmarks/harbor/scripts/run_comparison.py | 74 ++- benchmarks/harbor/tests/test_comparison.py | 106 +++ 5 files changed, 1033 insertions(+), 15 deletions(-) create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json create mode 100644 benchmarks/harbor/configs/dev12-baseline-20261011/preflight-v2-evidence.json diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/README.md b/benchmarks/harbor/configs/dev12-baseline-20261011/README.md index d549a6e..c1ce6c3 100644 --- a/benchmarks/harbor/configs/dev12-baseline-20261011/README.md +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/README.md @@ -15,18 +15,29 @@ is included. The fixed dataset revision is [Selection](selection.json) records the existing 22 tasks, selected revisions and rationales. The [original registered plan](experiment.json) records runtime, prompt, configuration and task-file hashes, work limits, preflight and queue -order. The active [pre-model revision](experiment-v2.json) repeats installation -preflight under a new job name and registration directory after infrastructure -failures; all model configurations, tasks, runtime, budgets and queue entries -remain identical. +order. The active [pre-model revision](experiment-v3.json) accepts twelve +successful native installation trials with hash-bound provenance. All model +configurations, tasks, runtime, budgets and queue entries remain identical. The [first preflight evidence](preflight-v1-evidence.json) retains five successful installations, six image-pull/environment-start errors, and one agent setup timeout while `apt-get update` was still downloading package indexes. No models or official verifiers ran. All twelve original images are now cached, including six downloaded with the verified image-cache utility. Their original registry -digests are recorded. The second preflight must pass all twelve installations -before any model trial starts; unsupported verifier profiles remain explicit. +digests are recorded. + +The [second preflight](experiment-v2.json) completed eleven successful +installations; POV-Ray's uv download ended with a partial-transfer error. +[Its evidence](preflight-v2-evidence.json) retains every result. The final plan +uses these eleven successes and POV-Ray's earlier successful installation, +without replaying successful installations because of another trial's network +failure. Before model work, the orchestrator requires exactly one accepted +installation-only result for every queued task and source arm. It verifies the +task revision, source snapshot, installed wheel, bootstrap status, input mode, +and recorded artifact hashes. A scored trial, failed installation, pending +registered verifier profile, incomplete set, or changed evidence is rejected. +Unsupported verifier profiles remain explicit. Both failed jobs and all eight +failed preflight trials remain in the evidence; none is a scored agent failure. | Task | Coverage | Work budget (seconds) | | --- | --- | ---: | @@ -65,20 +76,24 @@ credentials through the process environment, never configuration or logs. Set ```bash python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v2.json \ + --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json \ --phase preflight python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v2.json \ + --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json \ --phase model ``` The reusable orchestrator calls only the standard `uv run --project benchmarks/harbor python -m harbor_adapter run --config ...` workflow from the frozen repository root. It creates an immutable registration, -requires all twelve install-only results before model work, and refuses to retry -a job already recorded as started. Source snapshots and wheel manifests remain -in the normal Harbor job-input directories. +requires all twelve successful install-only records before model work, and +refuses to retry a job already recorded as started. With the final plan, +`--phase preflight` validates the recorded successful installations without +launching another installation job. Source snapshots and wheel manifests remain +in the normal Harbor job-input directories. A full model trial still installs +the agent in its own fresh container; preflight does not eliminate later +network failures. All twelve selected verifier dependency profiles are explicitly `not_configured`. Agent installation preflight does **not** certify verifier diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json b/benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json new file mode 100644 index 0000000..60f7489 --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json @@ -0,0 +1,625 @@ +{ + "schema_version": 1, + "registered_at": "2026-10-11T15:48:27.483536+00:00", + "experiment": "dev12-main-baseline-20261011", + "selection_file": "selection.json", + "selection_sha256": "cd51acea6592c01ce9ec8cde34cd46a4965a4164950194f2de7e8ca54649bd29", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "arms": { + "main": { + "source_root": "jobs/dev12-baseline-main-source", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "prompt_file": "main-prompt.txt", + "prompt_sha256": "37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d", + "prompt_words": 385 + } + }, + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "selection_rationale": "Native compilation and build tooling", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 12000, + "work_budget_seconds": 11820, + "verifier_timeout_seconds": 12000.0, + "category": "software-engineering", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/build-pov-ray:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "38380f64bc631916e264f80ce4d37feedbc783310001475b367d03989d114fdf", + "environment/Dockerfile": "44f2b5fc7311d8f336daaaca99dd4216b49355e67f50b494ebacf1b0f7c14e92", + "environment/deps/illum1.pov": "5a1f7f53ee0730c4923fcd21cd4a9c34719024d8557920441551ba7d96e75dc4", + "instruction.md": "56c8ecdd0084a07f8c97986c43af5c37ed03dd5aa994da6735cf59806162d1b2", + "solution/solve.sh": "3da9dbf589656512cd7a7004dc3752d3552a6af939cfe59eff79f91695fa42c7", + "task.toml": "b90a47ec26c1191d3c523cc094ea96b170e42fbfe176ef1f814c57d05390c097", + "tests/reference_illum1.png": "bb084f430bbc9d5b623a656920670dc06cee6d9132a9d550be4a1b99e41ab9e0", + "tests/test.sh": "928d1497ad21aef0c456294e3784f1fddf8493410f23584da89f9e02ec6bd134", + "tests/test_outputs.py": "64d6ce107503714ada346e5606832ee886be9950971cf81523e73d8ba28e8d37" + } + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "selection_rationale": "Legacy-language migration and behavioral compatibility", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "software-engineering", + "declared_difficulty": "easy", + "docker_image": "alexgshaw/cobol-modernization:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "c89318cf59efcc7040e3d946022e2616161b6733931a080cdeea7b1918e7ecc4", + "environment/Dockerfile": "8b175c83fc03ec7d16d6101516bc5100230b592f480e435279f4fe051c9280b2", + "environment/data/ACCOUNTS.DAT": "46f63eb29442ebc6716bcd8a0c36f4367910f2783f6b84a307ed9a61ccd1046e", + "environment/data/BOOKS.DAT": "db04da646207cd87f8e9c5b6f0b57d95953899f3eef22de1a81423f06ef4a9fd", + "environment/data/TRANSACTIONS.DAT": "1fa4cc5420ded94d585c563e8196b85166ebbae2bb8924b20014717fb2fb9ee3", + "environment/src/INPUT.DAT": "52fc984f0cf14af1495285d7efe02b9bae13159eea59255560ee938c4f78afee", + "environment/src/program.cbl": "27cb0e56f65f52fa75f346720b37a180b8e034ebe7d7622b979f195087bd6ebb", + "instruction.md": "a87074f1141b7345394cd88053d0debb0dc8f7de95308882c6538ee9a3b359da", + "solution/solve.sh": "7d4a0504e1686c1790bfbc074d4e598f4c631f54c2bc5f3d5fd786574b4810b9", + "task.toml": "4b620211dc3d6b1b989f0210c5ecd2f606974581c8bce27bd78659d48cf0cfbd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "009180a7092583fc8143bb3ffc3957923aa0d61d0aac3a471f76eeb4aae54a88" + } + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "selection_rationale": "Native memory management debugging", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "category": "debugging", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/custom-memory-heap-crash:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a06ed23a2c467aa36d0d4a88e5c020b1076cc25b8d483517239658e9c596dde9", + "environment/Dockerfile": "6ee81e5a4fa2540443d7f27d577ba840728fc6aab9f301ab7919103673c78080", + "environment/patches/locale_classes.tcc.patch": "16951f4a4df6bf253886ac75f5a39fdbc328e415194493f63cb7589a364aa817", + "environment/patches/locale_init.cc.patch": "e97f133ceebd86f8881f972e0e45b0ec0a13a9bfa428d8032008707683d54292", + "environment/program/main.cpp": "b94538c76887cc2b2ab739b5ac4731cddd0faa8f4a9892a6265c062bbbeede14", + "environment/program/user.cpp": "92b203786fed1bf0b3aa1675c9cd2d82b2651170675064da8ffee73b98ef14f9", + "environment/program/user.h": "f43d155995a56c22b9cd1417daef7c3b5a976cdc662a98eef0feb438ae1eee77", + "instruction.md": "53c68ae14a3c3af9a97c40bfd9bdc30a5d98b37eaa2f69f763475745f70f108d", + "solution/solve.sh": "b0b1d10bea1cd5be14a82b35d522df616df1e67028f90d2a6bef90aaa98511a1", + "task.toml": "ed474be1983e212913aa8866ebb66216acbe36aef2339ac375d232aa46617481", + "tests/test.sh": "70a2b55f0c5757f6b123a216dfd202be4a646680bced28dddcd31fd8959a058f", + "tests/test_outputs.py": "4e1cb33f777fa7fb5e1536f9bea9eace6deccd515c67d4f87570adbe692914ba" + } + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "selection_rationale": "Security bug repair and behavior preservation", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "security", + "declared_difficulty": "hard", + "docker_image": "alexgshaw/fix-code-vulnerability:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "87f28dbad183c3a634887696d328ca68a8a0858f132bc48cb7341c22267d38f7", + "environment/Dockerfile": "0917470986224df4daf4854a71302ae81c73ba5dd6e25be1232d29b758a5b3ab", + "instruction.md": "89b1abdf0af19399f720233b5cdf6b2e8e0ff7b6e3fe3e2a3af0d4d298a8641a", + "solution/solve.sh": "7a0131b369d83ae8546ecc3f9b4e23d38240f1bd1f0941e4430fa85b5182ee03", + "task.toml": "9aa2bd14ce9408289d199d0608bc3fa65b8b5e23374c9641020de62197dbf1e2", + "tests/test.sh": "f1611da693cb4cfc5e8482698c6a1858a5c67220ae9db89b50aed66584e9b17f", + "tests/test_outputs.py": "c006a5de95b69bd78055aa26cc055734dbd50718ede25d331f767bfee4ef7f26" + } + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "selection_rationale": "Service configuration and observable request behavior", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "system-administration", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/nginx-request-logging:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d79997586d1c14ed8dbfea050e3580b25500e04c8345acfd3f294152433c795a", + "environment/Dockerfile": "43003466d89ab83484a3fa618e35b58cdb74a002b1e8716ef30f12b3ac8302d8", + "instruction.md": "26a7ac98aced6107f147206790ada77bbf8ed3c25fb360cf323d5fc6889edf99", + "solution/solve.sh": "2454bbbf7c201a9a778165f286373ba2cb991a0eb1d026a8005d57ae71250dd9", + "task.toml": "b7a78d00be13656f75069197100b7b8c4fd268decf6f78929c853b3222a2729d", + "tests/test.sh": "27761195f53cbfe5941fe39b573631f6c926869e64b7c43220377e1e116528e7", + "tests/test_outputs.py": "045cc716c14efde3b0dcff5fc7c85ec5d18bfc6ce66f8b40a418fa2a3a4acda0" + } + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "selection_rationale": "Cross-branch repository operations", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "system-administration", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/git-multibranch:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3b9473429381d9eacabab498cb4a8915aa2bca24fa857ef862e5991d3f81f113", + "environment/Dockerfile": "88d9a9f4fef1ae7fddf632799dedf5a2fb6489460a93b610c33dd8b122982821", + "environment/default.conf": "a3a4c5d0ac9cfe79b0510e4225c80fa3988c27fbac13622505886bf66449e04a", + "instruction.md": "54014a38d7526d65faeb6c11d9b7ed3d4a5cb5d973d68d3da3b68b8c0289e125", + "solution/solve.sh": "300ca96b79db8d77f14742109aaf76048aafc4644ced750b1bc5c2761e8d0175", + "task.toml": "4b9981604578a31205257cca36a28d0c89c9d8d1058541330acd11d3f8d65f82", + "tests/test.sh": "ddbde3eea6ffa8fda105cbd86597a53bd409f41e13223900aa27e77a84b6b155", + "tests/test_outputs.py": "f03c6867663b8e8fb7899df51ee4df26411fab62ed491753964b801e591d16c5" + } + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "selection_rationale": "Database recovery and data integrity", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "file-operations", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + } + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "selection_rationale": "Data integration and cross-input consistency", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "data-processing", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/multi-source-data-merger:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9bf37162a4c1b88ed93627830bb8512ebae154325d7cb069c883144a0e4c69d6", + "environment/Dockerfile": "c95a40d3d35b473dc15ae1e8784ba7fb1725805c0583fc574dd2764325d75e4f", + "environment/data/source_a/users.json": "75e51e2d3ab0bdba2269d7cf3757576f84f3ee2c13407d2db94289a5958a1778", + "environment/data/source_b/users.csv": "8e87d16dc47d166edd8e56165955bad265b630d53d02762367c816affb5fa760", + "environment/data/source_c/users.parquet": "23d1ea51c4170bdba53febb885a0f495e28de525a52667347b2932136b9a72fe", + "instruction.md": "d1d28bb6e37f7a255a1b7f78aae8591f35c0494f33e4913392424ad954aab944", + "solution/solve.sh": "1b3d0d50e31930d042495d48f4c8e95f1a9c1a979c803b7f7cec651322537e1e", + "task.toml": "ee3964c0ee040f0a0cb121cefefbc15443d127684acb003fa5b61dd42f31c694", + "tests/test.sh": "d5a00e64413c0e1ae452f38a1aa1a47c3ed7b4416bbfccae1a4c053025a336d1", + "tests/test_outputs.py": "a070888930cf829aeca039771260b1386bda990ea805f313468d381ba3608ca5" + } + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "selection_rationale": "Database query performance and correctness", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 1800.0, + "category": "data-science", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/query-optimize:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a82c53e333a0ac0b0b410b5e26cb8e7305dece31db84d340248ffb13d5a45160", + "environment/Dockerfile": "064818771f200e0517d22068f87229a7a95d0a82b0778c84ca7203bf1926a0b8", + "environment/task-deps/my-sql-query.sql": "43382792f0f438625d643e34abe144febd5669518f4ab1ab550a15c6d9314a21", + "instruction.md": "473e12d320d2732f679fb631c2a4dab4a01deea6775686718ae0c417b8a39447", + "solution/solve.sh": "0d95f3cedb287ff9d47c796cbaf812c300d1222c10fa80c49e8acb0e81eb6978", + "task.toml": "6e88f2a86df699594bcb1bf81df4ed694505f79c9420f671335b02524bbe3868", + "tests/golden.sql": "4da1885d56b3ffa7b7b22b422996d018795611f6963499d551582881b74b84fd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2068ae80c57af8d52a7ae510f0666732c057806e556b480970f36f14bae683f6" + } + }, + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "selection_rationale": "Numerical sampling and statistical invariants", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "scientific-computing", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/adaptive-rejection-sampler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "95408217b9c484ffc18a02a91b7706323b6902919c72b24866a32e27a23543e2", + "environment/Dockerfile": "e644a3343e339303230f1441b7383f076d21c69a5a7aa6430b4f8a9d27f15c8b", + "environment/protected.tar.gz.enc": "7fc9bf1347bb97da8dede04b2c58638f225070913e222d76627068aebd1b1df2", + "instruction.md": "e6d28c28acd76bef61817427c74c2a29e3e8d75f535cbd708afcbfd02c54013f", + "solution/solve.sh": "fe443a8082b5183ba1623311ba0ee958888aefa633eff93766b9ae14e9bdd84d", + "task.toml": "6ed7b37f74beb46e9d47b0f5fad53025667c9548baff4e96d467b56d429fab1a", + "tests/test.sh": "38b43560d173cc2b952c3a3e17b8a480216d84e33450515e047bcb0d806b1e0a", + "tests/test_outputs.py": "547dc6e107f034f41703722aeceb6d0236e3fb69116fc3f2fbaba11884de352f" + } + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "selection_rationale": "Constrained algorithmic planning", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 1200, + "work_budget_seconds": 1020, + "verifier_timeout_seconds": 1200.0, + "category": "personal-assistant", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/constraints-scheduling:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d54a690970149e1216bd4fc066b45069c7948c13e3b1ef2a213a74342805f209", + "environment/Dockerfile": "7d9aba072a4a907229728b9739c8cf821fc2c98450babd64452682caf3759c84", + "environment/inputs/alice_calendar.ics": "5a5d45804bffd339c2479d49e76fe817b38244d14d389afe1de583443808ecbc", + "environment/inputs/bob_calendar.ics": "c97e85dbcdd756758f8a80f4db5fd5c1dcacee79298e4d2262f46de5ff680b50", + "environment/inputs/carol_calendar.ics": "32a0f8687637df0a12b8e5c56d49c4192bc3e790856943632d70f237c7c1363b", + "instruction.md": "342def78177f9bc41bea4d11af679bc385a9dda2ac8a1e425be384a5b7f937b6", + "solution/solve.sh": "802d81ec064fad3050bbade95a2f847ac24e0b7f80fa69b0bcee0347dc92fe77", + "task.toml": "7b9842de16d74e8faf8c7c0aa03b0b81145941cb12dd322d81ae23fc48607250", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "1422b345ba3aacdfc4184e962811b6379988435896c50d8cf62903e70990c5d1" + } + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "selection_rationale": "Model packaging and command-line contracts", + "cohort": "new_development_baseline", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "category": "model-training", + "declared_difficulty": "medium", + "docker_image": "alexgshaw/pytorch-model-cli:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "35a093fa5f1592737784ba718c2e3e9ae17ff8b5e53e003fae560ddce091039c", + "environment/Dockerfile": "8fd5c0f21a6914cd440d10c84c9db06fec606f20dc3e0363b17a3b8c220dc4d9", + "environment/task-deps/cJSON.c": "0e7995261640c96db16ed2c91a5d981efa917cf5e72d0f97ec45a1179ae3ad73", + "environment/task-deps/cJSON.h": "0909160541bcc510e0b2d8c7a9cc051fa1459a0f13be4d8e56346a15491f2f2f", + "environment/task-deps/image.png": "f4d5dfefe463939996541404340e7099d5957b56e1df3760d13b5d2de754db53", + "environment/task-deps/lodepng.cpp": "67873c5f6272f779a143a9c81070e8783327c5552d0b9de2aaee2cb779308917", + "environment/task-deps/lodepng.h": "b735b7e0357319973e9488d520d7f270d734a6e9e1bf924fb1d5b8d26d7e97ce", + "environment/task-deps/model.pth": "a32103eef09936e586b1a7dc7e64d279500a38ff30fdd6df2167e28eb954eee6", + "environment/task-deps/model.py": "e4d3a5c728449656d0efde7871ee227619348352769795ed1be1a8b7f835fddc", + "instruction.md": "ce36ee1f6c589f81ee6a6be4967e25c76a2ed697006928023f42dfb9f28a6750", + "solution/solve.sh": "3b34e561c9ef0a53c9948655847846eb4811aaac67b990480aa6aac1b04bfe66", + "task.toml": "fc1c018efbd5d0537ac0741d53d1ebad3886faa51d14cbde8ec953856411bc0d", + "tests/test.sh": "43fb67e537c7adc2df11c2367f47c876cb5d16a7c1ae16d0339c3a3364bb2b06", + "tests/test_outputs.py": "370702c57a4e0c85f70ee37a5c357ae3a5df8480a275bb6263b0376f2f6c093b" + } + } + ], + "total_model_runs": 12, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "preflight": [], + "queue": [ + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cobol-modernization", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/cobol-modernization.json", + "job_name": "dev12-cobol-modernization-baseline-20261011", + "config_sha256": "2680126c2f162d58948cd30565806bf6cb1f026d561c87b942ba4735173ea6a2" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/fix-code-vulnerability", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/fix-code-vulnerability.json", + "job_name": "dev12-fix-code-vulnerability-baseline-20261011", + "config_sha256": "4b2de43b547ddd92d748f7658c44b98b4fef25efd406710ace6d1713b9c789a7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/nginx-request-logging", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/nginx-request-logging.json", + "job_name": "dev12-nginx-request-logging-baseline-20261011", + "config_sha256": "4a79d4fc4431acd942548f36188c1820ac5406fc6a1385cf505daf02aa190da3" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/git-multibranch", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/git-multibranch.json", + "job_name": "dev12-git-multibranch-baseline-20261011", + "config_sha256": "ae9c6d5830c9dcb8aab8b1356441b47731957e2ad5e5747327b7caa1a6c037a1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/db-wal-recovery", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/db-wal-recovery.json", + "job_name": "dev12-db-wal-recovery-baseline-20261011", + "config_sha256": "475b7df31549e51f04f4f9171332da5a8ba03f7fbec2b3788fbf897340c0c91b" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/multi-source-data-merger", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/multi-source-data-merger.json", + "job_name": "dev12-multi-source-data-merger-baseline-20261011", + "config_sha256": "5eac4294149a11f037f732a32e1a8874613d8a28884843955481a9e8ef10b7bd" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/query-optimize", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/query-optimize.json", + "job_name": "dev12-query-optimize-baseline-20261011", + "config_sha256": "29ad0478282cd6fb86a1b209a93ffe33eee724c321916732db001b7beef8ea7e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/adaptive-rejection-sampler", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/adaptive-rejection-sampler.json", + "job_name": "dev12-adaptive-rejection-sampler-baseline-20261011", + "config_sha256": "e0fec8e3f1f8e077a5384058ea5b6e4733973a7475b02556f90cfc0f4ba96f6d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-cli", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/pytorch-model-cli.json", + "job_name": "dev12-pytorch-model-cli-baseline-20261011", + "config_sha256": "bc915a7de1286954473271e2232f9c83af983c23b885d099316b33dc3c1d445d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/constraints-scheduling", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/constraints-scheduling.json", + "job_name": "dev12-constraints-scheduling-baseline-20261011", + "config_sha256": "33813c2fd5b45efd3b7bf8a765fcb1f74c702821a3d4f473e4b479200433734e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/custom-memory-heap-crash", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/custom-memory-heap-crash.json", + "job_name": "dev12-custom-memory-heap-crash-baseline-20261011", + "config_sha256": "4678ab6961d82a2d597911bd8d4a53040a4be526c1e97afb27c614ecedb1c05f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-pov-ray", + "config": "benchmarks/harbor/configs/dev12-baseline-20261011/build-pov-ray.json", + "job_name": "dev12-build-pov-ray-baseline-20261011", + "config_sha256": "42e1bb7d2a1a6d5cf9f059a121140f9135d8b4632482a00d767a9aba2e1ac2e7" + } + ], + "log_dir": "jobs/dev12-baseline-20261011-v3-orchestration", + "budget_policy": "Native task agent timeout minus 180 seconds for runtime work; setup separately allows 1800 seconds. Native verifier limits remain unchanged. Build POV-Ray retains its unusually long 12000-second native limit (11820-second model work).", + "execution_policy": "One run per selected task. Run all twelve installation preflights before models. Queue is fixed by ascending native work budget, stable within ties; run consecutive pairs with at most two concurrent model trials.", + "failure_policy": "Retain every selected trial including setup failures, verifier errors and scored failures; no outcome-based substitutions or model reruns. Only verifier timeouts may retry, at most three total attempts on the same output with the native per-attempt limit.", + "preflight_policy": "All selected verifier dependency profiles are not_configured. Agent installation success does not certify verifier dependencies. Registered profiles, if present, must pass before model work.", + "information_policy": "Selection used task names before downloading. Only public metadata/instructions may be inspected before every agent output freezes; task files are hashed as bytes without inspecting hidden test or solution contents. Audit external references afterwards.", + "scope": "Development baseline for twelve new tasks, not a representative full-dataset score or a pass-rate improvement experiment. Existing 22 task records remain separate; remaining 55 tasks are not opened.", + "protocol_amendment": { + "supersedes": "experiment-v2.json", + "supersedes_sha256": "87f3c760051997e8957ae12efc77085040885053f44b4a9490565e4d3b371647", + "evidence_file": "preflight-v2-evidence.json", + "evidence_sha256": "8b4edad67647f78f4dd63ffafebee68bfbc5cfa19af91b8ded898826d8c14d46", + "previous_evidence_file": "preflight-v1-evidence.json", + "previous_evidence_sha256": "c723c858571a512e968db16babed482a2874f270561ae3ae8c426a2ad74bd73f", + "reason": "Use eleven successful installation-only trials from the second preflight plus the successful POV-Ray installation from the first. Preserve both failed job results and every failed trial. Each queued task/arm pair needs exactly one successful native installation-only result, bound to task revision, complete source snapshot, installed wheel, bootstrap, input configuration and artifact hashes. No model work has started.", + "changed_fields": [ + "registered_at", + "log_dir", + "preflight", + "accepted_preflight_trials", + "protocol_amendment" + ], + "unchanged": "Task selection and revisions, agent and adapter source, prompt, model, all task configurations and budgets, model queue and job names, concurrency, retry and information policies.", + "model_runs_started_at_amendment": 0 + }, + "accepted_preflight_trials": [ + { + "arm": "main", + "task": "terminal-bench/build-pov-ray", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-20261011/build-pov-ray__JBS6iy8/result.json", + "result_sha256": "c878116d4f2746c3269ceefba34b9f0d1c0d9672d3010adb43ff904767391c84", + "bootstrap_sha256": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4", + "manifest_sha256": "b36023c8c2ba8248ae41ab708b1bd39a8cb99f665522960170e51f20d1bb6227", + "input_config_sha256": "6dc58db5abd850d91f1e64d0b35a2a82ed85a36ffdff763b93ff60e5b050081a" + }, + { + "arm": "main", + "task": "terminal-bench/cobol-modernization", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/cobol-modernization__2A6Ldop/result.json", + "result_sha256": "f1010dc85724456c52f48644016afdf15a2a29dc68c57430270a01d16f75813b", + "bootstrap_sha256": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/custom-memory-heap-crash", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/custom-memory-heap-crash__D2vB9wp/result.json", + "result_sha256": "2151d83279d3fea4a998d1611cf7e523ae061bd1592ddb6df3825939d0477c1c", + "bootstrap_sha256": "cf6bb84c42557ba2a7b7e0ec653bbc1c7cf86c236d1b7e2a6852918dfb8eb7dc", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/fix-code-vulnerability", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/fix-code-vulnerability__m7qrvoe/result.json", + "result_sha256": "4914a240602a36fedab3ac6313643cb1beffc22de6724fd48b45f4974efa0339", + "bootstrap_sha256": "8d1fa3e359c1a2c87bb9b262c9113a423af3445aa5a20da8a2e6fb4d63b4f67a", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/nginx-request-logging", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/nginx-request-logging__LpiAC8p/result.json", + "result_sha256": "edc76f020a59073905b3d74d47b0bca454a3b5a43b13c0df23fc19b1bf0e0bd2", + "bootstrap_sha256": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/git-multibranch", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/git-multibranch__NsTYkJt/result.json", + "result_sha256": "66a667c7278b36d1bee52e1bf466d1464c7debdde77a288f0fb62d1ea7fbbd6f", + "bootstrap_sha256": "391e000959d10d8aa551310ebe7bddd0c70185a684ee8cb543b7e8bf57e5e63b", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/db-wal-recovery", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/db-wal-recovery__pJKJsai/result.json", + "result_sha256": "c0f6d3334e4f8bedd61a811ac5110a1776bf9a9d56bd1037910501d552b70289", + "bootstrap_sha256": "2ffe3efbf9fd9f423c3fc82102152f59af7f9180f8aaf39c46e61c96e2f02d76", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/multi-source-data-merger", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/multi-source-data-merger__3LWnqvT/result.json", + "result_sha256": "da74a33cea03bc9a54f5ff3e39c1ed8a3c07a9ed1ae7405ff73069c68f1a782c", + "bootstrap_sha256": "49ba4308d822fabb4ca040d48e8e24e7c3bd34137bc08e64c1e19b856268814c", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/query-optimize", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/query-optimize__ng6Cxoz/result.json", + "result_sha256": "0bdc5b650976ab994bea028e62b542682f8e9f13005fba846797f8ec11b8b580", + "bootstrap_sha256": "0a52c1c56dbec333ab809d00b69469021e06f49e0570249b280b6aacef2587f0", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/adaptive-rejection-sampler", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/adaptive-rejection-sampler__tXynuKz/result.json", + "result_sha256": "92c4b295be0b5890374c4b124ba339b19111969252a670547a5d47a46047d878", + "bootstrap_sha256": "eb4895cdc7076ee2da4b9519eb1d623465f7c0828b9afb96a5d685f89a8c1f1b", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/constraints-scheduling", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/constraints-scheduling__sEgWjQv/result.json", + "result_sha256": "9c845d39b1d6ee7db05df413559408a68f99711d9a89754fe1ea474fb72cb1d3", + "bootstrap_sha256": "c4d5399fb99e22fdaccbafaeca054f4d615062b69ddc2389af0ed1c19ba30325", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + }, + { + "arm": "main", + "task": "terminal-bench/pytorch-model-cli", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/pytorch-model-cli__Gm2gHNY/result.json", + "result_sha256": "4c3ba76c5299e13b0be55de1f63ec981933e126136026e2ce06fbbfe4d6811cd", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "7209a44f49fe68e5059dd8d36823546445ae8b86b7abc8c098ee0f6eaa0199f0", + "input_config_sha256": "901854a68691bc24f5d95b37f0d4d292f970ca691ee3e1f7a7f8c7ffe0afca18" + } + ] +} diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/preflight-v2-evidence.json b/benchmarks/harbor/configs/dev12-baseline-20261011/preflight-v2-evidence.json new file mode 100644 index 0000000..c1e99fe --- /dev/null +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/preflight-v2-evidence.json @@ -0,0 +1,206 @@ +{ + "recorded_at": "2026-10-11T15:48:27.483536+00:00", + "preflight_state_path": "jobs/dev12-baseline-20261011-v2-orchestration/dev12-main-install-only-v2-20261011.json", + "preflight_state_sha256": "8f4fb628bbee4faae150ef1932dff53a2a90d695c8d08fd551c72a6b93af82cd", + "preflight_exit_code": 1, + "summary": { + "setup_passed": 11, + "agent_install_network_errors": 1, + "model_runs_started": 0 + }, + "failure_diagnosis": "POV-Ray: curl returned 18 (transferred a partial file) while downloading the original uv 0.9.11 release archive. Its earlier installation succeeded.", + "trials": [ + { + "task": "terminal-bench/adaptive-rejection-sampler", + "trial": "adaptive-rejection-sampler__tXynuKz", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/adaptive-rejection-sampler__tXynuKz/result.json", + "result_sha256": "92c4b295be0b5890374c4b124ba339b19111969252a670547a5d47a46047d878", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/adaptive-rejection-sampler__tXynuKz/agent/bootstrap.json", + "bootstrap_sha256": "eb4895cdc7076ee2da4b9519eb1d623465f7c0828b9afb96a5d685f89a8c1f1b", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/build-pov-ray", + "trial": "build-pov-ray__55Ff9nd", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/build-pov-ray__55Ff9nd/result.json", + "result_sha256": "c3e7244e688ad45a31f8bafccdab1e9dccef1539189d9674640b37e7810681ac", + "exception_type": "NetworkConnectionError", + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/build-pov-ray__55Ff9nd/agent/bootstrap.json", + "bootstrap_sha256": "f906cdc7af870689cc72b31c415d66f9788fce687ea0f6b40d3b5e9e66e0fe1a", + "bootstrap_status": "failed", + "bootstrap_stage": "agent_install", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/cobol-modernization", + "trial": "cobol-modernization__2A6Ldop", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/cobol-modernization__2A6Ldop/result.json", + "result_sha256": "f1010dc85724456c52f48644016afdf15a2a29dc68c57430270a01d16f75813b", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/cobol-modernization__2A6Ldop/agent/bootstrap.json", + "bootstrap_sha256": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/constraints-scheduling", + "trial": "constraints-scheduling__sEgWjQv", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/constraints-scheduling__sEgWjQv/result.json", + "result_sha256": "9c845d39b1d6ee7db05df413559408a68f99711d9a89754fe1ea474fb72cb1d3", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/constraints-scheduling__sEgWjQv/agent/bootstrap.json", + "bootstrap_sha256": "c4d5399fb99e22fdaccbafaeca054f4d615062b69ddc2389af0ed1c19ba30325", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/custom-memory-heap-crash", + "trial": "custom-memory-heap-crash__D2vB9wp", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/custom-memory-heap-crash__D2vB9wp/result.json", + "result_sha256": "2151d83279d3fea4a998d1611cf7e523ae061bd1592ddb6df3825939d0477c1c", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/custom-memory-heap-crash__D2vB9wp/agent/bootstrap.json", + "bootstrap_sha256": "cf6bb84c42557ba2a7b7e0ec653bbc1c7cf86c236d1b7e2a6852918dfb8eb7dc", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__pJKJsai", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/db-wal-recovery__pJKJsai/result.json", + "result_sha256": "c0f6d3334e4f8bedd61a811ac5110a1776bf9a9d56bd1037910501d552b70289", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/db-wal-recovery__pJKJsai/agent/bootstrap.json", + "bootstrap_sha256": "2ffe3efbf9fd9f423c3fc82102152f59af7f9180f8aaf39c46e61c96e2f02d76", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "trial": "fix-code-vulnerability__m7qrvoe", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/fix-code-vulnerability__m7qrvoe/result.json", + "result_sha256": "4914a240602a36fedab3ac6313643cb1beffc22de6724fd48b45f4974efa0339", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/fix-code-vulnerability__m7qrvoe/agent/bootstrap.json", + "bootstrap_sha256": "8d1fa3e359c1a2c87bb9b262c9113a423af3445aa5a20da8a2e6fb4d63b4f67a", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/git-multibranch", + "trial": "git-multibranch__NsTYkJt", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/git-multibranch__NsTYkJt/result.json", + "result_sha256": "66a667c7278b36d1bee52e1bf466d1464c7debdde77a288f0fb62d1ea7fbbd6f", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/git-multibranch__NsTYkJt/agent/bootstrap.json", + "bootstrap_sha256": "391e000959d10d8aa551310ebe7bddd0c70185a684ee8cb543b7e8bf57e5e63b", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/multi-source-data-merger", + "trial": "multi-source-data-merger__3LWnqvT", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/multi-source-data-merger__3LWnqvT/result.json", + "result_sha256": "da74a33cea03bc9a54f5ff3e39c1ed8a3c07a9ed1ae7405ff73069c68f1a782c", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/multi-source-data-merger__3LWnqvT/agent/bootstrap.json", + "bootstrap_sha256": "49ba4308d822fabb4ca040d48e8e24e7c3bd34137bc08e64c1e19b856268814c", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/nginx-request-logging", + "trial": "nginx-request-logging__LpiAC8p", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/nginx-request-logging__LpiAC8p/result.json", + "result_sha256": "edc76f020a59073905b3d74d47b0bca454a3b5a43b13c0df23fc19b1bf0e0bd2", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/nginx-request-logging__LpiAC8p/agent/bootstrap.json", + "bootstrap_sha256": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/pytorch-model-cli", + "trial": "pytorch-model-cli__Gm2gHNY", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/pytorch-model-cli__Gm2gHNY/result.json", + "result_sha256": "4c3ba76c5299e13b0be55de1f63ec981933e126136026e2ce06fbbfe4d6811cd", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/pytorch-model-cli__Gm2gHNY/agent/bootstrap.json", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + }, + { + "task": "terminal-bench/query-optimize", + "trial": "query-optimize__ng6Cxoz", + "result_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/query-optimize__ng6Cxoz/result.json", + "result_sha256": "0bdc5b650976ab994bea028e62b542682f8e9f13005fba846797f8ec11b8b580", + "exception_type": null, + "bootstrap_path": "jobs/dev12-baseline-main-source/jobs/dev12-main-install-only-v2-20261011/query-optimize__ng6Cxoz/agent/bootstrap.json", + "bootstrap_sha256": "0a52c1c56dbec333ab809d00b69469021e06f49e0570249b280b6aacef2587f0", + "bootstrap_status": "ready", + "bootstrap_stage": "complete", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "model_started": false + } + ] +} diff --git a/benchmarks/harbor/scripts/run_comparison.py b/benchmarks/harbor/scripts/run_comparison.py index 9266f1f..1f2cfbc 100644 --- a/benchmarks/harbor/scripts/run_comparison.py +++ b/benchmarks/harbor/scripts/run_comparison.py @@ -114,6 +114,67 @@ def preflight_passed(states, expected_counts): ) +def accepted_preflights_passed(root, plan): + """Validate previously completed installation trials without replaying jobs. + + A failed sibling trial does not invalidate a successful installation. Every + task/arm pair still needs native, hash-bound evidence from the frozen source. + This mode cannot be combined with new preflight jobs in the same plan. + """ + if plan["preflight"]: + raise ValueError("Accepted installation evidence cannot mix with new preflight jobs") + expected = {(j["arm"], j["task"]) for j in plan["queue"]} + evidence = plan["accepted_preflight_trials"] + pairs = [(e["arm"], e["task"]) for e in evidence] + if not expected or set(pairs) != expected or len(pairs) != len(expected): + raise ValueError("Every queued task/arm pair needs exactly one accepted preflight") + tasks = {t["name"]: t for t in plan["tasks"]} + + def checked(path, expected_hash): + if digest(path) != expected_hash: + raise ValueError(f"Accepted preflight evidence changed: {path}") + return json.loads(path.read_text()) + + for item in evidence: + arm = plan["arms"][item["arm"]] + source = root / arm["source_root"] + result_path = (root / item["result_path"]).resolve() + if not result_path.is_relative_to((source / "jobs").resolve()) or result_path.name != "result.json": + raise ValueError("Accepted trial must belong to the frozen source checkout") + result = checked(result_path, item["result_sha256"]) + task = tasks[item["task"]] + task_id = result["task_id"] + if result["task_name"] != item["task"] or task_id["ref"] != task["ref"]: + raise ValueError("Accepted preflight has a different task revision") + if (not result.get("finished_at") or result.get("exception_info") + or result.get("agent_execution") is not None + or result.get("verifier") is not None + or result.get("verifier_result") is not None): + raise ValueError("Accepted evidence must be a successful installation-only trial") + bootstrap = checked(result_path.parent / "agent/bootstrap.json", item["bootstrap_sha256"]) + if (bootstrap.get("status") != "ready" or bootstrap.get("stage") != "complete" + or bootstrap.get("verifier_preflight", {}).get("status") not in {"passed", "not_configured"}): + raise ValueError("Accepted installation did not finish its required preflight") + job_dir = result_path.parent.parent + inputs = job_dir.with_name(job_dir.name + "-input") + manifest = checked(inputs / "manifest.json", item["manifest_sha256"]) + config = checked(inputs / "config.json", item["input_config_sha256"]) + if config.get("install_only") is not True or manifest["commit"] != arm["source_ref"]: + raise ValueError("Accepted installation used a different mode or source revision") + expected_sources = {} + for prefix, package in (("src", "nanopycodeagent"), ("benchmarks/harbor/src", "harbor_adapter")): + for path in (source / prefix / package).rglob("*.py"): + expected_sources[str(path.relative_to(source / prefix))] = digest(path) + if not expected_sources or manifest["source_sha256"] != expected_sources: + raise ValueError("Accepted installation source snapshot differs from the frozen source") + wheel = Path(result["config"]["agent"]["kwargs"]["wheel_path"]).resolve() + if not wheel.is_relative_to((inputs / "dist").resolve()): + raise ValueError("Accepted installation wheel must belong to its recorded inputs") + if not (digest(wheel) == manifest["wheel_sha256"] == bootstrap["wheel_sha256"]): + raise ValueError("Accepted installation wheel digest differs from its provenance") + return True + + def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--plan", type=Path, required=True) @@ -122,6 +183,10 @@ def main(): root, plan_path = Path.cwd(), args.plan.resolve() plan = json.loads(plan_path.read_text()) validate_plan(root, plan_path, plan) + accepted = ( + accepted_preflights_passed(root, plan) + if "accepted_preflight_trials" in plan else False + ) record = root / plan["log_dir"] record.mkdir(parents=True, exist_ok=True) record.chmod(0o700) @@ -138,10 +203,11 @@ def main(): ] if args.phase == "preflight": states = [run_job(root, plan, j, record, True) for j in plan["preflight"]] - return 0 if preflight_passed(states, expected_counts) else 1 - preflight = [json.loads((record / (j["job_name"] + ".json")).read_text()) for j in plan["preflight"]] - if not preflight_passed(preflight, expected_counts): - raise RuntimeError("Install-only checks must finish successfully before model work") + return 0 if accepted or preflight_passed(states, expected_counts) else 1 + if not accepted: + preflight = [json.loads((record / (j["job_name"] + ".json")).read_text()) for j in plan["preflight"]] + if not preflight_passed(preflight, expected_counts): + raise RuntimeError("Install-only checks must finish successfully before model work") states = [] with ThreadPoolExecutor(max_workers=2) as pool: for offset in range(0, len(plan["queue"]), 2): diff --git a/benchmarks/harbor/tests/test_comparison.py b/benchmarks/harbor/tests/test_comparison.py index 61faa56..7284900 100644 --- a/benchmarks/harbor/tests/test_comparison.py +++ b/benchmarks/harbor/tests/test_comparison.py @@ -55,3 +55,109 @@ def test_interrupted_job_cannot_silently_restart(tmp_path): (tmp_path / "interrupted.json").write_text(json.dumps({"status": "started"})) with pytest.raises(RuntimeError, match="refusing to retry"): comparison.run_job(tmp_path, {}, job, tmp_path, False) + + +@pytest.fixture +def accepted_installation(tmp_path): + source = tmp_path / "source" + module = source / "src/nanopycodeagent/agent.py" + module.parent.mkdir(parents=True) + module.write_text("# frozen runtime\n") + trial = source / "jobs/preflight/task__one" + (trial / "agent").mkdir(parents=True) + inputs = source / "jobs/preflight-input" + (inputs / "dist").mkdir(parents=True) + wheel = inputs / "dist/runtime.whl" + wheel.write_bytes(b"recorded wheel bytes") + files = { + "result": (trial / "result.json", { + "task_name": "org/task", "task_id": {"ref": "sha256:task"}, + "finished_at": "2026-10-11T00:00:00Z", "exception_info": None, + "agent_execution": None, "verifier": None, "verifier_result": None, + "config": {"agent": {"kwargs": {"wheel_path": str(wheel)}}}, + }), + "bootstrap": (trial / "agent/bootstrap.json", { + "status": "ready", "stage": "complete", + "verifier_preflight": {"status": "not_configured"}, + "wheel_sha256": comparison.digest(wheel), + }), + "manifest": (inputs / "manifest.json", { + "commit": "frozen", "wheel_sha256": comparison.digest(wheel), + "source_sha256": {"nanopycodeagent/agent.py": comparison.digest(module)}, + }), + "input_config": (inputs / "config.json", {"install_only": True}), + } + evidence = {"arm": "main", "task": "org/task", "result_path": str((trial / "result.json").relative_to(tmp_path))} + for name, (path, data) in files.items(): + path.write_text(json.dumps(data)) + evidence[name + "_sha256"] = comparison.digest(path) + plan = { + "preflight": [], "queue": [{"arm": "main", "task": "org/task"}], + "arms": {"main": {"source_root": "source", "source_ref": "frozen"}}, + "tasks": [{"name": "org/task", "ref": "sha256:task"}], + "accepted_preflight_trials": [evidence], + } + return plan, files, evidence, module, wheel + + +def test_accept_native_success_even_when_a_sibling_failed(tmp_path, accepted_installation): + plan, files, _, _, _ = accepted_installation + sibling = files["result"][0].parent.parent / "failed__two" + sibling.mkdir() + (sibling / "result.json").write_text(json.dumps({"exception_info": {"exception_type": "NetworkConnectionError"}})) + assert comparison.accepted_preflights_passed(tmp_path, plan) + assert (sibling / "result.json").exists() + + +@pytest.mark.parametrize("change", ["missing", "duplicate", "mixed"]) +def test_every_pair_needs_exactly_one_success(tmp_path, accepted_installation, change): + plan, _, evidence, _, _ = accepted_installation + if change == "missing": + plan["queue"].append({"arm": "main", "task": "org/second"}) + elif change == "duplicate": + plan["accepted_preflight_trials"].append(evidence.copy()) + else: + plan["preflight"].append({"job_name": "another"}) + with pytest.raises(ValueError): + comparison.accepted_preflights_passed(tmp_path, plan) + + +@pytest.mark.parametrize("field,value", [ + ("exception_info", {"exception_type": "AgentSetupTimeoutError"}), + ("agent_execution", {"started_at": "2026-10-11T00:00:00Z"}), + ("verifier_result", {"rewards": {"reward": 1}}), + ("finished_at", None), + ("task_id", {"ref": "sha256:different"}), +]) +def test_ineligible_trials_cannot_be_accepted(tmp_path, accepted_installation, field, value): + plan, files, evidence, _, _ = accepted_installation + path, data = files["result"] + data[field] = value + path.write_text(json.dumps(data)) + evidence["result_sha256"] = comparison.digest(path) + with pytest.raises(ValueError): + comparison.accepted_preflights_passed(tmp_path, plan) + + +@pytest.mark.parametrize("changed", ["result", "source", "wheel"]) +def test_evidence_and_installed_source_must_still_match(tmp_path, accepted_installation, changed): + plan, files, _, module, wheel = accepted_installation + path = files["result"][0] if changed == "result" else module if changed == "source" else wheel + path.write_bytes(path.read_bytes() + b" ") + with pytest.raises(ValueError): + comparison.accepted_preflights_passed(tmp_path, plan) + + +@pytest.mark.parametrize("name,field,value", [ + ("bootstrap", "verifier_preflight", {"status": "pending"}), + ("manifest", "commit", "another-revision"), + ("input_config", "install_only", False), +]) +def test_required_preflight_mode_and_revision_cannot_be_bypassed(tmp_path, accepted_installation, name, field, value): + plan, files, evidence, _, _ = accepted_installation + path, data = files[name] + data[field] = value + path.write_text(json.dumps(data)) + evidence[name + "_sha256"] = comparison.digest(path) + with pytest.raises(ValueError): + comparison.accepted_preflights_passed(tmp_path, plan) From 29cfddf06b2dca753e8f51d49016593df296d152 Mon Sep 17 00:00:00 2001 From: minixalpha Date: Sun, 11 Oct 2026 10:23:40 -0700 Subject: [PATCH 05/15] test(benchmarks): record completed 12-task main baseline --- .../configs/dev12-baseline-20261011/README.md | 5 + .../reports/dev12-main-baseline-20261011.md | 220 ++ .../results/dev12-main-baseline-20261011.json | 1865 +++++++++++++++++ 3 files changed, 2090 insertions(+) create mode 100644 benchmarks/harbor/reports/dev12-main-baseline-20261011.md create mode 100644 benchmarks/harbor/results/dev12-main-baseline-20261011.json diff --git a/benchmarks/harbor/configs/dev12-baseline-20261011/README.md b/benchmarks/harbor/configs/dev12-baseline-20261011/README.md index c1ce6c3..d116afa 100644 --- a/benchmarks/harbor/configs/dev12-baseline-20261011/README.md +++ b/benchmarks/harbor/configs/dev12-baseline-20261011/README.md @@ -6,6 +6,11 @@ packages, reading their instructions, or seeing outcomes. It is a purposive **development** cohort, not a representative random sample of Terminal-Bench. The other 55 tasks are left unopened by this experiment. +Completed result: **10/12 passed**, with two scored correctness failures and no +final setup or verifier errors. See the +[baseline report](../../reports/dev12-main-baseline-20261011.md) and +[machine-readable evidence](../../results/dev12-main-baseline-20261011.json). + The runtime is main at `2eb99b608dfb39464d17d109a4a8f11afda607a9`, isolated in `jobs/dev12-baseline-main-source`. This branch records configuration and evidence; it does not change the agent or adapter. Neither prompt candidate from PR #48 diff --git a/benchmarks/harbor/reports/dev12-main-baseline-20261011.md b/benchmarks/harbor/reports/dev12-main-baseline-20261011.md new file mode 100644 index 0000000..9113970 --- /dev/null +++ b/benchmarks/harbor/reports/dev12-main-baseline-20261011.md @@ -0,0 +1,220 @@ +# Twelve-task main baseline, 2026-10-11 + +The frozen main runtime passed **10 of 12 tasks (83.3%)**. Both remaining +tasks reached the official assertions and failed on output correctness. There +were no final setup or verifier errors in these twelve scored trials. This is +a new development baseline, with one run per task; it is not a comparison with +the prompt candidates in PR #48. + +The failures expose two concrete verification gaps: losing original evidence +before recovery, and validating an implementation against a reference that +shares its unproven assumption. More self-tests alone would not address either. + +## Runtime and cohort + +- Runtime: main `2eb99b608dfb39464d17d109a4a8f11afda607a9`, in the detached + `jobs/dev12-baseline-main-source` worktree. Agent and adapter source are + unchanged. The reporting branch contains only benchmark infrastructure, + configurations, and evidence. +- Model: `deepseek/deepseek-flash` at + `https://api.deepseek.com/anthropic`; 100 replies, 65,536 output tokens per + request; temperature and reasoning effort unset. The original 385-word + headless prompt is frozen in the configuration directory. +- Dataset: `terminal-bench/terminal-bench-2-1`, revision + `sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a`. + Twelve tasks were selected by public names and coverage before their packages + were downloaded or their outcomes known. They extend the previously exercised + 22 tasks to a 34-task development pool. The other 55 packages remain unopened + by this experiment. This purposive sample is not a full-dataset score or an + estimate of repeated-run reliability. +- Each task ran once, in the registered order, with concurrency two and a + barrier between consecutive pairs. No task was replaced and no complete + model trial was retried. Work limits are native agent limits minus 180 seconds; + setup has a separate 1,800-second allowance. Original verifier limits remain + unchanged, with at most three attempts and only timeouts permitting a retry. + +[Selection and configuration](../configs/dev12-baseline-20261011/README.md), +[active plan](../configs/dev12-baseline-20261011/experiment-v3.json), and +[machine-readable results](../results/dev12-main-baseline-20261011.json) contain +the revisions, budgets, artifact hashes, metrics, and post-run diagnosis. + +## Results + +Agent time below is Harbor's agent-execution stage, excluding environment +startup, agent installation, and the official verifier. Test counts refer to +the official task suite; the security task also ran 367 project tests. + +| Task | Reward | Official tests | Agent seconds | Observed self-verification | +| --- | ---: | --- | ---: | --- | +| `cobol-modernization` | 1 | 3 passed | 505.2 | Executed original COBOL as an oracle; byte comparisons, exit behavior, 155 differential and 120 fuzz cases. | +| `fix-code-vulnerability` | 1 | 6 passed | 66.3 | Reproduced a failing header-validation test, then 367 project tests passed; exercised the WSGI entry point. | +| `nginx-request-logging` | 1 | 8 passed | 34.1 | Real HTTP requests, 404 content, log fields, and a concurrent rate-limit probe. | +| `git-multibranch` | 1 | 1 passed | 232.1 | Password SSH clone/push, both HTTPS deployment paths, deployment timing, and service restart. | +| `db-wal-recovery` | 0 | 5 passed, 2 failed | 302.0 | Checked structure and consistency of reconstructed data, without the lost original WAL as an oracle. | +| `multi-source-data-merger` | 1 | 3 passed | 55.8 | Synthetic mixed-alias case found a dropped-record bug; fixed it and checked final values and conflicts. | +| `query-optimize` | 1 | 6 passed | 536.7 | Full output comparison with original SQL; separate Python reference covered all 17,834 filtered rows. | +| `adaptive-rejection-sampler` | 1 | 9 passed | 566.0 | Distribution moments, quantiles, KS checks, invalid inputs, reproducibility, and saved-sample checks. | +| `pytorch-model-cli` | 0 | 5 passed, 1 failed | 298.7 | 330 image comparisons passed against a reference using the same inferred normalization. | +| `constraints-scheduling` | 1 | 3 passed | 30.2 | Minute-resolution search for the earliest valid slot, then calendar structure and conflict checks. | +| `custom-memory-heap-crash` | 1 | 6 passed | 280.7 | Reproduced release crash; debug/release runs, debugger allocation-site checks, and zero-error Valgrind runs. | +| `build-pov-ray` | 1 | 3 passed | 393.0 | Official-source build, deterministic renders, height-field regression, and reproducible rebuild. | + +These observations come from tool outputs, not just the agents' final claims. +Examples are ATIF steps 71/83/91 for COBOL, 15/19/31 for the security repair, +13/15/17/19 for data merging, 21/31/43 for SQL, and 59/61/65 for the C++ repair. +The sampler corrected a mistaken self-test expectation at step 20 after a +10/11 result; its subsequent saved samples were checked separately at step 22. +Passing the official suite does not establish correctness outside its coverage. + +## Failure: database recovery destroyed its own evidence + +The original task required recovering all eleven records, **including WAL +changes**, and testing their specific values. At ATIF step 5, the agent ran +`sqlite3 main.db` before copying the database and WAL, concurrently with a +640-byte WAL hexdump. At step 7, opening `main.db-wal` failed with +`FileNotFoundError`; step 9 confirmed it was gone. The trace supports SQLite's +open/close as the cause, which the agent also recognized at step 11. + +The agent identified XOR key `0x42`, but only the captured prefix survived. It +then guessed the six missing rows from the alphabetical fruit names and +`value = id * 100` pattern, wrote replacement database/WAL files, and exported +that reconstruction. The final message disclosed the loss and guessing. + +Step 55 verified JSON shape, eleven sorted rows, and agreement between SQLite +and JSON. Those checks compared two artifacts derived from the same guesses. +They could not establish recovery of the original values. The official verifier +found row 1 still at `100`, while its committed WAL update required `150`; +`test_recovered_data_completeness` and `test_wal_was_decrypted` failed. The +failure therefore concerns data provenance and preservation, not JSON syntax +or a verifier dependency problem. + +A useful agent change is to preserve complete original inputs before opening +stateful formats, perform recovery on copies, and distinguish recovered facts +from inferred replacements. Verification must check committed updates as well +as inserted row counts. It must not treat agreement between two guessed +artifacts as evidence that the original data was recovered. + +## Failure: model conversion used a shared, unsupported assumption + +The supplied `model.py` defines a 784–16–16–10 ReLU network. The agent extracted +its weights, then searched preprocessing choices against public MNIST labels. +At steps 51/53, `(pixel / 255 - 0.5) / 0.5` achieved 92.07% label accuracy, so +the agent chose it over plain `pixel / 255`. This changed the requested model's +observable predictions to improve a different objective: digit-label accuracy. + +At steps 75/77/83/87, the C++ CLI matched a separately written Python reference +on 330 images, including small-margin cases. Both implementations read the +same inferred normalization from the generated weights file. This was useful +for detecting implementation discrepancies, but did not validate the assumption +itself. The agent's final message acknowledged that preprocessing was inferred. +The task instructions did not explicitly specify normalization, so this was a +real uncertainty that should have remained visible in its verification claim. + +The official verifier checked fixed model predictions and found 14 mismatches +among the first 50 MNIST test images. After all twelve outputs were frozen, a +separate diagnostic loaded the original float32 weights and implemented the +supplied architecture with standard-library float64 arithmetic: + +| Preprocessing | Mismatches against official expectations, first 50 images | +| --- | ---: | +| `pixel / 255` | 0 | +| `(pixel / 255 - 0.1307) / 0.3081` | 10 | +| `(pixel / 255 - 0.5) / 0.5` | 14 | + +The final variant reproduces every reported mismatch index. This confirms +preprocessing as a sufficient explanation for the observed failure. The +diagnostic used the original task model and a public MNIST fixture; their hashes +and all predictions are recorded in the results JSON. It did not edit the +submission, change the official score, or constitute another model trial. + +For model conversion, the agent should preserve source behavior and validate +the full input-to-output contract. An independent numerical implementation is +not an independent oracle when both programs share an unverified preprocessing +choice. Label accuracy should not substitute for behavioral equivalence. + +## Installation and verification accounting + +The original install-only preflight had five successes, six image-pull/start +errors, and one APT setup timeout. The second had eleven successes and one +partial uv download. Both complete histories remain in the +[first](../configs/dev12-baseline-20261011/preflight-v1-evidence.json) and +[second](../configs/dev12-baseline-20261011/preflight-v2-evidence.json) evidence +files. These eight preflight failures are separate from the twelve scored +trials. + +Before any model ran, plan v3 accepted exactly one successful native +installation per task: eleven from the second preflight and POV-Ray's success +from the first. The reusable orchestrator verified their task revision, source +snapshot, installed wheel, bootstrap state, installation-only mode, and artifact +hashes. No successful result was synthesized. The runtime, prompt, tasks, +budgets, and queue were unchanged across plan amendments. + +The branch also adds a verified cache for original public Docker Hub images. +Six stalled image downloads were completed through digest-checked parallel +ranges and imported without rebuilding or modifying images. Task scripts, +package sources, and official tests were preserved. All twelve verifier +dependency profiles remain **`not_configured`**: successful agent installation +was never reported as successful verifier-dependency preflight. + +The PyTorch verifier timed out once after 900 seconds during dependency setup; +its unchanged second attempt completed and reached the failing assertion. Every +other verifier completed on its first attempt. Each retry retained the native +per-attempt limit and the same agent output. All twelve final official suites +ran; there are no unresolved verifier errors to exclude from the denominator. + +The scored phase ran from **15:50:51 to 17:11:31 UTC**, approximately **80.7 +minutes** elapsed, excluding the earlier preflights. Summed across concurrent +trials, agent execution was 55.0 minutes, agent installation 39.9 minutes, and +verification 34.7 minutes. These sums are not wall-clock duration. The calendar +task alone spent 28.4 minutes installing, followed by 30.2 seconds of agent work. + +Recorded token-estimated cost totals **$0.600974**, with incomplete accounting: +PyTorch step 89 had a recoverable `RemoteProtocolError` and no usage record. +The sum covers known usage only, not an exact bill or complete total. Recorded +usage is 9,362,395 input tokens (9,171,328 cached) and 407,188 output tokens; +missing usage is not imputed as zero. This in-agent transport retry is separate +from the policy forbidding retries of entire model trials. + +## Audit boundaries and next experiments + +All twelve outputs had frozen by the observed checkpoint at 17:09:54 UTC, +before failure-content auditing began. Their trajectory hashes still match. +Final checks also matched the registered plan, prompt, task files, input +configurations, complete source snapshots, wheel digests, original image +identities/registry digests, and previously recorded completed-result hashes. +Queue barriers and verifier retry limits were verified from native artifacts. +No containers from the twelve scored trials remained after completion. + +Trace review found no observed retrieval of external benchmark solutions or +successful access to hidden verifier contents. POV-Ray used official upstream +archives; PyTorch used public MNIST images and labels, including examples later +used by the verifier. That overlap is recorded, not treated as benchmark-answer +retrieval. Several agents searched local files for references, and the Git and +database agents read bootstrap logs. These observations cannot establish that +the model had never encountered the benchmark in training. + +The database agent also read process environments at step 35, placing the +provider credential in its local trace. Raw logs must remain private or be +redacted before sharing. This report and the published JSON contain derived +observations and hashes, not raw environment values. Preventing credential +values from reaching tool observations is a separate agent-hardening follow-up. + +For a subsequent optimization experiment: + +1. Keep these twelve tasks intact. Use the two failures to develop preservation + and oracle-selection behavior; keep the ten passes as regression cases. +2. Require evidence for the decisive requirement: original data values for + recovery, source predictions for conversion. Track whether the oracle shares + assumptions or generated data with the implementation. +3. Add controlled development variants that change update values or input + contracts. Preserve the official baseline and label variants as custom tests. +4. Compare baseline and candidate with paired repeats and the same budgets; + measure task score, regressions, unsupported completion claims, and cost. + This run alone cannot show an improvement or estimate variance. +5. Leave the remaining 55 tasks unopened until a candidate and a fresh + evaluation protocol are fixed. Do not keep selecting fresh tasks after + seeing each development result. + +Infrastructure validation on this branch: **128 Harbor tests passed**, including +image-cache and accepted-preflight provenance tests. No agent optimization was +included in this baseline. diff --git a/benchmarks/harbor/results/dev12-main-baseline-20261011.json b/benchmarks/harbor/results/dev12-main-baseline-20261011.json new file mode 100644 index 0000000..0ce76fd --- /dev/null +++ b/benchmarks/harbor/results/dev12-main-baseline-20261011.json @@ -0,0 +1,1865 @@ +{ + "schema_version": 1, + "experiment": "dev12-main-baseline-20261011", + "collected_at": "2026-10-11T17:23:17.120884+00:00", + "cohort": "purposive_development", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "runtime_source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "plan_path": "benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json", + "plan_sha256": "9dcb64a3bbe76482dbb0aeb9297d2a1251071b704ad78e9f2b77ebc5040a2c9a", + "model_output_freeze": { + "observed_frozen_at": "2026-10-11T17:09:54.227372+00:00", + "model_outputs": 12, + "trials": [ + { + "task": "terminal-bench/cobol-modernization", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/agent/trajectory.json", + "trajectory_sha256": "4f8f83b3c39424085355f531f057d6479c145a52ed85f651be263ba292e7b04e", + "terminal": { + "status": "completed", + "duration_ms": 504192.364566, + "timestamp": "2026-10-11T15:59:56.042Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/agent/trajectory.json", + "trajectory_sha256": "7b9c7f7e56b664665dc86e4aeb960364b0548277a9b466b052683cb1cde49293", + "terminal": { + "status": "completed", + "duration_ms": 65406.544183, + "timestamp": "2026-10-11T15:52:31.589Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/nginx-request-logging", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/agent/trajectory.json", + "trajectory_sha256": "eae4b902126963ee4cddcdb378587f1795c83ffb3a6ed798c3151f7290199778", + "terminal": { + "status": "completed", + "duration_ms": 33079.926325, + "timestamp": "2026-10-11T16:01:44.208Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/git-multibranch", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/agent/trajectory.json", + "trajectory_sha256": "fddca6d5eb5b3bb341f539e1766a18e35367ab61082e198e740045a79a8e0300", + "terminal": { + "status": "completed", + "duration_ms": 231281.000847, + "timestamp": "2026-10-11T16:05:16.034Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/db-wal-recovery", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/agent/trajectory.json", + "trajectory_sha256": "99a8f85ab7c65218f67edf4c7a6ba15dd2ba2eae1bca0d4abdc74dcb66b7f1f9", + "terminal": { + "status": "completed", + "duration_ms": 300714.826752, + "timestamp": "2026-10-11T16:13:14.583Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/multi-source-data-merger", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/agent/trajectory.json", + "trajectory_sha256": "98254f523e1af69d46ce666756d5d0a0d735663a4a2651a3fa9534b70214dc2e", + "terminal": { + "status": "completed", + "duration_ms": 54941.240413, + "timestamp": "2026-10-11T16:08:57.994Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/query-optimize", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/agent/trajectory.json", + "trajectory_sha256": "d1e4af3f2b506732e2a39901e6608c88697f3e21b0202f26638b346abd40de6d", + "terminal": { + "status": "completed", + "duration_ms": 535870.372246, + "timestamp": "2026-10-11T16:28:00.417Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/adaptive-rejection-sampler", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/agent/trajectory.json", + "trajectory_sha256": "14c739071f532119612b23f97ceab1d79cdedd6e7f4eb23307b1627e38540019", + "terminal": { + "status": "completed", + "duration_ms": 565096.307358, + "timestamp": "2026-10-11T16:28:36.521Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/pytorch-model-cli", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/agent/trajectory.json", + "trajectory_sha256": "ac3a14da4dc07a62b48fb509f094003fc0913dfb1a05d65b3a44aa026e3e9bc4", + "terminal": { + "status": "completed", + "duration_ms": 297760.896708, + "timestamp": "2026-10-11T16:40:24.130Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/constraints-scheduling", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/agent/trajectory.json", + "trajectory_sha256": "2d1974f6890199437dc2a8719006f553907a8a9976c518a4a70b8aa13091b44d", + "terminal": { + "status": "completed", + "duration_ms": 29296.936721, + "timestamp": "2026-10-11T16:59:47.232Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/custom-memory-heap-crash", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/agent/trajectory.json", + "trajectory_sha256": "633e9240d85490bfba3bbe2fdc074f3a860b5da1bdd6fa545e21b78b83ac5ad7", + "terminal": { + "status": "completed", + "duration_ms": 279820.513108, + "timestamp": "2026-10-11T17:05:50.190Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + }, + { + "task": "terminal-bench/build-pov-ray", + "trajectory_path": "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/agent/trajectory.json", + "trajectory_sha256": "2a0e42851a8c3b0c8ee45f0dc3db1882eae3ba72cfd1fa883b181200d20588b3", + "terminal": { + "status": "completed", + "duration_ms": 392133.312638, + "timestamp": "2026-10-11T17:07:50.736Z", + "timestamp_source": "source_timestamp", + "outcome": "completed" + } + } + ] + }, + "integrity": { + "registration_unchanged": true, + "plan_source_prompt_configs_tasks_unchanged": true, + "accepted_preflights_verified": 12, + "source_snapshots_verified": 12, + "wheel_digests_verified": 12, + "trajectory_freeze_verified": 12, + "earlier_completed_artifact_hashes_verified": 8, + "all_native_test_suites_executed": true, + "queue_pair_barriers_verified": true, + "whole_model_trial_retries": 0, + "outcome_based_replacements": 0 + }, + "aggregate": { + "trials": 12, + "raw_passes": 10, + "scored_failures": 2, + "setup_errors": 0, + "verifier_errors": 0, + "unsupported_verifier_dependency_profiles": 12, + "pass_rate": 0.8333333333333334, + "verifier_timeout_retries": 1, + "estimated_known_cost_usd": 0.600973668, + "cost_complete": false, + "usage_incomplete_tasks": [ + "terminal-bench/pytorch-model-cli" + ], + "recorded_tokens": { + "prompt_tokens": 9362395, + "completion_tokens": 407188, + "cached_tokens": 9171328 + }, + "summed_stage_seconds": { + "environment_setup": 11.621372000000001, + "agent_setup": 2395.148702, + "agent_execution": 3300.751312, + "verifier": 2081.279393 + }, + "model_job_wall_seconds": 4839.714099 + }, + "trials": [ + { + "order": 1, + "task": "terminal-bench/cobol-modernization", + "task_ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "trial": "cobol-modernization__GQnVApW", + "job": "dev12-cobol-modernization-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791734414.6418216, + "stop": 1791734414.689376 + }, + "work_budget_seconds": 720, + "started_at": "2026-10-11T15:50:51.866843Z", + "finished_at": "2026-10-11T16:00:25.509548Z", + "seconds": { + "environment_setup": 0.933953, + "agent_setup": 38.24792, + "agent_execution": 505.160568, + "verifier": 18.268325 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 18.05617724900003 + } + ], + "verifier_timeout_seconds": 900.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 47, + "incomplete_model_calls": [], + "tool_calls": 47, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 2698556, + "completion_tokens": 82809, + "cached_tokens": 2672896 + }, + "estimated_known_cost_usd": 0.123106176, + "final_metrics": { + "total_steps": 95, + "total_prompt_tokens": 2698556, + "total_completion_tokens": 82809, + "total_cached_tokens": 2672896, + "total_cost_usd": 0.123106176, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.123106176 + } + } + }, + "task_checksum": "5e3806427eaaaa9f0f756a2a87d4f7f50ec1c8a0d329b26bab7dc09657b937cf", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/result.json": "ef3d278f8f7fb0a7af1b633dcf7266339016a620fcb7d77d38246e4cfcb73fc1", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/agent/trajectory.json": "4f8f83b3c39424085355f531f057d6479c145a52ed85f651be263ba292e7b04e", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/agent/bootstrap.json": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/verifier/retry-summary.json": "66e9157d51143a3d288ff4bcde44b055699fd01684ed07a7d2988b80af5fa076", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/verifier/test-stdout.txt": "f10b9a00a36e6d2fbf8bc3951c0b1630b91de506bdf0231a64ec95c6771cad1d", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/verifier/ctrf.json": "ae2aeb35fa7b1952ed2f64ff83cb1aa2341d1a3cb5a49b0d74c57bb56b33477e", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011-input/manifest.json": "a9ec1f89510ffacff9bef73cbf4decd7352edbd2bc54d2911382c1bdf5841b09", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011-input/config.json": "43e5dc00efba212c737ae12ccec37fd6e2e7da355f8b3b5f35f768089771d653", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-cobol-modernization-baseline-20261011.json": "94101e9a1f8045ea8791596acea9f2c8d993d889d7f61f269633350a77d9dcd7", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/verifier/attempts/1/ctrf.json": "ae2aeb35fa7b1952ed2f64ff83cb1aa2341d1a3cb5a49b0d74c57bb56b33477e", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-cobol-modernization-baseline-20261011/cobol-modernization__GQnVApW/verifier/attempts/1/test-stdout.txt": "f10b9a00a36e6d2fbf8bc3951c0b1630b91de506bdf0231a64ec95c6771cad1d" + } + }, + { + "order": 2, + "task": "terminal-bench/fix-code-vulnerability", + "task_ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "trial": "fix-code-vulnerability__8PT7Mmq", + "job": "dev12-fix-code-vulnerability-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791733961.4188828, + "stop": 1791733961.4372432 + }, + "work_budget_seconds": 720, + "started_at": "2026-10-11T15:50:51.470967Z", + "finished_at": "2026-10-11T15:52:52.233260Z", + "seconds": { + "environment_setup": 0.9524, + "agent_setup": 33.038588, + "agent_execution": 66.275879, + "verifier": 9.522895 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 9.333147841000027 + } + ], + "verifier_timeout_seconds": 900.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 16, + "incomplete_model_calls": [], + "tool_calls": 21, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 162083, + "completion_tokens": 7594, + "cached_tokens": 151424 + }, + "estimated_known_cost_usd": 0.013219044, + "final_metrics": { + "total_steps": 33, + "total_prompt_tokens": 162083, + "total_completion_tokens": 7594, + "total_cached_tokens": 151424, + "total_cost_usd": 0.013219044, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.013219044 + } + } + }, + "task_checksum": "80d707a96acfd0e19af96f0f0c8f44deefdbf167538fefbc0d1dc6391392090a", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/result.json": "53b9cf6b726ac1c7131ad7eabef0f4ce9e8e7c54134206ab0b02b29e69b39d2d", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/agent/trajectory.json": "7b9c7f7e56b664665dc86e4aeb960364b0548277a9b466b052683cb1cde49293", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/agent/bootstrap.json": "8d1fa3e359c1a2c87bb9b262c9113a423af3445aa5a20da8a2e6fb4d63b4f67a", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/verifier/retry-summary.json": "6da7d47a863e50dd5b2e910823ddbb81a6b2910ec0fc80c308d54c3bf532c4e3", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/verifier/test-stdout.txt": "5b1c1b620157e4e7ee7e5368a1cd9f8de20235ae042eaeeb7f266332d7e38f69", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/verifier/ctrf.json": "999592c0d6e4cfa5516e44db1e208a3f3df9f0f50356a9dd9968fdd1598e801d", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011-input/manifest.json": "20b441c2dfc5a7c9ece86670859a49d8f2db355165c400b691d45d69e539babd", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011-input/config.json": "1f6958d0ca945eca970719d64a87a2dc5994bc0254cd71937060689c3863d5de", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-fix-code-vulnerability-baseline-20261011.json": "c2f0153ace0b2eea1fc2597055651f836dd1c7b60f2ddd2ff3c14b903022cb0b", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/verifier/attempts/1/ctrf.json": "999592c0d6e4cfa5516e44db1e208a3f3df9f0f50356a9dd9968fdd1598e801d", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/verifier/attempts/1/original-repo-ctrf.json": "f4fc9648eec7ccf9a0891f567624303f7c6c6636b1cbec27917c92e8d3068eed", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-fix-code-vulnerability-baseline-20261011/fix-code-vulnerability__8PT7Mmq/verifier/attempts/1/test-stdout.txt": "5b1c1b620157e4e7ee7e5368a1cd9f8de20235ae042eaeeb7f266332d7e38f69" + } + }, + { + "order": 3, + "task": "terminal-bench/nginx-request-logging", + "task_ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "trial": "nginx-request-logging__EabhPTP", + "job": "dev12-nginx-request-logging-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 8, + "passed": 8, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791734525.1599152, + "stop": 1791734527.3533282 + }, + "work_budget_seconds": 720, + "started_at": "2026-10-11T16:00:31.803481Z", + "finished_at": "2026-10-11T16:02:18.275923Z", + "seconds": { + "environment_setup": 1.033164, + "agent_setup": 37.474728, + "agent_execution": 34.05106, + "verifier": 22.80059 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 22.611146957999154 + } + ], + "verifier_timeout_seconds": 900.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 11, + "incomplete_model_calls": [], + "tool_calls": 14, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 57586, + "completion_tokens": 3408, + "cached_tokens": 53120 + }, + "estimated_known_cost_usd": 0.00574812, + "final_metrics": { + "total_steps": 23, + "total_prompt_tokens": 57586, + "total_completion_tokens": 3408, + "total_cached_tokens": 53120, + "total_cost_usd": 0.00574812, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.00574812 + } + } + }, + "task_checksum": "913305d8f286ff121b30e4893217142190b27f12952343f22c0999dd2a4d725e", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/result.json": "9a431a143f83c13c17ad3c67d2bafaab9f7f90e7d0c7e3dd1b535ca28ee8c2b1", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/agent/trajectory.json": "eae4b902126963ee4cddcdb378587f1795c83ffb3a6ed798c3151f7290199778", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/agent/bootstrap.json": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/verifier/retry-summary.json": "99d7c7ccea0083dd20dcfbfda613b2501c9bd7f9ef5553862100d9e8a8cf6159", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/verifier/test-stdout.txt": "eb7475abfeecda63ffb1c40f78f99d8cfcba9aea5ca883f5f1fb8cdc63f6f839", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/verifier/ctrf.json": "b9bc0576b31ed205d3ea2fefd33d13542f23ad074d3972972d40a38f71af897d", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011-input/manifest.json": "ae8579625d2ceaf69be0f948815a88b2a71fd5c578d2344705c2c2a333aa92a6", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011-input/config.json": "262ad1af0bc0cc6271a0c66ec6cc3c206fbdbb6aca6bcfdb47556e018131b3d0", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-nginx-request-logging-baseline-20261011.json": "83cf7476987ec4d90ef54bffd87f75579128f1dfc0f0b2b7b45067eedae351dc", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/verifier/attempts/1/ctrf.json": "b9bc0576b31ed205d3ea2fefd33d13542f23ad074d3972972d40a38f71af897d", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-nginx-request-logging-baseline-20261011/nginx-request-logging__EabhPTP/verifier/attempts/1/test-stdout.txt": "eb7475abfeecda63ffb1c40f78f99d8cfcba9aea5ca883f5f1fb8cdc63f6f839" + } + }, + { + "order": 4, + "task": "terminal-bench/git-multibranch", + "task_ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "trial": "git-multibranch__69UpsQX", + "job": "dev12-git-multibranch-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791734736.7768967, + "stop": 1791734827.94831 + }, + "work_budget_seconds": 720, + "started_at": "2026-10-11T16:00:31.786222Z", + "finished_at": "2026-10-11T16:07:18.794990Z", + "seconds": { + "environment_setup": 1.010187, + "agent_setup": 51.263702, + "agent_execution": 232.131266, + "verifier": 111.543161 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 111.34293115500077 + } + ], + "verifier_timeout_seconds": 900.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 22, + "incomplete_model_calls": [], + "tool_calls": 28, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 330601, + "completion_tokens": 15423, + "cached_tokens": 319232 + }, + "estimated_known_cost_usd": 0.023833692, + "final_metrics": { + "total_steps": 45, + "total_prompt_tokens": 330601, + "total_completion_tokens": 15423, + "total_cached_tokens": 319232, + "total_cost_usd": 0.023833692, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.023833692 + } + } + }, + "task_checksum": "abe8a7f8f9dcb0e2170740e4735559b368cecd925ddee09c9ed81a7c67dce90a", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/result.json": "cb58d7f495ff58ee47582d995d8dff1c49128504f95a158989b5a6df841a06e2", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/agent/trajectory.json": "fddca6d5eb5b3bb341f539e1766a18e35367ab61082e198e740045a79a8e0300", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/agent/bootstrap.json": "391e000959d10d8aa551310ebe7bddd0c70185a684ee8cb543b7e8bf57e5e63b", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/verifier/retry-summary.json": "a1f62e3c9cb0d3870e9180b7d130f06ddfba26d49bbec118dd3dff07eec117f4", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/verifier/test-stdout.txt": "86aaa4516464d803d36759ab0ad510b1ae1f46dfc345dfacb929e60b2a014748", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/verifier/ctrf.json": "280c06fffd6f521a3e13bcbbd8f339f858d19b6e212f44d6ae25ee6a206f1c51", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011-input/manifest.json": "97d82709aadd19c986a324d22b13608104a091cd70f3a85e55aef0c00590db11", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011-input/config.json": "9c234f1e18fb14666fa52447f5b81266a57d88184b411a5dda7cc05c3b3212a5", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-git-multibranch-baseline-20261011.json": "b827bcbc0fad918f546cf7f52df2abcedc0649473493b4275fc377ec9b68d796", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/verifier/attempts/1/ctrf.json": "280c06fffd6f521a3e13bcbbd8f339f858d19b6e212f44d6ae25ee6a206f1c51", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-git-multibranch-baseline-20261011/git-multibranch__69UpsQX/verifier/attempts/1/test-stdout.txt": "86aaa4516464d803d36759ab0ad510b1ae1f46dfc345dfacb929e60b2a014748" + } + }, + { + "order": 5, + "task": "terminal-bench/db-wal-recovery", + "task_ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "trial": "db-wal-recovery__vQr4RTJ", + "job": "dev12-db-wal-recovery-baseline-20261011", + "raw_reward": 0.0, + "classification": "scored_failure", + "exception": null, + "official_test_summary": { + "tests": 7, + "passed": 5, + "failed": 2, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791735481.9039369, + "stop": 1791735481.930828 + }, + "work_budget_seconds": 720, + "started_at": "2026-10-11T16:07:27.182279Z", + "finished_at": "2026-10-11T16:18:12.844155Z", + "seconds": { + "environment_setup": 0.925319, + "agent_setup": 44.577398, + "agent_execution": 302.041538, + "verifier": 287.027836 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 286.8307059710005 + } + ], + "verifier_timeout_seconds": 900.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 28, + "incomplete_model_calls": [], + "tool_calls": 37, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 980554, + "completion_tokens": 54949, + "cached_tokens": 954624 + }, + "estimated_known_cost_usd": 0.079445544, + "final_metrics": { + "total_steps": 57, + "total_prompt_tokens": 980554, + "total_completion_tokens": 54949, + "total_cached_tokens": 954624, + "total_cost_usd": 0.079445544, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.079445544 + } + } + }, + "task_checksum": "c18abdc4fdc3a01bf374c55a9700708fe6a9662077d29db81abb692f0a3c5f6f", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/result.json": "8554a35eb18153678c22b6a66a80a209a8b29e515d5e5765d6c46496b4c7b64c", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/agent/trajectory.json": "99a8f85ab7c65218f67edf4c7a6ba15dd2ba2eae1bca0d4abdc74dcb66b7f1f9", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/agent/bootstrap.json": "2ffe3efbf9fd9f423c3fc82102152f59af7f9180f8aaf39c46e61c96e2f02d76", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/verifier/retry-summary.json": "e7e88e91a81f764ee441faf6f8a45a1f0ad31e1e4eb03df465d57bc16955ec43", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/verifier/test-stdout.txt": "64d88e5e6fef9ba8484dbbde478a1f1e9fd5a4218a44473e5fa34eb9aa2d94d6", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/verifier/ctrf.json": "68352fe9414a7e314215756dd4c5f2cc35a69fd783f34c698aa0d90e6942a3cd", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011-input/manifest.json": "48cf72c126f4bb349feb89bcf8e1c843b97d56d2295eb04300bd84002f7f86b8", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011-input/config.json": "b450c1e0756f85f1e56c7434f4fee72483c50fd26ccab908a0da7e1c3dbc3ef3", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-db-wal-recovery-baseline-20261011.json": "939be6d5761096e1df9ab3f83b0b29e30beae7bdfbbf30252c24d0889febbe3d", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/verifier/attempts/1/ctrf.json": "68352fe9414a7e314215756dd4c5f2cc35a69fd783f34c698aa0d90e6942a3cd", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev12-baseline-main-source/jobs/dev12-db-wal-recovery-baseline-20261011/db-wal-recovery__vQr4RTJ/verifier/attempts/1/test-stdout.txt": "64d88e5e6fef9ba8484dbbde478a1f1e9fd5a4218a44473e5fa34eb9aa2d94d6" + } + }, + { + "order": 6, + "task": "terminal-bench/multi-source-data-merger", + "task_ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "trial": "multi-source-data-merger__H5tGDAu", + "job": "dev12-multi-source-data-merger-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791735045.8595047, + "stop": 1791735046.8771257 + }, + "work_budget_seconds": 720, + "started_at": "2026-10-11T16:07:26.456836Z", + "finished_at": "2026-10-11T16:10:58.054981Z", + "seconds": { + "environment_setup": 0.968286, + "agent_setup": 34.912998, + "agent_execution": 55.807942, + "verifier": 108.678375 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 108.00855693199992 + } + ], + "verifier_timeout_seconds": 900.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 10, + "incomplete_model_calls": [], + "tool_calls": 10, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 88329, + "completion_tokens": 9546, + "cached_tokens": 83456 + }, + "estimated_known_cost_usd": 0.013417836, + "final_metrics": { + "total_steps": 21, + "total_prompt_tokens": 88329, + "total_completion_tokens": 9546, + "total_cached_tokens": 83456, + "total_cost_usd": 0.013417836, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.013417836 + } + } + }, + "task_checksum": "33fa3b988ff60ec62b6ce40ee455208cb083ac1ac46ddc5247e954c88b9d5e8e", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/result.json": "bf39627d51baef3ad24da8b801916d6612c2a5c59122da0394a8dc53a39cf042", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/agent/trajectory.json": "98254f523e1af69d46ce666756d5d0a0d735663a4a2651a3fa9534b70214dc2e", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/agent/bootstrap.json": "43369761e1c62328eb0c3890c99d441d724fe366a178abf9c568928eb85c4174", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/verifier/retry-summary.json": "9cba27413140de8aab08f28c7cdd299f371e776ee35f976ca1b6427e3c5bf53f", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/verifier/test-stdout.txt": "59e3a95584098847fca0c5263180174b6216f17028c800710c7e32198d1eec1a", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/verifier/ctrf.json": "694f1880784dfaa07c1c2f3d3869ded9385422222de599a1daec8a844ecc838c", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011-input/manifest.json": "054ba4498970383fe0ff6e14d30d04354b0126cce80ba5da094da500bc3e2793", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011-input/config.json": "caf972eaef443375ef4e532fd413bf6a871e98a0f0c2561a07b3b6d5470091d6", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-multi-source-data-merger-baseline-20261011.json": "c535138aa69a4a2afd0f0d09ddaaf8e93f19e00c8693992afbdf030f8b90b162", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/verifier/attempts/1/ctrf.json": "694f1880784dfaa07c1c2f3d3869ded9385422222de599a1daec8a844ecc838c", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-multi-source-data-merger-baseline-20261011/multi-source-data-merger__H5tGDAu/verifier/attempts/1/test-stdout.txt": "59e3a95584098847fca0c5263180174b6216f17028c800710c7e32198d1eec1a" + } + }, + { + "order": 7, + "task": "terminal-bench/query-optimize", + "task_ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "trial": "query-optimize__xW8CdxE", + "job": "dev12-query-optimize-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791736097.5172405, + "stop": 1791736228.6854308 + }, + "work_budget_seconds": 720, + "started_at": "2026-10-11T16:18:20.667016Z", + "finished_at": "2026-10-11T16:30:39.573437Z", + "seconds": { + "environment_setup": 0.965353, + "agent_setup": 42.219075, + "agent_execution": 536.72064, + "verifier": 147.905097 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 147.62170704900018 + } + ], + "verifier_timeout_seconds": 1800.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 22, + "incomplete_model_calls": [], + "tool_calls": 29, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 479388, + "completion_tokens": 24381, + "cached_tokens": 459008 + }, + "estimated_known_cost_usd": 0.038125248, + "final_metrics": { + "total_steps": 45, + "total_prompt_tokens": 479388, + "total_completion_tokens": 24381, + "total_cached_tokens": 459008, + "total_cost_usd": 0.038125248, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.038125248 + } + } + }, + "task_checksum": "593405b6a2f6970f0f4c2ca2aff455a3b32bce3eecf8b506cfeeb0fe93b14838", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/result.json": "a22011a054a45ffb2d3ee876630e77e83718bb017edb98415b4c040dbf21208a", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/agent/trajectory.json": "d1e4af3f2b506732e2a39901e6608c88697f3e21b0202f26638b346abd40de6d", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/agent/bootstrap.json": "0a52c1c56dbec333ab809d00b69469021e06f49e0570249b280b6aacef2587f0", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/verifier/retry-summary.json": "5e3b743fa2e363f6638a666bbd32d5fc8c1517443b0673b0d3ce5102cbcf7dc5", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/verifier/test-stdout.txt": "ef549e4b73a37d7bfb4fbfb7b1a4dcf6790f97efb4f49b2e44eddab30fe0d583", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/verifier/ctrf.json": "477608f55e8083337dda43de81168129271df20fe3e6ec36726c785933bf4feb", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011-input/manifest.json": "30be5279642e73ce991102a668792c93e9a3185fb63579899106040fe53c5f56", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011-input/config.json": "172d1e2fb4b548aacab61cd1b74e46ac8df6d01d92bea3274a8ad7422b7585d6", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-query-optimize-baseline-20261011.json": "b2977822e159e02c4c71cf7a0bd18c8b2840632b854a6aeb3a072f467bf38d1b", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/verifier/attempts/1/ctrf.json": "477608f55e8083337dda43de81168129271df20fe3e6ec36726c785933bf4feb", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-query-optimize-baseline-20261011/query-optimize__xW8CdxE/verifier/attempts/1/test-stdout.txt": "ef549e4b73a37d7bfb4fbfb7b1a4dcf6790f97efb4f49b2e44eddab30fe0d583" + } + }, + { + "order": 8, + "task": "terminal-bench/adaptive-rejection-sampler", + "task_ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "trial": "adaptive-rejection-sampler__AbftRyA", + "job": "dev12-adaptive-rejection-sampler-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 9, + "passed": 9, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791736140.2109442, + "stop": 1791736144.8185015 + }, + "work_budget_seconds": 720, + "started_at": "2026-10-11T16:18:20.643245Z", + "finished_at": "2026-10-11T16:29:15.879694Z", + "seconds": { + "environment_setup": 0.996828, + "agent_setup": 49.072429, + "agent_execution": 565.986258, + "verifier": 27.949146 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 27.708376808999674 + } + ], + "verifier_timeout_seconds": 900.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 11, + "incomplete_model_calls": [], + "tool_calls": 11, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 250525, + "completion_tokens": 102009, + "cached_tokens": 245632 + }, + "estimated_known_cost_usd": 0.125352492, + "final_metrics": { + "total_steps": 24, + "total_prompt_tokens": 250525, + "total_completion_tokens": 102009, + "total_cached_tokens": 245632, + "total_cost_usd": 0.125352492, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.125352492 + } + } + }, + "task_checksum": "a4726253ad23c3fc46b5cb9c3d21025874e367b18643a3920f636a1fa89e1207", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/result.json": "bf64380d0c4c4cdaf6492c30dbb8ae31752a53e46d00d01433706dcf70ae079e", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/agent/trajectory.json": "14c739071f532119612b23f97ceab1d79cdedd6e7f4eb23307b1627e38540019", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/agent/bootstrap.json": "eb4895cdc7076ee2da4b9519eb1d623465f7c0828b9afb96a5d685f89a8c1f1b", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/verifier/retry-summary.json": "3b768b047c61501aebbd6a5158cc0a79708b46123cfb79840c6c6ccf2ffd7630", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/verifier/test-stdout.txt": "97f76fc515fef3921a46ec13cb78f2cbf66ec54dbd3540fe1373ba4bd67bc0ec", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/verifier/ctrf.json": "9ef2a1b7e71afe901b69a01f156665f5d42b46b5c59154ce64d6be20e93b8456", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011-input/manifest.json": "de657438e6731aae6e4f50708044294a344cc40245c93bb9a16d468aa1587c01", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011-input/config.json": "71ffc80b450b239ea585ee4d269ff6e77e47dd582506ec98162adc502f6e15b7", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-adaptive-rejection-sampler-baseline-20261011.json": "4b31628d579b050eaa9df0de522a82c7a4efdb3042e5634322d40e0d07e8f4a9", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/verifier/attempts/1/ctrf.json": "9ef2a1b7e71afe901b69a01f156665f5d42b46b5c59154ce64d6be20e93b8456", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-adaptive-rejection-sampler-baseline-20261011/adaptive-rejection-sampler__AbftRyA/verifier/attempts/1/test-stdout.txt": "97f76fc515fef3921a46ec13cb78f2cbf66ec54dbd3540fe1373ba4bd67bc0ec" + } + }, + { + "order": 9, + "task": "terminal-bench/pytorch-model-cli", + "task_ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "trial": "pytorch-model-cli__asD2fTE", + "job": "dev12-pytorch-model-cli-baseline-20261011", + "raw_reward": 0.0, + "classification": "scored_failure", + "exception": null, + "official_test_summary": { + "tests": 6, + "passed": 5, + "failed": 1, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791737768.6626732, + "stop": 1791737790.0538428 + }, + "work_budget_seconds": 720, + "started_at": "2026-10-11T16:30:56.828069Z", + "finished_at": "2026-10-11T16:56:42.340334Z", + "seconds": { + "environment_setup": 0.964518, + "agent_setup": 267.780007, + "agent_execution": 298.721879, + "verifier": 966.520263 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "timeout", + "duration_seconds": 900.0019214439999 + }, + { + "attempt": 2, + "status": "completed", + "return_code": 0, + "duration_seconds": 66.32247663299859 + } + ], + "verifier_timeout_seconds": 900.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 46, + "incomplete_model_calls": [ + { + "step_id": 89, + "model_call_id": "model-38d54f7b-2d63-4ee3-a21a-98060dc808d7", + "error_type": "RemoteProtocolError", + "message": "peer closed connection without sending complete message body (incomplete chunked read)", + "generation_id": null, + "duration_ms": 1107.911214, + "will_retry": true, + "retry_delay_seconds": 1.0, + "source_timestamp": "2026-10-11T16:40:12.982Z", + "incomplete": true, + "started_at": "2026-10-11T16:40:11.874Z", + "started_at_source": "source_timestamp", + "timestamp_source": "source_timestamp" + } + ], + "tool_calls": 48, + "accounting": { + "usage_complete": false, + "recorded_tokens": { + "prompt_tokens": 1405137, + "completion_tokens": 37894, + "cached_tokens": 1384704 + }, + "estimated_known_cost_usd": 0.059910924, + "final_metrics": { + "total_steps": 93, + "extra": { + "usage_complete": false, + "known_cost_usd": 0.059910924, + "cost_is_partial": true, + "cost_is_estimated": true, + "estimated_cost_usd": 0.059910924 + } + } + }, + "task_checksum": "e8438573f503c2d98ad71905862638bfa485470010d15d5b092821e0096b9daa", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/result.json": "1e8182f609eaa0711cf791e3d0974fe380d6effe11447f9263483291bcc1d00c", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/agent/trajectory.json": "ac3a14da4dc07a62b48fb509f094003fc0913dfb1a05d65b3a44aa026e3e9bc4", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/verifier/retry-summary.json": "ac94dd7803a3444e1c1843a015e8a8bdb14cef580445144661fd28cbdaedc2bd", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/verifier/test-stdout.txt": "13431458fea4c92034f42c1caf51f2b8a607e1b8dddcd2c232902dc035f234f6", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/verifier/ctrf.json": "f5a8d18263e441da6ac66499220f9e14b954513c26ed1986ec43cf61bd98d299", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011-input/manifest.json": "a76f53880d39c493ec5e968691aa10639ff6622c35f5fb08db0673fe32787ff9", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011-input/config.json": "52fe8b50f8814caf106afc4c4e022363f39a64185f9c98171dfec1357c29f7d7", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-pytorch-model-cli-baseline-20261011.json": "8cdb2774c45999f9285460ea463ca13840a1b66836593283d3d769676613fe6f", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/verifier/attempts/1/test-stdout.txt": "ad3cc444bf23a016870f0b5d03f09e3e3c0a4010de2cb88f6583a88035501654", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/verifier/attempts/2/ctrf.json": "f5a8d18263e441da6ac66499220f9e14b954513c26ed1986ec43cf61bd98d299", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/verifier/attempts/2/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev12-baseline-main-source/jobs/dev12-pytorch-model-cli-baseline-20261011/pytorch-model-cli__asD2fTE/verifier/attempts/2/test-stdout.txt": "13431458fea4c92034f42c1caf51f2b8a607e1b8dddcd2c232902dc035f234f6" + } + }, + { + "order": 10, + "task": "terminal-bench/constraints-scheduling", + "task_ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "trial": "constraints-scheduling__NPnsKa5", + "job": "dev12-constraints-scheduling-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791738006.8966916, + "stop": 1791738007.2491832 + }, + "work_budget_seconds": 1020, + "started_at": "2026-10-11T16:30:54.073053Z", + "finished_at": "2026-10-11T17:00:18.177450Z", + "seconds": { + "environment_setup": 0.985738, + "agent_setup": 1702.177106, + "agent_execution": 30.152862, + "verifier": 19.670661 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 19.47129868399861 + } + ], + "verifier_timeout_seconds": 1200.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 6, + "incomplete_model_calls": [], + "tool_calls": 5, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 41911, + "completion_tokens": 5424, + "cached_tokens": 38016 + }, + "estimated_known_cost_usd": 0.007905396, + "final_metrics": { + "total_steps": 13, + "total_prompt_tokens": 41911, + "total_completion_tokens": 5424, + "total_cached_tokens": 38016, + "total_cost_usd": 0.007905396, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.007905396 + } + } + }, + "task_checksum": "5b311f974587adf67804532222c59395f41a6c984b54a5e96a56dd071ee801dc", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/result.json": "387f595a2e3e2583fcdd129ee0e2c39fb00cf36a5952350020e631a90cbf68ab", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/agent/trajectory.json": "2d1974f6890199437dc2a8719006f553907a8a9976c518a4a70b8aa13091b44d", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/agent/bootstrap.json": "c4d5399fb99e22fdaccbafaeca054f4d615062b69ddc2389af0ed1c19ba30325", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/verifier/retry-summary.json": "9e231af8730c507475558b0227f79427c294137e58dc76be04e77a8bbb6e4608", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/verifier/test-stdout.txt": "662f91bddfdd0ab684fde0983e4a47fe75c64990c9680373dd34d0bcc53e7f31", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/verifier/ctrf.json": "a5eea20278c556046d582c0e11e7ba3b326be02dc074c858a95d4bd715fe6449", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011-input/manifest.json": "3c54335b46550168c314fee8df3dd6c9b395f8fd749aa85869c229aceca23efc", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011-input/config.json": "72539609a630a73c5a417ee4ffec5bb8ad123b8cce58f60dc4b8fd88c9f769c9", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-constraints-scheduling-baseline-20261011.json": "3d6b4a3170e9294d2e707e891f261d527ce9ba43d75717578c06e878123fe673", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/verifier/attempts/1/ctrf.json": "a5eea20278c556046d582c0e11e7ba3b326be02dc074c858a95d4bd715fe6449", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-constraints-scheduling-baseline-20261011/constraints-scheduling__NPnsKa5/verifier/attempts/1/test-stdout.txt": "662f91bddfdd0ab684fde0983e4a47fe75c64990c9680373dd34d0bcc53e7f31" + } + }, + { + "order": 11, + "task": "terminal-bench/custom-memory-heap-crash", + "task_ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "trial": "custom-memory-heap-crash__sUefumj", + "job": "dev12-custom-memory-heap-crash-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791738676.7126513, + "stop": 1791738680.335953 + }, + "work_budget_seconds": 1620, + "started_at": "2026-10-11T17:00:25.666527Z", + "finished_at": "2026-10-11T17:11:31.185066Z", + "seconds": { + "environment_setup": 0.932353, + "agent_setup": 43.067384, + "agent_execution": 280.667347, + "verifier": 329.799339 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 329.5842780359999 + } + ], + "verifier_timeout_seconds": 1800.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 33, + "incomplete_model_calls": [], + "tool_calls": 49, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 1153770, + "completion_tokens": 33784, + "cached_tokens": 1128960 + }, + "estimated_known_cost_usd": 0.05475756, + "final_metrics": { + "total_steps": 67, + "total_prompt_tokens": 1153770, + "total_completion_tokens": 33784, + "total_cached_tokens": 1128960, + "total_cost_usd": 0.05475756, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.05475756 + } + } + }, + "task_checksum": "a0db695f342c1b72f1b3d0b6c234eb802e970c4758931355e4ceeea4c9e73c37", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/result.json": "464c0fb5a1d54336d978e84bd182545767bbc95a04d00c024826badb0b62ae7e", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/agent/trajectory.json": "633e9240d85490bfba3bbe2fdc074f3a860b5da1bdd6fa545e21b78b83ac5ad7", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/agent/bootstrap.json": "cf6bb84c42557ba2a7b7e0ec653bbc1c7cf86c236d1b7e2a6852918dfb8eb7dc", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/verifier/retry-summary.json": "c0e89aff5086b8fad0b1070b14253cf8510398bad2f85cc94a30269688a78e83", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/verifier/test-stdout.txt": "14f6704998da0aa999e43e00070c9294226313885c5ea80a07a43ca2b998d79f", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/verifier/ctrf.json": "f6a20756708b696647523c2aab2e1d11f092fe2ef4f353dbc322fd5a35eaf4be", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011-input/manifest.json": "59d97783114ebe4fa9f673a83d494a11f954a7e2695afba00249d58a21cc2645", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011-input/config.json": "199419c73ec813a4aab83df11085e36b5d51cc1aac7c410901bec30650e90783", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-custom-memory-heap-crash-baseline-20261011.json": "998ab1c22ebd6015842c0f9de6181544a9b61c00077c07d1709e1f124b79a7ef", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/verifier/attempts/1/ctrf.json": "f6a20756708b696647523c2aab2e1d11f092fe2ef4f353dbc322fd5a35eaf4be", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-custom-memory-heap-crash-baseline-20261011/custom-memory-heap-crash__sUefumj/verifier/attempts/1/test-stdout.txt": "14f6704998da0aa999e43e00070c9294226313885c5ea80a07a43ca2b998d79f" + } + }, + { + "order": 12, + "task": "terminal-bench/build-pov-ray", + "task_ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "trial": "build-pov-ray__gSzbLYX", + "job": "dev12-build-pov-ray-baseline-20261011", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791738494.844755, + "stop": 1791738502.6303985 + }, + "work_budget_seconds": 11820, + "started_at": "2026-10-11T17:00:25.604761Z", + "finished_at": "2026-10-11T17:08:33.715498Z", + "seconds": { + "environment_setup": 0.953273, + "agent_setup": 51.317367, + "agent_execution": 393.034073, + "verifier": 31.593705 + }, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 31.405323345999932 + } + ], + "verifier_timeout_seconds": 12000.0, + "verifier_dependency_profile": "not_configured", + "recorded_agent_steps": 53, + "incomplete_model_calls": [], + "tool_calls": 59, + "accounting": { + "usage_complete": true, + "recorded_tokens": { + "prompt_tokens": 1713955, + "completion_tokens": 29967, + "cached_tokens": 1680256 + }, + "estimated_known_cost_usd": 0.056151636, + "final_metrics": { + "total_steps": 107, + "total_prompt_tokens": 1713955, + "total_completion_tokens": 29967, + "total_cached_tokens": 1680256, + "total_cost_usd": 0.056151636, + "extra": { + "cost_is_estimated": true, + "estimated_cost_usd": 0.056151636 + } + } + }, + "task_checksum": "3426fb4714b5bb0463cc30bd872712a2400b23934b135c40ef7f88b27da4cd7a", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/result.json": "1b9b1ecca667330636ba0ca8ca5b6255fae7ea05627eee23326e461cfef27786", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/agent/trajectory.json": "2a0e42851a8c3b0c8ee45f0dc3db1882eae3ba72cfd1fa883b181200d20588b3", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/agent/bootstrap.json": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/verifier/retry-summary.json": "a617c5e8f570fde26f320b4ec71fe384e4a7eeb4e2831e160ddf13bdc51e90b2", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/verifier/test-stdout.txt": "a073573c5c4f7813cea76ae6e4938a09def1ba1d871c86b6eba3261324fd3a37", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/verifier/ctrf.json": "5fd556cc5936b1d417d20170cb2b5827ee41f14aa0bcffd558c8860099bb6a26", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011-input/manifest.json": "7e912ca1a4aafc9d9f3cc31da8a0e72cdff0bc037ad8873b6945fd66881a0a6d", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011-input/config.json": "2b2b3247f92d5a56bebef93e7d8007be8ac26807533cdb8eed3f555af7c3bb52", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "jobs/dev12-baseline-20261011-v3-orchestration/dev12-build-pov-ray-baseline-20261011.json": "a53545ba276d9b6fac51aec352d9a52d24e3169db5ef138361e13ef0f52bd4dd", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/verifier/attempts/1/ctrf.json": "5fd556cc5936b1d417d20170cb2b5827ee41f14aa0bcffd558c8860099bb6a26", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev12-baseline-main-source/jobs/dev12-build-pov-ray-baseline-20261011/build-pov-ray__gSzbLYX/verifier/attempts/1/test-stdout.txt": "a073573c5c4f7813cea76ae6e4938a09def1ba1d871c86b6eba3261324fd3a37" + } + } + ], + "post_run_pytorch_diagnostic": { + "kind": "post_run_diagnostic_not_benchmark_retry", + "fixture_url": "https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz", + "fixture_sha256": "8d422c7b0a1c1c79245a5bcf07fe86e33eeafee792b84584aec276f5a2dbc4e6", + "model_sha256": "a32103eef09936e586b1a7dc7e64d279500a38ff30fdd6df2167e28eb954eee6", + "method": "stdlib float64 forward pass; original packed float32 weights; original model.py ReLU architecture; compare first 50 images to official fixed expectations", + "expected_predictions": [ + 2, + 2, + 1, + 0, + 4, + 1, + 4, + 9, + 6, + 9, + 0, + 8, + 9, + 0, + 1, + 8, + 9, + 7, + 8, + 4, + 9, + 8, + 7, + 5, + 4, + 6, + 9, + 4, + 3, + 1, + 3, + 1, + 8, + 6, + 2, + 2, + 7, + 1, + 2, + 1, + 1, + 7, + 4, + 2, + 8, + 8, + 1, + 2, + 4, + 4 + ], + "variants": [ + { + "name": "raw01", + "mean": 0, + "std": 1, + "mismatch_count": 0, + "mismatch_indices": [], + "predictions": [ + 2, + 2, + 1, + 0, + 4, + 1, + 4, + 9, + 6, + 9, + 0, + 8, + 9, + 0, + 1, + 8, + 9, + 7, + 8, + 4, + 9, + 8, + 7, + 5, + 4, + 6, + 9, + 4, + 3, + 1, + 3, + 1, + 8, + 6, + 2, + 2, + 7, + 1, + 2, + 1, + 1, + 7, + 4, + 2, + 8, + 8, + 1, + 2, + 4, + 4 + ] + }, + { + "name": "mnist_standard", + "mean": 0.1307, + "std": 0.3081, + "mismatch_count": 10, + "mismatch_indices": [ + 0, + 18, + 21, + 22, + 25, + 26, + 28, + 32, + 34, + 44 + ], + "predictions": [ + 7, + 2, + 1, + 0, + 4, + 1, + 4, + 9, + 6, + 9, + 0, + 8, + 9, + 0, + 1, + 8, + 9, + 7, + 1, + 4, + 9, + 6, + 4, + 5, + 4, + 0, + 7, + 4, + 0, + 1, + 3, + 1, + 3, + 6, + 7, + 2, + 7, + 1, + 2, + 1, + 1, + 7, + 4, + 2, + 3, + 8, + 1, + 2, + 4, + 4 + ] + }, + { + "name": "agent_half", + "mean": 0.5, + "std": 0.5, + "mismatch_count": 14, + "mismatch_indices": [ + 0, + 11, + 15, + 18, + 21, + 22, + 25, + 26, + 28, + 32, + 33, + 34, + 44, + 45 + ], + "predictions": [ + 7, + 2, + 1, + 0, + 4, + 1, + 4, + 9, + 6, + 9, + 0, + 6, + 9, + 0, + 1, + 5, + 9, + 7, + 3, + 4, + 9, + 6, + 6, + 5, + 4, + 0, + 7, + 4, + 0, + 1, + 3, + 1, + 3, + 0, + 7, + 2, + 7, + 1, + 2, + 1, + 1, + 7, + 4, + 2, + 3, + 5, + 1, + 2, + 4, + 4 + ] + } + ] + }, + "final_image_integrity": { + "verified_at": "2026-10-11T17:22:55.161171+00:00", + "original_images_unchanged": 12, + "images": [ + { + "task": "terminal-bench/build-pov-ray", + "image": "alexgshaw/build-pov-ray:20251031", + "image_id": "sha256:874baf49341e2853a15a9184e308c7eb632e97f434359fcff2891c6e5ca1b39b", + "repo_digests": [ + "alexgshaw/build-pov-ray@sha256:874baf49341e2853a15a9184e308c7eb632e97f434359fcff2891c6e5ca1b39b" + ], + "size_bytes": 163066497 + }, + { + "task": "terminal-bench/cobol-modernization", + "image": "alexgshaw/cobol-modernization:20251031", + "image_id": "sha256:593ab9df3d83f771e927888b2d9436b380cdf7ec4d0af2a9f0802d541edebba0", + "repo_digests": [ + "alexgshaw/cobol-modernization@sha256:593ab9df3d83f771e927888b2d9436b380cdf7ec4d0af2a9f0802d541edebba0" + ], + "size_bytes": 162020401 + }, + { + "task": "terminal-bench/custom-memory-heap-crash", + "image": "alexgshaw/custom-memory-heap-crash:20251031", + "image_id": "sha256:1895d7d6240d5fb2ec774d6bfdd1057d170f8d199c15c582e4585d3c63802557", + "repo_digests": [ + "alexgshaw/custom-memory-heap-crash@sha256:1895d7d6240d5fb2ec774d6bfdd1057d170f8d199c15c582e4585d3c63802557" + ], + "size_bytes": 1267999268 + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "image": "alexgshaw/fix-code-vulnerability:20251031", + "image_id": "sha256:cac325252991f823713b2d0441502972901dd782bd67f66c03d9b1e410dac5c0", + "repo_digests": [ + "alexgshaw/fix-code-vulnerability@sha256:cac325252991f823713b2d0441502972901dd782bd67f66c03d9b1e410dac5c0" + ], + "size_bytes": 190696453 + }, + { + "task": "terminal-bench/nginx-request-logging", + "image": "alexgshaw/nginx-request-logging:20251031", + "image_id": "sha256:693e64431a0dbf45952e8b91a298d54fd60247e501c6623fb8fa3ebc07fc3d3d", + "repo_digests": [ + "alexgshaw/nginx-request-logging@sha256:693e64431a0dbf45952e8b91a298d54fd60247e501c6623fb8fa3ebc07fc3d3d" + ], + "size_bytes": 67285778 + }, + { + "task": "terminal-bench/git-multibranch", + "image": "alexgshaw/git-multibranch:20251031", + "image_id": "sha256:5bc449921c138295f6ddd8dca2977319fee2ce523811dbe56cec4ca663385fb5", + "repo_digests": [ + "alexgshaw/git-multibranch@sha256:5bc449921c138295f6ddd8dca2977319fee2ce523811dbe56cec4ca663385fb5" + ], + "size_bytes": 222134454 + }, + { + "task": "terminal-bench/db-wal-recovery", + "image": "alexgshaw/db-wal-recovery:20251031", + "image_id": "sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e", + "repo_digests": [ + "alexgshaw/db-wal-recovery@sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e" + ], + "size_bytes": 175276145 + }, + { + "task": "terminal-bench/multi-source-data-merger", + "image": "alexgshaw/multi-source-data-merger:20251031", + "image_id": "sha256:8b32782078ff7383a1b4e5d3cecca8ce287e30f50bb4c0e5db009a18064e666e", + "repo_digests": [ + "alexgshaw/multi-source-data-merger@sha256:8b32782078ff7383a1b4e5d3cecca8ce287e30f50bb4c0e5db009a18064e666e" + ], + "size_bytes": 220880128 + }, + { + "task": "terminal-bench/query-optimize", + "image": "alexgshaw/query-optimize:20251031", + "image_id": "sha256:28de7da87004eab5b29bd5eaf1823579bc5b73ccea4b80349a5623185770da14", + "repo_digests": [ + "alexgshaw/query-optimize@sha256:28de7da87004eab5b29bd5eaf1823579bc5b73ccea4b80349a5623185770da14" + ], + "size_bytes": 120494963 + }, + { + "task": "terminal-bench/adaptive-rejection-sampler", + "image": "alexgshaw/adaptive-rejection-sampler:20251031", + "image_id": "sha256:985bed39e8448bc95f138b5f766b59003a95c4be171c96af8e76448338f3dd8f", + "repo_digests": [ + "alexgshaw/adaptive-rejection-sampler@sha256:985bed39e8448bc95f138b5f766b59003a95c4be171c96af8e76448338f3dd8f" + ], + "size_bytes": 29731756 + }, + { + "task": "terminal-bench/constraints-scheduling", + "image": "alexgshaw/constraints-scheduling:20251031", + "image_id": "sha256:567ce5a189f8d11ac461790876e934cc7af38391baf89a78f95f4dafc1fec3b0", + "repo_digests": [ + "alexgshaw/constraints-scheduling@sha256:567ce5a189f8d11ac461790876e934cc7af38391baf89a78f95f4dafc1fec3b0" + ], + "size_bytes": 49852952 + }, + { + "task": "terminal-bench/pytorch-model-cli", + "image": "alexgshaw/pytorch-model-cli:20251031", + "image_id": "sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "repo_digests": [ + "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67" + ], + "size_bytes": 44349611 + } + ] + }, + "publication_policy": "Only derived results and hashes are published. Raw logs remain local; the database trace contains a provider credential and must not be shared unredacted." +} From 70da6f722caffb4581343f9fbfe807c78b855569 Mon Sep 17 00:00:00 2001 From: minixalpha Date: Mon, 12 Oct 2026 06:55:12 -0700 Subject: [PATCH 06/15] test(benchmarks): register a common-main 34-task baseline --- .../dev34-main-baseline-20261012/README.md | 79 + .../adaptive-rejection-sampler.json | 22 + .../build-cython-ext.json | 22 + .../build-pov-ray.json | 22 + .../caffe-cifar-10.json | 22 + .../cancel-async-tasks.json | 22 + .../circuit-fibsqrt.json | 22 + .../cobol-modernization.json | 22 + .../constraints-scheduling.json | 22 + .../custom-memory-heap-crash.json | 22 + .../db-wal-recovery.json | 22 + .../dna-assembly.json | 22 + .../experiment.json | 1356 +++++++++++++++++ .../fix-code-vulnerability.json | 22 + .../git-multibranch.json | 22 + .../image-digests.json | 311 ++++ .../install-only.json | 155 ++ .../kv-store-grpc.json | 22 + .../llm-inference-batching-scheduler.json | 22 + .../log-summary-date-ranges.json | 22 + .../main-prompt.txt | 9 + .../merge-diff-arc-agi-task.json | 22 + .../model-extraction-relu-logits.json | 22 + .../mteb-leaderboard.json | 22 + .../multi-source-data-merger.json | 22 + .../nginx-request-logging.json | 22 + .../openssl-selfsigned-cert.json | 22 + .../path-tracing.json | 22 + .../pypi-server.json | 22 + .../pytorch-model-cli.json | 22 + .../pytorch-model-recovery.json | 22 + .../qemu-alpine-ssh.json | 22 + .../query-optimize.json | 22 + .../regex-chess.json | 22 + .../regex-log.json | 22 + .../schemelike-metacircular-eval.json | 22 + .../selection.json | 150 ++ .../torch-pipeline-parallelism.json | 22 + .../torch-tensor-parallelism.json | 22 + .../write-compressor.json | 22 + 40 files changed, 2808 insertions(+) create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/adaptive-rejection-sampler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/build-cython-ext.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/build-pov-ray.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/caffe-cifar-10.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/cancel-async-tasks.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/circuit-fibsqrt.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/cobol-modernization.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/constraints-scheduling.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/custom-memory-heap-crash.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/db-wal-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/dna-assembly.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/fix-code-vulnerability.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/git-multibranch.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/image-digests.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/kv-store-grpc.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/llm-inference-batching-scheduler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/log-summary-date-ranges.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/main-prompt.txt create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/merge-diff-arc-agi-task.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/model-extraction-relu-logits.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/mteb-leaderboard.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/multi-source-data-merger.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/nginx-request-logging.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/openssl-selfsigned-cert.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/path-tracing.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/pypi-server.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-cli.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/qemu-alpine-ssh.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/query-optimize.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-chess.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-log.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/schemelike-metacircular-eval.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/selection.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-pipeline-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-tensor-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/write-compressor.json diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md new file mode 100644 index 0000000..9d17e36 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md @@ -0,0 +1,79 @@ +# Common-main baseline for the 34-task development pool + +This run evaluates **all 34 existing development tasks once**, with fresh model +outputs from main `2eb99b608dfb39464d17d109a4a8f11afda607a9`. Both local main +and origin/main pointed to that commit when the run was registered. Earlier +pilot20, repeat experiments, and the twelve-task baseline remain separate; +none of their model outputs or rewards count toward this run. + +[Selection](selection.json) is the union of the previously exercised 22 tasks +and the twelve additions. It uses the same pinned 89-task dataset revision. +The remaining 55 packages stay unopened. These are development tasks with prior +outcome and partial test/trace exposure, not a held-out evaluation. + +## Frozen inputs + +- [Plan](experiment.json): runtime, task revisions, complete task-file hashes, + prompt/configuration hashes, work budgets, preflight, and queue order. +- [Prompt](main-prompt.txt): current main's 385-word headless prompt. +- [Original images](image-digests.json): all 34 images were already cached; + identities and repository digests are recorded before execution. +- Runtime worktree: `jobs/dev34-baseline-main-source`, detached at the exact + main commit. The standard runner independently snapshots agent/adapter source + and records installed-wheel hashes for every job. + +Model: `deepseek/deepseek-flash`, official DeepSeek Anthropic-compatible endpoint, +100 replies, 65,536 output tokens per request. Temperature and reasoning effort +are unset. No prompt or agent optimization is included. + +Work budget is each task's native agent timeout minus 180 seconds: 23 tasks at +720 seconds, one at 1,020, four at 1,620, one at 2,220, four at 3,420, and +POV-Ray at 11,820. Setup independently allows 1,800 seconds. Original task images, +resources, package sources, and official test scripts remain unchanged. + +The model queue sorts by ascending work budget, then task name. Consecutive pairs +finish before the next pair starts; maximum concurrency is two. Every task gets +exactly one fresh model trial. There are no outcome-based replacements or retries +of entire model trials. Verifiers may retry **only timeouts**, at most three +total attempts, with the same agent output and original per-attempt limits. + +## Run + +From the repository root, with credentials in the process environment, set +`ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic` and leave +`ANTHROPIC_MODEL` unset. Do not put credentials in configuration or published +artifacts. Raw traces can contain environment values and require review/redaction +before sharing. + +```bash +python benchmarks/harbor/scripts/run_comparison.py \ + --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment.json \ + --phase preflight + +python benchmarks/harbor/scripts/run_comparison.py \ + --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment.json \ + --phase model +``` + +The orchestrator calls only the standard +`uv run --project benchmarks/harbor python -m harbor_adapter run --config ...` +entry point from the frozen repository root. Fresh install-only preflight runs +for all 34 tasks before model work. Only QEMU has a reviewed verifier dependency +profile, `qemu-alpine-ssh-v1`; the other 33 remain explicitly `not_configured`. +Agent installation success does not certify unsupported verifier dependencies. + +If installation infrastructure fails, preserve the failed job and register any +amendment before model work. Reuse a successful native install-only record only +after the existing orchestrator verifies complete task/source/wheel/bootstrap/ +configuration provenance. Never synthesize a successful job or reinterpret a +setup failure as an agent correctness failure. + +Keep all scored failures, setup errors, verifier errors, and incomplete usage +records. A native zero without executed assertions is a verifier error in the +analysis, while its raw reward remains unchanged. Inspect operational stage and +completion metadata while running; defer new trajectory and hidden-test content +review until all new model outputs freeze. Audit external benchmark-answer +exposure separately from raw scores. + +For reproduction, use new job names and a new registration directory. Existing +jobs and their immutable registration must not be overwritten. diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/adaptive-rejection-sampler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/adaptive-rejection-sampler.json new file mode 100644 index 0000000..381d3f9 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/adaptive-rejection-sampler.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/build-cython-ext.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/build-cython-ext.json new file mode 100644 index 0000000..222fc73 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/build-cython-ext.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/build-pov-ray.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/build-pov-ray.json new file mode 100644 index 0000000..c228cfd --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/build-pov-ray.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 11820 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/caffe-cifar-10.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/caffe-cifar-10.json new file mode 100644 index 0000000..1d3f832 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/caffe-cifar-10.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/cancel-async-tasks.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/cancel-async-tasks.json new file mode 100644 index 0000000..7a885de --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/cancel-async-tasks.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/circuit-fibsqrt.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/circuit-fibsqrt.json new file mode 100644 index 0000000..44280fc --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/circuit-fibsqrt.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/cobol-modernization.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/cobol-modernization.json new file mode 100644 index 0000000..459c25e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/cobol-modernization.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/constraints-scheduling.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/constraints-scheduling.json new file mode 100644 index 0000000..4958be1 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/constraints-scheduling.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1020 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/custom-memory-heap-crash.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/custom-memory-heap-crash.json new file mode 100644 index 0000000..3974e78 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/custom-memory-heap-crash.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/db-wal-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/db-wal-recovery.json new file mode 100644 index 0000000..3781ed1 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/db-wal-recovery.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/dna-assembly.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/dna-assembly.json new file mode 100644 index 0000000..3eed136 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/dna-assembly.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment.json new file mode 100644 index 0000000..25508b6 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment.json @@ -0,0 +1,1356 @@ +{ + "schema_version": 1, + "registered_at": "2026-10-12T13:52:35.779627+00:00", + "experiment": "dev34-main-baseline-20261012", + "selection_file": "selection.json", + "selection_sha256": "4572b1cc432a16b9c5d1dd554c8d2189ae80414f386a90bf03829bc4d0c3da85", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "arms": { + "main": { + "source_root": "jobs/dev34-baseline-main-source", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "prompt_file": "main-prompt.txt", + "prompt_sha256": "37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d", + "prompt_words": 385 + } + }, + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/adaptive-rejection-sampler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "95408217b9c484ffc18a02a91b7706323b6902919c72b24866a32e27a23543e2", + "environment/Dockerfile": "e644a3343e339303230f1441b7383f076d21c69a5a7aa6430b4f8a9d27f15c8b", + "environment/protected.tar.gz.enc": "7fc9bf1347bb97da8dede04b2c58638f225070913e222d76627068aebd1b1df2", + "instruction.md": "e6d28c28acd76bef61817427c74c2a29e3e8d75f535cbd708afcbfd02c54013f", + "solution/solve.sh": "fe443a8082b5183ba1623311ba0ee958888aefa633eff93766b9ae14e9bdd84d", + "task.toml": "6ed7b37f74beb46e9d47b0f5fad53025667c9548baff4e96d467b56d429fab1a", + "tests/test.sh": "38b43560d173cc2b952c3a3e17b8a480216d84e33450515e047bcb0d806b1e0a", + "tests/test_outputs.py": "547dc6e107f034f41703722aeceb6d0236e3fb69116fc3f2fbaba11884de352f" + } + }, + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/build-cython-ext:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d5f5113941c3963292bae396baa9369571026f46e59437f98ddcc93475dd7729", + "environment/Dockerfile": "cb6cf421cb9b69d43bd25cd491e49bda982fbb865d47515178d017eda727194e", + "instruction.md": "7e52874e45f88505bd21540229719de6639a401477557d814489981d472f0e34", + "solution/solve.sh": "41bbdde88617070366a70e518eedca466a4ea1acd7e96899b2fb7da988e40abc", + "task.toml": "c472f0e7522c128fb7f03e00b50c8cdf2e5e6474daa1e094e7441f4b54e770b2", + "tests/test.sh": "0156c9c4085258b696ce424cf1ce96cba44d1263c5d73bdf54083fd80237f915", + "tests/test_outputs.py": "6549f1a8e04b6163ec08877e6f6b9868bf94563f3a3c72ac1fba9c512c31689e" + } + }, + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cancel-async-tasks:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9adf86b1279cc44f83fe16e6d340595aa992240f1d87243d204cae32e453693b", + "environment/Dockerfile": "1ee7bdb4a7536f5f114c11a3ac6ff42790abfea901fcba6b2a8ea8685f19140e", + "instruction.md": "23d22f7039b50682941652c716f96dac314b8585c32c133614cac77dc067f47e", + "solution/solve.sh": "4938ca4ad5b7dcd36b414b42a1abb10c058ecefce7462b678f9bd05c9fae1844", + "task.toml": "14db0675e04408f2133b0067e8dbddf66a1d6e7baf6b8d38bef8ecccfc4e58a8", + "tests/test.py": "f533c0a64db068f582b4733c5b81f2e66b8fa52fcd3efa65752e6bd6d24780aa", + "tests/test.sh": "79600a6b863def31b4daa008297423010cd2d6243e689d67503e118e4dc53397", + "tests/test_outputs.py": "d6f510b51fafc81b4e9b5cc95535aa13491c8753e0e4171db42c45f4de460ee7" + } + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cobol-modernization:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "c89318cf59efcc7040e3d946022e2616161b6733931a080cdeea7b1918e7ecc4", + "environment/Dockerfile": "8b175c83fc03ec7d16d6101516bc5100230b592f480e435279f4fe051c9280b2", + "environment/data/ACCOUNTS.DAT": "46f63eb29442ebc6716bcd8a0c36f4367910f2783f6b84a307ed9a61ccd1046e", + "environment/data/BOOKS.DAT": "db04da646207cd87f8e9c5b6f0b57d95953899f3eef22de1a81423f06ef4a9fd", + "environment/data/TRANSACTIONS.DAT": "1fa4cc5420ded94d585c563e8196b85166ebbae2bb8924b20014717fb2fb9ee3", + "environment/src/INPUT.DAT": "52fc984f0cf14af1495285d7efe02b9bae13159eea59255560ee938c4f78afee", + "environment/src/program.cbl": "27cb0e56f65f52fa75f346720b37a180b8e034ebe7d7622b979f195087bd6ebb", + "instruction.md": "a87074f1141b7345394cd88053d0debb0dc8f7de95308882c6538ee9a3b359da", + "solution/solve.sh": "7d4a0504e1686c1790bfbc074d4e598f4c631f54c2bc5f3d5fd786574b4810b9", + "task.toml": "4b620211dc3d6b1b989f0210c5ecd2f606974581c8bce27bd78659d48cf0cfbd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "009180a7092583fc8143bb3ffc3957923aa0d61d0aac3a471f76eeb4aae54a88" + } + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "file-operations", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + } + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/fix-code-vulnerability:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "87f28dbad183c3a634887696d328ca68a8a0858f132bc48cb7341c22267d38f7", + "environment/Dockerfile": "0917470986224df4daf4854a71302ae81c73ba5dd6e25be1232d29b758a5b3ab", + "instruction.md": "89b1abdf0af19399f720233b5cdf6b2e8e0ff7b6e3fe3e2a3af0d4d298a8641a", + "solution/solve.sh": "7a0131b369d83ae8546ecc3f9b4e23d38240f1bd1f0941e4430fa85b5182ee03", + "task.toml": "9aa2bd14ce9408289d199d0608bc3fa65b8b5e23374c9641020de62197dbf1e2", + "tests/test.sh": "f1611da693cb4cfc5e8482698c6a1858a5c67220ae9db89b50aed66584e9b17f", + "tests/test_outputs.py": "c006a5de95b69bd78055aa26cc055734dbd50718ede25d331f767bfee4ef7f26" + } + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/git-multibranch:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3b9473429381d9eacabab498cb4a8915aa2bca24fa857ef862e5991d3f81f113", + "environment/Dockerfile": "88d9a9f4fef1ae7fddf632799dedf5a2fb6489460a93b610c33dd8b122982821", + "environment/default.conf": "a3a4c5d0ac9cfe79b0510e4225c80fa3988c27fbac13622505886bf66449e04a", + "instruction.md": "54014a38d7526d65faeb6c11d9b7ed3d4a5cb5d973d68d3da3b68b8c0289e125", + "solution/solve.sh": "300ca96b79db8d77f14742109aaf76048aafc4644ced750b1bc5c2761e8d0175", + "task.toml": "4b9981604578a31205257cca36a28d0c89c9d8d1058541330acd11d3f8d65f82", + "tests/test.sh": "ddbde3eea6ffa8fda105cbd86597a53bd409f41e13223900aa27e77a84b6b155", + "tests/test_outputs.py": "f03c6867663b8e8fb7899df51ee4df26411fab62ed491753964b801e591d16c5" + } + }, + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/kv-store-grpc:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "51fdc2ad06f0adb1270164f247ea41799b2ee2ad15490e48ddbf5e79d3767aaa", + "environment/Dockerfile": "47c9982789092437f950736110b2bd18899c39b8fd20f3af3c95b848e564a197", + "instruction.md": "f4b3facc199d5b1c78db6e603ffa29ec1b93a77ff7d291e1a92b42e6bf582bed", + "solution/solve.sh": "6bee2938ac9731d1156b0ca02b28c3d5793bf2cbe5f995c863d397093af01cb7", + "task.toml": "6ff0f30a526bfff86a67b0b53439b9965229c0f368167dc1a5d05f34d82299a3", + "tests/test.sh": "c348467a076bf614d0a850e25b2263f85bbf7b4c7a8d344fbd5454a0611a0df8", + "tests/test_outputs.py": "115de2e964bbe689d07930527113422ef8a9e4be4eb958ad84594e3fcd2b1b0e" + } + }, + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/log-summary-date-ranges:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "026000415cb577f3d3d5132902670262f7d5d8f69841903ec06a882292f756b4", + "environment/Dockerfile": "b479f6e987b63a6ec118fa1691b8bb85c437b2a6a0fe63feab5ed7154957fdfc", + "environment/log_generator_deterministic.py": "beb6eff59979e9aa0be63ce8a0f1b0df1178f92871f04ba9f76fa70cf8beb81c", + "instruction.md": "c1d4516ad4ec2238209740eaf01291df9ab48b0028a3fc5df383c6f831453484", + "solution/solve.sh": "9f6a8ee8947bdf1c0f31b530b6194ae998d66841a3b785fa2e246ee840350ab6", + "task.toml": "9a9593673d6805ef3a301ea734e9c4dbaf80da1b37c0fe9d1f2ceea2a8c57e18", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2fc06a6ce1944ad82f84fb437395fa75ed8c0eadd385b92e375c27e5bc9cb2b4" + } + }, + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/merge-diff-arc-agi-task:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d6b48ed0890b12eb68f72c12cd02689bd6bfae2191dd2688949cde6fe77f6a94", + "environment/Dockerfile": "4a03ae6f0c60b9c2f45c33f0413b5aa75c9811870254e3ff1968fd0119173fc6", + "environment/task-deps/bundle1.bundle": "a759f9b6ffc1a2eee50778d72a1899ad7d724becab84fa30e7aac71ee9e443a5", + "environment/task-deps/bundle2.bundle": "75e3e3916a01c980a03715de67b16c729e03661d94a694b5f4223d229bd6593a", + "environment/task-deps/examples.json": "5426569e4ea078bbb74039ec712a2d4d35f1f9fa3a5f26bf2a0de4a2c8ef5db8", + "instruction.md": "2d826479873e5ac1c94c1f802763bf85c22bb04624384685ebdf6b153fe0278e", + "solution/solve.sh": "48d4becc41fda5b6063fe96947474ffe778a29d6924d7b2bb235eb61e5ce555d", + "task.toml": "6ced40e06b43b169857509765eb7f07791db558f9437cc7367d2536771c2ca5b", + "tests/test.json": "20f116a6b718bbf4874bbe203d15a5c0f85a1d88cb622e2d7529c7e4fb2d4c28", + "tests/test.sh": "c9052954cd6ce25c4a375a2fb48e765b1efa5b39f7991b93993cc793af8ab9d2", + "tests/test_outputs.py": "2754ba6c5c5f75c9db5ed2c5729d9f5d7e21cb7fdb18aa0c65abdc3e7e85c149" + } + }, + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/model-extraction-relu-logits:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "mathematics", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9465accba6c57e5de9a196eed776a16eeb0852a8afc18fc3f8979e2de71b0c7b", + "environment/Dockerfile": "598b1f80d3b7194e97cbf02c8d245de30e303909e0cbe6465125d1d3179fddb5", + "environment/forward.py": "af1a23efd8e4390d16a29777c28116b2f8623ae3f7b628fe14db9f94dac26f5b", + "instruction.md": "c9e179dd099065a34c362a865b0eb9fab5f7d96be582cf117e7519580c036ebe", + "solution/solve.sh": "4cea5094f074ff21453d4423aa1b415a104ed90caa55a4acbb4f09941a3994ea", + "task.toml": "eb64dfaf8ae88868911ae28855cdc4f767de464001e2704919015b595be6dc29", + "tests/test.sh": "3b2d8f5e0353c2bab3515d6387468a863a4f9f241ee131c73f545f3fdf9e2d52", + "tests/test_outputs.py": "e5189c7eb9942ede55e1b48a80388590b80a486b75497613f257b3373ad9b2c9" + } + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/multi-source-data-merger:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9bf37162a4c1b88ed93627830bb8512ebae154325d7cb069c883144a0e4c69d6", + "environment/Dockerfile": "c95a40d3d35b473dc15ae1e8784ba7fb1725805c0583fc574dd2764325d75e4f", + "environment/data/source_a/users.json": "75e51e2d3ab0bdba2269d7cf3757576f84f3ee2c13407d2db94289a5958a1778", + "environment/data/source_b/users.csv": "8e87d16dc47d166edd8e56165955bad265b630d53d02762367c816affb5fa760", + "environment/data/source_c/users.parquet": "23d1ea51c4170bdba53febb885a0f495e28de525a52667347b2932136b9a72fe", + "instruction.md": "d1d28bb6e37f7a255a1b7f78aae8591f35c0494f33e4913392424ad954aab944", + "solution/solve.sh": "1b3d0d50e31930d042495d48f4c8e95f1a9c1a979c803b7f7cec651322537e1e", + "task.toml": "ee3964c0ee040f0a0cb121cefefbc15443d127684acb003fa5b61dd42f31c694", + "tests/test.sh": "d5a00e64413c0e1ae452f38a1aa1a47c3ed7b4416bbfccae1a4c053025a336d1", + "tests/test_outputs.py": "a070888930cf829aeca039771260b1386bda990ea805f313468d381ba3608ca5" + } + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/nginx-request-logging:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d79997586d1c14ed8dbfea050e3580b25500e04c8345acfd3f294152433c795a", + "environment/Dockerfile": "43003466d89ab83484a3fa618e35b58cdb74a002b1e8716ef30f12b3ac8302d8", + "instruction.md": "26a7ac98aced6107f147206790ada77bbf8ed3c25fb360cf323d5fc6889edf99", + "solution/solve.sh": "2454bbbf7c201a9a778165f286373ba2cb991a0eb1d026a8005d57ae71250dd9", + "task.toml": "b7a78d00be13656f75069197100b7b8c4fd268decf6f78929c853b3222a2729d", + "tests/test.sh": "27761195f53cbfe5941fe39b573631f6c926869e64b7c43220377e1e116528e7", + "tests/test_outputs.py": "045cc716c14efde3b0dcff5fc7c85ec5d18bfc6ce66f8b40a418fa2a3a4acda0" + } + }, + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/openssl-selfsigned-cert:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "5dfa88651890cda06fa2a25babd3b813818767635e88bbabecc8305d5bb3942f", + "environment/Dockerfile": "f590b523038be43afcaf5291d7b9e8ed1333c48d9994e77b8dee2f2c352f0f40", + "instruction.md": "38054a3371f7c2c853478a2f9af0bff858437d5b28a9a07483238f06e566921f", + "solution/solve.sh": "58cf3c844e907745cd81fb4bd75efb3d080e2f0e82b589c56c954ae6e1756510", + "task.toml": "75c8913f0fa15890786c11999cde920b8bfc10d4ce8a14edf1e76985a2ac5800", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "90970bb3c372e8e7fb6bb9ac5f649ba73eb38e1a5ea2e922a9057a72b6b4192d" + } + }, + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pypi-server:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "906a096a2cc7b9afc738ff5ad5fbd605be65c103f5069c6673915223b9b187e3", + "environment/Dockerfile": "17a4e67580e4f2c6a151d04b9d74eb020666e7eec8e5259c4a5411f9e93858d0", + "instruction.md": "1c537e4a311a8e71155e121f57bfc03f742fbfb99d05314cd03e433996a28a31", + "solution/solve.sh": "1143c0eade4932ec3aeaa5013fb5e23bbcccca15dead294aa03b6af88750775a", + "task.toml": "07c39d7a733c89adadf74bb63a3e906c3694feb6c490d5fc85ac187c003fd447", + "tests/test.sh": "d2b1220e42bcfdcce87907ac5bbd69ac210a3c94926545ca2439636786e7784e", + "tests/test_outputs.py": "82c6351722e9aa984ff1d701bf57df3150ce51466df2b70afa1aa62d2650be4c" + } + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-cli:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "35a093fa5f1592737784ba718c2e3e9ae17ff8b5e53e003fae560ddce091039c", + "environment/Dockerfile": "8fd5c0f21a6914cd440d10c84c9db06fec606f20dc3e0363b17a3b8c220dc4d9", + "environment/task-deps/cJSON.c": "0e7995261640c96db16ed2c91a5d981efa917cf5e72d0f97ec45a1179ae3ad73", + "environment/task-deps/cJSON.h": "0909160541bcc510e0b2d8c7a9cc051fa1459a0f13be4d8e56346a15491f2f2f", + "environment/task-deps/image.png": "f4d5dfefe463939996541404340e7099d5957b56e1df3760d13b5d2de754db53", + "environment/task-deps/lodepng.cpp": "67873c5f6272f779a143a9c81070e8783327c5552d0b9de2aaee2cb779308917", + "environment/task-deps/lodepng.h": "b735b7e0357319973e9488d520d7f270d734a6e9e1bf924fb1d5b8d26d7e97ce", + "environment/task-deps/model.pth": "a32103eef09936e586b1a7dc7e64d279500a38ff30fdd6df2167e28eb954eee6", + "environment/task-deps/model.py": "e4d3a5c728449656d0efde7871ee227619348352769795ed1be1a8b7f835fddc", + "instruction.md": "ce36ee1f6c589f81ee6a6be4967e25c76a2ed697006928023f42dfb9f28a6750", + "solution/solve.sh": "3b34e561c9ef0a53c9948655847846eb4811aaac67b990480aa6aac1b04bfe66", + "task.toml": "fc1c018efbd5d0537ac0741d53d1ebad3886faa51d14cbde8ec953856411bc0d", + "tests/test.sh": "43fb67e537c7adc2df11c2367f47c876cb5d16a7c1ae16d0339c3a3364bb2b06", + "tests/test_outputs.py": "370702c57a4e0c85f70ee37a5c357ae3a5df8480a275bb6263b0376f2f6c093b" + } + }, + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-recovery:20260430", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e080308a05703b40edd16f53aea41a0e6b9874750ecee35bb432029d0d44bf53", + "environment/Dockerfile": "8a6b7edbb9e7c241e447f15a71e13e5d043b25169be2fd9802b84f5be3d56eb6", + "environment/dataset.pt": "6db65e4ec1c0760209c2dd997ef582e4a1519ab38e0312339999c460a696960d", + "environment/weights.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080", + "instruction.md": "2bef07dc28be3ab2c35ae389584034c2e4dde92c2c31b3eb6bd829628c8c3f4a", + "solution/solve.sh": "175a7802777b43e0fa4f629d71f7c68fb40706813c342aaddc72108d89bd5b2c", + "task.toml": "851a51204f53482a31f47cdf8833128cf421113b0331011286940fc857c37dfd", + "tests/test.sh": "6005f3a75ccfe00d7bed1005264b4a84a1f8e30cd780f8a97b3a804be7cc7a8e", + "tests/test_outputs.py": "5d4d57c43e6913bf3d0081da764ad36d1b7275e3bb114a05abda529afe563905", + "tests/weights_gtruth.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080" + } + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/qemu-alpine-ssh:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "qemu-alpine-ssh-v1", + "task_file_sha256": { + "README.md": "72f1c83fa1e3e6381ef85dd82fd109ee18cdeac090311cf3212a0a8cff0a32a5", + "environment/Dockerfile": "015ce0601ff54758f66b33f4952e0d3a577d84c09aed8f3ec5945a570e8d168a", + "instruction.md": "8348c60490cb90bc690cba259dfe98996da4c3294c6d42bba1119b00c64de4de", + "solution/solve.sh": "1cd31186c427b91bb00499d22aa4f0214fc1ca93b40b6e4cfa969611b848db2a", + "task.toml": "10a43ec6ee732eeba852c377f12e23178f7fa5a4868143f3699a15565ffb64c7", + "tests/test.sh": "3b244479776c4ee3639631112799f2cde22cbb80ac10511715e3a8f32909c289", + "tests/test_outputs.py": "83ccb4bdcd259b342212bebefe395f289858b4dc55b2ee7ae0dab238ad2e9718" + } + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/query-optimize:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a82c53e333a0ac0b0b410b5e26cb8e7305dece31db84d340248ffb13d5a45160", + "environment/Dockerfile": "064818771f200e0517d22068f87229a7a95d0a82b0778c84ca7203bf1926a0b8", + "environment/task-deps/my-sql-query.sql": "43382792f0f438625d643e34abe144febd5669518f4ab1ab550a15c6d9314a21", + "instruction.md": "473e12d320d2732f679fb631c2a4dab4a01deea6775686718ae0c417b8a39447", + "solution/solve.sh": "0d95f3cedb287ff9d47c796cbaf812c300d1222c10fa80c49e8acb0e81eb6978", + "task.toml": "6e88f2a86df699594bcb1bf81df4ed694505f79c9420f671335b02524bbe3868", + "tests/golden.sql": "4da1885d56b3ffa7b7b22b422996d018795611f6963499d551582881b74b84fd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2068ae80c57af8d52a7ae510f0666732c057806e556b480970f36f14bae683f6" + } + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/regex-log:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "29ffd98c55dbb6f4844f1b9d8c1769b2155c117ee475e0df6bbc5ef330d239b3", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "4f7ac05e70cf9220ea0f1e5a052c5f908cd0fa884e847d80b0bd51bae2e96f9c", + "solution/solve.sh": "7e670d4f2b4bccb1e4db38f2a173e085ceda028c38167912b466b0a84fcc0999", + "task.toml": "6bc98525852755caf1fe34c174c2f10f4328adb4ce6c1968dcafe1c877e4e160", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "345c3bd09ab6f6fe8c8361a58c0a47bf0a13b3fcb38a5ac7824e44ff855e8f72" + } + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-pipeline-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "297ddea07c230dfd246a8ab07337dd7abcd8e5e2a2b2e26590cb0a8af779839c", + "environment/Dockerfile": "c357ea41e4a4013c5c5980b8fef04af6a5ae250ccc6697cf3bad3127401a931f", + "instruction.md": "989bfcf0c54baf6c8cdc391743dbb2f9e858c5779dd85112536875d4d1b2dcd2", + "solution/solve.sh": "787198e4178d88b887697aefdd29819f29435ffdfe356545d92f58a30fb38b70", + "task.toml": "8b4f9f525c08afbbc4eca9168e46238a61ccb903486773e9ca4da64bb7ffe857", + "tests/test.sh": "cc9fd28b6590993fc37e4a99da771cbf04726c8c62fe308e802612241da69dea", + "tests/test_outputs.py": "23574246a407163a8e6581c84287cc2f1eb745efbaad2a317a5a28c39958c64f" + } + }, + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-tensor-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "4c1b3c3be30548f1f94c6f5503b1bc74b006af7f192dc4cd026f411df1a54604", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "13bde2ab165d72a311e2bf066d5d6c3c8761bf9d3dd39d1b23eda23bc546b910", + "solution/solve.sh": "d083b7cbf9dbb6cc2ff38c8f2acc150f0059c04aea42d6c2183aeaecc5a19385", + "task.toml": "72fd2a5500d1fe8795dcae76cccb1a29fb25894881a7475f2fa519c87d425a8d", + "tests/test.sh": "a7e6c9408b2401e4e7448371f7ddc72f3284caccfb56277d3cd4e98051868cdf", + "tests/test_outputs.py": "4ae4285e15fba1fcf13d7b7c0638ac8c2e2b0797c937abab1fb6b136fd508290" + } + }, + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/write-compressor:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "603015a28500a28f0838fea2c43a5168613746bfd07737920f4bae40b5a326f7", + "environment/Dockerfile": "9a583d29d46818f6ff2be7c6a0879d03343106e5376fd5a48eb5926b402bb93b", + "environment/data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "environment/decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "environment/main.rs": "fa53d6572fca95903e30d2d953f1e428bb8b63513ff218ebee9bc60a425ab389", + "instruction.md": "c4fc3e73e44deeac95f1cd4a34a2d87e853b22ceec2b2824767f5f087b470bb2", + "solution/solve.sh": "01550b7de7ab1320b1898eae055226bb7c73bee259c75102d2240c974a730409", + "task.toml": "6066a509066433c97c417fea68c4e3e73996f698dcdcbabe07718835eea355a6", + "tests/original-data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "tests/original-decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "809f486053e3ffbce852111dee1f63d80ab31297bd33172389e50278bed384e2" + } + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "native_agent_timeout_seconds": 1200, + "work_budget_seconds": 1020, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/constraints-scheduling:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "personal-assistant", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d54a690970149e1216bd4fc066b45069c7948c13e3b1ef2a213a74342805f209", + "environment/Dockerfile": "7d9aba072a4a907229728b9739c8cf821fc2c98450babd64452682caf3759c84", + "environment/inputs/alice_calendar.ics": "5a5d45804bffd339c2479d49e76fe817b38244d14d389afe1de583443808ecbc", + "environment/inputs/bob_calendar.ics": "c97e85dbcdd756758f8a80f4db5fd5c1dcacee79298e4d2262f46de5ff680b50", + "environment/inputs/carol_calendar.ics": "32a0f8687637df0a12b8e5c56d49c4192bc3e790856943632d70f237c7c1363b", + "instruction.md": "342def78177f9bc41bea4d11af679bc385a9dda2ac8a1e425be384a5b7f937b6", + "solution/solve.sh": "802d81ec064fad3050bbade95a2f847ac24e0b7f80fa69b0bcee0347dc92fe77", + "task.toml": "7b9842de16d74e8faf8c7c0aa03b0b81145941cb12dd322d81ae23fc48607250", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "1422b345ba3aacdfc4184e962811b6379988435896c50d8cf62903e70990c5d1" + } + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/custom-memory-heap-crash:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a06ed23a2c467aa36d0d4a88e5c020b1076cc25b8d483517239658e9c596dde9", + "environment/Dockerfile": "6ee81e5a4fa2540443d7f27d577ba840728fc6aab9f301ab7919103673c78080", + "environment/patches/locale_classes.tcc.patch": "16951f4a4df6bf253886ac75f5a39fdbc328e415194493f63cb7589a364aa817", + "environment/patches/locale_init.cc.patch": "e97f133ceebd86f8881f972e0e45b0ec0a13a9bfa428d8032008707683d54292", + "environment/program/main.cpp": "b94538c76887cc2b2ab739b5ac4731cddd0faa8f4a9892a6265c062bbbeede14", + "environment/program/user.cpp": "92b203786fed1bf0b3aa1675c9cd2d82b2651170675064da8ffee73b98ef14f9", + "environment/program/user.h": "f43d155995a56c22b9cd1417daef7c3b5a976cdc662a98eef0feb438ae1eee77", + "instruction.md": "53c68ae14a3c3af9a97c40bfd9bdc30a5d98b37eaa2f69f763475745f70f108d", + "solution/solve.sh": "b0b1d10bea1cd5be14a82b35d522df616df1e67028f90d2a6bef90aaa98511a1", + "task.toml": "ed474be1983e212913aa8866ebb66216acbe36aef2339ac375d232aa46617481", + "tests/test.sh": "70a2b55f0c5757f6b123a216dfd202be4a646680bced28dddcd31fd8959a058f", + "tests/test_outputs.py": "4e1cb33f777fa7fb5e1536f9bea9eace6deccd515c67d4f87570adbe692914ba" + } + }, + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/dna-assembly:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e7855c23d9822f688eec4dd89ca0f1caf91c6a69e1c8f3c9eb544b4ab2a94821", + "environment/Dockerfile": "26e9e43a98f0b1d132bb638f44b7dc882b0d0ca32d56fa33048adeee00ab600d", + "environment/sequences.fasta": "ebdc361b5fba28eb0a2c7a8a8a444ccb12cc842c8cbc6df5a2c24254f130736f", + "instruction.md": "88998c98beabdd966ab46c04ede822bbc74796696aa1bf87c567a9131210533c", + "solution/solve.sh": "c976cb118c8ef1befd846ba05fd5cadb1b1f529bc76b9f2ea9cb3049e6e11a47", + "task.toml": "7b571544b64d9be1c91526837e5d805c2181afe0e6fd57a7f0676bba36c868d7", + "tests/test.sh": "271c8317c98539e3b49233d017fe51fb0e5fba8114e76830fe698565499b2b1d", + "tests/test_outputs.py": "5c2d0fd2e5f6303263e9750f67acda7db491ef04841832b4fc62bcfb61b08924" + } + }, + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/llm-inference-batching-scheduler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ecc9dd635357c184742cf72fc1e452a8946e207b183fc9a7eebbcf07133b911f", + "environment/Dockerfile": "10a6193904ee0022ac098504182989893b196dfc74e626a3ca48534a8eacde48", + "environment/task_file/input_data/requests_bucket_1.jsonl": "8b3cf1e2b06074162038064dd2c114abe308acd32cec7d365ee63ee81978f7de", + "environment/task_file/input_data/requests_bucket_2.jsonl": "e15e2d4c4d55f875c42a6042610d6282f6c0657c39d3e0d0f087268252525cb5", + "environment/task_file/scripts/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "environment/task_file/scripts/baseline_packer.py": "547d230d5e6d93197803480cb81cab0ec6ac63fcfa0a24b246f549523f216c4e", + "environment/task_file/scripts/cost_model.py": "0cf417fd5ff0112defcd46970d37a290c0763c5a9092e91cb3ebb86c3f20e6ce", + "instruction.md": "0817374767533fbf51bb93345ee1bba266263c216c6cc55c096e71b4a1d37dfc", + "solution/solve.sh": "cb9efb586d76a893c3027e62029e286eb6c7583dc92dbf1840a3adc236a926d7", + "task.toml": "52aaac942f917e6a91ff658e0272195bd1cd842d41d71e302414c52603cc588b", + "tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "tests/cost_model_for_tests.py": "d3e810a99d40327fb8317f6512d879c52c6aeebdf5dcdc0c5e117c7df61c9ab9", + "tests/test.sh": "005f980332355e2fe143d4ff4aac11828adf070594bbe7055399c238f06953ca", + "tests/test_outputs.py": "80b06b7c4ba0db28832b767d48be74bb462d7f5df5798d9cb58c4f1160d96ffa" + } + }, + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/path-tracing:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3e4fc952b3f9a860c7ee7837b3f3c686f1283ee7821b014265dba7c94d7fc849", + "environment/Dockerfile": "7aca02e4e20b04ed2269d627a29dd4e7337a02764a9652b641452bf09836a3d5", + "environment/orig.c": "665c1bde0907740890ac114d6b36af1f3fd00449fa287583817f778bb5da200c", + "instruction.md": "9f36e02d636f5dc0946619bc518f2e51c24fa75251da9d1f0ca9eca612cb9572", + "solution/solve.sh": "d107abde101eecee6838da80cc52a3d480c93956fff3a9343a299ea01b3fe9e6", + "task.toml": "cf47013f69bf1b43242da984f231d4daf364fdf44d3845f2eea2ca2cb9308c46", + "tests/test.sh": "a719dbb1ba7ae76df095e9d815941dfa4c55edaf11f173b9c3c3097afa838548", + "tests/test_outputs.py": "572e68b2843191d29592faf42a405861b7958d8e894868b74c5a8886b53eb879" + } + }, + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6", + "native_agent_timeout_seconds": 2400, + "work_budget_seconds": 2220, + "verifier_timeout_seconds": 2400.0, + "docker_image": "alexgshaw/schemelike-metacircular-eval:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "095d3ec6dcc0fe0c7a7e92355456dcea6992f91d1f8d397669b8cfe18475f506", + "environment/Dockerfile": "9b3df76222e16aad360fbd34cb0f71a0be8986052c065b10b92c0cdf417b25c7", + "environment/tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "environment/tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "environment/tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "environment/tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "environment/tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "environment/tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "environment/tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "environment/tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "environment/tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "environment/tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "environment/tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "environment/tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "environment/tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "environment/tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "environment/tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "environment/tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "environment/tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "environment/tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "environment/tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "environment/tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "environment/tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "environment/tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "environment/tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "environment/tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "environment/tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "environment/tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "environment/tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "environment/tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "environment/tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "environment/tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "environment/tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "environment/tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "environment/tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "environment/tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "environment/tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "environment/tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "environment/tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "environment/tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "environment/tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "environment/tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "environment/tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "environment/tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "environment/tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "environment/tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "environment/tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "environment/tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "environment/tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "environment/tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "environment/tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "environment/tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "environment/tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "environment/tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "environment/tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "environment/tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "environment/tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "environment/tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "environment/tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "environment/tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "environment/tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "environment/tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "environment/tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "environment/tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "environment/tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "environment/tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "environment/tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "environment/tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed", + "instruction.md": "3af5f9567b62f93683db7518ec6fcb609f83796258aa02166b5a7091c5b69031", + "solution/solve.sh": "e2c5b0ac2bef4801a1891e13c108813381877426c1a7f01bab86b4fc1f712510", + "task.toml": "8dae8eb226183b0ca92331c1106152a7abc93b03b4e5a95291b86a4ddc9b44e0", + "tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed" + } + }, + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/caffe-cifar-10:20260403", + "resources": { + "cpus": 4, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e918e6852c5c034422b81a6be22d7596fddcee53abe088f56727b2482da32728", + "environment/Dockerfile": "b3b8adc915a1afa035b604273af3bc1aa8b90999d2dfd72763218b34cb87abcd", + "instruction.md": "46907b0ee048f74269c29a15e0e086546b2b1220149597017da8b5ccc05a0cea", + "solution/solve.sh": "771887de1ed557269c7f3160c777519aa9538e1e9b59771b2f6ef38c8952490c", + "task.toml": "2831165ad9d9e8bbc646e6d85c0a2290640dcc4f13749547a2937c7675012a8b", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "e99b9f6284e923388fbc21488b1af99fdde46d4a809d3557b0788dbf1cb6b281" + } + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/circuit-fibsqrt:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "201f7c889dd6f73dd83ca078e347272b33720fa622f8c757ea6d5770453f4cbd", + "environment/Dockerfile": "780c23986778f3e5f842567868af30b1135ed095ca9449e1c90b7f0e0483d3d4", + "environment/gates.txt": "032786f94f16605767c62bb61ce4dbba7ab7d3f1046712c975f01c9a04b6ad29", + "environment/tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "environment/tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438", + "instruction.md": "a7f500266a878d04ac9de39fba76220cfad216d8f6b8063bba93bebb3d4ab9ea", + "solution/solve.sh": "a4ffab4183815a0eb7646abc1de1fc67695014c060d7ff16e8bca9ec94385d0b", + "task.toml": "9f5aa1f7ebbbcec90fd109121e9bd509af3cfff6b4cec701bd91947988f40972", + "tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438" + } + }, + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/mteb-leaderboard:20260430", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ed147a5bc31e9865b16848b2f01d2f9b1e677af57a7c632d7ea2311c0553ca76", + "environment/Dockerfile": "271fdb5b34520c0bf2cb6462365363ec52058f127cd9eea3f5da2589df5c1044", + "instruction.md": "06ea44106eca0405c25bab00f5b3cfeb611c872093f6c0f82ed057839d1d13de", + "solution/solve.sh": "4ed68831d4acf5d80d2d73eec1f21efd0896a6f51f23020f572cfc0af6da0410", + "task.toml": "86574bd1c6c4fefd2e64eee9c2caa53d87c4bcf2e51547912440a0ddbd0badb8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "b28f8fc8b7614801305813c402be52f4be1e75de6658ecdfdf9383aa793892cf" + } + }, + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/regex-chess:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9296cbee592fc996c58d228b51fee303647e5ecfae82d5666d42a40cb30ec355", + "environment/Dockerfile": "2616cd775252303839dfda0ee2f047eb862952b053b0103bd6b2460ef90c11b7", + "environment/check.py": "f9c416ae9ea01dcdd5f6a43170c8434b8c0a0b389a4a30876a2048e277d191c7", + "instruction.md": "a6ed3fcc54fc92d71a84af921942422d299ca89b410dcd92cd05a53d0d06436c", + "solution/solve.sh": "b728bf0ea392a3d244afbd0818d36933c8d3e190e47743c1382cd518d518a2e9", + "task.toml": "07fe52636fcc4bc9fff25d5630af145b97a5331219d719cc8decb4ca67ea21d4", + "tests/test.sh": "8f7f00a1c4dde7fcca6c918c768f2aba432b8c4007c15989716c68091950da87", + "tests/test_outputs.py": "4983c44d21c56dd8191381eff0304afc8b87bb714527ea9036b3319c015a30b4" + } + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "native_agent_timeout_seconds": 12000, + "work_budget_seconds": 11820, + "verifier_timeout_seconds": 12000.0, + "docker_image": "alexgshaw/build-pov-ray:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "38380f64bc631916e264f80ce4d37feedbc783310001475b367d03989d114fdf", + "environment/Dockerfile": "44f2b5fc7311d8f336daaaca99dd4216b49355e67f50b494ebacf1b0f7c14e92", + "environment/deps/illum1.pov": "5a1f7f53ee0730c4923fcd21cd4a9c34719024d8557920441551ba7d96e75dc4", + "instruction.md": "56c8ecdd0084a07f8c97986c43af5c37ed03dd5aa994da6735cf59806162d1b2", + "solution/solve.sh": "3da9dbf589656512cd7a7004dc3752d3552a6af939cfe59eff79f91695fa42c7", + "task.toml": "b90a47ec26c1191d3c523cc094ea96b170e42fbfe176ef1f814c57d05390c097", + "tests/reference_illum1.png": "bb084f430bbc9d5b623a656920670dc06cee6d9132a9d550be4a1b99e41ab9e0", + "tests/test.sh": "928d1497ad21aef0c456294e3784f1fddf8493410f23584da89f9e02ec6bd134", + "tests/test_outputs.py": "64d6ce107503714ada346e5606832ee886be9950971cf81523e73d8ba28e8d37" + } + } + ], + "total_model_runs": 34, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "preflight": [ + { + "arm": "main", + "job_name": "dev34-main-install-only-20261012", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only.json", + "config_sha256": "e65a0f4ee097d0231c8bc854484ba376b52eeb60fb7e5f120360310c92c347a4" + } + ], + "queue": [ + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/adaptive-rejection-sampler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/adaptive-rejection-sampler.json", + "job_name": "dev34-adaptive-rejection-sampler-main-20261012", + "config_sha256": "e0fec8e3f1f8e077a5384058ea5b6e4733973a7475b02556f90cfc0f4ba96f6d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-cython-ext", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/build-cython-ext.json", + "job_name": "dev34-build-cython-ext-main-20261012", + "config_sha256": "d5aaf7b0731b61618d2e3b017a3f0af8566a126da7159c5d536e66d4e87060d7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cancel-async-tasks", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/cancel-async-tasks.json", + "job_name": "dev34-cancel-async-tasks-main-20261012", + "config_sha256": "23b32b31a46e5857e0c4438aff6f1c7a5313ea41c208aba37ea30506d0c81e02" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cobol-modernization", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/cobol-modernization.json", + "job_name": "dev34-cobol-modernization-main-20261012", + "config_sha256": "2680126c2f162d58948cd30565806bf6cb1f026d561c87b942ba4735173ea6a2" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/db-wal-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/db-wal-recovery.json", + "job_name": "dev34-db-wal-recovery-main-20261012", + "config_sha256": "475b7df31549e51f04f4f9171332da5a8ba03f7fbec2b3788fbf897340c0c91b" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/fix-code-vulnerability", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/fix-code-vulnerability.json", + "job_name": "dev34-fix-code-vulnerability-main-20261012", + "config_sha256": "4b2de43b547ddd92d748f7658c44b98b4fef25efd406710ace6d1713b9c789a7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/git-multibranch", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/git-multibranch.json", + "job_name": "dev34-git-multibranch-main-20261012", + "config_sha256": "ae9c6d5830c9dcb8aab8b1356441b47731957e2ad5e5747327b7caa1a6c037a1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/kv-store-grpc", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/kv-store-grpc.json", + "job_name": "dev34-kv-store-grpc-main-20261012", + "config_sha256": "b13e70bc9a91788ad62db8445169f6cdab8deabcec07d1efebde77574e3af256" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/log-summary-date-ranges", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/log-summary-date-ranges.json", + "job_name": "dev34-log-summary-date-ranges-main-20261012", + "config_sha256": "9b01ea36ccb9b9ac9dea6a210e04aa95200d6a926676500f1d2a20066d22563b" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/merge-diff-arc-agi-task", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/merge-diff-arc-agi-task.json", + "job_name": "dev34-merge-diff-arc-agi-task-main-20261012", + "config_sha256": "762334e31f8710aa97ae88459acf208017ea121a15226f0c1e98a5ee662b4b2e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/model-extraction-relu-logits", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/model-extraction-relu-logits.json", + "job_name": "dev34-model-extraction-relu-logits-main-20261012", + "config_sha256": "10206628d392e6268c71af7d5b5217346aac0982234827e68574d3282b64d0ae" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/multi-source-data-merger", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/multi-source-data-merger.json", + "job_name": "dev34-multi-source-data-merger-main-20261012", + "config_sha256": "5eac4294149a11f037f732a32e1a8874613d8a28884843955481a9e8ef10b7bd" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/nginx-request-logging", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/nginx-request-logging.json", + "job_name": "dev34-nginx-request-logging-main-20261012", + "config_sha256": "4a79d4fc4431acd942548f36188c1820ac5406fc6a1385cf505daf02aa190da3" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/openssl-selfsigned-cert", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/openssl-selfsigned-cert.json", + "job_name": "dev34-openssl-selfsigned-cert-main-20261012", + "config_sha256": "c0d74fe7ddefdb8e849a38703f25724bbb69786b12a0081945986860e4e6898c" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pypi-server", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/pypi-server.json", + "job_name": "dev34-pypi-server-main-20261012", + "config_sha256": "748479904b2b4eb2af9afea22017037a46f5b1d8434e4eaf0d4649d0c50fe1f0" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-cli", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-cli.json", + "job_name": "dev34-pytorch-model-cli-main-20261012", + "config_sha256": "bc915a7de1286954473271e2232f9c83af983c23b885d099316b33dc3c1d445d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-recovery.json", + "job_name": "dev34-pytorch-model-recovery-main-20261012", + "config_sha256": "b0b7cf9c11e8fc0d2b88ff7e8b7e1d9f95da8be3a0b5cef65cd62e7380aef796" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/qemu-alpine-ssh", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/qemu-alpine-ssh.json", + "job_name": "dev34-qemu-alpine-ssh-main-20261012", + "config_sha256": "316693982d2ba8e9704de5a82a58530b1836d13c06ab53e72cf684b02b1aa3db" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/query-optimize", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/query-optimize.json", + "job_name": "dev34-query-optimize-main-20261012", + "config_sha256": "29ad0478282cd6fb86a1b209a93ffe33eee724c321916732db001b7beef8ea7e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-log", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-log.json", + "job_name": "dev34-regex-log-main-20261012", + "config_sha256": "34c42f2419bdfa381cd3cf0de7a3bc502bc8c295ae1301e3a0958ac6835c8532" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-pipeline-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-pipeline-parallelism.json", + "job_name": "dev34-torch-pipeline-parallelism-main-20261012", + "config_sha256": "a00e2f0c1d42b55c967eb96e4607f733d0290e5461a4bed5e49eb214462751c4" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-tensor-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-tensor-parallelism.json", + "job_name": "dev34-torch-tensor-parallelism-main-20261012", + "config_sha256": "14d7266c68e33a71b61cdde9d1f9f9d336c998287301303d15b507fb38564c9a" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/write-compressor", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/write-compressor.json", + "job_name": "dev34-write-compressor-main-20261012", + "config_sha256": "6d651666c02a7c8e2ed73c8fdc5b0cc9ed2ed475b7fa44333eb03c2e1c5159fd" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/constraints-scheduling", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/constraints-scheduling.json", + "job_name": "dev34-constraints-scheduling-main-20261012", + "config_sha256": "33813c2fd5b45efd3b7bf8a765fcb1f74c702821a3d4f473e4b479200433734e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/custom-memory-heap-crash", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/custom-memory-heap-crash.json", + "job_name": "dev34-custom-memory-heap-crash-main-20261012", + "config_sha256": "4678ab6961d82a2d597911bd8d4a53040a4be526c1e97afb27c614ecedb1c05f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/dna-assembly", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/dna-assembly.json", + "job_name": "dev34-dna-assembly-main-20261012", + "config_sha256": "00ae5d125dacb75c1fdfe0207b5749e23bb70bef71c185eda7573a27a6deccdf" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/llm-inference-batching-scheduler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/llm-inference-batching-scheduler.json", + "job_name": "dev34-llm-inference-batching-scheduler-main-20261012", + "config_sha256": "b62e9d33f024b01457d6163a7892f19187131b192fbd5a697f6d901b1792452d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/path-tracing", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/path-tracing.json", + "job_name": "dev34-path-tracing-main-20261012", + "config_sha256": "91e3cb3ecd0e524f97cc8340023fa979c926d2dd645c25cc5262a7e7b061be4b" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/schemelike-metacircular-eval", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/schemelike-metacircular-eval.json", + "job_name": "dev34-schemelike-metacircular-eval-main-20261012", + "config_sha256": "69fa2b257305717285fe5a32e89d78bda0ce94c4780ee7dfebe26b2bf3f512c8" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/caffe-cifar-10", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/caffe-cifar-10.json", + "job_name": "dev34-caffe-cifar-10-main-20261012", + "config_sha256": "d00004edd6d3e04f22276d5ce4f70445069b3dd3ca4c516b6e72e5b8efe42e0f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/circuit-fibsqrt", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/circuit-fibsqrt.json", + "job_name": "dev34-circuit-fibsqrt-main-20261012", + "config_sha256": "37fb3ec114fae088f27c3b2d0c4c22068fd7e3748e88fd97bec3a844458091d7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/mteb-leaderboard", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/mteb-leaderboard.json", + "job_name": "dev34-mteb-leaderboard-main-20261012", + "config_sha256": "c3e5b16d764f15fbe417622a0ffed1584e5e7f68f422449aea568420b711215c" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-chess", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-chess.json", + "job_name": "dev34-regex-chess-main-20261012", + "config_sha256": "0a9ebee2dd610cbd784be84f2989284e24cd8b4edd5a561629206528ec544bc6" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-pov-ray", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/build-pov-ray.json", + "job_name": "dev34-build-pov-ray-main-20261012", + "config_sha256": "42e1bb7d2a1a6d5cf9f059a121140f9135d8b4632482a00d767a9aba2e1ac2e7" + } + ], + "log_dir": "jobs/dev34-main-baseline-20261012-orchestration", + "image_digests_file": "image-digests.json", + "image_digests_sha256": "5fc06fa5fbac1cccd3a4fb2f560878dd79ce314a6b1005183bbdcb4e8e13381f", + "budget_policy": "Native task agent timeout minus 180 seconds, separate 1800-second setup. Native verifier per-attempt time limits; max three attempts, retry only timeouts against the same agent output.", + "execution_policy": "All 34 model trials are fresh. Fixed ascending work-budget order, then task name; consecutive pairs finish before the next pair. No whole-trial retries, task replacements, or reuse of previous model outputs.", + "failure_policy": "Keep all setup failures and verifier errors separate from scored failures. Retain native zeros even when assertions never start, but classify them as verifier errors in analysis.", + "preflight_policy": "Fresh installation-only preflight for all 34 tasks before model work. QEMU has a reviewed verifier dependency profile; the other 33 are explicitly not_configured. A pre-model infrastructure amendment may reuse successful native installation trials only with complete hash-bound provenance and retain all failures.", + "information_policy": "Use frozen current main without prompt or runtime tuning. Do not inspect new trajectories or hidden test contents until all new model outputs freeze. Prior development exposure is disclosed, and each completed trajectory will be audited for benchmark-answer exposure.", + "scope": "One common-main baseline of the existing 34-task development pool, not a comparison with prior mixed-version results and not a random or held-out benchmark." +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/fix-code-vulnerability.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/fix-code-vulnerability.json new file mode 100644 index 0000000..9e19897 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/fix-code-vulnerability.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/git-multibranch.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/git-multibranch.json new file mode 100644 index 0000000..5956b98 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/git-multibranch.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/image-digests.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/image-digests.json new file mode 100644 index 0000000..0749abd --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/image-digests.json @@ -0,0 +1,311 @@ +{ + "observed_at": "2026-10-12T13:52:35.659664+00:00", + "images": [ + { + "task": "terminal-bench/build-cython-ext", + "image": "alexgshaw/build-cython-ext:20251031", + "image_id": "sha256:3612a38fadb89a96f74a1a951fb0b0af734198fd160571eeaba6401593234594", + "repo_digests": [ + "alexgshaw/build-cython-ext@sha256:3612a38fadb89a96f74a1a951fb0b0af734198fd160571eeaba6401593234594" + ], + "size_bytes": 303222492 + }, + { + "task": "terminal-bench/caffe-cifar-10", + "image": "alexgshaw/caffe-cifar-10:20260403", + "image_id": "sha256:929a6d631b592c82e0d0f5d4a65cf03c946dc554cfc9bb09916e82076e7d3215", + "repo_digests": [ + "alexgshaw/caffe-cifar-10@sha256:929a6d631b592c82e0d0f5d4a65cf03c946dc554cfc9bb09916e82076e7d3215" + ], + "size_bytes": 213171982 + }, + { + "task": "terminal-bench/cancel-async-tasks", + "image": "alexgshaw/cancel-async-tasks:20251031", + "image_id": "sha256:84c7fae6b256dcc56a350790e2a9715eefc7dad662a9d8e8a472363aa71ef18d", + "repo_digests": [ + "alexgshaw/cancel-async-tasks@sha256:84c7fae6b256dcc56a350790e2a9715eefc7dad662a9d8e8a472363aa71ef18d" + ], + "size_bytes": 44171676 + }, + { + "task": "terminal-bench/circuit-fibsqrt", + "image": "alexgshaw/circuit-fibsqrt:20251031", + "image_id": "sha256:29783439f529eaed2145592f15af7a2281161528860392a382c825489030ae3a", + "repo_digests": [ + "alexgshaw/circuit-fibsqrt@sha256:29783439f529eaed2145592f15af7a2281161528860392a382c825489030ae3a" + ], + "size_bytes": 149136738 + }, + { + "task": "terminal-bench/dna-assembly", + "image": "alexgshaw/dna-assembly:20251031", + "image_id": "sha256:d1adf6835f1dd91205ba70e452c699d0aea601010038e5617f370716efb50569", + "repo_digests": [ + "alexgshaw/dna-assembly@sha256:d1adf6835f1dd91205ba70e452c699d0aea601010038e5617f370716efb50569" + ], + "size_bytes": 29727501 + }, + { + "task": "terminal-bench/kv-store-grpc", + "image": "alexgshaw/kv-store-grpc:20251031", + "image_id": "sha256:3399400800dcb207634daa42bc1b052e831e285cc9d221eea66c47bc0fc79791", + "repo_digests": [ + "alexgshaw/kv-store-grpc@sha256:3399400800dcb207634daa42bc1b052e831e285cc9d221eea66c47bc0fc79791" + ], + "size_bytes": 44171671 + }, + { + "task": "terminal-bench/llm-inference-batching-scheduler", + "image": "alexgshaw/llm-inference-batching-scheduler:20251031", + "image_id": "sha256:19e79aa49be4e55dccca1dde966515ebe6852314b6010bcb9230afb48b996774", + "repo_digests": [ + "alexgshaw/llm-inference-batching-scheduler@sha256:19e79aa49be4e55dccca1dde966515ebe6852314b6010bcb9230afb48b996774" + ], + "size_bytes": 44188631 + }, + { + "task": "terminal-bench/log-summary-date-ranges", + "image": "alexgshaw/log-summary-date-ranges:20251031", + "image_id": "sha256:cbeb6ba905c2fec294f16cd5e16e3ea7f2e04d38ac2484d51a11de262aa7dc51", + "repo_digests": [ + "alexgshaw/log-summary-date-ranges@sha256:cbeb6ba905c2fec294f16cd5e16e3ea7f2e04d38ac2484d51a11de262aa7dc51" + ], + "size_bytes": 44866158 + }, + { + "task": "terminal-bench/merge-diff-arc-agi-task", + "image": "alexgshaw/merge-diff-arc-agi-task:20251031", + "image_id": "sha256:bfc2a235f2ceea64ffb1ffc5529c2acebf570bff61f3668f0e5669ef570c17d4", + "repo_digests": [ + "alexgshaw/merge-diff-arc-agi-task@sha256:bfc2a235f2ceea64ffb1ffc5529c2acebf570bff61f3668f0e5669ef570c17d4" + ], + "size_bytes": 29728285 + }, + { + "task": "terminal-bench/model-extraction-relu-logits", + "image": "alexgshaw/model-extraction-relu-logits:20251031", + "image_id": "sha256:52fe1f089f38650f0dc22d7531ee6e01ebd526de1b349aaa2ecb331dca9fabce", + "repo_digests": [ + "alexgshaw/model-extraction-relu-logits@sha256:52fe1f089f38650f0dc22d7531ee6e01ebd526de1b349aaa2ecb331dca9fabce" + ], + "size_bytes": 83698971 + }, + { + "task": "terminal-bench/mteb-leaderboard", + "image": "alexgshaw/mteb-leaderboard:20260430", + "image_id": "sha256:a8538f4882ff132e0a20a75be3fffc07164318a7425ad19a6b4bf5268107feb4", + "repo_digests": [ + "alexgshaw/mteb-leaderboard@sha256:a8538f4882ff132e0a20a75be3fffc07164318a7425ad19a6b4bf5268107feb4" + ], + "size_bytes": 663825427 + }, + { + "task": "terminal-bench/openssl-selfsigned-cert", + "image": "alexgshaw/openssl-selfsigned-cert:20251031", + "image_id": "sha256:4c948a4e630af2435ae0a19108fc0814a946ac2fa29a512469e0fc77b38c8c12", + "repo_digests": [ + "alexgshaw/openssl-selfsigned-cert@sha256:4c948a4e630af2435ae0a19108fc0814a946ac2fa29a512469e0fc77b38c8c12" + ], + "size_bytes": 44173158 + }, + { + "task": "terminal-bench/path-tracing", + "image": "alexgshaw/path-tracing:20251031", + "image_id": "sha256:4b44725102cd627dfa895191a5a30d16dae3c5623bb43da6459f79135db6b482", + "repo_digests": [ + "alexgshaw/path-tracing@sha256:4b44725102cd627dfa895191a5a30d16dae3c5623bb43da6459f79135db6b482" + ], + "size_bytes": 399390468 + }, + { + "task": "terminal-bench/pypi-server", + "image": "alexgshaw/pypi-server:20251031", + "image_id": "sha256:d18bb30f47c7dcaa3acdbc31ac7413c98eadca2ea16540bbd380f2f063b95276", + "repo_digests": [ + "alexgshaw/pypi-server@sha256:d18bb30f47c7dcaa3acdbc31ac7413c98eadca2ea16540bbd380f2f063b95276" + ], + "size_bytes": 83726049 + }, + { + "task": "terminal-bench/pytorch-model-recovery", + "image": "alexgshaw/pytorch-model-recovery:20260430", + "image_id": "sha256:7c2bb52851cef25bdc3939e473496d25fbf11fd0cb7ab58bcf3cefcb751cdc0e", + "repo_digests": [ + "alexgshaw/pytorch-model-recovery@sha256:7c2bb52851cef25bdc3939e473496d25fbf11fd0cb7ab58bcf3cefcb751cdc0e" + ], + "size_bytes": 269661811 + }, + { + "task": "terminal-bench/qemu-alpine-ssh", + "image": "alexgshaw/qemu-alpine-ssh:20251031", + "image_id": "sha256:541e8214a8c4faa7b556531b899d45da21048fa2ca6dd328d8cb6c4cffa8e964", + "repo_digests": [ + "alexgshaw/qemu-alpine-ssh@sha256:541e8214a8c4faa7b556531b899d45da21048fa2ca6dd328d8cb6c4cffa8e964" + ], + "size_bytes": 1308553944 + }, + { + "task": "terminal-bench/regex-chess", + "image": "alexgshaw/regex-chess:20251031", + "image_id": "sha256:e3b6d61d2d3930fbbdc5f7bf091a7f8b7f6125e1951ef8a7dbe165b1ef62942c", + "repo_digests": [ + "alexgshaw/regex-chess@sha256:e3b6d61d2d3930fbbdc5f7bf091a7f8b7f6125e1951ef8a7dbe165b1ef62942c" + ], + "size_bytes": 55625584 + }, + { + "task": "terminal-bench/regex-log", + "image": "alexgshaw/regex-log:20251031", + "image_id": "sha256:90101b2e815323a8da20528a1439bebc407eb9761c9c68a3d557730856c878e9", + "repo_digests": [ + "alexgshaw/regex-log@sha256:90101b2e815323a8da20528a1439bebc407eb9761c9c68a3d557730856c878e9" + ], + "size_bytes": 29725514 + }, + { + "task": "terminal-bench/schemelike-metacircular-eval", + "image": "alexgshaw/schemelike-metacircular-eval:20251031", + "image_id": "sha256:b485be38bf80d56c1a4cd9da95bbf7ec129dc03f6786b86c0525518a96f6d1db", + "repo_digests": [ + "alexgshaw/schemelike-metacircular-eval@sha256:b485be38bf80d56c1a4cd9da95bbf7ec129dc03f6786b86c0525518a96f6d1db" + ], + "size_bytes": 44188336 + }, + { + "task": "terminal-bench/torch-pipeline-parallelism", + "image": "alexgshaw/torch-pipeline-parallelism:20251031", + "image_id": "sha256:3cb7b39d86f1704b20e668851da0b08b34b3270c01d583f136d65190984a8d1b", + "repo_digests": [ + "alexgshaw/torch-pipeline-parallelism@sha256:3cb7b39d86f1704b20e668851da0b08b34b3270c01d583f136d65190984a8d1b" + ], + "size_bytes": 29725531 + }, + { + "task": "terminal-bench/torch-tensor-parallelism", + "image": "alexgshaw/torch-tensor-parallelism:20251031", + "image_id": "sha256:c50ac169e11465f41b49749fe87e4487f24d31a0c39c0906cd3242cc5499c690", + "repo_digests": [ + "alexgshaw/torch-tensor-parallelism@sha256:c50ac169e11465f41b49749fe87e4487f24d31a0c39c0906cd3242cc5499c690" + ], + "size_bytes": 29725529 + }, + { + "task": "terminal-bench/write-compressor", + "image": "alexgshaw/write-compressor:20251031", + "image_id": "sha256:3618e1f8a997b437c09cd3dbaac736705809c05f6f12f584b65722175970ebe1", + "repo_digests": [ + "alexgshaw/write-compressor@sha256:3618e1f8a997b437c09cd3dbaac736705809c05f6f12f584b65722175970ebe1" + ], + "size_bytes": 291927726 + }, + { + "task": "terminal-bench/build-pov-ray", + "image": "alexgshaw/build-pov-ray:20251031", + "image_id": "sha256:874baf49341e2853a15a9184e308c7eb632e97f434359fcff2891c6e5ca1b39b", + "repo_digests": [ + "alexgshaw/build-pov-ray@sha256:874baf49341e2853a15a9184e308c7eb632e97f434359fcff2891c6e5ca1b39b" + ], + "size_bytes": 163066497 + }, + { + "task": "terminal-bench/cobol-modernization", + "image": "alexgshaw/cobol-modernization:20251031", + "image_id": "sha256:593ab9df3d83f771e927888b2d9436b380cdf7ec4d0af2a9f0802d541edebba0", + "repo_digests": [ + "alexgshaw/cobol-modernization@sha256:593ab9df3d83f771e927888b2d9436b380cdf7ec4d0af2a9f0802d541edebba0" + ], + "size_bytes": 162020401 + }, + { + "task": "terminal-bench/custom-memory-heap-crash", + "image": "alexgshaw/custom-memory-heap-crash:20251031", + "image_id": "sha256:1895d7d6240d5fb2ec774d6bfdd1057d170f8d199c15c582e4585d3c63802557", + "repo_digests": [ + "alexgshaw/custom-memory-heap-crash@sha256:1895d7d6240d5fb2ec774d6bfdd1057d170f8d199c15c582e4585d3c63802557" + ], + "size_bytes": 1267999268 + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "image": "alexgshaw/fix-code-vulnerability:20251031", + "image_id": "sha256:cac325252991f823713b2d0441502972901dd782bd67f66c03d9b1e410dac5c0", + "repo_digests": [ + "alexgshaw/fix-code-vulnerability@sha256:cac325252991f823713b2d0441502972901dd782bd67f66c03d9b1e410dac5c0" + ], + "size_bytes": 190696453 + }, + { + "task": "terminal-bench/nginx-request-logging", + "image": "alexgshaw/nginx-request-logging:20251031", + "image_id": "sha256:693e64431a0dbf45952e8b91a298d54fd60247e501c6623fb8fa3ebc07fc3d3d", + "repo_digests": [ + "alexgshaw/nginx-request-logging@sha256:693e64431a0dbf45952e8b91a298d54fd60247e501c6623fb8fa3ebc07fc3d3d" + ], + "size_bytes": 67285778 + }, + { + "task": "terminal-bench/git-multibranch", + "image": "alexgshaw/git-multibranch:20251031", + "image_id": "sha256:5bc449921c138295f6ddd8dca2977319fee2ce523811dbe56cec4ca663385fb5", + "repo_digests": [ + "alexgshaw/git-multibranch@sha256:5bc449921c138295f6ddd8dca2977319fee2ce523811dbe56cec4ca663385fb5" + ], + "size_bytes": 222134454 + }, + { + "task": "terminal-bench/db-wal-recovery", + "image": "alexgshaw/db-wal-recovery:20251031", + "image_id": "sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e", + "repo_digests": [ + "alexgshaw/db-wal-recovery@sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e" + ], + "size_bytes": 175276145 + }, + { + "task": "terminal-bench/multi-source-data-merger", + "image": "alexgshaw/multi-source-data-merger:20251031", + "image_id": "sha256:8b32782078ff7383a1b4e5d3cecca8ce287e30f50bb4c0e5db009a18064e666e", + "repo_digests": [ + "alexgshaw/multi-source-data-merger@sha256:8b32782078ff7383a1b4e5d3cecca8ce287e30f50bb4c0e5db009a18064e666e" + ], + "size_bytes": 220880128 + }, + { + "task": "terminal-bench/query-optimize", + "image": "alexgshaw/query-optimize:20251031", + "image_id": "sha256:28de7da87004eab5b29bd5eaf1823579bc5b73ccea4b80349a5623185770da14", + "repo_digests": [ + "alexgshaw/query-optimize@sha256:28de7da87004eab5b29bd5eaf1823579bc5b73ccea4b80349a5623185770da14" + ], + "size_bytes": 120494963 + }, + { + "task": "terminal-bench/adaptive-rejection-sampler", + "image": "alexgshaw/adaptive-rejection-sampler:20251031", + "image_id": "sha256:985bed39e8448bc95f138b5f766b59003a95c4be171c96af8e76448338f3dd8f", + "repo_digests": [ + "alexgshaw/adaptive-rejection-sampler@sha256:985bed39e8448bc95f138b5f766b59003a95c4be171c96af8e76448338f3dd8f" + ], + "size_bytes": 29731756 + }, + { + "task": "terminal-bench/constraints-scheduling", + "image": "alexgshaw/constraints-scheduling:20251031", + "image_id": "sha256:567ce5a189f8d11ac461790876e934cc7af38391baf89a78f95f4dafc1fec3b0", + "repo_digests": [ + "alexgshaw/constraints-scheduling@sha256:567ce5a189f8d11ac461790876e934cc7af38391baf89a78f95f4dafc1fec3b0" + ], + "size_bytes": 49852952 + }, + { + "task": "terminal-bench/pytorch-model-cli", + "image": "alexgshaw/pytorch-model-cli:20251031", + "image_id": "sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "repo_digests": [ + "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67" + ], + "size_bytes": 44349611 + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only.json new file mode 100644 index 0000000..7a66583 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only.json @@ -0,0 +1,155 @@ +{ + "n_concurrent_trials": 2, + "n_attempts": 1, + "install_only": true, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + }, + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12" + }, + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1" + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + }, + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307" + }, + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + }, + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d" + }, + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17" + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + }, + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302" + }, + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1" + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + }, + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc" + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + }, + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684" + }, + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e" + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + }, + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f" + }, + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67" + }, + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c" + }, + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6" + }, + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf" + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + }, + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c" + }, + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78" + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/kv-store-grpc.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/kv-store-grpc.json new file mode 100644 index 0000000..da86dd6 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/kv-store-grpc.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/llm-inference-batching-scheduler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/llm-inference-batching-scheduler.json new file mode 100644 index 0000000..64469e8 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/llm-inference-batching-scheduler.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/log-summary-date-ranges.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/log-summary-date-ranges.json new file mode 100644 index 0000000..e89bef2 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/log-summary-date-ranges.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/main-prompt.txt b/benchmarks/harbor/configs/dev34-main-baseline-20261012/main-prompt.txt new file mode 100644 index 0000000..56116f5 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/main-prompt.txt @@ -0,0 +1,9 @@ +You are nanoPyCodeAgent, a coding agent running non-interactively on a task handed to you up front. There is no user to reply to you: never ask a clarifying question, never stop to wait for confirmation, and never present a plan for approval — decide on your own and carry it out. Work the task through to the end, then check the result with the tools instead of assuming it worked. When it is done, answer with a short summary and no further tool calls: that reply is what ends the run. Runtime budget reminders report remaining time and model replies; when either is running low, prioritize saving the required deliverables, necessary verification, and a final summary. Do not start optional work after the requirements are satisfied. + +Identify the required deliverables, interfaces, and constraints. Treat examples as evidence, not as the full contract unless the task says so. Inspect relevant files, dependencies, and existing checks instead of guessing. Task-supplied workspace files, including source, executables, and tests, are available references unless the task restricts their use. Consult general documentation as needed. Do not seek or use task-specific reference solutions or hidden evaluation tests from outside the supplied materials unless the task explicitly requests them. Derive checks from the stated requirements. + +Verify the requested behavior through the real entry point when feasible. Check the content and meaning of outputs, not only command success or file existence. For logic or numerical work, choose a few representative and boundary cases justified by the task, using an independent reference, invariant, or consistency check when available. Do not derive every expectation from the implementation being tested or weaken requirements to make a check pass. Keep verification proportional to the task and remaining budget. + +When a check fails, inspect the error and relevant output, revise the underlying assumption or implementation, and rerun the affected check. Keep the task's constraints intact. Once appropriate checks support the required behavior, save the deliverables and finish. State what was verified and any remaining limitation without claiming unobserved success. + +Prefer the read tool for viewing files, the edit tool for changing part of an existing file, and the write tool for creating files or rewriting them whole. Use the bash tool to run commands, search with grep, and complete tasks that need real command output instead of guessing. \ No newline at end of file diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/merge-diff-arc-agi-task.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/merge-diff-arc-agi-task.json new file mode 100644 index 0000000..ad36630 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/merge-diff-arc-agi-task.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/model-extraction-relu-logits.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/model-extraction-relu-logits.json new file mode 100644 index 0000000..8561939 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/model-extraction-relu-logits.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/mteb-leaderboard.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/mteb-leaderboard.json new file mode 100644 index 0000000..35bf46c --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/mteb-leaderboard.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/multi-source-data-merger.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/multi-source-data-merger.json new file mode 100644 index 0000000..6c55a0a --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/multi-source-data-merger.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/nginx-request-logging.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/nginx-request-logging.json new file mode 100644 index 0000000..be8ea55 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/nginx-request-logging.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/openssl-selfsigned-cert.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/openssl-selfsigned-cert.json new file mode 100644 index 0000000..878614e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/openssl-selfsigned-cert.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/path-tracing.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/path-tracing.json new file mode 100644 index 0000000..6867ecd --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/path-tracing.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/pypi-server.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/pypi-server.json new file mode 100644 index 0000000..dec55d6 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/pypi-server.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-cli.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-cli.json new file mode 100644 index 0000000..0b76cb3 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-cli.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-recovery.json new file mode 100644 index 0000000..3ce4eb2 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/pytorch-model-recovery.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/qemu-alpine-ssh.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/qemu-alpine-ssh.json new file mode 100644 index 0000000..14dba05 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/qemu-alpine-ssh.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/query-optimize.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/query-optimize.json new file mode 100644 index 0000000..35c2998 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/query-optimize.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-chess.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-chess.json new file mode 100644 index 0000000..993cf47 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-chess.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-log.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-log.json new file mode 100644 index 0000000..64fa891 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/regex-log.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/schemelike-metacircular-eval.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/schemelike-metacircular-eval.json new file mode 100644 index 0000000..6df4632 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/schemelike-metacircular-eval.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 2220 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/selection.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/selection.json new file mode 100644 index 0000000..e4bf64e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/selection.json @@ -0,0 +1,150 @@ +{ + "schema_version": 1, + "selected_at": "2026-10-12T13:52:35.779627+00:00", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "catalog_sha256": "7fc7d95a2077ede76c4f47890a09de93c2fe558252a122ef830288ad293ceaaa", + "selection_method": "Entire existing 34-task development pool, explicitly requested by the user. No selection by latest score and no outcome-based replacements.", + "prior_selection_file": "../dev12-baseline-20261011/selection.json", + "prior_selection_sha256": "cd51acea6592c01ce9ec8cde34cd46a4965a4164950194f2de7e8ca54649bd29", + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + }, + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12" + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + }, + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf" + }, + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1" + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + }, + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f" + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + }, + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307" + }, + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67" + }, + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + }, + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d" + }, + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17" + }, + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c" + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + }, + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302" + }, + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c" + }, + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1" + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + }, + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc" + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + }, + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78" + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + }, + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6" + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + }, + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684" + }, + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e" + } + ], + "prior_exposure": "All tasks are development tasks with previous outcomes and some reviewed traces/tests. This is not a held-out capability evaluation. No fresh trace or hidden-test content is reviewed until every new agent output is frozen.", + "remaining_unopened_tasks": 55 +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-pipeline-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-pipeline-parallelism.json new file mode 100644 index 0000000..9c8a8f9 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-pipeline-parallelism.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-tensor-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-tensor-parallelism.json new file mode 100644 index 0000000..51e4893 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/torch-tensor-parallelism.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/write-compressor.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/write-compressor.json new file mode 100644 index 0000000..abc580f --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/write-compressor.json @@ -0,0 +1,22 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e" + } + ] +} From f4cdab9f19de2b9f39bf3044e645d53078cf407b Mon Sep 17 00:00:00 2001 From: minixalpha Date: Mon, 12 Oct 2026 08:52:13 -0700 Subject: [PATCH 07/15] test(benchmarks): cache verified apt indexes for common-main preflight --- benchmarks/harbor/README.md | 26 + benchmarks/harbor/README.zh-CN.md | 22 + .../dev34-main-baseline-20261012/README.md | 30 +- .../apt-cache.compose.yaml | 7 + .../apt-cache.conf | 8 + .../experiment-v2.json | 1377 +++++++++++++++++ .../preflight-v1-evidence.json | 135 ++ .../adaptive-rejection-sampler.json | 28 + .../transport-v2/build-cython-ext.json | 28 + .../transport-v2/build-pov-ray.json | 28 + .../transport-v2/caffe-cifar-10.json | 28 + .../transport-v2/cancel-async-tasks.json | 28 + .../transport-v2/circuit-fibsqrt.json | 28 + .../transport-v2/cobol-modernization.json | 28 + .../transport-v2/constraints-scheduling.json | 28 + .../custom-memory-heap-crash.json | 28 + .../transport-v2/db-wal-recovery.json | 28 + .../transport-v2/dna-assembly.json | 28 + .../transport-v2/fix-code-vulnerability.json | 28 + .../transport-v2/git-multibranch.json | 28 + .../transport-v2/install-only.json | 161 ++ .../transport-v2/kv-store-grpc.json | 28 + .../llm-inference-batching-scheduler.json | 28 + .../transport-v2/log-summary-date-ranges.json | 28 + .../transport-v2/merge-diff-arc-agi-task.json | 28 + .../model-extraction-relu-logits.json | 28 + .../transport-v2/mteb-leaderboard.json | 28 + .../multi-source-data-merger.json | 28 + .../transport-v2/nginx-request-logging.json | 28 + .../transport-v2/openssl-selfsigned-cert.json | 28 + .../transport-v2/path-tracing.json | 28 + .../transport-v2/pypi-server.json | 28 + .../transport-v2/pytorch-model-cli.json | 28 + .../transport-v2/pytorch-model-recovery.json | 28 + .../transport-v2/qemu-alpine-ssh.json | 28 + .../transport-v2/query-optimize.json | 28 + .../transport-v2/regex-chess.json | 28 + .../transport-v2/regex-log.json | 28 + .../schemelike-metacircular-eval.json | 28 + .../torch-pipeline-parallelism.json | 28 + .../torch-tensor-parallelism.json | 28 + .../transport-v2/write-compressor.json | 28 + benchmarks/harbor/scripts/apt_index_proxy.py | 245 +++ benchmarks/harbor/scripts/run_comparison.py | 4 + .../harbor/tests/test_apt_index_proxy.py | 162 ++ benchmarks/harbor/tests/test_comparison.py | 19 + 46 files changed, 3145 insertions(+), 3 deletions(-) create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.conf create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v2.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v1-evidence.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/adaptive-rejection-sampler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-cython-ext.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-pov-ray.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/caffe-cifar-10.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cancel-async-tasks.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/circuit-fibsqrt.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cobol-modernization.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/constraints-scheduling.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/custom-memory-heap-crash.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/db-wal-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/dna-assembly.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/fix-code-vulnerability.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/git-multibranch.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/install-only.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/kv-store-grpc.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/llm-inference-batching-scheduler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/log-summary-date-ranges.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/merge-diff-arc-agi-task.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/model-extraction-relu-logits.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/mteb-leaderboard.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/multi-source-data-merger.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/nginx-request-logging.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/openssl-selfsigned-cert.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/path-tracing.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pypi-server.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-cli.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/qemu-alpine-ssh.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/query-optimize.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-chess.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-log.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/schemelike-metacircular-eval.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-pipeline-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-tensor-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/write-compressor.json create mode 100644 benchmarks/harbor/scripts/apt_index_proxy.py create mode 100644 benchmarks/harbor/tests/test_apt_index_proxy.py diff --git a/benchmarks/harbor/README.md b/benchmarks/harbor/README.md index 056f454..fec388a 100644 --- a/benchmarks/harbor/README.md +++ b/benchmarks/harbor/README.md @@ -116,6 +116,32 @@ Keep the manifest lock records with the run evidence. This only warms Docker's image cache: continue using the standard benchmark entry point and installation preflight, and retain any earlier environment-start failures separately. +### Cache immutable APT indexes + +If original APT index downloads repeatedly stall, an optional host proxy can +fetch immutable `/by-hash/SHA256/` indexes using verified parallel ranges: + +```bash +python benchmarks/harbor/scripts/apt_index_proxy.py \ + --bind 172.17.0.1 --port 18742 \ + --cache .cache/nanopy-harbor/apt-indexes --workers 16 +``` + +Choose the host's actual Docker bridge address. Register a read-only APT +configuration mount through Docker's `extra_docker_compose`, with per-host +`Acquire::http::Proxy::` entries pointing to the proxy. The +[34-task registration](configs/dev34-main-baseline-20261012/README.md) contains +an example. Avoid global proxy environment variables: model/API traffic and +non-APT tools must retain their normal network path. + +The proxy accepts only the original public Debian/Ubuntu APT hosts. It validates +exact ranges, object sizes, and whole-object SHA256 before serving an index; +cache hits are rehashed. Mutable release metadata and package downloads are +forwarded uncached. APT still checks signed release metadata and current package +hashes. Original repositories, task images, and test scripts stay unchanged. +Keep the proxy running through preflight and model trials, record the mount and +utility hashes in the experiment plan, and retain earlier setup failures. + ### Direct Harbor invocation For adapter development or comparison with published versions, the lower-level diff --git a/benchmarks/harbor/README.zh-CN.md b/benchmarks/harbor/README.zh-CN.md index 0497dac..6c6f3cb 100644 --- a/benchmarks/harbor/README.zh-CN.md +++ b/benchmarks/harbor/README.zh-CN.md @@ -93,6 +93,28 @@ python benchmarks/harbor/scripts/cache_image.py \ 这个步骤只准备 Docker 镜像缓存；随后仍需使用标准 benchmark 入口并执行安装预检， 之前出现的环境启动失败应单独保留。 +### 缓存不可变的 APT 索引 + +如果原始 APT 索引下载反复停滞，可用可选的宿主机代理，通过校验过的并行分段请求 +获取 `/by-hash/SHA256/` 不可变索引： + +```bash +python benchmarks/harbor/scripts/apt_index_proxy.py \ + --bind 172.17.0.1 --port 18742 \ + --cache .cache/nanopy-harbor/apt-indexes --workers 16 +``` + +地址应使用本机实际的 Docker 网桥地址。通过 Docker 的 `extra_docker_compose` +登记只读 APT 配置挂载，以 `Acquire::http::Proxy::<原始主机名>` 分别指定代理。 +[34 题实验登记](configs/dev34-main-baseline-20261012/README.md) 提供了示例。 +不要设置全局代理环境变量，模型 API 和非 APT 工具应保留正常的网络路径。 + +代理仅允许原始公共 Debian/Ubuntu APT 主机。索引必须通过精确分段、大小和完整 +SHA256 校验后才能提供给客户端；缓存命中也会重新校验哈希。可变的 release 元数据 +和软件包下载直接转发、不由代理缓存。APT 仍校验签名元数据及当前软件包哈希。 +软件源、任务镜像和测试脚本保持原样。预检及模型运行期间需保持代理运行，将挂载 +文件与工具哈希登记进实验计划，并保留之前的安装失败记录。 + ### 直接调用 Harbor 开发 adapter 或对照已发布版本时，仍可使用底层 Harbor 入口，并固定容器内 agent 版本： diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md index 9d17e36..be47d13 100644 --- a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md @@ -13,7 +13,7 @@ outcome and partial test/trace exposure, not a held-out evaluation. ## Frozen inputs -- [Plan](experiment.json): runtime, task revisions, complete task-file hashes, +- [Original plan](experiment.json): runtime, task revisions, complete task-file hashes, prompt/configuration hashes, work budgets, preflight, and queue order. - [Prompt](main-prompt.txt): current main's 385-word headless prompt. - [Original images](image-digests.json): all 34 images were already cached; @@ -39,6 +39,30 @@ total attempts, with the same agent output and original per-attempt limits. ## Run +The active plan is [experiment-v2.json](experiment-v2.json). The original +install-only job ended with three setup timeouts, two uv-download TLS errors, +and one cancellation; 28 tasks never started. No models or official tests ran. +[Preserved evidence](preflight-v1-evidence.json) records the native artifacts +and their hashes. The amendment adds only an APT transport cache, keeping the +runtime source, original sources/images/scripts, budgets, and scoring fixed. + +Start the reusable host proxy before either phase and keep it running: + +```bash +python benchmarks/harbor/scripts/apt_index_proxy.py \ + --bind 172.17.0.1 --port 18742 \ + --cache .cache/nanopy-harbor/apt-indexes --workers 16 +``` + +[APT configuration](apt-cache.conf) routes only the named public APT HTTP +origins through this proxy. The [Compose override](apt-cache.compose.yaml) +mounts it read-only in each fresh container. Mutable release metadata and +packages are not proxy-cached; immutable SHA256 indexes require complete hash +verification. Model/API and other tool traffic retain their normal routes. +Both mount files and the reusable host utility are frozen by hashes in v2. +The recorded real-network probe fetched the original 8,790,396-byte Debian +index in 138.418 seconds cold and 0.008 seconds warm, with identical SHA256. + From the repository root, with credentials in the process environment, set `ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic` and leave `ANTHROPIC_MODEL` unset. Do not put credentials in configuration or published @@ -47,11 +71,11 @@ before sharing. ```bash python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment.json \ + --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v2.json \ --phase preflight python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment.json \ + --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v2.json \ --phase model ``` diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml b/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml new file mode 100644 index 0000000..6b0a28e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml @@ -0,0 +1,7 @@ +services: + main: + volumes: + - type: bind + source: /home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.conf + target: /etc/apt/apt.conf.d/99nanopy-index-cache + read_only: true diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.conf b/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.conf new file mode 100644 index 0000000..1bfac6b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.conf @@ -0,0 +1,8 @@ +// Route only public APT HTTP origins through the verified index cache. +Acquire::http::Proxy::deb.debian.org "http://172.17.0.1:18742/"; +Acquire::http::Proxy::security.debian.org "http://172.17.0.1:18742/"; +Acquire::http::Proxy::archive.debian.org "http://172.17.0.1:18742/"; +Acquire::http::Proxy::archive.ubuntu.com "http://172.17.0.1:18742/"; +Acquire::http::Proxy::security.ubuntu.com "http://172.17.0.1:18742/"; +Acquire::http::Proxy::ports.ubuntu.com "http://172.17.0.1:18742/"; +Acquire::http::Proxy::old-releases.ubuntu.com "http://172.17.0.1:18742/"; diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v2.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v2.json new file mode 100644 index 0000000..99c2761 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v2.json @@ -0,0 +1,1377 @@ +{ + "schema_version": 1, + "registered_at": "2026-10-12T15:49:06.643054+00:00", + "experiment": "dev34-main-baseline-20261012", + "selection_file": "selection.json", + "selection_sha256": "4572b1cc432a16b9c5d1dd554c8d2189ae80414f386a90bf03829bc4d0c3da85", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "arms": { + "main": { + "source_root": "jobs/dev34-baseline-main-source", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "prompt_file": "main-prompt.txt", + "prompt_sha256": "37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d", + "prompt_words": 385 + } + }, + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/adaptive-rejection-sampler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "95408217b9c484ffc18a02a91b7706323b6902919c72b24866a32e27a23543e2", + "environment/Dockerfile": "e644a3343e339303230f1441b7383f076d21c69a5a7aa6430b4f8a9d27f15c8b", + "environment/protected.tar.gz.enc": "7fc9bf1347bb97da8dede04b2c58638f225070913e222d76627068aebd1b1df2", + "instruction.md": "e6d28c28acd76bef61817427c74c2a29e3e8d75f535cbd708afcbfd02c54013f", + "solution/solve.sh": "fe443a8082b5183ba1623311ba0ee958888aefa633eff93766b9ae14e9bdd84d", + "task.toml": "6ed7b37f74beb46e9d47b0f5fad53025667c9548baff4e96d467b56d429fab1a", + "tests/test.sh": "38b43560d173cc2b952c3a3e17b8a480216d84e33450515e047bcb0d806b1e0a", + "tests/test_outputs.py": "547dc6e107f034f41703722aeceb6d0236e3fb69116fc3f2fbaba11884de352f" + } + }, + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/build-cython-ext:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d5f5113941c3963292bae396baa9369571026f46e59437f98ddcc93475dd7729", + "environment/Dockerfile": "cb6cf421cb9b69d43bd25cd491e49bda982fbb865d47515178d017eda727194e", + "instruction.md": "7e52874e45f88505bd21540229719de6639a401477557d814489981d472f0e34", + "solution/solve.sh": "41bbdde88617070366a70e518eedca466a4ea1acd7e96899b2fb7da988e40abc", + "task.toml": "c472f0e7522c128fb7f03e00b50c8cdf2e5e6474daa1e094e7441f4b54e770b2", + "tests/test.sh": "0156c9c4085258b696ce424cf1ce96cba44d1263c5d73bdf54083fd80237f915", + "tests/test_outputs.py": "6549f1a8e04b6163ec08877e6f6b9868bf94563f3a3c72ac1fba9c512c31689e" + } + }, + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cancel-async-tasks:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9adf86b1279cc44f83fe16e6d340595aa992240f1d87243d204cae32e453693b", + "environment/Dockerfile": "1ee7bdb4a7536f5f114c11a3ac6ff42790abfea901fcba6b2a8ea8685f19140e", + "instruction.md": "23d22f7039b50682941652c716f96dac314b8585c32c133614cac77dc067f47e", + "solution/solve.sh": "4938ca4ad5b7dcd36b414b42a1abb10c058ecefce7462b678f9bd05c9fae1844", + "task.toml": "14db0675e04408f2133b0067e8dbddf66a1d6e7baf6b8d38bef8ecccfc4e58a8", + "tests/test.py": "f533c0a64db068f582b4733c5b81f2e66b8fa52fcd3efa65752e6bd6d24780aa", + "tests/test.sh": "79600a6b863def31b4daa008297423010cd2d6243e689d67503e118e4dc53397", + "tests/test_outputs.py": "d6f510b51fafc81b4e9b5cc95535aa13491c8753e0e4171db42c45f4de460ee7" + } + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cobol-modernization:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "c89318cf59efcc7040e3d946022e2616161b6733931a080cdeea7b1918e7ecc4", + "environment/Dockerfile": "8b175c83fc03ec7d16d6101516bc5100230b592f480e435279f4fe051c9280b2", + "environment/data/ACCOUNTS.DAT": "46f63eb29442ebc6716bcd8a0c36f4367910f2783f6b84a307ed9a61ccd1046e", + "environment/data/BOOKS.DAT": "db04da646207cd87f8e9c5b6f0b57d95953899f3eef22de1a81423f06ef4a9fd", + "environment/data/TRANSACTIONS.DAT": "1fa4cc5420ded94d585c563e8196b85166ebbae2bb8924b20014717fb2fb9ee3", + "environment/src/INPUT.DAT": "52fc984f0cf14af1495285d7efe02b9bae13159eea59255560ee938c4f78afee", + "environment/src/program.cbl": "27cb0e56f65f52fa75f346720b37a180b8e034ebe7d7622b979f195087bd6ebb", + "instruction.md": "a87074f1141b7345394cd88053d0debb0dc8f7de95308882c6538ee9a3b359da", + "solution/solve.sh": "7d4a0504e1686c1790bfbc074d4e598f4c631f54c2bc5f3d5fd786574b4810b9", + "task.toml": "4b620211dc3d6b1b989f0210c5ecd2f606974581c8bce27bd78659d48cf0cfbd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "009180a7092583fc8143bb3ffc3957923aa0d61d0aac3a471f76eeb4aae54a88" + } + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "file-operations", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + } + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/fix-code-vulnerability:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "87f28dbad183c3a634887696d328ca68a8a0858f132bc48cb7341c22267d38f7", + "environment/Dockerfile": "0917470986224df4daf4854a71302ae81c73ba5dd6e25be1232d29b758a5b3ab", + "instruction.md": "89b1abdf0af19399f720233b5cdf6b2e8e0ff7b6e3fe3e2a3af0d4d298a8641a", + "solution/solve.sh": "7a0131b369d83ae8546ecc3f9b4e23d38240f1bd1f0941e4430fa85b5182ee03", + "task.toml": "9aa2bd14ce9408289d199d0608bc3fa65b8b5e23374c9641020de62197dbf1e2", + "tests/test.sh": "f1611da693cb4cfc5e8482698c6a1858a5c67220ae9db89b50aed66584e9b17f", + "tests/test_outputs.py": "c006a5de95b69bd78055aa26cc055734dbd50718ede25d331f767bfee4ef7f26" + } + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/git-multibranch:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3b9473429381d9eacabab498cb4a8915aa2bca24fa857ef862e5991d3f81f113", + "environment/Dockerfile": "88d9a9f4fef1ae7fddf632799dedf5a2fb6489460a93b610c33dd8b122982821", + "environment/default.conf": "a3a4c5d0ac9cfe79b0510e4225c80fa3988c27fbac13622505886bf66449e04a", + "instruction.md": "54014a38d7526d65faeb6c11d9b7ed3d4a5cb5d973d68d3da3b68b8c0289e125", + "solution/solve.sh": "300ca96b79db8d77f14742109aaf76048aafc4644ced750b1bc5c2761e8d0175", + "task.toml": "4b9981604578a31205257cca36a28d0c89c9d8d1058541330acd11d3f8d65f82", + "tests/test.sh": "ddbde3eea6ffa8fda105cbd86597a53bd409f41e13223900aa27e77a84b6b155", + "tests/test_outputs.py": "f03c6867663b8e8fb7899df51ee4df26411fab62ed491753964b801e591d16c5" + } + }, + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/kv-store-grpc:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "51fdc2ad06f0adb1270164f247ea41799b2ee2ad15490e48ddbf5e79d3767aaa", + "environment/Dockerfile": "47c9982789092437f950736110b2bd18899c39b8fd20f3af3c95b848e564a197", + "instruction.md": "f4b3facc199d5b1c78db6e603ffa29ec1b93a77ff7d291e1a92b42e6bf582bed", + "solution/solve.sh": "6bee2938ac9731d1156b0ca02b28c3d5793bf2cbe5f995c863d397093af01cb7", + "task.toml": "6ff0f30a526bfff86a67b0b53439b9965229c0f368167dc1a5d05f34d82299a3", + "tests/test.sh": "c348467a076bf614d0a850e25b2263f85bbf7b4c7a8d344fbd5454a0611a0df8", + "tests/test_outputs.py": "115de2e964bbe689d07930527113422ef8a9e4be4eb958ad84594e3fcd2b1b0e" + } + }, + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/log-summary-date-ranges:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "026000415cb577f3d3d5132902670262f7d5d8f69841903ec06a882292f756b4", + "environment/Dockerfile": "b479f6e987b63a6ec118fa1691b8bb85c437b2a6a0fe63feab5ed7154957fdfc", + "environment/log_generator_deterministic.py": "beb6eff59979e9aa0be63ce8a0f1b0df1178f92871f04ba9f76fa70cf8beb81c", + "instruction.md": "c1d4516ad4ec2238209740eaf01291df9ab48b0028a3fc5df383c6f831453484", + "solution/solve.sh": "9f6a8ee8947bdf1c0f31b530b6194ae998d66841a3b785fa2e246ee840350ab6", + "task.toml": "9a9593673d6805ef3a301ea734e9c4dbaf80da1b37c0fe9d1f2ceea2a8c57e18", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2fc06a6ce1944ad82f84fb437395fa75ed8c0eadd385b92e375c27e5bc9cb2b4" + } + }, + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/merge-diff-arc-agi-task:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d6b48ed0890b12eb68f72c12cd02689bd6bfae2191dd2688949cde6fe77f6a94", + "environment/Dockerfile": "4a03ae6f0c60b9c2f45c33f0413b5aa75c9811870254e3ff1968fd0119173fc6", + "environment/task-deps/bundle1.bundle": "a759f9b6ffc1a2eee50778d72a1899ad7d724becab84fa30e7aac71ee9e443a5", + "environment/task-deps/bundle2.bundle": "75e3e3916a01c980a03715de67b16c729e03661d94a694b5f4223d229bd6593a", + "environment/task-deps/examples.json": "5426569e4ea078bbb74039ec712a2d4d35f1f9fa3a5f26bf2a0de4a2c8ef5db8", + "instruction.md": "2d826479873e5ac1c94c1f802763bf85c22bb04624384685ebdf6b153fe0278e", + "solution/solve.sh": "48d4becc41fda5b6063fe96947474ffe778a29d6924d7b2bb235eb61e5ce555d", + "task.toml": "6ced40e06b43b169857509765eb7f07791db558f9437cc7367d2536771c2ca5b", + "tests/test.json": "20f116a6b718bbf4874bbe203d15a5c0f85a1d88cb622e2d7529c7e4fb2d4c28", + "tests/test.sh": "c9052954cd6ce25c4a375a2fb48e765b1efa5b39f7991b93993cc793af8ab9d2", + "tests/test_outputs.py": "2754ba6c5c5f75c9db5ed2c5729d9f5d7e21cb7fdb18aa0c65abdc3e7e85c149" + } + }, + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/model-extraction-relu-logits:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "mathematics", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9465accba6c57e5de9a196eed776a16eeb0852a8afc18fc3f8979e2de71b0c7b", + "environment/Dockerfile": "598b1f80d3b7194e97cbf02c8d245de30e303909e0cbe6465125d1d3179fddb5", + "environment/forward.py": "af1a23efd8e4390d16a29777c28116b2f8623ae3f7b628fe14db9f94dac26f5b", + "instruction.md": "c9e179dd099065a34c362a865b0eb9fab5f7d96be582cf117e7519580c036ebe", + "solution/solve.sh": "4cea5094f074ff21453d4423aa1b415a104ed90caa55a4acbb4f09941a3994ea", + "task.toml": "eb64dfaf8ae88868911ae28855cdc4f767de464001e2704919015b595be6dc29", + "tests/test.sh": "3b2d8f5e0353c2bab3515d6387468a863a4f9f241ee131c73f545f3fdf9e2d52", + "tests/test_outputs.py": "e5189c7eb9942ede55e1b48a80388590b80a486b75497613f257b3373ad9b2c9" + } + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/multi-source-data-merger:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9bf37162a4c1b88ed93627830bb8512ebae154325d7cb069c883144a0e4c69d6", + "environment/Dockerfile": "c95a40d3d35b473dc15ae1e8784ba7fb1725805c0583fc574dd2764325d75e4f", + "environment/data/source_a/users.json": "75e51e2d3ab0bdba2269d7cf3757576f84f3ee2c13407d2db94289a5958a1778", + "environment/data/source_b/users.csv": "8e87d16dc47d166edd8e56165955bad265b630d53d02762367c816affb5fa760", + "environment/data/source_c/users.parquet": "23d1ea51c4170bdba53febb885a0f495e28de525a52667347b2932136b9a72fe", + "instruction.md": "d1d28bb6e37f7a255a1b7f78aae8591f35c0494f33e4913392424ad954aab944", + "solution/solve.sh": "1b3d0d50e31930d042495d48f4c8e95f1a9c1a979c803b7f7cec651322537e1e", + "task.toml": "ee3964c0ee040f0a0cb121cefefbc15443d127684acb003fa5b61dd42f31c694", + "tests/test.sh": "d5a00e64413c0e1ae452f38a1aa1a47c3ed7b4416bbfccae1a4c053025a336d1", + "tests/test_outputs.py": "a070888930cf829aeca039771260b1386bda990ea805f313468d381ba3608ca5" + } + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/nginx-request-logging:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d79997586d1c14ed8dbfea050e3580b25500e04c8345acfd3f294152433c795a", + "environment/Dockerfile": "43003466d89ab83484a3fa618e35b58cdb74a002b1e8716ef30f12b3ac8302d8", + "instruction.md": "26a7ac98aced6107f147206790ada77bbf8ed3c25fb360cf323d5fc6889edf99", + "solution/solve.sh": "2454bbbf7c201a9a778165f286373ba2cb991a0eb1d026a8005d57ae71250dd9", + "task.toml": "b7a78d00be13656f75069197100b7b8c4fd268decf6f78929c853b3222a2729d", + "tests/test.sh": "27761195f53cbfe5941fe39b573631f6c926869e64b7c43220377e1e116528e7", + "tests/test_outputs.py": "045cc716c14efde3b0dcff5fc7c85ec5d18bfc6ce66f8b40a418fa2a3a4acda0" + } + }, + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/openssl-selfsigned-cert:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "5dfa88651890cda06fa2a25babd3b813818767635e88bbabecc8305d5bb3942f", + "environment/Dockerfile": "f590b523038be43afcaf5291d7b9e8ed1333c48d9994e77b8dee2f2c352f0f40", + "instruction.md": "38054a3371f7c2c853478a2f9af0bff858437d5b28a9a07483238f06e566921f", + "solution/solve.sh": "58cf3c844e907745cd81fb4bd75efb3d080e2f0e82b589c56c954ae6e1756510", + "task.toml": "75c8913f0fa15890786c11999cde920b8bfc10d4ce8a14edf1e76985a2ac5800", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "90970bb3c372e8e7fb6bb9ac5f649ba73eb38e1a5ea2e922a9057a72b6b4192d" + } + }, + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pypi-server:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "906a096a2cc7b9afc738ff5ad5fbd605be65c103f5069c6673915223b9b187e3", + "environment/Dockerfile": "17a4e67580e4f2c6a151d04b9d74eb020666e7eec8e5259c4a5411f9e93858d0", + "instruction.md": "1c537e4a311a8e71155e121f57bfc03f742fbfb99d05314cd03e433996a28a31", + "solution/solve.sh": "1143c0eade4932ec3aeaa5013fb5e23bbcccca15dead294aa03b6af88750775a", + "task.toml": "07c39d7a733c89adadf74bb63a3e906c3694feb6c490d5fc85ac187c003fd447", + "tests/test.sh": "d2b1220e42bcfdcce87907ac5bbd69ac210a3c94926545ca2439636786e7784e", + "tests/test_outputs.py": "82c6351722e9aa984ff1d701bf57df3150ce51466df2b70afa1aa62d2650be4c" + } + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-cli:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "35a093fa5f1592737784ba718c2e3e9ae17ff8b5e53e003fae560ddce091039c", + "environment/Dockerfile": "8fd5c0f21a6914cd440d10c84c9db06fec606f20dc3e0363b17a3b8c220dc4d9", + "environment/task-deps/cJSON.c": "0e7995261640c96db16ed2c91a5d981efa917cf5e72d0f97ec45a1179ae3ad73", + "environment/task-deps/cJSON.h": "0909160541bcc510e0b2d8c7a9cc051fa1459a0f13be4d8e56346a15491f2f2f", + "environment/task-deps/image.png": "f4d5dfefe463939996541404340e7099d5957b56e1df3760d13b5d2de754db53", + "environment/task-deps/lodepng.cpp": "67873c5f6272f779a143a9c81070e8783327c5552d0b9de2aaee2cb779308917", + "environment/task-deps/lodepng.h": "b735b7e0357319973e9488d520d7f270d734a6e9e1bf924fb1d5b8d26d7e97ce", + "environment/task-deps/model.pth": "a32103eef09936e586b1a7dc7e64d279500a38ff30fdd6df2167e28eb954eee6", + "environment/task-deps/model.py": "e4d3a5c728449656d0efde7871ee227619348352769795ed1be1a8b7f835fddc", + "instruction.md": "ce36ee1f6c589f81ee6a6be4967e25c76a2ed697006928023f42dfb9f28a6750", + "solution/solve.sh": "3b34e561c9ef0a53c9948655847846eb4811aaac67b990480aa6aac1b04bfe66", + "task.toml": "fc1c018efbd5d0537ac0741d53d1ebad3886faa51d14cbde8ec953856411bc0d", + "tests/test.sh": "43fb67e537c7adc2df11c2367f47c876cb5d16a7c1ae16d0339c3a3364bb2b06", + "tests/test_outputs.py": "370702c57a4e0c85f70ee37a5c357ae3a5df8480a275bb6263b0376f2f6c093b" + } + }, + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-recovery:20260430", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e080308a05703b40edd16f53aea41a0e6b9874750ecee35bb432029d0d44bf53", + "environment/Dockerfile": "8a6b7edbb9e7c241e447f15a71e13e5d043b25169be2fd9802b84f5be3d56eb6", + "environment/dataset.pt": "6db65e4ec1c0760209c2dd997ef582e4a1519ab38e0312339999c460a696960d", + "environment/weights.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080", + "instruction.md": "2bef07dc28be3ab2c35ae389584034c2e4dde92c2c31b3eb6bd829628c8c3f4a", + "solution/solve.sh": "175a7802777b43e0fa4f629d71f7c68fb40706813c342aaddc72108d89bd5b2c", + "task.toml": "851a51204f53482a31f47cdf8833128cf421113b0331011286940fc857c37dfd", + "tests/test.sh": "6005f3a75ccfe00d7bed1005264b4a84a1f8e30cd780f8a97b3a804be7cc7a8e", + "tests/test_outputs.py": "5d4d57c43e6913bf3d0081da764ad36d1b7275e3bb114a05abda529afe563905", + "tests/weights_gtruth.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080" + } + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/qemu-alpine-ssh:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "qemu-alpine-ssh-v1", + "task_file_sha256": { + "README.md": "72f1c83fa1e3e6381ef85dd82fd109ee18cdeac090311cf3212a0a8cff0a32a5", + "environment/Dockerfile": "015ce0601ff54758f66b33f4952e0d3a577d84c09aed8f3ec5945a570e8d168a", + "instruction.md": "8348c60490cb90bc690cba259dfe98996da4c3294c6d42bba1119b00c64de4de", + "solution/solve.sh": "1cd31186c427b91bb00499d22aa4f0214fc1ca93b40b6e4cfa969611b848db2a", + "task.toml": "10a43ec6ee732eeba852c377f12e23178f7fa5a4868143f3699a15565ffb64c7", + "tests/test.sh": "3b244479776c4ee3639631112799f2cde22cbb80ac10511715e3a8f32909c289", + "tests/test_outputs.py": "83ccb4bdcd259b342212bebefe395f289858b4dc55b2ee7ae0dab238ad2e9718" + } + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/query-optimize:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a82c53e333a0ac0b0b410b5e26cb8e7305dece31db84d340248ffb13d5a45160", + "environment/Dockerfile": "064818771f200e0517d22068f87229a7a95d0a82b0778c84ca7203bf1926a0b8", + "environment/task-deps/my-sql-query.sql": "43382792f0f438625d643e34abe144febd5669518f4ab1ab550a15c6d9314a21", + "instruction.md": "473e12d320d2732f679fb631c2a4dab4a01deea6775686718ae0c417b8a39447", + "solution/solve.sh": "0d95f3cedb287ff9d47c796cbaf812c300d1222c10fa80c49e8acb0e81eb6978", + "task.toml": "6e88f2a86df699594bcb1bf81df4ed694505f79c9420f671335b02524bbe3868", + "tests/golden.sql": "4da1885d56b3ffa7b7b22b422996d018795611f6963499d551582881b74b84fd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2068ae80c57af8d52a7ae510f0666732c057806e556b480970f36f14bae683f6" + } + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/regex-log:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "29ffd98c55dbb6f4844f1b9d8c1769b2155c117ee475e0df6bbc5ef330d239b3", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "4f7ac05e70cf9220ea0f1e5a052c5f908cd0fa884e847d80b0bd51bae2e96f9c", + "solution/solve.sh": "7e670d4f2b4bccb1e4db38f2a173e085ceda028c38167912b466b0a84fcc0999", + "task.toml": "6bc98525852755caf1fe34c174c2f10f4328adb4ce6c1968dcafe1c877e4e160", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "345c3bd09ab6f6fe8c8361a58c0a47bf0a13b3fcb38a5ac7824e44ff855e8f72" + } + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-pipeline-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "297ddea07c230dfd246a8ab07337dd7abcd8e5e2a2b2e26590cb0a8af779839c", + "environment/Dockerfile": "c357ea41e4a4013c5c5980b8fef04af6a5ae250ccc6697cf3bad3127401a931f", + "instruction.md": "989bfcf0c54baf6c8cdc391743dbb2f9e858c5779dd85112536875d4d1b2dcd2", + "solution/solve.sh": "787198e4178d88b887697aefdd29819f29435ffdfe356545d92f58a30fb38b70", + "task.toml": "8b4f9f525c08afbbc4eca9168e46238a61ccb903486773e9ca4da64bb7ffe857", + "tests/test.sh": "cc9fd28b6590993fc37e4a99da771cbf04726c8c62fe308e802612241da69dea", + "tests/test_outputs.py": "23574246a407163a8e6581c84287cc2f1eb745efbaad2a317a5a28c39958c64f" + } + }, + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-tensor-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "4c1b3c3be30548f1f94c6f5503b1bc74b006af7f192dc4cd026f411df1a54604", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "13bde2ab165d72a311e2bf066d5d6c3c8761bf9d3dd39d1b23eda23bc546b910", + "solution/solve.sh": "d083b7cbf9dbb6cc2ff38c8f2acc150f0059c04aea42d6c2183aeaecc5a19385", + "task.toml": "72fd2a5500d1fe8795dcae76cccb1a29fb25894881a7475f2fa519c87d425a8d", + "tests/test.sh": "a7e6c9408b2401e4e7448371f7ddc72f3284caccfb56277d3cd4e98051868cdf", + "tests/test_outputs.py": "4ae4285e15fba1fcf13d7b7c0638ac8c2e2b0797c937abab1fb6b136fd508290" + } + }, + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/write-compressor:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "603015a28500a28f0838fea2c43a5168613746bfd07737920f4bae40b5a326f7", + "environment/Dockerfile": "9a583d29d46818f6ff2be7c6a0879d03343106e5376fd5a48eb5926b402bb93b", + "environment/data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "environment/decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "environment/main.rs": "fa53d6572fca95903e30d2d953f1e428bb8b63513ff218ebee9bc60a425ab389", + "instruction.md": "c4fc3e73e44deeac95f1cd4a34a2d87e853b22ceec2b2824767f5f087b470bb2", + "solution/solve.sh": "01550b7de7ab1320b1898eae055226bb7c73bee259c75102d2240c974a730409", + "task.toml": "6066a509066433c97c417fea68c4e3e73996f698dcdcbabe07718835eea355a6", + "tests/original-data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "tests/original-decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "809f486053e3ffbce852111dee1f63d80ab31297bd33172389e50278bed384e2" + } + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "native_agent_timeout_seconds": 1200, + "work_budget_seconds": 1020, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/constraints-scheduling:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "personal-assistant", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d54a690970149e1216bd4fc066b45069c7948c13e3b1ef2a213a74342805f209", + "environment/Dockerfile": "7d9aba072a4a907229728b9739c8cf821fc2c98450babd64452682caf3759c84", + "environment/inputs/alice_calendar.ics": "5a5d45804bffd339c2479d49e76fe817b38244d14d389afe1de583443808ecbc", + "environment/inputs/bob_calendar.ics": "c97e85dbcdd756758f8a80f4db5fd5c1dcacee79298e4d2262f46de5ff680b50", + "environment/inputs/carol_calendar.ics": "32a0f8687637df0a12b8e5c56d49c4192bc3e790856943632d70f237c7c1363b", + "instruction.md": "342def78177f9bc41bea4d11af679bc385a9dda2ac8a1e425be384a5b7f937b6", + "solution/solve.sh": "802d81ec064fad3050bbade95a2f847ac24e0b7f80fa69b0bcee0347dc92fe77", + "task.toml": "7b9842de16d74e8faf8c7c0aa03b0b81145941cb12dd322d81ae23fc48607250", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "1422b345ba3aacdfc4184e962811b6379988435896c50d8cf62903e70990c5d1" + } + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/custom-memory-heap-crash:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a06ed23a2c467aa36d0d4a88e5c020b1076cc25b8d483517239658e9c596dde9", + "environment/Dockerfile": "6ee81e5a4fa2540443d7f27d577ba840728fc6aab9f301ab7919103673c78080", + "environment/patches/locale_classes.tcc.patch": "16951f4a4df6bf253886ac75f5a39fdbc328e415194493f63cb7589a364aa817", + "environment/patches/locale_init.cc.patch": "e97f133ceebd86f8881f972e0e45b0ec0a13a9bfa428d8032008707683d54292", + "environment/program/main.cpp": "b94538c76887cc2b2ab739b5ac4731cddd0faa8f4a9892a6265c062bbbeede14", + "environment/program/user.cpp": "92b203786fed1bf0b3aa1675c9cd2d82b2651170675064da8ffee73b98ef14f9", + "environment/program/user.h": "f43d155995a56c22b9cd1417daef7c3b5a976cdc662a98eef0feb438ae1eee77", + "instruction.md": "53c68ae14a3c3af9a97c40bfd9bdc30a5d98b37eaa2f69f763475745f70f108d", + "solution/solve.sh": "b0b1d10bea1cd5be14a82b35d522df616df1e67028f90d2a6bef90aaa98511a1", + "task.toml": "ed474be1983e212913aa8866ebb66216acbe36aef2339ac375d232aa46617481", + "tests/test.sh": "70a2b55f0c5757f6b123a216dfd202be4a646680bced28dddcd31fd8959a058f", + "tests/test_outputs.py": "4e1cb33f777fa7fb5e1536f9bea9eace6deccd515c67d4f87570adbe692914ba" + } + }, + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/dna-assembly:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e7855c23d9822f688eec4dd89ca0f1caf91c6a69e1c8f3c9eb544b4ab2a94821", + "environment/Dockerfile": "26e9e43a98f0b1d132bb638f44b7dc882b0d0ca32d56fa33048adeee00ab600d", + "environment/sequences.fasta": "ebdc361b5fba28eb0a2c7a8a8a444ccb12cc842c8cbc6df5a2c24254f130736f", + "instruction.md": "88998c98beabdd966ab46c04ede822bbc74796696aa1bf87c567a9131210533c", + "solution/solve.sh": "c976cb118c8ef1befd846ba05fd5cadb1b1f529bc76b9f2ea9cb3049e6e11a47", + "task.toml": "7b571544b64d9be1c91526837e5d805c2181afe0e6fd57a7f0676bba36c868d7", + "tests/test.sh": "271c8317c98539e3b49233d017fe51fb0e5fba8114e76830fe698565499b2b1d", + "tests/test_outputs.py": "5c2d0fd2e5f6303263e9750f67acda7db491ef04841832b4fc62bcfb61b08924" + } + }, + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/llm-inference-batching-scheduler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ecc9dd635357c184742cf72fc1e452a8946e207b183fc9a7eebbcf07133b911f", + "environment/Dockerfile": "10a6193904ee0022ac098504182989893b196dfc74e626a3ca48534a8eacde48", + "environment/task_file/input_data/requests_bucket_1.jsonl": "8b3cf1e2b06074162038064dd2c114abe308acd32cec7d365ee63ee81978f7de", + "environment/task_file/input_data/requests_bucket_2.jsonl": "e15e2d4c4d55f875c42a6042610d6282f6c0657c39d3e0d0f087268252525cb5", + "environment/task_file/scripts/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "environment/task_file/scripts/baseline_packer.py": "547d230d5e6d93197803480cb81cab0ec6ac63fcfa0a24b246f549523f216c4e", + "environment/task_file/scripts/cost_model.py": "0cf417fd5ff0112defcd46970d37a290c0763c5a9092e91cb3ebb86c3f20e6ce", + "instruction.md": "0817374767533fbf51bb93345ee1bba266263c216c6cc55c096e71b4a1d37dfc", + "solution/solve.sh": "cb9efb586d76a893c3027e62029e286eb6c7583dc92dbf1840a3adc236a926d7", + "task.toml": "52aaac942f917e6a91ff658e0272195bd1cd842d41d71e302414c52603cc588b", + "tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "tests/cost_model_for_tests.py": "d3e810a99d40327fb8317f6512d879c52c6aeebdf5dcdc0c5e117c7df61c9ab9", + "tests/test.sh": "005f980332355e2fe143d4ff4aac11828adf070594bbe7055399c238f06953ca", + "tests/test_outputs.py": "80b06b7c4ba0db28832b767d48be74bb462d7f5df5798d9cb58c4f1160d96ffa" + } + }, + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/path-tracing:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3e4fc952b3f9a860c7ee7837b3f3c686f1283ee7821b014265dba7c94d7fc849", + "environment/Dockerfile": "7aca02e4e20b04ed2269d627a29dd4e7337a02764a9652b641452bf09836a3d5", + "environment/orig.c": "665c1bde0907740890ac114d6b36af1f3fd00449fa287583817f778bb5da200c", + "instruction.md": "9f36e02d636f5dc0946619bc518f2e51c24fa75251da9d1f0ca9eca612cb9572", + "solution/solve.sh": "d107abde101eecee6838da80cc52a3d480c93956fff3a9343a299ea01b3fe9e6", + "task.toml": "cf47013f69bf1b43242da984f231d4daf364fdf44d3845f2eea2ca2cb9308c46", + "tests/test.sh": "a719dbb1ba7ae76df095e9d815941dfa4c55edaf11f173b9c3c3097afa838548", + "tests/test_outputs.py": "572e68b2843191d29592faf42a405861b7958d8e894868b74c5a8886b53eb879" + } + }, + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6", + "native_agent_timeout_seconds": 2400, + "work_budget_seconds": 2220, + "verifier_timeout_seconds": 2400.0, + "docker_image": "alexgshaw/schemelike-metacircular-eval:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "095d3ec6dcc0fe0c7a7e92355456dcea6992f91d1f8d397669b8cfe18475f506", + "environment/Dockerfile": "9b3df76222e16aad360fbd34cb0f71a0be8986052c065b10b92c0cdf417b25c7", + "environment/tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "environment/tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "environment/tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "environment/tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "environment/tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "environment/tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "environment/tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "environment/tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "environment/tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "environment/tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "environment/tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "environment/tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "environment/tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "environment/tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "environment/tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "environment/tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "environment/tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "environment/tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "environment/tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "environment/tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "environment/tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "environment/tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "environment/tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "environment/tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "environment/tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "environment/tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "environment/tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "environment/tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "environment/tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "environment/tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "environment/tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "environment/tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "environment/tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "environment/tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "environment/tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "environment/tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "environment/tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "environment/tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "environment/tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "environment/tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "environment/tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "environment/tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "environment/tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "environment/tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "environment/tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "environment/tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "environment/tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "environment/tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "environment/tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "environment/tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "environment/tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "environment/tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "environment/tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "environment/tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "environment/tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "environment/tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "environment/tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "environment/tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "environment/tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "environment/tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "environment/tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "environment/tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "environment/tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "environment/tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "environment/tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "environment/tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed", + "instruction.md": "3af5f9567b62f93683db7518ec6fcb609f83796258aa02166b5a7091c5b69031", + "solution/solve.sh": "e2c5b0ac2bef4801a1891e13c108813381877426c1a7f01bab86b4fc1f712510", + "task.toml": "8dae8eb226183b0ca92331c1106152a7abc93b03b4e5a95291b86a4ddc9b44e0", + "tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed" + } + }, + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/caffe-cifar-10:20260403", + "resources": { + "cpus": 4, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e918e6852c5c034422b81a6be22d7596fddcee53abe088f56727b2482da32728", + "environment/Dockerfile": "b3b8adc915a1afa035b604273af3bc1aa8b90999d2dfd72763218b34cb87abcd", + "instruction.md": "46907b0ee048f74269c29a15e0e086546b2b1220149597017da8b5ccc05a0cea", + "solution/solve.sh": "771887de1ed557269c7f3160c777519aa9538e1e9b59771b2f6ef38c8952490c", + "task.toml": "2831165ad9d9e8bbc646e6d85c0a2290640dcc4f13749547a2937c7675012a8b", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "e99b9f6284e923388fbc21488b1af99fdde46d4a809d3557b0788dbf1cb6b281" + } + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/circuit-fibsqrt:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "201f7c889dd6f73dd83ca078e347272b33720fa622f8c757ea6d5770453f4cbd", + "environment/Dockerfile": "780c23986778f3e5f842567868af30b1135ed095ca9449e1c90b7f0e0483d3d4", + "environment/gates.txt": "032786f94f16605767c62bb61ce4dbba7ab7d3f1046712c975f01c9a04b6ad29", + "environment/tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "environment/tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438", + "instruction.md": "a7f500266a878d04ac9de39fba76220cfad216d8f6b8063bba93bebb3d4ab9ea", + "solution/solve.sh": "a4ffab4183815a0eb7646abc1de1fc67695014c060d7ff16e8bca9ec94385d0b", + "task.toml": "9f5aa1f7ebbbcec90fd109121e9bd509af3cfff6b4cec701bd91947988f40972", + "tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438" + } + }, + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/mteb-leaderboard:20260430", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ed147a5bc31e9865b16848b2f01d2f9b1e677af57a7c632d7ea2311c0553ca76", + "environment/Dockerfile": "271fdb5b34520c0bf2cb6462365363ec52058f127cd9eea3f5da2589df5c1044", + "instruction.md": "06ea44106eca0405c25bab00f5b3cfeb611c872093f6c0f82ed057839d1d13de", + "solution/solve.sh": "4ed68831d4acf5d80d2d73eec1f21efd0896a6f51f23020f572cfc0af6da0410", + "task.toml": "86574bd1c6c4fefd2e64eee9c2caa53d87c4bcf2e51547912440a0ddbd0badb8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "b28f8fc8b7614801305813c402be52f4be1e75de6658ecdfdf9383aa793892cf" + } + }, + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/regex-chess:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9296cbee592fc996c58d228b51fee303647e5ecfae82d5666d42a40cb30ec355", + "environment/Dockerfile": "2616cd775252303839dfda0ee2f047eb862952b053b0103bd6b2460ef90c11b7", + "environment/check.py": "f9c416ae9ea01dcdd5f6a43170c8434b8c0a0b389a4a30876a2048e277d191c7", + "instruction.md": "a6ed3fcc54fc92d71a84af921942422d299ca89b410dcd92cd05a53d0d06436c", + "solution/solve.sh": "b728bf0ea392a3d244afbd0818d36933c8d3e190e47743c1382cd518d518a2e9", + "task.toml": "07fe52636fcc4bc9fff25d5630af145b97a5331219d719cc8decb4ca67ea21d4", + "tests/test.sh": "8f7f00a1c4dde7fcca6c918c768f2aba432b8c4007c15989716c68091950da87", + "tests/test_outputs.py": "4983c44d21c56dd8191381eff0304afc8b87bb714527ea9036b3319c015a30b4" + } + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "native_agent_timeout_seconds": 12000, + "work_budget_seconds": 11820, + "verifier_timeout_seconds": 12000.0, + "docker_image": "alexgshaw/build-pov-ray:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "38380f64bc631916e264f80ce4d37feedbc783310001475b367d03989d114fdf", + "environment/Dockerfile": "44f2b5fc7311d8f336daaaca99dd4216b49355e67f50b494ebacf1b0f7c14e92", + "environment/deps/illum1.pov": "5a1f7f53ee0730c4923fcd21cd4a9c34719024d8557920441551ba7d96e75dc4", + "instruction.md": "56c8ecdd0084a07f8c97986c43af5c37ed03dd5aa994da6735cf59806162d1b2", + "solution/solve.sh": "3da9dbf589656512cd7a7004dc3752d3552a6af939cfe59eff79f91695fa42c7", + "task.toml": "b90a47ec26c1191d3c523cc094ea96b170e42fbfe176ef1f814c57d05390c097", + "tests/reference_illum1.png": "bb084f430bbc9d5b623a656920670dc06cee6d9132a9d550be4a1b99e41ab9e0", + "tests/test.sh": "928d1497ad21aef0c456294e3784f1fddf8493410f23584da89f9e02ec6bd134", + "tests/test_outputs.py": "64d6ce107503714ada346e5606832ee886be9950971cf81523e73d8ba28e8d37" + } + } + ], + "total_model_runs": 34, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "preflight": [ + { + "arm": "main", + "job_name": "dev34-main-install-only-v2-20261012", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/install-only.json", + "config_sha256": "7d209b32e85d73a7be7197d1decbebd6c50d1add37b90ec5b35d7a6a2d06e33b" + } + ], + "queue": [ + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/adaptive-rejection-sampler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/adaptive-rejection-sampler.json", + "job_name": "dev34-adaptive-rejection-sampler-main-20261012", + "config_sha256": "7637bd048268072f49b8846661766edc8517c8101a667ad3bbc3d09966334685" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-cython-ext", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-cython-ext.json", + "job_name": "dev34-build-cython-ext-main-20261012", + "config_sha256": "1b9171f7155a340e6d3eb264749400c9a30f8872e67afe4b44d4fa686919b81b" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cancel-async-tasks", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cancel-async-tasks.json", + "job_name": "dev34-cancel-async-tasks-main-20261012", + "config_sha256": "ceacbecb49ad6588689a8955d66032e79eed89177faf7a782ef85ab5d802e5a9" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cobol-modernization", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cobol-modernization.json", + "job_name": "dev34-cobol-modernization-main-20261012", + "config_sha256": "081c3c24dcb8b249c288133977efba0efef1e0eaf7f21479703a6d973f648402" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/db-wal-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/db-wal-recovery.json", + "job_name": "dev34-db-wal-recovery-main-20261012", + "config_sha256": "2504427732ab221315f7e11a7eefdcd8ddde717a247939729ae1457cb77af491" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/fix-code-vulnerability", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/fix-code-vulnerability.json", + "job_name": "dev34-fix-code-vulnerability-main-20261012", + "config_sha256": "5dc9b2cf7e01e75bd54017fa1c652b1479f8c41374fbf44209cfb9059639a494" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/git-multibranch", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/git-multibranch.json", + "job_name": "dev34-git-multibranch-main-20261012", + "config_sha256": "f32823839420d04a6584a078bdeb0c07d3a7e4524b0bb5cf9606731c2419c7ed" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/kv-store-grpc", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/kv-store-grpc.json", + "job_name": "dev34-kv-store-grpc-main-20261012", + "config_sha256": "63be075e97d9fbb82eebea65817d8463b7f8fe119e147ca90fcbe4822992f29a" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/log-summary-date-ranges", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/log-summary-date-ranges.json", + "job_name": "dev34-log-summary-date-ranges-main-20261012", + "config_sha256": "94fc929689fa5cd9eedcde652db519348f5c37b315918674ea7571c0027e01d4" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/merge-diff-arc-agi-task", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/merge-diff-arc-agi-task.json", + "job_name": "dev34-merge-diff-arc-agi-task-main-20261012", + "config_sha256": "eec361dac1f523d07ced3d76ffbce2ccecada250a4510c2bc4f45083a10eede0" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/model-extraction-relu-logits", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/model-extraction-relu-logits.json", + "job_name": "dev34-model-extraction-relu-logits-main-20261012", + "config_sha256": "76672ff5283820f568467a46035aad80bf5e3bc46e268301695c507dee299d63" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/multi-source-data-merger", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/multi-source-data-merger.json", + "job_name": "dev34-multi-source-data-merger-main-20261012", + "config_sha256": "ed7a885c017cd7e9629522ee20dc39138107065cdf922f36d60582c512650420" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/nginx-request-logging", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/nginx-request-logging.json", + "job_name": "dev34-nginx-request-logging-main-20261012", + "config_sha256": "cb7af683b6cb2a8361ad5ec63eb5862e6b10d8d718191355720b1a2a17c5a77f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/openssl-selfsigned-cert", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/openssl-selfsigned-cert.json", + "job_name": "dev34-openssl-selfsigned-cert-main-20261012", + "config_sha256": "14367cf4b2a1e25240c1575df7738fccb3064c57e2aba361e0b928cfe0a57622" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pypi-server", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pypi-server.json", + "job_name": "dev34-pypi-server-main-20261012", + "config_sha256": "13a83dbf08f89d9a538174e3522007b2caf35cb4a9777a7797a5949a8fd1090c" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-cli", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-cli.json", + "job_name": "dev34-pytorch-model-cli-main-20261012", + "config_sha256": "80d5e27cd1b2621ebac680d2e74a59d941a558f6bdd6ac215a9a49e043d27149" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-recovery.json", + "job_name": "dev34-pytorch-model-recovery-main-20261012", + "config_sha256": "ad5f7ec41fe17f3ef44d3bf9b771adae193b4aabbf1ee00921ce9f3118d5374d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/qemu-alpine-ssh", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/qemu-alpine-ssh.json", + "job_name": "dev34-qemu-alpine-ssh-main-20261012", + "config_sha256": "5875cf6fc7ba45dd06dd4b2be0376b2bdbf0523a02ff8f67b5b85f7c06f3061a" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/query-optimize", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/query-optimize.json", + "job_name": "dev34-query-optimize-main-20261012", + "config_sha256": "bd6f22e5d0cd18af09950792d082cfdd34a08f422d55be03e0cb433ecb2f3799" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-log", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-log.json", + "job_name": "dev34-regex-log-main-20261012", + "config_sha256": "a16519a995fdc36b8da042657f1fc8839a329a32a85ebebe3f9f31192ecb09eb" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-pipeline-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-pipeline-parallelism.json", + "job_name": "dev34-torch-pipeline-parallelism-main-20261012", + "config_sha256": "4babd10c898a1ce2499a434d0bf31ffc1e9cb2d21f42c4d27246ef7502acc66e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-tensor-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-tensor-parallelism.json", + "job_name": "dev34-torch-tensor-parallelism-main-20261012", + "config_sha256": "c6f7ae3210dc116ad76c2a445eca945c9d0fc72077a274204f7c91b9414c3d03" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/write-compressor", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/write-compressor.json", + "job_name": "dev34-write-compressor-main-20261012", + "config_sha256": "244e7b084afe96eb3f79139f1fe2161cb732a9cad2f7c6080fd1d4e77623b0b2" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/constraints-scheduling", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/constraints-scheduling.json", + "job_name": "dev34-constraints-scheduling-main-20261012", + "config_sha256": "e892c7f481c2ff515aedfd6b4223ad47fdff24cc0ba24e501fb6c6cacd1c7d27" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/custom-memory-heap-crash", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/custom-memory-heap-crash.json", + "job_name": "dev34-custom-memory-heap-crash-main-20261012", + "config_sha256": "b2b5a47c9c3387998e3c0756b596f78f55ccd39d247fc0c92f2a3b7fe3a69179" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/dna-assembly", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/dna-assembly.json", + "job_name": "dev34-dna-assembly-main-20261012", + "config_sha256": "73a5776ac9f7fbd28b737bc9d0849fe2f8dceae9378f9b438b0d463a8bd73ff4" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/llm-inference-batching-scheduler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/llm-inference-batching-scheduler.json", + "job_name": "dev34-llm-inference-batching-scheduler-main-20261012", + "config_sha256": "30fabdfc5eadafb0fd510e7837e272077c781872bb5a843a2b8e927d6e3e1c78" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/path-tracing", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/path-tracing.json", + "job_name": "dev34-path-tracing-main-20261012", + "config_sha256": "c21b87c064f65b16dcd47731222e08efd81c0703a268b8fae963c86d33b63b4d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/schemelike-metacircular-eval", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/schemelike-metacircular-eval.json", + "job_name": "dev34-schemelike-metacircular-eval-main-20261012", + "config_sha256": "36ed3dcdb5859081e6445e07b65c539fa9f21133a4e15392121b1a566fb88348" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/caffe-cifar-10", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/caffe-cifar-10.json", + "job_name": "dev34-caffe-cifar-10-main-20261012", + "config_sha256": "fe005eeea1c9d30c252fe252cfbc3c94512bd8cc0d38652f111b355a78f2d264" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/circuit-fibsqrt", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/circuit-fibsqrt.json", + "job_name": "dev34-circuit-fibsqrt-main-20261012", + "config_sha256": "dd97cca17a635f60856af26f8fb67bb3f3f771e14b0917d73404fe0671c15559" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/mteb-leaderboard", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/mteb-leaderboard.json", + "job_name": "dev34-mteb-leaderboard-main-20261012", + "config_sha256": "025a1b1dba53f7359d00c71e72245c68762d76b17b8b265f464e81bc8f46bdcf" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-chess", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-chess.json", + "job_name": "dev34-regex-chess-main-20261012", + "config_sha256": "a845fc41273d9cf1cf81458184c5a87010eaeeba0e4cfaf58c3090ada731f8da" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-pov-ray", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-pov-ray.json", + "job_name": "dev34-build-pov-ray-main-20261012", + "config_sha256": "1fe9a8d4d8f4ef5f49f3c5ca257f6ce846c3429732812c398d3c03f269fe3bd7" + } + ], + "log_dir": "jobs/dev34-main-baseline-20261012-v2-orchestration", + "image_digests_file": "image-digests.json", + "image_digests_sha256": "5fc06fa5fbac1cccd3a4fb2f560878dd79ce314a6b1005183bbdcb4e8e13381f", + "budget_policy": "Native task agent timeout minus 180 seconds, separate 1800-second setup. Native verifier per-attempt time limits; max three attempts, retry only timeouts against the same agent output.", + "execution_policy": "All 34 model trials are fresh. Fixed ascending work-budget order, then task name; consecutive pairs finish before the next pair. No whole-trial retries, task replacements, or reuse of previous model outputs.", + "failure_policy": "Keep all setup failures and verifier errors separate from scored failures. Retain native zeros even when assertions never start, but classify them as verifier errors in analysis.", + "preflight_policy": "Fresh installation-only preflight for all 34 tasks before model work. QEMU has a reviewed verifier dependency profile; the other 33 are explicitly not_configured. A pre-model infrastructure amendment may reuse successful native installation trials only with complete hash-bound provenance and retain all failures.", + "information_policy": "Use frozen current main without prompt or runtime tuning. Do not inspect new trajectories or hidden test contents until all new model outputs freeze. Prior development exposure is disclosed, and each completed trajectory will be audited for benchmark-answer exposure.", + "scope": "One common-main baseline of the existing 34-task development pool, not a comparison with prior mixed-version results and not a random or held-out benchmark.", + "additional_file_sha256": { + "benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.conf": "d4a718524f7f7bccc6eb8ec06e73a0ddb0b580db0298f671c08ca4b5d37bd86a", + "benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml": "ed5e3a775ad575d84f04ef0f23825f87f2de5303610e15d0825e197ae6e283dc", + "benchmarks/harbor/scripts/apt_index_proxy.py": "233d2caaf2ec6cb0c5dbdc434d1c8369e4e52368376e1ae550ba2884f3d54d45", + "benchmarks/harbor/scripts/cache_image.py": "639709c03a7b0c3eb6aa78b4e87e6184b49b4b102853a31e33dd832d759180c0", + "benchmarks/harbor/scripts/run_comparison.py": "04f2106f0099a7b4d77e9206a36767f98a5e92c875ec78dd2de858c4936c12a8" + }, + "protocol_amendment": { + "prior_plan": "experiment.json", + "prior_plan_sha256": "171ab33ca7d88826217e8e2e21343905ab2ff84ad89f33acba17f732222d2d8b", + "reason": "Repair repeated installation download failures before any model work using a verified original-index transport cache.", + "transport_scope": "Read-only APT configuration mount for named public HTTP origins only. No general HTTP/HTTPS proxy variables, changed sources, or rebuilt task images. API traffic and non-APT tools use their normal network path.", + "original_runtime_unchanged": true, + "model_runs_started_before_amendment": 0, + "real_network_validation": { + "index_sha256": "9e0b5aabb2465b3d2e7a7fe27f9913846277833f7a2826e7767acccff5b588c5", + "bytes": 8790396, + "cold_seconds": 138.417564, + "warm_seconds": 0.008465 + } + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v1-evidence.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v1-evidence.json new file mode 100644 index 0000000..35d47f8 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v1-evidence.json @@ -0,0 +1,135 @@ +{ + "recorded_at": "2026-10-12T15:49:06.639441+00:00", + "original_plan_sha256": "171ab33ca7d88826217e8e2e21343905ab2ff84ad89f33acba17f732222d2d8b", + "state_path": "jobs/dev34-main-baseline-20261012-orchestration/dev34-main-install-only-20261012.json", + "state_sha256": "dae199b8142ab270e5c3ea13ca41c63807a78776c19fc7ceeb3bf20c6f204fa0", + "job_result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/result.json", + "job_result_sha256": "3ec2071c19091eb6042f650bd88bebb115a15dca31713018e4cd828123597ae4", + "model_runs_started": 0, + "official_verifier_runs_started": 0, + "started_trials": 6, + "unstarted_tasks": [ + "terminal-bench/git-multibranch", + "terminal-bench/kv-store-grpc", + "terminal-bench/log-summary-date-ranges", + "terminal-bench/merge-diff-arc-agi-task", + "terminal-bench/model-extraction-relu-logits", + "terminal-bench/multi-source-data-merger", + "terminal-bench/nginx-request-logging", + "terminal-bench/openssl-selfsigned-cert", + "terminal-bench/pypi-server", + "terminal-bench/pytorch-model-cli", + "terminal-bench/pytorch-model-recovery", + "terminal-bench/qemu-alpine-ssh", + "terminal-bench/query-optimize", + "terminal-bench/regex-log", + "terminal-bench/torch-pipeline-parallelism", + "terminal-bench/torch-tensor-parallelism", + "terminal-bench/write-compressor", + "terminal-bench/constraints-scheduling", + "terminal-bench/custom-memory-heap-crash", + "terminal-bench/dna-assembly", + "terminal-bench/llm-inference-batching-scheduler", + "terminal-bench/path-tracing", + "terminal-bench/schemelike-metacircular-eval", + "terminal-bench/caffe-cifar-10", + "terminal-bench/circuit-fibsqrt", + "terminal-bench/mteb-leaderboard", + "terminal-bench/regex-chess", + "terminal-bench/build-pov-ray" + ], + "trials": [ + { + "task": "terminal-bench/adaptive-rejection-sampler", + "trial": "adaptive-rejection-sampler__dLzh39D", + "finished_at": "2026-10-12T14:25:53.506160Z", + "exception_type": "AgentSetupTimeoutError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/adaptive-rejection-sampler__dLzh39D/result.json", + "result_sha256": "fc665e3c6e3b9fbade7c91b12639dca41e7759eb6639fef8fd4de2831e46b16e", + "bootstrap_sha256": "82878d7f8a339f6a19b1130e8160cbd8e23b500246887197f67728670a0e05a8" + }, + { + "task": "terminal-bench/build-cython-ext", + "trial": "build-cython-ext__eo75cET", + "finished_at": "2026-10-12T14:16:03.316850Z", + "exception_type": "NetworkConnectionError", + "bootstrap_status": "failed", + "stage": "agent_install", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/build-cython-ext__eo75cET/result.json", + "result_sha256": "9c08f2a8fe61ae948d16f3a1447caef2b5a17273666e578c0e0d3a38b698f732", + "bootstrap_sha256": "20c2b4bb8ee3bc17ebe99803d8086c15086ef9d07b0e9efb10813b47ef094842" + }, + { + "task": "terminal-bench/cancel-async-tasks", + "trial": "cancel-async-tasks__G8BCdbW", + "finished_at": "2026-10-12T14:46:25.596787Z", + "exception_type": "AgentSetupTimeoutError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/cancel-async-tasks__G8BCdbW/result.json", + "result_sha256": "2bb1ef9e0db51c408d0db621f4afa1cf96b239165d73011b6517b1ebffbf2642", + "bootstrap_sha256": "82878d7f8a339f6a19b1130e8160cbd8e23b500246887197f67728670a0e05a8" + }, + { + "task": "terminal-bench/cobol-modernization", + "trial": "cobol-modernization__eZhrhmk", + "finished_at": "2026-10-12T14:34:51.755049Z", + "exception_type": "NetworkConnectionError", + "bootstrap_status": "failed", + "stage": "agent_install", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/cobol-modernization__eZhrhmk/result.json", + "result_sha256": "9cf63cccf003bdbe7d434387ce8736dc4a0bea392febe9ee2c52c8bd0136dbcd", + "bootstrap_sha256": "c33d1442bdecb3f9e11d612e02b446d14ccbd0f7aed1108a4a0ff707ddd38fa4" + }, + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__U44UuJX", + "finished_at": "2026-10-12T15:05:22.642945Z", + "exception_type": "AgentSetupTimeoutError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/db-wal-recovery__U44UuJX/result.json", + "result_sha256": "ff7b455694c6d2f26f46014de0983b7ae86783dd465a8fb2af8e79a94b7e8fa7", + "bootstrap_sha256": "82878d7f8a339f6a19b1130e8160cbd8e23b500246887197f67728670a0e05a8" + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "trial": "fix-code-vulnerability__eY3pEt8", + "finished_at": "2026-10-12T15:05:49.249267Z", + "exception_type": "CancelledError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/fix-code-vulnerability__eY3pEt8/result.json", + "result_sha256": "d29687f27df846b6b1abbe53d36096bf044134e1c1fdcb1a4262eb0db2dad35d", + "bootstrap_sha256": "82878d7f8a339f6a19b1130e8160cbd8e23b500246887197f67728670a0e05a8" + } + ], + "diagnosis": "Native job ended after five failed installation trials and one cancelled installation; 28 tasks never started. Three setup timeouts, two uv-download TLS errors, and one cancellation are preserved. No models or official tests ran.", + "intervention": "Add an APT-only transport cache for original immutable SHA256 index URLs. Mutable release metadata and packages are not cached by the proxy. Original repositories, signed metadata validation, images, agent/adapter source, and all budgets remain unchanged. Start a fresh full installation preflight before model work." +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/adaptive-rejection-sampler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/adaptive-rejection-sampler.json new file mode 100644 index 0000000..702933c --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/adaptive-rejection-sampler.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-cython-ext.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-cython-ext.json new file mode 100644 index 0000000..e912c19 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-cython-ext.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-pov-ray.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-pov-ray.json new file mode 100644 index 0000000..6760851 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/build-pov-ray.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 11820 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/caffe-cifar-10.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/caffe-cifar-10.json new file mode 100644 index 0000000..35f2ea9 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/caffe-cifar-10.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cancel-async-tasks.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cancel-async-tasks.json new file mode 100644 index 0000000..9904548 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cancel-async-tasks.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/circuit-fibsqrt.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/circuit-fibsqrt.json new file mode 100644 index 0000000..9e61c98 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/circuit-fibsqrt.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cobol-modernization.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cobol-modernization.json new file mode 100644 index 0000000..a549048 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/cobol-modernization.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/constraints-scheduling.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/constraints-scheduling.json new file mode 100644 index 0000000..f70b6ec --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/constraints-scheduling.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1020 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/custom-memory-heap-crash.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/custom-memory-heap-crash.json new file mode 100644 index 0000000..8ceed2b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/custom-memory-heap-crash.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/db-wal-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/db-wal-recovery.json new file mode 100644 index 0000000..dd6adec --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/db-wal-recovery.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/dna-assembly.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/dna-assembly.json new file mode 100644 index 0000000..870db0c --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/dna-assembly.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/fix-code-vulnerability.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/fix-code-vulnerability.json new file mode 100644 index 0000000..517af3f --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/fix-code-vulnerability.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/git-multibranch.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/git-multibranch.json new file mode 100644 index 0000000..b00ca83 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/git-multibranch.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/install-only.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/install-only.json new file mode 100644 index 0000000..f80169b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/install-only.json @@ -0,0 +1,161 @@ +{ + "n_concurrent_trials": 2, + "n_attempts": 1, + "install_only": true, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + }, + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12" + }, + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1" + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + }, + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307" + }, + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + }, + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d" + }, + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17" + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + }, + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302" + }, + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1" + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + }, + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc" + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + }, + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684" + }, + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e" + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + }, + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f" + }, + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67" + }, + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c" + }, + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6" + }, + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf" + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + }, + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c" + }, + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78" + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/kv-store-grpc.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/kv-store-grpc.json new file mode 100644 index 0000000..406cd15 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/kv-store-grpc.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/llm-inference-batching-scheduler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/llm-inference-batching-scheduler.json new file mode 100644 index 0000000..d3f238e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/llm-inference-batching-scheduler.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/log-summary-date-ranges.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/log-summary-date-ranges.json new file mode 100644 index 0000000..d7f2f5b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/log-summary-date-ranges.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/merge-diff-arc-agi-task.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/merge-diff-arc-agi-task.json new file mode 100644 index 0000000..a91cc12 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/merge-diff-arc-agi-task.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/model-extraction-relu-logits.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/model-extraction-relu-logits.json new file mode 100644 index 0000000..a4fb110 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/model-extraction-relu-logits.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/mteb-leaderboard.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/mteb-leaderboard.json new file mode 100644 index 0000000..329019b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/mteb-leaderboard.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/multi-source-data-merger.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/multi-source-data-merger.json new file mode 100644 index 0000000..ad3a59b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/multi-source-data-merger.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/nginx-request-logging.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/nginx-request-logging.json new file mode 100644 index 0000000..d019ff8 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/nginx-request-logging.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/openssl-selfsigned-cert.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/openssl-selfsigned-cert.json new file mode 100644 index 0000000..56d2a92 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/openssl-selfsigned-cert.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/path-tracing.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/path-tracing.json new file mode 100644 index 0000000..ae4fb42 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/path-tracing.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pypi-server.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pypi-server.json new file mode 100644 index 0000000..561380c --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pypi-server.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-cli.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-cli.json new file mode 100644 index 0000000..118d3c8 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-cli.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-recovery.json new file mode 100644 index 0000000..700427d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/pytorch-model-recovery.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/qemu-alpine-ssh.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/qemu-alpine-ssh.json new file mode 100644 index 0000000..b941035 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/qemu-alpine-ssh.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/query-optimize.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/query-optimize.json new file mode 100644 index 0000000..974b14b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/query-optimize.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-chess.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-chess.json new file mode 100644 index 0000000..21e844a --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-chess.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-log.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-log.json new file mode 100644 index 0000000..64be37c --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/regex-log.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/schemelike-metacircular-eval.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/schemelike-metacircular-eval.json new file mode 100644 index 0000000..93998a4 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/schemelike-metacircular-eval.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 2220 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-pipeline-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-pipeline-parallelism.json new file mode 100644 index 0000000..d1b0cfd --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-pipeline-parallelism.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-tensor-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-tensor-parallelism.json new file mode 100644 index 0000000..3c30cf2 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/torch-tensor-parallelism.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/write-compressor.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/write-compressor.json new file mode 100644 index 0000000..cf99f5c --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v2/write-compressor.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/apt-cache.compose.yaml" + ] + } +} diff --git a/benchmarks/harbor/scripts/apt_index_proxy.py b/benchmarks/harbor/scripts/apt_index_proxy.py new file mode 100644 index 0000000..bc22965 --- /dev/null +++ b/benchmarks/harbor/scripts/apt_index_proxy.py @@ -0,0 +1,245 @@ +"""Accelerate original APT by-hash indexes without changing repositories. + +Mutable release metadata and package downloads are forwarded without caching. +Only /by-hash/SHA256/ objects are cached, after exact range, size and +whole-object SHA256 validation. APT still verifies signed release metadata and +the current index/package checksums. Bind only to a local Docker bridge. +""" + +import argparse +from datetime import datetime, timezone +from http.client import IncompleteRead +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +import importlib.util +import json +from pathlib import Path +import re +import threading +import time +from urllib.error import HTTPError, URLError +from urllib.parse import urlsplit +from urllib.request import HTTPRedirectHandler, ProxyHandler, Request, build_opener + + +spec = importlib.util.spec_from_file_location( + "verified_range_cache", Path(__file__).with_name("cache_image.py"), +) +ranges = importlib.util.module_from_spec(spec) +spec.loader.exec_module(ranges) + +HOSTS = frozenset({ + "deb.debian.org", "security.debian.org", "archive.debian.org", + "archive.ubuntu.com", "security.ubuntu.com", "ports.ubuntu.com", + "old-releases.ubuntu.com", +}) +BY_HASH = re.compile(r"/by-hash/SHA256/([0-9a-f]{64})$") +MAX_INDEX_BYTES = 128 * 1024 * 1024 +REQUEST_HEADERS = {"if-modified-since", "if-none-match", "range", "user-agent"} +RESPONSE_HEADERS = { + "content-type", "content-length", "content-range", "last-modified", + "etag", "cache-control", "expires", "accept-ranges", +} + + +def validate_url(url): + parsed = urlsplit(url) + if (parsed.scheme not in {"http", "https"} or parsed.hostname not in HOSTS + or parsed.username is not None or parsed.password is not None + or parsed.port not in {None, 80, 443} or parsed.query or parsed.fragment): + raise ValueError("Only original public APT origins without credentials are allowed") + return parsed + + +class AptRedirect(HTTPRedirectHandler): + def redirect_request(self, request, response, code, message, headers, url): + validate_url(url) + if urlsplit(request.full_url).scheme == "https" and urlsplit(url).scheme != "https": + raise ValueError("HTTPS redirects cannot downgrade transport") + return super().redirect_request(request, response, code, message, headers, url) + + +def open_origin(url, headers=None, method="GET"): + validate_url(url) + request = Request(url, headers={"Accept-Encoding": "identity", **(headers or {})}, method=method) + return build_opener(ProxyHandler({}), AptRedirect()).open(request, timeout=30) + + +def public_get(url, headers=None, limit=1024 * 1024): + for attempt in range(3): + try: + with open_origin(url, headers) as response: + data = response.read(limit + 1) + if len(data) > limit: + raise ValueError("Response exceeds requested range size") + return response.status, dict(response.headers.items()), data + except (URLError, OSError, TimeoutError, IncompleteRead) as error: + if attempt == 2: + raise RuntimeError(f"APT origin request failed: {type(error).__name__}") from None + time.sleep(attempt + 1) + + +# Reuse the image utility's tested exact-range and whole-object verification. +ranges.get = lambda url, headers=None, limit=1024 * 1024: public_get(url, headers, limit) + + +class IndexCache: + def __init__(self, root, workers=8, chunk_size=256 * 1024): + if not 1 <= workers <= 16 or not 1024 <= chunk_size <= 1024 * 1024: + raise ValueError("Invalid range worker count or chunk size") + self.root = Path(root) + self.workers, self.chunk_size = workers, chunk_size + self.guard = threading.Lock() + self.locks = {} + + def fetch(self, url): + parsed = validate_url(url) + match = BY_HASH.search(parsed.path) + if not match: + raise ValueError("Mutable URLs must not enter the index cache") + digest = match.group(1) + with self.guard: + lock = self.locks.setdefault(digest, threading.Lock()) + with lock: + path = self.root / "blobs/sha256" / digest + if path.exists(): + size = path.stat().st_size + if not 0 < size <= MAX_INDEX_BYTES: + raise ValueError("Cached index exceeds supported size") + ranges.verify(path, {"digest": "sha256:" + digest, "size": size}) + return path, True + status, headers, data = public_get(url, {"Range": "bytes=0-0"}, 1) + headers = {key.lower(): value for key, value in headers.items()} + content_range = re.fullmatch(r"bytes 0-0/(\d+)", headers.get("content-range", "")) + if status != 206 or data == b"" or len(data) != 1 or not content_range: + raise ValueError("Origin must return the exact size probe range") + size = int(content_range.group(1)) + if not 0 < size <= MAX_INDEX_BYTES: + raise ValueError("Index exceeds supported size") + event(kind="download_start", host=parsed.hostname, sha256=digest, size=size) + path = ranges.fetch_blob( + self.root, {"digest": "sha256:" + digest, "size": size}, url, {}, + workers=self.workers, chunk_size=self.chunk_size, + ) + return path, False + + +def byte_range(header, size): + if header is None: + return 0, size - 1, 200 + match = re.fullmatch(r"bytes=(\d+)-(\d*)", header) + if not match: + raise ValueError("Unsupported byte range") + start = int(match.group(1)) + end = min(int(match.group(2)), size - 1) if match.group(2) else size - 1 + if start >= size or end < start: + raise ValueError("Unsatisfiable byte range") + return start, end, 206 + + +def event(**fields): + print(json.dumps({"time": datetime.now(timezone.utc).isoformat(), **fields}), flush=True) + + +class Handler(BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def log_message(self, *_args): + pass # Do not publish arbitrary request URLs or headers. + + def do_GET(self): + self.respond(head=False) + + def do_HEAD(self): + self.respond(head=True) + + def respond(self, head): + self.close_connection = True + self.response_started = False + try: + parsed = validate_url(self.path) + except ValueError: + self.send_error(403, "Unsupported APT origin") + return + try: + if BY_HASH.search(parsed.path): + path, hit = self.server.cache.fetch(self.path) + size = path.stat().st_size + try: + start, end, status = byte_range(self.headers.get("Range"), size) + except ValueError: + self.send_response(416) + self.send_header("Content-Range", f"bytes */{size}") + self.send_header("Content-Length", "0") + self.send_header("Connection", "close") + self.end_headers() + return + self.send_response(status) + self.send_header("Content-Length", str(end - start + 1)) + self.send_header("Content-Type", "application/octet-stream") + self.send_header("Accept-Ranges", "bytes") + self.send_header("Connection", "close") + if status == 206: + self.send_header("Content-Range", f"bytes {start}-{end}/{size}") + self.end_headers() + if not head: + with path.open("rb") as stream: + stream.seek(start) + left = end - start + 1 + while left: + block = stream.read(min(left, 64 * 1024)) + self.wfile.write(block) + left -= len(block) + event(kind="index", host=parsed.hostname, sha256=path.name, size=size, cache_hit=hit) + else: + self.forward(head) + except (BrokenPipeError, ConnectionResetError): + event(kind="client_disconnected", host=parsed.hostname) + except Exception as error: + event(kind="error", host=parsed.hostname, error_type=type(error).__name__) + if not self.response_started: + self.send_error(502, "APT origin transfer or integrity validation failed") + + def end_headers(self): + self.response_started = True + super().end_headers() + + def forward(self, head): + headers = {k: v for k, v in self.headers.items() if k.lower() in REQUEST_HEADERS} + try: + response = open_origin(self.path, headers, "HEAD" if head else "GET") + except HTTPError as error: + response = error + with response: + self.send_response(response.status) + for key, value in response.headers.items(): + if key.lower() in RESPONSE_HEADERS: + self.send_header(key, value) + self.send_header("Connection", "close") + self.end_headers() + if not head: + while block := response.read(64 * 1024): + self.wfile.write(block) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--bind", required=True, help="Local Docker bridge address, not a public interface") + parser.add_argument("--port", type=int, default=18742) + parser.add_argument("--cache", type=Path, required=True) + parser.add_argument("--workers", type=int, default=8) + args = parser.parse_args() + if args.bind in {"0.0.0.0", "::", ""}: + parser.error("Binding all interfaces is not supported") + cache = IndexCache(args.cache, args.workers) + cache.root.mkdir(parents=True, exist_ok=True) + import fcntl + with (cache.root / ".lock").open("w") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + server = ThreadingHTTPServer((args.bind, args.port), Handler) + server.cache = cache + event(kind="ready", bind=args.bind, port=args.port, workers=args.workers) + server.serve_forever() + + +if __name__ == "__main__": + main() diff --git a/benchmarks/harbor/scripts/run_comparison.py b/benchmarks/harbor/scripts/run_comparison.py index 1f2cfbc..87ff60e 100644 --- a/benchmarks/harbor/scripts/run_comparison.py +++ b/benchmarks/harbor/scripts/run_comparison.py @@ -34,6 +34,10 @@ def prompt_from_source(path): def validate_plan(root, plan_path, plan): + for name, expected in plan.get("additional_file_sha256", {}).items(): + path = (root / name).resolve() + if not path.is_relative_to(root.resolve()) or digest(path) != expected: + raise ValueError(f"Registered infrastructure file changed: {name}") jobs = plan["preflight"] + plan["queue"] names = [job["job_name"] for job in jobs] if len(names) != len(set(names)): diff --git a/benchmarks/harbor/tests/test_apt_index_proxy.py b/benchmarks/harbor/tests/test_apt_index_proxy.py new file mode 100644 index 0000000..21e7d6e --- /dev/null +++ b/benchmarks/harbor/tests/test_apt_index_proxy.py @@ -0,0 +1,162 @@ +"""The APT transport cache must preserve original content and trust checks.""" + +import hashlib +import http.client +import importlib.util +import io +from pathlib import Path +from concurrent.futures import ThreadPoolExecutor +import threading +from urllib.error import HTTPError + +import pytest + +spec = importlib.util.spec_from_file_location( + "apt_index_proxy", Path(__file__).parents[1] / "scripts/apt_index_proxy.py", +) +proxy = importlib.util.module_from_spec(spec) +spec.loader.exec_module(proxy) + + +def fixture(payload): + digest = hashlib.sha256(payload).hexdigest() + return "http://deb.debian.org/debian/dists/bookworm/main/binary-amd64/by-hash/SHA256/" + digest + + +@pytest.mark.parametrize("url", [ + "http://127.0.0.1/private", "http://deb.debian.org.evil.example/file", + "http://user:secret@deb.debian.org/file", "file:///etc/passwd", + "http://deb.debian.org:8080/file", "http://deb.debian.org/file?token=secret", + "http://deb.debian.org/file#fragment", +]) +def test_restrict_origins(url): + with pytest.raises(ValueError): + proxy.validate_url(url) + + +def test_mutable_metadata_and_debs_never_enter_cache(tmp_path): + cache = proxy.IndexCache(tmp_path) + for suffix in ["dists/bookworm/InRelease", "dists/bookworm/main/Packages.xz", "pool/main/p/package.deb"]: + with pytest.raises(ValueError, match="Mutable"): + cache.fetch("http://deb.debian.org/debian/" + suffix) + assert not list(tmp_path.iterdir()) + + +def fake_origin(payload, requests): + def get(url, headers, limit): + start, end = map(int, headers["Range"].removeprefix("bytes=").split("-")) + requests.append((start, end)) + return 206, {"Content-Range": f"bytes {start}-{end}/{len(payload)}"}, payload[start:end + 1] + return get + + +def test_shared_index_is_verified_and_downloaded_once(tmp_path, monkeypatch): + payload = bytes(range(256)) * 20 + requests = [] + monkeypatch.setattr(proxy, "public_get", fake_origin(payload, requests)) + cache = proxy.IndexCache(tmp_path, workers=2, chunk_size=1024) + with ThreadPoolExecutor(max_workers=2) as pool: + results = list(pool.map(cache.fetch, [fixture(payload)] * 2)) + assert sorted(hit for _, hit in results) == [False, True] + assert len(requests) == 6 # Size probe plus five exact chunks. + assert all(path.read_bytes() == payload for path, _ in results) + + +def test_corrupt_cached_index_is_never_served(tmp_path, monkeypatch): + payload = b"trusted signed-index bytes" + monkeypatch.setattr(proxy, "public_get", fake_origin(payload, [])) + cache = proxy.IndexCache(tmp_path) + path, _ = cache.fetch(fixture(payload)) + path.write_bytes(b"x" * len(payload)) + with pytest.raises(ValueError, match="SHA256"): + cache.fetch(fixture(payload)) + + +def test_wrong_origin_content_is_never_committed(tmp_path, monkeypatch): + payload = b"original content" + monkeypatch.setattr(proxy, "public_get", fake_origin(b"modified content", [])) + cache = proxy.IndexCache(tmp_path) + with pytest.raises(ValueError, match="SHA256"): + cache.fetch(fixture(payload)) + digest = hashlib.sha256(payload).hexdigest() + assert not (tmp_path / "blobs/sha256" / digest).exists() + + +@pytest.mark.parametrize("status,header,body", [ + (200, "bytes 0-0/10", b"x"), (206, "bytes 1-1/10", b"x"), + (206, "bytes 0-0/10", b""), (206, "bytes 0-0/9999999999", b"x"), +]) +def test_reject_invalid_size_probe(tmp_path, monkeypatch, status, header, body): + monkeypatch.setattr(proxy, "public_get", lambda *_args: (status, {"Content-Range": header}, body)) + with pytest.raises(ValueError): + proxy.IndexCache(tmp_path).fetch(fixture(b"original")) + + +def test_resumed_range_matches_original_file_bytes(): + assert proxy.byte_range(None, 10) == (0, 9, 200) + assert proxy.byte_range("bytes=3-", 10) == (3, 9, 206) + assert proxy.byte_range("bytes=3-99", 10) == (3, 9, 206) + for bad in ["bytes=10-", "bytes=5-3", "bytes=-3", "bytes=0-1,3-4"]: + with pytest.raises(ValueError): + proxy.byte_range(bad, 10) + + +@pytest.fixture +def server(tmp_path): + server = proxy.ThreadingHTTPServer(("127.0.0.1", 0), proxy.Handler) + server.cache = proxy.IndexCache(tmp_path) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + yield server + server.shutdown() + server.server_close() + thread.join() + + +def request(server, url, method="GET", headers=None): + connection = http.client.HTTPConnection(*server.server_address, timeout=5) + try: + connection.request(method, url, headers=headers or {}) + response = connection.getresponse() + return response.status, dict(response.getheaders()), response.read() + finally: + connection.close() + + +def test_http_serves_only_verified_content_and_correct_ranges(server, monkeypatch): + payload = b"exact original index" + monkeypatch.setattr(proxy, "public_get", fake_origin(payload, [])) + url = fixture(payload) + assert request(server, url)[2] == payload + status, headers, body = request(server, url, headers={"Range": "bytes=3-7"}) + assert status == 206 and body == payload[3:8] + assert headers["Content-Range"] == f"bytes 3-7/{len(payload)}" + status, headers, body = request(server, url, method="HEAD") + assert status == 200 and body == b"" and int(headers["Content-Length"]) == len(payload) + assert request(server, url, headers={"Range": "bytes=999-"})[0] == 416 + assert request(server, "http://127.0.0.1/private")[0] == 403 + + +def test_http_rejects_origin_hash_mismatch(server, monkeypatch): + monkeypatch.setattr(proxy, "public_get", fake_origin(b"tampered index", [])) + status, _, body = request(server, fixture(b"original index")) + assert status == 502 and b"tampered index" not in body + + +def test_release_metadata_is_fetched_again_and_conditionals_preserved(server, monkeypatch): + calls = [] + def origin(url, headers, method): + calls.append((headers, method)) + if "If-Modified-Since" in headers: + raise HTTPError(url, 304, "Not Modified", {"Content-Length": "0"}, io.BytesIO()) + body = b"signed release " + str(len(calls)).encode() + response = io.BytesIO(body) + response.status = 200 + response.headers = {"Content-Length": str(len(body))} + return response + monkeypatch.setattr(proxy, "open_origin", origin) + url = "http://deb.debian.org/debian/dists/bookworm/InRelease" + assert request(server, url)[2] == b"signed release 1" + assert request(server, url)[2] == b"signed release 2" + assert request(server, url, headers={"If-Modified-Since": "original timestamp", "Authorization": "private"})[0] == 304 + assert calls[-1] == ({"If-Modified-Since": "original timestamp"}, "GET") diff --git a/benchmarks/harbor/tests/test_comparison.py b/benchmarks/harbor/tests/test_comparison.py index 7284900..fc2f5d6 100644 --- a/benchmarks/harbor/tests/test_comparison.py +++ b/benchmarks/harbor/tests/test_comparison.py @@ -57,6 +57,25 @@ def test_interrupted_job_cannot_silently_restart(tmp_path): comparison.run_job(tmp_path, {}, job, tmp_path, False) +def test_registered_transport_files_cannot_change(tmp_path): + path = tmp_path / "apt-cache.conf" + path.write_text("original configuration") + plan = {"additional_file_sha256": {path.name: comparison.digest(path)}, + "preflight": [], "queue": [], "total_model_runs": 0, + "max_concurrent_trials": 2, "whole_trial_retries": 0, + "arms": {}, "tasks": []} + comparison.validate_plan(tmp_path, tmp_path / "plan.json", plan) + path.write_text("changed proxy endpoint") + with pytest.raises(ValueError, match="infrastructure file changed"): + comparison.validate_plan(tmp_path, tmp_path / "plan.json", plan) + + +def test_registered_transport_paths_cannot_escape_repository(tmp_path): + plan = {"additional_file_sha256": {"../outside": "untrusted"}} + with pytest.raises(ValueError, match="infrastructure file changed"): + comparison.validate_plan(tmp_path, tmp_path / "plan.json", plan) + + @pytest.fixture def accepted_installation(tmp_path): source = tmp_path / "source" From 6ff949032157251aa74518d52472ef27ac195573 Mon Sep 17 00:00:00 2001 From: minixalpha Date: Mon, 12 Oct 2026 09:03:47 -0700 Subject: [PATCH 08/15] fix(benchmarks): isolate apt cache proxies in task sidecars --- benchmarks/harbor/README.md | 7 + benchmarks/harbor/README.zh-CN.md | 6 + .../dev34-main-baseline-20261012/README.md | 35 +- .../experiment-v3.json | 1374 +++++++++++++++++ .../preflight-v2-evidence.json | 97 ++ .../adaptive-rejection-sampler.json | 28 + .../transport-v3/apt-cache.compose.json | 56 + .../transport-v3/apt-cache.conf | 8 + .../transport-v3/build-cython-ext.json | 28 + .../transport-v3/build-pov-ray.json | 28 + .../transport-v3/caffe-cifar-10.json | 28 + .../transport-v3/cancel-async-tasks.json | 28 + .../transport-v3/circuit-fibsqrt.json | 28 + .../transport-v3/cobol-modernization.json | 28 + .../transport-v3/constraints-scheduling.json | 28 + .../custom-memory-heap-crash.json | 28 + .../transport-v3/db-wal-recovery.json | 28 + .../transport-v3/dna-assembly.json | 28 + .../transport-v3/fix-code-vulnerability.json | 28 + .../transport-v3/git-multibranch.json | 28 + .../transport-v3/install-only.json | 161 ++ .../transport-v3/kv-store-grpc.json | 28 + .../llm-inference-batching-scheduler.json | 28 + .../transport-v3/log-summary-date-ranges.json | 28 + .../transport-v3/merge-diff-arc-agi-task.json | 28 + .../model-extraction-relu-logits.json | 28 + .../transport-v3/mteb-leaderboard.json | 28 + .../multi-source-data-merger.json | 28 + .../transport-v3/nginx-request-logging.json | 28 + .../transport-v3/openssl-selfsigned-cert.json | 28 + .../transport-v3/path-tracing.json | 28 + .../transport-v3/pypi-server.json | 28 + .../transport-v3/pytorch-model-cli.json | 28 + .../transport-v3/pytorch-model-recovery.json | 28 + .../transport-v3/qemu-alpine-ssh.json | 28 + .../transport-v3/query-optimize.json | 28 + .../transport-v3/regex-chess.json | 28 + .../transport-v3/regex-log.json | 28 + .../schemelike-metacircular-eval.json | 28 + .../torch-pipeline-parallelism.json | 28 + .../torch-tensor-parallelism.json | 28 + .../transport-v3/write-compressor.json | 28 + benchmarks/harbor/scripts/apt_index_proxy.py | 19 +- .../harbor/tests/test_apt_index_proxy.py | 36 + 44 files changed, 2727 insertions(+), 24 deletions(-) create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v3.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v2-evidence.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/adaptive-rejection-sampler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-cython-ext.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-pov-ray.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/caffe-cifar-10.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cancel-async-tasks.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/circuit-fibsqrt.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cobol-modernization.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/constraints-scheduling.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/custom-memory-heap-crash.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/db-wal-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/dna-assembly.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/fix-code-vulnerability.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/git-multibranch.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/install-only.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/kv-store-grpc.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/llm-inference-batching-scheduler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/log-summary-date-ranges.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/merge-diff-arc-agi-task.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/model-extraction-relu-logits.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/mteb-leaderboard.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/multi-source-data-merger.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/nginx-request-logging.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/openssl-selfsigned-cert.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/path-tracing.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pypi-server.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-cli.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/qemu-alpine-ssh.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/query-optimize.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-chess.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-log.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/schemelike-metacircular-eval.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-pipeline-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-tensor-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/write-compressor.json diff --git a/benchmarks/harbor/README.md b/benchmarks/harbor/README.md index fec388a..bd264d1 100644 --- a/benchmarks/harbor/README.md +++ b/benchmarks/harbor/README.md @@ -142,6 +142,13 @@ hashes. Original repositories, task images, and test scripts stay unchanged. Keep the proxy running through preflight and model trials, record the mount and utility hashes in the experiment plan, and retain earlier setup failures. +If containers cannot reach the host listener, run the utility in an isolated +sidecar with `network_mode: service:main`, bind `127.0.0.1`, and use a readiness +health check. The registered example uses this mode. Mount only the proxy code +and index cache into the sidecar; do not share task files or credentials. Multiple +proxy processes coordinate cache writes with per-digest file locks. No host +proxy process or published port is needed in this mode. + ### Direct Harbor invocation For adapter development or comparison with published versions, the lower-level diff --git a/benchmarks/harbor/README.zh-CN.md b/benchmarks/harbor/README.zh-CN.md index 6c6f3cb..4405d1f 100644 --- a/benchmarks/harbor/README.zh-CN.md +++ b/benchmarks/harbor/README.zh-CN.md @@ -115,6 +115,12 @@ SHA256 校验后才能提供给客户端；缓存命中也会重新校验哈希 软件源、任务镜像和测试脚本保持原样。预检及模型运行期间需保持代理运行，将挂载 文件与工具哈希登记进实验计划，并保留之前的安装失败记录。 +如果容器无法访问宿主机监听端口，可将该工具运行在独立辅助容器中，设置 +`network_mode: service:main`，绑定 `127.0.0.1`，并配置就绪健康检查。上述登记示例 +采用这一方式。辅助容器只挂载代理代码和索引缓存，不共享题目文件或凭据。多个 +代理进程通过按摘要划分的文件锁协调缓存写入；此模式不需要宿主机代理进程，也 +不向宿主机发布端口。 + ### 直接调用 Harbor 开发 adapter 或对照已发布版本时，仍可使用底层 Harbor 入口，并固定容器内 agent 版本： diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md index be47d13..1a82bf0 100644 --- a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md @@ -39,27 +39,32 @@ total attempts, with the same agent output and original per-attempt limits. ## Run -The active plan is [experiment-v2.json](experiment-v2.json). The original +The active plan is [experiment-v3.json](experiment-v3.json). The original install-only job ended with three setup timeouts, two uv-download TLS errors, and one cancellation; 28 tasks never started. No models or official tests ran. [Preserved evidence](preflight-v1-evidence.json) records the native artifacts and their hashes. The amendment adds only an APT transport cache, keeping the runtime source, original sources/images/scripts, budgets, and scoring fixed. -Start the reusable host proxy before either phase and keep it running: - -```bash -python benchmarks/harbor/scripts/apt_index_proxy.py \ - --bind 172.17.0.1 --port 18742 \ - --cache .cache/nanopy-harbor/apt-indexes --workers 16 -``` - -[APT configuration](apt-cache.conf) routes only the named public APT HTTP -origins through this proxy. The [Compose override](apt-cache.compose.yaml) -mounts it read-only in each fresh container. Mutable release metadata and +The v2 host-proxy route proved unreachable from Docker containers and was +stopped before model work. [Native evidence](preflight-v2-evidence.json) retains +the errors and the subsequent successful sidecar connectivity probe. Historical +v2 utility hashes refer to branch commit `f4cdab9`. + +The active [APT configuration](transport-v3/apt-cache.conf) routes only the named +public APT HTTP origins through a loopback proxy. Its +[Compose override](transport-v3/apt-cache.compose.json) starts a separate proxy +sidecar in each task's own network namespace and waits for its health check. +The sidecar uses an already cached image pinned by digest, has no task filesystem +or agent credentials mounted, and publishes no ports. It has a separate 0.5 CPU / +256 MiB limit; task resources stay unchanged. Each sidecar shares the verified +host index cache, protected by per-digest cross-process locks. No host proxy +process is required for v3. + +The APT configuration is mounted read-only. Mutable release metadata and packages are not proxy-cached; immutable SHA256 indexes require complete hash verification. Model/API and other tool traffic retain their normal routes. -Both mount files and the reusable host utility are frozen by hashes in v2. +Both mount files and the reusable proxy utility are frozen by hashes in v3. The recorded real-network probe fetched the original 8,790,396-byte Debian index in 138.418 seconds cold and 0.008 seconds warm, with identical SHA256. @@ -71,11 +76,11 @@ before sharing. ```bash python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v2.json \ + --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v3.json \ --phase preflight python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v2.json \ + --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v3.json \ --phase model ``` diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v3.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v3.json new file mode 100644 index 0000000..0a348f0 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v3.json @@ -0,0 +1,1374 @@ +{ + "schema_version": 1, + "registered_at": "2026-10-12T16:02:56.129475+00:00", + "experiment": "dev34-main-baseline-20261012", + "selection_file": "selection.json", + "selection_sha256": "4572b1cc432a16b9c5d1dd554c8d2189ae80414f386a90bf03829bc4d0c3da85", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "arms": { + "main": { + "source_root": "jobs/dev34-baseline-main-source", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "prompt_file": "main-prompt.txt", + "prompt_sha256": "37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d", + "prompt_words": 385 + } + }, + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/adaptive-rejection-sampler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "95408217b9c484ffc18a02a91b7706323b6902919c72b24866a32e27a23543e2", + "environment/Dockerfile": "e644a3343e339303230f1441b7383f076d21c69a5a7aa6430b4f8a9d27f15c8b", + "environment/protected.tar.gz.enc": "7fc9bf1347bb97da8dede04b2c58638f225070913e222d76627068aebd1b1df2", + "instruction.md": "e6d28c28acd76bef61817427c74c2a29e3e8d75f535cbd708afcbfd02c54013f", + "solution/solve.sh": "fe443a8082b5183ba1623311ba0ee958888aefa633eff93766b9ae14e9bdd84d", + "task.toml": "6ed7b37f74beb46e9d47b0f5fad53025667c9548baff4e96d467b56d429fab1a", + "tests/test.sh": "38b43560d173cc2b952c3a3e17b8a480216d84e33450515e047bcb0d806b1e0a", + "tests/test_outputs.py": "547dc6e107f034f41703722aeceb6d0236e3fb69116fc3f2fbaba11884de352f" + } + }, + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/build-cython-ext:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d5f5113941c3963292bae396baa9369571026f46e59437f98ddcc93475dd7729", + "environment/Dockerfile": "cb6cf421cb9b69d43bd25cd491e49bda982fbb865d47515178d017eda727194e", + "instruction.md": "7e52874e45f88505bd21540229719de6639a401477557d814489981d472f0e34", + "solution/solve.sh": "41bbdde88617070366a70e518eedca466a4ea1acd7e96899b2fb7da988e40abc", + "task.toml": "c472f0e7522c128fb7f03e00b50c8cdf2e5e6474daa1e094e7441f4b54e770b2", + "tests/test.sh": "0156c9c4085258b696ce424cf1ce96cba44d1263c5d73bdf54083fd80237f915", + "tests/test_outputs.py": "6549f1a8e04b6163ec08877e6f6b9868bf94563f3a3c72ac1fba9c512c31689e" + } + }, + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cancel-async-tasks:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9adf86b1279cc44f83fe16e6d340595aa992240f1d87243d204cae32e453693b", + "environment/Dockerfile": "1ee7bdb4a7536f5f114c11a3ac6ff42790abfea901fcba6b2a8ea8685f19140e", + "instruction.md": "23d22f7039b50682941652c716f96dac314b8585c32c133614cac77dc067f47e", + "solution/solve.sh": "4938ca4ad5b7dcd36b414b42a1abb10c058ecefce7462b678f9bd05c9fae1844", + "task.toml": "14db0675e04408f2133b0067e8dbddf66a1d6e7baf6b8d38bef8ecccfc4e58a8", + "tests/test.py": "f533c0a64db068f582b4733c5b81f2e66b8fa52fcd3efa65752e6bd6d24780aa", + "tests/test.sh": "79600a6b863def31b4daa008297423010cd2d6243e689d67503e118e4dc53397", + "tests/test_outputs.py": "d6f510b51fafc81b4e9b5cc95535aa13491c8753e0e4171db42c45f4de460ee7" + } + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cobol-modernization:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "c89318cf59efcc7040e3d946022e2616161b6733931a080cdeea7b1918e7ecc4", + "environment/Dockerfile": "8b175c83fc03ec7d16d6101516bc5100230b592f480e435279f4fe051c9280b2", + "environment/data/ACCOUNTS.DAT": "46f63eb29442ebc6716bcd8a0c36f4367910f2783f6b84a307ed9a61ccd1046e", + "environment/data/BOOKS.DAT": "db04da646207cd87f8e9c5b6f0b57d95953899f3eef22de1a81423f06ef4a9fd", + "environment/data/TRANSACTIONS.DAT": "1fa4cc5420ded94d585c563e8196b85166ebbae2bb8924b20014717fb2fb9ee3", + "environment/src/INPUT.DAT": "52fc984f0cf14af1495285d7efe02b9bae13159eea59255560ee938c4f78afee", + "environment/src/program.cbl": "27cb0e56f65f52fa75f346720b37a180b8e034ebe7d7622b979f195087bd6ebb", + "instruction.md": "a87074f1141b7345394cd88053d0debb0dc8f7de95308882c6538ee9a3b359da", + "solution/solve.sh": "7d4a0504e1686c1790bfbc074d4e598f4c631f54c2bc5f3d5fd786574b4810b9", + "task.toml": "4b620211dc3d6b1b989f0210c5ecd2f606974581c8bce27bd78659d48cf0cfbd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "009180a7092583fc8143bb3ffc3957923aa0d61d0aac3a471f76eeb4aae54a88" + } + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "file-operations", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + } + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/fix-code-vulnerability:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "87f28dbad183c3a634887696d328ca68a8a0858f132bc48cb7341c22267d38f7", + "environment/Dockerfile": "0917470986224df4daf4854a71302ae81c73ba5dd6e25be1232d29b758a5b3ab", + "instruction.md": "89b1abdf0af19399f720233b5cdf6b2e8e0ff7b6e3fe3e2a3af0d4d298a8641a", + "solution/solve.sh": "7a0131b369d83ae8546ecc3f9b4e23d38240f1bd1f0941e4430fa85b5182ee03", + "task.toml": "9aa2bd14ce9408289d199d0608bc3fa65b8b5e23374c9641020de62197dbf1e2", + "tests/test.sh": "f1611da693cb4cfc5e8482698c6a1858a5c67220ae9db89b50aed66584e9b17f", + "tests/test_outputs.py": "c006a5de95b69bd78055aa26cc055734dbd50718ede25d331f767bfee4ef7f26" + } + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/git-multibranch:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3b9473429381d9eacabab498cb4a8915aa2bca24fa857ef862e5991d3f81f113", + "environment/Dockerfile": "88d9a9f4fef1ae7fddf632799dedf5a2fb6489460a93b610c33dd8b122982821", + "environment/default.conf": "a3a4c5d0ac9cfe79b0510e4225c80fa3988c27fbac13622505886bf66449e04a", + "instruction.md": "54014a38d7526d65faeb6c11d9b7ed3d4a5cb5d973d68d3da3b68b8c0289e125", + "solution/solve.sh": "300ca96b79db8d77f14742109aaf76048aafc4644ced750b1bc5c2761e8d0175", + "task.toml": "4b9981604578a31205257cca36a28d0c89c9d8d1058541330acd11d3f8d65f82", + "tests/test.sh": "ddbde3eea6ffa8fda105cbd86597a53bd409f41e13223900aa27e77a84b6b155", + "tests/test_outputs.py": "f03c6867663b8e8fb7899df51ee4df26411fab62ed491753964b801e591d16c5" + } + }, + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/kv-store-grpc:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "51fdc2ad06f0adb1270164f247ea41799b2ee2ad15490e48ddbf5e79d3767aaa", + "environment/Dockerfile": "47c9982789092437f950736110b2bd18899c39b8fd20f3af3c95b848e564a197", + "instruction.md": "f4b3facc199d5b1c78db6e603ffa29ec1b93a77ff7d291e1a92b42e6bf582bed", + "solution/solve.sh": "6bee2938ac9731d1156b0ca02b28c3d5793bf2cbe5f995c863d397093af01cb7", + "task.toml": "6ff0f30a526bfff86a67b0b53439b9965229c0f368167dc1a5d05f34d82299a3", + "tests/test.sh": "c348467a076bf614d0a850e25b2263f85bbf7b4c7a8d344fbd5454a0611a0df8", + "tests/test_outputs.py": "115de2e964bbe689d07930527113422ef8a9e4be4eb958ad84594e3fcd2b1b0e" + } + }, + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/log-summary-date-ranges:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "026000415cb577f3d3d5132902670262f7d5d8f69841903ec06a882292f756b4", + "environment/Dockerfile": "b479f6e987b63a6ec118fa1691b8bb85c437b2a6a0fe63feab5ed7154957fdfc", + "environment/log_generator_deterministic.py": "beb6eff59979e9aa0be63ce8a0f1b0df1178f92871f04ba9f76fa70cf8beb81c", + "instruction.md": "c1d4516ad4ec2238209740eaf01291df9ab48b0028a3fc5df383c6f831453484", + "solution/solve.sh": "9f6a8ee8947bdf1c0f31b530b6194ae998d66841a3b785fa2e246ee840350ab6", + "task.toml": "9a9593673d6805ef3a301ea734e9c4dbaf80da1b37c0fe9d1f2ceea2a8c57e18", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2fc06a6ce1944ad82f84fb437395fa75ed8c0eadd385b92e375c27e5bc9cb2b4" + } + }, + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/merge-diff-arc-agi-task:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d6b48ed0890b12eb68f72c12cd02689bd6bfae2191dd2688949cde6fe77f6a94", + "environment/Dockerfile": "4a03ae6f0c60b9c2f45c33f0413b5aa75c9811870254e3ff1968fd0119173fc6", + "environment/task-deps/bundle1.bundle": "a759f9b6ffc1a2eee50778d72a1899ad7d724becab84fa30e7aac71ee9e443a5", + "environment/task-deps/bundle2.bundle": "75e3e3916a01c980a03715de67b16c729e03661d94a694b5f4223d229bd6593a", + "environment/task-deps/examples.json": "5426569e4ea078bbb74039ec712a2d4d35f1f9fa3a5f26bf2a0de4a2c8ef5db8", + "instruction.md": "2d826479873e5ac1c94c1f802763bf85c22bb04624384685ebdf6b153fe0278e", + "solution/solve.sh": "48d4becc41fda5b6063fe96947474ffe778a29d6924d7b2bb235eb61e5ce555d", + "task.toml": "6ced40e06b43b169857509765eb7f07791db558f9437cc7367d2536771c2ca5b", + "tests/test.json": "20f116a6b718bbf4874bbe203d15a5c0f85a1d88cb622e2d7529c7e4fb2d4c28", + "tests/test.sh": "c9052954cd6ce25c4a375a2fb48e765b1efa5b39f7991b93993cc793af8ab9d2", + "tests/test_outputs.py": "2754ba6c5c5f75c9db5ed2c5729d9f5d7e21cb7fdb18aa0c65abdc3e7e85c149" + } + }, + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/model-extraction-relu-logits:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "mathematics", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9465accba6c57e5de9a196eed776a16eeb0852a8afc18fc3f8979e2de71b0c7b", + "environment/Dockerfile": "598b1f80d3b7194e97cbf02c8d245de30e303909e0cbe6465125d1d3179fddb5", + "environment/forward.py": "af1a23efd8e4390d16a29777c28116b2f8623ae3f7b628fe14db9f94dac26f5b", + "instruction.md": "c9e179dd099065a34c362a865b0eb9fab5f7d96be582cf117e7519580c036ebe", + "solution/solve.sh": "4cea5094f074ff21453d4423aa1b415a104ed90caa55a4acbb4f09941a3994ea", + "task.toml": "eb64dfaf8ae88868911ae28855cdc4f767de464001e2704919015b595be6dc29", + "tests/test.sh": "3b2d8f5e0353c2bab3515d6387468a863a4f9f241ee131c73f545f3fdf9e2d52", + "tests/test_outputs.py": "e5189c7eb9942ede55e1b48a80388590b80a486b75497613f257b3373ad9b2c9" + } + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/multi-source-data-merger:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9bf37162a4c1b88ed93627830bb8512ebae154325d7cb069c883144a0e4c69d6", + "environment/Dockerfile": "c95a40d3d35b473dc15ae1e8784ba7fb1725805c0583fc574dd2764325d75e4f", + "environment/data/source_a/users.json": "75e51e2d3ab0bdba2269d7cf3757576f84f3ee2c13407d2db94289a5958a1778", + "environment/data/source_b/users.csv": "8e87d16dc47d166edd8e56165955bad265b630d53d02762367c816affb5fa760", + "environment/data/source_c/users.parquet": "23d1ea51c4170bdba53febb885a0f495e28de525a52667347b2932136b9a72fe", + "instruction.md": "d1d28bb6e37f7a255a1b7f78aae8591f35c0494f33e4913392424ad954aab944", + "solution/solve.sh": "1b3d0d50e31930d042495d48f4c8e95f1a9c1a979c803b7f7cec651322537e1e", + "task.toml": "ee3964c0ee040f0a0cb121cefefbc15443d127684acb003fa5b61dd42f31c694", + "tests/test.sh": "d5a00e64413c0e1ae452f38a1aa1a47c3ed7b4416bbfccae1a4c053025a336d1", + "tests/test_outputs.py": "a070888930cf829aeca039771260b1386bda990ea805f313468d381ba3608ca5" + } + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/nginx-request-logging:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d79997586d1c14ed8dbfea050e3580b25500e04c8345acfd3f294152433c795a", + "environment/Dockerfile": "43003466d89ab83484a3fa618e35b58cdb74a002b1e8716ef30f12b3ac8302d8", + "instruction.md": "26a7ac98aced6107f147206790ada77bbf8ed3c25fb360cf323d5fc6889edf99", + "solution/solve.sh": "2454bbbf7c201a9a778165f286373ba2cb991a0eb1d026a8005d57ae71250dd9", + "task.toml": "b7a78d00be13656f75069197100b7b8c4fd268decf6f78929c853b3222a2729d", + "tests/test.sh": "27761195f53cbfe5941fe39b573631f6c926869e64b7c43220377e1e116528e7", + "tests/test_outputs.py": "045cc716c14efde3b0dcff5fc7c85ec5d18bfc6ce66f8b40a418fa2a3a4acda0" + } + }, + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/openssl-selfsigned-cert:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "5dfa88651890cda06fa2a25babd3b813818767635e88bbabecc8305d5bb3942f", + "environment/Dockerfile": "f590b523038be43afcaf5291d7b9e8ed1333c48d9994e77b8dee2f2c352f0f40", + "instruction.md": "38054a3371f7c2c853478a2f9af0bff858437d5b28a9a07483238f06e566921f", + "solution/solve.sh": "58cf3c844e907745cd81fb4bd75efb3d080e2f0e82b589c56c954ae6e1756510", + "task.toml": "75c8913f0fa15890786c11999cde920b8bfc10d4ce8a14edf1e76985a2ac5800", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "90970bb3c372e8e7fb6bb9ac5f649ba73eb38e1a5ea2e922a9057a72b6b4192d" + } + }, + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pypi-server:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "906a096a2cc7b9afc738ff5ad5fbd605be65c103f5069c6673915223b9b187e3", + "environment/Dockerfile": "17a4e67580e4f2c6a151d04b9d74eb020666e7eec8e5259c4a5411f9e93858d0", + "instruction.md": "1c537e4a311a8e71155e121f57bfc03f742fbfb99d05314cd03e433996a28a31", + "solution/solve.sh": "1143c0eade4932ec3aeaa5013fb5e23bbcccca15dead294aa03b6af88750775a", + "task.toml": "07c39d7a733c89adadf74bb63a3e906c3694feb6c490d5fc85ac187c003fd447", + "tests/test.sh": "d2b1220e42bcfdcce87907ac5bbd69ac210a3c94926545ca2439636786e7784e", + "tests/test_outputs.py": "82c6351722e9aa984ff1d701bf57df3150ce51466df2b70afa1aa62d2650be4c" + } + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-cli:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "35a093fa5f1592737784ba718c2e3e9ae17ff8b5e53e003fae560ddce091039c", + "environment/Dockerfile": "8fd5c0f21a6914cd440d10c84c9db06fec606f20dc3e0363b17a3b8c220dc4d9", + "environment/task-deps/cJSON.c": "0e7995261640c96db16ed2c91a5d981efa917cf5e72d0f97ec45a1179ae3ad73", + "environment/task-deps/cJSON.h": "0909160541bcc510e0b2d8c7a9cc051fa1459a0f13be4d8e56346a15491f2f2f", + "environment/task-deps/image.png": "f4d5dfefe463939996541404340e7099d5957b56e1df3760d13b5d2de754db53", + "environment/task-deps/lodepng.cpp": "67873c5f6272f779a143a9c81070e8783327c5552d0b9de2aaee2cb779308917", + "environment/task-deps/lodepng.h": "b735b7e0357319973e9488d520d7f270d734a6e9e1bf924fb1d5b8d26d7e97ce", + "environment/task-deps/model.pth": "a32103eef09936e586b1a7dc7e64d279500a38ff30fdd6df2167e28eb954eee6", + "environment/task-deps/model.py": "e4d3a5c728449656d0efde7871ee227619348352769795ed1be1a8b7f835fddc", + "instruction.md": "ce36ee1f6c589f81ee6a6be4967e25c76a2ed697006928023f42dfb9f28a6750", + "solution/solve.sh": "3b34e561c9ef0a53c9948655847846eb4811aaac67b990480aa6aac1b04bfe66", + "task.toml": "fc1c018efbd5d0537ac0741d53d1ebad3886faa51d14cbde8ec953856411bc0d", + "tests/test.sh": "43fb67e537c7adc2df11c2367f47c876cb5d16a7c1ae16d0339c3a3364bb2b06", + "tests/test_outputs.py": "370702c57a4e0c85f70ee37a5c357ae3a5df8480a275bb6263b0376f2f6c093b" + } + }, + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-recovery:20260430", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e080308a05703b40edd16f53aea41a0e6b9874750ecee35bb432029d0d44bf53", + "environment/Dockerfile": "8a6b7edbb9e7c241e447f15a71e13e5d043b25169be2fd9802b84f5be3d56eb6", + "environment/dataset.pt": "6db65e4ec1c0760209c2dd997ef582e4a1519ab38e0312339999c460a696960d", + "environment/weights.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080", + "instruction.md": "2bef07dc28be3ab2c35ae389584034c2e4dde92c2c31b3eb6bd829628c8c3f4a", + "solution/solve.sh": "175a7802777b43e0fa4f629d71f7c68fb40706813c342aaddc72108d89bd5b2c", + "task.toml": "851a51204f53482a31f47cdf8833128cf421113b0331011286940fc857c37dfd", + "tests/test.sh": "6005f3a75ccfe00d7bed1005264b4a84a1f8e30cd780f8a97b3a804be7cc7a8e", + "tests/test_outputs.py": "5d4d57c43e6913bf3d0081da764ad36d1b7275e3bb114a05abda529afe563905", + "tests/weights_gtruth.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080" + } + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/qemu-alpine-ssh:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "qemu-alpine-ssh-v1", + "task_file_sha256": { + "README.md": "72f1c83fa1e3e6381ef85dd82fd109ee18cdeac090311cf3212a0a8cff0a32a5", + "environment/Dockerfile": "015ce0601ff54758f66b33f4952e0d3a577d84c09aed8f3ec5945a570e8d168a", + "instruction.md": "8348c60490cb90bc690cba259dfe98996da4c3294c6d42bba1119b00c64de4de", + "solution/solve.sh": "1cd31186c427b91bb00499d22aa4f0214fc1ca93b40b6e4cfa969611b848db2a", + "task.toml": "10a43ec6ee732eeba852c377f12e23178f7fa5a4868143f3699a15565ffb64c7", + "tests/test.sh": "3b244479776c4ee3639631112799f2cde22cbb80ac10511715e3a8f32909c289", + "tests/test_outputs.py": "83ccb4bdcd259b342212bebefe395f289858b4dc55b2ee7ae0dab238ad2e9718" + } + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/query-optimize:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a82c53e333a0ac0b0b410b5e26cb8e7305dece31db84d340248ffb13d5a45160", + "environment/Dockerfile": "064818771f200e0517d22068f87229a7a95d0a82b0778c84ca7203bf1926a0b8", + "environment/task-deps/my-sql-query.sql": "43382792f0f438625d643e34abe144febd5669518f4ab1ab550a15c6d9314a21", + "instruction.md": "473e12d320d2732f679fb631c2a4dab4a01deea6775686718ae0c417b8a39447", + "solution/solve.sh": "0d95f3cedb287ff9d47c796cbaf812c300d1222c10fa80c49e8acb0e81eb6978", + "task.toml": "6e88f2a86df699594bcb1bf81df4ed694505f79c9420f671335b02524bbe3868", + "tests/golden.sql": "4da1885d56b3ffa7b7b22b422996d018795611f6963499d551582881b74b84fd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2068ae80c57af8d52a7ae510f0666732c057806e556b480970f36f14bae683f6" + } + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/regex-log:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "29ffd98c55dbb6f4844f1b9d8c1769b2155c117ee475e0df6bbc5ef330d239b3", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "4f7ac05e70cf9220ea0f1e5a052c5f908cd0fa884e847d80b0bd51bae2e96f9c", + "solution/solve.sh": "7e670d4f2b4bccb1e4db38f2a173e085ceda028c38167912b466b0a84fcc0999", + "task.toml": "6bc98525852755caf1fe34c174c2f10f4328adb4ce6c1968dcafe1c877e4e160", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "345c3bd09ab6f6fe8c8361a58c0a47bf0a13b3fcb38a5ac7824e44ff855e8f72" + } + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-pipeline-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "297ddea07c230dfd246a8ab07337dd7abcd8e5e2a2b2e26590cb0a8af779839c", + "environment/Dockerfile": "c357ea41e4a4013c5c5980b8fef04af6a5ae250ccc6697cf3bad3127401a931f", + "instruction.md": "989bfcf0c54baf6c8cdc391743dbb2f9e858c5779dd85112536875d4d1b2dcd2", + "solution/solve.sh": "787198e4178d88b887697aefdd29819f29435ffdfe356545d92f58a30fb38b70", + "task.toml": "8b4f9f525c08afbbc4eca9168e46238a61ccb903486773e9ca4da64bb7ffe857", + "tests/test.sh": "cc9fd28b6590993fc37e4a99da771cbf04726c8c62fe308e802612241da69dea", + "tests/test_outputs.py": "23574246a407163a8e6581c84287cc2f1eb745efbaad2a317a5a28c39958c64f" + } + }, + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-tensor-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "4c1b3c3be30548f1f94c6f5503b1bc74b006af7f192dc4cd026f411df1a54604", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "13bde2ab165d72a311e2bf066d5d6c3c8761bf9d3dd39d1b23eda23bc546b910", + "solution/solve.sh": "d083b7cbf9dbb6cc2ff38c8f2acc150f0059c04aea42d6c2183aeaecc5a19385", + "task.toml": "72fd2a5500d1fe8795dcae76cccb1a29fb25894881a7475f2fa519c87d425a8d", + "tests/test.sh": "a7e6c9408b2401e4e7448371f7ddc72f3284caccfb56277d3cd4e98051868cdf", + "tests/test_outputs.py": "4ae4285e15fba1fcf13d7b7c0638ac8c2e2b0797c937abab1fb6b136fd508290" + } + }, + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/write-compressor:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "603015a28500a28f0838fea2c43a5168613746bfd07737920f4bae40b5a326f7", + "environment/Dockerfile": "9a583d29d46818f6ff2be7c6a0879d03343106e5376fd5a48eb5926b402bb93b", + "environment/data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "environment/decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "environment/main.rs": "fa53d6572fca95903e30d2d953f1e428bb8b63513ff218ebee9bc60a425ab389", + "instruction.md": "c4fc3e73e44deeac95f1cd4a34a2d87e853b22ceec2b2824767f5f087b470bb2", + "solution/solve.sh": "01550b7de7ab1320b1898eae055226bb7c73bee259c75102d2240c974a730409", + "task.toml": "6066a509066433c97c417fea68c4e3e73996f698dcdcbabe07718835eea355a6", + "tests/original-data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "tests/original-decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "809f486053e3ffbce852111dee1f63d80ab31297bd33172389e50278bed384e2" + } + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "native_agent_timeout_seconds": 1200, + "work_budget_seconds": 1020, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/constraints-scheduling:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "personal-assistant", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d54a690970149e1216bd4fc066b45069c7948c13e3b1ef2a213a74342805f209", + "environment/Dockerfile": "7d9aba072a4a907229728b9739c8cf821fc2c98450babd64452682caf3759c84", + "environment/inputs/alice_calendar.ics": "5a5d45804bffd339c2479d49e76fe817b38244d14d389afe1de583443808ecbc", + "environment/inputs/bob_calendar.ics": "c97e85dbcdd756758f8a80f4db5fd5c1dcacee79298e4d2262f46de5ff680b50", + "environment/inputs/carol_calendar.ics": "32a0f8687637df0a12b8e5c56d49c4192bc3e790856943632d70f237c7c1363b", + "instruction.md": "342def78177f9bc41bea4d11af679bc385a9dda2ac8a1e425be384a5b7f937b6", + "solution/solve.sh": "802d81ec064fad3050bbade95a2f847ac24e0b7f80fa69b0bcee0347dc92fe77", + "task.toml": "7b9842de16d74e8faf8c7c0aa03b0b81145941cb12dd322d81ae23fc48607250", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "1422b345ba3aacdfc4184e962811b6379988435896c50d8cf62903e70990c5d1" + } + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/custom-memory-heap-crash:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a06ed23a2c467aa36d0d4a88e5c020b1076cc25b8d483517239658e9c596dde9", + "environment/Dockerfile": "6ee81e5a4fa2540443d7f27d577ba840728fc6aab9f301ab7919103673c78080", + "environment/patches/locale_classes.tcc.patch": "16951f4a4df6bf253886ac75f5a39fdbc328e415194493f63cb7589a364aa817", + "environment/patches/locale_init.cc.patch": "e97f133ceebd86f8881f972e0e45b0ec0a13a9bfa428d8032008707683d54292", + "environment/program/main.cpp": "b94538c76887cc2b2ab739b5ac4731cddd0faa8f4a9892a6265c062bbbeede14", + "environment/program/user.cpp": "92b203786fed1bf0b3aa1675c9cd2d82b2651170675064da8ffee73b98ef14f9", + "environment/program/user.h": "f43d155995a56c22b9cd1417daef7c3b5a976cdc662a98eef0feb438ae1eee77", + "instruction.md": "53c68ae14a3c3af9a97c40bfd9bdc30a5d98b37eaa2f69f763475745f70f108d", + "solution/solve.sh": "b0b1d10bea1cd5be14a82b35d522df616df1e67028f90d2a6bef90aaa98511a1", + "task.toml": "ed474be1983e212913aa8866ebb66216acbe36aef2339ac375d232aa46617481", + "tests/test.sh": "70a2b55f0c5757f6b123a216dfd202be4a646680bced28dddcd31fd8959a058f", + "tests/test_outputs.py": "4e1cb33f777fa7fb5e1536f9bea9eace6deccd515c67d4f87570adbe692914ba" + } + }, + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/dna-assembly:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e7855c23d9822f688eec4dd89ca0f1caf91c6a69e1c8f3c9eb544b4ab2a94821", + "environment/Dockerfile": "26e9e43a98f0b1d132bb638f44b7dc882b0d0ca32d56fa33048adeee00ab600d", + "environment/sequences.fasta": "ebdc361b5fba28eb0a2c7a8a8a444ccb12cc842c8cbc6df5a2c24254f130736f", + "instruction.md": "88998c98beabdd966ab46c04ede822bbc74796696aa1bf87c567a9131210533c", + "solution/solve.sh": "c976cb118c8ef1befd846ba05fd5cadb1b1f529bc76b9f2ea9cb3049e6e11a47", + "task.toml": "7b571544b64d9be1c91526837e5d805c2181afe0e6fd57a7f0676bba36c868d7", + "tests/test.sh": "271c8317c98539e3b49233d017fe51fb0e5fba8114e76830fe698565499b2b1d", + "tests/test_outputs.py": "5c2d0fd2e5f6303263e9750f67acda7db491ef04841832b4fc62bcfb61b08924" + } + }, + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/llm-inference-batching-scheduler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ecc9dd635357c184742cf72fc1e452a8946e207b183fc9a7eebbcf07133b911f", + "environment/Dockerfile": "10a6193904ee0022ac098504182989893b196dfc74e626a3ca48534a8eacde48", + "environment/task_file/input_data/requests_bucket_1.jsonl": "8b3cf1e2b06074162038064dd2c114abe308acd32cec7d365ee63ee81978f7de", + "environment/task_file/input_data/requests_bucket_2.jsonl": "e15e2d4c4d55f875c42a6042610d6282f6c0657c39d3e0d0f087268252525cb5", + "environment/task_file/scripts/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "environment/task_file/scripts/baseline_packer.py": "547d230d5e6d93197803480cb81cab0ec6ac63fcfa0a24b246f549523f216c4e", + "environment/task_file/scripts/cost_model.py": "0cf417fd5ff0112defcd46970d37a290c0763c5a9092e91cb3ebb86c3f20e6ce", + "instruction.md": "0817374767533fbf51bb93345ee1bba266263c216c6cc55c096e71b4a1d37dfc", + "solution/solve.sh": "cb9efb586d76a893c3027e62029e286eb6c7583dc92dbf1840a3adc236a926d7", + "task.toml": "52aaac942f917e6a91ff658e0272195bd1cd842d41d71e302414c52603cc588b", + "tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "tests/cost_model_for_tests.py": "d3e810a99d40327fb8317f6512d879c52c6aeebdf5dcdc0c5e117c7df61c9ab9", + "tests/test.sh": "005f980332355e2fe143d4ff4aac11828adf070594bbe7055399c238f06953ca", + "tests/test_outputs.py": "80b06b7c4ba0db28832b767d48be74bb462d7f5df5798d9cb58c4f1160d96ffa" + } + }, + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/path-tracing:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3e4fc952b3f9a860c7ee7837b3f3c686f1283ee7821b014265dba7c94d7fc849", + "environment/Dockerfile": "7aca02e4e20b04ed2269d627a29dd4e7337a02764a9652b641452bf09836a3d5", + "environment/orig.c": "665c1bde0907740890ac114d6b36af1f3fd00449fa287583817f778bb5da200c", + "instruction.md": "9f36e02d636f5dc0946619bc518f2e51c24fa75251da9d1f0ca9eca612cb9572", + "solution/solve.sh": "d107abde101eecee6838da80cc52a3d480c93956fff3a9343a299ea01b3fe9e6", + "task.toml": "cf47013f69bf1b43242da984f231d4daf364fdf44d3845f2eea2ca2cb9308c46", + "tests/test.sh": "a719dbb1ba7ae76df095e9d815941dfa4c55edaf11f173b9c3c3097afa838548", + "tests/test_outputs.py": "572e68b2843191d29592faf42a405861b7958d8e894868b74c5a8886b53eb879" + } + }, + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6", + "native_agent_timeout_seconds": 2400, + "work_budget_seconds": 2220, + "verifier_timeout_seconds": 2400.0, + "docker_image": "alexgshaw/schemelike-metacircular-eval:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "095d3ec6dcc0fe0c7a7e92355456dcea6992f91d1f8d397669b8cfe18475f506", + "environment/Dockerfile": "9b3df76222e16aad360fbd34cb0f71a0be8986052c065b10b92c0cdf417b25c7", + "environment/tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "environment/tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "environment/tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "environment/tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "environment/tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "environment/tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "environment/tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "environment/tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "environment/tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "environment/tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "environment/tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "environment/tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "environment/tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "environment/tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "environment/tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "environment/tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "environment/tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "environment/tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "environment/tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "environment/tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "environment/tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "environment/tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "environment/tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "environment/tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "environment/tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "environment/tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "environment/tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "environment/tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "environment/tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "environment/tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "environment/tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "environment/tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "environment/tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "environment/tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "environment/tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "environment/tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "environment/tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "environment/tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "environment/tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "environment/tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "environment/tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "environment/tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "environment/tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "environment/tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "environment/tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "environment/tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "environment/tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "environment/tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "environment/tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "environment/tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "environment/tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "environment/tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "environment/tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "environment/tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "environment/tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "environment/tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "environment/tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "environment/tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "environment/tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "environment/tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "environment/tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "environment/tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "environment/tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "environment/tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "environment/tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "environment/tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed", + "instruction.md": "3af5f9567b62f93683db7518ec6fcb609f83796258aa02166b5a7091c5b69031", + "solution/solve.sh": "e2c5b0ac2bef4801a1891e13c108813381877426c1a7f01bab86b4fc1f712510", + "task.toml": "8dae8eb226183b0ca92331c1106152a7abc93b03b4e5a95291b86a4ddc9b44e0", + "tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed" + } + }, + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/caffe-cifar-10:20260403", + "resources": { + "cpus": 4, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e918e6852c5c034422b81a6be22d7596fddcee53abe088f56727b2482da32728", + "environment/Dockerfile": "b3b8adc915a1afa035b604273af3bc1aa8b90999d2dfd72763218b34cb87abcd", + "instruction.md": "46907b0ee048f74269c29a15e0e086546b2b1220149597017da8b5ccc05a0cea", + "solution/solve.sh": "771887de1ed557269c7f3160c777519aa9538e1e9b59771b2f6ef38c8952490c", + "task.toml": "2831165ad9d9e8bbc646e6d85c0a2290640dcc4f13749547a2937c7675012a8b", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "e99b9f6284e923388fbc21488b1af99fdde46d4a809d3557b0788dbf1cb6b281" + } + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/circuit-fibsqrt:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "201f7c889dd6f73dd83ca078e347272b33720fa622f8c757ea6d5770453f4cbd", + "environment/Dockerfile": "780c23986778f3e5f842567868af30b1135ed095ca9449e1c90b7f0e0483d3d4", + "environment/gates.txt": "032786f94f16605767c62bb61ce4dbba7ab7d3f1046712c975f01c9a04b6ad29", + "environment/tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "environment/tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438", + "instruction.md": "a7f500266a878d04ac9de39fba76220cfad216d8f6b8063bba93bebb3d4ab9ea", + "solution/solve.sh": "a4ffab4183815a0eb7646abc1de1fc67695014c060d7ff16e8bca9ec94385d0b", + "task.toml": "9f5aa1f7ebbbcec90fd109121e9bd509af3cfff6b4cec701bd91947988f40972", + "tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438" + } + }, + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/mteb-leaderboard:20260430", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ed147a5bc31e9865b16848b2f01d2f9b1e677af57a7c632d7ea2311c0553ca76", + "environment/Dockerfile": "271fdb5b34520c0bf2cb6462365363ec52058f127cd9eea3f5da2589df5c1044", + "instruction.md": "06ea44106eca0405c25bab00f5b3cfeb611c872093f6c0f82ed057839d1d13de", + "solution/solve.sh": "4ed68831d4acf5d80d2d73eec1f21efd0896a6f51f23020f572cfc0af6da0410", + "task.toml": "86574bd1c6c4fefd2e64eee9c2caa53d87c4bcf2e51547912440a0ddbd0badb8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "b28f8fc8b7614801305813c402be52f4be1e75de6658ecdfdf9383aa793892cf" + } + }, + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/regex-chess:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9296cbee592fc996c58d228b51fee303647e5ecfae82d5666d42a40cb30ec355", + "environment/Dockerfile": "2616cd775252303839dfda0ee2f047eb862952b053b0103bd6b2460ef90c11b7", + "environment/check.py": "f9c416ae9ea01dcdd5f6a43170c8434b8c0a0b389a4a30876a2048e277d191c7", + "instruction.md": "a6ed3fcc54fc92d71a84af921942422d299ca89b410dcd92cd05a53d0d06436c", + "solution/solve.sh": "b728bf0ea392a3d244afbd0818d36933c8d3e190e47743c1382cd518d518a2e9", + "task.toml": "07fe52636fcc4bc9fff25d5630af145b97a5331219d719cc8decb4ca67ea21d4", + "tests/test.sh": "8f7f00a1c4dde7fcca6c918c768f2aba432b8c4007c15989716c68091950da87", + "tests/test_outputs.py": "4983c44d21c56dd8191381eff0304afc8b87bb714527ea9036b3319c015a30b4" + } + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "native_agent_timeout_seconds": 12000, + "work_budget_seconds": 11820, + "verifier_timeout_seconds": 12000.0, + "docker_image": "alexgshaw/build-pov-ray:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "38380f64bc631916e264f80ce4d37feedbc783310001475b367d03989d114fdf", + "environment/Dockerfile": "44f2b5fc7311d8f336daaaca99dd4216b49355e67f50b494ebacf1b0f7c14e92", + "environment/deps/illum1.pov": "5a1f7f53ee0730c4923fcd21cd4a9c34719024d8557920441551ba7d96e75dc4", + "instruction.md": "56c8ecdd0084a07f8c97986c43af5c37ed03dd5aa994da6735cf59806162d1b2", + "solution/solve.sh": "3da9dbf589656512cd7a7004dc3752d3552a6af939cfe59eff79f91695fa42c7", + "task.toml": "b90a47ec26c1191d3c523cc094ea96b170e42fbfe176ef1f814c57d05390c097", + "tests/reference_illum1.png": "bb084f430bbc9d5b623a656920670dc06cee6d9132a9d550be4a1b99e41ab9e0", + "tests/test.sh": "928d1497ad21aef0c456294e3784f1fddf8493410f23584da89f9e02ec6bd134", + "tests/test_outputs.py": "64d6ce107503714ada346e5606832ee886be9950971cf81523e73d8ba28e8d37" + } + } + ], + "total_model_runs": 34, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "preflight": [ + { + "arm": "main", + "job_name": "dev34-main-install-only-v3-20261012", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/install-only.json", + "config_sha256": "dacd02bf872aac235b3f0a11bef00635ff1a7f0dfcb7313884a92419f3e46137" + } + ], + "queue": [ + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/adaptive-rejection-sampler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/adaptive-rejection-sampler.json", + "job_name": "dev34-adaptive-rejection-sampler-main-20261012", + "config_sha256": "d1359abf44d0c0216525ed35b50e8c42697efb697948798f52bd157fedbde0eb" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-cython-ext", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-cython-ext.json", + "job_name": "dev34-build-cython-ext-main-20261012", + "config_sha256": "53bc1b1b9e16465e1cc3a3c96bf6c3e74fe2f534028f08a7fd1e8bbcb1e7cef1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cancel-async-tasks", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cancel-async-tasks.json", + "job_name": "dev34-cancel-async-tasks-main-20261012", + "config_sha256": "72f9a02b91d11206cc0ca6430a104b8edf98a9334e9f1be071d46d7f0cb329bd" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cobol-modernization", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cobol-modernization.json", + "job_name": "dev34-cobol-modernization-main-20261012", + "config_sha256": "f32e7f307a259f47bf647447430f1713b42a81c6b64ced1166c5ee642583cdc1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/db-wal-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/db-wal-recovery.json", + "job_name": "dev34-db-wal-recovery-main-20261012", + "config_sha256": "6a2cf1733ab98905ee2f783bb78d724a90b7f2abd22ce17e41626db3136afb91" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/fix-code-vulnerability", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/fix-code-vulnerability.json", + "job_name": "dev34-fix-code-vulnerability-main-20261012", + "config_sha256": "ab5fb163861aeaf2313cea0258631d11a1e959f1eeba4365f415f09fb0c48c94" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/git-multibranch", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/git-multibranch.json", + "job_name": "dev34-git-multibranch-main-20261012", + "config_sha256": "7b09493aa6d8200e7ab546a08f09fdc42bc3c12c669301f9208ae34740d774fc" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/kv-store-grpc", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/kv-store-grpc.json", + "job_name": "dev34-kv-store-grpc-main-20261012", + "config_sha256": "0f08c53566a5a9d4ccd84ce25828eb1728347b19a57431bf2ffdf36879ebe3c9" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/log-summary-date-ranges", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/log-summary-date-ranges.json", + "job_name": "dev34-log-summary-date-ranges-main-20261012", + "config_sha256": "30a4182133ed108fc8f5b10d07b25e0ed652feeb3f86dce7144f523de34f47a4" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/merge-diff-arc-agi-task", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/merge-diff-arc-agi-task.json", + "job_name": "dev34-merge-diff-arc-agi-task-main-20261012", + "config_sha256": "06808503acf8736a2e486b145526faede9f68a12f90d3b90f7077d5a2acfabf3" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/model-extraction-relu-logits", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/model-extraction-relu-logits.json", + "job_name": "dev34-model-extraction-relu-logits-main-20261012", + "config_sha256": "fcc810f03fd8340adcb1e2a6da7431074df24b880ea9af1aa94fb19a5de1c680" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/multi-source-data-merger", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/multi-source-data-merger.json", + "job_name": "dev34-multi-source-data-merger-main-20261012", + "config_sha256": "7d3cf5080e8a003b3cfffffb0b1bcadf3ea9af6d758496fc972bae30233e4880" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/nginx-request-logging", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/nginx-request-logging.json", + "job_name": "dev34-nginx-request-logging-main-20261012", + "config_sha256": "725a2695d1912ea68b82452910f4b64eb1d3cd9febe0282a4f02561968853fc1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/openssl-selfsigned-cert", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/openssl-selfsigned-cert.json", + "job_name": "dev34-openssl-selfsigned-cert-main-20261012", + "config_sha256": "10f28dab8a3b522416eecc04e550f70650c5ab2460631de9629e6aa86741f3d6" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pypi-server", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pypi-server.json", + "job_name": "dev34-pypi-server-main-20261012", + "config_sha256": "8874892c10910d72023c26fa3bed51085bec86fa7f82c1030f891215dc63920c" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-cli", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-cli.json", + "job_name": "dev34-pytorch-model-cli-main-20261012", + "config_sha256": "63da23dc3b29ee0473fd2c218facbbdee180316f23687f054d1021452c3b4f3b" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-recovery.json", + "job_name": "dev34-pytorch-model-recovery-main-20261012", + "config_sha256": "56689163c0bc7c6c37a70ef188f5b1ab84607cb948560c0956c749ffe93d90ab" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/qemu-alpine-ssh", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/qemu-alpine-ssh.json", + "job_name": "dev34-qemu-alpine-ssh-main-20261012", + "config_sha256": "530b95a3fdae98bd545f901566ce8de76369e27d6ebacaa305b08460a594ee76" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/query-optimize", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/query-optimize.json", + "job_name": "dev34-query-optimize-main-20261012", + "config_sha256": "432efe9929e524a97dfda535f904770f257e6f03895f5bd9c1a1ece195c1e491" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-log", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-log.json", + "job_name": "dev34-regex-log-main-20261012", + "config_sha256": "cdb02c5ebdfb68e8e8d297f751da02b46862acde4cc528f0916c3f2b261746b9" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-pipeline-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-pipeline-parallelism.json", + "job_name": "dev34-torch-pipeline-parallelism-main-20261012", + "config_sha256": "32c3e39b20d98b03d347bb11eccbf26764ff1c71402174001b9c4162ce2d942b" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-tensor-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-tensor-parallelism.json", + "job_name": "dev34-torch-tensor-parallelism-main-20261012", + "config_sha256": "cf5cb0b80ad4101f7236ec3c62b84d9659510c74a401a7b3ad73df9b28dbcbfe" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/write-compressor", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/write-compressor.json", + "job_name": "dev34-write-compressor-main-20261012", + "config_sha256": "8ea5e2bfb3f6008650f2df369d92d09e1f0dd726823e1966ed588ef796f1fc15" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/constraints-scheduling", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/constraints-scheduling.json", + "job_name": "dev34-constraints-scheduling-main-20261012", + "config_sha256": "69ddff2768f5eb50dc9d4d44a61c3935e18ac8c0b6296dda2949bafe6b8d2f97" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/custom-memory-heap-crash", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/custom-memory-heap-crash.json", + "job_name": "dev34-custom-memory-heap-crash-main-20261012", + "config_sha256": "46e30ac446a0e18cd07b188e54ce3c68fd1f5d87c4a86fbd3af9034e1a9f5269" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/dna-assembly", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/dna-assembly.json", + "job_name": "dev34-dna-assembly-main-20261012", + "config_sha256": "91d2c4d4998553a04a0a0a14ab30895ce6979555c03f8e09bc57fd14b23f7dc2" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/llm-inference-batching-scheduler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/llm-inference-batching-scheduler.json", + "job_name": "dev34-llm-inference-batching-scheduler-main-20261012", + "config_sha256": "0f957de890a4422c33c48ff8bf5411014f2407e91b70a6e6d9901902f9ef3923" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/path-tracing", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/path-tracing.json", + "job_name": "dev34-path-tracing-main-20261012", + "config_sha256": "60c29ff733fa5ae323fa8f8198bcfde093c8052a0d04acb67a801b88e69e7618" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/schemelike-metacircular-eval", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/schemelike-metacircular-eval.json", + "job_name": "dev34-schemelike-metacircular-eval-main-20261012", + "config_sha256": "9b0c0196fc12f6e64399d0ee7ef5ea5614b9eeb2de0e96032031be6b8b48a24c" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/caffe-cifar-10", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/caffe-cifar-10.json", + "job_name": "dev34-caffe-cifar-10-main-20261012", + "config_sha256": "f542b45b03a1e76ccb84b8f9d493315820057d4662017a2a645291e0e1593d96" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/circuit-fibsqrt", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/circuit-fibsqrt.json", + "job_name": "dev34-circuit-fibsqrt-main-20261012", + "config_sha256": "6a8b079ada286f0030203955d4e72b567aac67a73e738b2175708689fdd86075" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/mteb-leaderboard", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/mteb-leaderboard.json", + "job_name": "dev34-mteb-leaderboard-main-20261012", + "config_sha256": "b2a41e98eb4635bd4e5d5393a3eec7658e8f6f6d8a6da4a9219a1096ce7d5ef3" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-chess", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-chess.json", + "job_name": "dev34-regex-chess-main-20261012", + "config_sha256": "f893c8ea89e39a24136ef8b2fc8c673dc4df21c7dffc4dc5ceaa1d2f04ba588f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-pov-ray", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-pov-ray.json", + "job_name": "dev34-build-pov-ray-main-20261012", + "config_sha256": "3ae4e496a88bc7aa457a22b7a20a657a19edaf4b2d4f3adf6863dbe2d0be8625" + } + ], + "log_dir": "jobs/dev34-main-baseline-20261012-v3-orchestration", + "image_digests_file": "image-digests.json", + "image_digests_sha256": "5fc06fa5fbac1cccd3a4fb2f560878dd79ce314a6b1005183bbdcb4e8e13381f", + "budget_policy": "Native task agent timeout minus 180 seconds, separate 1800-second setup. Native verifier per-attempt time limits; max three attempts, retry only timeouts against the same agent output.", + "execution_policy": "All 34 model trials are fresh. Fixed ascending work-budget order, then task name; consecutive pairs finish before the next pair. No whole-trial retries, task replacements, or reuse of previous model outputs.", + "failure_policy": "Keep all setup failures and verifier errors separate from scored failures. Retain native zeros even when assertions never start, but classify them as verifier errors in analysis.", + "preflight_policy": "Fresh installation-only preflight for all 34 tasks before model work. QEMU has a reviewed verifier dependency profile; the other 33 are explicitly not_configured. A pre-model infrastructure amendment may reuse successful native installation trials only with complete hash-bound provenance and retain all failures.", + "information_policy": "Use frozen current main without prompt or runtime tuning. Do not inspect new trajectories or hidden test contents until all new model outputs freeze. Prior development exposure is disclosed, and each completed trajectory will be audited for benchmark-answer exposure.", + "scope": "One common-main baseline of the existing 34-task development pool, not a comparison with prior mixed-version results and not a random or held-out benchmark.", + "additional_file_sha256": { + "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf": "d0e1ffb6dbd02c54978ba2df386326311a59f6d3774990180f22bcc6adfaedbc", + "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json": "d73572545f3c8802b59e974577fe5af749cfb5cbdde0a4046a74258d157c951b", + "benchmarks/harbor/scripts/apt_index_proxy.py": "bc15f7adf1d7968d627ceca4683ab86742a7188a1b1955ba1fef37ba48e8a936", + "benchmarks/harbor/scripts/cache_image.py": "639709c03a7b0c3eb6aa78b4e87e6184b49b4b102853a31e33dd832d759180c0", + "benchmarks/harbor/scripts/run_comparison.py": "04f2106f0099a7b4d77e9206a36767f98a5e92c875ec78dd2de858c4936c12a8" + }, + "protocol_amendment": { + "prior_plan": "experiment-v2.json", + "prior_plan_sha256": "38403feae088a26a23b0f5567e30588fd8ac89249a787fec6a6c0796428acd19", + "reason": "The host proxy was unreachable from container networks; a loopback sidecar passed real cached-index and fresh-release-metadata integration probes.", + "transport_scope": "APT HTTP only. Sidecar shares the individual task network namespace, with no task filesystem, credentials, or agent artifacts mounted. No published ports. The cache uses per-digest process locks.", + "sidecar_image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "sidecar_cpu_limit": 0.5, + "sidecar_memory_limit": "256m", + "original_runtime_unchanged": true, + "model_runs_started_before_amendment": 0 + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v2-evidence.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v2-evidence.json new file mode 100644 index 0000000..fd493e2 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v2-evidence.json @@ -0,0 +1,97 @@ +{ + "recorded_at": "2026-10-12T16:02:56.125848+00:00", + "prior_plan_sha256": "38403feae088a26a23b0f5567e30588fd8ac89249a787fec6a6c0796428acd19", + "state_path": "jobs/dev34-main-baseline-20261012-v2-orchestration/dev34-main-install-only-v2-20261012.json", + "state_sha256": "358046ec5361482e77a490b67fa27b35d9de1b38e2126b1ab76ab5f1addaba92", + "model_runs_started": 0, + "official_verifier_runs_started": 0, + "trials": [ + { + "task": "terminal-bench/adaptive-rejection-sampler", + "trial": "adaptive-rejection-sampler__8GXdagL", + "finished_at": "2026-10-12T15:54:51.869152Z", + "exception_type": "NonZeroAgentExitCodeError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/adaptive-rejection-sampler__8GXdagL/result.json", + "result_sha256": "47f9f35a2402c483670b2d85888969a4693269b5218cb0786ef6ac411d8369cc", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b" + }, + { + "task": "terminal-bench/build-cython-ext", + "trial": "build-cython-ext__bKW8wMh", + "finished_at": "2026-10-12T15:54:51.777332Z", + "exception_type": "NonZeroAgentExitCodeError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/build-cython-ext__bKW8wMh/result.json", + "result_sha256": "19d978b009a3e9b3cd67c77481792a17c27ae51d80e2830c628f02c254334c5b", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b" + }, + { + "task": "terminal-bench/cancel-async-tasks", + "trial": "cancel-async-tasks__Br23k5i", + "finished_at": "2026-10-12T15:55:44.824754Z", + "exception_type": "NonZeroAgentExitCodeError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/cancel-async-tasks__Br23k5i/result.json", + "result_sha256": "b7f9ac46651382ec8bf8f76e6d07631058adf32d5948efcba92255a86ea61474", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b" + }, + { + "task": "terminal-bench/cobol-modernization", + "trial": "cobol-modernization__7zgjdZj", + "finished_at": "2026-10-12T15:56:25.006433Z", + "exception_type": "NetworkConnectionError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/cobol-modernization__7zgjdZj/result.json", + "result_sha256": "b2ddf6e87e017fb711d9ee031e7307edde820da541550a51294c9cc6aeaea173", + "bootstrap_sha256": "079f9d13e4af52880b5a1dd43c0432e9f1208b64fc6da106b3442630ac014e72" + }, + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__zpZVSsu", + "finished_at": "2026-10-12T15:56:34.815586Z", + "exception_type": "NonZeroAgentExitCodeError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/db-wal-recovery__zpZVSsu/result.json", + "result_sha256": "9776b92c1b1fb694b7feb3d6317ef698efaadcd804e5495246a240ba27144595", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b" + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "trial": "fix-code-vulnerability__RHxqE9D", + "finished_at": "2026-10-12T15:57:58.607062Z", + "exception_type": "NetworkConnectionError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/fix-code-vulnerability__RHxqE9D/result.json", + "result_sha256": "1d93ff9e819acd22331252c41fe41c8e0de2d0bf08e4a7ae6af1f0ba0bce9d5f", + "bootstrap_sha256": "7058299ec0c6f256d8eecf9a0da6b205ad7abf779e297bf17c9a230e10bb9ba6" + }, + { + "task": "terminal-bench/kv-store-grpc", + "trial": "kv-store-grpc__2aB6kU7", + "finished_at": "2026-10-12T15:58:49.628790Z", + "exception_type": "NonZeroAgentExitCodeError", + "bootstrap_status": "failed", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/kv-store-grpc__2aB6kU7/result.json", + "result_sha256": "b5abbaa52933dcb7791325b8df07e2b6336733a2c8c8508c4affe2b54c990f02", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b" + } + ], + "diagnosis": "Read-only APT mount was active, but TCP probes from containers to both docker0 and their own bridge gateway timed out. No requests reached the host proxy. Stopped installation-only orchestration with SIGINT; preserved native failures and cancellations.", + "intervention": "Run the same restricted proxy in a per-task sidecar sharing only that task network namespace. Bind loopback, share a verified host cache with cross-process digest locks, and require the sidecar health check before setup. No task image, source, agent, adapter, budget, or verifier edits.", + "integration_probe": { + "cached_index_http_status": 200, + "cached_index_bytes": 8790396, + "cached_index_sha256": "9e0b5aabb2465b3d2e7a7fe27f9913846277833f7a2826e7767acccff5b588c5", + "fresh_inrelease_status": 200, + "fresh_inrelease_bytes": 151075, + "fresh_inrelease_seconds": 2.54 + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/adaptive-rejection-sampler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/adaptive-rejection-sampler.json new file mode 100644 index 0000000..2b94efc --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/adaptive-rejection-sampler.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json new file mode 100644 index 0000000..e17e3d9 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json @@ -0,0 +1,56 @@ +{ + "services": { + "main": { + "volumes": [ + { + "type": "bind", + "source": "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf", + "target": "/etc/apt/apt.conf.d/99nanopy-index-cache", + "read_only": true + } + ] + }, + "apt-index-cache": { + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "network_mode": "service:main", + "entrypoint": [ + "python3", + "/proxy/apt_index_proxy.py", + "--bind", + "127.0.0.1", + "--port", + "18742", + "--cache", + "/cache", + "--workers", + "16" + ], + "user": "1000:1000", + "read_only": true, + "cap_drop": [ + "ALL" + ], + "security_opt": [ + "no-new-privileges:true" + ], + "mem_limit": "256m", + "cpus": 0.5, + "volumes": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/apt_index_proxy.py:/proxy/apt_index_proxy.py:ro", + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/cache_image.py:/proxy/cache_image.py:ro", + "/home/minix/projects/nanoPyCodeAgent/.cache/nanopy-harbor/apt-indexes:/cache" + ], + "healthcheck": { + "test": [ + "CMD", + "python3", + "-c", + "import socket; socket.create_connection((\"127.0.0.1\",18742), timeout=2).close()" + ], + "interval": "1s", + "timeout": "3s", + "retries": 30 + } + } + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf new file mode 100644 index 0000000..e191ddb --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf @@ -0,0 +1,8 @@ +// Route only public APT HTTP origins through the verified index cache. +Acquire::http::Proxy::deb.debian.org "http://127.0.0.1:18742/"; +Acquire::http::Proxy::security.debian.org "http://127.0.0.1:18742/"; +Acquire::http::Proxy::archive.debian.org "http://127.0.0.1:18742/"; +Acquire::http::Proxy::archive.ubuntu.com "http://127.0.0.1:18742/"; +Acquire::http::Proxy::security.ubuntu.com "http://127.0.0.1:18742/"; +Acquire::http::Proxy::ports.ubuntu.com "http://127.0.0.1:18742/"; +Acquire::http::Proxy::old-releases.ubuntu.com "http://127.0.0.1:18742/"; diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-cython-ext.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-cython-ext.json new file mode 100644 index 0000000..8e36110 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-cython-ext.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-pov-ray.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-pov-ray.json new file mode 100644 index 0000000..ecaf8b0 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/build-pov-ray.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 11820 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/caffe-cifar-10.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/caffe-cifar-10.json new file mode 100644 index 0000000..1d8708d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/caffe-cifar-10.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cancel-async-tasks.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cancel-async-tasks.json new file mode 100644 index 0000000..c325e01 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cancel-async-tasks.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/circuit-fibsqrt.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/circuit-fibsqrt.json new file mode 100644 index 0000000..481606d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/circuit-fibsqrt.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cobol-modernization.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cobol-modernization.json new file mode 100644 index 0000000..e4b1dec --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/cobol-modernization.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/constraints-scheduling.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/constraints-scheduling.json new file mode 100644 index 0000000..abed7d4 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/constraints-scheduling.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1020 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/custom-memory-heap-crash.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/custom-memory-heap-crash.json new file mode 100644 index 0000000..7b0dd2b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/custom-memory-heap-crash.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/db-wal-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/db-wal-recovery.json new file mode 100644 index 0000000..16b71c0 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/db-wal-recovery.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/dna-assembly.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/dna-assembly.json new file mode 100644 index 0000000..03d052f --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/dna-assembly.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/fix-code-vulnerability.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/fix-code-vulnerability.json new file mode 100644 index 0000000..3556a85 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/fix-code-vulnerability.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/git-multibranch.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/git-multibranch.json new file mode 100644 index 0000000..08d5d0b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/git-multibranch.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/install-only.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/install-only.json new file mode 100644 index 0000000..769a6fa --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/install-only.json @@ -0,0 +1,161 @@ +{ + "n_concurrent_trials": 2, + "n_attempts": 1, + "install_only": true, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + }, + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12" + }, + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1" + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + }, + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307" + }, + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + }, + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d" + }, + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17" + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + }, + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302" + }, + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1" + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + }, + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc" + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + }, + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684" + }, + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e" + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + }, + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f" + }, + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67" + }, + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c" + }, + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6" + }, + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf" + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + }, + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c" + }, + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78" + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/kv-store-grpc.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/kv-store-grpc.json new file mode 100644 index 0000000..ffb86f3 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/kv-store-grpc.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/llm-inference-batching-scheduler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/llm-inference-batching-scheduler.json new file mode 100644 index 0000000..5ba245e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/llm-inference-batching-scheduler.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/log-summary-date-ranges.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/log-summary-date-ranges.json new file mode 100644 index 0000000..d373542 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/log-summary-date-ranges.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/merge-diff-arc-agi-task.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/merge-diff-arc-agi-task.json new file mode 100644 index 0000000..c574826 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/merge-diff-arc-agi-task.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/model-extraction-relu-logits.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/model-extraction-relu-logits.json new file mode 100644 index 0000000..7a13dc0 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/model-extraction-relu-logits.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/mteb-leaderboard.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/mteb-leaderboard.json new file mode 100644 index 0000000..d98db22 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/mteb-leaderboard.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/multi-source-data-merger.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/multi-source-data-merger.json new file mode 100644 index 0000000..7d0228d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/multi-source-data-merger.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/nginx-request-logging.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/nginx-request-logging.json new file mode 100644 index 0000000..4e75381 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/nginx-request-logging.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/openssl-selfsigned-cert.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/openssl-selfsigned-cert.json new file mode 100644 index 0000000..9e2a702 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/openssl-selfsigned-cert.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/path-tracing.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/path-tracing.json new file mode 100644 index 0000000..b97136a --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/path-tracing.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pypi-server.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pypi-server.json new file mode 100644 index 0000000..fe8ee28 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pypi-server.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-cli.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-cli.json new file mode 100644 index 0000000..e37f862 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-cli.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-recovery.json new file mode 100644 index 0000000..976e8ab --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/pytorch-model-recovery.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/qemu-alpine-ssh.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/qemu-alpine-ssh.json new file mode 100644 index 0000000..6cd79a5 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/qemu-alpine-ssh.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/query-optimize.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/query-optimize.json new file mode 100644 index 0000000..5735b3f --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/query-optimize.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-chess.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-chess.json new file mode 100644 index 0000000..ba0c073 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-chess.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-log.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-log.json new file mode 100644 index 0000000..960657e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/regex-log.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/schemelike-metacircular-eval.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/schemelike-metacircular-eval.json new file mode 100644 index 0000000..9dd49ac --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/schemelike-metacircular-eval.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 2220 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-pipeline-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-pipeline-parallelism.json new file mode 100644 index 0000000..58800cd --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-pipeline-parallelism.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-tensor-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-tensor-parallelism.json new file mode 100644 index 0000000..dd6601a --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/torch-tensor-parallelism.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/write-compressor.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/write-compressor.json new file mode 100644 index 0000000..6b438c6 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/write-compressor.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/scripts/apt_index_proxy.py b/benchmarks/harbor/scripts/apt_index_proxy.py index bc22965..bf98ddd 100644 --- a/benchmarks/harbor/scripts/apt_index_proxy.py +++ b/benchmarks/harbor/scripts/apt_index_proxy.py @@ -8,6 +8,7 @@ import argparse from datetime import datetime, timezone +import fcntl from http.client import IncompleteRead from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import importlib.util @@ -99,7 +100,10 @@ def fetch(self, url): digest = match.group(1) with self.guard: lock = self.locks.setdefault(digest, threading.Lock()) - with lock: + lock_dir = self.root / "locks" + lock_dir.mkdir(parents=True, exist_ok=True) + with lock, (lock_dir / digest).open("a") as process_lock: + fcntl.flock(process_lock, fcntl.LOCK_EX) path = self.root / "blobs/sha256" / digest if path.exists(): size = path.stat().st_size @@ -223,7 +227,7 @@ def forward(self, head): def main(): parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--bind", required=True, help="Local Docker bridge address, not a public interface") + parser.add_argument("--bind", required=True, help="Loopback or local Docker bridge address") parser.add_argument("--port", type=int, default=18742) parser.add_argument("--cache", type=Path, required=True) parser.add_argument("--workers", type=int, default=8) @@ -232,13 +236,10 @@ def main(): parser.error("Binding all interfaces is not supported") cache = IndexCache(args.cache, args.workers) cache.root.mkdir(parents=True, exist_ok=True) - import fcntl - with (cache.root / ".lock").open("w") as lock: - fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) - server = ThreadingHTTPServer((args.bind, args.port), Handler) - server.cache = cache - event(kind="ready", bind=args.bind, port=args.port, workers=args.workers) - server.serve_forever() + server = ThreadingHTTPServer((args.bind, args.port), Handler) + server.cache = cache + event(kind="ready", bind=args.bind, port=args.port, workers=args.workers) + server.serve_forever() if __name__ == "__main__": diff --git a/benchmarks/harbor/tests/test_apt_index_proxy.py b/benchmarks/harbor/tests/test_apt_index_proxy.py index 21e7d6e..23beec4 100644 --- a/benchmarks/harbor/tests/test_apt_index_proxy.py +++ b/benchmarks/harbor/tests/test_apt_index_proxy.py @@ -4,6 +4,7 @@ import http.client import importlib.util import io +import multiprocessing from pathlib import Path from concurrent.futures import ThreadPoolExecutor import threading @@ -72,6 +73,41 @@ def test_corrupt_cached_index_is_never_served(tmp_path, monkeypatch): cache.fetch(fixture(payload)) +def test_separate_proxy_processes_share_verified_cache(tmp_path, monkeypatch): + context = multiprocessing.get_context("fork") + start, results, calls = context.Event(), context.Queue(), context.Value("i", 0) + payload = bytes(range(256)) * 20 + origin = fake_origin(payload, []) + + def get(*args): + with calls.get_lock(): + calls.value += 1 + return origin(*args) + + monkeypatch.setattr(proxy, "public_get", get) + + def fetch(): + start.wait(5) + path, hit = proxy.IndexCache(tmp_path, workers=2, chunk_size=1024).fetch(fixture(payload)) + results.put((path.read_bytes() == payload, hit)) + + processes = [context.Process(target=fetch) for _ in range(2)] + try: + for process in processes: + process.start() + start.set() + assert sorted(results.get(timeout=10) for _ in processes) == [(True, False), (True, True)] + for process in processes: + process.join(5) + assert process.exitcode == 0 + assert calls.value == 6 + finally: + for process in processes: + if process.is_alive(): + process.terminate() + process.join(5) + + def test_wrong_origin_content_is_never_committed(tmp_path, monkeypatch): payload = b"original content" monkeypatch.setattr(proxy, "public_get", fake_origin(b"modified content", [])) From ed2be4120ae8e8f6bca8bc89ddfa507be2f53e02 Mon Sep 17 00:00:00 2001 From: minixalpha Date: Mon, 12 Oct 2026 09:54:36 -0700 Subject: [PATCH 09/15] fix(benchmarks): cache original uv releases for reliable setup --- benchmarks/harbor/README.md | 27 +++ benchmarks/harbor/README.zh-CN.md | 23 ++ .../preflight-v2-evidence.json | 29 +++ .../adaptive-rejection-sampler.json | 28 +++ .../transport-v4/build-cython-ext.json | 28 +++ .../transport-v4/build-pov-ray.json | 28 +++ .../transport-v4/caffe-cifar-10.json | 28 +++ .../transport-v4/cancel-async-tasks.json | 28 +++ .../transport-v4/circuit-fibsqrt.json | 28 +++ .../transport-v4/cobol-modernization.json | 28 +++ .../transport-v4/constraints-scheduling.json | 28 +++ .../custom-memory-heap-crash.json | 28 +++ .../transport-v4/db-wal-recovery.json | 28 +++ .../dependency-cache.compose.json | 99 +++++++++ .../transport-v4/dna-assembly.json | 28 +++ .../transport-v4/fix-code-vulnerability.json | 28 +++ .../transport-v4/git-multibranch.json | 28 +++ .../transport-v4/kv-store-grpc.json | 28 +++ .../llm-inference-batching-scheduler.json | 28 +++ .../transport-v4/log-summary-date-ranges.json | 28 +++ .../transport-v4/merge-diff-arc-agi-task.json | 28 +++ .../model-extraction-relu-logits.json | 28 +++ .../transport-v4/mteb-leaderboard.json | 28 +++ .../multi-source-data-merger.json | 28 +++ .../transport-v4/nginx-request-logging.json | 28 +++ .../transport-v4/openssl-selfsigned-cert.json | 28 +++ .../transport-v4/path-tracing.json | 28 +++ .../transport-v4/pypi-server.json | 28 +++ .../transport-v4/pytorch-model-cli.json | 28 +++ .../transport-v4/pytorch-model-recovery.json | 28 +++ .../transport-v4/qemu-alpine-ssh.json | 28 +++ .../transport-v4/query-optimize.json | 28 +++ .../transport-v4/regex-chess.json | 28 +++ .../transport-v4/regex-log.json | 28 +++ .../schemelike-metacircular-eval.json | 28 +++ .../torch-pipeline-parallelism.json | 28 +++ .../torch-tensor-parallelism.json | 28 +++ .../transport-v4/write-compressor.json | 28 +++ .../uv-cache-integration-probe.json | 19 ++ .../uv-release-manifest.json | 24 +++ benchmarks/harbor/scripts/uv_release_cache.py | 201 ++++++++++++++++++ .../harbor/tests/test_uv_release_cache.py | 156 ++++++++++++++ 42 files changed, 1530 insertions(+) create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/adaptive-rejection-sampler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-cython-ext.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-pov-ray.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/caffe-cifar-10.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cancel-async-tasks.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/circuit-fibsqrt.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cobol-modernization.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/constraints-scheduling.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/custom-memory-heap-crash.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/db-wal-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dna-assembly.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/fix-code-vulnerability.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/git-multibranch.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/kv-store-grpc.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/llm-inference-batching-scheduler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/log-summary-date-ranges.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/merge-diff-arc-agi-task.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/model-extraction-relu-logits.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/mteb-leaderboard.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/multi-source-data-merger.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/nginx-request-logging.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/openssl-selfsigned-cert.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/path-tracing.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pypi-server.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-cli.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/qemu-alpine-ssh.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/query-optimize.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-chess.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-log.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/schemelike-metacircular-eval.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-pipeline-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-tensor-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/write-compressor.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-cache-integration-probe.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-release-manifest.json create mode 100644 benchmarks/harbor/scripts/uv_release_cache.py create mode 100644 benchmarks/harbor/tests/test_uv_release_cache.py diff --git a/benchmarks/harbor/README.md b/benchmarks/harbor/README.md index bd264d1..32da9aa 100644 --- a/benchmarks/harbor/README.md +++ b/benchmarks/harbor/README.md @@ -149,6 +149,33 @@ and index cache into the sidecar; do not share task files or credentials. Multip proxy processes coordinate cache writes with per-digest file locks. No host proxy process or published port is needed in this mode. +### Cache original uv release archives + +When the official uv installer repeatedly receives truncated GitHub downloads, +prepare its exact original release archives using the public release index: + +```bash +python benchmarks/harbor/scripts/uv_release_cache.py prepare \ + --cache .cache/nanopy-harbor/uv-releases \ + --manifest /path/to/new-run/uv-release-manifest.json \ + --version 0.9.11 --version 0.9.5 +``` + +The utility requires the original asset filename, size, and SHA256 from GitHub's +release metadata. It downloads verified ranges, checks the whole archive, and +preserves the metadata by content hash. Existing manifests cannot be overwritten. +Only the `x86_64-unknown-linux-gnu` archive is prepared; task images stay original. + +Run `uv_release_cache.py serve --cache /cache --manifest /manifest.json` in a +loopback sidecar sharing `service:main`, with a readiness health check and both +cache and manifest mounted read-only. Set only the official installer's supported +`UV_INSTALLER_GITHUB_BASE_URL=http://127.0.0.1:18743` environment variable in the +task container. Installer scripts, versions, package contents, and normal model +traffic remain unchanged. Registered archives are rehashed before serving; +uncached uv versions/platforms redirect to their original HTTPS GitHub URLs. +Record this transport setting, utility, manifest, and native preflight evidence +before model work. This cache does not certify unsupported verifier dependencies. + ### Direct Harbor invocation For adapter development or comparison with published versions, the lower-level diff --git a/benchmarks/harbor/README.zh-CN.md b/benchmarks/harbor/README.zh-CN.md index 4405d1f..3b0cb3c 100644 --- a/benchmarks/harbor/README.zh-CN.md +++ b/benchmarks/harbor/README.zh-CN.md @@ -121,6 +121,29 @@ SHA256 校验后才能提供给客户端；缓存命中也会重新校验哈希 代理进程通过按摘要划分的文件锁协调缓存写入；此模式不需要宿主机代理进程，也 不向宿主机发布端口。 +### 缓存原始 uv 发布压缩包 + +如果官方 uv 安装器反复遇到 GitHub 下载不完整，可按公共发布索引缓存对应的原始版本： + +```bash +python benchmarks/harbor/scripts/uv_release_cache.py prepare \ + --cache .cache/nanopy-harbor/uv-releases \ + --manifest /path/to/new-run/uv-release-manifest.json \ + --version 0.9.11 --version 0.9.5 +``` + +工具要求压缩包匹配 GitHub 发布元数据中的文件名、大小和 SHA256，通过分段及完整 +文件校验后才接受，并按内容哈希保留原始元数据。已有 manifest 不可覆盖。目前只准备 +`x86_64-unknown-linux-gnu` 压缩包，题目镜像保持原样。 + +在共享 `service:main` 网络命名空间的回环辅助容器中运行 +`uv_release_cache.py serve --cache /cache --manifest /manifest.json`，配置就绪健康 +检查，并以只读方式挂载缓存和 manifest。在题目容器中仅设置官方安装器支持的 +`UV_INSTALLER_GITHUB_BASE_URL=http://127.0.0.1:18743` 环境变量。安装脚本、版本、 +软件包内容及正常模型流量保持原样。已登记压缩包在提供下载前重新校验；未缓存的 +uv 版本或平台重定向至其原始 HTTPS GitHub 地址。在模型运行前登记此下载配置、 +工具、manifest 和原生预检证据；缓存不代表未配置的验证器依赖已通过预检。 + ### 直接调用 Harbor 开发 adapter 或对照已发布版本时，仍可使用底层 Harbor 入口，并固定容器内 agent 版本： diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v2-evidence.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v2-evidence.json index fd493e2..424e515 100644 --- a/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v2-evidence.json +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-v2-evidence.json @@ -93,5 +93,34 @@ "fresh_inrelease_status": 200, "fresh_inrelease_bytes": 151075, "fresh_inrelease_seconds": 2.54 + }, + "interrupted_trials_without_native_result": { + "recorded_at": "2026-10-12T16:05:50.616815+00:00", + "trials": [ + { + "trial": "git-multibranch__Ev3aCQ6", + "config_sha256": "32a2f147eaa242e5c68ab921261af8f76983a5cf965800b2d8947a13f12b38bd", + "native_result_present": false, + "reason": "Orchestration interrupted after confirmed host-proxy route failure; leftover installation container removed. No model calls or verifier execution.", + "removed_containers": [ + "286e73cd5c27" + ], + "removed_networks": [ + "a0dd770ff6db" + ] + }, + { + "trial": "log-summary-date-ranges__Ppd7Nns", + "config_sha256": "23b1a0ee37a691be4920a8206bef15327d0dbaf6306592ab031b475332308a37", + "native_result_present": false, + "reason": "Orchestration interrupted after confirmed host-proxy route failure; leftover installation container removed. No model calls or verifier execution.", + "removed_containers": [ + "198f807b0702" + ], + "removed_networks": [ + "4c1ac5b3ccb6" + ] + } + ] } } diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/adaptive-rejection-sampler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/adaptive-rejection-sampler.json new file mode 100644 index 0000000..44e60c8 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/adaptive-rejection-sampler.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-cython-ext.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-cython-ext.json new file mode 100644 index 0000000..50e0c47 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-cython-ext.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-pov-ray.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-pov-ray.json new file mode 100644 index 0000000..75826a3 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-pov-ray.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 11820 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/caffe-cifar-10.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/caffe-cifar-10.json new file mode 100644 index 0000000..3154348 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/caffe-cifar-10.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cancel-async-tasks.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cancel-async-tasks.json new file mode 100644 index 0000000..9a9e8ed --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cancel-async-tasks.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/circuit-fibsqrt.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/circuit-fibsqrt.json new file mode 100644 index 0000000..c1ec308 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/circuit-fibsqrt.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cobol-modernization.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cobol-modernization.json new file mode 100644 index 0000000..87c64bb --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cobol-modernization.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/constraints-scheduling.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/constraints-scheduling.json new file mode 100644 index 0000000..82d7084 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/constraints-scheduling.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1020 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/custom-memory-heap-crash.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/custom-memory-heap-crash.json new file mode 100644 index 0000000..8938ca8 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/custom-memory-heap-crash.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/db-wal-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/db-wal-recovery.json new file mode 100644 index 0000000..e1d97f3 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/db-wal-recovery.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json new file mode 100644 index 0000000..413dbe3 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json @@ -0,0 +1,99 @@ +{ + "services": { + "main": { + "volumes": [ + { + "type": "bind", + "source": "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf", + "target": "/etc/apt/apt.conf.d/99nanopy-index-cache", + "read_only": true + } + ], + "environment": { + "UV_INSTALLER_GITHUB_BASE_URL": "http://127.0.0.1:18743" + } + }, + "apt-index-cache": { + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "network_mode": "service:main", + "entrypoint": [ + "python3", + "/proxy/apt_index_proxy.py", + "--bind", + "127.0.0.1", + "--port", + "18742", + "--cache", + "/cache", + "--workers", + "16" + ], + "user": "1000:1000", + "read_only": true, + "cap_drop": [ + "ALL" + ], + "security_opt": [ + "no-new-privileges:true" + ], + "mem_limit": "256m", + "cpus": 0.5, + "volumes": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/apt_index_proxy.py:/proxy/apt_index_proxy.py:ro", + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/cache_image.py:/proxy/cache_image.py:ro", + "/home/minix/projects/nanoPyCodeAgent/.cache/nanopy-harbor/apt-indexes:/cache" + ], + "healthcheck": { + "test": [ + "CMD", + "python3", + "-c", + "import socket; socket.create_connection((\"127.0.0.1\",18742), timeout=2).close()" + ], + "interval": "1s", + "timeout": "3s", + "retries": 30 + } + }, + "uv-release-cache": { + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "network_mode": "service:main", + "entrypoint": [ + "python3", + "/proxy/uv_release_cache.py", + "serve", + "--cache", + "/cache", + "--manifest", + "/manifest.json" + ], + "user": "1000:1000", + "read_only": true, + "cap_drop": [ + "ALL" + ], + "security_opt": [ + "no-new-privileges:true" + ], + "cpus": 0.5, + "mem_limit": "256m", + "volumes": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/uv_release_cache.py:/proxy/uv_release_cache.py:ro", + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/cache_image.py:/proxy/cache_image.py:ro", + "/home/minix/projects/nanoPyCodeAgent/.cache/nanopy-harbor/uv-releases:/cache:ro", + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-release-manifest.json:/manifest.json:ro" + ], + "healthcheck": { + "test": [ + "CMD", + "python3", + "-c", + "import socket; socket.create_connection((\"127.0.0.1\",18743), timeout=2).close()" + ], + "interval": "1s", + "timeout": "3s", + "retries": 30 + } + } + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dna-assembly.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dna-assembly.json new file mode 100644 index 0000000..b16269b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dna-assembly.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/fix-code-vulnerability.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/fix-code-vulnerability.json new file mode 100644 index 0000000..747355b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/fix-code-vulnerability.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/git-multibranch.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/git-multibranch.json new file mode 100644 index 0000000..6f6702a --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/git-multibranch.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/kv-store-grpc.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/kv-store-grpc.json new file mode 100644 index 0000000..3681724 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/kv-store-grpc.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/llm-inference-batching-scheduler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/llm-inference-batching-scheduler.json new file mode 100644 index 0000000..365fda9 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/llm-inference-batching-scheduler.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/log-summary-date-ranges.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/log-summary-date-ranges.json new file mode 100644 index 0000000..a9245bd --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/log-summary-date-ranges.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/merge-diff-arc-agi-task.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/merge-diff-arc-agi-task.json new file mode 100644 index 0000000..df9226a --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/merge-diff-arc-agi-task.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/model-extraction-relu-logits.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/model-extraction-relu-logits.json new file mode 100644 index 0000000..3ce5408 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/model-extraction-relu-logits.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/mteb-leaderboard.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/mteb-leaderboard.json new file mode 100644 index 0000000..c6a7d83 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/mteb-leaderboard.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/multi-source-data-merger.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/multi-source-data-merger.json new file mode 100644 index 0000000..3e3288e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/multi-source-data-merger.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/nginx-request-logging.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/nginx-request-logging.json new file mode 100644 index 0000000..e50410c --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/nginx-request-logging.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/openssl-selfsigned-cert.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/openssl-selfsigned-cert.json new file mode 100644 index 0000000..ed26fea --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/openssl-selfsigned-cert.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/path-tracing.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/path-tracing.json new file mode 100644 index 0000000..3aa2ea8 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/path-tracing.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pypi-server.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pypi-server.json new file mode 100644 index 0000000..2a8ed0a --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pypi-server.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-cli.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-cli.json new file mode 100644 index 0000000..06b1243 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-cli.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-recovery.json new file mode 100644 index 0000000..2565280 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-recovery.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/qemu-alpine-ssh.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/qemu-alpine-ssh.json new file mode 100644 index 0000000..4351b79 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/qemu-alpine-ssh.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/query-optimize.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/query-optimize.json new file mode 100644 index 0000000..578fb18 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/query-optimize.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-chess.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-chess.json new file mode 100644 index 0000000..d44ed05 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-chess.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-log.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-log.json new file mode 100644 index 0000000..cffde2e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-log.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/schemelike-metacircular-eval.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/schemelike-metacircular-eval.json new file mode 100644 index 0000000..675f164 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/schemelike-metacircular-eval.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 2220 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-pipeline-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-pipeline-parallelism.json new file mode 100644 index 0000000..d5ac572 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-pipeline-parallelism.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-tensor-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-tensor-parallelism.json new file mode 100644 index 0000000..26450a0 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-tensor-parallelism.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/write-compressor.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/write-compressor.json new file mode 100644 index 0000000..4e97159 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/write-compressor.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-cache-integration-probe.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-cache-integration-probe.json new file mode 100644 index 0000000..3d1f5f6 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-cache-integration-probe.json @@ -0,0 +1,19 @@ +{ + "observed_at": "2026-10-12T16:50:17.208329+00:00", + "kind": "dependency_cache_integration_probe_not_a_benchmark_trial", + "main_image": "alexgshaw/git-multibranch@sha256:5bc449921c138295f6ddd8dca2977319fee2ce523811dbe56cec4ca663385fb5", + "sidecar_image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "setting": { + "UV_INSTALLER_GITHUB_BASE_URL": "http://127.0.0.1:18743" + }, + "original_script_sha256": { + "0.9.11": "16180030f576e0a23ed90e591d3e16db04a56264dc592b14313cf6c0f6f8ca27", + "0.9.5": "8402ab80d2ef54d7044a71ea4e4e1e8db3b20c87c7bffbc30bff59f1e80ebbd5" + }, + "observed_installed_versions": [ + "uv 0.9.11", + "uv 0.9.5" + ], + "release_manifest_sha256": "2b7a659f5031bff87ab22296320c0f49fd012dc28b7fb7bf111845149ba67d07", + "scope": "Unchanged official installer scripts, original release archives, no model calls or official task assertions." +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-release-manifest.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-release-manifest.json new file mode 100644 index 0000000..86c64e3 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-release-manifest.json @@ -0,0 +1,24 @@ +{ + "registered_at": "2026-10-12T16:43:49.880933+00:00", + "metadata_source": "Original public GitHub release API, fetched before this registration", + "assets": [ + { + "version": "0.9.11", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "url": "https://github.com/astral-sh/uv/releases/download/0.9.11/uv-x86_64-unknown-linux-gnu.tar.gz", + "digest": "sha256:817c0722b437b4b45b9a7e0231616a09db76bab1b8d178ba7a9680c690db19f0", + "size": 21699161, + "asset_id": 318993196, + "release_metadata_sha256": "af11222ef5d913ecba8a659d267238df719960dbaac4a09623c8f22c7163fd6a" + }, + { + "version": "0.9.5", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "url": "https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu.tar.gz", + "digest": "sha256:2cf10babba653310606f8b49876cfb679928669e7ddaa1fb41fb00ce73e64f66", + "size": 21370871, + "asset_id": 306970260, + "release_metadata_sha256": "fa1e1093fcc69e2398057de4819c64fae893bb04dd87afcb437f89899f13aa09" + } + ] +} diff --git a/benchmarks/harbor/scripts/uv_release_cache.py b/benchmarks/harbor/scripts/uv_release_cache.py new file mode 100644 index 0000000..78c9a2f --- /dev/null +++ b/benchmarks/harbor/scripts/uv_release_cache.py @@ -0,0 +1,201 @@ +"""Cache original uv release archives using GitHub's filename, size, and SHA256. + +The official installer remains unchanged. Its supported +UV_INSTALLER_GITHUB_BASE_URL setting can point to the loopback server. Uncached +release paths redirect to the original HTTPS GitHub URL. No TLS interception, +general proxy environment variables, or alternate packages are involved. +""" + +import argparse +from datetime import datetime, timezone +import fcntl +import hashlib +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +import importlib.util +import json +from pathlib import Path +import re + + +spec = importlib.util.spec_from_file_location( + "uv_verified_ranges", Path(__file__).with_name("cache_image.py"), +) +ranges = importlib.util.module_from_spec(spec) +spec.loader.exec_module(ranges) + +ASSET = "uv-x86_64-unknown-linux-gnu.tar.gz" +VERSION = re.compile(r"\d+\.\d+\.\d+") +RELEASE_PATH = re.compile( + r"/astral-sh/uv/releases/download/([0-9A-Za-z][0-9A-Za-z._+-]*)/" + r"(uv-[0-9A-Za-z._-]+\.(?:tar\.gz|zip)(?:\.sha256)?)", +) + + +def asset_path(version, name=ASSET): + return f"/astral-sh/uv/releases/download/{version}/{name}" + + +def release_asset(version, raw): + if not VERSION.fullmatch(version): + raise ValueError("Expected an exact stable uv version") + release = json.loads(raw) + if release.get("tag_name") != version: + raise ValueError("Release tag differs from the requested version") + matches = [asset for asset in release["assets"] if asset["name"] == ASSET] + if len(matches) != 1: + raise ValueError("Expected exactly one original Linux archive") + asset = matches[0] + url = "https://github.com" + asset_path(version) + if (asset.get("browser_download_url") != url + or not re.fullmatch(r"sha256:[0-9a-f]{64}", asset.get("digest") or "") + or type(asset.get("size")) is not int or not 0 < asset["size"] <= 128 * 1024 * 1024): + raise ValueError("Release archive lacks valid original provenance") + return dict(version=version, filename=ASSET, url=url, + digest=asset["digest"], size=asset["size"], asset_id=asset["id"], + release_metadata_sha256=hashlib.sha256(raw).hexdigest()) + + +def prepare(cache, manifest, versions, workers=16): + if manifest.exists(): + raise ValueError("A registered release manifest cannot be overwritten") + if not versions or len(versions) != len(set(versions)): + raise ValueError("Specify distinct exact uv versions") + if any(not VERSION.fullmatch(version) for version in versions): + raise ValueError("Expected exact stable uv versions") + cache.mkdir(parents=True, exist_ok=True) + with (cache / ".prepare.lock").open("a") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + entries = [] + for version in versions: + url = f"https://api.github.com/repos/astral-sh/uv/releases/tags/{version}" + status, _, raw = ranges.get(url, { + "User-Agent": "nanoPyCodeAgent-verified-cache", + "Accept": "application/vnd.github+json", + }, 4 * 1024 * 1024) + if status != 200: + raise ValueError("Original release metadata request failed") + entry = release_asset(version, raw) + metadata = cache / "metadata" / (entry["release_metadata_sha256"] + ".json") + metadata.parent.mkdir(exist_ok=True) + metadata.write_bytes(raw) + ranges.fetch_blob(cache, entry, entry["url"], {}, workers, 256 * 1024) + entries.append(entry) + print(json.dumps({"verified_version": version, "size": entry["size"], + "digest": entry["digest"]}), flush=True) + manifest.parent.mkdir(parents=True, exist_ok=True) + manifest.write_text(json.dumps({ + "registered_at": datetime.now(timezone.utc).isoformat(), + "metadata_source": "Original public GitHub release API, fetched before this registration", + "assets": entries, + }, indent=2) + "\n") + + +def registered_assets(cache, manifest): + entries = json.loads(manifest.read_text())["assets"] + assets = {} + if not entries: + raise ValueError("Release manifest is empty") + for entry in entries: + if not re.fullmatch(r"[0-9a-f]{64}", entry.get("release_metadata_sha256", "")): + raise ValueError("Invalid release metadata digest") + metadata = cache / "metadata" / (entry["release_metadata_sha256"] + ".json") + raw = metadata.read_bytes() + if release_asset(entry["version"], raw) != entry: + raise ValueError("Manifest differs from its original release metadata") + path = asset_path(entry["version"], entry["filename"]) + if path in assets: + raise ValueError("Duplicate registered release") + blob = ranges.blob_path(cache, entry) + ranges.verify(blob, entry) + assets[path] = (blob, entry) + return assets + + +class Handler(BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def log_message(self, *_args): + pass + + def do_GET(self): + self.respond(False) + + def do_HEAD(self): + self.respond(True) + + def respond(self, head): + self.close_connection = True + if not RELEASE_PATH.fullmatch(self.path): + self.send_error(403, "Only original uv release assets are supported") + return + cached = self.server.assets.get(self.path) + if cached is None: + self.send_response(302) + self.send_header("Location", "https://github.com" + self.path) + self.send_header("Content-Length", "0") + self.send_header("Connection", "close") + self.end_headers() + return + blob, entry = cached + try: + ranges.verify(blob, entry) + except (OSError, ValueError): + self.send_error(502, "Registered uv archive failed integrity validation") + return + start, end = 0, entry["size"] - 1 + requested = self.headers.get("Range") + if requested: + match = re.fullmatch(r"bytes=(\d+)-(\d*)", requested) + if match: + start = int(match.group(1)) + end = min(int(match.group(2)), end) if match.group(2) else end + if not match or start > end: + self.send_response(416) + self.send_header("Content-Range", f"bytes */{entry['size']}") + self.send_header("Content-Length", "0") + self.send_header("Connection", "close") + self.end_headers() + return + self.send_response(206 if requested else 200) + self.send_header("Content-Length", str(end - start + 1)) + self.send_header("Content-Type", "application/octet-stream") + self.send_header("Connection", "close") + if requested: + self.send_header("Content-Range", f"bytes {start}-{end}/{entry['size']}") + self.end_headers() + if not head: + try: + with blob.open("rb") as stream: + stream.seek(start) + remaining = end - start + 1 + while remaining: + block = stream.read(min(remaining, 64 * 1024)) + if not block: + raise OSError("Verified archive changed during transfer") + self.wfile.write(block) + remaining -= len(block) + except OSError: + return # Close a partial transfer; never fabricate successful bytes. + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("mode", choices=["prepare", "serve"]) + parser.add_argument("--cache", type=Path, required=True) + parser.add_argument("--manifest", type=Path, required=True) + parser.add_argument("--version", action="append", default=[]) + parser.add_argument("--bind", default="127.0.0.1", choices=["127.0.0.1"]) + parser.add_argument("--port", type=int, default=18743) + args = parser.parse_args() + if args.mode == "prepare": + prepare(args.cache, args.manifest, args.version) + else: + assets = registered_assets(args.cache, args.manifest) + server = ThreadingHTTPServer((args.bind, args.port), Handler) + server.assets = assets + print(json.dumps({"ready": True, "registered_archives": len(assets)}), flush=True) + server.serve_forever() + + +if __name__ == "__main__": + main() diff --git a/benchmarks/harbor/tests/test_uv_release_cache.py b/benchmarks/harbor/tests/test_uv_release_cache.py new file mode 100644 index 0000000..740f845 --- /dev/null +++ b/benchmarks/harbor/tests/test_uv_release_cache.py @@ -0,0 +1,156 @@ +"""The optional uv cache serves only archives bound to original release metadata.""" + +import hashlib +import http.client +import importlib.util +import json +from pathlib import Path +import threading + +import pytest + +spec = importlib.util.spec_from_file_location( + "uv_release_cache", Path(__file__).parents[1] / "scripts/uv_release_cache.py", +) +uv = importlib.util.module_from_spec(spec) +spec.loader.exec_module(uv) + + +def metadata(payload=b"original uv archive", version="0.9.11"): + return json.dumps({"tag_name": version, "assets": [{ + "name": uv.ASSET, "size": len(payload), + "digest": "sha256:" + hashlib.sha256(payload).hexdigest(), "id": 123, + "browser_download_url": "https://github.com" + uv.asset_path(version), + }]}).encode() + + +def make_cache(tmp_path, payload=b"original uv archive"): + raw = metadata(payload) + entry = uv.release_asset("0.9.11", raw) + (tmp_path / "metadata").mkdir() + (tmp_path / "metadata" / (entry["release_metadata_sha256"] + ".json")).write_bytes(raw) + blob = uv.ranges.blob_path(tmp_path, entry) + blob.parent.mkdir(parents=True) + blob.write_bytes(payload) + manifest = tmp_path / "manifest.json" + manifest.write_text(json.dumps({"assets": [entry]})) + return manifest, blob + + +@pytest.mark.parametrize("field,value", [ + ("name", "other.tar.gz"), ("size", 0), ("size", True), + ("digest", None), ("digest", "sha256:" + "z" * 64), + ("browser_download_url", "https://evil.example/uv.tar.gz"), +]) +def test_reject_incomplete_or_changed_release_provenance(field, value): + data = json.loads(metadata()) + data["assets"][0][field] = value + with pytest.raises(ValueError): + uv.release_asset("0.9.11", json.dumps(data).encode()) + + +def test_exact_version_and_unique_asset_required(): + with pytest.raises(ValueError, match="exact"): + uv.release_asset("latest", metadata()) + with pytest.raises(ValueError, match="tag"): + uv.release_asset("0.9.5", metadata()) + data = json.loads(metadata()) + data["assets"] *= 2 + with pytest.raises(ValueError, match="exactly one"): + uv.release_asset("0.9.11", json.dumps(data).encode()) + + +def test_registered_archive_and_metadata_are_revalidated(tmp_path): + manifest, blob = make_cache(tmp_path) + assert uv.registered_assets(tmp_path, manifest)[uv.asset_path("0.9.11")][0] == blob + blob.write_bytes(b"x" * blob.stat().st_size) + with pytest.raises(ValueError, match="SHA256"): + uv.registered_assets(tmp_path, manifest) + + +def test_manifest_cannot_change_original_size_or_digest(tmp_path): + manifest, _ = make_cache(tmp_path) + data = json.loads(manifest.read_text()) + data["assets"][0]["size"] += 1 + manifest.write_text(json.dumps(data)) + with pytest.raises(ValueError, match="metadata"): + uv.registered_assets(tmp_path, manifest) + data["assets"][0]["release_metadata_sha256"] = "../../outside" + manifest.write_text(json.dumps(data)) + with pytest.raises(ValueError, match="digest"): + uv.registered_assets(tmp_path, manifest) + + +def test_prepare_uses_fresh_original_index_and_verified_downloader(tmp_path, monkeypatch): + calls = [] + raw = metadata() + + def get(url, headers, limit): + calls.append(url) + assert headers["Accept"] == "application/vnd.github+json" + return 200, {}, raw + + def fetch(cache, entry, url, headers, workers, chunk): + assert entry == uv.release_asset("0.9.11", raw) + assert url == entry["url"] and headers == {} and workers == 16 + assert chunk == 256 * 1024 + calls.append("verified_download") + + monkeypatch.setattr(uv.ranges, "get", get) + monkeypatch.setattr(uv.ranges, "fetch_blob", fetch) + manifest = tmp_path / "plan.json" + uv.prepare(tmp_path / "cache", manifest, ["0.9.11"]) + assert calls == ["https://api.github.com/repos/astral-sh/uv/releases/tags/0.9.11", "verified_download"] + with pytest.raises(ValueError, match="overwritten"): + uv.prepare(tmp_path / "cache", manifest, ["0.9.11"]) + + +@pytest.fixture +def server(tmp_path): + manifest, blob = make_cache(tmp_path) + server = uv.ThreadingHTTPServer(("127.0.0.1", 0), uv.Handler) + server.assets = uv.registered_assets(tmp_path, manifest) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + yield server, blob + server.shutdown() + server.server_close() + thread.join() + + +def request(server, path, method="GET", headers=None): + connection = http.client.HTTPConnection(*server.server_address, timeout=5) + try: + connection.request(method, path, headers=headers or {}) + response = connection.getresponse() + return response.status, dict(response.getheaders()), response.read() + finally: + connection.close() + + +def test_http_exact_archive_head_and_resume(server): + server, blob = server + path = uv.asset_path("0.9.11") + assert request(server, path)[2] == blob.read_bytes() + status, headers, body = request(server, path, method="HEAD") + assert status == 200 and body == b"" and int(headers["Content-Length"]) == blob.stat().st_size + status, headers, body = request(server, path, headers={"Range": "bytes=3-8"}) + assert status == 206 and body == blob.read_bytes()[3:9] + assert headers["Content-Range"] == f"bytes 3-8/{blob.stat().st_size}" + assert request(server, path, headers={"Range": "bytes=999-"})[0] == 416 + + +def test_unknown_versions_keep_original_github_source(server): + server, _ = server + path = uv.asset_path("0.9.99") + status, headers, body = request(server, path) + assert status == 302 and headers["Location"] == "https://github.com" + path and body == b"" + for bad in ["/private", "//evil.example/asset", "/astral-sh/uv/releases/download/../../private", uv.asset_path(".."), path + "?token=private"]: + assert request(server, bad)[0] == 403 + + +def test_http_corruption_fails_closed_after_server_start(server): + server, blob = server + blob.write_bytes(b"x" * blob.stat().st_size) + status, _, body = request(server, uv.asset_path("0.9.11")) + assert status == 502 and b"original uv archive" not in body From 4559d076cbf822a2a81676c971ffc869c5e3a44a Mon Sep 17 00:00:00 2001 From: minixalpha Date: Mon, 12 Oct 2026 10:12:54 -0700 Subject: [PATCH 10/15] fix(benchmarks): verify original python archives before baseline setup --- benchmarks/harbor/README.md | 11 + benchmarks/harbor/README.zh-CN.md | 9 + .../dev34-main-baseline-20261012/README.md | 48 +- .../dependency-release-manifest.json | 34 + .../experiment-v4.json | 1380 +++++++++++++++++ .../install-only-v4.json | 45 + .../preflight-before-v4-evidence.json | 664 ++++++++ .../python-cache-integration-probe.json | 17 + .../dependency-cache.compose.json | 5 +- benchmarks/harbor/scripts/uv_release_cache.py | 81 +- .../harbor/tests/test_uv_release_cache.py | 65 + 11 files changed, 2321 insertions(+), 38 deletions(-) create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v4.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only-v4.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-before-v4-evidence.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/python-cache-integration-probe.json diff --git a/benchmarks/harbor/README.md b/benchmarks/harbor/README.md index 32da9aa..78bbf4a 100644 --- a/benchmarks/harbor/README.md +++ b/benchmarks/harbor/README.md @@ -176,6 +176,17 @@ uncached uv versions/platforms redirect to their original HTTPS GitHub URLs. Record this transport setting, utility, manifest, and native preflight evidence before model work. This cache does not certify unsupported verifier dependencies. +For an observed managed-Python download failure, add its exact original build +and archive to a new manifest, for example `--python-build 20251014 +--python-asset cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz`. +The same checks apply to the original `astral-sh/python-build-standalone` release +metadata. Set the documented +[`UV_PYTHON_INSTALL_MIRROR`](https://docs.astral.sh/uv/configuration/environment/#uv_python_install_mirror) +to `http://127.0.0.1:18743/astral-sh/python-build-standalone/releases/download`. +This changes the download route while retaining uv's original Python selection +and archive contents. Uncached Python builds redirect to their original GitHub +URLs; the mirror does not replace uv's embedded Python download index. + ### Direct Harbor invocation For adapter development or comparison with published versions, the lower-level diff --git a/benchmarks/harbor/README.zh-CN.md b/benchmarks/harbor/README.zh-CN.md index 3b0cb3c..3a551e2 100644 --- a/benchmarks/harbor/README.zh-CN.md +++ b/benchmarks/harbor/README.zh-CN.md @@ -144,6 +144,15 @@ python benchmarks/harbor/scripts/uv_release_cache.py prepare \ uv 版本或平台重定向至其原始 HTTPS GitHub 地址。在模型运行前登记此下载配置、 工具、manifest 和原生预检证据；缓存不代表未配置的验证器依赖已通过预检。 +若已观察到 uv 管理的 Python 下载失败，可将对应原始构建及文件名加入新的 manifest， +例如 `--python-build 20251014 +--python-asset cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz`。 +同样按原始 `astral-sh/python-build-standalone` 发布元数据校验。将官方文档中的 +[`UV_PYTHON_INSTALL_MIRROR`](https://docs.astral.sh/uv/configuration/environment/#uv_python_install_mirror) +设置为 `http://127.0.0.1:18743/astral-sh/python-build-standalone/releases/download`。 +这只改变下载路径，保留 uv 原来的 Python 版本选择和压缩包内容。未缓存的 Python +构建重定向至其原始 GitHub 地址，不替换 uv 内置的 Python 下载索引。 + ### 直接调用 Harbor 开发 adapter 或对照已发布版本时，仍可使用底层 Harbor 入口，并固定容器内 agent 版本： diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md index 1a82bf0..ca5a7fe 100644 --- a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md @@ -39,7 +39,10 @@ total attempts, with the same agent output and original per-attempt limits. ## Run -The active plan is [experiment-v3.json](experiment-v3.json). The original +The active installation plan is [experiment-v4.json](experiment-v4.json). +It checks only the five unresolved installations; do not use this partial +preflight plan for model work. A separate accepted-evidence plan must first +bind one successful native installation for every task. The original install-only job ended with three setup timeouts, two uv-download TLS errors, and one cancellation; 28 tasks never started. No models or official tests ran. [Preserved evidence](preflight-v1-evidence.json) records the native artifacts @@ -51,23 +54,42 @@ stopped before model work. [Native evidence](preflight-v2-evidence.json) retains the errors and the subsequent successful sidecar connectivity probe. Historical v2 utility hashes refer to branch commit `f4cdab9`. -The active [APT configuration](transport-v3/apt-cache.conf) routes only the named +The third preflight completed 29 installations, with two truncated uv archive +downloads and one QEMU Python-download TLS error. Its two remaining installations +were interrupted before model work after the new archive-cache integration probe +passed. [Preserved cumulative evidence](preflight-before-v4-evidence.json) +includes native results and the separately identified administrative interruptions. +V4 checks `log-summary-date-ranges`, `qemu-alpine-ssh`, `regex-log`, +`circuit-fibsqrt`, and `build-pov-ray` using the final transport configuration. + +The [APT configuration](transport-v3/apt-cache.conf) routes only the named public APT HTTP origins through a loopback proxy. Its -[Compose override](transport-v3/apt-cache.compose.json) starts a separate proxy -sidecar in each task's own network namespace and waits for its health check. -The sidecar uses an already cached image pinned by digest, has no task filesystem -or agent credentials mounted, and publishes no ports. It has a separate 0.5 CPU / -256 MiB limit; task resources stay unchanged. Each sidecar shares the verified +[Compose override](transport-v4/dependency-cache.compose.json) starts cache +sidecars in each task's own network namespace and waits for their health checks. +Each uses an already cached image pinned by digest, has no task filesystem or +agent credentials mounted, publishes no ports, and has a separate 0.5 CPU / +256 MiB limit. Task resources stay unchanged. The APT sidecar shares the verified host index cache, protected by per-digest cross-process locks. No host proxy -process is required for v3. +process is required. The APT configuration is mounted read-only. Mutable release metadata and packages are not proxy-cached; immutable SHA256 indexes require complete hash -verification. Model/API and other tool traffic retain their normal routes. -Both mount files and the reusable proxy utility are frozen by hashes in v3. +verification. Model/API traffic retains its normal route. The recorded real-network probe fetched the original 8,790,396-byte Debian index in 138.418 seconds cold and 0.008 seconds warm, with identical SHA256. +The archive sidecar serves only original files bound to the +[release manifest](dependency-release-manifest.json): uv 0.9.11, uv 0.9.5, and +CPython 3.13.9 from build 20251014. Filename, size, and full SHA256 must match the +original public release metadata. Its cache and manifest mounts are read-only. +The unchanged installers use their supported `UV_INSTALLER_GITHUB_BASE_URL` and +`UV_PYTHON_INSTALL_MIRROR` transport settings. Python version selection stays +unchanged; uncached versions redirect to their original GitHub URLs. The +[uv](uv-cache-integration-probe.json) and +[Python/pytest](python-cache-integration-probe.json) probes executed no models +or task assertions. All transport files, utilities, and the release manifest +are frozen by hashes in v4. + From the repository root, with credentials in the process environment, set `ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic` and leave `ANTHROPIC_MODEL` unset. Do not put credentials in configuration or published @@ -76,12 +98,8 @@ before sharing. ```bash python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v3.json \ + --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v4.json \ --phase preflight - -python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v3.json \ - --phase model ``` The orchestrator calls only the standard diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest.json new file mode 100644 index 0000000..3ff860f --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest.json @@ -0,0 +1,34 @@ +{ + "registered_at": "2026-10-12T17:04:03.742537+00:00", + "metadata_source": "Original public GitHub release API, fetched before this registration", + "assets": [ + { + "version": "0.9.11", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "url": "https://github.com/astral-sh/uv/releases/download/0.9.11/uv-x86_64-unknown-linux-gnu.tar.gz", + "digest": "sha256:817c0722b437b4b45b9a7e0231616a09db76bab1b8d178ba7a9680c690db19f0", + "size": 21699161, + "asset_id": 318993196, + "release_metadata_sha256": "c1a1378ece65ce1aa8343e57bdbd305289a8fc6d3d9180e8bb6b0f69acd91413" + }, + { + "version": "0.9.5", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "url": "https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu.tar.gz", + "digest": "sha256:2cf10babba653310606f8b49876cfb679928669e7ddaa1fb41fb00ce73e64f66", + "size": 21370871, + "asset_id": 306970260, + "release_metadata_sha256": "1921b17d3370aafb690b29d1d121ee8719ef3d3dbc0c7cf3b948c3059a927a5d" + }, + { + "version": "20251014", + "filename": "cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "url": "https://github.com/astral-sh/python-build-standalone/releases/download/20251014/cpython-3.13.9%2B20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "digest": "sha256:c0ccda275948c79996e46993c2c5476ff5cca606dee530f1dea712179131b348", + "size": 33517986, + "asset_id": 304305340, + "release_metadata_sha256": "8e5db21cfcc2bcafcb2127f269f00cbfdd1f74842a685acd2a14e3c1ad6f8394", + "repository": "astral-sh/python-build-standalone" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v4.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v4.json new file mode 100644 index 0000000..2724063 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v4.json @@ -0,0 +1,1380 @@ +{ + "schema_version": 1, + "registered_at": "2026-10-12T17:11:15.473051+00:00", + "experiment": "dev34-main-baseline-20261012", + "selection_file": "selection.json", + "selection_sha256": "4572b1cc432a16b9c5d1dd554c8d2189ae80414f386a90bf03829bc4d0c3da85", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "arms": { + "main": { + "source_root": "jobs/dev34-baseline-main-source", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "prompt_file": "main-prompt.txt", + "prompt_sha256": "37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d", + "prompt_words": 385 + } + }, + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/adaptive-rejection-sampler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "95408217b9c484ffc18a02a91b7706323b6902919c72b24866a32e27a23543e2", + "environment/Dockerfile": "e644a3343e339303230f1441b7383f076d21c69a5a7aa6430b4f8a9d27f15c8b", + "environment/protected.tar.gz.enc": "7fc9bf1347bb97da8dede04b2c58638f225070913e222d76627068aebd1b1df2", + "instruction.md": "e6d28c28acd76bef61817427c74c2a29e3e8d75f535cbd708afcbfd02c54013f", + "solution/solve.sh": "fe443a8082b5183ba1623311ba0ee958888aefa633eff93766b9ae14e9bdd84d", + "task.toml": "6ed7b37f74beb46e9d47b0f5fad53025667c9548baff4e96d467b56d429fab1a", + "tests/test.sh": "38b43560d173cc2b952c3a3e17b8a480216d84e33450515e047bcb0d806b1e0a", + "tests/test_outputs.py": "547dc6e107f034f41703722aeceb6d0236e3fb69116fc3f2fbaba11884de352f" + } + }, + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/build-cython-ext:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d5f5113941c3963292bae396baa9369571026f46e59437f98ddcc93475dd7729", + "environment/Dockerfile": "cb6cf421cb9b69d43bd25cd491e49bda982fbb865d47515178d017eda727194e", + "instruction.md": "7e52874e45f88505bd21540229719de6639a401477557d814489981d472f0e34", + "solution/solve.sh": "41bbdde88617070366a70e518eedca466a4ea1acd7e96899b2fb7da988e40abc", + "task.toml": "c472f0e7522c128fb7f03e00b50c8cdf2e5e6474daa1e094e7441f4b54e770b2", + "tests/test.sh": "0156c9c4085258b696ce424cf1ce96cba44d1263c5d73bdf54083fd80237f915", + "tests/test_outputs.py": "6549f1a8e04b6163ec08877e6f6b9868bf94563f3a3c72ac1fba9c512c31689e" + } + }, + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cancel-async-tasks:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9adf86b1279cc44f83fe16e6d340595aa992240f1d87243d204cae32e453693b", + "environment/Dockerfile": "1ee7bdb4a7536f5f114c11a3ac6ff42790abfea901fcba6b2a8ea8685f19140e", + "instruction.md": "23d22f7039b50682941652c716f96dac314b8585c32c133614cac77dc067f47e", + "solution/solve.sh": "4938ca4ad5b7dcd36b414b42a1abb10c058ecefce7462b678f9bd05c9fae1844", + "task.toml": "14db0675e04408f2133b0067e8dbddf66a1d6e7baf6b8d38bef8ecccfc4e58a8", + "tests/test.py": "f533c0a64db068f582b4733c5b81f2e66b8fa52fcd3efa65752e6bd6d24780aa", + "tests/test.sh": "79600a6b863def31b4daa008297423010cd2d6243e689d67503e118e4dc53397", + "tests/test_outputs.py": "d6f510b51fafc81b4e9b5cc95535aa13491c8753e0e4171db42c45f4de460ee7" + } + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cobol-modernization:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "c89318cf59efcc7040e3d946022e2616161b6733931a080cdeea7b1918e7ecc4", + "environment/Dockerfile": "8b175c83fc03ec7d16d6101516bc5100230b592f480e435279f4fe051c9280b2", + "environment/data/ACCOUNTS.DAT": "46f63eb29442ebc6716bcd8a0c36f4367910f2783f6b84a307ed9a61ccd1046e", + "environment/data/BOOKS.DAT": "db04da646207cd87f8e9c5b6f0b57d95953899f3eef22de1a81423f06ef4a9fd", + "environment/data/TRANSACTIONS.DAT": "1fa4cc5420ded94d585c563e8196b85166ebbae2bb8924b20014717fb2fb9ee3", + "environment/src/INPUT.DAT": "52fc984f0cf14af1495285d7efe02b9bae13159eea59255560ee938c4f78afee", + "environment/src/program.cbl": "27cb0e56f65f52fa75f346720b37a180b8e034ebe7d7622b979f195087bd6ebb", + "instruction.md": "a87074f1141b7345394cd88053d0debb0dc8f7de95308882c6538ee9a3b359da", + "solution/solve.sh": "7d4a0504e1686c1790bfbc074d4e598f4c631f54c2bc5f3d5fd786574b4810b9", + "task.toml": "4b620211dc3d6b1b989f0210c5ecd2f606974581c8bce27bd78659d48cf0cfbd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "009180a7092583fc8143bb3ffc3957923aa0d61d0aac3a471f76eeb4aae54a88" + } + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "file-operations", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + } + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/fix-code-vulnerability:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "87f28dbad183c3a634887696d328ca68a8a0858f132bc48cb7341c22267d38f7", + "environment/Dockerfile": "0917470986224df4daf4854a71302ae81c73ba5dd6e25be1232d29b758a5b3ab", + "instruction.md": "89b1abdf0af19399f720233b5cdf6b2e8e0ff7b6e3fe3e2a3af0d4d298a8641a", + "solution/solve.sh": "7a0131b369d83ae8546ecc3f9b4e23d38240f1bd1f0941e4430fa85b5182ee03", + "task.toml": "9aa2bd14ce9408289d199d0608bc3fa65b8b5e23374c9641020de62197dbf1e2", + "tests/test.sh": "f1611da693cb4cfc5e8482698c6a1858a5c67220ae9db89b50aed66584e9b17f", + "tests/test_outputs.py": "c006a5de95b69bd78055aa26cc055734dbd50718ede25d331f767bfee4ef7f26" + } + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/git-multibranch:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3b9473429381d9eacabab498cb4a8915aa2bca24fa857ef862e5991d3f81f113", + "environment/Dockerfile": "88d9a9f4fef1ae7fddf632799dedf5a2fb6489460a93b610c33dd8b122982821", + "environment/default.conf": "a3a4c5d0ac9cfe79b0510e4225c80fa3988c27fbac13622505886bf66449e04a", + "instruction.md": "54014a38d7526d65faeb6c11d9b7ed3d4a5cb5d973d68d3da3b68b8c0289e125", + "solution/solve.sh": "300ca96b79db8d77f14742109aaf76048aafc4644ced750b1bc5c2761e8d0175", + "task.toml": "4b9981604578a31205257cca36a28d0c89c9d8d1058541330acd11d3f8d65f82", + "tests/test.sh": "ddbde3eea6ffa8fda105cbd86597a53bd409f41e13223900aa27e77a84b6b155", + "tests/test_outputs.py": "f03c6867663b8e8fb7899df51ee4df26411fab62ed491753964b801e591d16c5" + } + }, + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/kv-store-grpc:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "51fdc2ad06f0adb1270164f247ea41799b2ee2ad15490e48ddbf5e79d3767aaa", + "environment/Dockerfile": "47c9982789092437f950736110b2bd18899c39b8fd20f3af3c95b848e564a197", + "instruction.md": "f4b3facc199d5b1c78db6e603ffa29ec1b93a77ff7d291e1a92b42e6bf582bed", + "solution/solve.sh": "6bee2938ac9731d1156b0ca02b28c3d5793bf2cbe5f995c863d397093af01cb7", + "task.toml": "6ff0f30a526bfff86a67b0b53439b9965229c0f368167dc1a5d05f34d82299a3", + "tests/test.sh": "c348467a076bf614d0a850e25b2263f85bbf7b4c7a8d344fbd5454a0611a0df8", + "tests/test_outputs.py": "115de2e964bbe689d07930527113422ef8a9e4be4eb958ad84594e3fcd2b1b0e" + } + }, + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/log-summary-date-ranges:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "026000415cb577f3d3d5132902670262f7d5d8f69841903ec06a882292f756b4", + "environment/Dockerfile": "b479f6e987b63a6ec118fa1691b8bb85c437b2a6a0fe63feab5ed7154957fdfc", + "environment/log_generator_deterministic.py": "beb6eff59979e9aa0be63ce8a0f1b0df1178f92871f04ba9f76fa70cf8beb81c", + "instruction.md": "c1d4516ad4ec2238209740eaf01291df9ab48b0028a3fc5df383c6f831453484", + "solution/solve.sh": "9f6a8ee8947bdf1c0f31b530b6194ae998d66841a3b785fa2e246ee840350ab6", + "task.toml": "9a9593673d6805ef3a301ea734e9c4dbaf80da1b37c0fe9d1f2ceea2a8c57e18", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2fc06a6ce1944ad82f84fb437395fa75ed8c0eadd385b92e375c27e5bc9cb2b4" + } + }, + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/merge-diff-arc-agi-task:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d6b48ed0890b12eb68f72c12cd02689bd6bfae2191dd2688949cde6fe77f6a94", + "environment/Dockerfile": "4a03ae6f0c60b9c2f45c33f0413b5aa75c9811870254e3ff1968fd0119173fc6", + "environment/task-deps/bundle1.bundle": "a759f9b6ffc1a2eee50778d72a1899ad7d724becab84fa30e7aac71ee9e443a5", + "environment/task-deps/bundle2.bundle": "75e3e3916a01c980a03715de67b16c729e03661d94a694b5f4223d229bd6593a", + "environment/task-deps/examples.json": "5426569e4ea078bbb74039ec712a2d4d35f1f9fa3a5f26bf2a0de4a2c8ef5db8", + "instruction.md": "2d826479873e5ac1c94c1f802763bf85c22bb04624384685ebdf6b153fe0278e", + "solution/solve.sh": "48d4becc41fda5b6063fe96947474ffe778a29d6924d7b2bb235eb61e5ce555d", + "task.toml": "6ced40e06b43b169857509765eb7f07791db558f9437cc7367d2536771c2ca5b", + "tests/test.json": "20f116a6b718bbf4874bbe203d15a5c0f85a1d88cb622e2d7529c7e4fb2d4c28", + "tests/test.sh": "c9052954cd6ce25c4a375a2fb48e765b1efa5b39f7991b93993cc793af8ab9d2", + "tests/test_outputs.py": "2754ba6c5c5f75c9db5ed2c5729d9f5d7e21cb7fdb18aa0c65abdc3e7e85c149" + } + }, + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/model-extraction-relu-logits:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "mathematics", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9465accba6c57e5de9a196eed776a16eeb0852a8afc18fc3f8979e2de71b0c7b", + "environment/Dockerfile": "598b1f80d3b7194e97cbf02c8d245de30e303909e0cbe6465125d1d3179fddb5", + "environment/forward.py": "af1a23efd8e4390d16a29777c28116b2f8623ae3f7b628fe14db9f94dac26f5b", + "instruction.md": "c9e179dd099065a34c362a865b0eb9fab5f7d96be582cf117e7519580c036ebe", + "solution/solve.sh": "4cea5094f074ff21453d4423aa1b415a104ed90caa55a4acbb4f09941a3994ea", + "task.toml": "eb64dfaf8ae88868911ae28855cdc4f767de464001e2704919015b595be6dc29", + "tests/test.sh": "3b2d8f5e0353c2bab3515d6387468a863a4f9f241ee131c73f545f3fdf9e2d52", + "tests/test_outputs.py": "e5189c7eb9942ede55e1b48a80388590b80a486b75497613f257b3373ad9b2c9" + } + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/multi-source-data-merger:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9bf37162a4c1b88ed93627830bb8512ebae154325d7cb069c883144a0e4c69d6", + "environment/Dockerfile": "c95a40d3d35b473dc15ae1e8784ba7fb1725805c0583fc574dd2764325d75e4f", + "environment/data/source_a/users.json": "75e51e2d3ab0bdba2269d7cf3757576f84f3ee2c13407d2db94289a5958a1778", + "environment/data/source_b/users.csv": "8e87d16dc47d166edd8e56165955bad265b630d53d02762367c816affb5fa760", + "environment/data/source_c/users.parquet": "23d1ea51c4170bdba53febb885a0f495e28de525a52667347b2932136b9a72fe", + "instruction.md": "d1d28bb6e37f7a255a1b7f78aae8591f35c0494f33e4913392424ad954aab944", + "solution/solve.sh": "1b3d0d50e31930d042495d48f4c8e95f1a9c1a979c803b7f7cec651322537e1e", + "task.toml": "ee3964c0ee040f0a0cb121cefefbc15443d127684acb003fa5b61dd42f31c694", + "tests/test.sh": "d5a00e64413c0e1ae452f38a1aa1a47c3ed7b4416bbfccae1a4c053025a336d1", + "tests/test_outputs.py": "a070888930cf829aeca039771260b1386bda990ea805f313468d381ba3608ca5" + } + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/nginx-request-logging:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d79997586d1c14ed8dbfea050e3580b25500e04c8345acfd3f294152433c795a", + "environment/Dockerfile": "43003466d89ab83484a3fa618e35b58cdb74a002b1e8716ef30f12b3ac8302d8", + "instruction.md": "26a7ac98aced6107f147206790ada77bbf8ed3c25fb360cf323d5fc6889edf99", + "solution/solve.sh": "2454bbbf7c201a9a778165f286373ba2cb991a0eb1d026a8005d57ae71250dd9", + "task.toml": "b7a78d00be13656f75069197100b7b8c4fd268decf6f78929c853b3222a2729d", + "tests/test.sh": "27761195f53cbfe5941fe39b573631f6c926869e64b7c43220377e1e116528e7", + "tests/test_outputs.py": "045cc716c14efde3b0dcff5fc7c85ec5d18bfc6ce66f8b40a418fa2a3a4acda0" + } + }, + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/openssl-selfsigned-cert:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "5dfa88651890cda06fa2a25babd3b813818767635e88bbabecc8305d5bb3942f", + "environment/Dockerfile": "f590b523038be43afcaf5291d7b9e8ed1333c48d9994e77b8dee2f2c352f0f40", + "instruction.md": "38054a3371f7c2c853478a2f9af0bff858437d5b28a9a07483238f06e566921f", + "solution/solve.sh": "58cf3c844e907745cd81fb4bd75efb3d080e2f0e82b589c56c954ae6e1756510", + "task.toml": "75c8913f0fa15890786c11999cde920b8bfc10d4ce8a14edf1e76985a2ac5800", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "90970bb3c372e8e7fb6bb9ac5f649ba73eb38e1a5ea2e922a9057a72b6b4192d" + } + }, + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pypi-server:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "906a096a2cc7b9afc738ff5ad5fbd605be65c103f5069c6673915223b9b187e3", + "environment/Dockerfile": "17a4e67580e4f2c6a151d04b9d74eb020666e7eec8e5259c4a5411f9e93858d0", + "instruction.md": "1c537e4a311a8e71155e121f57bfc03f742fbfb99d05314cd03e433996a28a31", + "solution/solve.sh": "1143c0eade4932ec3aeaa5013fb5e23bbcccca15dead294aa03b6af88750775a", + "task.toml": "07c39d7a733c89adadf74bb63a3e906c3694feb6c490d5fc85ac187c003fd447", + "tests/test.sh": "d2b1220e42bcfdcce87907ac5bbd69ac210a3c94926545ca2439636786e7784e", + "tests/test_outputs.py": "82c6351722e9aa984ff1d701bf57df3150ce51466df2b70afa1aa62d2650be4c" + } + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-cli:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "35a093fa5f1592737784ba718c2e3e9ae17ff8b5e53e003fae560ddce091039c", + "environment/Dockerfile": "8fd5c0f21a6914cd440d10c84c9db06fec606f20dc3e0363b17a3b8c220dc4d9", + "environment/task-deps/cJSON.c": "0e7995261640c96db16ed2c91a5d981efa917cf5e72d0f97ec45a1179ae3ad73", + "environment/task-deps/cJSON.h": "0909160541bcc510e0b2d8c7a9cc051fa1459a0f13be4d8e56346a15491f2f2f", + "environment/task-deps/image.png": "f4d5dfefe463939996541404340e7099d5957b56e1df3760d13b5d2de754db53", + "environment/task-deps/lodepng.cpp": "67873c5f6272f779a143a9c81070e8783327c5552d0b9de2aaee2cb779308917", + "environment/task-deps/lodepng.h": "b735b7e0357319973e9488d520d7f270d734a6e9e1bf924fb1d5b8d26d7e97ce", + "environment/task-deps/model.pth": "a32103eef09936e586b1a7dc7e64d279500a38ff30fdd6df2167e28eb954eee6", + "environment/task-deps/model.py": "e4d3a5c728449656d0efde7871ee227619348352769795ed1be1a8b7f835fddc", + "instruction.md": "ce36ee1f6c589f81ee6a6be4967e25c76a2ed697006928023f42dfb9f28a6750", + "solution/solve.sh": "3b34e561c9ef0a53c9948655847846eb4811aaac67b990480aa6aac1b04bfe66", + "task.toml": "fc1c018efbd5d0537ac0741d53d1ebad3886faa51d14cbde8ec953856411bc0d", + "tests/test.sh": "43fb67e537c7adc2df11c2367f47c876cb5d16a7c1ae16d0339c3a3364bb2b06", + "tests/test_outputs.py": "370702c57a4e0c85f70ee37a5c357ae3a5df8480a275bb6263b0376f2f6c093b" + } + }, + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-recovery:20260430", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e080308a05703b40edd16f53aea41a0e6b9874750ecee35bb432029d0d44bf53", + "environment/Dockerfile": "8a6b7edbb9e7c241e447f15a71e13e5d043b25169be2fd9802b84f5be3d56eb6", + "environment/dataset.pt": "6db65e4ec1c0760209c2dd997ef582e4a1519ab38e0312339999c460a696960d", + "environment/weights.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080", + "instruction.md": "2bef07dc28be3ab2c35ae389584034c2e4dde92c2c31b3eb6bd829628c8c3f4a", + "solution/solve.sh": "175a7802777b43e0fa4f629d71f7c68fb40706813c342aaddc72108d89bd5b2c", + "task.toml": "851a51204f53482a31f47cdf8833128cf421113b0331011286940fc857c37dfd", + "tests/test.sh": "6005f3a75ccfe00d7bed1005264b4a84a1f8e30cd780f8a97b3a804be7cc7a8e", + "tests/test_outputs.py": "5d4d57c43e6913bf3d0081da764ad36d1b7275e3bb114a05abda529afe563905", + "tests/weights_gtruth.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080" + } + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/qemu-alpine-ssh:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "qemu-alpine-ssh-v1", + "task_file_sha256": { + "README.md": "72f1c83fa1e3e6381ef85dd82fd109ee18cdeac090311cf3212a0a8cff0a32a5", + "environment/Dockerfile": "015ce0601ff54758f66b33f4952e0d3a577d84c09aed8f3ec5945a570e8d168a", + "instruction.md": "8348c60490cb90bc690cba259dfe98996da4c3294c6d42bba1119b00c64de4de", + "solution/solve.sh": "1cd31186c427b91bb00499d22aa4f0214fc1ca93b40b6e4cfa969611b848db2a", + "task.toml": "10a43ec6ee732eeba852c377f12e23178f7fa5a4868143f3699a15565ffb64c7", + "tests/test.sh": "3b244479776c4ee3639631112799f2cde22cbb80ac10511715e3a8f32909c289", + "tests/test_outputs.py": "83ccb4bdcd259b342212bebefe395f289858b4dc55b2ee7ae0dab238ad2e9718" + } + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/query-optimize:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a82c53e333a0ac0b0b410b5e26cb8e7305dece31db84d340248ffb13d5a45160", + "environment/Dockerfile": "064818771f200e0517d22068f87229a7a95d0a82b0778c84ca7203bf1926a0b8", + "environment/task-deps/my-sql-query.sql": "43382792f0f438625d643e34abe144febd5669518f4ab1ab550a15c6d9314a21", + "instruction.md": "473e12d320d2732f679fb631c2a4dab4a01deea6775686718ae0c417b8a39447", + "solution/solve.sh": "0d95f3cedb287ff9d47c796cbaf812c300d1222c10fa80c49e8acb0e81eb6978", + "task.toml": "6e88f2a86df699594bcb1bf81df4ed694505f79c9420f671335b02524bbe3868", + "tests/golden.sql": "4da1885d56b3ffa7b7b22b422996d018795611f6963499d551582881b74b84fd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2068ae80c57af8d52a7ae510f0666732c057806e556b480970f36f14bae683f6" + } + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/regex-log:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "29ffd98c55dbb6f4844f1b9d8c1769b2155c117ee475e0df6bbc5ef330d239b3", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "4f7ac05e70cf9220ea0f1e5a052c5f908cd0fa884e847d80b0bd51bae2e96f9c", + "solution/solve.sh": "7e670d4f2b4bccb1e4db38f2a173e085ceda028c38167912b466b0a84fcc0999", + "task.toml": "6bc98525852755caf1fe34c174c2f10f4328adb4ce6c1968dcafe1c877e4e160", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "345c3bd09ab6f6fe8c8361a58c0a47bf0a13b3fcb38a5ac7824e44ff855e8f72" + } + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-pipeline-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "297ddea07c230dfd246a8ab07337dd7abcd8e5e2a2b2e26590cb0a8af779839c", + "environment/Dockerfile": "c357ea41e4a4013c5c5980b8fef04af6a5ae250ccc6697cf3bad3127401a931f", + "instruction.md": "989bfcf0c54baf6c8cdc391743dbb2f9e858c5779dd85112536875d4d1b2dcd2", + "solution/solve.sh": "787198e4178d88b887697aefdd29819f29435ffdfe356545d92f58a30fb38b70", + "task.toml": "8b4f9f525c08afbbc4eca9168e46238a61ccb903486773e9ca4da64bb7ffe857", + "tests/test.sh": "cc9fd28b6590993fc37e4a99da771cbf04726c8c62fe308e802612241da69dea", + "tests/test_outputs.py": "23574246a407163a8e6581c84287cc2f1eb745efbaad2a317a5a28c39958c64f" + } + }, + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-tensor-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "4c1b3c3be30548f1f94c6f5503b1bc74b006af7f192dc4cd026f411df1a54604", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "13bde2ab165d72a311e2bf066d5d6c3c8761bf9d3dd39d1b23eda23bc546b910", + "solution/solve.sh": "d083b7cbf9dbb6cc2ff38c8f2acc150f0059c04aea42d6c2183aeaecc5a19385", + "task.toml": "72fd2a5500d1fe8795dcae76cccb1a29fb25894881a7475f2fa519c87d425a8d", + "tests/test.sh": "a7e6c9408b2401e4e7448371f7ddc72f3284caccfb56277d3cd4e98051868cdf", + "tests/test_outputs.py": "4ae4285e15fba1fcf13d7b7c0638ac8c2e2b0797c937abab1fb6b136fd508290" + } + }, + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/write-compressor:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "603015a28500a28f0838fea2c43a5168613746bfd07737920f4bae40b5a326f7", + "environment/Dockerfile": "9a583d29d46818f6ff2be7c6a0879d03343106e5376fd5a48eb5926b402bb93b", + "environment/data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "environment/decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "environment/main.rs": "fa53d6572fca95903e30d2d953f1e428bb8b63513ff218ebee9bc60a425ab389", + "instruction.md": "c4fc3e73e44deeac95f1cd4a34a2d87e853b22ceec2b2824767f5f087b470bb2", + "solution/solve.sh": "01550b7de7ab1320b1898eae055226bb7c73bee259c75102d2240c974a730409", + "task.toml": "6066a509066433c97c417fea68c4e3e73996f698dcdcbabe07718835eea355a6", + "tests/original-data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "tests/original-decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "809f486053e3ffbce852111dee1f63d80ab31297bd33172389e50278bed384e2" + } + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "native_agent_timeout_seconds": 1200, + "work_budget_seconds": 1020, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/constraints-scheduling:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "personal-assistant", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d54a690970149e1216bd4fc066b45069c7948c13e3b1ef2a213a74342805f209", + "environment/Dockerfile": "7d9aba072a4a907229728b9739c8cf821fc2c98450babd64452682caf3759c84", + "environment/inputs/alice_calendar.ics": "5a5d45804bffd339c2479d49e76fe817b38244d14d389afe1de583443808ecbc", + "environment/inputs/bob_calendar.ics": "c97e85dbcdd756758f8a80f4db5fd5c1dcacee79298e4d2262f46de5ff680b50", + "environment/inputs/carol_calendar.ics": "32a0f8687637df0a12b8e5c56d49c4192bc3e790856943632d70f237c7c1363b", + "instruction.md": "342def78177f9bc41bea4d11af679bc385a9dda2ac8a1e425be384a5b7f937b6", + "solution/solve.sh": "802d81ec064fad3050bbade95a2f847ac24e0b7f80fa69b0bcee0347dc92fe77", + "task.toml": "7b9842de16d74e8faf8c7c0aa03b0b81145941cb12dd322d81ae23fc48607250", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "1422b345ba3aacdfc4184e962811b6379988435896c50d8cf62903e70990c5d1" + } + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/custom-memory-heap-crash:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a06ed23a2c467aa36d0d4a88e5c020b1076cc25b8d483517239658e9c596dde9", + "environment/Dockerfile": "6ee81e5a4fa2540443d7f27d577ba840728fc6aab9f301ab7919103673c78080", + "environment/patches/locale_classes.tcc.patch": "16951f4a4df6bf253886ac75f5a39fdbc328e415194493f63cb7589a364aa817", + "environment/patches/locale_init.cc.patch": "e97f133ceebd86f8881f972e0e45b0ec0a13a9bfa428d8032008707683d54292", + "environment/program/main.cpp": "b94538c76887cc2b2ab739b5ac4731cddd0faa8f4a9892a6265c062bbbeede14", + "environment/program/user.cpp": "92b203786fed1bf0b3aa1675c9cd2d82b2651170675064da8ffee73b98ef14f9", + "environment/program/user.h": "f43d155995a56c22b9cd1417daef7c3b5a976cdc662a98eef0feb438ae1eee77", + "instruction.md": "53c68ae14a3c3af9a97c40bfd9bdc30a5d98b37eaa2f69f763475745f70f108d", + "solution/solve.sh": "b0b1d10bea1cd5be14a82b35d522df616df1e67028f90d2a6bef90aaa98511a1", + "task.toml": "ed474be1983e212913aa8866ebb66216acbe36aef2339ac375d232aa46617481", + "tests/test.sh": "70a2b55f0c5757f6b123a216dfd202be4a646680bced28dddcd31fd8959a058f", + "tests/test_outputs.py": "4e1cb33f777fa7fb5e1536f9bea9eace6deccd515c67d4f87570adbe692914ba" + } + }, + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/dna-assembly:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e7855c23d9822f688eec4dd89ca0f1caf91c6a69e1c8f3c9eb544b4ab2a94821", + "environment/Dockerfile": "26e9e43a98f0b1d132bb638f44b7dc882b0d0ca32d56fa33048adeee00ab600d", + "environment/sequences.fasta": "ebdc361b5fba28eb0a2c7a8a8a444ccb12cc842c8cbc6df5a2c24254f130736f", + "instruction.md": "88998c98beabdd966ab46c04ede822bbc74796696aa1bf87c567a9131210533c", + "solution/solve.sh": "c976cb118c8ef1befd846ba05fd5cadb1b1f529bc76b9f2ea9cb3049e6e11a47", + "task.toml": "7b571544b64d9be1c91526837e5d805c2181afe0e6fd57a7f0676bba36c868d7", + "tests/test.sh": "271c8317c98539e3b49233d017fe51fb0e5fba8114e76830fe698565499b2b1d", + "tests/test_outputs.py": "5c2d0fd2e5f6303263e9750f67acda7db491ef04841832b4fc62bcfb61b08924" + } + }, + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/llm-inference-batching-scheduler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ecc9dd635357c184742cf72fc1e452a8946e207b183fc9a7eebbcf07133b911f", + "environment/Dockerfile": "10a6193904ee0022ac098504182989893b196dfc74e626a3ca48534a8eacde48", + "environment/task_file/input_data/requests_bucket_1.jsonl": "8b3cf1e2b06074162038064dd2c114abe308acd32cec7d365ee63ee81978f7de", + "environment/task_file/input_data/requests_bucket_2.jsonl": "e15e2d4c4d55f875c42a6042610d6282f6c0657c39d3e0d0f087268252525cb5", + "environment/task_file/scripts/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "environment/task_file/scripts/baseline_packer.py": "547d230d5e6d93197803480cb81cab0ec6ac63fcfa0a24b246f549523f216c4e", + "environment/task_file/scripts/cost_model.py": "0cf417fd5ff0112defcd46970d37a290c0763c5a9092e91cb3ebb86c3f20e6ce", + "instruction.md": "0817374767533fbf51bb93345ee1bba266263c216c6cc55c096e71b4a1d37dfc", + "solution/solve.sh": "cb9efb586d76a893c3027e62029e286eb6c7583dc92dbf1840a3adc236a926d7", + "task.toml": "52aaac942f917e6a91ff658e0272195bd1cd842d41d71e302414c52603cc588b", + "tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "tests/cost_model_for_tests.py": "d3e810a99d40327fb8317f6512d879c52c6aeebdf5dcdc0c5e117c7df61c9ab9", + "tests/test.sh": "005f980332355e2fe143d4ff4aac11828adf070594bbe7055399c238f06953ca", + "tests/test_outputs.py": "80b06b7c4ba0db28832b767d48be74bb462d7f5df5798d9cb58c4f1160d96ffa" + } + }, + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/path-tracing:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3e4fc952b3f9a860c7ee7837b3f3c686f1283ee7821b014265dba7c94d7fc849", + "environment/Dockerfile": "7aca02e4e20b04ed2269d627a29dd4e7337a02764a9652b641452bf09836a3d5", + "environment/orig.c": "665c1bde0907740890ac114d6b36af1f3fd00449fa287583817f778bb5da200c", + "instruction.md": "9f36e02d636f5dc0946619bc518f2e51c24fa75251da9d1f0ca9eca612cb9572", + "solution/solve.sh": "d107abde101eecee6838da80cc52a3d480c93956fff3a9343a299ea01b3fe9e6", + "task.toml": "cf47013f69bf1b43242da984f231d4daf364fdf44d3845f2eea2ca2cb9308c46", + "tests/test.sh": "a719dbb1ba7ae76df095e9d815941dfa4c55edaf11f173b9c3c3097afa838548", + "tests/test_outputs.py": "572e68b2843191d29592faf42a405861b7958d8e894868b74c5a8886b53eb879" + } + }, + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6", + "native_agent_timeout_seconds": 2400, + "work_budget_seconds": 2220, + "verifier_timeout_seconds": 2400.0, + "docker_image": "alexgshaw/schemelike-metacircular-eval:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "095d3ec6dcc0fe0c7a7e92355456dcea6992f91d1f8d397669b8cfe18475f506", + "environment/Dockerfile": "9b3df76222e16aad360fbd34cb0f71a0be8986052c065b10b92c0cdf417b25c7", + "environment/tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "environment/tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "environment/tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "environment/tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "environment/tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "environment/tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "environment/tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "environment/tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "environment/tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "environment/tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "environment/tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "environment/tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "environment/tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "environment/tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "environment/tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "environment/tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "environment/tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "environment/tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "environment/tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "environment/tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "environment/tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "environment/tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "environment/tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "environment/tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "environment/tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "environment/tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "environment/tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "environment/tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "environment/tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "environment/tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "environment/tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "environment/tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "environment/tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "environment/tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "environment/tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "environment/tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "environment/tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "environment/tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "environment/tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "environment/tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "environment/tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "environment/tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "environment/tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "environment/tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "environment/tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "environment/tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "environment/tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "environment/tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "environment/tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "environment/tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "environment/tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "environment/tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "environment/tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "environment/tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "environment/tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "environment/tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "environment/tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "environment/tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "environment/tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "environment/tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "environment/tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "environment/tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "environment/tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "environment/tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "environment/tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "environment/tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed", + "instruction.md": "3af5f9567b62f93683db7518ec6fcb609f83796258aa02166b5a7091c5b69031", + "solution/solve.sh": "e2c5b0ac2bef4801a1891e13c108813381877426c1a7f01bab86b4fc1f712510", + "task.toml": "8dae8eb226183b0ca92331c1106152a7abc93b03b4e5a95291b86a4ddc9b44e0", + "tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed" + } + }, + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/caffe-cifar-10:20260403", + "resources": { + "cpus": 4, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e918e6852c5c034422b81a6be22d7596fddcee53abe088f56727b2482da32728", + "environment/Dockerfile": "b3b8adc915a1afa035b604273af3bc1aa8b90999d2dfd72763218b34cb87abcd", + "instruction.md": "46907b0ee048f74269c29a15e0e086546b2b1220149597017da8b5ccc05a0cea", + "solution/solve.sh": "771887de1ed557269c7f3160c777519aa9538e1e9b59771b2f6ef38c8952490c", + "task.toml": "2831165ad9d9e8bbc646e6d85c0a2290640dcc4f13749547a2937c7675012a8b", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "e99b9f6284e923388fbc21488b1af99fdde46d4a809d3557b0788dbf1cb6b281" + } + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/circuit-fibsqrt:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "201f7c889dd6f73dd83ca078e347272b33720fa622f8c757ea6d5770453f4cbd", + "environment/Dockerfile": "780c23986778f3e5f842567868af30b1135ed095ca9449e1c90b7f0e0483d3d4", + "environment/gates.txt": "032786f94f16605767c62bb61ce4dbba7ab7d3f1046712c975f01c9a04b6ad29", + "environment/tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "environment/tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438", + "instruction.md": "a7f500266a878d04ac9de39fba76220cfad216d8f6b8063bba93bebb3d4ab9ea", + "solution/solve.sh": "a4ffab4183815a0eb7646abc1de1fc67695014c060d7ff16e8bca9ec94385d0b", + "task.toml": "9f5aa1f7ebbbcec90fd109121e9bd509af3cfff6b4cec701bd91947988f40972", + "tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438" + } + }, + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/mteb-leaderboard:20260430", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ed147a5bc31e9865b16848b2f01d2f9b1e677af57a7c632d7ea2311c0553ca76", + "environment/Dockerfile": "271fdb5b34520c0bf2cb6462365363ec52058f127cd9eea3f5da2589df5c1044", + "instruction.md": "06ea44106eca0405c25bab00f5b3cfeb611c872093f6c0f82ed057839d1d13de", + "solution/solve.sh": "4ed68831d4acf5d80d2d73eec1f21efd0896a6f51f23020f572cfc0af6da0410", + "task.toml": "86574bd1c6c4fefd2e64eee9c2caa53d87c4bcf2e51547912440a0ddbd0badb8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "b28f8fc8b7614801305813c402be52f4be1e75de6658ecdfdf9383aa793892cf" + } + }, + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/regex-chess:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9296cbee592fc996c58d228b51fee303647e5ecfae82d5666d42a40cb30ec355", + "environment/Dockerfile": "2616cd775252303839dfda0ee2f047eb862952b053b0103bd6b2460ef90c11b7", + "environment/check.py": "f9c416ae9ea01dcdd5f6a43170c8434b8c0a0b389a4a30876a2048e277d191c7", + "instruction.md": "a6ed3fcc54fc92d71a84af921942422d299ca89b410dcd92cd05a53d0d06436c", + "solution/solve.sh": "b728bf0ea392a3d244afbd0818d36933c8d3e190e47743c1382cd518d518a2e9", + "task.toml": "07fe52636fcc4bc9fff25d5630af145b97a5331219d719cc8decb4ca67ea21d4", + "tests/test.sh": "8f7f00a1c4dde7fcca6c918c768f2aba432b8c4007c15989716c68091950da87", + "tests/test_outputs.py": "4983c44d21c56dd8191381eff0304afc8b87bb714527ea9036b3319c015a30b4" + } + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "native_agent_timeout_seconds": 12000, + "work_budget_seconds": 11820, + "verifier_timeout_seconds": 12000.0, + "docker_image": "alexgshaw/build-pov-ray:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "38380f64bc631916e264f80ce4d37feedbc783310001475b367d03989d114fdf", + "environment/Dockerfile": "44f2b5fc7311d8f336daaaca99dd4216b49355e67f50b494ebacf1b0f7c14e92", + "environment/deps/illum1.pov": "5a1f7f53ee0730c4923fcd21cd4a9c34719024d8557920441551ba7d96e75dc4", + "instruction.md": "56c8ecdd0084a07f8c97986c43af5c37ed03dd5aa994da6735cf59806162d1b2", + "solution/solve.sh": "3da9dbf589656512cd7a7004dc3752d3552a6af939cfe59eff79f91695fa42c7", + "task.toml": "b90a47ec26c1191d3c523cc094ea96b170e42fbfe176ef1f814c57d05390c097", + "tests/reference_illum1.png": "bb084f430bbc9d5b623a656920670dc06cee6d9132a9d550be4a1b99e41ab9e0", + "tests/test.sh": "928d1497ad21aef0c456294e3784f1fddf8493410f23584da89f9e02ec6bd134", + "tests/test_outputs.py": "64d6ce107503714ada346e5606832ee886be9950971cf81523e73d8ba28e8d37" + } + } + ], + "total_model_runs": 34, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "preflight": [ + { + "arm": "main", + "job_name": "dev34-main-install-only-v4-20261012", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only-v4.json", + "config_sha256": "4a2e610ad0141f756f7ac659def79d3684dee62ae5541716f300c824cb01be0e" + } + ], + "queue": [ + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/adaptive-rejection-sampler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/adaptive-rejection-sampler.json", + "job_name": "dev34-adaptive-rejection-sampler-main-20261012", + "config_sha256": "2616975f9e78d282814e12954e54ecb3af1195b5aa29ce1daedb273e92313197" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-cython-ext", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-cython-ext.json", + "job_name": "dev34-build-cython-ext-main-20261012", + "config_sha256": "a1e438042c729c9c6ea2ad7c791b31910c4621588ea954920f1018cf1e835399" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cancel-async-tasks", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cancel-async-tasks.json", + "job_name": "dev34-cancel-async-tasks-main-20261012", + "config_sha256": "1d7d2fd79228dafd24c6eb021ab05c63de5bf07e25c009e7939dc01004a845b9" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cobol-modernization", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/cobol-modernization.json", + "job_name": "dev34-cobol-modernization-main-20261012", + "config_sha256": "2c7457fc77c808c1d4e0b7c51ac2f2392bb420e8f8df2c9be2d21393ec49b967" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/db-wal-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/db-wal-recovery.json", + "job_name": "dev34-db-wal-recovery-main-20261012", + "config_sha256": "b2ec62ceee3296b103aad663b11d743845195b1f80c6eccc9913a8d51f41a694" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/fix-code-vulnerability", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/fix-code-vulnerability.json", + "job_name": "dev34-fix-code-vulnerability-main-20261012", + "config_sha256": "5a91764083988118dcdcfbbe1c6499895772adae0eb04f7755744242c3794b4a" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/git-multibranch", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/git-multibranch.json", + "job_name": "dev34-git-multibranch-main-20261012", + "config_sha256": "eb834f958725f7940f6480b102e8c802c7d8dd13cb68ae97e8c31e455b0f0d47" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/kv-store-grpc", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/kv-store-grpc.json", + "job_name": "dev34-kv-store-grpc-main-20261012", + "config_sha256": "7049d30361845b2bf3873bed435b06cff9628d96311b23355d3e8a46dc3b9b2a" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/log-summary-date-ranges", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/log-summary-date-ranges.json", + "job_name": "dev34-log-summary-date-ranges-main-20261012", + "config_sha256": "be091b29e4089f88ad1cf0b6418b172d4e8a82aa6d66b30120cd5f8c422a2c2d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/merge-diff-arc-agi-task", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/merge-diff-arc-agi-task.json", + "job_name": "dev34-merge-diff-arc-agi-task-main-20261012", + "config_sha256": "303847a54e011e784b01f43c06442aeb58d85508d686f07610b62f2218287de7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/model-extraction-relu-logits", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/model-extraction-relu-logits.json", + "job_name": "dev34-model-extraction-relu-logits-main-20261012", + "config_sha256": "9aeb1cc667e190dc1ad8f0b999a7a57296d364530e64f1b9a0607b292587bbcf" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/multi-source-data-merger", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/multi-source-data-merger.json", + "job_name": "dev34-multi-source-data-merger-main-20261012", + "config_sha256": "59fccfe416255b9d36fc66112c202243cd5f40a785e2cba1f336d6454846745c" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/nginx-request-logging", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/nginx-request-logging.json", + "job_name": "dev34-nginx-request-logging-main-20261012", + "config_sha256": "04a871c8a2c140b1975cc78db222bba9b75cd0c12369d705a35b1419543e7911" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/openssl-selfsigned-cert", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/openssl-selfsigned-cert.json", + "job_name": "dev34-openssl-selfsigned-cert-main-20261012", + "config_sha256": "af2d6b612a65f9a87d0346d09155bfe61b19003a8138cd5ef42f8a5012e51f6a" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pypi-server", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pypi-server.json", + "job_name": "dev34-pypi-server-main-20261012", + "config_sha256": "54fb4ff0449fe8bbde098aa3d0dddc49c5a83fdfa95679da24ab243115ace3c3" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-cli", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-cli.json", + "job_name": "dev34-pytorch-model-cli-main-20261012", + "config_sha256": "4efaeeb38d6124f5820cd38b63d7021f6d6bf8508afdeca16d0a0ea6b8b49165" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/pytorch-model-recovery.json", + "job_name": "dev34-pytorch-model-recovery-main-20261012", + "config_sha256": "8b6262a3678ed3d354d73bd1e99fe9518a766559bdbfd9a39f040edba86857e7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/qemu-alpine-ssh", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/qemu-alpine-ssh.json", + "job_name": "dev34-qemu-alpine-ssh-main-20261012", + "config_sha256": "58f37eb01ac75ffd5c4fc83d19c27b0018406fbecbb5481d24415a53d91eaafd" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/query-optimize", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/query-optimize.json", + "job_name": "dev34-query-optimize-main-20261012", + "config_sha256": "f1a466eceb8c0d9d32550fbf67f2b8e0de3aa0da268df919ddf360b2900dc97d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-log", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-log.json", + "job_name": "dev34-regex-log-main-20261012", + "config_sha256": "87c7135b321bdba30ddf59a860b5169078b4e6f5e6a4363265c167574da7fdc7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-pipeline-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-pipeline-parallelism.json", + "job_name": "dev34-torch-pipeline-parallelism-main-20261012", + "config_sha256": "8fa84a9022a9009b9ed4416cd5defcd73fbb7d5fbccc640cb528af379f81265a" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-tensor-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/torch-tensor-parallelism.json", + "job_name": "dev34-torch-tensor-parallelism-main-20261012", + "config_sha256": "968407435cbc131e766b583aa88b1e1b676d2a977157344b86a460d03632a23c" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/write-compressor", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/write-compressor.json", + "job_name": "dev34-write-compressor-main-20261012", + "config_sha256": "b47a2f35415400685be9361695b76fcf12478061e8912d96ebc4c2ae44baa996" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/constraints-scheduling", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/constraints-scheduling.json", + "job_name": "dev34-constraints-scheduling-main-20261012", + "config_sha256": "b24ec3c2e9c81ef502405c942f4f3fe1fe4960f7eb85c3d5b95726b970f152ea" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/custom-memory-heap-crash", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/custom-memory-heap-crash.json", + "job_name": "dev34-custom-memory-heap-crash-main-20261012", + "config_sha256": "3d3224f223b303b48317b38ee89d9f9064d35ca02eae7650d259428e17512ef1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/dna-assembly", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dna-assembly.json", + "job_name": "dev34-dna-assembly-main-20261012", + "config_sha256": "3c0d642744f904377d4e7cf8f8e53b2936085da4434964c7ecf876d1fb7a90b5" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/llm-inference-batching-scheduler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/llm-inference-batching-scheduler.json", + "job_name": "dev34-llm-inference-batching-scheduler-main-20261012", + "config_sha256": "0bb3d624b3f196781655c06d80088ec2d8f5f0109d04da9977cb0f06266f51ec" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/path-tracing", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/path-tracing.json", + "job_name": "dev34-path-tracing-main-20261012", + "config_sha256": "6a740f889c720c9913e63eaa41334b8049832f0474e5f92bfc0c7b01224a8ce7" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/schemelike-metacircular-eval", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/schemelike-metacircular-eval.json", + "job_name": "dev34-schemelike-metacircular-eval-main-20261012", + "config_sha256": "e84ee7384f1f55c2681168e35694de493a71dfba84eebf308f36099a8ccc356e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/caffe-cifar-10", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/caffe-cifar-10.json", + "job_name": "dev34-caffe-cifar-10-main-20261012", + "config_sha256": "cf47d82d7f104c58fe6876c81e8a16f47548c8430f2784d699320cfba5d0658a" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/circuit-fibsqrt", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/circuit-fibsqrt.json", + "job_name": "dev34-circuit-fibsqrt-main-20261012", + "config_sha256": "da5aae09f101bc7d51a554d9f96e810e97d7f9875379c063231b6c9f6e471335" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/mteb-leaderboard", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/mteb-leaderboard.json", + "job_name": "dev34-mteb-leaderboard-main-20261012", + "config_sha256": "e073768039f0f301df43354c723456d209c9ba18e9c9e3c93d302f64731607cf" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-chess", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/regex-chess.json", + "job_name": "dev34-regex-chess-main-20261012", + "config_sha256": "f3009de8fa16d81ad35549de4dff52ba446003a822ffa9fa72752f4543953aa6" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-pov-ray", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/build-pov-ray.json", + "job_name": "dev34-build-pov-ray-main-20261012", + "config_sha256": "ffad0ecc6df9b03b983420179da25733d874b33af348a8a605d24b551a6f7e2e" + } + ], + "log_dir": "jobs/dev34-main-baseline-20261012-v4-orchestration", + "image_digests_file": "image-digests.json", + "image_digests_sha256": "5fc06fa5fbac1cccd3a4fb2f560878dd79ce314a6b1005183bbdcb4e8e13381f", + "budget_policy": "Native task agent timeout minus 180 seconds, separate 1800-second setup. Native verifier per-attempt time limits; max three attempts, retry only timeouts against the same agent output.", + "execution_policy": "All 34 model trials are fresh. Fixed ascending work-budget order, then task name; consecutive pairs finish before the next pair. No whole-trial retries, task replacements, or reuse of previous model outputs.", + "failure_policy": "Keep all setup failures and verifier errors separate from scored failures. Retain native zeros even when assertions never start, but classify them as verifier errors in analysis.", + "preflight_policy": "Fresh installation-only preflight for all 34 tasks before model work. QEMU has a reviewed verifier dependency profile; the other 33 are explicitly not_configured. A pre-model infrastructure amendment may reuse successful native installation trials only with complete hash-bound provenance and retain all failures.", + "information_policy": "Use frozen current main without prompt or runtime tuning. Do not inspect new trajectories or hidden test contents until all new model outputs freeze. Prior development exposure is disclosed, and each completed trajectory will be audited for benchmark-answer exposure.", + "scope": "One common-main baseline of the existing 34-task development pool, not a comparison with prior mixed-version results and not a random or held-out benchmark.", + "additional_file_sha256": { + "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf": "d0e1ffb6dbd02c54978ba2df386326311a59f6d3774990180f22bcc6adfaedbc", + "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json": "af3b61b52b111bcde4ab47adfa2d562e5d6740d9db803a3c335b976e689149ec", + "benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest.json": "f4da90888edbec5f7e90ba9f4a0622229c654042a54275f3b50d700c1f7a0e35", + "benchmarks/harbor/scripts/apt_index_proxy.py": "bc15f7adf1d7968d627ceca4683ab86742a7188a1b1955ba1fef37ba48e8a936", + "benchmarks/harbor/scripts/cache_image.py": "639709c03a7b0c3eb6aa78b4e87e6184b49b4b102853a31e33dd832d759180c0", + "benchmarks/harbor/scripts/uv_release_cache.py": "993cc35b1f3a9e8e171b662ab6d4524c5fe059f0d1f0e275a5bb1930111e0ff5", + "benchmarks/harbor/scripts/run_comparison.py": "04f2106f0099a7b4d77e9206a36767f98a5e92c875ec78dd2de858c4936c12a8" + }, + "protocol_amendment": { + "prior_plan": "experiment-v3.json", + "prior_plan_sha256": "0cbbc776680ed1f3b0ca81a804c67ff8c46f49c01987a6b037900ab918e1f680", + "reason": "Preserve completed installation evidence and perform fresh installation-only checks for unresolved tasks. Add a verified cache of the original uv and managed-Python archives after repeated truncated downloads. No model outputs exist; agent/adapter source, images, installer scripts, package versions/content, budgets, and model queue order are unchanged.", + "transport": "Original APT cache plus a read-only uv release archive sidecar. UV_INSTALLER_GITHUB_BASE_URL and UV_PYTHON_INSTALL_MIRROR are set for the unchanged official installer; normal model traffic retains its original route. Cached filename, size, and SHA256 match original release metadata; uncached versions redirect to original GitHub URLs.", + "install_only_retry_tasks": [ + "terminal-bench/log-summary-date-ranges", + "terminal-bench/qemu-alpine-ssh", + "terminal-bench/regex-log", + "terminal-bench/circuit-fibsqrt", + "terminal-bench/build-pov-ray" + ], + "model_runs_started_before_amendment": 0, + "model_gate": "Do not run the model phase with this partial-preflight plan. First register exactly one successful native installation per task in an accepted-evidence plan and verify all 34 with the strict existing runner." + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only-v4.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only-v4.json new file mode 100644 index 0000000..80829cc --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/install-only-v4.json @@ -0,0 +1,45 @@ +{ + "n_concurrent_trials": 2, + "n_attempts": 1, + "install_only": true, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-before-v4-evidence.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-before-v4-evidence.json new file mode 100644 index 0000000..a4676b5 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/preflight-before-v4-evidence.json @@ -0,0 +1,664 @@ +{ + "recorded_at": "2026-10-12T17:11:15.471240+00:00", + "prior_plan_sha256": "0cbbc776680ed1f3b0ca81a804c67ff8c46f49c01987a6b037900ab918e1f680", + "native_successful_tasks": [ + "terminal-bench/adaptive-rejection-sampler", + "terminal-bench/build-cython-ext", + "terminal-bench/caffe-cifar-10", + "terminal-bench/cancel-async-tasks", + "terminal-bench/cobol-modernization", + "terminal-bench/constraints-scheduling", + "terminal-bench/custom-memory-heap-crash", + "terminal-bench/db-wal-recovery", + "terminal-bench/dna-assembly", + "terminal-bench/fix-code-vulnerability", + "terminal-bench/git-multibranch", + "terminal-bench/kv-store-grpc", + "terminal-bench/llm-inference-batching-scheduler", + "terminal-bench/merge-diff-arc-agi-task", + "terminal-bench/model-extraction-relu-logits", + "terminal-bench/mteb-leaderboard", + "terminal-bench/multi-source-data-merger", + "terminal-bench/nginx-request-logging", + "terminal-bench/openssl-selfsigned-cert", + "terminal-bench/path-tracing", + "terminal-bench/pypi-server", + "terminal-bench/pytorch-model-cli", + "terminal-bench/pytorch-model-recovery", + "terminal-bench/query-optimize", + "terminal-bench/regex-chess", + "terminal-bench/schemelike-metacircular-eval", + "terminal-bench/torch-pipeline-parallelism", + "terminal-bench/torch-tensor-parallelism", + "terminal-bench/write-compressor" + ], + "unresolved_tasks": [ + "terminal-bench/log-summary-date-ranges", + "terminal-bench/qemu-alpine-ssh", + "terminal-bench/regex-log", + "terminal-bench/circuit-fibsqrt", + "terminal-bench/build-pov-ray" + ], + "model_runs_started": 0, + "trials": [ + { + "task": "terminal-bench/adaptive-rejection-sampler", + "trial": "adaptive-rejection-sampler__dLzh39D", + "native_result_present": true, + "ready": false, + "exception_type": "AgentSetupTimeoutError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/adaptive-rejection-sampler__dLzh39D/result.json", + "result_sha256": "fc665e3c6e3b9fbade7c91b12639dca41e7759eb6639fef8fd4de2831e46b16e", + "bootstrap_sha256": "82878d7f8a339f6a19b1130e8160cbd8e23b500246887197f67728670a0e05a8", + "config_sha256": "c9f11d8a083ad237e7511257a27c55ebbea7148689bc1dc181b0e837d5536a99" + }, + { + "task": "terminal-bench/build-cython-ext", + "trial": "build-cython-ext__eo75cET", + "native_result_present": true, + "ready": false, + "exception_type": "NetworkConnectionError", + "stage": "agent_install", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/build-cython-ext__eo75cET/result.json", + "result_sha256": "9c08f2a8fe61ae948d16f3a1447caef2b5a17273666e578c0e0d3a38b698f732", + "bootstrap_sha256": "20c2b4bb8ee3bc17ebe99803d8086c15086ef9d07b0e9efb10813b47ef094842", + "config_sha256": "9fe0e9ff40ff73eda6853823c5f384f1c50ec84ea27657aaa491b8573743ae53" + }, + { + "task": "terminal-bench/cancel-async-tasks", + "trial": "cancel-async-tasks__G8BCdbW", + "native_result_present": true, + "ready": false, + "exception_type": "AgentSetupTimeoutError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/cancel-async-tasks__G8BCdbW/result.json", + "result_sha256": "2bb1ef9e0db51c408d0db621f4afa1cf96b239165d73011b6517b1ebffbf2642", + "bootstrap_sha256": "82878d7f8a339f6a19b1130e8160cbd8e23b500246887197f67728670a0e05a8", + "config_sha256": "fd1e813e2a22e094a2e122bbe7a62ebf5b4801166b743013dc51f59228c1626f" + }, + { + "task": "terminal-bench/cobol-modernization", + "trial": "cobol-modernization__eZhrhmk", + "native_result_present": true, + "ready": false, + "exception_type": "NetworkConnectionError", + "stage": "agent_install", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/cobol-modernization__eZhrhmk/result.json", + "result_sha256": "9cf63cccf003bdbe7d434387ce8736dc4a0bea392febe9ee2c52c8bd0136dbcd", + "bootstrap_sha256": "c33d1442bdecb3f9e11d612e02b446d14ccbd0f7aed1108a4a0ff707ddd38fa4", + "config_sha256": "7703d25e0e634c7b9d56899c62da12787e74fae3c95c4c242069064273c9a1ec" + }, + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__U44UuJX", + "native_result_present": true, + "ready": false, + "exception_type": "AgentSetupTimeoutError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/db-wal-recovery__U44UuJX/result.json", + "result_sha256": "ff7b455694c6d2f26f46014de0983b7ae86783dd465a8fb2af8e79a94b7e8fa7", + "bootstrap_sha256": "82878d7f8a339f6a19b1130e8160cbd8e23b500246887197f67728670a0e05a8", + "config_sha256": "d79f3a8c050eeaf2ca1fa2db456b5d73e3b7d40848b65811b579ff28c8facc13" + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "trial": "fix-code-vulnerability__eY3pEt8", + "native_result_present": true, + "ready": false, + "exception_type": "CancelledError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-20261012/fix-code-vulnerability__eY3pEt8/result.json", + "result_sha256": "d29687f27df846b6b1abbe53d36096bf044134e1c1fdcb1a4262eb0db2dad35d", + "bootstrap_sha256": "82878d7f8a339f6a19b1130e8160cbd8e23b500246887197f67728670a0e05a8", + "config_sha256": "c1a93436b00035f41d8f387e927240adb0935be19e2f685748beb126e0222204" + }, + { + "task": "terminal-bench/adaptive-rejection-sampler", + "trial": "adaptive-rejection-sampler__8GXdagL", + "native_result_present": true, + "ready": false, + "exception_type": "NonZeroAgentExitCodeError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/adaptive-rejection-sampler__8GXdagL/result.json", + "result_sha256": "47f9f35a2402c483670b2d85888969a4693269b5218cb0786ef6ac411d8369cc", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b", + "config_sha256": "f40de0f0f93370e52613a245cba01d8e49b36c5c7ee40a594778648fb9381264" + }, + { + "task": "terminal-bench/build-cython-ext", + "trial": "build-cython-ext__bKW8wMh", + "native_result_present": true, + "ready": false, + "exception_type": "NonZeroAgentExitCodeError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/build-cython-ext__bKW8wMh/result.json", + "result_sha256": "19d978b009a3e9b3cd67c77481792a17c27ae51d80e2830c628f02c254334c5b", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b", + "config_sha256": "5c2d4ac07390cfe82fc3ed716f47fb68cbaca845cac2e9452bda2984cfe77a59" + }, + { + "task": "terminal-bench/cancel-async-tasks", + "trial": "cancel-async-tasks__Br23k5i", + "native_result_present": true, + "ready": false, + "exception_type": "NonZeroAgentExitCodeError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/cancel-async-tasks__Br23k5i/result.json", + "result_sha256": "b7f9ac46651382ec8bf8f76e6d07631058adf32d5948efcba92255a86ea61474", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b", + "config_sha256": "2ad9bd8e5ea4dc08abdd5a1455d02c1079e9db981fa12a6492dcbb099abaf1ea" + }, + { + "task": "terminal-bench/cobol-modernization", + "trial": "cobol-modernization__7zgjdZj", + "native_result_present": true, + "ready": false, + "exception_type": "NetworkConnectionError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/cobol-modernization__7zgjdZj/result.json", + "result_sha256": "b2ddf6e87e017fb711d9ee031e7307edde820da541550a51294c9cc6aeaea173", + "bootstrap_sha256": "079f9d13e4af52880b5a1dd43c0432e9f1208b64fc6da106b3442630ac014e72", + "config_sha256": "a3856367eae10ae652b2b219c1ea854e56969110e3f7bcbbee7406dcc5b975a5" + }, + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__zpZVSsu", + "native_result_present": true, + "ready": false, + "exception_type": "NonZeroAgentExitCodeError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/db-wal-recovery__zpZVSsu/result.json", + "result_sha256": "9776b92c1b1fb694b7feb3d6317ef698efaadcd804e5495246a240ba27144595", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b", + "config_sha256": "6470a0e1ef24b59c0dc9bab6764ac0eaed7dacb210bdd677130307f98f936155" + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "trial": "fix-code-vulnerability__RHxqE9D", + "native_result_present": true, + "ready": false, + "exception_type": "NetworkConnectionError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/fix-code-vulnerability__RHxqE9D/result.json", + "result_sha256": "1d93ff9e819acd22331252c41fe41c8e0de2d0bf08e4a7ae6af1f0ba0bce9d5f", + "bootstrap_sha256": "7058299ec0c6f256d8eecf9a0da6b205ad7abf779e297bf17c9a230e10bb9ba6", + "config_sha256": "522ee4cb3c1f34fa9e67b810d66671a70ec35fd8e9b0b6b4aa283b643286ed01" + }, + { + "task": "terminal-bench/git-multibranch", + "trial": "git-multibranch__Ev3aCQ6", + "native_result_present": false, + "ready": false, + "exception_type": null, + "stage": null, + "result_path": null, + "result_sha256": null, + "bootstrap_sha256": null, + "config_sha256": "32a2f147eaa242e5c68ab921261af8f76983a5cf965800b2d8947a13f12b38bd" + }, + { + "task": "terminal-bench/kv-store-grpc", + "trial": "kv-store-grpc__2aB6kU7", + "native_result_present": true, + "ready": false, + "exception_type": "NonZeroAgentExitCodeError", + "stage": "system_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v2-20261012/kv-store-grpc__2aB6kU7/result.json", + "result_sha256": "b5abbaa52933dcb7791325b8df07e2b6336733a2c8c8508c4affe2b54c990f02", + "bootstrap_sha256": "a25f8ddb723b047f846596743973e13e470ee206b78070e19bc064f9d4ce0d0b", + "config_sha256": "5474477a8fad30e7e6ce0fbc7236e1b80b2158e784a57cfb18075e590607d974" + }, + { + "task": "terminal-bench/log-summary-date-ranges", + "trial": "log-summary-date-ranges__Ppd7Nns", + "native_result_present": false, + "ready": false, + "exception_type": null, + "stage": null, + "result_path": null, + "result_sha256": null, + "bootstrap_sha256": null, + "config_sha256": "23b1a0ee37a691be4920a8206bef15327d0dbaf6306592ab031b475332308a37" + }, + { + "task": "terminal-bench/adaptive-rejection-sampler", + "trial": "adaptive-rejection-sampler__5hjbBMn", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/adaptive-rejection-sampler__5hjbBMn/result.json", + "result_sha256": "3b6bfd13a6085baddbf8a9af3c55aa18b846e73abcac3f96a338f1cfb67b071e", + "bootstrap_sha256": "2adabcd7c4579775ee5370279e0e22ba6b11727a8084c156294b2ba124cf93a5", + "config_sha256": "7425ba7d31afdfc06efbb045bbdcb18c4af0fc4951fd7bc58327668aeac3d237" + }, + { + "task": "terminal-bench/build-cython-ext", + "trial": "build-cython-ext__88GjJpX", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/build-cython-ext__88GjJpX/result.json", + "result_sha256": "f86a832a9cc79551923a7afe79cdd5e92152afaa3e5abe64c4969cf1ccf2d71e", + "bootstrap_sha256": "63aa919cdf558f74286e12bcf6a3f154c7c2a1d9c89352e56303243af7950909", + "config_sha256": "0a8207137bceed0ed48fef4d0a1ad18cbcdd560c8b4f0e7a88f5393ad4359838" + }, + { + "task": "terminal-bench/build-pov-ray", + "trial": "build-pov-ray__FnvvxMD", + "native_result_present": false, + "ready": false, + "exception_type": null, + "stage": null, + "result_path": null, + "result_sha256": null, + "bootstrap_sha256": null, + "config_sha256": "56de0aa9db80d370730ad8a82f3bd0c81b051168a8437c1af46f1bd9de1efe0b" + }, + { + "task": "terminal-bench/caffe-cifar-10", + "trial": "caffe-cifar-10__QUGdMuN", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/caffe-cifar-10__QUGdMuN/result.json", + "result_sha256": "9b3f7ca9a37f1ed5f480e90cbe8893e1ad16e9e85cbeabe69a1442903766354a", + "bootstrap_sha256": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4", + "config_sha256": "0edb95ac57d33c964e20a73fb49b42f1ee4518b43c171d42f220ee7778107032" + }, + { + "task": "terminal-bench/cancel-async-tasks", + "trial": "cancel-async-tasks__HCzTua9", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/cancel-async-tasks__HCzTua9/result.json", + "result_sha256": "f3dbe582c6b0eac1d5abfe161dffe715a822c509f33badc67946d95dbb6e3397", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "config_sha256": "62286dd7c93a5667eeeec8d5c6c50ab4d7362b3d2fade833091c03775bfd3bd6" + }, + { + "task": "terminal-bench/circuit-fibsqrt", + "trial": "circuit-fibsqrt__RvJL8dy", + "native_result_present": false, + "ready": false, + "exception_type": null, + "stage": null, + "result_path": null, + "result_sha256": null, + "bootstrap_sha256": null, + "config_sha256": "06a7603789d43f705ce3faf6a04bb25eab717ea99955f536896a061bf192c78d" + }, + { + "task": "terminal-bench/cobol-modernization", + "trial": "cobol-modernization__aFZfWmu", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/cobol-modernization__aFZfWmu/result.json", + "result_sha256": "4204833ab5c63a62c1cf80ece0dd0c97b17656131df206367a2af86b61e997d0", + "bootstrap_sha256": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "config_sha256": "bd12c87ab92435df05fd8d20404f5f585276e9c9988b602084bbc82dc040e4fc" + }, + { + "task": "terminal-bench/constraints-scheduling", + "trial": "constraints-scheduling__uAF3dBy", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/constraints-scheduling__uAF3dBy/result.json", + "result_sha256": "59a970f740b7eeef09f12ca7b67e188bd12eaca5236c52e6d81197affbd167d5", + "bootstrap_sha256": "8ec0c76eef3467924ce96d81cc262bdea7775b259bbeeca2a0563a292696eb05", + "config_sha256": "5320fadf4b16e51ac32404610da832e7c2acf0e9f6fe2196a6ffd8bd1c10997a" + }, + { + "task": "terminal-bench/custom-memory-heap-crash", + "trial": "custom-memory-heap-crash__g5FutcN", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/custom-memory-heap-crash__g5FutcN/result.json", + "result_sha256": "83888f45d2814ba187bb9768ed323d151486f173697441c6eede910faebe3c21", + "bootstrap_sha256": "cf6bb84c42557ba2a7b7e0ec653bbc1c7cf86c236d1b7e2a6852918dfb8eb7dc", + "config_sha256": "50841911b59d3d802d88efd25c8f16bdf34d313e63f0069f8c3b71874fa247ec" + }, + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__6MChPGU", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/db-wal-recovery__6MChPGU/result.json", + "result_sha256": "2f6c09fec6385adc449183d61cd0fc627d106a139ef26f55705a03ff0c281322", + "bootstrap_sha256": "2ffe3efbf9fd9f423c3fc82102152f59af7f9180f8aaf39c46e61c96e2f02d76", + "config_sha256": "87bb8a0577b16894d59539ff8ea94dacca9c4f1693ef36065ae4fd754dee9d3d" + }, + { + "task": "terminal-bench/dna-assembly", + "trial": "dna-assembly__n2emDnw", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/dna-assembly__n2emDnw/result.json", + "result_sha256": "45c66c409797157136716053aa695e9330295d7865ee6756b861c49753d1750d", + "bootstrap_sha256": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "config_sha256": "6514104fde6444ca05d17c862c584d2beee642daca58cba6dc79a2ce67ca6cf3" + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "trial": "fix-code-vulnerability__RvHhCBe", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/fix-code-vulnerability__RvHhCBe/result.json", + "result_sha256": "d7ff8ef09f9d0eb3cfd0dcddcb5f29c69b2f8a0e711256511782806f735fd851", + "bootstrap_sha256": "8d1fa3e359c1a2c87bb9b262c9113a423af3445aa5a20da8a2e6fb4d63b4f67a", + "config_sha256": "80b9f8c6e2fe9ed0716e3b3137870e7d57925d7843aac0f8dcdfa7936fdb4c1c" + }, + { + "task": "terminal-bench/git-multibranch", + "trial": "git-multibranch__cbPbsmj", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/git-multibranch__cbPbsmj/result.json", + "result_sha256": "55a9a2c3d01c026a1a9a286b98fb04b97fff5c8dfcab598ecbfe0d7b6045f4d6", + "bootstrap_sha256": "391e000959d10d8aa551310ebe7bddd0c70185a684ee8cb543b7e8bf57e5e63b", + "config_sha256": "d05d690b47fbd52a2a2705454b3943e6f1b7abd5bf377c2ad64b3afeaa6c1cf2" + }, + { + "task": "terminal-bench/kv-store-grpc", + "trial": "kv-store-grpc__oJdqesS", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/kv-store-grpc__oJdqesS/result.json", + "result_sha256": "b42dbdc907ee1a6868c93f22a8a3ccb52fad14c6c9dba571fb76ae7b1766b761", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "config_sha256": "344824abed4da1371c8297604817122521ccdccf996f7095e734a25192da653d" + }, + { + "task": "terminal-bench/llm-inference-batching-scheduler", + "trial": "llm-inference-batching-scheduler__sUbNmpr", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/llm-inference-batching-scheduler__sUbNmpr/result.json", + "result_sha256": "b5daf0452dd173f3aac353a5edc9a58c6bd4e96a3120f7270c933fd43798a9f5", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "config_sha256": "10d22f8700f7313fe62cb19554846f4084fa5733ae73e1c797932f6679590533" + }, + { + "task": "terminal-bench/log-summary-date-ranges", + "trial": "log-summary-date-ranges__gB2sdP7", + "native_result_present": true, + "ready": false, + "exception_type": "NetworkConnectionError", + "stage": "agent_install", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/log-summary-date-ranges__gB2sdP7/result.json", + "result_sha256": "f410a95c652596adbe1db19d26d1df5cb78b439427dfb8dadd51ffe13223a1c5", + "bootstrap_sha256": "900b391964f7279e62d12ac815893963f5e3d02302b7c672403a5543388d2623", + "config_sha256": "0c16dec8043a2ccc8bb410fade61949a8348e9f5cfd50a4ed8e72e7a003c607f" + }, + { + "task": "terminal-bench/merge-diff-arc-agi-task", + "trial": "merge-diff-arc-agi-task__6DDLwSY", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/merge-diff-arc-agi-task__6DDLwSY/result.json", + "result_sha256": "7247da72307a6116b4576f71e8812a957a34256e682d21544fb035f9e6c0215b", + "bootstrap_sha256": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "config_sha256": "10ab7f8b5516c493a6e4cf718ddb4f366245d7e19b0ec3b7cced59b7e5c8d645" + }, + { + "task": "terminal-bench/model-extraction-relu-logits", + "trial": "model-extraction-relu-logits__VWhPix2", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/model-extraction-relu-logits__VWhPix2/result.json", + "result_sha256": "fae719c637528aeaa52cda57d02308e92721c97a93c6eb37af1b0319e4630f28", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "config_sha256": "18f79036c447e16654c4676e261b6f270f8fa02dd1db45dec9e102e83ffcae6f" + }, + { + "task": "terminal-bench/mteb-leaderboard", + "trial": "mteb-leaderboard__RVuUyDg", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/mteb-leaderboard__RVuUyDg/result.json", + "result_sha256": "ed5bec732ef3c67264700698308062f8aee2a161747eae5626c6719f8114cca5", + "bootstrap_sha256": "7c324cb3af6adf375e27d5cefcc67e8631dae65ac0f3c52687f893fa69b96c4e", + "config_sha256": "2531a9725721f919533b2b64acdbb4bb7e1dd73a18a0c51bb913f1ff85fed2eb" + }, + { + "task": "terminal-bench/multi-source-data-merger", + "trial": "multi-source-data-merger__Bm3JhqS", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/multi-source-data-merger__Bm3JhqS/result.json", + "result_sha256": "7e5daa82261b4b6416209c03bf23a532318e34592905c0c9e093747c18b5fc72", + "bootstrap_sha256": "43369761e1c62328eb0c3890c99d441d724fe366a178abf9c568928eb85c4174", + "config_sha256": "51ccefad26c36f70c9768899ee4067eace9f154ab65a89fe1244d4322434bcc6" + }, + { + "task": "terminal-bench/nginx-request-logging", + "trial": "nginx-request-logging__4X3hfes", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/nginx-request-logging__4X3hfes/result.json", + "result_sha256": "b7bec96262fb776976669e20710f5b0583b4556f577877b85d683e1b8a149eb0", + "bootstrap_sha256": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "config_sha256": "efa7829e411cc7e9ea89142cb622f49264cd4dfda6922e8a4b05a0c92138a976" + }, + { + "task": "terminal-bench/openssl-selfsigned-cert", + "trial": "openssl-selfsigned-cert__v5cNHDt", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/openssl-selfsigned-cert__v5cNHDt/result.json", + "result_sha256": "d8747031f715cb88df326b1517818ac30dcd603d0b52f0dcf45a8d3d554031c1", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "config_sha256": "ce0d672447f71c9ed169385ea290d2a3bc58dad8b1379215c98d3d4df7f897ba" + }, + { + "task": "terminal-bench/path-tracing", + "trial": "path-tracing__dVMT66x", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/path-tracing__dVMT66x/result.json", + "result_sha256": "f0d13abee1c7612b2ecb09ad78a210d21d2bc9b381f8dc555725520d5a65d25c", + "bootstrap_sha256": "f7c768324bf6e26e7f52bdec891623e4fc98e29f1f24ca02e3847723d8cb55bb", + "config_sha256": "e95e6495b7fcd237b631751f358e3be3ceb5bc427fba6f498c8bc8713110a098" + }, + { + "task": "terminal-bench/pypi-server", + "trial": "pypi-server__CFroJaU", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/pypi-server__CFroJaU/result.json", + "result_sha256": "de8683222335d13b0da701fe5b1c60d44ace7bd92c687afc595bcb97ab425a19", + "bootstrap_sha256": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "config_sha256": "7315e22c88302dde990deca959ebe225afda559491427c94a45030ab02d2929f" + }, + { + "task": "terminal-bench/pytorch-model-cli", + "trial": "pytorch-model-cli__DoBVGUe", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/pytorch-model-cli__DoBVGUe/result.json", + "result_sha256": "9a8729566b1c20cea1a04831c49930fde3c014138a5dbb1d836da7949d020750", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "config_sha256": "c05225556d9ee3ed825283a7900611d6a3983a087204806d2cd2a62bbaaf4689" + }, + { + "task": "terminal-bench/pytorch-model-recovery", + "trial": "pytorch-model-recovery__VjeKRPp", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/pytorch-model-recovery__VjeKRPp/result.json", + "result_sha256": "02cdb595a242ba446a09c3ef35c4c9e660225bf709ccf82187ea3692f3bca9ba", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "config_sha256": "534cac86cbecd723c96d98a5e893aabb843f47c89d9f1e3828ba9b32aaa6b896" + }, + { + "task": "terminal-bench/qemu-alpine-ssh", + "trial": "qemu-alpine-ssh__XkBzp28", + "native_result_present": true, + "ready": false, + "exception_type": "NonZeroAgentExitCodeError", + "stage": "verifier_dependencies", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/qemu-alpine-ssh__XkBzp28/result.json", + "result_sha256": "a6ee19abf5b4fa0887163564be0c2004de29201244009cf50b9b1fe63af3b900", + "bootstrap_sha256": "44255185320dae8a7fce3e03a7f20e41d1093a10df051503b104ae70b7f6a516", + "config_sha256": "705baa403b91b31fd259d5336144bd5a6eb99a90b84f6359c36a4fb167911da0" + }, + { + "task": "terminal-bench/query-optimize", + "trial": "query-optimize__frWo4fV", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/query-optimize__frWo4fV/result.json", + "result_sha256": "ea08b7612ff50f52e71c0427b2d5de603328351b1d4e6ef1b7f52f066e15f866", + "bootstrap_sha256": "c72c3ff607d38944df0e65d64c3a0a941fdc52c3dc17db638b246035ca2d8f89", + "config_sha256": "62a98a003a621cf0f3d7f7c1702bd8752a8dea20e1d43b4ef9e9751eb4ec70b7" + }, + { + "task": "terminal-bench/regex-chess", + "trial": "regex-chess__5SfYMHh", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/regex-chess__5SfYMHh/result.json", + "result_sha256": "0e519db59b54fe485678c721a78b8472d4a9fcb782cd79df9198652c6a9232fb", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "config_sha256": "d3a469cf23a63c9ff35aed1b2f7b617b6d47152f337d8ba0ec71fb46e2a4a9de" + }, + { + "task": "terminal-bench/regex-log", + "trial": "regex-log__gFFxNhF", + "native_result_present": true, + "ready": false, + "exception_type": "NetworkConnectionError", + "stage": "agent_install", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/regex-log__gFFxNhF/result.json", + "result_sha256": "3708bc980061aa792e3920c73e844274c48645054d9e75e9014d71c665730d83", + "bootstrap_sha256": "8834d987758c0df33d3fdd3ee6df628bac34c0541f331a6a574ae37bc29b305e", + "config_sha256": "4b771217af226ceee5805b0cfb6021103167b6e872314fd97445c0a234382e6d" + }, + { + "task": "terminal-bench/schemelike-metacircular-eval", + "trial": "schemelike-metacircular-eval__2XMSZho", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/schemelike-metacircular-eval__2XMSZho/result.json", + "result_sha256": "6d05110a8471af2db983de5cfaaac869c705b1fb0eaa702e03b39389054ad78e", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "config_sha256": "c793d8b8112f6dedf75c19771e6bbe40f5cc807c88ea9d95abb2b01e8eaa3ecf" + }, + { + "task": "terminal-bench/torch-pipeline-parallelism", + "trial": "torch-pipeline-parallelism__bSqFxiu", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/torch-pipeline-parallelism__bSqFxiu/result.json", + "result_sha256": "d106e2ab5083bb214256f6db0ee819c686ebe7d4ddec8c5755bdfd63e33b51e9", + "bootstrap_sha256": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "config_sha256": "58d133e12a0b11a12053e901aa56ad0a7dcaab61396f895bbdd5b6b49062b300" + }, + { + "task": "terminal-bench/torch-tensor-parallelism", + "trial": "torch-tensor-parallelism__xzKwyM9", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/torch-tensor-parallelism__xzKwyM9/result.json", + "result_sha256": "ab71b2eb2c7d4f63fe9df8fe2899beb1582e43a56f850e83af8141b349678511", + "bootstrap_sha256": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "config_sha256": "f44a88d9d7e9556fd4e8e04aed074c41d91769250ecb84a10741ba7d9821c60b" + }, + { + "task": "terminal-bench/write-compressor", + "trial": "write-compressor__gn7TLQW", + "native_result_present": true, + "ready": true, + "exception_type": null, + "stage": "complete", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/write-compressor__gn7TLQW/result.json", + "result_sha256": "63d5bf1fbc13df02d82f5dc6a1050a04fca46d7151f917de3b037fcab4707579", + "bootstrap_sha256": "78617809d0aabc5eaf4ecbb573bab09119e53b9c6a39a8890554717fa5823cf8", + "config_sha256": "2dda6994e91f2984afd84d414570b8597c4b7ad6b5b4d78ec10a0720b6dcd80f" + } + ], + "administrative_interruption_cleanup": { + "recorded_at": "2026-10-12T17:11:15.438714+00:00", + "trials": [ + { + "trial": "circuit-fibsqrt__RvJL8dy", + "config_sha256": "06a7603789d43f705ce3faf6a04bb25eab717ea99955f536896a061bf192c78d", + "native_result_present": false, + "reason": "Installation-only job interrupted to apply the verified uv/Python transport after three dependency network failures; no model or task assertions ran.", + "removed_containers": [ + "10cda5c127cc", + "648c2408789e" + ], + "removed_networks": [ + "21c70b780180" + ] + }, + { + "trial": "build-pov-ray__FnvvxMD", + "config_sha256": "56de0aa9db80d370730ad8a82f3bd0c81b051168a8437c1af46f1bd9de1efe0b", + "native_result_present": false, + "reason": "Installation-only job interrupted to apply the verified uv/Python transport after three dependency network failures; no model or task assertions ran.", + "removed_containers": [ + "3e4211a93548", + "119084734a35" + ], + "removed_networks": [ + "2b0a6af378ca" + ] + } + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/python-cache-integration-probe.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/python-cache-integration-probe.json new file mode 100644 index 0000000..fd6339f --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/python-cache-integration-probe.json @@ -0,0 +1,17 @@ +{ + "observed_at": "2026-10-12T17:08:56.647688+00:00", + "kind": "dependency_cache_integration_probe_not_a_benchmark_trial", + "main_image": "alexgshaw/git-multibranch@sha256:5bc449921c138295f6ddd8dca2977319fee2ce523811dbe56cec4ca663385fb5", + "sidecar_image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "settings": { + "UV_INSTALLER_GITHUB_BASE_URL": "http://127.0.0.1:18743", + "UV_PYTHON_INSTALL_MIRROR": "http://127.0.0.1:18743/astral-sh/python-build-standalone/releases/download" + }, + "dependency_request": "uvx -p 3.13 -w pytest==8.4.1 -w pytest-json-ctrf==0.3.5 pytest --version", + "observed_uv_version": "uv 0.9.5", + "observed_python_install": "cpython-3.13.9-linux-x86_64-gnu", + "observed_pytest_version": "pytest 8.4.1", + "manifest_sha256": "f4da90888edbec5f7e90ba9f4a0622229c654042a54275f3b50d700c1f7a0e35", + "utility_sha256": "993cc35b1f3a9e8e171b662ab6d4524c5fe059f0d1f0e275a5bb1930111e0ff5", + "scope": "Original uv version selection, installer scripts, and release contents; no model calls or task assertions." +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json index 413dbe3..9ddb17f 100644 --- a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v4/dependency-cache.compose.json @@ -10,7 +10,8 @@ } ], "environment": { - "UV_INSTALLER_GITHUB_BASE_URL": "http://127.0.0.1:18743" + "UV_INSTALLER_GITHUB_BASE_URL": "http://127.0.0.1:18743", + "UV_PYTHON_INSTALL_MIRROR": "http://127.0.0.1:18743/astral-sh/python-build-standalone/releases/download" } }, "apt-index-cache": { @@ -81,7 +82,7 @@ "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/uv_release_cache.py:/proxy/uv_release_cache.py:ro", "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/cache_image.py:/proxy/cache_image.py:ro", "/home/minix/projects/nanoPyCodeAgent/.cache/nanopy-harbor/uv-releases:/cache:ro", - "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/uv-release-manifest.json:/manifest.json:ro" + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest.json:/manifest.json:ro" ], "healthcheck": { "test": [ diff --git a/benchmarks/harbor/scripts/uv_release_cache.py b/benchmarks/harbor/scripts/uv_release_cache.py index 78c9a2f..3cfd81a 100644 --- a/benchmarks/harbor/scripts/uv_release_cache.py +++ b/benchmarks/harbor/scripts/uv_release_cache.py @@ -1,4 +1,4 @@ -"""Cache original uv release archives using GitHub's filename, size, and SHA256. +"""Cache uv and its managed Python using original release size and SHA256. The official installer remains unchanged. Its supported UV_INSTALLER_GITHUB_BASE_URL setting can point to the loopback server. Uncached @@ -15,6 +15,7 @@ import json from pathlib import Path import re +from urllib.parse import quote, unquote spec = importlib.util.spec_from_file_location( @@ -26,61 +27,87 @@ ASSET = "uv-x86_64-unknown-linux-gnu.tar.gz" VERSION = re.compile(r"\d+\.\d+\.\d+") RELEASE_PATH = re.compile( - r"/astral-sh/uv/releases/download/([0-9A-Za-z][0-9A-Za-z._+-]*)/" - r"(uv-[0-9A-Za-z._-]+\.(?:tar\.gz|zip)(?:\.sha256)?)", + r"/astral-sh/(?:uv|python-build-standalone)/releases/download/" + r"([0-9A-Za-z][0-9A-Za-z._+-]*)/" + r"((?:uv|cpython)-[0-9A-Za-z._+-]+\.(?:tar\.gz|zip)(?:\.sha256)?)", ) +PYTHON_ASSET = re.compile(r"cpython-[0-9][0-9a-z.+_-]*-x86_64-unknown-linux-gnu-install_only_stripped\.tar\.gz") -def asset_path(version, name=ASSET): - return f"/astral-sh/uv/releases/download/{version}/{name}" +def asset_path(version, name=ASSET, repository="astral-sh/uv"): + return f"/{repository}/releases/download/{quote(version, safe='')}/{quote(name, safe='')}" def release_asset(version, raw): if not VERSION.fullmatch(version): raise ValueError("Expected an exact stable uv version") + return original_asset("astral-sh/uv", version, ASSET, raw) + + +def python_asset(build, filename, raw): + if (not re.fullmatch(r"\d{8}", build) or not PYTHON_ASSET.fullmatch(filename) + or f"+{build}-" not in filename): + raise ValueError("Expected an exact original Python build and Linux archive") + return original_asset("astral-sh/python-build-standalone", build, filename, raw) + + +def original_asset(repository, version, filename, raw): release = json.loads(raw) if release.get("tag_name") != version: raise ValueError("Release tag differs from the requested version") - matches = [asset for asset in release["assets"] if asset["name"] == ASSET] + matches = [asset for asset in release["assets"] if asset["name"] == filename] if len(matches) != 1: raise ValueError("Expected exactly one original Linux archive") asset = matches[0] - url = "https://github.com" + asset_path(version) + url = "https://github.com" + asset_path(version, filename, repository) if (asset.get("browser_download_url") != url or not re.fullmatch(r"sha256:[0-9a-f]{64}", asset.get("digest") or "") or type(asset.get("size")) is not int or not 0 < asset["size"] <= 128 * 1024 * 1024): raise ValueError("Release archive lacks valid original provenance") - return dict(version=version, filename=ASSET, url=url, + entry = dict(version=version, filename=filename, url=url, digest=asset["digest"], size=asset["size"], asset_id=asset["id"], release_metadata_sha256=hashlib.sha256(raw).hexdigest()) + if repository != "astral-sh/uv": + entry["repository"] = repository + return entry -def prepare(cache, manifest, versions, workers=16): +def prepare(cache, manifest, versions, workers=16, python_build=None, python_filename=None): if manifest.exists(): raise ValueError("A registered release manifest cannot be overwritten") - if not versions or len(versions) != len(set(versions)): + if (not versions and not python_build) or len(versions) != len(set(versions)): raise ValueError("Specify distinct exact uv versions") if any(not VERSION.fullmatch(version) for version in versions): raise ValueError("Expected exact stable uv versions") + if bool(python_build) != bool(python_filename): + raise ValueError("Both Python build and archive are required") + if python_build and (not re.fullmatch(r"\d{8}", python_build) + or not PYTHON_ASSET.fullmatch(python_filename) + or f"+{python_build}-" not in python_filename): + raise ValueError("Invalid original Python release request") + requests = [("astral-sh/uv", version, ASSET) for version in versions] + if python_build: + requests.append(("astral-sh/python-build-standalone", python_build, python_filename)) cache.mkdir(parents=True, exist_ok=True) with (cache / ".prepare.lock").open("a") as lock: fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) entries = [] - for version in versions: - url = f"https://api.github.com/repos/astral-sh/uv/releases/tags/{version}" + for repository, version, filename in requests: + url = f"https://api.github.com/repos/{repository}/releases/tags/{version}" status, _, raw = ranges.get(url, { "User-Agent": "nanoPyCodeAgent-verified-cache", "Accept": "application/vnd.github+json", }, 4 * 1024 * 1024) if status != 200: raise ValueError("Original release metadata request failed") - entry = release_asset(version, raw) + entry = (release_asset(version, raw) if repository == "astral-sh/uv" + else python_asset(version, filename, raw)) metadata = cache / "metadata" / (entry["release_metadata_sha256"] + ".json") metadata.parent.mkdir(exist_ok=True) metadata.write_bytes(raw) ranges.fetch_blob(cache, entry, entry["url"], {}, workers, 256 * 1024) entries.append(entry) - print(json.dumps({"verified_version": version, "size": entry["size"], + print(json.dumps({"repository": repository, "verified_version": version, "size": entry["size"], "digest": entry["digest"]}), flush=True) manifest.parent.mkdir(parents=True, exist_ok=True) manifest.write_text(json.dumps({ @@ -100,9 +127,16 @@ def registered_assets(cache, manifest): raise ValueError("Invalid release metadata digest") metadata = cache / "metadata" / (entry["release_metadata_sha256"] + ".json") raw = metadata.read_bytes() - if release_asset(entry["version"], raw) != entry: + repository = entry.get("repository", "astral-sh/uv") + if repository == "astral-sh/uv": + original = release_asset(entry["version"], raw) + elif repository == "astral-sh/python-build-standalone": + original = python_asset(entry["version"], entry["filename"], raw) + else: + raise ValueError("Unsupported original release repository") + if original != entry: raise ValueError("Manifest differs from its original release metadata") - path = asset_path(entry["version"], entry["filename"]) + path = asset_path(entry["version"], entry["filename"], repository) if path in assets: raise ValueError("Duplicate registered release") blob = ranges.blob_path(cache, entry) @@ -125,13 +159,15 @@ def do_HEAD(self): def respond(self, head): self.close_connection = True - if not RELEASE_PATH.fullmatch(self.path): - self.send_error(403, "Only original uv release assets are supported") + path = unquote(self.path) + if not RELEASE_PATH.fullmatch(path): + self.send_error(403, "Only original uv and Python release assets are supported") return - cached = self.server.assets.get(self.path) + path = quote(path, safe="/-._~") + cached = self.server.assets.get(path) if cached is None: self.send_response(302) - self.send_header("Location", "https://github.com" + self.path) + self.send_header("Location", "https://github.com" + path) self.send_header("Content-Length", "0") self.send_header("Connection", "close") self.end_headers() @@ -184,11 +220,14 @@ def main(): parser.add_argument("--cache", type=Path, required=True) parser.add_argument("--manifest", type=Path, required=True) parser.add_argument("--version", action="append", default=[]) + parser.add_argument("--python-build") + parser.add_argument("--python-asset") parser.add_argument("--bind", default="127.0.0.1", choices=["127.0.0.1"]) parser.add_argument("--port", type=int, default=18743) args = parser.parse_args() if args.mode == "prepare": - prepare(args.cache, args.manifest, args.version) + prepare(args.cache, args.manifest, args.version, + python_build=args.python_build, python_filename=args.python_asset) else: assets = registered_assets(args.cache, args.manifest) server = ThreadingHTTPServer((args.bind, args.port), Handler) diff --git a/benchmarks/harbor/tests/test_uv_release_cache.py b/benchmarks/harbor/tests/test_uv_release_cache.py index 740f845..d6bcd5d 100644 --- a/benchmarks/harbor/tests/test_uv_release_cache.py +++ b/benchmarks/harbor/tests/test_uv_release_cache.py @@ -154,3 +154,68 @@ def test_http_corruption_fails_closed_after_server_start(server): blob.write_bytes(b"x" * blob.stat().st_size) status, _, body = request(server, uv.asset_path("0.9.11")) assert status == 502 and b"original uv archive" not in body + + +PYTHON_NAME = "cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz" + + +def python_metadata(payload=b"original managed Python"): + data = json.loads(metadata(payload, "20251014")) + data["assets"][0].update( + name=PYTHON_NAME, + browser_download_url="https://github.com" + uv.asset_path( + "20251014", PYTHON_NAME, "astral-sh/python-build-standalone"), + ) + return json.dumps(data).encode() + + +def test_python_archive_retains_exact_original_encoded_url(): + entry = uv.python_asset("20251014", PYTHON_NAME, python_metadata()) + assert entry["repository"] == "astral-sh/python-build-standalone" + assert "3.13.9%2B20251014" in entry["url"] + assert entry["filename"] == PYTHON_NAME + for build, name in [("latest", PYTHON_NAME), ("20251015", PYTHON_NAME), ("20251014", "../python.tar.gz")]: + with pytest.raises(ValueError, match="exact"): + uv.python_asset(build, name, python_metadata()) + + +def test_python_manifest_is_bound_to_metadata_and_bytes(tmp_path): + raw = python_metadata() + entry = uv.python_asset("20251014", PYTHON_NAME, raw) + (tmp_path / "metadata").mkdir() + (tmp_path / "metadata" / (entry["release_metadata_sha256"] + ".json")).write_bytes(raw) + blob = uv.ranges.blob_path(tmp_path, entry) + blob.parent.mkdir(parents=True) + blob.write_bytes(b"original managed Python") + manifest = tmp_path / "manifest.json" + manifest.write_text(json.dumps({"assets": [entry]})) + registered = uv.registered_assets(tmp_path, manifest) + assert next(iter(registered.values()))[0] == blob + blob.write_bytes(b"x" * blob.stat().st_size) + with pytest.raises(ValueError, match="SHA256"): + uv.registered_assets(tmp_path, manifest) + + +def test_python_mirror_preserves_unknown_original_downloads(server): + server, blob = server + path = uv.asset_path("20251014", PYTHON_NAME, "astral-sh/python-build-standalone") + server.assets[path] = (blob, uv.release_asset("0.9.11", metadata())) + assert request(server, path)[2] == blob.read_bytes() + assert request(server, path.replace("%2B", "+"))[2] == blob.read_bytes() + assert request(server, path.replace("%2B", "%2b"))[2] == blob.read_bytes() + uncached = path.replace("3.13.9", "3.12.9") + status, headers, _ = request(server, uncached) + assert status == 302 and headers["Location"] == "https://github.com" + uncached + assert request(server, path.replace("20251014/", "%2e%2e/"))[0] == 403 + + +def test_prepare_exact_python_release_without_uv_versions(tmp_path, monkeypatch): + calls = [] + raw = python_metadata() + monkeypatch.setattr(uv.ranges, "get", lambda url, *_args: (calls.append(url) or 200, {}, raw)) + monkeypatch.setattr(uv.ranges, "fetch_blob", lambda _cache, entry, url, *_args: calls.append((entry["filename"], url))) + manifest = tmp_path / "manifest.json" + uv.prepare(tmp_path / "cache", manifest, [], python_build="20251014", python_filename=PYTHON_NAME) + assert calls[0] == "https://api.github.com/repos/astral-sh/python-build-standalone/releases/tags/20251014" + assert calls[1][0] == PYTHON_NAME + assert json.loads(manifest.read_text())["assets"][0]["repository"] == "astral-sh/python-build-standalone" From 9cecf72e0a4b1d18892e351d030f481058052f21 Mon Sep 17 00:00:00 2001 From: minixalpha Date: Mon, 12 Oct 2026 10:28:49 -0700 Subject: [PATCH 11/15] test(benchmarks): freeze verified setup evidence for all 34 tasks --- .../dev34-main-baseline-20261012/README.md | 38 +- .../dependency-release-manifest-v5.json | 48 + .../experiment-v5.json | 1675 +++++++++++++++++ .../python-main-release-manifest.json | 16 + .../adaptive-rejection-sampler.json | 28 + .../transport-v5/build-cython-ext.json | 28 + .../transport-v5/build-pov-ray.json | 28 + .../transport-v5/caffe-cifar-10.json | 28 + .../transport-v5/cancel-async-tasks.json | 28 + .../transport-v5/circuit-fibsqrt.json | 28 + .../transport-v5/cobol-modernization.json | 28 + .../transport-v5/constraints-scheduling.json | 28 + .../custom-memory-heap-crash.json | 28 + .../transport-v5/db-wal-recovery.json | 28 + .../dependency-cache.compose.json | 100 + .../transport-v5/dna-assembly.json | 28 + .../transport-v5/fix-code-vulnerability.json | 28 + .../transport-v5/git-multibranch.json | 28 + .../transport-v5/kv-store-grpc.json | 28 + .../llm-inference-batching-scheduler.json | 28 + .../transport-v5/log-summary-date-ranges.json | 28 + .../transport-v5/merge-diff-arc-agi-task.json | 28 + .../model-extraction-relu-logits.json | 28 + .../transport-v5/mteb-leaderboard.json | 28 + .../multi-source-data-merger.json | 28 + .../transport-v5/nginx-request-logging.json | 28 + .../transport-v5/openssl-selfsigned-cert.json | 28 + .../transport-v5/path-tracing.json | 28 + .../transport-v5/pypi-server.json | 28 + .../transport-v5/pytorch-model-cli.json | 28 + .../transport-v5/pytorch-model-recovery.json | 28 + .../transport-v5/qemu-alpine-ssh.json | 28 + .../transport-v5/query-optimize.json | 28 + .../transport-v5/regex-chess.json | 28 + .../transport-v5/regex-log.json | 28 + .../schemelike-metacircular-eval.json | 28 + .../torch-pipeline-parallelism.json | 28 + .../torch-tensor-parallelism.json | 28 + .../transport-v5/write-compressor.json | 28 + 39 files changed, 2816 insertions(+), 13 deletions(-) create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest-v5.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v5.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/python-main-release-manifest.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/adaptive-rejection-sampler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-cython-ext.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-pov-ray.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/caffe-cifar-10.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cancel-async-tasks.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/circuit-fibsqrt.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cobol-modernization.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/constraints-scheduling.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/custom-memory-heap-crash.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/db-wal-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dna-assembly.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/fix-code-vulnerability.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/git-multibranch.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/kv-store-grpc.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/llm-inference-batching-scheduler.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/log-summary-date-ranges.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/merge-diff-arc-agi-task.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/model-extraction-relu-logits.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/mteb-leaderboard.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/multi-source-data-merger.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/nginx-request-logging.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/openssl-selfsigned-cert.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/path-tracing.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pypi-server.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-cli.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-recovery.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/qemu-alpine-ssh.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/query-optimize.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-chess.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-log.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/schemelike-metacircular-eval.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-pipeline-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-tensor-parallelism.json create mode 100644 benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/write-compressor.json diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md index ca5a7fe..a3b5b8d 100644 --- a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md @@ -39,10 +39,12 @@ total attempts, with the same agent output and original per-attempt limits. ## Run -The active installation plan is [experiment-v4.json](experiment-v4.json). -It checks only the five unresolved installations; do not use this partial -preflight plan for model work. A separate accepted-evidence plan must first -bind one successful native installation for every task. The original +The active model plan is [experiment-v5.json](experiment-v5.json). Before any +model call, the strict orchestrator validated exactly one successful native +installation per task: 29 from v3 and all five from v4. Task revisions, source +snapshots, wheel digests, bootstrap/profile states, and native artifact hashes +match the frozen main runtime. Earlier partial preflight plans are historical +evidence and must not be used to start model work. The original install-only job ended with three setup timeouts, two uv-download TLS errors, and one cancellation; 28 tasks never started. No models or official tests ran. [Preserved evidence](preflight-v1-evidence.json) records the native artifacts @@ -59,12 +61,13 @@ downloads and one QEMU Python-download TLS error. Its two remaining installation were interrupted before model work after the new archive-cache integration probe passed. [Preserved cumulative evidence](preflight-before-v4-evidence.json) includes native results and the separately identified administrative interruptions. -V4 checks `log-summary-date-ranges`, `qemu-alpine-ssh`, `regex-log`, -`circuit-fibsqrt`, and `build-pov-ray` using the final transport configuration. +V4 successfully checked `log-summary-date-ranges`, `qemu-alpine-ssh`, `regex-log`, +`circuit-fibsqrt`, and `build-pov-ray`. QEMU's registered verifier dependency +profile passed; the other 33 tasks remain explicitly `not_configured`. The [APT configuration](transport-v3/apt-cache.conf) routes only the named public APT HTTP origins through a loopback proxy. Its -[Compose override](transport-v4/dependency-cache.compose.json) starts cache +[Compose override](transport-v5/dependency-cache.compose.json) starts cache sidecars in each task's own network namespace and waits for their health checks. Each uses an already cached image pinned by digest, has no task filesystem or agent credentials mounted, publishes no ports, and has a separate 0.5 CPU / @@ -79,8 +82,12 @@ The recorded real-network probe fetched the original 8,790,396-byte Debian index in 138.418 seconds cold and 0.008 seconds warm, with identical SHA256. The archive sidecar serves only original files bound to the -[release manifest](dependency-release-manifest.json): uv 0.9.11, uv 0.9.5, and -CPython 3.13.9 from build 20251014. Filename, size, and full SHA256 must match the +[release manifest](dependency-release-manifest-v5.json): uv 0.9.11, uv 0.9.5, +CPython 3.13.9 from build 20251014, and CPython 3.14.0 from build 20251120. +The latter is listed by the original uv 0.9.11 download index and was warmed +before model work, after a long managed-Python installation was observed. +It adds a verified cache hit to the original-source fallback without changing +Python version selection. Filename, size, and full SHA256 must match the original public release metadata. Its cache and manifest mounts are read-only. The unchanged installers use their supported `UV_INSTALLER_GITHUB_BASE_URL` and `UV_PYTHON_INSTALL_MIRROR` transport settings. Python version selection stays @@ -88,7 +95,7 @@ unchanged; uncached versions redirect to their original GitHub URLs. The [uv](uv-cache-integration-probe.json) and [Python/pytest](python-cache-integration-probe.json) probes executed no models or task assertions. All transport files, utilities, and the release manifest -are frozen by hashes in v4. +are frozen by hashes in v5. Every model task uses this same final transport. From the repository root, with credentials in the process environment, set `ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic` and leave @@ -98,14 +105,19 @@ before sharing. ```bash python benchmarks/harbor/scripts/run_comparison.py \ - --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v4.json \ + --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v5.json \ --phase preflight + +python benchmarks/harbor/scripts/run_comparison.py \ + --plan benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v5.json \ + --phase model ``` The orchestrator calls only the standard `uv run --project benchmarks/harbor python -m harbor_adapter run --config ...` -entry point from the frozen repository root. Fresh install-only preflight runs -for all 34 tasks before model work. Only QEMU has a reviewed verifier dependency +entry point from the frozen repository root. The v5 preflight command validates +the accepted native records instead of replaying installation trials. Fresh +successful installations exist for all 34 tasks before model work. Only QEMU has a reviewed verifier dependency profile, `qemu-alpine-ssh-v1`; the other 33 remain explicitly `not_configured`. Agent installation success does not certify unsupported verifier dependencies. diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest-v5.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest-v5.json new file mode 100644 index 0000000..979c091 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest-v5.json @@ -0,0 +1,48 @@ +{ + "registered_at": "2026-10-12T17:27:47.967711+00:00", + "metadata_source": "Original release metadata preserved by the source manifests; no versions or payloads substituted.", + "source_manifests": { + "dependency-release-manifest.json": "f4da90888edbec5f7e90ba9f4a0622229c654042a54275f3b50d700c1f7a0e35", + "python-main-release-manifest.json": "f49570d0f1ef453261f37ac84e7ea72d46b48eb3a00839b44921e84902a6131c" + }, + "assets": [ + { + "version": "0.9.11", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "url": "https://github.com/astral-sh/uv/releases/download/0.9.11/uv-x86_64-unknown-linux-gnu.tar.gz", + "digest": "sha256:817c0722b437b4b45b9a7e0231616a09db76bab1b8d178ba7a9680c690db19f0", + "size": 21699161, + "asset_id": 318993196, + "release_metadata_sha256": "c1a1378ece65ce1aa8343e57bdbd305289a8fc6d3d9180e8bb6b0f69acd91413" + }, + { + "version": "0.9.5", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "url": "https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu.tar.gz", + "digest": "sha256:2cf10babba653310606f8b49876cfb679928669e7ddaa1fb41fb00ce73e64f66", + "size": 21370871, + "asset_id": 306970260, + "release_metadata_sha256": "1921b17d3370aafb690b29d1d121ee8719ef3d3dbc0c7cf3b948c3059a927a5d" + }, + { + "version": "20251014", + "filename": "cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "url": "https://github.com/astral-sh/python-build-standalone/releases/download/20251014/cpython-3.13.9%2B20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "digest": "sha256:c0ccda275948c79996e46993c2c5476ff5cca606dee530f1dea712179131b348", + "size": 33517986, + "asset_id": 304305340, + "release_metadata_sha256": "8e5db21cfcc2bcafcb2127f269f00cbfdd1f74842a685acd2a14e3c1ad6f8394", + "repository": "astral-sh/python-build-standalone" + }, + { + "version": "20251120", + "filename": "cpython-3.14.0+20251120-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "url": "https://github.com/astral-sh/python-build-standalone/releases/download/20251120/cpython-3.14.0%2B20251120-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "digest": "sha256:66f385305ae0eefd6b65e2cf942bc91d943a61e26fb7581e029f760a2b04f393", + "size": 35376131, + "asset_id": 318881850, + "release_metadata_sha256": "e3a737862d2dddf11c73bfef6c2f2579e9d4ce5bd271cca53790c998590dee58", + "repository": "astral-sh/python-build-standalone" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v5.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v5.json new file mode 100644 index 0000000..c95b5d1 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v5.json @@ -0,0 +1,1675 @@ +{ + "schema_version": 1, + "registered_at": "2026-10-12T17:27:48.114976+00:00", + "experiment": "dev34-main-baseline-20261012", + "selection_file": "selection.json", + "selection_sha256": "4572b1cc432a16b9c5d1dd554c8d2189ae80414f386a90bf03829bc4d0c3da85", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "arms": { + "main": { + "source_root": "jobs/dev34-baseline-main-source", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "prompt_file": "main-prompt.txt", + "prompt_sha256": "37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d", + "prompt_words": 385 + } + }, + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/adaptive-rejection-sampler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "95408217b9c484ffc18a02a91b7706323b6902919c72b24866a32e27a23543e2", + "environment/Dockerfile": "e644a3343e339303230f1441b7383f076d21c69a5a7aa6430b4f8a9d27f15c8b", + "environment/protected.tar.gz.enc": "7fc9bf1347bb97da8dede04b2c58638f225070913e222d76627068aebd1b1df2", + "instruction.md": "e6d28c28acd76bef61817427c74c2a29e3e8d75f535cbd708afcbfd02c54013f", + "solution/solve.sh": "fe443a8082b5183ba1623311ba0ee958888aefa633eff93766b9ae14e9bdd84d", + "task.toml": "6ed7b37f74beb46e9d47b0f5fad53025667c9548baff4e96d467b56d429fab1a", + "tests/test.sh": "38b43560d173cc2b952c3a3e17b8a480216d84e33450515e047bcb0d806b1e0a", + "tests/test_outputs.py": "547dc6e107f034f41703722aeceb6d0236e3fb69116fc3f2fbaba11884de352f" + } + }, + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/build-cython-ext:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d5f5113941c3963292bae396baa9369571026f46e59437f98ddcc93475dd7729", + "environment/Dockerfile": "cb6cf421cb9b69d43bd25cd491e49bda982fbb865d47515178d017eda727194e", + "instruction.md": "7e52874e45f88505bd21540229719de6639a401477557d814489981d472f0e34", + "solution/solve.sh": "41bbdde88617070366a70e518eedca466a4ea1acd7e96899b2fb7da988e40abc", + "task.toml": "c472f0e7522c128fb7f03e00b50c8cdf2e5e6474daa1e094e7441f4b54e770b2", + "tests/test.sh": "0156c9c4085258b696ce424cf1ce96cba44d1263c5d73bdf54083fd80237f915", + "tests/test_outputs.py": "6549f1a8e04b6163ec08877e6f6b9868bf94563f3a3c72ac1fba9c512c31689e" + } + }, + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cancel-async-tasks:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9adf86b1279cc44f83fe16e6d340595aa992240f1d87243d204cae32e453693b", + "environment/Dockerfile": "1ee7bdb4a7536f5f114c11a3ac6ff42790abfea901fcba6b2a8ea8685f19140e", + "instruction.md": "23d22f7039b50682941652c716f96dac314b8585c32c133614cac77dc067f47e", + "solution/solve.sh": "4938ca4ad5b7dcd36b414b42a1abb10c058ecefce7462b678f9bd05c9fae1844", + "task.toml": "14db0675e04408f2133b0067e8dbddf66a1d6e7baf6b8d38bef8ecccfc4e58a8", + "tests/test.py": "f533c0a64db068f582b4733c5b81f2e66b8fa52fcd3efa65752e6bd6d24780aa", + "tests/test.sh": "79600a6b863def31b4daa008297423010cd2d6243e689d67503e118e4dc53397", + "tests/test_outputs.py": "d6f510b51fafc81b4e9b5cc95535aa13491c8753e0e4171db42c45f4de460ee7" + } + }, + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/cobol-modernization:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "c89318cf59efcc7040e3d946022e2616161b6733931a080cdeea7b1918e7ecc4", + "environment/Dockerfile": "8b175c83fc03ec7d16d6101516bc5100230b592f480e435279f4fe051c9280b2", + "environment/data/ACCOUNTS.DAT": "46f63eb29442ebc6716bcd8a0c36f4367910f2783f6b84a307ed9a61ccd1046e", + "environment/data/BOOKS.DAT": "db04da646207cd87f8e9c5b6f0b57d95953899f3eef22de1a81423f06ef4a9fd", + "environment/data/TRANSACTIONS.DAT": "1fa4cc5420ded94d585c563e8196b85166ebbae2bb8924b20014717fb2fb9ee3", + "environment/src/INPUT.DAT": "52fc984f0cf14af1495285d7efe02b9bae13159eea59255560ee938c4f78afee", + "environment/src/program.cbl": "27cb0e56f65f52fa75f346720b37a180b8e034ebe7d7622b979f195087bd6ebb", + "instruction.md": "a87074f1141b7345394cd88053d0debb0dc8f7de95308882c6538ee9a3b359da", + "solution/solve.sh": "7d4a0504e1686c1790bfbc074d4e598f4c631f54c2bc5f3d5fd786574b4810b9", + "task.toml": "4b620211dc3d6b1b989f0210c5ecd2f606974581c8bce27bd78659d48cf0cfbd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "009180a7092583fc8143bb3ffc3957923aa0d61d0aac3a471f76eeb4aae54a88" + } + }, + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "file-operations", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + } + }, + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/fix-code-vulnerability:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "87f28dbad183c3a634887696d328ca68a8a0858f132bc48cb7341c22267d38f7", + "environment/Dockerfile": "0917470986224df4daf4854a71302ae81c73ba5dd6e25be1232d29b758a5b3ab", + "instruction.md": "89b1abdf0af19399f720233b5cdf6b2e8e0ff7b6e3fe3e2a3af0d4d298a8641a", + "solution/solve.sh": "7a0131b369d83ae8546ecc3f9b4e23d38240f1bd1f0941e4430fa85b5182ee03", + "task.toml": "9aa2bd14ce9408289d199d0608bc3fa65b8b5e23374c9641020de62197dbf1e2", + "tests/test.sh": "f1611da693cb4cfc5e8482698c6a1858a5c67220ae9db89b50aed66584e9b17f", + "tests/test_outputs.py": "c006a5de95b69bd78055aa26cc055734dbd50718ede25d331f767bfee4ef7f26" + } + }, + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/git-multibranch:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3b9473429381d9eacabab498cb4a8915aa2bca24fa857ef862e5991d3f81f113", + "environment/Dockerfile": "88d9a9f4fef1ae7fddf632799dedf5a2fb6489460a93b610c33dd8b122982821", + "environment/default.conf": "a3a4c5d0ac9cfe79b0510e4225c80fa3988c27fbac13622505886bf66449e04a", + "instruction.md": "54014a38d7526d65faeb6c11d9b7ed3d4a5cb5d973d68d3da3b68b8c0289e125", + "solution/solve.sh": "300ca96b79db8d77f14742109aaf76048aafc4644ced750b1bc5c2761e8d0175", + "task.toml": "4b9981604578a31205257cca36a28d0c89c9d8d1058541330acd11d3f8d65f82", + "tests/test.sh": "ddbde3eea6ffa8fda105cbd86597a53bd409f41e13223900aa27e77a84b6b155", + "tests/test_outputs.py": "f03c6867663b8e8fb7899df51ee4df26411fab62ed491753964b801e591d16c5" + } + }, + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/kv-store-grpc:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "51fdc2ad06f0adb1270164f247ea41799b2ee2ad15490e48ddbf5e79d3767aaa", + "environment/Dockerfile": "47c9982789092437f950736110b2bd18899c39b8fd20f3af3c95b848e564a197", + "instruction.md": "f4b3facc199d5b1c78db6e603ffa29ec1b93a77ff7d291e1a92b42e6bf582bed", + "solution/solve.sh": "6bee2938ac9731d1156b0ca02b28c3d5793bf2cbe5f995c863d397093af01cb7", + "task.toml": "6ff0f30a526bfff86a67b0b53439b9965229c0f368167dc1a5d05f34d82299a3", + "tests/test.sh": "c348467a076bf614d0a850e25b2263f85bbf7b4c7a8d344fbd5454a0611a0df8", + "tests/test_outputs.py": "115de2e964bbe689d07930527113422ef8a9e4be4eb958ad84594e3fcd2b1b0e" + } + }, + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/log-summary-date-ranges:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "026000415cb577f3d3d5132902670262f7d5d8f69841903ec06a882292f756b4", + "environment/Dockerfile": "b479f6e987b63a6ec118fa1691b8bb85c437b2a6a0fe63feab5ed7154957fdfc", + "environment/log_generator_deterministic.py": "beb6eff59979e9aa0be63ce8a0f1b0df1178f92871f04ba9f76fa70cf8beb81c", + "instruction.md": "c1d4516ad4ec2238209740eaf01291df9ab48b0028a3fc5df383c6f831453484", + "solution/solve.sh": "9f6a8ee8947bdf1c0f31b530b6194ae998d66841a3b785fa2e246ee840350ab6", + "task.toml": "9a9593673d6805ef3a301ea734e9c4dbaf80da1b37c0fe9d1f2ceea2a8c57e18", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2fc06a6ce1944ad82f84fb437395fa75ed8c0eadd385b92e375c27e5bc9cb2b4" + } + }, + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/merge-diff-arc-agi-task:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d6b48ed0890b12eb68f72c12cd02689bd6bfae2191dd2688949cde6fe77f6a94", + "environment/Dockerfile": "4a03ae6f0c60b9c2f45c33f0413b5aa75c9811870254e3ff1968fd0119173fc6", + "environment/task-deps/bundle1.bundle": "a759f9b6ffc1a2eee50778d72a1899ad7d724becab84fa30e7aac71ee9e443a5", + "environment/task-deps/bundle2.bundle": "75e3e3916a01c980a03715de67b16c729e03661d94a694b5f4223d229bd6593a", + "environment/task-deps/examples.json": "5426569e4ea078bbb74039ec712a2d4d35f1f9fa3a5f26bf2a0de4a2c8ef5db8", + "instruction.md": "2d826479873e5ac1c94c1f802763bf85c22bb04624384685ebdf6b153fe0278e", + "solution/solve.sh": "48d4becc41fda5b6063fe96947474ffe778a29d6924d7b2bb235eb61e5ce555d", + "task.toml": "6ced40e06b43b169857509765eb7f07791db558f9437cc7367d2536771c2ca5b", + "tests/test.json": "20f116a6b718bbf4874bbe203d15a5c0f85a1d88cb622e2d7529c7e4fb2d4c28", + "tests/test.sh": "c9052954cd6ce25c4a375a2fb48e765b1efa5b39f7991b93993cc793af8ab9d2", + "tests/test_outputs.py": "2754ba6c5c5f75c9db5ed2c5729d9f5d7e21cb7fdb18aa0c65abdc3e7e85c149" + } + }, + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/model-extraction-relu-logits:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "mathematics", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9465accba6c57e5de9a196eed776a16eeb0852a8afc18fc3f8979e2de71b0c7b", + "environment/Dockerfile": "598b1f80d3b7194e97cbf02c8d245de30e303909e0cbe6465125d1d3179fddb5", + "environment/forward.py": "af1a23efd8e4390d16a29777c28116b2f8623ae3f7b628fe14db9f94dac26f5b", + "instruction.md": "c9e179dd099065a34c362a865b0eb9fab5f7d96be582cf117e7519580c036ebe", + "solution/solve.sh": "4cea5094f074ff21453d4423aa1b415a104ed90caa55a4acbb4f09941a3994ea", + "task.toml": "eb64dfaf8ae88868911ae28855cdc4f767de464001e2704919015b595be6dc29", + "tests/test.sh": "3b2d8f5e0353c2bab3515d6387468a863a4f9f241ee131c73f545f3fdf9e2d52", + "tests/test_outputs.py": "e5189c7eb9942ede55e1b48a80388590b80a486b75497613f257b3373ad9b2c9" + } + }, + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/multi-source-data-merger:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9bf37162a4c1b88ed93627830bb8512ebae154325d7cb069c883144a0e4c69d6", + "environment/Dockerfile": "c95a40d3d35b473dc15ae1e8784ba7fb1725805c0583fc574dd2764325d75e4f", + "environment/data/source_a/users.json": "75e51e2d3ab0bdba2269d7cf3757576f84f3ee2c13407d2db94289a5958a1778", + "environment/data/source_b/users.csv": "8e87d16dc47d166edd8e56165955bad265b630d53d02762367c816affb5fa760", + "environment/data/source_c/users.parquet": "23d1ea51c4170bdba53febb885a0f495e28de525a52667347b2932136b9a72fe", + "instruction.md": "d1d28bb6e37f7a255a1b7f78aae8591f35c0494f33e4913392424ad954aab944", + "solution/solve.sh": "1b3d0d50e31930d042495d48f4c8e95f1a9c1a979c803b7f7cec651322537e1e", + "task.toml": "ee3964c0ee040f0a0cb121cefefbc15443d127684acb003fa5b61dd42f31c694", + "tests/test.sh": "d5a00e64413c0e1ae452f38a1aa1a47c3ed7b4416bbfccae1a4c053025a336d1", + "tests/test_outputs.py": "a070888930cf829aeca039771260b1386bda990ea805f313468d381ba3608ca5" + } + }, + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/nginx-request-logging:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d79997586d1c14ed8dbfea050e3580b25500e04c8345acfd3f294152433c795a", + "environment/Dockerfile": "43003466d89ab83484a3fa618e35b58cdb74a002b1e8716ef30f12b3ac8302d8", + "instruction.md": "26a7ac98aced6107f147206790ada77bbf8ed3c25fb360cf323d5fc6889edf99", + "solution/solve.sh": "2454bbbf7c201a9a778165f286373ba2cb991a0eb1d026a8005d57ae71250dd9", + "task.toml": "b7a78d00be13656f75069197100b7b8c4fd268decf6f78929c853b3222a2729d", + "tests/test.sh": "27761195f53cbfe5941fe39b573631f6c926869e64b7c43220377e1e116528e7", + "tests/test_outputs.py": "045cc716c14efde3b0dcff5fc7c85ec5d18bfc6ce66f8b40a418fa2a3a4acda0" + } + }, + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/openssl-selfsigned-cert:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "security", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "5dfa88651890cda06fa2a25babd3b813818767635e88bbabecc8305d5bb3942f", + "environment/Dockerfile": "f590b523038be43afcaf5291d7b9e8ed1333c48d9994e77b8dee2f2c352f0f40", + "instruction.md": "38054a3371f7c2c853478a2f9af0bff858437d5b28a9a07483238f06e566921f", + "solution/solve.sh": "58cf3c844e907745cd81fb4bd75efb3d080e2f0e82b589c56c954ae6e1756510", + "task.toml": "75c8913f0fa15890786c11999cde920b8bfc10d4ce8a14edf1e76985a2ac5800", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "90970bb3c372e8e7fb6bb9ac5f649ba73eb38e1a5ea2e922a9057a72b6b4192d" + } + }, + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pypi-server:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "906a096a2cc7b9afc738ff5ad5fbd605be65c103f5069c6673915223b9b187e3", + "environment/Dockerfile": "17a4e67580e4f2c6a151d04b9d74eb020666e7eec8e5259c4a5411f9e93858d0", + "instruction.md": "1c537e4a311a8e71155e121f57bfc03f742fbfb99d05314cd03e433996a28a31", + "solution/solve.sh": "1143c0eade4932ec3aeaa5013fb5e23bbcccca15dead294aa03b6af88750775a", + "task.toml": "07c39d7a733c89adadf74bb63a3e906c3694feb6c490d5fc85ac187c003fd447", + "tests/test.sh": "d2b1220e42bcfdcce87907ac5bbd69ac210a3c94926545ca2439636786e7784e", + "tests/test_outputs.py": "82c6351722e9aa984ff1d701bf57df3150ce51466df2b70afa1aa62d2650be4c" + } + }, + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-cli:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "35a093fa5f1592737784ba718c2e3e9ae17ff8b5e53e003fae560ddce091039c", + "environment/Dockerfile": "8fd5c0f21a6914cd440d10c84c9db06fec606f20dc3e0363b17a3b8c220dc4d9", + "environment/task-deps/cJSON.c": "0e7995261640c96db16ed2c91a5d981efa917cf5e72d0f97ec45a1179ae3ad73", + "environment/task-deps/cJSON.h": "0909160541bcc510e0b2d8c7a9cc051fa1459a0f13be4d8e56346a15491f2f2f", + "environment/task-deps/image.png": "f4d5dfefe463939996541404340e7099d5957b56e1df3760d13b5d2de754db53", + "environment/task-deps/lodepng.cpp": "67873c5f6272f779a143a9c81070e8783327c5552d0b9de2aaee2cb779308917", + "environment/task-deps/lodepng.h": "b735b7e0357319973e9488d520d7f270d734a6e9e1bf924fb1d5b8d26d7e97ce", + "environment/task-deps/model.pth": "a32103eef09936e586b1a7dc7e64d279500a38ff30fdd6df2167e28eb954eee6", + "environment/task-deps/model.py": "e4d3a5c728449656d0efde7871ee227619348352769795ed1be1a8b7f835fddc", + "instruction.md": "ce36ee1f6c589f81ee6a6be4967e25c76a2ed697006928023f42dfb9f28a6750", + "solution/solve.sh": "3b34e561c9ef0a53c9948655847846eb4811aaac67b990480aa6aac1b04bfe66", + "task.toml": "fc1c018efbd5d0537ac0741d53d1ebad3886faa51d14cbde8ec953856411bc0d", + "tests/test.sh": "43fb67e537c7adc2df11c2367f47c876cb5d16a7c1ae16d0339c3a3364bb2b06", + "tests/test_outputs.py": "370702c57a4e0c85f70ee37a5c357ae3a5df8480a275bb6263b0376f2f6c093b" + } + }, + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/pytorch-model-recovery:20260430", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "model-training", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e080308a05703b40edd16f53aea41a0e6b9874750ecee35bb432029d0d44bf53", + "environment/Dockerfile": "8a6b7edbb9e7c241e447f15a71e13e5d043b25169be2fd9802b84f5be3d56eb6", + "environment/dataset.pt": "6db65e4ec1c0760209c2dd997ef582e4a1519ab38e0312339999c460a696960d", + "environment/weights.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080", + "instruction.md": "2bef07dc28be3ab2c35ae389584034c2e4dde92c2c31b3eb6bd829628c8c3f4a", + "solution/solve.sh": "175a7802777b43e0fa4f629d71f7c68fb40706813c342aaddc72108d89bd5b2c", + "task.toml": "851a51204f53482a31f47cdf8833128cf421113b0331011286940fc857c37dfd", + "tests/test.sh": "6005f3a75ccfe00d7bed1005264b4a84a1f8e30cd780f8a97b3a804be7cc7a8e", + "tests/test_outputs.py": "5d4d57c43e6913bf3d0081da764ad36d1b7275e3bb114a05abda529afe563905", + "tests/weights_gtruth.pt": "71471e71c5a311a897362dd5b77413e7ca2ca59abd00a110bdc2cd3f8584d080" + } + }, + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/qemu-alpine-ssh:20251031", + "resources": { + "cpus": 1, + "memory_mb": 4096, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "system-administration", + "verifier_dependency_profile": "qemu-alpine-ssh-v1", + "task_file_sha256": { + "README.md": "72f1c83fa1e3e6381ef85dd82fd109ee18cdeac090311cf3212a0a8cff0a32a5", + "environment/Dockerfile": "015ce0601ff54758f66b33f4952e0d3a577d84c09aed8f3ec5945a570e8d168a", + "instruction.md": "8348c60490cb90bc690cba259dfe98996da4c3294c6d42bba1119b00c64de4de", + "solution/solve.sh": "1cd31186c427b91bb00499d22aa4f0214fc1ca93b40b6e4cfa969611b848db2a", + "task.toml": "10a43ec6ee732eeba852c377f12e23178f7fa5a4868143f3699a15565ffb64c7", + "tests/test.sh": "3b244479776c4ee3639631112799f2cde22cbb80ac10511715e3a8f32909c289", + "tests/test_outputs.py": "83ccb4bdcd259b342212bebefe395f289858b4dc55b2ee7ae0dab238ad2e9718" + } + }, + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/query-optimize:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a82c53e333a0ac0b0b410b5e26cb8e7305dece31db84d340248ffb13d5a45160", + "environment/Dockerfile": "064818771f200e0517d22068f87229a7a95d0a82b0778c84ca7203bf1926a0b8", + "environment/task-deps/my-sql-query.sql": "43382792f0f438625d643e34abe144febd5669518f4ab1ab550a15c6d9314a21", + "instruction.md": "473e12d320d2732f679fb631c2a4dab4a01deea6775686718ae0c417b8a39447", + "solution/solve.sh": "0d95f3cedb287ff9d47c796cbaf812c300d1222c10fa80c49e8acb0e81eb6978", + "task.toml": "6e88f2a86df699594bcb1bf81df4ed694505f79c9420f671335b02524bbe3868", + "tests/golden.sql": "4da1885d56b3ffa7b7b22b422996d018795611f6963499d551582881b74b84fd", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "2068ae80c57af8d52a7ae510f0666732c057806e556b480970f36f14bae683f6" + } + }, + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/regex-log:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-processing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "29ffd98c55dbb6f4844f1b9d8c1769b2155c117ee475e0df6bbc5ef330d239b3", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "4f7ac05e70cf9220ea0f1e5a052c5f908cd0fa884e847d80b0bd51bae2e96f9c", + "solution/solve.sh": "7e670d4f2b4bccb1e4db38f2a173e085ceda028c38167912b466b0a84fcc0999", + "task.toml": "6bc98525852755caf1fe34c174c2f10f4328adb4ce6c1968dcafe1c877e4e160", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "345c3bd09ab6f6fe8c8361a58c0a47bf0a13b3fcb38a5ac7824e44ff855e8f72" + } + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-pipeline-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "297ddea07c230dfd246a8ab07337dd7abcd8e5e2a2b2e26590cb0a8af779839c", + "environment/Dockerfile": "c357ea41e4a4013c5c5980b8fef04af6a5ae250ccc6697cf3bad3127401a931f", + "instruction.md": "989bfcf0c54baf6c8cdc391743dbb2f9e858c5779dd85112536875d4d1b2dcd2", + "solution/solve.sh": "787198e4178d88b887697aefdd29819f29435ffdfe356545d92f58a30fb38b70", + "task.toml": "8b4f9f525c08afbbc4eca9168e46238a61ccb903486773e9ca4da64bb7ffe857", + "tests/test.sh": "cc9fd28b6590993fc37e4a99da771cbf04726c8c62fe308e802612241da69dea", + "tests/test_outputs.py": "23574246a407163a8e6581c84287cc2f1eb745efbaad2a317a5a28c39958c64f" + } + }, + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-tensor-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "4c1b3c3be30548f1f94c6f5503b1bc74b006af7f192dc4cd026f411df1a54604", + "environment/Dockerfile": "5f264ff0bf62a49aee3c6398447de5108df141e7a2ad35c6de8ef39fa62049c6", + "instruction.md": "13bde2ab165d72a311e2bf066d5d6c3c8761bf9d3dd39d1b23eda23bc546b910", + "solution/solve.sh": "d083b7cbf9dbb6cc2ff38c8f2acc150f0059c04aea42d6c2183aeaecc5a19385", + "task.toml": "72fd2a5500d1fe8795dcae76cccb1a29fb25894881a7475f2fa519c87d425a8d", + "tests/test.sh": "a7e6c9408b2401e4e7448371f7ddc72f3284caccfb56277d3cd4e98051868cdf", + "tests/test_outputs.py": "4ae4285e15fba1fcf13d7b7c0638ac8c2e2b0797c937abab1fb6b136fd508290" + } + }, + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e", + "native_agent_timeout_seconds": 900, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/write-compressor:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "603015a28500a28f0838fea2c43a5168613746bfd07737920f4bae40b5a326f7", + "environment/Dockerfile": "9a583d29d46818f6ff2be7c6a0879d03343106e5376fd5a48eb5926b402bb93b", + "environment/data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "environment/decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "environment/main.rs": "fa53d6572fca95903e30d2d953f1e428bb8b63513ff218ebee9bc60a425ab389", + "instruction.md": "c4fc3e73e44deeac95f1cd4a34a2d87e853b22ceec2b2824767f5f087b470bb2", + "solution/solve.sh": "01550b7de7ab1320b1898eae055226bb7c73bee259c75102d2240c974a730409", + "task.toml": "6066a509066433c97c417fea68c4e3e73996f698dcdcbabe07718835eea355a6", + "tests/original-data.txt": "0fc483d0fb31128dfa41ed11adca56c88688888189826ef9daeafd7559b4fa4e", + "tests/original-decomp.c": "7f8d9ef66943f912a217aa48f4ef5efc967b44c2b21d0f56df029b5ab413f62a", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "809f486053e3ffbce852111dee1f63d80ab31297bd33172389e50278bed384e2" + } + }, + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "native_agent_timeout_seconds": 1200, + "work_budget_seconds": 1020, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/constraints-scheduling:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "personal-assistant", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "d54a690970149e1216bd4fc066b45069c7948c13e3b1ef2a213a74342805f209", + "environment/Dockerfile": "7d9aba072a4a907229728b9739c8cf821fc2c98450babd64452682caf3759c84", + "environment/inputs/alice_calendar.ics": "5a5d45804bffd339c2479d49e76fe817b38244d14d389afe1de583443808ecbc", + "environment/inputs/bob_calendar.ics": "c97e85dbcdd756758f8a80f4db5fd5c1dcacee79298e4d2262f46de5ff680b50", + "environment/inputs/carol_calendar.ics": "32a0f8687637df0a12b8e5c56d49c4192bc3e790856943632d70f237c7c1363b", + "instruction.md": "342def78177f9bc41bea4d11af679bc385a9dda2ac8a1e425be384a5b7f937b6", + "solution/solve.sh": "802d81ec064fad3050bbade95a2f847ac24e0b7f80fa69b0bcee0347dc92fe77", + "task.toml": "7b9842de16d74e8faf8c7c0aa03b0b81145941cb12dd322d81ae23fc48607250", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "1422b345ba3aacdfc4184e962811b6379988435896c50d8cf62903e70990c5d1" + } + }, + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/custom-memory-heap-crash:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "debugging", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "a06ed23a2c467aa36d0d4a88e5c020b1076cc25b8d483517239658e9c596dde9", + "environment/Dockerfile": "6ee81e5a4fa2540443d7f27d577ba840728fc6aab9f301ab7919103673c78080", + "environment/patches/locale_classes.tcc.patch": "16951f4a4df6bf253886ac75f5a39fdbc328e415194493f63cb7589a364aa817", + "environment/patches/locale_init.cc.patch": "e97f133ceebd86f8881f972e0e45b0ec0a13a9bfa428d8032008707683d54292", + "environment/program/main.cpp": "b94538c76887cc2b2ab739b5ac4731cddd0faa8f4a9892a6265c062bbbeede14", + "environment/program/user.cpp": "92b203786fed1bf0b3aa1675c9cd2d82b2651170675064da8ffee73b98ef14f9", + "environment/program/user.h": "f43d155995a56c22b9cd1417daef7c3b5a976cdc662a98eef0feb438ae1eee77", + "instruction.md": "53c68ae14a3c3af9a97c40bfd9bdc30a5d98b37eaa2f69f763475745f70f108d", + "solution/solve.sh": "b0b1d10bea1cd5be14a82b35d522df616df1e67028f90d2a6bef90aaa98511a1", + "task.toml": "ed474be1983e212913aa8866ebb66216acbe36aef2339ac375d232aa46617481", + "tests/test.sh": "70a2b55f0c5757f6b123a216dfd202be4a646680bced28dddcd31fd8959a058f", + "tests/test_outputs.py": "4e1cb33f777fa7fb5e1536f9bea9eace6deccd515c67d4f87570adbe692914ba" + } + }, + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/dna-assembly:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "scientific-computing", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e7855c23d9822f688eec4dd89ca0f1caf91c6a69e1c8f3c9eb544b4ab2a94821", + "environment/Dockerfile": "26e9e43a98f0b1d132bb638f44b7dc882b0d0ca32d56fa33048adeee00ab600d", + "environment/sequences.fasta": "ebdc361b5fba28eb0a2c7a8a8a444ccb12cc842c8cbc6df5a2c24254f130736f", + "instruction.md": "88998c98beabdd966ab46c04ede822bbc74796696aa1bf87c567a9131210533c", + "solution/solve.sh": "c976cb118c8ef1befd846ba05fd5cadb1b1f529bc76b9f2ea9cb3049e6e11a47", + "task.toml": "7b571544b64d9be1c91526837e5d805c2181afe0e6fd57a7f0676bba36c868d7", + "tests/test.sh": "271c8317c98539e3b49233d017fe51fb0e5fba8114e76830fe698565499b2b1d", + "tests/test_outputs.py": "5c2d0fd2e5f6303263e9750f67acda7db491ef04841832b4fc62bcfb61b08924" + } + }, + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/llm-inference-batching-scheduler:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ecc9dd635357c184742cf72fc1e452a8946e207b183fc9a7eebbcf07133b911f", + "environment/Dockerfile": "10a6193904ee0022ac098504182989893b196dfc74e626a3ca48534a8eacde48", + "environment/task_file/input_data/requests_bucket_1.jsonl": "8b3cf1e2b06074162038064dd2c114abe308acd32cec7d365ee63ee81978f7de", + "environment/task_file/input_data/requests_bucket_2.jsonl": "e15e2d4c4d55f875c42a6042610d6282f6c0657c39d3e0d0f087268252525cb5", + "environment/task_file/scripts/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "environment/task_file/scripts/baseline_packer.py": "547d230d5e6d93197803480cb81cab0ec6ac63fcfa0a24b246f549523f216c4e", + "environment/task_file/scripts/cost_model.py": "0cf417fd5ff0112defcd46970d37a290c0763c5a9092e91cb3ebb86c3f20e6ce", + "instruction.md": "0817374767533fbf51bb93345ee1bba266263c216c6cc55c096e71b4a1d37dfc", + "solution/solve.sh": "cb9efb586d76a893c3027e62029e286eb6c7583dc92dbf1840a3adc236a926d7", + "task.toml": "52aaac942f917e6a91ff658e0272195bd1cd842d41d71e302414c52603cc588b", + "tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "tests/cost_model_for_tests.py": "d3e810a99d40327fb8317f6512d879c52c6aeebdf5dcdc0c5e117c7df61c9ab9", + "tests/test.sh": "005f980332355e2fe143d4ff4aac11828adf070594bbe7055399c238f06953ca", + "tests/test_outputs.py": "80b06b7c4ba0db28832b767d48be74bb462d7f5df5798d9cb58c4f1160d96ffa" + } + }, + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c", + "native_agent_timeout_seconds": 1800, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "docker_image": "alexgshaw/path-tracing:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "3e4fc952b3f9a860c7ee7837b3f3c686f1283ee7821b014265dba7c94d7fc849", + "environment/Dockerfile": "7aca02e4e20b04ed2269d627a29dd4e7337a02764a9652b641452bf09836a3d5", + "environment/orig.c": "665c1bde0907740890ac114d6b36af1f3fd00449fa287583817f778bb5da200c", + "instruction.md": "9f36e02d636f5dc0946619bc518f2e51c24fa75251da9d1f0ca9eca612cb9572", + "solution/solve.sh": "d107abde101eecee6838da80cc52a3d480c93956fff3a9343a299ea01b3fe9e6", + "task.toml": "cf47013f69bf1b43242da984f231d4daf364fdf44d3845f2eea2ca2cb9308c46", + "tests/test.sh": "a719dbb1ba7ae76df095e9d815941dfa4c55edaf11f173b9c3c3097afa838548", + "tests/test_outputs.py": "572e68b2843191d29592faf42a405861b7958d8e894868b74c5a8886b53eb879" + } + }, + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6", + "native_agent_timeout_seconds": 2400, + "work_budget_seconds": 2220, + "verifier_timeout_seconds": 2400.0, + "docker_image": "alexgshaw/schemelike-metacircular-eval:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "095d3ec6dcc0fe0c7a7e92355456dcea6992f91d1f8d397669b8cfe18475f506", + "environment/Dockerfile": "9b3df76222e16aad360fbd34cb0f71a0be8986052c065b10b92c0cdf417b25c7", + "environment/tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "environment/tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "environment/tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "environment/tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "environment/tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "environment/tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "environment/tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "environment/tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "environment/tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "environment/tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "environment/tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "environment/tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "environment/tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "environment/tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "environment/tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "environment/tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "environment/tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "environment/tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "environment/tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "environment/tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "environment/tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "environment/tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "environment/tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "environment/tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "environment/tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "environment/tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "environment/tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "environment/tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "environment/tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "environment/tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "environment/tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "environment/tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "environment/tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "environment/tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "environment/tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "environment/tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "environment/tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "environment/tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "environment/tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "environment/tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "environment/tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "environment/tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "environment/tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "environment/tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "environment/tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "environment/tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "environment/tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "environment/tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "environment/tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "environment/tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "environment/tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "environment/tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "environment/tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "environment/tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "environment/tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "environment/tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "environment/tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "environment/tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "environment/tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "environment/tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "environment/tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "environment/tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "environment/tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "environment/tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "environment/tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "environment/tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed", + "instruction.md": "3af5f9567b62f93683db7518ec6fcb609f83796258aa02166b5a7091c5b69031", + "solution/solve.sh": "e2c5b0ac2bef4801a1891e13c108813381877426c1a7f01bab86b4fc1f712510", + "task.toml": "8dae8eb226183b0ca92331c1106152a7abc93b03b4e5a95291b86a4ddc9b44e0", + "tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1", + "tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5", + "tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac", + "tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235", + "tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230", + "tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c", + "tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a", + "tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3", + "tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c", + "tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff", + "tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07", + "tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47", + "tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c", + "tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612", + "tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080", + "tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e", + "tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238", + "tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24", + "tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f", + "tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec", + "tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103", + "tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856", + "tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05", + "tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712", + "tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923", + "tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62", + "tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e", + "tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75", + "tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd", + "tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f", + "tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f", + "tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860", + "tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad", + "tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4", + "tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b", + "tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6", + "tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0", + "tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216", + "tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b", + "tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8", + "tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7", + "tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae", + "tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f", + "tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae", + "tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61", + "tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6", + "tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff", + "tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7", + "tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d", + "tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8", + "tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b", + "tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06", + "tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d", + "tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d", + "tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9", + "tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b", + "tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2", + "tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216", + "tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a", + "tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce", + "tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186", + "tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb", + "tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809", + "tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e", + "tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed" + } + }, + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 1200.0, + "docker_image": "alexgshaw/caffe-cifar-10:20260403", + "resources": { + "cpus": 4, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "machine-learning", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "e918e6852c5c034422b81a6be22d7596fddcee53abe088f56727b2482da32728", + "environment/Dockerfile": "b3b8adc915a1afa035b604273af3bc1aa8b90999d2dfd72763218b34cb87abcd", + "instruction.md": "46907b0ee048f74269c29a15e0e086546b2b1220149597017da8b5ccc05a0cea", + "solution/solve.sh": "771887de1ed557269c7f3160c777519aa9538e1e9b59771b2f6ef38c8952490c", + "task.toml": "2831165ad9d9e8bbc646e6d85c0a2290640dcc4f13749547a2937c7675012a8b", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "e99b9f6284e923388fbc21488b1af99fdde46d4a809d3557b0788dbf1cb6b281" + } + }, + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/circuit-fibsqrt:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "201f7c889dd6f73dd83ca078e347272b33720fa622f8c757ea6d5770453f4cbd", + "environment/Dockerfile": "780c23986778f3e5f842567868af30b1135ed095ca9449e1c90b7f0e0483d3d4", + "environment/gates.txt": "032786f94f16605767c62bb61ce4dbba7ab7d3f1046712c975f01c9a04b6ad29", + "environment/tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "environment/tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438", + "instruction.md": "a7f500266a878d04ac9de39fba76220cfad216d8f6b8063bba93bebb3d4ab9ea", + "solution/solve.sh": "a4ffab4183815a0eb7646abc1de1fc67695014c060d7ff16e8bca9ec94385d0b", + "task.toml": "9f5aa1f7ebbbcec90fd109121e9bd509af3cfff6b4cec701bd91947988f40972", + "tests/sim.c": "700645ac6156cdc88ec1e210f7c7d8d9969bb42aec969c410597e8efca796173", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "6117b8edb7d567b30e32b16a9244bec4c131ec01e9065d52f6f53f87e36f0438" + } + }, + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/mteb-leaderboard:20260430", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "data-science", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "ed147a5bc31e9865b16848b2f01d2f9b1e677af57a7c632d7ea2311c0553ca76", + "environment/Dockerfile": "271fdb5b34520c0bf2cb6462365363ec52058f127cd9eea3f5da2589df5c1044", + "instruction.md": "06ea44106eca0405c25bab00f5b3cfeb611c872093f6c0f82ed057839d1d13de", + "solution/solve.sh": "4ed68831d4acf5d80d2d73eec1f21efd0896a6f51f23020f572cfc0af6da0410", + "task.toml": "86574bd1c6c4fefd2e64eee9c2caa53d87c4bcf2e51547912440a0ddbd0badb8", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "b28f8fc8b7614801305813c402be52f4be1e75de6658ecdfdf9383aa793892cf" + } + }, + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78", + "native_agent_timeout_seconds": 3600, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "docker_image": "alexgshaw/regex-chess:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "9296cbee592fc996c58d228b51fee303647e5ecfae82d5666d42a40cb30ec355", + "environment/Dockerfile": "2616cd775252303839dfda0ee2f047eb862952b053b0103bd6b2460ef90c11b7", + "environment/check.py": "f9c416ae9ea01dcdd5f6a43170c8434b8c0a0b389a4a30876a2048e277d191c7", + "instruction.md": "a6ed3fcc54fc92d71a84af921942422d299ca89b410dcd92cd05a53d0d06436c", + "solution/solve.sh": "b728bf0ea392a3d244afbd0818d36933c8d3e190e47743c1382cd518d518a2e9", + "task.toml": "07fe52636fcc4bc9fff25d5630af145b97a5331219d719cc8decb4ca67ea21d4", + "tests/test.sh": "8f7f00a1c4dde7fcca6c918c768f2aba432b8c4007c15989716c68091950da87", + "tests/test_outputs.py": "4983c44d21c56dd8191381eff0304afc8b87bb714527ea9036b3319c015a30b4" + } + }, + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "native_agent_timeout_seconds": 12000, + "work_budget_seconds": 11820, + "verifier_timeout_seconds": 12000.0, + "docker_image": "alexgshaw/build-pov-ray:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "38380f64bc631916e264f80ce4d37feedbc783310001475b367d03989d114fdf", + "environment/Dockerfile": "44f2b5fc7311d8f336daaaca99dd4216b49355e67f50b494ebacf1b0f7c14e92", + "environment/deps/illum1.pov": "5a1f7f53ee0730c4923fcd21cd4a9c34719024d8557920441551ba7d96e75dc4", + "instruction.md": "56c8ecdd0084a07f8c97986c43af5c37ed03dd5aa994da6735cf59806162d1b2", + "solution/solve.sh": "3da9dbf589656512cd7a7004dc3752d3552a6af939cfe59eff79f91695fa42c7", + "task.toml": "b90a47ec26c1191d3c523cc094ea96b170e42fbfe176ef1f814c57d05390c097", + "tests/reference_illum1.png": "bb084f430bbc9d5b623a656920670dc06cee6d9132a9d550be4a1b99e41ab9e0", + "tests/test.sh": "928d1497ad21aef0c456294e3784f1fddf8493410f23584da89f9e02ec6bd134", + "tests/test_outputs.py": "64d6ce107503714ada346e5606832ee886be9950971cf81523e73d8ba28e8d37" + } + } + ], + "total_model_runs": 34, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "preflight": [], + "queue": [ + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/adaptive-rejection-sampler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/adaptive-rejection-sampler.json", + "job_name": "dev34-adaptive-rejection-sampler-main-20261012", + "config_sha256": "0bf207305a6399d3cb34fa2641e4241b4ada0b64903aa7805d1d488aabf43fa0" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-cython-ext", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-cython-ext.json", + "job_name": "dev34-build-cython-ext-main-20261012", + "config_sha256": "12a06e23825689780bc08590dcb4b897e4d79ebeb14c7c311a0f881eca969d97" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cancel-async-tasks", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cancel-async-tasks.json", + "job_name": "dev34-cancel-async-tasks-main-20261012", + "config_sha256": "12a150f7e796291a660af75c52e0aba5b4b583c6e9f099762ef24e0be292e6a6" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/cobol-modernization", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cobol-modernization.json", + "job_name": "dev34-cobol-modernization-main-20261012", + "config_sha256": "6f26c686c9cbb8a70d9b9e27cccf19c98e6aa885bf482704d4e809910181b2fb" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/db-wal-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/db-wal-recovery.json", + "job_name": "dev34-db-wal-recovery-main-20261012", + "config_sha256": "98d43e3daf212326518eb56b5c436a01effb82768c6ee49a581a65bebe28af8e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/fix-code-vulnerability", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/fix-code-vulnerability.json", + "job_name": "dev34-fix-code-vulnerability-main-20261012", + "config_sha256": "5d174e8120286dcba211800d492543781141475e8abbdb3a5757dd0d5fe74ddf" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/git-multibranch", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/git-multibranch.json", + "job_name": "dev34-git-multibranch-main-20261012", + "config_sha256": "7753ce6817f8f232f9d6fb7c29655a00b1071050035338ba4086e4e9301a684e" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/kv-store-grpc", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/kv-store-grpc.json", + "job_name": "dev34-kv-store-grpc-main-20261012", + "config_sha256": "ef2504381558a7df9baa22961ddf39e09e810e6c6982618cf3ae8e66c2a66691" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/log-summary-date-ranges", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/log-summary-date-ranges.json", + "job_name": "dev34-log-summary-date-ranges-main-20261012", + "config_sha256": "259dc3bac158c4552735050d0dec8a3be0e45376eff2963800664d9b80ca949a" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/merge-diff-arc-agi-task", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/merge-diff-arc-agi-task.json", + "job_name": "dev34-merge-diff-arc-agi-task-main-20261012", + "config_sha256": "f4b32e6ca1da8b721eb1bb0a0c9f761ebe21955c4f7500bceb6ca7eb1814c20f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/model-extraction-relu-logits", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/model-extraction-relu-logits.json", + "job_name": "dev34-model-extraction-relu-logits-main-20261012", + "config_sha256": "a1096fdaaed222e978418a26293eb0d4a8fa8b33b28f4714e239b39bd8e329c9" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/multi-source-data-merger", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/multi-source-data-merger.json", + "job_name": "dev34-multi-source-data-merger-main-20261012", + "config_sha256": "032159219c798a2e419590cff2e321ca495dc1191a36de45d5dce4f72029e285" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/nginx-request-logging", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/nginx-request-logging.json", + "job_name": "dev34-nginx-request-logging-main-20261012", + "config_sha256": "0abbeb43a878db5f3278d37c63ad9976d69ec68b36589eba612e090101d6f172" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/openssl-selfsigned-cert", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/openssl-selfsigned-cert.json", + "job_name": "dev34-openssl-selfsigned-cert-main-20261012", + "config_sha256": "c161f338faf98c5f7912486ef91ed4a75fed16e9a2d379972912ea5a7fb7cc65" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pypi-server", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pypi-server.json", + "job_name": "dev34-pypi-server-main-20261012", + "config_sha256": "df765b4b735973479da446be44004d48247f35c1bcb63cec10f1d1379d50dca6" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-cli", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-cli.json", + "job_name": "dev34-pytorch-model-cli-main-20261012", + "config_sha256": "c0ceb466616fd07e4f072d297f72725d907315343159aaa3648a2fad0247a86c" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/pytorch-model-recovery", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-recovery.json", + "job_name": "dev34-pytorch-model-recovery-main-20261012", + "config_sha256": "4c23f7227327d5e31dfc04064699617b0a281cc2643ad2e4de7d3f420bd73849" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/qemu-alpine-ssh", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/qemu-alpine-ssh.json", + "job_name": "dev34-qemu-alpine-ssh-main-20261012", + "config_sha256": "3ff315e81e99b925ba5e648ee0a7f6c0701ba66dcb55bc5d9f8857cc793da9e9" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/query-optimize", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/query-optimize.json", + "job_name": "dev34-query-optimize-main-20261012", + "config_sha256": "2d836edcb1155bc43a4b0af29387cd4120c785e63d9c2359671235cc099b3719" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-log", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-log.json", + "job_name": "dev34-regex-log-main-20261012", + "config_sha256": "261e68f3c4f07ec633396a798b59cfeb2a6591791dc1a4083cb21a232225390f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-pipeline-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-pipeline-parallelism.json", + "job_name": "dev34-torch-pipeline-parallelism-main-20261012", + "config_sha256": "02f824e64249f601748039cb58f8837ae20fb90b063fa7f9a8614354801417df" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/torch-tensor-parallelism", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-tensor-parallelism.json", + "job_name": "dev34-torch-tensor-parallelism-main-20261012", + "config_sha256": "50e33cc9dcc017515a88dae1d30272dcc1c418a515d53667c2fc502b583c2c8f" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/write-compressor", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/write-compressor.json", + "job_name": "dev34-write-compressor-main-20261012", + "config_sha256": "d9cf9415e397a711e53c289f9b3e841cdb50c23a87b52c3c737ffaacc6c881e1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/constraints-scheduling", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/constraints-scheduling.json", + "job_name": "dev34-constraints-scheduling-main-20261012", + "config_sha256": "e120fe7a8dd1cb2e99a87a796655aec7f288447d5c9943746ea761e67dc09cd0" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/custom-memory-heap-crash", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/custom-memory-heap-crash.json", + "job_name": "dev34-custom-memory-heap-crash-main-20261012", + "config_sha256": "25dc53958e5d425ba17fff7c1bdf4b2e26be33b06d17a2f0c17099b5dec25053" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/dna-assembly", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dna-assembly.json", + "job_name": "dev34-dna-assembly-main-20261012", + "config_sha256": "590762383189a45b3aa8e7a213cb558970c844c023656e2bdda4c6a6f490c7b5" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/llm-inference-batching-scheduler", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/llm-inference-batching-scheduler.json", + "job_name": "dev34-llm-inference-batching-scheduler-main-20261012", + "config_sha256": "3b938028fd68e319692d7a8c8c128e5ad2c6e55c357eec13b36c3aa4120d9b7c" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/path-tracing", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/path-tracing.json", + "job_name": "dev34-path-tracing-main-20261012", + "config_sha256": "ddedc2f8d1fb81f89e808ca7ab9d7eb191fab4a0596fc83bdda703ef8eb863b1" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/schemelike-metacircular-eval", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/schemelike-metacircular-eval.json", + "job_name": "dev34-schemelike-metacircular-eval-main-20261012", + "config_sha256": "eee5401f718704d91c0ce459c77acd6d52ba57d8161569e447fb78c1d75c23f9" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/caffe-cifar-10", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/caffe-cifar-10.json", + "job_name": "dev34-caffe-cifar-10-main-20261012", + "config_sha256": "e3d2654e607e2532b51aa64f313c5f8e05fd35dc5a090eef1ef866ab58fa810d" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/circuit-fibsqrt", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/circuit-fibsqrt.json", + "job_name": "dev34-circuit-fibsqrt-main-20261012", + "config_sha256": "4f3861ec488fd1effb592c31cfb1bc51947d3b35dfcf100f89d2a7e0673d69d4" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/mteb-leaderboard", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/mteb-leaderboard.json", + "job_name": "dev34-mteb-leaderboard-main-20261012", + "config_sha256": "ec77c05d715cc821f6f9ea48541e56227e773b2b9b331bf0b8d3726749812fe4" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/regex-chess", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-chess.json", + "job_name": "dev34-regex-chess-main-20261012", + "config_sha256": "3a9ed5db6443cc1a5e77dd417083db59ab9ed49cc24ac4e43778dee51094a4aa" + }, + { + "arm": "main", + "repeat": 1, + "task": "terminal-bench/build-pov-ray", + "config": "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-pov-ray.json", + "job_name": "dev34-build-pov-ray-main-20261012", + "config_sha256": "55737d5c0867f00be330e35511709ce4eef8472f6c088504c5a688852dd62a8b" + } + ], + "log_dir": "jobs/dev34-main-baseline-20261012-v5-orchestration", + "image_digests_file": "image-digests.json", + "image_digests_sha256": "5fc06fa5fbac1cccd3a4fb2f560878dd79ce314a6b1005183bbdcb4e8e13381f", + "budget_policy": "Native task agent timeout minus 180 seconds, separate 1800-second setup. Native verifier per-attempt time limits; max three attempts, retry only timeouts against the same agent output.", + "execution_policy": "All 34 model trials are fresh. Fixed ascending work-budget order, then task name; consecutive pairs finish before the next pair. No whole-trial retries, task replacements, or reuse of previous model outputs.", + "failure_policy": "Keep all setup failures and verifier errors separate from scored failures. Retain native zeros even when assertions never start, but classify them as verifier errors in analysis.", + "preflight_policy": "Fresh installation-only preflight for all 34 tasks before model work. QEMU has a reviewed verifier dependency profile; the other 33 are explicitly not_configured. A pre-model infrastructure amendment may reuse successful native installation trials only with complete hash-bound provenance and retain all failures.", + "information_policy": "Use frozen current main without prompt or runtime tuning. Do not inspect new trajectories or hidden test contents until all new model outputs freeze. Prior development exposure is disclosed, and each completed trajectory will be audited for benchmark-answer exposure.", + "scope": "One common-main baseline of the existing 34-task development pool, not a comparison with prior mixed-version results and not a random or held-out benchmark.", + "additional_file_sha256": { + "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf": "d0e1ffb6dbd02c54978ba2df386326311a59f6d3774990180f22bcc6adfaedbc", + "benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json": "60d778603976454bb407b8273626851357ee41489cdb0395b06dcc7aab9d151e", + "benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest-v5.json": "bdf3d3ef8b7743700c5294c4a511c686a3928c178a77fd4abcba90ee4557ffcc", + "benchmarks/harbor/scripts/apt_index_proxy.py": "bc15f7adf1d7968d627ceca4683ab86742a7188a1b1955ba1fef37ba48e8a936", + "benchmarks/harbor/scripts/cache_image.py": "639709c03a7b0c3eb6aa78b4e87e6184b49b4b102853a31e33dd832d759180c0", + "benchmarks/harbor/scripts/uv_release_cache.py": "993cc35b1f3a9e8e171b662ab6d4524c5fe059f0d1f0e275a5bb1930111e0ff5", + "benchmarks/harbor/scripts/run_comparison.py": "04f2106f0099a7b4d77e9206a36767f98a5e92c875ec78dd2de858c4936c12a8" + }, + "protocol_amendment": { + "prior_plan": "experiment-v4.json", + "prior_plan_sha256": "e0fd1f09b9238d46419ce8f9b2fcc88d420726ca7eff645ae786e6e97e34f9dd", + "reason": "Accept one native successful installation per task with complete provenance; retain every preceding setup failure and interruption separately. No model outputs are reused or synthesized.", + "model_runs_started_before_amendment": 0, + "original_agent_adapter_source_unchanged": true, + "model_job_order_and_work_budgets_unchanged": true, + "cache_warmup": "Also cache the original CPython 3.14.0+20251120 archive listed by uv 0.9.11. Its filename, size, SHA256, and original release metadata are verified. Python selection and contents remain unchanged; this adds one cache hit to the previously tested original-source fallback. All 34 model tasks use the same final transport configuration." + }, + "accepted_preflight_trials": [ + { + "arm": "main", + "task": "terminal-bench/adaptive-rejection-sampler", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/adaptive-rejection-sampler__5hjbBMn/result.json", + "result_sha256": "3b6bfd13a6085baddbf8a9af3c55aa18b846e73abcac3f96a338f1cfb67b071e", + "bootstrap_sha256": "2adabcd7c4579775ee5370279e0e22ba6b11727a8084c156294b2ba124cf93a5", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/build-cython-ext", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/build-cython-ext__88GjJpX/result.json", + "result_sha256": "f86a832a9cc79551923a7afe79cdd5e92152afaa3e5abe64c4969cf1ccf2d71e", + "bootstrap_sha256": "63aa919cdf558f74286e12bcf6a3f154c7c2a1d9c89352e56303243af7950909", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/cancel-async-tasks", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/cancel-async-tasks__HCzTua9/result.json", + "result_sha256": "f3dbe582c6b0eac1d5abfe161dffe715a822c509f33badc67946d95dbb6e3397", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/cobol-modernization", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/cobol-modernization__aFZfWmu/result.json", + "result_sha256": "4204833ab5c63a62c1cf80ece0dd0c97b17656131df206367a2af86b61e997d0", + "bootstrap_sha256": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/db-wal-recovery", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/db-wal-recovery__6MChPGU/result.json", + "result_sha256": "2f6c09fec6385adc449183d61cd0fc627d106a139ef26f55705a03ff0c281322", + "bootstrap_sha256": "2ffe3efbf9fd9f423c3fc82102152f59af7f9180f8aaf39c46e61c96e2f02d76", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/fix-code-vulnerability", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/fix-code-vulnerability__RvHhCBe/result.json", + "result_sha256": "d7ff8ef09f9d0eb3cfd0dcddcb5f29c69b2f8a0e711256511782806f735fd851", + "bootstrap_sha256": "8d1fa3e359c1a2c87bb9b262c9113a423af3445aa5a20da8a2e6fb4d63b4f67a", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/git-multibranch", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/git-multibranch__cbPbsmj/result.json", + "result_sha256": "55a9a2c3d01c026a1a9a286b98fb04b97fff5c8dfcab598ecbfe0d7b6045f4d6", + "bootstrap_sha256": "391e000959d10d8aa551310ebe7bddd0c70185a684ee8cb543b7e8bf57e5e63b", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/kv-store-grpc", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/kv-store-grpc__oJdqesS/result.json", + "result_sha256": "b42dbdc907ee1a6868c93f22a8a3ccb52fad14c6c9dba571fb76ae7b1766b761", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/log-summary-date-ranges", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v4-20261012/log-summary-date-ranges__i56yJJQ/result.json", + "result_sha256": "7fd10679ae8d884a528e082d32a74a96de2c7e5557320c04b41ab1e0da684a13", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "80ebdb686c042dee53fe5b16fced130a617f78dae1b7836f7676d1a2b615fa7f", + "input_config_sha256": "b4272d950d707c12206d1b57803b4da597a9fa00e37523043e6b2950cbe37de6" + }, + { + "arm": "main", + "task": "terminal-bench/merge-diff-arc-agi-task", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/merge-diff-arc-agi-task__6DDLwSY/result.json", + "result_sha256": "7247da72307a6116b4576f71e8812a957a34256e682d21544fb035f9e6c0215b", + "bootstrap_sha256": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/model-extraction-relu-logits", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/model-extraction-relu-logits__VWhPix2/result.json", + "result_sha256": "fae719c637528aeaa52cda57d02308e92721c97a93c6eb37af1b0319e4630f28", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/multi-source-data-merger", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/multi-source-data-merger__Bm3JhqS/result.json", + "result_sha256": "7e5daa82261b4b6416209c03bf23a532318e34592905c0c9e093747c18b5fc72", + "bootstrap_sha256": "43369761e1c62328eb0c3890c99d441d724fe366a178abf9c568928eb85c4174", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/nginx-request-logging", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/nginx-request-logging__4X3hfes/result.json", + "result_sha256": "b7bec96262fb776976669e20710f5b0583b4556f577877b85d683e1b8a149eb0", + "bootstrap_sha256": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/openssl-selfsigned-cert", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/openssl-selfsigned-cert__v5cNHDt/result.json", + "result_sha256": "d8747031f715cb88df326b1517818ac30dcd603d0b52f0dcf45a8d3d554031c1", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/pypi-server", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/pypi-server__CFroJaU/result.json", + "result_sha256": "de8683222335d13b0da701fe5b1c60d44ace7bd92c687afc595bcb97ab425a19", + "bootstrap_sha256": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/pytorch-model-cli", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/pytorch-model-cli__DoBVGUe/result.json", + "result_sha256": "9a8729566b1c20cea1a04831c49930fde3c014138a5dbb1d836da7949d020750", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/pytorch-model-recovery", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/pytorch-model-recovery__VjeKRPp/result.json", + "result_sha256": "02cdb595a242ba446a09c3ef35c4c9e660225bf709ccf82187ea3692f3bca9ba", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/qemu-alpine-ssh", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v4-20261012/qemu-alpine-ssh__QWU7rTB/result.json", + "result_sha256": "41358b6d7bd947e6a0589f796a95973f0b0ce4b879692100127a7bd83a13c23a", + "bootstrap_sha256": "052b2e131d93ddb3117f1612f23e09e4a0216e77ac7924418a86072bbd3a5d8f", + "manifest_sha256": "80ebdb686c042dee53fe5b16fced130a617f78dae1b7836f7676d1a2b615fa7f", + "input_config_sha256": "b4272d950d707c12206d1b57803b4da597a9fa00e37523043e6b2950cbe37de6" + }, + { + "arm": "main", + "task": "terminal-bench/query-optimize", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/query-optimize__frWo4fV/result.json", + "result_sha256": "ea08b7612ff50f52e71c0427b2d5de603328351b1d4e6ef1b7f52f066e15f866", + "bootstrap_sha256": "c72c3ff607d38944df0e65d64c3a0a941fdc52c3dc17db638b246035ca2d8f89", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/regex-log", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v4-20261012/regex-log__fz29aAB/result.json", + "result_sha256": "3ce6c5a246eb5ac71e43cb97545964b71035a790981ee29152431db010c63a74", + "bootstrap_sha256": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "manifest_sha256": "80ebdb686c042dee53fe5b16fced130a617f78dae1b7836f7676d1a2b615fa7f", + "input_config_sha256": "b4272d950d707c12206d1b57803b4da597a9fa00e37523043e6b2950cbe37de6" + }, + { + "arm": "main", + "task": "terminal-bench/torch-pipeline-parallelism", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/torch-pipeline-parallelism__bSqFxiu/result.json", + "result_sha256": "d106e2ab5083bb214256f6db0ee819c686ebe7d4ddec8c5755bdfd63e33b51e9", + "bootstrap_sha256": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/torch-tensor-parallelism", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/torch-tensor-parallelism__xzKwyM9/result.json", + "result_sha256": "ab71b2eb2c7d4f63fe9df8fe2899beb1582e43a56f850e83af8141b349678511", + "bootstrap_sha256": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/write-compressor", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/write-compressor__gn7TLQW/result.json", + "result_sha256": "63d5bf1fbc13df02d82f5dc6a1050a04fca46d7151f917de3b037fcab4707579", + "bootstrap_sha256": "78617809d0aabc5eaf4ecbb573bab09119e53b9c6a39a8890554717fa5823cf8", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/constraints-scheduling", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/constraints-scheduling__uAF3dBy/result.json", + "result_sha256": "59a970f740b7eeef09f12ca7b67e188bd12eaca5236c52e6d81197affbd167d5", + "bootstrap_sha256": "8ec0c76eef3467924ce96d81cc262bdea7775b259bbeeca2a0563a292696eb05", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/custom-memory-heap-crash", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/custom-memory-heap-crash__g5FutcN/result.json", + "result_sha256": "83888f45d2814ba187bb9768ed323d151486f173697441c6eede910faebe3c21", + "bootstrap_sha256": "cf6bb84c42557ba2a7b7e0ec653bbc1c7cf86c236d1b7e2a6852918dfb8eb7dc", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/dna-assembly", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/dna-assembly__n2emDnw/result.json", + "result_sha256": "45c66c409797157136716053aa695e9330295d7865ee6756b861c49753d1750d", + "bootstrap_sha256": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/llm-inference-batching-scheduler", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/llm-inference-batching-scheduler__sUbNmpr/result.json", + "result_sha256": "b5daf0452dd173f3aac353a5edc9a58c6bd4e96a3120f7270c933fd43798a9f5", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/path-tracing", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/path-tracing__dVMT66x/result.json", + "result_sha256": "f0d13abee1c7612b2ecb09ad78a210d21d2bc9b381f8dc555725520d5a65d25c", + "bootstrap_sha256": "f7c768324bf6e26e7f52bdec891623e4fc98e29f1f24ca02e3847723d8cb55bb", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/schemelike-metacircular-eval", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/schemelike-metacircular-eval__2XMSZho/result.json", + "result_sha256": "6d05110a8471af2db983de5cfaaac869c705b1fb0eaa702e03b39389054ad78e", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/caffe-cifar-10", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/caffe-cifar-10__QUGdMuN/result.json", + "result_sha256": "9b3f7ca9a37f1ed5f480e90cbe8893e1ad16e9e85cbeabe69a1442903766354a", + "bootstrap_sha256": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/circuit-fibsqrt", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v4-20261012/circuit-fibsqrt__eEHjF3c/result.json", + "result_sha256": "4364cc1a2eae973ce8b0ed8dabe45823c97597c616196e1cb05bdc918a876516", + "bootstrap_sha256": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "manifest_sha256": "80ebdb686c042dee53fe5b16fced130a617f78dae1b7836f7676d1a2b615fa7f", + "input_config_sha256": "b4272d950d707c12206d1b57803b4da597a9fa00e37523043e6b2950cbe37de6" + }, + { + "arm": "main", + "task": "terminal-bench/mteb-leaderboard", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/mteb-leaderboard__RVuUyDg/result.json", + "result_sha256": "ed5bec732ef3c67264700698308062f8aee2a161747eae5626c6719f8114cca5", + "bootstrap_sha256": "7c324cb3af6adf375e27d5cefcc67e8631dae65ac0f3c52687f893fa69b96c4e", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/regex-chess", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v3-20261012/regex-chess__5SfYMHh/result.json", + "result_sha256": "0e519db59b54fe485678c721a78b8472d4a9fcb782cd79df9198652c6a9232fb", + "bootstrap_sha256": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "manifest_sha256": "30a540ad811a7cdce85137fc6a2f8c78601283485d8aac0dc0f581fd7d80aeb5", + "input_config_sha256": "f7b821f48eb1596b7d5658ca5c62a4573a481f5f05919bd415b5d5cb9566bc20" + }, + { + "arm": "main", + "task": "terminal-bench/build-pov-ray", + "result_path": "jobs/dev34-baseline-main-source/jobs/dev34-main-install-only-v4-20261012/build-pov-ray__zUzSVXp/result.json", + "result_sha256": "e20c6e192853dd9d2ee1b8532d9ea26425361b733248422c1f362265c52757c5", + "bootstrap_sha256": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4", + "manifest_sha256": "80ebdb686c042dee53fe5b16fced130a617f78dae1b7836f7676d1a2b615fa7f", + "input_config_sha256": "b4272d950d707c12206d1b57803b4da597a9fa00e37523043e6b2950cbe37de6" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/python-main-release-manifest.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/python-main-release-manifest.json new file mode 100644 index 0000000..dfaccad --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/python-main-release-manifest.json @@ -0,0 +1,16 @@ +{ + "registered_at": "2026-10-12T17:26:31.124593+00:00", + "metadata_source": "Original public GitHub release API, fetched before this registration", + "assets": [ + { + "version": "20251120", + "filename": "cpython-3.14.0+20251120-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "url": "https://github.com/astral-sh/python-build-standalone/releases/download/20251120/cpython-3.14.0%2B20251120-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "digest": "sha256:66f385305ae0eefd6b65e2cf942bc91d943a61e26fb7581e029f760a2b04f393", + "size": 35376131, + "asset_id": 318881850, + "release_metadata_sha256": "e3a737862d2dddf11c73bfef6c2f2579e9d4ce5bd271cca53790c998590dee58", + "repository": "astral-sh/python-build-standalone" + } + ] +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/adaptive-rejection-sampler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/adaptive-rejection-sampler.json new file mode 100644 index 0000000..42caf2d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/adaptive-rejection-sampler.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/adaptive-rejection-sampler", + "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-cython-ext.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-cython-ext.json new file mode 100644 index 0000000..db910b8 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-cython-ext.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-cython-ext", + "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-pov-ray.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-pov-ray.json new file mode 100644 index 0000000..bca06b9 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/build-pov-ray.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 11820 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/build-pov-ray", + "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/caffe-cifar-10.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/caffe-cifar-10.json new file mode 100644 index 0000000..cfdca4d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/caffe-cifar-10.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/caffe-cifar-10", + "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cancel-async-tasks.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cancel-async-tasks.json new file mode 100644 index 0000000..45694c8 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cancel-async-tasks.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cancel-async-tasks", + "ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/circuit-fibsqrt.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/circuit-fibsqrt.json new file mode 100644 index 0000000..e2f1bbb --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/circuit-fibsqrt.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/circuit-fibsqrt", + "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cobol-modernization.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cobol-modernization.json new file mode 100644 index 0000000..da8a130 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/cobol-modernization.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/cobol-modernization", + "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/constraints-scheduling.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/constraints-scheduling.json new file mode 100644 index 0000000..fa9807c --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/constraints-scheduling.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1020 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/constraints-scheduling", + "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/custom-memory-heap-crash.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/custom-memory-heap-crash.json new file mode 100644 index 0000000..a14150b --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/custom-memory-heap-crash.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/custom-memory-heap-crash", + "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/db-wal-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/db-wal-recovery.json new file mode 100644 index 0000000..f9aa117 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/db-wal-recovery.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json new file mode 100644 index 0000000..1503257 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json @@ -0,0 +1,100 @@ +{ + "services": { + "main": { + "volumes": [ + { + "type": "bind", + "source": "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v3/apt-cache.conf", + "target": "/etc/apt/apt.conf.d/99nanopy-index-cache", + "read_only": true + } + ], + "environment": { + "UV_INSTALLER_GITHUB_BASE_URL": "http://127.0.0.1:18743", + "UV_PYTHON_INSTALL_MIRROR": "http://127.0.0.1:18743/astral-sh/python-build-standalone/releases/download" + } + }, + "apt-index-cache": { + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "network_mode": "service:main", + "entrypoint": [ + "python3", + "/proxy/apt_index_proxy.py", + "--bind", + "127.0.0.1", + "--port", + "18742", + "--cache", + "/cache", + "--workers", + "16" + ], + "user": "1000:1000", + "read_only": true, + "cap_drop": [ + "ALL" + ], + "security_opt": [ + "no-new-privileges:true" + ], + "mem_limit": "256m", + "cpus": 0.5, + "volumes": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/apt_index_proxy.py:/proxy/apt_index_proxy.py:ro", + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/cache_image.py:/proxy/cache_image.py:ro", + "/home/minix/projects/nanoPyCodeAgent/.cache/nanopy-harbor/apt-indexes:/cache" + ], + "healthcheck": { + "test": [ + "CMD", + "python3", + "-c", + "import socket; socket.create_connection((\"127.0.0.1\",18742), timeout=2).close()" + ], + "interval": "1s", + "timeout": "3s", + "retries": 30 + } + }, + "uv-release-cache": { + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "network_mode": "service:main", + "entrypoint": [ + "python3", + "/proxy/uv_release_cache.py", + "serve", + "--cache", + "/cache", + "--manifest", + "/manifest.json" + ], + "user": "1000:1000", + "read_only": true, + "cap_drop": [ + "ALL" + ], + "security_opt": [ + "no-new-privileges:true" + ], + "cpus": 0.5, + "mem_limit": "256m", + "volumes": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/uv_release_cache.py:/proxy/uv_release_cache.py:ro", + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/scripts/cache_image.py:/proxy/cache_image.py:ro", + "/home/minix/projects/nanoPyCodeAgent/.cache/nanopy-harbor/uv-releases:/cache:ro", + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/dependency-release-manifest-v5.json:/manifest.json:ro" + ], + "healthcheck": { + "test": [ + "CMD", + "python3", + "-c", + "import socket; socket.create_connection((\"127.0.0.1\",18743), timeout=2).close()" + ], + "interval": "1s", + "timeout": "3s", + "retries": 30 + } + } + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dna-assembly.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dna-assembly.json new file mode 100644 index 0000000..2e7616d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dna-assembly.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/dna-assembly", + "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/fix-code-vulnerability.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/fix-code-vulnerability.json new file mode 100644 index 0000000..4ff2a03 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/fix-code-vulnerability.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/fix-code-vulnerability", + "ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/git-multibranch.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/git-multibranch.json new file mode 100644 index 0000000..61de0d0 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/git-multibranch.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/git-multibranch", + "ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/kv-store-grpc.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/kv-store-grpc.json new file mode 100644 index 0000000..eadc233 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/kv-store-grpc.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/kv-store-grpc", + "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/llm-inference-batching-scheduler.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/llm-inference-batching-scheduler.json new file mode 100644 index 0000000..772004a --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/llm-inference-batching-scheduler.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/llm-inference-batching-scheduler", + "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/log-summary-date-ranges.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/log-summary-date-ranges.json new file mode 100644 index 0000000..9dd1e1d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/log-summary-date-ranges.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/log-summary-date-ranges", + "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/merge-diff-arc-agi-task.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/merge-diff-arc-agi-task.json new file mode 100644 index 0000000..17219d5 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/merge-diff-arc-agi-task.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/merge-diff-arc-agi-task", + "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/model-extraction-relu-logits.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/model-extraction-relu-logits.json new file mode 100644 index 0000000..ce87f57 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/model-extraction-relu-logits.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/model-extraction-relu-logits", + "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/mteb-leaderboard.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/mteb-leaderboard.json new file mode 100644 index 0000000..a943b2f --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/mteb-leaderboard.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/mteb-leaderboard", + "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/multi-source-data-merger.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/multi-source-data-merger.json new file mode 100644 index 0000000..c2289eb --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/multi-source-data-merger.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/multi-source-data-merger", + "ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/nginx-request-logging.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/nginx-request-logging.json new file mode 100644 index 0000000..96595b4 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/nginx-request-logging.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/nginx-request-logging", + "ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/openssl-selfsigned-cert.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/openssl-selfsigned-cert.json new file mode 100644 index 0000000..f780b36 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/openssl-selfsigned-cert.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/openssl-selfsigned-cert", + "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/path-tracing.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/path-tracing.json new file mode 100644 index 0000000..bbc4fe3 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/path-tracing.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 1620 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/path-tracing", + "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pypi-server.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pypi-server.json new file mode 100644 index 0000000..8c9d695 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pypi-server.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pypi-server", + "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-cli.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-cli.json new file mode 100644 index 0000000..0a43e17 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-cli.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-cli", + "ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-recovery.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-recovery.json new file mode 100644 index 0000000..afad7c7 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/pytorch-model-recovery.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/pytorch-model-recovery", + "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/qemu-alpine-ssh.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/qemu-alpine-ssh.json new file mode 100644 index 0000000..c2c7871 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/qemu-alpine-ssh.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/qemu-alpine-ssh", + "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/query-optimize.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/query-optimize.json new file mode 100644 index 0000000..95dc238 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/query-optimize.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/query-optimize", + "ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-chess.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-chess.json new file mode 100644 index 0000000..3bef66e --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-chess.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 3420 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-chess", + "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-log.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-log.json new file mode 100644 index 0000000..7085c28 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/regex-log.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/regex-log", + "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/schemelike-metacircular-eval.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/schemelike-metacircular-eval.json new file mode 100644 index 0000000..b98d5e2 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/schemelike-metacircular-eval.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 2220 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/schemelike-metacircular-eval", + "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-pipeline-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-pipeline-parallelism.json new file mode 100644 index 0000000..797f01d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-pipeline-parallelism.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-tensor-parallelism.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-tensor-parallelism.json new file mode 100644 index 0000000..a7be20d --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/torch-tensor-parallelism.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/torch-tensor-parallelism", + "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/write-compressor.json b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/write-compressor.json new file mode 100644 index 0000000..fa42bf6 --- /dev/null +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/write-compressor.json @@ -0,0 +1,28 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "agents": [ + { + "name": "harbor_adapter:NanoPyCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "max_turns": 100, + "max_tokens": 65536, + "time_budget_seconds": 720 + } + } + ], + "tasks": [ + { + "name": "terminal-bench/write-compressor", + "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e" + } + ], + "environment": { + "type": "docker", + "extra_docker_compose": [ + "/home/minix/projects/nanoPyCodeAgent/benchmarks/harbor/configs/dev34-main-baseline-20261012/transport-v5/dependency-cache.compose.json" + ] + } +} From dd3f59f5980adb7f3e0f0351e9e1d7c3ea79bb95 Mon Sep 17 00:00:00 2001 From: minixalpha Date: Mon, 12 Oct 2026 15:24:49 -0700 Subject: [PATCH 12/15] test(benchmarks): record the audited 34-task common-main baseline --- .../dev34-main-baseline-20261012/README.md | 7 + .../reports/dev34-main-baseline-20261012.md | 300 + .../results/dev34-main-baseline-20261012.json | 5391 +++++++++++++++++ 3 files changed, 5698 insertions(+) create mode 100644 benchmarks/harbor/reports/dev34-main-baseline-20261012.md create mode 100644 benchmarks/harbor/results/dev34-main-baseline-20261012.json diff --git a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md index a3b5b8d..8af6414 100644 --- a/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md +++ b/benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md @@ -6,6 +6,13 @@ and origin/main pointed to that commit when the run was registered. Earlier pilot20, repeat experiments, and the twelve-task baseline remain separate; none of their model outputs or rewards count toward this run. +**Completed:** 28/34 passed (82.35%), with six scored failures, zero model-phase +setup errors, zero verifier errors, and no verifier retries. See the +[final report](../../reports/dev34-main-baseline-20261012.md) and +[audited results](../../results/dev34-main-baseline-20261012.json). +One passing tensor-parallelism trial accessed external benchmark revision +metadata; its raw score is retained with a separate exposure flag. + [Selection](selection.json) is the union of the previously exercised 22 tasks and the twelve additions. It uses the same pinned 89-task dataset revision. The remaining 55 packages stay unopened. These are development tasks with prior diff --git a/benchmarks/harbor/reports/dev34-main-baseline-20261012.md b/benchmarks/harbor/reports/dev34-main-baseline-20261012.md new file mode 100644 index 0000000..9cedb3c --- /dev/null +++ b/benchmarks/harbor/reports/dev34-main-baseline-20261012.md @@ -0,0 +1,300 @@ +# Common-main baseline: all 34 development tasks + +All 34 tasks received one fresh model trial on main +`2eb99b608dfb39464d17d109a4a8f11afda607a9`: **28 passed and six failed +(82.35%)**. Every official verifier executed tests. The model phase had zero +setup errors, zero verifier errors, and zero verifier retries. + +One passing trial, `torch-tensor-parallelism`, accessed external Terminal-Bench +revision metadata. Its official pass is retained and the exposure is reported +separately below. This is an already exposed development pool, not a held-out +benchmark or a claim of evaluation without external benchmark-material access. + +The [machine-readable results](../results/dev34-main-baseline-20261012.json) +contain all trial identifiers, source/task/configuration provenance, native +scores, test counts, timing, partial usage, failure diagnoses, and evidence +hashes. Raw trajectories and logs remain local because they can contain secrets. + +## Runtime and protocol + +Both local main and origin/main pointed to `2eb99b6` at registration. Experiment +files and transport utilities live on `benchmark/dev34-main-baseline-20261012`; +the evaluated agent and adapter came from a separate, unchanged checkout of +that main commit. No agent or prompt optimization was included. + +| Property | Value | +| --- | --- | +| Pool | Previously exercised 22 tasks plus the 12 additions | +| Dataset | `terminal-bench/terminal-bench-2-1`, 89-task catalog | +| Dataset revision | `sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a` | +| Model | `deepseek/deepseek-flash` | +| Endpoint | Official DeepSeek Anthropic-compatible endpoint | +| Prompt | Main's 385-word headless prompt | +| Limits | 100 replies; 65,536 output tokens per request | +| Sampling | Temperature and reasoning effort unset | +| Model trials | Exactly one per task; no replacements or whole-trial retries | +| Queue | Ascending work budget, then name; consecutive pairs; concurrency two | +| Setup | Separate 1,800-second budget | +| Agent work | Native task timeout minus 180 seconds | +| Verifier | Original per-attempt limit; timeout-only retries allowed, none used | + +The final [v5 registration](../configs/dev34-main-baseline-20261012/experiment-v5.json) +has SHA256 `f7bc7b329767f4b8007956462f6cdbd4407d386af493524df27ba313c0a4e4c3`. +All 34 jobs installed the same wheel, SHA256 +`f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350`. +The prompt SHA256 is +`37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d`. + +The standard runner was invoked from `jobs/dev34-baseline-main-source` through +`uv run --project benchmarks/harbor python -m harbor_adapter run --config ...`. +Source snapshots, installed wheel digests, complete task-file hashes, accepted +preflight provenance, configuration hashes, and queue barriers were checked for +every job. All 34 original image IDs and registry digests were unchanged in the +post-run audit, and no scored-trial containers remained. + +All outputs froze at **2026-10-12 22:12:55 UTC** before new trajectory or +hidden-test-content review. The freeze manifest is embedded in the results JSON. +Existing pilot, repeat, and twelve-task results were not reused in this score. +The remaining 55 packages stayed unopened. + +## Per-task results + +Test counts below come from official pytest stdout and include parameterized +cases. Native CTRF summaries are preserved separately: they deduplicate function +names for pipeline parallelism (3 instead of 4 tests) and tensor parallelism +(3 instead of 13). Agent seconds include its model/tool work, not setup or the +official verifier. + +| Task | Raw reward | Tests passed / failed | Agent seconds | +| --- | ---: | ---: | ---: | +| adaptive-rejection-sampler | 0 | 0 / 9 | 720.3 | +| build-cython-ext | 1 | 11 / 0 | 586.5 | +| cancel-async-tasks | 1 | 6 / 0 | 378.9 | +| cobol-modernization | 1 | 3 / 0 | 551.7 | +| db-wal-recovery | 0 | 5 / 2 | 359.6 | +| fix-code-vulnerability | 1 | 6 / 0 | 58.7 | +| git-multibranch | 1 | 1 / 0 | 238.2 | +| kv-store-grpc | 0 | 5 / 2 | 439.2 | +| log-summary-date-ranges | 1 | 2 / 0 | 23.1 | +| merge-diff-arc-agi-task | 1 | 5 / 0 | 123.2 | +| model-extraction-relu-logits | 1 | 1 / 0 | 380.0 | +| multi-source-data-merger | 1 | 3 / 0 | 33.1 | +| nginx-request-logging | 1 | 8 / 0 | 70.0 | +| openssl-selfsigned-cert | 1 | 6 / 0 | 70.0 | +| pypi-server | 1 | 1 / 0 | 129.1 | +| pytorch-model-cli | 1 | 6 / 0 | 392.7 | +| pytorch-model-recovery | 1 | 5 / 0 | 363.6 | +| qemu-alpine-ssh | 1 | 1 / 0 | 555.5 | +| query-optimize | 0 | 5 / 1 | 373.8 | +| regex-log | 1 | 1 / 0 | 72.6 | +| torch-pipeline-parallelism | 0 | 2 / 2 | 720.9 | +| torch-tensor-parallelism (exposure flagged) | 1 | 13 / 0 | 559.6 | +| write-compressor | 1 | 3 / 0 | 394.3 | +| constraints-scheduling | 1 | 3 / 0 | 32.0 | +| custom-memory-heap-crash | 1 | 6 / 0 | 245.3 | +| dna-assembly | 1 | 1 / 0 | 1162.7 | +| llm-inference-batching-scheduler | 1 | 6 / 0 | 614.5 | +| path-tracing | 1 | 5 / 0 | 259.9 | +| schemelike-metacircular-eval | 1 | 1 / 0 | 1777.1 | +| caffe-cifar-10 | 1 | 6 / 0 | 2448.9 | +| circuit-fibsqrt | 1 | 3 / 0 | 610.7 | +| mteb-leaderboard | 1 | 2 / 0 | 1380.4 | +| regex-chess | 0 | 2 / 2 | 2818.8 | +| build-pov-ray | 1 | 3 / 0 | 338.1 | + +Overall, pytest executed 155 cases: 137 passed and 18 failed, with no skips or +errors. The task score remains 28/34; individual test cases are not equally +weighted tasks. The previous 22-task subset scored 19/22 and the twelve additions +scored 9/12 in this run. + +## Why the six tasks failed + +Step references are ATIF `step_id` values in each locally retained trajectory. +The result JSON binds those files and official outputs by SHA256. + +| Task | Observed failure | Verification lesson | +| --- | --- | --- | +| adaptive-rejection-sampler | No `/app/ars.R` was submitted. Installation delays and interrupted model streams exhausted the 720-second work budget. | Produce a minimal deliverable early; bound installation and recovery work. This run provides no evidence about a submitted ARS algorithm. | +| db-wal-recovery | Opening the original SQLite database discarded its encrypted WAL before backup. Guessed values missed WAL updates. | Preserve all inputs and sidecars before stateful reads; validate recovered values against original evidence. | +| kv-store-grpc | `SetValRequest` used `val`; the task required `value`. The self-client used the same incorrect schema. | Derive an independent API contract from the specification. Self-consistency does not prove interface compliance. | +| query-optimize | Output was correct, but the query was too slow against the official optimized reference. | Establish repeatable target-runtime measurements and test candidate query plans after correctness is established. | +| torch-pipeline-parallelism | Layer calls lacked required rotary position embeddings. A real self-test found the error near the deadline. | Test the installed API with a tiny forward/backward pass early; propagate process failures and leave repair time. | +| regex-chess | Castling was allowed when the white king was already in check. | Supplement random differential tests with explicit negative cases for each special-rule precondition. | + +### Sampler: missing output after interrupted execution + +Installation calls at steps 5, 9, and 13 each hit the bash timeout, including +attempts to launch background work. R was available by step 15, roughly 6.5 +minutes into the work budget. The next two model calls ended in +`RemoteProtocolError` after approximately 241 and 29 seconds; the last call +reached the deadline after another 58 seconds. No implementation write completed. + +All nine official tests ran and failed on the missing file. This is a scored +failure with an operational cause, not a setup failure or a demonstrated +algorithmic error. The registered preflight did not cover R installation. + +### WAL recovery: preserving evidence must precede inspection + +Step 5 inspected part of the WAL and opened the original database concurrently. +SQLite discarded the invalid encrypted WAL. Backup/decode attempts at step 7 +came too late. The final JSON contained 11 plausible fruit records with values +following `id * 100`, and shape/count checks passed. Two official tests caught +incorrect values, including `100` where the recovered value should have been +`150` for id 1. The final agent message at step 53 disclosed the lost WAL and the +inference; this was not an undisclosed claim of successful recovery. + +### gRPC: an independent consumer would have caught the mismatch + +Step 7 defined `SetValRequest.val`. Tests at steps 9, 13, 23, and 37 used the +generated client and the same field. They exercised real RPCs, overwrites, +independent keys, empty keys, and an integer boundary, but never constructed the +request specified by the task. Official calls using `value=999` and `value=42` +failed before reaching the server. A descriptor assertion or independently +specified client would address the actual gap. + +### SQL: correct results are insufficient for a performance task + +The verifier measured a reference median of **0.291814 s** and a solution median +of **0.371624 s**, about 27.35% slower; the allowed multiplier was 1.05. The agent +did establish an exact 500-row output match and preserved the original database, +using indexes only on a copy to obtain reference output. Its attempted repeated +timing command at step 33 failed because `/usr/bin/time` was missing. Later +shell timings around 0.48 s looked good against the much slower original query. + +The optimized reference and precise threshold were hidden from the agent. +The actionable gap is reliable comparative performance work and optimization +margin, not expecting the agent to know the hidden reference. + +### Pipeline parallelism: verification started too late + +The implementation was written at step 11 before PyTorch/Transformers were +available. The first substantive test, written at step 27, failed in a reference +test hook. After that hook was repaired, step 31 exposed +`TypeError: cannot unpack non-iterable NoneType object` when attention unpacked +`position_embeddings`. The deadline arrived before a repair. + +Both official world-size tests reached submitted code and failed with the same +error; two structural checks passed. This is not a verifier installation error. +The self-test commands also piped output through `tail`, obscuring failing +process exit status. Early smoke tests, explicit process status, and bounded +distributed tests would make the remaining budget more useful. + +### Regex chess: a large test count missed a specific negative condition + +Two official games exposed an extra kingside castle from a position where the +white king on e1 was already checked: once by an e8 rook, once by a d3 knight. +The other game suite and the rule-size check passed. + +Steps 107, 115, and 117 changed castling generation and attack-filter ordering. +The intermediate X0-to-X1 state was still created before the first filter, so +it could survive even when the initial king position was attacked. Later +castling states could consequently escape that initial check. + +The final implementation had substantive independent verification against +`python-chess`: 618 sampled positions in one collection and 600 in another +passed after the final changes, as did the supplied check script. These counts +do not imply unique positions or complete rule coverage. Explicit examples of +castling out of check, through check, and into check were needed; the final claim +that all three were excluded exceeded the evidence. + +## Dependency preparation and environment preservation + +Slow or interrupted downloads required amendments **before any model trial**. +The historical attempts remain recorded separately from the 34 scored trials. + +| Install-only plan | Attempts | Successful | Setup errors | Cancelled / interrupted | +| --- | ---: | ---: | ---: | ---: | +| v1 | 6 | 0 | 5 | 1 | +| v2 | 9 | 0 | 7 | 2 | +| v3 | 34 | 29 | 3 | 2 | +| v4 | 5 | 5 | 0 | 0 | +| Total | 54 | 34 | 15 | 5 | + +V1 encountered setup timeouts and uv TLS errors. V2's host proxy was unreachable +from task containers and was stopped. V3 used isolated loopback sidecars and +completed 29 installations; two uv archive downloads and one managed-Python +download failed, and two remaining installations were interrupted to apply the +verified archive cache. V4 completed the five unresolved installations. + +V5 accepted the 29 successful v3 records and five v4 records only after complete +native provenance checks. QEMU's registered `qemu-alpine-ssh-v1` verifier +dependency profile passed. The other **33 profiles are `not_configured`**; +successful agent installation does not certify their verifier dependencies. +All install-only trials ran no models or official task assertions. + +The final transport cached original immutable APT indexes and original uv / GNU +CPython release archives with full digest verification. Mutable APT release +metadata and packages were not proxy-cached; native APT verification remained. +Release-cache hits required matching filename, size, and SHA256. Original +package origins, Python selection, task images, resources, official scripts, +and verifier per-attempt time limits were preserved. Model/API traffic used its +normal route. + +Each task had separate cache sidecars in its own network namespace, no published +ports, no task filesystem or credential mounts, and separate CPU/memory limits. +All 34 model trials used the final v5 transport. Details, original failure +evidence, manifests, and network integration probes are linked from the +[run configuration README](../configs/dev34-main-baseline-20261012/README.md). +The supporting Harbor workflow suite passed **167 tests** before model work; +the corresponding branch commit also passed all three CI checks. + +## Exposure and publication audit + +All 34 frozen trajectories were scanned for benchmark references, verifier +paths, external hosts, and exact provider credential values. Flagged commands +and returned observations were reviewed alongside the six failure traces. + +- **Tensor parallelism:** exact task-phrase searches at steps 15–23 reached a + Terminal-Bench dataset README. Step 23 returned revision metadata; step 25 + fetched the external README and displayed its pre-scattered-input clarification. + That requirement was already present in the supplied instruction. No solution + code or hidden test content was observed in these results. The pass remains + official raw reward 1, with benchmark-material exposure explicitly flagged. +- **Local test paths:** tensor parallelism probed an absent `/app/tests`; WAL + recovery and Git multibranch listed empty verifier-log directories. Other + flagged paths were public upstream tests or agent-created tests/data. + Memory-crash benchmark references were preserved source canary comments. +- **Other network activity:** reviewed accesses concerned dependencies, upstream + projects, CIFAR data, NEB documentation, and the public MTEB/Scandinavian + leaderboard sources relevant to those tasks. No other benchmark-answer + retrieval was observed in this review. +- **Credentials:** WAL recovery's environment inspection exposed the provider + credential to its own tool observations at steps 27 and 49. Those raw traces + remain local and are not included in the PR. No transmission of that credential + to another external destination was observed in the reviewed commands. + +This audit cannot prove absence of all exposure or training contamination. +It does not retroactively remove tasks, replace trials, or change native rewards. +Future experiments should declare and enforce an external benchmark-material +policy before execution and prevent task tools from reading provider secrets. + +## Time, cost, and comparison limits + +The model jobs ran from approximately **17:29 to 22:12 UTC on October 12**: +**283.25 minutes** elapsed, excluding dependency preflight. Summed agent work +was 321.38 minutes, agent installation 32.96 minutes, and verifier time 50.01 +minutes; these sums differ from wall time because of concurrency and pair +barriers. + +Recorded usage totals 60,464,780 prompt tokens (59,660,798 cached) and 1,603,196 +completion tokens. The **known estimated cost subtotal is $2.523**, not a complete +bill. Usage/cost records are incomplete for the sampler and pipeline trials; +missing calls have not been priced as zero. + +The [earlier twelve-task baseline](dev12-main-baseline-20261011.md) used the +same main commit and scored 10/12. Those tasks score 9/12 here: PyTorch CLI +changed from fail to pass, sampler and SQL changed from pass to fail, and WAL +recovery failed in both. Trial randomness and transport/environment history +differ; this is not evidence of an agent code improvement or regression. + +The 34-task pool remains useful for development: retain all 28 passing tasks as +regression checks and use the six failures for diagnosis. Agent experiments +should use matched baseline/candidate repeats under the same registered +conditions, measuring task pass rate, verification evidence, time, and exposure +separately. One pass does not establish task stability. Keep the remaining 55 +unopened tasks for a later, predeclared evaluation after choosing the candidate. + +The next agent changes suggested by this evidence are input preservation, +independent contract checks, early execution of small tests, reliable exit/error +reporting, targeted negative cases, and repeatable performance measurements. +These are hypotheses for a subsequent experiment; none was applied to this run. diff --git a/benchmarks/harbor/results/dev34-main-baseline-20261012.json b/benchmarks/harbor/results/dev34-main-baseline-20261012.json new file mode 100644 index 0000000..d71cff0 --- /dev/null +++ b/benchmarks/harbor/results/dev34-main-baseline-20261012.json @@ -0,0 +1,5391 @@ +{ + "schema_version": 1, + "experiment": "dev34-main-baseline-20261012", + "collected_at": "2026-10-12T22:13:05.117500+00:00", + "cohort": "existing_34_task_development_pool", + "dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a", + "model": "deepseek/deepseek-flash", + "endpoint": "https://api.deepseek.com/anthropic", + "runtime_source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "plan_path": "benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v5.json", + "plan_sha256": "f7bc7b329767f4b8007956462f6cdbd4407d386af493524df27ba313c0a4e4c3", + "model_output_freeze": { + "observed_at": "2026-10-12T22:12:55.450294+00:00", + "plan_path": "benchmarks/harbor/configs/dev34-main-baseline-20261012/experiment-v5.json", + "plan_sha256": "f7bc7b329767f4b8007956462f6cdbd4407d386af493524df27ba313c0a4e4c3", + "runtime_source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "boundary": "All 34 queued native trials ended before new trajectory or hidden-test content review. Setup errors, if any, have no model output and remain separate.", + "trials": [ + { + "task": "terminal-bench/adaptive-rejection-sampler", + "trial": "adaptive-rejection-sampler__zWw9zAU", + "finished_at": "2026-10-12T17:46:58.286796Z", + "model_started": true, + "trajectory_sha256": "d6b054f53f68a111f2d1b42ba3774eb2070acd798d02a441d86417b36f532069", + "raw_rewards": { + "reward": 0.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/agent/nanopycodeagent.txt": "a010f4fde40e491dfeaa0b565f6471c34388a4645c8169362f6ab19dae341054", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/agent/trajectory.json": "d6b054f53f68a111f2d1b42ba3774eb2070acd798d02a441d86417b36f532069", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/config.json": "6cc16382000a695512105efad71dc43285d6e298f2f14de50da49304f1dd52df", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/lock.json": "36641f7cbd1f7091eb2158b4ebd53bb1505955b7a250146b6209b1ab11d47a6d", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/result.json": "0442d30c596bc8ac89b91c4497d0653820a1e7209ecbca32c9c4f9cdc089788e", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/trial.log": "249f8bb7b1cc2b18adcad2ff83e9eee8b0e7a9a01949abeb5ca5714dab78182c", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/attempts/1/ctrf.json": "6b946a99b4473bb3ab0692440bfb35c38063c91701faa62fd7ce0bc6713c1504", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/attempts/1/test-stdout.txt": "e27a2fb2f987a71c07f8ff392065d03786823880ac9a44a4a3c131c4dbd3b0e1", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/ctrf.json": "6b946a99b4473bb3ab0692440bfb35c38063c91701faa62fd7ce0bc6713c1504", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/retry-summary.json": "07fb6406bef4efcfb135bd2e1c04ebbc64344c676d29b2afffd83d75bee8ff83", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/test-stdout.txt": "e27a2fb2f987a71c07f8ff392065d03786823880ac9a44a4a3c131c4dbd3b0e1", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-adaptive-rejection-sampler-main-20261012.json": "b1efcbfa4cff1b14211e959cc1fe565d13e99142e612df0b442eebc60e6b9a83" + } + }, + { + "task": "terminal-bench/build-cython-ext", + "trial": "build-cython-ext__vxC7HQK", + "finished_at": "2026-10-12T17:42:19.799897Z", + "model_started": true, + "trajectory_sha256": "e2a1ef6b2cd96535728ca6060e34d97bf8d8e1698c3c430f7d7baf4c0b60b7f6", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/agent/bootstrap.json": "63aa919cdf558f74286e12bcf6a3f154c7c2a1d9c89352e56303243af7950909", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/agent/nanopycodeagent.txt": "f4fc35b2352d08c2ab356de779e1313c585b8ae5ae6dde85d53402057fc88e4f", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/agent/trajectory.json": "e2a1ef6b2cd96535728ca6060e34d97bf8d8e1698c3c430f7d7baf4c0b60b7f6", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/config.json": "ec9d6e6cd305f38ee92c5d0856e1117623f629b6073f4d24c6bb5c210ef609b9", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/lock.json": "279c10bde567ecc135eb39374bcba163ecaf81ed4aad1dc3fa899158d354036b", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/result.json": "46c56969319287709f5819508f75cd6b99637b86d005c0bc22284232d95ba4b1", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/trial.log": "5fb10f86471c0bc1965e0630f11d313dfc725211e3a9439411e302869b654167", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/attempts/1/ctrf.json": "84a434bc4e7cfdf70179b50789419c739f5bdf32ad70a3334802d214236f742f", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/attempts/1/test-stdout.txt": "e7df7cbe3f62a09ceb0848405a356c3b092328438aa5cf49860825e85be4af51", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/ctrf.json": "84a434bc4e7cfdf70179b50789419c739f5bdf32ad70a3334802d214236f742f", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/retry-summary.json": "2e3b4876bc6fb35274f4aac59dd86690647ca84050b1737ce9f0759565a53b21", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/test-stdout.txt": "e7df7cbe3f62a09ceb0848405a356c3b092328438aa5cf49860825e85be4af51", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-build-cython-ext-main-20261012.json": "4be984fe0ce9c7af4ceff61103422a5894405a51a931abc968427edf4dd761ed" + } + }, + { + "task": "terminal-bench/cancel-async-tasks", + "trial": "cancel-async-tasks__7u3Kswi", + "finished_at": "2026-10-12T17:55:48.069843Z", + "model_started": true, + "trajectory_sha256": "f6d4e0ab94c0b096ea3ed731327b7ebd09b42c64b39b82f43f16a6505abb194b", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/agent/nanopycodeagent.txt": "27d5aa7fd02000414b5a6d0d4313c64f4ff5aed536ad1a7b823859aba39bd81c", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/agent/trajectory.json": "f6d4e0ab94c0b096ea3ed731327b7ebd09b42c64b39b82f43f16a6505abb194b", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/config.json": "2587c4c1262adda409e18393581cfd74a77a139648137bdc82865637663d7c30", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/lock.json": "cecb135e5633a20a377ec925b71f6876cadfdca718f1d9094cabd52455f3068b", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/result.json": "fdde6cfb40becf1e0a06a1afb3b065d16a935f21a33cc9a16aafe4e98cc06f3f", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/trial.log": "04a5a7c1d9d5c0e3f1b09b523dfe30d5dd36477e5542583577faa710a7a53b46", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/attempts/1/ctrf.json": "7d8f31aa11da4cd90906331c028f8db13dbc910a5f3a2cb880b1566052b872ed", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/attempts/1/test-stdout.txt": "387309447ada1c5ddc13e862f1e6901ab115497bdd564bbaf6a7522379cf4b44", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/ctrf.json": "7d8f31aa11da4cd90906331c028f8db13dbc910a5f3a2cb880b1566052b872ed", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/retry-summary.json": "15b57aee1acb8d1b95a862de49aaae4b77a4adc60312d5aafae0afbfdfffa7f6", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/test-stdout.txt": "387309447ada1c5ddc13e862f1e6901ab115497bdd564bbaf6a7522379cf4b44", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-cancel-async-tasks-main-20261012.json": "9d7eb4fbab558cffb75ecb3b1e874ac79c4864e3a1f494d6a22f5e51334f27e9" + } + }, + { + "task": "terminal-bench/cobol-modernization", + "trial": "cobol-modernization__KAQyLYX", + "finished_at": "2026-10-12T17:58:05.460421Z", + "model_started": true, + "trajectory_sha256": "48b7b6cde70d08fd954d8b487644b1299940df3bad5e22c845c4d3d23bebe637", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/agent/bootstrap.json": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/agent/nanopycodeagent.txt": "3745a26f1bdef14e1af9c1d7730e9c4f2e6051b1fb2d5e35056ce04f4deddb7f", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/agent/trajectory.json": "48b7b6cde70d08fd954d8b487644b1299940df3bad5e22c845c4d3d23bebe637", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/config.json": "b7b680164c8a678722c0e41131a7e1cd169025e7d5878c2f750cc3e6bd8fe3c7", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/lock.json": "9c57b7a4c9700f295b7fa43ec8a4cfb6441e6a629764c791a01fc6c71269036c", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/result.json": "20d76529caa7437735adbaba52a8b407c09f6d43fa66e81753bf5a62629b6ecf", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/trial.log": "beb0f5f8fbe2dd0a81f3b4ca1f09027bdf30f041464b1ebe0978fca2e9fb4664", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/attempts/1/ctrf.json": "49216d024b629a4d7419d3ef5e1989f0c26faae3e88cc2ce566970798abf3ea1", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/attempts/1/test-stdout.txt": "03508abf76daba9fa98d833225e5a7b0210f8666ab4248d483c3ebdfb1dd7a5b", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/ctrf.json": "49216d024b629a4d7419d3ef5e1989f0c26faae3e88cc2ce566970798abf3ea1", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/retry-summary.json": "33728273c4ce2ea6162774d3be98b1398bb485210aae801b7665fc577b50c0bc", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/test-stdout.txt": "03508abf76daba9fa98d833225e5a7b0210f8666ab4248d483c3ebdfb1dd7a5b", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-cobol-modernization-main-20261012.json": "082daaad921852710e090ba443f9d7b2f27255073560731ca16eb4d7a6368776" + } + }, + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__BCqJJMc", + "finished_at": "2026-10-12T18:06:57.459317Z", + "model_started": true, + "trajectory_sha256": "4ce2e6a8cc141829c6edb23d03654f06094a5f6cf064e0ffb09e0d4d5a2ec628", + "raw_rewards": { + "reward": 0.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/agent/bootstrap.json": "2ffe3efbf9fd9f423c3fc82102152f59af7f9180f8aaf39c46e61c96e2f02d76", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/agent/nanopycodeagent.txt": "7f64e609bf4dc26e61fe1c78225a3a98f9c005d678e4e0347d1be04278af7361", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/agent/trajectory.json": "4ce2e6a8cc141829c6edb23d03654f06094a5f6cf064e0ffb09e0d4d5a2ec628", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/config.json": "c37568b90dd79cdae4684f8f5ffc26d5125d7ecba84f4e9c6f3c7c172aee80ef", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/lock.json": "30cb07d551b480bf970792c1d771435a960c449b193da337e90e251f36e2356a", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/result.json": "9095e885a1d6d55e0b97d945400d09e9d6d184533fd76b81d496709b0a4808e1", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/trial.log": "bd4e169248c258ffe65f47201bd25b17e5b1c58a32a39cf150d27876d968143b", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/attempts/1/ctrf.json": "aae2ae3154dc2098385c4150e58dc2126e994e4e636359c638dc1472277684b1", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/attempts/1/test-stdout.txt": "b75e0d6910c411fe1818df7c823faf2fd1fd580216ab5b4a0a8f188bbf0c832c", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/ctrf.json": "aae2ae3154dc2098385c4150e58dc2126e994e4e636359c638dc1472277684b1", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/retry-summary.json": "873a8c8e40f06767941d32e45bb30b67920e02fec3ac38c4802e6e6b8b9ba082", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/test-stdout.txt": "b75e0d6910c411fe1818df7c823faf2fd1fd580216ab5b4a0a8f188bbf0c832c", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-db-wal-recovery-main-20261012.json": "e6dd4b153683abe9cbd565a6c1d8834e7b33ab6c36e179050a73f06950095da1" + } + }, + { + "task": "terminal-bench/fix-code-vulnerability", + "trial": "fix-code-vulnerability__TA7Ddkz", + "finished_at": "2026-10-12T18:00:41.014749Z", + "model_started": true, + "trajectory_sha256": "ada18e72edfca14062ea84b1f355fb46aedd2275e65810085e0bdabc5a522aca", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/agent/bootstrap.json": "8d1fa3e359c1a2c87bb9b262c9113a423af3445aa5a20da8a2e6fb4d63b4f67a", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/agent/nanopycodeagent.txt": "5cb666badbe2878546cafa8292a0c430799497fff83fa7f204649efe1e03ccaf", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/agent/trajectory.json": "ada18e72edfca14062ea84b1f355fb46aedd2275e65810085e0bdabc5a522aca", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/config.json": "1d46c6dbd31d1cf98fcab766ccf7f3b9275de9559274067ccd1c13100fc08d44", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/lock.json": "2390fc81c4243de841f2da3655693225d4991c5430bcb4db705e13e157c6224e", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/result.json": "6b89818abef5a2933957fa53ee2448d9add908ba3db59deb3c6553925a193d73", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/trial.log": "1017376a9c4cb22597eabfef51dcae9fc8acc64615740314c9e6cd314f5d4c6f", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/attempts/1/ctrf.json": "dd54b24cdf1a175daeabb64d50d2939e5981e97ae5000acfbfd7e62440cc63db", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/attempts/1/original-repo-ctrf.json": "00fd792f5c5085f1e97b41c91fe2a0cf788d7ee486b235d7a6073b154bf01210", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/attempts/1/test-stdout.txt": "5b1c1b620157e4e7ee7e5368a1cd9f8de20235ae042eaeeb7f266332d7e38f69", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/ctrf.json": "dd54b24cdf1a175daeabb64d50d2939e5981e97ae5000acfbfd7e62440cc63db", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/original-repo-ctrf.json": "00fd792f5c5085f1e97b41c91fe2a0cf788d7ee486b235d7a6073b154bf01210", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/retry-summary.json": "138ba5bcae551a441ead9637a5fa97e1890e9c8ae3a91d3052264810827abc1f", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/test-stdout.txt": "5b1c1b620157e4e7ee7e5368a1cd9f8de20235ae042eaeeb7f266332d7e38f69", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-fix-code-vulnerability-main-20261012.json": "52ce343d3ef2e0689367bb69be86a7f9914958075973aebc7a03265293d80255" + } + }, + { + "task": "terminal-bench/git-multibranch", + "trial": "git-multibranch__ufZejR2", + "finished_at": "2026-10-12T18:15:07.672075Z", + "model_started": true, + "trajectory_sha256": "82dd6175afcc59e979ff205d236025190448be510d9b6a3ff2f0988b73b98e2b", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/agent/bootstrap.json": "391e000959d10d8aa551310ebe7bddd0c70185a684ee8cb543b7e8bf57e5e63b", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/agent/nanopycodeagent.txt": "7d1743e6b7b0148bce43be28469dfc0b690a0633abca6bd41575deb6b4e1c4ac", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/agent/trajectory.json": "82dd6175afcc59e979ff205d236025190448be510d9b6a3ff2f0988b73b98e2b", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/config.json": "68c8686c93341ea65f1b7970cba8f3fcdda84754cedc0626c9fd038176762223", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/lock.json": "dae7b5396f4d02c1179bae72fd4bbd6a21eb51f8223c276b11ca7ad417d80f23", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/result.json": "ab3bd8f9b1b8ed390b735e0da3d9785d31eecbbf9d827dc72e9cfc23842c63a8", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/trial.log": "c32df93f45c9d4e288213da31eb2dbc3d50fa0e5edcdce2d980efdccba7c4053", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/attempts/1/ctrf.json": "eaf803a08e37b365f2d76d6e0f3e4e293285ae12cf77ba23cebea95a0827352a", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/attempts/1/test-stdout.txt": "2077df3f0a7e86f4f2671448876055446c091ddf02633af1e836f859bcad3d54", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/ctrf.json": "eaf803a08e37b365f2d76d6e0f3e4e293285ae12cf77ba23cebea95a0827352a", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/retry-summary.json": "4aedb7ab49a65872244c8f924814a60c804f11a84df3538e417f5d38d3802daa", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/test-stdout.txt": "2077df3f0a7e86f4f2671448876055446c091ddf02633af1e836f859bcad3d54", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-git-multibranch-main-20261012.json": "567b20f11373f7b55814bc113edbeed4b5b36bc6845ac74765fc890078a3120e" + } + }, + { + "task": "terminal-bench/kv-store-grpc", + "trial": "kv-store-grpc__pUAoWuK", + "finished_at": "2026-10-12T18:16:33.481604Z", + "model_started": true, + "trajectory_sha256": "36c4aa60b39c84ae4bb40e55a0bda69c6b13f5646feed1b02bf4f7bedc5ea4f6", + "raw_rewards": { + "reward": 0.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/agent/nanopycodeagent.txt": "bbea0cd5cbc04b477def384ef4ad0513e7e57ce07f304b026c9e88454dc65f49", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/agent/trajectory.json": "36c4aa60b39c84ae4bb40e55a0bda69c6b13f5646feed1b02bf4f7bedc5ea4f6", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/config.json": "745a642b6e7bd1f7a5bb7804b63a63b381236ad826a61c1b1d28f2ad3b59168d", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/lock.json": "a60bc07889acc431c04519e08c025b29d803cb926f1fba65fd0531f9c702e8f6", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/result.json": "028a8bba4fc04baed2f6a48651156a716700a8db1ba0c795283f1203f7fe068b", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/trial.log": "711b6ee7f9af520f69344cdd25a345fecfcdf181eb847ba721ab4c25c59d6cbd", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/attempts/1/ctrf.json": "e8c4f69bd06ba8bebaa3ac13170eca3f157053fc925a4d7d0db106ca1301c72c", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/attempts/1/test-stdout.txt": "d4e4fd05b333e67dd9028c53b042729e5f8b675da6bc22642e57d39de32cc2bd", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/ctrf.json": "e8c4f69bd06ba8bebaa3ac13170eca3f157053fc925a4d7d0db106ca1301c72c", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/retry-summary.json": "3134eb5510f040621b00d652d0fb1369d15629d26b7b7aade4baed7ee067cf79", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/test-stdout.txt": "d4e4fd05b333e67dd9028c53b042729e5f8b675da6bc22642e57d39de32cc2bd", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-kv-store-grpc-main-20261012.json": "4b65710461d78c82012a98fb727cdf01855c79e663b9dd9391d34e98714302ab" + } + }, + { + "task": "terminal-bench/log-summary-date-ranges", + "trial": "log-summary-date-ranges__Zr4DKzE", + "finished_at": "2026-10-12T18:18:53.646064Z", + "model_started": true, + "trajectory_sha256": "b18a9831cf624845a297db3c440a18443f5e4944eb6365aca9665d2d8d969620", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/agent/nanopycodeagent.txt": "64995fc7594eb28558207b736991ac5d29afdb48fec8ed642428682af16d814c", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/agent/trajectory.json": "b18a9831cf624845a297db3c440a18443f5e4944eb6365aca9665d2d8d969620", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/config.json": "00c2e401ab832fe2729b89ea9a62a485f5ece2091d4b332c04d00b7211eeae95", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/lock.json": "0008ebe87e928565afff7097f67e7cd7abdaa364beb1ff8eae78289dee9419b2", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/result.json": "4414fd2333fa08d97f82ffe106d194f68ef82fe883cc6e9c20eb95c8d3319fa1", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/trial.log": "c3ef91c507c13f5e06c871e695b4d4784477470a02d9fff2cdc24faa0cb65456", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/attempts/1/ctrf.json": "ae8736d6f2d5602be2154040dbe5411ae10ea888d91aff63231c21e065f08b80", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/attempts/1/test-stdout.txt": "0f30be1e7733def79fcdb50f13fc2403ced950850aa903dca0feb5bfcfc78758", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/ctrf.json": "ae8736d6f2d5602be2154040dbe5411ae10ea888d91aff63231c21e065f08b80", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/retry-summary.json": "b276dff563df4eaf9b96f01251b862980a75de090168328d1b794288c77d0323", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/test-stdout.txt": "0f30be1e7733def79fcdb50f13fc2403ced950850aa903dca0feb5bfcfc78758", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-log-summary-date-ranges-main-20261012.json": "97ce365cccfc2f9144113f7fa32ccc244f211ee36a154e38120fce9b829a80d3" + } + }, + { + "task": "terminal-bench/merge-diff-arc-agi-task", + "trial": "merge-diff-arc-agi-task__mQorUxv", + "finished_at": "2026-10-12T18:20:42.546934Z", + "model_started": true, + "trajectory_sha256": "36ff4b17aa2a379f8e86dfc64a0edad55a45999b92cb3c0c97988a884adea26d", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/agent/nanopycodeagent.txt": "b79e4d0dc77fc4dc9f40c87d3c7f0bdc7ee4eee79fc3a69f5a5100585bd28e54", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/agent/trajectory.json": "36ff4b17aa2a379f8e86dfc64a0edad55a45999b92cb3c0c97988a884adea26d", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/config.json": "090ee8c327afde3f23bcc08946b330ddc84c0f932131931fbcb3ece7dc6e744e", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/lock.json": "a476408b75b7271c4c9ef10b0a3005aa117822109b1d73a024aacd4dd3182dde", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/result.json": "8fbfcf8e310a656ccc8d18a8d011dc109125242781707391665ad6c065f61210", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/trial.log": "d2d1e1bb4bc5b115eb9d5ab6c9935cb963343306845d3ab84b3a84d6735152d6", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/attempts/1/ctrf.json": "d170545d104233003d44ea14e6792f79abff8fa024949f2f9e59fa5cf09db7d8", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/attempts/1/test-stdout.txt": "b71f9b52f2d5da5ae25ba50c329f4395ae179c8289b1920b7a5d3b32bcb512cb", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/ctrf.json": "d170545d104233003d44ea14e6792f79abff8fa024949f2f9e59fa5cf09db7d8", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/retry-summary.json": "4ddb2ae32cf6fcd2c10d9321d1f308e9c76514a26e15f140dcde3baf1623a26e", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/test-stdout.txt": "b71f9b52f2d5da5ae25ba50c329f4395ae179c8289b1920b7a5d3b32bcb512cb", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-merge-diff-arc-agi-task-main-20261012.json": "12a3c803dcc64564160f8227c937a310402d81670bb18cfc66470db2ce2b3f2c" + } + }, + { + "task": "terminal-bench/model-extraction-relu-logits", + "trial": "model-extraction-relu-logits__Zs75P6A", + "finished_at": "2026-10-12T18:28:44.245369Z", + "model_started": true, + "trajectory_sha256": "d644b512caf5b376a5546516ff6f0b56087bb76ec70f1c24e3ccf5acc0ad04c0", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/agent/nanopycodeagent.txt": "535786e780ed0c430735f3f60441ee29b85fc726df33789437ea07d62f3e06bc", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/agent/trajectory.json": "d644b512caf5b376a5546516ff6f0b56087bb76ec70f1c24e3ccf5acc0ad04c0", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/config.json": "87e9e3c37614f893c9a51574232644c7266d9370cea8ece102fef37199419104", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/lock.json": "063546eec02227267acb4ab319e948835be0da47f9dec5ea2feddb7167e85f17", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/result.json": "c55fae295c679400e1e75bd130524d8ae491dc3f35a7bb8ad954eb4468f18d2f", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/trial.log": "808f854039d999594bc8f1d1d65bedc3a6bb0d897f241000b7b808f83133b60d", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/attempts/1/ctrf.json": "c14b5b428a4dd45ca00a64cd49c9971f9d55659e0e433b0f52e2269c34234ff0", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/attempts/1/test-stdout.txt": "8d1edede3f588912e482bb88ce8f3574a3259875d858c244ad121673d8c4a8b5", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/ctrf.json": "c14b5b428a4dd45ca00a64cd49c9971f9d55659e0e433b0f52e2269c34234ff0", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/retry-summary.json": "edd97b3d515d07c80bd230faaf82bc3050fbe9319ccf2d60f8e20a9397e0bd43", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/test-stdout.txt": "8d1edede3f588912e482bb88ce8f3574a3259875d858c244ad121673d8c4a8b5", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-model-extraction-relu-logits-main-20261012.json": "69d6de1582880fd15888e3229d3b0315ad10eb01060d32942465b28614dfcfe8" + } + }, + { + "task": "terminal-bench/multi-source-data-merger", + "trial": "multi-source-data-merger__48opj3y", + "finished_at": "2026-10-12T18:23:13.855648Z", + "model_started": true, + "trajectory_sha256": "09a328e0d43c9571d3c9ae885416e44ddc539abc15182514c56a144960fe79c6", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/agent/bootstrap.json": "43369761e1c62328eb0c3890c99d441d724fe366a178abf9c568928eb85c4174", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/agent/nanopycodeagent.txt": "70f3092a8a32066d53ca60c821bba3580895df542f0eced615c9e93648b535a9", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/agent/trajectory.json": "09a328e0d43c9571d3c9ae885416e44ddc539abc15182514c56a144960fe79c6", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/config.json": "3115b56ed2d03d36ab296aa7c04c3ca53e43c6ee32f37f1e3147b1785c328432", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/lock.json": "156ec2828c7ae8420765e3360b23b6af77edd6c8a43928f8add8e5bbecb814c5", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/result.json": "59855416f2bcb6f956b40da7cca7dc113249e751dd2ae592d1933a7dbb4f1901", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/trial.log": "8ba3d5657e2429f4cf2662bb4565d43933ca069719971d0dbbfab5e505c22043", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/attempts/1/ctrf.json": "0e387ff7fedcc921602c20e046b44d4a79593301621922babc081f82377de1e9", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/attempts/1/test-stdout.txt": "075473c99116b68a1571c1a79d062857214cde8bbcc627d2099ec18dd54b9277", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/ctrf.json": "0e387ff7fedcc921602c20e046b44d4a79593301621922babc081f82377de1e9", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/retry-summary.json": "23d190ac0d7e2db6c64d9995114c1480c8504ca36eed68e3d65cfa8d588b8f0f", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/test-stdout.txt": "075473c99116b68a1571c1a79d062857214cde8bbcc627d2099ec18dd54b9277", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-multi-source-data-merger-main-20261012.json": "4bcc848bef41b25ae7617356783db36afa760293d91119741e62671f2ca84300" + } + }, + { + "task": "terminal-bench/nginx-request-logging", + "trial": "nginx-request-logging__StiD5Bg", + "finished_at": "2026-10-12T18:31:26.640772Z", + "model_started": true, + "trajectory_sha256": "728a824202e401d7a6358fe45813cf29fd126f829f69e24dc883a60bd3b9ddc1", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/agent/bootstrap.json": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/agent/nanopycodeagent.txt": "c9c2c8ea72d2f945b2080e4a6ad351c549fd10c20d88e5f5fb9ea0994981b674", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/agent/trajectory.json": "728a824202e401d7a6358fe45813cf29fd126f829f69e24dc883a60bd3b9ddc1", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/config.json": "8b9bd5c7be2e1c4f5146cf6ff961b7b353db33d8c0fe79651a40363afa6e8baf", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/lock.json": "ae86cceca7c8d888cc43faecd2ba9065926b11d16fea9b1faa6cab2fb9cb6a26", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/result.json": "545c441ef8faa7985d860e6ee87a8daf17dff8297ccfd1e763a6ad3ba4bc775c", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/trial.log": "71e2f4ec349e53838f484b004cf7ccb95cad96bad5897c2adc683da11b45a3c0", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/attempts/1/ctrf.json": "b1b30bbfeec6cdfdf6aec2b813dd92686565b7d11e090b82183c6d5cfd807d69", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/attempts/1/test-stdout.txt": "0743dc95a0638adc17912b3411bd2756b08c1a0f051c3cf0b990e00c5a3f5d3d", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/ctrf.json": "b1b30bbfeec6cdfdf6aec2b813dd92686565b7d11e090b82183c6d5cfd807d69", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/retry-summary.json": "6ad2eb1373733e495f4482718c70c07f19ffb352d747a6bcacb21dd59267bc46", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/test-stdout.txt": "0743dc95a0638adc17912b3411bd2756b08c1a0f051c3cf0b990e00c5a3f5d3d", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-nginx-request-logging-main-20261012.json": "5bdba723ee0aa8efad7491596628c9592f7d1dac6d1287797dfcdd391a18b064" + } + }, + { + "task": "terminal-bench/openssl-selfsigned-cert", + "trial": "openssl-selfsigned-cert__2a3CFWZ", + "finished_at": "2026-10-12T18:31:30.463236Z", + "model_started": true, + "trajectory_sha256": "dcfd891cd9e786fd461a25e4673a0689254879e6ec6afb9d574fbf4165a413f7", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/agent/nanopycodeagent.txt": "8ccea43c8900f9276e8dd40e6a6ce7d4cc036b2f3db6809e1ae72c395bcd603d", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/agent/trajectory.json": "dcfd891cd9e786fd461a25e4673a0689254879e6ec6afb9d574fbf4165a413f7", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/config.json": "4bc80622f50f13ba296a2f3508c1d4596deb3c306e533fea82d5dea2312f6506", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/lock.json": "60de1ecd6f04787b05315e5389e79297b6e4657864dca0f72c14bde6928030de", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/result.json": "8db19e1f9bdf16caad4ebaaf19524964476012abdfef06da99a14c816ef5c326", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/trial.log": "266d42dc2fc2bab466c29c6f172bd95745fdbc49202febf807074d1e8d546848", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/attempts/1/ctrf.json": "1d044484b65856cedf2c9cd62550a7def0b90363019d70f28f3e7bb304881815", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/attempts/1/test-stdout.txt": "1c0bf2eb7a1fa2ede4d9e4efabba3ef035fea2860fecce6ada34c8b139fd742d", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/ctrf.json": "1d044484b65856cedf2c9cd62550a7def0b90363019d70f28f3e7bb304881815", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/retry-summary.json": "bd6b2183686459bf1095e089e4be89b6a3d7edf90ac8fa769f87651d23645ef7", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/test-stdout.txt": "1c0bf2eb7a1fa2ede4d9e4efabba3ef035fea2860fecce6ada34c8b139fd742d", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-openssl-selfsigned-cert-main-20261012.json": "39eba9830e9a42e383bc7038e17b3db6402f486d9f9962549ffd06155309013e" + } + }, + { + "task": "terminal-bench/pypi-server", + "trial": "pypi-server__QydrX5C", + "finished_at": "2026-10-12T18:35:18.098613Z", + "model_started": true, + "trajectory_sha256": "80fe56d87e976ecd8d24be3a65054faac5dcc993b8ba61a12f07bc0f9b93310d", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/agent/bootstrap.json": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/agent/nanopycodeagent.txt": "a160adc3ebf081133e6d5d49009288ff20f186faf64e913cfa3aafbfaf8aab1f", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/agent/trajectory.json": "80fe56d87e976ecd8d24be3a65054faac5dcc993b8ba61a12f07bc0f9b93310d", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/config.json": "4477284e6dbe47a42acaa92fb78ea8fb9ceb8d95a7f9304124de37679f271556", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/lock.json": "697f391a655e2115100fc610a9646238e5606170810df6adf9937a6da677d7c7", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/result.json": "d22265a60535d74347b1fdb8ac0815a724bb52284aa02d43335e2ad064b1a188", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/trial.log": "da01cbe8e306b89addebe513d3ab457cff2b0f4c0edd7d8e50e0e6f5d3908847", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/attempts/1/ctrf.json": "93a123248fdada26e5c11dd867531e4a5f197e22d9e6a1a6cb20d9e6730be49c", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/attempts/1/test-stdout.txt": "ab215b9035200df1ef2bcca4ac7776f2161e447f8a954964fefb8b75c1139e37", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/ctrf.json": "93a123248fdada26e5c11dd867531e4a5f197e22d9e6a1a6cb20d9e6730be49c", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/retry-summary.json": "362cb1bae7630f31680639ad5dd0511e869b88ab923771b43167d828302af740", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/test-stdout.txt": "ab215b9035200df1ef2bcca4ac7776f2161e447f8a954964fefb8b75c1139e37", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-pypi-server-main-20261012.json": "6d9887df3fcddad2699d7c653d058db8de8df4e7a577ebbdc0db3196304385f1" + } + }, + { + "task": "terminal-bench/pytorch-model-cli", + "trial": "pytorch-model-cli__nZN835D", + "finished_at": "2026-10-12T18:51:02.784736Z", + "model_started": true, + "trajectory_sha256": "bdbc760feaf957839b033c06a57a80bb9c9cad12c9072fcdc7c04fdee2181112", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/agent/nanopycodeagent.txt": "2af25d3fe6aa86e61b1b59f0595a2c4d8005e2587f5f3d1b6c95f1165d00d6c7", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/agent/trajectory.json": "bdbc760feaf957839b033c06a57a80bb9c9cad12c9072fcdc7c04fdee2181112", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/config.json": "729877c79b842de8d80a512c7d37e02215bd1b5fbcefaa2ba6782265dd8212ea", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/lock.json": "9d018ffccb76ede5631f8fa29c7773b5eb319860e20ee288d6c876e1fe6c3a95", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/result.json": "9ecc4496c5ff9d9cbad2f2615ccf677b822ab638a63dc06f40239c515e7238b0", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/trial.log": "96f9413830ca1ab6c119b07011b3cec6acbd806d6817f96cb15a2fa51e2ef1fc", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/attempts/1/ctrf.json": "cb46570283985a7442f532115e442ab4ef280b5150654b4b739864241878226b", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/attempts/1/test-stdout.txt": "930d997650bd512f3b7455f03cd3f5cad5c3e579aa79ed0f01c8284b8e85bc81", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/ctrf.json": "cb46570283985a7442f532115e442ab4ef280b5150654b4b739864241878226b", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/retry-summary.json": "7111245c79cfcc4744768a5e97f8d943f3fe265c3d15876960f1607235eb4a45", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/test-stdout.txt": "930d997650bd512f3b7455f03cd3f5cad5c3e579aa79ed0f01c8284b8e85bc81", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-pytorch-model-cli-main-20261012.json": "61152ab9f7476bacf2c6f234a28d221cacd0b601b6c55ced03bc3ef5a6555fcc" + } + }, + { + "task": "terminal-bench/pytorch-model-recovery", + "trial": "pytorch-model-recovery__mzRpfsa", + "finished_at": "2026-10-12T19:05:34.440873Z", + "model_started": true, + "trajectory_sha256": "08e76d44befdbc664bf2a33ff188bad70a55eb8b82318240d21c0a6046e27726", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/agent/nanopycodeagent.txt": "e96d3ab473863de3eab6bf419d73c8f07691206cc165a74eadae7aec685c9580", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/agent/trajectory.json": "08e76d44befdbc664bf2a33ff188bad70a55eb8b82318240d21c0a6046e27726", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/config.json": "84d0e286f009d4aa9c2355b78f067911f8f67aea304b4f9fc042298983e57f83", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/lock.json": "ecb167bd590e26b85bc809e25f46a927981043cc5f3bf599d3c5a8ff90cf1cf7", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/result.json": "617eaa0b069c586340607be0f0eadfdfb77d4fabf2caf3978ffb547782433b2b", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/trial.log": "4a10b340bbb454f3fc43a7f2edbafdb7dcd25aec1df9f9d9fdf192a62dda803d", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/attempts/1/ctrf.json": "2981557d0b92325e3b9a879b39fbac4e83288e8d8343b5966b754fc5abf2bff5", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/attempts/1/test-stdout.txt": "4a65e6aca402273570aeb36040bc05bd5ff93073297484b3e8e3df9c58325226", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/ctrf.json": "2981557d0b92325e3b9a879b39fbac4e83288e8d8343b5966b754fc5abf2bff5", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/retry-summary.json": "1041f21d8dca87f6e334bfb8a95e544b334af14ec96f014f1d900a7773a6893b", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/test-stdout.txt": "4a65e6aca402273570aeb36040bc05bd5ff93073297484b3e8e3df9c58325226", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-pytorch-model-recovery-main-20261012.json": "d31e93e3691372cb325e86c9285206b264c90923b409d0a6659090dbc85a8567" + } + }, + { + "task": "terminal-bench/qemu-alpine-ssh", + "trial": "qemu-alpine-ssh__qwWToXq", + "finished_at": "2026-10-12T19:01:53.815107Z", + "model_started": true, + "trajectory_sha256": "d3f41720f2d00a9aef92fb049090d5c0be6d8cbc7e8f691d8d953d157fad60fc", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/agent/bootstrap.json": "052b2e131d93ddb3117f1612f23e09e4a0216e77ac7924418a86072bbd3a5d8f", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/agent/nanopycodeagent.txt": "32feaf149243f5604dc806f4dd821602ea5e6d33f0ea0f47951dcb8a4b133e79", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/agent/trajectory.json": "d3f41720f2d00a9aef92fb049090d5c0be6d8cbc7e8f691d8d953d157fad60fc", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/config.json": "4cb108544c3ea2115ad7c3af320ffe76a3fbd498b8572de6504b375f7b9ae142", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/lock.json": "6548703885791b083fa6cb32356f3a9db22c11ffd443826ce756f3a62a856903", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/result.json": "e7d337f5782f55e411f4fa5769bb120951a7625067d1a9b25d1721fae38e4935", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/trial.log": "3f94c97bdfbc189c617f67ec14e485340d457d9e5e0f83183f5e51181670c4e1", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/attempts/1/ctrf.json": "90e5b8c7beb8e8acb7ea2b69ece3a99812a20b4f117457674f6d5de876d0a11f", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/attempts/1/test-stdout.txt": "9faf2c88114688da695210864b175b9473c08da154c950afd3801cff423f6e99", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/ctrf.json": "90e5b8c7beb8e8acb7ea2b69ece3a99812a20b4f117457674f6d5de876d0a11f", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/retry-summary.json": "9d1e0ee6cb28e82dfe6d725ea0f5f2989d1d39de44b75e36c2549040667e8598", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/test-stdout.txt": "9faf2c88114688da695210864b175b9473c08da154c950afd3801cff423f6e99", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-qemu-alpine-ssh-main-20261012.json": "67352224f483b57231d000b2bd713c989aeaf2a1861423a729713a5beeb9354b" + } + }, + { + "task": "terminal-bench/query-optimize", + "trial": "query-optimize__ZVLSEuZ", + "finished_at": "2026-10-12T19:15:34.331956Z", + "model_started": true, + "trajectory_sha256": "0647111c3e036de1dd547f9075a19f2a289ebd1bf05e0bb3547b91c8acd7a3dd", + "raw_rewards": { + "reward": 0.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/agent/bootstrap.json": "c72c3ff607d38944df0e65d64c3a0a941fdc52c3dc17db638b246035ca2d8f89", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/agent/nanopycodeagent.txt": "26b78decf1c9102f4560b2953af932fca143109a32b2084b98032fbc28f70b18", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/agent/trajectory.json": "0647111c3e036de1dd547f9075a19f2a289ebd1bf05e0bb3547b91c8acd7a3dd", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/config.json": "109488acefd980d505a3eec5ced9035c58339f86dda67f7d0ee98de8b31dbf4b", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/lock.json": "28437ccaefb7eac85da6f226b5a52190dcf146f550293671e0dc4ea8335b7de0", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/result.json": "aa24b5aec7fb7f2602badc45676e33562c25c7973aff74087d05fc7ca185430b", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/trial.log": "74f611cde7992be8bc8ea3143fdf8e8b4a2a49d38c939f9497cd5bc595c16a2a", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/attempts/1/ctrf.json": "3162d237542641fede1219d6b116c920e6c11b94e2ee78692da9f4e79762c898", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/attempts/1/test-stdout.txt": "8365b8e093bc31dbf3fbfea6f8a7a7f5af31d64acbd38cbf9c577b26714e5026", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/ctrf.json": "3162d237542641fede1219d6b116c920e6c11b94e2ee78692da9f4e79762c898", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/retry-summary.json": "d551a5c48acc834182358ef4b789e9e000b02434b5748b09d1f4126083416ad0", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/test-stdout.txt": "8365b8e093bc31dbf3fbfea6f8a7a7f5af31d64acbd38cbf9c577b26714e5026", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-query-optimize-main-20261012.json": "09096d21d7e7c79824ab4109fb98cd69e4eda4d0e73e1977b82b1ceaafb6ee9a" + } + }, + { + "task": "terminal-bench/regex-log", + "trial": "regex-log__NmCmvYZ", + "finished_at": "2026-10-12T19:08:23.294264Z", + "model_started": true, + "trajectory_sha256": "b802eb010e3f944eb30a5a1db8e200849f97ae279154d321af891703676b1051", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/agent/nanopycodeagent.txt": "f2c4b70dff23cb6f09bf5b8ff70e14975e50576ed9488b35ace8460064530165", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/agent/trajectory.json": "b802eb010e3f944eb30a5a1db8e200849f97ae279154d321af891703676b1051", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/config.json": "f3591fca8a2e019be35b7ef39e44f803aae92ef1d218f99cf8da32f728921e3b", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/lock.json": "a1a5d783c569c4a2059da23ed933d6756d7bdd5b265cbf5fbaa65b5742c2b32b", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/result.json": "b262dc287135c516817075cab6b29d496eba59960742948591fbb22e544e10f9", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/trial.log": "0f95f577404646395522c12e900c273f70fd0a24cf5d02ed64804ef904d3704a", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/attempts/1/ctrf.json": "1d34d0c19544a38b207c8e96ab90039e84235034e6592b38c0cf8b3164322887", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/attempts/1/test-stdout.txt": "3c7cd32794830ce4b9c28bb6303111b0f99c0808e60941882da66b594d5af41b", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/ctrf.json": "1d34d0c19544a38b207c8e96ab90039e84235034e6592b38c0cf8b3164322887", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/retry-summary.json": "ea2c4dfd756be7347645686b615c1ac6c4830fea53cca68dc21cfe149f6cfdb2", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/test-stdout.txt": "3c7cd32794830ce4b9c28bb6303111b0f99c0808e60941882da66b594d5af41b", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-regex-log-main-20261012.json": "bbc06770ff4bcd59a059d326768e0db131a20b86b2f7a5cf09b9e4e7d47fa8e5" + } + }, + { + "task": "terminal-bench/torch-pipeline-parallelism", + "trial": "torch-pipeline-parallelism__Zr8yYNi", + "finished_at": "2026-10-12T19:36:25.229299Z", + "model_started": true, + "trajectory_sha256": "dff3b58fb3e3f31e71bce99effb2626a16b2fed1b0118ce73715d05e807bb644", + "raw_rewards": { + "reward": 0.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/agent/nanopycodeagent.txt": "6e70f72c1456a8af7b8ec4887f5a91388d943128a568c20ad1b0f32e5e0b364b", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/agent/trajectory.json": "dff3b58fb3e3f31e71bce99effb2626a16b2fed1b0118ce73715d05e807bb644", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/config.json": "ceb0998220b253e2ed3e624080c107504b85be71d111a0a824078d213be90df2", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/lock.json": "1b7203f339bde65255ca6af2d683de182c4a2e291af1e71b2e7417a36fe5e260", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/result.json": "ff8aeeb5d7dc6c19b748beda88cb1a4d6c47f7c431feea71c45465ed669c23ca", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/trial.log": "31db282f7383cae7d38cdbcc1b79d8045477679d7419a095562bdfc0150068c6", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/attempts/1/ctrf.json": "4977536344650bead5c9dc81b4f380caa585dfe084c145522d2a40e1a62314e1", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/attempts/1/test-stdout.txt": "c15cca6d1add0baf3aaac6f740be973087d869c25c5be5e698a5485f23702586", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/ctrf.json": "4977536344650bead5c9dc81b4f380caa585dfe084c145522d2a40e1a62314e1", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/retry-summary.json": "8f5c71517bce0af45f888b049dc6383f4e68e3990559d0d14fc3426d36430fa2", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/test-stdout.txt": "c15cca6d1add0baf3aaac6f740be973087d869c25c5be5e698a5485f23702586", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-torch-pipeline-parallelism-main-20261012.json": "94b1602f578343904543c39f57315b8d7d3a050fd8806c3aa2792f802f6abd78" + } + }, + { + "task": "terminal-bench/torch-tensor-parallelism", + "trial": "torch-tensor-parallelism__zyNwrLx", + "finished_at": "2026-10-12T19:33:11.489468Z", + "model_started": true, + "trajectory_sha256": "cfdde35ed77aa9e59cf9131e0185505db3495308cd3d0027045643995c42d259", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/agent/nanopycodeagent.txt": "ce291b7ee2fb846df1e5abc42257ae418a53cf1be15e05b8653bf4d3874dd2a0", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/agent/trajectory.json": "cfdde35ed77aa9e59cf9131e0185505db3495308cd3d0027045643995c42d259", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/config.json": "2bf096d510610bfb63398b301739d81d7e3d1d245c05cb565a3860c969d1ccf2", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/lock.json": "72296efdd008528b3b1d1b09437057ba576b1815ea306fb372dfde05177884ae", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/result.json": "85f294f3b1c96dda7778fbdb84bd042db813b8b66c5f4b176da5129a363f6916", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/trial.log": "70a71e385d7d561e5b6080a93adbb68afc91fd1dec1c1ec84f2dc095b60849a2", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/attempts/1/ctrf.json": "dbbdb9d0fd35369550da55e5392b7b70eeeb61fc222599a8a23e57199a913692", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/attempts/1/test-stdout.txt": "524654fa63db03030bd0cb213aa6666103d7988b050931143dbd91e16d3f434b", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/ctrf.json": "dbbdb9d0fd35369550da55e5392b7b70eeeb61fc222599a8a23e57199a913692", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/retry-summary.json": "76b8ba1d1484ee873a409533c8eaf39b3a9d04c838a4e0fe4ed03714f54c2410", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/test-stdout.txt": "524654fa63db03030bd0cb213aa6666103d7988b050931143dbd91e16d3f434b", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-torch-tensor-parallelism-main-20261012.json": "4c0b84e05d5c996b49a8d7cd82c975dfa5b5a5d0a8436081022df099731b7c54" + } + }, + { + "task": "terminal-bench/write-compressor", + "trial": "write-compressor__znNddyQ", + "finished_at": "2026-10-12T19:44:29.613941Z", + "model_started": true, + "trajectory_sha256": "de6f22093d839eb98bb87a7244f073efbe896c55dccafe830aeade3e95faf8ca", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/agent/bootstrap.json": "78617809d0aabc5eaf4ecbb573bab09119e53b9c6a39a8890554717fa5823cf8", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/agent/nanopycodeagent.txt": "1dd2d09e967fb4741a52993a2e088a065d2064af0e375df35f69d27c7dc8d486", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/agent/trajectory.json": "de6f22093d839eb98bb87a7244f073efbe896c55dccafe830aeade3e95faf8ca", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/config.json": "086a233dbae1fa2a966da96e418c8eba433e550c9aa7358e713e02855e847667", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/lock.json": "c57bb4fe3586d5bb089a19247d76741e235bf333cea08473aa96fc49848e0773", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/result.json": "6d7ae2b32f15abac90d962503ee3717ce4505c81f7ba08c1767b799257ed616c", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/trial.log": "a522af380f4c4a0ccc2d04ea0249b4ad6c27c8e387b1f5abaafe3ddddd2e9835", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/attempts/1/ctrf.json": "247608b6e85713f032d5e9f860303ce921698c8d75df2e3290d910168bd5c6af", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/attempts/1/test-stdout.txt": "3cb1542a661a9997d1837f73d257e7b0b47031883cb3faff8378eb7470f640a2", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/ctrf.json": "247608b6e85713f032d5e9f860303ce921698c8d75df2e3290d910168bd5c6af", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/retry-summary.json": "806c9fdb27880b4388b13d4a1875bea5dff3d2ae636c49c3ce5931121ef5f2b2", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/test-stdout.txt": "3cb1542a661a9997d1837f73d257e7b0b47031883cb3faff8378eb7470f640a2", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-write-compressor-main-20261012.json": "e9f51178f35fdcc16bd146a1a59498bcca15f994a363cb6b708523729689b7ed" + } + }, + { + "task": "terminal-bench/constraints-scheduling", + "trial": "constraints-scheduling__LyFmzLj", + "finished_at": "2026-10-12T19:38:36.004003Z", + "model_started": true, + "trajectory_sha256": "82dee44641b632a60a502ad8453b0997b7c91033f0aad87a50188578c7428f5e", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/agent/bootstrap.json": "8ec0c76eef3467924ce96d81cc262bdea7775b259bbeeca2a0563a292696eb05", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/agent/nanopycodeagent.txt": "da2ed1376689260fc56fbe8a0e39d5b7fb42e7cde432cb64183fec7a608105b1", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/agent/trajectory.json": "82dee44641b632a60a502ad8453b0997b7c91033f0aad87a50188578c7428f5e", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/config.json": "975a6e129536caf465f51a6ceebf36cefd35a8bdd0da265accb04923dd890533", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/lock.json": "276fd9bb0c189ad1bc828b0e1fe1fda5f578acb4a789f91043f64b4404a7d1f8", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/result.json": "674a5034a06448db2ee18c6046a85f860d0d15db50e55807e8b37cbb8b8d238b", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/trial.log": "eb67fa42a4fc408bf98121550743bbc3ee9c7611e4f699f1faed3bedc2411cd4", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/attempts/1/ctrf.json": "d6e10646c6d7bf0221c9c5341212c805f0a8c947665493fdca2e57f10bc83c97", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/attempts/1/test-stdout.txt": "c3bb6c75a50ce8e859989095cf3b5068a66fe5ded8e42bcdc92a74c3af270f70", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/ctrf.json": "d6e10646c6d7bf0221c9c5341212c805f0a8c947665493fdca2e57f10bc83c97", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/retry-summary.json": "51e5d4a3e03197b0410889d693406e9038bb176b0b3034f800c893170ff0b46f", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/test-stdout.txt": "c3bb6c75a50ce8e859989095cf3b5068a66fe5ded8e42bcdc92a74c3af270f70", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-constraints-scheduling-main-20261012.json": "7ee2b4b27af74185d2db9ee72026a97e8428f43b9a814ccc98f803fa4f097551" + } + }, + { + "task": "terminal-bench/custom-memory-heap-crash", + "trial": "custom-memory-heap-crash__fjqXaKf", + "finished_at": "2026-10-12T19:50:09.328682Z", + "model_started": true, + "trajectory_sha256": "eb311202739c38a935a3d67621c75ab32a4c9287b76cb23e2a337b308f58555f", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/agent/bootstrap.json": "cf6bb84c42557ba2a7b7e0ec653bbc1c7cf86c236d1b7e2a6852918dfb8eb7dc", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/agent/nanopycodeagent.txt": "1d691418241803bf49b351d5c515aa3f0caaede64af83938188febfe38e8a22f", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/agent/trajectory.json": "eb311202739c38a935a3d67621c75ab32a4c9287b76cb23e2a337b308f58555f", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/config.json": "956182b291e7064a2d10f5a643e393fd4e7ae07c644f964620e8bce28199baf4", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/lock.json": "b6724e58d36c9a4e15ab496c32ab06aad489da0ea54ad134decb2aa7cecbc41c", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/result.json": "2fd0df6ed0f13010b75c18c84b6fc35db117625d83b12ae4bc0a40b7869931dc", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/trial.log": "1aa3041d5b08d80a9c09299e6f4840f29fcc4d84f81223564e604215ae33854a", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/attempts/1/ctrf.json": "cacd14b02641488fd8c81cc7ce5c694803fc5451bd5193ccbbff0a0101dfd624", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/attempts/1/test-stdout.txt": "f06064c7a95100c8b4025ef8ae7fbb2a9fc627a0fbec0205a2b371ce683b7268", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/ctrf.json": "cacd14b02641488fd8c81cc7ce5c694803fc5451bd5193ccbbff0a0101dfd624", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/retry-summary.json": "9c383a7aedf097530f6995507834e8abb7394e76015c26d81c940909c4681d52", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/test-stdout.txt": "f06064c7a95100c8b4025ef8ae7fbb2a9fc627a0fbec0205a2b371ce683b7268", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-custom-memory-heap-crash-main-20261012.json": "307270949dadfcca82f611616f2b004483973466f0c46ab50ebfbfbc55f9417f" + } + }, + { + "task": "terminal-bench/dna-assembly", + "trial": "dna-assembly__mRFMZDo", + "finished_at": "2026-10-12T20:05:21.423836Z", + "model_started": true, + "trajectory_sha256": "3cfa4a57f7e84c83f9ccafc13bd27c0f3ccbd3e2b01d9ad10cd955fe9a2167df", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/agent/nanopycodeagent.txt": "7e1c56f8a4e2c99ddb1e88f8828d138ecc8f640a186d470e6e66fd97e3bf3a45", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/agent/trajectory.json": "3cfa4a57f7e84c83f9ccafc13bd27c0f3ccbd3e2b01d9ad10cd955fe9a2167df", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/config.json": "720c3c1cce90ea23dae90a63b676ef0cf15a55613bc27669b54e13ed622c6ec8", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/lock.json": "c1f3d8b5cde40654903c420408d4ba50384166ee21060d2ca169f444ad10e267", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/result.json": "44823a4c38d60cd831138c4ace5847673222a215ee3da864eb78170b89d0417e", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/trial.log": "a44b7a28ba28f5b446894c33863d4d4f3fd90e736e2b43cbd87861ca42aca045", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/attempts/1/ctrf.json": "f461e603ae7a442d7b97b937675bd42d92adf829695d7517a87cebf3e3d3cedd", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/attempts/1/test-stdout.txt": "724868261581dfd8b68d54e5d651d1c8b31afe4d85de06883d0cb40bb3e1e9a8", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/ctrf.json": "f461e603ae7a442d7b97b937675bd42d92adf829695d7517a87cebf3e3d3cedd", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/retry-summary.json": "a39ca3d19b542a2869ad624f7270aff3f7798e7255e1dc114cd4a04d2dc534aa", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/test-stdout.txt": "724868261581dfd8b68d54e5d651d1c8b31afe4d85de06883d0cb40bb3e1e9a8", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-dna-assembly-main-20261012.json": "7ec090f2433715b89ba9c581a89937d1dc89702e7ae12946c17fbb2d5155d011" + } + }, + { + "task": "terminal-bench/llm-inference-batching-scheduler", + "trial": "llm-inference-batching-scheduler__AvvLYtM", + "finished_at": "2026-10-12T20:16:53.143708Z", + "model_started": true, + "trajectory_sha256": "fb774af094770aa2b2a105029319accb9e7850f88e2c85722c6de6ff5f45c246", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/agent/nanopycodeagent.txt": "13696564ad5b563555594cdddddc7ba42c928901f45952dcd5d06c9199767c36", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/agent/trajectory.json": "fb774af094770aa2b2a105029319accb9e7850f88e2c85722c6de6ff5f45c246", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/config.json": "6de7527f665c887c8bc896eddeb556c7998ff7d98a8f70a37bc006dfe2459596", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/lock.json": "78a7340661de767356dc1b6bc12c2ca22fa28efb0a848b7349d1968c285acde3", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/result.json": "a35d23a179153f9ddc227f19310cfca1e6c603671bff397da2266ebdd76d9b4f", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/trial.log": "f8995b4e61627e44b457074daadd282b14b02ac820268b149a2df0b92a01f712", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/attempts/1/ctrf.json": "67c9af94662e1a5e1f3a4e7055b0f7804bc7258ac820ee7fbe946d90e6207bf5", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/attempts/1/test-stdout.txt": "6a580044057c6448fbedead4eb0e589a6ba9bb45ba53b7e53521b43ccfebd4ab", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/ctrf.json": "67c9af94662e1a5e1f3a4e7055b0f7804bc7258ac820ee7fbe946d90e6207bf5", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/retry-summary.json": "be3106a6486c29876fdb7f5914e23d3b4ba4b3e0e5866a1532b33ef208267c45", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/test-stdout.txt": "6a580044057c6448fbedead4eb0e589a6ba9bb45ba53b7e53521b43ccfebd4ab", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-llm-inference-batching-scheduler-main-20261012.json": "bb45a2ddb16fa5142bc3599915decd1f523d24f340c3a5c28fb5d7b9cc3c25f3" + } + }, + { + "task": "terminal-bench/path-tracing", + "trial": "path-tracing__LMykXqk", + "finished_at": "2026-10-12T20:11:19.706262Z", + "model_started": true, + "trajectory_sha256": "9a7054d0e81f0ee38e3d5e4c57d31a16ac2737a1a98f6f5c7b19a8b706ac01a4", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/agent/bootstrap.json": "f7c768324bf6e26e7f52bdec891623e4fc98e29f1f24ca02e3847723d8cb55bb", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/agent/nanopycodeagent.txt": "7dba04bb9efd7a6aa02e26e46024cf2b9286e614264061242951404f579bd969", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/agent/trajectory.json": "9a7054d0e81f0ee38e3d5e4c57d31a16ac2737a1a98f6f5c7b19a8b706ac01a4", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/config.json": "430c7f3928db3f3e16a19087a2e972c8f02e9d526cb0e448d605eccbc078a63e", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/lock.json": "e6caebcbb23e43b16173b7e3a09c926529c2088cbbc4c73e5117c661db0727c6", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/result.json": "8dbd3fa2cc44219b21c218cbd458fd6f5e0096343f2fe29e9a3fdd24d15b16f7", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/trial.log": "1470af8c2a19507dc9913b16de8083222aeaa7554c007743a3df95a7507290de", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/attempts/1/ctrf.json": "121ac3d70e6b4aca6d73d6b9b10fbdbcabe5f0aaeb4a012dfcd9dc8f83918a5d", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/attempts/1/test-stdout.txt": "2bc5a286c9aeccf54c38a581676c5212e12fc57ec97a63ba9f061c43a5a5e469", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/ctrf.json": "121ac3d70e6b4aca6d73d6b9b10fbdbcabe5f0aaeb4a012dfcd9dc8f83918a5d", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/retry-summary.json": "46126206f4f9d1ce74fd9988fb8d65700d55498cc73f132087d9d1e5b1a8fe47", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/test-stdout.txt": "2bc5a286c9aeccf54c38a581676c5212e12fc57ec97a63ba9f061c43a5a5e469", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-path-tracing-main-20261012.json": "beeb9a5f2fdfbd5dcf2388ba140e6f43e4a230acd75d08406e01c3e8f8f2df5a" + } + }, + { + "task": "terminal-bench/schemelike-metacircular-eval", + "trial": "schemelike-metacircular-eval__wpWP6A5", + "finished_at": "2026-10-12T20:48:39.961600Z", + "model_started": true, + "trajectory_sha256": "d944b83ab3e05e799c333b2b091949e99393c33be21c7106981dccaf64fd061e", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/agent/nanopycodeagent.txt": "ab95bbea507c1a35d48d97c479130608e5f6274bd16ba47278285087b6430e7c", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/agent/trajectory.json": "d944b83ab3e05e799c333b2b091949e99393c33be21c7106981dccaf64fd061e", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/config.json": "2e5dc6251620f258a13d21f946e1c77ad2a8107caa02fd331fde78355b332b14", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/lock.json": "1688ca2fc9bf8552ea287566f840a5bed16f1f0191f0f7c0be2c70b9ef9e0391", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/result.json": "9a4a2b210149b9930c62f3b713e900fe4c42e35fb765cc487831a7625667521e", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/trial.log": "c30bbeaeb87b548b737fa1287407ca8323e3f31b6c8cb8ac3750ac4bbb88f92c", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/attempts/1/ctrf.json": "e8d09bba9fca81d7bb88aa3e2bc201aa3d449cbc5fadda69d6cf318b2999848f", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/attempts/1/test-stdout.txt": "c8baf368ddf26533f5e2b003c41108730834d14daa3250fdb239657d4fadb5d2", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/ctrf.json": "e8d09bba9fca81d7bb88aa3e2bc201aa3d449cbc5fadda69d6cf318b2999848f", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/retry-summary.json": "3da4843558dc344c5aca75afadfb6c85a2753ee5bcd2e96c78d2690845ea5a5b", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/test-stdout.txt": "c8baf368ddf26533f5e2b003c41108730834d14daa3250fdb239657d4fadb5d2", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-schemelike-metacircular-eval-main-20261012.json": "5e800b0a0e8315bee49df78a5c0328f471421643a34873b5567f500929d7d90a" + } + }, + { + "task": "terminal-bench/caffe-cifar-10", + "trial": "caffe-cifar-10__Pw7u7Uu", + "finished_at": "2026-10-12T20:59:42.993886Z", + "model_started": true, + "trajectory_sha256": "6af6d661da6557cdaf2c53a82988069529a856346483272ad00520e110595963", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/agent/bootstrap.json": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/agent/nanopycodeagent.txt": "2d2352f7d0eea8d87c6301cc1acb06b4310c7c0ad6c105a4aa2dcf3b3768409f", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/agent/trajectory.json": "6af6d661da6557cdaf2c53a82988069529a856346483272ad00520e110595963", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/config.json": "6f9108f57ef992e13e6463bd4e678b7be7328110ea5b6f7ea3de749d7ddd250a", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/lock.json": "b1d31159d29ed8aa08d521b32285e0ede410a112c947587e1941a3c5815c9766", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/result.json": "d8107ddba84973ef8516febb5dc36af05186fcb5659acc33fc3f6a73081849cf", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/trial.log": "563494c998c997281b24399d137cd376dd0b04aa71fc99d91a0fdeda7f71759f", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/attempts/1/ctrf.json": "c4dcede6e70e7d407abe0e2b1f4eb2e800a7c73529f0b77b39d9d5bf6d67e0fb", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/attempts/1/test-stdout.txt": "48545966dc37a4a974d48c4c54b3009a8d8c5f37db9f96f6395b4318b7079d13", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/ctrf.json": "c4dcede6e70e7d407abe0e2b1f4eb2e800a7c73529f0b77b39d9d5bf6d67e0fb", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/retry-summary.json": "c48a6ecf2d2350e11f402ec92f08e6f45526e323748877771012ce0a53fb9350", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/test-stdout.txt": "48545966dc37a4a974d48c4c54b3009a8d8c5f37db9f96f6395b4318b7079d13", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-caffe-cifar-10-main-20261012.json": "4ec390e69d840cd7113a639dbd8c53b7b19d34ae5bf2198c5eebb1e02eec95c4" + } + }, + { + "task": "terminal-bench/circuit-fibsqrt", + "trial": "circuit-fibsqrt__JQ9zU3H", + "finished_at": "2026-10-12T21:11:07.895519Z", + "model_started": true, + "trajectory_sha256": "8387b2396f17131af58429660cb9a4e2f84e63d64915357e0fcdc67aa8bea377", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/agent/bootstrap.json": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/agent/nanopycodeagent.txt": "f4fa6636f9afb1442daadafd3c9f35de11b3c5f4fc3cf122bfaad9e786455930", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/agent/trajectory.json": "8387b2396f17131af58429660cb9a4e2f84e63d64915357e0fcdc67aa8bea377", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/config.json": "36c1e704b91f06543183d163096a88a5968ab11c56adc3f1f1a580f3788f3ad3", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/lock.json": "0bb71a5952d924995f5ffd262b1ef0d576d8b46de6270172382f9668ca6c7b0f", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/result.json": "dc9e6a340ee75ad1618cd257a407380bfcd9b8932aa9c0f71de3115d2df22e10", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/trial.log": "08c338576a42fed76b69572c198eddb1fce0db892fb95256937d0ced2f6c89a5", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/attempts/1/ctrf.json": "1e24704cd3410f4aab1eb463cb7368ed956e1475b7f241130164a958c70b1662", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/attempts/1/test-stdout.txt": "056af043e94d6bd337c2a488ab7f8783d0a180d8f2eed3628a4c273596770645", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/ctrf.json": "1e24704cd3410f4aab1eb463cb7368ed956e1475b7f241130164a958c70b1662", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/retry-summary.json": "e2b3b469bc1150219db9cd2074737653da6318117f1602f8f99d4b6f3d52125b", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/test-stdout.txt": "056af043e94d6bd337c2a488ab7f8783d0a180d8f2eed3628a4c273596770645", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-circuit-fibsqrt-main-20261012.json": "4a6e1e3752e88e7f38dfc9f59e51f95c96625a8fde59daf3b129db7520aa2a6a" + } + }, + { + "task": "terminal-bench/mteb-leaderboard", + "trial": "mteb-leaderboard__VD8EX2H", + "finished_at": "2026-10-12T21:24:06.475871Z", + "model_started": true, + "trajectory_sha256": "7e9ac54b565926862d872da405ae7fb7ae91323718990d8081b721a0ef52f775", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/agent/bootstrap.json": "7c324cb3af6adf375e27d5cefcc67e8631dae65ac0f3c52687f893fa69b96c4e", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/agent/nanopycodeagent.txt": "e0676a8e5570b5c0fb2bd989db783eac7b37f9a79e1eb21f8dcbcf6f46158dae", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/agent/trajectory.json": "7e9ac54b565926862d872da405ae7fb7ae91323718990d8081b721a0ef52f775", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/config.json": "d4281b3548cfeea355d2863a5dd937c121784a9770c05f0c4ad67f12c6e9dd46", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/lock.json": "5889a360552aa157819edf18c47d3dcbfb72506ea0e76da1982ad05632decfd7", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/result.json": "a56d61d05610b7314f1d51546e95ae9c386d718540321f8f6192171f33417f08", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/trial.log": "159b2ca5186d5862ae323f104eff755bb2018386c5f6d5f0fc2af49695915e8b", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/attempts/1/ctrf.json": "ff1b665a1eb3af71d74fc1ac2a70d3134a23073b48163549c8a7666080c1112a", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/attempts/1/test-stdout.txt": "684d4a2cee0edf5e75baad28bf20e67bb2fcf45a1fac503f1f1faaa21c770a00", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/ctrf.json": "ff1b665a1eb3af71d74fc1ac2a70d3134a23073b48163549c8a7666080c1112a", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/retry-summary.json": "086d38a33540dcf16467a587d8e956ecd02c3ac6d9eb5cc4b9bd903c044e2412", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/test-stdout.txt": "684d4a2cee0edf5e75baad28bf20e67bb2fcf45a1fac503f1f1faaa21c770a00", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-mteb-leaderboard-main-20261012.json": "78b955d930adde2127dd6cb63c6f154a30c03ecea1de8efa5cb0b9148db49b3d" + } + }, + { + "task": "terminal-bench/regex-chess", + "trial": "regex-chess__g85Ci69", + "finished_at": "2026-10-12T22:12:41.800871Z", + "model_started": true, + "trajectory_sha256": "05865541b5a961921d5ddfce1fca76aa52d3774ebf154f03b1c20f3bedc42c02", + "raw_rewards": { + "reward": 0.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/agent/nanopycodeagent.txt": "de5df9339de05415c456003db9f7e1a64d1aa1bb7ec26d88f7c78d7147e03714", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/agent/trajectory.json": "05865541b5a961921d5ddfce1fca76aa52d3774ebf154f03b1c20f3bedc42c02", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/config.json": "9952061ab6b5aab855cb20621194760be8e96c8565e31f90a5d8bb6664fb31bc", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/lock.json": "43ff96727a335229fc8f7cc32e372cadd4d201b540cb68e6ae5dd68a33ccbc80", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/result.json": "64dd8b01b2deef4b9dcd70b351a15515e3d940458b1246dac9c9d5729765e0bd", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/trial.log": "08cd3c2af5895f282d5cee4a7f3abfa1082dec87f5b7b4a04226f6d828bf1111", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/attempts/1/ctrf.json": "3fae26a1ae3b0a38e5dee9d71ba447e02538a4d806751dfa531e7472ba00bc42", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/attempts/1/test-stdout.txt": "5791480e7b0bb6088d3c03e25946254593b37cf0820d26fa1a1a13d54f87a275", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/ctrf.json": "3fae26a1ae3b0a38e5dee9d71ba447e02538a4d806751dfa531e7472ba00bc42", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/retry-summary.json": "d13938c17cb1fdc2b5c47580deb0a5f660a6e7731826b62092ab3cb25e0aa6e2", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/test-stdout.txt": "5791480e7b0bb6088d3c03e25946254593b37cf0820d26fa1a1a13d54f87a275", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-regex-chess-main-20261012.json": "75f77b46dd71a21942625d0aef383801fa2b49e81e71b4438c551e05059fa2c7" + } + }, + { + "task": "terminal-bench/build-pov-ray", + "trial": "build-pov-ray__SyyjSUB", + "finished_at": "2026-10-12T21:31:18.553016Z", + "model_started": true, + "trajectory_sha256": "9d6250cc4dfe75686648ee8e3abe158db9b2e19d43f951d1dc0200e346d20e94", + "raw_rewards": { + "reward": 1.0 + }, + "exception_type": null, + "files": { + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/agent/bootstrap.json": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/agent/nanopycodeagent.txt": "4ba28a554b57420903cc853fc8fa514598cebf504b7e374e589beb4476a0f74d", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/agent/trajectory.json": "9d6250cc4dfe75686648ee8e3abe158db9b2e19d43f951d1dc0200e346d20e94", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/config.json": "e059326fbbcc3bae7f3c1ddd099932269faa2cc23e241325926166bf9231822c", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/lock.json": "93bbbe4728d15e476862227556251297f6cedbfb3d488ea3a646e473c5691e7e", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/result.json": "482e91d41b666e3fa5305f2b6462ea4f30fd9c18447939ff2ab9de795e12c4e1", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/trial.log": "79218ae6a2ebe895e3db9447de508d896073c8ff5722b5fc1af964ae9b9ee525", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/attempts/1/ctrf.json": "f764c97cf5bb402548ec90b91837fad796d71f53a1404483506065a83570e66e", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/attempts/1/test-stdout.txt": "a98b4aea581b17d7ed32bbe6553563adc28e7c5693990631972d1a3e2206449a", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/ctrf.json": "f764c97cf5bb402548ec90b91837fad796d71f53a1404483506065a83570e66e", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/retry-summary.json": "a9c65509b2740af979f122272f6806061a6da5b4cbbc705e015d9fb056e94e9b", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/test-stdout.txt": "a98b4aea581b17d7ed32bbe6553563adc28e7c5693990631972d1a3e2206449a", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-build-pov-ray-main-20261012.json": "87d83e078c1e3e24b5c5272ce5fa625bfa427d8cf7a5b09af90ed8f6fde6474f" + } + } + ] + }, + "integrity": { + "registration_unchanged": true, + "plan_source_prompt_configs_tasks_unchanged": true, + "verified_preflight_tasks": 34, + "source_snapshots_verified": 34, + "wheel_digests_verified": 34, + "frozen_native_trial_artifacts_verified": 34, + "queue_pair_barriers_verified": true, + "whole_model_trial_retries": 0, + "outcome_based_replacements": 0 + }, + "aggregate": { + "trials": 34, + "models_started": 34, + "raw_passes": 28, + "raw_zeros": 6, + "classifications": { + "scored_failure": 6, + "scored_pass": 28 + }, + "verifier_timeout_retries": 0, + "estimated_known_cost_usd": 2.522994588, + "cost_complete": false, + "usage_incomplete_tasks": [ + "terminal-bench/adaptive-rejection-sampler", + "terminal-bench/torch-pipeline-parallelism" + ], + "recorded_tokens": { + "prompt_tokens": 60464780, + "completion_tokens": 1603196, + "cached_tokens": 59660798 + }, + "summed_stage_seconds": { + "environment_setup": 74.003943, + "agent_setup": 1977.897826, + "agent_execution": 19282.892401, + "verifier": 3000.789951 + }, + "model_job_wall_seconds": 16995.171695, + "raw_pass_rate": 0.8235294117647058, + "official_pytest_counts": { + "collected": 155, + "passed": 137, + "failed": 18, + "skipped": 0, + "error": 0 + } + }, + "trials": [ + { + "order": 1, + "task": "terminal-bench/adaptive-rejection-sampler", + "task_ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "trial": "adaptive-rejection-sampler__zWw9zAU", + "job": "dev34-adaptive-rejection-sampler-main-20261012", + "raw_reward": 0.0, + "classification": "scored_failure", + "exception_type": null, + "official_test_summary": { + "tests": 9, + "passed": 0, + "failed": 9, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791827196.8244386, + "stop": 1791827197.5578887 + }, + "started_at": "2026-10-12T17:29:27.080151Z", + "finished_at": "2026-10-12T17:46:58.286796Z", + "seconds": { + "environment_setup": 2.108817, + "agent_setup": 250.590434, + "agent_execution": 720.288542, + "verifier": 57.321946 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 57.1292674590004, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "time_budget_exhausted", + "recorded_agent_steps": 10, + "tool_calls": 8, + "incomplete_model_steps": [ + 17, + 19, + 21 + ], + "accounting": { + "usage_complete": false, + "cost_complete": false, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 21819, + "completion_tokens": 1516, + "cached_tokens": 20608 + }, + "estimated_known_cost_usd": 0.002306148 + }, + "task_checksum": "a4726253ad23c3fc46b5cb9c3d21025874e367b18643a3920f636a1fa89e1207", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/agent/nanopycodeagent.txt": "a010f4fde40e491dfeaa0b565f6471c34388a4645c8169362f6ab19dae341054", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/agent/trajectory.json": "d6b054f53f68a111f2d1b42ba3774eb2070acd798d02a441d86417b36f532069", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/config.json": "6cc16382000a695512105efad71dc43285d6e298f2f14de50da49304f1dd52df", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/lock.json": "36641f7cbd1f7091eb2158b4ebd53bb1505955b7a250146b6209b1ab11d47a6d", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/result.json": "0442d30c596bc8ac89b91c4497d0653820a1e7209ecbca32c9c4f9cdc089788e", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/trial.log": "249f8bb7b1cc2b18adcad2ff83e9eee8b0e7a9a01949abeb5ca5714dab78182c", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/attempts/1/ctrf.json": "6b946a99b4473bb3ab0692440bfb35c38063c91701faa62fd7ce0bc6713c1504", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/attempts/1/test-stdout.txt": "e27a2fb2f987a71c07f8ff392065d03786823880ac9a44a4a3c131c4dbd3b0e1", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/ctrf.json": "6b946a99b4473bb3ab0692440bfb35c38063c91701faa62fd7ce0bc6713c1504", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/retry-summary.json": "07fb6406bef4efcfb135bd2e1c04ebbc64344c676d29b2afffd83d75bee8ff83", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012/adaptive-rejection-sampler__zWw9zAU/verifier/test-stdout.txt": "e27a2fb2f987a71c07f8ff392065d03786823880ac9a44a4a3c131c4dbd3b0e1", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-adaptive-rejection-sampler-main-20261012.json": "b1efcbfa4cff1b14211e959cc1fe565d13e99142e612df0b442eebc60e6b9a83", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012-input/manifest.json": "5f8a39868185f3e0dcc96edb449c564f77ef0637b5b6b55a909b808638c856f1", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012-input/config.json": "7a51e445e67d644c59b429b39adf6d4c3eb195c541fdde94baf215192c761012", + "jobs/dev34-baseline-main-source/jobs/dev34-adaptive-rejection-sampler-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 9, + "passed": 0, + "failed": 9, + "skipped": 0, + "error": 0, + "summary_line": "9 failed in 0.73s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + }, + "failure_analysis": { + "cause": "No ars.R deliverable before the work deadline; R installation consumed repeated tool timeouts, then two provider stream interruptions and a deadline exhausted the remaining budget.", + "official_evidence": "All nine tests executed and failed on the missing /app/ars.R output.", + "agent_evidence": "Installation calls at steps 5, 9, and 13 timed out. R was available at step 15. Steps 17 and 19 ended in RemoteProtocolError; step 21 ended in DeadlineExceeded. No implementation write completed.", + "atif_steps": [ + 5, + 9, + 13, + 15, + 17, + 19, + 21 + ], + "improvement_hypothesis": "Bound dependency preparation, support reliable background jobs, checkpoint a minimal deliverable early, and account for stream interruption recovery within the deadline.", + "limitation": "This trial does not demonstrate a faulty sampling algorithm; none was submitted." + } + }, + { + "order": 2, + "task": "terminal-bench/build-cython-ext", + "task_ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12", + "trial": "build-cython-ext__vxC7HQK", + "job": "dev34-build-cython-ext-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 11, + "passed": 11, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791826909.6851988, + "stop": 1791826918.5973687 + }, + "started_at": "2026-10-12T17:29:26.629176Z", + "finished_at": "2026-10-12T17:42:19.799897Z", + "seconds": { + "environment_setup": 2.084114, + "agent_setup": 145.234043, + "agent_execution": 586.533592, + "verifier": 18.034162 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 17.832269572005316, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 45, + "tool_calls": 54, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 1041764, + "completion_tokens": 18119, + "cached_tokens": 1014272 + }, + "estimated_known_cost_usd": 0.036076032 + }, + "task_checksum": "b7e605f001726583c03bcf62835408e03cd413ae917a3a982d6beb4a0f3a01d2", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/agent/bootstrap.json": "63aa919cdf558f74286e12bcf6a3f154c7c2a1d9c89352e56303243af7950909", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/agent/nanopycodeagent.txt": "f4fc35b2352d08c2ab356de779e1313c585b8ae5ae6dde85d53402057fc88e4f", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/agent/trajectory.json": "e2a1ef6b2cd96535728ca6060e34d97bf8d8e1698c3c430f7d7baf4c0b60b7f6", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/config.json": "ec9d6e6cd305f38ee92c5d0856e1117623f629b6073f4d24c6bb5c210ef609b9", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/lock.json": "279c10bde567ecc135eb39374bcba163ecaf81ed4aad1dc3fa899158d354036b", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/result.json": "46c56969319287709f5819508f75cd6b99637b86d005c0bc22284232d95ba4b1", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/trial.log": "5fb10f86471c0bc1965e0630f11d313dfc725211e3a9439411e302869b654167", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/attempts/1/ctrf.json": "84a434bc4e7cfdf70179b50789419c739f5bdf32ad70a3334802d214236f742f", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/attempts/1/test-stdout.txt": "e7df7cbe3f62a09ceb0848405a356c3b092328438aa5cf49860825e85be4af51", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/ctrf.json": "84a434bc4e7cfdf70179b50789419c739f5bdf32ad70a3334802d214236f742f", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/retry-summary.json": "2e3b4876bc6fb35274f4aac59dd86690647ca84050b1737ce9f0759565a53b21", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012/build-cython-ext__vxC7HQK/verifier/test-stdout.txt": "e7df7cbe3f62a09ceb0848405a356c3b092328438aa5cf49860825e85be4af51", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-build-cython-ext-main-20261012.json": "4be984fe0ce9c7af4ceff61103422a5894405a51a931abc968427edf4dd761ed", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012-input/manifest.json": "b0a9ded39afc12eafbe30d0c3ad0dd431374b5b1851b915a89e0e6a7c76015f7", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012-input/config.json": "851c08e1532fec04ecb455e3949c7e32a7054b73f1741ed68c574f73313e1624", + "jobs/dev34-baseline-main-source/jobs/dev34-build-cython-ext-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 11, + "passed": 11, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "11 passed in 8.91s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [], + "reviewed_local_access": { + "atif_steps": [ + 61, + 81 + ], + "finding": "Tests came from the public pyknotid source tree, not the hidden verifier." + } + } + }, + { + "order": 3, + "task": "terminal-bench/cancel-async-tasks", + "task_ref": "sha256:a3d048d351136e48070696cda8bb79660dfd74db1fea3b6da88559f0332699c1", + "trial": "cancel-async-tasks__7u3Kswi", + "job": "dev34-cancel-async-tasks-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791827713.351878, + "stop": 1791827727.0202107 + }, + "started_at": "2026-10-12T17:47:09.606910Z", + "finished_at": "2026-10-12T17:55:48.069843Z", + "seconds": { + "environment_setup": 2.081936, + "agent_setup": 72.805723, + "agent_execution": 378.882392, + "verifier": 43.450678 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 43.254967877001036, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 23, + "tool_calls": 27, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 950862, + "completion_tokens": 55758, + "cached_tokens": 940928 + }, + "estimated_known_cost_usd": 0.075535368 + }, + "task_checksum": "44b08781bbffcbff0555fafbad034389009edeb85c2f83cd10620f91aa24974f", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/agent/nanopycodeagent.txt": "27d5aa7fd02000414b5a6d0d4313c64f4ff5aed536ad1a7b823859aba39bd81c", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/agent/trajectory.json": "f6d4e0ab94c0b096ea3ed731327b7ebd09b42c64b39b82f43f16a6505abb194b", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/config.json": "2587c4c1262adda409e18393581cfd74a77a139648137bdc82865637663d7c30", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/lock.json": "cecb135e5633a20a377ec925b71f6876cadfdca718f1d9094cabd52455f3068b", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/result.json": "fdde6cfb40becf1e0a06a1afb3b065d16a935f21a33cc9a16aafe4e98cc06f3f", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/trial.log": "04a5a7c1d9d5c0e3f1b09b523dfe30d5dd36477e5542583577faa710a7a53b46", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/attempts/1/ctrf.json": "7d8f31aa11da4cd90906331c028f8db13dbc910a5f3a2cb880b1566052b872ed", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/attempts/1/test-stdout.txt": "387309447ada1c5ddc13e862f1e6901ab115497bdd564bbaf6a7522379cf4b44", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/ctrf.json": "7d8f31aa11da4cd90906331c028f8db13dbc910a5f3a2cb880b1566052b872ed", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/retry-summary.json": "15b57aee1acb8d1b95a862de49aaae4b77a4adc60312d5aafae0afbfdfffa7f6", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012/cancel-async-tasks__7u3Kswi/verifier/test-stdout.txt": "387309447ada1c5ddc13e862f1e6901ab115497bdd564bbaf6a7522379cf4b44", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-cancel-async-tasks-main-20261012.json": "9d7eb4fbab558cffb75ecb3b1e874ac79c4864e3a1f494d6a22f5e51334f27e9", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012-input/manifest.json": "3bfd62e81ce15e5c91349fd6085c3ef878c1d47a5a13ab2fbb27f0256341abfb", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012-input/config.json": "0470ac4ad5af6d3229db044465d1c448cfd55102799dc1b5842a772bf166dde9", + "jobs/dev34-baseline-main-source/jobs/dev34-cancel-async-tasks-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "6 passed in 13.67s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 4, + "task": "terminal-bench/cobol-modernization", + "task_ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88", + "trial": "cobol-modernization__KAQyLYX", + "job": "dev34-cobol-modernization-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791827864.3949804, + "stop": 1791827864.4472044 + }, + "started_at": "2026-10-12T17:47:09.324122Z", + "finished_at": "2026-10-12T17:58:05.460421Z", + "seconds": { + "environment_setup": 2.213387, + "agent_setup": 56.022729, + "agent_execution": 551.69724, + "verifier": 25.019921 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 24.823700353997992, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 46, + "tool_calls": 50, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 2809456, + "completion_tokens": 84352, + "cached_tokens": 2780928 + }, + "estimated_known_cost_usd": 0.126466368 + }, + "task_checksum": "5e3806427eaaaa9f0f756a2a87d4f7f50ec1c8a0d329b26bab7dc09657b937cf", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/agent/bootstrap.json": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/agent/nanopycodeagent.txt": "3745a26f1bdef14e1af9c1d7730e9c4f2e6051b1fb2d5e35056ce04f4deddb7f", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/agent/trajectory.json": "48b7b6cde70d08fd954d8b487644b1299940df3bad5e22c845c4d3d23bebe637", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/config.json": "b7b680164c8a678722c0e41131a7e1cd169025e7d5878c2f750cc3e6bd8fe3c7", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/lock.json": "9c57b7a4c9700f295b7fa43ec8a4cfb6441e6a629764c791a01fc6c71269036c", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/result.json": "20d76529caa7437735adbaba52a8b407c09f6d43fa66e81753bf5a62629b6ecf", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/trial.log": "beb0f5f8fbe2dd0a81f3b4ca1f09027bdf30f041464b1ebe0978fca2e9fb4664", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/attempts/1/ctrf.json": "49216d024b629a4d7419d3ef5e1989f0c26faae3e88cc2ce566970798abf3ea1", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/attempts/1/test-stdout.txt": "03508abf76daba9fa98d833225e5a7b0210f8666ab4248d483c3ebdfb1dd7a5b", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/ctrf.json": "49216d024b629a4d7419d3ef5e1989f0c26faae3e88cc2ce566970798abf3ea1", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/retry-summary.json": "33728273c4ce2ea6162774d3be98b1398bb485210aae801b7665fc577b50c0bc", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012/cobol-modernization__KAQyLYX/verifier/test-stdout.txt": "03508abf76daba9fa98d833225e5a7b0210f8666ab4248d483c3ebdfb1dd7a5b", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-cobol-modernization-main-20261012.json": "082daaad921852710e090ba443f9d7b2f27255073560731ca16eb4d7a6368776", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012-input/manifest.json": "ff55fd1818c62f444b4802935429ae342300b0e7696eb6b5a3449b34d2ef727d", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012-input/config.json": "36accd91ea150e907d2bbd243251f12a0cd6b9fe2f9ea00e4889cc16d871ee40", + "jobs/dev34-baseline-main-source/jobs/dev34-cobol-modernization-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "3 passed in 0.05s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 5, + "task": "terminal-bench/db-wal-recovery", + "task_ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "trial": "db-wal-recovery__BCqJJMc", + "job": "dev34-db-wal-recovery-main-20261012", + "raw_reward": 0.0, + "classification": "scored_failure", + "exception_type": null, + "official_test_summary": { + "tests": 7, + "passed": 5, + "failed": 2, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791828396.3880813, + "stop": 1791828396.414536 + }, + "started_at": "2026-10-12T17:58:18.300826Z", + "finished_at": "2026-10-12T18:06:57.459317Z", + "seconds": { + "environment_setup": 2.119643, + "agent_setup": 116.052168, + "agent_execution": 359.595015, + "verifier": 20.162597 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 19.961623117000272, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 26, + "tool_calls": 37, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 1052775, + "completion_tokens": 55515, + "cached_tokens": 1016320 + }, + "estimated_known_cost_usd": 0.08365242 + }, + "task_checksum": "c18abdc4fdc3a01bf374c55a9700708fe6a9662077d29db81abb692f0a3c5f6f", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/agent/bootstrap.json": "2ffe3efbf9fd9f423c3fc82102152f59af7f9180f8aaf39c46e61c96e2f02d76", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/agent/nanopycodeagent.txt": "7f64e609bf4dc26e61fe1c78225a3a98f9c005d678e4e0347d1be04278af7361", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/agent/trajectory.json": "4ce2e6a8cc141829c6edb23d03654f06094a5f6cf064e0ffb09e0d4d5a2ec628", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/config.json": "c37568b90dd79cdae4684f8f5ffc26d5125d7ecba84f4e9c6f3c7c172aee80ef", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/lock.json": "30cb07d551b480bf970792c1d771435a960c449b193da337e90e251f36e2356a", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/result.json": "9095e885a1d6d55e0b97d945400d09e9d6d184533fd76b81d496709b0a4808e1", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/trial.log": "bd4e169248c258ffe65f47201bd25b17e5b1c58a32a39cf150d27876d968143b", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/attempts/1/ctrf.json": "aae2ae3154dc2098385c4150e58dc2126e994e4e636359c638dc1472277684b1", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/attempts/1/test-stdout.txt": "b75e0d6910c411fe1818df7c823faf2fd1fd580216ab5b4a0a8f188bbf0c832c", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/ctrf.json": "aae2ae3154dc2098385c4150e58dc2126e994e4e636359c638dc1472277684b1", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/retry-summary.json": "873a8c8e40f06767941d32e45bb30b67920e02fec3ac38c4802e6e6b8b9ba082", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012/db-wal-recovery__BCqJJMc/verifier/test-stdout.txt": "b75e0d6910c411fe1818df7c823faf2fd1fd580216ab5b4a0a8f188bbf0c832c", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-db-wal-recovery-main-20261012.json": "e6dd4b153683abe9cbd565a6c1d8834e7b33ab6c36e179050a73f06950095da1", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012-input/manifest.json": "9d3cb366d0e0328fb8b8e0b818af157c29e6e10be805a9a46764275b646bd1b7", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012-input/config.json": "5841ac720e9a6ebffc96e7eb29e9f679f02f619c4db8774091f9ac9c02141abf", + "jobs/dev34-baseline-main-source/jobs/dev34-db-wal-recovery-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 7, + "passed": 5, + "failed": 2, + "skipped": 0, + "error": 0, + "summary_line": "2 failed, 5 passed in 0.02s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [ + 27, + 49 + ], + "reviewed_local_access": { + "atif_steps": [ + 15 + ], + "finding": "The agent listed an empty /logs/verifier directory while looking for the lost WAL. No hidden verifier content was observed." + } + }, + "failure_analysis": { + "cause": "Opening the original SQLite database before preserving its encrypted WAL discarded the WAL. The final records used guessed values instead of recovering the lost updates.", + "official_evidence": "Five tests passed; two value checks failed, including value 100 instead of 150 for id 1.", + "agent_evidence": "Step 5 opened the original database while inspecting the WAL. Backup attempts followed the destructive read. Steps 41 and 51 produced and checked guessed rows for shape/count. The final message at step 53 disclosed the loss and inference.", + "atif_steps": [ + 5, + 7, + 9, + 11, + 41, + 43, + 51, + 53 + ], + "improvement_hypothesis": "Preserve the database and all sidecars before stateful inspection, work on copies, and validate values against preserved inputs rather than schema/count alone." + } + }, + { + "order": 6, + "task": "terminal-bench/fix-code-vulnerability", + "task_ref": "sha256:d31348aa16b533a15f013420e4d9726dc529e6086adc7d6d48067d22cc18fe71", + "trial": "fix-code-vulnerability__TA7Ddkz", + "job": "dev34-fix-code-vulnerability-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791828020.165812, + "stop": 1791828020.1834128 + }, + "started_at": "2026-10-12T17:58:20.389137Z", + "finished_at": "2026-10-12T18:00:41.014749Z", + "seconds": { + "environment_setup": 2.159411, + "agent_setup": 45.834256, + "agent_execution": 58.710773, + "verifier": 12.88474 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 12.68777713199961, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 13, + "tool_calls": 19, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 127168, + "completion_tokens": 5003, + "cached_tokens": 117760 + }, + "estimated_known_cost_usd": 0.00953256 + }, + "task_checksum": "80d707a96acfd0e19af96f0f0c8f44deefdbf167538fefbc0d1dc6391392090a", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/agent/bootstrap.json": "8d1fa3e359c1a2c87bb9b262c9113a423af3445aa5a20da8a2e6fb4d63b4f67a", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/agent/nanopycodeagent.txt": "5cb666badbe2878546cafa8292a0c430799497fff83fa7f204649efe1e03ccaf", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/agent/trajectory.json": "ada18e72edfca14062ea84b1f355fb46aedd2275e65810085e0bdabc5a522aca", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/config.json": "1d46c6dbd31d1cf98fcab766ccf7f3b9275de9559274067ccd1c13100fc08d44", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/lock.json": "2390fc81c4243de841f2da3655693225d4991c5430bcb4db705e13e157c6224e", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/result.json": "6b89818abef5a2933957fa53ee2448d9add908ba3db59deb3c6553925a193d73", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/trial.log": "1017376a9c4cb22597eabfef51dcae9fc8acc64615740314c9e6cd314f5d4c6f", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/attempts/1/ctrf.json": "dd54b24cdf1a175daeabb64d50d2939e5981e97ae5000acfbfd7e62440cc63db", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/attempts/1/original-repo-ctrf.json": "00fd792f5c5085f1e97b41c91fe2a0cf788d7ee486b235d7a6073b154bf01210", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/attempts/1/test-stdout.txt": "5b1c1b620157e4e7ee7e5368a1cd9f8de20235ae042eaeeb7f266332d7e38f69", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/ctrf.json": "dd54b24cdf1a175daeabb64d50d2939e5981e97ae5000acfbfd7e62440cc63db", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/original-repo-ctrf.json": "00fd792f5c5085f1e97b41c91fe2a0cf788d7ee486b235d7a6073b154bf01210", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/retry-summary.json": "138ba5bcae551a441ead9637a5fa97e1890e9c8ae3a91d3052264810827abc1f", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012/fix-code-vulnerability__TA7Ddkz/verifier/test-stdout.txt": "5b1c1b620157e4e7ee7e5368a1cd9f8de20235ae042eaeeb7f266332d7e38f69", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-fix-code-vulnerability-main-20261012.json": "52ce343d3ef2e0689367bb69be86a7f9914958075973aebc7a03265293d80255", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012-input/manifest.json": "2b512156333a1487d988bb873a46ec7994bfe6a7278dcf582c492ba62579a924", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012-input/config.json": "a9d3c02190a1191bceb7878851b21c7a8342cf922a59f17b4a652212cce33afb", + "jobs/dev34-baseline-main-source/jobs/dev34-fix-code-vulnerability-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "6 passed in 0.02s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 7, + "task": "terminal-bench/git-multibranch", + "task_ref": "sha256:7759a92a373c967dd31535975bad8659410644025673fe62385444fd2605c5fa", + "trial": "git-multibranch__ufZejR2", + "job": "dev34-git-multibranch-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791828785.4259021, + "stop": 1791828886.6810434 + }, + "started_at": "2026-10-12T18:07:06.135024Z", + "finished_at": "2026-10-12T18:15:07.672075Z", + "seconds": { + "environment_setup": 2.09602, + "agent_setup": 98.278143, + "agent_execution": 238.22446, + "verifier": 121.749228 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 121.55208354000206, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 34, + "tool_calls": 41, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 786406, + "completion_tokens": 26099, + "cached_tokens": 772608 + }, + "estimated_known_cost_usd": 0.040093848 + }, + "task_checksum": "abe8a7f8f9dcb0e2170740e4735559b368cecd925ddee09c9ed81a7c67dce90a", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/agent/bootstrap.json": "391e000959d10d8aa551310ebe7bddd0c70185a684ee8cb543b7e8bf57e5e63b", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/agent/nanopycodeagent.txt": "7d1743e6b7b0148bce43be28469dfc0b690a0633abca6bd41575deb6b4e1c4ac", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/agent/trajectory.json": "82dd6175afcc59e979ff205d236025190448be510d9b6a3ff2f0988b73b98e2b", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/config.json": "68c8686c93341ea65f1b7970cba8f3fcdda84754cedc0626c9fd038176762223", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/lock.json": "dae7b5396f4d02c1179bae72fd4bbd6a21eb51f8223c276b11ca7ad417d80f23", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/result.json": "ab3bd8f9b1b8ed390b735e0da3d9785d31eecbbf9d827dc72e9cfc23842c63a8", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/trial.log": "c32df93f45c9d4e288213da31eb2dbc3d50fa0e5edcdce2d980efdccba7c4053", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/attempts/1/ctrf.json": "eaf803a08e37b365f2d76d6e0f3e4e293285ae12cf77ba23cebea95a0827352a", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/attempts/1/test-stdout.txt": "2077df3f0a7e86f4f2671448876055446c091ddf02633af1e836f859bcad3d54", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/ctrf.json": "eaf803a08e37b365f2d76d6e0f3e4e293285ae12cf77ba23cebea95a0827352a", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/retry-summary.json": "4aedb7ab49a65872244c8f924814a60c804f11a84df3538e417f5d38d3802daa", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012/git-multibranch__ufZejR2/verifier/test-stdout.txt": "2077df3f0a7e86f4f2671448876055446c091ddf02633af1e836f859bcad3d54", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-git-multibranch-main-20261012.json": "567b20f11373f7b55814bc113edbeed4b5b36bc6845ac74765fc890078a3120e", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012-input/manifest.json": "ca519b9212bc78ceab531c628f8ad818f4b9143bd050a957cce3bc2877b10ef7", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012-input/config.json": "73bb4f7638d0f465cd277a0ad6c1e3490b63935197cd74385ad9df888149c620", + "jobs/dev34-baseline-main-source/jobs/dev34-git-multibranch-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "1 passed in 101.25s (0:01:41)", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [], + "reviewed_local_access": { + "atif_steps": [ + 11 + ], + "finding": "The agent read bootstrap metadata and listed an empty /logs/verifier directory. No hidden verifier content was observed." + } + } + }, + { + "order": 8, + "task": "terminal-bench/kv-store-grpc", + "task_ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307", + "trial": "kv-store-grpc__pUAoWuK", + "job": "dev34-kv-store-grpc-main-20261012", + "raw_reward": 0.0, + "classification": "scored_failure", + "exception_type": null, + "official_test_summary": { + "tests": 7, + "passed": 5, + "failed": 2, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791828972.348956, + "stop": 1791828972.4169147 + }, + "started_at": "2026-10-12T18:07:06.013437Z", + "finished_at": "2026-10-12T18:16:33.481604Z", + "seconds": { + "environment_setup": 2.150376, + "agent_setup": 58.76589, + "agent_execution": 439.211256, + "verifier": 46.123466 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 45.92737010899873, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 20, + "tool_calls": 26, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 239854, + "completion_tokens": 17895, + "cached_tokens": 232448 + }, + "estimated_known_cost_usd": 0.025090488 + }, + "task_checksum": "901a4dd5c3078d5155043875a0d07f3583f462c73f919b52855c025e680b1edd", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/agent/nanopycodeagent.txt": "bbea0cd5cbc04b477def384ef4ad0513e7e57ce07f304b026c9e88454dc65f49", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/agent/trajectory.json": "36c4aa60b39c84ae4bb40e55a0bda69c6b13f5646feed1b02bf4f7bedc5ea4f6", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/config.json": "745a642b6e7bd1f7a5bb7804b63a63b381236ad826a61c1b1d28f2ad3b59168d", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/lock.json": "a60bc07889acc431c04519e08c025b29d803cb926f1fba65fd0531f9c702e8f6", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/result.json": "028a8bba4fc04baed2f6a48651156a716700a8db1ba0c795283f1203f7fe068b", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/trial.log": "711b6ee7f9af520f69344cdd25a345fecfcdf181eb847ba721ab4c25c59d6cbd", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/attempts/1/ctrf.json": "e8c4f69bd06ba8bebaa3ac13170eca3f157053fc925a4d7d0db106ca1301c72c", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/attempts/1/test-stdout.txt": "d4e4fd05b333e67dd9028c53b042729e5f8b675da6bc22642e57d39de32cc2bd", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/ctrf.json": "e8c4f69bd06ba8bebaa3ac13170eca3f157053fc925a4d7d0db106ca1301c72c", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/retry-summary.json": "3134eb5510f040621b00d652d0fb1369d15629d26b7b7aade4baed7ee067cf79", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012/kv-store-grpc__pUAoWuK/verifier/test-stdout.txt": "d4e4fd05b333e67dd9028c53b042729e5f8b675da6bc22642e57d39de32cc2bd", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-kv-store-grpc-main-20261012.json": "4b65710461d78c82012a98fb727cdf01855c79e663b9dd9391d34e98714302ab", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012-input/manifest.json": "a2b2827a58b48a5be9aaaee33eda0db7da859c3a139bbc5d3a4a5d39d40c92df", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012-input/config.json": "ec41a3a69a5fe9f14aecadfd2bbd07bfdc971297878ef95cde9460a42cbf3dd1", + "jobs/dev34-baseline-main-source/jobs/dev34-kv-store-grpc-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 7, + "passed": 5, + "failed": 2, + "skipped": 0, + "error": 0, + "summary_line": "2 failed, 5 passed in 0.07s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + }, + "failure_analysis": { + "cause": "The request schema used val instead of the required value field; self-tests reused the same incorrect generated schema.", + "official_evidence": "Five tests passed; two failed while constructing SetValRequest(value=...), before the RPC.", + "agent_evidence": "Step 7 defined SetValRequest.val. Clients at steps 9, 13, 23, and 37 also used val. Real network checks therefore missed the contract mismatch; step 41 claimed complete verification.", + "atif_steps": [ + 7, + 9, + 13, + 23, + 37, + 41 + ], + "improvement_hypothesis": "Extract literal API names/types from the task into an independent contract fixture and validate descriptors and a client built from that fixture." + } + }, + { + "order": 9, + "task": "terminal-bench/log-summary-date-ranges", + "task_ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6", + "trial": "log-summary-date-ranges__Zr4DKzE", + "job": "dev34-log-summary-date-ranges-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 2, + "passed": 2, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791829112.6381407, + "stop": 1791829112.650529 + }, + "started_at": "2026-10-12T18:16:44.236373Z", + "finished_at": "2026-10-12T18:18:53.646064Z", + "seconds": { + "environment_setup": 2.10584, + "agent_setup": 60.463233, + "agent_execution": 23.138039, + "verifier": 22.508523 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 22.311106545006623, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 7, + "tool_calls": 8, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 96985, + "completion_tokens": 2471, + "cached_tokens": 80128 + }, + "estimated_known_cost_usd": 0.008503068 + }, + "task_checksum": "c833c594814ec7b8cb32eba3b9cb5ed648171efe5a074767aa64c25ea060f08f", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/agent/nanopycodeagent.txt": "64995fc7594eb28558207b736991ac5d29afdb48fec8ed642428682af16d814c", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/agent/trajectory.json": "b18a9831cf624845a297db3c440a18443f5e4944eb6365aca9665d2d8d969620", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/config.json": "00c2e401ab832fe2729b89ea9a62a485f5ece2091d4b332c04d00b7211eeae95", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/lock.json": "0008ebe87e928565afff7097f67e7cd7abdaa364beb1ff8eae78289dee9419b2", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/result.json": "4414fd2333fa08d97f82ffe106d194f68ef82fe883cc6e9c20eb95c8d3319fa1", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/trial.log": "c3ef91c507c13f5e06c871e695b4d4784477470a02d9fff2cdc24faa0cb65456", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/attempts/1/ctrf.json": "ae8736d6f2d5602be2154040dbe5411ae10ea888d91aff63231c21e065f08b80", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/attempts/1/test-stdout.txt": "0f30be1e7733def79fcdb50f13fc2403ced950850aa903dca0feb5bfcfc78758", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/ctrf.json": "ae8736d6f2d5602be2154040dbe5411ae10ea888d91aff63231c21e065f08b80", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/retry-summary.json": "b276dff563df4eaf9b96f01251b862980a75de090168328d1b794288c77d0323", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012/log-summary-date-ranges__Zr4DKzE/verifier/test-stdout.txt": "0f30be1e7733def79fcdb50f13fc2403ced950850aa903dca0feb5bfcfc78758", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-log-summary-date-ranges-main-20261012.json": "97ce365cccfc2f9144113f7fa32ccc244f211ee36a154e38120fce9b829a80d3", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012-input/manifest.json": "1daad4c01c07e9b72501cb76bc85609d67b05490aed5b3b6bf7647b3f5aaa089", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012-input/config.json": "373e00f57411c723de60e3762d636ffff3dcfb4172611047c90c4010d9fe700f", + "jobs/dev34-baseline-main-source/jobs/dev34-log-summary-date-ranges-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 2, + "passed": 2, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "2 passed in 0.01s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 10, + "task": "terminal-bench/merge-diff-arc-agi-task", + "task_ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d", + "trial": "merge-diff-arc-agi-task__mQorUxv", + "job": "dev34-merge-diff-arc-agi-task-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 5, + "passed": 5, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791829221.4425132, + "stop": 1791829221.4609864 + }, + "started_at": "2026-10-12T18:16:43.163376Z", + "finished_at": "2026-10-12T18:20:42.546934Z", + "seconds": { + "environment_setup": 2.171625, + "agent_setup": 68.233128, + "agent_execution": 123.162561, + "verifier": 24.519941 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 24.324041050000233, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 18, + "tool_calls": 19, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 301706, + "completion_tokens": 18026, + "cached_tokens": 291968 + }, + "estimated_known_cost_usd": 0.026304408 + }, + "task_checksum": "f6ac6f6b96613c5a110b0b0d51172b7cad781c0adc99b556425408d8d3278147", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/agent/nanopycodeagent.txt": "b79e4d0dc77fc4dc9f40c87d3c7f0bdc7ee4eee79fc3a69f5a5100585bd28e54", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/agent/trajectory.json": "36ff4b17aa2a379f8e86dfc64a0edad55a45999b92cb3c0c97988a884adea26d", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/config.json": "090ee8c327afde3f23bcc08946b330ddc84c0f932131931fbcb3ece7dc6e744e", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/lock.json": "a476408b75b7271c4c9ef10b0a3005aa117822109b1d73a024aacd4dd3182dde", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/result.json": "8fbfcf8e310a656ccc8d18a8d011dc109125242781707391665ad6c065f61210", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/trial.log": "d2d1e1bb4bc5b115eb9d5ab6c9935cb963343306845d3ab84b3a84d6735152d6", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/attempts/1/ctrf.json": "d170545d104233003d44ea14e6792f79abff8fa024949f2f9e59fa5cf09db7d8", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/attempts/1/test-stdout.txt": "b71f9b52f2d5da5ae25ba50c329f4395ae179c8289b1920b7a5d3b32bcb512cb", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/ctrf.json": "d170545d104233003d44ea14e6792f79abff8fa024949f2f9e59fa5cf09db7d8", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/retry-summary.json": "4ddb2ae32cf6fcd2c10d9321d1f308e9c76514a26e15f140dcde3baf1623a26e", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012/merge-diff-arc-agi-task__mQorUxv/verifier/test-stdout.txt": "b71f9b52f2d5da5ae25ba50c329f4395ae179c8289b1920b7a5d3b32bcb512cb", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-merge-diff-arc-agi-task-main-20261012.json": "12a3c803dcc64564160f8227c937a310402d81670bb18cfc66470db2ce2b3f2c", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012-input/manifest.json": "4a4646ea3f0c3f0692b704e18b358ad1a7655cd0d452bef1ad2f116d5983db81", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012-input/config.json": "97703fcef4605a879d2de7c3670bd19a738f318d780c54e44abbe6d268636d1d", + "jobs/dev34-baseline-main-source/jobs/dev34-merge-diff-arc-agi-task-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 5, + "passed": 5, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "5 passed in 0.02s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 11, + "task": "terminal-bench/model-extraction-relu-logits", + "task_ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17", + "trial": "model-extraction-relu-logits__Zs75P6A", + "job": "dev34-model-extraction-relu-logits-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791829701.4651358, + "stop": 1791829703.2166524 + }, + "started_at": "2026-10-12T18:20:52.207702Z", + "finished_at": "2026-10-12T18:28:44.245369Z", + "seconds": { + "environment_setup": 2.112638, + "agent_setup": 40.985069, + "agent_execution": 380.003553, + "verifier": 27.721415 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 27.52267157800088, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 14, + "tool_calls": 15, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 666388, + "completion_tokens": 63046, + "cached_tokens": 620800 + }, + "estimated_known_cost_usd": 0.0930564 + }, + "task_checksum": "8b3048320aa04676089f9240563c2d7d381be03d8810bcf5f5d336cab538f523", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/agent/nanopycodeagent.txt": "535786e780ed0c430735f3f60441ee29b85fc726df33789437ea07d62f3e06bc", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/agent/trajectory.json": "d644b512caf5b376a5546516ff6f0b56087bb76ec70f1c24e3ccf5acc0ad04c0", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/config.json": "87e9e3c37614f893c9a51574232644c7266d9370cea8ece102fef37199419104", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/lock.json": "063546eec02227267acb4ab319e948835be0da47f9dec5ea2feddb7167e85f17", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/result.json": "c55fae295c679400e1e75bd130524d8ae491dc3f35a7bb8ad954eb4468f18d2f", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/trial.log": "808f854039d999594bc8f1d1d65bedc3a6bb0d897f241000b7b808f83133b60d", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/attempts/1/ctrf.json": "c14b5b428a4dd45ca00a64cd49c9971f9d55659e0e433b0f52e2269c34234ff0", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/attempts/1/test-stdout.txt": "8d1edede3f588912e482bb88ce8f3574a3259875d858c244ad121673d8c4a8b5", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/ctrf.json": "c14b5b428a4dd45ca00a64cd49c9971f9d55659e0e433b0f52e2269c34234ff0", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/retry-summary.json": "edd97b3d515d07c80bd230faaf82bc3050fbe9319ccf2d60f8e20a9397e0bd43", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012/model-extraction-relu-logits__Zs75P6A/verifier/test-stdout.txt": "8d1edede3f588912e482bb88ce8f3574a3259875d858c244ad121673d8c4a8b5", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-model-extraction-relu-logits-main-20261012.json": "69d6de1582880fd15888e3229d3b0315ad10eb01060d32942465b28614dfcfe8", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012-input/manifest.json": "15d9d18dfa0865494a1eda833639fb866c522e2a3f30ad2b65655675a672ac53", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012-input/config.json": "e1e2006ef652f920ac66b485e8a3e4d849addf080a1d2e32f7239feafc6c28c0", + "jobs/dev34-baseline-main-source/jobs/dev34-model-extraction-relu-logits-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "1 passed in 1.75s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 12, + "task": "terminal-bench/multi-source-data-merger", + "task_ref": "sha256:70367c38732e1beda7b229968a48d60242277c2fa4db91339c3f064c4c230d49", + "trial": "multi-source-data-merger__48opj3y", + "job": "dev34-multi-source-data-merger-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791829371.6209648, + "stop": 1791829372.6695817 + }, + "started_at": "2026-10-12T18:20:50.179695Z", + "finished_at": "2026-10-12T18:23:13.855648Z", + "seconds": { + "environment_setup": 2.165042, + "agent_setup": 37.253593, + "agent_execution": 33.06938, + "verifier": 49.960229 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 49.75263675000315, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 6, + "tool_calls": 6, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 30503, + "completion_tokens": 5151, + "cached_tokens": 28288 + }, + "estimated_known_cost_usd": 0.007015428 + }, + "task_checksum": "33fa3b988ff60ec62b6ce40ee455208cb083ac1ac46ddc5247e954c88b9d5e8e", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/agent/bootstrap.json": "43369761e1c62328eb0c3890c99d441d724fe366a178abf9c568928eb85c4174", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/agent/nanopycodeagent.txt": "70f3092a8a32066d53ca60c821bba3580895df542f0eced615c9e93648b535a9", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/agent/trajectory.json": "09a328e0d43c9571d3c9ae885416e44ddc539abc15182514c56a144960fe79c6", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/config.json": "3115b56ed2d03d36ab296aa7c04c3ca53e43c6ee32f37f1e3147b1785c328432", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/lock.json": "156ec2828c7ae8420765e3360b23b6af77edd6c8a43928f8add8e5bbecb814c5", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/result.json": "59855416f2bcb6f956b40da7cca7dc113249e751dd2ae592d1933a7dbb4f1901", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/trial.log": "8ba3d5657e2429f4cf2662bb4565d43933ca069719971d0dbbfab5e505c22043", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/attempts/1/ctrf.json": "0e387ff7fedcc921602c20e046b44d4a79593301621922babc081f82377de1e9", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/attempts/1/test-stdout.txt": "075473c99116b68a1571c1a79d062857214cde8bbcc627d2099ec18dd54b9277", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/ctrf.json": "0e387ff7fedcc921602c20e046b44d4a79593301621922babc081f82377de1e9", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/retry-summary.json": "23d190ac0d7e2db6c64d9995114c1480c8504ca36eed68e3d65cfa8d588b8f0f", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012/multi-source-data-merger__48opj3y/verifier/test-stdout.txt": "075473c99116b68a1571c1a79d062857214cde8bbcc627d2099ec18dd54b9277", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-multi-source-data-merger-main-20261012.json": "4bcc848bef41b25ae7617356783db36afa760293d91119741e62671f2ca84300", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012-input/manifest.json": "8c018a9e77555653178582f4697d61ea23473aad82b7c37234a8f1173a953087", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012-input/config.json": "f670a1935f6b8599d81233462c6ded4f2774fd30bde3b375619fd6222c50c9b2", + "jobs/dev34-baseline-main-source/jobs/dev34-multi-source-data-merger-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "3 passed in 1.05s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 13, + "task": "terminal-bench/nginx-request-logging", + "task_ref": "sha256:9d1b8bebd989ea0bc8080c3b159480068caf183c5fd61385868b2574b206e097", + "trial": "nginx-request-logging__StiD5Bg", + "job": "dev34-nginx-request-logging-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 8, + "passed": 8, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791829863.4939477, + "stop": 1791829865.651691 + }, + "started_at": "2026-10-12T18:28:54.221510Z", + "finished_at": "2026-10-12T18:31:26.640772Z", + "seconds": { + "environment_setup": 2.116655, + "agent_setup": 40.956052, + "agent_execution": 69.991785, + "verifier": 18.200302 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 18.000908178000827, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 10, + "tool_calls": 12, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 53899, + "completion_tokens": 3859, + "cached_tokens": 50048 + }, + "estimated_known_cost_usd": 0.006086388 + }, + "task_checksum": "913305d8f286ff121b30e4893217142190b27f12952343f22c0999dd2a4d725e", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/agent/bootstrap.json": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/agent/nanopycodeagent.txt": "c9c2c8ea72d2f945b2080e4a6ad351c549fd10c20d88e5f5fb9ea0994981b674", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/agent/trajectory.json": "728a824202e401d7a6358fe45813cf29fd126f829f69e24dc883a60bd3b9ddc1", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/config.json": "8b9bd5c7be2e1c4f5146cf6ff961b7b353db33d8c0fe79651a40363afa6e8baf", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/lock.json": "ae86cceca7c8d888cc43faecd2ba9065926b11d16fea9b1faa6cab2fb9cb6a26", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/result.json": "545c441ef8faa7985d860e6ee87a8daf17dff8297ccfd1e763a6ad3ba4bc775c", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/trial.log": "71e2f4ec349e53838f484b004cf7ccb95cad96bad5897c2adc683da11b45a3c0", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/attempts/1/ctrf.json": "b1b30bbfeec6cdfdf6aec2b813dd92686565b7d11e090b82183c6d5cfd807d69", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/attempts/1/test-stdout.txt": "0743dc95a0638adc17912b3411bd2756b08c1a0f051c3cf0b990e00c5a3f5d3d", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/ctrf.json": "b1b30bbfeec6cdfdf6aec2b813dd92686565b7d11e090b82183c6d5cfd807d69", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/retry-summary.json": "6ad2eb1373733e495f4482718c70c07f19ffb352d747a6bcacb21dd59267bc46", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012/nginx-request-logging__StiD5Bg/verifier/test-stdout.txt": "0743dc95a0638adc17912b3411bd2756b08c1a0f051c3cf0b990e00c5a3f5d3d", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-nginx-request-logging-main-20261012.json": "5bdba723ee0aa8efad7491596628c9592f7d1dac6d1287797dfcdd391a18b064", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012-input/manifest.json": "691203cec6935b23176b49181a42c685d3a6371d5f09dc65c4b4fb52a92ccaf4", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012-input/config.json": "f3999a7ab6cca214253f1c2055d9cbd7133a2354c18dd739f1c8a2936ba303a1", + "jobs/dev34-baseline-main-source/jobs/dev34-nginx-request-logging-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 8, + "passed": 8, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "8 passed in 2.16s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 14, + "task": "terminal-bench/openssl-selfsigned-cert", + "task_ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302", + "trial": "openssl-selfsigned-cert__2a3CFWZ", + "job": "dev34-openssl-selfsigned-cert-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791829869.3702667, + "stop": 1791829869.4791772 + }, + "started_at": "2026-10-12T18:28:56.243841Z", + "finished_at": "2026-10-12T18:31:30.463236Z", + "seconds": { + "environment_setup": 2.10392, + "agent_setup": 41.411823, + "agent_execution": 70.040116, + "verifier": 19.51004 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 19.304975794999336, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 19, + "tool_calls": 18, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 167998, + "completion_tokens": 10046, + "cached_tokens": 161408 + }, + "estimated_known_cost_usd": 0.015000648 + }, + "task_checksum": "2b70d5535b5873f644fad37b76dbef86a1e42162e018c7bc06316e5e2521929a", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/agent/nanopycodeagent.txt": "8ccea43c8900f9276e8dd40e6a6ce7d4cc036b2f3db6809e1ae72c395bcd603d", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/agent/trajectory.json": "dcfd891cd9e786fd461a25e4673a0689254879e6ec6afb9d574fbf4165a413f7", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/config.json": "4bc80622f50f13ba296a2f3508c1d4596deb3c306e533fea82d5dea2312f6506", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/lock.json": "60de1ecd6f04787b05315e5389e79297b6e4657864dca0f72c14bde6928030de", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/result.json": "8db19e1f9bdf16caad4ebaaf19524964476012abdfef06da99a14c816ef5c326", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/trial.log": "266d42dc2fc2bab466c29c6f172bd95745fdbc49202febf807074d1e8d546848", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/attempts/1/ctrf.json": "1d044484b65856cedf2c9cd62550a7def0b90363019d70f28f3e7bb304881815", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/attempts/1/test-stdout.txt": "1c0bf2eb7a1fa2ede4d9e4efabba3ef035fea2860fecce6ada34c8b139fd742d", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/ctrf.json": "1d044484b65856cedf2c9cd62550a7def0b90363019d70f28f3e7bb304881815", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/retry-summary.json": "bd6b2183686459bf1095e089e4be89b6a3d7edf90ac8fa769f87651d23645ef7", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012/openssl-selfsigned-cert__2a3CFWZ/verifier/test-stdout.txt": "1c0bf2eb7a1fa2ede4d9e4efabba3ef035fea2860fecce6ada34c8b139fd742d", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-openssl-selfsigned-cert-main-20261012.json": "39eba9830e9a42e383bc7038e17b3db6402f486d9f9962549ffd06155309013e", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012-input/manifest.json": "21bc6a5f9ef2bf87ef6aba677e3dec9b27a04f696ad2bf50582b055dcfdf1e90", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012-input/config.json": "6ee5e40eb2650f9dca7eb9c26a3d2d07c8fdcceceedefe712d8e817dff988ffc", + "jobs/dev34-baseline-main-source/jobs/dev34-openssl-selfsigned-cert-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "6 passed in 0.11s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 15, + "task": "terminal-bench/pypi-server", + "task_ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1", + "trial": "pypi-server__QydrX5C", + "job": "dev34-pypi-server-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791830094.4410067, + "stop": 1791830097.019907 + }, + "started_at": "2026-10-12T18:31:37.721873Z", + "finished_at": "2026-10-12T18:35:18.098613Z", + "seconds": { + "environment_setup": 2.069702, + "agent_setup": 47.026743, + "agent_execution": 129.100977, + "verifier": 20.920622 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 20.703110141999787, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 17, + "tool_calls": 24, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 136735, + "completion_tokens": 8261, + "cached_tokens": 130560 + }, + "estimated_known_cost_usd": 0.01254906 + }, + "task_checksum": "9eb21236249d5b42ee64c7229bb1e5a3b3c0be7ed5ee9801bdd0d16ca575941a", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/agent/bootstrap.json": "bd617a3903bf50c818febe3abf77cc268952deb3cd9f0c6db99bc0cbf8146a94", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/agent/nanopycodeagent.txt": "a160adc3ebf081133e6d5d49009288ff20f186faf64e913cfa3aafbfaf8aab1f", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/agent/trajectory.json": "80fe56d87e976ecd8d24be3a65054faac5dcc993b8ba61a12f07bc0f9b93310d", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/config.json": "4477284e6dbe47a42acaa92fb78ea8fb9ceb8d95a7f9304124de37679f271556", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/lock.json": "697f391a655e2115100fc610a9646238e5606170810df6adf9937a6da677d7c7", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/result.json": "d22265a60535d74347b1fdb8ac0815a724bb52284aa02d43335e2ad064b1a188", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/trial.log": "da01cbe8e306b89addebe513d3ab457cff2b0f4c0edd7d8e50e0e6f5d3908847", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/attempts/1/ctrf.json": "93a123248fdada26e5c11dd867531e4a5f197e22d9e6a1a6cb20d9e6730be49c", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/attempts/1/test-stdout.txt": "ab215b9035200df1ef2bcca4ac7776f2161e447f8a954964fefb8b75c1139e37", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/ctrf.json": "93a123248fdada26e5c11dd867531e4a5f197e22d9e6a1a6cb20d9e6730be49c", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/retry-summary.json": "362cb1bae7630f31680639ad5dd0511e869b88ab923771b43167d828302af740", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012/pypi-server__QydrX5C/verifier/test-stdout.txt": "ab215b9035200df1ef2bcca4ac7776f2161e447f8a954964fefb8b75c1139e37", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-pypi-server-main-20261012.json": "6d9887df3fcddad2699d7c653d058db8de8df4e7a577ebbdc0db3196304385f1", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012-input/manifest.json": "d224ffc75c2e4eaa90a74e3d81c94b6df2c3d4a8c19c8c8a27839ede042b3c0a", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012-input/config.json": "38ecc7d0b1eccd6ee52cf2d476f3f50bb392ba57a285fd1ce974e79df0813e47", + "jobs/dev34-baseline-main-source/jobs/dev34-pypi-server-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "1 passed in 2.58s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [], + "reviewed_local_access": { + "atif_steps": [ + 29, + 31 + ], + "finding": "The /tests match was the agent-authored vectorops unit-test directory." + } + } + }, + { + "order": 16, + "task": "terminal-bench/pytorch-model-cli", + "task_ref": "sha256:6f11544dcc81a380dc4611902961f5520259c462d9fceb413419d66ab9a96c79", + "trial": "pytorch-model-cli__nZN835D", + "job": "dev34-pytorch-model-cli-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791831011.3410046, + "stop": 1791831039.5660374 + }, + "started_at": "2026-10-12T18:31:37.417131Z", + "finished_at": "2026-10-12T18:51:02.784736Z", + "seconds": { + "environment_setup": 2.125652, + "agent_setup": 42.488816, + "agent_execution": 392.68696, + "verifier": 705.626153 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 705.433562957005, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 24, + "tool_calls": 32, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 420051, + "completion_tokens": 19692, + "cached_tokens": 408832 + }, + "estimated_known_cost_usd": 0.029449092 + }, + "task_checksum": "e8438573f503c2d98ad71905862638bfa485470010d15d5b092821e0096b9daa", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/agent/nanopycodeagent.txt": "2af25d3fe6aa86e61b1b59f0595a2c4d8005e2587f5f3d1b6c95f1165d00d6c7", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/agent/trajectory.json": "bdbc760feaf957839b033c06a57a80bb9c9cad12c9072fcdc7c04fdee2181112", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/config.json": "729877c79b842de8d80a512c7d37e02215bd1b5fbcefaa2ba6782265dd8212ea", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/lock.json": "9d018ffccb76ede5631f8fa29c7773b5eb319860e20ee288d6c876e1fe6c3a95", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/result.json": "9ecc4496c5ff9d9cbad2f2615ccf677b822ab638a63dc06f40239c515e7238b0", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/trial.log": "96f9413830ca1ab6c119b07011b3cec6acbd806d6817f96cb15a2fa51e2ef1fc", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/attempts/1/ctrf.json": "cb46570283985a7442f532115e442ab4ef280b5150654b4b739864241878226b", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/attempts/1/test-stdout.txt": "930d997650bd512f3b7455f03cd3f5cad5c3e579aa79ed0f01c8284b8e85bc81", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/ctrf.json": "cb46570283985a7442f532115e442ab4ef280b5150654b4b739864241878226b", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/retry-summary.json": "7111245c79cfcc4744768a5e97f8d943f3fe265c3d15876960f1607235eb4a45", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012/pytorch-model-cli__nZN835D/verifier/test-stdout.txt": "930d997650bd512f3b7455f03cd3f5cad5c3e579aa79ed0f01c8284b8e85bc81", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-pytorch-model-cli-main-20261012.json": "61152ab9f7476bacf2c6f234a28d221cacd0b601b6c55ced03bc3ef5a6555fcc", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012-input/manifest.json": "d4b33002d807de0614127906f4377bcdb9db4656268cebe303af18431021f57a", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012-input/config.json": "a7e08a5da22529885e2208cb1bc3655d4b9aa5dd85f0da2ef05ae048b717e4ca", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-cli-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "6 passed in 28.23s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [], + "reviewed_local_access": { + "atif_steps": [ + 35 + ], + "finding": "The /tests match was an agent-authored /tmp/tests image directory for differential checking." + } + } + }, + { + "order": 17, + "task": "terminal-bench/pytorch-model-recovery", + "task_ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc", + "trial": "pytorch-model-recovery__mzRpfsa", + "job": "dev34-pytorch-model-recovery-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 5, + "passed": 5, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791831907.2906036, + "stop": 1791831911.9282436 + }, + "started_at": "2026-10-12T18:51:11.698162Z", + "finished_at": "2026-10-12T19:05:34.440873Z", + "seconds": { + "environment_setup": 2.146235, + "agent_setup": 39.597534, + "agent_execution": 363.582888, + "verifier": 435.240241 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 435.02409358399746, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 18, + "tool_calls": 19, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 269683, + "completion_tokens": 17306, + "cached_tokens": 262144 + }, + "estimated_known_cost_usd": 0.024601764 + }, + "task_checksum": "e321c0e7501b3f021cfca26520574b343dff3dee6919f2e9dd5ff9e28ffc3725", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/agent/nanopycodeagent.txt": "e96d3ab473863de3eab6bf419d73c8f07691206cc165a74eadae7aec685c9580", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/agent/trajectory.json": "08e76d44befdbc664bf2a33ff188bad70a55eb8b82318240d21c0a6046e27726", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/config.json": "84d0e286f009d4aa9c2355b78f067911f8f67aea304b4f9fc042298983e57f83", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/lock.json": "ecb167bd590e26b85bc809e25f46a927981043cc5f3bf599d3c5a8ff90cf1cf7", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/result.json": "617eaa0b069c586340607be0f0eadfdfb77d4fabf2caf3978ffb547782433b2b", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/trial.log": "4a10b340bbb454f3fc43a7f2edbafdb7dcd25aec1df9f9d9fdf192a62dda803d", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/attempts/1/ctrf.json": "2981557d0b92325e3b9a879b39fbac4e83288e8d8343b5966b754fc5abf2bff5", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/attempts/1/test-stdout.txt": "4a65e6aca402273570aeb36040bc05bd5ff93073297484b3e8e3df9c58325226", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/ctrf.json": "2981557d0b92325e3b9a879b39fbac4e83288e8d8343b5966b754fc5abf2bff5", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/retry-summary.json": "1041f21d8dca87f6e334bfb8a95e544b334af14ec96f014f1d900a7773a6893b", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012/pytorch-model-recovery__mzRpfsa/verifier/test-stdout.txt": "4a65e6aca402273570aeb36040bc05bd5ff93073297484b3e8e3df9c58325226", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-pytorch-model-recovery-main-20261012.json": "d31e93e3691372cb325e86c9285206b264c90923b409d0a6659090dbc85a8567", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012-input/manifest.json": "92d9fe3021577a7f2e4eb873f37bd2adfa5b5b5ef5487d895d28d8180d5d8113", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012-input/config.json": "ad20f34fb02d2ec3dd365fecd909bc3f67cef62046dd3be1e63fd2d1b7d65334", + "jobs/dev34-baseline-main-source/jobs/dev34-pytorch-model-recovery-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 5, + "passed": 5, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "5 passed, 2 warnings in 4.64s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [], + "reviewed_local_access": { + "atif_steps": [ + 15, + 17, + 23, + 25, + 27, + 29 + ], + "finding": "The flagged text was agent-authored solution/verification code and logs, not hidden verifier access." + } + } + }, + { + "order": 18, + "task": "terminal-bench/qemu-alpine-ssh", + "task_ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28", + "trial": "qemu-alpine-ssh__qwWToXq", + "job": "dev34-qemu-alpine-ssh-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791831692.3782322, + "stop": 1791831692.6773033 + }, + "started_at": "2026-10-12T18:51:11.044955Z", + "finished_at": "2026-10-12T19:01:53.815107Z", + "seconds": { + "environment_setup": 2.267099, + "agent_setup": 53.746348, + "agent_execution": 555.502567, + "verifier": 9.920006 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 9.683342746997369, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": "qemu-alpine-ssh-v1", + "status": "passed" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 45, + "tool_calls": 44, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 1804855, + "completion_tokens": 44714, + "cached_tokens": 1771520 + }, + "estimated_known_cost_usd": 0.07428642 + }, + "task_checksum": "48bb495efc325f96892d45c341d8997c4a13b71bbe7051119f28dcaa7970df4f", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/agent/bootstrap.json": "052b2e131d93ddb3117f1612f23e09e4a0216e77ac7924418a86072bbd3a5d8f", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/agent/nanopycodeagent.txt": "32feaf149243f5604dc806f4dd821602ea5e6d33f0ea0f47951dcb8a4b133e79", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/agent/trajectory.json": "d3f41720f2d00a9aef92fb049090d5c0be6d8cbc7e8f691d8d953d157fad60fc", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/config.json": "4cb108544c3ea2115ad7c3af320ffe76a3fbd498b8572de6504b375f7b9ae142", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/lock.json": "6548703885791b083fa6cb32356f3a9db22c11ffd443826ce756f3a62a856903", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/result.json": "e7d337f5782f55e411f4fa5769bb120951a7625067d1a9b25d1721fae38e4935", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/trial.log": "3f94c97bdfbc189c617f67ec14e485340d457d9e5e0f83183f5e51181670c4e1", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/attempts/1/ctrf.json": "90e5b8c7beb8e8acb7ea2b69ece3a99812a20b4f117457674f6d5de876d0a11f", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/attempts/1/test-stdout.txt": "9faf2c88114688da695210864b175b9473c08da154c950afd3801cff423f6e99", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/ctrf.json": "90e5b8c7beb8e8acb7ea2b69ece3a99812a20b4f117457674f6d5de876d0a11f", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/retry-summary.json": "9d1e0ee6cb28e82dfe6d725ea0f5f2989d1d39de44b75e36c2549040667e8598", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012/qemu-alpine-ssh__qwWToXq/verifier/test-stdout.txt": "9faf2c88114688da695210864b175b9473c08da154c950afd3801cff423f6e99", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-qemu-alpine-ssh-main-20261012.json": "67352224f483b57231d000b2bd713c989aeaf2a1861423a729713a5beeb9354b", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012-input/manifest.json": "251f549b47dab3182f36360921d15de2dc55686b238ac74c1235a1772388e12e", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012-input/config.json": "39677284445b77531b6874e2309a8905eb806079cac402deee25be0855b314a5", + "jobs/dev34-baseline-main-source/jobs/dev34-qemu-alpine-ssh-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "1 passed in 0.30s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 19, + "task": "terminal-bench/query-optimize", + "task_ref": "sha256:169496ea6843cb403b0860132675baf7aa0df0ac8b86221068e27d86395260f4", + "trial": "query-optimize__ZVLSEuZ", + "job": "dev34-query-optimize-main-20261012", + "raw_reward": 0.0, + "classification": "scored_failure", + "exception_type": null, + "official_test_summary": { + "tests": 6, + "passed": 5, + "failed": 1, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791832381.3178947, + "stop": 1791832513.314249 + }, + "started_at": "2026-10-12T19:05:42.440147Z", + "finished_at": "2026-10-12T19:15:34.331956Z", + "seconds": { + "environment_setup": 2.487392, + "agent_setup": 46.054077, + "agent_execution": 373.814475, + "verifier": 148.336553 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 1800.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 148.14902147300018, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 21, + "tool_calls": 30, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 308255, + "completion_tokens": 19054, + "cached_tokens": 299904 + }, + "estimated_known_cost_usd": 0.027169524 + }, + "task_checksum": "593405b6a2f6970f0f4c2ca2aff455a3b32bce3eecf8b506cfeeb0fe93b14838", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/agent/bootstrap.json": "c72c3ff607d38944df0e65d64c3a0a941fdc52c3dc17db638b246035ca2d8f89", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/agent/nanopycodeagent.txt": "26b78decf1c9102f4560b2953af932fca143109a32b2084b98032fbc28f70b18", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/agent/trajectory.json": "0647111c3e036de1dd547f9075a19f2a289ebd1bf05e0bb3547b91c8acd7a3dd", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/config.json": "109488acefd980d505a3eec5ced9035c58339f86dda67f7d0ee98de8b31dbf4b", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/lock.json": "28437ccaefb7eac85da6f226b5a52190dcf146f550293671e0dc4ea8335b7de0", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/result.json": "aa24b5aec7fb7f2602badc45676e33562c25c7973aff74087d05fc7ca185430b", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/trial.log": "74f611cde7992be8bc8ea3143fdf8e8b4a2a49d38c939f9497cd5bc595c16a2a", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/attempts/1/ctrf.json": "3162d237542641fede1219d6b116c920e6c11b94e2ee78692da9f4e79762c898", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/attempts/1/test-stdout.txt": "8365b8e093bc31dbf3fbfea6f8a7a7f5af31d64acbd38cbf9c577b26714e5026", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/ctrf.json": "3162d237542641fede1219d6b116c920e6c11b94e2ee78692da9f4e79762c898", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/retry-summary.json": "d551a5c48acc834182358ef4b789e9e000b02434b5748b09d1f4126083416ad0", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012/query-optimize__ZVLSEuZ/verifier/test-stdout.txt": "8365b8e093bc31dbf3fbfea6f8a7a7f5af31d64acbd38cbf9c577b26714e5026", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-query-optimize-main-20261012.json": "09096d21d7e7c79824ab4109fb98cd69e4eda4d0e73e1977b82b1ceaafb6ee9a", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012-input/manifest.json": "1957b523c1afd2b0541809a1cabfdf80289962f2351fd420cad6f52bc21d6c29", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012-input/config.json": "9af94b6964ac8c44965420245793f468412297eab0295dbdba3edde210ec06a0", + "jobs/dev34-baseline-main-source/jobs/dev34-query-optimize-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 6, + "passed": 5, + "failed": 1, + "skipped": 0, + "error": 0, + "summary_line": "1 failed, 5 passed in 131.99s (0:02:11)", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + }, + "failure_analysis": { + "cause": "The submitted query preserved the required output but did not meet the official performance threshold against an optimized reference.", + "official_evidence": "Five tests passed; performance failed. Reference median was 0.2918136419975781 seconds; solution median was 0.3716236660038703 seconds, above the 1.05x allowance.", + "agent_evidence": "The agent verified an exact 500-row match and preserved the original database, using an indexed copy to obtain reference output. Step 33 attempted repeated timings with a missing /usr/bin/time binary. Step 39 measured about 0.480 seconds; step 43 compared against the much slower original query.", + "atif_steps": [ + 33, + 39, + 43 + ], + "improvement_hypothesis": "Use a working repeatable timing harness, compare candidate query plans in the target runtime, and seek performance margin after establishing semantic equivalence.", + "limitation": "The optimized golden query and exact threshold were hidden from the agent. Correctness validation was substantive; the failure was not simply an absence of testing." + } + }, + { + "order": 20, + "task": "terminal-bench/regex-log", + "task_ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a", + "trial": "regex-log__NmCmvYZ", + "job": "dev34-regex-log-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791832082.1793313, + "stop": 1791832082.1918247 + }, + "started_at": "2026-10-12T19:05:42.325422Z", + "finished_at": "2026-10-12T19:08:23.294264Z", + "seconds": { + "environment_setup": 2.603675, + "agent_setup": 48.665346, + "agent_execution": 72.554623, + "verifier": 15.854588 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 15.647411710000597, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 8, + "tool_calls": 10, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 82222, + "completion_tokens": 12753, + "cached_tokens": 79104 + }, + "estimated_known_cost_usd": 0.016713624 + }, + "task_checksum": "31dc6115c061b96539a5287090ce41a7a89d3201c291b9b843bd70e416f35c39", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/agent/nanopycodeagent.txt": "f2c4b70dff23cb6f09bf5b8ff70e14975e50576ed9488b35ace8460064530165", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/agent/trajectory.json": "b802eb010e3f944eb30a5a1db8e200849f97ae279154d321af891703676b1051", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/config.json": "f3591fca8a2e019be35b7ef39e44f803aae92ef1d218f99cf8da32f728921e3b", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/lock.json": "a1a5d783c569c4a2059da23ed933d6756d7bdd5b265cbf5fbaa65b5742c2b32b", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/result.json": "b262dc287135c516817075cab6b29d496eba59960742948591fbb22e544e10f9", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/trial.log": "0f95f577404646395522c12e900c273f70fd0a24cf5d02ed64804ef904d3704a", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/attempts/1/ctrf.json": "1d34d0c19544a38b207c8e96ab90039e84235034e6592b38c0cf8b3164322887", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/attempts/1/test-stdout.txt": "3c7cd32794830ce4b9c28bb6303111b0f99c0808e60941882da66b594d5af41b", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/ctrf.json": "1d34d0c19544a38b207c8e96ab90039e84235034e6592b38c0cf8b3164322887", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/retry-summary.json": "ea2c4dfd756be7347645686b615c1ac6c4830fea53cca68dc21cfe149f6cfdb2", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012/regex-log__NmCmvYZ/verifier/test-stdout.txt": "3c7cd32794830ce4b9c28bb6303111b0f99c0808e60941882da66b594d5af41b", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-regex-log-main-20261012.json": "bbc06770ff4bcd59a059d326768e0db131a20b86b2f7a5cf09b9e4e7d47fa8e5", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012-input/manifest.json": "31428ecce931d904995f8540e71983fcb9abec3b919681fb741dafccaf51d16d", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012-input/config.json": "666e15363f2ad6e7482ecc180c32e1d5133e45c0efa41027f1adb58b54e73268", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-log-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "1 passed in 0.01s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 21, + "task": "terminal-bench/torch-pipeline-parallelism", + "task_ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "trial": "torch-pipeline-parallelism__Zr8yYNi", + "job": "dev34-torch-pipeline-parallelism-main-20261012", + "raw_reward": 0.0, + "classification": "scored_failure", + "exception_type": null, + "official_test_summary": { + "tests": 3, + "passed": 2, + "failed": 1, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791833686.9316525, + "stop": 1791833761.5373101 + }, + "started_at": "2026-10-12T19:15:44.928074Z", + "finished_at": "2026-10-12T19:36:25.229299Z", + "seconds": { + "environment_setup": 2.084403, + "agent_setup": 55.257321, + "agent_execution": 720.87391, + "verifier": 439.261999 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 439.01761015899683, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "time_budget_exhausted", + "recorded_agent_steps": 16, + "tool_calls": 20, + "incomplete_model_steps": [ + 33 + ], + "accounting": { + "usage_complete": false, + "cost_complete": false, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 672793, + "completion_tokens": 63889, + "cached_tokens": 668928 + }, + "estimated_known_cost_usd": 0.081839868 + }, + "task_checksum": "31e565558cd22598a3c2069f771d9d0eee26fb843b77f7b03e00ad4c336fbce9", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/agent/nanopycodeagent.txt": "6e70f72c1456a8af7b8ec4887f5a91388d943128a568c20ad1b0f32e5e0b364b", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/agent/trajectory.json": "dff3b58fb3e3f31e71bce99effb2626a16b2fed1b0118ce73715d05e807bb644", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/config.json": "ceb0998220b253e2ed3e624080c107504b85be71d111a0a824078d213be90df2", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/lock.json": "1b7203f339bde65255ca6af2d683de182c4a2e291af1e71b2e7417a36fe5e260", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/result.json": "ff8aeeb5d7dc6c19b748beda88cb1a4d6c47f7c431feea71c45465ed669c23ca", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/trial.log": "31db282f7383cae7d38cdbcc1b79d8045477679d7419a095562bdfc0150068c6", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/attempts/1/ctrf.json": "4977536344650bead5c9dc81b4f380caa585dfe084c145522d2a40e1a62314e1", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/attempts/1/test-stdout.txt": "c15cca6d1add0baf3aaac6f740be973087d869c25c5be5e698a5485f23702586", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/ctrf.json": "4977536344650bead5c9dc81b4f380caa585dfe084c145522d2a40e1a62314e1", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/retry-summary.json": "8f5c71517bce0af45f888b049dc6383f4e68e3990559d0d14fc3426d36430fa2", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012/torch-pipeline-parallelism__Zr8yYNi/verifier/test-stdout.txt": "c15cca6d1add0baf3aaac6f740be973087d869c25c5be5e698a5485f23702586", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-torch-pipeline-parallelism-main-20261012.json": "94b1602f578343904543c39f57315b8d7d3a050fd8806c3aa2792f802f6abd78", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012-input/manifest.json": "0fef56b7a41c6f13552f06d99c279ee34ec27026066a3c92a47d6f843016ff7e", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012-input/config.json": "f7c00ad0ea0a5b7b3d8a02f6883b52346fc398c429c5b18bf31a0a588f8fbabc", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-pipeline-parallelism-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 4, + "passed": 2, + "failed": 2, + "skipped": 0, + "error": 0, + "summary_line": "2 failed, 2 passed in 74.61s (0:01:14)", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + }, + "failure_analysis": { + "cause": "The submitted layer call omitted usable rotary position embeddings required by the installed Transformers API. The self-test exposed the same failure too late to repair it.", + "official_evidence": "Two structural tests passed; both world_size=1 and world_size=2 tests failed with TypeError unpacking position_embeddings=None.", + "agent_evidence": "Step 11 wrote the implementation before dependencies were available. Step 27 added a real test; step 29 hit a test-hook error. Step 31 repaired the harness and exposed the position_embeddings failure; step 33 reached the deadline.", + "atif_steps": [ + 11, + 27, + 29, + 31, + 33 + ], + "improvement_hypothesis": "Inspect the installed API and run a tiny one-rank forward/backward test early. Propagate worker and pipeline exit failures, bound test durations, and reserve time for repair before distributed checks." + } + }, + { + "order": 22, + "task": "terminal-bench/torch-tensor-parallelism", + "task_ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684", + "trial": "torch-tensor-parallelism__zyNwrLx", + "job": "dev34-torch-tensor-parallelism-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791833531.4903321, + "stop": 1791833568.976687 + }, + "started_at": "2026-10-12T19:15:44.702584Z", + "finished_at": "2026-10-12T19:33:11.489468Z", + "seconds": { + "environment_setup": 2.175414, + "agent_setup": 54.814839, + "agent_execution": 559.563603, + "verifier": 408.023715 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 407.77453560999857, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 35, + "tool_calls": 45, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 1303344, + "completion_tokens": 58753, + "cached_tokens": 1286656 + }, + "estimated_known_cost_usd": 0.083229936 + }, + "task_checksum": "d0160c1fe523063419b445f7b2974b64ff72a842ad5dc2b970d256e7629f0426", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/agent/nanopycodeagent.txt": "ce291b7ee2fb846df1e5abc42257ae418a53cf1be15e05b8653bf4d3874dd2a0", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/agent/trajectory.json": "cfdde35ed77aa9e59cf9131e0185505db3495308cd3d0027045643995c42d259", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/config.json": "2bf096d510610bfb63398b301739d81d7e3d1d245c05cb565a3860c969d1ccf2", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/lock.json": "72296efdd008528b3b1d1b09437057ba576b1815ea306fb372dfde05177884ae", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/result.json": "85f294f3b1c96dda7778fbdb84bd042db813b8b66c5f4b176da5129a363f6916", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/trial.log": "70a71e385d7d561e5b6080a93adbb68afc91fd1dec1c1ec84f2dc095b60849a2", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/attempts/1/ctrf.json": "dbbdb9d0fd35369550da55e5392b7b70eeeb61fc222599a8a23e57199a913692", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/attempts/1/test-stdout.txt": "524654fa63db03030bd0cb213aa6666103d7988b050931143dbd91e16d3f434b", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/ctrf.json": "dbbdb9d0fd35369550da55e5392b7b70eeeb61fc222599a8a23e57199a913692", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/retry-summary.json": "76b8ba1d1484ee873a409533c8eaf39b3a9d04c838a4e0fe4ed03714f54c2410", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012/torch-tensor-parallelism__zyNwrLx/verifier/test-stdout.txt": "524654fa63db03030bd0cb213aa6666103d7988b050931143dbd91e16d3f434b", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-torch-tensor-parallelism-main-20261012.json": "4c0b84e05d5c996b49a8d7cd82c975dfa5b5a5d0a8436081022df099731b7c54", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012-input/manifest.json": "94c547f141a4a0b98693c09b9f1c88bf6a63bc8a3f99a6ef6ef180401c3b99e5", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012-input/config.json": "6417b0bc61a645a24afceac5799e66fce062f6f99cc312be9373ddee3eb73253", + "jobs/dev34-baseline-main-source/jobs/dev34-torch-tensor-parallelism-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 13, + "passed": 13, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "13 passed, 1 warning in 37.49s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "observed", + "credential_observation_steps": [], + "reviewed_local_access": { + "atif_steps": [ + 3 + ], + "finding": "The agent probed /app/tests; it was absent. No local hidden verifier content was observed." + }, + "external_access": { + "atif_steps": [ + 15, + 17, + 19, + 21, + 23, + 25 + ], + "kind": "benchmark_revision_metadata", + "source_url": "https://huggingface.co/datasets/zai-org/terminal-bench-2-verified/raw/main/README.md", + "finding": "Exact task-phrase searches reached a Terminal-Bench dataset README. Step 23 returned revision metadata; step 25 fetched the README and displayed the row-parallel pre-scattered-input clarification, already present in the task instruction. No solution code or official hidden tests were observed in the returned content.", + "score_policy": "Retain the official raw pass and flag exposure separately; no clean-run claim or post-hoc replacement." + } + } + }, + { + "order": 23, + "task": "terminal-bench/write-compressor", + "task_ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e", + "trial": "write-compressor__znNddyQ", + "job": "dev34-write-compressor-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791834248.3099546, + "stop": 1791834248.4786124 + }, + "started_at": "2026-10-12T19:36:32.692827Z", + "finished_at": "2026-10-12T19:44:29.613941Z", + "seconds": { + "environment_setup": 2.405911, + "agent_setup": 44.460811, + "agent_execution": 394.275844, + "verifier": 14.440756 + }, + "work_budget_seconds": 720, + "verifier_timeout_seconds": 900.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 14.250457134003227, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 25, + "tool_calls": 26, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 1452941, + "completion_tokens": 68071, + "cached_tokens": 1445760 + }, + "estimated_known_cost_usd": 0.09251406 + }, + "task_checksum": "17ff45a3beaa04e5f08eb4f3c6136f4b56602bdcca52ade0896cfd341f374e99", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/agent/bootstrap.json": "78617809d0aabc5eaf4ecbb573bab09119e53b9c6a39a8890554717fa5823cf8", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/agent/nanopycodeagent.txt": "1dd2d09e967fb4741a52993a2e088a065d2064af0e375df35f69d27c7dc8d486", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/agent/trajectory.json": "de6f22093d839eb98bb87a7244f073efbe896c55dccafe830aeade3e95faf8ca", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/config.json": "086a233dbae1fa2a966da96e418c8eba433e550c9aa7358e713e02855e847667", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/lock.json": "c57bb4fe3586d5bb089a19247d76741e235bf333cea08473aa96fc49848e0773", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/result.json": "6d7ae2b32f15abac90d962503ee3717ce4505c81f7ba08c1767b799257ed616c", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/trial.log": "a522af380f4c4a0ccc2d04ea0249b4ad6c27c8e387b1f5abaafe3ddddd2e9835", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/attempts/1/ctrf.json": "247608b6e85713f032d5e9f860303ce921698c8d75df2e3290d910168bd5c6af", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/attempts/1/test-stdout.txt": "3cb1542a661a9997d1837f73d257e7b0b47031883cb3faff8378eb7470f640a2", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/ctrf.json": "247608b6e85713f032d5e9f860303ce921698c8d75df2e3290d910168bd5c6af", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/retry-summary.json": "806c9fdb27880b4388b13d4a1875bea5dff3d2ae636c49c3ce5931121ef5f2b2", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012/write-compressor__znNddyQ/verifier/test-stdout.txt": "3cb1542a661a9997d1837f73d257e7b0b47031883cb3faff8378eb7470f640a2", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-write-compressor-main-20261012.json": "e9f51178f35fdcc16bd146a1a59498bcca15f994a363cb6b708523729689b7ed", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012-input/manifest.json": "5503ac1c40a8453bb1de4120781e989f4aaf002537d55ac4549f7d9bd7371224", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012-input/config.json": "645c7c14ebd771f775e50d88c4b8d7d7b15846d25ab01050b8f286dc2b4122cb", + "jobs/dev34-baseline-main-source/jobs/dev34-write-compressor-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "3 passed in 0.17s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 24, + "task": "terminal-bench/constraints-scheduling", + "task_ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53", + "trial": "constraints-scheduling__LyFmzLj", + "job": "dev34-constraints-scheduling-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791833894.5543544, + "stop": 1791833894.908104 + }, + "started_at": "2026-10-12T19:36:32.991459Z", + "finished_at": "2026-10-12T19:38:36.004003Z", + "seconds": { + "environment_setup": 2.106804, + "agent_setup": 50.431794, + "agent_execution": 31.985042, + "verifier": 17.228763 + }, + "work_budget_seconds": 1020, + "verifier_timeout_seconds": 1200.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 17.00502801899711, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 5, + "tool_calls": 4, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 32045, + "completion_tokens": 5694, + "cached_tokens": 29184 + }, + "estimated_known_cost_usd": 0.007866204 + }, + "task_checksum": "5b311f974587adf67804532222c59395f41a6c984b54a5e96a56dd071ee801dc", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/agent/bootstrap.json": "8ec0c76eef3467924ce96d81cc262bdea7775b259bbeeca2a0563a292696eb05", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/agent/nanopycodeagent.txt": "da2ed1376689260fc56fbe8a0e39d5b7fb42e7cde432cb64183fec7a608105b1", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/agent/trajectory.json": "82dee44641b632a60a502ad8453b0997b7c91033f0aad87a50188578c7428f5e", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/config.json": "975a6e129536caf465f51a6ceebf36cefd35a8bdd0da265accb04923dd890533", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/lock.json": "276fd9bb0c189ad1bc828b0e1fe1fda5f578acb4a789f91043f64b4404a7d1f8", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/result.json": "674a5034a06448db2ee18c6046a85f860d0d15db50e55807e8b37cbb8b8d238b", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/trial.log": "eb67fa42a4fc408bf98121550743bbc3ee9c7611e4f699f1faed3bedc2411cd4", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/attempts/1/ctrf.json": "d6e10646c6d7bf0221c9c5341212c805f0a8c947665493fdca2e57f10bc83c97", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/attempts/1/test-stdout.txt": "c3bb6c75a50ce8e859989095cf3b5068a66fe5ded8e42bcdc92a74c3af270f70", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/ctrf.json": "d6e10646c6d7bf0221c9c5341212c805f0a8c947665493fdca2e57f10bc83c97", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/retry-summary.json": "51e5d4a3e03197b0410889d693406e9038bb176b0b3034f800c893170ff0b46f", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012/constraints-scheduling__LyFmzLj/verifier/test-stdout.txt": "c3bb6c75a50ce8e859989095cf3b5068a66fe5ded8e42bcdc92a74c3af270f70", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-constraints-scheduling-main-20261012.json": "7ee2b4b27af74185d2db9ee72026a97e8428f43b9a814ccc98f803fa4f097551", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012-input/manifest.json": "e6c7326d6b0b8461001ec7feb6514e894111dec073fc739ecbe3920c65b4aeae", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012-input/config.json": "ddb65353459fea46fea6f5f7cc46beb905b07cd7d643bea731bc088000680e0a", + "jobs/dev34-baseline-main-source/jobs/dev34-constraints-scheduling-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "3 passed in 0.35s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 25, + "task": "terminal-bench/custom-memory-heap-crash", + "task_ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e", + "trial": "custom-memory-heap-crash__fjqXaKf", + "job": "dev34-custom-memory-heap-crash-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791834584.7839317, + "stop": 1791834588.3243234 + }, + "started_at": "2026-10-12T19:44:36.052345Z", + "finished_at": "2026-10-12T19:50:09.328682Z", + "seconds": { + "environment_setup": 2.194168, + "agent_setup": 46.088412, + "agent_execution": 245.337894, + "verifier": 18.488955 + }, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 18.250786581003922, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 30, + "tool_calls": 41, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 810798, + "completion_tokens": 32860, + "cached_tokens": 790400 + }, + "estimated_known_cost_usd": 0.0502938 + }, + "task_checksum": "a0db695f342c1b72f1b3d0b6c234eb802e970c4758931355e4ceeea4c9e73c37", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/agent/bootstrap.json": "cf6bb84c42557ba2a7b7e0ec653bbc1c7cf86c236d1b7e2a6852918dfb8eb7dc", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/agent/nanopycodeagent.txt": "1d691418241803bf49b351d5c515aa3f0caaede64af83938188febfe38e8a22f", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/agent/trajectory.json": "eb311202739c38a935a3d67621c75ab32a4c9287b76cb23e2a337b308f58555f", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/config.json": "956182b291e7064a2d10f5a643e393fd4e7ae07c644f964620e8bce28199baf4", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/lock.json": "b6724e58d36c9a4e15ab496c32ab06aad489da0ea54ad134decb2aa7cecbc41c", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/result.json": "2fd0df6ed0f13010b75c18c84b6fc35db117625d83b12ae4bc0a40b7869931dc", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/trial.log": "1aa3041d5b08d80a9c09299e6f4840f29fcc4d84f81223564e604215ae33854a", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/attempts/1/ctrf.json": "cacd14b02641488fd8c81cc7ce5c694803fc5451bd5193ccbbff0a0101dfd624", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/attempts/1/test-stdout.txt": "f06064c7a95100c8b4025ef8ae7fbb2a9fc627a0fbec0205a2b371ce683b7268", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/ctrf.json": "cacd14b02641488fd8c81cc7ce5c694803fc5451bd5193ccbbff0a0101dfd624", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/retry-summary.json": "9c383a7aedf097530f6995507834e8abb7394e76015c26d81c940909c4681d52", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012/custom-memory-heap-crash__fjqXaKf/verifier/test-stdout.txt": "f06064c7a95100c8b4025ef8ae7fbb2a9fc627a0fbec0205a2b371ce683b7268", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-custom-memory-heap-crash-main-20261012.json": "307270949dadfcca82f611616f2b004483973466f0c46ab50ebfbfbc55f9417f", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012-input/manifest.json": "6fdf9c1bea068268338bb7462d982d91c51d7406b33d634046641a6d5d99e28c", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012-input/config.json": "85c2160bd2eabb6fe09b3a2096ec620957bea5791dccb8dfc22228452d335478", + "jobs/dev34-baseline-main-source/jobs/dev34-custom-memory-heap-crash-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "6 passed in 3.54s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [], + "reviewed_local_access": { + "atif_steps": [ + 15, + 39, + 53 + ], + "finding": "Benchmark references were the existing source canary preserved in rewritten user.cpp, not external answer retrieval." + } + } + }, + { + "order": 26, + "task": "terminal-bench/dna-assembly", + "task_ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f", + "trial": "dna-assembly__mRFMZDo", + "job": "dev34-dna-assembly-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791835500.3434703, + "stop": 1791835500.368095 + }, + "started_at": "2026-10-12T19:44:36.088493Z", + "finished_at": "2026-10-12T20:05:21.423836Z", + "seconds": { + "environment_setup": 2.157788, + "agent_setup": 44.338329, + "agent_execution": 1162.683039, + "verifier": 14.891571 + }, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 14.695995834001224, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 36, + "tool_calls": 44, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 5106013, + "completion_tokens": 208700, + "cached_tokens": 5040128 + }, + "estimated_known_cost_usd": 0.300446268 + }, + "task_checksum": "e2acbc34aa5f20bbe3b10d2d297cdec9e6f2086c94a70068f2325e7bb7ab1212", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/agent/bootstrap.json": "bbb1bebad6a2bcb02e064f6b8207cb9f0f39fce4b75c71fe3ae1d05a79e8d26b", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/agent/nanopycodeagent.txt": "7e1c56f8a4e2c99ddb1e88f8828d138ecc8f640a186d470e6e66fd97e3bf3a45", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/agent/trajectory.json": "3cfa4a57f7e84c83f9ccafc13bd27c0f3ccbd3e2b01d9ad10cd955fe9a2167df", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/config.json": "720c3c1cce90ea23dae90a63b676ef0cf15a55613bc27669b54e13ed622c6ec8", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/lock.json": "c1f3d8b5cde40654903c420408d4ba50384166ee21060d2ca169f444ad10e267", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/result.json": "44823a4c38d60cd831138c4ace5847673222a215ee3da864eb78170b89d0417e", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/trial.log": "a44b7a28ba28f5b446894c33863d4d4f3fd90e736e2b43cbd87861ca42aca045", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/attempts/1/ctrf.json": "f461e603ae7a442d7b97b937675bd42d92adf829695d7517a87cebf3e3d3cedd", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/attempts/1/test-stdout.txt": "724868261581dfd8b68d54e5d651d1c8b31afe4d85de06883d0cb40bb3e1e9a8", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/ctrf.json": "f461e603ae7a442d7b97b937675bd42d92adf829695d7517a87cebf3e3d3cedd", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/retry-summary.json": "a39ca3d19b542a2869ad624f7270aff3f7798e7255e1dc114cd4a04d2dc534aa", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012/dna-assembly__mRFMZDo/verifier/test-stdout.txt": "724868261581dfd8b68d54e5d651d1c8b31afe4d85de06883d0cb40bb3e1e9a8", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-dna-assembly-main-20261012.json": "7ec090f2433715b89ba9c581a89937d1dc89702e7ae12946c17fbb2d5155d011", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012-input/manifest.json": "ceb7f68039277f46c292651f3fc090ba1d86216dabef5ff49a213cec2d03bcf7", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012-input/config.json": "f4d2674d81190a904c1af88caf5c5730eecb82450730a35ad1c9bacfa11b664b", + "jobs/dev34-baseline-main-source/jobs/dev34-dna-assembly-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "1 passed in 0.02s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 27, + "task": "terminal-bench/llm-inference-batching-scheduler", + "task_ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67", + "trial": "llm-inference-batching-scheduler__AvvLYtM", + "job": "dev34-llm-inference-batching-scheduler-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791836192.134679, + "stop": 1791836192.1871674 + }, + "started_at": "2026-10-12T20:05:29.162969Z", + "finished_at": "2026-10-12T20:16:53.143708Z", + "seconds": { + "environment_setup": 2.2215, + "agent_setup": 32.966653, + "agent_execution": 614.543509, + "verifier": 13.098163 + }, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 12.895493742995313, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 28, + "tool_calls": 29, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 1307336, + "completion_tokens": 58416, + "cached_tokens": 1291264 + }, + "estimated_known_cost_usd": 0.082668384 + }, + "task_checksum": "c19d5af0dc6e54b45f66e3d8f25bfa783c51df3e1844253aa60c20ca0a827046", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/agent/nanopycodeagent.txt": "13696564ad5b563555594cdddddc7ba42c928901f45952dcd5d06c9199767c36", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/agent/trajectory.json": "fb774af094770aa2b2a105029319accb9e7850f88e2c85722c6de6ff5f45c246", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/config.json": "6de7527f665c887c8bc896eddeb556c7998ff7d98a8f70a37bc006dfe2459596", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/lock.json": "78a7340661de767356dc1b6bc12c2ca22fa28efb0a848b7349d1968c285acde3", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/result.json": "a35d23a179153f9ddc227f19310cfca1e6c603671bff397da2266ebdd76d9b4f", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/trial.log": "f8995b4e61627e44b457074daadd282b14b02ac820268b149a2df0b92a01f712", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/attempts/1/ctrf.json": "67c9af94662e1a5e1f3a4e7055b0f7804bc7258ac820ee7fbe946d90e6207bf5", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/attempts/1/test-stdout.txt": "6a580044057c6448fbedead4eb0e589a6ba9bb45ba53b7e53521b43ccfebd4ab", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/ctrf.json": "67c9af94662e1a5e1f3a4e7055b0f7804bc7258ac820ee7fbe946d90e6207bf5", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/retry-summary.json": "be3106a6486c29876fdb7f5914e23d3b4ba4b3e0e5866a1532b33ef208267c45", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012/llm-inference-batching-scheduler__AvvLYtM/verifier/test-stdout.txt": "6a580044057c6448fbedead4eb0e589a6ba9bb45ba53b7e53521b43ccfebd4ab", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-llm-inference-batching-scheduler-main-20261012.json": "bb45a2ddb16fa5142bc3599915decd1f523d24f340c3a5c28fb5d7b9cc3c25f3", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012-input/manifest.json": "a5c7476c202b8ddad85043d329b7b047c833beec1509f247f80237e37a7c90d7", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012-input/config.json": "3ef8115f79410deb8990618af5a5fa17b3eee6f573119da17326adba2f4dde55", + "jobs/dev34-baseline-main-source/jobs/dev34-llm-inference-batching-scheduler-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "6 passed in 0.05s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 28, + "task": "terminal-bench/path-tracing", + "task_ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c", + "trial": "path-tracing__LMykXqk", + "job": "dev34-path-tracing-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 5, + "passed": 5, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791835847.5987318, + "stop": 1791835857.9740386 + }, + "started_at": "2026-10-12T20:05:29.125963Z", + "finished_at": "2026-10-12T20:11:19.706262Z", + "seconds": { + "environment_setup": 2.218156, + "agent_setup": 39.742327, + "agent_execution": 259.882544, + "verifier": 27.491944 + }, + "work_budget_seconds": 1620, + "verifier_timeout_seconds": 1800.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 27.264476625001407, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 29, + "tool_calls": 34, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 1362598, + "completion_tokens": 36685, + "cached_tokens": 1319678 + }, + "estimated_known_cost_usd": 0.064816068 + }, + "task_checksum": "549aa348dde50ec3824205f402d9686da5d1dedf65ae2bd2ae8806513839fe93", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/agent/bootstrap.json": "f7c768324bf6e26e7f52bdec891623e4fc98e29f1f24ca02e3847723d8cb55bb", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/agent/nanopycodeagent.txt": "7dba04bb9efd7a6aa02e26e46024cf2b9286e614264061242951404f579bd969", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/agent/trajectory.json": "9a7054d0e81f0ee38e3d5e4c57d31a16ac2737a1a98f6f5c7b19a8b706ac01a4", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/config.json": "430c7f3928db3f3e16a19087a2e972c8f02e9d526cb0e448d605eccbc078a63e", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/lock.json": "e6caebcbb23e43b16173b7e3a09c926529c2088cbbc4c73e5117c661db0727c6", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/result.json": "8dbd3fa2cc44219b21c218cbd458fd6f5e0096343f2fe29e9a3fdd24d15b16f7", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/trial.log": "1470af8c2a19507dc9913b16de8083222aeaa7554c007743a3df95a7507290de", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/attempts/1/ctrf.json": "121ac3d70e6b4aca6d73d6b9b10fbdbcabe5f0aaeb4a012dfcd9dc8f83918a5d", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/attempts/1/test-stdout.txt": "2bc5a286c9aeccf54c38a581676c5212e12fc57ec97a63ba9f061c43a5a5e469", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/ctrf.json": "121ac3d70e6b4aca6d73d6b9b10fbdbcabe5f0aaeb4a012dfcd9dc8f83918a5d", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/retry-summary.json": "46126206f4f9d1ce74fd9988fb8d65700d55498cc73f132087d9d1e5b1a8fe47", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012/path-tracing__LMykXqk/verifier/test-stdout.txt": "2bc5a286c9aeccf54c38a581676c5212e12fc57ec97a63ba9f061c43a5a5e469", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-path-tracing-main-20261012.json": "beeb9a5f2fdfbd5dcf2388ba140e6f43e4a230acd75d08406e01c3e8f8f2df5a", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012-input/manifest.json": "70aeaf30fe65b6f5cd3b61e76e7841e92b1341fe973df9b6fa9798ef4ad099ca", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012-input/config.json": "9ad1788116481be46fc552c3c5cd3a69d7fa9408a2cff95600af7ff9dcae325b", + "jobs/dev34-baseline-main-source/jobs/dev34-path-tracing-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 5, + "passed": 5, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "5 passed in 10.37s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 29, + "task": "terminal-bench/schemelike-metacircular-eval", + "task_ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6", + "trial": "schemelike-metacircular-eval__wpWP6A5", + "job": "dev34-schemelike-metacircular-eval-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791838043.8282669, + "stop": 1791838098.886628 + }, + "started_at": "2026-10-12T20:17:00.598610Z", + "finished_at": "2026-10-12T20:48:39.961600Z", + "seconds": { + "environment_setup": 2.129193, + "agent_setup": 30.905831, + "agent_execution": 1777.106361, + "verifier": 67.848556 + }, + "work_budget_seconds": 2220, + "verifier_timeout_seconds": 2400.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 67.56629851100297, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 77, + "tool_calls": 82, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 11431862, + "completion_tokens": 173511, + "cached_tokens": 11383168 + }, + "estimated_known_cost_usd": 0.291120408 + }, + "task_checksum": "e4525988ae20585b6b90a2f1af2fcc9d18d6363d3cfcdd7be5305199428883d2", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/agent/nanopycodeagent.txt": "ab95bbea507c1a35d48d97c479130608e5f6274bd16ba47278285087b6430e7c", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/agent/trajectory.json": "d944b83ab3e05e799c333b2b091949e99393c33be21c7106981dccaf64fd061e", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/config.json": "2e5dc6251620f258a13d21f946e1c77ad2a8107caa02fd331fde78355b332b14", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/lock.json": "1688ca2fc9bf8552ea287566f840a5bed16f1f0191f0f7c0be2c70b9ef9e0391", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/result.json": "9a4a2b210149b9930c62f3b713e900fe4c42e35fb765cc487831a7625667521e", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/trial.log": "c30bbeaeb87b548b737fa1287407ca8323e3f31b6c8cb8ac3750ac4bbb88f92c", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/attempts/1/ctrf.json": "e8d09bba9fca81d7bb88aa3e2bc201aa3d449cbc5fadda69d6cf318b2999848f", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/attempts/1/test-stdout.txt": "c8baf368ddf26533f5e2b003c41108730834d14daa3250fdb239657d4fadb5d2", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/ctrf.json": "e8d09bba9fca81d7bb88aa3e2bc201aa3d449cbc5fadda69d6cf318b2999848f", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/retry-summary.json": "3da4843558dc344c5aca75afadfb6c85a2753ee5bcd2e96c78d2690845ea5a5b", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012/schemelike-metacircular-eval__wpWP6A5/verifier/test-stdout.txt": "c8baf368ddf26533f5e2b003c41108730834d14daa3250fdb239657d4fadb5d2", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-schemelike-metacircular-eval-main-20261012.json": "5e800b0a0e8315bee49df78a5c0328f471421643a34873b5567f500929d7d90a", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012-input/manifest.json": "5d8b8c41efee9504382f723cde3913e4ed0b1988a29f9a4d09e14c9dd2eb698b", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012-input/config.json": "ee675efc44640619137ca6ec97bd01f6e0626dc429588f7dc2710cb6ecc7d9f8", + "jobs/dev34-baseline-main-source/jobs/dev34-schemelike-metacircular-eval-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 1, + "passed": 1, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "1 passed in 55.06s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 30, + "task": "terminal-bench/caffe-cifar-10", + "task_ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf", + "trial": "caffe-cifar-10__Pw7u7Uu", + "job": "dev34-caffe-cifar-10-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791838720.706259, + "stop": 1791838759.4385488 + }, + "started_at": "2026-10-12T20:17:00.798366Z", + "finished_at": "2026-10-12T20:59:42.993886Z", + "seconds": { + "environment_setup": 2.095127, + "agent_setup": 34.850014, + "agent_execution": 2448.911944, + "verifier": 52.575337 + }, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 1200.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 52.33102223699825, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 73, + "tool_calls": 93, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 3820979, + "completion_tokens": 60421, + "cached_tokens": 3771264 + }, + "estimated_known_cost_usd": 0.110047284 + }, + "task_checksum": "38da84d32a37ce45b201f202f0deb320dd80b5482afbcf66ba8456c8ea814c69", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/agent/bootstrap.json": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/agent/nanopycodeagent.txt": "2d2352f7d0eea8d87c6301cc1acb06b4310c7c0ad6c105a4aa2dcf3b3768409f", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/agent/trajectory.json": "6af6d661da6557cdaf2c53a82988069529a856346483272ad00520e110595963", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/config.json": "6f9108f57ef992e13e6463bd4e678b7be7328110ea5b6f7ea3de749d7ddd250a", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/lock.json": "b1d31159d29ed8aa08d521b32285e0ede410a112c947587e1941a3c5815c9766", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/result.json": "d8107ddba84973ef8516febb5dc36af05186fcb5659acc33fc3f6a73081849cf", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/trial.log": "563494c998c997281b24399d137cd376dd0b04aa71fc99d91a0fdeda7f71759f", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/attempts/1/ctrf.json": "c4dcede6e70e7d407abe0e2b1f4eb2e800a7c73529f0b77b39d9d5bf6d67e0fb", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/attempts/1/test-stdout.txt": "48545966dc37a4a974d48c4c54b3009a8d8c5f37db9f96f6395b4318b7079d13", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/ctrf.json": "c4dcede6e70e7d407abe0e2b1f4eb2e800a7c73529f0b77b39d9d5bf6d67e0fb", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/retry-summary.json": "c48a6ecf2d2350e11f402ec92f08e6f45526e323748877771012ce0a53fb9350", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012/caffe-cifar-10__Pw7u7Uu/verifier/test-stdout.txt": "48545966dc37a4a974d48c4c54b3009a8d8c5f37db9f96f6395b4318b7079d13", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-caffe-cifar-10-main-20261012.json": "4ec390e69d840cd7113a639dbd8c53b7b19d34ae5bf2198c5eebb1e02eec95c4", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012-input/manifest.json": "5cba97421b492ea72a930452bfd399e54f87c13359ce35c83490b02448e2aba9", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012-input/config.json": "98dd5a75f67738422281d2cf707d0aa448a9afa2284452cf69e87728031310a4", + "jobs/dev34-baseline-main-source/jobs/dev34-caffe-cifar-10-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 6, + "passed": 6, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "6 passed in 38.73s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 31, + "task": "terminal-bench/circuit-fibsqrt", + "task_ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3", + "trial": "circuit-fibsqrt__JQ9zU3H", + "job": "dev34-circuit-fibsqrt-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791839446.5839653, + "stop": 1791839446.9585075 + }, + "started_at": "2026-10-12T20:59:49.982155Z", + "finished_at": "2026-10-12T21:11:07.895519Z", + "seconds": { + "environment_setup": 2.22626, + "agent_setup": 31.09974, + "agent_execution": 610.671386, + "verifier": 12.738614 + }, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 12.54025770699809, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 36, + "tool_calls": 36, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 2570971, + "completion_tokens": 88335, + "cached_tokens": 2541696 + }, + "estimated_known_cost_usd": 0.130034676 + }, + "task_checksum": "dc4c18df98c940d9ec11336861e07f7f5ca2a13b9e8e0f95e886c20f3d61bc0d", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/agent/bootstrap.json": "1812b1d7452b4ec930ab65be85cd8ba7f2e96342153c5e9e7d0dc948f342502f", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/agent/nanopycodeagent.txt": "f4fa6636f9afb1442daadafd3c9f35de11b3c5f4fc3cf122bfaad9e786455930", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/agent/trajectory.json": "8387b2396f17131af58429660cb9a4e2f84e63d64915357e0fcdc67aa8bea377", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/config.json": "36c1e704b91f06543183d163096a88a5968ab11c56adc3f1f1a580f3788f3ad3", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/lock.json": "0bb71a5952d924995f5ffd262b1ef0d576d8b46de6270172382f9668ca6c7b0f", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/result.json": "dc9e6a340ee75ad1618cd257a407380bfcd9b8932aa9c0f71de3115d2df22e10", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/trial.log": "08c338576a42fed76b69572c198eddb1fce0db892fb95256937d0ced2f6c89a5", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/attempts/1/ctrf.json": "1e24704cd3410f4aab1eb463cb7368ed956e1475b7f241130164a958c70b1662", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/attempts/1/test-stdout.txt": "056af043e94d6bd337c2a488ab7f8783d0a180d8f2eed3628a4c273596770645", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/ctrf.json": "1e24704cd3410f4aab1eb463cb7368ed956e1475b7f241130164a958c70b1662", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/retry-summary.json": "e2b3b469bc1150219db9cd2074737653da6318117f1602f8f99d4b6f3d52125b", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012/circuit-fibsqrt__JQ9zU3H/verifier/test-stdout.txt": "056af043e94d6bd337c2a488ab7f8783d0a180d8f2eed3628a4c273596770645", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-circuit-fibsqrt-main-20261012.json": "4a6e1e3752e88e7f38dfc9f59e51f95c96625a8fde59daf3b129db7520aa2a6a", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012-input/manifest.json": "7d1ee3b0b18e85f5304b9fe7358cff78f193884fc597bc450a088b4987eaca97", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012-input/config.json": "0667c3ff1c98ffe4435414776eb2509a4886ca54547289ca6f9e0d7642cf6d51", + "jobs/dev34-baseline-main-source/jobs/dev34-circuit-fibsqrt-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "3 passed in 0.37s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 32, + "task": "terminal-bench/mteb-leaderboard", + "task_ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c", + "trial": "mteb-leaderboard__VD8EX2H", + "job": "dev34-mteb-leaderboard-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 2, + "passed": 2, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791840218.1346626, + "stop": 1791840218.1500773 + }, + "started_at": "2026-10-12T20:59:50.007297Z", + "finished_at": "2026-10-12T21:24:06.475871Z", + "seconds": { + "environment_setup": 2.253718, + "agent_setup": 33.302559, + "agent_execution": 1380.35745, + "verifier": 12.001121 + }, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 11.79078324699367, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 79, + "tool_calls": 111, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 5224883, + "completion_tokens": 59715, + "cached_tokens": 5148288 + }, + "estimated_known_cost_usd": 0.125526228 + }, + "task_checksum": "f4c2fa1f189121ba4a0f654a270b92bfa2dad319c3315d89184f71c5c346a497", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/agent/bootstrap.json": "7c324cb3af6adf375e27d5cefcc67e8631dae65ac0f3c52687f893fa69b96c4e", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/agent/nanopycodeagent.txt": "e0676a8e5570b5c0fb2bd989db783eac7b37f9a79e1eb21f8dcbcf6f46158dae", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/agent/trajectory.json": "7e9ac54b565926862d872da405ae7fb7ae91323718990d8081b721a0ef52f775", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/config.json": "d4281b3548cfeea355d2863a5dd937c121784a9770c05f0c4ad67f12c6e9dd46", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/lock.json": "5889a360552aa157819edf18c47d3dcbfb72506ea0e76da1982ad05632decfd7", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/result.json": "a56d61d05610b7314f1d51546e95ae9c386d718540321f8f6192171f33417f08", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/trial.log": "159b2ca5186d5862ae323f104eff755bb2018386c5f6d5f0fc2af49695915e8b", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/attempts/1/ctrf.json": "ff1b665a1eb3af71d74fc1ac2a70d3134a23073b48163549c8a7666080c1112a", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/attempts/1/test-stdout.txt": "684d4a2cee0edf5e75baad28bf20e67bb2fcf45a1fac503f1f1faaa21c770a00", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/ctrf.json": "ff1b665a1eb3af71d74fc1ac2a70d3134a23073b48163549c8a7666080c1112a", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/retry-summary.json": "086d38a33540dcf16467a587d8e956ecd02c3ac6d9eb5cc4b9bd903c044e2412", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012/mteb-leaderboard__VD8EX2H/verifier/test-stdout.txt": "684d4a2cee0edf5e75baad28bf20e67bb2fcf45a1fac503f1f1faaa21c770a00", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-mteb-leaderboard-main-20261012.json": "78b955d930adde2127dd6cb63c6f154a30c03ecea1de8efa5cb0b9148db49b3d", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012-input/manifest.json": "826618237df0e667bff110b312b6d2817eccd20754d4d42e57ff4ea123df57ef", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012-input/config.json": "0a8ba9aeca7e3681b79c8c575603fe09518baea436f957e1a709742fd5f348e2", + "jobs/dev34-baseline-main-source/jobs/dev34-mteb-leaderboard-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 2, + "passed": 2, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "2 passed in 0.01s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + }, + { + "order": 33, + "task": "terminal-bench/regex-chess", + "task_ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78", + "trial": "regex-chess__g85Ci69", + "job": "dev34-regex-chess-main-20261012", + "raw_reward": 0.0, + "classification": "scored_failure", + "exception_type": null, + "official_test_summary": { + "tests": 4, + "passed": 2, + "failed": 2, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791843122.0810905, + "stop": 1791843140.8449337 + }, + "started_at": "2026-10-12T21:24:14.031223Z", + "finished_at": "2026-10-12T22:12:41.800871Z", + "seconds": { + "environment_setup": 2.083704, + "agent_setup": 31.724062, + "agent_execution": 2818.848399, + "verifier": 33.973729 + }, + "work_budget_seconds": 3420, + "verifier_timeout_seconds": 3600.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 33.77848868598812, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 82, + "tool_calls": 86, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 12529777, + "completion_tokens": 171249, + "cached_tokens": 12429952 + }, + "estimated_known_cost_usd": 0.310026012 + }, + "task_checksum": "64993ab339d437cea9397a033e825663a77d5527a42ef3c2fb2ad25133dcbd4e", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/agent/bootstrap.json": "9abae99e4800f37de01cc8c2eb9d13d0437c60d0c1c0724f648d551c7f382b77", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/agent/nanopycodeagent.txt": "de5df9339de05415c456003db9f7e1a64d1aa1bb7ec26d88f7c78d7147e03714", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/agent/trajectory.json": "05865541b5a961921d5ddfce1fca76aa52d3774ebf154f03b1c20f3bedc42c02", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/config.json": "9952061ab6b5aab855cb20621194760be8e96c8565e31f90a5d8bb6664fb31bc", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/lock.json": "43ff96727a335229fc8f7cc32e372cadd4d201b540cb68e6ae5dd68a33ccbc80", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/result.json": "64dd8b01b2deef4b9dcd70b351a15515e3d940458b1246dac9c9d5729765e0bd", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/trial.log": "08cd3c2af5895f282d5cee4a7f3abfa1082dec87f5b7b4a04226f6d828bf1111", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/attempts/1/ctrf.json": "3fae26a1ae3b0a38e5dee9d71ba447e02538a4d806751dfa531e7472ba00bc42", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/attempts/1/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/attempts/1/test-stdout.txt": "5791480e7b0bb6088d3c03e25946254593b37cf0820d26fa1a1a13d54f87a275", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/ctrf.json": "3fae26a1ae3b0a38e5dee9d71ba447e02538a4d806751dfa531e7472ba00bc42", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/retry-summary.json": "d13938c17cb1fdc2b5c47580deb0a5f660a6e7731826b62092ab3cb25e0aa6e2", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/reward.txt": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012/regex-chess__g85Ci69/verifier/test-stdout.txt": "5791480e7b0bb6088d3c03e25946254593b37cf0820d26fa1a1a13d54f87a275", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-regex-chess-main-20261012.json": "75f77b46dd71a21942625d0aef383801fa2b49e81e71b4438c551e05059fa2c7", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012-input/manifest.json": "950c73bb619248caf4afccd262c76dc34b91d68dd4591ffe642650f94cea0c58", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012-input/config.json": "b130e5de68a1ff0e81853c322ac9169219d43da4d6005756fa42fb23bf17c60c", + "jobs/dev34-baseline-main-source/jobs/dev34-regex-chess-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 4, + "passed": 2, + "failed": 2, + "skipped": 0, + "error": 0, + "summary_line": "2 failed, 2 passed in 18.76s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + }, + "failure_analysis": { + "cause": "Castling intermediate states survived the first attack filter, allowing white to castle while already in check.", + "official_evidence": "Two tests passed and two game suites failed, each with an extra white kingside castling move from an initially checked king.", + "agent_evidence": "Steps 107, 115, and 117 adjusted castling generation/filter order but left X0-to-X1 creation ahead of the first attack filter. Final independent python-chess comparisons covered 618 plus 600 sampled positions without finding the two official counterexamples. Step 165 nevertheless claimed castling out of check was excluded.", + "atif_steps": [ + 107, + 115, + 117, + 141, + 143, + 145, + 147, + 151, + 153, + 155, + 157, + 163, + 165 + ], + "improvement_hypothesis": "Map each special-rule precondition to an explicit negative case, especially castling out of, through, or into check; keep targeted regressions for every filter-order change.", + "limitation": "A genuinely independent oracle and more than 1,000 sampled positions were used. Broad random coverage still missed a required negative condition." + } + }, + { + "order": 34, + "task": "terminal-bench/build-pov-ray", + "task_ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb", + "trial": "build-pov-ray__SyyjSUB", + "job": "dev34-build-pov-ray-main-20261012", + "raw_reward": 1.0, + "classification": "scored_pass", + "exception_type": null, + "official_test_summary": { + "tests": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0, + "start": 1791840651.7265368, + "stop": 1791840657.432628 + }, + "started_at": "2026-10-12T21:24:13.971286Z", + "finished_at": "2026-10-12T21:31:18.553016Z", + "seconds": { + "environment_setup": 2.162618, + "agent_setup": 37.449986, + "agent_execution": 338.060282, + "verifier": 25.661377 + }, + "work_budget_seconds": 11820, + "verifier_timeout_seconds": 12000.0, + "verifier_attempts": [ + { + "attempt": 1, + "status": "completed", + "duration_seconds": 25.46187567799643, + "return_code": 0 + } + ], + "verifier_dependency_profile": { + "profile": null, + "status": "not_configured" + }, + "bootstrap_status": "ready", + "model_started": true, + "trajectory_terminal_outcome": "completed", + "recorded_agent_steps": 44, + "tool_calls": 67, + "incomplete_model_steps": [], + "accounting": { + "usage_complete": true, + "cost_complete": true, + "cost_is_estimated": true, + "recorded_tokens": { + "prompt_tokens": 1469056, + "completion_tokens": 28261, + "cached_tokens": 1433856 + }, + "estimated_known_cost_usd": 0.053076336 + }, + "task_checksum": "3426fb4714b5bb0463cc30bd872712a2400b23934b135c40ef7f88b27da4cd7a", + "source_ref": "2eb99b608dfb39464d17d109a4a8f11afda607a9", + "source_sha256": { + "nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1", + "nanopycodeagent/agent.py": "aa58960803be77797c2addf65d7cc785a4be54f40ad58d53b9eea9fca1cf4b56", + "nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270", + "nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f", + "nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932", + "nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70", + "nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f", + "nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d", + "nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814", + "nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3", + "nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e", + "nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04", + "nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5", + "nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe", + "nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206", + "harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b", + "harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d", + "harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c", + "harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02", + "harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2", + "harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493" + }, + "wheel_sha256": "f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350", + "provenance_checks_passed": true, + "evidence_sha256": { + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/agent/bootstrap.json": "40256d355dfe43492278a1a4a22b8a6093657509086254ddd27494162c85b0b4", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/agent/nanopycodeagent.txt": "4ba28a554b57420903cc853fc8fa514598cebf504b7e374e589beb4476a0f74d", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/agent/trajectory.json": "9d6250cc4dfe75686648ee8e3abe158db9b2e19d43f951d1dc0200e346d20e94", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/artifacts/manifest.json": "c1db653f487fb773c4f9247162ab3314e7f0584a486914b9874c4b3d6949e42d", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/config.json": "e059326fbbcc3bae7f3c1ddd099932269faa2cc23e241325926166bf9231822c", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/lock.json": "93bbbe4728d15e476862227556251297f6cedbfb3d488ea3a646e473c5691e7e", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/result.json": "482e91d41b666e3fa5305f2b6462ea4f30fd9c18447939ff2ab9de795e12c4e1", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/trial.log": "79218ae6a2ebe895e3db9447de508d896073c8ff5722b5fc1af964ae9b9ee525", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/attempts/1/ctrf.json": "f764c97cf5bb402548ec90b91837fad796d71f53a1404483506065a83570e66e", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/attempts/1/test-stdout.txt": "a98b4aea581b17d7ed32bbe6553563adc28e7c5693990631972d1a3e2206449a", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/ctrf.json": "f764c97cf5bb402548ec90b91837fad796d71f53a1404483506065a83570e66e", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/retry-summary.json": "a9c65509b2740af979f122272f6806061a6da5b4cbbc705e015d9fb056e94e9b", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012/build-pov-ray__SyyjSUB/verifier/test-stdout.txt": "a98b4aea581b17d7ed32bbe6553563adc28e7c5693990631972d1a3e2206449a", + "jobs/dev34-main-baseline-20261012-v5-orchestration/dev34-build-pov-ray-main-20261012.json": "87d83e078c1e3e24b5c5272ce5fa625bfa427d8cf7a5b09af90ed8f6fde6474f", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012-input/manifest.json": "c2e76aa2b0fa88a3a9169af6cf468cbb24f8e9c0e6fc5e7b72dc9865613cb91e", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012-input/config.json": "e23a81e11a8656f23ee3228362d5c333dafe44c6d10e0941b5ca32f8dbf8e3fa", + "jobs/dev34-baseline-main-source/jobs/dev34-build-pov-ray-main-20261012-input/working-tree.patch": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "official_pytest_summary": { + "collected": 3, + "passed": 3, + "failed": 0, + "skipped": 0, + "error": 0, + "summary_line": "3 passed in 5.70s", + "source": "verifier/test-stdout.txt" + }, + "audit": { + "benchmark_material_exposure": "not_observed_in_review", + "credential_observation_steps": [] + } + } + ], + "publication_policy": "Only derived results and hashes are published. Raw logs and trajectories remain local and require credential redaction before sharing.", + "final_image_integrity": { + "observed_at": "2026-10-12T22:16:08.238083+00:00", + "original_task_images_verified": 34, + "image_ids_and_registry_digests_unchanged": true, + "remaining_scored_trial_containers": [] + }, + "post_freeze_audit": { + "completed_at": "2026-10-12T22:21:48.045217+00:00", + "method": "Scan all 34 frozen trajectories for benchmark references, local verifier paths, external hosts, and exact credential values; review flagged commands/observations and six failure traces against official test output.", + "official_assertions_executed_tasks": 34, + "setup_errors_in_model_phase": 0, + "verifier_errors": 0, + "benchmark_material_exposure_tasks": [ + "terminal-bench/torch-tensor-parallelism" + ], + "hidden_solution_or_test_retrieval_observed": false, + "limitation": "This is a retrospective trace audit, not proof of absence of all exposure or model training contamination. All 34 tasks were already in the development pool. Raw reward is not adjusted by this audit.", + "credential_handling": "The provider credential appeared in db-wal-recovery tool observations at ATIF steps 27 and 49 after environment inspection. Raw logs remain local; only derived findings and hashes are published. No credential transmission to another external destination was observed in the reviewed commands.", + "parameterized_test_reporting": "Native CTRF deduplicates parameterized functions: pipeline reports 3 tests while pytest executes 4; tensor reports 3 while pytest executes 13. Both native CTRF and actual pytest counts are retained.", + "public_source_access": "Other reviewed network activity concerned dependencies, upstream projects, CIFAR data, NEB documentation, and the MTEB/Scandinavian leaderboard sources requested by those tasks." + }, + "preflight_history": { + "install_only_attempts": 54, + "successful": 34, + "setup_errors": 15, + "cancelled_or_administratively_interrupted": 5, + "by_plan": [ + { + "plan": "v1", + "attempts": 6, + "successful": 0, + "setup_errors": 5, + "cancelled_or_administratively_interrupted": 1 + }, + { + "plan": "v2", + "attempts": 9, + "successful": 0, + "setup_errors": 7, + "cancelled_or_administratively_interrupted": 2 + }, + { + "plan": "v3", + "attempts": 34, + "successful": 29, + "setup_errors": 3, + "cancelled_or_administratively_interrupted": 2 + }, + { + "plan": "v4", + "attempts": 5, + "successful": 5, + "setup_errors": 0, + "cancelled_or_administratively_interrupted": 0 + } + ], + "accepted": "The v5 plan accepts 29 successful native v3 installations and five v4 installations, with provenance verified before model work.", + "registered_verifier_profile_passed_tasks": [ + "terminal-bench/qemu-alpine-ssh" + ], + "verifier_profile_not_configured_tasks": 33, + "limitation": "Successful agent installation does not certify unregistered verifier dependencies. Preflight trials ran no models or official task assertions." + }, + "descriptive_subsets": [ + { + "subset": "previous_22", + "trials": 22, + "passes": 19 + }, + { + "subset": "twelve_additions", + "trials": 12, + "passes": 9 + } + ], + "historical_comparison_note": "The earlier twelve-task run used the same main commit and passed 10/12; this run passes 9/12 on those tasks. PyTorch CLI changed from fail to pass, sampler and SQL from pass to fail, and DB recovery failed both times. These are separate trials with different transport/environment history and do not estimate an agent code effect." +} From b7d4db49973095792f82dd8da9d08995cbf0445f Mon Sep 17 00:00:00 2001 From: minixalpha Date: Wed, 14 Oct 2026 06:36:54 -0700 Subject: [PATCH 13/15] docs: record the 34-task baseline in development notes --- docs/dev_notes/en/0.10.x.md | 87 ++++++++++++++++++++++++++++++++++ docs/dev_notes/zh-CN/0.10.x.md | 63 ++++++++++++++++++++++++ 2 files changed, 150 insertions(+) diff --git a/docs/dev_notes/en/0.10.x.md b/docs/dev_notes/en/0.10.x.md index e260b13..7f12bc2 100644 --- a/docs/dev_notes/en/0.10.x.md +++ b/docs/dev_notes/en/0.10.x.md @@ -4,6 +4,93 @@ ## 0.10.0 - YYYY.MM.DD +### Common-main baseline for 34 tasks: 28 passes and six failures + +On 2026-10-12, combine the previously exercised 22 tasks and the twelve additions +into one development pool, running one fresh trial per task on then-current main +`2eb99b608dfb39464d17d109a4a8f11afda607a9`. Experiment records live on +`benchmark/dev34-main-baseline-20261012`; the evaluated agent and Harbor adapter +come from a frozen main checkout. All 34 installations have identical source +snapshots and wheels. The runtime prompt is unchanged, and no outputs or scores +from pilot20, repeated experiments, or the twelve-task baseline are reused. + +Use `deepseek-flash` through DeepSeek's official Anthropic-compatible endpoint, +main's 385-word headless prompt, 100 replies, and 65,536 output tokens per +request. Each task's work budget is its native agent timeout minus 180 seconds; +installation has a separate 1,800-second limit, and native per-attempt verifier +limits remain unchanged. Fix the order by work budget and task name, completing +each pair before starting the next, with concurrency two. Generate one output +per task, without outcome-based replacements or model reruns. No allowed +verifier timeout retries are needed either. + +The final result is **28/34 passes (82.35%) and six scored failures**, with no +setup or verifier errors in the model phase. All 34 official suites execute +tests: 155 pytest cases yield 137 passes and 18 failures, with no skips or +errors. Parameterized cases use actual pytest execution counts; native CTRF +summaries are retained separately, and case counts do not replace task scores. +The original 22-task subset scores 19/22, and the twelve additions score 9/12. + +| Failed task | Observed cause | Future verification direction | +| --- | --- | --- | +| `adaptive-rejection-sampler` | R installation delays and two subsequent model-stream interruptions leave no `/app/ars.R` within 720 seconds; all nine official checks fail | Bound dependency preparation, launch background work reliably, and save a minimal deliverable early. This run does not demonstrate an incorrect sampling algorithm. | +| `db-wal-recovery` | Opening the original database before backup makes SQLite discard its encrypted WAL; guessed values miss WAL updates, with five checks passing and two failing | Preserve the database and every sidecar before working on copies; verify values against original evidence, not just format and row count. | +| `kv-store-grpc` | The request field is `val` instead of the required `value`; the self-test client reuses the same incorrect schema, with five checks passing and two failing | Derive the interface contract independently from the task, inspect descriptors, and verify with an independent client. | +| `query-optimize` | Results are correct but too slow: solution median 0.371624 seconds versus reference 0.291814 seconds exceeds the 1.05x allowance; five checks pass and one fails | Use a working, repeatable timing harness in the target environment, compare candidate query plans, and seek performance margin. The reference query and exact threshold were hidden from the agent. | +| `torch-pipeline-parallelism` | The implementation omits position embeddings required by the installed Transformers API; self-testing finds this near the deadline, with two checks passing and two failing | Inspect the actual API early, run a minimal single-process forward/backward test before distributed tests, and preserve exit status and repair time. | +| `regex-chess` | White can castle kingside while already in check; two checks pass and two fail | Explicitly test negative conditions for special rules. More than 1,000 sampled positions passing against independent `python-chess` do not replace checks for castling out of, through, or into check. | + +Keep dependency preparation separate from model scoring. Before any model work, +54 install-only attempts yield 34 successes, 15 setup errors, and five +cancellations or administrative interruptions. After verifying native provenance, +the final v5 plan accepts 29 successful v3 installations and five from v4. +QEMU's registered verifier dependency profile passes; the other 33 remain +explicitly `not_configured`. Agent installation success does not certify their +verifier dependencies. Verified caches of original APT indexes and uv/CPython +archives address download failures. Every model trial uses the final transport +configuration while preserving original images, package origins, task resources, +official scripts, and time limits. All 167 Harbor workflow tests pass, as do all +CI checks for the final result commit. + +Inspect new trajectories and hidden-test content only after all outputs freeze +at 22:12:55 UTC. The audit finds that the passing `torch-tensor-parallelism` trial +searched for and read external Terminal-Bench revision notes. The displayed +pre-scattered-input requirement was already in its instruction; no reference +code or hidden-test retrieval was observed. Preserve its raw pass and flag +external benchmark-material access separately; the complete run cannot be +described as an independent evaluation without such access. Environment +inspection in WAL recovery also placed the provider credential in tool +observations. Raw logs remain local; commits contain only derived findings and +hashes. The audit observed no transmission of that credential to another +external destination, but cannot prove absence of all exposure or training +contamination. + +Model-job wall time is approximately **283.25 minutes**, excluding dependency +preflight. The known token-cost estimate subtotal is **$2.523**. Sampler and +pipeline usage/cost records are incomplete: missing values are not zero, and +the subtotal is not an actual bill. The earlier twelve-task baseline scored +10/12 on the same main commit. Here PyTorch CLI changes from fail to pass, +sampler and SQL change from pass to fail, and WAL recovery fails both times. +Generation paths and environment history differ, so these changes cannot be +attributed to agent code changes. + +All 34 tasks have prior development exposure and are not a held-out set. Use the +six failures for diagnosis and retain passing tasks for regression checks; +one pass is not a reason to remove a task from the pool. Future candidates need +matched baseline/candidate repeats under the same conditions, tracking passes, +verification evidence, time, and material access separately. The other 55 tasks +remain unopened in this run, reserved for a separately registered evaluation +after selecting a candidate. Input preservation, independent contract checks, +earlier small tests, failure-status propagation, targeted negative cases, and +repeatable performance measurements remain hypotheses for the next experiment. + +The [complete per-task report](../../../benchmarks/harbor/reports/dev34-main-baseline-20261012.md), +[structured results](../../../benchmarks/harbor/results/dev34-main-baseline-20261012.json), +and [frozen configuration](../../../benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md) +retain all 34 task records, failure evidence, provenance hashes, preflight history, +and reproduction entry points. Add this entry on 2026-10-14 and synchronize the +Chinese and English development notes; this documentation update starts no new +benchmark runs. + ### Experiment: headless task-contract and independent-verification guidance On 2026-10-07, implement the prompt study's P0 candidate on diff --git a/docs/dev_notes/zh-CN/0.10.x.md b/docs/dev_notes/zh-CN/0.10.x.md index a541ea4..e8a32b8 100644 --- a/docs/dev_notes/zh-CN/0.10.x.md +++ b/docs/dev_notes/zh-CN/0.10.x.md @@ -4,6 +4,69 @@ ## 0.10.0 - YYYY.MM.DD +### 34 题统一主线 baseline：28 题通过，6 题失败 + +2026-10-12，将原先已运行的 22 题与新增的 12 题合为开发池，全部在当时的 main +`2eb99b608dfb39464d17d109a4a8f11afda607a9` 上各运行一次全新试次。实验记录位于 +`benchmark/dev34-main-baseline-20261012` 分支，实际 agent 和 Harbor adapter +来自固定的 main checkout；34 次安装的源码快照与 wheel 一致，没有修改运行 +提示词，也没有复用此前 pilot20、重复实验或 12 题 baseline 的输出和分数。 + +模型为 DeepSeek 官方 Anthropic 兼容端点的 `deepseek-flash`，使用 main 的 +385 词 headless 提示词、100 回合和每次请求 65,536 输出 token 上限。每题工作 +预算为原始 agent 时限减 180 秒，安装另设 1,800 秒，官方逐次 verifier 时限 +保持不变。按预算、题名固定顺序，每两题结束后再启动下一对，总并发为 2。 +每题只生成一次，没有按结果替换或重跑模型；允许的 verifier 超时重试也未发生。 + +最终为 **28/34 通过（82.35%）、6 题评分失败**，正式模型运行阶段没有 setup +或 verifier 错误。34 题都执行了官方测试，共 155 个 pytest 用例：137 通过、 +18 失败，没有跳过或错误。参数化用例按 pytest 实际执行数统计，原生 CTRF +摘要另外保留，不将测试用例数当作题目分数。原 22 题子集为 19/22，新增 12 题 +子集为 9/12。 + +| 未通过题目 | 本轮原因 | 后续验证方向 | +| --- | --- | --- | +| `adaptive-rejection-sampler` | R 安装耗时，随后模型流两次中断，720 秒内未写出 `/app/ars.R`；官方 9 项均失败 | 限制依赖准备耗时、可靠启动后台任务、提前保存最小交付物；本轮不能据此判断采样算法错误 | +| `db-wal-recovery` | 备份前打开原数据库，SQLite 丢弃了加密 WAL；最终推测的值遗漏 WAL 更新，5 项通过、2 项失败 | 先保存数据库及全部 sidecar，再在副本上操作；按原始证据核对值，不能只检查格式和行数 | +| `kv-store-grpc` | 请求字段写成 `val`，题面要求 `value`；自测客户端复用同一错误 schema，5 项通过、2 项失败 | 从题面独立提取接口契约，检查 descriptor，并用独立客户端验证 | +| `query-optimize` | 结果正确，性能不达标；解答中位数 0.371624 秒，参考 0.291814 秒，超过允许的 1.05 倍，5 项通过、1 项失败 | 使用可执行、可重复的目标环境计时，比较候选查询计划并保留性能余量；参考查询及精确门槛并未向 agent 公开 | +| `torch-pipeline-parallelism` | 实现未提供已安装 Transformers API 所需的位置编码参数；自测发现后已接近截止时间，2 项通过、2 项失败 | 尽早检查实际 API，先跑最小单进程前向与反向，再扩展分布式测试；保留退出码和修复时间 | +| `regex-chess` | 允许白王在已被将军时进行王翼易位，2 项通过、2 项失败 | 对特殊规则显式设计反例；即使对独立 `python-chess` 的一千多个采样局面自测通过，也不能替代“被将军、经过受攻击格、落到受攻击格”等检查 | + +依赖准备阶段与模型评分分开保存：模型启动前累计 54 次 install-only 尝试， +34 次成功、15 次 setup 错误、5 次取消或管理性中断。最终 v5 计划在核验原生 +来源后接受 v3 的 29 次成功和 v4 的 5 次成功。QEMU 的已注册 verifier 依赖 +profile 通过，其他 33 题仍明确为 `not_configured`，不把 agent 安装成功当作 +这些 verifier 依赖预检通过。下载问题通过校验原始内容的 APT 索引及 uv/CPython +归档缓存解决；正式运行统一使用最终传输配置，原始镜像、包来源、任务资源、 +官方脚本和时限保持不变。Harbor 流程测试 167 项通过，最终结果提交的 CI 全部通过。 + +全部输出于 22:12:55 UTC 冻结后才检查新轨迹和隐藏测试内容。审计发现，通过的 +`torch-tensor-parallelism` 曾搜索并读取外部 Terminal-Bench 修订说明；显示的 +输入预分片约定已存在于本题说明，未观察到获取参考代码或隐藏测试。保留其原始 +通过分数并单独标记外部基准资料接触，不能将整轮称为无外部接触的独立评测。 +WAL 题的环境检查还使 provider 凭据出现在工具观察中；原始日志留在本地，提交 +只包含派生结论和哈希。审计未观察到凭据被传到其他外部目的地，也不能据此证明 +完全不存在信息暴露或训练污染。 + +正式 job 墙钟耗时约 **283.25 分钟**，不含依赖预检。已知 token 费用估算小计 +为 **$2.523**，采样器和流水线两题 usage/费用不完整，缺失部分不能当作零， +也不能把小计当作实际账单。此前 12 题 baseline 虽然也是同一 main，却为 10/12： +本轮 PyTorch CLI 从失败变通过，采样器与 SQL 从通过变失败，WAL 两次都失败。 +生成路径和环境历史不同，不能把这些变化归因为 agent 代码改动。 + +这 34 题均已有开发暴露，不能当作保留集；六题用于诊断，其他通过题保留为回归 +检查，不能因为通过一次就从开发池移除。后续候选应在相同条件下与 baseline +配对重复，分别观察通过率、验证证据、耗时与资料接触。其余 55 题本轮保持未打开， +留待候选确定后另行登记评估。输入保护、独立契约检查、更早执行小测试、失败状态 +传递、针对性反例和可重复性能测量仍只是下一轮待验证的优化方向。 + +[完整逐题报告](../../../benchmarks/harbor/reports/dev34-main-baseline-20261012.md)、 +[结构化结果](../../../benchmarks/harbor/results/dev34-main-baseline-20261012.json) 和 +[固定配置](../../../benchmarks/harbor/configs/dev34-main-baseline-20261012/README.md) +保留全部 34 题明细、失败证据、来源哈希、预检历史和复现入口。本节于 2026-10-14 +补记，中英文开发笔记同步；本次文档更新没有启动新的 benchmark。 + ### 实验：headless 任务契约与独立验证提示词 2026-10-07 在 `experiment/headless-contract-verification` 分支落实提示词调研的 From 99f8ea8258701d28ead02befeb284bbd2198ff5f Mon Sep 17 00:00:00 2001 From: minixalpha Date: Wed, 14 Oct 2026 06:42:38 -0700 Subject: [PATCH 14/15] docs: include the bilingual common-main baseline prompt --- docs/dev_notes/zh-CN/0.10.x.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/docs/dev_notes/zh-CN/0.10.x.md b/docs/dev_notes/zh-CN/0.10.x.md index e8a32b8..25fb1a7 100644 --- a/docs/dev_notes/zh-CN/0.10.x.md +++ b/docs/dev_notes/zh-CN/0.10.x.md @@ -18,6 +18,39 @@ 保持不变。按预算、题名固定顺序，每两题结束后再启动下一对，总并发为 2。 每题只生成一次，没有按结果替换或重跑模型；允许的 verifier 超时重试也未发生。 +本轮实际发送给模型的完整静态 headless 系统提示词如下，包含末段共享工具指导； +逐题任务正文、工具 schema 和运行时预算提醒不属于这段固定文本。英文原文取自 +[本轮冻结的提示词文件](../../../benchmarks/harbor/configs/dev34-main-baseline-20261012/main-prompt.txt)， +共 385 词，SHA256 为 +`37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d`。 +中文译文仅供阅读，实验使用的是英文原文。 + +**英文原文** + +```text +You are nanoPyCodeAgent, a coding agent running non-interactively on a task handed to you up front. There is no user to reply to you: never ask a clarifying question, never stop to wait for confirmation, and never present a plan for approval — decide on your own and carry it out. Work the task through to the end, then check the result with the tools instead of assuming it worked. When it is done, answer with a short summary and no further tool calls: that reply is what ends the run. Runtime budget reminders report remaining time and model replies; when either is running low, prioritize saving the required deliverables, necessary verification, and a final summary. Do not start optional work after the requirements are satisfied. + +Identify the required deliverables, interfaces, and constraints. Treat examples as evidence, not as the full contract unless the task says so. Inspect relevant files, dependencies, and existing checks instead of guessing. Task-supplied workspace files, including source, executables, and tests, are available references unless the task restricts their use. Consult general documentation as needed. Do not seek or use task-specific reference solutions or hidden evaluation tests from outside the supplied materials unless the task explicitly requests them. Derive checks from the stated requirements. + +Verify the requested behavior through the real entry point when feasible. Check the content and meaning of outputs, not only command success or file existence. For logic or numerical work, choose a few representative and boundary cases justified by the task, using an independent reference, invariant, or consistency check when available. Do not derive every expectation from the implementation being tested or weaken requirements to make a check pass. Keep verification proportional to the task and remaining budget. + +When a check fails, inspect the error and relevant output, revise the underlying assumption or implementation, and rerun the affected check. Keep the task's constraints intact. Once appropriate checks support the required behavior, save the deliverables and finish. State what was verified and any remaining limitation without claiming unobserved success. + +Prefer the read tool for viewing files, the edit tool for changing part of an existing file, and the write tool for creating files or rewriting them whole. Use the bash tool to run commands, search with grep, and complete tasks that need real command output instead of guessing. +``` + +**中文译文** + +> 你是 nanoPyCodeAgent，一个以非交互方式运行的编码 agent，任务在开始时就已交给你。没有用户会回复你：不要提出澄清问题，不要停下来等待确认，也不要提交计划等待批准——自行决定并执行。将任务推进到完成，然后使用工具检查结果，不要假定它已经成功。完成后给出简短总结，不再调用工具：这条回复就是结束本次运行的信号。运行时预算提醒会报告剩余时间和模型回复次数；任一项所剩不多时，优先保存要求的交付物、完成必要验证并给出最终总结。要求满足后，不要再开始可选工作。 +> +> 明确要求的交付物、接口和约束。将示例视为证据；除非任务明确说明，否则不要把示例当作完整契约。检查相关文件、依赖和已有检查，不要靠猜测。任务提供的工作区文件，包括源码、可执行文件和测试，都可以作为参考，除非任务限制了它们的使用。根据需要查阅通用文档。除非任务明确要求，否则不得从所提供材料之外寻找或使用该任务专用的参考解答或隐藏评测测试。从明确给出的要求推导检查。 +> +> 可行时，通过真实入口验证要求的行为。检查输出的内容和含义，而不只是命令是否成功或文件是否存在。对于逻辑或数值任务，选择少量有任务依据的代表性用例和边界用例；如果有独立参考、不变量或一致性检查，就使用它们。不要从被测实现中推导全部预期结果，也不要为了让检查通过而降低要求。让验证投入与任务及剩余预算相称。 +> +> 检查失败时，查看错误和相关输出，修正底层假设或实现，并重新运行受影响的检查。保持任务约束不变。当适当的检查已为要求的行为提供支持时，保存交付物并结束。说明验证了什么，以及仍然存在的限制，不要声称未实际观察到的成功。 +> +> 查看文件时优先使用 read 工具，修改现有文件的部分内容时使用 edit 工具，创建文件或整体重写文件时使用 write 工具。使用 bash 工具运行命令、通过 grep 搜索，并完成需要真实命令输出的任务，不要靠猜测。 + 最终为 **28/34 通过（82.35%）、6 题评分失败**，正式模型运行阶段没有 setup 或 verifier 错误。34 题都执行了官方测试，共 155 个 pytest 用例：137 通过、 18 失败，没有跳过或错误。参数化用例按 pytest 实际执行数统计，原生 CTRF From 20b36f1656056c072f9d12ba48f67d897de740ec Mon Sep 17 00:00:00 2001 From: minixalpha Date: Wed, 14 Oct 2026 06:58:50 -0700 Subject: [PATCH 15/15] docs: regenerate english development notes for baseline merge --- docs/dev_notes/en/0.10.x.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/docs/dev_notes/en/0.10.x.md b/docs/dev_notes/en/0.10.x.md index 7f12bc2..6614821 100644 --- a/docs/dev_notes/en/0.10.x.md +++ b/docs/dev_notes/en/0.10.x.md @@ -23,6 +23,39 @@ each pair before starting the next, with concurrency two. Generate one output per task, without outcome-based replacements or model reruns. No allowed verifier timeout retries are needed either. +The complete static headless system prompt actually sent to the model follows, +including the shared tool guidance in the final paragraph. Per-task instructions, +tool schemas, and runtime budget reminders are separate from this fixed text. +The English original comes from the +[frozen prompt file for this run](../../../benchmarks/harbor/configs/dev34-main-baseline-20261012/main-prompt.txt), +contains 385 words, and has SHA256 +`37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d`. +The Chinese translation is for reading only; the experiment used the English +original. + +**English original** + +```text +You are nanoPyCodeAgent, a coding agent running non-interactively on a task handed to you up front. There is no user to reply to you: never ask a clarifying question, never stop to wait for confirmation, and never present a plan for approval — decide on your own and carry it out. Work the task through to the end, then check the result with the tools instead of assuming it worked. When it is done, answer with a short summary and no further tool calls: that reply is what ends the run. Runtime budget reminders report remaining time and model replies; when either is running low, prioritize saving the required deliverables, necessary verification, and a final summary. Do not start optional work after the requirements are satisfied. + +Identify the required deliverables, interfaces, and constraints. Treat examples as evidence, not as the full contract unless the task says so. Inspect relevant files, dependencies, and existing checks instead of guessing. Task-supplied workspace files, including source, executables, and tests, are available references unless the task restricts their use. Consult general documentation as needed. Do not seek or use task-specific reference solutions or hidden evaluation tests from outside the supplied materials unless the task explicitly requests them. Derive checks from the stated requirements. + +Verify the requested behavior through the real entry point when feasible. Check the content and meaning of outputs, not only command success or file existence. For logic or numerical work, choose a few representative and boundary cases justified by the task, using an independent reference, invariant, or consistency check when available. Do not derive every expectation from the implementation being tested or weaken requirements to make a check pass. Keep verification proportional to the task and remaining budget. + +When a check fails, inspect the error and relevant output, revise the underlying assumption or implementation, and rerun the affected check. Keep the task's constraints intact. Once appropriate checks support the required behavior, save the deliverables and finish. State what was verified and any remaining limitation without claiming unobserved success. + +Prefer the read tool for viewing files, the edit tool for changing part of an existing file, and the write tool for creating files or rewriting them whole. Use the bash tool to run commands, search with grep, and complete tasks that need real command output instead of guessing. +``` + +**Chinese translation** + +The [Chinese source](../zh-CN/0.10.x.md) includes a complete five-paragraph Chinese +translation immediately below the original. It covers the same instructions +quoted above: autonomous execution and budgets; task contracts and permitted +references; independent verification; diagnosing failures and reporting limits; +and use of the four tools. The English edition retains the original once rather +than translating that Chinese rendering back into a duplicate English quotation. + The final result is **28/34 passes (82.35%) and six scored failures**, with no setup or verifier errors in the model phase. All 34 official suites execute tests: 155 pytest cases yield 137 passes and 18 failures, with no skips or