Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions benchmarks/harbor/configs/truncation-recovery-20261005/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Bounded truncation recovery experiment

The [predeclared plan](experiment.json) selects the two failures from the
[v0.9.0 pilot20 baseline](../../reports/v0.9.0-baseline-20261005.md), with their
original revisions and budgets. Each task receives one fresh model run; no
score replaces a trial in the released baseline.

Use a checkout of agent commit
`e1be6ed1f81a17d73677b414d4d913adb81e9e25` for the recorded behavior and wheel
version. The standard runner builds its current checkout, including local
source changes. Configure the official DeepSeek credentials as described in
the [adapter README](../../README.md), set
`ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic`, and leave
`ANTHROPIC_MODEL` unset so the configuration chooses `deepseek-flash`.

Run dependency installation preflight first:

```bash
uv run --project benchmarks/harbor python -m harbor_adapter run \
--config benchmarks/harbor/configs/truncation-recovery-20261005/install-only.json \
--install-only
```

Both tasks lack a registered verifier dependency profile at this revision and
must explicitly report `not_configured`. After agent installation succeeds,
run the following two commands in separate terminals to reproduce the original
concurrent task pairing. Omit `--job-name` to generate fresh job names.

```bash
uv run --project benchmarks/harbor python -m harbor_adapter run \
--config benchmarks/harbor/configs/truncation-recovery-20261005/model-extraction.json
uv run --project benchmarks/harbor python -m harbor_adapter run \
--config benchmarks/harbor/configs/truncation-recovery-20261005/scheme.json
```

Record both task scores and actual recovery events. A task that passes without
reaching `max_tokens` does not demonstrate recovery effectiveness. The
[experiment report](../../reports/truncation-recovery-20261005.md) and
[structured results](../../results/tb21-truncation-recovery-20261005.json)
preserve that distinction.
150 changes: 150 additions & 0 deletions benchmarks/harbor/configs/truncation-recovery-20261005/experiment.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
{
"created_at": "2026-10-05T12:53:39.916356+00:00",
"question": "Can one bounded headless continuation after max_tokens preserve useful work and improve the two failed tasks without changing their budgets?",
"baseline_results": "benchmarks/harbor/results/tb21-v090-baseline-20261005.json",
"baseline_agent_ref": "c0605369d277352ac0d5ec20d02017b7cb149314",
"dataset_ref": "sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a",
"model": "deepseek/deepseek-flash",
"endpoint": "https://api.deepseek.com/anthropic",
"change": "One automatic truncation recovery per headless run. Keep the token cap and original deadline and turn count; skip truncated tools and incomplete thinking.",
"max_turns": 100,
"max_tokens": 65536,
"agent_setup_timeout_seconds": 1800,
"attempts_per_task": 1,
"whole_trial_retries": 0,
"execution": "Preflight both tasks, then launch the two disjoint single-task standard jobs together, at most two simultaneous trials. Each job retains n_concurrent_trials=2 but contains one task.",
"verifier_policy": "Original per-attempt limit; only verifier timeouts may retry, at most three attempts on the same agent output.",
"preflight": "Both task profiles are unsupported and must be recorded as not_configured; installation success does not certify verifier dependencies.",
"analysis": "Preserve every result. Count actual max_tokens responses and truncation_recovery injections. If no truncation occurs, a pass does not demonstrate recovery effectiveness. Report setup/verifier exceptions separately from scored failures. Do not replace the released pilot20 baseline or extrapolate a full-pilot score.",
"tasks": [
{
"name": "terminal-bench/model-extraction-relu-logits",
"ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17",
"native_agent_timeout_seconds": 900,
"work_budget_seconds": 720,
"verifier_timeout_seconds": 900.0,
"docker_image": "alexgshaw/model-extraction-relu-logits:20251031",
"docker_image_id": "sha256:52fe1f089f38650f0dc22d7531ee6e01ebd526de1b349aaa2ecb331dca9fabce",
"docker_repo_digests": [
"alexgshaw/model-extraction-relu-logits@sha256:52fe1f089f38650f0dc22d7531ee6e01ebd526de1b349aaa2ecb331dca9fabce"
],
"task_file_sha256": {
"instruction.md": "c9e179dd099065a34c362a865b0eb9fab5f7d96be582cf117e7519580c036ebe",
"task.toml": "eb64dfaf8ae88868911ae28855cdc4f767de464001e2704919015b595be6dc29",
"tests/test.sh": "3b2d8f5e0353c2bab3515d6387468a863a4f9f241ee131c73f545f3fdf9e2d52",
"tests/test_outputs.py": "e5189c7eb9942ede55e1b48a80388590b80a486b75497613f257b3373ad9b2c9"
}
},
{
"name": "terminal-bench/schemelike-metacircular-eval",
"ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6",
"native_agent_timeout_seconds": 2400,
"work_budget_seconds": 2220,
"verifier_timeout_seconds": 2400.0,
"docker_image": "alexgshaw/schemelike-metacircular-eval:20251031",
"docker_image_id": "sha256:b485be38bf80d56c1a4cd9da95bbf7ec129dc03f6786b86c0525518a96f6d1db",
"docker_repo_digests": [
"alexgshaw/schemelike-metacircular-eval@sha256:b485be38bf80d56c1a4cd9da95bbf7ec129dc03f6786b86c0525518a96f6d1db"
],
"task_file_sha256": {
"instruction.md": "3af5f9567b62f93683db7518ec6fcb609f83796258aa02166b5a7091c5b69031",
"task.toml": "8dae8eb226183b0ca92331c1106152a7abc93b03b4e5a95291b86a4ddc9b44e0",
"tests/interp.py": "efb8b5cdcf78b73afcf39d59600a1f355ca89b7bac9854bb01f110cc77bea2c1",
"tests/shadow_test/01-factorial.scm": "540e81fac476f04dc510ba107721d606ac980760088b2136889e7d4bf884e5c5",
"tests/shadow_test/02-fibonacci.scm": "628957171e0c4a3ee373635923fcd29e75bdf4bf13a402ac26fe94a5fc1443ac",
"tests/shadow_test/03-list-operations.scm": "fe8f17f59fe4045582fce363d52df95ba7c4de7345542c8bd4993a9f2f396235",
"tests/shadow_test/04-higher-order.scm": "9c2eba8c91bb7b224de56d789207d2eaad98bbe1f8c360d6eca4f014f42f9230",
"tests/shadow_test/05-simple-io.scm": "1eba30db30824fcf6048ea8e49b84e85c866cefce30862fce49b300e4416843c",
"tests/shadow_test/06-interactive-io.scm": "4fbc0e2d6e42c5a9495e549ccb13c02fcaeddf8fdaf8d2cf5de833116f12610a",
"tests/shadow_test/08-progn-sequencing.scm": "3bf8236dc5356d29ab8dbfe1b0f4c519354fbb23c8f3eaf946779c28b80feea3",
"tests/shadow_test/09-mutual-recursion.scm": "3bfa1f77e602c7039f8c8c70315e6d4fff6eed737dab0f0398e5bdf690137c2c",
"tests/shadow_test/10-advanced-features.scm": "b69ef497088fcf0aec388789a17561b783b8cdd5915719915a51c5ba189c31ff",
"tests/shadow_test/accumulator_patterns.scm": "8b9192796a6f9c3155cc5e44d0a0ba6a46d20a0d94e265b69b23ec8af43e4a07",
"tests/shadow_test/binary_tree.scm": "5a3dafc8844969cebb5809f253f4e54de7df480f378da8197f870badef03ee47",
"tests/shadow_test/church_numerals.scm": "31d075785f2f605bac3bd9ac2d99fcf7eaaed6686eb9afeab1d40bf5f35bbf2c",
"tests/shadow_test/closures.scm": "c7d025a376987f5ee22f529847b66eaf53f0af7033acbb4055cedb734168d612",
"tests/shadow_test/continuation_passing.scm": "79e7445c42227846f1adfaf08c17e278aac7a34df77482900b068be4fd248080",
"tests/shadow_test/currying.scm": "690a8cccbb1d249c8f6fcabc0193cecc5d0e4c1af39a0bb67c7e95fcff8e393e",
"tests/shadow_test/filter_operations.scm": "485a0563b81d002e368c5e4b2b7cc98a5306163701053630f416e3b783a2a238",
"tests/shadow_test/fold_operations.scm": "461b33db6b3f5580122d4c6dff0fd6d11c7f044865ffd684ade67d9b296a5a24",
"tests/shadow_test/function_composition.scm": "65ceee00188b12c3549134449e2f938d010c2c97dba090f5b53244abb3579b1f",
"tests/shadow_test/lazy_evaluation.scm": "edd95748de14aadf916c65209b5bcb8759d3e33b7ec935289a5128a656d33bec",
"tests/shadow_test/list_operations.scm": "660963851ecb49639f66557bd88db1e73db323186742145f33cb2e14d3af1103",
"tests/shadow_test/map_operations.scm": "974ac42d892f4b62b9b21ef7844581624dad811b38d35944f1005cc47f249856",
"tests/shadow_test/memoization.scm": "25482c1912fab1235651b1b7acd727f2ea6a55dc5c1919f841b44cfc32107b05",
"tests/shadow_test/mutual_recursion.scm": "2fbc9562e863cb2b325f86419fb2a7588265aaf5576295550722ee4274663712",
"tests/shadow_test/nested_defines.scm": "23ad3b2f936949501466da784511ea2e6215e2b198ca371bef86f6af877ca923",
"tests/shadow_test/oeis_sequences.scm": "1b586df6af2429a54808f7587f60021ba2b2c9bcc2ef2f9cbe44339699853c62",
"tests/shadow_test/oeis_sequences2.scm": "300da0b0dc20b883e01142adea9eb6da1a583a1b969ac0be47aa32c1ab66340e",
"tests/shadow_test/oeis_sequences3.scm": "dea198e874bf518b93c8406a843b64c1586cf4ce780fec8c444f8343e9322c75",
"tests/shadow_test/recursive_structures.scm": "64f1a430bf04a627f5e27bf004805544b60406f8a123d186516774dfcf3846cd",
"tests/shadow_test/test_read.scm": "fda448c22685e2f9af37ff7b986ee5e906d485a2db16f62eec113149f866870f",
"tests/shadow_test/variadic_functions.scm": "ac940ac0273e87ee06280fee192194ab5cdc14c88b61da9cfba92105e66b8f2f",
"tests/shadow_test/y_combinator.scm": "38d25b02d54342ab5610ad03fcef68c5f1d4eef8902295645f6acceb853c9860",
"tests/test/01-factorial.scm": "2623000a742c63d50ae1851854826247caee129c7d1613be6c7131d21f26a6ad",
"tests/test/02-fibonacci.scm": "db769cef446d5da9e9b1abccdf9ca644e239938095a98c21551aee25255a57d4",
"tests/test/03-list-operations.scm": "1a14b4100dedef30010fed62ea9fd295add1b0facb2e8e6e27c8d8eb3d90111b",
"tests/test/04-higher-order.scm": "59cd17bbffad641a37729b17c5bf9178b8bae7289fe89cc7135eb0bfec2b4fd6",
"tests/test/05-simple-io.scm": "3deeb3275df1b771449ebaebfcb9a934f375415e138c5e9f3685516c69a1f7a0",
"tests/test/06-interactive-io.scm": "aa3cae79b79dca90ac9d61a1babf64d6a5bb6559798c7727c11be02d8ddfd216",
"tests/test/08-progn-sequencing.scm": "4c0a4d980cf4223d1d17715d58adbe27dc120eced257a0d9f42de952c094bc5b",
"tests/test/09-mutual-recursion.scm": "939e18f4bd25b5ab0e2a4a767bf7eb6b0d8fb3530d28bf236f06cf27f2d90ee8",
"tests/test/10-advanced-features.scm": "c4cc5fb412df363875768c6a1e645255f6bc8f9ddf988d1ec2f4a266cedb8ec7",
"tests/test/README.md": "1f2174c7f12721f9f1898cda190c2886fb3ef9d35fd085f315da5814657104ae",
"tests/test/accumulator_patterns.scm": "50d246e00577d1f3ab3f0462c2b9618ea63680c33fcca12369cd1707b010cb5f",
"tests/test/binary_tree.scm": "c95f59ccac1ea4a9c846afddc993aa40ac081c0f9e6731544530109b269b67ae",
"tests/test/calculator.scm": "a09213ec8a2ec52486b9346c420ddb7c537576ef71b1cc5c6ec61d13d9139f61",
"tests/test/church_numerals.scm": "f37d858f32a9c5ba8868d094018c92516acadf0ea4e55b7c36f14320cb0c69a6",
"tests/test/closures.scm": "3ecfc35289479288876e20bf7f55e5b8eda84acd35937e89e3d062308a99aeff",
"tests/test/continuation_passing.scm": "01049581e582f7d35ef40ef68e1827d012da619beea57c1c74675a5f245318d7",
"tests/test/currying.scm": "2844d54eee9a94af05a11b4aae06d6e7823d193f17235c8f90bf98926cbca75d",
"tests/test/filter_operations.scm": "f6bc75f6da9ece96c0616e3570e4df59d0f2eb4655375d3a46b0349c86cc5dd8",
"tests/test/fold_operations.scm": "05aab492ad914baecd19cf454ed587e7a031e94b8990500eb1c869c15da6fc0b",
"tests/test/function_composition.scm": "9b56dc78823e7d35adb213744c0f8e95684074d6a12bb98c8c05755788be5b06",
"tests/test/lazy_evaluation.scm": "ea5912a53a35b65bf2bee9483dca90814b8baf52ae3ae50c2d4589f794dca54d",
"tests/test/list_operations.scm": "d7e70621afbbb787cf646e2089bccfbd3681fbb2e70325f3e50f56ccd30dbd9d",
"tests/test/map_operations.scm": "0cebddb3d4a50e4b6f6bfc0e8c12762d3f42609030aef077d8ab4bedda4718d9",
"tests/test/memoization.scm": "cf8202df22407dbba72ca7f660d8ab7cd4ca962b215c36cc491d5c1a474c160b",
"tests/test/mutual_recursion.scm": "a3dc23a96ec1cf05b0be4a4f0b6d3da9e2bad1aa3f7a83c7566f71f980a67fe2",
"tests/test/nested_defines.scm": "21faa0ccaf744cb8c9d283bbfd0c36e195c803c85f511e70a99267c0a85b6216",
"tests/test/oeis_sequences.scm": "d1f77161b367176b3dfab55a34466e511c602c7ae359e05f5a715627b93dca6a",
"tests/test/oeis_sequences2.scm": "5679ef0f99743114d8c06edcdcde1fea9474b71aeb57c2069f1ee33df1970bce",
"tests/test/oeis_sequences3.scm": "5dc8b44158b2648a6e9766f6bcdf21553a07bf487d79e64ebb418511ef3b8186",
"tests/test/recursive_structures.scm": "67838ff5bac26ea359139720adec1d5700176147b0062065c7def9715e8a2aeb",
"tests/test/test_read.scm": "5b9ee381c3cf45b2fc8c2e7b490857fb67f1ac78653b586d7f2848daf6853809",
"tests/test/variadic_functions.scm": "c9b847c50840c6977904e5033a07f6108cc3a54012b02d916e79025fb43e0f6e",
"tests/test/y_combinator.scm": "971144dcb2b04aaa3b34d9279459bd7d2e8256c585d17809465848c2589d36d8",
"tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334",
"tests/test_outputs.py": "fbb84e3c0c2f72ed7de90f54380f860052c16155e7d0c9076cd57c966e9be7ed"
}
}
],
"source_sha256": {
"src/nanopycodeagent/__init__.py": "b922299f65b3db2fb1514df5aeda4413e599f595d2595f45a353b4dc551a1eb1",
"src/nanopycodeagent/agent.py": "8e28a5c3fcd140441484c46d1e62ec811d325aaff484141a4b9837baead6fc52",
"src/nanopycodeagent/atif.py": "ca4793b971d125d904888b858452db8db30e21f0d9b87818133062fb6cebe270",
"src/nanopycodeagent/bash_tool.py": "dcf0366fb9ebdca258d4c40b1a50b78b2ec79fc4b85cbb253ac9a9ce9534574f",
"src/nanopycodeagent/cli.py": "e335789b3f09440abd441547a2e3e2feb32b90c68e9ba5fc8237de8f6bac2932",
"src/nanopycodeagent/cost.py": "30a8df17942c424796e8532f45e1cb0784ed6541cea44284b863db3dae2f7b70",
"src/nanopycodeagent/deadline.py": "26a861721e6897a4e02e6ce364cb3ef1f1dc7d4ad9fb2f772404276f5c9e9d5f",
"src/nanopycodeagent/edit_tool.py": "78e2bec911279082b081366ab05cb4cbb37d8910b20b6245cccd4b0424489d0d",
"src/nanopycodeagent/event_journal.py": "8974919c2d460afb52bc57ec37e7e271851290c0845d01b4ad0f9f760cde0814",
"src/nanopycodeagent/read_tool.py": "e0075da19f2e1c6ccf34a3056a0aec92398e5afe67869937a096c6f1e04181c3",
"src/nanopycodeagent/settings.py": "491bf5c6eca50eb4c5fe04b7bfe62a8bfc7d2e1fad05b8eeac1680c2fcf3674e",
"src/nanopycodeagent/terminal.py": "8d27b40976b2b6d476e5106dfea982b055115435b4ba16b7f758412a05b75f04",
"src/nanopycodeagent/tool_validation.py": "3a1a85a7222dd1504bdf58f37e65a51d11de13d16c4e7df42c72b4cd58b5e5e5",
"src/nanopycodeagent/transport.py": "152d3ff78be0e3b2d92e0abbbfb652d73466aff57ad316c06dc0e072411167fe",
"src/nanopycodeagent/write_tool.py": "7671e83464fa1e942f25742fc573d0712e32053b5fc0291257884b027f7f6206",
"benchmarks/harbor/src/harbor_adapter/__init__.py": "513ad49c721b3e73293095c6a0486a791483e9aa2b51e0e9ad4b7816af07e76b",
"benchmarks/harbor/src/harbor_adapter/__main__.py": "db89d582fba45aad72147317f595531af49c434e3b23c65f22d7491f0b10551d",
"benchmarks/harbor/src/harbor_adapter/_verifier_runner.py": "2eb73c3e76d7360f2fa16018f0e933797bf982e589236095b5f98750f3807a7c",
"benchmarks/harbor/src/harbor_adapter/adapter.py": "d1aa154ec6972d706f0725cfa69c56635506d4499dff2a129e2c957ea7034c02",
"benchmarks/harbor/src/harbor_adapter/bootstrap.py": "f499bfddfc3d29338cae15a7f8f6e9c66d19eb69ad9577e607ced4b1aa057be2",
"benchmarks/harbor/src/harbor_adapter/verifier.py": "ec26fdefd69be1c5c450c9ba69d749d4a9d320a2a170b8db34421f0573d9b493"
},
"configuration_sha256": {
"model-extraction.json": "f8baf52e182e6b6ab669644c904c1c961db5e6083c6c18b8d1479932a9b26588",
"scheme.json": "0bf57205bb0d82ea0342483a269eb995441a7c427fec40f0678f3a69b1a87b0b",
"install-only.json": "75e6b3f9d567c1c2db953deef82cc43ec8ea31c3c79ab15fa11599c68531aaaf"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"n_concurrent_trials": 2,
"n_attempts": 1,
"agents": [
{
"name": "harbor_adapter:NanoPyCodeAgent",
"model_name": "deepseek/deepseek-flash",
"override_setup_timeout_sec": 1800,
"kwargs": {
"max_turns": 100,
"max_tokens": 65536,
"time_budget_seconds": 720
}
}
],
"tasks": [
{
"name": "terminal-bench/model-extraction-relu-logits",
"ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17"
},
{
"name": "terminal-bench/schemelike-metacircular-eval",
"ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6"
}
],
"install_only": true
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"n_concurrent_trials": 2,
"n_attempts": 1,
"agents": [
{
"name": "harbor_adapter:NanoPyCodeAgent",
"model_name": "deepseek/deepseek-flash",
"override_setup_timeout_sec": 1800,
"kwargs": {
"max_turns": 100,
"max_tokens": 65536,
"time_budget_seconds": 720
}
}
],
"tasks": [
{
"name": "terminal-bench/model-extraction-relu-logits",
"ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17"
}
]
}
22 changes: 22 additions & 0 deletions benchmarks/harbor/configs/truncation-recovery-20261005/scheme.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"n_concurrent_trials": 2,
"n_attempts": 1,
"agents": [
{
"name": "harbor_adapter:NanoPyCodeAgent",
"model_name": "deepseek/deepseek-flash",
"override_setup_timeout_sec": 1800,
"kwargs": {
"max_turns": 100,
"max_tokens": 65536,
"time_budget_seconds": 2220
}
}
],
"tasks": [
{
"name": "terminal-bench/schemelike-metacircular-eval",
"ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Three additional runs per task

This [predeclared plan](experiment.json) follows the earlier
[two-task experiment](../../reports/truncation-recovery-20261005.md), which
passed both tasks without exercising truncation recovery. Run each task three
more times from scratch, preserving all six results. These are additional
samples, not replacements for earlier scores.

Keep the same agent/adapter Python source hashes, task revisions, images, model,
endpoint, token cap, reply cap, and work/setup/verifier budgets as the preceding
experiment. Runtime behavior is from commit
`e1be6ed1f81a17d73677b414d4d913adb81e9e25`; the runner records the exact checkout
commit and built wheel for this round.

From the repository root, configure official DeepSeek credentials, set
`ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic`, and leave
`ANTHROPIC_MODEL` unset. Run installation preflight first:

```bash
uv run --project benchmarks/harbor python -m harbor_adapter run \
--config benchmarks/harbor/configs/truncation-recovery-repeat3-20261005/install-only.json \
--install-only
```

Both verifier dependency profiles must explicitly report `not_configured`.
After installation succeeds, run the six entries in `job_queue`, taking each
configuration path relative to this directory. Limit concurrent jobs to two:
each configuration contains one task and one attempt. For example:

```bash
uv run --project benchmarks/harbor python -m harbor_adapter run \
--config benchmarks/harbor/configs/truncation-recovery-repeat3-20261005/scheme.json \
--job-name truncation-recovery-repeat3-scheme-1-20261005
```

Use fresh job names when reproducing the experiment; the runner never overwrites
existing results. No whole-trial retries or replacements are allowed. Only
verifier timeouts may retry, at most three attempts with the original per-attempt
limit and the same agent output. Record truncation, actual recovery, subsequent
tool execution, terminal outcome, score, and accounting completeness per trial.
A passing trial without truncation does not demonstrate recovery effectiveness.

The recorded run used checkout `382cb38413d7d5368ad1be97e85142f280172a5b`.
See the [six-run report](../../reports/truncation-recovery-repeat3-20261005.md)
and [structured results](../../results/tb21-truncation-recovery-repeat3-20261005.json).
Loading
Loading