Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 149 additions & 0 deletions docs/dev_notes/en/0.10.x.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,3 +157,152 @@ The [six-run report](../../../benchmarks/harbor/reports/truncation-recovery-repe
and [fixed configuration](../../../benchmarks/harbor/configs/truncation-recovery-repeat3-20261005/README.md)
preserve the recovery timestamp, remaining budget, subsequent model/tool
activity, and paths to the original logs for review.

### Next round: research general prompts from benchmark evidence

Start prompt research on the `research/agent-system-prompts` branch on
2026-10-05. Record the motivation first, then update the reference projects,
archive and analyze individual prompts, and derive directions that can be
validated independently.

The evidence now separates runtime mechanisms from solution quality. The
frozen v0.9.0 baseline is 18/20. The first truncation-recovery experiment passed
2/2 without exercising recovery. The six follow-up trials passed 4/6; the only
trial with natural recovery continued tool work and produced artifacts but
still failed official verification. Another extraction failure had no
truncation, and all six runs ended normally. Thus `completed`, artifact
existence, and a passing visible example do not individually establish task
correctness. Fresh results also cannot be combined into a revised baseline.

The original trajectories suggest a hypothesis to test. The two failed
extraction runs mainly checked the visible instance. The passing run also
checked different input conditions, network sizes, and output residuals, then
fixed issues exposed by those checks. An earlier PyTorch pipeline trial also
passed its own checks while the independent verifier rejected its numerical
results. These observations support investigating requirement coverage,
independent validation, and completion decisions. The small sample and
different solutions do not establish that stronger prompts will improve pass
rates or identify every algorithmic root cause.

The current `HEADLESS_SYSTEM_PROMPT` requires autonomous execution, tool-based
verification, and saving deliverables when budgets run low. The shared guidance
mainly explains the four tools. It does not yet spell out how to distinguish
examples from general requirements, check critical assumptions, select
independent or boundary checks, or decide completion from evidence. Long
reasoning that hits the output cap before creating a script also suggests a
separate experiment on obtaining executable feedback earlier. Successful
trajectories can contain long reasoning too, so this does not establish that
shorter reasoning is always better.

The aim is general agent capability: understand constraints, inspect the
environment, act appropriately, verify completeness, diagnose failures, and
report results honestly within the existing time and reply budgets. Task names,
hidden-test parameters, specific algorithms, and reference answers must not
enter runtime prompts. Separate system prompts, dynamic context, tool
protocols, and runtime mechanisms when surveying other projects. Avoid
attributing tool capabilities to wording alone or describing a chat product's
model prompt as the complete runtime prompt of a coding agent.

Research deliverables include each prompt's source, revision, original text or
canonical source location, Chinese explanation and corresponding English
version, plus the cross-project `agent_system_prompt.md`. Compare candidates
independently against current main with the same model, endpoint, task
revisions, and budgets; preserve every trial, then check the original pilot20
and tasks not used for tuning. Record prompt changes, observed results, and
claims of benefit separately.

### General prompt study: source archive and initial candidate

All six reference repositories were fetched and updated to their tracked
branches. Codex, OpenCode, Pi, and Grok Build had new commits; DeepSeek Harness
and the local claude-code repository were already current. Remotes, branches,
before-and-after SHAs, and update times are recorded, with individual originals
pinned to commits so historical files are not mistaken for current defaults.

The study adds 42 prompt records, each in Chinese and English: Codex 10,
OpenCode 13, Pi 4, Grok Build 4, DeepSeek Harness 4, this project's current
baseline 4, and official Claude pages 3. Complete texts, dynamic templates,
specialized-agent instructions, builder source, and test snapshots are
distinguished. The first version kept English originals and licenses in both
language versions and translated only the analysis. Prompt-body translations
are completed in the update recorded below.

The specified Claude pages describe claude.ai and mobile apps, not complete
Claude Code or API runtime prompts. The initial study covered only Opus 5.5,
Sonnet 5.5, and Fable 5.1, preserving official source links, short excerpts, and
bilingual analysis. Full webpage copies and translations were not archived at
that stage. The later update below records the full archive and translation of
text supplied by the user.

The comparison prioritizes requirement coverage, independent validation,
failure diagnosis, and evidence-based completion. Earlier runnable feedback
is a separate later variable, without directly limiting reasoning length.
Candidate text contains no specific tasks, hidden verification parameters,
reference algorithms, or answers, and introduces no unavailable tools or
runtime mechanisms.

The [comparative study](../../research/en/agent_system_prompt.md) provides
candidate English text, gaps in the current prompts, and an experiment design;
the [individual index](../../research/en/agent_system_prompts/README.md) links
every record. English versions have been generated in full and are in sync with
the Chinese sources. This change adds research and development records only:
runtime prompts are unchanged, no new paid benchmark has run, and no improvement
is claimed.

### Prompt archive: complete Chinese body translations

On 2026-10-06, following review feedback, the “Original text” sections in all
39 local-source Chinese records were replaced with “Chinese translation,”
translating prompt bodies, examples, and rules paragraph by paragraph. English
originals remain under `docs/research/en/agent_system_prompts/`, with translations
in the corresponding `zh-CN/` directory. Tool names, paths, code identifiers,
template placeholders, and machine-read format markers remain unchanged. The
Pi and budget-reminder builder records translate comments and prompt strings
while preserving code structure. They are explicitly source-code reading
translations, not captured runtime requests.

The manifest adds each translation's path, SHA256, corresponding original
SHA256, and date. English originals remain unchanged. English versions of the
individual analyses, indexes, comparative study, and these development notes
are synchronized. Validation separately checks English originals against their
pinned sources, translations against manifest hashes, template placeholders,
and local links.

Before the user supplied the bodies, the three Claude records still contained
only excerpts: `en/` held short English quotes and `zh-CN/` their Chinese
translations, alongside source links and scope notes. Full webpages had not
been fetched at that stage, and these three were excluded from the 39 full
translations above. The next section records their subsequent full archive.

This update still changes documentation only. It does not modify agent runtime
prompts or start a new benchmark.

### Claude prompts: archive and translate user-provided full originals

On 2026-10-06, the user placed the complete English Markdown bodies for Opus 5.5,
Sonnet 5.5, and Fable 5.1 under
`docs/research/en/agent_system_prompts/claude/`, requesting full translations and
a combined commit. This update uses those local files as translation inputs,
preserves their English bodies byte for byte, and translates all page metadata,
prompt paragraphs, and examples into the corresponding `zh-CN/` files. XML
tags, model identifiers, URLs, and placeholders such as `{{currentDateTime}}`
remain unchanged. Both originals and translations are included in version control.

The three bodies contain 183 paragraphs, including 120 distinct paragraphs.
Shared passages use consistent translations, while version differences in
refusal handling, formatting, crisis support, replies after tool calls, and
concision are preserved. Product information and behavior rules are translated
as supplied; they are not this project's active runtime instructions, and the
translation does not change their rules or factual assertions.

Outdated excerpt-only descriptions were removed from the research documents,
and the individual indexes, comparative study, and source manifest were updated.
The manifest explicitly records user-provided acquisition, document hashes at
receipt, and original and translated body hashes. It preserves the supplied
snapshots without claiming a new webpage fetch. All 42 records now contain the
complete selected original and its Chinese translation. English analysis and
these development notes are synchronized.

Validation covers unchanged user-provided English bodies, paragraph coverage,
XML tags and placeholders, model identifiers, URLs, Markdown fences, and local
links. This update changes documentation only and runs no new benchmark.
98 changes: 98 additions & 0 deletions docs/dev_notes/zh-CN/0.10.x.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,3 +119,101 @@ Scheme 都没有截断,仍不能据此判断恢复逻辑能否救回 Scheme;
[逐次结构化结果](../../../benchmarks/harbor/results/tb21-truncation-recovery-repeat3-20261005.json)
和[固定配置](../../../benchmarks/harbor/configs/truncation-recovery-repeat3-20261005/README.md)
保留了恢复时间点、剩余预算、后续模型与工具活动,以及原始日志的复核路径。

### 下一轮:从 benchmark 证据出发研究通用提示词

2026-10-05 在 `research/agent-system-prompts` 分支开始提示词调研。先记录本轮
动机,再更新参考项目、逐份保存和分析提示词,最后形成可独立验证的优化方向。

当前证据已经区分了运行机制与解答质量:0.9.0 的冻结 baseline 是 18/20;首次
截断恢复实验 2/2 通过,但均未触发恢复;追加实验 4/6 通过,其中唯一真实恢复的
运行继续执行了工具并生成产物,却仍未通过官方验证。另一次模型提取失败没有
截断,六次运行全部正常结束。因此,`completed`、文件存在、可见示例通过都不能
单独代表任务正确完成,也不能把复跑结果合并成新的 baseline。

原始轨迹提供了一个待验证的方向:两次失败主要验证当前可见实例;通过的模型
提取运行还检查了不同输入条件、模型规模和输出残差,并根据额外检查修复问题。
此前 PyTorch pipeline 也出现过自测通过、独立 verifier 拒绝数值结果的情况。
这些观察支持研究需求覆盖、独立验证和完成判断,但样本少、解法不同,尚不能
认定加强提示词一定提高通过率,或者已经确定所有错误的算法根因。

当前 `HEADLESS_SYSTEM_PROMPT` 要求自主执行、使用工具检查结果,并在预算不足时
保存交付物;共享提示词主要说明四个工具的用途。它们尚未具体表达如何区分示例
与一般要求、如何检查关键假设、如何选择独立或边界验证、以及如何用证据判断
已经完成。长推理在创建脚本前触及输出上限,也提示可以单独研究更早获得可运行
反馈的指导;成功轨迹同样可能包含长推理,因此不能直接推导出缩短推理一定更好。

本轮目标是提升 agent 的通用能力:理解约束、查阅环境、合理行动、验证完整性、
定位失败原因、诚实报告结果,并与现有时间和回合预算兼容。任务名称、隐藏测试
参数、特定算法和参考答案都不能进入运行提示词。调研会区分系统提示词、动态
上下文、工具协议和运行时机制,避免把别的项目依赖的工具能力当成文字本身的
效果,也避免把面向聊天产品的模型提示词误称为 coding-agent 的完整运行提示词。

调研产物包括各份提示词的来源、版本、原文或原文入口、中文解释和对应英文版,
以及 `agent_system_prompt.md` 的横向比较。候选方向需要在固定模型、端点、
任务版本和预算下与当前主线做独立对照,保留全部试次,再检查原 pilot20 回归
和未用于调优的任务。提示词改动、运行结果和收益结论应分别记录。

### 通用提示词调研:来源归档与首轮候选

六个参考仓库已 fetch 并更新到各自跟踪分支:Codex、OpenCode、Pi 和 Grok Build
有新提交;DeepSeek Harness 和本地 claude-code 已是最新。记录远端、分支、
更新前后 SHA 和更新时间,逐份原文固定到提交,避免把历史文件误当当前默认。

本轮新增 42 份提示词记录,各有中英文版本:Codex 10、OpenCode 13、Pi 4、
Grok Build 4、DeepSeek Harness 4、本项目当前基准 4、Claude 官方页面 3。
区分完整文本、动态模板、专用代理指令、生成器源码和测试快照。初版在两种语言
文档中均保留英文原文及许可证,仅将解读同步为双语;正文翻译在下节记录中补齐。

指定的 Claude 页面属于 claude.ai 与移动端,不是 Claude Code 或 API 的完整
运行提示词。初版只研究 Opus 5.5、Sonnet 5.5 和 Fable 5.1,并保存官方原文
入口、短引文与双语分析;当时未归档网页全文或全文翻译。用户随后提供正文的
完整归档与翻译见下节更新记录。

横向比较建议优先补充需求覆盖、独立验证、失败诊断和有证据的完成判断。
“更早获得可运行反馈”作为独立的后续变量,不直接限制推理长度。候选文本不含
特定题目、隐藏验证参数、参考算法或答案;不引入当前不存在的工具或运行机制。

[综合调研](../../research/zh-CN/agent_system_prompt.md)给出候选英文文本、现有
提示词差距和实验设计;[逐份索引](../../research/zh-CN/agent_system_prompts/README.md)
链接全部资料。对应英文版本已完整生成并与中文源同步。当前提交仅新增研究与
开发记录,没有修改运行提示词,没有运行新的付费 benchmark,也不宣称已有收益。

### 提示词归档:补齐中文正文翻译

2026-10-06 按评审要求,将 39 份本地来源记录的“原文”节替换为“中文译文”,
逐段翻译提示词正文、示例和规则。英文原文集中保留在 `docs/research/en/agent_system_prompts/`,
译文保存在对应的 `zh-CN/` 目录;工具名、路径、代码标识符、模板占位符和机器
读取的固定格式标记保持原样。Pi 和预算提醒的生成器资料翻译注释与提示词字符串,
保留代码结构,并明确它们是供阅读的源码译本,不是实际请求的运行捕获。

来源清单增加译文路径、译文 SHA256、对应原文 SHA256 和翻译日期;英文原文保持
不变。逐份解读、索引、综合调研和本开发笔记的英文版本一并同步。校验分别核对
英文原文与固定来源、译文与清单的哈希,以及模板占位符和本地链接。

在用户提供正文之前,三份 Claude 记录仍只有短引文:`en/` 保留英文短句,
`zh-CN/` 保留对应中文译文,来源链接和范围说明继续保留。当时未抓取网页全文,
这三份也未计入上述 39 份全文翻译。后续完整归档见下节。

本次仍仅修改文档,没有修改 agent 运行提示词或启动新的 benchmark。

### Claude 提示词:归档并翻译用户提供的完整原文

2026-10-06,用户将 Opus 5.5、Sonnet 5.5 和 Fable 5.1 的完整英文 Markdown
正文放入 `docs/research/en/agent_system_prompts/claude/`,要求全文翻译并整体
提交。本次以这些本地文件为翻译输入,逐字保留英文正文,将全部页面元数据、
提示词段落和示例翻译到对应 `zh-CN/` 文件;XML 标签、模型标识、网址和
`{{currentDateTime}}` 等占位符保持原样。原文与译文均纳入版本控制。

三份资料共 183 个正文段落,其中有 120 个不同段落;共享段落采用一致译法,
不同版本在拒绝处理、格式、危机支持、工具调用后回复和简洁程度上的差异分别
保留。提示词中的产品信息及行为规则按所提供版本翻译,不作为本项目当前运行
指令,也不借翻译修改其规则或事实陈述。

移除研究文档中已经过时的“仅短引文”说明,更新逐份索引、综合调研和来源清单。
清单明确记录原文由用户提供、提供时的文件哈希,以及原文和译文正文哈希;
这保存了用户提供的快照,不声称本次重新抓取了网页。全部 42 份记录现已具备
完整的选定原文和中文译文,英文解读及开发笔记同步更新。

校验覆盖用户提供的英文正文未变、逐段翻译覆盖、XML 标签与占位符、模型标识、
网址、Markdown 围栏和本地链接。本次只修改文档,未运行新的 benchmark。
6 changes: 6 additions & 0 deletions docs/research/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,9 @@ versions are in sync.
- How Pi, Claude Code, Codex, OpenCode, and Grok Build separate final output,
live JSON event streams, persistent sessions, and trajectory artifacts:
[English](en/agent_output_and_trajectory.md) | [Chinese](zh-CN/agent_output_and_trajectory.md)
- Updated agent prompt sources, their content directions, and general
improvements motivated by benchmark evidence:
[English](en/agent_system_prompt.md) | [Chinese](zh-CN/agent_system_prompt.md).
Individual prompt records: [English](en/agent_system_prompts/README.md) |
[Chinese](zh-CN/agent_system_prompts/README.md), with a shared
[source manifest and licenses](agent_system_prompts/README.md).
Loading
Loading