From 358998d77b1f455de45c8531f1239a3c9c82f4b2 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 2 Nov 2026 02:08:27 +0000 Subject: [PATCH] docs: promote CHANGELOG Unreleased (post-0.2.0) to 0.3.0 Move bullets added after the v0.2.0 tag into ## 0.3.0 - 2026-10-30 and backfill ## 0.2.0 - 2026-10-14 from the tag-time Unreleased section. The published v0.3.0 tag and GitHub Release are unchanged. Co-authored-by: Hongsheng Liu --- CHANGELOG.md | 186 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 116 insertions(+), 70 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 224de25..fae851b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,18 +4,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver ## Unreleased -### Fixed - -- Served OneJev uses one HTTP decision deadline across its permitted retry. A retry receives only the remaining - `ONEJEV_TIMEOUT_S` budget, and an exhausted budget fails without sending another request. - -### Changed - -- The direct TypeSafe path pins `jev-1.13.0` instead of the moving `jev-latest` alias (which moved to Jev 1.14.0 - after every published number here was recorded on 1.13), and `TYPESAFE_MODEL` now overrides the model on that - path too; an OpenRouter-style `typesafe/...` id there is a config error. Jev step records (tool and browser - ticks) keep `requested_model` next to `model`, so a saved run says which Jev was asked for and which answered. - The 1.13.0 vs 1.14.0 benchmark re-run is still to do. See #93. +## 0.3.0 - 2026-10-30 ### Added @@ -31,52 +20,71 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver without `--order`. `--order same:N` is the control: N more asks in file order, so a backend that disagrees with itself (sampling, a load-balanced server, GPU non-determinism) shows its own "flip" rate next to the reordered one. `--seed` or `--json` without `--order` is a usage error (exit 2) rather than silently ignored. See #90. + - `scripts/build_browsesafe_set.py`: a seeded, label-stratified BrowseSafe-Bench test sample (row ids in `evals/labelled/browsesafe-test-sample.json`, dataset not vendored) in the injection guard's labelled-set shape, and the share of page text the guard inspects at its 6,000-character clip (`docs/benchmarks.md`). Every command, including one given a local `--parquet`, refuses a file whose sha256 is not the pinned one. + - Offline quality consumer for shared Omni public231 raw, retaining planned denominators and original scoring. + - `s1a-mcp` tools carry titles and MCP behavior hints: `list_agents` and `decide` are read-only (so hosts with read-only auto-approval stop prompting for them), `run_agent` is destructive and open-world. `decide` and `list_agents` advertise typed output schemas; the returned content is unchanged. `docs/skills.md` lists every `decide` model (`jev`, `clm`, `laya`, `laya-served`, `cua`). -- Public231 decision-model benchmark with raw HTTP evidence, pinned upstream scoring, - complete-denominator quality metrics and per-round service latency. - -- Optional local Cua-S1 4B inference through the `cua-four-b` extra and `CUA_S1_VARIANT=4b`, supporting text and - multimodal input. Desktop `--pixel-target` offers named screenshot points; clicks remain bound to the observed - window and capture. - `jev-vl-served` in the browser front and decision-model factory: JEV-27B-VL through System1-Omni's native flat HTTP protocol, with a mandatory current PNG/JPEG, choice/noul mapping and a shared decision deadline. Local tests use mocks; single-H800 GPU checks and the bounded R3 browser fixture with same-trial video are validated. See the [validation scope](docs/decision-models.md#served-jev-vl). -- Verified desktop text input through `--text`, `--text-target`, and `--text-mode` (insert or replace). - Completion requires confirmed input and fresh field readback; `--verify-file` also requires matching contents - in a freshly written file. `--window-title` selects an exact window and `--app-path` launches a macOS app bundle. -- Snake recipe and game client: `evals/snake` vendors the laya-mlx snake CLI - (Apache-2.0) with single-game paced recording and a 16-game multigrid mode - against any `/v1/systemone` backend; `recipes/snake` documents setup, - verification, and recorded evidence (0 deaths; playback-speed-1 GIFs). + +- `evals/labelled/injection-public.jsonl`: 665 labelled tool outputs for the `injection_guard` rail from InjecAgent + and AgentDojo at pinned versions, 335 with an injected instruction. 65 of those read as an ordinary request that only + the user's task tells apart; their notes carry a `-u` source and they are reported on their own row. Negatives + include requests written for a human (`R2` in the note), which with the unmarked positives form a hard subset. + `uv run scripts/build_injection_dataset.py` rebuilds the file byte for byte; `--report ` prints precision and + recall per source without the `-u` positives, their own recall, and balanced accuracy on the hard subset against + fixed keyword baselines. The Jev run of 2026-10-05 behind `docs/benchmarks.md` is archived on #23 + ([records](https://github.com/user-attachments/files/33183041/injection-guard-jev-2026-10-05.zip)). ### Changed +- The direct TypeSafe path pins `jev-1.13.0` instead of the moving `jev-latest` alias (which moved to Jev 1.14.0 + after every published number here was recorded on 1.13), and `TYPESAFE_MODEL` now overrides the model on that + path too; an OpenRouter-style `typesafe/...` id there is a config error. Jev step records (tool and browser + ticks) keep `requested_model` next to `model`, so a saved run says which Jev was asked for and which answered. + The 1.13.0 vs 1.14.0 benchmark re-run is still to do. See #93. + - `mcp` is bounded below 2 (`mcp>=1.26,<2`): openjiuwen's fastmcp dependency still requires the 1.x SDK. `docs/architecture.md` § MCP SDK 2.x lists the 1.x-only call sites and the condition for porting. +- `Choice.confidence` / `Noul.confidence` now mean the same thing on every backend: `max_k p_k` for a choice and + `max(p, 1 - p)` for a noul, both computed in `validation.py` from the validated probabilities. Historical `conf=` + values from `laya` / `laya-served` (normalised entropy), `jev` / `clm` (service-reported) and `omnijev` (Jev margin) + are not comparable with new ones. Each backend's own number is kept as `backend_confidence` on the answer, in + `Decision.raw["backend_confidence"]`, and on tool/browser tick records when present (browser ticks also carry + `backend_confidences`, every answered head's number by question name). A backend's own `raw["backend_confidence"]` + is kept as `raw["backend_confidence_reply"]` instead of being overwritten. See #89. + ### Fixed +- Served OneJev uses one HTTP decision deadline across its permitted retry. A retry receives only the remaining + `ONEJEV_TIMEOUT_S` budget, and an exhausted budget fails without sending another request. + - JEV-VL browser raw traces stay scoped to `jev-vl-served`, retain only timing, request ID and content headers, and use null request IDs for local forced choices instead of implying an HTTP request was sent. + - `s1a-mcp` long runs: `run_agent` reports progress per episode or browser step when the request has a `progressToken`; a call made while another runs fails at once with a tool error instead of queueing past the host's timeout; a cancelled run finishes its browser and model teardown (it was cut at the first `await`) and records the cancellation in the job `result.json` or `answer.json` even when no episode finished; cancelled seeds count in `n_total_trials`. The server instructions put serialization and the timeout advice in their first 512 characters, and docs/skills.md gives the Codex and Claude Code settings. + - `decide` rejects repeated `--option` keys before model setup instead of silently replacing an earlier choice description. The usage error names the duplicate key without printing either description. + - The injection guard no longer passes the uninspected end of a long result: below the act band, a `browser_snapshot` or `fetch_webpage` result longer than the 6,000 characters its decision reads reaches the model as those characters plus a withheld notice. Rail ticks record `handling` (`pass`, `pass_uncertain`, @@ -84,56 +92,38 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver policy is in `docs/agents.md`. `RailSpec` gains optional `coverage_of` and `on_partial`. A failed decision's tick and warning log record the error's status and cause type (`error`, `error_code`) instead of its message, which can echo an HTTP error body quoting the guarded text. + - A rail's labelled-set evaluation sends each state through the spec's runtime clip (`RailSpec.labelled_state`; the injection guard cuts text to 6,000 characters) and records a failed decision as a `failure` row instead of aborting the run; the summary adds `failures` and per-label band counts. + - Malformed ticket-router JSONL now names the dataset, physical line and JSON column without printing ticket contents. -- Laya browser-state compaction preserves the recovery plan for the next decision. -- Desktop recovery resets its detection window on observed progress without resetting its cumulative budget. -- Screenshot-enabled desktop recovery detects pixel changes while ignoring capture IDs and PNG metadata. -- Recovery errors omit provider exception bodies, and tool episodes retain the full structured terminal and - permission-flow next step separately from shortened display output. -- Recovery evaluation preserves timeouts after verified submissions, escapes submitted result text, and keeps - missing decision usage distinct from reported zero tokens. Paired reports share call-count conventions. -- Custom recovery tasks now reach each trial's fixture with their requested routes, page behavior and submission - validation. -- Bounded recovery treats an empty, whitespace-only or otherwise blank planner answer (browser and desktop) as a - planner failure: the run stops with the existing reason and next action, the one attempt and its active seconds - stay charged and the fresh observation is kept, instead of recording a `planned` event with an empty plan. -- Fit-probe cases with no options, no accepted answer, or an accepted key outside the offered options now fail - input validation instead of skewing the fit verdict. -- Windows development checks: the smoke script accepts CRLF output, shell scripts and Git hooks retain LF - line endings, and tests check socket closure and invalid output directories without Unix-specific behavior. - The core CI matrix now covers Windows with Python 3.11. -- Rail, tool, and browser evaluations charge Jev-rate input tokens only when the decision backend declares - them billable. Local model token usage remains recorded without Jev API charges. -- Browser front: a WAIT whose in-page settle moved the page now records `page_changed: true` in the history, so - the next state no longer shows that wait as unmeasured. -### Changed +## 0.2.0 - 2026-10-14 -- `Choice.confidence` / `Noul.confidence` now mean the same thing on every backend: `max_k p_k` for a choice and - `max(p, 1 - p)` for a noul, both computed in `validation.py` from the validated probabilities. Historical `conf=` - values from `laya` / `laya-served` (normalised entropy), `jev` / `clm` (service-reported) and `omnijev` (Jev margin) - are not comparable with new ones. Each backend's own number is kept as `backend_confidence` on the answer, in - `Decision.raw["backend_confidence"]`, and on tool/browser tick records when present (browser ticks also carry - `backend_confidences`, every answered head's number by question name). A backend's own `raw["backend_confidence"]` - is kept as `raw["backend_confidence_reply"]` instead of being overwritten. See #89. +### Added -- Review guidance (`self-review`, `review-pr`) now treats merge readiness as a gate: authors confirm the pull - request reports mergeable with CI green on the recorded head before requesting review, and reviewers do not - approve a conflicting branch or a run pinned to an older commit. -- `review-pr` reads for readability as well as correctness: names that say what they do, reviewable function - length, and the repository's house idioms, judged against those conventions rather than personal style. -- Core Windows CI (`core (windows, 3.11)`) runs on pushes to `main` and the weekly schedule, not on every - pull request. Linux `core` and `full` still run on PRs. +- Public231 decision-model benchmark with raw HTTP evidence, pinned upstream scoring, + complete-denominator quality metrics and per-round service latency. -### Added +- Optional local Cua-S1 4B inference through the `cua-four-b` extra and `CUA_S1_VARIANT=4b`, supporting text and + multimodal input. Desktop `--pixel-target` offers named screenshot points; clicks remain bound to the observed + window and capture. + +- Verified desktop text input through `--text`, `--text-target`, and `--text-mode` (insert or replace). + Completion requires confirmed input and fresh field readback; `--verify-file` also requires matching contents + in a freshly written file. `--window-title` selects an exact window and `--app-path` launches a macOS app bundle. + +- Snake recipe and game client: `evals/snake` vendors the laya-mlx snake CLI + (Apache-2.0) with single-game paced recording and a 16-game multigrid mode + against any `/v1/systemone` backend; `recipes/snake` documents setup, + verification, and recorded evidence (0 deaths; playback-speed-1 GIFs). - `--model clm`: CLM's `clm-serve` behind the decision-model interface, over its `/v1/systemone`. The engine owns everything after a frozen Qwen3-8B encoder, so no torch and no cloud key are needed on this side. Offered everywhere the other HTTP models are: `run`, `decide`, `probe`, rails, the browser front and the MCP server. See [docs/clm.md](docs/clm.md) and the [ticket-router evidence](evals/ticket_router/CLM.md). + - Bounded recovery on the browser front: `--rethink on|off` (default off), `--rethink-attempts` (3) and `--rethink-timeout` (15 s). A stall (no page change, an A-B-A-B loop, a repeated URL, a WAIT that moved nothing) re-probes the page read-only through the same runtime permission and asks the chat model for a plan under a @@ -142,50 +132,65 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver seconds, history and ticks stay. `report()` and `decision_ticks.json` keep the recovery events, counts and termination, and a timed-out, failed or exhausted recovery is a clear `BLOCKED` even when a summary carries text. `--model llm` with `--rethink on` is rejected before the agent or browser is built. + - The desktop agent's bounded recovery takes the same `--rethink-attempts` and `--rethink-timeout` names and stalls after 3 actions without progress. Recovery failures include an operator next action; failed or cancelled chat calls remain counted, and incomplete token usage is reported as unknown cost. + - Small repeatable browser and native Windows recovery on/off fixtures, with independent completion checks, bounded failure cases and paired reports. These use scripted models to verify mechanisms, not model accuracy. + - `.coderabbit.yaml` and the `review-pr` skill: CodeRabbit reviews every non-draft pull request except Dependabot's, forks included, once an owner installs the CodeRabbit GitHub App, against `.claude/skills/review-pr/SKILL.md` and the linked issues. Maintainers can run the same skill by hand. + - Agent use-case recipe index, authoring template and contributor skill, with a runnable ticket-routing example and independent fixture verification. + - Contributor guidance for recording and attaching agent video demos, identifying the inference engine and checking supported System1-Omni paths, linked from the builder/self-review skills and PR template. + - `--model laya-served`: Laya served over HTTP by system1-omni's worker (or plain laya-serve), on every front that takes `laya`, in `decide` and in MCP `decide`. Configured by `LAYA_SERVED_URL` and optional `LAYA_SERVED_*` variables; no cloud key. Run records name the served checkpoint, revision and device. Design, API spec and the run steps: `docs/served-laya.md`, `docs/api/`. + - Tool-front ticks keep the answering model as `model`, and a served model's `served_by`, `url`, `request_id` and `server_timing`. + - `--model omnijev` on the browser agents: [OmniJev](https://github.com/tinnel123666888/OmniJev) (Apache-2.0), a Qwen3.5 vision-language decision model, in process behind `uv sync --extra omnijev` and a local clone named by `OMNIJEV_REPO`. It decides over a screenshot; `docs/decision-models.md`, `docs/configuration.md`. + - Browser front: a decision model that reads images (`supports_images`) gets a viewport PNG with every tick's observation, captured through the probe's run-code executor (a failed capture is tried once more; with no image OmniJev fails the call as `MODEL_CALL_FAILED`); the tick records its `screenshot_ms`. Jev, Laya and Cua-S1 read text only and see no change. + - `S1A_DECISION_TIMEOUT_S`: the deadline of one decision on the `jev` backend, 5 s when unset. A local System One server behind `TYPESAFE_API_URL` can be slower than Jev: on Google Flights, OneJev-27B on an A100 takes about 3.7 s a decision and more on the calendar page, so the 5 s deadline stopped every run at the eighth step; with 30 s it completed the task. `docs/configuration.md`. + - Every browser run saves the page the task ended on as `final.png` in `--logs-dir` before the browser is released, however the task ends, a timeout included. The answer and `answer.json` name the file in `screenshot`. A judge that grades the end state can read it, as Harbor's WebVoyager judge does at `/logs/agent/final.png`. A failed screenshot leaves `screenshot` null, records the exception type in `screenshot_error` and changes nothing else in the answer. A `final.png` an earlier run left in a reused `--logs-dir` is removed when the task starts. + - The MCP `decide` tool accepts `model="jev"|"laya"|"cua"`, defaulting to `jev`. Local backends use their optional extras and need no Jev API key. + - `docs/benchmarks.md`: the Google Flights driver comparison rerun on 2026-09-23 from Poland, every arm three times on both decision backends, next to the baseline rows in one table; the 24 S1A records, as one archive, and the chart under `docs/results/flights/rerun-2026-09-23/`. + - `laya_state` (`s1a/decision_models/laya.py`): folds a browser-front state to fit Laya's 512 to 1024 token window before every call — `page.text` dropped, one short line per element row instead of a JSON object, the last three actions instead of ten, a probe flag such as `"expanded": "false"` read as off, and "(no change)" only on an action measured as unchanged — roughly a tenfold reduction in the JSON-shaped state on the pages measured. On by default; `LAYA_COMPACT_BROWSER_STATE=0` turns it off. `docs/decision-models.md`. + - `laya_browser_question` (`s1a/decision_models/laya.py`): with a folded browser state, each browser question reaches Laya as the goal and the operation (the agent's long rules dropped) and each target option as its element's label and value. Laya fits a question's instruction and all its options into one `head_max_len` @@ -193,34 +198,75 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver element name. `text_value` gets its own ask; options that shorten alike keep their key; a blocked row keeps its overlay's name. Browser runs want `LAYA_MAX_LEN=1536` and `LAYA_HEAD_MAX_LEN=1024`: a calendar page's target head measures about 900 tokens. -- `evals/labelled/injection-public.jsonl`: 665 labelled tool outputs for the `injection_guard` rail from InjecAgent - and AgentDojo at pinned versions, 335 with an injected instruction. 65 of those read as an ordinary request that only - the user's task tells apart; their notes carry a `-u` source and they are reported on their own row. Negatives - include requests written for a human (`R2` in the note), which with the unmarked positives form a hard subset. - `uv run scripts/build_injection_dataset.py` rebuilds the file byte for byte; `--report ` prints precision and - recall per source without the `-u` positives, their own recall, and balanced accuracy on the hard subset against - fixed keyword baselines. The Jev run of 2026-10-05 behind `docs/benchmarks.md` is archived on #23 - ([records](https://github.com/user-attachments/files/33183041/injection-guard-jev-2026-10-05.zip)). ### Changed +- Review guidance (`self-review`, `review-pr`) now treats merge readiness as a gate: authors confirm the pull + request reports mergeable with CI green on the recorded head before requesting review, and reviewers do not + approve a conflicting branch or a run pinned to an older commit. + +- `review-pr` reads for readability as well as correctness: names that say what they do, reviewable function + length, and the repository's house idioms, judged against those conventions rather than personal style. + +- Core Windows CI (`core (windows, 3.11)`) runs on pushes to `main` and the weekly schedule, not on every + pull request. Linux `core` and `full` still run on PRs. + - Desktop click candidates omit disabled or unlabelled controls and application menu items, including in click-only tasks. + - Backend contribution guidance requires cold in-process model load timing, profiling above 5 s, and complete checkpoint coverage before skipping random weight initialization. + - `--model laya` loads in about 3 s instead of about 35 s: the encoder is built with transformers' weight init off, since the checkpoint replaces every weight. Weights and answers are unchanged. + - Important agent/inference PRs require an application + System1-Agents + System1-Omni video, following PR #35's worked example. Contributor skills, recipes and the PR template retain missing demos as review gaps. + - `--model laya` no longer draws the encoder's random weights before the checkpoint replaces them, which took most of a load of about 40 s on CPU. The `laya` extra now needs laya 0.3.9 or later, which skips the draw itself, and the lock moves from 0.3.5 to 0.3.20. Weights and answers are unchanged: laya 0.3.10 and later run a request of five or more questions in fp16 on MPS, which moves the answers, so `--model laya` keeps such requests in fp32 unless `LAYA_MPS_AMP_MIN_ROWS` is set. + - `--model` picks the model on every agent, on `decide` and on `probe`: `jev`, `laya`, `cua`, `llm`, `random` or `rule`. The results table's column, the replay page's badge data and a browser run's `answer.json` name it `model` as well; the replay still reads the `slot` key of records written by 0.1.0. +### Fixed + +- Laya browser-state compaction preserves the recovery plan for the next decision. + +- Desktop recovery resets its detection window on observed progress without resetting its cumulative budget. + +- Screenshot-enabled desktop recovery detects pixel changes while ignoring capture IDs and PNG metadata. + +- Recovery errors omit provider exception bodies, and tool episodes retain the full structured terminal and + permission-flow next step separately from shortened display output. + +- Recovery evaluation preserves timeouts after verified submissions, escapes submitted result text, and keeps + missing decision usage distinct from reported zero tokens. Paired reports share call-count conventions. + +- Custom recovery tasks now reach each trial's fixture with their requested routes, page behavior and submission + validation. + +- Bounded recovery treats an empty, whitespace-only or otherwise blank planner answer (browser and desktop) as a + planner failure: the run stops with the existing reason and next action, the one attempt and its active seconds + stay charged and the fresh observation is kept, instead of recording a `planned` event with an empty plan. + +- Fit-probe cases with no options, no accepted answer, or an accepted key outside the offered options now fail + input validation instead of skewing the fit verdict. + +- Windows development checks: the smoke script accepts CRLF output, shell scripts and Git hooks retain LF + line endings, and tests check socket closure and invalid output directories without Unix-specific behavior. + The core CI matrix now covers Windows with Python 3.11. + +- Rail, tool, and browser evaluations charge Jev-rate input tokens only when the decision backend declares + them billable. Local model token usage remains recorded without Jev API charges. + +- Browser front: a WAIT whose in-page settle moved the page now records `page_changed: true` in the history, so + the next state no longer shows that wait as unmeasured. + ## 0.1.0 - 2026-09-23 ### Added