[Feat] Find desktop click targets with Jev Box (--box-target) instead of hand-written --pixel-target

CHANGELOG.md

@@ -4,6 +4,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver
 
 ## Unreleased
 
+### Added
+
+- Experimental desktop `--box-target KEY=description` (#98): each step asks hosted Jev (1.14+, `S1A_JEV_IMAGES`) a
+  presence `noul` and a `box` per target, offers only those above uncalibrated `S1A_BOX_PRESENCE_THRESHOLD` /
+  `S1A_BOX_CONFIDENCE_THRESHOLD` (defaults 0.5), and clicks the box centre with the same capture binding as
+  `--pixel-target`. Decision-model layer gains `BoxQuestion` / `Box`. Offline tests use a fake transport; live
+  fixture measurement needs TypeSafe images. Helper: `scripts/measure_box_targets.py`. Depends on #97 / PR #99.
+
 ### Fixed
 
 - Served OneJev uses one HTTP decision deadline across its permitted retry. A retry receives only the remaining

docs/agents.md

@@ -50,7 +50,7 @@ If a call's token usage is unknown, the reported cost stays unknown rather than
 `--batch on|off`, `--prefetch on|off`, `--goal-values on|off`, `--rethink on|off`, `--rethink-attempts`,
 `--rethink-timeout`, `--profile-out` and `--logs-dir`; `desktop` takes `--app`, `--app-path`, `--window-title`,
 `--goal`, `--expect`, `--execute`, `--plan`, `--clear`, `--text`, `--text-target`, `--text-mode`, `--verify-file`,
-`--pixel-target` and the same three `--rethink` flags; `ticket_router` takes `--dataset` and `--batch-size`;
+`--pixel-target`, `--box-target` and the same three `--rethink` flags; `ticket_router` takes `--dataset` and `--batch-size`;
 `injection_guard` takes `--labelled-set`. The four games take no flag of their own.
 
 ### Desktop text input
@@ -71,6 +71,14 @@ among them when controls have no accessibility elements. Each click carries its
 Install the `cua-four-b` extra and set `CUA_S1_VARIANT=4b`, `CUA_S1_MODALITY=multimodal`.
 See [the macOS visual task](../evals/desktop/visual.md) for setup and verification commands.
 
+`--box-target KEY=description` (**experimental**, needs #97 / `S1A_JEV_IMAGES=1`, `TYPESAFE_MODEL=jev-1.14.0` or
+newer) locates each key with Jev's Box question instead of hand-written coordinates. Every step one request asks,
+per target, a presence `noul` and a `box` over the current screenshot. Targets below
+`S1A_BOX_PRESENCE_THRESHOLD` or `S1A_BOX_CONFIDENCE_THRESHOLD` (both default `0.5`, **uncalibrated**) are not
+offered; the click is the box centre, bound to the current capture like `--pixel-target`. Do not treat this as a
+general accuracy claim; measure on your own captures with `scripts/measure_box_targets.py`. A live run on the #30
+visual fixture is still required before calling the feature measured. Browser Box is out of scope.
+
 ## Allrecipes
 
 `allrecipes` is the first Allrecipes task of the [WebVoyager](https://github.com/MinorJerry/WebVoyager) task set

docs/capabilities.md

@@ -10,7 +10,7 @@ For issue trackers and serving support, see [System1-Omni model tracker #83](htt
 
 | Model / checkpoint | Input modality | Question types implemented | Scenario / front | Serving path | Hardware validation in this repository |
 |---|---|---|---|---|---|
-| TypeSafe Jev (`--model jev`; actual served model comes from the configured endpoint) | Text state and enumerated options; with `S1A_JEV_IMAGES=1` on direct TypeSafe, also one current screenshot (**experimental, TypeSafe danger zone**: offline fake-transport tests only, no recorded live run or quality result, #97) | `choice`, `noul`; service protocol also defines `score`, but Agents has no corresponding question type | Tool decisions, browser decisions, rails, `decide`/`probe` and MCP | HTTP through TypeSafe or configured OpenRouter-compatible endpoint; configured compatible endpoint can use System1-Omni `/v1/systemone` | The adapter can address configured Jev-compatible services; exact model identity is endpoint-reported. Historical artifacts include Open-Jev-9B and an Open-Jev-27B-v1.1 load attempt; those are endpoint configurations, not model names guaranteed by the `jev` adapter. See below. Other combinations **unverified here.** |
+| TypeSafe Jev (`--model jev`; actual served model comes from the configured endpoint) | Text state and enumerated options; with `S1A_JEV_IMAGES=1` on direct TypeSafe, also one current screenshot (**experimental, TypeSafe danger zone**: offline fake-transport tests only, no recorded live run or quality result, #97) | `choice`, `noul`; with `S1A_JEV_IMAGES` also experimental `box` (#98, Jev 1.14+); service protocol also defines `score`, but Agents has no corresponding question type | Tool decisions, browser decisions, rails, `decide`/`probe` and MCP | HTTP through TypeSafe or configured OpenRouter-compatible endpoint; configured compatible endpoint can use System1-Omni `/v1/systemone` | The adapter can address configured Jev-compatible services; exact model identity is endpoint-reported. Historical artifacts include Open-Jev-9B and an Open-Jev-27B-v1.1 load attempt; those are endpoint configurations, not model names guaranteed by the `jev` adapter. See below. Other combinations **unverified here.** |
 | JEV-27B-VL (`--model jev-vl-served`; `autotrust/JEV-27B-VL`) | Exactly one current PNG/JPEG plus text state and question context | `choice`, `noul` | Browser front and direct `build_model()` factory path; other fronts are not registered | HTTP native flat `/v1/systemone` through the System1-Omni frontend at `JEV_VL_URL` | Local adapter/registration tests use mocks. Single-H800 GPU checks and bounded R3 browser task completion with same-trial video are validated; other tasks and hardware remain unverified. See [validation scope and limits](decision-models.md#served-jev-vl). |
 | CLM-v0.1-8B (`--model clm`; default serving label `clm-latest`) | Text | `choice`, `noul` | Tool decisions, browser, rails, `decide`/`probe`, MCP | HTTP `/v1/systemone` to a compatible CLM service; real encoder is external | Adapter fixtures include stub responses. A historical ticket-router report records real Qwen3-8B through upstream `clm-serve` on an RTX 4090, but its Agents source SHA is not pinned in the report and the serving engine is not Omni native CLM. It reports 18/90 correct (20%) for the normal tool-loop framing and a 4/12 fit-probe verdict of “not a decision-model task”; the 90 decisions repeat the same 30-ticket set and are not 90 independent tickets. Treat this as poor, provenance-incomplete historical evidence, not current or broad validation. See [CLM report](../evals/ticket_router/CLM.md). |
 | LAYA (`--model laya`; default `convaiinnovations/laya`, English; `LAYA_SUBFOLDER=multilingual` selects multilingual) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`; browser state/question shaping is implemented | In-process Python `laya` | Historical English and multilingual CPU/MPS runs are documented below. Other/typed/custom checkpoints and CUDA are unverified here. |

docs/configuration.md

@@ -11,6 +11,8 @@ Variables can be exported in your shell or placed in a `.env` file at the root o
 | `TYPESAFE_API_KEY` | `jev` model | *(unset)* | API key for direct TypeSafe decisions endpoint (`https://api.typesafe.ai/v1/systemone`). |
 | `TYPESAFE_API_URL` | `jev` model | `https://openrouter.ai/api/alpha/decisions` | Endpoint URL for decisions; defaults to OpenRouter proxy, or can be overridden to a custom proxy URL. |
 | `TYPESAFE_MODEL` | `jev` model | `jev-1.13.0` direct, `typesafe/jev-1.13` through OpenRouter | Jev model id on either backend. Both defaults are pinned versions: no path sends the moving `jev-latest` or `jev-preview` aliases unless you set them here. On the direct path (`TYPESAFE_API_KEY` without `TYPESAFE_API_URL`) it takes a TypeSafe name such as `jev-1.14.0`; an OpenRouter id like `typesafe/jev-1.13` there is a config error. TypeSafe keeps `jev-1.13.0` available and promises 60 days' notice before deprecating it. Step records keep `model` (the id the response reports, else the one sent) and `requested_model` (the one sent). |
+| `S1A_BOX_PRESENCE_THRESHOLD` | desktop `--box-target` | `0.5` (uncalibrated) | Minimum Jev `noul` presence probability for a `--box-target` to be offered that step. Experimental; tune with `scripts/measure_box_targets.py`. |
+| `S1A_BOX_CONFIDENCE_THRESHOLD` | desktop `--box-target` | `0.5` (uncalibrated) | Minimum Jev Box confidence for a `--box-target` to be offered that step. Experimental; no recommended TypeSafe threshold yet. |
 | `S1A_JEV_IMAGES` | `jev` model (direct TypeSafe only) | *(unset: off)* | **Experimental (TypeSafe danger zone).** `1`/`true` lets hosted Jev read the observation's screenshot: the browser front then captures one per step, and the desktop `--pixel-target` path accepts `--model jev`. Each request carries one PNG/JPEG in TypeSafe's `assets` map, referenced once from `state.screenshot` (a fresh UUID reference per request), plus the `TypeSafe-Danger-Zone-Images: true` header. Screenshots whose longest side is over 1,568 px are scaled down (aspect kept, sent as PNG); a side under 280 px after that, a file over 16 MiB or a body over 32 MiB is a config error before anything is sent. Step records add `sent_image` (sent width, height, bytes, format, whether it was resized); image tokens are billed as `input_tokens`. Set with the OpenRouter proxy or `TYPESAFE_API_URL` it is a config error. An image request can take longer than the 5 s default deadline; raise `S1A_DECISION_TIMEOUT_S` if it times out. |
 | `S1A_DECISION_TIMEOUT_S` | `jev`, `jev-vl-served` models | `5` | Total deadline in seconds for one decision. Jev includes retries; JEV-VL shares it across sequential questions and does not retry HTTP requests. Must be finite and above zero. Raise it for a slower System One server behind `TYPESAFE_API_URL` (e.g. a local OneJev-27B or CLM-8B: `30`). |
 | `JEV_VL_URL` | `jev-vl-served` model | *(unset, required)* | Frontend base URL serving `autotrust/JEV-27B-VL` through System1-Omni's native flat `/v1/systemone` protocol, e.g. `http://127.0.0.1:8080`. Browser and direct factory paths only; requires exactly one current PNG/JPEG image. |

docs/decision-models.md

@@ -21,9 +21,11 @@ The registered paths, input modalities and question types are summarized in the
 
 Input: an `Observation(state, images=())` (a JSON object or plain text) and typed questions by name:
 `ChoiceQuestion(options, goal=, rules=, operation=)` picks one key among `options` (a string per key, or a dict for
-the browser's element rows); `NoulQuestion(question, criteria=)` asks whether a statement holds. Output: a
-`Decision(answers, latency_ms, usage, model, raw)` with one `Choice(key, probabilities, confidence)` or
-`Noul(p, confidence)` per question name. **`confidence` always means the same thing on every backend:** for a
+the browser's element rows); `NoulQuestion(question, criteria=)` asks whether a statement holds;
+`BoxQuestion(target)` (experimental, hosted Jev 1.14+ with images) asks where something is and returns a
+`Box(top, bottom, left, right, confidence)` as image fractions. Output: a
+`Decision(answers, latency_ms, usage, model, raw)` with one `Choice(key, probabilities, confidence)`,
+`Noul(p, confidence)` or `Box(...)` per question name. **`confidence` always means the same thing on every backend:** for a
 choice it is `max_k p_k` (the peak probability); for a noul it is `max(p, 1 - p)`. Validation computes both from
 the validated probabilities in `s1a/decision_models/validation.py`, so Laya's normalised entropy, Jev's normalised
 margin, and OmniJev's recompute never become the reported `confidence`. Each backend's own number is kept as
@@ -58,7 +60,7 @@ shorthands; `warm()` and `close()` open and release the backend.
 
 | `--model` | class | `name` | notes |
 |---|---|---|---|
-| `jev` | `JevModel(transport)` | `jev` | the request body every front sent before the layer existed, byte for byte; `from_env` picks TypeSafe or the OpenRouter proxy (see [configuration.md](configuration.md)). Text only unless `S1A_JEV_IMAGES=1` on the direct TypeSafe backend (**experimental, TypeSafe danger zone**, #97): then one current screenshot rides in `assets`, referenced from `state.screenshot`, with the danger-zone header; no quality result is claimed for it yet |
+| `jev` | `JevModel(transport)` | `jev` | the request body every front sent before the layer existed, byte for byte; `from_env` picks TypeSafe or the OpenRouter proxy (see [configuration.md](configuration.md)). Text only unless `S1A_JEV_IMAGES=1` on the direct TypeSafe backend (**experimental, TypeSafe danger zone**, #97): then one current screenshot rides in `assets`, referenced from `state.screenshot`, with the danger-zone header; with images on, `box` questions are accepted too (#98, Jev 1.14+); no quality result is claimed for screenshots or Box yet |
 | `jev-vl-served` | `ServedJevVLModel(url=, timeout_s=)` | `jev-vl-served` | `autotrust/JEV-27B-VL` through native flat `POST /v1/systemone`; exactly one current PNG/JPEG, choice and noul only; sequential questions share one deadline, with no HTTP retry or answer re-ask. Browser and direct factory paths; local tests use mocks, with separate [bounded R3 GPU/browser validation](#served-jev-vl). |
 | `clm` | `ClmModel(client, model=)` | `clm` | CLM behind `/v1/systemone`: a frozen Qwen3-8B encoder and two projection heads, scored by an engine that owns everything after the encoder. One POST per decision; `MODEL_SERVICE_CONFIG_ERROR` without `CLM_URL`, `MODEL_CALL_FAILED` when nothing is listening, on an error status, or on a body with no `answers`. Records `url`, the `X-Request-Id` it sent and `served_by` (the served names, read once — by `warm()` on the agent fronts, or on the first decision for `decide`/`probe`); choice and noul ([docs/clm.md](clm.md)) |
 | `laya` | `LayaModel(agent, model=)` | `laya` | one forward pass per call on a thread; `MODEL_SERVICE_CONFIG_ERROR` when `input_tokens` fills the window (Laya cuts the state silently; see `LAYA_MAX_LEN` and `LAYA_HEAD_MAX_LEN` in [configuration.md](configuration.md)); `ValueError` and `RuntimeError` from the library become `MODEL_CALL_FAILED` |

evals/desktop/visual.md

@@ -43,3 +43,11 @@ of the captured window. Every click carries its capture ID so the driver can rej
 
 The temporary selection file and ignored run outputs are generated locally. Screenshot bytes are passed
 to the model separately from the serialized observation state.
+
+## Experimental: `--box-target` instead of `--pixel-target`
+
+Requires hosted Jev 1.14+ with `S1A_JEV_IMAGES=1` (see #97 / PR #99) and a TypeSafe key. Replace the two
+`--pixel-target` flags with e.g. `--box-target left=the tile labelled Save` / `--box-target right=the tile labelled Cancel`
+only after measuring on this fixture; defaults for `S1A_BOX_*_THRESHOLD` are uncalibrated. This checkout has no
+recorded live Box fixture run (no TypeSafe image key / GPU on the agent box) — keep any such PR draft until one lands.
+