[Birfy]

Required human contributor answers

Per the repo owner's (Danyang, @Birfy) standing instruction, these are filled from the issue (#97) instead of waiting for separate human answers.

1. What outcome are you trying to achieve?

   Answer: --model jev (hosted TypeSafe Jev, the backend the README benchmarks most) can read the page or window screenshot, so it can be compared with and without one, and users without a GPU get a screenshot-reading option ([Feat] Let hosted Jev read screenshots (TypeSafe image input, opt-in) on the browser and desktop pixel-target paths #97).

2. What is the highest-leverage variable for that outcome?

   Answer: Whether the direct TypeSafe request carries the current screenshot in the documented assets format with the danger-zone header, within TypeSafe's image limits.

3. How will this work move that variable rather than create busywork?

   Answer: It flips JevModel.supports_images behind one opt-in setting, so the existing browser screenshot capture and desktop --pixel-target path work unchanged; nothing changes when the setting is unset.

4. Who is the customer or user, and what do they actually need?

   Answer: People running the browser or desktop agents on hosted Jev who need visual grounding (canvases, controls without accessibility entries) without running an in-process vision model.

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision and rationale (or explicit source reference): Yes; the request body is built in s1a/decision_models/jev.py, scoped by #97 items 1 to 4.

For a bug fix: link actual observed failure evidence.

Evidence: Not a bug fix: a new opt-in input path (#97).

Why

TypeSafe turned on image input for hosted Jev with Jev 1.14 (Images docs); jev-1.13.0, our pinned default, accepts it too. Here the jev path is text-only: JevModel never sets supports_images, so the browser front never captures a screenshot for it and the desktop --pixel-target path refuses it. Closes #97 (the live comparison, item 5, is still outstanding; see Demo / evidence).

How

Start with s1a/decision_models/jev.py, then s1a/decision_models/wire.py and tests/test_decision_models_jev_images.py.

• S1A_JEV_IMAGES=1 (read by wire.jev_images_from_env) is accepted only on the direct TypeSafe backend; with the OpenRouter proxy or a custom TYPESAFE_API_URL it is a config error, since the proxy is not known to forward assets. An unreadable value is a config error too. The TypeSafe client is then built with images=True, and JevModel.supports_images follows it.
• With images on and one screenshot in the observation, the body gets assets: {<fresh "screenshot-<uuid4>">: {type: image, format, data_base64}}, the reference goes once into state.screenshot (a text state becomes {text, screenshot}), and that request alone carries TypeSafe-Danger-Zone-Images: true (per-request headers= on JevDecisionsClient.decide). No screenshot means the plain text body; more than one is MODEL_CALL_FAILED.
• Image preparation follows the documented limits: PNG/JPEG still only (bytes must match the media type), longest side over 1,568 px scaled down with aspect kept and re-encoded as PNG, a side under 280 px after that rejected, 16 MiB per file and 32 MiB per body checked before sending.
• Step records add sent_image (width, height, bytes, format, resized) through Decision.provenance; input_tokens already comes from usage.
• Desktop: --pixel-target accepts --model jev when S1A_JEV_IMAGES is on (the env already sends exactly one current capture per step).

What

• New env var S1A_JEV_IMAGES (off by default), documented as experimental (TypeSafe danger zone) in docs/configuration.md, docs/decision-models.md and docs/capabilities.md, with no quality claim.
• JevDecisionsClient(images=...) and decide(body, *, headers=None); ScriptedTransport gains images and records headers.
• PROVENANCE_KEYS gains sent_image (present only when an image was sent).
• Nothing changes for callers with the setting unset: same bodies (the three oracle tests still pass), no header, no new record fields.

Verification

• uv run ruff format --check . && uv run ruff check . && uv run ty check: all clean
• uv run pytest -q and scripts/smoke.sh: 1273 passed, 130 skipped; smoke OK
• CHANGELOG.md and the docs say what the code does now

New offline tests (fake transport and httpx.MockTransport): assets and header only when on; every asset referenced once, fresh per request; text state wrapped; oversize 3136x1960 resized to 1568x980; 279 px and too-thin images rejected; GIF, mismatched bytes and garbage rejected; proxy and custom-URL backends refuse the setting; desktop --pixel-target accepted for jev only with images on.

Demo / evidence

• Video link, or exemption/blocker with follow-up: none yet. No TypeSafe key is available where this was built, so the issue's item 5 (text-only vs screenshot --model jev on the WebVoyager Allrecipes task, committed under docs/results/) is not done. Keeping this draft until that run lands or a maintainer accepts merging the opt-in path first.
• Application/task → agent actions → independently checked result: not recorded.
• System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: base main 06db629; hosted TypeSafe, no System1-Omni involved.

[Birfy]

Self-review (Birfy; cannot approve own PR).

Ready-to-merge-from-my-side for the opt-in wire path of #97: S1A_JEV_IMAGES defaults off; only the direct TypeSafe backend is accepted (proxy / custom URL are config errors, not silent text-only); assets + TypeSafe-Danger-Zone-Images are per-request; resize/reject follows the documented side and byte limits; provenance sent_image is present only when an image was sent; desktop --pixel-target accepts jev only with images on. Offline tests with fake / MockTransport cover the important reject paths. Main was merged in (8ec00f9); CI green.

What's left before fully closing #97 / leaving draft:

1. Issue item 5 — live text-only vs screenshot --model jev on WebVoyager Allrecipes task 0 under docs/results/ — still needs a TypeSafe key. Same blocker as noted in the PR body.
2. #113 (--box-target) is stacked on this branch; merging this first unblocks that stack.

From my side the experimental opt-in path is fine to merge if a maintainer accepts shipping without the live comparison row; otherwise stay draft until that run lands. No quality claim in the docs — keep it that way.