Repository navigation
feat: experiment progress visualization and watching - #1031
Draft
small-angel wants to merge 96 commits into
Draft
small-angel wants to merge 96 commits into
small-angel wants to merge 96 commits into
Conversation
Contributor
Author
|
(claude is still fixing something before it can push my bad) |
Draft: the run-monitoring stack as one commit for trying out. Agents instrument an experiment's stages and measures and write its view; runs show progress, time left and their charts above the chat and on the page, several runs share one summary, and an uninstrumented run says it is setting up and reads progress from its log. Assisted-by: claude-fable-5.2 Assisted-by: claude-opus-5.5 Assisted-by: claude-sonnet-5.5 Assisted-by: gpt-6.1-sol
small-angel
force-pushed
the
clo/run-monitoring
branch
from
November 14, 2026 02:34
664ecf3 to
b743eb0
Compare
A reusable test bed for the rules, tripwires and watcher that are about to land: an invented training run (payload.mjs) in twelve variants, each a child of a healthy parent, and a runner that launches one against the real orx up in the e2e sandbox and prints the line, strip, page, status text, monitor flag/sentence/watch fields and the wake message the agent gets at fixed seconds after the child starts. Scenarios live beside the payload so a dev project can use the same run command. The runner counts points from the run's startedAt, since a busy machine can take most of a minute to submit one. Assisted-by: claude-sonnet-5.5
An agent knows a run's whole lifetime before it starts, so the whole-run
bar no longer counts only the stages seen so far ('50%' during download,
'100% · 0s left' at the end of train with eval and SFT to come).
- Stored: the experiment's monitor settings gain plan: [{phase, of?,
unit?}], set with 'orx exp view set EXP --config PATH' ({"plan": …};
any other key is refused, naming what this orx reads), shown by 'view
get', copied into children with the rest of the settings. An empty plan
clears it. A stage that runs again is listed again where it recurs,
never twice in a row, since a log cannot tell two such occurrences
apart.
- Wire: '::orx:: {"plan": […]}' is the run's whole plan and replaces the
experiment's for that run; keys a stage does not know are ignored. The
checkpoint keeps it as 'plan'. orx_run.plan([...]) writes it.
- Stages: occurrences are laid over the plan in order. A stage the plan
does not name is shown as unplanned and the plan grows by it; planned
stages passed over, or never reached by a run that finished, are
skipped. Each stage's weight is how long the parent's run took over the
stage of the same name and count. When any stage has no such time, the
whole-run fraction is unknown (the bar is drawn as steps), never an
equal split; time left adds the parent's later stages and is a floor
('at least', naming the untimed stages) when one is unknown.
- Surfaces: the monitor lists every stage with planned/skipped,
segments span all of them, the summary carries segments for the line,
and 'orx exp status' prints a stages line. A run with no plan is
projected as before.
Assisted-by: claude-opus-5.5
The strip and the page list a plan's stages as the prototype does: done
stages with their time, the current one, the ones to come with 'about
Xm' from the parent's run, a skipped stage marked skipped (never done)
and an unplanned one marked unplanned. The strip lists them under its
chart now, not only when the view has no block for the stage.
The line's underline is segmented by stage from the summary's new
segments, so it needs no full monitor. A time left that leaves out
untimed stages names them ('No earlier run timed eval'); the summary
carries etaUntimed so the line and the strip say the same, and
'orx exp status' uses the same words. A stage that has begun with no
step yet reads as its name in the caption.
Assisted-by: claude-opus-5.5
'orx exp view get' printed a declared total of 3 shards as 3.0; the stored plan keeps the number the agent wrote. Assisted-by: claude-opus-5.5
A finished run whose last stage reported no step (chat) read no share at all, and with priors could read under 100%. Every stage of a plan that finished is done; the stages it skipped are not segments. Assisted-by: claude-opus-5.5
An agent picks, per view, what a run is compared with and what of the parent is drawn, since that depends on the experiment at hand: - vs="step|final" on <stats> and on a <signal> as its default, and vs="parent|step|final" on <table>: at the same stage and step, or with where the parent ended. Without one, a run compares at the same step while it goes and with the parent's final value once it ended. A stage that has ended compares at its last step, so a run in eval no longer measures its train loss against the parent's final one. - ghost="true|false" on <series>: a reference the run measures itself is not drawn twice. Every line of a chart with ghost="parent" has the parent's under it by default, over steps too. - x-range="run|parent" on a chart over steps: by default the axis spans the run's own total and the parent's line is cut at the edge, so a short child is no longer squashed under its parent's 5,000 steps, and the line no longer runs past a strip's edge. The monitor gains versusBy (both ways, for a block that names its own) and a step on each comparison made at one; the status line says which. Assisted-by: claude-opus-5.5
An agent asked mid-run to show something else runs `orx exp view set` with a new view and new patterns; the open strip and page redraw with no reload, and the patterns read what the log already said before they existed. The test records when each surface first shows the change: the page within the command's own return, the strip's new headline with its value at the next collection, about 0.8 s later. Assisted-by: claude-opus-5.5
The view spec read the base run's line after the agent had said what came of the run, when the line may already have left; it failed about half the full-suite runs. The result turn now holds until the line is read. Assisted-by: claude-opus-5.5
`versus` held each signal compared its own way and `versusBy` held it
both ways, so every own-way entry went out twice. Nothing has shipped,
so `versus` now carries the two ways, `{"step": …, "final": …}`, and
`versusBy` is gone without a compatibility path. The page picks the way
a block names, else the signal's own `vs`, else the same step while
the run goes and where the parent ended once it ended (`versusOf`); the
server keeps the same rule for the summary, the bindings and the status
text.
Assisted-by: claude-opus-5.5
A time left that is only a floor ("at least", from a counter in the log
or a plan with stages no earlier run timed) counted down to "At least 0s
left", which promises nothing. Under a second left it now reads "No
estimate", with the stages no earlier run timed as the detail ("No
earlier run timed eval, sft") or, for a counter read from the log, that
more work may follow it. `orx exp status` says the same. A plan whose
known stages are done and whose rest is untimed keeps a zero floor
instead of no estimate, so it names those stages too.
Assisted-by: claude-opus-5.5
How long a stage should take now comes from the experiment's own last run that finished, then from the parent's, stage by stage (PLAN §4 order). Before, only a run with a declared plan had stage times at all, and only from its parent. A run without a plan now gets them for every stage it reaches, so a run that goes silent at its first step stalls against the earlier pace (the `hang` fault: "no progress" after 90 s where it said "Estimating…" forever), and its first slices blend the earlier time as the estimator says. Assisted-by: claude-opus-5.5
A run that declared no plan and had reached a second stage read a share of the whole run from an equal split across the stages seen so far: 50% the moment train began, 98% just before an eval no one had seen yet. The stages still to come are unknown, so the share is too: the bar keeps its stages and the caption drops the percentage, as a plan with an untimed stage already does. Assisted-by: claude-opus-5.5
The line led with the first of the view's headline signals that held a number, so a finished run hid a NaN loss behind its eval score (`nan-400`) and an all-zero eval behind a level val loss (`eval-zeros`). The rule now: among the view's headline and then its other signals, the first non-finite value leads; else the first that is exactly zero where the parent's is not; otherwise the first headline with a number, as before. The server decides and sends the name as `headline` on the monitor and in its events, and the page draws it instead of choosing again. Assisted-by: claude-opus-5.5
A Python script with no shell around it could not name the stage it starts in: `orx_run.progress()` wrote its first report before any phase, so the stage list began with an unnamed stage the plan never listed. `progress(phase="train", …)` puts the name in that first report. Assisted-by: claude-opus-5.5
`orx exp status` and the line's summary said "eval · step 10" when the run printed steps without a total, while the stage list beside it had the plan's total. Both now take the plan's total when the run printed none: "eval · step 10 of 50". Assisted-by: claude-opus-5.5
Past its estimate with no progress since, a run's time left sat at "0s left" for as long as it stayed silent, beside a bar that had stopped (`hang-mid`, `step-stuck`: 40 to 80 s before the stall note, and then only in the strip). Once the countdown runs out with no progress since, the time slot reads how long it has been silent, "No progress for 40s", with "was due about 7:41 PM" as the detail, on the line, the strip and in `orx exp status`. A stage that had nothing left when estimated keeps "0s left"; a floor keeps its own words. Assisted-by: claude-opus-5.5
A run that failed without a message read "no message recorded" in `orx exp status`, "exit 1" on its page and nothing in its strip, though its log ended in a traceback that only the agent's wake carried. The log's last error now says it: a Python traceback's exception line, else the last line that names an error. `orx exp status` reads "reason: last error in the log: `torch.OutOfMemoryError: …`", the wake's Reason says the same, and the strip and page quote it under the Failed lead. A line the credential scan flags is never shown; a log that cannot be read leaves the reason pointing at `orx logs`. Assisted-by: claude-opus-5.5
The log tail that wakes an agent when its run ends was mostly `::orx::` progress records: an instrumented run left 7 human lines of its last 46. The tail now leaves the records out, reading up to 64 KiB back to find the last 40 lines a person would read (still at most 4 KiB), and says how many records it left out and that `orx logs RUN` has the whole log. Assisted-by: claude-opus-5.5
Nothing named a run whose every step took three times as long as its
parent's (`slow-3x`): its numbers were level, and "2m left" sat beside a
parent that ran 1m10s in all. Once four slices of a stage are measured,
a step that takes at least twice as long as one of the run that timed
the stage (the experiment's own last run, else the parent) is said in
the strip, on the page and in `orx exp status`: "3× slower than the
parent at this stage". It compares time per step, so a stage that only
has more steps is not slower; it is derived on every evaluation from
the stage times the estimator already has, never stored. The monitor
carries it as `pace: {times, than}`, rounded to a half.
Assisted-by: claude-opus-5.5
Each stage counts its steps from zero, but a series held every stage's points on one axis, and a value reported in a stage without steps sat at its time in seconds: a parent that reported its final loss in eval drew a ghost that jumped back to x≈0, and while a child downloaded its three shards the page drew the parent's train loss across that 0–3 axis. The fold now keeps one stage's points per series on the step axis (a series starts over when a later stage reports it, recorded as `seriesStages` in the checkpoint), and once a run counts steps a value from a stage without them is only the latest. The parent's line goes under the child's only when both are the same stage, or before the child has a point, the stage it counts steps in now. A checkpoint from before reads with its stages unknown and draws as it did. Assisted-by: claude-opus-5.5
A run that hung right after its stage's first report (`hang`: train step 0, then "compiling model" and silence) was never placed in time: the fold anchors `t` to the clock only at a report that advanced, and a stage's first report advances nothing, so the estimate counted from "now" forever and no stall could be measured, even with an earlier run's pace to go by. The first report of a stage now anchors too. Its cadence is unchanged: the gap after it still never counts as one. Assisted-by: claude-opus-5.5
An open strip keeps its monitor current through events, and their patch left out `logError`, so a strip that watched a run fail never showed why: the summary had it at once, the strip's monitor never did (the `crash` fault). Squash into "fix: say why a failed run failed". Assisted-by: claude-opus-5.5
An agent reads orx-experiment-runs before a baseline's first launch and whenever it changes how a run is watched. It teaches a run's whole lifetime in one home: read the entry point, declare every stage in a plan (--config for totals known now, orx_run.plan for totals the run learns), report progress and the deciding measures, write the view and choose per experiment how runs compare (vs), what the parent shadows (ghost) and the axis (x-range), patterns for logs the agent cannot edit, the first-minutes check, and what a mid-run view, pattern or plan change reaches. The name pairs with orx-experiment-tree (the tree; its runs) and avoids `orx runs`, the command that lists them. It replaces orx-experiment-tree/references/instrument.md and the reporting section of orx-compute, which now point at it; the helper stays at compute/assets/orx_run.py, where runs already vendor it from. The text test now also compiles every documented plan through the --config path and folds every documented ::orx:: line. Assisted-by: claude-opus-5.5
A cheap agent (claude-haiku-5-5), asked only to run a baseline and two learning-rate children of a small numpy project, did each thing the skill teaches, but left what it was unsure of to the user's eye. Five attempts moved the text where it went wrong: it blocked its turn on `orx exp wait` so a mid-run request waited for the run to end; it read its own script's log with patterns rather than reporting from it, which left the eval stage without progress; it drew a row per class but never stepped the stage; it left table columns unlabelled and put two scales on one chart. Assisted-by: claude-opus-5.5
Assisted-by: claude-opus-5.5
An agent declares the rules a run is stopped or asked by beside its stage
plan, with `orx exp view set EXP --config PATH` and `{"rules": [...]}`.
Each rule is {name, text, does: stop|ask, when}, and `when` is one bounded
condition, never an expression: nonFinite, compare, noImprovement,
worseThanParent, noProgress, costAbove and projectedFinishPastLimit, with
reports where a run of reports counts. Every mistake is refused at save in
its own words, and a condition a newer orx wrote stops a launch with what to
do.
Children start with a copy, as with the view. A run freezes the rules it
was launched with in `runs.monitor_config_json`, so editing the experiment
never changes what a running run checks. `orx exp run --budget USD` adds a
rule named budget, which the page words itself since orx wrote it.
The monitor lists a run's rules and its budget, and `orx exp status` gains a
`rules` line. Nothing checks them yet.
Assisted-by: claude-opus-5.5
The first cancel accepted for a run now keeps who asked for it, in
`runs.stop_context_json`, written in the same statement that accepts the
cancel under the run's cancel lock: a user, an agent, or (next) a rule.
A later stop of any kind, and a legacy cancel that came first, never
replace it, so a racing user and rule cannot rewrite the first actor. An
old cancellation has no one recorded and reads as unknown, never guessed
from its text.
`orx exp cancel --reason TEXT` keeps the agent's words (1 to 2,048
bytes). A cancel from an agent's chat is the agent's; one with no chat,
from the dashboard or a terminal, is the user's. A cancelled `Run` gains
`outcome: {by, at, reason?, rule?}` in both run projections, and
`orx exp status` a `stop` line in the page's leads: "Stopped by the
user.", "Auto-stopped by the agent: …".
Assisted-by: claude-opus-5.5
Words the walkthrough read badly: - "1 done · first done in at least 19s": once a run of a summary has ended, the soonest of the rest is the next done. - The strip named a run by its slug where every other place used its title; a run is named by its title everywhere. - "since epoch / 2 1s ago": the sentence's age runs on after a dot. - After Keep going the alarm sentence stayed in the all-clear colour with nothing beside it; until the next look rewrites it, the strip says who kept going after it. - A strip's chart beside a headline it doesn't draw was unlabeled; a chart now names what it draws, by caption where its legend has no room. Assisted-by: claude-opus-5.5
A pass read every live run's log while it held the watcher's state lock, which a finished chat's looks took too, so one slow host kept the watcher's bookkeeping waiting. The state is now the pass's own, and the set of chats being looked at, the one thing the look tasks share, has a lock of its own held for no read. From the review's smaller findings in these files: what a pass keeps of a run goes once the run is no longer read (F18), a tripwire's line and a flag's evidence are clipped as the digest's are (F13), and a look whose runs can't be republished says so in the log (F19). Assisted-by: claude-opus-5.5
The line and strip name a run by its experiment's title now, as the summary rows and the page already did; the specs looked them up by slug. Assisted-by: claude-opus-5.5
The walkthrough's looks cost 7k to 22k input tokens. Claude Code's one-shot ran in orx's Claude home, which links the user's CLAUDE.md, skills and plugins in, and Claude Code adds them to every request even under --system-prompt: on this machine 4,068 tokens before a look said anything, against 365 without them. A one-shot (a look, a title) now runs in a home of its own under agents/one-shot that links only the sign-in and settings, so it still signs in and honours the user's settings. --setting-sources "" would drop the memory too, but also a gateway or apiKeyHelper set in settings.json. Also here, belonging with the e2e change before it: the fault bed finds a run's line by its title. Assisted-by: claude-opus-5.5
A real look still sent about 4,400 tokens before its own prompt: the one-shot runs in a temporary folder, and where that folder sits under the user's home, Claude Code finds the home's .claude as a project and loads its memory and skills. One-shots now read only their home's own settings (--setting-sources user); measured, 4,379 extra tokens fell to none. Real looks also measured about two bytes of digest a token, not three: 4,327 bytes cost 1,760 tokens and 8,961 bytes 4,500. The budget counts two, so a look's request is held to 4,096 tokens as planned. Assisted-by: claude-opus-5.5
A Claude one-shot linked its account's .claude.json and
.credentials.json into its own home. On Windows without Developer
Mode the link is a hard link, so the next chat on that account
refused the account's files as linked: one look broke the account.
The sign-in is never linked now. Claude Code reads it where
CLAUDE_SECURESTORAGE_CONFIG_DIR points, which is the account's own
home on every OS, so a refreshed token lands there. The account file
is copied, private, on each one-shot: it holds a /login key on Linux
and Windows but nothing that rotates. Removing an account removes its
one-shot home with that copy.
A Codex one-shot ran in the account's own home, so it read the
user's AGENTS.md and skills. It now runs in a home that links only
config.toml and auth.json. Windows keeps the account's home: its
links fall back to hard links or copies, and a copied auth.json would
miss a refreshed token.
Neither runs the user's hooks: Claude takes --settings
'{"disableAllHooks":true}', which leaves their settings untouched,
and Codex features.hooks=false.
Not run on Windows; the new test runs there in CI.
Assisted-by: claude-opus-5.5
After the agent's own cancel or a rule's stop, a Copilot flag still read as waiting: the card offered Stop and Keep, both answered 409 "already settled", and the chat's sidebar kept its warning for as long as the engine took to deliver the cancel. A flag no one settled now reads stopped once the run's stop is accepted, with that stop's who, when, why and rule, marked moot. The journal says who stopped it (you, the agent, a rule by name, or someone, for a stop an older orx kept no record of), and a choice on it answers that the run is already stopping. Assisted-by: claude-opus-5.5
A run a chat only follows pulled the chat's own answered runs back into its banner: shown runs kept beside a run every run the chat launched in the turn that run started in, and for a followed run that turn is the follower's, not its launcher's. Only a run the chat launched now keeps the others from its turn. The runs of that turn are now found by a range on the run index, from the turn's start to the next turn's, instead of reading the launching turn of every later run of the chat, so a run stuck unsettled in an old turn no longer costs a scan of all the chat's runs since. Assisted-by: claude-opus-5.5
A pass read every live run's log, and ran its tripwires, one run after another, so one host slow to answer held up the tripwires and looks of every chat. A tripwire's read of a log's last lines, and the line numbers its evidence counts, also waited as long as the host did. A pass now reads up to 16 runs at once, each with what the passes keep of it (its scan, the conditions it raised, its stage and its parent comparisons), so a slow host delays only its own runs. The tripwire's reads and its evidence's line count give up after the 10 s every other read of a pass already had; the evidence then names bytes, not lines. Assisted-by: claude-opus-5.5
The far-worse tripwire compared single points: a ratio of 1.5 on three passes in a row. It flagged a sweep child with a lower learning rate whose loss sits above its parent's for the first thousand steps, and a loss of 0.002 against 0.004 on noise alone. Its count was keyed by run and signal, so a second stage, whose steps start at 0 again, was not counted until it passed the first stage's last step, and a count left from one stage flagged the next at its first bad point. Following the orchestrator's ruling: a verdict takes the median of the last five comparisons in the stage, and needs both a ratio of 1.5 and a gap past three times the parent's own scatter at that stage, judged from the parent's values the comparisons read. A better value within that noise of zero is judged on the gap alone. Nothing is judged in a stage's first tenth, where its total is known. The comparisons start over with each stage, and whenever the steps go back, as a relaunch from a checkpoint does. Looks a parent comparison asks for are left as they were. Assisted-by: claude-opus-5.5
A run's last look that failed was planned again on the next pass, five seconds on, and a provider outage or a timed-out call usually fails twice within the same minute, so both of its tries were spent at once. It is now taken again only once the experiment's least gap between looks has passed since the failure. Assisted-by: claude-opus-5.5
A look reads a run's log from its last cursor, and the credential scan tracked a private key from its BEGIN line, so a key whose BEGIN line fell before the cursor had its body lines sent. Lines shaped as a key's body (60 or more base64 characters alone on a line) are now held when they start a read or follow a line that was held. Assisted-by: claude-opus-5.5
A look still sent more than its 4,096 tokens: trimming touched only points, earlier sentences and values no signal names, never the briefs (up to 3.2 KB each), the watcher's notes (about 3 KB in CJK) or the declared signals, so one long brief and two runs left no room for log lines and sent about 1.5 times the budget. The retry sent all of it again with the reply. The digest now cuts, in order, older points, earlier sentences, values no signal names, the notes past 512 bytes, each brief past 1 KiB (its least needed labels first), the signals no sentence reads, then notes and briefs again to 256 and 512 bytes. Each is scanned for credentials before it is cut. The journal says what each run's part left out, and how many tokens it still went over when even that did not fit. A retry is fitted to the same budget with the reply inside it. Every later look also sent each run's first lines before its new ones, out of one shared room, so in a look over three runs the first run's unchanged head could leave the third run's new error out. New lines now go first, every run's; first lines get what is left, and only until one look has shown them. Measured on the test's three-run look (two 3 KB briefs, 3 KB of notes, 21 declared signals a run): 7,898 bytes, 3,738 input tokens. Assisted-by: claude-opus-5.5
A look's flag on a run whose earlier flag was still open was noted under it and woke no one: the Agentic agent read only the open flag's frozen wake, kept going, and "checkpoint write failed, disk full" never reached anyone. In Copilot a Keep settled both, and the look's cursor had moved on, so a finding only the model makes was lost for good. A noted flag now wakes the run's Agentic chats with its sentence and evidence, naming no flag to settle, so the agent weighs it before it settles the open one; the look's runs wake together as one turn. Once someone keeps going after the open flag, the run asks for a look soon, and that look is shown what was noted with the bytes it cited, which it may cite again to raise it as a flag of its own if it still holds. A look that weighed it once is not shown it again. Assisted-by: claude-opus-5.5
Assisted-by: claude-opus-5.5
An agent that put <signal> inside <view> read "<signal> is not a block" and dropped the declaration, losing labels, units and better for the session. The error now says to move it above <view> and keep it; a stray <series> says it belongs in a <chart>. Assisted-by: claude-opus-5.5
The sorting benchmark of the second walkthrough reported one table row
a case and no step. Its broken variant wrote correct 0 from the second
row on, yet no look came until the 60 s ceiling, and the look that came
was refused: "0" matched correct, a number that "still moves".
A row whose value falls to zero or stops being a number, in a column
whose first row was above zero, is now a change trigger ("correct in
cases fell to 0 at row 2"), doubling like the others when a look
answers same, and rows count as new output. The literal rule no longer
refuses zero, a 0/1 outcome column, a signal declared with values or
labels, or a value unchanged since its first report: a sentence that
says a check is stuck may state it. Any other number that still moves
is refused as before. Both are one change because the walkthrough case
needs both to raise its flag while the run runs.
Assisted-by: claude-opus-5.5
In the second walkthrough the agent waited on every run inside its turn with loops such as `for i in $(seq 1 120); do orx runs … | grep -qE "succeeded|…" && break; sleep 5; done`. `orx runs` prints done, so the first loop spun ten minutes after a 53 s run, holding back the user and the other launches. Every launch had told it "Follow it with `orx exp wait`", and the run skill said to read `orx exp status` every half minute. Inside a chat, `orx exp run` now says the result arrives as a message, to check the first minutes once with `orx exp view check`, and to end the turn without waiting or polling; a person at a terminal keeps the old hint. The playbook says the same where it introduces launching, and the run and compute skills drop the half-minute polling and name the loops not to write. The status words `orx runs` prints are unchanged. Assisted-by: claude-opus-5.5
Asked mid-run to chart a benchmark table as time against size, the agent waited out the run, said the chart needed a code change although the run already printed both timings, and then drew linear axes over 1k to 1M with all four input kinds on one zigzagging line. The skill now says a pattern over a line the run already prints reads it into rows, kept log included, so a chart asked for mid-run is a pattern and a view change on the running experiment in the same turn; that sizes spanning decades take log axes; that mixed kinds take `by=`; and shows the benchmark as a worked view with its pattern in a table of its own. The example view and pattern were compiled and fed a sample line. Assisted-by: claude-opus-5.5
A data directory made by an earlier build keeps that build's links in agents/one-shot/<account>: .claude.json and .credentials.json pointing at the account's own files. The first look then failed refusing to write the account file's copy over the symlink, and the stale credentials link shared the sign-in the one-shot home must only read, until someone deleted the directory by hand. Preparing the home now removes either file when it is a symlink, a managed link's file or a hard link to the account's, with its marker, before writing the private copy. The account's files are untouched. Assisted-by: claude-opus-5.5
The benchmark of the second walkthrough declared a plan of bench, 16 cases, and reported 20 table rows with no phase or step. Its bar read "needs a few steps" throughout, then 100%, and `orx exp status` said "bench skipped": nothing told the agent or the user that the plan counted work the run never reported as steps. The monitor now derives `disagrees` (stage, of, unit, table, rows) when a planned run reported rows but no step in any stage and the plan counts one. `orx exp status` prints it under plan, `orx exp view check` says the plan and the run disagree with what to report, the strip and the page say it in the note, and the run skill tells the first-minutes check to look for it. Optional field; old apps ignore it. Assisted-by: claude-opus-5.5
The broken benchmark of the second walkthrough ended, the agent replied to its result, and only then did the end look flag it. The strip had already gone with the reply; the summary dropped any flag once a run ended, so there was no amber line, no warning on the sidebar row, no choices, a green Done in the experiment list, and a page leading with the flag sentence in green. A flag no one kept going or stopped on now keeps its run among the chat's shown runs (or brings it back) until someone settles it, and the summary carries the flag after the run ends, toned as a flag when the run finished. So the line, strip, sidebar row, experiment list and page all show it, and a waiting flag opens the strip by itself. An ended run has nothing to stop: the strip offers Keep the result and Ask the agent and says why, the API refuses only a stop, and an agent woken with such a flag is told how to keep it, hand it to the user, or rerun. Agentic chats are woken by it as before, whether or not the agent already replied. The shown-runs query reads flags through a partial index on flag events, so it stays off the run history. Assisted-by: claude-opus-5.5
Every finished run of the second walkthrough woke the agent for a second reply such as "This is the cutoff-64 result I already reported." Each time the agent had waited inside its turn: the run ended mid-turn, the agent read its log and reported the result, and the run's result notice, queued at the end, woke it once the turn finished. No notice was duplicated. `orx logs <runId>` run by a chat's agent on an ended run whose log is fully kept now counts the chat's waiting result notice as delivered, so no turn repeats it; the run still leaves the composer only when a reply completes. A notice a turn already claimed, another chat's, or a live run's is left alone, so a result the agent has not read still wakes it. Assisted-by: claude-opus-5.5
Rerunning the second walkthrough's broken benchmark with real Haiku, both
looks of the run were refused (a bold word with no figure, then a
literal `{signal}`), each after its retry, and the run that returned
wrong output for most cases was never flagged. The sentence rules exist
to keep numbers current; a refused flag is the false negative the
watcher must never produce.
A reply that flags a run with a sentence the rules refuse now keeps the
flag in the watcher's own words with its bold and braces dropped, while
the retry is still asked for and wins if it writes a valid sentence. A
refused sentence that does not flag is still refused. A change look's
reason also prints a fallen value as the run wrote it ("fell to 0", not
"0.0").
Assisted-by: claude-opus-5.5
The experiment table, the experiment page and the run history still read a green Done for the walkthrough's flagged benchmark after its run ended, though the strip and the sidebar row showed the flag. A done run whose summary carries an unsettled flag now displays as flagged_ended: Flagged in amber without the live pulse, the amber square on the tree, the warning icon in a chat's experiment list and the flagged mark in the lineage. A failed or stopped run keeps its own status. Assisted-by: claude-opus-5.5
Rerunning the flagged benchmark left the first run's open flag in the workspace's Active Experiments with a warning after the rerun's flag was kept, though no surface offered to settle it any more: the composer and the experiment page follow the newer run. An ended run with an open flag is listed only while it is its experiment's newest run, the rule the composer's shown runs already follow. Assisted-by: claude-opus-5.5
The rerun of the walkthrough's benchmark read "Bench started · <its newest line>" for its whole run: its pattern named the stage and read each case into a row, but no step, and a named stage without a step counted as setting up. A run that has reported rows is under way, as one with a step is. Assisted-by: claude-opus-5.5
A run's page named "The agent" mid-sentence ("Ask The agent"); it now names the experiment's chat agent, or reads "the agent" when no chat is known.
Assisted-by: claude-sonnet-5.5
The run page now names the chat's agent, so the journal says who settled the flag by name. Assisted-by: claude-opus-5.5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiments should be as easy as possible to track for both the Human and the Agent. Right now, you have to watch logs yourself, or ask the agent "status pls?" in order for anything to actually happen. Instead, we're going to just tell you what's going on.
(these images are prototypes, but the system does work right now)
You can expand the card to reveal detailed graphs that live update. The agent builds these before running the experience, which is great because it can add the proper instrumentation into the experiment run script.
The agent can take advantage of this as well. We will now run Haiku/Luna, or another configured cheap model, to monitor your runs for you, which can be used to let your agent revise runs midway through. So now you don't have to bump your agents to fix problems as much. A step closer to true automated research.
Criteria
Closes #473
Closes #500 (original issue says a lot but the key is just that goal and experiment should not break. likely already the case)
Closes ORX-671
Closes ORX-678