MemoryReviewer kills healthy runs at a stale 240s wall and never retries a timeout; CortexHealth then calls one dropped run critical (9% of runs, measured)

[xmasyx]

Summary

MemoryReviewer.ts kills healthy reviewer runs at a wall clock fitted to a stale latency measurement, never retries a timeout (though it retries a parse failure), and CortexHealth.ts then reports the single dropped run as critical. The statusline paints PROBLEM · LATEST REVIEWER RUN IS TIMED-OUT. for up to an hour, until the next scheduled review happens to succeed. Nothing is actually broken.

Measured on one install over 45 consecutive reviewer runs (LIFEOS/MEMORY/OBSERVABILITY/reviewer-runs.jsonl):

metric	value
duration of successful runs	min 15s · p50 125s · p90 200s · max 250s
runs killed at the 240s wall	4 / 45 ≈ 9%
comment justifying 240s	"successful runs measure 50–115s" (dated 2026-08-03)

Prompt size does not explain it: a 39,691-byte prompt finished in 57s while a 29,199-byte one hit the wall. This is reasoning-latency variance under --effort high, so the 240s wall sits inside the working distribution rather than beyond it.

Root cause

1. LIFEOS/TOOLS/MemoryReviewer.ts:72 — the budget is fitted to a measurement the distribution has since outgrown:

const DEFAULT_TIMEOUT_MS = 240_000; // 120s timed out repeatedly on heavy sessions (2026-08-03); successful runs measure 50–115s, so 240s bounds the tail without masking hangs

2. LIFEOS/TOOLS/MemoryReviewer.ts:681-696 — a timeout is terminal on the first attempt. The asymmetry is the tell: a parse failure a few lines below gets a deliberate corrective retry (with a comment explaining why), while a wall-clock timeout, which is pure transient latency, gets none.

    const result = await inference({ ..., timeout: opts.timeoutMs ?? DEFAULT_TIMEOUT_MS });
    inferenceDuration = Date.now() - startedAt;
    if (!result.success) {
      const failed: ReviewResult = { ok: false, ..., error: `inference failed: ${result.error}` };
      logRunSummary({ ts: new Date().toISOString(), ...failed });
      return failed;      // <- no second attempt, ever
    }

3. LIFEOS/TOOLS/CortexHealth.ts:87 — severity asks the weaker question ("did the last run fail?") instead of the real one ("is the memory loop broken?"). 44 successes around one dropped run do not move the verdict:

else if (["failed", "parse-failed", "timed-out", "invalid"].includes(input.reviewer.status))
  add(..., "critical", `Latest reviewer run is ${input.reviewer.status}.`, input.reviewer);

The reviewer re-fires on the next quiet moment and repairs itself unattended, so a lone failure is warn-shaped, not critical-shaped. The evidence object already carries priorSuccesses but severity ignores it.

Suggested fix

a. Re-fit the budget to the measured tail (MemoryReviewer.ts:72) — 300s clears the observed max with headroom, and note in the comment that the tail beyond it is handled by the retry, not by a bigger number:

const DEFAULT_TIMEOUT_MS = 300_000;

b. One retry on a wall-clock timeout, mirroring the existing parse retry. Two independent draws beyond the wall are ~0.8% where one is ~9%; anything genuinely hung still fails both attempts and surfaces, now with the attempt count logged so a real hang cannot hide behind a silent second try:

const callInference = () => inference({ systemPrompt, userPrompt, level: "medium", expectJson: false, timeout: opts.timeoutMs ?? DEFAULT_TIMEOUT_MS });
const startedAt = Date.now();
let result = await callInference();
let attempts = 1;
if (!result.success && /timeout/i.test(String(result.error ?? ""))) { result = await callInference(); attempts = 2; }

Add attempts?: number to ReviewResult and log it on every terminal path, so reviewer-runs.jsonl distinguishes "succeeded on the second try" from "hung twice".

c. Reserve critical for a streak (CortexHealth.ts). Count the trailing non-ok, non-skipped rows (a skip is a healthy no-op and should neither break nor extend the streak), carry it on the evidence as consecutiveFailures, and branch:

else if (["failed", "parse-failed", "timed-out"].includes(input.reviewer.status)) {
  const streak = input.reviewer.consecutiveFailures ?? 1;
  const id = `reviewer-latest-${input.reviewer.status}`;
  if (streak >= 2) add(id, "critical", `Reviewer has failed ${streak} runs in a row (latest is ${input.reviewer.status}).`, input.reviewer);
  else add(id, "warn", `Reviewer ${input.reviewer.status} once; the next scheduled run repairs it.`, input.reviewer);
}

"invalid" should stay unconditionally critical — it is a contract violation, not latency. This is not a mute: there is no ack and no TTL, the count and age are carried forward, and a persistent failure escalates on its own at the second run.

The orphan-dir path (CortexHealth.ts:128, a run dir newer than any terminal row past reviewerRunGraceMs) is the current failure itself, so its streak is trailingFailures(rows) + 1.

Verification

Two poles against assessCortexEvidence, both observed after applying the above:

• consecutiveFailures: 1 → overall: warn, message "Reviewer timed-out once; the next scheduled run repairs it."
• consecutiveFailures: 2 and 3 → overall: critical, message naming the streak
• status: "invalid" → still critical

Replaying the 4 historical timeouts through the new logic: each had a successful run immediately before it, so all four would have been warn — matching what actually happened, since each was repaired by the next scheduled run without intervention.

[jacobo-ortiz]

Independent confirmation from a second install, plus one driver the report does not name.

Same failure, different numbers. 25 consecutive runs of reviewer-runs.jsonl, duration computed as ts minus the timestamp encoded in runId:

metric	value
successful runs	min 39s · max 235.0s
runs killed at the 240s wall	6 / 25
of those, in the last 8 hours	5

The 235.0s success is the part worth underlining: one second under the wall. The budget is not merely mis-fitted, it is inside the working distribution, exactly as the report says.

What drives the latency here: output size, not prompt size. The prompt held steady between 25 and 40 KB across both successes and timeouts (a 30,912-byte prompt finished in 166s; a 31,111-byte one hit the wall). What grew was the response — response.raw.txt went 10,546 → 13,170 → 13,722 bytes over two days. On this install that tracks a second signal: CortexHealth reports cap-pressure at 48/48, and a reviewer at the cap must re-emit the whole consolidated set every run and reason about what it merges. So the reasoning-variance framing is right, and there is a monotonic pressure underneath it that makes a fixed wall age badly on any install whose memory fills up. Worth a sentence in the comment next to the constant, so the next person to re-fit it knows the number drifts with cap pressure rather than being a one-time correction.

One tail case the retry math should account for. A run can exceed the wall by more than one multiple today, because the parse retry a few lines below spends its own full timeout and adds to the same inferenceDuration. One row here measured 328.5s against a 240s wall for that reason. With suggestion (b) applied, a worst case becomes two timeout draws plus a parse retry, so logging attempts matters for reading those rows correctly — a 600s row will otherwise look like a hang.

On suggestion (c), applied and observed. Both poles reproduce as described (consecutiveFailures: 1 → warn, >= 2 → critical, invalid unchanged). Concretely non-muting on this install: after applying it the check stayed critical, because the real streak was 3. The escalation happened on its own, which is the property that makes the change safe to take.

One note on the streak counter: a skipped row should be stepped over rather than treated as a success that breaks the streak. Two failures separated by a healthy no-op are still two failures in a row from the memory loop's point of view, and skips are common right after a /clear.

[xmasyx]

Thanks — that second install closes the part our report could only assert. Numbers from this install after applying (a)+(b)+(c), 7 days and 42 consecutive runs:

metric	value
runs since the change	42 / 42 ok
timeouts	0
runs that spent the retry (attempts: 2)	0
successful runs	min 8.8s · p50 86.9s · p90 174.2s · max 297.1s

The 297.1s is the headline, and it argues your point rather than ours. The wall is now 300s: that run finished 2.9 seconds under it, and under the old 240s wall it would have been killed outright. Two installs, two re-fits, and in both the tail lands within seconds of whatever number was chosen. So the constant is not the fix — it buys the run a chance, and the retry is what actually bounds the failure rate. Worth saying plainly in the comment above the constant: this number will drift; do not re-fit it without re-measuring, and do not treat a bigger number as a substitute for (b).

On the parse-retry tail — confirmed here in code. The corrective retry accumulates into the same counter (inferenceDuration += Date.now() - retryStarted), so one row conflates up to three inference calls once (b) is in. Agreed that attempts is what makes such a row readable; on this install it is logged on every terminal path — inference failure, parse failure, and success — so "succeeded on the second try" and "hung twice" never share a shape. A reader still needs to know the parse retry is not counted in attempts; that is arguably a second field rather than a bigger one.

On cap pressure as the underlying driver. Agreed, and it has an independent remedy worth pairing with the re-fit: gate the memory set below the cap, divert entries that would exceed it to an overflow log, and surface a health finding so a human sorts them. On this install the health check now reports cap-ok and overflow:empty, which decouples reviewer output size from memory growth. Without something like that the reviewer's response grows monotonically and every fixed wall is on a countdown.

On the streak counter and skipped. Agreed — and that behaviour is already in the proposal above: the counter steps over rows with skipped === true, so a skip neither breaks nor extends a streak. Two failures separated by a healthy no-op still read as two in a row, which is the case you name.

On (c) staying critical on your install because the real streak was 3 — that is the property that makes the change safe to take, and it is better evidence than either of our two-pole tests, since nobody constructed it.

[danielmiessler]

You were right on all three counts, and my own install shows the same shape: the slowest successful run in the last sixty finished six seconds under the 240s wall. Landed: one retry on a timeout with the attempt count logged on every terminal row, the health check moved to a streak so one dropped run is a warn and two in a row is critical, and the wall re-fit to 300s with a note that the number drifts and the retry is the real fix. Credited you and jacobo at each site. Next release. Thanks for the measurement.