Skip to content

Beat on startup and on a timer, so an idle watcher is not reported dead - #11

Open
Sujthr wants to merge 1 commit into
theschoolofai:mainfrom
Sujthr:fix/heartbeat-only-beats-on-events
Open

Sujthr wants to merge 1 commit into
theschoolofai:mainfrom
Sujthr:fix/heartbeat-only-beats-on-events

Conversation

@Sujthr

@Sujthr Sujthr commented Aug 10, 2026

Copy link
Copy Markdown

EventStore.beat() had exactly one caller: AutonomousEventEngine.process. The
heartbeat advanced only when an event arrived — never at startup, never on a timer.
So the counter meant "an event arrived", not "I am alive", which is the one thing
Section 9 needs it to mean.

What breaks

A healthy process that correctly receives nothing for STALE_AFTER_SECONDS (900)
reports itself dead. Observed live on 2026-08-10, minutes after a restart, while
the process served every other request normally:

$ curl http://127.0.0.1:8113/v1/agent/liveness
{"alive": false,
 "reason": "no heartbeat for 16235s; the watcher may have died",
 "beats": 15,
 "last_beat": "2026-08-10T13:54:30Z"}     <- the last EVENT, hours earlier

Three consequences:

  1. The alarm fires on correct behaviour. Section 8 says a quiet night is often
    the right answer; here a quiet night returns 503. Anyone paging on this endpoint
    is woken by every idle night, learns to ignore it, and then misses a real death.
  2. The morning report's header lies. The one line that tells an operator whether
    the watcher was up all night reads "NOT ALIVE ... the watcher may have died" for
    a process that never stopped.
  3. A fresh process is dead on arrival, reporting "no heartbeat has ever been
    recorded" until something happens to it.

Section 9 is explicit about both the mechanism — "the agent taps a counter every
time it wakes up and every time it handles an event" — and the reason: "a dead
process cannot send you an error, but it also cannot fake a pulse." A pulse that
exists only when work arrives cannot separate idleness from death, so 503 stops
being a signal an operator can act on.

The fix

  • beat once at startup, so a live process is never reported as never-having-lived
  • run a background beat every S16_HEARTBEAT_SECONDS (default 60, well inside the
    900s window), cancelled cleanly on shutdown
  • a failed beat is swallowed and retried on the next tick, so one bad write cannot
    stop the pulse — while a genuinely stuck store still goes stale and raises the
    alarm

Proof

tests/test_liveness_does_not_require_events.py: 3 of its 6 tests fail before this
change and all pass after. It also asserts the alarm still works — a genuinely
stale beat is still alive: false, and a store that never beat is still refused —
so the endpoint has not been made unconditionally green.

test_control_plane_auth.py::test_read_only_observability_is_available_to_an_operator
is updated: it asserted 503 and "NOT ALIVE" for a freshly started process,
which encoded the defect. It now asserts a beating process says so, and the alarm
case is covered directly against a stale beat in the new file.

`EventStore.beat()` had exactly one caller: `AutonomousEventEngine.process`. The
heartbeat advanced only when an event arrived — never at startup, never on a timer.
So the counter meant "an event arrived", not "I am alive", which is the one thing
Section 9 needs it to mean.

## What breaks

A healthy process that correctly receives nothing for `STALE_AFTER_SECONDS` (900)
reports itself dead. Observed live on 2026-08-10, minutes after a restart, while
the process served every other request normally:

    $ curl http://127.0.0.1:8113/v1/agent/liveness
    {"alive": false,
     "reason": "no heartbeat for 16235s; the watcher may have died",
     "beats": 15,
     "last_beat": "2026-08-10T13:54:30Z"}     <- the last EVENT, hours earlier

Three consequences:

1. **The alarm fires on correct behaviour.** Section 8 says a quiet night is often
   the right answer; here a quiet night returns 503. Anyone paging on this endpoint
   is woken by every idle night, learns to ignore it, and then misses a real death.
2. **The morning report's header lies.** The one line that tells an operator whether
   the watcher was up all night reads "NOT ALIVE ... the watcher may have died" for
   a process that never stopped.
3. **A fresh process is dead on arrival**, reporting "no heartbeat has ever been
   recorded" until something happens to it.

Section 9 is explicit about both the mechanism — "the agent taps a counter every
time it *wakes up* and every time it handles an event" — and the reason: "a dead
process cannot send you an error, but it also cannot fake a pulse." A pulse that
exists only when work arrives cannot separate idleness from death, so 503 stops
being a signal an operator can act on.

## The fix

- beat once at startup, so a live process is never reported as never-having-lived
- run a background beat every `S16_HEARTBEAT_SECONDS` (default 60, well inside the
  900s window), cancelled cleanly on shutdown
- a failed beat is swallowed and retried on the next tick, so one bad write cannot
  stop the pulse — while a genuinely stuck store still goes stale and raises the
  alarm

## Proof

`tests/test_liveness_does_not_require_events.py`: 3 of its 6 tests fail before this
change and all pass after. It also asserts the alarm still works — a genuinely
stale beat is still `alive: false`, and a store that never beat is still refused —
so the endpoint has not been made unconditionally green.

`test_control_plane_auth.py::test_read_only_observability_is_available_to_an_operator`
is updated: it asserted `503` and `"NOT ALIVE"` for a freshly started process,
which encoded the defect. It now asserts a beating process says so, and the alarm
case is covered directly against a stale beat in the new file.
@theschoolofai

Copy link
Copy Markdown
Owner

Session 16 — graded ✅

Score: +100.

Liveness only beat on event arrival, so an idle watcher was indistinguishable from a dead one. That is precisely the alarm Part 1 asks students to demonstrate, so it mattering here is fitting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants