Skip to content

Fixes for issues found during the hackathon - #281

Open
david-hummingbot wants to merge 5 commits into
mainfrom
fix/hackathon-fixes
Open

david-hummingbot wants to merge 5 commits into
mainfrom
fix/hackathon-fixes

Conversation

@david-hummingbot

@david-hummingbot david-hummingbot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A collection of fixes for issues found during the hackathon. More fixes will be added to this branch before it's ready to merge.

Notifications: "Clear all" replaces "Mark all read"

The notification bell's header action now deletes the user's notification history instead of only marking it read. Opening the bell still marks everything read, so the unread badge behaves as before.

  • Backend: new clear(user_id) in condor/notifications.py, scoped to the caller's own notifications and run under the store's write lock. Exposed as DELETE /api/v1/notifications.
  • Frontend: api.clearNotifications(). The bell shows Clear all with an inline two-step confirm (Clear all? · Clear · Cancel) that resets when the dropdown closes.
  • Test: clearing empties only the caller's notifications and leaves other users' untouched.

Agent code can no longer freeze or exit the main process

run_code snippets and routines run on Condor's main event loop, and the existing timeout could not stop synchronous code. During the hackathon an agent's snippet ran glob.glob("/**/_theta_derive.py", recursive=True), walked /sys on the main thread for hours, and froze the dashboard, every agent and the shutdown handler. Separately, exit() in a snippet or routine stopped Condor outright (exit code 0, nothing logged).

New condor/runtime/code_guard.py, scoped to the run and any tasks it spawns:

  • Timeout interrupts synchronous code via a sys.monitoring line/jump callback and an audit hook, so a stuck snippet is cut at its budget on the offending line.
  • Refused: listing /, /proc, /sys, /dev; input(); signalling Condor's own process; time.sleep over 1s; os._exit / os.abort / signal.raise_signal (checked before running).
  • exit() / sys.exit() / KeyboardInterrupt become an ordinary error.
  • Tasks a snippet left running are stopped when its run ends, but runs Condor starts on its behalf (start_routine) are detached and outlive it. help() no longer opens an interactive pager.
  • After a timeout, finally / except / with / async with cleanup gets up to 2s to run, whether the snippet was busy or awaiting, so locks and files are released.

New condor/runtime/loop_watchdog.py logs the loop thread's stack when the event loop is blocked for 10s or more.

Not covered in-process: a single long call into C, and memory use. Both need snippets in a separate worker process, which is a follow-up.

Test plan

  • uv run pytest tests/test_notifications.py (29 passed)
  • Frontend typecheck and npm run build
  • Local dashboard: Clear all button renders in the bell
  • uv run pytest (6468 passed), including 39 tests in tests/test_code_guard.py
  • Live dashboard via /api/v1/code/run: the recursive glob from / is refused in 2ms, while True: pass is cut at its 3s budget, exit() returns an error, and the dashboard keeps serving
  • Manual: Clear all → Cancel keeps the list; Clear all → Clear empties it and the list stays empty after a reload

The bell's header action now deletes the user's notification history
instead of only marking it read. Opening the bell still marks everything
read, so the badge behaves as before.

- condor/notifications.py: add clear(user_id), scoped to the caller's
  bucket and taken under the write lock so a concurrent record() is not
  lost or resurrected
- DELETE /api/v1/notifications route and api.clearNotifications()
- NotificationBell: "Clear all" with an inline two-step confirm
  (Clear all? · Clear · Cancel), reset when the dropdown closes
- test: clearing empties only the caller's bucket
@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[High impact] The PR appears safe to merge; the cleanup finding is addressed and no new blocking issue was found.

Summary

This PR adds notification deletion, guards against agent code freezing or stopping Condor, and a watchdog that logs blocked-loop stacks.

Since the last review:

  • Agent code gets checks that stop selected calls from taking Condor down.
  • A watchdog logs where Condor’s event loop gets stuck.
  • The notification bell lets users clear their history with confirmation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Snippet reaches its deadline] --> B{Handling an exception?}
  B -->|No| C[Interrupt at a safe instruction or audited operation]
  B -->|Yes| D{Cleanup grace ended?}
  D -->|No| E[Allow cleanup]
  D -->|Yes| C
  C --> F[Record timeout]
Loading

Reviews (5) · Last reviewed commit: "fix(runtime): let cleanup run when a sni..." · Reviewed by Greptile

Comment thread condor/notifications.py Outdated
Comment thread frontend/src/components/layout/NotificationBell.tsx
Comment thread frontend/src/components/layout/NotificationBell.tsx Outdated
Comment thread frontend/src/components/layout/NotificationBell.tsx
- clear() no longer swallows a failed write; the route logs it and
  returns 500, so the bell never shows success for a delete that did
  not happen
- cancel an in-flight list refresh before clearing, and re-read the
  store afterwards instead of leaving an empty snapshot that could hide
  a notice recorded just after the delete
- opening a notification closes the bell through close(), so the
  confirm step is disarmed on every way out of the dropdown
- show an error message when the clear fails, with Clear still
  available to retry
- tests: backend failure path, and a component test for the confirm,
  error and disarm behaviour
run_code snippets and routines execute on Condor's main event loop, so
synchronous code in them could not be interrupted by asyncio.wait_for.
On 2026-10-07 a snippet's recursive glob from / walked /sys on the main
thread for hours and froze the dashboard, every agent and the shutdown
handler. Separately, exit()/sys.exit() in a snippet or routine was
re-raised out of the event loop and stopped Condor with exit code 0.

condor/runtime/code_guard.py, scoped by a ContextVar to the run and the
tasks it spawns:
- the run's timeout now interrupts synchronous code: a sys.monitoring
  LINE/JUMP callback on the snippet's own code objects and an audit hook
  on everything it calls raise CodeRunDeadline once past the budget
- refuse listing /, /proc, /sys and /dev; input(); signalling Condor's
  own process; time.sleep over 1s
- refuse os._exit, os.abort and signal.raise_signal before running
- turn SystemExit/KeyboardInterrupt into an ordinary error
- stop tasks a snippet left running when its run ends

Also: help() no longer starts an interactive pager (the stdout proxy
reports isatty() False while capturing), and a loop watchdog logs the
loop thread's stack when the event loop is blocked for 10s or more.

Not covered in-process: a single long call into C, and memory use.
Those need snippets in a separate process.
Comment thread condor/routine_store.py
Comment thread condor/runtime/code_guard.py Outdated
Addresses review feedback on the code guard.

Background routines started from a snippet died with it. RoutineStore
started execute/start_continuous/schedule runs with asyncio.create_task,
which copied the snippet's guard scope, so a start_routine() run carried
the snippet's deadline and was cut once the snippet finished. The
CodeRunDeadline then bypassed _execute_and_record's except Exception and
left the instance "running" with no result.
- add code_guard.detached_context() and start the three background runs
  in it; call_routine (inline) still inherits the caller's deadline
- record a CodeRunDeadline in _execute_and_record as a failed run

A timed-out snippet's cleanup never ran, leaving e.g. a lock held.
The guard raised on every line after the deadline, and on CPython 3.12
an exception raised from a JUMP callback, on a compiler-generated
instruction or on a handler's first instruction (PUSH_EXC_INFO) skips
the frame's finally/except/__exit__.
- LINE and JUMP now only arm INSTRUCTION events; the INSTRUCTION
  callback raises at the first instruction where handlers still run,
  then disarms
- code handling an exception may run for up to INTERRUPT_GRACE (2s)
  before it is interrupted too; everything else is interrupted again,
  so a loop that swallows the interruption still ends

Tests: cleanup runs after a timeout for finally, a one-line loop, a
loop in a function, except and a with block; a hanging cleanup and a
swallowed interruption are still cut; a routine started from a snippet
completes and records its status.
A snippet that timed out mid-await is cancelled by wait_for, not by
the guard, so the guard had no interruption on record and treated the
cleanup's first line as the first interruption. Its finally, except or
async-with exit was cut, and a shared lock it held stayed held.

Code handling any exception (including that CancelledError) now gets
the INTERRUPT_GRACE cleanup window on the first interruption too,
measured from the deadline when the guard did not interrupt first.

Tests: cleanup runs after an awaiting timeout for finally, async with,
and a finally that does file I/O.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant