Repository navigation
OperationalError: disk I/O error #343
Description
Sentry Issue: GAB-11
OperationalError: disk I/O error
(3 additional frame(s) were not displayed)
...
File "github_agent_bridge/queue.py", line 612, in executor_paused
return self.state.executor_paused()
File "github_agent_bridge/persistence/state.py", line 103, in executor_paused
return self.executor_pause_state(connection=connection).paused
File "github_agent_bridge/persistence/state.py", line 83, in executor_pause_state
raw = self.get(EXECUTOR_PAUSE_STATE_KEY, "", connection=connection)
File "github_agent_bridge/persistence/state.py", line 61, in get
return self._get(con, key, default)
File "github_agent_bridge/persistence/state.py", line 117, in _get
row = con.execute("SELECT value FROM state WHERE key=?", (key,)).fetchone()
OperationalError: disk I/O error
(4 additional frame(s) were not displayed)
...
File "github_agent_bridge/persistence/runtime.py", line 71, in record_worker_heartbeat
with self.database.transaction(
File "contextlib.py", line 141, in __enter__
return next(self.gen)
File "github_agent_bridge/persistence/database.py", line 111, in transaction
con = self.read_write()
File "github_agent_bridge/persistence/database.py", line 70, in read_write
self._configure(con, read_only=False)
File "github_agent_bridge/persistence/database.py", line 185, in _configure
journal_mode = str(con.execute("PRAGMA journal_mode").fetchone()[0]).lower()
This issue was automatically created by Sentry via Github Issue
Reacted by GISCE Bot
Activity
giscebot commentedon Nov 11, 2026
Root cause confirmed: the SQLite database lives on
/dev/sdb, which had reached 100% usage. The simultaneous #344 alert reporteddatabase or disk is full; #343, #345, and #346 are follow-on I/O failures from the same incident.Operational recovery completed:
PRAGMA quick_checkreturnsokdisk I/O errorordatabase or disk is fullentries since recoveryPreventive fix: #349 adds a configurable 1 GiB free-space alert to the bridge monitor and skips monitor observability writes while the filesystem is below that threshold. Full suite: 716 passed.
Remaining operational risk: the volume still rounds to 100% usage. The worktree cleanup needs separate hardening; its latest run left 196 worktrees unverified.
giscebot commentedon Nov 11, 2026
Correction to the previous follow-up: #349 was superseded and closed without merge. The consolidated preventive fix is now #350 at commit 88a3ab5; it is open, mergeable, and measures the resolved parent filesystem of the SQLite database with fail-closed probe behavior.
I reviewed the consolidated diff and validated the current head locally: 60 focused monitor tests pass, and the full suite passes with 720 tests. The new regression explicitly fails if the capacity probe drifts to a filesystem other than the database mount. The latest GitHub Actions run is still queued: https://github.com/gisce/github-agent-bridge/actions/runs/43096761102
#343 remains open deliberately. Code validation is complete, but operational resolution still requires merging and deploying #350, reclaiming durable host capacity, and confirming a clean observation window. #348 separately covers fail-closed executor supervision for fatal heartbeat storage errors.