Repository navigation
Promote v1.19.3 to production - #3583
Conversation
Co-authored-by: Roomote <roomote@roomote.dev>
…rval (#3580) * [Fix] Re-create bullmq job schedulers whose offset outgrew their interval bullmq keeps an every-scheduler's stored offset when its interval changes and adds the whole offset when catching up on missed iterations, so a scheduler that once ran hourly and now runs every minute keeps firing about hourly. The heartbeat then goes stale between runs (failing /health/bullmq) and per-minute work such as custom automation checks runs late. On startup, remove any scheduler whose offset is not smaller than its interval before the upserts re-create it with a fresh offset. * fix: collect stale schedulers before removing them Schedulers are paged by sorted-set rank, so removing them while paging shifted later entries left and skipped stale ones past the first page. --------- Co-authored-by: Matt Rubens <2600+mrubens@users.noreply.github.com>
…er (#3581) * [Fix] Stop overlapping brain collector runs from starving the scheduler BrainCollectors runs on the shared scheduler worker every 15 minutes, and its historical backfill loop kept going as long as passes made progress. A long backfill (for example a large Notion workspace) left each run looping while the scheduler started a new one every tick, until every worker slot held a collector run. The heartbeat and per-minute jobs then never ran, the liveness watchdog exited the process, and stalled collector runs restarted on boot, repeating until the platform stopped restarting the service. - Only one BrainCollectors run collects at a time; a tick that finds one in flight finishes immediately. - A run stops looping after a 10-minute budget, well under the 15-minute cadence; the next tick resumes the backfill. * fix: resume PR facts from their exact cursor after a budget-ended run A collector run that ended on its time budget mid-backfill made the next run rewind PR facts by the overlap window. When one pass filled the budget, every tick rewrote the same facts and never advanced. Remember (in-process) that the last run ended on its budget and resume PR facts from the exact saved cursor; the overlap rewind still applies when a scan starts fresh or after a restart. --------- Co-authored-by: Matt Rubens <2600+mrubens@users.noreply.github.com>
Co-authored-by: @mrubens <2600+mrubens@users.noreply.github.com>
|
1 issue outstanding. See session
Reviewed e12f170 |
![roomote-community[bot]](https://avatars.githubusercontent.com/u/211522643?s=60&v=4){kind=small}
| }; | ||
| }, | ||
| }); | ||
| resumePullRequestFactsExactly = outcome === 'budget'; |
There was a problem hiding this comment.
[P2] Restore PR overlap when the PR scan itself has caught up
outcome describes all collectors, so a long Notion/Slack backfill sets resumePullRequestFactsExactly even when syncPullRequestFacts returned progressed: false and finished its scan. Subsequent scheduled ticks then keep skipping the PR overlap window for as long as that unrelated backfill exhausts the budget. The analytics writer deliberately shares one timestamp across sequential repository writes, so a later-committed fact behind the saved tuple is missed; if other facts advance the watermark by more than 24 hours before the backfill settles, the eventual overlap cannot recover it. An isolated probe of this orchestration with a settled PR scan and budget-consuming integration backfill produced overlap flags [true, false, false] across three scheduled runs. Preserve exact continuation only while PR facts themselves remain unfinished, and restore scheduled overlap once that source catches up.
Promote v1.19.3
Frozen at
e12f170159444cca3217a687887e6940d9a16b96— the commit where1.19.3was versioned. Commits merged todevelopafterward require an explicit candidate refresh or ship in the next release.v1.19.3and triggers the existing GHCRv*image publish (latestchannel).release/v1.19.3branch can be deleted after this PR merges.Changelog
1.19.3 (2026-10-24)
Roomote 1.19.3 keeps scheduled automations and worker health checks running reliably while memory history backfills make progress in the background.
Highlights
Patch changes