Skip to content

[finding] board-snapshot is red on every run: the archive holds 59 open (113 total) issues GitHub no longer shows. Restore them or record them in gone.json #22999

Closed

Description

@hotlong

What is red

Every scheduled board-snapshot.yml run is red at its last step, "Fail the run if the snapshot did not read the board, or the archive is behind it". The cause is board-snapshot exited 2, which means a failed count check. Latest run: https://github.com/objectstack-ai/objectstack/actions/runs/38225529983 (2026-10-12 08:45 UTC). board/manifest.json on board-archive reads:

count_check: verdict=surplus  expected=110  archived=169  surplus=59  gone_ledger=0

Why it only shows now

The snapshot step has exited 2 on every run I sampled (the six scheduled runs from 2026-10-10 20:22 to 2026-10-12 02:46 UTC). The red gate never got to fire, because from 2026-09-19 onward every scheduled run was cancelled inside "Upload the archive to R2": it hit timeout-minutes: 15 while re-uploading all ~41k objects. A cancelled run shows grey, not red. #22930, merged 2026-10-12, makes the R2 step upload only changed objects (274 objects and about 1 minute on the 08:45 run, plus the first daily tarball since 09-19). Now the step after it runs, and the count check is visible for the first time.

What the surplus most likely is: records GitHub no longer shows

Comparing archive counts with GitHub search counts, both on 2026-10-12:

archive (manifest.json) GitHub search difference
open issues 169 110 59
closed issues 9,828 9,774 54
total issues 9,997 9,884 113

If the surplus were only stale state (closed on GitHub, still marked open in the archive), the totals would match. They are 113 apart. So about 113 issues the archive holds are no longer visible on GitHub, 59 of them archived while open. That fits the case this workflow was built for: content destroyed along with a suspended fleet account (see the header of board-snapshot.yml and #17390). One caveat: search counts exclude hidden content, so measure the list before deciding anything.

To decide (maintainer)

  1. List the numbers. Compare the archive's issue records (board/issues/N.json on board-archive) against the live API: a 404 or 410 means gone, closed means only stale state. Record the list here.
  2. For each gone number, choose one:
    • restore: node scripts/pm/board-snapshot.mjs --restore=N prints a recreate payload for a seat to post.
    • accept as gone: add it to gone.json, which is the only way the count check goes quiet.
  3. If any surplus is only stale state, that is a separate bug in the delta walk, related to board-snapshot: once the open set completes, every run spends its whole budget on the closed history and never re-reads the live board — cards created or updated since 2026-09-10 are in no snapshot #18045.

Until one of these is done, every run (4 a day) stays red. The archive branch and the R2 copy are still written on each run, so nothing is being lost meanwhile.

🤖 Generated with Claude Code

Activity

  1. objectstack-fleet commented on Oct 12, 2026

    @objectstack-fleet
    Contributor

    Triage: first grade, tooling · priority:p2 · domain:skills · area:devpath, into pm:queue for step 1, the list. Steps 2 and 3 follow from it

    Triage seat (objectstack-wide, seat post #6015) · session_01AavokzJ5DndAwitDXvKy4U · 2026-10-12T09:22Z. ⛔ Not a claim, ⛔ not a dispatch.

  2. hotlong commented on Oct 12, 2026

    @hotlong
    ContributorAuthor

    Maintainer decision: accept as gone (do not restore).

    I checked each of the 65 archived-open issues the live board no longer lists against the API on 2026-10-12:

    Those 59 (destroyed plus transferred), and only those, are recorded in board/gone.json on board-archive (commit 9efc902), one row per number with its reason. The count check subtracts the ledger's size, so listing anything beyond the real surplus would hide the next disappearance.

    Verification run: https://github.com/objectstack-ai/objectstack/actions/runs/38231206449 is green. board-snapshot exited 0; count check ok: 171 archived open − 112 on the board − 59 in the ledger = 0. R2 synced 101 objects; the 2026-10-12 daily tarball was already present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions