Skip to content

Flaky: test_skill_listing_cost supplied_mtime test catches a stray config.json stat (Linux + Windows) #21079

Closed

Description

@CrysisDeu

What

test/test_skill_listing_cost.py::TestListingStaysCheapOnTheEventLoop::test_a_supplied_mtime_skips_the_frontmatter_cache_stat failed on 3 unrelated branches. It fails on both Linux and Windows, always in shard 3:

AssertionError: assert ['/tmp/pytest.../config.json'] == []

Why it's a flake

None of these branches touch skill listing. The test asserts that no stat happens on the event loop. The stat it catches is of a config.json under the pytest tmp dir, not a skill frontmatter file. That looks like a stat from some other path, like a config load in a background or earlier-scheduled task, landing inside the test's observation window. If so, the result depends on what else the worker ran first (determinism class 4, order dependence).

Likely fix direction

Narrow the stat spy to the frontmatter-cache paths the test is about. Or make sure nothing else on the loop loads config.json during the window, for example by pinning the config load before the spy starts.

(Filed by the CI flake patrol.)

Activity

  1. added
    tier:T1Bug: behaviour does not match intent; fix keeps the design
    on Oct 16, 2026
  2. dodgeblaster commented on Oct 20, 2026

    @dodgeblaster
    Contributor

    Recurrence on an unrelated branch (PR #22566, kc-wt-15009, a dashboard skill-picker fix that touches none of this test or the config.json stat path):

    • Backend Tests (Windows) shard 3/8, run https://github.com/kirodotdev/KiroCrew/actions/runs/39497603019
    • test/test_skill_listing_cost.py::TestListingStaysCheapOnTheEventLoop::test_a_supplied_mtime_skips_the_frontmatter_cache_stat
    • AssertionError: assert ['C:\\Users\\...\\config.json'] == [] — the same stray config.json stat leaking from another xdist worker, Windows shard 3, matching this issue exactly.
  3. amadsalmon commented on Oct 25, 2026

    @amadsalmon
    Member

    Seen on fork PR #9749 (a99b1329aa), Linux, Backend Tests (3.12, 3) (job 121140703831): test_a_supplied_mtime_skips_the_frontmatter_cache_stat - AssertionError: assert ['/tmp/pytest.../config.json'] == []. The PR touches no skill-listing code. Fix PR #21084 is still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area: testsbugSomething is not workingtier:T1Bug: behaviour does not match intent; fix keeps the design

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions