Skip to content

Fix mirror pull failing on nested skill bundles, concurrent pulls, and dashboard sync crash - #78

Open
Ascurse wants to merge 5 commits into
AMAP-ML:mainfrom
Ascurse:fix/skill-sync-robustness
Open

Fix mirror pull failing on nested skill bundles, concurrent pulls, and dashboard sync crash#78
Ascurse wants to merge 5 commits into
AMAP-ML:mainfrom
Ascurse:fix/skill-sync-robustness

Conversation

@Ascurse

@Ascurse Ascurse commented Aug 16, 2026

Copy link
Copy Markdown

Four independent failures hit while running SkillClaw against a Hermes skills directory. Each is reproducible and fixed separately.

1. Mirror pull fails permanently when a skill bundle contains nested SKILL.md files

_list_local_skill_dirs globs SKILL.md at any depth, so examples shipped inside a skill (e.g. claudeception/examples/*) are listed as standalone local skills. They are absent from the remote manifest, so mirror pull deletes them as stale — the parent directory first, then rmtree hits the now-missing nested paths with ignore_errors=False:

[SkillHub] mirror pull failed, restoring backup: [Errno 2] No such file or directory:
  '.../skills/claudeception/examples/nextjs-server-side-error-debugging'
[SkillHub] skill pull: 0 downloaded, 0 unchanged, 0 failed, 0 deleted, 101 total remote

Every pull rolled back, so nothing was ever downloaded. push_skills had the same unbounded glob and would have uploaded those nested examples to the hub as separate skills.

2. Concurrent pulls corrupt the skills directory

pull_skills is called by the launcher's auto-pull, the 30s reload poller and the CLI, with no mutual exclusion. Two overlapping pulls walk the same directory and one moves bundles out of staging while the other is still reading them, producing a burst of FileNotFoundError across unrelated skills.

3. Backup rollback fails exactly when needed

The rollback path does rmtree(skills_dir) then copytree(backup_dir, skills_dir). When the tree is being mutated concurrently, rmtree leaves directories behind and the restore aborts:

[SkillHub] backup restore failed: [Errno 66] Directory not empty: '.../skills/productivity'

4. dashboard sync crashes on date frontmatter

TypeError: Object of type date is not JSON serializable — PyYAML parses a bare date: value in skill frontmatter into datetime.date, which json.dumps rejects.

Changes

  • Bound skill discovery to the supported layouts (<root>/<name>, and <root>/<category>/<name> for the hermes root); nested SKILL.md files are bundle contents, not skills.
  • Extract that walk into skill_bundle.iter_skill_md_paths and route skill_hub, skill_manager and dashboard_ingest through it — they previously used three different rules, which is why the loader reported 105 skills and the dashboard 106 where the disk had 102.
  • Skip stale entries already removed together with a parent directory.
  • Guard pull_skills with a non-blocking flock; a busy lock returns locked_out and the cycle is skipped rather than racing. Surfaced in the CLI so a skipped pull is not mistaken for a successful empty one.
  • Rename the broken directory aside before restoring the backup, so rollback cannot trip over its own leftovers.
  • Serialize date/datetime values when writing the dashboard snapshot.

Verification

  • Fixture with a nested example bundle and a nested .claude/skills set: iter_skill_md_paths, SkillHub._list_local_skills and SkillManager all return the same 3 skills; before the change the latter two returned 5.
  • Lock: a second concurrent acquisition returns False and the lock is released afterwards. Four concurrent skillclaw skills pull runs against a live daemon — exactly one proceeds, three report Skipped: another pull is already running, and the skills directory is untouched.
  • Rollback: restoring over a directory holding leftovers from a half-written pull now succeeds and cleans up after itself; the previous code raised Directory not empty.
  • On a real 102-skill library the daemon now logs pull complete: 0 downloaded, 102 skipped, 0 deleted, 102 total remote on every cycle, with no rollbacks, and loader/hub/dashboard all report 102.

🤖 Generated with Claude Code

Ascurse and others added 5 commits August 16, 2026 18:56
_list_local_skill_dirs globbed SKILL.md at any depth, so example bundles
shipped inside a skill (e.g. claudeception/examples/*) were listed as
local skills. They are absent from the remote manifest, so a mirror pull
deleted them as stale: the parent directory went first, then rmtree hit
the now-missing nested paths and FileNotFoundError rolled back the whole
pull. With a nested-skill bundle present, every pull failed and restored
the backup, so no skill was ever downloaded.

Discovery is now bounded to the supported layouts (<root>/<name> and, for
the hermes root, <root>/<category>/<name>), push reuses the same walk, and
the stale-delete loop skips a directory already removed with its parent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
'skillclaw dashboard sync' crashed with "TypeError: Object of type date is
not JSON serializable" whenever a skill's YAML frontmatter contained a bare
date, since PyYAML parses those into datetime.date. Dump them as strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pull_skills is invoked from several places at once — the launcher's
auto-pull, the 30s reload poller, the CLI, and any second instance — all
against the same directory. Two overlapping pulls corrupt each other:
one moves a bundle out of staging while the other is still reading it,
producing a burst of FileNotFoundError.

Rollback then failed on its own: rmtree in place left partially removed
directories behind and copytree aborted with "Directory not empty", so the
safety net was gone exactly when it was needed.

Add a non-blocking flock around pull_skills (a busy lock returns
locked_out and the cycle is skipped), and rename the broken directory
aside before restoring the backup copy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The skills directory was walked in three places with three different
rules: skill_hub (sync), skill_manager (loading and injection) and
dashboard_ingest (snapshot). The latter two globbed SKILL.md recursively
without a depth bound, so example bundles nested inside a skill counted as
standalone skills — the agent was offered 105 skills instead of 102 and
the dashboard listed 106.

Move the walk into skill_bundle.iter_skill_md_paths and route all three
call sites through it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror pull unconditionally copied the whole skills directory into a
backup and then again into a staging dir before comparing hashes. With the
30s poller this rewrote every skill on disk twice a minute and kept ~200MB
of backups alive, even though nothing ever changed.

Compare hashes first and return early when there is no stale skill, no
duplicate local dir and every bundle matches the manifest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Upper9527

Copy link
Copy Markdown
Collaborator

The underlying reports are valuable, but this branch is not mergeable as-is:

  1. import fcntl is unconditional, so importing skillclaw.skill_hub fails on Windows. The repository has Windows users; the lock needs a cross-platform implementation or guarded fallback.
  2. iter_skill_md_paths() recognizes only the literal ~/.hermes/skills root, so it ignores custom HERMES_HOME/profile layouts and collides with the path work in feat(claw-adapter): resolve Hermes home from env (SKILLCLAW_HERMES_HOME / HERMES_HOME) #57/fix: honor HERMES_CONFIG/HERMES_HOME for multi-profile setups #72.
  3. The PR body describes several regression fixtures, but no tests are included in the diff. Please commit those tests, including nested bundle contents and concurrent/locked pulls.
  4. fix: dashboard snapshot sync crashes on date fields #75 has now landed, so please rebase and drop the duplicate dashboard serialization change.

I recommend splitting discovery/layout normalization from pull locking/rollback so each safety change can be reviewed and tested independently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants