Skip to content

feat: weekly sweep for CSA sources this repo isn't wiring up yet - #49

Merged
kurtseifried merged 2 commits into
mainfrom
feat/periodic-source-sweep
Sep 1, 2026
Merged

feat: weekly sweep for CSA sources this repo isn't wiring up yet#49
kurtseifried merged 2 commits into
mainfrom
feat/periodic-source-sweep

Conversation

@kurtseifried

Copy link
Copy Markdown
Contributor

Why

Nothing in CSA tells DesktopSetup when there is new tooling to wire up. A marketplace gets created, a plugin ships, an MCP server becomes ready — and this repo carries on installing the set it knew about the day someone last edited it. The gap is invisible from inside the repo: every script parses, every check passes, and new hires get a smaller toolset than the people who set their machines up by hand.

Three extension points drift independently, and none of them was checked by anything:

# What appeared Where it gets wired Cost
1 new plugin marketplace CSA_MARKETPLACES + plugin_marketplace_repo — 5 scripts edit + SCRIPT_VERSION bump ×5
2 new plugin in a known marketplace scripts/csa-plugins*.txt one commit, no version bump
3 new MCP server setups=() in setup_csa_internal_tools — 5 scripts edit + SCRIPT_VERSION bump ×5

What's here

  • tools/sweep-csa-sources.sh — probes the CSA orgs and diffs against all three. Flags --all-orgs, --quiet. Exit 0 no drift / 1 drift / 2 could not complete.
  • docs/periodic-sweep.md — runbook: what each finding means, which of the three places to fix it, and the two ways the sweep can under-report.
  • CLAUDE.md — a weekly-cadence pointer, plus a write-up of setup_csa_internal_tools, which was the one registration mechanism of the three with no documentation.

The sweep parses the lists out of macos-ai-tools.sh rather than restating them, so it can't itself become a sixth place the lists drift.

Findings on main today

1. Plugin marketplaces  all registered
   (CloudSecurityAlliance/Research-Plugins named and skipped — it's a
    stale private fork of the internal one; registering it would shadow
    the real marketplace)

2. Plugins              15 published, referenced by no install list
   csa-cino-plugins:    ai-activity-scanner claude-code-permissions
                        analyze-csa-artifact rename-project
                        csa-github-tools vendor-neutrality sense-making
                        landscape-mapper valid-ai-ted-evidence-based
   csa-research-plugins: artifact-control-mapper
   csa-training-plugins: csa-training-operations csa-slack-intelligence
                         csa-training-handoff
   csa-plugins-official: secid vulnerability-audit

3. MCP servers          none waiting to be wired
   csa-zendesk not ready — no internal-setup script yet

This PR only adds the sweep; it does not act on the findings. Whether any of those 15 plugins belong in a default install is a human call.

Two deliberate choices worth reviewing

Not in check-all.sh. It needs the network and a gh token with CSA-Internal access. CI has neither, and a check that cannot pass in CI is a check that gets deleted.

Probing is sequential. An earlier version used xargs -P 12 and reported three repos as having no marketplace.json when all three demonstrably do. Under load, a failed probe is indistinguishable from a genuine absence — so the sweep under-reports and the failure reads exactly like success. It now probes one at a time, separates 404 from other errors, and exits 2 if any probe fails, so an incomplete run can never be mistaken for a clean one. This is called out in the script header, the runbook, and CLAUDE.md, because it will look like an obvious thing to optimise. ~1 min/week.

Known blind spot

Section 3 requires a candidate to be named csa-* and mention "MCP" in its GitHub description — csa-* alone matched 13 csa-ai-exam-* / csa-research-* data repos that are not servers. A new server whose description omits "MCP" will be skipped; the script prints the skipped count rather than hiding it. Documented in both the script and the runbook.

Verification

./tools/check-all.sh — all checks pass (bash -n, shellcheck, duplication, native-call guards, paste safety, debug mode, PS parse, PSScriptAnalyzer, 26 Pester tests). shellcheck --severity=warning clean on the new script. Sweep run end-to-end against the live orgs (193 repos), output above.

Nothing in CSA notifies DesktopSetup when new tooling appears, so the
installed set silently lags what exists. Three extension points drift
independently and none of them is checked by anything:

  1. CSA_MARKETPLACES (5 scripts)      — new marketplace repos
  2. scripts/csa-plugins*.txt          — new plugins in known marketplaces
  3. setups=() (5 scripts)             — new local MCP servers

Adds tools/sweep-csa-sources.sh, which probes the CSA orgs and diffs
against all three. It parses the lists out of macos-ai-tools.sh rather
than restating them, so the sweep cannot itself become a sixth place
the lists drift.

Current findings on main: no unregistered marketplaces, 15 published
plugins referenced by no install list (9 of them in csa-cino-plugins),
and csa-zendesk present but not ready (no internal-setup script yet).

Not wired into check-all.sh: it needs network and a gh token with
CSA-Internal access, and a check that cannot pass in CI gets deleted.

Probing is sequential on purpose. An earlier version used xargs -P 12
and reported three repos as having no marketplace.json when all three
do — under load a failed probe is indistinguishable from a genuine
absence, so the sweep under-reports and reads as "no drift". It now
separates 404 from other errors and exits 2 if any probe fails, so an
incomplete run can never be mistaken for a clean one.

Also documents setup_csa_internal_tools in CLAUDE.md, which was the
one registration mechanism of the three with no write-up.
The weekly routine (trig_01TQh4GMWKRnt4L4QpM5mhJc, Mondays 15:04 UTC)
carries a deliberately short prompt that says 'read docs/periodic-sweep.md,
section Running this as a scheduled routine, and follow it exactly'. That
section did not exist yet. Adding it, so the job is defined in the repo
where it can be reviewed and changed, rather than inside a prompt nobody
can see from here.

Includes the constraint to verify on first run: the cloud token may not
have CSA-Internal read access, in which case the routine stops and says
so rather than reporting a false all-clear.
@kurtseifried
kurtseifried merged commit 5c74666 into main Sep 1, 2026
6 checks passed
@kurtseifried
kurtseifried deleted the feat/periodic-source-sweep branch September 1, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant