Repository navigation
Lazaretto, a hosted scanner for agent skills #69
Description
Scanner name
Lazaretto
Upstream URL
https://github.com/jamesdfinance-dev/lazaretto-plugins/tree/main/clawscan
Request type
New built-in scanner adapter
CLI/API shape
Following docs/scanners.md, I started with a user-defined command scanner for a custom profile rather than asking for a built-in adapter up front:
node clawscan/lazaretto-scanner.mjs {{target}}
{{target}} is a local skill directory (SKILL.md is scanned, every other file is listed as not scanned) or a published ClawHub skill as owner/name[@Version] (the whole package is scanned server side). One JSON object on stdout. Exit 0 when a scan completed, exit 1 with a reason when there is no result, so a failed scan never reads as clean. A sample profile with gate rules is in profile.example.yaml.
Credentials and setup
Env vars: LAZARETTO_API_KEY (declared under secretEnv). A free key: curl -s -X POST https://lazaretto.dev/v1/trial
Install: Node 20 or newer. No dependencies; the script reads files and makes one HTTPS request.
Expected artifact
JSON with status, verdict (clear, flagged, malicious), risk, findings (rule_id, severity, file, line, detail), scanned, not_scanned, complete, target_hash and scan_id. On failure: status "error", a reason, and not_an_all_clear: true.
Example proof
Fixtures and results are in the README. Against the live API on 2026-10-20, a skill that templates the conversation into a rendered image URL is flagged high (rule pi.markdown_exfil), and a plain skill comes back clear with its unscanned helper script listed. I have not yet run it under ClawScan itself.
Two questions before going further:
- Is a hosted, metered scanner (free tier, then paid per call) something you would consider for a profile, and at what volume would cost become the deciding factor?
- How should a scanner that is rate limited or unavailable report, so the harness treats it as skipped rather than clean?
Activity
clawsweeper commentedon Oct 20, 2026
Codex review: this still needs some work.
Summary
Keep this issue open for an adoption decision: current main supports custom scanners but does not include Lazaretto, so the repository’s implemented-only close policy is not met.
Reproducibility: This is an adoption proposal rather than an established ClawScan defect; the author supplies external 0.2.0 evaluation evidence, and no live reproduction was performed.
Maintainer decision needed
Next step
Assess current-main compatibility, benchmark value, service terms, cost, and ownership before choosing external-only, documented-profile, or built-in support.
Review details
Best possible solution:
Evaluate the existing custom profile on current main and shared benchmarks before committing to repository-maintained Lazaretto support.
Do we have a high-confidence way to reproduce the issue?
This is an adoption proposal rather than an established ClawScan defect; the author supplies external 0.2.0 evaluation evidence, and no live reproduction was performed.
Is this the best way to solve the issue?
Custom-profile evaluation follows the documented adoption process; built-in support should follow agreement on value, ownership, target coverage, and current failure semantics.
AGENTS.md: found and applied where relevant.
Remaining risk / open question:
Codex review notes: model internal, reasoning medium; reviewed against 3308fba908e3.
Label changes
Label changes:
No label changes.
Label justifications:
P3: Optional scanner adoption has an existing custom-profile path and establishes no urgent user impact or broken ClawScan contract.Evidence reviewed
What I checked:
AGENTS.md:1, 3308fba908e3)internal/runner/scanner_registry.go:201, 3308fba908e3)docs/scanners.md:93, 3308fba908e3)internal/runner/lifecycle.go:123, 3308fba908e3).github/workflows/clawscan-eval.yml:35, 42215c5bb2c5)Likely related people:
internal/runner/lifecycle.go)How this review workflow works
@clawsweeper re-reviewcomments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.@clawsweeper re-reviewor@clawsweeper re-runon an open PR or issue to request a fresh review only.@clawsweeper reviewto request a fresh review only.@clawsweeper explainto ask for more context, or@clawsweeper stopto stop active automation.Reviewed October 30, 2026, 3:40 PM ET / 19:40 UTC.
fix(clawscan): an unavailable scan never passes the gate, and prove i…
jamesdfinance-dev commentedon Oct 20, 2026
Thanks for the review. The gate finding was right: an unreachable service exited 1 with valid JSON, ClawScan recorded the scanner as completed, and the example profile had no rule for it, so the gate passed.
Fixed in jamesdfinance-dev/lazaretto-plugins@42215c5
The rate-limited and out-of-credits cases use a local stand-in, run with --sandbox off because the container cannot reach the host loopback. The key appears in no artifact; ClawScan records it as present. The evaluation runs on every change and weekly, and keeps the artifacts.
For the cost and data question: there is a free tier of 10 scans a day per key, then $0.03 a scan or prepaid packs. A local scan sends SKILL.md; a ClawHub reference sends only the slug. We keep the hash and verdict metadata, not the content.
Open question for maintainers: keep this as a documented custom profile, or evaluate it for a built-in adapter alongside the existing socket scanner?
183 remaining items