Quality tool: audit, overview & dedupe with HTML dashboard and agent skills - #11
Quality tool: audit, overview & dedupe with HTML dashboard and agent skills#11MirjamOdile wants to merge 14 commits into
Conversation
…nt skills Adds the read-only quality layer over data/<project>/ and its one mutating counterpart: - ./scrapai audit — coverage + extraction scoring per spider (CSV/MD), compliance lenses (robots/legal snapshots, conflict flags), external-PDF host analysis, and a self-contained HTML dashboard (Coverage · Compliance · PDFs tabs). PDFs follow the framework's row model: URL-only rows with content_type="pdf"; scraped/content% are judged over HTML rows only; same-org vs external is derived from allowed_domains. - ./scrapai overview — per-spider content profile: sections, date span, field coverage, thin/constant/off-domain checks. - ./scrapai dedupe — the ONE mutating command, kept separate so the audit never rewrites data; consolidates crawls/*.jsonl reversibly (originals kept as *.superseded). Ships with three slash-command skills (.claude/commands/: spider-review, spider-align, spider-slow) that consume the audit's status groups, docs (docs/quality.md, docs/skills-overview.md, request write-ups under docs/requests/), and the .gitignore pattern for teams to keep local skills/docs untracked alongside the shipped ones. Engine tests cover corpus scanning + caching, scoring (incl. PDF-row arithmetic and reviewed-stale guards), compliance capture/report, external-PDF partitioning, dashboard rendering, and an end-to-end synthetic-project flow.
3a4035b to
64d2f8a
Compare
2b2dba4 to
e411f49
Compare
|
Heads-up: two commits appended since you last looked — a bugfix bundle for the audit's own measurements ( What was broken: the audit scored spiders against corrupted numbers, so it flagged healthy spiders as broken:
What it does now: only 4xx count as dead; PDF citation rows get their own "pdf multi-ref" column while "versions" shows only genuine HTML changes (the counting itself is untouched, so Note: the media-loc and CF-robots fixes only fire on the sitemap FETCH path — they're logic-checked but still need one normal (fetching) audit run to be exercised live. Request doc: |
|
A short series of commits that make the compliance audit trustworthy on Cloudflare / proxy-gated sites: 1. Crawl axis — fall back to crawl-captured robots when the live probe is walled ( 2. The per-spider summary now honours the rescue ( 3. No-data rows sort to the TOP ( |
03a571f to
9d403bc
Compare
The audit scored against corrupted denominators/fingerprints, flagging healthy spiders as broken. All fixes are audit-only (reported numbers, never crawl behaviour or data): - A liveness: only 4xx count as dead; transient 5xx are excluded from the denominator (one spider: 810 transient 503s read as 'liveness 56%' when real coverage was ~97%; verified moved to ok). - B versions: the same PDF linked from N pages yields N rows differing only by found_on provenance; dedupe keeps them, so the counting is untouched — the scan now counts the PDF rows' own (url, fingerprint) pairs and the report splits 'pdf multi-ref' out of 'versions', which shows only HTML re-fetch churn (one spider: 10,932 phantom versions). Preserves the audit↔dedupe invariant that the tried counting change broke. - C: media attachment <loc>s (WP image sitemaps) excluded from the fetched coverage denominator (spider-side counter fixed in docs/requests/19). - D: a Cloudflare challenge page returned for robots.txt no longer reads as a valid robots with no Sitemap line — an HTML body triggers a browser retry (one site was mislabelled no-sitemap, hiding a ~50% loss). - E: default output is stage markers + a transient progress bar + summary; --verbose restores per-spider lines, the full external-PDF report, and per-org compliance detail. C/D fire only on the sitemap FETCH path and still need one live fetching audit to confirm (verification ran --no-fetch). docs/requests/22-audit-measurement-and-quiet.md
From this wave on the request number is the PR label (branch pr/<nn>-<slug>), continuing the 04-14 ledger; 15 (robots-obey default) was considered and withdrawn as a framework change (per-project preference, handled in spider configs), so the wave starts at 16. Requests 22 and 23 are patch commits on the open quality-tool and per-crawl-stats PRs, whose code they correct.
… walled
A capture-failure (the independent probe reached neither robots.txt nor the
homepage) is almost always a Cloudflare/TLS wall the audit's no-proxy probe
can't pass but the spider's own crawl (browser/curl_cffi/proxy) does. When that
spider captured robots.txt at crawl time we still hold an authoritative witness
for the CRAWL axis, so promote the domain into the report with a crawl-sourced
robots view ("(via crawl)") instead of leaving it "NOT CHECKED". The REUSE axis
stays a review — the homepage/licence was genuinely never read.
Report-time and read-only: the _capture_failed marker is left in place (the live
probe really did fail; --refresh still retries it) and old snapshots benefit
without re-capture. No crawl witness -> no rescue, so nothing is fabricated.
Extends the existing per-snapshot "rescued" path to the total-failure case.
9d403bc to
45f1d73
Compare
…port) compliance_summary() read has_capture_failed() directly and only loaded dated snapshots, so a capture-failed CF/proxy domain that build_report_data rescued via its crawl-captured robots still showed 'failed' in audit_<project>.md and the dashboard — disagreeing with compliance_<project>.md (which the docstring says can never happen). Feed it the already-rescued captured recs (computing them if not passed) so a domain answered via crawl-captured robots is not reported failed; no crawl witness -> still failed, nothing fabricated. +2 tests.
… → graded) The audit MD's per-spider compliance table and the dashboard's compliance tab both ranked‼️ capture-failed and ❓ not-checked rows LAST, so the most actionable rows (nothing known about them) sat at the bottom. Rank them first in both surfaces so the section reads most- to least-problematic.
Both tables appended NOT-CHECKED domains at the bottom; move them to the top so compliance_<project>.md reads most- to least-problematic, matching the audit MD and the dashboard's compliance tab.
The audit's third lens cost more than it returned, so it is cut before merge: external_pdf.py re-read the whole crawl corpus a second time with no scan cache, produced a multi-megabyte per-host report, and its dashboard tab needed a per-project host-exclusion layer (pdf_exclude.json plus a JS matcher, a chips UI and two JSON export paths) just to stay legible. Neither export had a consumer. Removed: core/quality/external_pdf.py, dashboard/pdfs_tab.py and their tests; the lens call in cli/audit.py; the PDFs tab, its select bar and the pdf_exclude scaffolding in dashboard/render.py; the PDF-only CSS/JS in dashboard/assets.py (URL island, exclusion matcher, share slider, include-list selbar mode, infobox). PDF handling itself is untouched: PDF_MODE still records every PDF link as a URL-only row, and the audit keeps the pdf / (N ext) column, the same-org vs external split, the pdf-only flag and the PDF-provenance split out of versions. overview's PDF stats and the robots "disallows PDFs" check are unaffected. The URLs stay in the crawl output, so the per-host view can be rebuilt if it ever earns its keep. render_dashboard() loses its pdf_spiders argument and write_dashboard() its pdf_result; nothing outside this branch calls them. Verified on a real 39-spider project (--no-fetch --no-compliance, static data): crawl_audit.csv and coverage.csv byte-identical before and after; audit_<project>.md differs only in the pdf glossary line that pointed at the deleted report; dashboard 3,853,820 -> 279,202 bytes.
Only the PDFs tab emitted `tr.fx-group`, so with that tab gone the group/collapse machinery in setupTable is unreachable: the `grouped` / `openG` state, the group branches in apply(), the group-header click handler and the fx-group CSS. Coverage and Compliance are flat tables and never entered any of it. Mechanical; kept out of the removal commit so it can be reviewed or reverted on its own.
The write-up led with a cost argument that was not the decision: the lens was dropped because it was convoluted and not particularly useful, and external-source scanning is moving elsewhere.
|
Dropped before merge: the external-PDF lens Two commits remove the audit's third lens — the external-PDF host report (core/quality/external_pdf.py → _audit/external_pdf_report.md) and the dashboard's PDFs tab, with its pdf_exclude.json host-exclusion layer. It was convoluted and not particularly useful in practice, and scanning external sources is moving elsewhere, so it does not need to ship here. PDF handling itself is unchanged — PDF_MODE still records every PDF link as a URL-only row, and the audit keeps the pdf / (N ext) column, the pdf-only flag, the PDF-provenance split out of versions, overview's PDF stats and the robots "disallows PDFs" check. The URLs stay in the crawl output.
render_dashboard() loses its pdf_spiders argument; nothing outside this branch calls it. Gate is green (black, flake8, unit suite). On a real project the audit's CSVs are byte-identical before and after, audit_.md differs only in the pdf glossary line that pointed at the deleted report, and the dashboard drops from 3.9 MB to 279 KB. |
The inert-template scaffolding went with the external-PDF lens; zero code references remain, so the quality doc's claim was stale.
a6fc6eb to
1f6ba85
Compare
… report The flag used to filter the spider set before the outputs were written, so a one-spider spot-check silently replaced the whole project report. --only now narrows only the work: un-scored spiders' rows merge back from the previous run's crawl_audit.csv (typed read via the shared CSV_FIELDS map — the CSV is a lossless copy of the audit rows) and, for the overview, from a new _audit/overview_rows.json row store (its CSV flattens the nested detail). The deleted-folder prune runs on the full spider set before the filter, and the compliance summary covers all spiders while capture stays narrowed to --only. Missing store: warn and fall back to the old narrowed output. No staleness stamping: rows carried over keep the semantics of the run that wrote them; re-run without --only after a scoring change.
|
Added one more commit: audit/overview --only previously narrowed the report, not just the work — a one-spider spot-check replaced the whole project-wide report. Now --only recomputes only the named spiders and merges every other spider's row back from the previous run's stored output (crawl_audit.csv, which is a lossless copy of the audit rows; a new _audit/overview_rows.json store for the overview, whose CSV flattens nested detail). The deleted-folder prune still works project-wide under --only, and the compliance summary covers all spiders while capture stays narrowed. If no previous report exists, it warns and falls back to the old narrowed output. Covered by tests/unit/test_only_merge.py. |
1 participant
What this adds
A project-wide quality layer over
data/<project>/crawl output — three CLIcommands, an HTML dashboard, three agent skills, docs, and tests. Built and
battle-tested against live production projects before this handover.
Commands
./scrapai audit --project <name>— read-only. Per-spider coverage +extraction scoring (
crawl_audit.csv/audit_<project>.md), compliancelenses (robots/llms/legal-page snapshots with conflict flagging),
external-PDF host analysis, and a self-contained HTML dashboard
(
_audit/dashboard_<project>.html, tabs: Coverage · Compliance · PDFs).Every spider is classified into a status worklist (
ok/manual review/too few pages/incomplete/extraction broken) — the goal, stated onthe report itself: every spider ends in
ok(or deliberatelydiscarded)../scrapai overview --project <name>— read-only. Per-spider contentprofile: sections, date span + per-year histogram + null %, field coverage,
thin/constant/off-domain checks.
./scrapai dedupe --project <name>— the ONE mutating command, keptseparate so the audit never rewrites data. Consolidates
crawls/*.jsonlreversibly (originals kept as
*.superseded).Design points reviewers will care about
links_onlyharvest —URL-only rows with
metadata_json.content_type="pdf". Extraction quality isjudged over HTML rows only; the PDFs lens splits same-org vs external by URL
host against
allowed_domains(finding an org's documents stored off itsown domain is the lens's whole purpose).
_audit/crawl_stats/when a crawl recorded them (no re-fetch, no drift, noCF problems); fetched sitemaps are the fallback.
_audit/audit_sitemap_skip.jsoncorrects theyardstick BEFORE scoring;
_audit/audit_notes.jsonrecords the verdictAFTER (promote/discard). Both are self-documenting (embedded, auto-refreshed
_instructions), both suggest-only for agents, and stale notes are policed(a note can never vouch for broken extraction or an empty corpus).
corpora re-audit in seconds; caches migrate themselves on format change.
dynamic value escaped; markdown/CSVs remain the untouched ground truth.
Also in this PR
.claude/commands/):/spider-review(evidence-based triage/repair of the audit's problem groups),
/spider-align(fleet-wide conformance sweep),
/spider-slow(live-crawl ops viacrawl-status/Pueue) — plus the
.gitignorepattern that tracks shippedskills while keeping teams' local ones untracked.
docs/quality.md(full reference),docs/skills-overview.md, CLAUDE.md§6.9 + README wiring, and
docs/requests/— the per-change design write-upsbehind this PR series (the earlier small PRs reference them).
Verification
~100 dedicated unit tests across corpus scanning + caching, scoring (including
PDF-row arithmetic and the reviewed-stale guards), compliance capture/report,
external-PDF partitioning, dashboard rendering, and an end-to-end synthetic
project flow. Full suite green. Restructures were gated by byte-identical
golden-master runs against real project output.