20 — show: read production crawl output by default, DB fallback - #18
Open
MirjamOdile wants to merge 2 commits into
Open
20 — show: read production crawl output by default, DB fallback#18MirjamOdile wants to merge 2 commits into
MirjamOdile wants to merge 2 commits into
Conversation
show queried only the scraped_items DB table, which holds test-crawl items — production crawls write crawls/*.jsonl, never DB rows — so it systematically misrepresented what a spider collected (one spider: show said 272 stale test items while the production corpus held 1,600 rows) and disagreed with overview/audit by design. Default source is now the spider's crawl files when they exist (newest items first, same --limit/--url/--title/--text filters), with a stated fallback to the DB otherwise — so Phase-4 test verification (no crawl files yet) works unchanged. --source db|crawls|auto overrides. The JSONL reader is self-contained (no quality-tool import), keeping this independent of the quality-tool PR. docs/requests/20-show-production-output.md
iRanadheer
force-pushed
the
pr/20-show-production-output
branch
from
July 30, 2026 08:53
e911780 to
dfa5912
Compare
iRanadheer
force-pushed
the
pr/20-show-production-output
branch
from
August 10, 2026 12:23
67b76ad to
5119a74
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was broken:
show— the command for eyeballing what a spider collected — read only the database. But production crawls deliberately write files, never DB rows; the database holds only the little--limittest samples. So for any spider that had run for real,showdisplayed stale test data as if it were the collection (observed: 272 old test items shown while the real corpus held 1,600 rows), and it disagreed withoverview/auditby design. Reviewers learned to distrust it.What it does now:
showreads the spider's real crawl files whenever they exist (newest items first, same--limit/--url/--title/--textfilters), clearly states which source it's showing, and falls back to the database for fresh spiders — so the standard "test 5 items, then check" workflow works unchanged.--source db|crawls|autooverrides (e.g.--source dbto re-verify a test crawl after production files exist).Details
cli/show.py: self-contained JSONL reader (newest file first, lines bottom-up — crawl files are append-only), no quality-tool import, so this PR is independent of the quality-tool stack.docs/requests/20-show-production-output.md.Behavior changes
show's default source changes for any spider that has production crawl files: it now shows the real collection instead of stale DB test samples. To inspect fresh test items on such a spider (e.g. re-testing selectors during a repair), use--source db.show" workflow needs no flag.Verified
5 unit tests (
tests/unit/test_show_crawls.py): newest-first ordering across files, all three filters (case-insensitive, matching the DB branch's ilike), limit, garbage-line tolerance. The DB branch is untouched.