Skip to content

Grow the verified quote tier by use, not by rebuild: the decision that replaces the wikiquote-pd-v2 reselect #213

Description

@holden

Rewritten 2026-09-27 after the linking reassessment. The earlier body proposed rebuilding the public-domain quote corpus as wikiquote-pd-v2 by reselecting from every concept the registry now holds: over 3,000 requests to verify lines for pages nobody has opened. That is bulk prepopulation of a dictionary that is meant to grow through use, and it is withdrawn. What remains is a decision about how the verified tier grows at all. The earlier blockers (a corpus hard-wired to v1, a builder with no budget) are kept below as facts, because either answer to the decision needs one of them fixed.

What the corpus is, and what it is for

wikiquote-pd-v1 is 1,053 quotations over 197 concepts, each Verified against a pre-1931 Gutenberg text by build 5's checks (#171), committed as a manifest and seeded as source records with a quotation fingerprint as identity (ADR 0003). On the Quotes shelf its lines carry Verified; live Wikiquote lines carry Plausible. Its value is provenance pinned to a primary text, a stable set that survives a database reset (restored, not regenerated), and lines that are there when Wikiquote is not.

Live discovery is what serves a page: a visit admits one Wikiquote run for the page's concept (or the concept the hop reaches), the result is cached under its refresh and retention policy, and a rule change re-versions the recipe. The corpus is a verified tier on top of that, never a substitute for it.

Why not rebuild

The decision owed by the owner

How does the Verified tier grow?

  1. By use (recommended). When a page's live Wikiquote run returns lines, a bounded background job runs build 5's verification on those lines under a request budget (Gutenberg paced as today) and records the ones that pass as verified source records. Verified lines then show as Verified on the next read; a page visited by nobody stays Plausible-only. Nothing is reselected, and the set grows exactly as fast as the site is used.
  2. By rebuild. Keep the manifest model and rebuild periodically with a budget. Predictable, reproducible, and mostly spent on unvisited words.
  3. Both. v1 stays as the committed seed; new verified lines are added by use and periodically folded into a committed manifest so a reset restores them.

Under 1 or 3, a second decision: are the verified lines a committed manifest (reproducible after a reset, one more file to maintain) or database rows only (restored from a backup like routing and curation state, #205's rule)? Recommended: rows under a wikiquote-pd-live source, folded into a committed manifest at each release.

What is needed under either answer (kept from the earlier body)

  • The corpus module hard-codes the v1 slug and manifest path (Quotations.Corpus @slug, @manifest_path); the manifest kind names v1; the catalog registers only v1; the shelf reads only v1. A second source row, whichever way it fills, needs the corpus to take its version from the manifest or the source row.
  • The builder and the verifier pace requests but have no budget. Any of the three answers needs one, with a per-stage ledger, like every discovery provider has.

Acceptance for this issue

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions