Skip to content

Quality-filter user manual (EN/TR) opens with a stale heuristic table, a per-row quality_flags list and flag names that do not exist (CR-A11-051) #481

Open
Open
Quality-filter user manual (EN/TR) opens with a stale heuristic table, a per-row quality_flags list and flag names that do not exist (CR-A11-051)#481
Labels
area: data-auditCode area: data-auditbugSomething isn't workingdependenciesDependency or ecosystem currencydocumentationImprovements or additions to documentationgood first issueGood for newcomersseverity: mediumReal defect with limited reach or a workaroundsource: review-2026-10Found by the October 2026 code and documentation reviewurgency: next-releaseTechnology currency: fix in the next releasewave: 2Roadmap wave 2: high severity, larger effort; next-release tech
Milestone
Wave 2 — larger fixes and technology updates

Description

@cemililik
cemililik
opened on Oct 9, 2026

Wave 2 · theme D2-5: Data audit, masking and ingestion: medium effort · highest severity Medium · 4 finding(s) · effort small

Summary

An operator gates CI on by_check.bullet_only or reads per-row quality_flags as the page's first section shows. Neither key ever appears, so the gate silently never fires.

Primary finding

CR-A11-051 — Quality-filter user manual (EN/TR) opens with a stale heuristic table, a per-row quality_flags list and flag names that do not exist

severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. quality-filter.md:10-32 lists alpha<55%, word length 3-10, short-content, bullet-only and symbol-density checks and flags short_response/bullet_only/abnormal_word_length, while _QUALITY_CHECKS (_quality.py:22-28) has five different checks and the report holds only aggregate counts.

Where.

Evidence.

  • _QUALITY_CHECKS (_quality.py:22-28) = low_alpha_ratio, low_punct_endings, abnormal_mean_word_length, short_paragraphs, repeated_lines. rg for short_response|bullet_only|abnormal_word_length in forgelm/ finds no match. Real split keys (e2e_zh_simhash_report.json): columns, languages_top3, near_duplicate_pairs, null_or_empty_count, null_or_empty_rate, quality_flags_counts, quality_samples_evaluated, quality_samples_flagged, sample_count, simhash_distinct, text_length.
  • forgelm audit q.jsonl gave by_check keys low_alpha_ratio, low_punct_endings, short_paragraphs, abnormal_mean_word_length and repeated_lines. There was no per-row quality_flags field and none of short_response, bullet_only or abnormal_word_length.

Recommended fix. Delete the stale table, the per-row claim and the quick example (EN and TR). Let the 'Tuning thresholds' table be the single source, and add the language-scope caveat from the quality-filter finding above.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

TECH-T06-005 — Data-audit methods: Quality heuristics (Gopher / C4 / RefinedWeb style) (aging)

severity Next release

Item. Quality heuristics (Gopher / C4 / RefinedWeb style) · verdict: aging · urgency: Next release

State in ForgeLM. forgelm/data_audit/_quality.py:23-27 defines five checks, and :142-191 implements them: alpha characters below 0.70 of non-whitespace; under 50% of lines ending in punctuation; mean word length outside 3.0-12.0; over 50% short paragraphs; top-3 repeated lines over 30%. docs/guides/data_audit.md:379-382 states these values correctly. docs/usermanuals/en/data/quality-filter.md:14-19 and :28-31 (and the TR mirror) list '<55%' alpha, '>10' word length, 'short content', 'bullet-only' and 'symbol density' checks and output keys short_response/bullet_only that do not exist, contradicting :54-60 of the same page.

State of the ecosystem on 2026-10-01. The Gopher paper (Rae et al., arXiv 2112.11446, quality filtering for MassiveWeb) removes documents outside 50-100,000 words or with mean word length outside 3-10, with a symbol-to-word ratio above 0.1 for '#' or ellipsis, with over 90% of lines starting with a bullet or over 30% ending with an ellipsis, with fewer than 80% of words containing a letter, or without at least two of a list of stop words. DataTrove's GopherQuality, GopherRepetition, C4 and FineWeb filters encode these and further rules; FineWeb-era pipelines also add model-based classifiers.

Impact. ForgeLM's thresholds are its own adaptations, not Gopher's: a character-level alpha ratio of 0.70 against Gopher's word-level 0.8, word length 3-12 against 3-10, and a top-3 repeated-line share against a duplicate-line fraction. The flags are advisory, but the user manual documents checks and keys that do not exist.

Recommendation. Rewrite quality-filter.md (EN and TR) from _quality.py. State explicitly which thresholds differ from Gopher/C4 and cite arXiv 2112.11446. Optionally add Gopher's word-count, stop-word and bullet/ellipsis checks.

Sources.

DOC-D3B07-046 — docs/usermanuals/en/data/quality-filter.md: false claim — What gets flagged: <55% alpha, mean word length <3 or >10, short content, bullet-only rows, symbol density; each row gets a quality_flags l…

severity Medium

Where. docs/usermanuals/en/data/quality-filter.md:12-21 (Turkish mirror: docs/usermanuals/tr/data/quality-filter.md:12-21) · type: false claim

The document says. What gets flagged: <55% alpha, mean word length <3 or >10, short content, bullet-only rows, symbol density; each row gets a quality_flags list in the audit report

What the code does. There are exactly five checks: low_alpha_ratio (<0.70), low_punct_endings, abnormal_mean_word_length (3–12), short_paragraphs, repeated_lines. There are no short-content, bullet or symbol checks. The report holds only per-split quality_flags_counts and the aggregate quality_summary; there is no per-row list. The page's own later table contradicts this one.

Code references. forgelm/data_audit/_quality.py:141-230, forgelm/data_audit/_aggregator.py:226-245

User impact. Users tune against checks that do not exist and look for per-row flags.

Proposed correction. Replace the first table with the five real checks (same as the 'Tuning thresholds' table) and state that only per-split/aggregate counts are reported.

Turkish mirror. the Turkish page has the same error; fix both in the same PR

DOC-D3B07-047 — docs/usermanuals/en/data/quality-filter.md: broken example — ⚠ quality flags: short_response / repeated_lines / abnormal_word_length / bullet_only; ✓ wrote audit/... (quality_summary: 45 / 12,400 fla…

severity Medium

Where. docs/usermanuals/en/data/quality-filter.md:27-31,41 (Turkish mirror: docs/usermanuals/tr/data/quality-filter.md:27-31,41) · type: broken example

The document says. ⚠ quality flags: short_response / repeated_lines / abnormal_word_length / bullet_only; ✓ wrote audit/... (quality_summary: 45 / 12,400 flagged)

What the code does. The flag names are low_alpha_ratio, low_punct_endings, abnormal_mean_word_length, short_paragraphs, repeated_lines. The text summary prints quality flags : … (samples_flagged=N) per split and Quality : N sample(s) flagged (overall score = X).

Code references. forgelm/data_audit/_summary.py:398-400,454-457

User impact. jq gates on short_response / bullet_only never fire.

Proposed correction. Quality : 45 sample(s) flagged (overall score = 0.9964) — with per-split quality flags : low_punct_endings=…, short_paragraphs=…

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

  • A regression test reproduces the failure scenario and asserts the corrected behaviour.
  • Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
  • Every finding in this issue is resolved or explicitly closed with a reason.

Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.

Activity

  1. cemililik
    added this to the v0.13.0 — Wave 2: larger fixes and technology updates milestone on Oct 9, 2026
  2. cemililik
    added
    bugSomething isn't working
    documentationImprovements or additions to documentation
    area: data-auditCode area: data-audit
    dependenciesDependency or ecosystem currency
    severity: mediumReal defect with limited reach or a workaround
    source: review-2026-10Found by the October 2026 code and documentation review
    urgency: next-releaseTechnology currency: fix in the next release
    wave: 2Roadmap wave 2: high severity, larger effort; next-release tech
    on Oct 9, 2026
  3. cemililik
    added this to ForgeLM Project Planon Oct 9, 2026
  4. cemililik
    added a parent issue on Oct 9, 2026
  5. cemililik
    added
    good first issueGood for newcomers
    on Oct 11, 2026

  6. cemililik commented on Oct 11, 2026

    @cemililik
    cemililik
    on Oct 11, 2026
    CollaboratorAuthor

    Where to start

    Thanks for taking a look. Please confirm the issue is unassigned and comment before starting so work does not overlap.

    Start with these files on development:

    Remove the stale opening threshold table, per-row quality_flags claim and invented output keys. Reuse the existing correct threshold section, describe aggregate counts and language limitations, and distinguish ForgeLM adaptations from upstream filters. Resolve all four finding IDs through this focused EN/TR rewrite; do not add heuristics.

    Run the focused tests:

    pytest tests/test_data_audit_phase12.py --no-cov

    Run these guards/checks as applicable to the changed files:

    ruff format --check forgelm/data_audit/_quality.py tests/test_data_audit_phase12.py
ruff check forgelm/data_audit/_quality.py tests/test_data_audit_phase12.py
python3 tools/check_bilingual_parity.py --strict
python3 tools/check_bilingual_code_blocks.py --strict
python3 tools/check_anchor_resolution.py --strict
python3 tools/check_tr_links_prefer_mirror.py --strict
python3 tools/check_usermanual_self_contained.py --strict
python3 tools/check_usermanual_schema_drift.py --strict

    If user-facing documentation changes, update the existing Turkish mirror in the same PR; keep commands, code and terminal output in English and structurally aligned. English-only contributor standards need no new mirror.

    Open the PR against development with Fixes #481 once every finding is resolved or explicitly closed with a reason; use Refs #481 for partial work and list the resolved finding IDs.

  7. cemililik commented on Oct 17, 2026

    @cemililik
    cemililik
    on Oct 17, 2026
    CollaboratorAuthor

    Update — re-verified on development (0766e94). Neither page has changed.

    Suggested scope, in both the English and the Turkish page: delete the opening threshold
    table, the sentence about a per-row quality_flags list and the first example. The
    "Tuning thresholds" table further down then becomes the single source. The only new prose
    is two short items:

    • a language caveat: the checks assume space-delimited words and ASCII end punctuation
      (a Chinese sample reports low_punct_endings because _PUNCT_END_PATTERN has no
      full-width punctuation);
    • one sentence saying the thresholds are ForgeLM's own (alpha ratio 0.70 over characters,
      mean word length 3–12), not the published Gopher ones.

    Replace the example with real forgelm audit output (CPU-only). If you do not write
    Turkish, do the English page and say so in the PR: a maintainer completes the Turkish
    mirror.

  8. cemililik commented on Oct 22, 2026

    @cemililik
    cemililik
    on Oct 22, 2026
    CollaboratorAuthor

    Update — re-verified on development (735a02f). Both pages are still unchanged, and the line numbers are the same in English and Turkish:

    • the stale table and the quality_flags sentence, lines 10-21;
    • the first example, lines 23-32;
    • the second example, lines 38-48.

    One gap in the scope above: the second code block also has an invented line, ✓ wrote audit/data_audit_report.json (quality_summary: 45 / 12,400 flagged) (line 41); forgelm audit prints no such line. The structure is decided as follows, in both languages:

    • Delete the whole ## What gets flagged section: heading, table and the quality_flags sentence.

    • Keep ## Quick example and delete only its first code block. In the second block, replace the output with what forgelm audit really prints (forgelm/data_audit/_summary.py, the quality flags and Quality lines). For example:

      • quality flags : low_punct_endings=4, short_paragraphs=3 (samples_flagged=4)
      • Quality : 4 sample(s) flagged (overall score = 0.3333)

      Run it on a small JSONL of your own and paste your real output.

    • Put the language caveat and the "thresholds are ForgeLM's own" sentence right after the "Tuning thresholds" table. A Chinese sample is flagged with both low_punct_endings and short_paragraphs, so mention both.

    Do not add heuristics. If you do not write Turkish, do the English page and say so in the PR. The Turkish edit is the same deletions plus those two short sentences, and a maintainer completes it. To claim the issue, start a comment with /assign.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: data-auditCode area: data-auditbugSomething isn't workingdependenciesDependency or ecosystem currencydocumentationImprovements or additions to documentationgood first issueGood for newcomersseverity: mediumReal defect with limited reach or a workaroundsource: review-2026-10Found by the October 2026 code and documentation reviewurgency: next-releaseTechnology currency: fix in the next releasewave: 2Roadmap wave 2: high severity, larger effort; next-release tech

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions