Phase 12 — Data Curation Maturity (v0.5.2)

.claude/skills/review-pr/SKILL.md

@@ -17,9 +17,9 @@ Do **not** use for:
 - Making the changes yourself (that's a different task)
 - Deciding whether to merge (human maintainer call)
 
-## The six-question review
+## The seven-question review
 
-Go through these **in order**. If any answer is "no" or "unclear," block the PR.
+Go through these **in order**. If any answer is "no" or "unclear," block the PR. Question 7 fires only when a regex changes; the other six apply to every PR.
 
 ### 1. Does it match the architecture?
 
@@ -88,6 +88,24 @@ Check against [docs/standards/documentation.md](../../../docs/standards/document
 - [ ] No broken links (relative paths resolve)
 - [ ] Bilingual mirrors structurally aligned (H2 count + order match)
 
+### 7. Does any new / modified regex pass [regex.md](../../../docs/standards/regex.md)?
+
+**Trigger:** any change to `re.compile`, `re.match`, `re.sub`, `re.findall`, `re.split`. Phase 11/11.5/12 review cycles burned ~10 iterations on regex correctness — this skill exists to spend zero on the next PR.
+
+Run `git diff --unified=0 origin/main..HEAD -- '*.py' | grep -E '^\+.*re\.(compile|match|search|sub|findall|split|fullmatch)'` to surface the deltas, then for each:
+
+- [ ] No `[A-Za-z0-9_]` (Sonar `python:S6353`); use `\w`.
+- [ ] No single-char character classes `[ ]`, `[\.]`, `[\\]` (Sonar `python:S6328`); use the bare character.
+- [ ] Quantifiers bounded where the spec allows it (`{1,6}` for ATX heading depth, not `+`).
+- [ ] **No two unbounded `*` / `+` / `*?` / `+?` competing for the same character class.** This is the #1 ReDoS shape we keep hitting (`[ \t]+(.+?)[ \t]*$` → 100ms at n=2000). Anchor on `\S` at body boundaries: `[ \t]+(\S(?:[^\n]*\S)?)[ \t]*$`.
+- [ ] No `.*?` + back-reference + `re.DOTALL` (Sonar `python:S5852`); replace with a state machine — see [`forgelm/data_audit.py::_strip_code_fences`](../../../forgelm/data_audit.py) and [`forgelm/ingestion.py::_markdown_sections`](../../../forgelm/ingestion.py).
+- [ ] `\s` under `re.MULTILINE` → prefer `[ \t]` (no newline ambiguity).
+- [ ] Operator-controlled input → 10K-char pathological-input wall-clock benchmark stays ≤ 10ms.
+- [ ] Test fixtures with credential-shaped strings built from inert fragments (see `FAKE_AWS_KEY` / `FAKE_GH_TOKEN` in `tests/test_data_audit_phase12.py`).
+- [ ] PEM / PGP block markers inside regex source split via concatenation (`r"-----" + r"BEGIN " + r"..."`) so repo-wide secret scanners don't false-positive.
+
+If the regex runs on operator-controlled input, **paste the pathological-input benchmark output into the PR description** so reviewers see the linearity proof.
+
 ## Scope and hygiene
 
 Beyond the six questions:

.github/workflows/ci.yml

@@ -25,7 +25,6 @@ jobs:
 
       - name: Ruff format check
         run: ruff format --check .
-        continue-on-error: true  # warn but don't block
 
   # --- Job 2: Test (matrix across Python versions) ---
   test:
@@ -147,6 +146,33 @@ jobs:
           print('synthetic.py import: OK')
           "
 
+      - name: Bilingual doc H2 parity check
+        run: |
+          python -c "
+          import re, sys
+          pairs = [
+              ('docs/reference/configuration.md',       'docs/reference/configuration-tr.md'),
+              ('docs/reference/usage.md',               'docs/reference/usage-tr.md'),
+              ('docs/reference/distributed_training.md','docs/reference/distributed_training-tr.md'),
+              ('docs/reference/data_preparation.md',    'docs/reference/data_preparation-tr.md'),
+              ('docs/reference/architecture.md',        'docs/reference/architecture-tr.md'),
+              ('docs/guides/ingestion.md',              'docs/guides/ingestion-tr.md'),
+              ('docs/guides/data_audit.md',             'docs/guides/data_audit-tr.md'),
+          ]
+          failed = False
+          for en, tr in pairs:
+              with open(en) as f:
+                  en_count = sum(1 for line in f if re.match(r'^## ', line))
+              with open(tr) as f:
+                  tr_count = sum(1 for line in f if re.match(r'^## ', line))
+              status = 'OK' if en_count == tr_count else 'FAIL'
+              print(f'{status}: {en} ({en_count} H2) vs {tr} ({tr_count} H2)')
+              if en_count != tr_count:
+                  failed = True
+          if failed:
+              sys.exit(1)
+          "
+
       - name: License check
         run: |
           test -f LICENSE || (echo "LICENSE file missing" && exit 1)

CLAUDE.md

@@ -15,6 +15,7 @@ Not a framework for training from scratch. Not an inference engine. Not a GUI. R
 1. **[docs/standards/README.md](docs/standards/README.md)** — index of all engineering standards
 2. **The specific standard** matching what you're about to change:
    - Python code → [coding.md](docs/standards/coding.md) + [architecture.md](docs/standards/architecture.md)
+   - **Any `re.compile` / regex change → [regex.md](docs/standards/regex.md)** (ReDoS exposure, fixture fragmentation, the 8 hard rules distilled from Phase 11/11.5/12 review cycles)
    - Error paths → [error-handling.md](docs/standards/error-handling.md)
    - Anything with output → [logging-observability.md](docs/standards/logging-observability.md)
    - Tests → [testing.md](docs/standards/testing.md)

CONTRIBUTING.md

@@ -78,10 +78,10 @@ forgelm/
 ├── webhook.py       # Notifications
 └── utils.py         # Auth & checkpoints
 
-tests/               # 22+ test files, 297+ tests
-notebooks/           # 5 Colab notebooks
+tests/               # 47 test files
+notebooks/           # 10 Colab notebooks
 configs/deepspeed/   # ZeRO presets
-docs/guides/         # 6 user guides
+docs/guides/         # user guides
 ```
 
 ### Running Tests
@@ -122,7 +122,7 @@ Configuration is in `pyproject.toml` under `[tool.ruff]`.
 - **Keep it simple.** ForgeLM's strength is simplicity. Don't add complexity unless necessary.
 - **Config-driven.** New features should be configurable via YAML. No hardcoded behavior.
 - **Optional dependencies.** Heavy dependencies go in optional groups: `pip install forgelm[feature]`.
-- **Tests required.** Every new feature or bugfix needs a test. We have 297+ tests — keep it growing.
+- **Tests required.** Every new feature or bugfix needs a test. Keep coverage growing.
 - **Ruff clean.** CI will reject code that doesn't pass `ruff check`.
 - **No secrets.** Never commit tokens, API keys, or credentials. Use env vars.