Skip to content

feat(core,ooxml,cli): tatweel-padding rule — M4 4.4 - #35

Merged
aenawi merged 1 commit into
mainfrom
tatweel-padding
Sep 5, 2026
Merged

aenawi merged 1 commit into
mainfrom
tatweel-padding

Conversation

@aenawi

@aenawi aenawi commented Sep 5, 2026

Copy link
Copy Markdown
Owner

The last item in M4. PLAN §4.4.

The hard part is that tatweel is legitimate

U+0640 is the kashida — the elongation Arabic typography has justified text with for a thousand years — and a font inserts it during layout, which is why it never reaches the stored string. A justified paragraph has none of it in the file. Every tatweel this tool can see was typed by somebody, and only some of those are a defect. A rule that concluded one from the character alone would be ADR 0004's first failure mode for the third time.

So crate::tatweel groups them into maximal runs and asks what each run is joined to.

Reported

  • Two or more in a row, joined to a letter — a word stretched to a width. No typography needs a second tatweel, because one is already enough to carry a mark or show a form, so the repetition is the evidence and two is where the line falls.
  • One wedged between two characters that already join — width and nothing else, because both neighbours take exactly the form they had without it. The wedge that fakes an alignment inside a word.

Silent, and each one is a real document

  • Carrying a harakat — the base a mark is written on, which is how a lone fatha appears in a table of harakat or a keyboard legend. Delete it and the mark lands on whatever precedes.
  • At a joining edge — how a letter's initial, medial or final form is written on its own, in a primer or a dictionary. The tatweel is the thing being shown.
  • Joined to nothing on either side — a rule or a separator, drawn with the character that draws rules. Reporting it would invent a stretched word where there is no word.

All three are committed as cases that a rule regressing to "any tatweel is a defect" fails.

A warning, not an error

ADR 0004's severity table read honestly: the text renders exactly as the author arranged it. What is wrong is the string behind it, and every cost is one a screenshot cannot show — a search for the heading no longer matches the heading, a spell-checker does not know the word, a screen reader reads the padding aloud, and the width it was measured against is gone the moment the box, the font or the point size changes.

evidence.offenders gives each run as U+0640 ARABIC TATWEEL ×5 @14 — the length and a byte offset into evidence.logical.

repair --strip-tatweel

The second repair ever to edit the characters an author typed rather than the properties around them, after literal-bullet, and opt-in for that rule's reason and more so.

Whether stripping is safe turned out to be the same question as the threshold, and it is asserted against joining::forms rather than restated: deleting a run a join crosses changes no letter's form, however long the run is. A stretched run at a word edge is not form-preserving — a noon padded to a width reverts to a plain noon — and that is the repair rather than a loss, because a plain noon is what the unpadded word says. The Carrier and Displayed cases are safe by never being touched.

One thing the adapter learned

RemoveControls and RemoveTatweel both carry byte offsets into the text as scanned. A paragraph with a bidi control in a padded heading would have had the second set applied to a string the first had already shortened, and the second deletion would have landed on a letter. rewrite.rs now deletes from one merged, descending sequence, and the case is committed in both fix orders — the planner's emission order is not a contract the rewriter may lean on.

Acceptance

Met in the conformance suite, so it holds for .pptx and .docx alike:

  • a heading padded with five typed tatweel comes back a tatweel-padding warning naming the run's offset and length in both formats
  • the same paragraph justified — where the kashida is the font's and the string holds none of it — comes back with nothing
  • so do a lone fatha on a tatweel, medial heh as a primer shows it, and a rule drawn with four of them

Through the writer, a padded heading in a run of its own is repaired to the word it was and re-audits clean; a tatweel carrying a fatha is neither reported nor planned against.

Corpus

The five repairable reports gained one options key each (strip_tatweel) and zero findings changed — the claim that nothing else moved. --strip-tatweel joins the golden repair, so every opt-in fix is exercised.

make verify is green: fmt, clippy -D warnings, all tests, docs -D warnings, release build.

Bookkeeping

M4 is now [x]. ROADMAP.md's M4 bullets are rewritten as landed items pointing at their plan sections.

U+0640 is legitimate — it is the kashida, and a font inserts it during
layout, which is why it never reaches the stored string. So the rule
judges what surrounds each run of typed tatweel rather than the
character, and `crate::tatweel` holds the argument.

Reported: two or more in a row joined to a letter (a word stretched to a
width — no typography needs a second tatweel, so the repetition is the
evidence), and one wedged between two characters that already join
(width, and no change to any letter's form).

Silent: one carrying a harakat, which is the base that mark is written
on; one at a joining edge, which is how a letter's contextual form is
written alone; and a run joined to nothing, which is a rule somebody
drew. All three are committed as cases a rule regressing to "any tatweel
is a defect" fails.

A warning, not an error: the text renders as arranged, and what is wrong
is the string behind it.

`repair --strip-tatweel` deletes exactly the runs the finding listed.
Opt-in, for `--convert-bullets`' reason and more so — it deletes
characters somebody typed. Safety is the same question as the threshold,
and is asserted against `joining::forms`: deleting a run a join crosses
changes no letter's form.

The adapter now deletes `RemoveControls` and `RemoveTatweel` from one
merged, descending sequence — both carry offsets into the text as
scanned, so applying them in turn would delete the second set from a
string the first had already shortened.
@aenawi
aenawi merged commit a6c9bd2 into main Sep 5, 2026
7 checks passed
@aenawi
aenawi deleted the tatweel-padding branch September 5, 2026 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant