Skip to content

Fix pretty() of constants so the output recompiles to the same model - #430

Open
binggao1230 wants to merge 1 commit into
neogeny:mainfrom
binggao1230:fix/constant-pretty-roundtrip
Open

binggao1230 wants to merge 1 commit into
neogeny:mainfrom
binggao1230:fix/constant-pretty-roundtrip

Conversation

@binggao1230

Copy link
Copy Markdown
Contributor

Constant._pretty() writes every literal bare between single backticks, which breaks the pretty round trip (compile(pretty(g)) == compile(g)) on current main (8f9d741) in three ways:

import tatsu
tatsu.compile(tatsu.compile('start = ```a`b``` $ ;').pretty())    # FailedParse
tatsu.compile(tatsu.compile('start = ```a\nb``` $ ;').pretty())   # FailedToken

and silently, because the '' =literal '' alternative of the constant rule is tried before the raw /(.*?)/ path:

g = tatsu.compile("start = `'42'` $ ;")   # Constant(literal='42')
tatsu.compile(g.pretty())                 # Constant(literal=42) -- now an int

Same for 'True'/'None'/'true'/'4.5', and ' x' loses its leading space to the word rule. Alert._pretty() delegates to Constant, so alerts inherit all of this.

The fix picks a source form, from the constant syntax the grammar already accepts, that reads back to the same value:

  • bare (output unchanged) whenever the value reads back as itself: plain words, spaced text, and all non-string literals;
  • repr() of the value when one quote kind suffices — escaped quotes are not an option because the lazy string patterns stop at the first quote character, so repr's delimiter choice is what makes this parse;
  • an '''...''' string, with quotes, backslashes, line breaks, tabs, and end whitespace escaped, when the value holds both quote kinds — multiline_string semantics apply trim(value.strip()).rstrip() before eval_escapes, so anything those touch has to be escaped.

Tests: a compile -> pretty -> compile harness asserting model equality (asjson) plus a pretty fixpoint, parametrized over 39 grammars covering the shapes above, and a check that already-stable literals keep their compact bare form. I ran the same round trip across the other node types (tokens, patterns, closures, joins, gathers, lookaheads, overrides, named, directives, comments): clean, except two pre-existing Pattern cases unrelated to this change (multiline patterns containing / fall back to ?"...", which cannot hold a raw newline, and multiline pattern text drifts to the trim() indentation fixpoint — the "weird indentation" note in grammar/tatsu.ebnf). Happy to file those separately.

uv run pytest tests: 641 passed, 6 skipped. ruff check --preview and ty check clean.

Constant._pretty() wrote every literal bare between single backticks.
Values holding a backtick or a line break produced grammars that fail
to recompile, and values that read back through the 'literal' rule were
silently reinterpreted ('42' -> 42, 'True' -> True, ' x' -> 'x').
Alert inherits the rendering.

Pick a source form that reads back to the same value: bare when safe,
repr() when a single quote kind suffices, and an escaped '''...'''
string when the value holds both kinds (multiline strings are trimmed
and stripped when read back, so line breaks, tabs, and end whitespace
are escaped as well).
@codspeed

codspeed Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Congrats! CodSpeed is installed 🎉

🆕 5 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


Open in CodSpeed

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant