Fix quoted tokens and comment handling - #323
Draft
git-hulk wants to merge 1 commit into
Draft
Conversation
Quoted identifiers stop at escaped delimiters, dollar strings become identifiers, and block comments end at the first nested closing marker. Hash comments are also rejected. Preserve identifier spelling, normalize complete dollar strings into the existing escaped StringLiteral representation, and recognize nested block and supported hash comments. Add lexer, invalid-input, error-position, and golden regressions. Verify both formatting modes against ClickHouse 26.7.1.1315 and pass make test, make lint, and the CLI build.
git-hulk
force-pushed
the
codex/fix-lexical-fidelity
branch
from
September 10, 2026 09:34
a692406 to
2199d5f
Compare
git-hulk
changed the base branch from
master
to
codex/fix-named-type-token-errors
September 10, 2026 09:34
git-hulk
added this pull request to stack #325
September 10, 2026 09:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The lexer splits escaped quoted identifiers, silently turning
"a""b"into column
"a"with alias"b". It classifies dollar-quoted stringsas identifiers, stops nested block comments at the first closing marker,
and rejects supported hash comments.
Reproduction
The first two produce incorrect ASTs; the last two fail parsing.
Fix
Handle doubled delimiters and backslash escapes in identifiers, normalize
complete dollar strings into the existing escaped string representation,
and track block-comment depth. Recognize
#and#!comments throughLF or EOF, matching ClickHouse's lexer.
Keep unmatched named dollar delimiters as identifiers, as ClickHouse does.
Escape literal line breaks so beautification preserves string values.
No exported types or fields are added.
Test
following separators, and formatting/reparsing. Quoted-identifier,
dollar-string, and nested-comment cases failed before the lexer fix.
Add an AST and compact/beautified SQL fixture.
all 58 compact/beautified outputs; both engines reject nine invalid cases.
make test(race and compatibility tests),make lint, andmakepass with Go 1.21.13. Tests/lint use external linking on macOS.