Skip to content

Fix quoted tokens and comment handling - #323

Draft
git-hulk wants to merge 1 commit into
codex/fix-named-type-token-errorsfrom
codex/fix-lexical-fidelity
Draft

Fix quoted tokens and comment handling#323
git-hulk wants to merge 1 commit into
codex/fix-named-type-token-errorsfrom
codex/fix-lexical-fidelity

Conversation

@git-hulk

@git-hulk git-hulk commented Sep 10, 2026

Copy link
Copy Markdown
Member

Problem

The lexer splits escaped quoted identifiers, silently turning "a""b"
into column "a" with alias "b". It classifies dollar-quoted strings
as identifiers, stops nested block comments at the first closing marker,
and rejects supported hash comments.

Reproduction

SELECT "a""b";
SELECT $$hello$$;
SELECT /* outer /* inner */ outer */ 1;
SELECT 1 # comment

The first two produce incorrect ASTs; the last two fail parsing.

Fix

Handle doubled delimiters and backslash escapes in identifiers, normalize
complete dollar strings into the existing escaped string representation,
and track block-comment depth. Recognize # and #! comments through
LF or EOF, matching ClickHouse's lexer.

Keep unmatched named dollar delimiters as identifiers, as ClickHouse does.
Escape literal line breaks so beautification preserves string values.
No exported types or fields are added.

Test

  • Lexer regressions cover token kind, escaped spelling, source positions,
    following separators, and formatting/reparsing. Quoted-identifier,
    dollar-string, and nested-comment cases failed before the lexer fix.
  • Extend invalid-input, error-position, and comment-termination tests.
    Add an AST and compact/beautified SQL fixture.
  • ClickHouse local 26.7.1.1315: 29 valid inputs produce matching ASTs for
    all 58 compact/beautified outputs; both engines reject nine invalid cases.
  • make test (race and compatibility tests), make lint, and make
    pass with Go 1.21.13. Tests/lint use external linking on macOS.

Quoted identifiers stop at escaped delimiters, dollar strings become
identifiers, and block comments end at the first nested closing marker.
Hash comments are also rejected.

Preserve identifier spelling, normalize complete dollar strings into the
existing escaped StringLiteral representation, and recognize nested block
and supported hash comments.

Add lexer, invalid-input, error-position, and golden regressions. Verify
both formatting modes against ClickHouse 26.7.1.1315 and pass make test,
make lint, and the CLI build.
@git-hulk
git-hulk force-pushed the codex/fix-lexical-fidelity branch from a692406 to 2199d5f Compare September 10, 2026 09:34
@git-hulk
git-hulk changed the base branch from master to codex/fix-named-type-token-errors September 10, 2026 09:34
@git-hulk
git-hulk added this pull request to stack #325 September 10, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant