Skip to content

Fix TOMLDecodeError position for an unfinished escape at EOF - #314

Open
fhgffy wants to merge 5 commits into
hukkin:masterfrom
fhgffy:fix-unfinished-escape-position
Open

fhgffy wants to merge 5 commits into
hukkin:masterfrom
fhgffy:fix-unfinished-escape-position

Conversation

@fhgffy

@fhgffy fhgffy commented Oct 4, 2026 •

Copy link
Copy Markdown

Fixes #312

An unfinished basic-string escape at the end of the document was reported one character past EOF. parse_basic_str_escape always advanced by two before noticing that no escape character remained, and that index was stored on TOMLDecodeError.

The error is now raised at the backslash itself, which is inside the document. The same path covers basic strings, multiline basic strings, and quoted keys.

The regression coverage also checks LF and CRLF documents containing a Unicode character before the unfinished escape, across basic strings, multiline basic strings, and quoted keys. It verifies the normalized document, character position, line, and column together. The test-only addition is @carlitogwapito's original commit, preserved with its authorship.

Validation on Windows, CPython 3.12.2, using the pure-Python source:

  • python -m unittest: 21 test methods, 20 passed and the existing Python 3.15+ lazy-import test skipped.
  • All six new LF/CRLF × string/key subcases fail against base 8479ed2902bafb7be7ac5f4431d42201ea640361 and pass at 5bd41ff6228a30bcbe28be49a1cd84970c41a9a0. The existing line/column control passes on both sources.
  • Black 26.1.0, isort 8.0.0, and flake8 core 7.3.0 checks pass for the changed test module. The full tox and pre-commit matrices were not run locally.

The parser advanced two characters before checking that an escape character was present, so a trailing backslash was reported one position past EOF.
@fhgffy
fhgffy force-pushed the fix-unfinished-escape-position branch from af6c2f9 to c70b6e8 Compare October 4, 2026 08:09
The unfinished-escape path no longer falls through to the unknown-escape branch, so that branch had no test and coverage dropped below 100%.
@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (8479ed2) to head (5bd41ff).
⚠️ Report is 13 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #314   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            4         4           
  Lines          535       537    +2     
  Branches       100       101    +1     
=========================================
+ Hits           535       537    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fhgffy and others added 3 commits October 4, 2026 18:55
Python 3.14 lists _types in the standard library, so flake8-builtins A005 fails on src/tomli/_types.py. That file is a private submodule and does not replace the stdlib module.
2026-10-05: resolve the configuration conflict without changing the escape-position fix.
@carlitogwapito

Copy link
Copy Markdown

I put together a small test-only commit on top of 14594c9, if it's useful here.

It adds six subcases: LF and CRLF inputs, each with a Unicode character in a basic string, multiline basic string, or quoted key. They check the normalized doc, backslash pos, line, and column together, following this PR's existing position choice.

On Python 3.14.4 with the checked-in pure Python source, all six passed on this PR's head. The same method on base 8479ed2 produced six expected position assertion failures: len(normalized) + 1 instead of len(normalized) - 1. The existing test_line_and_col control passed on both.

Only one test method is added; no parser changes. I haven't run the full suite, coverage, Black/pre-commit, type checks, compiled wheels, or the Python/platform matrix.

One small suggestion: keep these combined Unicode/newline cases with whichever position convention is ultimately chosen. They help catch accidental mixing of raw input offsets with normalized character coordinates.

@fhgffy

fhgffy commented Oct 17, 2026

Copy link
Copy Markdown
Author

Added your test-only commit 5bd41ff unchanged, preserving your authorship. I reran the six subcases with CPython 3.12.2: all six fail against base 8479ed2 and pass with the fix; the line/column control passes on both. The full native unittest run reports 21 methods, with the existing Python 3.15+ lazy-import test skipped. The PR description now includes these results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TOMLDecodeError reports an error position beyond EOF for an unfinished escape

2 participants