fix: recover every declared SQL routine from unparseable PL/pgSQL#2188
Open
Souptik96 wants to merge 1 commit into
Open
fix: recover every declared SQL routine from unparseable PL/pgSQL#2188Souptik96 wants to merge 1 commit into
Souptik96 wants to merge 1 commit into
Conversation
…aphify-Labs#2180) tree-sitter-sql cannot parse PL/pgSQL-only statements, and Graphify-Labs#1910's ERROR-node name recovery only covered one of the shapes that produces. Two others dropped the routine silently -- no node, no warning, exit code 0: 1. The statement is shredded into loose top-level tokens (keyword_create, keyword_function, object_reference, ..., keyword_begin) and the ERROR node holds only the offending body line, e.g. `PERFORM other_fn();` or `x := 1;`. No ERROR node contains any CREATE text, so scanning ERROR nodes finds nothing. This is what still dropped PERFORM and := after Graphify-Labs#1910. 2. The routine name is a quoted identifier -- CREATE OR REPLACE FUNCTION "public"."fn"(...) -- which the recovery's bare [\w$.]+ pattern cannot match, because it stops dead at the leading quote. Generated schema dumps quote every identifier, so whole files recovered nothing. Verified on the reported repro: the same body that drops under a quoted name is recovered fine under an unquoted one, which is why the drop looked like it depended only on the body statement. Fix mirrors the global REFERENCES fallback already in this extractor: after the tree walk, scan the raw source for every CREATE [OR REPLACE] FUNCTION/PROCEDURE and emit any routine the walk missed. Name parts accept bare or double-quoted identifiers. _add_node dedupes by node id, so routines already recovered from the tree are not emitted twice. Adds tests/fixtures/sample_plpgsql_quoted.sql -- generated-style quoted DDL whose bodies use RAISE, RAISE NOTICE, PERFORM, :=, IF..THEN and bare NULL; -- plus tests that every routine is recovered and that the file stays clean (tables before and after still extract, no duplicate ids or labels, no empty/ERROR labels, and every routine keeps its contains edge from the file node).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2180
Root cause: three parse shapes, only one was recovered
#1910 added ERROR-node name recovery, but PL/pgSQL breaks the parse in more than one
shape and the recovery only handled the first:
The whole
CREATElands in oneERRORnode — recovered before this PR.The statement is shredded into loose top-level tokens. For a body using
PERFORMor
:=, tree-sitter emitskeyword_create,keyword_or,keyword_replace,keyword_function,object_reference,function_arguments, …keyword_begin, and thenan
ERRORnode containing only the offending body line. Dumping the tree for thefixture in this PR:
No
ERRORnode contains anyCREATEtext, so scanning ERROR nodes finds nothing.This is why
PERFORMand:=still dropped after ERROR nodes on PostgreSQL/PLpgSQL functions #1910.The routine name is a quoted identifier. The recovery matched a bare
[\w$.]+,which stops dead at the leading quote, so
CREATE OR REPLACE FUNCTION "public"."fn"(...)matched nothing at all. Generated dumps quote every identifier, so entire files
recovered zero routines.
Confirmed shape 3 directly against
v8with your repro — the same PL/pgSQL body recoversunder an unquoted name and drops under a quoted one:
That's the missing piece in the issue's analysis: the body decides whether you hit the
error path at all, but the quoting and the shredding shape decide whether recovery can
see the routine. It also explains why the predicate wasn't clean at 186 functions —
unquoted or fully-parsed routines came through, quoted-and-shredded ones didn't.
Fix
Option 1 from the issue, implemented the way this extractor already handles the same class
of problem. There's an existing global regex fallback after the tree walk for
REFERENCESedges ("catch any REFERENCES missed due to ERROR nodes in the parse tree"); Imirrored it for routines: scan the raw source for every
CREATE [OR REPLACE] FUNCTION|PROCEDURE, with name parts accepting bare or double-quotedidentifiers, and emit any routine the tree walk missed.
_add_nodealready dedupes by nodeid, so routines recovered from the tree aren't emitted twice.
This is shape-agnostic — it fixes all three cases above, so a future grammar quirk can't
silently drop a declared routine again. After the fix your repro gives the expected 4 nodes
(2 file + 2 function).
I did not implement option 2 (the loud per-file "extracted N of M" warning). It's a good
idea, but a raw-text count can't tell a real declaration from
CREATE FUNCTIONinside acomment or a string literal, so it would need its own care to avoid warning on every clean
file. Happy to add it as a follow-up if you want it — and with this fix the count should now
be N == M anyway.
Testing
New fixture
tests/fixtures/sample_plpgsql_quoted.sql: generated-style DDL where everyidentifier is quoted and the bodies cover your matrix —
RAISE EXCEPTION,RAISE NOTICE,PERFORM,:=,IF..THEN, bareNULL;, plus aCREATE PROCEDURE— with tables before andafter.
Confirmed the tests catch the bug — same tests with
graphify/extractors/sql.pyreverted tov8:Full suite (this touches an extractor, so I ran everything):
Those 45 failures are pre-existing on Windows, not from this PR. My baseline on
unmodified
v8(66d8110) is 45 failed, 3578 passed, 15 skipped — the same 45, and passedmoves 3578 -> 3580, exactly the two tests added here. The pre-existing ones are Windows-specific
(
WinError 2/32, POSIX path-separator assertions,bash-dependent tests); none are intest_multilang.py. I have no Linux/macOS box to confirm they're green there.If you'd like, I'm happy to have the before/after count checked against your 186-function
schema — that's a better stress case than any fixture I can write.