Skip to content

fix: recover every declared SQL routine from unparseable PL/pgSQL#2188

Open
Souptik96 wants to merge 1 commit into
Graphify-Labs:v8from
Souptik96:fix/2180-sql-plpgsql-routine-recovery
Open

fix: recover every declared SQL routine from unparseable PL/pgSQL#2188
Souptik96 wants to merge 1 commit into
Graphify-Labs:v8from
Souptik96:fix/2180-sql-plpgsql-routine-recovery

Conversation

@Souptik96

Copy link
Copy Markdown

Fixes #2180

Root cause: three parse shapes, only one was recovered

#1910 added ERROR-node name recovery, but PL/pgSQL breaks the parse in more than one
shape and the recovery only handled the first:

  1. The whole CREATE lands in one ERROR node — recovered before this PR.

  2. The statement is shredded into loose top-level tokens. For a body using PERFORM
    or :=, tree-sitter emits keyword_create, keyword_or, keyword_replace,
    keyword_function, object_reference, function_arguments, … keyword_begin, and then
    an ERROR node containing only the offending body line. Dumping the tree for the
    fixture in this PR:

    keyword_function     line  27  FUNCTION
    object_reference     line  27  "public"."perform_fn"
    ...
    keyword_begin        line  28  BEGIN
    ERROR                line  29  PERFORM "public"."raise_notice_fn"();
    

    No ERROR node contains any CREATE text, so scanning ERROR nodes finds nothing.
    This is why PERFORM and := still dropped after ERROR nodes on PostgreSQL/PLpgSQL functions #1910.

  3. The routine name is a quoted identifier. The recovery matched a bare [\w$.]+,
    which stops dead at the leading quote, so CREATE OR REPLACE FUNCTION "public"."fn"(...)
    matched nothing at all. Generated dumps quote every identifier, so entire files
    recovered zero routines.

Confirmed shape 3 directly against v8 with your repro — the same PL/pgSQL body recovers
under an unquoted name and drops under a quoted one:

20260101000001_dropped.sql:   1 node(s) -> ['20260101000001_dropped.sql']                      # dropped
20260101000002_extracted.sql: 2 node(s) -> [..., '"public"."probe_with_return"()']
20260101000003_unquoted.sql:  2 node(s) -> [..., 'public.probe_unquoted()']                    # recovered!

That's the missing piece in the issue's analysis: the body decides whether you hit the
error path at all, but the quoting and the shredding shape decide whether recovery can
see the routine. It also explains why the predicate wasn't clean at 186 functions —
unquoted or fully-parsed routines came through, quoted-and-shredded ones didn't.

Fix

Option 1 from the issue, implemented the way this extractor already handles the same class
of problem. There's an existing global regex fallback after the tree walk for
REFERENCES edges ("catch any REFERENCES missed due to ERROR nodes in the parse tree"); I
mirrored it for routines: scan the raw source for every
CREATE [OR REPLACE] FUNCTION|PROCEDURE, with name parts accepting bare or double-quoted
identifiers, and emit any routine the tree walk missed. _add_node already dedupes by node
id, so routines recovered from the tree aren't emitted twice.

This is shape-agnostic — it fixes all three cases above, so a future grammar quirk can't
silently drop a declared routine again. After the fix your repro gives the expected 4 nodes
(2 file + 2 function).

I did not implement option 2 (the loud per-file "extracted N of M" warning). It's a good
idea, but a raw-text count can't tell a real declaration from CREATE FUNCTION inside a
comment or a string literal, so it would need its own care to avoid warning on every clean
file. Happy to add it as a follow-up if you want it — and with this fix the count should now
be N == M anyway.

Testing

New fixture tests/fixtures/sample_plpgsql_quoted.sql: generated-style DDL where every
identifier is quoted and the bodies cover your matrix — RAISE EXCEPTION, RAISE NOTICE,
PERFORM, :=, IF..THEN, bare NULL;, plus a CREATE PROCEDURE — with tables before and
after.

$ uv run pytest tests/test_multilang.py -q -k "quoted_plpgsql"
2 passed, 50 deselected, 1 warning in 0.38s

$ uv run pytest tests/test_multilang.py tests/test_languages.py tests/test_pg_introspect.py -q
391 passed, 1 warning in 2.08s

Confirmed the tests catch the bug — same tests with graphify/extractors/sql.py reverted to v8:

FAILED tests/test_multilang.py::test_sql_quoted_plpgsql_routines_are_recovered
  - AssertionError: perform_fn dropped from a quoted-DDL file (#2180)

Full suite (this touches an extractor, so I ran everything):

$ uv run pytest tests/ -q
45 failed, 3580 passed, 15 skipped, 4 warnings in 407.51s
$ uv run ruff check --config pyproject.toml graphify/extractors/sql.py tests/test_multilang.py
All checks passed!

$ uv run python -m tools.skillgen --check
check OK: 134 artifact(s) match committed output and expected/.

Those 45 failures are pre-existing on Windows, not from this PR. My baseline on
unmodified v8 (66d8110) is 45 failed, 3578 passed, 15 skipped — the same 45, and passed
moves 3578 -> 3580, exactly the two tests added here. The pre-existing ones are Windows-specific
(WinError 2/32, POSIX path-separator assertions, bash-dependent tests); none are in
test_multilang.py. I have no Linux/macOS box to confirm they're green there.

If you'd like, I'm happy to have the before/after count checked against your 186-function
schema — that's a better stress case than any fixture I can write.

…aphify-Labs#2180)

tree-sitter-sql cannot parse PL/pgSQL-only statements, and Graphify-Labs#1910's ERROR-node
name recovery only covered one of the shapes that produces. Two others dropped
the routine silently -- no node, no warning, exit code 0:

1. The statement is shredded into loose top-level tokens (keyword_create,
   keyword_function, object_reference, ..., keyword_begin) and the ERROR node
   holds only the offending body line, e.g. `PERFORM other_fn();` or `x := 1;`.
   No ERROR node contains any CREATE text, so scanning ERROR nodes finds
   nothing. This is what still dropped PERFORM and := after Graphify-Labs#1910.
2. The routine name is a quoted identifier -- CREATE OR REPLACE FUNCTION
   "public"."fn"(...) -- which the recovery's bare [\w$.]+ pattern cannot match,
   because it stops dead at the leading quote. Generated schema dumps quote
   every identifier, so whole files recovered nothing.

Verified on the reported repro: the same body that drops under a quoted name is
recovered fine under an unquoted one, which is why the drop looked like it
depended only on the body statement.

Fix mirrors the global REFERENCES fallback already in this extractor: after the
tree walk, scan the raw source for every CREATE [OR REPLACE] FUNCTION/PROCEDURE
and emit any routine the walk missed. Name parts accept bare or double-quoted
identifiers. _add_node dedupes by node id, so routines already recovered from
the tree are not emitted twice.

Adds tests/fixtures/sample_plpgsql_quoted.sql -- generated-style quoted DDL whose
bodies use RAISE, RAISE NOTICE, PERFORM, :=, IF..THEN and bare NULL; -- plus
tests that every routine is recovered and that the file stays clean (tables
before and after still extract, no duplicate ids or labels, no empty/ERROR
labels, and every routine keeps its contains edge from the file node).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQL extractor silently drops CREATE FUNCTION with PL/pgSQL-only bodies (PERFORM / := still dropped after #1910)

1 participant