"python3 -m pegen python" generates a Python parser that contains invalid code

[alexprengere]

Bug report

Bug description:

I am currently experimenting with the cpython parser/tokenizer, following the parser documentation:

$ python3.14 -m pegen python  ../../Grammar/python.gram

This generates a parser.py file, that contains remnants of C code:

# line 34
return _PyAST_FunctionType ( a , b , p -> arena )

# end of file
void *
_PyPegen_parse(Parser *p)
{
    // Initialize keywords
    p->keywords = reserved_keywords;
    p->n_keyword_lists = n_keyword_lists;
    p->soft_keywords = soft_keywords;

    // Run parser

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-133560: Clarified parser.md doc for pegen parser issue in using the existing Grammar/python.gram file. #139194

[sobolevn]

This looks like a possible duplicate of #96424

[alexprengere]

OK I get it now, this is definitely a duplicate.
I think perhaps the instructions in parser.md are a bit misleading, as they suggest that when you are "adding some new rules to the grammar" (e.g. updating the python.gram), "it is a good idea to experiment first by generating a Python parser", when in fact you cannot use python -m pegen python on the modified python.gram you may be working on.
I will see if I can improve the docs once I get a better understanding on how to leverage pegen to generate a Python parser.

[lysnikolaou]

"it is a good idea to experiment first by generating a Python parser", when in fact you cannot use python -m pegen python on the modified python.gram you may be working on.

You can do so using the pure Python version of the grammar. I don't think it's up-to-date with the latest updates to the CPython grammar though. I need to find some time to update it.

I think we can close this as won't fix for now. If you wanna update the docs in the future, please open a new issue for that.

[sobolevn]

However, since this is not the first time people ask this - I propose adding a clear error message that this can't be done.