gc.get_referrers() can be used to see objects before they are fully built

[sigdevel]

Crash report

That the problem is occurring in the Py_INCREF macro, it looks like that the error occurs because the code is trying to increment the reference count of an object that doesnt exist.

Steps to reproduce the behavior:
1.create a sample file that contains:

import gc

def g():
    marker = object()
    yield marker
    [tup] = [x for x in gc.get_referrers(marker) if type(x) is tuple]
    print(tup)
    print(tup[1])

tuple(g())

2. just run without opt:

./target/python/Python-3.11.1/builded/bin/python3.11 ./target/python/founded/sig11_sync_Python3111_52.py

Error messages

stderr:

(<object object at 0x7f9e66d10160>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>)
Segmentation fault (core dumped)

gdb:

After run with synthetical sample, binary file aborted with Segfault:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  Py_INCREF (op=0x0) at ./Include/object.h:502

warning: Source file is more recent than executable.
502	#ifdef Py_REF_DEBUG

Expected behavior

No assertion failed.

Additional context

For solution this problem added additional checking to ensure that the op argument is not a null pointer before trying to increment its reference count.

    if (op == NULL) {
        return;
    }

(also added in a patch)

Environment

• Tested on: Python 3.11.1
• Operating system and architecture: Centos-based system (RED OS release MUROM (7.3.2)) ; 5.15.78-2.el7.3.x86_64

Linked PRs

• gh-101855: Added additional checker Py_INCREF macro in object.h #101856

[corona10]

cc @pablogsal

[eltoder]

The issue here is not with Py_INCREF, but with PySequence_Tuple used to build the tuple from the generator. PySequence_Tuple should untrack the tuple it is building until the build is complete, so the unfinished tuple is not discovered by GC (including gc.get_referrers).

[kumaraditya303]

This is a known issue as evident by https://github.com/python/cpython/blob/main/Lib/test/crashers/gc_inspection.py. I don't see any point is fixing it at all, get_referrers should be used for debugging not otherwise.

[sigdevel]

Thanks for the clarification, then I think the issue may can be close.

[markshannon]

Why is this closed? It is an interpreter crash on legal Python with a known cause. Let's fix it.

[sigdevel]

Seemed to require testing on the current version.
upd: @markshannon found your comment unanswered, yep this is reproduced in the same way in recent versions, I assume this can be fixed so that you don't get out of this situation by an unexpected crash but do it more safely

[picnixz]

The fix doesn't seem to be easy enough:

A fix would include a whole-scale code review, possibly with an API
change to decouple object creation and GC registration, and according
fixes to the documentation for extension module writers.  It's unlikely
to happen, though.  So this is currently classified as
"gc.get_referrers() is dangerous, use only for debugging".

So I think we should still close this one. Unless someone finds a way to fix it "easily" without breaking the rest of Python.