Skip to content

Commit aac7360

Browse files
authored
[mypyc] Fix @property getter memory leak (#21230)
## Issue Mypyc can incorrectly borrow references from property getters (owned references). The borrow mechanism is safe for struct field access since they return a pointer into the object's memory; Property getters on the other hand are method calls that return new owned references i.e the caller must `DECREF` them. If mypyc mistakes a property for a struct field, it skips the `DECREF`, leaking one reference per call. I think this affects any expression context that enables borrowing (comparisons, `isinstance`, `is None` checks) when the borrowed expression is a property access on a cross-module class. We discovered this through OOM issues in SQLGlot, where `isinstance(self.this, Func)` (`this` is a `@property`) leaked on every call. ## Repro ```Python3 # base.py: class Bar: pass class Foo: def __init__(self) -> None: self.obj: object = Bar() @Property def val(self) -> object: return self.obj # derived.py: from base import Foo, Bar def check(foo: Foo) -> bool: return isinstance(foo.val, Bar) # test_leak.py: import sys from base import Foo from derived import check foo = Foo() init = sys.getrefcount(foo.obj) for _ in range(100): check(foo) print(f"Leaked refs: {sys.getrefcount(foo.obj) - init}") ``` Compile both with mypyc passing in `PYTHONHASHSEED=3` prints `Leaked refs: 100`. ## Root cause 1. `is_native_attr_ref()` decides if an attribute access can safely borrow by checking `has_attr(name) and not get_method(name)` i.e "if there's an attribute but no method, it's a struct field". 2. Although `has_attr()` is always fully populated, `get_method()` depends on whether the class's module (`base` in this case) has been compiled yet. 3. Modules within an SCC are compiled in nondeterministic order; If the module defining the property hasn't been compiled yet -> `get_method()` returns `None` -> `is_native_attr_ref` incorrectly treats the property as a borrowed struct field. I think there's a broader issue here: Even when `derived.py` imports from `base.py` (no cycle), mypyc can place them in the same SCC and compile derived before base. Since `ir.methods` is only populated when a module's function bodies are compiled, any code that reads `ir.methods` (i.e `get_methods`) during compilation of a different module in the same SCC could have similar order-dependent bugs. ## The suggested fix Check `ir.attributes` directly (struct fields only, always populated) instead of the unreliable `get_method`. I believe this is the getter-side counterpart to #21095 which fixed the same class of bug for property setters.
1 parent fc410cb commit aac7360

2 files changed

Lines changed: 45 additions & 2 deletions

File tree

mypyc/irbuild/builder.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1553,8 +1553,7 @@ def is_native_attr_ref(self, expr: MemberExpr) -> bool:
15531553
return (
15541554
isinstance(obj_rtype, RInstance)
15551555
and obj_rtype.class_ir.is_ext_class
1556-
and obj_rtype.class_ir.has_attr(expr.name)
1557-
and not obj_rtype.class_ir.get_method(expr.name)
1556+
and any(expr.name in ir.attributes for ir in obj_rtype.class_ir.mro)
15581557
)
15591558

15601559
def mark_block_unreachable(self) -> None:

mypyc/test-data/run-classes.test

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5925,3 +5925,47 @@ assert NonExtDict.BASE == {"x": 1}
59255925
assert NonExtDict.EXTENDED == {"x": 1, "y": 2}
59265926

59275927
assert NonExtChained.Z == {10, 20, 30}
5928+
5929+
[case testPropertyGetterLeak]
5930+
class Bar:
5931+
pass
5932+
5933+
class Foo:
5934+
def __init__(self) -> None:
5935+
self.obj: object = Bar()
5936+
5937+
@property
5938+
def val(self) -> object:
5939+
return self.obj
5940+
5941+
[file other.py]
5942+
import gc
5943+
from native import Foo, Bar
5944+
5945+
def check(foo: Foo) -> bool:
5946+
return isinstance(foo.val, Bar)
5947+
5948+
def test_property_getter_no_leak() -> None:
5949+
gc.collect()
5950+
before = gc.get_objects()
5951+
for _ in range(100):
5952+
foo = Foo()
5953+
check(foo)
5954+
gc.collect()
5955+
after = gc.get_objects()
5956+
diff = len(after) - len(before)
5957+
assert diff <= 2, diff
5958+
5959+
test_property_getter_no_leak()
5960+
5961+
[file driver.py]
5962+
import sys
5963+
from other import check
5964+
from native import Foo
5965+
5966+
foo = Foo()
5967+
init = sys.getrefcount(foo.obj)
5968+
for _ in range(100):
5969+
check(foo)
5970+
after = sys.getrefcount(foo.obj)
5971+
assert after - init == 0, f"Leaked {after - init} refs"

0 commit comments

Comments
 (0)