From a639ae78bfe0453c3eea03181ea364ce52b9a69b Mon Sep 17 00:00:00 2001 From: David Galloway Date: Wed, 19 Aug 2026 12:03:43 -0400 Subject: [PATCH 1/2] cephadm/tests: pre-import tracemalloc to avoid pyfakefs setup race Since 2026-08-18 run_tox_cephadm has been failing in roughly a quarter of arm64 make check runs (and ~8% of x86 runs) with RuntimeError: dictionary changed size during iteration pyfakefs/fake_filesystem_unittest.py:831 in Patcher._find_modules during `fs` fixture setup, followed by 80+ cascading AttributeError: 'NoneType' object has no attribute 'add_real_directory' because pyfakefs' Patcher is a ref-counted singleton and a failed setUp() never gets its tearDown(). There is no second thread involved. Several TestBootstrap tests run command_bootstrap() to completion, which leaves NamedTemporaryFile objects created on the fake filesystem to be garbage collected after the fake fs has been torn down. Their __del__ raises, which makes pytest's (>= 8.4) unraisable-exception hook run *inside the GC pass*, and the first time it does so it lazily imports tracemalloc (tracemalloc_message()). If that GC fires while pyfakefs 5.3.5 is doing `list(sys.modules.items())`, the iteration blows up. Newer pyfakefs (>= 5.10) snapshots sys.modules with dict.copy() and is immune, but 5.10.x is not a drop-in upgrade for this test suite, so import tracemalloc up front instead so that nothing new gets added to sys.modules from within a finalizer. Signed-off-by: David Galloway Assisted-by: Claude Fable 5 (claude-fable-5) (cherry picked from commit 79898177cb5f13da1631bb7e22f41248beeb54dc) (cherry picked from commit 44eff9c1937ea71ac3cd272b08c9dc16660aff10) --- src/cephadm/tests/conftest.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 src/cephadm/tests/conftest.py diff --git a/src/cephadm/tests/conftest.py b/src/cephadm/tests/conftest.py new file mode 100644 index 000000000000..7098e6c1bf0f --- /dev/null +++ b/src/cephadm/tests/conftest.py @@ -0,0 +1,13 @@ +# Pre-import tracemalloc so that pytest's unraisable-exception hook +# (_pytest.unraisableexception.unraisable_hook -> tracemalloc_message) does not +# lazily import it from inside a garbage-collection pass. +# +# Some tests (e.g. TestBootstrap) leave NamedTemporaryFile objects created on +# the pyfakefs fake filesystem to be garbage collected after the fake fs has +# been torn down; their __del__ raises, pytest's hook runs during GC and, the +# first time, imports tracemalloc. If that GC happens while pyfakefs' +# Patcher._find_modules() is iterating sys.modules, the `fs` fixture setup +# fails with "RuntimeError: dictionary changed size during iteration" and, +# because Patcher is a ref-counted singleton, every later fs-based test errors +# with "'NoneType' object has no attribute 'add_real_directory'". +import tracemalloc # noqa: F401 From 4a248ded0a074e6d58c954f6ab107164cb77fd2b Mon Sep 17 00:00:00 2001 From: Tatjana Dehler Date: Mon, 26 Oct 2026 12:59:38 +0100 Subject: [PATCH 2/2] release: record backport metadata Signed-off-by: Tatjana Dehler On-behalf-of: SAP Co-Authored-By: Claude Opus 5 (1M context) --- .../01a177f4-63f1-7e3d-a389-f72fcfbb888d.md | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 release-management/backports/01a177f4-63f1-7e3d-a389-f72fcfbb888d.md diff --git a/release-management/backports/01a177f4-63f1-7e3d-a389-f72fcfbb888d.md b/release-management/backports/01a177f4-63f1-7e3d-a389-f72fcfbb888d.md new file mode 100644 index 000000000000..704d6c9177e7 --- /dev/null +++ b/release-management/backports/01a177f4-63f1-7e3d-a389-f72fcfbb888d.md @@ -0,0 +1,69 @@ +--- +id: 01a177f4-63f1-7e3d-a389-f72fcfbb888d +title: 'tentacle: cephadm/tests: pre-import tracemalloc to avoid pyfakefs setup race' +component: build +category: fix +provenance: + type: upstream_pr + upstream_prs: + - https://github.com/ceph/ceph/pull/71189 + - https://github.com/ceph/ceph/pull/71163 + upstream_pr_state: merged-main + upstream_commits: + - 44eff9c1937ea71ac3cd272b08c9dc16660aff10 + - 79898177cb5f13da1631bb7e22f41248beeb54dc + upstream_trackers: + - https://tracker.ceph.com/issues/79677 +justification: + type: engineering + refs: + - cobaltcore-dev/cloud-storage#627 + - ceph/ceph#71189 + - ceph/ceph#71163 + - https://tracker.ceph.com/issues/79677 +risk: + # Stage-B — fill these during PR prep. See release-management/backports/RISK-RUBRIC.md. + blast: cosmetic + conflict: clean + coverage: weak +--- + +## Justification (public) + +Fixes an intermittent failure in the cephadm unit test suite that caused +`make check` to fail spuriously. No runtime behaviour changes. + +## Justification (internal) + +`run_tox_cephadm` fails on ~25% of arm64 and ~8% of x86 `make check` runs with +`RuntimeError: dictionary changed size during iteration` in pyfakefs +`Patcher._find_modules`, followed by ~80 cascading +`'NoneType' object has no attribute 'add_real_directory'` errors (the Patcher +is a ref-counted singleton, so a failed `setUp()` never gets its `tearDown()`). + +Root cause: pytest >= 8.4's unraisable-exception hook lazily imports +`tracemalloc` from inside a GC pass triggered by leaked `NamedTemporaryFile` +finalizers left on the fake filesystem; that mutates `sys.modules` while +pyfakefs 5.3.5 iterates it. pyfakefs >= 5.10 snapshots with `dict.copy()` and +is immune, but is not a drop-in upgrade here. Fix pre-imports `tracemalloc` in +conftest — one line, no production code touched. + +We carry the same pyfakefs 5.3.5 pin and unpinned pytest, so we are equally +exposed; without this, CI noise on the release branch is indistinguishable from +real regressions. + +Upstream state, checked 2026-10-26: #71163 is merged into `main` as +78193dc918711dee7ecb17f8f206659a3cbaa624. #71189 is its tentacle backport, +approved and labelled `releng-audit-pass`, but not yet merged. + +## Risk notes + +(Stage-B rationale: blast / conflict / coverage. Filled during PR prep.) + +## Behavior change + +(What changes for users? Empty if pure internal fix.) + +## Upgrade notes + +(Operator instructions, if any.)