Repository navigation
Conversation
This was referenced Nov 6, 2026
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 06e9d7e946
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
After an interrupt, parallel_map terminates and joins the workers, but the executor's manager thread may join them at the same time. Only one of the two gets the exit status from the OS; if it's the manager thread, the worker has exited and been reaped, but multiprocessing.active_children() still lists it until the manager thread has recorded its exit code (at most ~1 ms later). The test checked immediately, and failed in about 1 of 3 runs on Python 3.10. Allow a short delay (at most 5 s). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#566) Forked workers of a ProcessPoolExecutor inherit the registry of "wakeup" objects of all process pools in the parent (concurrent.futures.process. _threads_wakeups), and wake them up when they exit. In recent Python versions (e.g. 3.12.15, 3.13, 3.14) these objects have a lock; if another thread held such a lock while a worker was forked, the worker hung when exiting, and parallel_map hung in executor.shutdown(). This made test_overlapping_calls in tmulti_proc.py hang intermittently in CI. - Clear the inherited registry in the worker initializer (CPython does the same for thread pools in forked processes). - Add a regression test that triggers the race deterministically. - Add a 30-minute timeout to the CI job. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
**Bugs** The wavelet module incorrectly generated the `model_wavelets.fits` and `resid_wavelets.fits`. After Gaussian fitting, the residual image is further processed by wavelet Gaussian fitting. However, the implementation did not correctly account for the standard Gaussian model when saving the final outputs. - `model_wavelets.fits` contained only the additional model contributed by wavelet Gaussian fitting, rather than the complete source model. - `resid_wavelets.fits` contained the original image with only the wavelet model subtracted. So the standard Gaussian model was still present in this image, meaning it was not the actual residual. - This also caused incorrect results when wavelet Gaussian fitting was disabled (atrous_bdsm_do = False): the model image was 0 and the residual image was identical to the input image. **Fixed** - `model_wavelets.fits` contain the complete model (both standard and wavelet Gaussian components). - `resid_wavelets.fits` contain the final residual after subtraction of both model components. **Added** - `model_wavelets_component.fits` to preserve the wavelet-only model. Also updated the wavelet image filenames to distinguish wavelet components from remaining scale images when `atrous_sum = True`.
* Update reference outputs and make output comparison stricter - Regenerate `test/reference_outputs` with the current master. The old ones had drifted: `resid_gaus.fits` differed by 2.9% (close to the CI tolerance of 3%), and the wavelet model images by 12.7%. - `compare_results` now compares all subdirectories recursively (the images in `wavelet/model`, `wavelet/residual` and `wavelet/background`, and the top-level `ch0_*` images, were not compared), and fails if a file is missing in one of the runs (this was only logged). - `tbdsf_process_image.py` takes the tolerance as `--rtol`, with a default of 0.001 for runs on the same kind of system as the one that generated the reference outputs (x86-64 Linux). CI and cibuildwheel use 0.03. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Restore reference outputs from master --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Fix `inigaus_fbdsm` PA (#535) `fit_island()` converts the beam PA to the convention used by `gaus_2d()` by adding 90 deg. That beam is passed to `get_maxima()` and then to `mclean()`, which adds another 90 deg. The clean model is therefore rotated by 90 deg. * Fix wavelet output files (#548) **Bugs** The wavelet module incorrectly generated the `model_wavelets.fits` and `resid_wavelets.fits`. After Gaussian fitting, the residual image is further processed by wavelet Gaussian fitting. However, the implementation did not correctly account for the standard Gaussian model when saving the final outputs. - `model_wavelets.fits` contained only the additional model contributed by wavelet Gaussian fitting, rather than the complete source model. - `resid_wavelets.fits` contained the original image with only the wavelet model subtracted. So the standard Gaussian model was still present in this image, meaning it was not the actual residual. - This also caused incorrect results when wavelet Gaussian fitting was disabled (atrous_bdsm_do = False): the model image was 0 and the residual image was identical to the input image. **Fixed** - `model_wavelets.fits` contain the complete model (both standard and wavelet Gaussian components). - `resid_wavelets.fits` contain the final residual after subtraction of both model components. **Added** - `model_wavelets_component.fits` to preserve the wavelet-only model. Also updated the wavelet image filenames to distinguish wavelet components from remaining scale images when `atrous_sum = True`. * Update reference outputs and make output comparison stricter (#565) * Update reference outputs and make output comparison stricter - Regenerate `test/reference_outputs` with the current master. The old ones had drifted: `resid_gaus.fits` differed by 2.9% (close to the CI tolerance of 3%), and the wavelet model images by 12.7%. - `compare_results` now compares all subdirectories recursively (the images in `wavelet/model`, `wavelet/residual` and `wavelet/background`, and the top-level `ch0_*` images, were not compared), and fails if a file is missing in one of the runs (this was only logged). - `tbdsf_process_image.py` takes the tolerance as `--rtol`, with a default of 0.001 for runs on the same kind of system as the one that generated the reference outputs (x86-64 Linux). CI and cibuildwheel use 0.03. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Restore reference outputs from master --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Update reference outputs --------- Co-authored-by: Aleksander Kurek <aleksander.kurek@uj.edu.pl> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The multi-Gaussian function `MGFunction` and the fitter `lmder_fit` used by `gausfit.py` are now implemented in `bdsf/mgfunction.py`, using NumPy and MINPACK's LMDER through `scipy.optimize.leastsq`. - The function and its Jacobian are evaluated with exactly the same floating point operations as the C++ code (including the exponential in long double precision), because some fits are ill-conditioned and change with tiny rounding differences. - `lmder_fit` caches fit results per `MGFunction`: `fit_island` repeats most fits with identical input (75-84% of all fits in the benchmarks). - Remove the C++ sources, MINPACK and PORT3, and the Boost dependency (CMake, cibuildwheel, CI, README). - Add `test/tmgfunction.py` and benchmark scripts in `benchmark/replace-cbdsm`. Outputs of `test/tbdsf_process_image.py` are bit-identical to those of the C++ version; end-to-end runs are faster (see `benchmark/replace-cbdsm/DESCRIPTION.md`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `get_fitted_parameters`: use `flatten()`, which always returns a copy, instead of `ravel()` followed by an explicit `copy()`. - `_update`: drop `casting='unsafe'` from `N.exp`; the long double to float64 output cast is allowed by the default `same_kind` rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Write the fitted parameters through the cached flat view, like `residual` and `jacobian` already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`fit_island` stored its `MGFunction` in `isl.mg_fcn`, which is never used. With the pure-Python `MGFunction`, each instance keeps its evaluation caches (about 25 times the size of the island image), so with `ncores=1` memory grew with the total size of all fitted islands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When each forked worker imported `mgfunction` itself, it copied about 2.5 MiB of shared memory pages, and the test image needed 2-3 MiB more than with the C++ `_cbdsm`. Importing it in the parent, before the workers are forked, avoids that. Add memory measurements to the benchmark notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gmloose
force-pushed
the
replace_cbdsm
branch
from
November 9, 2026 16:25
1ea9bd3 to
af0bc75
Compare
- Drop the `MGFunction` methods that `gausfit.py` doesn't use (`remove_gaussian`, item access, the `parameters` setter, `errors`, `__len__`). - Always index the fitted parameters with an index array. - Write the evaluation of the Gaussians and the Jacobian as expressions instead of chains of in-place operations, with the same floating point operations in the same order. Replaying all 33097 fits of NGC6048 and 904 fits of the test image gives bit-identical results; the replay is 0.3% (NGC6048) and 4% (test image) faster (islands < 50 pixels: 4% faster, >= 1000 pixels: 0.9% slower), and peak memory is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `MGFunction._update` returns the residual (replaces `_residual`), and `_gaussian_scalars` returns only the numbers derived from the parameters. - `lmder_fit` handles new and cached fit results the same way. - `test/tmgfunction.py`: share the initial parameters of the tests, and shorten `test_repeated_fits`. - `record_fits.py` also works with `bdsf.mgfunction`, and `replay_fits.py` compares any two implementations (a Python file or a module such as `bdsf._cbdsm`), including peak memory of the largest fits. Replaying all 33097 fits of NGC6048 and 904 of the test image gives bit-identical results, with the same run time and peak memory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `replay_fits.py`: compare results with `repr()`, so that fits with identical NaN results count as identical, and count a NaN in only one of them as a difference. - `DESCRIPTION.md`: results on top of `master` 7ddb350 with the simplified `mgfunction`: all FITS outputs of both test runs are identical to those of C++, and the 8 non-identical fits of NGC6048 (2 distinct fits) diverge in both versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Written as expressions, the evaluation of the Gaussians and the Jacobian creates temporary arrays, which made fits of islands with >= 1000 pixels 1.7% slower than at af0bc75. Use in-place operations again, as before, and take the Jacobian rows of each Gaussian once (indexing creates a new view each time), which keeps small islands faster. Replaying all 33097 fits of NGC6048 and 904 of the test image gives bit-identical results; compared with af0bc75, fits of every island size are as fast or faster (NGC6048: -1.3% in total, >= 1000 pixels: -0.2%; test image: -3.8%), with the same peak memory. Update the benchmark notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the C++ extension
_cbdsm(Boost.Python, MINPACK, PORT3) with a pure-Python module,bdsf/mgfunction.py. It provides the part of theMGFunction,Gtypeandlmder_fitinterface thatgausfit.pyuses. The unused fittersdn2g_fitanddnsg_fit, and the unusedMGFunctionmethods for removing, getting and setting single Gaussians, were dropped. Ingausfit.pyonly the imports change;mgfunctionis imported at module level, so that the forked workers share it.How it works
lmder_fituses MINPACK's LMDER throughscipy.optimize.leastsq, with the same settings and analytical Jacobian as the C++ code.exp) made single fits about 2× cheaper than the final version, but it changed 2 of the 50 Gaussians of the test image (total flux by +67% and +10%), andtest/tbdsf_process_image.pyfailed. The Gaussians and their derivatives are therefore computed with exactly the same floating-point operations as the C++ code, including the exponential in long double precision.fit_islandrepeats most fits with identical input (74% of all fits for the test image, 84% for NGC6048).lmder_fitcaches results perMGFunction, which gives exactly the same results.Build
src/c++,src/minpackandsrc/port3.before_all.sh,before_build.sh, environment), the CI workflow and the README._pytesselate) and C (natgrid) extensions remain.Results
On a 20-core i7-12700 with NumPy 2.4.6 and SciPy 1.18.1, against a build of
master(7ddb350), 3 interleaved runs each:test/tbdsf_process_image.pybenchmark/run_pybdsf.py(NGC6048), totalOp_gausfitOp_wavelet_atrousmaster: 51 for the test image, 40 for NGC6048.gausfitrejects both. Earlier, on an oldermaster, such differences were traced to SciPy: since version 1.15, it uses a C translation of MINPACK, which in rare ill-conditioned fits rounds differently from the original Fortran; with the Fortran LMDER, built with f2py, those fits were bit-identical too.Memory (peak total PSS of the process tree, interleaved runs; measured on an older
master, 4fc79a2, before the later simplifications ofmgfunction, which don't change the peak memory of a fit):test/tbdsf_process_image.py(5 runs)Op_gausfitof the imagemgfunctionin each worker copied about 2.5 MiB of shared pages per worker (C++: 1.5 MiB for_cbdsm); a module-level import avoids that.Testing
New
test/tmgfunction.py, also added to CI and to the cibuildwheeltest-command. It covers:find_peakseterr(divide='raise')pip install .from a clean checkout builds without Boost;tmgfunction.py,tmulti_proc.pyandtbdsf_process_image.pypass.benchmark/replace-cbdsm/contains:record_fits.py: records all fits of a run, with_cbdsmormgfunctionreplay_fits.py: compares the results, speed and memory of single fits of two implementations (a Python file, such as an oldermgfunction.py, or a module such asbdsf._cbdsm)run_comparison.sh: an end-to-end comparison that checks every FITS output exactlyDetails are in its
DESCRIPTION.md.🤖 Generated with Claude Code