Skip to content

Replace C++ extension _cbdsm with a Python implementation - #564

Open
gmloose wants to merge 17 commits into
pure-python-implementationfrom
replace_cbdsm
Open

gmloose wants to merge 17 commits into
pure-python-implementationfrom
replace_cbdsm

Conversation

@gmloose

@gmloose gmloose commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Replaces the C++ extension _cbdsm (Boost.Python, MINPACK, PORT3) with a pure-Python module, bdsf/mgfunction.py. It provides the part of the MGFunction, Gtype and lmder_fit interface that gausfit.py uses. The unused fitters dn2g_fit and dnsg_fit, and the unused MGFunction methods for removing, getting and setting single Gaussians, were dropped. In gausfit.py only the imports change; mgfunction is imported at module level, so that the forked workers share it.

How it works

  • Fitting: lmder_fit uses MINPACK's LMDER through scipy.optimize.leastsq, with the same settings and analytical Jacobian as the C++ code.
  • Exact arithmetic: some island fits are ill-conditioned: a last-bit rounding difference can change a fitted Gaussian by tens of percent. A first NumPy version (one matrix multiplication for all Gaussians, float64 exp) made single fits about 2× cheaper than the final version, but it changed 2 of the 50 Gaussians of the test image (total flux by +67% and +10%), and test/tbdsf_process_image.py failed. The Gaussians and their derivatives are therefore computed with exactly the same floating-point operations as the C++ code, including the exponential in long double precision.
  • Speed: the Gaussians are evaluated one by one, in place where possible (temporary arrays slow down fits of large islands). A single fit of a small island is still slower than in C++ (NumPy overhead per operation), but fit_island repeats most fits with identical input (74% of all fits for the test image, 84% for NGC6048). lmder_fit caches results per MGFunction, which gives exactly the same results.
  • Errors: diverging fits give inf/nan as before, independent of global NumPy error settings.

Build

  • Removes src/c++, src/minpack and src/port3.
  • Removes Boost from CMake, cibuildwheel (before_all.sh, before_build.sh, environment), the CI workflow and the README.
  • The Fortran (_pytesselate) and C (natgrid) extensions remain.

Results

On a 20-core i7-12700 with NumPy 2.4.6 and SciPy 1.18.1, against a build of master (7ddb350), 3 interleaved runs each:

C++ Python
test/tbdsf_process_image.py 6.62–6.67 s 6.22–6.26 s
benchmark/run_pybdsf.py (NGC6048), total 63.8–64.1 s 54.2–54.8 s
… of which Op_gausfit 12.6–12.9 s 7.9–8.0 s
… of which Op_wavelet_atrous 40.1–40.2 s 35.5–35.8 s
  • Outputs: all FITS outputs, catalogs and images, are bit-identical to master: 51 for the test image, 40 for NGC6048.
  • Single fits (replaying all recorded fits, without the result cache): 904 of 904 fits of the test image and 33089 of 33097 of NGC6048 are bit-identical. The other 8 are 2 distinct fits (one repeated 7 times), which diverge in both versions: C++ ends with NaN parameters, Python with huge ones. Neither converges, so gausfit rejects both. Earlier, on an older master, such differences were traced to SciPy: since version 1.15, it uses a C translation of MINPACK, which in rare ill-conditioned fits rounds differently from the original Fortran; with the Fortran LMDER, built with f2py, those fits were bit-identical too.
  • A single fit takes 1.18× as long as in C++ in total for NGC6048: 7.7× for islands with < 50 pixels, 0.90× for islands with ≥ 1000 pixels.

Memory (peak total PSS of the process tree, interleaved runs; measured on an older master, 4fc79a2, before the later simplifications of mgfunction, which don't change the peak memory of a fit):

C++ Python
test/tbdsf_process_image.py (5 runs) 136–138 MiB 136–137 MiB
NGC6048 (6 runs) mean 3759 MiB mean 3759 MiB
… peak during Op_gausfit of the image 1404 MiB 1342 MiB
  • The peak of NGC6048 is in the wavelet stage, in the main process, which doesn't use the fitting code; it varies by ±50 MiB between runs of either version.
  • Importing mgfunction in each worker copied about 2.5 MiB of shared pages per worker (C++: 1.5 MiB for _cbdsm); a module-level import avoids that.
  • A fit of the largest NGC6048 island temporarily needs a few MB more in Python, because SciPy copies the Jacobian; Python frees it after each island, while C++ keeps its caches until the worker ends.

Testing

  • New test/tmgfunction.py, also added to CI and to the cibuildwheel test-command. It covers:

    • the residual and the Jacobian (against finite differences) for mixed Gaussian types
    • fitting, and fixed parameters
    • cached fits
    • find_peak
    • the Python interface and invalid input
    • degenerate fits: no exceptions or warnings, also with seterr(divide='raise')
  • pip install . from a clean checkout builds without Boost; tmgfunction.py, tmulti_proc.py and tbdsf_process_image.py pass.

  • benchmark/replace-cbdsm/ contains:

    • record_fits.py: records all fits of a run, with _cbdsm or mgfunction
    • replay_fits.py: compares the results, speed and memory of single fits of two implementations (a Python file, such as an older mgfunction.py, or a module such as bdsf._cbdsm)
    • run_comparison.sh: an end-to-end comparison that checks every FITS output exactly

    Details are in its DESCRIPTION.md.

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-08T14:02:59.548015Z 902bc59 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 06e9d7e946

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bdsf/mgfunction.py Resolved
@gmloose gmloose self-assigned this Nov 8, 2026
@gmloose
gmloose requested a review from aroffringa November 8, 2026 14:05
@gmloose
gmloose changed the base branch from master to pure-python-implementation November 8, 2026 18:36
gmloose and others added 13 commits November 9, 2026 11:48
After an interrupt, parallel_map terminates and joins the workers, but the
executor's manager thread may join them at the same time. Only one of the
two gets the exit status from the OS; if it's the manager thread, the
worker has exited and been reaped, but multiprocessing.active_children()
still lists it until the manager thread has recorded its exit code (at
most ~1 ms later). The test checked immediately, and failed in about 1 of
3 runs on Python 3.10. Allow a short delay (at most 5 s).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#566)

Forked workers of a ProcessPoolExecutor inherit the registry of "wakeup"
objects of all process pools in the parent (concurrent.futures.process.
_threads_wakeups), and wake them up when they exit. In recent Python
versions (e.g. 3.12.15, 3.13, 3.14) these objects have a lock; if another
thread held such a lock while a worker was forked, the worker hung when
exiting, and parallel_map hung in executor.shutdown(). This made
test_overlapping_calls in tmulti_proc.py hang intermittently in CI.

- Clear the inherited registry in the worker initializer (CPython does the
  same for thread pools in forked processes).
- Add a regression test that triggers the race deterministically.
- Add a 30-minute timeout to the CI job.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`fit_island()` converts the beam PA to the convention used by `gaus_2d()` by adding 90 deg. That beam is passed to `get_maxima()` and then to `mclean()`, which adds another 90 deg. The clean model is therefore rotated by 90 deg.
**Bugs**
The wavelet module incorrectly generated the `model_wavelets.fits` and `resid_wavelets.fits`. After Gaussian fitting, the residual image is further processed by wavelet Gaussian fitting. However, the implementation did not correctly account for the standard Gaussian model when saving the final outputs.

- `model_wavelets.fits` contained only the additional model contributed by wavelet Gaussian fitting, rather than the complete source model.

- `resid_wavelets.fits` contained the original image with only the wavelet model subtracted. So the standard Gaussian model was still present in this image, meaning it was not the actual residual.

- This also caused incorrect results when wavelet Gaussian fitting was disabled (atrous_bdsm_do = False): the model image was 0 and the residual image was identical to the input image.

**Fixed**
- `model_wavelets.fits` contain the complete model (both standard and wavelet Gaussian components).

- `resid_wavelets.fits` contain the final residual after subtraction of both model components.

**Added**
- `model_wavelets_component.fits` to preserve the wavelet-only model.

Also updated the wavelet image filenames to distinguish wavelet components from remaining scale images when `atrous_sum = True`.
* Update reference outputs and make output comparison stricter

- Regenerate `test/reference_outputs` with the current master. The old
  ones had drifted: `resid_gaus.fits` differed by 2.9% (close to the CI
  tolerance of 3%), and the wavelet model images by 12.7%.
- `compare_results` now compares all subdirectories recursively (the
  images in `wavelet/model`, `wavelet/residual` and `wavelet/background`,
  and the top-level `ch0_*` images, were not compared), and fails if a file
  is missing in one of the runs (this was only logged).
- `tbdsf_process_image.py` takes the tolerance as `--rtol`, with a default
  of 0.001 for runs on the same kind of system as the one that generated
  the reference outputs (x86-64 Linux). CI and cibuildwheel use 0.03.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Restore reference outputs from master

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Fix `inigaus_fbdsm` PA (#535)

`fit_island()` converts the beam PA to the convention used by `gaus_2d()` by adding 90 deg. That beam is passed to `get_maxima()` and then to `mclean()`, which adds another 90 deg. The clean model is therefore rotated by 90 deg.

* Fix wavelet output files (#548)

**Bugs**
The wavelet module incorrectly generated the `model_wavelets.fits` and `resid_wavelets.fits`. After Gaussian fitting, the residual image is further processed by wavelet Gaussian fitting. However, the implementation did not correctly account for the standard Gaussian model when saving the final outputs.

- `model_wavelets.fits` contained only the additional model contributed by wavelet Gaussian fitting, rather than the complete source model.

- `resid_wavelets.fits` contained the original image with only the wavelet model subtracted. So the standard Gaussian model was still present in this image, meaning it was not the actual residual.

- This also caused incorrect results when wavelet Gaussian fitting was disabled (atrous_bdsm_do = False): the model image was 0 and the residual image was identical to the input image.

**Fixed**
- `model_wavelets.fits` contain the complete model (both standard and wavelet Gaussian components).

- `resid_wavelets.fits` contain the final residual after subtraction of both model components.

**Added**
- `model_wavelets_component.fits` to preserve the wavelet-only model.

Also updated the wavelet image filenames to distinguish wavelet components from remaining scale images when `atrous_sum = True`.

* Update reference outputs and make output comparison stricter (#565)

* Update reference outputs and make output comparison stricter

- Regenerate `test/reference_outputs` with the current master. The old
  ones had drifted: `resid_gaus.fits` differed by 2.9% (close to the CI
  tolerance of 3%), and the wavelet model images by 12.7%.
- `compare_results` now compares all subdirectories recursively (the
  images in `wavelet/model`, `wavelet/residual` and `wavelet/background`,
  and the top-level `ch0_*` images, were not compared), and fails if a file
  is missing in one of the runs (this was only logged).
- `tbdsf_process_image.py` takes the tolerance as `--rtol`, with a default
  of 0.001 for runs on the same kind of system as the one that generated
  the reference outputs (x86-64 Linux). CI and cibuildwheel use 0.03.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Restore reference outputs from master

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* Update reference outputs

---------

Co-authored-by: Aleksander Kurek <aleksander.kurek@uj.edu.pl>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The multi-Gaussian function `MGFunction` and the fitter `lmder_fit` used
by `gausfit.py` are now implemented in `bdsf/mgfunction.py`, using NumPy
and MINPACK's LMDER through `scipy.optimize.leastsq`.

- The function and its Jacobian are evaluated with exactly the same
  floating point operations as the C++ code (including the exponential in
  long double precision), because some fits are ill-conditioned and change
  with tiny rounding differences.
- `lmder_fit` caches fit results per `MGFunction`: `fit_island` repeats
  most fits with identical input (75-84% of all fits in the benchmarks).
- Remove the C++ sources, MINPACK and PORT3, and the Boost dependency
  (CMake, cibuildwheel, CI, README).
- Add `test/tmgfunction.py` and benchmark scripts in
  `benchmark/replace-cbdsm`.

Outputs of `test/tbdsf_process_image.py` are bit-identical to those of the
C++ version; end-to-end runs are faster (see
`benchmark/replace-cbdsm/DESCRIPTION.md`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `get_fitted_parameters`: use `flatten()`, which always returns a copy,
  instead of `ravel()` followed by an explicit `copy()`.
- `_update`: drop `casting='unsafe'` from `N.exp`; the long double to
  float64 output cast is allowed by the default `same_kind` rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Write the fitted parameters through the cached flat view, like
`residual` and `jacobian` already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`fit_island` stored its `MGFunction` in `isl.mg_fcn`, which is never
used. With the pure-Python `MGFunction`, each instance keeps its
evaluation caches (about 25 times the size of the island image), so with
`ncores=1` memory grew with the total size of all fitted islands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When each forked worker imported `mgfunction` itself, it copied about
2.5 MiB of shared memory pages, and the test image needed 2-3 MiB more
than with the C++ `_cbdsm`. Importing it in the parent, before the
workers are forked, avoids that. Add memory measurements to the
benchmark notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gmloose and others added 4 commits November 9, 2026 22:13
- Drop the `MGFunction` methods that `gausfit.py` doesn't use
  (`remove_gaussian`, item access, the `parameters` setter, `errors`,
  `__len__`).
- Always index the fitted parameters with an index array.
- Write the evaluation of the Gaussians and the Jacobian as expressions
  instead of chains of in-place operations, with the same floating point
  operations in the same order.

Replaying all 33097 fits of NGC6048 and 904 fits of the test image gives
bit-identical results; the replay is 0.3% (NGC6048) and 4% (test image)
faster (islands < 50 pixels: 4% faster, >= 1000 pixels: 0.9% slower), and
peak memory is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `MGFunction._update` returns the residual (replaces `_residual`), and
  `_gaussian_scalars` returns only the numbers derived from the parameters.
- `lmder_fit` handles new and cached fit results the same way.
- `test/tmgfunction.py`: share the initial parameters of the tests, and
  shorten `test_repeated_fits`.
- `record_fits.py` also works with `bdsf.mgfunction`, and `replay_fits.py`
  compares any two implementations (a Python file or a module such as
  `bdsf._cbdsm`), including peak memory of the largest fits.

Replaying all 33097 fits of NGC6048 and 904 of the test image gives
bit-identical results, with the same run time and peak memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `replay_fits.py`: compare results with `repr()`, so that fits with
  identical NaN results count as identical, and count a NaN in only one of
  them as a difference.
- `DESCRIPTION.md`: results on top of `master` 7ddb350 with the simplified
  `mgfunction`: all FITS outputs of both test runs are identical to those
  of C++, and the 8 non-identical fits of NGC6048 (2 distinct fits) diverge
  in both versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Written as expressions, the evaluation of the Gaussians and the Jacobian
creates temporary arrays, which made fits of islands with >= 1000 pixels
1.7% slower than at af0bc75. Use in-place operations again, as before,
and take the Jacobian rows of each Gaussian once (indexing creates a new
view each time), which keeps small islands faster.

Replaying all 33097 fits of NGC6048 and 904 of the test image gives
bit-identical results; compared with af0bc75, fits of every island size are
as fast or faster (NGC6048: -1.3% in total, >= 1000 pixels: -0.2%; test
image: -3.8%), with the same peak memory. Update the benchmark notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants