Repository navigation
Normalize NumPy dtypes and preserve fractional multipoles - #685
Draft
marcpaterno wants to merge 12 commits into
Draft
marcpaterno wants to merge 12 commits into
marcpaterno wants to merge 12 commits into
Conversation
|
PR revision: No breaking API changes detected. PR public API verdictNo breaking API changes detected. Target: Static analysis only: no behavioral compatibility guarantee or external-dependency signature analysis. Public API changes: 15043d4 -> HEADExecutive summaryNo breaking API changes detected. Breaking changesNone. New public APINone. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## clarify-test-fixtures #685 +/- ##
=======================================================
Coverage 100.0% 100.0%
=======================================================
Files 163 165 +2
Lines 9979 10134 +155
Branches 1199 1218 +19
=======================================================
+ Hits 9979 10134 +155
🚀 New features to boost your workflow:
|
marcpaterno
force-pushed
the
normalize-np-dtypes
branch
from
October 25, 2026 01:06
b9cbcaa to
61c044b
Compare
marcpaterno
changed the title
marcpaterno
force-pushed
the
normalize-np-dtypes
branch
from
October 27, 2026 20:24
a798e36 to
3581802
Compare
Collaborator
|
Hello @marcpaterno :
|
marcpaterno
force-pushed
the
normalize-np-dtypes
branch
from
October 27, 2026 23:10
3581802 to
3891723
Compare
The array getters used `view()`, which reinterpreted stored bytes as the requested dtype instead of checking the stored type. An int64 array read through `get_float_array` returned reinterpreted bit patterns with no error. The getters now return the stored array without copying or converting it, and raise TypeError when its dtype is not native int64 or float64. Basic-dict conversion stores floating sequences explicitly as float64. Tests cover matching dtype and shared storage for empty and non-empty arrays, and rejection of mismatched dtypes, including byte-swapped, unsigned, bool, and complex storage.
Reject boolean, complex, string, object, structured, datetime, and masked inputs instead of silently discarding information. Convert integer and float inputs to float64 before subtraction so residuals no longer overflow int8 or lose precision in float32. Return independent copies from get_data_vector and get_theory_vector. Add tests for supported dtypes, rejected inputs, residual precision, and Gaussian ingress through read and create_ready. Cover NamedParameters basic-dict round trips for integer, float, and boolean values.
Add docs/numeric-types.md, a guide for physicists writing likelihoods that sets the intended NumPy dtype policy: float64 for real scientific values, int64 for exact multipoles and SACC indices, intp for native NumPy indices, bool for masks, and strings for identifiers. It covers converting inputs before arithmetic, accepted and rejected input types, precision limits, DataVector/TheoryVector handling, NamedParameters getter contracts, and a checklist for verifying a likelihood. Retitle GLOSSARY.md to "Firecrown Language" and broaden its scope to scientific likelihoods. Add terms distinguishing normalization (rescaling a distribution to unit integral) from dtype conversion (changing a numeric representation without rescaling), so the guide and glossary use these words consistently.
Coerce real integer and floating z/dndz inputs to float64 at tomography, SourceGalaxyArgs, NumberCountsArgs, WeakLensingArgs, the photo-z shift transforms, and SACC reads. Integer inputs previously made ones_like produce integer bias and alignment arrays, which truncated fractional values (e.g. constant IA alignment). Reject complex, object, string, datetime, structured, and masked arrays with TypeError before conversion, so their values are not silently dropped or altered. Conforming float64 inputs may still share storage. BREAKING CHANGE: source and tomography constructors and the photo-z shift functions now reject non-real or masked arrays that were previously accepted.
Document how TomographicBin, SourceGalaxyArgs, NumberCountsArgs, and WeakLensingArgs accept real integer and floating arrays for redshifts and distribution values, converting them to float64 storage before source calculations. Describe the public active and passive photo-z transforms, which convert before weighted averages and interpolation. Explain that conforming arrays may share storage, including strided arrays, while other real dtypes are converted and may round values. Masked arrays are rejected even when no entries are masked, since there is no missing-data policy. Note that SACC source reads keep their copy, flatten, and sort behavior, that tabulated bias and alignment fields have float64 contracts validated during source-argument construction, and that tracer preparation supplies double-valued arrays to pyccl. In the TomographicBin tutorial, create the example redshift grid with dtype=np.float64 and add a paragraph covering the accepted input dtypes, rejected dtypes (boolean, complex, string, object, structured, datetime, and masked), and the unchanged unit and normalization behavior.
Convert real integer and floating redshift arrays to float64 before evaluating ZDistLSSTSRD distributions and binned distributions, so results no longer depend on the input storage precision. Reject unsupported coordinate dtypes (bool, complex, string, object, structured, datetime, timedelta) and masked arrays with TypeError. Return float64 from LinearGrid1D, RawGrid1D, equal-area bin edges, and the Year 1 and Year 10 lens bin helpers. Add tests for real precision, unsupported inputs, grid and edge dtypes, and scalar and empty outputs. Update the tutorial to pass dtype=np.float64 explicitly.
- Reject noninteger, masked, and overflowing inputs before conversion across metadata, measurements, likelihoods, and SACC operations. - Validate multipoles before cache lookup and range-check temporary CosmoSIS C int arrays without changing stored domain multipoles. - Preserve fractional window labels and sampling coordinates; update fixtures, tutorials, and tests for integer storage, empty selections, covariance ordering, and incompatible example rejection. BREAKING CHANGE: Exact-multipole and index routes now reject floating arrays, including integral-valued floats. Fractional CMB-cross and cosmic shear examples fail at exact-support consumers rather than being accepted. Supply integer support or use a compatible real-label route; fractional coordinates are not automatically rounded.
- Add tests for _as_multipole_array rejecting masked arrays and arrays with non-integer, non-floating dtypes. - Add a test for cached_angular_cl rejecting ell tuples that contain non-numeric values.
Window weights were summed at their input precision, so reduced-precision SACC windows could lose accuracy before normalization. Window weights, effective labels, real angular coordinates, and measured values are now converted to float64 at their ingress points: TwoPointHarmonic, TwoPointReal, TwoPointMeasurement, and extract_window_function. Weight conversion happens before the zero-total check and column sums. Boolean, complex, string, object, and masked arrays are rejected. Conforming float64 arrays are shared rather than copied, and the covariance identifier still hashes the original SACC buffer. Adds workflow and contract tests for float32, int32, and int64 inputs, and updates the numeric types documentation.
marcpaterno
force-pushed
the
normalize-np-dtypes
branch
from
October 28, 2026 02:13
d3678da to
622a30c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make numeric array types explicit at Firecrown's data boundaries: scientific values, redshift samples, and covariance inputs use
float64; exact multipoles and SACC indices use checkedint64. Preserve fractional multipoles in unwindowed SACC spectra, including the CosmoSIS path, while keeping window grids and indices integral.The numeric-type contract is documented in
docs/numeric-types.md, with regression coverage for dtype validation, fractional multipoles, and connector behavior. This PR builds on the typed fixtures in the preceding stack PR.Evidence
TypeError.make lintpasses.clarify-test-fixturesreports no breaking public API changes. It does not establish behavioral compatibility.Merge Danger
Door: Two-way. The change can be reverted without a data migration.
Blast Radius: Numerical paths. Consumers that depended on implicit array dtypes or floating-point values in fields defined as exact integers may need adjustment.