Repository navigation
pandas3 support - #194
Open
MahmoodEtedadi wants to merge 48 commits into
Open
pandas3 support#194MahmoodEtedadi wants to merge 48 commits into
MahmoodEtedadi wants to merge 48 commits into
Conversation
MahmoodEtedadi
marked this pull request as draft
October 14, 2026 16:14
MahmoodEtedadi
added a commit
to MahmoodEtedadi/seismometer
that referenced
this pull request
Oct 15, 2026
The repository names fragments <ISSUE>.<TYPE>.rst, with the pull request number when no issue applies. The six free-text fragments become one per type for epic-open-source#194, each keeping the same sentences.
ydata-profiling (latest 4.18.4) requires pandas<3, numpy<2.4, matplotlib<=3.10 and Python<3.14, so as a hard dependency it blocks installing seismometer on the pandas 3 stack at all. Its imports were already lazy, so moving it to a `profiling` extra only affects the four profiling reports, which now point users at `pip install seismometer[profiling]`. setuptools moves with it: it was only a runtime dependency because ydata imports pkg_resources. The extra is not part of `all`, since it would make the resolver downgrade pandas (or fail on Python 3.14). Tests that patch objects inside ydata_profiling are skipped when it is not installed.
Widen the pins to pandas<4, pyarrow<26 and great-tables<2, keeping the existing lower bounds so platform runtimes on pandas 2 still work. Add Python 3.13 to the CI matrix. Fixes for what breaks on the new stack: - ParquetDataset no longer accepts use_legacy_dataset (removed in pyarrow 20); it was already the only behavior, so drop it. - as_percentages scaled its input in place, which fails on the read-only arrays copy-on-write returns from `.values`, and on pandas 2 silently mutated the caller's data. Return a new array. - rename(copy=False) is deprecated in pandas 3. Rename the freshly loaded events in place instead, which also avoids a copy on pandas 2. - Two groupbys over categorical cohort columns relied on the default `observed`, which pandas 3 flips to True. Pass observed=False so the plot counts and telemetry quantile groups match pandas 2. - Tests: SettingWithCopyWarning no longer exists, and string-parsed times default to datetime64[us]. The windowed event merge also fails on pandas 3 when prediction and event keys load as different string dtypes (object vs str); the next commit fixes that as part of reworking the merge.
The windowed asof merge dominates run_startup. Each event's merge_asof re-hashed the string entity keys and copied every prediction column into a new wide frame. Now: - merge_onto_predictions encodes the prediction and event keys once, jointly, into one integer id per row (KeyCodes). String keys are hashed as arrow strings, several times faster than Python objects. - forward/nearest merges run merge_asof on a slim frame of just the predict time and the key id, and attach the event columns back by position, since merge_asof keeps the left row order. The result is identical to merge_asof on the full frames, including dtypes, column order and the fresh RangeIndex. NaN keys still match each other, as in merge_asof. When equivalence isn't guaranteed (an event column already exists, or key dtypes differ in kind), the original merge_asof call runs instead. merge_onto_predictions now also gives the deduplicated predictions a RangeIndex up front. Before, when no event produced a merge, the sorted frame kept a permuted index, and create_cohorts then assigned cohort columns misaligned by label. Integer ids also fix the merge on pandas 3. merge_asof requires identical key dtypes, and predictions cast to `object` by the data dictionary no longer matched events read as `str`. The resulting MergeError was swallowed and surfaced as an UnboundLocalError. On the 3M-row diabetes example with pandas 3.0.6, run_startup took 30-32s, against 60-74s for main plus a minimal key-dtype fix. The loaded frame is identical to that reference on pandas 3, and to main on pandas 2.
Move the window filter, imputation, casting and reference-time cleanup that follow the merge into _finish_windowed_event, unchanged. A future commit reuses it for the polars merge, so both paths share these semantics instead of duplicating them.
run_startup(memory_efficient=True) loads the same values with less memory and time. Default loading is unchanged. - Strings stay arrow-backed (pandas 3's `str` dtype; on pandas 2 the equivalent string[pyarrow_numpy]) even where the data dictionary says `object`. On pandas 3, that cast is what turns the keys and string cohort sources into Python objects, the largest columns of the example frame. - If polars is installed (new `perf` extra), events are merged with polars.merge_onto_predictions. Polars only does the sort, dedupe and asof matching. Event values are taken from the pandas events frame, and windowing, imputation and casting reuse _finish_windowed_event, so the frame matches the pandas merge. The polars merge handles only what it reproduces exactly; otherwise it falls back to the pandas merge and logs why. Polars breaks `nearest` ties differently and never matches missing keys, so those fall back, as do mismatched key dtypes, events with no times and existing event columns. Tests run both merges on randomized data (duplicates, ties, NaT times, offsets, zero windows, multi-source events) and require identical frames. This takes the polars load from the memory-efficient-load branch without its parts that change results: float32 downcasts, integer downcasts, and polars `nearest` and null-key matching. Its parquet sidecar files are dropped too, since they wrote copies of the data next to the source. On the 1M-row diabetes example with pandas 3, the load took 4.7s instead of 8.6s and the frame was 337 MB instead of 540 MB, with identical values.
Rebase the image on python:3.13-slim and pin the latest releases that fit setup.cfg: pandas 3.0.6, numpy 2.5.3, pyarrow 25.0.1, polars 2.0.0 (perf extra), great-tables 1.0.0, matplotlib 3.11.2, scikit-learn 1.9.1 and IPython 9.17.1. ydata-profiling and numba (only needed by ydata) are dropped, since ydata requires pandas<3. great-tables was missing even though it is a runtime dependency.
Add 3.14 to the CI matrix. ydata-profiling was the only dependency without 3.14 support, and it is now optional. The suite passes on Python 3.14.8 and on 3.12.14 with the devcontainer pins (pandas 3.0.6, numpy 2.5.3, pyarrow 25.0.1, polars 2.0.0).
Internal environments are moving to 3.14. Every pinned package has 3.14 wheels, so only the base image changes. 3.12 and 3.13 are still covered by CI.
On the 1M-row diabetes example the polars merge raised the load's peak memory instead of lowering it: 2092 MB against 2009 MB for the default load (single runs). Polars allocates with jemalloc, which holds on to freed memory for a while; with jemalloc set to return it at once, the peak fell to 1919 MB. Merging with pandas instead peaked at 1881-1886 MB. memory_efficient now only keeps strings arrow-backed, which is what shrinks the loaded frame (322 MB against 515 MB). The merge is slower without polars (about 5 s against 1.8 s at 1M rows); a future commit removes some of that cost from the pandas merge. This drops the perf extra, the polars pin in the devcontainer and polars_merge with its tests.
_merge_event renamed each event's sources with replace() on the whole events frame, copying every column of every row once per configured event. It now selects the event's rows (its sources, plus rows already named after the display name, which replace() kept too) and renames only those. On the 1M-row diabetes example the six replace() calls took about 0.5 s of a 5 s merge. The new test covers rows already named after the display name.
merge_onto_predictions sorted the predictions, dropped repeated entity keys + prediction times and dropped rows without a prediction time in three steps, each copying the frame, and compared the string keys to find repeats. The keys are now encoded first, on all predictions instead of the remaining ones, and one pass compares the integer codes and prediction times, keeping the same rows. When the codes don't cover every entity key, or the key dtypes of predictions and events differ in kind, it compares the key columns as before. On the 100k and 1M-row diabetes examples the merged frames are identical to the previous code, with and without memory_efficient. With the previous commit, the 1M merge takes 4.1-4.8 s instead of 5.0-5.8 s.
memory_efficient was an opt-in run_startup flag that kept columns holding only strings as Arrow-backed strings (pandas 3's str dtype) instead of casting them to object. It is now how data always loads, and the flag is gone; it was never released. On pandas 3 this matches the dtype the events and any undeclared string columns already have, so prediction and event keys share one dtype. On the diabetes example the notebook outputs are the same as with object strings. The 1M and 3M-row notebooks ran 14% and 21% faster with 5-7% lower peak memory, and the loaded frame is 37% smaller. Code that edits sg.dataframe sees the new dtype: assigning a non-string into one of these columns raises TypeError, and .unique() returns an Arrow array. The changelog entry notes the change.
When an event's columns already exist in the predictions, the merge falls back to merge_asof on the full frames, and merge_asof rejects keys that are strings on both sides but in different dtypes. That happened with object keys on one side and str keys on the other, for example Arrow-string predictions and object events on pandas 2. The MergeError was caught as a ValueError, and the merge then failed with an UnboundLocalError. The fallback now casts those event keys to the predictions' dtype first. The result is what main produced in this case: the existing column and the event's get _x and _y suffixes.
_encode_keys concatenates each key column of the predictions and the events before factorizing them together. When the two dtypes differ, for example int64 predictions and a categorical with missing values in the events, pandas upcasts the concatenation to float64, where every id from 2**53 up can collide with its neighbor. The merge then attached events to the wrong entity without any error; main raised in this case. Key columns whose dtypes still differ after the string cast are factorized as objects, which keeps Python ints exact. Matching dtypes, the usual case, take the same path as before.
Installed at the declared minimum versions, the example notebook and the tests failed: - pandas 2.1: merge_asof rejects the int32 key codes, so run_startup fails. pandas 2.2 accepts them. - great-tables 0.11 and 0.12: the analytics and fairness tables use loc.title and loc.column_header, added in 0.13. - matplotlib 3.6: the first inline plot fails, because matplotlib-inline 0.2, which ipykernel installs, calls rcParams._get, added in matplotlib 3.7. The great-tables and matplotlib floors were already wrong on main. With these floors and the other dependencies at their minimums, the tests pass apart from two Windows-only int32 dtype asserts that fail on main too.
The test matrix installs the latest versions of every dependency, so nothing checked the floors in setup.cfg, and three of them had fallen behind. A new job installs install_requires at their minimum versions on Python 3.10, from .github/constraints-min.txt, and runs the tests. setup.cfg notes to update the file with the floors.
The fragments use towncrier's "+" prefix, for entries not tied to an issue; rename them to the pull request number once it exists.
_match_string_keys cast every object event key to the predictions' string dtype, so object keys holding ints became strings and 1 matched "1" in the fallback merge, while the integer-code merge kept them apart. It now casts only keys that hold just strings (or nothing), or casts to object, which never changes values. The first, last and count strategies, and the merge for events without times, now cast their event keys too. They use pd.merge, which on pandas 2 returned object keys when the events were object and the predictions Arrow strings, undoing the Arrow load. Their values are unchanged.
The entry called the dtype pandas' str dtype, which is only true on pandas 3, and did not say that assigning non-strings into these columns now raises TypeError.
seismometer.data.profile describes each column and raises the same data quality alerts as ydata-profiling 4.18 did under seismometer's report_config.yml, without depending on it: - Variable types come from the dtypes, as with ydata's infer_dtypes: false (Numeric, Categorical, Text, Boolean, DateTime, Unsupported). - Each column's description uses ydata's formulas and keys: counts, missing, distinct and unique, quantiles, moments, MAD, monotonicity, 50-bin histograms, imbalance, text lengths, characters and words. - Alerts use ydata's rules and thresholds (missing, zeros and infinite above 1%, imbalance above 0.5, |skew| above 20, more than 50 categories, unique, constant, unsupported, empty). The keys stay the same, so alert_config.yml thresholds keep working. Not reproduced: time-series detection (needs statsmodels), Unicode script and block names (ydata showed "(unknown)" without an optional package), and the features seismometer had turned off (correlations, interactions, missing diagrams, samples, duplicates). The tests compare against ydata-profiling 4.18.4's own output on a frame covering every type and alert path. make_profile_golden.py regenerates that output. Every statistic, type, histogram and alert matches on pandas 2. On pandas 3 only the order of tied value counts differs (a pandas change), so value counts are compared as sets. Against the 100k-row diabetes example, all 36 columns and 16 alerts matched ydata's description.
ProfileReport renders one profile, or two side by side, as a standalone HTML page laid out like the ydata-profiling reports: an overview (dataset statistics, variable types, alerts, reproduction), then a section per variable with the same tables, labels, tabs, frequency tables, histograms and category plots. Number formatting, alert sentences, badge colors and the plot style are ydata's, so values read the same: two significant digits, "< 0.1%", "x 10^5", and "has 22410 (22.6%) missing values". Comparisons show each value in a column per dataset, in ydata's two colors. Differences: no JavaScript (tabs and "More details" are plain HTML and CSS), no word clouds (they need the wordcloud package), and data values and column names are HTML-escaped.
feature_alerts, feature_summary, target_feature_summary and cohort_comparison_report now use seismometer.data.profile and ProfileReport. ydata-profiling was the last dependency blocking pandas 3 and Python 3.14, and its maintainers have deprecated the package. With it go the profiling extra and the ydata report_config.yml. The report functions keep their signatures, output files and on-disk caching. alert_config.yml filtering, alert severities, the alerts JSON and the telemetry metrics are unchanged; the telemetry source label stays "ydata_profiling_alerts" so existing queries still match. Profile alerts take the same arguments as ydata's Alert, which the alert config tests construct directly.
The date histograms bin naive dates as seconds since 1970 and labeled the ticks with datetime.fromtimestamp. On Windows that raises OSError for times before 1970, so a report with an older date column, such as a birth date, failed to render. Elsewhere it shifted the labels into the local time zone, which can show the day before. The ticks now add the seconds to 1970-01-01 directly.
ProfileReport.to_file opened the file for writing, which empties it, and then rendered the page. A render that failed left an empty file, and the comparison reports skip generating when the file exists, so they then showed that empty report until it was deleted.
Each variable's section, tabs and alert links used the column's slug as their id, and slugs ignore case, so columns such as age and Age shared one id. In the example report six pairs collide (age, race, gender, location, department, specialty), and the second column's tabs and alert links went to the first. The report now numbers repeated slugs (age, age-2) and links each alert to its column's anchor. alert_html takes the anchor as an optional argument and defaults to the plain slug, as the feature alerts outside the report use it.
It replaces the entry for the optional profiling extra, which this branch removes along with ydata-profiling.
With no key columns, _can_merge_on_codes returned True, since every key matched in kind, and _encode_keys then failed with an AttributeError. merge_windowed_event already rejects an empty key list, but _merge_with_strategy is called with one in test_merge_with_strategy_empty_pks_raises, added on main in epic-open-source#187, which expects merge_asof's error. With no keys it now takes the merge_asof path again.
The repository names fragments <ISSUE>.<TYPE>.rst, with the pull request number when no issue applies. The six free-text fragments become one per type for epic-open-source#194, each keeping the same sentences.
The macOS CI jobs failed test_column_statistics[float_mixed]: 170 of the column's 903 finite values differed from ydata_golden.json in the last bit. NumPy's rng.normal(2, 3) computes 2 + 3 * z with a fused multiply-add on macOS, so it rounds once instead of twice; recomputing the fused results reproduces exactly those 170 values. Draw standard normals and scale them with separate NumPy operations instead. The draws are the same and, off macOS, the values are bitwise identical, so the golden file doesn't change.
On pandas 3, object dates such as 9999-12-31 stay valid dates instead of becoming NaT. matplotlib then places a tick past year 9999, and its label raised OverflowError, failing the feature reports.
On pandas 3, a quantile of a Float64 column that interpolates between infinite values is NA rather than nan, and the IQR then raised TypeError. Casting the quantiles to float gives the plain float64 results on both pandas versions.
An Alert's values hold the column's description, including sample rows and value counts. The "Misconfigured alert" warning printed all of it, so a typo in alert_config.yml could write patient data to the log. ydata's Alert printed only a one-line description.
Codes from factorize_keys are looked up by index label, so passing events with relabeled rows silently pairs them with the wrong keys.
parse_alert now takes seismometer's own profile Alert, since ydata-profiling is no longer a dependency.
MahmoodEtedadi
force-pushed
the
metedadi/pandas3-support
branch
from
October 23, 2026 21:41
9d1aa5e to
569ea55
Compare
pandas 3.1 returns nan for the kurtosis and skewness of constant data, where earlier versions and ydata-profiling report 0. The profiler now returns 0 itself when a column holds one finite value, as before. A variance check would miss constant floats such as 0.1, whose variance rounds to about 1e-34 rather than 0.
pandas 3.1 deprecates inplace (PDEP-8), and plain rename copies the data on pandas 2. Setting the column labels directly does neither.
ydata-profiling applies its plot settings and then seaborn's white style on top. The report copied only the first layer, so its plots had a grey background with a white grid instead of ydata's white one, and its font list left out DejaVu Sans, matplotlib's bundled font. Where Arial and Liberation Sans aren't installed, as in the devcontainer, matplotlib then warned on every label (404 times for a 3-column report) before falling back to DejaVu Sans anyway. The style now ends with seaborn's white style, which gives the same 51 matplotlib settings as ydata's plotting context.
The report kept ydata-profiling's tabs but showed the items of every tab side by side. ydata lays some out differently, so: - Extreme values of numeric columns get a tab each for the minimum and maximum values; - the Categories tab of categorical columns shows the common values table above its plot; - the first tab of boolean columns is Common Values (Table); - the Alerts and Reproduction overview tabs start with a header.
The built-in report kept ydata-profiling's sections but not its look. The CSS now takes ydata's Bootstrap values: the light navbar with its links on the right, heading and body font sizes, the bordered card around the overview, the alert count as a badge, taller table rows with borders, the light More details button on the right, and the "?" hint at the end of its row. Plots keep their size, centered, and frequency bars take the dataset's color, as ydata draws them. A comparison is titled "Comparing <first> and <second>", as ydata titled main's comparison reports; the notebook link keeps the full report title. The report still loads no scripts or external files.
ydata-profiling's reports open the Variables section with a dropdown that shows only the chosen column, or all of them. The built-in report now has the same dropdown. Like ydata's, it runs a small inline script, so in JupyterLab's HTML viewer it works after Trust HTML; untrusted, all columns stay listed. The tabs still work without the script.
The PR checklist asks for full numpy-style docstrings on new methods. Also annotates the path argument of ProfileReport.to_file.
pandas 3.1 deprecates the inplace keyword (PDEP-8) and warns on each call. These frames are local, so assigning the result is equivalent.
pandas 3.1 deprecates sort_values' inplace keyword (PDEP-8), and pandas 4 removes it. With sort=True, the function now sorts copies, so the caller's frames are no longer reordered; the returned frame is the same. Seismometer's loader passes sort=False and is unaffected.
The final benchmark measured the feature reports at 2.9 to 3.3 times faster than ydata-profiling. Reports saved earlier are still reused, so upgrading users see the old ydata reports until they delete them.
The PR checklist asks for a versionchanged directive on breaking API changes. String columns now load as Arrow-backed strings, which reject non-string values.
A category column keeps the categories of values that load-time filters removed. create_cohorts counted those too, so filtering the example's specialty column to its top 18 values, with the column declared as category in the data dictionary, still counted all 38 and dropped the cohort for exceeding the limit of 25.
create_cohorts drops a cohort with too many values or too few rows, but its hierarchies were still resolved. Building their combinations checked the source columns, which exist, then read the missing cohort columns, so run_startup failed with a KeyError. The widgets would have failed the same way, since they read the combinations for every resolved hierarchy. Such a hierarchy is now skipped with a warning, and its cohorts that were created stay available as ordinary selections.
MahmoodEtedadi
marked this pull request as ready for review
October 28, 2026 22:08
The test resource is seismometer's own reference data now that ydata-profiling is no longer a dependency, so drop ydata from its name.
3 of 5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Related to #167:
run_startupnow peaks about 30% lower at 1M+ rows.Description of changes
Adds pandas 3 support. ydata-profiling blocks it (its latest release requires
pandas<3,numpy<2.4and Python<3.14), so this also replaces it with a built-in profiler.Dependencies
.github/constraints-min.txt).setuptools<81runtime pin from fix setuptools version #186, since nothing importspkg_resourcesnow. The build pin inpyproject.tomlstays.Data loading
stron pandas 3), even when the data dictionary saysobject. They use less memory.stringandobject). On pandas 3 this broke normal loading.Feature reports
seismometer.data.profile) and HTML report (seismometer.report.profile_report) keep ydata-profiling's statistics, alert rules,alert_config.ymlsettings and report layout. Tests compare them with ydata-profiling's output.Breaking changes
object. They load as Arrow-backed strings (pandas 3's defaultstrdtype), which saves memory and time. Code that writes non-string values into them, e.g.sg.dataframe.loc[rows, "col"] = 0, now raisesTypeError. Convert the column first with.astype(object).AlertConfigProvider.parse_alerttakes seismometer'sAlert(seismometer.data.profile.Alert) instead of ydata-profiling's, since ydata-profiling is gone. This only affects code that passed ydata-profiling alerts to it directly.merge_windowed_event(sort=True)no longer reorders the dataframes you pass in. It still returns the sorted, merged result. pandas 3.1 deprecates sorting in place. If your code relied on the inputs being sorted after the call, use the returned frame or sort them yourself.Benchmarks
The example notebook, end to end, median of 3 runs. Main uses Python 3.12 and pandas 2.2.3; this branch uses pandas 3.0.6. Times are per-context off / on, where on enables every per-context option in the notebook.
Testing
pip install -e .[dev].example-notebooks/binary-classifier/classifier_bin.ipynb. Delete itsoutputsfolder if there is one, since saved reports are reused. Then run all cells.findfontwarnings appear.sm.Seismogram().dataframe.dtypeslists string columns asstr, notobject.sm.feature_summary()andsm.target_feature_summary(), and one fromsm.cohort_comparison_report()(pick two cohorts and clickGenerate Report). ClickTrust HTMLin the viewer toolbar, then check that:Select Columnsdropdown under Variables shows one variable at a time.sm.feature_alerts()should show the same sections, statistics and alerts.Author Checklist
.. versionchanged:: NEXT_VERSION_PLACEHOLDERdirective to affected docstrings.changelog/ISSUE.TYPE.rstfiles; see changelog/README.md.