Skip to content

fix: preserve int64 dataframe indices on 32-bit Windows - #4492

Open
ianna wants to merge 6 commits into
scikit-hep:mainfrom
ianna:ianna/dataframe-index-width
Open

ianna wants to merge 6 commits into
scikit-hep:mainfrom
ianna:ianna/dataframe-index-width

Conversation

@ianna

@ianna ianna commented Oct 27, 2026

Copy link
Copy Markdown
Member

ak.to_dataframe can produce int32 index levels on Windows while pandas constructs the expected levels as int64, failing test_3694_to_dataframe_align_multindex.py::test_merge_option.

Create row coordinates as int64 independently of the repeat-count dtype, and explicitly restore merged index levels as int64. Keep the existing platform-compatible repeat counts, including the int32 conversion on 32-bit platforms. This addresses 32-bit Windows as well as 64-bit Windows without changing dependency requirements.

Related: #4458 removes the Windows-wide repeat-count cast when dropping NumPy 1 support. This fix keeps that compatibility path and separates index widths from repeat-count widths; it also replaces the platform-dependent astype(int) when restoring merged levels.

Validation: 23 targeted dataframe tests passed, including simulated Windows 32-bit and 64-bit branches and all merge modes. Ruff and whitespace checks passed. Native Windows execution remains pending in CI; simulated platform flags do not substitute for a native 32-bit runtime.

AI assistance was used to implement and validate this contribution.

@github-actions github-actions Bot added the type/fix PR title type: fix (set automatically) label Oct 27, 2026
@ianna
ianna requested a review from TaiSakuma October 27, 2026 14:56
Assisted-by: Claude Code:claude-opus-5-5
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.61%. Comparing base (24e8900) to head (f703275).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/operations/ak_to_dataframe.py 94.11% <100.00%> (+1.96%) ⬆️

... and 2 files with indirect coverage changes

…o int64

Rename the test to match the PR number and cover nested, mixed-depth
fields so that both shared and extra index levels are checked.

Assisted-by: Claude Code:claude-opus-5-5
@ariostas

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

I pushed two commits to this branch:

e252a87 – install pandas on Windows CI. pandas was excluded on win32 in every requirements file, so ak.to_dataframe was never tested on Windows. The failure in the integration tests (test_3694_to_dataframe_align_multindex.py::test_merge_option[right], int32 vs int64 index level) shows up with pandas 3 on 64-bit Windows (Python 3.13/3.14/3.14t), not on 32-bit; with pandas 2.3.3 on Python 3.10 it passes.

  • requirements-test-full.txt: pandas is now installed on Windows for Python >= 3.11. The last pandas with 32-bit Windows wheels (2.0.3) doesn't work with NumPy 2, and markers can't tell 32-bit Python apart on Windows, so the 3.10 jobs (which include the x86 one) stay without it.
  • requirements-test-nogil.txt: pandas on all platforms (there are cp314t Windows wheels).
  • requirements-test-numpy1.txt: added pandas, which resolves to 2.0.3 with NumPy 1.26 on the x86 job, so the dataframe tests also run on a real 32-bit interpreter.

1a70b39 – fixes and test.

  • counts is now cast to np.intp instead of int32 under win or bits32. numpy.repeat only needs counts that safely cast to intp; on 64-bit Windows intp is int64, so the old cast just truncated (lists with more than 2**31-1 elements would wrap). Since the index arrays are created as int64 now, the Windows special case isn't needed.
  • _merge_multi_indexed_dfs now casts all index levels to int64 after the merge, not only the extra ones, so the result doesn't depend on what dtype pandas' join gives the shared levels.
  • The test is renamed to test_4492_to_dataframe_index_dtype.py (it reused 3694) and rewritten. It no longer monkeypatches ak._util.win/bits32 (the code doesn't read them anymore); instead it relies on the Windows and 32-bit CI jobs above. It uses fields of depth 1, 2, and 3 with empty and missing lists, so it covers the row_arrays branch and merges with both shared and extra levels, checks that every index level is int64 for all how values, and checks the index values of the outer merge. Against the code before this PR (with win forced on), it fails for how=None and how="right".

@github-actions

Copy link
Copy Markdown
Contributor

The documentation preview is ready to be viewed at https://preview.awkward-array.org/PR4492/

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/fix PR title type: fix (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants