-
-
Notifications
You must be signed in to change notification settings - Fork 34.5k
gh-140557: Force alignment of empty bytearray and array.array buffers
#140559
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
6b37e66
Force alignment of empty `bytearray` and `array.array` buffers
jakelishman 1fea8e5
Add tests of buffer pointer alignment
jakelishman b7eed2b
Make NEWS more concise
jakelishman dd9e50b
Merge remote-tracking branch 'python/main' into max-align-buffers
jakelishman 9d454de
Avoid `ctypes` import on unsupported platforms
jakelishman de09bbe
Merge remote-tracking branch 'python/main' into max-align-buffers
jakelishman ad966dc
Revert Python-space `ctypes` tests
jakelishman b036215
Add alignment tests with `_testcapi` helper
jakelishman 2dec490
Slacken test to match 'size_t' only
jakelishman 92a5fff
Fixup review comments
jakelishman d3ca57f
Require max observable alignment in array test
jakelishman File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 2 additions & 0 deletions
2
Misc/NEWS.d/next/Core_and_Builtins/2025-10-24-17-30-51.gh-issue-140557.X2GETk.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| :class:`bytearray` buffers now have the same alignment | ||
| when empty as when allocated. Unaligned buffers can still be created by slicing. | ||
|
jakelishman marked this conversation as resolved.
|
||
2 changes: 2 additions & 0 deletions
2
Misc/NEWS.d/next/Library/2026-01-07-11-57-59.gh-issue-140557.3P6-nW.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| :class:`array.array` buffers now have the same alignment when empty as when | ||
| allocated. Unaligned buffers can still be created by slicing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Chatted with a Rust export on the "Rust for CPython Discord". Came to the conclusion with them that we should align
array.arrayto 8 bytes since it can store 8 byte things (typecodes: "qQd", https://docs.python.org/3/library/array.html#array.array). On 32 bit platformssize_twill only require 4 byte alignment.Rather than
struct.calcsize("N")I think we can just use the constant 8 here + have a comment pointing to the item size table in thearray.arraydoc?Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Chiming in to confirm that we had this discussion. 👍 for aligning to 8 bytes on all platforms; it seems simpler and less surprising than trying to marginally optimize for the subset of 4-byte platforms where
long longanddoubleare 4-byte aligned. Might also have benefits for other purposes, such as SIMD algorithms.Per https://en.wikipedia.org/wiki/Data_structure_alignment:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh yeah thanks - I was overly focused on how the buffer inside
PyBytesends up aligned and not thinking properly. I changed the test ofarray.arrayto take the max item size from all available formats for the platform, though I can change it to be hard-coded 8 if that's preferred.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Max of
array.array's elements I like / means both new element adding or buffer allocation changes would result in the test needing an intentional update / helps ensure changes are intentionalThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't fully paged in to this (awesome!) workstream yet, but I wanted to note that I'm incredibly inspired by the unstable rust
core::simdfeature, and I maintain both the https://docs rs/re2 and https://docs.rs/vectrorscan-async (hyperscan) rust wrapper crates.With regards to SIMD, I have two ideas:
core::simdcould be a great investigation for rust's external types proposal: Tracking issue for RFC 1861: Extern types rust-lang/rust#43467 (comment)I'm also right now doing a cleanroom implementation of zstd in rust (I can explain my concerns with ruzstd) and making use of alignment for SIMD to improve over the zstd C implementation (currently working on a ring buffer component), which I very much intend to plug into a python native module.
This comment is in full support of this PR and this workstream. Please let me know if I can follow up and learn more about this! Thanks! ^_^