Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions Lib/test/test_buffer.py
Original file line number Diff line number Diff line change
Expand Up @@ -4447,6 +4447,43 @@ def test_bytearray_release_buffer_read_flag(self):
with self.assertRaises(SystemError):
obj.__buffer__(inspect.BufferFlags.WRITE)

@support.cpython_only
@unittest.skipIf(_testcapi is None, "requires _testcapi")
@unittest.skipIf(struct is None, "requires struct")
Comment thread
cmaloney marked this conversation as resolved.
Outdated
Comment thread
jakelishman marked this conversation as resolved.
Outdated
def test_bytearray_alignment(self):
# gh-140557: pointer alignment of buffers including empty allocation
# should be at least to `size_t`.
align = struct.calcsize("N")
cases = [
bytearray(),
bytearray(1),
bytearray(b"0123456789abcdef"),
bytearray(16),
]
ptrs = [_testcapi.buffer_pointer_as_int(array) for array in cases]
self.assertEqual([ptr % align for ptr in ptrs], [0]*len(ptrs))

@support.cpython_only
@unittest.skipIf(_testcapi is None, "requires _testcapi")
@unittest.skipIf(struct is None, "requires struct")
Comment thread
jakelishman marked this conversation as resolved.
Outdated
def test_array_alignment(self):
# gh-140557: pointer alignment of buffers including empty allocation
# should be at least to `size_t`.
align = struct.calcsize("N")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chatted with a Rust export on the "Rust for CPython Discord". Came to the conclusion with them that we should align array.array to 8 bytes since it can store 8 byte things (typecodes: "qQd", https://docs.python.org/3/library/array.html#array.array). On 32 bit platforms size_t will only require 4 byte alignment.

Rather than struct.calcsize("N") I think we can just use the constant 8 here + have a comment pointing to the item size table in the array.arraydoc?

Copy link
Copy Markdown

@joshtriplett joshtriplett Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chiming in to confirm that we had this discussion. 👍 for aligning to 8 bytes on all platforms; it seems simpler and less surprising than trying to marginally optimize for the subset of 4-byte platforms where long long and double are 4-byte aligned. Might also have benefits for other purposes, such as SIMD algorithms.

Per https://en.wikipedia.org/wiki/Data_structure_alignment:

when compiling for 32-bit x86:
[...]
A double (eight bytes) will be 8-byte aligned on Windows and 4-byte aligned on Linux (8-byte with -malign-double compile time option).
A long long (eight bytes) will be 8-byte aligned on Windows and 4-byte aligned on Linux (8-byte with -malign-double compile time option).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah thanks - I was overly focused on how the buffer inside PyBytes ends up aligned and not thinking properly. I changed the test of array.array to take the max item size from all available formats for the platform, though I can change it to be hard-coded 8 if that's preferred.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Max of array.array's elements I like / means both new element adding or buffer allocation changes would result in the test needing an intentional update / helps ensure changes are intentional

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't fully paged in to this (awesome!) workstream yet, but I wanted to note that I'm incredibly inspired by the unstable rust core::simd feature, and I maintain both the https://docs rs/re2 and https://docs.rs/vectrorscan-async (hyperscan) rust wrapper crates.

With regards to SIMD, I have two ideas:

  1. Reflecting the FFI constraints into SIMD-compatible memory for rust core::simd could be a great investigation for rust's external types proposal: Tracking issue for RFC 1861: Extern types rust-lang/rust#43467 (comment)
  2. I have begun work that searches for SIMD instructions in the cpython configure script: https://discuss.python.org/t/add-zero-copy-conversion-of-bytearray-to-bytes-by-providing-bytes/79164/70
    • I would be very interested to discuss this sort of thing further. In that post I mention two specific use cases (one of them being url quoting in pip) which would produce very noticeable real-world speedups in foundational python tooling like pip.

I'm also right now doing a cleanroom implementation of zstd in rust (I can explain my concerns with ruzstd) and making use of alignment for SIMD to improve over the zstd C implementation (currently working on a ring buffer component), which I very much intend to plug into a python native module.

This comment is in full support of this PR and this workstream. Please let me know if I can follow up and learn more about this! Thanks! ^_^

cases = [array.array(fmt) for fmt in ARRAY]
# Empty arrays
self.assertEqual(
[_testcapi.buffer_pointer_as_int(case) % align for case in cases],
[0] * len(cases),
)
for case in cases:
case.append(0)
# Allocated arrays
self.assertEqual(
[_testcapi.buffer_pointer_as_int(case) % align for case in cases],
[0] * len(cases),
)

@support.cpython_only
def test_pybuffer_size_from_format(self):
# basic tests
Expand Down
1 change: 1 addition & 0 deletions Misc/ACKS
Original file line number Diff line number Diff line change
Expand Up @@ -1151,6 +1151,7 @@ Per Lindqvist
Eric Lindvall
Gregor Lingl
Everett Lipman
Jake Lishman
Mirko Liss
Alexander Liu
Hui Liu
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
:func:`bytearray` and :func:`array.array` buffers now have the same alignment
Comment thread
jakelishman marked this conversation as resolved.
Outdated
when empty as when allocated. Unaligned buffers can still be created by slicing.
Comment thread
jakelishman marked this conversation as resolved.
24 changes: 24 additions & 0 deletions Modules/_testcapi/buffer.c
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,27 @@ static PyTypeObject testBufType = {
.tp_members = testbuf_members
};

/* Get the pointer from a buffer-supporting object as a PyLong.
*
* Used to test alignment properties.*/
Comment thread
jakelishman marked this conversation as resolved.
Outdated
static PyObject *
buffer_pointer_as_int(PyObject *Py_UNUSED(module), PyObject *obj)
{
PyObject *out;
Py_buffer view;
if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) != 0) {
return NULL;
}
out = PyLong_FromVoidPtr(view.buf);
PyBuffer_Release(&view);
return out;
}

static PyMethodDef test_methods[] = {
{"buffer_pointer_as_int", buffer_pointer_as_int, METH_O},
{NULL},
};

int
_PyTestCapi_Init_Buffer(PyObject *m) {
if (PyType_Ready(&testBufType) < 0) {
Expand All @@ -106,6 +127,9 @@ _PyTestCapi_Init_Buffer(PyObject *m) {
if (PyModule_AddObjectRef(m, "testBuf", (PyObject *)&testBufType)) {
return -1;
}
if (PyModule_AddFunctions(m, test_methods) < 0) {
return -1;
}

return 0;
}
2 changes: 1 addition & 1 deletion Modules/arraymodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -2655,7 +2655,7 @@ array_ass_subscr(PyObject *op, PyObject *item, PyObject *value)
}
}

static const void *emptybuf = "";
static const _Py_ALIGNED_DEF(ALIGNOF_MAX_ALIGN_T, char) emptybuf[] = "";


static int
Expand Down
Loading