Skip to content

[FIX][LLVM] Handle non-unit-stride masked load and store - #245

Open
naye-gg wants to merge 6 commits into
apache:mainfrom
KarolayTamayoH:fix/masked-load-store-non-unit-stride
Open

naye-gg wants to merge 6 commits into
apache:mainfrom
KarolayTamayoH:fix/masked-load-store-non-unit-stride

Conversation

@naye-gg

@naye-gg naye-gg commented Nov 13, 2026 •

Copy link
Copy Markdown

Fixes apache/tvm#20779.

Problem

A masked load or store whose index is a Ramp with a non-unit stride passes the TIRx well-formedness check, but LLVM module verification fails:

Call parameter type does not match function signature!

For non-unit strides, BufferAccessHelper emits one access per lane, but each lane still receives the full vector predicate (<N x i1>). Passing only the lane's i1 is not enough either: llvm.masked.load and llvm.masked.store only accept vector types, so a scalar access is rejected with Intrinsic has incorrect return type!. Unit-stride accesses are not affected, since they are emitted as a single vector access.

Fix

src/codegen/llvm/codegen_llvm.cc:

  • BufferAccessHelper: when the index has more than one lane and the predicate has the same number of lanes, extract lane i of the predicate for the per-lane access. Scalable predicates are skipped, because lanes() is not defined for them.
  • CreateMaskedLoad: when the predicate is a scalar, emit the load as <1 x T> with a <1 x i1> mask and extract element 0. The load instruction is returned, so alias info is still attached to the memory access.
  • CreateMaskedStore: when the predicate is a scalar, store the value as <1 x T> with a <1 x i1> mask.

Tests

Added to tests/python/codegen/llvm/test_basic.py:

  • the reproducer from the issue (indices [0, 2]) now compiles;
  • stride-2 masked load and store run with all four 2-lane masks. Loads check only the active lanes, since inactive lanes are poison. Stores start from -1 sentinels to detect writes to inactive lanes.
  • 4-lane, stride-3 masked load and store with only the last lane active, for float32, int8 and float64. This catches a wrong predicate lane, which the 2-lane tests cannot, and covers the element type and alignment of the <1 x T> access.

tests/python/codegen/llvm on Linux x86_64, compared against main (c4fbd89):

LLVM 20.1.8 LLVM 23.1.1
main 539 passed, 4 failed 543 passed
this PR 554 passed, 4 failed 558 passed

The 4 failures on LLVM 20 are test_aarch64.py::test_min / test_max (float, float16) and are the same on main.

@naye-gg
naye-gg marked this pull request as ready for review November 15, 2026 22:06
Copilot AI balanced review requested due to automatic review settings November 15, 2026 22:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][TIRx][LLVM] Non-unit-stride masked load emits an invalid LLVM intrinsic

4 participants