Skip to content

activation: Add XPU backend to kernels-community/activation - #1071

Open
jiqing-feng wants to merge 4 commits into
huggingface:mainfrom
jiqing-feng:activation_xpu
Open

activation: Add XPU backend to kernels-community/activation#1071
jiqing-feng wants to merge 4 commits into
huggingface:mainfrom
jiqing-feng:activation_xpu

Conversation

@jiqing-feng

@jiqing-feng jiqing-feng commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a SYCL implementation of the activation kernels so the component can be
used on Intel GPUs. All eleven operators exposed by the CUDA backend are
implemented and registered under the XPU dispatch key, so existing callers
work unchanged on XPU devices.

Previously activation declared only cuda, rocm, metal and cpu. On XPU
the kernel could not be loaded at all, forcing every consumer to carry a
separate eager fallback path.

Operators: silu_and_mul, mul_and_silu, gelu_and_mul, gelu_tanh_and_mul,
fatrelu_and_mul, gelu_new, gelu_fast, gelu_quick, silu, gelu,
gelu_tanh. Supported dtypes: float32, float16, bfloat16.

New sources live in activation_xpu/, following the layout of the existing CPU
backend. tests/kernels/test_activation.py is parameterized over the available
accelerator instead of hard-coding CUDA, so the existing suite runs unmodified
on XPU.

Depends on huggingface/kernels#752, which makes kernel-builder request IEEE
correctly rounded fp32 division for SYCL kernels.

Correctness

Validated on Intel Arc Pro B60 with torch 2.13.0+xpu and oneAPI 2026.0.

The existing tests/kernels/test_activation.py passes in full on XPU (420
cases), including the gated operators that are compared against eager PyTorch
bit-for-bit with zero tolerance. Two cases were added to that file, covering
shapes and alignments that force a vectorized backend onto its scalar path, and
empty inputs. Both are device-parameterized like the rest of the suite.

Performance

Measured on one Intel Arc Pro B60. Speedup is eager time divided by kernel time;
values above 1.00 favour the kernel. Gated operators use a 16384 x 16384
input, element-wise operators 16384 x 8192; the 4096 columns are the same
measurement at 4096 x 8192 and 4096 x 4096.

Operator fp16 bf16 fp32 fp16 (4096) bf16 (4096) fp32 (4096)
silu_and_mul 1.66x 1.66x 1.68x 1.80x 1.67x 1.67x
mul_and_silu 1.66x 1.66x 1.68x 1.66x 1.67x 1.67x
gelu_and_mul 1.66x 1.66x 1.67x 1.68x 1.67x 1.67x
gelu_tanh_and_mul 1.87x 1.64x 1.71x 1.97x 1.69x 1.72x
fatrelu_and_mul 3.37x 3.55x 2.23x 3.40x 3.62x 2.23x
gelu_new 5.40x 3.86x 9.05x 5.19x 3.73x 8.53x
gelu_fast 6.20x 4.48x 10.51x 6.01x 4.35x 10.00x
gelu_quick 3.44x 3.47x 3.42x 3.54x 3.52x 3.49x
silu 0.98x 0.99x 0.98x 1.02x 1.00x 1.00x
gelu 0.98x 0.99x 0.97x 1.01x 1.01x 1.00x
gelu_tanh 1.18x 1.05x 1.02x 1.15x 1.01x 0.95x

Three regimes are visible:

  • Gated operators avoid two [n, d] intermediates and cut memory traffic
    from 6nd to 3nd. fatrelu_and_mul gains more because its eager form
    additionally materializes a boolean mask.
  • Composite activations have no native PyTorch operator and expand into
    eight or more element-wise launches in eager mode, so fusing them gives the
    largest speedups.
  • Single-operator activations read and write the same amount of data as the
    corresponding PyTorch operator and are therefore on par with it. They exist
    for interface completeness, so callers need no device-dependent branch.

Memory

Peak memory allocated while producing the result, via
torch.xpu.max_memory_allocated. The output tensor is included and the input
excluded, so both paths are charged for what they produce. The ratio is
independent of dtype and shape; absolute figures are float16 at
16384 x 16384 (gated) and 16384 x 8192 (element-wise).

Operator eager kernel reduction
silu_and_mul, mul_and_silu, gelu_and_mul, gelu_tanh_and_mul, fatrelu_and_mul 512 MiB 256 MiB 2.00x
gelu_quick 512 MiB 256 MiB 2.00x
gelu_new 768 MiB 256 MiB 3.00x
gelu_fast 1024 MiB 256 MiB 4.00x
silu, gelu, gelu_tanh 256 MiB 256 MiB 1.00x

The fused kernels never allocate beyond the output tensor. Eager gelu_fast
peaks at four times that because three intermediates are live simultaneously.

Build

Both XPU variants offered by kernel-builder build cleanly and pass the
manylinux_2_28 and Python abi3 compatibility checks:

nix build '.#redistributable.torch213-cxx11-xpu20260-x86_64-linux'
nix build '.#redistributable.torch212-cxx11-xpu20253-x86_64-linux'

Limitations

  • Inputs must be contiguous, matching the CUDA backend.

@github-actions github-actions Bot added the chore Version bumps, releases, misc maintenance label Aug 10, 2026
@jiqing-feng
jiqing-feng force-pushed the activation_xpu branch 2 times, most recently from 0854a71 to 8b437ac Compare August 10, 2026 03:43
Implement the eleven activation operators in SYCL and register them under
the XPU dispatch key, so existing callers work unchanged on Intel GPUs.
Previously the kernel declared only cuda, rocm, metal and cpu, and could not
be loaded on XPU at all.

Gated operators map one work-group to one token; element-wise operators use
a flat grid-stride loop. Both use 128-bit vectorized accesses with a runtime
alignment check and a scalar fallback, which is what keeps the element-wise
operators on par with eager PyTorch, whose TensorIterator already vectorizes.

Parameterize tests/kernels/test_activation.py over the available accelerator
rather than hard-coding CUDA, so the existing suite runs unmodified on XPU.
It passes in full on Intel Arc Pro B60 with torch 2.13.0+xpu, including the
gated operators that are compared against eager PyTorch with zero tolerance.
Add two cases to the same file for shapes and alignments that force the
scalar path, and for empty inputs.

Requires huggingface/kernels#752, which makes kernel-builder request IEEE
correctly rounded fp32 division for SYCL kernels; without it silu_and_mul
and mul_and_silu differ from eager PyTorch by one ULP.
@jiqing-feng

Copy link
Copy Markdown
Contributor Author

huggingface/kernels#752 merged. This PR is ready to be reviewed.

# Conflicts:
#	activation/tests/kernels/test_activation.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Version bumps, releases, misc maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant