SIGSEGV in Espresso cast_kernel_cpu / vDSP_vflt32 for transformer text encoders with a dynamic sequence dimension (static shape works)

[msluszniak]

Edited 2026-08-20. The original description blamed Core ML's CPU cast kernel and listed the CLIP text tower as affected. Both were wrong. The corrected analysis is below, and a self contained reproducer is in this comment. The original text is preserved at the bottom for reference.

Description

A Core ML model that applies a masked mean-pool over a dynamic sequence dimension segfaults on the first prediction whenever it actually runs on a non-CPU compute unit. The identical package on CPU_ONLY returns correct output, and compiling with a static sequence length also works.

Reproduces from plain coremltools predict, with no ExecuTorch involved.

What triggers it

Bisecting a sentence-transformers wrapper one piece at a time, over RangeDim(1, 382):

wrapper content	result
encoder only (last_hidden_state)	OK
encoder + CLS/SEP prepend	OK
encoder + masked mean-pool	SIGSEGV

The mean-pool is the trigger, not the encoder:

mask = attention_mask.unsqueeze(-1).expand(tokens.size()).float()
pooled = torch.sum(tokens * mask, 1) / torch.clamp(mask.sum(1), min=1e-9)

A ~20 line standalone model doing only this pattern reports, on a non-CPU path:

Espresso exception: "Invalid blob shape": Data-dependent shapes were disabled: var_39 - [?, 768]

Core ML infers the pooled output as data dependent [?, H] even though the reduction removes the dynamic axis and the true shape is [1, H]. In a small graph the backend rejects it and falls back. In a full encoder it segfaults.

Which compute paths fail

all-mpnet-base-v2 plus masked mean-pool, RangeDim(1, 382):

compute_units	precision	result
CPU_ONLY	fp16	OK
CPU_ONLY	fp32	OK
CPU_AND_NE	fp16	SIGSEGV
CPU_AND_NE	fp32	OK, but only because fp32 cannot run on the ANE and falls back to CPU
ALL	fp16	SIGSEGV
ALL	fp32	SIGSEGV

On the CPU path the output is exact against the torch reference: cosine 1.000000, max abs 0.000000 at L = 2, 8, 37, 128, 380.

Reproducing caveat: the compiled model cache silently serves an earlier compilation and will fake a pass. Clear it between runs.

Environment

coremltools 9.0, macOS 15 arm64 (M4), Python 3.10, torch 2.13.0. ExecuTorch 1.4.1 used only to produce the larger reproducer package; the crash itself reproduces without it.

---

Original description (superseded, kept for reference)

The original report attributed the crash to Core ML's CPU cast kernel, based on this stack from the failing execute():

libvDSP.dylib   vDSP_vflt32
Espresso        Espresso::cast_kernel_cpu::__launch(...)
Espresso        Espresso::layer::__launch(...)
Espresso        Espresso::interpreter_t::__launch_function(...)
Espresso        Espresso::net::__forward(...)
Espresso        EspressoLight::espresso_plan::dispatch_task_on_compute_batch(...)
Espresso        espresso_plan_execute_sync
CoreML          -[MLNeuralNetworkEngine executePlan:error:]

It reported all-mpnet-base-v2 and the openai/clip-vit-base-patch32 text tower as both segfaulting with a dynamic sequence dim and working with a static one, and hypothesised that the cause was integer tensors whose extent depends on the symbolic sequence dim (mpnet's relative position buckets, CLIP's causal mask and argmax gather). That hypothesis was explicitly flagged as unconfirmed, and it did not survive bisection: the encoder including that arithmetic is fine, and the CLIP text tower is not affected at all.

[TobyRoseman]

This is likely an issue with the Core ML Framework, not the coremltools Python package.

Can you give us a simple self contained code to reproduce the issue?

[msluszniak]

Thanks for looking. I have a self contained reproducer, and two corrections to my original report.

Correction 1: it is not the CPU cast kernel. The determinant is whether the model actually runs on a non-CPU unit. Only the CPU path is safe.

Correction 2: ExecuTorch is not required. It reproduces through plain coremltools predict, so your instinct that this sits below the Python package looks right.

I also narrowed the trigger. Bisecting the sentence-transformers wrapper one piece at a time over RangeDim(1, 382):

wrapper content	result
encoder only (last_hidden_state)	OK
encoder + CLS/SEP prepend	OK
encoder + masked mean-pool	SIGSEGV

So it is the masked mean-pool, not the encoder.

Minimal self contained repro

No transformers, no ExecuTorch. This one does not segfault, but on a non-CPU path it surfaces what I believe is the underlying complaint, and it is clean on CPU and with static shapes:

Espresso exception: "Invalid blob shape": Data-dependent shapes were disabled: var_39 - [?, 768]

import numpy as np, torch, coremltools as ct
MAXLEN, HID = 382, 768

class MeanPool(torch.nn.Module):
    def forward(self, tokens, attention_mask):
        mask = attention_mask.unsqueeze(-1).expand(tokens.size()).float()
        return torch.sum(tokens * mask, 1) / torch.clamp(mask.sum(1), min=1e-9)

ex = (torch.randn(1, 8, HID), torch.ones(1, 8, dtype=torch.int32))
ts = torch.jit.trace(MeanPool().eval(), ex)
m = ct.convert(
    ts,
    inputs=[
        ct.TensorType(name="tokens",
                      shape=ct.Shape(shape=(1, ct.RangeDim(1, MAXLEN), HID)),
                      dtype=np.float32),
        ct.TensorType(name="attention_mask",
                      shape=ct.Shape(shape=(1, ct.RangeDim(1, MAXLEN))),
                      dtype=np.int32),
    ],
    minimum_deployment_target=ct.target.iOS18,
    compute_precision=ct.precision.FLOAT32,
)
m.save("minimal_dyn.mlpackage")

mm = ct.models.MLModel("minimal_dyn.mlpackage", compute_units=ct.ComputeUnit.CPU_AND_GPU)
print(mm.predict({"tokens": np.random.rand(1, 8, HID).astype(np.float32),
                  "attention_mask": np.ones((1, 8), dtype=np.int32)}))

Core ML infers the pooled output as data dependent [?, 768] even though the reduction removes the dynamic axis and the true shape is [1, 768]. In this small graph the non-CPU backend rejects it and falls back. In a full encoder it segfaults.

Which compute paths fail

Full matrix for all-mpnet-base-v2 plus that same mean-pool, RangeDim(1, 382):

compute_units	precision	result
CPU_ONLY	fp16	OK
CPU_ONLY	fp32	OK
CPU_AND_NE	fp16	SIGSEGV
CPU_AND_NE	fp32	OK, but only because fp32 cannot run on the ANE and falls back to CPU
ALL	fp16	SIGSEGV
ALL	fp32	SIGSEGV

On the CPU path the output is exact against the torch reference, cosine 1.000000 and max abs 0.000000 at L = 2, 8, 37, 128, 380.

Caveat for reproducing: the compiled model cache will silently serve an earlier compilation and fake a pass. I cleared ~/Library/Caches/executorchcoreml between runs. Without that the matrix above is not reproducible.

Reproducer for the actual SIGSEGV

The crashing package is 416 MB so I have not attached it. This builds it, then five lines crash it:

import torch, coremltools as ct, torch.nn.functional as F
from torch.export import Dim, export as texport
from transformers import AutoModel

class Enc(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.m = AutoModel.from_pretrained("sentence-transformers/all-mpnet-base-v2")
    def forward(self, input_ids, attention_mask):
        t = self.m(input_ids=input_ids, attention_mask=attention_mask)[0]
        mask = attention_mask.unsqueeze(-1).expand(t.size()).float()
        return torch.sum(t * mask, 1) / torch.clamp(mask.sum(1), min=1e-9)

ex = (torch.ones(1, 8, dtype=torch.int64), torch.ones(1, 8, dtype=torch.int64))
d = Dim("tokens", min=1, max=382)
ep = texport(Enc().eval(), ex,
             dynamic_shapes={"input_ids": {1: d}, "attention_mask": {1: d}})
# lowered via ExecuTorch's Core ML backend only to obtain the .mlpackage;
# intercept coremltools.convert and .save() the MLModel it returns.

import numpy as np, coremltools as ct
m = ct.models.MLModel("pool.mlpackage", compute_units=ct.ComputeUnit.ALL)
n = [i.name for i in m.get_spec().description.input]
m.predict({k: np.ones((1, 8), dtype=np.int32) for k in n})   # SIGSEGV

Deterministic, 3 of 3 runs, at L = 2, 8 and 382. Switch to ComputeUnit.CPU_ONLY and the identical package returns correct output.

One narrowing on the original report

clip-vit-base-patch32's text tower, which I listed as affected, actually runs correctly on CPU_AND_NE at both precisions. So this looks specific to the mpnet style graph rather than to every model with a dynamic sequence dim.

Environment: coremltools 9.0, macOS 15 arm64 (M4), Python 3.10, torch 2.13.0. ExecuTorch 1.4.1 used only to produce the second package.