OpenVINO backend: Support Whisper.cpp

[mostafafaheem]

Overview

• Added support for unmasked attention cases (e.g., cross-attention and encoder self-attention) where src[3] (mask) is nullptr.
• Implemented reshape_flat_kv in flash_attn_ext to reshape and strided-slice 1D flat KV buffers ([1, 1, 1, N]) into multi-head attention inputs [B, n_head, S_k, head_size] based on dynamic/static attention sizes (op_case 1 & 2).
• Relaxed is_supported_flash_attn_pattern checks to accept PERMUTE -> RESHAPE (Whisper Q) and direct VIEW -> base (Whisper K/V) patterns.
• Added translation cases (op_case 5 & 6) using ov::op::v3::ScatterUpdate to write new token embeddings into flat 1D KV cache buffers.
• Added detection cases (cases 4 & 5) in compute_llm_params to extract token_len_per_seq, attention_size, and static cross-attention dimensions (attention_size_static) from direct GGML_OP_VIEW tensors without requiring PERMUTE nodes.
• Fixed output tensor shape resolution in create_ov_output_tensor for GGML_OP_CPY targeting a padded view_src, ensuring the output tensor matches the full underlying view_src buffer dimensions rather than the smaller slice shape.

TODO:

• NPU support
• Performance improvements
• Stateful execution
• Enabling --no-flash-attn

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, used AI to help with debugging (adding intermediate tensor logs, troubleshooting, etc.) and some of the implementation, a lot of the AI generated code was refactored or removed to reduce bloat. Manual review and validation were conducted.

[mostafafaheem]

whisper-bench results on LNL:

OpenVINO

CPU Plugin:

Click to expand

whisper_print_timings:     load time =   141.84 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (     0.00 ms per run)
whisper_print_timings:   encode time =   672.70 ms /     1 runs (   672.70 ms per run)
whisper_print_timings:   decode time =  2783.42 ms /   256 runs (    10.87 ms per run)
whisper_print_timings:   batchd time =   634.63 ms /   320 runs (     1.98 ms per run)
whisper_print_timings:   prompt time =  1396.20 ms /  4096 runs (     0.34 ms per run)
whisper_print_timings:    total time =  5488.13 ms

GPU Plugin

Click to expand

whisper_print_timings:     load time =   140.23 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (     0.00 ms per run)
whisper_print_timings:   encode time =   433.56 ms /     1 runs (   433.56 ms per run)
whisper_print_timings:   decode time =  3115.85 ms /   256 runs (    12.17 ms per run)
whisper_print_timings:   batchd time =   617.21 ms /   320 runs (     1.93 ms per run)
whisper_print_timings:   prompt time =   328.75 ms /  4096 runs (     0.08 ms per run)
whisper_print_timings:    total time =  4496.29 ms

GGML CPU

Click to expand

whisper_print_timings:     load time =   127.03 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (     0.00 ms per run)
whisper_print_timings:   encode time =   566.47 ms /     1 runs (   566.47 ms per run)
whisper_print_timings:   decode time =   679.57 ms /   256 runs (     2.65 ms per run)
whisper_print_timings:   batchd time =   455.48 ms /   320 runs (     1.42 ms per run)
whisper_print_timings:   prompt time =  3138.55 ms /  4096 runs (     0.77 ms per run)
whisper_print_timings:    total time =  4840.81 ms

Vulkan:

Click to expand

whisper_print_timings:     load time =   100.36 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (     0.00 ms per run)
whisper_print_timings:   encode time =    42.16 ms /     1 runs (    42.16 ms per run)
whisper_print_timings:   decode time =   851.30 ms /   256 runs (     3.33 ms per run)
whisper_print_timings:   batchd time =   260.00 ms /   320 runs (     0.81 ms per run)
whisper_print_timings:   prompt time =   217.96 ms /  4096 runs (     0.05 ms per run)
whisper_print_timings:    total time =  1371.77 ms

[mostafafaheem]

@cavusmustafa @ravi9 @wine99 can you please have a look? FYI: this PR is dependent on #261 and ggml-org/whisper.cpp#3997

[wine99]

Looks like OV CPU is much slower than GGML CPU

whisper_print_timings:   decode time =  2783.42 ms /   256 runs (    10.87 ms per run)
vs
whisper_print_timings:   decode time =   679.57 ms /   256 runs (     2.65 ms per run)

Is this the right number to look at?

Is the decoding itself actually slower, or does this timing include compilation time? Is there any graph splitting involved?

[wine99]

LGTM. The changes are mostly additive, so they shouldn’t break existing models. Great job. Thanks! cc @zhaixuejun1993

[mostafafaheem]

Looks like OV CPU is much slower than GGML CPU

whisper_print_timings:   decode time =  2783.42 ms /   256 runs (    10.87 ms per run)
vs
whisper_print_timings:   decode time =   679.57 ms /   256 runs (     2.65 ms per run)

Is this the right number to look at?

Is the decoding itself actually slower, or does this timing include compilation time? Is there any graph splitting involved?

@wine99 This is purely decode time. The model itself consists of 4 ggml_cgraphs (conv, encode, cross-attn, decode), which end up being 4 separate OV models. Only the decode graph runs autoregressively, and it reads from the cross-attention KV cache which might be part of the issue. I did some profiling and found some areas for improvement, like a plugin-emitted Convert node that casts the whole KV cache buffer from f16->f32 on every decode run for all layers. This is responsible for around 20% of the decode graphs execution time, so we might need to convert these buffers to f32 before execution and pass them in f32 instead. If you think this is a correct solution, please let me know and I will submit a pr with the changes. I will look for other areas to improve and report back.

[wine99]

Looks like OV CPU is much slower than GGML CPU

whisper_print_timings:   decode time =  2783.42 ms /   256 runs (    10.87 ms per run)
vs
whisper_print_timings:   decode time =   679.57 ms /   256 runs (     2.65 ms per run)

Is this the right number to look at?
Is the decoding itself actually slower, or does this timing include compilation time? Is there any graph splitting involved?

@wine99 This is purely decode time. The model itself consists of 4 ggml_cgraphs (conv, encode, cross-attn, decode), which end up being 4 separate OV models. Only the decode graph runs autoregressively, and it reads from the cross-attention KV cache which might be part of the issue. I did some profiling and found some areas for improvement, like a plugin-emitted Convert node that casts the whole KV cache buffer from f16->f32 on every decode run for all layers. This is responsible for around 20% of the decode graphs execution time, so we might need to convert these buffers to f32 before execution and pass them in f32 instead. If you think this is a correct solution, please let me know and I will submit a pr with the changes. I will look for other areas to improve and report back.

The conversion of KV cache from f16 to f32 is a known issue. The CPU plugin inserts these conversions because it lacks support for f16 arithmetic. The GPU plugin should not have such conversions. I think GPU performance should have a higher priority than CPU performance, since that’s usually what the team cares about more.

[mostafafaheem]

It is a much bigger issue in whisper since the KV cache is shared across layers, so the whole cache is converted for each layer. Thanks for reviewing, and I will keep you posted with any potential for improvement in performance.