ver4a/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

metal : optimize multi-sequence FA vec kernel (#13493)

Browse source 
* batched-bench : fix pp batch contents

* metal : optimize multi-sequence FA vec kernel

ggml-ci
This commit is contained in:
Georgi Gerganov 2025-05-13 18:04:00 +03:00 • committed by GitHub
parent 4f711afed5
commit c252e0c409
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 5 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

5
ggml/src/ggml-metal/ggml-metal.metal
View file

@ -3887,6 +3887,11 @@ kernel void kernel_flash_attn_ext_vec(
sm[tiisg] = pm[ic + tiisg]; sm[tiisg] = pm[ic + tiisg];
} }
// skip -INF blocks
if (simd_max(sm[tiisg]) == -INFINITY) {
continue;
}
// Q*K^T // Q*K^T
{ {
// each simdgroup processes 1 query and NE (NW/NL) head elements // each simdgroup processes 1 query and NE (NW/NL) head elements