ver4a/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

CUDA: don't convert BF16 weights to FP32 (ggml/1174)

Browse source 
* add bf16 support

* use convert_from_bf16_cuda instead of convert_unary_cuda for f32

* revert 7ec5085

* move functionality into convert_unary with constexpr
This commit is contained in:
Sigbjørn Skjæret 2025-04-04 21:05:12 +02:00 • committed by Georgi Gerganov
parent 995083e4ed
commit 36ca8b3628
3 changed files with 52 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

3
ggml/src/ggml-cuda/convert.cuh
View file

@ -7,7 +7,10 @@ using to_t_cuda_t = void (*)(const void * __restrict__ x, T * __restrict__ y, in
typedef to_t_cuda_t<float> to_fp32_cuda_t;
typedef to_t_cuda_t<half> to_fp16_cuda_t;
typedef to_t_cuda_t<nv_bfloat16> to_bf16_cuda_t;
to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type);
to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type);
to_fp32_cuda_t ggml_get_to_fp32_cuda(ggml_type type);