convert : fix gemma v1 tokenizer convert (#8248)

ggml-ci
This commit is contained in:
Georgi Gerganov 2024-07-04 10:41:03 +03:00 committed by GitHub
parent f619024764
commit 20fc3804bf
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
28 changed files with 85 additions and 4 deletions

View file

@ -40,4 +40,6 @@
56 56 56 56 56 56 56
56 56 56 56 56 56 56 56
56 56 56 56 56 56 56 56 56
72 34269 102 33245 17234 121
1236 12266
353 736 8886 221 10883 4238 16101 28540 222 3822 272 246 327 4434 46 18445 152 46030 45022 142 13878 327 12585 19884 33773 40920 751 46 41839 5954 137 271 3822 137 271 244 56 244 56 56 244 56 56 56 244 56 56 56 56 244 56 56 56 56 56 244 56 56 56 56 56 56 244 56 56 56 56 56 56 56 244 56 56 56 56 56 56 56 56 244 56 51 56 244 56 516 56 244 56 1198 56 244 14566 246 14566 152 14566 265 30428 257 14566 261 30428 248 14566 268 14566 153 14566 277 30428 247 14566 277 14566 133 14566 152 14566 251 36570 247 1037 4995 13379 2924 9515 17823 54 56 54 57 54 58 54 11904 47892 20895 16625 13047 8389 1059 9504 40216 13858 2073 8983 12571 1539 10721 5918 9643 13298 932 31723 31330 9221 3226 35426 10400 457 4783 2602 349 121 1477 957 1200 2038 49 349 632 863 3673 68 349 82 666 3673 457 4650 1949 580 49 349 73 863 2144 1649 35941 68 2726 44 7728 331 44 113 81