lisone/ds2api
1
0
Fork 0
mirror of https://github.com/CJackHwang/ds2api.git synced 2026-05-07 09:55:29 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
2f6b5ffda0299d8031ee748dc565a00edaec211f
ds2api/go.mod
shern-point bd41c8a90c feat: add tokenizer-based token counting utilities 
Use go-tiktoken with embedded vocabularies for accurate BPE token counting. CountPromptTokens applies conservative padding so returned context token counts stay slightly above the real value instead of undercounting.
2026-04-30 00:44:11 +08:00

784 B
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink