lisone/ds2api
1
0
Fork 0
mirror of https://github.com/CJackHwang/ds2api.git synced 2026-05-05 08:55:28 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
966f21211d59e372dfd4d393531ef7d3c00a2492
ds2api/go.mod
shern-point bd41c8a90c feat: add tokenizer-based token counting utilities 
Use go-tiktoken with embedded vocabularies for accurate BPE token counting. CountPromptTokens applies conservative padding so returned context token counts stay slightly above the real value instead of undercounting.
2026-04-30 00:44:11 +08:00

784 B
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink