lisone/ds2api
1
0
Fork 0
mirror of https://github.com/CJackHwang/ds2api.git synced 2026-05-06 09:25:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
822 Commits 2 Branches 63 Tags
966f21211d59e372dfd4d393531ef7d3c00a2492
Commit Graph

1 Commits

Author SHA1 Message Date
shern-point
bd41c8a90c feat: add tokenizer-based token counting utilities 
Use go-tiktoken with embedded vocabularies for accurate BPE token counting. CountPromptTokens applies conservative padding so returned context token counts stay slightly above the real value instead of undercounting.
 2026-04-30 00:44:11 +08:00