Files
ds2api/internal/util
shern-point bd41c8a90c feat: add tokenizer-based token counting utilities
Use go-tiktoken with embedded vocabularies for accurate BPE token counting. CountPromptTokens applies conservative padding so returned context token counts stay slightly above the real value instead of undercounting.
2026-04-30 00:44:11 +08:00
..
2026-04-27 02:09:11 +08:00
2026-04-26 01:58:15 +08:00
2026-04-27 02:09:11 +08:00