refactor: generalize DSML tag parsing to tolerate model noise; split tiktoken by build tags

Replace hardcoded DSML typo variant lists in Go/Node tool call parsers with
generalized prefix consumption that tolerates repeated leading <, repeated DSML
prefix noise, and trailing pipe terminators. Split tiktoken-dependent token
counting into a build-tagged file for non-cgo platform compatibility. Add /data
directory to Dockerfile for bind-mount permissions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
CJACK
2026-05-01 15:17:11 +08:00
parent 2671298439
commit fd0ec29991
21 changed files with 493 additions and 239 deletions

View File

@@ -20,4 +20,3 @@ internal/js/helpers/stream-tool-sieve/sieve-xml.js
internal/js/helpers/stream-tool-sieve/jsonscan.js
internal/js/helpers/stream-tool-sieve/parse.js
internal/js/helpers/stream-tool-sieve/format.js
internal/js/helpers/stream-tool-sieve/tool-keywords.js