diff --git a/.gitignore b/.gitignore
index c7c3919..2d70b75 100644
--- a/.gitignore
+++ b/.gitignore
@@ -29,6 +29,7 @@ yarn.lock
pnpm-lock.yaml
# Build artifacts
+dist/
*.tsbuildinfo
.cache/
.parcel-cache/
diff --git a/Dockerfile b/Dockerfile
index d5113f6..62269ec 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -29,7 +29,7 @@ WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates \
&& groupadd -r ds2api && useradd -r -g ds2api -d /app -s /sbin/nologin ds2api \
- && mkdir -p /app/data && chown -R ds2api:ds2api /app \
+ && mkdir -p /app/data /data && chown -R ds2api:ds2api /app /data \
&& rm -rf /var/lib/apt/lists/*
COPY --from=busybox-tools /bin/busybox /usr/local/bin/busybox
EXPOSE 5001
diff --git a/README.MD b/README.MD
index 41e584b..be7698f 100644
--- a/README.MD
+++ b/README.MD
@@ -247,6 +247,7 @@ docker-compose logs -f
默认 `docker-compose.yml` 会把宿主机 `6011` 映射到容器内的 `5001`。如果你希望直接对外暴露 `5001`，请设置 `DS2API_HOST_PORT=5001`（或者手动调整 `ports` 配置）。
同时默认把 `./config.json` 挂载到容器 `/data/config.json`，并设置 `DS2API_CONFIG_PATH=/data/config.json`，用于避免 `/app` 只读导致运行时 token 持久化失败。
+镜像会预创建 `/data` 并授权给非 root 的 `ds2api` 用户；如果使用单文件 bind mount，请确保宿主机 `config.json` 对容器用户可读写，例如 `chmod 644 config.json`。
更新镜像：`docker-compose up -d --build`
diff --git a/docs/DEPLOY.en.md b/docs/DEPLOY.en.md
index 4c1df13..a7716a3 100644
--- a/docs/DEPLOY.en.md
+++ b/docs/DEPLOY.en.md
@@ -131,6 +131,7 @@ docker-compose logs -f
The default `docker-compose.yml` directly uses `ghcr.io/cjackhwang/ds2api:latest` and maps host port `6011` to container port `5001`. If you want `5001` exposed directly, set `DS2API_HOST_PORT=5001` (or adjust the `ports` mapping).
The compose template also defaults to `DS2API_CONFIG_PATH=/data/config.json` with `./config.json:/data/config.json` mounted, so deployments avoid read-only `/app` persistence issues by default.
+The image pre-creates `/data` and grants it to the non-root `ds2api` user. If you bind-mount a single host file, make sure `config.json` is readable/writable by the container user, for example with `chmod 644 config.json`; otherwise Linux UID/GID mismatches can still cause `open /data/config.json: permission denied`.
Compatibility note: when `DS2API_CONFIG_PATH` is unset and runtime base dir is `/app`, newer versions prefer `/data/config.json`; if that file is missing but legacy `/app/config.json` exists, DS2API automatically falls back to the legacy path to avoid post-upgrade config loss.
If you want a pinned version instead of `latest`, you can also pull a specific tag directly:
diff --git a/docs/DEPLOY.md b/docs/DEPLOY.md
index 47dfd4b..3ff20ed 100644
--- a/docs/DEPLOY.md
+++ b/docs/DEPLOY.md
@@ -131,6 +131,7 @@ docker-compose logs -f
默认 `docker-compose.yml` 直接使用 `ghcr.io/cjackhwang/ds2api:latest`，并把宿主机 `6011` 映射到容器内的 `5001`。如果你希望直接对外暴露 `5001`，请设置 `DS2API_HOST_PORT=5001`（或者手动调整 `ports` 配置）。
Compose 模板还会默认设置 `DS2API_CONFIG_PATH=/data/config.json` 并挂载 `./config.json:/data/config.json`，优先避免 `/app` 只读带来的配置持久化问题。
+镜像内会预创建 `/data` 并授权给非 root 的 `ds2api` 用户；如果你使用 bind mount 单文件，请确保宿主机 `config.json` 至少可被容器用户读取/写入，例如 `chmod 644 config.json`，否则 Linux UID/GID 不一致时仍可能出现 `open /data/config.json: permission denied`。
兼容说明：若未设置 `DS2API_CONFIG_PATH` 且运行目录是 `/app`，新版本会优先使用 `/data/config.json`；当该文件不存在但检测到历史 `/app/config.json` 时，会自动回退读取旧路径，避免升级后“配置丢失”。
如需固定版本，也可以直接拉取指定 tag：
diff --git a/docs/toolcall-semantics.md b/docs/toolcall-semantics.md
index 5a3480b..ddef29b 100644
--- a/docs/toolcall-semantics.md
+++ b/docs/toolcall-semantics.md
@@ -26,7 +26,7 @@
```
-这不是原生 DSML 全链路实现。DSML 只作为 prompt 外壳和解析入口别名；进入 parser 前会被归一化成 `` / `` / ``，内部仍以现有 XML 解析语义为准。
+这不是原生 DSML 全链路实现。DSML 主要用于让模型有意识地输出协议标识，隔离普通 XML 语义；进入 parser 前会按固定本地标签名归一化成 `` / `` / ``，内部仍以现有 XML 解析语义为准。
约束：
@@ -39,7 +39,8 @@
兼容修复：
- 如果模型漏掉 opening wrapper，但后面仍输出了一个或多个 invoke 并以 closing wrapper 收尾，Go 解析链路会在解析前补回缺失的 opening wrapper。
-- 如果模型把 DSML 标签里的分隔符 `|` 写漏成空格（例如 `<|DSML tool_calls>` / `<|DSML invoke>` / `<|DSML parameter>`，或无 leading pipe 的 `` 形态），或把 `DSML` 与工具标签名直接黏连（例如 `` / `` / ``），或把最前面的 pipe 误写成全宽竖线（例如 `<｜DSML|tool_calls>` / `<｜DSML|invoke>` / `<｜DSML|parameter>`），Go / Node 会在固定工具标签名范围内归一化；相似但非工具标签名（如 `tool_calls_extra`）仍按普通文本处理。
+- Go / Node 解析层不再枚举每一种 DSML typo。它会把工具标签名前的 `DSML`、管道符 `|` / `｜`、空白、重复 leading `<` 视为可容忍的协议噪声，然后只匹配固定本地标签名 `tool_calls` / `invoke` / `parameter`。例如 ``、`<<|DSML|tool_calls>`、`<|DSML tool_calls>`、``、`<` 都会归一化；相似但非固定标签名（如 `tool_calls_extra`）仍按普通文本处理。
+- 如果模型在固定工具标签名后多输出一个尾部管道符，例如 `<|DSML|tool_calls|` / `<|DSML|invoke|` / `<|DSML|parameter|`，兼容层会把这个尾部 `|` 当作异常标签终止符并补齐缺失的 `>`；如果后面已经有 `>`，也会消费这个多余 `|` 后再归一化。
- 这是一个针对常见模型失误的窄修复，不改变推荐输出格式；prompt 仍要求模型直接输出完整 DSML 外壳。
- 裸 `` / `` 不会被当成“已支持的工具语法”；只有 `tool_calls` wrapper 或可修复的缺失 opening wrapper 才会进入工具调用路径。
@@ -53,7 +54,7 @@
在流式链路中（Go / Node 一致）：
-- DSML `<|DSML|tool_calls>` wrapper、兼容变体（``、`<｜tool_calls>`、`<|tool_calls>`、`<｜DSML|tool_calls>`）、窄容错空格分隔形态（如 `<|DSML tool_calls>`）、黏连形态（如 ``）和 canonical `` wrapper 都会进入结构化捕获
+- DSML `<|DSML|tool_calls>` wrapper、基于固定本地标签名的 DSML 噪声容错形态、尾部管道符形态（如 `<|DSML|tool_calls|`）和 canonical `` wrapper 都会进入结构化捕获
- 如果流里直接从 invoke 开始，但后面补上了 closing wrapper，Go 流式筛分也会按缺失 opening wrapper 的修复路径尝试恢复
- 已识别成功的工具调用不会再次回流到普通文本
- 不符合新格式的块不会执行，并继续按原样文本透传
@@ -94,7 +95,7 @@ node --test tests/node/stream-tool-sieve.test.js
- DSML `<|DSML|tool_calls>` wrapper 正常解析
- legacy canonical `` wrapper 正常解析
-- 别名变体（``、`<｜tool_calls>`、`<|tool_calls>`）、DSML 空格分隔 typo（如 `<|DSML tool_calls>`）和黏连 typo（如 ``）正常解析
+- 固定本地标签名的 DSML 噪声容错形态（如 ``、`<<|DSML|tool_calls>`、`<|DSML tool_calls>`、``、`<`）正常解析
- 混搭标签（DSML wrapper + canonical inner）归一化后正常解析
- 波浪线围栏 `~~~` 内的示例不执行
- 嵌套围栏（4 反引号嵌套 3 反引号）内的示例不执行
diff --git a/internal/deepseek/protocol/constants_shared.json b/internal/deepseek/protocol/constants_shared.json
index 1462280..08666e7 100644
--- a/internal/deepseek/protocol/constants_shared.json
+++ b/internal/deepseek/protocol/constants_shared.json
@@ -2,7 +2,7 @@
"client": {
"name": "DeepSeek",
"platform": "android",
- "version": "2.0.3",
+ "version": "2.0.4",
"android_api_level": "35",
"locale": "zh_CN"
},
@@ -24,4 +24,4 @@
"skip_exact_paths": [
"response/search_status"
]
-}
+}
\ No newline at end of file
diff --git a/internal/js/helpers/stream-tool-sieve/parse_payload.js b/internal/js/helpers/stream-tool-sieve/parse_payload.js
index 090cc77..40911bd 100644
--- a/internal/js/helpers/stream-tool-sieve/parse_payload.js
+++ b/internal/js/helpers/stream-tool-sieve/parse_payload.js
@@ -248,6 +248,9 @@ function replaceDSMLToolMarkupOutsideIgnored(text) {
if (tag) {
if (tag.dsmlLike) {
out += `<${tag.closing ? '/' : ''}${tag.name}${raw.slice(tag.nameEnd, tag.end + 1)}`;
+ if (raw[tag.end] !== '>') {
+ out += '>';
+ }
} else {
out += raw.slice(tag.start, tag.end + 1);
}
@@ -424,31 +427,42 @@ function scanToolMarkupTagAt(text, start) {
}
const lower = raw.toLowerCase();
let i = start + 1;
+ while (i < raw.length && raw[i] === '<') {
+ i += 1;
+ }
const closing = raw[i] === '/';
if (closing) {
i += 1;
}
- let dsmlLike = false;
- if (i < raw.length && isToolMarkupPipe(raw[i])) {
- dsmlLike = true;
- i += 1;
- }
- if (lower.startsWith('dsml', i)) {
- dsmlLike = true;
- i += 'dsml'.length;
- while (i < raw.length && isToolMarkupSeparator(raw[i])) {
- i += 1;
- }
- }
+ const prefix = consumeToolMarkupNamePrefix(raw, lower, i);
+ i = prefix.next;
+ const dsmlLike = prefix.dsmlLike;
const { name, len } = matchToolMarkupName(lower, i);
if (!name) {
return null;
}
- const nameEnd = i + len;
+ const originalNameEnd = i + len;
+ let nameEnd = originalNameEnd;
+ while (nameEnd < raw.length && isToolMarkupPipe(raw[nameEnd])) {
+ nameEnd += 1;
+ }
+ const hasTrailingPipe = nameEnd > originalNameEnd;
if (!hasXmlTagBoundary(raw, nameEnd)) {
return null;
}
- const end = findXmlTagEnd(raw, nameEnd);
+ let end = findXmlTagEnd(raw, nameEnd);
+ if (end < 0) {
+ if (!hasTrailingPipe) {
+ return null;
+ }
+ end = nameEnd - 1;
+ }
+ if (hasTrailingPipe) {
+ const nextLT = raw.indexOf('<', nameEnd);
+ if (nextLT >= 0 && end >= nextLT) {
+ end = nameEnd - 1;
+ }
+ }
if (end < 0) {
return null;
}
@@ -520,37 +534,94 @@ function findPartialToolMarkupStart(text) {
if (lastLT < 0) {
return -1;
}
- const tail = raw.slice(lastLT);
+ const start = includeDuplicateLeadingLessThan(raw, lastLT);
+ const tail = raw.slice(start);
if (tail.includes('>')) {
return -1;
}
- const lowerTail = tail.toLowerCase();
- const candidates = [
- ' 0 && text[out - 1] === '<') {
+ out -= 1;
}
- return -1;
+ return out;
}
function isToolMarkupPipe(ch) {
return ch === '|' || ch === '｜';
}
-function isToolMarkupSeparator(ch) {
- return ch === ' ' || ch === '\t' || ch === '\r' || ch === '\n' || isToolMarkupPipe(ch);
+function isPartialToolMarkupTagPrefix(text) {
+ const raw = toStringSafe(text);
+ if (!raw || raw[0] !== '<' || raw.includes('>')) {
+ return false;
+ }
+ const lower = raw.toLowerCase();
+ let i = 1;
+ while (i < raw.length && raw[i] === '<') {
+ i += 1;
+ }
+ if (i >= raw.length) {
+ return true;
+ }
+ if (raw[i] === '/') {
+ i += 1;
+ }
+ while (i <= raw.length) {
+ if (i === raw.length) {
+ return true;
+ }
+ if (hasToolMarkupNamePrefix(lower.slice(i))) {
+ return true;
+ }
+ if ('dsml'.startsWith(lower.slice(i))) {
+ return true;
+ }
+ const next = consumeToolMarkupNamePrefixOnce(raw, lower, i);
+ if (!next.ok) {
+ return false;
+ }
+ i = next.next;
+ }
+ return false;
+}
+
+function consumeToolMarkupNamePrefix(raw, lower, idx) {
+ let next = idx;
+ let dsmlLike = false;
+ while (true) {
+ const consumed = consumeToolMarkupNamePrefixOnce(raw, lower, next);
+ if (!consumed.ok) {
+ return { next, dsmlLike };
+ }
+ next = consumed.next;
+ dsmlLike = true;
+ }
+}
+
+function consumeToolMarkupNamePrefixOnce(raw, lower, idx) {
+ if (idx < raw.length && isToolMarkupPipe(raw[idx])) {
+ return { next: idx + 1, ok: true };
+ }
+ if (idx < raw.length && [' ', '\t', '\r', '\n'].includes(raw[idx])) {
+ return { next: idx + 1, ok: true };
+ }
+ if (lower.startsWith('dsml', idx)) {
+ return { next: idx + 'dsml'.length, ok: true };
+ }
+ return { next: idx, ok: false };
+}
+
+function hasToolMarkupNamePrefix(lowerTail) {
+ for (const name of TOOL_MARKUP_NAMES) {
+ if (lowerTail.startsWith(name) || name.startsWith(lowerTail)) {
+ return true;
+ }
+ }
+ return false;
}
function matchToolMarkupName(lower, start) {
diff --git a/internal/js/helpers/stream-tool-sieve/tool-keywords.js b/internal/js/helpers/stream-tool-sieve/tool-keywords.js
deleted file mode 100644
index ac47e4e..0000000
--- a/internal/js/helpers/stream-tool-sieve/tool-keywords.js
+++ /dev/null
@@ -1,55 +0,0 @@
-'use strict';
-
-const XML_TOOL_SEGMENT_TAGS = [
- '<|dsml|tool_calls>', '<|dsml|tool_calls\n', '<|dsml|tool_calls ',
- '<｜dsml|tool_calls>', '<｜dsml|tool_calls\n', '<｜dsml|tool_calls ',
- '<|dsml|invoke ', '<|dsml|invoke\n', '<|dsml|invoke\t', '<|dsml|invoke\r',
- '<|dsmltool_calls>', '<|dsmltool_calls\n', '<|dsmltool_calls ',
- '<|dsmlinvoke ', '<|dsmlinvoke\n', '<|dsmlinvoke\t', '<|dsmlinvoke\r',
- '<|dsml tool_calls>', '<|dsml tool_calls\n', '<|dsml tool_calls ',
- '<|dsml invoke ', '<|dsml invoke\n', '<|dsml invoke\t', '<|dsml invoke\r',
- '', '', '', '', '<｜tool_calls\n', '<｜tool_calls ',
- '<｜invoke ', '<｜invoke\n', '<｜invoke\t', '<｜invoke\r',
- '<|tool_calls>', '<|tool_calls\n', '<|tool_calls ',
- '<|invoke ', '<|invoke\n', '<|invoke\t', '<|invoke\r',
- '', '',
- '',
- '',
- '',
- '',
- '',
- '',
- '',
- '',
- '',
-];
-
-module.exports = {
- XML_TOOL_SEGMENT_TAGS,
- XML_TOOL_OPENING_TAGS,
- XML_TOOL_CLOSING_TAGS,
-};
diff --git a/internal/toolcall/toolcalls_dsml.go b/internal/toolcall/toolcalls_dsml.go
index c93e04c..c75702f 100644
--- a/internal/toolcall/toolcalls_dsml.go
+++ b/internal/toolcall/toolcalls_dsml.go
@@ -44,6 +44,9 @@ func rewriteDSMLToolMarkupOutsideIgnored(text string) string {
}
b.WriteString(tag.Name)
b.WriteString(text[tag.NameEnd : tag.End+1])
+ if text[tag.End] != '>' {
+ b.WriteByte('>')
+ }
i = tag.End + 1
continue
}
diff --git a/internal/toolcall/toolcalls_scan.go b/internal/toolcall/toolcalls_scan.go
index 099f73b..f1d3c4a 100644
--- a/internal/toolcall/toolcalls_scan.go
+++ b/internal/toolcall/toolcalls_scan.go
@@ -128,34 +128,39 @@ func scanToolMarkupTagAt(text string, start int) (ToolMarkupTag, bool) {
}
lower := strings.ToLower(text)
i := start + 1
+ for i < len(text) && text[i] == '<' {
+ i++
+ }
closing := false
if i < len(text) && text[i] == '/' {
closing = true
i++
}
- dsmlLike := false
- if next, ok := consumeToolMarkupPipe(text, i); ok {
- dsmlLike = true
- i = next
- }
- if strings.HasPrefix(lower[i:], "dsml") {
- dsmlLike = true
- i += len("dsml")
- for next, ok := consumeToolMarkupSeparator(text, i); ok; next, ok = consumeToolMarkupSeparator(text, i) {
- i = next
- }
- }
+ i, dsmlLike := consumeToolMarkupNamePrefix(lower, text, i)
name, nameLen := matchToolMarkupName(lower, i)
if nameLen == 0 {
return ToolMarkupTag{}, false
}
nameEnd := i + nameLen
+ nameEndBeforePipes := nameEnd
+ for next, ok := consumeToolMarkupPipe(text, nameEnd); ok; next, ok = consumeToolMarkupPipe(text, nameEnd) {
+ nameEnd = next
+ }
+ hasTrailingPipe := nameEnd > nameEndBeforePipes
if !hasToolMarkupBoundary(text, nameEnd) {
return ToolMarkupTag{}, false
}
end := findXMLTagEnd(text, nameEnd)
if end < 0 {
- return ToolMarkupTag{}, false
+ if !hasTrailingPipe {
+ return ToolMarkupTag{}, false
+ }
+ end = nameEnd - 1
+ }
+ if hasTrailingPipe {
+ if nextLT := strings.IndexByte(text[nameEnd:], '<'); nextLT >= 0 && end >= nameEnd+nextLT {
+ end = nameEnd - 1
+ }
}
trimmed := strings.TrimSpace(text[start : end+1])
return ToolMarkupTag{
@@ -171,6 +176,74 @@ func scanToolMarkupTagAt(text string, start int) (ToolMarkupTag, bool) {
}, true
}
+func IsPartialToolMarkupTagPrefix(text string) bool {
+ if text == "" || text[0] != '<' || strings.Contains(text, ">") {
+ return false
+ }
+ lower := strings.ToLower(text)
+ i := 1
+ for i < len(text) && text[i] == '<' {
+ i++
+ }
+ if i >= len(text) {
+ return true
+ }
+ if text[i] == '/' {
+ i++
+ }
+ for i <= len(text) {
+ if i == len(text) {
+ return true
+ }
+ if hasToolMarkupNamePrefix(lower[i:]) {
+ return true
+ }
+ if strings.HasPrefix("dsml", lower[i:]) {
+ return true
+ }
+ next, ok := consumeToolMarkupNamePrefixOnce(lower, text, i)
+ if !ok {
+ return false
+ }
+ i = next
+ }
+ return false
+}
+
+func consumeToolMarkupNamePrefix(lower, text string, idx int) (int, bool) {
+ dsmlLike := false
+ for {
+ next, ok := consumeToolMarkupNamePrefixOnce(lower, text, idx)
+ if !ok {
+ return idx, dsmlLike
+ }
+ idx = next
+ dsmlLike = true
+ }
+}
+
+func consumeToolMarkupNamePrefixOnce(lower, text string, idx int) (int, bool) {
+ if next, ok := consumeToolMarkupPipe(text, idx); ok {
+ return next, true
+ }
+ if idx < len(text) && (text[idx] == ' ' || text[idx] == '\t' || text[idx] == '\r' || text[idx] == '\n') {
+ return idx + 1, true
+ }
+ if strings.HasPrefix(lower[idx:], "dsml") {
+ return idx + len("dsml"), true
+ }
+ return idx, false
+}
+
+func hasToolMarkupNamePrefix(lowerTail string) bool {
+ for _, name := range toolMarkupNames {
+ if strings.HasPrefix(lowerTail, name) || strings.HasPrefix(name, lowerTail) {
+ return true
+ }
+ }
+ return false
+}
+
func matchToolMarkupName(lower string, start int) (string, int) {
for _, name := range toolMarkupNames {
if strings.HasPrefix(lower[start:], name) {
@@ -193,19 +266,6 @@ func consumeToolMarkupPipe(text string, idx int) (int, bool) {
return idx, false
}
-func consumeToolMarkupSeparator(text string, idx int) (int, bool) {
- if idx >= len(text) {
- return idx, false
- }
- if text[idx] == ' ' || text[idx] == '\t' || text[idx] == '\r' || text[idx] == '\n' {
- return idx + 1, true
- }
- if next, ok := consumeToolMarkupPipe(text, idx); ok {
- return next, true
- }
- return idx, false
-}
-
func hasToolMarkupBoundary(text string, idx int) bool {
if idx >= len(text) {
return true
diff --git a/internal/toolcall/toolcalls_test.go b/internal/toolcall/toolcalls_test.go
index b68955b..01de962 100644
--- a/internal/toolcall/toolcalls_test.go
+++ b/internal/toolcall/toolcalls_test.go
@@ -41,6 +41,52 @@ func TestParseToolCallsSupportsDSMLShell(t *testing.T) {
}
}
+func TestParseToolCallsToleratesDSMLTrailingPipeTagTerminator(t *testing.T) {
+ text := strings.Join([]string{
+ `<|DSML|tool_calls| `,
+ ` <|DSML|invoke name="terminal">`,
+ ` <|DSML|parameter name="command">`,
+ ` <|DSML|parameter name="timeout">`,
+ ` `,
+ ``,
+ }, "\n")
+ calls := ParseToolCalls(text, []string{"terminal"})
+ if len(calls) != 1 {
+ t.Fatalf("expected one trailing-pipe DSML call, got %#v", calls)
+ }
+ if calls[0].Name != "terminal" {
+ t.Fatalf("expected terminal tool, got %#v", calls[0])
+ }
+ if calls[0].Input["command"] != `find "/home" -type d` {
+ t.Fatalf("expected command argument, got %#v", calls[0].Input)
+ }
+ if calls[0].Input["timeout"] != float64(10) {
+ t.Fatalf("expected numeric timeout, got %#v", calls[0].Input)
+ }
+}
+
+func TestParseToolCallsToleratesExtraLeadingLessThanBeforeDSML(t *testing.T) {
+ text := `<<|DSML|tool_calls><<|DSML|invoke name="Bash"><<|DSML|parameter name="command">`
+ calls := ParseToolCalls(text, []string{"Bash"})
+ if len(calls) != 1 {
+ t.Fatalf("expected one extra-leading-less-than DSML call, got %#v", calls)
+ }
+ if calls[0].Name != "Bash" || calls[0].Input["command"] != "pwd" {
+ t.Fatalf("unexpected extra-leading-less-than DSML parse result: %#v", calls[0])
+ }
+}
+
+func TestParseToolCallsToleratesRepeatedDSMLPrefixNoise(t *testing.T) {
+ text := `<<<`
+ calls := ParseToolCalls(text, []string{"Bash"})
+ if len(calls) != 1 {
+ t.Fatalf("expected one repeated-prefix DSML call, got %#v", calls)
+ }
+ if calls[0].Name != "Bash" || calls[0].Input["command"] != "git status" {
+ t.Fatalf("unexpected repeated-prefix DSML parse result: %#v", calls[0])
+ }
+}
+
func TestParseToolCallsSupportsDSMLShellWithCanonicalExampleInCDATA(t *testing.T) {
content := `x`
text := `<|DSML|tool_calls><|DSML|invoke name="Write"><|DSML|parameter name="file_path">notes.md<|DSML|parameter name="content">`
diff --git a/internal/toolstream/tool_sieve_core.go b/internal/toolstream/tool_sieve_core.go
index a228c13..afe7c7f 100644
--- a/internal/toolstream/tool_sieve_core.go
+++ b/internal/toolstream/tool_sieve_core.go
@@ -1,10 +1,6 @@
package toolstream
-import (
- "strings"
-
- "ds2api/internal/toolcall"
-)
+import "ds2api/internal/toolcall"
func ProcessChunk(state *State, chunk string, toolNames []string) []Event {
if state == nil {
@@ -174,31 +170,27 @@ func findToolSegmentStart(state *State, s string) int {
if s == "" {
return -1
}
- lower := strings.ToLower(s)
offset := 0
for {
- bestKeyIdx := -1
- matchedTag := ""
- for _, tag := range xmlToolTagsToDetect {
- idx := strings.Index(lower[offset:], tag)
- if idx >= 0 {
- idx += offset
- if bestKeyIdx < 0 || idx < bestKeyIdx {
- bestKeyIdx = idx
- matchedTag = tag
- }
- }
- }
- if bestKeyIdx < 0 {
+ tag, ok := toolcall.FindToolMarkupTagOutsideIgnored(s, offset)
+ if !ok {
return -1
}
- if !insideCodeFenceWithState(state, s[:bestKeyIdx]) {
- return bestKeyIdx
+ start := includeDuplicateLeadingLessThan(s, tag.Start)
+ if !insideCodeFenceWithState(state, s[:start]) {
+ return start
}
- offset = bestKeyIdx + len(matchedTag)
+ offset = tag.End + 1
}
}
+func includeDuplicateLeadingLessThan(s string, idx int) int {
+ for idx > 0 && s[idx-1] == '<' {
+ idx--
+ }
+ return idx
+}
+
func consumeToolCapture(state *State, toolNames []string) (prefix string, calls []toolcall.ParsedToolCall, suffix string, ready bool) {
captured := state.capture.String()
if captured == "" {
diff --git a/internal/toolstream/tool_sieve_xml.go b/internal/toolstream/tool_sieve_xml.go
index f2a5718..8e728e3 100644
--- a/internal/toolstream/tool_sieve_xml.go
+++ b/internal/toolstream/tool_sieve_xml.go
@@ -153,27 +153,14 @@ func findPartialXMLToolTagStart(s string) int {
if lastLT < 0 {
return -1
}
- tail := s[lastLT:]
+ start := includeDuplicateLeadingLessThan(s, lastLT)
+ tail := s[start:]
// If there's a '>' in the tail, the tag is closed — not partial.
if strings.Contains(tail, ">") {
return -1
}
- lowerTail := strings.ToLower(tail)
- for _, tag := range []string{
- "", "", "", "", "", "", "", "", ""}
-
-// xmlToolCallBlockPattern matches a complete canonical XML tool call block.
-//
-//nolint:unused // reserved for future fast-path XML block detection.
-var xmlToolCallBlockPattern = regexp.MustCompile(`(?is)((?:]*>\s*(?:.*?)\s*(?:|))`)
-
-// xmlToolTagsToDetect is the set of XML tag prefixes used by findToolSegmentStart.
-var xmlToolTagsToDetect = []string{
- "<|dsml|tool_calls>", "<|dsml|tool_calls\n", "<|dsml|tool_calls ",
- "<｜dsml|tool_calls>", "<｜dsml|tool_calls\n", "<｜dsml|tool_calls ",
- "<|dsml|invoke ", "<|dsml|invoke\n", "<|dsml|invoke\t", "<|dsml|invoke\r",
- "<|dsmltool_calls>", "<|dsmltool_calls\n", "<|dsmltool_calls ",
- "<|dsmlinvoke ", "<|dsmlinvoke\n", "<|dsmlinvoke\t", "<|dsmlinvoke\r",
- "<|dsml tool_calls>", "<|dsml tool_calls\n", "<|dsml tool_calls ",
- "<|dsml invoke ", "<|dsml invoke\n", "<|dsml invoke\t", "<|dsml invoke\r",
- "", "", "", "", "<｜tool_calls\n", "<｜tool_calls ",
- "<｜invoke ", "<｜invoke\n", "<｜invoke\t", "<｜invoke\r",
- "<|tool_calls>", "<|tool_calls\n", "<|tool_calls ",
- "<|invoke ", "<|invoke\n", "<|invoke\t", "<|invoke\r",
- "", "` + "\n",
+ ` <|DSML|parameter name="command">` + "\n",
+ ` <|DSML|parameter name="timeout">` + "\n",
+ " \n",
+ "",
+ }
+ var events []Event
+ for _, c := range chunks {
+ events = append(events, ProcessChunk(&state, c, []string{"terminal"})...)
+ }
+ events = append(events, Flush(&state, []string{"terminal"})...)
+
+ var textContent strings.Builder
+ var calls []any
+ for _, evt := range events {
+ textContent.WriteString(evt.Content)
+ for _, call := range evt.ToolCalls {
+ calls = append(calls, call)
+ }
+ }
+ if text := textContent.String(); strings.Contains(strings.ToLower(text), "dsml") || strings.Contains(text, "terminal") {
+ t.Fatalf("trailing-pipe DSML tool call leaked to text: %q events=%#v", text, events)
+ }
+ if len(calls) != 1 {
+ t.Fatalf("expected one trailing-pipe DSML tool call, got %d events=%#v", len(calls), events)
+ }
+}
+
+func TestProcessToolSieveInterceptsExtraLeadingLessThanDSMLToolCallWithoutLeak(t *testing.T) {
+ var state State
+ chunks := []string{
+ "<<|DSML|tool_calls>\n",
+ ` <<|DSML|invoke name="Bash">` + "\n",
+ ` <<|DSML|parameter name="command">` + "\n",
+ " \n",
+ "",
+ }
+ var events []Event
+ for _, c := range chunks {
+ events = append(events, ProcessChunk(&state, c, []string{"Bash"})...)
+ }
+ events = append(events, Flush(&state, []string{"Bash"})...)
+
+ var textContent strings.Builder
+ toolCalls := 0
+ for _, evt := range events {
+ textContent.WriteString(evt.Content)
+ toolCalls += len(evt.ToolCalls)
+ }
+ if text := textContent.String(); strings.Contains(text, "<") || strings.Contains(text, "Bash") {
+ t.Fatalf("extra-leading-less-than DSML tool call leaked to text: %q events=%#v", text, events)
+ }
+ if toolCalls != 1 {
+ t.Fatalf("expected one extra-leading-less-than DSML tool call, got %d events=%#v", toolCalls, events)
+ }
+}
+
+func TestProcessToolSieveInterceptsRepeatedDSMLPrefixNoiseWithoutLeak(t *testing.T) {
+ var state State
+ chunks := []string{
+ "<\n",
+ ` <` + "\n",
+ ` <` + "\n",
+ " \n",
+ "",
+ }
+ var events []Event
+ for _, c := range chunks {
+ events = append(events, ProcessChunk(&state, c, []string{"Bash"})...)
+ }
+ events = append(events, Flush(&state, []string{"Bash"})...)
+
+ var textContent strings.Builder
+ toolCalls := 0
+ for _, evt := range events {
+ textContent.WriteString(evt.Content)
+ toolCalls += len(evt.ToolCalls)
+ }
+ if text := textContent.String(); strings.Contains(strings.ToLower(text), "dsml") || strings.Contains(text, "Bash") {
+ t.Fatalf("repeated-prefix DSML tool call leaked to text: %q events=%#v", text, events)
+ }
+ if toolCalls != 1 {
+ t.Fatalf("expected one repeated-prefix DSML tool call, got %d events=%#v", toolCalls, events)
+ }
+}
+
func TestProcessToolSieveHandlesLongXMLToolCall(t *testing.T) {
var state State
const toolName = "write_to_file"
@@ -442,6 +533,8 @@ func TestFindToolSegmentStartDetectsXMLToolCalls(t *testing.T) {
want int
}{
{"tool_calls_tag", "some text \n", 10},
+ {"dsml_trailing_pipe_tag", "some text <|DSML|tool_calls| \n", 10},
+ {"dsml_extra_leading_less_than", "some text <<|DSML|tool_calls>\n", 10},
{"invoke_tag_missing_wrapper", "some text \n", 10},
{"bare_tool_call_text", "prefix \n", -1},
{"xml_inside_code_fence", "```xml\n\n```", -1},
@@ -465,6 +558,8 @@ func TestFindPartialXMLToolTagStart(t *testing.T) {
want int
}{
{"partial_tool_calls", "Hello {
assert.deepEqual(calls[0].input, { path: 'README.MD' });
});
+test('parseToolCalls tolerates DSML trailing pipe tag terminator', () => {
+ const payload = [
+ '<|DSML|tool_calls| ',
+ ' <|DSML|invoke name="terminal">',
+ ' <|DSML|parameter name="command">',
+ ' <|DSML|parameter name="timeout">',
+ ' ',
+ '',
+ ].join('\n');
+ const calls = parseToolCalls(payload, ['terminal']);
+ assert.equal(calls.length, 1);
+ assert.equal(calls[0].name, 'terminal');
+ assert.deepEqual(calls[0].input, { command: 'find "/home" -type d', timeout: 10 });
+});
+
+test('parseToolCalls tolerates extra leading less-than before DSML tags', () => {
+ const payload = [
+ '<<|DSML|tool_calls>',
+ ' <<|DSML|invoke name="Bash">',
+ ' <<|DSML|parameter name="command">',
+ ' ',
+ '',
+ ].join('\n');
+ const calls = parseToolCalls(payload, ['Bash']);
+ assert.equal(calls.length, 1);
+ assert.equal(calls[0].name, 'Bash');
+ assert.deepEqual(calls[0].input, { command: 'pwd' });
+});
+
+test('parseToolCalls tolerates repeated DSML prefix noise', () => {
+ const payload = [
+ '<',
+ ' <',
+ ' <',
+ ' ',
+ '',
+ ].join('\n');
+ const calls = parseToolCalls(payload, ['Bash']);
+ assert.equal(calls.length, 1);
+ assert.equal(calls[0].name, 'Bash');
+ assert.deepEqual(calls[0].input, { command: 'git status' });
+});
+
test('parseToolCalls tolerates DSML space-separator typo', () => {
const payload = '<|DSML tool_calls><|DSML invoke name="Read"><|DSML parameter name="file_path">';
const calls = parseToolCalls(payload, ['Read']);
@@ -285,6 +328,39 @@ test('sieve emits tool_calls for DSML space-separator typo', () => {
assert.equal(text.includes('<|DSML invoke'), false);
});
+test('sieve emits tool_calls for DSML trailing pipe tag terminator', () => {
+ const events = runSieve([
+ '<|DSML|tool_calls| \n',
+ '<|DSML|invoke name="terminal">\n',
+ '<|DSML|parameter name="command">\n',
+ '<|DSML|parameter name="timeout">\n',
+ '\n',
+ '',
+ ], ['terminal']);
+ const finalCalls = events.filter((evt) => evt.type === 'tool_calls').flatMap((evt) => evt.calls || []);
+ const text = collectText(events);
+ assert.equal(finalCalls.length, 1);
+ assert.equal(finalCalls[0].name, 'terminal');
+ assert.deepEqual(finalCalls[0].input, { command: 'find "/home" -type d', timeout: 10 });
+ assert.equal(text.toLowerCase().includes('dsml'), false);
+});
+
+test('sieve emits tool_calls for extra leading less-than DSML tags without leaking prefix', () => {
+ const events = runSieve([
+ '<<|DSML|tool_calls>\n',
+ '<<|DSML|invoke name="Bash">\n',
+ '<<|DSML|parameter name="command">\n',
+ '\n',
+ '',
+ ], ['Bash']);
+ const finalCalls = events.filter((evt) => evt.type === 'tool_calls').flatMap((evt) => evt.calls || []);
+ const text = collectText(events);
+ assert.equal(finalCalls.length, 1);
+ assert.equal(finalCalls[0].name, 'Bash');
+ assert.deepEqual(finalCalls[0].input, { command: 'pwd' });
+ assert.equal(text.includes('<'), false);
+});
+
test('sieve keeps DSML space lookalike tag names as text', () => {
const input = '<|DSML tool_calls_extra><|DSML invoke name="Read"><|DSML parameter name="file_path">/tmp/input.txt';
const events = runSieve([input], ['Read']);