diff --git a/API.en.md b/API.en.md index 819cbce..19368ec 100644 --- a/API.en.md +++ b/API.en.md @@ -40,7 +40,7 @@ Docs: [Overview](README.en.md) / [Architecture](docs/ARCHITECTURE.en.md) / [Depl - OpenAI / Claude / Gemini protocols are now mounted on one shared `chi` router tree assembled in `internal/server/router.go`. - Adapter responsibilities are streamlined to: **request normalization → DeepSeek invocation → protocol-shaped rendering**, reducing legacy split-logic paths. -- Tool-calling semantics are aligned between Go and Node runtime: models should output the fullwidth-separator DSML shell `<｜DSML｜tool_calls>` → `<｜DSML｜invoke name="...">` → `<｜DSML｜parameter name="...">`; DS2API also accepts the halfwidth DSML wrapper `<|DSML|tool_calls>`, DSML wrapper aliases such as ``, `<|tool_calls>`, `<｜tool_calls>`, common DSML separator drift such as `<|DSML tool_calls>`, collapsed DSML local names such as ``, control-separator drift such as `` / raw STX `\x02`, arbitrary protocol prefixes such as ``, and legacy canonical XML `` → `` → ``. The scanner normalizes fixed local names (`tool_calls` / `invoke` / `parameter`) back to XML before parsing; only wrapped tool blocks or the narrow missing-opening-wrapper repair path enter the tool path, while bare `` does not count as supported syntax. JSON literal parameter bodies are preserved as structured values, explicit empty or whitespace-only parameters are preserved as empty strings, malformed complete wrappers are released as plain text, and loose CDATA is narrowly repaired at final parse/flush when it can preserve a complete outer tool call. +- Tool-calling semantics are aligned between Go and Node runtime: models should output the fullwidth-separator DSML shell `<｜DSML｜tool_calls>` → `<｜DSML｜invoke name="...">` → `<｜DSML｜parameter name="...">`; DS2API also accepts the halfwidth DSML wrapper `<|DSML|tool_calls>`, DSML wrapper aliases such as ``, `<|tool_calls>`, `<｜tool_calls>`, common DSML separator drift such as `<|DSML tool_calls>`, collapsed DSML local names such as ``, control-separator drift such as `` / raw STX `\x02`, CJK angle bracket and trailing attribute separator drift such as `...〈/DSM｜parameter〉`, arbitrary protocol prefixes such as ``, and legacy canonical XML `` → `` → ``. The scanner normalizes fixed local names (`tool_calls` / `invoke` / `parameter`) back to XML before parsing; only wrapped tool blocks or the narrow missing-opening-wrapper repair path enter the tool path, while bare `` does not count as supported syntax. JSON literal parameter bodies are preserved as structured values, explicit empty or whitespace-only parameters are preserved as empty strings, malformed complete wrappers are released as plain text, and loose CDATA is narrowly repaired at final parse/flush when it can preserve a complete outer tool call. - `Admin API` separates static config from runtime policy: `/admin/config*` for configuration state, `/admin/settings*` for runtime behavior. - When upstream returns a thinking-only response with no visible text, the Go main path for both streaming and non-streaming completions retries once in the same DeepSeek session: it appends the prompt suffix `"Previous reply had no visible output. Please regenerate the visible final answer or tool call now."` and sets `parent_message_id`. If that same-account retry would still end as `429 upstream_empty_output`, managed-account mode switches to the next available account, creates a fresh session, and retries the original payload once before returning 429. - Citation/reference marker boundary: streaming output hides upstream `[citation:N]` / `[reference:N]` placeholders by default; non-stream output converts DeepSeek search reference markers into Markdown links. @@ -355,7 +355,7 @@ When `tools` is present, DS2API performs anti-leak handling: Additional notes: -- The parser treats the recommended DSML shell tool blocks (`<｜DSML｜tool_calls>` / `<｜DSML｜invoke name="...">` / `<｜DSML｜parameter name="...">`), halfwidth DSML shell blocks (`<|DSML|tool_calls>` / `<|DSML|invoke name="...">` / `<|DSML|parameter name="...">`), DSML wrapper aliases (``, `<|tool_calls>`, `<｜tool_calls>`), common DSML separator drift (`<|DSML tool_calls>` / `<|DSML invoke>` / `<|DSML parameter>`), collapsed DSML local names (`` / `` / ``), control-separator drift (`` / raw STX `\x02`), arbitrary protocol prefixes (``), and legacy canonical XML tool blocks (`` / `` / ``) as executable tool calls. These shells normalize back to XML first, while internal parsing remains XML-based. Legacy ``, ``, ``, ``, ``, `tool_use`, antml variants, and standalone JSON `tool_calls` payloads are treated as plain text; complete but malformed wrappers are also released as plain text. +- The parser treats the recommended DSML shell tool blocks (`<｜DSML｜tool_calls>` / `<｜DSML｜invoke name="...">` / `<｜DSML｜parameter name="...">`), halfwidth DSML shell blocks (`<|DSML|tool_calls>` / `<|DSML|invoke name="...">` / `<|DSML|parameter name="...">`), DSML wrapper aliases (``, `<|tool_calls>`, `<｜tool_calls>`), common DSML separator drift (`<|DSML tool_calls>` / `<|DSML invoke>` / `<|DSML parameter>`), collapsed DSML local names (`` / `` / ``), control-separator drift (`` / raw STX `\x02`), CJK angle bracket and trailing attribute separator drift (`...〈/DSM｜parameter〉`), arbitrary protocol prefixes (``), and legacy canonical XML tool blocks (`` / `` / ``) as executable tool calls. These shells normalize back to XML first, while internal parsing remains XML-based. Legacy ``, ``, ``, ``, ``, `tool_use`, antml variants, and standalone JSON `tool_calls` payloads are treated as plain text; complete but malformed wrappers are also released as plain text. - The parser no longer drops tool calls solely because parameter values are empty; explicit empty strings or whitespace-only parameters become empty strings in structured `tool_calls`. Prompting still tells the model not to emit blank parameters, and missing/empty argument rejection belongs in the tool executor or client schema validation. - If the final visible response text is empty but the reasoning stream contains an executable tool call, Chat / Responses emits a standard OpenAI `tool_calls` / `function_call` output during finalization. If thinking/reasoning was not enabled by the client, that reasoning text is used only for detection and is not exposed as visible text or `reasoning_content`. - `tool_calls` shown inside fenced markdown code blocks (for example, ```json ... ```) are treated as examples, not executable calls. diff --git a/API.md b/API.md index 8241a02..8d8c827 100644 --- a/API.md +++ b/API.md @@ -40,7 +40,7 @@ - OpenAI / Claude / Gemini 三套协议已统一挂在同一 `chi` 路由树上，由 `internal/server/router.go` 负责装配。 - 适配器层职责收敛为：**请求归一化 → DeepSeek 调用 → 协议形态渲染**，减少历史版本中“同能力多处实现”的分叉。 -- Tool Calling 的解析策略在 Go 与 Node Runtime 间保持一致：推荐模型输出全角分隔符 DSML 外壳 `<｜DSML｜tool_calls>` → `<｜DSML｜invoke name="...">` → `<｜DSML｜parameter name="...">`；兼容层也接受半角 DSML wrapper `<|DSML|tool_calls>`、DSML wrapper 别名 ``、`<|tool_calls>`、`<｜tool_calls>`、常见 DSML 分隔符漏写形态（如 `<|DSML tool_calls>`）、`DSML` 与工具标签名黏连的常见 typo（如 ``）、控制分隔符漂移（如 `` / 原始 STX `\x02`）、任意协议前缀壳（如 ``），以及旧式 canonical XML `` → `` → ``。实现上采用结构扫描：只要固定本地标签名是 `tool_calls` / `invoke` / `parameter`，前缀壳会在解析入口归一化；只有 `tool_calls` wrapper 或可修复的缺失 opening wrapper 会进入工具路径，裸 `` 不计为已支持语法；流式场景继续执行防泄漏筛分。若参数体本身是合法 JSON 字面量（如 `123`、`true`、`null`、数组或对象），会按结构化值输出，不再一律当作字符串；显式空字符串和纯空白参数会结构化保留为空字符串，是否拒绝缺参由工具执行侧决定；完整但 malformed 的 wrapper 会作为普通文本释放，不会吞掉或伪造成工具调用；若 CDATA 偶发漏闭合，则会在最终 parse / flush 恢复阶段做窄修复，尽量保住已完整包裹的外层工具调用。 +- Tool Calling 的解析策略在 Go 与 Node Runtime 间保持一致：推荐模型输出全角分隔符 DSML 外壳 `<｜DSML｜tool_calls>` → `<｜DSML｜invoke name="...">` → `<｜DSML｜parameter name="...">`；兼容层也接受半角 DSML wrapper `<|DSML|tool_calls>`、DSML wrapper 别名 ``、`<|tool_calls>`、`<｜tool_calls>`、常见 DSML 分隔符漏写形态（如 `<|DSML tool_calls>`）、`DSML` 与工具标签名黏连的常见 typo（如 ``）、控制分隔符漂移（如 `` / 原始 STX `\x02`）、CJK 尖括号与属性尾部分隔符漂移（如 `...〈/DSM｜parameter〉`）、任意协议前缀壳（如 ``），以及旧式 canonical XML `` → `` → ``。实现上采用结构扫描：只要固定本地标签名是 `tool_calls` / `invoke` / `parameter`，前缀壳会在解析入口归一化；只有 `tool_calls` wrapper 或可修复的缺失 opening wrapper 会进入工具路径，裸 `` 不计为已支持语法；流式场景继续执行防泄漏筛分。若参数体本身是合法 JSON 字面量（如 `123`、`true`、`null`、数组或对象），会按结构化值输出，不再一律当作字符串；显式空字符串和纯空白参数会结构化保留为空字符串，是否拒绝缺参由工具执行侧决定；完整但 malformed 的 wrapper 会作为普通文本释放，不会吞掉或伪造成工具调用；若 CDATA 偶发漏闭合，则会在最终 parse / flush 恢复阶段做窄修复，尽量保住已完整包裹的外层工具调用。 - `Admin API` 将配置与运行时策略分开：`/admin/config*` 管静态配置，`/admin/settings*` 管运行时行为。 - 当上游返回 thinking-only 响应（模型输出了推理链但无可见文本）时，Go 主路径的流式与非流式补全都会先自动重试一次：以多轮对话 follow-up 方式追加 prompt 后缀 `"Previous reply had no visible output. Please regenerate the visible final answer or tool call now."` 并设置 `parent_message_id` 在同一 DeepSeek session 内让模型重新输出；同账号重试最大 1 次。若同账号重试后仍即将返回 `429 upstream_empty_output`，托管账号模式会在返回 429 前自动切换到下一个可用账号，新建 session，用原始 payload 再 fresh retry 一次。 - 引用标记处理边界：流式输出默认隐藏 `[citation:N]` / `[reference:N]` 这类上游内部占位符；非流式输出默认把 DeepSeek 搜索引用标记转换为 Markdown 引用链接。 @@ -357,7 +357,7 @@ data: [DONE] 补充说明： - **非代码块上下文**下，工具负载即使与普通文本混合，也会按特征识别并产出可执行 tool call（前后普通文本仍可透传）。 -- 解析器当前把推荐 DSML 外壳（`<｜DSML｜tool_calls>` / `<｜DSML｜invoke name="...">` / `<｜DSML｜parameter name="...">`）、半角 DSML 外壳（`<|DSML|tool_calls>` / `<|DSML|invoke name="...">` / `<|DSML|parameter name="...">`）、DSML wrapper 别名（``、`<|tool_calls>`、`<｜tool_calls>`）、常见 DSML 分隔符漏写形态（如 `<|DSML tool_calls>` / `<|DSML invoke>` / `<|DSML parameter>`）、`DSML` 与工具标签名黏连的常见 typo（如 `` / `` / ``）、控制分隔符漂移（如 `` / 原始 STX `\x02`）、任意协议前缀壳（如 ``）和旧式 canonical XML 工具块（`` / `` / ``）作为可执行调用解析；这些前缀壳会先归一化回 XML，内部仍以 XML 解析语义为准。旧式 ``、``、``、``、``、`tool_use`、antml 风格与纯 JSON `tool_calls` 片段默认都会按普通文本处理；完整但 malformed 的 wrapper 同样会作为普通文本释放。 +- 解析器当前把推荐 DSML 外壳（`<｜DSML｜tool_calls>` / `<｜DSML｜invoke name="...">` / `<｜DSML｜parameter name="...">`）、半角 DSML 外壳（`<|DSML|tool_calls>` / `<|DSML|invoke name="...">` / `<|DSML|parameter name="...">`）、DSML wrapper 别名（``、`<|tool_calls>`、`<｜tool_calls>`）、常见 DSML 分隔符漏写形态（如 `<|DSML tool_calls>` / `<|DSML invoke>` / `<|DSML parameter>`）、`DSML` 与工具标签名黏连的常见 typo（如 `` / `` / ``）、控制分隔符漂移（如 `` / 原始 STX `\x02`）、CJK 尖括号与属性尾部分隔符漂移（如 `...〈/DSM｜parameter〉`）、任意协议前缀壳（如 ``）和旧式 canonical XML 工具块（`` / `` / ``）作为可执行调用解析；这些前缀壳会先归一化回 XML，内部仍以 XML 解析语义为准。旧式 ``、``、``、``、``、`tool_use`、antml 风格与纯 JSON `tool_calls` 片段默认都会按普通文本处理；完整但 malformed 的 wrapper 同样会作为普通文本释放。 - 解析层不会因为参数值为空而丢弃工具调用；显式空字符串或纯空白参数会按空字符串进入结构化 `tool_calls`。Prompt 会要求模型不要主动输出空参数，缺参/空命令的拒绝应由工具执行侧或客户端 schema 校验负责。 - 当最终可见正文为空但思维链里包含可执行工具调用时，Chat / Responses 会在收尾阶段补发标准 OpenAI `tool_calls` / `function_call` 输出；如果客户端未开启 thinking / reasoning，该思维链只用于检测，不会作为可见正文或 `reasoning_content` 暴露。 - Markdown fenced code block（例如 ```json ... ```）中的 `tool_calls` 仅视为示例文本，不会被执行。 diff --git a/docs/prompt-compatibility.md b/docs/prompt-compatibility.md index c2a2906..dd8196a 100644 --- a/docs/prompt-compatibility.md +++ b/docs/prompt-compatibility.md @@ -168,7 +168,7 @@ OpenAI Chat / Responses 在标准化后、current input file 之前，会默认 4. 把这整段内容并入 system prompt。 工具调用正例现在优先示范全角分隔符 DSML 风格：`<｜DSML｜tool_calls>` → `<｜DSML｜invoke name="...">` → `<｜DSML｜parameter name="...">`。 -兼容层仍接受旧式纯 `` wrapper，并会容错若干 DSML 标签变体，包括短横线形式 `` / `` / ``、下划线形式 `` / `` / ``，以及其他前缀分隔形态如 `` / `` / ``；标签壳扫描还会把全角 ASCII 漂移归一化，例如 `<ｄＳＭＬ｜tool_calls>` 与全角 `＞` 结束符。更一般地，Go / Node tag 扫描以固定本地标签名 `tool_calls` / `invoke` / `parameter` 为准，标签名前任意协议前缀壳都会在解析入口剥离，例如 ``、`` 这类控制符或非 ASCII 分隔符漂移也会归一化回现有 XML 标签后继续走同一套 parser。但提示词会优先要求模型输出官方 DSML 标签，并强调不能只输出 closing wrapper 而漏掉 opening tag。需要注意：这是“兼容 DSML 外壳，内部仍以 XML 解析语义为准”，不是原生 DSML 全链路实现。解析器会先截获非代码块中的疑似工具 wrapper，完整解析失败或工具语义无效时再按普通文本放行。 +兼容层仍接受旧式纯 `` wrapper，并会容错若干 DSML 标签变体，包括短横线形式 `` / `` / ``、下划线形式 `` / `` / ``，以及其他前缀分隔形态如 `` / `` / ``；标签壳扫描还会把全角 ASCII 漂移归一化，例如 `<ｄＳＭＬ｜tool_calls>` 与全角 `＞` 结束符，也会容错 CJK 尖括号和属性尾部分隔符漂移，例如 `...〈/DSM｜parameter〉`。更一般地，Go / Node tag 扫描以固定本地标签名 `tool_calls` / `invoke` / `parameter` 为准，标签名前任意协议前缀壳都会在解析入口剥离，例如 ``、`` 这类控制符或非 ASCII 分隔符漂移也会归一化回现有 XML 标签后继续走同一套 parser。但提示词会优先要求模型输出官方 DSML 标签，并强调不能只输出 closing wrapper 而漏掉 opening tag。需要注意：这是“兼容 DSML 外壳，内部仍以 XML 解析语义为准”，不是原生 DSML 全链路实现。解析器会先截获非代码块中的疑似工具 wrapper，完整解析失败或工具语义无效时再按普通文本放行。 数组参数使用 `...` 子节点表示；当某个参数体只包含 item 子节点时，Go / Node 解析器会把它还原成数组，避免 `questions` / `options` 这类 schema 中要求 array 的参数被误解析成 `{ "item": ... }` 对象。除此之外，解析器还会回收一些更松散的列表写法，例如 JSON array 字面量或逗号分隔的 JSON 项序列，只要它们足够明确；但 `` 仍然是首选形态。若模型把完整结构化 XML fragment 误包进 CDATA，兼容层会在保护 `content` / `command` 等原文字段的前提下，尝试把非原文字段中的 CDATA XML fragment 还原成 object / array。不过，如果 CDATA 只是单个平面的 XML/HTML 标签，例如 `urgent` 这种行内标记，兼容层会保留原始字符串，不会强行升成 object / array；只有明显表示结构的 CDATA 片段，例如多兄弟节点、嵌套子节点或 `item` 列表，才会触发结构化恢复。对 `command` / `content` 等长文本参数，CDATA 内部的 Markdown fenced DSML / XML 示例会作为原文保护；示例里的 `]]>` 或 `` 不会截断外层工具调用，解析器会继续等待围栏外真正的参数 / wrapper 结束标签。 Go 侧读取 DeepSeek SSE 时不再依赖 `bufio.Scanner` 的固定 2MiB 单行上限；当写文件类工具把很长的 `content` 放在单个 `data:` 行里返回时，非流式收集、流式解析和 auto-continue 透传都会保留完整行，再进入同一套工具解析与序列化流程。 在 assistant 最终回包阶段，如果某个 tool 参数在声明 schema 中明确是 `string`，兼容层会在把解析后的 `tool_calls` / `function_call` 重新序列化成 OpenAI / Responses / Claude 可见参数前，递归把该路径上的 number / bool / object / array 统一转成字符串；其中 object / array 会压成紧凑 JSON 字符串。这个保护只对 schema 明确声明为 string 的路径生效，不会改写本来就是 `number` / `boolean` / `object` / `array` 的参数。这样可以兼容 DeepSeek 输出了结构化片段、但上游客户端工具 schema 又严格要求字符串参数的场景（例如 `content`、`prompt`、`path`、`taskId` 等）。 diff --git a/docs/toolcall-semantics.md b/docs/toolcall-semantics.md index 395f933..f2c148f 100644 --- a/docs/toolcall-semantics.md +++ b/docs/toolcall-semantics.md @@ -39,8 +39,8 @@ 兼容修复： - 如果模型漏掉 opening wrapper，但后面仍输出了一个或多个 invoke 并以 closing wrapper 收尾，Go 解析链路会在解析前补回缺失的 opening wrapper。 -- Go / Node 解析层不再枚举每一种 DSML typo。它以固定本地标签名 `tool_calls` / `invoke` / `parameter` 为准，把标签名前的任意协议前缀壳视为可容忍噪声，并继续兼容管道符 `|` / `｜`、空白、重复 leading `<`、可视控制符 `␂`、原始 STX `\x02`、非 ASCII 分隔符等漂移。例如 ``、`<<|DSML|tool_calls>`、`<|DSML tool_calls>`、``、`<`、``、`` 都会归一化；相似但非固定标签名（如 `tool_calls_extra`）仍按普通文本处理。 -- 如果模型在固定工具标签名后多输出一个尾部管道符，例如 `<|DSML|tool_calls|` / `<|DSML|invoke|` / `<|DSML|parameter|`，兼容层会把这个尾部 `|` 当作异常标签终止符并补齐缺失的 `>`；如果后面已经有 `>`，也会消费这个多余 `|` 后再归一化。 +- Go / Node 解析层不再枚举每一种 DSML typo。它以固定本地标签名 `tool_calls` / `invoke` / `parameter` 为准，把标签名前的任意协议前缀壳视为可容忍噪声，并继续兼容管道符 `|` / `｜`、空白、重复 leading `<`、可视控制符 `␂`、原始 STX `\x02`、非 ASCII 分隔符、CJK 尖括号 `〈` / `〉` 等漂移。例如 ``、`<<|DSML|tool_calls>`、`<|DSML tool_calls>`、``、`<`、``、``、`...〈/DSM｜tool_calls〉` 都会归一化；相似但非固定标签名（如 `tool_calls_extra`）仍按普通文本处理。 +- 如果模型在固定工具标签名后多输出一个尾部管道符，例如 `<|DSML|tool_calls|` / `<|DSML|invoke|` / `<|DSML|parameter|`，或在带属性标签的结束符前多输出一个尾部管道符（如 ``），兼容层会把这个尾部 `|` / `｜` 当作异常标签终止符并补齐或归一化；如果后面已经有 `>` / `〉`，也会消费这个多余分隔符后再归一化。 - 这是一个针对常见模型失误的窄修复，不改变推荐输出格式；prompt 仍要求模型直接输出完整 DSML 外壳。 - 裸 `` / `` 不会被当成“已支持的工具语法”；只有 `tool_calls` wrapper 或可修复的缺失 opening wrapper 才会进入工具调用路径。 @@ -102,7 +102,7 @@ go test -v -run 'TestParseToolCalls|TestProcessToolSieve' ./internal/toolcall ./ - DSML `<｜DSML｜tool_calls>` wrapper 正常解析 - legacy canonical `` wrapper 正常解析 -- 固定本地标签名的 DSML 噪声容错形态（如 ``、`<<|DSML|tool_calls>`、`<|DSML tool_calls>`、``、`<`）正常解析 +- 固定本地标签名的 DSML 噪声容错形态（如 ``、`<<|DSML|tool_calls>`、`<|DSML tool_calls>`、``、`<`、`...〈/DSM｜tool_calls〉`）正常解析 - 混搭标签（DSML wrapper + canonical inner）归一化后正常解析 - 波浪线围栏 `~~~` 内的示例不执行 - 嵌套围栏（4 反引号嵌套 3 反引号）内的示例不执行 diff --git a/internal/js/helpers/stream-tool-sieve/parse_payload.js b/internal/js/helpers/stream-tool-sieve/parse_payload.js index 5e4dcc2..ffd7742 100644 --- a/internal/js/helpers/stream-tool-sieve/parse_payload.js +++ b/internal/js/helpers/stream-tool-sieve/parse_payload.js @@ -1,6 +1,6 @@ 'use strict'; -const CDATA_PATTERN = /^|＞)$/i; +const CDATA_PATTERN = /^(?:<|〈)!\[CDATA\[([\s\S]*?)]](?:>|＞|〉)$/i; const XML_ATTR_PATTERN = /\b([a-z0-9_:-]+)\s*=\s*("([^"]*)"|'([^']*)')/gi; const TOOL_MARKUP_NAMES = [ { raw: 'tool_calls', canonical: 'tool_calls' }, @@ -431,26 +431,31 @@ function skipXmlIgnoredSection(lower, i) { function findCDATAEnd(text, from) { const ascii = text.indexOf(']]>', from); const fullwidth = text.indexOf(']]＞', from); - if (ascii < 0 && fullwidth < 0) { + const cjk = text.indexOf(']]〉', from); + if (ascii < 0 && fullwidth < 0 && cjk < 0) { return { index: -1, len: 0 }; } - if (ascii < 0) { - return { index: fullwidth, len: ']]＞'.length }; + let best = { index: -1, len: 0 }; + for (const candidate of [ + { index: ascii, len: ']]>'.length }, + { index: fullwidth, len: ']]＞'.length }, + { index: cjk, len: ']]〉'.length }, + ]) { + if (candidate.index >= 0 && (best.index < 0 || candidate.index < best.index)) { + best = candidate; + } } - if (fullwidth < 0 || ascii < fullwidth) { - return { index: ascii, len: ']]>'.length }; - } - return { index: fullwidth, len: ']]＞'.length }; + return best; } function scanToolMarkupTagAt(text, start) { const raw = toStringSafe(text); - if (!raw || start < 0 || start >= raw.length || raw[start] !== '<') { + if (!raw || start < 0 || start >= raw.length || normalizeFullwidthASCIIChar(raw[start]) !== '<') { return null; } const lower = raw.toLowerCase(); let i = start + 1; - while (i < raw.length && raw[i] === '<') { + while (i < raw.length && normalizeFullwidthASCIIChar(raw[i]) === '<') { i += 1; } const closing = raw[i] === '/'; @@ -822,6 +827,12 @@ function normalizeFullwidthASCIIChar(ch) { if (!ch) { return ch; } + if (ch === '〈') { + return '<'; + } + if (ch === '〉') { + return '>'; + } const code = ch.charCodeAt(0); if (code >= 0xff01 && code <= 0xff5e) { return String.fromCharCode(code - 0xfee0); @@ -858,9 +869,28 @@ function matchNormalizedASCII(raw, start, expected) { function normalizeToolMarkupTagTailForXML(tail) { let out = ''; - for (const ch of typeof tail === 'string' ? tail : String(tail || '')) { + const raw = typeof tail === 'string' ? tail : String(tail || ''); + let quote = ''; + for (let i = 0; i < raw.length; i += 1) { + const ch = raw[i]; const normalized = normalizeFullwidthASCIIChar(ch); - if (['>', '/', '=', '"', "'"].includes(normalized)) { + if (quote) { + out += normalized; + if (normalized === quote) { + quote = ''; + } + } else if (normalized === '"' || normalized === "'") { + quote = normalized; + out += normalized; + } else if (normalized === '|') { + let j = i + 1; + while (j < raw.length && [' ', '\t', '\r', '\n'].includes(raw[j])) { + j += 1; + } + if (normalizeFullwidthASCIIChar(raw[j] || '') !== '>') { + out += normalized; + } + } else if (['>', '/', '='].includes(normalized)) { out += normalized; } else { out += ch; diff --git a/internal/toolcall/toolcalls_dsml.go b/internal/toolcall/toolcalls_dsml.go index e712b66..e57a5ac 100644 --- a/internal/toolcall/toolcalls_dsml.go +++ b/internal/toolcall/toolcalls_dsml.go @@ -65,6 +65,7 @@ func normalizeToolMarkupTagTailForXML(tail string) string { } var b strings.Builder b.Grow(len(tail)) + quote := rune(0) for i := 0; i < len(tail); { r, size := utf8.DecodeRuneInString(tail[i:]) if r == utf8.RuneError && size == 1 { @@ -72,9 +73,38 @@ func normalizeToolMarkupTagTailForXML(tail string) string { i++ continue } - switch normalizeFullwidthASCII(r) { - case '>', '/', '=', '"', '\'': - b.WriteRune(normalizeFullwidthASCII(r)) + ch := normalizeFullwidthASCII(r) + if quote != 0 { + b.WriteRune(ch) + if ch == quote { + quote = 0 + } + i += size + continue + } + switch ch { + case '"', '\'': + quote = ch + b.WriteRune(ch) + case '|': + j := i + size + for j < len(tail) { + next, nextSize := utf8.DecodeRuneInString(tail[j:]) + if nextSize <= 0 { + break + } + if next == ' ' || next == '\t' || next == '\r' || next == '\n' { + j += nextSize + continue + } + break + } + next, _ := normalizedASCIIAt(tail, j) + if next != '>' { + b.WriteRune(ch) + } + case '>', '/', '=': + b.WriteRune(ch) default: b.WriteString(tail[i : i+size]) } diff --git a/internal/toolcall/toolcalls_markup.go b/internal/toolcall/toolcalls_markup.go index 718b9b0..03a6d6b 100644 --- a/internal/toolcall/toolcalls_markup.go +++ b/internal/toolcall/toolcalls_markup.go @@ -10,7 +10,7 @@ import ( var toolCallMarkupKVPattern = regexp.MustCompile(`(?is)<(?:[a-z0-9_:-]+:)?([a-z0-9_\-.]+)\b[^>]*>(.*?)`) // cdataPattern matches a standalone CDATA section. -var cdataPattern = regexp.MustCompile(`(?is)^|＞)$`) +var cdataPattern = regexp.MustCompile(`(?is)^(?:<|〈)!\[CDATA\[(.*?)]](?:>|＞|〉)$`) func parseMarkupKVObject(text string) map[string]any { matches := toolCallMarkupKVPattern.FindAllStringSubmatch(strings.TrimSpace(text), -1) diff --git a/internal/toolcall/toolcalls_parse_markup.go b/internal/toolcall/toolcalls_parse_markup.go index bee0404..e052e91 100644 --- a/internal/toolcall/toolcalls_parse_markup.go +++ b/internal/toolcall/toolcalls_parse_markup.go @@ -305,19 +305,23 @@ func indexToolCDATAClose(text string, from int) int { } asciiIdx := strings.Index(text[from:], "]]>") fullIdx := strings.Index(text[from:], "]]＞") - if asciiIdx < 0 && fullIdx < 0 { + cjkIdx := strings.Index(text[from:], "]]〉") + if asciiIdx < 0 && fullIdx < 0 && cjkIdx < 0 { return -1 } - if asciiIdx < 0 { - return from + fullIdx + best := -1 + for _, idx := range []int{asciiIdx, fullIdx, cjkIdx} { + if idx >= 0 && (best < 0 || idx < best) { + best = idx + } } - if fullIdx < 0 || asciiIdx < fullIdx { - return from + asciiIdx - } - return from + fullIdx + return from + best } func toolCDATACloseLenAt(text string, idx int) int { + if strings.HasPrefix(text[idx:], "]]〉") { + return len("]]〉") + } if strings.HasPrefix(text[idx:], "]]＞") { return len("]]＞") } diff --git a/internal/toolcall/toolcalls_scan.go b/internal/toolcall/toolcalls_scan.go index ca33ee0..f8001fd 100644 --- a/internal/toolcall/toolcalls_scan.go +++ b/internal/toolcall/toolcalls_scan.go @@ -134,12 +134,17 @@ func FindMatchingToolMarkupClose(text string, open ToolMarkupTag) (ToolMarkupTag } func scanToolMarkupTagAt(text string, start int) (ToolMarkupTag, bool) { - if start < 0 || start >= len(text) || text[start] != '<' { + next, ok := consumeToolMarkupLessThan(text, start) + if !ok { return ToolMarkupTag{}, false } - i := start + 1 - for i < len(text) && text[i] == '<' { - i++ + i := next + for { + next, ok := consumeToolMarkupLessThan(text, i) + if !ok { + break + } + i = next } closing := false if i < len(text) && text[i] == '/' { @@ -459,6 +464,14 @@ func consumeToolMarkupPipe(text string, idx int) (int, bool) { return idx, false } +func consumeToolMarkupLessThan(text string, idx int) (int, bool) { + ch, size := normalizedASCIIAt(text, idx) + if size <= 0 || ch != '<' { + return idx, false + } + return idx + size, true +} + func hasToolMarkupBoundary(text string, idx int) bool { if idx >= len(text) { return true @@ -488,6 +501,12 @@ func normalizedASCIIAt(text string, idx int) (byte, int) { } func normalizeFullwidthASCII(r rune) rune { + switch r { + case '〈': + return '<' + case '〉': + return '>' + } if r >= '！' && r <= '～' { return r - 0xFEE0 } diff --git a/internal/toolcall/toolcalls_test.go b/internal/toolcall/toolcalls_test.go index a47b03d..b66f047 100644 --- a/internal/toolcall/toolcalls_test.go +++ b/internal/toolcall/toolcalls_test.go @@ -132,6 +132,37 @@ func TestParseToolCallsSupportsFullwidthDSMLShell(t *testing.T) { } } +func TestParseToolCallsSupportsCJKAngleDSMDrift(t *testing.T) { + text := ` + +〈![CDATA[Show commits on local dev not on origin/dev]]〉〈/DSM｜parameter〉 +〈![CDATA[git log --oneline origin/dev..dev]]〉〈/DSM｜parameter〉 +〈/DSM｜invoke〉 + +〈![CDATA[Show commits on origin/dev not on local dev]]〉〈/DSM｜parameter〉 +〈![CDATA[git log --oneline dev..origin/dev]]〉〈/DSM｜parameter〉 +〈/DSM｜invoke〉 + +〈![CDATA[Check tracking branch status]]〉〈/DSM｜parameter〉 +〈![CDATA[git status -b --short]]〉〈/DSM｜parameter〉 +〈/DSM｜invoke〉 +〈/DSM｜tool_calls〉` + + calls := ParseToolCalls(text, []string{"Bash"}) + if len(calls) != 3 { + t.Fatalf("expected three CJK-angle DSM drift calls, got %#v", calls) + } + if calls[0].Name != "Bash" || calls[0].Input["command"] != "git log --oneline origin/dev..dev" { + t.Fatalf("unexpected first CJK-angle DSM drift call: %#v", calls[0]) + } + if calls[1].Name != "Bash" || calls[1].Input["description"] != "Show commits on origin/dev not on local dev" { + t.Fatalf("unexpected second CJK-angle DSM drift call: %#v", calls[1]) + } + if calls[2].Name != "Bash" || calls[2].Input["command"] != "git status -b --short" { + t.Fatalf("unexpected third CJK-angle DSM drift call: %#v", calls[2]) + } +} + func TestParseToolCallsIgnoresBareHyphenatedToolCallsLookalike(t *testing.T) { text := `pwd` calls := ParseToolCalls(text, []string{"Bash"}) diff --git a/internal/toolstream/tool_sieve_xml_test.go b/internal/toolstream/tool_sieve_xml_test.go index 6857e9a..e207969 100644 --- a/internal/toolstream/tool_sieve_xml_test.go +++ b/internal/toolstream/tool_sieve_xml_test.go @@ -1228,3 +1228,37 @@ func TestProcessToolSieveDSMLBarePrefixVariantDoesNotLeak(t *testing.T) { t.Fatalf("expected one tool call from DSML bare prefix variant, got %d events=%#v", toolCalls, events) } } + +func TestProcessToolSieveCJKAngleDSMDriftDoesNotLeak(t *testing.T) { + var state State + chunks := []string{ + "\n", + "\n", + "〈![CDATA[Check tracking branch status]]〉〈/DSM｜parameter〉\n", + "〈![CDATA[git status -b --short]]〉〈/DSM｜parameter〉\n", + "〈/DSM｜invoke〉\n", + "〈/DSM｜tool_calls〉", + } + var events []Event + for _, c := range chunks { + events = append(events, ProcessChunk(&state, c, []string{"Bash"})...) + } + events = append(events, Flush(&state, []string{"Bash"})...) + + var textContent string + var calls []toolcall.ParsedToolCall + for _, evt := range events { + textContent += evt.Content + calls = append(calls, evt.ToolCalls...) + } + + if strings.Contains(textContent, "DSM") || strings.Contains(textContent, "git status") { + t.Fatalf("CJK-angle DSM drift leaked to text: %q events=%#v", textContent, events) + } + if len(calls) != 1 { + t.Fatalf("expected one CJK-angle DSM drift tool call, got %d events=%#v", len(calls), events) + } + if calls[0].Name != "Bash" || calls[0].Input["command"] != "git status -b --short" { + t.Fatalf("unexpected CJK-angle DSM drift call: %#v", calls[0]) + } +} diff --git a/tests/node/stream-tool-sieve.test.js b/tests/node/stream-tool-sieve.test.js index 8c6fb0a..e6a07e5 100644 --- a/tests/node/stream-tool-sieve.test.js +++ b/tests/node/stream-tool-sieve.test.js @@ -129,6 +129,29 @@ test('parseToolCalls parses fullwidth DSML shell drift', () => { assert.deepEqual(calls[1].input, { file_path: '/Users/aq/Desktop/myproject/Personal_Blog/index.html' }); }); +test('parseToolCalls parses CJK-angle DSM drift', () => { + const payload = ` + +〈![CDATA[Show commits on local dev not on origin/dev]]〉〈/DSM｜parameter〉 +〈![CDATA[git log --oneline origin/dev..dev]]〉〈/DSM｜parameter〉 +〈/DSM｜invoke〉 + +〈![CDATA[Show commits on origin/dev not on local dev]]〉〈/DSM｜parameter〉 +〈![CDATA[git log --oneline dev..origin/dev]]〉〈/DSM｜parameter〉 +〈/DSM｜invoke〉 + +〈![CDATA[Check tracking branch status]]〉〈/DSM｜parameter〉 +〈![CDATA[git status -b --short]]〉〈/DSM｜parameter〉 +〈/DSM｜invoke〉 +〈/DSM｜tool_calls〉`; + const calls = parseToolCalls(payload, ['Bash']); + assert.equal(calls.length, 3); + assert.equal(calls[0].name, 'Bash'); + assert.equal(calls[0].input.command, 'git log --oneline origin/dev..dev'); + assert.equal(calls[1].input.description, 'Show commits on origin/dev not on local dev'); + assert.equal(calls[2].input.command, 'git status -b --short'); +}); + test('parseToolCalls parses DSML control separator drift', () => { for (const sep of ['␂', '\x02']) { const payload = ` @@ -523,6 +546,22 @@ test('sieve emits tool_calls for arbitrary-prefixed tool tags', () => { assert.equal(text.includes('💥'), false); }); +test('sieve emits tool_calls for CJK-angle DSM drift', () => { + const events = runSieve([ + '\n', + '\n', + '〈![CDATA[Check tracking branch status]]〉〈/DSM｜parameter〉\n', + '〈![CDATA[git status -b --short]]〉〈/DSM｜parameter〉\n', + '〈/DSM｜invoke〉\n', + '〈/DSM｜tool_calls〉', + ], ['Bash']); + const finalCalls = events.flatMap((evt) => (evt.type === 'tool_calls' ? evt.calls : [])); + assert.equal(finalCalls.length, 1); + assert.equal(finalCalls[0].name, 'Bash'); + assert.equal(finalCalls[0].input.command, 'git status -b --short'); + assert.equal(collectText(events), ''); +}); + test('sieve emits all-empty arbitrary-prefixed tool tags without leaking text', () => { const payload = [ '\n',