PR #460 introduced fullwidth pipe characters (｜) in DSML tool call formatting
to improve parsing robustness, but models exposed to these fullwidth pipes in
system prompts exhibit significantly higher rates of tool output hallucinations.
Reverting to halfwidth pipes (|) drastically reduces tokenizer/perplexity-driven
hallucinations while retaining the existing confusable-hardening in the parser.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Detect camelCase→PascalCase boundaries between arbitrary prefixes and fixed
local names (tool_calls/invoke/parameter), so that fused forms like
<DSmartToolCalls> are recognized without explicit separator characters.
Also add the underscore-free alias "toolcalls" as a valid DSML local name.
Includes lookalike rejection tests to ensure near-matches like
<DSmartToolCallsExtra> are not falsely accepted.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the hardcoded isToolMarkupPipe (matching |, ｜, ␂, \x02, !) and
isToolCDATAOpenSeparator (exclusion-based) with a single isToolMarkupSeparator
that treats any Unicode punctuation outside structural characters as a valid
DSML separator. This eliminates the need for a per-character allowlist — novel
separators like ※ are automatically supported without code changes. Also
removes the unused cdataPattern regexp and updates docs to use "non-structural
separator" terminology.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The closing tag format was <｜/DSML｜tag> but must be </｜DSML｜tag>.
The scanner's closing-tag detection checks text[1] == '/', so the
slash must come immediately after '<', before the first full-width
pipe (U+FF5C). Tags like <｜/DSML｜tool_calls> would not set
closing=true and would not match any tool markup name.
Files fixed:
- internal/toolcall/tool_prompt.go: all closing tags
- internal/promptcompat/prompt_build_test.go: 1 test expectation
Replace all strings.ToLower usage with ASCII case-insensitive matching
(hasASCIIPrefixFoldAt, indexASCIIFold, hasDSMLPrefix) to prevent slice
bounds errors when Unicode characters change byte length after case
folding (e.g., Turkish İ U+0130 → i + combining dot: 2 bytes → 3 bytes).
Root cause: code created a strings.ToLower(text) copy, found byte
positions in that copy, then used those positions to slice the
original text — byte offsets that were valid in the lowercased copy
became out-of-bounds in the original when case folding changed byte
lengths.
Files changed:
- toolcalls_scan.go: remove 5 lower usages, add hasDSMLPrefix
- toolcalls_parse_markup.go: remove 3 lower usages, add indexASCIIFold
- toolcalls_markup.go: SanitizeLooseCDATA lower removal
- toolcalls_parse.go: updateCDATAStateForStrip lower removal
- tool_prompt.go: align DSML pipe characters with tool call spec
- tool_prompt_test.go: fix pre-existing test character mismatch
The lower parameter was a footgun: callers had to keep it in sync with the
loop bound over text. Instead, skipXMLIgnoredSection now accepts only text
and constructs strings.ToLower(tail) internally for its prefix checks.
This eliminates the entire class of len(text) vs len(lower) boundary bugs
along with the min() workaround.
Also changes:
- findToolCDATAEnd: drop lower param, use text directly for closeMarker
search (]]> is ASCII, ToLower is a no-op for it)
- cdataEndLooksStructural: drop lower param, use raw text byte comparison
- All external callers: loop bound reverts to plain len(text)
The inner tag-matching functions (findXMLStartTagOutsideCDATA,
findMatchingXMLEndTagOutsideCDATA) retain their own local lower for
HasPrefix comparisons against the target tag name, keeping concerns
properly separated.
Fixes#435.
On Vercel, /var/task is read-only at runtime. ChatHistoryPath() now
auto-detects Vercel via IsVercel() and defaults to /tmp/chat_history.json
when no explicit DS2API_CHAT_HISTORY_PATH is set. Manual env var still
works as explicit override.
http.ServeFile relies on mime.TypeByExtension, which on Windows reads
HKEY_CLASSES_ROOT to resolve the MIME type. Third-party software (some
editors and registry-cleaning tools) can rewrite ".css" to
"application/xml", causing Chrome to refuse the stylesheet and breaking
the /admin panel with a fully unstyled page. The same class of bug
affects ".js" -> "text/plain" in some setups.
Pin the Content-Type by file extension before delegating to ServeFile,
covering the WebUI asset surface (css, js, mjs, html, json, map, svg,
common image and font formats, wasm). Unknown extensions still fall
through to ServeFile's default detection.
Tests cover the pinned types, case-insensitive extension matching, and
the unknown-extension passthrough.
- Stream: strip both and [reference:N] markers to prevent
leaking partial link metadata during incremental output
- Non-stream: convert citation/reference markers to Markdown links for
Claude Messages, Gemini generateContent, and OpenAI Chat/Responses
- Remove StripReferenceMarkers option from call sites; behavior is now
determined automatically by stream vs non-stream context
- Extend JS runtime stripReferenceMarkersText() to also match [citation:N]
- Add tests for streaming marker stripping and non-stream link conversion
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Support `<dsml_tool_calls>`, `<dsml_invoke>`, `<dsml_parameter>` in
addition to the existing pipe, space, hyphen, and collapsed forms.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- ValidateTurn no longer errors on thinking-only responses, deferring to
ShouldRetryEmptyOutput which now also covers thinking-only outputs.
- Empty output retry uses multi-turn follow-up with a regeneration prompt
suffix and parent_message_id in the same DeepSeek session.
- Centralize StripReferenceMarkersEnabled into textclean package to
eliminate duplicated hardcoded booleans across 4 protocol handlers.
- Log a deprecation warning when the legacy "compat" config key is used.
- Document thinking-only retry and reference marker stripping in API.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add compatibility for <dsml-tool-calls>/<dsml-invoke>/<dsml-parameter>
tag forms alongside the canonical pipe-prefixed DSML shell. Hyphenated
forms only activate when a DSML prefix is detected, preventing false
matches on bare XML lookalikes. Go and Node parsers aligned, with tests
covering here-doc CDATA, streaming sieve, and negative lookalike cases.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
