lisone/ds2api
1
0
Fork 0
mirror of https://github.com/CJackHwang/ds2api.git synced 2026-05-11 11:47:43 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor backend API structure

Browse Source
This commit is contained in:
CJACK
2026-04-26 06:58:20 +08:00
parent 8a91fef6ab
commit abc96a37d8
207 changed files with 2675 additions and 1344 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
docs/ARCHITECTURE.en.md
View File

@@ -21,28 +21,37 @@ ds2api/
├── docs/ # Project documentation
├── internal/ # Core implementation (non-public packages)
│ ├── account/ # Account pool, inflight slots, waiting queue
│ ├── adapter/ # Multi-protocol adapters
│ │ ├── claude/ # Claude protocol adapter
│ │ ├── gemini/ # Gemini protocol adapter
│ │ └── openai/ # OpenAI adapter and shared execution core
│ ├── admin/ # Admin API (config/accounts/ops)
│ ├── auth/ # Auth/JWT/credential resolution
│ ├── chathistory/ # Server-side conversation history storage/query
│ ├── claudeconv/ # Claude message conversion helpers
│ ├── compat/ # Compatibility and regression helpers
│ ├── config/ # Config loading/validation/hot reload
│ ├── deepseek/ # DeepSeek upstream client capabilities
│ ├── deepseek/ # DeepSeek upstream client/protocol/transport
│ │ ├── client/ # Login/session/completion/upload/delete calls
│ │ ├── protocol/ # DeepSeek URLs, constants, skip path/pattern
│ │ └── transport/ # DeepSeek transport details
│ ├── devcapture/ # Dev capture and troubleshooting
│ ├── format/ # Response formatting layer
│ │ ├── claude/ # Claude output formatting
│ │ └── openai/ # OpenAI output formatting
│ ├── httpapi/ # HTTP surfaces: OpenAI/Claude/Gemini/Admin
│ │ ├── admin/ # Admin API root assembly and resource packages
│ │ ├── claude/ # Claude HTTP protocol adapter
│ │ ├── gemini/ # Gemini HTTP protocol adapter
│ │ └── openai/ # OpenAI HTTP surface
│ │ ├── chat/ # Chat Completions execution entrypoint
│ │ ├── responses/ # Responses API and response store
│ │ ├── files/ # Files API and inline-file preprocessing
│ │ ├── embeddings/ # Embeddings API
│ │ ├── history/ # OpenAI history split
│ │ └── shared/ # OpenAI HTTP errors/models/tool formatting
│ ├── js/ # Node runtime related logic
│ │ ├── chat-stream/ # Node streaming bridge
│ │ ├── helpers/ # JS helper modules
│ │ │ └── stream-tool-sieve/ # JS implementation of tool sieve
│ │ └── shared/ # Shared semantics between Go/Node
│ ├── prompt/ # Prompt composition
│ ├── promptcompat/ # API request -> DeepSeek web-chat plain-text compatibility
│ ├── rawsample/ # Raw sample read/write and management
│ ├── server/ # Router and middleware assembly
│ │ └── data/ # Router/runtime helper data
@@ -51,6 +60,7 @@ ds2api/
│ ├── testsuite/ # Testsuite execution framework
│ ├── textclean/ # Text cleanup
│ ├── toolcall/ # Tool-call parsing and repair
│ ├── toolstream/ # Go streaming tool-call anti-leak and delta detection
│ ├── translatorcliproxy/ # Cross-protocol translation bridge
│ ├── util/ # Shared utility helpers
│ ├── version/ # Version query/compare
@@ -93,33 +103,34 @@ ds2api/
```mermaid
flowchart LR
C[Client/SDK] --> R[internal/server/router.go]
R --> OA[OpenAI Adapter]
R --> CA[Claude Adapter]
R --> GA[Gemini Adapter]
R --> AD[Admin API]
R --> OA[OpenAI HTTP API]
R --> CA[Claude HTTP API]
R --> GA[Gemini HTTP API]
R --> AD[Admin HTTP API]
CA --> BR[translatorcliproxy]
GA --> BR
BR --> CORE[internal/adapter/openai ChatCompletions]
BR --> CORE[internal/httpapi/openai/chat ChatCompletions]
OA --> CORE
CORE --> AUTH[internal/auth + config key/account resolver]
CORE --> POOL[internal/account queue + concurrency]
CORE --> TOOL[internal/toolcall parser + sieve]
CORE --> DS[internal/deepseek client]
CORE --> TOOL[internal/toolcall parser + internal/toolstream sieve]
CORE --> DS[internal/deepseek/client]
DS --> U[DeepSeek upstream]
```
## 3. Responsibilities in `internal/`
- `internal/server`: router tree + middlewares (health, protocol routes, Admin/WebUI).
- `internal/adapter/openai`: shared execution core (chat/responses/embeddings + tool semantics).
- `internal/adapter/{claude,gemini}`: protocol wrappers only (no duplicated upstream execution).
- `internal/httpapi/openai/*`: OpenAI HTTP surface split into chat, responses, files, embeddings, history, and shared packages.
- `internal/httpapi/{claude,gemini}`: protocol wrappers only (no duplicated upstream execution).
- `internal/promptcompat`: compatibility core for turning OpenAI/Claude/Gemini requests into DeepSeek web-chat plain-text context.
- `internal/translatorcliproxy`: structure translation between Claude/Gemini and OpenAI.
- `internal/deepseek`: upstream request/session/PoW/SSE handling.
- `internal/deepseek/{client,protocol,transport}`: upstream requests, sessions, PoW adaptation, protocol constants, and transport details.
- `internal/stream` + `internal/sse`: stream parsing and incremental assembly.
- `internal/toolcall`: canonical XML tool-call parsing + anti-leak sieve (the only executable format is `<tool_calls>` / `<invoke name="...">` / `<parameter name="...">`).
- `internal/admin`: config/accounts/vercel sync/version/dev-capture endpoints.
- `internal/toolcall` + `internal/toolstream`: canonical XML tool-call parsing + anti-leak sieve (the only executable format is `<tool_calls>` / `<invoke name="...">` / `<parameter name="...">`).
- `internal/httpapi/admin/*`: Admin API root assembly plus auth/accounts/config/settings/proxies/rawsamples/vercel/history/devcapture/version resource packages.
- `internal/chathistory`: server-side conversation history persistence, pagination, detail lookup, and retention policy.
- `internal/config`: config loading/validation + runtime settings hot-reload.
- `internal/account`: managed account pool, inflight slots, waiting queue.

47
docs/ARCHITECTURE.md
View File

@@ -21,28 +21,37 @@ ds2api/
├── docs/ # 项目文档目录
├── internal/ # 核心业务实现（不对外暴露）
│ ├── account/ # 账号池、并发槽位、等待队列
│ ├── adapter/ # 多协议适配层
│ │ ├── claude/ # Claude 协议适配
│ │ ├── gemini/ # Gemini 协议适配
│ │ └── openai/ # OpenAI 协议与统一执行核心
│ ├── admin/ # Admin API（配置/账号/运维）
│ ├── auth/ # 鉴权/JWT/凭证解析
│ ├── chathistory/ # 服务器端对话记录存储与查询
│ ├── claudeconv/ # Claude 消息格式转换工具
│ ├── compat/ # 兼容性辅助与回归支持
│ ├── config/ # 配置加载、校验、热更新
│ ├── deepseek/ # DeepSeek 上游客户端能力
│ ├── deepseek/ # DeepSeek 上游 client/protocol/transport
│ │ ├── client/ # 登录、会话、completion、上传/删除等上游调用
│ │ ├── protocol/ # DeepSeek URL、常量、skip path/pattern
│ │ └── transport/ # DeepSeek 传输层细节
│ ├── devcapture/ # 开发抓包与调试采集
│ ├── format/ # 响应格式化层
│ │ ├── claude/ # Claude 输出格式化
│ │ └── openai/ # OpenAI 输出格式化
│ ├── httpapi/ # HTTP surface：OpenAI/Claude/Gemini/Admin
│ │ ├── admin/ # Admin API 根装配与资源子包
│ │ ├── claude/ # Claude HTTP 协议适配
│ │ ├── gemini/ # Gemini HTTP 协议适配
│ │ └── openai/ # OpenAI HTTP surface
│ │ ├── chat/ # Chat Completions 执行入口
│ │ ├── responses/ # Responses API 与 response store
│ │ ├── files/ # Files API 与 inline file 预处理
│ │ ├── embeddings/ # Embeddings API
│ │ ├── history/ # OpenAI history split
│ │ └── shared/ # OpenAI HTTP 公共错误/模型/工具格式
│ ├── js/ # Node Runtime 相关逻辑
│ │ ├── chat-stream/ # Node 流式输出桥接
│ │ ├── helpers/ # JS 辅助函数
│ │ │ └── stream-tool-sieve/ # Tool sieve JS 实现
│ │ └── shared/ # Go/Node 共用语义片段
│ ├── prompt/ # Prompt 组装
│ ├── promptcompat/ # API 请求到 DeepSeek 网页纯文本上下文兼容层
│ ├── rawsample/ # raw sample 读写与管理
│ ├── server/ # 路由与中间件装配
│ │ └── data/ # 路由/运行时辅助数据
@@ -51,6 +60,7 @@ ds2api/
│ ├── testsuite/ # 测试集执行框架
│ ├── textclean/ # 文本清洗
│ ├── toolcall/ # 工具调用解析与修复
│ ├── toolstream/ # Go 流式 tool call 防泄漏与增量检测
│ ├── translatorcliproxy/ # 多协议互转桥
│ ├── util/ # 通用工具函数
│ ├── version/ # 版本查询/比较
@@ -93,33 +103,34 @@ ds2api/
```mermaid
flowchart LR
C[Client/SDK] --> R[internal/server/router.go]
R --> OA[OpenAI Adapter]
R --> CA[Claude Adapter]
R --> GA[Gemini Adapter]
R --> AD[Admin API]
R --> OA[OpenAI HTTP API]
R --> CA[Claude HTTP API]
R --> GA[Gemini HTTP API]
R --> AD[Admin HTTP API]
CA --> BR[translatorcliproxy]
GA --> BR
BR --> CORE[internal/adapter/openai ChatCompletions]
BR --> CORE[internal/httpapi/openai/chat ChatCompletions]
OA --> CORE
CORE --> AUTH[internal/auth + config key/account resolver]
CORE --> POOL[internal/account queue + concurrency]
CORE --> TOOL[internal/toolcall parser + sieve]
CORE --> DS[internal/deepseek client]
CORE --> TOOL[internal/toolcall parser + internal/toolstream sieve]
CORE --> DS[internal/deepseek/client]
DS --> U[DeepSeek upstream]
```
## 3. internal/ 子模块职责
- `internal/server`：路由树和中间件挂载（健康检查、协议入口、Admin/WebUI）。
- `internal/adapter/openai`：统一执行内核（chat/responses/embeddings 与 tool calling 语义）。
- `internal/adapter/{claude,gemini}`：协议输入输出适配，不重复实现上游调用逻辑。
- `internal/httpapi/openai/*`：OpenAI HTTP surface，按 chat、responses、files、embeddings、history、shared 拆分。
- `internal/httpapi/{claude,gemini}`：协议输入输出适配，不重复实现上游调用逻辑。
- `internal/promptcompat`：OpenAI/Claude/Gemini 请求到 DeepSeek 网页纯文本上下文的兼容内核。
- `internal/translatorcliproxy`：Claude/Gemini 与 OpenAI 结构互转。
- `internal/deepseek`：上游请求、会话、PoW、SSE 消费。
- `internal/deepseek/{client,protocol,transport}`：上游请求、会话、PoW 适配、协议常量与传输层。
- `internal/stream` + `internal/sse`：流式解析与增量处理。
- `internal/toolcall`：canonical XML 工具调用解析与防泄漏筛分（唯一可执行格式：`<tool_calls>` / `<invoke name="...">` / `<parameter name="...">`）。
- `internal/admin`：配置管理、账号管理、Vercel 同步、版本检查、开发抓包。
- `internal/toolcall` + `internal/toolstream`：canonical XML 工具调用解析与防泄漏筛分（唯一可执行格式：`<tool_calls>` / `<invoke name="...">` / `<parameter name="...">`）。
- `internal/httpapi/admin/*`：Admin API 根装配与 auth/accounts/config/settings/proxies/rawsamples/vercel/history/devcapture/version 等资源子包。
- `internal/chathistory`：服务器端对话记录持久化、分页、单条详情和保留策略。
- `internal/config`：配置加载、校验、运行时 settings 热更新。
- `internal/account`：托管账号池、并发槽位、等待队列。

4
docs/README.md
View File

@@ -21,7 +21,7 @@
### 文档维护约定
- 文档更新必须以实际代码实现为依据：路由看 `internal/*/handler_routes.go` 与 `internal/admin/handler.go`，配置默认值看 `internal/config/*`，模型/alias 看 `internal/config/models.go`，prompt 兼容链路看 `docs/prompt-compatibility.md` 列出的代码入口。
- 文档更新必须以实际代码实现为依据：总路由装配看 `internal/server/router.go`，协议/resource 路由看 `internal/httpapi/*/**/routes.go` 与 `internal/httpapi/admin/handler.go`，配置默认值看 `internal/config/*`，模型/alias 看 `internal/config/models.go`，prompt 兼容链路看 `docs/prompt-compatibility.md` 列出的代码入口。
- `README.MD` / `README.en.md`：面向首次接触用户，保留“是什么 + 怎么快速跑起来”。
- `docs/ARCHITECTURE*.md`：面向开发者，集中维护项目结构、模块职责与调用链。
- `API*.md`：面向客户端接入者，聚焦接口行为、鉴权和示例。
@@ -51,7 +51,7 @@ Recommended reading order:
### Maintenance conventions
- Documentation updates must be grounded in the actual implementation: routes live in `internal/*/handler_routes.go` and `internal/admin/handler.go`, config defaults in `internal/config/*`, models/aliases in `internal/config/models.go`, and the prompt compatibility pipeline in the code entrypoints listed by `docs/prompt-compatibility.md`.
- Documentation updates must be grounded in the actual implementation: root routing lives in `internal/server/router.go`, protocol/resource routes live in `internal/httpapi/*/**/routes.go` and `internal/httpapi/admin/handler.go`, config defaults in `internal/config/*`, models/aliases in `internal/config/models.go`, and the prompt compatibility pipeline in the code entrypoints listed by `docs/prompt-compatibility.md`.
- `README.MD` / `README.en.md`: onboarding-oriented (“what + quick start”).
- `docs/ARCHITECTURE*.md`: developer-oriented source of truth for module boundaries and execution flow.
- `API*.md`: integration-oriented behavior/contracts.

4
docs/TESTING.md
View File

@@ -209,8 +209,8 @@ go test -v -run TestParseToolCallsWithDeepSeekHallucination ./internal/toolcall/
# 运行 format 相关测试
go test -v ./internal/format/...
# 运行 adapter 相关测试
go test -v ./internal/adapter/openai/...
# 运行 HTTP API 相关测试
go test -v ./internal/httpapi/openai/...
```
### 调试 Tool Call 问题 | Debugging Tool Call Issues

64
docs/prompt-compatibility.md
View File

@@ -53,23 +53,23 @@ DS2API 当前的核心思路，不是把客户端传来的 `messages`、`tools`
对应的关键代码入口：
- OpenAI Chat / Responses：
[internal/adapter/openai/standard_request.go](../internal/adapter/openai/standard_request.go)
[internal/promptcompat/request_normalize.go](../internal/promptcompat/request_normalize.go)
- OpenAI prompt 组装：
[internal/adapter/openai/prompt_build.go](../internal/adapter/openai/prompt_build.go)
[internal/promptcompat/prompt_build.go](../internal/promptcompat/prompt_build.go)
- OpenAI 消息标准化：
[internal/adapter/openai/message_normalize.go](../internal/adapter/openai/message_normalize.go)
[internal/promptcompat/message_normalize.go](../internal/promptcompat/message_normalize.go)
- Claude 标准化：
[internal/adapter/claude/standard_request.go](../internal/adapter/claude/standard_request.go)
[internal/httpapi/claude/standard_request.go](../internal/httpapi/claude/standard_request.go)
- Claude 消息与 tool_use/tool_result 归一：
[internal/adapter/claude/handler_utils.go](../internal/adapter/claude/handler_utils.go)
[internal/httpapi/claude/handler_utils.go](../internal/httpapi/claude/handler_utils.go)
- Gemini 复用 OpenAI prompt builder：
[internal/adapter/gemini/convert_request.go](../internal/adapter/gemini/convert_request.go)
[internal/httpapi/gemini/convert_request.go](../internal/httpapi/gemini/convert_request.go)
- DeepSeek prompt 角色标记拼装：
[internal/prompt/messages.go](../internal/prompt/messages.go)
- prompt 可见 tool history XML：
[internal/prompt/tool_calls.go](../internal/prompt/tool_calls.go)
- completion payload：
[internal/util/standard_request.go](../internal/util/standard_request.go)
[internal/promptcompat/standard_request.go](../internal/promptcompat/standard_request.go)
## 4. 下游真正收到的东西
@@ -96,7 +96,7 @@ DS2API 当前的核心思路，不是把客户端传来的 `messages`、`tools`
- `prompt` 才是对话上下文主载体。
- `ref_file_ids` 只承载文件引用，不承载普通文本消息。
- `tools` 不会作为“原生工具 schema”直接下发给下游，而是被改写进 `prompt`。
- OpenAI Chat / Responses 原生走统一 OpenAI 标准化与 DeepSeek payload 组装；Claude / Gemini 会尽量复用 OpenAI prompt/tool 语义，其中 Gemini 直接复用 `openai.BuildPromptForAdapter`，Claude 消息接口在可代理场景会转换为 OpenAI chat 形态再执行。
- OpenAI Chat / Responses 原生走统一 OpenAI 标准化与 DeepSeek payload 组装；Claude / Gemini 会尽量复用 OpenAI prompt/tool 语义，其中 Gemini 直接复用 `promptcompat.BuildOpenAIPromptForAdapter`，Claude 消息接口在可代理场景会转换为 OpenAI chat 形态再执行。
- 客户端传入的 thinking / reasoning 开关会被归一到下游 `thinking_enabled`。Claude surface 没有 `thinking` 字段时按 Anthropic 语义视为关闭；Gemini `generationConfig.thinkingConfig.thinkingBudget` 会翻译成同一套 thinking 开关；关闭时即使上游返回 `response/thinking_content`，兼容层也不会把它当作可见正文输出。
## 5. prompt 是怎么拼出来的
@@ -148,10 +148,10 @@ DS2API 当前的核心思路，不是把客户端传来的 `messages`、`tools`
4. 把这整段内容并入 system prompt。
OpenAI 路径实现：
[internal/adapter/openai/handler_toolcall_format.go](../internal/adapter/openai/handler_toolcall_format.go)
[internal/promptcompat/tool_prompt.go](../internal/promptcompat/tool_prompt.go)
Claude 路径实现：
[internal/adapter/claude/handler_utils.go](../internal/adapter/claude/handler_utils.go)
[internal/httpapi/claude/handler_utils.go](../internal/httpapi/claude/handler_utils.go)
统一工具调用格式模板：
[internal/toolcall/tool_prompt.go](../internal/toolcall/tool_prompt.go)
@@ -217,9 +217,9 @@ tool / function role 的结果会作为 `<｜Tool｜>...<｜end▁of▁toolresul
OpenAI 文件相关实现：
- inline/base64/data URL 上传：
[internal/adapter/openai/file_inline_upload.go](../internal/adapter/openai/file_inline_upload.go)
[internal/httpapi/openai/files/file_inline_upload.go](../internal/httpapi/openai/files/file_inline_upload.go)
- 文件 ID 收集：
[internal/adapter/openai/file_refs.go](../internal/adapter/openai/file_refs.go)
[internal/promptcompat/file_refs.go](../internal/promptcompat/file_refs.go)
结论：
@@ -237,7 +237,7 @@ OpenAI 文件相关实现：
- 配置访问器：
[internal/config/store_accessors.go](../internal/config/store_accessors.go)
- 历史拆分：
[internal/adapter/openai/history_split.go](../internal/adapter/openai/history_split.go)
[internal/httpapi/openai/history/history_split.go](../internal/httpapi/openai/history/history_split.go)
触发后行为：
@@ -286,7 +286,7 @@ OpenAI 文件相关实现：
- top-level `system` 优先作为系统提示
- `tool_use` / `tool_result` 会被转换成统一的 assistant/tool 历史语义
- `tools` 同样会被并进 system prompt
- 常规执行通过 `internal/adapter/claude/handler_messages.go` 转到 OpenAI chat 路径，模型 alias 会先解析成 DeepSeek 原生模型
- 常规执行通过 `internal/httpapi/claude/handler_messages.go` 转到 OpenAI chat 路径，模型 alias 会先解析成 DeepSeek 原生模型
- 当前代码里没有像 OpenAI 那样完整的 `ref_file_ids` 附件链路
### 10.3 Gemini
@@ -295,7 +295,7 @@ OpenAI 文件相关实现：
- `systemInstruction`、`contents.parts`、`functionCall`、`functionResponse` 会先归一
- tools 会转成 OpenAI 风格 function schema
- prompt 构建复用 OpenAI 的 `BuildPromptForAdapter`
- prompt 构建复用 OpenAI 的 `promptcompat.BuildOpenAIPromptForAdapter`
- 未识别的非文本 part 会被安全序列化进 prompt，并对二进制/疑似 base64 内容做省略或截断处理
也就是说，Gemini 在“最终 prompt 语义”上，尽量和 OpenAI 保持一致。
@@ -348,31 +348,31 @@ OpenAI 文件相关实现：
优先检查这些文件：
- `internal/adapter/openai/standard_request.go`
- `internal/adapter/openai/prompt_build.go`
- `internal/adapter/openai/message_normalize.go`
- `internal/adapter/openai/handler_toolcall_format.go`
- `internal/adapter/openai/file_inline_upload.go`
- `internal/adapter/openai/file_refs.go`
- `internal/adapter/openai/history_split.go`
- `internal/adapter/openai/responses_input_normalize.go`
- `internal/adapter/claude/standard_request.go`
- `internal/adapter/claude/handler_utils.go`
- `internal/adapter/gemini/convert_request.go`
- `internal/adapter/gemini/convert_messages.go`
- `internal/adapter/gemini/convert_tools.go`
- `internal/promptcompat/request_normalize.go`
- `internal/promptcompat/prompt_build.go`
- `internal/promptcompat/message_normalize.go`
- `internal/promptcompat/tool_prompt.go`
- `internal/httpapi/openai/files/file_inline_upload.go`
- `internal/promptcompat/file_refs.go`
- `internal/httpapi/openai/history/history_split.go`
- `internal/promptcompat/responses_input_normalize.go`
- `internal/httpapi/claude/standard_request.go`
- `internal/httpapi/claude/handler_utils.go`
- `internal/httpapi/gemini/convert_request.go`
- `internal/httpapi/gemini/convert_messages.go`
- `internal/httpapi/gemini/convert_tools.go`
- `internal/prompt/messages.go`
- `internal/prompt/tool_calls.go`
- `internal/util/standard_request.go`
- `internal/promptcompat/standard_request.go`
## 13. 建议的最小验证
改动这条链路后，至少补齐或检查这些测试：
- `go test ./internal/prompt/...`
- `go test ./internal/adapter/openai/...`
- `go test ./internal/adapter/claude/...`
- `go test ./internal/adapter/gemini/...`
- `go test ./internal/httpapi/openai/...`
- `go test ./internal/httpapi/claude/...`
- `go test ./internal/httpapi/gemini/...`
- `go test ./internal/util/...`
如果改的是 tool call 相关兼容语义，还应同时检查：

4
docs/toolcall-semantics.md
View File

@@ -1,6 +1,6 @@
# Tool call parsing semantics（Go/Node 统一语义）
本文档描述当前代码中的**实际行为**，以 `internal/toolcall` 与 `internal/js/helpers/stream-tool-sieve` 为准。
本文档描述当前代码中的**实际行为**，以 `internal/toolcall`、`internal/toolstream` 与 `internal/js/helpers/stream-tool-sieve` 为准。
文档导航：[总览](../README.MD) / [架构说明](./ARCHITECTURE.md) / [测试指南](./TESTING.md)
@@ -58,7 +58,7 @@
可直接运行：
```bash
go test -v -run 'TestParseToolCalls|TestProcessToolSieve' ./internal/toolcall ./internal/adapter/openai
go test -v -run 'TestParseToolCalls|TestProcessToolSieve' ./internal/toolcall ./internal/toolstream ./internal/httpapi/openai/...
node --test tests/node/stream-tool-sieve.test.js
```