From ad80a57efac3a68db3ccfd3ed0c977bb706f78c2 Mon Sep 17 00:00:00 2001 From: CJACK Date: Sun, 3 May 2026 02:51:19 +0800 Subject: [PATCH] docs: add missing directory entries and package descriptions to architecture docs Fill gaps identified in architecture audit: add artifacts/ and static/ to directory tree, and document 7 auxiliary internal/ packages (textclean, claudeconv, compat, rawsample, devcapture, util, version) in Section 3. Co-Authored-By: Claude Opus 4.7 --- README.MD | 22 +++++----- README.en.md | 22 +++++----- docs/ARCHITECTURE.en.md | 50 ++++++++++++++++------ docs/ARCHITECTURE.md | 50 ++++++++++++++++------ docs/DeepSeekSSE行为结构说明-2026-04-05.md | 11 ++--- docs/prompt-compatibility.md | 5 ++- 6 files changed, 107 insertions(+), 53 deletions(-) diff --git a/README.MD b/README.MD index 2ef0143..60d76ec 100644 --- a/README.MD +++ b/README.MD @@ -76,13 +76,14 @@ flowchart LR subgraph Runtime["运行时核心能力"] Compat["PromptCompat\n(API -> 网页纯文本上下文)"] - Chat["Chat / Responses Runtime\n(统一工具调用与流式语义)"] + Completion["Completion Runtime\n(Session / PoW / Completion)"] + Turn["AssistantTurn\n(输出语义归一)"] Auth["Auth Resolver\n(API key / bearer / x-goog-api-key)"] Pool["Account Pool + Queue\n(并发槽位 + 等待队列)"] DSClient["DeepSeek Client\n(Session / Auth / Completion / Files)"] Pow["PoW 实现\n(纯 Go)"] Tool["Tool Sieve\n(Go/Node 语义对齐)"] - History["History Split\n(长历史文件化)"] + History["Current Input File\n(DS2API_HISTORY.txt)"] end end @@ -94,18 +95,19 @@ flowchart LR OA --> Compat CA & GA --> Compat - Compat --> Chat - Compat -.长历史.-> History - Vercel -.Go prepare.-> Chat + Compat --> Completion + Completion -.完整上下文.-> History + Completion --> Turn + Vercel -.Go prepare.-> Completion Vercel -.Node SSE.-> Tool - Chat --> Auth - Chat -.账号轮询.-> Pool - Chat -.工具调用解析.-> Tool - Chat -.PoW 计算.-> Pow + Completion --> Auth + Completion -.账号轮询.-> Pool + Completion -.工具调用解析.-> Tool + Completion -.PoW 计算.-> Pow Auth --> DSClient DSClient --> Upstream Upstream --> DSClient - Chat --> Client + Turn --> Client Vercel --> Client ``` diff --git a/README.en.md b/README.en.md index f7781ce..75b4848 100644 --- a/README.en.md +++ b/README.en.md @@ -73,13 +73,14 @@ flowchart LR subgraph Runtime["Runtime + Core Capabilities"] Compat["PromptCompat\n(API -> web-chat plain text context)"] - Chat["Chat / Responses Runtime\n(unified tools + stream semantics)"] + Completion["Completion Runtime\n(session / PoW / completion)"] + Turn["AssistantTurn\n(output semantic normalization)"] Auth["Auth Resolver\n(API key / bearer / x-goog-api-key)"] Pool["Account Pool + Queue\n(in-flight slots + wait queue)"] DSClient["DeepSeek Client\n(session / auth / completion / files)"] Pow["PoW Solver\n(Pure Go)"] Tool["Tool Sieve\n(Go/Node semantic parity)"] - History["History Split\n(long history as files)"] + History["Current Input File\n(DS2API_HISTORY.txt)"] end end @@ -91,18 +92,19 @@ flowchart LR OA --> Compat CA & GA --> Compat - Compat --> Chat - Compat -.long history.-> History - Vercel -.Go prepare.-> Chat + Compat --> Completion + Completion -.full context.-> History + Completion --> Turn + Vercel -.Go prepare.-> Completion Vercel -.Node SSE.-> Tool - Chat --> Auth - Chat -.account rotation.-> Pool - Chat -.tool-call parsing.-> Tool - Chat -.PoW solving.-> Pow + Completion --> Auth + Completion -.account rotation.-> Pool + Completion -.tool-call parsing.-> Tool + Completion -.PoW solving.-> Pow Auth --> DSClient DSClient --> Upstream Upstream --> DSClient - Chat --> Client + Turn --> Client Vercel --> Client ``` diff --git a/docs/ARCHITECTURE.en.md b/docs/ARCHITECTURE.en.md index c7ab027..ae06a46 100644 --- a/docs/ARCHITECTURE.en.md +++ b/docs/ARCHITECTURE.en.md @@ -15,6 +15,7 @@ ds2api/ │ └── workflows/ # GitHub Actions workflows ├── api/ # Serverless entrypoints (Vercel Go/Node) ├── app/ # Application-level handler assembly +├── artifacts/ # Debug artifacts (raw-stream-sim, stream-debug, etc.) ├── cmd/ # Executable entrypoints │ ├── ds2api/ # Main service bootstrap │ └── ds2api-tests/ # E2E testsuite CLI bootstrap @@ -40,13 +41,14 @@ ds2api/ │ │ ├── admin/ # Admin API root assembly and resource packages │ │ ├── claude/ # Claude HTTP protocol adapter │ │ ├── gemini/ # Gemini HTTP protocol adapter -│ │ └── openai/ # OpenAI HTTP surface -│ │ ├── chat/ # Chat Completions execution entrypoint -│ │ ├── responses/ # Responses API and response store -│ │ ├── files/ # Files API and inline-file preprocessing -│ │ ├── embeddings/ # Embeddings API -│ │ ├── history/ # OpenAI context file handling -│ │ └── shared/ # OpenAI HTTP errors/models/tool formatting +│ │ ├── openai/ # OpenAI HTTP surface +│ │ │ ├── chat/ # Chat Completions execution entrypoint +│ │ │ ├── responses/ # Responses API and response store +│ │ │ ├── files/ # Files API and inline-file preprocessing +│ │ │ ├── embeddings/ # Embeddings API +│ │ │ ├── history/ # OpenAI context file handling +│ │ │ └── shared/ # OpenAI HTTP errors/models/tool formatting +│ │ └── requestbody/ # HTTP body reading and UTF-8/JSON validation helpers │ ├── js/ # Node runtime related logic │ │ ├── chat-stream/ # Node streaming bridge │ │ ├── helpers/ # JS helper modules @@ -70,6 +72,7 @@ ds2api/ ├── plans/ # Stage plans and manual QA records ├── pow/ # PoW standalone implementation + benchmarks ├── scripts/ # Build/release helper scripts +├── static/ # Build artifacts (admin static resources) ├── tests/ # Test assets and scripts │ ├── compat/ # Compatibility fixtures + expected outputs │ │ ├── expected/ # Expected output samples @@ -78,9 +81,9 @@ ds2api/ │ │ └── toolcalls/ # Tool-call fixtures │ ├── node/ # Node unit tests │ ├── raw_stream_samples/ # Upstream raw SSE samples -│ │ ├── content-filter-trigger-20260405-jwt3/ # Content-filter terminal sample │ │ ├── continue-thinking-snapshot-replay-20260405/ # Continue-thinking sample -│ │ ├── guangzhou-weather-reasoner-search-20260404/ # Search/reference sample +│ │ ├── longtext-deepseek-v4-flash-20260429/ # Flash long-text/file-upload sample +│ │ ├── longtext-deepseek-v4-pro-20260429/ # Pro long-text/file-upload sample │ │ ├── markdown-format-example-20260405/ # Markdown sample │ │ └── markdown-format-example-20260405-spacefix/ # Space-fix sample │ ├── scripts/ # Test entry scripts @@ -93,6 +96,8 @@ ds2api/ ├── features/ # Feature modules │ ├── account/ # Account management page │ ├── apiTester/ # API tester page + │ ├── chatHistory/ # Server-side conversation history page + │ ├── proxy/ # Proxy management page │ ├── settings/ # Settings page │ └── vercel/ # Vercel sync page ├── layout/ # Layout components @@ -126,8 +131,11 @@ flowchart LR subgraph RUNTIME[Shared runtime] AUTH[internal/auth] POOL[internal/account queue + concurrency] + CR[internal/completionruntime] + TURN[internal/assistantturn] STREAM[internal/stream + internal/sse] TOOL[internal/toolcall + internal/toolstream] + FMT[internal/format/openai + claude] DS[internal/deepseek/client] POW[pow + internal/deepseek/protocol] end @@ -153,16 +161,24 @@ flowchart LR PC --> PROMPT PC -.long history.-> HIST PC --> AUTH + PC --> CR NCS -.Go prepare/release.-> CHAT NCS --> JS JS --> TOOL AUTH --> POOL - CHAT --> STREAM - RESP --> STREAM + CHAT --> CR + RESP --> CR + CA --> CR + GA --> CR + CR --> DS + CR --> STREAM + CR --> TURN + STREAM --> TURN STREAM --> TOOL - POOL --> DS + TURN --> FMT + POOL --> CR DS --> POW DS --> U[DeepSeek upstream] ``` @@ -171,7 +187,8 @@ flowchart LR - `internal/server`: router tree + middlewares (health, protocol routes, Admin/WebUI). - `internal/httpapi/openai/*`: OpenAI HTTP surface split into chat, responses, files, embeddings, history, and shared packages; chat/responses share the promptcompat, stream, and toolcall semantics. -- `internal/httpapi/{claude,gemini}`: protocol wrappers that normalize into the same prompt compatibility semantics without duplicating upstream execution. +- `internal/httpapi/{claude,gemini}`: protocol adapters that normalize into the same prompt compatibility semantics; direct paths share DeepSeek session/PoW/completion execution through `completionruntime`, while Vercel/proxy paths can still translate through `translatorcliproxy` into the OpenAI handler. +- `internal/httpapi/requestbody`: shared HTTP body reading, JSON pre-validation, and UTF-8 error helpers across protocol adapters. - `internal/promptcompat`: compatibility core for turning OpenAI/Claude/Gemini requests into DeepSeek web-chat plain-text context. - `internal/assistantturn`: Go output-side canonical semantics, converting DeepSeek SSE collection results and stream finalization state into assistant turns and centralizing thinking, tool call, citation, usage, stop/error behavior. - `internal/completionruntime`: shared Go completion execution helpers for DeepSeek session/PoW/call startup, non-stream collection, and empty-output retry; streaming paths use it to start upstream requests, continue to use `internal/stream` for real-time consumption, and use `assistantturn` during finalization. @@ -184,6 +201,13 @@ flowchart LR - `internal/chathistory`: server-side conversation history persistence, pagination, detail lookup, and retention policy. - `internal/config`: config loading/validation + runtime settings hot-reload. - `internal/account`: managed account pool, inflight slots, waiting queue. +- `internal/textclean`: text cleanup helpers, e.g. stripping `[reference: N]` markers. +- `internal/claudeconv`: Claude API request to DeepSeek format conversion. +- `internal/compat`: compatibility regression tests using SSE fixtures to verify output consistency. +- `internal/rawsample`: upstream raw response capture, read/write, and management. +- `internal/devcapture`: developer debug capture, storing HTTP request/response for troubleshooting. +- `internal/util`: cross-package utilities including JSON writing, type conversion, token counting, thinking parsing, etc. +- `internal/version`: version query and comparison, supporting build-time injection and runtime resolution. ## 4. WebUI Runtime Relation diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a7e7369..61557d4 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -15,6 +15,7 @@ ds2api/ │ └── workflows/ # GitHub Actions 工作流 ├── api/ # Serverless 入口（Vercel Go/Node） ├── app/ # 应用级 handler 装配层 +├── artifacts/ # 调试产物（raw-stream-sim, stream-debug 等） ├── cmd/ # 可执行程序入口 │ ├── ds2api/ # 主服务启动入口 │ └── ds2api-tests/ # E2E 测试集 CLI 入口 @@ -40,13 +41,14 @@ ds2api/ │ │ ├── admin/ # Admin API 根装配与资源子包 │ │ ├── claude/ # Claude HTTP 协议适配 │ │ ├── gemini/ # Gemini HTTP 协议适配 -│ │ └── openai/ # OpenAI HTTP surface -│ │ ├── chat/ # Chat Completions 执行入口 -│ │ ├── responses/ # Responses API 与 response store -│ │ ├── files/ # Files API 与 inline file 预处理 -│ │ ├── embeddings/ # Embeddings API -│ │ ├── history/ # OpenAI context file handling -│ │ └── shared/ # OpenAI HTTP 公共错误/模型/工具格式 +│ │ ├── openai/ # OpenAI HTTP surface +│ │ │ ├── chat/ # Chat Completions 执行入口 +│ │ │ ├── responses/ # Responses API 与 response store +│ │ │ ├── files/ # Files API 与 inline file 预处理 +│ │ │ ├── embeddings/ # Embeddings API +│ │ │ ├── history/ # OpenAI context file handling +│ │ │ └── shared/ # OpenAI HTTP 公共错误/模型/工具格式 +│ │ └── requestbody/ # HTTP 请求体读取与 UTF-8/JSON 校验辅助 │ ├── js/ # Node Runtime 相关逻辑 │ │ ├── chat-stream/ # Node 流式输出桥接 │ │ ├── helpers/ # JS 辅助函数 @@ -70,6 +72,7 @@ ds2api/ ├── plans/ # 阶段计划与人工验收记录 ├── pow/ # PoW 独立实现与基准 ├── scripts/ # 构建/发布/辅助脚本 +├── static/ # 构建产物（admin 等静态资源） ├── tests/ # 测试资源与脚本 │ ├── compat/ # 兼容性夹具与期望输出 │ │ ├── expected/ # 预期结果样本 @@ -78,9 +81,9 @@ ds2api/ │ │ └── toolcalls/ # toolcall 夹具 │ ├── node/ # Node 单元测试 │ ├── raw_stream_samples/ # 上游原始 SSE 样本 -│ │ ├── content-filter-trigger-20260405-jwt3/ # 风控终态样本 │ │ ├── continue-thinking-snapshot-replay-20260405/ # continue 样本 -│ │ ├── guangzhou-weather-reasoner-search-20260404/ # 搜索+引用样本 +│ │ ├── longtext-deepseek-v4-flash-20260429/ # flash 长文本/文件上传样本 +│ │ ├── longtext-deepseek-v4-pro-20260429/ # pro 长文本/文件上传样本 │ │ ├── markdown-format-example-20260405/ # Markdown 样本 │ │ └── markdown-format-example-20260405-spacefix/ # 空格修复样本 │ ├── scripts/ # 测试脚本入口 @@ -93,6 +96,8 @@ ds2api/ ├── features/ # 功能模块 │ ├── account/ # 账号管理页面 │ ├── apiTester/ # API 测试页面 + │ ├── chatHistory/ # 服务器端对话记录页面 + │ ├── proxy/ # 代理管理页面 │ ├── settings/ # 设置页面 │ └── vercel/ # Vercel 同步页面 ├── layout/ # 布局组件 @@ -126,8 +131,11 @@ flowchart LR subgraph RUNTIME[Shared runtime] AUTH[internal/auth] POOL[internal/account queue + concurrency] + CR[internal/completionruntime] + TURN[internal/assistantturn] STREAM[internal/stream + internal/sse] TOOL[internal/toolcall + internal/toolstream] + FMT[internal/format/openai + claude] DS[internal/deepseek/client] POW[pow + internal/deepseek/protocol] end @@ -153,16 +161,24 @@ flowchart LR PC --> PROMPT PC -.长历史.-> HIST PC --> AUTH + PC --> CR NCS -.Go prepare/release.-> CHAT NCS --> JS JS --> TOOL AUTH --> POOL - CHAT --> STREAM - RESP --> STREAM + CHAT --> CR + RESP --> CR + CA --> CR + GA --> CR + CR --> DS + CR --> STREAM + CR --> TURN + STREAM --> TURN STREAM --> TOOL - POOL --> DS + TURN --> FMT + POOL --> CR DS --> POW DS --> U[DeepSeek upstream] ``` @@ -171,7 +187,8 @@ flowchart LR - `internal/server`：路由树和中间件挂载（健康检查、协议入口、Admin/WebUI）。 - `internal/httpapi/openai/*`：OpenAI HTTP surface，按 chat、responses、files、embeddings、history、shared 拆分；chat/responses 共享 promptcompat、stream、toolcall 等核心语义。 -- `internal/httpapi/{claude,gemini}`：协议输入输出适配，归一到同一套 prompt compatibility 语义，不重复实现上游调用逻辑。 +- `internal/httpapi/{claude,gemini}`：协议输入输出适配，归一到同一套 prompt compatibility 语义；直连路径通过 `completionruntime` 共享 DeepSeek session/PoW/completion 调用，Vercel/代理路径仍可经 `translatorcliproxy` 转到 OpenAI handler。 +- `internal/httpapi/requestbody`：跨协议复用的请求体读取、JSON 解码前置校验与 UTF-8 错误处理辅助。 - `internal/promptcompat`：OpenAI/Claude/Gemini 请求到 DeepSeek 网页纯文本上下文的兼容内核。 - `internal/assistantturn`：Go 输出侧统一语义层，把 DeepSeek SSE 收集结果和流式收尾状态归一成 assistant turn，集中处理 thinking、tool call、citation、usage、stop/error 语义。 - `internal/completionruntime`：Go surface 共享的 completion 执行辅助，负责 DeepSeek session/PoW/call 启动、非流式 collect 和 empty-output retry；流式路径复用它启动上游请求，继续用 `internal/stream` 做实时消费，并在最终收尾阶段接入 `assistantturn`。 @@ -184,6 +201,13 @@ flowchart LR - `internal/chathistory`：服务器端对话记录持久化、分页、单条详情和保留策略。 - `internal/config`：配置加载、校验、运行时 settings 热更新。 - `internal/account`：托管账号池、并发槽位、等待队列。 +- `internal/textclean`：文本清洗，移除 `[reference: N]` 标记等噪声。 +- `internal/claudeconv`：Claude API 请求到 DeepSeek 格式的协议转换。 +- `internal/compat`：兼容性回归测试套件，用 SSE 夹具验证输出一致性。 +- `internal/rawsample`：上游原始响应的采集、读写与管理。 +- `internal/devcapture`：开发调试抓包，存储 HTTP 请求/响应用于问题排查。 +- `internal/util`：跨包通用工具，含 JSON 写入、类型转换、token 计数、thinking 解析等。 +- `internal/version`：版本号查询与比较，支持构建注入和运行时解析。 ## 4. WebUI 与运行时关系 diff --git a/docs/DeepSeekSSE行为结构说明-2026-04-05.md b/docs/DeepSeekSSE行为结构说明-2026-04-05.md index 007f4f4..081cd8b 100644 --- a/docs/DeepSeekSSE行为结构说明-2026-04-05.md +++ b/docs/DeepSeekSSE行为结构说明-2026-04-05.md @@ -1,7 +1,7 @@ # DeepSeek SSE 行为结构说明（第三方逆向版） > 说明：本文基于当前仓库 `tests/raw_stream_samples/` 下全部 `upstream.stream.sse` 原始流样本整理而成，属于第三方逆向观察文档，不是官方协议。 -> 当前 corpus 由 4 份原始流组成，覆盖搜索+引用、风控终态、Markdown 输出和空格敏感输出等行为。 +> 当前 corpus 由 5 份原始流组成，覆盖长文本生成、文件上传上下文、continue 接续、Markdown 输出和空格敏感输出等行为。 > 补充：文末还会注明少量“当前实现已确认、但 corpus 尚未完整覆盖”的行为，例如长思考场景下的自动续写状态。 文档导航：[文档总索引](./README.md) / [测试指南](./TESTING.md) / [样本目录说明](../tests/raw_stream_samples/README.md) @@ -12,8 +12,9 @@ | 样本 | 观察重点 | | --- | --- | -| [guangzhou-weather-reasoner-search-20260404](../tests/raw_stream_samples/guangzhou-weather-reasoner-search-20260404/upstream.stream.sse) | 搜索+思考流程，包含 `reference:N` 引用标记与工具片段 | -| [content-filter-trigger-20260405-jwt3](../tests/raw_stream_samples/content-filter-trigger-20260405-jwt3/upstream.stream.sse) | `CONTENT_FILTER` 终态分支，包含拒答模板与 `ban_regenerate` | +| [longtext-deepseek-v4-flash-20260429](../tests/raw_stream_samples/longtext-deepseek-v4-flash-20260429/upstream.stream.sse) | DeepSeek V4 flash 长文本流，包含 current input file 上传后的 completion 样本 | +| [longtext-deepseek-v4-pro-20260429](../tests/raw_stream_samples/longtext-deepseek-v4-pro-20260429/upstream.stream.sse) | DeepSeek V4 pro 长文本流，包含文件上传上下文和较长 reasoning/content 输出 | +| [continue-thinking-snapshot-replay-20260405](../tests/raw_stream_samples/continue-thinking-snapshot-replay-20260405/upstream.stream.sse) | 多轮 `completion + continue` 原始流，用于验证接续思考去重 | | [markdown-format-example-20260405](../tests/raw_stream_samples/markdown-format-example-20260405/upstream.stream.sse) | Markdown 输出的早期样本，用于观察 token 级输出形态 | | [markdown-format-example-20260405-spacefix](../tests/raw_stream_samples/markdown-format-example-20260405-spacefix/upstream.stream.sse) | Markdown 输出修正样本，用于验证空格 chunk 必须保留 | @@ -194,7 +195,7 @@ close ## 8. 终态行为 -当前 corpus 里有两条很重要的终态分支。 +当前 corpus 直接覆盖正常完成和 continue 接续；当前实现还兼容 `CONTENT_FILTER` 风控终态，相关分支由协议观察与兼容性 fixture 继续守护。 ### 8.1 正常完成 @@ -208,7 +209,7 @@ close ### 8.2 风控终态 -`content-filter-trigger-20260405-jwt3` 展示了另一种终态路径： +`CONTENT_FILTER` 不在当前 raw stream corpus 的目录样本中，但代码和兼容性测试仍按下面这种终态路径处理： 1. 先继续输出一段正常正文。 2. 出现提示类 fragment，例如 `TIP`。 diff --git a/docs/prompt-compatibility.md b/docs/prompt-compatibility.md index 2d6b4be..ac1c9fe 100644 --- a/docs/prompt-compatibility.md +++ b/docs/prompt-compatibility.md @@ -45,7 +45,8 @@ DS2API 当前的核心思路，不是把客户端传来的 `messages`、`tools` -> promptcompat 统一消息标准化 -> tool prompt 注入 -> DeepSeek 风格 prompt 拼装 - -> 文件收集 / inline 上传 / current input file（OpenAI 链路） + -> 文件收集 / inline 上传（OpenAI 文件链路） + -> current input file（completion runtime 全局入口） -> completion payload -> 下游网页对话接口 -> assistantturn 输出语义归一（Go 非流式 + 流式收尾） @@ -282,7 +283,7 @@ OpenAI 的文件上传现在不再是“只传文件本体”的通用路径， - 全局 completion runtime 应用点： [internal/completionruntime/nonstream.go](../internal/completionruntime/nonstream.go) -当前输入转文件启用并触发时，上传文件的真实文件名是 `DS2API_HISTORY.txt`，文件内容是完整 `messages` 上下文；它仍会先用 OpenAI 消息标准化和 DeepSeek 角色标记序列化，再按轮次编号成 `DS2API_HISTORY.txt` 风格的 transcript（不再注入文件边界标签）： +当前输入转文件启用并触发时，上传文件的真实文件名是 `DS2API_HISTORY.txt`，文件内容是完整 `messages` 上下文；它会使用 OpenAI-compatible 的消息/transcript 序列化规则和 DeepSeek 角色标记，再按轮次编号成 `DS2API_HISTORY.txt` 风格的 transcript（不再注入文件边界标签）： ```text [uploaded filename]: DS2API_HISTORY.txt