Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,11 @@ CNB_REPO_SLUG=your-org/ai-proxy
#QUOTA_ORG=your-org

# ---- optional tuning ----
# Comma-separated model ids advertised on /v1/models (routing is done by CNB).
#PROXY_MODELS=deepseek-v4-flash,glm-5.3-flash,kimi-k3
# Fallback model ids for /v1/models, used only until a response reveals the
# real upstream model name (which is then cached). See src/models.mjs.
#PROXY_MODELS=deepseek-v4.1-flash,glm-5.3-flash,kimi-k3
# Where the sniffed-model registry is persisted across workspace restarts.
#PROXY_MODELS_STATE=models-state.json
#PROXY_PORT=9001
# Upstream connect/first-byte timeout (ms)
#PROXY_UPSTREAM_TIMEOUT_MS=15000
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,5 @@ node_modules/
.DS_Store
__pycache__/
*.pyc
models-state.json
models-state.json.tmp
13 changes: 7 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,22 +100,22 @@ Three simple steps, only one API key to configure:
```bash
ANTHROPIC_BASE_URL=https://ai.example.com \
ANTHROPIC_AUTH_TOKEN=your-proxy-key \
ANTHROPIC_MODEL=deepseek-v4-flash \
ANTHROPIC_MODEL=deepseek-v4.1-flash \
claude -p "Hello! Introduce yourself."
```

### 2. OpenAI Clients & Gateways
Point BaseURL to `https://ai.example.com/v1` and set API Key to `PROXY_KEY`:
- **Chat Apps**: NextChat, LobeChat, Cherry Studio, Open WebUI
- **Gateways**: New API, One API (Channel type: OpenAI, model: `deepseek-v4-flash`)
- **Gateways**: New API, One API (Channel type: OpenAI, model: `deepseek-v4.1-flash`)
- **Coding Agents**: Cursor, Continue, Codex CLI

### 3. curl Verification
```bash
curl https://ai.example.com/v1/chat/completions \
-H "Authorization: Bearer your-proxy-key" \
-H "Content-Type: application/json" \
-d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hi"}],"stream":false}'
-d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"hi"}],"stream":false}'
```

---
Expand All @@ -131,7 +131,7 @@ Real production billing data measured over months of live operation (not marketi
| **Compute Core-Hours** | **1,600 core-hours** (shared dev + CI pool) | A `runner.cpus: 2` workspace running 24/7 for 30 days costs **1,440 core-hours** (safely within the 1,600 pool). No need to stop the proxy. |

> 💡 **The Math Behind 200M Free Tokens/Month**:
> 1. **Base Upstream Rate**: Live measurements on `deepseek-v4-flash` show an uncached 9,000-token request costs ~0.39 credit (approx **23,000 tokens / credit**).
> 1. **Base Upstream Rate**: Live measurements on `deepseek-v4.1-flash` show an uncached 9,000-token request costs ~0.39 credit (approx **23,000 tokens / credit**).
> 2. **Official Prompt Cache 30x Discount**: Once prompt prefixes hit CNB's prompt cache, cost drops to **~0.01 credit**, slashing cost to **1/30** of base price (~**700,000 tokens / credit**).
> 3. **Real-World Scenarios**: In coding assistants, Claude Code, and multi-turn Agent conversations with long system prompts and history reuse, cache hit rates typically sit at 80%~95% (assuming a realistic **90% cache hit rate**):
> $$\text{Blended Cost} = 10\% \times 1 + 90\% \times \frac{1}{30} \approx 13\% \text{ base cost} \implies \approx 177,000\text{ tokens / credit}$$
Expand Down Expand Up @@ -171,7 +171,8 @@ npm run quota # or: npx cnb2api-quota
| `PROXY_KEY` | — (required) | Bearer key / x-api-key sent by clients. Refuses to start if missing. |
| `CNB_TOKEN` | — (required) | Upstream token injected by CNB pipeline environment. |
| `CNB_REPO_SLUG` | (built-in) | `org/repo` used for upstream routing. Auto-filled by CNB. |
| `PROXY_MODELS` | `deepseek-v4-flash,glm-5.3-flash,kimi-k3` | Model IDs advertised on `/v1/models`. |
| `PROXY_MODELS` | `deepseek-v4.1-flash,glm-5.3-flash,kimi-k3` | **Fallback** model ids for `/v1/models` — used only until a response reveals the real upstream model name. |
| `PROXY_MODELS_STATE` | `models-state.json` | Where the sniffed-model registry is persisted across workspace restarts. Relative paths resolve against the run dir. |
| `PROXY_PORT` | `9001` | Listen port. |
| `PROXY_UPSTREAM_TIMEOUT_MS` | `15000` | Upstream connect / first-byte timeout (ms). |
| `PROXY_IDLE_TIMEOUT_MS` | `300000` | Per-stream idle watchdog (ms). |
Expand All @@ -183,7 +184,7 @@ npm run quota # or: npx cnb2api-quota
## FAQ

**Q: Why does the model list include glm and kimi when the backend is DeepSeek?**
A: The official CNB AI gateway currently routes all models to `deepseek-v4-flash`. Exposing multiple aliases ensures compatibility with various client presets out of the box.
A: `/v1/models` is **self-discovering**: the CNB gateway has no list endpoint, so the proxy learns the real model name from every completion response (the gateway echoes the resolved id, e.g. `deepseek-v4.1-flash`) and advertises that. `PROXY_MODELS` is only a fallback for the very first requests before anything has been sniffed — it exists so client presets that name glm/kimi keep working out of the box, and you can freeze it to whatever `/v1/models` has been returning.

**Q: Are native Tool Calls / Function Calling supported?**
A: Fully supported. In streaming mode, `input_json_delta` chunks are cleanly combined; in non-streaming mode, standard `tool_calls` structures are reassembled.
Expand Down
13 changes: 7 additions & 6 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,22 +100,22 @@ https://api.cnb.cool/<org>/<repo>/-/ai/chat/completions (官方 AI 核心端点)
```bash
ANTHROPIC_BASE_URL=https://ai.example.com \
ANTHROPIC_AUTH_TOKEN=your-proxy-key \
ANTHROPIC_MODEL=deepseek-v4-flash \
ANTHROPIC_MODEL=deepseek-v4.1-flash \
claude -p "你好,请介绍一下你自己"
```

### 2. OpenAI 客户端 / 聚合网关
将 BaseURL 指向 `https://ai.example.com/v1`,API Key 填入 `PROXY_KEY`:
- **聊天应用**:NextChat、LobeChat、Cherry Studio、Open WebUI
- **网关渠道**:New API、One API(渠道类型选择 OpenAI,模型填 `deepseek-v4-flash`)
- **网关渠道**:New API、One API(渠道类型选择 OpenAI,模型填 `deepseek-v4.1-flash`)
- **代码插件**:Cursor、Continue、Codex CLI

### 3. curl 调用验证
```bash
curl https://ai.example.com/v1/chat/completions \
-H "Authorization: Bearer your-proxy-key" \
-H "Content-Type: application/json" \
-d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hi"}],"stream":false}'
-d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"hi"}],"stream":false}'
```

---
Expand All @@ -131,7 +131,7 @@ curl https://ai.example.com/v1/chat/completions \
| **算力核时** | **1,600 核时**(dev + CI 共享池) | `runner.cpus: 2` 反代工作区整月 30 天 7×24h 不间断运行仅消耗 **1,440 核时**(完全在免费池内),无需关机省核时。 |

> 💡 **2 亿 Tokens/月的实测换算逻辑**:
> 1. **官方基础定价**:对 `deepseek-v4-flash` 实测,全新未命中缓存的请求约 9,000 tokens 花费 0.39 credit(约 **2.3 万 tokens / credit**)。
> 1. **官方基础定价**:对 `deepseek-v4.1-flash` 实测,全新未命中缓存的请求约 9,000 tokens 花费 0.39 credit(约 **2.3 万 tokens / credit**)。
> 2. **官方 Prompt Cache 30倍优惠**:命中 CNB 的 prompt 缓存后降至 **~0.01 credit**,相当于基准价格的 **1/30**(约 **70 万 tokens / credit**)。
> 3. **综合场景折算**:在 Cursor、Claude Code、Agent 编码等多轮对话和长 System Prompt 场景下,上下文缓存命中率通常在 80%~95%(取典型 **90% 命中率**):
> $$\text{综合成本} = 10\% \times 1 + 90\% \times \frac{1}{30} \approx 13\% \text{ 原价} \implies \approx 17.7\text{ 万 tokens / credit}$$
Expand Down Expand Up @@ -171,7 +171,8 @@ npm run quota # 或: npx cnb2api-quota
| `PROXY_KEY` | — (必填) | 客户端访问反代网关的 Bearer Token / x-api-key。 |
| `CNB_TOKEN` | — (必填) | CNB 内网端点鉴权令牌,流水线环境自动注入。 |
| `CNB_REPO_SLUG` | (内置) | 当前仓库 `org/repo`,流水线内置自动填充。 |
| `PROXY_MODELS` | `deepseek-v4-flash,glm-5.3-flash,kimi-k3` | `/v1/models` 暴露的模型列表。 |
| `PROXY_MODELS` | `deepseek-v4.1-flash,glm-5.3-flash,kimi-k3` | **兜底**模型列表:仅在尚未嗅探到真实上游模型名时用于 `/v1/models`。 |
| `PROXY_MODELS_STATE` | `models-state.json` | 嗅探到的模型名注册表的持久化路径,跨工作区重启保留。相对路径相对于运行目录解析。 |
| `PROXY_PORT` | `9001` | 服务监听端口。 |
| `PROXY_UPSTREAM_TIMEOUT_MS` | `15000` | 上游连接首字节超时 (ms)。 |
| `PROXY_IDLE_TIMEOUT_MS` | `300000` | 流式事件空闲超时看门狗 (ms)。 |
Expand All @@ -183,7 +184,7 @@ npm run quota # 或: npx cnb2api-quota
## 常见问题 (FAQ)

**Q: 既然上游是 DeepSeek,为什么模型列表有 glm 和 kimi?**
A: CNB 官方网关目前在底层统一将请求路由至 `deepseek-v4-flash`。暴露多个模型 ID 仅为了兼容各类客户端的默认预设,后续上游开放多模型时反代无需修改即可支持。
A: `/v1/models` 是**自发现**的:CNB 网关没有列表接口,所以反代从每一次补全响应里嗅探网关回带的真实模型 id(例如 `deepseek-v4.1-flash`)并对外暴露,且持久化到磁盘跨重启保留。`PROXY_MODELS` 只是首个请求发生前的**兜底**列表——它保证各类客户端默认预设开箱即用,你也可以随时把它固化成 `/v1/models` 实际返回过的值。

**Q: 原生函数/工具调用(Tool Calls)支持怎么样?**
A: 完整原生支持。流式模式下支持 `input_json_delta` 增量合并,非流式模式下忠实聚合出标准的 `tool_calls` 结构。
Expand Down
14 changes: 14 additions & 0 deletions docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ src/
├── config.mjs env loading + fail-fast validation (no default key, ever)
├── auth.mjs Bearer key check (timing-safe) + sliding-window fail limiter
├── sse.mjs SSE event parsing + non-stream aggregation
├── models.mjs passive model-name registry (sniffed from responses, disk-persisted)
├── proxy.mjs upstream forwarding: AbortSignal timeout + two-way cancel
├── server.mjs http server + routing (/health, /v1/models, /v1/chat/completions)
└── log.mjs structured single-line JSON logs, correlated by reqId
Expand Down Expand Up @@ -64,6 +65,19 @@ The non-stream aggregator is a small state machine, not a naive
- `id`/`model`/`created` → upstream values win
- comment/heartbeat lines (`: keep-alive`) → ignored

### Model discovery (no list endpoint upstream)

The CNB gateway exposes **no model-list endpoint** (every candidate path 404s),
so the only trustworthy source of the real model name is the `model` field the
gateway echoes on every completion. `models.mjs` therefore **sniffs passively**:
`proxy.mjs` calls `observe(j.model)` on streamed chunks and on the non-stream
aggregate, and the registry keeps first-seen order, dedups, and atomically
persists to `PROXY_MODELS_STATE` (`writeFileSync` to `.tmp` + `renameSync`) so a
workspace hot-restart keeps history. `/v1/models` returns the sniffed names, and
only falls back to the frozen `PROXY_MODELS` list while nothing has been sniffed
yet. There is deliberately **no startup probe** — we never spend credits on a
liveness request. Cost: zero.

## 4. Self-healing address (keepalive)

Three pieces, no long-lived CNB token in the loop:
Expand Down
18 changes: 13 additions & 5 deletions docs/SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,18 @@ the secrets-repo flow below — delete the inline `env:` line, uncomment
- A [cnb.cool](https://cnb.cool) account whose org has **AI credits** enabled
(the "天才程序员" / built-in AI offering). Without it the upstream endpoint
`https://api.cnb.cool/<org>/<repo>/-/ai/chat/completions` returns errors.
- Know which models your account exposes. Set `PROXY_MODELS` to that list —
the repo default is `deepseek-v4-flash,glm-5.3-flash,kimi-k3` (what our
account advertises; the gateway currently routes all three to the same
upstream model) and requests naming an unavailable model will be rejected.
- Know which models your account exposes. `/v1/models` **self-discovers** the
real upstream model name: the CNB gateway has no list endpoint, so the proxy
sniffs the resolved `model` id echoed on every completion response (currently
`deepseek-v4.1-flash`) and advertises that, persisting it across workspace
restarts. `PROXY_MODELS` is only the **fallback** used until the first
response has been sniffed — the repo default is
`deepseek-v4.1-flash,glm-5.3-flash,kimi-k3` (what our account advertises) so
client presets keep working before any request has run. As of 2026-10 CNB
upgraded the DeepSeek backend from `deepseek-v4-flash` to
`deepseek-v4.1-flash`; because the gateway echoes the resolved model id on
every response, the proxy picks up future renames automatically — no code or
config change needed.

**Quota & compute economics** (measured on our deployment):
- **Compute**: a workspace pinned at `runner.cpus: 2` burns 48 core-hours/day;
Expand Down Expand Up @@ -99,7 +107,7 @@ inlined in the code repo — is the cleaner layout for teams or public forks.
|---|---|
| both `imports:` stanzas | `https://cnb.cool/your-org/cnb2api-secrets/-/blob/main/proxy.yml` |
| vscode `env:` | delete the inline `PROXY_KEY` (it now comes from the secrets repo) |
| (secrets repo) | `PROXY_MODELS: "model-x,model-y"` if your models differ |
| (secrets repo) | `PROXY_MODELS: "model-x,model-y"` — fallback list only (optional) |

`REPO` for the keepalive needs no edit either way — it reads the built-in
`CNB_REPO_SLUG`. Note the keepalive cron lives under the `main:` branch key —
Expand Down
12 changes: 9 additions & 3 deletions src/config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,15 @@ export const config = {
repo: process.env.CNB_REPO_SLUG || process.env.CNB_BUILD_REPO || '',
proxyKey: process.env.PROXY_KEY || '',
upstreamToken: process.env.CNB_TOKEN || '',
// Model ids advertised on /v1/models. Actual routing is done by the CNB gateway;
// set PROXY_MODELS to whatever your account exposes.
models: (process.env.PROXY_MODELS || 'deepseek-v4-flash,glm-5.3-flash,kimi-k3').split(',').map((s) => s.trim()).filter(Boolean),
// Model ids advertised on /v1/models when nothing has been sniffed yet.
// The gateway has no list endpoint, so once a response reveals the real
// upstream model name (e.g. deepseek-v4.1-flash) it wins; this default is the
// frozen fallback (see src/models.mjs). Set PROXY_MODELS to whatever your
// account exposes.
models: (process.env.PROXY_MODELS || 'deepseek-v4.1-flash,glm-5.3-flash,kimi-k3').split(',').map((s) => s.trim()).filter(Boolean),
// Where the sniffed-model registry is persisted (survives a workspace hot
// restart). Relative paths resolve against the process CWD (the run dir).
modelsStatePath: process.env.PROXY_MODELS_STATE || 'models-state.json',
maxBodyBytes: 4 * 1024 * 1024,
upstreamTimeoutMs: Number(process.env.PROXY_UPSTREAM_TIMEOUT_MS || 15_000), // connect + first byte
idleTimeoutMs: Number(process.env.PROXY_IDLE_TIMEOUT_MS || 300_000), // per-stream idle cap (reset each chunk)
Expand Down
70 changes: 70 additions & 0 deletions src/models.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
// 上游模型名嗅探注册表。
//
// 背景:CNB 的 AI 网关没有「列模型」接口(/-/ai/models 等一律 404),
// 但每个 chat/completions 响应都会回显它真正路由到的模型名(如
// deepseek-v4.1-flash)。因此唯一的真实来源是「读响应里的 model 字段」。
//
// 策略(纯被动,零额外请求,不主动测活):
// - 每次转发成功的响应里观察到的 model 名登记进内存注册表(首见顺序去重);
// - 新模型出现时落盘持久化,重启后先加载,避免冷启动丢历史;
// - GET /v1/models 优先返回嗅探到的真实名;尚未嗅探到任何名字时回退
// config.models(即 PROXY_MODELS,人工固化自 /v1/models 的历史返回值)。
import fs from 'node:fs';
import { config } from './config.mjs';
import { log } from './log.mjs';

/** @type {string[]} 首见顺序的去重上游模型名 */
let observed = [];
const seen = new Set();
let loaded = false;

function load() {
loaded = true;
try {
const raw = fs.readFileSync(config.modelsStatePath, 'utf8');
const parsed = JSON.parse(raw);
const list = Array.isArray(parsed?.models) ? parsed.models : [];
for (const m of list) {
if (typeof m === 'string' && m && !seen.has(m)) { seen.add(m); observed.push(m); }
}
if (observed.length) log.info('-', 'models: loaded state', { count: observed.length, path: config.modelsStatePath });
} catch (e) {
if (e.code !== 'ENOENT') log.warn('-', 'models: state load failed', { err: String(e).slice(0, 160), path: config.modelsStatePath });
}
}

function persist() {
try {
const tmp = `${config.modelsStatePath}.tmp`;
fs.writeFileSync(tmp, JSON.stringify({ models: observed, updated_at: new Date().toISOString() }, null, 2));
fs.renameSync(tmp, config.modelsStatePath); // 原子替换,避免半写文件
} catch (e) {
// 持久化失败不致命:内存注册表仍可用,只是重启后可能冷启动
log.warn('-', 'models: state persist failed', { err: String(e).slice(0, 160), path: config.modelsStatePath });
}
}

// 观察一个上游回显的模型名。首次见到才登记 + 落盘(后续命中直接返回)。
export function observe(model) {
if (typeof model !== 'string') return;
const m = model.trim();
if (!m || m === 'unknown') return;
if (!loaded) load();
if (seen.has(m)) return;
seen.add(m);
observed.push(m);
log.info('-', 'models: discovered upstream model', { model: m });
persist();
}

// /v1/models 的模型列表:优先真实嗅探名,尚无样本时回退 PROXY_MODELS。
export function list() {
if (!loaded) load();
return observed.length ? [...observed] : [...config.models];
}

// 快照:供调试/测试观察注册表来源。
export function snapshot() {
if (!loaded) load();
return { source: observed.length ? 'upstream' : 'configured', observed: [...observed], fallback: [...config.models] };
}
3 changes: 3 additions & 0 deletions src/proxy.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { config } from './config.mjs';
import { log } from './log.mjs';
import { aggregate } from './sse.mjs';
import { record as recordUsage } from './usage.mjs';
import { observe as observeModel } from './models.mjs';

export async function forward({ reqId, payload, wantStream, res, transform }) {
// transform(可选):Anthropic /v1/messages 适配挂载点。
Expand Down Expand Up @@ -77,6 +78,7 @@ export async function forward({ reqId, payload, wantStream, res, transform }) {
let j;
try { j = JSON.parse(data); } catch { return; /* 心跳/非 JSON 行忽略 */ }
if (j.usage) usage = j.usage;
if (j.model) observeModel(j.model); // 嗅探上游真实模型名(无 list 接口,只能从响应读)
if (transform) writeEvents(transform.stream.feed(j));
};
const parseLines = (lines) => {
Expand Down Expand Up @@ -156,6 +158,7 @@ export async function forward({ reqId, payload, wantStream, res, transform }) {
}
clearTimeout(idleTimer);
const body = aggregate(text);
if (body.model) observeModel(body.model); // 嗅探上游真实模型名(非流式聚合结果里的 model)
recordUsage({ prompt: body.usage.prompt_tokens || 0, completion: body.usage.completion_tokens || 0 });
log.info(reqId, 'chat done', { ms: Date.now() - started, out: body.usage.completion_tokens });
if (transform) return { handled: false, status: 200, body: transform.response(body) };
Expand Down
Loading
Loading