Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ SAG_JEV_API_KEY=
SAG_JEV_RANKING_TIMEOUT_SECONDS=30
# error:失败即报错;fallback_to_score:允许粗排结果并显示降级提示。
SAG_JEV_FAILURE_POLICY=error
# 可选 SAG2 证据检索(zleap-sag2,SAG_SEARCH_STRATEGY=sag2):共用上面的 JEV 连接(必须有 Key),
# 只读直搜 SQLite / PostgreSQL 关系库,不依赖向量库;失败不回退,SAG_JEV_FAILURE_POLICY 对其无效。
# 调优参数见 apps/api/.env.example 的 SAG_SAG2_* 说明。

# 浏览器访问后端的地址(构建期内联;修改后需重建 web 镜像)。
NEXT_PUBLIC_API_BASE=http://localhost:8000
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@

## Unreleased

- 探索页新增「对比」:同一问题在 2–3 种检索模式下并排检索,显示耗时、召回数、SAG2 的 JEV 用量与共同命中的片段,可选 LLM 评审,并可一键按某一模式正式检索。
- 信源创建时固定选择「事件检索(SAG1)」或「证据检索(SAG2)」,创建后不可切换;已有信源保留 SAG1。SAG1 上传保留实体/事项抽取,SAG2 上传只解析、分块、建立原生索引。
- 搜索与问答默认按各知识库算法自动召回,混合范围合并去重,部分失败明确提示;显式具体模式只检索同算法的信源。问答工具栏仍支持按轮选择,探索页支持模式对比。
- 搜索/问答选择未配置的付费 JEV 模式时打开配置弹窗,保存成功后才切换;取消保留原模式与输入。创建 SAG2 信源和上传不需要 JEV Key。
- SAG2 原生持久化索引支持 SQLite / PostgreSQL,检索在专用线程执行,只扫描所选已发布文档;SQLite 使用独立 sag2.db,PostgreSQL 使用独立 sag2 schema,不修改 SAG1 表。OCTX 当前仅支持 SAG1,SAG2 导出与导入覆盖均禁用。
- 后端中文分词依赖由 `jieba-py` 改回 `jieba` 0.42.1,与 `zleap-sag2` 共用一份词典;分词结果不变。

## v1.9.2 · 2026-10-22

- 新增结构化输出模式设置和自动降级,兼容不支持严格结构化输出的模型;改善口语化提问的查询分析与语义结果保留(#248)。
Expand Down
15 changes: 11 additions & 4 deletions README-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ SAG 不是传统 RAG 与 GraphRAG 的融合,而是一套替代二者的原创
| 能力 | 解决的问题 |
| --- | --- |
| 知识导入 | 文件与网页信源、文档解析、分块、向量化、事件/实体抽取、后台处理 |
| 检索 | 跨信源或指定信源检索,支持快速(`vector` / `multi_es_fast`)、精确(`multi`)与可选 JEV 精排(`multi_jev`)模式 |
| 检索 | 跨信源或指定信源检索,支持快速(`vector` / `multi_es_fast`)、精确(`multi`)、可选 JEV 精排(`multi_jev`)与 SAG2 证据检索(`sag2`)模式 |
| 原文溯源 | 每条检索结果和引用都能打开对应的原文块 |
| 知识图谱 | 查看事件、实体及其可查询的关联关系 |
| Agent 对话 | 基于指定信源进行多轮问答,并提供可点击引用 |
Expand Down Expand Up @@ -566,12 +566,19 @@ config = EngineConfig.from_env()
| 快速(multi_es-fast) | `multi_es_fast` | BM25 实体召回与事件多跳检索,跳过 LLM 精排 |
| 精确 | `multi` | 结合实体关系与 LLM 精排,结果更完整 |
| JEV 精排 | `multi_jev` | 召回与图扩展后,通过独立 JEV 服务为候选证据评分 |
| SAG2 证据检索 | `sag2` | 由 [zleap-sag2](https://pypi.org/project/zleap-sag2/) 原生索引检索:BM25 召回后用 JEV 服务多轮评分,不依赖向量库 |

这里列出的是应用/API 策略名称;应用将 `multi` 映射到 Python 引擎的 `full_expand`,将 `multi_es_fast` 映射到 `pruned_expand_rff`。可用性取决于存储能力:`multi_es_fast` 需要词法检索支持。
这里列出的是应用/API 策略名称;应用将 `multi` 映射到 Python 引擎的 `full_expand`,将 `multi_es_fast` 映射到 `pruned_expand_rff`,`sag2` 由独立的 `zleap-sag2` 引擎执行。可用性取决于存储能力:`multi_es_fast` 与 `multi_jev` 需要词法检索支持,`sag2` 需要 SQLite 或 PostgreSQL 关系库。

在“设置 → 知识库 → 检索”配置 302.AI、TypeSafe 官方或自定义 JEV 服务(默认 `jev-latest`,独立 API Key),再选择 **JEV 精排**。仅保存服务不会启用;默认限时 30 秒、失败报错,可选择粗排降级。
创建信源时只选择两种固定算法:**事件检索(SAG1)**(默认)或 **证据检索(SAG2)**,创建后不可编辑切换。SAG1 后续上传沿用原有解析、分块、向量化和实体/事项提取;SAG2 后续上传只解析、分块并建立原文索引,不调用抽取或向量模型。已有信源统一保留 SAG1,既有数据无需重建。OCTX 导入/导出目前仅支持 SAG1,SAG2 不允许导出或被 OCTX 覆盖。

支持 LanceDB、Elasticsearch,暂不支持 pgvector;PostgreSQL 作为关系库不受影响。未启用时沿用原检索策略,已有文档无需重新入库。效果与成本见[更新日志](CHANGELOG.md#jev-reranking)。
搜索和问答默认使用**自动(按知识库算法)**:SAG1 按设置页的 SAG1 检索策略召回,SAG2 按证据检索召回,混合范围合并、去重后回答;某一算法失败会明确提示结果不完整。显式选择 SAG1 子模式只检索 SAG1 信源,选择 SAG2 只检索 SAG2 信源;Agent 人格中的显式策略也遵循此限制。探索页仍可并排对比 2–3 种具体模式。

在“设置 → 知识库 → 检索”配置 302.AI、TypeSafe 官方或自定义 JEV 服务(默认 `jev-latest`,独立 API Key)。JEV 精排和 SAG2 检索共用此服务:搜索/问答选择缺少配置的付费模式时打开配置弹窗,保存成功后才切换,取消保留当前模式和输入。创建 SAG2 信源及上传建索引不要求 JEV Key。JEV 精排默认限时 30 秒、失败报错,可选择粗排降级;SAG2 失败不降级。

**SAG2** 使用独立的 `zleap-sag2` 原生持久化索引,支持 SQLite / PostgreSQL(首期不支持 MySQL / OceanBase),BM25 召回后多轮调用 JEV 评分,不依赖向量库。仅已发布的所选文档进入检索范围和扫描计数。SQLite 数据位于引擎目录的 `sag2.db`;PostgreSQL 使用原数据库的独立 `sag2` schema,连接用户需有创建 schema 和其中表的权限。SAG1 数据表保持原样。候选数、超时、并发与范围上限等参数见 `apps/api/.env.example` 的 `SAG_SAG2_*`。

JEV 精排支持 LanceDB、Elasticsearch,暂不支持 pgvector;PostgreSQL 作为关系库不受影响。未启用时沿用原检索策略,已有文档无需重新入库。效果与成本见[更新日志](CHANGELOG.md#jev-reranking)。

#### 存储后端

Expand Down
15 changes: 11 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Upload a document once. SAG parses it, splits it into chunks, embeds it, extract
| Capability | What it gives you |
| --- | --- |
| Knowledge ingestion | File and web sources, document parsing, chunking, embedding, event/entity extraction, background processing |
| Search | Cross-source or source-scoped retrieval with Fast (`vector` / `multi_es_fast`), Precise (`multi`), and optional JEV reranking (`multi_jev`) modes |
| Search | Cross-source or source-scoped retrieval with Fast (`vector` / `multi_es_fast`), Precise (`multi`), optional JEV reranking (`multi_jev`), and SAG2 evidence retrieval (`sag2`) modes |
| Source tracing | Open any result or citation back to the exact original chunk |
| Knowledge graph | Inspect events, entities, and their queryable associations |
| Agent chat | Multi-turn answers grounded in selected sources, with clickable citations |
Expand Down Expand Up @@ -564,12 +564,19 @@ Pipeline request and result types are available from `zleap.sag.pipeline`: `Pars
| Fast (multi_es-fast) | `multi_es_fast` | BM25 entity recall and event multi-hop retrieval, skipping LLM reranking |
| Precise | `multi` | Combines entity relationships with LLM reranking for more complete results |
| JEV reranking | `multi_jev` | Scores recalled and graph-expanded evidence through an independent JEV service |
| SAG2 evidence retrieval | `sag2` | Native-index search by [zleap-sag2](https://pypi.org/project/zleap-sag2/) over the relational store: BM25 recall scored over several JEV rounds, no vector store involved |

These are the application/API strategy names; the application maps `multi` to the Python engine's `full_expand` and `multi_es_fast` to `pruned_expand_rff`. Availability depends on storage capabilities: `multi_es_fast` requires lexical search support.
These are the application/API strategy names; the application maps `multi` to the Python engine's `full_expand` and `multi_es_fast` to `pruned_expand_rff`, while `sag2` runs on the separate `zleap-sag2` engine. Availability depends on storage capabilities: `multi_es_fast` and `multi_jev` require lexical search support; `sag2` requires a SQLite or PostgreSQL relational store.

Configure 302.AI, TypeSafe, or a custom JEV service in **Settings → Knowledge → Retrieval** (default model: `jev-latest`, separate API key), then select **JEV reranking**. Saving the service alone does not enable it. The default deadline is 30 seconds; failures return an error unless coarse fallback is enabled.
Create each source with one immutable algorithm: **Event retrieval (SAG1)** (default) or **Evidence retrieval (SAG2)**. SAG1 uploads retain parsing, chunking, embeddings and entity/event extraction. SAG2 uploads only parse, chunk and build native text indexes, without extraction or embeddings. Existing sources stay on SAG1 without rebuilding. OCTX import/export remains SAG1-only; SAG2 export and OCTX overwrite are rejected.

Supported on LanceDB and Elasticsearch; pgvector is currently unsupported. PostgreSQL relational storage can use JEV with a supported vector backend. Existing retrieval settings remain until JEV is selected; documents do not need reindexing. See the [benchmark results and cost estimates](CHANGELOG.md#jev-reranking).
Search and chat default to **Automatic (by knowledge-base algorithm)**: SAG1 uses the SAG1 retrieval profile from Settings, SAG2 uses evidence retrieval, and mixed scopes merge and deduplicate both groups before answering. Partial failures explicitly identify the missing algorithm. Selecting a concrete SAG1 profile restricts retrieval to SAG1 sources; selecting SAG2 restricts it to SAG2 sources. Explicit agent-persona strategies follow the same scope rule. Explore still compares two or three concrete modes side by side.

Configure 302.AI, TypeSafe, or a custom JEV service in **Settings → Knowledge → Retrieval** (default model: `jev-latest`, separate API key). JEV reranking and SAG2 retrieval share this paid service. Selecting an unconfigured paid mode in search/chat opens a configuration dialog; only a successful save activates the mode. Cancel preserves the current mode and input. Creating or indexing a SAG2 source needs no JEV key. JEV reranking has a 30-second default deadline and optional coarse fallback; SAG2 never falls back.

**SAG2** uses separate durable native `zleap-sag2` indexes on SQLite or PostgreSQL (MySQL and OceanBase are not supported), BM25 recall and multiple JEV scoring rounds, without a vector store. Retrieval and scope counting include only selected, published inputs. SQLite uses `sag2.db` in the engine data directory; PostgreSQL uses a separate `sag2` schema in the configured database, requiring permission to create the schema and its tables. SAG1 tables remain unchanged. Tuning parameters are documented as `SAG_SAG2_*` in `apps/api/.env.example`.

JEV reranking is supported on LanceDB and Elasticsearch; pgvector is currently unsupported. PostgreSQL relational storage can use JEV with a supported vector backend. Existing retrieval settings remain until a JEV mode is selected; documents do not need reindexing. See the [benchmark results and cost estimates](CHANGELOG.md#jev-reranking).

#### Storage backends

Expand Down
14 changes: 14 additions & 0 deletions apps/api/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,20 @@ SAG_JEV_API_KEY=
SAG_JEV_RANKING_TIMEOUT_SECONDS=30
SAG_JEV_FAILURE_POLICY=error

# 创建信源时可选 SAG2 证据检索(zleap-sag2),创建后算法不可更改;搜索/问答默认按算法自动召回。
# 检索共用上面的 JEV 服务地址、模型与 API Key(付费);创建和上传建索引无需 JEV Key。
# 原生索引使用独立 SQLite sag2.db / PostgreSQL sag2 schema,不向量化或抽取实体事项;检索失败不回退,
# SAG_JEV_FAILURE_POLICY 对其无效。以下调优参数首期只经环境变量设置,默认值已控制用量。
# SAG_SAG2_CANDIDATE_K=50 # BM25 候选数(zleap-sag2 默认 100)
# SAG_SAG2_SEED_K=10 # 每轮扩展种子数
# SAG_SAG2_MAX_HOPS=2 # 扩展轮数 0–2
# SAG_SAG2_EXPANSION_TERM_LIMIT=20 # 每轮扩展词数
# SAG_SAG2_SEARCH_TIMEOUT_SECONDS=90 # 整次检索上限(含并发排队、扫描与全部评分轮次;设置发布期间的等待另计)
# SAG_SAG2_DECISION_TIMEOUT_SECONDS=60 # 单次决策请求超时
# SAG_SAG2_DECISION_MAX_CONCURRENCY=20 # 决策请求并发
# SAG_SAG2_MAX_CONCURRENT_SEARCHES=2 # 服务端同时执行的 SAG2 检索数
# SAG_SAG2_MAX_SCOPE_CHUNKS=20000 # 所选已发布文档分块上限(不含退役版本或未发布输入),0 不限制

# ── 知识宇宙镜头阈值与浏览器预算 ──────────────────────
SAG_UNIVERSE_LOD_ORBIT_PX=72
SAG_UNIVERSE_LOD_NEAR_PX=180
Expand Down
5 changes: 4 additions & 1 deletion apps/api/packaging/sag-api.spec
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ for package in (
# OpenAI, Anthropic, and Gemini, so freezing every LiteLLM module would pull
# unrelated server/CLI dependencies into the desktop sidecar.
datas += collect_data_files("litellm")
# zleap-sag2 的 Unicode 排序在运行时读取 pyuca 的 allkeys-*.txt,冻结包需显式收集。
datas += collect_data_files("pyuca")
datas += collect_data_files("certifi")
for package in (
"litellm.litellm_core_utils",
Expand All @@ -72,9 +74,10 @@ for package in (
hiddenimports += collect_submodules(package)

for package in (
"jieba-py",
"jieba",
"sag-api",
"zleap-sag",
"zleap-sag2",
"lancedb",
"litellm",
"markitdown",
Expand Down
3 changes: 2 additions & 1 deletion apps/api/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ dependencies = [
"litellm==1.103.0",
"orjson>=3.11.6,<4",
"zleap-sag==0.15.0",
"zleap-sag2==0.1.0",
"alembic>=1.13",
"trafilatura>=1.8",
"markitdown[docx,pdf,pptx,xls,xlsx]>=0.1.6,<0.2",
Expand All @@ -31,7 +32,7 @@ dependencies = [
"mcp>=1.28,<2",
"octx[vectors]==0.1.6",
"tzdata>=2024.1",
"jieba-py>=0.46.12,<0.47",
"jieba==0.42.1",
]

[project.optional-dependencies]
Expand Down
5 changes: 5 additions & 0 deletions apps/api/sag_api/api/v1/agents.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from sse_starlette.sse import EventSourceResponse

from sag_agent import AgentRuntime, EventType, RunHandle
from sag_api.core.config import settings
from sag_api.core.db import SessionLocal, get_session
from sag_api.core.deps import (
get_agent_runtime,
Expand Down Expand Up @@ -351,6 +352,9 @@ async def ask(
thread = await svc.get_thread(session, agent.id, thread_id)
if not llm.configured:
raise ConfigurationError("尚未配置 LLM,无法生成回答")
if body.strategy is not None:
# 本轮显式选择的模式必须可用(付费 JEV 模式需已配置 Key);在流开始前就拒绝。
EngineManager.require_strategy_available(settings, body.strategy)
plan = await svc.prepare_ask(
session,
agent=agent,
Expand All @@ -359,6 +363,7 @@ async def ask(
attachments=body.attachments,
source_ids=body.source_ids,
llm=llm,
strategy=body.strategy,
)
web_enabled = body.effective_web_enabled

Expand Down
5 changes: 3 additions & 2 deletions apps/api/sag_api/api/v1/dify.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
ServiceUnavailableError,
ValidationError,
)
from sag_api.enums import JEV_SCORED_STRATEGIES
from sag_api.sag import EngineManager
from sag_api.schemas.dify import (
DifyRetrievalRecord,
Expand Down Expand Up @@ -104,11 +105,11 @@ async def retrieval(
**({
"score_source": section.metadata.get("score_source", "unknown"),
"ranking_fallback_reason": (outcome.stats.get("ranking") or {}).get("fallback_reason"),
} if strategy == "multi_jev" else {}),
} if strategy in JEV_SCORED_STRATEGIES else {}),
},
)
for section in outcome.sections
if section.score >= body.retrieval_setting.score_threshold
and (strategy != "multi_jev" or fallback_used or section.metadata.get("score_source") == "jev")
and (strategy not in JEV_SCORED_STRATEGIES or fallback_used or section.metadata.get("score_source") == "jev")
]
return DifyRetrievalResponse(records=records)
13 changes: 10 additions & 3 deletions apps/api/sag_api/api/v1/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
from sag_api.core.errors import ApiError, ValidationError
from sag_api.core.logging import get_logger
from sag_api.db.models import Source, User
from sag_api.enums import JEV_SCORED_STRATEGIES
from sag_api.generation import LLMClient
from sag_api.sag import EngineManager, RetrievedSection, SearchOutcome
from sag_api.schemas.insight import EntityOut, GraphRelationOut
Expand All @@ -43,6 +44,7 @@
event_scores_from_evidence,
published_document_inputs,
recall_event_scores,
resolve_search_strategy,
retrieve_relevant_sections,
stream_synthesize_search_answer,
synthesize_search_answer,
Expand Down Expand Up @@ -169,16 +171,21 @@ async def _retrieve_search_evidence(
body: SearchRequest | GlobalSearchRequest,
) -> tuple[SearchOutcome, EventScoreMap, bool]:
configuration = getattr(engine_manager, "_settings", settings)
strategy = body.strategy or configuration.search_strategy
# Resolve once for the evidence-shape decision below; the retrieval service
# resolves the same inputs again and records where the mode came from.
strategy = resolve_search_strategy(requested=body.strategy, sources=sources, configuration=configuration).strategy
retrieval = retrieve_relevant_sections(
engine_manager,
sources,
body.query,
strategy=strategy,
strategy=body.strategy,
top_k=body.top_k,
include_ranked_candidates=body.include_ranked_candidates,
)
if strategy == "multi_jev":
if strategy in JEV_SCORED_STRATEGIES or (strategy == "auto" and (
configuration.search_strategy == "multi_jev"
or any(getattr(source, "engine_kind", "sag1") == "sag2" for source in sources)
)):
outcome = await retrieval
return outcome, event_scores_from_evidence(outcome.sections), True
outcome, scores = await asyncio.gather(
Expand Down
Loading
Loading