Skip to content

feat: add dataset adapters, resumable LLM Judge, and external benchmark integrations - #11

Merged
Zleap-Admin merged 5 commits into
Zleap-AI:mainfrom
lixiaoguai11:contrib/v2-release
Aug 28, 2026
Merged

Zleap-Admin merged 5 commits into
Zleap-AI:mainfrom
lixiaoguai11:contrib/v2-release

Conversation

@lixiaoguai11

Copy link
Copy Markdown
Contributor

Summary

This PR consolidates the v2 benchmark framework updates into a single, reviewable release branch.

The changes improve the project’s runtime architecture, dataset handling, evaluation reproducibility, external baseline integration, NarrativeQA support, and public-release documentation.

Main changes

1. Core runtime and storage architecture

  • Stabilize the SAG2 runtime lifecycle.
  • Clarify the boundaries between:
    • search orchestration;
    • runtime services;
    • storage facade;
    • event-universe storage;
    • database and Elasticsearch providers.
  • Keep search strategies independent from low-level storage clients.
  • Preserve compatibility imports through package-level __init__.py modules.
  • Improve configuration ownership and reduce duplicated storage logic.
  • Keep SAG2-specific runtime dependencies isolated from other search strategies.

2. Dataset adapters and LLM Judge

  • Add explicit adapters for supported datasets.
  • Normalize dataset records into a unified internal representation.
  • Add schema validation and clear errors for unsupported or malformed input.
  • Preserve document-level evidence semantics for retrieval evaluation.
  • Handle 2Wiki supporting-fact references safely while retaining valid document evidence.
  • Allow duplicate question text when records have different dataset IDs.
  • Add metric-level selection for generation, retrieval, and indexing metrics.
  • Add resumable Judge runs with failed-sample retry support.
  • Add validation for NaN, infinity, empty results, and invalid metric outputs.
  • Improve dataset digest, adapter version, and per-metric parameter validation during resume.
  • Classify non-retryable LLM request errors correctly.
  • Avoid redundant retry loops between the SDK, LLM client, and metric layer.
  • Print final aggregate values for each selected metric.

3. External baseline integrations

  • Integrate the supported external GraphRAG-style baselines into the benchmark workflow.
  • Keep project-specific adapters and output layouts explicit.
  • Improve reproducibility of external project evaluation.
  • Remove obsolete references to historical project layouts and inactive entry points.
  • Keep generated outputs, caches, datasets, virtual environments, and secrets outside version control.

4. NarrativeQA support

  • Add a consistent NarrativeQA dataset workflow.
  • Define the default metrics appropriate for inferred answers.
  • Normalize NarrativeQA records through the same adapter interface.
  • Keep unsupported evidence-dependent metrics disabled or explicitly rejected.

5. Public release and documentation

  • Update the public README and usage documentation.
  • Document upload, search benchmark, QA benchmark, and LLM Judge workflows.
  • Document supported CLI parameters and metric selection.
  • Add reproducibility instructions for SAG, BM25, embedding-based retrieval, Triple Indexing, and expansion ablation experiments.
  • Update release metadata for version 2.0.0.
  • Remove private paths, credentials, local run IDs, infrastructure details, and development-only instructions from public documentation.

Compatibility

  • Existing public import paths are preserved where practical.
  • Existing result artifacts remain readable through explicit path arguments.
  • New adapters and Judge workflows provide validation errors instead of silently accepting malformed data.
  • No private credentials or generated experiment outputs are included in this PR.

Validation

The following checks passed before preparing this PR:

  • Full test suite: 620 passed
  • Ruff lint check
  • Ruff format check
  • Python compilation check
  • Configuration and dataset-adapter type checks
  • Metric validation checks
  • Credential and public-repository hygiene checks

Review focus

Please review:

  1. The new storage/runtime boundaries.
  2. Dataset adapter contracts and document-level retrieval semantics.
  3. Judge resume and failed-sample retry behavior.
  4. Metric validation and error handling.
  5. Public documentation and release metadata.
  6. Backward compatibility of existing imports and CLI workflows.

Out of scope

  • Changes to private deployment infrastructure.
  • Changes to external service credentials or endpoints.
  • Generated datasets, caches, model outputs, and local experiment artifacts.
  • Unrelated changes to upstream algorithm implementations.

@Zleap-Admin
Zleap-Admin merged commit 942f498 into Zleap-AI:main Aug 28, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants