Repository navigation
[Quality] GLM-5.2 quality benchmarks (#36) merged with vng-benchmark - #39
Open
DinhKhangHoang wants to merge 48 commits into
Open
DinhKhangHoang wants to merge 48 commits into
DinhKhangHoang wants to merge 48 commits into
Conversation
Add GPQA-Diamond, MMLU-Pro, HLE, LiveCodeBench, BFCL v4, SciCode, SWE-bench Pro, and DeepSWE as new quality-eval scenario types. - 8 adapted benchmark scripts under benchmarks/single_node/quality/ - launch_quality-eval.sh: auto-setup (uv, venvs, repo clones), persistent cache at ~/.quality-eval-cache, result collection - launch_bench-client.sh: route SCENARIO_SUBDIR=quality/ to quality-eval - benchmark-tmpl.yml: quality-eval inputs, env vars, SCENARIO_SUBDIR routing, expanded upload patterns, --smoke in verify step - run-sweep.yml: sweep-quality-evals fan-out job - validation.py: 8 new Fields enum values, quality-eval Pydantic models - generate_sweep_configs.py: quality-eval generation in both sweep funcs - process_changelog.py: quality_evals bucket + routing - thresholds.yaml: thresholds for all 8 benchmarks (default + glm5.2) - validate_scores.py: --smoke flag, leniency when checked==0 - nvidia-master.yaml: glm5.2-quality-eval entry - runners.yaml: cluster:quality-eval -> bench-client_01 - ci-priority.yaml: quality-eval: 0.0 - Task configs: HLE YAMLs + utils.py, SWE-bench Pro config + runner
- nvidia-master.yaml: GPQA endpoint -> https://tokenplan.api.greennode.ai/v1 with smoke: true - benchmark-tmpl.yml: wire GN_TOKEN_PLAN secret as QUALITY_API_KEY - run-sweep.yml: eval-limit for sweep-enabled -> 10 (was 4) - perf-changelog.yaml: add glm5.2-quality-eval / quality-gpqa entry
The self-hosted runner on this machine is registered as bench-client_00, not bench-client_01.
…-name routing
Quality-eval jobs can be picked up by any runner with the cluster:quality-eval
label (e.g. h200-greennode_03), not just bench-client_00. The existing
launch_${RUNNER_NAME%%_*}.sh routing assumes a matching launch script exists
for every runner name prefix, which fails with exit code 127 when a runner
like h200-greennode_03 picks up the job.
Route quality-eval scenarios (SCENARIO_SUBDIR=quality/) directly to
launch_quality-eval.sh, independent of the runner name.
质量评估任务可被任何带有 cluster:quality-eval 标签的 runner 领取（如
h200-greennode_03），而非仅 bench-client_00。现有基于 runner 名称前缀的
launch 脚本路由在无匹配脚本时以退出码 127 失败。将质量评估场景直接路由至
launch_quality-eval.sh，不再依赖 runner 名称。
lm-eval's openai-chat-completions model requires the 'tenacity' package which is only installed via the 'api' extra (lm-eval[api]). Without it, the GPQA run fails with ModuleNotFoundError. Also bust the venv cache if tenacity is missing, so existing broken venvs are automatically repaired. lm-eval 的 openai-chat-completions 模型需要 'tenacity' 包，该包仅通过 'api' extra 安装。缺少时 GPQA 运行以 ModuleNotFoundError 失败。同时 在缺少 tenacity 时重建 venv 以自动修复。
Add smoke run configurations for MMLU-Pro (20 Qs, ccu4), HLE (8 Qs, ccu4), LiveCodeBench (4 Qs, ccu4), BFCL (4 Qs, ccu1), SciCode (2 Qs, ccu2), SWE-bench Pro (1 task), and DeepSWE (1 task) against GreenNode endpoint. Introduce num-concurrent and eval-limit fields to the quality-eval config schema so per-benchmark API concurrency and sample counts can be specified in the search-space. Wire NUM_CONCURRENT through launch_quality-eval.sh to each benchmark script's concurrency env var (MULTIPROCESS, NUM_THREADS, MAX_CONNECTIONS, WORKERS, CCU). 为其余 7 个基准添加冒烟运行配置：MMLU-Pro（20 题, ccu4）、HLE（8 题, ccu4）、LiveCodeBench（4 题, ccu4）、BFCL（4 题, ccu1）、SciCode（2 题, ccu2）、SWE-bench Pro（1 任务）、DeepSWE（1 任务），使用 GreenNode 端点。 在 quality-eval 配置模式中引入 num-concurrent 和 eval-limit 字段，以便在 search-space 中指定每个基准的 API 并发数和样本数。
deepswe: correct repo URL datacurve/deep-swe → datacurve-ai/deep-swe scicode: replace invalid --retry-on-error flag with --max-retries swebench: correct PyPI package minisweagent → mini-swe-agent hle: add Pillow to lm-eval venv (HLE dataset has image features) livecodebench: pin anthropic<0.40 (HUMAN_PROMPT removed in newer SDK) bfcl: add soundfile to BFCL venv (qwen_agent dependency) 修复 6 个基准冒烟设置的错误： deepswe 仓库地址 datacurve/deep-swe → datacurve-ai/deep-swe scicode 无效参数 --retry-on-error → --max-retries swebench PyPI 包名 minisweagent → mini-swe-agent hle lm-eval 虚拟环境缺少 Pillow lcb anthropic SDK 版本过新导致 HUMAN_PROMPT 导入失败 bfcl 虚拟环境缺少 soundfile
…e flag
venv: add --clear to all 'uv venv' calls so cache-bust rebuilds
don't fail on 'virtual environment already exists'.
Cache is still reused when the bust-check passes (no rebuild).
deepswe: map EVAL_LIMIT -> N_TASKS (was hardcoded to 6, ignoring
eval-limit: 1 from config); add num-concurrent: 1 to config.
swebench: add num-concurrent: 1 to config for consistency.
scicode: drop invalid --continue-on-fail flag (not in this inspect-ai
version); --no-fail-on-error already covers it.
修复虚拟环境与冒烟参数问题：
venv 重建时加 --clear，避免 'venv already exists' 报错；
缓存健康时仍跳过重建，直接复用。
deepswe 将 EVAL_LIMIT 映射到 N_TASKS（原先硬编码 6，忽略配置）；
配置补充 num-concurrent: 1。
scicode 移除不支持的 --continue-on-fail 参数。
swebench_pro: cache-bust check now tests 'import yaml' (pyyaml was
missing from stale cached venv).
livecodebench: drop anthropic<0.40 pin (LCB needs >=0.42.0); fix
cache-bust check to test 'import livecodebench'.
scicode: drop --metadata flag (unsupported on runner's inspect-ai).
bfcl: inject z-ai/glm-5.2-FC/-PROMPT into model_config.py after
clone (upstream BFCL doesn't have this model registered).
deepswe: collect_results now picks up result.json (singular) and
copies as results.json for benchmark-tmpl.yml's glob check.
job-naming: add benchmark-name to job display name so each quality
eval job is distinguishable in the CI dashboard.
config: comment out gpqa + mmlu_pro (smoke already passed in
run 33785845762); only dispatch the 6 remaining benchmarks.
…en_toks
swebench_pro: fix import path run.extra→run.benchmarks (mini-swe-agent
2.4.6 moved swebench module to run.benchmarks).
scicode: pin pyarrow<15 (PyExtensionType removed in pyarrow 15+,
cached venv had pyarrow 25.0.1); add 'import scicode'
cache-bust check.
hle: reduce max_gen_toks 32768→4096 in run_hle.sh + both task
yamls (HLE answers are ~400 tokens; 32K caused GreenNode
to allocate excessive resources and stall).
config: comment out deepswe (smoke passed in run 33832017880);
only dispatch 5 remaining: hle, livecodebench, bfcl,
scicode, swebench_pro.
BFCL CLI has no --limit flag. When LIMIT is set, generate test_case_ids_to_generate.json with first N IDs per requested category and pass --run-ids. Without this, EVAL_LIMIT=4 was ignored and BFCL ran full eval (hundreds of test cases).
…ults.json + scicode pyarrow
swebench_pro: clone with --recurse-submodules (SWE-agent is a git
submodule); init submodule if cache missed it; generate
instances.yaml via helper_code/generate_sweagent_instances.py
if missing (file is not in repo, must be generated from
HuggingFace dataset).
livecodebench: pin anthropic<0.40 (anthropic>=0.42 removed HUMAN_PROMPT
which lcb_runner still imports); cache-bust check now
verifies 'from anthropic import HUMAN_PROMPT'.
bfcl: write results.json wrapper from CSV scores so
benchmark-tmpl.yml's 'ls results*.json' check passes
(BFCL only produces CSVs, not JSONs).
scicode: install pyarrow<15 + datasets==2.14.4 BEFORE scicode
package (datasets pulls pyarrow>=15 which removes
PyExtensionType); cache-bust check now verifies
pyarrow.PyExtensionType exists.
…aw_sample path
livecodebench: instead of pinning anthropic<0.40 (conflicts with
livecodebench pyproject anthropic>=0.42.0), patch
self_repair.py and test_output_prediction.py to add
try/except ImportError fallback for HUMAN_PROMPT/AI_PROMPT
(code_generation.py already has it).
scicode: pin numpy<2 alongside pyarrow<15 + datasets==2.14.4.
pyarrow 14.0.2 is compiled against numpy 1.x; when scicode
pulls numpy>=2, pyarrow fails with 'AttributeError:
_ARRAY_API not found'.
swebench_pro: fix RAW_SAMPLE default path from
data/swebench_pro_raw_sample.jsonl (does not exist) to
helper_code/sweap_eval_full_v2.jsonl (731 instances,
shipped with the repo).
…name
livecodebench: LCB main.py has no --limit flag (uses --n for sample
count). Remove LIMIT handling entirely.
scicode: inspect-ai requires datasets>=2.16, but SciCode pyproject
pins unpinned 'datasets' → resolver picks 2.14.4.
Fix: install datasets>=2.16 + openai>=3.1 + anthropic +
config + other deps first, then scicode with --no-deps.
swebench_pro: OUT_DIR used 'swebench-pro' (hyphen) but collect_results
looks in 'swebench_pro' (underscore). Align to
swebench_pro so result files are found.
LCB has a hardcoded LanguageModelStore dict and z-ai/glm-5.2 is not in it, causing KeyError. Patch lm_styles.py to add z-ai/glm-5.2 as an OpenAIChat model (uses OpenAIRunner which respects OPENAI_BASE_URL env var for the endpoint).
…or swebench+lcb - LCB: patch code_generation.py to pass release_version as config name to load_dataset (cache has 'release_latest-version_tag=release_latest', not 'default-version_tag=release_latest') - SciCode: pin datasets==5.0.1 + pyarrow==25.0.1 (2.16.1 has TypeError bug with SciCode1/SciCode dataset in generate_from_dict) - SWE-bench Pro + LCB: copy eval_results.json / first .json as results.json so benchmark-tmpl.yml's 'ls results*.json' check passes LCB: 修复 load_dataset config name SciCode: 固定 datasets==5.0.1 SWE-bench Pro + LCB: 复制 results.json
…ection - LCB: patch main.py to read LCB_LIMIT env var and slice benchmark[:N] (LCB CLI has no --limit flag; without this it runs all 1055 problems) - SciCode: collect inspect-ai log files (.eval/.json) from logs/ subdir as results.json so benchmark-tmpl.yml's glob matches LCB: 添加 LCB_LIMIT 环境变量支持 SciCode: 收集 inspect-ai 日志文件
Patch all benchmark harnesses to use stream=True when calling the LLM API, avoiding proxy timeouts on long generations (32K tokens SciCode, 8K LCB, etc.). Each harness now accumulates SSE chunks into a response object matching the non-streaming format. - lm-eval (GPQA/MMLU-Pro/HLE): add stream=True to payload, parse SSE in model_call/amodel_call via _parse_sse_stream helpers - LiveCodeBench: stream=True in oai_runner.py, accumulate delta.content - BFCL: stream=True in generate_with_backoff, _accumulate_stream helper - SciCode/inspect_ai: async stream iteration in openai.py provider - SWE-bench Pro + DeepSWE: stream=True in litellm.completion, _accumulate_litellm_stream helper 为所有 8 个质量基准测试启用流式传输
LCB: patch path_utils.py to use LCB_OUTPUT_DIR env var as base for output path, so results land in $OUT_BASE (where collect_results looks) instead of $LCB_DIR/output/ (relative to cwd). SciCode: validate_scores.py now skips non-dict metrics values (inspect-ai log has results.total_samples=int, results.scores=list, not dict). Also convert inspect-ai log to validate_scores.py-compatible format in collect_results for scicode. 修复 LCB 输出路径 + SciCode 验证分数非字典指标
LCB _eval.json is a list [{"pass@1": 1.0, ...}, ...] but validate_scores.py
calls data.get("results", {}) which crashes on list. Add conversion in
collect_results for livecodebench: extract pass@k from list[0] into
{"results": {"livecodebench": {"pass@1": 1.0}}}.
转换 LCB 列表输出为字典格式以通过验证
为 HLE 启用正确的流式响应并保留推理增量，避免空响应。
仅重新运行 HLE 冒烟测试，并禁用已通过的 SWE-bench Pro 场景。
修复 HLE 最终答案提取，并提高 HLE 与 MMLU-Pro 的生成预算；仅调度三个 lm-eval 冒烟基准。
将 HLE 的 lm-eval HTTP 总超时提高到 1800 秒，避免长推理流在默认 300 秒后被 aiohttp 中止。仅保留 HLE smoke 场景重新验证，并补充回归测试与性能变更记录。
提高质量基准生成上限，使 BFCL 子集评分感知所选类别，并延长 SWE-bench Pro Docker 启动超时。
将本次重跑限定为 SciCode、BFCL 和 SWE-bench Pro 三个失败的质量基准。
恢复历史 DeepSWE 场景，并将新的重跑条目限定为三个失败的基准。
Port the quality-eval benchmarks onto the infx/ layout from vng-benchmark:
- matrix: quality scenarios in infx/matrix/{generate,validation,plan}.py,
quality rows go to a new quality_evals plan bucket
- evals/results: native quality scores in infx/evals/validate_scores.py,
custom quality metrics in infx/results/collect_eval_results.py,
SSE streaming patch in infx/evals/patches/lm_eval_sitecustomize.py,
GLM-5.2 and quality-benchmark thresholds in infx/evals/thresholds.yaml
- utils/ files stay the vng-benchmark compat shims
- workflows: quality env read from the row config (inputs.config),
quality rows routed to runners/launch_quality-eval.sh, sweep and e2e
quality jobs pass secrets explicitly (GREENNODE_API_KEY)
- configs/nvidia-master.yaml and perf-changelog.yaml: vng-benchmark plus
the glm5.2-quality-eval entries
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Khang Hoang <khanghd@greennode.ai>
|
Thanks for the contribution!
中文感谢你的贡献！
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #36 (by @Noridom1): the quality-eval benchmarks for GLM-5.2 on VNG Cloud MaaS HCM, merged with the current
vng-benchmarkto resolve the conflicts. #36's commits are kept as they are; this PR adds one merge commit.What the merge changes
vng-benchmarkmoved the matrix, eval and results code intoinfx/, and theutils/...files there are now compat shims. So the quality-eval code is ported to the new paths:infx/matrix/{generate,validation,plan}.py): adds the quality scenario types.build_plansends quality rows to a newquality_evalsbucket.infx/evals/validate_scores.py):infx/results/collect_eval_results.py):infx/evals/patches/lm_eval_sitecustomize.py):infx/evals/thresholds.yaml):glm5.2blockbenchmark-tmpl.ymlreads the quality env from the row (inputs.config) and routes quality rows torunners/launch_quality-eval.sh.run-sweep.ymlande2e-tests.ymlfollow the$/convention with explicit secrets, addingGREENNODE_API_KEY.configs/nvidia-master.yamlandperf-changelog.yaml: thevng-benchmarkversions plus theglm5.2-quality-evalentries from [Quality] Balanced Coding + Agentic Coding | GLM-5.2 | VNG Cloud MaaS HCM | CCU4 #36.Checks
generate test-config --config-keys glm5.2-quality-evalproduces 5 rows, identical to the generator on [Quality] Balanced Coding + Agentic Coding | GLM-5.2 | VNG Cloud MaaS HCM | CCU4 #36.infx.matrix.plan(basevng-benchmark) puts the same 5 rows inquality_evals.validate_perf_changelogpasses.ruff check infxis clean, andruff formatflags no new files.pytest utils/ runners/ experimental/CollectiveX/tests/has no new failures againstvng-benchmark.Notes
GREENNODE_API_KEYsecret for the quality jobs.🤖 Generated with Claude Code