Skip to content

feat: 补全本地 K8s Ray 与 Daft 可观测性 - #26

Merged
FANNG1 merged 5 commits into
mainfrom
feat/k8s-ray-observability
Sep 2, 2026
Merged

FANNG1 merged 5 commits into
mainfrom
feat/k8s-ray-observability

Conversation

@FANNG1

@FANNG1 FANNG1 commented Sep 2, 2026 •

Copy link
Copy Markdown
Owner

背景

本地 K8s/Ray 音频流水线需要统一查看 Ray 集群、Ray 进程和 Daft 作业指标,并提供无需重复登录的本地 Grafana 入口。

改动

  • 新增 Pushgateway,将 Daft 音频作业的吞吐、实时倍率、ASR/LLM 延迟、状态行数和 Ray RSS 峰值写入 Prometheus。
  • 扩展 Prometheus 抓取 Ray head/worker 与 Pushgateway 指标。
  • 新增 Daft/Ray 概览和 Ray 运行时进程诊断 Grafana Dashboard,支持节点与组件筛选、资源阈值和匿名只读访问。
  • 在 smoke 脚本中注入指标推送地址,并补充本地访问说明。

验证

  • python -m py_compile benchmark/audio/metrics.py benchmark/audio/pipeline.py
  • K8s 与 Grafana Dashboard YAML/JSON 解析通过。
  • python -m pytest tests/benchmark/test_submit.py tests/storage/test_io.py -q:16 passed。
  • 已执行真实 Daft-on-Ray 音频 smoke:4/4 status=ok,Lance Blob v2 校验通过。
  • 已验证 Pushgateway、Prometheus 和 Grafana API 均可查询 Daft/Ray 指标;Grafana 匿名访问为只读。

@FANNG1
FANNG1 merged commit aa91039 into main Sep 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant