Skip to content

feat: add detailed token usage monitoring and cache write metrics - #354

Merged
Ed1s0nZ merged 5 commits into
AIPentest:mainfrom
Sycun:codex/token-usage-monitoring
Oct 16, 2026
Merged

Ed1s0nZ merged 5 commits into
AIPentest:mainfrom
Sycun:codex/token-usage-monitoring

Conversation

@Sycun

@Sycun Sycun commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

The dashboard previously showed only a token total and sent its token card to chat, while cache-write usage was lost between model responses and persisted summaries. This adds a Token Usage subsection in System Settings with 7/30/90-day ranges, input/output/cache-read/cache-write/reasoning counts, cache-read ratio, model shares, daily trends and recent run summaries. The dashboard card opens that subsection and shows a fixed seven-day total, with only cache hit rate in the subtitle (cache-read tokens divided by input tokens). Call counts and daily usage remain in the detail view.

The detail view now separates four primary metrics from secondary counters, adds an interactive daily chart (tokens, calls or cache-read ratio), and visualizes cached versus remaining input in a ring. Model rows show proportional bars, daily numerical details are collapsed by default, and recent runs use status badges. Charts expose exact values on hover or keyboard focus, support mobile/dark themes, and use the API's server-local day keys, filling only gaps between returned dates. Chart metric changes require no extra API request. The design uses publicly verifiable LiteLLM/Langfuse dashboard patterns and DeepSeek cache semantics; the research and limits are documented. It loads independently, supports manual refresh, project scoping and cancellation, and rejects stale responses after rapid period changes or navigation away. Selecting a period does not refresh the whole dashboard. The subsection supports direct links and page refresh through #settings?section=token-usage, keyboard navigation, and Chinese/English/Russian labels. Dashboard polling and access filtering are retained.

Upgrade Eino to stable v0.9.21 and the corresponding agentic model adapters to propagate CacheWriteTokens through stream accumulation, SQLite persistence, APIs, chat summaries and WebShell timelines. Existing startup schema reconciliation adds the new column with a zero default. DeepSeek's prompt_cache_hit_tokens is mapped to standard OpenAI cache-read details for JSON and SSE responses, preserving existing standard fields and raw JSON number precision. Cache details and reasoning are subsets and are never added again to the total; missing totals fall back to input plus output.

The view is explicitly scoped to recorded Agent output usage, including collected sub-agent output. It is not a provider bill or a complete API-request ledger. Model grouping uses run configuration, so failover runs may contain mixed model usage. Unreported usage and internal calls without recorded output are excluded. Research, implementation details and follow-up directions are documented in docs/zh-CN/token-usage-monitoring.md.

Validation:

  • go test ./... passed for the backend changes.
  • node --test web/static/js/*.test.cjs — 298 passing tests after the final UI changes.
  • Regression coverage for cache-write persistence/aggregates, schema repair, conversation access isolation, stream deduplication, total fallback and DeepSeek JSON/SSE compatibility.
  • Added cache-ratio boundaries, daily timeline gap handling and late response/cancellation coverage for the independent detail loader.
  • Browser checks with synthetic data: dashboard cache ratio, card-to-settings navigation, direct-route restoration, keyboard selection, independent period request, chart metric toggles without fetching, keyboard chart details, input cache breakdown, untrusted provider labels, empty/unavailable states, desktop, 390px mobile and dark theme. Earlier checks covered untrusted model-name escaping, empty and unavailable states.
  • git diff --check passed for the changed files.

The frontend cache-version assertions were updated for changed assets; an obsolete HITL migration assertion now checks the canonical schema definition. No runtime configuration, API keys, databases or production deployment changes are included.

@Sycun
Sycun marked this pull request as ready for review October 11, 2026 14:48
@Ed1s0nZ
Ed1s0nZ merged commit 7ad837c into AIPentest:main Oct 16, 2026
6 checks passed
@Sycun
Sycun deleted the codex/token-usage-monitoring branch October 16, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants