perf: the first view of a detected_level volume over | json runs cheaper probes

[szibis]

Summary

Follow-up to #760. The first view of a window (nothing remembered; probed again
every five minutes) of sum by (detected_level) and sum by (level, detected_level) over | json, on rows with no stored level, ran the
stored-level query, the level probes and the presence probe before the chain,
and was several times slower than main before #760. Three cuts, none of which
changes an answer:

1. The two "two level keys" patterns of the level keys probe run behind a filter
   of phrases (logsqlTwiceFilter); a base too long for the gated text, or a
   VictoriaLogs that refuses it as too long, gets the probe without the gate.
2. The label probes (main's partial-parse probe among them) are remembered apart
   from the level probes, so the chain takes their verdict from the stored-level
   query instead of asking the same question again.

Explore's JSON volume on the harness's backfill without the level stream label, a
proxy started fresh before each request, VictoriaLogs capped at 2 CPUs, median
[min-max] of 7 (1h) and 5 (24h) requests, VictoriaLogs cgroup CPU in brackets.
Every sample is committed: bench/ab/results/oneoff/2026-10-16-dlfirstview-coldrestart.json,
produced by python3 bench/ab/cold_first_view.py cold --build before=054d1706 --build main=d12a062c --build head=HEAD --runs 1h=7,24h=5 --out <dir>:

shape	main before #760 054d1706	main now d12a062c	this branch
D (Explore, sum by (level, detected_level)) 1h	119 ms [106-175] (345 CPU-ms)	502 ms [439-526] (1096)	319 ms [314-414] (744)
D 24h	2.99 s [2.92-3.04] (6.02 CPU-s)	11.7 s [11.6-11.7] (23.5)	7.80 s [7.73-8.01] (15.7)
DLV5 (sum by (detected_level)) 1h	125 ms [109-214] (348)	403 ms [307-501] (870)	218 ms [190-291] (512)
DLV5 24h	2.99 s [2.80-3.11] (6.10)	8.30 s [8.20-8.53] (16.7)	4.90 s [4.68-5.15] (9.95)

It is still several times slower than main before #760 on this data (the chain
reads every line, where main's query answered with the wrong series; see "What
is left"). Warm times do not move: this branch against d12a062c, both orders,
D 24h on this data 2.80 s -> 2.84 s and 3.05 s -> 3.04 s (the 2.48 s -> 3.26 s
widening seen in #760's measurement was a busier host: the same two builds are
flat here).

Changes

change	what
perf: the two-keys patterns of the level keys probe run behind a gate of phrases	detectedLevelKeysRisk (a level key twice in a JSON or logfmt line, key(?s:.*)key) scans the rest of every line that holds a level key, which is every line of a body-only window: 325 CPU-ms of an hour of rows, the heaviest probe of the first view. It now runs behind logsqlTwiceFilter(detectedLevelKeyNames) (one source with the patterns): _msg:seq("a","a") for a name twice and (_msg:"a" (_msg:"b" or ...)) for two names, answered from the token index; 45 CPU-ms (the scan alone is 21). The gate is a superset of the patterns, never a second decision: each occurrence a pattern matches is a key name between non-word characters, which is a phrase of VictoriaLogs' tokenizer (getPhrasePos in lib/logstorage/filter_phrase.go, identical in v1.40, v1.47 and v1.52), and two occurrences do not overlap, so either seq finds the second name after the first or two names are both present. 2.5 KB of query text.
perf: the gate leaves out the pair of a dotted key name and its last word	The one occurrence of log.level holds the word level, so the pair of the two admitted every line with that single key. A line with both holds the word twice and passes seq("level","level"); the gate stays a superset.
fix: a selector too long for the gated keys probe gets the probe without the gate and fix: the keys probe is asked again without its gate when VictoriaLogs refuses the longer text	The gate adds 2.5 KB. A translated LogsQL base of 11.8 KB or more would have its gated probe refused by -search.maxQueryLen (16 KiB): it gets the text without the gate, the same verdict; the edge is pinned with bases of exactly the longest that fits and one byte more. An operator who lowered -search.maxQueryLen below the gated probe gets VictoriaLogs' 400 for a query that is too long; that refusal (and no other) sends the probe once more without the gate.
perf: the probes of the labels run once for the stored-level query and the chain	A plan that groups by a label and reads detected_level asks twice on rows with no stored level (the stored-level query, then the chain). The verdict of the label probes was remembered under a key that included whether the level probes run, so main's partial-parse probe (55 CPU-ms of an hour) ran in both asks. The label probes and the level probes are remembered apart (cachedStatsPushdownRisk, rememberedRisk, anyRisk); the verdict is risky if either is. The part of a window younger than five minutes is still asked in both, as every remembered verdict is.
test: the remembered risk verdicts are scoped to the base and to the filters	Pins what the two keys hold, which no test did (on main too): a window found clean for one base, or for one set of filters that reject unparsed lines, does not clear another in the same tenant and window.
test: the e2e of the keys gate writes to VictoriaLogs only	The e2e of the gate reads nothing from Loki and no longer pushes to it.
test: a script that measures the first view ..., test: the first view of a window, every sample ...	bench/ab/cold_first_view.py (usage in bench/ab/README.md) and its two outputs, 2026-10-16-dlfirstview-coldrestart.json and -probes.json under bench/ab/results/oneoff/.
test: A/B results ..., docs: ..., test: visual proof pages ...	Warm results under bench/ab/results/2026-10-16-dlfirstview-*.json (the other order and the body-only runs under oneoff/), the [Unreleased] entry, the registry row, two Explore pages and a 5m range for bench/visual/spec.json.

Per probe, VictoriaLogs CPU per request (one first view of D at 1h, each request
replayed 10 times on windows shifted by a minute, median; main now d12a062c and
this branch; 2026-10-16-dlfirstview-probes.json, from python3 bench/ab/cold_first_view.py probes --build main=d12a062c --build head=HEAD --out <dir>):

request	main now	this branch
keys probe	325 ms	46 ms
partial-parse probe (main's)	56 ms, twice per first view	55 ms, once
unpack probe	34 ms	35 ms
stream guard	0.2 ms	0.2 ms
presence probe	84 ms	86 ms
stored-level query	221 ms	221 ms
chain	265 ms	279 ms

Tests (each fails without its change)

• Gate: TestTwiceGateIsASupersetOfTheKeyPatterns (every ordered pair of the 14
  key names in 16 JSON and logfmt shapes, with tab, CR, form feed, newline,
  no-break space and near spellings, and 300,000 random lines; the gate evaluated
  from the text detectedLevelTwiceGate() returns, with getPhrasePos and
  matchSequence of VictoriaLogs written out from their source, an invalid byte
  counting as a token rune), TestTwiceGateRejectsSingleKeys (a single
  log.level key is rejected; two log.level keys, and log.level with
  level, pass), TestTwiceGateOmitsTheDottedPair, TestTwiceGateNarrowedIsCaught
  (six narrowings of the gate text, each caught by the corpus),
  FuzzTwiceGateIsASupersetOfTheKeyPatterns (30 s, no finding),
  TestKeysRiskProbeGatesTheTwoKeyPatterns. e2e on real VictoriaLogs v1.52.0
  and v1.47.0: TestRangeMetricCompatibilityDetectedLevelKeysGate (18 lines, one
  stream each: the probe the proxy sent and the same without the gate give the
  same verdict, which is the table's, per line).
• Long base and refusal: TestKeysRiskProbeGateEdgeOfTheQueryLength (bases of
  exactly 16384 minus the gated probe's length and one byte more),
  TestLongSelectorKeepsEveryRequestWithinTheQueryLimit (a 12.5 KB selector
  through the pushdown: every request within the limit, the right answer),
  TestKeysRiskProbeRetriesWithoutTheGateWhenRefusedAsTooLong (a fake
  VictoriaLogs that refuses long queries: one retry without the gate; a refusal of
  both is returned; another 400 is not retried).
• Label probes: TestLabelProbesRunOnceForTheAttemptAndTheChain (a cold window:
  one partial-parse request, none on the next ask; a line VictoriaLogs does not
  parse as Loki does, text after the closing brace, still sends the query to the
  raw-row evaluator), and the three guard-count tests that pinned the repeat
  (TestOrderedJSONFilterPushdownUsesStatsBuckets,
  TestOrderedJSONLogsVolumeUsesStatsBuckets,
  TestLevelVolumePlainAndLogfmtUseStatsBuckets) now expect one.
• Scope of the remembered verdicts: TestRiskVerdictOfOneBaseDoesNotClearAnother
  (two bases in one tenant and window, only the second holds two level keys: the
  clean one is asked first, the risky one goes to the raw-row evaluator with the
  answer it gets alone) and TestRiskVerdictOfOneFilterDoesNotClearAnother (the
  same for two filters over the same fields, one clean and one on a field a row
  stores while its body has none).
• Mutations (each fails the named tests): no same-key sequences in the gate, 3
  unit tests and the e2e; the last key out of the pairs, 3 unit tests and the e2e
  subtest of severity_text with SEVERITY_TEXT; the sequences never matching
  their own key, 3 e2e subtests; the dotted pair back in the gate, 2 tests; no
  fallback for a base too long, 2 tests; no retry on the refusal, 1 test (2
  subtests); the edge off by one, 1 test; the label key told apart by whether the
  level probes run again, 4 tests; the level key without the base, 1 test; the
  labels key without the filters, 1 test.

e2e against real Loki v3.7.7

Isolated compose project, image built from this branch, /ready and /health
checked, VictoriaLogs restart count 0 before and after every run, a fresh stack
per run (the two runs below are of the first-view cuts as pushed first; after the
last fixes TestRangeMetricCompatibilityDetectedLevelKeysGate was run again on a
fresh v1.52.0 stack: pass):

VictoriaLogs	pattern	result
v1.52.0	the semantics group of ci.yaml	45 tests, 852 subtests, all pass
v1.47.0	TestSetup_IngestLogs|TestRangeMetricCompatibility.*|TestDrilldown_LongRangeVolume_.*|TestCompat_HeavyQueryProtection|TestCompat_PatternLineFilter.*|TestCompat_DetectedLevel.*	30 tests, 361 subtests, all pass

A/B against d12a062c (this branch's own delta; VictoriaLogs v1.52.0, capped at 2 CPUs)

Warm p50 after a cold run (5 runs at 1h, 3 at 6h and 24h; a shape the first pass
moved is measured again), targets interleaved per request, one fresh stack per
run, docker update --cpus 2 on VictoriaLogs, noise settings are CI's (30% and
100 ms). Every run in both orders (base first, branch first), standard data
(a level stream label on every row) and the data without it (--no-level-label).
20 runs, restarts 0 -> 0 on every one, 0 new differences from Loki (the control
set's topk rate by app and DL6's sum by 0.2% to 0.4% differ on main too). The
control set at 1h: 20 shapes, all same, in both orders. Saved as
bench/ab/results/2026-10-16-dlfirstview-*.json (the other order and the body-only runs under
oneoff/). Runs: python3 bench/ab/pr_smoke.py --base d12a062c --ranges 1h --runs 5 --confirm-runs 7 --selection <sel.json> --out <dir> --loki-mem 3g --vl-mem 4g [--no-level-label].

Data with no stored level (what the change acts on)

Warm, with Loki in the loop at 1h:

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	120 ms	84 ms	-30%	135 ms	133 ms	-1%	same
DL6 explore volume, level filter before json	1h	277 ms	278 ms	+0%	287 ms	289 ms	+1%	differs

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
DLV1 drilldown logs volume, json	1h	512 ms	510 ms	-0%	497 ms	510 ms	+3%	same
DLV2 json error filter, detected_level	1h	2.29 s	2.29 s	+0%	2.17 s	2.18 s	+1%	same
DLV3 plain json, detected_level	1h	534 ms	550 ms	+3%	520 ms	515 ms	-1%	same
DLV4 explore json volume, level and detected_level	1h	42 ms	36 ms	-14%	37 ms	45 ms	+22%	same
DLV5 json error dropped, detected_level	1h	63 ms	88 ms	+40%	98 ms	64 ms	-35%	same
DLV6 drilldown detected_level breakdown, json	1h	83 ms	91 ms	+10%	93 ms	57 ms	-39%	same

6h and 24h (VictoriaLogs only):

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	6h	688 ms	692 ms	+1%	744 ms	696 ms	-6%
D volume, no filters	24h	2.80 s	2.84 s	+1%	3.05 s	3.04 s	-0%
DL6 explore volume, level filter before json	6h	1.58 s	1.51 s	-4%	1.67 s	1.72 s	+4%
DL6 explore volume, level filter before json	24h	6.17 s	6.30 s	+2%	6.50 s	6.57 s	+1%

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
DLV1 drilldown logs volume, json	6h	3.33 s	3.30 s	-1%	3.08 s	3.06 s	-1%
DLV1 drilldown logs volume, json	24h	10.17 s	10.37 s	+2%	9.62 s	9.60 s	-0%
DLV2 json error filter, detected_level	6h	8.05 s	7.93 s	-2%	7.61 s	7.63 s	+0%
DLV2 json error filter, detected_level	24h	7.94 s	7.98 s	+1%	7.78 s	7.66 s	-1%
DLV3 plain json, detected_level	6h	3.19 s	3.20 s	+0%	3.15 s	3.19 s	+1%
DLV3 plain json, detected_level	24h	9.97 s	9.89 s	-1%	9.87 s	10.04 s	+2%
DLV4 explore json volume, level and detected_level	6h	184 ms	193 ms	+5%	182 ms	202 ms	+11%
DLV4 explore json volume, level and detected_level	24h	1.04 s	952 ms	-8%	800 ms	795 ms	-1%
DLV5 json error dropped, detected_level	6h	698 ms	687 ms	-2%	692 ms	732 ms	+6%
DLV5 json error dropped, detected_level	24h	2.70 s	2.73 s	+1%	2.91 s	2.83 s	-3%
DLV6 drilldown detected_level breakdown, json	6h	680 ms	650 ms	-4%	736 ms	692 ms	-6%
DLV6 drilldown detected_level breakdown, json	24h	2.76 s	2.69 s	-2%	2.89 s	2.90 s	+0%

Warm, VictoriaLogs alone, 15 runs, 1h, against d12a062c (flat) and against main
before #760 054d1706 (the same gap #760 disclosed, unchanged on the warm path):

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	97 ms	98 ms	+1%	102 ms	95 ms	-7%
DLV5 json error dropped, detected_level	1h	97 ms	92 ms	-5%	98 ms	92 ms	-6%

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	88 ms	104 ms	+18%	83 ms	100 ms	+20%
DLV5 json error dropped, detected_level	1h	89 ms	96 ms	+8%	87 ms	97 ms	+11%

Cold, from the harness (the first request of a shape in a fresh process, one
sample per run; the restart measurement above is the figure to read):

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	517 ms	295 ms	-43%	498 ms	330 ms	-34%	same
DL6 explore volume, level filter before json	1h	656 ms	490 ms	-25%	583 ms	634 ms	+9%	differs

Data with a level stream label on every row (stored-level plan, untouched)

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	118 ms	88 ms	-25%	86 ms	128 ms	+49%	same
DL6 explore volume, level filter before json	1h	288 ms	300 ms	+4%	294 ms	273 ms	-7%	same series, sum ±0.25%

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	6h	641 ms	697 ms	+9%	585 ms	591 ms	+1%
D volume, no filters	24h	2.48 s	2.51 s	+1%	2.38 s	2.39 s	+0%
DL6 explore volume, level filter before json	6h	1.65 s	1.69 s	+2%	1.60 s	1.60 s	-0%
DL6 explore volume, level filter before json	24h	6.59 s	6.54 s	-1%	6.91 s	7.09 s	+3%

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
DLV1 drilldown logs volume, json	1h	540 ms	554 ms	+3%	526 ms	510 ms	-3%	same
DLV2 json error filter, detected_level	1h	2.23 s	2.21 s	-1%	2.31 s	2.30 s	-0%	same
DLV3 plain json, detected_level	1h	538 ms	526 ms	-2%	548 ms	542 ms	-1%	same
DLV4 explore json volume, level and detected_level	1h	52 ms	37 ms	-29%	37 ms	42 ms	+14%	same
DLV5 json error dropped, detected_level	1h	9 ms	9 ms	+0%	7 ms	8 ms	+14%	same
DLV6 drilldown detected_level breakdown, json	1h	13 ms	11 ms	-15%	7 ms	9 ms	+29%	same

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
DLV1 drilldown logs volume, json	6h	3.08 s	3.04 s	-1%	3.23 s	3.26 s	+1%
DLV1 drilldown logs volume, json	24h	9.38 s	9.39 s	+0%	10.24 s	10.15 s	-1%
DLV2 json error filter, detected_level	6h	7.63 s	7.65 s	+0%	8.22 s	8.13 s	-1%
DLV2 json error filter, detected_level	24h	7.70 s	7.69 s	-0%	7.99 s	8.12 s	+2%
DLV3 plain json, detected_level	6h	3.20 s	3.18 s	-1%	3.24 s	3.25 s	+0%
DLV3 plain json, detected_level	24h	9.97 s	9.91 s	-1%	10.06 s	10.03 s	-0%
DLV4 explore json volume, level and detected_level	6h	200 ms	174 ms	-13%	192 ms	192 ms	+0%
DLV4 explore json volume, level and detected_level	24h	740 ms	792 ms	+7%	786 ms	742 ms	-6%
DLV5 json error dropped, detected_level	6h	28 ms	40 ms	+43%	30 ms	31 ms	+3%
DLV5 json error dropped, detected_level	24h	199 ms	230 ms	+16%	189 ms	192 ms	+2%
DLV6 drilldown detected_level breakdown, json	6h	34 ms	28 ms	-18%	31 ms	30 ms	-3%
DLV6 drilldown detected_level breakdown, json	24h	200 ms	190 ms	-5%	190 ms	189 ms	-1%

Control set at 1h, both orders:

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
logs plain	1h	25 ms	21 ms	-16%	24 ms	25 ms	+4%	same
logs | json	1h	40 ms	36 ms	-10%	34 ms	41 ms	+21%	same
logs | logfmt	1h	57 ms	99 ms	+74%	82 ms	46 ms	-44%	same
json volume no filters	1h	56 ms	93 ms	+66%	89 ms	65 ms	-27%	same
logfmt volume	1h	135 ms	122 ms	-10%	123 ms	97 ms	-21%	same
plain volume	1h	13 ms	9 ms	-31%	11 ms	7 ms	-36%	same
stream-label count by app	1h	13 ms	15 ms	+15%	10 ms	11 ms	+10%	same
ungrouped rate	1h	7 ms	8 ms	+14%	8 ms	5 ms	-38%	same
bytes_over_time by app	1h	29 ms	32 ms	+10%	23 ms	24 ms	+4%	same
topk rate by app	1h	22 ms	20 ms	-9%	19 ms	19 ms	+0%	differs
detected_level volume	1h	21 ms	25 ms	+19%	19 ms	19 ms	+0%	same
by service_name	1h	15 ms	15 ms	+0%	14 ms	14 ms	+0%	same
drilldown level volume	1h	11 ms	7 ms	-36%	7 ms	10 ms	+43%	same
drilldown service landing	1h	14 ms	17 ms	+21%	13 ms	13 ms	+0%	same
drilldown detected_level	1h	21 ms	19 ms	-10%	19 ms	19 ms	+0%	same
json rate by level (no filter, no drop)	1h	3 ms	2 ms	-33%	3 ms	2 ms	-33%	same
logfmt count by level, filter, drop	1h	200 ms	245 ms	+22%	198 ms	171 ms	-14%	same
instant ungrouped rate	instant	4 ms	7 ms	+75%	5 ms	5 ms	+0%	same
instant json volume	instant	197 ms	215 ms	+9%	195 ms	190 ms	-3%	same
instant json filtered	instant	166 ms	175 ms	+5%	155 ms	155 ms	+0%	same

The three proofs

Isolated compose project (bench/visual/stack.py up --hours 168 --main-ref d12a062c,
body-only data: the generator's LOG_NO_LEVEL_LABEL=1; Loki holds the last 1.5 h as
the tool documents, so ranges beyond that compare main against this branch),
Grafana with the pinned plugins, three datasources: main (d12a062c), this branch,
Loki direct.

a. Visual and data-to-data

Two Explore pages were added to bench/visual/spec.json for the shapes this change
touches (explore-D, sum by (level, detected_level), and explore-DLV5, sum by (detected_level), both | json | drop __error__), next to the existing pages.
Screenshots of main, this branch and Loki side by side, per page and range
(80 montages, captured locally; the visual-smoke run of this pull request posts its own; for example
explore-D-1h.png, explore-DLV5-1h.png, dd-service-logs-1h.png,
dd-landing-1h.png), and the captured /api/ds/query bodies and responses with the
comparison (bench/visual/compare.py). This change has no
visible effect: the first view of a window is latency, and the data on screen is
the same, so main and this branch render the same panels (pixel difference
explore-D and explore-DLV5 0.04%, which is the datasource name in the header;
dd-service-logs 0.11%, dd-landing 0.14%, 2.1% at 7d). The proof is the data: in all 80 page and range combinations (16 pages x
5m, 1h, 6h, 24h, 7d) the requests of main and of this branch are identical
("main = PR" in the comparison; three cells were captured again after a one-off 500 on
main's high-cardinality span_id/trace_id breakdown at 24h and a retried request
at 7d, and are identical). Against Loki, at 5m and 1h (the ranges Loki holds), 163 of
236 captured requests are identical, the rest explained, history-dependent or
differences that main has too (for example Loki returns no series for the plain
sum by (level, detected_level) ... | drop __error__ on this data and the
logfmt level!="" count differs, 42 against 14). For the shapes this change reads,
Explore D and DLV5 at 1h, the series Grafana receives equal Loki's; at 5m one
series differs from Loki's in its timestamps (30 against 30 points), on main too.
The A/B harness's own comparison against Loki is same for every shape at 1h, on
both orders, for base and branch (series count and sums; DL6 within 0.2% to 0.4% on
both).

b. Real Grafana range matrix

The 16 pages (Explore logs and volumes: plain, | json, | logfmt, the grouped and
ungrouped sums; the label browser; Drilldown landing, service logs, labels, fields,
a field's values and patterns) at 5m, 1h, 6h, 24h and 7d, through Grafana's
datasources (its query splitting included), main against this branch against Loki. Result: main and this branch send the same requests and
receive the same answers in every cell, so no status differs from main's; all 80
captures completed.

c. Backend-safety load

240 s per build against the same VictoriaLogs (2 CPUs not applied here; the stack's
default), 8 concurrent clients drawing from nine shapes (D and DLV5 at 24h on the
body-only data, D at 6h, plain volume, rate by level and bytes at 24h, topk and the
json field volume at 6h, a json log query at 15m), each request with a window of its
own so that no cached answer is served; /labels probed every 2 s; VictoriaLogs
restart count read before and after. Both builds: every request 200, restart count
0 -> 0 (no OOM), /labels answered 119 of 119 times (p50 0.011
s for main and 0.009 s here, max 0.128 s and
0.062 s):

shape	main: requests	p50	max	this branch: requests	p50	max
D 24h	112	3.21 s	12.05 s	111	3.13 s	7.15 s
D 6h	95	2.18 s	5.88 s	96	2.3 s	5.4 s
DLV5 24h	111	3.05 s	8.36 s	111	3.02 s	6.54 s
bytes 24h	115	1.9 s	3.88 s	118	1.94 s	3.64 s
json field volume 6h	105	1.83 s	3.95 s	107	1.83 s	5.23 s
logs json 15m	123	0.07 s	0.53 s	123	0.06 s	0.31 s
plain volume 24h	105	1.67 s	3.41 s	106	1.66 s	3.5 s
rate by level 24h	119	1.7 s	3.54 s	121	1.74 s	3.4 s
topk 6h	108	1.59 s	3.3 s	111	1.61 s	3.07 s

Registry

The row changed (text only; nothing pending, no new rows):

id: semantics/json-metric-detected-level-derivation
title: sum by (detected_level) over a JSON pipeline follows the level Loki's ingest gave each line, by level key in document order, by keyword scan, with a detected_level stream label kept as stored (fixed; a few lines stay different, see semantics/detected-level-pipes-known-differences)
proves:
  - loki_api_v1_query_range
consumer: api
why: |
  Loki sets detected_level once per line at ingest (pkg/distributor/
  field_detection.go extractLogLevelFromLogLine, discover_log_levels on). A
  level stream label or structured metadata decides first. Otherwise a line
  whose first non-space character is `{` and last is `}` is read as JSON: the
  first string value of an allowed key (level, LEVEL, Level, log.level,
  severity, SEVERITY, Severity, SeverityText, lvl, LVL, Lvl, severity_text,
  Severity_Text, SEVERITY_TEXT) at depth 0 or 1, in document order, counts only
  when it is a known spelling (info, INFO, inf and information are `info`).
  Anything else (a number, a boolean, `notice`, no such key, a line that is not
  JSON) falls to a keyword scan of the whole line, which finds the first
  bounded level word; `unknown` is the answer only when the scan finds none.
  The value is structured metadata present on every line, so a body key named
  detected_level is detected_level_extracted, and a detected_level stream label
  keeps its stored value (the derived one moves to detected_level_extracted).

  Measured against Loki v3.7.7 on 50 probe lines (the e2e test pins each):
  `{"level":30,"msg":"hi"}` is unknown; a pino error line
  (`{"level":50,...,"err":{"type":"Error"}}`) is error; `{"level":"notice"}` is
  unknown but `{"level":"notice","msg":"error connecting"}` is error;
  `{"level":" info"}` is info; `{"severity":"ERROR"}`, `{"Level":"Information"}`,
  `{"LEVEL":"warn"}` and `{"log":{"level":"warn"}}` are error, info, warn and
  warn; `{"msg":"x","severity":"error","level":"info"}` is error (document
  order) and `{"level":"info","severity":"error"}` is info; a line without a
  level key that says "error" is error. A JSON line is read for keys only: the
  logfmt-looking text in `{"msg":"error: cannot set level=debug now"}` is no
  level, so it is error. A line that starts with a brace but does not end with
  one is logfmt (`{ error level=info` is info). Loki compares a JSON value's raw
  bytes, so `{"level":"\u0069nfo"}` is unknown, and decodes keys, so
  `{"\u006cevel":"warn"}` is warn. For logfmt the first level key in document
  order wins within a spelling family (`Level=info LEVEL=error` is info). An
  OTLP record is its severity number's level, or its text when the text is its
  own: `Unspecified` or no number is unknown (or the keyword scan of the line),
  number 10 is info, `notice` is notice. The proxy took the stream's spelling as
  stored, read only a body `level` key and made every other value a series of
  its own (`30`, `notice`, ` info`, `true`) or `unknown`.

  VictoriaLogs stores an OTLP record's severity in two shapes: from v1.50
  severity_text (the record's text, or a name it derives from the number:
  `Unspecified` for 0, `Info2` for 10) with severity_number, and before one
  severity field with no number. The derived text is no level field of Loki's,
  so the pipes skip it and the number decides (a record without a severity
  falls to the line, as in Loki). The e2e probes run on v1.47.0 and v1.52.0 and
  cover both shapes; the pairing of a derived name with its number is checked
  by family (a record sending `Info2` with number 12 is read as derived).

  Both routes of `sum by (detected_level) (... | json ...)` now give Loki's
  value for every probe line above, bar the differences recorded in
  semantics/detected-level-pipes-known-differences. The row evaluator (plain `| json`, `__error__!=""`, every pushdown
  fallback) gives each row Loki's structured metadata before any stage runs,
  using the derivation the log lines use (deriveRowLevel), so a body
  detected_level key is renamed and a filter on detected_level
  (`| json | detected_level="info"`) compares the derived value. The stats
  pushdown first groups the buckets by copies of the stored detected_level and
  level fields, taken before any unpack, and gives each bucket the level Loki's
  ingest gives it: the first non-empty of the two, a known spelling in any case
  as its level and any other word as stored (translator.NormalizeStoredLevel, tested against
  the chain's own spelling regexps and against a table of Loki v3.7.7's
  normalizeLogLevel answers). That is main's query with
  no chain and no level guard probe, and it answers every window whose rows
  all carry a stored level (a stream label or structured metadata, as Loki
  pipelines send it); the proxy reads a stored detected_level, then level, and
  a bucket with a level needs no other field of the proxy's list. Loki takes any
  allowed stream label before any structured metadata, so a stream `severity`
  with metadata `level` is error in Loki and info here: a recorded known
  difference (semantics/detected-level-pipes-known-differences), not part of
  this check. A filter on detected_level runs in VictoriaLogs on the stored
  spellings (translator.BuildStoredLevelCopyFilter, which keeps every row the
  stored fields do not decide so that those reach the chain) and again in Go on
  the normalised level. Loki's series limit is applied to the answer's series:
  the spellings of one level (`INFO`, `info`, `Information`) are one series,
  and a read of more than twice the limit of VictoriaLogs groups
  (`errStatsGroupGuard`, a memory guard) is asked again with the chain, whose
  groups are its series. A
  bucket without a stored level, or with a stored detected_level (a stream
  label keeps its spelling, structured metadata is normalised, and the stats
  cannot tell them apart), sends the query again with the pipes of the native
  metric route (translator.BuildDetectedLevelChain), built from the level
  fields and body keys the window holds (a presence probe: one stats query of
  counts per stored field and per body key word, over the same base and
  window, run on every request over the part of the window younger than five
  minutes; an escape sequence in a body makes every key present, a failed probe
  keeps the full chain; the narrowed query flags each row holding a field or key
  the probe left out, a line that arrived late inside the remembered part, and
  is then asked again with the full chain), placed before the unpack of any other label, and a
  guard (detectedLevelStreamRisk,
  detectedLevelKeysRisk, detectedLevelUnpackRisk, cached with the other
  pushdown probes) hands the query to the raw-row evaluator when the window
  holds a line those pipes read differently from Loki: a detected_level stream
  label, two mentions of a level key (the pipes read keys in a fixed order),
  a level key whose JSON value has an escape (Loki reads raw bytes),
  logfmt keys in a line that starts with a brace but does not end with one
  (Loki reads that as logfmt, the pipes do not unpack it), or a level key
  unpack_json yields no value for (nested other than log.level, leading
  whitespace, text after the closing brace, a syntax error). Differences it
  does not see are recorded as open in the known-differences row: a stream
  level field outranked in the pipes' list order by a stored metadata field
  (stream `severity` with metadata `level`), a logfmt level key in another case
  (`LvL=warn`), a logfmt syntax error before the level key, and a row
  VictoriaLogs unpacked from a JSON body. A filter
  on detected_level is applied after the pipes (`!= ""` always holds), also for
  Logs Drilldown's detected_level breakdown, which is not the native histogram
  (the label translator maps detected_level to level, so
  orderedJSONNativeGroupingIsExact is false for it). With
  -detected-level-body-scan=false the raw-row evaluator answers, as only the
  stored fields decide.

  The plain and `| logfmt` logs volume shapes (`sum by (level,
  detected_level)`) take the same plan, and the same pipes when a row has no
  stored level: a stream without a level label gets
  the level of its body (a `level=warn` key or a keyword), as in Loki, where
  main answered `unknown`. The plain shape has no raw-row fallback, so it runs
  without the guard (see semantics/detected-level-pipes-known-differences).

  Cost: a window whose rows all carry a stored level costs what main's query
  costs (two more grouped columns and one copy pipe). Measured against main on
  one hour of the A/B harness's backfill (about 290,000 lines, a level stream
  label on every row) with VictoriaLogs capped at 2 CPUs as on a CI runner,
  warm p50 of 5 to 7 runs, the two builds measured in each order (the second
  target of the harness runs slower by up to 30% even for the same binary):
  Explore's JSON volume 79 ms to 98 ms (101 ms to 76 ms in the other order),
  the volume behind a `detected_level="info"` filter before the parser
  (VictoriaLogs first unpacks logfmt for every line) 181 ms to 279 ms (259 ms to
  245 ms); at 6h and 24h 574 ms to 608 ms and 2.37 s to 2.40 s (596 ms to
  589 ms and 2.25 s to 2.33 s), 1.55 s to 1.60 s and 6.38 s to 6.90 s (1.50 s to
  1.59 s and 5.98 s to 6.45 s).

  A row with no stored level (the level only in the line: the same backfill
  with the level left out of the stream labels, which the harness seeds with
  `--no-level-label`) is read by the chain, which reads every line; main
  answers 4 series for Explore's volume there where Loki answers 8. The chain
  parses each line once, for the plan's labels and for the level: the plan's
  own `unpack_json` and the level keys' are one (translator.SharedUnpack: the
  chain's unpack reads the level keys and the plan's labels with
  result_prefix "__j_", the level keys are read only for rows no stored level
  decides, and before the closing delete each label is copied from its
  prefixed field where the field is not stored, which is what the plan's
  unpack with keep_original_fields wrote; the unpack parses a row only where a
  label is missing as a stored field or the chain reads the line). The keyword
  regexp runs only on lines that hold one of its nine words as a whole word of
  VictoriaLogs' tokenizer (`_msg:(i(trace) OR ... OR i(info))`, a superset of
  the regexp's matches: every byte Loki accepts around a keyword is a token
  separator and the folding of (?i) maps no non-ASCII rune onto the letters of
  the words), and the escape check of the presence probe and its witness is a
  substring filter (`_msg:*"\\u"*`), not a regexp. Measured on one hour of
  that data, VictoriaLogs capped at 2 CPUs, warm p50, VictoriaLogs alone, 15
  runs in each measurement order at 1h: Explore's volume 84 ms to 105 ms (87 ms
  to 102 ms with the branch measured first), the detected_level breakdown with
  the error rows dropped 88 ms to 99 ms (85 ms to 101 ms); at 6h and 24h (3
  runs, both orders) Explore's volume 541 ms to 692 ms and 2.39 s to 2.75 s
  (570 ms to 702 ms and 2.30 s to 2.84 s), the
  breakdown 585 ms to 641 ms and 2.24 s to 2.74 s (589 ms to 698 ms and 2.25 s
  to 2.79 s). The previous revision took 38 ms to 282 ms, 0.55 s to 1.5 s and
  2.3 s to 6.1 s for Explore's volume and 56 ms to 99 ms and 2.2 s to 3.8 s for
  the breakdown. On VictoriaLogs alone, one hour, per query: main's query 102
  ms and 225 CPU-ms (cgroup CPU time; the wall time at 2 CPUs moves in steps of
  the 100 ms quota period), the stored-level plan 100 ms, the previous chain
  290 ms and 550 CPU-ms, this one 107 ms and 260 CPU-ms. Where the CPU of the
  chain went (one group of pipes left out at a time): the plan's own unpack 194
  CPU-ms, the keyword regexp 144, the witness 102 before; the witness 87, the
  keyword regexp 48 and the json unpack 20 now. The detected_level breakdown of
  Logs Drilldown (a filter, `| json | drop __error__`), which main answered
  from raw rows, goes on that data from 2.2 s to 0.05-0.11 s at 1h and from
  7.5 s to 0.67 s (6h) and 2.6 s (24h), and on the data with a level stream
  label from 2.3 s to 7 ms at 1h. The first view of a window (nothing
  remembered: the stored-level attempt, the level probes, the presence probe,
  then the chain) takes 0.31 s at 1h for Explore's volume where main before #760 takes
  0.06 s (0.49 s before the first-view cuts) and 7.6 s at 24h against 2.9 s
  (11.2 s), medians of cold requests with each build restarted before them: the
  probes scan the window and share the 2 CPUs with the chain. The heaviest of
  them was detectedLevelKeysRisk (320-380 CPU-ms of an hour of rows: two level
  keys in a line, an escape in a level value, a brace without its closing
  brace), whose two-keys patterns now run behind a filter of phrases
  (logsqlTwiceFilter over detectedLevelKeyNames: one key name twice, or two of
  the names, answered from VictoriaLogs' token index and a superset of the
  patterns; 45-55 CPU-ms; a selector of 11.8 KB or more, which would push the
  longer text past -search.maxQueryLen, gets the probe without the gate), and
  the probes of the labels are remembered apart from the level probes
  (cachedStatsPushdownRisk), so the chain does not ask again what the
  stored-level query asked. A window with a row that has no stored level asks
  twice, the stored-level attempt first (it costs what main's query costs; its
  verdict is remembered for the base for five minutes so the next ask goes to
  the chain at once), and pays the chain with the three guard probes and the
  presence probe. Each guard probe stops at its first matching line (`limit 1`),
  which bounds what it returns, not the work: on a clean window it scans the
  lines that hold a level-key word (VictoriaLogs' word index narrows them) and
  the rows without a stored level. The guard is a superset of the lines the
  pipes read differently: a body with both `level` and `severity` (audit logs)
  trips it although the pipes might agree. One such line sends the whole query
  to the raw-row evaluator, which is exact and bounded by the row limit, so a
  long range answers 502 where the pushdown answered 200. The chain is about
  7.1 KB of LogsQL unpruned and 1.5 KB for a window with one stored level field
  and no body key, plus the witness (up to 1.5 KB, none when the window holds
  every field and key): with a `service_name` selector Explore's volume sends
  4.2 KB (9.8 KB unpruned) and a regexp selector of seven long names about
  9.6 KB with the witness (14.4 KB unpruned; those names match no row, so the
  chain is at its smallest), against VictoriaLogs' 16 KiB -search.maxQueryLen (a
  longer query answers 502; a window whose presence cannot be told keeps the
  full chain and its size; a unit test pins the witnessed request under the
  limit). The remembered verdicts (guard probes and presence) are kept per
  request scope, cover only the part of a window older than five minutes, and
  expire five minutes after the check of their oldest part, so a refreshing
  dashboard checks the last five minutes on every refresh and the whole window
  again every five minutes.
request:
  method: GET
  path: /loki/api/v1/query_range
  params:
    query: 'sum by (detected_level) (count_over_time({app="json-level-probe"} | json | drop __error__ [1m]))'
    start: now-10m
    end: now
    step: "60"
  headers:
    X-Scope-OrgID: "0"
  precondition: |
    One stream per probe line: pino numeric levels, notice, a padded level, the
    severity, Level, LEVEL and nested log.level keys, two level keys in both
    orders, a line with the word error and no level key, a detected_level
    stream label, a body detected_level key, structured metadata level and
    detected_level.
compare:
  mode: exact
  fields: [status, series_labels, values]
  expect: |
    The series Loki answers for each probe, on `| json` (raw-row evaluator) and
    `| json | drop __error__` (pushdown or its guarded fallback).
gap:
  status: fixed
  fixed_by: "#760"
  impact: explore-visible
  visual: none
  visual_reason: 'the generator data carries one lowercase spelling, the same in the stream label and the body, so main and the branch render the same panel; the e2e probes against Loki are the proof (v1.47 and v1.52)'
  effort: M
  area: 'internal/proxy ordered_json_metric.go (withDerivedLevel, setStatsPushdown, detectedLevel*Risk), stream_processing.go stringRow'
  loki: 'pkg/distributor/field_detection.go extractLogLevelFromLogLine, detectLevelFromLogLine, normalizeLogLevel; pkg/distributor/labels.go (detected_level_extracted)'
  proxy: 'the row evaluator and the stats pushdown read the stream level or a body level key and normalized it'
  planned_test: 'TestOrderedJSONDetectedLevelLikeLoki, TestOrderedJSONDetectedLevelMixedRows, TestOrderedJSONDetectedLevelKnownDifferences (proxy); TestRangeMetricCompatibilityJSONDetectedLevelProbes, TestRangeMetricCompatibilityJSONDetectedLevelSharedUnpack, TestRangeMetricCompatibilityDetectedLevelPipeVariants, TestRangeMetricCompatibilityEscapeSubstringFilter and TestRangeMetricCompatibilityJSONDetectedLevel (test/e2e-compat, group semantics); TestKeywordScanLikeLoki, TestKeywordGateIsASupersetOfTheRegexp, TestBuildDetectedLevelChainShared (translator); TestSharedUnpackKeepsThePlansLabelsLikeLoki, TestSharedLevelUnpackEligibility (proxy)'
  note: |
    Measured on VictoriaLogs v1.52.0 and v1.47.0 against Loki v3.7.7.
  clusters: []

What is left

• The first view is still several times slower than main before fix: sum by (detected_level) over a JSON pipeline groups the level the way Loki's ingest does #760 on this
  data, which is the cost of an exact answer (main's query returns 4 series where
  Loki returns 8): what runs before the chain is the stored-level query (221
  CPU-ms, which answers nothing on rows with no stored level), the presence probe
  (86), the partial-parse probe (55), the keys probe (46) and the unpack probe
  (35), and the stored-level query, the presence probe and the chain run one after
  the other. Estimates, not measured: running the stored-level query beside the
  probes and the presence probe takes its 70 to 140 ms off the wall time of a first
  view at the price of the same CPU, which on 2 CPUs saves about half of that;
  skipping it for a window the presence probe says has no stored level saves its
  221 CPU-ms and about 90 to 140 ms, but needs the presence probe first. Neither is
  done: they change the order of the plan, a separate decision.
• The part of a window younger than five minutes is probed on every ask, so for
  a window ending now the label probes still run twice over that part.
• The stream guard still sends its own request (1 CPU-ms). Reading the presence
  probe's detected_level count instead would save that request, but only if
  detected_level is a stored row field of every stream, which VictoriaLogs'
  ignore_fields breaks: not done.
• Everything fix: sum by (detected_level) over a JSON pipeline groups the level the way Loki's ingest does #760 listed under "Not done" still stands.

[github-actions]

⚪ Performance A/B: no change beyond noise

77 shape×range — 77 ⚪ same · results vs Loki: 62 same, 0 new difference(s), 15 pre-existing

Every shape is within noise of the base build (±30% and 100 ms).

77 unchanged shape×range (within noise)

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
⚪	bytes_over_time by app	1h	52 ms	49 ms	⚪ -6%	72 ms	0.68×	✅ same
⚪	drilldown service landing	1h	22 ms	25 ms	⚪ +14%	21 ms	1.19×	✅ same
⚪	instant json filtered	instant	287 ms	290 ms	⚪ +1%	58 ms	5.00×	✅ same
⚪	instant json volume	instant	347 ms	342 ms	⚪ -1%	77 ms	4.44×	✅ same
⚪	instant ungrouped rate	instant	4 ms	4 ms	⚪ +0%	58 ms	0.07×	✅ same
⚪	json rate by level (no filter, no drop)	1h	400	400	⚪ +0%	400	—	✅ same
⚪	json volume no filters	1h	141 ms	141 ms	⚪ +0%	2 ms	70×	✅ same
⚪	logfmt count by level, filter, drop	1h	340 ms	327 ms	⚪ -4%	43 ms	7.60×	✅ same
⚪	logfmt volume	1h	218 ms	221 ms	⚪ +1%	47 ms	4.70×	✅ same
⚪	logs plain	1h	53 ms	53 ms	⚪ +0%	33 ms	1.61×	✅ same
⚪	logs | json	1h	73 ms	79 ms	⚪ +8%	50 ms	1.58×	✅ same
⚪	logs | logfmt	1h	128 ms	124 ms	⚪ -3%	41 ms	3.02×	✅ same
⚪	stream-label count by app	1h	17 ms	21 ms	⚪ +24%	32 ms	0.66×	✅ same
⚪	topk rate by app	1h	33 ms	33 ms	⚪ +0%	69 ms	0.48×	➖ differs (as base)
⚪	ungrouped rate	1h	8 ms	8 ms	⚪ +0%	64 ms	0.12×	✅ same
⚪	DB3 json field breakdown user_id	1h	35 ms	34 ms	⚪ -3%	20 ms	1.70×	✅ same
⚪	DB4 json field breakdown batch_id	1h	70 ms	70 ms	⚪ +0%	46 ms	1.52×	✅ same
⚪	X1 json filter level_extracted, by level	1h	288 ms	276 ms	⚪ -4%	34 ms	8.12×	✅ same
⚪	X2 json group by level_extracted	1h	218 ms	215 ms	⚪ -1%	36 ms	5.97×	✅ same
⚪	X3 json rate, level_extracted equality	1h	200 ms	200 ms	⚪ +0%	28 ms	7.14×	✅ same
⚪	X4 json bytes, level_extracted regexp	1h	286 ms	283 ms	⚪ -1%	71 ms	3.99×	✅ same
⚪	X5 logfmt filter level_extracted, by level	1h	499 ms	499 ms	⚪ +0%	39 ms	13×	✅ same
⚪	E1 logs, json parser over stream labels	1h	87 ms	89 ms	⚪ +2%	48 ms	1.85×	✅ same
⚪	E2 logs, logfmt parser over stream labels	1h	55 ms	56 ms	⚪ +2%	40 ms	1.40×	✅ same
⚪	E3 logs, json, existence filter on level_extracted	1h	106 ms	100 ms	⚪ -6%	61 ms	1.64×	✅ same
⚪	E4 logs, logfmt, existence filter on level_extracted	1h	65 ms	68 ms	⚪ +5%	43 ms	1.58×	✅ same
⚪	E5 logs, json, drop level_extracted	1h	80 ms	86 ms	⚪ +7%	46 ms	1.87×	✅ same
⚪	E6 logfmt count by level_extracted	1h	34 ms	32 ms	⚪ -6%	6 ms	5.33×	✅ same
⚪	E7 json count by level_extracted	1h	104 ms	101 ms	⚪ -3%	11 ms	9.18×	✅ same
⚪	E8 instant logfmt by level_extracted	instant	6 ms	5 ms	⚪ -17%	7 ms	0.71×	✅ same
⚪	A explore volume, 2 filters, drop	1h	201 ms	144 ms	⚪ -28%	122 ms	1.18×	✅ same
⚪	B grouped sum, filter, no drop	1h	75 ms	74 ms	⚪ -1%	87 ms	0.85×	✅ same
⚪	C ungrouped sum, filter, no drop	1h	75 ms	74 ms	⚪ -1%	87 ms	0.85×	✅ same
⚪	D volume, no filters	1h	152 ms	156 ms	⚪ +3%	102 ms	1.53×	✅ same
⚪	E field breakdown pipeline (explore)	1h	74 ms	77 ms	⚪ +4%	29 ms	2.66×	✅ same
⚪	F field breakdown service_version (explore)	1h	108 ms	121 ms	⚪ +12%	34 ms	3.56×	✅ same
⚪	G field breakdown pipeline (drilldown)	1h	32 ms	32 ms	⚪ +0%	20 ms	1.60×	✅ same
⚪	H field breakdown service_version (drilldown)	1h	61 ms	63 ms	⚪ +3%	21 ms	3.00×	✅ same
⚪	I grafana volume of B	1h	73 ms	77 ms	⚪ +5%	28 ms	2.75×	✅ same
⚪	O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	51 ms	53 ms	⚪ +4%	29 ms	1.83×	✅ same
⚪	DL1 drilldown field breakdown json-expr, level filter	1h	363 ms	366 ms	⚪ +1%	19 ms	19×	✅ same
⚪	DL2 drilldown field breakdown, level + stream-label filter before json	1h	35 ms	39 ms	⚪ +11%	5 ms	7.80×	✅ same
⚪	DL3 drilldown logfmt field breakdown, level filter	1h	445 ms	459 ms	⚪ +3%	81 ms	5.67×	➖ differs (as base)
⚪	DL4 instant count, level filter, json+logfmt, 1h	instant	373 ms	373 ms	⚪ +0%	964 ms	0.39×	✅ sum ±0.27%
⚪	DL6 explore volume, level filter before json	1h	404 ms	408 ms	⚪ +1%	31 ms	13×	✅ sum ±0.26%
⚪	P1 logs, json, level equals	1h	90 ms	87 ms	⚪ -3%	62 ms	1.40×	✅ same
⚪	P2 logs, logfmt, level not equal	1h	22 ms	22 ms	⚪ +0%	4 ms	5.50×	✅ same
⚪	P3 logs, json, level existence	1h	96 ms	98 ms	⚪ +2%	51 ms	1.92×	✅ same
⚪	P4 logfmt count by level	1h	22 ms	21 ms	⚪ -5%	5 ms	4.20×	✅ same
⚪	P5 json count by level	1h	56 ms	59 ms	⚪ +5%	11 ms	5.36×	✅ same
⚪	P6 json count, level filter	1h	11 ms	10 ms	⚪ -9%	10 ms	1.00×	✅ same
⚪	P7 json rate by level, level regexp	1h	13 ms	12 ms	⚪ -8%	12 ms	1.00×	✅ same
⚪	P8 json count by a name that is no stream label	1h	61 ms	61 ms	⚪ +0%	10 ms	6.10×	✅ same
⚪	P9 instant logfmt by level	instant	4 ms	4 ms	⚪ +0%	7 ms	0.57×	✅ same
⚪	E1 json duration_ms sum by service	1h	61 ms	62 ms	⚪ +2%	12 ms	5.17×	➖ differs (as base)
⚪	E2 json duration_ms p95 by service	1h	289 ms	296 ms	⚪ +2%	12 ms	25×	✅ same
⚪	E3 logfmt latency_ms rate, production	1h	375 ms	374 ms	⚪ -0%	93 ms	4.02×	➖ differs (as base)
⚪	E4 logfmt size_bytes max by service	1h	32 ms	33 ms	⚪ +3%	12 ms	2.75×	➖ differs (as base)
⚪	E5 json duration_ms sum, instant	instant	25 ms	23 ms	⚪ -8%	22 ms	1.05×	✅ same
⚪	E6 logfmt ttl unit string (Loki 400)	1h	400	400	⚪ -2%	400	—	✅ same
⚪	E7 json hexadecimal trace_id (Loki 400)	1h	400	400	⚪ +2%	400	—	✅ same
⚪	E8 json duration_ms sum by service, namespace prod	1h	136 ms	136 ms	⚪ +0%	68 ms	2.00×	➖ differs (as base)
⚪	E9 logfmt latency_ms sum by service, namespace prod	1h	234 ms	231 ms	⚪ -1%	68 ms	3.40×	➖ differs (as base)
⚪	U1 json duration_ms sum by service	1h	59 ms	61 ms	⚪ +3%	12 ms	5.08×	➖ differs (as base)
⚪	U10 instant sum_over_time unwrap	instant	23 ms	24 ms	⚪ +4%	21 ms	1.14×	✅ same
⚪	U11 instant max_over_time missing label	instant	4 ms	4 ms	⚪ +0%	12 ms	0.33×	✅ same
⚪	U12 json duration_ms over every service	1h	171 ms	166 ms	⚪ -3%	33 ms	5.03×	➖ differs (as base)
⚪	U13 logfmt latency_ms over every service	1h	380 ms	372 ms	⚪ -2%	45 ms	8.27×	➖ differs (as base)
⚪	U14 duration() of a unit string	1h	31 ms	31 ms	⚪ +0%	5 ms	6.20×	➖ differs (as base)
⚪	U2 json duration_ms max by service	1h	68 ms	68 ms	⚪ +0%	28 ms	2.43×	➖ differs (as base)
⚪	U3 json duration_ms sliding sum	1h	58 ms	62 ms	⚪ +7%	26 ms	2.38×	➖ differs (as base)
⚪	U4 logfmt size_bytes sum by service	1h	31 ms	28 ms	⚪ -10%	6 ms	4.67×	➖ differs (as base)
⚪	U5 logfmt ttl unit string (no sample)	1h	28 ms	28 ms	⚪ +0%	5 ms	5.60×	✅ same
⚪	U6 missing label, no parser	1h	14 ms	14 ms	⚪ +0%	8 ms	1.75×	✅ same
⚪	U7 missing label, ungrouped sum	1h	13 ms	13 ms	⚪ +0%	8 ms	1.62×	✅ same
⚪	U8 rate of an unwrapped label	1h	64 ms	61 ms	⚪ -5%	13 ms	4.69×	✅ same
⚪	U9 bytes() conversion	1h	36 ms	37 ms	⚪ +3%	10 ms	3.70×	➖ differs (as base)

Cold (first-run) timings — what a user opening a panel sees

shape	range	base cold	PR cold	Loki cold	PR ÷ Loki cold
bytes_over_time by app	1h	51 ms	49 ms	519 ms	0.09×
drilldown service landing	1h	23 ms	25 ms	451 ms	0.06×
instant json filtered	instant	297 ms	291 ms	61 ms	4.77×
instant json volume	instant	348 ms	346 ms	99 ms	3.49×
instant ungrouped rate	instant	4 ms	6 ms	60 ms	0.10×
json rate by level (no filter, no drop)	1h	400	400	400	—
json volume no filters	1h	141 ms	140 ms	2 ms	70.00×
logfmt count by level, filter, drop	1h	332 ms	336 ms	811 ms	0.41×
logfmt volume	1h	173 ms	164 ms	885 ms	0.19×
logs plain	1h	73 ms	55 ms	31 ms	1.77×
logs | json	1h	85 ms	82 ms	50 ms	1.64×
logs | logfmt	1h	133 ms	146 ms	47 ms	3.11×
stream-label count by app	1h	17 ms	20 ms	466 ms	0.04×
topk rate by app	1h	31 ms	33 ms	536 ms	0.06×
ungrouped rate	1h	8 ms	8 ms	468 ms	0.02×
DB3 json field breakdown user_id	1h	49 ms	39 ms	483 ms	0.08×
DB4 json field breakdown batch_id	1h	67 ms	68 ms	480 ms	0.14×
X1 json filter level_extracted, by level	1h	2.62 s	2.57 s	586 ms	4.39×
X2 json group by level_extracted	1h	216 ms	219 ms	621 ms	0.35×
X3 json rate, level_extracted equality	1h	230 ms	254 ms	424 ms	0.60×
X4 json bytes, level_extracted regexp	1h	297 ms	296 ms	583 ms	0.51×
X5 logfmt filter level_extracted, by level	1h	501 ms	514 ms	693 ms	0.74×
E1 logs, json parser over stream labels	1h	106 ms	94 ms	73 ms	1.29×
E2 logs, logfmt parser over stream labels	1h	58 ms	53 ms	39 ms	1.36×
E3 logs, json, existence filter on level_extracted	1h	114 ms	107 ms	52 ms	2.06×
E4 logs, logfmt, existence filter on level_extracted	1h	64 ms	64 ms	42 ms	1.52×
E5 logs, json, drop level_extracted	1h	86 ms	84 ms	48 ms	1.75×
E6 logfmt count by level_extracted	1h	32 ms	33 ms	64 ms	0.52×
E7 json count by level_extracted	1h	1.04 s	1.02 s	140 ms	7.26×
E8 instant logfmt by level_extracted	instant	8 ms	5 ms	8 ms	0.62×
A explore volume, 2 filters, drop	1h	1.68 s	1.84 s	619 ms	2.97×
B grouped sum, filter, no drop	1h	88 ms	86 ms	793 ms	0.11×
C ungrouped sum, filter, no drop	1h	85 ms	84 ms	834 ms	0.10×
D volume, no filters	1h	147 ms	147 ms	1.31 s	0.11×
E field breakdown pipeline (explore)	1h	77 ms	81 ms	919 ms	0.09×
F field breakdown service_version (explore)	1h	998 ms	992 ms	498 ms	1.99×
G field breakdown pipeline (drilldown)	1h	37 ms	32 ms	441 ms	0.07×
H field breakdown service_version (drilldown)	1h	95 ms	100 ms	552 ms	0.18×
I grafana volume of B	1h	82 ms	82 ms	421 ms	0.19×
O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	121 ms	109 ms	421 ms	0.26×
DL1 drilldown field breakdown json-expr, level filter	1h	367 ms	366 ms	410 ms	0.89×
DL2 drilldown field breakdown, level + stream-label filter before json	1h	62 ms	67 ms	20 ms	3.35×
DL3 drilldown logfmt field breakdown, level filter	1h	476 ms	457 ms	801 ms	0.57×
DL4 instant count, level filter, json+logfmt, 1h	instant	378 ms	384 ms	967 ms	0.40×
DL6 explore volume, level filter before json	1h	784 ms	781 ms	563 ms	1.39×
P1 logs, json, level equals	1h	123 ms	86 ms	62 ms	1.39×
P2 logs, logfmt, level not equal	1h	21 ms	21 ms	24 ms	0.88×
P3 logs, json, level existence	1h	97 ms	98 ms	123 ms	0.80×
P4 logfmt count by level	1h	23 ms	23 ms	30 ms	0.77×
P5 json count by level	1h	66 ms	65 ms	140 ms	0.46×
P6 json count, level filter	1h	32 ms	34 ms	118 ms	0.29×
P7 json rate by level, level regexp	1h	32 ms	32 ms	138 ms	0.23×
P8 json count by a name that is no stream label	1h	69 ms	72 ms	107 ms	0.67×
P9 instant logfmt by level	instant	4 ms	4 ms	7 ms	0.57×
E1 json duration_ms sum by service	1h	77 ms	60 ms	159 ms	0.38×
E2 json duration_ms p95 by service	1h	293 ms	295 ms	169 ms	1.75×
E3 logfmt latency_ms rate, production	1h	377 ms	380 ms	757 ms	0.50×
E4 logfmt size_bytes max by service	1h	33 ms	33 ms	153 ms	0.22×
E5 json duration_ms sum, instant	instant	28 ms	23 ms	22 ms	1.05×
E6 logfmt ttl unit string (Loki 400)	1h	400	400	400	—
E7 json hexadecimal trace_id (Loki 400)	1h	400	400	400	—
E8 json duration_ms sum by service, namespace prod	1h	137 ms	136 ms	465 ms	0.29×
E9 logfmt latency_ms sum by service, namespace prod	1h	229 ms	229 ms	587 ms	0.39×
U1 json duration_ms sum by service	1h	77 ms	66 ms	149 ms	0.44×
U10 instant sum_over_time unwrap	instant	27 ms	24 ms	22 ms	1.09×
U11 instant max_over_time missing label	instant	4 ms	3 ms	11 ms	0.27×
U12 json duration_ms over every service	1h	167 ms	171 ms	605 ms	0.28×
U13 logfmt latency_ms over every service	1h	373 ms	375 ms	834 ms	0.45×
U14 duration() of a unit string	1h	35 ms	59 ms	35 ms	1.69×
U2 json duration_ms max by service	1h	67 ms	68 ms	624 ms	0.11×
U3 json duration_ms sliding sum	1h	58 ms	61 ms	161 ms	0.38×
U4 logfmt size_bytes sum by service	1h	28 ms	28 ms	33 ms	0.85×
U5 logfmt ttl unit string (no sample)	1h	28 ms	28 ms	34 ms	0.82×
U6 missing label, no parser	1h	14 ms	14 ms	56 ms	0.25×
U7 missing label, ungrouped sum	1h	13 ms	13 ms	60 ms	0.22×
U8 rate of an unwrapped label	1h	63 ms	68 ms	154 ms	0.44×
U9 bytes() conversion	1h	37 ms	40 ms	44 ms	0.91×

Why these shapes

• internal/proxy/ordered_json_metric.go → 73: control/bytes_over_time by app, control/instant json filtered, control/instant json volume, control/json rate by level (no filter, no drop), control/json volume no filters, control/logfmt count by level, filter, drop, control/logfmt volume, control/logs | json …
• control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app

_{base d12a062c75 → PR f14cdae532 · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 7.7 min. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.}

[github-actions]

✅ Visual smoke: passed

Base d12a062c75 vs PR df7609b61a, Grafana through the proxy builds and Loki on one fresh stack; run.
Core set: 5 capture(s) at 1h. Detailed set: 23 entries (explore-metric-plain, explore-A, explore-D, explore-DLV5, explore-B, explore-C, explore-E, explore-F, ...). 29 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.
Trimmed to the 1h range for the capture budget: explore-A, explore-D, explore-DLV5, explore-B, explore-C, explore-E, explore-F, explore-logs-json, explore-logfmt-metric, explore-logs-extracted, explore-extracted-metric, explore-extracted-filter-metric, explore-extracted-group-metric, explore-logs-plain-level, explore-plain-level-metric, explore-unwrap-sum, explore-unwrap-unit-string, explore-unwrap-missing-label, explore-unwrap-conversion-error, dd-field-service_version, dd-field-pipeline.

page	range	set	base = PR (data)	vs Loki (PR)	pixel diff	result
dd-fields	1h	core	✅ identical	identical (4 explained, 1 history-dependent)	0.11%	✅
dd-landing	1h	core	✅ identical	identical (3 explained)	0.13%	✅
dd-service-logs	1h	core	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
explore-logs-plain	1h	core	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-metric-plain	1h	core	✅ identical	1 difference(s), 1 on base too	0.04%	✅
dd-field-pipeline	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-field-service_version	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
explore-A	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-B	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-C	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-D	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-DLV5	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-E	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-F	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-filter-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-group-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-logfmt-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-logs-extracted	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-logs-json	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-logs-plain-level	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
explore-metric-plain	15m	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-metric-plain	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-plain-level-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-conversion-error	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.04%	✅
explore-unwrap-missing-label	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-sum	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.04%	✅
explore-unwrap-unit-string	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
tail-json-filter	live	detailed	✅ identical	entries in the window: PR 1828, Loki 663, in both 663	6.38%	✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

Core set montages (5): base | PR | Loki

dd-fields-1h

dd-landing-1h

dd-service-logs-1h

explore-logs-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-764.

[github-actions]

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	10321	10738	417
Coverage	87.9%	88.0%	+0.1% (stable)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal	Base	PR	Delta
QueryRange cache-hit CPU cost	3382.0 ns/op	3428.0 ns/op	+1.4% (stable)
QueryRange cache-hit memory	1240.0 B/op	1240.0 B/op	0.0% (stable)
QueryRange cache-hit allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
QueryRange cache-bypass CPU cost	4017.0 ns/op	4070.0 ns/op	+1.3% (stable)
QueryRange cache-bypass memory	1302.0 B/op	1303.0 B/op	+0.1% (stable)
QueryRange cache-bypass allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
Labels cache-hit CPU cost	923.0 ns/op	913.1 ns/op	-1.1% (stable)
Labels cache-hit memory	48.0 B/op	48.0 B/op	0.0% (stable)
Labels cache-hit allocations	3.0 allocs/op	3.0 allocs/op	0.0% (stable)
Labels cache-bypass CPU cost	1525.0 ns/op	1500.0 ns/op	-1.6% (stable)
Labels cache-bypass memory	400.0 B/op	399.0 B/op	-0.2% (stable)
Labels cache-bypass allocations	14.0 allocs/op	14.0 allocs/op	0.0% (stable)

State

• Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
• This is a delta report, not a release gate by itself. Required checks still decide merge safety.
• Performance is a smoke comparison, not a full benchmark lab run.
• Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.