fix: sum by (detected_level) over a JSON pipeline groups the level the way Loki's ingest does by szibis · Pull Request #760 · ReliablyObserve/loki-vl-proxy

szibis · 2026-10-13T18:41:22Z

Summary

sum by (detected_level) (...) over a | json pipeline now follows the level
Loki's ingest gave each line on every route that answers the shape, and a
filter on detected_level compares that value. A few lines stay different and are
registered (see "Not done").

Cost: a window whose rows all carry a stored level (a level stream label or
structured metadata, as Loki pipelines send it) is answered from VictoriaLogs
stats grouped by the stored fields, which is main's query: no chain, no level
guard probe, no probe beyond the ones main ran. Explore's JSON volume and the volume behind a
detected_level filter, which the CI perf smoke flagged (+52% and +186% on the
previous revision), stay within the harness's noise of main (see the A/B). A
window with a row that has no stored level, or a stored detected_level, asks
again with the chain, built from the fields and keys the window holds. That
path is still slower than main on the shapes whose answer main gets wrong, by
less than before (Explore's JSON volume on rows with no stored level,
VictoriaLogs alone, 15 runs in each measurement order: 84 ms to 105 ms and
87 ms to 102 ms at 1h; 0.54 s to 0.69 s at 6h and 2.39 s to 2.75 s at 24h; the
previous revision took 38 ms to 282 ms, 0.55 s to 1.5 s and 2.3 s to 6.1 s):
the chain parses each line once, for the plan's labels and for the level, and
scans a line for a keyword only when it holds one. The first view of a window
(nothing remembered) still runs the stored-level query, the level probes and
the presence probe before the chain: about 0.5 s at 1h where main takes 0.08 s.
The numbers and both measurement orders are in the A/B and "Not done".

Loki sets detected_level once per line (pkg/distributor/field_detection.go):
a level stream label or structured metadata first; otherwise, for a line that
starts with { and ends with }, the first string value of an allowed level
key (level, LEVEL, Level, log.level, severity, lvl, ...) at depth 0
or 1 in document order, counted only when it is a known spelling; anything
else (a number, notice, no key, a line that is not JSON) goes to a keyword
scan of the whole line, and only a scan that finds nothing is unknown. The
value is structured metadata present on every line, so a body detected_level
key is detected_level_extracted, and a detected_level stream label keeps its
stored spelling.

The row evaluator (plain | json, __error__!="", every pushdown fallback)
and the stats pushdown (| json | drop __error__, label filters) read only a
level stream label or body key and normalized it, so on plain | json:
{"level":30,"msg":"hi"} was 30 (Loki: unknown), a pino error line
({"level":50,...,"err":{...}}) was 50 (Loki: error), {"level":true}
was true, {"level":"notice"} was notice, {"level":"notice","msg":"error connecting"} was notice (Loki: error) and {"level":" info"} was
info. severity, Level, LEVEL and nested log.level keys and a line
without a level key that mentions error were unknown. A detected_level
stream label was normalized, and a body detected_level key replaced the
derived value.

Which clients send these shapes: Explore's logs volume (sum by (level, detected_level) (... | json | drop __error__ [$__auto])) and user-written
metrics. Logs Drilldown's volume over | json is answered by the row evaluator;
its detected_level breakdown (sum by (detected_level) (... | json | drop __error__, __error_details__ | detected_level != "" [$__auto])) has the shape of
its native field-breakdown histogram, but the label translator maps
detected_level to level, so that route is never exact for it
(orderedJSONNativeGroupingIsExact) and it was answered by the row evaluator
too. It is now served by the guarded stats pushdown (see the A/B: 2.4 s and 502
at 6h/24h before, 0.02-0.30 s now). The plain-selector detected_level volume
that Drilldown sends is the native route, which this change does not touch (see
"Not done").

Loki source

pkg/distributor/field_detection.go (v3.7.7): extractLogLevelFromLogLine,
getLevelUsingJSONParser (depth < 2, document order, string values only),
detectLevelFromLogLine (keyword scan with word boundaries),
normalizeLogLevel; pkg/distributor/labels.go (a colliding structured
metadata detected_level becomes detected_level_extracted).

Code

Row evaluator (internal/proxy/ordered_json_metric.go): a plan that groups
by, keeps (without) or filters on detected_level (derivesLevel) gives
each row Loki's structured metadata before any stage runs
(withDerivedLevel), using the derivation the log lines use
(deriveRowLevel, through logRowLevels.stringRow). Body detected_level
keys are renamed detected_level_extracted by the existing collision rule; a
stream label detected_level is kept as stored and the derived level is put
under detected_level_extracted (detectedLevelName). The level parser
hint and the normalization of the previous revision of this PR are gone
(lokiNormalizedLevel deleted): one derivation for log lines, metric rows
and the pushdown fallback.
Stored-level plan (same file, orderedJSONStatsBuckets with
storedLevel): a plan reading detected_level first copies the stored
detected_level and level fields before any unpack (| copy detected_level as __dl_d, level as __dl_l) and groups the buckets by the copies. Each
bucket's level is translator.NormalizeStoredLevel of the first non-empty of
the two (a known spelling in any case becomes its level, any other word
stays), which is the chain's spelling pipes in Go, tested against their
regexps and against a table of Loki v3.7.7's normalizeLogLevel answers
written out from its source. The proxy reads a stored detected_level, then
level, and a bucket with a level needs no other field of its list. Loki
takes any allowed stream label before any structured metadata, so a stream
severity with metadata level is error in Loki and info here: a
recorded difference (semantics/detected-level-pipes-known-differences), not
a property of this check. A bucket without a stored level, or with a stored
detected_level (a stream label keeps its stored spelling, metadata is
normalised, and the stats cannot tell them apart), returns
errLevelNotStored and the query is asked again with the chain; the verdict
is remembered for the base for five minutes (levelNeedsChain) so the next
ask goes there at once. The attempt runs only the label probes main ran, none
of the level guard probes (they exist for lines only the chain reads).
A filter on detected_level runs in VictoriaLogs on the stored spellings
(translator.BuildStoredLevelCopyFilter: the spellings of every level the
filter accepts, a word Loki does not know compared as stored, and every row
the stored fields do not decide kept, so that it reaches the chain) and again
in Go on the normalised level (storedLevelSeries). The series limit
(-max-stats-query-series) applies to the answer's series: the spellings of
one level are VictoriaLogs groups but one series (INFO, info and
Information are one, as in Loki), so the groups are counted after the merge
(mergeGroups). The read of the groups is bounded only as a memory guard,
twice the limit (splitGroupSlack): past it the attempt returns
errStatsGroupGuard, the verdict for that query is remembered, and the chain
route, whose groups are its series, counts them and answers Loki's 400 where
it is due. Before this, 300 pods with three levels each under a limit of 500
answered 400 for sum by (pod) (... | json | detected_level="error"), where
Loki answers 300 series.
Stats pushdown (same file), when the stored fields do not decide: the level is computed in VictoriaLogs with
translator.BuildDetectedLevelChain(..., DetectedLevelChainGrouping), the
pipes the native metric route uses, placed after the prefilters and before
the unpack of any other label (orderedJSONUnpackPipesAfter), so a body
level never looks stored. A filter on detected_level is applied after
them (!= "" always holds and is dropped). Three guard probes join the
pushdown's own and share their window cache: detectedLevelStreamRisk (a
detected_level stream label), detectedLevelKeysRisk (two level keys in a
line, which the pipes read in a fixed order; a level value with an escape,
which Loki reads as raw bytes; a line that starts with a brace but does not
end with one, which Loki reads as logfmt and the pipes do not unpack) and
detectedLevelUnpackRisk (a level key unpack_json yields no value for:
nested other than log.level, leading whitespace, text after the closing
brace, a syntax error). One such line in the window sends the query to the
raw-row evaluator. With -detected-level-body-scan=false the raw-row
evaluator answers.
The chain follows Loki on two storage shapes the first revision got wrong:
a line that is JSON for Loki (a brace first and last) is never unpacked as
logfmt ({"msg":"error: cannot set level=debug now"} is error, not
debug); the severity VictoriaLogs derives for an OTLP record without one
(severity_text Unspecified with severity_number 0, or Info2 with 10,
from v1.50; one severity field with no number before) is no level field, so
the number decides and a record without a severity falls to the line, as in
Loki. A scratch field __dl_h marks the rows a stored level decides once, so
the unpack pipes test it instead of repeating the stored-field filter.
Why not "always the raw-row evaluator" for detected_level groupings: the
pushdown serves Explore's logs volume at 24h in 0.28 s (DLV4) where the
raw-row evaluator answers 502 (row limit) at 6h and 24h on the same data
(DLV6 below). The pushdown gives Loki's value for every probe line the guard
clears; the lines it cannot see are registered in
semantics/detected-level-pipes-known-differences (see "Not done").
Pruned chain (translator.BuildDetectedLevelChain with a
DetectedLevelPresence, internal/proxy/detected_level_presence.go): each
pipe passes every block, so the chain is built from what the window holds. One
stats query (levelPresenceQuery) over the stats query's own base and window
counts every stored level field and the words of every body key, and the
chain keeps those. It may only over-report: a body key counts as present
when its words occur in the line of a row no definite stored level field
decides (the chain unpacks no other), an escape sequence (\u) in such a
line makes every key present (unpack_json decodes \u006cevel, which the word
test cannot see; Loki decodes it too), severity_number and severity keep
the OTLP pipes (v1.50+ and before), a probe error or an unexpected row keeps
the full chain. The rewrite happens in vlPostHTTP (pruneLevelChain) for
every stats_query_range request carrying the chain, so the native metric
route (the translator and its query cache are unchanged) and the pushdown
share it; a request holding the chain twice, and an instant query, keep the
full chain. A window with one stored level field and no body key sends a
chain of 1.5 KB where the full one is 7.1 KB. The probe runs before the
request takes its stats_query_range slot (pruneBeforeSlot), not while
holding one. The mask can be older than the data (a line that arrives late
for a timestamp inside the remembered part), so the narrowed query is also a
witness: pipes before the chain flag every row that holds a stored field or
body key the mask left out, with the probe's own predicates
(levelPresencePredicate), as a group-by field (__lw) that
levelWitnessResponse reads back and strips; a flagged row, or an answer too
large to check, drops the mask and sends the request again with the full
chain, which is exact. A filter on detected_level after the chain passes
flagged rows, so the flag reaches the stats.
Cheaper chain, three cuts, each exact (the cost is measured below; the
claim is that the answer of every row is the answer of the pipes before):
- Keyword gate (translator.LevelKeywordGate). The keyword regexp, which is
  Loki's scan (field_detection.go detectLevelFromLogLine: the first of
  trace, debug, fatal, critical, error, err, warning, warn, info by position,
  between a left boundary of space, tab, newline, [, (, {, ", = and a
  right boundary of those and ], ), }, :, ,, !, case folded), ran over
  every row left without a level after the key reads. It now runs behind
  _msg:(i(trace) OR ... OR i(info)), VictoriaLogs' case-insensitive word filter.
  The gate is a superset of the regexp, never a second decision: every byte
  Loki accepts around a keyword is a token separator of VictoriaLogs'
  tokenizer, and the folding of (?i) maps no non-ASCII rune onto the
  letters of the nine words, so a matched text is an ASCII word between
  separators, which i() finds. A line the gate lets through and Loki's
  boundaries reject (foo-error, error.) is rejected by the regexp as
  before. The ASCII-only fold of the regexp, and so İNFO, stay what they
  were (registered).
- Shared unpack (translator.BuildDetectedLevelChainShared,
  sharedLevelUnpack). A JSON plan that groups by labels (level for
  Explore's volume) unpacked them with its own unpack_json fields (...) keep_original_fields, and the chain unpacked the level keys again with
  another. The chain's unpack now reads the level keys and the plan's labels
  together (result_prefix "__j_"), and before the closing delete each
  label is copied from its prefixed field where the field is not stored,
  which is what the plan's unpack wrote. The level keys are read only for
  rows no stored level decides (a stored severity, lvl or
  severity_number decides a row but is no level, so the row's level
  label still comes from the body), the presence mask never drops a label of
  the plan, and the unpack parses a row only where a label is missing as a
  stored field or the chain reads the line (the plan's parsed every row).
  Only a JSON plan whose labels are plain names, with no name_extracted
  label, shares; pruneLevelChain finds the chain that holds the labels
  through the request's context and narrows and witnesses it as any other.
- Escape check. The presence probe and the witness look for a backslash
  followed by u in the lines no stored level decides (an escaped key
  unpack_json decodes and the word test cannot see) with a substring
  filter, _msg:*"\\u"*, where they ran a regexp over every such row. It
  selects the same lines.
  Not done, because the measurement made them moot on this data: a single
  format if cascade for the keyword scan (the gated regexp costs 48 CPU-ms
  of the chain's 269) and a one-key shortcut when the probe shows one body key
  (the presence-pruned chain of this window already unpacks only level, so there
  is nothing for it to cut).
Remembered windows (coveredWindowProbe, shared with the guard probes'
verdicts, whose logic moved there): a verdict covers only the part of a
window older than five minutes (statsPushdownRiskSettle), the younger part is
probed on every request, so a key that first appears in recent lines is seen
at once; the key now carries the request scope fingerprint (tenant, routing
and forwarded identity), which the guards' key did not; and extending an
entry no longer renews the expiry of the part checked earlier (it did, so a
dashboard that refreshed kept a verdict for ever and a line arriving late
inside it was never seen). Behaviour change for the guards too: a refreshing
dashboard checks the last five minutes on every refresh and the whole window
again every five minutes.
Cost, disclosed (the chain path; the stored-level plan has none of it): the three guard probes and the presence probe are
VictoriaLogs queries over the whole window when it is not remembered, and on
every request over the part of the window younger than five minutes. Each stops at its first matching line, which bounds its output, not
its work: on a clean window each scans the lines that hold a level-key word
and the rows without a stored level. The chain adds work on every row of the
window (about 70 pipes unpruned, 18 for a window with one stored level field; most
are skipped for a row with a stored level, but each pipe still passes every
block; the witness is a third of what is left, see the A/B). The guard is a superset: one line it
cannot clear sends the whole query to the raw-row evaluator, exact but
bounded by the row limit, so on a long range it answers 502 where the
pushdown answered 200. Ordinary data can trip it: a body with both level
and severity (audit logs), or a stream label detected_level. A message
that merely mentions level=debug or level: high does not.
The | logfmt and plain-selector logs volume plans (sum by (level, detected_level)) take the same pipes. That changes their answer: a stream
without a level label now gets the level of its body (a logfmt level=warn
key, a JSON key or a keyword), as Loki's ingest reads it, where main said
unknown. The logfmt plan has the guard; the plain-selector plan has no
raw-row fallback and runs without it (see "Not done").

Tests

Unit TestOrderedJSONDetectedLevelLikeLoki: 55 probe lines (pino numeric
levels, notice, a padded level, severity, Level, LEVEL and nested
log.level keys, two level keys in both orders, a level key mentioned inside
a string, stream level/severity/detected_level, stored level,
severity and detected_level fields, malformed lines with a body
detected_level key, a JSON message holding logfmt text, a line that only
starts with a brace, two logfmt spellings of level, escaped level values
and keys, and twelve OTLP records of both VictoriaLogs storage shapes:
severity_text with severity_number from v1.50, one severity field
before), each through the row evaluator (| json, __error__="") and the
pushdown (| json | drop __error__), whose fake evaluates the chain with a Go
emulation of the pipes (emulateDetectedLevelChain), so the test shows where
the chain would differ from Loki and asserts the guard hands exactly those
lines to the raw evaluator (rawRows). The emulation follows the chain's
text, which TestBuildDetectedLevelChain pins; the pipes themselves run only
in the e2e tests. TestOrderedJSONDetectedLevelKnownDifferences asserts the
raw-row evaluator reads two logfmt lines like Loki (LvL=warn, a syntax error
before the level key) and the pushdown keeps the pipes' value.
TestOrderedJSONDetectedLevelMixedRows mixes the lines in one window for
count and bytes. TestOrderedJSONDetectedLevelFilters runs filters on
detected_level (!= "", =, =~, !=, without the error drop, with
__error__=""), also as a Logs Drilldown request, over all probes and over the
lines the pipes read like Loki, asserting the guarded pushdown answers the
latter with the filter applied after the pipes.
Presence and stored-level tests (detected_level_presence_test.go, and the
probe table run twice, with the full and with the pruned chain, every probe
line asked of a fake that answers the presence probe from its rows):
TestDetectedLevelPresenceKeepsAKeyOfOneLateLine (a key on one late line of
the window is read), TestDetectedLevelPresenceProbeFailureKeepsFullChain
(backend error, no counts, a short row), TestDetectedLevelPresenceNeverTrustsRecentAbsence
(a key first seen in recent lines is read on the next request),
TestCoveredWindowProbeKeepsTheAgeOfTheCheck, ...RefreshChecksOnlyTheRecentTail,
...KeepsScopesApart, TestPruneLevelChainLeavesTwoChainsAlone,
TestLevelPresenceQuery/FromMask, TestDetectedLevelPresenceNativeRoute,
TestStoredLevelDecidesDetectedLevel (nine spellings through the json,
logfmt and plain plans and two filters on detected_level, one stats query,
no chain, no level guard, no presence probe),
TestStoredLevelAttemptFallsBackToChain (one row without a level; the
second ask goes to the chain at once), TestNormalizeStoredLevelMatchesTheChain,
TestBuildDetectedLevelChainPruned (the OTLP shapes included). The probe table
gained escaped level key, long spelling ({"\u006cevel":"Information"}:
info in Loki v3.7.7, which decodes keys; the keyword scan cannot find it, so
only the decoded key gives the answer) and stream detected_level wins over a stream level, in the unit table and the e2e table.
Mutations (each makes the named tests fail; tests / subtests that failed):
the presence probe reporting every field absent 6 / 19 (the probe table in
the pruned arm, the late-line, native-route and filter tests); body-key
flags ignored 4 / 1; a probe error answered with an empty presence 4 / 22;
the settle clamp removed 1 (a recent absence is trusted); the expiry renewed on
extension 1; the escape flag ignored 2 / 1; the number pipes pruned away when
only severity is present (VictoriaLogs before v1.50) 1 in the translator
test and 1 / 2 in the OTLP lines of the probe table; the scope dropped from
the key 1; two chains pruned with the first base 1; the stored-level
normalisation made case-sensitive 3 / 7; a bucket without a level accepted 7 /
104; a stored detected_level accepted 1 / 2; the verdict not remembered 1; the
filter on detected_level ignored 2 / 3; the stored-level attempt disabled 5 / 13.
Translator unit tests pin the chain (TestBuildDetectedLevelChain: the
synthesised-severity skip, the __dl_h marker, the logfmt condition),
BuildHasLevelFilter for both OTLP shapes and the stored-level filter for
the derived severity names.
Fail without the fix: with the guard's Level=info LEVEL=error clause removed
the probe logfmt level keys of two spellings fails; with the escaped-value
clause removed escaped level value fails; with the brace-without-closing
clause removed logfmt after a brace fails. In the e2e test, with the pipes
reading a derived OTLP severity as a level, six subtests fail on each
VictoriaLogs version (OTLP without severity, OTLP without severity, error word, OTLP fine severity number and the three OTLP filter groups). The
earlier revision's own mutations still hold (against 10b0941f the first 28
probes failed on 55 of 84 subtests; chain not inserted - 29; guard probes
off - 17).
The existing pushdown tests changed only in their expectations of the query
(the chain, three more probes), of the unpack fields (by (detected_level)
no longer unpacks level) and of a filter on detected_level (now pushed).
Their Loki-reference harness (lokiLevelVolumeReference) is a hand-built
model of Loki's answer for simple fixtures, not Loki's: it normalizes the
spelling and falls to the keyword scan as Loki's ingest does; the comparison
with real Loki is the e2e tests. The plain logs-volume test asserts the level
of a logfmt body.
e2e TestRangeMetricCompatibilityJSONDetectedLevelProbes against real Loki:
50 lines (the malformed ones valid only under drop __error__; seven are
OTLP records pushed to Loki and to VictoriaLogs' OTLP endpoint), one stream
each, | json and | json | drop __error__, the container proxy and an
in-process proxy; it asserts Loki's own answer equals the table and that the
pushdown serves the lines the pipes read like Loki and hands the others to the
raw-row evaluator. Ten more subtests run filters on detected_level and Logs
Drilldown's detected_level breakdown (with its X-Query-Tags header) against
Loki, three of them over the OTLP records, and assert the breakdown is one
guarded stats request without raw rows on clean lines. Loki v3.7.7 agreed
with the table on every line, including the escaped value (unknown), the
escaped key (warn), LvL=warn (warn) and the unterminated logfmt line
(error), measured separately.
TestRangeMetricCompatibilityJSONLogsVolume gained a plain stream line whose
body holds level=warn. ingestJSONVolumeFixture takes extra stream labels
(labels) and names the VictoriaLogs stream fields per app from the labels
its lines carry, so structured metadata (a stored level) is not made a
stream field.
Unit TestStoredLevelPlanCountsTheSeriesOfTheAnswer (the nine-spelling
fixture of TestStoredLevelDecidesDetectedLevel under -max-stats-query-series
3 to 6): a filter on one level keeps its spellings only (three series at a
limit of three, where the plan answered 400 for the nine groups), a regexp, a
negated and an unknown-word filter, the merged answer exactly at the limit
(200), one over it (400 with Loki's message), and a read past the group guard
that the chain answers. TestStoredLevelPlanManyPodsUnderTheSeriesLimit (30
pods of three levels under a limit of 50 and of 40: 200 with 30 series, and
the verdict of the guard remembered), TestStoredLevelCopyFilterIsExact (the
filter pipes against NormalizeStoredLevel for 20 filters and 25 stored
words, and rows the stored fields do not decide are kept).
TestNormalizeStoredLevelIsLokisNormalizeLogLevel is a table of Loki
v3.7.7's normalizeLogLevel answers written out from its source, and
TestStoredLevelDecidesDetectedLevel states its expectations literally
(they no longer come from NormalizeStoredLevel).
Unit TestDetectedLevelWitnessCatchesALateLine (a mask remembered for
level only, then a line {"msg":"error retry","severity":"warn"} stamped
inside the covered window: warn, Loki's, where the pruned chain answered
error; on the pushdown and the native route; the mask is probed again after
it), TestLevelWitnessResponse, TestPrunedChainWithWitnessFitsTheQueryLimit,
TestLevelPresenceProbeRunsBeforeTheStatsSlot (no slot is held while the
probe runs), TestCoveredWindowProbeKeepsTheAgeOfTheCheck (the cache clock,
no sleeps) and TestCache_SizeCountsTheKey.
Unit TestLevelPresenceSweepOnePositionAtATime: each of the 31 positions of
the presence mask (16 stored fields, 14 body keys, the escape flag) alone in
a window must give exactly its field or key in the pruned chain from one probe
and one chain query; forcing any one position to 0 in the probe fails its
subtest (31 of 31, where the earlier tests caught 10), because the witness
asks again with the full chain and the test counts the queries. The fake
VictoriaLogs counts with its own copies of the probe's tables.
FuzzStoredLevelCopyFilter, 30 s with FuzzTranslateLogQL and
FuzzParseDropConditions (no finding).
Cheaper chain (the three cuts above; a test per claim, each answer written out
from Loki v3.7.7's source or measured against it, not taken from the code):
TestKeywordScanLikeLoki (94 lines with Loki's literal answers from
detectLevelFromLogLine: case, errors/terror/warnings/information,
several keywords with the smallest position winning, warning: error,
critical error against error critical, every boundary byte on each
side including \r, a no-break space, é, ж, € and invalid UTF-8; the
chain's scan with the gate and the regexp alone both give them),
TestKeywordGateIsASupersetOfTheRegexp (400,000 generated lines: the regexp
matching implies the gate matching, the gate evaluated from the text
LevelKeywordGate() returns with VictoriaLogs' i() rules written out from
filter_any_case_phrase.go), FuzzKeywordGateIsASupersetOfTheRegexp (the
same, and the scan against a plain search for the first bounded keyword; 30 s,
no finding), TestLevelKeywordGateText, TestBuildDetectedLevelChainShared
(one unpack_json, the condition in each case, the level key reads gated by
-__dl_h:*, the copies before the delete, the mask never dropping a label),
TestSharedLevelUnpackEligibility, TestSharedUnpackKeepsThePlansLabelsLikeLoki
(ten lines, the unpruned and the pruned chain: a stored severity, lvl,
severity_number, Level or LEVEL with a body level, a stream level, a
stored level; the labels are Loki's, one unpack_json and the witness in
the request sent) and the 30 keyword lines in the probe tables, unit and e2e.
The unit fake of VictoriaLogs runs the unpack and the copies of a shared
chain as VictoriaLogs does and emulates the rest of the chain whole, with the
rows a stored level decides taken from the same stored-field list as the
chain's marker.
e2e (VictoriaLogs alone, TestRangeMetricCompatibilityDetectedLevelPipeVariants):
1,489 generated rows (89 bodies, 26 stored-field sets) per VictoriaLogs
version, the pipes as they were against the pipes as they are (gate only;
gate and shared unpack; full and pruned chain; the plan's unpack with and
without its condition), per row, identical on v1.47.0 and v1.52.0;
TestRangeMetricCompatibilityEscapeSubstringFilter (the regexp and the
substring filter select the same 7 of 14 lines);
TestRangeMetricCompatibilityJSONDetectedLevelSharedUnpack against Loki
(11 lines in windows that also hold a line no stored level decides, sum by (level, detected_level): the same two series as Loki, one chain request with
one unpack_json, no raw rows; OTLP severity numbers included).
Mutations (each makes the named tests fail; tests / subtests that failed):
the gate without err 5 tests (+ the e2e variants); no gate 2; the plan never
shares 5 / 27; the shared unpack parsing only the rows the chain reads 2 / 10
(+ 1); the shared unpack without the plan's labels 3 / 5; the labels not
copied 5 / 19; the escape check a regexp again 1; the pruner not knowing the
shared chain 1 / 10; the level key reads not gated by -__dl_h:* 1 (the
translator text test).

Registry

id: semantics/json-metric-detected-level-derivation
title: sum by (detected_level) over a JSON pipeline follows the level Loki's ingest gave each line, by level key in document order, by keyword scan, with a detected_level stream label kept as stored (fixed; a few lines stay different, see semantics/detected-level-pipes-known-differences)
proves:
  - loki_api_v1_query_range
consumer: api
why: |
  Loki sets detected_level once per line at ingest (pkg/distributor/
  field_detection.go extractLogLevelFromLogLine, discover_log_levels on). A
  level stream label or structured metadata decides first. Otherwise a line
  whose first non-space character is `{` and last is `}` is read as JSON: the
  first string value of an allowed key (level, LEVEL, Level, log.level,
  severity, SEVERITY, Severity, SeverityText, lvl, LVL, Lvl, severity_text,
  Severity_Text, SEVERITY_TEXT) at depth 0 or 1, in document order, counts only
  when it is a known spelling (info, INFO, inf and information are `info`).
  Anything else (a number, a boolean, `notice`, no such key, a line that is not
  JSON) falls to a keyword scan of the whole line, which finds the first
  bounded level word; `unknown` is the answer only when the scan finds none.
  The value is structured metadata present on every line, so a body key named
  detected_level is detected_level_extracted, and a detected_level stream label
  keeps its stored value (the derived one moves to detected_level_extracted).

  Measured against Loki v3.7.7 on 50 probe lines (the e2e test pins each):
  `{"level":30,"msg":"hi"}` is unknown; a pino error line
  (`{"level":50,...,"err":{"type":"Error"}}`) is error; `{"level":"notice"}` is
  unknown but `{"level":"notice","msg":"error connecting"}` is error;
  `{"level":" info"}` is info; `{"severity":"ERROR"}`, `{"Level":"Information"}`,
  `{"LEVEL":"warn"}` and `{"log":{"level":"warn"}}` are error, info, warn and
  warn; `{"msg":"x","severity":"error","level":"info"}` is error (document
  order) and `{"level":"info","severity":"error"}` is info; a line without a
  level key that says "error" is error. A JSON line is read for keys only: the
  logfmt-looking text in `{"msg":"error: cannot set level=debug now"}` is no
  level, so it is error. A line that starts with a brace but does not end with
  one is logfmt (`{ error level=info` is info). Loki compares a JSON value's raw
  bytes, so `{"level":"\u0069nfo"}` is unknown, and decodes keys, so
  `{"\u006cevel":"warn"}` is warn. For logfmt the first level key in document
  order wins within a spelling family (`Level=info LEVEL=error` is info). An
  OTLP record is its severity number's level, or its text when the text is its
  own: `Unspecified` or no number is unknown (or the keyword scan of the line),
  number 10 is info, `notice` is notice. The proxy took the stream's spelling as
  stored, read only a body `level` key and made every other value a series of
  its own (`30`, `notice`, ` info`, `true`) or `unknown`.

  VictoriaLogs stores an OTLP record's severity in two shapes: from v1.50
  severity_text (the record's text, or a name it derives from the number:
  `Unspecified` for 0, `Info2` for 10) with severity_number, and before one
  severity field with no number. The derived text is no level field of Loki's,
  so the pipes skip it and the number decides (a record without a severity
  falls to the line, as in Loki). The e2e probes run on v1.47.0 and v1.52.0 and
  cover both shapes; the pairing of a derived name with its number is checked
  by family (a record sending `Info2` with number 12 is read as derived).

  Both routes of `sum by (detected_level) (... | json ...)` now give Loki's
  value for every probe line above, bar the differences recorded in
  semantics/detected-level-pipes-known-differences. The row evaluator (plain `| json`, `__error__!=""`, every pushdown
  fallback) gives each row Loki's structured metadata before any stage runs,
  using the derivation the log lines use (deriveRowLevel), so a body
  detected_level key is renamed and a filter on detected_level
  (`| json | detected_level="info"`) compares the derived value. The stats
  pushdown first groups the buckets by copies of the stored detected_level and
  level fields, taken before any unpack, and gives each bucket the level Loki's
  ingest gives it: the first non-empty of the two, a known spelling in any case
  as its level and any other word as stored (translator.NormalizeStoredLevel, tested against
  the chain's own spelling regexps and against a table of Loki v3.7.7's
  normalizeLogLevel answers). That is main's query with
  no chain and no level guard probe, and it answers every window whose rows
  all carry a stored level (a stream label or structured metadata, as Loki
  pipelines send it); the proxy reads a stored detected_level, then level, and
  a bucket with a level needs no other field of the proxy's list. Loki takes any
  allowed stream label before any structured metadata, so a stream `severity`
  with metadata `level` is error in Loki and info here: a recorded known
  difference (semantics/detected-level-pipes-known-differences), not part of
  this check. A filter on detected_level runs in VictoriaLogs on the stored
  spellings (translator.BuildStoredLevelCopyFilter, which keeps every row the
  stored fields do not decide so that those reach the chain) and again in Go on
  the normalised level. Loki's series limit is applied to the answer's series:
  the spellings of one level (`INFO`, `info`, `Information`) are one series,
  and a read of more than twice the limit of VictoriaLogs groups
  (`errStatsGroupGuard`, a memory guard) is asked again with the chain, whose
  groups are its series. A
  bucket without a stored level, or with a stored detected_level (a stream
  label keeps its spelling, structured metadata is normalised, and the stats
  cannot tell them apart), sends the query again with the pipes of the native
  metric route (translator.BuildDetectedLevelChain), built from the level
  fields and body keys the window holds (a presence probe: one stats query of
  counts per stored field and per body key word, over the same base and
  window, run on every request over the part of the window younger than five
  minutes; an escape sequence in a body makes every key present, a failed probe
  keeps the full chain; the narrowed query flags each row holding a field or key
  the probe left out, a line that arrived late inside the remembered part, and
  is then asked again with the full chain), placed before the unpack of any other label, and a
  guard (detectedLevelStreamRisk,
  detectedLevelKeysRisk, detectedLevelUnpackRisk, cached with the other
  pushdown probes) hands the query to the raw-row evaluator when the window
  holds a line those pipes read differently from Loki: a detected_level stream
  label, two mentions of a level key (the pipes read keys in a fixed order),
  a level key whose JSON value has an escape (Loki reads raw bytes),
  logfmt keys in a line that starts with a brace but does not end with one
  (Loki reads that as logfmt, the pipes do not unpack it), or a level key
  unpack_json yields no value for (nested other than log.level, leading
  whitespace, text after the closing brace, a syntax error). Differences it
  does not see are recorded as open in the known-differences row: a stream
  level field outranked in the pipes' list order by a stored metadata field
  (stream `severity` with metadata `level`), a logfmt level key in another case
  (`LvL=warn`), a logfmt syntax error before the level key, and a row
  VictoriaLogs unpacked from a JSON body. A filter
  on detected_level is applied after the pipes (`!= ""` always holds), also for
  Logs Drilldown's detected_level breakdown, which is not the native histogram
  (the label translator maps detected_level to level, so
  orderedJSONNativeGroupingIsExact is false for it). With
  -detected-level-body-scan=false the raw-row evaluator answers, as only the
  stored fields decide.

  The plain and `| logfmt` logs volume shapes (`sum by (level,
  detected_level)`) take the same plan, and the same pipes when a row has no
  stored level: a stream without a level label gets
  the level of its body (a `level=warn` key or a keyword), as in Loki, where
  main answered `unknown`. The plain shape has no raw-row fallback, so it runs
  without the guard (see semantics/detected-level-pipes-known-differences).

  Cost: a window whose rows all carry a stored level costs what main's query
  costs (two more grouped columns and one copy pipe). Measured against main on
  one hour of the A/B harness's backfill (about 290,000 lines, a level stream
  label on every row) with VictoriaLogs capped at 2 CPUs as on a CI runner,
  warm p50 of 5 to 7 runs, the two builds measured in each order (the second
  target of the harness runs slower by up to 30% even for the same binary):
  Explore's JSON volume 79 ms to 98 ms (101 ms to 76 ms in the other order),
  the volume behind a `detected_level="info"` filter before the parser
  (VictoriaLogs first unpacks logfmt for every line) 181 ms to 279 ms (259 ms to
  245 ms); at 6h and 24h 574 ms to 608 ms and 2.37 s to 2.40 s (596 ms to
  589 ms and 2.25 s to 2.33 s), 1.55 s to 1.60 s and 6.38 s to 6.90 s (1.50 s to
  1.59 s and 5.98 s to 6.45 s).

  A row with no stored level (the level only in the line: the same backfill
  with the level left out of the stream labels, which the harness seeds with
  `--no-level-label`) is read by the chain, which reads every line; main
  answers 4 series for Explore's volume there where Loki answers 8. The chain
  parses each line once, for the plan's labels and for the level: the plan's
  own `unpack_json` and the level keys' are one (translator.SharedUnpack: the
  chain's unpack reads the level keys and the plan's labels with
  result_prefix "__j_", the level keys are read only for rows no stored level
  decides, and before the closing delete each label is copied from its
  prefixed field where the field is not stored, which is what the plan's
  unpack with keep_original_fields wrote; the unpack parses a row only where a
  label is missing as a stored field or the chain reads the line). The keyword
  regexp runs only on lines that hold one of its nine words as a whole word of
  VictoriaLogs' tokenizer (`_msg:(i(trace) OR ... OR i(info))`, a superset of
  the regexp's matches: every byte Loki accepts around a keyword is a token
  separator and the folding of (?i) maps no non-ASCII rune onto the letters of
  the words), and the escape check of the presence probe and its witness is a
  substring filter (`_msg:*"\\u"*`), not a regexp. Measured on one hour of
  that data, VictoriaLogs capped at 2 CPUs, warm p50, VictoriaLogs alone, 15
  runs in each measurement order at 1h: Explore's volume 84 ms to 105 ms (87 ms
  to 102 ms with the branch measured first), the detected_level breakdown with
  the error rows dropped 88 ms to 99 ms (85 ms to 101 ms); at 6h and 24h (3
  runs, both orders) Explore's volume 541 ms to 692 ms and 2.39 s to 2.75 s
  (570 ms to 702 ms and 2.30 s to 2.84 s), the
  breakdown 585 ms to 641 ms and 2.24 s to 2.74 s (589 ms to 698 ms and 2.25 s
  to 2.79 s). The previous revision took 38 ms to 282 ms, 0.55 s to 1.5 s and
  2.3 s to 6.1 s for Explore's volume and 56 ms to 99 ms and 2.2 s to 3.8 s for
  the breakdown. On VictoriaLogs alone, one hour, per query: main's query 102
  ms and 225 CPU-ms (cgroup CPU time; the wall time at 2 CPUs moves in steps of
  the 100 ms quota period), the stored-level plan 100 ms, the previous chain
  290 ms and 550 CPU-ms, this one 107 ms and 260 CPU-ms. Where the CPU of the
  chain went (one group of pipes left out at a time): the plan's own unpack 194
  CPU-ms, the keyword regexp 144, the witness 102 before; the witness 87, the
  keyword regexp 48 and the json unpack 20 now. The detected_level breakdown of
  Logs Drilldown (a filter, `| json | drop __error__`), which main answered
  from raw rows, goes on that data from 2.2 s to 0.05-0.11 s at 1h and from
  7.5 s to 0.67 s (6h) and 2.6 s (24h), and on the data with a level stream
  label from 2.3 s to 7 ms at 1h. The first view of a window (nothing
  remembered: the stored-level attempt, the level probes, the presence probe,
  then the chain) takes about 0.5 s at 1h for Explore's volume where main takes
  0.08 s (0.55-0.68 s in the previous revision, both builds restarted on one
  stack): the probes scan the window and share the 2 CPUs with the chain, the
  heaviest of them (detectedLevelKeysRisk, 260-300 ms: two level keys in a line,
  an escape in a level value, a brace without its closing brace) one this change
  adds. A window with a row that has no stored level asks
  twice, the stored-level attempt first (it costs what main's query costs; its
  verdict is remembered for the base for five minutes so the next ask goes to
  the chain at once), and pays the chain with the three guard probes and the
  presence probe. Each guard probe stops at its first matching line (`limit 1`),
  which bounds what it returns, not the work: on a clean window it scans the
  lines that hold a level-key word (VictoriaLogs' word index narrows them) and
  the rows without a stored level. The guard is a superset of the lines the
  pipes read differently: a body with both `level` and `severity` (audit logs)
  trips it although the pipes might agree. One such line sends the whole query
  to the raw-row evaluator, which is exact and bounded by the row limit, so a
  long range answers 502 where the pushdown answered 200. The chain is about
  7.1 KB of LogsQL unpruned and 1.5 KB for a window with one stored level field
  and no body key, plus the witness (up to 1.5 KB, none when the window holds
  every field and key): with a `service_name` selector Explore's volume sends
  4.2 KB (9.8 KB unpruned) and a regexp selector of seven long names about
  9.6 KB with the witness (14.4 KB unpruned; those names match no row, so the
  chain is at its smallest), against VictoriaLogs' 16 KiB -search.maxQueryLen (a
  longer query answers 502; a window whose presence cannot be told keeps the
  full chain and its size; a unit test pins the witnessed request under the
  limit). The remembered verdicts (guard probes and presence) are kept per
  request scope, cover only the part of a window older than five minutes, and
  expire five minutes after the check of their oldest part, so a refreshing
  dashboard checks the last five minutes on every refresh and the whole window
  again every five minutes.
request:
  method: GET
  path: /loki/api/v1/query_range
  params:
    query: 'sum by (detected_level) (count_over_time({app="json-level-probe"} | json | drop __error__ [1m]))'
    start: now-10m
    end: now
    step: "60"
  headers:
    X-Scope-OrgID: "0"
  precondition: |
    One stream per probe line: pino numeric levels, notice, a padded level, the
    severity, Level, LEVEL and nested log.level keys, two level keys in both
    orders, a line with the word error and no level key, a detected_level
    stream label, a body detected_level key, structured metadata level and
    detected_level.
compare:
  mode: exact
  fields: [status, series_labels, values]
  expect: |
    The series Loki answers for each probe, on `| json` (raw-row evaluator) and
    `| json | drop __error__` (pushdown or its guarded fallback).
gap:
  status: fixed
  fixed_by: "#760"
  impact: explore-visible
  visual: none
  visual_reason: 'the generator data carries one lowercase spelling, the same in the stream label and the body, so main and the branch render the same panel; the e2e probes against Loki are the proof (v1.47 and v1.52)'
  effort: M
  area: 'internal/proxy ordered_json_metric.go (withDerivedLevel, setStatsPushdown, detectedLevel*Risk), stream_processing.go stringRow'
  loki: 'pkg/distributor/field_detection.go extractLogLevelFromLogLine, detectLevelFromLogLine, normalizeLogLevel; pkg/distributor/labels.go (detected_level_extracted)'
  proxy: 'the row evaluator and the stats pushdown read the stream level or a body level key and normalized it'
  planned_test: 'TestOrderedJSONDetectedLevelLikeLoki, TestOrderedJSONDetectedLevelMixedRows, TestOrderedJSONDetectedLevelKnownDifferences (proxy); TestRangeMetricCompatibilityJSONDetectedLevelProbes, TestRangeMetricCompatibilityJSONDetectedLevelSharedUnpack, TestRangeMetricCompatibilityDetectedLevelPipeVariants, TestRangeMetricCompatibilityEscapeSubstringFilter and TestRangeMetricCompatibilityJSONDetectedLevel (test/e2e-compat, group semantics); TestKeywordScanLikeLoki, TestKeywordGateIsASupersetOfTheRegexp, TestBuildDetectedLevelChainShared (translator); TestSharedUnpackKeepsThePlansLabelsLikeLoki, TestSharedLevelUnpackEligibility (proxy)'
  note: |
    Measured on VictoriaLogs v1.52.0 and v1.47.0 against Loki v3.7.7.
  clusters: []

id: semantics/detected-level-pipes-known-differences
title: The detected_level pipes read level keys in a fixed order, do not decode logfmt like Loki and overwrite a detected_level stream label (open)
proves:
  - loki_api_v1_query_range
consumer: explore
why: |
  The native metric route (`sum by (detected_level) (count_over_time({...}
  [1m]))` over a plain selector, which Logs Drilldown sends, and Explore's
  plain-selector volume shape) computes detected_level in VictoriaLogs with
  translator.BuildDetectedLevelChain. Those pipes read the level keys of a body
  in a fixed order, do not search nested objects other than `log.level`, stop
  at nothing in a line VictoriaLogs cannot parse as a whole, fold only ASCII
  in the keyword scan, and write the derived level over a stored
  `detected_level`, where Loki takes the first level key in document order,
  searches objects at depth 1, keeps the keys read before a syntax error, folds
  Unicode and keeps a detected_level stream label as stored.

  The pipes also read the stored level fields in one list order without
  telling a stream label from structured metadata, where Loki and the raw-row
  evaluator read the stream's before the metadata: a stream `severity=error`
  with metadata `level=info` is error in Loki and info on the pipes. The JSON
  pipeline guard does not see it (an exact probe is an OR over pairs of names,
  and a cheap one would send every stream labelled severity to the raw-row
  evaluator), so it is recorded here too; the e2e and unit probe
  `stream severity before stored level metadata` assert the raw-row
  evaluator's answer only.

  Three more lines the pipes read differently, which no guard probe sees
  (an exact one would need a logfmt decoder per line):
  - A logfmt level key in another case, `msg="error x" LvL=warn`: Loki's
    decoder compares keys without case and reads `warn`, `unpack_logfmt` is
    case-sensitive and the keyword scan says `error`. Loki answers `warn`
    (measured on v3.7.7); the pushdown and this route answer `error`.
  - A logfmt syntax error before the level key,
    `msg="unterminated error level=info`: Loki's decoder stops at the error
    and the line falls to the keyword scan, `error` (measured); the pipes skip
    the error and read `level=info`.
  - A row VictoriaLogs unpacked from a JSON body (stored without a message,
    `{"level":"notice","msg":"error later"}` pushed with message parsing on),
    whose stored `level=notice` the pipes keep as the word `notice`, where Loki
    keeps the line and gives `error` (measured; the same on main).
  The raw-row evaluator reads the first two like Loki
  (TestOrderedJSONDetectedLevelKnownDifferences); the pushdown and the native
  route keep the pipes' value.

  Measured against Loki v3.7.7 on VictoriaLogs v1.52.0 with the e2e probe
  fixtures: `{"msg":"x","severity":"error","level":"info"}` is error in Loki and
  info in the proxy; a stream with the label `detected_level="INFO"` is INFO in
  Loki and info in the proxy; a stream `severity=error` with metadata
  `level=info` is error in Loki and under `| json`, and info under
  `| json | drop __error__`. Logs Drilldown's detected_level breakdown
  (`| json | drop __error__, __error_details__ | detected_level != ""`) is not
  on this route: the label translator maps detected_level to level, so the
  native histogram is never exact for it and the JSON pipeline route answers.
  The JSON pipeline route (`| json`, #760) probes
  for these lines and answers them with the raw-row evaluator, which is exact;
  the native route has no raw-row fallback for them.
request:
  method: GET
  path: /loki/api/v1/query_range
  params:
    query: 'sum by (detected_level) (count_over_time({app="json-level-probe"}[1m]))'
    start: now-10m
    end: now
    step: "60"
  headers:
    X-Scope-OrgID: "0"
  precondition: |
    A line holding `severity` before `level` with different values, a stream
    with a detected_level label, a stream `severity` with metadata `level`, a
    logfmt line with `LvL=warn` or a syntax error before its level key, or a
    row VictoriaLogs unpacked from a JSON body with a stored `level`.
compare:
  mode: exact
  fields: [series_labels, values]
  expect: |
    Loki: error for the line with severity first; INFO for the labelled stream;
    warn for `LvL=warn`; error for the unterminated line and for the unpacked
    row. Proxy: info for the first two, error for `LvL=warn`, info for the
    unterminated line, notice for the unpacked row.
gap:
  status: open
  impact: edge
  effort: M
  area: 'internal/translator detected_level_logsql.go BuildDetectedLevelChain; internal/proxy native metric route'
  loki: 'pkg/distributor/field_detection.go getLevelUsingJSONParser (document order, depth < 2), labels.go (detected_level_extracted)'
  proxy: 'BuildDetectedLevelChain reads keys in its list order and overwrites detected_level; the JSON pipeline route guards these lines instead'
  planned_test: 'an e2e on plain-selector probe lines once the route probes and falls back to a raw-row or keyed evaluation'
  clusters: []

Visual proof

visual: none: the generator keeps the stream label level equal to the body
level and uses one lowercase spelling, so main and the branch render the same
Explore and Drilldown panel (1h A/B: same series and sums as Loki on every
shape). The e2e probes against Loki are the proof.

e2e against real Loki v3.7.7

Isolated compose project dl5e (a fresh stack per VictoriaLogs version,
Grafana started), image built from this branch, /ready and /health checked
before any test, VictoriaLogs restart count 0 before and after every run. The
route checks count the stored-level attempt (isStoredLevelAttempt,
levelStatsOutcome) and the presence probe (isLevelPresenceCall).

VictoriaLogs	result
v1.52.0	all pass (29 tests, 337 passing subtests), including `TestRangeMetricCompatibilityJSONDetectedLevelSeriesLimit`, `TestRangeMetricCompatibilityJSONDetectedLevelSharedUnpack`, `TestRangeMetricCompatibilityDetectedLevelPipeVariants` and `TestRangeMetricCompatibilityEscapeSubstringFilter` (new), and `TestRangeMetricCompatibilityJSONDetectedLevelProbes` (131 subtests over 80 probe lines, through the stored-level plan or the chain), `...JSONDetectedLevel`, `...JSONFilterPushdown`, `...JSONLogsVolume`, `TestDrilldown_LongRangeVolume_` (Grafana), `TestCompat_DetectedLevel` (corpus: 552 lines, 552 agree with Loki)
v1.47.0	the same set, all pass (29 tests, 337 subtests)

TestRangeMetricCompatibilityJSONDetectedLevelSeriesLimit ingests 12 pods with
three spellings of error and one of info each into Loki and VictoriaLogs and
runs sum by (pod, detected_level) (24 series) and the same filtered to
detected_level="error" (12 series) through an in-process proxy whose
-max-stats-query-series is the answer's series count, one more and ten times
that, against the stack's Loki (limit 1000000): the same series, not a 400; and
one below the count: Loki's message. Loki's own boundary was checked on a throwaway Loki v3.7.7 with max_query_series 24 and 23 and the same 12-pod fixture (outside the suite, whose Loki has a limit of 1000000): 24 series under a limit of 24 is a 200, under 23 a 400, and the filtered query (12 series) a 200 under both; Loki's check is len(series) > maxSeries (pkg/logql/engine.go, pkg/querier/queryrange/limits.go), so the series count at the limit passes, as it does here. On the previous revision the limits equal to
the series count and one above it fail (a 400 where Loki answers 200). Each version ran on a fresh stack
(a second run on one stack re-ingests the fixtures and doubles VictoriaLogs'
counts), VictoriaLogs restart count 0 before and after.

A/B against origin/main (VictoriaLogs v1.52.0, capped at 2 CPUs)

Base 1dcb79e0 vs branch (the code of this revision), warm p50 after a cold run
(5 runs at 1h, 3 at 6h and 24h; a shape the first pass moved is measured again
over 7 runs at 1h or 3 at 6h and 24h on windows it did not use), targets
interleaved per request, same step-aligned windows, one fresh stack per run
(about 290,000 lines per hour of the harness's backfill); VictoriaLogs runs under
docker update --cpus 2 like the CI runner (the harness has no flag for it:
bench/ab/pr_smoke.py was wrapped to apply it right after stack.up), restarts
0 -> 0 on every run, noise settings are CI's (30% and 100 ms). Every run was
made twice, with the base build measured first and with the branch build
measured first: the position of a target in the pair moves a cheap shape by 20%
to 70% in either direction (Explore's volume on the data without a stored level:
base first 118 ms and branch second 97 ms, branch first 169 ms and base second 95
ms), and the two columns show it. Loki parity is checked at 1h (6h and 24h
seed VictoriaLogs only). Saved under bench/ab/results/
(2026-10-14-detlevel-*.json, replacing the previous revision's) and, for the
second order and the data without a stored level, bench/ab/results/oneoff/.
Runs: python3 bench/ab/pr_smoke.py --base 1dcb79e0 --ranges 1h --runs 5 --confirm-runs 7 --selection <sel.json> --out <dir> --loki-mem 3g --vl-mem 4g [--no-level-label]. All 16 runs: 0 new differences from Loki, restarts 0 -> 0.

Data with a level stream label on every row

A window whose rows all carry a stored level is answered by the stored-level
plan, which none of the cuts touches; the numbers are those of the previous
revision's, measured again. Explore's JSON volume (D) and the volume behind a
level filter before the parser (DL6) stay within the harness's noise of main:

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	79 ms	98 ms	+24%	101 ms	76 ms	-25%	same
DL6 explore volume, level filter before json	1h	181 ms	279 ms	+54%	259 ms	245 ms	-5%	same series, sum ±0.27%

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)
D volume, no filters	6h	574 ms	608 ms	+6%	596 ms	589 ms	-1%
D volume, no filters	24h	2.37 s	2.40 s	+1%	2.25 s	2.33 s	+4%
DL6 explore volume, level filter before json	6h	1.55 s	1.60 s	+3%	1.50 s	1.59 s	+6%
DL6 explore volume, level filter before json	24h	6.38 s	6.90 s	+8%	5.98 s	6.45 s	+8%

The detected_level shapes (DLV1-DLV6), 1h with Loki and 6h and 24h without:

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
DLV1 drilldown logs volume, json	1h	486 ms	500 ms	+3%	477 ms	526 ms	+10%	same
DLV2 json error filter, detected_level	1h	2.08 s	2.23 s	+7%	2.06 s	2.19 s	+7%	same
DLV3 plain json, detected_level	1h	508 ms	530 ms	+4%	487 ms	510 ms	+5%	same
DLV4 explore json volume, level and detected_level	1h	36 ms	48 ms	+33%	36 ms	43 ms	+19%	same
DLV5 json error dropped, detected_level	1h	55 ms	8 ms	-85%	67 ms	7 ms	-90%	same
DLV6 drilldown detected_level breakdown, json	1h	2.29 s	9 ms	-100%	2.22 s	6 ms	-100%	same

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)
DLV1 drilldown logs volume, json	6h	2.94 s	3.16 s	+7%	3.09 s	3.26 s	+6%
DLV1 drilldown logs volume, json	24h	9.03 s	9.63 s	+7%	9.65 s	10.15 s	+5%
DLV2 json error filter, detected_level	6h	7.34 s	7.96 s	+8%	7.53 s	8.15 s	+8%
DLV2 json error filter, detected_level	24h	7.48 s	7.99 s	+7%	7.48 s	8.04 s	+8%
DLV3 plain json, detected_level	6h	2.96 s	3.21 s	+9%	3.09 s	3.29 s	+7%
DLV3 plain json, detected_level	24h	9.17 s	9.57 s	+4%	9.19 s	9.86 s	+7%
DLV4 explore json volume, level and detected_level	6h	143 ms	106 ms	-26%	157 ms	184 ms	+17%
DLV4 explore json volume, level and detected_level	24h	689 ms	700 ms	+2%	746 ms	732 ms	-2%
DLV5 json error dropped, detected_level	6h	503 ms	28 ms	-94%	595 ms	26 ms	-96%
DLV5 json error dropped, detected_level	24h	2.41 s	174 ms	-93%	2.45 s	184 ms	-92%
DLV6 drilldown detected_level breakdown, json	6h	7.79 s	25 ms	-100%	7.83 s	26 ms	-100%
DLV6 drilldown detected_level breakdown, json	24h	7.70 s	178 ms	-98%	7.85 s	182 ms	-98%

DLV5 (the stored-level plan reads no body) and DLV6 (Logs Drilldown's breakdown,
the raw-row evaluator on main) are faster; DLV1-DLV3 are the raw-row evaluator,
unchanged. Control set at 1h, both orders (every shape is within noise; Loki
parity is the same as on main, topk rate by app differs on both):

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
logs plain	1h	24 ms	19 ms	-21%	22 ms	25 ms	+14%	same
logs \| json	1h	28 ms	30 ms	+7%	41 ms	36 ms	-12%	same
logs \| logfmt	1h	44 ms	42 ms	-5%	77 ms	61 ms	-21%	same
json volume no filters	1h	43 ms	105 ms	+144%	81 ms	60 ms	-26%	same
logfmt volume	1h	126 ms	111 ms	-12%	128 ms	159 ms	+24%	same
plain volume	1h	9 ms	8 ms	-11%	7 ms	8 ms	+14%	same
stream-label count by app	1h	13 ms	12 ms	-8%	11 ms	13 ms	+18%	same
ungrouped rate	1h	5 ms	7 ms	+40%	5 ms	6 ms	+20%	same
bytes_over_time by app	1h	26 ms	25 ms	-4%	24 ms	25 ms	+4%	same
topk rate by app	1h	19 ms	19 ms	+0%	26 ms	26 ms	+0%	differs
detected_level volume	1h	12 ms	20 ms	+67%	10 ms	23 ms	+130%	same
by service_name	1h	21 ms	16 ms	-24%	14 ms	15 ms	+7%	same
drilldown level volume	1h	7 ms	7 ms	+0%	9 ms	7 ms	-22%	same
drilldown service landing	1h	18 ms	14 ms	-22%	11 ms	13 ms	+18%	same
drilldown detected_level	1h	22 ms	18 ms	-18%	18 ms	17 ms	-6%	same
json rate by level (no filter, no drop)	1h	2 ms	2 ms	+0%	3 ms	2 ms	-33%	same
logfmt count by level, filter, drop	1h	170 ms	182 ms	+7%	205 ms	152 ms	-26%	same
instant ungrouped rate	instant	4 ms	6 ms	+50%	5 ms	5 ms	+0%	same
instant json volume	instant	177 ms	189 ms	+7%	200 ms	200 ms	+0%	same
instant json filtered	instant	157 ms	161 ms	+3%	184 ms	165 ms	-10%	same

Cold windows with a level stream label

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	64 ms	96 ms	+50%	87 ms	50 ms	-43%	same
DL6 explore volume, level filter before json	1h	540 ms	617 ms	+14%	510 ms	635 ms	+25%	same series, sum ±0.27%

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)
D volume, no filters	6h	934 ms	611 ms	-35%	669 ms	731 ms	+9%
D volume, no filters	24h	2.58 s	2.41 s	-7%	2.48 s	2.69 s	+8%
DL6 explore volume, level filter before json	6h	3.38 s	3.81 s	+13%	3.21 s	3.36 s	+5%
DL6 explore volume, level filter before json	24h	11.99 s	12.49 s	+4%	11.88 s	12.00 s	+1%

Data with no stored level (the level only in the line)

The same backfill seeded with --no-level-label (the generator leaves the level
out of the stream labels; the lines keep it in the JSON or logfmt body, or have
none). This is the data the chain reads, and where main answers 4 series for D
where Loki answers 8. The previous revision's numbers, for comparison (base ->
branch, base measured first): D 38 -> 282 ms at 1h, 547 ms -> 1.50 s at 6h, 2.29
s -> 6.09 s at 24h; DLV5 56 -> 99 ms, 532 ms -> 897 ms, 2.24 s -> 3.79 s.

Warm, VictoriaLogs alone (no Loki in the loop), 15 runs, 1h, each order:

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	84 ms	105 ms	+25%	87 ms	102 ms	+17%
DLV5 json error dropped, detected_level	1h	88 ms	99 ms	+13%	85 ms	101 ms	+19%

Warm, with Loki in the loop at 1h (the position in the pair moves D: 118 -> 97
ms with the base first, 95 -> 169 ms with the branch first), and at 6h and 24h
(VictoriaLogs only):

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	118 ms	97 ms	-18%	95 ms	169 ms	+78%	same
DL6 explore volume, level filter before json	1h	196 ms	296 ms	+51%	216 ms	269 ms	+25%	differs

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
DLV1 drilldown logs volume, json	1h	483 ms	507 ms	+5%	472 ms	498 ms	+6%	same
DLV2 json error filter, detected_level	1h	2.04 s	2.29 s	+12%	1.97 s	2.26 s	+14%	same
DLV3 plain json, detected_level	1h	498 ms	543 ms	+9%	482 ms	506 ms	+5%	same
DLV4 explore json volume, level and detected_level	1h	36 ms	40 ms	+11%	36 ms	41 ms	+14%	same
DLV5 json error dropped, detected_level	1h	68 ms	88 ms	+29%	91 ms	60 ms	-34%	same
DLV6 drilldown detected_level breakdown, json	1h	2.25 s	106 ms	-95%	2.14 s	52 ms	-98%	same

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)
D volume, no filters	6h	541 ms	692 ms	+28%	570 ms	702 ms	+23%
D volume, no filters	24h	2.39 s	2.75 s	+15%	2.30 s	2.84 s	+24%
DL6 explore volume, level filter before json	6h	1.45 s	1.59 s	+10%	1.44 s	1.50 s	+4%
DL6 explore volume, level filter before json	24h	6.04 s	6.24 s	+3%	6.00 s	6.27 s	+4%

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)
DLV1 drilldown logs volume, json	6h	2.92 s	3.15 s	+8%	3.08 s	3.21 s	+4%
DLV1 drilldown logs volume, json	24h	9.60 s	10.35 s	+8%	9.51 s	10.20 s	+7%
DLV2 json error filter, detected_level	6h	7.56 s	8.25 s	+9%	7.32 s	8.04 s	+10%
DLV2 json error filter, detected_level	24h	7.44 s	8.15 s	+9%	7.19 s	7.97 s	+11%
DLV3 plain json, detected_level	6h	2.92 s	3.15 s	+8%	3.07 s	3.27 s	+7%
DLV3 plain json, detected_level	24h	9.20 s	9.77 s	+6%	9.30 s	9.98 s	+7%
DLV4 explore json volume, level and detected_level	6h	150 ms	144 ms	-4%	142 ms	157 ms	+11%
DLV4 explore json volume, level and detected_level	24h	691 ms	734 ms	+6%	699 ms	743 ms	+6%
DLV5 json error dropped, detected_level	6h	585 ms	641 ms	+10%	589 ms	698 ms	+19%
DLV5 json error dropped, detected_level	24h	2.24 s	2.74 s	+22%	2.25 s	2.79 s	+24%
DLV6 drilldown detected_level breakdown, json	6h	7.49 s	666 ms	-91%	7.78 s	711 ms	-91%
DLV6 drilldown detected_level breakdown, json	24h	7.51 s	2.61 s	-65%	7.91 s	2.84 s	-64%

Cold (the first request of a shape in a fresh process: the stored-level attempt,
the level probes, the presence probe and the chain, none of them remembered):

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
D volume, no filters	1h	68 ms	591 ms	+769%	180 ms	458 ms	+154%	same
DL6 explore volume, level filter before json	1h	620 ms	502 ms	-19%	582 ms	615 ms	+6%	differs

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)	vs Loki
DLV1 drilldown logs volume, json	1h	503 ms	521 ms	+4%	474 ms	518 ms	+9%	same
DLV2 json error filter, detected_level	1h	1.96 s	2.26 s	+15%	1.95 s	2.20 s	+13%	same
DLV3 plain json, detected_level	1h	491 ms	574 ms	+17%	479 ms	509 ms	+6%	same
DLV4 explore json volume, level and detected_level	1h	21 ms	27 ms	+29%	21 ms	26 ms	+24%	same
DLV5 json error dropped, detected_level	1h	37 ms	307 ms	+730%	88 ms	316 ms	+259%	same
DLV6 drilldown detected_level breakdown, json	1h	2.29 s	39 ms	-98%	2.14 s	61 ms	-97%	same

shape	range	base	branch	change	base (other order)	branch (other order)	change (other order)
D volume, no filters	6h	868 ms	2.71 s	+212%	673 ms	2.93 s	+336%
D volume, no filters	24h	2.77 s	6.75 s	+144%	2.68 s	6.92 s	+159%
DL6 explore volume, level filter before json	6h	3.28 s	3.37 s	+3%	3.29 s	3.47 s	+5%
DL6 explore volume, level filter before json	24h	11.38 s	12.51 s	+10%	11.45 s	11.87 s	+4%

The cold request of D at 1h on the VictoriaLogs side, from a proxy restarted before
each request, three requests (duration of each call): the heaviest guard probe
264-296 ms, the stored-level attempt 64-127 ms, the other guard probes 29-188
ms, the presence probe 71-182 ms, the chain 200-296 ms (it shares the 2 CPUs
with the probes); 470-515 ms in all against 80 ms for main and 550-680 ms for
the previous revision (both builds restarted on one stack). The heaviest probe is
detectedLevelKeysRisk (two level keys in a line, an escape in a level value, a
brace without its closing brace: four regexp alternatives over the lines that
hold a level-key word), which this change added; main's cold request runs
its partial-parse probe (55-75 ms) and its stats query only. The probes are the
cold cost, not the chain: the first view of a window pays them once per five
minutes (see "Not done").

The harness's cold value is one sample per run, the first request in a fresh
process, and does not compare between sessions: this revision's table above says
591 ms and 458 ms for D, the previous revision's body said 304 ms and 218 ms. The
comparison to read is the restart measurement on one stack above: previous
revision 550-680 ms, this one 470-515 ms, main 80 ms.

Where the time goes (VictoriaLogs alone, 1h, body-only data)

Cgroup CPU time per query is the measure to read, not the wall time: at 2 CPUs
VictoriaLogs' wall time moves in steps of the 100 ms quota period, and at 6h and
24h the wall time follows the CPU time. Median of 12 runs, the queries the proxy
sends for Explore's volume (D); without X leaves one group of pipes out of the
chain, so the difference is the cost of X:

query	wall	CPU per query	series
main's	102 ms	215-225 ms	4 (Loki 8)
previous chain	290 ms	550 ms	8
this revision's chain	107-114 ms	260-280 ms	8
the same, without the witness	95 ms	182 ms	8

group of pipes	previous chain: CPU it costs	this revision: CPU it costs
the plan's own unpack_json	194 ms	0 (shared)
the keyword regexp (and its finals)	144 ms	48 ms
the witness	102 ms	87 ms
the chain's unpack_json	(in the plan's)	about 20 ms
unpack_logfmt and its picks	about 0	about 0

DLV5 (no label of the plan): main's 99 ms / 217 CPU-ms, previous chain 190 ms /
344 CPU-ms, this revision's 107 ms / 241-258 CPU-ms.

What is left on the warm path is the witness (a third of the chain's CPU) and
the marker pipe it reads. It evaluates 13 body-key words and 15 stored-field
tests per row; grouping the body-key words under one test of the marker
measured 10 to 20 CPU-ms less and is not done.

Quality gate: High-concurrency throughput

TestLoad_HighConcurrency_MemoryStability (100 goroutines x 1000 requests to
handleLabels against a fake backend, median of 3, measured for the head first
and for the base four minutes later on the same runner) does not touch this
change: handleLabels and the cache are unchanged, and the diff touches the
detected_level paths and vlPostHTTP, where every POST now costs one substring
check for the chain (a needle longer than the query returns at once). Six alternating local runs of 3 repetitions on the
base and the head binaries (8 CPUs, GOMAXPROCS=4) gave the same distribution:
the per-run medians, which is what the gate compares, were 1.09M-1.18M requests/s
for the base (median 1.146M) and 0.85M-1.18M for the head (median 1.156M); one
head run fell 26% below the others with the same code, as the gate's -27.8%
did. The same workflow passed on the two earlier runs of this branch. It is
runner noise.

Data-to-data match against Loki

Every row above carries "result vs Loki" from the harness: series count equal
and the sum of all points within the shown percentage of Loki's, for base and
branch alike (the harness fails a run whose branch differs from Loki where the
base did not: 0 new differences in every run; one pre-existing, topk rate by app, which differs on main too).

Not done

The native metric route for a plain selector (what Drilldown sends for
sum by (detected_level) without a parser) does not use the stored-level plan
(its query comes from the translator, which has no window): it keeps the chain,
pruned by the presence probe, unguarded:
{"msg":"x","severity":"error","level":"info"} is info where Loki says
error, and a detected_level stream label is normalized. Measured on the
same fixtures; registered open as
semantics/detected-level-pipes-known-differences. The Explore plain-selector
volume plan has no raw-row fallback either, so it runs the chain without the
guard.
Differences no guard sees, measured against Loki v3.7.7 and registered in
the same row: a stream level field outranked in the pipes' list order by a
metadata one (stream severity=error with metadata level=info: error in
Loki and under | json, info under | json | drop __error__); a logfmt
level key in another case (msg="error x" LvL=warn: Loki warn, pipes
error); a logfmt syntax error before the level key
(msg="unterminated error level=info: Loki error, pipes info); a row
VictoriaLogs unpacked from a JSON body with a stored level
({"level":"notice","msg":"error later"}: Loki error, pipes notice, the
same on main). The raw-row evaluator reads the first three like Loki.
The guard is a superset (see Code): a tenant whose bodies carry two level
keys gets the raw-row evaluator, and its row limit, on long ranges.
Rows with no stored level are still slower than main, by less. A window
with a row that has no stored level (plain text tenants, OTLP records, a
collector that does not promote the level) pays the chain, which reads every
line, after the stored-level attempt. On the harness's backfill seeded
without the level label (see the A/B), VictoriaLogs alone: Explore's JSON
volume (D) 84 ms -> 105 ms and 87 ms -> 102 ms at 1h (both orders, 15 runs),
0.54 s -> 0.69 s at 6h, 2.39 s -> 2.75 s at 24h; DLV5 88 -> 99 ms at 1h,
2.24 s -> 2.74 s at 24h; main answers 4 series for D there where Loki answers
8. What is left is the witness (87 of 269 CPU-ms), the keyword regexp (48) and
the level keys' parse (20). The first view of a window is the larger gap:
the probes (detectedLevelKeysRisk, which this change added, alone takes 290 ms),
the presence probe and the stored-level attempt run before and beside the chain: 470-515 ms for D at
1h against 80 ms for main (1h cold, harness: 591 ms and 458 ms against 68 ms
and 180 ms; 24h: 6.8 s against 2.7 s), once per five minutes for a window the
proxy remembers. Options, none taken because each is a trade of exactness or
a plan change: fold the presence probe into a guard probe (one scan less);
gate the heaviest guard probe's regexps behind word filters as the keyword scan is
(each needs its own exactness proof); trade the witness for a
shorter memory of the mask (the witness exists so a late line cannot be
misread); contains_any instead of nine word tests (VictoriaLogs v1.52 only,
so a capability-gated fallback). The native route pays the chain on every window (warm 12 ms -> 20 ms and
10 ms -> 23 ms in the two orders, the control set's detected_level volume; on
a cache miss 14 ms -> 106 ms, measured on the previous revision).
splitGroupSlack and errStatsGroupGuard also exist on branch
perf/json-valid-split, which this change does not touch; whichever lands second
reconciles the two.
Query sizes (the keyword gate adds 124 bytes to every chain, the figures below are
those of the previous revision plus that): the stored-level plan sends a few hundred bytes of pipes. The
chain path is 4.2 KB for Explore's volume with a service_name selector (9.8 KB
unpruned) and 8.0 KB with a regexp selector of seven long names (14.4 KB
unpruned; the generator's seven names match no row, so the chain is at its
smallest there: the bound is the full chain, 14.4 KB, and the presence probe
is the base plus 1.4 KB; the largest request is then a guard probe at 9.4 KB;
the witness adds up to 1.5 KB to a pruned request, and a unit test keeps the
witnessed request of a 6 KB selector under the limit), against
VictoriaLogs' 16 KiB -search.maxQueryLen; a window whose presence cannot be
told keeps the full chain and its size, and a longer query answers 502 as any
over-long query does.
Staleness: a remembered verdict of the guard probes of the part of a window
older than five minutes can miss a line that arrives later than five minutes
after its timestamp, until the entry expires (five minutes after the check of
its oldest part). The part younger than five minutes is always probed. The
presence mask has no such window: the narrowed query flags a row it would
misread (the witness) and is asked again with the full chain.
The presence probe is made before the request takes its
stats_query_range slot in the stats pushdown (collectRangeMetricHits); the
burst-fused Drilldown breakdown (drilldown_burst_coalescer.go) and the
breakdown handler (metric_binary.go) take a slot first and probe inside
it. A slot cannot deadlock on the probe (the probe takes no slot and holds
no admission token while the request waits), it can only be held for one
probe.
No visual capture, no real Grafana range matrix and no backend load test for
this change; the A/B and e2e above are the proofs.
DL6's | detected_level="info" before the parser keeps the translator's
rendering, which main uses too (unpack_logfmt | filter level:="info" in the
base of the stats query): the sum differs from Loki's by 0.2% to 0.4% on base
and branch alike (sum +-0.19%, +-0.36% above). The Summary's "a filter on
detected_level compares that value" holds for filters after the parser; this
one is not changed here.

github-actions · 2026-10-13T18:43:00Z

🟢 Performance A/B: faster, no regression

139 shape×range — 2 🟢 faster · 137 ⚪ same · results vs Loki: 119 same, 0 new difference(s), 20 pre-existing

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
🟢	DLV5 json error dropped, detected_level	1h	196 ms	18 ms ¹	🟢 -91%	43 ms	0.42×	✅ same
🟢	DLV6 drilldown detected_level breakdown, json	1h	5.41 s	18 ms ¹	🟢 -100%	28 ms	0.64×	✅ same

¹ re-measured with 7 runs, on windows the first pass did not use, after it moved beyond noise.

137 unchanged shape×range (within noise)

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
⚪	bytes_over_time by app	1h	71 ms	69 ms	⚪ -3%	90 ms	0.77×	✅ same
⚪	detected_level volume	1h	16 ms	89 ms	⚪ +456%	38 ms	2.34×	✅ same
⚪	drilldown detected_level	1h	81 ms	71 ms	⚪ -12%	25 ms	2.84×	✅ same
⚪	drilldown level volume	1h	14 ms	14 ms	⚪ +0%	25 ms	0.56×	✅ same
⚪	drilldown service landing	1h	33 ms	32 ms	⚪ -3%	25 ms	1.28×	✅ same
⚪	instant json filtered	instant	423 ms	426 ms	⚪ +1%	74 ms	5.76×	✅ same
⚪	instant json volume	instant	484 ms	514 ms	⚪ +6%	104 ms	4.94×	✅ same
⚪	instant ungrouped rate	instant	5 ms	7 ms	⚪ +40%	75 ms	0.09×	✅ same
⚪	json rate by level (no filter, no drop)	1h	400	400	⚪ -33%	400	—	✅ same
⚪	json volume no filters	1h	203 ms	203 ms	⚪ +0%	3 ms	68×	✅ same
⚪	logfmt count by level, filter, drop	1h	473 ms	447 ms	⚪ -5%	57 ms	7.84×	✅ same
⚪	logfmt volume	1h	307 ms	299 ms	⚪ -3%	63 ms	4.75×	✅ same
⚪	logs plain	1h	64 ms	66 ms	⚪ +3%	39 ms	1.69×	✅ same
⚪	logs \| json	1h	90 ms	88 ms	⚪ -2%	59 ms	1.49×	✅ same
⚪	logs \| logfmt	1h	148 ms	152 ms	⚪ +3%	52 ms	2.92×	✅ same
⚪	plain volume	1h	16 ms	21 ms	⚪ +31%	39 ms	0.54×	✅ same
⚪	stream-label count by app	1h	24 ms	27 ms	⚪ +12%	39 ms	0.69×	✅ same
⚪	topk rate by app	1h	43 ms	46 ms	⚪ +7%	87 ms	0.53×	➖ differs (as base)
⚪	ungrouped rate	1h	11 ms	11 ms	⚪ +0%	81 ms	0.14×	✅ same
⚪	DB3 json field breakdown user_id	1h	48 ms	51 ms	⚪ +6%	24 ms	2.12×	✅ same
⚪	DB4 json field breakdown batch_id	1h	97 ms	99 ms	⚪ +2%	49 ms	2.02×	✅ same
⚪	X1 json filter level_extracted, by level	1h	389 ms	391 ms	⚪ +1%	42 ms	9.31×	✅ same
⚪	X2 json group by level_extracted	1h	301 ms	308 ms	⚪ +2%	45 ms	6.84×	✅ same
⚪	X3 json rate, level_extracted equality	1h	275 ms	296 ms	⚪ +8%	36 ms	8.22×	✅ same
⚪	X4 json bytes, level_extracted regexp	1h	398 ms	397 ms	⚪ -0%	90 ms	4.41×	✅ same
⚪	X5 logfmt filter level_extracted, by level	1h	670 ms	677 ms	⚪ +1%	53 ms	13×	✅ same
⚪	E1 logs, json parser over stream labels	1h	120 ms	113 ms	⚪ -6%	64 ms	1.77×	✅ same
⚪	E2 logs, logfmt parser over stream labels	1h	72 ms	72 ms	⚪ +0%	53 ms	1.36×	✅ same
⚪	E3 logs, json, existence filter on level_extracted	1h	138 ms	140 ms	⚪ +1%	73 ms	1.92×	✅ same
⚪	E4 logs, logfmt, existence filter on level_extracted	1h	99 ms	91 ms	⚪ -8%	57 ms	1.60×	✅ same
⚪	E5 logs, json, drop level_extracted	1h	116 ms	114 ms	⚪ -2%	61 ms	1.87×	✅ same
⚪	E6 logfmt count by level_extracted	1h	47 ms	44 ms	⚪ -6%	7 ms	6.29×	✅ same
⚪	E7 json count by level_extracted	1h	144 ms	145 ms	⚪ +1%	14 ms	10×	✅ same
⚪	E8 instant logfmt by level_extracted	instant	6 ms	6 ms	⚪ +0%	9 ms	0.67×	✅ same
⚪	DLV1 drilldown logs volume, json	1h	1.23 s	1.27 s	⚪ +3%	12 ms	106×	✅ same
⚪	DLV2 json error filter, detected_level	1h	5.16 s	5.55 s	⚪ +8%	44 ms	126×	✅ same
⚪	DLV3 plain json, detected_level	1h	1.23 s	1.28 s	⚪ +5%	14 ms	92×	✅ same
⚪	DLV4 explore json volume, level and detected_level	1h	75 ms	76 ms	⚪ +1%	10 ms	7.60×	✅ same
⚪	A explore volume, 2 filters, drop	1h	157 ms	148 ms	⚪ -6%	108 ms	1.37×	✅ same
⚪	B grouped sum, filter, no drop	1h	122 ms	107 ms	⚪ -12%	38 ms	2.82×	✅ same
⚪	C ungrouped sum, filter, no drop	1h	103 ms	103 ms	⚪ +0%	35 ms	2.94×	✅ same
⚪	D volume, no filters	1h	222 ms	228 ms	⚪ +3%	47 ms	4.85×	✅ same
⚪	E field breakdown pipeline (explore)	1h	106 ms	103 ms	⚪ -3%	37 ms	2.78×	✅ same
⚪	F field breakdown service_version (explore)	1h	155 ms	171 ms	⚪ +10%	42 ms	4.07×	✅ same
⚪	G field breakdown pipeline (drilldown)	1h	44 ms	46 ms	⚪ +5%	23 ms	2.00×	✅ same
⚪	H field breakdown service_version (drilldown)	1h	116 ms	102 ms	⚪ -12%	25 ms	4.08×	✅ same
⚪	I grafana volume of B	1h	98 ms	100 ms	⚪ +2%	35 ms	2.86×	✅ same
⚪	O1 explore logs, underscore structured-metadata filter	1h	368 ms	371 ms	⚪ +1%	386 ms	0.96×	✅ same
⚪	O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	71 ms	69 ms	⚪ -3%	35 ms	1.97×	✅ same
⚪	O3 explore logs, dotted structured-metadata filter	1h	400	400	⚪ +0%	400	—	✅ same
⚪	O4 explore log volume (Grafana form), dotted structured-metadata filter	1h	400	400	⚪ +0%	400	—	✅ same
⚪	drilldown detected fields	1h	1 ms	1 ms	⚪ +0%	16 ms	0.06×	➖ differs (as base)
⚪	drilldown detected labels	1h	1 ms	1 ms	⚪ +0%	3 ms	0.33×	➖ differs (as base)
⚪	drilldown landing volume	1h	56 ms	58 ms	⚪ +4%	20 ms	2.90×	➖ differs (as base)
⚪	explore label values env	1h	1 ms	1 ms	⚪ +0%	9 ms	0.11×	✅ same
⚪	explore label values service_name	1h	1 ms	1 ms	⚪ +0%	12 ms	0.08×	✅ same
⚪	explore labels (label browser)	1h	1 ms	1 ms	⚪ +0%	3 ms	0.33×	✅ same
⚪	explore labels with selector	1h	1 ms	1 ms	⚪ +0%	11 ms	0.09×	✅ same
⚪	refresh label values env	1h	20 ms	20 ms	⚪ +0%	9 ms	2.22×	✅ same
⚪	refresh label values service_name	1h	9 ms	14 ms	⚪ +56%	11 ms	1.27×	✅ same
⚪	refresh labels (label browser)	1h	11 ms	14 ms	⚪ +27%	3 ms	4.67×	✅ same
⚪	refresh labels with selector	1h	14 ms	11 ms	⚪ -21%	11 ms	1.00×	✅ same
⚪	DL1 drilldown field breakdown json-expr, level filter	1h	501 ms	505 ms	⚪ +1%	22 ms	23×	✅ same
⚪	DL2 drilldown field breakdown, level + stream-label filter before json	1h	56 ms	51 ms	⚪ -9%	6 ms	8.50×	✅ same
⚪	DL3 drilldown logfmt field breakdown, level filter	1h	617 ms	621 ms	⚪ +1%	82 ms	7.57×	➖ differs (as base)
⚪	DL4 instant count, level filter, json+logfmt, 1h	instant	535 ms	521 ms	⚪ -3%	1.34 s	0.39×	✅ sum ±0.31%
⚪	DL6 explore volume, level filter before json	1h	526 ms	566 ms	⚪ +8%	39 ms	15×	✅ sum ±0.31%
⚪	OR1 logs, or alternatives	1h	77 ms	74 ms	⚪ -4%	60 ms	1.23×	✅ same
⚪	OR2 logs, negated or alternatives	1h	68 ms	71 ms	⚪ +4%	45 ms	1.58×	✅ same
⚪	OR3 logs, regexp or alternatives	1h	62 ms	61 ms	⚪ -2%	63 ms	0.97×	✅ same
⚪	OR4 count by level, or alternatives with parser	1h	79 ms	83 ms	⚪ +5%	29 ms	2.86×	✅ same
⚪	OR5 rate, negated regexp or alternatives	1h	55 ms	57 ms	⚪ +4%	83 ms	0.69×	✅ same
⚪	OR6 instant, or alternatives	instant	7 ms	7 ms	⚪ +0%	60 ms	0.12×	✅ same
⚪	P1 api-gateway empty pattern	1h	23 ms	23 ms	⚪ +0%	7 ms	3.29×	✅ same
⚪	P2 api-gateway not empty pattern, line filter	1h	75 ms	81 ms	⚪ +8%	18 ms	4.50×	✅ same
⚪	P3 nginx count_over_time, pattern with spaces	1h	12 ms	12 ms	⚪ +0%	6 ms	2.00×	✅ same
⚪	P4 nginx count_over_time, GET request pattern	1h	14 ms	15 ms	⚪ +7%	6 ms	2.50×	✅ same
⚪	A1 nginx include GET, log query	1h	20 ms	19 ms	⚪ -5%	11 ms	1.73×	✅ same
⚪	A2 nginx exclude GET, log query	1h	17 ms	16 ms	⚪ -6%	12 ms	1.33×	✅ same
⚪	A3 nginx include GET, count_over_time	1h	1 ms	1 ms	⚪ +0%	2 ms	0.50×	✅ same
⚪	A4 nginx exclude GET, count_over_time	1h	19 ms	18 ms	⚪ -5%	6 ms	3.00×	✅ same
⚪	A5 drilldown exclude, volume by detected_level	1h	22 ms	26 ms	⚪ +18%	5 ms	5.20×	✅ same
⚪	A6 nginx include ends on a literal, log query	1h	14 ms	13 ms	⚪ -7%	4 ms	3.25×	✅ same
⚪	A7 nginx exclude ends on a literal, log query	1h	29 ms	29 ms	⚪ +0%	11 ms	2.64×	✅ same
⚪	A8 nginx include two literals, count_over_time	1h	22 ms	23 ms	⚪ +5%	6 ms	3.83×	✅ same
⚪	A9 nginx include two literals, ends on a literal, count_over_time	1h	21 ms	20 ms	⚪ -5%	6 ms	3.33×	✅ same
⚪	P1 logs, json, level equals	1h	126 ms	136 ms	⚪ +8%	70 ms	1.94×	✅ same
⚪	P2 logs, logfmt, level not equal	1h	30 ms	30 ms	⚪ +0%	5 ms	6.00×	✅ same
⚪	P3 logs, json, level existence	1h	127 ms	120 ms	⚪ -6%	76 ms	1.58×	✅ same
⚪	P4 logfmt count by level	1h	32 ms	33 ms	⚪ +3%	6 ms	5.50×	✅ same
⚪	P5 json count by level	1h	83 ms	82 ms	⚪ -1%	13 ms	6.31×	✅ same
⚪	P6 json count, level filter	1h	15 ms	16 ms	⚪ +7%	13 ms	1.23×	✅ same
⚪	P7 json rate by level, level regexp	1h	16 ms	17 ms	⚪ +6%	14 ms	1.21×	✅ same
⚪	P8 json count by a name that is no stream label	1h	87 ms	86 ms	⚪ -1%	12 ms	7.17×	✅ same
⚪	P9 instant logfmt by level	instant	5 ms	6 ms	⚪ +20%	10 ms	0.60×	✅ same
⚪	SF1 logs, json extraction list	1h	91 ms	91 ms	⚪ +0%	40 ms	2.27×	✅ same
⚪	SF10 volume, structured-metadata filter	1h	13 ms	15 ms	⚪ +15%	28 ms	0.54×	✅ same
⚪	SF2 logs, logfmt extraction list	1h	154 ms	157 ms	⚪ +2%	42 ms	3.74×	✅ same
⚪	SF3 logs, json then line_format	1h	102 ms	100 ms	⚪ -2%	57 ms	1.75×	✅ same
⚪	SF4 logs, line_format without a parser	1h	68 ms	66 ms	⚪ -3%	38 ms	1.74×	✅ same
⚪	SF5 logs, regexp capture	1h	165 ms	158 ms	⚪ -4%	57 ms	2.77×	✅ same
⚪	SF6 logs, structured-metadata filter	1h	73 ms	66 ms	⚪ -10%	123 ms	0.54×	✅ same
⚪	SF7 logs, filter on a line key without a parser	1h	67 ms	71 ms	⚪ +6%	18 ms	3.94×	✅ same
⚪	SF8 volume, json extraction list	1h	21 ms	24 ms	⚪ +14%	44 ms	0.55×	✅ same
⚪	SF9 volume, logfmt label_format rename	1h	539 ms	571 ms	⚪ +6%	62 ms	9.21×	✅ same
⚪	E1 json duration_ms sum by service	1h	87 ms	86 ms	⚪ -1%	14 ms	6.14×	➖ differs (as base)
⚪	E2 json duration_ms p95 by service	1h	425 ms	426 ms	⚪ +0%	15 ms	28×	✅ same
⚪	E3 logfmt latency_ms rate, production	1h	501 ms	499 ms	⚪ -0%	133 ms	3.75×	➖ differs (as base)
⚪	E4 logfmt size_bytes max by service	1h	45 ms	45 ms	⚪ +0%	15 ms	3.00×	➖ differs (as base)
⚪	E5 json duration_ms sum, instant	instant	35 ms	37 ms	⚪ +6%	26 ms	1.42×	✅ same
⚪	E6 logfmt ttl unit string (Loki 400)	1h	400	400	⚪ +2%	400	—	✅ same
⚪	E7 json hexadecimal trace_id (Loki 400)	1h	400	400	⚪ +20%	400	—	✅ same
⚪	E8 json duration_ms sum by service, namespace prod	1h	191 ms	191 ms	⚪ +0%	82 ms	2.33×	➖ differs (as base)
⚪	E9 logfmt latency_ms sum by service, namespace prod	1h	308 ms	305 ms	⚪ -1%	91 ms	3.35×	➖ differs (as base)
⚪	U1 logfmt amount rate	1h	37 ms	38 ms	⚪ +3%	12 ms	3.17×	✅ same
⚪	U2 json duration_ms rate	1h	88 ms	88 ms	⚪ +0%	23 ms	3.83×	✅ same
⚪	U3 json duration_ms rate, error filter	1h	88 ms	85 ms	⚪ -3%	27 ms	3.15×	✅ same
⚪	U4 logfmt latency_ms rate, production	1h	347 ms	351 ms	⚪ +1%	124 ms	2.83×	➖ differs (as base)
⚪	U5 logfmt amount rate, instant	instant	7 ms	7 ms	⚪ +0%	10 ms	0.70×	✅ same
⚪	U6 json duration_ms rate, instant	instant	12 ms	12 ms	⚪ +0%	20 ms	0.60×	✅ same
⚪	U7 json duration_ms sum_over_time, instant	instant	15 ms	15 ms	⚪ +0%	20 ms	0.75×	✅ same
⚪	U8 logfmt amount rate by service (grouped)	1h	44 ms	41 ms	⚪ -7%	12 ms	3.42×	✅ same
⚪	U9 logfmt amount rate_counter (folded per series)	1h	287 ms	285 ms	⚪ -1%	32 ms	8.91×	➖ differs (as base)
⚪	U1 json duration_ms sum by service	1h	84 ms	85 ms	⚪ +1%	14 ms	6.07×	➖ differs (as base)
⚪	U10 instant sum_over_time unwrap	instant	37 ms	35 ms	⚪ -5%	27 ms	1.30×	✅ same
⚪	U11 instant max_over_time missing label	instant	4 ms	4 ms	⚪ +0%	13 ms	0.31×	✅ same
⚪	U12 json duration_ms over every service	1h	236 ms	235 ms	⚪ -0%	42 ms	5.60×	➖ differs (as base)
⚪	U13 logfmt latency_ms over every service	1h	493 ms	498 ms	⚪ +1%	58 ms	8.59×	➖ differs (as base)
⚪	U14 duration() of a unit string	1h	45 ms	47 ms	⚪ +4%	7 ms	6.71×	➖ differs (as base)
⚪	U2 json duration_ms max by service	1h	92 ms	93 ms	⚪ +1%	37 ms	2.51×	➖ differs (as base)
⚪	U3 json duration_ms sliding sum	1h	84 ms	85 ms	⚪ +1%	31 ms	2.74×	➖ differs (as base)
⚪	U4 logfmt size_bytes sum by service	1h	39 ms	40 ms	⚪ +3%	7 ms	5.71×	➖ differs (as base)
⚪	U5 logfmt ttl unit string (no sample)	1h	40 ms	40 ms	⚪ +0%	6 ms	6.67×	✅ same
⚪	U6 missing label, no parser	1h	17 ms	18 ms	⚪ +6%	9 ms	2.00×	✅ same
⚪	U7 missing label, ungrouped sum	1h	16 ms	16 ms	⚪ +0%	9 ms	1.78×	✅ same
⚪	U8 rate of an unwrapped label	1h	87 ms	87 ms	⚪ +0%	14 ms	6.21×	✅ same
⚪	U9 bytes() conversion	1h	51 ms	53 ms	⚪ +4%	12 ms	4.42×	➖ differs (as base)

Cold (first-run) timings — what a user opening a panel sees

shape	range	base cold	PR cold	Loki cold	PR ÷ Loki cold
DLV5 json error dropped, detected_level	1h	207 ms	17 ms	44 ms	0.39×
DLV6 drilldown detected_level breakdown, json	1h	5.41 s	25 ms	31 ms	0.81×
bytes_over_time by app	1h	71 ms	71 ms	760 ms	0.09×
detected_level volume	1h	16 ms	125 ms	776 ms	0.16×
drilldown detected_level	1h	80 ms	92 ms	731 ms	0.13×
drilldown level volume	1h	16 ms	14 ms	722 ms	0.02×
drilldown service landing	1h	36 ms	34 ms	669 ms	0.05×
instant json filtered	instant	408 ms	429 ms	71 ms	6.04×
instant json volume	instant	464 ms	489 ms	103 ms	4.75×
instant ungrouped rate	instant	6 ms	5 ms	74 ms	0.07×
json rate by level (no filter, no drop)	1h	400	400	400	—
json volume no filters	1h	197 ms	202 ms	3 ms	67.33×
logfmt count by level, filter, drop	1h	474 ms	448 ms	1.26 s	0.35×
logfmt volume	1h	225 ms	243 ms	1.32 s	0.18×
logs plain	1h	99 ms	67 ms	45 ms	1.49×
logs \| json	1h	92 ms	97 ms	63 ms	1.54×
logs \| logfmt	1h	158 ms	147 ms	56 ms	2.62×
plain volume	1h	21 ms	18 ms	771 ms	0.02×
stream-label count by app	1h	26 ms	30 ms	697 ms	0.04×
topk rate by app	1h	50 ms	48 ms	763 ms	0.06×
ungrouped rate	1h	11 ms	10 ms	752 ms	0.01×
DB3 json field breakdown user_id	1h	73 ms	50 ms	751 ms	0.07×
DB4 json field breakdown batch_id	1h	94 ms	101 ms	743 ms	0.14×
X1 json filter level_extracted, by level	1h	3.55 s	3.50 s	819 ms	4.28×
X2 json group by level_extracted	1h	299 ms	300 ms	954 ms	0.31×
X3 json rate, level_extracted equality	1h	269 ms	272 ms	624 ms	0.44×
X4 json bytes, level_extracted regexp	1h	413 ms	405 ms	779 ms	0.52×
X5 logfmt filter level_extracted, by level	1h	663 ms	682 ms	1.03 s	0.66×
E1 logs, json parser over stream labels	1h	151 ms	113 ms	60 ms	1.88×
E2 logs, logfmt parser over stream labels	1h	106 ms	78 ms	53 ms	1.47×
E3 logs, json, existence filter on level_extracted	1h	152 ms	132 ms	75 ms	1.76×
E4 logs, logfmt, existence filter on level_extracted	1h	89 ms	93 ms	71 ms	1.31×
E5 logs, json, drop level_extracted	1h	116 ms	118 ms	59 ms	2.00×
E6 logfmt count by level_extracted	1h	45 ms	46 ms	47 ms	0.98×
E7 json count by level_extracted	1h	1.39 s	1.40 s	184 ms	7.62×
E8 instant logfmt by level_extracted	instant	6 ms	6 ms	10 ms	0.60×
DLV1 drilldown logs volume, json	1h	1.22 s	1.29 s	185 ms	6.96×
DLV2 json error filter, detected_level	1h	5.09 s	5.50 s	893 ms	6.16×
DLV3 plain json, detected_level	1h	1.24 s	1.32 s	186 ms	7.12×
DLV4 explore json volume, level and detected_level	1h	77 ms	83 ms	200 ms	0.41×
A explore volume, 2 filters, drop	1h	2.88 s	1.91 s	1.20 s	1.59×
B grouped sum, filter, no drop	1h	122 ms	125 ms	611 ms	0.20×
C ungrouped sum, filter, no drop	1h	125 ms	120 ms	709 ms	0.17×
D volume, no filters	1h	216 ms	211 ms	876 ms	0.24×
E field breakdown pipeline (explore)	1h	123 ms	117 ms	746 ms	0.16×
F field breakdown service_version (explore)	1h	1.38 s	1.34 s	733 ms	1.83×
G field breakdown pipeline (drilldown)	1h	45 ms	44 ms	681 ms	0.06×
H field breakdown service_version (drilldown)	1h	155 ms	132 ms	698 ms	0.19×
I grafana volume of B	1h	109 ms	109 ms	688 ms	0.16×
O1 explore logs, underscore structured-metadata filter	1h	422 ms	373 ms	379 ms	0.98×
O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	172 ms	146 ms	607 ms	0.24×
O3 explore logs, dotted structured-metadata filter	1h	400	400	400	—
O4 explore log volume (Grafana form), dotted structured-metadata filter	1h	400	400	400	—
drilldown detected fields	1h	29 ms	26 ms	16 ms	1.62×
drilldown detected labels	1h	20 ms	16 ms	3 ms	5.33×
drilldown landing volume	1h	59 ms	54 ms	20 ms	2.70×
explore label values env	1h	73 ms	73 ms	9 ms	8.11×
explore label values service_name	1h	39 ms	36 ms	11 ms	3.27×
explore labels (label browser)	1h	23 ms	2 ms	3 ms	0.67×
explore labels with selector	1h	54 ms	56 ms	11 ms	5.09×
refresh label values env	1h	3 ms	3 ms	9 ms	0.33×
refresh label values service_name	1h	2 ms	2 ms	11 ms	0.18×
refresh labels (label browser)	1h	2 ms	2 ms	2 ms	1.00×
refresh labels with selector	1h	2 ms	2 ms	12 ms	0.17×
DL1 drilldown field breakdown json-expr, level filter	1h	499 ms	517 ms	637 ms	0.81×
DL2 drilldown field breakdown, level + stream-label filter before json	1h	84 ms	94 ms	30 ms	3.13×
DL3 drilldown logfmt field breakdown, level filter	1h	655 ms	638 ms	1.09 s	0.58×
DL4 instant count, level filter, json+logfmt, 1h	instant	526 ms	514 ms	1.36 s	0.38×
DL6 explore volume, level filter before json	1h	1.02 s	1.04 s	732 ms	1.42×
OR1 logs, or alternatives	1h	106 ms	74 ms	58 ms	1.28×
OR2 logs, negated or alternatives	1h	69 ms	68 ms	47 ms	1.45×
OR3 logs, regexp or alternatives	1h	55 ms	58 ms	63 ms	0.92×
OR4 count by level, or alternatives with parser	1h	124 ms	126 ms	502 ms	0.25×
OR5 rate, negated regexp or alternatives	1h	56 ms	58 ms	782 ms	0.07×
OR6 instant, or alternatives	instant	8 ms	7 ms	58 ms	0.12×
P1 api-gateway empty pattern	1h	44 ms	23 ms	85 ms	0.27×
P2 api-gateway not empty pattern, line filter	1h	101 ms	82 ms	20 ms	4.10×
P3 nginx count_over_time, pattern with spaces	1h	13 ms	12 ms	27 ms	0.44×
P4 nginx count_over_time, GET request pattern	1h	14 ms	14 ms	27 ms	0.52×
A1 nginx include GET, log query	1h	56 ms	32 ms	10 ms	3.20×
A2 nginx exclude GET, log query	1h	17 ms	18 ms	11 ms	1.64×
A3 nginx include GET, count_over_time	1h	1 ms	1 ms	2 ms	0.50×
A4 nginx exclude GET, count_over_time	1h	17 ms	18 ms	37 ms	0.49×
A5 drilldown exclude, volume by detected_level	1h	24 ms	37 ms	33 ms	1.12×
A6 nginx include ends on a literal, log query	1h	14 ms	15 ms	22 ms	0.68×
A7 nginx exclude ends on a literal, log query	1h	18 ms	18 ms	12 ms	1.50×
A8 nginx include two literals, count_over_time	1h	23 ms	21 ms	30 ms	0.70×
A9 nginx include two literals, ends on a literal, count_over_time	1h	20 ms	20 ms	24 ms	0.83×
P1 logs, json, level equals	1h	158 ms	129 ms	91 ms	1.42×
P2 logs, logfmt, level not equal	1h	31 ms	38 ms	37 ms	1.03×
P3 logs, json, level existence	1h	125 ms	122 ms	63 ms	1.94×
P4 logfmt count by level	1h	32 ms	33 ms	46 ms	0.72×
P5 json count by level	1h	94 ms	90 ms	182 ms	0.49×
P6 json count, level filter	1h	17 ms	14 ms	167 ms	0.08×
P7 json rate by level, level regexp	1h	16 ms	18 ms	184 ms	0.10×
P8 json count by a name that is no stream label	1h	102 ms	95 ms	153 ms	0.62×
P9 instant logfmt by level	instant	5 ms	6 ms	9 ms	0.67×
SF1 logs, json extraction list	1h	113 ms	99 ms	44 ms	2.25×
SF10 volume, structured-metadata filter	1h	13 ms	21 ms	460 ms	0.05×
SF2 logs, logfmt extraction list	1h	159 ms	159 ms	48 ms	3.31×
SF3 logs, json then line_format	1h	100 ms	98 ms	63 ms	1.56×
SF4 logs, line_format without a parser	1h	69 ms	73 ms	44 ms	1.66×
SF5 logs, regexp capture	1h	160 ms	160 ms	56 ms	2.86×
SF6 logs, structured-metadata filter	1h	68 ms	77 ms	127 ms	0.61×
SF7 logs, filter on a line key without a parser	1h	68 ms	71 ms	503 ms	0.14×
SF8 volume, json extraction list	1h	26 ms	23 ms	838 ms	0.03×
SF9 volume, logfmt label_format rename	1h	540 ms	549 ms	1.38 s	0.40×
E1 json duration_ms sum by service	1h	106 ms	99 ms	217 ms	0.46×
E2 json duration_ms p95 by service	1h	423 ms	424 ms	226 ms	1.88×
E3 logfmt latency_ms rate, production	1h	504 ms	509 ms	1.17 s	0.43×
E4 logfmt size_bytes max by service	1h	46 ms	44 ms	212 ms	0.21×
E5 json duration_ms sum, instant	instant	34 ms	34 ms	26 ms	1.31×
E6 logfmt ttl unit string (Loki 400)	1h	400	400	400	—
E7 json hexadecimal trace_id (Loki 400)	1h	400	400	400	—
E8 json duration_ms sum by service, namespace prod	1h	191 ms	191 ms	643 ms	0.30×
E9 logfmt latency_ms sum by service, namespace prod	1h	304 ms	305 ms	835 ms	0.37×
U1 logfmt amount rate	1h	57 ms	38 ms	58 ms	0.66×
U2 json duration_ms rate	1h	105 ms	87 ms	163 ms	0.53×
U3 json duration_ms rate, error filter	1h	85 ms	89 ms	194 ms	0.46×
U4 logfmt latency_ms rate, production	1h	368 ms	348 ms	1.12 s	0.31×
U5 logfmt amount rate, instant	instant	6 ms	8 ms	9 ms	0.89×
U6 json duration_ms rate, instant	instant	11 ms	12 ms	25 ms	0.48×
U7 json duration_ms sum_over_time, instant	instant	16 ms	30 ms	20 ms	1.50×
U8 logfmt amount rate by service (grouped)	1h	86 ms	42 ms	66 ms	0.64×
U9 logfmt amount rate_counter (folded per series)	1h	302 ms	285 ms	338 ms	0.84×
U1 json duration_ms sum by service	1h	117 ms	84 ms	201 ms	0.42×
U10 instant sum_over_time unwrap	instant	37 ms	36 ms	27 ms	1.33×
U11 instant max_over_time missing label	instant	5 ms	4 ms	14 ms	0.29×
U12 json duration_ms over every service	1h	246 ms	246 ms	869 ms	0.28×
U13 logfmt latency_ms over every service	1h	508 ms	490 ms	1.12 s	0.44×
U14 duration() of a unit string	1h	44 ms	45 ms	67 ms	0.67×
U2 json duration_ms max by service	1h	91 ms	94 ms	972 ms	0.10×
U3 json duration_ms sliding sum	1h	85 ms	87 ms	247 ms	0.35×
U4 logfmt size_bytes sum by service	1h	40 ms	42 ms	47 ms	0.89×
U5 logfmt ttl unit string (no sample)	1h	37 ms	43 ms	44 ms	0.98×
U6 missing label, no parser	1h	17 ms	18 ms	79 ms	0.23×
U7 missing label, ungrouped sum	1h	20 ms	24 ms	100 ms	0.24×
U8 rate of an unwrapped label	1h	87 ms	88 ms	215 ms	0.41×
U9 bytes() conversion	1h	52 ms	53 ms	54 ms	0.98×

Why these shapes

internal/translator/translator.go → 103: control/bytes_over_time by app, control/instant json filtered, control/instant json volume, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logfmt count by level, filter, drop, control/logfmt volume …
internal/proxy/stream_processing.go → 86: control/detected_level volume, control/drilldown detected_level, control/drilldown level volume, control/instant json filtered, control/instant json volume, control/json rate by level (no filter, no drop), control/json volume no filters, control/logfmt volume …
internal/proxy/ordered_json_metric.go → 73: control/bytes_over_time by app, control/instant json filtered, control/instant json volume, control/json rate by level (no filter, no drop), control/json volume no filters, control/logfmt count by level, filter, drop, control/logfmt volume, control/logs | json …
internal/translator/detected_level_logsql.go → 22: control/detected_level volume, control/drilldown detected_level, control/drilldown level volume, control/instant json volume, control/json volume no filters, control/logfmt volume, control/plain volume, json-detected-level/DLV1 drilldown logs volume, json …
internal/proxy/range_metric_compat.go → 21: control/bytes_over_time by app, control/instant json volume, control/instant ungrouped rate, control/json volume no filters, control/stream-label count by app, control/topk rate by app, control/ungrouped rate, json-filter-pushdown/A explore volume, 2 filters, drop …
internal/proxy/backend.go → 11: metadata-inventory/drilldown detected fields, metadata-inventory/drilldown detected labels, metadata-inventory/drilldown landing volume, metadata-inventory/explore label values env, metadata-inventory/explore label values service_name, metadata-inventory/explore labels (label browser), metadata-inventory/explore labels with selector, metadata-inventory/refresh label values env …
control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app
shape added or edited → 6: json-detected-level/DLV1 drilldown logs volume, json, json-detected-level/DLV2 json error filter, detected_level, json-detected-level/DLV3 plain json, detected_level, json-detected-level/DLV4 explore json volume, level and detected_level, json-detected-level/DLV5 json error dropped, detected_level, json-detected-level/DLV6 drilldown detected_level breakdown, json

_{base 054d17067e → PR dd8998ed21 · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 15.4 min. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.}

github-actions · 2026-10-13T18:52:03Z

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	0	10321	10321
Coverage	0%	87.9%	n/a

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal	Base	PR	Delta
QueryRange cache-hit CPU cost	3432.0 ns/op	3375.0 ns/op	-1.7% (stable)
QueryRange cache-hit memory	1240.0 B/op	1240.0 B/op	0.0% (stable)
QueryRange cache-hit allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
QueryRange cache-bypass CPU cost	4221.0 ns/op	4035.0 ns/op	-4.4% (stable)
QueryRange cache-bypass memory	1304.0 B/op	1303.0 B/op	-0.1% (stable)
QueryRange cache-bypass allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
Labels cache-hit CPU cost	925.9 ns/op	903.7 ns/op	-2.4% (stable)
Labels cache-hit memory	48.0 B/op	48.0 B/op	0.0% (stable)
Labels cache-hit allocations	3.0 allocs/op	3.0 allocs/op	0.0% (stable)
Labels cache-bypass CPU cost	1497.0 ns/op	1552.0 ns/op	+3.7% (stable)
Labels cache-bypass memory	399.0 B/op	399.0 B/op	0.0% (stable)
Labels cache-bypass allocations	14.0 allocs/op	14.0 allocs/op	0.0% (stable)

State

Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
This is a delta report, not a release gate by itself. Required checks still decide merge safety.
Performance is a smoke comparison, not a full benchmark lab run.
Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

github-actions · 2026-10-13T18:52:43Z

✅ Visual smoke: passed

Base 054d17067e vs PR acef15dd9e, Grafana through the proxy builds and Loki on one fresh stack; run.
Core set: 5 capture(s) at 1h. Detailed set: 29 entries (explore-logs-plain, explore-metric-plain, explore-A, explore-B, explore-C, explore-E, explore-F, explore-logs-json, ...). 37 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.
Trimmed to the 1h range for the capture budget: explore-A, explore-B, explore-C, explore-E, explore-F, explore-logs-json, explore-logfmt-metric, explore-logs-extracted, explore-extracted-metric, explore-extracted-filter-metric, explore-extracted-group-metric, explore-logs-plain-level, explore-plain-level-metric, explore-logs-line-format, explore-logs-line-filter-or, explore-metric-line-filter-or, explore-unwrap-sum, explore-unwrap-unit-string, explore-unwrap-missing-label, explore-unwrap-conversion-error, explore-unwrap-rate-sum, explore-empty-pattern-filter, explore-pattern-filter-anchored, dd-field-service_version, dd-field-pipeline.

Fix proof

Captures of the exact cases this pull request adds or fixes (registry case, base | PR | Loki). The verdict compares the PR with Loki where the base did not match. The gate fails a case unless one capture is fixed or improved, or, beyond the Loki data window, the base failed where Loki and the PR both answered with data (the label does not excuse it; only visual: none on the case does).

semantics/json-metric-detected-level-derivation: no visual proof, exempt: the generator data carries one lowercase spelling, the same in the stream label and the body, so main and the branch render the same panel; the e2e probes against Loki are the proof (v1.47 and v1.52)

Regression check

page	range	set	base = PR (data)	vs Loki (PR)	pixel diff	result
dd-fields	1h	core	✅ identical	identical (4 explained, 1 history-dependent)	0.11%	✅
dd-landing	1h	core	✅ identical	identical (3 explained)	0.13%	✅
dd-service-logs	1h	core	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
explore-logs-plain	1h	core	✅ identical	identical (1 explained)	0.07%	✅
explore-metric-plain	1h	core	✅ identical	identical (1 explained)	0.04%	✅
dd-field-pipeline	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-field-service_version	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-service-logs	15m	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-service-logs	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.12%	✅
explore-A	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-B	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-C	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-E	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-F	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-empty-pattern-filter	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-extracted-filter-metric	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-extracted-group-metric	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-extracted-metric	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-logfmt-metric	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-logs-extracted	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-logs-json	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-logs-line-filter-or	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-logs-line-format	1h	detailed	✅ identical	identical (2 explained)	0.07%	✅
explore-logs-plain	15m	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-logs-plain	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-logs-plain-level	1h	detailed	✅ identical	identical (2 explained)	0.07%	✅
explore-metric-line-filter-or	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-metric-plain	15m	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-metric-plain	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-pattern-filter-anchored	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-plain-level-metric	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-unwrap-conversion-error	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-unwrap-missing-label	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-unwrap-rate-sum	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-unwrap-sum	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.04%	✅
explore-unwrap-unit-string	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
tail-json-filter	live	detailed	✅ identical	entries in the window: PR 1905, Loki 646, in both 646	6.72%	✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

Core set montages (5): base | PR | Loki

dd-fields-1h

dd-landing-1h

dd-service-logs-1h

explore-logs-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-760.

…e way Loki's ingest does The row evaluator took the stream's level spelling as stored and never read a level key that only the JSON body holds. It now normalizes the stream's level and adds level to its parser hints for a detected_level grouping, as the stats pushdown already did. Only the level stream label and a body level key are read; Loki's other level fields and its ingest keyword scan are not.

…SON pipeline Adds the json-detected-level shape set (a Drilldown logs-volume shape, an __error__ filter shape and a plain | json shape) and the saved A/B results against origin/main on VictoriaLogs v1.52.0: the set at 1h with Loki and at 6h and 24h, plus the control set at 1h. Regenerates the performance evidence and gaps reports.

…ion on every route The row evaluator now gives each row the detected_level the log lines carry (deriveRowLevel) before any stage runs: a level key that is not a known string spelling falls to Loki's keyword scan instead of becoming its own series, the first level key in document order wins, a detected_level stream label keeps its stored value and a body detected_level key is renamed detected_level_extracted. The stats pushdown computes the level in VictoriaLogs with the pipes of the native metric route, placed before the unpack of any other label, and probes the window for the lines those pipes read differently from Loki (a detected_level stream label, two level keys, a level key unpack_json yields no value for); one such line sends the query to the raw-row evaluator.

…tion

…d stops tripping on text A filter on detected_level over a JSON pipeline (`| json | detected_level="info"`, Logs Drilldown's detected_level breakdown) now compares the derived level in the raw-row evaluator and in the stats pushdown, which applies it after the pipes that compute the level; `!= ""` always holds. The guard probe for two level keys reads quoted keys in a line that starts with a brace and logfmt keys otherwise, so a message mentioning `level=debug` or `level: high` no longer sends the query to the raw-row evaluator. The plain and logfmt volume tests assert the level of a body, as Loki's ingest reads it.

…detected_level

…xtures, and a known difference is recorded The e2e fixture named level a VictoriaLogs stream field for every app, so a metadata level was stored as a stream label. Each app now names only the stream labels its lines carry. A stream severity with metadata level is read in the pipes' list order by the stats pushdown and stream first by Loki and the raw-row evaluator; the probe asserts the raw-row answer and the case is registered as an open difference of the native pipes.

… JSON line is never read as logfmt VictoriaLogs writes severity_text (v1.50+, with severity_number) or severity (older, no number) for an OTLP record without a severity: Unspecified for 0, Info2 for 10. Loki stores neither, so it reads the number, or the line when the record has none. The pipes took the derived text as the level (Unspecified, Info2), a series Loki never has. They now skip a text derived from the number, and a scratch field marks the rows a stored level decides once, so the unpack pipes test it instead of repeating the stored-field filter (the chain is about 2 KB shorter, under VictoriaLogs' 16 KiB query limit with a service_name selector). A line Loki reads as JSON (a brace first and last) never goes to its logfmt parser, so the pipes no longer unpack it as logfmt: a JSON message holding level=debug as text stays error. The guard probe now also hands the raw-row evaluator a line that only starts with a brace (logfmt for Loki) and a level value with an escape (Loki reads raw bytes); the probe for escapes in keys is gone, Loki decodes them like VictoriaLogs. Probes: OTLP records of both VictoriaLogs storage shapes in the unit and e2e tables, the JSON-line, brace-logfmt, two-spelling logfmt and escape lines, and the two logfmt lines no guard sees (asserted on the raw-row evaluator).

… and what the chain costs The registry rows and the changelog entry describe OTLP severity, the JSON-line rule and the escape guard, record the logfmt and unpacked-row differences no guard sees, count the probe lines, and state the guard's cost as work (a limit 1 bounds the output, not the scan) and the chain's cost as measured.

…s at 8 and at 2 CPUs Replaces the results of the previous revision. The 1h and control runs use the branch's code with all 8 CPUs; Explore's JSON volume and the volume behind a detected_level filter, which the CI perf smoke flagged, are also measured with VictoriaLogs capped at 2 CPUs, where they take 144 ms (main 59 ms) and 794 ms (main 176 ms). The 6h and 24h run seeds VictoriaLogs only.

…nd the fake counts with its own copies of the probe's tables

…efore the stats slot, and the probe tests use the cache clock

…the request is sent

… the witness are described as they behave

…nst real Loki

…nt orders, on data with and without a stored level, and the cost they show

…hat hold a keyword The regexp that finds Loki's keyword ran over every row left without a level after the key reads. It now runs behind a filter on the nine words as whole case-insensitive words of VictoriaLogs' tokenizer (i()), which the block's token index answers. The filter is a superset of the regexp's matches, never a second decision: every byte Loki accepts around a keyword is a token separator and the folding of (?i) maps no non-ASCII rune onto the letters of the words, so the regexp still decides which rows are matched and what they read. On one hour of rows with no stored level (VictoriaLogs at 2 CPUs) the scan took about 90 ms of the 295 ms of the chain.

…string filter The check for an escape sequence (a backslash followed by u) in the lines no stored level decides ran as a regular expression over every such row, in the probe and in the witness of every pruned request. A substring filter selects the same lines (checked against VictoriaLogs v1.47.0 and v1.52.0) without the regexp engine.

…m and for detected_level The plan unpacked its labels (level, ...) with its own unpack_json and the detected_level pipes unpacked the level keys with another, so every line of a window without a stored level was parsed twice. The pipes now unpack the plan's labels with their level keys in one unpack_json (result_prefix "__j_") and, before the closing delete, copy each label from its prefixed field where the field is not stored, which is what the plan's own unpack with keep_original_fields wrote. The level keys are still read only from rows without a stored level, whatever rows the unpack parsed, and the presence mask never drops a label of the plan. The unpack parses a row only where a label is missing as a stored field or the pipes read the line; the plan's own parsed every row. The chain that holds the labels is found, narrowed and witnessed like any other.

…up of the e2e suite

…try and the registry row

…h measurement orders, on data with and without a stored level, and VictoriaLogs alone at 1h

… rows it proves, and the cold probe that is new

…n and this branch started fresh before each request

…per probes (#764) * perf: the two-keys patterns of the level keys probe run behind a gate of phrases The keys probe ran two regexps that scan the rest of the line after a level key over every line of a window that holds one, which is every line of a body-only window: 320 CPU-ms of 1h of rows, the heaviest probe of the first view of a window. Only a line holding two level keys can match them, so they now run behind a filter that passes the lines in which one key name occurs twice (a sequence of the phrase after itself) or two of the names occur, answered from VictoriaLogs' token index: 50 CPU-ms. The gate is a superset of the patterns, never a second decision; the probe's verdict is the patterns'. * fix: a selector too long for the gated keys probe gets the probe without the gate The gate adds 2.5 KB to the keys probe, so a selector of 11.8 KB to 14.3 KB, which fit VictoriaLogs' 16 KiB query length before, would have had its probe refused. The gate only narrows what the patterns run on, so the text without it gives the same verdict. * perf: the probes of the labels run once for the stored-level query and the chain A plan that groups by a label and reads detected_level asks twice on rows with no stored level, and the verdict of its label probes (main's partial-parse probe among them) was remembered under a key that told the two asks apart, so the same query ran twice in a first view. The label probes and the level probes are remembered apart now: the chain takes the verdict of the label probes from the stored-level query and asks only for the level probes. The verdict is risky if either is. * test: the e2e of the keys gate writes to VictoriaLogs only Its probes read nothing else, and the shared helper pushes the lines to Loki and flushes it, which other tests of the suite do not expect of a test that runs before them. * test: A/B results of the first-view cuts against main and against main before #760, in both measurement orders * docs: changelog entry and registry row for the first-view cuts * test: visual proof pages for the detected_level volumes, and a 5m range * perf: the gate leaves out the pair of a dotted key name and its last word The one occurrence of log.level holds the word level, so the pair of level and log.level admitted every line with that single key and the two-keys regexps ran over it. A line with both keys holds the word twice and passes the sequence of level after level; the gate stays a superset of the patterns. * fix: the keys probe is asked again without its gate when VictoriaLogs refuses the longer text An operator may have lowered -search.maxQueryLen below the gated probe, which is 2.5 KB longer than the probe was. VictoriaLogs' 400 for a query that is too long is told apart from other refusals, and the probe goes out once more without the gate, the same verdict. The edge of the 16 KiB default is pinned with bases of exactly the longest that fits and one byte more. * test: the remembered risk verdicts are scoped to the base and to the filters A window found clean for one base, or for one set of filters that reject unparsed lines, must not clear another base or set in the same tenant and window that holds a risky line: it goes to the raw-row evaluator with the answer it gets asked alone. Both scopes were unpinned: a key without the base, or without the filters, passed every test. * test: a script that measures the first view of a window, the builds started fresh before each request * test: the first-view script records its command without the output directory * test: the first view of a window, every sample: main before #760, main and this branch started fresh before each request * docs: the changelog entry and the registry row cite the first-view samples and say what the probes do on a refusal --------- Co-authored-by: szibis <szibis@users.noreply.github.com>

szibis added a commit that referenced this pull request Oct 13, 2026

docs: reference #760

10b0941

github-actions Bot added size/XL Extra large change scope/proxy Proxy core scope/docs Documentation scope/tests Tests bugfix Bug fix and removed size/XL Extra large change labels Oct 13, 2026

github-actions Bot added size/XL Extra large change scope/translator LogQL translator and removed size/XL Extra large change labels Oct 13, 2026

szibis added 12 commits October 14, 2026 23:36

docs: reference #760

466c549

test: A/B results and registry evidence for the detected_level deriva…

65da4d3

…tion

docs: count the probe lines of the detected_level derivation row

3800bde

test: A/B results for the detected_level derivation with a filter on …

008b865

…detected_level

szibis added 19 commits October 14, 2026 23:36

test: every position of the level presence mask is exercised alone, a…

0c21bba

…nd the fake counts with its own copies of the probe's tables

fix: the byte budget of the cache counts keys, the level probe runs b…

be2b6d2

…efore the stats slot, and the probe tests use the cache clock

test: a stack option that seeds rows with no stored level

397d0c5

fix: a level probe that found nothing to narrow is not repeated when …

1053ce2

…the request is sent

docs: the stored-level priority, the probe cost, the series limit and…

3325475

… the witness are described as they behave

test: an e2e case for the series limit of the stored-level plan, agai…

0a01b53

…nst real Loki

test: a fuzz target for the stored-level filter pipes

72b3818

refactor: the merge of stats groups into series is a method of the plan

5ef5d76

test: A/B results for the detected_level derivation in both measureme…

4dfdc14

…nt orders, on data with and without a stored level, and the cost they show

docs: the cache byte budget counts keys

7a0d5d8

test: the detected_level pipe variants tests run in the semantics gro…

1130f92

…up of the e2e suite

test: the keyword gate tests read the words from the text of the gate

bd92814

docs: the detected_level pipes' cost after the cuts, the changelog en…

054377f

…try and the registry row

test: A/B results for the detected_level pipes after the cuts, in bot…

8b2686f

…h measurement orders, on data with and without a stored level, and VictoriaLogs alone at 1h

docs: each new test of the detected_level pipes declares the registry…

ba0cb2d

… rows it proves, and the cold probe that is new

docs(changelog): keep this change's entry under Unreleased

acef15d

szibis force-pushed the fix/json-detected-level branch from 36e6dc8 to acef15d Compare October 14, 2026 21:47

github-actions Bot added size/XL Extra large change scope/cache Cache layer and removed size/XL Extra large change labels Oct 14, 2026

szibis marked this pull request as ready for review October 16, 2026 17:01

szibis merged commit 636e0f9 into main Oct 16, 2026
64 checks passed

szibis deleted the fix/json-detected-level branch October 16, 2026 17:02

This was referenced Oct 16, 2026

perf: | json | __error__="" metrics are counted by VictoriaLogs stats, reading only the doubtful lines #758

Open

perf: the first view of a detected_level volume over | json runs cheaper probes #764

Merged

szibis added a commit that referenced this pull request Oct 17, 2026

test: the first view of a window, every sample: main before #760, mai…

28f86e6

…n and this branch started fresh before each request

Navigation Menu

Repository navigation

fix: sum by (detected_level) over a JSON pipeline groups the level the way Loki's ingest does - #760

fix: sum by (detected_level) over a JSON pipeline groups the level the way Loki's ingest does#760
szibis merged 44 commits into
mainfrom
fix/json-detected-level

szibis commented Oct 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

szibis commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Loki source

Code

Tests

Registry

Visual proof

e2e against real Loki v3.7.7

A/B against origin/main (VictoriaLogs v1.52.0, capped at 2 CPUs)

Data with a level stream label on every row

Cold windows with a level stream label

Data with no stored level (the level only in the line)

Where the time goes (VictoriaLogs alone, 1h, body-only data)

Quality gate: High-concurrency throughput

Data-to-data match against Loki

Not done

Uh oh!

github-actions Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Performance A/B: faster, no regression

Uh oh!

github-actions Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Quality Report

Coverage and tests

Compatibility

Performance smoke

State

Uh oh!

github-actions Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Visual smoke: passed

Fix proof

Regression check

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

szibis commented Oct 13, 2026 •

edited

Loading

github-actions Bot commented Oct 13, 2026 •

edited

Loading

github-actions Bot commented Oct 13, 2026 •

edited

Loading

github-actions Bot commented Oct 13, 2026 •

edited

Loading