fix: sum by (detected_level) over a JSON pipeline groups the level the way Loki's ingest does

CHANGELOG.md

@@ -7,6 +7,113 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **`sum by (detected_level)` over a JSON pipeline follows the level Loki's
+  ingest gave each line (#760).** Loki sets detected_level once per line
+  (`distributor/field_detection.go`): a level stream label or structured
+  metadata, else the first string value of a level key (`level`, `LEVEL`,
+  `Level`, `log.level`, `severity`, `lvl`, ...) at depth 0 or 1 in document
+  order, counted only when it is a known spelling (`info`, `INFO`, `inf` and
+  `information` are one level), else the first level word of a keyword scan of
+  the whole line, else `unknown`. The row evaluator (plain `| json`,
+  `__error__!=""`, every pushdown fallback) grouped the stream's spelling as
+  stored and read only a body `level` key, and the stats pushdown read only
+  that key as well: pino's numeric levels (`{"level":30}` was `30`, Loki scans
+  the line and says `unknown`, or `error` for a line with an `Error` object),
+  `notice`, a padded ` info`, `severity` or `Level` keys and a line without a
+  level key that mentions `error` each made a series of their own or were
+  `unknown`. The row evaluator now derives the level with the derivation the
+  log lines use, and a body `detected_level` key is renamed
+  `detected_level_extracted` as in Loki; a `detected_level` stream label keeps
+  its stored spelling. A filter on detected_level
+  (`| json | detected_level="info"`, Logs Drilldown's detected_level
+  breakdown) compares that value too, where it compared the stream's level or
+  the body key; the breakdown, which main answered from raw rows (2.4 s at 1h,
+  502 at 6h and 24h), is served by the stats pushdown in 0.02 s to 0.30 s.
+  The stats pushdown first groups the buckets by copies of the stored
+  `detected_level` and `level` fields (taken before any unpack) and gives each
+  bucket the level Loki's ingest gives it, a known spelling in any case as its
+  level and any other word as stored: main's query, with no chain and no level
+  guard probe, for every window whose rows all carry a stored level. A filter on
+  detected_level runs in VictoriaLogs on the stored spellings; Loki's series
+  limit is applied to the answer's series, so `INFO`, `info` and `Information`
+  count as one, and a query whose VictoriaLogs groups pass twice the limit is
+  asked again with the chain, whose groups are its series, instead of failing
+  with a 400 Loki does not give. A
+  bucket without a stored level, or with a stored `detected_level` (a stream
+  label keeps its spelling), asks again with the pipes of the native metric
+  route, built from the level fields and body keys the window holds (a presence
+  probe: one more VictoriaLogs query of counts, over the whole window when it is
+  not remembered and on every request whose window reaches into the last five
+  minutes, because lines may still arrive there; a failed probe keeps the full
+  chain; the narrowed query also flags every row holding what the probe left
+  out, for a line that arrives late inside the remembered part, and is asked
+  again with the full chain), and probes the window for lines those pipes read
+  differently from Loki: two level keys, a level value with an escape (Loki
+  reads a JSON value's raw bytes), a line that starts with a brace but does not
+  end with one (logfmt for Loki), a level key VictoriaLogs does not unpack, a
+  `detected_level` stream label. One such line, and a line that only looks
+  like one (a body with both `level` and `severity`, as audit logs have), sends
+  the whole query to the raw-row evaluator: exact, but bounded by the row limit,
+  so a long range answers 502 where the pushdown answered 200. Each probe stops
+  at its first match, which bounds what it returns, not the work. On one hour
+  of the A/B harness's backfill (about 290,000 lines, a level stream label on
+  every row) with VictoriaLogs capped at 2 CPUs, warm p50, Explore's volume over
+  `| json` takes 98 ms where main takes 79 ms and a volume behind a
+  `detected_level` filter before the parser (which unpacks logfmt for every line
+  first) 279 ms where main takes 181 ms (the builds measured in the other order:
+  76 ms against 101 ms and 245 ms against 259 ms), and 6h and 24h stay within
+  8%. Rows with no stored level (the level only in the line) pay the chain's
+  own query, which reads every line; main answers 4 series there where Loki
+  answers 8. The chain parses each line once for the plan's labels and for the
+  level (one `unpack_json`, where the plan unpacked its labels itself and the
+  pipes the level keys again) and runs its keyword regexp only on lines that
+  hold one of the nine words as a whole word (VictoriaLogs' `i()` filter, a
+  superset of what the regexp matches), and the escape check of the presence
+  probe is a substring filter. On one hour of that data, VictoriaLogs alone
+  and capped at 2 CPUs, Explore's volume takes 105 ms where main takes 84 ms
+  (102 ms against 87 ms in the other order, 15 runs each) and was 282 ms
+  against 38 ms in the previous revision; at 6h and 24h it takes 0.69 s
+  against 0.54 s and 2.75 s against 2.39 s (was 1.5 s and 6.1 s). On the
+  VictoriaLogs side, per query: main's 102 ms and 225 CPU-ms, the previous
+  chain 290 ms and 550 CPU-ms, this one 107 ms and 260 CPU-ms. The first view
+  of a window (nothing remembered) still asks the stored-level query, the
+  level probes and the presence probe before the chain, about 0.5 s at 1h
+  where main takes 0.08 s.
+  With
+  `-detected-level-body-scan=false` the raw-row evaluator answers. A line that
+  is JSON for Loki (a brace first and last) is never read as logfmt by the
+  pipes either, and the severity VictoriaLogs derives for an OTLP record
+  without one (`Unspecified`, `Info2`, from v1.50 in `severity_text` with
+  `severity_number`, before in `severity`) is no level for them, as for Loki,
+  which reads the number or the line. A few lines stay different on the stats
+  pushdown and the native route, recorded in
+  `semantics/detected-level-pipes-known-differences` (open): a stream level
+  field outranked by a metadata one, a logfmt level key in another case, a
+  logfmt syntax error before the level key, and a row VictoriaLogs unpacked
+  from a JSON body.
+  The plain and `| logfmt` logs volume shapes
+  (`sum by (level, detected_level)`) take the same pipes, so a stream without a
+  level label now gets the level of its body (a `level=warn` key or a keyword)
+  as in Loki, where it was `unknown`. The native metric route for a plain
+  selector still reads level keys in the pipes' fixed order and overwrites a
+  `detected_level` stream label, with no raw-row fallback.
+  The verdicts of the probes of the JSON and logfmt pushdowns are remembered
+  per request scope (tenant, routing and forwarded identity, where the key held
+  the tenant only), and extending a remembered window no longer renews the
+  expiry of the part checked earlier, so a window kept alive by a refreshing
+  dashboard is checked again in full every five minutes (a refresh checks only
+  the part younger than five minutes, as before).
+  (`semantics/json-metric-detected-level-derivation`)
+- **The in-memory cache's byte budget (`-cache-max-bytes`) counts the key of an
+  entry as well as its value (#760).** Entries keyed by a long query (the
+  window verdicts of the pushdown probes, whose key holds the whole base, 8 to
+  14 KB) were counted as their one-byte value. An entry whose key and value
+  together pass a tenth of the budget is no longer stored, and a cache full of
+  long keys evicts sooner; the cache size metrics grow by the length of the
+  keys.
+
 ## [2.15.1] - 2026-10-14
 
 ### Fixed

bench/ab/README.md

@@ -84,6 +84,20 @@ python3 bench/ab/report.py compare bench/ab/results/2026-09-23-pr611-control.jso
 
 It lists only the shapes that changed beyond noise and prints one summary line.
 
+### Rows with no stored level
+
+The generator puts a `level` stream label on every row, so a run answers
+`detected_level` from the stored fields. To measure the routes that read the
+level from the line (JSON `level`, logfmt, plain text), seed a stack whose
+rows have no stored level and run the same sets on it:
+
+```bash
+python3 bench/ab/pr_smoke.py --base origin/main --no-level-label --out /tmp/ab-body-only
+```
+
+(`stack.py seed --no-level-label` does the same for a hand-run stack; the
+generator option is `LOG_NO_LEVEL_LABEL=1`.)
+
 ## Reading the results
 
 - Timings are warm p50 over the runs after the first; the first run is kept as

bench/ab/results/2026-10-14-detlevel-1h.json

@@ -0,0 +1,232 @@
+{
+ "label": "detlevel-1h",
+ "date": "2026-10-14",
+ "set": "json-detected-level",
+ "description": "sum by (detected_level) over a JSON pipeline (row evaluator, and the stats pushdown with the level computed in VictoriaLogs and its guard probes).",
+ "baseline": "base",
+ "candidate": "pr",
+ "reference": "loki",
+ "runs": 5,
+ "valid": true,
+ "restart_before": "0",
+ "restart_after": "0",
+ "noise": 0.3,
+ "timing": "warm",
+ "verdicts": {
+  "same": 5,
+  "faster": 1
+ },
+ "result_differs_from_reference": 0,
+ "result_differs_preexisting": 0,
+ "stale_answers": {
+  "base": 0,
+  "pr": 0,
+  "loki": 0
+ },
+ "rows": [
+  {
+   "shape": "DLV1 drilldown logs volume, json",
+   "range": "1h",
+   "p50": {
+    "base": 0.486,
+    "pr": 0.5,
+    "loki": 0.012
+   },
+   "p95": {
+    "base": 0.495,
+    "pr": 0.511,
+    "loki": 0.015
+   },
+   "cold": {
+    "base": 0.504,
+    "pr": 0.52,
+    "loki": 0.184
+   },
+   "status": {
+    "base": "200",
+    "pr": "200",
+    "loki": "200"
+   },
+   "vl_cpu_max": {
+    "base": 62.99,
+    "pr": 62.99,
+    "loki": 62.99
+   },
+   "parity_vs_reference": "same",
+   "baseline_parity_vs_reference": "same",
+   "parity_vs_baseline": "same",
+   "verdict": "same",
+   "speedup": 0.97
+  },
+  {
+   "shape": "DLV2 json error filter, detected_level",
+   "range": "1h",
+   "p50": {
+    "base": 2.079,
+    "pr": 2.228,
+    "loki": 0.025
+   },
+   "p95": {
+    "base": 2.124,
+    "pr": 2.244,
+    "loki": 0.034
+   },
+   "cold": {
+    "base": 2.059,
+    "pr": 2.312,
+    "loki": 0.372
+   },
+   "status": {
+    "base": "200",
+    "pr": "200",
+    "loki": "200"
+   },
+   "vl_cpu_max": {
+    "base": 34.19,
+    "pr": 37.49,
+    "loki": 30.93
+   },
+   "parity_vs_reference": "same",
+   "baseline_parity_vs_reference": "same",
+   "parity_vs_baseline": "same",
+   "verdict": "same",
+   "speedup": 0.93
+  },
+  {
+   "shape": "DLV3 plain json, detected_level",
+   "range": "1h",
+   "p50": {
+    "base": 0.508,
+    "pr": 0.53,
+    "loki": 0.008
+   },
+   "p95": {
+    "base": 0.549,
+    "pr": 0.549,
+    "loki": 0.009
+   },
+   "cold": {
+    "base": 0.474,
+    "pr": 0.526,
+    "loki": 0.087
+   },
+   "status": {
+    "base": "200",
+    "pr": "200",
+    "loki": "200"
+   },
+   "vl_cpu_max": {
+    "base": 30.89,
+    "pr": 23.32,
+    "loki": 23.32
+   },
+   "parity_vs_reference": "same",
+   "baseline_parity_vs_reference": "same",
+   "parity_vs_baseline": "same",
+   "verdict": "same",
+   "speedup": 0.96
+  },
+  {
+   "shape": "DLV4 explore json volume, level and detected_level",
+   "range": "1h",
+   "p50": {
+    "base": 0.036,
+    "pr": 0.048,
+    "loki": 0.005
+   },
+   "p95": {
+    "base": 0.048,
+    "pr": 0.063,
+    "loki": 0.01
+   },
+   "cold": {
+    "base": 0.027,
+    "pr": 0.024,
+    "loki": 0.106
+   },
+   "status": {
+    "base": "200",
+    "pr": "200",
+    "loki": "200"
+   },
+   "vl_cpu_max": {
+    "base": 24.12,
+    "pr": 24.12,
+    "loki": 24.12
+   },
+   "parity_vs_reference": "same",
+   "baseline_parity_vs_reference": "same",
+   "parity_vs_baseline": "same",
+   "verdict": "same",
+   "speedup": 0.75
+  },
+  {
+   "shape": "DLV5 json error dropped, detected_level",
+   "range": "1h",
+   "p50": {
+    "base": 0.055,
+    "pr": 0.008,
+    "loki": 0.021
+   },
+   "p95": {
+    "base": 0.059,
+    "pr": 0.011,
+    "loki": 0.021
+   },
+   "cold": {
+    "base": 0.07,
+    "pr": 0.007,
+    "loki": 0.424
+   },
+   "status": {
+    "base": "200",
+    "pr": "200",
+    "loki": "200"
+   },
+   "vl_cpu_max": {
+    "base": 28.32,
+    "pr": 28.32,
+    "loki": 28.32
+   },
+   "parity_vs_reference": "same",
+   "baseline_parity_vs_reference": "same",
+   "parity_vs_baseline": "same",
+   "verdict": "same",
+   "speedup": 6.88
+  },
+  {
+   "shape": "DLV6 drilldown detected_level breakdown, json",
+   "range": "1h",
+   "p50": {
+    "base": 2.29,
+    "pr": 0.009,
+    "loki": 0.017
+   },
+   "p95": {
+    "base": 2.336,
+    "pr": 0.011,
+    "loki": 0.019
+   },
+   "cold": {
+    "base": 2.265,
+    "pr": 0.009,
+    "loki": 0.047
+   },
+   "status": {
+    "base": "200",
+    "pr": "200",
+    "loki": "200"
+   },
+   "vl_cpu_max": {
+    "base": 36.32,
+    "pr": 29.07,
+    "loki": 29.07
+   },
+   "parity_vs_reference": "same",
+   "baseline_parity_vs_reference": "same",
+   "parity_vs_baseline": "same",
+   "verdict": "faster",
+   "speedup": 254.44
+  }
+ ]
+}