fix: an aggregation over a binary expression aggregates the whole binary, like Loki

CHANGELOG.md

@@ -7,6 +7,95 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **`label_replace` adds its destination label to instant answers, reads an
+  escaped regexp, and runs under `topk`, `sort` and a binary expression (#744).**
+  `label_replace(rate({app="api-gateway"}[5m]), "app_short", "$1", "app",
+  "(.*)-.*")` came back from `/loki/api/v1/query` without `app_short` on every
+  series (also through nested `label_replace`), where Loki adds it
+  (`pkg/logql/evaluator.go` `LabelReplaceEvaluator.Next`): the proxy applied the
+  transform to range matrices only. It now rewrites the labels of a vector and
+  of a matrix alike, keeps the rest of the answer, and expands the replacement
+  the way Loki does (`$1`, `${1}`, `$name`, `$$`; an empty expansion removes the
+  destination), with the anchored regexp. Two series that end with one label
+  set stay two samples on an instant query and become one series on a range
+  query, as `JoinSampleVector` does, and no error is raised, as in Loki. Also
+  fixed: the string arguments are unquoted like the LogQL lexer does, so a
+  regexp written `"(\\w+)\\.(\\w+)"` matches (the backslashes used to be kept,
+  and the regexp never matched) and an escaped quote no longer ends the
+  argument; `topk`, `bottomk`, `sort` and `stddev` over a `label_replace` sent
+  the translator's marker to VictoriaLogs and now rank what the transform
+  produced; and a `label_replace` around a binary expression on `query_range`
+  sent the binary marker to VictoriaLogs, as the instant path used to. An
+  invalid regexp under those wrappers is Loki's `parse error : invalid regex in
+  label_replace` 400. `label_replace(sum without (level) (...), ...)` kept the
+  excluded label and lost the transform, because the `without` marker swallowed
+  the `label_replace` marker after it; and a call without the four string
+  arguments of Loki's grammar, or with an invalid escape in one, was taken for a
+  log query (an empty `streams` answer on `query_range`) and is now a 400 parse
+  error on both endpoints.
+- **An instant query is answered with its failure status and a resultType, as
+  Loki does, not with a 200 that has neither (#735).** `stdvar without (app)
+  (...)` and `label_replace(avg by (app) (a + b / c) * 2, ...)` came back as
+  HTTP 200 with an error body and no `resultType`, which Grafana and every client
+  decoding by type (`pkg/loghttp/query.go` `QueryResponse.UnmarshalJSON` fails on
+  a missing or unknown type) cannot read, where Loki answers `vector`
+  (`pkg/logql/engine.go` `evalSample`, `JoinSampleVector`; `pkg/querier/queryrange/marshal.go`
+  `ResultToResponse`). Two causes. A query the proxy post-processes (`without`,
+  `label_replace`, `label_join`) was buffered, and a failure of it (a
+  VictoriaLogs error, a series limit, a rejected query) was written with the
+  status dropped, so every such error reached the client as a 200; query_range
+  already sent the failing status, the instant path now does too. And a
+  `label_replace` or `label_join` around a binary expression sent the
+  translator's internal binary marker to VictoriaLogs as a LogsQL query; the
+  expression under the transform now routes like any binary expression, its
+  operands run as instant queries and combine, and the transform applies to the
+  result. Also on the instant endpoint: a scalar literal (`1`, `-1`) answers a
+  `scalar` as Loki's `evalLiteral` does, where it was rejected as a log query,
+  also under a multi-tenant `X-Scope-OrgID`; `vector(1)`, `vector(1) + 2` and
+  `vector(1) + vector(2)` are evaluated as constants (a literal division by zero
+  is NaN, as in Loki's `MergeBinOp`) and stamp their sample in seconds like
+  every other sample (`pkg/util/marshal/query.go` `NewVector`) where they
+  carried nanoseconds; the first argument of a `label_replace` or `label_join`
+  is split outside strings and brackets, so a `)` or `,` in a line filter or a
+  matcher no longer breaks the translation; and
+  a log query sent with every tenant filtered out by `__tenant_id__` is rejected
+  with Loki's 400 for a log query as an instant query, not answered with an
+  empty `streams`. An aggregation over a binary expression (`sum by (job) (a /
+  b)`) is still not evaluated as Loki does; where the proxy rejects it, it now
+  does so with an error status, no longer with a typeless 200.
+
+- **A `sum`, `avg`, `min`, `max` or `count` over a binary expression aggregates
+  the whole binary, as Loki does, instead of only its first operand.**
+  `sum by (job) (rate({app="api-gateway"}[1m]) / count_over_time({app="api-gateway"}[1m]))`
+  answered the aggregated rate alone, `sum(rate({app="x"}[1m]) / 2)` lost the
+  `/ 2`, a comparison (`max(rate(...) > 0.5)`) filtered nothing, and `on()`,
+  `ignoring()` and `group_left` under an aggregation were ignored: all with a 200
+  and wrong values on query_range. On an instant query, a literal or a nested
+  binary as the first operand (`sum(2 * rate(...))`) answered 400 "log queries are
+  not supported as an instant query type". Loki 3.7.7 joins the operands per
+  series (`pkg/logql/evaluator.go:791` `newBinOpStepEvaluator`, `:955`
+  `vectorBinop`, `:1096` `resultMetric`; a comparison keeps the left value or,
+  with `bool`, answers 0 or 1) and then folds the joined series per step by the
+  aggregation's grouping (`VectorAggEvaluator.Next`, `:435-630`). The proxy sent
+  only a top-level binary to its binary evaluator; under an aggregation the
+  translator kept the first operand's stats chain and dropped the rest. Now
+  query and query_range send an aggregation (by, without or no grouping, nested
+  aggregations included) over a binary to the binary evaluator: each operand
+  stays a native VictoriaLogs stats query through its own handler, the proxy
+  joins the aggregated operand series and folds them per step, with no raw rows
+  and work bounded by the operands' series. An operand keeps its own parser when
+  the query names both `| json` and `| logfmt`, and an offset on one side only
+  is accepted as for a bare binary. A `label_replace` or `label_join` around
+  such an aggregation (the shape the differential corpus found on /query)
+  is evaluated the same way and the transform applies to the folded answer,
+  with each operand still reading the request's own parsers; a transform inside
+  an operand (`sum(label_replace(a, ...) / b)`) is applied by that operand, not
+  again to the answer. The series the proxy folds (this, and every `without`)
+  keep every label set apart, a value holding `,` or `=` included, and come in
+  Loki's order (`labels.Compare`) (#741).
+
 ## [2.11.2] - 2026-10-12
 
 ### CI

bench/visual/spec.json

@@ -330,6 +330,19 @@
     "semantics/unwrap-rate-ungrouped-sum-per-stream"
    ]
   },
+  {
+   "id": "explore-sum-by-over-binary",
+   "kind": "explore",
+   "query": "sum by (service_name) (rate({service_name=\"api-gateway\"}[$__auto]) / 2)",
+   "covers": [
+    "semantics/aggregation-over-binary-dropped",
+    "metric-series-identity",
+    "loki_api_v1_query_range"
+   ],
+   "fixes": [
+    "semantics/aggregation-over-binary-dropped"
+   ]
+  },
   {
    "id": "explore-label-browser",
    "kind": "label-browser",

conformance/registry/cases/semantics/aggregation-over-binary-dropped.yaml

@@ -0,0 +1,78 @@
+id: semantics/aggregation-over-binary-dropped
+title: A vector aggregation over a binary expression aggregates the whole binary, like Loki (fixed)
+proves:
+  - loki_api_v1_query_range
+  - loki_api_v1_query
+consumer: explore
+why: |
+  `sum by (job) (rate({app="api-gateway"}[1m]) / count_over_time({app="api-gateway"}[1m]))`
+  and `sum(rate({app="x"}[1m]) / 2)` are evaluated by Loki 3.7.7 as the binary
+  per series, then the aggregation per step: newBinOpStepEvaluator joins the
+  operands' samples (pkg/logql/evaluator.go:791 and vectorBinop:955, matching
+  on every label or on()/ignoring() with group_left/group_right, resultMetric:1096;
+  a literal on either side through newLiteralStepEvaluator:1130; a comparison
+  keeps the left value or, with bool, answers 0/1, pkg/logql/syntax/ast.go:1944
+  MergeBinOp), and VectorAggEvaluator.Next (pkg/logql/evaluator.go:435-630)
+  folds the joined series by the grouping.
+
+  The proxy routed only a top-level binary to its binary evaluator. Under an
+  aggregation the query went to the translator, whose outer-aggregation
+  extraction (internal/translator/translator.go extractOuterAggregation and
+  applyOuterAggregation) kept the first operand's stats chain and dropped the
+  operator and the second operand: `/ count_over_time(...)`, `/ 2`, `> 0.5` and
+  `/ on (job) group_left ...` all vanished with a 200 and wrong values on
+  query_range; an instant query whose first operand was a literal (`sum(2 *
+  rate(...))`) or a nested binary was translated to a line search and answered
+  400 "log queries are not supported as an instant query type".
+
+  Fixed: the routing of query_range and query sends a sum, avg, min, max or
+  count (by, without or no grouping, nested) over a binary to the binary
+  evaluator (proxyBinaryAggregation): each operand stays a native VictoriaLogs
+  stats query through its own handler, the evaluator joins the aggregated
+  operand series, and applyWithoutGrouping folds the joined series per step
+  with the aggregation's operator. The expression is read from the request, so
+  an operand keeps its parser when the whole query names `| json` and `|
+  logfmt` (the discovered cluster below). A mixed offset (one side only) is
+  accepted as for a bare binary.
+
+  A label_replace or label_join around the aggregation, the discovered shape
+  (cluster f2066991 on /query), routes by the expression inside it on both
+  endpoints and the transform applies to the folded answer. The request is
+  cloned to the expression inside the transform from the request's own query,
+  so an operand keeps its parser; a transform inside an operand is applied by
+  that operand, not again to the answer. The folded series keep every label set
+  apart (escaped keys: `a="b,c=d"` is not `a="b",c="d"`) and come in Loki's
+  order (labels.Compare: name then value, label by label).
+
+  A start off the step grid is pinned for a binary with a literal; a join of
+  two vectors answers on the aligned grid, registered apart
+  (semantics/binary-step-grid).
+
+  Owed after the PRs that own them land: the series limit counted on the folded
+  answer (semantics/non-sum-vector-aggregation-unreduced's fold) and NaN-skipping
+  min and max (semantics/binary-division-by-zero-nan).
+request:
+  method: GET
+  path: /loki/api/v1/query
+  params:
+    query: 'label_replace( sum by (job) ( sum_over_time( {service_name="api-gateway"} |= "level=error" | json | avg=5 and bar<25ms | unwrap duration(latency) | __error__!~".*" [5m] ) / count_over_time({service_name="api-gateway"} | logfmt | label_format foo=bar[5m]) ), "foo", "$1", "service", "(.*):.*" )'
+  headers:
+    X-Scope-OrgID: "0"
+compare:
+  mode: observed
+  fields: [resultType, series, values]
+  expect: |
+    Loki's vector: the binary per series, the aggregation per step, then the
+    label_replace; the same series and values as Loki.
+gap:
+  status: fixed
+  fixed_by: "#741"
+  impact: explore-visible
+  effort: M
+  area: 'internal/proxy binary_aggregation.go (proxyBinaryAggregation, foldBinaryAggregation); proxy.go query and query_range routing; vector_matching.go (fold keys and order)'
+  loki: 'pkg/logql/evaluator.go:791 newBinOpStepEvaluator, :955 vectorBinop, :1096 resultMetric, :400-630 newVectorAggEvaluator / VectorAggEvaluator.Next; pkg/logql/syntax/ast.go:1944 MergeBinOp'
+  proxy: 'the operands as native stats queries, the join and the fold over their aggregated series'
+  planned_test: 'TestCompat_AggregationOverBinary, TestCompat_AggregationOverBinaryMixedParsers and TestCompat_AggregationOverBinaryUnderLabelTransforms (test/e2e-compat, group core); TestAggregationOverBinaryEdgeCases, TestAggregationOverBinaryRange, TestAggregationOverBinaryDifferential, TestAggregationOverBinaryKeepsEachOperandsParser, TestAggregationOverBinaryMixedOffset, TestAggregationOverBinaryUniformOffset, TestAggregationOverBinaryUnderLabelTransforms, TestBinaryWithLabelReplaceOperandStaysBinary, TestWithoutFoldLabelOrderAndKeys and TestLabelTransformsKeepVectorAndWarnings (internal/proxy)'
+  note: 'The e2e is written and not yet run against the stack.'
+  clusters:
+    - f2066991

conformance/registry/cases/semantics/binary-division-by-zero-nan.yaml

@@ -0,0 +1,40 @@
+id: semantics/binary-division-by-zero-nan
+title: A binary division or modulo by zero answers ±Inf where Loki answers NaN (open)
+proves:
+  - loki_api_v1_query_range
+  - loki_api_v1_query
+consumer: explore
+why: |
+  Loki 3.7.7 guards both operators: `/` and `%` with a zero right-hand value
+  answer NaN (pkg/logql/syntax/ast.go:1978-2006 MergeBinOp, "guard against
+  divide by zero"), for a vector against a vector and against a literal
+  alike, and a NaN sample is encoded as "NaN". The proxy's binary evaluator
+  computes `a / b` and `math.Mod(a, b)` as Go floats (internal/proxy
+  binary_matching.go binarySampleValue), so `x / 0` answers "+Inf" or "-Inf"
+  (`0 / 0` and `x % 0` are NaN in both). A dashboard dividing by a count that
+  can be zero, or by a sum of values that can be zero, draws a different value.
+  Under an aggregation the difference carries on: Loki's min and max skip a NaN
+  after the first value of a group (pkg/logql/evaluator.go VectorAggEvaluator.Next,
+  `group.value < s.F || math.IsNaN(group.value)`), while the proxy's fold
+  propagates it (vector_matching.go withoutMerge.combine, math.Min/math.Max).
+request:
+  method: GET
+  path: /loki/api/v1/query
+  params:
+    query: 'sum by (app) (count_over_time({app="api-gateway"}[1m])) / 0'
+  headers:
+    X-Scope-OrgID: "0"
+compare:
+  mode: observed
+  fields: [values]
+  expect: |
+    "NaN" for every series, as Loki answers.
+gap:
+  status: open
+  impact: explore-visible
+  effort: S
+  area: 'internal/proxy binary_matching.go binarySampleValue (/ and %); vector_matching.go withoutMerge.combine (min, max over NaN)'
+  loki: 'pkg/logql/syntax/ast.go:1978-2006 MergeBinOp; pkg/logql/evaluator.go VectorAggEvaluator.Next (min, max)'
+  proxy: '±Inf for x/0; min and max propagate NaN'
+  planned_test: 'TestCompat_BinaryDivisionByZero (test/e2e-compat, group core); a binarySampleValue edge table and the aggregation-over-binary differential with zero divisors (internal/proxy)'
+  clusters: []

conformance/registry/cases/semantics/binary-step-grid.yaml

@@ -0,0 +1,49 @@
+id: semantics/binary-step-grid
+title: A binary joining two vectors answers on the step-aligned grid, a binary with a literal at start+k*step (open)
+proves:
+  - window-bounds-and-step-alignment
+  - loki_api_v1_query_range
+consumer: explore
+why: |
+  Loki 3.7.7 evaluates a range query at start+k*step (pkg/logql/engine.go,
+  the step evaluator from params.Start()); its query frontend moves start and
+  end down to step multiples first only with `align_queries_with_step: true`
+  (pkg/querier/queryrange/queryrangebase/step_align.go StepAlignMiddleware),
+  which the compose stack's Loki sets (test/e2e-compat/loki-local-config.yaml)
+  and a default Loki does not.
+
+  The proxy evaluates a metric at start+k*step, as a default Loki does, except
+  where a binary joins two vectors: alignBinaryRangeRequest
+  (internal/proxy/binary_logql.go) moves that request to the aligned grid so a
+  sliding-window operand (start+k*step) and a tumbling one (VictoriaLogs'
+  epoch buckets) share an axis (TestBinaryMixedWindowsShareStepAlignedAxis).
+  A binary with a literal is not moved. So with a start off the step grid,
+  `sum(rate(a[1m]) / count_over_time(a[1m]))` answers at aligned times, which
+  is the stack Loki's answer but not a default Loki's, and `sum(rate(a[1m]) /
+  2)` at start+k*step, which is a default Loki's but not the stack's. Grafana
+  sends step-aligned starts for most ranges, which hides it.
+request:
+  method: GET
+  path: /loki/api/v1/query_range
+  params:
+    query: 'sum(rate({app="api-gateway"}[1m]) / count_over_time({app="api-gateway"}[1m]))'
+    start: '1800000017'
+    end: '1800000617'
+    step: '60'
+  headers:
+    X-Scope-OrgID: "0"
+compare:
+  mode: observed
+  fields: [values]
+  expect: |
+    Samples at start+k*step on a default Loki; at step multiples with
+    align_queries_with_step. One rule for every binary, whichever is chosen.
+gap:
+  status: open
+  impact: edge
+  effort: M
+  area: 'internal/proxy binary_logql.go alignBinaryRangeRequest (vector joins only)'
+  loki: 'pkg/logql/engine.go step evaluation at start+k*step; pkg/querier/queryrange/queryrangebase/step_align.go StepAlignMiddleware (align_queries_with_step)'
+  proxy: 'vector joins aligned, literal binaries and single metrics at start+k*step'
+  planned_test: 'TestAggregationOverBinaryDifferential with a start off the step grid for vector joins (internal/proxy; skipped there today)'
+  clusters: []