Skip to content

fix: binary vector matching raises Loki's error text at Loki's step #742

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
szibis wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

fix: binary vector matching raises Loki's error text at Loki's step #742

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
405c463
fix: a vector join fails with Loki's duplicate-series text at the ste…
szibis Oct 12, 2026
72af131
chore(conformance): regenerate the parity reports for binary-ignoring…
szibis Oct 12, 2026
3c9ac5c
docs(changelog): name #742
szibis Oct 12, 2026
6c2ff79
test: skip the outer-sum ignoring e2e until the aggregation over a bi…
szibis Oct 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed

- **A vector-to-vector join fails with Loki's exact error text, at the step where
Loki fails.** When the right operand of `A + ignoring(machine) B` held several
series for one match key the proxy answered `found duplicate series on the
right hand-side; many-to-many matching not allowed: ...`, with a space after
the semicolon; Loki v3.7.7 joins the two halves with `;` alone
(pkg/logql/evaluator.go vectorBinop, :974-975), so a client matching the
message saw a different string (the `multiple matches for labels` texts already
matched). Loki joins one evaluation at a time and stops at the first whose
join fails (pkg/logql/engine.go reads the evaluator's error after every step);
the proxy checked the right-hand duplicates of every step before the left-hand
ones and before the group-label collisions, so a range query that fails at
several steps could name another error than Loki. The join now raises the error
of the earliest failing step, right-hand duplicates before left-hand multiple
matches inside a step. Pinned by a differential test over 6,000 generated
operand pairs and modifier sets (none, `on`, `ignoring`, `group_left`,
`group_right`, with and without `bool`) against a reference written from
vectorBinop, matchingSignature, resultMetric and MergeBinOp, an edge table for
`ignoring` over a label absent on both sides, on one side and on the right
only, and handler tests on range and instant queries; the live comparison with
Loki is `TestCompat_BinaryIgnoringAbsentLabel` (#742). Not changed: the query
from the conformance row `semantics/binary-ignoring-matching`, `sum by
(cluster) (sum by (cluster) (rate(..)) + ignoring(machine) sum by (cluster,
machine) (rate(..)))`, still answers the first operand's raw count (about 150
times Loki's value) because an outer aggregation over a binary expression is
translated from its first operand; the join itself matches Loki, and the row
stays open for that path.

## [2.11.2] - 2026-10-12

### CI
Expand Down
28 changes: 24 additions & 4 deletions conformance/registry/cases/semantics/binary-ignoring-matching.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,26 @@ why: |
ignoring(machine) sum by (cluster, machine) (rate({namespace="prod"}[5m])))`
is one series per cluster on Loki (2) with the sum of both sides; the proxy
returns 1 series with 150x Loki's value.

Root cause (measured against the real handler with a VictoriaLogs stand-in):
the join is not the defect. `sum by (cluster) (rate(..)) + ignoring(machine)
sum by (cluster, machine) (rate(..))` on its own is evaluated by the binary
evaluator and gives Loki's series: the ignored label is dropped from the
match key and from the result, an empty value counts as absent (Loki
pkg/logql/evaluator.go matchingSignature :945, resultMetric :1096). Under the
outer `sum by (cluster) (...)` the query never reaches the evaluator: the
translator takes the first operand of the binary and sends only
`stats by (cluster) count()` to VictoriaLogs, so the answer is that operand's
raw count, with no rate division and no second operand (a count of C against
Loki's 2C/300 is the 150x). The same happens for any outer aggregation over a
binary (`sum(A / B)`), with or without a vector-matching modifier.

Fixed in the matching code, found while pinning it to Loki: the duplicate
series text had a space after the semicolon (Loki joins the two halves of the
text with ";" alone, evaluator.go:974-975); and the join checked every
step's right-hand duplicates before any left-hand ones and before the
group-label collisions, where Loki stops at the first step whose join fails
(engine.go reads the evaluator's error after every step).
request:
method: GET
path: /loki/api/v1/query_range
Expand All @@ -28,10 +48,10 @@ gap:
status: open
impact: explore-visible
effort: S
area: 'internal/proxy/vector_matching.go (ignoring with a label absent on both sides)'
loki: 'pkg/logql/evaluator.go binary vector matching (ignoring)'
proxy: 'wrong join'
planned_test: 'TestCompat_BinaryIgnoringAbsentLabel (test/e2e-compat, group semantics)'
area: 'translator and handleQueryRange/handleQuery: an outer aggregation over a binary expression (the binary evaluator itself matches Loki)'
loki: 'pkg/logql/evaluator.go:791-1128 newBinOpStepEvaluator, matchingSignature, vectorBinop, resultMetric; pkg/logql/engine.go step loop'
proxy: 'sum by (..) (A + ignoring(..) B) is translated from its first operand alone: a raw count, no join'
planned_test: 'TestCompat_BinaryIgnoringOuterSum (the row's shape, test/e2e-compat, group semantics) and TestCompat_BinaryIgnoringAbsentLabel (the join itself); TestBinaryMatchingDifferential, TestBinaryMatchingErrorTextAndStepOrder, TestBinaryIgnoringMatchingEdgeTable, TestBinaryIgnoringThroughHandler (internal/proxy)'
clusters:
- 354c7d83
- 8fef8a78
29 changes: 26 additions & 3 deletions conformance/registry/generated/wiring.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,11 +181,13 @@
"test/e2e-compat/metadata_scan_admission_test.go"
],
"loki_api_v1_query": [
"internal/proxy/binary_ignoring_matching_test.go",
"internal/proxy/parenthesized_range_test.go",
"internal/proxy/series_limit_routes_test.go",
"internal/proxy/tenant_query_limits_test.go",
"internal/proxy/tumbling_axis_parity_test.go",
"internal/proxy/unwrap_conversion_error_test.go",
"test/e2e-compat/binary_ignoring_matching_test.go",
"test/e2e-compat/extracted_suffix_collision_test.go",
"test/e2e-compat/parenthesized_range_or_filters_test.go",
"test/e2e-compat/parser_capture_identity_test.go",
Expand All @@ -196,6 +198,8 @@
],
"loki_api_v1_query_range": [
"internal/proxy/backend_heavy_test.go",
"internal/proxy/binary_ignoring_matching_test.go",
"internal/proxy/binary_matching_differential_test.go",
"internal/proxy/drilldown_breakdown_exact_test.go",
"internal/proxy/drilldown_breakdown_matrix_test.go",
"internal/proxy/extracted_suffix_collision_test.go",
Expand All @@ -210,6 +214,7 @@
"internal/proxy/unwrap_conversion_error_test.go",
"internal/translator/extracted_suffix_test.go",
"internal/translator/regexp_escaping_test.go",
"test/e2e-compat/binary_ignoring_matching_test.go",
"test/e2e-compat/drilldown_field_breakdown_test.go",
"test/e2e-compat/extracted_suffix_collision_test.go",
"test/e2e-compat/heavy_query_protection_test.go",
Expand Down Expand Up @@ -361,6 +366,11 @@
"semantics/backend-timeout-argument": [
"internal/proxy/backend_heavy_test.go"
],
"semantics/binary-ignoring-matching": [
"internal/proxy/binary_ignoring_matching_test.go",
"internal/proxy/binary_matching_differential_test.go",
"test/e2e-compat/binary_ignoring_matching_test.go"
],
"semantics/client-cancellation-propagates": [
"internal/proxy/backend_heavy_test.go"
],
Expand Down Expand Up @@ -592,6 +602,15 @@
"operator-configurable-limits",
"semantics/backend-aborted-response"
],
"internal/proxy/binary_ignoring_matching_test.go": [
"loki_api_v1_query",
"loki_api_v1_query_range",
"semantics/binary-ignoring-matching"
],
"internal/proxy/binary_matching_differential_test.go": [
"loki_api_v1_query_range",
"semantics/binary-ignoring-matching"
],
"internal/proxy/compat_cache_freshness_test.go": [
"semantics/metadata-answers-include-last-24h-like-loki"
],
Expand Down Expand Up @@ -895,6 +914,11 @@
"test/e2e-compat/bad_request_status_parity_test.go": [
"loki-compatible-profile"
],
"test/e2e-compat/binary_ignoring_matching_test.go": [
"loki_api_v1_query",
"loki_api_v1_query_range",
"semantics/binary-ignoring-matching"
],
"test/e2e-compat/compat_option_matrix_test.go": [
"loki-compatible-profile",
"profiles/detected-fields-dotted-json-keys",
Expand Down Expand Up @@ -1064,7 +1088,7 @@
"profiles/structured-metadata-keys-per-profile"
]
},
"tests_with_markers": 96,
"tests_with_markers": 99,
"unknown_ids": {},
"unwired_items": [
"api_prom_label",
Expand Down Expand Up @@ -1131,7 +1155,6 @@
"quality/labels-backfill-into-cached-bucket",
"quality/stats-volume-freshness",
"semantics/absent-over-time-semantics",
"semantics/binary-ignoring-matching",
"semantics/count-values-unsupported",
"semantics/detected-fields-extracted-suffix",
"semantics/drop-keep-plain-name-after-parser",
Expand Down Expand Up @@ -1172,5 +1195,5 @@
"versions/capabilities-floor-vs-matrix",
"versions/rejected-stats-query-wording-vl-1.47"
],
"wired_items": 106
"wired_items": 107
}
2 changes: 1 addition & 1 deletion conformance/reports/parity-gaps.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Last run: 2026-10-05T20:16:15Z on `fcaccfcc`, window [1791220800, 1791224400].
| 9 | `profiles/json-extraction-list-exposes-only-listed` — Extraction lists (`\| json msg="msg"`, `\| logfmt op`) expose level from the line (open) | open | explore-visible | 18 | 1 | S | internal/proxy stage-field exposure (extraction lists) | pkg/logql/log/parser.go:671-700 JSONExpressionParser.Process (listed ids only) | TestCompat_JSONExtractionListExposesOnlyListed (test/e2e-compat, group semantics) |
| 10 | `semantics/first-last-over-time-translation` — first_over_time and last_over_time with offset or a line filter fail in VictoriaLogs (open) | open | explore-visible | 16 | 10 | S | internal/translator first/last_over_time emission (row_min/row_max with offset and filters) | pkg/logql/first_last_over_time.go | TestCompat_FirstLastOverTimeOffsetFilter (test/e2e-compat, group semantics) |
| 11 | `semantics/tail-history-start-limit` — Native tail ignores start and limit: Grafana Live shows no history where Loki shows its last 100 lines (open) | open | explore-visible | 15 | 1 | M | internal/proxy/tail.go openNativeTailStream (history replay before the live stream) | pkg/loghttp/tail.go:70-94 ParseTailQuery (limit, start); querier tailer sends history first | TestCompat_TailHistoryStartLimit (test/e2e-compat, group tail-multitenancy) + @explore-tail Playwright |
| 12 | `semantics/binary-ignoring-matching` — `+ ignoring(label)` between two sums joins the wrong series (open) | open | explore-visible | 10 | 2 | S | internal/proxy/vector_matching.go (ignoring with a label absent on both sides) | pkg/logql/evaluator.go binary vector matching (ignoring) | TestCompat_BinaryIgnoringAbsentLabel (test/e2e-compat, group semantics) |
| 12 | `semantics/binary-ignoring-matching` — `+ ignoring(label)` between two sums joins the wrong series (open) | open | explore-visible | 10 | 2 | S | translator and handleQueryRange/handleQuery: an outer aggregation over a binary expression (the binary evaluator itself matches Loki) | pkg/logql/evaluator.go:791-1128 newBinOpStepEvaluator, matchingSignature, vectorBinop, resultMetric; pkg/logql/engine.go step loop | TestCompat_BinaryIgnoringOuterSum (the row's shape, test/e2e-compat, group semantics) and TestCompat_BinaryIgnoringAbsentLabel (the join ... |
| 13 | `semantics/parse-error-labels-on-log-lines` — A line a parser rejects carries __error__ and __error_details__ in Loki's log answer (open) | open | explore-visible | 10 | 2 | M | internal/proxy log-row assembly (stage-field exposure) + translator parse-error predicate | pkg/logql/log/parser.go:97 addErrLabel(errJSON, err) -> :777-787 SetErr/SetErrorDetails; pkg/logql/log/error.go:5 errJSON | TestCompat_ParseErrorLabelsOnLogLines (test/e2e-compat, group semantics) |
| 14 | `semantics/absent-over-time-semantics` — absent_over_time answers when data exists and stays empty when it is absent, after a parser (open) | open | explore-visible | 8 | 6 | S | internal/translator absent_over_time (parser pipelines) | pkg/logql/evaluator.go:647-664, :1320 absentLabels | TestCompat_AbsentOverTimeWithParser (test/e2e-compat, group semantics) |
| 15 | `semantics/stddev-stdvar-grouped-values` — stddev and stdvar by a label compute different values (open) | open | explore-visible | 7 | 5 | S | internal/translator stddev/stdvar outer aggregation (by-group fold) | pkg/logql/evaluator.go VectorAggEvaluator stddev/stdvar | TestCompat_StddevStdvarBy (test/e2e-compat, group semantics) |
Expand Down
Loading
Loading