Skip to content

fix: binary vector matching raises Loki's error text at Loki's step - #742

Draft
szibis wants to merge 4 commits into
mainfrom
fix/binary-ignoring-matching
Draft

szibis wants to merge 4 commits into
mainfrom
fix/binary-ignoring-matching

Conversation

@szibis

@szibis szibis commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Join fixes for semantics/binary-ignoring-matching. The row stays open: its symptom, sum by (cluster) (A + ignoring(machine) B) answering about 150× Loki, is the aggregation-over-binary bug that #741 fixes. A top-level A + ignoring(machine) B already matched Loki.

This PR makes the join itself match Loki 3.7.7 (pkg/logql/evaluator.go:945-1128):

  • Error text. The duplicate-series error is now byte-exact: found duplicate series on the right hand-side;many-to-many matching not allowed: matching labels must be unique on one side. There is no space after ;, and the message says "left" for group_right (evaluator.go:974-975). The proxy wrote ; many-to-many.
  • Error order. Matching errors are now raised at the first failing step, in step order: right-hand duplicates, then left-hand multiple matches, within that step. The proxy checked every step for duplicates before the group-label check. The work moved to firstVectorMatchFailure and vectorMatchCounts.

No VictoriaLogs query changes. The join runs on the already-aggregated operand series.

Tests

  • TestBinaryMatchingErrorTextAndStepOrder and TestBinaryIgnoringMatchingEdgeTable (13 cases: label absent, empty or one-sided; comparison filter and bool; on; group_left include).
  • TestBinaryIgnoringThroughHandler, range and instant.
  • TestBinaryMatchingDifferential: 6,000 generated cases against a reference ported from vectorBinop, matchingSignature, resultMetric and MergeBinOp.
  • e2e vs Loki:

Pending before ready

…p where Loki fails

The right-hand duplicate error joined its two halves with "; " where Loki
v3.7.7 uses ";" (evaluator.go vectorBinop). The cardinality checks ran over
every step before the group-label checks, where Loki stops at the first
failing step. Adds a differential test against a reference written from
vectorBinop, an ignoring() edge table, handler tests on range and instant
queries and the live comparison with Loki.
@github-actions github-actions Bot added size/XL Extra large change scope/proxy Proxy core scope/docs Documentation scope/tests Tests bugfix Bug fix and removed size/XL Extra large change labels Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

⚪ Performance A/B: no change beyond noise

17 shape×range — 17 ⚪ same · results vs Loki: 14 same, 0 new difference(s), 3 pre-existing

Every shape is within noise of the base build (±30% and 100 ms).

17 unchanged shape×range (within noise)
shape range base p50 PR p50 change Loki p50 PR ÷ Loki result vs Loki
⚪ drilldown service landing 1h 34 ms 34 ms ⚪ +0% 26 ms 1.31× ✅ same
⚪ instant ungrouped rate instant 9 ms 9 ms ⚪ +0% 80 ms 0.11× ✅ same
⚪ json rate by level (no filter, no drop) 1h 400 400 ⚪ +0% 400 — ✅ same
⚪ json volume no filters 1h 249 ms 240 ms ⚪ -4% 51 ms 4.71× ✅ same
⚪ logs plain 1h 73 ms 80 ms ⚪ +10% 631 ms 0.13× ✅ same
⚪ logs | json 1h 104 ms 103 ms ⚪ -1% 60 ms 1.72× ✅ same
⚪ stream-label count by app 1h 28 ms 31 ms ⚪ +11% 43 ms 0.72× ✅ same
⚪ topk rate by app 1h 46 ms 52 ms ⚪ +13% 94 ms 0.55× ➖ differs (as base)
⚪ U1 logfmt amount rate 1h 39 ms 40 ms ⚪ +3% 13 ms 3.08× ✅ same
⚪ U2 json duration_ms rate 1h 99 ms 92 ms ⚪ -7% 26 ms 3.54× ✅ same
⚪ U3 json duration_ms rate, error filter 1h 91 ms 91 ms ⚪ +0% 29 ms 3.14× ✅ same
⚪ U4 logfmt latency_ms rate, production 1h 373 ms 372 ms ⚪ -0% 125 ms 2.98× ➖ differs (as base)
⚪ U5 logfmt amount rate, instant instant 9 ms 7 ms ⚪ -22% 14 ms 0.50× ✅ same
⚪ U6 json duration_ms rate, instant instant 12 ms 17 ms ⚪ +42% 24 ms 0.71× ✅ same
⚪ U7 json duration_ms sum_over_time, instant instant 18 ms 16 ms ⚪ -11% 24 ms 0.67× ✅ same
⚪ U8 logfmt amount rate by service (grouped) 1h 42 ms 44 ms ⚪ +5% 13 ms 3.38× ✅ same
⚪ U9 logfmt amount rate_counter (folded per series) 1h 236 ms 238 ms ⚪ +1% 35 ms 6.80× ➖ differs (as base)
Cold (first-run) timings — what a user opening a panel sees
shape range base cold PR cold Loki cold PR ÷ Loki cold
drilldown service landing 1h 37 ms 34 ms 578 ms 0.06×
instant ungrouped rate instant 10 ms 8 ms 83 ms 0.10×
json rate by level (no filter, no drop) 1h 400 400 400 —
json volume no filters 1h 241 ms 222 ms 760 ms 0.29×
logs plain 1h 214 ms 141 ms 1.24 s 0.11×
logs | json 1h 173 ms 176 ms 1.76 s 0.10×
stream-label count by app 1h 31 ms 25 ms 577 ms 0.04×
topk rate by app 1h 51 ms 47 ms 693 ms 0.07×
U1 logfmt amount rate 1h 57 ms 43 ms 58 ms 0.74×
U2 json duration_ms rate 1h 92 ms 98 ms 144 ms 0.68×
U3 json duration_ms rate, error filter 1h 90 ms 95 ms 168 ms 0.57×
U4 logfmt latency_ms rate, production 1h 376 ms 371 ms 1.10 s 0.34×
U5 logfmt amount rate, instant instant 10 ms 7 ms 13 ms 0.54×
U6 json duration_ms rate, instant instant 13 ms 16 ms 24 ms 0.67×
U7 json duration_ms sum_over_time, instant instant 14 ms 19 ms 24 ms 0.79×
U8 logfmt amount rate by service (grouped) 1h 62 ms 43 ms 64 ms 0.67×
U9 logfmt amount rate_counter (folded per series) 1h 238 ms 233 ms 565 ms 0.41×
Why these shapes
  • internal/proxy/vector_matching.go → 10: control/stream-label count by app, unwrap-ungrouped-sum/U1 logfmt amount rate, unwrap-ungrouped-sum/U2 json duration_ms rate, unwrap-ungrouped-sum/U3 json duration_ms rate, error filter, unwrap-ungrouped-sum/U4 logfmt latency_ms rate, production, unwrap-ungrouped-sum/U5 logfmt amount rate, instant, unwrap-ungrouped-sum/U6 json duration_ms rate, instant, unwrap-ungrouped-sum/U7 json duration_ms sum_over_time, instant …
  • control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app

base b20a550f6f → PR d84081b246 · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 3.0 min. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Visual smoke: passed

Base b20a550f6f vs PR 6c2ff79774, Grafana through the proxy builds and Loki on one fresh stack; run.
Core set: 5 capture(s) at 1h. Detailed set: 2 entries (explore-metric-plain, explore-unwrap-rate-sum). 10 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.

page range set base = PR (data) vs Loki (PR) pixel diff result
dd-fields 1h core ✅ identical identical (4 explained, 1 history-dependent) 0.11% ✅
dd-landing 1h core ✅ identical identical (3 explained) 0.13% ✅
dd-service-logs 1h core ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
explore-logs-plain 1h core ✅ identical identical (1 explained) 0.07% ✅
explore-metric-plain 1h core ✅ identical identical (1 explained) 0.04% ✅
explore-metric-plain 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-metric-plain 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-rate-sum 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-unwrap-rate-sum 1h detailed ✅ identical identical (1 explained) 0.04% ✅
explore-unwrap-rate-sum 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

Core set montages (5): base | PR | Loki

dd-fields-1h

dd-fields-1h

dd-landing-1h

dd-landing-1h

dd-service-logs-1h

dd-service-logs-1h

explore-logs-plain-1h

explore-logs-plain-1h

explore-metric-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-742.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal Base PR Delta
Test count 8648 8678 30
Coverage 87.7% 87.7% 0.0% (stable)

Compatibility

Track Base PR Delta
Loki API 100.0% 11/11 (100.0%) 0.0% (stable)
Logs Drilldown 100.0% 17/17 (100.0%) 0.0% (stable)
VictoriaLogs 100.0% 11/11 (100.0%) 0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal Base PR Delta
QueryRange cache-hit CPU cost 3384.0 ns/op 3414.0 ns/op +0.9% (stable)
QueryRange cache-hit memory 1240.0 B/op 1240.0 B/op 0.0% (stable)
QueryRange cache-hit allocations 27.0 allocs/op 27.0 allocs/op 0.0% (stable)
QueryRange cache-bypass CPU cost 4064.0 ns/op 4052.0 ns/op -0.3% (stable)
QueryRange cache-bypass memory 1300.0 B/op 1302.0 B/op +0.2% (stable)
QueryRange cache-bypass allocations 27.0 allocs/op 27.0 allocs/op 0.0% (stable)
Labels cache-hit CPU cost 904.7 ns/op 902.5 ns/op -0.2% (stable)
Labels cache-hit memory 48.0 B/op 48.0 B/op 0.0% (stable)
Labels cache-hit allocations 3.0 allocs/op 3.0 allocs/op 0.0% (stable)
Labels cache-bypass CPU cost 1490.0 ns/op 1479.0 ns/op -0.7% (stable)
Labels cache-bypass memory 399.0 B/op 399.0 B/op 0.0% (stable)
Labels cache-bypass allocations 14.0 allocs/op 14.0 allocs/op 0.0% (stable)

State

  • Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
  • This is a delta report, not a release gate by itself. Required checks still decide merge safety.
  • Performance is a smoke comparison, not a full benchmark lab run.
  • Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Bug fix scope/docs Documentation scope/proxy Proxy core scope/tests Tests size/XL Extra large change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant