Repository navigation
fix: sum without over an unwrap range function keeps a series per remaining label set, like Loki - #739
Merged
Conversation
szibis
added a commit
that referenced
this pull request
Oct 12, 2026
github-actions
Bot
added
size/XL
Contributor
⚪ Performance A/B: no change beyond noise120 shape×range — 120 ⚪ same · results vs Loki: 98 same, 0 new difference(s), 22 pre-existing Every shape is within noise of the base build (±30% and 100 ms). 120 unchanged shape×range (within noise)
Cold (first-run) timings — what a user opening a panel sees
Why these shapes
base |
Contributor
PR Quality ReportCompared against base branch Coverage and tests
Compatibility
Performance smokeLower CPU cost (
State
|
Contributor
✅ Visual smoke: passed with warningsBase Fix proofCaptures of the exact cases this pull request adds or fixes (registry case, base | PR | Loki). The verdict compares the PR with Loki where the base did not match. The gate fails a case unless one capture is fixed or improved, or, beyond the Loki data window, the base failed where Loki and the PR both answered with data (the label does not excuse it; only
explore-sum-without-rate-15m: improved
explore-sum-without-rate-1h: improved
explore-sum-without-rate-6h: undecided
Regression check
Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request explore-sum-without-rate-6h: no Loki data at this range; the same differences match Loki at a shorter range (1): query ('sum without (pod) (rate({ser
Core set montages (5): base | PR | Lokidd-fields-1h
dd-landing-1h
dd-service-logs-1h
explore-logs-plain-1h
explore-metric-plain-1h
All montages of this run: |
github-actions
Bot
added
size/XL
…aining label set, like Loki
`sum without (pod) (max_over_time({app="payment-service"} | logfmt | unwrap
amount [5m]))` answered one series on query_range and at an instant, where
Loki keeps every label but pod. The translation of an outer aggregation over
an unwrap function, and of a sum over rate, ended in a stats pipe with no
grouping, so VictoriaLogs returned one row and the without step had no labels
to group by. The last stats pipe of an `<op> without (...)` now keeps the
stream (the range aggregation's own grouping when it has one), VictoriaLogs
answers a row per stream from stats, and the proxy drops the excluded labels
and folds the rows with the outer operator, as Loki's VectorAggEvaluator does.
szibis
added a commit
that referenced
this pull request
Oct 17, 2026
szibis
force-pushed
the
fix/unwrap-sum-without
branch
from
October 17, 2026 16:26
0d3b2b4 to
e019f0a
Compare
github-actions
Bot
added
size/XL
szibis
marked this pull request as ready for review
October 17, 2026 16:43
…tion, like Loki Loki counts max_query_series on the answer of the expression, after the without fold. The proxy applied the limit to the per-stream rows it reads before the fold, so a `without` query failed as soon as the streams outnumbered the limit even when the answer fit it. A `without` request now reads the per-stream rows under the raw-row budget and applies the tenant's limit to the folded answer: Loki's 400 for a plain client, the busiest series with Loki's warning for Logs Drilldown, on query_range, at an instant and under topk, sort, stddev and stdvar.
… over an unwrap function Adds the unwrap-sum-without A/B set and its saved 1h and 6h/24h/7d results next to the control set, a visual page that fixes the registered case, and brings the series limit rows up to date: the limit counts the answer of the aggregation after the without fold, as Loki does. The harness leaves out the log limit and direction parameters for a range metric shape marked no_limit, which Loki holds an answer of thousands of series for until the client gives up.
…t over an unwrap function count and avg without after a parser keep one series where Loki keeps one per remaining label set; sum and sum without over quantile_over_time answer an empty result; sum by over quantile_over_time takes the quantile of the group instead of adding the per-series quantiles.
…eps its status and Loki's error format A test pins the status, the envelope and the redaction of the backend's echo of the query for a rejected and a failing backend, on the instant queries that are rewritten after VictoriaLogs answers (without, label_replace, label_join).
The translator records the operator of the without aggregation in the marker (__without__:<op>:pod) and the consumers fold with it, so a topk, sort, label_replace or a comparison around a max, min or count without no longer adds the series up (pods valued 10 and 4 gave 14 for a max). A binary expression folds its operands on their own and is no longer folded again over the combined answer, and the ranking of an instant topk or bottomk runs after the fold, not over the per-stream rows. The reads before the fold reuse the mark of an ungrouped sum folded afterwards instead of a key of their own. When the raw-row budget trips, not the tenant limit, the client reads the row-limit error naming -manual-range-metric-row-limit, still a rejection. The ranking of the top values of a grouping and the sliding window evaluator take the tenant's limit.
…and limit behaviour Drops the stray empty 2.11.1 heading, states the folded operator, the row-limit error and the wrappers, and points to the registry row for the rest.
…ed busiest series The raw-row budget replaces the tenant limit before the fold only for a request that withWithoutFold marked (it records the budget); an ungrouped sum folded afterwards keeps the tenant limit, as before. The Drilldown case of the folded series limit now asserts that the kept series is the busiest of the folded answer, which a limit applied to the streams before the fold would not keep. A topk of a comparison over a count without is pinned at an instant (on query_range a topk over a binary expression is answered 502 with or without without, and the range handler drops the fold for a binary inner like the instant one), and the translator test pins that stddev and stdvar without keep a single row.
…ing the earlier run
…ilure note main now carries
… of a without request The evaluator that answers a grouped range query from stats buckets cut the series at the tenant's limit while reading them. A request that folds its series afterwards (a without aggregation reads a row per stream) is limited on the folded answer, as Loki counts it, so the evaluator takes the bound the other routes take: resolvedMaxStatsQuerySeries.
…t is the outermost The proxy folds a row per stream by the without clause for sum, min, max, avg and count without, the outermost aggregation of the query (a topk, bottomk, sort, stddev or stdvar may wrap it, and a label_replace on query_range: the proxy applies it to the folded series). The marker of any other query that holds a without clause (topk, bottomk and stddev without; a without under another aggregation such as max(sum without ...)) names no operator: the translation, the fold and the series limit are what they were before the fold existed, and the tests pin those answers.
…ot by the limit value A without aggregation reads a series per stream before it folds them. The error it hits then was recognised as the raw-row budget by the budget being equal to the limit that tripped, so an operator who set both limits to one value, or a check against the tenant's own limit at that value, got the other text. The request records whether the budget is the bound it reads under (it is, unless the tenant's limit is lower), the error follows that, and the tenant's own limit check clears the fold from its context. The error answers 400, a rejection of the query, where the raw-row reads answer 502: the limit reference and the operations notes say both.
…ies and an offset on a without The label_replace wrapper of the without differential now asserts the dst label it adds, the comparison wrapper uses a threshold between the groups' values so it drops some and keeps others, and a new test pins sum, max and count without over a range function with an offset against Loki's window.
…er from Loki sum without () over an over-time function on query_range, a without over an inner sum by or count_over_time, a without under max or sum by, a comparison inside the aggregation, topk and bottomk without, label_replace around a vector at an instant and around a binary expression on query_range. Each row states the answer of the proxy and Loki's for the shapes a unit test reproduces.
szibis
force-pushed
the
fix/unwrap-sum-without
branch
from
October 17, 2026 17:35
e019f0a to
ba4b37c
Compare
github-actions
Bot
added
size/XL
…er's extraction list names A Loki series is the stream plus the labels its parsers extract, less the unwrapped one. With an extraction list (| logfmt amount, | logfmt amount, route) that set is known, so the inner stats of the sum, min, max, avg or count without that the proxy folds now groups by the stream and those labels instead of by _msg, which split the series per line: sum without (pod) (max_over_time(... | logfmt amount | unwrap amount [5m])) answered a sum of per-line maxima. The grouping applies to that fold only, on its instant, tumbling and sliding routes; every other query, whatever its parser, keeps its translation and its answers. A bare | logfmt or | json, label_format, drop, keep, a logfmt alias and a key that is also a label of the stream selector (Loki names it <key>_extracted) keep the previous grouping.
…he text after a peeled wrapper of a without
…ective series limit, and is a 400 the 5xx alert does not see
… keeps Loki's answer bounded A bare | logfmt makes every payment line a series of its own on Loki (thousands of series), so its capture never settles and the A/B shapes time out. | logfmt amount makes the stream the series, which Loki answers in two series and the proxy now answers alike. The fix-proof page is the rate over [$__auto]; the max_over_time page stays as a cover.
…r data, and register two more The requests use the level label and a bounded parser, so each reproduces on the generator data, with the values measured against Loki 3.7.7. count and avg without are right after an extraction list and stay open for a bare parser. Two shapes found on the way: sum by over a per-series function and a bare range function that exposes every parsed label.
… the known difference
…comparison threshold that splits the series, and the max-over-without numbers The two quantile rows keep their earlier requests.
…s previous answers after an extraction list, and register what differs The extraction list names the labels of a Loki series, but the proxy reads them for the without fold only. The differential test sends the other shapes (sum, avg, count, max, min and topk of per-series functions, rate and count_over_time, a without under another aggregation) through the handler on an emulated VictoriaLogs, as an instant query, a tumbling window and a sliding one, and pins the LogsQL sent and the answer to the ones origin/main produced. A second test pins the proxy's answers beside a reference evaluator that follows the Loki sources, and two open rows register the series identity outside without and a parsed key that is also a stream label (Loki names it key_extracted). The figures come from the emulator, not from a Loki run.
github-actions
Bot
added
size/XL
…n the final tree The affected set (W1 to W7 with | logfmt amount, W8, W9) at 1h, 6h, 24h and 7d and the control set, base cd174a7 against the branch on one stack, with VictoriaLogs at 2 CPUs and 0 restarts. The performance evidence report is regenerated from the saved runs.
The max_over_time page could not match Loki: its values follow the over-time window difference that is still open, so it stays out until that is fixed. The rate, topk and count pages match Loki exactly.
github-actions
Bot
added
size/XL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gap
semantics/unwrap-sum-without-collapses. Loki 3.7.7 keeps every label exceptpodforsum without (pod) (max_over_time({app="payment-service"} | logfmt amount | unwrap amount [5m])). The proxy answered a single series on query_range and query.Cause. For
<op> without (...)over an unwrap range function, the translator's last stats pipe had noby. VictoriaLogs returned one row, so the proxy'swithoutstep had no labels left to group by (Loki:pkg/logql/evaluator.go:451-466). The same applied to the stdvar branch and to tumbling rate windows.Fix.
bycan't express "every label except X".max_query_seriesis applied to the folded answer, which is what Loki counts (pkg/logql/engine.go:536-548,:575-590). Before this, anywithoutquery failed once the streams outnumbered the limit, even when the answer fit.__tenant_id__and the merged answer is what gets counted.-manual-range-metric-row-limitwhen it is not below the tenant's effectivemax_query_series(-max-stats-query-seriesunless the tenant has its own override); past it the client reads the row-limit message naming the flag as a 400 (a rejection of the query), where the raw-row reads of the other routes answer 502; the 400 is a client error and does not count toward theLokiVLProxyHighErrorRate5xx alert. The limits reference, the operations notes and the API reference state both. Which error a request hits is recorded on the request, not inferred from the two limits having the same value.withoutaggregation travels in the marker (__without__:<op>:pod), so amax,min,countoravgwithout folds with its own operator and not with a sum, also when atopk,bottomk,sort,stddevorstdvarwraps it (the wrapper is applied to the folded series), and a comparison around it. Alabel_replacearound it is applied on query_range; on an instant query it adds no label, as withoutwithout(registered below). A binary expression folds each operand on its own.| logfmt amount,| logfmt amount, currency) makes a Loki series the stream plus those labels, less the unwrapped one. The inner stats of thewithoutthe proxy folds, and of that fold only, over such a pipeline now groups by exactly that (_stream, plus the named labels), where it grouped by_msg, which stands for the labels of a parser that extracts every key and split the series per line:sum without (pod) (max_over_time(... | logfmt amount | unwrap amount [5m]))answered a sum of per-line maxima (870,792.27 against Loki's 252,870.11 on generator data). A bare| logfmtor| json,label_format,drop,keep,unpack, an alias of a logfmt list (r="route", which creates no field) and the names the proxy reads specially (level,service_name) keep the previous grouping, and so does a key that is also a label of the stream selector ({app="x", pod=~".+"} | logfmt amount, pod): Loki names the parsed onepod_extracted, which the grouping would not, so that query keeps the fold's earlier grouping by the stream. A stream label the selector does not name cannot be seen by the translation (open rowsemantics/unwrap-sum-without-extracted-label-collision). The labels named by the list are carried on the rate route and on the sliding-window route of the fold too. Every other query is unchanged by this: the grouping is read only when the translator marks the query for the fold (the marker with an operator), and the sliding route adds the parsed labels to a series only for a request that folds. Sum, avg, count, max, min and topk of per-series functions,rateandcount_over_time, awithoutthat is not folded (topk,bottomk,stddevandstdvar, or under another aggregation), after an extraction list, on an instant query, a tumbling window and a sliding one, send VictoriaLogs the same LogsQL and answer the same as origin/main (TestExtractionListOutsideWithoutKeepsItsGrouping, whose golden table was produced oncd174a7dand passes there too). On generator data, with Loki 3.7.7 and VictoriaLogs v1.52.0,sum without (pod)overmax_over_timeand overratewith| logfmt amountand with| logfmt amount, currency(2 and 10 series),count withoutovermax_over_time(118 and 150) andmax withoutovermax_over_timeanswer Loki's series and values at an instant, andrateon query_range answers the same series and last values. The over-time functions on query_range still read a step-sized bucket where Loki reads the sliding window (max withoutovermax_over_time: 9,722 against Loki's 9,988 at the last step), the open rowsemantics/unwrap-over-time-range-windows.sum,min,max,avgandcountthat are the outermost aggregation of the query.topk,bottomk,stddevandstdvarwith awithoutclause, and awithoutunder another aggregation (max(sum without ...)), carry a marker without an operator: their translation, fold and series limit are what they were before this change, and the tests pin those answers (TestWithoutFoldIsLimitedToTheOperatorsItProves, and after an extraction listTestExtractionListOutsideWithoutKeepsItsGrouping:max(sum without (pod) (max_over_time(... | logfmt amount | unwrap amount [5m])))answers main's 1691 on the fixture; the build that read the grouping there answered 358, and 45 of the 69 cases of the test then differ from main).Why the proof uses
| logfmt amount. With a bare| logfmtevery payment line carries a uniquetx_id,user_id,latency_msandmsg, so on Loki each line is a series of its own: about 6,300 series over 15 minutes and 19,600 over an hour for a rate, and Loki's Grafana capture never settles and its A/B timings reach the harness timeout. The proxy does not reproduce a parsed-label identity of that kind (open rowsemantics/unwrap-per-series-identity-raw-rows), and the over-time functions on query_range read a step-sized bucket where Loki reads the sliding window (open rowsemantics/unwrap-over-time-range-windows). With| logfmt amountthe parser extracts only the unwrapped label, which Loki drops from the series identity, so the series is the stream: Loki answers two series, bounded, and the proxy answers the same two with identical timestamps and values at 15 minutes and at 1 hour. The fix-proof page, the othersum withoutpages and the A/B shapes W1 to W7 use that form.Remaining differences. Shapes around
withoutthat are still wrong on origin/main and on this branch are registered, each with the proxy's answer and Loki's on generator data (measured against Loki 3.7.7, VictoriaLogs v1.52.0; the values of the fixture quoted in a row's text come from reading the v3.7.7 evaluator, not from a Loki run) and, where one exists, a unit test that reproduces it:sum without ()over an over-time function on query_range (without-empty-grouping-range-collapses), awithoutover an innersum byorcount_over_time(without-over-inner-aggregation-collapses), awithoutundermaxorsum by(without-nested-under-aggregation), a comparison inside the aggregation (without-comparison-inside-ignored),topk/bottomk without(topk-bottomk-without-strips-label),label_replacearound a vector at an instant (label-replace-instant-vector) and around a binary expression on query_range (label-replace-range-binary-marker).Outside a
without, the labels an extraction list names are not read yet, sosum(max_over_time(... | logfmt amount | unwrap amount [5m])),avg by,count byandtopkof per-series functions read other series than Loki (open rowsemantics/unwrap-extraction-list-series-identity; on a 4-stream fixture with three routes, from an emulator run with a reference evaluator that follows the Loki sources and not against a real Loki:sum(max_over_time)1691 against 358,count by (ns)overmax_over_time2 and 2 against 6 and 6,count by (ns)overcount_over_time2 and 2 against 6 and 6,avg by (ns)overavg_over_time55.94 and 42.75 against 55.94 and 42.06;TestExtractionListSeriesIdentityStillDifferspins both). Applying the grouping to those queries multiplies the series VictoriaLogs returns and is left to its own change.After a bare parser, Loki's series also carry every extracted label, so it answers thousands of series. The proxy groups by stream; projected onto the stream labels, it matches Loki on instant queries and for
rateon query_range. Range values ofsum_over_timeandmax_over_timefollow the open rowsemantics/unwrap-over-time-range-windows, and parser series identity issemantics/unwrap-per-series-identity-raw-rows. Two more shapes found with the extraction list are registered:sum by (level)overmax_over_timetakes the maximum of the group, not the sum of the per-series maxima (proxy 9,972 and 9,998 against Loki's 733,939 and 1,389,474;semantics/unwrap-sum-by-per-series-function-extraction-list), and a range function with unwrap and no outer aggregation exposes every parsed label although the parser names one key (2,035 series against Loki's 268 at an instant;semantics/unwrap-bare-range-exposes-parsed-labels).Instant
withoutanswers also lack thedetected_level=unknownlabel that Loki, and the proxy on query_range, add to a stream with no level label (semantics/without-instant-detected-level-unknown, open); the new tests drop that label before comparing instant answers.e2e vs Loki 3.7.7
Isolated stack (own compose project), image built from head
ee10d7dd, a fresh stack per VictoriaLogs version and per run, VictoriaLogs RestartCount 0 before and after every run. Every URL of the test environment points at that stack. Patterns are those ofci.yaml; the first row is the set the change touches.TestSetup_IngestLogs,TestCompat_Unwrap.*,TestRangeMetric.*,TestGrafanaClickout_.*(34 tests, 789 passes, 3 skips that are the open cases ofTestCompat_UnwrapConversionErrorParity)cd174a7d)TestCompat_UnwrapSumWithoutKeepsSeries,...ExtractionList,...UnderAnotherAggregation) also fail, as they mustsemanticsgroup (45 tests, 897 passes)coregroup (149 tests, 1539 passes)TestProxyCompatibilitySurface/vmauth_proxy_drilldown_limits_backend_detectionv1.138.0wherev1.52.is expected)The new tests pass on all three versions:
TestCompat_UnwrapSumWithoutKeepsSeries(120 subtests: fails 94 of 120 on main),TestCompat_UnwrapSumWithoutExtractionList(every step of theraterange queries, exact values and series against Loki),TestCompat_UnwrapSumWithoutUnderAnotherAggregationandTestCompat_UnwrapSumWithoutUnderMaxStillDiffers(pinsmaxover a foldedwithoutas the known difference).On v1.40.0 the 8 failing tests (
DrilldownFieldBreakdown,JSONDetectedLevel,JSONDetectedLevelSeriesLimit,JSONFilterPushdown,JSONLogsVolume,BareParserWindows,SlidingBucketEdges,TumblingAndShortWindows) fail with the same 57 subtests on main (compared by name with the run timestamps removed): the branch adds no failure on any version.A/B (main
cd174a7dvs branchee10d7dd, VictoriaLogs v1.52.0 capped at 2 CPUs)One fresh stack per run, targets interleaved per request, 5 runs (the first is cold), a shape the first pass moves is measured again with more runs, noise 30% and 100 ms. VictoriaLogs restarts 0 to 0 on every run. Saved results:
bench/ab/results/2026-10-17-unwrapwithout-{1h,long,control-1h,control-long}.json.No degradation. The control set (20 shapes) is unchanged: 20/20 within noise at 1h with Loki, 37/37 at 6h and 24h, no new difference from Loki (
topk rate by appdiffers from Loki on main too;json rate by levelanswers 400 on main, the branch and Loki).Affected shapes. Nothing is slower beyond noise. At 1h all 9 are within noise. At 24h and 7d the branch is faster on the range
max_over_time,sum_over_timeandtopkshapes (the inner stats now group by the stream and the named keys; main grouped by_msg):sum without (pod)overmax_over_timesum without (pod)oversum_over_timesum without (pod)overratemax without (pod)overmax_over_timetopk(5, sum without ...)sum withoutovercount_over_time(production)sum withoutovermax_over_time/ratelabel_replaceinstant controlEach cell is main / branch p50. The first pass flagged one cell as slower, W3 at 24h (1.01 against 1.39 s); measured again with more runs it reads 1.02 against 0.99 s, so it was noise. At 7d the windows of consecutive runs resolve to the same step-aligned buckets and the harness marks those warm timings as possibly cache-assisted; the cold timings agree in direction. The
count_over_timeproduction shape answers 502 at 24h on main and the branch alike (the response-size bound), unchanged.Against Loki at 1h (same data, Loki answers within the timeout now that the shapes use
| logfmt amount):W3(rate),W5andW6(instant) andW9match Loki's series and sums exactly.W1,W2,W4andW7(rangemax_over_timeandsum_over_time) still differ, the same difference main has: the series set is now Loki's (two series, theerrorandinfostreams), but the values and the time axis follow the open rowsemantics/unwrap-over-time-range-windows.W8is unanswered by Loki on this stack. The answers ofW1,W2,W4,W5,W6andW7differ from main's by design (main answered one series).benchstat
Test binaries of both trees run alternately, 10 rounds,
GOMAXPROCS=4,GOMEMLIMIT=2GiB,benchstatmain against branchee10d7dd. The main tree is a detachedcd174a7dworktree withinternal/proxy/sum_without_bench_test.gocopied in, uncommitted: the benchmark is new and uses only functions that exist on main.The hot paths of every query (
handleQueryRangewith the cache hit and bypass, the stats query, the range-metric collector, the manual window aggregations) show no change in time, bytes or allocations (all~, allocations identical). The translator (internal/translator, all benchmarks) shows no significant change in time, bytes or allocations (geomean time +1.7%, bytes and allocs +0.0%; oneTranslateLogQL_MultiLabelB/op line reads -0.06% at p=0.013).BenchmarkSumWithout_Querydrives the handlers with thesum withoutshapes and controls against a stub VictoriaLogs. Both builds receive the same 500-stream stats answer, so it compares what the proxy does with an answer, not what each translation makes VictoriaLogs compute:sum withoutovermax_over_time, query_rangesum withoutoverrate, query_rangesum withouttopk(5, sum without ...)sum by, ungrouped rate,label_replaceinstant, binarysum by+6.5% at p=0.035)The
withoutshapes allocate 4% to 23% more bytes and 2% to 3% more allocations (the folded rows are kept per stream before the fold); the time differences are within the spread of this machine (5 to 15% per side), with two cells at p<0.05 (the instantsum without, and thesum bycontrol whose bytes and allocations are identical, so it is noise). Output:bench739/benchstat-final-{proxy,translator,sumwithout}.txt.Real Grafana range matrix
Head
98c963c2(the code is that ofee10d7dd; the later commits add the A/B results and remove one Grafana page) against maincd174a7d. Isolated stack (bench/visual/stack.py up --hours 168: VictoriaLogs seeded for 7 days, Loki for the last 1.5 h), Grafana 13.2.1 with the pinned plugins, three datasources: main, this branch, Loki direct. 43 pages at 5m, 15m, 1h, 6h, 24h and 7d through Grafana's own query splitting (258 captures, all captured) plus Live tail;compare.pyover 260 page × range rows. VictoriaLogs restarts 0 to 0 on every run.sum without (pod) (rate(... | logfmt amount | unwrap amount [$__auto]))andtopk(5, ...)of it: main answers the single series{}, the branch the two stream series (level=errorandlevel=infoofpayment-service), at every range. Thecount_over_timepage is identical on both builds and on Loki.index/stats, which Grafana sends with every query and which differs from main the same way, also on control pages (Loki's estimate counts 11,234 entries where its owncount_over_timeand the proxy count 21,892). At 6h, 24h and 7d Loki holds no data; the branch's difference is the one proven closer to Loki at the shorter ranges.semantics/unwrap-sum-without-collapses): 4 captures improved, no failure.dd-fields(unchanged by this change, equal on both builds at 24h): Grafana was killed by the host's memory limit twice while loading main's 7d trace_id/span_id breakdown (about 257k frames), so that one cell is from the first capture and did not settle.max_over_timepage was removed from the spec: its series and values equal Loki's, but its samples sit one step early, the open rowsemantics/unwrap-over-time-range-windows(main has the same shift onsum(max_over_time ...)); it returns with the fix for that row.Backend-safety load
Same stack and VictoriaLogs, one build after the other, each proxy restarted fresh: 8 concurrent clients for 240 s over 49 request types (W1 to W9 of
bench/ab/shapes.jsonat 24h and 7d, and the control set), each request with a shifted window,/labelsprobed every second, proxy RSS sampled every 2 s, VictoriaLogs restart count and OOM flag read before and after./labelsThe errors are the same on both builds:
sum without (pod) (count_over_time({env="production"}[1m]))at 24h answers 502 (the 64 MiB response bound; 200 at 7d), andjson rate by levelwithout a filter answers 400 with Loki's own JSONParserErr. Two earlier 300 s runs of each build agree (branch 1,254 and 1,325 requests, p99 10.3 and 10.6 s; main 1,128, p99 10.0 s); the RSS peaks of all runs overlap (branch 3.5 to 5.6 GB, main 3.3 to 4.6 GB). Most 7d requests are served from the cache (5 distinct windows), so the load on VictoriaLogs is mostly the 24h shapes.Tests
TestSumWithoutDifferential: VictoriaLogs v1.52.0/v1.40.0, no parser and logfmt, rate and the over-time functions, sum/max/min outers and instant. 72 subtests fail without the fix.TestSumWithoutSeriesLimitOnFoldedAnswer: 16 cases, including Drilldown, topk and two tenants.TestWithoutKeepsSeriesInLastStatsPipeandTestKeepWithoutSeries(translator).TestWithoutWrappedByAnOperator(thelabel_replacewrapper is pinned by the label it adds, the comparison by a threshold between the groups' values that drops some and keeps others),TestWithoutOffset(sum, max and count without over a range function with an offset),TestWithoutBinaryOperands,TestWithoutFoldRowBudgetIsNotASeriesLimit,TestWithoutMarkerCarriesTheAggregation,TestWithoutComparisonAboveEveryValueDropsEverySeries.TestSumWithoutExtractionList:| logfmt amountand| logfmt amount, route, rate and the over-time functions, sum/max/min/count/avg outers, instant and query_range, VictoriaLogs v1.52.0 and v1.40.0, against Loki's evaluator over rows with a uniqueidin every line; each of the three code changes (inner grouping, rate grouping, sliding-window labels) fails it when reverted.TestParsedSeriesLabelsandTestUnwrapInnerGroupingFollowsParsedSeries(translator) pin the parsers that name their keys (logfmt, json, regexp, pattern), the ones that fall back, the selector-label collision and that a query that is not the fold groups by_msg.TestExtractionListOutsideWithoutKeepsItsGroupingpins 23 other shapes (69 cases, with the raw-row requests too) to the LogsQL and the answers ofcd174a7d; leaking the grouping to the general paths fails it andTestUnwrapInnerGroupingFollowsParsedSeries, and reverting the fold's grouping failsTestSumWithoutExtractionList.TestSumWithoutExtractedLabelCollisionpins the two collision cases.TestTopValueRankingIsBoundByTheTenantLimitWhenFolding: a folding request whose row budget is above the tenant's limit still hits the tenant's series limit on the busiest-values ranking.TestWithoutFoldIsLimitedToTheOperatorsItProvesandTestFoldingWithoutOp(translator, now with text after a wrapper,topk(5, sum without ...) > 10andstddev(sum without ...) by (level)): the shapes the fold covers, and the answers oftopk/bottomk withoutand of awithoutunder another aggregation, pinned byte for byte to the answers before this change.TestWithoutFoldRowBudgetError: the row-budget error is told by the request, with equal and unequal limits, and the tenant's own limit check at the budget's value is Loki's message.TestSeriesLimit_SlidingWindowStatsKeepsTheStreamsOfAFoldingRequest: the sliding window evaluator leaves the tenant's limit to the fold. It calls the evaluator directly: nowithoutquery is routed there today.TestWithoutShapesStillDifferFromLoki,TestInstantLabelReplaceOnAVectorAddsNoLabelandTestRangeLabelReplaceOfABinaryExpressionSendsTheMarkerreproduce the open rows below.TestInstantAggregationFailureIsRedactedpins the status, envelope and redaction of a failing instant aggregation (without,label_replace,label_join).Capped unit run (
GOMEMLIMIT=3GiB GOMAXPROCS=4 go test -short -p 1 ./..., peak RSS 1.55 GB),-raceon the touched tests (GOMEMLIMIT=2GiB),go vet(also-tags e2e),golangci-lint,gofmt -s, the conformance gate, the changelog gate and the CI script tests pass.Registry
semantics/unwrap-sum-without-collapsesisfixed(#739), withTestCompat_UnwrapSumWithoutKeepsSeries,TestSumWithoutDifferentialandTestSumWithoutSeriesLimitOnFoldedAnsweras its proofs.limits/series-limit-errorandseries-limits-and-partial-resultsdescribe the limit on the folded answer. The A/B setunwrap-sum-withoutand three Grafana pages (explore-sum-without-rate, the fix proof,explore-topk-sum-withoutandexplore-sum-without-count) are registered.Changed rows:
semantics/unwrap-sum-without-collapses(request is the bounded rate over| logfmt amount; the text states the extraction-list grouping, which is read for the fold only, and the collision fallback),semantics/unwrap-count-avg-without-after-parser(right after an extraction list, open for a bare parser; the request is an instantcount withoutover a bare| logfmt [1m], 415 series on Loki against the proxy's one), and the requests of the open rows aroundwithoutbelow (not the two quantile rows), which use the generator'slevellabel and a bounded parser, with the values measured against Loki.New open rows, found on the way:
count/avg withoutover unwrap after a parser.sumof a quantile answering[].sum byover quantile taking the group's quantile.semantics/without-empty-grouping-range-collapses:sum without () (max_over_time(... | unwrap ...))on query_range answers{}(1 series on the fixture), Loki keeps the 8 streams (without ()drops no label).semantics/without-over-inner-aggregation-collapses:sum without (pod) (sum by (pod, level) (rate(...)))and overcount_over_time, range and instant, answers{}; Loki a series perlevel.semantics/without-nested-under-aggregation:max(sum without (pod) (...))andsum by (level) (sum without (pod) (...))answer the total, Loki the largest per-level sum and a series per level.semantics/without-comparison-inside-ignored:sum without (pod) (rate(...) > 0.33)sums every series; Loki only the ones above the threshold (on origin/main an instant query also answers{}).semantics/topk-bottomk-without-strips-label:topk/bottomk without (pod) (1, ...)strips the label and answers other values; Loki keeps the top/bottom series with all labels.semantics/label-replace-instant-vector:label_replacearound a vector at an instant adds no label (applyLabelReplacehandles matrices only;semantics/label-replace-new-labelcovers query_range).semantics/label-replace-range-binary-marker:label_replacearound a binary expression on query_range sends the__binary__marker to VictoriaLogs: the proxy answers an empty matrix on v1.52.0 (Loki three series) and 502 from a backend that rejects the query.semantics/without-instant-detected-level-unknown: an instantwithoutanswer lacksdetected_level=unknownon a stream with no level label.semantics/unwrap-extraction-list-series-identity: sum, avg, count, max, min and topk of per-series functions, andcount_over_timeandrate, after an extraction list read other series than Loki outside awithout(figures above, from the emulator).semantics/unwrap-sum-without-extracted-label-collision: a parsed key that is also a stream label is not named<key>_extractedafter the fold.semantics/unwrap-sum-by-per-series-function-extraction-list:sum by (level) (max_over_time(... | logfmt amount | unwrap amount [5m]))answers the maximum of the level, not the sum of the per-series maxima.semantics/unwrap-bare-range-exposes-parsed-labels:rate(... | logfmt amount | unwrap amount [5m])exposes every parsed label (2,035 series against Loki's 268).The two quantile rows (the second and third above) are addressed in #731 (open). The series-limit rows are updated, and the limits reference states the row-budget error as a 400 next to the 502 of the raw-row reads.