docs: stop publishing macOS-measured numbers as Linux results - #819
Open
TinDang97 wants to merge 3 commits into
Open
docs: stop publishing macOS-measured numbers as Linux results#819TinDang97 wants to merge 3 commits into
TinDang97 wants to merge 3 commits into
Conversation
The most-repeated performance claims in the project trace to a table that docs/benchmarks.md:9 itself labels an Apple M4 Pro development reference, and docs/journey.md re-published those same values under a preamble asserting they came from GCloud Linux hosts. CLAUDE.md and docs/PRODUCTION-CONTRACT.md both require every benchmark number to come from a Linux host. Three claims are withdrawn, not relabelled, because no defensible provenance exists for them: - "45x / 23x better CPU" — macOS-measured, sampled with `ps -o %cpu=` (process-lifetime average, not steady-state load), and computed from a CPU numerator and an RPS denominator taken in different runs. Linux (§2.14) is a tie: 10.55 vs 11.33 us/op, inside Redis's own 11.9% spread. - "p50 latency 8-10x lower" — macOS-measured, and produced by redis-benchmark, a closed-loop tool. No Linux latency comparison exists. - "27-35% less memory" — macOS-measured, and contradicted by BENCHMARK.md §2.14, which concludes "memory is not won" (1.16x worse at 64 B, 1.26x worse on idle RSS) and retracts the per-key memory win as an artifact of redis-benchmark's default 3-byte value. Removed from eight sites. Everything else is relabelled with host, shard count, pipeline depth, client count, Redis version and Redis threading config taken from the source section — or, where the source run did not record a condition (§2.1 records neither the Redis io-threads setting nor the payload size), the table now says so instead of filling it in. Also corrected against moon's own interest: docs/journey.md's retracted "1->8 shards is flat-to-slightly negative" now carries the corrected Linux scaling of 1.42x / 2.14x / 3.79x. README's "hash-field TTL (Valkey-parity)" now reads as feature parity and 4-10% behind Valkey, which is what the three-way bench measured. The Moon-vs-Valkey throughput ratio is withdrawn as structurally asymmetric. docs/references.md no longer claims open-loop methodology for figures produced by a closed-loop tool. docs/index.md gained the scope caveat it previously had none of. Separately, removes an abandoned first-draft bullet left immediately above its replacement in the Unreleased #788 entry. Docs and CHANGELOG prose only. No new measurements were taken; no number was invented, adjusted or extrapolated — every surviving figure already existed in the tree. Refs #817 author: Tin Dang
|
ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing |
|
Warning Review limit reachedNext included review available in 40 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (14)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…asured 15-17% PR #819 deleted the "27-35% less memory" claim everywhere on the grounds that no Linux measurement backed it. The measurement now exists, so the claim is replaced rather than removed — with the losses published alongside the win. Measured 2026-09-04 on GCE c3-standard-8 (Linux 6.17, x86_64, 8 vCPU / 31 GB), moon d5f3501 at --shards 1, via scripts/bench-resources.sh with a fresh server per data point and `redis-benchmark -r N` for unique keys. Per-key = (loaded RSS - baseline RSS) / DBSIZE. The oracle had to be rebuilt for the comparison to be legitimate: both Redis binaries already installed on the host were libc-malloc builds, which inflate Redis RSS and would have biased every ratio in moon's favour. Redis 7.4.2 was rebuilt from source against jemalloc 5.3.0, and that is the only Redis these numbers are against. Result: 15-17% less memory per key at values >= 1 KB (9.5% at the smallest key count tested), a tie at 256 B, and a LOSS of 11-51% at 32 B — published, not omitted. Why the old number was wrong, and it was not moon that changed: the published 1M x 1 KB row was Redis 1,571 B / moon 1,153 B. Re-measured: Redis 1,380 B / moon 1,172 B. moon's own figure moved 1.6%; the oracle moved 12%. The inflation came from a Redis baseline measured on macOS and/or without jemalloc. BENCHMARK.md section 3 is now the single canonical provenance site — host, build, oracle, harness, formula and the full twelve-point table. Every other document states the number and links there. Also corrected against moon's interest: - BENCHMARK.md 3.1 empty-server RSS. Published "Redis 7.0 MB / moon 1 shard 7.0 MB - identical"; measured on all 12 points, Redis 7.5-7.7 MB and moon 12.6-12.9 MB, i.e. moon ~1.7x worse. Cause unknown, tracked in #821. The "moon (12 shards) 15.7 MB" row is kept and marked unverified/stale. - BENCHMARK.md 3.4 TTL overhead is marked UNVERIFIED: that section of bench-resources.sh omits `redis-benchmark -r`, so it loaded 1 key instead of 500,000. - BENCHMARK.md 2.14's "not met on memory" verdict is scoped to the value sizes it actually measured (<= 256 B), and its "moon --shards 1 beats Redis on idle RSS" observation is marked host-specific, since 3.1 measures the opposite on x86_64. Explicitly not published from this run: the throughput and CPU columns (the harness measures them incidentally and they are not a clean benchmark), any ARM result (the run was x86_64 only), and anything the run did not measure. Refs #817, #821 author: Tin Dang
The superseded-tables note credited only the 1M x 1 KB row. The 35% end of the range came from the 500K row (1,168/1,786 = 34.6%); the 27% end from the 1M row (1,153/1,571 = 26.6%). Both are now named with their arithmetic. author: Tin Dang
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #817.
Docs only —
git diff --stat origin/mainis 13 files, all.md. No newmeasurement was taken and no number was invented: every surviving figure was
verified to pre-exist in
origin/main(independent check over 82 distinctnumeric tokens in the added lines; the only one not in the base docs is
817,this issue's number).
Why
docs/benchmarks.md:9declares the tables below it an "Apple M4 Pro(12 cores, 24 GB) development reference". Those tables are numerically
identical to
BENCHMARK.md§3.2/§4/§5.1/§7/§8/§9 — anddocs/journey.md:18-21republishes the same values under a preamble asserting they came from GCE Linux
hosts.
CLAUDE.mdanddocs/PRODUCTION-CONTRACT.md:46-47both forbidpublishing a macOS number.
Deleted (please object if you disagree with any of these)
ps -o %cpu=(a process-lifetime average); numerator and denominator taken in different
runs. Replaced with the Linux §2.14 result, which is a tie
(10.55 vs 11.33 µs/op).
a closed-loop tool. No Linux latency comparison exists in the tree.
"27–35% less memory"— no longer deleted. It is now REPLACED with ameasured Linux number. See "Memory: measured, not deleted" at the bottom.
vs p=10, unrecorded vs 512 B payload, unrecorded vs 9 I/O threads, different
harnesses, and moon's side is its best two commands. No ratio replaces it.
Linux §7.3 figures (1.32×
everysecp16, 0.91×alwaysp16).docs/references.md's "follows open-loop principles" — every publishedfigure comes from
redis-benchmark, which is closed-loop.Corrected against moon's own interest
docs/journey.md:113-119still asserted "1→8 shards is flat-to-slightlynegative".
CHANGELOG.md:665-670retracts that as a one-literal-key harnessartifact; real scaling is 1.42× / 2.14× / 3.79×. Now stated, with Redis's
own io8/io1 alongside.
README.md:252-253"hash-field TTL (Valkey-parity)" now reads featureparity, with the measured result: Valkey 9.1.0 leads on every
HEXPIRE-family scenario (5–10% at p=16, 4–8% at p=1).
Relabelled, not removed
Provenance banners on
BENCHMARK.md§3/§4/§5/§7.1/§8/§9 naming the Linuxsection that supersedes each;
docs/benchmarks.md's executive summary is nowLinux-only with host / Redis version / shards / clients / pipeline depth /
payload per row;
docs/index.mdgains the scope warning it never had — thepipelined win is GET/SET only, and every other command family runs
0.40–0.67× Redis at p≥8.
Also fixed
The duplicated, truncated
#788bullet inCHANGELOG.md— an abandoned firstdraft left immediately above its replacement, whose dangling line re-published
a claim a later commit was written to retract. (It was at lines 98-99, not
191-194 as the issue estimated.)
Left alone — flagging for a human call
:15062,:15138,:6268-6269) stillcarry the retired claims. A changelog is a dated record of what was claimed
at the time, and the retraction is recorded in the same file. Rewriting
history is a judgement call.
docs/assets/journey-cpu.pngis now unreferenced; the other two are captionedwith provenance. Regenerating them is not a docs-text change.
Per-key memory on Linux has never been measured at 1 KB+ values— ithas now, on 2026-09-04. See the section at the bottom of this description.
scripts/still never sets Redisio-threads. That is a harness change; thedocs now say explicitly where the setting was not recorded rather than
assuming one.
Memory: measured, not deleted (added 2026-09-04, commit
af589672)The first draft of this PR deleted the memory claim because no Linux measurement
backed it. The measurement now exists, so the claim is replaced rather than
removed — and the losses are published alongside the win.
Provenance. GCE
c3-standard-8, Linux 6.17.0-1022-gcp, x86_64, 8 vCPU /31 GB. moon
d5f3501b(main, post-#786) at--shards 1.scripts/bench-resources.sh, fresh server instance per data point,redis-benchmark -r Nfor unique keys, per-key = (loaded RSS − baseline RSS) /DBSIZE.The oracle had to be rebuilt. Both Redis binaries already installed on the
host were libc-malloc builds. libc malloc inflates Redis RSS, and every ratio
taken against it would have been biased in moon's favour. Redis 7.4.2 was
rebuilt from source against jemalloc 5.3.0; that is the only Redis these
numbers are against.
Headline: 15–17% less per key at values ≥ 1 KB (9.5% at the smallest key
count tested). Not 27–35%.
Why the old claim was wrong — and moon is not what changed. The published
1M × 1 KB row was Redis 1,571 B / moon 1,153 B. Re-measured: Redis 1,380 B /
moon 1,172 B. moon's own figure moved 1.6%; the oracle moved 12%.
The old claim was inflated by a Redis baseline measured on macOS and/or without
jemalloc, not by anything moon did. That was this PR's thesis; it is now
demonstrated with numbers rather than asserted.
Also corrected, against moon's interest
BENCHMARK.md§3.1 empty-server RSS. Published "Redis 7.0 MB / moon1 shard 7.0 MB — identical". Measured on all 12 points: Redis 7.5–7.7 MB,
moon 12.6–12.9 MB — moon ~1.7× worse. Not silently swapped: the measured
figure is stated with its provenance and linked to moon's empty-server RSS is 12.8 MB, not the 7.0 MB BENCHMARK.md publishes — a 1.7x baseline regression nobody was watching #821, which tracks
finding the cause. The "moon (12 shards) 15.7 MB" row is kept and marked
unverified / stale — this run did not measure it.
BENCHMARK.md§3.4 TTL overhead is marked UNVERIFIED. That part of theharness omits
redis-benchmark -r, so it loaded 1 key instead of 500,000.The structural description is retained as a reading of the source, not a
measurement.
BENCHMARK.md§2.14's "not met on memory" verdict is scoped to the valuesizes it actually measured (≤ 256 B), and its "moon
--shards 1beats Redis onidle RSS" observation is marked host-specific, because §3.1 measures the
opposite on x86_64 and nothing in either run explains the gap.
Explicitly not published from this run
are not a clean benchmark.
now says x86_64 explicitly.
and 256 B, run-to-run spread (one point per cell, no repetitions).
Where the provenance lives
BENCHMARK.md§3 is the single canonical site: host, build, oracle, harness,formula, the full twelve-point table (§3.2), the corrected baseline RSS (§3.1),
and the superseded macOS tables kept in §3.2a so the retired claim can still be
traced to the numbers it came from. Every other document —
README.md,docs/index.md,docs/benchmarks.md,docs/journey.md,docs/design-advantages.md,docs/comparison-valkey.md,docs/architecture.md,docs/PRODUCTION-CONTRACT.md— states the number and links there.