Repository navigation
ci(nvim): measure wall-clock time around the startup benchmark - #44
Merged
Merged
Conversation
nvim's own --startuptime only instruments Lua/vimscript execution, so it can't explain why CI's numbers (up to 500ms+ on macos-14) are far higher than the same builds run locally (~25ms) -- that gap has to be happening before Lua even starts: process spawn, dynamic linker resolving nvim's own dylib/so deps, or macOS Gatekeeper/AMFI verifying freshly-downloaded nix-built binaries on a runner that's never seen them before. Wrap the benchmark with the shell's `time` builtin for an outside-in wall-clock figure to compare against nvim's internal mean, and print each run's individual time (not just aggregate stats) to see whether the cost is front-loaded (cold-start/Gatekeeper tax amortizing across the 10 runs) or uniform across every run.
Without a cache, "Restore plugins" does a full network git clone of all ~40 plugins on every single run, so the benchmark that follows always starts from a cold filesystem. Cache ~/.local/share/nvim/lazy keyed on runs-on (native build output like telescope-fzf-native's is arch-specific) and lazy-lock.json's hash, so a cache hit skips the clone entirely. Also corrects the comment above the benchmark step: the wall-clock measurement added in the previous commit shows the pre-Lua gap is small (~5-40ms/run) on both platforms, not the dominant factor -- the 100-600ms CI readings are real Lua/vimscript execution time, most likely from lazy.nvim's spec-resolution work getting hit by contention on weak/shared runners (worst on macos-14), evidenced by erratic non-monotonic per-run timings rather than a clean cold-cache warm-up curve.
…rmed Verified on a real pull_request-scoped run: caching lazy.nvim plugins alone took every matrix entry from 100-600ms down to ~20-27ms, matching local numbers almost exactly, with the erratic per-run spread gone too. That confirms the actual cause was cold disk reads for freshly-cloned plugin files landing inside nvim's own require()/sourcing timings -- not CPU contention on shared runners, which was our working theory before this data came in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #41. The CI benchmark reported dramatically higher numbers than the same builds run locally (e.g. macos-14 median ~300ms, up to 544ms, vs. ~25ms locally for the identical nix-built package).
nvim --startuptimeonly instruments Lua/vimscript execution — it can't see process spawn, dynamic-linker resolution of nvim's own.dylib/.sodependencies, or macOS Gatekeeper/AMFI scanning freshly-downloaded nix-built (unsigned/ad-hoc-signed) binaries on a runner that's never seen them before. That's the leading hypothesis for the gap, but it needs to actually be measured on the CI hardware to confirm.This adds:
time) around the whole 10-run benchmark, printed alongside nvim's own internal mean, so we can see if the gap is pre-Lua process-launch overhead.No change to what's being measured for the config itself — purely diagnostic, to decide what (if anything) is actually actionable here vs. CI-environment noise.
Test plan
nix develop .#cilint suite passesnix flake check --all-systemspassesTIMEFORMAT/timewrapping and per-run output locally🤖 Generated with Claude Code