Repository navigation
Read Gemfile.lock sections and DEPENDENCIES entries through formats::gem in hosted and vendored modes #780
Description
[agent] Filed by the scheduled architecture audit routine (ecosystems and formats). Register: discussion #560 register.
Kind: refactor. Source: review Part 5.4 ("Gem: … its own section model … two DEPENDENCIES-name parsers"), register E19 (gem half).
Problem
formats/gem says it is the one read model for Gemfile.lock, but hosted and vendored mode each re-scan the lock with their own section model and their own DEPENDENCIES-entry grammar. Verified on 045d7ec:
| Section model | DEPENDENCIES entry name | |
|---|---|---|
shared reader formats::gem::parse (inventory, VEX, lock_lists_direct_dependency, upstream restore) |
Section for every GEM/PATH/GIT/PLUGIN SOURCE header, with remotes and spec lines |
indent 2, split([' ', '(', '!']) (mod.rs#L286-L298) |
hosted formats::gem::hosted::converge_gem_lock_source |
its own GemLockSection (hosted.rs#L26-L35), GEM headers only, built by a second header walk |
gem_lock_dependency_name: trim_start, split(" ("), trim_end_matches('!') |
vendored vendor/gem.rs |
section_span(lines, header) / section_end: the first line equal to header, plus find_our_path_section and find_path_section (#L1926-L1942, #L2299-L2317) |
dep_entry_name: at_indent(2) + find([' ', '(', '!']), and spec_entry_name beside it |
So there are three section models and three DEPENDENCIES-name rules (the review counted two of the latter). The name rules agree on every Bundler-written entry I tried. The section models have already drifted: vendored section_span(&lines, "GEM") sees only the first GEM section, while the shared reader and hosted see all of them. Bundler 2 writes one GEM section per source, so a gem resolved from any source but the first can't be vendored. That bug is filed separately as #779, since it can be fixed before this refactor.
Symptoms and impact
- Vendored gem refuses a gem whose spec is not in the lock's first GEM section #779 (vendored gem refuses a gem outside the first
GEMsection). - Any future change to section or DEPENDENCIES handling, such as the platform-suffixed specs, a
GEMsection with several remotes, orPLUGIN SOURCE, has to be made three times.
Size: formats/gem is 1,149 lines, vendor/gem.rs 7,576 (2,461 production).
Proposed change
Make formats::gem the single source of line spans, and have both writers splice through it:
- Extend
GemfileLockwith what the writers need: eachSection's end line (exclusive, the next column-0 header), and the DEPENDENCIES entries as(line_no, name, pinned)using one name rule. Exposegem_sections(),path_sections(),dependencies()andsection(header). - Hosted: build
converge_gem_lock_source's section and DEPENDENCIES lookups fromparse(lk). DeleteGemLockSection, its header walk andgem_lock_dependency_name. - Vendored: replace
section_span,section_end,dep_entry_name,spec_entry_name,find_our_path_sectionandfind_path_sectionwith the parsed model. Delete them.edit_locklooks the spec up across everyGEMsection, which fixes Vendored gem refuses a gem whose spec is not in the lock's first GEM section #779 if it is still open.
Land it as two PRs, hosted first (smaller), then vendored. Neither changes output bytes.
Size and scope
Est. +120 / −250 production lines across formats/gem/mod.rs, formats/gem/hosted.rs and vendor/gem.rs. Out of scope: the Gemfile (Ruby source) models in formats/gem/manifest.rs and rewrite_gem's CHECKSUMS regexes (review Part 3, #340), and the hosted ↔ vendored takeover logic (#775, PR #776).
Acceptance criteria
-
GemLockSection,gem_lock_dependency_name,section_span,section_end,dep_entry_nameandspec_entry_nameno longer exist.grep -n '"DEPENDENCIES"' crates/socket-patch-core/srcfinds the header only informats/gem/mod.rsand tests. - Byte-identical output:
e2e_redirect_gem_build,e2e_vendor_gem_build, the hosted gem goldens and thevendor/gem.rsunit tests stay green unchanged. - New
formats::gemtests: section end lines with CRLF and trailing blank lines, a two-GEM-section lock, and DEPENDENCIES entriesrack,rack!,rack (~> 3.1),rack (= 3.2.6)!namingrackwith the rightpinnedflag. - A hosted test and a vendored test on a lock whose target gem is in the second
GEMsection.
Dependencies
Open PRs #768 and #776 edit vendor/gem.rs and formats/gem/mod.rs, so start after they merge. Blocks nothing; it makes #779's fix a deletion.
Backlog review — 2026-10-08
Priority: P1 → P3. Consolidating Gemfile.lock section models is useful maintenance, but this issue is not itself a reproduced urgent behavior failure.
Activity
audit-ecosystems: E19 filed (#780, #781); new E58 (#782), E59 (#779)
mikolalysenko commentedon Oct 4, 2026
[agent] Triaged as
priority:p1(Bundler refactor). Shares root cause with #779: vendored and hosted each keep a privateGemfile.locksection model instead of usingformats::gem. Per the issue, this waits for open PRs #768 and #776 (both editvendor/gem.rs), so it isn't eligible for an agent claim until they merge.Generated by Claude Code
mikolalysenko commentedon Oct 5, 2026
[agent] Re-checked on
main@4646693, after #805 (fixing #779) merged. The section models are still separate, and the vendored one has grown.vendor/gem.rskeepssection_span/section_end. Fix vendored gem lookup beyond the first GEM section (#779) #805 added a third private walker,gem_section_spans/record_gem_section, which re-derives the per-source GEM sections thatformats::gem::parsealready models.GemLockSection.The proposed change stands. It should now also delete
gem_section_spansandrecord_gem_section.Generated by Claude Code
mikolalysenko commentedon Oct 5, 2026
[agent] Re-checked on
9c43dfc. #750, #731 and #849 changed the gem readers, but none of them merged the section models. All three models are still present, and only their line numbers moved:GemLockSectionandgem_lock_dependency_name;section_span/section_end/gem_section_spans/record_gem_section,find_our_path_section,dep_entry_nameandspec_entry_name, andfind_path_section.formats/gem/gemfile.rsis new since the last check. It reads theGemfile, not the lock, so it doesn't change this issue's scope.Generated by Claude Code
mikolalysenko commentedon Oct 8, 2026
[agent] Re-checked on main @
e2d9633, after #768 (gem pair model) merged. #768 changed which manifest/lock pair is loaded, not how a lock is read, so the three section models and three DEPENDENCIES-name rules are unchanged:gem_lock_dependency_nameandGemLockSection;section_span/section_end,dep_entry_name/spec_entry_name,find_our_path_sectionandfind_path_section.The plan in the body still applies.
Generated by Claude Code
mikolalysenko commentedon Oct 9, 2026
[agent] Claiming this issue for the architecture refactor routine (highest leverage: with every vendored backend,
redirect/mod.rsandapi/client.rsbusy in open PRs, this hosted slice is the best free one. It deletes hosted's secondGemfile.locksection model and DEPENDENCIES-name rule, and it unblocks the vendored slice). Branch: arch-refactor/780-gem-lock-sections. Claim-ID: 2026-10-09T03:55:34Z-2022a0Slice: step 1 and step 2 of the proposed change.
formats::gem::parserecords section spans, remote lines and DEPENDENCIES entries, and hostedconverge_gem_lock_sourcereads them;GemLockSection, its header walk andgem_lock_dependency_nameare deleted. The vendored slice (step 3,vendor/gem.rs) remains: that file is changed by open PRs #1041 and #1026.Generated by Claude Code
mikolalysenko commentedon Oct 9, 2026
[agent] Draft PR for the hosted slice: #1221.
Generated by Claude Code
Locate hosted gem lock edits via formats::gem
mikolalysenko commentedon Oct 9, 2026
[agent] The hosted slice merged in #1221:
converge_gem_lock_sourcenow reads sections, remote lines and DEPENDENCIES entries throughformats::gem::parse, andGemLockSection/gem_lock_dependency_nameare gone. The vendored slice (vendor/gem.rssection_span/gem_section_spans) remains;vendor/gem.rsis changed by open PRs #1245 and #1026, so I'm releasing the claim until it frees up.Generated by Claude Code