Skip to content

Commit 8ce40d9

Browse files
committed
Demote hosted format-1 bun.lockb locks exactly on the vendor takeover
The bun backtest's binary job failed only for the format-1 writers (Bun 0.1.1 / 0.1.6, readers 0.5.9 and 1.4.2) after 65aa074: the hosted rewrite promotes a format-1 lock to format 2 (format 1 has no URL column), and the new native upstream restorer rebuilt the registry record inside that promoted lock. The vendor ledger then recorded the promoted bytes as its pre-vendor original, so `vendor --revert` returned a format-2 lock and e2e_bun_lockb's "the revert restores the pre-hosted bytes exactly" assertion failed. A promoted lock is byte-for-byte indistinguishable from one Bun 0.1.7+ wrote (same pool order, same metadata hash), and hosted mode keeps no ledger, so the restorer could not tell. The codec now marks a lock whenever an edit had to normalize it: seven magic bytes and a flag byte in the last eight bytes of the root package's resolution (the root resolution's value union, which no Bun reader reads; early writers leave uninitialized bytes there, and 1.4.2 / 0.5.9 read such locks). For a promoted lock those bytes are new, so the mark overwrites nothing. promote_legacy_format sets NORMALIZED_FORMAT_1; normalize_workspace_behaviors sets NORMALIZED_WORKSPACE when it changes a dependency behavior or workspace literal. The vendor ledger's layout_original path normalizes the original the same way, so its exact revert is unchanged. The bun.lockb upstream restorer then: - demotes a NORMALIZED_FORMAT_1 lock back to format 1 once every hosted record is rebuilt (BunLockb::demote_legacy_format, which drops the URL column and the URLs the promotion appended to the pool, and succeeds only if promoting the result again reproduces the lock byte for byte). Otherwise the pins are refused with the `git checkout -- bun.lockb` remedy. - refuses a NORMALIZED_WORKSPACE lock outright with that remedy, since clearing the workspace behavior bit cannot be undone. It no longer takes the lock over non-exactly. Tests: the upstream restorer's byte-exact test now covers 0.1.1 / 0.1.6. Workspace-normalized extension locks refuse, and so does a lock whose mark carries an unknown flag. The codec rebuild test checks the exact demotion. vendor_eject_bun_lockb adds the 0.1.1 / 0.1.6 takeover with an exact revert and a workspace-lock refusal. Locally with real Bun, the 1.4.2 reader x 0.1.1 / 0.1.6 writer cells now pass the takeover and the exact revert. They then stop at the 0.5.9 legacy reader, which segfaults on any networked install in this sandbox. The 1.1.45 / 1.2.0 / 1.4.2 cells pass everything except the `extensions` shape, which needs api.github.com (403 here). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tncU7u7FK48tubFNA12UQ
1 parent 698bef6 commit 8ce40d9

8 files changed

Lines changed: 272 additions & 50 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -194,11 +194,14 @@ into the new version's section — see docs/releasing.md.
194194
pre-hosted lock. The hosted rewrite now keeps the registry record's
195195
inactive bytes (padding, semver) in the tarball record it writes, and a
196196
re-pin to a later grant's URL drops the superseded URL from the string
197-
pool, so the rebuild is byte-exact — early writers' uninitialized padding included —
198-
except for a binary format 1 lock (kept promoted to format 2) and a
199-
workspace lock (dependency behaviors kept normalized). Those exceptions
200-
keep `rollback` / `remove` refusing a hosted `bun.lockb` pin with the
201-
`git checkout -- bun.lockb` remedy; an `--offline` vendor still refuses.
197+
pool, so the rebuild is byte-exact — early writers' uninitialized padding included.
198+
Where the hosted rewrite had to normalize the lock it marks it (in the
199+
root package's resolution bytes, which no Bun reader reads): a binary
200+
format 1 lock (Bun 0.1.1-0.1.6), promoted to format 2, is demoted back to
201+
its exact format-1 bytes, and a lock whose workspace dependency behaviors
202+
were normalized is refused with the `git checkout -- bun.lockb` remedy
203+
instead of taken over non-exactly. `rollback` / `remove` keep refusing a
204+
hosted `bun.lockb` pin with that remedy; an `--offline` vendor still refuses.
202205
- **`vendor_supersedes_redirect` is removed.** The vendored flows no longer
203206
warn about (or auto-reconcile, or unwind the `.npmrc` for) a stale hosted
204207
ledger record: once the lock routes a package to `.socket/vendor/`, no

‎crates/socket-patch-cli/CLI_CONTRACT.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -957,7 +957,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem
957957

958958
* **Scope.** The hosted pins are what lockfile discovery finds — `(purl, patch uuid, files wiring it)`, recognized only on `https://patch.socket.dev` or the `--patch-server-url` / `SOCKET_PATCH_SERVER_URL` origin. A scoped rollback (paths / identifiers / `--ecosystems`) restores exactly the pins in scope; each pin restores or refuses on its own (there is no whole-ledger replay, and a pre-v5 ledger's edits are never replayed). A pin discovery cannot see is out of reach: a lockless cargo `registry = "socket-patch-<uuid>"` pin, a nuget exact-id mapping with no `packages.lock.json`, a gem wired only in the `Gemfile` (pre-bundler-2.6 mixed state) — restore those files from version control.
959959
* **What a restore does.** Every file wiring the pin is rewritten back to the DEFAULT UPSTREAM registry entry for `name@version`, re-resolving whatever the entry pins (tarball URL, integrity, checksum, hashes) from the public registry; only the hosted entries change and every other byte stays the file's own. A pin is **all-or-nothing**: refused in one of its files, it is restored in none of them, so no pin is left half hosted. Nothing reaches disk until every pin has resolved, and `--dry-run` resolves exactly like a wet run — registry lookups included — and skips only the write. Per format:
960-
* **npm family** — `package-lock.json` / `npm-shrinkwrap.json`, `yarn.lock` (classic and berry), `pnpm-lock.yaml` / `shrinkwrap.yaml`, `bun.lock`: resolution + integrity (+ shasum where recorded) from the npm registry's version document (`SOCKET_NPM_REGISTRY`). Side settings: a project `.npmrc` that is exactly `allow-remote=all\n` is deleted once no root npm lock entry is hosted, otherwise a remaining top-level `allow-remote=all` warns `npm_allow_remote_left`; a `pnpm-workspace.yaml` that is exactly the scaffold hosted mode creates is deleted once `pnpm-lock.yaml` is no longer hosted, otherwise a remaining `trustLockfile: true` warns `pnpm_trust_lockfile_left`. **`bun.lockb` (binary)**: `rollback` and `remove` refuse it (the checkout remedy) — a rebuilt record is not byte-exact for every lock: a binary format 1 lock stays promoted to format 2 and a workspace lock keeps its normalized dependency behaviors. The hosted → vendored takeover and the eject DO restore it, since the vendor ledger then records the rebuilt record as its pre-vendor original: the native codec turns each hosted remote-tarball record back into Bun's npm registry record for `name@version` (the registry's `dist.tarball` + `dist.integrity`, the package metadata hash re-derived, the hosted URL string dropped from the string pool). The hosted rewrite keeps the registry record's inactive bytes (padding, semver) in the tarball record, so a lock it wrote comes back byte for byte — early writers' uninitialized padding included; a record without them (an older socket-patch or a Bun re-save) is rebuilt the way Bun writes one, and refused for a prerelease/build version.
960+
* **npm family** — `package-lock.json` / `npm-shrinkwrap.json`, `yarn.lock` (classic and berry), `pnpm-lock.yaml` / `shrinkwrap.yaml`, `bun.lock`: resolution + integrity (+ shasum where recorded) from the npm registry's version document (`SOCKET_NPM_REGISTRY`). Side settings: a project `.npmrc` that is exactly `allow-remote=all\n` is deleted once no root npm lock entry is hosted, otherwise a remaining top-level `allow-remote=all` warns `npm_allow_remote_left`; a `pnpm-workspace.yaml` that is exactly the scaffold hosted mode creates is deleted once `pnpm-lock.yaml` is no longer hosted, otherwise a remaining `trustLockfile: true` warns `pnpm_trust_lockfile_left`. **`bun.lockb` (binary)**: `rollback` and `remove` refuse it (the checkout remedy). The hosted → vendored takeover and the eject DO restore it, since the vendor ledger then records the rebuilt record as its pre-vendor original: the native codec turns each hosted remote-tarball record back into Bun's npm registry record for `name@version` (the registry's `dist.tarball` + `dist.integrity`, the package metadata hash re-derived, the hosted URL string dropped from the string pool). The hosted rewrite keeps the registry record's inactive bytes (padding, semver) in the tarball record, so a lock it wrote comes back byte for byte — early writers' uninitialized padding included; a record without them (an older socket-patch or a Bun re-save) is rebuilt the way Bun writes one, and refused for a prerelease/build version. A lock the hosted rewrite had to normalize is marked in the root package's resolution value bytes (which no Bun reader reads): a binary format 1 lock it promoted to format 2 is demoted back to its exact format-1 bytes (verified by promoting it again, otherwise refused), and a lock whose workspace dependency behaviors it normalized is refused with the `git checkout -- bun.lockb` remedy.
961961
* **vlt** — `vlt-lock.json`: slot [2] from the registry's `dist.integrity`, slot [3] per the lock's own convention (see the vlt hosted-mode contract); every hosted instance of the pin together.
962962
* **cargo** — `Cargo.lock` back on crates.io (source + the sparse index's checksum, `SOCKET_CRATES_INDEX`); every `Cargo.toml` declaration loses its `registry = "socket-patch-<uuid>"` pin (the shorthand the rewriter produced collapses back); the unreferenced `[registries.socket-patch-<uuid>]` block leaves the project cargo config. A declaration it cannot unpin refuses.
963963
* **golang** — the hosted `replace` and the socket module's go.sum lines go; the upstream module's two go.sum lines come back, hashed from the module proxy (`SOCKET_GOPROXY`, else `GOPROXY` / `GONOPROXY` / `GOPRIVATE` as go reads them) and cross-checked against the checksum database (`SOCKET_GOSUMDB_URL`, else `sum.golang.org` unless `GOSUMDB=off` / `GONOSUMDB` / `GOPRIVATE` say go would not ask it). A `replace` the user had before the hosted run is not recorded anywhere, so the restore lands on the plain upstream module.

‎crates/socket-patch-cli/tests/e2e_bun_lockb.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -116,8 +116,8 @@ fn cli_code(project: &Path, args: &[&str]) -> (i32, Value) {
116116
}
117117

118118
/// v5 keeps no hosted ledger, so undoing a hosted pin means restoring the
119-
/// entry's upstream registry form. For a binary `bun.lockb` that rebuilt
120-
/// record is not byte-exact for every lock (format 1, workspace locks), so
119+
/// entry's upstream registry form. For a binary `bun.lockb` only a vendor
120+
/// takeover rebuilds that record (it refuses a workspace-normalized lock), so
121121
/// `rollback` REFUSES the pin, naming the checkout remedy, and leaves the
122122
/// lock exactly as found; the test then applies that remedy (`git checkout --
123123
/// bun.lockb`, here: the original bytes written back).

‎crates/socket-patch-cli/tests/vendor_eject_bun_lockb.rs‎

Lines changed: 41 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,9 @@
88
//! per-purl takeover (`vendor` with the patch record staged) and the eject
99
//! (`vendor` in a manifest-less hosted project). Each vendors, records the
1010
//! REGISTRY record as the vendor ledger's pre-vendor original, and
11-
//! `vendor --revert` returns the exact pre-hosted bytes. `rollback` of the
11+
//! `vendor --revert` returns the exact pre-hosted bytes (a format-1 lock the
12+
//! hosted rewrite promoted is demoted back; a lock whose workspace behaviors
13+
//! it normalized is refused with the checkout remedy). `rollback` of the
1214
//! hosted pin still refuses with the `git checkout -- bun.lockb` remedy, as
1315
//! does an offline vendor (the registry cannot be asked).
1416
//!
@@ -251,12 +253,14 @@ fn assert_vendored(p: &Project, env: &Value) {
251253
assert_eq!(original["integrity"], UPSTREAM_INTEGRITY, "{original}");
252254
}
253255

254-
/// Bun 0.8.1 (uninitialized record padding), 1.1.38 (the last binary-only
255-
/// writer) and 1.2.0 (the legacy lock Bun 1.2 keeps): the takeover vendors
256-
/// over the hosted pin, and the revert gives back the pre-hosted bytes.
256+
/// Bun 0.1.1 / 0.1.6 (binary format 1, which the hosted rewrite promotes to
257+
/// format 2 and the takeover's restore demotes again), 0.8.1 (uninitialized
258+
/// record padding), 1.1.38 (the last binary-only writer) and 1.2.0 (the
259+
/// legacy lock Bun 1.2 keeps): the takeover vendors over the hosted pin, and
260+
/// the revert gives back the pre-hosted bytes.
257261
#[tokio::test]
258262
async fn takeover_vendors_over_a_hosted_bun_lockb_and_reverts_exactly() {
259-
for writer in ["0.8.1", "1.1.38", "1.2.0"] {
263+
for writer in ["0.1.1", "0.1.6", "0.8.1", "1.1.38", "1.2.0"] {
260264
let p = hosted_project(writer).await;
261265
stage_record(p.root());
262266
let hosted = p.lock();
@@ -311,6 +315,38 @@ async fn eject_vendors_a_hosted_bun_lockb_and_reverts_exactly() {
311315
assert!(p.lock() == p.pristine, "exact pre-hosted bytes");
312316
}
313317

318+
/// A hosted lock whose workspace dependency behaviors the rewrite had to
319+
/// normalize cannot be given back byte for byte: the takeover refuses with
320+
/// the checkout remedy (dry and wet alike) and writes nothing.
321+
#[tokio::test]
322+
async fn takeover_refuses_a_workspace_normalized_hosted_bun_lockb() {
323+
let p = hosted_project("1.1.45-extensions").await;
324+
stage_record(p.root());
325+
let hosted = p.lock();
326+
for extra in [&["--dry-run"][..], &[][..]] {
327+
let mut args = vec!["vendor"];
328+
args.extend_from_slice(extra);
329+
let (code, env) = p.run_json(&args);
330+
assert_eq!(code, 1, "{extra:?}: {env:#}");
331+
let refused = env["events"]
332+
.as_array()
333+
.and_then(|events| {
334+
events
335+
.iter()
336+
.find(|e| e["errorCode"] == "redirect_revert_failed")
337+
})
338+
.unwrap_or_else(|| panic!("expected redirect_revert_failed: {env:#}"));
339+
assert!(
340+
refused["error"].as_str().is_some_and(|e| e
341+
.contains("workspace dependency behaviors")
342+
&& e.contains("git checkout -- bun.lockb")),
343+
"{env:#}"
344+
);
345+
assert_eq!(p.lock(), hosted, "{extra:?}: a refused vendor writes nothing");
346+
assert!(!p.root().join(".socket/vendor").exists());
347+
}
348+
}
349+
314350
/// `rollback` of the hosted pin, and a vendor that cannot reach the
315351
/// registry, refuse with the checkout remedy and write nothing.
316352
#[tokio::test]

‎crates/socket-patch-core/src/patch/redirect/upstream/bun_lockb.rs‎

Lines changed: 86 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,20 @@
66
//! ([`BunLockb::set_registry_package`], which also re-derives the metadata
77
//! hash Bun's frozen install checks). Every other byte is the file's own.
88
//!
9+
//! Where the hosted rewrite had to normalize the lock, the codec marked it
10+
//! (see `vendor::bun_lockb`): a lock promoted from binary format 1 is
11+
//! demoted back to its exact format-1 bytes once every hosted record is
12+
//! rebuilt, and a lock whose workspace dependency behaviors were normalized
13+
//! (not invertible) is refused — the checkout remedy, never a non-exact lock.
14+
//!
915
//! Only reached under [`super::RestoreOptions::bun_lockb`]: see there for
1016
//! why `rollback` keeps refusing a binary lock.
1117
1218
use std::collections::BTreeSet;
1319

1420
use super::npm::{by_uuid, fetch_dists, refuse_all_in};
1521
use super::{Ctx, FormatResult, HostedPin, View};
16-
use crate::vendor::bun_lockb::BunLockb;
22+
use crate::vendor::bun_lockb::{BunLockb, NORMALIZED_FORMAT_1, NORMALIZED_WORKSPACE};
1723

1824
pub(super) async fn restore(
1925
view: &mut View<'_>,
@@ -45,6 +51,19 @@ pub(super) async fn restore(
4551
continue;
4652
}
4753
};
54+
let flags = lock.normalized_flags();
55+
if flags & NORMALIZED_WORKSPACE != 0 {
56+
refuse_all_in(
57+
&pins,
58+
rel,
59+
&mut result,
60+
format!(
61+
"the hosted rewrite normalized {rel}'s workspace dependency behaviors, \
62+
so its original bytes cannot be rebuilt"
63+
),
64+
);
65+
continue;
66+
}
4867
let packages = match lock.packages() {
4968
Ok(packages) => packages,
5069
Err(e) => {
@@ -85,6 +104,7 @@ pub(super) async fn restore(
85104
.collect();
86105
let dists = fetch_dists(&wanted, ctx, &mut result).await;
87106
let mut changed = false;
107+
let mut restored = Vec::new();
88108
for (id, uuid, name, version) in hits {
89109
if result.refused.contains_key(&uuid) {
90110
continue;
@@ -102,12 +122,27 @@ pub(super) async fn restore(
102122
// Transactional per record: a failed rebuild leaves `lock` as is.
103123
match lock.set_registry_package(id, &version, &dist.tarball, integrity) {
104124
Ok(()) => {
105-
result.handled.insert(uuid);
125+
restored.push(uuid);
106126
changed = true;
107127
}
108128
Err(e) => result.refuse(&uuid, format!("{rel} package #{id}: {e}")),
109129
}
110130
}
131+
if changed && flags & NORMALIZED_FORMAT_1 != 0 {
132+
// The hosted rewrite promoted a format-1 lock: give back its
133+
// exact original bytes, or nothing.
134+
match lock.demote_legacy_format() {
135+
Ok(Some(original)) => lock = original,
136+
Ok(None) => {}
137+
Err(e) => {
138+
for uuid in restored.drain(..) {
139+
result.refuse(&uuid, format!("{rel}: {e}"));
140+
}
141+
changed = false;
142+
}
143+
}
144+
}
145+
result.handled.extend(restored);
111146
if changed {
112147
view.write_bytes(rel, lock.bytes());
113148
}
@@ -212,13 +247,16 @@ mod tests {
212247
}
213248
}
214249

215-
/// Every binary writer whose lock the hosted rewrite leaves in the
216-
/// format it found gets its exact pre-hosted bytes back — including the
217-
/// uninitialized padding early writers leave in registry records.
250+
/// Every binary writer gets its exact pre-hosted bytes back — including
251+
/// the uninitialized padding early writers leave in registry records,
252+
/// and a format-1 lock (0.1.1 / 0.1.6), which the hosted rewrite
253+
/// promoted to format 2 and the restore demotes again.
218254
#[tokio::test]
219255
#[serial_test::serial]
220256
async fn hosted_record_restores_byte_exact_on_every_writer() {
221257
for dir in [
258+
"0.1.1",
259+
"0.1.6",
222260
"0.1.7",
223261
"0.5.9",
224262
"0.6.7",
@@ -253,40 +291,60 @@ mod tests {
253291
}
254292
}
255293

256-
/// Where the hosted rewrite had to normalize the lock (format 1 promoted
257-
/// to format 2; workspace dependency behaviors), the restore is the
258-
/// registry record, not the original bytes.
294+
/// Where the hosted rewrite had to normalize workspace dependency
295+
/// behaviors (not invertible), the restore refuses with the checkout
296+
/// remedy and writes nothing, instead of returning a non-exact lock.
259297
#[tokio::test]
260298
#[serial_test::serial]
261-
async fn normalized_locks_restore_the_registry_record() {
262-
for dir in [
263-
"0.1.1",
264-
"0.1.6",
265-
"1.1.45-extensions",
266-
"1.2.23-extensions",
267-
"1.4.2-extensions",
268-
] {
299+
async fn workspace_normalized_locks_refuse() {
300+
for dir in ["1.1.45-extensions", "1.2.23-extensions", "1.4.2-extensions"] {
269301
let original = fixture(dir);
270302
let upstream = minimist(&original);
271303
let server = registry(upstream.integrity.as_deref().unwrap()).await;
272304
std::env::set_var("SOCKET_NPM_REGISTRY", server.uri());
273-
let (outcome, after) = run(&hosted(&original), &vendor_opts()).await;
274-
std::env::remove_var("SOCKET_NPM_REGISTRY");
275-
assert_eq!(outcome.pins[0].status, PinStatus::Restored, "{dir}");
276-
let restored = minimist(&after);
277-
assert_eq!(restored.resolution, UPSTREAM_URL, "{dir}");
278-
assert_eq!(restored.integrity, upstream.integrity, "{dir}");
279-
BunLockb::parse(&after)
280-
.unwrap()
281-
.validate_mutation()
282-
.unwrap();
283-
assert!(
284-
!after.windows(16).any(|w| w == b"patch.socket.dev"),
305+
let lock = hosted(&original);
306+
assert_eq!(
307+
BunLockb::parse(&lock).unwrap().normalized_flags(),
308+
crate::vendor::bun_lockb::NORMALIZED_WORKSPACE,
285309
"{dir}"
286310
);
311+
let (outcome, after) = run(&lock, &vendor_opts()).await;
312+
std::env::remove_var("SOCKET_NPM_REGISTRY");
313+
let why: Vec<&str> = outcome.refused().map(|(_, why)| why).collect();
314+
assert_eq!(why.len(), 1, "{dir}: {why:?}");
315+
assert!(why[0].contains("workspace dependency behaviors"), "{why:?}");
316+
assert!(why[0].contains("git checkout -- bun.lockb"), "{why:?}");
317+
assert!(outcome.reverted_files.is_empty(), "{dir}");
318+
assert!(after == lock, "{dir}");
287319
}
288320
}
289321

322+
/// A promoted format-1 lock whose mark this codec did not write exactly
323+
/// (here: an unknown flag bit) is refused, never demoted to a guess.
324+
#[tokio::test]
325+
#[serial_test::serial]
326+
async fn promoted_format_1_lock_that_does_not_invert_refuses() {
327+
let original = fixture("0.1.6");
328+
let upstream = minimist(&original);
329+
let server = registry(upstream.integrity.as_deref().unwrap()).await;
330+
std::env::set_var("SOCKET_NPM_REGISTRY", server.uri());
331+
let mut lock = hosted(&original);
332+
let count = u64::from_le_bytes(lock[86..94].try_into().unwrap()) as usize;
333+
let package_start = u64::from_le_bytes(lock[110..118].try_into().unwrap()) as usize;
334+
// The root resolution's flag byte (its last).
335+
let flags_at = package_start + count * 16 + 63;
336+
assert_eq!(lock[flags_at], crate::vendor::bun_lockb::NORMALIZED_FORMAT_1);
337+
lock[flags_at] |= 0x40;
338+
BunLockb::parse(&lock).unwrap().validate_mutation().unwrap();
339+
let (outcome, after) = run(&lock, &vendor_opts()).await;
340+
std::env::remove_var("SOCKET_NPM_REGISTRY");
341+
let why: Vec<&str> = outcome.refused().map(|(_, why)| why).collect();
342+
assert_eq!(why.len(), 1, "{why:?}");
343+
assert!(why[0].contains("format-1"), "{why:?}");
344+
assert!(why[0].contains("git checkout -- bun.lockb"), "{why:?}");
345+
assert!(after == lock);
346+
}
347+
290348
/// Without the vendor opt-in (the `rollback` posture), or offline,
291349
/// nothing is written and the refusal names the checkout remedy; a dry
292350
/// run resolves the restore and writes nothing.

0 commit comments

Comments
 (0)