Skip to content

store/remotestore: add remote store client + server support - #22

Merged
bradfitz merged 1 commit into
mainfrom
bradfitz/remotestore
Jun 12, 2026
Merged

bradfitz merged 1 commit into
mainfrom
bradfitz/remotestore

Conversation

@bradfitz

@bradfitz bradfitz commented Feb 15, 2026 •

Copy link
Copy Markdown
Member
Add an HTTP-based remote store backend so ephemeral VMs (e.g. Windows
CI with WinFSP) can use a persistent Linux server's git-backed gomodfs
as their storage backend. The ephemeral VM doesn't need internet access;
the persistent server fetches modules on demand from proxy.golang.org.

The protocol has three endpoints at the ModuleVersion level:

  GET /api/v1/modmap/{mod}/@v/{ver}  — LZ4-compressed JSON file metadata map
  GET /api/v1/file/{mod}/@v/{ver}?path=...  — raw file bytes
  GET /api/v1/meta/{mod}/@v/{ver}.{info,mod,ziphash}  — metadata files

New packages and files:

  store/remotestore/protocol.go   — shared types (ModVersionMap, FileMeta)
                                    and URL path parsing
  store/remotestore/handler.go    — server-side HTTP handler backed by an
                                    FSBackend interface (satisfied by *gomodfs.FS)
                                    with in-memory LZ4 caching per module version
  store/remotestore/remotestore.go — client-side store.Store that fetches
                                     modmaps over HTTP, derives directory structure
                                     locally, and serves Stat/Readdir with zero
                                     additional HTTP calls
  remotestore_test.go             — integration tests in the root package
  store/remotestore/remotestore_test.go — unit tests for buildDirMap and
                                          acceptsLZ4

Changes to existing code:

  gomodfs.go — add ReadOnly bool field to FS; when set, getZipRoot and
               getMetaFile return ErrCacheMiss instead of attempting
               downloads. Add exported GetZipRootOrDownload,
               GetMetaFileByExt, and GetStore helpers for the handler.
  cmd/gomodfs/gomodfs-main.go — add --remote-store=URL and
               --serve-store-api=ADDR flags.

Updates tailscale/corp#24037
Updates tailscale/corp#34812

@bradfitz
bradfitz requested a review from tomhjp February 15, 2026 19:56
@bradfitz
bradfitz force-pushed the bradfitz/remotestore branch from 7ecce7a to 8e2de76 Compare February 15, 2026 20:00
Add an HTTP-based remote store backend so ephemeral VMs (e.g. Windows
CI with WinFSP) can use a persistent Linux server's git-backed gomodfs
as their storage backend. The ephemeral VM doesn't need internet access;
the persistent server fetches modules on demand from proxy.golang.org.

The protocol has three endpoints at the ModuleVersion level:

  GET /api/v1/modmap/{mod}/@v/{ver}  — LZ4-compressed JSON file metadata map
  GET /api/v1/file/{mod}/@v/{ver}?path=...  — raw file bytes
  GET /api/v1/meta/{mod}/@v/{ver}.{info,mod,ziphash}  — metadata files

New packages and files:

  store/remotestore/protocol.go   — shared types (ModVersionMap, FileMeta)
                                    and URL path parsing
  store/remotestore/handler.go    — server-side HTTP handler backed by an
                                    FSBackend interface (satisfied by *gomodfs.FS)
                                    with in-memory LZ4 caching per module version
  store/remotestore/remotestore.go — client-side store.Store that fetches
                                     modmaps over HTTP, derives directory structure
                                     locally, and serves Stat/Readdir with zero
                                     additional HTTP calls
  remotestore_test.go             — integration tests in the root package
  store/remotestore/remotestore_test.go — unit tests for buildDirMap and
                                          acceptsLZ4

Changes to existing code:

  gomodfs.go — add ReadOnly bool field to FS; when set, getZipRoot and
               getMetaFile return ErrCacheMiss instead of attempting
               downloads. Add exported GetZipRootOrDownload,
               GetMetaFileByExt, and GetStore helpers for the handler.
  cmd/gomodfs/gomodfs-main.go — add --remote-store=URL and
               --serve-store-api=ADDR flags.

Updates tailscale/corp#24037
Updates tailscale/corp#34812

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
@bradfitz
bradfitz force-pushed the bradfitz/remotestore branch from 8e2de76 to 161f3ac Compare June 12, 2026 16:21
@bradfitz
bradfitz merged commit 131ad61 into main Jun 12, 2026
7 checks passed
@bradfitz
bradfitz deleted the bradfitz/remotestore branch June 12, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants