perf(zccache): firmware compiles don't hit zccache cross-project even when args are workspace-relative

[zackees]

Context

Follow-up to #952 / PR #965. After making ESP32 fw-libs compiles workspace-relative (source/-o/-I relativized, cwd = workspace — the same treatment sketch/core already get), the #942 harness shows only a modest warm improvement (fw-libs 185s → 141s), not the expected drop to seconds.

Measuring a clean cold→warm in one container run (fresh ~/.fbuild before cold, fresh project copy for warm), none of the firmware phases hit the cache cross-project:

• warm compile-sketch ≈ 80 s (not a hit)
• warm compile-core-variant ≈ 10 s (not a hit)
• warm fw-libs ≈ 141 s (partial at best)

So the premise "sketch/core hit cross-project, make fw-libs like them" is only half true: sketch/core are workspace-relative but still don't reliably hit zccache across two project directories in a clean run. Workspace-relative args are necessary but not sufficient.

Investigate

• Does the embedded zccache key include the absolute compile CWD (or another project-specific input) in addition to the relativized args? If so, cross-project hits are impossible regardless of arg relativization.
• Does the embedded zccache persist its object store across builds within a container run at all, and where (~/.fbuild/<mode>/zccache/ vs the soldr-managed store)?
• Instrument the per-TU zccache lookup key for one sketch source across /tmp/nds (cold) and /tmp/nds2 (warm) and diff the exact differing component.

Acceptance criteria

• A clean cold→warm run shows warm compile-sketch, compile-core-variant, and fw-libs hitting zccache (sub-second / near-copy).
• Root cause (the project-specific key input) documented.
• Warm total drops toward config-parse + link once the compile phases hit.

Decisions

• Priority: P2 — this is the actual lever for the meta: profiling-driven build performance burndown (Docker Linux, NightDriverStrip, cold vs hot cache) #942 warm-scenario win; PR perf(build): compile ESP32 framework libs workspace-relative (cross-project cache stability, #952) #965 is the necessary precondition (workspace-relative args) but the payoff is gated here.

Part of #942. Follow-up to #952 / PR #965.

[zackees]

Empirical update — env-var approach does NOT work for the embedded service

Tried setting ZCCACHE_WORKTREE_ROOT=<compile_cwd> in the per-compile env (both the embedded svc.compile path for sketch/core and the zccache wrap CLI path for fw-libs). Measured in the harness: no change — a clean cold→warm still shows fw-libs≈161s, compile-sketch≈95s, compile-core-variant≈11s (no cross-project hits). Reverted.

Root cause of the non-effect: the embedded ZccacheService computes its cache key from its own context/config, not from the per-compile client env. The client env is forwarded to the spawned compiler subprocess, but does not reach resolve_worktree_root / key-root computation. So an env var on the compile request is inert for keying. (The earlier analysis conflated the CLI-wrap path, which reads process env, with the in-process service path.)

What the fix actually needs

The embedded ZccacheService::compile API must accept a per-request worktree_root (or key_root) so fbuild can pass each build's workspace explicitly — a daemon-level env var can't work because one long-lived daemon serves many project dirs. That is a zccache-crate change (add the field to the compile request + thread it into key computation), then fbuild passes compile_cwd through it.

Until then, PR #965's workspace-relative arg alignment is the extent of the fbuild-side improvement (warm fw-libs 185s→~141s). The sub-second warm win is gated on the zccache API change.

Decisions

• Priority stays P2, but now cross-repo — needs a change in the zccache crate (zackees/zccache) first, then a small fbuild follow-up to pass the per-request worktree root.

[zackees]

PR #985 makes ESP32 sketch + core compiles hit zccache cross-project. Root cause was two fbuild-side determinism leaks embedding the absolute project path into each TU's context key: (1) the object filename's hash suffix was derived from the absolute source path (so the compiler's -o arg differed per checkout), and (2) library include order came from unsorted read_dir. Both fixed to be project-workspace-relative / sorted.

Measured (clean cold /tmp/nds → warm fresh /tmp/nds2): warm compile-sketch 94.5 s → 0.73 s, compile-core-variant 12.9 s → 0.55 s — DIAG confirms identical context key + verdict=Hit across checkouts.

Framework-lib (fw-libs) compiles still miss cross-project via a separate non-arg input on the zccache wrap CLI path (args byte-identical yet context key differs) — split to #986. #966 stays open until #986 lands the fw-libs hit.

[zackees]

Fully resolved. #985 fixed sketch+core cross-project caching (object-filename + library-order determinism); #987 (#986) routed fw-libs through the embedded zccache service. All ESP32 firmware compile phases now hit zccache cross-project: warm sketch 0.8 s, core 0.5 s, fw-libs 2.2 s (from 94.5 s / 12.9 s / 152 s). No zccache-side change was needed — the mechanism worked; the leaks were all fbuild-side.