Repository navigation
pico2w: fix SD boot reset loop, add 288 MHz clock option, capture crash stack - #106
Merged
Merged
Conversation
Three fixes plus the investigation that produced them.
SD boot reset loop
------------------
Booting with a staged ROM that carried a rom_id reset the board forever.
It was not memory corruption: with no card, `embedded-sdmmc` takes ~17.1 s
to return CardNotFound (50 acquire retries x ~340 ms). The watchdog window
is 16 s, and the RP2350 counter is 24-bit MICROSECONDS so the window cannot
exceed 16.777 s. The call could never fit, and every boot reset mid-init,
leaving only WatchdogTimeout records and no fault.
* bound acquisition to 6 retries -> ~2.4 s with no card, ~0.8 s with one
* send the SD spec's 74 power-up clocks with CS deasserted; the crate
leaves this to the caller and we never did it
* skip the remaining boot SD reads once the card reports absent, instead
of paying the timeout twice more. The third one — the save-slot probe —
ran after the watchdog was re-armed and before the main loop's first
feed, so it was a guaranteed reset on its own.
Also feeds the watchdog during the ROM-list directory scan, which could
outlast the window on a slow card.
288 MHz clock option (oc-288)
-----------------------------
A frequency sweep by live PLL reprogramming put a failure cliff between
288 and 300 MHz: at 300 the SP-drift instrument logged 67.5 events per 1000
work-ticks, while 288/276/264/252/240 logged zero across 13,567 ticks. A
24-hour soak at 288 on the real firmware ran 3.56M ticks with zero crashes,
against ~43 predicted at the 300 MHz rate.
NOT a fix. The cliff is bracketed only to (288, 300], so 288 may have as
little as 0.3% margin, untested against temperature and part-to-part
spread, and the underlying defect still occurs at 150 MHz. 266 remains the
more conservative shipping point. Default is unchanged.
Crash record stack snapshot (schema v2)
---------------------------------------
WatchdogTimeout and wild-PC records carried no stack context, which is what
made them undiagnosable. Records now include six words from the pre-fault
SP. DMA channels 2-6 are dropped from the record to make room — a live read
of all 16 channels showed only ch0/ch1 are ever non-zero.
MPU hardening
-------------
Arm the RAM-resident .data code and thunks read-only, and core 0's stack
read-only to core 1.
Tests: 649 core + 191 pico2w pass. Verified on hardware: boots at 288 MHz,
mounts the card in 781 ms, loads a save state, enters the main loop at 3.8 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ab4PqF52wyVpp5S2Sg6RGK
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (73.58%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #106 +/- ##
==========================================
- Coverage 93.21% 93.06% -0.15%
==========================================
Files 82 83 +1
Lines 14055 14155 +100
==========================================
+ Hits 13102 13174 +72
- Misses 953 981 +28 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… clock profiles
Addresses three blocking defects found in review of the previous commit, plus
the cleanup items.
BLOCKING — the stack snapshot recorded the wrong bytes on every fault
--------------------------------------------------------------------
`commit_crash_and_reset` receives `sp_before_exception()`, which has ALREADY
added the frame size (32, or 104 for an FP-extended frame) and the STKALIGN
pad. The capture then added them a second time, so:
* the "stacked xPSR" probe at `sp + 0x1c` read an unrelated stack word, making
the +4 pad adjustment effectively random;
* the window landed 32 bytes past the return-address slot it exists to
capture — the feature never recorded what it was built for.
Fixed by passing `sp` through unchanged, which also removes an unguarded
dereference that ran inside the fault handler before any state was committed.
Renamed to `capture_stack_window` and corrected the bounds-check doc: the guard
clamps to physical SRAM, it does not verify the address is a stack.
BLOCKING — a corrupt save state destroyed the battery save
----------------------------------------------------------
`read_save_state` returns `Err` both when the card is absent AND when a
save-state file is present but fails validation (`InvalidValue`), or on
`OutOfMemory`. Classing every `Err` as "card absent" skipped the battery-save
read on a HEALTHY card, booted with blank cart RAM, and let the first flush
overwrite a good `.sav`.
Added `SdError::is_card_absent()`, true only for transport failures
(`DeviceError`/`NoSuchVolume`), and used it for the availability flag.
BLOCKING — the retry bound was narrower than documented
-------------------------------------------------------
`acquire_retries` bounds ONLY the CMD0 enter-SPI-mode loop; CMD8 and ACMD41 use
the crate's hardcoded 10_000-iteration budget. So the ~2.4 s figure holds for a
fully absent card, but a partially responsive one can still take ~7-8 s — inside
the 16 s window with ~2x margin, not the 6x the comment implied. Documented
honestly rather than silently relied upon.
Other fixes
-----------
* clock profiles: `TARGET_SYS_HZ` and `TARGET_CORE_VOLTAGE` were two cfg
cascades keyed on different feature sets and had already drifted — `oc-266`
selected 266 MHz but fell through to V1_30, NOT reproducing the "266 @
V1_25" point it is named for. Collapsed to one `ClockProfile` table so the
two cannot desync. Added a static assert that the watchdog window fits the
RP2350's 24-bit microsecond counter.
* SD mode pin (GP17) is now driven BEFORE the power-up clocks; it was set 31
lines after the first bus traffic this commit series introduced.
* removed `arm_core1_stack_fence()` (zero call sites) and the 25-line comment
describing a region it never armed; removed the unused `list_rom_page`.
* de-duplicated the MPU thunk-region arithmetic into `data_thunk_bounds()` /
`arm_ro_exec_region()`. Verified on hardware: both cores now report
identical bounds, which is the property the comment previously only
asserted.
* `stack_snapshot`: named indices, a static assert backing the padding claim,
and the documented limitation that a hardware watchdog reset has no capture
point (those records carry schema v2 with an all-zero window).
* decoder reports an uncaptured window as absent rather than six zeros.
* the capture base is logged instead of discarded, so the window has an anchor.
* apu: extracted `trusted_capacity()`; trimmed dated incident narrative.
Tests: 649 core + 194 pico2w. Three new ones cover the format change that had
none — v1 records must not reinterpret DMA bytes as a stack snapshot, the
snapshot must survive a round trip with distinct words, and the persisted byte
offsets are pinned absolutely.
Verified on hardware at 288 MHz: mounts, loads a save state, main loop at 3.8 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ab4PqF52wyVpp5S2Sg6RGK
The three structural findings from review. No behaviour change intended; each
is verified on hardware below.
Card-present state belongs to SdManager
---------------------------------------
`main` tracked a local `sd_available`, set from one `is_card_absent()` check at
the single boot site that happened to think of it. The object that knows
whether the card answered is the manager, so every other caller either
re-derived it or silently skipped the check.
`SdManager` now latches `card_absent` itself, in `open_volume_tracked()` —
every public operation opens volume 0 first, which makes it the one place an
absent card is observable regardless of which operation asked. Callers get
`card_present()` and `with_card()`.
The latch is deliberately one-way and documented as such: this type sees no
insertion event, and a re-seated card is not detectable without a full
re-acquire.
Watchdog bound once instead of threaded through 11 signatures
--------------------------------------------------------------
`&mut Watchdog` was a parameter on 11 functions across four modules, several of
which only passed it along. It is a hardware singleton with one owner and one
configured window, and every feed site wants the same thing, so the parameter
carried no information — but it did carry the window, and call sites had
already drifted to bare literals (`16_000`, `5_000`) with no relation to
`WATCHDOG_WINDOW_MS`.
New `wdt` module owns it behind a critical-section mutex:
* `wdt::feed()` — re-arm with the configured window; a caller cannot pick a
wrong one
* `wdt::feed_for(d)` — the two genuinely different windows in the flash
staging path, now visibly the exceptions
* `wdt::force_reset()` — reboot into a newly staged ROM
`pause_on_debug(true)` moves into `wdt::init` next to the `start()` it
compensates for; it was previously a loose call in `main` that a reordering
could have separated from its reason.
main() split 461 → 311 lines
----------------------------
Extracted `arm_core0_memory_guards`, `init_heap`, `init_sd_card`,
`restore_saves_on_boot`. Each has a name and a doc comment stating the
ordering constraint that kept it where it was.
`init_sd_card` returns the GP17 mode pin alongside the manager, and says why:
dropping that `Output` releases the pin and the card leaves SPI mode. That is
the one hazard in this extraction — the guard was a `let _sd_mode` binding
whose lifetime was load-bearing.
Verified
--------
649 core + 194 pico2w tests; clippy clean on the touched code; all four clock
profiles build.
On hardware at 288 MHz: boots, both cores arm identical MPU bounds, mounts the
card, and logs "save state loaded on boot" from inside the extracted
`restore_saves_on_boot` — so the SD path runs through the new latch and reports
the card present. Ran 150 s with no reset, which exercises `wdt::init` and the
main loop's `wdt::feed()`: a broken feed path resets within ~10 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ab4PqF52wyVpp5S2Sg6RGK
…, latch
Two silent defects, both introduced by the previous two commits and both
invisible to their hardware verification, plus the unanimous smaller findings.
Orphaned attributes in mpu.rs
-----------------------------
Extracting `data_thunk_bounds` inserted it BETWEEN `setup_core0_data_mpu`'s
attribute block and its `fn` line, so `#[inline(never)]` and the whole region-0
doc comment silently reattached to the helper. Confirmed in the ELF: the
3-line bounds helper kept an out-of-line symbol while the MPU-arming function
was inlined away. In a codebase whose debugging strategy is PC-to-symbol
attribution of RAM-code corruption, that is a regression in the instrument.
Restored; the symbols now flip back the correct way.
Nothing caught this — it builds on both targets and the lint that would fire
(`empty_line_after_outer_attr`) is nursery.
Core 1's MPU region overlapped the tail of `.uninit`
-----------------------------------------------------
The previous commit added a comment and a `const _: () = assert!(...)` claiming
`CRASH_STACK_SNAP`'s padding fenced the "last `.uninit` object falls inside core
1's MPU region" hazard. It did not. That array is not last:
20066a80 CRASH_STACK_SNAP (padded, not last)
20066ac0 DMA_CRASH_SNAPSHOT (last, unpadded, ends 20066b08)
20066b08 _stack_end / __euninit
`_stack_end & !0x1F` = 0x20066b00, so the final two words of
`DMA_CRASH_SNAPSHOT` were privileged read-only for core 1 — and
`capture_dma_snapshot()` writes all 18 unconditionally from the panic handler,
which runs in thread mode with the MPU live (unlike the HardFault path, where
MPU_CTRL.HFNMIENA=0 disables it). A core-1 panic therefore took a DACCVIOL that
escalated to HardFault, replacing the intended `CrashKind::Panic` record and
losing its file/line — on exactly the fault class under investigation.
Fixed at the source rather than by padding: `setup_core1_mpu` now rounds the
region base UP to the MPU granule, so it can never start below `__euninit` and
no `.uninit` object is reachable by it. Verified on hardware — the base moved
0x20066b00 -> 0x20066b20. Which object lands last is a link-order accident, so
padding could never have been the fix. The false assert is replaced with one
that checks something true.
Also corrected in that file: the `# Safety` block still claimed `base` is
"range-checked against the core-0 stack", contradicting the inline comment six
lines below it that correctly calls it a physical-SRAM clamp; and the window
only begins at the return slot on the FAULT path — on the panic path `base` is
the MSP inside the handler, so it starts on handler locals.
Card-presence latch: callers no longer decide
----------------------------------------------
`card_present()`/`with_card()` handed the decision back to call sites, produced
`Option<Option<_>>` and a `.flatten()` that erased "no card" from "no battery
file", and left every other SD site to wonder whether it should have checked.
The latch is now private and the READ paths consult it themselves —
`read_battery_save`/`read_save_state` return `Ok(None)`, `save_state_exists`
returns `Ok(false)`. Writes deliberately still attempt and report a real error:
a silent `Ok` on a write is data loss, not a fast path.
`is_card_absent` no longer matches `NoSuchVolume`. `open_raw_volume` returns it
only for `VolumeIdx >= 4` and this code always opens volume 0, so the arm was
unreachable; an unformatted card returns `FormatError`, correctly excluded.
Watchdog
--------
`feed_for(16_000)` was `WATCHDOG_WINDOW_MS` restated as a literal — the exact
drift the `wdt` module exists to prevent, in the one place it was aimed at. Now
`feed()`. Only `5_000` is a genuine exception, so the commit-message claim of
"two genuinely different windows" was wrong.
`force_reset()` could hang: if the watchdog was never published the `if let`
fell through and a `-> !` function spun forever. Falls back to
`SCB::sys_reset()`. Threading `&mut Watchdog` made that unrepresentable; a
global has to handle it explicitly. `feed()`'s doc also misattributed its own
safety — it is the missing peripheral that makes it silent, not the zero
window, which would write LOAD=0 and reset on the next tick.
Other
-----
* `init_sd_card` returns `SdBus` instead of `(PicoSdMgr, Output)`. The mode
pin's lifetime is now a type invariant instead of a `let _sd_mode` binding
whose leading underscore invites the cleanup that breaks it.
* comment de-duplication: the save-precedence rule was stated three times in
one 55-line function, and a six-line comment restated its own first
sentence verbatim (a paste artifact from the extraction).
* removed the stale `list_rom_page` doc left on `list_rom_page_with`.
Verified
--------
649 core + 194 pico2w tests; all four clock profiles build; lib clippy clean.
On hardware at 288 MHz: boots, core 1 region base 0x20066b20 (above
`__euninit`), both cores identical `.data` bounds, save state loaded from the
extracted restore path, main loop at 3.8 s, 150 s with no reset.
NOT fixed, pre-existing and out of scope: `cargo clippy --bins` fails on a
deny-by-default `modulo_one` at state/rom_list.rs:204 (from dda1ba8). The
previous commit's "clippy clean" claim was based on linting only the lib
target, which is why it read clean; the binary target does not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ab4PqF52wyVpp5S2Sg6RGK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three fixes, plus the investigation that produced them. The headline is a boot reset loop that turned out not to be memory corruption at all.
A round of review found three blocking defects in the first commit; all are fixed in
8ce3d1aand described at the bottom. Claims below have been corrected where review showed they were overstated.SD boot reset loop
Booting with a staged ROM carrying a
rom_idreset the board forever, and it looked like the long-running core-1 corruption bug. It isn't.With no card present,
embedded-sdmmctakes 17.1 s to returnCardNotFound(50 acquire retries × ~340 ms). The watchdog window is 16 s, and the RP2350 counter is 24-bit microseconds — so the window physically cannot exceed 16.777 s. The call could never fit. Every boot reset mid-init, leaving onlyWatchdogTimeoutrecords and no fault.What located it was PC sampling rather than the crash records: 24 halts gave 13 in
spi::blocking_transfer, 4 inblock_for, 1 inspi::flush— 75% in the SD path.embedded-sdmmcleaves this to the caller ("some cards do, some don't") and we never did itAlso feeds the watchdog during the ROM-list directory scan, which could outlast the window on a slow card.
288 MHz clock option (
oc-288)A frequency sweep by live PLL reprogramming (no rebuild, so no layout confound) put a failure cliff between 288 and 300 MHz:
A 24-hour soak at 288 on the real firmware ran 3.56M ticks with zero crashes, against ~43 predicted at the 300 MHz rate.
This is not a fix. The cliff is bracketed only to (288, 300], so 288 may have as little as 0.3% margin — untested against temperature and part-to-part spread — and the underlying defect still occurs at 150 MHz. 266 MHz remains the more conservative shipping point (8–11% below the cliff, and already 9 h clean at lower voltage). The default is unchanged; this only adds the option.
Crash record stack snapshot (schema v2)
Wild-PC and fault records carried no stack context, which is exactly what made them undiagnosable. Records now include six words from the pre-fault SP. DMA channels 2–6 are dropped to make room — a live read of all 16 showed only ch0/ch1 are ever non-zero.
MPU hardening
Arm the RAM-resident
.datacode and thunks read-only, and core 0's stack read-only to core 1.Review fixes (
8ce3d1a)Three blocking defects, all in the first commit:
sp_before_exception(), which has already added the frame size and STKALIGN pad; the capture added them a second time. The "stacked xPSR" probe read an unrelated word (making the pad adjustment random) and the window landed 32 bytes past the return-address slot it exists to capture. The feature never recorded what it was built for.read_save_statereturnsErrboth when the card is absent and when a save-state file fails validation. Classing everyErras "card absent" skipped the battery-save read on a healthy card, booted with blank cart RAM, and let the first flush overwrite a good.sav. AddedSdError::is_card_absent(), true only for transport failures.Also in that commit: the GP17 SD-mode pin is now driven before the first bus traffic (it was set 31 lines after);
TARGET_SYS_HZandTARGET_CORE_VOLTAGEwere twocfgcascades keyed on different feature sets that had already drifted (oc-266selected 266 MHz but fell through to V1_30, not reproducing the "266 @ V1_25" point it is named for) and are now oneClockProfiletable; the duplicated MPU thunk arithmetic is shared, verified on hardware by both cores reporting identical bounds; dead code removed (arm_core1_stack_fence,list_rom_page).One change lands outside
platform/pico2w:core/src/cpu/peripheral/apu.rsextracts atrusted_capacity()helper and trims dated incident narrative from comments. Behaviour is unchanged and it is covered by the existing 649 core tests, but it is shared code and worth a look.Structural refactors (
06aad77)The three design findings from review. No behaviour change intended.
Card-present state moved into
SdManager.maintracked a localsd_availableset from oneis_card_absent()check at the single boot site that happened to think of it; every other caller either re-derived it or skipped it. The manager now latchescard_absentitself inopen_volume_tracked()— every public operation opens volume 0 first, making it the one place an absent card is observable regardless of which operation asked. Callers getcard_present()andwith_card(). The latch is deliberately one-way and documented as such: this type sees no insertion event.Watchdog bound once instead of threaded through 11 signatures.
&mut Watchdogwas a parameter on 11 functions across four modules, several of which only passed it along. It is a hardware singleton with one owner and one window, so the parameter carried no information — but it did carry the window, and call sites had already drifted to bare literals (16_000,5_000) unrelated toWATCHDOG_WINDOW_MS. The newwdtmodule owns it behind a critical-section mutex and exposesfeed()(configured window, impossible to get wrong),feed_for(d)(the two genuinely different windows in the flash-staging path, now visibly exceptional), andforce_reset().pause_on_debug(true)moves intowdt::initnext to thestart()it compensates for, instead of being a loose call a reordering could separate from its reason.main()split, 461 → 311 lines. Extractedarm_core0_memory_guards,init_heap,init_sd_card, andrestore_saves_on_boot, each documenting the ordering constraint that kept it inline.init_sd_cardreturns the GP17 mode pin alongside the manager and says why: dropping thatOutputreleases the pin and the card leaves SPI mode — the guard was alet _sd_modebinding whose lifetime was load-bearing, and it is the one real hazard in the extraction.Second review round (
a7479d7)A second three-reviewer pass over
8ce3d1aand06aad77found no blocking runtime defect — the boot-save path was proven equivalent case-by-case, including the corrupt-save-state and OOM cases the branch exists to fix. It did find two silent defects introduced by those commits, both invisible to their own hardware verification.Orphaned attributes in
mpu.rs. Extractingdata_thunk_boundsinserted it betweensetup_core0_data_mpu's attribute block and itsfnline, so#[inline(never)]and the region-0 doc silently reattached to the helper. Confirmed in the ELF: the 3-line bounds helper kept an out-of-line symbol while the MPU-arming function was inlined away — a real regression in a codebase whose debugging strategy is PC-to-symbol attribution. Nothing caught it; it builds on both targets and the lint that would fire is nursery.Core 1's MPU region overlapped the tail of
.uninit.06aad77added a comment and a static assert claimingCRASH_STACK_SNAP's padding fenced this hazard. It didn't — that array isn't last:_stack_end & !0x1F=0x20066b00, so the last two words ofDMA_CRASH_SNAPSHOTwere privileged read-only for core 1 — andcapture_dma_snapshot()writes all 18 unconditionally from the panic handler, which runs in thread mode with the MPU live (the HardFault path is safe only becauseMPU_CTRL.HFNMIENA=0). A core-1 panic took a DACCVIOL that escalated to HardFault, replacing the intendedCrashKind::Panicrecord and losing its file/line — on exactly the fault class under investigation.Fixed at the source, not by padding: the region base now rounds up to the MPU granule, so it can never start below
__euninit. Verified on hardware, base moved0x20066b00→0x20066b20. Which object lands last is a link-order accident, so padding could never have been the fix; the false assert is replaced with one that checks something true.Smaller, unanimous:
feed_for(16_000)wasWATCHDOG_WINDOW_MSrestated as a literal — the exact driftwdtexists to prevent, in the one place it was aimed at (nowfeed(), so only5_000remains a genuine exception).force_reset()could hang forever if the watchdog was never published; it now falls back toSCB::sys_reset(). The card-presence latch is private again and the read paths consult it themselves, removing theOption<Option<_>>.flatten()that erased "no card" from "no battery file"; writes deliberately still attempt, because a silentOkon a write is data loss, not a fast path.is_card_absentno longer matchesNoSuchVolume, which was unreachable forVolumeIdx(0).init_sd_cardreturns a namedSdBusso the mode pin's lifetime is a type invariant rather than alet _sd_modebinding.Deferred by agreement: a
Devicesstruct to takemain()from 312 lines toward ~15, and ungatingsd.rsfrom ARM so the latch and error-classing become host-testable. Both are redesigns rather than defect fixes and belong in their own PR.Testing
oc-266,oc-288,oc-300); lib clippy cleancargo clippy --binsfails on a deny-by-defaultmodulo_oneatstate/rom_list.rs:204— pre-existing (fromdda1ba8), not introduced here, butstate/is touched by this branch and the crate-level clippy invocation does not pass. Worth a one-line follow-up.save state loaded on bootis logged from inside the extractedrestore_saves_on_boot, so the SD path runs through the new latch and reports the card present; and the board held 150 s with no reset, which exerciseswdt::initplus the main loop'swdt::feed()— a broken feed path resets within ~10 sNotes for review
docs/investigations/sp-drift-root-cause.mdis the distilled write-up;pico-crash-current-findings.mdis the raw log, included to match existing practice in that directory.heartbeatandfpuwere dropped too — their wiring lived in the revertedmulticore.rs, so keeping them would have shipped unwired dead code. They can return as a focused follow-up if wanted.cyw43-firmware/*.binare symlinks into a local cargo checkout and are intentionally not part of this commit. Note they are committed pointing at embassy rev6b01532, which is stale — a fresh checkout resolves to a different rev and the links dangle, so the WiFi build needs them replaced with real copies. Worth fixing separately.06aad77, described above.🤖 Generated with Claude Code
https://claude.ai/code/session_01Ab4PqF52wyVpp5S2Sg6RGK