Retry bulkcmd on transient failure rather than aborting the flash - #3
Retry bulkcmd on transient failure rather than aborting the flash#3bai-yi-bai wants to merge 1 commit into
Conversation
bulkcmd issued each command once and propagated any error to the caller, which aborts the entire flash. On the Spotify Car Thing (superbird) the USB burn link is unreliable: individual bulk commands intermittently time out or return `failed:` and then succeed when reissued. Because the send was single-shot, one such failure terminated the flash, observed most consistently while validating dtbo_a. Move the existing implementation to a private bulkcmd_once and have bulkcmd invoke it up to six times, with a 1500 ms delay between attempts, returning the final error only if every attempt fails. Behaviour on the success path is unchanged; a genuinely failing command still returns an error, after six attempts rather than one. Retries are logged at warn via the existing tracing integration.
|
I forgot to mention that on my deskthing-usb-host-mode repo that I instruct users to apply a patch which implements this retry change to the flashthing before building it. https://gitlab.com/baiyibai/deskthing-usb-host-mode/-/blob/main/install.sh?ref_type=heads#L20 I let AI write the patch on my repo, but I took the time to re-write the comment and craft this PR. |
|
i’m curious as to when this helps - in my experience (and when watching flashes with uart logs) if a bulkcmd fails the entire device is hung. i've not yet heard of anyone successfully retrying a failed bulkcmd. can you provide a bit more info? as a aside: i'll be releasing a new firmware image in the coming weeks on kernel 7.0 that should make it significantly easier to do usb host mode for your project. |
|
Hi, I can get a console log for you in a few days; I'll have to rebuild the stock flashthing. It will be great to see 7.0; who knows how many open CVEs are currently on these devices. Leaving them on 4.9 is a major risk. Plus, having a newer kernel should help with getting more usb network adapter drivers on them. |
|
i ai vibed this patch bishopdynamics/superbird-tool#18 it made things work on my machine tm when the original program nor this retry patch didnt work |
3 participants
Hi @JoeyEamigh,
First, thank you for all your work and for keeping the Car Thing alive.
I started hacking my Spotify Car Thing(s) over the past couple weeks and I've been posting my progress on the Thing Labs and DeskThing Discords. I'd be happy to connect there and share more about my future plans on how I want to improve DeskThing.
I'm contributing this PR because I experienced multiple failures while attempting to flash my devices. I tried all the advice: expensive cables, thunderbolt cables, cheap cables, hubs, no hubs, powered hubs, different systems, and different USB controllers. I know flash failures are a common problem regardless of the firmware being burned or code being used. For the NixOS flasher, I even dropped the block size down to 512 KB from 2,048 KB and enduring the long flash time. However, after lots of trial and error, I found that a simple retry routine eliminated the problem.
Summary
This change wraps
bulkcmdin a retry loop. Currently, each command is sent once and therefore any error propagates to the caller which causes the flash attempt to abort. I observed this behavior most frequently during the validating dtbo_a step, where one failure ended an otherwise-complete flash. Thebulkcmdcommand now calls abulkcmd_oncefunction up to six times with a 1500 ms delay between attempts. An error is only returned if every attempt fails.Adding retries eliminated the problem and saved me a lot of stress, hopefully others will benefit from this change.