Skip to content

fix(bluetooth): power off adapters via BlueZ before rfkill block to prevent PCIe driver lockup - #380

Open
JJRPF wants to merge 1 commit into
omacom:quattrofrom
JJRPF:fix/bluetooth-graceful-poweroff
Open

JJRPF wants to merge 1 commit into
omacom:quattrofrom
JJRPF:fix/bluetooth-graceful-poweroff

Conversation

@JJRPF

@JJRPF JJRPF commented Sep 9, 2026

Copy link
Copy Markdown

Summary

Powers down Bluetooth adapters through BlueZ before applying rfkill block bluetooth.

This resolves an issue on Apple Silicon Macs where toggling Bluetooth off via the menu/bar wedged the Broadcom PCIe controller (hci_bcm4377), causing 10 transfer ring destruction timeouts, leaving the controller dead, and preventing Bluetooth from turning back on without a manual driver rebind or reboot.

Upstream kernel bug report: AsahiLinux/linux#609


The Problem on Apple Silicon Macs

On Apple Silicon MacBooks (M-series running hci_bcm4377 with Broadcom BCM4377/BCM4378/BCM4387/BCM4388 chips), calling rfkill block bluetooth while the radio is actively scanning or exchanging data triggers a driver/DMA hang:

  1. Opening the Omarchy Bluetooth panel starts continuous discovery scanning (discoveryRetry in Panel.qml), keeping LE inquiry DMA queues active.
  2. Clicking the toggle switch calls omarchy-bluetooth-power off, which previously ran rfkill block bluetooth directly without asking BlueZ to power down first.
  3. The kernel's hci_rfkill_set_block immediately attempts an uncoordinated power-down (hci_dev_do_poweroff). Because the Broadcom PCIe hardware has active DMA transfers in flight, it times out on HCI opcodes (command 0x0c01 tx timeout, -110).
  4. Fallback teardown (bcm4377_hci_close) then attempts to destroy the 6 transfer rings and 4 completion rings. Because the controller firmware is locked up, every ring destruction times out (1s each, stalling kernel workqueues for 10 seconds):
    Bluetooth: hci0: Error when powering off device on rfkill (-110)
    hci_bcm4377 0000:01:00.1: failed to destroy transfer ring 6
    ...
    hci_bcm4377 0000:01:00.1: failed to destroy completion ring 1
    
  5. Because this teardown occurred inside the intentional close path, the kernel core marks the device closed without triggering an automatic device reset.
  6. When the user clicks the toggle to turn Bluetooth back on (omarchy-bluetooth-power on), rfkill unblock bluetooth succeeds, but bluetoothctl power on fails completely:
    bluetoothd: Failed to set mode: Failed (0x03)
    bluetoothctl: Failed to set power on: org.bluez.Error.Failed
    
  7. At this point, the controller is wedged. Only an unbind/bind of the PCIe driver via sysfs (echo 0000:01:00.1 > .../unbind && .../bind) or a full system reboot can recover the device.

Solution

  1. Introduce a power_off() helper in bin/omarchy-bluetooth-power:
    • Iterates through discovered controllers (controllers()) and requests a clean power off through BlueZ before cutting radio power.
    • Also executes a direct timeout 2s bluetoothctl power off for the default controller.
    • Uses timeout 2s guards so D-Bus hiccups can never block the script.
    • Applies rfkill block bluetooth afterward, preserving Omarchy's design where the soft block persists state across reboots via systemd-rfkill.
  2. Update test/shell.d/bluetooth-test.sh:
    • Updates the test assertion from expecting power off to be absent to verifying that controllers are cleanly powered down before the rfkill block.

By stopping discovery and draining DMA queues via BlueZ before rfkill block is applied, the kernel driver powers off the device cleanly with zero timeouts and zero ring destruction failures.


Verification

Tested on an Apple MacBook Pro (14-inch, M2 Pro, Mac14,9) running Omarchy on Linux 7.1.6-asahi with Broadcom BCM4388 (hci_bcm4377):

  1. Before fix: Opening the Bluetooth panel (triggering active LE scan) and running omarchy-bluetooth-power off reproduced the exact 10-ring teardown failure in dmesg, leaving hci0 in a state where omarchy-bluetooth-power on returned org.bluez.Error.Failed (0x03).
  2. After fix: Tested repeated omarchy-bluetooth-power off and omarchy-bluetooth-power on cycles while actively connected to Sennheiser MOMENTUM 4 headphones and scanning:
    • Paired audio devices disconnected gracefully.
    • dmesg confirmed zero HCI command timeouts and zero transfer ring destruction failures.
    • omarchy-bluetooth-power on powered on immediately, and headphones reconnected automatically without intervention.
  3. Unit tests: bash test/shell.d/bluetooth-test.sh passes all 47 assertions.

@JJRPF
JJRPF force-pushed the fix/bluetooth-graceful-poweroff branch from 432e806 to 5e6799a Compare September 9, 2026 11:40
@JJRPF

JJRPF commented Sep 9, 2026

Copy link
Copy Markdown
Author

Update: Rebased onto quattro following the merge of #377. All 4 checks (including Install VM) are now passing.

Previous CI status & isolation test (resolved)

Note for reviewers on CI status:

All checks exercising the code in this PR are 100% green:

  • CI/Syntax + shellcheck: PASSED
  • CI/test/all: PASSED (all 256 test suites pass, including the updated Bluetooth power-down tests)
  • CI/tests/all: PASSED

Isolating the Install VM Failure

To verify whether the single failing check (Install VM) was related to this PR, I ran an isolation test:

  1. Took the untouched upstream quattro branch (commit 291a6989) with zero code changes.
  2. Ran the exact same Install VM workflow in an isolated run on my fork:
    JJRPF/omarchy-mac Run #34405569067
  3. It failed identically:
    omarchy/hyprtoolkit installed=0.5.4-6 expected=0.5.4-5.1
    

Root Cause

  • Arch Linux ARM recently pushed hyprtoolkit 0.5.4-6 to upstream [extra], while Omarchy edge hosts 0.5.4-5.1.
  • During install.sh, pacman -Syu pulls the higher version 0.5.4-6, tripping the harness version check.
  • This is currently affecting all runs on quattro, including upstream's own scheduled run:
    omacom/omarchy-mac Run #34336377946

This package conflict is tracked and addressed in #377 / #379. Once either lands on quattro, I will gladly rebase to turn Install VM green here as well.

@JJRPF
JJRPF force-pushed the fix/bluetooth-graceful-poweroff branch from 5e6799a to 40c0ac2 Compare September 10, 2026 11:07
@duketopceo

Copy link
Copy Markdown

Verified on live hardware: MacBookPro18,1 with the BCM4387 (hci_bcm4377, pci 14e4:5f71) — one of the exact chips this targets.

Test: held an active LE discovery scan open (bluetoothctl scan on, Discovering: yes — the race window the bug needs), then ran the patched omarchy-bluetooth-power offon cycle:

  • Off: Powered: no, Soft blocked: yes, discovery stopped cleanly
  • On: Powered: yes, unblocked, controller fully back
  • dmesg through the whole cycle: zero HCI timeouts, zero transfer-ring destruction messages — just routine systemd-rfkill audit lines

The BlueZ power off first drains the scan/DMA queues before rfkill block cuts power, which is exactly the ordering fix the wedge needs. Also ran test/shell.d/bluetooth-test.sh — all cases pass.

(Didn't deliberately re-run the un-patched path under scan — a successful repro would leave the controller needing a driver rebind I can't sudo. The clean patched cycle on the right hardware is the evidence that matters anyway.)

@malik-na

Copy link
Copy Markdown
Member

Validated head 40c0ac2196c6862fee7b459f244e3ea33e5b4df4 against freshly fetched quattro 3246b6b4d84fd5339b4c5e8576a23d5199f7087a. The clean local merge has tree 7bcb3d26b81615a7fd3b40514ddfa093a6612f04.

Reproduced bug — P2: power-off submission is not a completion barrier

At bin/omarchy-bluetooth-power:73-78, piped interactive bluetoothctl exits on stdin EOF before its asynchronous power-off completes. The final noninteractive command waits only for the default controller, so it does not synchronize the preceding requests to other controllers.

I reproduced this using the installed BlueZ 5.87-2 CLI, the unchanged candidate helper, a private two-adapter D-Bus service, and an inert rfkill logger. Both fake adapters begin Powered=true. The default setter completes after 50ms and the secondary after 1000ms; the fake property changes only in the completion callback, immediately before the reply.

  • Piped secondary-controller command: CLI exits zero at 0.0053s; property changes/reply at 1.0047s.
  • Actual candidate: rfkill logger runs at 0.1078s; wrapper exits zero at 0.1130s; secondary property changes/reply at 1.0242s.
  • Baseline: calls rfkill directly, without graceful power-off attempts.

All four bounded experiments completed without harness errors. This demonstrates a missing completion barrier in the new implementation, not a newly reproduced kernel hang or worse hardware behavior than baseline. The included test checks command presence and misses this asynchronous case.

Before merging, observe bounded completion or Powered=false for each intended controller, explicitly handle failure/timeouts, and add delayed-secondary-controller coverage. Waiting for the current piped process alone is insufficient.

Verified pass

  • Baseline and candidate each pass 23 focused power-control cases plus 48 Bluetooth assertions, including real two-second timeouts of harmless stub processes, multi-controller ordering, failure paths, and rfkill error propagation. Syntax and diff checks pass. These synchronous fixture passes do not clear the asynchronous finding.
  • Clean merge; all four recorded hosted checks succeeded, including ARM installation. Those hosted checks date to September 10 and do not execute this synthetic merged tree.
  • Current open-PR path inventory identifies only Avoid connecting after Bluetooth pair failures #383 sharing bluetooth-test.sh. The previously tested combined tree retains both assertion sets and the identical fix(bluetooth): power off adapters via BlueZ before rfkill block to prevent PCIe driver lockup #380 power-helper blob. This is compatibility evidence, not approval of either PR's hardware claims.
  • Runtime-only change: no migration is needed. Existing installs use the new behavior when the helper is updated; new installs use the same helper.

Remaining coverage gaps

No live radio/rfkill operations were performed. Physical shutdown, DMA draining, active discovery/audio, lockup prevention, recovery and reboot persistence remain unverified; phone testing is paused. No full broad suite was rerun for this two-file change; ShellCheck is unavailable locally. GitHub reports MERGEABLE / BLOCKED with REVIEW_REQUIRED.

Verdict: FAIL — I do not recommend merging this head until the completion-ordering defect is addressed. No source correction was applied during this review.

@JJRPF
JJRPF marked this pull request as draft September 12, 2026 15:16
@JJRPF
JJRPF force-pushed the fix/bluetooth-graceful-poweroff branch from 40c0ac2 to 448068a Compare September 12, 2026 18:10
@JJRPF

JJRPF commented Sep 12, 2026

Copy link
Copy Markdown
Author

Thanks @malik-na for the sharp review and the reproducible test case! The asynchronous completion race on secondary controllers is completely valid, and I've addressed all three requirements:

Summary of Updates

  1. Bounded Completion Barrier (wait_unpowered):

    • Added wait_unpowered() in bin/omarchy-bluetooth-power modeled after wait_powered(), using the shared POWER_WAIT_SECONDS deadline.
    • In power_off(), after submitting power-down commands to all controllers, wait_unpowered() polls until all controllers reach Powered: no before rfkill block bluetooth is executed.
  2. Explicit Failure & Timeout Handling:

    • If any controller does not reach unpowered state within POWER_WAIT_SECONDS:
      • Emits a warning to stderr identifying the stuck controller (omarchy-bluetooth-power: warning: controller <mac> did not power down cleanly).
      • Applies rfkill block bluetooth as a safety fallback so radio power is never left on.
      • Returns non-zero status (1).
  3. Delayed Secondary Controller & Timeout Test Coverage:

    • Enhanced mock_bin/bluetoothctl in test/shell.d/bluetooth-test.sh to parse stdin for piped interactive commands (select <mac> and power off) and track per-controller power states.
    • Added test coverage verifying that omarchy-bluetooth-power off polls and waits for a delayed secondary controller to settle to Powered: no before rfkill block bluetooth is logged.
    • Added test coverage verifying that a stuck controller produces a non-zero exit code, emits the warning to stderr, and still applies the fallback rfkill block.
    • Verified that mutating power_off to skip wait_unpowered immediately fails the delayed secondary controller test.

Branch has been rebased cleanly onto latest quattro (3246b6b4) and all local tests pass.

@JJRPF
JJRPF marked this pull request as ready for review September 12, 2026 18:14
@wesleygrimes wesleygrimes added the cherry-pick-later Port to omacom/omarchy after the overlay merges; match mainline. label Sep 12, 2026
On Apple Silicon Macs (BCM4377/BCM4388 driven by hci_bcm4377), calling
rfkill block bluetooth directly while discovery or connection is active
causes HCI command timeouts (-110) followed by 10 transfer and completion
ring destruction timeouts. The controller is left in an unrecoverable state
where bluetoothctl power on fails until an unbind/bind or reboot.

Gracefully powering off controllers through BlueZ first stops discovery
and drains DMA queues cleanly, preventing driver timeouts while preserving
the rfkill soft block for reboot persistence.

Enforce a bounded completion barrier via wait_unpowered to ensure all
controllers (including secondary controllers submitted asynchronously)
reach Powered: no before applying the rfkill block. If any controller
fails to power down before the deadline, log a warning to stderr, apply
the rfkill block as a safety fallback, and return non-zero status.

Upstream kernel report: AsahiLinux/linux#609
@JJRPF
JJRPF force-pushed the fix/bluetooth-graceful-poweroff branch from 98208ba to 38189e9 Compare September 12, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cherry-pick-later Port to omacom/omarchy after the overlay merges; match mainline.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants