Repository navigation
Firewalla Crystal high CPU usage #345
Description
Hello,
I have recently installed Firewalla Crystal on a PC I had laying around and of course installed Controld on it right away.
However, after some time passes the process starts using between 80 to 100% of a core the entire time:
When I restart the ctrld process it stays lower for an hour at max before jumping to 80%+ again. I will email the logs/config to hello@controld.com.
Activity
matrix8967 commentedon Sep 28, 2026
Will take a look at this. In the mean time - could you try the suggestion from this comment and see if this keeps your memory profile down to a manageable level?
PietPiraat-1 commentedon Sep 28, 2026
I have started ctrld with the --silent flag. I will keep an eye on it.
PietPiraat-1 commentedon Sep 28, 2026
I started it with the --silent flag, but CPU is already at 80%+ again.
This is CPU usage after a restart:
And this is after ~20-30 minutes:
matrix8967 commentedon Sep 28, 2026
Thanks -- I had a deeper look at this and I think this is related to how
ctrlddiscovers client devices on the Firewalla platform.I'm working on verifying this internally and will follow up once I've confirmed the issue.
If you'd like to test a work around: please try this temporary change to see whether it stops the high CPU usage:
Copy the entire output, without the terminal prompt. Keep a copy as a backup. Don’t post it publicly, since it can contain your resolver ID and network details. If the command says the file doesn’t exist, let us know before continuing.
Open the Control D WebUI and find your Firewalla under Endpoints. Click its ctrld status/version button, then open TOML Config.
Paste the complete configuration into the editor. This editor needs the whole configuration, not just the setting below. If it already contains a custom configuration, back that up first and check that it matches before replacing anything.
Find [service] and add this line directly beneath it, or change the existing value to false:
Leave everything else unchanged.
DNS may briefly pause. Check that browsing works afterward, then monitor CPU usage for at least an hour.
This disables one device-discovery method, not IPv6 or your DNS filtering. Some devices may be identified less accurately during the test. Future changes to this custom configuration should be made in the WebUI, rather than the generated local file.
PietPiraat-1 commentedon Sep 29, 2026
Thank you for the fast responses!
I have added the discover_arp = false option. I will continue to monitor.
PietPiraat-1 commentedon Sep 29, 2026
It's been a couple of hours and I haven't seen the CPU spike like it has before!
Thank you!
matrix8967 commentedon Sep 29, 2026
@PietPiraat-1 Awesome - thanks for testing it out. I've started an internal bug report for this issue. Unfortunately the merge window for the next release is closed but we'll have a fix available in a future release.
hermesromeror commentedon Oct 2, 2026
Im having the same issue, with some Claude help I was able to detect the issue:
On Linux,
ctrldv1.5.7 goes to ~100% of one CPU core somewhere between 13 and 40 minutes after every restart and stays there until it is restarted again. The logs contain no warnings or errors when this happens. A goroutine dump (SIGQUIT) taken while it was spinning points toclientinfo.(*ndpDiscover).subscribe(internal/clientinfo/ndp_linux.go:42): its receive from the netlink neighbor-update channel never checks whether the channel was closed, so once netlink closes it theselectloop receives zero values forever.Environment
ctrld run --cd <redacted> --skip_self_checks --iface=auto --homedir=/data/controld --config=/data/controld/ctrld.tomlipv6 availability: falseat startup.discover_arp = true,discover_dhcp = true,discover_mdns = false, one listener on port 5354, 4 upstreamsObserved behavior
warnorerrorentries appear inctrld.logaround the time it starts spinning.topshowed the CPU split across Go runtime threads (for example 47% + 47%), which fits a single goroutine that is runnable all the time.Goroutine dump (SIGQUIT while spinning)
Only 1 of the 49 goroutines was
runnable. Apart from the goroutine on the netpoll/scheduler, the other goroutines were blocked inselect/chan receive/syscall.Goroutine 48 is the netlink helper that waits on
done. The netlink receive goroutine that feedschis no longer in the dump, so it has exited, andchis closed when it exits.Root cause
internal/clientinfo/ndp_linux.go:When the netlink receive loop in
vishvananda/netlinkstops, for example after a socket receive error such as ENOBUFS on a busy router, it closesch. A receive from a closed channel returns immediately with the zero value, andnu.Familyis0, so the loop hitscontinueand spins without blocking.Suggested fix
It would also help to pass
netlink.NeighSubscribeWithOptions(ch, done, netlink.NeighSubscribeOptions{ErrorCallback: ...})and log the error, so it is visible why the subscription ended. The same pattern may be worth checking in other netlink subscribers in the codebase.There is no config option to disable NDP discovery (the documented options are
discover_mdns/arp/dhcp/ptr/hosts), so the only workaround right now is to restart ctrld periodically. An option such asdiscover_ndp = falsewould also be a useful escape hatch.I can provide the full goroutine dump and debug log privately if needed.
ccpk1 commentedon Oct 12, 2026
Confirming I found this same issue independently and then landed at this thread before creating new. I'm adding a profiled measurement and a little more context that I think helps.
My interim workaround was to create a small watchdog script to restart ctrld when it spins out, but obviously that isn't ideal.
--- AI assisted debug below ---
I hit the same fault on a Firewalla Gold (x86_64, Ubuntu 22.04, 4 cores), running Local Config Mode. Relevant new data points:
1. It is not fixed in v1.5.8. I first saw it on v1.5.0, upgraded to v1.5.8 expecting the v1.5.1
dnsFromResolvConfloopback fix to help — the behaviour is identical on both. As of v1.5.8 the defect is still present, so the fix mentioned above has not landed yet.2. It never clears on its own. After one restart it spun for 3 days continuously at 99.2% before I intervened, with
go_goroutinesflat at 54 throughout. An earlier instance ran 27 days. It does not degrade, oscillate, or recover — consistent with a loop that has no exit condition.3. Measured onset (time since restart): 37 min, 56 min, 1 h 35 m, 1 h 47 m. One outlier lasted ~12 h. Matches the ~13–40 min and "an hour at max" reported elsewhere in this thread.
4. A symbolised CPU profile. This is the part I think is new to the thread. Previous evidence here is SIGQUIT goroutine dumps; this is a quantified profile of the spinning process.
perf record -F 199 -p <pid> -- sleep 15→ 2,990 samples, resolved to functions viago tool addr2line.runtime.selectgoruntime.unlock2…internal/clientinfo.(*ndpDiscover).subscriberuntime.selunlockruntime.lock2runtime.sellockruntime.memclrNoHeapPointerscontext.(*cancelCtx).DoneThe
select/lock machinery accounts for 77.15% of user samples, and the only application frame in the entire profile isndpDiscover.subscribe, with 265 samples (9.77%) on the single lineinternal/clientinfo/ndp_linux.go:44. That is thecase nu := <-ch:arm — the loop is not merely implicated, it is ~78% of all user CPU.Corroborating, consistent with the closed-channel mechanism:
go_sync_mutex_wait_total_seconds_totalrose only 1.48% of a core while spinning, andgo_sched_latencies_seconds_sumonly 0.13% — the locks are taken briefly and uncontended, not waited on, which is what an unblockedselectloop looks like.5. On the
discover_arp = falseworkaround — what it actually costs. I traced it, because it is the only workaround available and the tradeoff is not documented in this thread. NDP discovery has no config key of its own; it is gated by the ARP flag:so
discover_arp = falseis effective precisely because it stopssubscribebeing called at all. What it also disables: ARP/NDP populate the client-info table (IP ↔ MAC ↔ hostname), and that is forwarded to Control D by default for Control D/NextDNS DoH upstreams (UpstreamConfig.UpstreamSendClientInfo). So the cost is loss of per-device identification — MAC, LAN IP and hostname — in the Control D dashboard, and any per-device rules. DNS resolution and per-network (CIDR→upstream) policy are unaffected. Worth stating explicitly for anyone weighing it.6. One caveat on the WebUI instructions for Local Config Mode users. The
discover_arp = falsesteps given above go through the Control D WebUI TOML Config editor, which is a CD-mode path. This box runs Local Config Mode (no--cd), so nothing from the WebUI reaches it — the setting has to go directly into the local/data/controld/ctrld.toml. That is also safe here, because the regeneration concern in those instructions is specific to CD mode; in Local Config Mode the local file is authoritative. Might be worth noting in the guidance so Local Config Mode users don't follow the WebUI steps and see no effect.7. Also worth considering: an explicit
discover_ndpoption. Even after theokcheck lands, an explicit switch would help, since the only current escape hatch is disabling ARP as well. Aligning with the existingdiscover_mdns/arp/dhcp/ptr/hostsnaming would make it discoverable.Method note, in case it's useful for further profiling. The shipped binary is UPX-compressed and stripped (
-ldflags="-s -w"), which is whyperfcannot symbolise it. No rebuild was needed:upx -dunpacks it (-s -wremoves.symtabbut keeps.gopclntab, and the unpacked image retains section headers), thengo tool addr2lineagainst the unpacked image resolves sampled addresses. Validated by checking that the ELF entry point resolves to_rt0_amd64_linux. Happy to share the raw address list and the address→function mapping.Happy to provide the full goroutine dump, the 3-day 30-second metrics time series across the healthy→spun transition, and the profile.
hermesromeror commentedon Oct 13, 2026
Update from a second Firewalla Gold: root cause of the channel close, a tested fix, and a related second bug
Adding to the above (same setup as mine: Firewalla Gold, Control D mode, discover_arp = true).
Confirmed in v1.5.8 by source. At tag v1.5.8 (2d9f38c), internal/clientinfo/ndp_linux.go is byte-identical to v1.5.7: case nu := <-ch: at line 44, still without an ok check. The PR Release branch v1.5.8 #344 merge did not touch it.
Why the channel closes on a Firewalla. Firewalla’s own scanner (NmapSensor) sweeps every /24 on the box roughly every 12 minutes with ARP/ICMP/TCP. IPv4 ARP also goes to RTNLGRP_NEIGH, so the NDP subscriber receives a burst of neighbour events it then ignores (Family != V6; my box has no IPv6 neighbours). The default socket buffer (rmem 212992) overflows, Receive errors, and netlink closes ch silently because ctrld passes no ErrorCallback. The closes I logged (19 min after restart, then 11:30:43, 11:30:58, 11:31:43 and 11:54:53) each coincided with a scan start. I did not capture the ENOBUFS itself, but the next point supports the explanation.
Tested fix, 4 changes, ~40 lines. (a) re-subscribe on closed channel instead of spinning; (b) ErrorCallback that logs why the subscription ended; (c) 4 MiB receive buffer via SO_RCVBUFFORCE (fallback to SO_RCVBUF); (d) rescan of the neighbour table after each re-subscribe. Core of (a):
func (nd *ndpDiscover) subscribe(ctx context.Context) {
for {
if !nd.subscribeOnce(ctx) { return }
ctrld.ProxyLogger.Load().Warn().Msg("NDP neighbor subscription closed, re-subscribing")
select {
case <-ctx.Done(): return
case <-time.After(5 * time.Second):
}
}
}
// in subscribeOnce():
case nu, ok := <-ch:
if !ok { return true } // caller re-subscribes
Results: with only (a) the CPU stays at ~2.5% and re-subscribing works. With the larger buffer, a full NmapSensor sweep of every /24 no longer closes the channel. Running v1.5.8+patched since 2026-10-11 12:06 (and v1.5.7+patched since 2026-10-05), discover_arp = true: 3.0 to 3.3% CPU over 1 day 21 h, through a full reboot and multiple scans.
A second, independent bug: client-info discovery dies on any config reload. runWait() re-runs p.run(reload=true) on SIGUSR1, a control-server reload, or when the Control D config changes. The old run cancels its ctx, which ends ciTable.RefreshLoop(ctx) and logs NDP neighbor subscription Import code, preparing for release #1 ended: context done. The new run skips runClientInfoDiscover because of if !isMobile() && !reload. So after any reload, ARP, NDP, DHCP and mDNS discovery is dead until a restart (DNS keeps working; device names and X-Cd-Host degrade). This reproduces on v1.5.7 and still applies in v1.5.8. Fix: run discovery with a context tied to p.stopCh (and p.runAbortCh in 1.5.8), not the per-run ctx. Verified: with the original behaviour a SIGUSR1 reload killed the neighbour socket; with the fix it survived.
Related, Firewalla-specific: the dnsmasq config file is written 0600 while dnsmasq runs as user pi, which caused DNS outages of ~3 min on each hot reload (see Firewalla: dnsmasq_local/ctrld written with 0600 breaks DNS when Firewalla hot-reloads dnsmasq #346).
Happy to open a PR with these three patches (against v1.5.8) if that helps.
cuonglm commentedon Oct 19, 2026
@hermesromeror @ccpk1 Thanks for details report and analysis.
We will fix this in future release of ctrld.