FTL crashes under load

[nbx3]

Versions

Core version is v6.4 (Latest: v6.4)
Web version is v6.4.1 (Latest: v6.4.1)
FTL version is v6.5 (Latest: v6.5)

Platform

Raspberry Pi

Expected behavior

FTL should handle concurrent HTTP API requests to the web interface without crashing.

Actual behavior / bug

FTL crashes with a segmentation fault (SIGSEGV) in the civetweb-worker thread when handling multiple concurrent API requests. The null pointer dereference occurs at address 0 (SEGV_MAPERR), suggesting a race condition in the civetweb request handler.

The crash occurs intermittently, typically after 30-60 minutes of operation, when a Prometheus exporter makes ~19 concurrent authenticated GET requests to various /api/ endpoints on each scrape.

09:27:42 INFO: ----------------------------> FTL crashed! <----------------------------
09:27:42 INFO: FTL has been running for 2872 seconds
09:27:42 INFO: FTL version: v6.5
09:27:42 INFO: FTL commit: f888309
09:27:42 INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 15.2.0) 15.2.0
09:27:42 INFO: Received signal: Segmentation fault
09:27:42 INFO: at address: 0
09:27:42 INFO: with code: SEGV_MAPERR (Address not mapped to object)
09:27:42 INFO: Process details: MID: 68, PID: 68, TID: 238, Name: civetweb-worker

After the crash, FTL also logs high system load:
09:24:51 WARNING: Long-term load (15min avg) larger than number of processors: 4.2 > 4

Steps to reproduce

1. Run Pi-hole v6 (FTL v6.5) in Docker on a Raspberry Pi (arm64, 2GB RAM)
2. Configure a Prometheus exporter or similar tool that makes multiple concurrent authenticated API requests (e.g. /api/stats/summary, /api/dns/blocking, /api/stats/top_domains, /api/stats/top_clients, /api/stats/upstreams, /api/dhcp/leases, /api/network/devices, /api/info/system, etc.)
3. Wait 30-60 minutes — FTL crashes with SIGSEGV in civetweb-worker
4. The container restarts, and the cycle repeats

Reducing the concurrency of API requests (serialising instead of parallel) appears to mitigate the issue. I currently use my own exporter (https://github.com/nbx3/pihole-exporter) - i will add support for reducing the frequency, but this is still an issue

Debug Token

• URL:

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

• Platform: Raspberry Pi 4 (2GB), arm64, Raspberry Pi OS Bookworm
• Pi-hole: Core v6.4, Web v6.4.1, FTL v6.5 (Docker tag 2026.02.0)
• Docker: 28.4.0, network_mode: host

Compose file:

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    network_mode: "host"                                                                                                                                   
    environment:
      TZ: 'Europe/London'                                                                                                                                  
      FTLCONF_dns_listeningMode: 'ALL'
      FTLCONF_webserver_api_password: ${FTL_CONF_WEBSERVER_PASSWORD:-password}                                                                             
    volumes:                                                                                                                                               
      - 'etc-pihole:/etc/pihole'                                                                                                                           
      - 'etc-dnsmasq.d:/etc/dnsmasq.d'                                                                                                                     
      - 'var-log:/var/log/pihole'                                                                                                                          
    cap_add:
      - NET_ADMIN                                                                                                                                          
      - SYS_NICE
      - SYS_TIME
    restart: unless-stopped 

[rdwebdesign]

Please post a debug token:

• run docker exec -it <Pi-hole container name> pihole -d command.
• when asked to upload the log, answer Y.
• copy only the token and paste here.

[nbx3]

Sorry not sharing - too much sensitive data in that - if you need any specific config / versions happy to share

[DL6ER]

FTL crashes with a segmentation fault (SIGSEGV) in the civetweb-worker thread when handling multiple concurrent API requests. The null pointer dereference occurs at address 0 (SEGV_MAPERR), suggesting a race condition in the civetweb request handler.

This is a duplicate of #2786 with a proposed fix in #2833. Closing this in favor of the other issue ticket.

[DL6ER]

I think I have said this too early, I researched a bit further and found another issue, let us re-open this. The crash is caused by two thread-safety issues in the API request handling:

1. pi_hole_extra_headers global buffer race

pi_hole_extra_headers is a single global char[1024] buffer written by API handlers (Set-Cookie in src/api/auth.c:247, Location in src/api/list.c:509, Content-Disposition in src/api/teleporter.c:54) and read/cleared by send_additional_header() in src/webserver/civetweb/civetweb.c:4229. With up to 50 concurrent civetweb worker threads, one thread can overwrite or clear another's header data. This causes Set-Cookie headers to be sent to the wrong client (session leaking between connections) or dropped entirely. The buffer is a fixed-address global array, so the race causes data corruption but is unlikely to directly produce the NULL dereference.

Fix: Make pi_hole_extra_headers _Thread_local. Each civetweb worker thread gets its own buffer. This is safe because civetweb processes each HTTP request entirely within a single thread.

2. auth_data session array race (the primary crash cause)

The auth_data array (src/api/auth.c:32) is read and modified from multiple worker threads without synchronization — session lookups, validity timestamp updates, creation, and deletion all race against each other. A thread can observe partially-zeroed state if another thread calls delete_session() (which does memset() on the slot) concurrently. Additionally, JSON_REF_STR_IN_OBJECT stores raw pointers directly into auth_data memory (e.g. for sid, csrf, remote_addr in get_session_object and get_all_sessions), which can become stale if the underlying session is modified or cleared by another thread before cJSON serializes them.

Under the memory pressure described (2GB Raspberry Pi, 15min load average 4.2 > 4 cores, 19 concurrent authenticated requests), these races combined with potential allocation failures in civetweb's header handling (mg_strdup_ctx returning NULL) are the most likely path to the SIGSEGV at address 0 in the civetweb-worker thread. Without a full stack trace we can't pinpoint the exact crashing instruction, but these are the unprotected concurrent access patterns.

Fix: Add a pthread_mutex_t protecting all auth_data access (lookup, update, create, delete, iteration). Change JSON_REF_STR_IN_OBJECT to JSON_COPY_STR_TO_OBJECT for auth_data strings so the JSON tree owns copies rather than holding raw pointers into shared mutable state.

Serializing requests on the exporter side is a valid workaround but shouldn't be necessary — the server should handle parallel requests safely.

[DL6ER]

Proposed bug fix in #2835 you could test it using

sudo pihole checkout ftl fix/api-thread-safety