Voice: relay and codec work run on the main thread, so every tick stall breaks voice #335

Summary

All voice work goes through the main thread, on the server and the client. On the server, frames are relayed from the same Receive() loop as replication and RPCs, and that loop runs on the one thread that also runs scripting and the game's PostUpdate. On the client, capture, encode, send and decode all run in VoiceClient::Update on the game thread. So any main-thread stall longer than the jitter buffer, on the server or on either client, is heard as voice breaking up. A busy server stalls longer as player count rises, so voice gets worse exactly when many people gather.

FiveM avoids this by keeping voice off its game and script threads at both ends (details below). This issue proposes doing the same here, in steps.

What we measured

HogwartsMP 1.9.1 (Framework 34.2.0 / 4cf5bb97) on a community server, 2026-10-16, with 18–20 players. Players reported proximity voice breaking up when many of them were in one area, plus rubber-banding.

Server (the host's own metrics):

• Slow main-thread ticks during the busiest hour averaged ~850 ms. 41 took over 1 s, and the worst took 4.7 s. During a 2.5-minute boss fight they ran 700–850 ms, worst 1.5 s.
• The day before, with 11 players: ~220 ms average, 0.5 s peak. Twice the players gave stalls about 4× longer.
• The host machine was mostly idle the whole time: a 4-core VPS at 9.5% average CPU, with 5.3 of 7.7 GB of memory free and ~6.3 Mbit/s going out. The main thread blocked; the machine didn't.

Client (a Tracy capture of the same fight, 252 s at ~58 fps):

• Voice work on the main thread is cheap: Client::Voice averaged ~9 µs per frame, max 0.16 ms.
• The main thread itself hitched: 78 frames over 50 ms and 25 over 100 ms, plus a burst of seven freezes of up to 0.93 s caused by an unrelated engine call. Because capture/encode/send/decode live in VoiceClient::Update, each of those also stops that player's voice in both directions.
• Every zone in the capture was on the main thread; no voice or network worker thread recorded any.

Fan-out: the router relays every talker to every listener in range, with no cap. For example, 10 talkers × 19 listeners × 50 frames/s is about 9,500 packets/s, roughly 7.6 Mbit/s at kBitrate = 40000. That's the same order as the ~6.3 Mbit/s total the host measured, so bandwidth isn't the problem. The problem is when those packets get sent: only while the tick isn't stalled.

Why it breaks (code at 378182e0)

Server: one thread for everything

• Instance::Update runs Server::Networking, then _voiceServer.Update(), then Server::Scripting and Server::PostUpdate, all on the same thread.
• Voice frames come out of RakPeer::Receive() in NetworkPeer::Update. The unknown-packet handler dispatches ID_RAKVOICE_RELAY_DATA to VoiceServer::OnVoiceFrame, which relays the frame.
• So a 1 s tick, whether from slow JS or slow game logic, means 1 s with no voice relayed for anyone on the server, followed by a burst.

Client: the codec runs on the game thread

• VoiceClient::Update runs PumpCapture (drain the mic ring, denoise, VAD, encode, send) and PumpSpeakers (decode into the playout buffers). Only playback rendering runs on the audio callback.

How much slack there is

• The playout buffer starts at kJitterBufferFrames = 3 (~60 ms), adapts between 2 and 10 frames, and skips ahead past kJitterBufferMaxFrames = 12 (240 ms).
• Arrival gaps longer than kJitterTalkspurtGapMs = 250 are treated as the talker pausing. So the adaptive depth never learns about server stalls, which are almost all longer than that.
• The capture ring is 16384 samples, ~340 ms at 48 kHz. A longer client freeze loses the speaker's own audio.

MafiaNet already has network threads, but only for transport

• RakPeer does socket I/O, acks and resends on its own threads (UpdateNetworkLoop / RecvFromLoop). That's why connections survive multi-second ticks.
• Send() is safe from any thread: it goes through SendBuffered, which pushes onto bufferedCommands.
• But every message the application sees, RakVoice's included, comes out of Receive() on whichever thread calls it, and plugin OnReceive runs there too.

How FiveM does it

Links are to citizenfx/fivem @ ab3ee45.

Server. FiveM embeds a Mumble server (voip-server-mumble, derived from umurmur).

• Voice UDP shares the game port. The ENet host's intercept hook, called from inside enet_host_service, hands Mumble datagrams to the UdpInterceptor handler.
• That handler relays the frame right there (Client_voiceMsg).
• All of this runs in m_net->Process() on the svNetwork loop (process call). Scripts run on the svMain loop.
• Cleanup runs on its own [Mumble] Worker thread. Shared state sits behind one mutex.
• Hitches are recorded per thread (svMain, svNetwork), so you can tell which thread stalled.
• Who hears whom is Mumble channels and voice targets. Voice scripts only change targets, so a slow script delays a channel switch, not the audio.

Client (voip-mumble):

• Networking runs on its own "mumble" loop.
• Capture and Opus encode run on [Mumble] Audio Input Thread.
• Decoding happens on packet arrival, and playback goes through XAudio2.
• The game thread only supplies positions.

Background reading: FiveM voice docs, and the Mumble protocol (TCP control channel plus UDP voice, with a TCP tunnel fallback).

Proposal, smallest first

1. A client voice thread (no wire change).
   • Move the work in PumpCapture / PumpSpeakers (denoise, VAD, encode, send, decode) to a voice thread. Send() is already thread-safe.
   • The main-thread Receive() only copies voice payloads into an SPSC queue for that thread.
   • The main thread publishes listener and speaker positions to it.
   • Result: a hitching client no longer cuts its own outgoing voice. Incoming voice still waits for that client's main-thread Receive() until step 2.
2. Server voice off the tick (wire change, so MAJOR). Receive() can't be split per message: it also drives ReplicaManager3 and RPC4, which have to stay on the main thread with game state. So voice needs a path of its own:
   • Preferred: a second peer or socket owned by VoiceServer and pumped by its own thread. This is the FiveM model: a separate stack sharing the game port through an interceptor, or on a second port.
     • Each voice connection is tied to its player with a token issued over the game connection.
     • The main thread publishes recipient sets into a snapshot the voice thread reads. The router already refreshes them only every 250 ms (kRecipientRefreshMs), so this is cheap.
   • Worth checking first: whether a MafiaNet plugin hook on the network thread can see voice messages after the reliability layer. OnDirectSocketReceive gets raw datagrams before reliability, so it probably can't.
3. Instrumentation either way.
   • Tracy zones in the voice paths, plus thread names.
   • Plots or counters for playout underruns, skip-aheads, ring-full drops and capture-ring overflows.
   • On the server: frames relayed and recipients per frame.
   • With these, the next crowded event can be measured instead of inferred.

Not covered here: position rubber-banding. Replication also rides the main tick, so fixing that means fixing the tick stalls themselves.