Feature/camera stream v1 - #391
Open
jugurtha114 wants to merge 31 commits into
Open
Conversation
added 30 commits
August 15, 2026 20:44
Windows agents never showed the audio button because windows_audio.c was
never compiled: _KVM_AUDIO was only defined for Linux builds, and the
Windows KVM had no audio hooks. As a result the agent never sent
MNG_AUDIO_CAPS, so the browser kept the speaker icon hidden.
Agent (meshcore/KVM/Windows/windows_audio.c), rewritten for robustness:
- WASAPI shared-mode loopback capture of the default render endpoint
- Handles any mix format Windows may pick: float32 and 16/24/32-bit PCM,
arbitrary channel counts (downmixed to mono) and any sample rate
(linearly resampled to 48 kHz)
- Ring buffer guarantees exactly 960-sample frames reach opus_encode,
independent of the buffer sizes WASAPI returns
- Polls instead of using EVENTCALLBACK: Windows does not reliably signal
the event handle for an idle loopback stream, which could stall capture
- Correct AUDCLNT_BUFFERFLAGS_SILENT handling (buffer contents are
undefined when set; emit true silence with identical cursor arithmetic)
- Resampler state is per-session (thread-local) so a restart cannot reuse
stale interpolation phase
- CRITICAL_SECTION around encoder/handler, Interlocked shutdown flag,
COM balanced on every exit path including RPC_E_CHANGED_MODE
- <initguid.h> emits the COM GUIDs locally, avoiding a uuid.lib dependency
Windows KVM (meshcore/KVM/Windows/kvm.c):
- Handle MNG_AUDIO_START/STOP/QUERY in kvm_server_inputdata
- Call kvm_audio_init after hStdOut is acquired and next to
kvm_send_display_list; initialising earlier wrote CAPS to an invalid
handle in service (slave) mode, so the frame was silently lost
- kvm_audio_cleanup on KVM shutdown
Build integration (previously missing entirely):
- MeshService{,-2022}.vcxproj and MeshConsole{,-2022}.vcxproj: add
windows_audio.c and kvm_audio.h, define _KVM_AUDIO, add
lib-opus\includes, and link the per-architecture libopus.lib plus
Ole32/Winmm across all 38 KVM-enabled configurations (x86/x64/ARM64)
The wire format is byte-identical to the working Linux implementation
(only the CAPS platform byte differs: 2 = Windows), so the existing
browser code needs no changes.
Verified: windows_audio.c compiles clean under -Wall -Wextra against
Windows API stubs; the real capture thread was exercised end-to-end
against a mock WASAPI device across 48/44.1/96 kHz, stereo/5.1/mono and
float32/s16/s32, confirming one CAPS frame, correct frame counts, no
sequence gaps, valid headers and safe double start/stop/cleanup. The
Linux x86-64 build still compiles and links unchanged.
The Windows agent is built with Visual Studio, which cannot run on the
Linux dev host, so the WASAPI code previously could only be reviewed by
inspection or compiled against hand-written stubs. Stubs cannot catch a
mismatch against Microsoft's real COM interfaces, which is precisely the
risk in this file.
MinGW-w64 ships the genuine WASAPI headers (mmdeviceapi.h, audioclient.h)
and Wine can execute the resulting PE, so this adds a container that:
1. compiles windows_audio.c for x64 and x86 against the real Windows
headers with -Wall -Wextra
2. asserts the COM GUIDs resolve locally via <initguid.h> (so no
uuid.lib is required) and that the only external Win32 imports are
ones Ole32.lib provides
3. links an actual PE executable and runs it under Wine to check CAPS
emission and the start/stop/restart/cleanup lifecycle, including the
no-audio-device path
4. compiles kvm.c with and without _KVM_AUDIO and asserts the audio
hooks add no new errors
kvm.c cannot fully compile under GCC because it uses MSVC __try/__except,
so step 4 compares error counts between the two configurations rather
than requiring a clean build.
This does not replace a Visual Studio build or testing on real hardware,
where the audio endpoint actually exists; it catches API and lifecycle
regressions before that point.
Run with:
docker build -f docker/Dockerfile.windows-audio-check -t meshagent-winaudio .
docker run --rm -v "$PWD:/src" -w /src meshagent-winaudio \
bash docker/check-windows-audio.sh
The Windows CI build failed with 92 link errors. Two independent causes, both invisible to the MinGW cross-compile check because MinGW papers over each one. 1. Unresolved COM identifiers (LNK2001 on CLSID_MMDeviceEnumerator, IID_IMMDeviceEnumerator, IID_IAudioClient, IID_IAudioCaptureClient). MSVC only emits a GUID definition when INITGUID is defined *before* the declaring header is first seen. windows_audio.c included <initguid.h> ahead of <mmdeviceapi.h>, but mmdeviceapi.h had already been pulled in transitively via kvm_audio.h -> ILibParsers.h -> windows.h, so the definitions were never generated and only declarations remained. Rather than fight include order, the four identifiers are now defined directly in this translation unit under MESH_ names. Their values are fixed in the Windows SDK. MinGW hid this because its DEFINE_GUID always uses DECLSPEC_SELECTANY, which defines the GUID unconditionally. 2. libopus.lib was built with MinGW/GCC, not MSVC. The committed archives contain .o members referencing ___chkstk_ms, __mingw_vfprintf, __memcpy_chk and __memset_chk -- GCC runtime helpers the Microsoft CRT does not provide -- so MSVC could never link them. They were carried over from the previous branch and had not been exercised on Windows before. A static library must be produced by the toolchain that consumes it, so the Windows build now compiles Opus 1.5.2 from source with the same MSVC that builds the agent, for x86, x64 and ARM64, and caches the result per architecture. The unusable archives are removed, the path is gitignored, and lib-opus/windows/README.md documents the reasoning and the local equivalent. The Linux and macOS libraries stay committed, since GCC/Clang builds both the library and the agent there. Also tightens docker/check-windows-audio.sh: it previously asserted the GUIDs were "defined locally", which is true on MinGW regardless of the bug. It now asserts the object leaves no COM identifier unresolved, which is precisely the condition MSVC reports as LNK2001, so the check would have caught cause 1.
x64 and ARM64 now build, but x86 failed with:
MSVCRT.lib(chandler4gs.obj) : error LNK2019: unresolved external
symbol __except_handler4_common
The preceding LNK4217 warnings identify the cause: malloc/free were
"imported by" libopus, meaning Opus had been built against the dynamic
CRT (/MD) while the agent links the static CRT (/MT). That mismatch makes
libopus emit /DEFAULTLIB:MSVCRT, which pulls in chandler4gs.obj and its
reference to __except_handler4_common. That symbol is part of x86-only
SEH, so x64 and ARM64 tolerated the same mismatched library while x86
failed hard -- which is exactly the pattern CI reported.
CMAKE_MSVC_RUNTIME_LIBRARY was already being passed, but it is only
honoured when policy CMP0091 is NEW, so it was silently ignored. The
workflow now sets CMAKE_POLICY_DEFAULT_CMP0091=NEW, additionally forces
/MT through CMAKE_C_FLAGS_RELEASE, checks the cmake exit codes, and runs
dumpbin /directives over the result to fail immediately if the library
still carries a DEFAULTLIB:MSVCRT directive. Verifying rather than
assuming matters here because two of the three architectures link a
mismatched CRT without complaining.
Separately, MeshService{,-2022}.vcxproj set RuntimeLibrary to
MultiThreadedDebug in Release|Win32 while every other Release
configuration used MultiThreaded, and that same configuration links the
release OpenSSL (libcrypto32MT.lib). Mixing debug and release CRTs in one
image is unsupported and compounds the failure above, so Release|Win32
now matches the other Release configurations. This is pre-existing
upstream and unrelated to audio, but it is corrected here because adding
libopus to that link is what exposed it.
The CRT verification added in the previous commit did its job: it stopped
the x86 build with "Opus was built against the dynamic CRT" instead of
letting the mismatched library reach the linker. Two things went wrong.
CMake reported:
Manually-specified variables were not used by the project:
CMAKE_POLICY_DEFAULT_CMP0091
so the policy override never applied and CMAKE_MSVC_RUNTIME_LIBRARY was
ignored again. Opus also listed OPUS_STATIC_RUNTIME among its disabled
features -- it has a purpose-built option for this, which is a better fit
than overriding CMake's runtime handling from outside. Switched to
-DOPUS_STATIC_RUNTIME=ON and dropped the CMAKE_C_FLAGS_RELEASE override,
which was additionally clobbering Opus's own optimisation flags.
Because CMake only warns (never fails) about unknown variables, the
configure step now reads the resulting cache back and aborts unless
OPUS_STATIC_RUNTIME really is ON, rather than trusting the command line.
Hardened the CRT check too: it previously skipped verification entirely
if dumpbin was not found, which would quietly disable the guard that just
caught this. It now searches both Program Files locations, fails if
dumpbin is missing, and requires a positive LIBCMT directive rather than
only the absence of MSVCRT.
Bumped the cache key to opus-...-msvc-static-v2. The x64 and ARM64 jobs
succeeded on the previous run and cached dynamic-CRT libraries under the
old key; without the bump they would restore those and skip the fix. They
linked only because __except_handler4_common is x86-only SEH, so their
CRT mismatch was latent rather than harmless.
All three jobs failed with "OPUS_STATIC_RUNTIME was not applied" even though the log printed OPUS_STATIC_RUNTIME:BOOL=ON. The option was set correctly and Opus built fine -- the check I added was wrong and rejected a good build. `cmake -L` prints Opus's feature summary as well as the cache, so Select-String matched two lines and $cached was an array. In PowerShell -match/-notmatch FILTER when the left operand is an array rather than returning a boolean, so the expression evaluated to the feature-summary line (which contains no "=ON"), and a non-empty array is truthy, so the guard always threw. Both checks now join their output into a single string first, and the cache test matches the exact entry OPUS_STATIC_RUNTIME:BOOL=ON. Verified in pwsh against the literal strings from the failing run: the old form throws on a correct build, the new form passes on ON and still throws on OFF, and the dumpbin test accepts a static-CRT library while rejecting a dynamic-CRT one. Also switched the build from Opus 1.5.2 to 1.6.1. The headers vendored in lib-opus/includes turn out to be 1.6.1 verbatim -- all six compare byte-identical -- so 1.5.2 was compiling the agent against one version's API while linking another's library. 1.6.1 makes the two agree, which is what the version comment in the workflow now records. Cache key bumped to opus-1.6.1-...-v3 so nothing is restored from the earlier 1.5.2 attempts.
Building the agents in CI only helped if someone then downloaded the artifacts and committed them by hand, which is why the Windows agents the server was handing out were still the March binaries with no audio code while Linux worked: that one had been rebuilt and committed manually. Adds a deploy workflow that runs after a successful Windows or Linux build on this branch, downloads the artifacts, installs them into MeshCentral-jugu with the tools/install-*-agents.js scripts, and commits the result. Both platforms trigger the same workflow, so it is serialised with a concurrency group and retries the push after rebasing if the other platform landed first. The agents are committed to git rather than fetched by the server at container start. What a server hands out determines what every managed device installs, so it should be reviewable and revertible, the image should stay self-contained and reproducible, and the production host should not need GitHub credentials or a working network path to GitHub during boot. Also passes AUDIO=1 to the KVM Linux builds (ARCHID 5, 6, 26). Without it CI would have published Linux agents with the audio support silently removed, undoing what already works today. The NOKVM and Alpine targets are deliberately left alone: they have no KVM, and only x86, x86-64 and arm64 have a committed lib-opus to link against. Requires a MESHCENTRAL_DEPLOY_TOKEN secret with write access to MeshCentral-jugu; without it the deploy job fails at checkout and the build jobs are unaffected.
The deploy workflow only runs on completion of a Windows or Linux build, so it could not fire for the run that predated it. This empty commit starts a fresh build whose artifacts it can publish.
Adds the reverse of the existing speaker feature: an administrator can speak to the person at the managed device. Because that makes a remote voice audible in someone's room, playback is gated on the local user agreeing, and the gate is enforced in the agent rather than only in the browser or the server. Protocol (meshcore/meshdefines.h), using the next free command ids: MNG_MIC_QUERY 95, MNG_MIC_CAPS 96, MNG_MIC_START 97, MNG_MIC_STOP 98, MNG_MIC_DATA 99 MNG_MIC_DATA reuses the audio frame layout, so the two directions share one wire format. meshcore/KVM/kvm_mic.h with linux_mic.c (PulseAudio) and windows_mic.c (WASAPI shared render) decode Opus to 48 kHz mono and play it. Both refuse to open the output device without consent and discard every MNG_MIC_DATA frame while it is absent, so a client that skips the handshake gains nothing. Consent is per session: stopping playback, revoking, or ending the tunnel all clear it, and the next attempt prompts again. Timing out never counts as agreement. windows_mic.c declares the COM GUIDs locally for the same reason as windows_audio.c, and converts the decoded mono into whatever mix format the endpoint reports rather than assuming one. test/mic/run.sh compiles the real linux_mic.c against stub Opus and PulseAudio and asserts the gate directly: audio discarded before consent, start() refused without it, playback stopping the instant it is revoked, stop() clearing it, and malformed frames rejected safely. It needs no audio hardware, so it runs anywhere. Verified: Linux x86-64 builds clean, and windows_mic.c compiles without warnings for x64 and x86 against the real Windows headers.
…or's The first implementation had the direction backwards. It played the administrator's microphone through the device's speakers, which is the push-to-talk feature intended for later, not the one asked for. The browser consequently prompted the administrator for microphone access, which it should never do. Corrected so the device's own microphone is captured and streamed to the operator, letting them hear the user speaking and diagnose noises such as fans or clicking drives. This is the same direction as the existing audio feature; only the source differs, so linux_mic.c and windows_mic.c are now siblings of linux_audio.c and windows_audio.c rather than their inverse. Linux records from PulseAudio's default input rather than "@DEFAULT_MONITOR@", and Windows opens the eCapture endpoint without AUDCLNT_STREAMFLAGS_LOOPBACK. While fixing this I found the consent path could never have worked: MNG_MIC_START called kvm_mic_start(), which correctly refuses without consent, but nothing ever called kvm_mic_set_consent(1). Added MNG_MIC_CONSENT (100), sent only by the agent's own consent flow once the local user accepts. Keeping it separate from MNG_MIC_START means a browser frame can request capture but never grant permission for it, and meshcore.js drops command 100 if it ever arrives over the tunnel. kvm_mic_feed() is retained as a no-op so the KVM command switch stays symmetrical with the audio path; audio only travels device -> browser here, so anything inbound on MNG_MIC_DATA is discarded. test/mic now asserts the capture-direction properties: the microphone is not opened without consent, nothing is encoded before it is granted, capture stops when it is revoked, and stop() clears it so a later start re-prompts. Fixed a flaw in the harness itself, where the fake libpulse-simple referenced a counter in the test executable that dlopen cannot resolve, which made the library fail to load and the test report passes for the wrong reason.
The Windows agent sent a hard-coded 0x07 in the MNG_MIC_CAPS flags byte, left over from the audio implementation it was derived from. The browser reads bit0 as "microphone available" and bit1 as "consent granted", so 0x07 claimed a microphone that might not exist and, worse, always claimed consent had been given. The button state was therefore meaningless on Windows. It now probes for a default capture endpoint through the same COM path capture uses, and reports the live consent flag, matching what linux_mic.c already sent. A machine with no microphone hides the control instead of offering a button that cannot work.
The deploy workflow filtered workflow_run to feature/audio-stream, so the microphone work on feature/mic-stream-v2 built successfully and then deployed nothing. The failure was silent: the builds went green while the server carried on serving the previous agents, which is exactly the gap this workflow exists to close. The branch filter is removed, and the job now publishes to the MeshCentral branch of the same name when one exists, falling back to feature/audio-stream otherwise. Work on a feature branch therefore lands on its own counterpart instead of on the shared branch.
…om native code Windows kvm_mic_start() never checked g_consent at all: MNG_MIC_START from the browser started real microphone capture unconditionally, as long as an encoder existed and nothing was already running. The doc comment, the Linux implementation, and the whole design promised the opposite (fail closed without a local decision) -- this brings Windows in line with that. With the gate now real on both platforms, the consent prompt still needs a way to fire. Two attempts at that earlier this branch (a plain-object relay piped in place of the native KVM stream, and a wrapper on kvm.write) broke the desktop video stream, because JS has no safe way to intercept a multiplexed KVM tunnel from either side of pipe()/unpipe(). So this drives it from the one place that already sees every MNG_MIC_START and already knows the true consent state: native code. kvm_mic_start(), in linux_mic.c / windows_mic.c, now emits a new upward-only command, MNG_MIC_CONSENT_NEEDED, through the exact same slave-to-master channel MNG_MIC_CAPS/MNG_MIC_DATA already use in production -- when it refuses specifically because consent (not hardware) is missing. agentcore.c's existing KVM write-sink (already special-cases MNG_AUDIO_QUERY/MNG_MIC_QUERY there) intercepts it and calls into meshcore.js's onMicConsentNeeded() via duk_peval_string(), the same native-to-JS mechanism already used elsewhere in that file (RemoteDesktop_EndSink's SendCommand call) -- never touching the stream/pipe machinery that broke last time. kvm_mic_start() is now also called once, speculatively, right after the KVM slave starts on both platforms, so a device with a real microphone offers the consent prompt at session open instead of only after the operator clicks. Calling it there is a no-op unless consent is genuinely the only thing blocking it, so it costs nothing on devices with no microphone. test/mic/run.sh's consent-gate suite, run against the real linux_mic.c (now 21 assertions, up from 15), covers the new notify behavior: fires once per genuine refusal, not when hardware is absent, not while already capturing, and again after stop() revokes consent. Windows validated by CI/Wine only, per how this project already validates Windows changes.
Companion to the MeshCentral-jugu commit adding Audio/Mic quick-toggle buttons that connect without opening the Desktop panel. Without this, such a session still did full screen capture/diff/send under the hood, just discarded unread by the browser -- not the point of an audio/mic-only mode. g_remotepause was already set correctly by the existing MNG_KVM_PAUSE switch case on both platforms, but nothing ever read it: grepping the whole tree found zero readers. This wires a read into each platform's tile-capture loop, skipping straight to a paced sleep+continue when set. Linux: inserted right after the incoming-command read (so an unpause can still arrive) and before screen capture starts, explicitly closing the per-iteration X11 display handle first so it isn't leaked by skipping the end-of-iteration cleanup that normally closes it. Windows: KVM command handling runs on its own thread (kvm_mainloopinput), independent of this tile loop, so the check can sit right at the top and skip everything below unconditionally. Re-ran test/mic/run.sh against the real linux_mic.c (21/21) as a regression check -- this doesn't touch consent state, but it's adjacent code in the same file this session has broken twice before.
…indows) kvm_mic_start()'s refusal path notified the JS layer to show a consent prompt whenever consent was missing, without checking whether a real capture device even exists -- only g_enc (the Opus encoder, which is software and always creatable) gated it, not real hardware. On a machine with no microphone at all, the browser correctly never shows the mic button (mic_send_caps() already calls microphone_available() for that), but the operator opening a desktop session would still make the local user see "wants to listen through your microphone" for a device that was never offered as available in the first place. Fixed by gating the notify on the same microphone_available() COM probe already used for CAPS reporting, mirroring how the Linux side already gates on g_pa_lib (the PulseAudio library) before ever deciding to notify.
Two problems with the consent prompt, both reported from real use. First, the prompt fired at KVM session open, so opening a desktop session asked the local user about the microphone even when the operator only wanted the screen and never touched the mic -- a permission dialog for something nobody requested. The speculative kvm_mic_start() at session start is removed on both platforms; the prompt now happens only on a real MNG_MIC_START, which is the thing the local user is actually being asked about. Second, clicking the microphone button again to cancel (or after clicking it by mistake) left the dialog on the device screen, asking about a request nobody was waiting on any more. kvm_mic_stop() now emits MNG_MIC_CONSENT_CANCEL, the counterpart of the existing MNG_MIC_CONSENT_NEEDED, which agentcore.c turns into a call to onMicConsentCancelled() so the JS layer closes the dialog. Whether a prompt is outstanding is tracked explicitly (g_promptOutstanding) rather than inferred from consent/thread state. The test caught why that matters: inferring it made a second stop emit a stray cancel, which would later close an unrelated prompt. Set when the prompt is raised, cleared when it is answered or cancelled. test/mic/run.sh extended to cover both (23 assertions): a stop while awaiting consent cancels exactly once, and a stop with nothing pending does not cancel at all.
The mic encoder was fully hardcoded (28 kbps, FEC/DTX on, complexity 5, and even a cross-platform mismatch: Linux used VOIP application mode, Windows used AUDIO) with no way for the operator to trade quality for bandwidth. Extends MNG_MIC_START with an optional 8-byte settings payload (bitrate, application mode, VBR/CBR, bandwidth, frame size, complexity, DTX, FEC, packet loss hint) that kvm_mic_start() applies to the live encoder -- recreating it only when application mode changes, since that's not safely live-updatable, and applying everything else via opus_encoder_ctl without interrupting an already-running session. Fully backward compatible in both directions: a legacy 4-byte START (or one from a server that predates this) is treated as "keep current settings", and MNG_MIC_CAPS now reports a protocol-version byte so the browser can tell whether a given agent understands the extended frame at all before offering profile selection. Also unifies both platforms on OPUS_APPLICATION_VOIP as the compiled-in default (this captures speech/room noise, not music), and moves the capture frame size from a compile-time constant to a runtime value bounded by a max-size buffer, since profiles can now request up to 60ms frames instead of the previous fixed 20ms.
…efault Adds MNG_MIC_DEVICE_QUERY/MNG_MIC_DEVICE_LIST so the browser can ask an agent to enumerate its capture devices and offer them for selection, defaulting to the system's default input as before. The enumerated order is that session's index space; MNG_MIC_START grows a 9th settings byte (deviceIndex, 0xFF = default, out-of-range also falls back to default) referencing it. Linux: enumeration needs PulseAudio's async context API, which pa_simple never exposed -- added a second, independent dlopen of the full libpulse.so.0 (pa_simple-only capture still works unaffected if this one is missing) driving a bounded synchronous mainloop-iterate loop (3s timeout, own polling rather than trusting pa_mainloop_iterate's blocking variant). Struct layouts (pa_source_info et al) come from the real <pulse/pulseaudio.h> at compile time -- confirmed needed the hard way, since this file's own hand-rolled pa_sample_spec/pa_stream_direction_t now collided with it once both were in scope -- while every function is still resolved via dlsym(), never linked, so a system without PulseAudio still runs everything else in this binary. linux-build.yml gains libpulse-dev for the header only. Windows: enumeration is IMMDeviceEnumerator::EnumAudioEndpoints, already the same family of interfaces this file uses for capture; two more locally-defined GUIDs (IMMDeviceCollection, IPropertyStore) follow the file's existing INITGUID workaround. Neither platform's audio API has a live "switch device" primitive, so unlike every other MNG_MIC_START field, a device change makes mic_apply_params() report back to kvm_mic_start(), which stops and restarts the capture thread on the new endpoint -- without touching consent, so the operator isn't re-prompted just for changing input. Both native implementations were syntax/type-checked against their real platform headers before this commit (gcc -fsyntax-only against the actual libpulse-dev + this repo's own lib-opus/includes for Linux; MSVC itself isn't available to verify Windows, so that side leans on closely mirroring the already-working capture code's COM patterns).
Root cause of "Mic panel: consent prompt never shows, on both platforms": kvm_mic_query_devices() ran its PulseAudio/WASAPI enumeration directly on the same single-threaded KVM command dispatch loop that also processes MNG_MIC_START. The Mic panel sends SendMicDeviceQuery() immediately before SendMicStart() (see p22Toggle() in MeshCentral-jugu), so a slow or unresponsive audio daemon delayed the very command that triggers the prompt behind however long enumeration took (up to ~6s in the worst case) -- exactly the class of bug mic_capture_thread already exists to avoid for the streaming side, which this function didn't follow. Both platforms now spawn the actual enumeration on its own thread and return immediately; a query that arrives while one is already running is coalesced into a no-op rather than started concurrently, since two overlapping enumerations would stomp each other's device list. Also adds a way for the Mic panel specifically to skip the interactive local-user prompt entirely, per feedback: an operator on that panel isn't watching the desktop anyway, so there's nothing for an interactive prompt to add, whereas the Desktop panel's own mic button should keep going through it whenever server policy requires one. Implemented as a *request* bit in MNG_MIC_START's miscFlags, carried through MNG_MIC_CONSENT_NEEDED to the agent's JS layer -- native code still never grants its own consent (kvm_mic_start()'s fail-closed gate is unchanged), and the actual authorization decision is made by agents/meshcore.js's micConsentHandleStart(), the same trusted place that already decides whether policy requires a prompt at all, treating "panel asked to skip it" as a second way to reach the exact fast path policy-bypass already uses (still logged, still tells native to actually start capturing -- setting the flag without that was the original bug this fast path was written to fix). Separately, investigated the reported Windows Desktop-tab symptom (prompt doesn't appear until a second click, which cancels it, then it flashes and disappears): micConsentHandleStart()'s Windows "enhanced" dialog path calls server_getUserImage() -- a round-trip to the server -- before creating the actual win-userconsent dialog. A cancel that arrives during that window finds nothing to close yet (the dialog doesn't exist, so pr.close()/pr.__childPromise.close() are both still undefined) and is silently lost. Now closes the dialog immediately upon creation if a cancellation was already requested, reusing the exact close() path the normal cancel flow already relies on rather than a second bespoke one. This path is pre-existing and unrelated to the enumeration-threading fix above; not confirmed to be a regression from recent work, but it matches the reported symptom precisely.
Adds the device-side half of remote webcam viewing: MNG_CAM_* commands (105-116), the kvm_cam.h contract, and a V4L2 implementation built on the same shape linux_mic.c already proved -- a capture thread that owns the device, a fail-closed consent gate driven by the agent's JS layer, and enumeration that never runs on the KVM command thread. Two deliberate design choices worth recording: Streams complete JPEG frames rather than a video codec. Essentially every UVC webcam already emits MJPEG in hardware, so the common path forwards the camera's own bytes untouched and spends zero CPU encoding -- which is what makes this usable on the ARM boards this agent runs on, where a software H.264 encoder would saturate the machine it is meant to be diagnosing. Self-contained frames also mean a drop costs one frame instead of corrupting everything until the next keyframe. Cameras that cannot produce MJPEG fall back to YUYV plus libjpeg-turbo, already linked for desktop tiles, so no new third-party library is introduced. Does NOT reuse the desktop's tile differencing. That works because desktop content is static enough for most tiles to be byte-identical frame to frame; camera sensors perturb essentially every pixel of every frame with thermal and shot noise, so exact-match tiling would find almost nothing to skip while costing CPU to look. The equivalent that does work is whole-frame suppression with a tolerance -- decide whether the scene changed rather than whether the bytes did -- implemented via libjpeg-turbo's scaled (1/8) decode, which downscales during entropy decoding rather than after. An unattended room then costs near-zero bandwidth while staying live the instant anything moves. Frames exceeding the KVM header's 16-bit length are sent wrapped in MNG_JUMBO, which both receivers already understand generically, so nothing downstream needed changing to carry full-resolution stills. Gated behind CAMERA=1 / -D_KVM_CAMERA. Verified that camera-enabled, audio-only and plain-KVM configurations all still compile, and that linux_cam.c reduces to an empty translation unit when the flag is off.
Enables CAMERA=1 on the same three KVM-capable targets that already get AUDIO=1: x86 (5), x86-64 (6) and arm64 (26). The NOKVM and Alpine targets stay without it. No new package is required. V4L2 is kernel ioctls through linux/videodev2.h, which comes from libc6-dev and is already installed, and the JPEG encode/decode the camera path uses is the libjpeg-turbo the KVM build already links for desktop tiles -- so unlike AUDIO=1, which needed libpulse-dev added here, this adds nothing to the install step. Verified with "make -n" that CAMERA=1 compiles linux_cam.c and defines _KVM_CAMERA exactly as AUDIO=1 does for linux_mic.c, and that neither appears without the flag.
…rleave A pipe write is only atomic up to PIPE_BUF (4 KiB) and a camera frame is far larger, so two threads writing at once would interleave their bytes and corrupt both frames. This was reachable, not theoretical: the one-shot snapshot worker only spawns when nothing is streaming, but the operator can start the stream while that worker is still in flight, which puts two writers on the same fd at the same moment. send_caps() and the device-list sender were also writing directly rather than through cam_write_out(), so either could land in the middle of a frame the capture thread was emitting. All four senders now go through cam_write_out(), which serializes on its own mutex -- deliberately not g_lock, since callers reach it both holding and not holding that, and a frame write must not block the capture thread's per-frame settings checks behind it.
MNG_CAM_CONSENT carried no payload saying which kind of request it was answering, so its dispatcher case always called kvm_cam_start(), even when the local user was only granting a one-shot snapshot -- a photo request would silently turn on the live stream too, and any settings the original request carried were dropped across the consent round-trip since neither fail-closed branch ever applied them. Native now tracks what was actually pending (g_pendingAction, a bitmask so a rapid snapshot-then-start click before the user answers doesn't lose either one) and replays exactly that once consent is granted, via the new kvm_cam_consent_granted(). Snapshots also no longer piggyback on the live stream's resolution: cam_snapshot_worker now pauses a running stream, opens the device at its own resolution (the operator's request, or the camera's true maximum via the new VIDIOC_ENUM_FRAMESIZES-based cam_find_max_resolution() when none was given), and resumes the stream afterward. CAM_SNAPSHOT_QUALITY is bumped to 100 so a still defaults to the camera's best.
Implements windows_cam.c as the Windows counterpart to linux_cam.c: MJPEG passthrough when the camera offers it, YUY2/NV12 raw fallback via WIC otherwise, static-scene suppression, independent-resolution snapshots, device enumeration, and the same fail-closed consent gate. Live streaming uses an asynchronous IMFSourceReaderCallback (stopped via Flush(), never a blocking read) so a wedged or unplugged camera cannot hang session teardown; snapshots stay synchronous since they are already bounded by a small attempt count. kvm_cam_consent_granted() ports verbatim from Linux, which is what keeps Windows from reintroducing the "photo also starts the stream" bug already fixed there. Wires the MNG_CAM_* dispatcher into kvm.c using the already-fixed consent pattern (not the still-buggy kvm_mic_start(NULL,0) pattern next to it), and adds windows_cam.c/kvm_cam.h plus the Media Foundation and WIC import libs to every build configuration in both vcxproj files.
…rnal CI (all three architectures) failed with LNK2019 on MFGetAttributeSize -- this SDK/lib configuration doesn't provide it as the header-inline helper it was assumed to be. Replaced the one call site with a direct IMFAttributes::GetUINT64 vtable call and manual high/low-32-bit unpack of MF_MT_FRAME_SIZE, matching every other MF method call in this file that already links cleanly.
camera_available() called MFCreateAttributes()/MFEnumDeviceSources() without initializing COM on the calling thread. Every other MF-calling function in this file runs on a worker thread that wraps CoInitializeEx/CoUninitialize around its own body, but camera_available() is called from send_caps(), which also runs directly on the KVM command dispatch thread for kvm_cam_init()/kvm_cam_set_consent()/kvm_cam_stop()/kvm_cam_resend_caps() -- a thread nothing else here ever initializes COM on. MFCreateAttributes() failing with CO_E_NOTINITIALIZED on that thread made a real, present camera get reported as unavailable. Fixed by wrapping the function's own body with CoInitializeEx/CoUninitialize, mirroring windows_mic.c's microphone_available(), which already does this correctly.
Two reliability gaps, neither hit in normal testing but both real on hardware beyond a typical USB UVC webcam: - cam_dev_open() only ever tried MJPEG then YUYV. A camera offering neither (UYVY, common on some webcams/capture dongles; NV12, the ISP-native planar format on many ARM/CSI-connected cameras -- boards this agent explicitly targets) would enumerate fine in V4L2 yet be entirely unusable here. Extended the fallback chain to MJPEG -> YUYV -> UYVY -> NV12, refactored encode_yuyv_jpeg() into a generic encode_rgb_jpeg() plus per-format raw_to_rgb()/raw_thumbnail() dispatchers so adding a format means one new converter pair, not touching every call site. cam_find_max_resolution()'s fallback chain in cam_snapshot_worker() extended to match. - The live stream sent its very first frame(s) immediately after opening the device, unlike the snapshot path, which already discards the first 3 attempts while auto-exposure/white-balance settle. On a camera with real settling time, that meant the stream's first visible frame could be dark or half-exposed, and static-scene suppression would lock onto it as the baseline until a genuine scene change happened to clear it. Added the same 3-frame warmup discard to capture_thread(), counted on every grabbed frame independent of pacing. Verified via gcc -fsyntax-only -Wall -Wextra (zero warnings) -- same platform-neutral logic can't be exercised against real hardware in this environment, so behavioral verification is still a follow-up.
The camera code was added to meshconsole/MeshConsole*.vcxproj, but the binaries MeshCentral actually deploys are MeshService.exe / MeshService64.exe / MeshServiceARM64.exe, built from meshservice/MeshService*.vcxproj. Those projects carry windows_mic.c and _KVM_AUDIO but never got windows_cam.c or _KVM_CAMERA, so every camera symbol compiled to nothing and the agent had no MNG_CAM_* handling whatsoever -- which is why Windows reported "This device has no usable camera" no matter how many times the agent was reinstalled. Confirmed empirically rather than by inspection: the deployed MeshService64.exe imports ADVAPI32/COMCTL32/CRYPT32/dbghelp/GDI32/gdiplus/IPHLPAPI/KERNEL32/ ncrypt/ole32/sas/Shcore/SHELL32/User32/Ws2_32 and nothing else -- no Mf.dll, Mfplat.dll, Mfreadwrite.dll or Windowscodecs.dll -- while Opus (i.e. _KVM_AUDIO) is present, so the mic half shipped and the camera half did not. Applies the same four-part edit already made to the meshconsole projects, to all 12 configurations of MeshService-2022.vcxproj (which the CI solution builds for Win32/x64/ARM64) and all 8 of the legacy MeshService.vcxproj.
cam_dev_open() called IMFActivate::ShutdownObject() immediately after ActivateObject(), while still holding and intending to use the media source it had just created. Microsoft's contract is explicit -- "call ShutdownObject when you are done using the object" -- so this tore the source down before a single frame could be read, and every later call on it returned MF_E_SHUTDOWN. That is why Windows could detect a camera but never actually use one: device enumeration (MFEnumDeviceSources) never activates anything, so caps correctly reported a camera present, while both the live stream and the snapshot failed the instant they tried to open it. Releasing the activation object alone is correct and does not disturb the source. Also gives the live session's IMFMediaSource a deterministic lifetime: it is now held in g_liveSource and explicitly Shutdown() by cam_teardown_live() after the reader is released, instead of being dropped and left for COM to destroy whenever the reader's last reference went away. The device is single-open and cam_snapshot_worker() pauses the live stream specifically so it can reopen the same camera, so the device has to be free the moment teardown returns, not eventually.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In this pull request, the following changes are made:
Please follow this checklist to avoid unnecessary back and forth (click to expand)
I understand that I am responsible for and able to explain every line of code I submit.
the impact on platforms and architectures I could not test.
modules/, I re-embedded them so the compiled-in copies inmicroscript/ILibDuktape_Polyfills.cmatch (the agent runs the embedded copies, not the files on disk)..mshoptions table in readme.md.Testing
Screenshots for Visual Changes