diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index eff9506..5e75c37 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -49,7 +49,7 @@ body: id: headset attributes: label: Headset and client - description: For streaming. The headset, the ALVR client version, and USB or Wi-Fi. + description: The headset and how it is connected (the Mac's own display and USB, or streaming); for streaming, the ALVR client version and USB or Wi-Fi. placeholder: Quest 3, ALVR client v20.14.1, Wi-Fi - type: textarea id: logs diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3c57b68..1a3dfef 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,6 +8,9 @@ is built and what is missing. - **A game that does not work, or does**: open a *Game report*. It asks for the game, the Wine and Mullion versions, the headset and the logs. If the game is broken without VR too, it is a Mullion or Wine matter. +- **A headset MetalVR has not met yet**: follow its guide in [docs/headsets](docs/headsets/README.md) and open a + *Bug report* saying how far it got, with the server's log (`metalvrd --verbose`). Each guide lists what is still + unknown about that headset, which is what such a report answers. - **Bugs**: open a *Bug report* with the commit you built, the Mac and macOS version, and the output of `metalvrd --verbose` and of the application with `METALVR_LOG=1`. - **Security issues**: never in a public issue. See [SECURITY.md](SECURITY.md). diff --git a/README.md b/README.md index cdde454..14d37cc 100644 --- a/README.md +++ b/README.md @@ -2,62 +2,30 @@ [![CI](https://github.com/metalsharp/MetalVR/actions/workflows/ci.yml/badge.svg)](https://github.com/metalsharp/MetalVR/actions/workflows/ci.yml) -An open-source OpenXR runtime for Apple silicon Macs. +An open-source OpenXR runtime for Apple silicon Macs. PC VR games run under Wine with Direct3D translated to Metal by +[Mullion](https://github.com/dttdrv/mullion); MetalVR is the runtime they talk to and the server that shows their +frames on a headset. SteamVR games reach it through MetalVR's SteamVR driver, and native macOS OpenXR applications +through a Metal runtime. -PC VR games run under Wine with D3D12/11 translated to Metal by [Mullion](https://github.com/dttdrv/mullion). -MetalVR is the OpenXR runtime those games talk to, and the native server that composes their frames and shows -them on a headset: an Oculus Rift CV1 plugged into the Mac itself, or a standalone headset running the stock -[ALVR](https://github.com/alvr-org/ALVR) client, streamed with hardware encoding. SteamVR games reach it through -MetalVR's SteamVR driver or [OpenComposite](https://gitlab.com/znixian/OpenOVR), and native macOS OpenXR -applications use the same server through a Metal runtime. +**Status:** an Oculus Rift CV1 on the Mac's own HDMI and USB plays SteamVR games (Subnautica, ABE VR, Raw Data) with +its sensors and Touch controllers. Every other headset below is built and checked without the headset, and has not +met one yet: reports from owners are very welcome. -Current Status; +## Headsets -Testing is happening with an Oculus Rift CV1. First we had to create a vr server and apply specific commands for MacOS to recognize and load the headset display. We tied this to a 'Start VR Server' button on a dedicated MetalVR launch route. We tied Steam Launches to launch with mullion also, since this program relies on Mullion to load it's driver for d3d11/12. After this, we were able to get SteamVR to recognize the headset and render the tutorial through the headsets display at 90hz, and the headsets resolution. After more tests we were able to launch Subnautica through steam with SteamVR opened and the VR Sever running. BeamNG also was able to use the headset but currently experiences a crash in its driver. +| Headset | Tried on hardware | Guide | +|---|---|---| +| Oculus Rift CV1 | yes | [docs/headsets/RIFT_CV1.md](docs/headsets/RIFT_CV1.md) | +| Oculus Rift S | no | [docs/headsets/RIFT_S.md](docs/headsets/RIFT_S.md) | +| Meta Quest 1, 2, 3, 3S, Pro (ALVR) | no | [docs/headsets/QUEST.md](docs/headsets/QUEST.md) | +| Valve Steam Frame (ALVR) | no | [docs/headsets/STEAM_FRAME.md](docs/headsets/STEAM_FRAME.md) | +| Valve Index | no | [docs/headsets/VALVE_INDEX.md](docs/headsets/VALVE_INDEX.md) | +| HTC Vive, Vive Pro, Vive Pro 2 | no | [docs/headsets/HTC_VIVE.md](docs/headsets/HTC_VIVE.md) | +| Sony PlayStation VR2 | no | [docs/headsets/PSVR2.md](docs/headsets/PSVR2.md) | -Since then, the Rift's Touch controllers show in the headset and track, all three sensors (one of them on USB 2) track the headset and the controllers in position and show in SteamVR as base stations, and the flashing coloured blocks in the VR composition are fixed. ABE VR and Raw Data play through SteamVR. +Each guide covers what you need, setting up and playing, what is checked in code, and what is still unknown. -Still to do; - -- Optimize the experience for other headsets, such as CV2, Oculus Rift S, Meta Quest 1/2/3, Valve Index, Steam Frame. -- BeamNG's crash in its driver. - -Contribution is very welcome because I do not own every headset. The experience will be optimized for an Oculus Rift Cv1 first, followed by conclusive updates from users and myself where applicable. - -## Oculus Rift CV1 - -A Rift CV1 on the Mac's own HDMI and USB, with its sensors and Touch controllers, needs no Oculus software. The -release's `metalvr-rift` script does each step: - -```sh -metalvr-rift install # from the unpacked release: into ~/.metalsharp/metalvr-runtime -metalvr-rift display # once per Mac: let macOS show the headset's display (asks for your password) -metalvr-rift setup # once: register MetalVR's driver with SteamVR in MetalSharp's Steam -metalvr-rift steamvr # start the Rift server, then SteamVR -metalvr-rift game APPID # start the Rift server, then a Steam game in VR -``` - -**[docs/RIFT_CV1.md](docs/RIFT_CV1.md)** walks through it: what you need, setting up, playing, and what to do when -the headset stays dark. SteamVR and its games need MetalSharp-Wine's Mullion at -[aaf2tbz/mullion@998c32a6](https://github.com/aaf2tbz/mullion/commit/998c32a61095985e7034c7b56553d79aa81c7193) or -later, which shares Direct3D 11 buffers between processes for VR; `metalvr-rift setup` switches that on -(`MULLION_METALVR=1`). - -## How it works - -``` -game.exe ─ OpenXR ─▶ metalvr_openxr.dll ─┐ ┌─▶ simulated headset (tests) - (Direct3D 11 via Mullion) ├─ loopback ─▶ metalvrd ──┤ -mac app ── OpenXR ─▶ libmetalvr_openxr ──┘ compose · encode └─▶ ALVR ─▶ headset - (Metal) -``` - -- One runtime source tree builds as a Windows DLL and a macOS dylib. -- Swapchain images never leave the GPU: under Wine they are Mullion shared textures, natively they are - IOSurfaces. -- `metalvrd` composes both eyes into one frame and encodes H.264 or HEVC with VideoToolbox. - -## Quick start +## Building ```sh scripts/fetch-deps.sh @@ -66,50 +34,31 @@ cmake -S . -B build/win64 -G Ninja -DCMAKE_TOOLCHAIN_FILE=cmake/mingw-x86_64.cma scripts/test-all.sh ``` -Then run the server and make MetalVR a Wine prefix's OpenXR runtime: - -```sh -build/native/daemon/metalvrd --trust 1234.client # the name the headset's ALVR app shows -scripts/install-prefix.sh /path/to/wine /path/to/prefix -``` - -[docs/BUILDING.md](docs/BUILDING.md) has the requirements and the rest. +Requirements, the optional parts (ALVR, libsurvive, Basalt, OpenComposite) and the test suites: +[docs/BUILDING.md](docs/BUILDING.md). ## Documentation | | | |---|---| +| [Headset guides](docs/headsets/README.md) | one per headset | | [Building, running and testing](docs/BUILDING.md) | requirements, builds, options, the test suites | -| [Architecture](docs/ARCHITECTURE.md) | the runtime, the server, the protocol between them, and what is not there yet | -| [MetalSharp-Wine integration](docs/METALSHARP_WINE.md) | what a Wine distribution ships and sets, and the contract with Mullion | -| [Headset streaming with ALVR](docs/ALVR.md) | building the driver, pairing a headset, how ALVR's API is used | -| [SteamVR games with OpenComposite](docs/OPENCOMPOSITE.md) | building `openvr_api.dll`, installing it for a game, known gaps | -| [Oculus Rift CV1 with MetalSharp](docs/RIFT_CV1.md) | the `metalvr-rift` script: from a release to a game in the headset | -| [Oculus Rift CV1 driver](docs/RIFT.md) | how the CV1 driver works: display, IMU, sensors, tracking, pacing | +| [Architecture](docs/ARCHITECTURE.md) | the runtime, the server, the protocol between them, what works and what is not there yet | | [SteamVR](docs/STEAMVR.md) | MetalVR's SteamVR driver: direct mode, requirements, its messages | - -## What works - -- OpenXR 1.0 and 1.1 core: sessions, spaces, swapchains, the frame loop, projection and quad layers, actions, - haptics. -- `XR_KHR_D3D11_enable` under Wine with Mullion, and `XR_KHR_metal_enable` on macOS. -- Hardware H.264 and HEVC encoding, at a headset's resolution and refresh rate. -- OpenVR applications through OpenComposite: rendering, legacy and action-manifest input, haptics. -- The ALVR driver, against ALVR's own client networking code on one machine. -- An Oculus Rift CV1 on the Mac's HDMI and USB: its display with lens correction, head rotation from its IMU, - position tracking with up to four Rift Sensors (USB 3 or USB 2), Touch controllers, and SteamVR through - MetalVR's own driver. - -Not yet: a standalone headset over ALVR in practice, Direct3D 12 / Vulkan / OpenGL applications, hand tracking, -audio, 32-bit games. +| [The direct-display driver](docs/RIFT.md) | how headsets on the Mac's own display are driven: display, IMU, sensors, pacing | +| [ALVR](docs/ALVR.md) | streaming to standalone headsets: the driver, pairing, MetalVR's patches | +| [OpenComposite](docs/OPENCOMPOSITE.md) | SteamVR games without SteamVR | +| [MetalSharp-Wine integration](docs/METALSHARP_WINE.md) | what a Wine distribution ships and sets, and the contract with Mullion | +| [Headset research](docs/HEADSETS.md) | each headset's specifications, how it is recognized, and the sources | ## Contributing -Game reports, bug reports and pull requests are welcome: see [CONTRIBUTING.md](CONTRIBUTING.md). Report -security issues privately, as [SECURITY.md](SECURITY.md) describes. +Game reports, headset reports, bug reports and pull requests are welcome: see [CONTRIBUTING.md](CONTRIBUTING.md). +Report security issues privately, as [SECURITY.md](SECURITY.md) describes. Known problems: BeamNG.drive crashes in its +graphics driver. ## License -MIT; see [LICENSE](LICENSE). MetalVR is written from scratch against public specifications and open-source -code. The components it builds against or beside keep their own licenses, and the patches under `patches/` -carry the license of what they patch: see [patches/README.md](patches/README.md). +MIT; see [LICENSE](LICENSE). MetalVR is written from scratch against public specifications and open-source code. The +components it builds against or beside keep their own licenses, and the patches under `patches/` carry the license of +what they patch: see [patches/README.md](patches/README.md). diff --git a/daemon/CMakeLists.txt b/daemon/CMakeLists.txt index aa22b35..697e3a9 100644 --- a/daemon/CMakeLists.txt +++ b/daemon/CMakeLists.txt @@ -22,10 +22,64 @@ add_executable(metalvrd encoder.mm shared_texture.mm driver_sim.cpp + headsets.cpp + foveation.cpp + json_value.cpp driver_rift.mm + direct_models.cpp + backend_rift_s.mm + rift_s_protocol.cpp + backend_lighthouse.mm + backend_psvr2.mm + psvr2_protocol.cpp + lighthouse_protocol.cpp + lighthouse_tracking.cpp + camera_passthrough.mm + camera_tracking.cpp + rift_s_tracking.cpp + rift_s_cameras.mm window_capture.mm ) +# The Rift S's camera tracking loads Basalt (scripts/build-basalt.sh) when it starts, as Monado does: looked for in +# this build's output first, then next to metalvrd. Nothing is linked. +set(METALVR_BASALT_LIBRARY_DIR "${CMAKE_SOURCE_DIR}/build/basalt/prefix/lib" CACHE PATH + "where scripts/build-basalt.sh puts libbasalt.dylib") +set_property(SOURCE camera_tracking.cpp APPEND PROPERTY COMPILE_DEFINITIONS + "METALVR_BASALT_DIR=\"${METALVR_BASALT_LIBRARY_DIR}\"") + +# Position tracking for the Lighthouse headsets with their base stations is libsurvive (MIT) on hidapi (BSD), which +# scripts/build-libsurvive.sh builds statically into build/libsurvive. Its parts register themselves as the program +# starts, so each is linked whole. Without it the Lighthouse driver tracks rotation only. +set(METALVR_LIBSURVIVE_DIR "${CMAKE_SOURCE_DIR}/build/libsurvive" CACHE PATH "scripts/build-libsurvive.sh's output") +if(EXISTS "${METALVR_LIBSURVIVE_DIR}/build/src/libsurvive.a") + set(libsurvive_default ON) +else() + set(libsurvive_default OFF) +endif() +option(METALVR_WITH_LIBSURVIVE "Track the Lighthouse headsets' position with libsurvive" ${libsurvive_default}) +if(METALVR_WITH_LIBSURVIVE) + set(ls "${METALVR_LIBSURVIVE_DIR}/build") + set(ls_src "${CMAKE_SOURCE_DIR}/external/libsurvive") + if(NOT EXISTS "${ls}/src/libsurvive.a") + message(FATAL_ERROR "METALVR_WITH_LIBSURVIVE needs ${ls}; run scripts/build-libsurvive.sh") + endif() + message(STATUS "metalvrd: with libsurvive for Lighthouse positions (${ls})") + set_source_files_properties(lighthouse_tracking.cpp PROPERTIES COMPILE_DEFINITIONS METALVR_LIBSURVIVE=1) + target_include_directories(metalvrd PRIVATE "${ls_src}/include/libsurvive" "${ls_src}/include" "${ls_src}/redist" + "${ls_src}/libs/cnmatrix/include" "${ls_src}/libs/cnkalman/include" + "${METALVR_LIBSURVIVE_DIR}/prefix/include/hidapi") + foreach(plugin driver_vive driver_simulator disambiguator_statebased poser_kalman_only poser_mpfit + driver_global_scene_solver) + target_link_options(metalvrd PRIVATE "-Wl,-force_load,${ls}/src/lib${plugin}.a") + endforeach() + target_link_libraries(metalvrd PRIVATE "${ls}/src/libsurvive.a" "${ls}/redist/libmpfit.a" + "${ls}/libs/cnkalman/src/libcnkalman.a" "${ls}/libs/cnmatrix/src/libcnmatrix.a" + "${METALVR_LIBSURVIVE_DIR}/prefix/lib/libhidapi.a") +else() + message(STATUS "metalvrd: without libsurvive (scripts/build-libsurvive.sh builds it); Lighthouse rotation only") +endif() + if(METALVR_WITH_ALVR) if(NOT EXISTS "${METALVR_ALVR_LIBRARY}") message(FATAL_ERROR "METALVR_WITH_ALVR needs ${METALVR_ALVR_LIBRARY}; run scripts/build-alvr.sh") @@ -131,6 +185,9 @@ target_link_libraries(metalvrd PRIVATE "-framework UniformTypeIdentifiers" "-framework AppKit" "-framework IOKit" + "-framework IOUSBHost" "-framework QuartzCore" "-framework ScreenCaptureKit" + "-framework AVFoundation" + z ) diff --git a/daemon/backend_lighthouse.mm b/daemon/backend_lighthouse.mm new file mode 100644 index 0000000..154fe3f --- /dev/null +++ b/daemon/backend_lighthouse.mm @@ -0,0 +1,479 @@ +// The Lighthouse headsets' USB side for the direct-display driver (docs/HEADSETS.md, phases F, G and H): finds an +// HTC Vive, Vive Pro, Vive Pro 2 or Valve Index, reads the calibration it keeps in its firmware (its lenses and its +// IMU), turns it on, switches a Vive Pro 2 to the mode asked for, and hands the driver its IMU samples, its IPD and +// whether it is worn, and its controllers' inputs and rotation through their Watchman receivers. The report layouts +// are in lighthouse_protocol.h. +// +// What it does not do yet: track position with the base stations (the Lighthouse pulses are not read; libsurvive +// would do it), so the head and the controllers have their rotation only, as a Rift CV1 without its sensors does. +#import +#import + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common.h" +#include "direct_backend.h" +#include "hid_util.h" +#include "lighthouse_protocol.h" +#include "lighthouse_tracking.h" + +namespace metalvr { +namespace { + +using namespace lighthouse; + +// The device's 48 MHz clock onto the server's: anchored at a report's arrival, carried by the ticks between samples, +// and anchored again when the two drift apart. +class TickClock { +public: + int64_t toNs(uint32_t ticks, int64_t received_ns) { + if (!anchored_) { + anchored_ = true; + base_ticks_ = ticks; + base_ns_ = received_ns; + return received_ns; + } + const uint32_t delta = ticks - base_ticks_; // wraps + int64_t t = base_ns_ + static_cast(delta / kTicksPerSecond * 1e9); + if (t > received_ns || received_ns - t > 50'000'000) { + base_ticks_ = ticks; + base_ns_ = received_ns; + t = received_ns; + } + return t; + } + void reset() { anchored_ = false; } + +private: + bool anchored_ = false; + uint32_t base_ticks_ = 0; + int64_t base_ns_ = 0; +}; + +struct Controller { + IOHIDDeviceRef device = nullptr; + bool gen2 = false; + bool configured = false; + ControllerConfig config; + ImuRanges ranges; + ControllerState state; + int hand = -1; + TickClock clock; + std::chrono::steady_clock::time_point retry{}; + uint8_t buffer[64]; + class LighthouseBackend *owner = nullptr; +}; + +class LighthouseBackend final : public DirectBackend { +public: + explicit LighthouseBackend(const LighthouseOptions &options) : options_(options) {} + const char *name() const override { return "lighthouse"; } + + void run(DirectHost &host, const std::atomic &running) override { + host_ = &host; + bool told = false; + while (running) { + if (!find()) { + if (!told) { + told = true; + logInfo("lighthouse: waiting for an HTC Vive, Vive Pro, Vive Pro 2 or Valve Index on USB"); + } + close(); + CFRunLoopRunInMode(kCFRunLoopDefaultMode, 1.0, false); + continue; + } + if (!configure()) { + close(); + CFRunLoopRunInMode(kCFRunLoopDefaultMode, 3.0, false); + continue; + } + drive(running); + close(); + } + } + + bool trackedPose(int device, TrackedPose &out) override { + std::lock_guard lock(tracking_mutex_); + return tracking_ && tracking_->pose(device, out); + } + + uint32_t trackingReferences(mvr_tracking_reference *out) override { + std::lock_guard lock(tracking_mutex_); + return tracking_ ? tracking_->references(out) : 0; + } + + void haptic(uint32_t hand, float duration_s, float frequency_hz, float amplitude) override { + std::lock_guard lock(controllers_mutex_); + for (const auto &c : controllers_) + if (c->configured && c->hand == static_cast(hand)) { + const auto pulse = hapticReport(duration_s <= 0 ? 0 : std::max(duration_s, 0.05f), frequency_hz, amplitude); + hid::setFeature(c->device, pulse.data(), pulse.size()); + } + } + + void controllerInput(Controller &c, uint32_t id, const uint8_t *raw, CFIndex length) { + uint8_t report[64]; + const size_t n = hid::withId(id, raw, length, report, sizeof(report)); + const int64_t received = nowNs(); + if (report[0] == kControllerDisconnect) { + logInfo("lighthouse: a controller went off"); + return; + } + ControllerImu imu[4]; + const size_t samples = decodeControllerReport(report, n, c.gen2, c.config, c.ranges, c.state, imu); + if (c.hand < 0) + return; + host_->directControllerInput(c.hand, inputOf(c), received); + for (size_t i = 0; i < samples; ++i) + host_->directControllerImu(c.hand, c.clock.toNs(imu[i].ticks, received), imu[i].accel, imu[i].gyro); + } + +private: + bool find() { + manager_ = IOHIDManagerCreate(kCFAllocatorDefault, kIOHIDOptionsTypeNone); + NSArray *matching = @[ @{@kIOHIDVendorIDKey : @(kHtcVendor)}, @{@kIOHIDVendorIDKey : @(kValveVendor)} ]; + IOHIDManagerSetDeviceMatchingMultiple(manager_, (__bridge CFArrayRef)matching); + IOHIDManagerOpen(manager_, kIOHIDOptionsTypeNone); + std::vector list = hid::devices(manager_); + std::vector ids; + for (IOHIDDeviceRef d : list) + ids.push_back({hid::vendor(d), hid::product(d)}); + model_ = identify(ids); + if (model_ == Model::None) { + hid::release(list); + return false; + } + for (IOHIDDeviceRef d : list) { + const uint16_t vendor = hid::vendor(d), product = hid::product(d); + const int interface = hid::interfaceNumber(d); + if (vendor == kValveVendor && (product == kViveSensors || product == kLhrSensors) && interface == 0 && !sensors_) + sensors_ = take(d); + else if (vendor == kHtcVendor && (product == kViveMainboard || product == kViveProMainboard || + product == kViveProMainboardB || product == kVivePro2Mainboard) && + interface == 0 && !mainboard_) + mainboard_ = take(d); + else if (vendor == kValveVendor && (product == kWatchmanGen1 || product == kWatchmanGen2) && interface == 0) { + if (IOHIDDeviceRef opened = take(d)) { + auto c = std::make_unique(); + c->device = opened; + c->gen2 = product == kWatchmanGen2; + c->owner = this; + controllers_.push_back(std::move(c)); + } + } + } + hid::release(list); + if (!sensors_) { + logInfo("lighthouse: %s on USB, but its IMU interface did not open", nameOf(model_)); + return false; + } + return true; + } + + IOHIDDeviceRef take(IOHIDDeviceRef d) { + if (IOHIDDeviceOpen(d, kIOHIDOptionsTypeNone) != kIOReturnSuccess) + return nullptr; + CFRetain(d); + return d; + } + + void close() { + { + std::lock_guard lock(controllers_mutex_); + for (auto &c : controllers_) { + IOHIDDeviceUnscheduleFromRunLoop(c->device, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + IOHIDDeviceClose(c->device, kIOHIDOptionsTypeNone); + CFRelease(c->device); + } + controllers_.clear(); + } + for (IOHIDDeviceRef *d : {&sensors_, &mainboard_}) { + if (!*d) + continue; + IOHIDDeviceUnscheduleFromRunLoop(*d, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + IOHIDDeviceClose(*d, kIOHIDOptionsTypeNone); + CFRelease(*d); + *d = nullptr; + } + if (manager_) { + IOHIDManagerClose(manager_, kIOHIDOptionsTypeNone); + CFRelease(manager_); + manager_ = nullptr; + } + have_sequence_ = false; + clock_.reset(); + } + + // a device's configuration: feature report 0x10 starts it, 0x11 reports carry it until an empty one + static std::optional readConfig(IOHIDDeviceRef device) { + uint8_t report[64]; + if (!hid::getFeature(device, kConfigStartReport, report, sizeof(report))) + return std::nullopt; + ConfigAssembler assembler; + for (int i = 0; i < 512 && !assembler.done(); ++i) { + const size_t n = hid::getFeature(device, kConfigReadReport, report, sizeof(report)); + if (!n || !assembler.add(report, n)) + return std::nullopt; + } + return assembler.json(); + } + + bool configure() { + const auto json = readConfig(sensors_); + const auto config = json ? parseHeadsetConfig(*json) : std::nullopt; + if (!config) { + logInfo("lighthouse: %s on USB, but its configuration (lenses, IMU) %s; it is not driven", nameOf(model_), + json ? "is not one MetalVR reads" : "could not be read"); + return false; + } + if (config->model != model_) + logInfo("lighthouse: USB said %s, its configuration says %s (\"%s\"); going by the configuration", + nameOf(model_), nameOf(config->model), config->model_number.c_str()); + model_ = config->model; + config_ = *config; + uint8_t report[64]; + const size_t n = hid::getFeature(sensors_, kImuRangeReport, report, sizeof(report)); + if (const auto ranges = parseImuRanges(report, n)) + ranges_ = *ranges; + else + logInfo("lighthouse: no IMU range report; taking 500 degrees/s and 4 g"); + + if (model_ == Model::VivePro2 && options_.pro2_mode >= 0 && !pro2_switched_) { + if (!mainboard_) { + logInfo("lighthouse: a Vive Pro 2's mode is switched through its HTC mainboard, which is not on USB"); + } else { + const Pro2Mode &mode = pro2Modes()[std::min(options_.pro2_mode, pro2Modes().size() - 1)]; + logInfo("lighthouse: switching the Vive Pro 2 to %dx%d at %.0f Hz%s; it restarts", mode.width, mode.height, + mode.refresh_hz, mode.needs_dsc ? " (needs Display Stream Compression, which macOS may not do)" : ""); + char dtd[16]; + std::snprintf(dtd, sizeof(dtd), "dtd,%d", mode.index); + for (const char *command : {"wireless,0", static_cast(dtd), "chipreset"}) { + const auto r = pro2Command(command); + hid::setFeature(mainboard_, r.data(), r.size()); + } + pro2_switched_ = true; + return false; // it goes away and comes back; look for it again + } + } + pro2_mode_ = pro2_switched_ ? options_.pro2_mode : 2; + host_->directModel(lighthouseModel(config_, pro2_mode_)); + logInfo("lighthouse: %s (%s), panel %dx%d per eye", nameOf(model_), config_.serial.c_str(), config_.eye_width, + config_.eye_height); + return true; + } + + void drive(const std::atomic &running) { + if (mainboard_ && model_ != Model::Index) { + const auto on = powerReport(true); + if (!hid::setFeature(mainboard_, on.data(), on.size())) + logInfo("lighthouse: the mainboard did not take the power-on report"); + } + host_->directOn(true); + // positions from the base stations, when libsurvive is built in; it opens the same devices alongside + if (lighthouseTrackingBuilt()) { + auto tracking = startLighthouseTracking(); + std::lock_guard lock(tracking_mutex_); + tracking_ = std::move(tracking); + } else { + logInfo("lighthouse: built without libsurvive (scripts/build-libsurvive.sh); rotation only"); + } + IOHIDDeviceRegisterInputReportCallback(sensors_, sensors_buffer_, sizeof(sensors_buffer_), onSensors, this); + IOHIDDeviceScheduleWithRunLoop(sensors_, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + if (mainboard_) { + IOHIDDeviceRegisterInputReportCallback(mainboard_, mainboard_buffer_, sizeof(mainboard_buffer_), onMainboard, this); + IOHIDDeviceScheduleWithRunLoop(mainboard_, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + } + gone_ = false; + IOHIDManagerRegisterDeviceRemovalCallback(manager_, onRemoved, this); + IOHIDManagerScheduleWithRunLoop(manager_, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + while (running && !gone_) { + CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, false); + serviceControllers(); + } + IOHIDManagerUnscheduleFromRunLoop(manager_, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + { + std::unique_ptr tracking; + { + std::lock_guard lock(tracking_mutex_); + tracking = std::move(tracking_); + } + tracking.reset(); // libsurvive stops and lets the devices go + } + if (mainboard_ && model_ != Model::Index) { + const auto off = powerReport(false); + hid::setFeature(mainboard_, off.data(), off.size()); + } + host_->directOn(false); + } + + // A receiver's controller answers once it is on: its configuration says which it is (a wand, a left or right + // Index controller). Until then it is asked again every two seconds. + void serviceControllers() { + const auto now = std::chrono::steady_clock::now(); + std::lock_guard lock(controllers_mutex_); + for (auto &c : controllers_) { + if (c->configured || now < c->retry) + continue; + c->retry = now + std::chrono::seconds(2); + const auto json = readConfig(c->device); + const auto config = json ? parseControllerConfig(*json) : std::nullopt; + if (!config || config->kind == ControllerKind::Unknown) { + if (json) + logInfo("lighthouse: a controller MetalVR does not know (\"%s\")", config ? config->model_number.c_str() : "?"); + continue; + } + c->config = *config; + uint8_t report[64]; + if (const auto ranges = parseImuRanges(report, hid::getFeature(c->device, kImuRangeReport, report, sizeof(report)))) + c->ranges = *ranges; + c->hand = config->kind == ControllerKind::IndexLeft ? 0 + : config->kind == ControllerKind::IndexRight ? 1 + : freeHand(); + c->configured = true; + IOHIDDeviceRegisterInputReportCallback(c->device, c->buffer, sizeof(c->buffer), onController, c.get()); + IOHIDDeviceScheduleWithRunLoop(c->device, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + host_->directControllerGrip(c->hand, mvr_pose{{0, 0, 0, 1}, {0, 0, 0}}, mvr_pose{{0, 0, 0, 1}, {0, 0, 0}}); + logInfo("lighthouse: %s controller: %s", c->hand == 0 ? "left" : "right", config->model_number.c_str()); + } + } + + // a Vive wand has no side: the first one on is the left hand, the next the right + int freeHand() const { + bool used[2] = {false, false}; + for (const auto &c : controllers_) + if (c->configured && c->hand >= 0) + used[c->hand] = true; + return used[0] ? 1 : 0; + } + + static DirectControllerInput inputOf(const Controller &c) { + const ControllerState &s = c.state; + DirectControllerInput in; + const auto bit = [&](int n) { return (s.buttons >> n) & 1; }; + in.trigger = s.trigger; + if (c.config.kind == ControllerKind::Wand) { + // trigger 0, trackpad touch 1, trackpad click 2, system 3, grip 4, menu 5 + in.buttons = (bit(0) ? MVR_BUTTON_TRIGGER : 0u) | (bit(2) ? MVR_BUTTON_TRACKPAD : 0u) | + (bit(3) ? MVR_BUTTON_SYSTEM : 0u) | (bit(4) ? MVR_BUTTON_SQUEEZE : 0u) | + (bit(5) ? MVR_BUTTON_MENU : 0u); + in.touches = bit(1) ? MVR_BUTTON_TRACKPAD : 0u; + in.squeeze = bit(4) ? 1.0f : 0.0f; + in.trackpad_x = s.pad_x; + in.trackpad_y = s.pad_y; + } else { + // trigger 0, trackpad touch 1, thumbstick click 2, system 3, A 4, B 5; touched: system 3, A 4, B 5, stick 6 + const auto touched = [&](int n) { return (s.touch >> n) & 1; }; + in.buttons = (bit(0) ? MVR_BUTTON_TRIGGER : 0u) | (bit(2) ? MVR_BUTTON_THUMBSTICK : 0u) | + (bit(3) ? MVR_BUTTON_SYSTEM : 0u) | (bit(4) ? MVR_BUTTON_PRIMARY : 0u) | + (bit(5) ? MVR_BUTTON_SECONDARY : 0u); + in.touches = (bit(1) ? MVR_BUTTON_TRACKPAD : 0u) | (touched(3) ? MVR_BUTTON_SYSTEM : 0u) | + (touched(4) ? MVR_BUTTON_PRIMARY : 0u) | (touched(5) ? MVR_BUTTON_SECONDARY : 0u) | + (touched(6) ? MVR_BUTTON_THUMBSTICK : 0u); + // the trackpad and the stick report on the same axes: the trackpad's while it is touched (Monado's choice) + if (bit(1)) { + in.trackpad_x = s.pad_x; + in.trackpad_y = s.pad_y; + } else { + in.thumbstick_x = s.pad_x; + in.thumbstick_y = s.pad_y; + } + if (s.has_fingers) { + in.has_fingers = true; + for (int f = 0; f < 4; ++f) + in.fingers[f] = s.fingers[f] / 255.0f; + in.squeeze = (s.fingers[1] + s.fingers[2] + s.fingers[3]) / (3.0f * 255.0f); + in.squeeze_force = s.squeeze_force / 255.0f; + in.trackpad_force = s.trackpad_force / 255.0f; + } + } + if (s.has_battery) { + in.has_battery = true; + in.battery = s.battery / 100.0f; + in.charging = s.charging; + } + return in; + } + + static void onSensors(void *context, IOReturn result, void *, IOHIDReportType, uint32_t id, uint8_t *report, + CFIndex length) { + if (result != kIOReturnSuccess) + return; + auto *self = static_cast(context); + uint8_t buffer[64]; + const size_t n = hid::withId(id, report, length, buffer, sizeof(buffer)); + if (buffer[0] != kImuReport) + return; // Lighthouse pulses (0x25, 0x27, 0x28): not used yet + ImuSample samples[3]; + const int64_t received = nowNs(); + const size_t count = parseImuReport(buffer, n, self->config_, self->ranges_, self->sequence_, self->have_sequence_, + samples); + for (size_t i = 0; i < count; ++i) + self->host_->directImu(self->clock_.toNs(samples[i].ticks, received), samples[i].accel, samples[i].gyro); + } + + static void onMainboard(void *context, IOReturn result, void *, IOHIDReportType, uint32_t id, uint8_t *report, + CFIndex length) { + if (result != kIOReturnSuccess) + return; + auto *self = static_cast(context); + uint8_t buffer[64]; + const size_t n = hid::withId(id, report, length, buffer, sizeof(buffer)); + const auto status = parseMainboardStatus(buffer, n); + if (!status) + return; + if (status->ipd > 0.05f && status->ipd < 0.08f) + self->host_->directIpd(status->ipd); + self->host_->directWorn(status->proximity > kWornProximity); + } + + static void onController(void *context, IOReturn result, void *, IOHIDReportType, uint32_t id, uint8_t *report, + CFIndex length) { + if (result != kIOReturnSuccess) + return; + auto *c = static_cast(context); + c->owner->controllerInput(*c, id, report, length); + } + + static void onRemoved(void *context, IOReturn, void *, IOHIDDeviceRef device) { + auto *self = static_cast(context); + if (device == self->sensors_ || device == self->mainboard_) { + logInfo("lighthouse: the headset went away from USB"); + self->gone_ = true; + } + } + + const LighthouseOptions options_; + DirectHost *host_ = nullptr; + IOHIDManagerRef manager_ = nullptr; + IOHIDDeviceRef sensors_ = nullptr, mainboard_ = nullptr; + std::mutex controllers_mutex_; + std::mutex tracking_mutex_; + std::unique_ptr tracking_; + std::vector> controllers_; + Model model_ = Model::None; + HeadsetConfig config_; + ImuRanges ranges_; + uint8_t sequence_ = 0; + bool have_sequence_ = false; + TickClock clock_; + bool gone_ = false; + bool pro2_switched_ = false; + int pro2_mode_ = 2; + uint8_t sensors_buffer_[64]; + uint8_t mainboard_buffer_[64]; +}; + +} // namespace + +std::unique_ptr makeLighthouseBackend(const LighthouseOptions &options) { + return std::make_unique(options); +} + +} // namespace metalvr diff --git a/daemon/backend_psvr2.mm b/daemon/backend_psvr2.mm new file mode 100644 index 0000000..dd59037 --- /dev/null +++ b/daemon/backend_psvr2.mm @@ -0,0 +1,543 @@ +// The PlayStation VR2's USB side for the direct-display driver (docs/HEADSETS.md, phase J), through Sony's PC +// adapter: finds the headset (054c:0cde) with IOUSBHost, reads its lens calibration and serial number, and hands +// the driver its IMU (2 kHz), whether it is worn, its IPD dial, and its position and heading, which the headset +// tracks itself from its cameras and sends finished. Its Sense controllers (054c:0e45 left, 0e46 right), paired with +// the Mac over Bluetooth, are read here too: buttons, stick, trigger, touches, battery, rotation from their IMU, +// and vibration. The report layouts are in psvr2_protocol.h. +// +// What it does not do: track the controllers' positions (the headset's cameras see their LEDs; no open driver +// solves that yet), eye tracking, passthrough, the headset's own rumble, or the adaptive triggers. +#import +#import +#import +#import + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common.h" +#include "direct_backend.h" +#include "hid_util.h" +#include "psvr2_protocol.h" + +namespace metalvr { +namespace { + +using namespace psvr2; + +constexpr int64_t kImuPeriodNs = static_cast(1e9 / kImuHz); +constexpr double kSlamFreshSeconds = 0.5; + +long registryNumber(io_registry_entry_t entry, CFStringRef key) { + long value = -1; + if (CFTypeRef ref = IORegistryEntryCreateCFProperty(entry, key, kCFAllocatorDefault, 0)) { + if (CFGetTypeID(ref) == CFNumberGetTypeID()) + CFNumberGetValue(static_cast(ref), kCFNumberLongType, &value); + CFRelease(ref); + } + return value; +} + +// one endpoint read over and over, each completed transfer handed to `handle` +struct Reader { + IOUSBHostPipe *pipe = nil; + NSMutableData *data = nil; + std::function handle; + const char *name = ""; +}; + +class Psvr2Backend final : public DirectBackend { +public: + const char *name() const override { return "psvr2"; } + + ~Psvr2Backend() override { closeControllers(); } + + void run(DirectHost &host, const std::atomic &running) override { + host_ = &host; + openControllers(); + bool told = false; + while (running) { + if (!open()) { + if (!told) { + told = true; + logInfo("psvr2: waiting for a PlayStation VR2 on USB (054c:0cde), through Sony's PC adapter"); + } + CFRunLoopRunInMode(kCFRunLoopDefaultMode, 1.0, false); + continue; + } + drive(running); + close(); + } + } + + bool trackedPose(int device, TrackedPose &out) override { + if (device != LighthouseTracking::kHead) + return false; + std::lock_guard guard(slam_mutex_); + if (!have_slam_) + return false; + out.orientation = head_.orientation; + out.position = head_.position; + out.velocity = velocity_; + out.position_tracked = (nowNs() - slam_received_ns_) / 1e9 < kSlamFreshSeconds; + return true; + } + + void haptic(uint32_t hand, float duration_s, float frequency_hz, float amplitude) override { + std::lock_guard guard(controllers_mutex_); + for (auto &entry : controllers_) { + Controller &c = *entry.second; + if (c.hand != static_cast(hand)) + continue; + c.vibration_until_ns = duration_s > 0 ? nowNs() + static_cast(std::max(duration_s, 0.02f) * 1e9) : 0; + c.vibration = duration_s > 0 ? std::clamp(amplitude, 0.0f, 1.0f) : 0.0f; + sendVibration(c); + } + } + +private: + // ---- the headset ---- + + bool open() { + CFMutableDictionaryRef matching = [IOUSBHostDevice createMatchingDictionaryWithVendorID:@(kVendor) + productID:@(kProduct) + bcdDevice:nil + deviceClass:nil + deviceSubclass:nil + deviceProtocol:nil + speed:nil + productIDArray:nil]; + io_service_t service = IOServiceGetMatchingService(kIOMainPortDefault, matching); // takes `matching` + if (!service) + return false; + queue_ = dispatch_queue_create("metalvr.psvr2.usb", DISPATCH_QUEUE_SERIAL); + NSError *error = nil; + // the vendor requests go to the device; if macOS's composite driver keeps it, through an interface instead + device_ = [[IOUSBHostDevice alloc] initWithIOService:service + options:IOUSBHostObjectInitOptionsNone + queue:queue_ + error:&error + interestHandler:nil]; + if (!device_) + logInfo("psvr2: the device itself did not open (%s); requests go through an interface", + error.localizedDescription.UTF8String); + for (int number : {kSlamInterface, kCameraInterface, kStatusInterface, kLedDetectorInterface, + kRelocalizerInterface, kVdInterface}) + if (!openInterface(service, number)) { + IOObjectRelease(service); + logInfo("psvr2: interface %d of the PlayStation VR2 did not open; is another program using it?", number); + close(); + return false; + } + IOObjectRelease(service); + return true; + } + + bool openInterface(io_service_t device, int number) { + io_iterator_t it = 0; + if (IORegistryEntryGetChildIterator(device, kIOServicePlane, &it) != KERN_SUCCESS) + return false; + io_service_t found = 0; + while (io_service_t child = IOIteratorNext(it)) { + if (!found && IOObjectConformsTo(child, "IOUSBHostInterface") && + registryNumber(child, CFSTR("bInterfaceNumber")) == number) + found = child; + else + IOObjectRelease(child); + } + IOObjectRelease(it); + if (!found) + return false; + NSError *error = nil; + IOUSBHostInterface *object = [[IOUSBHostInterface alloc] initWithIOService:found + options:IOUSBHostObjectInitOptionsNone + queue:queue_ + error:&error + interestHandler:nil]; + IOObjectRelease(found); + if (!object) + return false; + const NSUInteger alternate = number == kStatusInterface ? kStatusAltSetting : 0; + if (![object selectAlternateSetting:alternate error:&error]) { + logInfo("psvr2: interface %d refused alternate setting %lu: %s", number, static_cast(alternate), + error.localizedDescription.UTF8String); + return false; + } + interfaces_[number] = object; + return true; + } + + void close() { + for (Reader &r : readers_) + [r.pipe abortWithOption:IOUSBHostAbortOptionSynchronous error:nil]; + readers_.clear(); + for (auto &entry : interfaces_) + [entry.second destroy]; + interfaces_.clear(); + [device_ destroy]; + device_ = nil; + queue_ = nullptr; + std::lock_guard guard(slam_mutex_); + have_slam_ = false; + } + + // a vendor request (psvr2::ControlRequest) on the default pipe; how many bytes came back for a read, -1 if it + // failed + long control(const ControlRequest &r, std::vector *reply = nullptr) { + IOUSBDeviceRequest request{}; + request.bmRequestType = r.request_type; + request.bRequest = r.request; + request.wValue = r.value; + request.wIndex = r.index; + request.wLength = static_cast(r.data.size()); + const bool in = (r.request_type & 0x80) != 0; + NSMutableData *buffer = in ? [NSMutableData dataWithLength:r.data.size()] + : [NSMutableData dataWithBytes:r.data.data() length:r.data.size()]; + IOUSBHostObject *via = device_ ? static_cast(device_) : interfaces_[kStatusInterface]; + NSUInteger done = 0; + NSError *error = nil; + if (![via sendDeviceRequest:request data:buffer bytesTransferred:&done completionTimeout:0.2 error:&error]) { + logInfo("psvr2: request %02x for report %02x failed: %s", r.request, r.value, + error.localizedDescription.UTF8String); + return -1; + } + if (reply) + reply->assign(static_cast(buffer.bytes), static_cast(buffer.bytes) + done); + return static_cast(done); + } + + bool startReader(int interface, int endpoint, size_t size, const char *label, + std::function handle) { + IOUSBHostInterface *object = interfaces_[interface]; + NSError *error = nil; + Reader r; + r.pipe = [object copyPipeWithAddress:endpoint error:&error]; + r.data = r.pipe ? [object ioDataWithCapacity:size error:&error] : nil; + if (!r.data) { + logInfo("psvr2: the %s endpoint (%02x) did not open: %s", label, endpoint, error.localizedDescription.UTF8String); + return false; + } + r.data.length = size; + r.handle = std::move(handle); + r.name = label; + readers_.push_back(r); + return true; + } + + void submit(size_t index) { + Reader &r = readers_[index]; + IOUSBHostPipe *pipe = r.pipe; + NSMutableData *data = r.data; + NSError *error = nil; + const BOOL queued = + [pipe enqueueIORequestWithData:data + completionTimeout:0 + error:&error + completionHandler:^(IOReturn status, NSUInteger bytes) { + if (status == kIOReturnAborted) + return; + if (status == kIOReturnNoDevice || status == kIOReturnNotResponding || + status == kIOReturnNotOpen) { + gone_ = true; + return; + } + if (status == kIOReturnSuccess || status == kIOReturnUnderrun) + if (index < readers_.size()) + readers_[index].handle(static_cast(data.bytes), bytes); + if (!gone_ && index < readers_.size()) + submit(index); + }]; + if (!queued) { + logInfo("psvr2: reading the %s endpoint failed: %s", r.name, error.localizedDescription.UTF8String); + gone_ = true; + } + } + + void drive(const std::atomic &running) { + gone_ = false; + std::vector reply; + const ControlRequest info = firmwareInfoRequest(); + if (control(info, &reply) > 0) { + const auto payload = readPayload(reply.data(), reply.size(), 84); + if (const auto firmware = parseFirmwareInfo(payload.data(), payload.size())) + logInfo("psvr2: PlayStation VR2 %s, firmware %08x", firmware->pcb_id.c_str(), firmware->version); + } + std::array lens = defaultLensCalibration(); + if (control(lensCalibrationRequest(), &reply) > 0) { + const auto payload = readPayload(reply.data(), reply.size(), 0x100); + lens = lensCalibration(payload.data(), payload.size()); + } else { + logInfo("psvr2: no lens calibration from the headset; using Monado's default"); + } + host_->directModel(psvr2Model(lens)); + control(brightnessRequest(1.0f)); + // the headset tracks itself with its cameras: they run as Monado runs them, and what they send is read and + // dropped, as are the other streams Monado drains + control(cameraModeRequest(kCameraTracking)); + + last_imu_ns_ = 0; + const bool started = + startReader(kStatusInterface, kStatusEndpoint, kStatusTransferSize, "status", + [this](const uint8_t *p, size_t n) { status(p, n); }) && + startReader(kSlamInterface, kSlamEndpoint, kSlamTransferSize, "tracking", + [this](const uint8_t *p, size_t n) { slam(p, n); }) && + startReader(kCameraInterface, kCameraEndpoint, kCameraTransferSize, "camera", [](const uint8_t *, size_t) {}) && + startReader(kLedDetectorInterface, kLedDetectorEndpoint, kLedDetectorTransferSize, "LED detector", + [](const uint8_t *, size_t) {}) && + startReader(kRelocalizerInterface, kRelocalizerEndpoint, kRelocalizerTransferSize, "relocalizer", + [](const uint8_t *, size_t) {}) && + startReader(kVdInterface, kVdEndpoint, kVdTransferSize, "VD", [](const uint8_t *, size_t) {}); + if (!started) { + logInfo("psvr2: the PlayStation VR2's endpoints did not all open; looking for it again"); + CFRunLoopRunInMode(kCFRunLoopDefaultMode, 1.0, false); + return; + } + for (size_t i = 0; i < readers_.size(); i++) + submit(i); + host_->directOn(true); + for (int hand = 0; hand < 2; hand++) + host_->directControllerGrip(hand, senseGrip(hand), senseAim(hand)); + + while (running && !gone_) { + CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.05, false); + stopFinishedVibrations(); + } + if (gone_) + logInfo("psvr2: the PlayStation VR2 went away; looking for it again"); + else + control(cameraModeRequest(kCameraOff)); + host_->directOn(false); + } + + // on the USB queue + void status(const uint8_t *p, size_t n) { + const auto s = parseStatus(p, n); + if (!s) + return; + const int64_t received = nowNs(); + const size_t count = s->imu.size(); + for (size_t i = 0; i < count; i++) { + // the samples are half a millisecond apart, and the last arrived just now + const int64_t t = received - static_cast(count - 1 - i) * kImuPeriodNs; + if (t <= last_imu_ns_) + continue; + last_imu_ns_ = t; + host_->directImu(t, s->imu[i].accel, s->imu[i].gyro); + } + if (s->worn != worn_ || !told_worn_) { + worn_ = s->worn; + told_worn_ = true; + host_->directWorn(worn_); + logInfo(worn_ ? "psvr2: the PlayStation VR2 is worn" : "psvr2: the PlayStation VR2 is off the face"); + } + if (s->ipd_mm >= 55 && s->ipd_mm <= 75 && s->ipd_mm != ipd_mm_) { + ipd_mm_ = s->ipd_mm; + host_->directIpd(ipd_mm_ / 1000.0f); + logInfo("psvr2: IPD %u mm", ipd_mm_); + } + if (s->display == 0 && display_ready_) { + display_ready_ = false; + logInfo("psvr2: the headset's DisplayPort input lost its signal"); + } else if (s->display != 0 && !display_ready_) { + display_ready_ = true; + logInfo("psvr2: the headset's DisplayPort input has a signal"); + } + } + + // on the USB queue + void slam(const uint8_t *p, size_t n) { + std::lock_guard guard(slam_mutex_); + const auto s = parseSlam(p, n, slam_.orientation); + if (!s) + return; + const int64_t now = nowNs(); + const mvr_pose head = headFromSlam(s->pose); + if (have_slam_) { + const float dt = static_cast(static_cast(s->vts_us - slam_vts_us_)) / 1e6f; + if (dt > 0 && dt < 0.2f) + velocity_ = {(head.position.x - head_.position.x) / dt, (head.position.y - head_.position.y) / dt, + (head.position.z - head_.position.z) / dt}; + } else { + logInfo("psvr2: the headset is tracking itself"); + } + slam_ = s->pose; + slam_vts_us_ = s->vts_us; + head_ = head; + slam_received_ns_ = now; + have_slam_ = true; + } + + // ---- the Sense controllers ---- + + struct Controller { + IOHIDDeviceRef device = nullptr; + int hand = -1; + bool bluetooth = true; + std::optional calibration; + uint8_t buffer[128]; + uint8_t sequence = 0, counter = 0; + float vibration = 0; + int64_t vibration_until_ns = 0; + uint32_t last_ticks = 0; + bool have_ticks = false; + int64_t imu_ns = 0; + Psvr2Backend *owner = nullptr; + }; + + void openControllers() { + manager_ = IOHIDManagerCreate(kCFAllocatorDefault, kIOHIDOptionsTypeNone); + NSArray *matches = @[ + @{@kIOHIDVendorIDKey : @(kVendor), @kIOHIDProductIDKey : @(kSenseLeft)}, + @{@kIOHIDVendorIDKey : @(kVendor), @kIOHIDProductIDKey : @(kSenseRight)} + ]; + IOHIDManagerSetDeviceMatchingMultiple(manager_, (__bridge CFArrayRef)matches); + IOHIDManagerRegisterDeviceMatchingCallback(manager_, onController, this); + IOHIDManagerRegisterDeviceRemovalCallback(manager_, onControllerGone, this); + IOHIDManagerScheduleWithRunLoop(manager_, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + IOHIDManagerOpen(manager_, kIOHIDOptionsTypeNone); + } + + void closeControllers() { + if (!manager_) + return; + std::lock_guard guard(controllers_mutex_); + for (auto &entry : controllers_) + IOHIDDeviceRegisterInputReportCallback(entry.second->device, entry.second->buffer, sizeof(entry.second->buffer), + nullptr, nullptr); + controllers_.clear(); + IOHIDManagerClose(manager_, kIOHIDOptionsTypeNone); + CFRelease(manager_); + manager_ = nullptr; + } + + static void onController(void *context, IOReturn, void *, IOHIDDeviceRef device) { + static_cast(context)->addController(device); + } + static void onControllerGone(void *context, IOReturn, void *, IOHIDDeviceRef device) { + auto *self = static_cast(context); + std::lock_guard guard(self->controllers_mutex_); + if (self->controllers_.erase(device)) + logInfo("psvr2: a Sense controller went away"); + } + + void addController(IOHIDDeviceRef device) { + auto c = std::make_unique(); + c->device = device; + c->owner = this; + c->hand = hid::product(device) == kSenseLeft ? 0 : 1; + NSString *transport = (__bridge NSString *)IOHIDDeviceGetProperty(device, CFSTR(kIOHIDTransportKey)); + c->bluetooth = !transport || ![transport isEqualToString:@"USB"]; + // the calibration, in two parts; reading it also switches the controller into the reports read here + uint8_t a[64] = {}, b[64] = {}; + if (hid::getFeature(device, kSenseCalibrationReport, a, sizeof(a)) == sizeof(a) && + hid::getFeature(device, kSenseCalibrationReport, b, sizeof(b)) == sizeof(b)) + c->calibration = parseSenseCalibration(a, b); + logInfo("psvr2: %s Sense controller over %s%s", c->hand == 0 ? "left" : "right", + c->bluetooth ? "Bluetooth" : "USB", c->calibration ? "" : " (no IMU calibration: rotation off)"); + IOHIDDeviceRegisterInputReportCallback(device, c->buffer, sizeof(c->buffer), onReport, c.get()); + std::lock_guard guard(controllers_mutex_); + controllers_[device] = std::move(c); + } + + static void onReport(void *context, IOReturn result, void *, IOHIDReportType, uint32_t id, uint8_t *report, + CFIndex length) { + if (result != kIOReturnSuccess) + return; + auto *c = static_cast(context); + c->owner->controllerReport(*c, id, report, length); + } + + void controllerReport(Controller &c, uint32_t id, const uint8_t *raw, CFIndex length) { + uint8_t report[128]; + const size_t n = hid::withId(id, raw, length, report, sizeof(report)); + const auto s = parseSenseReport(report, n, c.hand == 0); + if (!s) + return; + const int64_t received = nowNs(); + DirectControllerInput in; + in.buttons = s->buttons; + in.touches = s->touches; + in.trigger = s->trigger; + in.squeeze = s->squeeze; + in.thumbstick_x = s->stick_x; + in.thumbstick_y = s->stick_y; + in.has_battery = s->battery_valid; + in.battery = s->battery; + in.charging = s->charging; + host_->directControllerInput(c.hand, in, received); + if (!c.calibration) + return; + // the IMU's own clock, thirds of a microsecond, laid on the Mac's from the first report + if (!c.have_ticks) { + c.have_ticks = true; + c.imu_ns = received; + } else { + c.imu_ns += static_cast(static_cast(s->imu_ticks - c.last_ticks)) * 1000 / 3; + // never ahead of now, nor far behind it + c.imu_ns = std::clamp(c.imu_ns, received - 20'000'000, received); + } + c.last_ticks = s->imu_ticks; + float accel[3], gyro[3]; + senseImu(*s, *c.calibration, accel, gyro); + host_->directControllerImu(c.hand, c.imu_ns, accel, gyro); + } + + // controllers_mutex_ held + void sendVibration(Controller &c) { + if (!c.bluetooth) + return; // over USB the controller takes another report, which is not sent here + const auto report = senseOutputReport(c.vibration, c.sequence, c.counter++, + static_cast(nowNs() / 1000)); + c.sequence = (c.sequence + 1) % 16; + IOHIDDeviceSetReport(c.device, kIOHIDReportTypeOutput, report[0], report.data(), + static_cast(report.size())); + } + + void stopFinishedVibrations() { + std::lock_guard guard(controllers_mutex_); + const int64_t now = nowNs(); + for (auto &entry : controllers_) { + Controller &c = *entry.second; + if (c.vibration > 0 && now >= c.vibration_until_ns) { + c.vibration = 0; + sendVibration(c); + } + } + } + + DirectHost *host_ = nullptr; + dispatch_queue_t queue_ = nullptr; + IOUSBHostDevice *device_ = nil; + std::map interfaces_; + std::vector readers_; + std::atomic gone_{false}; + int64_t last_imu_ns_ = 0; + bool worn_ = false, told_worn_ = false, display_ready_ = false; + uint8_t ipd_mm_ = 0; + + std::mutex slam_mutex_; + bool have_slam_ = false; + mvr_pose slam_{{0, 0, 0, 1}, {0, 0, 0}}, head_{{0, 0, 0, 1}, {0, 0, 0}}; + mvr_vec3 velocity_{}; + uint32_t slam_vts_us_ = 0; + int64_t slam_received_ns_ = 0; + + IOHIDManagerRef manager_ = nullptr; + std::mutex controllers_mutex_; + std::map> controllers_; +}; + +} // namespace + +std::unique_ptr makePsvr2Backend() { return std::make_unique(); } + +} // namespace metalvr diff --git a/daemon/backend_rift_s.mm b/daemon/backend_rift_s.mm new file mode 100644 index 0000000..97e0962 --- /dev/null +++ b/daemon/backend_rift_s.mm @@ -0,0 +1,420 @@ +// The Oculus Rift S's USB side for the direct-display driver (docs/HEADSETS.md, phase I): finds the headset +// (2833:0051) and opens its three HID interfaces, reads its IMU's scales and calibration, turns it on and keeps it +// on, and hands the driver its IMU samples, whether it is worn, and its Touch controllers' buttons and rotation. +// The report layouts are in rift_s_protocol.h. +// +// The headset's position comes from its cameras (rift_s_tracking.h): Basalt, when scripts/build-basalt.sh has built +// it and macOS lets metalvrd read the cameras; otherwise the headset tracks rotation only. What it does not do: +// track the controllers' positions (no open driver does), or play haptics on them (the command is not known). +#import +#import + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common.h" +#include "direct_backend.h" +#include "hid_util.h" +#include "rift_s_cameras.h" +#include "rift_s_protocol.h" +#include "rift_s_tracking.h" + +namespace metalvr { +namespace { + +using namespace rift_s; + +constexpr float kPi = 3.14159265358979323846f; + +class RiftSBackend final : public DirectBackend { +public: + const char *name() const override { return "rift-s"; } + + bool trackedPose(int device, TrackedPose &out) override { + if (device != LighthouseTracking::kHead) + return false; + std::lock_guard guard(tracking_mutex_); + return tracking_ && tracking_->pose(out); + } + + void run(DirectHost &host, const std::atomic &running) override { + host_ = &host; + bool told = false; + while (running) { + if (!open()) { + if (!told) { + told = true; + logInfo("rift: waiting for a Rift S on USB (2833:0051, interfaces 6, 7 and 8)"); + } + CFRunLoopRunInMode(kCFRunLoopDefaultMode, 1.0, false); + continue; + } + drive(running); + close(); + } + } + +private: + bool open() { + IOHIDManagerRef manager = IOHIDManagerCreate(kCFAllocatorDefault, kIOHIDOptionsTypeNone); + IOHIDManagerSetDeviceMatching(manager, + (__bridge CFDictionaryRef) @{@kIOHIDVendorIDKey : @(kVendor), @kIOHIDProductIDKey : @(kProduct)}); + IOHIDManagerOpen(manager, kIOHIDOptionsTypeNone); + std::vector list = hid::devices(manager); + for (IOHIDDeviceRef d : list) { + const int interface = hid::interfaceNumber(d); + IOHIDDeviceRef *slot = interface == kInterfaceHmd ? &hmd_ : interface == kInterfaceStatus ? &status_ + : interface == kInterfaceControllers ? &controllers_ : nullptr; + if (!slot || *slot || IOHIDDeviceOpen(d, kIOHIDOptionsTypeNone) != kIOReturnSuccess) + continue; + CFRetain(d); + *slot = d; + } + hid::release(list); + manager_ = manager; + if (!hmd_) { + close(); + return false; + } + if (!status_ || !controllers_) + logInfo("rift: the Rift S's %s interface did not open; %s", !status_ ? "status" : "controller", + !status_ ? "the screen follows nothing and stays on" : "its Touch controllers are off"); + return true; + } + + void close() { + for (IOHIDDeviceRef *d : {&hmd_, &status_, &controllers_}) { + if (!*d) + continue; + IOHIDDeviceUnscheduleFromRunLoop(*d, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + IOHIDDeviceClose(*d, kIOHIDOptionsTypeNone); + CFRelease(*d); + *d = nullptr; + } + if (manager_) { + IOHIDManagerClose(manager_, kIOHIDOptionsTypeNone); + CFRelease(manager_); + manager_ = nullptr; + } + controller_state_.clear(); + devices_.clear(); + imu_configs_.clear(); + } + + void drive(const std::atomic &running) { + uint8_t buffer[256]; + size_t n = hid::getFeature(hmd_, kReportDeviceInfo, buffer, sizeof(buffer)); + if (const auto info = parseDeviceInfo(buffer, n)) + logInfo("rift: Rift S panel %ux%u at %u Hz", info->width, info->height, info->refresh_hz); + else + logInfo("rift: the Rift S did not report its panel; taking 1440x2560 at 80 Hz"); + n = hid::getFeature(hmd_, kReportImuConfig, buffer, sizeof(buffer)); + if (const auto config = parseImuConfig(buffer, n)) + imu_config_ = *config; + else + logInfo("rift: the Rift S did not report its IMU's scales; using defaults"); + const std::string json = readFirmwareBlock(kFirmwareImuCalibration); + if (const auto calibration = parseImuCalibration(json)) + calibration_ = *calibration; + else + logInfo("rift: no IMU calibration from the Rift S (%zu bytes read); its rotation may drift more", json.size()); + startTracking(); + + host_->directModel(riftSModel()); + for (const std::vector &report : enableReports(true)) + if (!hid::setFeature(hmd_, report.data(), report.size())) + logInfo("rift: the Rift S did not take report 0x%02x", report[0]); + if (!status_) { + // no proximity reports: the screen stays on + const auto on = screenReport(true); + hid::setFeature(hmd_, on.data(), on.size()); + } + host_->directOn(true); + for (int hand = 0; hand < 2; ++hand) { + // Touch's grip and aim are its own frame turned 40 degrees up about x (Monado's rift_s_controller.c) + const float half = 20.0f * kPi / 180.0f; + const mvr_pose up{{std::sin(half), 0, 0, std::cos(half)}, {0, 0, 0}}; + host_->directControllerGrip(hand, up, up); + } + + IOHIDDeviceRef interfaces[3] = {hmd_, status_, controllers_}; + for (int i = 0; i < 3; ++i) { + if (!interfaces[i]) + continue; + // a buffer each: IOHID fills it before the callback + IOHIDDeviceRegisterInputReportCallback(interfaces[i], input_buffers_[i], sizeof(input_buffers_[i]), onInput, this); + IOHIDDeviceScheduleWithRunLoop(interfaces[i], CFRunLoopGetCurrent(), kCFRunLoopDefaultMode); + } + last_imu_ns_ = 0; + auto last_camera_look = std::chrono::steady_clock::now() - std::chrono::seconds(10); + bool told_cameras = false; + auto last_keep_alive = std::chrono::steady_clock::now() - std::chrono::seconds(2); + gone_ = false; + while (running && !gone_) { + CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.05, false); + const auto now = std::chrono::steady_clock::now(); + if (now - last_keep_alive >= std::chrono::milliseconds(kKeepAliveMs)) { + last_keep_alive = now; + const auto keep = keepAliveReport(); + if (!hid::setFeature(hmd_, keep.data(), keep.size()) && ++keep_alive_failures_ > 5) { + logInfo("rift: the Rift S stopped answering; looking for it again"); + gone_ = true; + } + } + serviceControllers(); + if (tracking() && !cameras_ && now - last_camera_look >= std::chrono::seconds(2)) { + // the camera device appears once the headset is on + last_camera_look = now; + startCameras(); + if (!cameras_ && !told_cameras) { + told_cameras = true; + logInfo("rift: no \"Rift S Sensor\" camera among macOS's yet; looking again every 2 s"); + } + } + uint16_t exposure; + uint8_t gain; + if (tracking() && cameras_ && tracking()->exposureChange(exposure, gain)) { + const auto report = cameraReport(true, exposure, gain, true); + hid::setFeature(hmd_, report.data(), report.size()); + } + } + cameras_.reset(); // no frame arrives after this + { + std::lock_guard guard(tracking_mutex_); + tracking_.reset(); + } + for (const std::vector &report : enableReports(false)) + hid::setFeature(hmd_, report.data(), report.size()); + host_->directOn(false); + } + + RiftSTracking *tracking() { + std::lock_guard guard(tracking_mutex_); + return tracking_.get(); + } + + // Basalt with this headset's cameras' calibration (firmware block 0x0f) and its IMU's + void startTracking() { + const std::string json = readFirmwareBlock(kFirmwareCameraCalibration); + const auto cameras = parseCameraCalibration(json); + if (!cameras) { + logInfo("rift: no camera calibration from the Rift S (%zu bytes read); it tracks rotation only", json.size()); + return; + } + std::string why; + auto tracking = RiftSTracking::start(imu_config_, calibration_, *cameras, &why); + if (!tracking) { + logInfo("rift: the Rift S tracks rotation only: %s", why.c_str()); + return; + } + std::lock_guard guard(tracking_mutex_); + tracking_ = std::move(tracking); + } + + void startCameras() { + RiftSTracking *tracking = this->tracking(); + cameras_ = startRiftSCameras([tracking](const uint8_t *grey, uint32_t width, uint32_t height, size_t stride) { + tracking->frame(grey, width, height, stride); + }); + } + + // A firmware block, read 56 bytes at a time after its 12-byte header (OpenHMD's rift_s_read_firmware_block). + std::string readFirmwareBlock(uint8_t block) { + uint8_t reply[64]; + if (!readFirmware(block, 0, 0x0c, reply)) + return {}; + const uint32_t length = firmwareBlockLength(reply, sizeof(reply)); + std::string out; + for (uint32_t at = 0; at < length; at += 56) { + const uint8_t chunk = static_cast(std::min(56, length - at)); + if (!readFirmware(block, at + 0x0c, chunk, reply)) + return {}; + out.append(reinterpret_cast(reply + 8), chunk); + } + return out; + } + + bool readFirmware(uint8_t block, uint32_t offset, uint8_t length, uint8_t reply[64]) { + const auto request = firmwareRequest(block, offset, length); + bool send = true; + for (int attempt = 0; attempt < 20; ++attempt) { + if (send && !hid::setFeature(hmd_, request.data(), request.size())) + return false; + const size_t n = hid::getFeature(hmd_, kReportFirmwareBlock, reply, 64); + if (n && firmwareReplyMatches(reply, n, request)) + return true; + // 1: the read is being done, ask again for its result; 0: the request was not taken, send it again + send = n < 2 || reply[1] == 0x00; + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + } + return false; + } + + static void onInput(void *context, IOReturn result, void *, IOHIDReportType, uint32_t id, uint8_t *report, + CFIndex length) { + if (result == kIOReturnSuccess) + static_cast(context)->input(id, report, length); + } + + void input(uint32_t id, const uint8_t *raw, CFIndex length) { + uint8_t report[256]; + const size_t n = hid::withId(id, raw, length, report, sizeof(report)); + const int64_t received = nowNs(); + keep_alive_failures_ = 0; + if (report[0] == kInHmd) { + const auto parsed = parseHmdReport(report, n, imu_config_, calibration_); + if (!parsed || parsed->count == 0) + return; + // the samples are a tick apart and the last arrived just now + if (RiftSTracking *tracking = this->tracking()) + tracking->imu(*parsed); + const int64_t tick = 1'000'000'000 / std::max(imu_config_.hz, 1); + for (int i = 0; i < parsed->count; ++i) { + const int64_t t = received - (parsed->count - 1 - i) * tick; + if (t <= last_imu_ns_) + continue; + last_imu_ns_ = t; + host_->directImu(t, parsed->samples[i].accel, parsed->samples[i].gyro); + } + } else if (report[0] == kInStatus) { + const auto worn = parseStatus(report, n); + if (!worn || *worn == worn_) + return; + worn_ = *worn; + // the screen follows the proximity sensor, as Oculus's runtime has it + const auto screen = screenReport(worn_); + hid::setFeature(hmd_, screen.data(), screen.size()); + host_->directWorn(worn_); + logInfo(worn_ ? "rift: the Rift S is worn; its screen is on" : "rift: the Rift S is off the face; screen off"); + } else if (report[0] == kInController) { + ControllerState probe; + const uint64_t device = parseControllerReport(report, n, probe); + if (device == 0) + return; + ControllerState &state = controller_state_[device]; + parseControllerReport(report, n, state); + const int hand = handOf(device); + if (hand < 0) + return; + const TouchValues v = touchValues(state); + DirectControllerInput in; + in.buttons = (v.primary ? MVR_BUTTON_PRIMARY : 0u) | (v.secondary ? MVR_BUTTON_SECONDARY : 0u) | + (v.stick ? MVR_BUTTON_THUMBSTICK : 0u) | + (v.menu ? (hand == 0 ? MVR_BUTTON_MENU : MVR_BUTTON_SYSTEM) : 0u); + in.touches = (v.primary_touch ? MVR_BUTTON_PRIMARY : 0u) | (v.secondary_touch ? MVR_BUTTON_SECONDARY : 0u) | + (v.stick_touch ? MVR_BUTTON_THUMBSTICK : 0u) | (v.trigger_touch ? MVR_BUTTON_TRIGGER : 0u); + in.trigger = std::clamp(v.trigger, 0.0f, 1.0f); + in.squeeze = std::clamp(v.grip, 0.0f, 1.0f); + in.thumbstick_x = std::clamp(v.stick_x, -1.0f, 1.0f); + in.thumbstick_y = std::clamp(v.stick_y, -1.0f, 1.0f); + host_->directControllerInput(hand, in, received); + const auto config = imu_configs_.find(device); + if (probe.has_imu && config != imu_configs_.end()) { + float accel[3], gyro[3]; + controllerImu(state, config->second, accel, gyro); + host_->directControllerImu(hand, received, accel, gyro); + } + } + } + + int handOf(uint64_t device) { + for (const Device &d : devices_) + if (d.id == device) + return d.type == kLeftController ? 0 : d.type == kRightController ? 1 : -1; + wanted_devices_ = true; + return -1; + } + + // the radio's slow work, between reports: which controller is which hand, and each one's IMU scales + void serviceControllers() { + if (!controllers_ && !hmd_) + return; + const auto now = std::chrono::steady_clock::now(); + if (wanted_devices_ && now - devices_read_ > std::chrono::milliseconds(500)) { + devices_read_ = now; + uint8_t buffer[200]; + const size_t n = hid::getFeature(hmd_, kReportDevices, buffer, sizeof(buffer)); + const std::vector list = parseDevices(buffer, n); + if (!list.empty()) { + devices_ = list; + wanted_devices_ = false; + for (const Device &d : devices_) + if (d.type == kLeftController || d.type == kRightController) + logInfo("rift: %s Touch controller %016llx", d.type == kLeftController ? "left" : "right", + static_cast(d.id)); + } + } + for (const Device &d : devices_) { + if (imu_configs_.count(d.id) || (d.type != kLeftController && d.type != kRightController)) + continue; + if (now < radio_retry_[d.id]) + continue; + if (const auto config = readControllerImuConfig(d.id)) { + imu_configs_[d.id] = *config; + logInfo("rift: the %s Touch controller's IMU is on", d.type == kLeftController ? "left" : "right"); + } else { + radio_retry_[d.id] = now + std::chrono::seconds(2); // asleep, or not answering yet + } + } + } + + // one radio command and its reply (OpenHMD's rift_s_radio_update): sent with 0x12, its reply polled on 0x0b + // until the radio is no longer busy and its sequence number has moved + std::optional readControllerImuConfig(uint64_t device) { + const auto command = radioCommand(device, kControllerConfigRequest, sizeof(kControllerConfigRequest)); + if (!hid::setFeature(hmd_, command.data(), 61)) + return std::nullopt; + uint8_t reply[200]; + for (int attempt = 0; attempt < 50; ++attempt) { + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + const size_t n = hid::getFeature(hmd_, kReportRadioResponse, reply, sizeof(reply)); + if (n < 4) + return std::nullopt; + const int seqnum = reply[1]; + const bool busy = reply[2] != 0; + if (busy || seqnum == last_radio_seqnum_) { + if (busy) + last_radio_seqnum_ = seqnum; + continue; + } + last_radio_seqnum_ = seqnum; + return parseControllerImuConfig(reply + 3, n - 3); + } + return std::nullopt; + } + + DirectHost *host_ = nullptr; + IOHIDManagerRef manager_ = nullptr; + IOHIDDeviceRef hmd_ = nullptr, status_ = nullptr, controllers_ = nullptr; + uint8_t input_buffers_[3][256]; + ImuConfig imu_config_; + ImuCalibration calibration_; + int64_t last_imu_ns_ = 0; + bool worn_ = false; + bool gone_ = false; + int keep_alive_failures_ = 0; + std::map controller_state_; + std::vector devices_; + bool wanted_devices_ = true; + std::chrono::steady_clock::time_point devices_read_{}; + std::map imu_configs_; + std::map radio_retry_; + int last_radio_seqnum_ = -1; + std::mutex tracking_mutex_; + std::unique_ptr tracking_; + std::unique_ptr cameras_; +}; + +} // namespace + +std::unique_ptr makeRiftSBackend() { return std::make_unique(); } + +} // namespace metalvr diff --git a/daemon/camera_passthrough.h b/daemon/camera_passthrough.h new file mode 100644 index 0000000..7399539 --- /dev/null +++ b/daemon/camera_passthrough.h @@ -0,0 +1,26 @@ +// The headset's own cameras, shown in its home (docs/HEADSETS.md, phase F): a Valve Index's front cameras (one USB +// video device giving both pictures side by side), or a Vive's or Vive Pro's, captured with AVFoundation as +// macOS's camera stack owns them. The pictures are drawn as straight-ahead views of a fixed field of view: the +// cameras' own calibration (in the headset's configuration) is not applied yet. +#pragma once + +#include + +#include + +namespace metalvr { + +class CameraPassthrough { +public: + virtual ~CameraPassthrough() = default; + // the newest picture, retained for the caller; null until one comes + virtual CVPixelBufferRef latest() = 0; + // whether the picture holds both eyes side by side (an Index's), or one picture for both + virtual bool stereo() const = 0; +}; + +// The camera of the headset `headset` (mvr_headset), started; null if macOS has no such camera or does not let +// metalvrd use it (System Settings, Privacy & Security, Camera). +std::unique_ptr startCameraPassthrough(uint32_t headset); + +} // namespace metalvr diff --git a/daemon/camera_passthrough.mm b/daemon/camera_passthrough.mm new file mode 100644 index 0000000..8c71bd5 --- /dev/null +++ b/daemon/camera_passthrough.mm @@ -0,0 +1,130 @@ +#import "camera_passthrough.h" + +#import + +#include +#include +#include + +#include "common.h" +#include "metalvr_protocol.h" + +@interface MetalVRCameraSink : NSObject { +@public + std::mutex *mutex; + CVPixelBufferRef *latest; +} +@end + +@implementation MetalVRCameraSink +- (void)captureOutput:(AVCaptureOutput *)output + didOutputSampleBuffer:(CMSampleBufferRef)sample + fromConnection:(AVCaptureConnection *)connection { + CVPixelBufferRef buffer = CMSampleBufferGetImageBuffer(sample); + if (!buffer) + return; + CVPixelBufferRetain(buffer); + std::lock_guard lock(*mutex); + if (*latest) + CVPixelBufferRelease(*latest); + *latest = buffer; +} +@end + +namespace metalvr { +namespace { + +class Capture final : public CameraPassthrough { +public: + bool start(AVCaptureDevice *device, bool stereo) { + stereo_ = stereo; + NSError *error = nil; + AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:device error:&error]; + if (!input) { + logInfo("passthrough: cannot open %s: %s", device.localizedName.UTF8String, + error.localizedDescription.UTF8String); + return false; + } + session_ = [AVCaptureSession new]; + if (![session_ canAddInput:input]) + return false; + [session_ addInput:input]; + AVCaptureVideoDataOutput *output = [AVCaptureVideoDataOutput new]; + output.videoSettings = @{(id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA), + (id)kCVPixelBufferMetalCompatibilityKey : @YES}; + output.alwaysDiscardsLateVideoFrames = YES; + sink_ = [MetalVRCameraSink new]; + sink_->mutex = &mutex_; + sink_->latest = &latest_; + queue_ = dispatch_queue_create("metalvr.passthrough", DISPATCH_QUEUE_SERIAL); + [output setSampleBufferDelegate:sink_ queue:queue_]; + if (![session_ canAddOutput:output]) + return false; + [session_ addOutput:output]; + [session_ startRunning]; + logInfo("passthrough: %s (%s)", device.localizedName.UTF8String, stereo ? "both eyes side by side" : "one picture"); + return true; + } + + ~Capture() override { + [session_ stopRunning]; + std::lock_guard lock(mutex_); + if (latest_) + CVPixelBufferRelease(latest_); + latest_ = nullptr; + } + + CVPixelBufferRef latest() override { + std::lock_guard lock(mutex_); + if (latest_) + CVPixelBufferRetain(latest_); + return latest_; + } + + bool stereo() const override { return stereo_; } + +private: + AVCaptureSession *session_ = nil; + MetalVRCameraSink *sink_ = nil; + dispatch_queue_t queue_ = nullptr; + std::mutex mutex_; + CVPixelBufferRef latest_ = nullptr; + bool stereo_ = false; +}; + +} // namespace + +std::unique_ptr startCameraPassthrough(uint32_t headset) { + // how each headset's camera names itself to macOS (an Index's is Etron's "3D Camera"; guessed for the Vives) + std::vector names; + bool stereo = false; + if (headset == MVR_HEADSET_VALVE_INDEX) + names = {"3D Camera", "Etron", "Index"}, stereo = true; + else if (headset == MVR_HEADSET_VIVE_PRO || headset == MVR_HEADSET_VIVE_PRO_2) + names = {"VIVE Pro", "Vive Pro", "HTC"}, stereo = true; + else if (headset == MVR_HEADSET_VIVE) + names = {"VIVE", "Vive", "HTC"}; + else + return nullptr; + if ([AVCaptureDevice authorizationStatusForMediaType:AVMediaTypeVideo] == AVAuthorizationStatusDenied) { + logInfo("passthrough: macOS does not let metalvrd use cameras (System Settings, Privacy & Security, Camera)"); + return nullptr; + } + NSArray *types = @[ AVCaptureDeviceTypeExternal ]; + AVCaptureDeviceDiscoverySession *discovery = + [AVCaptureDeviceDiscoverySession discoverySessionWithDeviceTypes:types + mediaType:AVMediaTypeVideo + position:AVCaptureDevicePositionUnspecified]; + for (AVCaptureDevice *device in discovery.devices) + for (const std::string &name : names) + if ([device.localizedName rangeOfString:@(name.c_str()) options:NSCaseInsensitiveSearch].location != NSNotFound) { + auto capture = std::make_unique(); + if (capture->start(device, stereo)) + return capture; + return nullptr; + } + logInfo("passthrough: no camera of this headset among macOS's cameras"); + return nullptr; +} + +} // namespace metalvr diff --git a/daemon/camera_tracking.cpp b/daemon/camera_tracking.cpp new file mode 100644 index 0000000..2c7a5d5 --- /dev/null +++ b/daemon/camera_tracking.cpp @@ -0,0 +1,302 @@ +#include "camera_tracking.h" + +#include +#include + +#include +#include +#include +#include + +#include "common.h" +#include "vit/vit_interface.h" + +#ifndef METALVR_BASALT_DIR +#define METALVR_BASALT_DIR "" +#endif + +namespace metalvr { +namespace { + +// the VIT functions, from the loaded library +struct Vit { + void *library = nullptr; +#define VIT_FUNCTION(name) PFN_vit_##name name = nullptr; + VIT_FUNCTION(api_get_version) + VIT_FUNCTION(tracker_create) + VIT_FUNCTION(tracker_destroy) + VIT_FUNCTION(tracker_start) + VIT_FUNCTION(tracker_stop) + VIT_FUNCTION(tracker_push_imu_sample) + VIT_FUNCTION(tracker_push_img_sample) + VIT_FUNCTION(tracker_add_imu_calibration) + VIT_FUNCTION(tracker_add_camera_calibration) + VIT_FUNCTION(tracker_pop_pose) + VIT_FUNCTION(pose_destroy) + VIT_FUNCTION(pose_get_data) +#undef VIT_FUNCTION + + bool load(const std::string &path, std::string &why) { + library = dlopen(path.c_str(), RTLD_NOW | RTLD_LOCAL); + if (!library) { + why = dlerror(); + return false; + } + bool all = true; +#define VIT_FUNCTION(name) \ + name = reinterpret_cast(dlsym(library, "vit_" #name)); \ + all = all && name; + VIT_FUNCTION(api_get_version) + VIT_FUNCTION(tracker_create) + VIT_FUNCTION(tracker_destroy) + VIT_FUNCTION(tracker_start) + VIT_FUNCTION(tracker_stop) + VIT_FUNCTION(tracker_push_imu_sample) + VIT_FUNCTION(tracker_push_img_sample) + VIT_FUNCTION(tracker_add_imu_calibration) + VIT_FUNCTION(tracker_add_camera_calibration) + VIT_FUNCTION(tracker_pop_pose) + VIT_FUNCTION(pose_destroy) + VIT_FUNCTION(pose_get_data) +#undef VIT_FUNCTION + if (!all) { + why = path + " is not a VIT tracker"; + return false; + } + uint32_t major = 0, minor = 0, patch = 0; + if (api_get_version(&major, &minor, &patch) != VIT_SUCCESS || major != VIT_HEADER_VERSION_MAJOR) { + why = path + ": VIT " + std::to_string(major) + "." + std::to_string(minor) + ", not " + + std::to_string(VIT_HEADER_VERSION_MAJOR); + return false; + } + return true; + } +}; + +std::vector libraryCandidates() { + std::vector out; + if (const char *path = std::getenv("METALVR_BASALT_LIBRARY"); path && *path) out.emplace_back(path); + char executable[4096]; + uint32_t size = sizeof(executable); + if (_NSGetExecutablePath(executable, &size) == 0) { + const std::filesystem::path dir = std::filesystem::path(executable).parent_path(); + out.push_back((dir / "libbasalt.dylib").string()); + out.push_back((dir / "../lib/libbasalt.dylib").string()); + } + if (*METALVR_BASALT_DIR) out.push_back(std::string(METALVR_BASALT_DIR) + "/libbasalt.dylib"); + return out; +} + +// Basalt's room has gravity down its -z; OpenXR's has +y up: a quarter turn about x (as libsurvive's, in +// lighthouse_tracking.cpp) +constexpr double kHalf = 0.70710678118654752440; + +class Basalt final : public CameraTracking { +public: + bool start(const std::vector &cameras, const TrackerImu &imu, std::string &why) { + for (const std::string &path : libraryCandidates()) { + std::error_code error; + if (!std::filesystem::exists(path, error)) continue; + if (vit_.load(path, why)) { + path_ = path; + break; + } + logInfo("camera tracking: %s", why.c_str()); + } + if (!vit_.library) { + if (why.empty()) why = "libbasalt.dylib not found (scripts/build-basalt.sh)"; + return false; + } + if (cameras.size() < 2) { + why = "Basalt needs two cameras or more"; + return false; + } + imu_position_[0] = imu.position[0]; + imu_position_[1] = imu.position[1]; + imu_position_[2] = imu.position[2]; + vit_config_t config{}; + config.file = nullptr; // the calibration below and Basalt's default pipeline, as Monado does with a driver's + config.cam_count = static_cast(cameras.size()); + config.imu_count = 1; + if (vit_.tracker_create(&config, &tracker_) != VIT_SUCCESS || !tracker_) { + why = "Basalt did not create a tracker"; + return false; + } + // 8-bit grey is what Basalt's push_frame takes; its has_image_format answers no to everything (it overwrites + // its own answer), so it is not asked, as Monado does not ask it + for (size_t i = 0; i < cameras.size(); i++) { + const TrackerCamera &c = cameras[i]; + vit_camera_calibration_t calibration{}; + calibration.camera_index = static_cast(i); + calibration.width = static_cast(c.width); + calibration.height = static_cast(c.height); + calibration.frequency = c.hz; + calibration.fx = c.fx; + calibration.fy = c.fy; + calibration.cx = c.cx; + calibration.cy = c.cy; + calibration.model = VIT_CAMERA_DISTORTION_KB4; + calibration.distortion_count = 4; + for (int k = 0; k < 4; k++) calibration.distortion[k] = c.kb4[k]; + for (int k = 0; k < 16; k++) calibration.transform[k] = c.imu_from_camera[k]; + if (vit_.tracker_add_camera_calibration(tracker_, &calibration) != VIT_SUCCESS) { + why = "Basalt refused camera " + std::to_string(i) + "'s calibration"; + return false; + } + } + // the samples are given already rectified: identity transforms, no offsets (as Monado gives the Rift S's) + vit_imu_calibration_t calibration{}; + calibration.imu_index = 0; + calibration.frequency = imu.hz; + for (vit_inertial_calibration *inertial : {&calibration.accel, &calibration.gyro}) { + inertial->transform[0] = inertial->transform[4] = inertial->transform[8] = 1; + const bool accel = inertial == &calibration.accel; + for (int a = 0; a < 3; a++) { + inertial->noise_std[a] = accel ? imu.accel_noise : imu.gyro_noise; + inertial->bias_std[a] = accel ? imu.accel_bias : imu.gyro_bias; + } + } + if (vit_.tracker_add_imu_calibration(tracker_, &calibration) != VIT_SUCCESS) { + why = "Basalt refused the IMU's calibration"; + return false; + } + if (vit_.tracker_start(tracker_) != VIT_SUCCESS) { + why = "Basalt did not start"; + return false; + } + running_ = true; + logInfo("camera tracking: Basalt (%s) with %zu cameras", path_.c_str(), cameras.size()); + return true; + } + + ~Basalt() override { + if (tracker_) { + if (running_) vit_.tracker_stop(tracker_); + vit_.tracker_destroy(tracker_); + } + // libbasalt stays loaded: its threads' static state is not made to be unloaded + } + + void pushImu(int64_t time_ns, const float accel[3], const float gyro[3]) override { + std::lock_guard guard(push_mutex_); + if (time_ns <= last_imu_ns_) return; // Basalt takes only increasing times + last_imu_ns_ = time_ns; + vit_imu_sample_t sample{}; + sample.timestamp = time_ns; + sample.ax = accel[0], sample.ay = accel[1], sample.az = accel[2]; + sample.wx = gyro[0], sample.wy = gyro[1], sample.wz = gyro[2]; + vit_.tracker_push_imu_sample(tracker_, &sample); + } + + void pushFrames(int64_t time_ns, const std::vector &pictures) override { + { + std::lock_guard guard(push_mutex_); + if (time_ns <= last_frame_ns_) return; + last_frame_ns_ = time_ns; + for (size_t i = 0; i < pictures.size(); i++) { + vit_img_sample_t sample{}; + sample.cam_index = static_cast(i); + sample.timestamp = time_ns; + sample.data = const_cast(pictures[i]); + sample.format = VIT_IMAGE_FORMAT_L8; + sizes(i, sample); + vit_.tracker_push_img_sample(tracker_, &sample); + } + } + drainPoses(); + } + + bool pose(TrackedPose &out, int64_t *time_ns) override { + drainPoses(); + std::lock_guard guard(pose_mutex_); + if (!have_pose_) return false; + out = pose_; + if (time_ns) *time_ns = pose_time_ns_; + return true; + } + + void setSizes(const std::vector &cameras) { + for (const TrackerCamera &c : cameras) { + widths_.push_back(c.width); + heights_.push_back(c.height); + } + } + +private: + void sizes(size_t i, vit_img_sample_t &sample) const { + sample.width = widths_[i]; + sample.height = heights_[i]; + sample.stride = widths_[i]; + sample.size = widths_[i] * heights_[i]; + } + + void drainPoses() { + vit_pose_t *pose = nullptr; + while (vit_.tracker_pop_pose(tracker_, &pose) == VIT_SUCCESS && pose) { + vit_pose_data_t data{}; + if (vit_.pose_get_data(pose, &data) == VIT_SUCCESS) take(data); + vit_.pose_destroy(pose); + pose = nullptr; + } + } + + // Basalt's pose of the IMU (its body: the headset's axes, at the IMU) in its room, to the headset's origin in + // OpenXR's axes + void take(const vit_pose_data_t &d) { + const double qx = d.ox, qy = d.oy, qz = d.oz, qw = d.ow; + // the headset's origin is -imu_position in the body's frame: rotate it into the room + const double v[3] = {-imu_position_[0], -imu_position_[1], -imu_position_[2]}; + double r[3]; + rotate(qx, qy, qz, qw, v, r); + const double p[3] = {d.px + r[0], d.py + r[1], d.pz + r[2]}; + // c = a quarter turn about x, -90 degrees: (x, y, z) -> (x, z, -y); q' = c q + const double cx = -kHalf, cw = kHalf; + TrackedPose out; + out.orientation = {static_cast(cw * qx + cx * qw), static_cast(cw * qy - cx * qz), + static_cast(cw * qz + cx * qy), static_cast(cw * qw - cx * qx)}; + out.position = {static_cast(p[0]), static_cast(p[2]), static_cast(-p[1])}; + out.velocity = {d.vx, d.vz, -d.vy}; + out.position_tracked = std::isfinite(out.position.x) && std::isfinite(out.position.y) && + std::isfinite(out.position.z); + std::lock_guard guard(pose_mutex_); + pose_ = out; + pose_time_ns_ = d.timestamp; + have_pose_ = true; + } + + static void rotate(double x, double y, double z, double w, const double v[3], double out[3]) { + // v + 2 w (q x v) + 2 q x (q x v) + const double tx = 2 * (y * v[2] - z * v[1]), ty = 2 * (z * v[0] - x * v[2]), tz = 2 * (x * v[1] - y * v[0]); + out[0] = v[0] + w * tx + (y * tz - z * ty); + out[1] = v[1] + w * ty + (z * tx - x * tz); + out[2] = v[2] + w * tz + (x * ty - y * tx); + } + + Vit vit_; + std::string path_; + vit_tracker_t *tracker_ = nullptr; + bool running_ = false; + double imu_position_[3] = {}; + std::vector widths_, heights_; + std::mutex push_mutex_, pose_mutex_; + int64_t last_imu_ns_ = 0, last_frame_ns_ = 0; + TrackedPose pose_; + int64_t pose_time_ns_ = 0; + bool have_pose_ = false; +}; + +} // namespace + +std::unique_ptr startCameraTracking(const std::vector &cameras, const TrackerImu &imu, + std::string *why) { + auto basalt = std::make_unique(); + basalt->setSizes(cameras); + std::string reason; + if (!basalt->start(cameras, imu, reason)) { + if (why) *why = reason; + return nullptr; + } + return basalt; +} + +} // namespace metalvr diff --git a/daemon/camera_tracking.h b/daemon/camera_tracking.h new file mode 100644 index 0000000..bdc501c --- /dev/null +++ b/daemon/camera_tracking.h @@ -0,0 +1,55 @@ +// Position tracking from a headset's own cameras and IMU (inside-out; the Rift S, docs/HEADSETS.md phase I): Basalt, +// Monado's fork (BSD-3-Clause), built by scripts/build-basalt.sh as libbasalt.dylib and loaded when tracking starts +// through Monado's VIT interface (daemon/vit/vit_interface.h, BSL-1.0), as Monado loads it. metalvrd neither links +// it nor needs it: without it a headset tracks rotation only. +// +// The tracker is given the IMU's samples in the headset's frame (OpenXR's axes, +y up, +z back), the cameras' +// pictures with their lenses as Kannala-Brandt (KB4) and where each sits from the IMU, all on the headset's clock. +// It hands back where the headset is in a room it builds as it goes, turned into OpenXR's axes. +#pragma once + +#include +#include +#include +#include +#include + +#include "lighthouse_tracking.h" + +namespace metalvr { + +struct TrackerCamera { + uint32_t width = 0, height = 0; + double fx = 0, fy = 0, cx = 0, cy = 0; + std::array kb4{}; + double imu_from_camera[16] = {}; // row-major isometry + double hz = 30; +}; + +struct TrackerImu { + double hz = 1000; + // the noise and bias walk Monado gives the Rift S's IMU (rift_s_fill_slam_imu_calibration) + double accel_noise = 0.016, accel_bias = 0.001; + double gyro_noise = 0.000282, gyro_bias = 0.0001; + float position[3] = {}; // where the IMU sits in the headset, meters: the pose handed back is the headset's origin +}; + +class CameraTracking { +public: + virtual ~CameraTracking() = default; + // One IMU sample: specific force (m/s^2) and angular velocity (rad/s), in the headset's frame. Times are the + // headset's, in nanoseconds, increasing. + virtual void pushImu(int64_t time_ns, const float accel[3], const float gyro[3]) = 0; + // One picture from every camera, taken together; each `width * height` bytes of grey, rows packed + virtual void pushFrames(int64_t time_ns, const std::vector &pictures) = 0; + // The newest pose of the headset's origin: rotation and position in the tracker's room (OpenXR's axes, +y up); + // false until it has one. `time_ns` is the headset time it is for. + virtual bool pose(TrackedPose &out, int64_t *time_ns = nullptr) = 0; +}; + +// Basalt, loaded and started with these sensors; null (with `why` set) if libbasalt.dylib is not found or refuses +// them. It is looked for in $METALVR_BASALT_LIBRARY, next to metalvrd, and in the build's build/basalt/prefix/lib. +std::unique_ptr startCameraTracking(const std::vector &cameras, const TrackerImu &imu, + std::string *why); + +} // namespace metalvr diff --git a/daemon/compositor.h b/daemon/compositor.h index b12df7c..26dcafd 100644 --- a/daemon/compositor.h +++ b/daemon/compositor.h @@ -7,6 +7,7 @@ #include #include +#include "foveation.h" #include "metalvr_protocol.h" namespace metalvr { @@ -43,8 +44,9 @@ class Compositor { explicit Compositor(id device); ~Compositor(); - // the size of the whole frame; a change drops the buffer pool - bool configure(uint32_t frame_width, uint32_t frame_height); + // the size of the whole frame; a change drops the buffer pool. With foveation enabled, the eyes are composed at + // its eye size and squeezed into a frame of this size. + bool configure(uint32_t frame_width, uint32_t frame_height, const Foveation &foveation = {}); uint32_t frameWidth() const { return width_; } uint32_t frameHeight() const { return height_; } @@ -59,6 +61,9 @@ class Compositor { id library_; id sampler_; id pipelines_[2]; // opaque, blended + id squeeze_ = nil; + Foveation foveation_; + id full_ = nil; // the eyes at their composed size, when foveated CVPixelBufferPoolRef pool_ = nullptr; uint32_t width_ = 0; uint32_t height_ = 0; diff --git a/daemon/compositor.mm b/daemon/compositor.mm index 818110b..b90c48f 100644 --- a/daemon/compositor.mm +++ b/daemon/compositor.mm @@ -52,6 +52,45 @@ static float3 srgb_to_linear(float3 c) { return select(hi, lo, c <= float3(0.04045)); } +// ALVR's foveated encoding (foveation.h, CompressAxisAlignedPixelShader.hlsl): each pixel of the squeezed frame +// reads the composed frame where ALVR's client will put it back +struct Squeeze { + float2 frame_size; + float2 eye_size_ratio; + float2 center_size; + float2 center_shift; + float2 edge_ratio; +}; + +vertex Varyings squeeze_vertex(uint vid [[vertex_id]]) { + float2 p = float2((vid << 1) & 2, vid & 2); + Varyings out; + out.position = float4(p * 2.0 - 1.0, 0.0, 1.0); + out.uv = float2(p.x, 1.0 - p.y); + return out; +} + +fragment float4 squeeze_fragment(Varyings in [[stage_in]], texture2d full [[texture(0)]], + sampler smp [[sampler(0)]], constant Squeeze &f [[buffer(0)]]) { + float2 uv = in.position.xy / f.frame_size; + bool right = uv.x > 0.5; + float2 eye = float2((uv.x + (right ? 1.0 - 2.0 * uv.x : 0.0)) * 2.0, uv.y) / f.eye_size_ratio; + float2 c0 = (1.0 - f.center_size) / 2.0; + float2 c1 = (f.edge_ratio - 1.0) * c0 * (f.center_shift + 1.0) / f.edge_ratio; + float2 c2 = (f.edge_ratio - 1.0) * f.center_size + 1.0; + float2 lo = c0 * (f.center_shift + 1.0) / c2; + float2 hi = c0 * (f.center_shift - 1.0) / c2 + 1.0; + float2 center = eye * c2 / f.edge_ratio + c1; + float2 d2 = eye * c2; + float2 d3 = (eye - 1.0) * c2 + 1.0; + float2 g1 = eye / lo; + float2 g2 = (1.0 - eye) / (1.0 - hi); + float2 left_edge = g1 * center + (1.0 - g1) * d2; + float2 right_edge = g2 * center + (1.0 - g2) * d3; + float2 source = select(select(center, right_edge, eye > hi), left_edge, eye < lo); + return full.sample(smp, float2(source.x * 0.5 + (right ? 1.0 - source.x : 0.0), source.y)); +} + fragment float4 layer_fragment(Varyings in [[stage_in]], texture2d source [[texture(0)]], sampler smp [[sampler(0)]], constant Uniforms &u [[buffer(0)]]) { float4 c = source.sample(smp, in.uv); @@ -155,9 +194,43 @@ void project(const mvr_fov &fov, mvr_vec3 p, float out[4]) { return pipelines_[index]; } -bool Compositor::configure(uint32_t frame_width, uint32_t frame_height) { - if (frame_width == width_ && frame_height == height_ && pool_) +bool Compositor::configure(uint32_t frame_width, uint32_t frame_height, const Foveation &foveation) { + const bool same_foveation = foveation.enabled == foveation_.enabled && + (!foveation.enabled || (foveation.eye_width == foveation_.eye_width && + foveation.eye_height == foveation_.eye_height && + foveation.center_size[0] == foveation_.center_size[0] && + foveation.center_size[1] == foveation_.center_size[1] && + foveation.center_shift[0] == foveation_.center_shift[0] && + foveation.center_shift[1] == foveation_.center_shift[1] && + foveation.edge_ratio[0] == foveation_.edge_ratio[0] && + foveation.edge_ratio[1] == foveation_.edge_ratio[1])); + if (frame_width == width_ && frame_height == height_ && pool_ && same_foveation) return true; + foveation_ = foveation; + full_ = nil; + if (foveation_.enabled) { + MTLTextureDescriptor *desc = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm_sRGB + width:foveation_.eye_width * 2 + height:foveation_.eye_height + mipmapped:NO]; + desc.usage = MTLTextureUsageRenderTarget | MTLTextureUsageShaderRead; + desc.storageMode = MTLStorageModePrivate; + full_ = [device_ newTextureWithDescriptor:desc]; + if (!squeeze_ && library_) { + MTLRenderPipelineDescriptor *pipeline = [MTLRenderPipelineDescriptor new]; + pipeline.vertexFunction = [library_ newFunctionWithName:@"squeeze_vertex"]; + pipeline.fragmentFunction = [library_ newFunctionWithName:@"squeeze_fragment"]; + pipeline.colorAttachments[0].pixelFormat = MTLPixelFormatBGRA8Unorm_sRGB; + NSError *error = nil; + squeeze_ = [device_ newRenderPipelineStateWithDescriptor:pipeline error:&error]; + if (!squeeze_) + logInfo("compositor: foveation pipeline failed: %s", error.localizedDescription.UTF8String); + } + if (!full_ || !squeeze_) + return false; + logInfo("compositor: foveated: %ux%u per eye squeezed to %ux%u", foveation_.eye_width, foveation_.eye_height, + frame_width / 2, frame_height); + } if (pool_) { CVPixelBufferPoolRelease(pool_); pool_ = nullptr; @@ -207,8 +280,14 @@ void project(const mvr_fov &fov, mvr_vec3 p, float out[4]) { return nullptr; } + // foveated: the layers go to the eyes at their full size first, and are squeezed into the frame after + const bool squeezed = foveation_.enabled && full_ && squeeze_; + id layers_target = squeezed ? full_ : target; + const uint32_t compose_width = static_cast(layers_target.width); + const uint32_t compose_height = static_cast(layers_target.height); + MTLRenderPassDescriptor *pass = [MTLRenderPassDescriptor renderPassDescriptor]; - pass.colorAttachments[0].texture = target; + pass.colorAttachments[0].texture = layers_target; pass.colorAttachments[0].loadAction = MTLLoadActionClear; pass.colorAttachments[0].storeAction = MTLStoreActionStore; pass.colorAttachments[0].clearColor = MTLClearColorMake(0, 0, 0, 1); @@ -217,7 +296,7 @@ void project(const mvr_fov &fov, mvr_vec3 p, float out[4]) { id encoder = [commands renderCommandEncoderWithDescriptor:pass]; [encoder setFragmentSamplerState:sampler_ atIndex:0]; - const uint32_t eye_width = width_ / 2; + const uint32_t eye_width = compose_width / 2; bool first = true; for (const ComposeLayer &layer : frame.layers) { const bool alpha = (layer.flags & MVR_LAYER_BLEND_ALPHA) != 0; @@ -274,9 +353,9 @@ void project(const mvr_fov &fov, mvr_vec3 p, float out[4]) { uniforms.opaque = !blend; MTLViewport viewport = {static_cast(eye * eye_width), 0, static_cast(eye_width), - static_cast(height_), 0, 1}; + static_cast(compose_height), 0, 1}; [encoder setViewport:viewport]; - MTLScissorRect scissor = {eye * eye_width, 0, eye_width, height_}; + MTLScissorRect scissor = {eye * eye_width, 0, eye_width, compose_height}; [encoder setScissorRect:scissor]; [encoder setVertexBytes:vertices length:sizeof(vertices) atIndex:0]; [encoder setFragmentBytes:&uniforms length:sizeof(uniforms) atIndex:0]; @@ -287,6 +366,30 @@ void project(const mvr_fov &fov, mvr_vec3 p, float out[4]) { } [encoder endEncoding]; + if (squeezed) { + struct { + float frame_size[2]; + float eye_size_ratio[2]; + float center_size[2]; + float center_shift[2]; + float edge_ratio[2]; + } squeeze = {{static_cast(width_), static_cast(height_)}, + {foveation_.eye_size_ratio[0], foveation_.eye_size_ratio[1]}, + {foveation_.center_size[0], foveation_.center_size[1]}, + {foveation_.center_shift[0], foveation_.center_shift[1]}, + {foveation_.edge_ratio[0], foveation_.edge_ratio[1]}}; + MTLRenderPassDescriptor *out = [MTLRenderPassDescriptor renderPassDescriptor]; + out.colorAttachments[0].texture = target; + out.colorAttachments[0].loadAction = MTLLoadActionDontCare; + out.colorAttachments[0].storeAction = MTLStoreActionStore; + id squeeze_encoder = [commands renderCommandEncoderWithDescriptor:out]; + [squeeze_encoder setRenderPipelineState:squeeze_]; + [squeeze_encoder setFragmentTexture:full_ atIndex:0]; + [squeeze_encoder setFragmentSamplerState:sampler_ atIndex:0]; + [squeeze_encoder setFragmentBytes:&squeeze length:sizeof(squeeze) atIndex:0]; + [squeeze_encoder drawPrimitives:MTLPrimitiveTypeTriangle vertexStart:0 vertexCount:3]; + [squeeze_encoder endEncoding]; + } [commands commit]; [commands waitUntilCompleted]; if (commands.status != MTLCommandBufferStatusCompleted) { diff --git a/daemon/direct_backend.h b/daemon/direct_backend.h new file mode 100644 index 0000000..8890924 --- /dev/null +++ b/daemon/direct_backend.h @@ -0,0 +1,130 @@ +// A headset on the Mac's own video output and USB, other than the Rift CV1 (whose code is in driver_rift.mm): the +// device side of the direct-display driver (docs/HEADSETS.md). The driver owns the window on the headset's display, +// the lens correction, the home, frame pacing and the head's rotation; a backend finds the headset on USB, turns it +// on, and hands the driver what the headset says: which model it is, its IMU samples, whether it is worn, and its +// controllers. +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "lighthouse_tracking.h" +#include "metalvr_protocol.h" + +namespace metalvr { + +// how a headset's panel is laid out and its lenses corrected +struct DirectLens { + enum Model : uint32_t { CatmullRomCV1 = 0, Panotools = 1, Vive = 2, Psvr2 = 3 }; + uint32_t model = CatmullRomCV1; + // panotools (Rift S) + float pano_k[5] = {}; + float pano_aberration[3] = {1, 1, 1}; + float pano_scale = 0; + float pano_center[2] = {}; + float pano_viewport[2] = {}; + // Vive (Monado's u_vive_values) + float vive_aspect = 0.9f; + float vive_grow = 0.5f; + float vive_center[3][2] = {}; + float vive_coefficients[3][4] = {}; + bool flip_y = false; // the Vive Pro 2's image is upside down + // PlayStation VR2 (psvr2::lensCalibration): this eye's x and y offset, then cos and sin of its tilt + float psvr2[4] = {0, 0, 1, 0}; +}; + +struct DirectModel { + uint32_t headset = MVR_HEADSET_UNKNOWN; + uint32_t controllers = MVR_CONTROLLERS_UNKNOWN; + std::string name; + uint32_t display_vendor = 0; + std::vector display_products; // any of these is the headset's display + float refresh_hz = 90; + std::vector refresh_rates; + uint32_t eye_width = 0, eye_height = 0; // the render size applications are asked for + mvr_fov fov[2] = {}; + mvr_quat eye_rotation[2] = {{0, 0, 0, 1}, {0, 0, 0, 1}}; // canted lenses (Index): each eye's view in the head + float ipd = 0.0635f; + bool portrait_eyes = false; // the eyes are the panel's top and bottom halves, each turned a quarter right (Rift S) + DirectLens lens[2]; +}; + +// a controller's report, in the controller's own frame (+x right, +y up, +z back) +struct DirectControllerInput { + uint32_t buttons = 0; // mvr_button + uint32_t touches = 0; + float trigger = 0, squeeze = 0; + float thumbstick_x = 0, thumbstick_y = 0; + float trackpad_x = 0, trackpad_y = 0, trackpad_force = 0, squeeze_force = 0; + bool has_fingers = false; + float fingers[4] = {}; + bool has_battery = false; + float battery = 0; + bool charging = false; +}; + +class DirectHost { +public: + virtual ~DirectHost() = default; + // the headset is identified: which it is and how to draw for it + virtual void directModel(const DirectModel &model) = 0; + // the headset is on (its display and IMU running) or off + virtual void directOn(bool on) = 0; + // one IMU sample of the head, in the headset's frame as OpenXR has it; time on the server clock + virtual void directImu(int64_t time_ns, const float accel[3], const float gyro[3]) = 0; + virtual void directWorn(bool worn) = 0; + virtual void directIpd(float meters) = 0; + // a controller's IMU sample and its inputs; hand 0 left, 1 right. `grip` is the grip pose in the controller's + // own frame, for its kind (Monado's offsets) + virtual void directControllerImu(int hand, int64_t time_ns, const float accel[3], const float gyro[3]) = 0; + virtual void directControllerInput(int hand, const DirectControllerInput &input, int64_t time_ns) = 0; + virtual void directControllerGrip(int hand, const mvr_pose &grip, const mvr_pose &aim) = 0; +}; + +class DirectBackend { +public: + virtual ~DirectBackend() = default; + virtual const char *name() const = 0; + // Runs on the driver's USB thread until `running` turns false: finds the headset, turns it on, reports. Returns + // having turned it off again. + virtual void run(DirectHost &host, const std::atomic &running) = 0; + virtual void haptic(uint32_t hand, float duration_s, float frequency_hz, float amplitude) {} + // Where a tracker outside the headset (the base stations) has the head (LighthouseTracking::kHead) or a hand, in + // the tracker's space; false while it has nothing. The driver keeps its own rotation from the IMU and takes the + // position from here, as it does with a Rift's sensors. Any thread. + virtual bool trackedPose(int device, TrackedPose &out) { return false; } + // that tracker's base stations, in its space + virtual uint32_t trackingReferences(mvr_tracking_reference *out) { return 0; } +}; + +struct LighthouseOptions { + int pro2_mode = -1; // the Vive Pro 2's mode to switch to (0..5); -1 leaves it as it is +}; + +std::unique_ptr makeRiftSBackend(); +std::unique_ptr makeLighthouseBackend(const LighthouseOptions &options); +std::unique_ptr makePsvr2Backend(); + +namespace lighthouse { +struct HeadsetConfig; +} + +// the model the Rift S backend reports, and a Lighthouse headset's from its configuration (direct_models.cpp), so +// that what is drawn can be checked without a headset. `pro2_mode` is the Vive Pro 2's mode (pro2Modes()). +DirectModel riftSModel(); +DirectModel lighthouseModel(const lighthouse::HeadsetConfig &config, int pro2_mode); +DirectModel riftCv1Model(); +// the PlayStation VR2 with its lens calibration (psvr2::lensCalibration) +DirectModel psvr2Model(const std::array &calibration); + +// the lens-check grid for `model` with the head straight ahead, drawn as the headset is drawn, into a panel-sized +// BGRA picture (driver_rift.mm): the lens models checked without a headset (metalvrd --panel-check) +// With `passthrough`, the home instead, showing a stereo camera picture red on the left and blue on the right. +bool drawTestPanel(const DirectModel &model, uint32_t width, uint32_t height, std::vector &bgra, + bool passthrough = false); + +} // namespace metalvr diff --git a/daemon/direct_models.cpp b/daemon/direct_models.cpp new file mode 100644 index 0000000..3644282 --- /dev/null +++ b/daemon/direct_models.cpp @@ -0,0 +1,131 @@ +// What the direct-display driver is told of the Rift S, the Lighthouse headsets and the PlayStation VR2 +// (direct_backend.h). +#include + +#include "direct_backend.h" +#include "headsets.h" +#include "lighthouse_protocol.h" +#include "psvr2_protocol.h" +#include "rift_s_protocol.h" + +namespace metalvr { + +namespace { + +void fromSpec(DirectModel &m, uint32_t headset) { + const HeadsetSpec &spec = headsetSpec(headset); + const mvr_headset_description *description = mvr_describe_headset(headset); + m.headset = headset; + m.controllers = description->controllers; + m.name = description->name; + m.refresh_hz = spec.default_refresh; + m.refresh_rates.assign(spec.refresh_rates, spec.refresh_rates + spec.refresh_rate_count); + m.eye_width = spec.eye_width; + m.eye_height = spec.eye_height; + m.fov[0] = spec.fov[0]; + m.fov[1] = spec.fov[1]; + m.ipd = spec.ipd; +} + +} // namespace + +DirectModel riftSModel() { + DirectModel m; + fromSpec(m, MVR_HEADSET_RIFT_S); + m.display_vendor = rift_s::kDisplayVendor; + m.display_products = {rift_s::kDisplayProduct}; + m.portrait_eyes = true; + for (int eye = 0; eye < 2; eye++) { + rift_s::fieldOfView(eye, m.fov[eye].left, m.fov[eye].right, m.fov[eye].up, m.fov[eye].down); + const rift_s::Panotools p = rift_s::panotools(eye); + DirectLens &lens = m.lens[eye]; + lens.model = DirectLens::Panotools; + for (int i = 0; i < 5; i++) lens.pano_k[i] = p.distortion_k[i]; + for (int i = 0; i < 3; i++) lens.pano_aberration[i] = p.aberration_k[i]; + lens.pano_scale = p.scale; + lens.pano_center[0] = p.lens_center[0]; + lens.pano_center[1] = p.lens_center[1]; + lens.pano_viewport[0] = p.viewport[0]; + lens.pano_viewport[1] = p.viewport[1]; + } + return m; +} + +DirectModel psvr2Model(const std::array &calibration) { + DirectModel m; + fromSpec(m, MVR_HEADSET_PSVR2); + // Sony's vendor id; the PS VR2's own product code in its EDID is not known yet (docs/HEADSETS.md), so any Sony + // display is taken, and named in the log + m.display_vendor = psvr2::kDisplayVendor; + m.display_products = {}; + for (int eye = 0; eye < 2; eye++) { + m.fov[eye] = psvr2::fieldOfView(eye); + DirectLens &lens = m.lens[eye]; + lens.model = DirectLens::Psvr2; + lens.psvr2[0] = calibration[eye ? 2 : 0]; + lens.psvr2[1] = calibration[eye ? 3 : 1]; + lens.psvr2[2] = calibration[eye ? 6 : 4]; + lens.psvr2[3] = calibration[eye ? 7 : 5]; + } + return m; +} + +DirectModel lighthouseModel(const lighthouse::HeadsetConfig &config, int pro2_mode) { + using namespace lighthouse; + DirectModel m; + fromSpec(m, headsetOf(config.model)); + switch (config.model) { + case Model::Vive: + m.display_vendor = 0x22d2; // "HVR" + m.display_products = {0xaa01}; + break; + case Model::VivePro: + m.display_vendor = 0x22d2; + m.display_products = {0xaa02, 0xaa03}; // which is the Vive Pro's is a guess (docs/HEADSETS.md) + break; + case Model::VivePro2: + m.display_vendor = 0x22d2; + m.display_products = {0xaa04}; + break; + case Model::Index: + m.display_vendor = 0x5996; // "VLV" + m.display_products = {0x91a8}; + for (uint32_t p = 0x91b0; p <= 0x91bf; p++) m.display_products.push_back(p); + break; + case Model::None: + break; + } + // the panel's pixels for one eye, about 1.2 times over, as the CV1's render size is + const auto render = [](int pixels) { return static_cast(std::lround(pixels * 1.2 / 16.0) * 16); }; + int eye_w = config.eye_width, eye_h = config.eye_height; + if (config.model == Model::VivePro2) { + const auto &modes = pro2Modes(); + const Pro2Mode &mode = modes[pro2_mode >= 0 && pro2_mode < static_cast(modes.size()) ? pro2_mode : 2]; + eye_w = mode.width / 2; + eye_h = mode.height; + m.refresh_hz = mode.refresh_hz; + } + if (eye_w > 0 && eye_h > 0) { + m.eye_width = render(eye_w); + m.eye_height = render(eye_h); + } + for (int eye = 0; eye < 2; eye++) { + m.fov[eye] = config.fov[eye]; + m.eye_rotation[eye] = config.eye_rotation[eye]; + const Distortion &d = config.distortion[eye]; + DirectLens &lens = m.lens[eye]; + lens.model = DirectLens::Vive; + lens.vive_aspect = d.aspect_x_over_y; + lens.vive_grow = d.grow_for_undistort; + for (int c = 0; c < 3; c++) { + lens.vive_center[c][0] = d.center[c][0]; + lens.vive_center[c][1] = d.center[c][1]; + for (int k = 0; k < 4; k++) lens.vive_coefficients[c][k] = d.coefficients[c][k]; + } + lens.flip_y = config.model == Model::VivePro2; + } + if (config.lens_separation > 0.05f && config.lens_separation < 0.08f) m.ipd = config.lens_separation; + return m; +} + +} // namespace metalvr diff --git a/daemon/driver.h b/daemon/driver.h index 0a18232..0e90cc4 100644 --- a/daemon/driver.h +++ b/daemon/driver.h @@ -12,6 +12,7 @@ #include +#include "foveation.h" #include "metalvr_protocol.h" namespace metalvr { @@ -25,6 +26,9 @@ struct VideoParams { uint32_t frame_height = 0; float fps = 72.0f; uint64_t bitrate_bps = 30'000'000; + // when enabled, each eye is composed at foveation's eye size and squeezed to frame_width/2 x frame_height, which + // are its optimized size (ALVR's foveated encoding, foveation.h) + Foveation foveation; }; // an encoded frame's identity and history; times are on the server clock @@ -68,6 +72,12 @@ class Driver { // the tracking system's sensors, at most MVR_MAX_TRACKING_REFERENCES; how many were written virtual uint32_t trackingReferences(mvr_tracking_reference *out) { return 0; } + // the hands' joints for `target_time_ns`, when the headset tracks bare hands (systemInfo's + // MVR_SYSTEM_HAND_TRACKING); a hand it does not see now has no MVR_HAND_TRACKED + virtual void hands(int64_t target_time_ns, mvr_hand out[2]) { + out[0].flags = out[1].flags = 0; + } + virtual VideoParams videoParams() = 0; // true once when the headset needs a keyframe (a new connection, packet loss) @@ -100,6 +110,11 @@ struct SimOptions { uint64_t bitrate_bps = 30'000'000; std::string dump_path; // empty: encoded video is counted and dropped bool animate = true; + // the headset it stands in for (--headset): its name, refresh rates and controllers, which the simulated + // controllers then work like (a Vive wand's trackpad, an Index controller's fingers). Its flags still say + // MVR_SYSTEM_SIMULATED. + uint32_t headset = MVR_HEADSET_SIMULATED; + bool foveated_encoding = false; // squeeze each eye as ALVR's foveated encoding with its default shape does }; std::unique_ptr makeSimDriver(const SimOptions &options); @@ -110,17 +125,34 @@ struct AlvrOptions { // ALVR streams only to clients marked trusted, and nothing is trusted that is not named here or already // trusted in the config directory: a headset that announces itself is logged with its name and left out. std::vector trusted_clients; + // The headset, when the person running metalvrd names it (--headset). Otherwise it is the model the headset's + // ALVR client reports (docs/HEADSETS.md), and a client that reports none it knows is an unknown headset. + uint32_t headset = MVR_HEADSET_UNKNOWN; + // The adb a headset on USB (the trusted client "client.wired") is reached through. Empty: the adb on PATH. Wired + // mode does not start without one, so that ALVR never downloads one of its own. + std::string adb; + // ALVR's foveated encoding: each eye squeezed toward its edges before encoding and stretched back by the client, + // fewer pixels for the decoder (--foveated-encoding; ALVR's own foveated_encoding settings set the shape) + bool foveated_encoding = false; }; std::unique_ptr makeAlvrDriver(const AlvrOptions &options); -// An Oculus Rift CV1 on the Mac's own video output and USB (docs/RIFT.md): its HID interface turns the headset on -// and gives its rotation, and frames are drawn on its display with its lens distortion. Head rotation only. +// A headset on the Mac's own video output and USB, drawn on directly with its lens correction (docs/RIFT.md, +// docs/HEADSETS.md). The Rift family (--driver rift): an Oculus Rift CV1, whose HID interface turns it on and gives +// its rotation (its sensors, if there are any, its position), or a Rift S. The Lighthouse family (--driver +// lighthouse): an HTC Vive, Vive Pro, Vive Pro 2 or Valve Index, by its rotation; their base stations are not +// used yet. Which headset it is comes from USB; one the driver cannot identify is not driven. struct RiftOptions { - uint32_t eye_width = 1344; // the per-eye render size applications are asked for - uint32_t eye_height = 1600; + bool lighthouse = false; // the Lighthouse family instead of the Rift family + bool psvr2 = false; // the PlayStation VR2 instead of the Rift family + int pro2_mode = -1; // a Vive Pro 2's mode to switch it to (0..5, docs/HEADSETS.md); -1 leaves it as it is + float refresh_hz = 0; // the display mode to set the headset's display to (Index 80..144); 0 leaves it as it is + uint32_t eye_width = 0; // the per-eye render size applications are asked for; 0: the headset's (CV1 1344x1600) + uint32_t eye_height = 0; float ipd = 0.0f; // meters; 0: the headset's own lens separation float head_height = 1.6f; // meters above the floor, for a seated head that has no position tracking bool test_scene = false; // between applications, the lens-check grid instead of the home + bool passthrough = false; // lighthouse: the home shows the room through the headset's cameras std::string home_logo; // the PNG shown in the home; empty: /../share/metalvr/home-logo.png }; std::unique_ptr makeRiftDriver(const RiftOptions &options); diff --git a/daemon/driver_alvr.cpp b/daemon/driver_alvr.cpp index 01ce4b1..a292025 100644 --- a/daemon/driver_alvr.cpp +++ b/daemon/driver_alvr.cpp @@ -8,6 +8,7 @@ // initializes or shuts ALVR down after start(). tracking(), sendFrame() and the other Driver calls may come from // any daemon thread. #include "driver.h" +#include "headsets.h" #include "alvr_server_core.h" @@ -33,7 +34,9 @@ #include #include #include +#include #include +#include #include #include @@ -353,6 +356,10 @@ struct Pin { string_view expect; // how it reads there }; +// With --foveated-encoding the client is told to expect ALVR's foveated encoding, and metalvrd squeezes each eye +// to match (foveation.h): that pin turns on, and its shape is ALVR's own settings'. +bool g_foveated_encoding = false; + const std::vector &pins() { static const std::vector table = { {{"video", "foveated_encoding", "enabled"}, "false", {"video", "foveated_encoding"}, "\"Disabled\""}, @@ -398,7 +405,13 @@ const std::vector &pins() { {"connection", "client_discovery", "Enabled", "auto_trust_clients"}, "false"}, }; - return table; + // the same, foveated encoding on (its settings are an object then, which is not checked) + static const std::vector foveated = [] { + std::vector t = table; + t[0] = {{"video", "foveated_encoding", "enabled"}, "true", {"video", "foveated_encoding"}, ""}; + return t; + }(); + return g_foveated_encoding ? foveated : table; } std::string dotted(std::span path) { @@ -469,10 +482,24 @@ constexpr string_view kTrustedClient = R"({"display_name":"Unknown","current_ip":null,"manual_ips":[],"trusted":true,"connection_state":"Disconnected"})"; struct ClientList { + struct Client { + std::string hostname; + std::string display_name; // what the client calls its headset ("Quest 3"), "Unknown" until it has said + std::string state; // alvr_common's ConnectionState: "Disconnected", "Connecting", "Streaming", ... + }; std::vector trusted; std::vector untrusted; + std::vector trusted_clients; }; +// a JSON string's text without its quotes; empty if it is not a string or holds escapes +std::string plainString(string_view raw) { + if (raw.size() < 2 || raw.front() != '"' || raw.back() != '"') return {}; + raw = raw.substr(1, raw.size() - 2); + if (raw.find('\\') != string_view::npos) return {}; + return std::string(raw); +} + std::optional readClientList(const std::filesystem::path &session_path) { const auto text = readFile(session_path); if (!text) return std::nullopt; @@ -481,11 +508,40 @@ std::optional readClientList(const std::filesystem::path &session_pa const bool ok = json::forEachMember(clients, [&](string_view name, string_view client) { const bool trusted = json::boolean(json::find(client, {"trusted"})).value_or(false); (trusted ? list.trusted : list.untrusted).emplace_back(name); + if (trusted) + list.trusted_clients.push_back({std::string(name), plainString(json::find(client, {"display_name"})), + plainString(json::find(client, {"connection_state"}))}); }); if (!ok) return std::nullopt; return list; } +// ALVR's name for a headset on USB, reached through adb (alvr_sockets' WIRED_CLIENT_HOSTNAME) +constexpr string_view kWiredClient = "client.wired"; + +bool isExecutable(const std::filesystem::path &path) { + struct stat st {}; + return ::stat(path.c_str(), &st) == 0 && S_ISREG(st.st_mode) && ::access(path.c_str(), X_OK) == 0; +} + +// the adb on PATH, as ALVR would find it; empty if there is none +std::filesystem::path adbOnPath() { + const char *path = std::getenv("PATH"); + if (!path) return {}; + string_view rest = path; + while (!rest.empty()) { + const size_t colon = rest.find(':'); + const string_view dir = rest.substr(0, colon); + if (!dir.empty()) { + const std::filesystem::path candidate = std::filesystem::path(dir) / "adb"; + if (isExecutable(candidate)) return candidate; + } + if (colon == string_view::npos) break; + rest.remove_prefix(colon + 1); + } + return {}; +} + // Writes the object holding entries[begin, end), which share their first `depth` keys and are grouped by key. struct SeedEntry { std::vector path; @@ -514,9 +570,15 @@ void appendObject(std::string &out, const std::vector &entries, size_ // A session that does not exist yet: the pins, the trusted headsets, and HEVC instead of ALVR's default H.264 // (ALVR's default frame for a Quest is 4288 pixels wide, more macroblocks than H.264 allows below level 6). ALVR // fills in everything else when it loads the file. -std::string sessionSeed(const std::vector &trust_keys) { +std::string sessionSeed(const std::vector &trust_keys, uint32_t headset) { std::vector entries; entries.push_back({{"session_settings", "video", "preferred_codec", "variant"}, "\"Hevc\""}); + // a headset named on the command line asks for its usual PC VR rate; ALVR's settings decide it from then on + char fps[16] = {}; + if (headsetSpec(headset).stream_refresh > 0) { + std::snprintf(fps, sizeof(fps), "%.1f", headsetSpec(headset).stream_refresh); + entries.push_back({{"session_settings", "video", "preferred_fps"}, fps}); + } for (const Pin &pin : pins()) { SeedEntry entry{{"session_settings"}, pin.value}; entry.path.insert(entry.path.end(), pin.session.begin(), pin.session.end()); @@ -530,10 +592,11 @@ std::string sessionSeed(const std::vector &trust_keys) { } // Runs before ALVR loads the session, which it does once per process. -bool prepareSession(const std::filesystem::path &session_path, const std::vector &trust_keys) { +bool prepareSession(const std::filesystem::path &session_path, const std::vector &trust_keys, + uint32_t headset) { auto text = readFile(session_path); if (!text || text->find_first_not_of(" \t\r\n") == std::string::npos) { - if (!writeFileAtomic(session_path, sessionSeed(trust_keys))) { + if (!writeFileAtomic(session_path, sessionSeed(trust_keys, headset))) { logLine("cannot write %s", session_path.c_str()); return false; } @@ -618,7 +681,7 @@ std::optional readSettings() { for (size_t i = 0; i < pin.setting.size() && !switched_off; i++) switched_off = pin.setting[i] == "Enabled" && json::find(text, std::span(pin.setting).first(i)) == "\"Disabled\""; - if (switched_off) continue; + if (switched_off || pin.expect.empty()) continue; if (!json::same(json::find(text, pin.setting), pin.expect)) settings.problems += ' ' + dotted(pin.session); } return settings; @@ -638,6 +701,7 @@ struct StreamConfig { bool ten_bit = false; bool hdr = false; bool foveated = false; + float foveation[6] = {}; // center size x, y, center shift x, y, edge ratio x, y float gamma = 1.0f; }; @@ -672,6 +736,11 @@ std::optional readStreamConfig(const std::filesystem::path &sessio stream.ten_bit = json::boolean(json::find(config, {"use_10bit_encoder"})).value_or(false); stream.hdr = json::boolean(json::find(config, {"enable_hdr"})).value_or(false); stream.foveated = json::boolean(json::find(config, {"enable_foveated_encoding"})).value_or(false); + static const string_view foveation_keys[6] = {"foveation_center_size_x", "foveation_center_size_y", + "foveation_center_shift_x", "foveation_center_shift_y", + "foveation_edge_ratio_x", "foveation_edge_ratio_y"}; + for (int i = 0; i < 6; i++) + stream.foveation[i] = static_cast(json::number(json::find(config, {foveation_keys[i]})).value_or(0.0)); stream.gamma = static_cast(json::number(json::find(config, {"encoding_gamma"})).value_or(1.0)); return stream; } @@ -786,6 +855,8 @@ struct Sample { AlvrDeviceMotion head{}; bool hand_valid[2] = {false, false}; AlvrDeviceMotion hand[2]{}; + bool skeleton_valid[2] = {false, false}; // the client tracks that bare hand (no controller in it) + AlvrPose skeleton[2][MVR_HAND_JOINTS]{}; }; // ALVR's server_core is a set of process-wide singletons: one environment (config directory, logger) for the life @@ -804,8 +875,19 @@ class AlvrDriver final : public Driver { system_.ipd = kDefaultIpd; system_.fov[0] = kDefaultFov[0]; system_.fov[1] = kDefaultFov[1]; - std::snprintf(system_.system_name, sizeof(system_.system_name), "MetalVR ALVR Headset"); setViewOffsets(kDefaultIpd); + setModel(options_.headset, "named on the command line"); + if (options_.headset != MVR_HEADSET_UNKNOWN) { + const HeadsetSpec &spec = headsetSpec(options_.headset); + system_.eye_width = spec.eye_width; + system_.eye_height = spec.eye_height; + system_.refresh_hz = spec.default_refresh; + system_.fov[0] = spec.fov[0]; + system_.fov[1] = spec.fov[1]; + video_.frame_width = spec.eye_width * 2; + video_.frame_height = spec.eye_height; + video_.fps = spec.default_refresh; + } } ~AlvrDriver() override { stop(); } @@ -831,6 +913,8 @@ class AlvrDriver final : public Driver { logLine("'%s' is not a hostname a headset can be trusted by", hostname.c_str()); return false; } + if (!prepareWired()) return false; + g_foveated_encoding = options_.foveated_encoding; { std::lock_guard lock(g_process_mutex); @@ -839,7 +923,7 @@ class AlvrDriver final : public Driver { return false; } if (!g_environment_ready) { - if (!prepareSession(session_path_, trustKeys(options_.trusted_clients))) return false; + if (!prepareSession(session_path_, trustKeys(options_.trusted_clients), options_.headset)) return false; const std::string session_log = (dir / "session_log.txt").string(); const std::string crash_log = (dir / "crash_log.txt").string(); alvr_initialize_environment(dir.c_str(), dir.c_str()); @@ -890,6 +974,8 @@ class AlvrDriver final : public Driver { std::lock_guard lock(state_mutex_); mvr_system_info info = system_; info.flags = connected_ ? MVR_SYSTEM_HMD_CONNECTED : 0; + // every Quest tracks bare hands, and ALVR passes their joints on; a Steam Frame has no hand tracking + if (system_.headset != MVR_HEADSET_STEAM_FRAME) info.flags |= MVR_SYSTEM_HAND_TRACKING; return info; } @@ -914,6 +1000,10 @@ class AlvrDriver final : public Driver { if (have_sample) out.flags |= MVR_TRACKING_HEAD_VALID; if (connected && have_sample && now - sample_.received_ns < kTrackingFreshNs) out.flags |= MVR_TRACKING_PRESENTING; + if (connected && battery_[0].valid) { + out.flags |= MVR_TRACKING_BATTERY_VALID | (battery_[0].plugged ? MVR_TRACKING_CHARGING : 0u); + out.head_battery = battery_[0].level; + } out.head = have_sample ? toPose(sample_.head.pose) : mvr_pose{{0, 0, 0, 1}, {0, kDefaultHeadHeight, 0}}; // ALVR's client sends the head pose already predicted and without velocities out.head_linear_velocity = have_sample ? toVec3(sample_.head.linear_velocity) : mvr_vec3{}; @@ -937,10 +1027,14 @@ class AlvrDriver final : public Driver { controller.thumbstick_y = input.thumbstick_y; controller.grip = {{0, 0, 0, 1}, {0, 0, 0}}; controller.aim = controller.grip; + if (connected && battery_[1 + hand].valid) { + controller.flags |= MVR_CONTROLLER_BATTERY_VALID | (battery_[1 + hand].plugged ? MVR_CONTROLLER_CHARGING : 0u); + controller.battery = battery_[1 + hand].level; + } if (!connected || !have_sample || !sample_.hand_valid[hand]) continue; const AlvrDeviceMotion &motion = sample_.hand[hand]; - controller.flags = MVR_CONTROLLER_CONNECTED | MVR_CONTROLLER_POSE_VALID; + controller.flags |= MVR_CONTROLLER_CONNECTED | MVR_CONTROLLER_POSE_VALID; controller.grip = toPose(motion.pose); controller.aim = compose( controller.grip, {{std::sin(kAimPitch * 0.5f), 0.0f, 0.0f, std::cos(kAimPitch * 0.5f)}, kAimOffset}); @@ -952,6 +1046,21 @@ class AlvrDriver final : public Driver { } } + // the hands the client tracks, as it predicted them for the newest packet, like the head + void hands(int64_t, mvr_hand out[2]) override { + std::lock_guard lock(state_mutex_); + const bool fresh = connected_ && sample_.timestamp_ns != 0 && nowNs() - sample_.received_ns < kTrackingFreshNs; + for (int hand = 0; hand < 2; hand++) { + out[hand].flags = 0; + if (!fresh || !sample_.skeleton_valid[hand]) continue; + for (uint32_t j = 0; j < MVR_HAND_JOINTS; j++) { + out[hand].joints[j] = toPose(sample_.skeleton[hand][j]); + out[hand].radii[j] = handJointRadius(j); + } + out[hand].flags = MVR_HAND_TRACKED; + } + } + void haptic(uint32_t hand, float duration_s, float frequency_hz, float amplitude) override { if (!ready_ || !connected_ || hand > 1) return; if (!std::isfinite(duration_s) || !std::isfinite(frequency_hz) || !std::isfinite(amplitude)) return; @@ -1096,8 +1205,26 @@ class AlvrDriver final : public Driver { logLine("no headset is trusted: one that announces itself will be named here and refused"); else logLine("trusted headsets: %s", names.c_str()); + // Before a headset connects, the model its client reported last time, when every trusted headset that has + // reported one is the same model. Anything else stays unknown until a headset connects. + if (options_.headset == MVR_HEADSET_UNKNOWN) { + uint32_t known = MVR_HEADSET_UNKNOWN; + bool agree = true; + for (const ClientList::Client &client : clients->trusted_clients) { + const uint32_t model = headsetFromAlvrName(client.display_name); + if (model == MVR_HEADSET_UNKNOWN) continue; + if (known != MVR_HEADSET_UNKNOWN && known != model) agree = false; + known = model; + } + if (agree && known != MVR_HEADSET_UNKNOWN) setModel(known, "reported by its ALVR client when it last connected"); + } StreamConfig stream; + if (const uint32_t model = currentModel(); model != MVR_HEADSET_UNKNOWN) { + const HeadsetSpec &spec = headsetSpec(model); + stream.stream_width = stream.render_width = spec.eye_width; + stream.stream_height = stream.render_height = spec.eye_height; + } if (const auto stored = readStreamConfig(session_path_); stored && !isPlaceholder(*stored)) { stream = *stored; } else { @@ -1237,7 +1364,11 @@ class AlvrDriver final : public Driver { bringDown(); break; case ALVR_EVENT_BATTERY: - case ALVR_EVENT_PLAYSPACE_SYNC: // ALVR recenters its tracking space itself; the bounds are not used + onBattery(event.battery); + break; + case ALVR_EVENT_PLAYSPACE_SYNC: // ALVR recenters its tracking space itself; only the size is passed on + onPlayArea(event.playspace_sync[0], event.playspace_sync[1]); + break; case ALVR_EVENT_CAPTURE_FRAME: default: break; @@ -1245,16 +1376,45 @@ class AlvrDriver final : public Driver { } void onConnected() { + // which headset this is: what its client reported, which ALVR saved with the client before this event + if (const auto clients = readClientList(session_path_)) { + // the client furthest into its connection; one stuck in "Connecting" from an earlier attempt ranks last + const auto rank = [](const std::string &state) { + return state == "Streaming" ? 3 : state == "Connected" ? 2 : state == "Connecting" ? 1 : 0; + }; + const ClientList::Client *client = nullptr; + for (const ClientList::Client &c : clients->trusted_clients) + if (rank(c.state) > 0 && (!client || rank(c.state) > rank(client->state))) client = &c; + const uint32_t reported = client ? headsetFromAlvrName(client->display_name) : MVR_HEADSET_UNKNOWN; + if (client) logLine("headset '%s' reports itself as '%s'", client->hostname.c_str(), client->display_name.c_str()); + if (options_.headset != MVR_HEADSET_UNKNOWN) { + if (reported != MVR_HEADSET_UNKNOWN && reported != options_.headset) + logLine("warning: the headset reports itself as %s, but --headset names %s; using %s", + mvr_describe_headset(reported)->name, mvr_describe_headset(options_.headset)->name, + mvr_describe_headset(options_.headset)->name); + } else { + setModel(reported, reported != MVR_HEADSET_UNKNOWN ? "reported by its ALVR client" + : "its ALVR client reports no model MetalVR knows"); + } + } // ALVR wrote what it negotiated with this client to session.json before reporting the connection if (const auto stream = readStreamConfig(session_path_)) { applyStream(*stream); logLine("headset connected: %ux%u per eye at %.0f Hz, %s", stream->stream_width, stream->stream_height, stream->refresh_hz, stream->codec == Codec::H264 ? "H.264" : "HEVC"); if (stream->av1) logLine("the headset negotiated AV1, which metalvrd does not encode; sending HEVC"); + const HeadsetSpec &spec = headsetSpec(currentModel()); + if (!withinDecoder(spec, stream->stream_width * 2, stream->stream_height, stream->refresh_hz)) + logLine("warning: %ux%u at %.0f Hz is more than %s's video decoder is known to handle; lower the stream's " + "resolution in ALVR's settings if the picture stutters", + stream->stream_width * 2, stream->stream_height, stream->refresh_hz, + mvr_describe_headset(spec.headset)->name); // the pins rule these out; they can only differ if the settings were changed while metalvrd was running - if (stream->foveated || stream->full_range || stream->ten_bit || stream->hdr || stream->gamma != 1.0f) + const bool unexpected_foveation = stream->foveated != options_.foveated_encoding; + if (unexpected_foveation || stream->full_range || stream->ten_bit || stream->hdr || stream->gamma != 1.0f) logLine("warning: the headset was told to expect%s%s%s%s%s, which metalvrd does not produce; restart metalvrd", - stream->foveated ? " foveated encoding" : "", stream->full_range ? " full-range video" : "", + unexpected_foveation ? (stream->foveated ? " foveated encoding" : " no foveated encoding") : "", + stream->full_range ? " full-range video" : "", stream->ten_bit ? " 10-bit video" : "", stream->hdr ? " HDR" : "", stream->gamma != 1.0f ? " an encoding gamma" : ""); } else { @@ -1280,25 +1440,134 @@ class AlvrDriver final : public Driver { std::lock_guard lock(state_mutex_); input_[0] = input_[1] = {}; sample_.hand_valid[0] = sample_.hand_valid[1] = false; + battery_[0] = battery_[1] = battery_[2] = {}; } generation_++; } void applyStream(const StreamConfig &stream) { + // foveated: each eye is composed at the stream's view size and squeezed to ALVR's optimized size, which is what + // the client decodes and stretches back + Foveation foveation; + if (stream.foveated && options_.foveated_encoding) + foveation = computeFoveation(stream.stream_width, stream.stream_height, stream.foveation[0], stream.foveation[1], + stream.foveation[2], stream.foveation[3], stream.foveation[4], stream.foveation[5]); + const uint32_t frame_width = foveation.enabled ? foveation.optimized_eye_width * 2 : stream.stream_width * 2; + const uint32_t frame_height = foveation.enabled ? foveation.optimized_eye_height : stream.stream_height; std::lock_guard lock(state_mutex_); const bool changed = system_.eye_width != stream.render_width || system_.eye_height != stream.render_height || system_.refresh_hz != stream.refresh_hz || video_.codec != stream.codec || - video_.frame_width != stream.stream_width * 2 || video_.frame_height != stream.stream_height; + video_.frame_width != frame_width || video_.frame_height != frame_height || + video_.foveation.enabled != foveation.enabled; system_.eye_width = stream.render_width; system_.eye_height = stream.render_height; system_.refresh_hz = stream.refresh_hz; + setRefreshRatesLocked(); video_.codec = stream.codec; - video_.frame_width = stream.stream_width * 2; - video_.frame_height = stream.stream_height; + video_.frame_width = frame_width; + video_.frame_height = frame_height; + video_.foveation = foveation; + if (foveation.enabled && changed) + logLine("foveated encoding: %ux%u per eye sent as %ux%u", stream.stream_width, stream.stream_height, + foveation.optimized_eye_width, foveation.optimized_eye_height); video_.fps = stream.refresh_hz; if (changed) generation_++; } + // --- the headset's model --- + + uint32_t currentModel() { + std::lock_guard lock(state_mutex_); + return system_.headset; + } + + // Names the headset (MVR_HEADSET_UNKNOWN: a headset MetalVR does not know). Its controllers are what ALVR + // reports for every client, the Touch family's inputs, unless the model is known to have others. + void setModel(uint32_t headset, const char *why) { + std::lock_guard lock(state_mutex_); + if (system_.headset == headset && system_.system_name[0]) return; + const mvr_headset_description *description = mvr_describe_headset(headset); + system_.headset = headset; + system_.controllers = headset == MVR_HEADSET_UNKNOWN ? MVR_CONTROLLERS_TOUCH : description->controllers; + std::snprintf(system_.system_name, sizeof(system_.system_name), "%s", + headset == MVR_HEADSET_UNKNOWN ? "MetalVR ALVR Headset" : description->name); + setRefreshRatesLocked(); + generation_++; + if (headset != MVR_HEADSET_UNKNOWN) logLine("headset: %s (%s)", description->name, why); + } + + // the model's refresh rates, with the one the stream runs at among them + void setRefreshRatesLocked() { + const HeadsetSpec &spec = headsetSpec(system_.headset); + std::vector rates(spec.refresh_rates, spec.refresh_rates + spec.refresh_rate_count); + if (system_.refresh_hz > 0 && std::none_of(rates.begin(), rates.end(), [&](float r) { + return std::fabs(r - system_.refresh_hz) < 0.5f; + })) + rates.push_back(system_.refresh_hz); + std::sort(rates.begin(), rates.end()); + if (rates.size() > MVR_MAX_REFRESH_RATES) rates.resize(MVR_MAX_REFRESH_RATES); + system_.refresh_rate_count = static_cast(rates.size()); + std::fill(std::begin(system_.refresh_rates), std::end(system_.refresh_rates), 0.0f); + std::copy(rates.begin(), rates.end(), system_.refresh_rates); + } + + // --- what the headset says of itself besides video and tracking --- + + struct Battery { + bool valid = false; + bool plugged = false; + float level = 0.0f; + }; + + void onBattery(const AlvrBatteryInfo &info) { + int slot = -1; // 0 head, 1 left hand, 2 right hand + if (info.device_id == head_id_) slot = 0; + if (info.device_id == hand_id_[0]) slot = 1; + if (info.device_id == hand_id_[1]) slot = 2; + if (slot < 0 || !std::isfinite(info.gauge_value)) return; + std::lock_guard lock(state_mutex_); + battery_[slot] = {true, info.is_plugged, std::clamp(info.gauge_value, 0.0f, 1.0f)}; + } + + void onPlayArea(float width, float depth) { + if (!std::isfinite(width) || !std::isfinite(depth) || width < 0.0f || depth < 0.0f || width > 100.0f || + depth > 100.0f) + return; + std::lock_guard lock(state_mutex_); + if (std::fabs(system_.play_area_width - width) < 1e-3f && std::fabs(system_.play_area_depth - depth) < 1e-3f) + return; + system_.play_area_width = width; + system_.play_area_depth = depth; + generation_++; + } + + // A trusted "client.wired" is a headset on USB, which ALVR reaches with adb. ALVR would download Google's + // platform-tools when it finds no adb; here an adb must exist before ALVR starts, so it never does. + bool prepareWired() { + const bool wired = std::any_of(options_.trusted_clients.begin(), options_.trusted_clients.end(), + [](const std::string &name) { return sameHostname(name, kWiredClient); }); + if (!wired) return true; + std::filesystem::path adb = options_.adb.empty() ? adbOnPath() : std::filesystem::path(options_.adb); + if (adb.empty() || !isExecutable(adb)) { + logLine("a headset on USB (%s) needs adb: install Android's platform-tools (brew install android-platform-tools) " + "or name one with --adb; MetalVR does not let ALVR download it", + std::string(kWiredClient).c_str()); + return false; + } + if (!options_.adb.empty()) { + // ALVR runs "adb" from PATH: the one named goes first there + if (adb.filename() != "adb") { + logLine("--adb must name a program called adb, not %s", adb.c_str()); + return false; + } + const char *path = std::getenv("PATH"); + const std::string dir = std::filesystem::absolute(adb).parent_path().string(); + ::setenv("PATH", (dir + (path && *path ? ":" + std::string(path) : std::string())).c_str(), 1); + } + logLine("a headset on USB is reached with %s", adb.c_str()); + return true; + } + void setViewOffsets(float ipd) { view_offset_[0] = {{0, 0, 0, 1}, {-ipd * 0.5f, 0, 0}}; view_offset_[1] = {{0, 0, 0, 1}, {ipd * 0.5f, 0, 0}}; @@ -1327,8 +1596,12 @@ class AlvrDriver final : public Driver { sample.timestamp_ns = timestamp_ns; sample.received_ns = nowNs(); if (timestamp_ns == 0 || !alvr_get_device_motion(head_id_, timestamp_ns, &sample.head)) return; - for (int hand = 0; hand < 2; hand++) + for (int hand = 0; hand < 2; hand++) { sample.hand_valid[hand] = alvr_get_device_motion(hand_id_[hand], timestamp_ns, &sample.hand[hand]); + // ALVR's 26 joints are OpenXR's, in its order + sample.skeleton_valid[hand] = alvr_get_hand_skeleton(hand == 0 ? ALVR_HAND_TYPE_LEFT : ALVR_HAND_TYPE_RIGHT, + timestamp_ns, sample.skeleton[hand]); + } std::lock_guard lock(state_mutex_); sample_ = sample; } @@ -1416,6 +1689,7 @@ class AlvrDriver final : public Driver { mvr_pose view_offset_[2]{}; Sample sample_; ControllerInput input_[2]; + Battery battery_[3]; // head, left hand, right hand bool controllers_enabled_ = false; std::mutex video_mutex_; // the parameter sets last given to sendVideoConfig diff --git a/daemon/driver_rift.mm b/daemon/driver_rift.mm index a6e344b..e0e7aaf 100644 --- a/daemon/driver_rift.mm +++ b/daemon/driver_rift.mm @@ -32,7 +32,14 @@ #include #include "common.h" +#include "camera_passthrough.h" +#include "direct_backend.h" #include "driver.h" +#include "headsets.h" +#include "metalvr_headsets.h" +#include "psvr2_lens_table.h" +#include "psvr2_protocol.h" +#include "rift_s_protocol.h" #include "window_capture.h" #include "metalvr_math.h" @@ -73,6 +80,8 @@ constexpr int64_t kSampleNs = 1'000'000; // 1000 Hz constexpr float kRefreshHz = 90.0f; +// the headset cameras' half field of view, taken as 55 degrees until their calibration is read (tan 55) +constexpr float kPassthroughHalfFovTan = 1.428f; // Monado's CV1 distortion (rift_driver.c CV1_DISTORTIONS): the default eye relief's Catmull-Rom magnification // curve, colour fringing correction, and lens diameter @@ -191,6 +200,30 @@ mvr_quat ahead(int64_t time_ns, float *seconds) const { float k[11]; }; +// The lens of a headset other than the CV1 (direct_backend.h DirectLens), and how its eye sits on the panel +struct Lens { + uint model; // 0: the CV1's curve (Distortion); 1: panotools (Rift S); 2: Vive, Vive Pro, Index; + // 3: PlayStation VR2 (its lens table, kPsvr2Lens) + uint portrait; // the eye's view is turned a quarter right on the panel (Rift S) + uint flip_y; // the Vive Pro 2's image is upside down + uint pad; + float4 eye_rotation; // the eye's view in the head, a quaternion (x, y, z, w): the Index's lenses are canted + float4 pano_k; // panotools k0..k3 + float4 pano_k4_ab; // k4, then the aberration of red, green, blue + float4 pano; // scale, lens center x, y (meters) + float4 pano_viewport; // the view's size (meters) + float4 vive; // aspect x over y, grow for undistort + float4 vive_center[3]; // per colour + float4 vive_coeff[3]; // per colour, k1..k4 + float4 psvr2; // the eye's lens offset x, y, and cos and sin of its tilt + float4 psvr2_center; // the lens centre in the rendered image x, y, and its scale x, y +}; + +static float3 qrot(float4 q, float3 v) { + float3 t = 2.0 * cross(q.xyz, v); + return v + q.w * t + cross(q.xyz, t); +} + struct VOut { float4 position [[position]]; }; @@ -239,9 +272,69 @@ static float magnification(constant Distortion &d, float rho) { return hermite(p0, m0, p1, m1, t); } +// Where each colour of this pixel is in the rendered eye image (0..1 across and down), for the lens models other +// than the CV1's: Monado's u_compute_distortion_panotools and u_compute_distortion_vive. +static void lens_uv(constant Distortion &d, constant Lens &lens, float2 position, thread float2 src[3]) { + float2 uv = (position - d.viewport_origin) / d.viewport_size; + // the panel's view of the eye turned back to the eye's own (Monado's u_device_rotation_right) + if (lens.portrait != 0) { + float2 c = uv - 0.5; + uv = float2(c.y, -c.x) + 0.5; + } + if (lens.model == 3) { + // Monado's psvr2_compute_distortion_asymmetric + float2 p = float2(lens.psvr2.x + (2.0 * uv.x - 1.0) * 0.9803922 * 0.9394987, + lens.psvr2.y + (2.0 * uv.y - 1.0) * 0.9394987); + float r = dot(p, p) * 1023.0; + int i = int(r); + float3 scale; + if (i > -1 && i + 1 < 1024) { + float f = r - float(i); + scale = f * kPsvr2Lens[i + 1] + (1.0 - f) * kPsvr2Lens[i]; + } else { + float d = r - 1023.0; + scale = float3(d * 0.003648639 + 1.592586, d * 0.004862547 + 1.692979, d * 0.007425308 + 1.841771); + } + for (int c = 0; c < 3; ++c) { + float2 q = float2(p.x, c == 1 ? p.y - 0.0002302693 : p.y) * scale[c]; + src[c] = float2(lens.psvr2.z * q.x - lens.psvr2.w * q.y, lens.psvr2.w * q.x + lens.psvr2.z * q.y) * + lens.psvr2_center.zw + lens.psvr2_center.xy; + } + return; + } + if (lens.model == 1) { + float2 r = (uv * lens.pano_viewport.xy - lens.pano.yz) / lens.pano.x; + float m = length(r); + float k = lens.pano_k.x + lens.pano_k.y * m + lens.pano_k.z * m * m + lens.pano_k.w * m * m * m + + lens.pano_k4_ab.x * m * m * m * m; + for (int c = 0; c < 3; ++c) + src[c] = (r * k * lens.pano.x * lens.pano_k4_ab[1 + c] + lens.pano.yz) / lens.pano_viewport.xy; + return; + } + float aspect = lens.vive.x; + float factor = 0.5 / (1.0 + lens.vive.y); + for (int c = 0; c < 3; ++c) { + float2 t = float2(2.0 * uv.x - 1.0, (2.0 * uv.y - 1.0) / aspect) - lens.vive_center[c].xy; + float r2 = dot(t, t); + float4 k = lens.vive_coeff[c]; + float s = 1.0 / (1.0 + r2 * (k.x + r2 * (k.y + r2 * k.z))) + k.w; + src[c] = 0.5 + (t * s + lens.vive_center[c].xy) * float2(factor, factor * aspect); + if (lens.flip_y != 0) src[c].y = 1.0 - src[c].y; + } +} + // The direction each colour channel of this pixel shows, as a tangent from the eye's axis (x right, y down): the // lens bends red, green and blue apart, so each is looked up where the lens takes it. -static void channel_tangents(constant Distortion &d, float2 position, thread float2 out[3]) { +static void channel_tangents(constant Distortion &d, constant Lens &lens, float2 position, thread float2 out[3]) { + if (lens.model != 0) { + // a position in the rendered image is a direction: the image spans the eye's field of view + float2 src[3]; + lens_uv(d, lens, position, src); + for (int c = 0; c < 3; ++c) + out[c] = float2(src[c].x * (d.fov_tan.x + d.fov_tan.y) - d.fov_tan.x, + src[c].y * (d.fov_tan.z + d.fov_tan.w) - d.fov_tan.z); + return; + } float2 uv = (position - d.viewport_origin) / d.viewport_size; float2 ndc = uv * 2.0 - 1.0; // the halves of the panel are mirror images. The lens geometry (lens_center) is the right eye's, where the lens @@ -263,16 +356,19 @@ static void channel_tangents(constant Distortion &d, float2 position, thread flo }; fragment float4 distort(VOut in [[stage_in]], constant Distortion &d [[buffer(0)]], - constant SceneRotation &rotation [[buffer(1)]], texture2d frame [[texture(0)]], - sampler s [[sampler(0)]]) { + constant SceneRotation &rotation [[buffer(1)]], constant Lens &lens [[buffer(2)]], + texture2d frame [[texture(0)]], sampler s [[sampler(0)]]) { float2 t[3]; - channel_tangents(d, in.position.xy, t); + channel_tangents(d, lens, in.position.xy, t); float3x3 now_to_rendered = float3x3(rotation.columns[0].xyz, rotation.columns[1].xyz, rotation.columns[2].xyz); + float4 eye = lens.eye_rotation; + float4 eye_inverse = float4(-eye.xyz, eye.w); float3 color; for (int c = 0; c < 3; ++c) { // The direction this pixel shows, as the head is now, seen from the head the frame was rendered for: the frame - // is turned by how far the head has turned since. The eye looks down -z with +y up; tangent y runs down. - float3 v = now_to_rendered * float3(t[c].x, -t[c].y, -1.0); + // is turned by how far the head has turned since. The eye looks down -z with +y up; tangent y runs down. A + // canted eye's direction goes into the head's frame for the turn, and back. + float3 v = qrot(eye_inverse, now_to_rendered * qrot(eye, float3(t[c].x, -t[c].y, -1.0))); if (v.z > -1e-4) { color[c] = 0.0; continue; @@ -314,6 +410,8 @@ static float3 grid_color(float3 w) { float4 eye[2]; // each eye's position in the world, meters (the head is at the origin) float4 window; // x: 1 when a window is shown; y: its height over its width float4 logo; // x: 1 when there is a logo; y: its height over its width + float4 passthrough; // x: 1 when the headset's cameras are shown; y: the tangent of their half field of view; + // z: 1 when the picture holds both eyes side by side // each Touch controller, in the world with the head at the origin: where it is, its three axes, and its inputs. // hand_in0 is the thumbstick's x and y, the trigger and the grip; hand_in1 is the buttons pressed and touched // (mvr_button bits), 1 when the controller is on, and -1 for the left hand or 1 for the right @@ -656,12 +754,27 @@ static float4 hands(float3 origin, float3 dir, constant Home &home) { return result; } +// The headset's cameras, as straight-ahead views of a fixed field of view: what the eye looks at in the head's frame, +// found in the camera's picture (its own eye's half of an Index's side-by-side picture) +static float3 passthrough_color(float3 dir_head, uint eye, constant Home &home, texture2d camera, sampler s) { + if (dir_head.z > -1e-3) return float3(0.0); + float2 t = float2(dir_head.x / -dir_head.z, -dir_head.y / -dir_head.z) / home.passthrough.y; + float2 uv = t * 0.5 + 0.5; + if (any(uv < 0.0) || any(uv > 1.0)) return float3(0.0); + if (home.passthrough.z > 0.5) uv.x = (float(eye) + uv.x) * 0.5; + return camera.sample(s, uv).rgb; +} + fragment float4 scene(VOut in [[stage_in]], constant Distortion &d [[buffer(0)]], constant Home &home [[buffer(1)]], - texture2d logo [[texture(0)]], texture2d window [[texture(1)]], + constant Lens &lens [[buffer(2)]], texture2d logo [[texture(0)]], + texture2d window [[texture(1)]], texture2d camera [[texture(2)]], sampler s [[sampler(0)]]) { float2 t[3]; - channel_tangents(d, in.position.xy, t); - float3x3 head_to_world = float3x3(home.columns[0].xyz, home.columns[1].xyz, home.columns[2].xyz); + channel_tangents(d, lens, in.position.xy, t); + // a canted eye looks a little aside from the head: its view into the head's frame first + float3x3 eye_to_head = float3x3(qrot(lens.eye_rotation, float3(1, 0, 0)), qrot(lens.eye_rotation, float3(0, 1, 0)), + qrot(lens.eye_rotation, float3(0, 0, 1))); + float3x3 head_to_world = float3x3(home.columns[0].xyz, home.columns[1].xyz, home.columns[2].xyz) * eye_to_head; float3 color; if (home.test_scene) { for (int c = 0; c < 3; ++c) { @@ -673,7 +786,9 @@ fragment float4 scene(VOut in [[stage_in]], constant Distortion &d [[buffer(0)]] } // the space is far away and soft: one look-up for all three colours; the panels are near and sharp: one per colour const float3 origin = home.eye[d.eye].xyz; - const float3 background = space_color(normalize(head_to_world * float3(t[1].x, -t[1].y, -1.0))); + const float3 background = home.passthrough.x > 0.5 + ? passthrough_color(normalize(eye_to_head * float3(t[1].x, -t[1].y, -1.0)), d.eye, home, camera, s) + : space_color(normalize(head_to_world * float3(t[1].x, -t[1].y, -1.0))); for (int c = 0; c < 3; ++c) { float3 dir = normalize(head_to_world * float3(t[c].x, -t[c].y, -1.0)); float4 panel = panels(origin, dir, home, logo, window, s); @@ -687,6 +802,21 @@ fragment float4 scene(VOut in [[stage_in]], constant Distortion &d [[buffer(0)]] } )"; +// The PlayStation VR2's lens table (psvr2_lens_table.h) as a Metal constant, put in front of kShaderSource +NSString *psvr2LensSource() { + static NSString *source = [] { + std::string text = "#include \nusing namespace metal;\nconstant float3 kPsvr2Lens[1024] = {\n"; + char entry[64]; + for (const auto &row : psvr2::kLensTable) { + std::snprintf(entry, sizeof(entry), "float3(%.6f, %.6f, %.6f),\n", row[0], row[1], row[2]); + text += entry; + } + text += "};\n"; + return [NSString stringWithUTF8String:text.c_str()]; + }(); + return source; +} + struct DistortionUniforms { float lens_center[2]; float tan_scale[2]; @@ -701,11 +831,30 @@ fragment float4 scene(VOut in [[stage_in]], constant Distortion &d [[buffer(0)]] }; static_assert(sizeof(DistortionUniforms) == 128, "the shader's Distortion struct"); +struct LensUniforms { + uint32_t model; + uint32_t portrait; + uint32_t flip_y; + uint32_t pad; + float eye_rotation[4]; + float pano_k[4]; + float pano_k4_ab[4]; + float pano[4]; + float pano_viewport[4]; + float vive[4]; + float vive_center[3][4]; + float vive_coeff[3][4]; + float psvr2[4]; + float psvr2_center[4]; +}; +static_assert(sizeof(LensUniforms) == 240, "the shader's Lens struct"); + struct HomeUniforms { float columns[3][4]; // head to world float eye[2][4]; // each eye's position in the world, meters float window[4]; // 1 when a window is shown, its height over its width float logo[4]; // 1 when there is a logo, its height over its width + float passthrough[4]; // 1 when the cameras are shown, the tangent of their half field of view, 1 for stereo float hand_pos[2][4]; // each controller: where it is, seen from the head float hand_x[2][4]; // its axes float hand_y[2][4]; @@ -715,7 +864,7 @@ fragment float4 scene(VOut in [[stage_in]], constant Distortion &d [[buffer(0)]] uint32_t test_scene; uint32_t pad[3]; }; -static_assert(sizeof(HomeUniforms) == 320, "the shader's Home struct"); +static_assert(sizeof(HomeUniforms) == 336, "the shader's Home struct"); // ---- the Touch controllers and the remote, through the headset's radio ---- // @@ -1676,19 +1825,294 @@ void serviceHaptics(int hand, int64_t now_ns) { std::atomic recenter_requested_{false}; }; -class RiftDriver final : public Driver { +// ---- the controllers of the headsets other than the CV1: rotation from each one's IMU, placed beside the body ---- + +class DirectControllers { +public: + void imu(int hand, int64_t time_ns, const float accel[3], const float gyro[3]) { + if (hand < 0 || hand > 1) + return; + std::lock_guard lock(mutex_); + Hand &h = hands_[hand]; + h.fusion.update(time_ns, accel, gyro); + h.last_ns = time_ns; + h.has_imu = true; + } + + void input(int hand, const DirectControllerInput &input, int64_t time_ns) { + if (hand < 0 || hand > 1) + return; + std::lock_guard lock(mutex_); + Hand &h = hands_[hand]; + if (!h.last_ns || time_ns - h.last_ns > kTouchGoneNs) + logInfo("direct: %s controller is on", hand == 0 ? "left" : "right"); + h.input = input; + h.last_ns = time_ns; + } + + void grip(int hand, const mvr_pose &grip, const mvr_pose &aim) { + if (hand < 0 || hand > 1) + return; + std::lock_guard lock(mutex_); + hands_[hand].grip = grip; + hands_[hand].aim = aim; + } + + void recenter() { + std::lock_guard lock(mutex_); + for (Hand &h : hands_) + h.aligned = false; + } + + void clear() { + std::lock_guard lock(mutex_); + for (Hand &h : hands_) + h = Hand{}; + } + + // As TouchRadio::fill: a controller's rotation is its own IMU's. With a tracker's pose (the base stations) it is + // where the tracker puts it, its heading pulled toward the tracker's; without, its heading is lined up with the + // head's when it is first seen, and it is placed where an arm would hold a controller pointing that way. + void fill(mvr_controller_state out[2], const mvr_pose &body, const TrackingSpace &space, int64_t time_ns, + DirectBackend *tracker = nullptr) { + const mvr_quat heading = space.heading; + std::lock_guard lock(mutex_); + for (int hand = 0; hand < 2; ++hand) { + Hand &h = hands_[hand]; + mvr_controller_state &c = out[hand]; + std::memset(&c, 0, sizeof(c)); + if (!h.last_ns || time_ns - h.last_ns > kTouchGoneNs) + continue; + const DirectControllerInput &in = h.input; + c.flags = MVR_CONTROLLER_CONNECTED; + c.buttons = in.buttons; + c.touches = in.touches; + c.trigger = in.trigger; + c.squeeze = in.squeeze; + c.thumbstick_x = in.thumbstick_x; + c.thumbstick_y = in.thumbstick_y; + c.trackpad_x = in.trackpad_x; + c.trackpad_y = in.trackpad_y; + c.trackpad_force = in.trackpad_force; + c.squeeze_force = in.squeeze_force; + if (in.has_fingers) { + c.flags |= MVR_CONTROLLER_FINGERS_VALID; + std::memcpy(c.fingers, in.fingers, sizeof(c.fingers)); + } + if (in.has_battery) { + c.flags |= MVR_CONTROLLER_BATTERY_VALID | (in.charging ? MVR_CONTROLLER_CHARGING : 0u); + c.battery = in.battery; + } + if (c.trigger > 0.75f) + c.buttons |= MVR_BUTTON_TRIGGER; + if (c.squeeze > 0.75f) + c.buttons |= MVR_BUTTON_SQUEEZE; + if (!h.has_imu || !h.fusion.ready()) + continue; + TrackedPose tracked; + if (tracker && tracker->trackedPose(hand == 0 ? LighthouseTracking::kLeft : LighthouseTracking::kRight, + tracked) && + tracked.position_tracked) { + const mvr_quat own = h.fusion.rotation(); + const mvr_quat target = quatYawOnly(quatMul(tracked.orientation, quatConj(own))); + const int64_t now = nowNs(); + if (!h.on_tracker) { + h.yaw_align = target; + h.on_tracker = true; + } else if (now > h.align_ns) { + const float dt = std::min(static_cast(now - h.align_ns) / 1e9f, 0.1f); + h.yaw_align = yawToward(h.yaw_align, target, 1.0f - std::exp(-dt / kYawPullSeconds)); + } + h.align_ns = now; + const mvr_quat turn = h.fusion.ahead(time_ns, nullptr); + const mvr_pose hand_pose{quatNormalize(quatMul(heading, quatMul(h.yaw_align, quatMul(own, turn)))), + space.toApp(tracked.position)}; + c.linear_velocity = quatRotate(heading, tracked.velocity); + c.angular_velocity = quatRotate(hand_pose.orientation, h.fusion.gyro()); + c.grip = poseMul(hand_pose, h.grip); + c.aim = poseMul(hand_pose, h.aim); + c.flags |= MVR_CONTROLLER_POSE_VALID; + h.aligned = false; + continue; + } + h.on_tracker = false; + mvr_vec3 angular{}; + mvr_quat q = h.fusion.predict(time_ns, &angular); + if (!h.aligned) { + const mvr_quat controller_yaw = quatYawOnly(quatMul(heading, q)); + h.yaw_fix = quatMul(quatYawOnly(body.orientation), quatConj(controller_yaw)); + h.aligned = true; + } + q = quatNormalize(quatMul(h.yaw_fix, quatMul(heading, q))); + const float side = hand == 0 ? -1.0f : 1.0f; + const mvr_vec3 pointing = quatRotate(quatConj(body.orientation), quatRotate(q, mvr_vec3{0, 0, -1})); + const float raised = pointing.y > 0 ? pointing.y : 0; + const mvr_vec3 elbow{side * 0.20f, -0.32f + 0.10f * raised, -0.10f - 0.06f * raised}; + const float forearm = 0.28f; + const mvr_vec3 in_body{elbow.x + pointing.x * forearm, elbow.y + pointing.y * forearm, + elbow.z + pointing.z * forearm}; + const mvr_pose hand_pose{q, quatRotate(body.orientation, in_body) + body.position}; + c.angular_velocity = quatRotate(h.yaw_fix, quatRotate(heading, angular)); + c.grip = poseMul(hand_pose, h.grip); + c.aim = poseMul(hand_pose, h.aim); + c.flags |= MVR_CONTROLLER_POSE_VALID; + } + } + +private: + struct Hand { + int64_t last_ns = 0; + bool has_imu = false; + Fusion fusion; + bool aligned = false; + mvr_quat yaw_fix = quatIdentity(); + bool on_tracker = false; + mvr_quat yaw_align = quatIdentity(); + int64_t align_ns = 0; + DirectControllerInput input; + mvr_pose grip{quatIdentity(), {0, 0, 0}}; + mvr_pose aim{quatIdentity(), {0, 0, 0}}; + }; + std::mutex mutex_; + Hand hands_[2]; +}; + +// what the driver draws for, before and after the headset is identified +DirectModel cv1Model() { + DirectModel m; + m.headset = MVR_HEADSET_RIFT_CV1; + m.controllers = MVR_CONTROLLERS_TOUCH; + m.name = "Oculus Rift CV1"; + m.display_vendor = kDisplayVendor; + m.display_products = {kDisplayProductCV1}; + m.refresh_hz = kRefreshHz; + m.refresh_rates = {kRefreshHz}; + m.eye_width = 1344; + m.eye_height = 1600; + m.fov[0] = kFov[0]; + m.fov[1] = kFov[1]; + return m; +} + +// a family whose headset is not identified yet: nothing is assumed of it but a size to render at +DirectModel unidentifiedModel(const RiftOptions &options) { + if (options.psvr2) { + // the PlayStation VR2 is the only headset of its driver: its lens calibration comes when it is found + DirectModel m = psvr2Model(psvr2::defaultLensCalibration()); + m.name = "PlayStation VR2 (not found yet)"; + return m; + } + const bool lighthouse = options.lighthouse; + DirectModel m; + m.name = lighthouse ? "Lighthouse headset (not identified)" : "Rift (not identified)"; + m.controllers = MVR_CONTROLLERS_UNKNOWN; + m.refresh_hz = 90; + m.refresh_rates = {90}; + m.eye_width = lighthouse ? 1728 : 1344; + m.eye_height = lighthouse ? 1920 : 1600; + m.fov[0] = kFov[0]; + m.fov[1] = kFov[1]; + return m; +} + +class RiftDriver final : public Driver, public DirectHost { public: - explicit RiftDriver(const RiftOptions &options) : options_(options) { + explicit RiftDriver(const RiftOptions &options) : options_(options), model_(unidentifiedModel(options)) { space_.offset = {0.0f, options_.head_height, 0.0f}; touch_.setSensors(&sensors_); + refresh_hz_ = model_.refresh_hz; } ~RiftDriver() override { stop(); } - const char *name() const override { return "rift"; } + const char *name() const override { return options_.psvr2 ? "psvr2" : options_.lighthouse ? "lighthouse" : "rift"; } bool displaysFrames() const override { return true; } bool needsMainRunLoop() const override { return true; } bool start() override { + if (!setUpGpu()) + return false; + if (!options_.test_scene) { + logo_ = loadLogo(); + window_capture_ = makeWindowCapture(); + } + + running_ = true; + hid_thread_ = std::thread([this] { hidThread(); }); + scene_thread_ = std::thread([this] { sceneThread(); }); + return true; + } + + // Draws the lens-check grid for `model`, with the head straight ahead, through the same pipeline and uniforms the + // headset is drawn with, into a panel-sized BGRA picture. For checking the lens models without a headset. + bool drawTestPanel(const DirectModel &model, uint32_t width, uint32_t height, std::vector &bgra, + bool passthrough = false) { + if (!device_ && !setUpGpu()) + return false; + // for passthrough: a stereo camera picture whose left half is red and right half blue + id camera = nil; + if (passthrough) { + MTLTextureDescriptor *cd = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm_sRGB + width:64 + height:32 + mipmapped:NO]; + camera = [device_ newTextureWithDescriptor:cd]; + std::vector pixels(64 * 32); + for (int y = 0; y < 32; ++y) + for (int x = 0; x < 64; ++x) + pixels[y * 64 + x] = x < 32 ? 0xffff0000u : 0xff0000ffu; // BGRA in memory: red, blue + [camera replaceRegion:MTLRegionMake2D(0, 0, 64, 32) mipmapLevel:0 withBytes:pixels.data() bytesPerRow:64 * 4]; + } + setModel(model); + MTLTextureDescriptor *td = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm_sRGB + width:width + height:height + mipmapped:NO]; + td.usage = MTLTextureUsageRenderTarget; + td.storageMode = MTLStorageModeShared; + id target = [device_ newTextureWithDescriptor:td]; + MTLRenderPassDescriptor *pass = [MTLRenderPassDescriptor renderPassDescriptor]; + pass.colorAttachments[0].texture = target; + pass.colorAttachments[0].loadAction = MTLLoadActionClear; + pass.colorAttachments[0].clearColor = MTLClearColorMake(0, 0, 0, 1); + pass.colorAttachments[0].storeAction = MTLStoreActionStore; + id cmd = [queue_ commandBuffer]; + id enc = [cmd renderCommandEncoderWithDescriptor:pass]; + [enc setRenderPipelineState:scene_pipeline_]; + [enc setFragmentTexture:blank_ atIndex:0]; + [enc setFragmentTexture:blank_ atIndex:1]; + [enc setFragmentTexture:(camera ?: blank_) atIndex:2]; + [enc setFragmentSamplerState:sampler_ atIndex:0]; + HomeUniforms home{}; + const float columns[3][4] = {{1, 0, 0, 0}, {0, 1, 0, 0}, {0, 0, 1, 0}}; + std::memcpy(home.columns, columns, sizeof(columns)); + home.test_scene = passthrough ? 0 : 1; + if (passthrough) { + home.passthrough[0] = 1; + home.passthrough[1] = kPassthroughHalfFovTan; + home.passthrough[2] = 1; + } + [enc setFragmentBytes:&home length:sizeof(home) atIndex:1]; + for (uint32_t eye = 0; eye < 2; ++eye) { + const MTLViewport viewport = eyeViewport(model, eye, width, height); + DistortionUniforms u = uniforms(model, eye, viewport); + LensUniforms lens = lensUniforms(model, eye); + [enc setViewport:viewport]; + [enc setFragmentBytes:&u length:sizeof(u) atIndex:0]; + [enc setFragmentBytes:&lens length:sizeof(lens) atIndex:2]; + [enc drawPrimitives:MTLPrimitiveTypeTriangle vertexStart:0 vertexCount:3]; + } + [enc endEncoding]; + [cmd commit]; + [cmd waitUntilCompleted]; + if (cmd.status != MTLCommandBufferStatusCompleted) + return false; + bgra.resize(static_cast(width) * height * 4); + [target getBytes:bgra.data() bytesPerRow:width * 4 fromRegion:MTLRegionMake2D(0, 0, width, height) mipmapLevel:0]; + return true; + } + +private: + bool setUpGpu() { device_ = MTLCreateSystemDefaultDevice(); if (!device_) { logInfo("rift: no Metal device"); @@ -1697,7 +2121,9 @@ bool start() override { queue_ = [device_ newCommandQueue]; queue_.label = @"metalvr-rift"; NSError *error = nil; - id library = [device_ newLibraryWithSource:kShaderSource options:nil error:&error]; + // the PlayStation VR2's lens table goes in as a constant ahead of the rest + NSString *source = [psvr2LensSource() stringByAppendingString:kShaderSource]; + id library = [device_ newLibraryWithSource:source options:nil error:&error]; if (!library) { logInfo("rift: shader compile failed: %s", error.localizedDescription.UTF8String); return false; @@ -1728,17 +2154,11 @@ bool start() override { const uint32_t black = 0; [blank_ replaceRegion:MTLRegionMake2D(0, 0, 1, 1) mipmapLevel:0 withBytes:&black bytesPerRow:4]; } - if (!options_.test_scene) { - logo_ = loadLogo(); - window_capture_ = makeWindowCapture(); - } - - running_ = true; - hid_thread_ = std::thread([this] { hidThread(); }); - scene_thread_ = std::thread([this] { sceneThread(); }); return true; } +public: + void stop() override { if (!running_.exchange(false)) return; @@ -1769,15 +2189,23 @@ void stop() override { bool connected() override { return hid_open_ && fusion_ready_ && layer_ready_; } mvr_system_info systemInfo() override { + const DirectModel m = model(); mvr_system_info info{}; info.flags = hid_open_ ? MVR_SYSTEM_HMD_CONNECTED : 0; - info.eye_width = options_.eye_width; - info.eye_height = options_.eye_height; - info.refresh_hz = kRefreshHz; + info.eye_width = options_.eye_width ? options_.eye_width : m.eye_width; + info.eye_height = options_.eye_height ? options_.eye_height : m.eye_height; + info.refresh_hz = m.refresh_hz; info.ipd = ipd(); - info.fov[0] = kFov[0]; - info.fov[1] = kFov[1]; - std::snprintf(info.system_name, sizeof(info.system_name), "Oculus Rift CV1"); + info.fov[0] = m.fov[0]; + info.fov[1] = m.fov[1]; + std::snprintf(info.system_name, sizeof(info.system_name), "%s", m.name.c_str()); + info.headset = m.headset; + info.controllers = m.controllers; + if (passthrough_.load()) + info.flags |= MVR_SYSTEM_PASSTHROUGH; + for (float rate : m.refresh_rates) + if (info.refresh_rate_count < MVR_MAX_REFRESH_RATES) + info.refresh_rates[info.refresh_rate_count++] = rate; return info; } @@ -1800,17 +2228,25 @@ void tracking(int64_t target_time_ns, mvr_tracking_response &out) override { out.head_linear_velocity = head.linear_velocity; out.head_angular_velocity = head.angular_velocity; const float ipd_m = ipd(); + const DirectModel m = model(); for (int eye = 0; eye < 2; ++eye) { - out.views[eye].pose = poseMul(out.head, mvr_pose{quatIdentity(), {(eye == 0 ? -0.5f : 0.5f) * ipd_m, 0, 0}}); - out.views[eye].fov = kFov[eye]; + out.views[eye].pose = + poseMul(out.head, mvr_pose{m.eye_rotation[eye], {(eye == 0 ? -0.5f : 0.5f) * ipd_m, 0, 0}}); + out.views[eye].fov = m.fov[eye]; } // the controllers are carried along with the head's heading, as a body turns with it const mvr_pose body{quatYawOnly(out.head.orientation), out.head.position}; - touch_.fill(out.controllers, body, space, time); + if (DirectBackend *backend = backend_.load()) + direct_controllers_.fill(out.controllers, body, space, time, backend); + else + touch_.fill(out.controllers, body, space, time); } void haptic(uint32_t hand, float duration_s, float frequency_hz, float amplitude) override { - touch_.haptic(hand, duration_s, frequency_hz, amplitude); + if (DirectBackend *backend = backend_.load()) + backend->haptic(hand, duration_s, frequency_hz, amplitude); + else + touch_.haptic(hand, duration_s, frequency_hz, amplitude); } uint32_t trackingReferences(mvr_tracking_reference *out) override { @@ -1819,14 +2255,25 @@ uint32_t trackingReferences(mvr_tracking_reference *out) override { std::lock_guard lock(fusion_mutex_); space = space_; } - return sensors_.references(out, space); + DirectBackend *backend = backend_.load(); + if (!backend) + return sensors_.references(out, space); + // the base stations, from the tracker's space to the application's, as the Rift's sensors are + const uint32_t count = backend->trackingReferences(out); + for (uint32_t i = 0; i < count && i < MVR_MAX_TRACKING_REFERENCES; ++i) + if (out[i].flags & MVR_REFERENCE_POSE_VALID) { + out[i].pose.orientation = quatNormalize(quatMul(space.heading, out[i].pose.orientation)); + out[i].pose.position = space.toApp(out[i].pose.position); + } + return count; } VideoParams videoParams() override { + const mvr_system_info info = systemInfo(); VideoParams params; - params.frame_width = options_.eye_width * 2; - params.frame_height = options_.eye_height; - params.fps = kRefreshHz; + params.frame_width = info.eye_width * 2; + params.frame_height = info.eye_height; + params.fps = info.refresh_hz; return params; } @@ -1895,7 +2342,8 @@ void displayFrame(const FrameInfo &frame, CVPixelBufferRef composed) override { // `home_head` and `home_space` place the head and the controllers in the home, whose floor is the head's height // below its origin. void present(CAMetalLayer *layer, id texture, const mvr_quat *rotation, CVPixelBufferRef keep, - id window = nil, mvr_vec3 home_head = {}, const TrackingSpace &home_space = {}) { + id window = nil, mvr_vec3 home_head = {}, const TrackingSpace &home_space = {}, + id camera = nil, bool camera_stereo = false) { std::lock_guard draw(draw_mutex_); id drawable = [layer nextDrawable]; if (!drawable) @@ -1916,6 +2364,7 @@ void present(CAMetalLayer *layer, id texture, const mvr_quat *rotati [enc setRenderPipelineState:scene_pipeline_]; [enc setFragmentTexture:(logo_ ?: blank_) atIndex:0]; [enc setFragmentTexture:(window ?: blank_) atIndex:1]; + [enc setFragmentTexture:(camera ?: blank_) atIndex:2]; [enc setFragmentSamplerState:sampler_ atIndex:0]; } // the rotation's columns: for the frame, the head now to the head it was rendered for; for the home, head to world @@ -1941,10 +2390,18 @@ void present(CAMetalLayer *layer, id texture, const mvr_quat *rotati home.logo[1] = static_cast(logo_.height) / logo_.width; } home.test_scene = options_.test_scene ? 1 : 0; + if (camera) { + home.passthrough[0] = 1; + home.passthrough[1] = kPassthroughHalfFovTan; + home.passthrough[2] = camera_stereo ? 1 : 0; + } // the controllers, where the server places them, seen from the head mvr_controller_state hands[2]; const mvr_pose body{quatYawOnly(q), home_head}; - touch_.fill(hands, body, home_space, nowNs()); + if (DirectBackend *backend = backend_.load()) + direct_controllers_.fill(hands, body, home_space, nowNs(), backend); + else + touch_.fill(hands, body, home_space, nowNs()); for (int hand = 0; hand < 2; ++hand) { if (!(hands[hand].flags & MVR_CONTROLLER_POSE_VALID)) continue; @@ -1969,11 +2426,14 @@ void present(CAMetalLayer *layer, id texture, const mvr_quat *rotati } } [enc setFragmentBytes:&home length:sizeof(home) atIndex:1]; - const float half_w = drawable.texture.width / 2.0f, h = drawable.texture.height; + const DirectModel m = model(); for (uint32_t eye = 0; eye < 2; ++eye) { - DistortionUniforms u = uniforms(eye, half_w, h); - [enc setViewport:MTLViewport{eye * half_w, 0, half_w, h, 0, 1}]; + const MTLViewport viewport = eyeViewport(m, eye, drawable.texture.width, drawable.texture.height); + DistortionUniforms u = uniforms(m, eye, viewport); + LensUniforms lens = lensUniforms(m, eye); + [enc setViewport:viewport]; [enc setFragmentBytes:&u length:sizeof(u) atIndex:0]; + [enc setFragmentBytes:&lens length:sizeof(lens) atIndex:2]; [enc drawPrimitives:MTLPrimitiveTypeTriangle vertexStart:0 vertexCount:3]; } [enc endEncoding]; @@ -2014,12 +2474,16 @@ void present(CAMetalLayer *layer, id texture, const mvr_quat *rotati [ce setRenderPipelineState:scene_pipeline_]; [ce setFragmentTexture:(logo_ ?: blank_) atIndex:0]; [ce setFragmentTexture:(window ?: blank_) atIndex:1]; + [ce setFragmentTexture:(camera ?: blank_) atIndex:2]; [ce setFragmentSamplerState:sampler_ atIndex:0]; [ce setFragmentBytes:&home length:sizeof(home) atIndex:1]; for (uint32_t eye = 0; eye < 2; ++eye) { - DistortionUniforms u = uniforms(eye, half_w, h); - [ce setViewport:MTLViewport{eye * half_w, 0, half_w, h, 0, 1}]; + const MTLViewport viewport = eyeViewport(m, eye, drawable.texture.width, drawable.texture.height); + DistortionUniforms u = uniforms(m, eye, viewport); + LensUniforms lens = lensUniforms(m, eye); + [ce setViewport:viewport]; [ce setFragmentBytes:&u length:sizeof(u) atIndex:0]; + [ce setFragmentBytes:&lens length:sizeof(lens) atIndex:2]; [ce drawPrimitives:MTLPrimitiveTypeTriangle vertexStart:0 vertexCount:3]; } [ce endEncoding]; @@ -2086,7 +2550,7 @@ void sceneThread() { TrackingSpace home_space; { std::lock_guard lock(fusion_mutex_); - head = headLocked(nowNs() + static_cast(2e9 / kRefreshHz)); + head = headLocked(nowNs() + static_cast(2e9 / refresh_hz_.load())); home_space = space_; } const mvr_quat q = head.orientation; @@ -2139,7 +2603,13 @@ void sceneThread() { // the home, with the window of the game that is starting CVPixelBufferRef captured = window_capture_ ? window_capture_->latest() : nullptr; id window = captured ? textureFor(captured) : nil; - present(layer, nil, &q, captured, window, home_head, home_space); + // the room through the headset's cameras, with --passthrough + CameraPassthrough *cameras = passthrough_.load(); + CVPixelBufferRef room = cameras ? cameras->latest() : nullptr; + id camera = room ? textureFor(room) : nil; + present(layer, nil, &q, captured, window, home_head, home_space, camera, cameras && cameras->stereo()); + if (room) + CVPixelBufferRelease(room); if (captured) CVPixelBufferRelease(captured); } @@ -2160,13 +2630,14 @@ int pacing() { if (off || frame_interval_ns_ <= 0) return 1; const double rate = 1e9 / frame_interval_ns_; + const double refresh = refresh_hz_.load(); int target = 4; for (int n = 1; n <= 3; ++n) - if (rate >= kRefreshHz / n * 0.92) { + if (rate >= refresh / n * 0.92) { target = n; break; } - if (target < pace_ && rate < kRefreshHz / target * 0.97) + if (target < pace_ && rate < refresh / target * 0.97) target = pace_; pace_ = target; return pace_; @@ -2233,7 +2704,130 @@ float ipd() const { if (options_.ipd > 0.0f) return options_.ipd; const float measured = ipd_m_.load(); - return measured > 0.05f && measured < 0.08f ? measured : panel_.lens_separation_m; + if (measured > 0.05f && measured < 0.08f) + return measured; + return backend_.load() ? model().ipd : panel_.lens_separation_m; + } + + DirectModel model() const { + std::lock_guard lock(model_mutex_); + return model_; + } + + void setModel(const DirectModel &m) { + { + std::lock_guard lock(model_mutex_); + model_ = m; + } + refresh_hz_ = m.refresh_hz > 1 ? m.refresh_hz : 90.0f; + ++generation_; + } + + // --- what a backend reports (DirectHost) --- + + void directModel(const DirectModel &m) override { + setModel(m); + if (options_.passthrough && !passthrough_owner_) { + passthrough_owner_ = startCameraPassthrough(m.headset); + passthrough_ = passthrough_owner_.get(); + } + logInfo("%s: %s, %ux%u per eye at %.0f Hz", name(), m.name.c_str(), m.eye_width, m.eye_height, m.refresh_hz); + } + + void directOn(bool on) override { + if (hid_open_.exchange(on) == on) + return; + if (!on) + direct_controllers_.clear(); + ++generation_; + // the scripts wait for these lines (scripts/metalsharp/metalvr-rift) + logInfo(on ? "rift: headset on" : "rift: headset off"); + } + + void directImu(int64_t time_ns, const float accel[3], const float gyro[3]) override { + std::lock_guard lock(fusion_mutex_); + fusion_.update(time_ns, accel, gyro); + settleLocked(time_ns, pending_put_on_.exchange(false)); + ++imu_reports_; + } + + void directWorn(bool worn) override { + if (worn && !worn_) + pending_put_on_ = true; + worn_ = worn; + } + + void directIpd(float meters) override { ipd_m_.store(meters); } + + void directControllerImu(int hand, int64_t time_ns, const float accel[3], const float gyro[3]) override { + direct_controllers_.imu(hand, time_ns, accel, gyro); + } + + void directControllerInput(int hand, const DirectControllerInput &input, int64_t time_ns) override { + direct_controllers_.input(hand, input, time_ns); + } + + void directControllerGrip(int hand, const mvr_pose &grip, const mvr_pose &aim) override { + direct_controllers_.grip(hand, grip, aim); + } + + // where an eye is on the panel: the left and right halves, or for a portrait panel (Rift S) the top and bottom + static MTLViewport eyeViewport(const DirectModel &m, uint32_t eye, double width, double height) { + if (m.portrait_eyes) + return MTLViewport{0, eye * height / 2.0, width, height / 2.0, 0, 1}; + return MTLViewport{eye * width / 2.0, 0, width / 2.0, height, 0, 1}; + } + + static LensUniforms lensUniforms(const DirectModel &m, uint32_t eye) { + const DirectLens &l = m.lens[eye]; + LensUniforms u{}; + u.model = l.model; + u.portrait = m.portrait_eyes ? 1 : 0; + u.flip_y = l.flip_y ? 1 : 0; + const mvr_quat &q = m.eye_rotation[eye]; + const float rotation[4] = {q.x, q.y, q.z, q.w}; + std::memcpy(u.eye_rotation, rotation, sizeof(rotation)); + std::memcpy(u.pano_k, l.pano_k, sizeof(u.pano_k)); + u.pano_k4_ab[0] = l.pano_k[4]; + std::memcpy(u.pano_k4_ab + 1, l.pano_aberration, sizeof(l.pano_aberration)); + u.pano[0] = l.pano_scale; + u.pano[1] = l.pano_center[0]; + u.pano[2] = l.pano_center[1]; + u.pano_viewport[0] = l.pano_viewport[0]; + u.pano_viewport[1] = l.pano_viewport[1]; + u.vive[0] = l.vive_aspect; + u.vive[1] = l.vive_grow; + for (int c = 0; c < 3; ++c) { + u.vive_center[c][0] = l.vive_center[c][0]; + u.vive_center[c][1] = l.vive_center[c][1]; + std::memcpy(u.vive_coeff[c], l.vive_coefficients[c], sizeof(u.vive_coeff[c])); + } + std::memcpy(u.psvr2, l.psvr2, sizeof(u.psvr2)); + // Monado's centre of each eye's image and its scale (psvr2_distortion.c) + u.psvr2_center[0] = eye == 0 ? 0.6603788f : 0.3396213f; + u.psvr2_center[1] = 0.5f; + u.psvr2_center[2] = 0.3585564f; + u.psvr2_center[3] = 0.3762281f; + return u; + } + + DistortionUniforms uniforms(const DirectModel &m, uint32_t eye, const MTLViewport &viewport) const { + if (m.lens[eye].model != DirectLens::CatmullRomCV1) { + // the other lens models need the eye's field of view and where it is on the panel (Lens in the shader) + DistortionUniforms u{}; + const mvr_fov &fov = m.fov[eye]; + u.fov_tan[0] = std::tan(-fov.left); + u.fov_tan[1] = std::tan(fov.right); + u.fov_tan[2] = std::tan(fov.up); + u.fov_tan[3] = std::tan(-fov.down); + u.eye = eye; + u.viewport_origin[0] = static_cast(viewport.originX); + u.viewport_origin[1] = static_cast(viewport.originY); + u.viewport_size[0] = static_cast(viewport.width); + u.viewport_size[1] = static_cast(viewport.height); + return u; + } + return uniforms(eye, static_cast(viewport.width), static_cast(viewport.height)); } DistortionUniforms uniforms(uint32_t eye, float viewport_w, float viewport_h) const { @@ -2276,7 +2870,7 @@ Head headLocked(int64_t time_ns) { const mvr_quat own = fusion_.rotation(); mvr_vec3 raw_position = space_.origin, raw_velocity{}; SensorPose tracked; - if (sensors_.pose(RiftSensors::kHmd, tracked)) { + if (headTracked(tracked)) { // The head's rotation stays this driver's own, from the IMU, which is smooth; the tracker's takes in every // camera fix and shakes as the head turns. Its heading is turned to the tracker's, and pulled along slowly // while a sensor sees the headset, which takes out the IMU's drift. @@ -2320,11 +2914,26 @@ Head headLocked(int64_t time_ns) { return head; } + // where a position tracker has the head: the Rift's sensors, or a backend's (the base stations) + bool headTracked(SensorPose &out) { + if (DirectBackend *backend = backend_.load()) { + TrackedPose p; + if (!backend->trackedPose(LighthouseTracking::kHead, p)) + return false; + out.orientation = p.orientation; + out.position = p.position; + out.velocity = p.velocity; + out.position_tracked = p.position_tracked; + return true; + } + return sensors_.pose(RiftSensors::kHmd, out); + } + // Straight ahead becomes where the headset faces, and where it is becomes the seated head's place; // fusion_mutex_ is held. void recenterLocked(int64_t now_ns) { SensorPose tracked; - if (head_on_sensors_ && sensors_.pose(RiftSensors::kHmd, tracked)) { + if (head_on_sensors_ && headTracked(tracked)) { space_.heading = quatConj(quatYawOnly(quatMul(yaw_align_, fusion_.rotation()))); if (head_seen_) space_.origin = tracked.position; @@ -2333,6 +2942,7 @@ void recenterLocked(int64_t now_ns) { } ++generation_; touch_.recenter(); + direct_controllers_.recenter(); } // The headset's infrared LEDs, and where its IMU sits, from its position reports: one LED (or the IMU) a read, @@ -2511,19 +3121,99 @@ void input(uint32_t id, const uint8_t *report, CFIndex length) { unpackSample(report + 11 + i * 16 + 8, 0.0001f, gyro); fusion_.update(received - (samples - 1 - i) * kSampleNs, accel, gyro); } + settleLocked(received, put_on); + } + + // After new samples of the head: what the headset faces as it starts, and again each time it is put on, is + // straight ahead. fusion_mutex_ is held. + void settleLocked(int64_t now_ns, bool put_on) { if ((!fusion_ready_ || put_on) && fusion_.ready()) { - // what the headset faces as it starts, and again each time it is put on, is straight ahead - recenterLocked(received); + recenterLocked(now_ns); if (!fusion_ready_) - logInfo("rift: tracking head rotation"); + logInfo("%s: tracking head rotation", name()); else - logInfo("rift: the headset was put on; straight ahead is where it faces"); + logInfo("%s: the headset was put on; straight ahead is where it faces", name()); fusion_ready_ = true; } } + // Which headset of the family is on USB, then that headset's own loop until the driver stops. A Lighthouse + // headset and a Rift S have a backend of their own (direct_backend.h); the CV1 is driven here. void hidThread() { hid_loop_ = CFRunLoopGetCurrent(); + if (options_.psvr2) { + runBackend(makePsvr2Backend()); + return; + } + if (options_.lighthouse) { + LighthouseOptions lighthouse; + lighthouse.pro2_mode = options_.pro2_mode; + runBackend(makeLighthouseBackend(lighthouse)); + return; + } + bool told = false; + while (running_) { + // a Rift S (2833:0051) or a CV1 (2833:0031); nothing else of Oculus's is a headset here + IOHIDManagerRef manager = IOHIDManagerCreate(kCFAllocatorDefault, kIOHIDOptionsTypeNone); + IOHIDManagerSetDeviceMatching(manager, (__bridge CFDictionaryRef) @{@kIOHIDVendorIDKey : @(kVendor)}); + IOHIDManagerOpen(manager, kIOHIDOptionsTypeNone); + bool cv1 = false, rift_s = false; + if (CFSetRef set = IOHIDManagerCopyDevices(manager)) { + std::vector values(CFSetGetCount(set)); + CFSetGetValues(set, values.data()); + for (const void *v : values) { + const int product = intProperty(static_cast(const_cast(v)), CFSTR(kIOHIDProductIDKey)); + cv1 = cv1 || product == kProductCV1; + rift_s = rift_s || product == rift_s::kProduct; + } + CFRelease(set); + } + IOHIDManagerClose(manager, kIOHIDOptionsTypeNone); + CFRelease(manager); + if (cv1) { + setModel(cv1Model()); + cv1Thread(); + return; + } + if (rift_s) { + runBackend(makeRiftSBackend()); + return; + } + if (!told) { + told = true; + logInfo("rift: waiting for a Rift CV1 (2833:0031) or a Rift S (2833:0051) on USB"); + } + CFRunLoopRunInMode(kCFRunLoopDefaultMode, 1.0, false); + } + } + + // A headset with a backend: the backend's loop on this thread, while this thread also watches its display. + void runBackend(std::unique_ptr backend) { + DirectBackend *b = backend.get(); + backend_owner_ = std::move(backend); + backend_ = b; + std::thread watcher([this] { + while (running_) { + watchDisplay(); + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + } + }); + b->run(*this, running_); + watcher.join(); + hid_open_ = false; + ++generation_; + } + + static int intProperty(IOHIDDeviceRef device, CFStringRef key) { + int value = 0; + CFTypeRef p = IOHIDDeviceGetProperty(device, key); + if (p && CFGetTypeID(p) == CFNumberGetTypeID()) + CFNumberGetValue(static_cast(p), kCFNumberIntType, &value); + return value; + } + + // the Rift CV1: its HID interfaces, sensors and radio, driven by this driver itself + void cv1Thread() { IOHIDManagerRef manager = IOHIDManagerCreate(kCFAllocatorDefault, kIOHIDOptionsTypeNone); NSDictionary *match = @{@kIOHIDVendorIDKey : @(kVendor), @kIOHIDProductIDKey : @(kProductCV1)}; IOHIDManagerSetDeviceMatching(manager, (__bridge CFDictionaryRef)match); @@ -2710,13 +3400,31 @@ static int interfaceNumber(IOHIDDeviceRef dev) { // --- the headset's display --- - static CGDirectDisplayID findDisplay() { + // the headset's display, by the vendor and product its EDID gives: the model's, once it is identified + CGDirectDisplayID findDisplay() { + const DirectModel m = model(); + if (!m.display_vendor) + return kCGNullDirectDisplay; CGDirectDisplayID ids[16]; uint32_t count = 0; CGGetOnlineDisplayList(16, ids, &count); - for (uint32_t i = 0; i < count; ++i) - if (CGDisplayVendorNumber(ids[i]) == kDisplayVendor && CGDisplayModelNumber(ids[i]) == kDisplayProductCV1) + for (uint32_t i = 0; i < count; ++i) { + if (CGDisplayVendorNumber(ids[i]) != m.display_vendor) + continue; + // no product codes: any display of the maker's (the PlayStation VR2's own code is not known yet) + if (m.display_products.empty()) { + static uint32_t told = 0; + if (told != CGDisplayModelNumber(ids[i])) { + told = CGDisplayModelNumber(ids[i]); + logInfo("%s: taking display %04x:%04x (%zux%zu) as the headset's", name(), m.display_vendor, told, + CGDisplayPixelsWide(ids[i]), CGDisplayPixelsHigh(ids[i])); + } return ids[i]; + } + for (uint32_t product : m.display_products) + if (CGDisplayModelNumber(ids[i]) == product) + return ids[i]; + } return kCGNullDirectDisplay; } @@ -2789,6 +3497,7 @@ void openWindow(CGDirectDisplayID display) { display_ = kCGNullDirectDisplay; // look again return; } + useDisplayMode(display); NSWindow *window = [[NSWindow alloc] initWithContentRect:screen.frame styleMask:NSWindowStyleMaskBorderless backing:NSBackingStoreBuffered @@ -2827,6 +3536,46 @@ void openWindow(CGDirectDisplayID display) { logInfo("rift: drawing on the headset's display, %.0fx%.0f", layer.drawableSize.width, layer.drawableSize.height); } + // The refresh rate asked for (--refresh), if the display has a mode of its size at it; then the rate the display + // really runs at becomes the headset's, which frames are paced to. The CV1 stays at its 90 Hz. + void useDisplayMode(CGDirectDisplayID display) { + if (!backend_.load()) + return; + if (options_.refresh_hz > 0) { + CGDisplayModeRef current = CGDisplayCopyDisplayMode(display); + CFArrayRef modes = CGDisplayCopyAllDisplayModes(display, nullptr); + CGDisplayModeRef chosen = nullptr; + for (CFIndex i = 0; modes && current && i < CFArrayGetCount(modes); ++i) { + CGDisplayModeRef mode = static_cast(const_cast(CFArrayGetValueAtIndex(modes, i))); + if (CGDisplayModeGetPixelWidth(mode) == CGDisplayModeGetPixelWidth(current) && + CGDisplayModeGetPixelHeight(mode) == CGDisplayModeGetPixelHeight(current) && + std::fabs(CGDisplayModeGetRefreshRate(mode) - options_.refresh_hz) < 1.0) + chosen = mode; + } + if (!chosen) + logInfo("rift: the headset's display has no %.0f Hz mode at its size; it stays as it is", options_.refresh_hz); + else if (CGDisplaySetDisplayMode(display, chosen, nullptr) != kCGErrorSuccess) + logInfo("rift: macOS did not switch the headset's display to %.0f Hz", options_.refresh_hz); + if (modes) + CFRelease(modes); + if (current) + CGDisplayModeRelease(current); + } + CGDisplayModeRef mode = CGDisplayCopyDisplayMode(display); + const double hz = mode ? CGDisplayModeGetRefreshRate(mode) : 0; + if (mode) + CGDisplayModeRelease(mode); + DirectModel m = model(); + if (hz > 30 && std::fabs(hz - m.refresh_hz) > 0.5) { + logInfo("rift: the headset's display runs at %.2f Hz", hz); + m.refresh_hz = static_cast(hz); + if (std::none_of(m.refresh_rates.begin(), m.refresh_rates.end(), + [&](float r) { return std::fabs(r - hz) < 0.5; })) + m.refresh_rates.push_back(m.refresh_hz); + setModel(m); + } + } + void closeWindow() { layer_ready_ = false; { @@ -2851,6 +3600,16 @@ void removeScreenObserver() { } RiftOptions options_; + // the headset drawn for: the CV1's, a backend's (direct_backend.h), or nothing assumed until one is identified + mutable std::mutex model_mutex_; + DirectModel model_; + std::atomic refresh_hz_{90.0f}; + std::unique_ptr backend_owner_; // the USB thread's, set once + std::atomic backend_{nullptr}; + DirectControllers direct_controllers_; + std::unique_ptr passthrough_owner_; // the USB thread's, set once + std::atomic passthrough_{nullptr}; + std::atomic pending_put_on_{false}; Panel panel_; id device_ = nil; id queue_ = nil; @@ -2931,4 +3690,14 @@ void removeScreenObserver() { std::unique_ptr makeRiftDriver(const RiftOptions &options) { return std::make_unique(options); } +bool drawTestPanel(const DirectModel &model, uint32_t width, uint32_t height, std::vector &bgra, + bool passthrough) { + RiftOptions options; + options.test_scene = true; + RiftDriver driver(options); + return driver.drawTestPanel(model, width, height, bgra, passthrough); +} + +DirectModel riftCv1Model() { return cv1Model(); } + } // namespace metalvr diff --git a/daemon/driver_sim.cpp b/daemon/driver_sim.cpp index ace12ad..abb26f6 100644 --- a/daemon/driver_sim.cpp +++ b/daemon/driver_sim.cpp @@ -9,6 +9,7 @@ #include "common.h" #include "driver.h" +#include "headsets.h" #include "metalvr_math.h" namespace metalvr { @@ -53,14 +54,33 @@ class SimDriver final : public Driver { mvr_system_info systemInfo() override { mvr_system_info info{}; - info.flags = MVR_SYSTEM_HMD_CONNECTED | MVR_SYSTEM_SIMULATED; + info.flags = MVR_SYSTEM_HMD_CONNECTED | MVR_SYSTEM_SIMULATED | MVR_SYSTEM_HAND_TRACKING; info.eye_width = options_.eye_width; info.eye_height = options_.eye_height; info.refresh_hz = options_.refresh_hz; info.ipd = kIpd; info.fov[0] = {-0.942f, 0.698f, 0.768f, -0.960f}; info.fov[1] = {-0.698f, 0.942f, 0.768f, -0.960f}; - std::snprintf(info.system_name, sizeof(info.system_name), "MetalVR Simulated Headset"); + const mvr_headset_description *description = mvr_describe_headset(options_.headset); + info.headset = options_.headset; + info.controllers = description->controllers; + if (options_.headset == MVR_HEADSET_SIMULATED) + std::snprintf(info.system_name, sizeof(info.system_name), "MetalVR Simulated Headset"); + else + std::snprintf(info.system_name, sizeof(info.system_name), "%s (simulated)", description->name); + const HeadsetSpec &spec = headsetSpec(options_.headset); + info.refresh_rate_count = 0; + bool listed = false; + for (uint32_t i = 0; i < spec.refresh_rate_count; ++i) { + info.refresh_rates[info.refresh_rate_count++] = spec.refresh_rates[i]; + listed = listed || std::fabs(spec.refresh_rates[i] - options_.refresh_hz) < 0.5f; + } + if (!listed) { + info.refresh_rate_count = 1; + info.refresh_rates[0] = options_.refresh_hz; + } + info.play_area_width = 2.0f; + info.play_area_depth = 1.5f; return info; } @@ -102,6 +122,8 @@ class SimDriver final : public Driver { const mvr_pose body{quatYawOnly(out.head.orientation), out.head.position}; c.grip = poseMul(body, in_head); c.aim = c.grip; + c.flags |= MVR_CONTROLLER_BATTERY_VALID; + c.battery = hand == 0 ? 0.8f : 0.6f; if (options_.animate) { // the right trigger ramps over four seconds; the left primary button is down every other second if (hand == 1) { @@ -116,10 +138,38 @@ class SimDriver final : public Driver { c.buttons |= MVR_BUTTON_PRIMARY; c.touches |= MVR_BUTTON_PRIMARY; } + // what the controllers of the headset stood in for have besides + const uint32_t kind = mvr_describe_headset(options_.headset)->controllers; + if (kind == MVR_CONTROLLERS_VIVE_WAND) { + // no stick: the thumb's circle is on the trackpad + c.trackpad_x = c.thumbstick_x; + c.trackpad_y = c.thumbstick_y; + c.thumbstick_x = c.thumbstick_y = 0; + if (c.trackpad_x != 0 || c.trackpad_y != 0) + c.touches |= MVR_BUTTON_TRACKPAD; + } else if (kind == MVR_CONTROLLERS_INDEX) { + c.flags |= MVR_CONTROLLER_FINGERS_VALID; + for (int finger = 0; finger < 4; ++finger) + c.fingers[finger] = 0.5f + 0.5f * std::sin(t + finger * 0.5f); + c.squeeze = c.fingers[MVR_FINGER_MIDDLE]; + c.trackpad_x = 0.5f * std::cos(t); + c.trackpad_y = 0.5f * std::sin(t); + c.touches |= MVR_BUTTON_TRACKPAD; + } else if (kind == MVR_CONTROLLERS_STEAM_FRAME && static_cast(t) % 3 == 2) { + c.buttons |= hand == 0 ? MVR_BUTTON_DPAD_UP : MVR_BUTTON_BUMPER; + } } } } + // the hands where the controllers are, open + void hands(int64_t target_time_ns, mvr_hand out[2]) override { + mvr_tracking_response tracking{}; + this->tracking(target_time_ns, tracking); + for (int hand = 0; hand < 2; ++hand) + openHand(tracking.controllers[hand].grip, hand == 0, out[hand]); + } + void haptic(uint32_t hand, float duration_s, float frequency_hz, float amplitude) override { ++haptics_; logDebug("sim: haptic hand %u %.3fs %.0fHz amplitude %.2f", hand, duration_s, frequency_hz, amplitude); @@ -130,6 +180,12 @@ class SimDriver final : public Driver { params.codec = options_.codec; params.frame_width = options_.eye_width * 2; params.frame_height = options_.eye_height; + if (options_.foveated_encoding) { + // ALVR's default foveated_encoding settings + params.foveation = computeFoveation(options_.eye_width, options_.eye_height, 0.45f, 0.4f, 0.4f, 0.1f, 4.0f, 5.0f); + params.frame_width = params.foveation.optimized_eye_width * 2; + params.frame_height = params.foveation.optimized_eye_height; + } params.fps = options_.refresh_hz; params.bitrate_bps = options_.bitrate_bps; return params; diff --git a/daemon/foveation.cpp b/daemon/foveation.cpp new file mode 100644 index 0000000..7f53a5b --- /dev/null +++ b/daemon/foveation.cpp @@ -0,0 +1,99 @@ +#include "foveation.h" + +#include + +namespace metalvr { + +Foveation computeFoveation(uint32_t eye_width, uint32_t eye_height, float center_size_x, float center_size_y, + float center_shift_x, float center_shift_y, float edge_ratio_x, float edge_ratio_y) { + Foveation f; + if (!eye_width || !eye_height || !(edge_ratio_x >= 1.0f) || !(edge_ratio_y >= 1.0f)) return f; + const double size[2] = {static_cast(eye_width), static_cast(eye_height)}; + const double center[2] = {center_size_x, center_size_y}; + const double shift[2] = {center_shift_x, center_shift_y}; + const double edge[2] = {edge_ratio_x, edge_ratio_y}; + uint32_t optimized[2]; + for (int a = 0; a < 2; a++) { + const double edge_size = size[a] - center[a] * size[a]; + const double center_aligned = 1.0 - std::ceil(edge_size / (edge[a] * 2.0)) * (edge[a] * 2.0) / size[a]; + const double edge_aligned = size[a] - center_aligned * size[a]; + const double shift_aligned = + edge_aligned > 0 ? std::ceil(shift[a] * edge_aligned / (edge[a] * 2.0)) * (edge[a] * 2.0) / edge_aligned : 0; + const double scale = center_aligned + (1.0 - center_aligned) / edge[a]; + const double squeezed = scale * size[a]; + optimized[a] = static_cast(std::ceil(squeezed / 32.0) * 32.0); + f.eye_size_ratio[a] = static_cast(squeezed / optimized[a]); + f.center_size[a] = static_cast(center_aligned); + f.center_shift[a] = static_cast(shift_aligned); + f.edge_ratio[a] = static_cast(edge[a]); + } + f.enabled = true; + f.eye_width = eye_width; + f.eye_height = eye_height; + f.optimized_eye_width = optimized[0]; + f.optimized_eye_height = optimized[1]; + return f; +} + +void foveationSource(const Foveation &f, const float frame_uv[2], float out[2]) { + // CompressAxisAlignedPixelShader.hlsl: the right eye is mirrored, so both are squeezed alike from the nose out + const bool right = frame_uv[0] > 0.5f; + float eye[2] = {(frame_uv[0] + (right ? 1.0f - 2.0f * frame_uv[0] : 0.0f)) * 2.0f, frame_uv[1]}; + float source[2]; + for (int a = 0; a < 2; a++) { + const float uv = eye[a] / f.eye_size_ratio[a]; + const float size = f.center_size[a], shift = f.center_shift[a], edge = f.edge_ratio[a]; + const float c0 = (1.0f - size) / 2.0f; + const float c1 = (edge - 1.0f) * c0 * (shift + 1.0f) / edge; + const float c2 = (edge - 1.0f) * size + 1.0f; + const float lo = c0 * (shift + 1.0f) / c2; + const float hi = c0 * (shift - 1.0f) / c2 + 1.0f; + const float center = uv * c2 / edge + c1; + const float d2 = uv * c2; + const float d3 = (uv - 1.0f) * c2 + 1.0f; + const float g1 = uv / lo; + const float g2 = (1.0f - uv) / (1.0f - hi); + if (uv < lo) + source[a] = g1 * center + (1.0f - g1) * d2; + else if (uv > hi) + source[a] = g2 * center + (1.0f - g2) * d3; + else + source[a] = center; + } + out[0] = source[0] * 0.5f + (right ? 1.0f - source[0] : 0.0f); + out[1] = source[1]; +} + +void foveationClientSample(const Foveation &f, int eye, const float eye_uv[2], float out[2]) { + // stream.rs's constants and stream.wgsl's fragment_main + float uv[2] = {eye == 1 ? 1.0f - eye_uv[0] : eye_uv[0], eye_uv[1]}; + for (int a = 0; a < 2; a++) { + const double size = f.center_size[a], shift = f.center_shift[a], edge = f.edge_ratio[a]; + const double c0 = (1.0 - size) * 0.5; + const double c1 = (edge - 1.0) * c0 * (shift + 1.0) / edge; + const double c2 = (edge - 1.0) * size + 1.0; + const double lo_bound = c0 * (shift + 1.0); + const double hi_bound = c0 * (shift - 1.0) + 1.0; + const double lo_c = c0 * (shift + 1.0) / c2; + const double hi_c = c0 * (shift - 1.0) / c2 + 1.0; + const double a_left = c2 * (1.0 - edge) / (edge * lo_c); + const double b_left = (c1 + c2 * lo_c) / lo_c; + const double a_right = c2 * (edge - 1.0) / (edge * (1.0 - hi_c)); + const double b_right = + (c2 - edge * c1 - 2.0 * edge * c2 + c2 * edge * (1.0 - hi_c) + edge) / (edge * (1.0 - hi_c)); + const double c_right = (c2 * edge - c2) * (c1 - hi_c + c2 * hi_c) / (edge * (1.0 - hi_c) * (1.0 - hi_c)); + const double v = uv[a]; + double r; + if (v < lo_bound) + r = (-b_left + std::sqrt(b_left * b_left + 4.0 * a_left * v)) / (2.0 * a_left); + else if (v > hi_bound) + r = (-b_right + std::sqrt(b_right * b_right - 4.0 * (c_right - a_right * v))) / (2.0 * a_right); + else + r = (v - c1) * edge / c2; + uv[a] = static_cast(r * f.eye_size_ratio[a]); + } + out[0] = eye == 1 ? 1.0f - uv[0] : uv[0]; + out[1] = uv[1]; +} + +} // namespace metalvr diff --git a/daemon/foveation.h b/daemon/foveation.h new file mode 100644 index 0000000..57f37bd --- /dev/null +++ b/daemon/foveation.h @@ -0,0 +1,33 @@ +// ALVR's foveated encoding (docs/HEADSETS.md, phase E): each eye of the composed frame is squeezed toward its edges +// before it is encoded, the center kept whole, and the headset's ALVR client stretches it back. The server's half +// is ALVR's server_openvr FFR.cpp and CompressAxisAlignedPixelShader.hlsl; the client's, which this must match, is +// graphics/src/stream.rs (foveated_encoding_shader_constants) and graphics/resources/stream.wgsl (MIT, alvr-org). +#pragma once + +#include + +namespace metalvr { + +struct Foveation { + bool enabled = false; + uint32_t eye_width = 0, eye_height = 0; // each eye as composed (the stream's view size) + uint32_t optimized_eye_width = 0, optimized_eye_height = 0; // each eye as encoded, multiples of 32 + float eye_size_ratio[2] = {1, 1}; // the squeezed eye's share of its aligned size + float center_size[2] = {}; // aligned, as ALVR's client aligns them + float center_shift[2] = {}; + float edge_ratio[2] = {1, 1}; +}; + +// ALVR's CalculateFoveationVars, from the eye's size and ALVR's foveated_encoding settings +Foveation computeFoveation(uint32_t eye_width, uint32_t eye_height, float center_size_x, float center_size_y, + float center_shift_x, float center_shift_y, float edge_ratio_x, float edge_ratio_y); + +// The server's warp: for a point of the encoded frame (uv 0..1 over the whole side-by-side frame), the point of +// the composed frame it shows (uv over the whole composed frame). The compositor's shader does the same. +void foveationSource(const Foveation &f, const float frame_uv[2], float out[2]); + +// The client's unwarp, for checking: for a point of an eye as the headset shows it (uv 0..1 over that eye), the +// point of the encoded eye it is read from (uv over that eye of the encoded frame). +void foveationClientSample(const Foveation &f, int eye, const float eye_uv[2], float out[2]); + +} // namespace metalvr diff --git a/daemon/headsets.cpp b/daemon/headsets.cpp new file mode 100644 index 0000000..7673168 --- /dev/null +++ b/daemon/headsets.cpp @@ -0,0 +1,193 @@ +#include "headsets.h" + +#include +#include +#include +#include + +namespace metalvr { +namespace { + +constexpr float kPi = 3.14159265358979323846f; +constexpr float rad(float degrees) { return degrees * kPi / 180.0f; } + +// a left eye's field of view and its mirror for the right +constexpr HeadsetSpec withFov(HeadsetSpec spec, float left, float right, float up, float down) { + spec.fov[0] = {rad(left), rad(right), rad(up), rad(down)}; + spec.fov[1] = {rad(-right), rad(-left), rad(up), rad(down)}; + return spec; +} + +constexpr uint64_t k4k60 = 3840ull * 2160ull * 60ull; // Snapdragon 835 +constexpr uint64_t k4k120 = 3840ull * 2160ull * 120ull; // Snapdragon XR2 Gen 1 and Gen 2 + +// Sources and what is a guess: docs/HEADSETS.md. Streamed headsets: the size is what ALVR negotiates for the +// headset's default view by default (2144 wide, the height from the headset's aspect, a multiple of 32), so that +// what OpenXR is told before a headset connects is what it will be told after. Headsets on the Mac's own display: +// about 1.2 times the panel, as the CV1's 1344x1600 is for its 1080x1200. +HeadsetSpec makeSpec(uint32_t headset) { + HeadsetSpec s; + s.headset = headset; + const auto rates = [&](std::initializer_list list, float preferred) { + s.refresh_rate_count = 0; + for (float rate : list) + if (s.refresh_rate_count < MVR_MAX_REFRESH_RATES) s.refresh_rates[s.refresh_rate_count++] = rate; + s.default_refresh = preferred; + }; + switch (headset) { + case MVR_HEADSET_SIMULATED: + s.eye_width = 1440, s.eye_height = 1584, rates({72}, 72); + return withFov(s, -54.0f, 40.0f, 44.0f, -55.0f); + case MVR_HEADSET_RIFT_CV1: + s.eye_width = 1344, s.eye_height = 1600, rates({90}, 90), s.ipd = 0.0635f; + return withFov(s, -46.6f, 40.0f, 43.4f, -46.6f); // guess: the CV1 driver computes its own from the lens + case MVR_HEADSET_RIFT_S: + // Oculus's own default render size for the Rift S + s.eye_width = 1648, s.eye_height = 1776, rates({80}, 80), s.ipd = 0.0635f; + return withFov(s, -48.0f, 42.0f, 45.0f, -48.0f); // guess; open drivers assume about 105 degrees across + case MVR_HEADSET_QUEST_1: + s.eye_width = 2144, s.eye_height = 2368, rates({72}, 72), s.ipd = 0.0635f; + s.max_decode_pixels_per_second = k4k60, s.stream_refresh = 72; + return withFov(s, -52.0f, 42.0f, 47.0f, -53.0f); + case MVR_HEADSET_QUEST_2: + s.eye_width = 2144, s.eye_height = 2336, rates({72, 80, 90, 120}, 72); + s.max_decode_pixels_per_second = k4k120, s.stream_refresh = 90; + return withFov(s, -52.0f, 45.0f, 48.0f, -51.0f); + case MVR_HEADSET_QUEST_3: + s.eye_width = 2144, s.eye_height = 2240, rates({72, 80, 90, 120}, 72); + s.max_decode_pixels_per_second = k4k120, s.av1_decode = true, s.stream_refresh = 90; + return withFov(s, -55.0f, 55.0f, 48.0f, -48.0f); + case MVR_HEADSET_QUEST_3S: + s.eye_width = 2144, s.eye_height = 2240, rates({72, 80, 90, 120}, 72); + s.max_decode_pixels_per_second = k4k120, s.av1_decode = true, s.stream_refresh = 90; + return withFov(s, -50.0f, 46.0f, 45.0f, -45.0f); + case MVR_HEADSET_QUEST_PRO: + s.eye_width = 2144, s.eye_height = 2240, rates({72, 90}, 72); + s.max_decode_pixels_per_second = k4k120, s.av1_decode = false, s.stream_refresh = 90; + return withFov(s, -53.0f, 53.0f, 48.0f, -48.0f); + case MVR_HEADSET_STEAM_FRAME: + // its panels are 2160x2160; the clients that run on it decode HEVC 8-bit only + s.eye_width = 2160, s.eye_height = 2160, rates({72, 80, 90, 96, 108, 120}, 90), s.stream_refresh = 90; + return withFov(s, -52.0f, 52.0f, 52.0f, -52.0f); // guess: Valve says "up to 110 degrees" + case MVR_HEADSET_VALVE_INDEX: + s.eye_width = 1728, s.eye_height = 1920, rates({80, 90, 120, 144}, 90); + return withFov(s, -54.0f, 50.0f, 55.0f, -55.0f); // guess: the driver reads its own from the headset + case MVR_HEADSET_VIVE: + s.eye_width = 1344, s.eye_height = 1600, rates({90}, 90); + return withFov(s, -52.0f, 50.0f, 55.0f, -55.0f); // guess: the driver reads its own from the headset + case MVR_HEADSET_VIVE_PRO: + s.eye_width = 1728, s.eye_height = 1920, rates({90}, 90); + return withFov(s, -52.0f, 50.0f, 55.0f, -55.0f); + case MVR_HEADSET_VIVE_PRO_2: + // in its 3264x1632 mode, the largest one an Apple silicon Mac is expected to drive (docs/HEADSETS.md) + s.eye_width = 1952, s.eye_height = 1952, rates({90, 120}, 90); + return withFov(s, -58.0f, 55.0f, 55.0f, -55.0f); + case MVR_HEADSET_PSVR2: + // 2000x2040 OLED panels at 90 or 120 Hz through Sony's PC adapter, rendered at the panel's own size (1.2 times + // it would be nearly 6 million pixels an eye); Monado's field of view (psvr2.c), wider toward the temple + s.eye_width = 2000, s.eye_height = 2040, rates({90, 120}, 90); + return withFov(s, -61.5f, 43.5f, 53.0f, -53.0f); + default: + s.headset = MVR_HEADSET_UNKNOWN; + return s; + } +} + +} // namespace + +const HeadsetSpec &headsetSpec(uint32_t headset) { + static const auto table = [] { + struct Table { + HeadsetSpec specs[MVR_HEADSET_COUNT]; + } t; + for (uint32_t i = 0; i < MVR_HEADSET_COUNT; i++) t.specs[i] = makeSpec(i); + return t; + }(); + return table.specs[headset < MVR_HEADSET_COUNT ? headset : MVR_HEADSET_UNKNOWN]; +} + +uint32_t headsetFromAlvrName(std::string_view name) { + if (name == "Quest 1") return MVR_HEADSET_QUEST_1; + if (name == "Quest 2") return MVR_HEADSET_QUEST_2; + if (name == "Quest 3") return MVR_HEADSET_QUEST_3; + if (name == "Quest 3S") return MVR_HEADSET_QUEST_3S; + if (name == "Quest Pro") return MVR_HEADSET_QUEST_PRO; + // the name MetalVR's patched ALVR client gives itself on a Steam Frame (patches/alvr/0003); the stock client says + // "Android (unknown)" there + if (name == "Steam Frame") return MVR_HEADSET_STEAM_FRAME; + return MVR_HEADSET_UNKNOWN; +} + +uint32_t headsetFromKey(std::string_view key) { + for (uint32_t i = 1; i < MVR_HEADSET_COUNT; i++) + if (key == mvr_describe_headset(i)->key) return i; + return MVR_HEADSET_UNKNOWN; +} + +void applyHeadsetSpec(const HeadsetSpec &spec, mvr_system_info &info) { + const mvr_headset_description *description = mvr_describe_headset(spec.headset); + info.headset = spec.headset; + info.controllers = description->controllers; + std::snprintf(info.system_name, sizeof(info.system_name), "%s", description->name); + if (spec.headset == MVR_HEADSET_UNKNOWN) return; + info.eye_width = spec.eye_width; + info.eye_height = spec.eye_height; + info.refresh_hz = spec.default_refresh; + info.refresh_rate_count = spec.refresh_rate_count; + std::memcpy(info.refresh_rates, spec.refresh_rates, sizeof(info.refresh_rates)); + info.fov[0] = spec.fov[0]; + info.fov[1] = spec.fov[1]; +} + +float handJointRadius(uint32_t joint) { + // palm, wrist; thumb metacarpal..tip; then index, middle, ring, little: metacarpal, proximal, intermediate, + // distal, tip + static const float radii[MVR_HAND_JOINTS] = {0.018f, 0.020f, 0.017f, 0.014f, 0.011f, 0.009f, 0.012f, 0.010f, + 0.009f, 0.008f, 0.007f, 0.012f, 0.010f, 0.009f, 0.008f, 0.007f, + 0.011f, 0.009f, 0.008f, 0.007f, 0.006f, 0.010f, 0.008f, 0.007f, + 0.006f, 0.005f}; + return joint < MVR_HAND_JOINTS ? radii[joint] : 0.008f; +} + +void openHand(const mvr_pose &wrist, bool left, mvr_hand &out) { + // where each joint is from the wrist, in the hand's frame (+x right, +y up, -z toward the fingertips), meters + const float side = left ? -1.0f : 1.0f; + float local[MVR_HAND_JOINTS][3]; + local[0][0] = 0, local[0][1] = 0, local[0][2] = -0.045f; // palm + local[1][0] = 0, local[1][1] = 0, local[1][2] = 0; // wrist + for (int j = 0; j < 4; ++j) { // thumb, toward the inside + local[2 + j][0] = -side * (0.020f + 0.018f * j); + local[2 + j][1] = 0.005f; + local[2 + j][2] = -0.020f - 0.020f * j; + } + for (int finger = 0; finger < 4; ++finger) { + const float x = side * (0.025f - 0.016f * finger); + const float length = finger == 1 ? 1.08f : finger == 3 ? 0.8f : 1.0f; + for (int j = 0; j < 5; ++j) { + float *p = local[6 + finger * 5 + j]; + p[0] = x; + p[1] = 0; + p[2] = -0.02f - (j == 0 ? 0.0f : 0.06f + 0.025f * (j - 1)) * length; + } + } + for (uint32_t j = 0; j < MVR_HAND_JOINTS; ++j) { + // the joint's position turned with the wrist; every joint faces as the wrist does + const mvr_quat &q = wrist.orientation; + const float vx = local[j][0], vy = local[j][1], vz = local[j][2]; + const float tx = 2 * (q.y * vz - q.z * vy), ty = 2 * (q.z * vx - q.x * vz), tz = 2 * (q.x * vy - q.y * vx); + out.joints[j].orientation = q; + out.joints[j].position = {wrist.position.x + vx + q.w * tx + (q.y * tz - q.z * ty), + wrist.position.y + vy + q.w * ty + (q.z * tx - q.x * tz), + wrist.position.z + vz + q.w * tz + (q.x * ty - q.y * tx)}; + out.radii[j] = handJointRadius(j); + } + out.flags = MVR_HAND_TRACKED; +} + +bool withinDecoder(const HeadsetSpec &spec, uint32_t frame_width, uint32_t frame_height, float fps) { + if (spec.max_decode_pixels_per_second == 0 || !(fps > 0.0f)) return true; + const double rate = static_cast(frame_width) * frame_height * fps; + return rate <= static_cast(spec.max_decode_pixels_per_second) * 1.02; // rounding of sizes and rates +} + +} // namespace metalvr diff --git a/daemon/headsets.h b/daemon/headsets.h new file mode 100644 index 0000000..f49b5c4 --- /dev/null +++ b/daemon/headsets.h @@ -0,0 +1,54 @@ +// What metalvrd knows of each headset before the headset itself says (docs/HEADSETS.md): the render size, +// refresh rates and field of view OpenXR is told while none is connected, and what the stream to a standalone +// headset must stay within. A driver replaces each value with the headset's own as soon as it has it. +#pragma once + +#include +#include + +#include "metalvr_headsets.h" +#include "metalvr_protocol.h" + +namespace metalvr { + +struct HeadsetSpec { + uint32_t headset = MVR_HEADSET_UNKNOWN; + uint32_t eye_width = 0; // the per-eye render size applications are asked for + uint32_t eye_height = 0; + float refresh_rates[MVR_MAX_REFRESH_RATES] = {}; // ascending; the first `default_refresh` is the default + uint32_t refresh_rate_count = 0; + float default_refresh = 0; + // a streamed headset's rate for PC VR, which a new ALVR session asks for (0: ALVR's default) + float stream_refresh = 0; + mvr_fov fov[2] = {}; + float ipd = 0.063f; + // a standalone headset's video decoder: the most pixels per second it decodes (0: no limit known), and AV1 + uint64_t max_decode_pixels_per_second = 0; + bool av1_decode = false; +}; + +// the spec of a headset, or one with no values for MVR_HEADSET_UNKNOWN +const HeadsetSpec &headsetSpec(uint32_t headset); + +// the headset an ALVR client's display_name stands for (alvr_system_info's Platform names: "Quest 2", "Quest 3S", +// ...), or MVR_HEADSET_UNKNOWN; matched whole, so "Quest 3" is never a "Quest 3S" +uint32_t headsetFromAlvrName(std::string_view display_name); + +// the headset a command-line key names ("quest-3", "index"; mvr_headset_description.key), or MVR_HEADSET_UNKNOWN +uint32_t headsetFromKey(std::string_view key); + +// Fills a system info's identity and defaults from a spec: name, headset, controllers, render size, refresh rates +// and field of view. Flags, IPD and the play area are left as they are. +void applyHeadsetSpec(const HeadsetSpec &spec, mvr_system_info &info); + +// A hand joint's usual radius in meters (OpenXR's joint order), for headsets that send joints without one. +float handJointRadius(uint32_t joint); + +// A relaxed open hand whose wrist is at `wrist` (the joints in OpenXR's order, x toward the thumb side for the +// right hand), for the simulated headset. +void openHand(const mvr_pose &wrist, bool left, mvr_hand &out); + +// Whether a stream of frame_width x frame_height at fps is within what the headset decodes; true when unknown. +bool withinDecoder(const HeadsetSpec &spec, uint32_t frame_width, uint32_t frame_height, float fps); + +} // namespace metalvr diff --git a/daemon/hid_util.h b/daemon/hid_util.h new file mode 100644 index 0000000..9eba5ff --- /dev/null +++ b/daemon/hid_util.h @@ -0,0 +1,108 @@ +// IOHIDManager access shared by the direct-display backends (backend_rift_s.mm, backend_lighthouse.mm): devices by +// vendor and product, their USB interface, and feature and input reports always with their report id first, which +// is how the protocol parsers take them (macOS hands numbered reports over with or without it). +#pragma once + +#import + +#include +#include +#include +#include + +namespace metalvr::hid { + +inline int intProperty(IOHIDDeviceRef device, CFStringRef key) { + int value = 0; + CFTypeRef p = IOHIDDeviceGetProperty(device, key); + if (p && CFGetTypeID(p) == CFNumberGetTypeID()) + CFNumberGetValue(static_cast(p), kCFNumberIntType, &value); + return value; +} + +inline uint16_t vendor(IOHIDDeviceRef d) { return static_cast(intProperty(d, CFSTR(kIOHIDVendorIDKey))); } +inline uint16_t product(IOHIDDeviceRef d) { return static_cast(intProperty(d, CFSTR(kIOHIDProductIDKey))); } + +// the USB interface a HID device is, from the registry above it; 99 when unknown +inline int interfaceNumber(IOHIDDeviceRef dev) { + int number = 99; + io_service_t service = IOHIDDeviceGetService(dev); + CFTypeRef value = service ? IORegistryEntrySearchCFProperty(service, kIOServicePlane, CFSTR("bInterfaceNumber"), + kCFAllocatorDefault, + kIORegistryIterateRecursively | kIORegistryIterateParents) + : nullptr; + if (value) { + if (CFGetTypeID(value) == CFNumberGetTypeID()) + CFNumberGetValue(static_cast(value), kCFNumberIntType, &number); + CFRelease(value); + } + return number; +} + +// the devices a manager matches now, retained (release each with CFRelease) +inline std::vector devices(IOHIDManagerRef manager) { + std::vector out; + CFSetRef set = IOHIDManagerCopyDevices(manager); + if (!set) + return out; + std::vector values(CFSetGetCount(set)); + CFSetGetValues(set, values.data()); + for (const void *v : values) { + IOHIDDeviceRef d = static_cast(const_cast(v)); + CFRetain(d); + out.push_back(d); + } + CFRelease(set); + return out; +} + +inline void release(std::vector &list) { + for (IOHIDDeviceRef d : list) + CFRelease(d); + list.clear(); +} + +// a feature report whose first byte is its id +inline bool setFeature(IOHIDDeviceRef device, const uint8_t *report, size_t size) { + if (!device || size == 0) + return false; + return IOHIDDeviceSetReport(device, kIOHIDReportTypeFeature, report[0], report, static_cast(size)) == + kIOReturnSuccess; +} + +// feature report `id` into `out`, its id first; how many bytes, 0 if none came +inline size_t getFeature(IOHIDDeviceRef device, uint8_t id, uint8_t *out, size_t size) { + if (!device || size < 2) + return 0; + std::memset(out, 0, size); + out[0] = id; + CFIndex length = static_cast(size); + if (IOHIDDeviceGetReport(device, kIOHIDReportTypeFeature, id, out, &length) != kIOReturnSuccess || length < 1) + return 0; + size_t n = static_cast(length); + if (out[0] != id) { + // handed over without its id: put it in front + n = std::min(n + 1, size); + std::memmove(out + 1, out, n - 1); + out[0] = id; + } + return n; +} + +// an input report as the callback gives it, with its id first; `buffer` must hold length + 1 bytes +inline size_t withId(uint32_t id, const uint8_t *report, CFIndex length, uint8_t *buffer, size_t capacity) { + if (length <= 0) + return 0; + size_t n = static_cast(length); + if (report[0] == id) { + n = std::min(n, capacity); + std::memcpy(buffer, report, n); + return n; + } + n = std::min(n + 1, capacity); + buffer[0] = static_cast(id); + std::memcpy(buffer + 1, report, n - 1); + return n; +} + +} // namespace metalvr::hid diff --git a/daemon/json_value.cpp b/daemon/json_value.cpp new file mode 100644 index 0000000..f07d9cd --- /dev/null +++ b/daemon/json_value.cpp @@ -0,0 +1,234 @@ +#include "json_value.h" + +#include +#include +#include + +namespace metalvr { + +namespace { +const JsonValue &nullValue() { + static const JsonValue none; + return none; +} +} // namespace + +class JsonParser { +public: + explicit JsonParser(std::string_view text) : s_(text) {} + + bool document(JsonValue &out) { + skip(); + if (!value(out, 0)) return false; + skip(); + return i_ == s_.size(); + } + +private: + static constexpr int kMaxDepth = 64; + + void skip() { + while (i_ < s_.size() && (s_[i_] == ' ' || s_[i_] == '\t' || s_[i_] == '\r' || s_[i_] == '\n')) i_++; + } + + bool literal(std::string_view word) { + if (s_.substr(i_, word.size()) != word) return false; + i_ += word.size(); + return true; + } + + bool value(JsonValue &out, int depth) { + if (i_ >= s_.size() || depth > kMaxDepth) return false; + switch (s_[i_]) { + case '{': + return object(out, depth); + case '[': + return array(out, depth); + case '"': + out.type_ = JsonValue::Type::String; + return string(out.string_); + case 't': + out.type_ = JsonValue::Type::Bool; + out.boolean_ = true; + return literal("true"); + case 'f': + out.type_ = JsonValue::Type::Bool; + out.boolean_ = false; + return literal("false"); + case 'n': + out.type_ = JsonValue::Type::Null; + return literal("null"); + default: + return number(out); + } + } + + bool number(JsonValue &out) { + const size_t begin = i_; + if (i_ < s_.size() && s_[i_] == '-') i_++; + while (i_ < s_.size() && (std::strchr("0123456789.eE+-", s_[i_]) != nullptr) && s_[i_] != '\0') i_++; + if (i_ == begin || i_ - begin > 64) return false; + char text[65]; + std::memcpy(text, s_.data() + begin, i_ - begin); + text[i_ - begin] = 0; + char *end = nullptr; + const double v = std::strtod(text, &end); + if (end != text + (i_ - begin) || !std::isfinite(v)) return false; + out.type_ = JsonValue::Type::Number; + out.number_ = v; + return true; + } + + static void appendUtf8(std::string &out, uint32_t c) { + if (c < 0x80) { + out += static_cast(c); + } else if (c < 0x800) { + out += static_cast(0xC0 | (c >> 6)); + out += static_cast(0x80 | (c & 0x3F)); + } else { + out += static_cast(0xE0 | (c >> 12)); + out += static_cast(0x80 | ((c >> 6) & 0x3F)); + out += static_cast(0x80 | (c & 0x3F)); + } + } + + bool string(std::string &out) { + if (i_ >= s_.size() || s_[i_] != '"') return false; + i_++; + while (i_ < s_.size()) { + const char c = s_[i_++]; + if (c == '"') return true; + if (c != '\\') { + out += c; + continue; + } + if (i_ >= s_.size()) return false; + const char e = s_[i_++]; + switch (e) { + case '"': out += '"'; break; + case '\\': out += '\\'; break; + case '/': out += '/'; break; + case 'b': out += '\b'; break; + case 'f': out += '\f'; break; + case 'n': out += '\n'; break; + case 'r': out += '\r'; break; + case 't': out += '\t'; break; + case 'u': { + if (i_ + 4 > s_.size()) return false; + uint32_t code = 0; + for (int k = 0; k < 4; k++) { + const char h = s_[i_++]; + code <<= 4; + if (h >= '0' && h <= '9') code |= h - '0'; + else if (h >= 'a' && h <= 'f') code |= h - 'a' + 10; + else if (h >= 'A' && h <= 'F') code |= h - 'A' + 10; + else return false; + } + appendUtf8(out, code); // a surrogate pair stays two code units; nothing here needs more + break; + } + default: + return false; + } + } + return false; + } + + bool array(JsonValue &out, int depth) { + out.type_ = JsonValue::Type::Array; + i_++; + skip(); + if (i_ < s_.size() && s_[i_] == ']') { + i_++; + return true; + } + for (;;) { + JsonValue element; + skip(); + if (!value(element, depth + 1)) return false; + out.array_.push_back(std::move(element)); + skip(); + if (i_ >= s_.size()) return false; + if (s_[i_] == ',') { + i_++; + continue; + } + if (s_[i_] != ']') return false; + i_++; + return true; + } + } + + bool object(JsonValue &out, int depth) { + out.type_ = JsonValue::Type::Object; + i_++; + skip(); + if (i_ < s_.size() && s_[i_] == '}') { + i_++; + return true; + } + for (;;) { + skip(); + std::string key; + if (!string(key)) return false; + skip(); + if (i_ >= s_.size() || s_[i_] != ':') return false; + i_++; + skip(); + JsonValue member; + if (!value(member, depth + 1)) return false; + out.object_.emplace_back(std::move(key), std::move(member)); + skip(); + if (i_ >= s_.size()) return false; + if (s_[i_] == ',') { + i_++; + continue; + } + if (s_[i_] != '}') return false; + i_++; + return true; + } + } + + std::string_view s_; + size_t i_ = 0; +}; + +std::optional JsonValue::parse(std::string_view text) { + // firmware blocks end in a NUL, which is not part of the document + while (!text.empty() && text.back() == '\0') text.remove_suffix(1); + JsonValue value; + JsonParser parser(text); + if (!parser.document(value)) return std::nullopt; + return value; +} + +const JsonValue &JsonValue::operator[](std::string_view key) const { + if (type_ != Type::Object) return nullValue(); + for (const auto &member : object_) + if (member.first == key) return member.second; + return nullValue(); +} + +const JsonValue &JsonValue::operator[](size_t index) const { + if (type_ != Type::Array || index >= array_.size()) return nullValue(); + return array_[index]; +} + +bool JsonValue::numbers(float *out, size_t n) const { + if (type_ != Type::Array || array_.size() < n) return false; + for (size_t i = 0; i < n; i++) + if (!array_[i].isNumber()) return false; + for (size_t i = 0; i < n; i++) out[i] = static_cast(array_[i].number_); + return true; +} + +bool JsonValue::numbers(double *out, size_t n) const { + if (type_ != Type::Array || array_.size() < n) return false; + for (size_t i = 0; i < n; i++) + if (!array_[i].isNumber()) return false; + for (size_t i = 0; i < n; i++) out[i] = array_[i].number_; + return true; +} + +} // namespace metalvr diff --git a/daemon/json_value.h b/daemon/json_value.h new file mode 100644 index 0000000..a6384b6 --- /dev/null +++ b/daemon/json_value.h @@ -0,0 +1,53 @@ +// A JSON document as values, for the configuration headsets keep in their own firmware (a Vive's or an Index's +// lens and IMU calibration, a Rift S's IMU calibration). Small and strict: what does not parse is rejected whole. +#pragma once + +#include +#include +#include +#include +#include +#include + +namespace metalvr { + +class JsonValue { +public: + enum class Type { Null, Bool, Number, String, Array, Object }; + + // the document, or nothing if `text` is not one JSON value (with only white space around it) + static std::optional parse(std::string_view text); + + Type type() const { return type_; } + bool isNumber() const { return type_ == Type::Number; } + bool isString() const { return type_ == Type::String; } + bool isArray() const { return type_ == Type::Array; } + bool isObject() const { return type_ == Type::Object; } + + double number(double otherwise = 0.0) const { return type_ == Type::Number ? number_ : otherwise; } + const std::string &string() const { return string_; } + bool boolean(bool otherwise = false) const { return type_ == Type::Bool ? boolean_ : otherwise; } + + // an object's member, or null if there is none (or this is not an object) + const JsonValue &operator[](std::string_view key) const; + // an array's element, or null + const JsonValue &operator[](size_t index) const; + size_t size() const { return type_ == Type::Array ? array_.size() : type_ == Type::Object ? object_.size() : 0; } + const std::vector &elements() const { return array_; } + const std::vector> &members() const { return object_; } + + // the first `n` numbers of an array into `out`; false unless it is an array of at least `n` numbers + bool numbers(float *out, size_t n) const; + bool numbers(double *out, size_t n) const; + +private: + friend class JsonParser; + Type type_ = Type::Null; + bool boolean_ = false; + double number_ = 0.0; + std::string string_; + std::vector array_; + std::vector> object_; +}; + +} // namespace metalvr diff --git a/daemon/lighthouse_protocol.cpp b/daemon/lighthouse_protocol.cpp new file mode 100644 index 0000000..8e569bb --- /dev/null +++ b/daemon/lighthouse_protocol.cpp @@ -0,0 +1,557 @@ +#include "lighthouse_protocol.h" + +#include + +#include +#include +#include +#include + +#include "json_value.h" + +namespace metalvr::lighthouse { + +namespace { + +constexpr float kPi = 3.14159265358979323846f; +constexpr float kGravity = 9.80665f; +constexpr size_t kMaxCompressed = 16 * 1024; +constexpr size_t kMaxJson = 64 * 1024; + +uint16_t le16(const uint8_t *p) { return static_cast(p[0] | (p[1] << 8)); } +uint32_t le32(const uint8_t *p) { return p[0] | (p[1] << 8) | (p[2] << 16) | (static_cast(p[3]) << 24); } + +bool vec3(const JsonValue &v, Vec3 &out) { + float f[3]; + if (!v.numbers(f, 3)) return false; + out = {f[0], f[1], f[2]}; + return true; +} + +// a row-major 3x3 matrix from an array of three rows +bool matrix3(const JsonValue &v, float out[9]) { + if (!v.isArray() || v.size() != 3) return false; + for (size_t row = 0; row < 3; row++) + if (!v[row].numbers(out + 3 * row, 3)) return false; + return true; +} + +void readImu(const JsonValue &root, Vec3 &acc_bias, Vec3 &acc_scale, Vec3 &gyro_bias, Vec3 &gyro_scale) { + // early devices (the original Vive) keep the IMU's numbers at the root + const JsonValue &imu = root["imu"].isObject() ? root["imu"] : root; + vec3(imu["acc_bias"], acc_bias); + vec3(imu["acc_scale"], acc_scale); + vec3(imu["gyro_bias"], gyro_bias); + vec3(imu["gyro_scale"], gyro_scale); // not on every device +} + +std::string modelNumber(const JsonValue &root) { + if (root["model_number"].isString()) return root["model_number"].string(); + if (root["model_name"].isString()) return root["model_name"].string(); + return {}; +} + +// a rotation matrix (row-major) as a quaternion +mvr_quat quatFromMatrix(const float m[9]) { + const float trace = m[0] + m[4] + m[8]; + mvr_quat q{}; + if (trace > 0) { + const float s = 0.5f / std::sqrt(trace + 1.0f); + q = {(m[7] - m[5]) * s, (m[2] - m[6]) * s, (m[3] - m[1]) * s, 0.25f / s}; + } else if (m[0] > m[4] && m[0] > m[8]) { + const float s = 2.0f * std::sqrt(1.0f + m[0] - m[4] - m[8]); + q = {0.25f * s, (m[1] + m[3]) / s, (m[2] + m[6]) / s, (m[7] - m[5]) / s}; + } else if (m[4] > m[8]) { + const float s = 2.0f * std::sqrt(1.0f + m[4] - m[0] - m[8]); + q = {(m[1] + m[3]) / s, 0.25f * s, (m[5] + m[7]) / s, (m[2] - m[6]) / s}; + } else { + const float s = 2.0f * std::sqrt(1.0f + m[8] - m[0] - m[4]); + q = {(m[2] + m[6]) / s, (m[5] + m[7]) / s, 0.25f * s, (m[3] - m[1]) / s}; + } + const float n = std::sqrt(q.x * q.x + q.y * q.y + q.z * q.z + q.w * q.w); + if (!(n > 0)) return {0, 0, 0, 1}; + return {q.x / n, q.y / n, q.z / n, q.w / n}; +} + +// Monado's _calculate_fov: the angles from the view's center to the panel's edges, from the lens intrinsics +void computeFov(HeadsetConfig &c) { + const float width = static_cast(c.eye_width), height = static_cast(c.eye_height); + for (int eye = 0; eye < 2; eye++) { + const Distortion &d = c.distortion[eye]; + const float scale = d.grow_for_undistort + 1.0f; + const float focus_x = d.intrinsic_focus[0] / scale * width / 2.0f; + const float focus_y = d.intrinsic_focus[1] / scale * height / 2.0f; + const float center_x = (d.intrinsic_center[0] / scale - 1.0f) * -1.0f * width / 2.0f; + const float center_y = (d.intrinsic_center[1] / scale + 1.0f) * height / 2.0f; + if (!(focus_x > 0) || !(focus_y > 0)) { + c.fov[eye] = {}; + continue; + } + c.fov[eye].up = std::atan(center_y / focus_y); + c.fov[eye].down = -std::atan((height - center_y) / focus_y); + c.fov[eye].left = -std::atan(center_x / focus_x); + c.fov[eye].right = std::atan((width - center_x) / focus_x); + } +} + +// MPU-6500 full-scale codes to rad/s and m/s^2 +void rangesFromCodes(uint8_t gyro, uint8_t accel, ImuRanges &out) { + out.gyro = kPi / 180.0f * static_cast(250 << gyro); + out.accel = kGravity * static_cast(2 << accel); +} + +void scaled(const int16_t raw[3], float range, const Vec3 &scale, const Vec3 &bias, float out[3]) { + const float k = range / 32768.0f; + out[0] = k * scale.x * raw[0] - bias.x; + out[1] = k * scale.y * raw[1] - bias.y; + out[2] = k * scale.z * raw[2] - bias.z; +} + +// Monado's convert_imu_to_openxr: each model's IMU is mounted its own way +void headsetToOpenXr(Model model, float v[3]) { + const float x = v[0], y = v[1], z = v[2]; + switch (model) { + case Model::Vive: + v[0] = x, v[1] = -y, v[2] = -z; + break; + case Model::VivePro: + case Model::VivePro2: + v[0] = -x, v[1] = y, v[2] = -z; + break; + case Model::Index: + v[0] = -y, v[1] = -x, v[2] = -z; + break; + case Model::None: + break; + } +} + +void controllerToOpenXr(ControllerKind kind, float v[3]) { + const float x = v[0], y = v[1], z = v[2]; + switch (kind) { + case ControllerKind::Wand: + v[0] = -x, v[1] = -z, v[2] = -y; + break; + case ControllerKind::IndexRight: + v[0] = z, v[1] = -y, v[2] = x; + break; + case ControllerKind::IndexLeft: + v[0] = -z, v[1] = x, v[2] = -y; + break; + case ControllerKind::Unknown: + break; + } +} + +} // namespace + +uint32_t headsetOf(Model model) { + switch (model) { + case Model::Vive: return MVR_HEADSET_VIVE; + case Model::VivePro: return MVR_HEADSET_VIVE_PRO; + case Model::VivePro2: return MVR_HEADSET_VIVE_PRO_2; + case Model::Index: return MVR_HEADSET_VALVE_INDEX; + case Model::None: break; + } + return MVR_HEADSET_UNKNOWN; +} + +const char *nameOf(Model model) { + switch (model) { + case Model::Vive: return "HTC Vive"; + case Model::VivePro: return "HTC Vive Pro"; + case Model::VivePro2: return "HTC Vive Pro 2"; + case Model::Index: return "Valve Index"; + case Model::None: break; + } + return "no Lighthouse headset"; +} + +Model identify(const std::vector &devices) { + const auto has = [&](uint16_t vendor, uint16_t product) { + return std::any_of(devices.begin(), devices.end(), + [&](const UsbId &d) { return d.vendor == vendor && d.product == product; }); + }; + // the HTC mainboard names an HTC headset; its IMU device has to be there too for it to be of any use + if (has(kHtcVendor, kViveMainboard)) return has(kValveVendor, kViveSensors) ? Model::Vive : Model::None; + if (has(kHtcVendor, kVivePro2Mainboard)) return has(kValveVendor, kLhrSensors) ? Model::VivePro2 : Model::None; + if (has(kHtcVendor, kViveProMainboard) || has(kHtcVendor, kViveProMainboardB)) + return has(kValveVendor, kLhrSensors) ? Model::VivePro : Model::None; + if (has(kValveVendor, kLhrSensors)) return Model::Index; // until its configuration confirms it + return Model::None; +} + +Model modelFromConfig(std::string_view n) { + if (n == "Utah MP" || n == "Index") return Model::Index; + if (n == "Vive MV" || n == "Vive MV." || n == "Vive. MV") return Model::Vive; + if (n == "Vive_Pro MV" || n == "VIVE_Pro MV") return Model::VivePro; + if (n == "Vive_Pro 2 MV" || n == "VIVE_Pro 2 MV" || n == "VIVE_Pro 2 PV" || n == "Vive_Pro 2 PV") + return Model::VivePro2; + return Model::None; +} + +bool ConfigAssembler::add(const uint8_t *report, size_t size) { + if (done_ || size < 2 || report[0] != kConfigReadReport) return false; + const size_t length = report[1]; + if (length > 62 || length + 2 > size || compressed_.size() + length > kMaxCompressed) return false; + if (length == 0) { + done_ = true; + return true; + } + compressed_.insert(compressed_.end(), report + 2, report + 2 + length); + return true; +} + +std::optional ConfigAssembler::json() const { + if (!done_) return std::nullopt; + return inflateConfig(compressed_); +} + +std::optional inflateConfig(const std::vector &compressed) { + if (compressed.empty()) return std::nullopt; + std::string out(kMaxJson, '\0'); + z_stream stream{}; + stream.next_in = const_cast(compressed.data()); + stream.avail_in = static_cast(compressed.size()); + stream.next_out = reinterpret_cast(out.data()); + stream.avail_out = static_cast(out.size()); + if (inflateInit(&stream) != Z_OK) return std::nullopt; + const int result = inflate(&stream, Z_FINISH); + const size_t produced = stream.total_out; + inflateEnd(&stream); + if (result != Z_STREAM_END) return std::nullopt; + out.resize(produced); + return out; +} + +std::optional parseHeadsetConfig(std::string_view text) { + const auto root = JsonValue::parse(text); + if (!root || !root->isObject()) return std::nullopt; + HeadsetConfig c; + c.model_number = modelNumber(*root); + c.model = modelFromConfig(c.model_number); + if (c.model == Model::None) return std::nullopt; + readImu(*root, c.acc_bias, c.acc_scale, c.gyro_bias, c.gyro_scale); + if ((*root)["device_serial_number"].isString()) c.serial = (*root)["device_serial_number"].string(); + if ((*root)["lens_separation"].isNumber()) + c.lens_separation = static_cast((*root)["lens_separation"].number()); + else if ((*root)["ipd"]["default_mm"].isNumber()) + c.lens_separation = static_cast((*root)["ipd"]["default_mm"].number() / 1000.0); + const JsonValue &device = (*root)["device"]; + if (device["eye_target_width_in_pixels"].isNumber()) + c.eye_width = static_cast(device["eye_target_width_in_pixels"].number()); + if (device["eye_target_height_in_pixels"].isNumber()) + c.eye_height = static_cast(device["eye_target_height_in_pixels"].number()); + for (Distortion &d : c.distortion) + d.aspect_x_over_y = static_cast(device["physical_aspect_x_over_y"].number(d.aspect_x_over_y)); + + const JsonValue &eyes = (*root)["tracking_to_eye_transform"]; + for (int eye = 0; eye < 2; eye++) { + const JsonValue &e = eyes[static_cast(eye)]; + if (!e.isObject()) return std::nullopt; // no lens, nothing to draw with + Distortion &d = c.distortion[eye]; + float eye_to_head[9]; + if (matrix3(e["eye_to_head"], eye_to_head)) c.eye_rotation[eye] = quatFromMatrix(eye_to_head); + d.grow_for_undistort = static_cast(e["grow_for_undistort"].number(d.grow_for_undistort)); + d.undistort_r2_cutoff = static_cast(e["undistort_r2_cutoff"].number(d.undistort_r2_cutoff)); + float k[9]; + if (!matrix3(e["intrinsics"], k)) return std::nullopt; + // a pinhole camera's matrix: [fx 0 cx; 0 fy cy; 0 0 -1] + d.intrinsic_focus[0] = k[0]; + d.intrinsic_focus[1] = k[4]; + d.intrinsic_center[0] = k[2]; + d.intrinsic_center[1] = k[5]; + static const char *const names[3] = {"distortion_red", "distortion", "distortion_blue"}; + for (int channel = 0; channel < 3; channel++) { + const JsonValue &dist = e[names[channel]]; + if (!dist.isObject()) continue; + d.center[channel][0] = static_cast(dist["center_x"].number()); + d.center[channel][1] = static_cast(dist["center_y"].number()); + // a Vive has 8 coefficients of which 3 are used, an Index 4 + const JsonValue &coeffs = dist["coeffs"]; + for (size_t i = 0; i < 4 && i < coeffs.size(); i++) + d.coefficients[channel][i] = static_cast(coeffs[i].number()); + } + } + computeFov(c); + for (const mvr_fov &f : c.fov) + if (!(f.left < 0 && f.right > 0 && f.up > 0 && f.down < 0)) return std::nullopt; + return c; +} + +void computeDistortion(const Distortion &d, float u, float v, bool flip_y, float out[3][2]) { + const float factor = 0.5f / (1.0f + d.grow_for_undistort); + for (int i = 0; i < 3; i++) { + float x = 2.0f * u - 1.0f; + float y = (2.0f * v - 1.0f) / d.aspect_x_over_y; + x -= d.center[i][0]; + y -= d.center[i][1]; + const float r2 = x * x + y * y; + const float k1 = d.coefficients[i][0], k2 = d.coefficients[i][1], k3 = d.coefficients[i][2]; + const float k4 = d.coefficients[i][3]; + const float scale = 1.0f / (1.0f + r2 * (k1 + r2 * (k2 + r2 * k3))) + k4; + out[i][0] = 0.5f + (x * scale + d.center[i][0]) * factor; + out[i][1] = 0.5f + (y * scale + d.center[i][1]) * factor * d.aspect_x_over_y; + if (flip_y) out[i][1] = 1.0f - out[i][1]; + } +} + +std::optional parseImuRanges(const uint8_t *report, size_t size) { + if (size < 3 || report[0] != kImuRangeReport) return std::nullopt; + const uint8_t gyro = report[1], accel = report[2]; + if (gyro == 0 || accel == 0 || gyro > 4 || accel > 4) return std::nullopt; + ImuRanges ranges; + rangesFromCodes(gyro, accel, ranges); + return ranges; +} + +size_t parseImuReport(const uint8_t *report, size_t size, const HeadsetConfig &config, const ImuRanges &ranges, + uint8_t &sequence, bool &have_sequence, ImuSample out[3]) { + if (size < kImuReportSize || report[0] != kImuReport) return 0; + // {accel 3 x int16, gyro 3 x int16, time u32, sequence u8} x 3, updated round robin + const auto at = [&](int i) { return report + 1 + 17 * i; }; + const uint8_t s0 = at(0)[16], s1 = at(1)[16], s2 = at(2)[16]; + // Monado's oldest_sequence_index: start at the oldest + int i = 0; + if (s0 == static_cast(s1 + 2)) i = 1; + else if (s1 == static_cast(s2 + 2)) i = 2; + size_t n = 0; + // what was seen is judged against the newest sequence number before this report, as Monado does + const uint8_t last = sequence; + const bool had = have_sequence; + for (int j = 0; j < 3; j++, i = (i + 1) % 3) { + const uint8_t *s = at(i); + const uint8_t seq = s[16]; + if (had && (seq == last || seq == static_cast(last - 1) || seq == static_cast(last - 2))) + continue; + const int16_t acc[3] = {static_cast(le16(s)), static_cast(le16(s + 2)), + static_cast(le16(s + 4))}; + const int16_t gyro[3] = {static_cast(le16(s + 6)), static_cast(le16(s + 8)), + static_cast(le16(s + 10))}; + ImuSample &sample = out[n++]; + sample.ticks = le32(s + 12); + scaled(acc, ranges.accel, config.acc_scale, config.acc_bias, sample.accel); + scaled(gyro, ranges.gyro, config.gyro_scale, config.gyro_bias, sample.gyro); + headsetToOpenXr(config.model, sample.accel); + headsetToOpenXr(config.model, sample.gyro); + sequence = seq; + have_sequence = true; + } + return n; +} + +std::optional parseMainboardStatus(const uint8_t *report, size_t size) { + // {id, u16, len, lens separation u16, u16, button, audio button, 2, proximity change, 1, proximity u16, ipd u16} + if (size < 18 || report[0] != kMainboardStatusReport) return std::nullopt; + MainboardStatus status; + status.lens_separation = le16(report + 4) * 1e-5f; // hundredths of a millimeter + status.proximity = le16(report + 14); + status.ipd = le16(report + 16) * 1e-5f; + return status; +} + +std::array powerReport(bool on) { + // {id 4, type 0x2978, len 56, 9 bytes, 32 reserved, 1 byte, 18 reserved}: Monado's power_on_report + std::array r{}; + r[0] = 0x04; + r[1] = 0x78; + r[2] = 0x29; + r[3] = 56; + const uint8_t on_bytes[9] = {0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x01}; + const uint8_t off_bytes[9] = {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00}; + std::memcpy(r.data() + 4, on ? on_bytes : off_bytes, 9); + r[4 + 9 + 32] = on ? 0x7a : 0x7c; + return r; +} + +const std::array &pro2Modes() { + static const std::array modes = {{ + {0, 2448, 1224, 90.03f, false}, + {1, 2448, 1224, 120.05f, false}, + {2, 3264, 1632, 90.0f, false}, + {3, 3680, 1836, 90.02f, false}, + {4, 4896, 2448, 90.02f, true}, + {5, 4896, 2448, 120.02f, true}, + }}; + return modes; +} + +std::array pro2Command(std::string_view text) { + std::array r{}; + r[0] = 0x04; + r[1] = 0x70; // 0x2970, little-endian + r[2] = 0x29; + const size_t n = std::min(text.size(), r.size() - 4); + r[3] = static_cast(n); + std::memcpy(r.data() + 4, text.data(), n); + return r; +} + +std::optional parseControllerConfig(std::string_view text) { + const auto root = JsonValue::parse(text); + if (!root || !root->isObject()) return std::nullopt; + ControllerConfig c; + c.model_number = modelNumber(*root); + const std::string &n = c.model_number; + if (n == "Vive. Controller MV" || n == "Vive Controller MV") + c.kind = ControllerKind::Wand; + else if (n == "Knuckles Right" || n == "Knuckles EV3.0 Right") + c.kind = ControllerKind::IndexRight; + else if (n == "Knuckles Left" || n == "Knuckles EV3.0 Left") + c.kind = ControllerKind::IndexLeft; + readImu(*root, c.acc_bias, c.acc_scale, c.gyro_bias, c.gyro_scale); + return c; +} + +namespace { + +void controllerImu(const uint8_t *p, uint32_t message_ticks, const ControllerConfig &config, const ImuRanges &ranges, + ControllerImu &out) { + // {timestamp high byte, accel 3 x int16, gyro 3 x int16}; the time is in 48 MHz ticks, its low byte missing + out.ticks = message_ticks | (static_cast(p[0]) << 8); + const int16_t acc[3] = {static_cast(le16(p + 1)), static_cast(le16(p + 3)), + static_cast(le16(p + 5))}; + const int16_t gyro[3] = {static_cast(le16(p + 7)), static_cast(le16(p + 9)), + static_cast(le16(p + 11))}; + scaled(acc, ranges.accel, config.acc_scale, config.acc_bias, out.accel); + scaled(gyro, ranges.gyro, config.gyro_scale, config.gyro_bias, out.gyro); + controllerToOpenXr(config.kind, out.accel); + controllerToOpenXr(config.kind, out.gyro); +} + +constexpr size_t kImuSampleSize = 13; +constexpr size_t kTouchForceSize = 8; +constexpr uint8_t kTypeFlagTouchForce = 161; + +void battery(uint8_t b, ControllerState &s) { + s.battery = b & 0x7f; + s.charging = (b & 0x80) != 0; + s.has_battery = true; +} + +void touchForce(const uint8_t *p, ControllerState &s) { + // {type flag, touched, middle, ring, pinky, index on the trigger, squeeze force, trackpad force} + s.touch = p[1]; + s.fingers[1] = p[2]; + s.fingers[2] = p[3]; + s.fingers[3] = p[4]; + s.fingers[0] = p[5]; + s.squeeze_force = p[6]; + s.trackpad_force = p[7]; + s.has_fingers = true; +} + +void pad(const uint8_t *p, ControllerState &s) { + s.pad_x = static_cast(static_cast(le16(p))) / 32767.0f; + s.pad_y = static_cast(static_cast(le16(p + 2))) / 32767.0f; +} + +// Monado's vive_controller_decode_watchmanv1 +size_t decodeV1(const uint8_t *buf, const uint8_t *end, uint32_t ticks, const ControllerConfig &config, + const ImuRanges &ranges, ControllerState &s, ControllerImu *imu) { + size_t samples = 0; + if ((*buf & 0xe0) != 0xe0 || buf >= end) return 0; + const uint8_t flags = *buf++; + const bool has_battery = (flags & 0x10) != 0x10 && (flags & 0x1) == 0x1; + const bool has_trigger = (flags & 0x10) == 0x10 && (flags & 0x4) == 0x4; + const bool has_trackpad = (flags & 0x10) == 0x10 && (flags & 0x2) == 0x2; + const bool has_buttons = (flags & 0x10) == 0x10 && (flags & 0x1) == 0x1; + const bool has_imu = (flags & 0x08) == 0x8; + const auto room = [&](size_t n) { return buf + n <= end + 1; }; + if (has_battery && room(1)) battery(*buf++, s); + if (has_buttons && room(1)) s.buttons = *buf++; + if (has_trigger && room(1)) s.trigger = *buf++ / 255.0f; + if (has_trackpad && room(4)) { + pad(buf, s); + buf += 4; + } + if (has_imu && room(kImuSampleSize)) { + controllerImu(buf, ticks, config, ranges, imu[samples++]); + buf += kImuSampleSize; + } + return samples; +} + +// Monado's vive_controller_decode_watchmanv2 +size_t decodeV2(const uint8_t *buf, const uint8_t *end, uint32_t ticks, const ControllerConfig &config, + const ImuRanges &ranges, ControllerState &s, ControllerImu *imu) { + size_t samples = 0; + const auto room = [&](size_t n) { return buf + n <= end + 1; }; + if (buf < end && *buf == 0xe1) { + buf++; + if (room(1)) battery(*buf++, s); + } + if (buf < end && *buf == 0xf0) { + buf++; + if (room(kTouchForceSize)) { + touchForce(buf, s); + buf += kTouchForceSize; + } + } + if (buf < end && *buf == 0xe8) { + buf++; + if (room(kImuSampleSize)) { + controllerImu(buf, ticks, config, ranges, imu[samples++]); + buf += kImuSampleSize; + } + } + if (buf < end - 1 && (*buf & 0xf0) == 0xf0) { + const uint8_t flags = *buf++; + if ((flags & 0x1) && room(1)) s.buttons = *buf++; + if ((flags & 0x4) && room(1)) s.trigger = *buf++ / 255.0f; + if ((flags & 0x2) && room(4)) { + pad(buf, s); + buf += 4; + } + if ((flags & 0x8) && room(kTouchForceSize) && *buf == kTypeFlagTouchForce) { + touchForce(buf, s); + buf += kTouchForceSize; + } + if (buf < end && end - buf >= static_cast(kImuSampleSize)) { + controllerImu(buf, ticks, config, ranges, imu[samples++]); + buf += kImuSampleSize; + } + } + return samples; +} + +} // namespace + +size_t decodeControllerReport(const uint8_t *report, size_t size, bool gen2, const ControllerConfig &config, + const ImuRanges &ranges, ControllerState &state, ControllerImu imu[4]) { + if (size < 1) return 0; + int messages = 0; + if (report[0] == kControllerReport1 && size >= 30) messages = 1; + if (report[0] == kControllerReport2 && size >= 59) messages = 2; + size_t samples = 0; + for (int m = 0; m < messages; m++) { + // {timestamp high, length, timestamp low, payload 26} + const uint8_t *message = report + 1 + 29 * m; + const uint8_t length = message[1]; + if (length < 1 || length > 26) continue; + const uint8_t *payload = message + 3; + const uint8_t *end = payload + length - 1; + const uint32_t ticks = (static_cast(message[0]) << 24) | (static_cast(message[2]) << 16); + samples += gen2 ? decodeV2(payload, end, ticks, config, ranges, state, imu + samples) + : decodeV1(payload, end, ticks, config, ranges, state, imu + samples); + } + return samples; +} + +std::array hapticReport(float duration_s, float frequency_hz, float amplitude) { + // a pulse train: frequency = 1e6 / (high + low) microseconds, amplitude the low part's share, Monado's mapping + if (!(frequency_hz > 0)) frequency_hz = 150.0f; + frequency_hz = std::clamp(frequency_hz, 1.0f, 1000.0f); + amplitude = std::clamp(std::isfinite(amplitude) ? amplitude : 0.0f, 0.0f, 1.0f); + duration_s = std::clamp(std::isfinite(duration_s) ? duration_s : 0.0f, 0.0f, 10.0f); + const float high_plus_low = 1e6f / frequency_hz; + uint16_t low = static_cast(amplitude * high_plus_low / 2.0f); + if (low == 0) low = 1; // a Vive wand does not move with 0 + const uint16_t high = static_cast(high_plus_low - low); + const uint16_t repeat = duration_s > 0 ? static_cast(std::max(1.0f, duration_s * frequency_hz)) : 0; + return {0xff, 0x8f, 7, 0, static_cast(high & 0xff), static_cast(high >> 8), + static_cast(low & 0xff), static_cast(low >> 8), static_cast(repeat & 0xff), + static_cast(repeat >> 8)}; +} + +} // namespace metalvr::lighthouse diff --git a/daemon/lighthouse_protocol.h b/daemon/lighthouse_protocol.h new file mode 100644 index 0000000..bf737a2 --- /dev/null +++ b/daemon/lighthouse_protocol.h @@ -0,0 +1,212 @@ +// The USB side of Valve's Lighthouse headsets and controllers (the HTC Vive, Vive Pro and Vive Pro 2, and the +// Valve Index): which one is plugged in, the calibration each keeps in its own firmware, its IMU reports, and the +// Watchman radio its controllers report through (docs/HEADSETS.md). +// +// Nothing here touches a device: these are the reports' layouts, taken in and made as bytes, so that each can be +// checked without a headset (tests/headset_protocol_check.cpp). The layouts and the arithmetic follow Monado's Vive +// driver (src/xrt/drivers/vive, src/xrt/auxiliary/vive, src/xrt/drivers/vp2 and u_distortion_mesh.c; Copyright +// 2016-2019 Philipp Zabel, 2019-2023 Collabora, Ltd. and others; BSL-1.0, https://gitlab.freedesktop.org/monado/monado). +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include "metalvr_protocol.h" + +namespace metalvr::lighthouse { + +constexpr uint16_t kHtcVendor = 0x0bb4; +constexpr uint16_t kValveVendor = 0x28de; +constexpr uint16_t kViveMainboard = 0x2c87; // HTC Vive +constexpr uint16_t kViveProMainboard = 0x0309; // HTC Vive Pro (Monado, OpenHMD) +constexpr uint16_t kViveProMainboardB = 0x030e; // HTC Vive Pro (libsurvive) +constexpr uint16_t kVivePro2Mainboard = 0x0342; // HTC Vive Pro 2 +constexpr uint16_t kViveSensors = 0x2000; // the Vive's IMU and Lighthouse receiver +constexpr uint16_t kLhrSensors = 0x2300; // the Vive Pro's, Vive Pro 2's and Index's +constexpr uint16_t kWatchmanGen1 = 0x2101; // a receiver for Vive wands +constexpr uint16_t kWatchmanGen2 = 0x2102; // a receiver for Index controllers + +enum class Model { None, Vive, VivePro, VivePro2, Index }; + +// mvr_headset of a model +uint32_t headsetOf(Model model); +const char *nameOf(Model model); + +struct UsbId { + uint16_t vendor = 0; + uint16_t product = 0; +}; + +// Which headset the devices on USB are, or None. An HTC mainboard names its headset; Valve's IMU device 2300 with no +// HTC mainboard is an Index only once its own configuration says so (modelFromConfig), since a Vive Pro whose +// mainboard is not seen, or a Tracker, shows the same device: here it is Index, to be confirmed. +Model identify(const std::vector &devices); + +// the model a headset's configuration names (its "model_number"), or None +Model modelFromConfig(std::string_view model_number); + +// ---- the configuration in a device's firmware: feature report 0x10 starts it, 0x11 reports carry it zlib-compressed + +constexpr uint8_t kConfigStartReport = 0x10; +constexpr uint8_t kConfigReadReport = 0x11; + +// Gathers the 0x11 reports' payloads. add() takes one report (with its id); false when it is not one or the data +// is too much. done() once a report with no payload has come. +class ConfigAssembler { +public: + bool add(const uint8_t *report, size_t size); + bool done() const { return done_; } + // the JSON, inflated; nothing if it does not inflate + std::optional json() const; + +private: + std::vector compressed_; + bool done_ = false; +}; + +std::optional inflateConfig(const std::vector &compressed); + +struct Vec3 { + float x = 0, y = 0, z = 0; +}; + +// one eye's lens, as the configuration gives it (Monado's u_vive_values) +struct Distortion { + float aspect_x_over_y = 0.9f; + float grow_for_undistort = 0.5f; + float intrinsic_focus[2] = {}; + float intrinsic_center[2] = {}; + float undistort_r2_cutoff = 1.0f; + float center[3][2] = {}; // red, green, blue + float coefficients[3][4] = {}; +}; + +struct HeadsetConfig { + std::string model_number; + std::string serial; + Model model = Model::None; + Vec3 acc_bias, acc_scale{1, 1, 1}, gyro_bias, gyro_scale{1, 1, 1}; + int eye_width = 1080; // the panel's pixels for one eye + int eye_height = 1200; + float lens_separation = 0; // meters; 0 if the configuration has none (the IPD is the headset's to report) + Distortion distortion[2]; + mvr_fov fov[2] = {}; + // each eye's view in the head ("eye_to_head"): the Index's lenses are canted + mvr_quat eye_rotation[2] = {{0, 0, 0, 1}, {0, 0, 0, 1}}; +}; + +std::optional parseHeadsetConfig(std::string_view json); + +// Where each colour of the point (u, v) of an eye's view on the panel (0..1 across and down) is in the rendered +// eye image (0..1 across and down): Monado's u_compute_distortion_vive. The Vive Pro 2's image is upside down +// against the others' (flip_y). +void computeDistortion(const Distortion &d, float u, float v, bool flip_y, float out[3][2]); + +// ---- the IMU: report 0x20, three samples of an MPU-6500 in round robin; its ranges from feature report 0x01 + +constexpr uint8_t kImuReport = 0x20; +constexpr size_t kImuReportSize = 52; +constexpr uint8_t kImuRangeReport = 0x01; + +struct ImuRanges { + float gyro = 8.726646f; // rad/s at full scale (500 degrees/s) + float accel = 39.2266f; // m/s^2 at full scale (4 g) +}; + +std::optional parseImuRanges(const uint8_t *report, size_t size); + +struct ImuSample { + uint32_t ticks = 0; // the device's 48 MHz clock + float accel[3] = {}; // m/s^2, in the headset's frame as OpenXR has it (+x right, +y up, +z back) + float gyro[3] = {}; // rad/s +}; + +constexpr double kTicksPerSecond = 48e6; + +// The samples of an IMU report not seen before, oldest first. `sequence` is the newest sequence number taken, +// `have_sequence` whether there is one yet; both are updated. +size_t parseImuReport(const uint8_t *report, size_t size, const HeadsetConfig &config, const ImuRanges &ranges, + uint8_t &sequence, bool &have_sequence, ImuSample out[3]); + +// ---- the HTC mainboard (Vive Pro, Vive Pro 2): status report 0x03, and feature report 0x04 commands + +constexpr uint8_t kMainboardStatusReport = 0x03; + +struct MainboardStatus { + float ipd = 0; // meters + float lens_separation = 0; // meters + uint16_t proximity = 0; +}; + +std::optional parseMainboardStatus(const uint8_t *report, size_t size); + +// whether a proximity reading means the headset is on a face (a guess from the community Vive Pro 2 driver) +constexpr uint16_t kWornProximity = 150; + +// feature report 0x04 of type 0x2978: the headset's power on and off (Vive, Vive Pro) +std::array powerReport(bool on); + +// the Vive Pro 2's modes (docs/HEADSETS.md), which `dtd,N` selects +struct Pro2Mode { + int index; + int width, height; // both eyes + float refresh_hz; + bool needs_dsc; // only with Display Stream Compression +}; +const std::array &pro2Modes(); +// feature report 0x04 of type 0x2970 carrying a text command ("wireless,0", "dtd,2", "chipreset") +std::array pro2Command(std::string_view text); + +// ---- controllers, through a Watchman receiver: reports 0x23 (one message) and 0x24 (two) + +constexpr uint8_t kControllerReport1 = 0x23; +constexpr uint8_t kControllerReport2 = 0x24; +constexpr uint8_t kControllerDisconnect = 0x26; + +enum class ControllerKind { Unknown, Wand, IndexLeft, IndexRight }; + +struct ControllerConfig { + std::string model_number; + ControllerKind kind = ControllerKind::Unknown; + Vec3 acc_bias, acc_scale{1, 1, 1}, gyro_bias, gyro_scale{1, 1, 1}; +}; + +std::optional parseControllerConfig(std::string_view json); + +// what a controller has reported so far +struct ControllerState { + // Vive wand bits: trigger click 0, trackpad touch 1, trackpad click 2, system 3, grip 4, menu 5. + // Index bits: trigger click 0, trackpad touch 1, thumbstick click 2, system 3, A 4, B 5. + uint8_t buttons = 0; + // Index: touched, as buttons' bits 3..6 (system 3, A 4, B 5, thumbstick 6) + uint8_t touch = 0; + float trigger = 0; + float pad_x = 0, pad_y = 0; // the trackpad, or on an Index the thumbstick when the pad is not touched + uint8_t battery = 0; // percent + bool charging = false; + bool has_battery = false; + uint8_t fingers[4] = {}; // index, middle, ring, pinky: 0 open .. 255 around the handle + uint8_t squeeze_force = 0, trackpad_force = 0; + bool has_fingers = false; +}; + +// One message's IMU sample, already in the controller's frame (+x right, +y up, +z back); `ticks` from the message. +struct ControllerImu { + uint32_t ticks = 0; + float accel[3] = {}, gyro[3] = {}; +}; + +// Decodes the messages of a report into `state`, and its IMU samples into `imu` (at most four, two a message); +// returns how many IMU samples there were. `gen2` for an Index controller's receiver. +size_t decodeControllerReport(const uint8_t *report, size_t size, bool gen2, const ControllerConfig &config, + const ImuRanges &ranges, ControllerState &state, ControllerImu imu[4]); + +// the haptic pulse command for a controller, sent as a feature report to its receiver +std::array hapticReport(float duration_s, float frequency_hz, float amplitude); + +} // namespace metalvr::lighthouse diff --git a/daemon/lighthouse_tracking.cpp b/daemon/lighthouse_tracking.cpp new file mode 100644 index 0000000..3780cda --- /dev/null +++ b/daemon/lighthouse_tracking.cpp @@ -0,0 +1,203 @@ +#include "lighthouse_tracking.h" + +#ifdef METALVR_LIBSURVIVE + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include "common.h" + +namespace metalvr { +namespace { + +constexpr double kSeenSeconds = 0.5; + +// Monado's survive_driver.c pose_to_relation: libsurvive's world has +z up; its objects look down their -z, which +// is turned forward here +mvr_quat rotationToOpenXr(const LinmathQuat q) { + const double s = std::sqrt(0.5); + // down = (x s, y 0, z 0, w -s), applied on the left: down * q + const double qw = q[0], qx = q[1], qy = q[2], qz = q[3]; + const double dw = -s, dx = s; + mvr_quat out{static_cast(dw * qx + dx * qw), static_cast(dw * qy - dx * qz), + static_cast(dw * qz + dx * qy), static_cast(dw * qw - dx * qx)}; + const float n = std::sqrt(out.x * out.x + out.y * out.y + out.z * out.z + out.w * out.w); + if (!(n > 0)) return {0, 0, 0, 1}; + return {out.x / n, out.y / n, out.z / n, out.w / n}; +} + +mvr_vec3 pointToOpenXr(const LinmathVec3d p) { + return {static_cast(p[0]), static_cast(p[2]), static_cast(-p[1])}; +} + +void logFromSurvive(SurviveSimpleContext *, SurviveLogLevel level, const char *message) { + if (level == SURVIVE_LOG_LEVEL_ERROR || level == SURVIVE_LOG_LEVEL_WARNING) + logInfo("libsurvive: %s", message); + else + logDebug("libsurvive: %s", message); +} + +class Survive final : public LighthouseTracking { +public: + bool start(bool simulated) { + // The Lighthouse backend opens the same HID devices: hidapi must not take them for itself. hid_init sets + // exclusive opening once, at its first call; libsurvive's own call after this one leaves it as set here. + if (hid_init() != 0) + return false; + hid_darwin_set_open_exclusive(0); + // libsurvive keeps where the base stations stand (its calibration) beside MetalVR's other settings, so that a + // room is solved once + // (the simulated room of a check is kept apart, in a file of its own that starts empty) + const char *home = std::getenv("HOME"); + config_path_ = simulated ? std::filesystem::temp_directory_path().string() + : std::string(home ? home : ".") + "/Library/Application Support/MetalVR"; + std::error_code error; + std::filesystem::create_directories(config_path_, error); + config_path_ += simulated ? "/metalvr-libsurvive-simulated.json" : "/libsurvive.json"; + if (simulated) + std::filesystem::remove(config_path_, error); + char *args[] = {const_cast("metalvrd"), + const_cast("--lighthouse-gen"), + const_cast("0"), // 1.0 or 2.0 base stations, whichever are there + const_cast("--globalscenesolver"), + const_cast("1"), + const_cast("--configfile"), + config_path_.data(), + const_cast("--simulator"), + const_cast("1")}; + const int count = sizeof(args) / sizeof(args[0]) - (simulated ? 0 : 2); + context_ = survive_simple_init_with_logger(count, args, logFromSurvive); + if (!context_) + return false; + survive_simple_start_thread(context_); + logInfo("lighthouse: libsurvive tracks positions with the base stations (its calibration in %s)", + config_path_.c_str()); + return true; + } + + ~Survive() override { + if (context_) + survive_simple_close(context_); + } + + bool pose(int device, TrackedPose &out) override { + std::lock_guard guard(mutex_); + const SurviveSimpleObject *object = find(device); + if (!object) + return false; + SurvivePose pose{}; + const FLT time = survive_simple_object_get_latest_pose(object, &pose); + if (time <= 0 || (pose.Rot[0] == 0 && pose.Rot[1] == 0 && pose.Rot[2] == 0 && pose.Rot[3] == 0)) + return false; + SurviveVelocity velocity{}; + survive_simple_object_get_latest_velocity(object, &velocity); + out.orientation = rotationToOpenXr(pose.Rot); + out.position = pointToOpenXr(pose.Pos); + out.velocity = pointToOpenXr(velocity.Pos); + out.position_tracked = survive_simple_run_time(context_) - time < kSeenSeconds; + return true; + } + + uint32_t references(mvr_tracking_reference *out) override { + std::lock_guard guard(mutex_); + uint32_t count = 0; + for (const SurviveSimpleObject *o = survive_simple_get_first_object(context_); + o && count < MVR_MAX_TRACKING_REFERENCES; o = survive_simple_get_next_object(context_, o)) { + if (survive_simple_object_get_type(o) != SurviveSimpleObject_LIGHTHOUSE) + continue; + mvr_tracking_reference &r = out[count++]; + std::memset(&r, 0, sizeof(r)); + const char *serial = survive_simple_serial_number(o); + std::snprintf(r.serial, sizeof(r.serial), "%s", serial && *serial ? serial : survive_simple_object_name(o)); + SurvivePose pose{}; + if (survive_simple_object_get_latest_pose(o, &pose) > 0 && + (pose.Rot[0] != 0 || pose.Rot[1] != 0 || pose.Rot[2] != 0 || pose.Rot[3] != 0)) { + r.flags = MVR_REFERENCE_POSE_VALID; + r.pose.orientation = rotationToOpenXr(pose.Rot); + r.pose.position = pointToOpenXr(pose.Pos); + } else { + r.pose.orientation = {0, 0, 0, 1}; + } + } + return count; + } + + std::vector> objects() override { + std::lock_guard guard(mutex_); + std::vector> out; + for (const SurviveSimpleObject *o = survive_simple_get_first_object(context_); o; + o = survive_simple_get_next_object(context_, o)) { + SurvivePose pose{}; + const FLT time = survive_simple_object_get_latest_pose(o, &pose); + if (time <= 0 || (pose.Rot[0] == 0 && pose.Rot[1] == 0 && pose.Rot[2] == 0 && pose.Rot[3] == 0)) + continue; + TrackedPose p; + p.orientation = rotationToOpenXr(pose.Rot); + p.position = pointToOpenXr(pose.Pos); + p.position_tracked = survive_simple_run_time(context_) - time < kSeenSeconds; + out.emplace_back(survive_simple_object_name(o), p); + } + return out; + } + +private: + // the head is libsurvive's HMD; a controller is an Index controller of its side, or a Vive wand, the first one + // seen the left hand + const SurviveSimpleObject *find(int device) { + const SurviveSimpleObject *wands[2] = {nullptr, nullptr}; + int wand_count = 0; + for (const SurviveSimpleObject *o = survive_simple_get_first_object(context_); o; + o = survive_simple_get_next_object(context_, o)) { + const auto type = survive_simple_object_get_type(o); + const auto subtype = survive_simple_object_get_subtype(o); + if (device == kHead && type == SurviveSimpleObject_HMD) + return o; + if (type != SurviveSimpleObject_OBJECT) + continue; + if (subtype == SURVIVE_OBJECT_SUBTYPE_KNUCKLES_L && device == kLeft) + return o; + if (subtype == SURVIVE_OBJECT_SUBTYPE_KNUCKLES_R && device == kRight) + return o; + if (subtype == SURVIVE_OBJECT_SUBTYPE_WAND && wand_count < 2) + wands[wand_count++] = o; + } + return device == kLeft ? wands[0] : device == kRight ? wands[1] : nullptr; + } + + SurviveSimpleContext *context_ = nullptr; + std::string config_path_; + std::mutex mutex_; +}; + +} // namespace + +bool lighthouseTrackingBuilt() { return true; } + +std::unique_ptr startLighthouseTracking(bool simulated) { + auto survive = std::make_unique(); + if (!survive->start(simulated)) { + logInfo("lighthouse: libsurvive did not start; rotation only"); + return nullptr; + } + return survive; +} + +} // namespace metalvr + +#else + +namespace metalvr { +bool lighthouseTrackingBuilt() { return false; } +std::unique_ptr startLighthouseTracking(bool) { return nullptr; } +} // namespace metalvr + +#endif diff --git a/daemon/lighthouse_tracking.h b/daemon/lighthouse_tracking.h new file mode 100644 index 0000000..5ea7c12 --- /dev/null +++ b/daemon/lighthouse_tracking.h @@ -0,0 +1,44 @@ +// Position tracking for the Lighthouse headsets with their base stations (docs/HEADSETS.md): libsurvive (MIT, +// https://github.com/cntools/libsurvive), built by scripts/build-libsurvive.sh and linked when it is there. It reads +// the headset's and the controllers' light sensors through hidapi, solves where each is and where the base stations +// stand, and this hands those poses over in OpenXR's axes (+y up, -z forward), in the tracker's own space. +// +// The Lighthouse backend keeps the rest of the headset (its lenses, its IMU, which turns the head, its controllers' +// inputs); the driver takes positions from here, and pulls the head's heading toward the tracker's, as it does with +// a Rift's sensors. +#pragma once + +#include +#include +#include +#include +#include + +#include "metalvr_protocol.h" + +namespace metalvr { + +struct TrackedPose { + mvr_quat orientation{0, 0, 0, 1}; + mvr_vec3 position{}, velocity{}; + bool position_tracked = false; // the base stations saw it in the last half second +}; + +class LighthouseTracking { +public: + virtual ~LighthouseTracking() = default; + enum Device { kHead = 0, kLeft = 1, kRight = 2 }; + // the latest pose of a device; false while libsurvive has none + virtual bool pose(int device, TrackedPose &out) = 0; + // the base stations libsurvive knows, where it has them + virtual uint32_t references(mvr_tracking_reference *out) = 0; + // every object libsurvive has a pose for, by its name (for checking) + virtual std::vector> objects() = 0; +}; + +// libsurvive, started: nothing if MetalVR was built without it or it does not start. `simulated`: libsurvive's own +// simulated device and base stations instead of USB, for checking the whole path without a headset. +std::unique_ptr startLighthouseTracking(bool simulated = false); +bool lighthouseTrackingBuilt(); + +} // namespace metalvr diff --git a/daemon/main.mm b/daemon/main.mm index 3398ba5..6b7cf58 100644 --- a/daemon/main.mm +++ b/daemon/main.mm @@ -9,8 +9,21 @@ #include #include +#import +#import + +#include +#include + #include "common.h" +#include "compositor.h" +#include "direct_backend.h" +#include "psvr2_protocol.h" #include "driver.h" +#include "foveation.h" +#include "headsets.h" +#include "lighthouse_protocol.h" +#include "lighthouse_tracking.h" #include "server.h" namespace metalvr { @@ -35,21 +48,36 @@ void usage() { std::fprintf( stderr, "usage: metalvrd [options]\n" - " --driver sim|alvr|rift headset driver (default: alvr if built in, else sim); rift: an Oculus Rift CV1\n" - " on this Mac's video output and USB (docs/RIFT.md)\n" + " --driver sim|alvr|rift|lighthouse|psvr2\n" + " headset driver (default: alvr if built in, else sim). rift: an Oculus Rift CV1\n" + " or Rift S on this Mac's video output and USB (docs/RIFT.md); lighthouse: an HTC\n" + " Vive, Vive Pro, Vive Pro 2 or Valve Index there; psvr2: a PlayStation VR2\n" + " through Sony's PC adapter (docs/HEADSETS.md)\n" " --port N loopback port runtimes connect to (default %d, or $%s)\n" " --config-dir PATH where the ALVR driver keeps its session and logs\n" " --trust HOSTNAME let this headset connect (as its ALVR client shows it, e.g. 1234.client.local.);\n" " repeatable. A headset that is not trusted is logged with its name and refused.\n" + " --headset MODEL which headset this is (docs/HEADSETS.md): quest-1, quest-2, quest-3, quest-3s,\n" + " quest-pro, steam-frame; alvr: instead of the model its client reports; sim: the\n" + " headset to stand in for, also rift-cv1, rift-s, index, vive, vive-pro, vive-pro-2,\n" + " psvr2\n" + " --adb PATH alvr: the adb a headset on USB (--trust client.wired) is reached with (default:\n" + " adb on PATH)\n" + " --foveated-encoding alvr: squeeze each eye toward its edges before encoding, as ALVR's foveated\n" + " encoding does (its shape from ALVR's settings); fewer pixels for the headset\n" " --verbose log per-swapchain and per-interval detail\n" - "Rift CV1:\n" + "rift, lighthouse and psvr2:\n" " --ipd MM the distance between the eyes (default: the headset's lens separation)\n" - " --test-scene rift: between applications, a grid for checking the lens correction instead of\n" + " --passthrough lighthouse: the home shows the room through the headset's cameras\n" + " --refresh HZ set the headset's display to this rate, if it has it (an Index: 80 to 144)\n" + " --pro2-mode N lighthouse: switch a Vive Pro 2 to mode N first (0 2448x1224@90, 1 @120,\n" + " 2 3264x1632@90, 3 3680x1836@90; 4 and 5, 4896x2448, need DSC)\n" + " --test-scene between applications, a grid for checking the lens correction instead of\n" " the home\n" - " --home-logo PATH rift: the logo the home shows (default: ../share/metalvr/home-logo.png beside\n" + " --home-logo PATH the logo the home shows (default: ../share/metalvr/home-logo.png beside\n" " metalvrd)\n" - "simulated headset and Rift CV1:\n" - " --eye WxH per-eye render size (default 1440x1584; Rift CV1 1344x1600)\n" + "simulated headset, rift, lighthouse and psvr2:\n" + " --eye WxH per-eye render size (default 1440x1584; the others: the headset's)\n" "simulated headset:\n" " --fps N refresh rate (default 72)\n" " --codec h264|hevc (default h264)\n" @@ -60,10 +88,260 @@ void usage() { " --snapshot FILE.png write one composed frame as a PNG\n" " --snapshot-frame N which composed frame (default 30)\n" " --exit-after-session quit when the client that ran a session disconnects\n" + " --passthrough-check draw an Index's home with a test camera picture, check each eye, and exit\n" + " --survive-check [sim] start libsurvive (Lighthouse positions) for a few seconds, list what it tracks;\n" + " sim: its simulated device and base stations, which must be tracked\n" + " --foveation-check squeeze a test picture for foveated encoding on the GPU, check it, and exit\n" + " --panel-check MODEL PNG draw the lens-check grid for rift-cv1, rift-s, index, vive, vive-pro, vive-pro-2\n" + " or psvr2 into PNG, check each eye's straight-ahead mark, and exit\n" " --software-encoder encode on the CPU; for machines with no hardware encoder, not for a headset\n", MVR_DEFAULT_PORT, MVR_PORT_ENV); } +// --panel-check: the lens-check grid for a headset's model, drawn as the headset is drawn but into a picture; each +// eye's straight-ahead mark (the red dot of the grid) must be inside that eye's part of the panel. The Lighthouse +// headsets' lenses come from their firmware: here a stand-in lens, barrel-shaped, with the model's layout. +int panelCheck(const std::string &key, const std::string &png) { + const uint32_t headset = headsetFromKey(key); + DirectModel model; + uint32_t width = 0, height = 0; + if (headset == MVR_HEADSET_RIFT_CV1) { + model = riftCv1Model(); + width = 2160, height = 1200; + } else if (headset == MVR_HEADSET_RIFT_S) { + model = riftSModel(); + width = 1440, height = 2560; + } else if (headset == MVR_HEADSET_PSVR2) { + // the lens as an uncalibrated headset has it (Monado's default), on the 4000x2040 panel + model = psvr2Model(psvr2::defaultLensCalibration()); + width = 4000, height = 2040; + } else if (headset == MVR_HEADSET_VALVE_INDEX || headset == MVR_HEADSET_VIVE || headset == MVR_HEADSET_VIVE_PRO || + headset == MVR_HEADSET_VIVE_PRO_2) { + lighthouse::HeadsetConfig config; + config.model = headset == MVR_HEADSET_VALVE_INDEX ? lighthouse::Model::Index + : headset == MVR_HEADSET_VIVE ? lighthouse::Model::Vive + : headset == MVR_HEADSET_VIVE_PRO ? lighthouse::Model::VivePro + : lighthouse::Model::VivePro2; + config.eye_width = headset == MVR_HEADSET_VIVE ? 1080 : headset == MVR_HEADSET_VIVE_PRO_2 ? 1632 : 1440; + config.eye_height = headset == MVR_HEADSET_VIVE ? 1200 : headset == MVR_HEADSET_VIVE_PRO_2 ? 1632 : 1600; + for (int eye = 0; eye < 2; ++eye) { + lighthouse::Distortion &d = config.distortion[eye]; + d.intrinsic_focus[0] = d.intrinsic_focus[1] = 1.5f; // 45 degrees to each edge + for (int c = 0; c < 3; ++c) + d.coefficients[c][0] = 0.08f + 0.02f * c; + config.fov[eye] = {-0.785f, 0.785f, 0.785f, -0.785f}; + if (headset == MVR_HEADSET_VALVE_INDEX) { + // canted 5 degrees outward + const float half = (eye == 0 ? 2.5f : -2.5f) * 3.14159265f / 180.0f; + config.eye_rotation[eye] = {0, std::sin(half), 0, std::cos(half)}; + } + } + model = lighthouseModel(config, 2); + width = static_cast(config.eye_width * 2), height = static_cast(config.eye_height); + } else { + std::fprintf(stderr, "--panel-check: %s is not a headset drawn on this Mac's display\n", key.c_str()); + return 2; + } + std::vector bgra; + if (!drawTestPanel(model, width, height, bgra)) { + std::fprintf(stderr, "--panel-check: drawing failed\n"); + return 1; + } + int failures = 0; + for (int eye = 0; eye < 2; ++eye) { + const uint32_t x0 = model.portrait_eyes ? 0 : eye * width / 2, y0 = model.portrait_eyes ? eye * height / 2 : 0; + const uint32_t w = model.portrait_eyes ? width : width / 2, h = model.portrait_eyes ? height / 2 : height; + double sx = 0, sy = 0; + uint64_t mark = 0, lit = 0; + for (uint32_t y = y0; y < y0 + h; ++y) + for (uint32_t x = x0; x < x0 + w; ++x) { + const uint8_t *p = &bgra[(static_cast(y) * width + x) * 4]; + const uint8_t b = p[0], g = p[1], r = p[2]; + lit += r > 8 || g > 8 || b > 8; + if (r > 230 && g < 110 && b < 110) { + ++mark; + sx += x - x0; + sy += y - y0; + } + } + const double fx = mark ? sx / mark / w : -1, fy = mark ? sy / mark / h : -1; + const bool ok = mark >= 20 && fx > 0.1 && fx < 0.9 && fy > 0.1 && fy < 0.9 && lit > static_cast(w) * h / 4; + std::printf("%s eye: straight ahead at %.2f, %.2f of its part of the panel (%llu pixels), %.0f%% drawn: %s\n", + eye == 0 ? "left" : "right", fx, fy, static_cast(mark), + 100.0 * lit / (static_cast(w) * h), ok ? "ok" : "FAIL"); + failures += !ok; + } + if (!png.empty()) { + CGColorSpaceRef space = CGColorSpaceCreateWithName(kCGColorSpaceSRGB); + CGContextRef context = CGBitmapContextCreate(bgra.data(), width, height, 8, width * 4, space, + static_cast(kCGImageAlphaNoneSkipFirst) | + static_cast(kCGBitmapByteOrder32Little)); + CGImageRef image = context ? CGBitmapContextCreateImage(context) : nullptr; + CGImageDestinationRef out = image ? CGImageDestinationCreateWithURL( + (__bridge CFURLRef)[NSURL fileURLWithPath:@(png.c_str())], + CFSTR("public.png"), 1, nullptr) + : nullptr; + if (out) { + CGImageDestinationAddImage(out, image, nullptr); + CGImageDestinationFinalize(out); + CFRelease(out); + } + CGImageRelease(image); + if (context) + CGContextRelease(context); + CGColorSpaceRelease(space); + } + std::printf("%s: %s panel %ux%u\n", failures ? "FAILED" : "PASSED", key.c_str(), width, height); + return failures ? 1 : 0; +} + +// --foveation-check: each eye an image whose colour is its own position (red across, green down), composed and +// squeezed for ALVR's foveated encoding on the GPU; every pixel of the squeezed frame must show the point of the +// eye the CPU's port of ALVR's shader says (foveationSource), which ALVR's client's stretch undoes +// (tests/headset_protocol_check.cpp). +int foveationCheck() { + id device = MTLCreateSystemDefaultDevice(); + const uint32_t eye_w = 1024, eye_h = 1024; + const Foveation f = computeFoveation(eye_w, eye_h, 0.45f, 0.4f, 0.4f, 0.1f, 4.0f, 5.0f); + Compositor compositor(device); + const uint32_t out_w = f.optimized_eye_width * 2, out_h = f.optimized_eye_height; + if (!device || !compositor.configure(out_w, out_h, f)) { + std::fprintf(stderr, "--foveation-check: no compositor\n"); + return 1; + } + // linear-light float textures, so the values come back as written + MTLTextureDescriptor *td = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRGBA32Float + width:eye_w + height:eye_h + mipmapped:NO]; + td.storageMode = MTLStorageModeShared; + ComposeFrame frame; + ComposeLayer layer; + layer.type = MVR_LAYER_PROJECTION; + std::vector pixels(static_cast(eye_w) * eye_h * 4); + for (int eye = 0; eye < 2; ++eye) { + for (uint32_t y = 0; y < eye_h; ++y) + for (uint32_t x = 0; x < eye_w; ++x) { + float *p = &pixels[(static_cast(y) * eye_w + x) * 4]; + p[0] = (x + 0.5f) / eye_w; + p[1] = (y + 0.5f) / eye_h; + p[2] = eye ? 1.0f : 0.0f; + p[3] = 1; + } + id t = [device newTextureWithDescriptor:td]; + [t replaceRegion:MTLRegionMake2D(0, 0, eye_w, eye_h) mipmapLevel:0 withBytes:pixels.data() bytesPerRow:eye_w * 16]; + layer.views[eye].texture = t; + layer.views[eye].rect = {0, 0, static_cast(eye_w), static_cast(eye_h)}; + } + frame.layers.push_back(layer); + CVPixelBufferRef buffer = compositor.compose(frame); + if (!buffer) { + std::fprintf(stderr, "--foveation-check: compose failed\n"); + return 1; + } + CVPixelBufferLockBaseAddress(buffer, kCVPixelBufferLock_ReadOnly); + const uint8_t *base = static_cast(CVPixelBufferGetBaseAddress(buffer)); + const size_t stride = CVPixelBufferGetBytesPerRow(buffer); + // the frame is 8-bit sRGB: back to linear, as the position was written + const auto linear = [](uint8_t v) { + const double c = v / 255.0; + return c <= 0.04045 ? c / 12.92 : std::pow((c + 0.055) / 1.055, 2.4); + }; + double worst = 0; + uint64_t wrong_eye = 0, counted = 0; + for (uint32_t y = 2; y + 2 < out_h; y += 7) + for (uint32_t x = 2; x + 2 < out_w; x += 7) { + if (x + 3 > out_w / 2 && x < out_w / 2 + 3) + continue; // the seam between the eyes + const uint8_t *p = base + y * stride + x * 4; // BGRA + const float uv[2] = {(x + 0.5f) / out_w, (y + 0.5f) / out_h}; + float source[2]; + foveationSource(f, uv, source); + const int eye = uv[0] > 0.5f ? 1 : 0; + const double expected_x = source[0] * 2.0 - eye, expected_y = source[1]; + if (expected_x < 0.01 || expected_x > 0.99 || expected_y < 0.01 || expected_y > 0.99) + continue; // the squeeze reads past the eye there: clamped at its edge + wrong_eye += (linear(p[0]) > 0.5) != (eye == 1); + const double dx = std::fabs(linear(p[2]) - expected_x), dy = std::fabs(linear(p[1]) - expected_y); + // 8 bits of sRGB are about 1/400 at the bright end + worst = std::max(worst, std::max(dx, dy)); + ++counted; + } + CVPixelBufferUnlockBaseAddress(buffer, kCVPixelBufferLock_ReadOnly); + CVPixelBufferRelease(buffer); + const bool ok = counted > 1000 && wrong_eye == 0 && worst < 0.01; + std::printf("%s: %ux%u per eye squeezed to %ux%u; %llu pixels, worst %.4f off, %llu from the wrong eye\n", + ok ? "PASSED" : "FAILED", eye_w, eye_h, f.optimized_eye_width, f.optimized_eye_height, + static_cast(counted), worst, static_cast(wrong_eye)); + return ok ? 0 : 1; +} + +// --passthrough-check: an Index's home drawn with a stereo camera picture (left half red, right half blue); the +// middle of each eye must show its own half +int passthroughCheck() { + lighthouse::HeadsetConfig config; + config.model = lighthouse::Model::Index; + config.eye_width = 1440, config.eye_height = 1600; + for (int eye = 0; eye < 2; ++eye) { + config.distortion[eye].intrinsic_focus[0] = config.distortion[eye].intrinsic_focus[1] = 1.5f; + config.fov[eye] = {-0.785f, 0.785f, 0.785f, -0.785f}; + } + const DirectModel model = lighthouseModel(config, 2); + std::vector bgra; + if (!drawTestPanel(model, 2880, 1600, bgra, true)) { + std::printf("FAILED: drawing\n"); + return 1; + } + const auto at = [&](uint32_t x, uint32_t y) { return &bgra[(static_cast(y) * 2880 + x) * 4]; }; + const uint8_t *left = at(720, 800), *right = at(2160, 800); + const bool ok = left[2] > 200 && left[0] < 50 && right[0] > 200 && right[2] < 50; + std::printf("%s: left eye shows (%u %u %u), right eye (%u %u %u)\n", ok ? "PASSED" : "FAILED", left[2], left[1], + left[0], right[2], right[1], right[0]); + return ok ? 0 : 1; +} + +// --survive-check: libsurvive started in this process for a few seconds, as the Lighthouse driver starts it, and +// stopped again; what it tracks (nothing without a headset and base stations) is listed +int surviveCheck(bool simulated) { + if (!lighthouseTrackingBuilt()) { + std::printf("metalvrd was built without libsurvive (scripts/build-libsurvive.sh)\n"); + return 2; + } + auto tracking = startLighthouseTracking(simulated); + if (!tracking) { + std::printf("FAILED: libsurvive did not start\n"); + return 1; + } + std::this_thread::sleep_for(std::chrono::seconds(simulated ? 8 : 4)); + // the simulator's device (SM0) is posed, and its base stations stand where it puts them: 1 m up, which only + // comes out so when libsurvive's +z up is turned into OpenXR's +y up + bool device_posed = false; + int stations_placed = 0; + for (const auto &[name, p] : tracking->objects()) { + std::printf("%s at %.3f %.3f %.3f m%s\n", name.c_str(), p.position.x, p.position.y, p.position.z, + p.position_tracked ? ", tracked" : ""); + device_posed = device_posed || name == "SM0"; + stations_placed += name.rfind("LH", 0) == 0 && std::fabs(p.position.y - 1.0f) < 0.05f; + } + mvr_tracking_reference references[MVR_MAX_TRACKING_REFERENCES]; + const uint32_t stations = tracking->references(references); + static const char *const names[3] = {"head", "left hand", "right hand"}; + for (int device = 0; device < 3; ++device) { + TrackedPose p; + const bool has = tracking->pose(device, p); + std::printf("%s: %s\n", names[device], + has ? (p.position_tracked ? "tracked" : "known, not seen lately") : "not there"); + } + std::printf("base stations: %u\n", stations); + tracking.reset(); + if (simulated && (!device_posed || stations_placed < 2 || stations < 2)) { + std::printf("FAILED: libsurvive's simulated device and its two base stations were not all placed\n"); + return 1; + } + std::printf("PASSED: libsurvive started%s and stopped\n", simulated ? ", tracked its simulated device," : ""); + return 0; +} + } // namespace int main(int argc, char **argv) { @@ -96,6 +374,17 @@ int main(int argc, char **argv) { alvr.config_dir = value(); else if (arg == "--trust") alvr.trusted_clients.push_back(value()); + else if (arg == "--headset") { + const uint32_t headset = headsetFromKey(value()); + if (headset == MVR_HEADSET_UNKNOWN) { + std::fprintf(stderr, "--headset: %s is not a headset MetalVR knows (docs/HEADSETS.md)\n", argv[i]); + return 2; + } + alvr.headset = sim.headset = headset; + } else if (arg == "--adb") + alvr.adb = value(); + else if (arg == "--foveated-encoding") + alvr.foveated_encoding = sim.foveated_encoding = true; else if (arg == "--verbose") g_verbose = true; else if (arg == "--eye") { @@ -112,6 +401,17 @@ int main(int argc, char **argv) { rift.home_logo = value(); else if (arg == "--ipd") rift.ipd = static_cast(std::atof(value())) / 1000.0f; + else if (arg == "--passthrough") + rift.passthrough = true; + else if (arg == "--refresh") + rift.refresh_hz = static_cast(std::atof(value())); + else if (arg == "--pro2-mode") { + rift.pro2_mode = std::atoi(value()); + if (rift.pro2_mode < 0 || rift.pro2_mode > 5) { + std::fprintf(stderr, "--pro2-mode wants 0 to 5\n"); + return 2; + } + } else if (arg == "--fps") sim.refresh_hz = static_cast(std::atof(value())); else if (arg == "--codec") { @@ -134,7 +434,16 @@ int main(int argc, char **argv) { server_options.snapshot_path = value(); else if (arg == "--snapshot-frame") server_options.snapshot_frame = static_cast(std::atoi(value())); - else if (arg == "--exit-after-session") + else if (arg == "--passthrough-check") + return passthroughCheck(); + else if (arg == "--survive-check") + return surviveCheck(i + 1 < argc && std::string(argv[i + 1]) == "sim"); + else if (arg == "--foveation-check") + return foveationCheck(); + else if (arg == "--panel-check") { + const std::string key = value(); + return panelCheck(key, value()); + } else if (arg == "--exit-after-session") server_options.exit_after_session = true; else if (arg == "--software-encoder") server_options.software_encoder = true; @@ -144,11 +453,28 @@ int main(int argc, char **argv) { } } + // a streamed headset is named for the ALVR driver; one on this Mac's own display is found by its driver + if (alvr.headset != MVR_HEADSET_UNKNOWN && (driver_name.empty() || driver_name == "alvr")) { + const uint32_t h = alvr.headset; + const bool streamed = h == MVR_HEADSET_QUEST_1 || h == MVR_HEADSET_QUEST_2 || h == MVR_HEADSET_QUEST_3 || + h == MVR_HEADSET_QUEST_3S || h == MVR_HEADSET_QUEST_PRO || h == MVR_HEADSET_STEAM_FRAME; + if (!streamed) { + std::fprintf(stderr, "--headset %s is not streamed: use --driver %s, which finds it on USB\n", + mvr_describe_headset(h)->key, + h == MVR_HEADSET_RIFT_CV1 || h == MVR_HEADSET_RIFT_S ? "rift" + : h == MVR_HEADSET_PSVR2 ? "psvr2" + : "lighthouse"); + return 2; + } + } + std::unique_ptr driver; - if (driver_name == "rift") { + if (driver_name == "rift" || driver_name == "lighthouse" || driver_name == "psvr2") { + rift.lighthouse = driver_name == "lighthouse"; + rift.psvr2 = driver_name == "psvr2"; driver = makeRiftDriver(rift); if (!driver) { - std::fprintf(stderr, "no Rift CV1 driver\n"); + std::fprintf(stderr, "no %s driver\n", driver_name.c_str()); return 1; } } else if (driver_name.empty() || driver_name == "alvr") { diff --git a/daemon/psvr2_lens_table.h b/daemon/psvr2_lens_table.h new file mode 100644 index 0000000..cd8d14b --- /dev/null +++ b/daemon/psvr2_lens_table.h @@ -0,0 +1,354 @@ +// The PlayStation VR2 lens's magnification for red, green and blue over the squared radius, 1024 steps, from +// Monado's drivers/psvr2/psvr2_distortion.c (Copyright 2024 Joel Valenciano, 2025 Beyley Cardellio; BSL-1.0), +// copied by value. Used by psvr2::computeDistortion and, as a Metal constant, by the direct-display driver's shader. +#pragma once + +namespace metalvr::psvr2 { + +constexpr int kLensTableSize = 1024; +constexpr float kLensTable[kLensTableSize][3] = { + {0.856803, 0.863602, 0.870955}, {0.857020, 0.863827, 0.871189}, {0.857236, 0.864052, 0.871423}, + {0.857453, 0.864277, 0.871657}, {0.857670, 0.864503, 0.871892}, {0.857888, 0.864728, 0.872127}, + {0.858106, 0.864954, 0.872362}, {0.858324, 0.865181, 0.872598}, {0.858542, 0.865407, 0.872834}, + {0.858760, 0.865634, 0.873070}, {0.858979, 0.865861, 0.873307}, {0.859198, 0.866088, 0.873543}, + {0.859417, 0.866316, 0.873780}, {0.859636, 0.866544, 0.874018}, {0.859856, 0.866772, 0.874255}, + {0.860076, 0.867000, 0.874493}, {0.860296, 0.867229, 0.874731}, {0.860516, 0.867458, 0.874970}, + {0.860737, 0.867687, 0.875208}, {0.860958, 0.867916, 0.875447}, {0.861179, 0.868146, 0.875686}, + {0.861400, 0.868376, 0.875926}, {0.861622, 0.868606, 0.876166}, {0.861844, 0.868837, 0.876406}, + {0.862066, 0.869068, 0.876646}, {0.862288, 0.869299, 0.876887}, {0.862511, 0.869530, 0.877128}, + {0.862734, 0.869761, 0.877369}, {0.862957, 0.869993, 0.877611}, {0.863180, 0.870225, 0.877852}, + {0.863404, 0.870458, 0.878095}, {0.863627, 0.870690, 0.878337}, {0.863852, 0.870923, 0.878580}, + {0.864076, 0.871156, 0.878823}, {0.864301, 0.871390, 0.879066}, {0.864525, 0.871623, 0.879309}, + {0.864751, 0.871857, 0.879553}, {0.864976, 0.872091, 0.879797}, {0.865201, 0.872326, 0.880042}, + {0.865427, 0.872561, 0.880287}, {0.865653, 0.872796, 0.880532}, {0.865880, 0.873031, 0.880777}, + {0.866106, 0.873267, 0.881022}, {0.866333, 0.873502, 0.881268}, {0.866560, 0.873739, 0.881514}, + {0.866788, 0.873975, 0.881761}, {0.867015, 0.874212, 0.882008}, {0.867243, 0.874448, 0.882255}, + {0.867471, 0.874686, 0.882502}, {0.867700, 0.874923, 0.882750}, {0.867929, 0.875161, 0.882998}, + {0.868157, 0.875399, 0.883246}, {0.868387, 0.875637, 0.883494}, {0.868616, 0.875876, 0.883743}, + {0.868846, 0.876115, 0.883992}, {0.869076, 0.876354, 0.884242}, {0.869306, 0.876593, 0.884492}, + {0.869536, 0.876833, 0.884742}, {0.869767, 0.877073, 0.884992}, {0.869998, 0.877313, 0.885243}, + {0.870229, 0.877554, 0.885493}, {0.870461, 0.877795, 0.885745}, {0.870692, 0.878036, 0.885996}, + {0.870925, 0.878277, 0.886248}, {0.871157, 0.878519, 0.886500}, {0.871389, 0.878761, 0.886753}, + {0.871622, 0.879003, 0.887005}, {0.871855, 0.879245, 0.887258}, {0.872089, 0.879488, 0.887512}, + {0.872322, 0.879731, 0.887765}, {0.872556, 0.879974, 0.888019}, {0.872790, 0.880218, 0.888274}, + {0.873025, 0.880462, 0.888528}, {0.873259, 0.880706, 0.888783}, {0.873494, 0.880951, 0.889038}, + {0.873729, 0.881195, 0.889294}, {0.873965, 0.881440, 0.889550}, {0.874200, 0.881686, 0.889806}, + {0.874436, 0.881931, 0.890062}, {0.874673, 0.882177, 0.890319}, {0.874909, 0.882424, 0.890576}, + {0.875146, 0.882670, 0.890833}, {0.875383, 0.882917, 0.891091}, {0.875620, 0.883164, 0.891349}, + {0.875858, 0.883411, 0.891608}, {0.876096, 0.883659, 0.891866}, {0.876334, 0.883907, 0.892125}, + {0.876572, 0.884155, 0.892384}, {0.876811, 0.884404, 0.892644}, {0.877050, 0.884652, 0.892904}, + {0.877289, 0.884902, 0.893164}, {0.877529, 0.885151, 0.893425}, {0.877769, 0.885401, 0.893686}, + {0.878009, 0.885651, 0.893947}, {0.878249, 0.885901, 0.894208}, {0.878489, 0.886151, 0.894470}, + {0.878730, 0.886402, 0.894732}, {0.878972, 0.886654, 0.894995}, {0.879213, 0.886905, 0.895257}, + {0.879455, 0.887157, 0.895521}, {0.879697, 0.887409, 0.895784}, {0.879939, 0.887661, 0.896048}, + {0.880181, 0.887914, 0.896312}, {0.880424, 0.888167, 0.896576}, {0.880667, 0.888420, 0.896841}, + {0.880911, 0.888674, 0.897106}, {0.881154, 0.888928, 0.897371}, {0.881398, 0.889182, 0.897637}, + {0.881642, 0.889436, 0.897903}, {0.881887, 0.889691, 0.898170}, {0.882132, 0.889946, 0.898436}, + {0.882377, 0.890202, 0.898703}, {0.882622, 0.890457, 0.898971}, {0.882868, 0.890713, 0.899238}, + {0.883114, 0.890970, 0.899507}, {0.883360, 0.891226, 0.899775}, {0.883606, 0.891483, 0.900044}, + {0.883853, 0.891740, 0.900313}, {0.884100, 0.891998, 0.900582}, {0.884347, 0.892256, 0.900852}, + {0.884595, 0.892514, 0.901122}, {0.884843, 0.892772, 0.901392}, {0.885091, 0.893031, 0.901663}, + {0.885339, 0.893290, 0.901934}, {0.885588, 0.893550, 0.902205}, {0.885837, 0.893810, 0.902477}, + {0.886087, 0.894070, 0.902749}, {0.886336, 0.894330, 0.903022}, {0.886586, 0.894591, 0.903295}, + {0.886836, 0.894852, 0.903568}, {0.887087, 0.895113, 0.903841}, {0.887338, 0.895375, 0.904115}, + {0.887589, 0.895636, 0.904389}, {0.887840, 0.895899, 0.904664}, {0.888092, 0.896161, 0.904939}, + {0.888344, 0.896424, 0.905214}, {0.888596, 0.896687, 0.905489}, {0.888849, 0.896951, 0.905765}, + {0.889102, 0.897215, 0.906042}, {0.889355, 0.897479, 0.906318}, {0.889608, 0.897743, 0.906595}, + {0.889862, 0.898008, 0.906872}, {0.890116, 0.898273, 0.907150}, {0.890370, 0.898539, 0.907428}, + {0.890625, 0.898805, 0.907706}, {0.890880, 0.899071, 0.907985}, {0.891135, 0.899337, 0.908264}, + {0.891391, 0.899604, 0.908544}, {0.891647, 0.899871, 0.908824}, {0.891903, 0.900138, 0.909104}, + {0.892159, 0.900406, 0.909384}, {0.892416, 0.900674, 0.909665}, {0.892673, 0.900943, 0.909946}, + {0.892930, 0.901212, 0.910228}, {0.893188, 0.901481, 0.910510}, {0.893446, 0.901750, 0.910792}, + {0.893704, 0.902020, 0.911075}, {0.893963, 0.902290, 0.911358}, {0.894222, 0.902560, 0.911642}, + {0.894481, 0.902831, 0.911925}, {0.894740, 0.903102, 0.912210}, {0.895000, 0.903373, 0.912494}, + {0.895260, 0.903645, 0.912779}, {0.895521, 0.903917, 0.913064}, {0.895782, 0.904190, 0.913350}, + {0.896043, 0.904463, 0.913636}, {0.896304, 0.904736, 0.913922}, {0.896566, 0.905009, 0.914209}, + {0.896828, 0.905283, 0.914496}, {0.897090, 0.905557, 0.914784}, {0.897353, 0.905832, 0.915072}, + {0.897616, 0.906106, 0.915360}, {0.897879, 0.906381, 0.915649}, {0.898143, 0.906657, 0.915938}, + {0.898406, 0.906933, 0.916227}, {0.898671, 0.907209, 0.916517}, {0.898935, 0.907486, 0.916807}, + {0.899200, 0.907763, 0.917097}, {0.899465, 0.908040, 0.917388}, {0.899731, 0.908317, 0.917680}, + {0.899997, 0.908595, 0.917971}, {0.900263, 0.908874, 0.918263}, {0.900529, 0.909152, 0.918556}, + {0.900796, 0.909431, 0.918849}, {0.901063, 0.909711, 0.919142}, {0.901331, 0.909990, 0.919436}, + {0.901598, 0.910270, 0.919730}, {0.901866, 0.910551, 0.920024}, {0.902135, 0.910832, 0.920319}, + {0.902404, 0.911113, 0.920614}, {0.902673, 0.911394, 0.920910}, {0.902942, 0.911676, 0.921206}, + {0.903212, 0.911958, 0.921502}, {0.903482, 0.912241, 0.921799}, {0.903752, 0.912524, 0.922096}, + {0.904023, 0.912807, 0.922394}, {0.904294, 0.913091, 0.922692}, {0.904565, 0.913375, 0.922990}, + {0.904837, 0.913659, 0.923289}, {0.905109, 0.913944, 0.923588}, {0.905382, 0.914229, 0.923888}, + {0.905654, 0.914514, 0.924188}, {0.905927, 0.914800, 0.924488}, {0.906201, 0.915087, 0.924789}, + {0.906474, 0.915373, 0.925090}, {0.906748, 0.915660, 0.925392}, {0.907023, 0.915947, 0.925694}, + {0.907298, 0.916235, 0.925996}, {0.907573, 0.916523, 0.926299}, {0.907848, 0.916812, 0.926602}, + {0.908124, 0.917100, 0.926906}, {0.908400, 0.917390, 0.927210}, {0.908677, 0.917679, 0.927515}, + {0.908953, 0.917969, 0.927820}, {0.909230, 0.918260, 0.928125}, {0.909508, 0.918550, 0.928431}, + {0.909786, 0.918841, 0.928737}, {0.910064, 0.919133, 0.929043}, {0.910343, 0.919425, 0.929350}, + {0.910621, 0.919717, 0.929658}, {0.910901, 0.920009, 0.929966}, {0.911180, 0.920302, 0.930274}, + {0.911460, 0.920596, 0.930583}, {0.911740, 0.920890, 0.930892}, {0.912021, 0.921184, 0.931202}, + {0.912302, 0.921478, 0.931512}, {0.912583, 0.921773, 0.931822}, {0.912865, 0.922069, 0.932133}, + {0.913147, 0.922364, 0.932444}, {0.913430, 0.922660, 0.932756}, {0.913712, 0.922957, 0.933068}, + {0.913995, 0.923254, 0.933381}, {0.914279, 0.923551, 0.933694}, {0.914563, 0.923849, 0.934007}, + {0.914847, 0.924147, 0.934321}, {0.915132, 0.924445, 0.934636}, {0.915416, 0.924744, 0.934950}, + {0.915702, 0.925043, 0.935266}, {0.915987, 0.925343, 0.935581}, {0.916273, 0.925643, 0.935897}, + {0.916560, 0.925944, 0.936214}, {0.916847, 0.926244, 0.936531}, {0.917134, 0.926546, 0.936848}, + {0.917421, 0.926847, 0.937166}, {0.917709, 0.927149, 0.937485}, {0.917997, 0.927452, 0.937804}, + {0.918286, 0.927755, 0.938123}, {0.918575, 0.928058, 0.938443}, {0.918864, 0.928362, 0.938763}, + {0.919154, 0.928666, 0.939083}, {0.919444, 0.928970, 0.939405}, {0.919734, 0.929275, 0.939726}, + {0.920025, 0.929581, 0.940048}, {0.920316, 0.929887, 0.940371}, {0.920608, 0.930193, 0.940693}, + {0.920900, 0.930499, 0.941015}, {0.921192, 0.930806, 0.941333}, {0.921485, 0.931114, 0.941648}, + {0.921778, 0.931422, 0.941962}, {0.922071, 0.931730, 0.942278}, {0.922365, 0.932039, 0.942594}, + {0.922659, 0.932348, 0.942910}, {0.922954, 0.932657, 0.943227}, {0.923249, 0.932967, 0.943545}, + {0.923545, 0.933278, 0.943863}, {0.923843, 0.933589, 0.944181}, {0.924146, 0.933900, 0.944500}, + {0.924451, 0.934211, 0.944819}, {0.924757, 0.934524, 0.945139}, {0.925063, 0.934836, 0.945459}, + {0.925368, 0.935149, 0.945779}, {0.925675, 0.935462, 0.946100}, {0.925982, 0.935776, 0.946422}, + {0.926289, 0.936091, 0.946744}, {0.926597, 0.936405, 0.947066}, {0.926905, 0.936720, 0.947389}, + {0.927214, 0.937036, 0.947712}, {0.927523, 0.937352, 0.948036}, {0.927832, 0.937669, 0.948361}, + {0.928142, 0.937985, 0.948685}, {0.928452, 0.938303, 0.949011}, {0.928763, 0.938621, 0.949336}, + {0.929074, 0.938939, 0.949663}, {0.929385, 0.939257, 0.949989}, {0.929697, 0.939577, 0.950316}, + {0.930009, 0.939896, 0.950644}, {0.930322, 0.940216, 0.950972}, {0.930635, 0.940537, 0.951301}, + {0.930949, 0.940858, 0.951630}, {0.931263, 0.941179, 0.951959}, {0.931577, 0.941501, 0.952289}, + {0.931892, 0.941823, 0.952620}, {0.932208, 0.942146, 0.952951}, {0.932523, 0.942469, 0.953283}, + {0.932840, 0.942793, 0.953615}, {0.933156, 0.943117, 0.953947}, {0.933473, 0.943442, 0.954280}, + {0.933791, 0.943767, 0.954613}, {0.934109, 0.944092, 0.954947}, {0.934427, 0.944418, 0.955282}, + {0.934746, 0.944745, 0.955617}, {0.935065, 0.945072, 0.955952}, {0.935385, 0.945399, 0.956288}, + {0.935705, 0.945727, 0.956625}, {0.936025, 0.946055, 0.956962}, {0.936346, 0.946384, 0.957299}, + {0.936668, 0.946713, 0.957637}, {0.936990, 0.947043, 0.957976}, {0.937312, 0.947373, 0.958315}, + {0.937635, 0.947704, 0.958654}, {0.937958, 0.948035, 0.958994}, {0.938282, 0.948367, 0.959335}, + {0.938606, 0.948699, 0.959676}, {0.938931, 0.949032, 0.960017}, {0.939256, 0.949365, 0.960359}, + {0.939581, 0.949698, 0.960702}, {0.939907, 0.950032, 0.961045}, {0.940234, 0.950367, 0.961388}, + {0.940561, 0.950702, 0.961733}, {0.940888, 0.951038, 0.962077}, {0.941216, 0.951374, 0.962422}, + {0.941544, 0.951710, 0.962768}, {0.941873, 0.952047, 0.963114}, {0.942202, 0.952385, 0.963461}, + {0.942532, 0.952723, 0.963808}, {0.942862, 0.953061, 0.964156}, {0.943193, 0.953400, 0.964505}, + {0.943524, 0.953740, 0.964854}, {0.943855, 0.954080, 0.965203}, {0.944187, 0.954420, 0.965553}, + {0.944520, 0.954761, 0.965903}, {0.944853, 0.955103, 0.966254}, {0.945186, 0.955445, 0.966606}, + {0.945520, 0.955788, 0.966958}, {0.945855, 0.956131, 0.967311}, {0.946190, 0.956474, 0.967664}, + {0.946525, 0.956818, 0.968018}, {0.946861, 0.957163, 0.968372}, {0.947198, 0.957508, 0.968727}, + {0.947535, 0.957854, 0.969083}, {0.947872, 0.958200, 0.969438}, {0.948210, 0.958547, 0.969795}, + {0.948548, 0.958894, 0.970152}, {0.948887, 0.959242, 0.970510}, {0.949226, 0.959590, 0.970868}, + {0.949566, 0.959939, 0.971227}, {0.949906, 0.960288, 0.971586}, {0.950247, 0.960638, 0.971946}, + {0.950589, 0.960988, 0.972306}, {0.950930, 0.961339, 0.972668}, {0.951273, 0.961691, 0.973029}, + {0.951616, 0.962043, 0.973391}, {0.951959, 0.962395, 0.973754}, {0.952303, 0.962748, 0.974117}, + {0.952647, 0.963102, 0.974481}, {0.952992, 0.963456, 0.974846}, {0.953338, 0.963811, 0.975211}, + {0.953683, 0.964166, 0.975577}, {0.954030, 0.964522, 0.975943}, {0.954377, 0.964878, 0.976310}, + {0.954724, 0.965235, 0.976677}, {0.955072, 0.965593, 0.977045}, {0.955421, 0.965951, 0.977414}, + {0.955770, 0.966309, 0.977783}, {0.956119, 0.966668, 0.978153}, {0.956469, 0.967028, 0.978523}, + {0.956820, 0.967388, 0.978894}, {0.957171, 0.967749, 0.979266}, {0.957523, 0.968110, 0.979638}, + {0.957875, 0.968472, 0.980010}, {0.958228, 0.968835, 0.980384}, {0.958581, 0.969198, 0.980758}, + {0.958935, 0.969561, 0.981132}, {0.959289, 0.969926, 0.981508}, {0.959644, 0.970290, 0.981883}, + {0.959999, 0.970656, 0.982260}, {0.960355, 0.971022, 0.982637}, {0.960712, 0.971388, 0.983015}, + {0.961069, 0.971755, 0.983393}, {0.961426, 0.972123, 0.983772}, {0.961785, 0.972491, 0.984151}, + {0.962143, 0.972860, 0.984531}, {0.962503, 0.973229, 0.984912}, {0.962862, 0.973599, 0.985294}, + {0.963223, 0.973970, 0.985676}, {0.963584, 0.974341, 0.986058}, {0.963945, 0.974713, 0.986442}, + {0.964307, 0.975085, 0.986826}, {0.964670, 0.975458, 0.987210}, {0.965033, 0.975832, 0.987595}, + {0.965397, 0.976206, 0.987981}, {0.965761, 0.976581, 0.988368}, {0.966126, 0.976956, 0.988755}, + {0.966491, 0.977332, 0.989143}, {0.966857, 0.977709, 0.989531}, {0.967224, 0.978086, 0.989920}, + {0.967591, 0.978464, 0.990310}, {0.967959, 0.978842, 0.990701}, {0.968327, 0.979221, 0.991092}, + {0.968696, 0.979601, 0.991483}, {0.969065, 0.979981, 0.991876}, {0.969435, 0.980362, 0.992269}, + {0.969806, 0.980744, 0.992663}, {0.970177, 0.981126, 0.993057}, {0.970549, 0.981509, 0.993452}, + {0.970921, 0.981892, 0.993848}, {0.971294, 0.982276, 0.994244}, {0.971668, 0.982661, 0.994641}, + {0.972042, 0.983046, 0.995039}, {0.972417, 0.983432, 0.995438}, {0.972792, 0.983819, 0.995837}, + {0.973168, 0.984206, 0.996237}, {0.973545, 0.984594, 0.996637}, {0.973922, 0.984982, 0.997038}, + {0.974300, 0.985371, 0.997440}, {0.974678, 0.985761, 0.997843}, {0.975057, 0.986152, 0.998246}, + {0.975437, 0.986543, 0.998650}, {0.975817, 0.986935, 0.999055}, {0.976198, 0.987327, 0.999460}, + {0.976579, 0.987720, 0.999867}, {0.976962, 0.988114, 1.000273}, {0.977344, 0.988508, 1.000681}, + {0.977728, 0.988904, 1.001089}, {0.978112, 0.989299, 1.001498}, {0.978496, 0.989696, 1.001908}, + {0.978882, 0.990093, 1.002318}, {0.979267, 0.990491, 1.002729}, {0.979654, 0.990889, 1.003141}, + {0.980041, 0.991288, 1.003554}, {0.980429, 0.991688, 1.003967}, {0.980817, 0.992089, 1.004381}, + {0.981206, 0.992490, 1.004796}, {0.981596, 0.992892, 1.005212}, {0.981987, 0.993294, 1.005628}, + {0.982378, 0.993698, 1.006045}, {0.982769, 0.994102, 1.006463}, {0.983162, 0.994506, 1.006881}, + {0.983555, 0.994911, 1.007300}, {0.983948, 0.995318, 1.007720}, {0.984343, 0.995724, 1.008141}, + {0.984738, 0.996132, 1.008563}, {0.985133, 0.996540, 1.008985}, {0.985529, 0.996949, 1.009408}, + {0.985926, 0.997359, 1.009832}, {0.986324, 0.997769, 1.010256}, {0.986722, 0.998180, 1.010682}, + {0.987121, 0.998592, 1.011108}, {0.987521, 0.999004, 1.011535}, {0.987921, 0.999417, 1.011963}, + {0.988322, 0.999831, 1.012391}, {0.988724, 1.000246, 1.012820}, {0.989126, 1.000661, 1.013250}, + {0.989530, 1.001078, 1.013681}, {0.989933, 1.001494, 1.014113}, {0.990338, 1.001912, 1.014545}, + {0.990743, 1.002330, 1.014978}, {0.991149, 1.002750, 1.015412}, {0.991555, 1.003169, 1.015847}, + {0.991963, 1.003590, 1.016283}, {0.992371, 1.004011, 1.016719}, {0.992779, 1.004434, 1.017156}, + {0.993189, 1.004856, 1.017594}, {0.993599, 1.005280, 1.018033}, {0.994010, 1.005704, 1.018473}, + {0.994421, 1.006130, 1.018914}, {0.994833, 1.006555, 1.019355}, {0.995246, 1.006982, 1.019797}, + {0.995660, 1.007410, 1.020240}, {0.996074, 1.007838, 1.020684}, {0.996490, 1.008267, 1.021129}, + {0.996905, 1.008697, 1.021574}, {0.997322, 1.009127, 1.022021}, {0.997739, 1.009559, 1.022468}, + {0.998158, 1.009991, 1.022916}, {0.998576, 1.010424, 1.023365}, {0.998996, 1.010858, 1.023815}, + {0.999416, 1.011292, 1.024266}, {0.999837, 1.011728, 1.024716}, {1.000259, 1.012164, 1.025167}, + {1.000682, 1.012601, 1.025621}, {1.001105, 1.013039, 1.026081}, {1.001529, 1.013477, 1.026544}, + {1.001954, 1.013917, 1.027011}, {1.002380, 1.014357, 1.027479}, {1.002806, 1.014798, 1.027948}, + {1.003233, 1.015240, 1.028418}, {1.003661, 1.015683, 1.028888}, {1.004090, 1.016126, 1.029360}, + {1.004519, 1.016571, 1.029833}, {1.004950, 1.017016, 1.030306}, {1.005381, 1.017462, 1.030781}, + {1.005813, 1.017909, 1.031256}, {1.006245, 1.018357, 1.031733}, {1.006678, 1.018805, 1.032210}, + {1.007113, 1.019255, 1.032688}, {1.007549, 1.019705, 1.033168}, {1.007984, 1.020156, 1.033648}, + {1.008417, 1.020608, 1.034130}, {1.008846, 1.021061, 1.034612}, {1.009275, 1.021515, 1.035095}, + {1.009705, 1.021970, 1.035579}, {1.010135, 1.022425, 1.036065}, {1.010566, 1.022882, 1.036551}, + {1.010998, 1.023339, 1.037038}, {1.011431, 1.023797, 1.037527}, {1.011864, 1.024256, 1.038016}, + {1.012298, 1.024716, 1.038506}, {1.012733, 1.025177, 1.038998}, {1.013169, 1.025639, 1.039490}, + {1.013605, 1.026101, 1.039984}, {1.014043, 1.026565, 1.040478}, {1.014481, 1.027029, 1.040974}, + {1.014920, 1.027495, 1.041470}, {1.015360, 1.027961, 1.041968}, {1.015800, 1.028428, 1.042466}, + {1.016242, 1.028896, 1.042966}, {1.016684, 1.029365, 1.043467}, {1.017127, 1.029835, 1.043969}, + {1.017571, 1.030306, 1.044472}, {1.018015, 1.030778, 1.044976}, {1.018461, 1.031251, 1.045481}, + {1.018907, 1.031724, 1.045987}, {1.019354, 1.032199, 1.046494}, {1.019802, 1.032675, 1.047002}, + {1.020251, 1.033151, 1.047512}, {1.020701, 1.033629, 1.048022}, {1.021152, 1.034107, 1.048534}, + {1.021603, 1.034587, 1.049047}, {1.022055, 1.035067, 1.049561}, {1.022509, 1.035549, 1.050076}, + {1.022962, 1.036031, 1.050592}, {1.023417, 1.036514, 1.051109}, {1.023873, 1.036999, 1.051628}, + {1.024330, 1.037484, 1.052147}, {1.024787, 1.037970, 1.052668}, {1.025245, 1.038458, 1.053190}, + {1.025705, 1.038946, 1.053713}, {1.026165, 1.039436, 1.054237}, {1.026626, 1.039926, 1.054762}, + {1.027088, 1.040417, 1.055289}, {1.027550, 1.040910, 1.055816}, {1.028014, 1.041403, 1.056345}, + {1.028479, 1.041897, 1.056875}, {1.028944, 1.042393, 1.057407}, {1.029410, 1.042889, 1.057939}, + {1.029878, 1.043387, 1.058473}, {1.030346, 1.043885, 1.059008}, {1.030815, 1.044385, 1.059544}, + {1.031285, 1.044886, 1.060081}, {1.031756, 1.045387, 1.060619}, {1.032228, 1.045890, 1.061159}, + {1.032701, 1.046394, 1.061700}, {1.033175, 1.046899, 1.062243}, {1.033649, 1.047405, 1.062786}, + {1.034125, 1.047912, 1.063331}, {1.034601, 1.048420, 1.063877}, {1.035079, 1.048929, 1.064424}, + {1.035557, 1.049439, 1.064973}, {1.036037, 1.049951, 1.065522}, {1.036517, 1.050463, 1.066074}, + {1.036999, 1.050977, 1.066626}, {1.037481, 1.051492, 1.067180}, {1.037964, 1.052007, 1.067735}, + {1.038448, 1.052524, 1.068291}, {1.038934, 1.053042, 1.068849}, {1.039420, 1.053562, 1.069407}, + {1.039907, 1.054082, 1.069968}, {1.040395, 1.054603, 1.070529}, {1.040884, 1.055126, 1.071092}, + {1.041374, 1.055650, 1.071657}, {1.041865, 1.056175, 1.072222}, {1.042358, 1.056701, 1.072789}, + {1.042851, 1.057228, 1.073358}, {1.043345, 1.057756, 1.073928}, {1.043840, 1.058286, 1.074499}, + {1.044336, 1.058816, 1.075071}, {1.044833, 1.059348, 1.075645}, {1.045332, 1.059881, 1.076220}, + {1.045831, 1.060416, 1.076797}, {1.046331, 1.060951, 1.077375}, {1.046833, 1.061488, 1.077955}, + {1.047335, 1.062026, 1.078536}, {1.047838, 1.062565, 1.079118}, {1.048343, 1.063105, 1.079702}, + {1.048848, 1.063647, 1.080287}, {1.049355, 1.064189, 1.080874}, {1.049863, 1.064733, 1.081462}, + {1.050371, 1.065279, 1.082052}, {1.050881, 1.065825, 1.082643}, {1.051392, 1.066373, 1.083236}, + {1.051904, 1.066922, 1.083830}, {1.052417, 1.067472, 1.084426}, {1.052931, 1.068023, 1.085023}, + {1.053446, 1.068576, 1.085622}, {1.053963, 1.069130, 1.086222}, {1.054480, 1.069686, 1.086823}, + {1.054999, 1.070242, 1.087427}, {1.055518, 1.070800, 1.088032}, {1.056039, 1.071359, 1.088638}, + {1.056561, 1.071920, 1.089246}, {1.057084, 1.072482, 1.089855}, {1.057608, 1.073045, 1.090467}, + {1.058133, 1.073609, 1.091079}, {1.058660, 1.074175, 1.091694}, {1.059187, 1.074742, 1.092309}, + {1.059716, 1.075311, 1.092927}, {1.060246, 1.075881, 1.093546}, {1.060777, 1.076452, 1.094167}, + {1.061309, 1.077024, 1.094789}, {1.061842, 1.077598, 1.095413}, {1.062376, 1.078174, 1.096039}, + {1.062912, 1.078750, 1.096666}, {1.063449, 1.079328, 1.097296}, {1.063987, 1.079908, 1.097926}, + {1.064526, 1.080489, 1.098559}, {1.065067, 1.081071, 1.099193}, {1.065608, 1.081655, 1.099829}, + {1.066151, 1.082240, 1.100466}, {1.066695, 1.082826, 1.101106}, {1.067240, 1.083414, 1.101747}, + {1.067787, 1.084004, 1.102390}, {1.068334, 1.084595, 1.103034}, {1.068883, 1.085187, 1.103680}, + {1.069434, 1.085781, 1.104329}, {1.069985, 1.086376, 1.104979}, {1.070538, 1.086973, 1.105630}, + {1.071091, 1.087571, 1.106284}, {1.071647, 1.088171, 1.106939}, {1.072203, 1.088772, 1.107596}, + {1.072761, 1.089375, 1.108255}, {1.073320, 1.089979, 1.108916}, {1.073880, 1.090585, 1.109579}, + {1.074441, 1.091192, 1.110244}, {1.075004, 1.091801, 1.110910}, {1.075568, 1.092411, 1.111579}, + {1.076134, 1.093023, 1.112249}, {1.076700, 1.093637, 1.112921}, {1.077268, 1.094252, 1.113595}, + {1.077838, 1.094868, 1.114272}, {1.078408, 1.095486, 1.114950}, {1.078980, 1.096106, 1.115630}, + {1.079554, 1.096727, 1.116312}, {1.080128, 1.097350, 1.116996}, {1.080704, 1.097975, 1.117682}, + {1.081282, 1.098601, 1.118370}, {1.081860, 1.099229, 1.119060}, {1.082440, 1.099859, 1.119752}, + {1.083022, 1.100490, 1.120446}, {1.083605, 1.101123, 1.121142}, {1.084189, 1.101757, 1.121840}, + {1.084774, 1.102393, 1.122541}, {1.085361, 1.103031, 1.123243}, {1.085950, 1.103671, 1.123948}, + {1.086540, 1.104312, 1.124655}, {1.087131, 1.104955, 1.125363}, {1.087723, 1.105599, 1.126075}, + {1.088318, 1.106246, 1.126788}, {1.088913, 1.106894, 1.127503}, {1.089510, 1.107544, 1.128221}, + {1.090108, 1.108195, 1.128940}, {1.090708, 1.108848, 1.129663}, {1.091309, 1.109504, 1.130387}, + {1.091912, 1.110160, 1.131113}, {1.092516, 1.110819, 1.131842}, {1.093122, 1.111480, 1.132573}, + {1.093729, 1.112142, 1.133307}, {1.094338, 1.112806, 1.134042}, {1.094948, 1.113472, 1.134780}, + {1.095560, 1.114140, 1.135521}, {1.096173, 1.114809, 1.136263}, {1.096788, 1.115481, 1.137008}, + {1.097404, 1.116154, 1.137756}, {1.098022, 1.116830, 1.138506}, {1.098641, 1.117507, 1.139258}, + {1.099262, 1.118186, 1.140013}, {1.099884, 1.118867, 1.140770}, {1.100508, 1.119550, 1.141530}, + {1.101133, 1.120234, 1.142292}, {1.101761, 1.120921, 1.143057}, {1.102389, 1.121610, 1.143824}, + {1.103019, 1.122300, 1.144594}, {1.103651, 1.122993, 1.145366}, {1.104285, 1.123688, 1.146141}, + {1.104920, 1.124384, 1.146919}, {1.105556, 1.125083, 1.147699}, {1.106195, 1.125783, 1.148482}, + {1.106835, 1.126486, 1.149267}, {1.107476, 1.127191, 1.150055}, {1.108119, 1.127898, 1.150846}, + {1.108764, 1.128606, 1.151640}, {1.109411, 1.129317, 1.152436}, {1.110059, 1.130030, 1.153235}, + {1.110709, 1.130745, 1.154037}, {1.111360, 1.131463, 1.154842}, {1.112014, 1.132182, 1.155649}, + {1.112669, 1.132903, 1.156460}, {1.113325, 1.133627, 1.157273}, {1.113984, 1.134353, 1.158089}, + {1.114644, 1.135081, 1.158908}, {1.115306, 1.135811, 1.159730}, {1.115969, 1.136543, 1.160555}, + {1.116635, 1.137278, 1.161383}, {1.117302, 1.138015, 1.162214}, {1.117971, 1.138754, 1.163048}, + {1.118641, 1.139495, 1.163885}, {1.119314, 1.140238, 1.164725}, {1.119988, 1.140984, 1.165568}, + {1.120664, 1.141732, 1.166414}, {1.121342, 1.142483, 1.167264}, {1.122021, 1.143236, 1.168117}, + {1.122703, 1.143991, 1.168973}, {1.123386, 1.144748, 1.169832}, {1.124071, 1.145508, 1.170694}, + {1.124758, 1.146270, 1.171560}, {1.125447, 1.147035, 1.172429}, {1.126137, 1.147802, 1.173301}, + {1.126830, 1.148571, 1.174177}, {1.127524, 1.149343, 1.175056}, {1.128221, 1.150117, 1.175939}, + {1.128919, 1.150894, 1.176825}, {1.129619, 1.151673, 1.177715}, {1.130321, 1.152455, 1.178608}, + {1.131025, 1.153239, 1.179505}, {1.131731, 1.154026, 1.180405}, {1.132439, 1.154815, 1.181309}, + {1.133149, 1.155607, 1.182217}, {1.133861, 1.156402, 1.183128}, {1.134574, 1.157199, 1.184043}, + {1.135290, 1.157998, 1.184962}, {1.136008, 1.158800, 1.185885}, {1.136728, 1.159605, 1.186811}, + {1.137449, 1.160413, 1.187742}, {1.138173, 1.161223, 1.188676}, {1.138899, 1.162036, 1.189615}, + {1.139627, 1.162852, 1.190557}, {1.140356, 1.163670, 1.191504}, {1.141088, 1.164491, 1.192455}, + {1.141822, 1.165315, 1.193410}, {1.142558, 1.166142, 1.194369}, {1.143296, 1.166972, 1.195332}, + {1.144037, 1.167804, 1.196300}, {1.144779, 1.168639, 1.197272}, {1.145523, 1.169477, 1.198248}, + {1.146270, 1.170318, 1.199229}, {1.147018, 1.171162, 1.200214}, {1.147769, 1.172008, 1.201204}, + {1.148522, 1.172858, 1.202199}, {1.149277, 1.173711, 1.203198}, {1.150034, 1.174566, 1.204202}, + {1.150794, 1.175425, 1.205211}, {1.151555, 1.176286, 1.206224}, {1.152319, 1.177151, 1.207243}, + {1.153085, 1.178018, 1.208266}, {1.153853, 1.178889, 1.209295}, {1.154623, 1.179763, 1.210329}, + {1.155396, 1.180640, 1.211368}, {1.156170, 1.181520, 1.212412}, {1.156947, 1.182403, 1.213462}, + {1.157727, 1.183290, 1.214517}, {1.158508, 1.184179, 1.215578}, {1.159292, 1.185072, 1.216645}, + {1.160078, 1.185968, 1.217717}, {1.160866, 1.186867, 1.218795}, {1.161657, 1.187770, 1.219879}, + {1.162450, 1.188676, 1.220969}, {1.163245, 1.189585, 1.222066}, {1.164043, 1.190498, 1.223169}, + {1.164842, 1.191414, 1.224278}, {1.165645, 1.192333, 1.225390}, {1.166449, 1.193256, 1.226512}, + {1.167256, 1.194183, 1.227652}, {1.168065, 1.195112, 1.228796}, {1.168877, 1.196046, 1.229907}, + {1.169690, 1.196983, 1.230970}, {1.170506, 1.197923, 1.232009}, {1.171325, 1.198867, 1.233051}, + {1.172146, 1.199814, 1.234098}, {1.172969, 1.200766, 1.235149}, {1.173795, 1.201721, 1.236204}, + {1.174623, 1.202679, 1.237264}, {1.175453, 1.203642, 1.238327}, {1.176286, 1.204608, 1.239396}, + {1.177122, 1.205577, 1.240469}, {1.177959, 1.206551, 1.241547}, {1.178799, 1.207528, 1.242629}, + {1.179641, 1.208510, 1.243715}, {1.180486, 1.209495, 1.244806}, {1.181333, 1.210484, 1.245902}, + {1.182183, 1.211477, 1.247003}, {1.183035, 1.212474, 1.248109}, {1.183889, 1.213475, 1.249219}, + {1.184746, 1.214480, 1.250334}, {1.185605, 1.215489, 1.251454}, {1.186467, 1.216502, 1.252578}, + {1.187330, 1.217520, 1.253708}, {1.188197, 1.218541, 1.254843}, {1.189065, 1.219567, 1.255983}, + {1.189936, 1.220596, 1.257128}, {1.190809, 1.221630, 1.258278}, {1.191685, 1.222669, 1.259432}, + {1.192563, 1.223711, 1.260593}, {1.193444, 1.224759, 1.261759}, {1.194325, 1.225810, 1.262930}, + {1.195210, 1.226866, 1.264105}, {1.196097, 1.227926, 1.265286}, {1.196989, 1.228991, 1.266473}, + {1.197882, 1.230060, 1.267665}, {1.198768, 1.231134, 1.268863}, {1.199654, 1.232212, 1.270067}, + {1.200559, 1.233295, 1.271275}, {1.201504, 1.234383, 1.272489}, {1.202482, 1.235475, 1.273709}, + {1.203477, 1.236572, 1.274935}, {1.204476, 1.237674, 1.276167}, {1.205478, 1.238781, 1.277404}, + {1.206486, 1.239892, 1.278647}, {1.207497, 1.241008, 1.279896}, {1.208512, 1.242130, 1.281150}, + {1.209532, 1.243256, 1.282411}, {1.210556, 1.244387, 1.283679}, {1.211584, 1.245523, 1.284952}, + {1.212616, 1.246665, 1.286231}, {1.213652, 1.247811, 1.287515}, {1.214693, 1.248963, 1.288807}, + {1.215739, 1.250119, 1.290105}, {1.216789, 1.251281, 1.291409}, {1.217842, 1.252449, 1.292720}, + {1.218901, 1.253621, 1.294036}, {1.219964, 1.254799, 1.295359}, {1.221031, 1.255983, 1.296688}, + {1.222104, 1.257171, 1.298025}, {1.223181, 1.258366, 1.299368}, {1.224262, 1.259565, 1.300718}, + {1.225348, 1.260771, 1.302074}, {1.226439, 1.261982, 1.303436}, {1.227534, 1.263198, 1.304806}, + {1.228635, 1.264421, 1.306183}, {1.229740, 1.265649, 1.307567}, {1.230850, 1.266883, 1.308957}, + {1.231965, 1.268123, 1.310355}, {1.233085, 1.269368, 1.311759}, {1.234209, 1.270620, 1.313171}, + {1.235340, 1.271877, 1.314591}, {1.236475, 1.273141, 1.316017}, {1.237615, 1.274411, 1.317450}, + {1.238760, 1.275687, 1.318890}, {1.239910, 1.276968, 1.320338}, {1.241065, 1.278257, 1.321795}, + {1.242226, 1.279551, 1.323258}, {1.243393, 1.280852, 1.324728}, {1.244564, 1.282159, 1.326206}, + {1.245741, 1.283473, 1.327692}, {1.246923, 1.284793, 1.329186}, {1.248111, 1.286119, 1.330687}, + {1.249304, 1.287452, 1.332197}, {1.250503, 1.288792, 1.333713}, {1.251708, 1.290138, 1.335238}, + {1.252917, 1.291492, 1.336771}, {1.254133, 1.292852, 1.338312}, {1.255354, 1.294218, 1.339861}, + {1.256582, 1.295592, 1.341418}, {1.257815, 1.296972, 1.342982}, {1.259054, 1.298360, 1.344555}, + {1.260299, 1.299754, 1.346137}, {1.261549, 1.301156, 1.347728}, {1.262806, 1.302565, 1.349325}, + {1.264069, 1.303981, 1.350931}, {1.265338, 1.305404, 1.352546}, {1.266614, 1.306834, 1.354169}, + {1.267895, 1.308272, 1.355802}, {1.269182, 1.309717, 1.357442}, {1.270476, 1.311170, 1.359091}, + {1.271776, 1.312630, 1.360748}, {1.273083, 1.314097, 1.362414}, {1.274397, 1.315572, 1.364090}, + {1.275716, 1.317055, 1.365773}, {1.277042, 1.318546, 1.367465}, {1.278374, 1.320044, 1.369166}, + {1.279714, 1.321550, 1.370876}, {1.281060, 1.323064, 1.372596}, {1.282413, 1.324586, 1.374324}, + {1.283772, 1.326115, 1.376060}, {1.285138, 1.327653, 1.377805}, {1.286512, 1.329199, 1.379560}, + {1.287892, 1.330753, 1.381325}, {1.289280, 1.332315, 1.383097}, {1.290674, 1.333885, 1.384879}, + {1.292075, 1.335463, 1.386669}, {1.293483, 1.337050, 1.388469}, {1.294899, 1.338645, 1.390279}, + {1.296323, 1.340248, 1.392098}, {1.297753, 1.341860, 1.393924}, {1.299191, 1.343480, 1.395761}, + {1.300635, 1.345109, 1.397607}, {1.302088, 1.346746, 1.399463}, {1.303548, 1.348392, 1.401327}, + {1.305016, 1.350046, 1.403200}, {1.306491, 1.351709, 1.405083}, {1.307974, 1.353381, 1.406976}, + {1.309464, 1.355061, 1.408878}, {1.310962, 1.356750, 1.410789}, {1.312469, 1.358448, 1.412708}, + {1.313983, 1.360155, 1.414639}, {1.315505, 1.361871, 1.416579}, {1.317034, 1.363595, 1.418528}, + {1.318572, 1.365329, 1.420486}, {1.320118, 1.367071, 1.422454}, {1.321673, 1.368823, 1.424432}, + {1.323235, 1.370583, 1.426421}, {1.324805, 1.372353, 1.428419}, {1.326383, 1.374131, 1.430425}, + {1.327970, 1.375919, 1.432442}, {1.329566, 1.377716, 1.434471}, {1.331169, 1.379522, 1.436509}, + {1.332780, 1.381337, 1.438557}, {1.334401, 1.383161, 1.440615}, {1.336030, 1.384995, 1.442685}, + {1.337668, 1.386838, 1.444767}, {1.339314, 1.388690, 1.446859}, {1.340968, 1.390552, 1.448962}, + {1.342630, 1.392423, 1.451077}, {1.344302, 1.394303, 1.453206}, {1.345984, 1.396193, 1.455347}, + {1.347674, 1.398093, 1.457499}, {1.349371, 1.400002, 1.459665}, {1.351078, 1.401920, 1.461847}, + {1.352794, 1.403849, 1.464044}, {1.354520, 1.405787, 1.466257}, {1.356254, 1.407734, 1.468483}, + {1.357996, 1.409692, 1.470727}, {1.359748, 1.411660, 1.472993}, {1.361508, 1.413638, 1.475277}, + {1.363279, 1.415625, 1.477581}, {1.365059, 1.417624, 1.479907}, {1.366846, 1.419633, 1.482258}, + {1.368643, 1.421652, 1.484638}, {1.370450, 1.423682, 1.487046}, {1.372267, 1.425723, 1.489482}, + {1.374093, 1.427776, 1.491953}, {1.375927, 1.429840, 1.494466}, {1.377770, 1.431916, 1.497020}, + {1.379624, 1.434004, 1.499624}, {1.381488, 1.436104, 1.502279}, {1.383360, 1.438218, 1.504959}, + {1.385242, 1.440344, 1.507651}, {1.387133, 1.442485, 1.510364}, {1.389035, 1.444640, 1.513095}, + {1.390947, 1.446810, 1.515845}, {1.392869, 1.448996, 1.518612}, {1.394798, 1.451199, 1.521399}, + {1.396739, 1.453420, 1.524205}, {1.398692, 1.455659, 1.527027}, {1.400655, 1.457918, 1.529865}, + {1.402627, 1.460199, 1.532719}, {1.404609, 1.462502, 1.535588}, {1.406603, 1.464830, 1.538471}, + {1.408609, 1.467184, 1.541364}, {1.410625, 1.469568, 1.544267}, {1.412652, 1.471983, 1.547174}, + {1.414690, 1.474433, 1.550055}, {1.416742, 1.476922, 1.552939}, {1.418807, 1.479452, 1.555810}, + {1.420883, 1.482031, 1.558314}, {1.422972, 1.484667, 1.560539}, {1.425076, 1.487341, 1.563313}, + {1.427194, 1.490029, 1.566682}, {1.429327, 1.492735, 1.569993}, {1.431474, 1.495465, 1.573299}, + {1.433637, 1.498220, 1.576672}, {1.435819, 1.500998, 1.580071}, {1.438020, 1.503796, 1.583505}, + {1.440238, 1.506615, 1.586975}, {1.442475, 1.509458, 1.590479}, {1.444736, 1.512327, 1.594026}, + {1.447021, 1.515221, 1.597614}, {1.449331, 1.518138, 1.601235}, {1.451667, 1.521079, 1.604890}, + {1.454031, 1.524043, 1.608593}, {1.456429, 1.527032, 1.612341}, {1.458865, 1.530045, 1.616129}, + {1.461338, 1.533084, 1.619961}, {1.463852, 1.536148, 1.623836}, {1.466414, 1.539234, 1.627757}, + {1.469037, 1.542335, 1.631730}, {1.471706, 1.545451, 1.635751}, {1.474390, 1.548591, 1.639810}, + {1.477091, 1.551764, 1.643927}, {1.479820, 1.554971, 1.648105}, {1.482580, 1.558207, 1.652319}, + {1.485368, 1.561476, 1.656591}, {1.488178, 1.564781, 1.660923}, {1.491010, 1.568118, 1.665312}, + {1.493871, 1.571486, 1.669762}, {1.496769, 1.574889, 1.674272}, {1.499700, 1.578334, 1.678833}, + {1.502662, 1.581813, 1.683470}, {1.505656, 1.585325, 1.688174}, {1.508689, 1.588873, 1.692932}, + {1.511759, 1.592463, 1.697761}, {1.514872, 1.596094, 1.702671}, {1.518083, 1.599767, 1.707651}, + {1.521310, 1.603481, 1.712708}, {1.524670, 1.607233, 1.717836}, {1.528899, 1.611031, 1.723047}, + {1.532588, 1.614877, 1.728356}, {1.535429, 1.618766, 1.733737}, {1.538513, 1.622694, 1.739186}, + {1.541688, 1.626676, 1.744760}, {1.544828, 1.630712, 1.750413}, {1.548018, 1.634784, 1.756165}, + {1.551245, 1.638914, 1.762017}, {1.554496, 1.643097, 1.767979}, {1.557778, 1.647332, 1.774063}, + {1.561099, 1.651627, 1.780222}, {1.564455, 1.655977, 1.786529}, {1.567842, 1.660377, 1.792957}, + {1.571263, 1.664843, 1.799490}, {1.574720, 1.669375, 1.806173}, {1.578213, 1.673955, 1.812989}, + {1.581748, 1.678608, 1.819965}, {1.585325, 1.683328, 1.827049}, {1.588937, 1.688116, 1.834346}, + {1.592586, 1.692979, 1.841771}, +}; + +} // namespace metalvr::psvr2 diff --git a/daemon/psvr2_protocol.cpp b/daemon/psvr2_protocol.cpp new file mode 100644 index 0000000..6f61c0a --- /dev/null +++ b/daemon/psvr2_protocol.cpp @@ -0,0 +1,359 @@ +#include "psvr2_protocol.h" + +#include +#include +#include + +#include "psvr2_lens_table.h" + +namespace metalvr::psvr2 { + +namespace { + +constexpr float kPi = 3.14159265358979323846f; +constexpr float kGravity = 9.80665f; +constexpr float kGyroScale = 2000.0f / 32767.0f; // degrees/s per count +constexpr float kAccelScale = 4.0f * kGravity / 32767.0f; // m/s^2 per count + +uint16_t le16(const uint8_t *p) { return static_cast(p[0] | (p[1] << 8)); } +uint32_t le32(const uint8_t *p) { return p[0] | (p[1] << 8) | (p[2] << 16) | (static_cast(p[3]) << 24); } +float lef32(const uint8_t *p) { + const uint32_t bits = le32(p); + float f; + std::memcpy(&f, &bits, sizeof(f)); + return f; +} +void put16(uint8_t *p, uint32_t v) { p[0] = static_cast(v), p[1] = static_cast(v >> 8); } +void put32(uint8_t *p, uint32_t v) { + for (int i = 0; i < 4; i++) p[i] = static_cast(v >> (8 * i)); +} + +mvr_quat mul(const mvr_quat &a, const mvr_quat &b) { + return {a.w * b.x + a.x * b.w + a.y * b.z - a.z * b.y, a.w * b.y - a.x * b.z + a.y * b.w + a.z * b.x, + a.w * b.z + a.x * b.y - a.y * b.x + a.z * b.w, a.w * b.w - a.x * b.x - a.y * b.y - a.z * b.z}; +} +mvr_quat conj(const mvr_quat &q) { return {-q.x, -q.y, -q.z, q.w}; } +mvr_vec3 rotate(const mvr_quat &q, const mvr_vec3 &v) { + const float tx = 2 * (q.y * v.z - q.z * v.y), ty = 2 * (q.z * v.x - q.x * v.z), tz = 2 * (q.x * v.y - q.y * v.x); + return {v.x + q.w * tx + (q.y * tz - q.z * ty), v.y + q.w * ty + (q.z * tx - q.x * tz), + v.z + q.w * tz + (q.x * ty - q.y * tx)}; +} +mvr_quat normalize(mvr_quat q) { + const float n = std::sqrt(q.x * q.x + q.y * q.y + q.z * q.z + q.w * q.w); + if (!(n > 0)) return {0, 0, 0, 1}; + return {q.x / n, q.y / n, q.z / n, q.w / n}; +} +mvr_quat aboutX(float degrees) { + const float half = degrees * kPi / 360.0f; + return {std::sin(half), 0, 0, std::cos(half)}; +} +// a's inverse composed with b: b in a's frame +mvr_pose relative(const mvr_pose &a, const mvr_pose &b) { + const mvr_quat inv = conj(a.orientation); + const mvr_vec3 d{b.position.x - a.position.x, b.position.y - a.position.y, b.position.z - a.position.z}; + return {normalize(mul(inv, b.orientation)), rotate(inv, d)}; +} + +ControlRequest request(bool read, uint16_t report_id, uint16_t subcommand, const uint8_t *payload, uint32_t length) { + ControlRequest r; + r.request_type = static_cast(0x40 | 0x02 | (read ? 0x80 : 0)); // vendor, to an endpoint + r.request = read ? 0x01 : 0x09; + r.value = report_id; + r.data.assign(8 + length, 0); + put16(r.data.data(), report_id); + put16(r.data.data() + 2, subcommand); + put32(r.data.data() + 4, length); + if (payload && length) std::memcpy(r.data.data() + 8, payload, length); + return r; +} + +} // namespace + +// ---- status ---- + +std::optional parseStatus(const uint8_t *r, size_t size) { + // {DisplayPort receiver, proximity, function button, 2 empty, IPD mm, 26 more} then 24-byte IMU records: + // {vts u32, accel i16 x 3, gyro i16 x 3, DisplayPort frame u16, line u16, IMU time u16, status u16} + constexpr size_t kHeader = 32, kRecord = 24; + if (size < kHeader) return std::nullopt; + Status s; + s.display = r[0]; + s.worn = r[1] != 0; + s.function = r[2] != 0; + s.ipd_mm = r[5]; + const float degrees = kPi / 180.0f; + for (size_t at = kHeader; at + kRecord <= size; at += kRecord) { + const uint8_t *p = r + at; + int16_t a[3], g[3]; + for (int i = 0; i < 3; i++) { + a[i] = static_cast(le16(p + 4 + 2 * i)); + g[i] = static_cast(le16(p + 10 + 2 * i)); + } + ImuSample sample; + sample.vts_us = le32(p); + // the IMU's axes into the headset's (Monado's process_imu_record) + sample.gyro[0] = -g[1] * kGyroScale * degrees; + sample.gyro[1] = g[2] * kGyroScale * degrees; + sample.gyro[2] = -g[0] * kGyroScale * degrees; + sample.accel[0] = -a[1] * kAccelScale; + sample.accel[1] = a[2] * kAccelScale; + sample.accel[2] = -a[0] * kAccelScale; + s.imu.push_back(sample); + } + return s; +} + +// ---- tracking ---- + +std::optional parseSlam(const uint8_t *r, size_t size, const mvr_quat &previous) { + // {"SLA", 1, size u32, vts u32, u32, position f32 x 3, orientation f32 x 4 (w first), ...}: 512 bytes + if (size != 512 || std::memcmp(r, "SLA", 3) != 0) return std::nullopt; + SlamPose s; + s.vts_us = le32(r + 8); + const float p[3] = {lef32(r + 16), lef32(r + 20), lef32(r + 24)}; + const float o[4] = {lef32(r + 28), lef32(r + 32), lef32(r + 36), lef32(r + 40)}; + for (float v : p) + if (!std::isfinite(v)) return std::nullopt; + for (float v : o) + if (!std::isfinite(v)) return std::nullopt; + s.pose.position = {p[2], p[1], -p[0]}; + mvr_quat q{-o[2], -o[1], o[3], o[0]}; + if (q.x * previous.x + q.y * previous.y + q.z * previous.z + q.w * previous.w < 0) q = {-q.x, -q.y, -q.z, -q.w}; + s.pose.orientation = q; + return s; +} + +mvr_pose headFromSlam(const mvr_pose &slam) { + const float h = std::sqrt(0.5f); + const mvr_quat correction{0, 0, h, h}; + const mvr_quat q = normalize(mul(correction, slam.orientation)); + const mvr_vec3 offset = rotate(q, mvr_vec3{0.000247f, -0.000273f, 0.104826f}); + return {q, {slam.position.x + offset.x, slam.position.y + offset.y, slam.position.z + offset.z}}; +} + +// ---- requests ---- + +ControlRequest sendRequest(uint16_t report_id, uint16_t subcommand, const uint8_t *payload, uint32_t length) { + return request(false, report_id, subcommand, payload, length); +} +ControlRequest readRequest(uint16_t report_id, uint16_t subcommand, uint32_t length) { + return request(true, report_id, subcommand, nullptr, length); +} +std::vector readPayload(const uint8_t *reply, size_t size, uint32_t length) { + if (size < 8) return {}; + const size_t n = std::min(length, size - 8); + return std::vector(reply + 8, reply + 8 + n); +} + +ControlRequest cameraModeRequest(uint32_t mode) { + uint8_t payload[8]; + put32(payload, 1); + put32(payload + 4, mode); + return sendRequest(0x0b, 1, payload, sizeof(payload)); +} + +ControlRequest brightnessRequest(float brightness) { + const uint8_t level = static_cast(std::clamp(brightness * 31.0f, 0.0f, 31.0f)); + return sendRequest(0x12, 1, &level, 1); +} + +ControlRequest firmwareInfoRequest() { return readRequest(0x81, 1, 84); } +ControlRequest lensCalibrationRequest() { return readRequest(0x8f, 1, 0x100); } + +std::optional parseFirmwareInfo(const uint8_t *p, size_t size) { + // {version u32, u32, commit 40, u32, u32, PCB id 16, ...} + if (size < 72) return std::nullopt; + FirmwareInfo f; + f.version = le32(p); + const char *id = reinterpret_cast(p + 56); + f.pcb_id.assign(id, strnlen(id, 16)); + return f; +} + +// ---- lenses ---- + +std::array defaultLensCalibration() { return {-0.09919293f, 0, 0.09919293f, 0, 1, 0, 1, 0}; } + +std::array lensCalibration(const uint8_t *payload, size_t size) { + // {version u8, 7 bytes, f32 x 32} + if (size < 8 + 7 * 4 || payload[0] < 4) return defaultLensCalibration(); + float p[7]; + for (int i = 0; i < 7; i++) p[i] = lef32(payload + 8 + 4 * i); + for (float v : p) + if (!std::isfinite(v)) return defaultLensCalibration(); + std::array c{}; + c[0] = (((-p[0] - p[6]) * 29.9f + 14.95f) / 1000.0f - 3.22f) / 32.46199f; + c[1] = (((-p[1] * 29.9f) + 14.95f) / 1000.0f) / 32.46199f; + c[2] = (((p[6] - p[2]) * 29.9f + 14.95f) / 1000.0f + 3.22f) / 32.46199f; + c[3] = (((-p[3] * 29.9f) + 14.95f) / 1000.0f) / 32.46199f; + const float left = -p[4] * kPi / 180.0f, right = -p[5] * kPi / 180.0f; + c[4] = std::cos(left), c[5] = std::sin(left); + c[6] = std::cos(right), c[7] = std::sin(right); + return c; +} + +void computeDistortion(const std::array &cal, int eye, float u, float v, float out[3][2]) { + const float fx = 0.3585564f, fy = 0.3762281f, cy = 0.5f; + const float k1 = cal[eye ? 2 : 0], k2 = cal[eye ? 3 : 1], k3 = cal[eye ? 6 : 4], k4 = cal[eye ? 7 : 5]; + const float cx = eye ? 0.3396213f : 0.6603788f; + const float x = k1 + ((u + u) - 1.0f) * 0.9803922f * 0.9394987f; + const float y = k2 + ((v + v) - 1.0f) * 0.9394987f; + const float r = (x * x + y * y) * 1023.0f; + const int i = static_cast(r); + float scale[3]; + if (i > -1 && i + 1 < kLensTableSize) { + const float f = r - static_cast(i); + for (int c = 0; c < 3; c++) scale[c] = f * kLensTable[i + 1][c] + (1.0f - f) * kLensTable[i][c]; + } else { + const float d = r - 1023.0f; + scale[0] = d * 0.003648639f + 1.592586f; + scale[1] = d * 0.004862547f + 1.692979f; + scale[2] = d * 0.007425308f + 1.841771f; + } + for (int c = 0; c < 3; c++) { + const float a = x * scale[c]; + const float b = (c == 1 ? y - 0.0002302693f : y) * scale[c]; + out[c][0] = (k3 * a - k4 * b) * fx + cx; + out[c][1] = (k4 * a + k3 * b) * fy + cy; + } +} + +mvr_fov fieldOfView(int eye) { + const float d = kPi / 180.0f; + return eye == 0 ? mvr_fov{-61.5f * d, 43.5f * d, 53.0f * d, -53.0f * d} + : mvr_fov{-43.5f * d, 61.5f * d, 53.0f * d, -53.0f * d}; +} + +// ---- Sense ---- + +uint32_t crc32(uint32_t crc, const uint8_t *p, size_t size) { + crc ^= 0xffffffffu; + while (size--) { + crc ^= *p++; + for (int i = 0; i < 8; i++) crc = (crc >> 1) ^ ((crc & 1) ? 0xedb88320u : 0); + } + return crc ^ 0xffffffffu; +} + +std::optional parseSenseReport(const uint8_t *r, size_t size, bool left) { + const uint8_t *c = nullptr; // the part both carry (Monado's pssense_input_report_common) + if (size == kSenseBluetoothLength && r[0] == kSenseBluetoothReport) { + const uint8_t seed = 0xa1; + uint32_t crc = crc32(0, &seed, 1); + crc = crc32(crc, r, kSenseBluetoothLength - 4); + if (crc != le32(r + kSenseBluetoothLength - 4)) return std::nullopt; + c = r + 2; + } else if (size == kSenseUsbLength && r[0] == kSenseUsbReport) { + c = r + 1; + } else { + return std::nullopt; + } + // {stick x, stick y, trigger, trigger proximity, squeeze proximity, 2, buttons x 3, 1, sequence u32, + // gyro i16 x 3, accel i16 x 3, IMU ticks u32, temperature, 7, trigger feedback x 2, battery, plug, ...} + const uint8_t *b = c + 7; + SenseState s; + s.stick_x = (c[0] - 128) / 128.0f; + s.stick_y = (c[1] - 128) / -128.0f; + s.trigger = c[2] / 255.0f; + const bool ps = b[1] & 16, squeeze_touch = b[2] & 8, trigger_touch = b[1] & 128, stick_touch = b[2] & 4; + bool primary, primary_touch, secondary, secondary_touch, squeeze, trigger_click, stick, menu; + if (left) { + // square is X, triangle Y; Create is the menu + primary = b[0] & 1, primary_touch = b[2] & 2, secondary = b[0] & 8, secondary_touch = b[2] & 1; + squeeze = b[0] & 16, trigger_click = b[0] & 64, stick = b[1] & 4, menu = b[1] & 1; + } else { + // cross is A, circle B; Options is the menu + primary = b[0] & 2, primary_touch = b[2] & 2, secondary = b[0] & 4, secondary_touch = b[2] & 1; + squeeze = b[0] & 32, trigger_click = b[0] & 128, stick = b[1] & 8, menu = b[1] & 2; + } + s.squeeze = squeeze ? 1.0f : 0.0f; + s.buttons = (primary ? MVR_BUTTON_PRIMARY : 0u) | (secondary ? MVR_BUTTON_SECONDARY : 0u) | + (menu ? MVR_BUTTON_MENU : 0u) | (ps ? MVR_BUTTON_SYSTEM : 0u) | (stick ? MVR_BUTTON_THUMBSTICK : 0u) | + (squeeze ? MVR_BUTTON_SQUEEZE : 0u) | (trigger_click ? MVR_BUTTON_TRIGGER : 0u); + s.touches = (primary_touch ? MVR_BUTTON_PRIMARY : 0u) | (secondary_touch ? MVR_BUTTON_SECONDARY : 0u) | + (stick_touch ? MVR_BUTTON_THUMBSTICK : 0u) | (trigger_touch ? MVR_BUTTON_TRIGGER : 0u) | + (squeeze_touch ? MVR_BUTTON_SQUEEZE : 0u); + for (int i = 0; i < 3; i++) { + s.gyro[i] = static_cast(le16(c + 15 + 2 * i)); + s.accel[i] = static_cast(le16(c + 21 + 2 * i)); + } + s.imu_ticks = le32(c + 27); + const uint8_t battery = c[41]; + const uint8_t state = battery >> 4; + s.battery = std::min(1.0f, (battery & 0x0f) * 0.1f + 0.05f); + s.battery_valid = state <= 2; + s.charging = state == 1 || state == 2; + if (state == 2) s.battery = 1.0f; + return s; +} + +std::optional parseSenseCalibration(const uint8_t a[64], const uint8_t b[64]) { + // each part {5, part id, 58 bytes, crc}: part 0x00 first in the calibration, 0x81 second + if (a[0] != kSenseCalibrationReport || b[0] != kSenseCalibrationReport) return std::nullopt; + const uint8_t *first = a[1] == 0x00 ? a : b[1] == 0x00 ? b : nullptr; + const uint8_t *second = a[1] == 0x81 ? a : b[1] == 0x81 ? b : nullptr; + if (!first || !second) return std::nullopt; + uint8_t d[116]; + std::memcpy(d, first + 2, 58); + std::memcpy(d + 58, second + 2, 58); + const auto i16 = [&](size_t at) { return static_cast(static_cast(le16(d + at))); }; + const float accel_plus[3] = {i16(0x00), i16(0x0e), i16(0x1c)}, accel_minus[3] = {i16(0x06), i16(0x14), i16(0x22)}; + const float gyro_bias[3] = {i16(0x24), i16(0x26), i16(0x28)}; + const float gyro_plus[3] = {i16(0x2a), i16(0x38), i16(0x46)}, gyro_minus[3] = {i16(0x30), i16(0x3e), i16(0x4c)}; + float reference = (i16(0x4e) + i16(0x50)) * 0.5f * kPi / 180.0f; + if (!(reference > 0)) reference = kPi * 3.0f; // Monado's fallback + SenseCalibration cal; + for (int i = 0; i < 3; i++) { + const float span = (std::fabs(gyro_plus[i] - gyro_bias[i]) + std::fabs(gyro_minus[i] - gyro_bias[i])) / 2.0f; + const float bias = (accel_plus[i] + accel_minus[i]) / 2.0f; + if (!(span > 0) || !(accel_plus[i] - bias != 0)) return std::nullopt; + cal.gyro_bias[i] = gyro_bias[i]; + cal.gyro_scale[i] = reference / span; + cal.accel_bias[i] = bias; + cal.accel_scale[i] = kGravity / (accel_plus[i] - bias); + } + return cal; +} + +void senseImu(const SenseState &s, const SenseCalibration &c, float accel[3], float gyro[3]) { + for (int i = 0; i < 3; i++) { + gyro[i] = (s.gyro[i] - c.gyro_bias[i]) * c.gyro_scale[i]; + accel[i] = (s.accel[i] - c.accel_bias[i]) * c.accel_scale[i]; + } +} + +std::array senseOutputReport(float amplitude, uint8_t sequence, uint8_t counter, uint32_t host_time_us) { + // {0x31, sequence << 4, 0x10, settings (38), counter, PCM haptics (32), crc}; settings: {flag1, flag2, amplitude, + // 0, trigger (11), host time u32, LEDs {phase, sequence, period, position u32, length u32, blink x 4}, 4} + std::array r{}; + r[0] = kSenseBluetoothReport; + r[1] = static_cast((sequence & 0x0f) << 4); + r[2] = 0x10; + uint8_t *settings = r.data() + 3; + const uint8_t level = static_cast(std::clamp(amplitude, 0.0f, 1.0f) * 255.0f + 0.5f); + settings[0] = 0x01 | 0x02; // vibration on (with rumble emulation), at 120 Hz + settings[2] = level; + put32(settings + 15, host_time_us); + settings[19] = 5; // tracking LEDs all off + r[41] = counter; + const uint8_t seed = 0xa2; + uint32_t crc = crc32(0, &seed, 1); + crc = crc32(crc, r.data(), r.size() - 4); + put32(r.data() + 74, crc); + return r; +} + +namespace { +// Monado's T_steamvrroot_model, _xrgrip and _xraim, for a hand +mvr_pose senseRoot(int hand, int which) { + const float sx = hand == 0 ? 1.0f : -1.0f; + if (which == 0) return {aboutX(-39.0f), {-sx * 0.01432f, 0.007713f, 0.10399f}}; + if (which == 1) return {aboutX(20.6f), {sx * 0.007f, -0.00182941f, 0.1019482f}}; + return {aboutX(-39.4f), {sx * 0.007f, -0.03894766f, 0.00949694f}}; +} +} // namespace + +mvr_pose senseGrip(int hand) { return relative(senseRoot(hand, 0), senseRoot(hand, 1)); } +mvr_pose senseAim(int hand) { return relative(senseRoot(hand, 0), senseRoot(hand, 2)); } + +} // namespace metalvr::psvr2 diff --git a/daemon/psvr2_protocol.h b/daemon/psvr2_protocol.h new file mode 100644 index 0000000..6e4e3f5 --- /dev/null +++ b/daemon/psvr2_protocol.h @@ -0,0 +1,149 @@ +// The PlayStation VR2's USB side and its Sense controllers' reports (docs/HEADSETS.md, phase J). +// +// Nothing here touches a device: reports are taken in and made as bytes, so that each can be checked without a +// headset (tests/headset_protocol_check.cpp). The layouts and the axis conventions follow Monado's drivers +// (src/xrt/drivers/psvr2 and pssense; Copyright 2023 Jan Schmidt, 2023 Collabora, 2023 Jarett Millard, 2024 Joel +// Valenciano, 2025-2026 Beyley Cardellio; BSL-1.0). +// +// The headset is a USB device of vendor-specific interfaces, not HID: its status and IMU arrive on an interrupt +// endpoint, and the headset tracks itself (inside-out, on its own processor) and sends its pose on a bulk endpoint. +// It is controlled with vendor requests on the default pipe. The Sense controllers pair with the Mac over Bluetooth +// as HID devices. +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "metalvr_protocol.h" + +namespace metalvr::psvr2 { + +constexpr uint16_t kVendor = 0x054c; // Sony +constexpr uint16_t kProduct = 0x0cde; +constexpr uint32_t kDisplayVendor = 0x4dd9; // "SNY" in the EDID + +// interfaces and their IN endpoints (Monado's psvr2_protocol.h) +constexpr int kSlamInterface = 3, kSlamEndpoint = 0x83; +constexpr int kGazeInterface = 5, kGazeEndpoint = 0x85; +constexpr int kCameraInterface = 6, kCameraEndpoint = 0x87; +constexpr int kStatusInterface = 7, kStatusEndpoint = 0x88, kStatusAltSetting = 1; +constexpr int kLedDetectorInterface = 8, kLedDetectorEndpoint = 0x89; +constexpr int kRelocalizerInterface = 9, kRelocalizerEndpoint = 0x8a; +constexpr int kVdInterface = 10, kVdEndpoint = 0x8b; +constexpr size_t kSlamTransferSize = 1024, kStatusTransferSize = 1024, kCameraTransferSize = 1040640, + kLedDetectorTransferSize = 36944, kRelocalizerTransferSize = 821120, kVdTransferSize = 32768; + +constexpr float kImuHz = 2000.0f; + +// ---- the headset's status and IMU (interrupt endpoint 8) ---- + +struct ImuSample { + uint32_t vts_us = 0; // the headset's clock + float accel[3] = {}; // m/s^2, in the headset's frame (OpenXR's: +x right, +y up, +z back) + float gyro[3] = {}; // rad/s +}; + +struct Status { + uint8_t display = 0; // the DisplayPort receiver: 0 not ready + bool worn = false; // the proximity sensor + bool function = false; // the button under the headset + uint8_t ipd_mm = 0; // the IPD dial, 59..72 + std::vector imu; +}; +std::optional parseStatus(const uint8_t *report, size_t size); + +// ---- the headset's own tracking (bulk endpoint 3) ---- + +struct SlamPose { + uint32_t vts_us = 0; + mvr_pose pose{{0, 0, 0, 1}, {0, 0, 0}}; // the headset's tracker, in its room (OpenXR's axes) +}; +// A 512-byte "SLA" record, its axes turned as Monado turns them. `previous` keeps the quaternion on the same side +// as the last one: the headset's tracker does not, and a flip would read as a full turn. +std::optional parseSlam(const uint8_t *record, size_t size, const mvr_quat &previous); +// From the tracker's pose to the head's: Monado's SLAM correction (a quarter turn about z) and the tracker's place +// in the head (T_imu_head) +mvr_pose headFromSlam(const mvr_pose &slam); + +// ---- vendor requests (Monado's send_psvr2_control and get_psvr2_control) ---- + +struct ControlRequest { + uint8_t request_type = 0; // vendor, to an endpoint; 0x80 set for a read + uint8_t request = 0; // 9 to send, 1 to read + uint16_t value = 0; // the report id + uint16_t index = 0; + std::vector data; // {report id u16, subcommand u16, length u32, payload} +}; +ControlRequest sendRequest(uint16_t report_id, uint16_t subcommand, const uint8_t *payload, uint32_t length); +ControlRequest readRequest(uint16_t report_id, uint16_t subcommand, uint32_t length); +// what a read returned: the payload after the 8-byte header +std::vector readPayload(const uint8_t *reply, size_t size, uint32_t length); + +enum CameraMode : uint32_t { kCameraOff = 0, kCameraTracking = 0x0a }; // 0x0a is the mode Monado runs +ControlRequest cameraModeRequest(uint32_t mode); +ControlRequest brightnessRequest(float brightness); // 0..1 +ControlRequest firmwareInfoRequest(); +ControlRequest lensCalibrationRequest(); + +struct FirmwareInfo { + uint32_t version = 0; + std::string pcb_id; // the headset's serial number +}; +std::optional parseFirmwareInfo(const uint8_t *payload, size_t size); + +// ---- the lenses ---- + +// The eye's lens offsets and tilt from the headset's calibration (report 0x8f), as Monado's +// psvr2_setup_distortion_and_fovs works them out: x and y offsets, then cos and sin of the tilt, left then right. +// Before calibration version 4, or without one, Monado's default. +std::array lensCalibration(const uint8_t *payload, size_t size); +std::array defaultLensCalibration(); +// Monado's psvr2_compute_distortion_asymmetric: where each colour of the eye's panel point (u, v) (0..1 across and +// down) is in the rendered image +void computeDistortion(const std::array &calibration, int eye, float u, float v, float out[3][2]); +// each eye's field of view, OpenXR's signs, radians (Monado's) +mvr_fov fieldOfView(int eye); + +// ---- the Sense controllers ---- + +constexpr uint16_t kSenseLeft = 0x0e45, kSenseRight = 0x0e46; +constexpr uint8_t kSenseBluetoothReport = 0x31, kSenseUsbReport = 0x01, kSenseCalibrationReport = 0x05; +constexpr size_t kSenseBluetoothLength = 78, kSenseUsbLength = 64, kSenseFeatureLength = 64; + +uint32_t crc32(uint32_t crc, const uint8_t *data, size_t size); // the controllers' CRC-32 (reflected, 0xedb88320) + +struct SenseState { + uint32_t buttons = 0; // mvr_button: A/X (cross, square), B/Y (circle, triangle), menu, system, stick, squeeze, trigger + uint32_t touches = 0; + float trigger = 0, squeeze = 0; + float stick_x = 0, stick_y = 0; // -1..1, +y up + int16_t gyro[3] = {}, accel[3] = {}; // raw + uint32_t imu_ticks = 0; // thirds of a microsecond + float battery = 0; // 0..1 + bool charging = false, battery_valid = false; +}; +// An input report, Bluetooth (0x31, its CRC checked) or USB (0x01), of the `left` or right controller +std::optional parseSenseReport(const uint8_t *report, size_t size, bool left); + +struct SenseCalibration { + float gyro_bias[3] = {}, gyro_scale[3] = {}; // (raw - bias) * scale = rad/s + float accel_bias[3] = {}, accel_scale[3] = {}; // (raw - bias) * scale = m/s^2 +}; +// Feature report 5, read twice: its two parts (0x00 and 0x81) in either order. Reading it switches the controller +// from its compatibility mode into the one with these reports. +std::optional parseSenseCalibration(const uint8_t part_a[64], const uint8_t part_b[64]); +void senseImu(const SenseState &state, const SenseCalibration &calibration, float accel[3], float gyro[3]); + +// The Bluetooth output report (78 bytes) with a vibration of `amplitude` (0..1, 0 stops it), its tracking LEDs off +// (MetalVR does not track them), stamped with `sequence` (0..15) and `counter`, its CRC filled in +std::array senseOutputReport(float amplitude, uint8_t sequence, uint8_t counter, uint32_t host_time_us); + +// the grip and aim poses in the controller's own frame (Monado's T_steamvrroot_* offsets, for each hand) +mvr_pose senseGrip(int hand); +mvr_pose senseAim(int hand); + +} // namespace metalvr::psvr2 diff --git a/daemon/rift_s_cameras.h b/daemon/rift_s_cameras.h new file mode 100644 index 0000000..db2fd3f --- /dev/null +++ b/daemon/rift_s_cameras.h @@ -0,0 +1,26 @@ +// The Rift S's cameras as macOS sees them: a UVC video device named "Rift S Sensor" (Monado's rift_s_camera.c finds +// it by that name), captured with AVFoundation. Its frames are 8-bit grey, 3200 wide, though the device calls them +// YUYV at 1600: they are asked for as YUYV so that macOS hands the bytes over unconverted, and read as grey. If +// macOS instead gives full-width pictures (from an MJPEG mode), their brightness is taken. +#pragma once + +#include +#include +#include +#include + +namespace metalvr { + +class RiftSCameras { +public: + virtual ~RiftSCameras() = default; +}; + +// grey bytes, their width and height, and the bytes from one row to the next; on AVFoundation's queue +using RiftSFrameHandler = std::function; + +// the capture started; null if macOS has no "Rift S Sensor" camera (yet: it appears after the headset is turned +// on) or does not let metalvrd use cameras (System Settings, Privacy & Security, Camera) +std::unique_ptr startRiftSCameras(RiftSFrameHandler handler); + +} // namespace metalvr diff --git a/daemon/rift_s_cameras.mm b/daemon/rift_s_cameras.mm new file mode 100644 index 0000000..13a524f --- /dev/null +++ b/daemon/rift_s_cameras.mm @@ -0,0 +1,134 @@ +#import "rift_s_cameras.h" + +#import + +#include + +#include "common.h" +#include "rift_s_protocol.h" + +@interface MetalVRRiftSCameraSink : NSObject { +@public + metalvr::RiftSFrameHandler handler; + std::vector grey; +} +@end + +@implementation MetalVRRiftSCameraSink +- (void)captureOutput:(AVCaptureOutput *)output + didOutputSampleBuffer:(CMSampleBufferRef)sample + fromConnection:(AVCaptureConnection *)connection { + CVPixelBufferRef buffer = CMSampleBufferGetImageBuffer(sample); + if (!buffer || CVPixelBufferLockBaseAddress(buffer, kCVPixelBufferLock_ReadOnly) != kCVReturnSuccess) + return; + const OSType format = CVPixelBufferGetPixelFormatType(buffer); + const uint32_t width = static_cast(CVPixelBufferGetWidth(buffer)); + const uint32_t height = static_cast(CVPixelBufferGetHeight(buffer)); + const bool planar = CVPixelBufferIsPlanar(buffer); + const uint8_t *base = static_cast(planar ? CVPixelBufferGetBaseAddressOfPlane(buffer, 0) + : CVPixelBufferGetBaseAddress(buffer)); + const size_t stride = planar ? CVPixelBufferGetBytesPerRowOfPlane(buffer, 0) : CVPixelBufferGetBytesPerRow(buffer); + if (base && width * 2 == metalvr::rift_s::kCameraFrameWidth && + (format == kCVPixelFormatType_422YpCbCr8_yuvs || format == kCVPixelFormatType_422YpCbCr8)) { + // the device's own bytes: grey at twice the width it states + handler(base, width * 2, height, stride); + } else if (base && width == metalvr::rift_s::kCameraFrameWidth) { + // full-width pictures: their brightness, from the luma plane or every other byte of packed 4:2:2 + if (planar) { + handler(base, width, height, stride); + } else if (format == kCVPixelFormatType_422YpCbCr8_yuvs || format == kCVPixelFormatType_422YpCbCr8) { + const int luma = format == kCVPixelFormatType_422YpCbCr8_yuvs ? 0 : 1; + grey.resize(static_cast(width) * height); + for (uint32_t y = 0; y < height; y++) + for (uint32_t x = 0; x < width; x++) grey[y * width + x] = base[y * stride + 2 * x + luma]; + handler(grey.data(), width, height, width); + } + } + CVPixelBufferUnlockBaseAddress(buffer, kCVPixelBufferLock_ReadOnly); +} +@end + +namespace metalvr { +namespace { + +class Capture final : public RiftSCameras { +public: + bool start(AVCaptureDevice *device, RiftSFrameHandler handler) { + NSError *error = nil; + AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:device error:&error]; + if (!input) { + logInfo("rift: cannot open the Rift S's cameras: %s", error.localizedDescription.UTF8String); + return false; + } + session_ = [AVCaptureSession new]; + if (![session_ canAddInput:input]) + return false; + [session_ addInput:input]; + // the widest format the device offers, at its highest rate (the 60 Hz feed of interleaved tracking and + // controller frames) + AVCaptureDeviceFormat *best = nil; + for (AVCaptureDeviceFormat *format in device.formats) { + const CMVideoDimensions size = CMVideoFormatDescriptionGetDimensions(format.formatDescription); + const CMVideoDimensions best_size = + best ? CMVideoFormatDescriptionGetDimensions(best.formatDescription) : CMVideoDimensions{0, 0}; + if (size.width > best_size.width || (size.width == best_size.width && size.height > best_size.height)) + best = format; + } + if (best && [device lockForConfiguration:&error]) { + device.activeFormat = best; + [device unlockForConfiguration]; + } + AVCaptureVideoDataOutput *output = [AVCaptureVideoDataOutput new]; + output.videoSettings = @{(id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_422YpCbCr8_yuvs)}; + output.alwaysDiscardsLateVideoFrames = NO; // every tracking frame matters to the tracker + sink_ = [MetalVRRiftSCameraSink new]; + sink_->handler = std::move(handler); + queue_ = dispatch_queue_create("metalvr.rift-s-cameras", DISPATCH_QUEUE_SERIAL); + [output setSampleBufferDelegate:sink_ queue:queue_]; + if (![session_ canAddOutput:output]) + return false; + [session_ addOutput:output]; + [session_ startRunning]; + const CMVideoDimensions size = + best ? CMVideoFormatDescriptionGetDimensions(best.formatDescription) : CMVideoDimensions{0, 0}; + logInfo("rift: reading the Rift S's cameras (%s, %dx%d)", device.localizedName.UTF8String, size.width, + size.height); + return true; + } + + ~Capture() override { + [session_ stopRunning]; + if (queue_) + dispatch_sync(queue_, ^{ + }); // the last frame handled before the handler goes + } + +private: + AVCaptureSession *session_ = nil; + MetalVRRiftSCameraSink *sink_ = nil; + dispatch_queue_t queue_ = nullptr; +}; + +} // namespace + +std::unique_ptr startRiftSCameras(RiftSFrameHandler handler) { + if ([AVCaptureDevice authorizationStatusForMediaType:AVMediaTypeVideo] == AVAuthorizationStatusDenied) { + logInfo("rift: macOS does not let metalvrd use cameras (System Settings, Privacy & Security, Camera); the " + "Rift S tracks rotation only"); + return nullptr; + } + AVCaptureDeviceDiscoverySession *discovery = + [AVCaptureDeviceDiscoverySession discoverySessionWithDeviceTypes:@[ AVCaptureDeviceTypeExternal ] + mediaType:AVMediaTypeVideo + position:AVCaptureDevicePositionUnspecified]; + for (AVCaptureDevice *device in discovery.devices) + if ([device.localizedName rangeOfString:@"Rift S Sensor" options:NSCaseInsensitiveSearch].location != NSNotFound) { + auto capture = std::make_unique(); + if (capture->start(device, std::move(handler))) + return capture; + return nullptr; + } + return nullptr; +} + +} // namespace metalvr diff --git a/daemon/rift_s_protocol.cpp b/daemon/rift_s_protocol.cpp new file mode 100644 index 0000000..8a65951 --- /dev/null +++ b/daemon/rift_s_protocol.cpp @@ -0,0 +1,532 @@ +#include "rift_s_protocol.h" + +#include +#include +#include +#include + +#include "json_value.h" + +namespace metalvr::rift_s { + +namespace { + +constexpr float kGravity = 9.80665f; +constexpr float kDegrees = 3.14159265358979323846f / 180.0f; + +uint16_t le16(const uint8_t *p) { return static_cast(p[0] | (p[1] << 8)); } +uint32_t le32(const uint8_t *p) { return p[0] | (p[1] << 8) | (p[2] << 16) | (static_cast(p[3]) << 24); } +uint64_t le64(const uint8_t *p) { return le32(p) | (static_cast(le32(p + 4)) << 32); } +float lef32(const uint8_t *p) { + const uint32_t bits = le32(p); + float f; + std::memcpy(&f, &bits, sizeof(f)); + return f; +} + +void apply(const float m[9], const float in[3], float out[3]) { + const float x = in[0], y = in[1], z = in[2]; + out[0] = m[0] * x + m[1] * y + m[2] * z; + out[1] = m[3] * x + m[4] * y + m[5] * z; + out[2] = m[6] * x + m[7] * y + m[8] * z; +} + +} // namespace + +std::optional parseDeviceInfo(const uint8_t *r, size_t size) { + // {id, height u16, width u16, u16, refresh u8, ...}: the panel's resolution as the computer sees it + if (size < 8 || r[0] != kReportDeviceInfo) return std::nullopt; + DeviceInfo info; + info.height = le16(r + 1); + info.width = le16(r + 3); + info.refresh_hz = r[7]; + if (!info.width || !info.height || !info.refresh_hz) return std::nullopt; + return info; +} + +std::optional parseImuConfig(const uint8_t *r, size_t size) { + // {id, imu hz u32, gyro scale f32, accel scale f32, temperature scale f32, temperature offset f32} + if (size < 21 || r[0] != kReportImuConfig) return std::nullopt; + ImuConfig c; + c.hz = le32(r + 1); + c.gyro_scale = lef32(r + 5); + c.accel_scale = lef32(r + 9); + if (!c.hz || !(c.gyro_scale > 0) || !(c.accel_scale > 0) || !std::isfinite(c.gyro_scale) || + !std::isfinite(c.accel_scale)) + return std::nullopt; + return c; +} + +std::optional parseImuCalibration(std::string_view text) { + const auto root = JsonValue::parse(text); + if (!root || !root->isObject()) return std::nullopt; + const JsonValue &version = (*root)["FileFormat"]["Version"]; + if (!version.isString() || std::strtof(version.string().c_str(), nullptr) != 1.0f) return std::nullopt; + const JsonValue &imu = (*root)["ImuCalibration"]; + ImuCalibration c; + // a row-major 4x4 (or 3x4) isometry: the rotation is its upper left 3x3 + float m[12]; + if (!imu["DeviceFromImu"].numbers(m, 12)) return std::nullopt; + const float rotation[9] = {m[0], m[1], m[2], m[4], m[5], m[6], m[8], m[9], m[10]}; + std::memcpy(c.device_from_imu, rotation, sizeof(rotation)); + c.imu_position[0] = m[3]; + c.imu_position[1] = m[7]; + c.imu_position[2] = m[11]; + const JsonValue &gyro = imu["Gyroscope"]; + const JsonValue &accel = imu["Accelerometer"]; + if (!gyro["RectificationMatrix"].numbers(c.gyro_rectification, 9) || + !gyro["Offset"]["ConstantOffset"].numbers(c.gyro_offset, 3) || + !accel["RectificationMatrix"].numbers(c.accel_rectification, 9) || + !accel["Offset"]["OffsetAtZeroDegC"].numbers(c.accel_offset, 3)) + return std::nullopt; + return c; +} + +std::optional parseHmdReport(const uint8_t *r, size_t size, const ImuConfig &config, + const ImuCalibration &calibration) { + // {id, u16, timestamp u32, three {marker, accel 3 x i16, gyro 3 x i16, temperature i16}, ...}: 64 bytes + if (size != 64 || r[0] != kInHmd) return std::nullopt; + HmdReport report; + report.timestamp_us = le32(r + 3); + const float gyro_scale = kDegrees / config.gyro_scale; + const float accel_scale = kGravity / config.accel_scale; + for (int i = 0; i < 3; i++) { + const uint8_t *s = r + 7 + 15 * i; + if (s[0] & 0x80) break; // this sample and the ones after it are not valid + float accel[3], gyro[3]; + for (int a = 0; a < 3; a++) { + accel[a] = accel_scale * static_cast(le16(s + 1 + 2 * a)) - calibration.accel_offset[a]; + gyro[a] = gyro_scale * static_cast(le16(s + 7 + 2 * a)) - calibration.gyro_offset[a]; + } + float rectified_accel[3], rectified_gyro[3]; + apply(calibration.accel_rectification, accel, rectified_accel); + apply(calibration.gyro_rectification, gyro, rectified_gyro); + ImuSample &out = report.samples[report.count++]; + apply(calibration.device_from_imu, rectified_accel, out.accel); + apply(calibration.device_from_imu, rectified_gyro, out.gyro); + } + return report; +} + +std::optional parseStatus(const uint8_t *r, size_t size) { + if (size < 2 || r[0] != kInStatus) return std::nullopt; + return r[1] != 0; +} + +std::vector> enableReports(bool on) { + std::vector> reports; + if (on) reports.push_back({0x07, 0xa3, 0x01}); // unknown, sent by Oculus's runtime first + reports.push_back({0x14, static_cast(on ? 1 : 0)}); // unknown + reports.push_back({0x0a, static_cast(on ? 2 : 0)}); // the radio to the controllers + if (!on) reports.push_back(screenReport(false)); + reports.push_back({0x02, static_cast(on ? 1 : 0)}); // the proximity sensor and the IMU + reports.push_back(cameraReport(on)); + return reports; +} + +std::vector cameraReport(bool on, uint16_t exposure, uint8_t gain, bool radio_sync) { + // {id, uvc enable, radio sync, exposure u16 x 5, gain x 5, 26 00 40, u32 x 5} (OpenHMD's rift_s_send_camera_report) + std::vector camera = {0x05, 0x01, 0x00, 0xb3, 0x36, 0xb3, 0x36, 0xb3, 0x36, 0xb3, 0x36, 0xb3, 0x36, 0xf0, + 0xf0, 0xf0, 0xf0, 0xf0, 0x26, 0x00, 0x40, 0x7a, 0x04, 0x00, 0x00, 0xa7, 0x04, 0x00, + 0x00, 0xa7, 0x04, 0x00, 0x00, 0xa5, 0x04, 0x00, 0x00, 0xa8, 0x04, 0x00, 0x00}; + camera[1] = on ? 1 : 0; + camera[2] = radio_sync ? 1 : 0; + for (int i = 0; i < kCameraCount; i++) { + camera[3 + 2 * i] = static_cast(exposure); + camera[4 + 2 * i] = static_cast(exposure >> 8); + camera[13 + i] = gain; + } + return camera; +} + +std::vector keepAliveReport() { + // report 0x93: a 3000 ms timeout, renewed every second + return {0x93, 0x01, 0xb8, 0x0b, 0x00, 0x00}; +} + +std::vector screenReport(bool on) { return {0x08, static_cast(on ? 1 : 0)}; } + +std::array firmwareRequest(uint8_t block, uint32_t offset, uint8_t length) { + std::array r{}; + r[0] = kReportFirmwareBlock; + r[2] = block; + r[3] = static_cast(offset); + r[4] = static_cast(offset >> 8); + r[5] = static_cast(offset >> 16); + r[6] = static_cast(offset >> 24); + r[7] = length; + return r; +} + +bool firmwareReplyMatches(const uint8_t *reply, size_t size, const std::array &request) { + return size >= 8 + static_cast(request[7]) && std::memcmp(reply, request.data(), 7) == 0; +} + +uint32_t firmwareBlockLength(const uint8_t *reply, size_t size) { + // the header: 8 bytes of checksum, then the length + if (size < 20) return 0; + const uint32_t length = le32(reply + 16); + if (length < 0xC || length == 0xFFFFFFFFu || length > 1u << 20) return 0; + return length; +} + +std::vector parseDevices(const uint8_t *r, size_t size) { + std::vector devices; + if (size < 3 || r[0] != kReportDevices) return devices; + size_t count = (size - 3) / 28; + if (count > r[2]) count = r[2]; + for (size_t i = 0; i < count && i < 7; i++) { + const uint8_t *d = r + 3 + 28 * i; + devices.push_back({le64(d), le32(d + 8)}); + } + return devices; +} + +uint64_t parseControllerReport(const uint8_t *r, size_t size, ControllerState &state) { + // {id, device u64, data length, flags, log 3, blocks...} + if (size < 10 || r[0] != kInController) return 0; + const uint64_t device = le64(r + 1); + size_t avail = r[9]; + if (device == 0 || avail < 4) return device; + if (avail > size - 10) avail = size - 10; + const uint8_t *p = r + 14; + avail -= 4; + while (avail > 1) { + size_t length = 0; + switch (p[0]) { + case 0x08: case 0x0c: case 0x0d: case 0x0e: length = 2; break; // one byte + case 0x1b: length = 4; break; // trigger and grip + case 0x22: length = 5; break; // stick + case 0x27: length = 5; break; // capacitive sensors + case 0x91: length = 19; break; // IMU + default: break; + } + if (length == 0 || avail < length) break; + switch (p[0]) { + case 0x0c: + state.buttons = p[1]; + break; + case 0x0d: + state.fingers = p[1]; + break; + case 0x1b: + state.trigger = static_cast((p[2] & 0x0f) << 8 | p[1]); + state.grip = static_cast((p[2] & 0xf0) >> 4 | (p[3] << 4)); + break; + case 0x22: + state.stick_x = static_cast(le16(p + 1)); + state.stick_y = static_cast(le16(p + 3)); + break; + case 0x91: + // {id, timestamp u32, u16, accel 3 x i16, gyro 3 x i16} + state.has_imu = true; + state.imu_timestamp = le32(p + 1); + for (int a = 0; a < 3; a++) { + state.accel[a] = static_cast(le16(p + 7 + 2 * a)); + state.gyro[a] = static_cast(le16(p + 13 + 2 * a)); + } + break; + default: + break; + } + p += length; + avail -= length; + } + return device; +} + +TouchValues touchValues(const ControllerState &s) { + constexpr uint8_t kAxStrong = 0x01, kByStrong = 0x02, kStickStrong = 0x04, kTriggerStrong = 0x08; + constexpr uint8_t kAxWeak = 0x10, kByWeak = 0x20, kStickWeak = 0x40, kTriggerWeak = 0x80; + TouchValues v; + v.primary = s.buttons & 0x01; + v.secondary = s.buttons & 0x02; + v.stick = s.buttons & 0x04; + v.menu = s.buttons & 0x08; + const uint8_t f = s.fingers; + v.primary_touch = (f & kAxStrong) || ((f & kAxWeak) && !(f & (kByStrong | kStickStrong))); + v.secondary_touch = (f & kByStrong) || ((f & kByWeak) && !(f & (kAxStrong | kStickStrong))); + v.stick_touch = (f & kStickStrong) || ((f & kStickWeak) && !(f & (kAxStrong | kByStrong))); + v.trigger_touch = f & (kTriggerWeak | kTriggerStrong); + v.trigger = 1.0f - s.trigger / 4096.0f; + v.grip = 1.0f - s.grip / 4096.0f; + if (s.trigger == 0 && s.grip == 0 && s.buttons == 0 && f == 0) v.trigger = v.grip = 0; // nothing reported yet + v.stick_x = s.stick_x / 32768.0f; + v.stick_y = s.stick_y / 32768.0f; + return v; +} + +std::array radioCommand(uint64_t device, const uint8_t *bytes, size_t size) { + // {0x12, device u64, command 52} + std::array r{}; + r[0] = kReportRadioCommand; + for (int i = 0; i < 8; i++) r[1 + i] = static_cast(device >> (8 * i)); + std::memcpy(r.data() + 9, bytes, size < 52 ? size : 52); + return r; +} + +std::optional parseControllerImuConfig(const uint8_t *bytes, size_t size) { + // {u32, length, accel limit u16, gyro limit u16, accel hz u16, gyro hz u16, accel scale f32, gyro scale f32} + if (size < 5 + 16 || bytes[4] < 16) return std::nullopt; + ControllerImuConfig c; + c.accel_scale = lef32(bytes + 5 + 8); + c.gyro_scale = lef32(bytes + 5 + 12); + if (!(c.accel_scale > 0) || !(c.gyro_scale > 0) || !std::isfinite(c.accel_scale) || !std::isfinite(c.gyro_scale)) + return std::nullopt; + return c; +} + +void controllerImu(const ControllerState &s, const ControllerImuConfig &config, float accel[3], float gyro[3]) { + for (int a = 0; a < 3; a++) { + accel[a] = kGravity * config.accel_scale * s.accel[a]; + gyro[a] = kDegrees * config.gyro_scale * s.gyro[a]; + } +} + +namespace { + +constexpr double kPanelWidth = 0.149760 / 2.0; // one eye's, in the view +constexpr double kPanelHeight = 0.093600; +constexpr double kLensSeparation = 0.074; +constexpr double kHorizontalFov = 105.0 * 3.14159265358979323846 / 180.0; + +// Monado's math_solve_triangle: a width split at w1, seen across theta_total from a point: the angles to either end +bool solveTriangle(double w_total, double w_1, double theta_total, double &theta_1, double &theta_2, double &d) { + const double w_2 = w_total - w_1; + const double u = w_2 / w_1; + const double v = std::tan(theta_total); + const double b = u + 1.0; + const double root = std::sqrt(b * b + 4 * u * v * v); + const double plus = std::atan((-b + root) / (2 * v)), minus = std::atan((-b - root) / (2 * v)); + if (plus > 0 && plus < theta_total) + theta_2 = plus; + else if (minus > 0 && minus < theta_total) + theta_2 = minus; + else + return false; + theta_1 = theta_total - theta_2; + d = w_2 / std::tan(theta_2); + return true; +} + +} // namespace + +Panotools panotools(int eye) { + Panotools p; + const double center_x = kLensSeparation / 2.0; + p.scale = static_cast(kPanelWidth - kLensSeparation / 2.0); + // the left eye's lens is the right eye's moved the other way + p.lens_center[0] = static_cast(eye == 0 ? kPanelWidth - center_x : center_x); + p.lens_center[1] = static_cast(kPanelHeight / 2.0); + p.viewport[0] = static_cast(kPanelWidth); + p.viewport[1] = static_cast(kPanelHeight); + return p; +} + +void fieldOfView(int eye, float &left, float &right, float &up, float &down) { + const double center_x = kLensSeparation / 2.0, center_y = kPanelHeight / 2.0; + double theta_1 = 0, theta_2 = 0, d = 1; + // Monado's: the right eye's horizontal center of projection is center_x, the left eye's the panel's width less it + solveTriangle(kPanelWidth, eye == 0 ? kPanelWidth - center_x : center_x, kHorizontalFov, theta_1, theta_2, d); + left = static_cast(-theta_1); + right = static_cast(theta_2); + down = static_cast(-std::atan(center_y / d)); + up = static_cast(std::atan((kPanelHeight - center_y) / d)); +} + +void computeDistortion(const Panotools &p, float u, float v, float out[3][2]) { + const float rx = (u * p.viewport[0] - p.lens_center[0]) / p.scale; + const float ry = (v * p.viewport[1] - p.lens_center[1]) / p.scale; + const float r = std::sqrt(rx * rx + ry * ry); + const float k = p.distortion_k[0] + p.distortion_k[1] * r + p.distortion_k[2] * r * r + + p.distortion_k[3] * r * r * r + p.distortion_k[4] * r * r * r * r; + for (int c = 0; c < 3; c++) { + const float dx = rx * k * p.scale * p.aberration_k[c], dy = ry * k * p.scale * p.aberration_k[c]; + out[c][0] = (dx + p.lens_center[0]) / p.viewport[0]; + out[c][1] = (dy + p.lens_center[1]) / p.viewport[1]; + } +} + +std::optional> parseCameraCalibration(std::string_view text) { + const auto root = JsonValue::parse(text); + if (!root || !root->isObject()) return std::nullopt; + const JsonValue &version = (*root)["FileFormat"]["Version"]; + if (!version.isString() || std::strtof(version.string().c_str(), nullptr) != 1.0f) return std::nullopt; + const JsonValue &cameras = (*root)["CameraCalibration"]; + if (!cameras.isArray() || cameras.size() != kCameraCount) return std::nullopt; + std::array out; + for (int i = 0; i < kCameraCount; i++) { + const JsonValue &camera = cameras[static_cast(i)]; + CameraCalibration &c = out[i]; + if (!camera["Id"].isString() || std::strtol(camera["Id"].string().c_str(), nullptr, 10) != i) return std::nullopt; + double size[2], projection[3], distortion[8]; + if (!camera["ImageSize"].numbers(size, 2) || !camera["DeviceFromCamera"].numbers(c.device_from_camera, 16) || + camera["Projection"]["Model"].string() != "PinholeSymmetric" || + !camera["Projection"]["Coefficients"].numbers(projection, 3) || + camera["Distortion"]["Model"].string() != "Fisheye62" || + !camera["Distortion"]["Coefficients"].numbers(distortion, 8)) + return std::nullopt; + if (!(size[0] > 0 && size[0] <= 640 && size[1] > 0 && size[1] <= 480) || !(projection[0] > 0)) return std::nullopt; + c.width = static_cast(size[0]); + c.height = static_cast(size[1]); + c.x_offset = static_cast(i) * c.width; // the pictures sit side by side in their ids' order + c.f = projection[0]; + c.cx = projection[1]; + c.cy = projection[2]; + std::memcpy(c.k, distortion, sizeof(c.k)); + c.p1 = distortion[6]; + c.p2 = distortion[7]; + } + return out; +} + +void fisheye62Project(const CameraCalibration &c, const double point[2], double pixel[2]) { + // Monado's fisheye62_undistort_func: a point on the plane z = 1 to its pixel + const double x = point[0], y = point[1]; + const double r = std::sqrt(x * x + y * y); + if (r < 1e-8) { + pixel[0] = c.f * x + c.cx; + pixel[1] = c.f * y + c.cy; + return; + } + const double theta = std::atan(r), theta2 = theta * theta; + const double xp = x * theta / r, yp = y * theta / r; + double radial = c.k[5]; + for (int i = 4; i >= 0; i--) radial = radial * theta2 + c.k[i]; + radial = radial * theta2 + 1; + const double dx = 2 * c.p1 * xp * yp + c.p2 * (theta2 + 2 * xp * xp); + const double dy = 2 * c.p2 * xp * yp + c.p1 * (theta2 + 2 * yp * yp); + pixel[0] = c.f * (xp * radial + dx) + c.cx; + pixel[1] = c.f * (yp * radial + dy) + c.cy; +} + +void kb4Project(const CameraCalibration &c, const std::array &k, const double point[2], double pixel[2]) { + const double x = point[0], y = point[1]; + const double r = std::sqrt(x * x + y * y); + if (r < 1e-8) { + pixel[0] = c.f * x + c.cx; + pixel[1] = c.f * y + c.cy; + return; + } + const double theta = std::atan(r), theta2 = theta * theta; + const double d = theta * (1 + theta2 * (k[0] + theta2 * (k[1] + theta2 * (k[2] + theta2 * k[3])))); + pixel[0] = c.f * x * d / r + c.cx; + pixel[1] = c.f * y * d / r + c.cy; +} + +// Newton's method from the pinhole guess +bool fisheye62Unproject(const CameraCalibration &c, const double pixel[2], double point[2]) { + point[0] = (pixel[0] - c.cx) / c.f; + point[1] = (pixel[1] - c.cy) / c.f; + for (int iteration = 0; iteration < 50; iteration++) { + double at[2]; + fisheye62Project(c, point, at); + const double ex = at[0] - pixel[0], ey = at[1] - pixel[1]; + if (ex * ex + ey * ey < 1e-14) return true; + const double h = 1e-7; + double ax[2], ay[2]; + const double px[2] = {point[0] + h, point[1]}, py[2] = {point[0], point[1] + h}; + fisheye62Project(c, px, ax); + fisheye62Project(c, py, ay); + const double j00 = (ax[0] - at[0]) / h, j10 = (ax[1] - at[1]) / h; + const double j01 = (ay[0] - at[0]) / h, j11 = (ay[1] - at[1]) / h; + const double det = j00 * j11 - j01 * j10; + if (!(std::fabs(det) > 1e-18)) return false; + point[0] -= (j11 * ex - j01 * ey) / det; + point[1] -= (j00 * ey - j10 * ex) / det; + } + double at[2]; + fisheye62Project(c, point, at); + return std::hypot(at[0] - pixel[0], at[1] - pixel[1]) < 1e-3; +} + +std::array kb4FromFisheye62(const CameraCalibration &c) { + // KB4 puts (x, y) at f (x, y) / r theta (1 + k1 theta^2 + .. + k4 theta^8) + c: linear in k, so the fit over the + // grid is a 4x4 least-squares solve + constexpr int kSteps = 21; + double ata[4][4] = {}, atb[4] = {}; + for (int yi = 0; yi < kSteps; yi++) + for (int xi = 0; xi < kSteps; xi++) { + const double pixel[2] = {static_cast(xi * (static_cast(c.width) - 1) / (kSteps - 1)), + static_cast(yi * (static_cast(c.height) - 1) / (kSteps - 1))}; + double point[2]; + if (!fisheye62Unproject(c, pixel, point)) continue; + const double r = std::sqrt(point[0] * point[0] + point[1] * point[1]); + if (r < 1e-8) continue; + const double theta = std::atan(r), theta2 = theta * theta; + for (int axis = 0; axis < 2; axis++) { + const double base = c.f * point[axis] / r * theta; // the pixel's offset with k = 0 + const double target = pixel[axis] - (axis ? c.cy : c.cx) - base; + double row[4]; + double power = theta2; + for (int i = 0; i < 4; i++, power *= theta2) row[i] = base * power; + for (int i = 0; i < 4; i++) { + atb[i] += row[i] * target; + for (int j = 0; j < 4; j++) ata[i][j] += row[i] * row[j]; + } + } + } + // Gaussian elimination with partial pivoting + double m[4][5]; + for (int i = 0; i < 4; i++) { + for (int j = 0; j < 4; j++) m[i][j] = ata[i][j]; + m[i][4] = atb[i]; + } + for (int col = 0; col < 4; col++) { + int pivot = col; + for (int r = col + 1; r < 4; r++) + if (std::fabs(m[r][col]) > std::fabs(m[pivot][col])) pivot = r; + if (!(std::fabs(m[pivot][col]) > 1e-30)) return {0, 0, 0, 0}; + for (int j = 0; j < 5; j++) std::swap(m[col][j], m[pivot][j]); + for (int r = 0; r < 4; r++) { + if (r == col) continue; + const double factor = m[r][col] / m[col][col]; + for (int j = col; j < 5; j++) m[r][j] -= factor * m[col][j]; + } + } + return {m[0][4] / m[0][0], m[1][4] / m[1][1], m[2][4] / m[2][2], m[3][4] / m[3][3]}; +} + +std::optional parseCameraFrameInfo(const uint8_t *grey, uint32_t width, uint32_t height) { + // Monado's parse_frame_data: 50 bytes, a bit per 8x8 block, read across the middle (row 4) of the top blocks + if (width != kCameraFrameWidth || height < 8 || !grey) return std::nullopt; + const uint8_t *row = grey + width * 4; + uint8_t raw[50] = {}; + for (uint32_t block = 0; block < 400; block++) + if (row[block * 8 + 4] > 128) raw[block / 8] |= static_cast(0x80 >> (block % 8)); + // {type, abcd, counter u16, QHWH u32, 7 zero, time u64 us, pair counter u32, exposure u16 x 5, 2 zero, gain x 5, + // zero, u16, face} + if (le16(raw + 1) != 0xabcd || le16(raw + 48) != 0xface) return std::nullopt; + CameraFrameInfo info; + info.tracking = (raw[0] & 0x80) != 0; + info.counter = le16(raw + 3); + info.time_us = le64(raw + 16); + for (int i = 0; i < kCameraCount; i++) { + info.exposure[i] = le16(raw + 28 + 2 * i); + info.gain[i] = raw[40 + i]; + } + return info; +} + +bool splitTrackingFrame(const uint8_t *grey, uint32_t width, uint32_t height, size_t stride, + const CameraFrameInfo &info, const std::array &calibration, + std::array, kCameraCount> &pictures) { + for (int i = 0; i < kCameraCount; i++) { + const CameraId id = kTrackerCameraOrder[i]; + const CameraCalibration &c = calibration[id]; + const uint32_t top = cameraYOffset(info.exposure[id]); + if (c.x_offset + c.width > width || top + c.height > height) return false; + std::vector &out = pictures[i]; + out.resize(static_cast(c.width) * c.height); + for (uint32_t y = 0; y < c.height; y++) + std::memcpy(out.data() + static_cast(y) * c.width, grey + (top + y) * stride + c.x_offset, c.width); + } + return true; +} + +uint32_t cameraYOffset(uint16_t exposure) { + const int offset = (static_cast(exposure) + 275) / 38; + return static_cast(offset < 8 ? 8 : offset > 375 ? 375 : offset); +} + +} // namespace metalvr::rift_s diff --git a/daemon/rift_s_protocol.h b/daemon/rift_s_protocol.h new file mode 100644 index 0000000..88bd688 --- /dev/null +++ b/daemon/rift_s_protocol.h @@ -0,0 +1,215 @@ +// The Oculus Rift S's USB side (docs/HEADSETS.md, phase I): its three HID interfaces (6 the headset and its IMU, +// 7 its status, 8 the Touch controllers' radio), the reports on them and the commands that turn it on. +// +// Nothing here touches a device: the reports are taken in and made as bytes, so that each can be checked without a +// headset (tests/headset_protocol_check.cpp). The layouts follow OpenHMD's Rift S driver (external/OpenHMD-rift/ +// src/drv_oculus_rift_s; Copyright 2019 Lucas Teske, 2019-2020 Jan Schmidt; BSL-1.0) and Monado's +// (src/xrt/drivers/rift_s; Copyright 2020-2022 Jan Schmidt; BSL-1.0). +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +namespace metalvr::rift_s { + +constexpr uint16_t kVendor = 0x2833; +constexpr uint16_t kProduct = 0x0051; +constexpr uint32_t kDisplayVendor = 0x3ed2; // "OVR" +constexpr uint32_t kDisplayProduct = 0x0012; +constexpr int kInterfaceHmd = 6, kInterfaceStatus = 7, kInterfaceControllers = 8; + +constexpr uint8_t kReportDeviceInfo = 0x06; +constexpr uint8_t kReportImuConfig = 0x09; +constexpr uint8_t kReportDevices = 0x0c; +constexpr uint8_t kReportFirmwareBlock = 0x4a; +constexpr uint8_t kReportRadioCommand = 0x12; +constexpr uint8_t kReportRadioResponse = 0x0b; +constexpr uint8_t kInHmd = 0x65, kInStatus = 0x66, kInController = 0x67; +constexpr uint8_t kFirmwareImuCalibration = 0x0e; + +// ---- the headset ---- + +struct DeviceInfo { + uint16_t width = 1440, height = 2560; // the panel as the computer sees it: portrait + uint8_t refresh_hz = 80; +}; +std::optional parseDeviceInfo(const uint8_t *report, size_t size); // report 6, its id first + +struct ImuConfig { + uint32_t hz = 1000; + float gyro_scale = 16.4f; // raw / scale = degrees/s + float accel_scale = 2048.0f; // raw * g / scale = m/s^2 +}; +std::optional parseImuConfig(const uint8_t *report, size_t size); // report 9, its id first + +// The IMU's calibration in firmware block 0x0e (JSON): offsets and rectifications, and where the IMU sits in the +// headset (its rotation into the headset's frame, which is OpenXR's: +x right, +y up, +z back). +struct ImuCalibration { + float device_from_imu[9] = {1, 0, 0, 0, 1, 0, 0, 0, 1}; // row-major rotation + float gyro_rectification[9] = {1, 0, 0, 0, 1, 0, 0, 0, 1}; + float gyro_offset[3] = {}; + float accel_rectification[9] = {1, 0, 0, 0, 1, 0, 0, 0, 1}; + float accel_offset[3] = {}; + float imu_position[3] = {}; // where the IMU sits in the headset, meters +}; +std::optional parseImuCalibration(std::string_view json); + +struct ImuSample { + float accel[3] = {}; // m/s^2, in the headset's frame + float gyro[3] = {}; // rad/s +}; + +struct HmdReport { + uint32_t timestamp_us = 0; // the first sample's + ImuSample samples[3]; + int count = 0; // the valid ones, in order +}; +// IN report 0x65: three IMU samples of the headset, rectified and turned into its frame +std::optional parseHmdReport(const uint8_t *report, size_t size, const ImuConfig &config, + const ImuCalibration &calibration); + +// IN report 0x66: whether the proximity sensor sees a face +std::optional parseStatus(const uint8_t *report, size_t size); + +// what turns the headset on and keeps it on (OpenHMD's rift_s_hmd_enable), as feature reports, in order +std::vector> enableReports(bool on); +std::vector keepAliveReport(); // every second: the headset stops after 3 without one +std::vector screenReport(bool on); +constexpr int kKeepAliveMs = 1000; + +// a request for part of a firmware block (report 0x4a); reading block `id` starts with its 12-byte header +std::array firmwareRequest(uint8_t block, uint32_t offset, uint8_t length); +// whether a 0x4a reply answers `request`, and its payload (bytes 8 on) +bool firmwareReplyMatches(const uint8_t *reply, size_t size, const std::array &request); +// the block's length from its header reply (bytes 16..19), 0 if invalid +uint32_t firmwareBlockLength(const uint8_t *header_reply, size_t size); + +// ---- the Touch controllers, through interface 8 ---- + +constexpr uint32_t kLeftController = 0x13001101, kRightController = 0x13011101; + +// report 0x0c: the radio's devices, each an id and a type +struct Device { + uint64_t id = 0; + uint32_t type = 0; +}; +std::vector parseDevices(const uint8_t *report, size_t size); + +struct ControllerState { + uint8_t buttons = 0; // 1 A/X, 2 B/Y, 4 stick, 8 menu/Oculus + uint8_t fingers = 0; // capacitive touches, strong and weak + uint16_t trigger = 0, grip = 0; // raw: 4095 released .. 0 pressed + int16_t stick_x = 0, stick_y = 0; + bool has_imu = false; + uint32_t imu_timestamp = 0; + int16_t accel[3] = {}, gyro[3] = {}; // raw; scaled by the controller's config, read over the radio +}; +// IN report 0x67: one controller's blocks, into `state`; returns its device id, or 0 for an empty report +uint64_t parseControllerReport(const uint8_t *report, size_t size, ControllerState &state); + +// Touch values from a state: 0..1 trigger and grip, -1..1 stick, and the touches (Monado's mapping) +struct TouchValues { + bool primary = false, secondary = false, stick = false, menu = false; // pressed + bool primary_touch = false, secondary_touch = false, stick_touch = false, trigger_touch = false; + float trigger = 0, grip = 0, stick_x = 0, stick_y = 0; +}; +TouchValues touchValues(const ControllerState &state); + +// The controller's IMU config, read over the radio: {0x32, 0x20, 0xe8, 0x03, 0...} to it as a radio command +std::array radioCommand(uint64_t device, const uint8_t *bytes, size_t size); +constexpr uint8_t kControllerConfigRequest[10] = {0x32, 0x20, 0xe8, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}; +struct ControllerImuConfig { + float accel_scale = 0, gyro_scale = 0; // raw * accel_scale * g, raw * gyro_scale degrees/s +}; +// the response bytes of the config command (after the radio reply's 3-byte header) +std::optional parseControllerImuConfig(const uint8_t *bytes, size_t size); +// a controller's raw IMU sample in its frame, scaled; Touch's frame is its own, the grip pose 40 degrees up from it +void controllerImu(const ControllerState &state, const ControllerImuConfig &config, float accel[3], float gyro[3]); + +// ---- the lenses and the panel ---- +// +// No correct Rift S lens model has been published: OpenHMD's and Monado's drivers both use the CV1's panotools +// coefficients with the Rift S's panel size, marked as wrong. MetalVR starts from Monado's numbers +// (rift_s_hmd.c) and leaves them to be tuned on a headset (docs/HEADSETS.md, phase I). +// +// The panel is one portrait 1440x2560 display: the left eye's view is its top half, the right eye's its bottom half, +// each turned a quarter right (Monado's u_device_rotation_right): an eye's view is 1280 wide and 1440 high. + +struct Panotools { + float distortion_k[5] = {0.819f, -0.241f, 0.324f, 0.098f, 0.0f}; + float aberration_k[3] = {0.9952420f, 1.0f, 1.0008074f}; + float scale = 0; // meters + float lens_center[2] = {}; // meters, in the eye's view + float viewport[2] = {}; // meters +}; + +// Monado's values for each eye: the panel's 149.76 x 93.6 mm, lenses 74 mm apart, 105 degrees across +Panotools panotools(int eye); +// each eye's field of view from those (Monado's math_compute_fovs); radians, OpenXR's signs +void fieldOfView(int eye, float &left, float &right, float &up, float &down); +// where each colour of the view's point (u, v) (0..1 across and down, as the application renders it) is in the +// rendered image (Monado's u_compute_distortion_panotools) +void computeDistortion(const Panotools &p, float u, float v, float out[3][2]); + +// ---- the cameras (camera tracking, docs/HEADSETS.md phase I) ---- +// +// The five cameras reach the computer as one UVC video device, "Rift S Sensor", whose frames are the five pictures +// side by side, 8-bit grey, though the device calls them YUYV at half the width (Monado's rift_s_camera.c). Every +// other frame is exposed for tracking the room, the others for the controllers' LEDs; a row of 8x8 blocks at the top +// of each frame says which, with its time on the headset's clock and each camera's exposure. + +constexpr int kCameraCount = 5; +enum CameraId { kCameraTop = 0, kCameraSideLeft = 1, kCameraFrontRight = 2, kCameraFrontLeft = 3, kCameraSideRight = 4 }; +// the order the tracker is given them in (Monado's CAM_IDX_TO_ID): the front pair first, as its stereo pair +constexpr CameraId kTrackerCameraOrder[kCameraCount] = {kCameraFrontLeft, kCameraFrontRight, kCameraSideLeft, + kCameraSideRight, kCameraTop}; +constexpr uint8_t kFirmwareCameraCalibration = 0x0f; +constexpr uint32_t kCameraFrameWidth = 50 * 8 * 8; // the frame's width as 8-bit grey: five 640-wide pictures + +// one camera's calibration from firmware block 0x0f (JSON) +struct CameraCalibration { + uint32_t width = 640, height = 480; + uint32_t x_offset = 0; // where its picture starts in the frame + double device_from_camera[16] = {}; // row-major isometry, in the headset's frame (meters) + double f = 0, cx = 0, cy = 0; // PinholeSymmetric, pixels + double k[6] = {}, p1 = 0, p2 = 0; // Fisheye62 +}; +std::optional> parseCameraCalibration(std::string_view json); + +// The tracker takes a Kannala-Brandt lens (KB4) and the headset keeps Fisheye62: fitted as Monado's +// rift_s_get_cam_calib does, over a 21x21 grid of the picture. +void fisheye62Project(const CameraCalibration &c, const double point[2], double pixel[2]); +// the point on the plane z = 1 that Fisheye62 puts at `pixel`; false if it does not converge +bool fisheye62Unproject(const CameraCalibration &c, const double pixel[2], double point[2]); +std::array kb4FromFisheye62(const CameraCalibration &c); +void kb4Project(const CameraCalibration &c, const std::array &kb4, const double point[2], double pixel[2]); + +// the 8x8-block row at the top of a frame (8-bit grey, kCameraFrameWidth wide) +struct CameraFrameInfo { + bool tracking = false; // exposed for the room (else for the controllers) + uint16_t counter = 0; + uint64_t time_us = 0; // the headset's clock, as its IMU's + uint16_t exposure[kCameraCount] = {}; + uint8_t gain[kCameraCount] = {}; +}; +std::optional parseCameraFrameInfo(const uint8_t *grey, uint32_t width, uint32_t height); +// how far down the frame a camera's picture starts, from its exposure (Monado's get_y_offset) +uint32_t cameraYOffset(uint16_t exposure); +// report 5: the cameras on or off, with every camera's exposure (38..14022) and gain (16..255); `radio_sync` as +// Monado sends it while it reads the cameras +std::vector cameraReport(bool on, uint16_t exposure = 0x36b3, uint8_t gain = 0xf0, bool radio_sync = false); +constexpr uint16_t kMinExposure = 38, kMaxExposure = 14022; +constexpr uint8_t kMinGain = 16; + +// a tracking frame's five pictures, each `width * height` bytes packed, in kTrackerCameraOrder; false if the frame +// is too small for where the calibration and exposures put them (Monado's rift_s_camera_extract_frame) +bool splitTrackingFrame(const uint8_t *grey, uint32_t width, uint32_t height, size_t stride, + const CameraFrameInfo &info, const std::array &calibration, + std::array, kCameraCount> &pictures); + +} // namespace metalvr::rift_s diff --git a/daemon/rift_s_tracking.cpp b/daemon/rift_s_tracking.cpp new file mode 100644 index 0000000..22d05f7 --- /dev/null +++ b/daemon/rift_s_tracking.cpp @@ -0,0 +1,124 @@ +#include "rift_s_tracking.h" + +#include +#include + +namespace metalvr { + +using namespace rift_s; + +std::vector riftSTrackerCameras(const std::array &cameras, + const ImuCalibration &imu) { + std::vector out; + for (CameraId id : kTrackerCameraOrder) { + const CameraCalibration &c = cameras[id]; + TrackerCamera t; + t.width = c.width; + t.height = c.height; + t.fx = t.fy = c.f; + t.cx = c.cx; + t.cy = c.cy; + t.kb4 = kb4FromFisheye62(c); + t.hz = 30; // every other frame of the 60 Hz feed is for tracking (Monado's CAMERA_FREQUENCY) + // the IMU's frame has the headset's axes, at the IMU: the camera's rotation is the headset's, its position + // moved by where the IMU is + std::copy(c.device_from_camera, c.device_from_camera + 16, t.imu_from_camera); + for (int a = 0; a < 3; a++) t.imu_from_camera[4 * a + 3] -= imu.imu_position[a]; + out.push_back(t); + } + return out; +} + +std::unique_ptr RiftSTracking::start(const ImuConfig &config, const ImuCalibration &imu, + const std::array &cameras, + std::string *why) { + TrackerImu tracker_imu; + tracker_imu.hz = config.hz; + std::copy(imu.imu_position, imu.imu_position + 3, tracker_imu.position); + auto tracker = startCameraTracking(riftSTrackerCameras(cameras, imu), tracker_imu, why); + if (!tracker) return nullptr; + return std::make_unique(std::move(tracker), config, cameras); +} + +RiftSTracking::RiftSTracking(std::unique_ptr tracker, const ImuConfig &config, + const std::array &cameras) + : tracker_(std::move(tracker)), config_(config), cameras_(cameras) {} + +void RiftSTracking::imu(const HmdReport &report) { + int64_t first_us; + { + std::lock_guard guard(mutex_); + if (have_imu_ && report.timestamp_us < last_imu_us_ && last_imu_us_ - report.timestamp_us > 0x80000000u) + imu_high_++; // the 32-bit clock wrapped (every 71 minutes) + have_imu_ = true; + last_imu_us_ = report.timestamp_us; + first_us = (imu_high_ << 32) + report.timestamp_us; + } + const int64_t tick_ns = 1'000'000'000 / std::max(config_.hz, 1); + int64_t last_us = first_us; + for (int i = 0; i < report.count; i++) { + const int64_t t_ns = first_us * 1000 + i * tick_ns; + tracker_->pushImu(t_ns, report.samples[i].accel, report.samples[i].gyro); + last_us = t_ns / 1000; + } + std::lock_guard guard(mutex_); + latest_imu_us_ = last_us; +} + +bool RiftSTracking::frame(const uint8_t *grey, uint32_t width, uint32_t height, size_t stride) { + const auto info = parseCameraFrameInfo(grey, width, height); + if (!info || !info->tracking) return false; + int64_t imu_us; + { + std::lock_guard guard(mutex_); + if (!have_imu_) return false; + imu_us = latest_imu_us_; + } + // The frame's time is 64-bit and the IMU's 32-bit on the same clock: the frame's low 32 bits, placed in the + // 2^32 window nearest the IMU's time (Monado's camera_ts_offset) + const int64_t low = static_cast(info->time_us & 0xffffffffu); + int64_t frame_us = (imu_us & ~static_cast(0xffffffff)) + low; + if (frame_us - imu_us > 0x80000000LL) frame_us -= 0x100000000LL; + if (imu_us - frame_us > 0x80000000LL) frame_us += 0x100000000LL; + if (!splitTrackingFrame(grey, width, height, stride, *info, cameras_, pictures_)) return false; + std::vector pointers; + for (const auto &picture : pictures_) pointers.push_back(picture.data()); + tracker_->pushFrames(frame_us * 1000, pointers); + frames_tracked_++; + adjustExposure(pictures_[0]); + return true; +} + +bool RiftSTracking::pose(TrackedPose &out, int64_t *time_ns) { return tracker_->pose(out, time_ns); } + +void RiftSTracking::adjustExposure(const std::vector &picture) { + // an exposure takes effect two frames after it is sent: change it every fourth tracking frame + if (++frames_since_exposure_ < 4 || picture.empty()) return; + frames_since_exposure_ = 0; + uint64_t sum = 0; + for (size_t i = 0; i < picture.size(); i += 7) sum += picture[i]; + const double mean = static_cast(sum) / static_cast((picture.size() + 6) / 7); + const double target = 255.0 / 3.0; + if (std::fabs(mean - target) < 12) return; + const double factor = std::clamp(target / std::max(mean, 1.0), 0.5, 2.0); + std::lock_guard guard(mutex_); + const double wanted = exposure_ * factor; + if (wanted > kMaxExposure && gain_ < 255) { + exposure_ = kMaxExposure; + gain_ = static_cast(std::min(255.0, gain_ * factor)); + } else if (factor < 1 && gain_ > 127) { + gain_ = static_cast(std::max(127.0, gain_ * factor)); + } else { + exposure_ = static_cast(std::clamp(wanted, double(kMinExposure), double(kMaxExposure))); + } +} + +bool RiftSTracking::exposureChange(uint16_t &exposure, uint8_t &gain) { + std::lock_guard guard(mutex_); + if (exposure_ == sent_exposure_ && gain_ == sent_gain_) return false; + sent_exposure_ = exposure = exposure_; + sent_gain_ = gain = gain_; + return true; +} + +} // namespace metalvr diff --git a/daemon/rift_s_tracking.h b/daemon/rift_s_tracking.h new file mode 100644 index 0000000..af31e89 --- /dev/null +++ b/daemon/rift_s_tracking.h @@ -0,0 +1,67 @@ +// The Rift S's position from its cameras (docs/HEADSETS.md, phase I), as Monado's rift_s_tracker.c does it: the +// headset's IMU and its tracking frames on the headset's clock, given to Basalt (camera_tracking.h) with the +// cameras' calibration from the headset's firmware. Nothing here touches the device: the backend +// (backend_rift_s.mm) gives it the reports and frames it receives, and sends the exposure it asks for, so that the +// same path runs on synthetic input (tests/rift_s_tracking_check.cpp). +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "camera_tracking.h" +#include "rift_s_protocol.h" + +namespace metalvr { + +// The cameras as Basalt is given them, in rift_s::kTrackerCameraOrder: each lens fitted to KB4 and placed from the +// IMU, whose frame is the headset's axes at the IMU's position. +std::vector riftSTrackerCameras(const std::array &cameras, + const rift_s::ImuCalibration &imu); + +class RiftSTracking { +public: + // Basalt, started for this headset; null with `why` set if it cannot be + static std::unique_ptr start(const rift_s::ImuConfig &config, const rift_s::ImuCalibration &imu, + const std::array &cameras, + std::string *why); + // The same with a tracker given (for checking the timing and frame handling without Basalt) + RiftSTracking(std::unique_ptr tracker, const rift_s::ImuConfig &config, + const std::array &cameras); + + // one IMU report, as parsed (its samples in the headset's frame) + void imu(const rift_s::HmdReport &report); + // one frame from the camera device: 8-bit grey, rift_s::kCameraFrameWidth wide. Controller frames and frames + // before the IMU's first are skipped. Returns whether it was given to the tracker. + bool frame(const uint8_t *grey, uint32_t width, uint32_t height, size_t stride); + // where the headset is, in the tracker's room (OpenXR's axes) + bool pose(TrackedPose &out, int64_t *time_ns = nullptr); + // The exposure and gain the cameras should take, when they changed since last asked (an auto exposure aiming the + // front-left picture at a third of full brightness, as tracking prefers dark pictures with little blur) + bool exposureChange(uint16_t &exposure, uint8_t &gain); + + uint64_t framesTracked() const { return frames_tracked_; } + +private: + void adjustExposure(const std::vector &picture); + + std::unique_ptr tracker_; + rift_s::ImuConfig config_; + std::array cameras_; + std::mutex mutex_; + // the IMU's 32-bit microsecond clock, extended + bool have_imu_ = false; + uint32_t last_imu_us_ = 0; + int64_t imu_high_ = 0; // multiples of 2^32 us + int64_t latest_imu_us_ = 0; + uint64_t frames_tracked_ = 0; + std::array, rift_s::kCameraCount> pictures_; + uint16_t exposure_ = 6000, sent_exposure_ = 0; + uint8_t gain_ = 127, sent_gain_ = 0; + int frames_since_exposure_ = 0; +}; + +} // namespace metalvr diff --git a/daemon/server.mm b/daemon/server.mm index e7410b9..e99f878 100644 --- a/daemon/server.mm +++ b/daemon/server.mm @@ -288,7 +288,8 @@ uint32_t endFrame(Client &client, const mvr_frame_end_request &request, const mv if (!params.frame_width || !params.frame_height) return MVR_OK; const bool displays = driver->displaysFrames(); - if (!compositor->configure(params.frame_width, params.frame_height) || (!displays && !encoder->configure(params))) + if (!compositor->configure(params.frame_width, params.frame_height, displays ? Foveation{} : params.foveation) || + (!displays && !encoder->configure(params))) return MVR_ERR_INTERNAL; const int64_t begin = nowNs(); @@ -442,6 +443,16 @@ bool dispatch(Client &client, const mvr_header &header, const std::vector 0 ? request.time_ns : nowNs(); + driver->hands(response.time_ns, response.hands); + response.result = MVR_OK; + return respond(client, header.type, response); + } case MVR_MSG_CLOCK: { mvr_clock_response response{}; response.server_time_ns = nowNs(); diff --git a/daemon/vit/vit_interface.h b/daemon/vit/vit_interface.h new file mode 100644 index 0000000..c53e5a1 --- /dev/null +++ b/daemon/vit/vit_interface.h @@ -0,0 +1,497 @@ +// Copyright 2023-2024, Collabora, Ltd. +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief Visual-Intertial Tracking interface header. + * @author Mateo de Mayo + * @author Simon Zeni + * @author Jakob Bornecrantz + * + * This file contains the declaration of the @ref vit_tracker struct. This + * header is intended to appear in both consumers (e.g., Monado) and external + * visual-inertial tracking (VIT) systems (e.g., Basalt). The implementation of + * `vit_interface` is provided by the external system. Additional data types are + * declared for the communication between consumers and the system. + */ + +#pragma once + +#if defined(__cplusplus) +extern "C" { +#endif + +#include +#include + +//! Compatibility with these values should be checked against @ref vit_api_get_version. +#define VIT_HEADER_VERSION_MAJOR 2 //!< API Breakages +#define VIT_HEADER_VERSION_MINOR 0 //!< Backwards compatible API changes +#define VIT_HEADER_VERSION_PATCH 1 //!< Backw. comp. .h-implemented changes + +#define VIT_CAMERA_CALIBRATION_DISTORTION_MAX_COUNT 32 + +/*! + * Result type used by the VIT system + * + * 0 is @ref VIT_SUCCESS, positive values are non fatal results and negative values are errors. + */ +typedef enum vit_result { + /*! + * Operation suceeded + */ + VIT_SUCCESS = 0, + + /*! + * The operation is not available on the current version + */ + VIT_ERROR_INVALID_VERSION = -1, + + /*! + * The operation received an invalid value. + */ + VIT_ERROR_INVALID_VALUE = -2, + + /*! + * The operation was not able to allocate memory to pursue the operation. + */ + VIT_ERROR_ALLOCATION_FAILURE = -3, + + /*! + * The operation requires an extension that is not supported. + */ + VIT_ERROR_NOT_SUPPORTED = -4, + + /*! + * The operation requires an extension that is not enabled. + */ + VIT_ERROR_NOT_ENABLED = -5, +} vit_result_t; + +/*! + * Image formats. + */ +typedef enum vit_image_format { + //! 8-bit luminance + VIT_IMAGE_FORMAT_L8 = 1, + //! 16-bit luminance + VIT_IMAGE_FORMAT_L16 = 2, + //! 24-bit rgb, tightly packed. + VIT_IMAGE_FORMAT_R8G8B8 = 3, +} vit_image_format_t; + +/*! + * Camera calibration types. + */ +typedef enum vit_camera_distortion { + //! No distortion (pre-disotorted). + VIT_CAMERA_DISTORTION_NONE, + //! Distortion radial-tangential (OpenCV), 4 parameters. + VIT_CAMERA_DISTORTION_RT4, + //! Distortion radial-tangential (OpenCV), 5 parameters. + VIT_CAMERA_DISTORTION_RT5, + //! Distortion radial-tangential (OpenCV), 8 parameters. + VIT_CAMERA_DISTORTION_RT8, + //! Distortion Kannala-Brandt (OpenCV fisheye), 4 parameters. + VIT_CAMERA_DISTORTION_KB4, +} vit_camera_distortion_t; + +/*! + * Optional functionality a tracker can provide. + */ +typedef enum vit_tracker_extension { + //! Allows to specify camera calibration data programatically + VIT_TRACKER_EXTENSION_ADD_CAMERA_CALIBRATION, + //! Allows to specify IMU calibration data programatically + VIT_TRACKER_EXTENSION_ADD_IMU_CALIBRATION, + //! Optionally provide timing information with the returned poses + VIT_TRACKER_EXTENSION_POSE_TIMING, + //! Optionally provide feature count information with the returned poses + VIT_TRACKER_EXTENSION_POSE_FEATURES, + //! Number of extensions + VIT_TRACKER_EXTENSION_COUNT, +} vit_tracker_extension_t; + +/*! + * Set of extensions. Field order matches enumerators in @ref vit_tracker_extension. + */ +typedef struct vit_tracker_extension_set { + bool has_add_camera_calibration; + bool has_add_imu_calibration; + bool has_pose_timing; + bool has_pose_features; +} vit_tracker_extension_set_t; + +/*! + * Visual-Inertial Tracking interface, opaque type. + */ +struct vit_tracker; +typedef struct vit_tracker vit_tracker_t; + +/*! + * Pose interface, opaque type. + */ +struct vit_pose; +typedef struct vit_pose vit_pose_t; + +/*! + * Names of the timestamps returned by `vit_pose_get_timings` + */ +typedef struct vit_tracker_timing_titles { + uint32_t count; //! Number of titles + const char **titles; //! Names of the measures timestamps +} vit_tracker_timing_titles; + +/*! + * Parameters for creating the system pipeline. + */ +typedef struct vit_config { + //! Path to a implementation-specific config file. If null, use defaults. + const char *file; + + //! Number of cameras to use. Required. + uint32_t cam_count; + + //! Number of IMU to use. Required. + uint32_t imu_count; + + //! If supported, whether to open the system's UI. + bool show_ui; +} vit_config_t; + +/*! + * IMU sample type feed into VIT tracker + */ +typedef struct vit_imu_sample { + //! In nanoseconds + int64_t timestamp; + + //! Acceleration in meters per second squared (m/s²) + float ax, ay, az; + + //! Gyro in radians per second (rad/s) + float wx, wy, wz; +} vit_imu_sample_t; + +/*! + * Region in image space that this mask covers. + */ +typedef struct vit_mask { + //! In pixels. + float x, y, w, h; +} vit_mask_t; + +/*! + * Image sample type feed into VIT tracker + * + * Can easily be converted into an OpenCV Matrix for processing. + */ +typedef struct vit_img_sample { + uint32_t cam_index; + + //! In nanoseconds, must increase monotonically. + int64_t timestamp; + + // !Image data + uint8_t *data; + uint32_t width, height; + uint32_t stride, size; + vit_image_format_t format; + + //! Regions to ignore + uint32_t mask_count; + vit_mask_t *masks; +} vit_img_sample_t; + +/*! + * Data that is always returned from tracker. + */ +typedef struct vit_pose_data { + //! In nanoseconds, must increase monotonically. + int64_t timestamp; + + //! Position vector. + float px, py, pz; + + //! Orientation quaternion. + float ox, oy, oz, ow; + + //! Linear velocity. + float vx, vy, vz; +} vit_pose_data_t; + +/*! + * Result of pose timing request function. + */ +typedef struct vit_pose_timing { + uint32_t count; + const int64_t *timestamps; +} vit_pose_timing_t; + +/*! + * One single feature, element of @ref vit_pose_features result. + */ +typedef struct vit_pose_feature { + int64_t id; + float u, v, depth; +} vit_pose_feature_t; + +/*! + * Result of pose feature request function. + */ +typedef struct vit_pose_features { + uint32_t count; + const struct vit_pose_feature *features; +} vit_pose_features_t; + +/*! + * Container of parameters for a pinhole camera calibration (fx, fy, cx, cy) + * with an optional distortion. + * + *`distortion_model` and its corresponding `distortion` parameters are not + * standardized in this struct to facilitate implementation prototyping. + */ +typedef struct vit_camera_calibration { + uint32_t camera_index; // 1 cameras, the N frames must be pushed following @ref cam_index order. + * The bundle of N frames must have the same timestamps. + */ +vit_result_t vit_tracker_push_img_sample(vit_tracker_t *tracker, const vit_img_sample_t *sample); + +/*! + * Adds an inertial measurement unit calibration to the tracker. The tracker must not be started. + * + * Returns `VIT_ERROR_NOT_SUPPORTED` if the tracker doesn't offer the extension. + * + * @see vit_tracker_get_supported_extensions + * @see vit_tracker_extension_t + */ +vit_result_t vit_tracker_add_imu_calibration(vit_tracker_t *tracker, const vit_imu_calibration_t *calibration); + +/*! + * Adds a camera calibration to the tracker. The tracker must not be started. + * + * Returns `VIT_ERROR_NOT_SUPPORTED` if the tracker doesn't offer the extension. + * + * @see vit_tracker_get_supported_extensions + * @see vit_tracker_extension_t + */ +vit_result_t vit_tracker_add_camera_calibration(vit_tracker_t *tracker, const vit_camera_calibration_t *calibration); + +/*! + * Get the pose from the front of the tracking queue from the VIT tracker + * + * This function must be non-blocking and consumed by a single consummer. + * + * If @p out_pose is NULL the pose will be immediately destroyed. + * + * @param[out] out_pose Pose returned to the caller, NULL if there is not pose available or on error. + */ +vit_result_t vit_tracker_pop_pose(vit_tracker_t *tracker, vit_pose_t **out_pose); + +/*! + * Get the titles of the timestamps measured by the pose timings. + * + * Returns `VIT_ERROR_NOT_SUPPORTED` if the tracker doesn't offer the pose timing extension. + */ +vit_result_t vit_tracker_get_timing_titles(const vit_tracker_t *tracker, vit_tracker_timing_titles *out_titles); + +/*! + * Destroys a pose. All of the data, timing and features associated to it will be invalidated. + */ +void vit_pose_destroy(vit_pose_t *pose); + +/*! + * Gets the data form a given `vit_pose_t`. + * + * The data becomes invalid when the associated pose gets destroyed. + */ +vit_result_t vit_pose_get_data(const vit_pose_t *pose, vit_pose_data_t *out_data); + +/*! + * Gets the timing form a given `vit_pose_t`. + * + * The timing data becomes invalid when the associated pose gets destroyed. + */ +vit_result_t vit_pose_get_timing(const vit_pose_t *pose, vit_pose_timing_t *out_timing); + +/*! + * Gets the features form a given `vit_pose_t`. + * + * The features data becomes invalid when the associated pose gets destroyed. + */ +vit_result_t vit_pose_get_features(const vit_pose_t *pose, uint32_t camera_index, vit_pose_features_t *out_features); + +#endif + +#if defined(__cplusplus) +} +#endif diff --git a/docs/ALVR.md b/docs/ALVR.md index 73ebafc..07f7a77 100644 --- a/docs/ALVR.md +++ b/docs/ALVR.md @@ -57,6 +57,15 @@ The header compiles as C++20 under `-Wall -Wextra`. Under `-Wpedantic` clang rep - `0002-web-server-listens-on-loopback.patch` binds ALVR's dashboard API (TCP 8082) to 127.0.0.1 instead of every interface. That API takes unauthenticated requests that trust a client or change any setting, so left on the network it would let any machine there decide which headsets may connect. +- `0003-client-runs-on-the-steam-frame.patch` changes ALVR's headset client, not the server. Valve's OpenXR runtime + on the Steam Frame has no `XR_KHR_convert_timespec_time`, which the stock client requires, so it shows a blank + screen there; patched, the client asks for that extension only when the runtime has it, and does without the + runtime's clock when it does not (every place that reads it already handles having none). It also names a headset + whose maker is Valve "Steam Frame", so MetalVR recognizes it. `scripts/build-alvr.sh` applies it with the others but + builds only `server_core`; `scripts/build-alvr-client.sh` builds the client as + `build/alvr-client/alvr_client_android.apk` (fetching the Android NDK, SDK, a JDK 17 and cargo-apk into + `build/android`; Google's SDK license is shown for you to accept), to install with + `adb install -r build/alvr-client/alvr_client_android.apk` on a Steam Frame in developer mode. Nothing else in ALVR is changed. @@ -94,6 +103,42 @@ The first connection after the negotiated stream changes (always the very first the client "the streamer is restarting", logs `Cannot restart SteamVR. No dashboard process found`, and the client connects again about two seconds later. That is ALVR expecting SteamVR to restart; nothing needs to. +## Which headset it is + +The stock client tells ALVR what it runs on, and ALVR keeps that in `session.json` as the client's +`display_name`: "Quest 1", "Quest 2", "Quest 3", "Quest 3S", "Quest Pro" (ALVR's `alvr_system_info`). The driver +reads it when the headset connects, and before that from what the trusted headset reported last time, and names +the headset by it (`metalvr_headsets.h`): OpenXR's `systemName` and `vendorId`, SteamVR's model and controllers, +the interaction profile games are given first (Touch Plus on a Quest 3, 3S or Pro), the refresh rates +`XR_FB_display_refresh_rate` lists, and the warning when the negotiated stream is more than the headset's video +decoder is known to take (ALVR's default is too much for a Quest 1). The name is matched whole: a "Quest 3S" is +never taken for a "Quest 3". Anything else (a Pico, a Vive Focus, "Android (unknown)") is an unknown headset with +Touch-shaped controllers, which is what ALVR's client sends for every headset; never a Rift. + +`--headset quest-2` (or `quest-1`, `quest-3`, `quest-3s`, `quest-pro`, `steam-frame`) names the headset instead, +for a client that does not say: the stock ALVR client on a Steam Frame reports "Android (unknown)"; the client built +with `0003-client-runs-on-the-steam-frame.patch` reports "Steam Frame", which MetalVR names by itself. A headset that reports +another model than the one named is logged, and the one named is used. A session MetalVR creates for a named +headset asks for its usual PC VR rate (90 Hz; 72 Hz for a Quest 1) instead of ALVR's 72 Hz; an existing session +keeps what its settings say. + +The headset's battery and its controllers' (`ALVR_EVENT_BATTERY`) go to OpenXR applications through the protocol +and to SteamVR as battery levels; the size of its play-area boundary (`ALVR_EVENT_PLAYSPACE_SYNC`) is the STAGE +space's bounds (`xrGetReferenceSpaceBoundsRect`). + +## A headset on USB + +ALVR reaches a headset on USB through adb, under the client name `client.wired`: + +``` +metalvrd --trust client.wired +``` + +The headset needs USB debugging on (its developer mode), and the Mac needs Android's platform-tools +(`brew install android-platform-tools`), or `--adb PATH` naming an `adb`. Without one, metalvrd does not start: +ALVR would otherwise download Google's platform-tools by itself at run time, which MetalVR does not allow. ALVR's +wired mode forwards its ports over adb; it has not been tried from a Mac with a headset. + ## How the C API is used Initialization, once per process, in `start()`: @@ -220,7 +265,7 @@ starts, and `start()` fails if ALVR's effective settings still disagree: | setting | pinned to | why | |---|---|---| -| `video.foveated_encoding` | off | the client would undo a distortion the frame never had | +| `video.foveated_encoding` | off, or on with `--foveated-encoding` | the client undoes the squeeze, so the frame must have it exactly when the client is told it does (below) | | `video.encoder_config.use_full_range` (+ server override) | limited range | with full range the client rescales the decoded picture, assuming its decoder treated the stream as limited | | `use_10bit`, `hdr.enable_hdr`, `encoding_gamma` (+ server overrides) | 8-bit, off, 1.0 | not produced | | `audio.game_audio`, `audio.microphone` | off | not implemented | @@ -235,12 +280,27 @@ and rewrites the file), or through ALVR's dashboard API from the same machine. **The encoder must therefore produce video-range (16–235) 8-bit 4:2:0.** +### Foveated encoding + +`--foveated-encoding` turns ALVR's foveated encoding on: each eye of the composed frame is squeezed toward its edges +before it is encoded, its center kept whole, and the client stretches it back, so the headset decodes fewer pixels +(a 2144x2336 eye is sent as 1280x1216 with ALVR's default shape). The shape is ALVR's own +`video.foveated_encoding` settings, read from the negotiated stream; metalvrd's squeeze is ALVR's server shader +ported (`daemon/foveation.h`, the compositor's `squeeze_fragment`), and checked against ALVR's client's stretch: +`tests/headset_protocol_check.cpp` (the formulas, both ways), `metalvrd --foveation-check` (the GPU's squeeze +against the formula) and the `native-foveated` test (a squeezed stream end to end). The focus is fixed at the +eye's center as ALVR shapes it; following the eye's gaze (a Steam Frame's eye tracking) would need a client that +sends the gaze and stretches around it. + ## Not implemented - Audio in either direction. -- Hand tracking (skeletons), body, face and eye tracking, battery levels, play-area bounds. +- Body, face and eye tracking. (Bare hands are passed on: the joints ALVR's client sends are OpenXR's + `XR_EXT_hand_tracking`.) +- Changing the refresh rate: `XR_FB_display_refresh_rate` lists the headset's rates, and the stream stays at the + one negotiated. - AV1 (HEVC is sent instead). -- Foveated encoding, 10-bit, HDR, server-side color correction. +- Gaze-following foveation, 10-bit, HDR, server-side color correction. - ALVR's frame pacing hint (`alvr_duration_until_next_vsync`). - Prediction on the server: poses are the client's. - Trusting a headset while metalvrd runs: ALVR reads its client list once per process. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 5c9ecc9..92037bf 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -20,7 +20,9 @@ MetalVR has two halves and one protocol between them. ├─ encoder: VideoToolbox H.264 / HEVC, frame in / frame out └─ driver: where tracking comes from and video goes ├─ sim deterministic headset, optional stream dump - └─ alvr ALVR's server_core → stock ALVR client on the headset + ├─ alvr ALVR's server_core → stock ALVR client on the headset + └─ rift, lighthouse, psvr2 + a headset on the Mac's own display and USB (RIFT.md) ``` ## One runtime, two targets @@ -113,19 +115,34 @@ The runtime's state machine follows the server's flags, not assumptions: ## Input -The server reports one kind of controller: two hands, laid out like Touch (`mvr_controller_state`). Every -interaction profile an application suggests is read as a naming of those physical inputs — `a`/`x` are the -primary button, a trackpad is the thumbstick — so a game that only ships Index or Vive bindings still gets -input, without the runtime claiming to be that hardware. The profile reported as current is the first of Touch, -Index, WMR, Vive, simple controller that the application suggested bindings for. +The server says which headset it drives and which controllers are in the user's hands (`mvr_system_info`'s +`headset` and `controllers`, described by `protocol/metalvr_headsets.h`), and reports each controller's inputs in +one layout (`mvr_controller_state`: buttons, touches, trigger, squeeze, stick, trackpad, forces, finger curls). Their +own interaction profile is preferred when the application suggested bindings for it (Touch, Touch Plus, Index, Vive, +Steam Frame); every other profile is read as a naming of the same physical inputs — `a`/`x` are the primary button, +a Vive profile's trackpad is a Touch controller's thumbstick and a Touch profile's thumbstick a Vive wand's trackpad +— so a game that ships bindings only for other controllers still gets input, without the runtime claiming to be +that hardware. With the controllers unknown, the order is Touch, Index, WMR, Vive, simple controller, Touch Plus, +Steam Frame. + +## What works + +- OpenXR 1.0 and 1.1 core: sessions, spaces, swapchains, the frame loop, projection and quad layers, actions, + haptics; `XR_EXT_hand_tracking`, `XR_FB_display_refresh_rate`, and the Touch Plus and Steam Frame profiles. +- `XR_KHR_D3D11_enable` under Wine with Mullion, and `XR_KHR_metal_enable` on macOS. +- Hardware H.264 and HEVC encoding at a headset's resolution and refresh rate, with ALVR's foveated encoding. +- OpenVR applications through MetalVR's SteamVR driver, or through OpenComposite: rendering, legacy and + action-manifest input, haptics. +- The headsets in [headsets/README.md](headsets/README.md), each with what is checked and what is not. ## What is not there yet - Graphics bindings other than Direct3D 11 (Windows) and Metal (macOS): Direct3D 12, Vulkan, OpenGL. - Layer types beyond projection and quad (cylinder, equirect, cube); depth submitted with a projection layer is accepted and unused. -- Hand tracking, eye tracking, passthrough, foveation. +- Eye tracking, passthrough for applications (the Lighthouse headsets' home shows the room with `--passthrough`), + gaze-following foveation. Hand tracking is `XR_EXT_hand_tracking`, from headsets that track hands. - 32-bit Windows applications. -- The headset's play-area bounds (`xrGetReferenceSpaceBoundsRect` reports them unavailable) and recentering. +- Play-area bounds from the headsets MetalVR drives itself (an ALVR headset's are reported), and recentering. - Game audio and microphone. - Starting the server on demand; today it must be running before the application creates its instance. diff --git a/docs/BUILDING.md b/docs/BUILDING.md index 546bf74..8d64556 100644 --- a/docs/BUILDING.md +++ b/docs/BUILDING.md @@ -5,7 +5,7 @@ - An Apple silicon Mac. MetalVR is developed and tested on macOS 27. - Xcode command line tools, CMake 3.24 or later, Ninja. - MinGW-w64 for the Windows runtime: `brew install mingw-w64`. -- A Rust toolchain, only for the ALVR driver. +- A Rust toolchain, only for the ALVR driver (and the Android client, `scripts/build-alvr-client.sh`). - A Wine with [Mullion](https://github.com/dttdrv/mullion), only to run Windows applications and the Wine tests. [MetalSharp-Wine](https://github.com/metalsharp/MetalSharp-Wine) is what the tests default to. @@ -27,6 +27,9 @@ Two optional pieces are built by their own scripts, which fetch what they need i |---|---| | `scripts/build-alvr.sh` | ALVR's `server_core`, for streaming to a headset. Configure `build/native` again afterwards and `metalvrd` links it. See [ALVR.md](ALVR.md). | | `scripts/build-opencomposite.sh` | OpenComposite's `openvr_api.dll`, for SteamVR games. See [OPENCOMPOSITE.md](OPENCOMPOSITE.md). | +| `scripts/build-libsurvive.sh` | libsurvive and hidapi, for the Lighthouse headsets' positions from their base stations; configure `build/native` again afterwards. See [HEADSETS.md](HEADSETS.md). | +| `scripts/build-alvr-client.sh` | ALVR's headset client with MetalVR's patches, as an APK for a standalone headset (the Steam Frame); fetches the Android NDK, SDK, a JDK 17 and cargo-apk into `build/android`, and shows Google's SDK license for you to accept. See [ALVR.md](ALVR.md). | +| `scripts/build-basalt.sh` | Basalt (Monado's fork) with oneTBB, OpenCV and fmt, into `build/basalt`, for the Rift S's position from its cameras. `metalvrd` loads `libbasalt.dylib` when a Rift S starts; nothing is linked, so it needs no reconfigure. See [HEADSETS.md](HEADSETS.md). | Without ALVR, `metalvrd` has only the simulated headset. @@ -38,7 +41,9 @@ build/native/daemon/metalvrd --trust 1234.client The name is the one the headset's ALVR client shows. Only headsets named with `--trust` may connect; one that is not is logged with its name and refused. `metalvrd --driver sim` runs with the simulated headset instead, and -`metalvrd --help` lists every option. +`metalvrd --help` lists every option. A headset on the Mac's own display and USB runs with +`metalvrd --driver rift` (Rift CV1, Rift S), `--driver lighthouse` (Index, Vives) or `--driver psvr2`; each one's +guide is in [headsets/](headsets/README.md). The server must be running before an application creates its OpenXR instance. @@ -76,8 +81,21 @@ scripts/test-native.sh # Metal application → metalvrd scripts/test-wine.sh # Direct3D 11 application under Wine + Mullion → metalvrd scripts/test-openvr.sh # OpenVR application → OpenComposite → MetalVR, under Wine + Mullion scripts/test-failures.sh # no server, server killed mid-session, second application +scripts/test-alvr-models.sh # which headset the ALVR driver names from what its client reported; wired mode's adb ``` +The headsets MetalVR drives itself are checked without one: `build/native/tests/headset_protocol_check` (the Rift +S's, the Lighthouse headsets', the PlayStation VR2's and the Sense controllers' reports, configuration and +commands) and `metalvrd --panel-check MODEL OUT.png` +(each model's panel drawn through its lens correction, with each eye's straight-ahead mark where it belongs). +`metalvrd --survive-check sim` runs libsurvive on its own simulated room and checks the poses that come out; +`build/native/tests/rift_s_tracking_check` moves a synthetic Rift S through a textured room and checks the path +Basalt tracks from its rendered camera frames and IMU against the true one; and +`metalvrd --passthrough-check` draws an Index's home with a test camera picture. The simulated headset tracks hands, +which `xr_smoke` locates through `XR_EXT_hand_tracking` in every test that runs it. +`scripts/test-all.sh` runs these, and `--headset index`, `--headset vive` or `--headset psvr2` with the simulated +headset for the interaction profiles of those controllers. Each headset's guide lists which of these cover it. + All of them use the simulated headset. CI runs the native ones on GitHub's macOS runners, which have no hardware video encoder, with `METALVR_DAEMON_ARGS="--software-encoder --eye 640x704"`; that option is for such machines only and is far too slow for a headset. A case passes only if @@ -100,7 +118,7 @@ protocol/ the wire protocol and pose math shared by both halves runtime/ the OpenXR runtime: Windows DLL and macOS dylib from one tree daemon/ metalvrd: server, compositor, encoder, headset drivers tests/ test applications and the stream and frame checkers -scripts/ dependency fetch, ALVR and OpenComposite builds, prefix install, tests -patches/ the changes MetalVR's builds make to ALVR and OpenComposite -docs/ this documentation +scripts/ dependency fetch, the optional builds (ALVR, OpenComposite, libsurvive, Basalt), prefix install, tests +patches/ the changes MetalVR's builds make to ALVR, OpenComposite and libsurvive +docs/ this documentation; docs/headsets/ the headset guides ``` diff --git a/docs/HEADSETS.md b/docs/HEADSETS.md new file mode 100644 index 0000000..281c9aa --- /dev/null +++ b/docs/HEADSETS.md @@ -0,0 +1,544 @@ +# Headset support plan + +The research and plan behind MetalVR's headset support: each headset's specifications, how MetalVR reaches and +recognizes it, and what could be built and checked before one is in hand. To set a headset up, see its guide in +[headsets/](headsets/README.md). The plan started from one real headset, the Oculus Rift CV1 ([RIFT.md](RIFT.md)), +and added ten, one phase each: + +| Phase | Headset | How it reaches the Mac | Driver | +|---|---|---|---| +| A | Meta Quest 1 | ALVR over Wi-Fi, or USB-C through adb | `alvr` | +| B | Meta Quest 2 | ALVR over Wi-Fi, or USB-C through adb | `alvr` | +| C | Meta Quest 3 | ALVR over Wi-Fi, or USB-C through adb | `alvr` | +| D | Meta Quest 3S | ALVR over Wi-Fi, or USB-C through adb | `alvr` | +| E | Valve Steam Frame | ALVR's client, patched by MetalVR to run there, over Wi-Fi | `alvr` | +| F | Valve Index | DisplayPort and USB on the Mac, Lighthouse base stations | new `lighthouse` | +| G | HTC Vive | HDMI (or DisplayPort) and USB on the Mac, Lighthouse 1.0 | `lighthouse` | +| H | HTC Vive Pro / Vive Pro 2 | DisplayPort and USB on the Mac, Lighthouse | `lighthouse` | +| I | Oculus Rift S | DisplayPort and USB on the Mac, inside-out cameras | `rift` (extended) | +| J | Sony PlayStation VR2 | Sony's PC adapter: DisplayPort and USB on the Mac; the headset tracks itself | new `psvr2` | + +None of these headsets has been tried with MetalVR. Everything below comes from published specifications, the open +drivers that support them (ALVR, OpenHMD, Monado, libsurvive, the community Vive Pro 2 driver) and MetalVR's own code. +Facts are marked **(guess)** where no source confirms them and **(checked here)** where they were confirmed on a +development Mac. Each phase ends with what can be built and tested without the headset, and what cannot be known until +one is plugged in. + +## What is built + +Every phase's work that can be done without the headset is in the code and checked without one. Each headset's +guide in [headsets/](headsets/README.md) says how to run it, what is checked and by which test, and what is unknown +until it meets real hardware. Only the Rift CV1 has. + +The shared work S1 to S10 below is built. S7 took a different shape than planned: rather than a presenter moved +into its own file, the CV1's driver became the one direct-display driver, and each other headset's USB side is a +backend behind a narrow interface (`daemon/direct_backend.h`): the window, pacing, home, lens correction and head +rotation are written once. + +## What had to change first + +Read across the first nine phases, MetalVR's code assumed a Rift in several places (all changed since, see above): + +- **The headset had no identity.** `mvr_system_info` carried a name string and nothing else, and the ALVR driver + named every headset "MetalVR ALVR Headset", although ALVR records the exact model the client reported ("Quest 2", + "Quest 3S") in `session.json` under `client_connections..display_name`. +- **SteamVR was always shown Rift CV1 controllers**, with the CV1's grip offset and `willDriftInYaw`; the HMD's + manufacturer was "Oculus" only when the name contained "Rift". +- **One refresh rate.** `xrEnumerateDisplayRefreshRatesFB` returned the single negotiated rate. +- **Controllers were Touch-shaped.** `mvr_controller_state` had no trackpad, finger curl, grip force, bumper, D-pad + or View button, and the runtime reported whichever profile the game suggested first, so a Vive wand could be handed + Touch bindings with A/B buttons it does not have. +- **The direct-display code was CV1-only**: the CV1's display IDs (`3ed2:0004`), a fixed 90 Hz, the panel split + into left and right halves. +- **ALVR's battery and play-area events were dropped.** (Hand tracking is still not there.) + +Each was fixed once, in the first phase that needed it, and reused by the phases after it: + +| # | Shared work | First needed in | Reused by | +|---|---|---|---| +| S1 | A headset model and a controller kind in `mvr_system_info` (protocol version bump). SteamVR, OpenXR's `systemName` and the interaction profile are chosen from them; the `strstr(name, "Rift")` test goes. | A | all | +| S2 | ALVR model detection: read `display_name` from `session.json` on connect, look it up in a per-model table (name, defaults before `VIEWS_CONFIG`, refresh list, codec limits, controller kind). An unknown name gives generic defaults, never a Rift. | A | B–E | +| S3 | ALVR battery (`ALVR_EVENT_BATTERY`) and play area (`PLAYSPACE_SYNC`) through to OpenXR stage bounds and SteamVR battery properties. | A | B–E | +| S4 | Wired ALVR: document `--trust client.wired`, and require an `adb` the user installed. ALVR otherwise downloads Google's platform-tools at runtime (`alvr/adb/src/commands.rs`), which MetalVR must refuse. | A | B–D | +| S5 | The headset's refresh-rate list in `mvr_system_info` and `XR_FB_display_refresh_rate`. Changing the rate on an ALVR headset means renegotiating the stream, so the list may start read-only. | B | C–I | +| S6 | Touch Plus: `/interaction_profiles/meta/touch_plus_controller` and `XR_META_touch_controller_plus`. | C | D | +| S7 | One direct-display presenter, moved out of `driver_rift.mm` now that a second driver needs it: the window on the headset's display, pacing, the home and test scenes, `Fusion`. The display's IDs, refresh rate, panel rotation and eye split become per-model parameters. | F | G, H, I | +| S8 | `display` for every direct-display headset: the macOS override that shows it, keeping any EDID patches Apple's own override carries (the Vive Pro 2's). | F | G, H, I | +| S9 | Lighthouse core: read the headset's zlib-compressed JSON config, its lens distortion (Monado's Vive model), the IMU, the Watchman radio for controllers, and haptics. Base-station tracking through libsurvive after that. | F | G, H | +| S10 | Controller protocol: trackpad axes and touch, finger curl, grip and trackpad force, bumper, D-pad and View. | E or F | E–H | + +## Recognizing the headset + +MetalVR never assumes a headset. Each driver identifies one by evidence that cannot belong to another, and a driver +that finds nothing it recognizes reports no headset. It does not fall back to a Rift or to another model. + +| Headset | Identified by | macOS display override | +|---|---|---| +| Rift CV1 (today) | USB `2833:0031` | `3ed2:0004` | +| Quest 1 | ALVR `display_name` "Quest 1"; over adb, `ro.product.device` = `monterey` | streamed, none | +| Quest 2 | "Quest 2"; `hollywood` | streamed, none | +| Quest 3 | "Quest 3" (exact match; "Quest 3S" shares the prefix); `eureka` | streamed, none | +| Quest 3S | "Quest 3S"; `panther` | streamed, none | +| Steam Frame | "Steam Frame" from MetalVR's patched client (`patches/alvr/0003`); the stock client says "Android (unknown)" | streamed, none | +| Valve Index | USB `28de:2300` with no HTC mainboard present, confirmed by the model in its config JSON | `5996:91a8` **(checked here)** | +| HTC Vive | USB `0bb4:2c87` (HTC mainboard) with `28de:2000` | `22d2:aa01` **(checked here)** | +| Vive Pro | USB `0bb4:0309` (Monado, OpenHMD) or `0bb4:030e` (libsurvive) with `28de:2300` | `22d2:aa02` (guess) | +| Vive Pro 2 | USB `0bb4:0342` with `28de:2300` | `22d2:aa04` (inferred from its timings) | +| Rift S | USB `2833:0051`, confirmed by feature report 6 (1440×2560, 80 Hz) | `3ed2:0012` **(checked here)** | +| PlayStation VR2 | USB `054c:0cde` (Monado); Sense controllers `054c:0e45` (left) and `0e46` (right) | Sony's `4dd9`, product not known | + +Quest USB product IDs change with the USB mode (MTP, adb, Link) and are shared across models, so they are not used. +`28de:2300` is also used by the Vive Pro line and a Vive Tracker, so the Index is only recognized once its config +confirms it. + +macOS marks every direct-display headset in this plan as a head-mounted display and keeps it off the desktop, as it +does the CV1: `/System/Library/Displays/Contents/Resources/Overrides` on a development Mac holds `is-hmd` overrides for +`3ed2` products 1, 2, 3, 4 and 12, `22d2` products aa01 to aa04, and `5996:91a8`. It holds none for Sony's displays +(**checked here**): the PlayStation VR2's display should appear as an ordinary one, with no override. + +--- + +## Phase A: Meta Quest 1 + +| | | +|---|---| +| Model | Oculus Quest (1st generation), 2019; codename `monterey`; 64 GB and 128 GB. Discontinued 2020; Meta stopped updates in 2024. | +| Resolution | Two PenTile OLED panels, 1440×1600 per eye (2880×1600) | +| Refresh | 72 Hz only | +| Lenses | Fresnel-type, from the Oculus Go **(guess: not stated as Fresnel by Meta)**; IPD slider 58–72 mm | +| Field of view | about 93° × 93° visible; the runtime renders about 104° × 100° | +| Tracking | Inside-out, 4 cameras (Oculus Insight); controllers tracked by the cameras through IR LEDs in their rings; hand tracking since 2020; no eye or face tracking | +| Connectivity | Wireless (Wi-Fi 5) or USB-C. Standalone: Snapdragon 835, 4 GB | +| Audio | Speakers in the strap, two 3.5 mm jacks, microphones | +| Controllers | Touch (2nd generation, ring on top): A/B or X/Y (click and touch), Menu or Oculus, trigger (analog, touch), grip (analog), thumbstick (2 axes, click, touch), thumbrest touch; one AA each | +| Haptics | One motor per controller; none in the headset | +| Expandability | Straps and batteries only | +| Video decoding | H.264 and HEVC up to 2160p60; no AV1 | + +**Path in MetalVR.** ALVR over Wi-Fi works as it is, with the ALVR client sideloaded (it is not on Meta's store for +Quest 1). Wired streaming over USB-C uses ALVR's adb mode, which forwards its ports through `adb` on the Mac. Meta's own +Link and Air Link use a closed protocol and Windows-only software; they cannot be used from a Mac. + +**Work.** +1. S1, S2: a Quest 1 entry in the model table. 1216×1344 per eye **(guess)** as the default before the client's + values arrive, 72 Hz, HEVC only, and a frame size the Snapdragon 835 can decode: about 6.9 Mpx at 72 Hz, for example + 2 × 1664 × 1856 **(derived)**. ALVR's default 4288-wide frame is too wide for it. +2. S1: SteamVR shows Quest Touch controllers (render model names from ALVR's `server_openvr/src/props.rs`, + manufacturer "Oculus"), with no CV1 grip offset and `willDriftInYaw` false. +3. S3: battery and play area. +4. S4: wired mode with a user-installed `adb`; `docs/ALVR.md` covers sideloading and wired setup. + +**Testable now:** model detection from fixture `session.json` files, the table, the seed it produces, SteamVR +properties per model, battery and play-area handling from synthetic ALVR events, the encoder at Quest 1 sizes on the +simulated headset. +**Needs the headset:** the resolution and refresh list the client really reports (the client falls back to 90 Hz when +the runtime lacks `XR_FB_display_refresh_rate`, which a Quest 1 panel cannot do), the decoder's real ceiling, the +controller grip and aim offsets, whether ALVR's current client still runs on Quest 1 firmware, wired mode on macOS. + +**Risks:** ALVR keeps a separate OpenXR loader for Quest 1 and may drop it. Every frame size above the decoder's +limit fails on the headset, not on the Mac. + +## Phase B: Meta Quest 2 + +| | | +|---|---| +| Model | Meta (Oculus) Quest 2, 2020; codename `hollywood`; 64/128/256 GB. Discontinued 2024. | +| Resolution | One fast-switch LCD, 1832×1920 per eye (3664×1920) | +| Refresh | 72, 90, 120 Hz (120 experimental); 60 and 80 Hz unconfirmed | +| Lenses | Fresnel; IPD in three positions, 58/63/68 mm | +| Field of view | about 97° × 93° visible (community measurements; no official figure) | +| Tracking | Inside-out, 4 IR cameras; controllers through IR LEDs in their rings; hand tracking; no eye or face tracking | +| Connectivity | Wireless (Wi-Fi 6) or USB-C. Standalone: Snapdragon XR2 Gen 1, 6 GB | +| Audio | Speakers in the strap, 3.5 mm jack, 2 microphones | +| Controllers | Touch (3rd generation, ring): same inputs as Quest 1; one AA, about 30 h | +| Haptics | One LRA per controller **(guess)** | +| Expandability | Elite Strap with or without battery | +| Video decoding | HEVC and H.264; no AV1 | + +**Path in MetalVR.** As Quest 1: ALVR over Wi-Fi, or wired through adb. Link is not usable. + +**Work.** +1. A Quest 2 entry in the model table (S2), and S5: the refresh list (72/90/120). The ALVR driver's current defaults + (2144×2336 at 72 Hz) were meant as a Quest 2's but match neither its native nor its default render size; replace + them with the table's. +2. SteamVR: `oculus_quest2_controller_left/right` and ALVR's Quest 2 model number. +3. Seed 90 Hz as the preferred rate for a known Quest 2. + +**Testable now:** as phase A, plus the refresh list through `xrEnumerateDisplayRefreshRatesFB`. +**Needs the headset:** the real recommended size and refresh list, HEVC decoding at 120 Hz, controller offsets, +haptics, wired mode. + +**Risks:** the client's model string comes from ALVR's code and could change between ALVR versions; MetalVR matches +the version it pins and falls back to generic. + +## Phase C: Meta Quest 3 + +| | | +|---|---| +| Model | Meta Quest 3, 2023; codename `eureka`; 128 GB and 512 GB | +| Resolution | Two LCDs, 2064×2208 per eye (4128×2208); 1680×1760 default render size | +| Refresh | 72, 80, 90, 120 Hz; 96 and 100 Hz for newer OpenXR apps; up to 144 Hz and beyond in extended modes | +| Lenses | Two-element pancake; IPD wheel, 53–75 mm | +| Field of view | 110° × 96° | +| Tracking | Inside-out: 4 IR cameras, 2 color cameras, a depth projector; color passthrough; Touch Plus controllers without rings, tracked by LEDs in their faces; hand tracking; no eye or face tracking | +| Connectivity | Wireless (Wi-Fi 6E) or USB-C. Standalone: Snapdragon XR2 Gen 2, 8 GB | +| Audio | Speakers in the strap, 3.5 mm jack, microphones | +| Controllers | Touch Plus: A/B or X/Y (click, touch), Menu, Oculus, trigger (value, touch, and in OpenXR proximity, force, curl, slide), grip, thumbstick (axes, click, touch), thumbrest touch; one AA, about 30 h | +| Haptics | TruTouch, one voice-coil motor per controller; none in the headset | +| Expandability | Straps, battery straps, charging dock | +| Video decoding | AV1, HEVC, H.264 | + +**Path in MetalVR.** ALVR over Wi-Fi or adb. The Mac cannot use the Quest 3's AV1: the encoders on an Apple M6 are +H.264, HEVC, ProRes and JPEG **(checked here)**, and Apple silicon decodes AV1 but does not encode it. MetalVR already +turns AV1 requests into HEVC. + +**Work.** +1. A Quest 3 entry (S2), with an exact match on "Quest 3". +2. S6: the Touch Plus profile ahead of `oculus/touch_controller`. ALVR's own client binds every Quest's controllers as + plain Touch, so trigger proximity maps to trigger touch, and force, curl and slide are reported as absent. Say so in + `docs/ALVR.md`. +3. SteamVR: `oculus_quest_plus_controller_*`, "Meta Quest 3 (Left Controller)", manufacturer "Meta". +4. Later: hand tracking through `alvr_get_hand_skeleton` and `XR_EXT_hand_tracking`. + +**Testable now:** detection and table; Touch Plus bindings and profile reporting with synthetic input; SteamVR +properties. +**Needs the headset:** the reported resolution and refresh list, HEVC decoding at about 4288×2240, controller offsets, +haptics. + +**Risks:** the Touch Plus profile promises inputs (force, curl, slide) that ALVR does not send. + +## Phase D: Meta Quest 3S + +| | | +|---|---| +| Model | Meta Quest 3S, 2024; codename `panther`; 128 GB and 256 GB; an Xbox edition (2025) with the same hardware | +| Resolution | One LCD, 1832×1920 per eye (3664×1920) | +| Refresh | 72, 80, 90, 120 Hz (96 and 100 Hz possibly, sources disagree); no extended modes | +| Lenses | Fresnel; IPD in three positions, 58/63/68 mm | +| Field of view | about 96° × 90° | +| Tracking | Inside-out: 4 IR cameras, 2 color cameras, no depth sensor; Touch Plus controllers; hand tracking | +| Connectivity | Wireless (Wi-Fi 6E) or USB-C. Standalone: Snapdragon XR2 Gen 2, 8 GB | +| Audio | Speakers in the strap, microphones, **no headphone jack** (audio over USB-C) | +| Controllers | Touch Plus, as Quest 3 | +| Haptics | TruTouch in the controllers | +| Expandability | Straps, battery straps; an Action button on the headset | +| Video decoding | AV1, HEVC, H.264 | + +**Path in MetalVR.** As Quest 3. The 3S is told apart from the Quest 3 only by ALVR's "Quest 3S", or `panther` over adb. + +**Work.** A Quest 3S entry in the table: Quest 3 behavior with the Quest 2's panel size and lens steps, and a refresh +list that stops at 120 Hz. Everything else comes from phases A to C. + +**Testable now:** detection, especially that "Quest 3S" is not taken for "Quest 3". +**Needs the headset:** the reported resolution, refresh list (96 and 100 Hz) and IPD per lens step. + +## Phase E: Valve Steam Frame + +| | | +|---|---| +| Model | Valve Steam Frame; 256 GB and 1 TB. Announced November 2025; launched September 2026 through a reservation queue, with orders running into 2027 (sources as of November 2026). | +| Resolution | Two LCDs, 2160×2160 per eye | +| Refresh | 72, 80, 90, 96, 108, 120 Hz, and 144 Hz experimental | +| Lenses | Pancake; IPD dial, 60–70 mm per Valve (57.6–69.6 mm per one review) | +| Field of view | "up to 110°" (Valve gives no breakdown) | +| Tracking | Inside-out: 4 monochrome fisheye cameras with IR illuminators; controllers tracked by the cameras through IR LEDs; 2 eye-tracking cameras for foveated streaming; no hand tracking; no Lighthouse support found | +| Connectivity | Wireless: the headset has Wi-Fi 7, and the bundled USB adapter for the PC is a Wi-Fi 6E card (`28de:2432`, a Realtek RTL8852CU). Wired: one USB-C 2.0 port. Standalone: Snapdragon 8 Gen 3, 16 GB, SteamOS on arm64; Android apps run in a container ("Lepton") | +| Audio | Two drivers per ear in the strap; microphone array | +| Controllers | Ringless, IR LEDs, IMU, TMR thumbsticks. A/B/X/Y all on the right controller, a D-pad and View on the left; each has a bumper, trigger, squeeze, Menu/System; capacitive sensing on every input and the handle; one AA, about 40 h | +| Haptics | In the controllers | +| Expandability | A 44-pin front port for cameras or PCIe, with published CAD files; developer mode gives ssh, adb and remote desktop | + +**Path in MetalVR.** This is the one phase that depends on a client running on the headset. +- **Valve's own streaming is closed.** Steam Link VR is undocumented and changes with Steam updates, and needs Windows + SteamVR with a PC GPU encoder. The adapter is a plain Wi-Fi card with no macOS driver, and Wine cannot pass a network + card through. Valve does not support Mac hosts. This route is closed. +- **ALVR is the only route.** The stock ALVR 20.14.1 client shows a blank screen in Lepton (Valve's runtime lacks + `XR_KHR_convert_timespec_time`); MetalVR patches the client to do without it (work item 4 below). Lepton has only + software video decoders; a native Linux arm64 client, like WiVRn's, would reach the hardware decoder. Any client has + to speak the protocol of the ALVR version MetalVR pins. +- **Wired:** USB-C 2.0 has the bandwidth for 80–150 Mb/s HEVC, but the class of USB network it offers, and so whether + macOS can use it, is undocumented. +- **No direct display:** the Frame is a computer, not a display. + +**Work.** +1. A Steam Frame entry in the table, chosen explicitly (`--headset steam-frame`) until a client identifies itself: + HEVC 8-bit only, 2160×2160 per eye, 90 Hz default, 72–120 Hz allowed. +2. S10: bumper, D-pad and View bits; the Frame's own profile + (`/interaction_profiles/valve/frame_controller_valve`, behind Valve's `XR_VALVE_frame_controller_interaction`), + with Touch as the fallback. SteamVR's controller names for the Frame are unknown until they can be read from a + SteamVR install. +3. Foveated streaming, in two steps: ALVR's static foveated encoding in `daemon/compositor.mm`, matched to the client's + unwarp shader; then gaze-driven foveation, which needs a modified client to send gaze and undo the warp. +4. A client: ALVR's own, patched (`patches/alvr/0003-client-runs-on-the-steam-frame.patch`) to ask for + `XR_KHR_convert_timespec_time` only when the runtime has it, and to name itself "Steam Frame", built as an APK + by `scripts/build-alvr-client.sh` and installed with adb. It still decodes in + software in Lepton; a native client that reaches the hardware decoder remains for later. + +**Testable now:** the explicit model choice and table, the controller bits and profile, static foveation against +ALVR's shader math on the simulated headset. +**Needs the headset (and a working client):** everything about the stream: decoding speed, latency, the real field of +view, the USB network class, the headset's USB IDs, SteamVR's controller names. + +**Risks:** the patched client has not run on a Frame, and decodes in software there; the vendor controller extension +has already been renamed once. + +## Phase F: Valve Index + +| | | +|---|---| +| Model | Valve Index, 2019; sold as headset, controllers, base stations, or kits. Succeeded by the Steam Frame. | +| Resolution | Two LCDs (full RGB), 1440×1600 per eye (2880×1600) | +| Refresh | 80, 90, 120 Hz, and 144 Hz experimental | +| Lenses | Dual-element, canted; IPD slider 58–70 mm; eye-relief knob (changes the field of view) | +| Field of view | about 108° (unofficial; Valve says "about 20° more than Vive") | +| Tracking | Lighthouse (SteamVR Tracking 2.0 sensors): works with 1.0 base stations (up to 2) and 2.0 (up to 4); IMU. Stereo 960×960 global-shutter color cameras for passthrough, not tracking | +| Connectivity | Tethered: DisplayPort 1.2, USB 3.0 and 12 V in one 5 m cable with a breakaway. On a Mac: a USB-C/Thunderbolt to DisplayPort cable or adapter; Valve does not support HDMI adapters | +| Audio | Off-ear BMR speakers, 3.5 mm jack, dual microphones | +| Controllers | Index controllers: A, B, System, analog trigger, thumbstick, track button with force sensor, grip force sensor, per-finger capacitive sensing (finger tracking), IMU; USB-C charging, 1100 mAh, 7+ h. Vive wands and Vive Trackers also work | +| Haptics | HD LRA haptics in each controller | +| Expandability | Front USB port ("frunk") for peripherals; face gaskets | + +**Path in MetalVR.** A direct-display headset like the CV1: frames drawn on its display through the Mac's +DisplayPort, its USB devices opened through IOHIDManager. The Index is the first Lighthouse headset, so this phase +builds what G and H reuse. + +**Work.** +1. S7: the shared direct-display presenter out of `daemon/driver_rift.mm`. +2. S8: `display` for the Index: `/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-5996/DisplayProductID-91a8` + with `DisplayVendorID` 22934, `DisplayProductID` 37288, `is-hmd` false. The command and its guide move from + `metalvr-rift` to a script that covers every direct-display headset. +3. S9: `daemon/driver_lighthouse.mm` with a per-model table, `--driver lighthouse` in `daemon/main.mm`: + - recognize the Index as above, and report no headset when unsure; + - read the config (feature report 0x10, then 0x11 until a 0-length chunk; zlib; JSON); + - the Index lens distortion (per eye and color channel, ported from Monado's `u_compute_distortion_vive`, BSL-1.0) + and its canted eye transforms; + - the IMU (report 0x20) into `Fusion`, for rotation first; + - refresh choice, 80 to 144 Hz, through `CGDisplaySetDisplayMode`. +4. S10 and the Watchman radio (`28de:2102`): Index controller input, finger curl and force into the protocol; + `/interaction_profiles/valve/index_controller` with no trackpad folding; SteamVR's `{indexcontroller}` profile and + skeletal input; haptics. +5. Position: libsurvive (MIT; Lighthouse 1.0 and 2.0) fed through IOHIDManager, since macOS's own HID driver holds the + interfaces libusb would need; base stations as `trackingReferences`. +6. Later: passthrough from the cameras through AVFoundation. + +**Testable now:** config reassembly and parsing on fixtures, distortion against Monado's output, IMU decoding, +identification over mock device lists (Index vs Vive Pro vs Tracker vs Rift), the controller protocol and bindings, +the override plist. +**Needs the headset:** whether macOS brings its display up through a USB-C to DisplayPort adapter at all (the Rift +notes already record macOS reading a headset's EDID and not showing it through a hub), which modes it offers (120, +144 Hz; 144 Hz only just fits DisplayPort 1.2), whether the display needs a power-on command, HID access, tracking +quality, controller pairing, haptics. + +**Risks:** the display path through an adapter is the largest unknown in the plan. libsurvive had never been built for +macOS before MetalVR (`scripts/build-libsurvive.sh`), and has run only in its own simulated room. The Index controllers' radio protocol is partly reverse-engineered. The headset is discontinued. + +## Phase G: HTC Vive + +| | | +|---|---| +| Model | HTC Vive, 2016 (a lighter revision in 2017; the Vive Pre was the developer edition) | +| Resolution | Two AMOLED panels (PenTile), 1080×1200 per eye (2160×1200) | +| Refresh | 90 Hz | +| Lenses | Fresnel; IPD knob, about 60.8–74.6 mm measured | +| Field of view | about 110° | +| Tracking | Lighthouse 1.0 only (32 headset sensors, 24 per wand; 2.0 base stations do not work with it); IMU; proximity sensor; one front camera | +| Connectivity | Tethered through a link box: HDMI 1.4 or mini-DisplayPort, USB, power. On a Mac: its own HDMI port | +| Audio | 3.5 mm jack and microphone; the Deluxe Audio Strap adds headphones | +| Controllers | Vive wands: trackpad (axes, touch, click), two-stage trigger, grip (one button), Menu, System; micro-USB, about 6 h | +| Haptics | One actuator per wand, driven by a USB report to the receiver in the headset, which forwards it by radio | +| Expandability | Deluxe Audio Strap, Wireless Adapter, TPCast, Vive Trackers. Index controllers need 2.0 base stations, so not with a Vive's 1.0 stations | + +**Path in MetalVR.** The `lighthouse` driver from phase F, with a Vive model entry. + +**Work.** +1. A Vive entry: `0bb4:2c87` (mainboard) and `28de:2000` (IMU and receiver), 1080×1200 at 90 Hz, Lighthouse 1.0, + wands. Power-on is a feature report to the mainboard (`04 78 29 38 01 …`); whether the display needs it to light is + unknown. +2. `display` for the Vive: `DisplayVendorID-22d2/DisplayProductID-aa01` (8914 / 43521). +3. Wands: trackpad, grip, Menu, System into the protocol (S10); the runtime offers `htc/vive_controller` when wands are + attached, instead of the game's first suggested profile; SteamVR's `vive_controller` profile, manufacturer "HTC". + Haptics as a pulse report through the receiver (`ff 8f 07 00` and pulse timings, tuned on hardware). + +**Testable now:** the model entry and identification, the wand mapping and profile choice, haptic report encoding, +the override plist. +**Needs the headset:** the HDMI mode on a Mac, power-on, IMU axes, the real config, wand pairing (assumed already +stored in the receiver), Lighthouse 1.0 tracking quality. + +**Risks:** Lighthouse 1.0 base stations are scarce secondhand; without base stations the Vive has only rotation. + +## Phase H: HTC Vive Pro and Vive Pro 2 + +| | Vive Pro (2018; Vive Pro Eye 2019) | Vive Pro 2 (2021) | +|---|---|---| +| Resolution | Two AMOLED panels, 1440×1600 per eye | Two LCDs, 2448×2448 per eye (4896×2448) | +| Refresh | 90 Hz | 90 and 120 Hz | +| Lenses | IPD knob with sensor; adjustable eye relief | IPD 57–70 mm with sensor; adjustable eye relief | +| Field of view | 110° | up to 120° horizontal | +| Tracking | Lighthouse 1.0 and 2.0; IMU, proximity, IPD sensors; stereo front cameras; the Pro Eye adds 120 Hz eye tracking | Lighthouse 1.0 and 2.0; IMU, proximity, IPD sensors | +| Connectivity | Link box: DisplayPort 1.2, USB 3.0, power | Link Box 2.0: DisplayPort with Display Stream Compression for full resolution, USB 3.0, power | +| Audio | Removable on-ear headphones, microphones | Removable on-ear headphones, dual microphones | +| Controllers | Vive wands (2018 kit); Index controllers also work | Wands or Index controllers | +| Haptics | In the controllers | In the controllers | +| Expandability | Wireless Adapter, Vive Trackers, Facial Tracker, USB-C port | Wireless Adapter (up to 3264×1632 at 90 Hz), Vive Trackers, Facial Tracker, USB-C port | + +The Vive Pro 2 has six display modes: 2448×1224 at 90 and 120 Hz, 3264×1632 at 90 Hz, 3680×1836 at 90 Hz, and +4896×2448 at 90 and 120 Hz. Only the last two need Display Stream Compression (DSC). + +**Path in MetalVR.** The `lighthouse` driver, from DisplayPort over USB-C or Thunderbolt (not the Mac's HDMI port). + +**Work.** +1. Model entries: Vive Pro by `0bb4:0309` or `0bb4:030e`, Vive Pro 2 by `0bb4:0342`; both with `28de:2300`, the + presence of the HTC mainboard telling them from an Index. +2. The Vive Pro uses the Lighthouse distortion model from its config. The Vive Pro 2 uses HTC's own lens correction + (Monado's code for it), read from the HTC mainboard's JSON. +3. The Vive Pro 2's mode select: a feature report (`wireless,0`, `dtd,N`, `chipreset`), after which the headset + re-enumerates and the driver waits for it. Default to 3264×1632 at 90 Hz; 2448×1224 as the safe floor. Treat the + 4896-wide modes as unavailable until a Mac is seen to drive them: macOS's use of DSC with a third-party display is + undocumented, and a cap near 600 MHz may rule out 3680×1836 (reported, not confirmed). +4. IPD and proximity from the HTC status report. +5. `display`: macOS's own override for `22d2:aa04` does not only hide the display, it adds the Pro 2's non-DSC display + timings to its EDID. MetalVR's override must carry those patches along, or macOS may have no usable mode. Whether a + `/Library` override replaces Apple's or merges with it is unknown, so copy them either way. +6. Eye tracking (Pro Eye) stays out of scope: its tracker uses Windows-only SDKs. + +**Testable now:** identification, config parsing, status and mode-select reports, the override plist with its EDID +patches checked against Apple's file. +**Needs the headset:** whether macOS shows its display and which modes it offers, the pixel-clock limit, DSC, HID access, +`chipreset` re-enumeration, distortion, tracking. + +**Risks:** the Vive Pro 2 at full resolution may never run on a Mac. The `aa02` and `aa03` assignments are guesses. +Without base stations there is only rotation. + +## Phase I: Oculus Rift S + +| | | +|---|---| +| Model | Oculus Rift S, made with Lenovo, 2019; discontinued 2021 | +| Resolution | One LCD, 1280×1440 per eye (2560×1440). The panel is portrait, 1440×2560: the left eye is the top half, the right eye the bottom, each rotated | +| Refresh | 80 Hz | +| Lenses | From the Oculus Go; no hardware IPD (software only, lenses fixed near 63.5 mm) | +| Field of view | 115° claimed (uncited); open drivers assume 105° horizontal | +| Tracking | Inside-out (Oculus Insight): 5 cameras in one USB video device, and an IMU; controllers tracked by the cameras | +| Connectivity | Tethered: DisplayPort 1.2 and USB 3.0 in one 5 m cable, powered from USB; no HDMI. On a Mac: USB-C/Thunderbolt to DisplayPort or mini-DisplayPort | +| Audio | Speakers in the strap, 3.5 mm jack, microphone | +| Controllers | Touch (2nd generation, as Quest 1): A/B/X/Y, Menu or Oculus, trigger, grip, thumbstick, touch on face buttons, stick and trigger; one AA | +| Haptics | In the controllers; no open driver implements them | +| Expandability | None | + +**Rift S against the CV1, as MetalVR sees them.** USB `2833:0051` (CV1 `0031`); display `3ed2:0012` (CV1 `0004`); +DisplayPort instead of HDMI; 80 Hz instead of 90; a rotated portrait panel; its own power-on (reports 0x07, 0x14, 0x0A, +0x02, 0x05) and a keep-alive every second instead of every ten; the screen switched by its proximity sensor; its own +IMU report (0x65, calibrated from a JSON block in the headset); controllers on its own interface (report 0x67); no +external sensors. + +**Path in MetalVR.** The existing `rift` driver, choosing the model by USB product ID. OpenHMD's Rift S driver is +already in MetalVR's fetched sources (`external/OpenHMD-rift/src/drv_oculus_rift_s`, BSL-1.0); Monado's (BSL-1.0) is +the other reference. + +**Work.** +1. A model table in `daemon/driver_rift.mm` on top of S7: IDs, display, refresh, field of view, eye size, panel + rotation and eye split. +2. The Rift S power-on, keep-alive, proximity-driven screen, IMU and calibration, as a parser file of its own. +3. The distortion shader for the rotated, top-and-bottom split. No correct Rift S distortion is published: both open + drivers reuse the CV1's and mark it wrong. Start from theirs, with a calibration hook for tuning by eye. +4. Controllers: input from report 0x67 into the existing Touch mapping; rotation only (no open driver tracks their + position); SteamVR's Rift S render models; no haptics until the command is known. +5. `metalvr-rift`: an override for `3ed2:0012` (16082 / 18); detect and report the model by USB product ID (49 or 81) + rather than by the name "Rift", which a Rift S does not match. +6. Position tracking from the cameras, as Monado does it: the five cameras arrive as one UVC device ("Rift S + Sensor"), read with AVFoundation (`daemon/rift_s_cameras.mm`); their calibration (firmware block 0x0f, Fisheye62 + lenses fitted to KB4) and the IMU go to Basalt, Monado's fork, loaded at run time through its VIT interface + (`daemon/camera_tracking.cpp`, `daemon/rift_s_tracking.cpp`); an auto exposure keeps the pictures dark enough to + track. `scripts/build-basalt.sh` builds Basalt with oneTBB, OpenCV and fmt into `build/basalt`. Checked on a + synthetic Rift S (`tests/rift_s_tracking_check`): rendered camera frames in the device's own layout, IMU reports + across the IMU clock's wrap, and the tracked path against the true one. + +**Testable now:** the parsers on constructed bytes, the model table, IMU scaling, the rotated eye split rendered +offscreen and checked, the script's override and detection with mocked `ioreg` output. +**Needs the headset:** whether macOS brings the DisplayPort link up and at 80 Hz without rotating it, the order of +power-on and proximity, USB names and interfaces on macOS, distortion, axis signs. + +**Risks:** the distortion has to be tuned by eye. Rotation-only controllers will disappoint in games built for tracked +hands. DisplayPort through USB-C is unreliable even on Windows for this headset. + +--- + +## Phase J: Sony PlayStation VR2 + +| | | +|---|---| +| Model | PlayStation VR2 (CFI-ZVR1), 2023; on PCs through Sony's PlayStation VR2 PC adapter (2024) | +| Resolution | 2000×2040 per eye (one 4000×2040 image across both panels, Monado), OLED | +| Refresh | 90 or 120 Hz | +| Lenses | Fresnel, with an IPD dial (59 to 72 mm, reported in the headset's status) | +| Field of view | about 110° (Sony); Monado's per eye: 61.5° toward the temple, 43.5° toward the nose, 53° up and down | +| Tracking | Inside-out with four cameras, solved on the headset itself, which sends its pose over USB; eye tracking | +| Connectivity | One USB-C cable carrying DisplayPort, USB and 12 V power. A PC needs Sony's adapter (DisplayPort 1.4 in, USB 3 in, its own power), since few ports give DisplayPort and 12 V together; on a Mac the same: the adapter's DisplayPort from a USB-C or Thunderbolt port through a USB-C to DisplayPort cable, its USB into the Mac | +| Audio | 3.5 mm jack on the headset | +| Controllers | Sense controllers: cross/circle (right) and square/triangle (left), Options and Create, PS button, stick, trigger with touch, grip, finger touch on the buttons; adaptive triggers; haptics; tracking LEDs seen by the headset's cameras; Bluetooth to the computer | +| Haptics | In the controllers; a rumble motor in the headset | +| Expandability | None | + +**Path in MetalVR.** The direct-display driver with a backend of its own (`daemon/backend_psvr2.mm`), from Monado's +`psvr2` and `pssense` drivers (BSL-1.0): +- The headset's USB side is vendor interfaces, not HID: status and IMU on an interrupt endpoint (interface 7, + alternate 1), its own tracking's pose on a bulk endpoint (interface 3), and the camera, LED-detector, relocalizer and + VD streams, which Monado reads and drops while the cameras run in mode 0x0a; MetalVR does the same. Controls (camera + mode, brightness, firmware information, the lens calibration) are vendor requests on the default pipe. MetalVR opens + it with IOUSBHost. +- The head's pose: the headset's own, its axes turned and corrected as Monado does (`psvr2::headFromSlam`), handed to + the driver as a tracker's pose, which takes the position from it and pulls the IMU's heading toward it, as it does + with the Rift's sensors and the base stations. +- The lens: Monado's model, a radial lookup table per colour with each eye's offset and tilt from the headset's + calibration (report 0x8f), as lens model 3 in the driver's shader. +- The Sense controllers over Bluetooth: their input report 0x31 (CRC-checked), their IMU calibration (feature report + 5, which also switches them into those reports), and vibration through output report 0x31. Games are given the Touch + profile, as Monado binds them; SteamVR shows them as Touch controllers named for the Sense. + +**Testable now:** every report and request, on constructed bytes (`headset_protocol_check`: status and IMU axes, the +tracking record and its quaternion continuity, requests, firmware, lens calibration and model, the controllers' reports, +CRC, calibration and vibration); the panel through its lens (`--panel-check psvr2`); the model as OpenXR and SteamVR +see it (`native-psvr2-sense`); the driver waiting for the headset and stopping cleanly. +**Needs the headset:** whether macOS lets IOUSBHost open its interfaces and take the vendor requests (through the +device, or else an interface); whether the adapter's display comes up at 4000×2040 and 90 or 120 Hz on a Mac (it may +need DisplayPort's compression); its EDID product code; the Sense controllers' pairing with a Mac and their IMU axes. + +**Risks:** the display path through the adapter is the largest unknown. The controllers are rotation only, and games +built for tracked hands will feel it, as with the Rift S. + +--- + +## Order and dependencies + +The phases run A to I, but not all of them can finish in that order: + +- **A to D** depend only on MetalVR and the ALVR version it pins. Each needs a headset only for its last check. +- **E** depends on ALVR's client running on the Frame. MetalVR patches the client for Valve's runtime; whether it + streams well with software decoding is known only on a headset. +- **F** builds the shared direct-display and Lighthouse code. G and H are small once F exists. +- **I** reuses S7 from F and otherwise stands alone. +- **J** reuses the direct-display driver from F and I; its tracking is the headset's own. +- The largest unknowns are on the Mac: whether macOS shows a headset's display through a USB-C to DisplayPort adapter + (F, H, I), whether it drives the Vive Pro 2 at all (H), and whether libsurvive works on macOS (F, G, H). + +## Sources + +The research reports behind this plan cite their sources fact by fact. The main ones: + +- ALVR (vendored at `external/ALVR`): `alvr/system_info/src/lib.rs` (model detection), `client_core/src/connection.rs`, + `client_openxr/src/lib.rs` and `interaction.rs`, `server_core/src/connection.rs`, `adb/src/commands.rs`, + `server_openvr/src/props.rs`. +- OpenHMD (vendored at `external/OpenHMD-rift`), Monado's driver documentation + (https://monado.pages.freedesktop.org/monado/), libsurvive (https://github.com/cntools/libsurvive), the community + Vive Pro 2 driver (https://github.com/CertainLach/VivePro2-Linux-Driver), Valve's udev rules + (https://github.com/ValveSoftware/steam-devices), Linux's EDID quirks (`drivers/gpu/drm/drm_edid.c`). +- Khronos OpenXR registry (`XR_META_touch_controller_plus`); Valve's Steam Frame input documentation + (https://partner.steamgames.com/doc/steamhardware/steamframe/input); Meta's refresh-rate documentation + (https://developers.meta.com/horizon/documentation/native/android/mobile-display-refresh-rate/). +- Specifications: Meta, Valve and HTC product pages, Road to VR, UploadVR, Tom's Hardware, Wikipedia; for the Steam + Frame, https://vr.org/steam-frame-specs (updated November 2026) and ALVR issue + https://github.com/alvr-org/ALVR/issues/3411. +- macOS's own display overrides in `/System/Library/Displays/Contents/Resources/Overrides`. +- PlayStation VR2: Monado's `drivers/psvr2` and `drivers/pssense` (in the reference checkout of Monado), the Linux VR + Adventures wiki (https://vronlinux.org/docs/hardware/psvr2/), and the PSVR2 Toolkit + (https://github.com/BnuuySolutions/PSVR2Toolkit). diff --git a/docs/RIFT.md b/docs/RIFT.md index 8658102..4c01b42 100644 --- a/docs/RIFT.md +++ b/docs/RIFT.md @@ -1,4 +1,7 @@ -# Oculus Rift CV1 +# The direct-display driver and the Rift CV1 + +How MetalVR drives a headset plugged into the Mac itself, told through the Rift CV1, on which it was built. To set a +headset up, see its guide in [headsets/](headsets/README.md). `metalvrd --driver rift` drives an Oculus Rift CV1 plugged into the Mac itself: its HID interface turns the headset on and gives its rotation, and frames are drawn on its display with its lens distortion. Oculus's own @@ -6,9 +9,16 @@ software is not involved, and none is needed. With Rift Sensors the headset and the Touch controllers are tracked in position as well as rotation (below); without them, rotation only, and the head stays where a seated user's would be. `--ipd` sets the eye distance, otherwise the -headset's slider is read. [RIFT_CV1.md](RIFT_CV1.md) is the short path from a release to a game, with the +headset's slider is read. [headsets/RIFT_CV1.md](headsets/RIFT_CV1.md) is the short path from a release to a game, with the `metalvr-rift` script. +The same driver drives a **Rift S** (2833:0051, which USB decides), as `--driver lighthouse` an HTC Vive, Vive Pro, +Vive Pro 2 or Valve Index, and as `--driver psvr2` a PlayStation VR2: the window on the headset's display, the pacing, the home and the head's +rotation are this driver's for all of them, and each one's USB side is a backend of its own +(`daemon/direct_backend.h`, [HEADSETS.md](HEADSETS.md)). This page is the CV1's. A Rift S has no sensors: its +position comes from its own cameras through Basalt, when `scripts/build-basalt.sh` has built it and macOS lets +`metalvrd` use cameras (System Settings, Privacy & Security, Camera); otherwise it tracks rotation only. + ## Setting the Mac up 1. **The headset's HDMI goes into one of the Mac's own HDMI ports.** Through a USB-C hub or adapter, macOS may read @@ -19,12 +29,11 @@ headset's slider is read. [RIFT_CV1.md](RIFT_CV1.md) is the short path from a re lifts that, once: ```sh - sudo mkdir -p /Library/Displays/Contents/Resources/Overrides/DisplayVendorID-3ed2 - sudo /usr/libexec/PlistBuddy -c "Add :DisplayVendorID integer 16082" -c "Add :DisplayProductID integer 4" \ - -c "Add :is-hmd bool false" /Library/Displays/Contents/Resources/Overrides/DisplayVendorID-3ed2/DisplayProductID-4 + scripts/metalsharp/metalvr display rift # the CV1's and the Rift S's; asks for your password ``` - Deleting that file undoes it. + It copies Apple's override to `/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-3ed2/` with + `is-hmd` false. `metalvr display rift --undo`, or deleting those files, undoes it. 3. The headset's display is dark until software turns it on, so it appears only once `metalvrd` runs. The first time, macOS may place it among the other displays and make it the main one: in System Settings, Displays, put the menu bar back on a monitor. macOS remembers that arrangement. @@ -41,7 +50,8 @@ horizon and a red mark straight ahead, for checking head tracking and the lens c Straight ahead is the way the headset faces when `metalvrd` starts, and again each time the headset is put on (its proximity sensor). Applications then use MetalVR as their OpenXR runtime as with any other driver (BUILDING.md, METALSHARP_WINE.md, OPENCOMPOSITE.md). -`--eye WxH` changes the per-eye render size applications are asked for (default 1344x1600). +`--eye WxH` changes the per-eye render size applications are asked for (default the headset's: 1344x1600 for the +CV1). ## How it works diff --git a/docs/STEAMVR.md b/docs/STEAMVR.md index e2bd596..6866fa5 100644 --- a/docs/STEAMVR.md +++ b/docs/STEAMVR.md @@ -38,7 +38,9 @@ lines. The compositor's next frame starts at the headset's next refresh, as `met `metalvr steamvr --off` takes it out again. The file as it was before the first change is kept beside it as `openvrpaths.vrpath.metalvr-orig`. The same command keeps the Steam client itself out of VR (below); restart Steam after it. -4. Start `metalvrd` (for a Rift: `Rift Server.command`, or `metalvr rift --verbose`), then start SteamVR. +4. Start `metalvrd` (for a headset on the Mac's own display, its script: `metalvr-rift start`, + `metalvr-lighthouse start` or `metalvr-psvr2 start`; see its guide in [headsets/](headsets/README.md)), then + start SteamVR. SteamVR needs `metalvrd` running when it starts. Without it the driver reports no headset, and SteamVR says so. @@ -60,9 +62,15 @@ SteamVR needs `metalvrd` running when it starts. Without it the driver reports n bindings, gets no answer, so applications get no controller input. `metalvr steamvr` keeps `steam.exe` alone from loading that DLL (a per-application Wine override; games use their own copy). The cost: Steam does not show itself in the headset, and writes an assert dump about the DLL when it starts. -- Oculus Touch controllers are registered with the headset and follow what `metalvrd` hears from them over the - headset's radio (buttons, triggers, thumbsticks, haptics). With the Rift's sensors they and the headset are tracked - in position; without sensors the headset is tracked in rotation only, at a fixed seated height, and the +- The controllers are registered as the ones in the user's hands, which `metalvrd` names (`metalvr_headsets.h`, + [HEADSETS.md](HEADSETS.md)): SteamVR's own Touch profile with the CV1's, Rift S's, Quest 2's or Quest 3's models; + its Index controller profile, with each finger's curl and the hand skeleton; or its Vive controller profile. + The Steam Frame's controllers are shown as ringless Touch controllers until SteamVR's own names for them are known, + and the PlayStation VR2's Sense controllers as Touch controllers named for the Sense. + Which ones they are is read when SteamVR starts the driver: a headset that connects later with other controllers + needs SteamVR started again. Battery levels show where the headset reports them (ALVR's headsets). +- With the Rift's sensors the controllers and the headset are tracked in position; without them, and on the other + headsets MetalVR drives itself, the headset is tracked in rotation only, at a fixed seated height, and the controllers are placed beside the body. - SteamVR covers the whole view with a fade color when it judges the application's frames or the tracking not good enough, and with MetalVR it never lifts that fade: the headset shows one flat color (teal). `metalvr steamvr` diff --git a/docs/headsets/HTC_VIVE.md b/docs/headsets/HTC_VIVE.md new file mode 100644 index 0000000..be7e59d --- /dev/null +++ b/docs/headsets/HTC_VIVE.md @@ -0,0 +1,82 @@ +# HTC Vive, Vive Pro and Vive Pro 2 + +**Tried on hardware:** no. Built from the open Lighthouse drivers (Monado's, OpenHMD's, libsurvive, and the +community Vive Pro 2 driver) and checked without the headsets. Reports from owners are very welcome. + +The Vives use the `lighthouse` driver, shared with the [Valve Index](VALVE_INDEX.md), and the `metalvr-lighthouse` +script, which finds which one is on USB. Their position and their controllers' come from your base stations through +libsurvive, which the release includes. + +| | Vive | Vive Pro | Vive Pro 2 | +|---|---|---|---| +| Video into the Mac | HDMI (the Mac's own HDMI port) or mini DisplayPort | DisplayPort (USB-C or Thunderbolt to DisplayPort) | DisplayPort; its sharpest modes need DisplayPort's compression (DSC) | +| USB | `0bb4:2c87` with `28de:2000` | `0bb4:0309` or `030e` with `28de:2300` | `0bb4:0342` with `28de:2300` | +| Display macOS hides as a headset's | `22d2:aa01` | `22d2:aa02` or `aa03` (which is a guess) | `22d2:aa04` | +| Base stations | 1.0 | 1.0 or 2.0 | 1.0 or 2.0 | +| Controllers | Vive wands (or Index controllers) | Vive wands or Index controllers | Vive wands or Index controllers | + +## What you need + +- An Apple silicon Mac with the video output in the table, the headset's link box (Vive, Vive Pro) or its adapter + (Vive Pro 2), its USB in a USB 3 port, and its power. +- Base stations, and controllers already paired with the headset (pairing is done once in SteamVR on Windows or + Linux; the pairing lives in the headset's receivers). +- MetalSharp, its Windows Steam, and SteamVR there for SteamVR games. + +## Setting up and playing + +As for the Index ([VALVE_INDEX.md](VALVE_INDEX.md#setting-up-and-playing)): `metalvr-lighthouse install`, `display`, +`setup`, then `steamvr` or `game APPID`. `metalvr-lighthouse status` names the Vive it finds on USB. + +**The Vive Pro 2's modes.** `metalvr-lighthouse start --pro2-mode N` switches the headset to a mode first: + +| N | Size | Rate | +|---|---|---| +| 0 | 2448×1224 | 90 Hz | +| 1 | 2448×1224 | 120 Hz | +| 2 | 3264×1632 | 90 Hz | +| 3 | 3680×1836 | 90 Hz | +| 4 | 4896×2448 | 90 Hz (needs DSC) | +| 5 | 4896×2448 | 120 Hz (needs DSC) | + +Mode 2 is the largest an Apple silicon Mac is expected to drive without DSC. + +`--passthrough` shows the room through the headset's front camera (or the Vive Pro's two) in MetalVR's home. + +## What works, checked without the headset + +| | Checked by | +|---|---| +| Finding each on USB and telling them apart (Vive, Vive Pro, Vive Pro 2, Index; a Rift is never taken for one) | `headset-protocols` | +| Their configuration from their firmware: lenses, IMU calibration | `headset-protocols` | +| Each one's lens correction, the Vive Pro 2's upside-down image, drawn offscreen | `panel-vive`, `panel-vive-pro`, `panel-vive-pro-2` | +| Head rotation from the IMU; the Vive Pro's and Pro 2's IPD and proximity | `headset-protocols` | +| The Vive Pro 2's mode commands | `headset-protocols` | +| Vive wands through their Watchman radio: buttons, trackpad, trigger, rotation, battery, haptics | `headset-protocols` | +| The Vive controller profile in OpenXR | `native-vive-wands` | +| Positions from the base stations, through libsurvive | `libsurvive` (libsurvive's own simulated room) | +| The display overrides the script writes, including the Vive Pro 2's EDID patches copied from Apple's | `metalvr display` with `METALVR_DISPLAY_DRY_RUN=1`, by hand | + +Built, but not covered by a test: SteamVR's Vive wands from MetalVR's SteamVR driver. + +## Unknown until tried on one + +- Whether macOS shows each one's display and in which modes; the pixel-clock limit; DSC for the Vive Pro 2. +- Power-on, the IMU's axes, and USB access to their interfaces on macOS. +- Which of `aa02` and `aa03` is the Vive Pro's display. +- The Vive Pro 2's `chipreset` re-enumeration after a mode change. +- libsurvive's tracking with real base stations on a Mac, and Lighthouse 1.0 tracking quality. +- The lens correction on real lenses, and haptics. + +## Not supported + +- The original Vive's cameras through anything but MetalVR's home; eye tracking (the Vive Pro Eye's). +- Without base stations, rotation only. + +## When it doesn't work + +As for the Index ([VALVE_INDEX.md](VALVE_INDEX.md#when-it-doesnt-work)). A Vive's HDMI must go into the Mac's own +HDMI port: through a hub or adapter, macOS may read the display and never turn it on. + +The research and sources: [HEADSETS.md](../HEADSETS.md#phase-g-htc-vive) and +[HEADSETS.md](../HEADSETS.md#phase-h-htc-vive-pro-and-vive-pro-2). diff --git a/docs/headsets/PSVR2.md b/docs/headsets/PSVR2.md new file mode 100644 index 0000000..74cc5de --- /dev/null +++ b/docs/headsets/PSVR2.md @@ -0,0 +1,72 @@ +# Sony PlayStation VR2 + +**Tried on hardware:** no. Built from Monado's PlayStation VR2 and Sense controller drivers and checked without the +headset. If you have one, a report of how far it gets is the most useful thing you can send. + +The PS VR2 has its own driver, `psvr2`. It tracks itself: its cameras and processor work out where it is and send +the finished pose over USB, so MetalVR needs no base stations and no camera tracker for it. + +## What you need + +- **Sony's PlayStation VR2 PC adapter**, with its power supply. The headset's one USB-C cable carries video, USB and + 12 V power, which no Mac port gives together. +- An Apple silicon Mac with a **DisplayPort output**: a USB-C or Thunderbolt port with a USB-C to DisplayPort cable + into the adapter's DisplayPort input. The adapter's USB plug into a USB 3 port on the Mac. +- The Sense controllers paired with the Mac over Bluetooth (System Settings, Bluetooth; hold the controller's + Create or Options button and the PS button until its light blinks). +- MetalSharp, its Windows Steam, and SteamVR there for SteamVR games. + +## Setting up and playing + +The steps are the CV1's ([RIFT_CV1.md](RIFT_CV1.md)) with `metalvr-psvr2` in place of `metalvr-rift`, except that +there is no display step: macOS does not hide the PS VR2's display (it ships no override for Sony's displays). + +```sh +metalvr-psvr2 install # from the unpacked release, if not installed yet +metalvr-psvr2 setup # once: SteamVR in MetalSharp's Steam +metalvr-psvr2 steamvr # the PlayStation VR2 server, then SteamVR +metalvr-psvr2 game APPID # the server, then a Steam game in VR +``` + +Or run the server by itself: `metalvr psvr2` (`metalvrd --driver psvr2`). + +## What works, checked without the headset + +| | Checked by | +|---|---| +| Its status report: worn, the function button, the IPD dial, and its IMU at 2 kHz turned into the headset's axes | `headset-protocols` | +| Its own tracking's pose: the record, its axes, Monado's correction, and the quaternion kept continuous | `headset-protocols` | +| The requests that run it: camera mode, brightness, firmware and serial number, lens calibration | `headset-protocols` | +| Its lens: Monado's model with its colour table and each eye's calibration, on the CPU and in the shader | `headset-protocols`, `panel-psvr2` | +| The Sense controllers' reports: buttons (cross and circle as A and B, square and triangle as X and Y), stick, trigger, touches, battery, the Bluetooth CRC | `headset-protocols` | +| Their IMU calibration, and their vibration report | `headset-protocols` | +| The PS VR2 and its Sense controllers as OpenXR sees them (Touch) | `native-psvr2-sense` | +| The driver waiting for the headset without one, and stopping cleanly | by hand | + +Built, but not covered by a test: opening the headset with IOUSBHost and reading its endpoints, the Sense +controllers over IOHID, and SteamVR showing them as Touch controllers named for the Sense. + +## Unknown until tried on one + +- Whether macOS lets MetalVR open the headset's USB interfaces and take its vendor requests. +- Whether the adapter's display comes up on a Mac at 4000×2040 and 90 or 120 Hz (it may need DisplayPort's + compression), and its display's product code (until it is known, MetalVR takes any Sony display and names it in + the log). +- The Sense controllers' pairing with a Mac, and their IMU's axes. +- Whether the headset's tracking starts as it does on Linux, with the cameras run as Monado runs them. + +## Not supported + +- The controllers' position: the headset's cameras see their LEDs, but no open driver solves that yet. They are + rotation only, held where a hand would be. +- Eye tracking, passthrough, the headset's rumble, and the adaptive triggers. + +## When it doesn't work + +- **No headset found.** `metalvr-psvr2 status` says whether a PlayStation VR2 (`054c:0cde`) is on USB; the log says + which interface or request failed. +- **The headset is found, its display stays dark.** The log says `psvr2: the headset's DisplayPort input has a + signal` once the adapter's video reaches it; check the DisplayPort cable and the adapter's power. +- **No controllers.** The log names each Sense controller when it connects; pair them in System Settings first. + +The research and sources: [HEADSETS.md](../HEADSETS.md#phase-j-sony-playstation-vr2). diff --git a/docs/headsets/QUEST.md b/docs/headsets/QUEST.md new file mode 100644 index 0000000..c92ca30 --- /dev/null +++ b/docs/headsets/QUEST.md @@ -0,0 +1,72 @@ +# Meta Quest 1, 2, 3, 3S and Pro + +**Tried on hardware:** no. The streaming path has run end to end against ALVR's own client code on one Mac, but no +Quest has shown a picture from MetalVR yet. If you have one, a report is very welcome. + +A Quest runs the stock [ALVR](https://github.com/alvr-org/ALVR) client (version 20.x) and MetalVR streams to it: +`metalvrd`'s `alvr` driver links ALVR's own server, and encodes with the Mac's hardware encoder. Nothing Meta makes +runs on the Mac. + +## What you need + +- An Apple silicon Mac and the Quest on the same network (5 GHz or 6 GHz Wi-Fi, the Mac preferably wired), or the + Quest on USB. +- ALVR's client **v20.x** on the Quest (from the Meta store, or sideloaded from ALVR's releases). +- MetalSharp, its Windows Steam, and SteamVR there for SteamVR games. + +## Setting up + +1. Install MetalVR (`metalvr-rift install` from an unpacked release, which also makes MetalVR the Steam prefix's + OpenXR runtime; [RIFT_CV1.md](RIFT_CV1.md#setting-up-once)), and for SteamVR, `metalvr-rift setup`. +2. Start the ALVR client on the Quest. It shows its name, such as `1234.client`. +3. Start the server, trusting that headset (once; it is remembered): + + ```sh + ~/.metalsharp/metalvr-runtime/bin/metalvr start --trust 1234.client + ``` + + A headset that is not trusted is refused and named in the server's output. + +**On USB instead of Wi-Fi:** turn on the Quest's developer mode and USB debugging, install Android's platform-tools +(`brew install android-platform-tools`), and trust the wired name: `metalvr start --trust client.wired` (or +`--adb PATH` for another `adb`). + +Then start games from MetalSharp as usual, or SteamVR with `metalvr launch steamvr`. + +## Which Quest it is + +The client says which Quest it runs on, and MetalVR names the headset by it: its name and refresh rates in OpenXR and +SteamVR, its controllers (Touch on a Quest 1 or 2, Touch Plus on a 3, 3S or Pro), and a warning when the stream is +more than its video decoder takes (ALVR's default is too much for a Quest 1). `--headset quest-2` (or `quest-1`, +`quest-3`, `quest-3s`, `quest-pro`) names it instead, for a client that does not say. + +## What works, checked without the headset + +| | Checked by | +|---|---| +| The model from what the client reports, never mistaking a 3S for a 3 | `alvr-models` | +| The driver starting, trusting only the named headsets, and stopping; wired mode refusing to start without an `adb` | `alvr-driver`, `alvr-models` | +| The stream (hardware HEVC, the handshake, poses, inputs, haptics), against ALVR's own client code on the same Mac | by hand, not in the suite: [ALVR.md](../ALVR.md#what-has-been-verified-and-what-has-not) | +| Hand tracking: the joints the client sends, as `XR_EXT_hand_tracking` and SteamVR's hand skeletons | `native` (the simulated headset's hands) | +| ALVR's foveated encoding (`--foveated-encoding`): each eye squeezed toward its edges as ALVR's client expects | `foveation`, `native-foveated` | +| Each Quest's controllers as OpenXR profiles and SteamVR models | `headset-protocols`, the simulated headset | + +Built, but not covered by a test (ALVR sends them only from a headset): the headset's and controllers' battery +levels, and the play area's bounds as OpenXR's stage bounds. + +## Unknown until tried on one + +- That the Quest's decoder accepts the stream and shows it; latency, reprojection and judder. +- The resolution and refresh rates the client really reports, and HEVC at 120 Hz. +- The controllers' grip and aim poses (the aim offset is an estimate) and their buttons from real hardware. +- Wi-Fi streaming and discovery (the checks used the same machine), and wired mode through adb from a Mac. +- Whether ALVR's current client still runs on a Quest 1. + +## Not supported + +- Audio in either direction; eye and face tracking; changing the refresh rate (the stream stays at the negotiated + one); AV1; trusting a headset while the server runs (start it again with `--trust`). +- Touch Plus's force, curl and slide, which ALVR does not send. + +How the ALVR driver works: [ALVR.md](../ALVR.md). The research and sources: +[HEADSETS.md](../HEADSETS.md#phase-a-meta-quest-1). diff --git a/docs/headsets/README.md b/docs/headsets/README.md new file mode 100644 index 0000000..2b30a38 --- /dev/null +++ b/docs/headsets/README.md @@ -0,0 +1,23 @@ +# Headsets + +One guide per headset: what you need, how to set it up and play, what MetalVR's tests check without the headset, +and what is still unknown until someone tries one. Only the Rift CV1 has been run on real hardware so far. The others +are built from their specifications and the open drivers that support them, and reports from owners are very +welcome ([CONTRIBUTING.md](../../CONTRIBUTING.md)). + +| Headset | How it reaches the Mac | Tried on hardware | Guide | +|---|---|---|---| +| Oculus Rift CV1 | HDMI and USB on the Mac; Rift Sensors | yes | [RIFT_CV1.md](RIFT_CV1.md) | +| Oculus Rift S | DisplayPort and USB on the Mac; its own cameras | no | [RIFT_S.md](RIFT_S.md) | +| Meta Quest 1, 2, 3, 3S, Pro | streamed with ALVR, over Wi-Fi or USB | no | [QUEST.md](QUEST.md) | +| Valve Steam Frame | streamed with ALVR's client, patched by MetalVR | no | [STEAM_FRAME.md](STEAM_FRAME.md) | +| Valve Index | DisplayPort and USB on the Mac; Lighthouse base stations | no | [VALVE_INDEX.md](VALVE_INDEX.md) | +| HTC Vive, Vive Pro, Vive Pro 2 | HDMI or DisplayPort and USB on the Mac; Lighthouse base stations | no | [HTC_VIVE.md](HTC_VIVE.md) | +| Sony PlayStation VR2 | Sony's PC adapter: DisplayPort and USB on the Mac | no | [PSVR2.md](PSVR2.md) | + +Every headset needs the same things on the game's side: [MetalSharp](https://github.com/metalsharp/MetalSharp) with +its Windows Steam, and for SteamVR games, SteamVR in that Steam and MetalVR's SteamVR driver +([STEAMVR.md](../STEAMVR.md)). OpenXR games use MetalVR directly. + +The research behind each headset (specifications, how each is recognized, sources) is in +[HEADSETS.md](../HEADSETS.md). diff --git a/docs/RIFT_CV1.md b/docs/headsets/RIFT_CV1.md similarity index 74% rename from docs/RIFT_CV1.md rename to docs/headsets/RIFT_CV1.md index ae2f4dc..ebf432f 100644 --- a/docs/RIFT_CV1.md +++ b/docs/headsets/RIFT_CV1.md @@ -1,8 +1,12 @@ -# Oculus Rift CV1 with MetalSharp +# Oculus Rift CV1 -The fastest path from a MetalVR release to a game in a Rift CV1, with the `metalvr-rift` script that comes in the -release (`bin/metalvr-rift`; in this repository, `scripts/metalsharp/metalvr-rift`). How the Rift driver works is in -[RIFT.md](RIFT.md), and SteamVR's side in [STEAMVR.md](STEAMVR.md). +**Tried on hardware:** yes. This is the headset MetalVR is developed on: SteamVR's tutorial, Subnautica, ABE VR and +Raw Data have played on it at 90 Hz. + +The path from a MetalVR release to a game in a Rift CV1 is the `metalvr-rift` script that comes in the release +(`bin/metalvr-rift`; in this repository, `scripts/metalsharp/metalvr-rift`). How the Rift driver works is in +[RIFT.md](../RIFT.md), and SteamVR's side in [STEAMVR.md](../STEAMVR.md). The other headsets have guides of their own +([README.md](README.md)). ## What you need @@ -16,7 +20,7 @@ release (`bin/metalvr-rift`; in this repository, `scripts/metalsharp/metalvr-rif [aaf2tbz/mullion@998c32a6](https://github.com/aaf2tbz/mullion/commit/998c32a61095985e7034c7b56553d79aa81c7193) or later (MetalSharp-Wine's `src/mullion`). SteamVR's compositor and its games share Direct3D 11 buffers between processes, which that Mullion does with `MULLION_METALVR=1`; `metalvr-rift setup` sets it for the Steam prefix - ([STEAMVR.md](STEAMVR.md)). With an older Mullion, SteamVR and its games show black in the headset. + ([STEAMVR.md](../STEAMVR.md)). With an older Mullion, SteamVR and its games show black in the headset. Oculus's own software is not used, and none is needed. @@ -90,7 +94,24 @@ directly. display. If only the first appears, check the HDMI is in the Mac's own port and run `metalvr-rift display`. - **SteamVR or the game is black in the headset.** Run `metalvr-rift setup` again (it sets `MULLION_METALVR=1`) and restart Steam, and check the runtime's Mullion is new enough (see What you need). -- **SteamVR shows no headset.** Start the server before SteamVR; [STEAMVR.md](STEAMVR.md) lists the driver's +- **SteamVR shows no headset.** Start the server before SteamVR; [STEAMVR.md](../STEAMVR.md) lists the driver's messages. - **Steam's windows load forever after SteamVR starts.** `metalvr-rift setup` was not run, or Steam was not restarted after it. + +## What works + +- The display at 2160×1200 and 90 Hz with the CV1's lens correction, and the head's rotation from its IMU. +- Position tracking of the headset and the Touch controllers with up to four Rift Sensors, on USB 3 or USB 2; the + sensors show in SteamVR as base stations. +- The Touch controllers' buttons, sticks, triggers, grips, touches and haptics, in OpenXR and SteamVR. +- The proximity sensor (putting the headset on re-centers) and the remote. + +Checked by `scripts/test-all.sh` as well: the CV1's lens grid drawn offscreen (`panel-rift-cv1`) and the OpenXR and +SteamVR paths on the simulated headset. + +## Known problems + +- BeamNG.drive crashes in its graphics driver. +- Through a USB-C hub or adapter, macOS may read the headset's display and never turn it on: use the Mac's own HDMI. + diff --git a/docs/headsets/RIFT_S.md b/docs/headsets/RIFT_S.md new file mode 100644 index 0000000..e14ae25 --- /dev/null +++ b/docs/headsets/RIFT_S.md @@ -0,0 +1,57 @@ +# Oculus Rift S + +**Tried on hardware:** no. Built from the open Rift S drivers (OpenHMD's and Monado's) and checked without the +headset. If you have one, a report of how far it gets is the most useful thing you can send. + +The Rift S uses the same driver and the same `metalvr-rift` script as the [Rift CV1](RIFT_CV1.md): whichever is on +USB decides. It has no external sensors: its position comes from its own five cameras. + +## What you need + +- An Apple silicon Mac with a **DisplayPort output**: a USB-C or Thunderbolt port with a USB-C to DisplayPort cable + or adapter (the Rift S has no HDMI). Its USB in a USB 3 port. +- A MetalVR release (it includes Basalt, the camera tracker; building from source, `scripts/build-basalt.sh`). +- MetalSharp, its Windows Steam, and SteamVR there for SteamVR games, as for the CV1. + +## Setting up and playing + +The steps are the CV1's ([RIFT_CV1.md](RIFT_CV1.md)): `metalvr-rift install`, `metalvr-rift display` (the Rift S's +display is `3ed2:0012`, which macOS hides as a headset's, as it does the CV1's), `metalvr-rift setup`, then +`metalvr-rift steamvr` or `metalvr-rift game APPID`. `metalvr-rift status` says "a Rift S is on USB" once it sees it. + +**Allow the camera.** The first time the server starts with a Rift S, macOS asks whether the app that started it +(Terminal, or MetalSharp) may use the camera: the Rift S's cameras reach macOS as a camera named "Rift S Sensor". +Without that, the headset tracks rotation only. + +## What works, checked without the headset + +| | Checked by | +|---|---| +| Finding it on USB (`2833:0051`), its three HID interfaces, power-on and keep-alive, the screen following the proximity sensor | `headset_protocol_check` | +| Its IMU with its own calibration (firmware block 0x0e) | `headset_protocol_check` | +| The portrait 1440×2560 panel split into two eyes, each turned, with Monado's lens values | `panel-rift-s` | +| Touch controllers: buttons, stick, trigger, grip, touches and rotation, through the headset's radio | `headset_protocol_check` | +| Position from its cameras: their calibration (firmware block 0x0f), their lenses fitted for Basalt, the camera frames' layout, auto exposure, and Basalt's visual-inertial tracking | `headset_protocol_check`, `rift-s-tracking` (a synthetic Rift S tracked to 0.2 cm over a 5 m path) | + +## Unknown until tried on one + +- Whether macOS brings its DisplayPort link up at 80 Hz, unrotated, through a USB-C adapter. +- The order of power-on and proximity, and its USB interfaces as macOS presents them. +- Whether macOS's camera driver hands over the "Rift S Sensor" frames unconverted, and how Basalt tracks on real + pictures. +- The lens correction: no correct Rift S lens model is published; Monado's values are a starting point to tune by eye. +- The IMU's axis signs. + +## Not supported + +- The controllers' position (no open driver tracks it): they are rotation only, held where a hand would be. +- The controllers' haptics (the command is not known). + +## When it doesn't work + +- **No position, only rotation.** The log says why: `the Rift S tracks rotation only: ...` (no Basalt, or no camera + calibration), or no `reading the Rift S's cameras` line (macOS has no "Rift S Sensor" camera, or camera access was + refused: System Settings, Privacy & Security, Camera). +- Everything else is as for the CV1 ([RIFT_CV1.md](RIFT_CV1.md#when-it-doesnt-work)). + +The research and sources: [HEADSETS.md](../HEADSETS.md#phase-i-oculus-rift-s). diff --git a/docs/headsets/STEAM_FRAME.md b/docs/headsets/STEAM_FRAME.md new file mode 100644 index 0000000..b7cc7a6 --- /dev/null +++ b/docs/headsets/STEAM_FRAME.md @@ -0,0 +1,71 @@ +# Valve Steam Frame + +**Tried on hardware:** no. MetalVR streams to the Frame with ALVR, like a Quest, through ALVR's client patched by +MetalVR to run there. The patched client is built and checked, but has not run on a Frame. + +Valve's own streaming (Steam Link) needs Windows SteamVR and a PC's GPU encoder, so it is not an option on a Mac. +The stock ALVR client shows a blank screen on the Frame: it requires an OpenXR extension +(`XR_KHR_convert_timespec_time`) that Valve's runtime does not have. MetalVR's patch +(`patches/alvr/0003-client-runs-on-the-steam-frame.patch`) makes the client do without it, and name the headset +"Steam Frame" so that MetalVR recognizes it. + +## What you need + +- An Apple silicon Mac and the Frame on the same network (Wi-Fi 6E or 7 on the Frame's side; the Mac preferably + wired). +- The Frame in developer mode, for installing the client with `adb`, and Android's platform-tools on the Mac + (`brew install android-platform-tools`). +- The patched client's APK. Build it from this repository: + + ```sh + scripts/build-alvr-client.sh # fetches the Android tools into build/android; shows Google's SDK license + ``` + + which writes `build/alvr-client/alvr_client_android.apk`. +- MetalSharp, its Windows Steam, and SteamVR there for SteamVR games. + +## Setting up + +1. Install the client on the Frame: + + ```sh + adb install -r build/alvr-client/alvr_client_android.apk + ``` + +2. Install MetalVR and set SteamVR up as for a Quest ([QUEST.md](QUEST.md#setting-up)). +3. Start the client on the Frame, note its name (such as `1234.client`), and start the server trusting it: + + ```sh + ~/.metalsharp/metalvr-runtime/bin/metalvr start --trust 1234.client + ``` + +The patched client names the headset "Steam Frame" by itself. With the stock client (which reports "Android +(unknown)"), add `--headset steam-frame`. + +## What works, checked without the headset + +| | Checked by | +|---|---| +| The patched client: it builds as an APK (verified: signed, arm64, the "Steam Frame" name compiled in), and its Android code compiles | `scripts/build-alvr-client.sh`, by hand | +| MetalVR naming the headset from the client's "Steam Frame" | `alvr-models`, `headset-protocols` | +| The Frame's entry in the model table (2160×2160 per eye, 72 to 120 Hz) | `headset-protocols` | +| ALVR's foveated encoding (`--foveated-encoding`), fixed at the centre | `foveation`, `native-foveated` | + +Built, but not covered by a test: its controllers' extra inputs (the bumper, D-pad, View button, and X and Y on the +right) through OpenXR (Valve's `frame_controller_valve` profile, with Touch as the fallback) and SteamVR. The +simulated headset presses them when it stands in for a Frame (`metalvrd --driver sim --headset steam-frame`). + +## Unknown until tried on one + +- Whether the patched client runs and streams on the Frame. The Frame's Android container has only software video + decoders, so its decoding speed decides the usable resolution and rate. +- Latency, the real field of view, and the headset's USB network (whether a cable could carry the stream on a Mac). +- SteamVR's own names for the Frame's controllers (they are shown as Touch controllers until then). + +## Not supported + +- Foveation following the eyes' gaze (it needs the client to send the gaze). +- Everything a Quest lacks too ([QUEST.md](QUEST.md#not-supported)). + +The research and sources: [HEADSETS.md](../HEADSETS.md#phase-e-valve-steam-frame); the patch: +[ALVR.md](../ALVR.md#the-patches). diff --git a/docs/headsets/VALVE_INDEX.md b/docs/headsets/VALVE_INDEX.md new file mode 100644 index 0000000..237d8c8 --- /dev/null +++ b/docs/headsets/VALVE_INDEX.md @@ -0,0 +1,73 @@ +# Valve Index + +**Tried on hardware:** no. Built from the open Lighthouse drivers (Monado's, OpenHMD's and libsurvive) and checked +without the headset. If you have one, a report is very welcome. + +The Index uses the `lighthouse` driver, which it shares with the HTC Vives ([HTC_VIVE.md](HTC_VIVE.md)), and the +`metalvr-lighthouse` script (the release's `metalvr-rift` under another name, which looks for Lighthouse headsets). +Its position and its controllers' come from your base stations, through libsurvive, which the release includes. + +## What you need + +- An Apple silicon Mac with a **DisplayPort output**: a USB-C or Thunderbolt port with a USB-C to DisplayPort cable + or adapter. The Index's USB in a USB 3 port, and its power adapter. +- Base stations (1.0 or 2.0), and the Index controllers already paired with the headset (pairing is done once in + SteamVR on Windows or Linux; the pairing lives in the headset's receivers). +- MetalSharp, its Windows Steam, and SteamVR there for SteamVR games. + +## Setting up and playing + +The steps are the CV1's ([RIFT_CV1.md](RIFT_CV1.md)) with `metalvr-lighthouse` in place of `metalvr-rift`: + +```sh +metalvr-lighthouse install # from the unpacked release, if not installed yet +metalvr-lighthouse display # once per Mac: macOS hides the Index's display (5996:91a8) as a headset's +metalvr-lighthouse setup # once: SteamVR in MetalSharp's Steam +metalvr-lighthouse steamvr # the Lighthouse server, then SteamVR +metalvr-lighthouse game APPID # the Lighthouse server, then a Steam game in VR +``` + +- `metalvr-lighthouse start --refresh 120` (or 80, 90, 144) sets the Index's refresh rate. +- `--passthrough` shows the room through the Index's front cameras in MetalVR's home; macOS asks for camera access + the first time. +- The first time, libsurvive works out where your base stations stand: keep the headset still in view of them for a + few seconds. It remembers them in `~/Library/Application Support/MetalVR/libsurvive.json`. + +## What works, checked without the headset + +| | Checked by | +|---|---| +| Finding the Index on USB (`28de:2300`, confirmed by the model in its configuration) | `headset-protocols` | +| Its configuration from its firmware: lenses, canted displays, IMU calibration | `headset-protocols` | +| Its lens correction and canted eyes, drawn offscreen | `panel-index` | +| Head rotation from its IMU, the proximity sensor, the IPD | `headset-protocols` | +| The Index controllers through their Watchman radio: buttons, trackpad, each finger's curl, grip force, rotation, battery, haptics | `headset-protocols` | +| The Index controller profile in OpenXR | `native-index-touch` | +| Positions of the head and controllers from the base stations, through libsurvive | `libsurvive` (libsurvive's own simulated room) | +| The home showing the room through the front cameras | `passthrough` | + +Built, but not covered by a test: SteamVR's Index controllers with each finger's curl and the hand skeleton, from +MetalVR's SteamVR driver. + +## Unknown until tried on one + +- Whether macOS shows the Index's display through a USB-C to DisplayPort adapter, and which refresh rates it offers + (144 Hz only just fits DisplayPort 1.2). +- Whether the display needs a power-on command, and USB access to its interfaces on macOS. +- libsurvive's tracking with real base stations on a Mac (it has run only in its simulated room). +- The controllers' pairing state and haptics on real hardware. +- The cameras are drawn without their own calibration: the passthrough view is a fixed field of view. + +## Not supported + +- Eye tracking (the Index has none) and passthrough for games (only MetalVR's home shows it). + +## When it doesn't work + +- **The headset stays dark.** `metalvr-lighthouse status`, then `metalvr-lighthouse display` if the display is + hidden; check the DisplayPort cable and the headset's power. +- **Rotation only.** The log says whether libsurvive started and what it sees; check the base stations are on and + in view. +- Everything else is as for the CV1 ([RIFT_CV1.md](RIFT_CV1.md#when-it-doesnt-work)). + +The research and sources: [HEADSETS.md](../HEADSETS.md#phase-f-valve-index). diff --git a/patches/README.md b/patches/README.md index 97af7f7..ba54245 100644 --- a/patches/README.md +++ b/patches/README.md @@ -9,6 +9,7 @@ A patch is a change to someone else's work and carries that work's license, not |---|---|---| | `alvr/` | [ALVR](https://github.com/alvr-org/ALVR) v20.14.1 | MIT, as ALVR | | `opencomposite/` | [OpenComposite](https://gitlab.com/znixian/OpenOVR) | GPL-3.0, as OpenComposite | +| `libsurvive/` | [libsurvive](https://github.com/cntools/libsurvive) | MIT, as libsurvive | ALVR's `server_core` is linked into `metalvrd`. OpenComposite is built as its own `openvr_api.dll` and is never linked into MetalVR. diff --git a/patches/alvr/0003-client-runs-on-the-steam-frame.patch b/patches/alvr/0003-client-runs-on-the-steam-frame.patch new file mode 100644 index 0000000..fb7fc9b --- /dev/null +++ b/patches/alvr/0003-client-runs-on-the-steam-frame.patch @@ -0,0 +1,52 @@ +diff --git a/alvr/client_openxr/src/lib.rs b/alvr/client_openxr/src/lib.rs +index 3e35add..d98840c 100644 +--- a/alvr/client_openxr/src/lib.rs ++++ b/alvr/client_openxr/src/lib.rs +@@ -185,7 +185,8 @@ pub fn entry_point() { + { + exts.khr_android_create_instance = true; + } +- exts.khr_convert_timespec_time = true; ++ // Valve's runtime on the Steam Frame has no timespec conversion; the clock is then not the runtime's ++ exts.khr_convert_timespec_time = available_extensions.khr_convert_timespec_time; + exts.khr_opengl_es_enable = true; + exts.other = available_extensions + .other +@@ -525,6 +526,8 @@ pub fn entry_point() { + + #[allow(unused)] + fn xr_runtime_now(xr_instance: &xr::Instance) -> Option { ++ // without XR_KHR_convert_timespec_time (the Steam Frame) the runtime's clock cannot be read ++ xr_instance.exts().khr_convert_timespec_time.as_ref()?; + xr_instance + .now() + .ok() +diff --git a/alvr/system_info/src/lib.rs b/alvr/system_info/src/lib.rs +index 87aa4e4..175e852 100644 +--- a/alvr/system_info/src/lib.rs ++++ b/alvr/system_info/src/lib.rs +@@ -35,6 +35,7 @@ pub enum Platform { + Yvr, + PlayForDreamMR, + Lynx, ++ SteamFrame, + AndroidUnknown, + AppleHeadset, + WindowsPc, +@@ -104,6 +105,7 @@ impl Display for Platform { + Platform::Yvr => "YVR", + Platform::PlayForDreamMR => "Play For Dream MR", + Platform::Lynx => "Lynx Headset", ++ Platform::SteamFrame => "Steam Frame", + Platform::AndroidUnknown => "Android (unknown)", + Platform::AppleHeadset => "Apple Headset", + Platform::WindowsPc => "Windows PC", +@@ -153,6 +155,8 @@ pub fn platform() -> Platform { + ("YVR", _, _, _) => Platform::Yvr, + ("Play For Dream", _, _, _) => Platform::PlayForDreamMR, + ("Lynx Mixed Reality", _, _, _) => Platform::Lynx, ++ // the Steam Frame's Android container ("Lepton") reports Valve as its maker ++ ("Valve", _, _, _) => Platform::SteamFrame, + _ => Platform::AndroidUnknown, + } + } diff --git a/patches/libsurvive/0001-simulator-keep-base-station-tracker.patch b/patches/libsurvive/0001-simulator-keep-base-station-tracker.patch new file mode 100644 index 0000000..12dbbc3 --- /dev/null +++ b/patches/libsurvive/0001-simulator-keep-base-station-tracker.patch @@ -0,0 +1,17 @@ +diff --git a/src/driver_simulator.c b/src/driver_simulator.c +index d8a28d4..e0dd6bd 100644 +--- a/src/driver_simulator.c ++++ b/src/driver_simulator.c +@@ -739,7 +739,12 @@ int DriverRegSimulator(SurviveContext *ctx) { + lh_count = 2; + + for (int i = 0; i < lh_count; i++) { ++ // the record's tracker and OOTX decoder belong to the context, which made them in survive_init ++ struct SurviveKalmanLighthouse *tracker = ctx->bsd[i].tracker; ++ void *ootx_data = ctx->bsd[i].ootx_data; + ctx->bsd[i] = simulated_bsd[i]; ++ ctx->bsd[i].tracker = tracker; ++ ctx->bsd[i].ootx_data = ootx_data; + + for (int axis = 0; axis < 2; axis++) { + for (int cal_idx = 0; cal_idx < sizeof(fcalNoise) / sizeof(FLT); cal_idx++) { diff --git a/protocol/metalvr_headsets.h b/protocol/metalvr_headsets.h new file mode 100644 index 0000000..744394c --- /dev/null +++ b/protocol/metalvr_headsets.h @@ -0,0 +1,48 @@ +/* + * What each headset MetalVR can identify is called and what it comes with, shared by the server, the OpenXR + * runtime and the SteamVR driver so that all three describe a headset the same way. Which headset is connected is + * decided by the server's driver (mvr_system_info.headset); this table only describes it. + */ +#ifndef METALVR_HEADSETS_H +#define METALVR_HEADSETS_H + +#include "metalvr_protocol.h" + +#ifdef __cplusplus +extern "C" { +#endif + +typedef struct mvr_headset_description { + const char *name; /* as a person would say it, e.g. "Meta Quest 3" */ + const char *manufacturer; /* as SteamVR shows it */ + const char *key; /* for command lines, e.g. "quest-3" */ + uint32_t controllers; /* mvr_controller_kind it ships with */ + uint32_t vendor_id; /* the maker's USB vendor ID, which OpenXR reports as the system's vendor; 0 if none */ +} mvr_headset_description; + +static inline const mvr_headset_description *mvr_describe_headset(uint32_t headset) { + static const mvr_headset_description table[MVR_HEADSET_COUNT] = { + {"MetalVR Headset", "MetalVR", "unknown", MVR_CONTROLLERS_UNKNOWN, 0}, + {"MetalVR Simulated Headset", "MetalVR", "sim", MVR_CONTROLLERS_TOUCH, 0}, + {"Oculus Rift CV1", "Oculus", "rift-cv1", MVR_CONTROLLERS_TOUCH, 0x2833}, + {"Oculus Rift S", "Oculus", "rift-s", MVR_CONTROLLERS_TOUCH, 0x2833}, + {"Oculus Quest", "Oculus", "quest-1", MVR_CONTROLLERS_TOUCH, 0x2833}, + {"Meta Quest 2", "Oculus", "quest-2", MVR_CONTROLLERS_TOUCH, 0x2833}, + {"Meta Quest 3", "Meta", "quest-3", MVR_CONTROLLERS_TOUCH_PLUS, 0x2833}, + {"Meta Quest 3S", "Meta", "quest-3s", MVR_CONTROLLERS_TOUCH_PLUS, 0x2833}, + {"Meta Quest Pro", "Meta", "quest-pro", MVR_CONTROLLERS_TOUCH_PLUS, 0x2833}, + {"Valve Steam Frame", "Valve", "steam-frame", MVR_CONTROLLERS_STEAM_FRAME, 0x28de}, + {"Valve Index", "Valve", "index", MVR_CONTROLLERS_INDEX, 0x28de}, + {"HTC Vive", "HTC", "vive", MVR_CONTROLLERS_VIVE_WAND, 0x0bb4}, + {"HTC Vive Pro", "HTC", "vive-pro", MVR_CONTROLLERS_VIVE_WAND, 0x0bb4}, + {"HTC Vive Pro 2", "HTC", "vive-pro-2", MVR_CONTROLLERS_VIVE_WAND, 0x0bb4}, + {"PlayStation VR2", "Sony", "psvr2", MVR_CONTROLLERS_PS_SENSE, 0x054c}, + }; + return &table[headset < (uint32_t)MVR_HEADSET_COUNT ? headset : (uint32_t)MVR_HEADSET_UNKNOWN]; +} + +#ifdef __cplusplus +} +#endif + +#endif /* METALVR_HEADSETS_H */ diff --git a/protocol/metalvr_protocol.h b/protocol/metalvr_protocol.h index 01e2454..890c7f0 100644 --- a/protocol/metalvr_protocol.h +++ b/protocol/metalvr_protocol.h @@ -21,7 +21,7 @@ extern "C" { #endif -#define MVR_PROTOCOL_VERSION 1u +#define MVR_PROTOCOL_VERSION 2u #define MVR_MAGIC 0x3152564du /* "MVR1" */ #define MVR_DEFAULT_PORT 37913 #define MVR_PORT_ENV "METALVR_PORT" @@ -55,6 +55,7 @@ enum mvr_message { MVR_MSG_HAPTIC = 10, MVR_MSG_SYSTEM = 11, /* the headset as it is now; it changes when one connects */ MVR_MSG_REFERENCES = 12, /* the tracking sensors that watch the play space, where they stand */ + MVR_MSG_HANDS = 13, /* the hands' joints, from a headset that tracks hands (MVR_SYSTEM_HAND_TRACKING) */ }; /* every response starts with one of these */ @@ -95,16 +96,62 @@ enum mvr_graphics_api { enum mvr_system_flags { MVR_SYSTEM_HMD_CONNECTED = 1u << 0, MVR_SYSTEM_SIMULATED = 1u << 1, + MVR_SYSTEM_HAND_TRACKING = 1u << 2, /* the headset can track bare hands (MVR_MSG_HANDS) */ + MVR_SYSTEM_PASSTHROUGH = 1u << 3, /* the headset's home shows the room through its cameras */ }; +/* Which headset the server drives. A driver names one only when it has identified it (metalvr_headsets.h lists + * what each is); otherwise it says UNKNOWN, and nothing downstream assumes a model. */ +enum mvr_headset { + MVR_HEADSET_UNKNOWN = 0, + MVR_HEADSET_SIMULATED = 1, + MVR_HEADSET_RIFT_CV1 = 2, + MVR_HEADSET_RIFT_S = 3, + MVR_HEADSET_QUEST_1 = 4, + MVR_HEADSET_QUEST_2 = 5, + MVR_HEADSET_QUEST_3 = 6, + MVR_HEADSET_QUEST_3S = 7, + MVR_HEADSET_QUEST_PRO = 8, + MVR_HEADSET_STEAM_FRAME = 9, + MVR_HEADSET_VALVE_INDEX = 10, + MVR_HEADSET_VIVE = 11, + MVR_HEADSET_VIVE_PRO = 12, + MVR_HEADSET_VIVE_PRO_2 = 13, + MVR_HEADSET_PSVR2 = 14, + MVR_HEADSET_COUNT +}; + +/* The physical controllers in the user's hands, which decides the interaction profile an application is given and + * the controllers SteamVR shows. */ +enum mvr_controller_kind { + MVR_CONTROLLERS_UNKNOWN = 0, + MVR_CONTROLLERS_TOUCH = 1, /* Oculus Touch: Rift CV1, Rift S, Quest 1 and 2 */ + MVR_CONTROLLERS_TOUCH_PLUS = 2, /* Meta Touch Plus and Touch Pro: Quest 3, 3S and Pro */ + MVR_CONTROLLERS_INDEX = 3, /* Valve Index controllers */ + MVR_CONTROLLERS_VIVE_WAND = 4, /* HTC Vive controllers */ + MVR_CONTROLLERS_STEAM_FRAME = 5, + MVR_CONTROLLERS_PS_SENSE = 6, /* PlayStation VR2 Sense controllers */ + MVR_CONTROLLERS_COUNT +}; + +#define MVR_MAX_REFRESH_RATES 8u + typedef struct mvr_system_info { uint32_t flags; uint32_t eye_width; /* recommended per-eye render size */ uint32_t eye_height; - float refresh_hz; + float refresh_hz; /* the rate the headset runs at now */ float ipd; mvr_fov fov[2]; char system_name[64]; + uint32_t headset; /* mvr_headset */ + uint32_t controllers; /* mvr_controller_kind */ + /* the rates the headset can run at, refresh_hz among them; 0 entries when only refresh_hz is known */ + uint32_t refresh_rate_count; + float refresh_rates[MVR_MAX_REFRESH_RATES]; + /* the play area the headset's own boundary encloses, meters; 0 when the headset has none or has not said */ + float play_area_width; + float play_area_depth; } mvr_system_info; typedef struct mvr_hello_request { @@ -198,14 +245,22 @@ enum mvr_tracking_flags { MVR_TRACKING_HEAD_VALID = 1u << 1, /* the headset is worn and showing this session: the runtime may go VISIBLE and FOCUSED */ MVR_TRACKING_PRESENTING = 1u << 2, + /* head_battery holds the headset's charge */ + MVR_TRACKING_BATTERY_VALID = 1u << 3, + MVR_TRACKING_CHARGING = 1u << 4, }; enum mvr_controller_flags { MVR_CONTROLLER_CONNECTED = 1u << 0, MVR_CONTROLLER_POSE_VALID = 1u << 1, + /* battery holds the controller's charge */ + MVR_CONTROLLER_BATTERY_VALID = 1u << 2, + MVR_CONTROLLER_CHARGING = 1u << 3, + /* fingers[] are measured (Index controllers), not derived from the buttons */ + MVR_CONTROLLER_FINGERS_VALID = 1u << 4, }; -/* one bit per digital input of a two-handed, Touch-like controller */ +/* one bit per digital input; which of them a controller has depends on its kind (mvr_controller_kind) */ enum mvr_button { MVR_BUTTON_PRIMARY = 1u << 0, /* A / X */ MVR_BUTTON_SECONDARY = 1u << 1, /* B / Y */ @@ -215,8 +270,21 @@ enum mvr_button { MVR_BUTTON_SQUEEZE = 1u << 5, MVR_BUTTON_THUMBSTICK = 1u << 6, MVR_BUTTON_THUMBREST = 1u << 7, + MVR_BUTTON_TRACKPAD = 1u << 8, /* Index and Vive: the trackpad pressed in */ + MVR_BUTTON_BUMPER = 1u << 9, /* Steam Frame */ + MVR_BUTTON_DPAD_UP = 1u << 10, /* Steam Frame, left hand */ + MVR_BUTTON_DPAD_DOWN = 1u << 11, + MVR_BUTTON_DPAD_LEFT = 1u << 12, + MVR_BUTTON_DPAD_RIGHT = 1u << 13, + MVR_BUTTON_VIEW = 1u << 14, /* Steam Frame, left hand */ + /* Steam Frame, right hand: X and Y sit beside A and B */ + MVR_BUTTON_TERTIARY = 1u << 15, + MVR_BUTTON_QUATERNARY = 1u << 16, }; +/* fingers[] of a controller that senses them, 0 open to 1 curled around the handle */ +enum mvr_finger { MVR_FINGER_INDEX = 0, MVR_FINGER_MIDDLE = 1, MVR_FINGER_RING = 2, MVR_FINGER_PINKY = 3 }; + typedef struct mvr_controller_state { uint32_t flags; uint32_t buttons; /* mvr_button, pressed */ @@ -229,6 +297,12 @@ typedef struct mvr_controller_state { mvr_pose aim; mvr_vec3 linear_velocity; mvr_vec3 angular_velocity; + float trackpad_x; /* -1..1; a controller whose only pad is a trackpad (Vive) reports it here, not as a stick */ + float trackpad_y; + float trackpad_force; /* 0..1, Index */ + float squeeze_force; /* 0..1 beyond a full squeeze, Index */ + float fingers[4]; /* mvr_finger */ + float battery; /* 0..1 when MVR_CONTROLLER_BATTERY_VALID */ } mvr_controller_state; typedef struct mvr_view_state { @@ -252,6 +326,8 @@ typedef struct mvr_tracking_response { mvr_vec3 head_angular_velocity; mvr_view_state views[2]; mvr_controller_state controllers[2]; /* left, right */ + float head_battery; /* 0..1 when MVR_TRACKING_BATTERY_VALID */ + uint32_t reserved; } mvr_tracking_response; enum mvr_layer_type { @@ -318,6 +394,32 @@ typedef struct mvr_references_response { mvr_tracking_reference references[MVR_MAX_TRACKING_REFERENCES]; } mvr_references_response; +/* A hand's joints in OpenXR's order (XR_HAND_JOINT_COUNT_EXT: palm, wrist, then each finger from its base), in the + * tracking space, with each joint's radius in meters. */ +#define MVR_HAND_JOINTS 26u + +enum mvr_hand_flags { + MVR_HAND_TRACKED = 1u << 0, /* the headset sees this hand now */ +}; + +typedef struct mvr_hand { + uint32_t flags; + uint32_t reserved; + mvr_pose joints[MVR_HAND_JOINTS]; + float radii[MVR_HAND_JOINTS]; +} mvr_hand; + +typedef struct mvr_hands_request { + int64_t time_ns; /* the server time to predict for; 0 for now */ +} mvr_hands_request; + +typedef struct mvr_hands_response { + uint32_t result; + uint32_t reserved; + int64_t time_ns; + mvr_hand hands[2]; /* left, right */ +} mvr_hands_response; + typedef struct mvr_haptic_request { uint32_t hand; /* 0 left, 1 right */ float duration_s; /* <= 0 stops */ diff --git a/runtime/CMakeLists.txt b/runtime/CMakeLists.txt index 046f927..60a7bc1 100644 --- a/runtime/CMakeLists.txt +++ b/runtime/CMakeLists.txt @@ -2,6 +2,7 @@ set(RUNTIME_SOURCES instance.cpp session.cpp input.cpp + hands.cpp dispatch.cpp ipc.cpp ) diff --git a/runtime/api.h b/runtime/api.h index c0324b9..defea0f 100644 --- a/runtime/api.h +++ b/runtime/api.h @@ -109,6 +109,9 @@ constexpr uint32_t kMaxSampleCount = 1; X(xrGetInputSourceLocalizedName) \ X(xrApplyHapticFeedback) \ X(xrStopHapticFeedback) \ + X(xrCreateHandTrackerEXT) \ + X(xrDestroyHandTrackerEXT) \ + X(xrLocateHandJointsEXT) \ MVR_PLATFORM_FUNCTIONS(X) // Each entry point is declared from the OpenXR header's own function-pointer type, so a definition whose diff --git a/runtime/hands.cpp b/runtime/hands.cpp new file mode 100644 index 0000000..7fbd341 --- /dev/null +++ b/runtime/hands.cpp @@ -0,0 +1,111 @@ +// XR_EXT_hand_tracking: the hands' joints as the headset tracks them (MVR_MSG_HANDS), located in any space. +#include "api.h" + +#include + +namespace metalvr { +namespace api { + +MVR_API xrCreateHandTrackerEXT(XrSession handle, const XrHandTrackerCreateInfoEXT *info, XrHandTrackerEXT *out) { + MVR_SESSION(handle); + if (!info || !out || info->type != XR_TYPE_HAND_TRACKER_CREATE_INFO_EXT) + return XR_ERROR_VALIDATION_FAILURE; + if (!instance->hasExtension(XR_EXT_HAND_TRACKING_EXTENSION_NAME)) + return XR_ERROR_FUNCTION_UNSUPPORTED; + if (info->hand != XR_HAND_LEFT_EXT && info->hand != XR_HAND_RIGHT_EXT) + return XR_ERROR_VALIDATION_FAILURE; + if (info->handJointSet != XR_HAND_JOINT_SET_DEFAULT_EXT) + return XR_ERROR_VALIDATION_FAILURE; + if (!(instance->system.flags & MVR_SYSTEM_HAND_TRACKING)) + return XR_ERROR_FEATURE_UNSUPPORTED; + auto tracker = std::make_unique(); + tracker->session = session; + tracker->hand = info->hand == XR_HAND_LEFT_EXT ? 0 : 1; + *out = reinterpret_cast(tracker.get()); + session->hand_trackers.push_back(std::move(tracker)); + return XR_SUCCESS; +} + +MVR_API xrDestroyHandTrackerEXT(XrHandTrackerEXT handle) { + HandTracker *tracker = lookup(handle); + if (!tracker) + return XR_ERROR_HANDLE_INVALID; + Session *session = tracker->session; + std::lock_guard lock(session->instance->mutex); + auto &list = session->hand_trackers; + list.erase(std::remove_if(list.begin(), list.end(), [&](const auto &t) { return t.get() == tracker; }), list.end()); + return XR_SUCCESS; +} + +MVR_API xrLocateHandJointsEXT(XrHandTrackerEXT handle, const XrHandJointsLocateInfoEXT *info, + XrHandJointLocationsEXT *locations) { + HandTracker *tracker = lookup(handle); + if (!tracker) + return XR_ERROR_HANDLE_INVALID; + Session *session = tracker->session; + Instance *instance = session->instance; + std::lock_guard lock(instance->mutex); + if (!info || !locations || info->type != XR_TYPE_HAND_JOINTS_LOCATE_INFO_EXT || + locations->type != XR_TYPE_HAND_JOINT_LOCATIONS_EXT || !locations->jointLocations) + return XR_ERROR_VALIDATION_FAILURE; + if (locations->jointCount != XR_HAND_JOINT_COUNT_EXT) + return XR_ERROR_VALIDATION_FAILURE; + const Space *base = lookup(info->baseSpace); + if (!base) + return XR_ERROR_HANDLE_INVALID; + if (info->time <= 0) + return XR_ERROR_TIME_INVALID; + MVR_ALIVE(); + + XrHandJointVelocitiesEXT *velocities = nullptr; + for (auto *next = static_cast(locations->next); next; next = next->next) + if (next->type == XR_TYPE_HAND_JOINT_VELOCITIES_EXT) + velocities = reinterpret_cast(next); + if (velocities && (velocities->jointCount != XR_HAND_JOINT_COUNT_EXT || !velocities->jointVelocities)) + return XR_ERROR_VALIDATION_FAILURE; + + locations->isActive = XR_FALSE; + for (uint32_t j = 0; j < XR_HAND_JOINT_COUNT_EXT; ++j) { + locations->jointLocations[j].locationFlags = 0; + locations->jointLocations[j].pose = {{0, 0, 0, 1}, {0, 0, 0}}; + locations->jointLocations[j].radius = 0; + if (velocities) + velocities->jointVelocities[j].velocityFlags = 0; // the server sends no joint velocities + } + + mvr_hands_request request{info->time}; + mvr_hands_response response{}; + if (!instance->connection.call(MVR_MSG_HANDS, request, response) || response.result != MVR_OK) { + instance->markLost(); + return XR_ERROR_INSTANCE_LOST; + } + const mvr_hand &hand = response.hands[tracker->hand]; + if (!(hand.flags & MVR_HAND_TRACKED)) + return XR_SUCCESS; + + // the joints are in the tracking space; the base space is where the server's tracking for that time puts it + mvr_tracking_response sample{}; + if (!session->tracking(info->time, sample)) + return XR_ERROR_INSTANCE_LOST; + const Located located = session->locate(*base, sample); + if (!located.orientation_valid || !located.position_valid) + return XR_SUCCESS; + const mvr_pose to_base = poseInverse(located.pose); + constexpr XrSpaceLocationFlags kValid = XR_SPACE_LOCATION_ORIENTATION_VALID_BIT | + XR_SPACE_LOCATION_POSITION_VALID_BIT | + XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT | + XR_SPACE_LOCATION_POSITION_TRACKED_BIT; + for (uint32_t j = 0; j < XR_HAND_JOINT_COUNT_EXT; ++j) { + const mvr_pose p = poseMul(to_base, hand.joints[j]); + XrHandJointLocationEXT &out = locations->jointLocations[j]; + out.pose.orientation = {p.orientation.x, p.orientation.y, p.orientation.z, p.orientation.w}; + out.pose.position = {p.position.x, p.position.y, p.position.z}; + out.radius = hand.radii[j]; + out.locationFlags = kValid; + } + locations->isActive = XR_TRUE; + return XR_SUCCESS; +} + +} // namespace api +} // namespace metalvr diff --git a/runtime/input.cpp b/runtime/input.cpp index b7d24fd..e04f9fd 100644 --- a/runtime/input.cpp +++ b/runtime/input.cpp @@ -1,8 +1,9 @@ // Actions: sets, suggested bindings, state, action spaces and haptics. // -// The server reports one kind of controller, a two-handed one laid out like Touch. Every interaction profile an -// application suggests is read as a naming of those same physical inputs, so a game written for Index or Vive -// controllers gets working bindings without the runtime pretending to be that hardware. +// The server says which controllers the user holds (mvr_system_info.controllers). Their own interaction profile is +// preferred when the application suggests bindings for it; every other profile is read as a naming of the same +// physical inputs, so a game written for other controllers still gets working bindings without the runtime +// pretending to be that hardware. #include "api.h" #include @@ -14,6 +15,10 @@ namespace metalvr { namespace { +constexpr const char *kTouchPlusProfile = "/interaction_profiles/meta/touch_plus_controller"; +constexpr const char *kTouchPlusExtensionProfile = "/interaction_profiles/meta/touch_controller_plus"; +constexpr const char *kFrameProfile = "/interaction_profiles/valve/frame_controller_valve"; + // the profiles this runtime binds, most specific layout first, and whether each has a thumbstick of its own struct Profile { const char *path; @@ -25,12 +30,49 @@ const Profile kProfiles[] = { {"/interaction_profiles/microsoft/motion_controller", true}, {"/interaction_profiles/htc/vive_controller", false}, {"/interaction_profiles/khr/simple_controller", false}, + {kTouchPlusProfile, true}, + {kTouchPlusExtensionProfile, true}, + {kFrameProfile, true}, }; -// The order the profiles are tried in. METALVR_INTERACTION_PROFILE names one to try first, for a game whose -// bindings for another controller are the better ones ("/interaction_profiles/htc/vive_controller"). -std::vector profileOrder() { +// the profiles that name the controllers of a kind as they are, best first +std::vector nativeProfiles(uint32_t controllers) { + switch (controllers) { + case MVR_CONTROLLERS_TOUCH: + // the Sense controllers have no OpenXR profile of their own: Touch, which has their layout (Monado binds them so) + case MVR_CONTROLLERS_PS_SENSE: + return {"/interaction_profiles/oculus/touch_controller"}; + case MVR_CONTROLLERS_TOUCH_PLUS: + return {kTouchPlusProfile, kTouchPlusExtensionProfile, "/interaction_profiles/oculus/touch_controller"}; + case MVR_CONTROLLERS_INDEX: + return {"/interaction_profiles/valve/index_controller"}; + case MVR_CONTROLLERS_VIVE_WAND: + return {"/interaction_profiles/htc/vive_controller"}; + case MVR_CONTROLLERS_STEAM_FRAME: + return {kFrameProfile}; + default: + return {}; + } +} + +bool hasThumbstick(uint32_t controllers) { return controllers != MVR_CONTROLLERS_VIVE_WAND; } +bool hasTrackpad(uint32_t controllers) { + return controllers == MVR_CONTROLLERS_INDEX || controllers == MVR_CONTROLLERS_VIVE_WAND; +} + +// The order the profiles are tried in: METALVR_INTERACTION_PROFILE names one to try first, for a game whose +// bindings for another controller are the better ones ("/interaction_profiles/htc/vive_controller"); then the +// profiles of the controllers in the user's hands; then the rest. +std::vector profileOrder(uint32_t controllers) { std::vector order; + const auto add = [&](const char *path) { + for (const Profile &have : order) + if (!std::strcmp(have.path, path)) + return; + for (const Profile &profile : kProfiles) + if (!std::strcmp(profile.path, path)) + order.push_back(profile); + }; const char *preferred = std::getenv("METALVR_INTERACTION_PROFILE"); if (preferred && *preferred) { bool known = false; @@ -42,9 +84,10 @@ std::vector profileOrder() { if (!known) logLine("METALVR_INTERACTION_PROFILE=%s is not a profile this runtime binds; ignored", preferred); } + for (const char *path : nativeProfiles(controllers)) + add(path); for (const Profile &profile : kProfiles) - if (order.empty() || std::strcmp(order.front().path, profile.path)) - order.push_back(profile); + add(profile.path); return order; } @@ -74,46 +117,87 @@ bool parseBinding(const std::string &path, int &hand, std::string &identifier, s return !identifier.empty(); } -// `has_thumbstick` is whether the profile being read names a thumbstick of its own +struct PadSources { + Source both, x, y, click, touch, force; +}; +constexpr PadSources kStick = {Source::Thumbstick, Source::ThumbstickX, Source::ThumbstickY, + Source::ThumbstickClick, Source::ThumbstickTouch, Source::None}; +constexpr PadSources kPad = {Source::Trackpad, Source::TrackpadX, Source::TrackpadY, + Source::TrackpadClick, Source::TrackpadTouch, Source::TrackpadForce}; + +Source padSource(const PadSources &pad, const std::string &component, XrActionType type) { + if (component == "x") + return pad.x; + if (component == "y") + return pad.y; + if (component == "click") + return pad.click; + if (component == "touch") + return pad.touch; + if (component == "force") + return pad.force; + return type == XR_ACTION_TYPE_VECTOR2F_INPUT ? pad.both + : type == XR_ACTION_TYPE_BOOLEAN_INPUT ? pad.click + : Source::None; +} + +// What a binding of a profile reads on the controllers in the user's hands. `has_thumbstick` is whether the +// profile names a thumbstick of its own; `controllers` is the physical kind (mvr_controller_kind). Source sourceFor(const std::string &identifier, const std::string &component, XrActionType type, - bool has_thumbstick) { + bool has_thumbstick, const char *profile, uint32_t controllers, int hand) { const bool wants_bool = type == XR_ACTION_TYPE_BOOLEAN_INPUT; - // On a controller with no stick the trackpad is where the thumb steers, and it is read as the stick. On one - // that has both, the trackpad is a second input this controller does not have: bound to the stick as well, one - // push would fire the game's stick and trackpad actions together. - if (identifier == "trackpad" && has_thumbstick) - return Source::None; + if (identifier == "trackpad") { + // a trackpad of its own (Index, Vive) is read as itself + if (hasTrackpad(controllers)) + return padSource(kPad, component, type); + // On a controller with no trackpad, the trackpad of a profile without a stick (Vive's) is where the thumb + // steers, and it is read as the stick. Of a profile that has both, it is a second input these controllers do + // not have: bound to the stick as well, one push would fire the game's stick and trackpad actions together. + return has_thumbstick ? Source::None : padSource(kStick, component, type); + } + if (identifier == "thumbstick") + // a game written for sticks, on Vive wands: the thumb steers on the trackpad + return padSource(hasThumbstick(controllers) ? kStick : kPad, component, type); if (identifier == "trigger" || identifier == "select") { if (component == "click") return Source::TriggerClick; - if (component == "touch") + if (component == "touch" || component == "proximity") return Source::TriggerTouch; if (component == "value") return Source::TriggerValue; + if (component == "force") + return Source::None; // Touch Plus: no controller reports it through the server return wants_bool ? Source::TriggerClick : Source::TriggerValue; } + if (identifier == "trigger_curl" || identifier == "trigger_slide") + return Source::None; if (identifier == "squeeze") { if (component == "click") return Source::SqueezeClick; - if (component == "value" || component == "force") + if (component == "force") + return controllers == MVR_CONTROLLERS_INDEX ? Source::SqueezeForce : Source::SqueezeValue; + if (component == "value") return Source::SqueezeValue; return wants_bool ? Source::SqueezeClick : Source::SqueezeValue; } - if (identifier == "thumbstick" || identifier == "trackpad") { - if (component == "x") - return Source::ThumbstickX; - if (component == "y") - return Source::ThumbstickY; - if (component == "click") - return Source::ThumbstickClick; - if (component == "touch") - return Source::ThumbstickTouch; - if (component == "force") - return Source::None; - return type == XR_ACTION_TYPE_VECTOR2F_INPUT ? Source::Thumbstick - : wants_bool ? Source::ThumbstickClick - : Source::None; - } + if (identifier == "bumper") + return component == "touch" ? Source::BumperTouch : Source::BumperClick; + if (identifier == "dpad_up") + return Source::DpadUp; + if (identifier == "dpad_down") + return Source::DpadDown; + if (identifier == "dpad_left") + return Source::DpadLeft; + if (identifier == "dpad_right") + return Source::DpadRight; + if (identifier == "view") + return Source::ViewClick; + // the Steam Frame's right controller has X and Y beside A and B + const bool frame_right = hand == 1 && !std::strcmp(profile, kFrameProfile); + if (frame_right && identifier == "x") + return component == "touch" ? Source::TertiaryTouch : Source::TertiaryClick; + if (frame_right && identifier == "y") + return component == "touch" ? Source::QuaternaryTouch : Source::QuaternaryClick; if (identifier == "a" || identifier == "x") return component == "touch" ? Source::PrimaryTouch : Source::PrimaryClick; if (identifier == "b" || identifier == "y") @@ -122,7 +206,7 @@ Source sourceFor(const std::string &identifier, const std::string &component, Xr return Source::MenuClick; if (identifier == "system") return component == "touch" ? Source::None : Source::SystemClick; - if (identifier == "thumbrest") + if (identifier == "thumbrest" || identifier == "thumb_resting_surfaces") return Source::ThumbrestTouch; if (identifier == "grip" || identifier == "palm_ext" || identifier == "grip_surface") return Source::GripPose; @@ -190,6 +274,61 @@ void readSource(const mvr_controller_state &c, Source source, float &x, float &y case Source::ThumbrestTouch: x = bit(c.touches, MVR_BUTTON_THUMBREST); break; + case Source::Trackpad: + x = c.trackpad_x; + y = c.trackpad_y; + break; + case Source::TrackpadX: + x = c.trackpad_x; + break; + case Source::TrackpadY: + x = c.trackpad_y; + break; + case Source::TrackpadClick: + x = bit(c.buttons, MVR_BUTTON_TRACKPAD); + break; + case Source::TrackpadTouch: + x = bit(c.touches, MVR_BUTTON_TRACKPAD); + break; + case Source::TrackpadForce: + x = c.trackpad_force; + break; + case Source::SqueezeForce: + x = c.squeeze_force; + break; + case Source::BumperClick: + x = bit(c.buttons, MVR_BUTTON_BUMPER); + break; + case Source::BumperTouch: + x = bit(c.touches | c.buttons, MVR_BUTTON_BUMPER); + break; + case Source::DpadUp: + x = bit(c.buttons, MVR_BUTTON_DPAD_UP); + break; + case Source::DpadDown: + x = bit(c.buttons, MVR_BUTTON_DPAD_DOWN); + break; + case Source::DpadLeft: + x = bit(c.buttons, MVR_BUTTON_DPAD_LEFT); + break; + case Source::DpadRight: + x = bit(c.buttons, MVR_BUTTON_DPAD_RIGHT); + break; + case Source::ViewClick: + x = bit(c.buttons, MVR_BUTTON_VIEW); + break; + case Source::TertiaryClick: + x = bit(c.buttons, MVR_BUTTON_TERTIARY); + break; + case Source::TertiaryTouch: + x = bit(c.touches, MVR_BUTTON_TERTIARY); + break; + case Source::QuaternaryClick: + x = bit(c.buttons, MVR_BUTTON_QUATERNARY); + break; + case Source::QuaternaryTouch: + x = bit(c.touches, MVR_BUTTON_QUATERNARY); + break; default: break; } @@ -262,7 +401,8 @@ void attachBindings(Session &session) { for (int hand = 0; hand < 2; ++hand) session.active_profile[hand] = XR_NULL_PATH; - for (const Profile &profile : profileOrder()) { + const uint32_t controllers = instance.system.controllers; + for (const Profile &profile : profileOrder(controllers)) { auto id = instance.path_ids.find(profile.path); if (id == instance.path_ids.end()) continue; @@ -281,8 +421,8 @@ void attachBindings(Session &session) { std::string identifier, component; if (!path || !parseBinding(*path, binding_hand, identifier, component) || binding_hand != hand) continue; - const Source source = - sourceFor(identifier, component, binding.action->action_type, profile.has_thumbstick); + const Source source = sourceFor(identifier, component, binding.action->action_type, profile.has_thumbstick, + profile.path, controllers, hand); if (source == Source::None) continue; binding.action->bindings.push_back({hand, source, binding.binding}); diff --git a/runtime/instance.cpp b/runtime/instance.cpp index 19cc849..1361037 100644 --- a/runtime/instance.cpp +++ b/runtime/instance.cpp @@ -1,5 +1,6 @@ // Instance, system, paths, events and the object registry. #include "api.h" +#include "metalvr_headsets.h" #include #include @@ -129,6 +130,12 @@ const ExtensionInfo kExtensions[] = { {XR_KHR_VISIBILITY_MASK_EXTENSION_NAME, XR_KHR_visibility_mask_SPEC_VERSION}, {XR_EXT_LOCAL_FLOOR_EXTENSION_NAME, XR_EXT_local_floor_SPEC_VERSION}, {XR_FB_DISPLAY_REFRESH_RATE_EXTENSION_NAME, XR_FB_display_refresh_rate_SPEC_VERSION}, + // interaction profiles: suggested bindings for them are read on whatever controllers the user holds + {XR_META_TOUCH_CONTROLLER_PLUS_EXTENSION_NAME, XR_META_touch_controller_plus_SPEC_VERSION}, + // the hands' joints, when the headset tracks bare hands (hands.cpp) + {XR_EXT_HAND_TRACKING_EXTENSION_NAME, XR_EXT_hand_tracking_SPEC_VERSION}, + // Valve's, for the Steam Frame's controllers; not in the Khronos registry (docs/HEADSETS.md) + {"XR_VALVE_frame_controller_interaction", 1}, }; bool extensionSupported(const char *name) { @@ -380,7 +387,7 @@ MVR_API xrGetSystemProperties(XrInstance handle, XrSystemId system_id, XrSystemP if (system_id != kSystemId) return XR_ERROR_SYSTEM_INVALID; properties->systemId = kSystemId; - properties->vendorId = 0; + properties->vendorId = mvr_describe_headset(instance->system.headset)->vendor_id; copyString(properties->systemName, std::string(instance->system.system_name, strnlen(instance->system.system_name, sizeof(instance->system.system_name)))); @@ -389,6 +396,10 @@ MVR_API xrGetSystemProperties(XrInstance handle, XrSystemId system_id, XrSystemP properties->graphicsProperties.maxLayerCount = MVR_MAX_LAYERS; properties->trackingProperties.orientationTracking = XR_TRUE; properties->trackingProperties.positionTracking = XR_TRUE; + for (auto *next = static_cast(properties->next); next; next = next->next) + if (next->type == XR_TYPE_SYSTEM_HAND_TRACKING_PROPERTIES_EXT) + reinterpret_cast(next)->supportsHandTracking = + (instance->system.flags & MVR_SYSTEM_HAND_TRACKING) ? XR_TRUE : XR_FALSE; return XR_SUCCESS; } diff --git a/runtime/runtime.h b/runtime/runtime.h index 9dde14a..f73d84e 100644 --- a/runtime/runtime.h +++ b/runtime/runtime.h @@ -66,7 +66,7 @@ struct Action; // ---- handles ------------------------------------------------------------------------------------------------- -enum class ObjectType : uint32_t { Instance, Session, Swapchain, Space, ActionSet, Action }; +enum class ObjectType : uint32_t { Instance, Session, Swapchain, Space, ActionSet, Action, HandTracker }; struct Object { explicit Object(ObjectType t); @@ -142,6 +142,24 @@ enum class Source : uint8_t { MenuClick, SystemClick, ThumbrestTouch, + Trackpad, // both axes + TrackpadX, + TrackpadY, + TrackpadClick, + TrackpadTouch, + TrackpadForce, + SqueezeForce, + BumperClick, + BumperTouch, + DpadUp, + DpadDown, + DpadLeft, + DpadRight, + ViewClick, + TertiaryClick, // Steam Frame: X on the right hand + TertiaryTouch, + QuaternaryClick, // Steam Frame: Y on the right hand + QuaternaryTouch, GripPose, AimPose, Haptic, @@ -203,6 +221,14 @@ struct Space : Object { mvr_pose offset = poseIdentity(); }; +// an XR_EXT_hand_tracking hand tracker (hands.cpp) +struct HandTracker : Object { + static constexpr ObjectType kType = ObjectType::HandTracker; + HandTracker() : Object(kType) {} + Session *session = nullptr; + int hand = 0; // 0 left, 1 right +}; + // where a space is in the server's tracking space at some time struct Located { mvr_pose pose = poseIdentity(); @@ -251,6 +277,7 @@ struct Session : Object { std::vector> swapchains; std::vector> spaces; + std::vector> hand_trackers; uint32_t next_swapchain_id = 1; // frame loop diff --git a/runtime/session.cpp b/runtime/session.cpp index ad11a16..0e7e771 100644 --- a/runtime/session.cpp +++ b/runtime/session.cpp @@ -448,7 +448,14 @@ MVR_API xrGetReferenceSpaceBoundsRect(XrSession handle, XrReferenceSpaceType typ return XR_ERROR_VALIDATION_FAILURE; if (!referenceSpaceSupported(*instance, type)) return XR_ERROR_REFERENCE_SPACE_UNSUPPORTED; - // the headset's play area is not carried over the stream + // the play area the headset's own boundary encloses, when it has one and has said; it is centered on the + // tracking space's origin, which is where STAGE is + if (type == XR_REFERENCE_SPACE_TYPE_STAGE && instance->system.play_area_width > 0 && + instance->system.play_area_depth > 0) { + bounds->width = instance->system.play_area_width; + bounds->height = instance->system.play_area_depth; + return XR_SUCCESS; + } bounds->width = 0; bounds->height = 0; return XR_SPACE_BOUNDS_UNAVAILABLE; @@ -983,11 +990,25 @@ MVR_API xrGetVisibilityMaskKHR(XrSession handle, XrViewConfigurationType type, u } // ---- XR_FB_display_refresh_rate ------------------------------------------------------------------------------ -// The headset's rate is whatever the stream was negotiated at; it is reported, and only that rate can be asked for. +// The rates the headset can run at are reported. It runs at the one it was started at (for a streamed headset, +// the one the stream was negotiated at): a request for another rate it has is accepted and not acted on, which the +// extension allows, and one for a rate it does not have is refused. + +namespace { +std::vector refreshRates(const mvr_system_info &system) { + std::vector rates; + for (uint32_t i = 0; i < system.refresh_rate_count && i < MVR_MAX_REFRESH_RATES; ++i) + if (system.refresh_rates[i] > 0) + rates.push_back(system.refresh_rates[i]); + if (rates.empty()) + rates.push_back(system.refresh_hz); + return rates; +} +} // namespace MVR_API xrEnumerateDisplayRefreshRatesFB(XrSession handle, uint32_t capacity, uint32_t *count, float *rates) { MVR_SESSION(handle); - return fillArray(capacity, count, rates, {instance->system.refresh_hz}); + return fillArray(capacity, count, rates, refreshRates(instance->system)); } MVR_API xrGetDisplayRefreshRateFB(XrSession handle, float *rate) { @@ -1000,9 +1021,16 @@ MVR_API xrGetDisplayRefreshRateFB(XrSession handle, float *rate) { MVR_API xrRequestDisplayRefreshRateFB(XrSession handle, float rate) { MVR_SESSION(handle); - if (rate != 0.0f && std::fabs(rate - instance->system.refresh_hz) > 0.5f) - return XR_ERROR_DISPLAY_REFRESH_RATE_UNSUPPORTED_FB; - return XR_SUCCESS; + if (rate == 0.0f) + return XR_SUCCESS; + for (float have : refreshRates(instance->system)) + if (std::fabs(rate - have) <= 0.5f) { + if (std::fabs(rate - instance->system.refresh_hz) > 0.5f) + logLine("xrRequestDisplayRefreshRateFB: %.0f Hz asked for; the headset stays at %.0f Hz", rate, + instance->system.refresh_hz); + return XR_SUCCESS; + } + return XR_ERROR_DISPLAY_REFRESH_RATE_UNSUPPORTED_FB; } } // namespace api diff --git a/scripts/build-alvr-client.sh b/scripts/build-alvr-client.sh new file mode 100755 index 0000000..9dddf69 --- /dev/null +++ b/scripts/build-alvr-client.sh @@ -0,0 +1,75 @@ +#!/bin/bash +# Builds ALVR's headset client with MetalVR's patches (patches/alvr, among them 0003, which lets it run on the +# Steam Frame and name itself there) as an APK to install on a standalone headset: build/alvr-client/ +# alvr_client_android.apk. Everything it needs is fetched into build/android, nothing is installed elsewhere: +# the Android NDK r27c, the SDK's command-line tools with platform 32, build-tools 34 and platform-tools, +# a JDK 17 (Eclipse Temurin), cargo-apk (ALVR's fork), and Khronos's OpenXR loader for Android 1.0.34. +# Google's SDK license is shown by sdkmanager the first time, for you to accept or decline; nothing is accepted +# for you. Only the generic OpenXR loader is bundled, which is the one the Steam Frame (and any headset ALVR has no +# vendor loader for) uses. +# +# scripts/build-alvr-client.sh then: adb install -r build/alvr-client/alvr_client_android.apk +set -euo pipefail +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +A="$ROOT/build/android" +NDK_VERSION=r27c +TOOLS_ZIP=commandlinetools-mac-11076708_latest.zip +CARGO_APK_COMMIT=0fd3126dad5aa1c5f0f26cdae3410f2e5af62c60 +LOADER_URL=https://github.com/KhronosGroup/OpenXR-SDK-Source/releases/download/release-1.0.34/openxr_loader_for_android-1.0.34.aar +say() { printf 'build-alvr-client: %s\n' "$*" >&2; } + +# the server's build applies the patches and checks ALVR's commit +bash "$ROOT/scripts/build-alvr.sh" >/dev/null +mkdir -p "$A" + +if [ ! -d "$A/android-ndk-$NDK_VERSION" ]; then + say "fetching the Android NDK $NDK_VERSION" + curl -fsSL -o "$A/ndk.zip" "https://dl.google.com/android/repository/android-ndk-$NDK_VERSION-darwin.zip" + unzip -q "$A/ndk.zip" -d "$A" && rm "$A/ndk.zip" +fi +JAVA_HOME="$(ls -d "$A"/jdk-dl/jdk-17*/Contents/Home 2>/dev/null | head -1 || true)" +if [ -z "$JAVA_HOME" ]; then + say "fetching a JDK 17" + mkdir -p "$A/jdk-dl" + curl -fsSL -o "$A/jdk-dl/jdk.tar.gz" \ + "https://api.adoptium.net/v3/binary/latest/17/ga/mac/aarch64/jdk/hotspot/normal/eclipse" + tar -xzf "$A/jdk-dl/jdk.tar.gz" -C "$A/jdk-dl" && rm "$A/jdk-dl/jdk.tar.gz" + JAVA_HOME="$(ls -d "$A"/jdk-dl/jdk-17*/Contents/Home | head -1)" +fi +export JAVA_HOME +SDKMANAGER="$A/sdk/cmdline-tools/latest/bin/sdkmanager" +if [ ! -x "$SDKMANAGER" ]; then + say "fetching the Android SDK's command-line tools" + mkdir -p "$A/sdk/cmdline-tools" + curl -fsSL -o "$A/tools.zip" "https://dl.google.com/android/repository/$TOOLS_ZIP" + rm -rf "$A/sdk/cmdline-tools/latest" "$A/sdk/cmdline-tools-dl" + unzip -q "$A/tools.zip" -d "$A/sdk/cmdline-tools-dl" && rm "$A/tools.zip" + mv "$A/sdk/cmdline-tools-dl/cmdline-tools" "$A/sdk/cmdline-tools/latest" && rmdir "$A/sdk/cmdline-tools-dl" +fi +if [ ! -d "$A/sdk/platforms/android-32" ] || [ ! -d "$A/sdk/build-tools/34.0.0" ]; then + say "Google's Android SDK license follows; installing needs it accepted" + "$SDKMANAGER" --sdk_root="$A/sdk" --licenses + "$SDKMANAGER" --sdk_root="$A/sdk" "platforms;android-32" "build-tools;34.0.0" "platform-tools" +fi +if [ ! -x "$A/tools/bin/cargo-apk" ]; then + say "building cargo-apk" + cargo install --quiet --git https://github.com/zarik5/cargo-apk --rev "$CARGO_APK_COMMIT" cargo-apk --root "$A/tools" +fi +rustup target add aarch64-linux-android >/dev/null 2>&1 +loader="$ROOT/external/ALVR/deps/android_openxr/arm64-v8a/libopenxr_loader.so" +if [ ! -f "$loader" ]; then + say "fetching Khronos's OpenXR loader for Android" + rm -rf "$A/loader-dl" && mkdir -p "$A/loader-dl" "$(dirname "$loader")" + curl -fsSL -o "$A/loader-dl/loader.aar" "$LOADER_URL" + unzip -q "$A/loader-dl/loader.aar" -d "$A/loader-dl/x" + cp "$A/loader-dl/x/prefab/modules/openxr_loader/libs/android.arm64-v8a/libopenxr_loader.so" "$loader" +fi + +export ANDROID_HOME="$A/sdk" ANDROID_SDK_ROOT="$A/sdk" +export ANDROID_NDK_ROOT="$A/android-ndk-$NDK_VERSION" ANDROID_NDK_HOME="$A/android-ndk-$NDK_VERSION" +export PATH="$A/tools/bin:$PATH" +say "building the client (release)" +(cd "$ROOT/external/ALVR" && cargo xtask build-client --release) +mkdir -p "$ROOT/build/alvr-client" +cp "$ROOT/external/ALVR/build/alvr_client_android/alvr_client_android.apk" "$ROOT/build/alvr-client/" +say "built $ROOT/build/alvr-client/alvr_client_android.apk" diff --git a/scripts/build-basalt.sh b/scripts/build-basalt.sh new file mode 100755 index 0000000..c6f7adb --- /dev/null +++ b/scripts/build-basalt.sh @@ -0,0 +1,53 @@ +#!/bin/sh +# Builds the Rift S's camera tracking (docs/HEADSETS.md, phase I) into build/basalt/prefix, without installing anything +# outside the build folder: +# Basalt (BSD-3-Clause), Monado's fork, as libbasalt.dylib with Monado's VIT interface: the visual-inertial tracker +# metalvrd loads when it starts a Rift S, and gives the headset's cameras and IMU +# oneTBB (Apache-2.0), OpenCV 4.10 core/imgproc/calib3d/features2d/flann (Apache-2.0) and fmt (MIT), what Basalt needs +# scripts/fetch-deps.sh fetches all of it into external/. metalvrd finds the libraries in build/basalt/prefix/lib (or +# next to itself in a release) when it runs; it is not linked against them. +# +# scripts/build-basalt.sh +set -eu +root=$(cd "$(dirname "$0")/.." && pwd) +out=$root/build/basalt +prefix=$out/prefix +jobs=$(sysctl -n hw.ncpu) +for dir in basalt oneTBB opencv fmt; do + [ -d "$root/external/$dir" ] || { echo "external/$dir is missing: run scripts/fetch-deps.sh" >&2; exit 1; } +done +mkdir -p "$out" + +common="-G Ninja -DCMAKE_POLICY_VERSION_MINIMUM=3.5 -DCMAKE_BUILD_TYPE=Release -DCMAKE_OSX_ARCHITECTURES=arm64 + -DCMAKE_OSX_DEPLOYMENT_TARGET=15.0 -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_PREFIX_PATH=$prefix + -DCMAKE_INSTALL_NAME_DIR=@rpath" + +step() { # name, then the source directory and its cmake options + name=$1; shift + src=$1; shift + if [ -f "$out/$name.done" ]; then echo "$name: built"; return; fi + echo "$name: building" + # shellcheck disable=SC2086 + { cmake -S "$src" -B "$out/$name-build" $common "$@" && cmake --build "$out/$name-build" -j "$jobs" && + cmake --install "$out/$name-build"; } >"$out/$name.log" 2>&1 || { tail -40 "$out/$name.log"; exit 1; } + touch "$out/$name.done" +} + +step oneTBB "$root/external/oneTBB" -DTBB_TEST=OFF -DTBB_EXAMPLES=OFF -DTBB_STRICT=OFF -DBUILD_SHARED_LIBS=ON +step fmt "$root/external/fmt" -DFMT_TEST=OFF -DFMT_DOC=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON +step opencv "$root/external/opencv" -DBUILD_LIST=core,imgproc,calib3d,features2d,flann,highgui \ + -DBUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DWITH_TBB=OFF -DWITH_OPENCL=OFF -DWITH_FFMPEG=OFF \ + -DWITH_JPEG=OFF -DWITH_PNG=OFF -DWITH_TIFF=OFF -DWITH_WEBP=OFF -DWITH_OPENJPEG=OFF -DWITH_JASPER=OFF \ + -DWITH_OPENEXR=OFF -DWITH_IMGCODEC_HDR=OFF -DWITH_IMGCODEC_SUNRASTER=OFF -DWITH_IMGCODEC_PXM=OFF \ + -DWITH_IMGCODEC_PFM=OFF -DWITH_AVFOUNDATION=OFF -DWITH_ITT=OFF -DWITH_IPP=OFF -DWITH_LAPACK=OFF -DWITH_EIGEN=OFF \ + -DWITH_PROTOBUF=OFF -DWITH_QT=OFF -DWITH_GTK=OFF -DWITH_COCOA=OFF -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF \ + -DBUILD_EXAMPLES=OFF -DBUILD_opencv_apps=OFF -DBUILD_JAVA=OFF -DBUILD_opencv_python3=OFF -DBUILD_ZLIB=ON \ + -DWITH_ADE=OFF -DBUILD_opencv_gapi=OFF +step basalt "$root/external/basalt" -DBASALT_BUILD_SHARED_LIBRARY_ONLY=ON -DBASALT_BUILD_VISUALIZATION=OFF \ + -DBASALT_ENABLE_ROSBAG2=OFF -DBASALT_INSTANTIATIONS_DOUBLE=OFF -DBUILD_TESTING=OFF \ + -DSKIP_PERFORMANCE_COMPARISON=ON -DBUILD_SANDBOX=OFF -DBUILD_DOC=OFF \ + -DEIGEN_ROOT="$root/external/basalt/thirdparty/basalt-headers/thirdparty/eigen" \ + -DOpenCV_DIR="$prefix/lib/cmake/opencv4" -DCMAKE_INSTALL_RPATH=@loader_path + +echo "basalt $(git -C "$root/external/basalt" rev-parse --short HEAD) built into $prefix" +ls "$prefix/lib" | grep -E '^lib(basalt|tbb).*\.dylib$' | sed 's/^/ /' diff --git a/scripts/build-libsurvive.sh b/scripts/build-libsurvive.sh new file mode 100755 index 0000000..e643d94 --- /dev/null +++ b/scripts/build-libsurvive.sh @@ -0,0 +1,46 @@ +#!/bin/sh +# Builds libsurvive (MIT), the Lighthouse tracker metalvrd's Lighthouse driver takes positions from (docs/HEADSETS.md), +# statically into build/libsurvive, with hidapi (BSD) as its USB layer: macOS's own HID driver holds the headsets' +# HID interfaces, which hidapi reaches through IOHIDManager and libusb could not claim. scripts/fetch-deps.sh fetches +# both into external/. Configure build/native again afterwards and metalvrd links it. +# +# scripts/build-libsurvive.sh +set -eu +root=$(cd "$(dirname "$0")/.." && pwd) +out=$root/build/libsurvive +for dir in libsurvive hidapi; do + [ -d "$root/external/$dir" ] || { echo "external/$dir is missing: run scripts/fetch-deps.sh" >&2; exit 1; } +done +# the changes MetalVR needs (patches/libsurvive), applied once +for patch in "$root"/patches/libsurvive/*.patch; do + [ -f "$patch" ] || continue + if git -C "$root/external/libsurvive" apply --reverse --check "$patch" 2>/dev/null; then + : + elif git -C "$root/external/libsurvive" apply --check "$patch" 2>/dev/null; then + echo "applying $(basename "$patch")" + git -C "$root/external/libsurvive" apply "$patch" + else + echo "patch does not apply to external/libsurvive: $patch" >&2 + exit 1 + fi +done + +common="-G Ninja -DCMAKE_POLICY_VERSION_MINIMUM=3.5 -DCMAKE_BUILD_TYPE=Release -DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_OSX_DEPLOYMENT_TARGET=15.0" + +# shellcheck disable=SC2086 +cmake -S "$root/external/hidapi" -B "$out/hidapi-build" $common -DBUILD_SHARED_LIBS=OFF \ + -DCMAKE_INSTALL_PREFIX="$out/prefix" >"$out.hidapi.log" 2>&1 || { cat "$out.hidapi.log"; exit 1; } +cmake --build "$out/hidapi-build" >>"$out.hidapi.log" 2>&1 && cmake --install "$out/hidapi-build" >>"$out.hidapi.log" 2>&1 + +# the core and the plugins Lighthouse tracking needs, each a static library that registers itself when linked whole +# shellcheck disable=SC2086 +PKG_CONFIG_PATH="$out/prefix/lib/pkgconfig" cmake -S "$root/external/libsurvive" -B "$out/build" $common \ + -DBUILD_STATIC=ON -DDO_CORE_BUILD=ON -DUSE_HIDAPI=ON -DBUILD_APPLICATIONS=OFF -DENABLE_TESTS=OFF \ + -DBUILD_GATT_SUPPORT=OFF -DUSE_ASAN=OFF \ + -DENABLE_driver_vive=ON -DENABLE_driver_simulator=ON -DENABLE_disambiguator_statebased=ON -DENABLE_poser_kalman_only=ON \ + -DENABLE_poser_mpfit=ON -DENABLE_driver_global_scene_solver=ON \ + >"$out.log" 2>&1 || { cat "$out.log"; exit 1; } +cmake --build "$out/build" --target survive driver_vive driver_simulator disambiguator_statebased poser_kalman_only poser_mpfit \ + driver_global_scene_solver >>"$out.log" 2>&1 || { tail -40 "$out.log"; exit 1; } +echo "libsurvive $(git -C "$root/external/libsurvive" rev-parse --short HEAD) built into $out/build" +find "$out/build" -name "*.a" | sed "s|$out/build/| |" diff --git a/scripts/fetch-deps.sh b/scripts/fetch-deps.sh index 0b4952d..31e27e5 100755 --- a/scripts/fetch-deps.sh +++ b/scripts/fetch-deps.sh @@ -3,6 +3,10 @@ # OpenXR-SDK Khronos headers and loader (Apache-2.0), needed by everything # OpenHMD-rift OpenHMD with the Rift's positional tracker (BSL-1.0), which metalvrd's Rift driver compiles in for # tracking with the Rift's sensors; without it the driver tracks rotation only +# libsurvive, hidapi the Lighthouse tracker (MIT) and its USB layer (BSD), which scripts/build-libsurvive.sh builds +# for tracking the Lighthouse headsets with their base stations +# basalt and what it needs (oneTBB, OpenCV, fmt) Monado's Basalt (BSD-3-Clause), the visual-inertial tracker +# scripts/build-basalt.sh builds for the Rift S's camera tracking # ALVR (scripts/build-alvr.sh) and OpenComposite (scripts/build-opencomposite.sh) fetch themselves. set -eu root=$(cd "$(dirname "$0")/.." && pwd) @@ -33,3 +37,29 @@ if [ "$found" != "$openhmd_commit" ]; then exit 1 fi echo "OpenHMD-rift: $openhmd_commit" + +# libsurvive and hidapi: commits, as above +fetch_commit() { # directory, repository, commit + if [ ! -d "$root/external/$1/.git" ]; then + git init -q "$root/external/$1" + git -C "$root/external/$1" fetch -q --depth 1 "$2" "$3" + git -C "$root/external/$1" checkout -q FETCH_HEAD + fi + found=$(git -C "$root/external/$1" rev-parse HEAD) + if [ "$found" != "$3" ]; then + echo "external/$1 is at $found, not the pinned $3" >&2 + exit 1 + fi + echo "$1: $3" +} +fetch_commit libsurvive https://github.com/cntools/libsurvive.git 44b1e2f0cb95e5697bc0b86f58fd8520a813b713 +fetch_commit hidapi https://github.com/libusb/hidapi.git d3013f0af3f4029d82872c1a9487ea461a56dee4 # hidapi-0.14.0 +# Basalt, Monado's fork with its VIT interface, and only the submodules its library uses +fetch_commit basalt https://gitlab.freedesktop.org/mateosss/basalt.git df6e970c8da7636eb401a09e3317fbeaaf829b9a +git -C "$root/external/basalt" submodule update -q --init --depth 1 thirdparty/basalt-headers thirdparty/opengv \ + thirdparty/magic_enum thirdparty/CLI11 +git -C "$root/external/basalt/thirdparty/basalt-headers" submodule update -q --init --depth 1 thirdparty/Sophus \ + thirdparty/cereal thirdparty/eigen +fetch_commit oneTBB https://github.com/uxlfoundation/oneTBB.git 0c0ff192a2304e114bc9e6557582dfba101360ff # v2022.0.0 +fetch_commit opencv https://github.com/opencv/opencv.git 71d3237a093b60a27601c20e9ee6c3e52154e8b1 # 4.10.0 +fetch_commit fmt https://github.com/fmtlib/fmt.git 0c9fce2ffefecfdce794e1859584e25877b7b592 # 11.0.2 diff --git a/scripts/metalsharp/home-logo.png b/scripts/metalsharp/home-logo.png index b665d25..a296f0a 100644 Binary files a/scripts/metalsharp/home-logo.png and b/scripts/metalsharp/home-logo.png differ diff --git a/scripts/metalsharp/metalvr b/scripts/metalsharp/metalvr index f769e3b..52e40cc 100755 --- a/scripts/metalsharp/metalvr +++ b/scripts/metalsharp/metalvr @@ -4,7 +4,12 @@ # # metalvr start [--trust NAME] [metalvrd options] run the server for a headset (ALVR). NAME is what the # headset's ALVR app shows; it is remembered once given -# metalvr rift [metalvrd options] run the server for an Oculus Rift CV1 on this Mac's HDMI and USB +# metalvr rift [metalvrd options] run the server for an Oculus Rift CV1 or Rift S on this Mac's +# video output and USB +# metalvr lighthouse [metalvrd options] the same for an HTC Vive, Vive Pro, Vive Pro 2 or Valve Index +# metalvr psvr2 [metalvrd options] the same for a PlayStation VR2, through Sony's PC adapter +# metalvr display rift|lighthouse|all [--undo] let macOS show those headsets' displays (asks for your +# password once), or hide them again # metalvr sim [metalvrd options] run the server with the simulated headset (no headset needed) # metalvr prefix [PREFIX] [--off] make MetalVR the prefix's OpenXR runtime (default: the # Steam prefix every MetalSharp Steam game shares) @@ -41,9 +46,61 @@ start) mkdir -p $CONF exec $H/bin/metalvrd --config-dir $CONF "$@" ;; -rift) +rift|lighthouse|psvr2) + D=$1 shift - exec $H/bin/metalvrd --driver rift "$@" + exec $H/bin/metalvrd --driver $D "$@" + ;; +display) + # macOS marks each of these headsets' displays a head-mounted display and keeps it off the desktop. An override in + # /Library takes precedence: Apple's own is copied (it may carry EDID patches the display needs: the Vive Pro 2's + # modes) with is-hmd set false, or one is written with the display's IDs. docs/RIFT.md, docs/HEADSETS.md. + shift + F=${1:-all}; U= + [[ $F == --undo ]] && { F=all; U=1; } + [[ ${2:-} == --undo ]] && U=1 + case $F in + rift) IDS=(3ed2:4 3ed2:12) ;; # Rift CV1, Rift S + lighthouse) IDS=(22d2:aa01 22d2:aa02 22d2:aa03 22d2:aa04 5996:91a8) ;; # Vive, Vive Pro (aa02 or aa03), Pro 2, Index + all) IDS=(3ed2:4 3ed2:12 22d2:aa01 22d2:aa02 22d2:aa03 22d2:aa04 5996:91a8) ;; + *) echo >&2 "!! usage: metalvr display rift|lighthouse|all [--undo]"; exit 1 ;; + esac + SYS=/System/Library/Displays/Contents/Resources/Overrides + LIB=/Library/Displays/Contents/Resources/Overrides + lifted() { [[ -f $1 ]] && /usr/libexec/PlistBuddy -c 'Print :is-hmd' $1 2>/dev/null | grep -q false; } + SCRIPT=$(mktemp) + trap 'rm -f $SCRIPT' EXIT + print 'set -e' >$SCRIPT + todo=0 + for id in $IDS; do + v=${id%%:*} p=${id##*:} + f=$LIB/DisplayVendorID-$v/DisplayProductID-$p + if [[ -n $U ]]; then + [[ -f $f ]] || continue + print -r -- "rm -f '$f'" >>$SCRIPT + else + lifted $f && continue + print -r -- "mkdir -p '${f:h}'; rm -f '$f'" >>$SCRIPT + if [[ -f $SYS/DisplayVendorID-$v/DisplayProductID-$p ]]; then + print -r -- "cp '$SYS/DisplayVendorID-$v/DisplayProductID-$p' '$f'" >>$SCRIPT + print -r -- "/usr/libexec/PlistBuddy -c 'Set :is-hmd false' '$f' 2>/dev/null || /usr/libexec/PlistBuddy -c 'Add :is-hmd bool false' '$f'" >>$SCRIPT + print -r -- "/usr/libexec/PlistBuddy -c 'Print :DisplayVendorID' '$f' >/dev/null 2>&1 || /usr/libexec/PlistBuddy -c 'Add :DisplayVendorID integer $((16#$v))' -c 'Add :DisplayProductID integer $((16#$p))' '$f' >/dev/null" >>$SCRIPT + else + print -r -- "/usr/libexec/PlistBuddy -c 'Add :DisplayVendorID integer $((16#$v))' -c 'Add :DisplayProductID integer $((16#$p))' -c 'Add :is-hmd bool false' '$f' >/dev/null" >>$SCRIPT + fi + fi + todo=$((todo + 1)) + done + if (( todo == 0 )); then + [[ -n $U ]] && print -r -- "nothing to undo: macOS's own overrides apply" || print -r -- "macOS already shows these headsets' displays" + exit 0 + fi + # METALVR_DISPLAY_DRY_RUN=1: what would run, without running it (for checking) + [[ -n ${METALVR_DISPLAY_DRY_RUN:-} ]] && { cat $SCRIPT; exit 0; } + print -r -- "macOS needs an administrator's password to write $todo override(s) in $LIB" + sudo /bin/zsh $SCRIPT + if [[ -n $U ]]; then print -r -- "removed: macOS hides those displays again (after they are plugged in again)" + else print -r -- "done: unplug the headset's video cable and plug it in again (or restart) for macOS to see it"; fi ;; sim) shift @@ -217,6 +274,6 @@ status) if [[ -s $CONF/games ]]; then print -r -- "games with OpenComposite in place:"; sed 's/^/ /' $CONF/games; fi ;; *) - sed -n '2,17p' $0 | sed 's/^# \{0,1\}//' + sed -n '2,21p' $0 | sed 's/^# \{0,1\}//' ;; esac diff --git a/scripts/metalsharp/metalvr-rift b/scripts/metalsharp/metalvr-rift index 2f7cbef..e0f4e78 100755 --- a/scripts/metalsharp/metalvr-rift +++ b/scripts/metalsharp/metalvr-rift @@ -1,11 +1,14 @@ #!/bin/zsh -# metalvr-rift: an Oculus Rift CV1 on this Mac, from a MetalVR release to a game in the headset (docs/RIFT_CV1.md). +# metalvr-rift: a headset on this Mac's own display and USB, from a MetalVR release to a game in the headset: an +# Oculus Rift CV1 or Rift S (docs/headsets/); run as metalvr-lighthouse, an HTC Vive, Vive Pro, Vive Pro 2 or +# Valve Index; run as metalvr-psvr2, a PlayStation VR2 through Sony's PC adapter (docs/HEADSETS.md). # # metalvr-rift install copy this release into ~/.metalsharp/metalvr-runtime and make MetalVR the Steam # prefix's OpenXR runtime # metalvr-rift display [--undo] let macOS show the headset's display (asks for your password; once per Mac) # metalvr-rift setup register MetalVR's SteamVR driver with the Steam prefix's SteamVR -# metalvr-rift start start the Rift server and wait for the headset +# metalvr-rift start [options] start the server and wait for the headset (options go to metalvrd, e.g. +# --refresh 120 for an Index, --pro2-mode 2 for a Vive Pro 2) # metalvr-rift stop stop the server (the headset's display turns off) # metalvr-rift steamvr start the server if needed, then SteamVR on the headset # metalvr-rift game APPID start the server if needed, then a Steam game's VR entry on the headset @@ -14,9 +17,16 @@ set -e H=${0:A:h:h} # this release, unpacked or installed MS=$HOME/.metalsharp T=$MS/metalvr-runtime # where MetalSharp looks for MetalVR -LOG=$MS/logs/rift-server.log -OVR=/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-3ed2/DisplayProductID-4 PORT=${METALVR_PORT:-37913} +if [[ ${0:t} == metalvr-lighthouse ]]; then + FAMILY=lighthouse SERVER="Lighthouse server" VIDEO="DisplayPort (or a Vive's HDMI)" +elif [[ ${0:t} == metalvr-psvr2 ]]; then + FAMILY=psvr2 SERVER="PlayStation VR2 server" VIDEO="PC adapter's DisplayPort and USB (and its power)" +else + FAMILY=rift SERVER="Rift server" VIDEO="HDMI (CV1) or DisplayPort (Rift S)" +fi +LOG=$MS/logs/$FAMILY-server.log +LIB=/Library/Displays/Contents/Resources/Overrides die() { print -r -- "!! $*" >&2; exit 1; } need_metalsharp() { @@ -24,32 +34,80 @@ need_metalsharp() { die "no MetalSharp Steam prefix in $MS/prefix-steam: install MetalSharp and its Windows Steam first" } need_installed() { [[ -x $T/bin/metalvrd ]] || die "MetalVR is not installed: run '${0:t} install' from the release"; } -rift_plugged() { ioreg -p IOUSB -l -w0 2>/dev/null | grep -q '"USB Product Name" = "Rift"$'; } + +# whether a USB device of this vendor and product (decimal) is plugged in +usb_has() { + ioreg -p IOUSB -l -w0 2>/dev/null | awk -v V=$1 -v P=$2 ' + /\+-o / { if (v == V && p == P) f = 1; v = ""; p = "" } + /"idVendor" = / { v = $NF } + /"idProduct" = / { p = $NF } + END { if (v == V && p == P) f = 1; exit !f }' +} +# the headset on USB, by its IDs (docs/HEADSETS.md); empty if none of this family is +headset_plugged() { + if [[ $FAMILY == psvr2 ]]; then + usb_has 1356 3294 && { print "PlayStation VR2"; return; } # 054c:0cde + elif [[ $FAMILY == rift ]]; then + usb_has 10291 81 && { print "Rift S"; return; } # 2833:0051 + usb_has 10291 49 && { print "Rift CV1"; return; } # 2833:0031 + else + usb_has 3028 834 && { print "Vive Pro 2"; return; } # 0bb4:0342 + { usb_has 3028 777 || usb_has 3028 782; } && { print "Vive Pro"; return; } # 0bb4:0309, 030e + usb_has 3028 11399 && { print "Vive"; return; } # 0bb4:2c87 + usb_has 10462 8960 && { print "Valve Index"; return; } # 28de:2300 without an HTC mainboard + fi + return 0 +} sensors() { ioreg -p IOUSB -l -w0 2>/dev/null | grep -c '"USB Product Name" = "Rift Sensor"$' || true; } -override_on() { [[ -f $OVR ]] && /usr/libexec/PlistBuddy -c 'Print :is-hmd' $OVR 2>/dev/null | grep -q false; } +# the display overrides this family's headsets need, one a line: vendor:product:name +overrides() { + # macOS does not hide the PlayStation VR2's display (it ships no override for it): none + if [[ $FAMILY == psvr2 ]]; then + return 0 + elif [[ $FAMILY == rift ]]; then + print -l "3ed2:4:Rift CV1" "3ed2:12:Rift S" + else + print -l "22d2:aa01:Vive" "22d2:aa02:Vive Pro" "22d2:aa03:Vive Pro" "22d2:aa04:Vive Pro 2" "5996:91a8:Index" + fi +} +lifted() { + local f=$LIB/DisplayVendorID-$1/DisplayProductID-$2 + [[ -f $f ]] && /usr/libexec/PlistBuddy -c 'Print :is-hmd' $f 2>/dev/null | grep -q false +} +# whether macOS shows any of this family's displays +override_on() { + [[ $FAMILY == psvr2 ]] && return 0 + local o rest + overrides | while IFS= read -r o; do + rest=${o#*:} + lifted ${o%%:*} ${rest%%:*} && return 0 + done + return 1 +} server_up() { pgrep -x metalvrd >/dev/null && nc -z 127.0.0.1 $PORT 2>/dev/null; } start_server() { need_installed - if server_up; then print -r -- "the Rift server is running"; return; fi + if server_up; then print -r -- "the $SERVER is running"; return; fi pgrep -x metalvrd >/dev/null && die "metalvrd is running but not answering on port $PORT: stop it first ('${0:t} stop')" override_on || print -r -- "warning: macOS may keep the headset's display hidden: run '${0:t} display' once" >&2 - rift_plugged || print -r -- "warning: no Rift CV1 on USB yet: plug in its USB and HDMI (the Mac's own HDMI port)" >&2 + local model=$(headset_plugged) + [[ -n $model ]] || print -r -- "warning: no headset of this kind on USB yet: plug in its USB and its $VIDEO" >&2 mkdir -p ${LOG:h} - nohup $T/bin/metalvr rift --verbose >$LOG 2>&1 &! + nohup $T/bin/metalvr $FAMILY --verbose "$@" >$LOG 2>&1 &! local n=0 until server_up || (( n >= 30 )); do sleep 1; n=$((n + 1)); done - server_up || die "the Rift server did not start: see $LOG" - print -r -- "the Rift server is running (log: $LOG)" + server_up || die "the $SERVER did not start: see $LOG" + print -r -- "the $SERVER is running (log: $LOG)" # the headset comes on once its HID answers; its display appears in macOS a few seconds later n=0 until grep -q "rift: drawing on the headset's display" $LOG 2>/dev/null || (( n >= 20 )); do sleep 1; n=$((n + 1)); done if grep -q "rift: drawing on the headset's display" $LOG; then print -r -- "the headset is on and showing MetalVR's home" elif grep -q 'rift: headset on' $LOG; then - print -r -- "warning: the headset is on, but macOS has not shown its display: check '${0:t} display' and that its HDMI is in the Mac's own port" >&2 + print -r -- "warning: the headset is on, but macOS has not shown its display: check '${0:t} display' and its video cable" >&2 else - print -r -- "waiting for the headset: plug in its USB and HDMI (the server keeps waiting; log: $LOG)" >&2 + print -r -- "waiting for the headset: plug in its USB and its $VIDEO (the server keeps waiting; log: $LOG)" >&2 fi } @@ -69,21 +127,11 @@ install) print -r -- "next: ${0:t} display, then ${0:t} setup" ;; display) - if [[ ${2:-} == --undo ]]; then - [[ -f $OVR ]] || { print -r -- "nothing to undo: macOS's own override applies"; exit 0; } - sudo rm -f $OVR - print -r -- "removed $OVR: macOS hides the headset's display again (after it is plugged in again)" - exit 0 - fi - if override_on; then print -r -- "macOS already shows the headset's display ($OVR)"; exit 0; fi - # macOS's own override for Oculus's displays marks the CV1 a head-mounted display (is-hmd), which keeps it off - # the desktop; one in /Library takes precedence and lifts it (docs/RIFT.md) - print -r -- "macOS needs an administrator's password to write $OVR" - sudo mkdir -p ${OVR:h} - sudo rm -f $OVR - sudo /usr/libexec/PlistBuddy -c "Add :DisplayVendorID integer 16082" -c "Add :DisplayProductID integer 4" \ - -c "Add :is-hmd bool false" $OVR >/dev/null - print -r -- "done: unplug the headset's HDMI and plug it in again (or restart) for macOS to see it" + shift + [[ $FAMILY == psvr2 ]] && { print -r -- "macOS shows the PlayStation VR2's display as it is: nothing to do"; exit 0; } + # the release's own helper when this runs before it is installed + B=$T/bin/metalvr; [[ -x $B ]] || B=$H/bin/metalvr + exec $B display $FAMILY "$@" ;; setup) need_installed; need_metalsharp @@ -92,10 +140,11 @@ setup) pgrep -f 'vrserver\.exe' >/dev/null && die "quit SteamVR first: it writes its settings back when it quits" $T/bin/metalvr prefix $T/bin/metalvr steamvr - print -r -- "SteamVR is set up for the Rift. Quit Steam if it is running, so it starts again with these settings." + print -r -- "SteamVR is set up for the headset. Quit Steam if it is running, so it starts again with these settings." ;; start) - start_server + shift + start_server "$@" ;; stop) if pgrep -x metalvrd >/dev/null; then @@ -104,7 +153,7 @@ stop) pgrep -x metalvrd >/dev/null && die "metalvrd did not stop" print -r -- "stopped" else - print -r -- "the Rift server is not running" + print -r -- "the $SERVER is not running" fi ;; steamvr) @@ -118,16 +167,20 @@ game) ;; status) print -r -- "release: $(cat $T/lib/metalvr/VERSION 2>/dev/null || print 'not installed')" - override_on && print -r -- "display: macOS shows the headset's display" || - print -r -- "display: hidden by macOS (${0:t} display)" - rift_plugged && print -r -- "headset: a Rift CV1 is on USB" || print -r -- "headset: no Rift CV1 on USB" - print -r -- "sensors: $(sensors) Rift Sensor(s) on USB" + overrides | while IFS= read -r o; do + rest=${o#*:}; v=${o%%:*} p=${rest%%:*} name=${rest#*:} + lifted $v $p && print -r -- "display: macOS shows the $name's display ($v:$p)" || + print -r -- "display: the $name's ($v:$p) is hidden by macOS (${0:t} display)" + done + model=$(headset_plugged) + [[ -n $model ]] && print -r -- "headset: a $model is on USB" || print -r -- "headset: none of this kind on USB" + [[ $FAMILY == rift && $model == "Rift CV1" ]] && print -r -- "sensors: $(sensors) Rift Sensor(s) on USB" if server_up; then print -r -- "server: running (log: $LOG)"; else print -r -- "server: not running (${0:t} start)"; fi VP=$(find $MS/prefix-steam/drive_c/users -maxdepth 5 -name openvrpaths.vrpath 2>/dev/null | head -1) if [[ -n $VP ]] && grep -q 'metalvr' $VP; then print -r -- "steamvr: MetalVR's driver is registered" else print -r -- "steamvr: not set up (${0:t} setup)"; fi ;; *) - sed -n '2,12p' $0 | sed 's/^# \{0,1\}//' + sed -n '2,15p' $0 | sed 's/^# \{0,1\}//' ;; esac diff --git a/scripts/package-release.sh b/scripts/package-release.sh index 0f1556f..d3d5074 100755 --- a/scripts/package-release.sh +++ b/scripts/package-release.sh @@ -2,12 +2,15 @@ # Builds a MetalVR release for MetalSharp and packs it as dist/metalvr-VERSION-macos-arm64.tar.gz, laid out as # MetalSharp installs it (~/.metalsharp/metalvr-runtime): # bin/metalvrd the server (arm64), with ALVR's server_core and the Rift tracker -# bin/metalvr, bin/metalvr-rift the helper, and the Rift CV1 script (docs/RIFT_CV1.md) +# bin/metalvr, bin/metalvr-rift the helper, and the script for a headset on this Mac's own display +# bin/metalvr-lighthouse, bin/metalvr-psvr2 (docs/headsets/): the Rift's, the Lighthouse's +# and the PlayStation VR2's +# lib/libbasalt.dylib, lib/libtbb.12.dylib Basalt, which metalvrd loads for the Rift S's camera tracking # lib/metalvr/x86_64-windows/ the OpenXR runtime and its manifest, and OpenComposite's openvr_api.dll # lib/metalvr/steamvr/metalvr/ the SteamVR driver # lib/metalvr/LICENSE*, OpenComposite/ the licenses of what is redistributed # share/metalvr/home-logo.png the logo the Rift's home shows -# docs/ the Rift CV1, Rift and SteamVR guides +# docs/ the headset guides (docs/headsets), the driver, ALVR and SteamVR # # package-release.sh VERSION e.g. 0.1.0; builds in build/release-native and build/release-win64 # @@ -24,6 +27,8 @@ MACOS_FLOOR=15.0 # MetalSharp's sh scripts/fetch-deps.sh bash scripts/build-alvr.sh +sh scripts/build-libsurvive.sh +sh scripts/build-basalt.sh cmake -S . -B build/release-native -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_OSX_ARCHITECTURES=arm64 \ -DCMAKE_OSX_DEPLOYMENT_TARGET=$MACOS_FLOOR -DMETALVR_WITH_ALVR=ON cmake --build build/release-native --target metalvrd @@ -36,33 +41,58 @@ cmake --build build/release-native --target metalvrd D=build/release-native/daemon/metalvrd W=build/release-win64 -for f in $D $W/runtime/metalvr_openxr.dll $W/runtime/metalvr_openxr.json build/opencomposite/openvr_api.dll \ +B=build/basalt/prefix/lib +for f in $D $B/libbasalt.dylib $B/libtbb.12.dylib $W/runtime/metalvr_openxr.dll $W/runtime/metalvr_openxr.json build/opencomposite/openvr_api.dll \ $W/steamvr/metalvr/bin/win64/driver_metalvr.dll; do [[ -f $f ]] || { echo >&2 "!! not built: $f"; exit 1; } done $D --help 2>&1 | grep -q -- '--trust' || { echo >&2 "!! metalvrd was built without the ALVR driver"; exit 1; } $D --help 2>&1 | grep -q -- 'rift' || { echo >&2 "!! metalvrd was built without the Rift driver"; exit 1; } +$D --survive-check >/dev/null 2>&1 || { echo >&2 "!! metalvrd was built without libsurvive"; exit 1; } rm -rf $OUT; mkdir -p $OUT/bin $OUT/lib/metalvr/x86_64-windows $OUT/lib/metalvr/steamvr $OUT/share/metalvr $OUT/docs cp $D $OUT/bin/metalvrd cp scripts/metalsharp/metalvr scripts/metalsharp/metalvr-rift $OUT/bin/ -chmod +x $OUT/bin/* +ln -s metalvr-rift $OUT/bin/metalvr-lighthouse # the same script, for the Lighthouse headsets +ln -s metalvr-rift $OUT/bin/metalvr-psvr2 # and for the PlayStation VR2 +chmod +x $OUT/bin/metalvrd $OUT/bin/metalvr $OUT/bin/metalvr-rift +# Basalt (with OpenCV and fmt inside it) and oneTBB, found by metalvrd as bin/../lib/libbasalt.dylib; Basalt finds +# oneTBB beside it (@loader_path) +cp -L $B/libbasalt.dylib $OUT/lib/libbasalt.dylib +cp -L $B/libtbb.12.dylib $OUT/lib/libtbb.12.dylib cp $W/runtime/metalvr_openxr.dll $W/runtime/metalvr_openxr.json build/opencomposite/openvr_api.dll $OUT/lib/metalvr/x86_64-windows/ cp -R $W/steamvr/metalvr $OUT/lib/metalvr/steamvr/ cp scripts/metalsharp/home-logo.png $OUT/share/metalvr/ -cp docs/RIFT_CV1.md docs/RIFT.md docs/STEAMVR.md $OUT/docs/ -# what is redistributed, with its license: MetalVR (MIT), ALVR's server_core (MIT) and OpenHMD's Rift tracker -# (BSL-1.0) inside metalvrd, and OpenComposite (GPL-3.0) with the Khronos loader and OpenVR headers compiled in +cp -R docs/headsets $OUT/docs/ +cp docs/RIFT.md docs/STEAMVR.md docs/HEADSETS.md docs/ALVR.md $OUT/docs/ +# what is redistributed, with its license: MetalVR (MIT), ALVR's server_core (MIT), OpenHMD's Rift tracker +# (BSL-1.0), libsurvive (MIT) and hidapi (BSD) inside metalvrd, Basalt (BSD-3-Clause) with what it builds in +# (OpenCV, Apache-2.0; fmt, MIT; Sophus, MIT; cereal, BSD; Eigen, MPL-2.0; OpenGV, BSD; CLI11, BSD; magic_enum, MIT) +# and oneTBB (Apache-2.0), and OpenComposite (GPL-3.0) with the Khronos loader and OpenVR headers compiled in cp LICENSE $OUT/lib/metalvr/LICENSE cp external/ALVR/LICENSE $OUT/lib/metalvr/LICENSE.ALVR cp external/OpenHMD-rift/LICENSE $OUT/lib/metalvr/LICENSE.OpenHMD +cp external/libsurvive/LICENSE $OUT/lib/metalvr/LICENSE.libsurvive +cp external/hidapi/LICENSE-bsd.txt $OUT/lib/metalvr/LICENSE.hidapi +mkdir -p $OUT/lib/metalvr/Basalt +BT=external/basalt/thirdparty +cp external/basalt/LICENSE $OUT/lib/metalvr/Basalt/LICENSE.Basalt +cp external/oneTBB/LICENSE.txt $OUT/lib/metalvr/Basalt/LICENSE.oneTBB +cp external/opencv/LICENSE $OUT/lib/metalvr/Basalt/LICENSE.OpenCV +cp external/fmt/LICENSE $OUT/lib/metalvr/Basalt/LICENSE.fmt +cp $BT/basalt-headers/thirdparty/Sophus/LICENSE.txt $OUT/lib/metalvr/Basalt/LICENSE.Sophus +cp $BT/basalt-headers/thirdparty/cereal/LICENSE $OUT/lib/metalvr/Basalt/LICENSE.cereal +cp $BT/basalt-headers/thirdparty/eigen/COPYING.MPL2 $OUT/lib/metalvr/Basalt/LICENSE.Eigen +cp $BT/opengv/License.txt $OUT/lib/metalvr/Basalt/LICENSE.OpenGV +cp $BT/CLI11/LICENSE $OUT/lib/metalvr/Basalt/LICENSE.CLI11 +cp $BT/magic_enum/LICENSE $OUT/lib/metalvr/Basalt/LICENSE.magic_enum O=external/OpenComposite mkdir -p $OUT/lib/metalvr/OpenComposite cp $O/LICENSE.txt $O/LICENSE_*.txt $O/LICENCE_*.txt $OUT/lib/metalvr/OpenComposite/ cp $O/libs/openxr-sdk/LICENSE $OUT/lib/metalvr/OpenComposite/LICENSE_OPENXR_SDK.txt -echo "MetalVR $V ($REV, metalsharp/MetalVR), ALVR $(git -C external/ALVR describe --tags --always), OpenHMD $(git -C external/OpenHMD-rift rev-parse --short HEAD), OpenComposite $(git -C $O rev-parse --short HEAD)" \ +echo "MetalVR $V ($REV, metalsharp/MetalVR), ALVR $(git -C external/ALVR describe --tags --always), OpenHMD $(git -C external/OpenHMD-rift rev-parse --short HEAD), libsurvive $(git -C external/libsurvive rev-parse --short HEAD), Basalt $(git -C external/basalt rev-parse --short HEAD), OpenComposite $(git -C $O rev-parse --short HEAD)" \ > $OUT/lib/metalvr/VERSION -codesign --force --sign - $OUT/bin/metalvrd +codesign --force --sign - $OUT/bin/metalvrd $OUT/lib/libbasalt.dylib $OUT/lib/libtbb.12.dylib tar -C $ROOT/dist -czf $ROOT/dist/$N.tar.gz $N (cd $ROOT/dist && shasum -a 256 $N.tar.gz > $N.tar.gz.sha256) # the format shasum -c reads diff --git a/scripts/test-all.sh b/scripts/test-all.sh index 04502c1..f0ad642 100755 --- a/scripts/test-all.sh +++ b/scripts/test-all.sh @@ -45,6 +45,33 @@ native native-flip --frames 150 --no-quad --flip run native-headset-size env METALVR_TEST_NAME=native-headset-size \ METALVR_DAEMON_ARGS="--eye 2144x2336 --codec hevc" "$root/scripts/test-native.sh" --frames 300 run failures "$root/scripts/test-failures.sh" +# the headsets (docs/HEADSETS.md): each controller kind's interaction profile, the protocols of the headsets MetalVR +# drives itself, and each one's panel drawn through its lens model +run native-vive-wands env METALVR_TEST_NAME=native-vive-wands METALVR_DAEMON_ARGS="--headset vive" \ + "$root/scripts/test-native.sh" --frames 120 --expect-profile /interaction_profiles/htc/vive_controller +run native-index-touch env METALVR_TEST_NAME=native-index-touch METALVR_DAEMON_ARGS="--headset index" \ + "$root/scripts/test-native.sh" --frames 120 +run native-psvr2-sense env METALVR_TEST_NAME=native-psvr2-sense METALVR_DAEMON_ARGS="--headset psvr2" \ + "$root/scripts/test-native.sh" --frames 120 --expect-profile /interaction_profiles/oculus/touch_controller +run headset-protocols "$root/build/native/tests/headset_protocol_check" +run foveation "$root/build/native/daemon/metalvrd" --foveation-check +run libsurvive "$root/build/native/daemon/metalvrd" --survive-check sim +if [ -f "$root/build/basalt/prefix/lib/libbasalt.dylib" ] && [ -x "$root/build/native/tests/rift_s_tracking_check" ]; then + run rift-s-tracking "$root/build/native/tests/rift_s_tracking_check" 24 +else + echo "skip rift-s-tracking (Basalt is not built; scripts/build-basalt.sh)" +fi +run passthrough "$root/build/native/daemon/metalvrd" --passthrough-check +run native-foveated env METALVR_TEST_NAME=native-foveated METALVR_DAEMON_ARGS="--foveated-encoding --codec hevc" \ + "$root/scripts/test-native.sh" --frames 120 +for model in rift-cv1 rift-s index vive vive-pro vive-pro-2 psvr2; do + run "panel-$model" "$root/build/native/daemon/metalvrd" --panel-check "$model" "$out/panel-$model.png" +done +if [ -x "$root/build/native/tests/alvr_driver_smoke" ]; then + run alvr-models "$root/scripts/test-alvr-models.sh" +else + echo "skip alvr-models (metalvrd was built without ALVR; scripts/build-alvr.sh)" +fi if [ -x "$root/build/native/tests/alvr_driver_smoke" ]; then rm -rf "$out/alvr-smoke" run alvr-driver "$root/build/native/tests/alvr_driver_smoke" "$out/alvr-smoke" 2 1234.client diff --git a/scripts/test-alvr-models.sh b/scripts/test-alvr-models.sh new file mode 100755 index 0000000..e7699b4 --- /dev/null +++ b/scripts/test-alvr-models.sh @@ -0,0 +1,56 @@ +#!/bin/zsh +# Which headset the ALVR driver names, without a headset: for each model an ALVR client reports, a session whose +# trusted client last reported it, and the driver must name that headset before anything connects. A name MetalVR +# does not know (a Pico, "Android (unknown)" from the stock client on a Steam Frame) must stay an unknown headset, +# never a Rift; MetalVR's patched client names a Steam Frame (patches/alvr/0003). +# +# scripts/test-alvr-models.sh (needs build/native with the ALVR driver: scripts/build-alvr.sh) +set -eu +root=${0:A:h:h} +build=${METALVR_BUILD:-$root/build/native} +smoke=$build/tests/alvr_driver_smoke +[[ -x $smoke ]] || { print -r -- "no $smoke: build with the ALVR driver first" >&2; exit 1; } +work=$(mktemp -d) +trap 'rm -rf $work' EXIT +host=1234.client + +# a full session as ALVR writes it, made once +"$smoke" $work/seed 0.5 $host >$work/seed.log 2>&1 || { cat $work/seed.log; exit 1; } + +failed=0 +for pair in "Quest 1:quest-1" "Quest 2:quest-2" "Quest 3:quest-3" "Quest 3S:quest-3s" "Quest Pro:quest-pro" \ + "Steam Frame:steam-frame" "Pico 4:unknown" "Android (unknown):unknown" "Unknown:unknown"; do + name=${pair%%:*} key=${pair##*:} + dir=$work/${key}-${name// /_} + cp -R $work/seed $dir + python3 - "$dir/session.json" "$host" "$name" <<'PY' +import json, sys +path, host, name = sys.argv[1:4] +session = json.load(open(path)) +session["client_connections"][host]["display_name"] = name +json.dump(session, open(path, "w"), indent=2) +PY + if METALVR_EXPECT_HEADSET=$key "$smoke" $dir 0.3 $host >$dir.log 2>&1; then + print -r -- "ok \"$name\" -> $key" + else + print -r -- "FAIL \"$name\" -> $key"; grep -E "FAIL|headset before" $dir.log; failed=1 + fi +done + +# a headset on USB: without an adb the driver refuses to start (ALVR would otherwise download one); with one it +# starts and ALVR downloads nothing +if PATH=/usr/bin:/bin "$smoke" $work/wired-none 0.3 client.wired >$work/wired-none.log 2>&1; then + print -r -- "FAIL wired without adb started"; failed=1 +elif grep -q "needs adb" $work/wired-none.log; then + print -r -- "ok wired without adb refused" +else + print -r -- "FAIL wired without adb: no reason given"; cat $work/wired-none.log; failed=1 +fi +mkdir -p $work/bin && print '#!/bin/sh\nexit 0' >$work/bin/adb && chmod +x $work/bin/adb +if PATH=$work/bin:/usr/bin:/bin "$smoke" $work/wired 0.3 client.wired >$work/wired.log 2>&1 && + [[ ! -e $work/wired/platform-tools ]]; then + print -r -- "ok wired with adb started" +else + print -r -- "FAIL wired with adb"; grep -E "FAIL|adb" $work/wired.log; failed=1 +fi +exit $failed diff --git a/steamvr/driver_metalvr.cpp b/steamvr/driver_metalvr.cpp index 940b808..5e87a39 100644 --- a/steamvr/driver_metalvr.cpp +++ b/steamvr/driver_metalvr.cpp @@ -13,6 +13,7 @@ #include +#include #include #include #include @@ -26,7 +27,9 @@ #include #include +#include "hand_skeleton.h" #include "ipc.h" +#include "metalvr_headsets.h" #include "metalvr_protocol.h" namespace metalvr { @@ -603,6 +606,96 @@ mvr_quat quatMultiply(const mvr_quat &a, const mvr_quat &b) { a.w * b.z + a.x * b.y - a.y * b.x + a.z * b.w, a.w * b.w - a.x * b.x - a.y * b.y - a.z * b.z}; } +// What SteamVR is told the controllers are: its own profile and models for the controllers in the user's hands, +// so games' bindings for them apply as they are. +struct ControllerLook { + const char *profile; + const char *type; + const char *render_model[2]; // left, right + const char *model_number[2]; + const char *manufacturer; +}; + +ControllerLook lookFor(uint32_t headset, uint32_t kind) { + switch (kind) { + case MVR_CONTROLLERS_INDEX: + return {"{indexcontroller}/input/index_controller_profile.json", + "knuckles", + {"{indexcontroller}valve_controller_knu_1_0_left", "{indexcontroller}valve_controller_knu_1_0_right"}, + {"Knuckles (Left Controller)", "Knuckles (Right Controller)"}, + "Valve"}; + case MVR_CONTROLLERS_VIVE_WAND: + return {"{htc}/input/vive_controller_profile.json", + "vive_controller", + {"vr_controller_vive_1_5", "vr_controller_vive_1_5"}, + {"Vive. Controller MV", "Vive. Controller MV"}, + "HTC"}; + case MVR_CONTROLLERS_TOUCH_PLUS: + if (headset == MVR_HEADSET_QUEST_PRO) + return {"{oculus}/input/touch_profile.json", + "oculus_touch", + {"oculus_quest_pro_controller_left", "oculus_quest_pro_controller_right"}, + {"Meta Quest Pro (Left Controller)", "Meta Quest Pro (Right Controller)"}, + "Meta"}; + return {"{oculus}/input/touch_profile.json", + "oculus_touch", + {"oculus_quest_plus_controller_left", "oculus_quest_plus_controller_right"}, + {"Meta Quest 3 (Left Controller)", "Meta Quest 3 (Right Controller)"}, + "Meta"}; + case MVR_CONTROLLERS_PS_SENSE: + // Sony's SteamVR driver's names are its own: shown as Touch controllers, whose bindings games have, with the + // Sense's face buttons on Touch's A, B, X and Y + return {"{oculus}/input/touch_profile.json", + "oculus_touch", + {"oculus_quest2_controller_left", "oculus_quest2_controller_right"}, + {"PlayStation VR2 Sense (Left Controller)", "PlayStation VR2 Sense (Right Controller)"}, + "Sony"}; + case MVR_CONTROLLERS_STEAM_FRAME: + // SteamVR's own names for these are not known yet (docs/HEADSETS.md): shown as ringless Touch controllers, + // whose bindings games have, with the Frame's extra buttons on Touch's nearest inputs + return {"{oculus}/input/touch_profile.json", + "oculus_touch", + {"oculus_quest_plus_controller_left", "oculus_quest_plus_controller_right"}, + {"Steam Frame (Left Controller)", "Steam Frame (Right Controller)"}, + "Valve"}; + default: + if (headset == MVR_HEADSET_RIFT_S || headset == MVR_HEADSET_QUEST_1) + return {"{oculus}/input/touch_profile.json", + "oculus_touch", + {"oculus_rifts_controller_left", "oculus_rifts_controller_right"}, + {"Oculus Rift S (Left Controller)", "Oculus Rift S (Right Controller)"}, + "Oculus"}; + if (headset == MVR_HEADSET_QUEST_2) + return {"{oculus}/input/touch_profile.json", + "oculus_touch", + {"oculus_quest2_controller_left", "oculus_quest2_controller_right"}, + {"Miramar (Left Controller)", "Miramar (Right Controller)"}, + "Oculus"}; + return {"{oculus}/input/touch_profile.json", + "oculus_touch", + {"oculus_cv1_controller_left", "oculus_cv1_controller_right"}, + {"Oculus Rift CV1 (Left Controller)", "Oculus Rift CV1 (Right Controller)"}, + "Oculus"}; + } +} + +// whether a headset's tracking can drift in yaw: one tracked by its rotation alone (MetalVR's Rift S, a Rift CV1 +// without sensors, a Lighthouse headset without base stations) can; inside-out tracking cannot +bool driftsInYaw(uint32_t headset) { + switch (headset) { + case MVR_HEADSET_QUEST_1: + case MVR_HEADSET_QUEST_2: + case MVR_HEADSET_QUEST_3: + case MVR_HEADSET_QUEST_3S: + case MVR_HEADSET_QUEST_PRO: + case MVR_HEADSET_STEAM_FRAME: + case MVR_HEADSET_PSVR2: + return false; + default: + return true; + } +} + class Controller final : public TrackedDevice { public: Controller(Link &link, uint32_t hand) : link_(link), hand_(hand) {} @@ -611,49 +704,102 @@ class Controller final : public TrackedDevice { auto *p = vr::VRProperties(); container_ = p->TrackedDeviceToPropertyContainer(id); const bool left = hand_ == 0; + const mvr_system_info &system = link_.system(); + headset_ = system.headset; + kind_ = system.controllers; + const ControllerLook look = lookFor(headset_, kind_); p->SetInt32Property(container_, vr::Prop_ControllerRoleHint_Int32, left ? vr::TrackedControllerRole_LeftHand : vr::TrackedControllerRole_RightHand); - // SteamVR's own Oculus Touch profile and models, so games' Touch bindings apply as they are - p->SetStringProperty(container_, vr::Prop_InputProfilePath_String, "{oculus}/input/touch_profile.json"); - p->SetStringProperty(container_, vr::Prop_ControllerType_String, "oculus_touch"); - p->SetStringProperty(container_, vr::Prop_RenderModelName_String, - left ? "oculus_cv1_controller_left" : "oculus_cv1_controller_right"); - p->SetStringProperty(container_, vr::Prop_ModelNumber_String, - left ? "Oculus Rift CV1 (Left Controller)" : "Oculus Rift CV1 (Right Controller)"); - p->SetStringProperty(container_, vr::Prop_ManufacturerName_String, "Oculus"); + p->SetStringProperty(container_, vr::Prop_InputProfilePath_String, look.profile); + p->SetStringProperty(container_, vr::Prop_ControllerType_String, look.type); + p->SetStringProperty(container_, vr::Prop_RenderModelName_String, look.render_model[hand_]); + p->SetStringProperty(container_, vr::Prop_ModelNumber_String, look.model_number[hand_]); + p->SetStringProperty(container_, vr::Prop_ManufacturerName_String, look.manufacturer); p->SetStringProperty(container_, vr::Prop_TrackingSystemName_String, "metalvr"); p->SetStringProperty(container_, vr::Prop_SerialNumber_String, left ? "MetalVR-Touch-Left" : "MetalVR-Touch-Right"); p->SetStringProperty(container_, vr::Prop_RegisteredDeviceType_String, left ? "metalvr/MetalVR-Touch-Left" : "metalvr/MetalVR-Touch-Right"); p->SetBoolProperty(container_, vr::Prop_DeviceProvidesBatteryStatus_Bool, false); + log("metalvr: %s controller: %s (%s)", left ? "left" : "right", look.model_number[hand_], look.type); auto *in = vr::VRDriverInput(); - const char *first = left ? "/input/x" : "/input/a"; - const char *second = left ? "/input/y" : "/input/b"; - char path[64]; - std::snprintf(path, sizeof(path), "%s/click", first); - in->CreateBooleanComponent(container_, path, &primary_click_); - std::snprintf(path, sizeof(path), "%s/touch", first); - in->CreateBooleanComponent(container_, path, &primary_touch_); - std::snprintf(path, sizeof(path), "%s/click", second); - in->CreateBooleanComponent(container_, path, &secondary_click_); - std::snprintf(path, sizeof(path), "%s/touch", second); - in->CreateBooleanComponent(container_, path, &secondary_touch_); - in->CreateBooleanComponent(container_, "/input/system/click", &system_click_); - in->CreateBooleanComponent(container_, "/input/system/touch", &system_touch_); - in->CreateBooleanComponent(container_, "/input/joystick/click", &stick_click_); - in->CreateBooleanComponent(container_, "/input/joystick/touch", &stick_touch_); - in->CreateScalarComponent(container_, "/input/joystick/x", &stick_x_, vr::VRScalarType_Absolute, - vr::VRScalarUnits_NormalizedTwoSided); - in->CreateScalarComponent(container_, "/input/joystick/y", &stick_y_, vr::VRScalarType_Absolute, - vr::VRScalarUnits_NormalizedTwoSided); - in->CreateScalarComponent(container_, "/input/trigger/value", &trigger_, vr::VRScalarType_Absolute, - vr::VRScalarUnits_NormalizedOneSided); - in->CreateBooleanComponent(container_, "/input/trigger/touch", &trigger_touch_); - in->CreateScalarComponent(container_, "/input/grip/value", &grip_, vr::VRScalarType_Absolute, - vr::VRScalarUnits_NormalizedOneSided); - in->CreateBooleanComponent(container_, "/input/grip/touch", &grip_touch_); - in->CreateBooleanComponent(container_, "/input/thumbrest/touch", &thumbrest_touch_); + const auto boolean = [&](const char *path) { + vr::VRInputComponentHandle_t handle = 0; + in->CreateBooleanComponent(container_, path, &handle); + return handle; + }; + const auto scalar = [&](const char *path, bool two_sided) { + vr::VRInputComponentHandle_t handle = 0; + in->CreateScalarComponent(container_, path, &handle, vr::VRScalarType_Absolute, + two_sided ? vr::VRScalarUnits_NormalizedTwoSided : vr::VRScalarUnits_NormalizedOneSided); + return handle; + }; + if (kind_ == MVR_CONTROLLERS_VIVE_WAND) { + // the paths of SteamVR's Vive controller profile + c_.system_click = boolean("/input/system/click"); + c_.menu_click = boolean("/input/application_menu/click"); + c_.grip_click = boolean("/input/grip/click"); + c_.trigger_click = boolean("/input/trigger/click"); + c_.trigger = scalar("/input/trigger/value", false); + c_.pad_x = scalar("/input/trackpad/x", true); + c_.pad_y = scalar("/input/trackpad/y", true); + c_.pad_click = boolean("/input/trackpad/click"); + c_.pad_touch = boolean("/input/trackpad/touch"); + } else if (kind_ == MVR_CONTROLLERS_INDEX) { + // the paths of SteamVR's Index controller profile + c_.primary_click = boolean("/input/a/click"); + c_.primary_touch = boolean("/input/a/touch"); + c_.secondary_click = boolean("/input/b/click"); + c_.secondary_touch = boolean("/input/b/touch"); + c_.system_click = boolean("/input/system/click"); + c_.system_touch = boolean("/input/system/touch"); + c_.trigger_click = boolean("/input/trigger/click"); + c_.trigger = scalar("/input/trigger/value", false); + c_.trigger_touch = boolean("/input/trigger/touch"); + c_.pad_x = scalar("/input/trackpad/x", true); + c_.pad_y = scalar("/input/trackpad/y", true); + c_.pad_touch = boolean("/input/trackpad/touch"); + c_.pad_force = scalar("/input/trackpad/force", false); + c_.grip = scalar("/input/grip/value", false); + c_.grip_force = scalar("/input/grip/force", false); + c_.grip_touch = boolean("/input/grip/touch"); + c_.stick_x = scalar("/input/thumbstick/x", true); + c_.stick_y = scalar("/input/thumbstick/y", true); + c_.stick_click = boolean("/input/thumbstick/click"); + c_.stick_touch = boolean("/input/thumbstick/touch"); + for (int finger = 0; finger < 4; ++finger) { + static const char *const paths[4] = {"/input/finger/index", "/input/finger/middle", "/input/finger/ring", + "/input/finger/pinky"}; + c_.finger[finger] = scalar(paths[finger], false); + } + in->CreateSkeletonComponent(container_, left ? "/input/skeleton/left" : "/input/skeleton/right", + left ? "/skeleton/hand/left" : "/skeleton/hand/right", "/pose/raw", + vr::VRSkeletalTracking_Partial, nullptr, 0, &skeleton_); + } else { + // the Touch family's paths, which the Steam Frame's controllers are shown with too + const char *first = left ? "/input/x" : "/input/a"; + const char *second = left ? "/input/y" : "/input/b"; + char path[64]; + std::snprintf(path, sizeof(path), "%s/click", first); + c_.primary_click = boolean(path); + std::snprintf(path, sizeof(path), "%s/touch", first); + c_.primary_touch = boolean(path); + std::snprintf(path, sizeof(path), "%s/click", second); + c_.secondary_click = boolean(path); + std::snprintf(path, sizeof(path), "%s/touch", second); + c_.secondary_touch = boolean(path); + c_.system_click = boolean("/input/system/click"); + c_.system_touch = boolean("/input/system/touch"); + c_.stick_click = boolean("/input/joystick/click"); + c_.stick_touch = boolean("/input/joystick/touch"); + c_.stick_x = scalar("/input/joystick/x", true); + c_.stick_y = scalar("/input/joystick/y", true); + c_.trigger = scalar("/input/trigger/value", false); + c_.trigger_touch = boolean("/input/trigger/touch"); + c_.grip = scalar("/input/grip/value", false); + c_.grip_touch = boolean("/input/grip/touch"); + c_.thumbrest_touch = boolean("/input/thumbrest/touch"); + } in->CreateHapticComponent(container_, "/output/haptic", &haptic_); // only now may the pose thread update it: its inputs exist id_ = id; @@ -684,13 +830,18 @@ class Controller final : public TrackedDevice { pose.deviceIsConnected = (c.flags & MVR_CONTROLLER_CONNECTED) != 0; pose.poseIsValid = (c.flags & MVR_CONTROLLER_POSE_VALID) != 0; pose.result = pose.poseIsValid ? vr::TrackingResult_Running_OK : vr::TrackingResult_Running_OutOfRange; - pose.willDriftInYaw = true; + pose.willDriftInYaw = driftsInYaw(headset_); if (pose.poseIsValid) { - // the controller's own frame, which SteamVR's Touch models place their grip and tip from: the grip pose - // without its offset (RiftLinux: 60 degrees about x, at 0, -0.032, 0.040) - const mvr_quat grip_rotation{0.5f, 0, 0, 0.8660254f}; - const mvr_quat q = quatMultiply(c.grip.orientation, quatConjugate(grip_rotation)); - const mvr_vec3 offset = rotate(q, {0, -0.03207f, 0.04045f}); + mvr_quat q = c.grip.orientation; + mvr_vec3 offset{0, 0, 0}; + if (kind_ != MVR_CONTROLLERS_INDEX && kind_ != MVR_CONTROLLERS_VIVE_WAND) { + // the controller's own frame, which SteamVR's Touch models place their grip and tip from: the grip pose + // without its offset (RiftLinux: 60 degrees about x, at 0, -0.032, 0.040). Lighthouse controllers report + // their own frame as the grip already. + const mvr_quat grip_rotation{0.5f, 0, 0, 0.8660254f}; + q = quatMultiply(c.grip.orientation, quatConjugate(grip_rotation)); + offset = rotate(q, {0, -0.03207f, 0.04045f}); + } pose.qRotation = {q.w, q.x, q.y, q.z}; pose.vecPosition[0] = c.grip.position.x - offset.x; pose.vecPosition[1] = c.grip.position.y - offset.y; @@ -707,22 +858,52 @@ class Controller final : public TrackedDevice { vr::VRServerDriverHost()->TrackedDevicePoseUpdated(id_, pose, sizeof(pose)); if (!pose.deviceIsConnected) return; + updateBattery(c); auto *in = vr::VRDriverInput(); - in->UpdateBooleanComponent(primary_click_, c.buttons & MVR_BUTTON_PRIMARY, 0); - in->UpdateBooleanComponent(primary_touch_, c.touches & MVR_BUTTON_PRIMARY, 0); - in->UpdateBooleanComponent(secondary_click_, c.buttons & MVR_BUTTON_SECONDARY, 0); - in->UpdateBooleanComponent(secondary_touch_, c.touches & MVR_BUTTON_SECONDARY, 0); - in->UpdateBooleanComponent(system_click_, c.buttons & (MVR_BUTTON_MENU | MVR_BUTTON_SYSTEM), 0); - in->UpdateBooleanComponent(system_touch_, c.buttons & (MVR_BUTTON_MENU | MVR_BUTTON_SYSTEM), 0); - in->UpdateBooleanComponent(stick_click_, c.buttons & MVR_BUTTON_THUMBSTICK, 0); - in->UpdateBooleanComponent(stick_touch_, c.touches & MVR_BUTTON_THUMBSTICK, 0); - in->UpdateScalarComponent(stick_x_, c.thumbstick_x, 0); - in->UpdateScalarComponent(stick_y_, c.thumbstick_y, 0); - in->UpdateScalarComponent(trigger_, c.trigger, 0); - in->UpdateBooleanComponent(trigger_touch_, (c.touches & MVR_BUTTON_TRIGGER) || c.trigger > 0.05f, 0); - in->UpdateScalarComponent(grip_, c.squeeze, 0); - in->UpdateBooleanComponent(grip_touch_, c.squeeze > 0.05f, 0); - in->UpdateBooleanComponent(thumbrest_touch_, c.touches & MVR_BUTTON_THUMBREST, 0); + const auto set = [&](vr::VRInputComponentHandle_t handle, bool value) { + if (handle) + in->UpdateBooleanComponent(handle, value, 0); + }; + const auto value = [&](vr::VRInputComponentHandle_t handle, float v) { + if (handle) + in->UpdateScalarComponent(handle, v, 0); + }; + const bool frame = kind_ == MVR_CONTROLLERS_STEAM_FRAME; + // Steam Frame on Touch's paths: its right hand's X and Y as A and B, its left hand's D-pad up and down as X + // and Y, the bumper as the grip's click + const uint32_t primary = MVR_BUTTON_PRIMARY | (frame ? uint32_t{MVR_BUTTON_TERTIARY | MVR_BUTTON_DPAD_UP} : 0u); + const uint32_t secondary = + MVR_BUTTON_SECONDARY | (frame ? uint32_t{MVR_BUTTON_QUATERNARY | MVR_BUTTON_DPAD_DOWN} : 0u); + set(c_.primary_click, c.buttons & primary); + set(c_.primary_touch, c.touches & primary); + set(c_.secondary_click, c.buttons & secondary); + set(c_.secondary_touch, c.touches & secondary); + const uint32_t system = static_cast(MVR_BUTTON_SYSTEM) | + (kind_ == MVR_CONTROLLERS_VIVE_WAND ? 0u : static_cast(MVR_BUTTON_MENU)) | + (frame ? static_cast(MVR_BUTTON_VIEW) : 0u); + set(c_.system_click, c.buttons & system); + set(c_.system_touch, c.buttons & system); + set(c_.menu_click, c.buttons & MVR_BUTTON_MENU); + set(c_.stick_click, c.buttons & MVR_BUTTON_THUMBSTICK); + set(c_.stick_touch, c.touches & MVR_BUTTON_THUMBSTICK); + value(c_.stick_x, c.thumbstick_x); + value(c_.stick_y, c.thumbstick_y); + set(c_.trigger_click, (c.buttons & MVR_BUTTON_TRIGGER) || c.trigger > 0.95f); + value(c_.trigger, c.trigger); + set(c_.trigger_touch, (c.touches & MVR_BUTTON_TRIGGER) || c.trigger > 0.05f); + const float squeeze = frame && (c.buttons & MVR_BUTTON_BUMPER) ? 1.0f : c.squeeze; + value(c_.grip, squeeze); + value(c_.grip_force, c.squeeze_force); + set(c_.grip_touch, squeeze > 0.05f || (c.touches & MVR_BUTTON_SQUEEZE)); + set(c_.grip_click, (c.buttons & MVR_BUTTON_SQUEEZE) || c.squeeze > 0.7f); + set(c_.thumbrest_touch, c.touches & MVR_BUTTON_THUMBREST); + value(c_.pad_x, c.trackpad_x); + value(c_.pad_y, c.trackpad_y); + set(c_.pad_click, c.buttons & MVR_BUTTON_TRACKPAD); + set(c_.pad_touch, c.touches & MVR_BUTTON_TRACKPAD); + value(c_.pad_force, c.trackpad_force); + if (kind_ == MVR_CONTROLLERS_INDEX) + updateFingers(c); } // SteamVR's vibration event for this controller, to metalvrd @@ -736,16 +917,103 @@ class Controller final : public TrackedDevice { } private: + // the finger curls, as SteamVR's finger inputs and as the hand skeleton + void updateFingers(const mvr_controller_state &c) { + float curl[5]; + const bool measured = (c.flags & MVR_CONTROLLER_FINGERS_VALID) != 0; + // without measured fingers: the index finger follows the trigger, the others the grip + curl[1] = measured ? c.fingers[MVR_FINGER_INDEX] : c.trigger; + curl[2] = measured ? c.fingers[MVR_FINGER_MIDDLE] : c.squeeze; + curl[3] = measured ? c.fingers[MVR_FINGER_RING] : c.squeeze; + curl[4] = measured ? c.fingers[MVR_FINGER_PINKY] : c.squeeze; + curl[0] = (c.touches & (MVR_BUTTON_PRIMARY | MVR_BUTTON_SECONDARY | MVR_BUTTON_THUMBSTICK | MVR_BUTTON_TRACKPAD)) + ? 1.0f + : 0.0f; + auto *in = vr::VRDriverInput(); + for (int finger = 0; finger < 4; ++finger) + if (c_.finger[finger]) + in->UpdateScalarComponent(c_.finger[finger], std::clamp(curl[finger + 1], 0.0f, 1.0f), 0); + if (!skeleton_) + return; + vr::VRBoneTransform_t bones[skeleton::kBoneCount]; + const skeleton::Bone *open = hand_ == 0 ? skeleton::kLeftOpen : skeleton::kRightOpen; + const skeleton::Bone *closed = hand_ == 0 ? skeleton::kLeftClosed : skeleton::kRightClosed; + for (int bone = 0; bone < skeleton::kBoneCount; ++bone) { + // which finger a bone belongs to: 0 thumb .. 4 pinky; the root and wrist to none + int finger = -1; + if (bone >= 2 && bone <= 25) + finger = bone < 6 ? 0 : 1 + (bone - 6) / 5; + else if (bone >= 26) + finger = bone - 26; + const float t = finger < 0 ? 0.0f : std::clamp(curl[finger], 0.0f, 1.0f); + bones[bone] = blendBone(open[bone], closed[bone], t); + } + in->UpdateSkeletonComponent(skeleton_, vr::VRSkeletalMotionRange_WithController, bones, skeleton::kBoneCount); + in->UpdateSkeletonComponent(skeleton_, vr::VRSkeletalMotionRange_WithoutController, bones, + skeleton::kBoneCount); + } + + static vr::VRBoneTransform_t blendBone(const skeleton::Bone &a, const skeleton::Bone &b, float t) { + vr::VRBoneTransform_t out{}; + for (int i = 0; i < 3; ++i) + out.position.v[i] = a.position[i] + (b.position[i] - a.position[i]) * t; + out.position.v[3] = 1; + // normalized lerp along the shorter way: close enough to a slerp between two poses of one finger + float bq[4] = {b.orientation[0], b.orientation[1], b.orientation[2], b.orientation[3]}; + const float dot = a.orientation[0] * bq[0] + a.orientation[1] * bq[1] + a.orientation[2] * bq[2] + + a.orientation[3] * bq[3]; + if (dot < 0) + for (float &v : bq) + v = -v; + float q[4]; + float length = 0; + for (int i = 0; i < 4; ++i) { + q[i] = a.orientation[i] + (bq[i] - a.orientation[i]) * t; + length += q[i] * q[i]; + } + length = length > 0 ? std::sqrt(length) : 1.0f; + out.orientation = {q[0] / length, q[1] / length, q[2] / length, q[3] / length}; + return out; + } + + // the controller's charge, when the headset reports it, a few times a minute + void updateBattery(const mvr_controller_state &c) { + const bool valid = (c.flags & MVR_CONTROLLER_BATTERY_VALID) != 0; + const ULONGLONG now = GetTickCount64(); + if (valid == battery_valid_ && now - battery_set_ < 10000) + return; + battery_set_ = now; + battery_valid_ = valid; + auto *p = vr::VRProperties(); + p->SetBoolProperty(container_, vr::Prop_DeviceProvidesBatteryStatus_Bool, valid); + if (!valid) + return; + p->SetFloatProperty(container_, vr::Prop_DeviceBatteryPercentage_Float, std::clamp(c.battery, 0.0f, 1.0f)); + p->SetBoolProperty(container_, vr::Prop_DeviceIsCharging_Bool, (c.flags & MVR_CONTROLLER_CHARGING) != 0); + } + + struct Components { + vr::VRInputComponentHandle_t primary_click = 0, primary_touch = 0, secondary_click = 0, secondary_touch = 0; + vr::VRInputComponentHandle_t system_click = 0, system_touch = 0, menu_click = 0; + vr::VRInputComponentHandle_t stick_click = 0, stick_touch = 0, stick_x = 0, stick_y = 0; + vr::VRInputComponentHandle_t trigger = 0, trigger_click = 0, trigger_touch = 0; + vr::VRInputComponentHandle_t grip = 0, grip_force = 0, grip_touch = 0, grip_click = 0, thumbrest_touch = 0; + vr::VRInputComponentHandle_t pad_x = 0, pad_y = 0, pad_click = 0, pad_touch = 0, pad_force = 0; + vr::VRInputComponentHandle_t finger[4] = {}; + }; + Link &link_; const uint32_t hand_; + uint32_t headset_ = MVR_HEADSET_UNKNOWN; + uint32_t kind_ = MVR_CONTROLLERS_UNKNOWN; std::atomic id_{vr::k_unTrackedDeviceIndexInvalid}; vr::PropertyContainerHandle_t container_ = 0; std::mutex mutex_; vr::DriverPose_t pose_{}; - vr::VRInputComponentHandle_t primary_click_ = 0, primary_touch_ = 0, secondary_click_ = 0, secondary_touch_ = 0; - vr::VRInputComponentHandle_t system_click_ = 0, system_touch_ = 0, stick_click_ = 0, stick_touch_ = 0; - vr::VRInputComponentHandle_t stick_x_ = 0, stick_y_ = 0, trigger_ = 0, trigger_touch_ = 0, grip_ = 0; - vr::VRInputComponentHandle_t grip_touch_ = 0, thumbrest_touch_ = 0, haptic_ = 0; + Components c_; + vr::VRInputComponentHandle_t skeleton_ = 0, haptic_ = 0; + bool battery_valid_ = false; + ULONGLONG battery_set_ = 0; }; // ---- the tracking system's sensors ---- @@ -753,25 +1021,27 @@ class Controller final : public TrackedDevice { // A sensor that stands in the room (a Rift sensor): SteamVR shows it as a base station where metalvrd says it is. class Reference final : public TrackedDevice { public: - explicit Reference(const char *serial) : serial_(serial) {} + // a Lighthouse headset's base station, or a Rift's sensor + Reference(const char *serial, bool lighthouse) : serial_(serial), lighthouse_(lighthouse) {} const std::string &serial() const { return serial_; } vr::EVRInitError Activate(uint32_t id) override { auto *p = vr::VRProperties(); const vr::PropertyContainerHandle_t c = p->TrackedDeviceToPropertyContainer(id); - p->SetStringProperty(c, vr::Prop_ModelNumber_String, "Oculus Rift CV1 Sensor"); - p->SetStringProperty(c, vr::Prop_ManufacturerName_String, "Oculus"); + p->SetStringProperty(c, vr::Prop_ModelNumber_String, lighthouse_ ? "Valve SR Imp" : "Oculus Rift CV1 Sensor"); + p->SetStringProperty(c, vr::Prop_ManufacturerName_String, lighthouse_ ? "Valve" : "Oculus"); p->SetStringProperty(c, vr::Prop_TrackingSystemName_String, "metalvr"); p->SetStringProperty(c, vr::Prop_SerialNumber_String, serial_.c_str()); - p->SetStringProperty(c, vr::Prop_RenderModelName_String, "rift_camera"); + p->SetStringProperty(c, vr::Prop_RenderModelName_String, lighthouse_ ? "lh_basestation_vive" : "rift_camera"); p->SetStringProperty(c, vr::Prop_ModeLabel_String, ""); - // what the sensor's camera takes in: about 100 by 70 degrees, from 0.4 to 3.5 m - p->SetFloatProperty(c, vr::Prop_FieldOfViewLeftDegrees_Float, 50.0f); - p->SetFloatProperty(c, vr::Prop_FieldOfViewRightDegrees_Float, 50.0f); - p->SetFloatProperty(c, vr::Prop_FieldOfViewTopDegrees_Float, 35.0f); - p->SetFloatProperty(c, vr::Prop_FieldOfViewBottomDegrees_Float, 35.0f); - p->SetFloatProperty(c, vr::Prop_TrackingRangeMinimumMeters_Float, 0.4f); - p->SetFloatProperty(c, vr::Prop_TrackingRangeMaximumMeters_Float, 3.5f); + // what it takes in: a Rift sensor's camera about 100 by 70 degrees, from 0.4 to 3.5 m; a base station sweeps + // about 150 by 110 degrees, to 5 m + p->SetFloatProperty(c, vr::Prop_FieldOfViewLeftDegrees_Float, lighthouse_ ? 75.0f : 50.0f); + p->SetFloatProperty(c, vr::Prop_FieldOfViewRightDegrees_Float, lighthouse_ ? 75.0f : 50.0f); + p->SetFloatProperty(c, vr::Prop_FieldOfViewTopDegrees_Float, lighthouse_ ? 55.0f : 35.0f); + p->SetFloatProperty(c, vr::Prop_FieldOfViewBottomDegrees_Float, lighthouse_ ? 55.0f : 35.0f); + p->SetFloatProperty(c, vr::Prop_TrackingRangeMinimumMeters_Float, lighthouse_ ? 0.2f : 0.4f); + p->SetFloatProperty(c, vr::Prop_TrackingRangeMaximumMeters_Float, lighthouse_ ? 5.0f : 3.5f); p->SetBoolProperty(c, vr::Prop_DeviceProvidesBatteryStatus_Bool, false); p->SetBoolProperty(c, vr::Prop_CanWirelessIdentify_Bool, false); p->SetBoolProperty(c, vr::Prop_WillDriftInYaw_Bool, false); @@ -817,6 +1087,7 @@ class Reference final : public TrackedDevice { private: const std::string serial_; + const bool lighthouse_; std::atomic id_{vr::k_unTrackedDeviceIndexInvalid}; std::mutex mutex_; vr::DriverPose_t pose_{}; @@ -834,9 +1105,9 @@ class Headset final : public TrackedDevice { vr::PropertyContainerHandle_t c = vr::VRProperties()->TrackedDeviceToPropertyContainer(id); auto *p = vr::VRProperties(); const char *name = system.system_name[0] ? system.system_name : "MetalVR headset"; + headset_ = system.headset; p->SetStringProperty(c, vr::Prop_ModelNumber_String, name); - p->SetStringProperty(c, vr::Prop_ManufacturerName_String, - std::strstr(name, "Rift") ? "Oculus" : "MetalVR"); + p->SetStringProperty(c, vr::Prop_ManufacturerName_String, mvr_describe_headset(system.headset)->manufacturer); p->SetStringProperty(c, vr::Prop_TrackingSystemName_String, "metalvr"); p->SetStringProperty(c, vr::Prop_SerialNumber_String, "MetalVR-HMD"); p->SetStringProperty(c, vr::Prop_RenderModelName_String, "generic_hmd"); @@ -850,6 +1121,7 @@ class Headset final : public TrackedDevice { p->SetBoolProperty(c, vr::Prop_DriverDirectModeSendsVsyncEvents_Bool, false); p->SetBoolProperty(c, vr::Prop_DeviceProvidesBatteryStatus_Bool, false); p->SetBoolProperty(c, vr::Prop_DeviceCanPowerOff_Bool, false); + container_ = c; p->SetUint64Property(c, vr::Prop_CurrentUniverseId_Uint64, 0x4d56); // "MV": one fixed play space p->SetBoolProperty(c, vr::Prop_DisplayDebugMode_Bool, false); p->SetBoolProperty(c, vr::Prop_NeverTracked_Bool, false); @@ -904,7 +1176,7 @@ class Headset final : public TrackedDevice { pose.deviceIsConnected = (sample.flags & MVR_TRACKING_HMD_CONNECTED) != 0; pose.poseIsValid = valid; pose.result = valid ? vr::TrackingResult_Running_OK : vr::TrackingResult_Running_OutOfRange; - pose.willDriftInYaw = true; + pose.willDriftInYaw = driftsInYaw(headset_); pose.shouldApplyHeadModel = false; pose.qRotation = {sample.head.orientation.w, sample.head.orientation.x, sample.head.orientation.y, sample.head.orientation.z}; @@ -931,6 +1203,7 @@ class Headset final : public TrackedDevice { mvr_tracking_response sample{}; const vr::DriverPose_t head = pose(&sample); vr::VRServerDriverHost()->TrackedDevicePoseUpdated(id_, head, sizeof(vr::DriverPose_t)); + updateBattery(sample); if (head.poseIsValid != last_valid_ || head.deviceIsConnected != last_connected_) { last_valid_ = head.poseIsValid; last_connected_ = head.deviceIsConnected; @@ -946,7 +1219,27 @@ class Headset final : public TrackedDevice { } } + // the headset's charge, when it reports one, a few times a minute + void updateBattery(const mvr_tracking_response &sample) { + const bool valid = (sample.flags & MVR_TRACKING_BATTERY_VALID) != 0; + const ULONGLONG now = GetTickCount64(); + if (valid == battery_valid_ && now - battery_set_ < 10000) + return; + battery_set_ = now; + battery_valid_ = valid; + auto *p = vr::VRProperties(); + p->SetBoolProperty(container_, vr::Prop_DeviceProvidesBatteryStatus_Bool, valid); + if (!valid) + return; + p->SetFloatProperty(container_, vr::Prop_DeviceBatteryPercentage_Float, std::clamp(sample.head_battery, 0.0f, 1.0f)); + p->SetBoolProperty(container_, vr::Prop_DeviceIsCharging_Bool, (sample.flags & MVR_TRACKING_CHARGING) != 0); + } + bool last_valid_ = false, last_connected_ = false; + bool battery_valid_ = false; + ULONGLONG battery_set_ = 0; + uint32_t headset_ = MVR_HEADSET_UNKNOWN; + vr::PropertyContainerHandle_t container_ = 0; public: Controller *controllers_[2] = {}; @@ -1038,7 +1331,11 @@ class Provider final : public vr::IServerTrackedDeviceProvider { for (auto &slot : references_) { if (slot) continue; - slot = std::make_unique(r.serial); + const uint32_t headset = link_.system().headset; + slot = std::make_unique(r.serial, headset == MVR_HEADSET_VALVE_INDEX || + headset == MVR_HEADSET_VIVE || + headset == MVR_HEADSET_VIVE_PRO || + headset == MVR_HEADSET_VIVE_PRO_2); if (vr::VRServerDriverHost()->TrackedDeviceAdded(r.serial, vr::TrackedDeviceClass_TrackingReference, slot->asDriver())) { log("metalvr: tracking sensor %s", r.serial); diff --git a/steamvr/hand_skeleton.h b/steamvr/hand_skeleton.h new file mode 100644 index 0000000..7153d7a --- /dev/null +++ b/steamvr/hand_skeleton.h @@ -0,0 +1,153 @@ +// Two poses of each hand for SteamVR's skeletal input, open and closed around a controller's handle, for +// controllers that sense the fingers (Index). A finger's bones are blended between the two by its curl. +// +// The poses are ALVR's (server_openvr/cpp/alvr_server/Controller.cpp; MIT, Copyright (c) 2018-2019 polygraphene, +// (c) 2020-2024 alvr-org, lib/metalvr/LICENSE.ALVR): open is its pose with nothing touched, closed is its pose with +// the thumb resting, the trigger pulled and the grip squeezed, with the controller in the hand. +#pragma once + +namespace metalvr::skeleton { + +struct Bone { + float position[3]; + float orientation[4]; // w, x, y, z +}; + +// SteamVR's hand skeleton: root, wrist, then thumb 2-5, index 6-10, middle 11-15, ring 16-20, pinky 21-25, and +// the aux bones 26 (thumb) to 30 (pinky) +constexpr int kBoneCount = 31; + +const Bone kLeftOpen[31] = { + {{0.000000f, 0.000000f, 0.000000f}, {1.000000f, 0.000000f, 0.000000f, 0.000000f}}, + {{-0.034038f, 0.036503f, 0.164722f}, {-0.055147f, -0.078608f, -0.920279f, 0.379296f}}, + {{-0.012083f, 0.028070f, 0.025050f}, {0.464112f, 0.567418f, 0.272106f, 0.623374f}}, + {{0.040406f, 0.000000f, -0.000000f}, {0.994838f, 0.082939f, 0.019454f, 0.055130f}}, + {{0.032517f, 0.000000f, 0.000000f}, {0.974793f, -0.003213f, 0.021867f, -0.222015f}}, + {{0.030464f, -0.000000f, -0.000000f}, {1.000000f, -0.000000f, 0.000000f, 0.000000f}}, + {{0.000632f, 0.026866f, 0.015002f}, {0.644251f, 0.421979f, -0.478202f, 0.422133f}}, + {{0.074204f, -0.005002f, 0.000234f}, {0.995332f, 0.007007f, -0.039124f, 0.087949f}}, + {{0.043930f, -0.000000f, -0.000000f}, {0.997891f, 0.045808f, 0.002142f, -0.045943f}}, + {{0.028695f, 0.000000f, 0.000000f}, {0.999649f, 0.001850f, -0.022782f, -0.013409f}}, + {{0.022821f, 0.000000f, -0.000000f}, {1.000000f, 0.000000f, -0.000000f, 0.000000f}}, + {{0.002177f, 0.007120f, 0.016319f}, {0.546723f, 0.541277f, -0.442520f, 0.460749f}}, + {{0.070953f, 0.000779f, 0.000997f}, {0.980294f, -0.167261f, -0.078959f, 0.069368f}}, + {{0.043108f, 0.000000f, 0.000000f}, {0.997947f, 0.018493f, 0.013192f, 0.059886f}}, + {{0.033266f, 0.000000f, 0.000000f}, {0.997394f, -0.003328f, -0.028225f, -0.066315f}}, + {{0.025892f, -0.000000f, 0.000000f}, {0.999195f, 0.000000f, 0.000000f, 0.040126f}}, + {{0.000513f, -0.006545f, 0.016348f}, {0.516692f, 0.550144f, -0.495548f, 0.429888f}}, + {{0.065876f, 0.001786f, 0.000693f}, {0.990420f, -0.058696f, -0.101820f, 0.072495f}}, + {{0.040697f, 0.000000f, 0.000000f}, {0.999545f, -0.002240f, 0.000004f, 0.030081f}}, + {{0.028747f, -0.000000f, -0.000000f}, {0.999102f, -0.000721f, -0.012693f, 0.040420f}}, + {{0.022430f, -0.000000f, 0.000000f}, {1.000000f, 0.000000f, 0.000000f, 0.000000f}}, + {{-0.002478f, -0.018981f, 0.015214f}, {0.526918f, 0.523940f, -0.584025f, 0.326740f}}, + {{0.062878f, 0.002844f, 0.000332f}, {0.986609f, -0.059615f, -0.135163f, 0.069132f}}, + {{0.030220f, 0.000000f, 0.000000f}, {0.994317f, 0.001896f, -0.000132f, 0.106446f}}, + {{0.018187f, 0.000000f, 0.000000f}, {0.995931f, -0.002010f, -0.052079f, -0.073526f}}, + {{0.018018f, 0.000000f, -0.000000f}, {1.000000f, -0.000000f, -0.000000f, -0.000000f}}, + {{-0.006059f, 0.056285f, 0.060064f}, {0.737238f, 0.202745f, 0.594267f, 0.249441f}}, + {{-0.040416f, -0.043018f, 0.019345f}, {-0.290330f, 0.623527f, -0.663809f, -0.293734f}}, + {{-0.039354f, -0.075674f, 0.047048f}, {-0.187047f, 0.678062f, -0.659285f, -0.265683f}}, + {{-0.038340f, -0.090987f, 0.082579f}, {-0.183037f, 0.736793f, -0.634757f, -0.143936f}}, + {{-0.031806f, -0.087214f, 0.121015f}, {-0.003659f, 0.758407f, -0.639342f, -0.126678f}}, +}; +const Bone kLeftClosed[31] = { + {{0.000000f, 0.000000f, 0.000000f}, {1.000000f, 0.000000f, 0.000000f, 0.000000f}}, + {{-0.034038f, 0.036503f, 0.164722f}, {-0.055147f, -0.078608f, -0.920279f, 0.379296f}}, + {{-0.017303f, 0.032567f, 0.025281f}, {0.317609f, 0.528344f, 0.213134f, 0.757991f}}, + {{0.040406f, 0.000000f, -0.000000f}, {0.991742f, 0.085317f, 0.019416f, 0.093765f}}, + {{0.032517f, -0.000000f, 0.000000f}, {0.959385f, -0.012202f, -0.031055f, 0.280120f}}, + {{0.030464f, -0.000000f, -0.000000f}, {1.000000f, -0.000000f, 0.000000f, 0.000000f}}, + {{-0.003925f, 0.027171f, 0.014640f}, {0.666448f, 0.430031f, -0.455947f, 0.403772f}}, + {{0.076015f, -0.005124f, 0.000239f}, {-0.956011f, -0.000025f, 0.158355f, -0.246913f}}, + {{0.043930f, -0.000000f, -0.000000f}, {-0.944138f, -0.043351f, 0.014947f, -0.326345f}}, + {{0.028695f, 0.000000f, 0.000000f}, {-0.912149f, 0.003626f, 0.039888f, -0.407898f}}, + {{0.022821f, 0.000000f, -0.000000f}, {1.000000f, -0.000000f, -0.000000f, 0.000000f}}, + {{0.002177f, 0.007120f, 0.016319f}, {0.529359f, 0.540512f, -0.463783f, 0.461011f}}, + {{0.070953f, 0.000779f, 0.000997f}, {-0.831727f, 0.270927f, 0.175647f, -0.451638f}}, + {{0.043108f, 0.000000f, 0.000000f}, {-0.854886f, -0.008231f, -0.028107f, -0.517990f}}, + {{0.033266f, -0.000000f, 0.000000f}, {-0.825759f, 0.085208f, 0.086456f, -0.550805f}}, + {{0.025892f, -0.000000f, 0.000000f}, {0.999195f, -0.000000f, 0.000000f, 0.040126f}}, + {{0.000513f, -0.006545f, 0.016348f}, {0.500244f, 0.530784f, -0.516215f, 0.448939f}}, + {{0.065876f, 0.001786f, 0.000693f}, {0.831617f, -0.242931f, -0.139695f, 0.479461f}}, + {{0.040697f, 0.000000f, 0.000000f}, {0.769163f, -0.001746f, 0.001363f, 0.639049f}}, + {{0.028747f, -0.000000f, -0.000000f}, {0.968615f, -0.064537f, -0.046586f, 0.235477f}}, + {{0.022430f, -0.000000f, 0.000000f}, {1.000000f, 0.000000f, -0.000000f, -0.000000f}}, + {{-0.002478f, -0.018981f, 0.015214f}, {0.474671f, 0.434670f, -0.653212f, 0.398827f}}, + {{0.062878f, 0.002844f, 0.000332f}, {0.798788f, -0.199577f, -0.094418f, 0.559636f}}, + {{0.030220f, 0.000002f, -0.000000f}, {0.853087f, 0.001644f, -0.000913f, 0.521765f}}, + {{0.018187f, -0.000002f, 0.000000f}, {0.974249f, 0.052491f, 0.003591f, 0.219249f}}, + {{0.018018f, 0.000000f, -0.000000f}, {1.000000f, 0.000000f, 0.000000f, 0.000000f}}, + {{0.006629f, 0.026690f, 0.061870f}, {0.805084f, -0.018369f, 0.584788f, -0.097597f}}, + {{-0.007882f, -0.040478f, 0.039337f}, {-0.322494f, 0.932092f, 0.121861f, 0.111140f}}, + {{0.016642f, -0.029992f, 0.083200f}, {-0.094577f, 0.694550f, 0.702845f, 0.121100f}}, + {{0.011144f, -0.028727f, 0.108366f}, {-0.076328f, 0.788280f, 0.605097f, 0.081527f}}, + {{0.011333f, -0.026044f, 0.128585f}, {-0.144791f, 0.737451f, 0.656958f, -0.060069f}}, +}; +const Bone kRightOpen[31] = { + {{0.000000f, 0.000000f, 0.000000f}, {1.000000f, 0.000000f, 0.000000f, 0.000000f}}, + {{0.034038f, 0.036503f, 0.164722f}, {-0.055147f, -0.078608f, 0.920279f, -0.379296f}}, + {{0.012330f, 0.028661f, 0.025049f}, {0.571059f, -0.451277f, 0.630056f, -0.270685f}}, + {{-0.040406f, -0.000000f, 0.000000f}, {0.994565f, 0.078280f, 0.018282f, 0.066177f}}, + {{-0.032517f, -0.000000f, -0.000000f}, {0.977658f, -0.003039f, 0.020722f, -0.209156f}}, + {{-0.030464f, 0.000000f, 0.000000f}, {1.000000f, -0.000000f, 0.000000f, 0.000000f}}, + {{-0.000632f, 0.026866f, 0.015002f}, {0.421833f, -0.643793f, 0.422458f, 0.478661f}}, + {{-0.074204f, 0.005002f, -0.000234f}, {0.994784f, 0.007053f, -0.041286f, 0.093009f}}, + {{-0.043930f, 0.000000f, 0.000000f}, {0.998404f, 0.045905f, 0.002780f, -0.032767f}}, + {{-0.028695f, -0.000000f, -0.000000f}, {0.999704f, 0.001955f, -0.022774f, -0.008282f}}, + {{-0.022821f, -0.000000f, 0.000000f}, {1.000000f, 0.000000f, -0.000000f, 0.000000f}}, + {{-0.002177f, 0.007120f, 0.016319f}, {0.541874f, -0.547427f, 0.459996f, 0.441701f}}, + {{-0.070953f, -0.000779f, -0.000997f}, {0.979837f, -0.168061f, -0.075910f, 0.076899f}}, + {{-0.043108f, -0.000000f, -0.000000f}, {0.997271f, 0.018278f, 0.013375f, 0.070266f}}, + {{-0.033266f, -0.000000f, -0.000000f}, {0.998402f, -0.003143f, -0.026423f, -0.049849f}}, + {{-0.025892f, 0.000000f, -0.000000f}, {0.999195f, 0.000000f, 0.000000f, 0.040126f}}, + {{-0.000513f, -0.006545f, 0.016348f}, {0.548983f, -0.519068f, 0.426914f, 0.496920f}}, + {{-0.065876f, -0.001786f, -0.000693f}, {0.989791f, -0.065882f, -0.096417f, 0.081716f}}, + {{-0.040697f, -0.000000f, -0.000000f}, {0.999102f, -0.002168f, -0.000020f, 0.042317f}}, + {{-0.028747f, 0.000000f, 0.000000f}, {0.998584f, -0.000674f, -0.012714f, 0.051653f}}, + {{-0.022430f, 0.000000f, -0.000000f}, {1.000000f, 0.000000f, 0.000000f, 0.000000f}}, + {{0.002478f, -0.018981f, 0.015214f}, {0.518597f, -0.527304f, 0.328264f, 0.587580f}}, + {{-0.062878f, -0.002844f, -0.000332f}, {0.987294f, -0.063356f, -0.125964f, 0.073274f}}, + {{-0.030220f, -0.000000f, -0.000000f}, {0.993413f, 0.001573f, -0.000147f, 0.114578f}}, + {{-0.018187f, -0.000000f, -0.000000f}, {0.997047f, -0.000695f, -0.052009f, -0.056495f}}, + {{-0.018018f, -0.000000f, 0.000000f}, {1.000000f, -0.000000f, -0.000000f, -0.000000f}}, + {{0.005198f, 0.054204f, 0.060030f}, {0.747318f, 0.182508f, -0.599586f, -0.220688f}}, + {{0.038779f, -0.042973f, 0.019824f}, {-0.297445f, 0.639373f, 0.648910f, 0.285734f}}, + {{0.038027f, -0.074844f, 0.046941f}, {-0.199898f, 0.698218f, 0.635767f, 0.261406f}}, + {{0.036845f, -0.089781f, 0.081973f}, {-0.190960f, 0.756469f, 0.607591f, 0.148733f}}, + {{0.030251f, -0.086056f, 0.119887f}, {-0.018948f, 0.779249f, 0.612180f, 0.132846f}}, +}; +const Bone kRightClosed[31] = { + {{0.000000f, 0.000000f, 0.000000f}, {1.000000f, 0.000000f, 0.000000f, 0.000000f}}, + {{0.034038f, 0.036503f, 0.164722f}, {-0.055147f, -0.078608f, 0.920279f, -0.379296f}}, + {{0.017303f, 0.032567f, 0.025281f}, {0.528344f, -0.317609f, 0.757991f, -0.213134f}}, + {{-0.040406f, -0.000000f, 0.000000f}, {0.991742f, 0.085317f, 0.019416f, 0.093765f}}, + {{-0.032517f, 0.000000f, -0.000000f}, {0.959385f, -0.012202f, -0.031055f, 0.280120f}}, + {{-0.030464f, 0.000000f, 0.000000f}, {1.000000f, -0.000000f, 0.000000f, 0.000000f}}, + {{-0.003925f, 0.027171f, 0.014640f}, {0.666448f, 0.430031f, -0.455947f, 0.403772f}}, + {{0.076015f, -0.005124f, 0.000239f}, {-0.956011f, -0.000025f, 0.158355f, -0.246913f}}, + {{0.043930f, -0.000000f, -0.000000f}, {-0.944138f, -0.043351f, 0.014947f, -0.326345f}}, + {{0.028695f, 0.000000f, 0.000000f}, {-0.912149f, 0.003626f, 0.039888f, -0.407898f}}, + {{0.022821f, 0.000000f, -0.000000f}, {1.000000f, -0.000000f, -0.000000f, 0.000000f}}, + {{0.002177f, 0.007120f, 0.016319f}, {0.529359f, 0.540512f, -0.463783f, 0.461011f}}, + {{0.070953f, 0.000779f, 0.000997f}, {-0.831727f, 0.270927f, 0.175647f, -0.451638f}}, + {{0.043108f, 0.000000f, 0.000000f}, {-0.854886f, -0.008231f, -0.028107f, -0.517990f}}, + {{0.033266f, -0.000000f, 0.000000f}, {-0.825759f, 0.085208f, 0.086456f, -0.550805f}}, + {{0.025892f, -0.000000f, 0.000000f}, {0.999195f, -0.000000f, 0.000000f, 0.040126f}}, + {{0.000513f, -0.006545f, 0.016348f}, {0.500244f, 0.530784f, -0.516215f, 0.448939f}}, + {{0.065876f, 0.001786f, 0.000693f}, {0.831617f, -0.242931f, -0.139695f, 0.479461f}}, + {{0.040697f, 0.000000f, 0.000000f}, {0.769163f, -0.001746f, 0.001363f, 0.639049f}}, + {{0.028747f, -0.000000f, -0.000000f}, {0.968615f, -0.064537f, -0.046586f, 0.235477f}}, + {{0.022430f, -0.000000f, 0.000000f}, {1.000000f, 0.000000f, -0.000000f, -0.000000f}}, + {{-0.002478f, -0.018981f, 0.015214f}, {0.474671f, 0.434670f, -0.653212f, 0.398827f}}, + {{0.062878f, 0.002844f, 0.000332f}, {0.798788f, -0.199577f, -0.094418f, 0.559636f}}, + {{0.030220f, 0.000002f, -0.000000f}, {0.853087f, 0.001644f, -0.000913f, 0.521765f}}, + {{0.018187f, -0.000002f, 0.000000f}, {0.974249f, 0.052491f, 0.003591f, 0.219249f}}, + {{0.018018f, 0.000000f, -0.000000f}, {1.000000f, 0.000000f, 0.000000f, 0.000000f}}, + {{0.006629f, 0.026690f, 0.061870f}, {0.805084f, -0.018369f, 0.584788f, -0.097597f}}, + {{-0.007882f, -0.040478f, 0.039337f}, {-0.322494f, 0.932092f, 0.121861f, 0.111140f}}, + {{0.016642f, -0.029992f, 0.083200f}, {-0.094577f, 0.694550f, 0.702845f, 0.121100f}}, + {{0.011144f, -0.028727f, 0.108366f}, {-0.076328f, 0.788280f, 0.605097f, 0.081527f}}, + {{0.011333f, -0.026044f, 0.128585f}, {-0.144791f, 0.737451f, 0.656958f, -0.060069f}}, +}; + +} // namespace metalvr::skeleton diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index fad367e..7902940 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -23,6 +23,23 @@ elseif(APPLE) target_link_libraries(frame_check PRIVATE "-framework Foundation" "-framework CoreGraphics" "-framework ImageIO") + # what MetalVR reads from and sends to the headsets it drives itself, without a headset (docs/HEADSETS.md) + add_executable(headset_protocol_check headset_protocol_check.cpp + "${CMAKE_SOURCE_DIR}/daemon/headsets.cpp" "${CMAKE_SOURCE_DIR}/daemon/json_value.cpp" + "${CMAKE_SOURCE_DIR}/daemon/foveation.cpp" + "${CMAKE_SOURCE_DIR}/daemon/lighthouse_protocol.cpp" "${CMAKE_SOURCE_DIR}/daemon/rift_s_protocol.cpp" + "${CMAKE_SOURCE_DIR}/daemon/psvr2_protocol.cpp") + target_include_directories(headset_protocol_check PRIVATE "${CMAKE_SOURCE_DIR}/daemon" "${CMAKE_SOURCE_DIR}/protocol") + target_link_libraries(headset_protocol_check PRIVATE z) + + # the Rift S's camera tracking end to end on a synthetic headset; Basalt is loaded when it runs + add_executable(rift_s_tracking_check rift_s_tracking_check.cpp + "${CMAKE_SOURCE_DIR}/daemon/rift_s_tracking.cpp" "${CMAKE_SOURCE_DIR}/daemon/camera_tracking.cpp" + "${CMAKE_SOURCE_DIR}/daemon/rift_s_protocol.cpp" "${CMAKE_SOURCE_DIR}/daemon/json_value.cpp") + target_include_directories(rift_s_tracking_check PRIVATE "${CMAKE_SOURCE_DIR}/daemon" "${CMAKE_SOURCE_DIR}/protocol") + target_compile_definitions(rift_s_tracking_check PRIVATE + "METALVR_BASALT_DIR=\"${CMAKE_SOURCE_DIR}/build/basalt/prefix/lib\"") + if(TARGET rift_tracking) # the Rift sensors by themselves: opens them and runs their cameras add_executable(rift_tracking_smoke rift_tracking_smoke.c) @@ -31,7 +48,8 @@ elseif(APPLE) if(METALVR_WITH_ALVR) # the ALVR driver by itself: starts it, reads its defaults, stops it, starts it again - add_executable(alvr_driver_smoke alvr_driver_smoke.cpp "${CMAKE_SOURCE_DIR}/daemon/driver_alvr.cpp") + add_executable(alvr_driver_smoke alvr_driver_smoke.cpp "${CMAKE_SOURCE_DIR}/daemon/driver_alvr.cpp" + "${CMAKE_SOURCE_DIR}/daemon/headsets.cpp" "${CMAKE_SOURCE_DIR}/daemon/foveation.cpp") target_include_directories(alvr_driver_smoke PRIVATE "${CMAKE_SOURCE_DIR}/daemon" "${CMAKE_SOURCE_DIR}/protocol" "${METALVR_ALVR_INCLUDE_DIR}") target_link_libraries(alvr_driver_smoke PRIVATE "${METALVR_ALVR_LIBRARY}") diff --git a/tests/alvr_driver_smoke.cpp b/tests/alvr_driver_smoke.cpp index 02ec245..5a5e1b3 100644 --- a/tests/alvr_driver_smoke.cpp +++ b/tests/alvr_driver_smoke.cpp @@ -4,7 +4,11 @@ // // With a headset running the ALVR client on the same network, and its hostname given, the same program shows the // connection, the negotiated stream and live tracking; it sends no video. +// +// METALVR_EXPECT_HEADSET= (quest-3, unknown, ...) checks which headset the driver names before any headset +// connects: the model ALVR's session remembers its trusted client reporting (scripts/test-alvr-models.sh). #include "driver.h" +#include "headsets.h" #include #include @@ -28,6 +32,10 @@ void print(metalvr::Driver &driver, double elapsed) { static_cast(driver.generation())); std::printf(" system: \"%s\" flags=0x%x eye=%ux%u refresh=%.1fHz ipd=%.4fm\n", system.system_name, system.flags, system.eye_width, system.eye_height, system.refresh_hz, system.ipd); + std::printf(" headset=%s controllers=%u rates:", mvr_describe_headset(system.headset)->key, system.controllers); + for (uint32_t i = 0; i < system.refresh_rate_count && i < MVR_MAX_REFRESH_RATES; i++) + std::printf(" %.0f", system.refresh_rates[i]); + std::printf("\n"); for (int eye = 0; eye < 2; eye++) std::printf(" fov[%d]: left=%.3f right=%.3f up=%.3f down=%.3f rad\n", eye, system.fov[eye].left, system.fov[eye].right, system.fov[eye].up, system.fov[eye].down); @@ -84,6 +92,20 @@ int main(int argc, char **argv) { } std::printf("started\n"); + if (const char *key = std::getenv("METALVR_EXPECT_HEADSET")) { + const mvr_system_info system = driver->systemInfo(); + const uint32_t wanted = std::string(key) == "unknown" ? MVR_HEADSET_UNKNOWN : metalvr::headsetFromKey(key); + std::printf("headset before a connection: %s (expected %s)\n", mvr_describe_headset(system.headset)->key, key); + expect(system.headset == wanted, "the driver names the headset its client last reported"); + expect(system.controllers != MVR_CONTROLLERS_UNKNOWN, "ALVR's controllers are of a known kind"); + expect(std::string(system.system_name).find("Rift") == std::string::npos, "an ALVR headset is never a Rift"); + expect(system.refresh_rate_count >= 1, "the headset's refresh rates are listed"); + bool listed = false; + for (uint32_t i = 0; i < system.refresh_rate_count; i++) + listed = listed || system.refresh_rates[i] == system.refresh_hz; + expect(listed, "the refresh rate is among the listed ones"); + } + // every headset asked for is trusted in ALVR's session, under its plain name however it was typed; nobody else { std::ifstream file(options.config_dir + "/session.json"); diff --git a/tests/headset_protocol_check.cpp b/tests/headset_protocol_check.cpp new file mode 100644 index 0000000..0d9229a --- /dev/null +++ b/tests/headset_protocol_check.cpp @@ -0,0 +1,760 @@ +// Checks, without a headset, what MetalVR reads from and sends to the headsets it drives directly (docs/HEADSETS.md): +// the Lighthouse headsets' and controllers' reports and configuration, the Rift S's reports, the JSON reader both +// use, and the table of headsets the server describes. +// +// headset_protocol_check prints each check; exits 1 if any fails +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "foveation.h" +#include "headsets.h" +#include "json_value.h" +#include "lighthouse_protocol.h" +#include "psvr2_protocol.h" +#include "rift_s_protocol.h" + +using namespace metalvr; + +namespace { + +int g_failures = 0; +int g_checks = 0; + +void check(bool ok, const char *what) { + ++g_checks; + if (!ok) { + ++g_failures; + std::printf("FAIL: %s\n", what); + } +} + +bool near(float a, float b, float eps = 1e-4f) { return std::fabs(a - b) <= eps; } +uint32_t le32(const uint8_t *p) { return p[0] | (p[1] << 8) | (p[2] << 16) | (static_cast(p[3]) << 24); } + +void put16(std::vector &v, size_t at, int value) { + v[at] = static_cast(value & 0xff); + v[at + 1] = static_cast((value >> 8) & 0xff); +} +void put32(std::vector &v, size_t at, uint32_t value) { + for (int i = 0; i < 4; i++) v[at + i] = static_cast(value >> (8 * i)); +} +void putf(std::vector &v, size_t at, float f) { + uint32_t bits; + std::memcpy(&bits, &f, 4); + put32(v, at, bits); +} + +// ---- JSON ---- + +void json() { + const auto doc = JsonValue::parse(R"( {"a": [1, 2.5, -3e2], "b": {"c": "x\"yé"}, "d": true, "e": null} )"); + check(doc && doc->isObject(), "json: an object parses"); + if (!doc) return; + float a[3]; + check((*doc)["a"].numbers(a, 3) && a[0] == 1 && a[1] == 2.5f && a[2] == -300.0f, "json: numbers"); + check((*doc)["b"]["c"].string() == "x\"y\xc3\xa9", "json: escapes"); + check((*doc)["d"].boolean() && (*doc)["e"].type() == JsonValue::Type::Null, "json: true and null"); + check((*doc)["missing"]["deeper"].type() == JsonValue::Type::Null, "json: a missing member is null"); + check(!JsonValue::parse("{\"a\": 1,}"), "json: a trailing comma is refused"); + check(!JsonValue::parse("{\"a\": 1} x"), "json: text after the document is refused"); + check(!JsonValue::parse("[1, 2"), "json: an unclosed array is refused"); + std::string deep(200, '['); + check(!JsonValue::parse(deep), "json: nesting is bounded"); + check(JsonValue::parse(std::string("{}\0\0", 4)).has_value(), "json: a firmware block's trailing NULs are allowed"); +} + +// ---- Lighthouse ---- + +// a headset configuration as the Index's firmware has it, cut down to what MetalVR reads +const char *kIndexConfig = R"({ + "model_number": "Index", + "device_serial_number": "LHR-TEST0001", + "imu": {"acc_bias": [0.1, -0.2, 0.3], "acc_scale": [1.0, 1.01, 0.99], "gyro_bias": [0.001, 0.002, -0.003], + "gyro_scale": [1, 1, 1], "plus_x": [1,0,0], "plus_z": [0,0,1], "position": [0,0,0]}, + "device": {"eye_target_width_in_pixels": 1440, "eye_target_height_in_pixels": 1600, + "physical_aspect_x_over_y": 0.9}, + "tracking_to_eye_transform": [ + {"grow_for_undistort": 0.6, "undistort_r2_cutoff": 1.1, + "intrinsics": [[0.92, 0, -0.08], [0, 1.02, 0.0], [0, 0, -1]], + "distortion": {"center_x": -0.04, "center_y": 0.01, "coeffs": [0.15, 0.1, 0.02, 0]}, + "distortion_red": {"center_x": -0.04, "center_y": 0.01, "coeffs": [0.14, 0.1, 0.02, 0]}, + "distortion_blue": {"center_x": -0.04, "center_y": 0.01, "coeffs": [0.17, 0.1, 0.02, 0]}}, + {"grow_for_undistort": 0.6, "undistort_r2_cutoff": 1.1, + "intrinsics": [[0.92, 0, 0.08], [0, 1.02, 0.0], [0, 0, -1]], + "distortion": {"center_x": 0.04, "center_y": 0.01, "coeffs": [0.15, 0.1, 0.02, 0]}, + "distortion_red": {"center_x": 0.04, "center_y": 0.01, "coeffs": [0.14, 0.1, 0.02, 0]}, + "distortion_blue": {"center_x": 0.04, "center_y": 0.01, "coeffs": [0.17, 0.1, 0.02, 0]}} + ] +})"; + +std::vector deflate(const std::string &text) { + uLongf size = compressBound(text.size()); + std::vector out(size); + compress2(out.data(), &size, reinterpret_cast(text.data()), text.size(), 9); + out.resize(size); + return out; +} + +void lighthouseIdentify() { + using namespace lighthouse; + const UsbId vive_board{kHtcVendor, kViveMainboard}, vive_imu{kValveVendor, kViveSensors}; + const UsbId pro_board{kHtcVendor, kViveProMainboard}, pro_board_b{kHtcVendor, kViveProMainboardB}; + const UsbId pro2_board{kHtcVendor, kVivePro2Mainboard}, lhr{kValveVendor, kLhrSensors}; + const UsbId rift{0x2833, 0x0031}, rift_s{0x2833, 0x0051}, dongle{kValveVendor, kWatchmanGen2}; + check(identify({vive_board, vive_imu}) == Model::Vive, "identify: an HTC Vive"); + check(identify({vive_board}) == Model::None, "identify: a Vive mainboard without its IMU device is nothing"); + check(identify({pro_board, lhr}) == Model::VivePro, "identify: a Vive Pro (0309)"); + check(identify({pro_board_b, lhr}) == Model::VivePro, "identify: a Vive Pro (030e)"); + check(identify({pro2_board, lhr, dongle}) == Model::VivePro2, "identify: a Vive Pro 2"); + check(identify({lhr, dongle, dongle}) == Model::Index, "identify: 2300 alone is an Index, to be confirmed"); + check(identify({rift}) == Model::None && identify({rift_s}) == Model::None, "identify: a Rift is never a Lighthouse"); + check(identify({}) == Model::None, "identify: nothing is nothing"); + check(modelFromConfig("Index") == Model::Index && modelFromConfig("Utah MP") == Model::Index, "config: Index"); + check(modelFromConfig("Vive MV") == Model::Vive && modelFromConfig("Vive_Pro MV") == Model::VivePro && + modelFromConfig("VIVE_Pro 2 MV") == Model::VivePro2, + "config: the Vive models"); + check(modelFromConfig("Knuckles Right") == Model::None, "config: a controller is no headset"); + check(headsetOf(Model::Index) == MVR_HEADSET_VALVE_INDEX && headsetOf(Model::None) == MVR_HEADSET_UNKNOWN, + "headsetOf"); +} + +void lighthouseConfig() { + using namespace lighthouse; + // the configuration arrives as 0x11 reports of up to 62 bytes, the last one empty + const std::vector compressed = deflate(kIndexConfig); + ConfigAssembler assembler; + bool ok = true; + for (size_t at = 0; at < compressed.size(); at += 62) { + const size_t n = std::min(62, compressed.size() - at); + uint8_t report[64] = {kConfigReadReport, static_cast(n)}; + std::memcpy(report + 2, compressed.data() + at, n); + ok = ok && assembler.add(report, sizeof(report)); + } + check(ok && !assembler.done(), "config: the chunks are taken"); + const uint8_t last[64] = {kConfigReadReport, 0}; + check(assembler.add(last, sizeof(last)) && assembler.done(), "config: an empty report ends it"); + const auto json = assembler.json(); + check(json && *json == kIndexConfig, "config: it inflates to the JSON that was deflated"); + const uint8_t bad[64] = {kConfigReadReport, 63}; + ConfigAssembler other; + check(!other.add(bad, sizeof(bad)), "config: a chunk longer than 62 bytes is refused"); + + const auto config = parseHeadsetConfig(kIndexConfig); + check(config.has_value(), "config: an Index configuration parses"); + if (!config) return; + check(config->model == Model::Index && config->serial == "LHR-TEST0001", "config: model and serial"); + check(config->eye_width == 1440 && config->eye_height == 1600, "config: the panel's size"); + check(near(config->acc_bias.y, -0.2f) && near(config->acc_scale.y, 1.01f), "config: IMU calibration"); + check(near(config->distortion[0].coefficients[2][0], 0.17f) && near(config->distortion[1].center[1][0], 0.04f), + "config: distortion coefficients and centers, per colour"); + // Monado's _calculate_fov with these intrinsics + const float scale = 1.6f; + const float fx = 0.92f / scale * 720.0f, fy = 1.02f / scale * 800.0f; + const float cx = (-0.08f / scale - 1.0f) * -720.0f, cy = (0.0f / scale + 1.0f) * 800.0f; + check(near(config->fov[0].left, -std::atan(cx / fx)) && near(config->fov[0].right, std::atan((1440 - cx) / fx)) && + near(config->fov[0].up, std::atan(cy / fy)) && near(config->fov[0].down, -std::atan((1600 - cy) / fy)), + "config: the field of view from the intrinsics"); + check(config->fov[0].left < 0 && config->fov[0].right > 0 && near(config->fov[0].left, -config->fov[1].right), + "config: the eyes' fields of view mirror each other"); + check(!parseHeadsetConfig(R"({"model_number": "Knuckles Left"})"), "config: a controller's is no headset's"); + check(!parseHeadsetConfig(R"({"model_number": "Index"})"), "config: a headset without lenses is refused"); + + // the lens: a point on the lens axis stays there, and further out it maps further out (barrel) + float out[3][2]; + Distortion flat; + computeDistortion(flat, 0.5f, 0.5f, false, out); + check(near(out[1][0], 0.5f) && near(out[1][1], 0.5f), "distortion: the center maps to the center"); + computeDistortion(config->distortion[1], 0.5f, 0.5f, false, out); + float edge[3][2]; + computeDistortion(config->distortion[1], 0.95f, 0.5f, false, edge); + check(edge[1][0] > out[1][0], "distortion: further out on the panel is further out in the image"); + check(edge[0][0] > edge[2][0], "distortion: red and blue are bent apart (blue more)"); + float flipped[3][2]; + computeDistortion(config->distortion[1], 0.3f, 0.2f, true, flipped); + computeDistortion(config->distortion[1], 0.3f, 0.2f, false, out); + check(near(flipped[1][1], 1.0f - out[1][1]), "distortion: the Vive Pro 2's image is upside down"); +} + +void lighthouseImu() { + using namespace lighthouse; + check(!parseImuRanges((const uint8_t[]){0x01, 0, 0}, 3), "ranges: zero codes are refused"); + const auto ranges = parseImuRanges((const uint8_t[]){0x01, 1, 1}, 3); + check(ranges && near(ranges->gyro, 500.0f * 3.14159265f / 180.0f) && near(ranges->accel, 4 * 9.80665f, 1e-3f), + "ranges: 500 degrees/s and 4 g"); + HeadsetConfig config; + config.model = Model::Index; + // a report whose samples have sequence numbers 11, 12, 10 (round robin); x on the accelerometer only + std::vector report(kImuReportSize, 0); + report[0] = kImuReport; + const uint8_t seq[3] = {11, 12, 10}; + for (int i = 0; i < 3; i++) { + const size_t at = 1 + 17 * i; + put16(report, at, 8192); // accel x: a quarter of full scale + put16(report, at + 6 + 4, -16384); // gyro z: minus half + put32(report, at + 12, 1000 * (seq[i] - 9)); + report[at + 16] = seq[i]; + } + ImuRanges r; + ImuSample out[3]; + uint8_t sequence = 0; + bool have = false; + size_t n = parseImuReport(report.data(), report.size(), config, r, sequence, have, out); + check(n == 3 && sequence == 12 && out[0].ticks == 1000 && out[2].ticks == 3000, + "imu: all three samples, oldest first"); + // the Index's IMU: x' = -y, y' = -x, z' = -z + check(near(out[0].accel[1], -0.25f * r.accel, 1e-3f) && near(out[0].accel[0], 0) && + near(out[0].gyro[2], 0.5f * r.gyro, 1e-4f), + "imu: the Index's axes into OpenXR's"); + report[1 + 16] = 13; // the next report: one new sample + n = parseImuReport(report.data(), report.size(), config, r, sequence, have, out); + check(n == 1 && sequence == 13, "imu: samples already seen are skipped"); + config.model = Model::Vive; + have = false; + n = parseImuReport(report.data(), report.size(), config, r, sequence, have, out); + check(n == 3 && near(out[0].accel[0], 0.25f * r.accel, 1e-3f) && near(out[0].gyro[2], 0.5f * r.gyro, 1e-4f), + "imu: the Vive's axes into OpenXR's"); + check(parseImuReport(report.data(), 40, config, r, sequence, have, out) == 0, "imu: a short report is refused"); +} + +void lighthouseCommands() { + using namespace lighthouse; + const auto on = powerReport(true); + const uint8_t expected[13] = {0x04, 0x78, 0x29, 0x38, 0x01, 0, 0, 0, 0, 0, 0x02, 0x00, 0x01}; + check(std::memcmp(on.data(), expected, sizeof(expected)) == 0 && on[45] == 0x7a, + "power: the power-on report libsurvive and Monado send"); + check(powerReport(false)[4] == 0 && powerReport(false)[45] == 0x7c, "power: off"); + const auto mode = pro2Command("dtd,2"); + check(mode[0] == 0x04 && mode[1] == 0x70 && mode[2] == 0x29 && mode[3] == 5 && std::memcmp(mode.data() + 4, "dtd,2", 5) == 0, + "pro 2: a mode command"); + check(pro2Modes()[2].width == 3264 && !pro2Modes()[2].needs_dsc && pro2Modes()[4].needs_dsc, + "pro 2: its modes, the 4896-wide ones with DSC"); + std::vector status(64, 0); + status[0] = kMainboardStatusReport; + put16(status, 4, 6350); // lens separation 63.50 mm + put16(status, 14, 210); // proximity + put16(status, 16, 6420); // IPD 64.20 mm + const auto s = parseMainboardStatus(status.data(), status.size()); + check(s && near(s->ipd, 0.0642f, 1e-6f) && near(s->lens_separation, 0.0635f, 1e-6f) && s->proximity == 210, + "mainboard: IPD, lens separation, proximity"); + const auto pulse = hapticReport(0.1f, 200.0f, 0.5f); + // 1e6 / 200 = 5000 us a period; half amplitude: low 1250, high 3750; 20 repeats + check(pulse[0] == 0xff && pulse[1] == 0x8f && pulse[2] == 7 && (pulse[4] | pulse[5] << 8) == 3750 && + (pulse[6] | pulse[7] << 8) == 1250 && (pulse[8] | pulse[9] << 8) == 20, + "haptic: a 200 Hz pulse of half amplitude for 0.1 s"); + check((hapticReport(0.1f, 200.0f, 0.0f)[6]) == 1, "haptic: amplitude 0 is the weakest pulse, not none"); +} + +void lighthouseControllers() { + using namespace lighthouse; + const auto wand = parseControllerConfig(R"({"model_number": "Vive. Controller MV", "acc_scale": [1,1,1]})"); + check(wand && wand->kind == ControllerKind::Wand, "controller config: a Vive wand"); + const auto left = parseControllerConfig(R"({"model_number": "Knuckles EV3.0 Left", "imu": {}})"); + check(left && left->kind == ControllerKind::IndexLeft, "controller config: a left Index controller"); + ImuRanges ranges; + + // a wand: flags 1111 0111 (trigger, trackpad, buttons), buttons, trigger, trackpad x and y + { + uint8_t report[30] = {kControllerReport1, 0x12, 9, 0x34, 0xF7, 0x05, 0x80, 0x00, 0x40, 0x00, 0xC0}; + ControllerState state; + ControllerImu imu[4]; + const size_t n = decodeControllerReport(report, sizeof(report), false, *wand, ranges, state, imu); + check(n == 0 && state.buttons == 0x05 && near(state.trigger, 128 / 255.0f) && near(state.pad_x, 0.5f, 1e-3f) && + near(state.pad_y, -0.5f, 1e-3f), + "watchman 1: buttons, trigger, trackpad"); + } + // a wand's battery then its IMU: 1110 1001 = battery and IMU + { + uint8_t report[30] = {kControllerReport1, 0x12, 16, 0x34, 0xE9, 0x80 | 77, 0x56}; + std::vector v(report, report + 30); + put16(v, 7, 0); // accel x + put16(v, 9, 16384); // accel y: half of full scale + put16(v, 11, 0); + std::memcpy(report, v.data(), 30); + ControllerState state; + ControllerImu imu[4]; + const size_t n = decodeControllerReport(report, sizeof(report), false, *wand, ranges, state, imu); + check(state.has_battery && state.battery == 77 && state.charging, "watchman 1: battery and charging"); + // the wand's IMU: x' = -x, y' = -z, z' = -y + check(n == 1 && near(imu[0].accel[2], -0.5f * ranges.accel, 1e-3f) && imu[0].ticks == (0x12u << 24 | 0x34u << 16 | 0x56u << 8), + "watchman 1: an IMU sample, in the wand's frame, with its time"); + } + // an Index controller: 0xf0 then touch and force + { + uint8_t report[30] = {kControllerReport1, 0x01, 11, 0x02, 0xF0, 161, 0x30, 10, 20, 30, 40, 50, 60}; + ControllerState state; + ControllerImu imu[4]; + decodeControllerReport(report, sizeof(report), true, *left, ranges, state, imu); + check(state.has_fingers && state.touch == 0x30 && state.fingers[0] == 40 && state.fingers[1] == 10 && + state.fingers[3] == 30 && state.squeeze_force == 50 && state.trackpad_force == 60, + "watchman 2: touches, fingers and forces"); + } + // two messages in one report + { + uint8_t report[59] = {kControllerReport2, 0, 3, 0, 0xF1, 0x09}; + report[30] = 0; + report[31] = 3; + report[33] = 0xF1; + report[34] = 0x21; + ControllerState state; + ControllerImu imu[4]; + decodeControllerReport(report, sizeof(report), true, *left, ranges, state, imu); + check(state.buttons == 0x21, "watchman 2: both messages of a report, the second last"); + } + ControllerState state; + ControllerImu imu[4]; + uint8_t junk[30] = {kControllerReport1, 0, 200}; + check(decodeControllerReport(junk, sizeof(junk), true, *left, ranges, state, imu) == 0, + "watchman: a message longer than its report is ignored"); +} + +// ---- Rift S ---- + +void riftS() { + using namespace rift_s; + std::vector info(32, 0); + info[0] = kReportDeviceInfo; + put16(info, 1, 2560); + put16(info, 3, 1440); + info[7] = 80; + const auto device = parseDeviceInfo(info.data(), info.size()); + check(device && device->width == 1440 && device->height == 2560 && device->refresh_hz == 80, + "rift s: the panel, portrait, at 80 Hz"); + + std::vector imu_config(32, 0); + imu_config[0] = kReportImuConfig; + put32(imu_config, 1, 1000); + putf(imu_config, 5, 16.4f); + putf(imu_config, 9, 2048.0f); + const auto config = parseImuConfig(imu_config.data(), imu_config.size()); + check(config && config->hz == 1000 && near(config->gyro_scale, 16.4f) && near(config->accel_scale, 2048.0f), + "rift s: the IMU's scales"); + + const char *calibration_json = R"({"FileFormat": {"Version": "1.0"}, "ImuCalibration": { + "DeviceFromImu": [0, -1, 0, 0.01, 1, 0, 0, 0.02, 0, 0, 1, 0.03, 0, 0, 0, 1], + "Gyroscope": {"RectificationMatrix": [1, 0, 0, 0, 1, 0, 0, 0, 1], "Offset": {"ConstantOffset": [0.01, 0, 0]}}, + "Accelerometer": {"RectificationMatrix": [1, 0, 0, 0, 1, 0, 0, 0, 1], + "Offset": {"OffsetAtZeroDegC": [0, 0.1, 0], "OffsetTemperatureCoefficient": [0, 0, 0]}}}})"; + const auto calibration = parseImuCalibration(calibration_json); + check(calibration && near(calibration->device_from_imu[1], -1) && near(calibration->device_from_imu[3], 1) && + near(calibration->gyro_offset[0], 0.01f), + "rift s: the IMU calibration, its rotation from the 4x4"); + check(calibration && near(calibration->imu_position[0], 0.01f) && near(calibration->imu_position[2], 0.03f), + "rift s: the IMU calibration, where the IMU sits"); + check(!parseImuCalibration(R"({"FileFormat": {"Version": "2.0"}, "ImuCalibration": {}})"), + "rift s: a calibration of another version is refused"); + if (!config || !calibration) return; + + std::vector hmd(64, 0); + hmd[0] = kInHmd; + put32(hmd, 3, 123456); + // sample 0: accel y = 1 g (2048), gyro x = 16.4 (1 degree/s); sample 2 invalid + put16(hmd, 7 + 1 + 2, 2048); + put16(hmd, 7 + 7, 164); + hmd[7 + 30] = 0x80; + const auto report = parseHmdReport(hmd.data(), hmd.size(), *config, *calibration); + check(report && report->count == 2 && report->timestamp_us == 123456, "rift s: valid samples up to the first invalid"); + if (report) { + // the IMU's y is the device's -x (DeviceFromImu row 0 is [0 -1 0]); accel offset 0.1 on y first + check(near(report->samples[0].accel[0], -(9.80665f - 0.1f), 1e-3f), "rift s: accel into the headset's frame"); + const float gyro_x = 10.0f * 3.14159265f / 180.0f - 0.01f; + check(near(report->samples[0].gyro[1], gyro_x, 1e-4f), "rift s: gyro into the headset's frame"); + } + check(!parseHmdReport(hmd.data(), 63, *config, *calibration), "rift s: a short IMU report is refused"); + check(parseStatus((const uint8_t[]){kInStatus, 1}, 2).value_or(false), "rift s: worn"); + const auto enable = enableReports(true); + check(enable.size() == 5 && enable[0] == std::vector({0x07, 0xa3, 0x01}) && enable[2][0] == 0x0a && + enable[2][1] == 2 && enable[4].size() == 41 && enable[4][1] == 1, + "rift s: the power-on sequence"); + check(keepAliveReport() == std::vector({0x93, 0x01, 0xb8, 0x0b, 0x00, 0x00}), "rift s: keep-alive"); + + const auto request = firmwareRequest(kFirmwareImuCalibration, 0x0c, 56); + check(request[0] == 0x4a && request[2] == 0x0e && request[3] == 0x0c && request[7] == 56, "rift s: firmware request"); + std::vector reply(64, 0); + std::memcpy(reply.data(), request.data(), 8); + check(firmwareReplyMatches(reply.data(), reply.size(), request), "rift s: a reply that answers the request"); + reply[3] = 0; + check(!firmwareReplyMatches(reply.data(), reply.size(), request), "rift s: a reply for another offset"); + put32(reply, 16, 2400); + check(firmwareBlockLength(reply.data(), reply.size()) == 2400, "rift s: a block's length"); + + std::vector devices(200, 0); + devices[0] = kReportDevices; + devices[2] = 2; + put32(devices, 3, 0xAAAA); + put32(devices, 3 + 8, kLeftController); + put32(devices, 31, 0xBBBB); + put32(devices, 31 + 8, kRightController); + const auto list = parseDevices(devices.data(), devices.size()); + check(list.size() == 2 && list[0].type == kLeftController && list[1].id == 0xBBBB, "rift s: the radio's devices"); + + // a controller report: buttons 0x05, trigger/grip, stick, IMU + std::vector c(62, 0); + c[0] = kInController; + put32(c, 1, 0xAAAA); + size_t at = 14; + const uint8_t blocks[] = {0x0c, 0x05, 0x0d, 0x11, 0x1b, 0x00, 0x08, 0x40, 0x22}; + std::memcpy(c.data() + at, blocks, sizeof(blocks)); + at += sizeof(blocks); + put16(c, at, 16384); + put16(c, at + 2, static_cast(-16384)); + at += 4; + c[at] = 0x91; + put32(c, at + 1, 777); + put16(c, at + 7, 1024); + at += 19; + c[9] = static_cast(at - 10); + ControllerState state; + check(parseControllerReport(c.data(), c.size(), state) == 0xAAAA, "rift s: a controller report names its device"); + const TouchValues v = touchValues(state); + check(v.primary && v.stick && !v.secondary && v.primary_touch, "rift s: buttons and touches"); + // trigger: 0x800 of 4096 travel left; grip: 0x400 + check(near(v.trigger, 0.5f, 1e-3f) && near(v.grip, 0.75f, 1e-3f), "rift s: trigger and grip"); + check(near(v.stick_x, 0.5f) && near(v.stick_y, -0.5f), "rift s: the stick"); + check(state.has_imu && state.imu_timestamp == 777 && state.accel[0] == 1024, "rift s: the controller's IMU block"); + + std::vector response(32, 0); + response[4] = 16; + putf(response, 5 + 8, 1.0f / 1024.0f); + putf(response, 5 + 12, 0.1220703f); + const auto imu = parseControllerImuConfig(response.data(), response.size()); + check(imu && near(imu->accel_scale, 1.0f / 1024.0f, 1e-7f), "rift s: the controller's IMU config"); + if (imu) { + float accel[3], gyro[3]; + controllerImu(state, *imu, accel, gyro); + check(near(accel[0], 9.80665f, 1e-3f), "rift s: the controller's accel, scaled"); + } + // the lenses: each eye sees about 105 degrees across, the left eye's mirrored from the right's + float fov[2][4]; + fieldOfView(0, fov[0][0], fov[0][1], fov[0][2], fov[0][3]); + fieldOfView(1, fov[1][0], fov[1][1], fov[1][2], fov[1][3]); + check(near(fov[0][1] - fov[0][0], 105.0f * 3.14159265f / 180.0f, 1e-4f) && near(fov[0][0], -fov[1][1]) && + fov[0][2] > 0 && fov[0][3] < 0 && near(fov[0][2], -fov[0][3]), + "rift s: the field of view, 105 degrees across, the eyes mirrored"); + const Panotools lens = panotools(1); + float center[3][2], outer[3][2]; + computeDistortion(lens, lens.lens_center[0] / lens.viewport[0], 0.5f, center); + check(near(center[1][0], lens.lens_center[0] / lens.viewport[0]) && near(center[1][1], 0.5f), + "rift s: the lens center stays put"); + computeDistortion(lens, 0.95f, 0.5f, outer); + float mid[3][2]; + computeDistortion(lens, 0.75f, 0.5f, mid); + check(outer[1][0] > mid[1][0] && mid[1][0] > center[1][0] && outer[2][0] > outer[0][0], + "rift s: further out on the panel is further out in the image, blue furthest"); + + const auto command = radioCommand(0xAAAA, kControllerConfigRequest, sizeof(kControllerConfigRequest)); + check(command[0] == 0x12 && command[1] == 0xAA && command[2] == 0xAA && command[9] == 0x32, "rift s: a radio command"); +} + +// ---- the Rift S's cameras ---- + +void riftSCameras() { + using namespace rift_s; + std::string json = R"({"FileFormat": {"Version": "1.0"}, "CameraCalibration": [)"; + for (int i = 0; i < kCameraCount; i++) { + char camera[512]; + std::snprintf(camera, sizeof(camera), + R"(%s{"Id": "%d", "ImageSize": [640, 480], "DeviceFromCamera": [1, 0, 0, %d, 0, 1, 0, 0, 0, 0, 1, + 0, 0, 0, 0, 1], "Projection": {"Model": "PinholeSymmetric", "Coefficients": [300, 320, 240]}, + "Distortion": {"Model": "Fisheye62", "Coefficients": [-0.03, 0.012, -0.006, 0.0013, 0, 0, 0.0004, + -0.0003]}})", + i ? "," : "", i, i); + json += camera; + } + json += "]}"; + const auto cameras = parseCameraCalibration(json); + check(cameras && (*cameras)[3].x_offset == 1920 && near((*cameras)[2].device_from_camera[3], 2) && + near((*cameras)[0].f, 300) && near((*cameras)[0].k[0], -0.03f) && near((*cameras)[0].p2, -0.0003f), + "rift s cameras: the calibration's five cameras, side by side in the frame"); + std::string wrong = json; + wrong.replace(wrong.find("Fisheye62"), 9, "Fisheye64"); + check(!parseCameraCalibration(wrong), "rift s cameras: another lens model is refused"); + if (!cameras) return; + + // the KB4 lens fitted to Fisheye62 puts every point of the picture within a pixel of where it was (KB4 has no + // tangential terms; the rest of the difference is a few hundredths) + const CameraCalibration &c = (*cameras)[0]; + const auto kb4 = kb4FromFisheye62(c); + double worst = 0; + bool inverted = true; + for (int y = 0; y < 480; y += 30) + for (int x = 0; x < 640; x += 30) { + const double pixel[2] = {double(x), double(y)}; + double point[2], fisheye[2], fitted[2]; + inverted = inverted && fisheye62Unproject(c, pixel, point); + fisheye62Project(c, point, fisheye); + kb4Project(c, kb4, point, fitted); + worst = std::max({worst, std::hypot(fisheye[0] - pixel[0], fisheye[1] - pixel[1]), + std::hypot(fitted[0] - pixel[0], fitted[1] - pixel[1])}); + } + std::printf("rift s cameras: KB4 %.5f %.5f %.5f %.5f, %.3f px at worst\n", kb4[0], kb4[1], kb4[2], kb4[3], worst); + check(inverted && worst < 1.0, "rift s cameras: the KB4 fit matches the Fisheye62 lens within a pixel"); + + // a frame: its header row, and the pictures where the exposure puts them + std::vector frame(kCameraFrameWidth * 700, 0); + uint8_t raw[50] = {0x86, 0xcd, 0xab, 0x34, 0x12}; + const uint64_t time = 0x0000000512345678ULL; + for (int i = 0; i < 8; i++) raw[16 + i] = static_cast(time >> (8 * i)); + for (int i = 0; i < kCameraCount; i++) raw[28 + 2 * i] = static_cast(1000 + 1000 * i), // exposures + raw[29 + 2 * i] = static_cast((1000 + 1000 * i) >> 8), + raw[40 + i] = static_cast(0x40 + i); + raw[48] = 0xce, raw[49] = 0xfa; + for (int block = 0; block < 400; block++) + if (raw[block / 8] & (0x80 >> (block % 8))) + for (int y = 0; y < 8; y++) std::memset(&frame[y * kCameraFrameWidth + block * 8], 200, 8); + const auto info = parseCameraFrameInfo(frame.data(), kCameraFrameWidth, 700); + check(info && info->tracking && info->counter == 0x1234 && info->time_us == time && info->exposure[4] == 5000 && + info->gain[2] == 0x42, + "rift s cameras: a frame's header row: tracking, its counter, time, exposures and gains"); + check(!parseCameraFrameInfo(frame.data(), 1600, 700), "rift s cameras: a frame of another width is refused"); + check(cameraYOffset(6000) == 165 && cameraYOffset(0) == 8 && cameraYOffset(20000) == 375, + "rift s cameras: the picture's offset from the exposure, within 8..375"); + // each picture marked with its camera id at its corner + for (int id = 0; id < kCameraCount; id++) + frame[cameraYOffset(info->exposure[id]) * kCameraFrameWidth + (*cameras)[id].x_offset] = static_cast(10 + id); + std::array, kCameraCount> pictures; + const bool split = splitTrackingFrame(frame.data(), kCameraFrameWidth, 700, kCameraFrameWidth, *info, *cameras, pictures); + check(split && pictures[0][0] == 10 + kCameraFrontLeft && pictures[1][0] == 10 + kCameraFrontRight && + pictures[4][0] == 10 + kCameraTop && pictures[2].size() == 640 * 480, + "rift s cameras: the pictures, cut out in the tracker's order (front left, front right, sides, top)"); + check(!splitTrackingFrame(frame.data(), kCameraFrameWidth, 400, kCameraFrameWidth, *info, *cameras, pictures), + "rift s cameras: a frame too short for its pictures is refused"); + + const auto report = cameraReport(true, 0x1234, 0x56, true); + const auto off = cameraReport(false); + check(report.size() == 41 && report[0] == 5 && report[1] == 1 && report[2] == 1 && report[3] == 0x34 && + report[4] == 0x12 && report[11] == 0x34 && report[17] == 0x56 && report[18] == 0x26 && off[1] == 0 && + off[3] == 0xb3 && off[13] == 0xf0, + "rift s cameras: the camera report with every camera's exposure and gain"); +} + +// ---- the PlayStation VR2 ---- + +void psvr2Headset() { + using namespace psvr2; + // status: worn, function button, IPD 64, then two IMU records + std::vector status(32 + 2 * 24, 0); + status[0] = 2, status[1] = 1, status[2] = 1, status[5] = 64; + put32(status, 32, 1000); + put16(status, 32 + 4, 0); // accel x (the IMU's) + put16(status, 32 + 6, 0); // accel y + put16(status, 32 + 8, 8192); // accel z: 1 g up the IMU's z, which is the headset's +y + put16(status, 32 + 10, 16384); // gyro x: 1000 degrees/s about the IMU's x, the headset's -z + put32(status, 56, 1500); + const auto st = parseStatus(status.data(), status.size()); + check(st && st->worn && st->function && st->ipd_mm == 64 && st->display == 2 && st->imu.size() == 2 && + st->imu[1].vts_us == 1500, + "psvr2: status: worn, button, IPD and its IMU records"); + check(st && near(st->imu[0].accel[1], 9.80665f, 0.01f) && near(st->imu[0].accel[0], 0) && + near(st->imu[0].gyro[2], -1000.0f * 3.14159265f / 180.0f * 16384.0f * 2000.0f / 32767.0f / 1000.0f, 0.01f), + "psvr2: the IMU's axes turned into the headset's (Monado's), 1 g up"); + check(!parseStatus(status.data(), 20), "psvr2: a short status report is refused"); + + // tracking: the headset's position (1, 2, 3) and orientation w=1 + std::vector record(512, 0); + std::memcpy(record.data(), "SLA", 3); + put32(record, 8, 777); + putf(record, 16, 1), putf(record, 20, 2), putf(record, 24, 3); + putf(record, 28, -1); // w = -1: the same rotation as +1, on the far side + const auto slam = parseSlam(record.data(), record.size(), mvr_quat{0, 0, 0, 1}); + check(slam && slam->vts_us == 777 && near(slam->pose.position.x, 3) && near(slam->pose.position.y, 2) && + near(slam->pose.position.z, -1) && near(slam->pose.orientation.w, 1), + "psvr2: the headset's own pose, its axes turned, kept on the last quaternion's side"); + record[0] = 'X'; + check(!parseSlam(record.data(), record.size(), mvr_quat{0, 0, 0, 1}), "psvr2: a record without SLA is refused"); + const mvr_pose head = headFromSlam(mvr_pose{{0, 0, 0, 1}, {0, 0, 0}}); + check(near(head.orientation.z, std::sqrt(0.5f)) && near(head.orientation.w, std::sqrt(0.5f)) && + near(std::sqrt(head.position.x * head.position.x + head.position.y * head.position.y + + head.position.z * head.position.z), + 0.104827f, 0.0005f), + "psvr2: the head from the tracker: Monado's correction and the tracker's 10.5 cm offset"); + + // requests + const ControlRequest mode = cameraModeRequest(kCameraTracking); + check(mode.request_type == 0x42 && mode.request == 9 && mode.value == 0x0b && mode.data.size() == 16 && + mode.data[0] == 0x0b && mode.data[2] == 1 && mode.data[4] == 8 && mode.data[8] == 1 && mode.data[12] == 0x0a, + "psvr2: the camera mode request"); + const ControlRequest read = lensCalibrationRequest(); + check(read.request_type == 0xc2 && read.request == 1 && read.value == 0x8f && read.data.size() == 8 + 0x100 && + brightnessRequest(1.0f).data[8] == 31, + "psvr2: a read request, and brightness"); + std::vector fw(8 + 84, 0); + put32(fw, 8, 0x06000102); + std::memcpy(fw.data() + 8 + 56, "SERIAL123", 9); + const auto payload = readPayload(fw.data(), fw.size(), 84); + const auto firmware = parseFirmwareInfo(payload.data(), payload.size()); + check(firmware && firmware->version == 0x06000102 && firmware->pcb_id == "SERIAL123", "psvr2: firmware and serial"); + + // lenses: an old calibration is Monado's default; a version 4 one is worked out + std::vector lens(0x100, 0); + lens[0] = 3; + const auto old = lensCalibration(lens.data(), lens.size()); + lens[0] = 4; + putf(lens, 8 + 4 * 4, 10.0f); // the left eye tilted 10 degrees + const auto cal = lensCalibration(lens.data(), lens.size()); + check(near(old[0], -0.09919293f) && near(old[2], 0.09919293f) && near(cal[4], std::cos(10 * 3.14159265f / 180)) && + near(cal[5], -std::sin(10 * 3.14159265f / 180)) && near(cal[1], 14.95f / 1000 / 32.46199f), + "psvr2: the lens calibration, its offsets and tilt"); + float center[3][2], edge[3][2], mid[3][2]; + computeDistortion(old, 1, 0.5f, 0.5f, center); + computeDistortion(old, 1, 0.95f, 0.5f, edge); + computeDistortion(old, 1, 0.75f, 0.5f, mid); + check(center[1][0] > 0.3f && center[1][0] < 0.4f && near(center[1][1], 0.5f, 0.01f) && edge[1][0] > mid[1][0] && + mid[1][0] > center[1][0] && edge[2][0] > edge[0][0], + "psvr2: the lens: further out on the panel is further out in the image, blue furthest"); + float left_center[3][2]; + computeDistortion(old, 0, 0.5f, 0.5f, left_center); + check(near(left_center[1][0], 1.0f - center[1][0], 0.001f) && near(fieldOfView(0).left, -fieldOfView(1).right), + "psvr2: the eyes mirror each other"); +} + +void psvr2Sense() { + using namespace psvr2; + const uint8_t check_value[] = {'1', '2', '3', '4', '5', '6', '7', '8', '9'}; + check(psvr2::crc32(0, check_value, 9) == 0xcbf43926u, "sense: CRC-32 (the standard check value)"); + + // a Bluetooth report from the right controller: cross pressed and touched, trigger half, stick right + std::vector r(78, 0); + r[0] = kSenseBluetoothReport; + uint8_t *c = r.data() + 2; + c[0] = 255, c[1] = 128, c[2] = 128; + c[7] = 2; // cross + c[9] = 2 | 4; // cross touched, stick touched + c[8] = 16; // PS + put16(r, 2 + 15, 100); // gyro x + put16(r, 2 + 21, 200); // accel x + put32(r, 2 + 27, 3000); + c[41] = 0x15; // charging, level 5 + const uint8_t seed = 0xa1; + uint32_t crc = psvr2::crc32(psvr2::crc32(0, &seed, 1), r.data(), 74); + put32(r, 74, crc); + const auto right = parseSenseReport(r.data(), r.size(), false); + check(right && (right->buttons & MVR_BUTTON_PRIMARY) && (right->buttons & MVR_BUTTON_SYSTEM) && + (right->touches & MVR_BUTTON_PRIMARY) && (right->touches & MVR_BUTTON_THUMBSTICK) && + near(right->trigger, 128 / 255.0f) && near(right->stick_x, 127 / 128.0f) && near(right->stick_y, 0) && + right->gyro[0] == 100 && right->accel[0] == 200 && right->imu_ticks == 3000 && right->charging && + near(right->battery, 0.55f), + "sense: a right controller's Bluetooth report: cross is A, the PS button, touches, trigger, stick, battery"); + const auto as_left = parseSenseReport(r.data(), r.size(), true); + check(as_left && !(as_left->buttons & MVR_BUTTON_PRIMARY), "sense: the left controller's buttons are its own"); + r[10] ^= 1; + check(!parseSenseReport(r.data(), r.size(), false), "sense: a report that fails its CRC is refused"); + + // calibration: accel +-8192 about 0, gyro bias 10, +-1010 at 2000 degrees/s + uint8_t a[64] = {kSenseCalibrationReport, 0x00}, b[64] = {kSenseCalibrationReport, 0x81}; + uint8_t d[116] = {}; + const auto put = [&](size_t at, int16_t v) { d[at] = static_cast(v), d[at + 1] = static_cast(v >> 8); }; + put(0x00, 8192), put(0x06, -8192), put(0x0e, 8192), put(0x14, -8192), put(0x1c, 8192), put(0x22, -8192); + put(0x24, 10), put(0x26, 10), put(0x28, 10); + put(0x2a, 1010), put(0x30, -990), put(0x38, 1010), put(0x3e, -990), put(0x46, 1010), put(0x4c, -990); + put(0x4e, 2000), put(0x50, 2000); + std::memcpy(a + 2, d, 58); + std::memcpy(b + 2, d + 58, 58); + const auto calibration = parseSenseCalibration(b, a); // either order + check(calibration && near(calibration->accel_scale[0], 9.80665f / 8192) && near(calibration->gyro_bias[1], 10) && + near(calibration->gyro_scale[2], 2000 * 3.14159265f / 180 / 1000, 1e-4f), + "sense: the IMU calibration from its two parts"); + if (calibration) { + SenseState state; + state.gyro[0] = 1010, state.accel[1] = 8192; + float accel[3], gyro[3]; + senseImu(state, *calibration, accel, gyro); + check(near(gyro[0], 2000 * 3.14159265f / 180, 0.01f) && near(accel[1], 9.80665f, 0.01f), + "sense: a raw sample scaled with it"); + } + + const auto out = senseOutputReport(0.5f, 3, 9, 1234); + const uint8_t out_seed = 0xa2; + check(out[0] == 0x31 && out[1] == 0x30 && out[2] == 0x10 && out[3] == 0x03 && out[5] == 128 && out[22] == 5 && + out[41] == 9 && le32(out.data() + 74) == psvr2::crc32(psvr2::crc32(0, &out_seed, 1), out.data(), 74), + "sense: the vibration report, LEDs off, its CRC"); + const mvr_pose grip = senseGrip(0), grip_right = senseGrip(1); + check(near(grip.position.x, -grip_right.position.x) && std::fabs(grip.orientation.x) > 0.4f, + "sense: grip poses mirror between the hands, tilted from the controller"); +} + +// ---- the headsets table ---- + +void headsetTable() { + check(headsetFromAlvrName("Quest 3") == MVR_HEADSET_QUEST_3 && headsetFromAlvrName("Quest 3S") == MVR_HEADSET_QUEST_3S, + "alvr names: Quest 3 and Quest 3S are told apart"); + check(headsetFromAlvrName("Steam Frame") == MVR_HEADSET_STEAM_FRAME, "alvr names: the patched client on a Steam Frame"); + check(headsetFromAlvrName("quest 3") == MVR_HEADSET_UNKNOWN && headsetFromAlvrName("Android (unknown)") == + MVR_HEADSET_UNKNOWN, + "alvr names: anything else is unknown"); + check(headsetFromKey("index") == MVR_HEADSET_VALVE_INDEX && headsetFromKey("rift") == MVR_HEADSET_UNKNOWN, + "keys"); + for (uint32_t h = 1; h < MVR_HEADSET_COUNT; h++) { + const HeadsetSpec &spec = headsetSpec(h); + char what[96]; + std::snprintf(what, sizeof(what), "spec of %s: sizes, rates and field of view", mvr_describe_headset(h)->key); + bool ok = spec.eye_width > 0 && spec.eye_height > 0 && spec.refresh_rate_count > 0 && spec.default_refresh > 0; + bool listed = false; + for (uint32_t i = 0; i < spec.refresh_rate_count; i++) listed = listed || spec.refresh_rates[i] == spec.default_refresh; + ok = ok && listed && spec.fov[0].left < 0 && spec.fov[0].right > 0 && spec.fov[0].up > 0 && spec.fov[0].down < 0; + check(ok, what); + check(std::string(mvr_describe_headset(h)->key) != "unknown", "each headset has a key"); + } + check(!withinDecoder(headsetSpec(MVR_HEADSET_QUEST_1), 4288, 2368, 72.0f), + "decoder: ALVR's default frame is too much for a Quest 1"); + check(withinDecoder(headsetSpec(MVR_HEADSET_QUEST_1), 3328, 1856, 72.0f), "decoder: a smaller one is not"); + check(withinDecoder(headsetSpec(MVR_HEADSET_QUEST_3), 4288, 2240, 90.0f), "decoder: a Quest 3 at 90 Hz"); + check(withinDecoder(headsetSpec(MVR_HEADSET_UNKNOWN), 9999, 9999, 120.0f), "decoder: no limit known, none applied"); + mvr_system_info info{}; + applyHeadsetSpec(headsetSpec(MVR_HEADSET_VIVE_PRO_2), info); + check(info.headset == MVR_HEADSET_VIVE_PRO_2 && info.controllers == MVR_CONTROLLERS_VIVE_WAND && + std::string(info.system_name) == "HTC Vive Pro 2" && info.refresh_rate_count == 2, + "applyHeadsetSpec: identity and rates"); +} + +// ---- foveated encoding: the server's squeeze and ALVR's client's stretch undo each other ---- + +void foveation() { + // ALVR's default shape, for a Quest 2-sized eye + const Foveation f = computeFoveation(2144, 2336, 0.45f, 0.4f, 0.4f, 0.1f, 4.0f, 5.0f); + check(f.enabled && f.optimized_eye_width % 32 == 0 && f.optimized_eye_height % 32 == 0 && + f.optimized_eye_width < 2144 && f.optimized_eye_height < 2336, + "foveation: a smaller frame, in multiples of 32"); + check(f.eye_size_ratio[0] > 0.9f && f.eye_size_ratio[0] <= 1.0f, "foveation: the squeezed eye nearly fills it"); + float worst = 0; + for (int eye = 0; eye < 2; eye++) + for (int i = 0; i <= 20; i++) + for (int j = 0; j <= 20; j++) { + const float p[2] = {0.02f + 0.96f * i / 20.0f, 0.02f + 0.96f * j / 20.0f}; + float q[2], back[2]; + foveationClientSample(f, eye, p, q); // where the client reads this point in the encoded eye + const float frame[2] = {(eye + q[0]) * 0.5f, q[1]}; + foveationSource(f, frame, back); // what the server put there, in the composed frame + const float expected[2] = {(eye + p[0]) * 0.5f, p[1]}; + worst = std::max(worst, std::max(std::fabs(back[0] - expected[0]), std::fabs(back[1] - expected[1]))); + } + std::printf("foveation: worst round trip %.6f of the frame\n", worst); + check(worst < 1e-3f, "foveation: the client's stretch undoes the server's squeeze everywhere"); + check(!computeFoveation(2144, 2336, 0.45f, 0.4f, 0.4f, 0.1f, 0.5f, 5.0f).enabled, + "foveation: an edge ratio under 1 is refused"); +} + +} // namespace + +int main() { + json(); + lighthouseIdentify(); + lighthouseConfig(); + lighthouseImu(); + lighthouseCommands(); + lighthouseControllers(); + riftS(); + riftSCameras(); + psvr2Headset(); + psvr2Sense(); + headsetTable(); + foveation(); + std::printf("%s: %d checks, %d failed\n", g_failures ? "FAILED" : "PASSED", g_checks, g_failures); + return g_failures ? 1 : 0; +} diff --git a/tests/rift_s_tracking_check.cpp b/tests/rift_s_tracking_check.cpp new file mode 100644 index 0000000..f439e88 --- /dev/null +++ b/tests/rift_s_tracking_check.cpp @@ -0,0 +1,395 @@ +// The Rift S's camera tracking end to end, without a headset (docs/HEADSETS.md, phase I): a synthetic Rift S moves +// through a textured room along a known path. Its five Fisheye62 cameras' pictures are rendered and packed into the +// camera device's frames as the headset sends them (the 8x8-block header row, each picture lowered by its exposure, +// controller frames between the tracking ones), and its IMU's reports are made from the same path, with the IMU's +// 32-bit clock wrapping part way. All of it goes through RiftSTracking (the backend's path) into Basalt, and the +// path Basalt reports, in OpenXR's axes, is compared with the true one. +// +// rift_s_tracking_check [seconds] (needs scripts/build-basalt.sh) +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "rift_s_protocol.h" +#include "rift_s_tracking.h" + +using namespace metalvr; +using namespace metalvr::rift_s; + +namespace { + +struct Vec { + double x = 0, y = 0, z = 0; +}; +Vec operator+(Vec a, Vec b) { return {a.x + b.x, a.y + b.y, a.z + b.z}; } +Vec operator-(Vec a, Vec b) { return {a.x - b.x, a.y - b.y, a.z - b.z}; } +Vec operator*(double s, Vec a) { return {s * a.x, s * a.y, s * a.z}; } +double dot(Vec a, Vec b) { return a.x * b.x + a.y * b.y + a.z * b.z; } +Vec normalize(Vec a) { return (1.0 / std::sqrt(dot(a, a))) * a; } + +struct Mat { + double m[3][3] = {{1, 0, 0}, {0, 1, 0}, {0, 0, 1}}; + Vec operator*(Vec v) const { + return {m[0][0] * v.x + m[0][1] * v.y + m[0][2] * v.z, m[1][0] * v.x + m[1][1] * v.y + m[1][2] * v.z, + m[2][0] * v.x + m[2][1] * v.y + m[2][2] * v.z}; + } + Mat operator*(const Mat &o) const { + Mat r; + for (int i = 0; i < 3; i++) + for (int j = 0; j < 3; j++) r.m[i][j] = m[i][0] * o.m[0][j] + m[i][1] * o.m[1][j] + m[i][2] * o.m[2][j]; + return r; + } + Mat t() const { + Mat r; + for (int i = 0; i < 3; i++) + for (int j = 0; j < 3; j++) r.m[i][j] = m[j][i]; + return r; + } +}; +Mat rotX(double a) { + Mat r; + r.m[1][1] = r.m[2][2] = std::cos(a); + r.m[1][2] = -std::sin(a); + r.m[2][1] = std::sin(a); + return r; +} +Mat rotY(double a) { + Mat r; + r.m[0][0] = r.m[2][2] = std::cos(a); + r.m[0][2] = std::sin(a); + r.m[2][0] = -std::sin(a); + return r; +} +Mat rotZ(double a) { + Mat r; + r.m[0][0] = r.m[1][1] = std::cos(a); + r.m[0][1] = -std::sin(a); + r.m[1][0] = std::sin(a); + return r; +} +Mat fromQuat(double x, double y, double z, double w) { + Mat r; + r.m[0][0] = 1 - 2 * (y * y + z * z), r.m[0][1] = 2 * (x * y - z * w), r.m[0][2] = 2 * (x * z + y * w); + r.m[1][0] = 2 * (x * y + z * w), r.m[1][1] = 1 - 2 * (x * x + z * z), r.m[1][2] = 2 * (y * z - x * w); + r.m[2][0] = 2 * (x * z - y * w), r.m[2][1] = 2 * (y * z + x * w), r.m[2][2] = 1 - 2 * (x * x + y * y); + return r; +} +// the angle of a rotation matrix +double angleOf(const Mat &r) { + return std::acos(std::clamp((r.m[0][0] + r.m[1][1] + r.m[2][2] - 1) / 2, -1.0, 1.0)); +} + +// ---- the path: at rest for a second, then moving about in all six axes ---- + +double ramp(double t) { + const double s = std::clamp((t - 1.0) / 2.0, 0.0, 1.0); + return s * s * (3 - 2 * s); +} +Vec truePosition(double t) { + const double r = ramp(t); + return {r * 0.45 * std::sin(0.55 * t), 1.6 + r * 0.12 * std::sin(0.8 * t + 0.3), r * 0.4 * std::sin(0.37 * t + 1.0)}; +} +Mat trueRotation(double t) { + const double r = ramp(t); + return rotY(r * 0.7 * std::sin(0.31 * t)) * rotX(r * 0.25 * std::sin(0.47 * t)) * rotZ(r * 0.12 * std::sin(0.6 * t)); +} + +// ---- the room: a box textured with smooth noise at two scales ---- + +double hash(int a, int b, int c) { + uint32_t h = static_cast(a) * 374761393u + static_cast(b) * 668265263u + + static_cast(c) * 2246822519u; + h = (h ^ (h >> 13)) * 1274126177u; + return ((h ^ (h >> 16)) & 0xffff) / 65535.0; +} +double noise(double u, double v, double scale, int seed) { + const double x = u / scale, y = v / scale; + const int ix = static_cast(std::floor(x)), iy = static_cast(std::floor(y)); + double fx = x - ix, fy = y - iy; + fx = fx * fx * (3 - 2 * fx); + fy = fy * fy * (3 - 2 * fy); + const double a = hash(ix, iy, seed), b = hash(ix + 1, iy, seed), c = hash(ix, iy + 1, seed), + d = hash(ix + 1, iy + 1, seed); + return a + (b - a) * fx + (c - a) * fy + (a - b - c + d) * fx * fy; +} +const Vec kRoomMin{-3.5, 0.0, -3.5}, kRoomMax{3.5, 2.8, 3.5}; +uint8_t shade(Vec origin, Vec direction) { + double t = 1e9; + int face = 0; + const double o[3] = {origin.x, origin.y, origin.z}, d[3] = {direction.x, direction.y, direction.z}; + const double lo[3] = {kRoomMin.x, kRoomMin.y, kRoomMin.z}, hi[3] = {kRoomMax.x, kRoomMax.y, kRoomMax.z}; + for (int a = 0; a < 3; a++) { + if (std::fabs(d[a]) < 1e-12) continue; + const double ta = ((d[a] > 0 ? hi[a] : lo[a]) - o[a]) / d[a]; + if (ta > 0 && ta < t) t = ta, face = a * 2 + (d[a] > 0); + } + const Vec p = origin + t * direction; + double u, v; + if (face / 2 == 0) u = p.z, v = p.y; + else if (face / 2 == 1) u = p.x, v = p.z; + else u = p.x, v = p.y; + const double value = 0.5 * noise(u, v, 0.21, face) + 0.35 * noise(u, v, 0.07, face + 10) + + 0.15 * noise(u, v, 0.6, face + 20); + return static_cast(std::clamp(20.0 + 230.0 * (value - 0.15) / 0.7, 0.0, 255.0)); +} + +// ---- the headset: five cameras, as a Rift S's firmware would describe them ---- + +struct Placement { + Vec position; + double yaw, pitch; // where it looks: yaw left of straight ahead, pitch up +}; +// in the headset's frame (+x right, +y up, +z back), by camera id: top, side left, front right, front left, side right +const Placement kPlacements[kCameraCount] = { + {{0.0, 0.05, -0.06}, 0.0, 0.75}, {{-0.09, 0.0, -0.03}, 1.25, -0.35}, {{0.035, 0.0, -0.08}, -0.15, -0.2}, + {{-0.035, 0.0, -0.08}, 0.15, -0.2}, {{0.09, 0.0, -0.03}, -1.25, -0.35}}; + +std::string calibrationJson(Mat device_from_camera[kCameraCount]) { + std::string json = "{\"FileFormat\":{\"Version\":\"1.0\"},\"CameraCalibration\":["; + for (int i = 0; i < kCameraCount; i++) { + const Placement &p = kPlacements[i]; + // the camera looks down its +z with +y down (OpenCV's camera); turned to look along `yaw` and `pitch` + Mat look; + look.m[0][0] = 1, look.m[1][1] = -1, look.m[2][2] = -1; // camera (x, y, z) -> headset (x, -y, -z): ahead + const Mat r = rotY(p.yaw) * rotX(p.pitch) * look; + device_from_camera[i] = r; + char camera[1024]; + std::snprintf(camera, sizeof(camera), + "%s{\"Id\":\"%d\",\"ImageSize\":[640,480],\"DeviceFromCamera\":[%.9f,%.9f,%.9f,%.9f,%.9f,%.9f," + "%.9f,%.9f,%.9f,%.9f,%.9f,%.9f,0,0,0,1],\"Projection\":{\"Model\":\"PinholeSymmetric\"," + "\"Coefficients\":[%.3f,%.3f,%.3f]},\"Distortion\":{\"Model\":\"Fisheye62\",\"Coefficients\":" + "[-0.028,0.011,-0.006,0.0013,0,0,0.00042,-0.00031]}}", + i ? "," : "", i, r.m[0][0], r.m[0][1], r.m[0][2], p.position.x, r.m[1][0], r.m[1][1], r.m[1][2], + p.position.y, r.m[2][0], r.m[2][1], r.m[2][2], p.position.z, 290.0 + i, 321.5 - i, 239.0 + i * 0.5); + json += camera; + } + return json + "]}"; +} + +std::string imuJson(const float imu_position[3]) { + char json[1024]; + std::snprintf(json, sizeof(json), + "{\"FileFormat\":{\"Version\":\"1.0\"},\"ImuCalibration\":{\"DeviceFromImu\":[1,0,0,%f,0,1,0,%f,0,0," + "1,%f,0,0,0,1],\"Gyroscope\":{\"RectificationMatrix\":[1,0,0,0,1,0,0,0,1],\"Offset\":" + "{\"ConstantOffset\":[0,0,0]}},\"Accelerometer\":{\"RectificationMatrix\":[1,0,0,0,1,0,0,0,1]," + "\"Offset\":{\"OffsetAtZeroDegC\":[0,0,0]}}}}", + imu_position[0], imu_position[1], imu_position[2]); + return json; +} + +// a camera device frame: the header row's 50 bytes as 8x8 blocks, then each camera's picture where it belongs +void packFrame(std::vector &frame, uint32_t height, bool tracking, uint16_t counter, uint64_t time_us, + uint16_t exposure, const std::array, kCameraCount> *pictures, + const std::array &calibration) { + frame.assign(static_cast(kCameraFrameWidth) * height, 0); + uint8_t raw[50] = {}; + raw[0] = tracking ? 0x86 : 0x06; + raw[1] = 0xcd, raw[2] = 0xab; + raw[3] = static_cast(counter), raw[4] = static_cast(counter >> 8); + for (int i = 0; i < 8; i++) raw[16 + i] = static_cast(time_us >> (8 * i)); + for (int c = 0; c < kCameraCount; c++) { + raw[28 + 2 * c] = static_cast(exposure); + raw[29 + 2 * c] = static_cast(exposure >> 8); + raw[40 + c] = 0xf0; + } + raw[48] = 0xce, raw[49] = 0xfa; + for (int block = 0; block < 400; block++) + if (raw[block / 8] & (0x80 >> (block % 8))) + for (int y = 0; y < 8; y++) std::memset(&frame[y * kCameraFrameWidth + block * 8], 255, 8); + if (!pictures) return; + const uint32_t top = cameraYOffset(exposure); + for (int id = 0; id < kCameraCount; id++) { + const CameraCalibration &c = calibration[id]; + const std::vector &picture = (*pictures)[id]; + for (uint32_t y = 0; y < c.height; y++) + std::memcpy(&frame[(top + y) * kCameraFrameWidth + c.x_offset], &picture[y * c.width], c.width); + } +} + +} // namespace + +int main(int argc, char **argv) { + const double seconds = argc > 1 ? std::atof(argv[1]) : 24.0; + int failures = 0; + auto check = [&](bool ok, const char *what) { + std::printf("%s %s\n", ok ? "ok " : "FAIL", what); + if (!ok) failures++; + }; + + // the headset's description, as its firmware would hand it over, through the parsers the backend uses + const float imu_position[3] = {0.012f, -0.008f, -0.045f}; + Mat device_from_camera[kCameraCount]; + const auto calibration = parseCameraCalibration(calibrationJson(device_from_camera)); + const auto imu_calibration = parseImuCalibration(imuJson(imu_position)); + check(calibration.has_value() && imu_calibration.has_value(), "the synthetic firmware's calibration parses"); + if (!calibration || !imu_calibration) return 1; + ImuConfig config; + config.hz = 1000; + + std::string why; + auto tracking = RiftSTracking::start(config, *imu_calibration, *calibration, &why); + if (!tracking) { + std::printf("FAIL Basalt did not start: %s\n", why.c_str()); + return 1; + } + + // each camera's rays, once: the true lens (Fisheye62) undone at every pixel + std::array, kCameraCount> rays; + for (int id = 0; id < kCameraCount; id++) { + const CameraCalibration &c = (*calibration)[id]; + rays[id].resize(static_cast(c.width) * c.height); + for (uint32_t y = 0; y < c.height; y++) + for (uint32_t x = 0; x < c.width; x++) { + double point[2] = {0, 0}; + const double pixel[2] = {static_cast(x), static_cast(y)}; + fisheye62Unproject(c, pixel, point); + rays[id][y * c.width + x] = device_from_camera[id] * normalize(Vec{point[0], point[1], 1.0}); + } + } + + // the IMU's clock starts 6 s before its 32-bit wrap; the camera's 64-bit clock has other high bits + const uint64_t imu_start_us = 0x100000000ULL - 6'000'000ULL; + const uint64_t camera_high = 0x500000000ULL; + const Vec gravity{0, -9.80665, 0}; + std::mt19937 random(7); + std::normal_distribution accel_noise(0, 0.02), gyro_noise(0, 0.002); + const double h = 1e-4; + auto specificForce = [&](double t) { + const Vec a = (1 / (h * h)) * (truePosition(t + h) - 2.0 * truePosition(t) + truePosition(t - h)); + return trueRotation(t).t() * (a - gravity); + }; + auto angularVelocity = [&](double t) { + // R(t)^T R(t + h) = I + h [w]x + const Mat d = trueRotation(t - h).t() * trueRotation(t + h); + return (1 / (4 * h)) * Vec{d.m[2][1] - d.m[1][2], d.m[0][2] - d.m[2][0], d.m[1][0] - d.m[0][1]}; + }; + + const uint32_t frame_height = cameraYOffset(6000) + 480 + 8; + std::array, kCameraCount> pictures; + std::vector frame; + struct Sample { + double t; + TrackedPose pose; + }; + std::vector estimates; + int64_t last_pose_ns = 0; + const int imu_steps = static_cast(seconds * 1000); + uint16_t counter = 0; + int skipped_controller_frames = 0; + const auto started = std::chrono::steady_clock::now(); + for (int step = 0; step < imu_steps; step += 3) { + // an IMU report: three samples a millisecond apart, the first's time in it + HmdReport report; + report.timestamp_us = static_cast(imu_start_us + static_cast(step) * 1000); + report.count = 3; + for (int i = 0; i < 3; i++) { + const double t = (step + i) / 1000.0; + const Vec f = specificForce(t), w = angularVelocity(t); + ImuSample &s = report.samples[i]; + s.accel[0] = static_cast(f.x + accel_noise(random)); + s.accel[1] = static_cast(f.y + accel_noise(random)); + s.accel[2] = static_cast(f.z + accel_noise(random)); + s.gyro[0] = static_cast(w.x + gyro_noise(random)); + s.gyro[1] = static_cast(w.y + gyro_noise(random)); + s.gyro[2] = static_cast(w.z + gyro_noise(random)); + } + tracking->imu(report); + // the camera device's frames at 60 Hz, tracking and controller frames in turn, once the IMU has passed them + const int frame_index_now = (step + 2) * 60 / 1000, frame_index_before = (step - 1) * 60 / 1000; + if (step == 0 || frame_index_now == frame_index_before) continue; + const int index = frame_index_now; + const double t = index / 60.0; + const bool is_tracking = index % 2 == 0; + const uint64_t time_us = camera_high + ((imu_start_us + static_cast(t * 1e6)) & 0xffffffffULL); + if (!is_tracking) { + packFrame(frame, frame_height, false, counter++, time_us, 6000, nullptr, *calibration); + if (!tracking->frame(frame.data(), kCameraFrameWidth, frame_height, kCameraFrameWidth)) skipped_controller_frames++; + continue; + } + // render the five pictures from the true pose at this time + const Mat world_from_device = trueRotation(t); + const Vec device_position = truePosition(t); + std::vector workers; + for (int id = 0; id < kCameraCount; id++) + workers.emplace_back([&, id] { + const CameraCalibration &c = (*calibration)[id]; + pictures[id].resize(static_cast(c.width) * c.height); + const Vec origin = device_position + world_from_device * kPlacements[id].position; + for (size_t p = 0; p < pictures[id].size(); p++) + pictures[id][p] = shade(origin, world_from_device * rays[id][p]); + }); + for (std::thread &w : workers) w.join(); + packFrame(frame, frame_height, true, counter++, time_us, 6000, &pictures, *calibration); + tracking->frame(frame.data(), kCameraFrameWidth, frame_height, kCameraFrameWidth); + TrackedPose pose; + int64_t pose_ns = 0; + if (tracking->pose(pose, &pose_ns) && pose_ns != last_pose_ns) { + last_pose_ns = pose_ns; + // the pose's time back to the path's: the headset's extended microseconds since the IMU's first + estimates.push_back({(pose_ns / 1000 - static_cast(imu_start_us)) / 1e6, pose}); + } + } + // the last poses, as Basalt finishes + for (int wait = 0; wait < 40; wait++) { + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + TrackedPose pose; + int64_t pose_ns = 0; + if (tracking->pose(pose, &pose_ns) && pose_ns != last_pose_ns) { + last_pose_ns = pose_ns; + estimates.push_back({(pose_ns / 1000 - static_cast(imu_start_us)) / 1e6, pose}); + wait = 0; + } + } + const double elapsed = std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + std::printf(" %.0f s of headset time in %.1f s; %llu tracking frames given, %zu poses back\n", seconds, elapsed, + static_cast(tracking->framesTracked()), estimates.size()); + check(skipped_controller_frames > 0 && tracking->framesTracked() > 0, + "controller frames are skipped and tracking frames are given to the tracker"); + check(estimates.size() > tracking->framesTracked() / 2, "Basalt reports a pose for most tracking frames"); + if (estimates.size() < 10) return 1; + + // The tracker's room has its own origin and heading: align it to the true one by the first pose, by heading only + // (both have +y up, which this checks), then compare the whole path. + const Sample &first = estimates.front(); + const Mat est0 = fromQuat(first.pose.orientation.x, first.pose.orientation.y, first.pose.orientation.z, + first.pose.orientation.w); + const Mat true0 = trueRotation(first.t); + const Mat align_full = true0 * est0.t(); + const double heading = std::atan2(align_full.m[0][2] - align_full.m[2][0], align_full.m[0][0] + align_full.m[2][2]); + const Mat align = rotY(heading); + const Vec est_p0{first.pose.position.x, first.pose.position.y, first.pose.position.z}; + const double tilt = angleOf(align.t() * align_full); + std::printf(" the tracker's up is %.2f degrees from the true up\n", tilt * 180 / M_PI); + check(tilt < 2 * M_PI / 180, "the tracker's room has +y up, as OpenXR (within 2 degrees)"); + + double sum_sq = 0, worst = 0, worst_angle = 0, path = 0; + Vec previous = truePosition(first.t); + for (const Sample &s : estimates) { + const Vec est{s.pose.position.x, s.pose.position.y, s.pose.position.z}; + const Vec mapped = truePosition(first.t) + align * (est - est_p0); + const Vec truth = truePosition(s.t); + const Vec error = mapped - truth; + const double e = std::sqrt(dot(error, error)); + sum_sq += e * e; + worst = std::max(worst, e); + path += std::sqrt(dot(truth - previous, truth - previous)); + previous = truth; + const Mat r = align * fromQuat(s.pose.orientation.x, s.pose.orientation.y, s.pose.orientation.z, s.pose.orientation.w); + worst_angle = std::max(worst_angle, angleOf(r.t() * trueRotation(s.t))); + } + const double rms = std::sqrt(sum_sq / estimates.size()); + std::printf(" path %.2f m: position error %.1f cm RMS, %.1f cm at worst; rotation %.2f degrees at worst\n", path, + rms * 100, worst * 100, worst_angle * 180 / M_PI); + check(path > 2.0, "the headset moved more than 2 m"); + check(rms < 0.05, "the tracked path follows the true one (under 5 cm RMS)"); + check(worst_angle < 3 * M_PI / 180, "the tracked rotation follows the true one (under 3 degrees)"); + std::printf(failures ? "FAILED: %d\n" : "PASSED\n", failures); + return failures ? 1 : 0; +} diff --git a/tests/xr_smoke.cpp b/tests/xr_smoke.cpp index 0a345de..57f4a18 100644 --- a/tests/xr_smoke.cpp +++ b/tests/xr_smoke.cpp @@ -218,17 +218,20 @@ int main(int argc, char **argv) { REQUIRE_XR(xrEnumerateInstanceExtensionProperties(nullptr, 0, &extension_count, nullptr)); std::vector extensions(extension_count, {XR_TYPE_EXTENSION_PROPERTIES}); REQUIRE_XR(xrEnumerateInstanceExtensionProperties(nullptr, extension_count, &extension_count, extensions.data())); - bool has_graphics = false; - for (const auto &extension : extensions) + bool has_graphics = false, has_hands = false; + for (const auto &extension : extensions) { has_graphics = has_graphics || !std::strcmp(extension.extensionName, gfx->extensionName()); + has_hands = has_hands || !std::strcmp(extension.extensionName, XR_EXT_HAND_TRACKING_EXTENSION_NAME); + } CHECK(has_graphics, "the runtime does not offer %s", gfx->extensionName()); // ---- instance and system ------------------------------------------------------------------------------------ - const char *enabled[] = {gfx->extensionName()}; + // hand tracking, when the runtime offers it: the simulated headset holds open hands where its controllers are + const char *enabled[] = {gfx->extensionName(), XR_EXT_HAND_TRACKING_EXTENSION_NAME}; XrInstanceCreateInfo instance_info{XR_TYPE_INSTANCE_CREATE_INFO}; std::snprintf(instance_info.applicationInfo.applicationName, XR_MAX_APPLICATION_NAME_SIZE, "xr_smoke"); instance_info.applicationInfo.apiVersion = XR_API_VERSION_1_0; - instance_info.enabledExtensionCount = 1; + instance_info.enabledExtensionCount = has_hands ? 2 : 1; instance_info.enabledExtensionNames = enabled; XrInstance instance = XR_NULL_HANDLE; REQUIRE_XR(xrCreateInstance(&instance_info, &instance)); @@ -248,8 +251,21 @@ int main(int argc, char **argv) { XrSystemId system = XR_NULL_SYSTEM_ID; REQUIRE_XR(xrGetSystem(instance, &system_info, &system)); XrSystemProperties system_properties{XR_TYPE_SYSTEM_PROPERTIES}; + XrSystemHandTrackingPropertiesEXT hand_properties{XR_TYPE_SYSTEM_HAND_TRACKING_PROPERTIES_EXT}; + if (has_hands) + system_properties.next = &hand_properties; REQUIRE_XR(xrGetSystemProperties(instance, system, &system_properties)); - std::printf("system: %s\n", system_properties.systemName); + std::printf("system: %s%s\n", system_properties.systemName, + hand_properties.supportsHandTracking ? " (tracks hands)" : ""); + PFN_xrCreateHandTrackerEXT xrCreateHandTrackerEXT = nullptr; + PFN_xrDestroyHandTrackerEXT xrDestroyHandTrackerEXT = nullptr; + PFN_xrLocateHandJointsEXT xrLocateHandJointsEXT = nullptr; + const bool hands_wanted = has_hands && hand_properties.supportsHandTracking; + if (hands_wanted) { + REQUIRE_XR(get_proc(instance, "xrCreateHandTrackerEXT", reinterpret_cast(&xrCreateHandTrackerEXT))); + REQUIRE_XR(get_proc(instance, "xrDestroyHandTrackerEXT", reinterpret_cast(&xrDestroyHandTrackerEXT))); + REQUIRE_XR(get_proc(instance, "xrLocateHandJointsEXT", reinterpret_cast(&xrLocateHandJointsEXT))); + } uint32_t view_count = 0; XrViewConfigurationView config_views[2] = {{XR_TYPE_VIEW_CONFIGURATION_VIEW}, {XR_TYPE_VIEW_CONFIGURATION_VIEW}}; @@ -344,6 +360,15 @@ int main(int argc, char **argv) { attach.countActionSets = 1; attach.actionSets = &action_set; REQUIRE_XR(xrAttachSessionActionSets(session, &attach)); + XrHandTrackerEXT hand_trackers[2] = {XR_NULL_HANDLE, XR_NULL_HANDLE}; + for (int hand = 0; hand < 2 && hands_wanted; ++hand) { + XrHandTrackerCreateInfoEXT info{XR_TYPE_HAND_TRACKER_CREATE_INFO_EXT}; + info.hand = hand == 0 ? XR_HAND_LEFT_EXT : XR_HAND_RIGHT_EXT; + info.handJointSet = XR_HAND_JOINT_SET_DEFAULT_EXT; + REQUIRE_XR(xrCreateHandTrackerEXT(session, &info, &hand_trackers[hand])); + } + uint32_t hands_seen[2] = {0, 0}; + float worst_wrist = 0; XrSpace grip_space[2] = {XR_NULL_HANDLE, XR_NULL_HANDLE}; for (int hand = 0; hand < 2; ++hand) { XrActionSpaceCreateInfo info{XR_TYPE_ACTION_SPACE_CREATE_INFO}; @@ -502,6 +527,25 @@ int main(int argc, char **argv) { (location.locationFlags & XR_SPACE_LOCATION_POSITION_VALID_BIT)) { ++grip_valid[hand]; last_grip[hand] = location.pose; + if (hand_trackers[hand]) { + // the hand's wrist is where its controller's grip is (the simulated headset's hands) + XrHandJointLocationEXT joints[XR_HAND_JOINT_COUNT_EXT]; + XrHandJointLocationsEXT located{XR_TYPE_HAND_JOINT_LOCATIONS_EXT}; + located.jointCount = XR_HAND_JOINT_COUNT_EXT; + located.jointLocations = joints; + XrHandJointsLocateInfoEXT where{XR_TYPE_HAND_JOINTS_LOCATE_INFO_EXT}; + where.baseSpace = local; + where.time = frame_state.predictedDisplayTime; + REQUIRE_XR(xrLocateHandJointsEXT(hand_trackers[hand], &where, &located)); + if (located.isActive && (joints[XR_HAND_JOINT_WRIST_EXT].locationFlags & XR_SPACE_LOCATION_POSITION_VALID_BIT)) { + ++hands_seen[hand]; + const XrVector3f w = joints[XR_HAND_JOINT_WRIST_EXT].pose.position; + const float d = std::sqrt((w.x - location.pose.position.x) * (w.x - location.pose.position.x) + + (w.y - location.pose.position.y) * (w.y - location.pose.position.y) + + (w.z - location.pose.position.z) * (w.z - location.pose.position.z)); + worst_wrist = std::fmax(worst_wrist, d); + } + } } } if (rendered == 10) { @@ -673,6 +717,15 @@ int main(int argc, char **argv) { REQUIRE_XR(xrDestroySwapchain(eyes[eye].swapchain)); if (use_quad) REQUIRE_XR(xrDestroySwapchain(quad_chain)); + if (hands_wanted) { + std::printf("hands located: left %u, right %u of %u frames; wrist at most %.3f m from the grip\n", hands_seen[0], + hands_seen[1], rendered, worst_wrist); + if (std::strstr(system_properties.systemName, "Simulated") && frames_wanted >= 100) + CHECK(hands_seen[0] > 0 && hands_seen[1] > 0 && worst_wrist < 0.03f, + "the simulated hands were not located where the controllers are"); + for (XrHandTrackerEXT tracker : hand_trackers) + REQUIRE_XR(xrDestroyHandTrackerEXT(tracker)); + } REQUIRE_XR(xrDestroySession(session)); { // a session destroyed with its state changes unread takes them with it: an event must not name a dead session