Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ body:
id: headset
attributes:
label: Headset and client
description: For streaming. The headset, the ALVR client version, and USB or Wi-Fi.
description: The headset and how it is connected (the Mac's own display and USB, or streaming); for streaming, the ALVR client version and USB or Wi-Fi.
placeholder: Quest 3, ALVR client v20.14.1, Wi-Fi
- type: textarea
id: logs
Expand Down
3 changes: 3 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ is built and what is missing.

- **A game that does not work, or does**: open a *Game report*. It asks for the game, the Wine and Mullion
versions, the headset and the logs. If the game is broken without VR too, it is a Mullion or Wine matter.
- **A headset MetalVR has not met yet**: follow its guide in [docs/headsets](docs/headsets/README.md) and open a
*Bug report* saying how far it got, with the server's log (`metalvrd --verbose`). Each guide lists what is still
unknown about that headset, which is what such a report answers.
- **Bugs**: open a *Bug report* with the commit you built, the Mac and macOS version, and the output of
`metalvrd --verbose` and of the application with `METALVR_LOG=1`.
- **Security issues**: never in a public issue. See [SECURITY.md](SECURITY.md).
Expand Down
119 changes: 34 additions & 85 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,62 +2,30 @@

[![CI](https://github.com/metalsharp/MetalVR/actions/workflows/ci.yml/badge.svg)](https://github.com/metalsharp/MetalVR/actions/workflows/ci.yml)

An open-source OpenXR runtime for Apple silicon Macs.
An open-source OpenXR runtime for Apple silicon Macs. PC VR games run under Wine with Direct3D translated to Metal by
[Mullion](https://github.com/dttdrv/mullion); MetalVR is the runtime they talk to and the server that shows their
frames on a headset. SteamVR games reach it through MetalVR's SteamVR driver, and native macOS OpenXR applications
through a Metal runtime.

PC VR games run under Wine with D3D12/11 translated to Metal by [Mullion](https://github.com/dttdrv/mullion).
MetalVR is the OpenXR runtime those games talk to, and the native server that composes their frames and shows
them on a headset: an Oculus Rift CV1 plugged into the Mac itself, or a standalone headset running the stock
[ALVR](https://github.com/alvr-org/ALVR) client, streamed with hardware encoding. SteamVR games reach it through
MetalVR's SteamVR driver or [OpenComposite](https://gitlab.com/znixian/OpenOVR), and native macOS OpenXR
applications use the same server through a Metal runtime.
**Status:** an Oculus Rift CV1 on the Mac's own HDMI and USB plays SteamVR games (Subnautica, ABE VR, Raw Data) with
its sensors and Touch controllers. Every other headset below is built and checked without the headset, and has not
met one yet: reports from owners are very welcome.

Current Status;
## Headsets

Testing is happening with an Oculus Rift CV1. First we had to create a vr server and apply specific commands for MacOS to recognize and load the headset display. We tied this to a 'Start VR Server' button on a dedicated MetalVR launch route. We tied Steam Launches to launch with mullion also, since this program relies on Mullion to load it's driver for d3d11/12. After this, we were able to get SteamVR to recognize the headset and render the tutorial through the headsets display at 90hz, and the headsets resolution. After more tests we were able to launch Subnautica through steam with SteamVR opened and the VR Sever running. BeamNG also was able to use the headset but currently experiences a crash in its driver.
| Headset | Tried on hardware | Guide |
|---|---|---|
| Oculus Rift CV1 | yes | [docs/headsets/RIFT_CV1.md](docs/headsets/RIFT_CV1.md) |
| Oculus Rift S | no | [docs/headsets/RIFT_S.md](docs/headsets/RIFT_S.md) |
| Meta Quest 1, 2, 3, 3S, Pro (ALVR) | no | [docs/headsets/QUEST.md](docs/headsets/QUEST.md) |
| Valve Steam Frame (ALVR) | no | [docs/headsets/STEAM_FRAME.md](docs/headsets/STEAM_FRAME.md) |
| Valve Index | no | [docs/headsets/VALVE_INDEX.md](docs/headsets/VALVE_INDEX.md) |
| HTC Vive, Vive Pro, Vive Pro 2 | no | [docs/headsets/HTC_VIVE.md](docs/headsets/HTC_VIVE.md) |
| Sony PlayStation VR2 | no | [docs/headsets/PSVR2.md](docs/headsets/PSVR2.md) |

Since then, the Rift's Touch controllers show in the headset and track, all three sensors (one of them on USB 2) track the headset and the controllers in position and show in SteamVR as base stations, and the flashing coloured blocks in the VR composition are fixed. ABE VR and Raw Data play through SteamVR.
Each guide covers what you need, setting up and playing, what is checked in code, and what is still unknown.

Still to do;

- Optimize the experience for other headsets, such as CV2, Oculus Rift S, Meta Quest 1/2/3, Valve Index, Steam Frame.
- BeamNG's crash in its driver.

Contribution is very welcome because I do not own every headset. The experience will be optimized for an Oculus Rift Cv1 first, followed by conclusive updates from users and myself where applicable.

## Oculus Rift CV1

A Rift CV1 on the Mac's own HDMI and USB, with its sensors and Touch controllers, needs no Oculus software. The
release's `metalvr-rift` script does each step:

```sh
metalvr-rift install # from the unpacked release: into ~/.metalsharp/metalvr-runtime
metalvr-rift display # once per Mac: let macOS show the headset's display (asks for your password)
metalvr-rift setup # once: register MetalVR's driver with SteamVR in MetalSharp's Steam
metalvr-rift steamvr # start the Rift server, then SteamVR
metalvr-rift game APPID # start the Rift server, then a Steam game in VR
```

**[docs/RIFT_CV1.md](docs/RIFT_CV1.md)** walks through it: what you need, setting up, playing, and what to do when
the headset stays dark. SteamVR and its games need MetalSharp-Wine's Mullion at
[aaf2tbz/mullion@998c32a6](https://github.com/aaf2tbz/mullion/commit/998c32a61095985e7034c7b56553d79aa81c7193) or
later, which shares Direct3D 11 buffers between processes for VR; `metalvr-rift setup` switches that on
(`MULLION_METALVR=1`).

## How it works

```
game.exe ─ OpenXR ─▶ metalvr_openxr.dll ─┐ ┌─▶ simulated headset (tests)
(Direct3D 11 via Mullion) ├─ loopback ─▶ metalvrd ──┤
mac app ── OpenXR ─▶ libmetalvr_openxr ──┘ compose · encode └─▶ ALVR ─▶ headset
(Metal)
```

- One runtime source tree builds as a Windows DLL and a macOS dylib.
- Swapchain images never leave the GPU: under Wine they are Mullion shared textures, natively they are
IOSurfaces.
- `metalvrd` composes both eyes into one frame and encodes H.264 or HEVC with VideoToolbox.

## Quick start
## Building

```sh
scripts/fetch-deps.sh
Expand All @@ -66,50 +34,31 @@ cmake -S . -B build/win64 -G Ninja -DCMAKE_TOOLCHAIN_FILE=cmake/mingw-x86_64.cma
scripts/test-all.sh
```

Then run the server and make MetalVR a Wine prefix's OpenXR runtime:

```sh
build/native/daemon/metalvrd --trust 1234.client # the name the headset's ALVR app shows
scripts/install-prefix.sh /path/to/wine /path/to/prefix
```

[docs/BUILDING.md](docs/BUILDING.md) has the requirements and the rest.
Requirements, the optional parts (ALVR, libsurvive, Basalt, OpenComposite) and the test suites:
[docs/BUILDING.md](docs/BUILDING.md).

## Documentation

| | |
|---|---|
| [Headset guides](docs/headsets/README.md) | one per headset |
| [Building, running and testing](docs/BUILDING.md) | requirements, builds, options, the test suites |
| [Architecture](docs/ARCHITECTURE.md) | the runtime, the server, the protocol between them, and what is not there yet |
| [MetalSharp-Wine integration](docs/METALSHARP_WINE.md) | what a Wine distribution ships and sets, and the contract with Mullion |
| [Headset streaming with ALVR](docs/ALVR.md) | building the driver, pairing a headset, how ALVR's API is used |
| [SteamVR games with OpenComposite](docs/OPENCOMPOSITE.md) | building `openvr_api.dll`, installing it for a game, known gaps |
| [Oculus Rift CV1 with MetalSharp](docs/RIFT_CV1.md) | the `metalvr-rift` script: from a release to a game in the headset |
| [Oculus Rift CV1 driver](docs/RIFT.md) | how the CV1 driver works: display, IMU, sensors, tracking, pacing |
| [Architecture](docs/ARCHITECTURE.md) | the runtime, the server, the protocol between them, what works and what is not there yet |
| [SteamVR](docs/STEAMVR.md) | MetalVR's SteamVR driver: direct mode, requirements, its messages |

## What works

- OpenXR 1.0 and 1.1 core: sessions, spaces, swapchains, the frame loop, projection and quad layers, actions,
haptics.
- `XR_KHR_D3D11_enable` under Wine with Mullion, and `XR_KHR_metal_enable` on macOS.
- Hardware H.264 and HEVC encoding, at a headset's resolution and refresh rate.
- OpenVR applications through OpenComposite: rendering, legacy and action-manifest input, haptics.
- The ALVR driver, against ALVR's own client networking code on one machine.
- An Oculus Rift CV1 on the Mac's HDMI and USB: its display with lens correction, head rotation from its IMU,
position tracking with up to four Rift Sensors (USB 3 or USB 2), Touch controllers, and SteamVR through
MetalVR's own driver.

Not yet: a standalone headset over ALVR in practice, Direct3D 12 / Vulkan / OpenGL applications, hand tracking,
audio, 32-bit games.
| [The direct-display driver](docs/RIFT.md) | how headsets on the Mac's own display are driven: display, IMU, sensors, pacing |
| [ALVR](docs/ALVR.md) | streaming to standalone headsets: the driver, pairing, MetalVR's patches |
| [OpenComposite](docs/OPENCOMPOSITE.md) | SteamVR games without SteamVR |
| [MetalSharp-Wine integration](docs/METALSHARP_WINE.md) | what a Wine distribution ships and sets, and the contract with Mullion |
| [Headset research](docs/HEADSETS.md) | each headset's specifications, how it is recognized, and the sources |

## Contributing

Game reports, bug reports and pull requests are welcome: see [CONTRIBUTING.md](CONTRIBUTING.md). Report
security issues privately, as [SECURITY.md](SECURITY.md) describes.
Game reports, headset reports, bug reports and pull requests are welcome: see [CONTRIBUTING.md](CONTRIBUTING.md).
Report security issues privately, as [SECURITY.md](SECURITY.md) describes. Known problems: BeamNG.drive crashes in its
graphics driver.

## License

MIT; see [LICENSE](LICENSE). MetalVR is written from scratch against public specifications and open-source
code. The components it builds against or beside keep their own licenses, and the patches under `patches/`
carry the license of what they patch: see [patches/README.md](patches/README.md).
MIT; see [LICENSE](LICENSE). MetalVR is written from scratch against public specifications and open-source code. The
components it builds against or beside keep their own licenses, and the patches under `patches/` carry the license of
what they patch: see [patches/README.md](patches/README.md).
57 changes: 57 additions & 0 deletions daemon/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,64 @@ add_executable(metalvrd
encoder.mm
shared_texture.mm
driver_sim.cpp
headsets.cpp
foveation.cpp
json_value.cpp
driver_rift.mm
direct_models.cpp
backend_rift_s.mm
rift_s_protocol.cpp
backend_lighthouse.mm
backend_psvr2.mm
psvr2_protocol.cpp
lighthouse_protocol.cpp
lighthouse_tracking.cpp
camera_passthrough.mm
camera_tracking.cpp
rift_s_tracking.cpp
rift_s_cameras.mm
window_capture.mm
)

# The Rift S's camera tracking loads Basalt (scripts/build-basalt.sh) when it starts, as Monado does: looked for in
# this build's output first, then next to metalvrd. Nothing is linked.
set(METALVR_BASALT_LIBRARY_DIR "${CMAKE_SOURCE_DIR}/build/basalt/prefix/lib" CACHE PATH
"where scripts/build-basalt.sh puts libbasalt.dylib")
set_property(SOURCE camera_tracking.cpp APPEND PROPERTY COMPILE_DEFINITIONS
"METALVR_BASALT_DIR=\"${METALVR_BASALT_LIBRARY_DIR}\"")

# Position tracking for the Lighthouse headsets with their base stations is libsurvive (MIT) on hidapi (BSD), which
# scripts/build-libsurvive.sh builds statically into build/libsurvive. Its parts register themselves as the program
# starts, so each is linked whole. Without it the Lighthouse driver tracks rotation only.
set(METALVR_LIBSURVIVE_DIR "${CMAKE_SOURCE_DIR}/build/libsurvive" CACHE PATH "scripts/build-libsurvive.sh's output")
if(EXISTS "${METALVR_LIBSURVIVE_DIR}/build/src/libsurvive.a")
set(libsurvive_default ON)
else()
set(libsurvive_default OFF)
endif()
option(METALVR_WITH_LIBSURVIVE "Track the Lighthouse headsets' position with libsurvive" ${libsurvive_default})
if(METALVR_WITH_LIBSURVIVE)
set(ls "${METALVR_LIBSURVIVE_DIR}/build")
set(ls_src "${CMAKE_SOURCE_DIR}/external/libsurvive")
if(NOT EXISTS "${ls}/src/libsurvive.a")
message(FATAL_ERROR "METALVR_WITH_LIBSURVIVE needs ${ls}; run scripts/build-libsurvive.sh")
endif()
message(STATUS "metalvrd: with libsurvive for Lighthouse positions (${ls})")
set_source_files_properties(lighthouse_tracking.cpp PROPERTIES COMPILE_DEFINITIONS METALVR_LIBSURVIVE=1)
target_include_directories(metalvrd PRIVATE "${ls_src}/include/libsurvive" "${ls_src}/include" "${ls_src}/redist"
"${ls_src}/libs/cnmatrix/include" "${ls_src}/libs/cnkalman/include"
"${METALVR_LIBSURVIVE_DIR}/prefix/include/hidapi")
foreach(plugin driver_vive driver_simulator disambiguator_statebased poser_kalman_only poser_mpfit
driver_global_scene_solver)
target_link_options(metalvrd PRIVATE "-Wl,-force_load,${ls}/src/lib${plugin}.a")
endforeach()
target_link_libraries(metalvrd PRIVATE "${ls}/src/libsurvive.a" "${ls}/redist/libmpfit.a"
"${ls}/libs/cnkalman/src/libcnkalman.a" "${ls}/libs/cnmatrix/src/libcnmatrix.a"
"${METALVR_LIBSURVIVE_DIR}/prefix/lib/libhidapi.a")
else()
message(STATUS "metalvrd: without libsurvive (scripts/build-libsurvive.sh builds it); Lighthouse rotation only")
endif()

if(METALVR_WITH_ALVR)
if(NOT EXISTS "${METALVR_ALVR_LIBRARY}")
message(FATAL_ERROR "METALVR_WITH_ALVR needs ${METALVR_ALVR_LIBRARY}; run scripts/build-alvr.sh")
Expand Down Expand Up @@ -131,6 +185,9 @@ target_link_libraries(metalvrd PRIVATE
"-framework UniformTypeIdentifiers"
"-framework AppKit"
"-framework IOKit"
"-framework IOUSBHost"
"-framework QuartzCore"
"-framework ScreenCaptureKit"
"-framework AVFoundation"
z
)
Loading
Loading