Skip to content

feat: MetalFX spatial upscaling of the eyes in VR - #12

Merged
aaf2tbz merged 4 commits into
metalsharp:mainfrom
multision:feat/metalfx-spatial-vr
Nov 20, 2026
Merged

aaf2tbz merged 4 commits into
metalsharp:mainfrom
multision:feat/metalfx-spatial-vr

Conversation

@multision

@multision multision commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

MetalSharp's MetalFX Spatial toggle works in flat games but does nothing in VR games. It sets Mullion's d3d11.metalSpatialUpscale, and Mullion applies that only to DXGI swapchains. A VR game's eyes go to MetalVR's OpenXR swapchains or its SteamVR driver's textures instead, so the toggle only upscaled the desktop mirror window. This adds the VR half in MetalVR:

  • Setting (runtime/upscale.{h,cpp}): METALVR_METALFX_SPATIAL=<factor>; on uses Mullion's factor, or 2 if that is 1, and 0/off turns it off. When it is unset, MetalVR reads Mullion's own setting the way Mullion does:

    • on with d3d11.metalSpatialUpscale or MULLION_METALFX_SPATIAL_SWAPCHAIN=1, at d3d11.metalSpatialUpscaleFactor, parsed with Mullion's locale-free rules;
    • from the file MULLION_CONFIG_FILE names (else mullion.conf or dxmt.conf in the working directory) and from MULLION_CONFIG, honouring Mullion's [game.exe] sections, with each MULLION_ variable also answering to its DXMT_ name;
    • under Wine, variables and the exe name are read wide as UTF-8, and a macOS path is opened through wine_get_dos_file_name.

    A factor of 1, which is MetalSharp's default, changes nothing, and the logs say so.

  • Render size: xrEnumerateViewConfigurationViews recommends the headset's eye size divided by the factor, and the max sizes are unchanged. The SteamVR driver's GetRecommendedRenderTargetSize does the same. Both log the size once.

  • Compositor (daemon/compositor.mm): when the bottom layer is a projection whose views are smaller than the output eye, each view is scaled with an MTLFXSpatialScaler, then drawn 1:1. Details:

    • The view's rect is first blit-copied out of the shared texture, because MetalFX reads from the origin.
    • sRGB formats go to MetalFX as their UNORM view in perceptual mode. Other 8/10-bit formats also use perceptual mode, and float formats use HDR mode.
    • Mirrored rects are flipped when drawn.
    • Each eye has one scaler of its format and the eye's size, with an eye-sized input, and MetalFX is told each frame how much of it the view fills. Layers that come and go (SteamVR's dashboard) and dynamic resolution reuse the same scalers. Unused ones are dropped after about ten seconds. A format MetalFX refuses is drawn bilinearly from then on; a texture that can't be allocated is retried the next frame.
    • Each new size is logged once: compositor: MetalFX spatial upscaling 960x1056 to 1440x1584 per eye (...).
    • The bilinear path is kept when supportsDevice: is false, when MetalFX refuses a format, and with METALVR_METALFX_SPATIAL=0 in metalvrd's environment.
    • Layers blended over the bottom one are sampled as before. In SteamVR these are the dashboard and overlays: blended UI that MetalFX isn't made for, and scaling them too doubled the cost.
    • Quad layers and foveated encoding are unchanged. With foveated encoding on, eyes are upscaled to its composed eye size and then squeezed as before.
  • metalvrd --upscale-check, added to scripts/test-all.sh, checks that both eyes go through MetalFX and every pixel lands where it belongs. The eyes are rendered at 2/3 size, side by side in one texture with the right eye's rect off the origin and mirrored, in BGRA8 sRGB and RGBA16Float. On a GPU without MetalFX it prints SKIPPED.

  • MetalFX is linked in daemon/CMakeLists.txt. Docs: docs/METALSHARP_WINE.md (new section MetalFX upscaling in VR), docs/BUILDING.md, docs/STEAMVR.md.

The driver change is two small hunks in steamvr/driver_metalvr.cpp (GetRecommendedRenderTargetSize and one log line). They should not conflict with #11.

Testing

  • Built for macOS and for Windows with CMake (build/native with ALVR v20.14.1 linked, build/win64), no warnings. Earlier also built by hand with the CMake flags.
  • scripts/test-native.sh:
    • METALVR_METALFX_SPATIAL=1.5: the runtime recommends 960x1056 for the 1440x1584 sim headset, metalvrd logs MetalFX spatial upscaling 960x1056 to 1440x1584 per eye, and xr_smoke, stream_check and frame_check pass.
    • The same with --flip, --unorm, --array, and with --foveated-encoding --codec hevc.
    • The baseline without the variable still recommends 1440x1584 and passes.
  • scripts/test-wine.sh on MetalSharp-Wine x86_64 (the copy MetalSharp installs in ~/.metalsharp/runtime) with Mullion, Direct3D 11. It passes with upscaling switched on either way: METALVR_METALFX_SPATIAL=1.5, or a Mullion config that has d3d11.metalSpatialUpscale = True and factor 1.50. Either way the game gets 960x1056 eyes and metalvrd scales them up with MetalFX. (The build tree had to be copied to a path without spaces: the script's --runtime path splits on a space under MetalSharp's launcher. That problem predates this change.)
  • metalvrd --upscale-check and --foveation-check pass on Apple silicon. --upscale-check checks the pixels after frames of a larger size, so the scalers are reused with a new content size, and finds no scaler made for the new size. It also composes 20 frames that alternate between one projection layer and two of different sizes. The first commit's code made 36 scalers after the first two frames; the current code makes none and leaves the blended layer unscaled.
  • On a headset (Quest 3 over ALVR, MetalSharp's toggle at factor 2.00). The first three commits were tried there; the fourth was tested only on the Mac:
    • BONELAB (OpenXR) rendered 1072x1120 to 1180x1232 per eye instead of 2144x2240 to 2360x2464, and went from about 28 fps on average to a steady 59–60 once playing. MetalFX added about 2 ms per frame to metalvrd's compose time.
    • Blade & Sorcery (SteamVR): the driver recommended 1072x1120, but SteamVR's own resolution setting raised that to 1312x1372 and 1608x1680. Changes in its layer layout rebuilt the scalers; the second commit fixes that, and a later headset run showed each size set up once.
    • B&S still ran at about 44 fps with MetalFX, against 70–72 fps without it at 1740x1820. Both of SteamVR's layers were being scaled: four passes per frame, raising compose time from about 1.4 to 4.5 ms. metalvrd composes while the application waits on its frame, so the game missed its frame time. The third commit scales only the bottom layer, and with it B&S held a steady 72 fps on the headset.
    • Upscaling pays only in a game held back by the size it renders at (BONELAB). It costs about 1 ms per eye in the application's frame time, so a game that already holds 72 at full size (B&S) gets slower. Composing without the application waiting would remove that cost; that is a larger change to frame pacing and is not in this PR.
  • Unit check of the setting reader. Cases: on, off, factor 1, .5 (default 2, as Mullion), sections, MULLION_CONFIG, DXMT_CONFIG_FILE, DXMT_METALFX_SPATIAL_SWAPCHAIN, mullion.conf/dxmt.conf in the working directory, env overrides, a comma-decimal locale, and an unparseable value. The Windows build also ran under MetalSharp-Wine with a config path containing a non-ASCII character.
  • scripts/test-failures.sh: all four cases pass, with and without METALVR_METALFX_SPATIAL=1.5. (Run through a build path without spaces: the script word-splits the client's path. That problem predates this change.)
  • scripts/test-openvr.sh with OpenComposite cff07db (scripts/build-opencomposite.sh), on MetalSharp-Wine with Mullion: all five cases pass. With METALVR_METALFX_SPATIAL=1.5, each case was recommended 960x1056 and scaled back to 1440x1584 by MetalFX.

Checklist

  • The change is my own work, or open-source code whose license allows it here; nothing is taken from a closed-source product
  • Changes to ALVR or OpenComposite are patches under patches/, not edits in external/ (none)
  • New downloads are pinned to a commit (none)
  • Documentation is updated where behaviour, options or the protocol change
  • No Wine prefixes, logs with personal paths, or other personal data

MetalSharp's MetalFX toggle sets Mullion's d3d11.metalSpatialUpscale, which
Mullion applies only to DXGI swapchains; a VR game's eyes never pass through
one, so the toggle only upscaled the desktop mirror.

- runtime/upscale.{h,cpp}: METALVR_METALFX_SPATIAL=<factor|on|off>, or
  Mullion's d3d11.metalSpatialUpscale(Factor) from MULLION_CONFIG_FILE (a
  macOS path, opened through Wine when needed) and MULLION_CONFIG, with
  Mullion's [exe] sections.
- The OpenXR runtime's recommended eye size and the SteamVR driver's
  recommended render target size are divided by the factor; maximums stay.
- metalvrd scales each projection view smaller than its eye up with an
  MTLFXSpatialScaler (perceptual for 8/10-bit, HDR for float), its rect
  copied out first, mirrored rects flipped, then draws it 1:1. Bilinear
  stays for GPUs without MetalFX, formats it refuses, and
  METALVR_METALFX_SPATIAL=0. Quads and foveated encoding are unchanged.
- metalvrd --upscale-check, run by scripts/test-all.sh.
…ition

SteamVR's projection layers come and go with its dashboard and overlays, and
in Blade & Sorcery the larger of its two layer sizes was sometimes alone in a
frame. Upscalers were kept per view position in the frame, so each change of
layout made new scalers (and their textures) for every eye, logged as the
sizes alternating, with a frame-rate dip after each burst.

Each view now takes an upscaler of its own size and format that the frame has
not taken yet, kept across frames and dropped after about ten seconds
unused; each size is logged once. --upscale-check composes 20 frames
alternating one layer and two of different sizes: the old code made 36
scalers after the first two frames, this one none.
@multision
multision force-pushed the feat/metalfx-spatial-vr branch from 4ef98d6 to d8255de Compare November 19, 2026 17:55
In Blade & Sorcery through SteamVR, both of SteamVR's projection layers were
scaled: four MetalFX passes at 2144x2240 a frame, raising compose from about
1.4 to 4.5 ms. metalvrd composes while the application waits on its frame, so
that pushed a game that held 72 frames a second past its frame time (about 44
fps). The layers over the bottom one are SteamVR's dashboard and overlays,
blended UI that MetalFX is not made for; they are sampled as before.

--upscale-check's changing-layers frames now also check the blended layer is
not scaled. METALSHARP_WINE.md says where upscaling pays and where it does not.
…n's setting as Mullion does

Review findings:

- Upscalers were keyed by input size, so an application whose render size
  changes every frame inside a fixed swapchain (dynamic resolution; SteamVR's
  own bounds moved from 1312x1372 to 1010x1056 in Blade & Sorcery) made a new
  scaler and two eye-sized textures for every size, each kept ten seconds:
  gigabytes within seconds. Each eye now has one scaler of the eye's size and
  format, its input the eye's size too, told each frame how much of it the
  view fills (inputContentWidth/Height, as MetalFX means for dynamic
  resolution). --upscale-check composes larger eyes first, then checks the
  smaller ones pixel by pixel with no scaler made for the new size.
- A texture that could not be made (memory pressure) turned MetalFX off for
  that pixel format until restart; only a format MetalFX refuses does now, and
  the textures are tried again next frame.
- runtime/upscale.cpp now reads Mullion's setting as Mullion does: each
  MULLION_ variable answers to its DXMT_ name, MULLION_METALFX_SPATIAL_SWAPCHAIN=1
  turns it on, mullion.conf or dxmt.conf in the working directory is read when
  no file is named, numbers are parsed with Mullion's own locale-free rules,
  and on Windows the variables and the executable's name are read wide and
  turned into UTF-8 (a home folder with an accent opened nothing before).
- METALVR_METALFX_SPATIAL=on with Mullion's factor at 1 means factor 2, and
  the runtime and driver log when upscaling is on even at factor 1, where it
  changes nothing.
@multision
multision marked this pull request as ready for review November 19, 2026 19:05
@multision
multision requested a review from aaf2tbz as a code owner November 19, 2026 19:05
@aaf2tbz
aaf2tbz merged commit 1c2496b into metalsharp:main Nov 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants