Repository navigation
feat: MetalFX spatial upscaling of the eyes in VR - #12
Merged
Merged
Conversation
MetalSharp's MetalFX toggle sets Mullion's d3d11.metalSpatialUpscale, which
Mullion applies only to DXGI swapchains; a VR game's eyes never pass through
one, so the toggle only upscaled the desktop mirror.
- runtime/upscale.{h,cpp}: METALVR_METALFX_SPATIAL=<factor|on|off>, or
Mullion's d3d11.metalSpatialUpscale(Factor) from MULLION_CONFIG_FILE (a
macOS path, opened through Wine when needed) and MULLION_CONFIG, with
Mullion's [exe] sections.
- The OpenXR runtime's recommended eye size and the SteamVR driver's
recommended render target size are divided by the factor; maximums stay.
- metalvrd scales each projection view smaller than its eye up with an
MTLFXSpatialScaler (perceptual for 8/10-bit, HDR for float), its rect
copied out first, mirrored rects flipped, then draws it 1:1. Bilinear
stays for GPUs without MetalFX, formats it refuses, and
METALVR_METALFX_SPATIAL=0. Quads and foveated encoding are unchanged.
- metalvrd --upscale-check, run by scripts/test-all.sh.
…ition SteamVR's projection layers come and go with its dashboard and overlays, and in Blade & Sorcery the larger of its two layer sizes was sometimes alone in a frame. Upscalers were kept per view position in the frame, so each change of layout made new scalers (and their textures) for every eye, logged as the sizes alternating, with a frame-rate dip after each burst. Each view now takes an upscaler of its own size and format that the frame has not taken yet, kept across frames and dropped after about ten seconds unused; each size is logged once. --upscale-check composes 20 frames alternating one layer and two of different sizes: the old code made 36 scalers after the first two frames, this one none.
multision
force-pushed
the
feat/metalfx-spatial-vr
branch
from
November 19, 2026 17:55
4ef98d6 to
d8255de
Compare
In Blade & Sorcery through SteamVR, both of SteamVR's projection layers were scaled: four MetalFX passes at 2144x2240 a frame, raising compose from about 1.4 to 4.5 ms. metalvrd composes while the application waits on its frame, so that pushed a game that held 72 frames a second past its frame time (about 44 fps). The layers over the bottom one are SteamVR's dashboard and overlays, blended UI that MetalFX is not made for; they are sampled as before. --upscale-check's changing-layers frames now also check the blended layer is not scaled. METALSHARP_WINE.md says where upscaling pays and where it does not.
…n's setting as Mullion does Review findings: - Upscalers were keyed by input size, so an application whose render size changes every frame inside a fixed swapchain (dynamic resolution; SteamVR's own bounds moved from 1312x1372 to 1010x1056 in Blade & Sorcery) made a new scaler and two eye-sized textures for every size, each kept ten seconds: gigabytes within seconds. Each eye now has one scaler of the eye's size and format, its input the eye's size too, told each frame how much of it the view fills (inputContentWidth/Height, as MetalFX means for dynamic resolution). --upscale-check composes larger eyes first, then checks the smaller ones pixel by pixel with no scaler made for the new size. - A texture that could not be made (memory pressure) turned MetalFX off for that pixel format until restart; only a format MetalFX refuses does now, and the textures are tried again next frame. - runtime/upscale.cpp now reads Mullion's setting as Mullion does: each MULLION_ variable answers to its DXMT_ name, MULLION_METALFX_SPATIAL_SWAPCHAIN=1 turns it on, mullion.conf or dxmt.conf in the working directory is read when no file is named, numbers are parsed with Mullion's own locale-free rules, and on Windows the variables and the executable's name are read wide and turned into UTF-8 (a home folder with an accent opened nothing before). - METALVR_METALFX_SPATIAL=on with Mullion's factor at 1 means factor 2, and the runtime and driver log when upscaling is on even at factor 1, where it changes nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MetalSharp's MetalFX Spatial toggle works in flat games but does nothing in VR games. It sets Mullion's
d3d11.metalSpatialUpscale, and Mullion applies that only to DXGI swapchains. A VR game's eyes go to MetalVR's OpenXR swapchains or its SteamVR driver's textures instead, so the toggle only upscaled the desktop mirror window. This adds the VR half in MetalVR:Setting (
runtime/upscale.{h,cpp}):METALVR_METALFX_SPATIAL=<factor>;onuses Mullion's factor, or 2 if that is 1, and0/offturns it off. When it is unset, MetalVR reads Mullion's own setting the way Mullion does:d3d11.metalSpatialUpscaleorMULLION_METALFX_SPATIAL_SWAPCHAIN=1, atd3d11.metalSpatialUpscaleFactor, parsed with Mullion's locale-free rules;MULLION_CONFIG_FILEnames (elsemullion.confordxmt.confin the working directory) and fromMULLION_CONFIG, honouring Mullion's[game.exe]sections, with eachMULLION_variable also answering to itsDXMT_name;wine_get_dos_file_name.A factor of 1, which is MetalSharp's default, changes nothing, and the logs say so.
Render size:
xrEnumerateViewConfigurationViewsrecommends the headset's eye size divided by the factor, and the max sizes are unchanged. The SteamVR driver'sGetRecommendedRenderTargetSizedoes the same. Both log the size once.Compositor (
daemon/compositor.mm): when the bottom layer is a projection whose views are smaller than the output eye, each view is scaled with anMTLFXSpatialScaler, then drawn 1:1. Details:compositor: MetalFX spatial upscaling 960x1056 to 1440x1584 per eye (...).supportsDevice:is false, when MetalFX refuses a format, and withMETALVR_METALFX_SPATIAL=0in metalvrd's environment.metalvrd --upscale-check, added toscripts/test-all.sh, checks that both eyes go through MetalFX and every pixel lands where it belongs. The eyes are rendered at 2/3 size, side by side in one texture with the right eye's rect off the origin and mirrored, in BGRA8 sRGB and RGBA16Float. On a GPU without MetalFX it prints SKIPPED.MetalFX is linked in
daemon/CMakeLists.txt. Docs:docs/METALSHARP_WINE.md(new section MetalFX upscaling in VR),docs/BUILDING.md,docs/STEAMVR.md.The driver change is two small hunks in
steamvr/driver_metalvr.cpp(GetRecommendedRenderTargetSizeand one log line). They should not conflict with #11.Testing
build/nativewith ALVR v20.14.1 linked,build/win64), no warnings. Earlier also built by hand with the CMake flags.scripts/test-native.sh:METALVR_METALFX_SPATIAL=1.5: the runtime recommends 960x1056 for the 1440x1584 sim headset, metalvrd logsMetalFX spatial upscaling 960x1056 to 1440x1584 per eye, and xr_smoke, stream_check and frame_check pass.--flip,--unorm,--array, and with--foveated-encoding --codec hevc.scripts/test-wine.shon MetalSharp-Wine x86_64 (the copy MetalSharp installs in~/.metalsharp/runtime) with Mullion, Direct3D 11. It passes with upscaling switched on either way:METALVR_METALFX_SPATIAL=1.5, or a Mullion config that hasd3d11.metalSpatialUpscale = Trueand factor 1.50. Either way the game gets 960x1056 eyes and metalvrd scales them up with MetalFX. (The build tree had to be copied to a path without spaces: the script's--runtimepath splits on a space under MetalSharp's launcher. That problem predates this change.)metalvrd --upscale-checkand--foveation-checkpass on Apple silicon.--upscale-checkchecks the pixels after frames of a larger size, so the scalers are reused with a new content size, and finds no scaler made for the new size. It also composes 20 frames that alternate between one projection layer and two of different sizes. The first commit's code made 36 scalers after the first two frames; the current code makes none and leaves the blended layer unscaled..5(default 2, as Mullion), sections,MULLION_CONFIG,DXMT_CONFIG_FILE,DXMT_METALFX_SPATIAL_SWAPCHAIN,mullion.conf/dxmt.confin the working directory, env overrides, a comma-decimal locale, and an unparseable value. The Windows build also ran under MetalSharp-Wine with a config path containing a non-ASCII character.scripts/test-failures.sh: all four cases pass, with and withoutMETALVR_METALFX_SPATIAL=1.5. (Run through a build path without spaces: the script word-splits the client's path. That problem predates this change.)scripts/test-openvr.shwith OpenComposite cff07db (scripts/build-opencomposite.sh), on MetalSharp-Wine with Mullion: all five cases pass. WithMETALVR_METALFX_SPATIAL=1.5, each case was recommended 960x1056 and scaled back to 1440x1584 by MetalFX.Checklist
patches/, not edits inexternal/(none)