Skip to content

feat(examples): add a GPU hillshade of Mapterhorn PMTiles - #817

Merged
kylebarron merged 31 commits into
mainfrom
kyle/pmtiles-terrain-example
Oct 28, 2026
Merged

kylebarron merged 31 commits into
mainfrom
kyle/pmtiles-terrain-example

Conversation

@kylebarron

@kylebarron kylebarron commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Note

This PR was written by Claude (Claude Code), not by @kylebarron.

Adds examples/experimental/pmtiles-terrain. It computes relief shading on the GPU from Mapterhorn's global terrain, read straight from one PMTiles archive by RasterTileLayer. The archive is planet.pmtiles: 356 GB covering zooms 0-12, as Terrarium-encoded 512 px lossless WebP.

It shows three things:

  1. deck.gl-raster reads PMTiles too. Any tiled source works, not only COG and Zarr.
  2. Hillshade is computed live on the GPU. The sun is a slider, and moving it fetches nothing.
  3. Reads are merged. A clustered PMTiles archive keeps a view's tiles together in the file, so the range reader merges them into a few streamed requests. A "Merge requests" switch compares that with one request per tile.

This becomes the hillshade example. It ports and reviews #647's Horn kernel, halo and modes; #647 stays parked. Ref #647.

The design is in dev-docs/specs/2026-10-27-pmtiles-terrain-design.md and the example's README.

Built on the range reader

It reads PMTiles through @developmentseed/range-reader (#806) and uses #809's protected _abortUnwantedLoads hook, both on main. Outside the example, the PR changes only the spec, one docs entry, the card image and lockfile additions.

How it works

  • No loaders.gl. The example uses the pmtiles library directly. @loaders.gl/pmtiles takes only a URL or a Blob, so its reads can't merge; it calls getZxy without a signal; and it pins pmtiles 3.x.
  • RangeReaderSource implements pmtiles' Source on the range reader.
    • Tile-data reads use the data purpose; the header and directories use metadata.
    • Bytes are copied out, because the reader's are views into shared buffers.
    • A replaced archive is caught with pmtiles' own etag check.
  • PMTilesSource extends RasterSource describes a 512 px WebMercatorQuad grid bounded to the archive.
    • The layer takes its tilesetDescriptor; question 10 of the Sources spec stays as it is.
    • If RasterTileLayer learns to take a source, this plugs in there.
  • Halo. Each tile is a 514 x 514 r32float texture whose outer ring comes from its eight neighbors.
    • Nothing is fetched for a halo. A tile loads only itself and draws at once, its ring filled from the neighbors that have loaded.
    • A side whose neighbor hasn't loaded repeats the tile's own edge. That neighbor isn't in view, so neither is the side.
    • When a neighbor loads later, the tile rewrites only its ring, four strips one texel wide.
    • TileBorders keeps each loaded tile's border (8 KiB, not 1 MiB), so a tile still finds a neighbor's border after deck.gl has unloaded the neighbor.
  • Ground cell size is computed per fragment from Mercator latitude. One value per tile would make the shading jump 7% between z6 tile rows near 46°N.
  • Exact decoding. There's no premultiplied alpha or color conversion. Pixels are read back the way MapLibre reads DEM tiles: through a canvas, or through VideoFrame when canvas reads are altered. Where neither gives exact bytes, tiles fail with a message in the panel instead of drawing wrong elevations.
  • PMTilesTerrainLayer exists only to turn on feat(deck.gl-geotiff): coalesce COG tile reads by default #809's protected _abortUnwantedLoads hook, because the example raises maxRequests. It can go once a public prop exists.

Measured

The opening view at the Matterhorn: 12 tiles.

  • Each mode was measured cold, two or three times, from a toggle of the switch after the basemap had loaded, in headed Chrome 156.
  • Times come from polling the readout, so they're +/- 0.25 s.
Connection Mode HTTP requests Data All tiles drawn
fast per tile 14 1.9 MB 1.9 s
fast merged 7 2.0 MB 1.2-1.9 s
150 ms round trips, 20 Mbit/s per tile 14 1.9 MB 2.1-2.3 s
150 ms round trips, 20 Mbit/s merged 7 2.0 MB 1.8-1.9 s

The merged opening view is seven requests:

  • one 64 KiB header block;
  • one 64 KiB block holding the leaf directory;
  • five tile-data requests: 951 KB and 303 KB (one run of eight adjacent tiles, split by the reader's 1 MiB request cap), 305 KB (two tiles), 166 KB and 160 KB.

Mapterhorn serves over HTTP/2, so merging saves more requests than time. Decoding takes about 1.25 ms per tile on the main thread.

Halos without fetches. At first, each tile also fetched its eight neighbors for its halo. Kyle found the example slow, and filling halos only from neighbors that load anyway made it faster:

Measured Fetching neighbors Loaded neighbors only
Opening view, merged 4.8 MB 2.0 MB
Opening view, merged, 150 ms round trips all drawn at 2.9 s all drawn at 1.8-1.9 s
One-screen pan, fast 13 tiles fetched for 6 shown, 1.57 MB 6 tiles, 0.91 MB, all drawn at 0.6 s
One-screen pan, 150 ms round trips last tile at 1.27 s last tile at 0.93 s

Verified

  • Tests: the example's 82 tests, typecheck, Biome and vite build. Before the halo change, deck.gl-raster (251) and range-reader (99) also passed on feat(deck.gl-geotiff): coalesce COG tile reads by default #809's head.
  • Headed Chrome:
    • All four modes render.
    • Moving the sun or changing the mode fetches nothing.
    • There are no seams at tile edges or between tile rows at z5 (the Himalaya).
    • After the halo change, full-resolution crops across tile edges and a corner show no seams. Every opening view rewrote 29 rings: one per pair of adjacent tiles, counting diagonals, among its 4 x 3 tiles.
    • Tilted to 60° at z12.5 and 65° at z9.6, no seams were visible.
    • Across the antimeridian (Taveuni, Fiji, at z12), the rings read back from the GPU equal the neighbors' edge columns exactly. Hillshade still shows a faint line at 180°, and it comes from the data: there, the step between Mapterhorn's two edge columns is 20-60% of the steps beside it, while at other tile edges it matches them.
    • With canvas noise simulated, decoding switches to the VideoFrame path.
  • Headed Firefox 155 with privacy.resistFingerprinting: canvas reads come back random and VideoFrame is undefined. Tiles fail with the panel message; before the fix they drew random stripes.

Review

A fresh reviewer found four Important issues, and I raised two minors to Important. All are fixed, each verified by a failing test or browser check first.

  • A drawn tile's halo reads were never cancelled. Fixed then with a signal per halo, released when the tile unloaded. The halo change has since removed those reads.
  • A replaced archive could leave pmtiles unable to read it until reload. Fixed by using pmtiles' own etag check.
  • Each cached tile kept its 1 MiB halo array alive. luma.gl keeps createTexture's data prop, so the array is now uploaded with writeData. The COG layers do the same: render-pipeline.ts and multi-cog-layer.ts.
  • The stats poll rebuilt every tile's sublayers about four times a second while tiles loaded. The readout now polls on its own; tileset setOptions calls during a pan load went from 4 to 0.
  • Firefox with fingerprinting resistance drew random stripes as terrain. Tiles now fail, and the panel says why.

Also found in the browser: #647's shader takes atan(0, 0) on flat ground, and the resulting NaN drew lakes black. It's fixed here, but #647 still has it.

Open questions

  • Should there be a WebAssembly WebP decoder as a fallback, so Firefox with fingerprinting resistance (and probably Safari private browsing) can draw tiles? It would be a new dependency.
  • Should there be an issue for the COG layers keeping createTexture's data?
  • pmtiles caches failed directory fetches, so a network blip blanks a region until reload. That's upstream behavior and isn't filed.
Calls made without asking
  • Built on feat(deck.gl-geotiff): coalesce COG tile reads by default #809's newest head rather than the head named in the plan. Cost if wrong: one rebase --onto.
  • Tests use hand-derived literal values.
  • Borders are kept after their tiles unload, up to 2048 (16 MiB). Cost if wrong: memory, or seams where a forgotten neighbor's border is missing.
  • Added tests beyond the plan: the counting FetchSource, getUniforms (proven by a mutation), a released tile's refine, and getZxy returning nothing.
  • Two comment fixes in Kyle/elevation example #647's small modules: ASCII, and Terrarium units.
  • Gave TerrainDerivative explicit uniform and binding types.
  • loadTerrariumElevation lives in get-tile-data.ts.
  • The experimental badge sits inline in the panel title, because ControlPanel has no experimental prop on this base.
  • A refined halo repaints through map.triggerRepaint. Cost if wrong: a refined ring could show a frame late.
  • Retuned the colormap ranges so valleys don't look like water: [500, 4500], [2000, 8850], [0, 2800] and [400, 3000].
  • The layer allows null tiles, and renderTile draws nothing for them.
  • Fixed atan(0, 0) on flat ground.
  • Verified in my own headed Chrome over CDP, because the chrome-devtools profile was busy.
  • Each opened archive gets its own range reader, so both sides of the switch start cold.
  • Timed every pass from a toggle after the basemap loaded. An earlier run timed from page load was confounded by the basemap still loading, so it's excluded.
  • Captured a fresh 1800 x 1013 docs card.
  • Raised the stats-poll and canvas-fallback minors to Important.
  • Superseded by review fixes:
    • warning once and falling back to a distorted canvas;
    • adopting a new validator after one mismatch;
    • simulating the Firefox check in Chrome.
  • Left as is:
    • deck.gl's default tile cache size. Cost: hundreds of MiB of GPU memory on large high-DPR screens.
    • Blockiness past z12 wasn't checked. Cost: possibly blocky shading at zooms 13-14.
    • The colormap range follows the chosen place, not the current view. Cost: a poor range after panning far.
    • Borders can survive an archive swap. Cost: a stray seam right after a swap.
    • Safari is untested. Its private browsing would now get the error message.
    • StrictMode opens the archive twice in development.
    • A few unused deck.gl dependencies, copied from titiler-cog.
    • A ring is rewritten once per neighbor that loads after its tile, up to eight times. Cost: four small writeData calls each.
    • Where tiles of two zoom levels meet, the finer tile's neighbor at its own zoom may never load, so that side keeps a repeated edge. The tilted views above stayed at one tile zoom, so I didn't see it happen.
    • Decoding stays on the main thread. Cost: about 15 ms for the 12-tile opening view.
Deferred minors
  • The center texel isn't clamped at the tile edge, which could draw a speck there. None was seen.
  • performance.measure entries pile up over a session.

🤖 Written by Claude Code

kylebarron and others added 27 commits October 28, 2026 14:39
…layers

`DEFAULT_RANGE_READER`, a module-level `RangeReader` next to the default
limiter, so every COG layer on a page shares one byte cache and one
batching window. `COGSource.open`, `MultiCOGSource.open` and `MosaicLayer`
take `reader` and `requestInit` (headers, credentials, mode for data
behind authentication) and forward them to `GeoTIFF.fromUrl`; the mosaic
passes them to `getSource` like the limiter.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e task

`COGLayer` and `MultiCOGLayer` default to `maxRequests: 256` and
`debounceTime: 50`: deck.gl's six-at-a-time throttle would starve the
range reader, which merges a selection's reads into few streamed requests
and is capped by the per-origin limiter instead; the debounce lets a pan
settle before anything is requested. `RasterTileLayer` and `MosaicLayer`
keep deck.gl's defaults (#562).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`RasterTileset2D` gains `abortUnwantedLoads`: after every update, a
loading tile that is neither selected nor visible is aborted. deck.gl does
that only while more than `maxRequests` loads are in flight, which the COG
layers' high `maxRequests` disables; without it a merged, streamed request
would run on for tiles that left the view. `RasterTileLayer` exposes it as
a protected hook, off by default; `COGLayer` and `MultiCOGLayer` turn it
on. `ZarrLayer` keeps deck.gl's behaviour until its own reader adapter
lands: zarrita's coalescer fails every read of a merged request when one
aborts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Requests, caching and authentication for the COG layers: the shared
reader, tuning it or isolating a layer, `requestInit` for data behind
authentication, and the new throttling defaults.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…x docs

Findings of the whole-branch review:

- deck.gl evicts unselected loading tiles inside `update` (its cache
  resize) before the tileset's own abort pass can see them, so a tile that
  left the view and fell out of the cache in one update kept its stream
  alive and later made a texture nobody would free. `RasterTileset2D` now
  wraps `onTileUnload` in `setOptions` (which `TileLayer` calls on every
  props change) to abort a loading tile first. Test added for both the
  option on and off.
- The deck.gl-geotiff README showed `reader` and `requestInit` as
  `COGLayer` props, which do not exist: a TypeScript user got a compile
  error and a JavaScript user's auth header was silently dropped. The
  examples now open the source with `COGSource.open` and pass it to the
  layer; `MosaicLayer` takes both as props.
- `finalize`'s docstring had been displaced above `update`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Docstrings state what each option or hook is and the rule a caller
needs; the reasoning behind the defaults and the abort rule is in the PR.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Designs an experimental example that does for PMTiles what cog-terrain
does for COGs: Mapterhorn's terrain archive feeds both a TerrainLayer
mesh and a RasterTileLayer drape that decodes Terrarium elevation on
the GPU. Both read through the range reader, since a clustered archive
keeps a view's tiles together: 20 tiles at the Matterhorn sit in 2
contiguous byte runs. Assumes question 10 of the Sources spec stays as
it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rework the spec around #647's hillshade instead of 3D terrain: the same
Horn kernel, halo and sun controls, fed by Mapterhorn's PMTiles through
the range reader. The view and its halo ring (42 tiles) sit in 7
contiguous runs at the Matterhorn. Adds what Web Mercator needs (a
ground cell size per fragment), a cache of tile borders with counted
cancellation, MapLibre's exact pixel readback, and why the pmtiles
library beats loaders.gl here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kyle settled four of the five open questions: no loaders.gl, this
replaces #647 as the hillshade example (#647 is parked), the merge
switch stays, and the pmtiles adapter lives in the example until it
moves to library code. How a directly used RasterTileLayer aborts
unwanted loads is still open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kyle chose a small RasterTileLayer subclass that overrides #809's
protected _abortUnwantedLoads hook over a new public prop, for now. That
closes the spec's last open decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The range reader now calls its structural purpose "metadata", and #807's
chunkd adapter tracks validators. The PMTiles adapter follows both, and
PMTilesSource.open takes the byte source so the merge switch can swap it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PMTiles hillshade example needs a WebMercatorQuad tile matrix set
with 512 px tiles (morecantile ships none), each tile's Mercator y
range, and the ground cell size, which shrinks with latitude. The app
itself is a placeholder map until the shading lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hillshade needs each tile's exact bytes, so tiles decode without
premultiplied alpha or color conversion, and pixels are read back the
way MapLibre reads DEM tiles: through a canvas, or through VideoFrame
when the browser adds fingerprinting noise to canvas reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ported from #647's halo, but each neighbor contributes only its border
(its outermost rows and columns), which is all the ring needs and all a
cache has to keep. Columns wrap across the antimeridian; rows past the
poles replicate the tile's own edge. ringWrites gives the four strips
that refresh just the ring once late neighbors arrive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each tile and its eight neighbors share one load per tile through a
cache that keeps only a settled tile's border (8 KiB, not the 1 MiB of
decoded elevation). Cancellation is counted as pmtiles counts it for
directories: a load aborts once every waiter's signal has aborted, so
panning stops the halo reads for tiles that left the view. #647's cache
let those run to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pmtiles Source backed by the range reader merges the reads of one
task, such as a view's tiles once their leaf directory arrives, into a
few streamed requests. Tile data reads use the "data" purpose and
headers and directories "metadata". The bytes are copied out so a worker
transfer can't detach the reader's cache. A changed validator raises
pmtiles' EtagMismatch, on which pmtiles refetches its directories.
PMTilesSource opens an archive as a RasterSource on a 512 px
WebMercatorQuad grid. A counting FetchSource is the per-tile baseline
the merge switch compares against.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Horn kernel, hillshade, slope, elevation and tinted-relief modules
come from #647. TerrainDerivative changes in two ways: the ground cell
size is computed per fragment from its Mercator y (one value per tile
jumps 7% between z6 rows near 46°N), and the nodata substitution is
gone, since Terrarium has no nodata value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
getTileData asks for a tile and its eight neighbors in one synchronous
pass, so their reads merge. It draws the tile as soon as it decodes,
then rewrites only the halo ring when late neighbors land, as #647 did
for the whole texture. The layer subclass turns on #809's protected
abortUnwantedLoads hook, because the example raises maxRequests and
deck.gl then stops aborting loads for tiles that left the view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hillshade app: Mapterhorn's planet.pmtiles opened through the range
reader, four places from the Matterhorn to Yosemite, and #647's modes
and sun controls. In headed Chrome the opening view loads in 7 HTTP
requests, moving the sun refetches nothing, and there are no seams at
tile edges, between tile rows at z5, or across the antimeridian.

Also fixes a bug carried over from #647: on flat ground such as a lake,
the shader took atan(0, 0), which GLSL leaves undefined, and the NaN
drew lakes black in tinted relief. The aspect is now 0 there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A readout shows tiles drawn and decoded, HTTP requests and bytes, and a
"Merge requests" switch swaps the range reader for pmtiles' own
FetchSource and deck.gl's six-at-a-time default. Each opened archive
gets its own reader, so both sides of the switch start cold. Measured
at the Matterhorn: 7 requests merged against 33-35 per tile for the same
~5 MB, and the view completes about 7% sooner at 150 ms round trips.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README explains what the example shows and how: merged reads
through the range reader, the halo and its border cache, the
latitude-aware cell size, exact decoding. It records the measured
request counts and says what's provisional. The docs list it with the
other experimental examples, with a card of the Matterhorn in tinted
relief.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found that deck.gl stops aborting a tile once it has drawn, so a
tile's neighbor reads ran on after it left the view, was evicted, or
the merge switch replaced the layer. They also ran on when the archive
had no tile to draw. The halo reads now hold their own signal, which is
aborted on those paths and by releaseHalo, which onTileUnload calls.

luma.gl keeps a texture's props, so createTexture({ data }) held each
tile's 1 MiB halo array for as long as the tile was cached. The array
is now written with writeData instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found that the adapter's own validator tracking could leave pmtiles
permanently unable to read a replaced archive. It threw on the read that
first saw the new version, then accepted stale reads in flight, and could
throw from the header re-read, which pmtiles caches as a failure. Use
pmtiles' own check instead: it passes the header's etag with every later
read, so throw EtagMismatch only when a response's validator differs
from it. The header read carries no etag and always adopts the new
version. A test drives a real PMTiles across the swap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found that the 250 ms stats poll re-rendered the whole app, and
with it the layer. Each tick handed deck.gl's TileLayer a new renderTile,
so it re-traversed and rebuilt every cached tile's sublayers, about four
times a second while tiles loaded, the window the readout measures. The
readout now polls on its own, and renderTile and the tile callbacks are
memoized. Measured in headed Chrome: tileset setOptions calls during a
pan load went from 4 to 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review asked for a real check in Firefox with privacy.resistFingerprinting
rather than the earlier simulation. Firefox 155 randomizes every canvas
read and switches WebCodecs off, so decoding fell back to the canvas and
drew random stripes as terrain. Unlike MapLibre, the decode now never
falls back to an altered canvas: tiles fail with ExactPixelsUnavailableError,
and the app shows one message in the panel. The README says so, and notes
that a drawn tile's halo reads are cancelled only when it's unloaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured again on the final code, after the readout stopped rebuilding
the layer during loads: merged reads finish the opening view in
1.3-1.4 s against 1.6-1.9 s per tile on a fast connection, still with
7 requests against 33-35.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the feat label Oct 28, 2026
Base automatically changed from kyle/range-reader-layers to main October 28, 2026 19:09
Each tile used to fetch its eight neighbors too, so its halo was complete.
That more than doubled the opening view's bytes (4.8 MB instead of 2.0 MB)
and made the tiles in view wait behind the off-screen ring in the merged
requests.

Now a tile loads only itself. Its ring is filled from the borders of
neighbors that have loaded, and a side with no loaded neighbor repeats the
tile's own edge; that neighbor isn't in view, so neither is the side. When
a neighbor loads later, the tile rewrites only its ring. TileBorders keeps
loaded tiles' borders and tells loaded neighbors when one arrives; it
replaces BorderCache, whose shared loads and cancellation are no longer
needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kylebarron and others added 2 commits October 28, 2026 15:37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#809 squash-merged as ed7a7f1, with the same tree as its head 98bf4eb, which this branch was built on. Merging main lets the PR's diff show only the example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't include the information modal in the screenshot

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This reply was written by Claude (Claude Code), not by @kylebarron.

Recaptured without the panel in 6738ba1: the same Matterhorn view and size (1800 x 1013), with the panel hidden before the screenshot. The map's attribution stays in the corner, since Mapterhorn's terms ask for it.

🤖 Written by Claude Code

@kylebarron kylebarron left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall lgtm

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kylebarron
kylebarron merged commit 2003ff6 into main Oct 28, 2026
3 checks passed
@kylebarron
kylebarron deleted the kyle/pmtiles-terrain-example branch October 28, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant