Repository navigation
feat: add Worldview Rainbow 4 and NAQFC colormaps - #200
Conversation
Register custom colormaps from `src/titiler/multidim/cmap_data/*.npy` and expose them through both the tile endpoints (`colormap_dependency`) and `/colorMaps` (`supported_colormaps`), the same wiring veda-backend uses. Three colormaps ship, generated by `scripts/make_cmaps.py` (numpy only): - `worldview_rainbow_4`: NASA Worldview's "Rainbow 4" palette, verbatim from nasa-gibs/worldview (255 colors, top color repeated), for TEMPO NO2 with e.g. `rescale=0,2e16`. - `naqfc_o3`: NOAA Air Quality Forecast Guidance Viewer ozone ramp over 0-200 ppb (`rescale=0,200`, or `0,0.2` for ppm). - `naqfc_pm25`: the ozone ramp stretched so each EPA 24-hour PM2.5 AQI category spans the same colors as its 8-hour ozone category, over 0-225.4 ug/m3 (`rescale=0,225.4`). `tests/test_cmaps.py` regenerates every file and fails if a committed `.npy` differs from the script, checks the Worldview palette is verbatim, and requests a tile and the `/colorMaps` entries through the app. Near-term counterpart to NASA-IMPACT/veda-backend#665 for data served by titiler-multidim. Longer term these move to a shared colormap package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Thanks for putting this together, Max. The colormaps themselves look great, and I especially like that the Worldview palette is a verbatim copy of the original and that the known limits are included. My one real concern is the generated I'm sorry to send you another gist. I started with just that change, and as I worked through it I kept finding small things to tidy, so it grew. The three colormaps come out exactly the same, and the notes at the top of the gist explain each change. Please treat everything beyond the Since @hrodmn has already approved, feel free to merge this as is if that's easier, and I can open a follow-up PR with these changes, if you like. https://gist.github.com/chuckwondo/d97813dd8889d648bb2cb8ce31774d39 |
chuckwondo
left a comment
chuckwondo
left a comment
There was a problem hiding this comment.
Thanks @maxrjones. My review is in this comment.
|
I should've mentioned in the beginning that I think this is a temporary solution, and adds a bit of tech debt. See https://github.com/NASA-IMPACT/veda-architecture/issues/899 for a proposal for a more sustainable approach. @chuckwondo thanks for your comments. I similarly dislike committed .npy files. I'll look into your suggestions, and maybe we can port them to veda-backend as well in the future, which also uses the .npy files. |
Thanks Max. A shared library sounds like the right home for these, and I added a few thoughts on NASA-IMPACT/veda-architecture#899. |
Per review on #200: the three colormaps are now built when the package is imported (`titiler/multidim/colormaps.py`), so there are no generated files to drift from their source. This removes `scripts/make_cmaps.py`, the `cmap_data/` .npy files, and the drift test. The generated tables are byte-identical to the previous .npy files. Also from the review: - the dict passed to `cmap.register` is the only list of colormaps (previously a script dict plus a directory glob in main.py) - ramp stops and AQI breakpoints are dicts, so pairs can't misalign - rio-tiler's `parse_color` replaces the hand-written hex parsers - `rainbow_4.json` lives in the package and is read via importlib.resources - the Worldview source link is pinned to commit 7d16ee5 - the colormap table and known limits move into the main README - tests check every rainbow color, catch out-of-order interpolation points, and are parametrized per colormap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every colormap is now a `cmap_data/<name>.json` file with a `source` and either `stops` (data value -> "#rrggbb") or `ramp` + `breakpoints` (data value -> value on another file's stops). `colormaps.py` registers each file under its name and interpolates it into a 256-row table, so the ozone ramp and PM2.5 breakpoints are data rather than Python constants. Worldview's Rainbow 4 is converted to one stop per row (alpha was ff for all 255 colors; the top color is repeated for row 255), so the upstream file is no longer kept verbatim; its `source` pins the upstream commit. All three generated tables are identical to the previous ones. Tests now check every file against the schema and every stop/breakpoint list for increasing order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add an optional `notes` field to the cmap_data schema and move the NAQFC screenshot-sampling limit into naqfc_o3.json (naqfc_pm25.json points to it), next to the stops it describes. The README keeps what API users need, since /colorMaps serves only the table: each colormap's `rescale`, a one-line description linking to its file, and the limits that apply to every custom colormap. Provenance details live only in each file's `source`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each colormap file now has a `reference` (an https URL) alongside its `source` text, which no longer embeds links: the pinned Worldview file, veda-backend#665 for the NAQFC ozone stops, and the AirNow AQI Technical Assistance Document for the PM2.5 breakpoints. The schema test checks it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PM2.5 (88101, 24 HOUR) and 8-hour ozone (44201) breakpoints in naqfc_pm25.json and the naqfc_o3 notes match this table exactly (9.0/35.4/55.4/125.4/225.4 ug/m3; 54/70/85/105/200 ppb). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks again for your comments, @chuckwondo. I've re-requested a review because I applied your suggestions, but then made some minor changes because I'd rather have the colormaps in a schematized format rather than a .py module. So, the current implementation isn't exactly what you recommended. |
chuckwondo
left a comment
chuckwondo
left a comment
There was a problem hiding this comment.
Thanks @maxrjones. Looks great! I like the move of the tables from python code into separate resource-loadable json files. Nice adjustment.
I'm approving, but there are 2 more suggestions I've made to avoid a couple of traps.
| [options] | ||
| exclude-newer = "0001-01-01T00:00:00Z" # This has no effect and is included for backwards compatibility when using relative exclude-newer values. | ||
| exclude-newer-span = "P1W" | ||
|
|
There was a problem hiding this comment.
It looks like you have somehow set exclude-newer, perhaps in your own global uv config file, or the UV_EXCLUDE_NEWER env var, which causes the addition of these lines.
Unfortunately, this is not portable because someone who does not also have the same setting will cause this entry to be removed (via uv sync), resulting in unnecessary churn in the lock file.
However, this can be robustly remedied by adding the following block in pyproject.toml, which will ensure the lines remain in place even when someone else does not have the same config setting you have:
[tool.uv]
exclude-newer = "1 week"There was a problem hiding this comment.
NOTE: I edited the comment above. Previously I suggested removing the extra uv.lock lines, but that was a mistake. That's what would happen without having the additional block I suggest adding to pyproject.toml and also without having the same global config you have.
There was a problem hiding this comment.
Moving the tables to JSON files is a nice improvement!
The only trap we need to avoid is the potential for accidental duplicate keys, which can go silently unnoticed. For example, if we were to accidentally write the "110" key as "100", it would be a duplicate of the preceding, correct entry for "100", but json.loads doesn't complain about duplicate keys. It simply allows the last key to "win".
However, if we add the following to .pre-commit-config.yaml, it will fail pre-commit if any JSON file contains duplicate keys anywhere (I suggest listing it as the first repo in the repos list. Although order doesn't matter, it's simply conventional for pre-commit's own hooks.):
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: check-json
Co-authored-by: Chuck Daniels <cjdaniels4@gmail.com>
Set exclude-newer to "1 week" under [tool.uv]. The [options] block in uv.lock then comes from the project instead of a contributor's global uv config. A plain `uv sync` no longer strips it. Add the check-json hook from pre-commit-hooks. json.loads keeps the last of two duplicate keys without warning. A mistyped breakpoint key in cmap_data would otherwise replace an earlier entry silently. Co-authored-by: Chuck Daniels <cjdaniels4@gmail.com>
Summary
This PR adds 3 custom colormaps requested for TEMPO and NAQFC rendering for the VEDA portal.
Testing
Tested using by appending this to the dev deployment URL:
WebMercatorQuad/map.html?url=s3%3A%2F%2Fairquality-data-store-develop%2Ftempo%2Fno2%2Fv04%2F&variable=vertical_column_troposphere&sel=time%3Dnearest%3A%3A2026-10-06T17%3A37%3A16Z&algorithm=mean&where=eff_cloud_fraction<%3D0.5&where=solar_zenith_angle<%3D80&where=main_data_quality_flag<%3D1&tilesize=512&datetime=2020-01-01T00%3A00%3A00Z%2F2026-10-06T17%3A37%3A16Z&colormap_name=worldview_rainbow_4&rescale=0%2C20000000000000000&resampling=nearestPR checks
run-cdk-checkslabel to this PR.deploy-devlabel. It smoke-tests tiles from the native MUR, virtual MUR, and virtual NLDAS Icechunk stores after deployment.