Skip to content

performance: add benchmarks #191

Open

Description

@maxrjones

Motivation

We want every performance PR to show how much it helped. That's not currently possible because:

Current limitations

  • The draft benchmarks in chore: add pytest benchmarks #165 cover only XarrayReader, on a small Zarr store with 4 time steps. They don't cover the mosaic backend or the /info endpoint, where fix: open each dataset once per request #171 and the info() slowness live.
  • They have no Icechunk tests.
  • They have no long time axis. TEMPO has 17,000+ time steps, and the info() problem only shows up with that many.

Propose solution

  1. Count tests, with no new dependency. Ordinary pytest tests that count the expensive things a request does and assert a limit. For example:
    • "a tile request opens each dataset once"
    • "a warm tile request makes one store request"
    • "a tile request creates no boto3.Session"
  2. Timing benchmarks (revive chore: add pytest benchmarks #165) for CPU-bound changes:
    • rebase it and drop dask (it was only there to compare against feat: filtering based on multiple variables #151);
    • add a long time-axis fixture (17,000+ steps; coordinates only, so it stays small);
    • add a local Icechunk fixture;
    • add app-level cases through TestClient (/tiles, /point, /info, /info?show_times=true).

Each PR compares before and after with pytest-benchmark compare.

CodSpeed could later run the same benchmarks and post a comparison on each PR, because pytest-codspeed accepts the same benchmark fixture. Its default mode measures CPU instructions, so request-bound improvements still need the count tests.

Acceptance criteria

  • Counting helpers exist for dataset opens, store requests and boto3.Session constructions.
  • There is at least one count test each for /tiles, /point and /info.
  • chore: add pytest benchmarks #165's benchmarks run on main and include the long-time-axis, Icechunk and app-level cases.
  • The benchmarks README explains how a PR reports its before/after.

Activity

  1. changed the title [-]Add performance benchmarks[/-] [+]performance: add benchmarks[/+] on Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions