Seamless era5 connection through Zarr

[Daafip]

Saw on LinkedIn that era5 is now accessible as Zarr, making forcing much simpler possibly? https://github.com/ecmwf-training/dss-notebooks/blob/main/datasets%2Freanalysis-era5-single-levels%2Farco-access.ipynb.

[BSchilperoort]

The main difficulty here would be to have ESMValTool use those zarr stores. @bouweandela any idea if this has been discussed or not?

Otherwise it would be good as it requires less local data, the chunk sizes seem to be appropriate for our hydrological use case (time: 67584, lat: 4, lon: 4. This is very useful for non-ML applications!).

A possible downside is that it requires connection to an external server which can be down, and it requires CDS credentials to be set up. This is fine for a PhD student, but not as convenient for education use cases.

[bouweandela]

Yes, I've been playing with it, I'll try to make a draft pull request today so we can explore feasibility.

ARCO ERA5 data is also available from gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3 without an account and probably quite reliable, described at https://github.com/google-research/arco-era5.

[bouweandela]

Opened ESMValGroup/ESMValCore#3179 to demonstrate that this is possible and easy to implement. It's holiday time now, so it may be a while before I have time to finish it, but you can already experiment with it to see if it fits your use case. Please feel free to leave comments/requests on the PR.

[BSchilperoort]

For the eWaterCycle use case the CDS geoChunked zarr is essential; otherwise there is way too much bandwidth and RAM required for producing catchment/region scale model input data. And you might as well store the full maps anyway if you're already downloading it all.

I am able to generate Makkink potential evaporation based on the CDS geoChunked Zarr (simple input data, less accurate). For the De Bruin method we require the shortwave radiation at the top of atmosphere, which is not in the CDS variables 😞 *

Additionally, we require orography for downscaling. It's not in the CDS data, but it is available in the google zarr, which works, and can be combined with the CDS data.

GCS for orog

projects:
  native6:
    data:
      gcs-era5-zarr:
        type: esmvalcore.io.xarray.XarrayDataSource
        store: "gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3/"
        options:
          chunks: "auto"
          engine: "zarr"
          storage_options:
            token: anon
        variables:
          geopotential_at_surface: orog
        facets:
          dataset: "ERA5"
          mip: "fx"

tas, pr, rsds for Makkink from CDS

# Read ERA5 data in zarr format on Climate Data Store.
# Go to
# https://github.com/ecmwf-training/dss-notebooks/blob/main/datasets%2Freanalysis-era5-single-levels%2Farco-access.ipynb
# for further information.
projects:
  native6:
    data:
      cds-era5-zarr:
        type: esmvalcore.io.xarray.XarrayDataSource
        store: "https://arco.datastores.ecmwf.int/cadl-arco-geo-002/arco/reanalysis_era5_single_levels/sfc/geoChunked.zarr"
        options:
          chunks: "auto"
          consolidated: true
          engine: "zarr"
          storage_options:
            headers:
              Authorization: "Bearer <enter auth here>"
        variables:
          t2m: tas
          tp: pr
          ssrd: rsds
        facets:
          dataset: "ERA5"
          frequency: "1hr"

* I do feel like it's silly for distributed models, which model land surface processes, to require potential evaporation. They should just solve this themselves based on incoming radiation, air temperature and the vapor pressure.