add IMERG Late and Early

[Thomas Stanley (thomasastanley)]

Assuming what I see at https://planetarycomputer.microsoft.com/dataset/gpm-imerg-hhr is final IMERG, it would be nice to have a record of Late and early IMERG as well.

[Tom Augspurger (TomAugspurger)]

Sean Harkins (@sharkinsspatial) can you confirm that we pulled the "final" IMERG data?

Thomas Stanley (@thomasastanley) can you share a bit about why early and late would be useful in addition to final?

[Tom Augspurger (TomAugspurger)]

Thanks. At this moment, https://planetarycomputer.microsoft.com/dataset/gpm-imerg-hhr is a historical snapshot. See pangeo-forge/pangeo-forge-recipes#37 for the issue in pangeo-forge for supporting dynamic datsaets.

[Tom Augspurger (TomAugspurger)]

Thomas Stanley (@thomasastanley) (and oliviercics) is there any value to the Early / Late runs after the Final is available? One option that balances the timeliness of early / late against the cost of hosting them is to only store a rolling window of the last X day's worth of Early and Late (which could at a minimum cover up to the latest Final plus a buffer). But Early / Late files older than X days would be deleted. Would that work for your use case?

[oliviercics]

Yes. In our application, we will replace the Late Runs by the Final Runs when they become available (after 3.5 or 4 month to the present date). For our use case two things are important: 1) the availability of the Late Run as soon as it is available from NASA (we are doing near-real time monitoring) and 2) replace the Late Run files as soon as the Final Run files are available (this can be done once a month).

One additional comment. As we are developing our use case, we will compare a full run performed using Late Runs files and a full run using Final Run files. But that will be a one time thing.

[Thomas Stanley (thomasastanley)]

Tom Augspurger (@TomAugspurger), yes, the existence of an archive of NRT products enables us to fit and test hindcasts before making a real-time application. While it would still be of value to have the last 60 days of imerg available for real-time applications, my hunch is that (like GEE) this platform is better for retrospective work than putting out real-time alerts.

Of course, it's up to you what to host. I am not planning a specific project here at the moment, so you may not want to host unused data. (but part of the reason i don't work here is that i would have to upload the relevant datasets.) Just trying to give my perspective on what would make MPC more useful.

Thanks,
Thomas

[Tom Augspurger (TomAugspurger)]

Thanks for the context on both of those. Training on old realtime forecast data makes sense.

[Sean Harkins (sharkinsspatial)]

Tom Augspurger (@TomAugspurger) I can confirm that we pulled the final IMERG product https://github.com/pangeo-forge/gpm-imerge-hhr-feedstock/blob/main/feedstock/recipe.py#L13-L14.