zarr and object store like formats as an alternative to netcdf

[paolap]

As this is a hot topic in the community it would be nice to have a page somewhere where pros and cons and implications are discussed. It's important for people who managed data to have a t east a starting point to look into this when taking decision around formats

see https://docs.unidata.ucar.edu/netcdf-c/current/md__media_psf_Home_Desktop_netcdf_releases_v4_9_2_release_netcdf_c_docs_nczarr.html

[hot007]

Cross ref this page https://acdguide.github.io/BigData/data/data-zarr.html but also I think that page should be updated seeing it was written for 4.8.0!

Quick note for this book:
Pros - improves read efficiency making analysis more tractable, highly performant with python/dask
Cons - doesn't inherently support metadata standards (unless used as a back-end for netCDF), uses A LOT of inodes which can cause quota problems unless you work with zipstore, basically only works with python which means it doesn't meet the I part of FAIR.

[hot007]

Note we need to update/fill out this page! Data formats