Python experts from across the world came together at ECMWF on 30 and 31 October 2018 to review progress in developing Python frameworks for Earth system sciences.
This was the second workshop of its kind. ECMWF organises these workshops to engage with the wider Python community at a time when the language is increasingly used as an interface to interact with ECMWF and Copernicus data and services related to numerical weather prediction and climate science.
Nineteen speakers gave updates on recent developments. At ECMWF, these chiefly relate to the Copernicus Climate Data Store (CDS) toolbox and the meteorological visualisation and computation package Metview.
The CDS is the gateway to data and information for users of the Copernicus Climate Change Service (C3S) implemented by ECMWF on behalf of the EU.
The main news from ECMWF was that the CDS toolbox has been open to the public since the summer; that the Python interface to Metview is available for beta testing; and that the cfgrib package can now be used to read GRIB files of meteorological data into xarray data arrays.
A large part of the workshop was dedicated to working group discussions on challenges in the provision of Python frameworks.
The optimal setup to make use of cloud and high-performance computing systems for large-scale calculations was discussed, with a particular focus on the Pangeo initiative and the use of distributed processing through the Dask Python package.
Much of the discussion focused on how the interoperability between packages can be improved. There was a lot of useful feedback to guide future Python developments at ECMWF.
Building on the outcomes of the meeting, ECMWF will continue its work to provide Python packages while focusing on the migration to Python 3 and packaging through community channels, such as PyPi and conda-forge.
Participants agreed that the community should develop showcases and ‘cookbooks’ to show how the various frameworks can work together.
ECMWF will try to participate in the ongoing explorative work of the Pangeo initiative to demonstrate the processing of large-scale data in our community.
Further information
The presentations and recordings of the talks are available on the workshop web page.
On 28 November from 14:00 GMT, ECMWF will live-stream a webinar on ‘Working with ECMWF data in Python’. The webinar is open to the public and will include a demonstration of how to use ECMWF Python packages to process ECMWF data, including from the MARS meteorological archive and the CDS.
The workshop was attended by nearly 50 participants.
