A joint project by ECMWF and EUMETSAT to create a cloud computing infrastructure for the meteorological community is entering a new phase. The focus will be on working with users currently trialling applications on the pilot infrastructures established by the two organisations to evaluate the platform and to shape the future operational services.
The ‘European Weather Cloud’ project aims to make the full extent of ECMWF’s and EUMETSAT’s vast data holdings available to users in their Member States and other members of the European Meteorological Infrastructure.
It will lay the foundations for a strong, domain-specific European IT infrastructure enabling direct access to and effective processing of data. This means that observation data, forecast data and meteorological products can be accessed together as if they were collocated. Federation with relevant cloud infrastructures from other partners will make it possible to further widen the range of accessible data.
A key advantage of cloud technology is that it can be used to execute processing workloads where the data is, without the need to transfer large volumes of data. Compared to other existing solutions such as high-performance computing facilities, the cloud allows for a greater degree of flexibility and customisation by the end user of the IT infrastructure underpinning the data processing.
The pilot is demonstrating how users can bring their processing to the data and apply new forms of data analytics, like machine learning, on the large datasets they require.
Example scenarios of how "service users" may interact with the cloud. The "system users" of the national meteorological services of a Member State offer hosted processing/services to internal service users (left, red), or hosted processing services for locally hosted services (middle, green) or hosted in another cloud (middle, purple) to internal or external service users. Finally, hosted services can be made accessible directly to external users and branded as a "Member State service".
As the initial two-year pilot period draws to a close, substantial progress has been made in addressing the project’s governance and infrastructure aspects and in trialling use cases. The European Weather Cloud is expected to become operational in 2022, with the pilot phase having been extended by one year so that the first operational infrastructure can be deployed in ECMWF’s new data centre in Bologna, Italy.
The usefulness of the European Weather Cloud has already been demonstrated by the role it played in backing up Croatia’s forecast production following an earthquake earlier this year. Other use cases are also showing great promise by demonstrating the value of combining datasets and processing data in the cloud.
Working with users in the Member States is crucial to ensure the platform is tailored for the meteorological community, allowing users to build weather and climate services for their users and providing fast access to a wide range of datasets such as forecast products, archive data and observations.
Laying the foundations
The infrastructures supporting the European Weather Cloud at ECMWF and at EUMETSAT are in place, tested and interconnected.
Many other technical and user support features and tools have been added to support and facilitate Member State users, including a joint landing page and a knowledge base and service portal.
Currently, all ECMWF and EUMETSAT Member States can submit use cases and gain access to the cloud infrastructures, create virtual resources and start their computational and data analytics workloads.
Pilot use cases
For the pilot project, 30 use cases have been defined and are at various stages of development. Covering a wide range of applications, they play an important role in helping to improve the platform and increase the available documentation for all users.
Regular user workshops provide opportunities to engage with users in Member States. 200 participants joined the cloud user workshop in May 2020.
Use application cases range from porting existing web applications (from other clouds or users’ IT centres) and building new data services on forecast data, to hosting web-based notebooks for NWP training and machine learning data analytics.
- The German weather service DWD is already feeding maps generated by a server it deployed on the cloud into its public GeoPortal service.
- A joint EUMETSAT and ECMWF use case will assess bias correction schemes for the assimilation of radiance data based on several satellite data time series.
- The Royal Netherlands Meteorological Institute (KNMI) will host a climate explorer web application based on KNMI climate explorer data and ECMWF weather and climate reanalyses.
- The Royal Meteorological Institute of Belgium will prepare ECMWF forecast data for use in a local atmospheric dispersion model.
- The EUMETSAT Numerical Weather Prediction Satellite Application Facility (NWP SAF) will develop a training module for a fast radiative transfer model (RTTOV) based on ERA5 reanalysis data.
Six of this year’s ECMWF Summer of Weather Code (ESoWC) projects also used cloud credits on the European Weather Cloud.
Next steps
The focus for the year ahead will be on further developing and evaluating the use cases. User workshops have already provided valuable opportunities for collaboration with the Member States and feedback for configuring the future operational cloud services and more user workshops are planned for 2021. ECMWF is also hosting a Weather and climate in the cloud workshop in February 2021 to explore ideas for the use of cloud technologies for meteorological applications.
Work will also continue to develop the technical infrastructure as well as specific governance and security aspects.
The European Weather Cloud is set in a wider context of related activities involving ECMWF. These include the Copernicus Climate and Atmosphere Data Stores (CDS/ADS) and the Centre’s contribution to the WEkEO Data and Information Access Services, as well as the data lake infrastructure of the new Destination Earth (DestinE) initiative of the European Commission, which will interface with the European Open Science Cloud. This context allows for synergies and consolidation of experience and expertise in a fast-evolving technology area, so that the European Weather Cloud will be able to deliver the best value for the investment of the European weather community.