HiDALGO

ECMWF and its partners in the HiDALGO project aim to implement a platform for on-demand simulation of complex global challenges which require massive high-performance computing resources alongside state-of-the-art data analytics and visualization. The key use-cases are in the areas of human migration, urban air pollution and social networking. HiDALGO has also applied its technology to provide an agent-based model for Flu and Coronavirus Simulation (FACS) for the 2020 pandemic.

The HiDALGO use-case workflows are comprised of four main components: pre-processing, numerical simulation, post-processing and visualization. The core simulations are ideally suited to running in a dedicated HPC environment, due to their large computational demands, coupled with the heavy communication overhead between parallel processes. However, the pre-/post-processing and visualisation tasks generally do not demand more than a few cores to compute and do not require message passing between instances, hence they are good candidates to run in a cloud environment. Enabling and efficiently managing and orchestrating the integration of both HPC and cloud environments to improve overall performance and functionality is the key goal of HiDALGO

ECMWF's role in the project is to enable seamless integration of the pilot applications with our meteorological data and services (such as data exploration, analysis and visualisation) delivered via the European Weather Cloud and orchestrated by bespoke HiDALGO workflows. During the project, ECMWF will develop the technology to serve real-time and archive data to the cloud, demonstrating the advantages of collocated cloud and HPC resources for data-centric workflows.

​