On 18 October 2022, ECMWF successfully migrated all operational forecast services from its data centre in Reading (UK) to its new data centre in Bologna (Italy). On 27 June 2023, it successfully upgraded its operations to Integrated Forecasting System (IFS) Cycle 48r1. This article provides a review of one year of operational service in Bologna.
Resilience and performance
Full handover of the data centre from the Regione Emilia-Romagna to ECMWF took place on 29 April 2022. Since March 2022, when the Copernicus Climate Data Store became operational in Bologna as the first operational service, the data centre has been running smoothly, providing the environment to host ECMWF operational activities.
The mechanical infrastructure has been very reliable and the electrical continuity system successfully covered any voltage losses during the first year of operation. The GARR Consortium, the Italian national research and academic network, provides a resilient 100 Gigabits per second (Gbps) service for dissemination. Alongside GARR, resilient 1 Gbps and ADSL network services are provided by Lepida. These are intended for analyst remote access to the data centre independent of the GARR service as well as for emergency access during incidents.
Each of the data centre’s two data halls has an ‘IP Fabric’ network topology, and the two data stores have a third production network. There is also a Management and Monitoring network supporting all network and server devices, and a limited-scope out-of-band emergency network as a last resort. Application-specific security zones make it possible to carefully manage the access between computer systems. This modern design has already proved its worth during the first year of operations. ECMWF operates a modern network, server, and security infrastructure in Bologna. Some stability issues were experienced in the first year of operation, which will be learned from and contribute to continual improvement.
A new HPCF and move of the data handling system
The greater power and cooling capacity provided by the Bologna data centre compared to what was available in Reading made it possible to install a new high-performance computing facility (HPCF) from Atos. Without the larger HPCF, it would not have been possible to deliver IFS Cycle 48r1. Implementing the Atos HPCF has not been without challenges. However, it is now facilitating reliable forecast production and supporting the research and project activities at ECMWF as well as the workload of ECMWF’s Member and Co-operating States.
The switch of forecast production and all related services, such as observation acquisition and pre‑processing, product generation and dissemination services, to the HPCF in Bologna was carried out on 18 October 2022. ECMWF was running IFS Cycle 47r3 at the time. On 27 June 2023, a major scientific upgrade to Cycle 48r1 was carried out. The changes included a horizontal resolution increase of the medium-range ensemble (ENS) from 18 to 9 km as well as a major upgrade of the configuration of the extended-range ensemble.
In the autumn of 2022, the Data Handling System was moved from Reading to Bologna: three tape libraries, 17 disk systems, 150 servers, 290 tape drives, and almost 30,000 tapes were transported by lorry. Despite the complexity of the task, it was possible to provide a continuous archive and retrieval service for all critical production components throughout the whole migration process, and the services have continued to perform reliably over the year.
A successful migration
The last year in Bologna has shown that we successfully migrated operations to the new data centre and the new HPCF as well as implementing a major cycle upgrade – altogether a huge undertaking for ECMWF. The migration was used to move from infrastructure that had evolved organically to a design that aims to accommodate the current requirements and support enhancements in the near future without compromising any of our modern design principles.
Such principles are, for example, improved operational resiliency, better IT security, higher levels of efficiency, and reduced environmental impact, to name just a few. In addition, and to react to the new situation of having a remote data centre, new software tools and working processes have been introduced. The improvements, delivered through careful design work, improved tool-chains, and efficient work processes, are clearly visible in the daily results of delivering high-quality forecasts and services to our Member and Co‑operating States and other external and internal stakeholders of ECMWF.