Every other summer Member and Co-operating States report on the application and verification of ECMWF’s forecast products for the previous two years. ECMWF also gathers feedback in other fora throughout the year. This report summarises feedback collected between summer 2019 and summer 2021.
Usage of ECMWF forecast products remains widespread across National Met Services (NMSs), from short range through to seasonal timescales. Very favourable comments regarding outputs and accuracy are commonplace, notably for the short range and the medium range. There are again plenty of examples of forecasting success in severe weather situations, e.g. for wintry precipitation, but also some examples of events being missed. Flash flooding remains very challenging for models in general, with moisture budget terms in certain situations still hard to disentangle.
In the short range and early medium range ECMWF output is ordinarily used alongside the output of high resolution, limited area, convection-resolving models (LAMs) and ensembles. Activities here that have expanded since 2019 are the use of LAM ensembles, downscaling, and automated (and manual) blending techniques. Blending is designed to extract maximum benefit, and seamlessness, from modelling systems with different strengths. Post-processing (prior to any blending) is also commonplace, and there are again examples of simple approaches like bias correction delivering significant gains. Machine-learning (ML) is still just on the horizon for most NMSs, although Switzerland do already deliver some ML-based operational products. Switzerland are also running what appears to be the highest resolution LAM ensemble (1.1km).
National Met Services have again performed comparative verification for LAMs, HRES (ECMWF’s 9km model) and ENS (ECMWF’s 18km ensemble), notably for surface weather. The difference in spatial scale between LAMs and global models can affect verification results, and countries use a variety of approaches to account for this. Overall, using a neighbourhood approach and a combined index for precipitation and wind gust, France report AROME-FRANCE consistently outperforms the global HRES and ARPEGE models. Israel report their LAM ensemble to be clearly superior to ENS for all variables. More generally, relative to the IFS, LAMs have a clear advantage for 10m wind prediction, especially for mountains, and some advantages for 2m temperature. Problem areas for 2m temperature, for many models, are clear calm nights, hot sunny days, and springtime. The scale issues noted above are especially relevant for fields exhibiting high spatial variability, such as precipitation (and some other moisture-related variables). Nevertheless, even if not accounting for related ‘double penalty’ issues, the reports generally show that the LAMs also perform well for precipitation, while for low-level moisture variables results are less consistent, with IFS often performing better. Interestingly Belgium identified a multi-day drift in several surface weather variables in IFS forecasts for Belgian sites; we had some awareness of these but are now investigating further.
New and innovative NMS-designed diagnostics include fire ignition risk from Portugal, based on dry thunderstorm probability, hail predictors developed in Hungary, and stacked probability bar charts for visibility from The Netherlands. Services were very positive about ECMWF’s “progressive” forecast products, such as precipitation type charts, ensemble vertical profiles and ecPoint output, but all manner of new requests continue to be lodged. The ecCharts tool was widely praised, with a marked and reassuring reduction in the number of complaints about speed. The Open Charts initiative was also warmly welcomed, even if some issues still need addressing.
There is some customer dissatisfaction with the weak or incorrect signals seen in extended range (monthly) and (particularly) seasonal forecasts. This dissatisfaction seems to stem from unrealistic expectations linked to the considerable societal benefits that could be realised if accuracy were achieved. Regular reference to verification statistics alongside new forecasts could quell the misplaced optimism. ECMWF should make this easier; indeed user desire for more and better verification data, for longer ranges, was highlighted by an online survey in 2021.
The increased uptake of certain Copernicus products (e.g. from ERA5 and CAMS-IFS) was very good to see, although we would encourage NMSs to consider making more use of the C3S multi-model seasonal forecasts.
Every other summer Member and Co-operating States report on the application and verification of ECMWF’s forecast products for the previous two years. ECMWF also gathers feedback in other fora throughout the year. This report summarises feedback collected between summer 2019 and summer 2021.
Usage of ECMWF forecast products remains widespread across National Met Services (NMSs), from short range through to seasonal timescales. Very favourable comments regarding outputs and accuracy are commonplace, notably for the short range and the medium range. There are again plenty of examples of forecasting success in severe weather situations, e.g. for wintry precipitation, but also some examples of events being missed. Flash flooding remains very challenging for models in general, with moisture budget terms in certain situations still hard to disentangle.
In the short range and early medium range ECMWF output is ordinarily used alongside the output of high resolution, limited area, convection-resolving models (LAMs) and ensembles. Activities here that have expanded since 2019 are the use of LAM ensembles, downscaling, and automated (and manual) blending techniques. Blending is designed to extract maximum benefit, and seamlessness, from modelling systems with different strengths. Post-processing (prior to any blending) is also commonplace, and there are again examples of simple approaches like bias correction delivering significant gains. Machine-learning (ML) is still just on the horizon for most NMSs, although Switzerland do already deliver some ML-based operational products. Switzerland are also running what appears to be the highest resolution LAM ensemble (1.1km).
National Met Services have again performed comparative verification for LAMs, HRES (ECMWF’s 9km model) and ENS (ECMWF’s 18km ensemble), notably for surface weather. The difference in spatial scale between LAMs and global models can affect verification results, and countries use a variety of approaches to account for this. Overall, using a neighbourhood approach and a combined index for precipitation and wind gust, France report AROME-FRANCE consistently outperforms the global HRES and ARPEGE models. Israel report their LAM ensemble to be clearly superior to ENS for all variables. More generally, relative to the IFS, LAMs have a clear advantage for 10m wind prediction, especially for mountains, and some advantages for 2m temperature. Problem areas for 2m temperature, for many models, are clear calm nights, hot sunny days, and springtime. The scale issues noted above are especially relevant for fields exhibiting high spatial variability, such as precipitation (and some other moisture-related variables). Nevertheless, even if not accounting for related ‘double penalty’ issues, the reports generally show that the LAMs also perform well for precipitation, while for low-level moisture variables results are less consistent, with IFS often performing better. Interestingly Belgium identified a multi-day drift in several surface weather variables in IFS forecasts for Belgian sites; we had some awareness of these but are now investigating further.
New and innovative NMS-designed diagnostics include fire ignition risk from Portugal, based on dry thunderstorm probability, hail predictors developed in Hungary, and stacked probability bar charts for visibility from The Netherlands. Services were very positive about ECMWF’s “progressive” forecast products, such as precipitation type charts, ensemble vertical profiles and ecPoint output, but all manner of new requests continue to be lodged. The ecCharts tool was widely praised, with a marked and reassuring reduction in the number of complaints about speed. The Open Charts initiative was also warmly welcomed, even if some issues still need addressing.
There is some customer dissatisfaction with the weak or incorrect signals seen in extended range (monthly) and (particularly) seasonal forecasts. This dissatisfaction seems to stem from unrealistic expectations linked to the considerable societal benefits that could be realised if accuracy were achieved. Regular reference to verification statistics alongside new forecasts could quell the misplaced optimism. ECMWF should make this easier; indeed user desire for more and better verification data, for longer ranges, was highlighted by an online survey in 2021.
The increased uptake of certain Copernicus products (e.g. from ERA5 and CAMS-IFS) was very good to see, although we would encourage NMSs to consider making more use of the C3S multi-model seasonal forecasts.