The following is a summary of various studies based on daily precipitation events. These include the verification studies from 2017 and 2019 as well as the provider comparison analysis.
Daily precipitation sums are used to classify precipitation events. There are four different daily precipitation event thresholds defined: Depending on the study, precipitation amount above 0.1 mm, 1 mm, 10 mm and 50 mm are estimated. For all options, it will be evaluated, if precipitation was forecasted and if it was measured, leading to different cases displayed in the following contingency table.
|Forecast||YES||a "Hit"||b "False Alarm"|
|NO||c "Miss"||d "Correct Rejection"|
POD (Probability of Detection)
POD calculates the probability if a measured precipitation event was forecasted. Values range between 0 and 1, with 1 being the best value.
FAR (False Alarm Ratio)
FAR calculates the ratio of a non-precipitation-event to all forecasted precipitation events. The values range between 0 and 1, with 0 being the best value.
The previously described error metrics give an idea of the accuracy of the forecast, showing how precise the forecast was in comparison to the measurements. The skill, however, is a measure used to evaluate the capability and competency traits of the forecast over the standard forecast. POD and FAR are error metrics which do not estimate the skill of a model. For example, the false alarm rate in deserts would also be 0 or very close to 0 as it rarely or never rains. In this case the forecast accuracy would have additionally been around or above 99%. However, a skill score quantifies the model performance to randomly correct estimates. Hence, meteoblue additionally considers the Heidke-Skill-Scrore as a skill score.
The HSS is calculated from the proportion of correctly predicted values (PC) out of all values and the random luck value (E), which can also be calculated from the values of the contingency table (a-d). Values range from - ∞ to 1, with 1 means a good forecast.
Global verification study 2017
Throughout this study, 6505 METAR precipitation measurements (worldwide distributed) of the year 2017 were used to verify and compare our meteoblue MultiModel with other forecast models like ERA5, GFS, CHIRPS2 and NEMS.
For historical data, the model performance of ERA5 and the meteoblue MultiModel is significantly better than the satellite observation CHIRPS2. Satellite observations typically perform better than numerical weather forecast models for heavy precipitation and for areas close to the equator. The following table shows also, that the metoblue MultiModel has a higher (or same) HSS than ERA5.
|Daily precipitation > 1mm||Daily precipitation > 10mm||Daily precipitation > 50mm|
Global verification study 2019
The year 2019 was analyzed in detail in a separate bachelor thesis (Huonder, 2020). Precipitation measurement data of 8112 WMO stations (worldwide distributed) were compared with the 24h precipitation forecast of different models (ERA5, NEMSGLOBAL, GFS05, MFGLOBAL) for the year 2019 (Bachelor thesis Huonder, 2020).
A comparison of the HSS for different months shows that ERA5 performs very well and achieves the highest HSS for the majority of the months with the exception of November and December, where GFS has a higher HSS. The highest HSS of all models were achieved in December, the lowest HSS values occur in the northern hemispheric summer months.
|Daily precipitation > 1 mm (2017)||Daily precipitaiton > 1mm (2019)|
The competition analysis between our forecast and different competitors allows to evaluate the performance of meteoblue’s forecast in the market. The following results refer to daily precipitation events. The analysis is based on 24h precipitation forecast. The analysis was conducted for January to June 2021 for 100 locations in North America based on precipitation measurements from METAR and for 500 locations worldwide based on GSOD measurements.
|Events > 0.1mm||Events > 1mm||Events > 10mm|
Note, that the HSS for North America is typically larger than for the entire world, where the meteoblue forecast reaches HSS values of 0.44 (based on the analysis between January and June).
|Raw Model||Events > 0.1mm||Events > 1mm||Events > 10mm|
For all precipitation events, the meteoblue forecast has a higher HSS than the other chosen providers.