Meteorology 5540
Mini-Lecture 6: Forecasting Skill Scores

I. Common Skill Scores

  • ABS(Forecast-Observed) Score: The observed value is merely subtracted from that forecast with the absolute value representing the forecast error. Frequently used to evaluate temperature forecasts.
  • (Forecast-Observed)**2 Score: This approach squares the difference between the forecast and observed values. Large forecast errors are severely penalized. Frequently used to evaluate probability forecasts.
  • Threat Score: This is a categorical scoring system that is frequently used to score precipitaiton forecasts. The technique might consider the following categories: (A) the number of times a forecaster predicted rain and rain was observed (commonly called a "hit"), (B) the number of times the forecaster predicted no rain and rain was observed, (C) the number of times the forecaster predicted rain and no rain fell (commonly called a "false alarm") and (D) the number of times the forecaster predicted no rain and no rain fell. The resulting table looks like this:

    . Rain Forecast Rain Not Forecast
    Rain Observed A B
    Rain Not Observed C D

    The percent correct score is (A+D)/(A + B + C + D) while the threat score is given by A/(A + B + C). Note that the threat score is highly dependent on the threshold used to define a "hit" or "miss." For example, a numerical model may have outstanding threat scores when it comes to forecasting events that produce more than .01" of rain, but relatively poor scores for forecasting rain events > 1.00".

    II. Optimizing Skill Scores:

  • To limit error points a forecaster must understand what events may occur at the forecast site, determine the probability of those events occuring, and choose a forecast value based on the probability distribution that optimizes the forecast skill scores.
  • Optimizing ABS(Forecast-Observed) Score: Forecast the median event of the predicted probability distribution.
  • Optimizing (Forecast-Observed)**2 Score: Forecast the mean of the predicted probability distribution.
  • Optimizing Threat Score: An event should be forecast to occur when the probability is 50% or greater; otherwise it should be forcast to not occur.

    V. References and on-line links

  • Vislocky, R. L., and G. S. Young, 198?: Improving your weather forecasts through a bettern knowledge of skill scores. Nat. Wea. Dig., 13, 15-17.
  • Murphy, A. H., and R. W. Katz, 1985: Probability, Statistics, and Decision Making in the Atmospheric Sciences. Westview Press, Boulder, Co. 545 pp.
    Updated May 8, 1997