Accuracy of three major weather forecasting services

For the past month, I’ve been slowly working my way through Nate Silver’s book, The Signal and the Noise. It’s really a great read, but if you’re a regular reader on this blog, I’d imagine you’ve already read it. This book is loaded with all kinds of great examples of where predictive analytics succeeds and fails, and I decided to highlight his weather forecasting example because of how surprising it was to me.

For those who aren’t in the know: Most of the weather forecasts out there for the U.S. are originally based on data from the U.S. National Weather Service, a government-run agency tasked with measuring and predicting everything related to weather across all of North America. Commercial companies like The Weather Channel then build off of those data and forecasts and try to produce a “better” forecast — a fairly lucky position to be in, if you consider that the NWS does a good portion of the heavy lifting for them.

We all rely on these weather forecasts to plan our day-to-day activities. For example, before planning a summer grill out over the weekend, we’ll check our favorite weather web site to see whether it’s going to rain. Of course, we’re always left to wonder: Just how accurate are these forecasts? Plotted below is the accuracy of three major weather forecasting services. Note that a perfect forecast means that, e.g., the service forecasted a 20% chance of rain for 40 days of the year, and exactly 8 (20%) of those days actually had rain.


There’s some pretty startling trends here. For one, The Weather Service is pretty accurate for the most part, and that’s because they consistently try to provide the most accurate forecasts possible. They pride themselves on the fact that if you go to and it says there’s a 60% chance of rain, there really is a 60% chance of rain that day.

With the advantage of having The Weather Service’s forecasts and data as a starting point, it’s perhaps unsurprising that The Weather Channel manages to be slightly more accurate in their forecasts. The only major inaccuracy they have, which is surprisingly consistent, is in the lower and higher probabilities of raining: often forecasts that there’s a higher probability of raining than there really is.

This phenomenon is commonly known as a wet bias, where weather forecasters will err toward predicting more rain than there really is. After all, we all take notice when forecasters say there won’t be rain and it ends up raining (= ruined grill out!); but when they predict rain and it ends up not raining, we’ll shrug it off and count ourselves lucky.

The worst part of this graph is the performance of local TV meteorologists. These guys consistently over-predict rain so much that it’s difficult to place much confidence in their forecasts at all. As Silver notes:

TV weathermen they aren’t bothering to make accurate forecasts because they figure the public won’t believe them anyway. But the public shouldn’t believe them, because the forecasts aren’t accurate.

Even worse, some meteorologists have admitted that they purposely fudge their rain forecasts to improve ratings. What’s a better way to keep you tuning in every day than to make you think it’s raining all the time, and they’re the only ones saving you from soaking your favorite outfit?

For me, the big lesson learned from this chapter in Silver’s book is that I’ll be tuning in to for my weather forecasts from now on. Most notably because, as Silver puts it:

The further you get from the government’s original data, and the more consumer facing the forecasts, the worse this bias becomes. Forecasts “add value” by subtracting accuracy.

Dr. Randy Olson is an AI Scientist at Absci using data science and deep learning to make medicines better and make better medicines.

Tagged with: , , , , ,
56 comments on “Accuracy of three major weather forecasting services
  1. marc says:

    What about replacing the “probability” term on the verticla axis by something that sound more statistical ? (I mean, descriptive stats). “Observed probability” sounds strange to me, even if I am highly Bayesian-compatible.

  2. Rob Dale says:

    Very few TV meteorologists give forecast rain chances for a point like NWS forecasts do. How did you compare them?

    Your second paragraph is wholly inaccurate. I think you confused NWS forecasts made by meteorologists, and NWS computer model data. Nobody takes the NWS forecast and “tweaks” it better. We (private sector and NWS) use the same basic set of observations and model data to create our own forecasts.

    • Randy Olson says:

      Very few TV meteorologists give forecast rain chances for a point like NWS forecasts do. How did you compare them?

      The source for that comparison is here:

      • mark says:

        Since the NWS probability of precipitation is based on .01 or more, and this study only verified for .10 inch or more, the statistics mean nothing. The NWS is forecasting one thing, and this person was verifying something totally different. There’s many times when we know the rain will be light…less than .10 inches…but know it will be over .01, so the POP will be high as it should be.

  3. Jayson Prentice says:

    To say that private weather companies use the National Weather Services forecast and “build off of those forecasts” isn’t entirely correct. Nearly every forecast, both NWS and private, is translated from a number of numerical weather prediction models (computer models) in which a forecaster, or computer, then takes these forecasts to create the final forecast product.

    Each NWS forecast is derived from the models and other sources, but is ‘hand-crafted’ by a meteorologist to produce the forecast you see on Private companies will use the same model data (often provided by government sources) to produce their own forecast, either through a computer or a meteorologist, many times not even referencing or using the actual NWS forecast itself.

    • Randy Olson says:

      Thanks for pointing this out, Jayson. I’ve changed my wording slightly in the beginning paragraph to better reflect the truth of the matter.

  4. Robert says:

    I’m currently reading Silver’s book too! The most interesting so far is being able to predict the likelihood of earthquakes. Few theories have been introduced but none have been successful. Since those times, any heed for an algorithm that can predict likelihood for them goes largely ignored. I believe the and earthquake analytics talk in his book are in the same chapter. Won’t spoil for anyone, but it’s a good read!

    • Randy Olson says:

      The “for years you’ve been telling us that rain is green!” story in that chapter had me cracking up. Silver’s an entertaining writer.

  5. bcl says:

    If you’re interested in the science (and some of the problems we have in the US) read UW Professor Cliff Mass’ blog –

  6. Sara says:

    This is really interesting. However, I wonder if an ROC curve (or similar plot) would be more informative. The NWS may have more “accurate” long-run probabilities, but this should translate into a higher rate of Type I error than that of local forecasters. As you point out, the ratings penalty for a Type I error may be higher than that for a Type II error, and so erring “on the side of caution” may actually provide a better service to the public.

    • Randy Olson says:

      That is indeed TWC and the like’s gig: “Commercial forecasts ‘add value’ by subtracting accuracy.” In this case, subtracting accuracy may not always be a bad thing. Otherwise TWC and co. likely wouldn’t still be around. ๐Ÿ™‚

      • Sara says:

        Well, actually, I guess my point was that the information here isn’t informative about binary precipitation forecast accuracy, which is what’s more interesting to most people. This instead displays weighting of probabilities vs long-run true probabilities, which may just reflect overweighting of smaller probabilities by the target audience. What I think would be more informative re:accuracy is to look at this as a binary classification problem. For that, though, you’d need a different type of plot and probably a different type of data to actually say anything conclusive about “accuracy.”

  7. Ryan Wichman says:

    Isn’t it a bit irresponsible to include all broadcast meteorologists into a study of one city? While that study appears fair in KC, it’s a gross generalization to base a nationwide profession with significantly different weather on one location.

    • Randy Olson says:

      It’s hard to say if the generalization holds or not. Do you track the accuracy of your forecasts? I’d be curious to see this study expanded to local TV meteorologists across the entire U.S.

  8. Stephen says:

    This article is a science fail. As a local TV meteorologist, I wonder whose forecasts you have actually looked at. I can also tell you this whole “fudge the forecast to improve ratings” is absolutely not the case. We are not in the business for that. This is a good example of shoddy science with what appear to be made up “facts” to support your claims. You clearly looked at a very small and unrepresentative sample of TV stations and NWS offices for your “study.” No forecaster, television or otherwise, is or will ever be 100% accurate. Thankfully some of the others who have commented on this post seem to notice the bias and inaccuracy with which you have reported. I would never say the National Weather Service does a bad job, but frequently they aren’t even in the area of the local TV station, and the tv meteorologists in those areas tend to perform quite well. To say NWS “consistently” outperforms tv meteorologists is, as much of your article seems to be, a gross and incorrect generalization.

    • Randy Olson says:

      Considering your profession, I understand the agitated tone in your comment here. But to call this article a science fail is a bit much. For one, the accuracy of the data on TWC and NWS is very difficult to dispute. After all, the data is from, who has made a career out of checking the accuracy of weather forecasts. The data for the local TV meteorologists is from the Freakonomics blog, which was indeed limited to several local channels in Kansas. Are all local TV meteorologists as bad as the Kansas ones? That’s hard to say. Maybe local TV meteorologists should start recording and reporting on their own accuracy if they’re worried about being trusted.

      • mark says:

        NWS forecasts the probability of .01 or greater of precipitation for a 12-hour forecast period from 00Z-12Z and from 12Z-00Z (6-6 standard time in the central time zone). The Freakonomics blog only counted rain of .10 or greater, so that cannot be used to determine the accuracy of NWS forecasts. NWS forecast high temperatures are valid from 7 am-7 pm local standard time, and forecast low temperatures are valid from 7 pm- 8 am local standard time. Unless the verification data followed those exact time definitions, then the temperature data would be invalid as well. There are many times when the “daily” low falls outside those time limits, and a lesser number of times when the “daily” high does..

      • Daniel Sackney says:

        Guessing it will rain is all good and fine.. I mean anyone who looks at the signs can predict with a decent certainty the chance it will rain… the amount and days down the road are what I think haven’t been perfected or even intermediatly mastered. Mother nature is too unpredictable for the tech we have.. find me a “meteorologist ” who can accurately “predict” the weather in the coming week.. the fact they say predict takes away all credibility. Its an educated guess. And the fact is meteorologists aren’t very educated . Through no fault of their own.. the science isn’t there yet. Its a matter of probability. Which is never certain.

        • Rob Dale says:

          Funny you say that… A few weeks ago we had a Friday with temperatures in the 50s and mostly sunny skies. Putting your thoughts into action, you’d say Saturday would probably be mild. I posted a forecast of 5-8″ of snow and the number of “It’ll never happen” replies was at an all-time high ๐Ÿ™‚

          We ended up with 8.1″ the next day.

          So give me my credibility back and remember that I’m highly educated…

          • Daniel Sackney says:

            Not being a qualified professional I wouldn’t pressure to tell u your job but, I’m sure many factors affected your prediction of precipitation for the following day. As I said educated guess. And hey! Ya got one. Albeit only one day away.. my criticism really focuses on the ability to accurately predict anything past 2 or 3 days ..

    • Marine Vet says:

      Figures, this article and data would offend the locals.

    • Chuck Cox says:

      Au contraire, mon TV meteorologist . . .
      You ARE in the business of improving ratings . . . otherwise you’d be out of a job. Just listen to the emotive adjectives you use to describe the weather conditions . . . And please don’t compare yourself to the atmospheric scientists at the NWS . . . you are a “broadcast meteorologist” only in the eyes of the trade association American Meteorological Society.

      • Rob Dale says:

        Good French – but the rest of your post was crap. For most broadcast meteorologists they get paid the same regardless of ratings. And it’s the same AMS who defines “meteorologists” that work in NWS.

        • Chuck Cox says:

          Thanks for your reply, but if the TV meteorologist is attractive and popular, with a good personality, the ratings will go up. And if the ratings go up, the TV station wants to retain the meteorologist, so the station pays he / she more to help retain the personality. Simple.
          Also, I don’t mean to offend.

          • Ward Drennan says:

            Yea, the post was pretty offensive to broadcast meteorologists. Just because they look good on air and talk well to maintain viewer interest, that doesn’t mean they aren’t good meteorologists. The super egg head PhDs usually aren’t in broadcasting!

      • Thibeinn says:

        “TV meteorologists” are not meteorologists, they’re weather casters. There’s a huge difference. Basically all they do is report on what the real meteorologists say the weather forecast will be (with “improve the ratings” added in).

        • Rob Dale says:

          Way to dig up an old thread with totally invalid information ๐Ÿ™‚ TV meteorologists are true meteorologists who make their own forecasts. There are TV weather casters who just report weather forecasts made by others. All NWS employees (forecasters) are meteorologists.

      • HarleyRdr says:

        I don’t think giving false forecasts is likely to improve ratings since people do make, or change, plans based on forecasts. If I want to plan an outdoor activity but decide against it because our local guy has predicted rain, if it doesn’t rain, I’m not particularly happy. If that happens a couple of times, I’ll start comparing his forecasts to others. If I find the others are more accurate, I will no longer view his channel and will start watching another. It’s not about personalities, though a pretty face and pleasant personality is enjoyable, it’s about having the best forecast possible for my plans. So, if they are in the “business of improving ratings” the best way to do it is to give the most accurate forecast…that’s THEIR contribution.

    • David Shimshon Heller says:

      The NWS offices located at most every major airport in the country do not forecast the weather ALL forecasts come from NOAA headquarters at least that was the way it used to be in 1980 when I worked for the NWS at the Oakland Airport in 1980 at the time the local NWS personal were not allowed to forecast anything!

      • Rob Dale says:

        No, that was never the case. In the 80s and early 90s there were state forecast offices (Michigan’s was in Ann Arbor) and then local offices (called Weather Service Offices.) The state forecast office made the forecasts, and the local office helped interpret that for pilots / public / etc. They could tweak the short-range outlook. But in any case, forecasts do not come from NOAA HQ at all.

        • David Shimshon Heller says:

          I will take your word for it as its been 36 years!! They did let me release the weather balloon once though if I promised not to tell anyone. LOL! And I did leave the NWS with a vial of Mt. St Helen’s volcanic ash. So it was a fun time I had. Sadly it was only a 6 month CO-OP with the local community college and the NWS man that put it together died about the time I started working. The idea was to train new technicians and promote them into the weather service as regular full time employees.

    • n5ifi says:

      I agree. The local guys are much better than stated. This bias might have been true in the 60’s but no so much any more. The worst service that I see consistently is weather underground.

    • Andy Johnson says:

      You may want to monitor Kalamazoo Channel 3 TV. for an example. A month ago I was sitting on the front porch watching 100% chance of rain followed by 2 hours of 80% chance of rain. The dryway was DRY over the entire time>>>>>>.

    • Jj Slyde says:

      How about me sir. I don’t know anything about weather forcasts. Total ignorance. I’m on this page, because Weather Underground is constantly about 10 degrees under the actual. My predicted day of 74 degrees today: 10am, it is already 84. Pretty hard to plan around that. I’m trying to find a source that can more accurately predict my up coming day. Simple. Just like me. (And just about everyone else).

  9. Ward Drennan says:

    I’d like to see real data on temperature accuracy as well as precipitation and data on the accuracy as a function of duration of time prior to the event. For example, is this for the “tomorrow” forecast. What about 2, 3, 4, 5, 10 days out?

  10. Piggyghost says:

    I would like to know how climate change is effecting weather predictions.

    • Rob Dale says:

      Not at all. Climate is long term – weather is short.

      • Piggyghost says:

        Of course. But climate change is affecting weather. For instance with the El Nina, we are seeing odd weather. And then there is the “polar express.” Also severity of storms. I suppose with short term forecasting you can still see it coming, regardless. But these changes do bring surprises.

        • Rob Dale says:

          El Niรฑo and the Polar Vortex are not climate change, those are natural occurrences. Climate is bigger picture than those. But yes, changes can bring surprises.

  11. LoveIsEqual1 says:

    Sorry but in my neck of the woods is almost NEVER right! They go from say 1″ of rain, to nothing, back to 1″, then 1/4″, then 1/2″, back to 1″ again and then down to 1/4″, all in the last couple of days before the “event”! is the most unreliable agency I think I have ever seen!

    They do similar with their temperature forecasts for high & low. I guess they figure if they change it enough times, literally up to a few short hours before the final ACTUAL results are in, they will finally hit it right. Ironically they are often wrong even with all the changes they constantly make!

    I understand that weather is very fluid and often hard to predict but this is such a common practice with that there is RARELY an exception to what I stated above.

    And I do know a bit about weather and also have a high end weather station at my home (high end for a home environment,not implying a professional station) which ran about $2000 total with all my sensors, etc.

    I always have to balance my outlook by looking at, NWS and at least 2 local network channels to feel like I even have any confidence whatsoever in the forecast!

  12. Krystal Kalleen says:

    I am so thankful to you for this information! Everyone always wonders how local forecasts are always inaccurate! I haven’t had time to research myself. Thank you!

  13. larrybud says:

    How long of a time out are these numbers? In other words, is this a 24 hour forecast? What’s the radius which we’re talking about here?

    In other words, I can go to the Weather Channel and get a prediction by zip code. Heck, with Accuweather they’ll even predict by address.

    Turn on TV weather, and they’re talking to a 50 mile radius.

  14. Indie says:

    Wunderground is very accurate because it uses *local* weather stations. I’ve believed for years that The Weather Channel doesn’t do much if any of its own forecasting. However, local station meteorologists do NOT fudge their forecasts for ratings.

  15. josephz2va says:

    The Weather Channel is inaccurate 3 out of 4 times in many areas. When I pull up the weather app (any phone OS even Google Chrome and Bing), weather channel says it’s pure sunshine or cloudy and it’s raining like no tomorrow. Weather channel says no snow predicted or cloudy forecast, and it snows.

  16. Clifford H. Poyneer says:

    Please include Weather Underground. They use their own network of privately owned weather stations.

  17. JustASoccerDad says:

    Yesterday evening as I got out of my convertible, I checked the Weather Channel app to see if it would rain overnight. To my joy, the forecast was 0% chance of precip for the next THREE days. I went to bed assured the morning would be extraordinary, driving to work with the wind blowing through my hair…only to awake to find nearly 1/2 of an inch of water in my cup holder and my car’s interior soaking wet. Had the forecast been 5%, ok. But 0%…and we got THAT much rain? Looks like I need to find a different app to consider.

  18. Ron Dvorak says:

    They should have a vertical line at 50% representing a random guess.

  19. Jeff Tappan says:

    A comedian some years ago said ” Most TV stations have a weather dog that they send out shortly before the broadcast. If it comes back wet, it’s gonna rain. If it’s got icicles on it, it’s gonna be cold. If it doesn’t come back, it’s gonna be windy ” . Facetious, I know. But, as an OTR truck driver. I rely on accurate forecasts, and I just don’t see that many.

  20. HarleyRdr says:

    It seems to me that the ratings for a local station, if the weatherman is giving consistently wrong forecasts of rain, would drop since people would be unable to trust that forecast. What I find strange is that if I don’t like a forecast all I need to do is check another forecast. Eventually, I’ll find one I like better…though it may not be accurate. Example: The Weather Channel, in our area, has proven to me to be highly inaccurate when forecasting rain. They often show a 50% – 60% chance of rain over the weekend, when checking 2 days out or more, where the Accuweather forecast shows 10% – 20%. On the day in question, there is no rain.
    This is not a scientific experiment but, as a result of my experience, I will not watch or view The Weather Channels forecast for our local area. Assuming others are similiar in thought, that would negatively affect their ratings. So if a local weatherman wants good ratings, give accurate forecasts.

6 Pings/Trackbacks for "Accuracy of three major weather forecasting services"
  1. […] Local weather channels consistently over-predict rain on purpose […]

  2. […] fascinating point from Nate Silver’s The Signal and the Noise is where he talks about how far into the future […]

  3. […] fascinating point from Nate Silver’s The Signal and the Noise is where he talks about how far into the future […]

  4. […] For those who arenโ€™t in the know: Most of the weather forecasts out there for the U.S. are origina… […]

  5. […] came across an interesting little story. Blogger Randy Olson reviews Nate Silverโ€™s book, The Signal and the Noise. Itโ€™s a study of how good (and bad) […]

  6. […] it going to rain? Your local weatherman exaggerates the worst. Hyperbole sells.”โ€ฆ […]