Why are models so bad?

[Analog96]

I know this sounds strange, but it just seems like recently, the models have gotten even worse! The ECMWF (the supposed King of the Models) had a complete garbage solution yesterday, that everyone figured was real, because after all, it was the ECMWF. Even the GGEM showed a similar solution. The GFS HAD shown this, but days and days ago. It seems like the modeling really isn't any better than it was in earlier years right now. Not only are they going to screw up the position of the low, but in earlier runs, there was a BOMB out to sea. Now the OTS low is a strung out piece of garbage, so even if it DID come up the coast, it would not be a big deal! I guess, the GFS and GGEM deserve the most credit for how they handlded this storm, but even they were far from stellar.

[michsnowfreak]

I know this sounds strange, but it just seems like recently, the models have gotten even worse!

IMO they have gotten WAY worse. I have never been as into following/decoding the models as many on these boards, only following when a storm threat is nearing (yes, I do casually glance at the LR GFS every day...."casually glance" the key words). But I actually spent a decent amount of time watching the models for our Dec 12th storm threat. I saw them waffle from direct hit, to WAY north, to WAY south. In the end, I was pleased with over 6 inches of snow, but funny thing is it occurred because of a scenario not ONE model saw coming until LESS than 12 hours before the storm began. A secondary L formed over Lake Erie, enhancing heavy snow over eastern MI and in the process taking the oomph out of the deform snow from the main L (after it dumped major snow on MN/WI). It was the main L that had 100% of my attention in the week leading up to the storm, because the scenario that actually panned out was never even forecast as a possibility. You cant even count on the models a day before anymore.

[mdsnowlover]

Sir I think to be fair, models are written by humans, who do their best to compile a mathematical representation fo what is going on in the atmosphere. You can't see air, and I,m sure they do their best. Maybe that is why you are trained so well. I , for one, take a model with a big grain of salt. We seem to be chasing model runs like their gospel. I remember that some of our best storms were NOT PREDICTED in advance. I am distressed that our gvt wont put into effect the assimilation system used on the euro. Meteorology is an important science for all concerned. It is also very good that, you as a trained met, can tell us when a model run is crap. Thank you for your efforts and time spent on this board. Maybe, in the future, the gfs will be changed with the new assimilation system. Until then, you and your fellow mets, are an invaluable assests to the public and this board as you weed out what is not possible by the laws of physics. I read a lot of the posts on this board. Everyone does their best to read the models. In the end, MOTHER NATURE will have her way. Have great holidays

[Analog96]

IMO they have gotten WAY worse. I have never been as into following/decoding the models as many on these boards, only following when a storm threat is nearing (yes, I do casually glance at the LR GFS every day...."casually glance" the key words). But I actually spent a decent amount of time watching the models for our Dec 12th storm threat. I saw them waffle from direct hit, to WAY north, to WAY south. In the end, I was pleased with over 6 inches of snow, but funny thing is it occurred because of a scenario not ONE model saw coming until LESS than 12 hours before the storm began. A secondary L formed over Lake Erie, enhancing heavy snow over eastern MI and in the process taking the oomph out of the deform snow from the main L (after it dumped major snow on MN/WI). It was this main L that had 100% of my attention in the week leading up to the storm, because the scenario that actually panned out was never even forecast as a possibility.

I agree. Models have been routinely waffling by hundreds of miles every 6-12 hours.

[Analog96]

Sir I think to be fair, models are written by humans, who do their best to compile a mathematical representation fo what is going on in the atmosphere. You can't see air, and I,m sure they do their best. Maybe that is why you are trained so well. I , for one, take a model with a big grain of salt. We seem to be chasing model runs like their gospel. I remember that some of our best storms were NOT PREDICTED in advance. I am distressed that our gvt wont put into effect the assimilation system used on the euro. Meteorology is an important science for all concerned. It is also very good that, you as a trained met, can tell us when a model run is crap. Thank you for your efforts and time spent on this board. Maybe, in the future, the gfs will be changed with the new assimilation system. Until then, you and your fellow mets, are an invaluable assests to the public and this board as you weed out what is not possible by the laws of physics. I read a lot of the posts on this board. Everyone does their best to read the models. In the end, MOTHER NATURE will have her way. Have great holidays

I'm not saying that people don't try their best. I'm just surprised that with all the advances in technology and science over the years, it's amazing that models really are still so bad.

[michsnowfreak]

I agree. Models have been routinely waffling by hundreds of miles every 6-12 hours.

Even when the event is there. Really hard on mets I imagine. The Dec 12th scenario made Detroits "Winter Weather Advisory" turn into 6.3" of snow at DTW, first heavy wet snow, then blizzard conditions, and Milwaukees "Blizzard Warning" turn into 1.1" of snow at MKE.

[Analog96]

Even when the event is there. Really hard on mets I imagine. The Dec 12th scenario made Detroits "Winter Weather Advisory" turn into 6.3" of snow at DTW, first heavy wet snow, then blizzard conditions, and Milwaukees "Blizzard Warning" turn into 1.1" of snow at MKE.

Wow! I didn't follow that area that closely! I didn't realize the busts were that bad out there!

[Confuzzled]

I suppose that is what you get when you want to forecast the weather reliably beyond 60 hours or think models 4-5 days into the future are actually reliable. Or reliable enough to be dissapointed when they don't verify

I too am a victim of the weenie casting when the Euro shows a bomb dropping a foot plus on me but in reality all we ever really knew with a high degree confidence is that a storm was developing in the southeastern us and was moving in a northeastward direction. The track and strength of the low were largely unknown.

Our grid spacing provides us with huge sampling errors (even larger over the ocean), we have limitations on processessing time, mathematical approximations abound and the non-linear nature of our atmosphere all fighting against us. Not to mention the tweaking of initial variables provides radically different solutions (ensembles).

I'm certainly dissapointed, but not in the models really. I just love snow. Its just basic chaos theory here. Something everyone interested in meteorology should be well aware of.

[The Iceman]

my guess this year would be because of la nina. much faster flow than normal and little changes make huge differences in the run.

[mdsnowlover]

I'm not saying that people don't try their best. I'm just surprised that with all the advances in technology and science over the years, it's amazing that models really are still so bad.

In all honesty, I,m not surprised. I keep hearing the word chaos, and I think that is really the answer. You would think that when the models smell out a storm, they would stick with what they originally determined. But as DT said last night, things change. I,m sure the models reflect this word very well. Unfortunately, that is the way things are. I agree with you, but in reality, it s the way it is. It's too bad. Thanks your all your efforts

[usedtobe]

I know this sounds strange, but it just seems like recently, the models have gotten even worse! The ECMWF (the supposed King of the Models) had a complete garbage solution yesterday, that everyone figured was real, because after all, it was the ECMWF. Even the GGEM showed a similar solution. The GFS HAD shown this, but days and days ago. It seems like the modeling really isn't any better than it was in earlier years right now. Not only are they going to screw up the position of the low, but in earlier runs, there was a BOMB out to sea. Now the OTS low is a strung out piece of garbage, so even if it DID come up the coast, it would not be a big deal! I guess, the GFS and GGEM deserve the most credit for how they handlded this storm, but even they were far from stellar.

If you read my discussion/blog, I tried to explain at least partially why the models have been having such a hard time. With the monster block and only a flat ridge in the west there is a constant stream of shortwave coming across the u.s. I fast flow like this, the models have always had big problems for as long as I've been forecasting. The models haven't gotten suddenly worse though as the resolution increases the butterfly effect becomes more pronounced so in this kind of flow, they can appear to be terrible but give the FEb 5th of last year and they'l seem great.

[vortmax]

It not so much the models getting worse as it is the very unique and persistent pattern we are experiencing. The models are not programmed to handle this type of rare pattern very well. Hopefully the 'humans' will recognize this and improve their performance during patterns such as these. JMHO...

[donsutherland1]

IMO, the recent poor model performance likely has to do with the difficult forecasting environment (synoptic situation in a quite strong La Niña event). Minor errors in details can lead to dramatic forecast errors. Last winter (strong blocking and moderate El Niño) presented a much easier forecasting environment. Overall,, I don't believe models have become worse, even as the current difficult forecasting environment leads to a bad performance. I suspect that earlier versions of the GFS and Euro would be faring even worse were they still running.

[Srain]

If you read my discussion/blog, I tried to explain at least partially why the models have been having such a hard time. With the monster block and only a flat ridge in the west there is a constant stream of shortwave coming across the u.s. I fast flow like this, the models have always had big problems for as long as I've been forecasting. The models haven't gotten suddenly worse though as the resolution increases the butterfly effect becomes more pronounced so in this kind of flow, they can appear to be terrible but give the FEb 5th of last year and they'l seem great.

Experience with patterns certainly helps the forecaster in understanding the limitations of numerical guidance, IMO Wes.

IMO, the recent poor model performance likely has to do with the difficult forecasting environment (synoptic situation in a quite strong La Niña event). Minor errors in details can lead to dramatic forecast errors. Last winter (strong blocking and moderate El Niño) presented a much easier forecasting environment. Overall,, I don't believe models have become worse, even as the current difficult forecasting environment leads to a bad performance. I suspect that earlier versions of the GFS and Euro would be faring even worse were they still running.

Don, I agree. This is an excellent teaching tool for the younger forecasters and will benefit them when such environments repeat in the future.

[DaculaWeather]

It not so much the models getting worse as it is the very unique and persistent pattern we are experiencing. The models are not programmed to handle this type of rare pattern very well. Hopefully the 'humans' will recognize this and improve their performance during patterns such as these. JMHO...

I agree, I think the extremes that the models are having to deal with is throwing them off. That's why I tend to trust the mets more than the models, at least there you have some human interpretation based on history and experience. And another reason I'm not discounting the weekend system, at this time of year in the south, the models nor anyone else REALLY has a clue what's going to happen.

[psuhoffman]

OMG we go through this every time there is a storm like this where we all know its a tricky situation with lots of moving parts that require phasing and perfect timing and blah blah blah...then the models struggle (big surprise) and someone freaks out about it. The models are not getting worse and did not suddenly go to crap. They are having a hard time because this was a very delicate situation where minor adjustments in the placement and strength of fairly meso scale features made enormous differences to the actual end result. Some storms are easy to see...when you have a massive STJ wave blasting into a block...others that require threading the needle and rely on small peices of energy timing up correctly and on and on are much more problamatic for humans AND computers to forecast. What boggles my mind is most of the really well respected minds the past week have stated this, and have pointed out how his was a long shot due to all of the different factors that would have to come together perfectly and there was NEVER a lot of model consensus in favor of this storm. The most we ever had was one or two models at a time showing a storm. And yet last night we still got the total freak out when the final death blow was dealt. That flip flop by some humans is way more disconcerting to me then the model flip flops.

[dtk]

IMO, the recent poor model performance likely has to do with the difficult forecasting environment (synoptic situation in a quite strong La Niña event). Minor errors in details can lead to dramatic forecast errors. Last winter (strong blocking and moderate El Niño) presented a much easier forecasting environment. Overall,, I don't believe models have become worse, even as the current difficult forecasting environment leads to a bad performance. I suspect that earlier versions of the GFS and Euro would be faring even worse were they still running.

This. All you have to do is look at a long enough time series of verification to see how much better the models are today (even compared to only 5 years ago). I also (strongly) suspect that the pre-July version of the GFS would have fared even worse than the current version did....

[michsnowfreak]

Wow! I didn't follow that area that closely! I didn't realize the busts were that bad out there!

Whats funny is that the northern burbs of Detroit were in a warning anyway, but again, it was for the original forecast of deform snow from the main L, not the new secondary that developed. Also, it cause the area to go into a near gridlock because of how unusual the storm was (and how poorly the models handled it until the last minute). The storm started as heavy rain and temps actually in the upper 30s, it quickly turned to heavy wet snow, and after about 4" of wet snow, calm winds, and the temp sitting at 33F for hours, cold front slammed through, the snow continued falling heavily but its consistency immediately turned powdery, with visibility nearing zero as winds gusted to near 40mph turning it into a blizzard and the temp began a 20-degree fall. It is very rare here to see temps fall 20 degrees during a storm. The wet snow from earlier then flash froze to the roads (many residential roads still today are caked with frozen snow the plows cant move) while the final 2" came in the form of powder, blowing and drifting everywhere. Most schools closed the next day, and it literally gridlocked a major metro area thats used to dealing with snow. Ive seen twice as much snow cause half the hassle.

[usedtobe]

IMO, the recent poor model performance likely has to do with the difficult forecasting environment (synoptic situation in a quite strong La Niña event). Minor errors in details can lead to dramatic forecast errors. Last winter (strong blocking and moderate El Niño) presented a much easier forecasting environment. Overall,, I don't believe models have become worse, even as the current difficult forecasting environment leads to a bad performance. I suspect that earlier versions of the GFS and Euro would be faring even worse were they still running.

You said it more eloquently than I did.

[MACoastWx]

The pattern has obviously made this weekend's potential weather difficult for the models to portray. At this range then is it safe to say the models are coming to a concensus given the obs data in the west or is the wx threat still completely up in the air?

[Harry]

This. All you have to do is look at a long enough time series of verification to see how much better the models are today (even compared to only 5 years ago). I also (strongly) suspect that the pre-July version of the GFS would have fared even worse than the current version did....

And we go back to this. Sure maybe they are doing better at 500mb but what good is that when it comes to individual events busting by 100s of miles? People are more concerned about each event vs how much they do better overall at 500mb. There is a problem and even wes mentioned it with how the models handle this type of pattern and so it IS a known issue with them. So what can be done or can anything be done to help correct this obvious issue they have with patterns such as this?

Oh and btw.. It is great to have your input here. I agree the models have gotten better ( mainly the GFS ) though over time just not with stuff like this.

[ohleary]

I'm not saying that people don't try their best. I'm just surprised that with all the advances in technology and science over the years, it's amazing that models really are still so bad.

I wouldn't be surprised by a statement like that out of a member of the general public, but out of a trained Met its quite surprising to me. I know when I learned the entire process of numerical weather modeling in detail, its shortcomings, and what is actually going on, I was amazed how good it was. I still am amazed that they do so well with something so incredibly complicated, and they ARE getting better. Occasionally they all waffle on solutions but what should be expected, perfection? The entire process is fraught with imperfections. But show me anyone else in the world who so accurately can predict the future about anything.

Another thing saying things like that does is stir up the weenies, they see a Met saying something like that and they go spewing it around the forum how the models suck, etc. I'll say it here, the models don't suck, they do a great job, sometimes amazing, at predicting an enormous, chaotic set of physical processes with relatively little to go on. To me, it's one of the top scientific achievements of the last 30 years.

[usedtobe]

I wouldn't be surprised by a statement like that out of a member of the general public, but out of a trained Met its quite surprising to me. I know when I learned the entire process of numerical weather modeling in detail, its shortcomings, and what is actually going on, I was amazed how good it was. I still am amazed that they do so well with something so incredibly complicated, and they ARE getting better. Occasionally they all waffle on solutions but what should be expected, perfection? The entire process is fraught with imperfections. But show me anyone else in the world who so accurately can predict the future about anything.

Another thing saying things like that does is stir up the weenies, they see a Met saying something like that and they go spewing it around the forum how the models suck, etc. I'll say it here, the models don't suck, they do a great job, sometimes amazing, at predicting an enormous, chaotic set of physical processes with relatively little to go on. To me, it's one of the top scientific achievements of the last 30 years.

I agree wholeheartedly. The models do an amazing job. People truly have gotten spoiled or they wouldn't complain when a forecast of a storm made Thursday for Sunday busts. It's quite a coup that the models can predict that there will be a storm somewhere off the east coast. Chaos is real. Similar threads get started almost every year when someone gets discouraged that their snow storm is not going to happen. It's like the models become the scapegoats.

[dtk]

And we go back to this. Sure maybe they are doing better at 500mb but what good is that when it comes to individual events busting by 100s of miles? People are more concerned about each event vs how much they do better overall at 500mb. There is a problem and even wes mentioned it with how the models handle this type of pattern and so it IS a known issue with them. So what can be done or can anything be done to help correct this obvious issue they have with patterns such as this?

Oh and btw.. It is great to have your input here. I agree the models have gotten better ( mainly the GFS ) though over time just not with stuff like this.

But that's the point....I strongly suspect that the older version(s) of the GFS/Euro would have done even worse with this situation. If I had the resources and time to do it, I'd run a weeks worth of analyses and forecasts myself with the old GFS model just to prove it.

BTW....I'm not just talking about improvements to 500 mb AC, the ability for global models to forecast everything from TC tracks (even TC genesis now with some usefulness), QPF, etc., is so much better recently it's not even debatable. Events like this (which have extremely large error growth rates) expose a predictability issue ..... the red flags for this storm have been up for days. People should have been far more focused on the ensembles and probabilities instead of clinging to deterministic model runs. If you go back and read some of Wes's posts, I think he summed up the uncertainty (and probabilities of the event actually coming to fruition) quite nicely.

Thanks for the kind words. Even though I work in NWP/Data assimilation, I'm still a weather weenie at heart. I"m thrilled to be able to clarify issues or answer questions (to the best of my ability). Keep the questions/comments coming (honestly, interacting with people and hearing complaints sometimes gives me things to look more closely at or even new ideas to try).

[Harry]

I wouldn't be surprised by a statement like that out of a member of the general public, but out of a trained Met its quite surprising to me. I know when I learned the entire process of numerical weather modeling in detail, its shortcomings, and what is actually going on, I was amazed how good it was. I still am amazed that they do so well with something so incredibly complicated, and they ARE getting better. Occasionally they all waffle on solutions but what should be expected, perfection? The entire process is fraught with imperfections. But show me anyone else in the world who so accurately can predict the future about anything.

Another thing saying things like that does is stir up the weenies, they see a Met saying something like that and they go spewing it around the forum how the models suck, etc. I'll say it here, the models don't suck, they do a great job, sometimes amazing, at predicting an enormous, chaotic set of physical processes with relatively little to go on. To me, it's one of the top scientific achievements of the last 30 years.

I don't think anyone is expecting perfection. Atleast i hope not. Outside of last winter/tropics they have not done all that well though. I lost count of the number of short term busts via the NWS etc biting into them and then having a suprise or two. Ofcourse part of this is probably a result of known biases as all pretty much knew what they were BEFORE the upgrades/updates. Example.. In 2007-08 it was almost a given models would trend nw with a system the closer we got to it. Thus now we have a bit more guess work involved again. So i suppose that could be why there is the perception they are not much better and perhaps are worse by some?

[Analog96]

I wouldn't be surprised by a statement like that out of a member of the general public, but out of a trained Met its quite surprising to me. I know when I learned the entire process of numerical weather modeling in detail, its shortcomings, and what is actually going on, I was amazed how good it was. I still am amazed that they do so well with something so incredibly complicated, and they ARE getting better. Occasionally they all waffle on solutions but what should be expected, perfection? The entire process is fraught with imperfections. But show me anyone else in the world who so accurately can predict the future about anything.

Another thing saying things like that does is stir up the weenies, they see a Met saying something like that and they go spewing it around the forum how the models suck, etc. I'll say it here, the models don't suck, they do a great job, sometimes amazing, at predicting an enormous, chaotic set of physical processes with relatively little to go on. To me, it's one of the top scientific achievements of the last 30 years.

Then what are we supposed to do? Lie and say how good the models are? Heck, numerical temperature predictions on day 1 and 2 from the MOS sets have been off by 3-7 degrees. Maybe if more mets like me would complain, something could actually get accomplished.

[Analog96]

But that's the point....I strongly suspect that the older version(s) of the GFS/Euro would have done even worse with this situation. If I had the resources and time to do it, I'd run a weeks worth of analyses and forecasts myself with the old GFS model just to prove it.

BTW....I'm not just talking about improvements to 500 mb AC, the ability for global models to forecast everything from TC tracks (even TC genesis now with some usefulness), QPF, etc., is so much better recently it's not even debatable. Events like this (which have extremely large error growth rates) expose a predictability issue ..... the red flags for this storm have been up for days. People should have been far more focused on the ensembles and probabilities instead of clinging to deterministic model runs. If you go back and read some of Wes's posts, I think he summed up the uncertainty (and probabilities of the event actually coming to fruition) quite nicely.

Thanks for the kind words. Even though I work in NWP/Data assimilation, I'm still a weather weenie at heart. I"m thrilled to be able to clarify issues or answer questions (to the best of my ability). Keep the questions/comments coming (honestly, interacting with people and hearing complaints sometimes gives me things to look more closely at or even new ideas to try).

I agree that the models are better with TC forecasting, but I don't think they're any better at all with mid-latitude cyclone forecasting.

[Confuzzled]

The pattern has obviously made this weekend's potential weather difficult for the models to portray. At this range then is it safe to say the models are coming to a concensus given the obs data in the west or is the wx threat still completely up in the air?

Gfs is initializing now

[dtk]

I agree that the models are better with TC forecasting, but I don't think they're any better at all with mid-latitude cyclone forecasting.

But they are actually (and it's not even close)....I'll try to compile some stats if/when I get a chance.

[Terpeast]

I don't think the models are that bad, and I believe that a model is generally as good as its A ) input data, and B ) resolution. I know that's probably an oversimplification, but it needs to be kept in perspective. Take a global model that plots out height and wind forecasts at different pressure levels, from 1000 mb to 200 mb, every six hours from now to several days out... one has to ask this question:

Is a 100 mile shift in a mid-latitude cyclone track 4 days out really significant in the grand scheme of things?

Looking at the the general weather forecast for the entire CONUS or even the entire northern hemisphere, I would say no. However, if we're looking at IMBY snowfall accumulations and where the rain/snow line will be, then the answer is YES. That's where the challenge is.