Why are models so bad?

[RitualOfTheTrout]

I think chaos plays a huge factor but in some patterns (like last winter) while chaos still exists the number of potential outcomes is less so it is easier for the models to predict what will happen. Sparse data input from critical areas also has to play a huge role in why models flip flop.

Don't forget this year we are also in a rather rare environment with a Strong Nina, and record blocking at the same time.

It is amazing to think that we can take in observed data from the atmosphere, input it into a computer models which runs complex algorithms then displays a possible outcome in graphical form, and while it's not pinpoint accurate, ends up usually being damn close to what happens.

[GD0815]

I agree wholeheartedly. The models do an amazing job. People truly have gotten spoiled or they wouldn't complain when a forecast of a storm made Thursday for Sunday busts. It's quite a coup that the models can predict that there will be a storm somewhere off the east coast. Chaos is real. Similar threads get started almost every year when someone gets discouraged that their snow storm is not going to happen. It's like the models become the scapegoats.

you said it perfectly, models are made to be scapegoats when people are unhappy! never mind the fact that verification scores are getting better and better---who needs facts anyway It is just depressing that a professional met would make a thread like this, you'd like to think he would know better.

[LakeEffectKing]

IMO, the recent poor model performance likely has to do with the difficult forecasting environment (synoptic situation in a quite strong La Niña event). Minor errors in details can lead to dramatic forecast errors. Last winter (strong blocking and moderate El Niño) presented a much easier forecasting environment. Overall,, I don't believe models have become worse, even as the current difficult forecasting environment leads to a bad performance. I suspect that earlier versions of the GFS and Euro would be faring even worse were they still running.

Great post Don, and might I add to your point that such a pattern inherently has large gyres of UL's, where pieces of energy are interacting at various speeds, and tight rotational components may indeed cause wider range of solutions at T+xx than a normal sinusoidal type flow. Think about the physical equations and how they break down in the core of a hurricane wrt modeling/forecasting (basically requiring grid spacing on the scale of meters in order to have the ability to model such extreme parameters)

In Homer Simpson type vocabulary, "spinning things" are harder to model.....they are more dynamic (vorticity interactions within the circulation envelope and in general, tend to have faster flow in a larger areal coverage than normal stream flow.

[Guest someguy]

I know this sounds strange, but it just seems like recently, the models have gotten even worse! The ECMWF (the supposed King of the Models) had a complete garbage solution yesterday, that everyone figured was real, because after all, it was the ECMWF. Even the GGEM showed a similar solution. The GFS HAD shown this, but days and days ago. It seems like the modeling really isn't any better than it was in earlier years right now. Not only are they going to screw up the position of the low, but in earlier runs, there was a BOMB out to sea. Now the OTS low is a strung out piece of garbage, so even if it DID come up the coast, it would not be a big deal! I guess, the GFS and GGEM deserve the most credit for how they handlded this storm, but even they were far from stellar.

I dont see the point of this thread at all . Not from a professional stand point.

Its Kind of something JI would post.

Its a La Nina winter ... if you have ever forecasted in a Mid / strong la Nina winter the crappy Model performance is very common

[Harry]

But they are actually (and it's not even close)....I'll try to compile some stats if/when I get a chance.

That would be great. Thanks.

Is it possible to do that and say starting at November 1st? Again i notice the performance seems to go down when winter is setting in.

BTW.. I would take the older euro in a heartbeat. Sorry but the so called King has not been that king it was prior to it's update. Yes it is still very good but some issues with how it handles some events seem to have creeped in since it's so called update/upgrade. The ensembles have been doing alot better though with both the GFS/Euro..

[Guest someguy]

Then what are we supposed to do? Lie and say how good the models are? Heck, numerical temperature predictions on day 1 and 2 from the MOS sets have been off by 3-7 degrees. Maybe if more mets like me would complain, something could actually get accomplished.

or one could become a better meteorologist instead of a Modelologist

[Harry]

I dont see the point of this thread at all . Not from a professional stand point.

Its Kind of something JI would post.

Its a La Nina winter ... if you have ever forecasted in a Mid / strong la Nina winter the crappy Model performance is very common

Thats the point of the thread Dave. Note dtk and what he does.

[isohume]

I wouldn't be surprised by a statement like that out of a member of the general public, but out of a trained Met its quite surprising to me. I know when I learned the entire process of numerical weather modeling in detail, its shortcomings, and what is actually going on, I was amazed how good it was. I still am amazed that they do so well with something so incredibly complicated, and they ARE getting better. Occasionally they all waffle on solutions but what should be expected, perfection? The entire process is fraught with imperfections. But show me anyone else in the world who so accurately can predict the future about anything.

Another thing saying things like that does is stir up the weenies, they see a Met saying something like that and they go spewing it around the forum how the models suck, etc. I'll say it here, the models don't suck, they do a great job, sometimes amazing, at predicting an enormous, chaotic set of physical processes with relatively little to go on. To me, it's one of the top scientific achievements of the last 30 years.

Excellent points and I agree.

[donsutherland1]

I agree with what many have written about the models. They have improved markedly over the years to the extent that they are indispensable. They aren't perfect, but they are remarkable given the complexities they deal with.

Having said that, ensemble forecasting can be very helpful during periods such as the current one when the modeling does not perform as well.

With respect to the MOS errors, if one tracks MOS forecasts/results and keeps a moving average of errors, one can also better identify periods/patterns when MOS performance is somewhat weaker and compensate accordingly. That's in addition to compensating for current synoptic characteristics that require adjustments.

[uncle W]

I remember many a blown forecast 24hrs in advance...Now that is very rare...What looks bad is seeing a forecast change every 12 hours for an event a week away...For snow lovers a forecast for a snowstorm a week away should come with a multitude of disclaimers...I watched a video with Joe Lundburg this morning for free and got the facts not the fiction...

[tmagan]

Its because of events like these, that I don't think NHC forecasts should extend beyond three days, especially since most of the extreme weather happens closer to the center of the tropical cyclone than in regards to the extratropical cyclone.

[Ellinwood]

I'm 100% on board with the "The models are constantly improving, and why would a met start this thread?" sentiment.

Then what are we supposed to do? Lie and say how good the models are? Heck, numerical temperature predictions on day 1 and 2 from the MOS sets have been off by 3-7 degrees. Maybe if more mets like me would complain, something could actually get accomplished.

I DARE you to say that to the developers. Improvements and theories are constantly being worked on, tested and corrected. There are BIG limitations to what you can actually get out of a computer model. I would hardly think that complaining about it would make things any better.

Its because of events like these, that I don't think NHC forecasts should extend beyond three days, especially since most of the extreme weather happens closer to the center of the tropical cyclone than in regards to the extratropical cyclone.

Because no one wants forecasts beyond 3 days, anyway... The 4-7 window is a big deal in the trade and risk businesses and in oil production.

[baroclinic_instability]

GEM has some simple verification here vs. radiosondes/height field. http://www.weatherof...nthly_ts_e.html

Doesn't look any worse than previous years and is in fact a tad better and fits the general trend that has been going on for what, 40 years?

First, I think some have such amazingly high expections of weather models to be perfect since they have become so good. Second, the pattern we have been in has been rather sensitive and there certainly has been a number of high profile busts lately. This happens in events where the potential for rapid positive feedback cyclogenesis can occur. What normally would have been a track error of 100 miles can suddenly result in a storm orders of magnitude larger/stronger, for instance (last weeks storm comes to mind).

This nor'easter was another example of "threading the needle" as many expected. I think most forecasters kept a very low probability of this storm happening even when a consensus came in suggesting otherwise. Why? So much had to happen at the exact right time. It is impossible for models to initialize perfectly and model a perfect solution under those conditions.

[Vandy]

Maybe if more mets like me would complain, something could actually get accomplished.

This implies that there aren't whole groups of folks who ARE continuously trying to improve the models. IMO, any system where you are forced to parametrize important things such as land-surface interaction and convection is bound to have errors. Considering those limitations, I think it's impressive they work as well as they do. Remember, errors propagate upward in scale.

[SnowGoose69]

I agree that the models are better with TC forecasting, but I don't think they're any better at all with mid-latitude cyclone forecasting.

The major improvement inside 3 days, outside of that I'm not sure they're much better than they were 15 years ago...25 years ago? Yes probably.....a big reason for the improvement inside 3 days is probably due to the NGM and LFM being replaced....as I said a week ago, I think 1988 to 1995 or so was the biggest improvement period we'll ever see in the models. Believe me, its a strong possibility than if this 1990 and not 2010 that many of the models still show this storm being a hit, we just don't know....the 2/5 event last year is a good example of generally how strong the models are today in the short range....thats an event I'm pretty sure the old ETA and AVN would have missed and brought well up the coast...the NGM would have certainly done it since the NGM did not understand the concept of cold dry air suppressing systems.

[baroclinic_instability]

I'm not saying that people don't try their best. I'm just surprised that with all the advances in technology and science over the years, it's amazing that models really are still so bad.

Sorry but I have to call you out here. This is just such an uninformed and arrogant comment to make considering the advancements in technology and modeling. The simple fact is models have become so good I think some mets just think they should be perfect. Can't lie about the statistics. The improvements in weather forecasting over the last 50 years has saved innumerable lives and there is really no such thing as "surprise" winter storms that cripples cities (think emergency preparation) and kills hundreds.

Weather forecasting has generally changed (see the HPC thread pinned in this forum) and weather forecasters are becoming short-term impact forecasters since models have improved so much. I don't think this storm was a bust, and the models did a good enough job showing the forecasters what all needed to happen for it to occur. As I said, most forecasters were going very low probability. This was by no means a "bust" forecast or a model bust.

[Analog96]

I don't think the models are that bad, and I believe that a model is generally as good as its A ) input data, and B ) resolution. I know that's probably an oversimplification, but it needs to be kept in perspective. Take a global model that plots out height and wind forecasts at different pressure levels, from 1000 mb to 200 mb, every six hours from now to several days out... one has to ask this question:

Is a 100 mile shift in a mid-latitude cyclone track 4 days out really significant in the grand scheme of things?

Looking at the the general weather forecast for the entire CONUS or even the entire northern hemisphere, I would say no. However, if we're looking at IMBY snowfall accumulations and where the rain/snow line will be, then the answer is YES. That's where the challenge is.

A 100 mile shift in 24 hours is fine. A 700 mile shift in 12 hours is disturbing.

[Analog96]

Sorry but I have to call you out here. This is just such an uninformed and arrogant comment to make considering the advancements in technology and modeling. The simple fact is models have become so good I think some mets just think they should be perfect. Can't lie about the statistics. The improvements in weather forecasting over the last 50 years has saved innumerable lives and there is really no such thing as "surprise" winter storms that cripples cities (think emergency preparation) and kills hundreds.

Weather forecasting has generally changed (see the HPC thread pinned in this forum) and weather forecasters are becoming short-term impact forecasters since models have improved so much. I don't think this storm was a bust, and the models did a good enough job showing the forecasters what all needed to happen for it to occur. As I said, most forecasters were going very low probability. This was by no means a "bust" forecast or a model bust.

I'm talking about 10-20 years, not 50 years. I think 50 years is obvious.

[Analog96]

The major improvement inside 3 days, outside of that I'm not sure they're much better than they were 15 years ago...25 years ago? Yes probably.....a big reason for the improvement inside 3 days is probably due to the NGM and LFM being replaced....as I said a week ago, I think 1988 to 1995 or so was the biggest improvement period we'll ever see in the models. Believe me, its a strong possibility than if this 1990 and not 2010 that many of the models still show this storm being a hit, we just don't know....the 2/5 event last year is a good example of generally how strong the models are today in the short range....thats an event I'm pretty sure the old ETA and AVN would have missed and brought well up the coast...the NGM would have certainly done it since the NGM did not understand the concept of cold dry air suppressing systems.

I think you understand my point perfectly! I was talking about the late 1990s vs now.

[Analog96]

I'm 100% on board with the "The models are constantly improving, and why would a met start this thread?" sentiment.

I DARE you to say that to the developers. Improvements and theories are constantly being worked on, tested and corrected. There are BIG limitations to what you can actually get out of a computer model. I would hardly think that complaining about it would make things any better.

Because no one wants forecasts beyond 3 days, anyway... The 4-7 window is a big deal in the trade and risk businesses and in oil production.

Why would you dare me? Would they threaten me?

[baroclinic_instability]

I'm talking about 10-20 years, not 50 years. I think 50 years is obvious.

We need a stats person here to dig some up. I am pretty sure the general statistical trend has been WAY up in a relatively consistent fashion for the last 10-20 years as well.

[Ellinwood]

Why would you dare me? Would they threaten me?

Because they put more effort into it than you'll ever know. There's a reason why things have only been gradually improving over the last 10-20 years... because of the limitations that are a part of computing. Have you even looked at the code and/or have taken a modeling class? Even the smallest errors can cause catastrophe in the long run, and these small errors are spawned in the data gathering and initialization processes in addition to the truncation error, boundary conditions, etc. that is part of the computing proccess, so to ask for leaps and bounds at this stage in the game is ludicrous.

[baroclinic_instability]

Why would you dare me? Would they threaten me?

Read Deterministic Nonperiodic Flow by Lorenz. Perfection will never happen. BEsides the fact perfect data assimilation will never occur unless we have infinite and continuous observations with NO error associated with them, the model equations are truncated because we have not solved the full Navier Stokes equations.

If you want to win a million dollars, it is a Millenium Prize Problem. Until then, some amount of turbulence will exist...and therefore chaos will as well.

http://en.wikipedia...._and_smoothness

[OSUmetstud]

[baroclinic_instability]

I think the arguments in this thread claiming suddenly crappy numerical guidance have all officially been squashed with pretty solid evidence.

[Cheeznado]

I will play middle of the road in this discussion- of course the models have gotten better, the statistics show it. With that said however, there is still a very very long way to go before you will get to a point where the 4-7 day period has any sort of true reliability which would prevent the type of extreme swings and poor performance exhibited recently- it may never happen, at least in my lifetime. The atmosphere is just way too chaotic IMHO.

[Hoosier]

Read Deterministic Nonperiodic Flow by Lorenz. Perfection will never happen. BEsides the fact perfect data assimilation will never occur unless we have infinite and continuous observations with NO error associated with them, the model equations are truncated because we have not solved the full Navier Stokes equations.

If you want to win a million dollars, it is a Millenium Prize Problem. Until then, some amount of turbulence will exist...and therefore chaos will as well.

http://en.wikipedia...._and_smoothness

No doubt there's been some great advances but this brings up an interesting point. Are we going to see the rate of improvement plateau since there is only so much data that can be collected? Seems pretty likely we're going to hit the "wall" so to speak.

[baroclinic_instability]

No doubt there's been some great advances but this brings up an interesting point. Are we going to see the rate of improvement plateau since there is only so much data that can be collected? Seems pretty likely we're going to hit the "wall" so to speak.

Yeah definitely agree. The overall size of the improvements will become smaller. I think this is why organizations such as the NWS for instance are changing to impact based forecasting in the short term for this very reason. Forecasting roles in general are changing. The typical role a met used to play will no longer exist, but forecasters in general will always exist (See my discussion in the "HPC" pinned thread). It is the same with private weather forecasting. Long range teleconnections and whatnot are also a mainstay, but that is a different discussion.

[baroclinic_instability]

Another thing not mentioned much (or at all here) is the work of ensembling which has decreased variability even more and well beyond the individual model improvements themselves.

[Rainshadow]

This. All you have to do is look at a long enough time series of verification to see how much better the models are today (even compared to only 5 years ago). I also (strongly) suspect that the pre-July version of the GFS would have fared even worse than the current version did....

I agree.