Why are models so bad?

[Guest someguy]

PERFECTLY stated Wes

if Greg / analog 96 had Made THIS point.... then he would be 100% correct

we are all going to have to graps this is -- at for the 1st half - a Moderate la nina wintern and it means a LOTS less model reliability and performance for THIS WINTER

last year the models handled that pattern much better overall. The feb 5th storm was one of the best long range forecasts of amjor dc area snowstorm that I can remember. It was on the radar when it was snowing Jan 31. The feb 10th was not quite as well forecast but still was way better than the last two this year. Maybe later in the month there was a northward shift to some of the storms but overall, they were forecast really well compared to this year. I guess our memories are different because of the difference in perceptions based on where we live.

[Guest someguy]

If I'm not mistaken, the LFM's greatest hit was the February Blizzard of 1978.

YES It was ... I STILL have the Model riuns from that

I will scan them in and posted them one day

soon

[Guest someguy]

I think so too.

Perhaps I'm starting to gain some experience when it comes to model watching, but I never expected this storm to bring any snow at all to the NYC area or anyone near or NW of that zone. Most of the model runs were offshore, especially the EC, despite a few blips. Most storms are going to have extreme forecast hiccups before the event. The key is to note the overall trend or consistency.

QUINCY!!!!

ya made it.... wow good for you!!!

always knew you had the pasison for this and I am galkd to see you had the determination

Bully

[Guest someguy]

They have been awful inside 96 hours this month. I think inaccuracies like we've just experienced are inexcusable in the year 2010.

wow... you mindset is Just awful

the model flips flops DID tell us something

the fact that you views the flips flops as failures is why you dont see this

[donsutherland1]

the model flips flops DID tell us something

Assuming nothing was lost in translation, for me, they were screaming high uncertainty/low confidence.

[dtk]

see PINNED thread

see 0z model thread...

basically for east coast snwostorm the GFS is useless piece of crap past 84 hrs

always has been

always will be until they go with 4DVAR and there aint no money for that

This is a pretty ridiculous overstatement. Also, there is no guarantee that 4DVAR solves anything necessarily (see NOGAPS, Canadian global GEM, etc.).

BTW, NCEP (myself included) does have 4dVAR development going on right now, in addition to new science beyond 4dvar. There are other threads/discussions about this.

Okay, back on topic now.....

[ohleary]

00Z DEC 20 GFS blinked

MODEL 1 says ABC and it said that foir 12 straight runs ( model 1 is GFS)

MODEL 2 says XYZ

MODEL 3 says XYZ

MODEL 4 says XYZ

MODEL 5 says XYZ

MODEL1 ensembles over the 12 Model runs do NOT say ABC... they say XYZ

MODEL 2 ensembles over the past 8 model runs all say XYZ

MODEL 3 ensembles all say XYZ for 8 runs in a row

then MODEL 1 new runs out and says XYZ

what does that tell us ?

Model has a clue or the that Model 1 is a Piece of crap?

Having Mets call specific models (mostly GFS and NAM) a piece of crap is going to drive me off this forum. It's something I expect from weenies that don't know any better, but from a Met? C'mon guys. If you don't like a particular run's solution, for whatever reason, just say that. Models are not pieces of crap, that sounds like a 12 year-old talking.

FYI, the GFS first sniffed this system out about 4 days ago, the same cycle of the Euro had nothing in its solution. I didn't hear anyone saying the Euro was crap at that point.

There is great model consensus with this system, and I doubt that any of them are absolutely correct at this point. And please tell me if the GFS trended toward the golden child Euro on this run, why it's "stupid"? The logic makes absolutely no sense.

[Ian]

Having Mets call specific models (mostly GFS and NAM) a piece of crap is going to drive me off this forum. It's something I expect from weenies that don't know any better, but from a Met? C'mon guys. If you don't like a particular run's solution, for whatever reason, just say that. Models are not pieces of crap, that sounds like a 12 year-old talking.

FYI, the GFS first sniffed this system out about 4 days ago, the same cycle of the Euro had nothing in its solution. I didn't hear anyone saying the Euro was crap at that point.

There is great model consensus with this system, and I doubt that any of them are absolutely correct at this point. And please tell me if the GFS trended toward the golden child Euro on this run, why it's "stupid"? The logic makes absolutely no sense.

you make good points... plz dont let him get to you. he has a history of thinking he is the only person in the world who knows anything.

[dtk]

Having Mets call specific models (mostly GFS and NAM) a piece of crap is going to drive me off this forum. It's something I expect from weenies that don't know any better, but from a Met? C'mon guys. If you don't like a particular run's solution, for whatever reason, just say that. Models are not pieces of crap, that sounds like a 12 year-old talking.

This! +1

[donsutherland1]

This thread should probably be closed and allowed to drop into oblivion. It's at a point of generating negative marginal returns.

[Chagrin Falls]

you make good points... plz dont let him get to you. he has a history of thinking he is the only person in the world who knows anything.

So "Mr. Administrator" attacks D.T. personally because D.T. criticized a product of the federal government? A product that clearly has medium-range verification problems when put up against the competition?

So much for AmericanWX. Same tender personalities and easily damaged egos as EasternUSWX.

[dissident]

winter weather advisory turned into 7 inches here in brookings, sd and two counties to the south which were supposed to get 2 or 3 inches... only one of the models on this site

Earl Barker's Regional Snowfall Model Page

came close for our area.. the GFS model forecast, second from the bottom.. most of them had the snow area too far north. Usually they over-predict.. they say we are gonna get 6 and we get 3.. under-predicting is relatively rare. Of course it doesn't take that long underneath yellow for it to pile up.Never been in orange level snowfall before, only orange that was sleet.. that would have been something to be under.. no sleet with any of this, all snow. The GEM one on the very bottom was the least reliable, the other ones somewhere in the middle... they mostly had the track too far north.

Fun fun, more on the way for thursday, but I'll be vacationing in WI by then.

[Bluescat1]

This thread should probably be closed and allowed to drop into oblivion. It's at a point of generating negative marginal returns.

Don you have always been the adult in the room.

[NJwinter23]

IMO, the recent poor model performance likely has to do with the difficult forecasting environment (synoptic situation in a quite strong La Niña event). Minor errors in details can lead to dramatic forecast errors. Last winter (strong blocking and moderate El Niño) presented a much easier forecasting environment. Overall,, I don't believe models have become worse, even as the current difficult forecasting environment leads to a bad performance. I suspect that earlier versions of the GFS and Euro would be faring even worse were they still running.

Great explanations from you and Wes. And I definitely get how the many moving parts in a situation such as this weekend's non-event is a prime cause for poor model performance. But I always thought that blocking patterns presented some severe model forecasting issues as well because of difficulty handling cutoffs and what not? I can also see how the slower flow as a result of blocking would increase performance in some instances as well though.

[A-L-E-X]

That may be a small part of it. But the nature of chaos is probably the driver that creates these "fits". Just look at the verification scores for the models...note that they have "ups" and "downs", and I suspect that the "downs" are during more chaotic type flow regimes.

Quick analogy: Take a bottle (that floats) and place it in a fairly slow, flat, steady stream and count to 20....note where it ends up at T+20.....do that 10 times and the "end points" will encompass a certain circle. Do the same thing but in a stream with rapids, eddies and curves, and I suspect that circle would be quite a bit larger.....

Point is, the atmosphere has varying states encompassing various degrees of chaos at any given time, and since the models remain the same, expecting performance to remain the same when added chaos is introduced is an expectation surely to disappoint.

Couple that with your noted point, and those errors are then magnified. Remember, initialization of ANY model is a "best approximation" state. It is why scrutiny of the T+0 panels is conducted at every run by the HPC at NCEP. It in effect, tries to warn forecasters when and why there may be larger error potential at T+xx.

Thanks for making this analogy, this is the clearest way I've ever seen it depicted. The circle of probability that you've described is probably the reason why ensemble forecasting is so valuable.

[LakeEffectKing]

Thanks for making this analogy, this is the clearest way I've ever seen it depicted. The circle of probability that you've described is probably the reason why ensemble forecasting is so valuable.

Yeah, and for the most part, my analogy is a 2 dimensional "circle" where as with the atmosphere we are dealing more with a 3D "sphere" (no pun intended....until I read it!! )

[smerby]

I haven't read the entire thread...but I'll throw out a theory if I may...is it possible that as we trend towards higher resolution models, that actually hurts us? Could it be that we're getting to the point where we're trying to model things down to a scale that we really have no business trying to model? Perhaps not because it can't ultimately done...but maybe because it can't be done at this point in time due to various reasons such as sparsity of input data, computing power, and an oversimplification of the processes that are actually occurring at the scales we're trying to predict? Yeah we've got these cool complex equations to try to predict atmospheric motion...but the smaller the scale and the more features you attempt to model...the more muddy and chaotic the solution becomes, right?

In other words...perhaps we really are only to the point where we can model systems with some success on a synoptic scale...but the more we try to model down to meso-scale or even microscale the more inaccurate we get because the current and future state of the atmosphere is that much more complex.

I have a similar same take on this. I have been an operational met since 84 and have seen many changes in the models, almost all of them good. Over the last several years though I have seen too many large changes and flip flops in the short range (12hrs to 2 days) when dealing precip events. These are the same issues we used to see and expect in the 3-5-7 day range back in the 80s and 90s. I agree with Mr. W that we may have bitten off more than we can chew trending toward higher res models without the computing power or proper initialization to catch up with them. The days of keeping continuity are gone as the models rarely keep a similar solution for a day or two. In my earlier years we analyzed upper air and surface maps by hand and tracked 500 MB short waves and 850 MB temps and could put out more accurate short range forecasts (Hi/Lo temp, precip start and end times) than the short range models of today. That may not be the best method from a business standpoint. Another problem I have noticed in the past several years is the increasing inconsistency in model bias. What used to work in the past does not always work now and what worked with last weeks storm may not work with this weeks. Could this come from the higher resolution trend? Our models are so sensitive now that the solutions are ever changing so it takes a different approach to forecasting. There is more frustration in forecasting now than before when it comes to being accurate so you have to be able to roll with it and have a thicker skin.

Smerby

www.accuweather.com

[Guest someguy]

Having Mets call specific models (mostly GFS and NAM) a piece of crap is going to drive me off this forum. It's something I expect from weenies that don't know any better, but from a Met? C'mon guys. If you don't like a particular run's solution, for whatever reason, just say that. Models are not pieces of crap, that sounds like a 12 year-old talking.

FYI, the GFS first sniffed this system out about 4 days ago, the same cycle of the Euro had nothing in its solution. I didn't hear anyone saying the Euro was crap at that point.

There is great model consensus with this system, and I doubt that any of them are absolutely correct at this point. And please tell me if the GFS trended toward the golden child Euro on this run, why it's "stupid"? The logic makes absolutely no sense.

You are right

I should NOT of callled it that ....

Pushed for time I spoke rashly

My bad

[greg ralls]

You are right

I should NOT of callled it that ....

Pushed for time I spoke rashly

My bad

I liked it, actually. I think people take the humor and sarcasm that you use the wrong way, and get all offended & mad without really paying attention to your points - which are usually excellent.

[A-L-E-X]

I liked it, actually. I think people take the humor and sarcasm that you use the wrong way, and get all offended & mad without really paying attention to your points - which are usually excellent.

I agree, I think he just speaks his mind. He doesnt have an anti-NWS bias, as I've seen him go after JB when JB makes one of his ridiculous anti-NWS rants.

Basically, DT is just being honest on which models perform the best. I'm sure the model performance hierarchy will change as we get more computational power + some funding.

[A-L-E-X]

I have a similar same take on this. I have been an operational met since 84 and have seen many changes in the models, almost all of them good. Over the last several years though I have seen too many large changes and flip flops in the short range (12hrs to 2 days) when dealing precip events. These are the same issues we used to see and expect in the 3-5-7 day range back in the 80s and 90s. I agree with Mr. W that we may have bitten off more than we can chew trending toward higher res models without the computing power or proper initialization to catch up with them. The days of keeping continuity are gone as the models rarely keep a similar solution for a day or two. In my earlier years we analyzed upper air and surface maps by hand and tracked 500 MB short waves and 850 MB temps and could put out more accurate short range forecasts (Hi/Lo temp, precip start and end times) than the short range models of today. That may not be the best method from a business standpoint. Another problem I have noticed in the past several years is the increasing inconsistency in model bias. What used to work in the past does not always work now and what worked with last weeks storm may not work with this weeks. Could this come from the higher resolution trend? Our models are so sensitive now that the solutions are ever changing so it takes a different approach to forecasting. There is more frustration in forecasting now than before when it comes to being accurate so you have to be able to roll with it and have a thicker skin.

Smerby

www.accuweather.com

The problem with working with ultra high resolution is that, at some point, "noise" overtakes the signal, and the system falls to the law of diminishing returns with a reduced signal to noise ratio. As others have noted, ensemble forecasting will mitigate this somewhat, as you can stack a whole range of possible scenarios and find patterns in the way they cluster towards certain solutions. I believe ensemble forecasting is more important than higher resolution.

[A-L-E-X]

Yeah, and for the most part, my analogy is a 2 dimensional "circle" where as with the atmosphere we are dealing more with a 3D "sphere" (no pun intended....until I read it!! )

Haha I see what you mean-- I was thinking of the circumference of the circle being the sample space of the range of possible solutions, and as you expand the circle outward you increase the range of possibilities in direct proportion to the circumference.

The holographic principle says that you can fully recreate the physics of any geometry in one dimension less than the geometry you're trying to represent...... therefore, since what we perceive of as our reality exists in 3 spatial dimensions plus time, you should be able to accurately represent it in 2 spatial dimensions plus time. The extra spatial dimension can be seen as a projection that emerges from the other ones.

[thegreatdr]

Much of the improvement made in short and medium range forecasting within the deterministic models like the GFS and ECMWF are due to better initialization. Whether that is due to better schemes used, better ways of dealing with errant initial data, or dependence on a model guess from 6 to 12 hours ago, I'm not sure.

What I am sure about is that even if the models agree in respect to the 500 hPa, 700 hPa, and surface patterns, QPF will be quite different between these pieces of guidance due to the inadequate understanding of the physics of rainfall and/or its modeling. Models such as former version of the GFS, NAM, and Canadian which allow too many convective/gridscale feedback bull's eyes blow up onto the synoptic scale do so at their own peril during the warm season. Those that don't do it quite often enough (like the ECMWF) will miss tropical cyclones at the surface more than the other guidance. I can think of a few occasions where this has happenned, where the H5 pattern of the ECMWF screamed development of a mid-level warm core cyclone, while the surface pressures had nothing. The fact that tropical cyclone rainfall analog techniques still have skill over the deterministic guidance should give all of us pause as to the rainfall forecasting accuracy in numerical weather forecasting.

We just had a QPF experiment this past summer with high resolution (close to 4 km) models. All of them were way to heavy in amounts. However, one or two did produce better axes (or placement). It will take years to get access to the best models of that grouping on a regular basis due to computational demands on the current supercomputer. Whenever we can finally figure out how to model rainfall/precipitation properly, the stage will be set for another set of significant model improvements, because right now, that's the Achilles heel of the model guidance. It's sad (though sometimes darkly funny) to realize that forecasters have a better chance of nailing the 168 hour pressure forecast than a rainfall forecast during the next 24-36 hours. However, it is true.

DR

I have a similar same take on this. I have been an operational met since 84 and have seen many changes in the models, almost all of them good. Over the last several years though I have seen too many large changes and flip flops in the short range (12hrs to 2 days) when dealing precip events. These are the same issues we used to see and expect in the 3-5-7 day range back in the 80s and 90s. I agree with Mr. W that we may have bitten off more than we can chew trending toward higher res models without the computing power or proper initialization to catch up with them. The days of keeping continuity are gone as the models rarely keep a similar solution for a day or two. In my earlier years we analyzed upper air and surface maps by hand and tracked 500 MB short waves and 850 MB temps and could put out more accurate short range forecasts (Hi/Lo temp, precip start and end times) than the short range models of today. That may not be the best method from a business standpoint. Another problem I have noticed in the past several years is the increasing inconsistency in model bias. What used to work in the past does not always work now and what worked with last weeks storm may not work with this weeks. Could this come from the higher resolution trend? Our models are so sensitive now that the solutions are ever changing so it takes a different approach to forecasting. There is more frustration in forecasting now than before when it comes to being accurate so you have to be able to roll with it and have a thicker skin.

Smerby

www.accuweather.com

[Ed Lizard]

Because I was a test subject and got in on PPV AccuWx early, I pay less than $100 a year, and now I see a whole host of Euro maps on 6 hour intervals and see MOS QPF for anyplace in 6 hour intervals.

A NYC sub-forum regular must pay more, AccuWx Euro data shows up about the same time as ECMWF site free maps show up. About an hour later than the play by play of what the short wave over the Southwest is starting to do starts.

My only complaint, as a taxpayer, I help pay for the GFS and NAM, and most of their data can be found for free on either an NCEP or university site. I can make forecast skew-Ts from them on the NIU site, and as a thunderstorm weenie (never chased, however) I like that. SPC site is quite amateur friendly, skew-Ts, SREF maps, RUC analysis. General outlooks and mesoscale discussions.

Someone in Europe can see GFS data for free, but the free Euro data is poor graphics resolution (best model resolution, lowest graphics resolution, interesting). Semi-free ride for European taxpayers.

That is why when I see mets discuss cutting off model data to us amateurs, I think, if it isn't national security related, and I'm paying for it, I want it.

Semi-related, I use HPC and local NWS office forecast discos the way some people use ensemble guidance, as a form of uncertainty detection. They know the inherent model error modes better than I do as an amateur, and if one has been reading the NWS BOX discos, for example, one would have known the weekend blizzard was not high confidence. BTW, the BOX forecaster who wrote the discussion knows amateurs read the discos, because besides the mention of low confidence and model spreads, he or she also cautioned against reliance on single operational model runs, which had to be meant for us amateurs.

Apparently another epic failure for the Euro on the East Coast snow storm, although there are a couple of 12Z GFS peturbations that show a sub 984 mb low near or inside the Benchmark, so maybe the 12Z run was a hiccup. I doubt it.

BTW, I'm glad to see NCEP forecasters, those who help improve the models, and those who interpret the models. The people that know the models the best. And until someone develops a supersize version of the FSU Super-ensemble, that takes many different operational models and accurately corrects for all known biases, and produces the perfect prog, that can account for the model not being initialized with data in thousands of vertical layers every square kilometer perfectly, the talk about NWS mets becoming now-casters and warning-casters, IMHO, seems premature.

[baroclinic_instability]

Another thread the needle phase event. It should be worth noting that even as the models began to "converge" a little yesterday at 0-12Z on a better solution, they were also continuing the weakening and slowing trend with the southern PV, enough so it showed us as forecasters that the window of opportunity/timeframe for the phase to occur was shrinking if a good outcome was to be expected since it was obvious there would be weak GOM cyclogenesis. The UK showed that when it had the PV way down into the GOM yet the solution was out-to-sea. NAM/GFS to a degree showed that a solution N of the GOM would also yield a solution that would hook too late given the orientation of the cold air advection to the gulf stream. It is the probability of an event discussed in the forecaster bias thread, and this is a good example model "convergence" or a developing consensus does not necessarily equate to a higher probability of an event.

The models were not bad with this system. Once again, any good forecasters knew based on the situation it was still a rather low probability of an even--likely 30% or less. A lot of folks were cautioning on this and saw the potential areas that could throw this system off course and OTS, and they were right to have a low probability as a result. The models did fine with this system. The forecasters who promised east coast destruction? Perhaps not.

[Yeoman]

So much for AmericanWX. Same tender personalities and easily damaged egos as EasternUSWX.

Well, at least one. I wouldn't say the same for the rest of the staff

[Ian]

Well, at least one. I wouldn't say the same for the rest of the staff

lolz

maybe that needs to be size 6 and in caps given the situation

[A-L-E-X]

Much of the improvement made in short and medium range forecasting within the deterministic models like the GFS and ECMWF are due to better initialization. Whether that is due to better schemes used, better ways of dealing with errant initial data, or dependence on a model guess from 6 to 12 hours ago, I'm not sure.

What I am sure about is that even if the models agree in respect to the 500 hPa, 700 hPa, and surface patterns, QPF will be quite different between these pieces of guidance due to the inadequate understanding of the physics of rainfall and/or its modeling. Models such as former version of the GFS, NAM, and Canadian which allow too many convective/gridscale feedback bull's eyes blow up onto the synoptic scale do so at their own peril during the warm season. Those that don't do it quite often enough (like the ECMWF) will miss tropical cyclones at the surface more than the other guidance. I can think of a few occasions where this has happenned, where the H5 pattern of the ECMWF screamed development of a mid-level warm core cyclone, while the surface pressures had nothing. The fact that tropical cyclone rainfall analog techniques still have skill over the deterministic guidance should give all of us pause as to the rainfall forecasting accuracy in numerical weather forecasting.

We just had a QPF experiment this past summer with high resolution (close to 4 km) models. All of them were way to heavy in amounts. However, one or two did produce better axes (or placement). It will take years to get access to the best models of that grouping on a regular basis due to computational demands on the current supercomputer. Whenever we can finally figure out how to model rainfall/precipitation properly, the stage will be set for another set of significant model improvements, because right now, that's the Achilles heel of the model guidance. It's sad (though sometimes darkly funny) to realize that forecasters have a better chance of nailing the 168 hour pressure forecast than a rainfall forecast during the next 24-36 hours. However, it is true.

DR

DR, it almost sounds like the models' "dynamic range" are a bit too narrow to match that of the atmosphere/ocean couplet, which is why when you tweak a model to fix one thing (like TC blowups) it develops a bias in something else (a cold bias, for example.) This is probably why it's best to look at a range of models and develop some sort of consensus-- no one model, on its own, has the dynamic range to fully model the extremes of weather.

[A-L-E-X]

Another thread the needle phase event. It should be worth noting that even as the models began to "converge" a little yesterday at 0-12Z on a better solution, they were also continuing the weakening and slowing trend with the southern PV, enough so it showed us as forecasters that the window of opportunity/timeframe for the phase to occur was shrinking if a good outcome was to be expected since it was obvious there would be weak GOM cyclogenesis. The UK showed that when it had the PV way down into the GOM yet the solution was out-to-sea. NAM/GFS to a degree showed that a solution N of the GOM would also yield a solution that would hook too late given the orientation of the cold air advection to the gulf stream. It is the probability of an event discussed in the forecaster bias thread, and this is a good example model "convergence" or a developing consensus does not necessarily equate to a higher probability of an event.

The models were not bad with this system. Once again, any good forecasters knew based on the situation it was still a rather low probability of an even--likely 30% or less. A lot of folks were cautioning on this and saw the potential areas that could throw this system off course and OTS, and they were right to have a low probability as a result. The models did fine with this system. The forecasters who promised east coast destruction? Perhaps not.

Yes, even the euro's own ensembles raised some huge red flags.

[Yeoman]

lolz

maybe that needs to be size 6 and in caps given the situation

I'd love the enthusiasm assuming it wasn't some annoying weenie.. there are bigger fish to fry