Why are models so bad?

[Guest someguy]

I haven't read the entire thread...but I'll throw out a theory if I may...is it possible that as we trend towards higher resolution models, that actually hurts us? Could it be that we're getting to the point where we're trying to model things down to a scale that we really have no business trying to model? Perhaps not because it can't ultimately done...but maybe because it can't be done at this point in time due to various reasons such as sparsity of input data, computing power, and an oversimplification of the processes that are actually occurring at the scales we're trying to predict? Yeah we've got these cool complex equations to try to predict atmospheric motion...but the smaller the scale and the more features you attempt to model...the more muddy and chaotic the solution becomes, right?

In other words...perhaps we really are only to the point where we can model systems with some success on a synoptic scale...but the more we try to model down to meso-scale or even microscale the more inaccurate we get because the current and future state of the atmosphere is that much more complex.

DT strong agrees with this.

the real issue IMO is that as the models have gotten better .... we have become more dependent on them but the synoptic skills of younger mets have turned to crap

The younger Mets students are great on the Meso scale stuff but synopotic patterns... climatology of Midwest winter storms or KU ... or east coast Hurriucane patterns

forget about it.

IMO the real issue is the decline in synoptic scale forecasting

[Yeoman]

The weather is unpredictable. How a supposed "met" could ask this question boggles the mind and further confirms my theory that he is anything but one.

[baroclinic_instability]

DT strong agrees with this.

the real issue IMO is that as the models have gotten better .... we have become more dependent on them but the synoptic skills of younger mets have turned to crap

The younger Mets students are great on the Meso scale stuff but synopotic patterns... climatology of Midwest winter storms or KU ... or east coast Hurriucane patterns

forget about it.

IMO the real issue is the decline in synoptic scale forecasting

I am young, but I think I am ok at least

No I totally agree. I have found a lot of mets are unable to fully analyze and break down weather patterns. Combined with a general lack of knowledge regarding the models themselves, I have found a lot of mets struggle in general, especially those coming out of college. I was just lucky to have some seriously smart profs who helped me along.

I know at my University (Univ. of North Dakota) they are trying to work on program changes to deal with this very issue.

[baroclinic_instability]

Yeah...I'm kinda just throwing ideas around...but I guess what I'm getting at is that no, LFM and NGM would lose in almost all scenarios to the GFS and NAM...but perhaps they would offer more stability in more complex situations. Maybe they are still wrong in the end...but maybe they trend more smoothly to the correct solution as opposed to flip-flopping several times back and forth.

Yeah I see this point. In response, I do know the ensembles, like the GFS ensemble, for instance, besides having perturbed initial conditions and slight tweaks in parameterizations/physics, were still showing potential bombs through this forecast period as well. The ensemble members run at lower resolutions than the operational run, and some of the ensemble members had the biggest bomb solutions.

[baroclinic_instability]

I am amazed that a computer, electricity and metal, can tell me on Monday that there will be a storm in the MA on Thursday. And then it happens. A storm that hasn't formed, doesn't exist. That's pretty amazing. Now it doesn't always get the details right, but it's impressive nonetheless.

I think in all the discussion on this topic, this may be the most insightful.

[SnowGoose69]

I see where you are trying to go, but no, I disagree. There is good reason we don't use models such as the LFM and NGM anymore

Seriously though, the latest GFS update is a good example that shows an increased resolution doesn't result in worse forecasts per se. Also, some synoptic scale systems are highly sensitive to sub-synoptic scale forcings, so if we went back to real low res models, we would simply be unable to even model those solutions. I think that would be a step back.

Model updates do not always mean better...the Euro update a few years back clearly made it worse until they did another upgrade which seemed to right it again.

[JoMo]

High resolution models are used with decent success in complex terrain like the intermountain west and Pacific ranges. If initialized properly, they can be quite useful within 48 hours. Of course, they have signficant issues and their errors tend to compound rapidly with time because less agressive filters need to be used to preserve the high resolution details. As we know with chaos, those details can also become an issue since we know some of it is noise. I think this is why we won't be seeing any 4 km global models anytime soon.

High res models have a place though, and they can be used quite effectively. U of Washington has a nice implementation out W. They get some insane complex weather with the straight of juan de fuca, the olympic mtns, the crazy convergence zones, and the carved out mtn ranges that can create forecasting headaches in general.

http://www.atmos.washington.edu/mm5rt/

Not really a reply to this post but you remember how horrible the models were in the lead up to the system on your profile picture? Nearly every model (GFS, UKMET, NOGAPS, etc) had the storm developing way far to the S and SE of where it actually did. The only model that had it right was the Euro and even it had one run where it agreed with the rest of the models.

[KEITH L.I]

one good thing about better models..say 24-48 hours..you don't have heavy snow warnings that end up as partly cloudy the next morning anymore..now that's a letdown!..happened many times in the 60's and early 70's

[baroclinic_instability]

Not really a reply to this post but you remember how horrible the models were in the lead up to the system on your profile picture? Nearly every model (GFS, UKMET, NOGAPS, etc) had the storm developing way far to the S and SE of where it actually did. The only model that had it right was the Euro and even it had one run where it agreed with the rest of the models.

See my earlier comments

I do very well remember and there will always be potential storm busts in a highly chaotic environment where the growth of a disturbance can become almost exponential.

[SnowGoose69]

one good thing about better models..say 24-48 hours..you don't have heavy snow warnings that end up as partly cloudy the next morning anymore..now that's a letdown!..happened many times in the 60's and early 70's

It happened many times in the 80s...its amazing that with the LFM going away and the ETA replacing it as well as the AVN probably getting more usage than the NGM how those busts dropped off markedly in the early 90s as I posted in the beginning of this thread...the LFM basically had one great score, the Thanksgiving event in 1989 which it had well in advance....the busts now more come due to warm air advection errors where snow is forecast and you get more sleet or freezing rain, but yeah, snow forecasts turning into partly cloudy are rare, of course you have the December 2000 incident but they're pretty infrequent now.

[Rainshadow]

It happened many times in the 80s...its amazing that with the LFM going away and the ETA replacing it as well as the AVN probably getting more usage than the NGM how those busts dropped off markedly in the early 90s as I posted in the beginning of this thread...the LFM basically had one great score, the Thanksgiving event in 1989 which it had well in advance....the busts now more come due to warm air advection errors where snow is forecast and you get more sleet or freezing rain, but yeah, snow forecasts turning into partly cloudy are rare, of course you have the December 2000 incident but they're pretty infrequent now.

From a timing perspective for some of the yewth, the lfm model run ended at 48 hours. I remember even back then, the avn was too suppressed.

[uncle W]

one good thing about better models..say 24-48 hours..you don't have heavy snow warnings that end up as partly cloudy the next morning anymore..now that's a letdown!..happened many times in the 60's and early 70's

yea like January 1970...There were a few good busts back then also...6" of snow flurries in Feb. 74...

[Rainshadow]

yea like January 1970...There were a few good busts back then also...6" of snow flurries in Feb. 74...

January 1978 was no picnic either, the first one was suppose to be a massive snowstorm, the second change to rain quickly.

[LakeEffectKing]

I haven't seen mentioned (I don't think) that we also must consider our limitations as to the sampling of the atmosphere at any given time to input into the models. We sample, what....like .0000xxxxxx% of the atmosphere at any given 00z/12z dump time, then three dimensionally extrapolate inbetween data points, then approximate the equations that govern the thermal dynamics and dynamics as we integrate over time. There certainly are going to be times, patterns, regimes that outputs get a bit out of wack.

The rennaissance period for modeling is over....(as stated before). We are nearing the point on the "improvement curve" where the diminishment of returns is too great to invest as we have in the past (Until some "new/unknown" breakthrough) comes along.....ie, secret time machine.

[uncle W]

January 1978 was no picnic either, the first one was suppose to be a massive snowstorm, the second change to rain quickly.

that was a wild month...Six major storms in a months time...

[Rainshadow]

that was a wild month...Six major storms in a months time...

Yup senior year of high school for me, my first winter driving a car. The old garbage pail wars started early for me.

Still nothing beat the Lindsey storm as far as forecasts that went awry.

[A-L-E-X]

I think in all the discussion on this topic, this may be the most insightful.

I know this thread didnt start off in the best way, but perhaps some good can come of it, as there are probably many people out there who have the same views as the thread starter but dont voice them-- hopefully they learned from this discussion.

[Rainshadow]

I haven't seen mentioned (I don't think) that we also must consider our limitations as to the sampling of the atmosphere at any given time to input into the models. We sample, what....like .0000xxxxxx% of the atmosphere at any given 00z/12z dump time, then three dimensionally extrapolate inbetween data points, then approximate the equations that govern the thermal dynamics and dynamics as we integrate over time. There certainly are going to be times, patterns, regimes that outputs get a bit out of wack.

The rennaissance period for modeling is over....(as stated before). We are nearing the point on the "improvement curve" where the diminishment of returns is too great to invest as we have in the past (Until some "new/unknown" breakthrough) comes along.....ie, secret time machine.

Yeah, you can tell by the decay curves that the improvements have slowed.

[Rainshadow]

I know this thread didnt start off in the best way, but perhaps some good can come of it, as there are probably many people out there who have the same views as the thread starter but dont voice them-- hopefully they learned from this discussion.

Maybe a better title would have been why are the models having such a difficult time with this pattern?

[A-L-E-X]

THE BLACK SWAN by Taleb

\ and

FOOLED BY RANDOMNESS also by taleb

if you are into day 3 to day 30 forecasting and you have NOT read those books ...twice... your are screwed

One of Taleb's inspirations, Dr. Mandlebrot (he of fractal fame) passed away this year-- what a loss. Taleb said of Mandlebrot "He is a Greek among Romans" (an allusion to the fact that Greeks were innovators and the Romans were imitators.)

It's amazing how much of nature behaves fractally-- from stock markets, to super conductors to the inside of black holes.

[A-L-E-X]

Maybe a better title would have been why are the models having such a difficult time with this pattern?

Yeah I think frustration overtook his common sense when he did that lol.

[Harry]

It happened many times in the 80s...its amazing that with the LFM going away and the ETA replacing it as well as the AVN probably getting more usage than the NGM how those busts dropped off markedly in the early 90s as I posted in the beginning of this thread...the LFM basically had one great score, the Thanksgiving event in 1989 which it had well in advance....the busts now more come due to warm air advection errors where snow is forecast and you get more sleet or freezing rain, but yeah, snow forecasts turning into partly cloudy are rare, of course you have the December 2000 incident but they're pretty infrequent now.

I believe the MSP system last week qualifies for such a event or very darn close. NWS GRR had warnings out for 6+ for the whole area and a number of spots just to my north here did not even get a inch. You can see where the 6+ line was.

Ofcourse a part of the problem with this was the warmer air was slower to leave and the deform snows never took off. They ended up in the se part of the state which had advisories from Detroit to the Ohio line and warnings north of there.

But yeah it still does happen and more often then a few might think. Have seen it go the other way as well.

Found the AFD..

.SHORT TERM...(407 AM EST SAT DEC 11 2010)

(TODAY THROUGH MONDAY)

FORECAST CONCERNS DEAL WITH THE APPROACHING STORM THAT CURRENTLY IS

NEAR THE NE/IA BORDER. LATEST MODEL GUIDANCE SHOWS THE LOW TRACK

TRENDING SOUTH ALONG THE IN/MI BORDER. THE OPERATIONAL GFS AND ECMWF

ARE NOW IN MUCH BETTER AGREEMENT WITH THEIR ENSEMBLE MEANS AND HAVE

CONVERGED ON A TRACK ALONG THE BORDER. HOWEVER THERE REMAINS ROOM

FOR MORE MODIFICATION TODAY. IF IT KEEPS TRENDING SOUTH...IT MAY

MEAN LESS SNOW OVER OUR NRN CWA. AS SUCH WE KEPT THE WATCH INTACT

AND ADDED THE REST OF THE COUNTIES TO IT. THE NAM CONTINUES TO BE

THE FARTHEST NORTH...ALTHOUGH THE 06Z RUN IS TRENDING SOUTH...AND SO

HAS THE LEAST AMOUNT OF SNOW IN THE SOUTH.

THE PCPN WILL BEGIN THIS AFTERNOON AS SOME RAIN BUT TURN TO SNOW BY

EVENING. MODERATE TO STRONG FGEN FORCING IS PROGD FROM NEAR MKG TO

MOP FROM 09Z TO 18Z SUNDAY. THERE SHOULD BE QUITE A BIT OF

DEFORMATION SNOW DEVELOP TONIGHT AND SUNDAY AS THE TROWAL MOVES

THROUGH. IT/S ENTIRELY POSSIBLE...BUT NOT CERTAIN...THAT WE/LL SEE A

FOOT OF SNOW OVER THE NRN 1/2 OF THE CWA FROM THIS STORM...AND

THAT/S BEFORE THE LAKE EFFECT SNOW DEVELOPS.

STRONG NORTH WINDS WILL DEVELOP LATE TONIGHT AS THE LOW PASSES TO

THE SOUTH. THIS COULD ACTUALLY BE A MITIGATING EFFECT TO LAKE SNOW

AS H8 WINDS IN THE 50-60KT RANGE WILL PROHIBIT BANDS FROM

ESTABLISHING. EVEN SO...STRONG OMEGA IN A FAIRLY WIDE DGZ WILL

CREATE QUITE A BIT OF SNOW. RH FIELDS SHOW DEEP MOISTURE WITH THIS

SYSTEM. WINDS GUSTING TO 50 MPH ALONG THE LAKESHORE WILL CREATE NEAR

BLIZZARD CONDITIONS LATE SUNDAY NIGHT AND MONDAY.

OVERALL...SNOW TOTALS ALONG THE LAKE SHORE COULD TOTAL A FOOT OR TWO

BY LATE MONDAY AND 6 TO 9 INCHES SOUTH OF I-96/69 AND EAST OF US-131

INCLUDING GRAND RAPIDS AND LANSING . ACROSS THE NRN CWA AROUND A

FOOT IS POSSIBLE.

WIND CHILLS WILL FALL BELOW 0 LATE SUNDAY NIGHT AND MONDAY.

&&

[OKpowdah]

THE BLACK SWAN by Taleb

\ and

FOOLED BY RANDOMNESS also by taleb

if you are into day 3 to day 30 forecasting and you have NOT read those books ...twice... your are screwed

A New Kind of Science by Wolfram

[Mencken_Fan]

IMO, the recent poor model performance likely has to do with the difficult forecasting environment (synoptic situation in a quite strong La Niña event). Minor errors in details can lead to dramatic forecast errors. Last winter (strong blocking and moderate El Niño) presented a much easier forecasting environment. Overall,, I don't believe models have become worse, even as the current difficult forecasting environment leads to a bad performance. I suspect that earlier versions of the GFS and Euro would be faring even worse were they still running.

You're on the mark. For people to suggest the models have gotten worse is folly; the models haven't changed, the environment has.

For any event (climate change, weather, even plane and car accidents) a threshold of likelihood must first be crossed. Triggers lie in wait all around us; and weather models are much like sensors. With some atmospheric environments we roar past the threshold, all triggers are fired, and the model alarms go off. With others we hover at the threshold; periodically stepping over it with meek, momemtumless steps.

Looking back, the shortwave was crap from day 1; there never was a solid Low in or near the GOM; etc. At times, the models sounded the alarm without a full complement of triggers going off. Many forecasters noticed this and warned of a "low confidence forecast." To them, we say "job well done."

[Mencken_Fan]

This is one thing I don't fully understand regarding the human drive towards perfection...variability is what makes everything so much better. A day we have perfect models will be a day where life just became suddenly more boring. This applies, as you said, to almost everything in science. We don't need to be perfect or have perfect solutions to everything.

Bravo! What you're describing, I call "the secret of life."

[Chagrin Falls]

Hey Greg (Analog96),

I don't agree that the models have been doing badly. MODELS ARE FANTASTIC! Any measure has shown they have improved significantly. Forecasts continue to become more accurate further into the future. MOS technology has greatly improved, computing power allowing more model runs and ensembles. Often the models make us meteorologists quite humble. Don't believe it? Try using the LFM and 60 hr spectral model to make a 5 day forecast and let me know how that works for you. If the models improve more more meteorologists may be out of jobs (I'm thinking this is inevitable as time moves on, but anyway...).

I think it is impossible to determine why the models went bad. They make billions of calculations. Unless retroactively the models are run again with different data and different outcomes we will never know for sure. Every excuse is just speculation. For goodness sakes, perhaps there was some convective feeback in Estonia. Who knows?

I think a better question (and more relevant one) to ask is, why has the performance of meteorologists been so poor? A few days ago I posted something that was quite critical of the HPC, well, needless to say I could have been more careful with my language and for that I apologize and hope offended board members can forgive me. Ego sometimes gets in the way in the field of meteorology (including mine) -- sadly, this prevents learning from mistakes and prevents forecasting progress.

We could say the models behaved badly, but honestly, "I'm sorry officer but the model made me do it"? No excuses...without getting into which office said what & when, look at the discussions from the last few days. Literally, every excuse in the book was made to make a big East Coast storm. I saw one that talked about a "aclimatological solution is needed" and one that discussed "baroclinic instability" near the Gulfstream. Well, this time of year there is often baroclinic instability near the Gulfstream, but obviously most short-waves do not bomb and go up the coast. What made this one different? I'm not even sure what an aclimatological solution is.

Meteorologists rely on models too much, they hug them, they have preconceived notions of what they want to happen and they selectively analyze data to provide evidence for their reasoning. This is human behavior, we all do it (including me), but it seems to happen more and more these days. Perhaps because at least one model run or one ensemble member will show what the meteorologist WANTS to occur. Maybe there is just too much information out there?

When the models are flip flopping, the ensembles show wide standard deviations & the flow is very fast I think meteorologists should be able to look at that and determine what is most likely to occur. Honestly, I don't see that happening very much. Why follow the GFS when it has a poor medium range track record? Why follow the deterministic/operational model runs and not the ensembles? Why follow the ECMWF if it suddenly flip flops? Why follow the ECMWF deterministic but not the ensembles?

I see a lot of bad meteorology going on & I don't think it is fair to blame the models because they can't speak for themselves and also because we as meteorologists are almost totally dependent upon them.

I often bomb forecasts, I bombed today in Ohio because it snowed more than I thought, especially away from the lake. It happens, I try to learn from it and move on. Sometimes I can't determine why my forecasts suck so bad. I think it is very difficult for many meteorologists to realize their own mistakes and learn from them, instead a lot of defensiveness occurs and blame gets redirected. Not good for the profession.

Anyway, I hope I didn't offend anyone with this post. Occasionally, every meteorologist gets something wrong. It is embarrassing, but it will happen every once in a while. I do see a lot of meteorologists defending thoughts that are not reasonable and I think this is a big problem.

Models will continue to improve and do their magic, but humans, will they improve as well? That is the question that is difficult to answer.

[CT Rain]

I agree that the models are better with TC forecasting, but I don't think they're any better at all with mid-latitude cyclone forecasting.

Disgaree 100%

You can't honestly think that, can you?

Not only were we dealing with a crazy fast flow with any number of shortwaves flying through we also had an exceptionally anomalous pattern with a massive block and a retrograding PV across southern Canada. It was a strange setup and it was very clear that we were going to have some trouble figuring this one out on the models.

[Vandy]

I haven't seen mentioned (I don't think) that we also must consider our limitations as to the sampling of the atmosphere at any given time to input into the models. We sample, what....like .0000xxxxxx% of the atmosphere at any given 00z/12z dump time, then three dimensionally extrapolate inbetween data points, then approximate the equations that govern the thermal dynamics and dynamics as we integrate over time. There certainly are going to be times, patterns, regimes that outputs get a bit out of wack.

The rennaissance period for modeling is over....(as stated before). We are nearing the point on the "improvement curve" where the diminishment of returns is too great to invest as we have in the past (Until some "new/unknown" breakthrough) comes along.....ie, secret time machine.

Great post. And I think this ties into what others have said about increased resolution in models not helping. If you are computing things on a 1km grid, but at best sampling at every 10km, you are going to get errors. The model will want to generate all kinds of mesoscale phenomena at those 1km grid points that you can't ever hope to successfully feed information to with your extrapolated 10m observation grid. I mean, the things that are done with data assimilation these days is amazing, but still can't compensate for gaps in data coverage. And that's just on land. Think about the number of radiosondes going up/down over the oceans....

[uncle W]

Yup senior year of high school for me, my first winter driving a car. The old garbage pail wars started early for me.

Still nothing beat the Lindsey storm as far as forecasts that went awry.

I was 20 when the Lindsay storm hit...The day before I noticed hundred's of high flying birds from the south...I thought it was odd and never saw that before or since...That evening I went to a concert at the Filmore East in lower Manhattan...It was about 10pm and we were standing on line and the winds were coming from the water which was from the southeast...All day the temperatures were in the high 30's under increasing cloudiness...It was about 36 degrees that evening as we went in to see the show...Canned Heat was one of the performers...A great show and at 3am when we left it was snowing...It was a wet snow and wasn't sticking...I drove home and got to my house around 4am...I went to sleep wishing it would be a big storm...I was a big weenie even then...I woke up at 8am looked out the window and got up and dressed quickly...I drove around 11am while the roads were still passable...The heavy wet snow on tree limbs made them droop so low you couldn't walk down the block with out stooping under them...I got stuck driving around a week later...After the storm a series of storms under preformed especially the March 3rd 69 event...

[usedtobe]

It does seem to me that modeling in general seems to do better in el nino winters than la nina winters for the local area we forecast. Maybe statistically, globally its not that different. We looked up the gfs runs for the first three big winter storms last season and it had double digit snows forecast for the PHI CWA 60hrs in advance. Some of the more difficult events, Feb 89 2 feet at the shore, Jan 2000, March 2001 hyperstorm have happened coincidentally or not in la nina winters.

This is where I think the ensembling has helped considerably in both the deterministic and confidence spectrum of forecasting. The last two ECMWF operational runs were at opposite ends of the ensemble spectrum, one way west, the other way east. Either way it should have raised a flag. We all know that the ECMWF likes to hold back energy in the desert southwest too long and you could see how yesterday's 12z run could happen, it holds the short wave back, if its being held back it must be stronger and chances slower. This slower, stronger evolution made it possible for the northern stream short wave to catch the southern stream short wave and phase and there we went.

Looking at the decay curves off of the EMC site the models have gained about a day to a day and a half of forecast skill at 500mb since 2001, so there is slow but steady improvement that does go on. I believe the models are better this winter than if we would have used the models as they were from the 2007-8 la nina winter, we're too close to realize it.

As usual I totally agree with Don Sutherland's posts, he even says it better than I would.

Tony, good post. I agree. Also Don usually is more articulate than we are.