Dissecting The Bust

[wxman]

And as you mentioned with Irene and sandy when the next big blizzard is forecast are people going to take it seriously?

There is no real choice here.  Sandy stood as good a chance of busting (maybe more) than what just happened.  Should forecasters have downplayed the event?  The only thing that should have been better is honest mentioning of the uncertainty.  I didn't see much of that. Perhaps going forward extreme forecasts should be accompanied by a confidence level. 

[mranger48]

I thought this was an interesting article. What do u all think about what this guy is saying? Is he right?

http://www.weathernj.com/how-every-meteorologist-including-myself-got-it-wrong/

Sent from my iPhone using Tapatalk

[Derecho!]

Replace the nws staff with an automated, gfs, ukmet, cmc, euro average. Eliminates

weenie biases. Also to eliminate stupidity, throw the Nam out of a skyscraper.

 

I realize there is some tongue-in-cheek to the above, but there's a lot to this.

 

The problem isn't too MUCH "modelology." It's too LITTLE.

 

Note I'm explicitly leaving aside two issues which were a huge influence:

 

1) IMBY weenism and wishcasting - note that the exciting attraction of forecasting a historic storm for NYC creates the equivalent of IMBY for mets not geographically in NYC. And being a met simply means that your justifications for your weenieism sound better.

 

2) If your forecasts actually affect decisionmaking, you will be criticized far more forecasting 8" and getting 24" than the reverse.

 

Again, I realize tropical is different, but NHC forecasters, even with PhDs and decades of experience and the ability to view a vast panopoly of satellte products and observations, openly admit they can't beat verbatim model consensus tracks.

 

If you put a verbatim tropical consensus track model up against NHC forecasters that had the restriction they had to chose a single verbatim model track each forecast cycle, the consensus model would almost certainly win.

 

Mutual Fund managers with multiple advanced degrees and decades of experience can't beat simple Index Funds comprised of entire stock markets.

 

For each storm, one model will have done BETTER than the "consensus" but you will never know which model it will be beforehand. An automated model consensus is boring, and people have a romantic infatuation with "intuition and experience" allowing a given met to diagnose which models to "chose" for each forecast cycle (as you see HPC doing). I submit that a straight consensus will end up doing better.

 

 

So here's what is going to happen with this storm:

 

1) Of the dozens of people that offered Central Park snowfall total forecasts, the one that happened to be closest to correct will receive "kudos" and will be seen as the "guru" for the next storm (where they will probably fail miserably because their forecast was mostly dumb luck).

 

2) The "Best" model for NYC for this storm (Whichever it was) will be proclaimed the "king" and the rest will be deemed "garbage" (either for "this winter" or for "this pattern.") This model, of course, will likely fail miserably at the next chance for significant snowfall this winter.

 

3) Repeat Steps 1 & 2. Over and over again.

[nflwxman]

I didn't want to post this in the midst of the storm yesterday since I didn't want to seem like I'm trolling, but I agree with many of the posts above.  Clipper to Miller B storms are rarely top 3 events from NJ (I'm from New Brunswick).  Truthfully, the Euro (and a few runs of the NAM) were the only models bullish on big totals for KNYC.  I looked through the archives of this storm at the hi-res GFS, RGEM, GGEM, NAMOf the 28 runs of those 4 models over the 48 hours leading up to the storm- The following snowfall predictions for KNYC were taken from the models (assuming a 12 to 1 ratio)- I made sure each model encompassed the entire storm.

 

5 out of 28- Showed  4-8 inches

 

12 out of 28- Showed 8-12 inches 

 

5 out of 28- Showed 12-16 inches

 

4 out of 28- Showed 16-20 inches

 

2 out of 28-Showed 20+

 

 

We all understand that mesoscale banding often sets up NW of modeled, but I just don't believe that was a compelling enough story to ignore these selective model runs. However, What was particularly complicating about this was that the hi-res GFS and Euro were both equally strong in towing their own lines.  The Euro had a 5 inch standard deviation in it's last 4 runs for KNYC and the GFS had a 4 inch standard deviation in the same runs.

 

Perhaps this should give some credence to the SREFs and other Ensemble approaches next time.

[snywx]

Yes but then it evolved into 24 to 36 was unreasonable .

 

The bust was simple , it jumped east at 6pm to the best convection . Happens . Was not really seen that well and most models

printed out 1.5- 1.75 and 2 on the western side and that's where we were at 9am yesterday

 

Happens , we just missed this one .

 

Now what was the reason for the ridiculous forecast of 24-36" deep into Orange county when it was clear the Euro was the ONLY model even remotely close to that. Sure the NAM was fairly similar but every other single piece of guidance was probably 30-40% of what the Euro had. Its obvious they had "euro vision" ( including us) and ran with it. 

[PB GFI]

Now what was the reason for the ridiculous forecast of 24-36" deep into Orange county when it was clear the Euro was the ONLY model even remotely close to that. Sure the NAM was fairly similar but every other single piece of guidance was probably 30-40% of what the Euro had. Its obvious they had "euro vision" ( including us) and ran with it. 

Euro 10 straight runs

NAM  24 hours out

[The_Global_Warmer]

The gfs was crapped on all weekend.

The differences in model verification at 48, 36, and 24 hours has to be minuscule.

The confirmation bias with this got way out of control

[packfan98]

1) The nam has a tendency to over amp storms which also results in a further west track.

2) The Euro has the same tendency to a lesser extent.  Remember the "Santa Bomb" storm that never happened.

 

I agree that a GFS/EURO blend is reasonable most times.  The problem with this one is that NYC with millions of people were on the dividing line between the the westward edges of the heavy precip on the major models.  

 

Bottom Line:  I have not lost confidence in any of the mets or NWS.  The forecasts were reasonable with the given guidance.  (Now the hype machine was in full effect and I don't care for all of the hoopla, but that's what we have to deal with on any topic that gets steamrolled into hysteria and foolishness.)

[nzucker]

Making an initial forecast of 24-36" for the entire NYC metro, from central NJ to eastern CT, was pretty irresponsible.

 

How many storms have dumped such large amounts of snow over such a widespread area? Even the Blizzard of 96 had a sharp cut-off to the northwest, and most of the amounts were in the 20-25" range, not 30-36". PDII was around 16-20" for the large majority of NYC metro, with some places like JFK receiving up to 2'. March 1960 dumped 28" here in Westchester but significantly less in the City. History tells us that it's not easy to get 2-3' to verify across such a large forecast zone, so why call for such huge amounts? A forecast of 12-18" would have conveyed the seriousness of the storm without being over the top...

 

Also, the trend yesterday was to cut back on amounts. Anytime you see that trend you have to realize amounts are probably going to be lower than the current forecast as everything is gradually cutting back. Forecasts went from 24-36" to 18-24" to 12-18" to 9-16" here in Westchester...that obviously shows that the system is not performing as expecting. The 100-150 mile jog east at the last minute cost us the larger amounts, and that trend was apparent on radar and models last night. Some chose to ignore it at their own folly.

[Stormlover74]

I realize there is some tongue-in-cheek to the above, but there's a lot to this.

The problem isn't too MUCH "modelology." It's too LITTLE.

Note I'm explicitly leaving aside two issues which were a huge influence:

1) IMBY weenism and wishcasting - note that the exciting attraction of forecasting a historic storm for NYC creates the equivalent of IMBY for mets not geographically in NYC. And being a met simply means that your justifications for your weenieism sound better.

2) If your forecasts actually affect decisionmaking, you will be criticized far more forecasting 8" and getting 24" than the reverse.

Again, I realize tropical is different, but NHC forecasters, even with PhDs and decades of experience and the ability to view a vast panopoly of satellte products and observations, openly admit they can't beat verbatim model consensus tracks.

If you put a verbatim tropical consensus track model up against NHC forecasters that had the restriction they had to chose a single verbatim model track each forecast cycle, the consensus model would almost certainly win.

Mutual Fund managers with multiple advanced degrees and decades of experience can't beat simple Index Funds comprised of entire stock markets.

For each storm, one model will have done BETTER than the "consensus" but you will never know which model it will be beforehand. An automated model consensus is boring, and people have a romantic infatuation with "intuition and experience" allowing a given met to diagnose which models to "chose" for each forecast cycle (as you see HPC doing). I submit that a straight consensus will end up doing better.

So here's what is going to happen with this storm:

1) Of the dozens of people that offered Central Park snowfall total forecasts, the one that happened to be closest to correct will receive "kudos" and will be seen as the "guru" for the next storm (where they will probably fail miserably because their forecast was mostly dumb luck).

2) The "Best" model for NYC for this storm (Whichever it was) will be proclaimed the "king" and the rest will be deemed "garbage" (either for "this winter" or for "this pattern.") This model, of course, will likely fail miserably at the next chance for significant snowfall this winter.

3) Repeat Steps 1 & 2. Over and over again.

Yep the freakin weather channel and the gfs..lord help us

[snywx]

Euro 10 straight runs

NAM  24 hours out

 

5 days out? 

[wxman]

The gfs was crapped on all weekend.

The differences in model verification at 48, 36, and 24 hours has to be minuscule.

The confirmation bias with this got way out of control

As mentioned above, the difference in the GFS v Euro (sensible wx) was mainly the NJ TPK.  A small area.  I do not view this as a "blend the models" situation.  The euro had the capture and close off marginally different from the GFS (and Canadian) and, in my view, was more likely to get those features right.  Again, but for highlighting the uncertainty more, not sure this could have been handled better

[PB GFI]

The gfs was crapped on all weekend.

The differences in model verification at 48, 36, and 24 hours has to be minuscule.

The confirmation bias with this got way out of control

Fair , but it missed the 2 inches of liquid here last Sunday and only caught it at 48 hours out  while  the Euro had it for 5 days

So that had to be fresh in their minds

 

Plus the GFS has a Se bias on the EC and has for many years , so that had to be a factor .

Even more than skill score the Euro showed such tremendous continuity showing the same solution for 10 runs it was hard not to be at the top of the list .

[pazzo83]

Fair , but it missed the 2 inches of liquid here last Sunday and only caught it at 48 hours out  while  the Euro had it for 5 days

So that had to be fresh in their minds

 

Plus the GFS has a Se bias on the EC and has for many years , so that had to be a factor .

Even more than skill score the Euro showed such tremendous continuity showing the same solution for 10 runs it was hard not to be at the top of the list .

 

Maybe that bias has been dealt with in this new version of the GFS.

[The_Global_Warmer]

As mentioned above, the difference in the GFS v Euro (sensible wx) was mainly the NJ TPK. A small area. I do not view this as a "blend the models" situation. The euro had the capture and close off marginally different from the GFS (and Canadian) and, in my view, was more likely to get those features right. Again, but for highlighting the uncertainty more, not sure this could have been handled better

That's exactly right.

There is nothing wrong with explaining this storm could range from 6-24" and here's why.

[PB GFI]

5 days out? 

I cant post the map, they will not let me  find Jan 20 hour 84 - 90  pull it up . It is in my files but it will not load .

4  plus DAYS OUT and it never moved .

[PB GFI]

Maybe that bias has been dealt with in this new version of the GFS.

On this one , but  not on it`s first chance out of the gate - it was not fixed . So they had every reason to believe it still existed .

 

It was not like the GFS did not see this  the GEFS was going back and forth between 1.25- 1.50 -1.75 for the 24 hours to the run up.

They were only looking for .25 more and it matched the Euro

 

You guys are making it sound like it had .5 at the park  it did not

[Brian5671]

I cant post the map, they will not let me  find Jan 20 hour 84 - 90  pull it up . It is in my files but it will not load .

4  plus DAYS OUT and it never moved .

I think it was the Sat 0z run that showed a partial hit, it was 12z Sat where it really delivered the goods, so 12z Sat, 0z Sun, 12z Sunday, and 0z Monday.   By 12z Monday it was shifting east, but still had 20-25 inch totals for Western CT.    Amazing to see the Euro bust that bad 12-24 hrs out, I'm astounded really.

[wxman]

None of this shakes my faith in the euro.  When it locks onto an east coast storm it is usually money.  It was this time too, but for our small area.  It was the first to sniff this out...last Friday the GFS thought this whole thing was a clipper heading south!  

[cleetussnow]

That's exactly right.

There is nothing wrong with explaining this storm could range from 6-24" and here's why.

I wonder if because the GFS upgrade had been so erratic, the runs that showed an eastern progression were discounted. In other words, the dissenting voice was muted. Maybe if it was the old gfs, more of a spread in forecasts would be introduced.

[IsentropicLift]

I put my faith in the Euro and its ensembles. It was really the only model that didn't make huge shifts. It's hard to go against the best model even when the writing was on the wall. I still never expected this. I probably received less than 0.25" WE out here.

I knew something was up when the 00z EPS members shifted East on Sunday night.

[PB GFI]

I think it was the Sat 0z run that showed a partial hit, it was 12z Sat where it really delivered the goods, so 12z Sat, 0z Sun, 12z Sunday, and 0z Monday.   By 12z Monday it was shifting east, but still had 20-25 inch totals for Western CT.    Amazing to see the Euro bust that bad 12-24 hrs out, I'm astounded really.

Again the SREFs were 1.75 the GEFS were 1.50 the NAM was 2 the Euro was 2.5 .

It wasn't the only one .

The GFS 0z OP Sunday night  backed down to .9 at the park  but at noon yesterday the rest of the guidance held .

 

24 - 36 means if 24 falls that's a good forecast .  24 plus  was modeled on the Euro SREF NAM 18z GEFS

In the end they all sucked west of LGA  and the 0z GFS the night before turned out right . 

What about all its Sunday runs that had 1 inch back to EWR and .75 to KMMU is that lost on people .

The model 36 hours out was not as strong but its members were not that far off .

The Euro stands out because it was given the most weight in peoples minds but if you the WPC disco Sunday they weighted the SREF as number 1 .

 

You cant blame the offices all 3 collaborated and agreed .  24 plus was modeled and they were off by 75 miles South of 40 - %^& happens .

[cleetussnow]

It was a smallish error in the big picture. But if you can't trust the euro, who the hell can you trust? Lol

[SnowGoose69]

It was a smallish error in the big picture. But if you can't trust the euro, who the hell can you trust? Lol

The problem was the Euro was clearly moving east late in the game gradually, I think in hindsight the best idea was knock off it's numbers a tad and shift it east 40 miles on yesterday's 12Z run and that would have been very close. The GFS and RGEM didn't exactly do a spectacular job either, the RGEM run's late yesterday were not far enough west with some of the heavier snow although they were good in far western areas.

[JERSEYSNOWROB]

I think we're all taking a beating today from family, friends and co-workers, not just the NWS. ha ha ha  Is this our group therapy session?? ha ha ha

[IsentropicLift]

It's not that bad because LI was hit hard. At least some areas lived up to the hype. That will save face for a lot of mets. Out here in NJ it was the most disapointing storm in many years. I would have taken my 12" that every model had yesterday and ran with it. I didn't measure but my eyeball final total is 3.5".

[WintersGrasp]

The problem was the Euro was clearly moving east late in the game gradually, I think in hindsight the best idea was knock off it's numbers a tad and shift it east 40 miles on yesterday's 12Z run and that would have been very close. The GFS and RGEM didn't exactly do a spectacular job either, the RGEM run's late yesterday were not far enough west with some of the heavier snow although they were good in far western areas.

The RGEM for the most part showed us on the western edge of things (in NNJ/NYC).....other models were east and even the euro trending east (which it seems to do more gradually than other models usually). I feel as if many times forecasts tend to put the upper end of what they see possible as the lower end. In the 24-36" forecast, there was not much support for the 36". There was, however, support for the 24" in some areas with a possible trend east due to being on the western fringes as well as the models shifting east. 12-18" with higher amounts in the city would have been a much better call (although still on the bullish end) and 18-24" with higher amounts for central and eastern LI. People still would take that very seriously, it would be the more accurate prediction supportive of the models, and it leaves open the potential for a little bit more in some areas. Going 24-36" for such a wide area was unreasonable, especially considering that the potential for this to dip well below 24" was much greater than the potential for this to go above 36"

[real]

As I said on another thread, forecasters probably should have more care before predicting a "historic" snowfall or a "record-breaking" event.  When any major East Coast city gets to about foot, most people understand its a big deal. So what is wrong with starting out conservative.

 

So even if you are thinking a storm could top 2 feet, you start out with a forecast of 10 to 20 or 14 to 18, etc.  And then as event draws clearer - and in the case of a Miller B actually forms -- you gradually inch up storm totals if conditions and trends warrant. 

 

If the worry is making sure local and state governments/road crews, etc, are adequately prepared, aren't there non-public  conference calls in which NWS can share worst case scenarios, in private?  But when you are predicting 2 to 3 feet for any storm on East Coast, it just screams potential for a bust and an upset public.

[IsentropicLift]

The GFS was just as awful. I think the 18z run yesterday had most of LI under 1" liquid.

[WintersGrasp]

The GFS was just as awful. I think the 18z run yesterday had most of LI under 1" liquid.

Oh yeah no doubts about that.....but the key was the defined eastward trend with the heavy precip on pretty much all of the models within 2-3 days and even the euro in the last day. Combining the euro's formation of the storm and the NAM's qpf numbers for a forecast is like juicing apples and oranges and hoping it comes out peanut butter.