Are very specific snowstorm postmortems (lessons learned) all that valuable?

[gymengineer]

After every significant event, the Capital Weather Gang (usually Jason) writes a postmortem with lessons learned from the event. Reading through the ones from recent seasons, I'm kind of doubting the value in being so specific about lessons learned. In other words, I don't think there are always reliable lessons to carry from one event into the next one since what caused a forecast to bust in one instance could be completely missing from another case. 

 

I'll start with a good, general lesson that has worked out well in subsequent cases for the CWG:

During the 3/1-2/09 snowstorm, they were very skittish, with good reason, about the lack of snow in the western suburbs up through midnight, so they lowered the totals west of DC in a now-cast. The super-band in the morning made the now-cast bust. 

Later, they were less "jumpy" during storms, even when the storms were slower than predicted in getting going- like in 2/9-10/10 and 1/21/14.  

 

 

But, last winter:

12/10: all sources predicted a high-impact snow during morning commute that ended up having trouble sticking at all on downtown streets. Lesson drawn? Don't ignore the lesson from 3/6/13, low elevations inside the beltway suck at marginal temperature situations, start adding in the DCA bubble on snow maps. 

1/2-3: skeptical of the how quickly the temps would drop (so internalizing the lessons learned from 3/13 and 12/10/13), the snow forecast is very conservative the day of the event- up to an inch for DC southward bumping up to 1-3" in the afternoon with the DCA bubble of 0.5-2". Of course, the storm over-performed. One lesson learned was about the track of the upper-low being near ideal for precip over the area.

1/21: lesson learned- even when deep cold is modeled funneling in during the onset, be skeptical about how well snow will accumulate on streets in the urban areas.

2/12: lesson learned- underforecast for the western 'burbs, so if all of the other models are showing a ton of precip, don't over-weigh the lighter GFS into the forecast

3/2: a couple of now-casting issues, but generally steady forecast (so again, lesson learned from 3/09)

3/16: taking into account the "too warm" lessons learned before, the forecast was for 1-3" in the DCA bubble, 2-4" for the metro area... result was beyond even the "boom" scenario. What happened? Under-estimated the cold, so the opposite of 12/10 and 1/21. 

 

And, after the 3/6/13 bust, there was of course the temp lessons applied immediately to 3/25/13. But that just resulted in forecasts being too conservative (0-2" for DC eastward should have been 1-3", T-3" for west of the city should have been 3-5").

 

I love that CWG does own up to mistakes and tries to improve, but I think the back-and-forth lessons from last season shows that nailing all the details in a snowstorm is difficult in general, and perhaps not doable in the current state of modeling. Even this season, we see that the models can bust in one direction in one storm and then bust in the opposite direction in the next storm. So maybe lessons learned are more effective when collected across several seasons so that they are broader and less storm-specific. 

[Deck Pic]

Aggregating knowledge is certainly more helpful than lessons learned from discrete events.  I think with almost all busts, there was model warning for how the bust happened, that was ignored or diminished in importance.  I can't think of a single bust where there weren't warning signs.  I think it is more helpful to create a checklist before a storm, and maybe that would make a good thread.  One thing that I want to learn more about is flake size and lift...A storm like 3/3, where we had such bad ratios, and kind of had to claw our way to a respectable semi-bust with a +SN band at the end.  Coastalwx and some others saw the signs, but I am not as familiar with the science behind it.  As far as the other stuff it is usually there.  Doesn't mean we should be overly cautious or pessimistic but we shouldn't be blindsided.  3/5/13 we were blindsided. BL/SFC temps, warm nose, mixing, late developers, Miller B's, things that need to back in from the coast, relying on a cutoff to make our own cold air. All the pitfalls are almost always evident.  We might get blindsided legitimately at some point, but a 3/5/13 should never have happened...

[gymengineer]

Aggregating knowledge is certainly more helpful than lessons learned from discrete events.  I think with almost all busts, there was model warning for how the bust happened, that was ignored or diminished in importance.  I can't think of a single bust where there weren't warning signs.  I think it is more helpful to create a checklist before a storm, and maybe that would make a good thread.  One thing that I want to learn more about is flake size and lift...A storm like 3/3, where we had such bad ratios, and kind of had to claw our way to a respectable semi-bust with a +SN band at the end.  Coastalwx and some others saw the signs, but I am not as familiar with the science behind it.  As far as the other stuff it is usually there.  Doesn't mean we should be overly cautious or pessimistic but we shouldn't be blindsided.  3/5/13 we were blindsided. BL/SFC temps, warm nose, mixing, late developers, Miller B's, things that need to back in from the coast, relying on a cutoff to make our own cold air. All the pitfalls are almost always evident.  We might get blindsided legitimately at some point, but a 3/5/13 should never have happened...

 

It's the trend... that's how we got blindsided. Heading into 3/13, there were red flags everywhere about BL temps and even precip type (on the Euro), but it was that last set of 0Z model runs that made everyone excited. Trend was for a better result, so it was pretty easy to disregard the red flags. 

Similar situation happened with 2/22/11-- the last set of runs as precip was beginning suggested a solid hit of snow after the rain-to-sleet-to-snow transition. But the red flags were there with us being on the southern edge and the mid-level warmth. 

[Ian]

I don't think a lot of single-storm lessons are returned to that heavily. If you see a repeating pattern maybe.. for a few storms after the bust maybe.  Focusing on a lesson from one event could probably also hurt a forecast as easily as help.  Obviously lessons that repeatedly return become more like rules. I think psychology is probably a factor in an event like 3/13.. lots of us are still little kids that like snow at heart. If we've been trained to assume a shorter range model is best and the short range all goes nuts what's to stop us from going to snowville.

[gymengineer]

I don't think a lot of single-storm lessons are returned to that heavily. If you see a repeating pattern maybe.. for a few storms after the bust maybe.  Focusing on a lesson from one event could probably also hurt a forecast as easily as help.  Obviously lessons that repeatedly return become more like rules. I think psychology is probably a factor in an event like 3/13.. lots of us are still little kids that like snow at heart. If we've been trained to assume a shorter range model is best and the short range all goes nuts what's to stop us from going to snowville.

Well, the urban core temperature issue still spooks forecasters.... I know I was totally in support of the "DCA bubble" on snow maps, but now I think that it may have been overused by CWG last season. 

In retrospect, I see that forecasters bust too warm and too cold regularly for snowstorms/ice storms/mix storms. Maybe the storms that end up warmer than predicted happen more often than vice versa, but the storms that go the other way don't seem to stick in our heads as much. I think that's because weather weenies don't ever get up in arms about positive busts. 

[T. August]

Well, the urban core temperature issue still spooks forecasters.... I know I was totally in support of the "DCA bubble" on snow maps, but now I think that it may have been overused by CWG last season. 

In retrospect, I see that forecasters bust too warm and too cold regularly for snowstorms/ice storms/mix storms. Maybe the storms that end up warmer than predicted happen more often than vice versa, but the storms that go the other way don't seem to stick in our heads as much. I think that's because weather weenies don't ever get up in arms about positive busts. 

This is at least true for me. Even something as simple as a high temp bust I remember (like today), but the colder than modeled days all seem to blend together.

[Bob Chill]

Good thread. It's too complicated to lump one set of factors and/or previous events over another. We walk the line too often. 3/13 did teach me some lessons thoug. Marginal events on the edge of climo that require dynamics can only succeed when dynamics are realized in real time. We lost on that one when dynamics shifted during nowcast time. It's unpredictable and could have gone the other way just as easily

Walking the line in the heart of climo is much easier than the margins. Last March wasn't walking the line. It was a prime sequence of events for late season snow. And it worked out. It's really tricky in the MA to pre-judge an event based on previous events on the margins. It's a little easier during the heart of climo but risk will always be in the mix.

[Deck Pic]

It's the trend... that's how we got blindsided. Heading into 3/13, there were red flags everywhere about BL temps and even precip type (on the Euro), but it was that last set of 0Z model runs that made everyone excited. Trend was for a better result, so it was pretty easy to disregard the red flags. 

Similar situation happened with 2/22/11-- the last set of runs as precip was beginning suggested a solid hit of snow after the rain-to-sleet-to-snow transition. But the red flags were there with us being on the southern edge and the mid-level warmth. 

 

we talked about all those factors and just pretended like they didn't matter...The 0z NAM was insane, but the 0z Euro was actually not that bullish, and I don't think 12z was either...It's just that nobody pays much attention to the Euro within 24 hours because we get bombarded with constant data and only get one euro run during the day.  By 0z everyone was asleep.  I remember the 3/2 0z Euro made a lot of us nervous...but we all got Nam poisoned...It was so pretty...18z NAM gave me 1.90"....Trend for sure, but we never learn with the NAM because it scores a coup every once in a while...People forget just how many times the NAM has screwed us..

[Amped]

Yes most busts are pretty big lessons. 3/5/13 had about 20 different lessons all packed into one. Every detail was handled the dumbest way possible by everyone involved NWS, news media and social media. Thinking about it still makes my head one to explode.

 

Small events that have high impact should also have some lessons, like last weeks clipper.

[gymengineer]

Good thread. It's too complicated to lump one set of factors and/or previous events over another. We walk the line too often. 3/13 did teach me some lessons thoug. Marginal events on the edge of climo that require dynamics can only succeed when dynamics are realized in real time. We lost on that one when dynamics shifted during nowcast time. It's unpredictable and could have gone the other way just as easily

Walking the line in the heart of climo is much easier than the margins. Last March wasn't walking the line. It was a prime sequence of events for late season snow. And it worked out. It's really tricky in the MA to pre-judge an event based on previous events on the margins. It's a little easier during the heart of climo but risk will always be in the mix.

And that "prime sequence" was evident only part-way into the 3/16-17 event, especially since 3/3 was a bit of an underachiever.

 

Two approaches for 3/16-17: NWS looked at the model outputs, adjusted them somewhat but minimally, and put out a blanket 4-8" amount across the region as of the afternoon of 3/16. Of course they were aware of the 3/13 and 12/10/13 busts, but didn't factor those in, in the end, given what the models were showing verbatim.

CWG looked at the model outputs, but also applied the specific "warm bust" lessons to the accumulation forecast resulting in a 3/16 afternoon forecast of 1-3" for the DCA bubble and 2-4" for the rest of the city and suburbs (other than parts of Fairfax County, which was at the edge of their heaviest band). There was no overlap between the two forecasts for the immediate metro area, which is unusual, and of course it turned out that the NWS's forecast was much better. 

I think CWG was prudent to consider temps being warmer than modeled, especially in the urban areas.. it's just that they (and all other outlets) had more than one bust in both directions last winter, so it was almost as if the lessons canceled each other out storm-by-storm. The models will run too warm and too cold again in the future, so in the end, it's still just each forecaster doing their best interpreting the guidance for each storm.