NWS Product Debate

[winterymix]

No.  I didn't receive one.  I see ERS is still posting this morning, which I'm glad for.  

Yeah, I think the truth of the matter is that the products that he oversees are a good start

and I've looked at them for each storm.  Is there room for improvement?  Sure, most concepts can be elevated a level or more with additional thought but I very much prefer a welcoming tone towards the LWX staff and I doubt I'm the only one. 

[Deck Pic]

Great job to the bashers of the NWS.

You bashed what turned out to be a great forecast, drove away a red tagger and made a$$e$ out of yourselves.

There's debating, disagreeing with an official forecast and then there's trashing it. Such class. These same posters are welcome to this forum, while red taggers are chased away. This isn't the first instance certain members chased members away, yet they are invincible as

To disciplinary actions.

 

ummm..no..sorry you got almost shut out in Philly...actually...no I'm not

[donsutherland1]

As my explanation related to the percentile products was moved here (I had to look for it as I had posted it elsewhere), I'll post my comments here.

 

1. Storm total snowfall fell between the 10th and 90th percentile amounts as one should have expected. The maps depicted thresholds for which only 10% of values would fall below the lowest threshold and 10% would wind up higher than the highest threshold. They did not forecast probabilities of given amounts. This is a great probabilistic tool if one is considering scenarios. Hopefully, the tool will be used going forward.

 

2. The guidance, which was more aggressive than recent climatology, proved accurate.

 

IMO, NWS Sterling did a terrific job. This latest performance offers yet another reminder of what a valuable resource NWS is.

[NaoPos]

ummm..no..sorry you got almost shut out in Philly...actually...no I'm not

this has nothing to do with philly. Great redirect though.

I simply voiced an opinion (after all, thats what this thread was based upon) about what had transpired. Seeing as that the event just ended this am, I wouldn't call it an old wound.

Some people bashed the NWS in a disrespectful manner, and nothing was done about it. Doesn't matter that I'm from philly. I post weather info in this forum at times, and no one is limited to posting to their own region.

[Deck Pic]

Yea, how dare LWX issue such "preposterous" products/forecasts.    

 

LWX did great with this storm and deserve kudos, but the products need to be judged based on the information available at the time...I approached it too harshly, but I stand by it 100%.  Even if not intended for the public (not so sure about this...it is readily available), a 10th percentile amount of 3.4" was totally unsupported by the envelope of data.  We know now that it is an experimental product, and they are largely hamstrung by an automated process heavily influence by SREFS and rounded out by some other American guidance.  I don't think they can completely toss a 32 member ensemble suite...only tweak.  It is good to know they are folding in some euro ensembles next year and it will make a flawed, but very cool and formidable product even better.  I don't think you will find a single operational met in DC metro who thinks 3.4" was a low goalpost.  CWG is at the very least on par with LWX as forecasters for DC metro, and at the time these products were being issued with those numbers, CWG had a 10th percentile goal post of less than 1".  I realize this is probably over your head, but I figured I'd recap.

[donsutherland1]

Yeah, I think the truth of the matter is that the products that he oversees are a good start

and I've looked at them for each storm.  Is there room for improvement?  Sure, most concepts can be elevated a level or more with additional thought but I very much prefer a welcoming tone towards the LWX staff and I doubt I'm the only one. 

I like the increasing use of probabalistic tools. They offer a degree of rich insight that one doesn't see in the deterministic forecasts.

[Deck Pic]

As my explanation related to the percentile products was moved here (I had to look for it as I had posted it elsewhere), I'll post my comments here.

 

1. Storm total snowfall fell between the 10th and 90th percentile amounts as one should have expected. The maps depicted thresholds for which only 10% of values would fall below the lowest threshold and 10% would wind up higher than the highest threshold. They did not forecast probabilities of given amounts. This is a great probabilistic tool if one is considering scenarios. Hopefully, the tool will be used going forward.

 

2. The guidance, which was more aggressive than recent climatology, proved accurate.

 

IMO, NWS Sterling did a terrific job. This latest performance offers yet another reminder of what a valuable resource NWS is.

 

They did, but it was a different product, and equally flawed.

 

The 10th percentile amount is suggested as an amount that represents the least amount of snow the storm can produce or that one should expect this amount as a minimum.  I don't think any operational met would agree that at the time this was issued that 3.4" was representative of the least amount of snow the storm could produce.  I didn't follow these products as closely earlier in the winter, but I am curious what this number was for 12/10.

[Deck Pic]

this has nothing to do with philly. Great redirect though.

I simply voiced an opinion (after all, thats what this thread was based upon) about what had transpired. Seeing as that the event just ended this am, I wouldn't call it an old wound.

Some people bashed the NWS in a disrespectful manner, and nothing was done about it. Doesn't matter that I'm from philly. I post weather info in this forum at times, and no one is limited to posting to their own region.

 

Thanks. I always appreciate the input of someone outside this region who has no skin in the game on matters that have been resolved.

[chris87]

They did, but it was a different product, and equally flawed.

 

The 10th percentile amount is suggested as an amount that represents the least amount of snow the storm can produce or that one should expect this amount as a minimum.  I don't think any operational met would agree that at the time this was issued that 3.4" was representative of the least amount of snow the storm could produce.  I didn't follow these products as closely earlier in the winter, but I am curious what this number was for 12/10.

 

Probably been said earlier -- the problem doesn't lie with logic of the products but the flawed inputs -- I think someone mentioned future plans to move to something like a "super ensemble" where these products are based on more than just the SREFs -- sometimes I think the NWS moves towards products such as these (and for example, the Warn-on-Forecast movement for severe weather) without asking themselves are we ready to use these products, is the science ready for such products and can we effectively communicate what the products mean. 

[donsutherland1]

They did, but it was a different product, and equally flawed.

 

The 10th percentile amount is suggested as an amount that represents the least amount of snow the storm can produce or that one should expect this amount as a minimum.  I don't think any operational met would agree that at the time this was issued that 3.4" was representative of the least amount of snow the storm could produce.  I didn't follow these products as closely earlier in the winter, but I am curious what this number was for 12/10.

 

No disagreement.

 

I understood that the product represented LWX's assessment, and fully realize others might well have had differing risk assessments. I don't believe anyone should have assumed that the figures were maximum or minimum snowfall amounts, but only that LWX estimated that 10% of cases might wind up above or below the given thresholds and 80% would fall within the parameters. If one used the figures to write off forecasts that might have been more conservative, in this case, I don't believe that is a good use of the product. After all, this is only the 2nd snowstorm since 1950 to bring 6" or more snow to DCA after March 15.

 

Moreover, like any product, it has its own limitations. The real test over time will be whether snowfall amounts wind up above or below the thresholds in 10% of the cases/fall within the parameters 80% of the time. If so, the probabalistic assessment will be reasonable. If not, then the percentiles will need to be refined.

 

Finally, like you, I would definitely like to know how the product has performed in other cases. For now, it's a work in progress.

[Deck Pic]

Yeah, I think the truth of the matter is that the products that he oversees are a good start

and I've looked at them for each storm.  Is there room for improvement?  Sure, most concepts can be elevated a level or more with additional thought but I very much prefer a welcoming tone towards the LWX staff and I doubt I'm the only one. 

 

A lot of storms completely bust here..that is DC...even this winter of several overperformers, the WSW on 12/10 busted in a lot of places - badly.  That isn't to say it was necessarily a bad forecast.. I agreed with it at the time.  Turns out it was another learning experience.  The signs of bust were there.

 

Now I don't think it is useful for a <6" storm to have a lower goalpost of complete fail from a certain range even if complete fail is probably 10th percentile for a lot of storms.  Saying every <3-6" storm could crap out is not useful guidance for planners and decision makers. Besides the things that have already been  exhaustively covered, at a certain range I'd probably use a 25%-75% range for planning purposes rather than 90-10, since 10% if portrayed accurately is often <=1".  25% for this storm was probably ~1-2" for immediate DC metro at the time these products were issued.

[WxUSAF]

I think that product has real merit, but it shouldn't be 100% based off the SREFs for sure.  

[Deck Pic]

Probably been said earlier -- the problem doesn't lie with logic of the products but the flawed inputs -- I think someone mentioned future plans to move to something like a "super ensemble" where these products are based on more than just the SREFs -- sometimes I think the NWS moves towards products such as these (and for example, the Warn-on-Forecast movement for severe weather) without asking themselves are we ready to use these products, is the science ready for such products and can we effectively communicate what the products mean. 

 

I think it is both.  Even if euro ensembles were folded in, a 10th percentile number is usually going to be higher than it should be since 10th percentile from a range for planning purposes is usually <1" except for our bigger events.  A lot of the public is dumb , but there are enough smart people out there who based on their understanding of percentiles, would consider a goal post or a number that is 10th percentile to be a virtual lock.

[Deck Pic]

Didn't see ERS say he was leaving until now.  That's disappointing.  

 

fortunately we are both adults, and ERS was kind enough to accept my public and private apology and return to the board.  I often PM people I get in spats with, usually with resolution,  and I don't CC the entire board.  

[WxUSAF]

fortunately we are both adults, and ERS was kind enough to accept my public and private apology and return to the board.  I often PM people I get in spats with, usually with resolution,  and I don't CC the entire board.  

Glad to hear it.  I wasn't around too much yesterday, so I only saw a little of the discussion and didn't realize he got upset about it until this AM.  

[ers-wxman1]

They did, but it was a different product, and equally flawed.

The 10th percentile amount is suggested as an amount that represents the least amount of snow the storm can produce or that one should expect this amount as a minimum. I don't think any operational met would agree that at the time this was issued that 3.4" was representative of the least amount of snow the storm could produce. I didn't follow these products as closely earlier in the winter, but I am curious what this number was for 12/10.

10th percentile graphic should be examined as follows...

A 10 percent probability you will receive at least xx amount of snow and a 90 percent probability it will be greater than that number. The 3.4 was right in line and a good minimum number.

[chris87]

I think it is both.  Even if euro ensembles were folded in, a 10th percentile number is usually going to be higher than it should be since 10th percentile from a range for planning purposes is usually <1" except for our bigger events.  A lot of the public is dumb , but there are enough smart people out there who based on their understanding of percentiles, would consider a goal post or a number that is 10th percentile to be a virtual lock.

 

very true -- in general the public understands deterministic forecasts much better than probabilistic -- i think as a community were stuck in between transitioning to more probabilistic forecasts, but it's held back due to the fact that even the best ensemble systems aren't as robust as we'd like and we haven't found a constructive way to communicate uncertainty to the general public. 

[ers-wxman1]

LWX did great with this storm and deserve kudos, but the products need to be judged based on the information available at the time...I approached it too harshly, but I stand by it 100%. Even if not intended for the public (not so sure about this...it is readily available), a 10th percentile amount of 3.4" was totally unsupported by the envelope of data. We know now that it is an experimental product, and they are largely hamstrung by an automated process heavily influence by SREFS and rounded out by some other American guidance. I don't think they can completely toss a 32 member ensemble suite...only tweak. It is good to know they are folding in some euro ensembles next year and it will make a flawed, but very cool and formidable product even better. I don't think you will find a single operational met in DC metro who thinks 3.4" was a low goalpost. CWG is at the very least on par with LWX as forecasters for DC metro, and at the time these products were being issued with those numbers, CWG had a 10th percentile goal post of less than 1". I realize this is probably over your head, but I figured I'd recap.

There was more human element involved in adjusting the graphic for this storm. We just don't plug and play. There is a level of adjustment with each run... 08 and 20z plus periodic updates at forecaster discretion. First year running these. Concept is great, improvements will be made all spring and summer with expansions to include ice next year and as WXMAN1 pointed out there will be adjustments with adding numerous ENS members as well. Your feedback and ALL feedback is appreciated.

[ers-wxman1]

very true -- in general the public understands deterministic forecasts much better than probabilistic -- i think as a community were stuck in between transitioning to more probabilistic forecasts, but it's held back due to the fact that even the best ensemble systems aren't as robust as we'd like and we haven't found a constructive way to communicate uncertainty to the general public.

Great Point! You nailed it. We are trying to take an in exact science, make sense of many solutions that are heavily physics and math based with scientific analytical meteorologists and developers and trying to communicate uncertainties in a simpler way and more social sciency way. Sometimes we are our own worst enemy. But with any good thing... It takes practice and an openness to change. We are attempting things that haven't been done on a local Cwa scale. And boy has it been tough but we will push through til we have perfected the science. End goal is much less surprises for storms

[Deck Pic]

10th percentile graphic should be examined as follows...

A 10 percent probability you will receive at least xx amount of snow and a 90 percent probability it will be greater than that number. The 3.4 was right in line and a good minimum number.

 

It's too bad you think that.  There will be less improvement and less human tweaking (even if much is allowed) if you think that is true. A number shouldn't be judged by the result.  I wonder what the number was for 12/10 or 3/5/13 or 12/26/10 (I dont think they are around) and if you still think they were good numbers regardless of the result.

[Fozz]

It's too bad you think that.  There will be less improvement and less human tweaking (even if much is allowed) if you think that is true. A number shouldn't be judged by the result.  I wonder what the number was for 12/10 or 3/5/13 or 12/26/10 (I dont think they are around) and if you still think they were good numbers regardless of the result.

 

I don't remember the exact number for 3/5/13 (I'm guessing 0.5" but I could be off), but I've seen the 10th percentile map for that storm and it looked very close to what actually happened.

[Deck Pic]

It's too bad you think that.  There will be less improvement and less human tweaking (even if much is allowed) if you think that is true. A number shouldn't be judged by the result.  I wonder what the number was for 12/10 or 3/5/13 or 12/26/10 (I dont think they are around) and if you still think they were good numbers regardless of the result

 

10th percentile graphic should be examined as follows...

A 10 percent probability you will receive at least xx amount of snow and a 90 percent probability it will be greater than that number. The 3.4 was right in line and a good minimum number.

 

 

It's too bad you think that.  There will be less improvement and less human tweaking (even if much is allowed) if you think that is true. A number shouldn't be judged by the result.  I wonder what the number was for 12/10 or 3/5/13 or 12/26/10 (I dont think they are around) and if you still think they were good numbers regardless of the result.

 

 

I should rephrase, and say we can agree to disagree.  You're certainly entitled to your opinion and it is as valid as anyone else's. Question, since these products were rolled out have you ever thought the 10th percentile goal post was a bad number and if so would you have been willing to say so publicly?

[ers-wxman1]

I should rephrase, and say we can agree to disagree. You're certainly entitled to your opinion and it is as valid as anyone else's. Question, since these products were rolled out have you ever thought the 10th percentile goal post was a bad number and if so would you have been willing to say so publicly?

Which number would you use as a min case?

[Deck Pic]

Which number would you use as a min case?

 

for this storm at the range the minimum case was issued, I would have used 0.5".  Keep in mind I am not sure that is a useful number to convey to officials, but in terms of the letter of the guidance of 10th percentile, that is the number I would have used for DC/ARL/Alexandria/PG.  Based on CWG's forecast at the time, their 10th percentile was also under 1".  

[ers-wxman1]

I should rephrase, and say we can agree to disagree. You're certainly entitled to your opinion and it is as valid as anyone else's. Question, since these products were rolled out have you ever thought the 10th percentile goal post was a bad number and if so would you have been willing to say so publicly?

We had to set the bar within a range. We picked 10 and 90. Could it be tweaked? Absolutely. Out of your objections which are fine, what would be your recommendation for the range?

[ers-wxman1]

for this storm at the range the minimum case was issued, I would have used 0.5". Keep in mind I am not sure that is a useful number to convey to officials, but in terms of the letter of the guidance of 10th percentile, that is the number I would have used for DC/ARL/Alexandria/PG. Based on CWG's forecast at the time, their 10th percentile was also under 1".

We would be doing zero service if we had communicated to any high profile customer that in the next 6 hours you can expect to see 0.5 but as much as 12!! They would say the confidence is low yet we had high confidence with a warning in place. The objective is to narrow the posts with time. Low end was 3.4 high end near 12...those are your planning posts. Most likely verified between 6 and 8 in the district which falls in between the two! The case was a hit.

[Deck Pic]

A lot of storms completely bust here..that is DC...even this winter of several overperformers, the WSW on 12/10 busted in a lot of places - badly.  That isn't to say it was necessarily a bad forecast.. I agreed with it at the time.  Turns out it was another learning experience.  The signs of bust were there.

 

Now I don't think it is useful for a <6" storm to have a lower goalpost of complete fail from a certain range even if complete fail is probably 10th percentile for a lot of storms.  Saying every <3-6" storm could crap out is not useful guidance for planners and decision makers. Besides the things that have already been  exhaustively covered, at a certain range I'd probably use a 25%-75% range for planning purposes rather than 90-10, since 10% if portrayed accurately is often <=1".  25% for this storm was probably ~1-2" for immediate DC metro at the time these products were issued.

 

 

We had to set the bar within a range. We picked 10 and 90. Could it be tweaked? Absolutely. Out of your objections which are fine, what would be your recommendation for the range?

 

see above.  

[Deck Pic]

We would be doing zero service if we had communicated to any high profile customer that in the next 6 hours you can expect to see 0.5 but as much as 12!! They would say the confidence is low yet we had high confidence with a warning in place. The objective is to narrow the posts with time. Low end was 3.4 high end near 12...those are your planning posts. Most likely verified between 6 and 8 in the district which falls in between the two! The case was a hit.

 

agreed, which I have now stated in multiple posts, which is why 90-10 goal posts are misleading or inaccurate.

[ers-wxman1]

agreed, which I have now stated in multiple posts, which is why 90-10 goal posts are misleading or inaccurate.

Stop switching the message! 10 and 90 posts were 3.4 at minimum.

[WxUSAF]

I think you guys are talking past each other.