NWS Product Debate

[Deck Pic]

Stop switching the message! 10 and 90 posts were 3.4 at minimum.

 

right.  What I am saying is this...I agree that conveying that a complete bust as a minimum from the range that is useful for high level planning is not helpful, EVEN if a <1" amount is in fact what should be the 10%.  Which is my "beef" and since the product seeks feedback, I am offering my feedback.  So in my opinion you are dealing with a flawed product, but a really cool one.  That's all.  I don't agree that at the range you issued the product that 3.4" was a figure that would be met or exceeded 90% of the time.  and YET if you had used a number that better reflects what is in fact the 10th percentile (imo), it wouldn't be helpful for planning purposes.  We can reasonably disagree on the number that has a 90% probability of being exceeded, but I don't think I am crazy for thinking it should have been way lower than 3.4" at the range the product was issued.  CWG (who busted badly imo and their grade of C is way too generous) are very talented mets, one of whom is a retired government met who is considered a winter weather expert, and at the time the 3.4" was issued, 3.4" was the higher end of their 1-4" (15th - 85th) forecast range.

[Deck Pic]

I think you guys are talking past each other. 

 

Instead of mediating what I consider a healthy debate, perhaps you'd like to help clarify what is being said for purposes of advancing the debate, or even render an opinion.  Having read my last several posts, I am unclear where I am talking past anyone.  Doesn't mean I am being clear.  I am happy for help. I manage people at work and sometimes they don't know what the hell I am saying, and I will ask someone for an assist.

[ers-wxman1]

right. What I am saying is this...I agree that conveying that a complete bust as a minimum from the range that is useful for high level planning is not helpful, EVEN if a <1" amount is in fact what should be the 10%. Which is my "beef" and since the product seeks feedback, I am offering my feedback. So in my opinion you are dealing with a flawed product, but a really cool one. That's all. I don't agree that at the range you issued the product that 3.4" was a figure that would be met or exceeded 90% of the time. and YET if you had used a number that better reflects what is in fact the 10th percentile (imo), it wouldn't be helpful for planning purposes. We can reasonably disagree on the number that has a 90% probability of being exceeded, but I don't think I am crazy for thinking it should have been way lower than 3.4" at the range the product was issued. CWG (who busted badly imo and their grade of C is way too generous) are very talented mets, one of whom is a retired government met who is considered a winter weather expert, and at the time the 3.4" was issued, 3.4" was the higher end of their 1-4" (15th - 85th) forecast range.

Ok thanks for your side. Did you get my pm?

[Deck Pic]

Ok thanks for your side. Did you get my pm?

 

yes...I responded...I am a cool guy.  I think in person, even "I sleep with a government met" and "IrishRob" would like me (or at least be able to tolerate me).

 

BTW, I totally understand that especially govt employees and even employees of private orgs, have to be careful not to throw their colleagues under the bus, and have to measure what they say and what they are "allowed" to convey.  So in a way it may not be a fair playing field, since I don't have to measure my words or worry about sabotaging the people I work with.

[WxUSAF]

Instead of mediating what I consider a healthy debate, perhaps you'd like to help clarify what is being said for purposes of advancing the debate, or even render an opinion.  Having read my last several posts, I am unclear where I am talking past anyone.  Doesn't mean I am being clear.  I am happy for help. I manage people at work and sometimes they don't know what the hell I am saying, and I will ask someone for an assist.

I haven't done any mediating.  That was my way of suggesting to both of you to perhaps rephrase how you both are making your points.  

[ers-wxman1]

The retired met is well known and a great guy. I very much respect what he has to say. I appreciate all members of CWG. They were even at our open house two years ago. With forecasting though we don't collaborate with them or any private company for that matter. Strictly cannot.

[Deck Pic]

I haven't done any mediating.  That was my way of suggesting to both of you to perhaps rephrase how you both are making your points.  

 

right..I think it is resolved now, but having read my posts, I would have appreciated feedback on where I was being unclear or talking pas someone.

[Deck Pic]

The retired met is well known and a great guy. I very much respect what he has to say. I appreciate all members of CWG. They were even at our open house two years ago. With forecasting though we don't collaborate with them or any private company for that matter. Strictly cannot.

 

totally understood

[Ian]

Some day NWS will become more open to a discussion with the outside.. taking comments on Facebook or talking to EMs doesn't really count.  They had a great forecast though and I am sorry I was so vocal that parts of it were stupid. I will make it more of a point to praise them when they deserve it rather than just going about my way. I think I and others made a number of valid points intermixed as well though. Hopefully someone else within the organization will also think some of them up among other things.

[SerialDerecho]

LWX did great with this storm and deserve kudos, but the products need to be judged based on the information available at the time...I approached it too harshly, but I stand by it 100%.  Even if not intended for the public (not so sure about this...it is readily available), a 10th percentile amount of 3.4" was totally unsupported by the envelope of data.  We know now that it is an experimental product, and they are largely hamstrung by an automated process heavily influence by SREFS and rounded out by some other American guidance.  I don't think they can completely toss a 32 member ensemble suite...only tweak.  It is good to know they are folding in some euro ensembles next year and it will make a flawed, but very cool and formidable product even better.  I don't think you will find a single operational met in DC metro who thinks 3.4" was a low goalpost.  CWG is at the very least on par with LWX as forecasters for DC metro, and at the time these products were being issued with those numbers, CWG had a 10th percentile goal post of less than 1".  I realize this is probably over your head, but I figured I'd recap.

 

That's not over my head at all, but thanks for being condescending yet again. 

[chris87]

Stop switching the message! 10 and 90 posts were 3.4 at minimum.

 

just a clarification that may put some things about these maps into perspective -- how much latitude does a forecaster have to deviate from the initial SREF-based analysis -- i assume the product is initially populated with direct SREF output? Also, from a time constraint how much time does a forecaster be allowed/able to dedicate, assuming his/her other responsibilities to add human input to the maps?

 

I think people like to compare to say the CWG "boom/bust" ranges -- which I assume (those close to them can comment) are done after hours of analysis/debate and are the sole product produced by CWG. 

[WEATHER53]

Majority hung their hats on climo. I went with persistence, winter seasons can show once every 10 to 20 years that everytime it can possiby snow-it does snow. Tried to say over and over that nightime events in mid March can deliver because temps only have to be about 8 degrees below average at night to deliver. Even with that, my original forecast of 3"+ on Thursday was too low and my final update at midnight of 4-7" was still too low.

[Deck Pic]

Majority hung their hats on climo. I went with persistence, winter seasons can show once every 10 to 20 years that everytime it can possiby snow-it does snow. Tried to say over and over that nightime events in mid March can deliver because temps only have to be about 8 degrees below average at night to deliver. Even with that, my original forecast of 3"+ on Thursday was too low and my final update at midnight of 4-7" was still too low.

 

most of us were nearing or already in that range by midnight...a little too much back patting for a busted forecast, though some of us were certainly more bearish than you on Thursday

[MN Transplant]

I'm not as concerned as zwyts with the 90/10 products.  People will end up getting more used to the probabilistic nature of forecasting.  I do think the titles on the webpage and on the graphs themselves should match.  (either go with maximum/minimum or 90/10)  

 

What I am happy to hear is that there will be less of a reliance on the SREFs.  I think everyone is still a little stung by the event below.  In most situations, the 10% in our minds will be a trace.  Everything worked out great today, but we have plenty of cautionary tales from the past. 

 

 

 

[mkellerm]

Great Point! You nailed it. We are trying to take an in exact science, make sense of many solutions that are heavily physics and math based with scientific analytical meteorologists and developers and trying to communicate uncertainties in a simpler way and more social sciency way. Sometimes we are our own worst enemy. But with any good thing... It takes practice and an openness to change. We are attempting things that haven't been done on a local Cwa scale. And boy has it been tough but we will push through til we have perfected the science. End goal is much less surprises for storms

 

As a social scientist with zero weather expertise, I've read this discussion with interest.  I think that the probabilistic forecasts are great, and I used one from a previous storm this winter as a question on a problem set for an applied stats class. 

 

Based on the description in one of the earlier posts, it seems like LWX is doing a sort of empirical Bayesian model averaging to generate the probability maps.  That seems like a good approach, but it has some drawbacks:

1) The probabilities are only as good as the underlying models, as has been acknowledged.  Not much to be done about that.

2) It is always harder to estimate extreme quantiles, so there is just a lot more error in 10/90 estimates than in the median.  If there were 200 ensemble members, the estimates of the extremes would be more accurate.  They'd also jump around a lot less from run to run, which is one of the things that is confusing about the current maps - some of those changes are presumably simulation error.

3) Most people overestimate low probability events, particularly those with no background in statistics.  This is how you get Anne Arundel Emergency Management sending out robocalls for a forecast of up to 14 inches on 3/3 when even taking the probabilistic forecasts at face value that was a very unlikely event. 

 

With all of that being said, I think you'd have fewer misunderstandings with 20/80 maps than the current product.  The estimates themselves would be more accurate, and the problems with interpretation would be less severe.

[WEATHER53]

Don't know about the backpatting stuff from grumpster but there was nowhere near 7" just before midnight and no one had put an accumulation on it as early as Thursday and while we are at it, the weazel worded "low confidence-medium-high" is bullsh*t parachute stuff. Make the forecast, give a decent range of 5-7" and not this newly emerging 3-8" crap, take credits for your hits if you actually give a specific and non bail-me-out forecast and if you bust say you did. 4-7" last call with 5-10 realized is Not a bsuted forecst, it's not an excellent one but no bust.

[Deck Pic]

As a social scientist with zero weather expertise, I've read this discussion with interest.  I think that the probabilistic forecasts are great, and I used one from a previous storm this winter as a question on a problem set for an applied stats class. 

 

Based on the description in one of the earlier posts, it seems like LWX is doing a sort of empirical Bayesian model averaging to generate the probability maps.  That seems like a good approach, but it has some drawbacks:

1) The probabilities are only as good as the underlying models, as has been acknowledged.  Not much to be done about that.

2) It is always harder to estimate extreme quantiles, so there is just a lot more error in 10/90 estimates than in the median.  If there were 200 ensemble members, the estimates of the extremes would be more accurate.  They'd also jump around a lot less from run to run, which is one of the things that is confusing about the current maps - some of those changes are presumably simulation error.

3) Most people overestimate low probability events, particularly those with no background in statistics.  This is how you get Anne Arundel Emergency Management sending out robocalls for a forecast of up to 14 inches on 3/3 when even taking the probabilistic forecasts at face value that was a very unlikely event. 

 

With all of that being said, I think you'd have fewer misunderstandings with 20/80 maps than the current product.  The estimates themselves would be more accurate, and the problems with interpretation would be less severe.

 

I like this....when you are tethered (somewhat) to guidance that is not infrequently an outlier (usually on the bullish side), with 90-10 you take a greater risk of a high bust (forecast more than occurs), which is the kind of  bust which  the public seems to be the least forgiving of.

[Deck Pic]

Don't know about the backpatting stuff from grumpster but there was nowhere near 7" just before midnight and no one had put an accumulation on it as early as Thursday and while we are at it, the weazel worded "low confidence-medium-high" is bullsh*t parachute stuff. Make the forecast, give a decent range of 5-7" and not this newly emerging 3-8" crap, take credits for your hits if you actually give a specific and non bail-me-out forecast and if you bust say you did. 4-7" last call with 5-10 realized is Not a bsuted forecst, it's not an excellent one but no bust.

 

fair enough...good call on Thursday...

[Ian]

I like this....when you are tethered (somewhat) to guidance that is not infrequently an outlier (usually on the bullish side), with 90-10 you take a greater risk of a high bust (forecast more than occurs), which is the kind of  bust which  the public seems to be the least forgiving of.

I noted almost exactly the same thing the other night as well.. and think those numbers are closer to CWG boom/bust.

 

My assumption is the biggest problem with the product is there is likely very little forecaster input.. otherwise it would be freaking tedious considering how often it updates.

[ers-wxman1]

just a clarification that may put some things about these maps into perspective -- how much latitude does a forecaster have to deviate from the initial SREF-based analysis -- i assume the product is initially populated with direct SREF output? Also, from a time constraint how much time does a forecaster be allowed/able to dedicate, assuming his/her other responsibilities to add human input to the maps?

I think people like to compare to say the CWG "boom/bust" ranges -- which I assume (those close to them can comment) are done after hours of analysis/debate and are the sole product produced by CWG.

For everyone just FYI here is a running list of what goes into the min and max maps....remember this is the starting point...

32 members

21 SREF

6 GEFS

1 GFS

1 NAM

2 Euro

1 Canadian

32 members.

Then updates by forecaster and or collaborate with WPC if changes are significant. When all is well they are published, shipped to winter page. In addition we get WPC probs for >t through >18 and we tweak those as well. Top two products make up the table at the bottom.

[Deck Pic]

I noted almost exactly the same thing the other night as well.. and think those numbers are closer to CWG boom/bust.

 

My assumption is the biggest problem with the product is there is likely very little forecaster input.. otherwise it would be freaking tedious considering how often it updates.

 

you know how I feel about that

[ers-wxman1]

I noted almost exactly the same thing the other night as well.. and think those numbers are closer to CWG boom/bust.

My assumption is the biggest problem with the product is there is likely very little forecaster input.. otherwise it would be freaking tedious considering how often it updates.

See post above... There is input. Whether or not the change is significant enough is forecaster discretion but note we want a consistent message with WPC since the guidance is being generated from that end. At times it required meeting collaboration with bigger changes.

[WxUSAF]

For everyone just FYI here is a running list of what goes into the min and max maps....remember this is the starting point...

32 members

21 SREF

6 GEFS

1 GFS

1 NAM

2 Euro

1 Canadian

32 members.

Then updates by forecaster and or collaborate with WPC if changes are significant. When all is well they are published, shipped to winter page. In addition we get WPC probs for >t through >18 and we tweak those as well. Top two products make up the table at the bottom.

And those are all given equal weightings?  

[ers-wxman1]

Some day NWS will become more open to a discussion with the outside.. taking comments on Facebook or talking to EMs doesn't really count. They had a great forecast though and I am sorry I was so vocal that parts of it were stupid. I will make it more of a point to praise them when they deserve it rather than just going about my way. I think I and others made a number of valid points intermixed as well though. Hopefully someone else within the organization will also think some of them up among other things.

Someday? :-). I'm doing it now lol

[chris87]

And those are all given equal weightings?  

 

i suppose when you're trying to resolve the pdf you probably have to include as much you can with equal weighting -- now a weighted ensemble mean would be interesting if you could tie it to past performance (but not sure you can do that with the perturbed ensemble members)...

[Ian]

See post above... There is input. Whether or not the change is significant enough is forecaster discretion but note we want a consistent message with WPC since the guidance is being generated from that end. At times it required meeting collaboration with bigger changes.

Well unless you guys become much more transparent it's hard to judge exactly what that means from the outside. Input could be glancing and approving at least in some cases it seems. Otherwise there must be a team that only creates those maps given all the points and products.

I don't really blame NWS for being heavily internalized with their processes. I think it's just a big gov thing and there is not really any outside org pushing for policy changes etc like there is with most other big gov entities. Not to mention the org is hugely balkanized which presents a ton of problems in itself.

[WxUSAF]

i suppose when you're trying to resolve the pdf you probably have to include as much you can with equal weighting -- now a weighted ensemble mean would be interesting if you could tie it to past performance (but not sure you can do that with the perturbed ensemble members)...

Yeah, good point RE: the perturbed members.  

 

Given those members, most of your distribution is going to come from the SREFs in almost each scenario.  But, based on past performance, I'm going to give the Euro, GFS and GGEM much more weighting than any individual SREF member and even more than the SREF mean.  Not sure what the best solution is there, but giving most weight to the SREFs is going to create issues in most storms.  

[Deck Pic]

Yeah, good point RE: the perturbed members.  

 

Given those members, most of your distribution is going to come from the SREFs in almost each scenario.  But, based on past performance, I'm going to give the Euro, GFS and GGEM much more weighting than any individual SREF member and even more than the SREF mean.  Not sure what the best solution is there, but giving most weight to the SREFs is going to create issues in most storms.  

 

which is why they are adding a bunch of euro ens members next year (according to wxman)

[chris87]

Well unless you guys become much more transparent it's hard to judge exactly what that means from the outside. Input could be glancing and approving at least in some cases it seems. Otherwise there must be a team that only creates those maps given all the points and products.

I don't really blame NWS for being heavily internalized with their processes. I think it's just a big gov thing and there is not really any outside org pushing for policy changes etc like there is with most other big gov entities. Not to mention the org is hugely balkanized which presents a ton of problems in itself.

 

are you talking about from a verification standpoint? if so, i'd agree, i have never understood why WFO-based verification stats are kept so secret -- because performance within the office is heavily scrutinized internally by regional and national leadership. 

[Ian]

Someday? :-). I'm doing it now lol

And again I commend you for discussing things here as I'm sure you do in many places.. from what I've seen you are a rare commodity in the organization. And I know enough people in it at various levels that I'm not just talking out my arse or something here. 

 

NWS is moving in the direction that will probably eventually resolve a lot of the issues currentl seen... I am just not sure it will ever happen fast enough to keep up with the rest of the world unless there is more outside interference run.  Unfortunately, IMO, everyone is so heavily focused on climate change that it's hard to find people who are as interested in similar policy ideas for simple weather.  I do think that will change as well eventually.. but I don't have 90% confidence yet.