NWS Product Debate

[Ian]

Constructive criticism is a net plus for any gov agency in particular. NWS is one of the rare places that products are often just fully disseminated without any questioning.. I think that's perhaps why there tends to be many in the org taken aback when anyone has anything but glowing praise.  It is probably among the most efficient federal government agencies at the least.

[winterymix]

..A min of 1" and a max of 16" does me little good to know. It's like well, yeah. Why even bother?

When that sort of range pops up, it is an indication of

 

1.  lack of model support; either the models are fluctuating from run to run or are not converging on a solution

2.  there is enough precipitable water vapor forecast to move into the area and enough lift to ring out 16" but confidence is low because of volatility in the models.

3.  good old fashioned synoptic meteorology is in cognitive dissonance with the models.  For instance, a meteorologist may examine all the standard levels of the atmosphere and then evaluate which troughs and which ridges may strengthen/weaken; all this based upon what is taught in meteorology school and then reinforced through on the job experience.   So what does a meteorologist do if they feel confident in their synoptic analysis and their analysis clashes with numerical modeling?  The meteorologist calculates the minimum snow that the system can deliver and the maximum that the system can deliver.  Such a range of potential snowfall doesn't imply that the meteorologist doesn't know what they are doing.  It implies that at that particular time those predictions are conceivable.  Note that forecaster has every intention of tightening the upper and lower limits as the storm approaches.   Note that with the last system, LWX went with a minimum of 3.4" for DCA.   Those meteorologists must have been particularly confident that 3.4" would be an achievable minimum.   They must have been really confident.

 

Well...probabilistic forecasting allows the meteorologist to communication what he or she thinks will probably occur and also allows them to express their confidence level.

 

Maybe...LWX can put some thought into defining each winter storm event by the specific forecast lead times for that system.  Simply put, some systems have model convergence way out in front and LWX could share with the public that it is appropriate for that particular storm to give a preliminary detailed forecast 36 hours before the start with fine tuning 24 hours out.  Before anyone balks at the idea, meteorologists can state with conviction that for a well-modeled storm, the start of the art and science governs such a lead time.  For a tricky system with poor model convergence, LWX reserves the right to state that they are watching the system closely and a draft forecast is available at 36 hours with a preliminary detailed forecast at 24 hours and fine tuning 16 hours before the start.   If those lead times are state of the art, it is integrity in science to express those lead times to the public.  People need to accept that some systems don't show their hand until 12 to 16 hours before the start, (and sometimes shorter).

[isohume]

Constructive criticism is a net plus for any gov agency in particular. NWS is one of the rare places that products are often just fully disseminated without any questioning.

That's not true at all. We receive constant feedback through various channels from our partners and customers on our products and the way we perform our mission. I agree constructive criticism is always a net plus for any agency...public or private.

[Ian]

That's not true at all. We receive constant feedback through various channels from our partners and customers on our products and the way we perform our mission. I agree constructive criticism is always a net plus for any agency...public or private.

Well, I'm sure I simplified it there.. but what I mean is more like when does NWS issue a watch or a warning that all media etc doesn't rebroadcast it?  There are plenty of times at CWG for instance when we've disagreed with it but still tell everyone about it -- not like they wouldn't hear from other sources.  Once or twice we've made a note to say we strongly disagree and somehow someone at NWS or elsewhere always seems to pop up saying "how could they." To me, NWS is a base or at least a comparative for most public forecasts whether people cop to that or not.  Not to mention that most of the forecasting tools are owned by NWS/NOAA. 

 

When your mission is at least partly defined as saving lives and property I think you by nature receive less criticism than say the IRS or any number of other federal agencies. 

 

I'm not really here to diss NWS or anything. This is an area that I find interesting given my day job and some ideas I've had in recent years about the lack of private policy research pointed to weather and the government orgs that focus on it. 

[isohume]

Well, I'm sure I simplified it there.. but what I mean is more like when does NWS issue a watch or a warning that all media etc doesn't rebroadcast it?

Number one on our mission statement is being charged with providing watches and warnings to protect life, property, and commerce. We develop trusting partnerships with media outlets to help disemminate these products. They are basically obliged to pass this information in the interest in public safety. Unless or until society feels they are not being served adequately...the NWS will be the voice of authority on svr watches/warnings.

[Ian]

Number one on our mission statement is being charged with providing watches and warnings to protect life, property, and commerce. We develop trusting partnerships with media outlets to help disemminate these products. They are basically obliged to pass this information in the interest in public safety. Unless or until society feels they are not being served adequately...the NWS will be the voice of authority on svr watches/warnings.

Yes, and I think that's how it should be myself. It would be a potential disaster to have a bunch of competing organizations releasing their own products like that. Again, not quite my point.. but I've come to expect these kind of responses as 'the usual' which is why I am glad someone like ers is here who is willing to have an open discussion.  Hopefully more of him end up in the organization. 

[winterymix]

Number one on our mission statement is being charged with providing watches and warnings to protect life, property, and commerce. We develop trusting partnerships with media outlets to help disemminate these products. They are basically obliged to pass this information in the interest in public safety. Unless or until society feels they are not being served adequately...the NWS will be the voice of authority on svr watches/warnings.

Its basically here:   http://www.nws.noaa.gov/wsom/manual/archives/NA027045.HTML

 

It is socially responsible to have only one authority that issues warnings.   BTW, does the NWS make public any verification scores for watches and warning?

[isohume]

Yes, and I think that's how it should be myself. It would be a potential disaster to have a bunch of competing organizations releasing their own products like that. Again, not quite my point.. but I've come to expect these kind of responses as 'the usual' which is why I am glad someone like ers is here who is willing to have an open discussion.  Hopefully more of him end up in the organization.

I can only give you the facts as I know them. So what was your point?

[Ian]

I can only give you the facts as I know them. So what was your point?

I made it for now.. you skipped by whatever you weren't interested in.  It's cool.

[Ian]

Its basically here:   http://www.nws.noaa.gov/wsom/manual/archives/NA027045.HTML

 

It is socially responsible to have only one authority that issues warnings.   BTW, does the NWS make public any verification scores for watches and warning?

That wasn't my point at all. I'd be strongly against any media org or the like issuing their own watches. isohume just built a strawman so he could knock it down.

[isohume]

Its basically here:   http://www.nws.noaa.gov/wsom/manual/archives/NA027045.HTML

 

It is socially responsible to have only one authority that issues warnings.   BTW, does the NWS make public any verification scores for watches and warning?

IEM used to have an unofficial app, but it doesn't work anymore. I'm not sure where else they may be available. You can always ask your local WFO for their numbers.

[WxMan1]

Yes, and I think that's how it should be myself. It would be a potential disaster to have a bunch of competing organizations releasing their own products like that. Again, not quite my point.. but I've come to expect these kind of responses as 'the usual' which is why I am glad someone like ers is here who is willing to have an open discussion.  Hopefully more of him end up in the organization. 

Good points Ian, and I agree. "Stakeholders" (I hate that term), perhaps a better one is weather partners -- we're all part of the Global Weather Industry.  Communication is the key if we as an agency want to be relevant for years to come -- relavent to the public and emergency officials if we want them to pay attention to the watches/warnings, and relavent to private weather entities if we would like them to incorporate our products as part of guidance for their forecast model(s).  As long as the NWS remains within a government agency, funded by taxpayer money, it will continue to be a non-profit organization.

 

As for the watches and warnings -- one of the main reasons why the NWS issues them is due to the liability.  It's extremely difficult to sue the federal government for an un-warned event unless negligence can be proven, which is awfully tough (and expensive) to prove.

[isohume]

I made it for now.. you skipped by whatever you weren't interested in.  It's cool.

 

Okay, I'll respond to your other points. 

 

Do you mean we get less criticism than other agencies because of our core mission? That's not the case. If anything we are under the microscope moreso on every little move we make because of our mission. There's been cases of folks almost losing their jobs because they sent out a false warning (training) after outcries from various sources. We make a mistake and it's widely known real fast and we pay the price for it.

Or did you mean CWG sometimes doesn't agree with a svr watch/warning yet disseminates the info to their customers anyway? That's probably the right thing to do, but it really is your call as a private entity.

[Ian]

Good points Ian, and I agree. "Stakeholders" (I hate that term), perhaps a better one is weather partners -- we're all part of the Global Weather Industry.  Communication is the key if we as an agency want to be relevant for years to come -- relavent to the public and emergency officials if we want them to pay attention to the watches/warnings, and relavent to private weather entities if we would like them to incorporate our products as part of guidance for their forecast model(s).  As long as the NWS remains within a government agency, funded by taxpayer money, it will continue to be a non-profit organization.

 

As for the watches and warnings -- one of the main reasons why the NWS issues them is due to the liability.  It's extremely difficult to sue the federal government for an un-warned event unless negligence can be proven, which is awfully tough (and expensive) to prove.

 

Again, I hold NWS -- local and national -- under the highest regard. SPC, WPC, NHC, and the other big national offices in particular are IMO the best in the business. I think more of the real 'issues' within the organization are in the local offices and partly because there are so many of them and there does not seem to be a cohesive plan in a lot of instances.  And, for whatever reason, the northeast corridor tends to have more issues than many.. I often wonder if it's because of the proximity to HQ etc.  If you look at the current move into social media etc., by far the best offices are in the central US or other far flung locations. But that's one area it's easy to point to the lack of a cohesive plan.. it's all very grassroots and years behind the rest of the population. 

 

I'm not looking for any wholesale changes to anything. I've been at Brookings for almost a decade so I've sort of learned to live life in a way where private researchers aid the government in its mission.  I'm willing to admit I don't know all the ins and outs of NWS.. but I do know enough that I think I have a decent grasp on it... plus I know a fair number of younger smart mets in NWS who clearly have many of the same thoughts I do about the organization.  I think a lot of that will resolve with time, but it may need more outside pushing and prodding to do so.

 

All you have to do is look at the debate after Sandy to see how criticism can get things moving in the right direction with more money flowing for model upgrades etc.  My thoughts are never really about any individual as much as processes and bureaucracy and the like. Public/private partnerships should work both ways in a perfect world. Given NWS can't openly collaborate on forecasts etc it's still mostly a NWS to everyone else type of relationship in many ways.  I think most organizations have trouble solving all their own deficiencies from within.. and I think the organization understands that based on the move to bring in more social scientists for comm help and the like.  It's just a matter of efficiency, structure, etc.. and if that's really the whole story.  

 

That is an interesting note on the liability.. I never even thought about it from that angle.  I know Accu Enterprises issues their own "warnings" to their subscribers.  I don't necessarily have a big problem with that given the way it's set up but I have long feared a company like TWC would eventually try to start issuing their own NWS-like products to the population.  Something like that would be ugly.. though perhaps your liability point will keep that from happening. 

 

I think by and large the watch/warning process is great.  That said, you can certainly tell a place like SPC is not quite as good in this area as they are in their own backyard.  Add in hedging because of where we live and all the population and you can end up with a lot of seemingly CYA products at times.  Alas, a watch is a watch I suppose.. though of course we've seen some blanket warnings that were probably unnecessary as well.  I'm not completely sure how that can be easily resolved given all the factors of the area though. It's a lot easier to hedge low on something that might be borderline in North Dakota than it is around here.

[Ian]

Okay, I'll respond to your other points. 

 

Do you mean we get less criticism than other agencies because of our core mission? That's not the case. If anything we are under the microscope moreso on every little move we make. There's been cases of folks almost losing their jobs because they sent out a false warning (training) after outcries from various sources. We make a mistake and it's widely known real fast and we pay the price for it.

Or did you mean CWG sometimes doesn't agree with a svr watch/warning yet disseminates the info to their customers anyway? That's probably the right thing to do, but it really is your call as a private entity.

I don't know if NWS truly gets less criticism but it feels like whenever there is criticism there is someone who is very defensive about it. Almost ever single time.  That is an interesting note on the warning thing. 

 

CWG would never not disseminate NWS products.. and I think that's the case for all local media.  We have taken some flak from a few folks in NWS for noting that we see less of a risk of something or etc.. mainly because they can't say the same back when they disagree.. or that's the claim. 

 

I might be wrong in my viewpoint on the defensiveness of NWS -- it could just be a handful of people.  I have seen from a number of folks close or within that NWS is a very IMBY (not sure that's the right phrase, but "it comes from within or it's useless") type of place. I dunno if there's anything wrong with that per se. Many orgs private or public are like that.  I think the weather enterprise as a whole needs to cooperate more.. and that perhaps includes NWS taking input as much as output. Yeah, some input is taken.. sure.. but it's still more out than in by a good margin I'd guess.

[isohume]

I don't know if NWS truly gets less criticism but it feels like whenever there is criticism there is someone who is very defensive about it. Almost ever single time.  That is an interesting note on the warning thing. 

 

CWG would never not disseminate NWS products.. and I think that's the case for all local media.  We have taken some flak from a few folks in NWS for noting that we see less of a risk of something or etc.. mainly because they can't say the same back when they disagree.. or that's the claim. 

 

I might be wrong in my viewpoint on the defensiveness of NWS -- it could just be a handful of people.  I have seen from a number of folks close or within that NWS is a very IMBY (not sure that's the right phrase, but "it comes from within or it's useless") type of place. I dunno if there's anything wrong with that per se. Many orgs private or public are like that.  I think the weather enterprise as a whole needs to cooperate more.. and that perhaps includes NWS taking input as much as output. Yeah, some input is taken.. sure.. but it's still more out than in by a good margin I'd guess.

 

The NWS is all about outside input and service to our customers. It's why we exist. This is why we installed NWSChat for the media/em communities and FB/Twitter for the public at large. We coordinate fcsts through chat when needed or wanted and we see how the public is responding or wanting through social media. All our products are always freely available to be used by anyone. We always solicit comments on new products or services and we are highly transparent and accessible for a fed agency.  

[Ian]

The NWS is all about outside input and service to our customers. It's why we exist. This is why we installed NWSChat for the media/em communities and FB/Twitter for the public at large. We coordinate fcsts through chat when needed or wanted and we see how the public is responding or wanting through social media. All our products are always freely available to be used by anyone. We are highly transparent and accessible for a fed agency.  

Not quite what I'm getting at again. Perhaps I need to reformulate my thoughts when it's not midnight.  I'm not sure I count reading FB messages and inviting the media to open houses etc as equal in and out.  But if it's uber transparent I guess I'm missing something here. Maybe I'm totally off base.

[isohume]

Not quite what I'm getting at again. Perhaps I need to reformulate my thoughts when it's not midnight.  I'm not sure I count reading FB messages and inviting the media to open houses etc as equal in and out.  But if it's uber transparent I guess I'm missing something here. Perhaps I'm totally off base.

 

I have to admit I have never met anyone as suspicious as you thinking the NWS is hiding some secret agenda from everyone. But then again, maybe I need some sleep too.  

[Ian]

I have to admit I have never met anyone as suspicious as you thinking the NWS is hiding some secret agenda from everyone. But then again, maybe I need some sleep too.  

lol.. pretty sure I didn't say that in the least, but you do have a way with words.

[isohume]

lol.. pretty sure I didn't say that in the least, but you do have a way with words.

 

[ers-wxman1]

I have nothing but praise for LWX. I find their forecasts to be very good most of the time. Impossible job. One thing related to some of the disco from a couple days ago...I can't seem to find a good reason to put out min/max possibilities that seem so out of range that they are almost locks as far as what actually happens. A min of 1" and a max of 16" does me little good to know. It's like well, yeah. Why even bother?

Look at it this way... If the range is 1 for min and 16 for max, we train our planners to see the uncertainty in the guidance. Obviously the goal is to narrow the goal posts in time but if they are wide that means its uncertain. Main thing is the most likely falls in between the two and offers high confidence. Guidance says this, guidance says that but here is what we are thinking. If the high end is up there it tells a planner that they need to at least prepare for that possibility even though the most likely says another thing. Feedback from hundreds of partners says they would rather prepare for the worst case anyway rather than be surprised later. Our aviation customers to the same thing! If our TAF says tempo 1/4sm prevailing 3/4 BR then they plan for LIFR to VLIFR and your flight to Canada will not take off. It's the reason airlines carry extra fuel for alternate routes! Plan for the worst. Same concept here. It's all about preparedness not what we measure in our backyards! Planners make decisions that affect millions of people! They want to know the possibilities.

[ers-wxman1]

I don't know if NWS truly gets less criticism but it feels like whenever there is criticism there is someone who is very defensive about it. Almost ever single time. That is an interesting note on the warning thing.

CWG would never not disseminate NWS products.. and I think that's the case for all local media. We have taken some flak from a few folks in NWS for noting that we see less of a risk of something or etc.. mainly because they can't say the same back when they disagree.. or that's the claim.

I might be wrong in my viewpoint on the defensiveness of NWS -- it could just be a handful of people. I have seen from a number of folks close or within that NWS is a very IMBY (not sure that's the right phrase, but "it comes from within or it's useless") type of place. I dunno if there's anything wrong with that per se. Many orgs private or public are like that. I think the weather enterprise as a whole needs to cooperate more.. and that perhaps includes NWS taking input as much as output. Yeah, some input is taken.. sure.. but it's still more out than in by a good margin I'd guess.

It's federal that's the big point and its cultural. There are things we just cannot do. It's not whether or not we agree with it or not it just is. I absolutely luv what I do for a living. I'm dedicated to the protection of life and property. I've given 14 years of my life in 4 states plus national level deployments to Hurricane Ike, BP spill, DNC, Inauguration, Sandy etc... Many 12-18 hour days and the February storm I went to state headquarters in MD and worked at 16 hours straight then crashed on a cot for 2 hours, no shower or anything and buried with 18" in 12 hours. Get right up and head back in. It's nuts but its me and I will continue to do it until I can no longer. I do it for the EM's and other county, state and federal entities. I focus on my customer and serving them.

[ers-wxman1]

Point is we do a TON, the workload is immense especially in the NE. There is rotating shift work involved, working Christmas Day, going on damage surveys while your children have a recital. It's tough stuff and we get badgered on top of that. It's not fair in many regards but its understandable as inevitably we serve the public.

[ers-wxman1]

I'm just getting home now from a shift, I will turn right around and teach training at an eoc in the morning. It's 1:30 now lol.

[Ellinwood]

To argue against Ian's point of the governments mets being defensive, I would say that they at least have a good reason to be, even if the experimental/final product appears to be flawed. Every regular product is researched, tested, implemented in experimental mode and then put into operation. This process takes months to years and is open to feedback throughout the process. Government folks have put in a lot of time and effort to make a product operational after using as much feedback as they can get, so it would at least to them seem odd that an operational product would come under such regular scrutiny. I certainly have some issues sometimes with some of the operational products, but I don't blame the government mets for defending their stuff.

I like all of the constructive feedback on the experimental products in this thread, and I think the high/low percentiles will work a lot better once the already-noted changes get implemented. I try to do my part and send feedback to the NWS, WPC, SPC, etc. through the official means when the opportunity arises, and I encourage everyone else who wants something changed to do so. If you want the government folks to put in the effort to make a better product, then you have to put in the effort to give proper feedback.

[ers-wxman1]

This is an important conversation and I am glad we are having it. Thank you for all you do. Public employees get a bad rap imo but a lot of us are incredibly dedicated and really care about improving the lives of people. I could talk about that all day...

As for the products, I have been on record before saying that I think they are good. In terms of the goalposts, the product will be even stronger once you use input from other guidance and make more tweaks by hand when necessary. I personally like probabilistic forecasting for complex events--the public is smart enough to handle it imo--and I think that serves people much better than calling for 2-4 (many including this guy ) and then having to adjust up once the storm has started. By then, it is too late. There is no saving face at that point. I would say however that the majority of events probably can be forecasted the same way they are now.

Finally, I can't speak to whether or not the NWS is transparent, but I'm sure they are more transparent about their decision making process than most agencies. I trust them (not saying that anyone here doesn't), even if I don't always agree.

The way I see it, the NWS has to hedge and be defensive a lot of the time. As much as a lot of us amateurs (putting it nicely) dumb it down, meteorology is an insanely technical field and requires exceptional attention to detail. In law, if you miss a certain feature or get too wrapped up in trends your tail will get scorched. Between the guidance (statutes/opinions), trends (what happens in certain jurisdictions), and moving parts (the actual facts/politics), there is often no right answer--especially for the complex stuff where you earn your living. There have been plenty of times where I was just about certain of something and had to hedge just because. I expect this to be the same at the NWS, and probably more to the extreme than local mets just because they are gov. Screw up too much, and you are staring at trouble (talk of funding cuts) on the Hill. That takes a toll. You have to be defensive about being defensive.

Like I used to say in my sig--if you want to be in a profession where too many people don't trust you for no good reason and many think you're wrong because they know next to nothing about what you actually do, this is the field for you .

I appreciate you and your posts! Good stuff.

Issuing actual warnings, products etc takes a lot of analysis, coordination both internal and external. It is FAR from easy! Believe me. It is easy for ANYONE looking from the outside in to what should have been done or what they would have done. Ultimately we are responsible for 10 million people, and a myriad of customers. All it takes to cause problems is an inch of snow on cold pavement at the right time to cause problems so sometimes headlines go out for impacts and timing vs more so on the amount. We learn these impacts by building trust with our partners. I've been involved in this process for many years. Also it is important to remember that we also serve a LOT of different customers... Fire, aviation, marine, hydro, outreach, climate, air quality, upper air, GIS, emergency response and more!

[ers-wxman1]

The NWS is all about outside input and service to our customers. It's why we exist. This is why we installed NWSChat for the media/em communities and FB/Twitter for the public at large. We coordinate fcsts through chat when needed or wanted and we see how the public is responding or wanting through social media. All our products are always freely available to be used by anyone. We always solicit comments on new products or services and we are highly transparent and accessible for a fed agency.

Absolutely true. We are all about feedback but I have realized over the years that we simply cannot be all things to all people. It is just not possible. Can't please everyone no matter what we do.

[MN Transplant]

ers - I just left my feedback on the NWS site.  One thought was to take the table of "chance of snow accumulation" and turn it into a graph.  So, for the last storm, you would probably have something like below (that isn't drawn by a 3 year old).  I think that would communicate exactly what you are going for, that the best chances are 6-9", with lower probabilities, but still a chance at higher or lower values.  You could do a point/click on a map, or on certain dots on a map, to make the graph.

 

[PivotPoint]

ers - I just left my feedback on the NWS site.  One thought was to take the table of "chance of snow accumulation" and turn it into a graph.  So, for the last storm, you would probably have something like below (that isn't drawn by a 3 year old).  I think that would communicate exactly what you are going for, that the best chances are 6-9", with lower probabilities, but still a chance at higher or lower values.  You could do a point/click on a map, or on certain dots on a map, to make the graph.

 

Untitled.png

Coming from a novice's perspective -- I really like this idea/graphic. It conveys to me the likelihood of the various scenarios in one convenient and digestible snapshot.

[ers-wxman1]

ers - I just left my feedback on the NWS site. One thought was to take the table of "chance of snow accumulation" and turn it into a graph. So, for the last storm, you would probably have something like below (that isn't drawn by a 3 year old). I think that would communicate exactly what you are going for, that the best chances are 6-9", with lower probabilities, but still a chance at higher or lower values. You could do a point/click on a map, or on certain dots on a map, to make the graph.

Untitled.png

I like this as well... Now I see what you guys are talking about as far as having us show the min and max as lower probabilities on a curve. Our original thought was to use the product that snow would be greater then a certain threshold to give our confidence in the higher numbers.