Clinch Leatherwood Posted February 19, 2012 Share Posted February 19, 2012 In retrospect based on what we know at this stage, and based on what we have seen at the Euro at 228+ hours would we say that the G4 data helped? The Euro obviously did not have this data at D7 and still did pretty well from there on in. But the GFS actually seemed to be more erratic after data ingestion. Curious as to the opinions. Personally I'm not sure this data helped the NCEP models much and I'd hate to think if it did help what the solutions would have been like otherwise. Great writeup Phil. Link to comment Share on other sites More sharing options...
Srain Posted February 19, 2012 Share Posted February 19, 2012 In retrospect based on what we know at this stage, and based on what we have seen at the Euro at 228+ hours would we say that the G4 data helped? The Euro obviously did not have this data at D7 and still did pretty well from there on in. But the GFS actually seemed to be more erratic after data ingestion. Curious as to the opinions. Personally I'm not sure this data helped the NCEP models much and I'd hate to think if it did help what the solutions would have been like otherwise. Great writeup Phil. While some may think that the G-IV data was a waste of NOAA funds and offered no help to the almighty Euro, I suggest that the data did provide meaningful help for those of us beyond the I-95 corridor... Link to comment Share on other sites More sharing options...
Clinch Leatherwood Posted February 19, 2012 Share Posted February 19, 2012 Srain...but the Euro had kind of nailed it at 228 and really everywhere under that. Granted it did pin it better after the data. What I'm more interested in figuring out is whether it helped the NCEP models. Given their variability I'd argue no, partly because one of the problematic s/w's for the NCEP models was coming from south central canada. Link to comment Share on other sites More sharing options...
dtk Posted February 19, 2012 Share Posted February 19, 2012 Srain...but the Euro had kind of nailed it at 228 and really everywhere under that. Granted it did pin it better after the data. What I'm more interested in figuring out is whether it helped the NCEP models. Given their variability I'd argue no, partly because one of the problematic s/w's for the NCEP models was coming from south central canada. This will be impossible to know until someone can go back and run OSEs (and we simply don't have the resources to do it). These things always have to be evaluated on a per-case basis. I started to go on about this in a regional thread the other day, and plan to do a longer write-up about this when I get a chance. One thing about DA is that it is really statistical (and/or probabilistic) in nature. We are combining a lot of information, and every single bit of this information has errors associated with it (yes, even the observations). The nice thing about radiosondes/dropsondes is that they are more direct measurements of the model fields, so the representativeness error (errors associated with interpolation, transforming the observation into model space, etc.) isn't as extreme as most other observations types. The idea behind DA is to come up with a best estimate of the state, one that minimizes the errors associated with each bit of information. For NWP, things are even more complicated, because the best analysis (i.e. the state that truly minimizes the errors) is NOT the best initial condition. This is because of model error/bias. The one thing I want to try to hit home....because of all aforementioned, not all observations improve forecasts (or reduce forecast error). In fact, in terms of counts....only a small majority of observations (say between 51-60%) actually reduce forecast error (and yes, that implies that for any given cycle, as much as 49% of the observations actually can have a "negative" impact on the forecast). That is not to say that those observations are bad, just that they can cause things to happen through the assimilation that can act to increase forecast error (through representativeness or sensor errors, through a mismatch with something in the model climatology, through over/under extrapolation of information through the assimilation process, by incorrectly projecting the multi-variate information with in the assimilation process). I'm going to show a figure, with one example, to highlight sort of what I'm talking about. We (the community, not NCEP, yet) actually have a means of estimating the actual impact every observation has in reducing short term forecast error (either through adjoint or ensemble based methods). Here is an example of the impact of assimilating dropsonde observations into the NASA General Circulation Model (they use the NCEP DA system) for the past month (figure is updated quasi-regularly): Basically, anything negative means that the observations reduced the forecast error (i.e. had a positive impact on the system). Any cases for which the number is positive, that subset of observations actually increased the short term forecast error. First, note that the impact isn't always positive for dropsondes (in their system, for this estimation/metric). Again, it is not to say that the observations are actually bad for the cases where the impact was negative. Secondly, note how variable the amplitude of the impact can be (this is related to how the information projects on to growing/decaying modes, as well as the error growth characteristics of each case). Lastly, even though "negative" information is regularly brought into the system....the net impact when summed up is actually positive as expected. Here it is broken down for the NASA system by observation type: The relative amplitude and ranking is going to be different for each operational center, so you have to take the information for what it's worth. Also, the method is only an estimate of the impact (and doesn't require running actual data denial experiments). Link to comment Share on other sites More sharing options...
phil882 Posted February 19, 2012 Author Share Posted February 19, 2012 This will be impossible to know until someone can go back and run OSEs (and we simply don't have the resources to do it). These things always have to be evaluated on a per-case basis. I started to go on about this in a regional thread the other day, and plan to do a longer write-up about this when I get a chance. One thing about DA is that it is really statistical (and/or probabilistic) in nature. We are combining a lot of information, and every single bit of this information has errors associated with it (yes, even the observations). The nice thing about radiosondes/dropsondes is that they are more direct measurements of the model fields, so the representativeness error (errors associated with interpolation, transforming the observation into model space, etc.) isn't as extreme as most other observations types. The idea behind DA is to come up with a best estimate of the state, one that minimizes the errors associated with each bit of information. For NWP, things are even more complicated, because the best analysis (i.e. the state that truly minimizes the errors) is NOT the best initial condition. This is because of model error/bias. The one thing I want to try to hit home....because of all aforementioned, not all observations improve forecasts (or reduce forecast error). In fact, in terms of counts....only a small majority of observations (say between 51-60%) actually reduce forecast error (and yes, that implies that for any given cycle, as much as 49% of the observations actually can have a "negative" impact on the forecast). That is not to say that those observations are bad, just that they can cause things to happen through the assimilation that can act to increase forecast error (through representativeness or sensor errors, through a mismatch with something in the model climatology, through over/under extrapolation of information through the assimilation process, by incorrectly projecting the multi-variate information with in the assimilation process). I'm going to show a figure, with one example, to highlight sort of what I'm talking about. We (the community, not NCEP, yet) actually have a means of estimating the actual impact every observation has in reducing short term forecast error (either through adjoint or ensemble based methods). Here is an example of the impact of assimilating dropsonde observations into the NASA General Circulation Model (they use the NCEP DA system) for the past month (figure is updated quasi-regularly): Basically, anything negative means that the observations reduced the forecast error (i.e. had a positive impact on the system). Any cases for which the number is positive, that subset of observations actually increased the short term forecast error. First, note that the impact isn't always positive for dropsondes (in their system, for this estimation/metric). Again, it is not to say that the observations are actually bad for the cases where the impact was negative. Secondly, note how variable the amplitude of the impact can be (this is related to how the information projects on to growing/decaying modes, as well as the error growth characteristics of each case). Lastly, even though "negative" information is regularly brought into the system....the net impact when summed up is actually positive as expected. Here it is broken down for the NASA system by observation type: The relative amplitude and ranking is going to be different for each operational center, so you have to take the information for what it's worth. Also, the method is only an estimate of the impact (and doesn't require running actual data denial experiments). Excellent Information... really interesting to see that dropsondes had quite a positive impact in some periods (especially between January 29th - Feb 5) while in other periods the impact was negligible. It is good to see that the vast majority of the time, the impact is positive as one would expect Do you have a source link to the figures you have provided above, or is this information restricted to the public? Link to comment Share on other sites More sharing options...
dtk Posted February 19, 2012 Share Posted February 19, 2012 Excellent Information... really interesting to see that dropsondes had quite a positive impact in some periods (especially between January 29th - Feb 5) while in other periods the impact was negligible. It is good to see that the vast majority of the time, the impact is positive as one would expect Do you have a source link to the figures you have provided above, or is this information restricted to the public? The NASA GMAO site: http://gmao.gsfc.nasa.gov/forecasts/systems/fp/obs_impact/ NAVY: http://www.nrlmry.navy.mil/obsens/fnmoc/obsens_main_od.html I think that ECMWF has an internal monitoring package. An ensemble-based package is in development for NCEP. Lastly, here is a figure from the GMAO that shows the fraction of observations that are beneficial (getting to my previous point that only a small majority of observations contribute positively to a given system....I've seen fractions as high as 60% for some systems, but the point remains): Link to comment Share on other sites More sharing options...
baroclinic_instability Posted February 19, 2012 Share Posted February 19, 2012 Totally curious, you mentioned representativeness, but I was also wondering if issues arise regarding uneven distribution of data (i.e., a lone dropsonde into a Pacific cyclone, etc.)? What types of schemes are in place now to deal with extreme data distribution differences within a various domain? Link to comment Share on other sites More sharing options...
Rainshadow Posted February 19, 2012 Share Posted February 19, 2012 The NASA GMAO site: http://gmao.gsfc.nas.../fp/obs_impact/ NAVY: http://www.nrlmry.na...ns_main_od.html I think that ECMWF has an internal monitoring package. An ensemble-based package is in development for NCEP. Lastly, here is a figure from the GMAO that shows the fraction of observations that are beneficial (getting to my previous point that only a small majority of observations contribute positively to a given system....I've seen fractions as high as 60% for some systems, but the point remains): Are Dropsonde missions routinely scheduled? The ones the week of Jan 29th I believe were because at one time it looked like there was going to be a major event on Super Bowl weekend. The Euro did a great job with this event, but up to this one it didn't appear to me to be any better (or worse) than most of the other medium range models beyond day4 this winter. I know the previous sentence is subjective and is based on the PHI CWA (Well Walt Drag agrees with me, so that must count for something.). The GFS overphased solution on 15/00z, once something that off (in retrospect always easier) gets into the modeling, does it take a while for it to be corrected? It seemed every sounding run after that it did trend closer to what was in the end the Euro solution. Link to comment Share on other sites More sharing options...
dtk Posted February 19, 2012 Share Posted February 19, 2012 Totally curious, you mentioned representativeness, but I was also wondering if issues arise regarding uneven distribution of data (i.e., a lone dropsonde into a Pacific cyclone, etc.)? What types of schemes are in place now to deal with extreme data distribution differences within a various domain? I don't really think this is as much of an issue, per say, but does yield an analysis/IC that is of variable quality for different parts of the globe. How the observational information is used to correct the guess (short term model forecast) is highly dependent on the background error covariance information. This is one of the reasons why ensemble methods have become so popular within the DA community.....as they allow you to sample/estimate through Monte Carlo type methods where you need to spread observational informational out over larger spatial distances or with larger amplitude (in addition to inherit model-driven multivariate aspects of the correlations). In the case of 4DVAR, this can be somewhat accounted for implicitly through the linearized model that is used as part of the update. There are ways to account for this within 3DVAR, but it's more difficult. Ensemble methods are imperfect, however, due to the inherit issues related to sampling a huge space (the background error covariance for a modern day NWP models is huge, ~10^7 or greater....state space squared.....and we're trying to sample this with only ~50-100 ensemble members). This is why various places (UKMet, NCEP, CMC, others) are investing in hybrid type technologies where you combine ensemble-based information with the variational (with a full rank, static background error estimate) framework. As an example, there have been projects that utilized ensemble-based methods to construct historical reanalysis....from a limited subset of surface pressure data only (and I'm talking full 3D analyses). That's about as extreme as it gets. Link to comment Share on other sites More sharing options...
dtk Posted February 19, 2012 Share Posted February 19, 2012 Are Dropsonde missions routinely scheduled? The ones the week of Jan 29th I believe were because at one time it looked like there was going to be a major event on Super Bowl weekend. The Euro did a great job with this event, but up to this one it didn't appear to me to be any better (or worse) than most of the other medium range models beyond day4 this winter. I know the previous sentence is subjective and is based on the PHI CWA (Well Walt Drag agrees with me, so that must count for something.). The GFS overphased solution on 15/00z, once something that off (in retrospect always easier) gets into the modeling, does it take a while for it to be corrected? It seemed every sounding run after that it did trend closer to what was in the end the Euro solution. The big ones for the US are of course the WSR and Hurricane type missions. Any other observations labeled "recon" could be from missions of opportunity, international field experiments/missions, or who knows (I'd have to dig on a case by case basis). We pretty much share all observations with each other these days (even experimental). In terms of your last question, that's hard to say because you are also talking about the model forecast grabbing onto a solution. Models have errors to, and even if given perfect initial conditions could be prone to grabbing onto certain types of solutions. It's possible that something finds its way into the analysis/short-term forecast that takes a while for the assimilation to completely get rid of. Link to comment Share on other sites More sharing options...
bluewave Posted February 19, 2012 Share Posted February 19, 2012 The big ones for the US are of course the WSR and Hurricane type missions. Any other observations labeled "recon" could be from missions of opportunity, international field experiments/missions, or who knows (I'd have to dig on a case by case basis). We pretty much share all observations with each other these days (even experimental). In terms of your last question, that's hard to say because you are also talking about the model forecast grabbing onto a solution. Models have errors to, and even if given perfect initial conditions could be prone to grabbing onto certain types of solutions. It's possible that something finds its way into the analysis/short-term forecast that takes a while for the assimilation to completely get rid of. Was there an upgrade applied to the OP GFS this week or only just the ensembles? http://www.nws.noaa....ncrease_aab.htm Link to comment Share on other sites More sharing options...
dtk Posted February 19, 2012 Share Posted February 19, 2012 Was there an upgrade applied to the OP GFS this week or only just the ensembles? http://www.nws.noaa....ncrease_aab.htm The OP GFS remained unchanged (there is a GFS/GDAS upgrade coming, likely in May....but it's mostly on the data assimilation component and not the model itself). The version of the model used in the GEFS was finally upgraded to use the same options that are used in the current OP GFS. Link to comment Share on other sites More sharing options...
bluewave Posted February 19, 2012 Share Posted February 19, 2012 The OP GFS remained unchanged (there is a GFS/GDAS upgrade coming, likely in May....but it's mostly on the data assimilation component and not the model itself). The version of the model used in the GEFS was finally upgraded to use the same options that are used in the current OP GFS. Thanks for the information. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.