Data ingestion discussion

[PhineasC]

Pretty sure the new data on the s/w that made it into the Euro didn't get into the 12Z NAM, which means the 18Z doesn't have it either, AFAIK.

[IsentropicLift]

Anyway to verify it? This 18Z run is a serious mess.

Reliable sources mentioned on the 12z thread that the new data wasn't available for the 12z NAM

[ohleary]

Reliable sources mentioned on the 12z thread that the new data wasn't available for the 12z NAM

I missed that post, what kind of "new data" was missing from the 12z NAM?

[Ji]

I missed that post, what kind of "new data" was missing from the 12z NAM?

the data that would make it snow over I95:)

[Sundog]

Pretty sure the new data on the s/w that made it into the Euro didn't get into the 12Z NAM, which means the 18Z doesn't have it either, AFAIK.

What are the cutoff times for receiving new data for the 12z runs of the Euro and NAM/GFS?

[PhineasC]

I missed that post, what kind of "new data" was missing from the 12z NAM?

There is a theory that the data on the shortwave in question was ingested into 12Z Euro but not NAM or GFS.

[stormtracker]

There is a theory that the data on the shortwave in question was ingested into 12Z Euro but not NAM or GFS.

But how is that? All models use the same set of data I thought?

[PhineasC]

But how is that? All models use the same set of data I thought?

I am not sure. I though the beef all the time was that the Euro had better data or something?

[ORH_wxman]

But how is that? All models use the same set of data I thought?

They do for raob data, but other stuff like sfc obs and satellite it can be different, I think the later start time of the Euro gives it an advantage on those latter data. But I'm not 100% sure.

[ohleary]

What are the cutoff times for receiving new data for the 12z runs of the Euro and NAM/GFS?

Don't know about the Euro but the 12Z NAM dump runs at 1315Z, for the other cycles add/subtract 6 hours. The GFS 12Z dump is at 1446Z, again do the math for the other cycles.

[PhineasC]

FINAL 12Z MDL EVALUATION INCLUDING THE ECMWF

ANY INITIALIZATION ERRORS APPEAR MINOR AND DO NOT SEEM TO

SIGNIFICANTLY INFLUENCE THE FORECAST

http://www.hpc.ncep.noaa.gov/discussions/pmdhmd.html

That is not what we are talking about.

[baroclinic_instability]

But how is that? All models use the same set of data I thought?

No not true. This is where EURO/UK and other global models has a key advantage with their 4DVAR data assimilation system which statistically analyzes the data differently and considers time dependence. Also, the NAM with its early runtime requires that its boundary conditions be initialized by the previous GFS run plus it has an earlier data cutoff. This is in some way why the NAM can be garbage in fast westerly flow...and why it is usually pretty worthless for the West coast.

DTK or anyone else with more info can correct me because I am not a total expert.

[ohleary]

That is not what we are talking about.

All I can see as far as raobs is a few Alaska stations and one from Cuba not making it into the NAM analysis.

[PhineasC]

All I can see as far as raobs is a few Alaska stations and one from Cuba not making it into the NAM analysis.

The question is whether or not the Euro had "more data" on the shortwave that was coming ashore just as the NAM started running.

[LocoAko]

The question is whether or not the Euro had "more data" on the shortwave that was coming ashore just as the NAM started running.

What time does the Euro ingest its data?

[PhineasC]

What time does the Euro ingest its data?

I wish I knew. I am hoping some more smart red-taggers will find this thread.

[A-L-E-X]

No not true.  This is where EURO/UK and other global models has a key advantage with their 4DVAR data assimilation system which statistically analyzes the data differently and considers time dependence.  Also, the NAM with its early runtime requires that its boundary conditions be initialized by the previous GFS run plus it has an earlier data cutoff.  This is in some way why the NAM can be garbage in fast westerly flow...and why it is usually pretty worthless for the West coast.  

DTK or anyone else with more info can correct me because I am not a total expert.

Why do we put out an inferior product?  It's high time we got with the program and used higher resolution and 4DVAR data assimilation scheme.  It's embarassing that we let others take the lead. 

[CoastalWx]

I'm pretty sure any difference is related to the initialization scheme. 12z RAOBS all get incorporated into the models. Will might have a point about stuff like obs and satellite imagery getting ingested, but I think part of the late start time to the Euro may be due to the extra time needed with the initialization scheme?

[mitchnick]

looks to me that the short wave in question just made in on the beach in S. CA late this afternoon and is now over AZ per this sat pic (or is it still off shore?)

if I'm correct with the ID of the short wave, none of the 12Z model runs had data specific to the short wave while over land

http://wxweb.meteost...tellite/SPN/WV/

[am19psu]

Why do we put out an inferior product?  It's high time we got with the program and used higher resolution and 4DVAR data assimilation scheme.  It's embarassing that we let others take the lead. 

It's not that easy. I'll let dtk/wxbadger finish my post.

[am19psu]

looks to me that the short wave in question just made in on the beach in S. CA late this afternoon and is now over AZ per this sat pic (or is it still off shore?)

if I'm correct with the ID of the short wave, none of the 12Z model runs had data specific to the short wave while over land

http://wxweb.meteost...tellite/SPN/WV/

I specifically looked at 715a and it hadn't made landfall. I don't actually know how that affects the initialization though.

[Amped]

The real shortwave splits into Oregon tonight.

[UNCCmetgrad]

Why do we put out an inferior product? It's high time we got with the program and used higher resolution and 4DVAR data assimilation scheme. It's embarassing that we let others take the lead.

Seems like I read in another thread or even maybe another board that we simply do not have the computing power to run 4DVAR assimilation currently. It would take something on the order of twice as long to run the models with this scheme.

[dtk]

Seems like I read in another thread or even maybe another board that we simply do not have the computing power to run 4DVAR assimilation currently. It would take something on the order of twice as long to run the models with this scheme.

I didn't see this in the main thread, but I started replying to the 18z data nonsense, as well as 4DVAR questions in a regional thread. Before I dig those comments up, I'll reply to the 4DVAR thing: given the same resolution, it would cost at LEAST 10x (probably more) to run the analysis. Also keep in mind that the NCEP products are distributed earlier than the other centers....and that is not going to change. We're going to have to be creative when it comes to improving things given the strict cut of times and lack of resources.

This is just the computing issue, and doesn't even take into account the manpower needed to finish developing the thing.

[A-L-E-X]

I didn't see this in the main thread, but I started replying to the 18z data nonsense, as well as 4DVAR questions in a regional thread.  Before I dig those comments up, I'll reply to the 4DVAR thing:  given the same resolution, it would cost at LEAST 10x (probably more) to run the analysis.  Also keep in mind that the NCEP products are distributed earlier than the other centers....and that is not going to change.  We're going to have to be creative when it comes to improving things given the strict cut of times and lack of resources.

This is just the computing issue, and doesn't even take into account the manpower needed to finish developing the thing.

Thanks for the replies in the regional thread, it was very informative.  How do the euros defray the costs?  Do they have less cost for some reason, or is it because you have to pay to access their site?  Hopefully some of the advances in parallel processing and one day, quantum computing, will take care of the processing end of things.....

[dtk]

Thanks for the replies in the regional thread, it was very informative. How do the euros defray the costs? Do they have less cost for some reason, or is it because you have to pay to access their site? Hopefully some of the advances in parallel processing and one day, quantum computing, will take care of the processing end of things.....

Well, first of all, they invested in the manpower to develop the thing a LONG time ago (mid 90s). The fact also remains that they are a much more single-focused organization than we are at NCEP....and I don't mean that as a good or bad thing. The European Center was formed as a consortium with a single focus, and their forecasts today are a result of that. They don't have the wide variety of applications and products that we at NCEP put out, which stretches us thin both in terms of human resources and computing.

However, other centers have developed and implemented 4DVAR with much fewer resources (UK Met office, Canada, etc.), and I'm not going to make excuses for anyone. All I can say is that we are working hard, with few resources, to improve the system as best we can. We have several things on the table in terms of data assimilation, including hybrid ensemble-variation DA (both 3d and 4d) as well as a prototype 4DVAR system. Unfortunately, I can't speak specifically to exact timelines for implementation, but they hybrid stuff is much more mature than the 4DVAR code at this point.

[A-L-E-X]

Well, first of all, they invested in the manpower to develop the thing a LONG time ago (mid 90s).  The fact also remains that they are a much more single-focused organization than we are at NCEP....and I don't mean that as a good or bad thing.  The European Center was formed as a consortium with a single focus, and their forecasts today are a result of that.  They don't have the wide variety of applications and products that we at NCEP put out, which stretches us thin both in terms of human resources and computing.

However, other centers have developed and implemented 4DVAR with much fewer resources (UK Met office, Canada, etc.), and I'm not going to make excuses for anyone.  All I can say is that we are working hard, with few resources, to improve the system as best we can.  We have several things on the table in terms of data assimilation, including hybrid ensemble-variation DA (both 3d and 4d) as well as a prototype 4DVAR system.  Unfortunately, I can't speak specifically to exact timelines for implementation, but they hybrid stuff is much more mature than the 4DVAR code at this point.

Fair enough-- I know you guys are working hard and doing the best with the resources you have.  Based on what you said earlier, I think it's pretty safe to say we'll have this implemented in 5 years or so.  I dont take the European Center's single focus as a positive or negative either-- the fact is that when you have so many areas you need to allocate your resources to, you get stretched thin in some areas, but it's really nice to hear that there is progress being made.

[A-L-E-X]

 Wow mid 90s, that's hard to imagine-- in the private sector we were running some (what would be considered today) archaic computer architecture  lol. 

[Dr No]

Is this why the GFS seems oftern to "trend' toward the Euro within 12-24 hours (especially in a fast flow)? The 4DVAR assimilation allows for improved accuracy a with respect to apecific atmospheric feature, that otherwise the standard initialization scheme wouldn't pick up for several runs yet?

[pcbjr]

Don't know if this is the right place to post this, but the run-to-run comparisons of the GFS are bipolar:

http://www.wxforecaster.com/runtorun/all.html