GFS Model Data Extraction

[weatherwiz]

Not necessarily myself but a few people I'm associated with who are worlds better at computer programming (and have the brain for it unlike me) have dabbled in linux and other ways to extract model data and generate graphics and such.  I know there are these links:

https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/global-forcast-system-gfs

http://www.nco.ncep.noaa.gov/pmb/products/gfs/

Does anyone know any other sources where you can grab model data (more specifically for the GFS) and easily choose which parameters and data you specifically want?

 

[MegaMike]

On 8/21/2018 at 5:04 PM, weatherwiz said:

Not necessarily myself but a few people I'm associated with who are worlds better at computer programming (and have the brain for it unlike me) have dabbled in linux and other ways to extract model data and generate graphics and such.  I know there are these links:

https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/global-forcast-system-gfs

http://www.nco.ncep.noaa.gov/pmb/products/gfs/

Does anyone know any other sources where you can grab model data (more specifically for the GFS) and easily choose which parameters and data you specifically want?

 

I've been working on extracting operational variables myself for modeling purposes. I believe the first url ('https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/global-forcast-system-gfs') archives only analysis data (is this what you want?). Regardless, the second url ('http://www.nco.ncep.noaa.gov/pmb/products/gfs/') stores/archives both analysis and operational (forecast/predicted) data.

When you write "choose which parameters and data," I'm assuming you mean NWP variables such as temperature, wind speed, etc... That being stated;

Typically, NWP data is stored by grib files. Very rarely are these grib files stored for specific variable such as 'temperature' or 'wind speed' separately. If they are, however, all you'd need to do is connect to a particular server then download a variable-specific grib file for 'x' different files. You can then convert that file into a .txt/.csv/etc... file afterwards in order to manipulate the data as you wish. What I think you'll need to do, however, is download a grib file (which contains all variables) for one timestamp from a server, then you'll have to extract a specific variable from a grib file itself. Therefore you'll need to a) connect to a server to obtain grib files likely by ftp b) extract a variable from the grib file and c) store the extracted variable as a .txt/.csv/etc... file to manipulate its data.

This is how I'm planning on accomplishing this task for my project:

a) Firstly, I'd recommend this server ('ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/'). The server hosts multiple NWP models (analysis/operational) in a user friendly and organized manner. It'll be easy to cd and loop through certain directories in order to obtain multiple NWP models for multiple variables == It's convenient . You can use this link 'https://www.weather.gov/mdl/ndfd_data_grid' to determine the abbreviations and the formatting of the directories.

b) As for extracting certain variables, I'd recommend NCAR Command Language (NCL). The program can be run on Linux (recommended) or a Linux Bash Shell via Windows 10. It takes only a couple lines of code to extract data using this program.

Alternatively, you can use MATLAB by utilizing nctoolbox or MeteoLab (likely more functions than this). Both functions accomplish the same procedure in a slightly more intensive way.

If you want a more interactive program and have the patience of a Jets or Browns fan (~1/2 a century worth of patience), you can use the NOAA Weather and Climate Toolkit. This program is 10x more tedious and longer than simply using NCL or MATLAB. You'd have to download the grib files manually (or automatically by a loop using a program of your choice) before being able to process the data through the Toolkit. I first used this program when I was an undergraduate. I definitely wouldn't recommend this program now, unfortunately. I'd say it's useful for case studies of one or two events.

- You can also use Python and probably R to obtain, extract, and manipulate grib files, as well. My knowledge is very limited for these programs though.

I hope this helps and I welcome anyone else who has other methods, ideas, and or corrections!

 

[weatherwiz]

This is incredible and extremely helpful information! Thank you!!! 

[MegaMike]

1 hour ago, weatherwiz said:

This is incredible and extremely helpful information! Thank you!!! 

No problem, Wiz! I'm glad to help. 

I may keep my process in this thread for the sake of reference.

[ohleary]

On 8/21/2018 at 5:04 PM, weatherwiz said:

Not necessarily myself but a few people I'm associated with who are worlds better at computer programming (and have the brain for it unlike me) have dabbled in linux and other ways to extract model data and generate graphics and such.  I know there are these links:

https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/global-forcast-system-gfs

http://www.nco.ncep.noaa.gov/pmb/products/gfs/

Does anyone know any other sources where you can grab model data (more specifically for the GFS) and easily choose which parameters and data you specifically want?

 

Wiz,

NOMADS is exactly what you're looking for. The grib filter is what you use and you can cut any hours, paramaters and even specify your domain out of any of the models. Plus it can be scripted. Lemme know if you have questions.

 

http://nomads.ncep.noaa.gov/

 

 

[SN_Lover]

It's a pain to display on GRADS. Me and my research mentor had to contact NCEP. You have to do something with a file to get it to display right.

[weatherwiz]

On 9/7/2018 at 6:39 PM, ohleary said:

Wiz,

NOMADS is exactly what you're looking for. The grib filter is what you use and you can cut any hours, paramaters and even specify your domain out of any of the models. Plus it can be scripted. Lemme know if you have questions.

 

http://nomads.ncep.noaa.gov/

 

 

This is fantastic! Thank you!!