NAVGEM Model vs others

[jgf]

Is NAVGEM much different from the NOGAPS it replaces?

 

If you are interested in MSLP and 10m wind over the NW Atlantic Ocean, would there be any reason, in general, to prefer NAVGEM over, say, GFS?

 

How about in comparison with the Canadian GEM?

 

I realize that on any particular day, and for any particular event, one model may be preferred over another.., but I am asking about the general situation.

[Quincy]

Is NAVGEM much different from the NOGAPS it replaces?

 

If you are interested in MSLP and 10m wind over the NW Atlantic Ocean, would there be any reason, in general, to prefer NAVGEM over, say, GFS?

 

How about in comparison with the Canadian GEM?

 

I realize that on any particular day, and for any particular event, one model may be preferred over another.., but I am asking about the general situation.

I may not have information on the NOGAPS switch, but here's some information you might find useful. FNO is, I'm assuming, the current NOGAPS.

 

If you're wondering about model verification, the CMC has consistently ranked somewhere between the Euro (best verification) and the GFS (poorer verification) with large scale patterns over the northern hemisphere. In terms of wind, in the small sample I've reviewed, the CMC verifies near or slowly below the GFS.

 

It also depends on what time frame you're looking at. Attached are two maps, both for 5-day forecasts. The first is for MSLP and the second is for 850mb wind.

[jgf]

Thanks.

 

my time span is 0hrs out to 96 or 120, area of interest is the western North Atlantic.

 

I am mostly interested in models for which I can get actual GRIB surface wind forecasts, so EURO is out.

 

Global models I can get are GFS, CMC Global, NAVGEM.

 

I can also get COAMPS, NAM, and sometimes various RAMS or WRF models, but i assume that even though the domains of some of these models extend pretty far offshore, they are best used for terrestrial or coastal/nearshore winds.

 

I get a little bit confused over the terminology for CMC models - I assume that because those plots are comparing global models, the CMC model in question is the GDPS or Global Deterministic Prediction System - which is available in 25 and 66km resolutions. As I'm sure you know, they also have regional models at higher resolutions.

 

Anyway, with respect to the plots, FNO is the NOGAPS (edit - i meant to say NAVGEM), and it is occasionally better than GFS for PMSL and 850mb wind.

 

what I usually do, is read the area forecast discussion, the model discussion, or the Offshore Forecast discussion, to see what model they are preferring, but they are typically focusing on GFS vs EURO vs NAM, and as I said, I can't get EURO, and I am not sure about using NAM offshore.

[ohleary]

Is NAVGEM much different from the NOGAPS it replaces?

 

If you are interested in MSLP and 10m wind over the NW Atlantic Ocean, would there be any reason, in general, to prefer NAVGEM over, say, GFS?

 

How about in comparison with the Canadian GEM?

 

I realize that on any particular day, and for any particular event, one model may be preferred over another.., but I am asking about the general situation.

 

In response to your first question, looking at the changes I'd say it's significantly different...with a new core, several dynamic and several physical scheme changes, in addition to a resolution increase.  In pre-implementation verification, it outperformed the NOGAPS. 

 

As far as your second question goes, I'd say no, but you can glean the info you need from the chart Quincy posted.

[Quincy]

As far as your second question goes, I'd say no, but you can glean the info you need from the chart Quincy posted.

Here's a link for all the verification stats. You can choose your location and variable(s).

http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

[jgf]

so this seems interesting...

 

this northern hemisphere wind speed bias at 1000mb - so a decent proxy for surface winds

 

seems like NAVGEM has been about as good as Euro over the last month - and both were better than GFS

[dtk]

so this seems interesting...

 

this northern hemisphere wind speed bias at 1000mb - so a decent proxy for surface winds

 

seems like NAVGEM has been about as good as Euro over the last month - and both were better than GFS

 

You have to be very careful with analysis-based verification metrics, particularly for winds and especially in the tropics.  The plot you are showing is not bias relative to "truth" (unknown), but bias relative to its own model's analysis.  If something is done at the analysis stage that the model disagrees with, it will manifest itself as bias in a metric as is shown.  Also, the plot is only showing the speed bias.  I'm not sure this is a particularly meaningful metric if you want to know how "useful" something is.  These numbers are pretty small compared to the actual error (by component, vector, or in comparison to observations).

 

Having said that, RMSE can be even more problematic.  For winds, you can get great verification statistics through post-processing (i.e. cheating), by doing things like smoothing the analysis for example.  I encourage you to look at the following presentation for some more information on the topic:

http://www.emc.ncep.noaa.gov/gmb/wx24fy/doc/RMSE_decomposition.pdf

 

So then you might think to look at anomalies instead....except our climatologies are not great.  How about consensus analyses?  Not all analyses are created equal. 

 

To make a long story short, verification is messy and tricky.  You need to combine analysis-based verification with a comparison to observations, other models/analyses, and qualitative (forecaster) evaluation.

 

Although NAVGEM is a significant improvement over NOGAPS for the reasons cited by ohleary (and improved data assimilation), it still lags quite a bit behind the other operational globals.

[jgf]

 

 

Also, the plot is only showing the speed bias.  I'm not sure this is a particularly meaningful metric if you want to know how "useful" something is. 

 

wind speed is extremely useful to me.

 

i also looked at the zonal and meridional bias, as direction is also very important.

 

MSLP is interesting.., but in the end i don't actually need it - what i need is wind speed and direction

 

but i will look at the RMSE 

[dtk]

wind speed is extremely useful to me.

 

i also looked at the zonal and meridional bias, as direction is also very important.

 

MSLP is interesting.., but in the end i don't actually need it - what i need is wind speed and direction

 

but i will look at the RMSE 

 

Sorry, I agree that wind speed is important in general, and of course we consider it as part of our standard verification.   What I meant to say is that bias computed using self-analysis may not be entirely useful, as the analysis itself is likely to have similar biases given that the analysis procedure is incremental.  It is much more useful to look at bias relative to observations or perhaps consensus analysis.

[jgf]

Ok - so the forecast verification plots are typically comparing a model forecast with the model analysis file that eventually gets produced for that forecast hour, presumably when they are preparing to run the model.

 

 

 

 It is much more useful to look at bias relative to observations or perhaps consensus analysis.

 

Sure - are there any quantitative evaluations of model forecast skill based on either consensus analysis or observations?

 

I often do both of those evaluations for my own purposes  - I compare MSLP from model forecasts with a surface analysis, and I compare 10m wind speed and direction from model forecasts with scatterometer winds

 

it would be great to have a more rigorous measure than my eyeball...

 

here are two examples:

 

the first is the 3-16-2014 12Z GFS (blue) and GEM (red) forecast for today 3-18-2014 at 12Z overlain on today's OPC surface analysis valid at 12Z. The OPC analysis is partly based on observations, but I think it is also is a consensus analysis in the sense that the forecaster probably looks at analysis  files, or at least short range forecasts, from several models.

 

The GFS forecasts the location of the L off FL better than the GEM, and it closes the 1012mb contour, which the GEM doesn't. neither model gets the depth of the L correct - my guess is that the 48hr GFS was a better prediction of wind speed and direction near that L than the GEM.

 

 

the second is GFS 10m winds (green) vs Scatterometer winds (red) from 3-15-2014 at 02:30Z - the GFS is interpolated between 00Z and 03Z - it's hard to see the scatterometer unless you zoom in, but the agreement is pretty good S of LI in speed and direction, but not so great in the SE portion. there seems to be some sort of squall (although no rain flag) near the "1024", but of course no model is forecasting squalls. it might be interesting to look at CAPE there though.