inaccurate temperature data is passing QA and is being archived into GHCN-Daily

[wfgarnett3]

To summarize the below inquiry:

On July 11, 2019 the South Jersey Regional airport inaccurate TMIN of 50 degrees Fahrenheit was 16 and 18 degrees lower than the values reported from nearby Neshaminy Falls and Trenton Mercer airport and appropriately failed the spatial consistency QA test.
Yet on October 2, 2019 the Neshaminy Falls inaccurate TMIN of 52 degrees Fahrenheit was 15 and 18 degrees lower than the values reported from nearby Trenton Mercer and South Jersey Regional airport yet did not fail the spatial consistency QA test.

Was a spatial consistency test really done by NOAA on the October 2 TMIN value for Neshaminy Falls?
If not, why not?
If so, why did it pass where the similar example from July 11 for South Jersey failed the test?
Does it have to do with the way the math behind the spatial regression and spatial corroboration tests are structured?

(I e-mailed NOAA also and will post an update when I hear from them)
--------------------------------------------------

For September 17, 2019 a minimum temperature value of 49 degrees Fahrenheit was reported for the Neshaminy Falls (GHCND ID: USC00366194) COOP weather station located in Pennsylvania just outside Philadelphia.

On September 18, 2019 I contacted The National Weather Service (NWS) / NOAA and I informed them this value was impossible.

NWS / NOAA agreed and changed the TMIN value for September 17 for Neshaminy Falls from 49 degrees to 55 degrees Fahrenheit which was the At Observation temperature (TOBS) reported just before end of day midnight.

On October 2, 2019 a TMIN value of 52 degrees Fahrenheit was reported for Neshaminy Falls.

I contacted NOAA once again and I informed them this value was impossible (I did not hear back from them).

October 2 was the day the TMAX temperature was over 90 degrees Fahrenheit from the Southeastern United States all the way up to the New York City area.

No weather station in the Philadelphia area, not even Mount Pocono near 2000 feet elevation and much further north got anywhere near a TMIN of 52 degrees Fahrenheit that day.

In addition for the ASOS stations in the entire area the hourly reported dewpoint values for the whole day were over 60 degrees Fahrenheit thefore since actual temperature can never go below dewpoint temperature it was scientifically impossible to have a TMIN of 52 degrees at Neshaminy Falls.

At the link below it states archive-ready data sources are available 45 to 60 days after the close of a data month, and for COOP weather stations the sflag will flip from "H" to "7".

Today is the 60th day from the close of October and the flag has not flipped - I called NCEI and they stated there is a delay for the reprocessing but that the QA would have already been done.

Thus for Neshaminy Falls the TMIN of 52 degrees Fahrenheit (11.1 degrees Celsius) on October 2 would be official.

https://www.ncdc.noaa.gov/ghcn-daily-description

We can compare the Neshaminy Falls (40 feet elevation) station's October 2 TMIN value of 52 degrees to ASOS Trenton Mercer Airport, New Jersey weather station (USW00014792 190 feet elevation), which is located 11.46 miles away, and ASOS South Jersey Regional Airport, NJ (USW00093780 49 feet elevation), which is located 15.53 miles away.

All 3 weather stations report nearly on a daily midnight to midnight schedule for TMAX and TMIN so are appropriate for comparisons.

ASOS Northeast Philadelphia Airport and ASOS Philadelphia International Airport sometimes have an urban heat island effect -- they reported TMIN values of 72 and 70 Fahrenheit respectively that day.

Trenton Mercer Airport for October 2 reported a TMIN of 67 degrees Fahrenheit while South Jersey Regional Aiport reported a TMIN of 70 degrees Fahrenheit.

Thus Neshaminy Falls inaccurate TMIN value of 52 degrees on October 2 is 15 and 18 degrees lower than these 2 other nearby New Jersey weather stations.

Over at the link below at climate.gov NOAA states:

"Observers submit their raw data to NOAA’s National Center for Environmental Information (NCEI) on a daily or monthly schedule for further quality checks."
"Weather data are also checked for consistency across a region. Scientists observe data sets from comparable stations to see if the data makes sense for the region and time of year."
"Once the COOP data has passed quality control, it becomes part of the larger data record known as the Global Historical Climate Network-Daily (GHCN-D) database. The data can then be processed to generate climate products such as maps and graphs."

https://www.climate.gov/maps-data/primer/processing-climate-data

Below are the links detailing the QA process for GHCN-Daily, including the Durre et al. paper "Comprehensive Automated Quality Assurance of Daily Surface Observations" which is easy to read and understand:

https://www.ncdc.noaa.gov/ghcn-daily-methods
https://journals.ametsoc.org/doi/full/10.1175/2010JAMC2375.1

I understand as stated that the test thresholds are chosen to yield the highest error detection rate without exceeding 20% false-positive rate.
Also obviously, for instance, flagging temperature values that might be only 1 degree off is imprudent.

Section 6 and Table 4 detail the "spatial consistency" tests (regression and corroboration).

I read the explanations and I get the "idea" of the math behind the tests but note the Neshaminy Falls TMIN on October 2 exceeds a 8 degree Celsius (14.4 degree Fahrenheit) difference from the reported TMIN for BOTH Trenton Mercer aiport and South Jersey Regional airport that day and also has exactly a 10 degree Celsius (18 degree Fahrenheit) difference from South Jersey Regional airport.

Also note in the GHCN-Daily on July 11, 2019 for South Jersey Regional airport the TMIN of 50 degrees Fahrenheit (10 degrees Celsius) has a qflag of "S" and thus failed the spatial consistency test.

On that same July 11 day Neshaminy Falls reported a TMIN of 66 degrees Fahrenheit while Trenton Mercer Airport reported 68 degrees Fahrenheit (20 degrees Celsius).

So we have the case where on July 11 South Jersey Regional airport inaccurate TMIN was 16 and 18 degrees Fahrenheit lower than the values reported from Neshaminy Falls and Trenton Mercer airport and appropriately failed the spatial consistency test, yet on October 2 Neshaminy Falls inaccurate TMIN was 15 and 18 degrees Fahrenheit lower than the values reported from Trenton Mercer airport and South Jersey Regional airport yet did not fail the spatial consistency test.

Also note the last time Neshaminy Falls failed a spatial consistency test was the TMIN value of 24 degrees Fahrenheit on October 23, 2016 (that Trenton Mercer airport and South Jersey Regional airport reported respectively 43 degrees and 46 degrees Fahrenheit).

Focusing on the October 2, 2019 TMIN value of 52 degrees Fahrenheit for Neshaminy Falls, the next logical questions I have are:

1. Did NOAA really run a spatial consistency test on the TMIN value on October 2 for Neshaminy Falls?
    a) If the answer is no, why not?
2. If a spatial consistency test was done and it passed QA does this mean regardless of the fact the value had a difference from nearby weather stations that exceeded 8 degrees Celsius is this due to the way the math behind the spatial consistency tests (both regression and corroboration) are structured?
    a) In light of the examples above would it make sense to modify the math behind the spatial consistency tests?
3. Why did the July 11 example from South Jersey Regional airport fail the spatial consistency test yet the similar October 2 example from Neshaminy Falls did not?
4. Do you think since the eastern United States had hot 90 degree plus Fahrenheit afternoon temperatures on October 2 that somehow that TMIN of 52 by Neshaminy Falls should have been caught by some test since it so "obviously" wrong?
5. Furthermore, not counting the unarchived November 1 TMAX of 73 and the December 11 TMIN of 15 for Neshaminy Falls that are obviously wrong I have 2 other recent archived examples from Neshaminy Falls.
September 15 TMIN of 54 Fahrenheit (not sure about this one since it did rain).
October 1 TMIN of 53 degrees Fahrenheit (that is obviously wrong for the same reasons stated above for October 2).
For September 15 and October 1 for Neshaminy Falls the difference from the 2 nearby Trenton Mercer airport and South Jersey Regional airport weather stations was only 7-9 degrees Fahrenheit.
Would it make "prudent" sense to make the test threshold even tighter or will this lead to more false positives/more workload/not feasible with budget constraints/etc. ?
6. Also in regards to (4) note as stated there were two days in a row (October 1 and October 2) that the TMIN values were obviously wrong for Neshaminy Falls yet that was not enough to cause either the spatial regression or the spatial corroboration check tests to fail for at least one of those days?
7. Finally note for South Jersey Regional airport the inaccurate TMIN of 28 degrees Fahrenheit on September 27 appropriately failed the spatial consistency test. (Neshaminy Falls and Trenton Mercer airport were 47 degrees and 50 degrees Fahrenheit respectively).
However the South Jersey Regional inaccurate TMAX of 94 degrees Fahrenheit on July 23, 2019 DID NOT fail the spatial consistency test.
It is an inaccurate value -- Neshaminy Falls And Trenton Mercer airport reported respectively 76 and 75 degrees Fahrenheit on that day (and thus the difference is at least 10 degrees Celsius for both cases).
Notice too no hourly temperature values were reported for that entire day for this ASOS station so there must have been some issue going on:
https://www7.ncdc.noaa.gov/CDO/cdoselect.cmd
Regardless, since a TMAX of 94 was submitted why did it not fail the spatial consistency test?

[wfgarnett3]

NOAA replied with the below feedback:

--------------------------------------------------------------

Hi William,

 

Our GHCN-Daily POC Matt Menne has provided the following feedback in italics to your questions regarding Neshaminy Falls, PA's and other stations' data values and GHCN-Daily's QC of them:

 

************************************** 

************************************** 

   

"Please see my responses to the questions below (in italics):

 

1. Did NOAA really run a spatial consistency test on the TMIN value on October 2 for Neshaminy Falls?

Yes, the checks are run daily on all data automatically.  As long as the minimum data requirements are met for a QC test, the test is applied
a) If the answer is no, why not?

2. If a spatial consistency test was done and it passed QA does this mean regardless of the fact the value had a difference from nearby weather stations that exceeded 8 degrees Celsius is this due to the way the math behind the spatial consistency tests (both regression and corroboration) are structured?

As noted in Durre et al. 2010, (Table 4), the absolute value of the residual for the spatial regression check needs to exceed 8 deg C and the normalized residual must exceed 4.0 standard deviations using neighboring values for the day before, the day of and the day after the value being tested in order to fail the test.  Likewise the spatial collaration test compares anomalies on the day before, the day of and the day after the value in question and the anomalies must differ by 10 deg C. If these conditions are not met, the value will pass the test.

a) In light of the examples above would it make sense to modify the math behind the spatial consistency tests?

The spatial checks are conservative by design to avoid flagging valid observations.  Altering the thresholds would inevitably lead to an increase in legitimate values failing the checks.  While we would like to catch only errors, the nature of these observations is such that no set of checks can be assured to flag only errors.  The spatial consistency checks are particularly susceptible to flagging too many legitimate values, which is why the thresholds are set to be conservative.

 

3. Why did the July 11 example from South Jersey Regional airport fail the spatial consistency test yet the similar October 2 example from Neshaminy Falls did not?
It is all a question of exceeding or not exceeding the thresholds described in Durre et a. 2010 (Table 4), assuming the criteria for having the minimum number of neighboring observations are met.

 

4. Do you think since the eastern United States had hot 90 degree plus Fahrenheit afternoon temperatures on October 2 that somehow that TMIN of 52 by Neshaminy Falls should have been caught by some test since it so "obviously" wrong?

We would certainly like to flag all errors, however the observational day for COOP observers is sometimes somewhat irregular if the observer is unable to read the thermometers at exactly the same time each day.  This is why many of the daily quality checks consider a 3-day window centered on the day in question.  Note in this example, that a front was nearby separating the very warm air in the mid-Atlantic from a cooler air mass in New England, and that much cooler temperatures followed on Oct. 3, which was likely a factor in the value passing the spatial check.  In a case like this, NOAA's Datzilla process can be used to flag a value based on additional scrutiny and forensics.   

 

5. Furthermore, not counting the unarchived November 1 TMAX of 73 and the December 11 TMIN of 15 for Neshaminy Falls that are obviously wrong I have 2 other recent archived examples from Neshaminy Falls.
September 15 TMIN of 54 Fahrenheit (not sure about this one since it did rain).

October 1 TMIN of 53 degrees Fahrenheit (that is obviously wrong for the same reasons stated above for October 2).
For September 15 and October 1 for Neshaminy Falls the difference from the 2 nearby Trenton Mercer airport and South Jersey Regional airport weather stations was only 7-9 degrees Fahrenheit.
Would it make "prudent" sense to make the test threshold even tighter or will this lead to more false positives/more workload/not feasible with budget constraints/etc. ?

 

Based on extensive testing and years of user feedback, we have opted for the thresholds in place that prevent over-flagging the data but require the occasional need to submit a "Datzilla" data ticket to requiring us to flag or unflag a particular value. 

 

6. Also in regards to (4) note as stated there were two days in a row (October 1 and October 2) that the TMIN values were obviously wrong for Neshaminy Falls yet that was not enough to cause either the spatial regression or the spatial corroboration check tests to fail for at least one of those days?

 

As the response noted for the first question (3), is all a question of exceeding or not exceeding the thresholds described in Durre et a. 2010 (Table 4), assuming the criteria for having the minimum number of neighboring observations are met.  

7. Finally note for South Jersey Regional airport the inaccurate TMIN of 28 degrees Fahrenheit on September 27 appropriately failed the spatial consistency test. (Neshaminy Falls and Trenton Mercer airport were 47 degrees and 50 degrees Fahrenheit respectively).

 

However the South Jersey Regional airport inaccurate TMAX of 94 degrees Fahrenheit on July 23, 2019 DID NOT fail the spatial consistency test.
It is an inaccurate value -- Neshaminy Falls And Trenton Mercer airport reported respectively 76 and 75 degrees Fahrenheit on that day (and thus the difference is at least 10 degrees Celsius for both cases).
Notice too no hourly temperature values were reported for that entire day for this ASOS station so there must have been some issue going on:
https://www7.ncdc.noaa.gov/CDO/cdoselect.cmd

 

Regardless, since a TMAX of 94 was submitted why did it not fail the spatial consistency test?

Again, it is all a question of exceeding or not exceeding the thresholds described in Durre et a. 2010 (Table 4), assuming the criteria for having the minimum number of neighboring observations are met.  

 

**************************************

**************************************

In Matt's reply to your Question #4 and #5, he has mentioned that we (NCEI) have the ability to flag (or unflag if it applies) a data value; doing so has to be based on supporting evidence to do so.  For more recent data values such as these for the past data-year of 2019, we very much depend on the National Weather Service (NWS) and its Forecast Office at NWS-Mt. Holly, NJ (WFO-PHI) to inform us via the proper channels of whether these actions are necessary.  We're going to depend on their expertise in the case of these stations' questioned data temperature values to do so, if needed.

 

We (NCEI) very much depend on NWS to verify and properly document via Station-Level Metadata Updates that the equipment and observing practices at each station meets the standards of the well established NWS Directives pertaining to the data observing network in question (COOP, ASOS, etc.).   Having said that, overall issues with the data observing practices with the COOP Site at Neshaminy Falls, PA (GHCND ID: USC00366194; COOP ID #366194) and the other stations in this general local area that you've mentioned here should be further discussed with NWS-Mount Holly, NJ (WFO-PHI) and NWS Eastern Region HQ all of whom are cc'd on this email reply.

 

I hope Matt's answers to your questions and our further advisement help your understanding of GHCN-Daily's QC of NCEI's archived data.  We greatly appreciate your email inquiry and have a great day!

 

Best Regards,
Bryant Korzeniewski