December 2016 Temperature Forecast Contest

[Roger Smith]

Summary of Contest results

Congratulations to DonSutherland.1 for winning all three contests. His margin in the "all nine" was fairly comfortable, but he came from second to first in the western section and just managed to hold off some charging competition in the "original six" portion. Don also had high score in the central portion of that contest, and for NYC, DEN and SEA. He won the May contests in original six and all nine, November in all nine, and April in the western contest. 

Damage in Tolland had second place in the all nine scoring, was third in the original six and seventh in the west. He had high scores for DCA and ATL. He won in June for both all nine and western, and in July and November for the original six portion. 

Wxdude64 was third overall and second in the original six, by a margin that was about equal to small totals of time penalties there. He was eighth in the western contest. He had overall high scores for BOS and the eastern portion of the contest, and won January in all nine and the western contest. 

Mallow finished fourth overall and in the western contest, and sixth in the original six. He won September and November in the western contest.

Maxim was fifth in the all nine and western contests, and was eighth in the original six. Maxim won three months in both all nine and original six contests, March, September and October, and was also top score in the west in September.

RodneyS finished sixth overall, fourth in the original six and tied eleventh in the west. He had high score for IAH and for May in the western contest.

Wxallannj was seventh overall, and finished third in the west (high score PHX) after leading most of the year. He was tenth in the original six where he had high score for April. He also won April in the all nine segment. 

Rjay was eighth in the all nine, with high score for August; sixth in the west with high score for October, and ninth in the original six with high score in August.

BKViking was ninth overall,  fifth in the original six and tenth in the west. He had high score for ORD, and won January and December in the original six, as well as December in the all nine. BKV had substantial time penalties in three months and his raw score was very close to the totals of DonSutherland and wxdude64. Those details can be seen for all forecasters back in the summary of annual scoring (immediately after December scoring).

Midlo Snow Maker was tenth overall with high score in February, and surged to second place in the west including high score in March. He was 13th in the original six but had high score in February there too.

Highlights for the rest of the crew ... BLAZESS556 was 11th (all nine), and 7th in the original six, tied 11th in the west. Then it was HUDSONVALLEY21 in 12th (all nine), 11th in the original six and ninth in the west, winning July there. ... DMILLZ25 had a steady climb to finish 13th overall, 12th in the original six, and 15th in the west but finished with a win for December there. ... ROGER SMITH 14th (16th and 13th, high score August in west). ... TOM was 15th overall, and won June in the original six ... SD was 16th overall, and won July in the west. ... STEBO finished 17th (14th in original six) and began to score well in the west after a rough start. ... OHweather would have been around tenth had he continued on after July, where he had high score in west and all nine. TENMAN JOHNSON played ten months and had high score for February in the west. 

I think this was the most competitive year so far with the top ten really mixing it up and making it tough for Don all the way to the end, but it was quite an achievement to win all segments despite all that. Hope you enjoyed it, now on to 2017 ... 

[Damage In Tolland]

Another 2nd place finish..Once to Isotherm and one to Donny baseball. Two great minds

Congrats Don..well deserved. Now we track the snow

[donsutherland1]

Thanks all. Thank you Roger for holding this fun contest. I also second Kevin's idea that "now we track the snow."

Have a great 2017.

 

[wxdude64]

I'll take my third and realize that I could have done a hair better paying attention to the time lol. Good tight contest minus Don who pulled away from the field.

Congrats Don!

[RodneyS]

On 1/1/2017 at 11:22 PM, Roger Smith said:

Summary of Contest results

Congratulations to DonSutherland.1 for winning all three contests.

Very impressive accomplishment to win all three!

[Roger Smith]

The average error per forecast ranges from about 1.8 F deg at the top of the table to 3.1 F deg for random "Normal" estimates. The range is something like 1.6 to 2.8 in the original six, and 2.1 to 3.7 in the west, although in the west Normal is not the largest error range as in the original six and all nine, so in the west Normal's average error is 3.4. 

Your point total will correlate with this fairly well, but there are cases during the year where you might score zero but have an error larger than 5.0, or where the anomaly was larger than 5 making the points deduction smaller for errors of various sizes. If you want to estimate your own average error, take the percentage of points lost (e.g., if you scored 6500 in the all nine, a perfect score is 10,800 so you lost 4300 or about 40%) -- that would imply an average error of 2.0, but then adjust that by ten per cent (2.2 F deg) to make up for those oddball scoring cases, or scan back and take them case by case, some people probably have fewer missing deductions than others. 

Our group consensus is probably around 2.5 F error, 2.2 in the east/central and 2.8 in the west. As a group, we improved on random about 80% of the time (had the right anomaly sign) and by an average of 0.6 deg. Given the number of extreme forecasts (over half the total qualified) you could say we have a rather conservative bias in general. 

Perhaps more useful information (in terms of improving one's forecasting) could be gleaned from studying the error trends, some people are probably consistent and some are likely showing a warm or cold bias. If you can identify a bias, you can improve your results, those who scatter at random on either side won't find any tips they can use from that factoid. I will post again and reveal what other "norms" would have scored. Those other norms are always predicting +0.5, +1.0, +1.5 etc. I think one or two of those might have kicked most of our butts. And I will report on what the average anomaly was for each of the nine stations. I suspect that most had a positive average, possibly higher than +1.0 in some cases. 

[Roger Smith]

So, I scored the various Norms and ... the gains on Normal are shown, with ranking for just all nine where Normal finished behind all seventeen of the forecasters who entered every month ...

 

FORECASTER _____________ EAST __ CENTRAL __ ORIGINAL SIX __ WESTERN ___ ALL NINE ___ RANK

Norm +0.5 (all forecasts 0.5) __ 262 ____ 179 _______ 441 ________ 132 _____ 573 ___________ 18th

Norm +1.0 (all forecasts 1.0) __ 428 ____ 344 _______ 772 ________ 259 _____1031 ___________ 14th

Norm +1.5 (all forecasts 1.5) __ 511 ____ 469 _______ 980 ________ 344 _____1324 ___________ 13th

Norm +2.0 (all forecasts 2.0) __ 561 ____ 556 _______1117 ________ 407_____1524 ____________ 9th

Norm +2.5 (all forecasts 2.5) __ 533 ____ 654 _______1187 ________ 445 ____ 1632 ____________ 6th

Norm +3.0 (all forecasts 3.0) __ 443 ____ 658 _______1101 ________ 371 ____ 1472 ____________ 9th

so it would appear that Norm + 2.7 or +2.8 might have done the best of all the "norms" and come closest to our top five. You would score higher by going +4.0 for every forecast, than to predict normal. Normal +3.0 would have won February, July, and October (when it would have won all segments), and would have finished last in May, as well as last in the west for December. So I'm thinking that a strategy where you predict +3.0 unless you feel sure it will go cold might be a very good one (a lot better than mine, anyway).

 

Meanwhile, these are the average departures from normal for each station over the twelve months. 

 

DCA __ NYC __ BOS ____ ORD __ ATL __ IAH ____ DEN __ PHX __ SEA

+2.2__ +2.2__ +2.0 ___ +2.4 __+3.3 __+1.6 ____ +1.9 __+1.7 __+2.5

Out of the 108 months, only 20 were below normal, and one was exactly normal. 

The average absolute departure was 2.6 degrees. 

 

[Scotty Lightning]

That's not good when two mets finished toward the bottom......

[Roger Smith]

You can't be 100% sure from tags, but I think almost all of the top ten are non-mets technically speaking (identified as mets would be Mallow, who was in the top five or six, Blazess556, SD and Stebo). I think everyone knows that Don Sutherland has a solid amount of climatology background so probably he's by experience almost a meteorologist in terms of what he knows about the science behind long range forecasting (which, let's face it, is not all that much of a science). I could say the same thing minus the reputation for insight, and in fact I do make daily forecasts for an audience, just not paid to do so (nor is that audience in North America); although this year my finish was not very great at all, but I did manage to win this contest once. But I do follow weather patterns quite closely every day.  I believe some of our regulars are long-time weather enthusiasts who have picked up most if not all of what there is to know about pattern recognition and then from there it's mostly your hunches, what do you think the second half of the month will be like? The first half is more or less a partially revealed "known quantity" although we would probably give a range of estimates, my guess is that the range would be about half what the monthly forecasts usually span. I kept seeing phantom cold outbreaks and going too low with my estimates, when I was right it would be a good score, but it didn't happen often enough to make it an advantage and it turned into a disadvantage. There is probably a random factor to all this although Don always seems to be near the top of these forecast contests so I really believe he has a bit of a demonstrated edge. I wish more mets would participate in this contest. It would be interesting to see what would happen in the scoring if let's say every active member of the forum participated. My guess is a completely random distribution. I do notice with the scoring that some people join in and get consistently better with time, they must be picking up ideas that work as they go along. In my case I seem to be going in the opposite direction. There is no sign of improvement over time, I would say, this year's average error and scoring looks about the same as any given year in the past five. A score of 500 in the original six or 250 in the west usually wins a month, and consensus is usually in the low 400s (or low 200s). It varies a little, the hardest months for scoring are the extreme outcomes and that is why I relaxed scoring for big anomalies, so anyone with half a clue about it would at least get some points. If we used the same scoring metric for every month, some recent winter months would have been a whole lot of zeroes and a few 10s and maybe a 20. 

I think the scoring method measures skill accurately but it is an approximation of a more detailed method; I tested it out and found it was about .95 correlation so as it's a lot easier to manage, I go with it. 

About the late penalties -- I hate them too, but I have to impose them. I think they represent 80% punishment and 20% score adjustment for actual advantage gained. I base that on how much (if any) improvement I can make on my own score when I project the anomalies from day 5 or day 10. Sometimes those projections gain 20-40 points; my own subjective bias is probably still working its way through but I would say that somebody who entered on the fourth or fifth (when we have very large penalties) probably has only a 10% points advantage from that late entry. That's why I provided a table of actual scores before penalties, so you could see how people really compare in skill, but I would say the exact comparison would need to be adjusted, maybe remove 20% of the returned points.