Eskimo Joe Posted November 10, 2011 Share Posted November 10, 2011 Just an FYI: NCEP: UNSHCHEDULED POWER OUTAGE AT FAIRMONT...WHERE THE PROD SUPERCOMPUTER IS...HAS NECESSITATED A PROD MACHINE SWITCH TO THE BACKUP SUPERCOMPUTER. EXPECT NCEP MODEL DELAYS OF 60 TO 90 MIN. WE REGRET THE INCONVENIENCE THIS WILL CAUSE. WE APPRECIATE YOUR PATIENCE. Link to comment Share on other sites More sharing options...
Ellinwood Posted November 10, 2011 Share Posted November 10, 2011 Just an FYI: NCEP: UNSHCHEDULED POWER OUTAGE AT FAIRMONT...WHERE THE PROD SUPERCOMPUTER IS...HAS NECESSITATED A PROD MACHINE SWITCH TO THE BACKUP SUPERCOMPUTER. EXPECT NCEP MODEL DELAYS OF 60 TO 90 MIN. WE REGRET THE INCONVENIENCE THIS WILL CAUSE. WE APPRECIATE YOUR PATIENCE. That popped up at the start of the 12z GFS Op. run, and it looks like it's been resolved. Link to comment Share on other sites More sharing options...
dtk Posted November 10, 2011 Share Posted November 10, 2011 That popped up at the start of the 12z GFS Op. run, and it looks like it's been resolved. It's not 'resolved', they lost power to the facility that hosts one of our two supercomputers (the one in West Virginia). So they had to do an emergency, unplanned computer switch (to the machine that was in backup/development mode) and restart the cycle that was running at the time (expect glitches). As of now, we're basically operating without a backup. Fun day for NCO. Link to comment Share on other sites More sharing options...
Ellinwood Posted November 10, 2011 Share Posted November 10, 2011 It's not 'resolved', they lost power to the facility that hosts one of our two supercomputers (the one in West Virginia). So they had to do an emergency, unplanned computer switch (to the machine that was in backup/development mode) and restart the cycle that was running at the time (expect glitches). As of now, we're basically operating without a backup. Fun day for NCO. Sorry... I meant that the model delay had been resolved, not the underlying issue of the power outage. Sounds like a fun day... at least the backup's working Link to comment Share on other sites More sharing options...
icebreaker5221 Posted November 10, 2011 Share Posted November 10, 2011 It's not 'resolved', they lost power to the facility that hosts one of our two supercomputers (the one in West Virginia). So they had to do an emergency, unplanned computer switch (to the machine that was in backup/development mode) and restart the cycle that was running at the time (expect glitches). As of now, we're basically operating without a backup. Fun day for NCO. At what forecast hour did the restart occur? Link to comment Share on other sites More sharing options...
dtk Posted November 10, 2011 Share Posted November 10, 2011 At what forecast hour did the restart occur? I don't know exactly when the thing went down. I'm 99% sure they just restarted from the analysis (and didn't just re-fire the forecast mid-stream from say 42 hours or something). Link to comment Share on other sites More sharing options...
icebreaker5221 Posted November 10, 2011 Share Posted November 10, 2011 I don't know exactly when the thing went down. I'm 99% sure they just restarted from the analysis (and didn't just re-fire the forecast mid-stream from say 42 hours or something). Oh ok, that makes sense! I was thinking WRF terminology as in "they ran a restart". Link to comment Share on other sites More sharing options...
Eskimo Joe Posted November 10, 2011 Author Share Posted November 10, 2011 SENIOR DUTY METEOROLOGIST NWS ADMINISTRATIVE MESSAGE NWS NCEP CENTRAL OPERATIONS CAMP SPRINGS MD 1750Z THU NOV 10 2011 THE FAILOVER FROM THE NCEP PRIMARY TO BACKUP SUPERCOMPUTER WAS SUCCESSFUL...ALL 12Z MODELS ARE EXPECTED TO RUN TO COMPLETION WITH EXCEPTION OF THE GLOBAL RTOFS...WITH EXPECTED DELAYS. AT THIS TIME, THE 18Z SUITE IS EXPECTED TO RUN ON-TIME. $ OREILLY/SDM/NCO/NCEP Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.