ITS Service Status Report

Scheduled Maintenance — Resolved

MACC Data Center Outage

Services Affected: Data Center Services, Data Center Tier 1, MACC Data Center Outage

Start Time: 02/18/2011 6:00 pm

Maintenance Completed: 02/19/2011 8:20 pm

02/19/11 8:30 The second and final testing phase of Eaton hardware is expected to be complete within the next 30 minutes. This phase consisted of removing a ground loop that was discovered in the planning/developement stages of this project. The Eaton phase one test were redone without failure.
We expect to have the Data Center up and available for system start up at approximately 10:00 p.m.
Thank you:
Mark Linsenman

02/19/4:00 p.m. all Cyberx checks are nearing completion 10 grounds required tightening. We are just completing the first phase of the Eaton testing and have not experienced any failures. We remain on schedule.
Mark Linsenman

02/19/11 12:32 p.m. Thomas & Betts Cyberex testing is on schedule.
Eaton set up for testing is now complete including all verification steps.
We remain on schedule as we start the Eaton testing.
Mark Linsenman

2/19/11 4:16 a.m. All generator testing was completed on schedule.The generators were tested under multiple scenarios, multiple times and in each instance the system worked as designed. We noted small documentation additions for one condition.
One problem was noted with the duplex switch gear that was determined not to be critical but will require further research.
Due to the level and type of work being during the next phase of testing the next status update will be at noon.
Mark Linsenman

2/19/11 12:20 p.m. Generator testing started on time testing is proceeding as planned....Mark Linsenman

The MACC Data Center Outage schedule is as follows:

Friday February 18th starting at 8:00 p.m. thru Sunday February 20th at noon. This will be a complete outage of this facility. All systems should be down by 8:00 p.m. when testing starts.

All communications will be done at major milestones and/or no more than 4 hr intervals via the ITS Status page and associated email distribution

This outage is intended to identify the cause of the previous unplanned outages that occurred in 2009 and 2010. Testing will also be done to determine the generators "failure to start" condition that occurred in the July 2010 instance.

Based on the scope of the remediation required these repairs may be done within this window so retesting and validation of remediation can occur.

Thank you
Mark Linsenman

Technical Details

Service Type: Production

Groups Notified:

  • macc-notify
  • fln
  • macc-ops
  • ITS.Lead.Team
  • dannance
  • macc-users
  • ias.annarbor@gmail.com
  • macc2011.02.outage.workers
  • cluster-admins
  • mikko.kinnunen@ithaka.org
  • tim.bruce@ithaka.org
  • larry
  • maio@ithaka.org
  • macc.2011.02.outage.pagers@umich.edu

Report Additional Impacts