ITS Service Status Report

Service Degradation — Resolved

U-M Data Center Degradation Impacting Multiple Services

Services Affected: Data Center Services

Start Time: 02/13/2021 11:50 am

Service Restored: 02/13/2021 9:15 pm

Issue Symptoms: Degradation

9 pm Update: A  problem with power distribution at the Michigan Academic Computing Center (MACC) disrupted many U-M services.  The power has been restored, and technicians have made substantial progress in restoring services. Services still impacted include:

  • ARC-TS - A Yottabyte switch died, ARC-TS is attempting to get the service back to a degraded state. No vendor support for this circumstance.
  • Web Hosting - only one known problem at this time: M-Marketsite Browse-Only is still inaccessible through Wolverine Access.
  • IAM - Identity Governance service is suffering from a DB issue
    • Functionality impacted is door access

7 pm Update: A  problem with power distribution at the Michigan Academic Computing Center (MACC) disrupted many U-M services.  The power has been restored, and technicians have made substantial progress in restoring services. Services still impacted include:

  • ARC-TS - Yottabyte and Conflux (small user community) are the last two systems to be restored. A Yottabyte switch died, ARC-TS is attempting to get the service back to a degraded state. No vendor support for this circumstance.

    • Progress made on Yottabyte Mais, but still no user access. Looking more positive than earlier.

  • Data center -the team believes we are no longer vulnerable to a DTE outage and we can safely fall back on the generators if needed

  • Unified Communications - Phone services are restored

  • LSA - Web Apps are down, storage appliance is running parity checks, SharePoint coming online is dependent on storage

  • Web Hosting - M-Marketsite Browse-Only is still inaccessible through Wolverine Access.  There may or may not be a problem with our “shim application” for sending users to M-Marketsite/SciQuest.

  • M-Marketsite is down for vendor maintenance from 6pm to 2am per https://solutions.sciquest.com/, which means troubleshooting cannot continue right now. Troubleshooting will resume at 2:00am.

  • Pure Storage network fiber replication path between ASB and MACC is down, a field service tech is in route to take a look a the fiber path

  • Merit has a switch down at the MACC impacting early child care at Dearborn

5 pm Update: A  problem with power distribution at the Michigan Academic Computing Center (MACC) disrupted many U-M services.  The power has been restored, and technicians have made substantial progress in restoring services. Services still impacted include:

  • Microsoft SharePoint - affects U-M COVID-19 Quarantine & Isolation Coordination Center operations
  • MyLINC and OARS, and Targeted Email
  • Some ARC-TS Services: Yottabyte, Lighthouse
  • Access to Marketsite through WolverineAccess
  • Some Dentistry systems
  • Some LSA systems

Services Restored as of 4 pm:

  • www.umich.edu and most other U-M websites
  • Web sites (redirects) for connecting U-M users to U-M / cloud services (Google, Canvas, etc.)
  • U-M Enterprise Financials, including Marketsite
  • U-M Enterprise Student, HR services
  • U-M Research Administration (eResearch) systems
  • Some ARC-TS Services: ARMIS, Turbo, and Locker
  • AFS storage and MiStorage
  • ITS Exchange Mailboxes & voicemail
  • Phone se

Who is Impacted? Faculty, staff, students, and other users of many U-M Services

Next Update: 9:30 pm.

Technical Details

Service Type: Production

Comments:

Merit - a switch down at the MACC impacting early child care at Dearborn. Ticket open with Merit.

Report Additional Impacts

Contact the ITS Service Center for more information or to report additional impacts.