ITS Service Status Report

Significant Incident — Resolved

U-M Google, U-M Email

Start Time: 01/26/2018 6:26 am

Service Restored: 01/26/2018 1:54 pm

Users are reporting issues receiving email. The mail gateway hosts are currently experiencing much higher than normal load, which is causing this issue. The root cause is currently being identified.

Issue Symptoms: Degradation

Technical Details

Service Type: Production

Comments:
3:15pm Update The backlog of email has been processed. There still may be some external email servers outside of the university that will retry and send some email over the next day or two. This Significant Incident and outage is considered resolved.

1:15pm Update: Email services have been restored. ITS has added more infrastructure to help process the backlog. It is going to take some time and we expect most mail to be delivered by the end of the day today. It is possible some messages may take a couple of days to be delivered. It is likely you will received a large influx of email in a short period of time as a result.

Additionally, for most recipients, emails will display the time when the message was successfully delivered rather than the time it was sent. We will monitor throughout the afternoon, and will provide another update after our 3:00pm conference call.

11:35am Update: We are continuing to work on releasing the impacted email (including voicemail messages) in the queue. Another update will be provided after our 1:00pm conference call.

10:45am Update: We have determined that the problem is the result of an anti-virus software update that did not process correctly.
ITS is correcting the problem and we hope to have impacted email that is currently being queued released shortly. Our next conference call is at 11:30am and we will provide another update after that call.


9:30am Update: This issue has been declared a Significant Incident. ITS is continuing to investigate and working to find root cause as to why email to umich.edu email addresses is delayed. Our next conference call is scheduled for 10:30AM and we will provide another update after that call.

Report Additional Impacts