ITS Service Status Report

Service Degradation

Exchange Email & Calendar

Start Time: 12/06/2018 1:03 pm

Anticipated End Time: Unknown

Issue Symptoms: Degradation

12/9 - 6:00pm ITS has worked with Microsoft the majority of today to develop possible workarounds. Work continues and will resume tomorrow, with further discussions with other ITS teams related to this issue. Our next update will be provided at 10:00 AM. 12/9 - 2:00pm ITS is actively working with Microsoft engineering teams on this issue. Our next update will be provided at 6:00 PM. 12/9 -10:00am The degradation is still being experienced by some ITS voicemail users, and ITS is continuing their engagement with Microsoft. Our next update will be provided at 2:00 PM. 12/8 - 6:00pm We continue to connect and work with Microsoft engineers to collaborate on this issue. We will provide another update at 10:00 AM tomorrow. 12/8 - 2:00pm ITS continues to engage with both Microsoft and Google teams on this issue. Expect to see our next status update at 6:00 PM. 12/8 - 10:00am ITS continues to engage with Google to repair the current degradation of email notifications related to Exchange-based voicemail. ITS has also reached out to Microsoft in regard to this matter. Expect to see our next status update at 2:00 PM. 12/7 - 6:00pm ITS continues to engage with Google to address the current degradation of email notifications related to voicemail. Google has raised the priority of our case and it has been escalated to their Support Engineers. From our most recent analysis, it appears that 80% of the notifications are being successfully delivered, while 20% are not. Our team will continue to provide updates on this situation until it is resolved. Expect to see those updates at 10:00 AM, 2:00 PM and 6:00 PM. 12/7 - 12:30pm We believe the issue is related to an adjustment Google made to their spam detection algorithms. We have a case open with Google and are working with them. Our investigation determined that the failure was happening after delivery from our Mail Gateway servers to Google gmail. Some notification emails were accepted, some were rejected, and some were marked as spam. We are working with Google to jointly examine the failures and identify possible causes. 12/6 - 1:00pm We have confirmed reports where email notifications about the arrival of new voicemail are not being delivered to the individual's personal mailbox. We are investigating. This issue is not affecting the delivery of voicemails to the voicemail mailbox. They can be accessed as normal by dialing in to 734-615-4242 and accessing your mailbox.

Technical Details

Service Type: Production

Server Name: ITS-ExchPrdA01 ITS-ExchPrdA02

Comments:

12/8 - 10:00am It appears that Google made recent changes in their SPAM detection methodology that are resulting in these rejections. ITS is working with Microsoft to investigate adjustment of the message headers that are causing Google's rejection of the voicemail notification emails.

Groups Notified:

Report Additional Impacts