ITS Service Status Report

Service Degradation — Resolved

Voicemail notifications not delivered to email

Services Affected: Exchange Email & Calendar, specifically Exchange Voicemail

Start Time: 12/06/2018 1:03 pm

Service Restored: 12/19/2018 10:58 am

Issue Symptoms: Degradation

12/14 - 3:32pm
After additional problem investigation, we have discovered that some mailboxes are still affected by this issue. Specifically, the problem still exists for mailbox 'redirect' rules that were manually created by users. The fix for this is to convert 'redirect' type rules to 'forward' type rules, and instructions for doing so can be found here:

https://documentation.its.umich.edu/node/960

Once the steps are completed, the flow of voicemail notifications should return to normal for their intended target(s).

12/13 - 8:30am
The workaround is now in place. You should once again be receiving email notifications sent to your U-M Google email account about new voicemail messages and missed calls. The messages will have a slightly different format, so if you created a filter in Gmail to handle voicemail notification messages, you will want to adjust it accordingly. If you have any ongoing issues related to this, please contact the ITS Service Center. Many thanks again for your patience and understanding as we worked to get this addressed.

12/12 - 2:30pm
We have a workaround that we will be putting into place this evening to resolve this issue. Those who are affected should receive a message once the workaround is in place. We will provide our next update on this issue no later 12:00PM tomorrow, December 13th. Thank you again for your patience as we have worked to resolve this issue.

12/11 - 5:00pm
A potential workaround is in development and testing. We won't have any additional updates this evening, but expect our next status update no later than 12:00PM tomorrow.

12/11 - 3:30pm
With Microsoft's assistance, ITS continues to develop a potential workaround to address the degradation related to email notification about new voicemails and missed calls. We appreciate your patience as we continue our work to get a fix into place. We'll update this status again around 6:00PM.

12/11 - 12:00pm
ITS continues to work with Microsoft and Google to repair delivery of voicemail notification emails. Until this is resolved, and to ensure that you are not missing important voicemails we recommend that Exchange voicemail users on the UM-Ann Arbor campus use Outlook Web Access (https://exchange.umich.edu) or call into your voicemail by dialing into 734-615-4242 and access your mailbox via telephone. If you need help to reset your voicemail password or phone PIN, please call 734-763-2000 and select option 1 for assistance. We will provide our next status update around 3:30PM.

12/10 - 6:00pm
Microsoft and ITS continue to work to address this issue collaboratively. We will keep this status page updated with the latest news. Expect our next update on this matter no later than 12:00PM tomorrow.

12/10 - 3:30pm
ITS continues to work directly with Microsoft to address this issue and to try to put a solution in place. There are no other updates at this time. We will post another update at 6:00PM.

12/10 - 12:00pm
ITS continues to work to address the degradation of voicemail notification emails. Until this is corrected, and to ensure that you are not missing important voicemails we recommend that Exchange voicemail users on the UM-Ann Arbor campus use Outlook Web Access (https://exchange.umich.edu) or call into your voicemail by dialing into 734-615-4242 and access your mailbox via telephone. We will provide the next status update around 3:30PM.

12/9 - 6:00pm
ITS has worked with Microsoft the majority of today to develop possible workarounds. Work continues and will resume tomorrow, with further discussions with other ITS teams related to this issue. Our next update will be provided at 10:00 AM.

12/9 - 2:00pm
ITS is actively working with Microsoft engineering teams on this issue. Our next update wil

Technical Details

Service Type: Production

Server Name: ITS-ExchPrdA01 ITS-ExchPrdA02

Comments:

During the first week of December, Google made a security change to the way they accept email. Due to the way voicemail notifications are processed, the delivery of some voicemail notifications to U-M Google email accounts was affected. ITS has taken steps to remedy this for the majority of accounts. As of today, December 19th, Google has temporarily rolled back the change that was made but will be re-implementing this change in January.

There remain a number of accounts with manually added redirect rules that are affected. ITS will be sending a communication to those users with instructions on how to update these rules prior to the re-implementation of the Google security change. These steps are also documented at the following link, should you like to proceed with the fix before the communication is sent:

https://documentation.its.umich.edu/node/960

Groups Notified:

Report Additional Impacts