ITS Service Status Report

Service Degradation — Resolved

MiServer - Intermittent Network Connectivity Issues

Services Affected: MiDatabase, MiServer

Start Time: 01/13/2022 8:35 am

Service Restored: 01/17/2022 10:00 am

Issue Symptoms: Intermittent issues

We're receiving reports of intermittent connectivity issues with some MiServer virtual machines.  If you are experiencing similar symptoms, please let us send a message to miserver.support@umich.edu and we work with you to get help resolve the problem and obtain additional information for further investigation.

Who is Impacted? Miserver and MiDatabase customers

Next Update: 01/17/22 - end of business day

Technical Details

Service Type: Production

Server Name: vcenter.umich.edu

Comments:

1-17-22 11:00am

We have made settings changes that address the VMware bug that was causing the issues. Please contact miserver.support@umich.edu if you experience any network issues. We continue to monitor the cluster and the VMs. 

1-17-22 9:00am

We are continuing to work with VMware on root cause. We believe we have identified the root cause but are working to confirm before we deploy any changes to the environment. 


1-14-22 5:00pm

We have implemented a change that should minimize the likelihood of VMs experiencing the intermittent connectivity issues. The windows team is also conducting emergency maintenance tomorrow night to change settings on VMware tools. https://status.its.umich.edu/report.php?id=152281

We continue to work with VMware on a long term fix. If teams do experience an issue with connectivity, please open a ticket by emailing miserver.support@umich.edu. That will place the ticket directly in our team queue and we will work to resolve the connectivity issues. 


1-14-22 8:00am

We continue to gather information with the vendor on the root cause. It appears that there are two issues, one dealing specifically with a configuration option and VMware tools on Windows servers. The second issue we were able to narrow down to the NSX-t firewall specifically. We were able to collect the logs today as it was happening and are awaiting vendor analysis. 

Report Additional Impacts

Contact the ITS Service Center for more information or to report additional impacts.