ITS Service Status Report

Service Degradation — Resolved

MRequest/FootPrints slowness

Services Affected: MRequest

Start Time: 11/27/2018 6:00 am

Service Restored: 12/15/2018 5:58 pm

Issue Symptoms: Intermittent issues

Users of the MRequest FootPrints service may experience slow page loads and occasional error pages due to ongoing, sporadic performance issues on the request.umich.edu server.

System administrators are monitoring the server and making improvements where possible.

We understand that these issues are an impediment to performing tasks related to your work are are working diligently to improve the experience for our users.

Technical Details

Service Type: Production

Server Name: request.umich.edu

Comments:

12/3/18: revised mail server configuration to no longer call (defunct) dspam and restarted mail server

12/5/18: webserver configured to use longer Timeout parameter

12/7/18: shunting spam destined for a particular workspace that was being hit hard to the bit-bucket

12/10/18: metrics show the server is performing better than the previous two weeks, but there are still periods of the day with high numbers of timeouts loading pages. Further increased the Timeout parameter in an attempt to better manage the user experience.

12/11/18: Restarted the database server to turn off query logging to improve database performance. While the number of web server timeouts observed is down significantly, those we are seeing appear to be due to individual workspace's tables being locked to insert a record for an extended period of time. While the locking is expected, the length of time the table is locked is not. It is hope that disabling query logging will reduce the length of time that tables are locked, therefore reducing the timeouts.

12/15/18 We feel that performance has returned to level similar to what existed prior to Thanksgiving. We consider this degradation resolved.

We are not satisfied with the current level of performance are are taking additional steps to improve overall response times. Provided our current testing proceeds satisfactorily, we will probably schedule additional maintenance in the next few days to implement our planned changes.

Groups Notified:

  • mrequest-community

Report Additional Impacts