ITS Service Status Report

Scheduled Maintenance — Resolved

Satellite Server Patching

Services Affected: Red Hat Satellite at U-M

Start Time: 09/28/2022 5:00 pm

Service Restored: 09/29/2022 2:32 pm

Issue Symptoms: Outage

2:30 pm: We have reverted back to make the service operational again, please report any issues that arise via tickets to the "ITS-Satellite" TDx group.

2:00 pm: Red Hat has discovered a hotfix that applies to our current problem when upgrading to 6.10, we are rolling back and once we verify the production environment is in a healthy state again we will update the status page

1:00 pm: We are planning to revert back if there's no progress made with Red Hat and we will update again once the prod satellite environment is fully operational again.

10:53 am: Severity 1 case is open with Red Hat and we are actively engaged with an engineer. The prod satellite server is still currently down at this time. Apologies for any inconvenience.

9:00 am: We are escalating out to Red Hat engineers as the upgrade work is going outside the estimated time provided to us.

6:30 am: We are unfortunately still waiting on the process to complete. We will update again at 9 am.

10:00 pm: Extending the window further as we are still waiting for a process to successfully run to completion. We are continuing to monitor and will provide more updates.

7:56 pm: Extending maintenance window until 10 pm and will again update progress then.

Update @ 7:23 pm:

We are still monitoring and waiting on the final content migration to complete before switching over to continue the 6.10 upgrade work.

Patching and rebooting of the Red Hat Satellite at U-M server

Who is Impacted? Users of the Red Hat Satellite at U-M Service

Next Update: 2:30 pm

Technical Details

Service Type: Production

Server Name: calipso.linux.it.umich.edu

Comments:

Reverted back to the snapshot prior to any upgrade work. The production environment service is back up and operational.

Groups Notified:

  • satellite.notify

Report Additional Impacts

Contact the ITS Service Center for more information or to report additional impacts.