ITS Service Status Report

Service Degradation — Resolved

ASB Turbo Outage

Services Affected: Research Storage, Turbo Research File Service

Start Time: 07/12/2021 8:00 am

Service Restored: 09/16/2021 11:03 am

Issue Symptoms: None

Update 9/16/2021 @ 11 am

The last sync is complete.  All policies are syncing as expected.  We are closing this status, and will be working with the vendor to reschedule the upgrade from July.

Update 9/15/2021 @ 10am

Progress continues on the data copies from MACC to ASB.  28 of 29 are complete.  We could be finished within the next 8 days. The next update will be Friday AM

Update 9/13/2021 @ 3pm

Progress continues on the data copies from MACC to ASB.  27 of 29 are complete.  Some performance improvements were made over the last week and the pace of data being copied is much faster.  We could be finished as early as within the next 10 days.

Update 9/10/2021 @ 9 am

Progress continues on the data copies from MACC to ASB.  24 of the 29 are complete.  We are approximately half way through the copying of data, but it will take longer to complete due to the process taking longer.  The next update will be Monday AM.

Update 9/7/2021 @ 10:30 am

Progress continues on the data copies from MACC to ASB.  23 of the 29 policies are complete.  The next update will be Thursday AM.  

Update 9/3/2021 @ 5pm

Progress continues on the data copies from MACC to ASB.  18 of the 29 policies are complete.  The next update will be Tuesday AM.  

Update 9/1/2021 @ 11am

We have changed our strategy for fixing the issue as syncs are still intermittently failing.   We are working with the vendor to change how we sync data from MACC to ASB and resyncing the data one new policy at a time.   We have finished with about half of the new policies at this point and about 10-15% of the data on the cluster.  This new syncing process will take about 40 days to complete for the largest policies (the policies for MM volumes and LSA volumes respectively).  While the new data is being synced over, the old data is deleted, so there is a possbility that we will not hold a backup copy on the ASB cluster on your behalf (the data will still be in snapshots as applicable however).  We regret having to go this route at this late date, but the vendor did not have an effective way of determining if a file was good or bad after our first pass solution, so we had to go to this 'start over' method.  We apologize for the long delay between updates, and will update this every couple of days going forward as the copies work toward completion.

Update 8/17/2021 @ noon

Both syncs are successfully completed this morning.  We are going to ensure another two syncs complete, and then we are going ot disable syncs for 48 hours to do a few final checks on file integrity.

update 8/11/2021 @ noon

one sync has successfully completed.  The other continues to run.  

Update 08/09/2021 @ 6pm

We are still finding files to synchronize to ASB.  Work will continue through tomorrow.  

Update 08/05/2021 @1pm

One of the individual files fix scripts has completed, the other one may be close to finishing. We may have updates later today. 

Update 08/03/2021 @ 11am

We are still fixing individual files today in order to start what will be the final sync run. The bad drives have been replaced, so this should be the last step to completion.

Update 08/01/2021 @11 am

We restarted the syncs on Friday, but some files are still failing because we didn't fix them properly.  We are going through and fixing individual files now.

Update 07/30/2021 @ 2:00pm

The main scans ha

Who is Impacted? Users of the Turbo Service

Next Update: 9/17/2021 @~11am

Technical Details

Service Type: Production

Comments:

This took a long time to recover, but fixing the policies to be smaller and spreading the work contributed this to being finished much faster than expected.  

Report Additional Impacts

Contact the ITS Service Center for more information or to report additional impacts.