A message from Scott Claverie, Director of Computing and Communication Services:
For those of you who use our Microsoft Skype service for instant messaging, presence and telephony, we experienced an outage yesterday due to a Microsoft software issue. More specifically, the Microsoft security patch that was installed on the servers had an ill effect on the servers themselves and created a wide spread interruption for all of our users. We have reexamined our process for patching and have identified a flaw where this incident could have been avoided. We have already made the necessary adjustments to our process to prevent this scenario from happening again. We understand that with any enterprise communication tool we offer we must make every effort to ensure its reliability 24 hours a day, 7 days a week and Skype has no exception to that effort. Our promise to you is to always improve upon the services we provide in support of our campus mission.
I apologize for any inconvenience this may have caused to either you or your department.
Director, Computing and Communications Services
California State University, Chico
Unannounced changes to the Zoom service require an emergency reconfiguration and restart of our Shibboleth Identity Provider. This will occur at 12:15 PM today.
Authentication and authorization services to federated applications will be unavailable for 5-10 minutes. Users will lose any active single sign-on sessions. Shibboleth federated systems effected by this outage include TeamDynamix, CFS, Lynda.com, and CSYou.calstate.edu.
During scheduled maintenance on the Bay file server this morning there were technical difficulties resulting in some data loss. System administrators are currently working to identify the scope of the data loss. Please review your data on Bay and if necessary submit a request at https://support.csuchico.edu/TDClient/Requests/ServiceDet?ID=8065 for file recovery.
The Development and Training System (DTS) is experiencing intermittent issues and is currently unavailable. Technicians are working to resolve the issue but there is currently no estimate for full restoration of service. System Status on www.csuchico.edu/itss will be updated when DTS is back online.
Technicians are currently troubleshooting an issue with campus voicemail. Callers may be unable to check or leave messages, and call routing may not be working properly. We hope to have the issue resolved shortly.
IT Support Services is twitter at https://twitter.com/chicoitss. Follow us for tech tips, maintenance information, and outage notifications.
Bay will be down for emergency maintenance this weekend from 7:00pm Friday until the maintenance is completed. In the worst case, Bay will be unavailable until 7:00am Tuesday morning.
After the issue with the Bay file server this morning, Microsoft is recommending that we run a full disk check to make sure everything is working properly. Due to the size of the disks on the Bay server, the estimated time for completion of this disk check is unknown. Bay will be unavailable during the disk check.
Since this is likely to take a long time, we’re planning on starting the check at 7:00pm Friday evening. This will allow a full three-day weekend for the disk check to complete. At the end of the process, the server will be restarted. If the disk check is not completed by 7:00am Tuesday morning it will be canceled and the server will be restarted at that time and available shortly thereafter.
If you have a critical need to work on files on Bay over the weekend, please contact ITSS for options to have files available locally on your workstation. Please note that these options have some potential issues and will require care on your part to avoid data loss. We highly recommend that you just enjoy your three-day weekend if possible and save your Bay work for Tuesday.
Update 9:55 AM: Bay is back up. As near as we can tell, no corruption was found, so all files should be intact.
The campus file server (Bay) was patched this morning. During this process the operating system found some corruption on the main data drive. The system automatically started a disk check process in an attempt to repair the corruption. Bay will unavailable until this process is complete. Due the size of the drives on Bay, there is no estimated time that services will be available again; it is very possible that this could be a full day outage.
Check http://status.csuchico.edu for current information on system outages.
Some people are unable to log into Outlook Online (Outlook Web Access). This does not seem to be affecting access via Outlook or other email clients. Technicians are aware of the problem and working to correct it.
Update 7/28 4:30 PM: Outlook Online is now working.