The University’s systems, in particular the SUN-e-HR human resources system and selected portal applications, have been unstable over the past two weeks and inaccessible during this week. This was caused by a computer security threat which placed a high risk on our systems. However, the risk has now been averted and we can give more feedback on the initial problem.
The cause is the so-called “Poodle man-in-the-middle vulnerability” (see http://en.wikipedia.org/wiki/POODLE, https://securityblog.redhat.com/2014/10/20/can- ssl-3-0-be-fixed-an-analysis-of-the-Poodle-attack /).
Poodle is a vulnerability in computer systems that expose it to potential break-ins. This was discovered in the US in September and the first evidence of it’s existence was when Google adapted it’s Chrome and Mozilla it’s Firefox web browsers to withdraw the outdated SSL3 encryption, which posed the threat.
The direct result for the University was that Chrome and Firefox users could no longer access the SUN-e-HR system or portal applications. For example, students could not access their exam results and staff weren’t able to apply for leave.
We had no control over this. Chrome and Firefox were automatically updated by Google and Mozilla respectively.
Oracle released updates (“patches”) to address their part of the risk. In cases like these IT has no choice but to install the patches. The risk when not installing them is too great. At this stage users mostly had no access to systems already due to Chrome and Mozilla without SSL 3.
The upgrade was first tested in a development environment, and during a scheduled maintenance weekend (6/7 December) put into production. The upgrade’s installation went smoothly and has been tested as thoroughly as possible before the start of the week.
What we could not foresee, is that the Oracle upgrade would break Oracle’s own program code and configuration optimisations – this only became evident under the load when in production. Due to this, any process requiring an Oracle login, failed.
IT systems staff worked through the night and, with the assistance of Oracle, tried to locate the cause. It was first identified on Thursday, December 11 and could then be corrected within two hours.
The impact of the upgrade on staff and students was larger than expected. If the upgrade would have been postponed until after recess, the error would have only occurred a week before registration, which would have caused a bigger crisis.
It is not possible to schedule upgrades of this magnitude in recess time due to the interdependence of systems and the amount of people needed for installation and testing – from system administrators to users.
Today’s computer systems are significantly more complex than a decade ago. The result is that errors are inevitable.
We can only, to the best of our abilities, try and manage incidents like these. We learn from our mistakes – and the most important part is to communicate. IT will also implement an alternative backup plan for the future.
Thank you for your understanding and support.
