The University’s systems, in particular the SUN-e-HR human resources system and selected portal applications, have been unstable over the past two weeks and inaccessible during this week. This was caused by a computer security threat which placed a high risk on our systems. However, the risk has now been averted and we can give more feedback on the initial problem.
The cause is the so-called “Poodle man-in-the-middle vulnerability” (see http://en.wikipedia.org/wiki/POODLE, https://securityblog.redhat.com/2014/10/20/can- ssl-3-0-be-fixed-an-analysis-of-the-Poodle-attack /).
Poodle is a vulnerability in computer systems that expose it to potential break-ins. This was discovered in the US in September and the first evidence of it’s existence was when Google adapted it’s Chrome and Mozilla it’s Firefox web browsers to withdraw the outdated SSL3 encryption, which posed the threat.
The direct result for the University was that Chrome and Firefox users could no longer access the SUN-e-HR system or portal applications. For example, students could not access their exam results and staff weren’t able to apply for leave.
We had no control over this. Chrome and Firefox were automatically updated by Google and Mozilla respectively.
Oracle released updates (“patches”) to address their part of the risk. In cases like these IT has no choice but to install the patches. The risk when not installing them is too great. At this stage users mostly had no access to systems already due to Chrome and Mozilla without SSL 3.
The upgrade was first tested in a development environment, and during a scheduled maintenance weekend (6/7 December) put into production. The upgrade’s installation went smoothly and has been tested as thoroughly as possible before the start of the week.
What we could not foresee, is that the Oracle upgrade would break Oracle’s own program code and configuration optimisations – this only became evident under the load when in production. Due to this, any process requiring an Oracle login, failed.
IT systems staff worked through the night and, with the assistance of Oracle, tried to locate the cause. It was first identified on Thursday, December 11 and could then be corrected within two hours.
The impact of the upgrade on staff and students was larger than expected. If the upgrade would have been postponed until after recess, the error would have only occurred a week before registration, which would have caused a bigger crisis.
It is not possible to schedule upgrades of this magnitude in recess time due to the interdependence of systems and the amount of people needed for installation and testing – from system administrators to users.
Today’s computer systems are significantly more complex than a decade ago. The result is that errors are inevitable.
We can only, to the best of our abilities, try and manage incidents like these. We learn from our mistakes – and the most important part is to communicate. IT will also implement an alternative backup plan for the future.
Thank you for your understanding and support.
Die Universiteit se stelsels, en spesifiek die SUN-e-HR menslike hulpbronstelsel asook die portaal, was die afgelope week of twee onstabiel en die grootste deel van hierdie week buite aksie weens ‘n rekenaar sekuriteitsbedreiging – wat ‘n groot risiko op ons stelsels geplaas het. Die risiko is nou afgeweer en ons gee graag meer agtergrond:
Die oorsaak is die sogenaamde “Poodle man-in-the-middle vulnerability” (sien http://en.wikipedia.org/wiki/POODLE, https://securityblog.redhat.com/2014/10/20/can-ssl-3-0-be-fixed-an-analysis-of-the-poodle-attack/ ).
Dit is ‘n leemte in rekenaarstelsels wat dit blootstel aan inbrake. Dis in September 2014 in die VSA ontdek en van die eerste gevolge was dat Google hul Chrome webblaaiers en Mozille hul Firefox aangepas het om die verouderde SSL3 enkripsie, wat groot leemtes het, te onttrek .
Die direkte gevolg vir die Universiteit was dat Chrome en Firefox gebruikers nie meer die SUN-e–HR stelsel of portaal sou kon gebruik nie. Studente kon byvoorbeeld nie by hulle punte kom nie, en personeel nie verlof insleutel nie. Ons het geen beheer hieroor gehad nie. Chrome en Firefox is outomaties deur Google en Mozilla onderskeidelik aangepas.
Oracle het opgraderings (“patches”) vrygestel wat hulle deel van die risiko aanspreek. IT het in sulke gevalle nie ‘n keuse nie – ons moet dit installeer. Die risikos is te groot om dit nie te installeer. Gebruikers het op daardie stadium grootliks nie meer toegang tot die stelsels gehad nie (weens Chrome en Mozilla sonder SSL 3).
Die opgradering is eers op die ontwikkelingsomgewing getoets en daarna tydens ‘n geskeduleerde onderhoudsnaweek (6/7 Desember) op produksie uitgevoer. Die opgradering se installasie het goed verloop en is so deeglik as moontlik getoets voor die begin van die week.
Wat ons nie kon voorsien, is dat die Oracle opgradering Oracle se eie program kode en opstelling optimisasies sou breek – wat eers onder produksielas sigbaar geword het.
Die simptome was dat enige proses wat die Oracle login gebruik, gefaal het. IT se stelselpersoneel het nagte deurgewerk en met die hulp van Oracle die oorsaak probeer opspoor. Dit is eers op Donderdag 11 Desember geïdentifiseer en kon daarna binne 2 ure reggestel word.
Die impak van die opgradering was groter as verwag, op personeel, sowel as op studente. Sou die opgradering egter uitgestel word tot na die resestyd, sou die fout ‘n week voor registrasie sigbaar geword het en was dit ‘n veel groter krisis. Dit is egter nie meer moontlik om sulke werk in die resestyd te doen omdat die interafhanklikheid van stelsels ‘n groot span mense kort vir die installasie en toets; vanaf stelseladministrateurs tot gebruikers.
Vandag se rekenaarstelsels is aansienlik meer kompleks as ‘n dekade gelede. Die gevolg is dat foute onvermydelik voorkom.
Ons kan net na die beste van ons vermoëns rondom dit bestuur. Ons leer uit ons foute – en kommunikasie is kardinaal. IT gaan egter ook ‘n beter noodplan vir sulke gevalle ontwikkel.
Dankie vir die begrip en ondersteuning.
