Recent Downtime…
Incident Report
Incident Date: July 6, 2010
Prepared By: Service Assurance Centre
HIGH LEVEL EXPLANATION
At approximately 12:45pm on July 6, 2010 power to our data centre in Thornhill was disrupted. Upon loss of power, we confirmed that all 3 generators had started properly. PowerStream was alerted and dispatched a repair team.
At about 1:25 our Generator “C” overheated and shut down. At about 1:40 our Generator “B” also overheated and shut down. Our Generator maintenance company arrived at 1:30 and commenced diagnosis on both systems.
We were unable to restart the generators before the UPS batteries were depleted causing loss of power to the data centres.
Our telephone system is connected to both generators so was operational until the second generator failure. Our phone system was down from approximately 2:05 to 2:35.
Generator “B” was restored at approximately 2:25pm. Incoming utility power was restored at approximately 2:37pm and generator “C” was repaired at 4:00pm.
Cause of the Incident:
- A loss of power from the utility combined with overheating of two generators caused loss of power to the data centre.
Call to Action
- Both generators have been repaired with interim fixes for additional venting to facilitate better cooling. Interim procedures to facilitate air flow have been established.
- A temporary generator is in transit and will be installed this evening.
- Our generating company is reviewing all generators for ability to operate properly during periods of extreme heat.
- We consider this a very serious incident and deeply regret the difficulties this has caused you. Senior management are initiating a full external engineering analysis of the event and of power and cooling design and operation. Actions from this analysis will be published upon completion.
If you have any questions or require any further information please contact
