Terry Rhode remembers Saturday Dec. 9, the day of his 2000 office Christmas party, all too well — it was that day he received a phone call IT managers most dread.
The caller told him that the head office of Roseneau Transport Ltd., an Edmonton freight shipping company where Rhode worked as assistant controller and IT manager, was on fire. Even as he drove toward the office, he was convinced the call was an elaborate joke.
“I didn’t believe it until I turned the corner and saw the fire trucks and smoke,” he said.
Rhode later learned that a power surge protector in the accounts receivable office arced, starting a $2-million fire that quickly destroyed the entire office, including its main AS/400 unit, as well as all other servers, PCs and telecom equipment. Its entire back office infrastructure was gone.
As the caller earlier told Rhode, “Now (you) have a chance to put (your) disaster recovery plan to the test.”
Roseneau did have such a plan in place, albeit one that was neither formalized nor contained in a document. Still, Rhode saw to it that tapes and data containing daily data backups from the AS/400 were stored off-site. As a result, the company lost only 10 hours of data.
Rhode had also been careful to choose only those suppliers with the ability to respond quickly to outages. With the help of IBM and Roseneau’s main computer reseller, Rhode got his hands on a temporary AS/400 the next day and had remote locations up and running within 36 hours, and all systems up and running in 48 hours. The newly located office was back in business 72 hours after the fire.
It wasn’t all smooth-sailing: the biggest hole in Rhode’s plan was a lack of backups for the office’s PCs, something that is today done routinely.
However, it appears few companies are as well equipped to handle disasters as was Roseneau. A survey released last month found that 43 per cent of companies worldwide were “unprepared” to handle a major fire disaster.
In the event of a fire, for instance, 45 per cent of the 1,259 IT managers polled admitted they had no idea how long it would take to get systems up and running in the wake of a major outage.
Thirty-eight per cent of respondents said they maintain a detailed disaster recovery plan that accounts for both technology and business continuity. Only slightly more use some form of software to help in those efforts.
The study, backed by enterprise software make Veritas Corp. and conducted by a British polling firm, also found that more than half of respondents keep their backup plans in their data centres. Fifty per cent say they had to, at some time, employ some or all of their backup plan in the wake of an outage or disaster — caused mostly by hardware or software failure, followed by viruses or hackers, natural disasters and “internal threats,” including employee actions and man made disasters, such as war or terrorism.
A Canadian survey conducted by Athabasca University (in conjunction with IT World Canada affiliate CIO Canada) and released in November reported similar numbers — it found that 44 per cent of the more than 2,500 senior IT managers polled did not believe their organizations had a disaster recovery plan in place.
According to Rhode, even a well-documented plan can account for only some of the challenges that arise after disaster strikes. “(Our recovery) was possible because of the partnerships I had in place with IBM and our computer supplier,” he said. “They weren’t the cheapest … (but) they could do what they say they can do.” That’s why he recommends discussing the topic during contract negotiations, and to avoid simply looking for the least-expensive option.
Rhode also recommends taking into account the obvious but also easily overlooked items that become crucial in the event of a disaster, such as the availability of an updated list of staff and their contact information; a list of current suppliers and what items they cover; and a supply of corporate paperwork and accounting items, such as cheques, so suppliers can be paid in a pinch.
The executive summary of the Veritas report can be found at: http://eval.veritas.com/mktginfo/products/White_Papers/High_Availability/dynamic_markets_executive_summary.pdf.