As we watched the horrors of Hurricane Katrina unfold from the safety of our Canadian homes, it was tempting to believe that such a thing couldn’t happen here. But disasters do happen — everywhere — and Canada is certainly not immune. In 2003 Hurricane Juan ripped through Atlantic Canada, causing extensive damage and killing at least eight; that same year large areas of the country were plunged into darkness by the Northeast Blackout, which left 50 million people without power throughout North America; and in 1998 the Great Ice Storm devastated parts of Quebec and Ontario, taking a huge toll on the economy of both provinces.
From a business perspective, any of these disasters could have been devastating, which is why smart companies take disaster recovery very seriously.
Canadian companies and Katrina
Because many Canadian companies have operations in the U.S., several Canadian firms were directly impacted by Katrina. One was Trimac, a bulk transportation company that hauls unpackaged products such as petroleum, chemicals and cement, usually from one manufacturer to another. While computer operations and head-office administration is located in Calgary, the company has more than 100 branch terminals and other assets across North America.
With Katrina threatening, measures were taken at about half a dozen Trimac branches in Louisiana and Texas to protect staff and prevent disruptions in deliveries and orders. Trimac helped branch staff in backing up data onto tapes, which were removed off site or sent to Calgary. Branch systems were shut down to ensure no loss of critical information, and vital equipment was made as safe as possible before staff evacuated the premises.
“Every potential disaster is slightly different,” says Trimac CIO Janet Topic. “We have emergency plans regarding what to do in the event of these threats so our operation keeps running, staff are safe, and customers are serviced.”
Every Trimac branch has a standard protocol of what to do in the event of a disaster. Regional offices are responsible for presenting branches with those directions. The disaster plans detail who will provide backup service to the customers so there are no service failures if a branch is unable to conduct business. The plans outline what to do with the drivers, trucks and branch personnel.
Topic says that the company’s plans are geared to all manner of disasters, not just those as dramatic as Katrina. If a fire next door rendered a branch inaccessible, dispatchers could go to a place with Internet access or the closest Trimac branch and perform their duties from there. If dispatch wasn’t available, another branch would fulfill the orders until full service was restored. Other branch staff could log onto the network from PCs and access the services they need from central IT systems at head office.
Trimac’s detailed IT disaster recovery plan involves the stationing of development and test servers in an offsite location remote from the facility housing production servers. Twice a year the company practices bringing those test and development servers up to perform as production equipment in the event of an emergency in the main data centre.
The last time the IT staff conducted such a mock disaster they went to another company in Calgary and brought the systems up from that location. It’s part of a reciprocal disaster recovery agreement Topic has arranged with another CIO. Topic describes the IT recovery plan as conservative, affordable and one that minimizes risk. It has been in existence for three years and took about the same time to get into place.
“We’re getting better at the recovery process because we are very committed to ensuring our business resumption plan is viable. As new systems come along there are different requirements and priorities, so it’s important to keep testing. We’ve integrated the disaster recovery considerations as deliverables in our normal development cycle or product installation cycle.”
Topic believes it’s a good model, and one that can be applied in other areas of the business. Taking a cue from IT, Trimac’s head office is now working on a detailed business resumption plan.
Disaster-proofing the workforce Central to the disaster recovery philosophy of Brampton, Ontario-based Nortel Networks is high system availability and worker mobility. These factors came into play in Nortel offices in regions affected by Katrina, where the company had employees working out of their homes. This extended even to North Carolina, where the governor urged people not to drive because of the gas shortages.
With the SARS outbreak in Toronto and the Northeast Blackout, Nortel had much of its workforce working out of their homes, says Don Dixon, vice-president network and security services. Employees’ home computers were equipped with software to tunnel into the company’s network securely and access everything they needed to be productive. Part of this mobility strategy flows from so many employees having wireless devices and portable computers to begin with.
By being in the business of designing networks for so long, Nortel has been able to build smarter designs into its own systems, he says. It architects the environment to leverage not so much redundancy as resiliency, whether with servers or networks with diverse circuitry. At the end of the day, it pays for itself with higher availability and reliability, two attributes that go a long way toward disaster recovery.
“I don’t want to downplay the need for business continuity planning, proper scenarios, correct documentation and contact lists and everything else. You absolutely have to have them. But in a lot of cases you can minimize the amount of times you get into those scenarios just by the proper design and architecture of your services.”
Disaster planning at Mountain Equipment Co-operative
Many companies have recently turned to more comprehensive disaster recovery preparations because of such events as the September 11 terrorist attack, but business continuity has been a constant pursuit for Mountain Equipment Co-operative (MEC), Vancouver, since the 1990s. It provides outdoor gear for wilderness activities.
MEC was formed in the early 1970s by avid mountaineers who were having trouble getting climbing equipment in Canada. It now has two million members in 128 countries, but it is most widely known in Canada. The co-op is a multi-channel retailer with about $200 million annually in sales. It has stores in major centres across Canada, has operated an ecommerce site since 2001, and maintains a call centre for phone orders.
In the 1990s MEC commissioned IBM Canada to do a thorough business impact assessment. From that, it evaluated all business functions to assess how long the company could be down before sustaining unacceptable losses. It determined that all systems related to critical business functions had to be completely restored within 48 hours.
It secured a hot site contract with IBM where its systems are backed up on similar systems at the IBM data centre in Markham, ON. All the core inventory, financial, payroll and human resource systems that must be restored in 48 hours sit on this iSeries technology. </
