While it’s important to commit a disaster recovery plan (DRP) to paper, it’s even more critical to actually test the process and ensure its effectiveness. Nowhere is this more important than when it comes to your company’s data centre emergency evacuation plans.
According to Ross Armstrong, senior research analyst with London, Ont.-based Info-Tech Research Group, no DRP is complete without data centre-specific evacuation guidelines. After all, the most important asset in your company’s data centre is the employees that help keep it up and running.
“An actual emergency situation is not the time to be finding out that your plans are ineffective, insufficient and have gaps,” Armstrong said. “So, while most companies have the proper procedures committed to policy, a lot of them are not testing the plan to see how it works.”
Armstrong added that testing should encompass all elements of the plan — not just the emergency notification and evacuation procedures. “It also means you should be testing your backups and seeing that you can restore them to a server in an off-site location.”
Gaining stakeholder acceptance of the plan is also crucial, he said. Evacuation plans involve a lot more than IT, and planning sessions should include co-ordination with health and safety committees, HR, and other applicable business units.
For companies that are having difficulty gaining corporate interest in DRP, being able to determine the probability of a potential disaster and what kind of effect it could have on the data centre is another important exercise, Armstrong said. After conducting a threat and risk assessment, you must communicate these findings to both your stakeholders and your staff. You must let them know that there are very real reasons behind the evacuation plans, he added. “Without that, people tend to tune this stuff out and say that fire drills remind them of school,” Armstrong said.
Threat and risk assessments should also include an element of scenario planning. “I’ve heard horror stories of companies that went so far as to purchase backup diesel generators to provide them with off-the-grid power in the event of an outage,” Armstrong said. But when a blackout affected half the city, the company went to get the diesel at the gas station and found the place was sold out. “Every company in the area was doing the same thing,” he said. “They were down anyway, despite their investment in the generators.”
To get a closer look at how effectively emergency notification and evacuation plans can be integrated into a DRP, we talked to two large North American enterprises that aren’t taking chances with employee safety. Enbridge Inc. and Waste Management Inc. face different challenges, but both have implemented plans that can deal with the risks most likely to affect them.
Enbridge
At Enbridge’s Toronto-based data centre, the company has set up the Enbridge East crisis response committee to go over all the different scenarios that can possibly happen in the event of a disaster.
“There are individuals within every department and floor designated to be part of that response team, and the evacuation exercise is practiced and improved upon on an ongoing basis,” said Peter Rapini, manager of IT technical services for Enbridge.
The team runs quarterly fire drills, which includes a full evacuation of the data centre. As part of the company’s crisis response manual, Rapini said, employees can read up on what they might be responsible for during an evacuation.
“There’s an actual binder that depicts who on the second floor holds the door open, the sequence in which people leave, and who’s responsible for checking the cubicles and making sure everybody is evacuated,” he said.
The most effective evacuation plans, according to Rapini, are the ones that don’t surprise employees. “Things will always go smoother if your employees know exactly which door to exit and where to go once they’re outside the building.
“The amount of effort you put into it beforehand will be how much of a pay-off you’ll see in the event of a disaster,” he added.
In addition to the crisis response team, Enbridge has also established a committee looking at avian flu and its potential effect on employees.
“We have a whole plan around that, if it were to take place,” he said. “Every department has a list of critical employees who could work from home. We’re actually deploying a lot more laptops to enable connectivity from home in order to facilitate a better work-life balance, and also leverage that technology in the event of a disaster.”
This type of planning can also prepare a company for a prolonged data centre evacuation, he added.
The company houses its production servers in its Victoria Park Building in Toronto, while the majority of its developmental servers can be found at nearby Thorold, Ont., less than 200 kilometres away. In the event of a disaster in Toronto, all the company’s applications and data can be remotely routed to Thorold and controlled at the company’s off-site emergency response centre, Rapini said.
“Once an evacuation is declared, even though we physically empty out the data centre, we don’t necessarily migrate activities to the backup data centre until we deem it can’t operate remotely anymore,” he said.
In 2003, the company endured a significant flood in the basement of its Victoria Park offices. The building was evacuated, but the data centre stayed up and running remotely.
The company now has a data replication strategy in place, which means that 80 per cent of all the company’s data is now backed up on disk, and directly to Thorold. Rapini said the contingency measure lessens the potential business impact of a disaster.
Ultimately, he said, companies should have the safety of their employees as their first and highest priority, and keep that in mind when developing a business resumption plan.
Waste Management
Gaining corporate buy-in for data centre safety was one of the most important factors in the success of Houston-based Waste Management’s evacuation procedures. And, unlike most emergency evacuation plans, this one has been put into practice for real.
Last September, Hurricane Ike caused an estimated US$31 billion worth of damage, and was the third most destructive hurricane to ever make landfall in the United States. On its path of destruction, it made a pass over Waste Management headquarters, causing a large-scale evacuation.
The company’s hurricane plan, which is referred to as the “corporate emergency response plan,” is carried out by Waste Management’s “Go Team.” The unit is comprised of employees across multiple business units, such as payroll, corporate communications and IT operations. It also has alternate employees fully trained and prepared in case a Go Tea