Beth Everett’s brushes with history have left her with better insights into worst-case scenarios.
Everett worked at Novartis AG in 2001 and saw the company’s global connectivity cut off when the network lines under the World Trade Center were destroyed.
Then she worked as CIO at Organon International Inc., which shared space in a Roseland, N.J., office park with a large financial services institution high on the list of potential terrorist targets. What would happen, she remembers thinking, if officials shut down the area and employees couldn’t get to work?
And now, as a consultant at Network Inference Inc. in Carlsbad, Calif., she wonders how a potential hazardous materials situation — a chemical spill or attack — might affect IT. Worries like these would have been unimaginable even five years ago. But world events have changed the way many CIOs think about their disaster recovery plans.
“There are definitely new threats out there,” says Steven W. Agnoli, CIO at law firm Kirkpatrick & Lockhart Nicholson Graham LLP in Pittsburgh.
Some CIOs are imagining potential disasters that go well beyond the everyday hiccups that can disrupt applications and networks. Others, recognizing how integral IT is to business today, are focusing on the need to recover instantaneously from any unforeseen event. Many are trying to do both. But CIOs agree that disaster recovery planning has taken on an immediacy that didn’t exist in the ’90s.
And they expect the threats to get worse. “The things you think about are, What will the virus/hacker people be able to do 10 years from now? What do I need to do to keep my capabilities ahead of the game?” says Rob Reeg, senior vice president of global operations at MasterCard International Inc.
“We need to be prepared for the next level of alert,” says Joseph Daluz, vice president and CIO at Computer Horizons Corp. in Mountain Lakes, N.J.
Old worries and new
When it comes to disaster recovery, the concerns are diverse. CIOs say they still worry about the traditional problems, from those manual errors and little snafus that can crash a system to natural disasters like fire and flood. But they’ve also added new concerns that range from catastrophic power loss and network attacks to employee sabotage and terrorist attacks. World events have changed the way many CIOs think about their disaster recovery plans. They agree that disaster recovery planning has taken on an immediacy that didn’t exist in the ’90s.Text
Raj Sampath, chief technology officer at LoanCity, a wholesale residential mortgage lender in San Jose, has considered just about all those scenarios. He says his biggest fear is a hacker attack. “It’s the unknown part — I don’t know how or when it’s going to be,” Sampath says.
He says a successful attack could not only corrupt his system but also compromise the personal data of the company’s customers. That’s why he has a firewall manager — a combination of hardware and software that acts as the first point of contact for the external world, manages the security of the company’s systems, protects servers from hackers and allows only certain specified transactions. Sampath also diligently keeps security software updated.
He worries about other scenarios, too, such as earthquakes knocking out his primary data center and employees downloading infectious programs. So he sets up redundant systems, continually updates antivirus software, monitors employee computer use and uses technology from San Jose-based Sonasoft Corp. that automates the backup and recovery process for Microsoft Exchange and SQL and Windows servers.
Other CIOs draw their new list of concerns from current events such as the 9/11 terrorist attacks and the August 2003 blackout that affected the Northeast. They ask, “What if someone sets off a dirty bomb? Or launches a bioterrorist attack? What happens if the country’s aging power grid fails?”
“It’s a different world. There are so many more things to consider than the traditional fire, flood and theft,” says Robert Rosen, a Bethesda, Md.-based CIO in the U.S. government and president of Share Inc., an IBM user group.
For example, as he toured a disaster recovery site last year, Rosen was impressed by its meticulous planning and features. Still, he was concerned when he heard a low-flying plane overhead, noting that a site’s proximity to an airport — even a small one — means there’s an increased risk of it being hit by a crashing aircraft. Disaster has taken on new meaning in this era of Sarbanes-Oxley. If regulators come knocking, they now expect companies to produce all the required data within hours – not weeks as they once did.Mike Kahn>Text
“There’s some risk there, even if the risk is pretty small. But it’s one of the things you factor into your analysis,” he says. “If I were [with] an organization that had to have total uptime no matter what, maybe I couldn’t live with that risk.”
But that doesn’t mean CIOs have to prepare for every scenario they can imagine. Companies usually make their disaster recovery decisions based on cost and risk analysis, says David Palermo, vice president of marketing at SunGard Availability Services LP in Wayne, Pa. They prioritize risks according to the likelihood of various scenarios and the effect each one might have. “At some point, you’re out of money and you have to make your choice,” Palermo says.
A key to risk analysis is that it’s not always about full-blown system failures; even small problems can have significant consequences.
“Disaster has taken on new meaning in this era of Sarbanes-Oxley and all this government regulation,” says Mike Kahn, managing director of The Clipper Group Inc., a technology acquisition consultancy in Wellesley, Mass. If regulators come knocking, they now expect companies to produce all the required data within hours – not weeks as they once did, he says.
But regulations aside, “data is now just so important to ongoing operations [that] if you lose data and it’s real-time data, that could actually impact your business,” says Kahn.
Anne M. Candreva, CIO at the Carnegie Library of Pittsburgh, agrees. “Technology is central to what we do, and if we lose our technology capability, we’ll come to a halt.”
Speedy recovery
The best-prepared executives recognize that speed is essential in recovering from whatever disaster might come to pass, Kahn says. That’s why there’s a trend toward enabling technology users to restore their own documents, so a lawyer, for example, can retrieve a brief that took weeks to write but an instant to accidentally delete.
And in addition to regularly backing up to tape that’s then stored off-site, companies are employing newer technologies to take snapshots — every five minutes, or every hour, depending on the business — to reduce the risk of potential loss, Kahn says.
But while the technology exists to ensure that a company doesn’t experience a catastrophic loss of data even if its systems go down, experts say executives need to make disaster recovery a priority — and fund it appropriately — if they want to guarantee business continuity during almost any situation. The best-prepared executives recognize that speed is essential in recovering from whatever disaster might come to pass.Text
“There’s no lack of products, and no lack of salespeople out there who want to sell them to you,” Rosen says. “So it’s not a technological issue any more. It’s really a management issue and a will to do it. One of the problems is it’s not cheap.”
Robert Borr, CIO at Quincy Medical Center Inc. in Quincy, Mass., has tried to protect h
