At 2 a.m. on Aug. 27, two days before Hurricane Katrina devastated the Gulf Coast, Tim Babco grabbed a red binder containing the latest version of SCP Pool Corp.’s disaster recovery plan, put his dog and cat in the car, locked up his house and drove 500 miles from Covington, La., to the company’s emergency operations center in Dallas.
Babco, senior director of IT at Covington-based SCP, a US$1.3 billion wholesale distributor of swimming pool supplies, had relocated his operations on two earlier occasions when hurricanes threatened neighbouring New Orleans. Both of those storms turned out to be near misses, but Babco said this week that the practice runs helped him fine-tune his plan for when the real thing finally hit.
“People would be lying to say these things always go perfectly,” Babco said. “But has it succeeded in allowing our business operations to continue to buy, sell and distribute products? It certainly has, and that’s what disaster recovery is all about.”
However, the kind of disaster recovery planning done by Babco isn’t universal — especially for a calamity as massive as this week’s. Gartner Inc. analyst Simon Mingay said that about 40 percent of Fortune 1,000 companies aren’t prepared for a regional disaster. And small and midsize businesses are even less ready, he added.
“Obviously, we’re looking at a level of devastation here that few would have considered,” Mingay said. “But most still believe that these are things that don’t happen to them.”
Mingay said companies that have prepared properly for disasters, such as SCP, have extensive emergency communications plans, hot sites from which they can continue business operations for an extended period and some level of IT systems redundancy outside of their headquarters region.
But even companies that are well-prepared might not take into account a crisis of the magnitude as the one spawned by Katrina and the flooding that followed the storm.
For example, John Wade, CIO at Saint Luke’s Health System Inc. in Kansas City, Mo., said his contract with a disaster recovery vendor allows the IT department to work out of a hot-site facility for up to six weeks. But in a disaster similar to Katrina, the six-week limit could “pose a hardship,” Wade said.
Joe Hartman, an application development manager at HydroChem Industrial Services Inc. in the Houston suburb of Deer Park, said the company’s disaster recovery plan involves moving corporate operations to facilities on the city’s north side. “Anything that would completely wipe out Houston would leave us in a bad way,” he added.
Babco said SCP’s headquarters were unscathed by Katrina and the flooding that followed but lost all data and voice communications links. Fifteen of the company’s 40 IT workers have relocated to Dallas for now, while another 15 were dispersed among offices around the country.
Three years ago, Babco decided to flip-flop SCP’s primary and secondary data centers, placing its critical systems in the Dallas facility, which is run by Houston-based VeriCenter Inc. He also created an IT disaster recovery team for functions such as coordinating help desk services and relocating hardware. In addition, SCP has set up an internal business-continuity Web site that posts corporate alerts and provides toll-free numbers and an extensive list of employee contact information.
Not everything went completely smoothly this week. On Tuesday, Babco made an unplanned trip from Dallas to SCP’s headquarters to retrieve 12 application servers, including ones supporting human resources and e-mail.
He also said he has been unable to get a response from Boston-based Iron Mountain Inc., which handles off-site data storage for SCP. The area around Iron Mountain’s Kenner, La., facility is inaccessible, said Babco, who added that he would have liked to have had SCP’s backup tapes sent to Dallas in advance of the storm.
“They’ve not been able to provide us any information about when they will be able to get tapes out of their facility,” Babco said. “I think they weren’t proactive in seeing the event looming and getting the tapes out of harm’s way.” He vowed to switch to a different storage archiving vendor.
Ken Rubin, senior vice president of marketing at Iron Mountain, said the company moved requested data tapes to customers’ facilities beginning Aug. 26. It also put the tapes in its sites on higher shelves. “We don’t know once the levees broke what impact or not there is to any of the contents of the buildings,” Rubin said. “We’re trying to get in.”
Richard Kerley, chief operating officer at Shutts & Bowen LLP, said three of the Miami-based law firm’s offices in Florida were hit by multiple hurricanes last year. “We could have had a situation where our main [data center] site in Miami was taken out, and the cold site in Orlando was taken out, and we were decapitated,” Kerley said.
Six months ago, Shutts & Bowan signed up for data center space in a reinforced concrete bunker owned by Terremark Worldwide Inc. in Miami. Kerley said the law firm’s neighbors in the bunker include the State Department, the Department of Defense and the Federal Emergency Management Agency.
( Carol Sliwa contributed to this report.)
Hot sites get heavy usage
Saint Luke’s Health System conducted a test of its disaster recovery procedures in August and planned to do another one late this month, said CIO John Wade. But in a sign of Hurricane Katrina’s effect on IT operations, the health care provider’s hot-site vendor canceled the September test this week because so many of its other clients have declared disaster situations, Wade said.
Bob DiLossi, manager of SunGard Data Systems Inc.’s crisis management center, said the Wayne, Pa.-based vendor has contracts for disaster recovery services with 146 companies in the area hit by Katrina. Of those, 24 had declared disasters as of Thursday, and 120 had issued disaster alerts, DiLossi said.
SunGard has three emergency data centers in Texas, and it’s sending mobile data centers to the Gulf Coast region. In addition, DiLossi said SunGard has been trying to reach out to customers in the New Orleans and Biloxi, Miss., areas. But it has had trouble making contact with them because of telecommunications outages.
Belinda Wilson, executive director of business continuity services for the Americas region at Hewlett-Packard Co., said most of the customers seeking HP’s help are asking for offices and work areas that they can use. “It’s really been tough,” she said. “It’s just chaos.”
The job ahead appears enormous for IT, said Bill Oates, CIO at Starwood Hotels and Resorts Worldwide Inc. Disaster planning typically “isn’t assuming that you’re going to have extended problems” of the sort caused by Katrina, he said.
(Lucas Mearian, Carol Sliwa and Todd R. Weiss contributed to this report.)
HP forced to put off user conference
Bowing to the flooding catastrophe that has befallen New Orleans, Hewlett-Packard Co. postponed the inaugural HP Technology Forum user conference it was scheduled to hold in the city next week.
HP initially said in a statement posted on the conference Web site that it was monitoring the situation in New Orleans and assessing the viability of going ahead with the event. But as the extent of the flooding became apparent, HP updated the statement, saying it was working “to determine a time in the fall when we can host the conference.” <asset width="2
