At BusinessWeek’s CIO Summit here this morning, Steven Ruegnitz tried his best to move beyond emotion to present his peers with an analytical look at how his company, Morgan Stanley, Dean Witter & Co., used its systems to respond to and recover from the Sept. 11 terrorist attacks on the World Trade Center.
But as Ruegnitz, managing director of enterprise applications security and telecommunications for Morgan Stanley’s institutional securities business, explained how the company’s IT infrastructure and emergency response plan saved thousands of his colleagues’ lives, he had trouble separating the emotional and analytical.
“I don’t know that any of you have lived until you thought that 3,000 of your friends are dead,” he said.
But, agreed Ruegnitz and Paul Lacouture, president of network services at Verizon Communications, which had offices in and around the Twin Towers, there was no time for emotions or even for planning that day. The reason their companies recovered as quickly and completely as they did was because of disaster plans their businesses already had in place.
The most important success factor: employees at both companies were empowered ahead of time to take action. Lacouture stressed the need for field operations leaders and communications leaders who can devote their full attention to those tasks.
Ruegnitz is convinced that the quick action of leaders saved employees’ lives. “We didn’t talk about it,” he said. “We kicked into gear and we activated.”
Using internal charts and graphs, Ruegnitz illustrated how exactly one minute after the first plane hit 1 World Trade Center at 8:48 a.m., Morgan Stanley’s security staff began evacuating its 3,828 employees from 2 World Trade Center. By 9:03 a.m., the second plane hit 2 World Trade Center.
At 9:15 a.m., senior management was moved to a command facility on Seventh Avenue. Ten minutes later, Morgan Stanley’s Varick Street backup system was activated and its Harborside, N.J., offices started buying emergency IT equipment.
At 9:50 a.m., Morgan Stanley evacuated its last employee from 2 World Trade Center. Nine minutes later, 2 World Trade Center collapsed, said Ruegnitz.
Morgan Stanley lost six employees in the disaster — four of whom had gone back in to make sure everyone was out. Another 31 employees were treated for injuries, said Ruegnitz.
Both companies credited their speedy recovery to underlying telecommunications infrastructures, said Ruegnitz and Lacouture.
Two years ago, Morgan Stanley changed its telecommunications policy to emphasize disaster readiness over cost-efficiency. That meant the company’s telecommunications infrastructure was built to survive daily disruptions, said Ruegnitz.
Of Morgan Stanley’s entire technology infrastructure, telephone service was most critical just after the attacks, said Ruegnitz. The company had its own emergency telephone lines installed so employees could get outside lines.
And because Morgan Stanley had four Internet service providers in New York alone, it was able to stay online throughout the disaster (although three of the four lines went down, something Ruegnitz said was a little too “white knuckle” for him – he has since signed on with a fifth ISP).
Telecommunications redundancy was key, said Lacouture. At Verizon’s office building across the street, the Manhattan 911 center was destroyed. But emergency calls still got through because Verizon had a backup center in place.
He also stressed the need for an equipment cleaning plan. While Verizon’s building remained standing, the offices were nearly destroyed by smoke and water damage, said Lacouture. “Every piece of electronics was coated with dust, debris and water,” he said.
Although about 90 percent of the equipment was eventually replaced, Lacouture said much of it was cleaned off enough that it could be put to use for short-term recovery. “Don’t give up on it,” he said. “Even though it looks terrible.”
Lacouture and Ruegnitz shared other lessons learned during and after the attacks:
— Turn customer call centers into emergency contact centers. Morgan Stanley, for instance, had call center reps call all of its World Trade Center employees.
— Ensure access to emergency areas.
— Prioritize which systems to restore first.
— Maintain constant communication/coordination to update priorities.
— Upgrade physical security at key locations. Companies may have secure rooms with pass-code entry, but make sure the code isn’t openly available.
— Keep equipment up to date. Discontinued equipment is more difficult to replace — a fact that can serve as a helpful tool in rationalizing upgrades to CEOs, said Lacouture.
— Design next-generation networks with different hub locations.
— Keep current records backed up. Without that, restoration time can double or triple.
— Make sure employee assistance programs can expand resources in emergencies.
— Use diverse office locations to help employees get back to work quickly.
Verizon is still 12 to 15 months away from full recovery, and its anticipated disaster-recovery expenditures will total US$1.5 billion, said Lacouture.
Ruegnitz said Morgan Stanley’s previous disaster recovery plan was centered around a single incident that would put the company out of business for three to four days. Its new plan, he said, is aimed at keeping operations running through any type of disaster.
He said Morgan Stanley is spending an additional $30 million on business continuity planning this year, on top of its regular budget.
“We will likely face this again,” said Ruegnitz. “We hope not at this magnitude.”