In the year since the terrorist attacks, network executives have pragmatically revisited their disaster-recovery plans and core network technologies, haunted by visions of last Sept. 11.
“I don’t think anybody viewed the world prior to 9/11 as we do now,” says Denny Groh, acting associate commissioner for service delivery at the General Services Administration, which delivers IT and telecom products and services to federal agencies. “No one thought that two buildings could bring that kind of chaos economically, or technologically, to our country. We found out differently.”
“Since Sept. 11 there is more of a focus on preparation,” says Larry Godec, CIO at First American Title Corp. in Santa Ana, Calif. “There’s more a sense of things can happen. And there is more awareness about the importance of business resumption and disaster recovery.”
Network executives have begun turning the trend toward consolidation on its head by looking for ways to spread out critical data, IT systems and employees. Other changes: Businesses that turned to videoconferencing to reduce corporate travel after the attacks seem to have stuck with it; others have bought into voice over IP as being more resilient than traditional telephony.
At the same time, the country’s major carriers have been reassessing their networks and stretching to meet the changing demands of their corporate customers.
Backing up business
Dave Purdy, director of business continuity at storage vendor EMC Corp., says his company has seen a change in the way companies look at business continuity.
“Even customers that thought they were doing things well are revisiting their assumptions,” he says. “People historically would attempt to identify the risk areas – fire, earthquake, power outage. Sept. 11 ingrained in people: ‘Don’t try to plan for a particular type of outage. Plan for the impact of the outage.’ It’s a different mindset.”
Rich Arenaro, corporate vice president and general manager of Windows and Unix systems at Commerzbank AG in New York City, agrees. Commerzbank, located in 2 World Financial Center, is across the street from where the towers stood. Commerzbank uses EMC’s Symmetrix Remote Data Facility (SRDF) software to replicate data from one storage-area network in the city to another in Rye, N.Y., about 30 miles away. The company had completed the bulk of the project just prior to Sept. 11 and got most of its critical systems back online within four hours after the attacks.
Even so, Arenaro says since Sept. 11 there has been a greater focus on ensuring not only that data will be protected but also that business will be able to continue when disaster strikes.
“Our strategy had been based on a false one-to-one ratio of technology, meaning if I buy a server here and one for Rye, I’m protected,” Arenaro says. “The reality is when you are faced with that situation, having hardware really is the least of your worries. It’s really having your data and your systems available and ready to use.”
Enabling business to continue immediately was not the focus for law firm Harris Beach LLP, which had an office on the 85th floor of World Trade Center Tower 2. The firm lost six employees, its office and its files during the attacks.
“As a law firm, if we suffer a cataclysmic disaster, we don’t need to be up in an hour,” says Alan Winchester, a technology partner at the firm. “People aren’t ready to work in a day or two. . . . The only thing you want to get going quickly is your e-mail and telephone [so people can stay connected].”
Harris Beach also went to work installing a system to scan all incoming documents and turn them into digital files it stores at local offices and at headquarters in Rochester, N.Y. The firm had considered a paperless system before the attacks, he says, but the project never got going because of the time and energy associated with scanning every document.
While each business must determine for itself what degree of business continuity and disaster recovery is necessary, some network executives say that getting the financial go-ahead to fund such projects has gotten much easier since Sept. 11. This despite an economy that IDC says has let IT spending this year sneak up just 3 percent to US$436 billion.
“What Sept. 11 did was open the eyes of executives and make it easier for me to justify security perimeters and projects,” First American Title’s Godec says. Godec is overseeing a revamp of his company’s network to support an enterprisewide IP-based application and a new redundant site in Dallas. “It really made it easier to push a lot of the initiatives that were already on the table,” he says.
A job for new technologies
While disaster recovery and business continuity may have received the bulk of post-Sept. 11 attention, there are many creative initiatives under way at companies that have turned to technologies that were little used before the attacks.
At chemicals and materials company Grace in Columbia, Md., CEO Paul Norris clamped down on corporate travel following the attacks. As a result, employees who are spread out in offices in the U.S., South America, Asia and Europe were forced to make use of the company’s Polycom Web conferencing software.
While the technology had been available to employees for about four years, its use since Sept. 11 has spiked. Just this July, the company held 42 meetings via the Web, avoiding international travel to meetings that could have cost as much as $40,000 apiece. Guy Welty, manager of global media networks and collaborative services for Grace, says videoconferencing is expected to save the company more than $1.2 million in the corporate headquarters alone this year.
Another technology that is getting more attention is VoIP. At the Benjamin School, K-12 private school in North Palm Beach, Fla., with about 25 buildings on a campus about the size of a city block, a new VoIP system was installed in the wake of Sept. 11.
While the school was not directly affected by the events in New York and Washington, D.C., it found its phone lines down because of a lightning strike with no way for parents to reach their children on the day of the attacks. With 3Com Corp.’s VoIP system that was installed in February, the school has put phones in each classroom and has updated its disaster plan.
“We now have a mass communications system where we can page all the phones simultaneously, as well as the outside horns and inside speakers,” says Dustin Durbin, technology coordinator at the school. “We haven’t had any issues since we installed the system, which runs over a managed network rather than over a bunch of analog cables in the ground.”
Security is another hot topic, especially because the focus now is on geographically distributed network storage and data replication.
“People will be moving to online backup and mirroring over a broad geographic area. That means information will have to traverse electronic networks, and that’s an unsafe environment,” says Bill Crowell, CEO of security firm Cylink Corp., and formerly of the National Security Agency, where he oversaw information security systems development and the agency’s operational intelligence mission. “Security comes to the fore.”
Newmont Mining Inc. in Denver, with offices worldwide, was already looking at ways to better protect its network before Sept. 11.
“What Sept. 11 did was reinforce that security had to become more of a key piece of what we do rather than something done on the side,” says Dan Kesl, manager of information security. “The security of our information and our employees around the world is a high priority.”
Because Kesl wanted round-the-clock monitoring of firewalls, intrusion detection, antivirus and other services, he decided to outsource that responsibility to managed security service provider Guardent.
“To train and bring up internal staff would have been very time-consuming and very expensive,” Kesl says. “To get the service levels we needed we had to go outside.”
One more area that is getting increased attention since Sept. 11 is how to ensure communication within companies and between the public and private sectors in the event of disaster.
Technology firm Candle Corp. created the National Center for Crisis and Continuity Coordination in February to provide collaboration assistance to the private and public sectors during emergencies.
“We’re not replacing the disaster-recovery firms,” says Jim Montagnino, the center’s general manager. “What we’re involved in and what we’re finding is, especially in the wake of 9/11, businesses are realizing they need ways of getting information from the police, fire departments and emergency personnel.”
“And the public sector needs to know when businesses have closed down and evacuated,” Montagnino says.
Carriers respond
As businesses modify their network strategies, carriers are seeing a change in customer demand. Cable & Wireless PLC says it has seen demand for its IP VPN servicedouble since the end of last year; WorldCom Inc. reports seeing double-digit growth in demand for collocation and managed hosting in the past year, a trend it attributes to customers’ heightened sense of security.
“The big change since 9/11 is customers are actually looking for redundancy,””says Audrey Wells, senior manager of global VPN services at WorldCom. “We don’t need to tell them why they need it.”
At the same time, carriers have been reviewing their own network architectures. The hardest hit of all the carriers on Sept. 11 was Verizon Communications Inc., which had its central switching station next to the World Trade Center. Verizon has completed most of the repairs to its heavily damaged West Street central office and to lines that were cut in lower Manhattan, says Marlene Grant, executive director of network planning for Verizon.
The carrier has installed more than 40 miles of new fiber downtown, and Grant says that voice and data service there has returned to normal levels. But the phone giant still has some significant work ahead of it to get the network permanently restored. Verizon estimates it will not be completely finished rebuilding its network until 2004.
Three of the West Street switches that were damaged in the Sept. 11 attack are being replaced – Verizon already has replaced one of them – without disrupting customer service, Grant says.
Finally, Verizon is building more reliability and survivability into its network, and part of that effort involves reducing the number of circuit hops. For example, Grant says a circuit terminating in White Plains, N.Y., north of Manhattan, might typically be routed through a central office in the Bronx and two more central offices in upper Manhattan before reaching lower Manhattan.
“We’re looking now at taking that circuit and making it maybe one, or at most, two hops,” she says.
Up to 20 percent of the circuits that ran through the West Street central office were pass-through circuits, meaning they didn’t need to run through the office, and Verizon wants to significantly lower that number in all its lower Manhattan central offices, Grant says.
Another improvement Verizon is making is adding more SONET rings, which are designed to survive cuts, in lower Manhattan. Verizon is not only adding more rings to the company’s backbone network but also is pushing them to customer premises wherever possible.
Other carriers, including WorldCom and C&W, say they were satisfied with how their network architectures performed. Both, however, say they underwent extensive reviews following the attacks to see where improvements could be made.
WorldCom has created an organization dedicated to business continuance and emergency response.
The carrier’s hazardous materials team, which can be sent in to fix equipment when there are biological dangers, also is available nationwide and has been trained to deal with new threats, such as anthrax.
Another big focus among the carriers is an effort to ensure smooth partnering in the event of another disaster. On Sept. 11, the carriers helped each other by moving customers onto operating lines.
“A lot of carriers have come together because they realize that when an incident like that occurs we need to work closely together to get communication established. That way we can bring up our customers and get some kind of service restored to them in a more timely manner,” says Jim Weber, vice president of network operations at C&W.
Communication and diversity might be the key to how businesses, carriers and public agencies can steel themselves against network problems in the event of future disasters. The GSA’s Groh says organizations need to understand that diversity means more than multiple service providers, it means having network support that ranges from landline to wireless.
“So if switching stations are taken out, or central office switches are taken out, business can be rerouted instantaneously and not just through hardwire assets but also through satellite, microwave and other ways. The thing is to ensure that whatever bad people do you have the ability to overcome it,” Groh says.
