The fire at a Bell Canada switching centre in Toronto on July 16 wreaked havoc on some businesses, but many firms with solid disaster recovery plans in place made it through the crisis without serious disruptions.
Internet service and business solution provider Passport Online managed to keep many of its clients up and running despite the Bell Canada outage, which affected voice and data connections in Toronto’s downtown core.
“Business ISPs need redundancy and we have it,” said Earl Epstein, Passport’s CEO. Epstein explained that Passport has three connections from its premises to the Internet. One is a line from Sprint Canada which uses Bell Canada’s infrastructure; one is a link to MetroNet Communications which employs MetroNet’s own infrastructure; and the other is a line to UUNET Canada which uses leased dark fibre.
There was no interruption to servers hosted at Passport’s Toronto site and no connection problems for those Passport clients who had last-mile connections that didn’t rely on Bell infrastructure.
Epstein said Passport makes it a practice to give its clients the option of installing alternate last-mile connections. “But there are many places where Bell is the only option,” he added. Those Passport clients that had only a Bell last-mile connection (including LTI, ComputerWorld Canada‘s parent company) lost their Internet connections.
As for voice, Passport had cell phones available, so the company was still able to call out despite the fact its voice landline was down.
“As soon as we went down, we called our customers and told them what was going on,” Epstein said.
Royal Bank spokesperson Teresa Pagnutti told a similar story. “We have a policy of multiple service providers, so that really helped our situation,” she said.
Royal lost approximately 250 bank machines and fewer than 30 bank branches were affected at the peak of the outage. “Most of our other services were intact,” Pagnutti said.
The bank is content with the way it dealt with the outage and won’t be making major changes to its disaster planning, Pagnutti noted.
The Bell network outage occurred when a worker dropped a tool into electrical equipment during some regular maintenance work at Bell’s central office exchange on Simcoe Ave. in Toronto.
“We had a contractor who was doing an upgrade for a back-up power source and that work led to an electrical explosion which led to a short circuit and a fire,” said Bell Canada spokesperson Samantha Ouimet.
The short circuit damaged the power supply used to keep switches at the central office running. Back-up plans called for Bell to use a diesel generator to maintain power.
However, the fire had caused sprinklers to activate and because of the resulting water on the floor of the central office, emergency officials would not allow Bell employees to activate the diesel generator. As a result, the switches continued to run for only a few more hours as the power in their batteries drained away. Once the batteries died, those portions of the network that relied on the building’s switches went down.
The switches stayed down for several hours until Bell workers hooked up an external power generator.
Ouimet said Bell is reviewing the circumstances surrounding the network outage. “We’re doing a full review to get to the bottom of events and once we get there we’ll review the procedures as needed.”
Bob Hafner, an analyst with consultancy firm Gartner Group Canada Inc. in Mississauga, Ont., said it doesn’t sound as if Bell did anything wrong, given the circumstances.
“It was just a case of all the wrong things that could have happened did happen,” he said.
There are precautions enterprises can take to ensure they are not caught off-guard, Hafner said. For data connections, businesses should ensure they are not relying on one provider, Hafner explained. And when looking for a back-up, firms should ensure the provider runs over a different infrastructure than the primary provider. This is an important point, Hafner said, because some alternate providers rely on Bell’s infrastructure, which means a Bell outage could affect the alternate as well.
Having a back-up local service provider is an option for voice redundancy, but Hafner noted it’s an expensive option. A better alternative is cell phones, he said.