A fire at Bell Canada’s Simcoe Street central office in Toronto has created a series of problems for some securities firms trying to place orders and trade through the TSE’s trading systems.
MTT customers are advised that long-distance service to a large number of exchanges in downtown Toronto has been impaired due to a fire at a Bell Canada switching station located on Simcoe Street.
These two Canada NewsWire excerpts would seem to relate to the same event. Alas, they do not. Fire hit Bell Canada’s switching station twice in less than 10 months. If one of Canada’s most technologically advanced companies had the scourge of fire hit twice in less than a year, where does your company sit in the realm of disaster prevention and recovery?
The odds are pretty good your are not prepared, at least not as well as you may think. A recent IBM Corp. survey of 226 business-recovery corporate managers found that only eight per cent of Internet businesses are ready to deal with a computer system disaster.
But you do not work for an Internet business, you say. Well, in the opinion of other experts, disaster, in its myriad forms, is a distinct threat for the vast majority of corporate Canada. And with the 24 by 7 mentality now being inflicted on the business world, data emergencies can be the death knell for a company. In essence, if you go down you may not recover.
“The reality is that [companies] are not prepared and the ones that are prepared, or think that they are prepared, really aren’t,” said Sam Nicoletta, manager with the global professional services division of Compaq Canada Inc. “The fact is that it will get worse as we move forward, as people’s expectations get dialled up,” he added. “In the near future, if you have to recover your business is dead.”
Part of the problem is today’s companies are afforded little or no time for down time. It used to be if you had a power failure or a computer crash it could be hidden from the customer for a few hours or days. Business was more local, and if a fire ravaged your site and destroyed the mainframe there were feeling of sympathy from the community.
“You are in a brand new world now…there is something called trust, and trust, when it is wrecked, is really hard to regain,” said Mark Fabro, senior scientist and managing director of Guardent Canada Inc. in Toronto.
Once a problem is in the public eye it is difficult to repair the damage. He said the problem lies in the fact that the average news reader may not have the technical knowledge to realize that if a financial institution’s Web site was attacked, it doesn’t necessarily mean financial data or information was ever in jeopardy.
To add salt to these wounds, with the hyper-connected world customers can be more informed than you are.
“The world can actually know before you do, which is scary…I couldn’t imagine getting that call,” Nicoletta said.
prevention, prevention, prevention
If an ounce of prevention was worth a pound of cure in the days of yore, today that ounce is worth a tonne. The problem is to sell that ounce to corporate executives. It really shouldn’t be as difficult as it is, given computers are machines and machines are prone to failure, according to Tom Keenan, dean of the faculty of continuing education at the University of Calgary.
“You take a PC and you run it and I guarantee the hard disk will fail. It is not a question of if, it is just a question of when,” he said.
Paul Tew, technical manager at PowerQuest Corp. in Orem, Utah, agrees. “I think it is just a matter of teaching good computing habits and a good computing habit is the fact that this is a machine and machines are subject to failure, so matter how well built they are…they will wear out eventually.” He added that today’s hard drive technology is vastly superior to just a few years ago, which is a bit of a double-edged sword since people tend to believe in the units’ infallibility.
“A lot of [users] are having to learn the hard way, unfortunately.” Tew said, in his experience, 95 per cent of companies burn their fingers before they learn to fear the flame.
That said, the question is where to start. If it is not fire or flood, it could be human error or a virus that causes the system to crash. The guardians of today’s IT world are surrounded by potential causes of heartache and corporate mayhem. “From a disaster standpoint the playing field is so much larger right now because of information systems and open systems connectivity,” Fabro said.
Pop quiz: Where does your company store its nightly back tapes? (If, indeed, it even makes them.) If your answer, like many, is right beside or on top of the server, you are making error number one, and it is an error almost every IT person knows he or she should not be committing. But it is done nonetheless; call it stupidity or laziness or just avoiding one more daily time-consuming hassle.
While most companies can start by making sure nightly backups are stored off site or at least away from the server, there is a more urgent requirement.
get a plan, keep it current
Companies need to have a current corporate policy dealing with all aspects of disaster prevention and recovery. Some dusty manual propping up a wonky server table just won’t cut it anymore.
Fabro said companies need a policy in place and that it should start with a clear and concise overall understanding of the company’s system architecture, including where systems are physically located, what each one does and who monitors them. He goes so far as suggesting a separation of duties to ensure that no one person can completely debilitate the system.
“Without the policies and the agreement from management that those policies have to be enforced, it is very difficult to enforce any preventative measures,” said Deborah Harrop, a disaster recovery and business continuity planning specialist working for a large company in Edmonton.
Fabro sees the need for top-level management to become more educated on data security. “For the C-level executives, their understanding of information security must go beyond concepts of things like revenue, they have to understand the implications of information security breaches at all levels within the organization,” he explained.
Testing, testing
But having the world’s greatest plan is of no use if it won’t work properly under pressure. Remember those fire drills at school when you were a kid? Disaster run-throughs should be as common.
“These drills actually serve a purpose because with so much turnover in companies the odds are very high that there will be somebody there who has no idea what this is all about,” Keenan said. “Do [drills] every six months.”
But getting senior management to buy into a costly and expensive disaster-readiness implementation can be a tough sell. Fabro suggests a simple reminder. “The cost of reactive mitigation is always one, maybe two, orders of magnitude more than proactive.”
Much like insurance, a company should have a system in place it hopes to never have to use. But unlike insurance, part of the disaster system is designed to help prevent a catastrophe from ever occurring.
To see where your company stands, answer yes or no to the following 10 questions from the DTL Tape Web site.
1. Do you have a written disaster recovery plan?
2. Have you tested it?
3. Did you pass your test?
4. Have you qualified and ranked the financial risk of outages to all vital functions?
5. Are you prepared to address the liabilities and fiduciary responsibilities in case of disaster?
6. Are disaster recovery plans kept current and updated for business changes?
7. Do you perform backups faithfully and include every server and hard disk?
8. Do you regularly send your backups to a safe, off-site archive?
9. Have you standardized on a proven media, drive, software and automation backup solution?
10. Does disaster recovery readiness have support of top management in your organization?
If you responded in the negative for any question, it is time to address those issues.
creating a happy medium
It is possible to create an almost Fort Knox-like system that will stay up under every situation this side of a direct nuclear hit. But is it worth it, what are the non-monetary costs and how much will it effect business processes?
“[The security solution] has to be in balance with the criticality of what you are trying to protect. Your security measures should never outpace the business need. It should be an enabler, not a disabler,” Harrop explained. She added that there is always the debate of how much preventative measures stand in the way of business processes. Since data security typically creates some business-activity slowdowns, it is very important that corporate continuity policies have complete support from management.
Fabro agrees it is a difficult line to walk. “The problem is trying to find that sweet spot where is effective security that doesn’t slow anybody down,” he said. But he stressed that going from an insecure environment to a secure one will only slow most processes down a second or two. “But organizations are more concerned with the delivery of quick and effective financials…than they are actually about security,” he stated.
Keenan talked of days gone by when companies would put all mission-critical information on a parallel site at least 20km away so it was within a separate power grid and weather pattern. “[Today] nobody really thinks first about security or disaster prevention, they think about business functionality,” he said. His solution is for companies to look where it hurts most: the wallet.
“One way to do it, that certainly builds attention for e-commerce companies, is to say ‘Well, what would it cost us if we were down from 9 to 10 am tomorrow morning’? You do that and all of a sudden the priority of the budget allocation of this backup thing moves up a couple of notches,” he said with a hint of laugher.
Nicoletta agrees that it is never an easy sell. “To have a capital outlay as a result of something you hope is never going to happen is hard to stomach,” he explained.
“It is a little bit like the analogy of insurance. I pay all this money every year for my house and I don’t get the value because my house doesn’t burn down,” said Bob Mahood, executive vice-president of corporate development of Jawz Inc. in Toronto.
Though cost can be a stumbling block, he sees business process slowdown as a non-issue. “I think that if you do it right it is not that big of an impediment to actually doing business. It is like the idea of physically showing a security guard a card…at the end of the day it doesn’t really slow you down that much,” he said.
backing up mobile data
Backing up your personal data seems like the ultimate no-brainer. But do you do it? Always? Even on business trips?
With huge numbers of corporate travellers and telecommuters there are hundreds of millions of dollars worth of data sitting on laptops.
“I think there is huge exposure in data sitting on laptops,” Keenan said.
Dan McLean, network analyst with IDC Canada in Toronto, agrees. “The data stored on the localized drive is more important than ever, but it is highly vulnerable and [the risk for companies] to lose that data through some corruption of the local device…is just way too great.”
Yet very few people consistently back up their data. Test yourself. Remove the hard drive from your laptop. Now throw it in the garbage. Feel comfortable? Probably not.
Tew uses a lifestyle analogy to explain why we don’t back up as much as we should. “It is like eating right or getting enough exercise. We know what we are supposed to do, but hey, it is too much bother.”
“If you take anybody’s list of the top 100 things to do today, backing up their files, I guarantee you, will not be on it,” Keenan said.
McLean thinks companies should be less trusting when it comes to data. “I think what you want to try and achieve in back up is to remove that function from the end user themselves, being something they need to initiate…because I think history has shown that they just don’t do it,” he said.
The solution, according to many, is to move the entire corporate backup process to the Internet.
“That takes your worry, theoretically, away from having to manage the back-ups and all the other nightmares that come along with managing the infrastructure of a network,” Tew said. “Data is exploding exponentially with the Internet…there needs to be places that it is stored, and I think the SSPs (storage service providers) are going to do a lot to help that.”
away from the public’s eye
Regardless, disasters do and will occur, again and again, but whether the public ever hears about them is another matter.
“On a daily basis, we are called into events that will never reach the public eye,” Fabro said.
He concluded with a bit of unsettling information. “Without actually giving names or locations, I can still say, with all confidence, that based on my recent experience I do know that things like some of the largest financial exchanges in the world are literally, literally inches from complete disaster.”
“The cost required to maintain an effective information security program is trivial compared to trying to recoup the cost from damages incurred by a security breach.”