“If there’s no business, there’s no need for IT,” stresses Elizabeth Beaver, senior manager, Business Recovery for the CIBC Mellon Global Securities Services Company.
Beaver’s office near the main door in the elegantly restored 1929 banking hall on Toronto’s Bay St. has a brass plate identifying it as the Crisis Command Centre. A classic wood table with a half dozen chairs, dwarfed by the distant height of the ceilings, is where executives meet to discuss the business continuance for CIBC Mellon’s two operating entities: CIBC Mellon Global Securities Services Company, a global custody provider and CIBC Mellon Trust Company, a supplier of transfer agency and corporate trust services. The company’s presence amid the heritage building’s pillars and arches that silently assure a solid foundation underscores CIBC Mellon’s new tag line The Freedom to Focus on Your World.
“Clients and customers aren’t sitting back anymore being quiet,” says Beaver, speaking also as president of the Toronto chapter of Canada’s Disaster Recovery Information Exchange (DRIE). “They’re being very vocal on how much they are wanting to be protected. If they’re coming to you for a particular service, they want to make sure that you’re here today and tomorrow no matter what else is going on. They want to make sure their interests are protected. We’re seeing new clients say ‘yes’ or ‘no’ depending on the recoverability of organizations and how they can protect themselves.”
Seeming to be as certain as death and taxes is a competitive necessity for financial institutions. In a world still mindful of the Sept.11, 2001 terrorist attacks and concerned with current political global tensions, a heightened tension greets news of a stolen hard drive, pilfered credit card numbers and Internet attacks such as the Slammer worm. At stake is customers’ confidence in their financial institutions’ ongoing protection of their personal information.
The ‘always on’ nature of the Internet and the increasing speed of the financial world – even before achieving straight-through processing – leaves no tolerance for data loss or down time. Not when there are mega dollars in transit between financial institutions, notes Anna Frazatto, VP of Professional Services, Agility Recovery Solutions (formerly GE Capital), Mississauga, Ont. (www.agilityrecovery.com)
“If any of those services are not available for even a short period of time, if you cannot meet customer satisfaction, you can lose the faith of your customer base and that spells death to a business.”
Financial institutions are now setting data loss and outage time goals, reports Ralph Dunham, manager, Business Continuity & Disaster Recovery Services at IBM Canada. “One bank in Canada has internally published that it will have no more than six hours outage and zero data loss,” he confides. He refused to name the bank, but noted that reaching those goals will require running two physical locations and mirroring in real-time.
He cites the risk management and governance issues involving Enron and WorldCom as also pressuring boards of directors and regulatory bodies to reassess a company’s ability to survive. Compounding all this is the U.S. white paper published by the Federal Reserve Board (see related coverage in Interview section and Keeping Current – Firms add business to continuity plans) which highlighted how financial institutions could have a higher resiliency and caused much discussion in the industry.
Dunham adds that “the resiliency concept goes beyond the disaster recovery, which was all IT-based, and business continuity, which included people and their access to IT, to design and build an environment to take a blow but not bring the whole system down. The system would just shift and adapt as events occur.”
More than IT
The broadening shift from a singular business continuance focus on just IT to including business units has been a lesson there for those willing to learn it. Even before joining CIBC Mellon in 2000, Beaver took note with the ice storms in eastern Ontario and Quebec in January 1998.
“The IT plans were there and the IT professionals got things up and running. They knew how to do their stuff. They had done the risk analysis. They knew that one of their single points of failure would be hydro. They had brought in diesel generators. So during the ice storm, yes they had the diesel generator, but they needed the diesel to run a generator and those trucks could not get through.” The lesson? “We can’t work in isolation.”
She also points out how the Sept. 11 attacks illustrated the importance of planning beyond recovering just IT. “Information services has always been the leader in business recovery. We saw the IT departments quickly recover after Sept. 11. They had well-documented plans. They’d been well tested. They got their data moved and up and running. But the human side of dealing with such tragedy was much slower.”
A disaster in only the data centre is now a very small part of her focus. “If you go into most organizations, you’re going to find that the IT budget is a much larger proportion than the rest of the business budget,” she admits. “That is just the nature of technology. It is just expensive. In the long run, when you take a look and do a proper business impact analysis, in the business units you’re going to lose more if they are not up and running.
“Even the vendors have learned that we just can’t focus on IT, so they are also looking at moving their plans to be more business focused,” she continues. “There are vendors out there that still really focus on the IT world which we need. IT is a very large portion of anybody’s business, but if you take a look at SunGard or IBM, they’re just not focused on recovering the data centres any more.
“To have a really good plan, it has to be comprehensive,” she adds. “It has to take in your IT. It has to have critical business units. It has to take into account who your vendors are; your suppliers. Now most importantly, it needs to take into consideration your employees – their skill sets and how you recover those. There’s no sense covering the IT plan if you don’t have someone there to use it. We have to work in conjunction with public authorities, the government, our landlords, any outsourcers that we may work with. We’ve also seen lately a lot of viruses that have been shutting down ATM machines and our access to Web sites. That means our recovery plan must work more closely with security than we’ve done before.”
“Security is a key part of business continuity as is the ability to isolate and insulate an incident,” Dunham adds. “You supplement your production environment so that your performance doesn’t degrade when some pieces are taken out.”
He notes that more regulatory involvement affecting the integration of processes can involve seven or eight organizations. When looking at reaching to all these, it becomes an issue beyond in-house. Service level agreements must be very strong. “All you need is one component that doesn’t take it seriously and the entire process is at risk.”
As information has come down to the desktop level, the focus of disaster recovery has shifted from recovering data and technology to recovering people and functionality, concurs Agility’s Frazatto. “It is important not only to have a replacement server, but to have a critical person, at a desk, usually speaking to the outside world,” she stresses. “Businesses are more dependent on 24 x 7 sales, customer service, etc…and therefore must concern themselves with end user recovery. When you are dealing with people, and not just machines, traditional recovery at a remote hotsite becomes a logistical problem – how do you transport people? Can you get them to leave their homes and families? Can we afford to house/feed all these people in a remote location? Recovery options are increasingly tending towards local and onsite options. Recent studies have indicated that people are not willing to travel more than 20 minutes more than their normal commute to affect a recovery.”
“We’re finding people are working together more in a community situation,” adds Beaver. “Businesses aren’t working in isolation anymore. They are taking a look at ‘what if this business disappeared? What impact is that going to have on me? What impact is that going to have on our economy?'”
IBM’s Dunham claims many companies are turning to third parties to design and construct environments which are always available. He says IBM’s workload to confirm that clients’ business continuity plans actually work has increased by more than 60 per cent over this past year. IBM has increased its number of employees who are skilled in testing and recovery, and expanded localized capabilities. At one time it could accommodate 100 of a customer’s personnel moving to its facility. Now they are expanding that number toward 700, he says, with its recovery centre in Markham, Ont., and local access centres in Montreal, Winnipeg and Calgary.
Beaver also reports more members are joining DRIE (www.drie.org) where disaster recovery tactics and experiences are shared and kept in confidence. The Toronto chapter now numbers 340 members. A new chapter formed this year in the Atlantic brings to seven DRIE Canada’s chapters coast to coast. DRIE Canada provides a number of courses and certifications. It also supports the Business Continuity Institute in the UK (www.thebci.org) which has a 10-step process for different membership levels of certification.
DRIE has vendors sponsor a quarterly session or become a yearly sponsor with a particular chapter, thus bringing their services to the community that needs them. Vendors include SunGard, IBM, Infostream Technologies Inc. and Agility, plus auditing companies. Beaver also keeps informed via the Canadian Emergency Preparedness Group (www.ccep.ca), Disaster Recovery Journal (DRJ), GlobalContinuity.com and vendors such as SunGard.
A common message among all these groups is to be prepared and have plans in place as to how you will respond to a fire, major downtown evacuation and even a major loss of life. “They can be generic enough that you can mould them into whatever event you’re faced with,” Beaver advises. “That’s what a business recovery person brings to a company and it’s what the DRIE organization assists those professionals in doing.”
Putting it into practice
It is Beaver’s role at CIBC Mellon to help determine that the teams and comprehensive plans are in place for the company right across Canada.
“This process is never complete, but I make sure there isn’t a group working in isolation and that that we are pooling the expertise at the time of the event. On Sept. 11, it showed how well it worked at CIBC Mellon. The crisis communication went out promptly to our clients, customers and employees. We had the crisis counsellors in here that day,” she stresses. “They were here for a week providing counselling in Toronto and to all our branch offices across Canada. It was very proactive.”
The business continuance plans at CIBC Mellon are checked annually, she reports. It is important for companies to do so, whether it entails sitting around the table and going through documented procedures or actually going out to a recovery site and recovering data or selecting several critical business units and performing what they would have performed on a particular day in their business world. She finds it also helps the business units remember that they need to continue this process.
Twice a year CIBC Mellon’s critical business units make sure its call trees are accurate so that employees can be reached in an emergency.
IBM’s Dunham sees a need to build more automated processes, such as mass call outs to employees, and have these in place instead of relying on human thought. He points out that there are tools to automate restoring business, to watch for outages in network and to identify hacks, isolate their damage and switch to back up. “This movement is what IBM refers to as their autonomic computing initiative, building knowledge into the environment so it is performed automatically,” he adds.
“The more you remove the human element, the better your plans will be,” concurs Andrew Steen, vice-president, Technology Speciality Insurance, Chubb Insurance Company of Canada.
Declaring himself “a big advocate of automated back-up,” Steen warns “relying on one individual is a critical weakness.”
He still finds companies’ managements too often think that manually backing up data is adequate. He cites examples where the data integrity was so compromised that only a fraction of data could be retrieved. Or, management may delegate the task but it never gets done.
He says among the best practices advice Chubb gives clients are recommendations for automated solutions from business continuity companies such as the newly created Traxion Technologies Inc. of Mississauga, Ont. (www.traxion.ca) Steen says there are many automated options, from many times a day to once a day to mirroring in real time offsite.
“As we continue to become a more data focused society, the need for data protection is magnified. If a company is based on its intellectual property and can’t access its data again, it’s probably lights out,” he concludes.
From a backup perspective there are plenty of tools available that allow for minimal downtime affecting production systems, adds Agility’s Frazatto. “The ability to snapshot databases has been around for years but the ability to have those snapshots offsite on a timely basis is more in the forefront now. Local recovery from a mobile recovery is new to the market. End users do not have to relocate to a distant recovery site.”
A trend to real time processing and a faster financial world has added pressure to create real time solutions, but solutions for recovery in minutes are expensive and should be minimized where the need really exists, counsels Agility’s Frazatto. “Immediate needs may be things such as stock trades, either individual or corporate. When a mutual fund manager’s ability to make a trade for an organization is compromised, he/she may lose that company thousands, even millions of dollars if the trade is delayed. To the contrary, a loan approval might be something that can wait for 24 or 48 hours to be processed.”
She also warns that although there are many software tools to assist in planning and establishing a business continuity and disaster recovery plan, fancy tools should not distract from the discipline of planning, managing and exercising your recovery capability. “The old standby of Keep it Simple applies. Many of the excellent programs that we see are based on word processing documentation. It can be accessed by all those designated with responsibility to maintain the recovery plans. No specialized knowledge is required to update information. It is cheap. Too many of the business continuity coordinators become software specialists and lose focus on the real target.”
Beaver’s focus is clear: being ready for even the large ‘what ifs’, such as having “to stand in front of the media and say what has happened and how we’re going to get through this and give a comfort level to our clients.”
To that end, her main challenge is “making sure that as our business grows and changes and our clients’ needs change that we keep our business continuity and technology plans in sync to meet requirements.”
She admits it’s a moving target, but she says her job is facilitated by another essential element in successfully safeguarding the company: an executive truly committed to business continuance.