The terrorist attacks of 9/11, the outbreak of Severe Acute Respiratory Syndrome (SARS) and last year’s Ontario blackout have undoubtedly fuelled much of the recent surge of interest in DR – disaster recovery — planning. But one CIO from Western Canada learned to build a little redundancy into his network way back in the 1980s.
Pat Ryan didn’t have to wrestle with a travel ban by the World Health Organization, nor did he lose hydroelectric power. On July 31, 1987, Ryan, currently CIO of B.C.’s Interior Health Authority, was working for Strathcona County, just east of Edmonton, when an F4 tornado tore through Alberta’s capital, killing 27 people and wreaking $330 million worth of destruction.
Ryan has never forgotten that killer storm — in fact, his organization’s DR strategy was on the books before 9/11 — and 17 years later he still looks to the DR lessons he and his colleagues learned back then. Of particular interest are the procedures that he now implements so organizations can survive sudden, unexpected challenges and swiftly return to normal operations.
According to Ryan, an important step in enabling a DR plan and building redundancy into the network is consolidating data. This was a challenging task for B.C.’s Interior Health Authority (IHA), which was formed about 30 months ago. Comprised of 19 different health care entities, the IHA serves about 30 cities and towns from 183 locations in B.C.’s interior. These 183 locations were also served by a whopping 51 data centres — which, Ryan said, were servicing different aspects of health care independently and thus weren’t connected to each other.
None of these data centres had what Ryan considered a DR plan — an unfortunate situation, but one common in Canada’s health care sector, he added.
“I knew we were vulnerable, but I also knew that all my peers were just as vulnerable. As a result, there isn’t quite that sense of urgency to get your redundancy up to where it should be if no one else is doing it. That said, it was certainly high on my list of things to do.” Each of these 51 data centres was a separate silo of information; when data is stored this way, each silo needs its own backup and recovery system. Consolidation, however, allows all the corporate data to be backed up and managed centrally.
You can’t have a DR plan without data consolidation, Ryan said.
“What we used to do — and we called this a disaster plan — was have each data centre do backup tapes and they’d move them offsite once a day,” he said. “The problem is, if the computer room is ruined by a flood or a fire or an earthquake then the tape is not good for anything.” The bottom line, he said, was that although they were storing information safely, they lacked a plan to restart operations and make use of the safely stored data.
Of course, centralizing data can be a scary thing because it is like staking all your bets on one horse — if the nag stumbles you’ve lost all your money. “Our vulnerability before was in 51 locations,” Ryan said, “but our degree of impact was small. Now the degree of impact is huge but the risk is only one data centre.” That’s exactly why it’s important for organizations to have a redundant data centre, said Bruce Yott, director of marketing at Getronics Canada Inc., which provides data backup services.
This data centre should be housed in a different building, ideally about 200 kilometres away from the primary site, in an area with minimal environmental hazards. For example, Winnipeg would not be an ideal place to house a data centre because it’s below the flood plain, said Jacques Flipot, district manager, Eastern Canada, at Veritas Software Corp. in Ottawa. Also, you don’t want your DR centre under the flight path of an airplane, he added. But selecting the location of a secondary data centre depends on what you want to guard against. Flipot said large consulting and auditing firms generally stick with 200 kilometres — a distance that would protect against geological issues like earthquakes and floods.
In the case of a widespread power grid failure, Yott recommends backup diesel generators, something that helped Getronics keep its networks running during last year’s Ontario blackout. In B.C., Ryan said, the IHA selected its Kelowna headquarters to house its primary data centre because it had a sophisticated storage infrastructure. The IHA designated Kamloops as the secondary data centre because it also had a highly developed storage environment. Kamloops is, coincidentally, about 200 kilometres from Kelowna. It’s also one of the least likely places in B.C. to have an earthquake or flood, Ryan said.
A challenge for any publicly funded organization embarking on a DR project is securing money – especially in the health care field, which, Ryan said, is not highly funded for IT. It’s hard to justify spending money on a new data centre when the money could go to an MRI machine. “It’s not easy for a CIO to raise money for a secondary data centre,” Ryan said. “But our board saw through that. They saw advantages to long-term preparedness. Our organization was created to run as a business and there’s such a strong business case for it.” So Ryan was able to convince them that in the long run, reducing risk through emergency preparedness was less expensive.
Getronics’ Yott echoed Ryan’s concerns. Public sector organizations are in a tight spot because they are responsible to taxpayers who demand reliable service from government-funded entities. But unfortunately the public rarely understands the significant that role IT networks play in delivering their services.
Previously, the IHA relied on server-attached storage, which was costly to maintain because each server had to be backed up. The IHA’s data centres now consist of storage area networks (SANs), which let the IHA purchase storage and servers separately as needed. The IHA stores the data on EMC Corp.’s Symmetrix boxes and uses EMC Corp.’s Connectrix, which sits on either end to keep the SANs in synch and manage the data. Additionally, the IHA uses EMC’s Centera, which electronically sends backup tapes off site at time intervals selected by the IHA. Centera also takes daily and weekly “snapshots” of data that goes offsite, stored on tape. Additionally, the IHA purchased a high-speed gigabit fibre optic line running between the Kelowna and Kamloops sites from Telus Corp. Even though owning fibre was more expensive than paying per-volume, it was worth it because the data lines weren’t taxed, Ryan said.
“If you’ve got slow speed fibre then you can’t [send data between the two datacentres] in real time,” he said. “You have to pump it down the line when the line isn’t in use or late at night, so you can’t replicate your data and keep it as current. With our systems we’ll send the updates in five minute blocks.” Hospitals frequently send high-resolution diagnostic images for tests like x-rays and CT scans, which can include up to 300 different files, Ryan said. If the IHA had to wait until off-peak hours to back up data, it could be lost between creation and backup. But having its own high-speed connection means the data can be backed up in real-time, he added.
Also, because hospitals are constantly renovating, the IHA ensured there were two points of entry at the Kelowna and Kamloops data centre for the fibre optic cable in case it gets cut accidentally during construction. “You just can’t be sure when someone is going to run a backhoe through it,” Ryan explained. So far, the IHA has managed to consolidate about 11 of its data centres but won’t be finished the project until about June 2005, so Ryan is not quite ready to throw the switch — but after the project is finished, that’s the next step.
“That’s a whole complex game in itself,” he said. “Once you’ve moved off site A, to site B, how do you move back to A again? That’s where EMC’s expertise is. Not only does [the company] help you jump over to the second site, but it helps you get back to your primary site once you’re up and running again.” But one thing is certain, a DR plan is like housework — there’s always more to do.
“What I’ve learned through the DR plan we’ve been rolling out is that there may come a time where you feel comfortable throwing the switch, but it’s not like you buy the thing, snap it together and now it’s there,” Ryan said. “It’s a lot of effort to make these systems mirror one another because each system is independent and unique.” Getronics’ Yott agreed. He said organizations should be constantly refining their DR plans and reviewing them at least annually. Officials at a health authority in Whitehorse, N.W.T. have been so impressed with the IHA’s work that they’re looking to the IHA to host their backup — a relatively inexpensive option, since storage is so cheap, Ryan said. The IHA is also discussing a similar partnership with two other health authorities, he added.
Not only are health care organizations becoming more savvy when it comes to DR but so are government departments. Industry insiders are now noticing fewer differences between the way public sector organizations and private firms approach DR. In fact, public sector organizations might even been more vigilant.
“There is a general public perception that in the last couple of years, public awareness has (met) higher standards than private industry,” Veritas’ Flipot said. “We’d be hard-pressed to say we can’t defend ourselves because the computer is down.”
Calvin Stone, director of Global Outsourcing for Avaya Canada Corp., agreed that the Canadian public sector is becoming just as diligent as the private sector. Flipot said the best DR strategies have a lot of automation built in so that they require minimal human intervention in case the situation is so severe that no employees are available. For example, during SARS many employees were instructed to work from home. But that poses a problem if IT personnel don’t have the technology to manage the network remotely.
Other considerations include the organization’s operating system and document requirements, application risk, followed by network, antivirus and firewall, said Al Rocek, director, worldwide professional and education services, enterprise administration for Symantec Corp. in Orem, Utah.
Planning is the most crucial aspect to any DR strategy because every conceivable disaster scenario needs to be accounted for, from the smallest software bug, to power outages, to floods, fires and severe weather. Any public sector agency that goes to an independent software vendor (ISV) or managed service provider with assistance to develop a DR approach will be asked the same types of questions they ask private companies — questions that help focus the DR plan to that organizations particular needs. A few key points to consider are:
– What are the organization’s core competencies?
– The most crucial systems?
– Does your organization have a front door and a back door for your telecommunications lines?
– And do you have a backup means of communication in case the phone lines are knocked down?
For example, Ryan said the IHA is designed to run as a business, meaning the good business case his team made for a DR plan was the ultimate reason it was approved.
“What board could, in their right mind, turn down something that saves money, ensures reliability and reduces risk?” Ryan said.
Rebecca Reid (rreid@itworldcanada.com) is a staff writer with IT World Canada in Toronto.
SIDEBAR: Plan B may need a Plan B
Organizations without the means to run their own redundant data centre might want to consider the path taken by Toronto’s University Health Network (UHN) —signing up with a managed services provider.
Unlike the IHA, it was the glut of unfortunate incidents that afflicted North America over the past year, especially the blackout and computer viruses, that kicked the UHN’s DR plans into action, said Jennifer Willis, manager of technical services.
The UHN is made up of the Toronto General Hospital, Toronto Western Hospital and Princess Margaret Hospital, but it had some challenges as a publicly funded organization. “From a public sector perspective…it’s being able to identify available dollars and costs and trying to find more creative solutions for DR processes,” Willis said. “(We also needed to find ways] to leverage some of our other activities and ensure everything we do has a DR component to it.”
The UHN has a sophisticated data centre compared with other health care centres, Willis said, including full redundancy for power and cooling. It also has its own diesel generator separate from other hospital systems and the supports the infrastructure in-house. The UHN wanted to ensure a robust environment that could withstand any environmental test, she added.
With the cooling infrastructure aging, the UHN needed to consider putting more money into the data centre. Additionally, it was starting to run out of space, limiting the expansion of core patient care systems, she said.
At the time, the UHN’s five-year contract with its managed services provider, Hewlett-Packard Canada Co., was up for renewal. The UHN signed a new $18 million agreement with HP and expanded the breadth of its services to include an off-site data centre, which was cheaper than going it alone, Willis said. The UHN’s new data centre will be housed in Mississauga, Ont., and be in production by September.
— Reid
