Backing up huge amounts of data from one storage area network (SAN) to another may be the norm in the Canadian financial industry, but when Roy Southby, director of technology services for B.C.’s Interior Health Authority, went hunting for SAN-based replication examples in the Canadian healthcare industry, he came to the conclusion he was about to become a pioneer.
“Health most probably has been the last bastion to move over to a disaster recovery situation,” Southby explained. “The financial guys have been doing it for years, but health generally doesn’t have anything in place aside from backup storage data.”
IHA needed a SAN to SAN system to give the agency, which coordinates healthcare for about 690,000 people in B.C.’s interior, a disaster-proof plan for the healthcare data stored in its two main data centres in Kelowna and Kamloops.
IHA manages over six terabytes of operational data and patient information. The agency’s two key applications are the Meditech Health Care Information System database, which covers every aspect of the health system ranging from payroll to patient lab results, and the Picture Archiving Communications System (PAX), which stores images like patient X-rays.
Ross Allen, managing director of EMC Canada, said IHA is at the leading edge in terms of data replication in the Canadian healthcare industry.
“I think the thing that’s most unique about this deal is the vertical sector in terms of health care and the use of the technology,” he explained. “The use of PAX, the archiving communications system, really delivered some immediate benefits in terms of service levels to patients in the IHA.” Instead of waiting for X-ray information to be sent to specialists, the specialists can now view that information online, he noted, saving time and money.
IHA has been working to consolidate its storage since 2001, when the government of B.C. pared down the number of regional health districts. IHA was responsible for centralizing the storage of 183 health sites and 40 data centres at its two SAN locations. Once the agency finished the storage consolidation, it began putting together its disaster recovery plan.
“We didn’t want to put everything in one basket in Kelowna,” Southby said. “We realized the modern trend, after experiences with terrorist acts, etc., is you have to look at disaster situations.”
IHA issued a RFP in July of 2003 and selected EMC, which isn’t surprising given that both of IHA’s SANs are based on EMC technology.
The Kamloops data centre made an ideal backup site, because it was geographically diverse from Kelowna, being around 200 kilometers distant, and it was located in an area with low seismic activity, meaning the risk of an earthquake was low. Also, because both Kamloops and Kelowna had EMC Symmetrix SAN platforms, IHA could use EMC’s Symmetrix Remote Data Facility replication software, which is designed for large, asynchronous transfers and EMC’s TimeFinder remote storage software in its disaster recovery plans.
“With TimeFinder at any time you can just take off another copy of a database without having to stop the existing database,” Southby explained. “It just takes a snapshot and sends over any data that’s changed.”
To handle the data replication, IHA had to upgrade the WAN connection between Kamloops and Kelowna from a 30Mbps link to a full duplex Gigabit Ethernet connection running over Dense Wavelength Division Multiplexing. “In our tests our ping times were two milliseconds,” Southby said. “Over 200 kilometers, we were really thrilled with that.”
IHA plans to do backups once an hour. That means data will be an hour out of date, but that’s still a lot better than IHA’s tape backup system, which only had one backup a day.
“We had a disaster the year before last and with the tape backup it took us about 40 hours to rebuild the data,” Southby said. Rebuilding a health system can be tricky, he noted, because, for example, a patient’s lab result could end up getting lost and the patient might have to be tested over again.
With SAN to SAN replication happening once an hour, any future rebuilding and lost data should be minimal, he noted.
If IHA’s main site in Kelowna goes down, the agency has arranged for its provider, Telus, to point the agency’s IP addresses to the secondary site in Kamloops.
“Users would be totally unaware of which site they were sourcing data from,” Southby said. “It’s a pretty smooth operation.”
Although IHA relies on about 20 servers to run the Meditech application in Kelowna, there are only about half that number of servers available for backup in Kamloops.
“Meditech is a modular system, so the more servers you have, the faster it runs,” Southby said. “In a disaster situation we didn’t feel like we needed to run at full speed, therefore we could put more modules on each server.”
IHA has completed initial testing of the disaster recovery system, and other than a problem with the Gigabit Ethernet connection, which was subsequently fixed, there have been no serious problems. The agency hopes to have the system in operation by July.
In the meantime, Southby noted, the Kamloops data centre is still handling some non-disaster recovery related activities and will continue to do so in the future.
“It’s not as if all the technology in Kamloops is sitting there waiting for a disaster,” he said.
