Ontario’s Smart Systems for Health Agency (SSHA) may be a fledgling IT infrastructure provider, but this agency of the Ontario Ministry of Health and Long Term Care is ready for almost any disaster.
Created to enable the secure electronic exchange of health information among healthcare providers and patients, SSHA accomplishes its mandate through two highly available data centres in the Greater Toronto Area.The SSHA data centres were built with high availability as a guiding principle of design. The goal, as an IT operation, is to never lose data, and to never go down.Text
Designed and built in partnership with several private sector firms, these centres represent a new commitment by the province’s healthcare sector to use IT to its fullest advantage.
Smart Systems is building for a future where public health information, laboratory test results, prescriptions, diagnostic images and patient files will be instantly available to healthcare providers province-wide.
The SSHA data centres, which went live in November 2003, were built with high availability as a guiding principle of design. The goal, as an IT operation, is to never lose data, and to never go down.
Linda Weaver, chief technology officer at SSHA puts it succinctly, “you have zero time to recover, and you can’t lose anything.”
Zero tolerance for downtime is understandable when you consider that these systems could potentially be used to support life and death decisions in a health crisis such as SARS, or a natural disaster. They must be constantly available to respond to healthcare and public health emergencies.
SSHA used a variety of methods to ensure this level of availability. For instance, the two data centres are totally synchronized and mirrored.
Systems are duplicated within each data centre, and there is redundancy between the two centres, so they actually have four copies of all critical data.
A multiple redundant network was built between the data centres to guarantee instant real-time mirroring. But Weaver admits that the weak link in the system is the wide-area network connecting SSHA’s data centres to the province’s far-flung hospitals and doctors’ offices.
“There’s a lot of technical capability in most of the urban centres,” she explains, “but when you get into the more remote sites, that same type of technical support and capability to instantly respond to things doesn’t exist.”
A new, fully redundant network would solve this problem, but may be some time coming as SSHA, working with other provincial and federal healthcare agencies, struggles to integrate existing legacy systems across the province. Planning for an enhanced network is under way, however, fuelled by consumer demand for a level of service they have come to expect from the private sector.
In contrast to the network, SSHA’s data centres are discreet, controllable environments offering healthcare clients a highly available and secure location for their critical applications and data.
They host a variety of applications for clients, ranging from public health information systems to the province’s children’s health information system. In addition, SSHA runs a highly secure e-mail system that allows healthcare professionals to rapidly exchange patient information.
Not all of the province’s health information will be hosted at SSHA however, nor is this envisioned. Such a centralized system would not only be impractical, it would also create a single point of failure.
Lessons like this were learned from the private sector, but there are peculiar nuances to running a health information data centre, adds Linda Weaver. “If you can’t access your bank account, you wait ten minutes and try it again. But if you’re trying to manage a heart attack, sometimes you can’t wait ten minutes and try it again.”
SSHA prepares for potential threats by working through a variety of scenarios and developing systems and procedures that guide their response to each one. In a real disaster, practiced responses are the ones people tend to fall back on. Guessing under pressure does not produce the reliable responses demanded by a systematic disaster recovery plan.
Despite its readiness for natural disasters and other attacks, SSHA’s biggest threats have been virtual — viruses and worms, denial of service attacks and attempted intrusions. In response, the Agency has developed rapid-response systems and procedures to isolate infected systems.
Not all threats have been virtual however. Although their production data centres were not fully operational at the time, SSHA learned a lot from the blackout of 2003, including the procedures required for a safe and orderly shutdown. This is less of a requirement in the new, fully provisioned data centres, but they now know what’s required to keep their backup generators running. SSHA prepares for potential threats by working through a variety of scenarios and developing systems and procedures that guide their response to each one. In a real disaster, practiced responses are the ones people tend to fall back on.Text
SSHA did take a direct hit during the last SARS outbreak, when a number of staff members were quarantined. This taught them the need for staff redundancy, and for designated backups for executives and managers. “If something were to happen to me,” asserts Linda Weaver, “people know who my first designate is, and they know who my second designate is.”
By adopting best practices from a variety of industries, SSHA is bringing what’s already known about IT reliability management into the healthcare sector. They may not be developing anything new, but they have vastly improved the level of IT reliability and responsiveness in a system that we may all need when the next disaster strikes.
