Like many municipalities, Ontario’s Grey County had a disaster recovery plan. Or thought it did.
Actually, admits Jody MacEachern, the county’s manager of information technology, it was “a really old document that was developed, stapled sat on a shelf. Everything in that document was out of date.”
And, he added, if IT couldn’t recover data needed from a backup it wasn’t going to be restored.
Data theft, power outage closing servers, ransomware — all these and more can be information technology disasters for any organization. So having an IT disaster recovery plan is essential.
But having a disaster recovery plan isn’t a business continuity plan, according to an expert who does both for municipalities.
In fact, said Gary Walker, of Toronto-based Perry Group Consulting, which helps Ontario municipalities with their IT strategies, too many counties, towns and cities think backing up data to a secondary site like a fire hall or community centre is a DR plan.
It’s not.
Disaster recovery is merely part of a business continuity plan, he said in an interview. A BCP encompasses a wide range of strategies for problems, from a natural disaster closing a building to a flu outbreak that reduces the number of staff in the office. In short, as its name suggests, business continuity is about planning how the organization continues to deliver service in the face of an issue.
So in creating a BCP, Walker said, organizations have to know what departments are responsible for which services, the technology/applications that support those services.
Then departments can set recovery time objectives (RTO — how long can we go without each service) and recovery point objectives (RPO — how long can we do without data for each service, which then determines backup windows).
Walker and MacEachern spoke about Grey Country’s voyage to new a new disaster recovery and business continuity strategy at last month’s annual IT security conference of the Ontario Municipal Information Systems Association (MISA).
Related content: Six steps to keep disaster recovery real
Two hours’ drive north of Toronto, the county covers some 100,000 people, most of whom live in nine municipalities. Although each of them has its own IT departments, the county has a number of important responsibilities under its $155 million annual budget including looking after 877 km of roads, a paramedics service, economic development and tourism and three long-term care facilities.
About two years ago, MacEachern said, as the IT department was changing to a “hyper-converged” infrastructure, it began thinking of updating its disaster recovery strategy. Choices included managing DR on their own by putting a replication site in a county building. One problem, though, was getting reliable internet to a secondary site in this largely rural area.
More importantly, IT realized the county didn’t have a good handle on the basics: How many applications did each business unit have, which were critical, what are the RTO (recovery time objectives) and RPOs (recovery point objectives) and more.
So the county put out a request for proposals for help with a DR plan, and eventually hired Perry Group. Through the consulting firm the county realized what it really needed was a business continuity plan.
“IT does not do itself any favours by inferring to the organization they have disaster recovery in place because they’re replicating from the fire hall or community centre,” said Walker. “All it means is you replicate some workloads. That might be part of a DR strategy, but on its own it’s not disaster recovery.”
Meanwhile, he said, IT is guessing at what it thinks should be replicated. Sometimes it doesn’t even ask what the business units think is important.
As part of a business continuity plan, in addition to determining what applications are critical an organization needs a data management strategy, Walker said. That’s because often 60 per cent of the data it stores is stale (meaning it hasn’t been accessed in over a year). It costs a lot to backup/replicate/recover stale or unstructured data.
What’s important, he said, is that BCP is driven by business (or municipal) leaders, who make risk assessment decisions. “IT is only there to facilitate the technology to satisfy the needs of the business,” said Walker. Often organizations toss decisions in the lap of IT.
“IT needs to push back and say, ‘Business, figure it out and then come to us and tell us what your critical services are. Then we will put in the technology to satisfy that.'”
MacEachern said Grey County’s experience showed how hard it can be for departments to establish priorities. Sometimes people can genuinely differ on what’s important.
“For example when we asked how important the document management system is, they’d say ‘I don’t need it, I get all my stuff through this [county] web site.'” But, of course, the website is fed by the document management system.
To drive the conversation away from IT, ask each business unit what services it delivers, and how quickly each would need to be restored if service was cut.
That, McEachern said, made Grey County staff think about priorities in different terms — this service needs to be restored fast for (fill in the blank — health and safety reasons, financial or reputational risk, etc.)
“It was transformative to our success because right away people understand you’re asking about the business, not technology, and they can usually tie it to specific reasons why they need services restored, he said. Walker adds that for disaster recovery purposes some services may be able to be delivered at less than 100 per cent for a short period.
Walker recalled dealing with a company that said 330 of its applications were critical. After a discussion that got whittled down to 10.
“If everything’s critical, nothing is,” he observed.
Even finding and agreeing on a name for all of an organization’s applications can be daunting, he added. For example, some staff may know an application by its brand name (Great Plains) while others know it as “the HR app.”
In the end, looking at options with Perry Group, Grey County decided to buy “disaster recovery as a service” from a provider rather than to DR itself. Meanwhile it is also working on a full business continuity plan.
Finally, Walker emphasizes that a BCP needs to be regularly reviewed — departments often buy new technology — and taken seriously.