Arguably, the most dreaded task an information security pro has to face is to rip and replace IT infrastructure. But the Canadian-based chief information security officer (CISO) of an international firm says many leaders have to face an even bigger job: Ripping up and replacing their business continuity plan for surviving a major regional — or bigger — IT outage.
“We all — whether we want to admit it or not — have business continuity plans that are wildly out of date, wildly incomplete,” James Arlen, CISO and chief information officer (CIO) at Helsinki-based Aiven, a database-as-a-service provider, told the SecTor conference Thursday.
“The business impact assessments were done by people that don’t understand the businesses because you couldn’t get one of the business people interested in having a conversation with you about what happens when their tools die. They don’t care. They’re like, ‘Just make it work.’ The business side says to IT, ‘Computers are magical. Just click some things! That’s what you do over there.'”
The fact is, Arlen said, applications these days are dependent on other applications — particularly cloud apps.
What infosec leaders need to do is carefully map those dependencies in a new continuity plan. Otherwise, he warned, they won’t really know what to do when there is a major collapse of a major cloud provider.
It has happened, Arlen pointed out: In December, 2020 Google applications that required Google OAuth authentication services — including Gmail and Workspace apps — were unavailable for 47 minutes.
When a power grid goes down, electric utilities have to know how to bring the infrastructure back online. Similarly, Arlen said, IT and infosec administrators have to know how to bring their infrastructure back from a major collapse. But, he added, if they don’t have a full inventory of their hardware and software — including dependencies — any plan is crippled.
What has to be created is similar to what the utility industry calls a Black Start plan — starting when the power grid is black — Arlen said. He calls it a Cyber Black Start.
Don’t think about modifying your existing business continuity plan, he stressed. Start from scratch. The existing plan can be used for reference material. “But you do have to start over,” he maintained. “You have to think deeply about it as you go. Putting together a Cyber Black Start will not take a couple of days or a couple of weeks or even months. It’s a year’s worth of work.”
A dependencies graph or map — especially in a hybrid infrastructure — will be “almost terrifyingly giant,” he warned. That’s because a major cloud-based app your firm relies on may itself rely on an platform-as-a-service provider, for example.
How many Canadian organizations have outdated plans? Most medium and small firms, Arlen said in a post-speech interview.
“Most information security professionals don’t consider the inter-relatedness” of applications, he said. “There’s been a creeping level of complexity that’s occurred over the last 10 years. It’s accelerated a lot in the last two or three, especially because of the pandemic where they’ve been adding new systems without considering the implications of them and how staff becomes dependent on them.” For example, videoconferencing used to be nice to have. Now, in many organizations it’s a must. But few organizations have updated their continuity plans to take that into account, he said.
The result is, in a big internet crisis, most organizations will become “materially dysfunctional for a period of time.”
Many employees now work from home, he noted. Do they know what to do if they can’t log in as usual one morning? Do they know the phone number for IT support? Does the organization have an alternate communications messaging system, like SMS text?
“We pat ourselves on the back and say, ‘We did a business impact assessment and we can be fine for 24 hours,'” Arlen said in the interview. But one staff member may think their inability to log in means they’ve been fired.
What to do?
First, Arlen said, infosec leaders need to compile a full list of IT assets — which, he said, they may think they already have, but odds are it isn’t complete. Arlen’s team recently figured that the company, directly or indirectly, has 197 tools and services, including infrastructure- and platform-as-a-service providers — and each has some data attached to it.
European-based firms have an advantage, he added: They have to meet certain provisions of the General Data Protection Regulation, so have to maintain data flow diagrams about how personally identifiable information moves internally. That helps in understanding where and how applications and tools are interlinked.
Don’t follow GDPR? Then start by making a list of known applications, then go to each business unit and ask if there is anything to add — or delete. When you’re sure you have every app and tool, start creating the dependency graph.
Arlen cautions that some dependencies may be discovered only by searching a product’s marketing material. Every tool has dependencies, and there may be latent dependencies that can only be found in marketing material or a SOC 2 report.
Playbooks are still needed, Arlen added. But they have to be regularly updated. And you may find there are duplicates of the same playbook written by different people.
