Like all CIOs, Darryl Lemecha worries about viruses and hackers, data centre problems and technology meltdowns. But what separates his worried mind from many others is a detailed incident response plan that will guide him, his IT staff and his organization through whatever problems may arise.
“The more you get that down on paper, the better you’re going to be in a real crisis,” says Lemecha, CIO and senior vice-president of shared services for ChoicePoint Inc., a data aggregator based in Atlanta.
An incident response plan takes its place beside business continuity and disaster recovery plans as a key corporate document that helps guarantee companies will survive whatever glitch, emergency or calamity comes their way.
“A lot of companies have that mentality, ‘We have some really good people in our organization, things are running well, the chances of something happening are small, and if something does happen, we’ll be able to deal with it.’ But in the event of a real crisis, people won’t know what to do,” says George McBride, director of IT risk consulting with Aon Consulting Worldwide in Chicago.
The typical response to trouble – the deer caught in the headlights – is exactly why companies need such a plan, McBride says. And while a business continuity plan aims to preserve operations in the face of adversity and a disaster recovery plan details what to do in case of a disaster, McBride says an incident response plan is broader, laying out how to respond to scenarios as diverse as data security breaches and network crashes.
Given their breadth and specificity, these documents are usually lengthy and in need of regular upkeep. They will vary from company to company and even among departments within the same corporation, but here are five points that all IT-specific plans should contain.
1. A sense of what can happen
You can’t possibly anticipate what will happen in a crisis or during the aftermath – that’s the nature of the beast. But that doesn’t mean you can’t plan for one, says Ian I. Mitroff, a senior investigator at the Center for Catastrophic Risk Management at University of California, Berkeley, and the author of Crisis Leadership: Planning for the Unthinkable.
Well-prepared companies pick potential incidents representative of the various crises that could occur and then devise strategies to handle them, Mitroff explains.
2. A well-chosen team
CIOs need to name names, says Janice Malaszenko, an IT executive who has held the CIO position at several Fortune 1,000 companies. They need to identify which departments have roles to play when something happens.
Think broadly, she says, lining up people from the human resources, public relations, legal and purchasing departments, to pitch in during an incident. Go outside the company, too, and identify the key suppliers and service groups most likely to play a part during a crisis. “Identify secondary or backup people, too, in case [the first-tier] people are unavailable,” she adds.
3. A communication plan
Bridge lines, conference call numbers and Intranet sites will be crucial for getting team members together when they’re trying to fix problems that might have them working in diverse geographical locations, Malaszenko says.
The plan should also include the individual contact information for team members that goes well beyond office e-mail addresses and phone extensions, she says. The document needs to contain home phone numbers and e-mails along with mobile phone numbers.
Finally, Malaszenko adds, the plan needs to say which team member owns communications, so when the time comes, there’s no delay in getting everyone talking.
4. A list of who does what (and when)
Good incident response plans don’t just name the members of the response team; rather, they lay out who will have which responsibilities and authority so they can get right to work, says Joe Brennan who, as Ohio University’s executive director of communication and marketing, played a key role in the aftermath of data security breaches that hit the college in 2006.
“In a crisis, a CIO can’t run around and say, ‘Hey, do I have permission to do this?’ A public relations person can’t run around and say, ‘Who’s going to approve my release?'” he explains. The plan must give them the power to make those decisions quickly.
But the plan should also give them guidelines to help them make the best decisions. “It should spell out the values and principles that will guide the response and the communications,” he says.
A hospital CIO might establish that patient safety is the top priority, so that the response team knows that its actions must first align with that goal. Or a university CIO might state that communicating promptly and honestly with students and faculty is a top concern, thereby establishing for team members that they need to put that above other priorities.
It’s important, too, to assign key roles to specific team members in advance, says Mike Tainter, the IT service management practice director at Forsythe Solutions Group Inc. in Skokie, Ill.
Determine who will handle communications with the public, internal business colleagues and external partners. Pick a particular person to track spending. And assign someone to document the team’s response to an incident. These notes will be valuable when it comes time to update the incident response plan.
“Nothing works better than to have a go-to team that’s trained and ready to resolve the problem,” Tainter says.
5. A safe, accessible home
Good incident response plans will have detailed, often proprietary, corporate information along with personal contact information for team members. That kind of document should be kept under lock and key, or at least secured deep in the corporate computer system.
On the other hand, if your IT system goes down and the plan is inaccessible, then it doesn’t do any good. The best approach is to thoroughly think out how and where the information is stored to guarantee access during all sorts of scenarios.
Lemecha, for example, has copies of his company’s incident response plan in three spots. Everything is on ChoicePoint’s Intranet, a second copy is on an encrypted CD that’s given to all the team leaders, and a third copy is kept off-site at one of the company’s locations (the exact location is undisclosed).
Plan to revisit and revise
An incident response plan is never really done. Rather, it needs to be revisited and revised as an organization grows, new threats develop, and team members change, Malaszenko says.
Start by putting someone in charge of managing the document. According to Malaszenko, IT security executives are often in charge of incident-response plans in larger organizations.
Whatever the title, the plan’s manager should update the document not only with everyday items, such as the names of new team members as employees come and go, but also with revisions to policies and procedures as incidents happen. The manager should also train new team members as they come on board and organize regularly scheduled drills, tests and simulations.
Testing requirements
You don’t want to find holes and glitches in your incident response plan when you’re dealing with a denial-of-service attack or a downed server. That’s why it’s so important to test it ahead of time.
Start with a desktop-type test, just walking through and acting out the plan; that will help identify any glaring problems with the document before going through the time and expense of a simulation, Malaszenko says. Then move to the next level by simulating an actual event.
Brennan worked at one university that tested its plan by simulating a hostage situation in which a gunman barricaded himself in a fraternity house. Among other things, simulations like that can test how fast the IT response team can set up a bank of toll-free telephone numbers and put together a new Web site for communications. Brennan says that test took half a day, with debriefing taking the remainder of the day.
Related content:
Security on her mind: Interview with Julie Spallin, manager CCIRC Centre
