Sunday evening. The phone rings.
Caller: “There’s a problem, Steve. An alarm went off for the Data Centre. A breaker tripped. Power was cut.”
Me: “OK, tell me more.”
Caller: “The Data Centre is completely dark.”
Me: “Dark, as in the lights are off, right? The UPS then diesel generator kicked on, right?”
Caller: “No. The breaker shut out the UPS and Generator. EVERYTHING IS OFF. There was a hard shut-down of ALL of all servers, core network switches and SAN. What do we do now?”. Then he says quietly, “I’ll bet some heads are going to roll.”
IT Services were then completely offline for ten thousand clients and a couple thousand staff. Network, phone system, email, ERP’s. Everything. They would be livid. I didn’t know how long it would take to get the Data Centre live again. And I didn’t know what physical damage was done by the hard shut-down. And worst of all, I didn’t know if we lost any data.
This is an example of a moment of truth for a CIO. A moment that will reveal your character.
As CIO’s we’ve all faced crisis situations, big and small. And given the nature of IT systems, things go wrong. Systems fail. Vendors under-perform. Projects fail to deliver. And, sometimes, team members don’t deliver on their promises. Clients will be frustrated and angry.
Here are some things to consider the next time crisis strikes. I’m not focusing here on what to do to address the crisis, but on how you address the crisis.
1. Microscope. Remember that your team is watching you closely especially during a crisis. You are setting an example for how to behave when things go wrong. Do you stay cool, or do you blow up? Think about how you would want your team members to react the next time they are faced with a crisis. Act that way.
2. Respect. Treat everyone involved with respect. Remember that nobody intentionally created this crisis, but the people around you will be working toward a fix. Remember to thank your team early and often. Recognize if they had to come in after hours. Even if “it is their job” and they “are being paid for it”. Recognize that the stress they feel is often greater than the stress you feel.
3. Blame. Don’t be quick to find the culprit. If it is very clear that, for example, a staff member’s mistake caused the crisis, now is not the time to call attention to the person. There will be plenty of time to address mistakes afterwards. Focus on resolution. Consider asking the person that made the mistake to take an important role in solving the crisis. Remember, you want them to avoid making mistakes in the future so this doesn’t happen again. This is a good way to drive accountability. If others on the team recognize that person’s role in causing the crisis, they will see what you are doing and appreciate it. And, remember, that if you begin to lay blame, your team will do the same the next time they are faced with a crisis.
4. Just the facts. Remember that during a crisis, people will react and start to build theories about the issue. Be open and honest with the resolution team, and expect likewise. Make sure that your team is sharing blame-less facts. If someone forgot to check the backups, that is simply a fact at this point. Now is not the time to deal with missed procedures or errors. You want everyone to understand precisely what happened, or they may not be heading in the right direction to get you out of the crisis. You will be demonstrating that you value openness and honesty. Now, you need to be open and honest with your stakeholders. If you don’t know something, or it appears that something was done incorrectly, tell them. Remember that you want your team to do likewise with you.
5. Long Run. Keep in mind that this isn’t the first crisis and it won’t be the last. As you are leading the resolution, you are also teaching your team how to behave. You need to continue to be professional and focussed. If training opportunities arise, you may want to use them.
6. But, you are angry! Allow yourself to feel and express any anger or frustration. But choose the time to express it. Often you need to focus on the solution first, then address the anger or frustration afterwards. This is the hardest thing to do for many of us. But, as my Mom says, “nobody said this is supposed to be easy”.
The situation above actually happened to me. The power shut-down came as a result of new power distribution in the building that overloaded a circuit. It shouldn’t have happened but it did. Although there were some tense moments, in time all systems were restored, no data was lost and all hardware proved undamaged.
I learned something much more important than ensuring a full review before power distribution changes. I got a glimpse into my own character. I would like to say that I followed all my advice above, but I didn’t. Not entirely at least. But, I’m learning.
P.S. Thanks for reading my post. If you have any other suggestions for how to behave in a crisis, I’d be grateful if you included them in the comment section below.
