Information is power, someone once said. It could also destroy an IT security manager’s career if it isn’t understood or presented properly.
That’s the problem IT staff face when they want to include security metrics in a report to their team or senior management, digital forensics expert George McBride told the SC Congress security conference Wednesday in Toronto: Network security software and devices collect a tremendous amount of data, but only a slice of it may be needed.
McBride, vice-president of New York-based consulting firm Stroz Friedberg, gave a classic example: An organization created a bar chart comparing the number of IT security events its two gateways experienced over a 12 month period. The number of incidents ranged as high as 1.8 billion.
But, McBride said, that only shows the infrastructure’s sensors were working. A number in the billions is meaningless. More significant would have been to measure security incidents that required action or analysis. Just as important would have been correlate either to the organization’s risk profile. And why measure both gateways separately when the total both faced is likely more important?
In short, when deciding on a metric think, “So what?”
The biggest mistake security practitioners make is focusing on information they have no capability of changing, he said in an interview. “They present on external threats, they may present on other things that they have no capability to change and have no effect on the security or risk posture of the organization.
“We love numbers, and we love to be able to present all of the information we have. The challenge lies in being able to translate that information into data that is actionable, that is meaningful and can be used to effect change in the organization.”
Things like the risk posture of the servers in your DMZ or how many days they gone since being patched are much more important to the security team, and possibly senior management, he said, since it will indicate if resources need to be re-allocated.
Ideally, you want to show what IT staff is doing is the right thing, it can be measured and progress is being made.
Another problem is that the source data is either wrong or is being presented in an improper way, so the message becomes misunderstood.
Sharing raw data at the operational level may make sense, he added, but not when presented to senior managers or the board. On the other hand, McBride added, even within the security team it may be better to reduce data to answering significant questions: Are all systems operational, what was their availability? “If it doesn’t matter, you shouldn’t present on it. Just because you have data doesn’t make it meaningful or actionable.”
Metrics often need context, he said — for example, showing the percentage of employees who fell for a phishing exploit by itself isn’t important; comparing it to the percentage who fell for it after staff training is.
Given that security software has no problem churning out reports, but the number of reported network breaches continues to rise suggests a lot of reports are meaningless. McBride disagreed. Some organizations focus more on compliance to law and regulations than security, he said. “Compliance doesn’t equal security.”
If the right reason for presenting metrics is to effect change, he told his audience, the wrong reason is you have to fill a report with something.
Best practices include knowing the audience you’re presenting to — your team can understand more data; the board less so. And your time before the board may be short, so you have distill information tightly). Generally, senior leaders want to know the state of IT security — what’s the risk posture, are you on budget; are we meeting regulatory compliance; are projects on time? And sometimes you might be asked to produce a metric about something you don’t know about but management does.
Managers may want to know more details: the security posture of the organization, percent of systems patched (maybe by asset function); incident response status; employee training on security; percent of malicious messages or Web traffic being blocked.
Security/risk awareness teams want operational details: CPU usage, trending threats; blocked Web site attempts (a sign of a botnet) and security violations; number of inactive accounts; number of files or SharePoint sites without an owner; elevated privilege requests.
Tip: Have your source data handy in case you’re questioned on it.
There is no standard for presenting metrics, McBride cautioned. Beyond learning what the audience wants, figure out how to introduce new metrics in your presentation so the audience doesn’t get complacent; obtain buy-in from colleagues (validate the sources of information you will be presenting. They will help correct errors, and show you trends).
Finally, when in doubt, ask yourself: what does this metric mean? would the audience want to know about this metric?
