By John Pescatore and Steve Bittinger
Gartner Inc.
Choosing the appropriate metrics will enable IT security directors to focus on effective security practices that address critical business needs, such as reducing downtime.
What You Need to Know
IT security directors should use selected metrics – chosen according to business impact, enterprise control and measurability – to drive organizational change and process improvements, improve reporting capabilities, and justify technology investments.
Analysis
What are the key criteria and standards for selecting security metrics?
IT security directors who are evaluating security metrics should consider three closely related criteria:
– Business impact – Apply security metrics to areas that have a significant impact on important business processes.
– Enterprise control – Apply metrics to areas where you can exercise control or choice.
– Measurability – Be able to obtain the data that is required for a given metric, as well as measure phenomena on which you can take action (for example, vulnerabilities and the time required to reduce them).
The most important goal of a security system is to help the enterprise meet its business performance goals. Security metrics should easily translate in terms of direct impact on the enterprise in areas such as the downtime of critical systems, regulatory compliance and customer trust. Security metrics also must be meaningful to the enterprise’s security and IS organizations. However, the primary consideration must be business relevancy.
Security metrics should be used to drive meaningful change in increased efficiency or improved security. Therefore, you should measure values that will drive change. Measuring the number of malicious-code attacks or attackers, for example, is not productive because these are factors beyond your control. However, the number of vulnerabilities and the time needed to reduce them are under the control of the IT security organization. Thus, they are meaningful metrics.
There are no true standards for security metrics. Related efforts include:
– The (U.S.) National Institute of Standards and Technology Security Metrics Guide for Information Technology Systems (Special Publication 800-55) – This document provides guidelines for establishing security metrics, especially in response to regulatory compliance pressures (see http://csrc.nist.gov/publications/nistpubs/800-55/sp800-55.pdf).
– International Organization for Standardization 17799 – This comprehensive IT standard includes a list of common security areas. Map your metrics against these areas to determine their coverage.
– (U.S.) Regulatory reporting requirements – Many legislative and regulatory initiatives mandate reporting requirements for security. This can influence your security metric decision. These initiatives include the U.S. Gramm-Leach-Bliley Financial Services Modernization Act of 1999, the U.S. Health Insurance Portability and Accountability Act, the U.S. Public Company Accounting Reform and Investor Protection Act of 2002 (also known as the Sarbanes-Oxley Act) and California Senate Bill 1386.
What are examples of effective security metrics?
Gartner has identified a set of high-level metrics – many of which represent “roll-ups” of lower-level measurements – that can be effective for large enterprises. By weighting and aggregating these metrics, you can develop security-service-level tracking programs that show their progress against goals. The primary metrics are:
Resistance to Attack
What percentage of known attacks is the enterprise vulnerable to? When did you last check for vulnerabilities to known attacks? These metrics require that you perform periodic vulnerability assessment and threat monitoring. The results will change whenever a new vulnerability is discovered in Microsoft Windows or other software products, driving you to take action.
Internal Protections
What percentage of the enterprise’s software, employees and suppliers has been reviewed for security compliance? What percentage of the enterprise’s critical data is strongly protected? These metrics consider key internal preparedness issues. Have you established requirements for your software and service providers to demonstrate the delivery of secure products and services? Have background checks been conducted on employees who have access to sensitive data and systems? Are databases that store sensitive information protected with encryption or two-factor authentication?
Process Improvement
How many PCs or other machines were involved in the most-recent virus incident? How many days typically elapse between the release of a critical patch and its implementation on your desktops and servers? Some metrics, such as the number of machines affected by a virus attack, are key indicators of improvement in security processes. Viruses are the most likely attack that you will experience. Thus, reducing the number of desktops and servers that are affected by each virus is an important measurement. Faster patching is the next best thing to buying software that requires fewer patches.
Efficiency and Effectiveness
What is the enterprise’s security spending as a percentage of revenue? What percentage of system downtime is caused by security lapses? Gartner estimates that the average enterprise will spend 5.4 per cent of its IT budget on security in 2003. The average enterprise will spend seven per cent of revenue on IT in the same period. Thus, security spending will consume an average of 0.38 per cent of revenue. If this percentage of revenue grows faster than security-derived downtime decreases, something is wrong with your security program.
What should be done with the data that is derived from security metrics?
Security metrics should be used to:
– Allocate resources (people and funding)
– Report to management about the progress and effectiveness of security programs
– Drive change and process improvement in security enterprisewide
In addition, lower-level metrics should be used by internal organizations that are responsible for deploying and operating security systems to track improvements. The IS organization can use metrics to determine if new security measures are having the desired impact, and to set the levels of performance expected of outsourcing providers. Finally, the chief information security officer can create security-service-delivery charts for presentation to the CEO and the board of directors by weighting and combining individual metrics.
Resource allocation should be driven primarily by business criticality, although the gap between the required service level and the state of security is also an important consideration. For example, many enterprises spend more on detecting attacks than on preventing them. Appropriate security metrics quickly would show that investments in antivirus technologies or application-level firewalls would be more cost-effective.
How can metrics be used to drive security improvements?
IT security directors that face budget constraints can use security metrics to justify increases in security budgets or staffing. You may need security metrics to gain management backing for changes in processes, such as locking down desktops or requiring new e-business systems to use host-based intrusion prevention systems. Many enterprises need security metrics to make sound insourcing or outsourcing decisions.
A key element of security metrics is a value scale that indicates what constitutes “good enough” security for the enterprise. Larger enterprises can establish the acceptable level by providing metrics for each business area and determining where the dividing line should be. For example, an enterprise can perform backward analysis to determine the minimum reaction time that would have been required to completely protect it from recent threats, and use this data to set proactive mitigation and blocking time goals for the IT security organization.
In the long term, benchmarking will be the most effective mechanism for turning security metrics into process improvements. An enterprise whose security metrics are significantly different from its peers may need to re-evaluate its systems management and security management practices. Industry norms will drive generally accepted security service levels, although each enterprise’s business requirements and IT deployment decisions will justify deviations above or below the norms.
Key Issue
What are the policies, best practices, organizational structures and security architectures required for effective IT risk management?
Entire contents