Information Technology, especially the next generation of Social, Mobile, Analytic and Cloud (SMAC) technologies, is a complicated beast (the tale of the blind men describing the elephant seems to fit well). Organizations slice and dice IT roles and responsibilities, and then further subdivide the tasks so that technical people can focus and specialize. Security is, without doubt, one of those highly specialized areas but it is also an area that is tough to define and measure success.
In the past decade or so some organizations have adopted a service-oriented approach to IT – internal customers “buy” one or more of the services offered by the IT department. Each IT service includes a “package” of hardware, software, networks and underpinning services (such as security) that, when combined, deliver solutions that are useful for the business (email, for example).
Security overlaps and impacts almost all other IT functions in one way or another, so it is often difficult to separate what the security functions do and what they deliver.
Being analytical engineers, we always like to specify what is being provided (the services), to establish the delivery objectives and targets (the service level objectives), and then to measure (and report on) how well the system worked.
IT service metrics play a crucial role in this.
I like to think that metrics are meta-measurements: they specify what is to be measured, what measurement methods are appropriate, and how the measurement is to be captured. A measurement is an instance of a metric. Another more general definition is provided here.
A draft document from NIST (National Institute of Standards and Technology) in the USA, Special Publication 500-307, addresses the topic of Cloud Services Metrics Description. An emerging ISO standard, ISO/IEC 19086-1, which provides a framework and terminology for cloud computing service level agreements, also includes a definition for metrics.
The formulation and standardization of service metrics, including those for security, has become an important topic in the road to cloud computing maturity.
As is illustrated by the diagram above, metrics can be defined at three levels:
Service elements (which could also be thought of as microservices) are the “piece parts” of a complete service; for example, a local access link would be a service element for Email-as-a-Service, and can be characterized by bandwidth, delay and jitter metrics;
Services provide a self-contained IT capability that is useful to its consumers; for example, an Email-as-a-Service provides for email storage, forwarding, and delivery; an email service could be characterized by mail delivery delay, probability of misrouting, mailbox reachability, etc.;
Solution, which can be defined as a set of services that, when combined, deliver business results (such as a solution for bank-to-bank funds transfer over email); solution metrics would primarily be business-oriented and could include funds transfer delay, transfer capacity, service resilience, and the level of privacy protection.
Needless to say, there are no universal metrics – there is always a context associated with using any metric. However, metric can be examined from different perspectives.
For example, service metrics may exhibit the following characteristics:
- Time-based – these metrics involve time and include rates, delays, periods, dates, etc.;
- Location-based – these metrics are location-based using such things as addresses, nationality or geography;
- People-based – these metrics include personnel-oriented items such as clearance levels, training, rank, etc.;
- Quantity-based – these metrics involve counting in some way such as the number of objects, numbers, features, lists or capacities;
- Quality-based – these metrics are associated with the quality of a service in some way, including efficiency, effectiveness or performance; and
- Qualitative-based – these metrics can be derived from non-quantifiable characteristics or human judgements (e.g., mean opinion score for voice quality).
Metrics can also be divided into different topic areas:
- Governance metrics for items such as risk, privacy, compliance and effectiveness;
- Process metrics for the overall business service (e.g., a supply chain or high level process);
- Functional metrics that list the service’s capabilities and capacities (i.e., WHAT is included);
- Performance metrics that relate to how well the service is operating;
- Security metrics that capture the level of access controls, privacy protection, non-repudiation, delivery assurance, etc.; and
- Management metrics that relate to the efficiency and effectiveness of the IT provider’s administration and maintenance support.
Security services and metrics
To define security metrics, it is first necessary to ask what is within scope for the security services. It is then possible to identify the context, the service level objectives, the metrics that should be applied, and the measurements that are needed.
Security can be a very general topic but also very pervasive in that touches almost all aspects of IT (i.e., it is a horizontal service). Security metrics can be both technical and business-oriented, and can be interpreted very broadly or quite narrowly.
For example, the following may or may not be in scope for a specific security service (depending on the context and specific solution):
- Physical security – Is the location of a data centre a security question?
- Service availability – is availability a security or performance issue, or both?
- Access rules and policies – is access to a system a functional or a security issue?
- Network security – are third party service elements included as part of the security service?
- Data protection – Is disposition of information a security responsibility?
- Privacy – is protecting personal information considered to be a security-related objective?
A number of standards and guides for security techniques have been developed, including:
In “A Guide to Security Metrics,” the SANS Institute recommends the following seven key steps to guide the process of establishing a security metrics program:
- Define the metrics program goal(s) and objectives;
- Decide which metrics to generate;
- Develop strategies for generating the metrics;
- Establish benchmarks and targets;
- Determine how the metrics will be reported;
- Create an action plan and act on it; and
- Establish a formal program review/refinement cycle.
In summary, it should now be clear that establishing your own set of security metrics won’t be as easy as it might seem at first!
These are my thoughts. What do you think?
