Security Metrics

Replacing Fear, Uncertainty, and Doubt

Author: Andrew Jaquith

Publisher: Addison Wesley, 2007

Pages: 306

ISBN: 0-321-34998-9

Buy it from: Amazon

Security Metrics is promoted as a comprehensive guide to security metrics best practices. The coverage is certainly broad, ranging from the mathematical considerations underpinning metrics to their use in measuring technical security elements such as antivirus and application security, and managing security programs. However, most of the metrics are IT/technological metrics measuring security technologies, technical processes and their inputs: the broader aspects of information security as opposed to IT or cyber-security are not so well represented.

The final chapter on designing security versions of Kaplan and Norton’s Balanced Scorecard is as challenging and provocative as it is helpful. The original Balanced Scorecard used a handful of metrics on each of four specific perspectives (financial, customer, internal business processes, and learning and growth) as a decision support tool for corporate performance management. Andrew proposes modifying the perspectives to suit security management (for instance changing the customer perspective from that of customers of the business to internal customers of the security function), and suggests a selection of metrics for the altered perspectives. Although he is reasonably explicit about the proposed changes, readers are encouraged to consider and adapt the suggestions to suit their circumstances rather than simply follow a generic menu.

Andrew is a prolific, highly-regarded and influential writer, widely acknowledged as a guru in the field of security metrics. He writes in a forthright style that appears to discount, perhaps even discredit alternative approaches and opinions that he clearly disagrees with, and as such he has at times led the field down a narrow, somewhat introspective path. His strident descriptions of good metrics, bad metrics and non-metrics, for example, have perhaps unwittingly constrained the professional dialog to number theory and statistics. According to Andrew, metrics must be expressed as a cardinal number or percentage, not with qualitative labels like “high”, “medium” and “low”, and expressed using at least one unit of measure, such as “defects”, “hours” or “dollars”* We consciously favor a more practical though less academically rigorous approach reflecting the way metrics are actually being used to measure and manage information security in organizations around the globe. Andrew is quite right to point out that it is mathematically unsound to calculate compound security risk scores using simple arithmetic based on “High = 3, Medium = 2 and Low = 1” … but in the absence of a better approach to assessing security risks, such analysis arguably serves a valuable practical purpose. Traffic-light colors are indeed facile … but as a means of focusing management attention on serious risk and security matters, they clearly have a legitimate rôle in business - albeit as part of a more comprehensive, well-rounded approach to metrics.

* The same bias/prejudice affects many proponents of FAIR, although FAIR’s daddy - Jack Jones - does acknowledge that qualitative measures, done well (an important caveat), have value too: quantitative approaches are not the only way.