 IT Security Metrics
A Practical Framework for
Measuring Security & Protecting Data
Author: Lance Hayden
Pages: 368
ISBN: 978-0-07-171340-5
Publisher: McGraw Hill, 2011
Buy it from: Amazon
Part I of the book is a general introduction to security metrics, which are commonly described both as the results or output of the measurement activity (better defined as “measurements”!) and the measurement activity itself. Lance’s key point here is that the measurements - meaning the data values - are merely the mechanistic end products of the measurement activities, while the overall process is the more interesting bit. The metrics process as a whole involves not just literally measuring stuff, but earlier deciding what is of concern, why it needs to be measured, who will measure it and how and when, what the measurements tell us, and ultimately what we do as a result of measuring stuff. It’s all about gleaning meaning from data and using the knowledge in support of the business.
In part I, Lance eloquently describes how to apply the GQM (Goal-Question-Metric) method in this context, with helpful examples. Despite being quite simple, GQM is valuable in that it forces us to make explicit links between business objectives and metrics, using as middlemen the rhetorical questions that managers might typically pose as a way to envisage how certain metrics will address certain goals. [We would argue that framing the issues and posing sensible questions is the true art to metrics, and is a prime reason why published metrics lists or catalogs are generally of such limited value. Mostly they ‘jump directly to the answers without showing the workings’. Sure, we know we could measure X, but why would we even want to do that? What would the measurements of X tell us, and is that something we actually needed to know? Using the GQM method ‘backward’ means reverse-engineering the kinds of business goals that the suggested metrics might address ... which then presents the opportunity to use GQM conventionally and so identify a whole bunch of possible metrics, aside from those suggested.
Part I ends by delving into nominals, ordinals, intervals and ratios, the basic principles of numbering that Jaquith covers at length. If you still cling to the idea that risk values can be calculated by adding or multiplying the values assigned to high, medium or low categories of threats, vulnerabilities and impacts, you should study this part very carefully. I get it and I really want to move ahead now please. There’s so much more to security metrics than boring number theory and statistics, important though they are!
Part II covers implementation through a framework - the SPM (Security Process Management) Framework - that once again associates the business with the metrics. Deming’s PDCA (Plan-Do-Check-Act) continuous improvement approach is the fundamental concept here, one that fans of ISO9k and ISO27k will immediately recognize and appreciate.
At one point, Lance suggests improving security through a sequence of security improvement projects, each supported by a suite of metrics. While I agree that metrics are a vital plank in any project, I’m not entirely convinced that security can or should be chunked into discrete activities, since there are so many overlaps and dependencies in practice. Sure, there are implementation projects to install, configure and launch IT security subsystems (such as intrusion detection systems, user authentication technologies or whatever) but a well conceived security strategy treats those IT projects as mere steps on the way: the grand plan blurs a coherent sequence or suite of projects together to deliver a coherent security architectural vision that is BUSINESS-as-usual.
Later in Part II, Lance talks about the Security Measurement Project, which again implies a discrete project structure with pre-defined objectives, a beginning, a middle and an end. Its true that some security metrics have limited utility: they are measured and used for a while until their purpose is served, and then things move on. Many others, however, are of value on an indefinite basis, albeit some may evolve, gradually adapting in line with the organization’s progress towards security and metrics maturity.
Part II also skims through basic statistical analysis - normal distributions, sampling, confidence limits, correlation and so on - plus the scientific aspects such as testable hypotheses. It’s a handy reminder if you know this stuff already, but a barely adequate introduction if you don’t. Statistics and science are a lot like law: if you don’t truly know what you’re doing, they are best left to the experts.
Part III on security measurement projects discusses metrics in four IT security domains:
-
Security operations: the strictly IT perspective is very evident here too, for example suggesting “Percent of IT budget devoted to IT security” as a metric. Lance might be a little shocked to discover that IT security, and in fact IT, are mere incidentals to most business people! Security metrics do have value and purpose within IT, but that’s just part of a much bigger picture. That said, the worked examples demonstrate an approach that can be applied more widely.
-
Compliance and conformance: talks about mapping and aligning disparate compliance obligations, and Flesch readability scores ... a curiously narrow take on the topic.
-
Security cost and value: you might expect this to cover financial metrics relating to the costs and benefits of security, but in practice it slips back into statistics such as Poisson distributions and Monte Carlo simulations.
-
People, organizations and culture: it is a refreshing sign to see an IT security specialist seriously considering the human element, even mentioning ethnography at one point.
Part IV takes us ‘beyond security metrics’, meaning (apparently) the utility of metrics in managing programs of security improvement projects rather than just isolated projects. Lance’s continued reference to projects is my biggest beef with the book. Lance comments “In most of the companies visit during consulting engagements, security is managed on a project basis, whether the purpose of those protects are assessment, development, or implementation. We all understand security projects, but many of my clients complain that the project approach to security meets only some of their needs.” (page 308). How true! It reminds me that the book’s title concerns IT security. Many IT professionals see life as a series of projects. Real life is more than that.
There is a case study in each of the four parts, contributed by other authors. Although these are not bad in themselves, they are not particularly well integrated into the context of the book - they don’t really illustrate the specific points made in each part - but perhaps I am being harsh since so few organizations would claim to have mature security metrics processes, and what are the chances that any of them would have followed the approach written in this book before it was even published?!
Overall, IT Security Metrics is well-written, thought provoking, and definitely a valuable contribution to the field. Lance’s eloquent description of GQM is particularly worthwhile.
|
