“Metametrics” are to metrics what “metadata” are to data.
In other words, metametrics are information-about-metrics.
Metametrics include various characteristics of metrics. For example, metrics lists and metrics catalogs typically describe each metric in terms of its scope, purpose, parameters, sources and calculations: these are all metametrics.
Who decides that these particular characteristics of metrics are important enough to specify? How do they decide? Which are the most/least important factors? On what basis are certain metametrics taken into account while others are ignored? Unfortunately, such important considerations are seldom discussed by the people that publish metrics lists and catalogs. Generally, all we see is the end result without the opportunity to understand the thinking process behind it.
Most of all metametrics provide an objective basis for designing and selecting metrics that are actually worth measuring, analyzing, reporting and using. Without metametrics, we’re saddled with hocus-pocus, guesswork and blind faith.
In the information security context, metametrics are valuable:
-
To assess different security metrics on a meaningful, rational and comparable basis, helping us decide objectively which (if any!) might be worth adopting, developing further or parking;
-
When reviewing and reassessing the security metrics currently in use, for example if management is unsatisfied with the current crop or needs factual information in order to manage other security risks;
-
To consider the merits of security metrics recommended by others, and to explain the basis for recommending specific security metrics to anyone, not least our own management;
-
For benchmarking security metrics used by different organizations, or by business units within a large organization, particularly to identify and share more creative and productive ways of measuring information security;
-
In support of a more rigorous, professional approach to information security measurement, which is itself essential for the governance (direction, management and control) of information security.
Just as security metrics are used to measure, manage and improve our information security management system, metametrics help us measure, manage and improve an information security measurement system. Applying systems design to measurement is an example of the innovative thinking behind the PRAGMATIC approach.
Information-about-metrics includes metrics-about-metrics. For example the number of security metrics used by the organization is a relatively crude metametric, while the quality of the organization’s security metrics is potentially much more informative and useful.
 The book identifies nine key criteria defining information security metrics. PRAGMATIC metrics are:
They help us deal with situations, make decisions and improve things for the future;
To the subject matter i.e. information security, governance, risk, compliance, control ...;
They tell us things that we can actually do something about, apart from saying “Oh that’s nice”!;
The numbers are fact-based and cannot easily be faked or manipulated for some hidden agenda;
To the intended audience/s, without creating a lot of head-scratching and hand-waving;
Sufficiently true and precise to allow proportional control (not just stop/go but how fast?);
Security is a dynamic area, so we need up-to-date information at the point decisions have to be made;
Measured dispassionately and objectively, based on verifiable evidence; and
Generate more value than they cost to gather, analyze, present and use.
The book expands on the nine PRAGMATIC criteria, explaining why these are such important factors to take into account when selecting metrics. Chapter 6 lays out the PRAGMATIC method, explaining in straightforward terms how to use the criteria in practice to assess, compare and select metrics. At long last, management has a rational and straightforward way to sift out those few metrics that directly support and facilitate information security from the huge pool of possible (candidate) metrics.
By the way, if you are so new to the field that you don’t already have a long shopping list of security metrics, Chapter 5 tells you where to go looking for them. It is easy to come up with hundreds, even thousands of secuity metrics. To get you started, we describe more than 150 in Chapter 7.
The PRAGMATIC method also has application both in designing security metrics from scratch, and in systematically improving your current metrics. If you are using security metrics that ‘ought to work’ in theory but for some reason don’t seem to work out so well in practice, the PRAGMATIC method helps you understand why they don’t work and identify what would need to change to make them more valuable. Simply altering the way the security metrics are analyzed and presented may be sufficient, otherwise it may be worth exploring whether changing the phrasing or definition of metrics will turn things around.
At the end of the day, some security metrics are so poor they are simply irredeemable: the PRAGMATIC method gives you a way to put lame metrics out of their misery, saving money and encouraging management to focus their attention on the remaining fit-for-purpose metrics. Lacking this crucial step, metrics systems tend to grow like Topsy. As David Lynas put it, we end up “measuring what we can, not what we should”.
PRAGMATIC Security Metrics covers new ground in terms of building a coherent portfolio of information security security metrics, in other words an information security measurement system supporting the organization’s information security management system. Chapter 8 concerns how to design, build and manage the measurement system, while using it is covered in Chapter 10.
NB: if you are familiar with existing approaches to security metrics, you will appreciate that this deceptively simple concept is a significant advance - so novel in fact that we have applied for a patent to control its commercial exploitation. Rest assured, you are very welcome to use the PRAGMATIC method within your organization. Simply buy the book, maybe read the FAQ, and jump right in!
|
