Metrics are good, but just because they’re precise doesn’t mean they’re useful:
I’ve been thinking a little bit about “threat/vulnerability” pairing. You know the drill, go out, get a scan – match the scan data to existing exploits, and voila! You’ve got risk.
Now regular readers and FAIR practitioners know that I don’t believe this exercise gives you risk at all. In fact, in FAIR terms, I’m not sure this exercise does much for finding Vulnerability.
My Assertion To You: The industry loves T/V pairing because it is precise. It looks good on paper, and if you’re a consultant doing it, it looks like you’ve earned your hourly rate. We love The precision of T/V pairing gives us a false sense of accuracy.
Accuracy, Precision, And Threat/Vulnerability Pairing,
23 July 2007
He goes on to point out you also need to consider who’s likely
to attack you, as in such Threat Agents, as he calls htem,
may be too stupid to use a given exploit, or too smart to use it
because they’ve got a better way.
He recommends some statistical analysis to help out.
I’d also recommend more basic steps, such as not using IE
and shifting away from other monoculture software until you’ve
got a mix of software from different sources.
Those things will usually get you in trouble with sales and marketing,
however, because hey, they’ve never had any problems, well, not
many, and it’s not their job to fix them.
The precise thing isn’t necessarily the right thing.