6.4 Security and vulnerability management metrics

Overview and motivation

This topic closes Part 6 by extending the same reliability discipline this part has built, target-setting, guardrail-pairing, honest incident reporting, to a distinct but closely related risk: not whether a system fails on its own, but whether someone makes it fail, or exploits it, deliberately. Vulnerability management metrics measure how well an organisation finds and fixes security weaknesses before they are exploited: how many vulnerabilities exist, how severe they are, and critically, how quickly they get remediated once discovered, since a known but unpatched vulnerability is a standing, quantifiable risk the organisation has chosen to carry, whether deliberately or through neglect.

This topic’s central concern parallels topic 4.4’s treatment of static analysis findings directly: a raw vulnerability count is a poor metric, conflating trivial and critical issues, and it is exposed to exactly the same gaming risks, definition narrowing, suppression, and threshold gaming, that topic 1.2 describes generally. The specific addition security metrics require is time-to-remediate tracked against severity, since a critical vulnerability sitting unpatched for months represents a fundamentally different risk than the same vulnerability caught and fixed within a day, information a simple count alone cannot convey.

For large teams, security metrics carry consequences beyond the immediate technical risk: enterprise organisations face contractual and reputational exposure from a breach, and government organisations face national security, legal, and public trust consequences that make security metrics a matter of genuine public interest, not merely an internal engineering concern. This topic treats vulnerability management with the same rigor and the same guardrail-pairing discipline this book applies throughout, because security metrics are exposed to every gaming risk this book describes, with correspondingly higher stakes when that gaming succeeds.

Key principles

Time-to-remediate by severity matters more than a raw vulnerability count. A critical issue unpatched for months is a fundamentally different risk than the same issue caught and fixed quickly.
Security metrics are exposed to the same gaming risks as static analysis findings (topic 4.4), with higher stakes when gaming succeeds.
Severity classification needs external, standardised criteria wherever possible, not purely internal judgement that can drift toward leniency.
A vulnerability disclosed and fixed quickly is a sign of a healthy process, not a failure to hide. Punishing disclosure discourages the reporting this whole system depends on.
Security debt is a category of technical debt (topic 4.5) and should compete for prioritised remediation capacity on the same explicit, quantified basis.

Recommendations

Track time-to-remediate by severity as the primary metric

For every discovered vulnerability, record its severity (using a standardised scale such as the Common Vulnerability Scoring System, CVSS, where applicable) and track the time from discovery to genuine remediation, not to a ticket being closed or a fix being merged but not yet deployed. Set explicit remediation-time targets by severity, commonly measured in days for critical issues and weeks for lower-severity ones, and track compliance against those targets as the primary security health metric, rather than a raw, unweighted vulnerability count.

Use standardised severity scoring rather than purely internal judgement

Where a standardised external scoring system like CVSS is available, use it as the primary basis for severity classification rather than relying entirely on internal, potentially inconsistent judgement. This mirrors topic 5.1’s escaped-defect classification discipline and topic 6.2’s incident classification discipline, applied here to security specifically, and it resists the same lenient-drift risk those topics warn against, since an externally anchored score is harder to quietly redefine downward than a purely internal one.

Build a genuinely non-punitive vulnerability disclosure and internal

reporting culture

Apply topic 6.2’s blameless postmortem principle directly to security: an engineer who discovers and reports a vulnerability they introduced, or a researcher who responsibly discloses one found externally, should be treated as providing a valuable service, not as confessing a failure. Punishing disclosure, internally or from external researchers, reliably discourages exactly the reporting the entire vulnerability management system depends on, driving real risk underground rather than into a managed remediation process.

Treat security debt as a category within your technical debt backlog

Fold known, accepted-risk vulnerabilities, ones deliberately not yet remediated due to competing priorities, into the same visible, quantified technical debt backlog described in topic 4.5, with the same cost-to-fix versus cost-to-carry framing. This prevents security risk from either disappearing into an invisible, undocumented “we know about it” status or competing unfairly against feature work without an explicit, quantified case for its priority.

Combine vulnerability metrics with exposure and exploitability context

Not every vulnerability with the same nominal severity score carries the same actual risk: a critical vulnerability in an internal tool with no external network exposure is a different risk than the same nominal severity in an internet-facing service handling customer data. Where feasible, weight prioritisation by actual exposure and exploitability context, not severity score alone, so remediation capacity concentrates on genuinely highest-risk items first.

Trade-offs: pros and cons

Approach	Pros	Cons
Raw vulnerability count	Simple to report	Conflates trivial and critical issues; easily gamed through suppression
Severity-weighted, time-to-remediate tracking	Reflects actual risk exposure over time	Requires disciplined, consistent classification and tracking
Purely internal severity judgement	Flexible, tailored to context	Prone to lenient drift and inconsistency across teams
Standardised external scoring (e.g., CVSS) plus context weighting	Consistent, externally anchored, resists gaming	Requires additional context analysis for genuinely accurate prioritisation

The central tension is consistency versus context. A purely standardised scoring approach is consistent and resistant to gaming but can miss genuine context, exposure and exploitability, that determines actual risk; a purely contextual, internally judged approach captures nuance but is prone to the same lenient-drift risk this book warns against for every other classification-dependent metric. Resolve the tension by anchoring on standardised scoring as the consistent baseline, then applying documented, auditable context weighting on top of it, rather than either extreme alone.

Questions to discuss with your team

Do we track time-to-remediate by severity, or only a raw vulnerability count? Pull your actual current metric and check whether it distinguishes a critical issue sitting unpatched for months from one fixed within a day, since a raw count treats these very differently risky situations identically.
Do we use a standardised external severity scoring system, or does classification rely on purely internal, potentially inconsistent judgement? If purely internal, discuss what adopting a standard like CVSS would change about your current classification practice.
Would an engineer who introduced and then reported a vulnerability feel safe doing so, or would they fear punishment? This is the direct security-specific version of topic 6.2’s blameless-culture question, and an honest answer here matters enormously for whether your vulnerability data can be trusted at all.
Do we have a visible, quantified backlog of known, accepted-risk vulnerabilities, or does “we know about it” status quietly become invisible and unaddressed over time? Check whether your security debt is tracked with the same rigor as your general technical debt backlog (topic 4.5).
Does our remediation prioritisation account for actual exposure and exploitability, or does it rely purely on a nominal severity score regardless of context? Pick a real example where two vulnerabilities with similar nominal severity carried very different actual risk, and discuss whether your current process would have prioritised them correctly.
Has a vulnerability’s severity classification ever drifted downward over time with no clear justification? This mirrors the definition-gaming pattern topic 1.2 and topic 6.2 both warn about; audit a sample of your recent classifications for this specific risk.

Sector lens

Startup. Formal vulnerability management processes are often unnecessary very early, but adopting basic automated dependency scanning and a simple, honest internal reporting norm from the start costs little and prevents security debt from accumulating invisibly before the team has the capacity to address it systematically.

Small business. Most modern development platforms include free or low-cost automated vulnerability scanning for dependencies; enable this early and track time-to-remediate for anything flagged as critical, even without a dedicated security function or sophisticated tooling.

Enterprise. Consistent, standardised severity scoring and genuinely non-punitive disclosure culture are both essential and both harder to maintain at scale, where inconsistency across dozens of teams and cultural drift toward blame-seeking after a serious incident are constant risks. Invest in a dedicated security governance function to maintain classification consistency and actively protect disclosure culture.

Government. Security metrics here often intersect directly with national security, regulatory compliance, and public trust, and a serious, mishandled vulnerability can have consequences well beyond a typical private-sector breach. Maintain rigorous, externally anchored severity classification, protect internal and external disclosure culture actively, and treat security debt with the transparency and prioritisation rigor this topic recommends, since an undocumented, quietly accepted critical vulnerability in public infrastructure is a genuinely serious, auditable risk.

Examples

Enterprise. A software company’s security team had, for years, reported only a raw vulnerability count to leadership, a number that had been trending flat, giving a false sense of stability. A revised severity-weighted, time-to-remediate analysis revealed that while the total count was flat, critical vulnerabilities were taking an average of over ninety days to remediate, well beyond any reasonable target, because they were competing unsuccessfully against feature work in every planning cycle with no dedicated, protected capacity. Establishing a hard 7-day remediation target for critical vulnerabilities, backed by protected security-debt remediation capacity mirroring topic 4.5’s technical debt allocation model, brought average critical remediation time down to under five days within two quarters.

Government. A national infrastructure agency discovered, following an external security audit, that internal engineers had been informally avoiding reporting vulnerabilities they discovered in their own code, fearing it would reflect poorly on their performance reviews, a clear parallel to topic 6.2’s blame-driven incident under-reporting pattern. The agency instituted an explicit, publicly communicated policy protecting internal vulnerability reporters from any performance consequence, modelled directly on blameless incident-response practice, and internal vulnerability reports rose substantially within the following year, a result the agency’s leadership correctly interpreted as evidence of improved detection and honest reporting, not evidence of declining code quality, avoiding the natural but mistaken conclusion that a rising number must mean things had gotten worse.

Business case: motivations, ROI, and TCO

The return on rigorous, well-classified, honestly reported vulnerability management is avoided breach cost, which for a serious security incident frequently dwarfs the cost of proactive remediation many times over, alongside avoided regulatory, contractual, and reputational damage. The software company example above shows the specific mechanism: security debt had been silently losing the prioritisation competition against feature work for years, exactly the pattern topic 4.5 warns about for technical debt generally, until protected remediation capacity fixed it directly.

The total cost of ownership includes automated scanning tooling, the protected remediation capacity this topic recommends allocating, and the sustained cultural investment in non-punitive disclosure practice. That cost is modest compared to the cost of a serious, successfully exploited vulnerability that proactive, well-prioritised remediation would have caught and fixed well before it could be exploited.

Anti-patterns and pitfalls

Tracking only a raw vulnerability count: conflates trivial and critical issues and gives a false sense of stability or crisis regardless of actual risk.
Purely internal, unstandardised severity classification: prone to lenient drift and inconsistency across teams.
Punishing vulnerability disclosure, internal or external: drives real risk underground rather than into a managed remediation process.
Security debt with no visible, quantified backlog: loses the prioritisation competition against feature work by default.
Prioritising by nominal severity score alone, ignoring exposure and exploitability context: misdirects limited remediation capacity.
Interpreting a rising vulnerability report count as evidence of declining quality without checking whether reporting itself improved: a specific instance of topic 1.6’s confounding-variable trap.

Maturity model

Level 1, Initiate: Vulnerabilities are tracked, if at all, as a raw count with no severity weighting, no remediation-time tracking, and a punitive disclosure culture.
Level 2, Develop: Some severity classification exists, but standards are inconsistent and remediation time is not tracked against explicit targets.
Level 3, Standardise: Standardised, externally anchored severity scoring and explicit remediation-time targets by severity are applied consistently, with a genuinely non-punitive disclosure culture.
Level 4, Manage: Security debt is tracked in a visible, quantified backlog with protected remediation capacity; prioritisation accounts for exposure and exploitability context, not severity alone.
Level 5, Orchestrate: The organisation can point to specific, measurable reductions in critical remediation time and can demonstrate a sustained, trusted disclosure culture that produces honest, comprehensive vulnerability data.

Ideas for discussion

What is our current average time-to-remediate for critical vulnerabilities, and does it meet an explicit target?
Would an engineer who introduced a vulnerability feel safe reporting it themselves?
Do we have a visible, quantified backlog of known, accepted-risk security debt?
Does our remediation prioritisation account for actual exposure, or only nominal severity?
Has a severity classification ever drifted downward over time without clear justification?

Key takeaways

Track time-to-remediate by severity, not a raw vulnerability count, as the primary security health metric.
Use standardised external severity scoring (such as CVSS) as a consistent baseline, resistant to the lenient-drift risk purely internal judgement invites.
Build a genuinely non-punitive disclosure culture; punishing reporting drives real risk underground.
Treat security debt as a category of technical debt (topic 4.5), competing fairly for protected remediation capacity.
Weight prioritisation by actual exposure and exploitability, not severity score alone.

References and further reading

FIRST.org’s Common Vulnerability Scoring System (CVSS) specification: the standardised severity scoring framework referenced throughout this topic.
OWASP Foundation resources on vulnerability management and secure software development lifecycle practice.
Site Reliability Engineering: How Google Runs Production Systems, by Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds. (the blameless culture principles this topic applies to security disclosure).
NIST Special Publication 800-40, Guide to Enterprise Patch Management Planning: authoritative guidance on vulnerability remediation practice.