An Analytical Framework for Measuring Cybersecurity Defense Systems
Jothy Rosenberg, PhD Founder & CEO
Dover Microsystems
Abstract
This paper presents an analytical framework to measure the effectiveness of cybersecurity defense systems. The analysis focuses on the specific types of software and firmware vulnerabilities a system protects against, as well as their corresponding level of severity. A system is most effective when it protects against the largest number (or percentage) of the highest severity vulnerabilities. To create this framework, we considered The MITRE Corporation and NIST (National Institute of Standards and Technology) Common Vulnerabilities and Exposures (CVEs) list as the normative set of vulnerabilities. We also used the Common Vulnerability Scoring System (CVSS), an open framework for communicating the characteristics and impacts of IT vulnerabilities, as the standard severity score for each vulnerability. In addition, we used MITRE’s Common Weakness Enumeration (CWE) list—which is sponsored by the U.S. Department of Homeland Security (DHS) Cybersecurity and Infrastructure Security Agency (CISA)—to aggregate large groups of CVEs under specific CWE categories. In this paper, we describe our analytical framework, and how it applies to the cybersecurity defense mechanisms one will find in commercial products. For each defense mechanism a system offers, we show how we calculate the number of CVEs it protects against, and how we calculate an overall system-protection score. As far as we know, this level of in-depth analysis has never been done for any cybersecurity system, but we hope to see it broadly applied in the future.
The need for an objective analytical framework What is an analytical framework? DEEP says “Analytical frameworks are designed to structure an analyst's 1
thinking, and to help logical thinking in a systematic manner. In short, analytical frameworks are models that aim to guide and facilitate sense making and understanding.” One major problem in the cybersecurity arena is that there are currently no fair and consistent ways to measure or compare cybersecurity defense systems. That is because there is no structure to guide an analyst’s thinking about cybersecurity defense concepts. The terminology used by vendors, analysts, and users is often inconsistent and misleading, and some vendors use this confusion to their advantage. The result is that there is no objective measure to compare and contrast one system to another, and with 3,000 cybersecurity vendors , we are long past due for an objective analytical framework to compare 2
and contrast them.
The expanding universe of software vulnerabilities To measure the effectiveness of a cybersecurity mechanism, we must first understand the vulnerabilities that the mechanism is attempting to mitigate. The vulnerabilities that we are focusing on are the result of errors in the software running a given system. A programming error is commonly referred to as a “bug”, and attackers look for bugs that may be exploited to either damage a system or exfiltrate sensitive information. According to Steve McConnell in his seminal book, Code Complete, there are on average 15-50 bugs per 1,000 lines of delivered code. To give that statistic some context, consider this: the Android operating system has 15 million lines of 3
source code, Windows 7 has 40 million, and a Ford F-150 has 150 million lines of code. Cybersecurity Ventures 4
estimates that there are 111 billion lines of new software code produced each year. That is, roughly, over 1.6 5
billion bugs added to the world’s software per year. The FBI told this author they estimate at least 2% of such bugs can be exploited by remote attackers which means 32 million new exploitable attack points in new deployed software per year. This is a huge, and ever-expanding universe of exploitable vulnerabilities. It is impossible to recreate every known vulnerability in order to test a particular cybersecurity product. Instead, we need a way to build up an analytical framework using elements based on these real-world vulnerabilities but in such a way as to be manageable and be based on broadly accepted and adopted elements.
The foundational elements of this analytical framework Thankfully, an enormous amount of work has been done documenting existing vulnerabilities, quantifying the severity of each vulnerability, and identifying common features in groups of vulnerabilities. We use three highly regarded datasets as the basis for our framework: CVE (Common Vulnerabilities and Exposures), CVSS (Common Vulnerability Scoring System), and CWE (Common Weakness Enumeration).
CVE is an open standard that provides globally unique identifiers for known cybersecurity vulnerabilities in software. It was started by MITRE in 1999 to address the profusion of different names to describe the same vulnerabilities. CVE entries (also referred to as “CVE IDs” or simply “CVEs”) are assigned by CVE Numbering Authorities from around the world, with each having an identification number, a description, and at least one reference to an instance of that vulnerability in a specific piece of software. The use of CVEs ensures confidence among parties when discussing or sharing information about a unique software or firmware vulnerability, provides a baseline for tool evaluation, and enables automated data exchange. MITRE Corporation, an FFRDC , maintains the system with funding from the National Cyber Security Division of the 6
United States Department of Homeland Security. The CVE list is posted on MITRE's site as well as in the US 7
National Vulnerability Database (NVD) which is part of NIST, the National Institute of Standards and 8
Technology. Since CVE is a fast moving list—entries are added every day—we must specify a date on which we based
our analysis. For this paper that date was 18 July 2019.
CWE — Common Weakness Enumeration
While building CVE, MITRE developed a preliminary classification of vulnerabilities, attacks, and faults. This work evolved into the CWE list, which was first published in 2005. Where CVE lists specific instances of vulnerabilities (numbering 119,306), CWE defines categories of software weaknesses (currently numbering 800). A software weakness is an error that can lead to a software vulnerability. Software weaknesses include, for example, buffer overflows, code evaluation and injection, and insufficient verification of data. Like CVE, CWE is maintained by MITRE. The most recent version of the CWE list was released in June 2019.
MITRE created CWE with code assessment in mind. It “serves as a common language for describing software security weaknesses, a standard measuring stick for software security tools targeting these vulnerabilities, and a baseline standard for weakness identification, mitigation, and prevention efforts.” CWE is 9
also used to organize the massive list of CVEs. As CVEs are formally installed into the list, a CWE is associated with each one to create groupings that map potentially thousands of CVEs to the much smaller number of CWEs. For example, there are 12,605 CVEs due to buffer errors, and all of those CVEs map to one CWE: ID 119, Improper Restriction of Operations within the Bounds of a Memory Buffer (i.e. buffer overflow).
CVSS — Common Vulnerability Scoring System
CVSS provides a free and open industry standard for assessing the severity of computer system security vulnerabilities. Its quantitative model ensures repeatable accurate measurements while enabling users to see the underlying vulnerability characteristics that were used to generate the scores. Scores are calculated based on a
Federally funded research and development centers (FFRDCs) are public-private partnerships which conduct research for the United 6
formula that depends on several metrics that approximate ease and impact of exploit. Scores range from 0 to 10, with 10 being the most severe. CVSS is well suited as a standard measurement system for industries, organizations, and governments that need accurate and consistent vulnerability impact scores. The National Vulnerability Database (NVD) provides CVSS scores for almost all known vulnerabilities. CVSS Severity and the numeric range associated with that label (Low, Medium, High, Critical) are listed in Table 1. In our analytical framework, we pay close attention to vulnerabilities with a High or Critical CVSS score greater than 7.0, which we will refer to collectively as “Severe” because these frequently allow attackers to completely take over the software, steal data, or prevent the software from working at all and it is most important that our defense mechanisms are effective at blocking these vulnerabilities.
Building the analytical framework so it is objective and will last To make this analytical framework objective and ensure that it will last—and because cybersecurity defense mechanisms must focus on weaknesses in software and firmware (that is, on CWEs), while actual attacks are represented by CVEs—four key principles drove its construction:
• Consider only CVEs with an assigned CWE;• Utilize only CWEs that have one or more associated CVEs;• Value defense mechanisms that block CVEs with higher CVSS scores;
and when applying this framework to a specific cybersecurity system,• Group CWEs together that correspond to a single defense mechanism.
The following four sections will rationalize and explain each of these principles.
Consider only CVEs with an assigned CWE
Because the CVE list was begun in 1999 and the CWE and the formality of CVE Numbering Authorities came much later—around 2005—about half of the 119,306 CVEs do not have a CWE assigned to them. This is because many of them are incomplete, not reproducible, or generally of low quality. (See relevant Rapid7 blog on this topic. ) More importantly, defense mechanisms are closely tied to the software weaknesses the CWE was 10
created to describe. Therefore, for this analytical framework we consider only the CVEs that have an assigned CWE. This prunes the CVE dataset used in our analysis down to 63,002 CVEs.
Utilize only CWEs that have one or more associated CVEs
From the complete list of 808 CWEs, the number that have been associated with at least one CVE is 84. We refer to this as our Active-CWE list. One of the goals of CWE is to support finding these common types of software security flaws in code prior to fielding even if it is a CWE that has not yet been associated with a real attack in the wild. Also included in the CWE list are “views” that further group CWEs into views such as by Research Concepts, by Development Concepts, by Architectural Concepts, by Top 25 most severe, by the Seven Pernicious Kingdoms of weaknesses, by Software Fault Pattern Clusters, and many other views, each of which
cover a subset of the 808 total entries. Our Active-CWE list of 84 ignores these views and only considers individual weaknesses. Our framework scores security approaches based on which CWE classes are mitigated by the approach. We base the score of a given CWE on the total value (according to CVSS) of the vulnerabilities included in the CWE.Therefore we ignore the CWEs with no CVEs as they do not add value to this analytical framework (i.e. they can, for now, be considered theoretical). We verified the number of vulnerabilities (i.e. CVEs) for each CWE in our Active-CWE list measured on 18 July 2019 at the National Vulnerability Database (nvd.nist.gov).
Value defense mechanisms that block CVEs with higher CVSS scores
A cybersecurity defense that blocks the most severe attacks, as measured by CVSS score, is more valuable than one that blocks lesser attacks. This is because a CVSS score greater than 7.0 is defined to mean the following:
• The attack comes over the network; • The complexity of the attack (the attacker’s sophistication level, or how hard they have to work) is low; • No special privilege level is required to accomplish the attack; • No user interaction is required to enable the attack; • One or more of confidentiality, integrity, or availability is adversely affected, which is likely to have a
catastrophic adverse effect on the targeted organization or individuals within the organization; and • No workaround is possible (the issue must be fixed and patched to prevent further instances of attack).
Group CWEs together that correspond to a single defense mechanism
When applying this framework to a specific cybersecurity system we will group the CWEs together that correspond to a single defense mechanism in that system. This is because many CWEs are related such that a single defense mechanism applies. For example, in the general category of “memory safety” there are ten CWEs. If a single defense mechanism handles all of these, they can be grouped together since the goal of this framework is to measure each unique defense mechanism and not each individual CWE.
Table 2 includes all 84 CWEs that have one or more CVEs to which they are assigned. These have been grouped by a defense mechanism that, if effective, could block most if not all CVEs in that grouping (because they all relate to a small number of CWEs representing a software weakness that a defense can be built to overcome). Table 2 has a row for each CWE and has the following columns:
CWE ID the assigned identifier of a particular CWE; Name the official name (or nickname) of the CWE; Description the summary of this CWE; CVE count the vulnerability count (number of CVEs) for this CWE; CVSS > 7.0 the number of associated CVEs with a CVSS score greater than 7; Grouping a group of one or more CWEs that can be blocked by a defense mechanism; CVEs in Group the sum of the vulnerability count for all the CWEs in a grouping; CVSS > 7 the sum of the vulnerability count in a grouping with SVSS > 7; % Severe the percentage of the total vulnerability count for a grouping that have SVSS > 7.
In Table 2 we chose to consider nine groupings (that can be addressed by what we are calling defense mechanisms) based on our experiences working in the field and seeing how others in the field tend to group vulnerabilities. Those nine groupings are defined as:
Any specific security product will address some specific sets of CWEs. Table 2 also includes in aggregate the remaining 27 CWEs not included in these groupings covering a total of 63,002 CVEs (of which 21,903 are Severe with CVSS > 7). The table identifies the total number of CVEs and Severe CVEs for each grouping, and the percentage of all CVEs that are considered Severe. This table can be directly used to evaluate a particular defense mechanism against all CVEs and against Severe CVEs. When doing so, some CWEs may need to be
Memory safety Memory safety is the state of being protected from various software bugs and security vulnerabilities when dealing with memory access, such as buffer overflows and dangling pointers.
Sanitization Sanitization (sometimes called Input Validation) is the process of examining a stream of text (frequently in the form of a scripting language) and producing a new stream that preserves only whatever constructs are designated "safe" and desired. SQL sanitization can prevent SQL injection attacks. A defense mechanism in this category can enforce and verify that a user-provided sanitization routine was executed prior to the text being utilized as specified. This does not verify the “goodness” of the sanitization function. That requires a formal methods proof and is rarely done; careful inspection will be used.
Read-Write-Execute Read-Write-Execute (RWX) refers to access control settings on data. Read – allows access to data to view the data only. Write – allows the data to be modified. Execute – allows the data to be treated as instructions to be executed by the processor.
Web protection Very similar to Sanitization except focused exclusively on Web browsers and servers. This includes HTML sanitization as well as processing of URLs and HTTP constructs to protect against attacks aimed at Web-based systems.
Compartmentalization Compartmentalization separates information belonging to one execution thread from other execution threads and controls access to both data and instructions between all compartments. The idea is to have a compartment that contains trusted and vital software be protected from malicious code in other compartments.
Access Control Access control permits or denies users the use of certain data where use specifies whether the data can be read and written or just read. Users must be authenticated such that their identity is confirmed at whatever level of certainty required by the device owners.
Privacy Data privacy deals with the ability an individual has to determine what data can be shared with third parties. Frequently this is called Data Exfiltration protection.
Cryptographic issues This grouping collects issues related to the use of cryptography but that usage is either done incorrectly or uses weak versions of functions when better ones exist.
Architectural, Configuration, or User Error
This grouping are software weaknesses that are classified by the CWE creators as relating to how the software was architected (and needs to be changed at the source level), how it has been configured, or the use of that software can create security vulnerabilities through user error.
JOTHY ROSENBERG OF 7 13
moved to a different grouping if they don’t apply to that defense mechanism but they must still be considered and not dropped from the analysis. The list of 27 not grouped in this table can be examined to see if they 11
apply to a particular defense mechanism being considered.
To examine the uncategorized 27, go to cvedetails.com, use the view ‘CWE Definitions’ and look at the bottom 27 entries on page 2.11
Improper Restriction of Operations within the Bounds of a Memory Buffer
The software performs operations on a memory buffer, but it can read from or write to a memory location that is outside of the intended boundary of the buffer.
12,386 7,563
Memory Safety 17,663 9,147 51.8%
125 Out-of-bounds Read The software reads data past the end, or before the beginning, of the intended buffer. 1,411 176
416 Use After FreeReferencing memory after it has been freed can cause a program to crash, use unexpected values, or execute code.
1,084 526
190Integer Overflow or Wraparound
The software performs a calculation that can produce an integer overflow or wraparound, when the logic assumes that the resulting value will always be larger than the original value. This can introduce other weaknesses when the calculation is used for resource management or execution control.
1,010 230
476NULL pointer dereference
A NULL (or uninitialized) pointer dereference occurs when the application dereferences a pointer that it expects to be valid, but is NULL, typically causing a crash or exit.
810 167
787 Out-of-bounds Write The software writes data past the end, or before the beginning, of the intended buffer. 556 276
134 Uncontrolled Format String
The software uses a function that accepts a format string as an argument, but the format string originates from an external source.
202 109
415 Double freeThe product calls free() twice on the same memory address, potentially leading to modification of unexpected memory locations.
152 70
191 Integer Underflow
The product subtracts one value from another, such that the result is less than the minimum allowable integer value, which produces a value that is not equal to the correct result.
45 24
123Write-what-where Condition
Any condition where the attacker has the ability to write an arbitrary value to an arbitrary location, often as the result of a buffer overflow.
The product does not validate or incorrectly validates input that can affect the control flow or data flow of a program.
7,078 2,332
Sanitization 14,261 7,785 54.6%
89 SQL Injecion
The software constructs all or part of an SQL command using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the intended SQL command when it is sent to a downstream component.
5,416 4,263
77 Command Injection
The software constructs all or part of a command using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the intended command when it is sent to a downstream component.
776 534
78 OS Command Injection
The software constructs all or part of an OS command using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the intended OS command when it is sent to a downstream component.
468 402
74 Injection
The software constructs all or part of a command, data structure, or record using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify how it is parsed or interpreted when it is sent to a downstream component.
236 83
502Deserialization of Untrusted Data
The application deserializes untrusted data without sufficiently verifying that the resulting data will be valid.
211 142
428 Unquoted Search Path or Element
The product uses a search path that contains an unquoted element, in which the element contains whitespace or other separators. This can cause the product to access resources in a parent path.
32 11
91 Blind Path Injection
The software does not properly neutralize special elements that are used in XML, allowing attackers to modify the syntax, content, or commands of the XML before it is processed by an end system.
18 3
88Argument Injection or Modification
The software does not sufficiently delimit the arguments being passed to a component in another control sphere, allowing alternate arguments to be provided, leading to potentially security-relevant changes.
15 10
90 LDAP Injection
The software constructs all or part of an LDAP query using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the intended LDAP query when it is sent to a downstream component.
The software constructs all or part of a code segment using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the syntax or behavior of the intended code segment.
2,345 1,543
Read-Write-Execute 2,765 1,748 63.2%
434Unrestricted Upload of File with Dangerous Type
The software allows the attacker to upload or transfer files of dangerous types that can be automatically processed within the product's environment.
314 156
129Improper Validation of Array Index
The product uses untrusted input when calculating or using an array index, but the product does not validate or incorrectly validates the index to ensure the index references a valid position within the array.
55 34
345Insufficient Verification of Data Authenticity
The software does not sufficiently verify the origin or authenticity of data, in a way that causes it to accept invalid data.
51 15
79 Cross-site Scripting
The software does not neutralize or incorrectly neutralizes user-controllable input before it is placed in output that is used as a web page that is served to other users.
10,863 9
Web Protection 14,800 917 6.2%
22 Path Traversal
The software uses external input to construct a pathname that is intended to identify a file or directory that is located underneath a restricted parent directory, but the software does not properly neutralize special elements within the pathname that can cause the pathname to resolve to a location that is outside of the restricted directory.
2,810 723
59 Link Following
The software attempts to access a file based on the filename, but it does not properly prevent that filename from identifying a link or shortcut that resolves to an unintended resource.
506 62
611Information Leak Through XML External Entity File Disclosure
The software processes an XML document that can contain XML entities with URIs that resolve to documents outside of the intended sphere of control, causing the product to embed incorrect documents into its output.
345 113
601 Open RedirectA web application accepts a user-controlled input that specifies a link to an external site, and uses that link in a Redirect. This simplifies phishing attacks.
222 2
93 CRLF Injection
The software uses CRLF (carriage return line feeds) as a special element, e.g. to separate lines or records, but it does not neutralize or incorrectly neutralizes CRLF sequences from inputs.
29 2
613 Insufficient Session Expiration
Insufficient Session Expiration is when a web site permits an attacker to reuse old session credentials or session IDs for authorization.
The software receives data from an upstream component, but does not neutralize or incorrectly neutralizes CR and LF characters before the data is included in outgoing HTTP headers.
20 1
Web Protection 37 5 13.5%
444HTTP Request Smuggling
When malformed or abnormal HTTP requests are interpreted by one or more entities in the data flow between the user and the web server, such as a proxy or firewall, they can be interpreted inconsistently, allowing the attacker to "smuggle" a request to one device without the other device being aware of it.
17 4
284 Access Control (Authorization) Issues
The software does not restrict or incorrectly restricts access to a resource from an unauthorized actor. 3,503 726
Compartmentalization 3,505 727 20.7%216 Containment Errors This tries to cover various problems in which
improper data are included within a "container." 2 1
285Improper Access Control (Authorization)
The software does not perform or incorrectly performs an authorization check when an actor attempts to access a resource or perform an action.
187 30
Access Control 296 75 25.3%306 Missing Authentication
for Critical Function
The software does not perform any authentication for functionality that requires a provable user identity or consumes a significant amount of resources.
53 35
346 Origin Validation Error The software does not properly verify that the source of data or communication is valid. 29 3
693Protection Mechanism Failure
The product does not use or incorrectly uses a protection mechanism that provides sufficient defense against directed attacks against the product.
27 7
200 Information Exposure
An information exposure is the intentional or unintentional disclosure of information to an actor that is not explicitly authorized to have access to that information.
6,071 196 Privacy 6,071 196 3.2%
326Inadequate Encryption Strength
The software stores or transmits sensitive data using an encryption scheme that is theoretically sound, but is not strong enough for the level of protection required.
92 10
Cryptographic issues 202 31 15.3%347
Improper Verification of Cryptographic Signature
The software does not verify, or incorrectly verifies, the cryptographic signature for data. 56 12
327Use of a Broken or Risky Cryptographic Algorithm
The use of a broken or risky cryptographic algorithm is an unnecessary risk that may result in the exposure of sensitive information.
34 8
338Use of Cryptographically Weak PRNG
The product uses a Pseudo-Random Number Generator (PRNG) in a security context, but the PRNG's algorithm is not cryptographically strong.
When an actor claims to have a given identity, the software does not prove or insufficiently proves that the claim is correct.
1,535 737
Architectural, Configuration, or User error
3,262 1,236 37.9%
362 Race Condition
The program contains a code sequence that can run concurrently with other code, and the code sequence requires temporary, exclusive access to a shared resource, but a timing window exists in which the shared resource can be modified by another code sequence that is operating concurrently.
576 120
400 Resource Exhaustion
The software does not properly control the allocation and maintenance of a limited resource thereby enabling an actor to influence the amount of resources consumed, eventually leading to the exhaustion of available resources.
394 113
798 Use of Hard-coded Credentials
The software contains hard-coded credentials, such as a password or cryptographic key, which it uses for its own inbound authentication, outbound communication to external components, or encryption of internal data.
263 196
704 Incorrect Type Conversion or Cast
The software does not correctly convert an object, resource, or structure from one type to a different type.
159 38
369 Divide By Zero The product divides a value by zero. 97 4
532Information Leak Through Log Files
Information written to log files can be of a sensitive nature and give valuable guidance to an attacker or expose sensitive user information.
89 0
640
Weak Password Recovery Mechanism for Forgotten Password
The software contains a mechanism for users to recover or change their passwords without knowing the original password, but the mechanism is weak. 49 6
427Uncontrolled Search Path Element
The product uses a fixed or controlled search path to find resources, but one or more locations in that path can be under the control of unintended actors.
40 13
754
Improper Check for Unusual or Exceptional Conditions
The software does not check or incorrectly checks for unusual or exceptional conditions that are not expected to occur frequently during day to day operation of the software.
24 1
358Improperly Implemented Security Check for Standard
The software does not implement or incorrectly implements one or more security-relevant checks as specified by the design of a standardized algorithm, protocol, or technique.
18 4
330 Use of Insufficiently Random Values
When software generates predictable values in a context requiring unpredictability, it may be possible for an attacker to guess the next value that will be generated, and use this guess to impersonate another user or access sensitive information.
16 3
398Indicator of Poor Code Quality
Poor code quality leads to unpredictable behavior. From a user's perspective that often manifests itself as poor usability. For an adversary it provides an opportunity to stress the system in unexpected ways.
1 1
485Insufficient Encapsulation
Encapsulation is about drawing strong boundaries. In a web browser that might mean ensuring that your mobile code cannot be abused by other mobile code.
Conclusion An analytical framework to measure the effectiveness of a cybersecurity defense system was described. This framework is based on MITRE’s Common Vulnerabilities and Exposures (CVE), the Common Vulnerability Scoring System (CVSS), and the Common Weakness Enumeration (CWE) list. We showed an approach to map CWEs to large numbers of CVEs and then to associate a cybersecurity defense mechanism to groupings of related CWEs. A CVSS score on each CVE allows us to highlight the defense mechanism’s abilities against the most damaging attacks. This approach allows us to postulate that a particular defense mechanism, if effective, can block a fairly precise number of vulnerabilities—including how many severe vulnerabilities—and thereby measure overall effectiveness of a cybersecurity technology. This level of rigorous analysis has rarely been done for any cybersecurity system. The cybersecurity ecosystem has at least 3,000 vendors in it and a framework like the one we propose could improve the entire ecosystem with objective measures of each system’s effectiveness.
