Abstract Periodicity in key processes related to softwarevulnerabilities need to be taken into account for assessingsecurity at a given time. Here, we examine the actual multi-year field datasets for someof themost used software systems(operating systems and Web-related software) for poten-tial annual variations in vulnerability discovery processes.We also examine weekly periodicity in the patching andexploitation of the vulnerabilities. Accurate projections ofthe vulnerability discovery process are required to optimallyallocate the effort needed to develop patches for handling dis-covered vulnerabilities. A time series analysis that combinesthe periodic pattern and longer-term trends allows the devel-opers to predict future needs more accurately. We analyzeeighteen datasets of software systems for annual seasonal-ity in their vulnerability discovery processes. This analysisshows that there are indeed repetitive annual patterns. Next,some of the datasets from a large number of major organi-zations that record the result of daily scans are examinedfor potential weekly periodicity and its statistical signifi-cance. The results show a 7-day periodicity in the presenceof unpatched vulnerabilities, as well as in the exploitationpattern. The seasonal index approach is used to examine thestatistical significance of the observed periodicity. The auto-correlation function is used to identify the exact periodicity.The results show that periodicity needs to be considered for
1 Department of Computer Engineering, Kyungil University,Gyeongsan, Korea
2 Computer Science Department, Colorado State University,Fort Collins, CO 80523, USA
optimal resource allocations and for evaluation of securityrisks.
Keywords Vulnerability · Laws of vulnerabilities ·Seasonality · Periodicity · Operating system
1 Introduction
As software systems become larger and more complex, thenumber of defects they contain has become a major consid-eration. A fraction of all defects are security related and aretermed vulnerabilities. Vulnerabilities are major concerns inall systems whether they are open source or commercial. Afamous quotation is attributed to Lord Kelvin: “If you can-not measure it, you cannot improve it.” Both developers andusers need to be able to assess the security level of softwaresystems in order to improve them. The security of a softwaresystem is impacted by the presence of unpatched vulnera-bilities which are discovered from time to time and can bepotentially exploited.
A number of studies related to the evaluation of computersecurity have been carried out. However, most of them havebeen qualitative. Quantitativemethods arewell established infields such as performance assessment, metric measurement,functional evaluation, or statistical modeling, but these arerelatively new in the area of security evaluation. Quantitativestatistical analysis methods permit developers to estimatefuture trends much more accurately because they provideactual data-driven estimation approaches [6]. Quantitativemethods will make it possible for end-users to objectivelyassess the risk posed by vulnerabilities in software sys-tems and their potential breaches. Researchers have begunto explore some of the quantitative characteristics of vulner-abilities as the required datasets are becoming sufficiently
large to be analyzed [20,31]. Nevertheless, quantitative riskanalysis in information security systems is still in its infancy.
Security vulnerabilities in major applications that havebeen discovered and disclosed but not remedied representgreat risk of both organizations and individuals. They includemajor classes of software systems such as operating systems(OSes), Web servers and browsers. The OSes form the com-plex foundation for all computing systems,whileWeb serversand browsers provide the connectivity among modern com-puting systems due to the Internet. Unfortunately, each year,a large number of vulnerabilities are detected in OSes andWeb-related software systems that represent a major secu-rity risk [33]. If we could predict the expected vulnerabilitydiscovery pattern and the attributes of the vulnerabilities dis-covered, the needed resources could be allocated at the righttime for corrective measures, which would greatly reducesecurity risks. Such methods could also be utilized by end-users to assess risks and be prepared to take remedial actionwhen potential security breaches occur.
Figure 1 represents the box plots for the number of newlydiscovered vulnerabilities in the three types of software sys-tems grouped for each month. Table 1 shows the numberof known vulnerabilities for each software system, togetherwith the examined periods. Here, software systems are cat-
egorized into Windows OSes (Windows NT, Windows 95,Windows 98, Windows 2000, Windows XP, and Windows7), non-Windows OSes (iPhone OS, MAC OSX, Red HatLinux Enterprise, AIX, Android, and ChromeOS), andWeb-related software systems [Apache http server, IIS, InternetExplorer, Firefox, Safari, and Java (JRE)]. Figure 2 showsthe number of vulnerabilities found each calendar month.All datasets were extracted from the National Vulnerabil-ity Database (NVD; http://nvd.nist.gov) on August 2014. InTable 1, periods are determined by the time when the firstand the last vulnerabilities were reported in each softwaresystem in the NVD datasets.
In Fig. 1, it is observed visually that all the three softwaregroups peak in certain months. For the Windows OSes, astrong year-end peak and a somewhat weaker mid-year peakare observed. On the other hand, the Web-related softwaresystems have relatively fewer vulnerabilities reported year-end, but a strong mid-year peak. The non-Windows OSeshave strong peaks between mid-year and year-end, Marchand September. We will analyze these patterns later in thispaper using statistical methods to see whether the differ-ences are statistically significant. A consistent variation inannual box plots suggests a seasonal pattern that should betaken into account inmakingmore accurate predictions about
Periodicity in software vulnerability discovery, patching and exploitation 675
Windows NTVu
lner
abili
ties
0
5
10
15
20
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
Windows 95
Vuln
erab
ilitie
s
0
1
2
3
4
5
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
Windows 98
Vuln
erab
ilitie
s
0
2
4
6
8
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
Windows 2000
Vuln
erab
ilitie
s
0
5
10
15
20
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
Windows XP
Vuln
erab
ilitie
s0
10
20
30
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Windows 7
Vuln
erab
ilitie
s
05
101520253035
2009
2010
2011
2012
2013
2014
AIX
Vuln
erab
ilitie
s
0
2
4
6
8
10
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
RHEL
Vuln
erab
ilitie
s
0
5
10
15
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
iPhone OS
Vuln
erab
ilitie
s
0102030405060
2007
2008
2009
2010
2011
2012
2013
2014
OSX
Vuln
erab
ilitie
s
0
10
20
30
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Android
Vuln
erab
ilitie
s
0.00.51.01.52.02.53.0
2009
2010
2011
2012
2013
2014
Chrome OS
Vuln
erab
ilitie
s
0
5
10
15
2010
2011
2012
2013
2014
Apache HTTP Server
Vuln
erab
ilitie
s
0
2
4
6
8
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
IIS
Vuln
erab
ilitie
s
0
2
4
6
8
10
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Internet Explorer
Vuln
erab
ilitie
s
01020304050
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Firefox
Vuln
erab
ilitie
s
05
101520253035
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Safari Web Browser
Vuln
erab
ilitie
s
0
20
40
60
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
JRE
Vuln
erab
ilitie
s
0
10
20
30
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
Fig. 2 Run chart for the number of monthly vulnerabilities, with calendar time
the number of vulnerabilities expected to be discovered in afuture period. We examine the question of the existence ofannual and weekly patterns and their significance for activi-ties (discovery, patching and exploitation) related to software
vulnerabilities using actual datasets in a quantitative manner[15,16].
The paper is organized as follows. Section 2 discussesseasonality-related research in other fields. Section 3 presents
123
676 H. Joh, Y. K. Malaiya
the background research related to security vulnerabilities.Section 4 concisely describes the statistical methods used inthis paper. The next section presents an analysis of potentialseasonality in the eighteen software systems. In Sect. 6, wewill examine the 7-day periodic behavior in the presence ofunpatched vulnerabilities, as well as the exploitation pattern.Section 7 discusses possible factors affecting the periodicbehavior in the vulnerability-related activities. Conclusionswill be given in Sect. 8.
2 Related works on seasonality
Periodic behavior such as seasonal or weekend effects arewell-established research areas in other disciplines such asthe stock market [12], high-performance computing systems[34], epidemiology [30], power transmission [32], marinebiology [21], and birth defects [7]. This section reviews thembriefly.
Stocks tend to have relatively higher returns for certainspecific calendarmonths. The higher return betweenNovem-ber and April is termed the Halloween Effect (http://ssrn.com/abstract=901088). Heston and Sadka [12] have identi-fied a repetitive and distinctive pattern with lags of 12, 24and 36 months in the data of [14] caused by the persistentseasonal effect in stock returns. The results could potentiallybe useful for developing seasonal stock market strategies.Tran and Reed [34] have examined the software applicationperformance, using the fact that application I/O patterns arebursty in scientific codes due either to periodic checkpointsor nested loops, and such bursty patterns could cause anoverflow in system resource usage. The authors have triedto predict the resulting I/O request patterns using time seriesmodels. This kind of access pattern forecast could be used tomake pre-fetch decisions during application execution time.
In epidemiology, Rios et al. [30] attempted to determinewhether pulmonary tuberculosis has an annual seasonal pat-tern by using the autoregressive integrated moving average(ARIMA) time series model using autocorrelation function(ACF) and partial ACF. The seasonal trend for the diseasecould be caused by a rise in indoor activities inwinter, therebyincreasing the risk of exposure of healthy persons to bacillifrom other infected persons. Another reason could be the factthat infections of viral etiology are more frequent in wintercausing immunological deficiency. The model developed bythe authors can also determine whether the incidence of thedisease is greater than that forecast by the model so that themodel could be used to assess the quality of the existing pre-ventative measures.
For power transmission in utilities, Salehian [32] hasattempted to model thermal rating patterns that are influ-enced by weather, using the ARIMA time series model forforecasting. Forecasting the thermal capabilities of the line
would allowmeeting contractual power delivery obligations.In marine biology, Maes et al. [21] have tried to determinehow elements affect the abundance of fish, which are exposedto great environmental variabilities in the form of dissolvedoxygen, temperature, water quality, salinity, prey, etc. Theresults show that the life cycles of marine organisms haveclear seasonal patterns in growth, reproduction, and abun-dance.
Carrion-Baralt et al. [7] have investigated whether thebirth of a schizophrenic infant, which occursmore frequentlyduring winter, is actually due to severe winter temperatures,as was thought previously. In contrast to previous studies, theauthors conducted their work on a tropical island having nosevere winter weather and drew the conclusion that extremetemperatures are not a sufficient explanation for this phenom-enon. Lastly, in [35], the authors tried to find any seasonalpatterns of tobacco smoking behavior using online searchinformation, Google Trends. They first collected the searchquery of “smoking” from the search engine in six nations.Then, periodogrammethodwas applied to determinewhetherthe seasonality of smoking exists in the data. They also calcu-lated the pairwise cross-correlations among the six datasets.The result shows that the similar weather condition on thesame hemisphere leads to the similar smoking-related searchbehavior. Since seasonal periodicity analysis is a well-knownstatistical approach in the repertoire of many researchers andanalysts in several disciplines as discussed above, somewell-developed analytical techniques exist that can be applied tothe information security field.
3 Recent research in security vulnerabilities
A software vulnerability can be defined as a defect or weak-ness in the security system which might be exploited bya malicious user causing loss or harm [26]. Thus far, onlylimited work has been done to characterize security vulner-abilities in a quantitative manner. There are a few standardscommonly used by researchers to render the vulnerabili-ties foundmeasurable. These include CommonVulnerabilityExposures (CVE; http://cve.mitre.org), Common WeaknessEnumeration (CWE; http://cwe.mitre.org), and CommonVulnerability Scoring System (CVSS; http://www.first.org).
CVE is a publicly available, free-for-use list or dictionaryof standardized identifiers concerning common computervulnerabilities and exposures. CVE’s common identifiersenable data exchange among security products and providea baseline index point for evaluating coverage of tools andservices.
CWE is a list of softwareweakness types that is intended tobe a complete dictionary of software weaknesses. A uniquenumber is assigned to each weakness type which allows aclear description, and hence better management of software
Periodicity in software vulnerability discovery, patching and exploitation 677
weaknesses related to architecture and design.CWEprovidesa very detailed classification using programming, design, andarchitectural errors that led to vulnerabilities.
CVSS, the Common Vulnerability Scoring System, hasbeen adopted by many vendors since its launch in 2004. Thissystem, whose initial design was finalized in 2007, is now inits secondversion.CVSS includes threemetric groups,whichare termedbase, temporal, and environmental.While the basemetric group is required for the final CVSS score, the othertwo groups are optional. The score is in the range of [0.0,10.0]. Scores close to 0.0 are relatively benign vulnerabili-ties,whereas scores close to 10.0mean that the vulnerabilitiesaremore likely to encounter exploitation causing serious con-sequences. On May 2012, the CVSS Special Interest Groupstarted to develop the third version, and on June 2014, pre-view of version 3 has been available.
Some vulnerability discovery models (VDMs) that modelthe vulnerability discovery process have been proposedrecently. These include Andersons thermodynamic model[3],Alhazmi-MalaiyaLogistic (AML)model [1], andRescor-las linear/exponential models [28]. Some classical softwarereliability growth models have also been used as [25].
Recently, the impact of evolution caused by successivesoftware versions on the vulnerability discovery process havebeen examined [8,18]. In their study,Kimet al. [18] have con-sidered the impact of sharing code across successive versions.They have proposed an enhanced version of the AMLmodelby taking into consideration the superposition [10] of the dis-covery processes for multiple version software systems andverified their model by examining two open source softwaresystems. Chen et al. [8] have presented a multi-cycle vulner-ability discovery model incorporating a sinusoidal behavior.The multi-cycle model attempts to model the relationshipbetween the number of vulnerabilities and their release timeand is compared with other VDMs using datasets from eightWindows OSes.
Ozment [24] has proposed a standard set of terms for mea-suring characteristics of vulnerabilities and their discoveryprocesses. Alhazmi and Malaiya [1] have compared severalVDMs by fitting the data for major operating systems andhave shown that AML fits better than other models in mostcases. However, since AML is a symmetrical model, it maynot perform well with asymmetric discovery patterns. Johand Malaiya [17] have examined some S-shaped distribu-tions which can model an asymmetrical behavior. Condon etal. [9] have used software security incident datasets to com-pare forecasts using time series models and using softwarereliability growth models. There are a large number of vul-nerability databases and security advisories on theWeb. As aresult, selecting the appropriate data source for quantitativeanalysis is not an easy task. Massacci and Nguyen [22] haveexamined the accuracy of sampling by the researchers fortheir quantitative vulnerability analyses. Many vulnerability
databases provide de facto standard information such as CVEidentifier numbers and CVSS. The best known vulnerabilitydatabases include NVD, Open Source Vulnerability Data-base (OSVDB) (http://osvdb.org), US-CERT (http://www.kb.cert.org/vuls), IBM X-Force (http://xforce.iss.net) andSecunia (http://secunia.com).
In this paper, we have used the NVD database for long-term annual periodicity analysis, because it provides themostextensive datasets, and the data is collected and organized byusing specific standards. Since the NVD project is sponsoredby the USDepartment of Homeland Security andmaintainedby National Institute of Standard and Technology, it can beconsidered a standard source. Moreover, NVD uses CVEidentifiers, so that any updates to CVE immediately appearon NVD. Also, in Sect. 6, which addresses shorter term peri-odicity, an extensive amount of data gathered globally byQualys [27] using automated scans, such as exploitation pat-terns and the presence of unpatched vulnerabilities, is used toexamine potential weekly periodicity in vulnerability-relatedactivities.
4 Statistical methods in seasonality analysis
The seasonal index is the most commonmeasure used in sea-sonality analysis. A seasonal index states to what extent theaverage for a particular period tends to be above (or below)the expected value. The monthly seasonal index values aregiven by
si = did
(1)
where, si is the seasonal index for the i th month, di isthe mean value of the i th month, and d is a grand aver-age (http://home.ubalt.edu/ntsbarsh/business-stat/stat-data/forecast.htm). Thus, for example, a monthly seasonal indexof 1.25 indicates that the expected value for that month is25 greater than 1/12 of the overall annual average wherethe expected value is 1. To determine whether the seasonalindices are statistically significant, the Chi-square test is alsoapplied. To evaluate the significance of the non-uniformityof the distribution in calculated indices, we tested the grandtotal of each month against the expected value (total vulnera-bilities divided by 12). The Chi-square statistic is calculatedas
χ2s =
n∑
i=1
(oi − ei )2
ei(2)
where oi and ei are the observed and expected values atthe i th time point, respectively. For the test to be accept-able, the Chi-square statistic value (χ2
the corresponding Chi-square critical value (χ2c ) with the
given alpha level and degrees of freedom. The p value pro-duced by the test represents the probability that a value ofthe statistic, at least as high as the value calculated in Eq. 2could have occurred by chance. In this paper, to confirm thenon-uniformity of the distribution in calculated indices, pvalues need to be smaller than 0.05 since we are using thealpha level of 0.05. Goonatilake et al. [11] describe how toapply Chi-square test in software security. To pinpoint whichmonths seasonal index is statistically greater or less thanothers, analysis of variance (ANOVA) with Fisher’s leastsignificant difference (LSD) tests [23] is conducted on thecalculated seasonal indices. When a result from the ANOVAtest is significant, it indicates there is at least one group thatdiffers from the other groups. However, an ANOVA analysiscan only determine whether the indices among the 12monthsare statistically the same or not, it cannot determine whichmonths index is higher or lower than others.
LSD is a test for comparing average values from treat-ment groups after theANOVAnull hypothesis of equalmeansamong the groups has been rejected. Once the ANOVA nullhypothesis is rejected, researchers need to check pairwisecomparisons of means from each group.
When there are twelve seasonal indices for each of themonths and μi is the mean seasonal index value for monthi , then LSD is used to test the null hypothesis that μi = μ j
where 1 ≤ {i and j} ≤ 12, i and j are integer, and i �= j .When the calculated LSDi, j is |μi −μ j | ≥ LSDi, j , then thenull hypothesis of μi = μ j will be rejected, confirming thatthere is a statistically significant difference between the twogroups. LSDi, j can be obtained by [23]:
LSDi, j = tα/2,d f
√
MS
(1
ni+ 1
n j
)(3)
where d f is degrees of freedom, MS is mean square valuefrom the ANOVA test, and ni is number of observations ingroup i . tα/2,d f value can be obtained from the Student’st-distribution table.
The other approach to characterize periodicity is to usethe autocorrelation function (ACF). ACF analysis gives usspecific relationship information in terms of the related timeunits such as month or day. With time series values ofzb, zb+1, . . . , zn , the ACF at time lag k, denoted by rk , is[5]:
rk =∑n−k
t=b (zt− z̄)(zt+k− z̄)∑nt=b(zt− z̄)2
, where z̄=∑n
t=b ztn−b+1
(4)
ACF measures the linear relationship between time seriesobservations separated by a lag of k time units.When anACFvalue is located outside chosen upper or lower confidence
intervals, there is a statistically significant periodic relation-ship associated with that time lag. An event occurring at timet + k(k > 0) is said to lag behind an event occurring at timet , the extent of the lag being k. In this paper, the seasonalindex analysis with the Chi-square test, ANOVA test, LSDtest, and the ACF analysis is used to establish the annual andweekly periodic behaviors discussed below.
5 Annual seasonal variations in vulnerabilitydiscovery processes
Figure 2 gives plots for the number of vulnerabilities dis-closed each month for the eighteen software products. Itcan be seen that, for each software group, certain monthstend to have more vulnerabilities then others. For exam-ple, in Windows group, year-end has peak values, whilemid-year (summer) peaks are shown in theWeb-related soft-ware group, suggesting the possibility of seasonality. Inthis section, we examine the significance of this observedannual seasonality. We group the software systems into threecategories for convenience—Windows OSes, non-WindowsOSes, and Web-related systems. We will examine the nullhypothesis H0 that the seasonal indices for the 12monthsare not all significantly different from each other. The samemethods will be used in the next section for checking weeklyperiodic behaviors in the exploitation pattern and prevalenceof unpatched vulnerabilities.
5.1 Seasonal index analysis
A time series data is not uniformly distributed, and peri-odic patterns are present in a dataset when certain monthshave significantly more incidents of reported vulnerabilitiesthan other months. Table 2 shows seasonal indices for the12months from each system, as described in Sect. 4 above.In Fig. 3, for Windows OSes, seasonal index values for mid-year (summer) and year-end (winter) tend to have highervalues than index value of 1.0, which represents the expectedvalue. Consequently, the months between the two peaks tendto have the expected seasonal index value less than 1.0. Thevalues for March and September are lower than others.
For the non-Windows OSes, there is no clear consis-tent mid-year seasonal patterns, but March, September, andDecember seem to have higher values. In contrast, clearconsistent mid-year seasonal patterns are found among theindices fromWeb-related software systems. As for the Web-related software systems, IIS and IE datasets display a patternsimilar to that of Windows OSes, which are the native plat-forms for the two Web-related systems; the mid-year andyear-end periods tend to have more vulnerabilities than othertimes for the four systems. The mid-year peak may explainthe higher third quarter advisories for Microsoft products
123
Periodicity in software vulnerability discovery, patching and exploitation 679
Table 2 Vulnerability discovery process seasonal indices
p value 1.20641E−35 8.7302E−35 3.8201E−245 4.4737E−239 7.6726E−213 4.6663E−161
JAN
FEB
MA
R
AP
R
MAY
JUN
JUL
AUG
SE
P
OC
T
NO
V
DE
C
Sea
sona
l Ind
ex
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5 Windows NTWindows 95Windows 98Windows 2000Windows XPWindows 7
JAN
FEB
MA
R
AP
R
MAY
JUN
JUL
AUG
SE
P
OC
T
NO
V
DE
C
Sea
sona
l Ind
ex
0
1
2
3
4iPhone OSMAC OS XRHELAIXAndroidChrome OS
JAN
FEB
MA
R
AP
R
MAY
JUN
JUL
AUG
SE
P
OC
T
NO
V
DE
C
Sea
sona
l Ind
ex
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5 ApacheIISIEFirefoxSafariJRE
(a) (b) (c)
Fig. 3 Seasonal indices of Windows OSes, non-Windows OSes and Web-related software systems. Individual values are in Table 2. a WindowsOSes. b non-Windows OSes. cWeb-related software
Table 3 ANOVA table for seasonal indices from Windows OSes
Windows SS df MS F p value Fcrit
Between groups 9.314879458 11 0.846807223 4.655838669 4.16254E−05 1.952211939
Within groups 10.91284235 60 0.181880706
Total 20.22772181 71
[13]. To evaluate the significance of the non-uniformity of thedistribution among the seasonal indices, we applied the Chi-square test to the grand total of each month against the meanvalue (total vulnerabilities divided by 12). In this paper, thelevel of alpha chosen is 0.05. Hence, when the p value of theChi-square test is below 0.05, the null hypothesis that there isno seasonality in the dataset will be rejected. In Table 2, wesee that the systems yield extremely small p values, therebyproviding strong evidence of the non-uniformity of the dis-tributions of vulnerability discovery rates contrary to the nullhypothesis.
To determine whichmonths actually have statistically sig-nificant higher or lower indices, the ANOVA test along withthe Fisher’s LSD test are conducted on the mean index val-ues from each month grouped by software categories. Asobserved previously, since the ANOVA test can only tellwhether the mean index values among the 12months arethe same or not, Fisher’s LSD test is also conducted afterconfirming the unequal performance via the ANOVA test.
Tables 3, 4 and 5 show ANOVA tables for each softwaregroup. Here, the alpha level is 0.05 for the F-test. To be
statistically significant, the F value needs to be greater thanthe corresponding F critical with sufficiently small p values(less than 0.05). In Tables 3 and 5, F values greater than the Fcritical value, confirm that not all themonths have equalmeanvalueswhereas ANOVA test for the non-WindowsOSes doesnot produce statistically significant value. In addition, the Fvalue from theWindows OSes is larger than the one from theWeb-related software, implying that seasonal fluctuations inWindows OSes are more dynamic.
Tables 6, 7 and 8 give the absolute differences among theseasonal indices and the significance of pairwise compar-isons, with italicized cells representing statistically signifi-cant differences. To be a italicized cell, differences betweentwo compared mean values need to be greater than the cor-responding calculated LSD value. The LSD values for eachtable are LSDWindows = 0.4924, LSDnon-Windows = 0.7901and LSDWeb = 0.6811, respectively.
For the seasonal index values, Table 6 confirms that, inWindowsOSes,December is greater than all the othermonthsand September is less than January, February, April, May,June, and August. Table 7 confirms that, in non-Windows
123
Periodicity in software vulnerability discovery, patching and exploitation 681
Table 4 ANOVA table for seasonal indices from non-Windows OSes
Non-windows SS df MS F p value Fcrit
Between groups 6.761092828 11 0.614644803 1.312574578 0.240035623 1.952211939
Within groups 28.09645163 60 0.468274194
Total 34.85754446 71
Table 5 ANOVA table for seasonal indices from Web-related software
Web SS df MS F p value Fcrit
Between groups 7.988696315 11 0.72624512 2.087102541 0.035136421 1.952211939
Within groups 20.8780864 60 0.347968107
Total 28.86678272 71
Table 6 LSD test for Windows seasonal index; LSDWindows = 0.4924
Month Mean JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
MAY 0.7175 0 0.9776 0.7522 0.0593 0.0440 0.4428 0.0562 0.5986
JUN 1.6950 0 0.2254 0.9182 1.0216 0.5347 0.9214 0.3790
JUL 1.4697 0 0.6929 0.7962 0.3094 0.6960 0.1536
AUG 0.7768 0 0.1033 0.3835 0.0032 0.5392
SEP 0.6735 0 0.4868 0.1002 0.6426
OCT 1.1603 0 0.3867 0.1557
NOV 0.7736 0 0.5424
DEC 1.3160 0
OSes, March is greater than all the other months, except Jan-uary, September, and December. Table 8 shows that Juneand July are greater than January, March, August, Septem-ber, and November. Plus, June is also greater than April andMay. Also, the table says December is greater than January.
5.2 Autocorrelation function analysis
The autocorrelation function (ACF) in time series analysisis calculated by computing the correlation between a vari-able value and the successive values of the same variablewith some time lags. Thus, ACF measures the linear rela-tionship between time series observations separated by a lagof k time units [5,30]. When an ACF value is located outsideof defined confidence intervals at a lag k, there is a significantrelationship associated with that time lag.
Tables 9, 10 and 11 show the ACF values with 95% confi-dence intervals for the three software groups, respectively. Intables, the bold font indicates a value outside of confidenceintervals, and superscripts represent time lags in month rang-ing from 0 to 23. For the Windows OSes, since the year-endperiod tends to have the majority of higher seasonal indicesand September has particularly low values, we expect thatlags corresponding to about 3months or around its multipleswould have their corresponding ACF values outside the con-fidence interval. In Table 9, we observe that for WindowsNT, the lags for 0, 1, 2, 5, 6, 7, and 11 months are outside ofconfidence interval; in other words, there are strong autocor-relations with the lags that are multiples of around three oreleven, confirming a seasonal pattern.
For Windows 95, lags for 0, 3, 6, and 7 months; for Win-dows 2000, 0, 1, 2, 3, 4, 5 and 6 months; for Windows XP, 0,2, 3, 4, 5, 6, 10, 14, 16, 18, and 22 months are significantlydifferent from the zero of ACF which confirms a seasonalpattern. For Windows 98 and Windows 7, only the lag of 0
is outside the confidence interval. As we expected, roughly a3 and 11month periodicity is observed in most of the cases.The same approach was earlier applied in [30,34] to demon-strate seasonality in datasets pertaining to other researchareas.
In Table 10, for iPhoneOS, lags for 0, 6, and 18 are outsidethe confidence interval. For OSX, 0, 3, 4, 7, 8, 10, 11, 12, 15,17, 18, 20, 21, 22, and 23 months; for RHEL, 0, 1, 2, 3, 4, 5,6, 7, 8, and 9; for AIX, 0, 1, 2, 3, 4, 5, 6, 7, 14, and 21; forAndroid, lags of 0 and 16 are outside the confidence interval,while Chrome OS has only one lag which is 0.
In Table 11, for the ApacheWeb server, lags of 0 and 7 areoutside the confidence interval. For the IIS, lags of 0, 1, 4, 5,6, 7, 8, 10, 11, 14, 18, 21, and 23 are outside the confidenceinterval. For Internet Explorer, lags of 0, 1, 2, 3, 4, and 8 arelocated outside the boundary. For Firefox, lags of 0, 2, 3, 11,and 14 are located outside the confidence interval. For theSafari web browser, only lags number 0 and 12 are outside.Interestingly, all the significant legs for JRE are multiples offour, 0, 4, 8, 12, 16, 20.
6 Seven-day periodicity in the institutionalvulnerability scans
In this section, another periodic behavior related to softwarevulnerabilities with a much shorter periodicity is examined.We examine the available data for the presence of weeklyperiodic trends. Periodic scanning is a major part of corpo-rate security strategy. Some security service vendors such asQualys [27] collect large amount of data which is quite valu-able because it comes from real systems in major industrialorganizations. We have mined one such data collection toexamine periodicity in the presence of unpatched vulnerabil-ities and exploitations in the case of a worm.
123
Periodicity in software vulnerability discovery, patching and exploitation 683
Table9
IndividualACFvalues
forWindowsOSes
WindowsNT;
95%
confi
denceinterval
=(−
0.1414482,
0.1414482)
100.1461
0.1522
0.07
30.0844
0.2275
0.1786
0.1487
0.0178
0.0019
0.0411
00.2251
1
0.0151
20.1151
30.08
140.0361
5−0
.001
16−0
.019
170.0451
80.0161
90.0112
0−0
.008
21−0
.032
2−0
.042
23
Windows95;9
5%
confi
denceinterval
=(−
0.1569227,
0.1569227)
100.1151
0.1462
0.1883
0.1124
0.1435
0.2286
0.1987
0.0688
0.1429
0.1521
00.0761
1
0.0661
20.1731
30.0921
40.0731
50.04
160.0931
7−0
.059
180.0481
90.0362
00.0462
10.0372
20.0152
3
Windows98;9
5%
confi
denceinterval
=(−
0.170593,0
.170593)
100.1661
0.1542
0.0613
−0.006
40.1695
0.1376
0.13
70.07
80.1139
0.0561
00.0631
1
0.1051
2−0
.004
13−0
.025
14−0
.039
15−0
.144
160.02
17−0
.041
8−0
.038
190.0612
0−0
.103
210.0292
2−0
.054
23
Windows2000;9
5%
confi
denceinterval
=(−
0.1414482,
0.1414482)
100.1631
0.2942
0.1623
0.1654
0.3115
0.2066
0.1047
0.1168
0.0959
0.1051
00.0931
1
0.0911
2−0
.003
130.0471
4−0
.072
150.0231
6−0
.006
170.0961
8−0
.138
190.0022
0−0
.118
21−0
.094
22−0
.017
23
WindowsXP;
95%
confi
denceinterval
=(−
0.1460871,
0.1460871)
100.0651
0.2562
0.16
30.1734
0.1655
0.2816
0.1167
0.2178
0.0489
0.1491
00.0621
1
0.14
120.0811
30.1781
4−0
.053
150.1581
6−0
.068
170.1791
80.0321
90.0812
0−0
.022
10.2022
2−0
.021
23
Windows7;
95%
confi
denceinterval
=(−
0.230984,0
.230984)
100.0121
0.1182
0.14
30.1544
−0.042
50.0956
0.0437
0.2098
−0.095
90.0111
0−0
.058
11
−0.062
12−0
.112
130.0481
4−0
.069
15−0
.012
16−0
.139
17−0
.037
18−0
.002
19−0
.096
20−0
.112
10.2212
2−0
.085
23
123
684 H. Joh, Y. K. Malaiya
Table10
IndividualACFvalues
fornon-WindowsOSes
iPhone
OS;
95%
confi
denceinterval
=(−
0.2138496,
0.2138496)
10−0
.078
1−0
.068
2−0
.043
30.1154
0.0625
0.28
6−0
.057
70.0568
−0.047
90.2031
0−0
.032
11
0.19
12−0
.058
130.0681
4−0
.019
15−0
.018
16−0
.089
170.46
18−0
.029
19−0
.081
20−0
.022
10.0412
20.0062
3
OSX
;95%
confi
denceinterval
=(−
0.1372249,
0.1372249)
10−0
.006
1−0
.017
20.2743
0.2874
0.1075
0.0996
0.2847
0.1568
0.1019
0.15
100.21
11
0.2031
20.0941
30.13
140.2351
50.0591
60.1661
70.2361
80.1351
90.1672
00.1392
10.0552
20.1532
3
RHEL;9
5%
confi
denceinterval
=(−
0.1333587,
0.1333587)
100.6171
0.41
20.4313
0.4774
0.2895
0.1436
0.2137
0.2878
0.1439
0.0161
00.0681
1
0.0631
20.0171
3−0
.009
140.0231
50.0481
6−0
.004
17−0
.043
18−0
.044
19−0
.042
20−0
.013
21−0
.025
22−0
.029
23
AIX
;95%
confi
denceinterval
=(−
0.1206274,
0.1206274)
100.1341
0.1942
0.1723
0.1724
0.14
50.1866
0.2467
0.0688
0.1169
0.1311
00.0661
1
0.1351
20.0471
30.1511
40.0211
50.1031
60.0111
70.0781
80.0991
90.0012
00.1822
1−0
.057
220.0372
3
Android;9
5%
confi
denceinterval
=(−
0.2530303,
0.2530303)
100.0621
0.0792
0.1863
−0.093
4−0
.132
5−0
.104
60.0387
−0.172
8−0
.109
90.1481
0−0
.108
11
−0.182
120.1191
3−0
.092
14−0
.029
150.2611
60.1181
70.1011
8−0
.018
190.1012
0−0
.075
21−0
.059
220.0042
3
Chrom
eOS;
95%
confi
denceinterval
=(−
0.2828964,
0.2828964)
100.2191
−0.013
2−0
.084
30.0564
−0.066
5−0
.102
6−0
.101
70.0078
−0.042
9−0
.041
100.0361
1
−0.026
120.0021
3−0
.032
14−0
.044
15−0
.049
160.0031
70.0041
80.0431
9−0
.044
20−0
.038
21−0
.052
2−0
.012
23
123
Periodicity in software vulnerability discovery, patching and exploitation 685
Table11
IndividualACFvalues
forWeb-related
software
Apacheweb
server;9
5%
confi
denceinterval
=(−
0.1333587,
0.1333587)
100.0331
0.1222
0.0553
0.1314
0.08
50.0146
0.1897
0.0458
0.08
90.0221
00.0611
1
0.0521
2−0
.041
30.0361
40.04
15−0
.034
160.04
170.1181
8−0
.059
190.1222
00.0332
10.1212
2−0
.016
23
IIS;
95%
confi
denceinterval
=(−
0.1333587,
0.1333587)
100.17
10.0492
0.1133
0.2084
0.1525
0.1976
0.3847
0.2018
0.0499
0.2021
00.2181
1
0.1141
20.0721
30.27
140.06
150.1161
60.1231
70.3131
80.0061
9−0
.032
200.1422
10.06
220.1672
3
IE;9
5%
confi
denceinterval
=(−
0.1372249,0.1372249
)
100.2531
0.1512
0.1623
0.2974
0.1025
0.1076
0.1357
0.1648
0.1069
0.1081
00.1321
1
0.2831
20.0741
30.0391
40.0441
50.0631
6−0
.014
170.0361
80.0061
90.0282
0−0
.037
210.0132
2−0
.031
23
Firefox;
95%
confi
denceinterval
=(−
0.170593,0
.170593)
100.0581
0.2242
0.3393
0.1374
0.1335
0.0916
0.1597
0.0998
0.1489
0.1481
00.2421
1
0.0391
20.0851
30.2351
4−0
.013
15−0
.003
160.1611
7−0
.026
1801
90.0812
0−0
.002
21−0
.012
2−0
.053
23
Safari;9
5%
confi
denceinterval
=(−
0.170593,0
.170593)
100.01
1−0
.019
20.0633
0.0734
0.0125
−0.081
6−0
.039
70.0498
0.0819
−0.061
10−0
.014
11
0.3351
20.0861
3−0
.042
14−0
.039
150.0891
60.05
17−0
.082
18−0
.049
190.0822
00.0722
10.0672
2−0
.042
23
JRE;9
5%
confi
denceinterval
=(−
0.1633303,
0.1633303)
10−0
.126
10.0632
−0.066
30.5044
0.0255
0.1426
−0.112
70.4558
−0.069
0.0181
00.0571
1
0.2641
2−0
.055
130.0511
4−0
.031
50.3061
6−0
.053
170.0421
8−0
.004
190.2422
0−0
.046
210.0932
2−0
.018
23
123
686 H. Joh, Y. K. Malaiya
(a)(b) (c)
(f)(e)(d)
(g) (h) (i)
Fig. 4 Run charts for the seven half-life plots (critical vulnerabilities during 2008, by industries of finance, service, retail, manufacturing, andhealth, and overall critical vulnerabilities), patch level and exploitation
Qualys has been involved in collecting and plotting suchdata for several years. In a 2009 report [27], the organizationpresented the data collected during 2008, which represents104 million global vulnerability scans, including 82 millioninternal scans and 22 million external Internet-based scans.The data demonstrates the encountering of more than 72mil-lion critical vulnerabilities among the 680million detections.About 3500 organizations that represented major industrysectors of Financial, Health, Manufacturing, Service, andWholesale/Retail were scannedworldwide. Four distinct andquantifiable attributes related to software vulnerabilitieswereintroduced by the company:
– Half-life: the time interval measuring the reduction byhalf of a vulnerability’s occurrence. Since a shorter half-life indicates faster remediation, over time, this metricdemonstrates how successful efforts have been to eradi-cate vulnerabilities.
– Prevalence: the turnover rate of vulnerabilities in the “Top20” list during a year. The prevalence of such vulnerabili-ties are dangerous because they represent ongoing potentrisks to computing environments. Risk rises as the preva-lence rate increases because of the larger total number ofthe top 20 risks tallied during a year.
– Persistence: the measure of the total life span of vul-nerabilities. The fact that vulnerabilities persist and donot conclusively die off is a red flag for security admin-istrators. It underscores the importance of patching allsystems and ensuring that old vulnerabilities are not inad-vertently installed on new systems.
– Exploitation: the time interval between an exploitannouncement and the first attack. This metric indicatesthe probable reaction time prior to the discovery of themethod of exploiting the vulnerability. The worst sce-nario is a “zero day” attack because there is no reactiontime.
Qualys terms the above four attributes “the Laws of Vul-nerabilities”. At the 2009 Black Hat USA conference (http://www.blackhat.com/html/bh-usa-09/bh-us-09-main.html),the following observations for the each law were presented.The average duration of a vulnerability’s half-life is about 30days, varying by industry sector. Prevalence has increased,with 60 remaining in the list in 2008 compared to 50 in2004. Persistence remains virtually unlimited. Exploitationnow occurs more rapidly, often within <10days comparedto 60 days in 2004.
We observed that in the report, most of the plots visuallysuggest a short-term 7-day periodicity, as shown in Fig. 4.This section examines the statistical significance of the peri-odic pattern of selected plots in the report by using theseasonal index and autocorrelation analysis.
Figure 4 shows nine run charts from the report plotteddaily. The plots are normalized using the maximum valueset as 100 . Even visually, it is clearly observed that there arecertain periodic patterns in the data. The values decline as theresult of a remediation and increase due to new installations.Figure 5 displays corresponding ACF values from Fig. 4.In the figure, lags of seven (or its multiples) tend to either
have higher values or values outside the ±95% confidenceintervals shownby the dashed lines. This demonstrates strongautocorrelations with lags that are multiples of seven, whichconfirms a 7-day periodicity in the data.
Table 12 shows the calculated weekly seasonal indexvalues from Fig. 4 with Chi-square test. Since there is noinformation as to day of the week except (h) and (i) fromFig. 4, it is labeled as day1, day2, . . ., day7, while the twocases are mentioned on specific weekdays (Sun. throughSat.). From (a) to (g), it is observed that values generallytend to be clustered into high or low values consecutively.For example, in (g), higher values appear on day7, day1,
123
688 H. Joh, Y. K. Malaiya
and day2 successively. For (h) and (i), weekdays (Monday–Thursday) tend to have higher index values for the number ofincidents. The observed patterns could be related to softwarevendors’ patch release policies, organizations’ patch man-agement strategies, or the behavior of specific individual. Tobe statistically significant for the calculated seasonal indexvalues, the Chi-square statistic values must be greater thanthe corresponding critical values with a sufficiently smallp value. In the table, the small p values confirm the non-uniform distributions.
7 Possible factors causing periodic behaviors
Seasonality in many natural biological systems can be easilyexplained in terms of the annual seasons due to the rotation ofthe earth. Sometimes the causes of seasonality may be harderto pin down. For vulnerabilities, Rescorla [29] hasmentioneda possible cause for the observed year-end seasonality, sug-gesting that the large number of vulnerabilities reported atthe year-end may be a result of the end-of-year cleanup. Hedid not, however, discuss this possibility in detail. One possi-bility is that it may be related to year-end reports whichmanyorganizations require to be completed before the year’s end.
Further research is needed to determine why the dis-covery of vulnerabilities in Microsoft products tends topeak in the mid-year months in addition to the year-endmonths. One possibility is that Defcon (https://www.defcon.org/main.html), a major computer security-related confer-ence that originated in 1993 takes place mainly in July orAugust. While it originally stated as a hackers convention, itis attended by a large number of security professionals. The
potential conference participants might have a higher incen-tive [4] to find the vulnerabilities before the conference, tobrag about, especially in popular Microsoft products.
Figure 6a shows the number of occurrences of Defconand Black Hat; the two best known conventions at whichsecurity vulnerabilities are announced. At the same time, theAugust–November period appears to be associated with therelease of a large number of new Microsoft products. Fig-ure 6b shows the products’ releasemonths for major versionsof Windows OSes and Internet Explorer. The major versionsof Windows and Internet Explorer tend to be released dur-ing the period between June and November. This may berelated to the starting of school or to Christmas and NewYear shopping seasons, when many people buy new com-puters with new operating systems, known as IT seasonality.Condon et al. [9] also observed that occurrences of softwaresecurity incidents increase during the academic calendar, andthe most relevant form of institutional type of seasonality issummer vacation from school [19]. In December, emphasismay shift to identifying and handling vulnerabilities.
The reason for similar seasonality for the Windows oper-ating systems, IIS, and Internet Explorer might be due to thefact that IIS and Internet Explorer are distributed only forthe Windows platform, while vulnerabilities of Web serversand browsers may be correlated to parent operating systemplatforms.
Figure 7a, b shows the number of vulnerabilities groupedby day of the week in terms of disclosure date from OSVDB(data on 2010-10-06; relatively old (∵) stop providing dumpdata) and published date from NVD (data on 2014-08-05),respectively. “Disclosure date” refers to the date on whichvulnerabilities are publicly disclosed, whereas “published
JAN
FEB
MA
R
AP
R
MAY
JUN
JUL
AUG
SE
P
OC
T
NO
V
DE
C
Freq
uenc
y
0
5
10
15
20
25Black Hat: 1997~2014Defcon: 1994~2014
JAN
FEB
MA
R
AP
R
MAY
JUN
JUL
AUG
SE
P
OC
T
NO
V
DE
C
Win
200
0, S
erve
r 200
8
IE5,
IE8,
IE9
Ser
ver 2
003
Win
98
Win
NT
4.0 W
in 9
5, IE
1, IE
3, IE
6, W
in8,
Win
8.1
Win
ME
, IE
4, IE
10
Win
XP,
IE7,
Win
7, I
E11
Win
Vis
ta, I
E2
(a) (b)
Fig. 6 Frequency of Black Hat and Defcon by month, and major Microsoft software system release time by month. a Black Hat and Defcon bymonth. bMS release by month
Periodicity in software vulnerability discovery, patching and exploitation 689
Freq
uenc
y
Freq
uenc
y
SUN MON TUE WED THU FRI SAT SUN MON TUE WED THU FRI SAT SUN MON TUE WED THU FRI SAT
Freq
uenc
y
020
0040
0060
0080
0010
000
1200
014
000
020
0040
0060
0080
0010
000
1200
014
000
010
020
030
040
050
060
070
0
(a) (b) (c)
Fig. 7 Frequency of vulnerability disclosure date, published date, and data loss report data by day of the week. a Disclosure date (https://blog.osvdb.org) no. of vuln.: 67325. b Published date (https://nvd.nist.gov) no. of vul.: 63647. c Report date (https://blog.datalossdb.org) no. of reports:3047
date” is the date onwhich vulnerabilities are published on thedatabase according to OSVDB and NVD, respectively; tech-nically, the two dates have the same meaning. It is observedthat both the disclosure date’s and the published date’s val-ues peak on Tuesday, values generally increasing as Tuesdayapproaches and decreasing thereafter. Figure 7c representsthe number of data loss incidents reported by organizationsin terms of day of the week by the Open Security Foundation(data on 2010-12-28; relatively old (∵) stop providing dumpdata). Although the data does not directly reflect vulnera-bility information, it clearly epitomizes the weekday versusweekend phenomenon.
Anbalagan andVouk [2] suggest a possible weekly patternfor fixing ordinary defects. Those reported on Tuesdays tendto be fixed faster. Their graph displaying the average correc-tion time shows an opposite pattern to the first two plots inFig. 7, i.e., values decreasing toward Tuesday and increasingafter that point. This could be because developers know fromexperience that they require a higher level of effort on thatday; consequently, in order not to fall behind, more effort isdone on Tuesdays.
The findings of annual seasonality in this paper are sup-ported by the results of a survey conducted by Tufin Techno-logies (http://www.net-security.org/secworld.php?id=7928)related to hackers’ habits, with information provided byseventy-nine hackers attending the Defcon 17 conferencein 2009. Analysis of the survey reveals that Christmas andNew Year holiday seasons are popular with hackers target-ing Western countries and that hackers spend time hackingon weekdays rather than weekends. Here are some numbersfrom the survey related to our study:
– 89 said taking a summer vacation would have littleimpact on their hacking activities.
– 81 said they are far more active during the winter hol-idays, with 56 citing Christmas as the best time forhacking and 25 naming New Year’s Eve.
– 52 said that they hack other systems during weekdayevenings; 32 said that their operations take place duringweekday working hours; and only 15 of hackers do theirhacking on weekends.
8 Conclusions
Analysis of the vulnerability data using seasonal indexand autocorrelation function approaches shows that there isindeed a statistically significant annual and weekly periodicpattern in software vulnerability-related activities. In the firstpart of this paper, for all three software groups examined, ahigher vulnerability discovery rate is encountered in certainmonths. In Microsoft products, a higher incidence duringthe mid-year periods is also observed. Also, 7-day periodicbehavior was observed in the vulnerability scan data; higheractivity duringweekdays thanweekends has been confirmed.Specifically, vulnerability activity values corresponding toTuesday tend to be higher than other days of the week.
One of the main contributions of this paper is that it pro-vides a variety of evidence that there actually do exist shortand long term seasonal patterns which have been but vaguelyrecognized among security researchers thus far. The resultsfound in this paper should be used to optimize resourceallocations, patch management practices, and on the gen-eral determination of IT-related risks. For example, systemadministrators should apply patches prior to the time whenseasonal indices are relatively high for both short and longterm strategies.
Further work is needed to develop methods for the pre-diction of future vulnerability discovery trends using theBox–Jenkins time series Model (ARIMA), which uses auto-correlation function, periodogram, spectral analysis, andpartial autocorrelation function analysis. Chen et al. [8] havealready taken a periodic factor into their multi-cycle vul-nerability discovery model. However, their periodic factor
does not directly consider long or short term seasonality, butrather the vulnerability discovery rate. The findings in thispapermay be used in conjunctionwith the longer-term trendsdescribed by the vulnerability discovery models to improvevulnerability discovery predictions and to optimize resourceallocation.
References
1. Alhazmi, O.H.,Malaiya, Y.K.: Application of vulnerability discov-ery models to major operating systems. IEEE Trans. Reliab. 57(1),14–22 (2008)
2. Anbalagan, P., Vouk, M.: “Days of the week” effect in predictingthe time taken to fix defects. In: DEFECTS’09: Proceedings of the2nd InternationalWorkshop onDefects in Large Software Systems,pp. 29–30, New York, NY, USA. ACM (2009)
3. Anderson, R: Security in open versus closed systems—the danceof boltzmann, coase and moore. In: Conference on Open SourceSoftware, Economics, Law and Policy, pp. 1–15 (2002)
4. Arora, A., Telang, R.: Economics of software vulnerability disclo-sure. IEEE Secur. Priv. 3(1), 20–25 (2005)
5. Bowerman, B.L., O’connell, R.T.: Time Series Forecsting: UnifiedConcepts and Computer Implementation, 2nd edn. Duxbury Press,Boston (1987)
6. Bozorgi, M., Saul, L.K., Savage, S., Voelker, G.M.: Beyond heuris-tics: learning to classify vulnerabilities and predict exploits. In:Proceedings of the 16th ACM SIGKDD International ConferenceonKnowledgeDiscovery andDataMining, KDD’10, pp. 105–114,New York, NY, USA. ACM (2010)
7. Carrion-Baralt, J.R., Smith, C.J., Rossy-Fullana, E., Lewis-Femandez, R., Davis, K.L., Silverman, J.M.: Seasonality effectson schizophrenic births in multiplex families in a tropical island.Psychiatry Res. 142(1), 93–97 (2006)
8. Chen, K., Feng, D.-G., Su, P.-R., Nie, C.-J., Zhang, X.-F.: Multi-cycle vulnerability discovery model for prediction. J. Softw. 21(9),2367–2375 (2010)
9. Condon, E., He, A., Cukier, M.: Analysis of computer securityincident data using time series models. In: ISSRE’08: Proceedingsof the 2008 19th International Symposium on Software ReliabilityEngineering, pp. 77–86, Washington, DC, USA. IEEE ComputerSociety (2008)
11. Goonatilake, R., Herath, A., Herath, S., Herath, S., Herath, J.:Intrusion detection using the chi-square goodness-of-fit test forinformation assurance, network, forensics and software security. J.Comput. Small Coll. 23, 255–263 (2007)
12. Heston, S.L., Sadka, R.: Seasonality in the cross-section of stockreturns. J. Financ. Econ. 87(2), 418–445 (2008)
16. Joh, H., Malaiya, Y. K.: Seasonal variation in the vulnerability dis-covery process. In: ICST’09: International Conference on SoftwareTesting, Verification, and Validation, pp. 191–200, Los Alamitos,CA, USA. IEEE Computer Society (2009)
18. Kim, J., Malaiya, Y.K., Ray, I.: Vulnerability discovery in multi-version software systems. In: HASE’07: Proceedings of the 10thIEEE High Assurance Systems Engineering Symposium, pp. 141–148, Washington, DC, USA. IEEE Computer Society (2007)
19. Koc, E., Altinay, G.: An analysis of seasonality in monthly perperson tourist spending in Turkish inbound tourism from a marketsegmentation perspective. Tour. Manag. 28(1), 227–237 (2007)
20. Kozina, M., Golub, M., Groš, S.: A method for identifying webapplications. Int. J. Inf. Secur. 8(6), 455–467 (2009)
21. Maes, J., Van Damme, S., Meire, P., Ollevier, F.: Statistical mod-eling of seasonal and environmental influences on the populationdynamics of an estuarine fish community. Mar. Biol. 145, 1033–1042 (2004)
22. Massacci, F., Nguyen, V.H.: Which is the Right Source for Vul-nerability Studies? An Empirical Analysis on Mozilla Firefox.Technical report. University of Trento, Italy (2010)
23. Ott, R.L., Longnecker,M.T.:An Introduction toStatisticalMethodsand Data Analysis, 5th edn. Duxbury press, North Scituate (2000)
24. Ozment, A.: Improving vulnerability discovery models. In:QoP’07: Proceedings of the 2007 ACM Workshop on Quality ofProtection, pp. 6–11, New York, NY, USA. ACM (2007)
25. Ozment, A., Schechter, S.E.: Milk or wine: does software securityimprove with age? In: USENIX-SS’06: Proceedings of the 15thConference onUSENIXSecuritySymposium,Berkeley,CA,USA.USENIX Association (2006)
26. Pfleeger, C.P., Pfleeger, S.L.: Security in Computing, 3rd edn. Pren-tice Hall PTR, Upper Saddle River (2003)
27. Qualys, I.: The laws of vulnerabilities 2.0. In Black Hat 2009,Presented by Wolfgang Kandek (CTO) (July 28, 2009)
28. Rescorla, E.: Security holes.who cares? In: SSYM’03: Proceedingsof the 12th Conference on USENIX Security Symposium, pp. 75–90, Berkeley, CA, USA. USENIX Association (2003)
29. Rescorla, E.: Is finding security holes a good idea? IEEE Secur.Priv. 3, 14–19 (2005)
30. Rios,M., Garcia, J.M., Sanchez, J.A., Perez, D.: A statistical analy-sis of the seasonality in pulmonary tuberculosis. Eur. J. Epidemiol.16(5), 483-8 (2000)
31. Romanov, A., Tsubaki, H., Okamoto, E.: An approach to performquantitative information security risk assessment in it landscapes.JIP 18, 213–226 (2010)
32. Salehian, A.: Arima time series modeling for forecasting thermalrating of transmission lines. In: Transmission andDistributionCon-ference andExposition, 2003 IEEEPES, vol. 3, pp. 875–879 (2003)
33. Symantec. Symantec global internet security threat report: trendsfor 2009, vol. XV (2010)
34. Tran, N., Reed, D.: Automatic arima time series modeling foradaptive i/o prefetching. IEEE Trans. Parallel Distrib. Syst. 15(4),362–377 (2004)
35. Zhang, Z., Zheng, X., Zeng, D., Cui, K., Luo, C., He, S., Leischow,S.: Discovering seasonal patterns of smoking behavior using onlinesearch information. In: Intelligence and Security Informatics (ISI),2013 IEEE International Conference on, pp. 371–373 (2013)
![Vulnerability Discovery - resources.sei.cmu.edu3 Vulnerability Discovery October 25, 2016 © 2016 Carnegie Mellon University [DISTRIBUTION STATEMENT A] This material has been approved](https://static.documents.pub/doc/80x56/5fa47084ff186c548922fb0a/vulnerability-discovery-3-vulnerability-discovery-october-25-2016-2016-carnegie.jpg)
