Tripwire Combating Patch Fatigue White Paper

8/19/2019 Tripwire Combating Patch Fatigue White Paper

1/19

WHITE PAPERCONFIDENCE:SECURED

ADVANCED THREAT PROTECTION, SECURITY AND COMPLIANCE

LANE THAMES, SECURITY RESEARCHER, TRIPWIRE &TYLER REGULY, MANAGER, SECURITY RESE ARCH, TRIPWIRE

COMBATINGPATCH FATIGUE

ARE WE OVERWHELMING IT TO THE DETRIMENT OF ENTERPRISE SECURITY?


2/192 Combating Patch Fatigue

The logical response is that a singlepatch generally resolves multiple vulner-abilities. Take, for example, MS15-112,the November security bulletin forInternet Explorer, which resolved 26 vul-

nerabilities. It’s all too common to hearthe statement, “just apply the MS15-112patch.” This statement leads to theassumption that a single patch resolves

multiple vulnerabilities. While it’s truethat the application of a single patch willresolve multiple vulnerabilities, withinMS15-112 there are 32 patches avail-able for download and four more thatare referenced. If we assume that this isnormal, we can then conclude that thereare more patches issued annually than

there are vulnerabilities.

These numbers bring us to the conceptof Patch Fatigue, which can be summedup in a single question: Are we overbur-dened with patches?

Based on a recent survey we conductedof 483 IT professionals who are involved

in the patch management process acrossorganizations of all sizes, the answeris a resounding “yes.” Here are some

of the data points that led us to thisconclusion:

» Almost 20 percent of organizationsmanage their patching process withoutpatch management software

»Nearly half of all individuals surveyedadmit that at times they struggle tokeep up—or find themselves com-pletely overwhelmed with the volumeof patches released

»More than two-thirds of organizationssurveyed have fewer than five peopleactively involved in their patch man-

agement process

On top of the negative impacts toemployees, overburdened IT and secu-rity teams lead to poor security hygiene

within the enterprise. If teams cannotinstall security patches as quickly asthey are released then vulnerabilities

will l inger, providing additional attackvectors for malicious actors to use duringa data breach. This can result is substan-tial losses, as a report1 by IBM indicatesthat the average cost of a data breach is$3.8 million.

Employees that find themselves over-burdened by their workload tend to

be stressed and anxious according to areport published by Workforce2. Thisstress decreases employee productivity

and leads to a loss of talented, skilledemployees. According to WebMD3,stress can lead to heart disease, asthma,obesity, diabetes, headaches, depression,and a number of other health-relatedissues. In a follow-up report4 pub-

lished by Workforce, they showed that

employees that are stressed and feelingpressured are generally unhappy andend up looking at other employmentopportunities where their happiness willincrease.

With the impact of Patch Fatigue clearlydefined and rather self evident, we willtake a look at the reasons for and causesof Patch Fatigue in the remainder of this

white paper. More specifically, we willinvestigate the historical trends in patchmanagement and the current shiftingtrends across vendors. We also highlight

a number of factors contributing topatch fatigue on both the vendor andenterprise sides of the equation. Finally,

we offer a number of solutions that bothvendors and enterprises can employ to

lessen the pain of Patch Fatigue.

SETTING THE STAGEPatch management is the process of

acquiring, testing and installing softwarepatches for information technologyassets. Patch management plays a criticalrole in maintaining the overall securityposture for enterprise information tech-nology systems. Unfortunately, it seems

INTRODUCTIONA vulnerability is a bu or flaw in software or hardware that can be

exploited for malicious ains. In order to avoid miscommunication and

facilitate coordinated discussion, Mitre maintains the CVE (Common

Vulnerabilities and Exposures) database, which establishes a namin

standard for all vulnerabilities. In 2015, over 6,000 new CVEs were

assined. If only one-tenth of those vulnerabilities affected devices in your

area of responsibility, you would have been responsible for resolvin 630

vulnerabilities annually or 2.5 vulnerabilities each business day.

S of tw ar e C om pon en t S ecur it y

Patches in 2015Windows 7 120

Internet Explorer 13

Chrome 16

Microsoft Office 2013Professional

13

Oracle Java 4

Adobe Flash 13

Adobe Shockwave 3

Microsoft Silverlight 3

Adobe Reader 3

uTable 1: Security patch statistics for thegold image


3/19Combating Patch Fatigue

like every day we hear about a newdata breach, many of which occur orescalate due to improper patch manage-ment. Moreover, the footprint of assetsthat IT departments have to manage isexploding due to business needs revolv-

ing around new technology trends suchas mobile computing and the Internetof Things. Given the constant flux ofnew security events happening acrossthe globe along with rapidly chang-ing environments of modern day ITsystems, we should evaluate the currentstate of affairs with respect to patch

management.

To begin this study, we consideredthe amount of workload required tomaintain patches for a “gold” desktop

image representing a typical enterprise workstation. The gold image contained acollection of baseline software commonto many enterprise organizations. Next,

we calculated the number of securitypatches released for each software ele-ment in 2015. In this white paper, we areconcerned mostly with security-basedpatches. We defined a security patch as apatch that addresses at least one knownvulnerability. The results are shown inTable 1.

This very basic enterprise desktopconfiguration required a total of188 security patches during 2015, orapproximately 15 security patches permonth. Fifteen security patches per assetper month can accumulate rapidly fororganizations with a large number ofassets. For example, 46 percent of our

survey respondents were responsible fora number of IT endpoints ranging from500 to 5000.

Organizations have software footprintsbased on business needs. Respondents

were asked about their involvement withsecurity patching for various types ofsoftware used within their organization,

and these results are shown in Figure 1.

Microsoft Windows, Microsoft Office, Adobe Flash Player, Adobe Reader,Oracle Java, VMware vCenter andGoogle Chrome ranked the highest interms of our respondents’ patch manage-ment responsibilities. As a result, we will

dig deeper into these various productsthroughout the remainder of this whitepaper in order to better understand

Patch Fatigue.

Given the wide array of products thatmake up the modern IT ecosystem, weasked respondents about their comfortlevels in terms of patch management.

Particularly, we asked respondents to

rank various products in terms of thosethat are easiest to patch and those thatare hardest to patch. Figure 2 shows howrespondents rank the easiest to patchplatforms, and Figure 3 reveals howrespondents rank the hardest. According

to the results, the top five easiest plat-forms to patch are Microsoft Windows,Google Chrome, Red Hat Enterprise

Linux, WordPress and VMware vCenter.The top five hardest platforms to patchare Oracle Database, Oracle Java, CiscoIOS, VMware vCenter and Microsoft

Windows. It is interesting to see theoverlap for Microsoft Windows and

VMware vCenter, which made it into

u FIG. 2 Rank of platforms by patching ease

u FIG. 1 Product distribution



both top five lists. When comparingthe results, we see that overall Microsoft

Windows is viewed as the easiestplatform to patch. Conversely, Oracledatabase “won” for the most difficultplatform to patch.

Now that we have an understanding ofhow difficult various platforms are topatch, let us consider the relationshipbetween difficulty and patch quantity.For this, we evaluated the top five plat-forms and calculated the total number ofpatches delivered for them in 2015. The

results are provided in Table 2.

When coupling the data from Table2 with the difficulty levels associated

with each platform, as described by the

rankings from Figures 2 and 3, oneobservation is immediately obvious:Patch difficulty is not a result of thenumber of patches per year. For exam-

ple, Oracle Database had substantiallyfewer patches than Microsoft Windowsin 2015, yet it ranked as the most dif-ficult platform to patch, while Microsoft

Windows ranked as the easiest.

STRUCTURED

PATCH RELEASE CYCLESOne of the more interesting patchmanagement changes over the pastdecade has been the introduction of the

structured patch release cycle. Microsoftintroduced the world to the conceptof structured patch release cycles inOctober 2003 when they launched PatchTuesday. This regular cadence allowedenterprises to plan and schedule theirupdates. Some companies even intro-duced the concept of “Patch Saturday,”

a day IT teams set aside two weeks afterpatches are released to regularly installneeded updates.

A couple of years after Microsoft started with a structured cycle, Oracle joinedthe game, announcing in 2005 quarterlyupdates. In 2008, Cisco joined in by ini-tiating biannual updates for IOS. Adobe,

who has become closely entwined withMicrosoft, started following Microsoft’sPatch Tuesday schedule in late 2012.

While Adobe doesn’t strictly abide bythis schedule and unscheduled updatesstill drop, they’ve done a good job ofbeing in line with Microsoft’s patchrelease schedule.

With structured updates comes struc-

tured information. All of the vendorslisted above started preannouncing theirsecurity updates to inform customersof what was coming so that they could

properly prepare for the patch before itdropped. This put enterprises IT andsecurity teams in a much better position,as they were able to ensure they hadadequate resources available to deal withthe patches when they were released.

While many vendors continue this cycle,Microsoft decided to discontinue the

advanced notification, now delivering anunknown number of updates affectingvarious platforms.

Our enterprise patch managementsurvey looked to garner feedback fromrespondents on structured patch cyclesand found that nearly two-thirds oforganizations prefer Microsoft’s monthly

patch cycle over longer intervals. One ofthe more interesting discoveries was thatone-third of organizations would prefer

individual patches be released as they areavailable, similar to the cycle that RedHat uses for its RHEL patches.

It should come as no surprise to read-ers that less than two percent of those

Product Platform SecurityPatches in 2015

RHEL 2859

Windows 2804

Oracle Database 276

Oracle Java 116

IBM AIX 176

Cisco IOS 62

Google Chrome 16

VMware vCenter 12

WordPress 6

uTable 2: Security patches delivered in2015 for the top five easiest and hardestplatforms to patch

u FIG. 3 Rank of platforms by patching difficulty



surveyed preferred the Cisco (quar-terly) and Apple (unscheduled bundle)approaches to patch releases. This shouldbe an eye opener for vendors like Applethat don’t adhere to a schedule. Cisco’sextended cycle can greatly increase risk

by increasing the window where a vul-nerability can be exploited, increasingthe attack surface. On the other hand,

Apple’s cycle allows for no preparationor preplanning, causing IT organiza-tions to rush to apply unannouncedupdates. Both of these methods shouldbe recognized as contributing factors to

the increasing Patch Fatigue that we’reseeing across enterprises.

When our survey looked at activelyexploited vulnerabilities, the focus

shifted. Eighty percent of those surveyed would like to see vendors test patchesand then release them immediately.Unsurprisingly, fewer than one in 10

respondents advocated for vendors tomaintain their regular schedule whencritical fixes are needed to mitigateactive attacks.

One thing is clear: Structure is greatlypreferred in the enterprise world. Manyvendors have strived to provide this, and

it’s critical that it be maintained goingforward in order to ensure we limit theeffects of Patch Fatigue on employees

working in IT.

SHIFTING TRENDS WITH MICROSOFT When you start to investigate PatchFatigue, it’s impossible to discuss theconcept without considering some of the

shifting trends affecting both vendorsand enterprises. As we discussed above,numerous vendors have introduced

scheduled patch drops into their process,making it easier for enterprises to planfor and manage new updates. However,it is clear that these processes are living,breathing entities that change over time.The shift to scheduled patch drops is

u FIG. 4

u FIG. 5

evidence of how the process changes. Byscheduling their releases, vendors allowed

enterprise IT and security teams to planaccordingly. This ensured that majorprojects were not impacted, available

resources were properly scheduled, andthat potential downtime was announced.

When considering these benefits, it’svery easy to see how a shift in the trendcan positively or negatively impact PatchFatigue within an organization.

One of the more interesting shiftingtrends has been Microsoft’s stance

on enterprise security and the releaseof patches. One such example is theinclusion of Adobe Flash bundled with

Internet Explorer and Edge. Flash in Windows XP proved to be a challenge,as we’ll see in section Adobe FlashPlayer: The Battle of the Bundle, somany were surprised to see it return.Initially, the inclusion of Flash packaged



with Windows lead to a single securityadvisory that persisted over three years.Microsoft’s shift away from issuingsecurity bulletins and instead con-tinuously updating the same securityadvisory made things more challenging

for Windows administrators. However,Microsoft resolved this issue as it relatesto Flash in the February 2016 PatchTuesday drop when they released a secu-rity bulletin that replaced the securityadvisory. Normally, you don’t have thistype of replacement, indicating thatMicrosoft realized they could improve

the process and sought to resolve thispain point.

Windows 10 patch releases have demon-strated a shift away from straightforward

Windows patch management. Gone is asingle line of security patches as used byprevious versions of Windows; instead

we see a shift toward multiple release

branches, all with different rules forpatching. The three branches includeCurrent Branch (CB), Current BusinessBranch (CBB), and the Long-TermServicing Branch (LTSB) and eachrequires a commitment to a differentrelease cycle that is unlike anythingMicrosoft has offered before. The pro-

cess is complex enough that Microsofthas published a multi-page document5 tohelp explain the process.

The new line of servicing options isconfusing and reminiscent of Cisco’sversioning. As we’re discussing PatchFatigue, it’s probably worth noting thatonly one-third of Cisco IOS administra-

tors are able to decipher which updatesto install without contacting Cisco’stechnical support team. Windows 10

appears to be heading down a similarpath, with only one-third of thosesurveyed feeling that Windows 10 hasimproved patch management. Thisnumber is surprisingly similar to thenumber of individuals who can decipher

Cisco IOS updates. These results areeven more telling when you consider

that 41 percent of those surveyed feltthat Windows 10 was making enter-prise patch management more difficult.Microsoft’s decision to combine servicingoptions with a single cumulative updatethat Microsoft releases for Windows

10, which allows for no control over theindividual updates installed may explain

why this number is so high.

WHAT TO DO ABOUT JAVA Java was introduced to the world in1995. At that time, the Internet, along

with the World Wide Web, was boom-

ing, and the Web was largely built withstatic HTML pages. Java introduced theability to add dynamics to the Web viaits graphical capabilities with applets.This single capability drove very wide

adoption of the Java language, andplayed a huge role in its success. In a

world hungry to build distributed net- worked applications using the Web, Java

quickly dominated the scene. However,there was one other feature that alsoplayed a big role: security. Upon itsdebut, Java was advertised as a highlysecure language. Security was especiallyimportant when building applicationsfor the Web, but we know that there isno such thing as absolute security and,

over time, the number of insecurities

u FIG. 6 Java vulnerabilities over time

and weaknesses discovered in Java beganto increase.

Java originally debuted as a secureplatform because of several features.One was that the language was built

with mechanisms capable of enforcingruntime constraints such as preventingbuffer overflows. Java checks the boundsof buffers and will prevent access toany memory beyond those bounds. Javaalso contains a security managementmechanism that uses a sandbox to isolateuntrusted code from the overall system.

However, all of these systems are built with software, and all software containsflaws that can lead to security vulner-abilities. Figure 6 shows the approximatecount of core Java vulnerabilities that

have been discovered since 2000.

Between 2000 and 2005, Java experi-enced linear growth in the number of

vulnerabilities being discovered. From2005 until 2008, that growth followedan exponential trend, and since 2008the trend has been stochastic. We cansee a peak in 2013, which represents 193distinct vulnerabilities. The continualdiscovery of new Java vulnerabilitiesmeans that outdated versions of Java

cannot be considered secure. The need



u FIG. 7 Adobe Flash vulnerabilities over time

to constantly manage these installationsappeared to be a pain point amongsurvey respondents.

It turns out that Java patches introducePatch Fatigue-inducing challenges for

IT and patch management teams. Thedecoupled aspect of managing applica-tions written with Java separately fromthe Java platform (e.g. Java RuntimeEnvironment (JRE)) is one of these chal-lenges. In particular, IT departmentsand patch management teams often needto wait for developers to update their

Java applications before they are ableto apply security patches for the Javaplatform. This is a common problemconnected with legacy applications, andthis timing delay places organizations

that depend on Java-based applicationsat risk. This scenario is of great concernfor organizations—86 percent of respon-dents stated that they are concerned

about the security of Java-based appli-cations. When asked for more detailsabout their concerns, the sheer numberof reported Java vulnerabilities ranked asthe highest concern, followed by the factthat the Oracle Java updater does notremove older, more vulnerable versions6.

When asked to provide specif ic concerns

related to Java, respondents clearlysignaled that issues with applicationcompatibility and the need to run olderversions of Java for legacy applications

were particularly troubling. As a result,27 percent of respondents say they are tobe phasing out Java-based applications

within their IT environments.

ADOBE FLASH PLAYER:THE BATTLE OF THE BUNDLE

Adobe Flash Player is another product

that is often on the minds of IT depart-ments and patch management teams.The continuous discovery of new AdobeFlash vulnerabilities is definitely a keyissue. Figure 7 shows the number ofknown vulnerabilities that have been

discovered in Adobe Flash since 2000.

Another key issue is the consistentappearance of exploits for Adobe Flashvulnerabilities within exploit kits. There

was a time when Java was a consistent

exploit kit target but Adobe Flashappears to be the favorite today. We ana-lyzed the vulnerabilities included in the

Angler exploit kit dating back to 2013and found that 76 percent of the exploitstargeted Adobe Flash. The remaining24 percent targeted predominantly

Java, alongside Internet Explorer and

Microsoft Silverlight.

The above data clearly indicates whyadministrators are worried aboutnew vulnerabilities in Adobe Flash.Unfortunately, managing Adobe Flashsecurity patches is not easy because thissoftware is now bundled with otherproducts. Bundled software can raise

the level of difficulty for administrators who need to understand which partsof the application need to be updated

and which vendor is responsible for theupdates. Adobe Flash has seen its shareof patch difficulty over the years due tothis bundling scenario.

One particular issue that highlights the

difficulty caused by bundled softwarebecomes evident when determining

the ownership of security updates. Agreat example of this problem occurredbetween Microsoft and Adobe back in2010 when KB9792677 was released.

Microsoft had bundled Adobe FlashPlayer 6 with Windows XP, but didnot ship security updates as Adobeissued patches. After multiple vulner-abilities surfaced in this version of Flash,Microsoft released the security advisory

warning users to uninstall Adobe Flash6 and upgrade to a newer version. Once

Microsoft stopped bundling Flash, theboundaries became clear: users wereresponsible for the installation of Flashalong with its patch management via onesingle source, Adobe. Unfortunately, itdidn’t take long for the Flash bundlingsituation to occur again with multiplevendors. Google began bundling AdobeFlash Player within the Chrome browser,

and then Microsoft began bundling it with Internet Explorer. Administrators were placed in a dif ficult situation where

attributing Adobe Flash vulnerabilitiesbecame problematic. They were backinto a patch management scenario whereFlash vulnerabilities might need to bepatched by either the browser’s vendor or

Adobe, or in some cases, both. Microsoft

has recently made an effort to ease thispatch management pain. Until February



2016, Microsoft maintained a singlesecurity advisory that detailed AdobeFlash vulnerabilities related to bundledinstallations for Microsoft products.However, MS16-022 marked the firstsecurity bulletin to directly address

bundled versions of Adobe Flash Playerin Microsoft products.

Obviously, software vendors feel thatbundling software has its benefits.However, what do IT professionals thinkof this patch management paradigm?Eighty-six percent of our survey respon-

dents stated that products with multipledistribution methods (standalone andbundled in other products) such as

Adobe Flash create challenges in under-standing the impact of security patches.

CONTRIBUTING FACTORS: PATCHPRIORITIZATION AND TIMING

Patch management plays a critical role

in strengthening the security postureof most organizations. Fifty-nine per-cent of our respondents claimed thatsecurity-related patches take priority overnon-security related patches, but thisis not the only set of priorities. Figure8 shows how organizations prioritizepatches based on various categories. The

data shows that issues such as knownattacks, public exploit availability andreboot requirements play significant roles

when prioritizing security patches.

It is interesting that the “RebootRequired” category ranked third inimportance when prioritizing patches.However, it makes sense when you con-

sider that server footprints are very largein modern IT infrastructures. Fromour survey, we found that 90 percent

of respondents had responsibility forpatching server endpoints. Another note-

worthy point is related to CVSS scoring.CVSS is an open standard used forassessing the severity of known vulner-abilities, and many security vendors and

various industry standards, such as thePayment Card Industry Data Security

u FIG. 8 Patch prioritization categories

u FIG. 9 Participant involvement with industry standards

Standard (PCI DSS), use it. PCI, whichrequires that all vulnerabilities with a

CVSS score of 4.0 or higher be patched,is a retail industry standard that’s appli-cable to all companies processing credit

card transactions. Respondents wereasked about their organizations’ adher-ence to industry standards. Figure 9shows these results.

Even though 39 percent of respondents

must adhere to PCI DSS standards,CVSS ranks next to last in terms of howadministrators prioritize patches.

Timing and prioritization are importantaspects of patch management. Figure 8

shows that security patches related tovulnerabilities with known attacks or

with publicly available exploits are most

important to those involved with patchmanagement. When exploits surface,vendors should respond accordinglyby providing patches for the associatedvulnerabilities. However, our surveyindicated that when it comes to patches

released for vulnerabilities with in-the-wild exploits, IT teams consider



prudence to be more important thanurgency. Survey respondents stronglypreferred thoroughly tested patches, and

we found that 80 percent of the respon-dents wanted to receive security patchesfor in-the-wild exploits as soon as the

patch was developed but only after it wasfully tested. Comparatively, only six per-cent of those surveyed wanted the patchas soon as it was available regardless oftesting, and 10 percent were simply satis-fied with delivery during the vendor’snormal patch delivery cycle.

The survey clearly indicated thatthorough testing of security patches

was desirable, yet we observed a smalldiscrepancy between the viewpointsof executives and individual contribu-

tors. While executives tend to be moreconcerned with risks associated withunpatched vulnerabilities that haveknown exploits, individual contribu-

tors tend to be more concerned withrisks associated with the deployment ofuntested security patches. When askedif a security patch for an in-the-wildexploit should be delivered as soon asit is available even without being fullytested, 11 percent of executives agreedto that approach versus four percent of

individual contributors. This may be dueto a difference in responsibilities. Whileexecutives are responsible for higher-level concerns (such as the cost of a databreach), individual contributors are con-cerned about day-to-day operations andbetter recognize the risks associated withdeploying untested patches on criticalinfrastructure.

The constant influx of patches acrossthe many different types of IT assets

means that time becomes a criticalfactor. Respondents were asked about theamount of time considered acceptablebetween the release of a security patchand its installation in their environ-ments. They were also asked how long it

takes to deploy security patches in their

environments. The results are shown inFigure 10.

As the data suggests, organizations arecurrently on track with the amount of

time deemed acceptable and the actualamount of time needed to deploy secu-rity patches. The majority of respondents

feel that security patches should betested and deployed within seven days ofrelease. Together, 93 percent of respon-dents take no longer than one month todeploy security patches. This participantperception doesn’t seem to mesh with

the reports published by other vendors

and researchers. A report8 published byNopSec indicates that the financial andeducation sectors take an average of 176days to remediate a vulnerability.

One key component affecting theamount of time needed to deploy patch-es is testing. Respondents were asked if

they tested patches before deployment,and 47 percent said they did for desktopsand 55 percent for servers. Figure 11shows the amount of time taken by ourrespondents to test patches.

u FIG. 10 Timing aspects of security patches

u FIG. 11 Patch test times



As Figure 11 reveals, respondents tendto spend less time testing patches fordesktops and slightly more time testingpatches for servers. When consideringthe impact that a faulty patch can haveon IT environments, it is obvious that

administrators want more time to testserver patches. Although the vast major-ity of respondents are comfortable withdeploying patches within seven days, weobserved a small discrepancy betweenexecutives and individual contributorsin terms of immediate patch deploymentrequirements. Of those respondents

who feel that security patches should bedeployed immediately, 12 percent wereexecutives, and only five percent wereindividual contributors. This shows aclear distinction between motivations

of executives versus administrators—executives are more concerned aboutthe potential impact of a security event,and administrators are more concerned

with the potential impact of deploying afaulty patch since this can impact avail-ability and reliability of critical businesssystems.

CONTRIBUTING FACTORS:RECOGNIZING VULNERABILITIES

When discussing security-related

updates, it’s important to remember thegoal of patches: remediating a vulner-ability rather than f ixing a functionalbug or adding new features. For thoseon the security side, that may seem likea straightforward concept, but there’soften a disconnect between security andoperations teams on exactly what needsto be done. This disconnect is one of

the major contributing factors of PatchFatigue.

Figure 12a shows the responses to thesurvey question, “Does your IT staffhave difficulties understanding the dif-ference between applying a patch andresolving a vulnerability?” If the answeris yes, then you can represent the data as

illustrated in Figure 12b.

u FIG. 12a Does your IT staff have difficulty understanding the differencebetween applying a patch and resolving a vulnerability?

u FIG. 12b Does your IT staff have difficulty understanding the differencebetween applying a patch and resolving a vulnerability?

A great example in the diff iculty present-

ed when attempting to understand thedifference between applying a patch andresolving a vulnerability is MS15-1249,

the December 2015 Internet Explorercumulative update that resolved 30CVEs. In most cases, Windows adminsexpect to install the update and be done,but one CVE in this bulletin contained aspecial note.

The bulletin laid out details on how

to take the additional steps requiredto truly mitigate the vulnerability. Inmany cases, this additional step was not

taken, leaving systems in a vulnerablestate. This meant that companies thatverify with Vulnerability Managementproducts rather than Patch Managementproducts left their internal teams withthe additional overhead of verifying if

systems were truly vulnerable. This may



u Important: Your system is

not protected from this ASLR

Bypass unless you install the

applicable updates and then

enable the User32 Exception

Handler Hardening Featureu

CVE-2015-6161

or may not lead to external requests tovendors, consultants, or others.

In order to better recognize individual

vulnerabilities, administrators shouldcompletely review vendor security bulle-tins. There are varying degrees of usefulinformation in the bulletins providedby the vendors, which we’ll investigate

in an upcoming section. However,understanding bulletins is important toproperly resolve vulnerabilities via patchapplication.

Take a look at the team around you. According to the survey results, half ofyour team do not understand if a vulner-ability is resolved after applying a patch.Think about the extra cycles caused bythat lack of understanding, the addi-tional work done by individuals within

your team, and by the vendors thatsupport you. This is clearly a widespreadissue within the industry, and it’s easierto understand how this contributes tothe overburdening of teams responsiblefor enterprise patch management.

In the security world we often talk aboutend-user education as the key to good

security hygiene, but it may be that within patch management, education isa missing piece. Many post-secondary

institutions talk about cryptography whenever the word security is men-tioned, while others are starting to bringin courses focused on exploit develop-ment. These lessons don’t seem to crosspaths with operations-related teachings.

Security conferences and local meet-upspresent the perfect place to provide thistype of education. Unfortunately, manyof the events are focused on introducing

cutting-edge or “cool” concepts ratherthan solidifying core knowledge. It’s theresponsibility of the community to createa solid forum for sharing this knowledgeand educating others.

Internally, companies should look tocreate and promote knowledge sharing.

If half of your team understands what isgoing on, they should be spreading thatknowledge to the other half. Brown baglunches are a great way to start a pro-gram like this, and those can be furtherimproved if the company pays for lunch.This expense is a relatively small invest-ment to help reduce the Patch Fatiguethat the knowledge gap contributes to.

Trainers can be brought in to provideadditional knowledge transfer, as well asa recognized expert to answer questions.

There are plenty of training organiza-tions available, and many vendors areable to provide indirect guidance on thissubject during product training.

u FIG. 13

CONTRIBUTING FACTORS:SECURITY BULLETINSIt’s practically impossible to talk aboutsecurity patching and Patch Fatigue

without considering the role that vendorbulletins play. When you’re patching asystem, bulletins tell you what to patch,how to patch it, and which vulner-abilities are resolved. These should bea critical source of information for alladministrators, but many find them tobe more of a hindrance than help.

We looked at major vendors and askedsurvey respondents to classify their topvendors for both best and worst securitybulletins. Microsoft provides the bestcontent (see Figure 13), but it simultane-ously ranks near the top of the list for

worst information providers. Microsoftlikely appears as the best information

provider because it is among a short listof vendors that clearly call out post-con-figuration steps, provide details on the

nature of the vulnerability resolved, andprovide work-arounds (when available) tothose that can’t be patched immediately.

When looking at the worst informationproviders, some respondents provided



additional commentary. While onlyone individual expressed dissatisfaction

with Microsoft bulletin quality, severalrespondents wanted to call out Oracle.This is unsurprising since an Oracle bul-letin can contain several hundred links

to patches—a number that appears tobe unrivalled by any other vendor. Thismethod of dumping updates withoutadequate information clearly doesn’t sit

well with survey respondents. A propersecurity bulletin would remove thisconcern, which is potentially harmingOracle’s reputation and unnecessar-

ily increasing the workload of theircustomers.

The sheer quantity of security patchesconsumes an enormous amount of an

enterprise’s resources in part because ITteams are often unsure of when to applyspecific patches. Figure 14 shows thatonly 34 percent of enterprise patch man-

agement teams are “always confident”that they understand which patchesapply to which systems. This numberis disturbingly low, and is also a clearindicator that the remaining 66 percentend up doing additional work. Thisadditional work could include trial anderror patch installations, phone calls and

tickets to vendors, and internal meetingsto discuss patch deployment. All of theseadditional tasks add to patch deploymenttimes and increase the teams collectivePatch Fatigue.

To further investigate this issue, welooked deeper into Cisco patches.Conference presentations and entire

books have been published on the com-plexity of the Cisco release model. Theirbulletins with lists of affected software

were so complex that the lists wereremoved and a tool was written that nowallows you to enter your software versionto determine if you are affected.

u FIG. 14

u FIG. 15

When you break it down, nearly two-thirds of administrators require outside

assistance to update their Cisco IOSdevices, which slows down the patchcycle and increases the burden on others.

The data makes it clear that one vendorstands out with more than four positiveresponses for every negative: VMwarestands above every other vendor in theeyes of those surveyed. A quick look at

a VMware Security Advisory (VMSA)reveals why. They are clearly organized

without information overload, and com-municate sufficient detail quite well.

Within a few minutes of reviewing these

bulletins, we were able to understand what was f ixed, identify the productsthat we were running, and found thefixes that should be applied. This is amajor improvement over the bulletins ofthe other vendors we reviewed.



While many vendors have started tomove to CVRF and OVAL for machine-readable bulletin content, very few havestandardized the human-readable webinterfaces. In fact, even though patchvolumes and complexity have increased,

over the years many vendors havedecreased the information they makeavailable to their users. This makes itharder for administrators and securityteams to tell what an update is doing, ifit is resolving specific issues that concernyou, and makes it difficult to identify

which updates you need to apply. These

communication failures are all majorcontributors to the build-up of PatchFatigue within an organization.

In this case, the majority of the onus is

on the vendors. Every vendor needs tocommit to making their security bul-letins and all patch documentation easierto read and understand. Standardizing

human-readable content is clearly animportant step in improving our patch-ing ecosystem—and thereby reducingenterprise Patch Fatigue.

That doesn’t mean that administratorsand operations teams are off the hook.Investments in training and education

can go a long way in improving the abil-ity of your team to understand securitybulletins. If your organization is luckyenough to have an employee in the twopercent of Cisco administrators whofully understands bulletins, have themcross-train the rest of the team. If not,find someone and bring them in-houseto perform training. While it would be

nice to wait for vendors to delivery betterdocumentation, there are definitely stepsthat can be taken within the enterprise

to alleviate Patch Fatigue and decreasethe overall burden on security andoperations teams.

PATCH MANAGEMENT vs.VULNERABILITY MANAGEMENT

When evaluating enterprise patch man-agement programs the prevalence ofpatch management versus vulnerabilitymanagement technologies is an interest-

ing factor. The two terms are often usedinterchangeably and many would behard-pressed to explain the difference.Before we investigate the applicabilityof both systems to the enterprise patchmanagement program, let’s discuss theunique aspects of these tools.

Patch management usually acts on oneof two levels, depending on the func-tionality of the product involved. At thehighest level, it looks at vendor bulletins,but these products tend to be the least

accurate offering as vendors seldomrelease bulletins with a 1:1 mapping topatches. Higher-end patch managementsoftware looks at the individual patches,

often rolling them up to the bulletinlevel. This approach more accuratelytracks the deployment of patches acrossthe enterprise.

On the other hand, vulnerability man-agement breaks patches and bulletinsdown to the individual vulnerabilities,

often using CVE as the standard identi-fier. One common misconception is thatvulnerability management requires theexploitation of vulnerabilities. In reality,the concepts applied within vulner-ability management are similar—if notidentical—to patch management. Thisincludes checking for indicators that aspecific patch has been applied. Where

vulnerability management differs frompatch management is in checking forpost-patch application steps that may be

required.

A proper enterprise patch managementprogram should utilize both vulner-ability and patch management tools toensure a holistic solution. Using only one

of the tools can often leave you withoutenough information to ascertain yourtrue security posture, and while 87 per-cent of organizations surveyed use patchmanagement software, only 43 percentunderstand the difference between

applying a patch and resolving a vulner-ability. These numbers indicate that amature vulnerability management pro-gram could help reduce Patch Fatigue.

One of the better examples of the limita-tions of patch management tools withminimal functionality is Microsoft

Baseline Security Analyzer (MBSA).This tool has the potential to lead ITorganizations astray by indicating thatsystems were fully patched. MBSAfails to report issues in software that

is no longer supported (such as olderversions of the .NET Framework andthe antiquated Microsoft Java). Thiscauses confusion among organizations

with mature vulnerability managementprograms because they are led to believethere are issues with their vulnerabilitymanagement products, but in reality theissue is caused by discrepancies in the

way unsupported software products arereported.

A more recent example of why organiza-tions need both patch and vulnerabilitymanagement software is MS15-124,the December 2015 Internet Explorersecurity bulletin that we discussedearlier. While patch management soft-

ware may indicate whether or not thepatches associated with MS15-124 areinstalled, the reporting generally stops

there; the reports don’t indicate if theadditional step required to resolve one ofthe vulnerabilities contained within the

bulletin has been taken. This is wherethe difference between vulnerabilitymanagement and patch managementis further highlighted. As noted inthe section Contributing Factors:Recognizing Vulnerabilities, one of the



u FIG. 16

vulnerabilities, CVE-2015-6161, requiresusers enable a registry setting in orderto enable the mitigation installed by thepatch. Without this change, the systemremains vulnerable.

The flip side of this conversation isthat while vulnerability managementis great at finding the one-off condi-tions that patch management misses, itdoesn’t necessarily show the easiest pathto resolution. It’s much easier to haveone line item that says, “Install patch x from MS15-124,” than to have 30 line

items for each CVE identified withinMS15-124. This is why patch manage-ment should be used hand-in-hand withvulnerability management to ensureadequate security and reduce Patch

Fatigue.

COMBATING PATCH FATIGUEWITH A MATURE PATCHMANAGEMENT PROGRAM

One of the more surprising statisticsfrom this survey was that nearly one-fifth of those surveyed don’t use patchmanagement software (Figure 16). Animportant part of combating PatchFatigue is a mature patch managementprogram. While other contributing

factors to Patch Fatigue require vendorchanges or extensive training, this spe-cific contributing factor can be resolved

with changes to your internal patchmanagement process.

The first step in this process is theproper use of software, which issomething we’ve already discussed.

Vulnerability and patch managementsolutions should be used together to geta picture of the enterprise security pos-

ture, including both current patch levelsand known risks. When either of thesetools is missing, additional responsibilityis placed on individuals instead of tech-nology. Without these tools, individualsmust be aware of every asset and every

application installed on every asset. Asmentioned in the section Setting theStage, a typical enterprise workstation

could require nearly 200 patches annual-ly. Expecting any employee to manuallytrack and manage this is simply unfeasi-ble, and would add additional stress thatthose working in security and operationsdon’t need.

Moreover, as we discussed above, sched-uling and testing are important factorsfor many enterprises in the deployment

of patches. While many enterprises haveclear policies around this, which is a suresign of a mature patch management pro-gram, some don’t. Setting up schedules,assigning responsibilities and definingroles are easy steps to take in combating

u FIG. 17



Patch Fatigue. It’s much easier for anadministrator to plan for server down-time and patch installation if they knowthat patches are installed on the thirdSaturday of every month, as opposed tounscheduled events.

Consider your plan for unexpectedissues. If a patch installation fails ortakes a system offline, what is yourrecovery plan? It’s not uncommon forvendors to release bad patches that pro-duce completely unexpected results. Arecent Windows 10 update broke Citrix

functionality, which was a potentialnightmare for enterprises unaware of thisnegative interaction. This is why testingis so important and why back-up systemsfor mission-critical roles are a great way

to reduce the stress an organization feels when deploying an update.

One of the major contributing factors

to Patch Fatigue is the lack of adequateheadcount. Staffing to appropriate levelsis important. Do employees have time toreview bulletins? Are they rushed whenthey deploy patches? Do they have timeto understand and apply post-installationsteps or to make sure they understandchanges before deploying an update?

These are important questions for man-agement to ask. If your employees sufferfrom Patch Fatigue, employee morale

will drop, your enterprise security pos-ture will be negatively impacted, and thepotential for downtime will increase.

If you feel that your enterprise is “doingfine,” consider Figure 17, which dem-

onstrates that while more than half ofenterprises are managing the volume ofpatches, a substantial number are not.

Ultimately, the first step in resolvingPatch Fatigue is identifying it, so sittingdown with your team and identifyingpotential points of failure and stress isbeneficial to any discussion. Once you

start to resolve the points identifiedabove, you’re on your way to a moremature enterprise patch managementprogram, which will subsequentlyimprove your security posture.

In conclusion, while we’ve thrown a lotof numbers and statistics at you withthis paper, it’s important to rememberthe end goal. Patch Fatigue is very realfor many organizations, and resolvingit will lead to happier, more productiveemployees and, ultimately, more secureenvironments. Security should be at the

top of every company’s priorities giventoday’s threat landscape and all improve-ments, especially low-hanging fruit likethis, should be seen as positive gains. Soplan that first meeting and figure out

if Patch Fatigue is affecting your teamtoday.

FOOTNOTES

1 http://www-03.ibm.com/security/data-breach/

2 http://www.workforce.com/articles/today-s-workforce-pressed-and-stressed

3 http://www.webmd.com/balance/stress-manaement/features/10-fix-able-stress-related-health-problems

4 http://www.workforce.com/articles/20310-two-years-later-still-stressed-and-pressed

5 https://technet.microsoft.com/en-us/library/mt598226(v=vs.85).aspx

6 https://www.ftc.ov/news-events/press-releases/2015/12/oracle-arees-settle-ftc-chares-it-deceived-consumers-about-java

7 https://technet.microsoft.com/en-us/library/security/979267.aspx

8 http://info.nopsec.com/2015StateofVulnerabilityRiskManaement_ThinkLikeaHacker.html

9 https://technet.microsoft.com/en-us/library/security/ms15-124.aspx

http://www-03.ibm.com/security/data-breach/http://www-03.ibm.com/security/data-breach/http://www.workforce.com/articles/today-s-workforce-pressed-and-stressedhttp://www.workforce.com/articles/today-s-workforce-pressed-and-stressedhttp://www.webmd.com/balance/stress-management/features/10-fixable-stress-related-health-problemshttp://www.webmd.com/balance/stress-management/features/10-fixable-stress-related-health-problemshttp://www.webmd.com/balance/stress-management/features/10-fixable-stress-related-health-problemshttp://www.workforce.com/articles/20310-two-years-later-still-stressed-and-pressedhttp://www.workforce.com/articles/20310-two-years-later-still-stressed-and-pressedhttps://technet.microsoft.com/en-us/library/mt598226(v=vs.85).aspxhttps://technet.microsoft.com/en-us/library/mt598226(v=vs.85).aspxhttps://www.ftc.gov/news-events/press-releases/2015/12/oracle-agrees-settle-ftc-charges-it-deceived-consumers-about-javahttps://www.ftc.gov/news-events/press-releases/2015/12/oracle-agrees-settle-ftc-charges-it-deceived-consumers-about-javahttps://www.ftc.gov/news-events/press-releases/2015/12/oracle-agrees-settle-ftc-charges-it-deceived-consumers-about-javahttps://technet.microsoft.com/en-us/library/security/979267.aspxhttps://technet.microsoft.com/en-us/library/security/979267.aspxhttp://info.nopsec.com/2015StateofVulnerabilityRiskManagement_ThinkLikeaHacker.htmlhttp://info.nopsec.com/2015StateofVulnerabilityRiskManagement_ThinkLikeaHacker.htmlhttp://info.nopsec.com/2015StateofVulnerabilityRiskManagement_ThinkLikeaHacker.htmlhttps://technet.microsoft.com/en-us/library/security/ms15-124.aspxhttps://technet.microsoft.com/en-us/library/security/ms15-124.aspxhttps://technet.microsoft.com/en-us/library/security/ms15-124.aspxhttps://technet.microsoft.com/en-us/library/security/ms15-124.aspxhttp://info.nopsec.com/2015StateofVulnerabilityRiskManagement_ThinkLikeaHacker.htmlhttp://info.nopsec.com/2015StateofVulnerabilityRiskManagement_ThinkLikeaHacker.htmlhttp://info.nopsec.com/2015StateofVulnerabilityRiskManagement_ThinkLikeaHacker.htmlhttps://technet.microsoft.com/en-us/library/security/979267.aspxhttps://technet.microsoft.com/en-us/library/security/979267.aspxhttps://www.ftc.gov/news-events/press-releases/2015/12/oracle-agrees-settle-ftc-charges-it-deceived-consumers-about-javahttps://www.ftc.gov/news-events/press-releases/2015/12/oracle-agrees-settle-ftc-charges-it-deceived-consumers-about-javahttps://www.ftc.gov/news-events/press-releases/2015/12/oracle-agrees-settle-ftc-charges-it-deceived-consumers-about-javahttps://technet.microsoft.com/en-us/library/mt598226(v=vs.85).aspxhttps://technet.microsoft.com/en-us/library/mt598226(v=vs.85).aspxhttp://www.workforce.com/articles/20310-two-years-later-still-stressed-and-pressedhttp://www.workforce.com/articles/20310-two-years-later-still-stressed-and-pressedhttp://www.webmd.com/balance/stress-management/features/10-fixable-stress-related-health-problemshttp://www.webmd.com/balance/stress-management/features/10-fixable-stress-related-health-problemshttp://www.webmd.com/balance/stress-management/features/10-fixable-stress-related-health-problemshttp://www.workforce.com/articles/today-s-workforce-pressed-and-stressedhttp://www.workforce.com/articles/today-s-workforce-pressed-and-stressedhttp://www-03.ibm.com/security/data-breach/http://www-03.ibm.com/security/data-breach/



STUDY DEMOGRAPHICS

u FIG. 18

u FIG. 19

u FIG. 20



u FIG. 21

u FIG. 22



u FIG. 24

u FIG. 23


19/19

u Tripwire is a leading provider of endpoint detection and response, security, compliance and IT operation solutions for

enterprises, service providers and government agencies. Tripwire solutions are based on high-fidelity asset visibility and

deep endpoint intelligence combined with business context; together these solutions integrate and automate security and

IT operations. Tripwire’s portfolio of enterprise-class solutions includes configuration and policy management, file integrity

monitoring, vulnerability management, log management, and reporting and analytics. Learn more at tripwire.com u

SECURITY NEWS, TRENDS AND INSIGHTS AT TRIPWIRE.COM/BLOG u FOLLOW US @TRIPWIREINC ON TWITTER

©2016 Tripwire Inc Tripwire is a registered trademark of Tripwire Inc
http://www.tripwire.com/bloghttps://www.twitter.com/tripwireinchttps://www.twitter.com/tripwireinchttp://www.tripwire.com/blog

Date post:	07-Jul-2018
Category:	Documents
Upload:	andrew-thompson
View:	216 times
Download:	0 times

Tripwire Combating Patch Fatigue White Paper

Documents