With the sophistication and sheer volume of exploits targeting
major applications and operating systems, the speed of
assessment and deployment of security patches across
your complex IT infrastructure is key to mitigating
risks and remediating vulnerabilities. Here are the
Lumension-recommended steps to cure your
patch management headache.
Lumension® Guide toPatch Management Best Practices
WP-EN-04-17-12
April 2012
Lumension® Guide to Patch Management Best Practices
2
Introduction
Laying the Groundwork1. Discover Assets 3
2. Agent Maintenance 4
3. Classify Value and Risk 8
4. Establish Workflow and Groups 8
5. Identify Test Groups 12
6. Staff Training 14
Before Patch Tuesday7. Schedule Resources 15
8. Reserve Down-Time for Servers 15
9. Watch for Pre-Announcements 15
10. Confirm Reporting Up-to-Date 16
11. Deploy missing updates and prerequisites 17
On Patch Tuesday12. Study Vendor Information and Patch Tuesday Security Briefings 19
13. Prioritize Potential Patches 19
14. Change Control 20
15. Staged Testing 21
16. Installation of the Patches 22
After Patch Tuesday 17. Deployment History 24
18. Calculate Time to Deploy 25
19. Monitor for Compliance 26
20. Checks and Balances 27
21. Metrics Improvement 28
Lumension® Guide to Patch Management Best Practices
3
IntroductionPatch and vulnerability management is a core component of your risk mitigation strategy. It is the first and
last line of defense against existing and new exploits – laying the foundation from which your AV and other
security technologies work. As the sophistication and sheer volume of exploits targeting operating systems
and major applications increases, the speed of assessment and deployment of security patches is key to
mitigating risks and remediating vulnerabilities – and reducing costs.
In this best practice guide, we are going to take a deep dive into a best practice process for patch and vul-
nerability management, developed by Lumension over thousands of customer engagements. This process
– which is flexible and simple enough to be adapted into your environment – revolves around the well-known
monthly release of security updates from Microsoft known as Patch Tuesday, and includes:
» Laying the Groundwork for a Successful Patch Process
» Before Patch Tuesday
» On Patch Tuesday
» After Patch Tuesday
Every company’s Patch Management process is going to be a little bit different, but what’s important about these
best practices are: It’s a repeatable cycle. It’s based on calendar events – in this case Microsoft’s Patch Tuesday.
It’s iterative – it can be tweaked based on what’s learned from previous patch cycles. It’s measureable.
Documenting a process for the organization is really the best way to communicate the importance of patch-
ing your environment to the rest of the organization. In this best practice guide we chose to base the process
on the well-known Patch Tuesday event, but you can align your patch process with other recurring IT tasks
– with equally effective results – that works best for your organization.
Laying the GroundworkThis section is about gaining an understanding of the machines under management and preparing the Patch
and Remediation process. At a high level, this means identifying the systems to be managed, defining the
patch-roll out plan, and training the organization on the Patch and Remediation process.
1. Discover Assets Within Lumension® Endpoint Management and Security Suite (L.E.M.S.S.), identify all hardware
and sof tware on the network and categorize them
by plat form, applications, depar tment, etc.
Practical Steps: » In L.E.M.S.S., navigate to Discover > Assets
Lumension® Guide to Patch Management Best Practices
4
» Follow the Discover Assets wizard to set up an Asset Discovery job.
» As a best practice, administrators will want to schedule a more frequent recurring scan to identify
new endpoints that enter the network, then a less frequent scan as the number of machines under
management stabilizes
2. Agent Maintenance Ensure that all endpoint assets in the network have been fully installed with an automated patch solution.
Install new patch management agents where required, if this task has not yet been fully automated with
a group policy, login script or other technique. Identify offline agents and last contact date – either inside
L.E.M.S.S. or by running the Endpoint Check-in report in Lumension® Reporting Services (LRS), a free,
integrated add-on to L.E.M.S.S.
Lumension® Guide to Patch Management Best Practices
5
Practical Steps: » You can either set up a recurring Asset Scan or
an Asset Scan/Install Agents job.
• In L.E.M.S.S., navigate to Discover > Assets
or Discover > Assets and Install Agents and
follow the wizard to set up a recurring or one-
time job.
Lumension® Guide to Patch Management Best Practices
6
» We also recommend verifying agent availability and last check-in via LRS:
• Run the Asset Management report “Endpoint Check-in” in LRS.
• Select the desired date of last endpoint check-in (“Last Contact Date on or before”) – typically your
current date.
• The report displays the list of endpoints that have not checked-in with the server in a given
timeframe.
• Ensure that agent communication is established with all the endpoints in your environment.
• Review endpoints that have not checked in recently and verify which endpoints need follow-up or
attention prior to rolling out updates (training computers that are off vs. sales guy in field that needs
to check in)
Lumension® Guide to Patch Management Best Practices
7
» It may also be useful to verify the agent versions and operating systems of your endpoints through
LRS, especially if you are planning to perform an upgrade to a newer version of L.E.M.S.S.:
• Run the Operational Report “Agent Version and Operating System Distribution” in LRS
• The report displays the mix of agent versions and operating systems in the endpoint environment,
along with a detailed endpoint count.
• Ensure that all desired endpoints are listed, have the expected agent version(s) and communicate
properly.
Lumension® Guide to Patch Management Best Practices
8
3. Classify Value and Risk Determine which systems are most critical to protect based on the assets housed and/or the function they
provide. Define the level of risk by criticality of system and how prone it is to attack.
Practical Steps: » Review your network topology and classify your assets by level of criticality.
4. Establish Workflow and GroupsDetermine ownership, permissions needed and responsibilities for threat identification, testing and reme-
diation across security, IT and business units. Define correlating system groups. L.E.M.S.S. will predefine
system groups based on desktops, servers, physical or virtual hardware, as well as operating systems. If
more granular management is required, IT managers can create additional groups based on specific re-
quirements, e.g. if servers are internet-facing, they may be grouped as high-risk but also as limited down-
time. Use RBAC controls and set up permissions for desktop patch admin, server patch admin, as well as
individuals who have reporting access only.
Practical Steps: » Determine system ownership, uptime requirements,
and patch windows for these machines. Define the
patch cycle for different managed systems.
» Define users and roles within your organization and
who needs access to which systems.
• On the Tools > Users and Roles page, select the
Roles tab and either select and assign existing
role(s) or create new roles.
Lumension® Guide to Patch Management Best Practices
9
• Next, assign users to the selected role(s) from the Users tab.
» Set up your categorized assets in custom groups in L.E.M.S.S.
• On the Manage Groups page, click on Custom Groups.
Navigate to View in the upper right corner and select Group
Membership to create a custom group.
• Navigate to View in the upper right corner and select Endpoint
Membership to assign endpoints to that group.
• Click on Manage to assign endpoints to that group.
Lumension® Guide to Patch Management Best Practices
10
» Set Hours of Operation (HOP) for managed endpoints that
require a specific patch window.
• On the Manage > Agent Policy Sets page, create a new agent
policy and define the hours of operation.
• Then, apply that policy to specific endpoints or groups that
require this HOP policy.
Lumension® Guide to Patch Management Best Practices
11
» For machines managed over the WAN, it is recommended to set up a caching proxy per remote
location to cache the package content.
• Deploy “Lumension Caching Proxy 2.7 for Windows” to a target machine in the remote location
• Create Agent Policy and set FastPath Servers – Both Interval and Define Servers
• Manage > Agent Policy Sets > Select Create and Save when completed
• Apply Agent Policy to your custom group
• Manage Groups > Right-click on the group > Select Policies > Select Add > Select Agent Policy and
click Save
◦ Note: Policy will not set until the next check-in to L.E.M.S.S.
• For more information on setting up a caching proxy please review the following resources:
◦ Best Practices Fast Path: KB article 523
◦ Distribution Point (PDP) Does not Cache Large Deployment: KB article 231
Lumension® Guide to Patch Management Best Practices
12
5. Identify Test GroupsBuild a representative sample set of each type of machine based on steps 2 (Agent Maintenance) and 3
(Classify Value and Risk), in readiness for patch testing step 15 (Staged Testing). Make sure your test group
includes a representative sample of platforms under management and includes a representative sample of
applications in the environment, especially machines that have custom, in-house developed applications.
As a best practice, at least one machine from each major group in the organization should be included in a
test group.
Practical Steps: » Once test groups have been identified, create custom groups for
those test groups.
• On the Manage Groups page, click on Custom Groups.
• Navigate to View in the upper right corner and select Group
Membership to create a custom group.
Lumension® Guide to Patch Management Best Practices
13
• Navigate to View in the upper right corner and select Endpoint Membership, then click on Manage,
select the desired endpoints and click Assign to assign these endpoints to that group.
Lumension® Guide to Patch Management Best Practices
14
6. Staff Training Train applicable staff on vulnerability monitoring and remediation techniques. At a minimum, administrators
responsible for deploying Patch updates need to be trained in the Patch and Remediation application. As a
best practice, there should be an internal resource for all employees to learn more about why it is important
to keep machines in the organization fully patched.
Practical Steps: » Use Lumension® Learning resources to help build your internal staff training.
Continued »
Lumension® Guide to Patch Management Best Practices
15
Before Patch TuesdayThis section is about preparing the environment for the monthly patch deployment, including industry re-
search on what is expected to be released by Microsoft and other application vendors and assess the impact
of those planned releases to your managed machines.
7. Schedule Resources Allocate IT resources for Patch Tuesday while also integrating additional patch release schedules from third-
party software, such as Adobe, Apple (ad hoc), Java and so forth. In addition, review the patching needs of
any internally-developed applications and/or custom patches and consider deploying these patches as part
of the monthly patch cycle.
8. Reserve Down-Time for Servers Reserve time slots to be able to deploy patch updates to any mission critical servers within 72 hours of the
Patch Tuesday release.
9. Watch for Pre-Announcements Monitor security sites for pre-announcements of patches and discussion of vulnerabilities and possible
zero-day exploits that they may address from sources such as Lumension Endpoint Intelligence Center
(LEIC), Microsoft Security Response Center (MSRC), SANS Internet Storm Center, National Vulnerability
Database (NVD), etc.
Practical Steps: » In addition to reviewing vendor sites, we recommend setting up email notifications within L.E.M.S.S. to
receive an email when new vulnerabilities have been replicated to L.E.M.S.S.
Lumension® Guide to Patch Management Best Practices
16
10. Confirm Reporting Up-to-Date Review last deployment reports via Lumension Reporting Services (LRS) and make sure all computers
are being regularly scanned. Validate the L.E.M.S.S. application server is actively communicating with the
global subscription service (GSS).
Practical Steps: » To confirm recent deployments and ongoing scanning in LRS:
• Run the operational report “Deployment Detail”
• Select the group(s) that you are monitoring
• Review success/failure results (Patched and Complete %)
» To confirm communication with GSS in L.E.M.S.S.:
Lumension® Guide to Patch Management Best Practices
17
• Go to the Tools > Subscription Updates page.
• Confirm that the “Successful” column shows “true”, indicating successful replication.
• If “false” is shown in any of the rows, troubleshoot to ensure replication.
11. Deploy missing updates and prerequisites Determine if your software is fully updated or if there are any missing Service Packs, hotfixes or rollups
from prior months that are still outstanding. Remember that some patches won’t install if you have miss-
ing prerequisites. Check that each machine in the defined group has received the latest Service Pack
or update needed.
Practical Steps: » To verify if your software is fully
updated:
• In L.E.M.S.S., go to the Review
> Software > Service Packs
(Software Installers / Updates)
page and investigate any
missing service packs, hotfixes
or rollups from prior months that
Lumension® Guide to Patch Management Best Practices
18
are still outstanding.
» Deploy missing updates:
• Deploy any missing updates directly from the page above by selecting the missing patches and
clicking on Deploy.
Lumension® Guide to Patch Management Best Practices
19
On Patch TuesdayThis section outlines the steps to prioritize the Security Patches released by Microsoft and other application
vendors and to deploy those patches out to the machines managed in your environment.
12. Study Vendor Information and Patch Tuesday Security Briefings Microsoft and other vendors provide webinars, email alerts and comprehensive online information on all new Patch
Tuesday updates.
Lumension offers a monthly Patch Tuesday Security Briefing as well as other patching guidance on the
Lumension® Optimal Security Blog, the Lumension® Patch Tuesday Alerts webpage and in the Patch Tuesday
newsletter.
Important information to consider when understanding the impact of Patch Tuesday on your environment includes:
» What is the bulletin severity rating?
» Is the vulnerability known / publicly disclosed at the time of release?
» Does the vendor know of any active exploits at the time of release?
» How easily can the vulnerability be exploited once the bulletin is been released?
13. Prioritize Potential Patches With the vendor information gathered in step 12 (Study Vendor Information and Patch Tuesday Security Brief-
ings), use patch impact (Critical, Important, etc.), asset risk and value to prioritize your systems for patch testing
and deployment. Understand the applicability and impact of deploying these patches to your environment, espe-
cially critical machines. When making this assessment, consider:
1. Threat Level;
2. Known Active Exploits in the Wild;
3. Risk of Compromise;
4. Consequences of Compromise.
Lumension® Guide to Patch Management Best Practices
20
Practical Steps: » To review the released Patch Tuesday patches and their applicability in your endpoint
environment, we recommend you use LRS and run the report “Patch Release by Vendor”
• The report provides a high-level overview of the applicability of the released bulletins to your
managed endpoints and groups. It reflects the severity of and expected workload for that month’s
Patch Tuesday release and the organization’s patch status.
• When choosing your parameters, we recommend selecting all the criticalities and the first day of the
month. The report will then display the number of vulnerability patches and content released by each
vendor in the top section and the vulnerability patches and content applicable to your environment in
the “Applicable” section directly below.
14. Change Control Follow any internal planning and approval processes for agreeing on patch deployment. This may include
following different processes for the server side than for the desktop side. Some organizations will have dif-
ferent change control processes for desktop machines than for server machines due to high uptime require-
ments for servers or to limit reboot interruptions for desktop users.
Lumension® Guide to Patch Management Best Practices
21
15. Staged Testing Testing each patch is vital; automated deployment is very risky and not advised. Be certain to test the patch
in each environment of your previously defined groups and deploy the patches in phases. In addition, be-
fore remediation, and especially if there is a lack of time or resources to perform a test on the patch before
deploying it on a production system, there is great benefit in joining patch user forums and learning what
experiences others have had in installing or using the patch.
Practical Steps: » Deploy applicable bulletins to test groups configured in step 5 (Identify Test Groups) above.
» Ensure successful deployment before rollout to additional groups in the environment.
» Pay special attention to impact to custom-developed, internal applications, especially when deploying
Java updates.
Lumension® Guide to Patch Management Best Practices
22
16. Installation of the Patches Stage deployments by system groups and prioritization. Start with smaller, low-risk groups, and validate that
no problems occur, and then work your way to larger and higher-risk areas of the network. As a best prac-
tice, and especially if your servers have a limited maintenance window, it is recommended to cache all the
patch content before deployment. If deployments are scheduled off-hours, take advantage of Wake-on-LAN
settings to wake up any powered-down endpoints and ensure that they receive the content.
Practical Steps: » In L.E.M.S.S., go to the Review >
Vulnerabilities > New Vulnerabilities page,
select content applicable to your environment
and cache the packages associated with
those binaries by selecting the bulletins and
clicking on the “Update Cache” button.
Lumension® Guide to Patch Management Best Practices
23
» Go to the Manage Groups page under the Vulnerabilities view
and filter for new critical bulletins. Deploy bulletins that are
applicable to that target group.
» After successful deployment, move on to other groups in your
patching plan.
Lumension® Guide to Patch Management Best Practices
24
After Patch TuesdayThis section is about assessing the success of the Patch and Remediation deployments in your environ-
ment.
17. Deployment History Maintain accurate records of all patches deployed. Validate that any necessary reboot(s) occurred and/or
that your endpoints don’t require a reboot.
Practical Steps: » To confirm recent deployments in LRS:
• Run the operational report “Deployment Detail”
• Select the group(s) that you are monitoring
• Review success/failure results (Patched and Complete %)
Lumension® Guide to Patch Management Best Practices
25
18. Calculate Time to Deploy Measure how long it takes to get all servers, desktops and laptops fully patched in your organization. This is
a great metric to measure against. Remain vigilant for laptops and VPN-connected systems that may con-
nect days (or weeks) after the initial deployment.
Fully patched and time to deploy success metrics may be defined differently for different organizations de-
pending on the mobility of the machines being managed, how often the machines are online, or the type of
machines under management, such as desktop or server.
Practical Steps: » To strategize and organize patch deployments to the appropriate endpoints and endpoint groups, use
LRS as follows:
• Run the report “Patch Tuesday Monitoring Report”
• Select the group(s) that you are monitoring
• The report provides a summary of the patch status for a selected group of machines for the critical
patches released in the selected Patch Tuesday cycle.
• Set the Auto Refresh parameter to monitor the progress of deployments on endpoints in near-real-time.
Lumension® Guide to Patch Management Best Practices
26
19. Monitor for Compliance Make certain that new or rebuilt systems are “base-lined” for their appropriate systems group. Monitor for
removal of patches. Create or update an existing mandatory baseline for future deployments.
Practical Steps: » Upon successful deployment of bulletin content, add bulletins to
mandatory baseline policies.
• Go the Manage > Groups page
• Select the Mandatory Baseline View
• Click on the “Manage” button
• Select bulletins to add to the mandatory baseline
• Click on the “Assign” button
Lumension® Guide to Patch Management Best Practices
27
20. Checks and Balances Review the Effectiveness of Patch Tuesday Remediations report in LRS to validate the deployment.
Practical Steps: » To review the patch progress and effectiveness of deploying Patch Tuesday remediations and to
understand the security posture and vulnerability compliance of the enterprise for Patch Tuesday
patches released by Microsoft for the selected patch cycle, use LRS as follows:
• Run the report “Effectiveness of Patch Tuesday Remediations Report”
• Select the group(s) that you are monitoring
» The report provides an executive overview of the Patch Tuesday deployment status while also allowing
drill-throughs to operational endpoint details.
Lumension® Guide to Patch Management Best Practices
28
21. Metrics Improvement Modify system settings, distribution parameters and so forth to further optimize the system for next month’s
updates. WAN optimization, polling frequency and minimizing the patches being detected can all help further
optimize performance. Look for computers that did not receive updates at all or those that took unusually
long to receive updates.
Practical Steps: » Go the Manage > Groups page
» Identify any endpoints that are offline and/or have not been remediated.
» Troubleshoot the endpoints to determine why endpoints were not updated and modify deployments
accordingly
Lumension® Guide to Patch Management Best Practices
29
About Lumension Security, Inc.Lumension Security, Inc., a global leader in endpoint manage-
ment and security, develops, integrates and markets security
software solutions that help businesses protect their vital infor-
mation and manage critical risk across network and endpoint
assets. Lumension enables more than 5,100 customers world-
wide to achieve optimal security and IT success by delivering a
proven and award-winning solution portfolio that includes Vul-
nerability Management, Endpoint Protection, Data Protection,
Antivirus and Reporting and Compliance offerings. Lumension
is known for providing world-class customer support and servic-
es 24x7, 365 days a year. Headquartered in Scottsdale, Arizona,
Lumension has operations worldwide, including Texas, Florida,
Washington D.C., Ireland, Luxembourg, Singapore, the United
Kingdom, and Australia. Lumension: IT Secured. Success Opti-
mized.™ More information can be found at www.lumension.com.
Lumension, Lumension Patch and Remediation, Lumen-
sion Vulnerability Management, “IT Secured. Success Op-
timized.”, and the Lumension logo are trademarks or reg-
istered trademarks of Lumension Security, Inc. All other
trademarks are the property of their respective owners.
Global Headquarters
8660 East Hartford Drive, Suite 300
Scottsdale, AZ 85255 USA
phone: +1.480.970.1025
fax: +1.480.970.6323
www.lumension.comVulnerability Management | Endpoint Protection | Data Protection | Compliance and IT Risk Management