Building and Maintaining a Successful Vulnerability and Patch Management

Program

Presented to Western Regional Educause

April 2, 2008

2

Naval Postgraduate School Established in Annapolis in 1909

Moved to Monterey in 1951

3

Academic Programs

Academic Schools• Graduate School of Business and Public Policy• Graduate School of Engineering and Applied Sciences• Graduate School of Operations and Information Sciences• School of International Graduate Studies

Research Institutes• Cebrowski Institute for Information Innovation and

Superiority• Wayne E. Meyer Institute of Systems Engineering• Modeling Virtual Environments and Simulation Institute

4

NPS Population

• 1700 resident students– Less than 50% are Navy– 30% international officers – Remainder Air Force, Army, US Marine Corps,

US Coast Guard, and civilians

• 1700 faculty and staff

• 880 distance learning students

5

Accreditation

Regional

• Western Association of Schools and Colleges (WASC)

Programmatic

• ABET (some programs)

• AACSB

• NASPAA

6

Research-related Organizations

• CSU-Monterey Bay

• Monterey Peninsula College

• Monterey Institute of International Studies

• Hopkins Marine Station – Stanford University

• Monterey Bay Education Science and Technology Center, University of California

• National Undersea Research Program (NOAA)

• Moss Landing Marine Lab (CSU)

• University of California Sea Grant Extension

• National Weather Service

• Monterey Bay National Marine Sanctuary, NOAA Pacific Fisheries Environmental Lab, NOAA

• Defense Language Institute• Fleet Numerical Meteorology

and Oceanography Center• Monterey Bay Aquarium

Research Institute• Naval Research Laboratory• Defense Manpower and Data

Center• Naval Postgraduate School• Monterey College of Law• Chapman College• Golden Gate University

7

8

NPS Systems

• Multiple Networks – 7 and growing

• Web services– Extranet: 326MB per day; 55M hits per day– Intranet: 786MB per day; 88M hits per day

• 5425 systems (computer, printer, scanner) connected to NPS network

• 6500 campus computer accounts

• 800 software applications

• 3150 active phone lines

9

Definition: Information Assurance

• “Measures that protect and defend information and information systems by ensuring their confidentiality, integrity, availability, non-repudiation and authentication. This includes providing for restoration of information systems by incorporating protection, detection and reaction capabilities.” (DoDD 8500.1)

10

IA Attributes

• Confidentiality – Disclosure of Information• Integrity- Unauthorized modification of data• Availability- timely, reliable access to data• Non-repudiation- proof of delivery and

identity• Authentication – proof of identity

11

IA Tools

• Computer Network Defense (CND)– Vulnerability Management

• Alerts, Bulletins, Tech. Advisories – Navy’s IAVM program• Network Vulnerability Detection Tools- Retina/REM• Patch Management Tools

– LANDesk– WSUS

– Antivirus Tools• Centrally managed Symantec Antivirus• Barracuda Spam Filter

– Network Access Control• Bradford Network Appliance

– Intrusion Detection• Snort• StealthWatch

12

Motivators

Why did NPS create a Vulnerability and Patch Management Program?• Attacks

• Welchia and Blaster – 2003

• Other attacks have followed and continue to pose a significant threat.

• Mandates• DoD/Navy - Information Assurance Vulnerability

Management (IAVM) Process – CJCSM 6510.01

• Best practices

13

Vulnerabilities over the last 10 years

Reference: http://nvd.nist.gov/statistics.cfm

0

1000

2000

3000

4000

5000

6000

7000

8000

Year

# of Vuln

se High Risk

Medium Risk

Low Risk

14

Scoping the Problem

• Our EDU network poses the biggest challenge– Largest network at NPS– Transient systems– Many locally administered systems

• New vulnerabilities emerge daily.

• A strategy is needed that protects not only servers and network services, but workstations as well.

15

How to manage?

• Effectively managing this problem requires NPS to:– Maintain awareness of our vulnerability posture.

• Scan regularly to ensure compliance

• Obtain Local access to all NPS assets

• Update vulnerability audits.

– communicate vulnerabilities/remediation to system owners.

– Close the loop (feedback and documentation).

16

Where we were – May 2007

• In-house system bridged gap between Foundstone and Remedy– One remedy ticket = One vulnerability on one system– Vulnerability Technician did not work directly with system owners.– Feedback system was almost non-existant (duplicate tickets, false

positives)

• Vulnerability scanner was not properly configured.

• We reached a critical decision point– Foundstone License was soon to expire. – Given the expense to continue with Foundstone, we needed a more

cost effective solution.

17

Leveraging our unique position

• Is NPS a University or a Naval Command?– We are both!

• As a Navy Command, Retina/REM were available to us for free!

• Rather than face a coverage gap, we began learning Retina/REM.

18

Configuration

• One dedicated system allows us to scan for vulnerabilities.– 1 Dell PE1950 (Windows 2003)– eEye Retina Vulnerability Scanner– eEye Retina Enterprise Management (REM)

Console– SQL Server 2000

19

Developing a scan schedule

• We recommend a more frequent scan schedule than once a month.– Maintain better awareness of your vulnerability posture.

– NPS scans our class B address space weekly.

• Avoid scanning a large IP space in one session.– Find a scan schedule that promotes easy troubleshooting.

– We scan by building, and scan 2-3 buildings per day.

– Scan DMZ when load is lowest.

20

Developing a Scan Schedule (cont)

• Maximize your coverage– Do your users powerdown at night?

• Many of ours do.

– We scan DHCP zones during the day and static IP ranges at night.

21

Host-based Vulnerability Scans

• Requires local access to the machine– For windows (local or domain admin)– Other OSes (SSH account)

• Vulnerability audits are usually dependent upon examination of registry settings, file version, or package.

• At NPS – the majority of our systems are Windows-based and belong to our Windows domain – to those machines access is easy… but…

22

Host-based Vulnerability Scans (cont)

• Access to Researchers’ systems often presents a challenge.

• We have overcome these challenges by:– Establishing collegial relationships with our

researchers – we try not to be “the man behind the curtain”.

– Establishing a configuration management process that requires systems be rid of medium and high risk vulnerabilities.

23

Knowing who to contact

• Scans are of little value if the results are not shared with system owners for remediation.

• Determining a system owner for every system can be challenging and difficult to keep updated.– We use SQL triggers to automate the

discovery/assignment of enterprise workstations.– Other systems are matched to an owner once a

quarter.

24

Communication/Remediation

• We choose different remediation Strategy dependent upon asset type:– Enterprise Servers– Enterprise Workstations– Researchers / non-Enterprise administered systems

25

Enterprise Servers

• Administered by Server Management and Business Solutions Group– Server management applies OS patches– BSG applies application specific patches

• Patching is performed as a part of maintenance – coincides with Patch Tuesdays.

26

Enterprise Workstations

• Several tools exist which aide patch deployment to our workstations– Group Policy– LANDesk

• Patch Management

• Remote Control

• Inventory Scanner

27

Enterprise Workstations (cont)

– WSUS• Solves problem where LANDesk MS updates caused

auto-reboot.

– Remote Desktop to the machines• Some failed pushes are easy to fix – but cannot be

deployed to all systems.

– When all else fails – re-image the machine.

28

Non-enterprise administered systems

• It is the responsibility of the administrator to patch their system.

• But, what about people who just won’t patch?– We deny their operational requests until our

security requests have been met (most common request type is firewall related).

– We avoid threatening system disconnect unless absolutely necessary.

29

Closing the loop

• Document the feedback you receive.– Sometimes recommended fixes fail.– Occasionally false positives are reported.

• Documenting this information provides you with a clearer picture of what your actual vulnerability posture is.– We have created a separate database which

contains NPS specific fix information and false positives.

30

Where are we now? April 2008

• For the time being, remedy is not a part of the vulnerability management process – all vulnerabilities are tracked within Retina/REM.

• Local access has been obtained on approximately 90% of all network devices.

• Retina/REM appears to be a good fit for NPS and we have no licensing worries for the foreseeable future.

• Much closer relationships between Network Security and the rest of campus.

31

Summary

• Significant milestones in our IA program– Deploying a vulnerability scanner (2003)

• Brought visibility to the gap between patched systems an those at risk

– Deploying LanDesk “push” (2004)• Shortened timeline between vulnerability and patch

• Reduced sysadmin time accomplishing patching

• Immediately saw a drop in virus infections.

– Adding the WSUS or “pull” component to patching• Reduced the patch time again.

32

Future Work

Next step:

• Bradford network appliance – compliance appliance

Challenges:

• Scanning is limited to the access rights to each system.

• Research networks protected by a Firewall or behind a separate gateway are no visible without admin rights.

33

Questions?

Contact Information

Terri Brutzman

tbrutzman@nps.edu

Jason Cullum

jcullum@nps.edu