2005 FNAL Computer Security Peer Review andSelf Assessment

Networking –Current Status

FNAL Computer Security

Peer Review

Phil DeMar

March 22, 2005

2005 FNAL Computer Security Peer Review andSelf Assessment

Outline

• FNAL Network Overview• Perimeter Controls & Tools• Internal Network Controls & Tools• Network Critical System*

* Termed ‘Major Application’ in the new CSPP under development

2005 FNAL Computer Security Peer Review andSelf Assessment

FNAL Network Overview

• A centrally-managed campus-wide network– Restricted central services (FNAL Policy on Computing…):

• Routing & bridging– Separately admin’ed AD network grandfathered in policy

• Address, name, & time services• Exemptions rarely granted

• Architecture based on work group model:– Affinity groups w/ their own dedicated LANs

• Based on experiment, organization, geography• Mostly physical LANs; a few vLANs w/ trunking• Detachable from campus network, if necessary

2005 FNAL Computer Security Peer Review andSelf Assessment

Core Network Facilities & Essential Network Services

• Core network facilities:– FCC collapsed backbone– WH core router – Border router

• Essential network services– Name service– Address allocation services

• Static addresses• DHCP service

– Time service– VPN service

ADLAN

Site 38

Off-Site[Internet]

FCC Offices

FCCComputingResources

WH OfficeLANs

FCCCollapsedBackbone

Switch/Router

WHCollapsedBackbone

Switch/Router

SiteBorderRouter

622Mb/s

TD/IC

Village

CDF

D0

SDSS

MiniBoone

CMS

FTArea

MINOS

2005 FNAL Computer Security Peer Review andSelf Assessment

Internal Network

• A single, general network access zone:– No customized access restrictions for individual

work groups

• Critical System* LANs:– Networks supporting collection of related systems

who’s compromise could seriously impact the laboratory’s science programmatic operations

• Designated by the CSExec– Individual plans, typically with customized network

access & protections

* Termed ‘Major Applications’ in the new CSPP under development

2005 FNAL Computer Security Peer Review andSelf Assessment

Critical Systems (aka Major Applications)

Critical System Network Access Protection Operational Management

Accelerator controls network

Firewall w/ VPN AD

Business systems network

Firewall w/ border router ACLs

BSS

CDF Online network Router ACLs CD Networking

D0 Online network Router ACLs CD Networking

Network Firewall w/ VPN CD Networking

Authentication systems Host-based protections CD Security Team

MetaSys building controls

Isolated vLAN w/ Firewall & VPN

CD Networking

2005 FNAL Computer Security Peer Review andSelf Assessment

Off-site Network Access (I)

• Current site perimeter access policy:– Open inbound access with a few protections:

– Netbios (TCP ports 135, 137 – 139, 445)

– SunRPC* (TCP/UDP port 111)

– Web Servers (TCP ports 80, 443)

» Exemption process available– SMTP (TCP port 25) except for facility mail servers– DNS (TCP port 53) except for facility DNS servers– SNMP* (UDP port 161)

– Open outbound access with minimal restrictions:– IRC (TCP default ports 6667-6669)

* also blocked outbound

2005 FNAL Computer Security Peer Review andSelf Assessment

Off-site Network Access (II)

• An alternate very high bandwidth offsite path now in place:– Via dark fiber connection

to StarLight– Intended use – high

impact data movement– Redundant path for

production offsite link

StarLight

ESnet

FNALBorderRouter

ESnetRouter

CERN

SD1648 SMCommunication Subsystem Shelf

SD1648 SMCommunication Subsystem Shelf

FNALDWDM

gear

FNALDWDM

gear

Onsite

Off-site

FNALDark Fiber

to StarLight FNAL

FNAL6500

@StarLight

FNALStarLight

Router

622

Mb

/s

FNAL

Network

Abilene

GeneralInternet

Production Network (10GE)

StarLight 10GE Path

Production Network (1GE)

(NBC Bldg)

UltraScience

Net

UltraLight UKLight

CAnet4

• Default-deny inbound access w/ ACL exceptions- Redundant path traffic goes thru border router

2005 FNAL Computer Security Peer Review andSelf Assessment

Border router flow data

• Logs all off-site network connections– Useful for investigating computer security incidents

• Generates daily & hourly Top 20 reports on:– Top talkers, top listeners, top conversations– Breakouts by number of flows, bytes, or packets– Unusual traffic patterns

• Large numbers of offsite hosts contacted• Large amounts of data transferred• Unusual consumption of network resources

• Now collecting flow data on internal routers

2005 FNAL Computer Security Peer Review andSelf Assessment

AutoBlocker

• Based on quasi-realtime flow record analysis• Blocks “greedy” users (perceived as scanners…)

– Outbound or inbound scanners– Address-based scans or port-based scans– Automated unblocked after behavior stops

• Proven useful in blocking infected local systems– Alerts for out-of-ordinary flow patterns– Occasionally blocks “greedy”, but legit apps

• Mostly nuisance apps, such as P2P, games…• New version should minimize those disruptions

2005 FNAL Computer Security Peer Review andSelf Assessment

Telecommuting Access

• VPN service available– Encrypted tunnel capability to the Laboratory– Assigns virtual local Fermilab address– Allows site access to protocols blocked at Border – Must use Cisco VPN client & FNAL-provided profile

• Standard configuration forced onto users• Split-tunneling restricts tunnel data flows to

FNAL-related traffic

• Dial-up: – Uses Radius authentication – Limited to on-site access only

2005 FNAL Computer Security Peer Review andSelf Assessment

Node Registration

• System registration is required to be granted a usable address on the facility network

– Permanent registration in MISCOMP database for either static or automatic DHCP address:

• Key information required: MACs, sysadmin – Temporary DHCP service available for transient

users not registered in MISCOMP:• Provides DHCP lease good for rest of the day• Re-registration necessary every day

– 5 day limit per 30 day period

2005 FNAL Computer Security Peer Review andSelf Assessment

Node Registration Monitoring

• Currently checking for unregistered static IP systems via simple ping utility– Doesn’t work so well with software firewalls…– Not useful at all for DHCP subnets

• Have developed a prototype to check ARP table information for proper registration:– Verifies IP/MAC tuples observed on network

correlates to registered MISCOMP information– 2-3 months away from being production use tool

2005 FNAL Computer Security Peer Review andSelf Assessment

Node Tracking

• Router ARP & switch FDB tables gathered every 20 minutes

• Node Locator utility manipulates ARP & switch FDB data to:– Identify location of IP or MAC address on the network– Provide switch port information for the system– Provide traffic utilization for switch port

2005 FNAL Computer Security Peer Review andSelf Assessment

Infrastructure Monitoring & Response

• Network management stations monitor status of network devices & servers:– Device and server reachability & uptime monitored– Service response (DNS, DHCP, & NTP) also monitored

• Off-hours support:– Automated device/service paging during off-hours

• Two people on call at all times– Escalation procedures to Section, Dept., then Division Heads

– User problem reporting via HelpDesk off-hours service

2005 FNAL Computer Security Peer Review andSelf Assessment

Wireless Support

• WLANs cover major work areas of the site• Not treated differently than wired access

– Broadcast SSID– Authentication not required– Encryption not required– Node registration required

• But tightening down on vulnerabilities:– Migrating to wireless subnets (70% complete)– Rogue detection based on Cisco Wireless LAN

Solution Engine (WLSE) & war drives– Site border scans checking for offsite bleed-thru

2005 FNAL Computer Security Peer Review andSelf Assessment

The Network Critical System*• Network Critical System*:

– “Those parts or components of the network necessary to sustain the operation of the general facility network as a functioning entity”

– “Those parts or components of the network that are an integral part of an activity or operation whose compromise could seriously impact the Laboratory’s science programmatic operations”

• CSPP Network Critical System* Plan:– Protects network critical system components themselves– Current plan is version 2; revised 4/7/2003

• Next revision due in line with new CSPP * also known as Major Application

2005 FNAL Computer Security Peer Review andSelf Assessment

Components

• Facility core network devices:– FCC & WH core routers

– Border router

• Servers for essential network services:– DNS, DHCP, NTP

• Run-II experiment network “core” routers– Off-line network core router

– On-line network router

2005 FNAL Computer Security Peer Review andSelf Assessment

Network Management LAN

• Isolated LAN to controlled access to:

– Network Critical System* core & border routers

• Also other major network devices in the FCC & WH– Enterprise DNS/DHCP server & NTP time sources

• Misc other servers (ie., Radius server…)

• Used for:– Remote console access & configuration management– O/S upgrades– snmp/statistical data collection

* also known as Major Application

2005 FNAL Computer Security Peer Review andSelf Assessment

Network Mgmt LAN Figure

Cisco PIX

WH FCC

WH FCC

EnterpriseDNS/DHCP

server

GPStime

servers

X

NetworkMgmt

system

NetworkManagement

LAN

General Facility

LAN

GPStime

servers

VPNConc.

PIX Firewall

BorderRtr

<Off-Site>

RadiusServer

DNSServer

DHCPServer

DNSServer

DHCPServer

2005 FNAL Computer Security Peer Review andSelf Assessment

Network Mgmt LAN (cont)

• Physically separate from campus LAN– Dedicated fiber; dedicated switches

• Firewall protected w/ default deny inbound– Exceptions for necessary server traffic & monitoring:

• DNS/DHCP traffic• NTP traffic w/ stratum-2 NTP servers (ie., routers)

• Remote terminal access via VPN• Network management system dual-homed to

general LAN & network management LAN

2005 FNAL Computer Security Peer Review andSelf Assessment

Questions…

?