37
CIS 5373 – Systems Security Topic 3.1: OS Security – Basics of secure design Endadul Hoque
CIS 5373 – Systems SecurityTopic 3.1: OS Security – Basics of secure design
Endadul Hoque
Slide Acknowledgment
• Contents are based on slides from Ninghui Li (Purdue), John Mitchell (Stanford), Dan Boneh(Stanford)
2
What security goals does an operating system provide?
3
• Originally: time-sharing computers: enabling multiple users to securely share a computer
– Separation and sharing of processes, memory, files, devices, etc.
• What is the threat model?
– Users may be malicious, users have terminal access to computers, software may be malicious/buggy, and so on
• Security mechanisms
– Memory protection
– Processor modes
– User authentication
– File access control
• More recent past and present: Networked desktop computers: ensure secure operation in networked environment
• New threat?
– Attackers coming from the network. Network-facing programs on computers may be buggy. Users may be hurt via online communication.
• Security mechanisms
– Authentication; Access Control
– Secure Communication (using cryptography)
– Logging & Auditing
– Intrusion Prevention and Detection
– Recovery 4
What security goals does an operating system provide?
• Present and near future: mobile computing devices
• New threat?
– Apps (programs) may be malicious or questionable.
– More tightly connected with personal life of the owner.
• Security mechanisms?
– Isolation of each app.
– Help users assess risks of apps.
– Risk communication. 5
What security goals does an operating system provide?
Principles of Secure Design
• Compartmentalization– Isolation
– Principle of least privilege
• Defense in depth– Use more than one security mechanism
– Secure the weakest link
– Fail securely
• Keep it simple
6
Principles of Least Privilege
• Principle of Least Privilege
– A system module should only have the minimal privileges needed for its intended purposes
• What’s a privilege?
– Ability to access or modify a resource
• Assumes compartmentalization and isolation
– Separate the system into isolated compartments
– Limit interaction between compartments
7
Monolithic design
8
System
Network
User input
File system
Network
User device
File system
Monolithic design
9
System
Network
User input
File system
Network
User device
File system
Monolithic design
10
System
Network
User input
File system
Network
User device
File system
Component design
11
Network
User input
File system
Network
User display
File system
Component design
12
Network
User input
File system
Network
User display
File system
Component design
13
Network
User input
File system
Network
User display
File system
Example: Mail Agent
• Requirements
– Receive and send email over external network
– Place incoming email into local user inbox files
• Sendmail
– Traditional Unix
– Monolithic design
– Historical source of many vulnerabilities
• Qmail
– Compartmentalized design
14
Memory Protection: Access Control to Memory
• Ensures that one user’s process cannot access other’s memory– fence
– relocation
– base/bounds register
– segmentation
– paging
– …
• Operating system and user processes need to have different privileges
15Reading: http://flylib.com/books/en/4.270.1.46/1/
CPU modes (aka Processor modes or privilege)
• System mode (privileged mode, master mode, supervisor mode, kernel mode)
– Can execute any instruction
– Can access any memory locations, e.g., accessing hardware devices,
– Can enable and disable interrupts,
– Can change privileged processor state,
– Can access memory management units,
– Can modify registers for various descriptor tables
16
Reading: http://en.wikipedia.org/wiki/CPU_modes
CPU modes (aka Processor modes or privilege)
• User mode
– Access to memory is limited,
– Cannot execute some instructions
– Cannot disable interrupts,
– Cannot change arbitrary processor state,
– Cannot access memory management units
• Transition from user mode to system mode can only happen via well defined entry points, i.e., through system calls
17
Reading: http://en.wikipedia.org/wiki/CPU_modes
System Calls
• Guarded gates from user mode (space, land) into kernel mode (space, land)– use a special CPU instruction (often an
interruption) to transfer control to predefined entry point in more privileged code
– allows the more privileged code to specify where it will be entered as well as important processor state at the time of entry
– the higher privileged code, by examining processor state set by the less privileged code and/or its stack, determines what is being requested and whether to allow it
18
Reading: http://en.wikipedia.org/wiki/System_call
Kernel space vs User space
• Part of the OS runs in the kernel model
– known as the OS kernel
• Other parts of the OS run in the user mode, including service programs (daemon programs), user applications, etc.
– they run as processes
– they form the user space (or the user land)
• Why is root user more powerful than normal users?
19
Privilege Levels
• Security is often achieved by running control/protection code at a higher privilege level
• Components running at the same level can be isolated by a higher-privilege component
• If attack and defense are at the same level, then it is an arms’ race and there can be no guarantee
20
ISOLATION – THE CONFINEMENT PRINCIPLE
OS Security
21
Running untrusted code
• We often need to run buggy/unstrusted code:
– programs from untrusted Internet sites:• apps, extensions, plug-ins, codecs for media player
– exposed applications: pdf viewers, outlook
– legacy daemons: sendmail, bind
– Honeypots
• Goal: if application “misbehaves” ⇒ kill it
22
Approach: Confinement
Confinement: ensure misbehaving app cannot harm rest of system
Can be implemented at many levels:
– Hardware: run application on isolated hw (air gap)
⇒ difficult to manage23
air gap network 1Network 2
app 1 app 2
Approach: Confinement
Confinement: ensure misbehaving app cannot harm rest of system
Can be implemented at many levels:
– Virtual machines: isolate OS’s on a single machine
24
Virtual Machine Monitor (VMM)
OS1 OS2
app1 app2
Approach: Confinement
Confinement: ensure misbehaving app cannot harm rest of system
Can be implemented at many levels:
– Process: System Call Interposition
Isolate a process in a single operating system
25
Operating System
process 2
process 1
Approach: Confinement
Confinement: ensure misbehaving app cannot harm rest of system
Can be implemented at many levels:
– Threads: Software Fault Isolation (SFI)
• Isolating threads sharing same address space
– Application: e.g. browser-based confinement
26
Implementing Confinement
• Key component: reference monitor
– Mediates requests from applications• Implements protection policy
• Enforces isolation and confinement
– Must always be invoked:• Every application request must be mediated
– Tamperproof:• Reference monitor cannot be killed
• … or if killed, then monitored process is killed too
– Small enough to be analyzed and validated
27
System call interposition
• Observation: to damage host system (e.g. persistent changes) app must make system calls:
– To delete/overwrite files: unlink, open, write
– To do network attacks: socket, bind, connect, send
• Idea: monitor app’s system calls and block unauthorized calls
• Implementation options:
– Completely kernel space (e.g. GSWTK)
– Completely user space (e.g. program shepherding)
– Hybrid (e.g. Systrace)
28
System call interposition
Linux ptrace: process tracing
process calls: ptrace (… , pid_t pid , …)
and wakes up when pid makes sys call.
Monitor kills application if request is disallowed29
OS Kernel
monitoredapplication
(browser)monitor
user space
open(“/etc/passwd”, “r”)
System call interposition
• If app forks, monitor must fork
– Forked-monitor monitors forked-app
• If monitor crashes, app must be killed
• Monitor must maintain all OS state associated with app
– current-working-dir (CWD), UID, EUID, GID
– When app does “cd path” • monitor must update its CWD
• otherwise: relative path requests interpreted incorrectly
30
cd(“/tmp”)open(“passwd”, “r”)
cd(“/etc”)open(“passwd”, “r”)
Complications:
Virtual Machines
31
Virtual Machine Monitor (VMM)
Guest OS 2
Apps
Guest OS 1
Apps
Hardware
Host OS
VM2 VM1
VMM security assumption
VMM Security assumption:
– Malware can infect guest OS and guest apps
– But malware cannot escape from the infected VM
• Cannot infect host OS
• Cannot infect other VMs on the same hardware
Requires that VMM protect itself and is not buggy
– VMM is much simpler than full OS
… but device drivers run in Host OS32
Problem: Covert channels
• Covert channel: unintended communication channel between isolated components
– Can be used to leak classified data from secure component to public component
33
Classified VM Public VM
secretdoc
ma
lwa
re
listenercovert
channel
VMM
An example covert channel
• Both VMs use the same underlying hardware
• To send a bit b ∈ {0,1} malware does:
– b = 1: at 1:00am do CPU intensive calculation
– b = 0: at 1:00am do nothing
• At 1:00am listener does CPU intensive calc. and measures completion time
– b = 1 ⇒ completion-time > threshold
• Many covert channels exist in running system:
– File lock status, cache contents, interrupts, …
– Difficult to eliminate all34
VMM introspection – protecting IDS/AV systems
Intrusion detection / Anti-virus
• Runs as part of OS kernel and user space process
– Kernel root kit can shutdown protection system
– Common practice for modern malware
• Standard solution: run IDS system in the network
– Problem: insufficient visibility into user’s machine
35
VMM introspection – protecting IDS/AV systems
Better: run IDS as part of VMM (protected from malware)
– VMM can monitor virtual hardware for anomalies
– VMI: Virtual Machine Introspection
• Allows VMM to check Guest OS internals
36
Infected VMma
lwa
re
VMM
Guest OS
Hardware
IDS
Subversion of VMI is possible
Coming up
• Access control
37
