Marten van Dijk
Syed Kamran Haider, Chenglu Jin, Phuong Ha Nguyen
Department of Electrical & Computer Engineering
University of Connecticut
CSE 5095 & ECE 4451 & ECE 5451 – Spring 2017
Intel SGX: High Level Overview & Memory Organization & Page Swapping
With the help of:
1. Intel SGX Tutorial (Reference Number: 332680-002) presented at ISCA 2015
2. “Intel SGX Explained”, Victor Costan and Srinivas Devadas, CSAIL MIT
Lecture 3b
• Slide deck extracted from Kamran’s tutorial on SGX
presented during ECE 6095 Spring 2017 on Secure
Computation and Storage, a precursor to this course
Outline
Introduction
SGX High Level Overview
SGX Memory Organization
2
Why Aren’t Compute Devices Trustworthy?
Protected Mode (Privilege Levels i.e., Rings) protects OS from apps …
3
Why Aren’t Compute Devices Trustworthy?
Protected Mode (Privilege Levels i.e., Rings) protects OS from apps …
… and protects apps from each other …
4
Why Aren’t Compute Devices Trustworthy?
Protected Mode (Privilege Levels i.e., Rings) protects OS from apps …
… and protects apps from each other …
… UNTIL a malicious app exploits a flaw to gain full privileges and then tampers with the OS or other apps.
Apps are typically not protected from privileged code attacks
5
Why Aren’t Compute Devices Trustworthy?
Protected Mode (Privilege Levels i.e., Rings) protects OS from apps …
… and protects apps from each other …
… UNTIL a malicious app exploits a flaw to gain full privileges and then tampers with the OS or other apps.
Apps are typically not protected from privileged code attacks
6
Attack surface today…
In current systems, a large code base constitutes the part that can be exploited…
Application Codes
OS code
Virtual Machine Manager code
Hence, millions of lines of code need to be inspected for exploitable bugs etc…
7
Reduced attack surface with SGX Enclaves
With SGX, Application gains ability to defend its own secrets
Smallest attack surface (App + processor)
Malware that subverts OS/VMM, BIOS, Drivers etc. cannot steal app secrets
8
What is SGX?
9
The key concept behind Intel’s Software Guard Extensions (SGX) is an Enclave.
Enclave:
A protected environment that contains the code and data of a security-sensitive computation.
There can be many enclaves in the system at a time!
What is SGX?
Isolation
Each enclave’s environment is isolated from the untrusted software outside the enclave, as well as from other enclaves.
Attestation
A software attestation scheme that allows a remote party to authenticate the software running inside an enclave.
10
SGX enabled processors offer the following two crucial properties:
Protects the privacy and integrity of the computation!
How SGX Secure Enclaves Work
App is built with trusted and untrusted parts
11
How SGX Secure Enclaves Work
App is built with trusted and untrusted parts
App runs & creates enclave which is placed in trusted memory
12
How SGX Secure Enclaves Work
App is built with trusted and untrusted parts
App runs & creates enclave which is placed in trusted memory
Trusted function is called; code running inside enclave sees data in clear
13
How SGX Secure Enclaves Work
App is built with trusted and untrusted parts
App runs & creates enclave which is placed in trusted memory
Trusted function is called; code running inside enclave sees data in clear;
Function returns; enclave data remains in trusted memory
External access to enclave data is denied
14
SGX High Level Overview• Programming Environment
• Access Control (Isolation)
• Attestation & Sealing
• Memory Snooping Protection
15
SGX Programming Environment
Trusted execution environment embedded in a process
16
SGX Programming Environment
Trusted execution environment embedded in a process
The process creates a secure Enclave
17
SGX Programming Environment
Trusted execution environment embedded in a process
The process creates a secure Enclave
Contains Code & Data
Provides Confidentiality
Provides Integrity & Freshness
Controlled Entry Points
18
SGX Programming Environment
Trusted execution environment embedded in a process
The process creates a secure Enclave
Contains Code & Data
Provides Confidentiality
Provides Integrity
Controlled Entry Points
Supporting multiple threads
Full access to application’s memory
19
SGX High-level HW/SW Picture
20
SGX Access Control
21
SGX Access Control
22
SGX Access Control
23
SGX Access Control
24
SGX Access Control
25
Critical Feature: Attestation and Sealing
26
Critical Feature: Attestation and Sealing
Enclave built & measured
27
Critical Feature: Attestation and Sealing
Enclave built & measured
Enclave requests REPORT (HW-signed blob that includes enclave identity information)
28
Critical Feature: Attestation and Sealing
Enclave built & measured
Enclave requests REPORT (HW-signed blob that includes enclave identity information)
REPORT sent to server & verified by the server
29
Critical Feature: Attestation and Sealing
Enclave built & measured
Enclave requests REPORT (HW-signed blob that includes enclave identity information)
REPORT sent to server & verified by the server
30
Critical Feature: Attestation and Sealing
Enclave built & measured
Enclave requests REPORT (HW-signed blob that includes enclave identity information)
REPORT sent to server & verified by the server
Attestation Key sent to enclave, first secret provisioned
31
Critical Feature: Attestation and Sealing
Enclave built & measured
Enclave requests REPORT (HW-signed blob that includes enclave identity information)
REPORT sent to server & verified by the server
Attestation Key sent to enclave, first secret provisioned
Enclave-platform-specific Sealing Key generated (EGETKEY)
32
Critical Feature: Attestation and Sealing
Enclave built & measured
Enclave requests REPORT (HW-signed blob that includes enclave identity information)
REPORT sent to server & verified by the server
Attestation Key sent to enclave, first secret provisioned
Enclave-platform-specific Sealing Key generated (EGETKEY)
Attestation Key encrypted via Sealing Key & stored for later use.33
Protection against Memory Snooping Attacks
34
Protection against Memory Snooping Attacks
Security perimeter is the CPU package boundary
35
Protection against Memory Snooping Attacks
Security perimeter is the CPU package boundary
Data and code is unencrypted inside CPU package
36
Protection against Memory Snooping Attacks
Security perimeter is the CPU package boundary
Data and code is unencrypted inside CPU package
Data and code outside CPU package is encrypted and/or integrity checked
37
Protection against Memory Snooping Attacks
Security perimeter is the CPU package boundary
Data and code is unencrypted inside CPU package
Data and code outside CPU package is encrypted and/or integrity checked
External memory reads and bus snoops see only encrypted data
SGX does NOT protect against leakage via access patterns to the external memory!
38
SGX Memory Organization • Physical Memory Organization
• Memory Layout of SGX Enclave
39
SGX Physical Memory Organization
The Processor’s Reserved Memory (PRM) is a reserved region in DRAM.
40
SGX Physical Memory Organization
The Processor’s Reserved Memory (PRM) is a reserved region in DRAM.
The Enclave Page Cache (EPC) contains enclave’s code & data.
41
SGX Physical Memory Organization
The Processor’s Reserved Memory (PRM) is a reserved region in DRAM.
The Enclave Page Cache (EPC) contains enclave’s code & data.
The Enclave Page Cache Map (EPCM) contains an entry to point to each EPC page.
42
Trusted
Memory
Enclave Page Cache Map (EPCM)
EPCM entries are used by SGX hardware to perform checks that ensure that the (untrusted) OS is behaving as expected
E.g., the same EPC page cannot be allocated to two enclaves
The EPCM’s content is only used by SGX’s security checks
The application and OS programmer can ignore it.
EPCM Entry Fields
VALID EPC page is available or allocated
PT Page type, e.g. Regular (PT_REG), or SECS Page
ENCLAVESECS Points to the SECS Page of owner enclave
43
Trusted
Memory
Valid Page Type Owner Enclave…
SGX Enclave Control Structure (SECS)
The SGX Enclave Control Structure (SECS) stores critical metadata of each SGX enclave
E.g., enclave’s measurement for software attestation
Enclave Attributes
Each SECS is stored in a dedicated EPC page with the page type PT_SECS.
SECS Pages cannot be accessed by:
System Software (OS/Hypervisor etc.)
Even the enclave’s code itself.
44
Regular Page
Regular Page
Regular Page
SECS Page
SECS Page
The Enclave Linear Address Range (ELRANGE)
EPC pages are accessed using a dedicated region in the enclave’s virtual address space, called ELRANGE.
45
The Enclave Linear Address Range (ELRANGE)
EPC pages are accessed using a dedicated region in the enclave’s virtual address space, called ELRANGE.
The rest of the virtual address space is used to access the memory of the host process.
46
The Enclave Linear Address Range (ELRANGE)
EPC pages are accessed using a dedicated region in the enclave’s virtual address space, called ELRANGE.
The rest of the virtual address space is used to access the memory of the host process.
The memory mappings are established using the page tables managed by system software.
47
SGX Enclave Attributes
An enclave’s attributes are the sub-fields in the ATTRIBUTES field of the enclave’s SECS.
DEBUG Enables Read/Write enclave’s memory in Debug mode.
XFRM Defines Extended Features Request Mask to specify architectural extensions.
MODE64BIT Set to true for enclaves that use the 64-bit Intel architecture.
48
Address Translation for SGX Enclaves
The OS and hypervisor are in full control of the page tables and EPTs.
Each enclave’s code uses this address translation Possible Security Problems
When an EPC page is allocated, its intended virtual address is recorded in the EPCM entry for the page, in the ADDRESS field.
Upon address translation, given virtual address is verified against the stored one!
Also, R/W/X attributes from EPCM entry override the permissions specified in page tables.
49
EPCM Entry:
The Thread Control Structure (TCS)
It is possible for multiple logical processors (Threads) to concurrently execute the same enclave’s code at the same time, via different threads.
SGX implementation uses a Thread Control Structure (TCS) for each thread that executes an enclave’s code.
Each TCS is stored in a dedicated EPC Page.
The contents of an EPC page that holds a TCS cannot be directly accessed, even by the code of the enclave that owns the TCS
Similar restriction as EPC pages holding SECS instances.
50
TCS Page
Regular Page
Regular Page
SECS Page
SECS Page
EPC Page Swapping• High Level Overview
• Paging Instructions
• Examples
51
EPC Page Swapping
EPC memory is set by BIOS and limited from size perspective
We need a way to remove an EPC page, place into unprotected memory, and restore it later.
Page must maintain same security properties (confidentiality, anti-replay, and integrity) when restored
EPC paging instructions provide ability to encrypt page and produce meta data needed to meet requirements
52
EPC Page Swapping
An enclave page must be evicted only after all cached translations to that page have been evicted from all logical processors.
Content is swapped on 4KByte page basis
Each 4KByte EPC page produces
4KByte of encrypted content
128Byte of meta-data (PCMD).
53
Paging Operations at a High Level
When a page is evicted from EPC
It is assigned a unique version number which is recorded in a new type of EPC page called Version Array (VA)
Encrypted page, metadata, and EPCM information are written out to system memory
When page is reloaded
The processor decrypts, and integrity checks the page, using crypto metadata
The processor verifies that version is the same version that was last written out
54
EPC Paging Instructions
EPA
Allocates a 4KByte page in EPC for holding an array of page versions (VA) for anti-replay protection
VA contains versions of paged out enclave pages, size of each version slot is 64 bits.
EBLOCK
Blocks a page from being accessed in preparation for swapping it out
Any future accesses by owner enclave to BLOCKED page result in #PF
Returns indication that page previously blocked
ETRACK
Sets a tracking mechanism to verify that all TLB entries for the blocked page has been flushed
55
EPC Paging Instructions
EWB
Securely evicts a 4KByte page from the EPC along with it’s page information
Assigning a unique version value for the page and storing it in the VA page.
Encrypt EPC page, create MAC over the encrypted page, version counter, and meta data. And write it out to external memory
Enclave page must be first prepared for eviction:
I.e. Blocked and no TLB entry refer to that page.
56
EPC Paging Instructions
ELDU/B
Securely loads a page back from memory into the EPC into an unblocked or blocked state
Verify the MAC on the meta data, version counter from specific VA entry, and encrypted enclave page content
If verification succeed, decrypt the enclave page content into EPC page allocated by system memory and clear the VA entry.
57
Page-out Example
Instruction EWB writes back a page from EPC to system memory
Assume the page is ready (Blocked, no TLB entries)
58
Page-out Example
Instruction EWB writes back a page from EPC to system memory
Assume the page is ready (Blocked, no TLB entries)
EWB Parameters:
Pointer to EPC page that needs to be paged out
Pointer to empty version slot
Pointers outside EPC location
59
Page-out Example
Instruction EWB writes back a page from EPC to system memory
Assume the page is ready (Blocked, no TLB entries)
EWB Parameters:
Pointer to EPC page that needs to be paged out
Pointer to empty version slot
Pointers outside EPC location
EWB Operation
Remove page from the EPC
Populate version slot
Write encrypted version to outside
All pages, including SECS and Version Array can be paged out
60
Page-in Example
Instruction ELD loads a page from system memory into EPC
61
Page-in Example
Instruction ELD loads a page from system memory into EPC
ELD Parameters:
Encrypted page
Free EPC page
SECS (for an enclave page)
Populated version slot
62
Page-in Example
Instruction ELD loads a page from system memory into EPC
ELD Parameters:
Encrypted page
Free EPC page
SECS (for an enclave page)
Populated version slot
ELD Operation
Verify and decrypt the page using version
Populate the EPC slot
Free-up version slot
63