61
Department of Computer Sciences, University of Wisconsin Madison DADA – Dynamic Allocation of Disk Area Jayaram Bobba Vivek Shrivastava
Department of Computer Sciences, University of Wisconsin Madison
DADA – Dynamic Allocation of Disk Area
Jayaram Bobba
Vivek Shrivastava
Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction Existing Framework Implementation Issues Results Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Problem
Disk Volumes Allocate physical space on creation
Multiple Disk Volumes on a physical disk Unused space Disk space not a premium What If?
Department of Computer Sciences, University of Wisconsin Madison
Motivation
Virtual Machine Environment Add Some points
Storage Area Networks Storage area is the bo
Dynamic Physical Allocation (LFS style)
Department of Computer Sciences, University of Wisconsin Madison
Related Work
HP AutoRAID Dynamically change redundancy levels Hot Data – Mirrored Cold Data – RAID 5 Dynamic migration of data
Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction
Existing Framework Implementation Issues Results Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Existing Framework
Logical Volume Manager (LVM) User space tool that enables creation of logical
volumes of a logical partition. Supports resizing of volumes.
Device Mapper (DM) Kernel driver that provides a level of indirection in
address translation
Department of Computer Sciences, University of Wisconsin Madison
LVM design
Volume Group (VG) Physical Volume (PV) Logical Volume (LV) Physical Extent (PE) Logical Extent (LE)
Department of Computer Sciences, University of Wisconsin Madison
LVM design
Physical Device
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
LV 1
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
LE
LV 1LV2
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
LE
LV 1LV2
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM designVG
PV
LE
LV 1LV2
PE
Department of Computer Sciences, University of Wisconsin Madison
DM designUserspace Applications
log kcopyd
emc
ioctl interface filesystem interface
control interface block interface
core device-mapper
mapping/target interface
linear striped snapshot multipath
round-robin
path selectors
hardwarehandlers
Department of Computer Sciences, University of Wisconsin Madison
LVM – DM interaction
Kernel
Userspace
Device Mapper
libdevmapper
LVM2
Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction Existing Framework
Implementation Issues Results Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Modifications
A trap mechanism from kernel driver to user-level LVM code.
Support for multiple segments in LV
Support for virtual mappings in driver
Department of Computer Sciences, University of Wisconsin Madison
Initial State
LVM
Device Mapper
User space
Kernel space
Unallocated Disk Space
LVM daemon
Department of Computer Sciences, University of Wisconsin Madison
Volume Group Creation
LVM
Device Mapper
User space
Kernel space
Unallocated Disk Space
LVM daemon
PE (4MB) 25 Extents
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon Create LV
32MB
User A
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon
User A
Map 0-7 extents for user A
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon
User A
Map 0-7 extents for user A
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
Create LV
16 MB
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
Create LV
16 MB
map 8-11 extents for B
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
map 8-11 extents for B
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User BRead/write
Read/write
Department of Computer Sciences, University of Wisconsin Madison
Volume Group Creation
LVM
Device Mapper
User space
Kernel space
Unallocated Disk Space
LVM daemon
PE (4MB) 25 Extents
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon Create LV1
32MB
User A
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon
User A
Virtually map 0-7 extents for user A
{0-7 error}
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
Create LV
16 MB
{0-7 error}
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
Create LV2
16 MB
Virtually map 8-11 extents for B
{0-7 error} {8-11 error}
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
Write 6MB
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
writetrap
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
writetrap
On Demand allocation of disk
area
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
Write 6MB
trap
On Demand allocation of disk
area
Allocate 2 extents (8MB)
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
User B
{0-1 linear}{2-7 error}
{8-11 error}
Write 6MB
trap
On Demand allocation of disk
area
Allocate 2 extents (8MB)
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
User B
{0-1 linear}{2-7 error}{8-11 error}
Write
10MB
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
(0-1 linear}{0-7 error}{8-11 error}
Write 10 MB trap
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
writetrap
On Demand allocation of disk
area
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
User B
{0-1 linear}{2-7 error}{8-10 linear}
Write 10 MBtrap
On Demand allocation of disk
area
Allocate 3 extents (12 MB)
{11 error}
Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction Existing Framework Implementation
Issues Results Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Issues
Not the most efficient implementation On a trap,
How much to allocate? On demand Pre-allocate
Where to allocate? Temporal locality Spatial locality
Free unused space
Department of Computer Sciences, University of Wisconsin Madison
How much?
On-demand Allocate only as many blocks as needed
Pre-allocate Pre-allocate physical space for blocks likely
to be accessed in the future. How much? Where?
Department of Computer Sciences, University of Wisconsin Madison
How much?
Tradeoff – Performance vs. Disk Space Depends on the workload
I/O on critical path Asynchronous I/O Multithreaded Applications
Department of Computer Sciences, University of Wisconsin Madison
Pre-Allocation
Think: Filesystem Block Allocation Dumb Allocation
Stride Allocation
Multi-Strided Allocation
Department of Computer Sciences, University of Wisconsin Madison
Dumb Allocation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
{0-1 linear}{2-7 error}
Write 6MB
trap
On Demand allocation of disk
area
Allocate 2 extents (8MB)
Department of Computer Sciences, University of Wisconsin Madison
Stride Allocation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
{0-1 linear}{2-7 error}
Write 6MB
trap
On Demand allocation of disk
area
Allocate 4 extents (16MB)
PreAllocated Extent
Department of Computer Sciences, University of Wisconsin Madison
Ext2 access pattern
Department of Computer Sciences, University of Wisconsin Madison
Ext3 access pattern
Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction Existing Framework Implementation Issues
Results Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Results
Micro Benchmarks mkfs
Trace based analysis Trace collected by HP Research Lab
Department of Computer Sciences, University of Wisconsin Madison
Micro Benchmark : mkfs
ExecutionTimes
0
20
40
60
80
100
120
140
8K 16K 32K 64K 128K 256K 512K 1M 2M 4M
Extent Size
Tim
e T
aken
(in
sec
s)
#Traps Taken
0
50
100
150
200
250
300
350
400
450
8K 16K 32K 64K 128K 256K 512K 1M 2M 4M
Extent Size
# T
rap
s T
aken
Department of Computer Sciences, University of Wisconsin Madison
Micro Benchmark : Asynchronous I/O
Department of Computer Sciences, University of Wisconsin Madison
Micro Benchmark : Asynchronous I/O
Department of Computer Sciences, University of Wisconsin Madison
Trace Based Analysis
Department of Computer Sciences, University of Wisconsin Madison
Trace Based Analysis
Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction Existing Framework Implementation Issues Results
Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Conclusions
Tradeoff between performance penalty and space
Useful in environments where space is the bottleneck resource
Techniques like Pre Allocation reduce the performance penalty to some extent
For practical implementation, LVM must be placed in kernel address space
Department of Computer Sciences, University of Wisconsin Madison
Conclusions
mkfs performance degrades substantially due to traps.
No degradation in performance for CPU intensive applications with asynchronous I/O
Extent size in the range of 512K-1M provide best results for the trace
This evaluation will be more realistic, if LVM is moved into Kernel Address space.
