Date post: | 14-Jul-2015 |
Category: |
Software |
Upload: | youn-sung-park |
View: | 3,077 times |
Download: | 3 times |
SwiftHighly available, distributed, eventually consistent object
storage
Youn Sung Park
March 16, 2015 Presentation Title
2
1. Introduction
2. Features - Basic
3. Features - Advanced
Contents
Introduction
1
Contents
• What is Swift?
Object Storage
Swift component
March 16, 2015
Swift
4
Introduction
Object Storage
• Object Storage – What is Object Storage?
March 16, 2015
Swift
5
Introduction
Swift
- Highly available, distributed, eventually consistent object storage
Object Storage
• Object Storage – What is Object Storage?
March 16, 2015
Swift
6
Introduction
Object Storage
• Object Storage – What is Object Storage?
March 16, 2015
Swift
7
Introduction
Object storage vs File storagevs Volume storage
Object Storage
• Object Storage – Legacy Storage
March 16, 2015
Swift
8
Introduction
WAS WAS WAS
NAS / SAN NAS / SAN NAS / SAN NAS / SAN…
…
Object Storage
• Object Storage – Legacy Storage
March 16, 2015
Swift
9
Introduction
WAS WAS WAS
NAS / SAN NAS / SAN NAS / SAN NAS / SAN…
…
Storage manager
Object Storage
• Object Storage – Legacy Storage
• Fault tolerance – RAID5, 6, 10, ...
• Interface – NFS for file, iSCSI for volume
• Multi tenancy - ??
• QoS - ??
• Authentication - ??
• Data Placement - ??
• Metadata management - ??
• Capacity management - ??
• Rebalancing - ??
March 16, 2015
Swift
10
Introduction
Object Storage
• Object Storage
March 16, 2015
Swift
11
Introduction
WAS WAS WAS
Object Storage
…
Object Storage
• Object Storage – Legacy Storage
• Fault tolerance – Replication
• Interface – REST API
• Multi tenancy – Defaut. SWAuth, Keystone, Tempauth, LDAP(?)
• QoS – Rate limit, Bandwidth limit, Software defined ...
• Authentication - SWAuth, Keystone, Tempauth
• Data Placement – The Ring algorithm
• Metadata management – Ring and database
• Capacity management – The Ring algorithm
• Rebalancing – The Ring algorithm
March 16, 2015
Swift
12
Introduction
Components
• Swift component – Server application
March 16, 2015
Swift
13
Introduction
Proxy Server
Account Server
Container Server
Object Server
Sqlite
Container List
Object List
Objects
Sqlite
Components
• Swift component – Daemon
• Replicator – Replicate objects and make a system in a consistent state.
• Recover disk failure, network outages situation.
• Updater – Update metadata
• Recover failure caused by container, account metadata high load.
• Auditor – Delete problematic account, container or objects and replicate from other server
• Recover dbs or files which have bit rot problem.
March 16, 2015
Swift
14
Introduction
Components
• Swift component – Key algorithm: Data placement, Metadata management
March 16, 2015
Swift
15
Introduction
RingContainer name/file name Location of a file
ex) 3 copyServer A / sdcServer H / sdyServer D / sdp
Features - Basic
2
Contents
• Swift features
History – Releases
Architecture
High availability
High scalability
Eventually consistent & Self healing
REST API
March 16, 2015
Swift
17
Features - Basic
History - Releases
March 16, 2015
Swift
18
Features - Basic
Architecture
• Architecture
March 16, 2015
Swift
19
Features - Basic
Proxy Server Proxy Server Proxy Server
A/C/O Server
A/C/O Server
A/C/O Server
Zone1
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone2
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone3
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone4
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone5
A/C/O Server
…
…
…
Sqlite
Account server
Container server
Object server
Sqlite
Ring Ring Ring
Features
• High availability, Fault tolerance
March 16, 2015
Swift
20
Features - Basic
Proxy Server Proxy Server Proxy Server
A/C/O Server
A/C/O Server
A/C/O Server
Zone1
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone2
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone3
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone4
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone5
A/C/O Server
…
…
…
Sqlite
Account server
Container server
Object server
Sqlite
Ring Ring Ring
Features
• High scalability
March 16, 2015
Swift
21
Features - Basic
A/C/O Server
HDDHDDHDDHDD
A/C/O Server
HDDHDDHDDHDDHDDHDDHDDHDD
More disk
Zone1Zone1
Zone1Zone1
Zone5
Zone1Zone1
Zone1Zone1
Zone1Zone1
Zone7
More zones
A/C/O Server
A/C/O Server
Zone
A/C/O Server
A/C/O Server
Zone
A/C/O Server
A/C/O Server
More servers
Features
• Eventually consistent & self healing
March 16, 2015
Swift
22
Features - Basic
Proxy Server Proxy Server Proxy Server
A/C/O Server
A/C/O Server
A/C/O Server
Zone1
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone2
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone3
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone4
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone5
A/C/O Server
…
…
…
Sqlite
Account server
Container server
Object server
Sqlite
Fail
Eventual consistency
Ring Ring Ring
Features
• Eventually consistent & self healing
March 16, 2015
Swift
23
Proxy Server Proxy Server
A/C/O Server
A/C/O Server
Zone1
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone2
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone3
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone4
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone5
A/C/O Server
…
…
…
Sqlite
Account server
Container server
Object server
Sqlite
Server failure
Self healing - replication
A/C/O Server
Proxy Server
Features - Basic
Ring RingRing
Features - Advanced
3
Contents
• Swift features
History – Features
Features
Large object support
Static web hosting
S3 compatible API
Object expiration
Temp url
Object versioning
Global cluster
Storage policy
March 16, 2015
Swift
25
Features - Advanced
History - Features
March 16, 2015
Swift
26
Features - Advanced
Swift 2.2 ReleaseStorage policy support
Swift 1.10 ReleaseGlobal clusters support
Swift 1.8 ReleaseDifferent replica policy for A/C/OStatsD update
Swift 1.7 ReleaseObject versioning supportStatd integration
Swift 1.4.8 ReleaseObject expiration supportTemp url support
Swift 1.4.3 ReleaseS3 compatible API support
Swift 1.3 ReleaseStatic website support
Swift 1.2 ReleaseLarge object support
Swift 1.0 Release
Swift 1.12 ReleaseAccount ACL support
Features
• Large object support
• Swift limitation
• Single object: 5GB
• Split object & manage large object
• Manage segmented objects by manifest file
• Ref
• http://docs.openstack.org/developer/swift/overview_large_objects.html
March 16, 2015
Swift
27
Features - Advanced
Features
• Static web hosting
• Upload static web file and make web site
• Upload web site file with index and error files
• Use statiscweb middleware
• Ref
• http://docs.openstack.org/developer/swift/middleware.html#staticweb
March 16, 2015
Swift
28
Features - Advanced
Features
• S3 compatible API
• Support S3 API
• Support limited API less than 40%
• Use swift3 middleware
• Ref
• https://github.com/stackforge/swift3
March 16, 2015
Swift
29
Features - Advanced
Features
• Object expiration
• Schedule deletion of objects
• Use X-Delete-At and X-Delete-After header while using an object PUT or POST
•X-Delete-At: Delete object at specified time
•X-Delete-After: Delete object after specified time
• Ref
• http://docs.openstack.org/developer/swift/overview_expiring_objects.html
March 16, 2015
Swift
30
Features - Advanced
Features
• Temp url
• Provide url to access in limited time
• Need temp_url_expires time in header
• Use temporary URL middleware
• Ref
• http://docs.openstack.org/developer/swift/api/temporary_url_middleware.html
March 16, 2015
Swift
31
Features - Advanced
Features
• Global cluster
• Make a single cluster in distant region
• Read/Write affinity
• Deferred replication
• Ref
• http://docs.openstack.org/developer/swift/admin_guide.html
• https://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/
March 16, 2015
Swift
32
Features - Advanced
Features
• Storage policy
• Support various policy in sing storage cluster
• Use multiple ring file
• Ref
• http://docs.openstack.org/developer/swift/admin_guide.html
March 16, 2015
Swift
33
Features - Advanced
Thanks