You can rely on whenever you need
Yosuke Hara Oct 30, 2012
Tuesday, October 30, 12
1. Motivation
Tuesday, October 30, 12
Need to store and managehuge amount of “media files” at low-cost.
Tuesday, October 30, 12
1. Low ROI2. Possibility of SPOF3. Storage Expansion is difficult during increasing data
?
Tuesday, October 30, 12
X 3.5
Face the situation
*ZDNet: http://www.zdnet.com/blog/btl/amazon-web-services-hosts-more-than-762-billion-objects-in-s3/68308
Tuesday, October 30, 12
“Object Storage System”
Tuesday, October 30, 12
5MB 100MB a few GB
From Photo Storage To Cloud Storage
1st step as Cloud StorageSpecialize in “Photo”
Tuesday, October 30, 12
5MB 100MB a few GB
From Photo Storage To Cloud Storage
Aim to “DATA-HUB” in the CloudHandle various unstructured data
Tuesday, October 30, 12
S3FS-C
Tuesday, October 30, 12
Aim To
Tuesday, October 30, 12
2. Overview
Tuesday, October 30, 12
Storage
Gateway
Manager
ConcurrencyDistributionFault tolerance
Using in Telecom, Banking, e-commerce, Instant messaging,...
Tuesday, October 30, 12
LeoFS-Manager
LeoFS-Gateway
LeoFS-Storage
REST over HTTPRPC
Request from Web Application(s) or Browser
META Object Store
Storage Engine/Router
META Object Store META Object Store
RPC
Storage Engine/Router Storage Engine/Router
Load Balancer
S3-API
SNMP
GUI Console
LeoFS Overview
Gateway (Stateless Proxy)
HTTP Request/Response Handling+
w/Object Cache
ManagerSystem Management
Ring MonitorNode State Monitor
StorageObject Storage, Meta data Storage
+Replicator/Recoverer, Qeueue
Tuesday, October 30, 12
LeoFS-Manager
LeoFS-Gateway
LeoFS-Storage
REST over HTTP (80/443) RPC
(4369)
Request from Web Application(s) or Browser
META Object Store
Storage Engine/Router
META Object Store META Object Store
RPC (4369)
Storage Engine/Router Storage Engine/Router
Load Balancer
S3-API
Monitor(SNMP)
GUI Console
(4000,4010,4020)
(10020, 10021)
Tuesday, October 30, 12
1 - Minimum for Development
Manager x 1 Gateway x 1 Storage x 1
20 - 50TB Storage System (# of replicas = 3)
Manager x 2 Gateway x 3 .. Storage x 8 - 15
10TB .. 20TB / server
50 - 300TB Storage System (# of replicas = 3)
Manager x 2 Gateway x 4 .. Storage x 45 - 90
10TB .. 20TB / server
on XEN, VMWare
Tuesday, October 30, 12
Media Platform
Application / Log Collector
Search / Analysis
PaaS / IaaS
DATA-HUB
Tuesday, October 30, 12
Tuesday, October 30, 12
“DATA-HUB”
Tuesday, October 30, 12
Inside LeoFS
Tuesday, October 30, 12
LeoFS ArchitectureGatewayHTTP
Tuesday, October 30, 12
LeoFS ArchitectureGatewayHTTP
Storage Cluster
Erlang RPC
Erlang RPC
Manager Cluster
State/Process MonitorErlang RPC
Object Cache
Tuesday, October 30, 12
LeoFS Architecture - Gateway / Storage
Storage Engine
Object Storage Metadata
StorageRPC RPC
membership (fault-detection)
redundant-manager replicator
queue
read-repairer
Gateway REST over HTTP (S3-API)
redundant-manager membership (fault-detection)
get put delete head
RPC
Tuesday, October 30, 12
LeoFS Gateway
Tuesday, October 30, 12
*Cowboy: Erlang light-weight HTTP-Server - http://http://www.ninenines.eu/
Gateway
From ApplicationsS3-API
Object Cache
Replicate when using RPC
Consistent HashingHorizontal Distribution
Storage Nodes
[ LRU, Slab allocator, Skip graph ]“Cowboy”
“Stateless Proxy”
Tuesday, October 30, 12
LeoFS Storage
Tuesday, October 30, 12
...
LeoFS Storage
leo-object-storage
LeoFS Storage Engine
Metadata : Keeps an in-memory index of all data.Object Storage : Log structured (append-only) object store.
Request From Gateway
replicatorrepairer
queue...
Tuesday, October 30, 12
LeoFS Storage Engine - Data Structure
Offset Version Time-stamp{VNodeId, Key}
<Metadata>
Checksum
for Sync
KeySize CustomMeta Size File Size
for Retrieve an File (Object)
Footer (8B)
Checksum KeySize DataSize Offset Version Time-stamp
{VNodeId,Key} User-Meta Footer
<Object>
Header (Metadata - Fixed length) Body (Variable Length)
User-MetaSize
ActualFile
Supe
r-bl
ock
Obj
ect-1
Obj
ect-2
Obj
ect-3
<Object Container>
Obj
ect-4
Obj
ect-5
Tuesday, October 30, 12
ObjectID
Leo Object StorageTraditional File System
directory inode
owner infosize
timestampblocks
directory data
inodefilename
file inode
owner infosize
timestampblocks
File
LeoFS Storage Engine
Tuesday, October 30, 12
ObjectID
Leo Object Storage
LeoFS Storage Engine
Robustness
High Performance
Tuesday, October 30, 12
< META DATA >IDFilenameOffsetSizeChecksum
“Object Container”
Header
File
Footer
< META DATA >IdFilenameOffset, SizeChecksum (MD5)Version#
Storage Engine
LeoFS Storage Engine - Retrieve an object from the storage
Metadata Storage
Tuesday, October 30, 12
Storage Engine
Data
LeoFS Storage Engine - Insert an object into the storage
Insert a metadata
Append an object into the object container
Metadata Storage
Tuesday, October 30, 12
Compaction
OLD Object ContainerOLD Metadata
NEW Object ContainerNew Metadata
LeoFS Storage Engine - Remove unnecessary objects from the storage
Storage Engine
Tuesday, October 30, 12
Large object Support
Tuesday, October 30, 12
Large-object Support (LeoFS v0.12)
Gateway
chunk-0
chunk-1
chunk-2
chunk-3Client
1. Able to equalize disk usage of each storage node2. High I/O efficiency
Original fils’s Metadata #metadata{ ... dsize = 10485768, cnumber = 4, csize = 3670016, ...}
Storage Cluster
Tuesday, October 30, 12
LeoFS Manager
Tuesday, October 30, 12
LeoFS Manager
Manager
For Administrators
Monitor
Operate
RING, Node State
status, suspend,resume, detach, whereis, ...
TCP/IP
for continue operation
Tuesday, October 30, 12
LeoFS-Manager
LeoFS-Gateway
LeoFS-Storage
REST over HTTP (80/443) RPC
(4369)
Request from Web Application(s) or Browser
META Object Store
Storage Engine/Router
META Object Store META Object Store
RPC (4369)
Storage Engine/Router Storage Engine/Router
Load Balancer
S3-API
Monitor(SNMP)
GUI Console
(4000,4010,4020)
(10020, 10021)
Verticalized Storage System
Tuesday, October 30, 12
3. Future Works
Tuesday, October 30, 12
Improve Performance & Functionality
High I/O efficiency
1. Multi-layer Cache (Using SSD)
HIGH-Performance (Low latency)
Primary Cache
Secondary Cache
Gateway
Reduce traffic between Gateway and Storage
Storage
Tuesday, October 30, 12
Improve Performance & Functionality
2. Multi-Datacenter for Geographical Optimization
HIGH-Scalability
HIGH-Availability
Tuesday, October 30, 12
System Console
Log Analyzer / Searcher
3. LeoFS’s Tools
Improve Performance & Functionality
Tuesday, October 30, 12
4. Wrap Up
Tuesday, October 30, 12
LeoFS will realize 3-HIGHs, soonHIGH Cost PerformanceHIGH ReliabilityHIGH Scalability
Tuesday, October 30, 12
Thank you for your time
LeoFS - http://www.leofs.org
Tuesday, October 30, 12