1. 1. 1 Copyright 2014 MPSTOR LTD. All rights reserved. Getting performance & scalability on standard platforms, the Object vs Block storage debate Speakers William Oppermann, CEO, MPSTOR Mo Hassine, Director of Product and Marketing, MPSTOR
2. 2. 2 Copyright 2014 MPSTOR LTD. All rights reserved. Storage requirements in the datacenter Wide range of application workloads provisioned at scale cost effectively Provisioning ALL the Datacenter consumers Virtual machines Tenant spaces Storage centric services Consumer nodes Opex Complexity & management Automation of provisioning Volume services management (snapshot, thin volumes, replication, backup) Capex Use of Open Platforms Proprietary platforms Storage Efficiency (the cost of redundancy in storage) Availibility&Resiliency Component Redundancy Data Consistency& Integrity IDA (Information Dispersal Algorithms) Scalability Scale up Scale scale Reducing the Impact of failures through IDA Performance per workload type BW performance IO performance Caching acceleration Fabrics Tiering SLA/QOS
3. 3. 3 Copyright 2014 MPSTOR LTD. All rights reserved. The BIG 6 issues Storage technologies for the cloud struggle in the datacenter because there is a BIG 6 requirements list that is very difficult to fulfil. 1) Storage must be resilient to component failure 2) Storage must be scalable (addition of capacity and amount of data stored) 3) Storage must cater for a wide range of application workloads 4) Storage provisioning to a wide range of storage consumer types Virtual machines Tenant spaces Storage centric services Consumer nodes 5) Storage must deliver 1,2,3,4 at a CAPEX and OPEX compatible with Utility cloud computing. Open platforms Highly automated provisioning End user provisioning tools Simple and easy to administer by the cloud operator 6) Storage must be secure
4. 4. 4 Copyright 2014 MPSTOR LTD. All rights reserved. The BIG 6 storage challenges RESILIENCY SCALABILITY WORKLOADS CONSUMER TYPES TCO (CAPEX and OPEX) DATA Security
5. 5. 5 Copyright 2014 MPSTOR LTD. All rights reserved. BIG 6 ranking out of 10 for Block and Object 0 2 4 6 8 10 12 RESILIENCY DIVERSE WORKLOADS TCO (Cloud CAPEX and OPEX) STORAGE SECURITY SCALABILITY MULTIPLE CONSUMER TYPES Block Object
6. 6. 6 Copyright 2014 MPSTOR LTD. All rights reserved. ISSUES BLOCK OBJECT ? COMMENTS RESILIENCY Object has large windows of failure & poor storage efficiency DIVERSE WORKLOADS Object suitable for narrow range of workload types TCO (Cloud CAPEX and OPEX) Block storage high cost to Administer STORAGE SECURITY Both object and block storage needs managed encryption and security access SCALABILITY BLOCK storage very difficult to manage for scale out multi fabric, multi tier storage. MULTIPLE CONSUMER TYPES Missing paradigm in both Block , File and Object storage for many of the datacenter consumer types Full Support Partial Support Poor Support One paradigm does not fit all
7. 7. 7 Copyright 2014 MPSTOR LTD. All rights reserved. Object & Block storage Object storage is resilient to component failures, scalable and can be delivered on open hardware platforms cost effectively for some workload types. Strong point is cost, scalability and managing the impact of disk failures but object storage has poor performance in workloads requiring high IO & low latency => It can deliver partially on the BIG 6 issues Block storage is a resilient & somewhat less scalable technology that is usually delivered on proprietary platforms that supports all workload types. Strong point is very good mixed work load performance, support for different media types & fabrics. Scaling can be difficult and the recovery time from failures in large silos can prohibitive. => It can deliver partially on the BIG 6 issues
8. 8. 8 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#1 (using object storage) Storage needs to be resilient to component failure Object storage keeps multiple copies of data objects within its storage pool (3 copies => storage efficiency = 100/3 = 33%) Each copy is replicated over time T, during this T minutes the data is not protected, T can be long multiples of minutes and is not acceptable in many mission critical enterprise class configurations. If a disk fails the object store notes which objects were dependent on that disk and makes new copies over time of the lost objects. => At the cost of low storage efficiency and a wide window in time of non protection object storage delivers resiliency. For the designed for USE CASE (upload/download outside the datacenter of digital media (photos, videos, documents) this can be acceptable
9. 9. 9 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#2 (using object storage) Storage needs to be scalable (Scalable in terms of overall capacity and amount of data stored) Adding storage capacity is relatively simple but usually requires a major re- balancing operation when storage is added to the pool (all the data gets reshuffled between its disks) Data objects are stored using data base technology. Each object has a unique ID, the ID is used as a database KEY, as the object store grows the lookup cost of keys grows with the number of objects Object storage has a high overhead every time the object is accessed. This overhead grows with the number of stored objects. Object storage delivers partially the requirement of increasing the pool capacity size at the cost of rebalancing itself when capacity is added. Object storage struggles with the ever increasing number of objects it has to store.
10. 10. 10 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#3 (using object storage) Storage needs to cater for a wide range of end user workloads The SEMANTICS of Object storage make it suitable for only a restricted (but important) set of workloads, its semantics also make provisioning an end user task which improves the provisioning OPEX cost. Storing photos/videos/documents in object storage works well, taking seconds to load a photo/video is acceptable if once loaded the data can stream. For data processing workloads the object storage look-up costs make it unusable. Adding BLOCK interfaces on OBJECT stores only makes the problems worse in terms of performance but also increases the complexity of the solution.
11. 11. 11 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#4 (using object storage) Storage delivery to a wide range of storage consumer types Object storage uses a very specific API making it useless at provisioning multiple consumer types which all need block storage to run/boot but can in some cases use object storage when in operation for certain workloads.
12. 12. 12 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#5 (using object storage) Storage needs to deliver 1,2,3,4 at a CAPEX and OPEX compatible with Utility cloud computing pricing. Storing photos/videos/documents using object storage works well. Simple interface, end user can provision, storage looks like a big pool => Object storage on low cost media & open hardware platforms even with low storage efficiency due to its multiple copies works well for upload/download of large media files. Object storage is poor at transaction data processing, low latency and high IO type workloads Object storage partially delivers on the big5 requirements !
13. 13. 13 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#1 (using Block storage) Storage needs to be resilient to component failure Block storage uses RAID or ERASURE code technology to store data resiliently. Block storage uses many techniques to improve resiliency to failures such as multipathing, dual controllers, cache coherency techniques which store data with a ZERO time window of non redundancy. Data is secured and consistent, i.e a block storage controller can die mid flight of an IO and the system will store work. This very complex hardware and software is a considerable technical challenge and is reflected in the high cost and proprietary nature of block storage. RAID&ERASURE techniques require special consideration so that the BUILD and REBUILD recovery times stay within defined limits.
14. 14. 14 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#2 (using Block storage) Storage needs to be scalable (Scalable in terms of overall capacity and amount of data stored) Adding storage usually requires a new RAID, a RAID ADD cost depends on the size of the RAID and the RAID level. Disks sizes are now getting so large the RAID build time is becoming a major problem to admins, during a RAID rebuild the raid is not fully redundant and if enough disks are lost due to failure the entire dataset can be lost. Block storage has been designed for a high setup cost and a very low transaction cost during operation. Unlike object storage the transaction cost of accessing data scales perfectly with the amount of data stored since the transaction cost is a FIXED cost. Block storage works very efficiently when managing a storage SILO (SCALE-UP by adding more storage to the SILO), SCALE-OUT (i,e adding more storage SILOS in parallel) is difficult. This scale out issue in contrast to OBJECT storage puts BLOCK storage into an "all the eggs in one basket" type technology. The basket may be resilient and redundant but there is only one basket.
15. 15. 15 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#3 (using object storage) Storage needs to cater for a wide range of end user workloads Block storage has inbuilt into its semantics and implementation the ability to cover all workload types. This is BLOCK storages strongest capability and is one of its major advantage over Object storage. Additional advantages are its ability to transition across multiple high speed fabrics and provide the raw building blocks for other protocols such as File and Object storage iself.
16. 16. 16 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#4 (using Block storage) Storage delivery to a wide range of storage consumer types Block storage does not in it self provision all the consumer types in the datacenter, as a base technology its far easier and higher a performing base to develop tools that can provision all the consumer types of Virtual machines Tenant spaces Storage centric services Consumer nodes
17. 17. 17 Copyright 2014 MPSTOR LTD. All rights reserved. BIG5 Issue#5 (using Block storage) Storage needs to deliver 1,2,3,4 at a CAPEX and OPEX compatible with Utility cloud computing pricing. Block storage scales both in capacity and quantity of data stored. Block scales up very easily, scales out with difficulty, manages all workloads very well and is very resilient to failures with a ZERO window of time when the data is not redundant. Block supports wide range of FABRICS and protocols (SAS, FC, IB, FCoE, Eth) in comparison to Object storage (Eth, IP). Block storage has good support for media tiering and accelerated caching using SSD technology. Block storage is complex and in most cases uses proprietary hardware. Block storage is more difficult to administer than object storage.
18. 18. 18 Copyright 2014 MPSTOR LTD. All rights reserved. Block ? Object, Both? or something else ? Is that the end of the debate? Who wins Block or Object ? Do we need both ? Do we need something else ? What are the hard limitations
19. 19. 19 Copyright 2014 MPSTOR LTD. All rights reserved. Block v/s Object semantics LBA@,LEN Volume SCSI CDB ByteArrayInputStream input = new ByteArrayInputStream("Hello World!".getBytes()); conn.putObject(bucket.getName(), "hello.txt", input, new ObjectMetadata()); S3 API
20. 20. 20 Copyright 2014 MPSTOR LTD. All rights reserved. Block, Object, Block over Object disk HD driver FS Mgt DB+Files API Head Block Layer Object Layer Block over Object Layer Obj App LBA Map 1 3 2 User Land Kernel S3 Backer Rados Block Device (RDB)
21. 21. 21 Copyright 2014 MPSTOR LTD. All rights reserved. Semantic cost & Performance 100% CPUUtilisation IO/sec 50K IO/sec @8K 300K IO/sec @8K Referece: http://www.mellanox.com/related- docs/whitepapers/WP_Deploying_Ceph_over_High_Performance_Networks.pdf
22. 22. 22 Copyright 2014 MPSTOR LTD. All rights reserved. Scale out storage 10G/16G/40G 8PB 8PB 8PB 8PB 192TB 128 1 40G/100G Scale out Cluster up to 10 nodes Exporting BLOCK FILE OBJECT Scale out Storage Nodes Up to 128 nodes
23. 23. 23 Copyright 2014 MPSTOR LTD. All rights reserved. Scale Out Scale out allows a storage service to scale in real time without service interruption in both capacity and performance Scale out storage can be Block (ScaleIO, Orkestra-IDA (information dispersal) Object (CEPH, SWIFT) File (GPFS, Gluster)
24. 24. 24 Copyright 2014 MPSTOR LTD. All rights reserved. Object IDA (Information Dispersal Algorithm) D#1 Proxy D#1 C#2 C#3 C#1 C#2 IO_Write Ring1 5 3 FIG 2 Zone1 Zone2 Zone3 Zone4 D 4 5 Proxy Load Balancer 2
25. 25. 25 Copyright 2014 MPSTOR LTD. All rights reserved. Block IDA (Information Dispersal Algorithm) D/3 VBS D/3 D/3 D/3 D/3 D/3 IO_Write Group1 5 3 FIG 2 Zone1 Zone2 Zone3 Zone4 D 3 3 VBS2 P P 4 VBD
26. 26. 26 Copyright 2014 MPSTOR LTD. All rights reserved. Storage Resiliency Object storage makes copies over (large window of non protection)across multiple storage arrays Weak real time consistent data Scale out block storage stores data redundantly in real time across multiple storage arrays Strong real time consistent data
27. 27. 27 Copyright 2014 MPSTOR LTD. All rights reserved. IDA Volume create RG1 RG2 RG3 RG4 RG1 RG2 RG3 RG4 RG1 RG2 RG3 RG4 RG1 RG2 RG3 RG4 Space is reserved according to the RAID QOS on independent Arrays and built into an RAID on the VBS layer Array 1 Array 2 Array 3 Array 4 IDA RAID IDA VOLVBS Node Storage Array Nodes QoS1 QoS2 QoS3 VBD
28. 28. 28 Copyright 2014 MPSTOR LTD. All rights reserved. Managed pools is not Scale-Out BASIC STANDARD 1G iSCSI 10G iSCSI 6G SAS 4/8/16G FC Fabrics Storage Containers PREMIUM Orkestra SDS Automation StoragePools ScaleoutComputeGroups Compute Containers SDS automates the provisioning of storage per group Each GROUP has a configurable QoS & SLA BRONZE compute GROUP SILVER compute GROUP GOLD compute GROUP
29. 29. 29 Copyright 2014 MPSTOR LTD. All rights reserved. Block & Object Storage efficiency 100% 50% 33% 100% 94% 88% 0% 20% 40% 60% 80% 100% 120% 1 2 3 StorageEfficiency # parity drives/copies Series1 Series2Block Object
30. 30. 30 Copyright 2014 MPSTOR LTD. All rights reserved. Multi-Tenancy using throttling MPSTOR Data Center Storage Automation SOS supports Bandwidth Throttling (both IOPS and MB/s)
31. 31. 31 Copyright 2014 MPSTOR LTD. All rights reserved. Feature Benefits Snapshot Allows users to take point of time copies Thin Provisioning Allows capacity to be added as demand increases Rate limiting Allows multi tenancy of high speed media Replication Resilient to failure Tiered Management Allows wide range of workloads Solid State Disk Caching Speeds up IO by caching High Availibility No single point of failure, no loss of functionality or data FC/FCoE SAN Automated management of high speed fabric SAS SAN Automated management of high speed fabric iSCSI SAN TAutomated management of high speed fabric NFS protocol File access protocol for LINUX CIFS protocol File access protocol for Windows Storage Features in the data-center NFS CIFS FC SAS iSCSI
32. 32. 32 Copyright 2014 MPSTOR LTD. All rights reserved. SOFTWARE DEFINED STORAGE Storage categories & Contenders Converged Storage Converged Infrastructure Server SAN VSAN Block File Object BIG IRON Evolving set of terms and definitions which are frequently a source of confusion
33. 33. 33 Copyright 2014 MPSTOR LTD. All rights reserved. ISSUES BLOCK OBJECT ? COMMENTS RESILIENCY Strong or weak real time consistency DIVERSE WORKLOADS Block performance TCO (Cloud CAPEX and OPEX) Use standard platforms for BLOCK or OBJECT with SDS automation STORAGE SECURITY ? Gaps exist SCALABILITY Scale out technologies exist for both Block and Object storage MULTIPLE CONSUMER TYPES ? Missing paradigm in both block , file and object storage for many of the datacenter consumer types Full Support Partial Support Poor Support One paradigm does not fit all
34. 34. 34 Copyright 2014 MPSTOR LTD. All rights reserved. Thank You