Post on 31-Aug-2019
transcript
0 © Fujitsu 2014
Webtalk Storage Trends
Sepp Stieger- Technology Watch Fujitsu
Webtalk Channel TechCommunity
New Storage Technologies– which ones are hype and reality?
How soon will these new trends impact your data center?
1 © Fujitsu 2014
Storage Trends
Trends / Disruptions
Storage System Challenges
Key Requirements
Fujitsus ETERNUS CD10000 Technology
OpenStack
2 © Fujitsu 2014
Storage Industry Distruption
Flash/ SSD
Cloud
Services
commercial-
on premise
Software
Defined
Convergence
e.g. Mobile Phones
Distributed
Architectures
Driving Transformation on Premise
Source: IDC 04/14
3 © Fujitsu 2014
IDC 3rd Platform: Opportunities at the intersection of Mobile, Cloud, Social and Big Data
From 2013 through 2020, 90% of IT industry growth will be driven by 3rd Platform technologies that, today, represent just 22% of ICT spending
Services will be build on innovative mash-ups of cloud, mobile devices/apps, social technologies, big data/analytics, and more
Data Center Transforming Converged systems will account for over 1/3 of
enterprise cloud deployments by 2016
Software-defined networks will penetrate 35% of Ethernet switching in the data center
Growing importance of mega DC, Service Source: IDC 12/12
4 © Fujitsu 2014
And the others?
This page is intentionally left blank. Please listen.
7 © Fujitsu 2014
Modular scalability of capacity and performance up to 100s of PB
Zero planned or unplanned downtime
Technology refreshes without system downtime or migration
Self-optimizing
Efficient management of exponential data growth
Lower total costs of capacity
The new unified data access (Block, File, Object, future access formats)
Key requirements for future storage architectures
10 © Fujitsu 2014
ETERNUS CD10000 Technology
11 © Fujitsu 2014
Next Generation Storage from Fujitsu
Copyright 2014 FUJITSU
ETERNUS CD10000
ETERNUS
CD10000
Unlimited
Scalability
Open
Standards
Cost
optimized
The new
unified
Immortal
System
Zero
Downtime
12 © Fujitsu 2014
Immortal System
Node1 Node2 Node(n)
+
Adding nodes
with new generation
of hardware
……… +
Adding nodes
New Node1
Non-disruptive add / remove / exchange of hardware (disks and nodes)
Mix of nodes of different generations, online technology refresh
Very long lifecycle reduces migration efforts and costs
13 © Fujitsu 2014
Node2
Zero downtime
Node1 Node2 Node(n)
Data is automatically distributed over disks and nodes in a self optimizing way
2,3,4.. data copies (replicas) protect against disk and node failures (instead of RAID)
In case of a disk or node failure lost data copies are automatically rebuilt
A very fast node interconnect enables a fast rebuild
Maintenance without any interruption
14 © Fujitsu 2014
Unlimited Scalability
Cluster of storage nodes
Capacity scales by adding storage nodes
Each nodes has integrated compute
power adding also performance
Three node types differing in capacity
and performance attributes –any mix
possible
Theoretically scalable to exabytes of data
1st version of CD10000 is released for
max. 224 nodes enabling a max. capacity
of 50 Petabyte
Basic node 12 TB Performance node 35 TB Capacity node 252 TB
15 © Fujitsu 2014
How Nature Overcomes the Central Access Bottleneck
Swarm of birds or fish Source: wikipedia
16 © Fujitsu 2014
How ETERNUS CD10000 Overcomes the Central Access Bottleneck
Disks assume the role of fish in
the storage swarm
every disk is represented by an
OSD (Object Storage Device)
Clients can directly access the
OSDs (disks, fish ) without a
central instance
19 © Fujitsu 2014
From Scalability to Reliability
CLIENT 1 VMs CLIENT 2 VMs
Redundant Client Interconnect (IP based)
Redundant Cluster Interconnect (IP based)
CLIENT 3 VMs CLIENT 4
VMs
Stor Node
Distributed Redundant Storage
Intelligent data Distribution
across all nodes and spindles
= wide striping (64KB – 16MB)
Redundancy with replica
= 2, 3 … 8
Thin provisioning
Fast distributed rebuild
Availability, Fault tolerance
Disk, Node, Interconnect
Automatic rebuild
Distributed HotSpare Space
Transparent Block, File access
Reliability and Consistency
Scalable Performance
Pure PCIe-SSD for extreme
Transaction processing
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Block Object Block Block
20 © Fujitsu 2014
From Reliability Back To Scalability
CLIENT 1 VMs CLIENT 2 VMs
Block
Redundant Client Interconnect (IP based)
Redundant Cluster Interconnect (IP based)
CLIENT 3 VMs
Object
CLIENT 4 VMs
Block
Stor Node
Distributed Redundant Storage
Add nodes without
reconfiguration
For more Performance
For more Capacity
Automatic rebalancing
avoids hot spots PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Block
Rebalance
Rebalance
Rebalance
Rebalance
21 © Fujitsu 2014
Seamless Technology Hardware Refresh
CLIENT 1 VMs CLIENT 2 VMs
Block
Redundant Client Interconnect (IP based)
Redundant Cluster Interconnect (IP based)
CLIENT 3 VMs
Object
CLIENT 4 VMs
Block
Add storage nodes based
on new technology
without reconfiguration
Seamlessly phase out old
technology storage nodes
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Stor Node
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
Block
Rebalance
Rebalance
Restore redundancy
Restore redundancy
22 © Fujitsu 2014
DRS HA/DR Design, up to 80km
CLIENT 1 VMs CLIENT 2 VMs
Object
Redundant Client Interconnect (IP)
Redundant Cluster Interconnect (IP)
CLIENT 3 VMs
Block
CLIENT n VMs
Object
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Block
Redundant Client Interconnect (IP)
Redundant Cluster Interconnect (IP)
DR Node
(at 3rd site)
2 replica will be placed locally
The 3rd/4th replica will go to the remote site
On a site failure, all data will be available
at the remote site to minimize the RTO
Recovery Time Objective
23 © Fujitsu 2014
DRS HA/DR Design, up to 80km
CLIENT 1 VMs CLIENT 2 VMs
Object
Redundant Client Interconnect (IP)
Redundant Cluster Interconnect (IP)
CLIENT 3 VMs
Block
CLIENT n VMs
Object
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Block
Redundant Client Interconnect (IP)
Redundant Cluster Interconnect (IP)
DR Node
(at 3rd site)
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
Stor Node
SSD
SSD
SAS
SAS
SAS
SAS
… 16 ×
Storage
Nodes
… 16 ×
Storage
Nodes
24 © Fujitsu 2014
Unified Storage Mgmt. for Block, Object, & File
Librados A library allowing
apps to directly
access RADOS,
with support for
C, C++, Java,
Python, Ruby,
and PHP
Ceph Object
Gateway
(RGW) A bucket-based
REST gateway,
compatible with
S3 and Swift
Ceph Block
Device
(RBD) A reliable and fully-
distributed block
device, with a Linux
kernel client and a
QEMU/KVM driver
Ceph File
System
(CephFS) A POSIX-compliant
distributed file
system, with a Linux
kernel client and
support for FUSE
App App
Object
Host / VM
Virtual Disk
Client
Files & Dirs
Ceph Storage Cluster (RADOS) A reliable, autonomous, distributed object store comprised
of self-healing, self-managing, intelligent storage nodes
33 © Fujitsu 2014
Software Defined Storage ETERNUS CD10000 Application Areas
34 © Fujitsu 2014
Application Areas for Scale-Out Storage
Cloud Services
Service Providers
Telcos
Scientific Computing
Universities
Research Institutions
Life Sciences, Chemical Industry, Automotive, Oil &
Gas, …
Multi Media
Video Surveillance
Broadcasting
35 © Fujitsu 2014
Cloud
Cloud Services
Focus on IaaS virtualization platforms with OpenStack /
KVM
Sync & Share
S3 compatible Object Store
Cloud service / Telco provider with fast growing
expansion strategy based on Open Souce
software
Enterprise companies with in-house cloud service and
additional focus on providing their own services for external
customers
36 © Fujitsu 2014
Scientific Computing
Universities, Research Institutions, Industries
depending on scientic computing (Life Sciences (e.g.
Genomic Sequencing), Weather Forecasting,
Climate Research, Automotive, Seismic Analysis, Oil
& Gas, …)
High performance storage for HPC
Mid to long term storage of larger data quantities
(historical data) addressing 500TB+ capacities
file system
object storage
37 © Fujitsu 2014
Multi Media
Video Surveillance
Media and Broadcasting companies
Fast online streaming services of PB`s of video and audio data
Realtime rendering and streaming process of high quality video data
Fast growing data environment and secure of consistent availability
Secure handling of PB data volumes - always available
HA- no downtime scenario
38 © Fujitsu 2014
Cloud Services
Public cloud offering with high degree of standardisation
Private cloud offering with significant portions of Open Source / Linux
Customers who are open for new infrastructure concepts (OpenStack, Ethernet
based storage access)
Customers interested in storing mid to long term data via an object oriented
interface (e.g. Amazon S3)
Scientific Computing / Multi Media
larger quantities of unstructured data
significant portions of inactive data
high throughput for sequential data
39 © Fujitsu 2014
OpenStack
40 © Fujitsu 2014
What is OpenStack?
Open Source Cloud Platform
Simple to implement
powerful implementation frameworks
Massively scalable
Fastest growing cloud platform
1000+ Developers
Biggest industry support
180+ Participating Organisations
Platin
Gold
Corporate
…
…
41 © Fujitsu 2014
OpenStack Cloud Layers
OpenStack Architecture
Physical Server (CPU, Memory, SSD, HDD) and Network
Base Operating System (Linux)
OAM
-dhcp
-Deploy
-LCM
Hypervisor
KVM (ESXi,
Hyper-V)
Compute (Nova)
Network
(Neutron) +
plugins
Dashboard (Horizon)
Billing Portal
OpenStack
Cloud APIs
RADOS
Block
(RBD)
S3
(Rados-GW)
Object (Swift) Volume (Cinder)
Authentication
(Keystone) Images (Glance)
EC2 API
Metering (Ceilometer)
42 © Fujitsu 2014
OpenStack Cloud Layers
OpenStack and ETERNUS CD10000
Physical Server (CPU, Memory, SSD, HDD) and Network
Base Operating System (Linux)
OAM
-dhcp
-Deploy
-LCM
Hypervisor
KVM (ESXi,
Hyper-V)
Compute (Nova)
Network
(Neutron) +
plugins
Dashboard (Horizon)
Billing Portal
OpenStack
Cloud APIs
RADOS
Block
(RBD)
S3
(Rados-GW)
Object (Swift) Volume (Cinder)
Authentication
(Keystone) Images (Glance)
EC2 API
Metering (Ceilometer)
Fujitsu
Storage
ETERNUS CD10000
43 © Fujitsu 2014
44 © Fujitsu 2014
Scale out storage – customer profiles
45 © Fujitsu 2014
Typical usage areas
Data volumes
500TB and beyond
Exponential and
unknown data growth
Typical usage areas
(Cloud) service providers
Telecommunications providers
Financial institutions
R&D, public institutions, universities Companies with large R&D activities
Public institutions with
huge document repositories
Media, broadcasting /
streaming companies
Business analytics tasks
needing fast access to large
scale (historical) data
46 © Fujitsu 2014
(Cloud) service providers
Typical usage scenarios
1 2 3 4 5
IaaS providers
Collaboration, Email, Sharepoint, DropBox services
(National) telco providers who are expanding in IT services
Large enterprises companies with in-house cloud services
IT organizations extending their DC usage in order to provide services for external customers as well
Providers with open source platform strategy (KVM, OpenStack, etc.)
47 © Fujitsu 2014
Typical usage scenarios
Companies, public institutions, universities etc. with large online archives 1 2 3 4 5
Need online fast access to large amounts of current and historical data,
e.g. research or legal documents, contracts
Use object data for efficient archiving
Reliable online storage for 500TB of data and beyond –when data migration to tape is not applicable
48 © Fujitsu 2014
Typical usage scenarios
Industry specific service providers (e.g. midsize banking, insurance companies) 1 2 3 4 5
High availability and a fast online store
Disaster recovery site for storage
Fast access for customer data and profiles
Zero downtime requirements
49 © Fujitsu 2014
Typical usage scenarios
Media and broadcasting/streaming companies 1 2 3 4 5
Fast online streaming services with petabytes of video and audio data / mediastores
Real-time rendering and streaming process of high quality video data
Creation of fast copies for media with high access frequency
Fast growing data environment; secure consistent availability
Secure handling of petabyte data volumes
Zero downtime requirements
50 © Fujitsu 2014
Typical usage scenarios
Big data/business analytics 1 2 3 4 5
Fast access to high volumes of historical data
High I/O performance to upload data into analytical servers for number crunching
Reduction of storage costs by using highly distributed data on disks instead of using SSDs
Using one storage system for production and analytics enabling so that current data can be analyzed
without slowing down productive environments
51 © Fujitsu 2014
Summary: typical usage scenarios
ETERNUS CD10000 for enterprise environments
High capacity demand
Unpredictable data growth
Keep high volumes even of historical data online
52 © Fujitsu 2014
ETERNUS CD10000 – Example Configurations
OpenStack and VDI
Video Surveillance and Broadcasting
DropBox, Synch&Share
OpenStack, VDI, DropBox, Synch&Share
53 © Fujitsu 2014
ETERNUS CD10000 example configuration (1)
Client VMs Client VMs
librbd
Redundant Client Interconnect (IP based)
Redundant Cluster Interconnect (IP based)
Client VMs
librbd
Client VMs
librbd
Perf Node
OpenStack compute, VDI
8 × Performance nodes
Raw: 8 × 6TB PCI-SSD = 48TB
Raw: 8 × 48TB SAS = 384TB
SSD as Cache pool with replica
= 3 ~16TB usable
2.5” SAS as Tiered pool with
EC ~275TB usable
Bandwidth ↑
IOPS ↑
Perf Node Perf Node Perf Node Perf Node Perf Node Perf Node Perf Node
librbd
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
Cache Pool Tiered Pool Capacity Pool
54 © Fujitsu 2014
ETERNUS CD10000 example configuration (3)
Client Apps Client Apps
Object
Redundant Client Interconnect (IP based)
Redundant Cluster Interconnect (IP based)
Client Apps
Object
Cap.Node
Dropbox, Sync & Share
4 × Capacity nodes
Raw: 4 × 0.8TB PCIe-SSD
Journal
Raw: 4 × 240TB SATA = 960TB
3,5” HDD as Capacity pool with
EC ~680TB usable
Bandwidth →
IOPS ↓
Cap.Node Cap.Node Cap.Node
Object
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
Cache Pool Tiered Pool Capacity Pool
55 © Fujitsu 2014
ETERNUS CD10000 example configuration (4)
Client VMs Client VMs
librbd
Redundant Client Interconnect (IP based)
Redundant Cluster Interconnect (IP based)
Client Apps
File
Client Apps
File
Perf Node
OpenStack compute, VDI
Dropbox, Sync & Share
4 × Performance,
4 × Capacity nodes
Raw: 8 × 6TB PCI-SSD = 48TB
Raw: 4 × 48TB SAS = 192TB
Raw: 4 × 240TB SATA = 960TB
SSD as Cache pool with replica
= 3 ~16TB usable
SAS 2.5” as Tiered pool with EC
~135TB usable
3,5” HDD as Capacity pool with
EC ~680TB usable
Bandwidth ↑
IOPS ↑
Perf Node Perf Node Perf Node Cap.Node Cap.Node CapaNode Cap.Node
librbd
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
PCI SSD
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
NL-SAS
SA NL-SAS TA
NL-SAS
NL-SAS
NL-SAS
NL-SAS
Cache Pool Tiered Pool Capacity Pool
56 © Fujitsu 2014
Next Generation Storage from Fujitsu
ETERNUS CD10000
ETERNUS
CD10000
Unlimited
Scalability
Open
Standards
Cost
optimized
The new
unified
Immortal
System
Zero
Downtime
57 © Fujitsu 2014
Unlimited Scalability
Cluster of storage nodes
Capacity scales by adding storage nodes
Each nodes has integrated compute
power adding also performance
Three node types differing in capacity
and performance attributes –any mix
possible
Theoretically scalable to exabytes of data
1st version of CD10000 is released for
max. 224 nodes enabling a max. capacity
of 50 Petabyte
Basic node 12 TB Performance node 35 TB Capacity node 252 TB