2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Scaling Oracle with pNFS: Improving
Database Efficiency for Scale-out
Architecture
Bikash Roy Choudhury
Solutions Architect, NetApp
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Agenda
Comparison betweenNFSv3, NFSv4 and
NFSv4.1
How is pNFS different
pNFS Implementation
Oracle on NetApp Cluster-Mode(Scale-out)
Oracle, pNFS and NetApp Cluster-Mode
2
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Personality
Semantics
Authentication
Identification
Permissions
Transport
Caching
Data Locality
Stateful
Supports UNIX & WINDOWS
Strong (Kerberos)
Stringbased ([email protected])
WINDOWS like access
TCP Only
File Delegations
Referrals
NFSv4
Stateless
UNIX Only
Weak(AUTH_SYS)
32 bit UID/GID
UNIX based
UDP & TCP
Ad-hoc
No Remote Access
NFSv3
Stateful
Supports UNIX & WINDOWS
Strong (Kerberos)
Stringbased ([email protected])
More WINDOWS friendly
TCP Only
Files & Directory Delegations*
Referrals
NFSv4.1
Comparison of NFSv3, NFSv4 and NFSv4.1
3
NFSv4.1
• Is a minor release of NFSv4
• Does not modify any NFSv4 features
• Is only in DOT8.1 c-mode
• Bug fixes
• Supports pNFS
• Supports sessions and trunking *
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Challenges with regular NFS
Meta-data, data and control share the same
IO path
Single server bottleneck - Limited CPU
File system resides on a single NFS server
Restricts to distribute the workload
Single server design with a single IP
assigned to a single physical interface
With large file size data transmission
becomes the limiting factor
4
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
5
pNFS
pNFS Protocol
− Minor version of NFSv4.1
− Transparent to applications
Storage Access Protocol
− files (NFSv4.1)
− blocks (FC, iSCSI, FCoE)
− objects (OSD2)
Control Protocol
− Outside of the
pNFS standard
User Applications
NFSv4.1
Generic pNFS layout
File layout
OSD layout
Block layout
SUN RPC iSCSI
SCSI
TCP FCP RDMA
pNFS Protocol
Control Protocol
Metadata Server Data Servers
Client
Storage Access Protocol
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Regular NFS Vs. pNFS
pNFS with Data Ontap 8.1
6
Meta-data, Control,
Data use the same path
NFS
Single Stack NFS, Meta-Data, Data , Locks
NFS Client
Meta-Data Path
Data Path
Network
NFS pNFS
Control Path
Network
NFS pNFS
Data Meta-Data, Data
Locks
Data Meta-Data, Data
Locks
pNFS Client
pNFS Client
pNFS Client
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
pNFS Architecture and Core
Protocols
Isolates Meta-data from data into
different IO paths
Meta-data Server (MDS)
Handles all non-data; pure meta-data
File layout information
Data servers (DS)
Stores meta-data and data; no protocol meta-data operations
Clients read and write directly to data server
Need to have network/LIFs configured on each data server
7
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
pNFS Architecture and Core Protocols continued
pNFS client
Supported pNFS client (RHEL6.2)
File Layout driver
Protocols pNFS / 4.1 protocol for clients to access MDS
for retrieving layouts
Storage access protocol (NFSv4.1) for clients to access the DS
Control protocol to provide synchronization between MDS and DS
NetApp Confidential – Internal Use
Only
8
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
CLIENT
SIDE
SWITCH
pNFS implementation in Data Ontap
8.1
NetApp Confidential - Internal Use
Only
9
Meta-Data Path
Data Path
Control Path
N-Blade
pNFS VIFMgr
D-Blade Meta-Data, Data
Locks
N-Blade
pNFS VIFMgr
D-Blade Meta-Data, Data
Locks
pNFS Client
pNFS Client
pNFS Client
N-Blade
pNFS VIFMgr
D-Blade Meta-Data, Data
Locks
N-Blade
pNFS VIFMgr
D-Blade Meta-Data, Data
Locks
VLDB
VLDB
VLDB
VLDB
vol1
vol3
vol2
root
Cluster Interconnect Switch
Cluster Interconnect Switch
Cluster Interconnect Switch
LIF2
LIF1
LIF3
LIF4
LIF=<ip_address>
Mounting /vol1 using LIF1
Mounting /vol2 using LIF2
Mounting /vol3 using LIF4
Vol Move vol1
Client accessing /vol1
pNFS client
Mounting /vol1 using LIF3
root
vol1 vol2 vol3
Provides Data Locality
Client un -mount is not required
Transparent to client
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
pNFS workflow
The LIF that mounts on the client becomes the Meta-data server
(MDS)
pNFS operation flow
Opens the file with a OPEN() ,
Client gets the layout of the file, layout has information about DS
When a READ or WRITE request goes to DS
pNFS layout has information about
Constituents of a volume
For flexvol, there is only one constituent
Network location of constituent volumes
Basically, network address that is hosted on the same controller
as the volume
Nodes need to have a LIF configured for the client to get to the remote volumes
10
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Challenges With Traditional Storage
for Oracle
11
Data
Data
Logs
Capacity is limited within storage controller
IP is always attached to physical network ports
Re-distribution of load leads to disruption
While adding Oracle scales out (RAC), storage scales up
Moving data for more capacity, maintenance, load balancing needs downtime or outage
Managing disparate islands of storage are not efficient
home
NFS
Real Application Cluster
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Cluster-Mode or Scale-Out …
What is it buying for Oracle?
Increased flexibility − Storage virtualization – v-servers
− Cluster namespace
− Unified storage architecture
− Re-distribution of Oracle data across nodes
− IP address is not tied to physical network port
− Data Mobility and Non-disruptive operations
Tiered Storage
− Archival or less frequently used data
− Data for compliance purposes
− Isolate sequential workloads like logs & parallel queries
12
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
What’s more…
Agility
Quick deployment for new customers
Dynamically adjust to changing SLAs without user/
database disruption
Oracle databases with different performance
needs
With respect to cost and resiliency
characteristics
Scalability – Scale-up/scale-out
Capacity scaling
Spread out the flexvols across the nodes
Network load balancing 13
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Still provides the traditional efficiency
Storage efficiency
RAID-DP
Snapshot
Thin Provisioning
Space efficient Cloning
De-duplication
Data Compression
Data Protection
SnapMirror between v-servers
SnapMirror between clusters
14
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Icing on the cake…
Manageability
A single point of management
Tracking and monitoring
User interfacing tools like
SnapCreator 3.5 for
backup/restore/clone/replicate
15
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
NetApp
Cluster-mode
• Balance applications and
data among storage systems
Without Disruption!
• Add controllers and storage
Without Disruption!
• Move temporary resources
to critical applications –
Without disruption!
• Move data between types
of storage to optimize
storage utilization
Without Disruption!
Checklist for Oracle DB With Cluster-Mode
16
Storage– Seamless
Unified – NAS, SAN
Immort– Data is alive and active
Multi-T– Storage Virtualization
Mobilit– Data Motion of Volumes
Efficie – Parity with 7g/7-mode
Manag– Simple & Easy
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Oracle on Cluster-Mode with pNFS
Oracle can definitely benefit from pNFS
Less OPEN/CLOSE
Large IOs
Oracle over pNFS is not tested yet
Patches for o_direct just got out
Fedora17 and Linux3.5 kernel has it
Validating Oracle over pNFS is in the roadmap
17
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Thank You
18
2012 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
19