Post on 04-Dec-2014
description
transcript
1
Red Hat Storage Cluster
(GFS, CLVM, GNBD)
GFS allows multiple nodes to share storage at a block level as if the
storage were connected locally to each cluster node.
Schubert Zhang, Guangxian Liao
Jul, 2008
2
Deployment
GNBD Client
import
/dev/gnbd/x
CLVM/LVM
vg-lv
GFS
/mnt/gfs
GNBD Server
export
GNBD Server
export
TCP/IP Network
Block Devices
Disks...
Block Devices
Disks...
GNBD Server Node GNBD Server Node
Applications
GNBD Client
import
/dev/gnbd/x
CLVM/LVM
vg-lv
GFS
/mnt/gfs
Applications
GNBD Client
import
/dev/gnbd/x
GFS
/mnt/gfs
Applications
CLVM/LVM
vg-lv
GFS node
Storage Level
GFS node GFS node
“Economy and Performance” deployment.
3
Architecture
• Pay attention to the relationship of cluster software components. – Cluster Infrastructure (Common): CMAN, DLM, CCS, Fencing
– GNBD: Server and Client
– CLVM
– GFS
4
Cluster Infrastructure • We only concern cluster type for storage.
• Necessary infrastructure components: – Cluster management (CMAN)
• It’s a distributed cluster manager and runs in each cluster node.
• Keeps track of cluster quorum, avoids "split-brain". (Like chubby’s Paxos Protocol?)
• Keeps track of membership.
• libcman.so library, cman_tool
• dlm_controld: started by cman init script to manage dlm in kernel
• gfs_controld : started by cman init script to manage gfs in kernel
• groupd: started by cman init script to interface between openais/cman and dlm_controld/gfs_controld/fenced , group_tool
– Lock management (DLM) • To synchronize access to shared resources (shared storage, etc.).
• Runs in each cluster node.
• libdlm.so library.
• GFS and CLVM use locks from DLM.
• GFS uses locks from the lock manager to synchronize access to file system metadata (on shared storage).
• CLVM uses locks from the lock manager to synchronize updates to LVM volumes and volume groups (also on shared storage).
• Like chubby’s lock function?
5
Cluster Infrastructure (cont.)
– Cluster configuration management (CCS) • Runs in each cluster node. (ccsd)
• Synchronization/up-to-date of cluster configuration file. Propagates modification in a cluster.
• Other components (eg. CMAN) access configuration info via CCS.
• /etc/cluster/cluster.conf (Cluster Name, Cluster nodes, Fence, Resources, etc.)
• ccs_tool: to make online updates of CCS configuration files
• ccs_test: to retrieve information from configuration files through ccsd.
– Fencing • Fencing is the disconnection of a node from the cluster's shared storage.
Fencing cuts off I/O from shared storage, thus ensuring data integrity. The cluster infrastructure performs fencing through the fence daemon, fenced.
• GNBD fencing: fence_gnbd?
• fence_tool
• cluster configuration file: fencing-method, fencing agent, fencing device for each node in the cluster.
6
GNBD
• An ancillary component of GFS that exports
block-level storage to Ethernet.
• Global Network Block Device
– GNBD provides block-device access for Red Hat GFS
over TCP/IP. GNBD is similar in concept to NBD;
– GNBD is GFS-specific and tuned solely for use with
GFS.
• Two major components
– GNBD Server
– GNBD Client
7
GNBD (server)
• Exports block-level storage from its local
storage.
• gnbd_serv process
– GNBD server need not join the cluster
manager.
• gnbd_export
– Export block devices.
8
GNBD (client)
• A GNBD client runs in a node with GFS.
• Imports block device exported by GNBD server.
• Multiple GNBD clients can access a device exported by a GNBD server, thus making a GNBD suitable for use by a group of nodes running GFS.
• gnbd.ko – A kernel module
• gnbd_import – Import remote block devices form GNBD server.
9
CLVM
• Provides volume management of cluster storage.
• A cluster-wide version of LVM2
• CLVM provides the same capabilities as LVM2 on a single node, but makes the logical volumes created with CLVM available to all nodes in a cluster.
• CLVM uses the lock-management service provided by the cluster infrastructure.
• Using CLVM requires minor changes to /etc/lvm/lvm.conf for cluster-wide locking.
• clvmd: – A daemon that provides clustering extensions to the standard LVM2 tool
set and allows LVM2 commands to manage shared storage.
– Runs on each cluster node.
– Distributes LVM metadata updates in a cluster, thereby presenting each cluster node with the same view of the logical volumes
10
CLVM (cont.)
• lvm: LVM2 command line tools.
• /etc/lvm/lvm.conf
• pvcreate
– block devices/partitions -> PV
• vgcreate
– PV(s) -> VG
• lvcreate
– VG -> LV(s)
• Ready for make file system on LV.
11
GFS
• To simultaneously access a block device that is shared among the nodes.
• Single, consistent view of the FS name space across GFS nodes in a cluster.
• Native FS under VFS, POSIX interface to applications.
• Distributed metadata and multiple journals.
• Uses lock manager to coordinate I/O.
• When one node changes data on a GFS file system, that change is immediately visible to the other cluster nodes using that file system.
• Scale the cluster seamlessly by adding servers or storage on the fly.
• We use a “Economy and Performance” deployment.
12
GFS (cont.)
• gfs.ko: kernel module, loaded on each GFS cluster node.
• gfs_mkfs: create a GFS on a storage device.
• gfs_tool: configures or tunes a GFS.
• gfs_grow: grows a mounted GFS.
• gfs_jadd: adds journals to a mounted GFS.
• gfs_quota: manages quotas on a mounted GFS.
• gfs_fsck: repairs an unmounted GFS.
• mount.gfs: mount helper called by mount.
13
Fencing
• We must configure each GFS node in the
cluster for at least one form of fencing.
14
Setup a Cluster
(prepare) • Software Installation
– Install default packages of “Clustering” and “Storage Clustering” in each node.
– The rpms in cdrom: /ClusterStorage and /Cluster
– cman’s rpm in cdrom: /Server
– Major RPM (if need dependence, install depended rpm) • cman-2.0.60-1.el5.i386.rpm
• modcluster-0.8-27.el5.i386.rpm
• gnbd-1.1.5-1.el5.i386.rpm
• kmod-gnbd-0.1.3-4.2.6.18_8.el5.i686.rpm
• kmod-gfs-0.1.16-5.2.6.18_8.el5.i686.rpm
• lvm2-cluster-2.02.16-3.el5.i386.rpm
• Global_File_System-en-US-5.0.0-4.noarch.rpm
• gfs-utils-0.1.11-1.el5.i386.rpm
• gfs2-utils-0.1.25-1.el5.i386.rpm
• etc.
• Network – Disable firewall and SELinux
– Enable multicast and IGMP.
– Configure /etc/hosts or DNS for hostname (import!)
• Machine hostnames – 192.168.1.251 test1 (gnbd server)
– 192.168.1.252 test2 (gfs node)
– 192.168.1.253 test3 (gfs node)
– 192.168.1.254 test4 (gnbd server)
15
Setup a Cluster
(GNBD server) • In GNBD server node, need not start cman, i.e., the GNBD server is not a member of
the cluster.
(1) Start GNBD server process (man gnbd_serv) # gnbd_serv -n
(2) Export block device (man gnbd_export) # gnbd_export -v -d /dev/sda3 -e gnbdnode1 –c
Note1: must enable cache since no cman.
Note2: the block device should be a disk partition (how about LV? The document says LV is not supported).
(3) Check the export
# gnbd_export -l
(4) Add to /etc/rc.local gnbd_serv –n
gnbd_export -v /dev/sda3 -e gnbdnode1 -c
16
Setup a Cluster
(cluster infrastructure) • Initially configure a cluster
– /etc/cluster/cluster.conf: generated by “system-config-cluster” or manually. (only config clustername and node members)
– In one GFS node, use “system-config-cluster” to create a new cluster(name:cluster1) and add a node (name: test2)
– The cluster.conf <?xml version="1.0" ?>
<cluster alias="cluster1" config_version="5" name="cluster1">
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="test2" nodeid="1" votes="1">
<fence/>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices/>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
• Start infrastructure components – service cman start (refer to /etc/init.d/cman)
• Load kernel modules (configfs, dlm, lock_dlm)
• Mount configfs (I think it like chubby’s files ystem space)
• Start ccsd daemon
• Start cman (no daemon, use cman_tool join to join this node to cluster)
• Start daemons (start groupd, fenced, dlm_controld, gfs_controld)
• Start fencing (start fenced daemon, and use fence_tool to join this node to fence domain)
17
Setup a Cluster
(GNBD client) • Load kernel module gnbd.ko
# echo “modprobe gnbd” >/etc/sysconfig/modules/gnbd.modules
# chmod 755 /etc/sysconfig/modules/gnbd.modules
# modprobe gnbd
Then, the gnbd.ko will be loaded when the node boot-up.
• Import GNBD # gnbd_import -i test1
Then, we can find a block device /dev/gnbd/gnbdnode1, and it is same as /dev/gnbd0.
create a /etc/init.d/gnbd-client as a service script # chmod 755 /etc/init.d/gnbd-client
# chkconfig --add gnbd-client
Since gnbd_import -i should be done early then clvmd and gnbd_import –R should be done later then clvmd, we assign special start-number (23<24) and stop-number(77>76) in script /etc/init.d/gnbd-client.
Thus, the gnbd will be imported automatically when node boot-up.
18
Setup a Cluster
(CLVM) • Start clvmd
# service clvmd start
# chkconfig --level 35 clvmd
• pvcreate # pvcreate /dev/gnbd0, or
# pvcreate /dev/gnbd/gnbdnode1
Use lvmdiskscan or pvdisplay or pvscan to display status.
• vgcteare # vgcreate vg1 /dev/gnbd0
Use vgdisplay or vgscan to display status.
• lvcreate # lvcreate -l 100%FREE -n lv1 vg1
Use lvdisplay and lvscan to display status.
• Restart clvmd # service clvmd rerestart
clvmd is responsible for sync the lvm configuration among the cluster nodes.
Now, we can find a new logical volume block device @ /dev/vg1/lv1, and we can make file system on it.
19
Setup a Cluster
(GFS) • Make sure that the clocks on the GFS nodes are synchronized. (Use
NTP)
• Make GFS file system # gfs_mkfs -p lock_dlm -t cluster1:testgfs -j 4 /dev/vg1/lv1
Note: -j number same node number. One journal is required for each node that mounts a GFS file system. Make sure to account for additional journals needed for future expansion.
• Mount GFS # mkdir /mnt/gfs
(1) # mount -t gfs -o acl /dev/vg1/lv1 /mnt/gfs
(2) add to /etc/fstab
/dev/vg1/lv1 /mnt/gfs gfs defaults,acl 0 0
# mount -a -t gfs
# chkconfig --level 35 gfs
(refer to /etc/init.d/gfs)
Now, the GFS file system is accessible @ /mnt/gfs/
20
Setup a Cluster
(add a new GFS node) • Install packages in the new node
• Add new node information in cluster.conf in a existing node. – Use ccs_tool addnode, or
– Use system-config-cluster
• Copy (scp) the /etc/cluster.conf to new node.
• Import GNBD on the new node
• Stop components on all running node (if there are more than 2 node existing, need not do this step) # service gfs stop
# service clvmd stop
# service cman stop
• Then start components on all running node and the new added node # service cman start
# service clvmd start
# service gfs start
The clvmd will sync the metadata of logical volumes to the new added node, so, when clvmd is started, the /dev/vg1/lv1 will be visible on the now node.
21
Setup a Cluster
(add a new GNBD node) • Setup a new GNBD server (machine test4)
and export a new block device. # gnbd_serv -n
# gnbd_export -v -d /dev/sda3 -e gnbdnode2 –c
# gnbd_export -l
• Import the new GNBD (on all of the cluster nodes) # gnbd_import -i test4
# gnbd_import –l
• Make a new PV (on one of the cluster node) # pvcreate –v /dev/gnbd1
Then, we can find the new PV on all nodes in the cluster, by pvdisplay or lvmdiskscan or pvscan or pvs.
• Extend the VG (on one of the cluster node) # vgextend -v vg1 /dev/gnbd1
Then, we can find the extended VG and changed PV on all nodes in the cluster, by vgdisplay or vgfscan or vgs.
• Extend the LV (on one of the cluster node) # lvextend -v -l +100%FREE
/dev/vg1/lv1
Then, we can find the extended LV on all nodes in the cluster, by lvdisplay or lvfscan or lvs.
• Grow the GFS (on one of the cluster node) (next page)
# gnbd_import -l
Device name : gnbdnode1
----------------------
Minor # : 0
sysfs name : /block/gnbd0
Server : test1
Port : 14567
State : Open Connected Clear
Readonly : No
Sectors : 3984120
Device name : gnbdnode2
----------------------
Minor # : 1
sysfs name : /block/gnbd1
Server : test4
Port : 14567
State : Close Connected Clear
Readonly : No
Sectors : 4273290
22
Setup a Cluster
(grow the GFS)
• The gfs_grow command must be run on a
mounted file system. Only needs to be run
on one node in a cluster.
• Grow
# gfs_grow /mnt/gfs
• Sometimes after gfs_grow, df and
lvdisplay are hung. Need reboot the
system.
23
Evaluation
• Size – GFS is based on a 64-bit architecture, which can theoretically
accommodate an 8 EB file system. However, the current supported maximum size of a GFS file system is 25 TB. (But there is a note in Red Hat document: “If your system requires GFS file systems larger than 25 TB, contact your Red Hat service representative.”)
• Essential benchmarks – Refer to GlusterFS’s benchmarks:
http://www.gluster.org/docs/index.php/GlusterFS#GlusterFS_Benchmarks
• Iozone benchmarks – Refer to http://www.iozone.org
• Lock test – Always blocks the process when it set a new lock which conflict with
other process.
24
Conclusions
• The cluster infrastructure is too complex and weak, sometimes fail (such as cman, fencing, etc).
• The GNDB is simple and robust, but lacks flexibility.
• The CLVM is ok, but too complex to use.
• The GFS is very weak, sometime fail. (mount is ok, but umount often fail)
• The “two levels” (GNBDStorage+GFSCluster) deployment is not meet the “cloud” goal.
• Not easy to add a new GFS cluster node or a GNBD storage node.
• No data replicas for safety.
• Risks when a GNBD node fails. When one GNBD node fails, the data GFS is not accessible.
The Red Hat Cluster solution is not based on the Assumptions like GoogleFS and GlusterFS, (i.e. “The system is built from many computer that often fail.”), it is not easy and not safe to use in a moderate scale cluster. So, I think the Red Hat Storage Cluster is not a good designed solution, and has no good future.