Module 3 - Continuent - Multi-Site Active-Active MySQL Clusters (15 Minutes)...............37Preparing the Environment for Module 3............................................................... 38Module Introduction .............................................................................................. 41MySQL Cluster Topology........................................................................................ 43Verify Cross-Cluster Data Replication .................................................................... 48Verify Data Replication across Topology changes.................................................. 56
Module 4 - Continuent - Big Data (Hadoop) Integration (15 Minutes) .............................62Preparing the Environment for Module 4............................................................... 63Module Introduction .............................................................................................. 65Verify Data Replication into Hadoop...................................................................... 67Verify Data Access in Hadoop via Hive.................................................................. 72Summary and For More Information...................................................................... 74
HOL-HBD-1686
Page 1HOL-HBD-1686
Lab Overview - HOL-HBD-1686 - VMwarevCloud® Air™ Data
Services with Continuent
HOL-HBD-1686
Page 2HOL-HBD-1686
Lab GuidanceHOL2015-1686 : "vCloud Air Data Services - Continuent"
Description:
We will demonstrate key capabilities of VMware Continuent around MySQL Clustering/HA- Clustering across a single site, near real-time replication/failover to a backup Clusteron a different site, Multi-Site Active-Active Clusters and Big Data (Hadoop) Integration.
MySQL Clustering across a single site to handle Planned and Unplanned downtime withno impact to Client Applications. Extending the same capabilities to a backup MySQLCluster on another site.
Module 3 : "Multi-Site Active-Active MySQL Clusters" [Advanced Level, 15minutes]
Active-Active MySQL Clusters across sites, to enable Client Applications to do databasereads/writes across sites in parallel.
Data replication near real-time from MySQL into Hadoop to enable Hadoop analytics tobe performed over real-time data.
Activation Prompt or Watermark
When you first start your lab, you may notice a watermark on the desktop indicatingthat Windows is not activated.
One of the major benefits of virtualization is that virtual machines can be moved andrun on any platform. The Hands-on Labs utilizes this benefit and we are able to run thelabs out of multiple datacenters. However, these datacenters may not have identicalprocessors, which triggers a Microsoft activation check through the Internet.
HOL-HBD-1686
Page 3HOL-HBD-1686
Rest assured, VMware and the Hands-on Labs are in full compliance with Microsoftlicensing requirements. The lab that you are using is a self-contained pod and does nothave full access to the Internet, which is required for Windows to verify the activation.Without full access to the Internet, this automated process fails and you see this
watermark.
This cosmetic issue has no effect on your lab. If you have any questions or concerns,please feel free to use the support made available to you either at VMworld in theHands-on Labs area, in your Expert-led Workshop, or online via the survey comments aswe are always looking for ways to improve your hands on lab experience.
HOL-HBD-1686
Page 4HOL-HBD-1686
Module 1 - Introduction toContinuent (30 Minutes)
HOL-HBD-1686
Page 5HOL-HBD-1686
Lab IntroductionBusiness needs around Agility and Customer Experience is driving demand for the nextgeneration of “SMAC” ("Social Mobile Analytics Cloud") Applications. IT Architects arelooking to leverage new and emerging IT solutions like Cloud Native Applications,Database as a Service (“DBaaS”), Platform as a Service (“PaaS”), etc. to deliver theseapplications.
VMware has launched a number of “Application Services” on its Public Cloud offering“vCloud Air” to enable IT Architects to build these applications. These ApplicationServices include a set of “Data Services” that customers can leverage to build the DataTier of their applications.
VMware has observed some key design patterns around the Data Tier, as customers arebuilding their next-generation Public Cloud-based “Cloud Native” applications:
• Applications are looking to leverage Relational Databases as a Managed Service(DBaaS), rather than installing/maintaining their own Database instances
• Applications are looking to seed these DBaaS instances from their existing On-Premise database instances, rather than start from scratch
• Applications are looking for the Public Cloud to provide Clustering/HA/DRcapabilities for the Data Tier, if such capabilities are not available/insufficientnatively within the database solution
This Hands On Lab will walk you through one of the key VMware solutions relevant tothese design patterns - VMware Continuent. VMware Continuent is a database clusteringand replication technology for vCloud Air and on-premises environments. It providesenterprise-class high availability, globally redundant disaster recovery, performancescaling and real-time heterogeneous data integration.
Some of its key benefits include:
• High Availability:Replace a failed master DBMS server within seconds whilemaintaining continuous availability and without losing any transactions.
• Zero-Downtime Maintenance:Perform database maintenance, such as schemachanges, upgrades to a new version of MySQL or adding new hardware, withoutapplication or service interruptions.
• Improved Performance:Increase replication performance 5x over native MySQLand increase transaction volumes using intelligent parallel methods. (VMwareinternal testing using 130GB/30 database sysbench test)
• Deployment Flexibility:Build database clusters in any bare-metal, private cloud(VMware, Open Stack) and public cloud (vCloud Air, Amazon AWS, Red HatOpenShift) environments.
For more information, please refer to http://www.vmware.com/products/continuent/
Continuent OverviewVMware Continuent allows enterprises running business-critical database applications toaffordably achieve commercial-grade high availability (HA), globally redundant disasterrecovery (DR), and performance scaling. VMware Continuent makes it simple to createdatabase clusters in the cloud or in your private data center, to keep the data availableeven when systems fail.
In addition, VMware Continuent provides data replication from relational databases tohigh-performance NoSQL and data analytics engines to derive insight from big data forbetter business decisions.
Key Use Cases & Benefits
This is a list of key benefits and use cases where Continuent can be used to solvebusiness & technical challenges.
• Deployment Flexibility - Build database clusters in any bare-metal, private cloud(VMware, Open Stack) and public cloud (vCloud Air, Amazon AWS, Red HatOpenShift) environments.
• Zero-Downtime Maintenance - Perform database maintenance, such as schemachanges, upgrades to a new version of MySQL or adding new hardware, withoutapplication or service interruptions.
• High Availability - Replace a failed master DBMS server within seconds whilemaintaining continuous availability and without losing any transactions.
• Improved Performance - Increase replication performance five times over nativeMySQL and increase transaction volumes using intelligent parallel methods.
• Geo-Clustering - Implement multi-master, cross-site database clusters that spansites and are ready for immediate failover.
• Disaster Recovery - Switch to a hot-standby disaster recovery site running aVMware Continuent cluster with a single command without losing applicationconnectivity.
• Affordable Oracle Replication - Replicate data from Oracle to MySQL, from MySQLto Oracle, and from Oracle to Oracle. Think “Oracle GoldenGate without the pricetag!”
• Real-Time Integration Between RDBMS and Analytics - Replicate quickly andefficiently to high-performance NoSQL and data analytics engines to deriveinsight from big data for better business decisions.
• Supports off-the-shelf MySQL, MariaDB and Percona Server - No softwareupgrade, data migration, application or DDL changes are required.
HOL-HBD-1686
Page 8HOL-HBD-1686
Continuent Overview Video (2 minutes 7 seconds)
Here is a short video that gives a simple visualization of the Continuent solution. Thisvideo contains audio. If you will be using headphones please make sure the plug ispushed all the way into the headphone jack. Low volume or no audio could be anindication that the headphones are not plugged in properly.
<div class="player-unavailable"><h1 class="message">An error occurred.</h1><div class="submessage"><ahref="http://www.youtube.com/watch?v=xUNCK8j3syo" target="_blank">Try watching this video on www.youtube.com</a>, or enableJavaScript if it is disabled in your browser.</div></div>
HOL-HBD-1686
Page 9HOL-HBD-1686
Module 2 - Continuent -Key Capabilities (30
Minutes)
HOL-HBD-1686
Page 10HOL-HBD-1686
Preparing the Environment for Module2To make this lab as seamless as possible and to ensure that users may go through eachmodule individually, a set of scripts have been created. At the beginning of each modulea script will need to be executed that sets up the topology for the lab. At any time youmay jump to the next module and run the corresponding setup script.
For this module you will execute a script named "module2". Continue on to the nextpage for the instructions.
HOL-HBD-1686
Page 11HOL-HBD-1686
Load PuTTY
1. Double-click the shortcut on the desktop to open our SSH client, PuTTY.2. Double-click the db1.vsphere.corp.local server name to open an ssh session to
that server.
Note: We've configured certificate authentication on the ControlCenter and allof the servers in this lab environment. This means that we do not have toenter in credentials to log into the systems when we launch an SSH sessionwhich should makes things a little easier.
HOL-HBD-1686
Page 12HOL-HBD-1686
Execute Module 2 Setup Script
1. Type the following command at the command prompt and hit the <enter> key:module2
2. The script will take a minute or two to complete. Once it completes and you areback to the command prompt type the following command and hit <enter>:source ~/.bash_profile
You'll notice that there is some output of the script (Cleanup & Install) during the firststep but there is no output after the second command. There should be no errors but ifthere are please seek assistance.
HOL-HBD-1686
Page 13HOL-HBD-1686
Close PuTTY
Now Module 2 is setup and ready. Please close PuTTY before moving on to the nextsection to start Module 2.
1. Click the Red X in the upper right of the PuTTY window.2. Click OK.
HOL-HBD-1686
Page 14HOL-HBD-1686
Module IntroductionThis module will guide you through the key capabilities of Continuent with regards toMySQL Clustering and Failover, within one site and across multiple (two) geographicsites.
1. We’ll exercise the following capabilities of Continuent:1. Ability to administratively reconfigure the MySQL Cluster topology without
impacting application uptime or any Client Applications connected to theCluster.
2. Ability to automatically discover failures of MySQL instances and promotenew MySQL instance to the Master role (as necessary), with no impact toClient Applications connected to the Cluster.
3. Ability to recover a previously failed MySQL instance and re-introduce itback into the Cluster.
4. Ability to administratively migrate Master to a different geographic site,with no impact to any Client Applications.
2. We will configure two Clusters of MySQL instances across two geographic sites.We will configure Continuent in a “Multi-Site Single-Master” topology. A single“Master” MySQL instance across the two sites will handle all Writes operations,transparent to the Client Applications. Read operations can be performed fromany MySQL instance across the two sites.
The diagram on the next page illustrates the topology.
For more information on various MySQL Clustering topologies supported by Continuent,please refer to https://www.vmware.com/files/pdf/products/continuent/VMware-Continuent-Benefits-And-Configurations-Whitepaper.pdf.
The estimated time to complete this module is 30 minutes.
MySQL Cluster TopologyLet’s first connect to our MySQL environment and examine its topology.
Load PuTTY
Open a PuTTY session to db1.vsphere.corp.local:
1. Double-click the shortcut on the desktop to open our SSH client, PuTTY.2. Double-click the db1.vsphere.corp.local server name to open an ssh session to
that server.
Note: We've configured certificate authentication on the ControlCenter and allof the servers in this lab environment. This means that we do not have toenter in credentials to log into the systems when we launch an SSH sessionwhich should makes things a little easier.
HOL-HBD-1686
Page 17HOL-HBD-1686
Examine MySQL Cluster Topology
1. Enter the following command at the command prompt within PuTTY and then hitthe <Enter> key: cctrl -multi
2. Enter ls (small L) when the “[LOGICAL] / >” prompt appears and then hit<Enter>.
You should now see information on the overall topology of our environment. We havetwo geographic sites “vsphere” and “vca” that are part of a Continuent Cluster called“global”.
“cctrl” is a Continuent administrative command that provides clustermanagement for your installed cluster, providing a command-line shellinterface to obtain information and manage your cluster and structure. "ls" isa command to list information about the current scope.
HOL-HBD-1686
Page 18HOL-HBD-1686
Examine MySQL Cluster Topology - Master & Slave
Here we'll take a look at the overall topology called global which consists of two sitescalled vsphere & vca.
1. Enter the following command and then hit the <Enter> key: use global2. Enter the following command and then hit the <Enter> key: ls3. You can see we have a "Composite" topology where the Master is running on the
site called vsphere.4. Verify the Slave is running on the site called vca.
Examine MySQL Cluster Topology - vsphere
We'll connect to the site named vsphere and find out a little bit about its topology. Notethat the screenshot is truncated due to size but you will see some additional informationon the other nodes in the vsphere site which are db2.vsphere.corp.local &db3.vsphere.corp.local.
1. Enter the following command at the command prompt with PuTTY and then hitthe <Enter> key: use vsphere
2. Enter the following command and then hit the <Enter> key: ls
The output will display the topology of the site called vsphere as shown in thescreenshot. Pay particular attention to the key information:
• “COORDINATOR” - member of the Cluster responsible for making decisions onthe state of the Cluster. It will not always be the Master.
• “CONNECTOR” - Continuent service that sits between Client applications and theMySQL Cluster. In our case, we have six Connectors across the two sites, which sitin front of each of the six MySQL instances. Continuent keeps track of the entire
HOL-HBD-1686
Page 19HOL-HBD-1686
topology so that it can disseminate any topology change information across allconcerned parties.
• “MASTER” - member of the Cluster (“db1.vsphere.corp.local” in this case) that ishandling all the Write transactions.
• “DATASOURCES” - members of the Cluster. In our case here, SITE_A Cluster hasthree members, out of which “db-01a.corp.local” is the Master. The field“progress” is a sequence number of the last record replicated to this MySQLinstance (should match or be close to the sequence number of the Master). Thefield “latency” is the applied latency of transactions: for Master, it’s thedifference between commit time and extraction time. For a Slave, differencebetween commit time in Master and commit time in this Slave.
HOL-HBD-1686
Page 20HOL-HBD-1686
Examine MySQL Cluster Topology - vca
1. Enter the following command at the command prompt with PuTTY and then hitthe <Enter> key: use vca
2. Enter the following command and then hit the <Enter> key: ls
This site has its own coordinator “db1.vca.corp.local” and note the role “relay” for theMySQL instance “db1.vca.corp.local”.
HOL-HBD-1686
Page 21HOL-HBD-1686
Close PuTTY
1. Before moving onto the next set of steps lets close PuTTY by hitting the X in theupper right of the PuTTY window.
2. Click the OK button.
HOL-HBD-1686
Page 22HOL-HBD-1686
Testing Graceful Failover OperationsIn this series of steps we'll take a look at failover operations with both an administrativefailover (graceful) and an automatic failover (ungraceful). An administrative failovermay occur when we have planned maintenance on our Master node so we'll gracefullyshutdown the site called vsphere and move the instance to the site called vca. Theautomatic failover would occur when the current Master node, vsphere in our case,experiences some sort of failure such as a server crash and the instance is moved tovca without intervention. This is considered an ungraceful shutdown and Continuentautomatically moves the instance for us in a consistent and safe manner.
Planned Migration
We will now administratively migrate the Master to a different MySQL instance andobserve how a client application seamlessly reconnects to the new Master.
Again, this is the typical process you would execute if you need to shutdown the Masterto perform any maintenance tasks (patching, upgrade, etc.).
HOL-HBD-1686
Page 23HOL-HBD-1686
Load PuTTY
In this case, since it is planned maintenance we should know which node is our Master- db1.vsphere.corp.local. Since we'll be taking that node offline, let's connect to ourMySQL instance through one of our Slave nodes.
1. Double-click the shortcut on the desktop to open our SSH client, PuTTY.2. Double-click the db2.vsphere.corp.local server name to open an ssh session to
that server.
HOL-HBD-1686
Page 24HOL-HBD-1686
Select Master Node
Open a MySQL connection to the local Continuent Connector service:
1. At the command prompt within PuTTY, type the Continuent command and hit the<enter key>: tpm connector
2. At the command prompt type the Continuent command and hit the <enter key>:select @@hostname;
Note: Be sure to include the semi-colon (;) at the end of the 'select @@hostname;' command.
You will see that there is a connection to db1.vsphere.corp.local (highlighed inyellow). The local Connector service automatically routed you to the Master instance.
Migrate the Master Node
Migrate the Master from db1.vsphere.corp.local to db3.vsphere.corp.local by executingthe following command:
1. At the command prompt, type the following command and hit the <enter key> toexit the MySQL cli: exit
2. At the command prompt, type the Continuent command and hit the <enter key>:cctrl -multi
HOL-HBD-1686
Page 25HOL-HBD-1686
3. At the command prompt, type the Continuent command and hit the <enter key>:use vsphere
4. At the command prompt, type the Continuent command and hit the <enter key>:switch to db3.vsphere.corp.local
5. Verify that you see that the migration was successful.
Now the MySQL instance is running on db3.vsphere.corp.local. As a reminder, this was agraceful, planned migration. Next we'll look at this same mode of migration but migratethe instance to a remote site.
Migrate the Master Node to the Remote Site
Here we will migrate our MySQL instance from the local site (vsphere) to the remote site(vca).
HOL-HBD-1686
Page 26HOL-HBD-1686
Migrate to vca
1. At the command prompt within PuTTY, type the Continuent command and hit the<enter key>: use global
2. At the command prompt type the Continuent command and hit the <enter key>:switch to vca
3. Verify the transfer was successful
HOL-HBD-1686
Page 27HOL-HBD-1686
Verify Master on vca
1. At the command prompt type the command and hit the <enter key>: exit2. At the command prompt type the command and hit the <enter key>: tpm
connector3. At the command prompt type the command and hit the <enter key>: select
@@hostname;[again, note the semi-colon (;) at the end of this command]4. You will see that we have been reconnected to the Master Node which is now
running on the remote site called vca on db1.vca.corp.local.
HOL-HBD-1686
Page 28HOL-HBD-1686
Close PuTTY
This concludes the current section. Let's close our PuTTY session to make sure we'reready for the next lesson.
1. Click the Red X in the upper right of the PuTTY window.2. Click OK.
HOL-HBD-1686
Page 29HOL-HBD-1686
Testing Automatic FailoverWe will now walk through the scenario where the Master crashes unexpectedly.Continuent will automatically detect the failure and migrate the Master to a differentnode within the Continuent cluster.
Simulating Failure
In order to test the scenario of a node failure we will stop the mysql service on thecurrent master node. Once the service is stopped Continuent will elect a new Masternode and migrate the MySQL instance to that node.
If you haven't skipped any steps the current Master node should be db1.vca.corp.local.We'll verify this so we make sure this task runs smoothly.
Let's begin!
HOL-HBD-1686
Page 30HOL-HBD-1686
Load PuTTY
1. Double-click the shortcut on the desktop to open our SSH client, PuTTY.2. Double-click the db1.vca.corp.local server name to open an ssh session to that
server.
HOL-HBD-1686
Page 31HOL-HBD-1686
Verify the Master Node
With a couple of commands we can confirm where the Master node is currently residing.
1. At the command prompt type the command and hit the <enter key>: tpmconnector
2. At the command prompt type the command and hit the <enter key>: select@@hostname;[again, note the semi-colon (;) at the end of this command]
3. You should see that we have the Master Node running on db1.vca.corp.local.
If you see a different node listed as the Master that's ok. You may proceed by openingup a PuTTY session to that node and continuing (substituting that node in the followinginstructions) or go back to the instructions to move the Master to db1.vca.corp.local.
Simulating Failure
Now that we've confirmed we're working with the Master node on db1.vca.corp.locallet's simulate a failure by stopping the mysql service.
1. At the command prompt type the command and hit the <enter key>: exit2. At the command prompt type the command and hit the <enter key>: sudo /etc/
init.d/mysql stop
HOL-HBD-1686
Page 32HOL-HBD-1686
You will see the message that the MySQL service is shutting down with "SUCCESS" afterjust a couple of moments.
HOL-HBD-1686
Page 33HOL-HBD-1686
Checking for a New Master Node
Now that the mysql service on db1.vca.corp.local has been stopped Continuent willelect a new master and move the MySQL instance to that new Master. Let's see whereour instance is running.
1. At the command prompt type the command and hit the <enter key>: tpmconnector
2. At the command prompt type the command and hit the <enter key>: select@@hostname;[again, note the semi-colon (;) at the end of this command]
3. You should see that we have the Master Node running on eitherdb2.vca.corp.local or db3.vca.corp.local.
HOL-HBD-1686
Page 34HOL-HBD-1686
Close PuTTY
This concludes Module 2. Let's close our PuTTY session to make sure we're ready for thenext lesson.
1. Click the Red X in the upper right of the PuTTY window.2. Click OK.
We hope you enjoyed this lesson and that you'll continue on to another module!
HOL-HBD-1686
Page 35HOL-HBD-1686
Next StepsPlease refer to http://docs.continuent.com/continuent-tungsten-4.0/deployment-composite.html for more information on Composite Cluster topology.
Please refer to http://docs.continuent.com/continuent-tungsten-4.0/introduction.html fordetails on the various member roles and responsibilities in a Cluster.
Preparing the Environment for Module3To make this lab as seamless as possible and to ensure that users may go through eachmodule individually, a set of scripts have been created. At the beginning of each modulea script will need to be executed that sets up the topology for the lab. At any time youmay jump to the next module and run the corresponding setup script.
For this module you will execute a script named "module3". Continue on to the nextpage for the instructions.
HOL-HBD-1686
Page 38HOL-HBD-1686
Load PuTTY
1. Double-click the shortcut on the desktop to open our SSH client, PuTTY.2. Double-click the db1.vsphere.corp.local server name to open an ssh session to
that server.
Note: We've configured certificate authentication on the ControlCenter and allof the servers in this lab environment. This means that we do not have toenter in credentials to log into the systems when we launch an SSH sessionwhich should makes things a little easier.
Execute Module 3 Setup Script
1. Type the following command at the command prompt and hit the <enter> key:module3
2. The script will take a minute or two to complete. Once it completes and you areback to the command prompt type the following command and hit <enter>:source ~/.bash_profile
You'll notice that there is some output of the script (Cleanup & Install) during the firststep but there is no output after the second command. There should be no errors but ifthere are please seek assistance.
HOL-HBD-1686
Page 39HOL-HBD-1686
Now Module 3 is setup and ready.
HOL-HBD-1686
Page 40HOL-HBD-1686
Module IntroductionContinuent provides you the ability to configure Multi-Site “Active-Active” MySQLClusters - this enables Client Applications to perform Write operations concurrentlyacross two geographic sites.
Please note Client Applications must be designed appropriately to performconcurrent Writes across two data centers. Some Best Practicesrecommended by Continuent can be found at http://docs.continuent.com/continuent-tungsten-4.0/deployment-msmm.html#deployment-msmm-prepare.
This topology is referred to as “Multisite - Multimaster" topology. The diagram on thenext page illustrates the topology we’ll configure in our lab.
For more information on various MySQL Clustering topologies supported by Continuent,please refer to https://www.vmware.com/files/pdf/products/continuent/VMware-Continuent-Benefits-And-Configurations-Whitepaper.pdf.
In this module, we’ll exercise the following capabilities of Continuent:
• Ability to configure bi-directional replication of data between 2 MySQL Clusters,where each Cluster is running on a separate geographic site.
• Ability of Client Applications to concurrently perform write operations againsttheir respective MySQL Clusters. And be able to read data written by the otherClient Application on the other site.
• Ability to detect and failover a crashed Master, with no impact to the ClientApplication performing the write operations or the replication ongoing with theother site.
MySQL Cluster TopologyLet’s first connect to our MySQL environment and examine its topology.
Load PuTTY
You should already have a PuTTY session opened to db1.vsphere.corp.local from theprevious step so you may skip this step. If you do not have a PuTTY session opened todb1.vsphere.corp.local please continue.
1. Double-click the shortcut on the desktop to open our SSH client, PuTTY.2. Double-click the db1.vsphere.corp.local server name to open an ssh session to
that server.
Note: We've configured certificate authentication on the ControlCenter and allof the servers in this lab environment. This means that we do not have toenter in credentials to log into the systems when we launch an SSH sessionwhich should makes things a little easier.
HOL-HBD-1686
Page 43HOL-HBD-1686
Examine Local MySQL Cluster topology
1. Enter the following command at the command prompt within PuTTY and then hitthe <Enter> key: cctrl -multi
2. Enter ls when the “[LOGICAL] / >” prompt appears and then hit <Enter>.
“cctrl” is a Continuent administrative command that provides clustermanagement for your installed cluster, providing a command-line shellinterface to obtain information and manage your cluster and structure. "ls" isa command to list information about the current scope.
You should now see information on the topology of our local site. Note the informationdisplayed in PuTTY is different from the previous module (Module 2) in that in theprevious instance we had configured a “Single Master” topology across the two sites. InModule 3 we have a Multisite Multimaster topology.
In our current setup, we have configured two MySQL Clusters, one each at the two sites,independent of each other. That is why we only see one site, vsphere, instead of bothwhen we perform the ls command above.
HOL-HBD-1686
Page 44HOL-HBD-1686
Examine Remote MySQL Cluster topology
Now let's open up a new PuTTY session to one of the remote servers. PuTTY will allowmultiple sessions so you do not need to close out of the prior session. To connect to theremote site you can choose any of the dbx.vca servers but we'll connect todb1.vca.corp.local just as an example.
1. Double-click the shortcut on the desktop to open our SSH client, PuTTY.2. Double-click the db1.vca.corp.local server name to open an ssh session to that
server.
HOL-HBD-1686
Page 45HOL-HBD-1686
Examine Remote MySQL Cluster Topology
1. Enter the following command at the command prompt within PuTTY and then hitthe <Enter> key: cctrl -multi
2. Enter ls when the “[LOGICAL] / >” prompt appears and then hit <Enter>.
You'll notice we only see the remote site, vca. To summarize module 3 up to this pointwe've just validated that we have 2 separate clusters each with their own Master nodes.
Let's go one step further and look at how this Multisite Multimaster topology workstogether.
Examine Global topology
Let's stick with the PuTTY session on db1.vca.corp.local. Before running this nextcommand, expand the PuTTY window so that the displayed data is a bit easier to read.You may maximize the window as well by clicking the square icon in the upper right ofthe PuTTY window.
1. Enter the following command at the command prompt within PuTTY and then hitthe <Enter> key: exit
2. Enter the following command at the command prompt within PuTTY and then hitthe <Enter> key: multi_trepctl --by-service
Before moving on, note the highlighted servers in the screenshot. These arethe Master nodes in this topology. They should be db1.vsphere.corp.local &db1.vca.corp.local. Due to the nature of the Hands On Labs, your Masternodes could be different and that's ok. Just note which nodes are listed as theMasters so you can use those in the following steps.
You'll notice that the output produced shows each of the six nodes as part of eachcluster. While at a local level, each cluster only sees its local nodes, at a global level we
HOL-HBD-1686
Page 46HOL-HBD-1686
can see here that the clusters use all the nodes so write operations to the database canbe concurrent across the sites providing a robust Active-Active infrastructure.
Through the next two sections we'll see how data replicates across the two sites andwhat happens in the event of a failure (hint: no impact to the application!)
HOL-HBD-1686
Page 47HOL-HBD-1686
Verify Cross-Cluster Data ReplicationNow let’s examine how Continuent replicates data between these two clusters. At thispoint you should still have two PuTTY sessions open - one to db1.vsphere.corp.local&db1.vca.corp.local. If you do not please re-open a PuTTY session to each of those twoservers.
Startup Client Applications for Both Clusters
From the previous step we know which nodes are our Masters. These Master nodes arethe nodes that our client application is connected to in order to write or read to ourMySQL database.
HOL-HBD-1686
Page 48HOL-HBD-1686
Select Master Node
In the PuTTY session connected to db1.vsphere.corp.local open a MySQL connection tothe local Continuent Connector service:
1. At the command prompt within PuTTY, type the Continuent command and hit the<enter key>: tpm connector
2. At the command prompt type the Continuent command and hit the <enter key>:select @@hostname; [Note semi-colon (;) at the end]
Note: Be sure to include the semi-colon (;) at the end of the 'select@@hostname;' command. If you omit the semi-colon (;) at the end, and unableto exit out of the erroneous condition, please do a Ctrl-C and restart with step1.
You will see that db1.vsphere.corp.local (highlighed in yellow) is still the Master.
Insert Schema into Local Cluster
We'll now simulate an application adding data to our database. Let's create a newSchema called vsphere_fooon the local vsphere cluster. Continue using your PuTTYsession that is connected to db1.vsphere.corp.local.
HOL-HBD-1686
Page 49HOL-HBD-1686
Make sure you are still in the mysql command line interface (CLI). Your PuTTY sessionshould show the mysql> as in the screenshot. If it does not, issue the tpm connectorcommand and hit <Enter>.
1. Enter the following command at the command prompt within PuTTY and then hitthe <Enter> key: create schema vsphere_foo;[Note semi-colon (;) at the end]
2. Enter the following command at the command prompt and hit the <Enter> key:show schemas; [Note semi-colon (;) again]
Note: Be sure to include the semi-colon (;) at the end of the commands. If youomit the semi-colon (;) at the end, and unable to exit out of the erroneouscondition, please do a Ctrl-C and restart with step 1.
You can now see that we've added the vsphere_foo schema to the database (highlightedarea). Let's see if it has been replicated to the remote site.
Data Replication to Remote MySQL Cluster
Switch to the PuTTY session connected to db1.vca.corp.local and open a MySQLconnection to the local Continuent Connector service:
1. At the command prompt within PuTTY, type the Continuent command and hit the<enter key>: tpm connector
2. At the command prompt type the Continuent command and hit the <enter key>:select @@hostname;
HOL-HBD-1686
Page 50HOL-HBD-1686
Note: Be sure to include the semi-colon (;) at the end of the 'select@@hostname;' command. If you omit the semi-colon (;) at the end, andunable to exit out of the erroneous condition, please do a Ctrl-C and restartwith step 1.
You will see that db1.vca.corp.local (highlighed in yellow) is the Master.
HOL-HBD-1686
Page 51HOL-HBD-1686
Data Replication to Remote MySQL Cluster
Let's check the remote MySQL cluster, vca, to make sure the schema we just createdhas replicated from the vsphere instance. Switch PuTTY windows so you're working inthe session connected to db1.vca.corp.local.
1. At the mysql> prompt type the following command and hit the <Enter> key:show schemas; [Note the semi-colon (;)]
Note: if you omit the semi-colon (;) at the end, and unable to exit out of theerroneous condition, please do a Ctrl-C and restart with step 1.
You see that the vsphere_foo schema (highlighted in yellow) has indeed been replicatedto the remote instance.
Data Replication to Local Site
This replication is bi-directional so let's create a new schema on the remote cluster(vca). After creating the new schema (vca_foo) we'll check that it appears in the schemalist on both the vca& vsphere clusters.
1. In the PuTTY session for db1.vca.corp.local type the following command at themysql> prompt and hit the <Enter> key: create schema vca_foo; [Note thesemi-colon (;)]
HOL-HBD-1686
Page 52HOL-HBD-1686
2. At the mysql> prompt type the following command and hit the <Enter> key:show schemas; [Note the semi-colon (;)]
3. Switch to the PuTTY session connected to db1.vsphere.corp.local and type thefollowing command at the mysql> prompt and hit the <Enter> key: showschemas;[Note the semi-colon (;)]
Note: if you omit the semi-colon (;) at the end, and unable to exit out of theerroneous condition, please do a Ctrl-C and restart with the step.
You will see the schema vca_foo (highlighted) in both lists as shown in the screenshot.
Row-level Replication Across Sites
Now let’s insert some data into some schemas and observe how it gets replicated to theother site.
On the local site (vsphere), we’ll create a new table called “names” and insert two rowsinto it. Shortly thereafter, this data will be visible on the remote site (vca). For reference,here are the MySQL commands we’ll execute which you can copy-paste from theREADME.txt on the desktop into your mysql> command prompt. Note that in Putty,you can simply right click to paste the previously copied text.
use vsphere_foo;
create table names (id int primary key auto_increment, name varchar(32));
HOL-HBD-1686
Page 53HOL-HBD-1686
insert into names (name) values ('jon'), ('alan');
select * from vsphere_foo.names;
Be sure to use the PuTTY session for db1.vsphere.corp.local for the following series ofcommands.
1. At the mysql> prompt type the following command and hit the <Enter> key: usevsphere_foo;
2. At the mysql> prompt type the following command and hit the <Enter> key:create table names (id int primary key auto_increment, name varchar(32));
3. At the mysql> prompt type the following command and hit the <Enter> key:insert into names (name) values ('jon'), ('alan');
4. At the mysql> prompt type the following command and hit the <Enter> key:select * from vsphere_foo.names;
5. You will see your table with your two rows containing 'jon' and 'alan'.
HOL-HBD-1686
Page 54HOL-HBD-1686
Row-level Replication Across Sites
Now let's check our vca cluster to make sure the table and its data have been replicatedover. Be sure to use the PuTTY session for db1.vca.corp.local for the followingcommands.
1. At the mysql> prompt type the following command and hit the <Enter> key:select * from vsphere_foo.names;
2. You will see that the table with the two rows containing 'jon' and 'alan' have beenreplicated successfully.
HOL-HBD-1686
Page 55HOL-HBD-1686
Verify Data Replication acrossTopology changesWe’ve seen how Continuent is able to replicate data across the two sites. Let’s nowintroduce changes to the Cluster topology and observe how the replication is able tocontinue seamlessly across these changes.
HOL-HBD-1686
Page 56HOL-HBD-1686
Examine Replication Configuration
We'll first take a look at our local site, vsphere. You should already have a PuTTYconnection open to db1.vsphere.corp.local. The following procedures produce quite a bitof text so you may want to increase the size of your PuTTY window by clicking theMaximize button in the upper right of the window.
1. At the mysql> prompt type exit and hit the <Enter> key2. Enter the following command at the command prompt and hit the <Enter> key:
mm_trepctl status
The “mm_trpctl status” command is used to obtain details of the replicationconfiguration at the current host. We can see potential replication partners, whichone(s) we are currently replicating from, and the status of replication.
Observe the following configuration settings and metrics (highlighted in the screenshot):
• appliedLastSeqno: last replication record received• masterConnectUri: list of MySQL instances on the other site we can replicate
from• pipelineSource: MySQL instance on the other site we are obtaining replicated
data from
HOL-HBD-1686
Page 57HOL-HBD-1686
Remote Site Topology Change
Let's switch over to our remote site, vca, and create a topology change by moving theMaster role to a different node in the cluster. Make sure you switch to your PuTTYsession that is connected to db1.vca.corp.local.
1. At the mysql> prompt type exit and hit the <Enter> key2. Enter the following command at the command prompt and hit the <Enter> key:
cctrl -multi3. Enter the following command at the command prompt and hit the <Enter> key:
use vca[Note: This ensures we're using the correct site. If you receive an errorthen you may be using the wrong PuTTY session]
4. Enter the following command at the command prompt and hit the <Enter> key:switch to db3.vca.corp.local
Observe from the output that the new Master is db3.vca.corp.local and thatdb1.vca.corp.local is now a Slave node in the cluster. Let's find out how this has affectedour replication.
Examine Remote Site Replication Configuration
We'll check the replication status of the vca site and verify that replication is stillhappening as expected even with the cluster topology change from the previous step.Since the following command will display a large amount of information it may help toMaximize the PuTTY window just as we did in the previous step.
1. At the [LOGICAL] /vca > prompt type ls and hit the <Enter> key.2. Locate the node which is the current Master: db3.vca.corp.local. There are a few
highlighted areas which are important to note. First, we can see thatdb3.vca.corp.local is indeed the Master node and that it is Online. The seconditem of interest is the progress=6 item. This correlates to the appliedLastSeqno
HOL-HBD-1686
Page 58HOL-HBD-1686
from the mm_trepctl status command. This should now be consistent betweenthe two sites and we'll validate that in the next step. The last item of interest isthe Status [OK] message showing that the cluster and replication is functioningnormally. Also note that we don't necessarily need to look at the Master node tosee this information. It is consistent across the Slave nodes as well.
Note that the number you show for progress and appliedLastSeqno may not be 6 due to the nature of the HOL. But whatever that number is should be consistent across the clusters. For example, you may see progress=8 in your particular lab. That means you should also show appliedLastSeqno to be 8 in the next step.
HOL-HBD-1686
Page 59HOL-HBD-1686
Observe Updated Replication Topology
Now go back to the local site, vsphere, and you’ll observe the replication continuedthrough the Master migration at the remote site, vca. The replication topology onvsphere was automatically adjusted based on the topology change we introduced in theprevious step.
1. Enter the following command at the command prompt and hit the <Enter> key:mm_trepctl status
2. Observe the appliedLastSeqno and verify it matches the progress= number fromthe prior step.
3. Observe the pipelineSource parameter.
As you can see we are now replicating from db1.vca.corp.local and the replication iscurrent - last replicated record had a sequence # of 6. (Reminder, your exact numbermay be different but it should be the same number as shown in the progress= area ofthe prior step.)
Close PuTTY
This concludes Module 3. Let's close our PuTTY session to make sure we're ready for thenext lesson.
1. Click the Red X in the upper right of the PuTTY windows.2. Click OK.
HOL-HBD-1686
Page 60HOL-HBD-1686
We hope you enjoyed this lesson and that you'll continue on to another module!
HOL-HBD-1686
Page 61HOL-HBD-1686
Module 4 - Continuent -Big Data (Hadoop)
Integration (15 Minutes)
HOL-HBD-1686
Page 62HOL-HBD-1686
Preparing the Environment for Module4To make this lab as seamless as possible and to ensure that users may go through eachmodule individually, a set of scripts have been created. At the beginning of each modulea script will need to be executed that sets up the topology for the lab. At any time youmay jump to the next module and run the corresponding setup script.
For this module you will execute a script named "module4". Continue on to the nextpage for the instructions.
Load PuTTY
1. Double-click the shortcut on the desktop to open our SSH client, PuTTY.2. Double-click the db1.vsphere.corp.local server name to open an ssh session to
that server.
Note: We've configured certificate authentication on the ControlCenter and all of the servers in this lab environment. This means that we do not have to enter in credentials to log into the systems when we launch an SSH session which should makes things a little easier.
HOL-HBD-1686
Page 63HOL-HBD-1686
Execute Module 4 Setup Script
1. Type the following command at the command prompt and hit the <enter> key:module4
2. The script will take a minute or two to complete. Once it completes and you areback to the command prompt type the following command and hit <enter>:source ~/.bash_profile
You'll notice that there is some output of the script (Cleanup & Install) during the firststep but there is no output after the second command. There should be no errors but ifthere are please seek assistance.
HOL-HBD-1686
Page 64HOL-HBD-1686
Module IntroductionContinuent provides you the ability to deliver Real-Time Integration between RDBMS andAnalytics. You can quickly and efficiently replicate data from your Relation Database intoNoSQL and data analytics engines. You can thus gain insights from big data to makebetter business decisions.
Please note Continuent supports this capability for MySQL and Oracle databases.
In this Module, we will demonstrate Continuent’s ability to replicate from MySQL toHadoop:
• We’ll configure 1 MySQL Cluster on Site A and a Hadoop cluster on Site B.• Client will do write operations to the MySQL Cluster.• You will observe data written previously by Client showing up on Hadoop HDFS
file system.
The graphic on the next page shows the flow of data from the MySQL database intoHadoop:
HOL-HBD-1686
Page 65HOL-HBD-1686
Data Flow from Relation Database into Hadoop
HOL-HBD-1686
Page 66HOL-HBD-1686
Verify Data Replication into HadoopWe will now insert some records into the MySQL database and observe them beingreplicated over to Hadoop.
Verify MySQL Cluster topology
1. Verify connectivity to MySQL and cluster topology by typing the followingcommand at the command prompt and hit the <enter> key: cctrl
2. Enter ls (small L) when the “[LOGICAL] / >” prompt appears and then hit<Enter>.
HOL-HBD-1686
Page 67HOL-HBD-1686
Generate Cluster Heartbeats
Now we'll generate some cluster heartbeat events. The cluster heartbeat commandsends a heartbeat event through the local cluster to demonstrate that all replicators areworking. This increments the sequence number which validates the cluster is workingproperly.
1. Generate a cluster heartbeat by typing the following command at the prompt thatsays "[LOGICAL] /vsphere >" and hit the <enter> key: cluster heartbeat
2. Repeat the cluster heartbeat command3. Exit out of "cctrl" command by typing "exit" and hitting the <enter> key.
HOL-HBD-1686
Page 68HOL-HBD-1686
Insert data into MySQL database
1. Insert data into the database by typing the following command on the commandprompt and hit the <enter> key: tungsten_create_load --threads=10
Hit "Ctrl-C" after about 10 seconds, as sufficient data would've been inserted for our usecase.
2. Verify data insertion by typing the following command on the command prompt andhit the <enter> key: trepctl status
3. Observe the value of field "appliedLastSeqno", that's how many records were insertedinto the database. The number in your lab will vary from the screenshot based on howlong you let the load script run in step 1.
HOL-HBD-1686
Page 69HOL-HBD-1686
Verify data records in MySQL database
Verify the data records in the database by running the commands below:
1. tpm db2. show schemas;3. use tungsten_create_load;4. show tables;5. select * from tungsten_create_load.values;
Verify data records on Hadoop server
Now verify the records have been replicated over to the server running Hadoop. Log intothe "hdfs" server and check the replication status and HDFS, by running the commandsbelow:
HOL-HBD-1686
Page 70HOL-HBD-1686
1. ssh hdfs2. trepctl status
Verify data records within Hadoop Filesystem (HDFS)
Finally, we can verify the records have been inserted into the Hadoop filesystem (HDFS),for subsequent analytics operations. This can be done by executing the command"hadoop fs -ls /user/tungsten/staging/vsphere/tungsten_create_load/values", as above.
HOL-HBD-1686
Page 71HOL-HBD-1686
Verify Data Access in Hadoop via HiveApache Hive™ is a data warehouse software facility for querying and managing largedatasets residing in distributed storage. Hive provides a mechanism to project structureonto this data and query the data using a SQL-like language called HiveQL. At the sametime this language also allows traditional map/reduce programmers to plug in theircustom mappers and reducers when it is inconvenient or inefficient to express this logicin HiveQL.
As with any other data set in Hadoop, you can access the data replicated in viaContinuent with Hive as well, by executing the commands highlighted below.
As these commands are quite long, you can copy & paste them from this manualthrough the Send Text button in the upper left of your browser window or also find themin the README.txt file where you can also copy/paste them from.
Create DB view in Hive
Enter the following command as shown above: sudo hive -e "CREATE DATABASEvsphere_tungsten_create_load;"
1. sudo hive2. select * from vsphere_tungsten_create_load.stage_xxx_values;
This concludes Module 4 and this Hands On Lab. Thank you!
HOL-HBD-1686
Page 73HOL-HBD-1686
Summary and For More InformationVMware Continuent provides data replication from relational databases to high-performance NoSQL and data analytics engines to derive insight from big data for betterbusiness decisions. In addition, VMware Continuent allows enterprises running business-critical database applications to affordably achieve commercial-grade high availability(HA), globally redundant disaster recovery (DR) and performance scaling. VMwareContinuent makes it simple to create database clusters in the cloud or in your privatedatacenter, to keep the data available even when systems fail.
For more information and to sign-up for your free 60-day trial, visit usat http://www.vmware.com/products/continuent/. Or connect with our specialists for adiscovery call to discuss your specific needs and environment at [email protected].
