· Table of Contents i DCA Expansion Guide – Contents DCA Expansion Guide - Contents Preface

0

EMC® Greenplum® Data Computing ApplianceExpansion Guide

Appliance Version 1.2.0.

P/N: 300-013-848Rev: A01

EMC Confidential - for EMC use only

Copyright © 2012 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com

All other trademarks used herein are the property of their respective owners.

DCA Expansion Guide – Contents

DCA Expansion Guide - ContentsPreface ............................................................................................... 1

About This Guide.............................................................................. 1Document Conventions .................................................................... 1

Text Conventions........................................................................ 1Command Syntax Conventions ................................................... 2

Getting Support ............................................................................... 3Product information .................................................................... 3Technical support ....................................................................... 3

Chapter 1: Greenplum DCA Expansion Overview.................... 4Expansion task overview .................................................................. 4Expansion Requirements .................................................................. 5

Chapter 2: Add additional hardware .......................................... 6Identify expansion order and existing configuration ......................... 6Install and cable new hardware........................................................ 7

Determine population order ........................................................ 7Attach rails to the new server ..................................................... 8Install the server in the cabinet .................................................. 8Cable new servers to switches and power................................... 9

Position and connect new racks.......................................................10Position new rack.......................................................................10Add aggregation switches for 7+ racks ......................................11Connect new racks ....................................................................12

Label new servers and record MAC addresses .................................12Connect new bays to the power supply ...........................................14

Upgrade an existing rack to 2 or more modules.........................14Connect power to new rack .......................................................18

Chapter 3: Configure switches and image new hardware ..19Set switch IP addresses ..................................................................19

Configure the Administration switch...........................................19Configure Brocade 8000 Interconnect switches..........................20Configure aggregation switches .................................................21

Image Servers ................................................................................22

Chapter 4: Expand and Validate Greenplum DCA ..................24Expand with DCA Setup ..................................................................24Validate system performance ..........................................................25Validate the Greenplum DCA Host Operating Systems.....................26

Chapter 5: Expanding Greenplum Database ...........................27Stop health monitoring and revert secure mode..............................27Back up your Greenplum Databases................................................28Change failure group mapping ........................................................28Plan the Greenplum Database expansion ........................................29

Set expectations with the customer ...........................................29Evaluate customer data and space requirements .......................30

Initialize new segment resources ....................................................31Rebalance table data.......................................................................32

Set the order of table rebalancing..............................................32

Table of Contents i


Run table rebalancing ................................................................33Remove the expansion schema..................................................33

Resume health monitoring, secure mode and restart GPDB.............34

Chapter 6: Configure Greenplum DIA Modules ......................35Configure external network access ..................................................35Restart Greenplum Database ..........................................................36Configure the Expanded DIA Modules for Loading ...........................36

Chapter 7: Expand Hadoop modules .........................................37Set VLAN IP addresses on new hosts .........................................37Restart Hadoop and Greenplum Database..................................38

Appendix A: Connect to the DCA ...................................................39Laptop Prerequisites........................................................................39Connecting Your Laptop to the DCA Admin Network........................39

Laptop is running Windows 7 .....................................................39Laptop is running Windows XP ...................................................41

Connecting via SSH to the Greenplum Master .................................41Putty Example ...........................................................................42Cygwin Example ........................................................................42

Using SCP to Copy a File to the Greenplum Master..........................43WinSCP Example .......................................................................43Cygwin Example ........................................................................44

Use Putty to Create a Serial Connection ..........................................45

Appendix B: DCA reference information .......................................47Greenplum DCA configurations........................................................47

DCA configuration rules .............................................................47System rack components...........................................................49Aggregation Rack Components ..................................................50Expansion Rack Components .....................................................51

Power Supply Reference..................................................................52Network and Cabling Configurations................................................53

Interconnect Switch Reference ..................................................53Administration Switch Reference................................................57Network Hostname and IP Configuration ...................................59Multiple Rack - Aggregation Cabling...........................................62Multiple Rack Cabling Information .............................................64

Configuration Files ..........................................................................66Default Passwords...........................................................................67

Appendix C: Working with the Integrated Dell Remote Access Controller (iDRAC) .............................................................................................68

Default iDRAC IP Addresses for the DCA .........................................68Setting the iDRAC IP .......................................................................68Accessing a Server via its iDRAC Interface ......................................70Create SSH Tunnel to the iDRAC Web Interface ..............................71

Appendix D: Linux and vi Command Reference...........................75Common Linux Commands..............................................................75vi Quick Reference ..........................................................................77

Table of Contents ii


Appendix E: Enable Secured Mode ................................................78Enable Secured Mode on the DCA ...................................................78Revert to Unsecured Mode on the DCA............................................79

Appendix F: Brocade VDX-6720 configuration .............................80

Table of Contents iii

DCA Expansion Guide – Preface

Preface

This guide is intended for EMC field personnel responsible for expanding an Greenplum Data Computing Appliance (Greenplum DCA) with GPDB, DIA or HD modules. This guide provides instructions on adding additional hardware resources to a Greenplum DCA, configuring the operating system of the new resources, validating the configuration and expanding Greenplum Database on the Greenplum DCA to make use of the new resources.

• About This Guide

• Document Conventions

• Getting Support

About This GuideThis guide provides procedural and reference information to help technical field personnel and system administrators expand a Greenplum Data Computing Appliance and Greenplum Database system or Greenplum Data Integration Accelerator. The instructions in this guide reference installation procedures in the SVR-D2U-R510 Installation and Replacement Guide (P/N 300-011-447).

This guide assumes knowledge of Linux/UNIX system administration, network administration, and database administration.

Document ConventionsThe following conventions are used throughout the Greenplum Database documentation to help you identify certain types of information.

• Text Conventions

• Command Syntax Conventions

Text Conventions

Table 0.1 Text Conventions

Text Convention Usage Examples

bold Button, menu, tab, page, and field names in GUI applications

Click Cancel to exit the page without saving your changes.

italics New terms where they are defined

Database objects, such as schema, table, or columns names

The master instance is the postgres process that accepts client connections.

Catalog information for Greenplum Database resides in the pg_catalog schema.

About This Guide 1



Command Syntax Conventions

monospace File names and path names

Programs and executables

Command names and syntax

Parameter names

Edit the postgresql.conf file.

Use gpstart to start Greenplum Database.

monospace italics Variable information within file paths and file names

Variable information within command syntax

/home/gpadmin/config_file

COPY tablename FROM 'filename'

monospace bold Used to call attention to a particular part of a command, parameter, or code snippet.

Change the host name, port, and database name in the JDBC connection URL:

jdbc:postgresql://host:5432/mydb

UPPERCASE Environment variables

SQL commands

Keyboard keys

Make sure that the Java /bin directory is in your $PATH.

SELECT * FROM my_table;

Press CTRL+C to escape.

Table 0.1 Text Conventions


Table 0.2 Command Syntax Conventions


{ } Within command syntax, curly braces group related command options. Do not type the curly braces.

FROM { 'filename' | STDIN }

[ ] Within command syntax, square brackets denote optional arguments. Do not type the brackets.

TRUNCATE [ TABLE ] name

... Within command syntax, an ellipsis denotes repetition of a command, variable, or option. Do not type the ellipsis.

DROP TABLE name [, ...]

Document Conventions 2



Getting SupportEMC support, product, and licensing information can be obtained as follows.

Product information

For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Powerlink website (registration required) at:

http://Powerlink.EMC.com

Technical support

For technical support, go to Powerlink and choose Support. On the Support page, you will see several options, including one for making a service request. Note that to open a service request, you must have a valid support agreement. Please contact your EMC sales representative for details about obtaining a valid support agreement or with questions about your account.

| Within command syntax, the pipe symbol denotes an “OR” relationship. Do not type the pipe symbol.

VACUUM [ FULL | FREEZE ]

$ system_command

# root_system_command

=> gpdb_command

=# su_gpdb_command

Denotes a command prompt - do not type the prompt symbol. $ and # denote terminal command prompts. => and =# denote Greenplum Database interactive program command prompts (psql or gpssh, for example).

$ createdb mydatabase

# chown gpadmin -R /datadir

=> SELECT * FROM mytable;

=# SELECT * FROM pg_database;

Table 0.2 Command Syntax Conventions


Getting Support 3




DCA Expansion Guide – Chapter 1: Greenplum DCA Expansion Overview

1. Greenplum DCA Expansion Overview

Expanding Greenplum DCA modules increases the scan rate, data ingest rate, and storage capacity of the DCA. All aspects of the expansion process are handled by EMC field personnel. The following sections are included in this chapter:

• Expansion task overview

• Expansion Requirements

Expansion task overviewThis diagram provides a quick overview of the steps required for a Greenplum DCA expansion.

Expansion task overview 4


DCA Expansion Guide – Chapter 1: Greenplum DCA Expansion Overview

Expansion RequirementsThe following are recommended or required for the expansion.

Table 1.1 Expansion Requirements

Tools People

EMC Lift Tool1

1. Viewing the Lift Tool Training VILT from education services is recommended prior to using the EMC lift tool.

A minimum of two people is recommended if the EMC lift tool is not available.

#2 Philips Screwdriver

Flathead Screwdriver

Expansion Requirements 5


DCA Expansion Guide – Chapter 2: Add additional hardware

2. Add additional hardware

This chapter explains the steps involved in adding new modules or racks to a DCA. The following sections are included:

• Identify expansion order and existing configuration

• Install and cable new hardware

• Position and connect new racks

• Label new servers and record MAC addresses

• Connect new bays to the power supply

Identify expansion order and existing configurationBefore beginning the upgrade, you need to understand what was shipped in the upgrade, and the current configuration. The upgrade needs to be configured to sequentially resume from the current configuration.

1. Identify the current configuration by counting the server labels. Labels read sdw for GPDB, etl for DIA and hdm/hdw/hbw for GP HD modules.

Figure 2.1 Server Label Location

2. Remove the new servers from their packaging.

3. Determine what module type to configure the new servers. Servers with 2TB drives can be imaged as Greenplum Database (GPDB) capacity, Data Integration Accelerator (DIA) or Greenplum Hadoop (GP HD) modules. There are three types of GP HD modules: Master, Worker and HBase. Servers with 600GBdrives can be imaged as GPDB balanced modules.

Identify expansion order and existing configuration 6



Figure 2.2 Drive Type Label Location

4. Using Table 2.1, “Drive Labels for Server Types” on page 7 , verify the server types that were shipped match what is expected in the order.

Table 2.1 Drive Labels for Server Types

Server Type Drive Label

GPDB Balanced SAS 6Gb 600GB 15k

GPDB Capacity SATA 3Gb 2TB 7.2k

DIA SATA 3Gb 2TB 7.2k

GP HD SATA 3Gb 2TB 7.2k

Install and cable new hardwareThis section explains how to install the new servers into an existing rack with free space. If the upgrade contains a new rack, and there is no free space in the existing rack, continue to the next section.

Determine population order

Specific population rules need to be followed when upgrading a cluster with mixed modules. Follow the rules below to create a plan to install the new servers:

• Add GPDB modules starting at the lowest free position in an existing rack.

• Add DIA modules until all modules have been used up.

• Add GP HD modules until all modules have been used up. Add Hadoop Masters first, Workers second and Hbase third.

• 1st rack is always the system rack - labeled DCA1-SYSRACK

• 2nd rack is always an aggregation rack - labeled DCA1-AGGREG

• Rack 3 through 12 are expansion racks - labeled DCA1-EXPAND

• Balanced and capacity GPDB modules cannot be mixed in the same rack or system

Install and cable new hardware 7



• If a new rack was shipped, and there is free space in the existing rack, move modules to fill the existing rack first.

Attach rails to the new server

1. Attach the inner rails to the new server as shown in Figure 2.3 .

2. On the new server, align the protruding rail notches with the connection studs on the server as shown in Figure 2.3 .

Click!

CL4347

Figure 2.3 Attaching Inner Rails to the New Server

3. Push the rail along the server until the studs fit securely into the notches; an audible click indicates that the rail is secure.

Install the server in the cabinet

The enclosure is heavy and should be installed into a rack by two people. To avoid personal injury and/or damage to the equipment, do not attempt to lift and install the enclosure into a rack without a mechanical lift and/or help from another person.

1. From the front of the cabinet, align the inner rails attached to the server with the mounting channels on the slide rails. Refer to Figure 2.4 .

2. Slide the server into the slide rails.




CL4349

Figure 2.4 Inserting the Server in the Cabinet

3. Tighten the captive screws at the front of the server to secure it to the rack.

Cable new servers to switches and power

Each server connects to two interconnect switches, an administration switch and two power zones. Follow these steps to complete cabling for each server. Cabling is pre-installed in the bay.

1. Connect cabling from the Interconnect switches in the rack to the server:

a. Connect the lower Interconnect switch cables to the converged network adapter (CNA) in each server. Use Table B.2, “Interconnect Switch Port Reference” on page 56 and Figure B.6, “Bottom Interconnect Switch Cabling Reference” on page 54 for reference.

b. Connect the upper Interconnect switch cables to the CNA in each server. Use Table B.2, “Interconnect Switch Port Reference” on page 56 and Figure B.7, “Top Interconnect Switch Cabling Reference” on page 55 for reference.

2. Connect cabling from the Administration switch at the top of the rack to each server:

a. Locate the GB1 port on each server.

b. Connect the correct cable from the Administration switch. Refer to Table B.3, “Admin Switch Port Reference” on page 58 and Figure B.9, “Rack 1 Admin Switch Cabling Reference” on page 57 .

3. Connect power to each server:




a. Locate the power cables for zone A and B. Zone A cables connect to the lower power supply. Zone B cables connect to the upper power supply.

b. Refer to Figure B.4, “Greenplum DCA Power Cable Configuration” on page 52 .

Position and connect new racksThis sections explains how to position and connect a new rack. If the upgrade does not include a rack, only loose servers, skip this section.

Position new rack

Follow these instructions to position a rack. If the upgrade order only contains loose server that were installed into existing bays, skip this section.

4. Identify the rack based on its label on the REAR of the rack: aggregation - DCA1-AGGREG, or expansion - DCA1-EXPAND.

Figure 2.5 Rack identification label

5. Position the aggregation rack 2nd and 7th, and the expansion as 3rd - 12th as shown in Figure B.2, “1 to 12 Rack Configuration” on page 48 .

6. Use a 1 3/8’’ wrench to lower the leveling feet on each rack. Turn the feet counter-clockwise to lower - once lowered, secure the feet by tightening the nut against the bottom of the rack. Refer to Figure 2.6.

Position and connect new racks 10



Figure 2.6 Lower leveling feet

Add aggregation switches for 7+ racks

If you are upgrading a 6 rack or smaller cluster to a 7 rack or larger cluster, you will need to remove the existing and add new aggregation switches.

This activity will require Greenplum Database and Hadoop services to be stopped.

1. Connect your laptop to the DCA. See “Connecting Your Laptop to the DCA Admin Network” on page 39 for detailed connection instructions. Log in as the user gpadmin.

2. Verify there are no errors with the Greenplum Database:

$ gpstate -e

3. Stop Greenplum Database:

$ gpstop -af

4. Connect to a Hadoop master, hdm1 as the user gpadmin:

$ ssh gpadmin@hdm1

5. Verify there are no errors with Hadoop services:

$ dca_hadoop --status

6. Stop Hadoop services:

$ dca_hadoop --stop

7. Locate the Aggregation switches in rack 2. Refer to “Aggregation Rack Components” on page 50 .

8. Label and disconnect the cables connected to the Aggregation switches. Refer to Figure B.10, “1 to 6 Rack Interconnect Switch to Aggregation Switch Cabling” on page 62 .

Position and connect new racks 11



9. Remove the Aggregation switches from the rack.

10. Install one Brocade VDX-6720 Aggregation switch into rack 2. Refer to Figure B.3, “Brocade VDX-6720 Aggregation Switch Locations” on page 48 .

11. Install the second Brocade VDX-6720 switch into rack 7.

Connect new racks

In a multiple rack cluster, the Interconnect switches in each Rack connect to the Aggregation switches.

1. Locate the first aggregation switch, aggr-sw-1. Refer to “Aggregation Rack Components” on page 59 for a system diagram.

2. Connect four inter-rack cables to ports 20-23 on the LOWER Interconnect switch of each rack. Refer to Table B.6, “DCA Multi-Rack Cable Kits” on page 64 to determine the correct kit to use between each rack.

3. Route and connect cables to the first Aggregation switch, aggr-sw-1. Refer to Figure B.11, “7+ Rack Aggregation Switch Cabling” on page 63 .

4. Repeat steps 1. through 3. to connect each rack to the second Aggregation switch, aggr-sw-2.

5. Connect the Administration switches in each rack to the Aggregation Administration switch in rack 2.

Label new servers and record MAC addressesThis section explains how to label the new servers being installed, and record the MAC Addresses of the new hardware.

1. Locate the label kit included on the rear door. The label kit contains labels for GPDB, DIA and HD. The GPDB labels read sdw, DIA - etl and GP HD hdm/hdw/hbw.

2. Affix a label to the front and rear of each new server to mark the module type.

Figure 2.7 Rear Server ID Label

Label new servers and record MAC addresses 12



Figure 2.8 Front Server ID Label

3. Use the numbered labels to mark the new servers sequentially from the existing server. Place the numbered label ON TOP of the server type label.

For example:

• If you are installing a module of 4 GPDB servers into a system with 8 existing GPDB servers, mark the new servers with sdw labels starting at 9.

• If you are installing a module of 4 DIA servers into a system with 8 existing GPDB servers, mark the new servers starting at 1.

• If you are installing a module of 4 GP HD servers into a system with 8 existing GPDB servers and 4 DIA servers, mark the new servers starting at 1.

4. Pull out the server’s service tag. Record the server type, number and MAC Address. The MAC Address can be found UNDERNEATH the service tag, and is noted as NIC1.

5. Repeat steps 2. through 4. for each additional new server that was installed.

6. Once the MAC Addresses of the new hardware have been recorded, populate the dhcpd.conf file on the Primary Master Server.

a. Connect to the Primary Master Server as the user root

Label new servers and record MAC addresses 13



b. Open the file /etc/dhcpd.conf for editing

c. Scroll to the entry for the first NEW server and change the value in BOLD to the MAC Address of the corresponding server.host sdw9-cm{

hardware ethernet 84:2B:2B:4E:81:A3;

fixed-address 172.28.4.9;

}

d. The dhcpd.conf file is populated with hostnames for the first 16 servers of each type - GPDB, DIA and GP HD. Add additional entries if required, using the existing entries as a template.

e. Enter the remaining MAC Addresses in the dhcpd.conf file, then save and exit the editor.

Connect new bays to the power supplyWhen installing a new rack, the customer power source must be connected. When upgrading a bay with one modules to two, three or four modules, the power distribution panel (PDP) to power distribution unit (PDU) connections may need to be re-routed. The customer power feed connects to PDPs, which feed PDUs. The servers and switches connect to PDUs.

Choose section that applies to your situation:

• Upgrade an existing rack to 2 or more modules

• Connect power to new rack

Upgrade an existing rack to 2 or more modules

A rack with one module may have the PDU to PDP cabling configured to require two power connections from the customer. When upgrading the rack to two or more modules, the PDU to PDP cabling must be re-routed to accept four power connections.

Note: Before beginning this procedure, verify the LEDs of the power supplies on all switches are green. If any power supply LEDs are amber or OFF, replace the component before continuing.

1. From the rear of the system, locate the power zone B and power zone A upper PDPs using the labeling. Refer to Figure 2.9 for the correct location of power components.

Connect new bays to the power supply 14



Figure 2.9 PDP and PDU Cabling on a 1 module DCA

2. Verify the power zone B and power zone A upper PDP switches are in the OFF position.

3. Connect the two new customer supplied power connections to both upper PDPs




Figure 2.10 Upper PDP connector locations

4. On the zone B upper PDU, flip all three breaker switches to the OFF position.

Figure 2.11 PDU Breaker Switch Locations

5. Re-route the zone B upper PDU cable from the zone B lower PDP to the zone B upper PDP. Figure 2.12 shows the BEFORE state and Figure 2.13 shows the AFTER state.




Figure 2.12 PDP to PDU Cabling Before Re-Routing

Figure 2.13 PDP to PDU Cabling After Re-Routing

6. On the zone B upper PDU, flip the three breaker switches to the ON position.

7. On the zone B upper PDP, flip the switch to the ON position.

8. Verify the power supply of each components attached to the zone B upper PDU has a green LED indicating a power connection.

9. Repeat steps 4. through 8. for power zone A.




Connect power to new rack

1. Connect the upper and lower PDPs from side A and side B to the customer-supplied PDPs. Verify with the customer that A and B utilize separate power feed sources to ensure redundancy.

2. Flip all PDP switches to the ON position.

3. From the back of the appliance, verify that each server displays a green LED light for both power supplies.

4. If a LED is not lit or is amber, trace the power cord to the PDP and verify that the cable is securely connected.



DCA Expansion Guide – Chapter 3: Configure switches and image new hardware

3. Configure switches and image new

hardwareNew servers from manufacturing arrive with no software image. New switches from manufacturing arrive with no configuration. Follow the steps in this chapter to image and configure the new hardware.

• Set switch IP addresses

• Image Servers

Set switch IP addressesSwitches are configured by manufacturing in racks 1 through 3; any additional racks must be configured on-site. Follow the steps in this section to configure switches in racks 4 to 12.

The follow tasks are included:

• Configure the Administration switch

• Configure Brocade 8000 Interconnect switches

• Configure aggregation switches

Figure 3.1 Need to configure switches in rack 4 to 12

Configure the Administration switch

1. Locate the Allied Telesis Administration switch in the first new rack. Refer to “Greenplum DCA configurations” on page 47 for the location of the switch.

2. Connect to the Allied Telesis Administration switch. The switch uses an RJ-45 plug as the serial connector - a DB-9 to RJ-45 serial adapter can be used if a native RJ-45 serial cable is not available.

3. Establish a serial connection to the switch. Login as the user manager with password friend. Refer to “Use Putty to Create a Serial Connection” on page 45.

Set switch IP addresses 19



4. Enter the following commands to set the IP address and hostname:

set conf=none

restart reboot

Wait while switch reboots, then log back in.

enable ip

add ip int=vlan1 ipaddress=172.28.0.193 mask=255.255.248.0

set system name=a-sw-4

create config=gp.cfg

set config=gp.cfg

restart reboot

Modify the IP Address and hostname in BOLD with the Admin switch values from “Network Hostname and IP Configuration” on page 59.

5. Disconnect the serial cable.

6. Repeat this section for the administration switch in each new rack.

Configure Brocade 8000 Interconnect switches

1. Start in the first new rack - connect to the lower Brocade 8000 switch using a serial connection The switch uses an RJ-45 plug as the serial connector - a DB-9 to RJ-45 serial adapter can be used if a native RJ-45 serial cable is not available.

2. Establish a serial connection to the switch. Log in as the user admin with password changeme. Refer to “Use Putty to Create a Serial Connection” on page 45.

3. Identify the switch you are configuring by its rack number and position. Use “Network Hostname and IP Configuration” on page 59 to determine the hostname and IP address.

4. Set the IP address based on the switch location:

MP_8000B:Admin> ipaddrset

Ethernet IP Address [10.77.77.77]: 172.28.0.173

Ethernet Subnetmask [255.255.255.0]: 255.255.248.0

Gateway IP Address [none]:ENTER

DHCP [Off]:ENTER

5. Rename the switch based on the position and rack:

i-sw-5:admin> switchname i-sw-7

i-sw-5:admin> cmsh

i-sw-5# config t

i-sw-5(config)# hostname i-sw-7

i-sw-5(config)# write mem




Overwrite Changes? y

6. Log out of the switch:

i-sw-7(config)# exit

i-sw-7# exit

i-sw-7:admin> exit

7. Repeat this section for the UPPER interconnect switch.

8. Repeat this section for the interconnect switches in each new rack.

Configure aggregation switches

Follow the steps in this section to configure aggregation switches

Brocade 8000 aggregation switches

There is no configuration required for the Brocade 8000 aggregation switches used in a 2-6 rack DCA.

Brocade VDX-6720 aggregation switches

Configuring the VDX-6720 aggregation switches is a two part process - first the switch must be given an IP address in order to be accessed over SSH, then a configuration must be loaded to the switch.

Set switch IP addresses

1. Connect to the first Brocade VDX switch using a serial connection The switch uses an RJ-45 plug as the serial connector - a DB-9 to RJ-45 serial adapter can be used if a native RJ-45 serial cable is not available.

2. Establish a serial connection to the switch. Log in as the user admin with password changeme. Refer to “Use Putty to Create a Serial Connection” on page 45.

3. Set the startup configuration as the default configuration:

# copy default-config startup-config

4. Reload the switch. You will be logged out.

# reload

5. Log back into the system as the user admin with password password.

6. Set the switch IP address. For the first aggregation switch use 172.28.0.248, for the second use 172.28.0.249.

sw0# configure terminal

sw0(config)# interface Management 1/0

sw0(config-Management-1/0)# ip address 172.28.0.248/24

sw0(config-Management-1/0)# exit

7. Set the username and password:

sw0(config)# username admin password changeme




8. Enter the following commands to save changes and exit:

sw0(config)# write memOverwrite Changes? ysw0(config)# exitsw0# exit

9. Repeat steps 2. through 8. for the second aggregation switch.

Load configuration

1. Connect your service laptop to the administration switch in the system rack. For instructions on how to connect to the DCA refer to “Connecting Your Laptop to the DCA Admin Network” on page 39.

2. Log in to the Primary Master server as the user root.

3. Connect to the first aggregation switch as the user admin:

ssh admin@aggr-sw-1

4. Enter config mode:

sw0# configure terminal

5. Change the user Admin’s password:

sw0(config)# username admin password changeme

6. Enter the configuration listed in Appendix F, “Brocade VDX-6720 configuration”. For the first aggregation switch, use VLAN ID 199 where noted in BOLD. For the second aggregation switch, use VLAN ID 299.

7. Once the ports have been configured, enter the following commands to save changes and exit:

sw0(config)# write memOverwrite Changes? ysw0(config)# exit

sw0# exit

8. Repeat these steps for the second aggregation switch.

Image ServersBy default, the new servers have no software installed. Each server must be imaged for its type and position in the correct rack.


2. Check if the DCA is in Secured Mode. If 1.2.0.0s is returned, the DCA is in Secured Mode. Follow instructions in “Revert to Unsecured Mode on the DCA” on page 79.

$ cat /etc/gpdb-appliance-version

3. Enable the PXE boot services on the Primary Master server

# service httpd start

Image Servers 22



# service dhcpd start

# service xinetd start

4. Power on each new server, up to 16 at a time to start the image process. If there are more than 16 servers, wait for the first 16 to complete, then image the remaining servers in groups of 16 until all new servers have been imaged.

The image process will take 25-30 minutes for each server. Continue to the next step after 25-30 minutes.

5. Issue the ping command to each server that was imaged. Wait for a response before continuing. For example:

# ping sdw9

6. Disable PXE boot services once all the servers have completed imaging:

# service httpd stop

# service dhcpd stop

# service xinetd stop

Image Servers 23


DCA Expansion Guide – Chapter 4: Expand and Validate Greenplum DCA

4. Expand and Validate Greenplum DCA

The tests described in this chapter validate that the baseline hardware performance is as expected before you complete installation. Poor performance results can indicate hardware problems such as a failed network interface, disk drive, or CPU.

The following sections are included:

• Expand with DCA Setup

• Validate system performance

• Validate the Greenplum DCA Host Operating Systems

Expand with DCA Setup

1. Connect your laptop to the Greenplum DCA and ssh to the master host as root. See “Connecting via SSH to the Greenplum Master” on page 79 for detailed connection instructions.

2. Launch the DCA Setup utility:

# dca_setup

3. Select option 2: Modify DCA Settings.

4. Select option 9: Expand DCA. Answer the following prompts:

• Confirm the detected configuration is correct.

• Enter the number of new GPDB Segment hosts.

• Enter the number of new DIA hosts.

• Enter the number of new Hadoop Worker hosts.

• Enter the number of new Hadoop HBase hosts.

• Enter the root user password for the new hosts. Use the default password changeme unless the customer has specified otherwise.

• Enter the gpadmin user password for the new hosts. Use the default password changeme unless the customer has specified otherwise.

• Enter the iDRAC password for the new hosts. Use the default password calvin unless the customer has specified otherwise.

• Enter y to regenerate the cluster map.

• Enter y to continue with expansion.

5. Restart Greenplum Command Center after the expansion:

# su - gpadmin

$ gpcmdr --restart

Expand with DCA Setup 24



Validate system performanceCOPY FROM INSTALL GUIDE

1. Verify you are logged in to the Primary Master server as the user root.

2. Navigate to the /home/gpadmin/gpconfigs directory:

# cd /home/gpadmin/gpconfigs

3. Run the following performance tests. Verify the results match the values in Table 4.1, “Expected Performance Results” on page 26.If any of the results are below the expected result, this may indicate a hardware issue.

For GPDB and DIA Modules:

a. Run the network test on interconnect one:# dcaperfcheck -f hostfile_gpdb_ic1 -r N -d /tmp

b. After the first test completes, run the network test on interconnect two:# dcaperfcheck -f hostfile_gpdb_ic2 -r N -d /tmp

c. Run the disk I/O and stream test:# dcaperfcheck -f hostfile_segments -r ds -D -d /data1/primary -d /data1/mirror -d /data2/primary -d /data2/mirror

Change the hostfile name in BOLD to the module type you are validating.

For GP HD Modules:

a. Run the network test on Interconnect 1:# dcaperfcheck -f hostfile_hadoop_ic1 -r N -d /tmp

b. Run the network test on Interconnect 2:# dcaperfcheck -f hostfile_hadoop_ic2 -r N -d /tmp

c. Run the disk I/O and stream test on Hadoop Masters:# dcaperfcheck -f hostfile_hdm -r ds -d /data -S 50GB

d. Run the disk I/O and stream test on Hadoop Workers:# dcaperfcheck -r ds -f hostfile_hdw -d /data1 -d /data2 -d /data3 -d /data4 -d /data5 -d /data6 -d /data7 -d /data8 -d /data9 -d /data10 -d /data11 -d /data12

If there are Hadoop HBase modules in the cluster, replace hostfile_hdw with hostfile_hbw in the command above.

Module Type Validation Test Hostfile to Use Result

GPDB Standard

GPDB Capacity

Network hostfile_gpdb_ic1

hostfile_gpdb_ic2

>950 MB/s

GPDB Standard Disk I/O hostfile_segments > 1000 MB/s Write

> 1200 MB/s Read

Validate system performance 25



Table 4.1 Expected Performance Results

Validate the Greenplum DCA Host Operating SystemsThe dcacheck utility validates several configuration settings and checks software versions on the Greenplum Data Computing Appliance.

1. Verify you are logged in to the Primary Master server as the user root.

2. Run the dcacheck utility, verify no ERROR messages are reported.

# dcacheck

GPDB Capacity Disk I/O hostfile_segments > 500 MB/s Write

> 500 MB/s Read

GPDB Standard

GPDB Capacity

Stream hostfile_segments > 2000 MB/s

DIA Network hostfile_dia_ic1

hostfile_dia_ic2

> 950 MB/s

DIA Disk I/O hostfile_dia > 500 MB/s Write

> 500 MB/s Read

DIA Stream hostfile_dia > 2000 MB/s

GP HD Network hostfile_hadoop_ic1

hostfile_hadoop_ic2

> 950 MB/s

GP HD Disk I/O - Masters hostfile_hdm > 20 MB/s Write

> 1800 MB/s Read

GP HD Disk I/O - Workers/Hbase hostfile_hdw > 750 MB/s Write

> 2000 MB/s Read

GP HD Stream hostfile_hadoop > 2000 MB/s

Module Type Validation Test Hostfile to Use Result

Validate the Greenplum DCA Host Operating Systems 26


DCA Expansion Guide – Chapter 5: Expanding Greenplum Database

5. Expanding Greenplum Database

This chapter explains the steps involved in expanding a Greenplum Database system once the additional hardware resources have been added to the DCA. Expanding Greenplum Database involves the following high-level steps:

• Stop health monitoring and revert secure mode

• Back up your Greenplum Databases

• Change failure group mapping

• Plan the Greenplum Database expansion

• Initialize new segment resources

• Rebalance table data

• Resume health monitoring, secure mode and restart GPDB

Note: Skip to Configure Greenplum DIA Modules or Expand Hadoop modules if you are expanding DIA or GP HD modules.

Stop health monitoring and revert secure modeIn order to stop false alarm dial-homes from being sent to EMC Support, the health monitoring process needs to be stopped prior to Greenplum Database expansion.


2. Stop the healthmon daemon

# dca_healthmon_ctl -d

3. Disabled automated Master server failover:

a. Check if the automated master server failover feature is enabled:# ps -ef | grep dbwatcherd

If the following text is returned, the feature is enabled, and needs to be disabled. If the following text is not returned, the feature is not enabled, skip to step 4.

root 30889 1 0 14:10 ? 00:00:00 /opt/dca/bin/dbwatcherd

b. Launch the DCA Setup utility:# dca_setup

c. Select option 2 to Modify DCA Settings.

d. Select option 12 for DB Watcher.

e. Select option 2 for Disable dbwatcherd.

Stop health monitoring and revert secure mode 27



f. Enter y to confirm dbwatcherd will be disabled.

4. Revert the DCA from secure mode if it is enabled. Follow the instructions to revert to unsecured mode in “Revert to Unsecured Mode on the DCA” on page 79.

Back up your Greenplum DatabasesBefore beginning the expansion procedure, back up of all customer databases, as well as a snapshot of the existing system configuration.

1. Connect your laptop to the Greenplum DCA and ssh to the master host as gpadmin. See “Connecting via SSH to the Greenplum Master” on page 41 for detailed connection instructions.

2. Back up the database, configuration files, and global database objects using gpcrondump:

$ gpcrondump -x database_name -c -g -G

3. If your Greenplum Database system has multiple databases, backup each additional database:

$ gpcrondump -x database_name

4. Capture a snapshot of the system configuration using gpstate:

$ gpstate -s >> gpstate.out

Timestamped dump files are created for the master and each primary segment within their respective data directories in a directory called db_dumps. The dump files should be moved off the Greenplum DCA to a secure location.

Change failure group mappingIf the DCA is using spread mirroring, it will need to be re-mapped to group mirroring. The DCA may be using spread mirroring if it was installed at software version 1.0.3.0 or earlier.

1. Connect to the Primary Master Server as the user gpadmin.

2. Navigate to the /home/gpadmin/gpconfigs directory:

$ cd /home/gpadmin/gpconfigs

3. Check if the move mirrors configuration file was generated:

$ ll move_mirrors_failure_group.conf

• If the file exists, DCA Setup detected your failure group mapping needs to be re-configured. Follow steps in this section.

• If the file does not exist, skip to “Plan the Greenplum Database expansion” on page 29.

4. Start the Greenplum Database if it is not started:

$ gpstart -a

Back up your Greenplum Databases 28



5. Run the gpmovemirrors command:

$ $GPHOME/bin/gpmovemirrors -i move_mirrors_failure_groups.conf -d $MASTER_DATA_DIRECTORY

6. Monitor re-mapping progress:

$ gpstate -m

The status will read Resynchronizing while re-mapping is occurring, and Synchronized when it is complete.

Based on the size of the existing system, the re-mapping can take from 30 minutes to a few hours.

Plan the Greenplum Database expansionBefore you begin the process of expanding Greenplum Database, you should make sure you have evaluated the data to be expanded, and discussed with the customer the implications of expansion to their database processing and throughput. This section covers the considerations of expanding a Greenplum Database system in an active production customer environment.

• Set expectations with the customer

• Evaluate customer data and space requirements

Set expectations with the customer

Before you begin the database expansion, the customer must be aware of the impact to their ongoing database workloads.

Greenplum Database expansion is a two-part process:

1. Initializing the new segment resources and adding them into your Greenplum Database configuration. This process causes a brief system outage (about 30 minutes).

2. Rebalancing table data across the expanded system, which includes the newly added segment resources. The system is online during rebalancing, but performance may be degraded until rebalancing is complete.

When you run the command to initialize the new segments, Greenplum Database table distribution policies are reset to random distribution for all tables. This can affect the customer’s query performance until the table rebalancing process is complete. The Greenplum Database query planner uses distribution policy information to optimize query execution plans. When a table has a random distribution policy, it essentially has no distribution policy that the query planner can use to optimize certain database operations, such as joins. Customers may experience slower response times for their query workloads until the table rebalancing process is complete.

The table rebalancing process can be resource intensive, as all data of a table is scanned and rewritten. For this reason, customers should plan to do table rebalancing during low-usage times. Although the database is online during rebalancing, response

Plan the Greenplum Database expansion 29



times and throughput will degrade while rebalancing is running. Also, a table is not available for write operations while it is in the process of being rebalanced. Any automated data loads should be temporarily suspended until rebalancing completes.

Evaluate customer data and space requirements

Before you perform the database expansion, you need to understand the data being expanded so that you can determine the best rebalancing strategy. If disk space is not a consideration, rebalance tables in order of importance to the customer. If the existing segment systems have minimal free space, you should rebalance small tables first to free up space for the larger tables. While a table is being rebalanced, any load processes (ETL, DML queries) to that table must be suspended until after the rebalancing is complete.

Consult with the customer about the order in which to rebalance tables and low usage maintenance time windows when the rebalancing process can run. The rebalancing process is resource intensive; therefore, it is not recommended to perform table rebalancing when the customer is running heavy query workloads.

As part of your evaluation, you should have an inventory of all customer tables, their size, and ranking of customer importance. This information is used to determine the rebalancing schedule with the customer.

• Verify available disk space on existing systems

• Perform a table analysis

Verify available disk space on existing systems

The gpexand utility rebalances one table at a time. During rebalancing, you need enough space to hold a duplicate copy of the table being rebalanced. Before you begin, it is recommended to check the amount of free space on the existing segment hosts. Determine the size of the largest table in the customer’s database. Ensure there is enough free space to hold a duplicate of the largest table. If there is not enough free space, you can rebalance small tables first to free up space. If the existing systems are completely full, contact the EMC Support Center for a workaround strategy.

To determine free disk space on existing segment hosts:

1. Connect your laptop to the Greenplum DCA and ssh to the Primary Master server as the user gpadmin. See “Connecting via SSH to the Greenplum Master” on page 41 for detailed connection instructions.

2. Start a psql session by connecting to the customer’s database. If the customer has multiple databases, you will have to evaluate each database independently. For example:

$ psql database_name

3. Once in psql, run the following command to determine the amount of free space on each segment host file system:

=# SELECT DISTINCT dfdevice AS seg_filesystem,

dfhostname AS seg_host,

dfspace/1024 AS KB_free_space

Plan the Greenplum Database expansion 30



FROM gp_toolkit.gp_disk_free

ORDER BY seg_host;

Perform a table analysis

As a part of your evaluation, you should take an inventory of all customer tables in each database in the system. For each table, you should capture the following information:

• Database name

• Schema name

• Table name

• Partition Status (not partitioned, parent table, or child partition)

• Storage Type (heap, append-only, append-only compressed)

• Size (total table size with indexes)

• Ranking (level of importance to the customer)

How to look up table information in the system catalog


2. Start a psql session by connecting to the customer’s database. If the customer has multiple databases, you will have to evaluate each database independently. For example:

$ psql database_name

3. To see a list of all tables, use the \dt meta-command:

=# \dt

4. To see the details about a particular table, use the \d+ meta-command:

=# \d+ table_name

5. To check the size of all tables:

=# SELECT relname as name, sotdsize as size, sotdtoastsize as toast, sotdadditionalsize as other

FROM gp_toolkit.gp_size_of_table_disk as sotd, pg_class

WHERE sotd.sotdoid=pg_class.oid ORDER BY size;

6. Exit psql

=# \q

Initialize new segment resourcesThis process involves initializing new segment instances and adding them into the Greenplum Database system configuration.

Initialize new segment resources 31



To initialize new segments:


2. Navigate to the gpconfigs directory:

$ cd gpconfigs

3. Create a temporary expansion database

$ createdb dca_expansion_db

4. Run gpexpand using the gpexpand_map file for your expansion configuration:

$ gpexpand -i gpexpand_map -D dca_expansion_db

5. When prompted, type Y to confirm the configuration and execute the expansion. A success message will display when the process is complete.

Rebalance table dataAfter you have successfully initialized the new segment resources, the Greenplum Database system will be back online. At this point, table data only resides on the original segments, not the new segments you just added. Each table must be rebalanced in order to distribute its data across all segments, and to restore the original table distribution policy.

• Set the order of table rebalancing

• Run table reblancing

• Remove the expansion schema

Set the order of table rebalancing

The expansion initialization process creates an expansion schema called gpexpand in the selected database. This schema contains a number of tables and views that can be used to manage and track the table rebalancing process.

Using the gpexpand.status_detail table, you can rank tables so that they are rebalanced in preferred order. If space is a concern, you should rebalance smaller tables first. If space is not an issue, you should rebalance tables in order of importance to the customer.

To rank tables for rebalancing


2. Start psql:

$ psql

Rebalance table data 32



3. Update the rank column for each table in the gpexpand.status_detail table. For example:

=> UPDATE gpexpand.status_detail SET rank=1

WHERE fq_name='table1_name';

=> UPDATE gpexpand.status_detail SET rank=2

WHERE fq_name='table2_name';

Run table reblancing

Tables should be rebalanced during low-use hours when the necessary CPU usage and table locks will have the least impact on database operations. You have the option to run the rebalancing process for a set time interval.

While the rebalancing of a table is in progress, the table or partition currently being rebalanced will be locked and unavailable for read or write operations.

To run the rebalancing process


2. Run the gpexpand utility using the -d option to define the time period for the rebalancing session. For example, to run for a maximum of 10 consecutive hours:

$ gpexpand -D dca_expansion_db -d 10:00:00

3. The utility rebalances tables in ranked order until all tables are completed, or until the specified time duration is reached. Each time a session is started or finished, the utility updates the status and updated time in the gpexpand.status table.

Check status of table rebalancing

At any time during the table rebalancing process, you can query the expansion schema. The view gpexpand.expansion_progress provides a summary of the current progress, including calculations of the estimated rebalancing rate and estimated time to completion. For example:

=# SELECT * FROM gpexpand.expansion_progress;

Because the estimates in gpexpand.expansion_progress are based on the rates achieved for each table, the view cannot calculate an accurate estimate until the first table has completed. Calculations are restarted each time you re-run gpexpand to start a new table rebalancing session.

The table gpexpand.status_detail can be queried for per-table status information. For example:

=# SELECT status, expansion_started, source_bytes FROM gpexpand.status_detail WHERE fq_name='table_name';

Remove the expansion schema

The expansion schema can safely be removed after all tables are rebalanced and verified.

Rebalance table data 33



To remove the expansion schema


2. Run the gpexpand utility with the -c option:

$ gpexpand -c -D dca_expansion_db

Resume health monitoring, secure mode and restart GPDB

After the expansion process completes, health monitoring and secure mode (if it was used) should be resumed so the DCA can dial-home to EMC Support with errors.

1. Connect your laptop to the Greenplum DCA and ssh to the Primary Master server as the user root. See “Connecting via SSH to the Greenplum Master” on page 41 for detailed connection instructions.

2. Start the healthmon daemon:

# dca_healthmon_ctl -e

3. Enable automated master server failover. This should only be done if the feature was previously enabled.

a. Launch the DCA Setup utility: # dca_setup

b. Select option 2 to Modify DCA Settings.

c. Select option 12 for DB Watcher.

d. Select option 1 for Enable dbwatcherd.

e. Enter y to confirm dbwatcherd will be enabled.

4. If the DCA previous sly was setup with secure mode , enable it by following the instructions in “Enable Secured Mode on the DCA” on page 78.

5. Restart the Greenplum Database. This is necessary for Command Center metrics to detect the new hardware:

$ su - gpadmin

$ gpstop -r


$ gpcmdr --restart

7. If the DCA was previously in Secured Mode, follow the steps in “Enable Secured Mode on the DCA” on page 78.

Resume health monitoring, secure mode and restart GPDB 34


DCA Expansion Guide – Chapter 6: Configure Greenplum DIA Modules

6. Configure Greenplum DIA Modules

In order to be able to access the new hosts in the DIA, you need to assign IP addresses and a hostname to each of the DIA hosts external interface (eth1) and configure domain name services (DNS). This section applies to the Greenplum DIA only.

The following sections are included:

• Configure external network access

• Restart Greenplum Database

• Configure the Expanded DIA Modules for Loading

Configure external network access


2. SSH into the first newly added Greenplum DIA host

# ssh etl9

3. Edit the file /etc/sysconfig/network-scripts/ifcfg-eth1 and add the IPADDR and NETMASK settings using the values provided by the customer. For example, the settings in bold would be the ones to edit as per the customer’s environment. All other settings should remain as is:

DEVICE=eth1

BOOTPROTO=static

IPADDR=10.6.193.46

NETMASK=255.255.252.0

ONBOOT=YES

MTU=1500

HWADDR=b8:ac:6f:15:92:73

4. Edit the file /etc/hosts and add a line for the new IP you just added. This IP should resolve to the hostname etl9-ext1 and whatever other alias the customer wants to use (in bold). For example:

10.6.193.46 etl9-ext1 etl9-gpdb

5. Edit the file /etc/sysconfig/network and set the GATEWAY setting to the customer-provided default gateway IP address. For example, the setting in bold would be the one to add or edit as per the customer’s environment. All other settings should remain as is:

NETWORKING=yes

NETWORKING_IPV6=no

HOSTNAME=etl9

Configure external network access 35


DCA Expansion Guide – Chapter 6: Configure Greenplum DIA Modules

GATEWAY=10.6.193.1

6. Use the route command to add the default gateway to the operating system’s IP routing tables. For example, where 10.6.193.1 is the IP address of the gateway, and eth1 is the name of the network interface used to connect to it (change the information in bold as per the customer’s environment):

# route add default gw 10.6.193.1 eth1

7. Bring up the interface after it is configured, then disconnect.

# ifup eth1

# exit

8. Back on the DCA the master server, ssh to the next DIA host:

# ssh etl10

9. Repeat steps 3. and 4. to configure a different external IP for the remaining DIA hosts - etl10 through etl16.

10. Repeat steps 5., 6. and 7. to set the same default gateway on the remaining DIA hosts - eth10 through etl16.

Restart Greenplum DatabaseThe Greenplum Database must be restarted in order for Command Center to begin monitoring the new hardware:

1. Connect to the Primary Master server as the user gpadmin.

2. Restart the Greenplum Database:

$ gpstop -r


Configure the Expanded DIA Modules for LoadingDIA configuration is specific for every customer. If the DIA modules were added to a DCA with existing DIA modules, the same configuration - loading directories, external tables can be used. If the new DIA module is the first module of this type, use the following resources for configuration:

• Greenplum Database 4.2 Administrator Guide

• EMC Greenplum Data Computing Appliance Installation and Configuration Guide

Restart Greenplum Database 36


DCA Expansion Guide – Chapter 7: Expand Hadoop modules

7. Expand Hadoop modules

Follow the steps in this chapter to add Hadoop modules to a DCA. The following sections are included:

• Set VLAN IP addresses on new hosts

• Restart Hadoop and Greenplum Database

Note: If the DCA does not already contain Hadoop modules, you must follow instructions to install Hadoop in the DCA Installation Guide - Configure Hadoop Modules.

Set VLAN IP addresses on new hosts

1. Connect your laptop to the DCA. Refer to “Connecting via SSH to the Greenplum Master” on page 41 for detailed connection instructions.


3. Launch the DCA Setup utility:

# dca_setup


5. Select option 14 for Networking: Segments.

6. Selection option 4 to Edit Custom VLAN Configuration.

7. Select option 4 to change Interconnect 1 VLAN IP Addresses.

8. A list of all hosts in the cluster will be displayed. Starting with the first NEW Hadoop host, enter the corresponding number to modify it’s Interconnect 1 VLAN IP Address.

9. Enter the customer provided IP Address for the host on Interconnect 1 VLAN:

Enter IP for host 'hdm1':

10. Select the next Hadoop host. Repeat this until all new Hadoop hosts have been provided IP Addresses for the Interconnect 1 VLAN.

11. From the IC 1 Virtual IP Configuration menu, select option 1 to Apply Changes.

12. From the DCA Customer VLAN Configuration menu, select option 1 to Apply Changes.

13. From the Network Configuration: Hosts menu, select option 1 again to Apply Changes.

14. Enter y to continue:

37


DCA Expansion Guide – Chapter 7: Expand Hadoop modules

Enter yes to continue or no to exit this application: (Yy|Nn): y

Restart Hadoop and Greenplum Database

1. From the Primary Master server as the user root, launch the DCA Setup utility:

# dca_setup


3. Select option 16 for Hadoop.

4. Select option 3 to Configure namenode failover.

5. Press ENTER to use the existing virtual IP address in BOLD:

Enter the VIP for namenode: [10.6.193.83]: ENTER

6. Enter y to continue. The Hadoop namenodes will be configured using the virtual IP address.

7. Log in to hdm1 as the user gpadmin:

ssh gpadmin@hdm1

8. Restart Hadoop services:

$ dca_hadoop --stop

$ dca_hadoop --start

9. Verify the new hardware has been added to the Hadoop configuration:

$ dca_hadoop --status

10. Log in to the Primary Master server as the user gpadmin:

$ ssh mdw

11. Restart the Greenplum Database:

$ gpstop -r


$ gpcmdr --restart


38


DCA Expansion Guide – Appendix A: Connect to the DCA

A. Connect to the DCA

This section contains information on how to connect to the DCA for various maintenance tasks. Administration is always performed from the Primary Master server (hostname mdw) This appendix assumes you are using a Windows laptop and the Putty application.

• Laptop Prerequisites

• Connecting Your Laptop to the DCA Admin Network

• Connecting via SSH to the Greenplum Master

• Using SCP to Copy a File to the Greenplum Master

• Use Putty to Create a Serial Connection

Laptop PrerequisitesThe laptop you use to connect to the Greenplum DCA must have the following requirements met in order to perform Greenplum DCA administration.

• RJ-45 Ethernet port

• Administrator access on the laptop

• An ssh client such as Putty or Cygwin with the OpenSSH package enabled

• An scp client such as WinSCP, Putty PSCP or Cygwin with the OpenSSH package enabled

Connecting Your Laptop to the DCA Admin NetworkThe procedure to change your laptop’s IP address is different for a system running Windows XP or Windows 7. Choose the correct procedure based on the version of Windows you are running.

Laptop is running Windows 7

These instructions should be used if your laptop is running Windows 7:

1. Looking at the back of the appliance, locate the red service cable. This cable is connected to port 23 of the first administration switch (a-sw-1). Connect the service cable to your laptop.

2. Open Network Connections by clicking the Start button, and then clicking Control Panel. In the search box, type adapter, and then, under Network and Sharing Center, click View network connections.

3. Right-click the connection that you want to change, and then click Properties. If you're prompted for an administrator password or confirmation, type the password or provide confirmation.

Laptop Prerequisites 39


http://www.chiark.greenend.org.uk/~sgtatham/putty/

http://www.cygwin.com/

http://winscp.net

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

http://www.cygwin.com/


4. Click the Networking tab. Under This connection uses the following items, click Internet Protocol Version 4 (TCP/IPv4) and then click Properties.

5. Click Use the following IP address, and then, in the IP address, and Subnet mask field, ype the IP address settings:

• IP address: 172.28.3.253

• Subnet mask: 255.255.248.0

6. Click OK

Connecting Your Laptop to the DCA Admin Network 40



Laptop is running Windows XP

These instructions should be use if your laptop is running Windows XP:


2. On your Windows laptop, open the Control Panel.

3. Double-click on Network Connections to open the Network Connections window.

4. Right-click on Local Area Connection and select Properties.

5. Select Internet Protocol (TCP/IP) and click Properties.

6. Enter 172.28.3.253 as the IP address and 255.255.248.0 as the Subnet mask, and click OK.

Connecting via SSH to the Greenplum MasterThe process for establishing an ssh connection to the master host depends on the ssh client you are using (Putty, Cygwin, etc.). Regardless of the client, you want to connect using the following connection information.

Table A.1 Master Host Connection Information

hostname 172.28.4.250

username root

root password changeme (or whatever the customer has selected for their root password)

Connecting via SSH to the Greenplum Master 41



Putty Example

1. Open Putty and enter 172.28.4.250 in the Host Name (or IP address) field. Select SSH as the Connection type.

2. Click Open.

3. If this is the first time you have connected to this server, a security alert will display. Click Yes to continue.

4. Next you will be presented with the SSH Login window, where you will be prompted to enter your username and password. For example:

login as: root

[email protected] password: changeme

Cygwin Example

If using Cygwin, you must have enabled the OpenSSH package when you installed Cygwin. Open a Cygwin terminal window and type the following at the prompt:

$ ssh [email protected]

When prompted, type the root password (default is changeme).

Connecting via SSH to the Greenplum Master 42



Using SCP to Copy a File to the Greenplum MasterThe process for copying a file from your local laptop to the Greenplum master host will depend on the scp client you are using (WinSCP, Cygwin, etc.). Regardless of the client, you want to connect using the following connection information.

Table A.2 Master Host Connection Information

hostname 172.28.4.250

username gpadmin

root password changeme (or whatever the customer has selected for their gpadmin password)

copy to location on master /home/gpadmin

WinSCP Example

1. Log in to the master host IP 172.28.4.250 as user gpadmin. Select SFTP as the File protocol.

Using SCP to Copy a File to the Greenplum Master 43



2. Locate the file you want to copy on your local host and choose the destination of /home/gpadmin on the master host:

3. Click Copy. Progress of the file copy is shown.

Cygwin Example

If using Cygwin, you must have enabled the OpenSSH package when you installed Cygwin. Open a Cygwin terminal window and type the following at the prompt:

$ cd <local_location_of_file>

$ scp <file_name> [email protected]:/home/gpadmin

When prompted, type the gpadmin password (default is changeme).

Using SCP to Copy a File to the Greenplum Master 44



Use Putty to Create a Serial ConnectionThis sections contains the procedure to connect your service laptop to a serial port on the Brocade 8000 or Allied Telesis 9924TL switch. This procedure is required when the switch contains factory settings, or cannot be accessed by telnet or ssh through the DCA administrative network.

1. Connect a serial cable between the serial port (console port in Figure A.1 ) on the new switch and your workstation. If your workstation does not have a serial port, it is OK to use a USB to Serial Adapter.

2. Open the Putty application.

3. Select Serial in Basic Options under the Session section.

Figure A.1 Serial Option in a Putty Session

4. Expand the connection section and select Serial. Verify the settings for the COM port are 9600 Baud, 8 data bits, and no hardware flow control.

Use Putty to Create a Serial Connection 45



5. Click Open to connect.

6. Press <Enter> to display the login prompt.

Use Putty to Create a Serial Connection 46


DCA Expansion Guide – Appendix B: DCA reference information

B. DCA reference information

This section contains reference information on the DCA hardware and software configurations and settings. The following information is included:

• Greenplum DCA configurations

• Power Supply Reference

• Network and Cabling Configurations

• Configuration Files

• Default Passwords

Greenplum DCA configurationsThis section includes information on the available DCA configurations, and components included. The following information is included:

• DCA configuration rules

• System rack components.

• Aggregation Rack Components

• Expansion Rack Components

DCA configuration rules

There are three basic types of racks that can be shipped from Manufacturing: System - DCA1-SYSRACK, Aggregation - DCA1-AGGREG and Expansion - DCA1-EXPAND. The following rules apply for population of racks:

• A configuration of two racks or smaller will use one System and one Aggregation rack.

• A configuration of one to six racks will use a System and Aggregation for the first two racks and Expansion for each remaining rack.

• A configuration of one to twelve racks will use a System as the first rack, aggregation as the second rack, and expansion for the remaining racks.

• The largest configuration allowed is 12 racks.

• The smallest configuration allowed is 1/4 rack

• Each rack can contain 4, 8, 12 or 16 servers.

• Racks must be fully populated with 16 servers before an additional rack is added.

Greenplum DCA configurations 47



Figure B.1 1 to 6 Rack Configuration

Figure B.2 1 to 12 Rack Configuration

Figure B.3 Brocade VDX-6720 Aggregation Switch Locations




System rack components.

Table B.1 System Rack Components

Component Quantity

Master Host 2 (one primary and one standby)

Hosts 4-16

Interconnect Switch 2

Administration Switch 1




Aggregation Rack Components

Component Quantity

Master Host 0

Hosts 4-16



Aggregation Switch 2 - 1 to 6 Rack Cluster

1 - 1 to 12 Rack Cluster. Second aggregation switch located in rack 7.

In a 1 to 12 rack configuration, the Brocade 8000 aggregation switches, shown as aggr-sw-1 and aggr-sw-2 are replaced by a single 2U Brocade VDX switch. The Brocade VDX switch is placed in rack 2 and 7.




Expansion Rack Components

Component Quantity

Master Host 0

Hosts 4-16






Power Supply ReferenceFigure B.4 shows the external power cables connected to their respective PDPs (Power Distribution Panel), while each PDP is connected to an internal PDU (Power Distribution Unit). Viewed from the rear of the DCA, Power Zone A is located on the right hand side, while Power Zone B is located on the left hand side of the system. The figure shows a full system rack.

Figure B.4 Greenplum DCA Power Cable Configuration

Power Supply Reference 52



Network and Cabling ConfigurationsThis section describes the network cabling configurations for the administration and interconnect networks. The following information is included:

• Interconnect Switch Reference

• Administration Switch Reference

• Network Hostname and IP Configuration

• Multiple Rack - Aggregation Cabling

Interconnect Switch Reference

The DCA contains two Brocade 8000 switches in each rack for the Greenplum Interconnect. Ports are numbered starting at 0. Ports 0-23 are used for segment server and rack connectivity - not including ports 18 and 19. Ports 18 and 19 are reserved for external connectivity. Ports 0-7 on the far right of the switch are fibre channel and unused by default.

Figure B.5 Brocade 8000 Portmap

Network and Cabling Configurations 53



Interconnect 1 - Bottom Brocade Switch References

The bottom Brocade Interconnect switch connects servers to the first interconnect. The bottom Brocade switch will always use odd-numbered hostnames, i-sw-1, i-sw-3, i-sw-5, etc.

Figure B.6 Bottom Interconnect Switch Cabling Reference




Interconnect 2 - Top Brocade Switch References

The top Brocade Interconnect switch connects servers to the second interconnect. The top Brocade switch will always use even-numbered hostnames, i-sw-2, i-sw-4, i-sw-6, etc.

Figure B.7 Top Interconnect Switch Cabling Reference




Interconnect Switch Port References

Table B.2 Interconnect Switch Port Reference

Switch Port System RackAggregation and Expansion Racks

Server Port

0 sdw1 Server 1

Interconnect 1 Switches (bottom) connect to RIGHT server port:

Interconnect 2 switches (top) connect to LEFT server port:

1 sdw2 Server 2

2 sdw3 Server 3

3 sdw4 Server 4

4 Server 5 Server 5

5 Server 6 Server 6

6 Server 7 Server 7

7 Server 8 Server 8

8 Server 9 Server 9

9 Server 10 Server 10







16 mdw ---

17 smdw ---

18 Reserved for external Connectivity

Reserved for external Connectivity

19 Reserved for external Connectivity

Reserved for external Connectivity




Administration Switch Reference

The DCA contains one Allied Telesis switch per rack, except for the Aggregation rack which contains two. The Allied Telesis Administration switch is used for management traffic. This switch connects all of the hosts and switches in a DCA, and allows service connectivity through a red service cable..

Figure B.8

Figure B.9 Rack 1 Admin Switch Cabling Reference

Allied Telesis AT9924TL Portmap




Table B.3 Admin Switch Port Reference

Switch Port System RackAggregation and Expansion Racks

server port

1 sdw1 Server 1

Top NIC (GB1):

Left NIC (GB1):

2 sdw2 Server 2

3 sdw3 Server 3

4 sdw4 Server 4

5 Server 5 Server 5

6 Server 6 Server 6

7 Server 7 Server 7

8 Server 8 Server 8

9 Server 9 Server 9








17 smdw ---

18 mdw ---

19 Bottom Brocade Interconnect Switch

Bottom Brocade Interconnect Switch

20 Top Brocade Interconnect Switch

Top Brocade Interconnect Switch

21 aggr-a-sw-1 aggr-a-sw-1




Network Hostname and IP ConfigurationTable B.4 DCA Network Configuration

Component hostnameiDRAC IPhost-sp

NIC 1 IPhost-cm

IC 1host-1

IC 2host-2

Administration switch in rack 1

a-sw-1 172.28.0.190


a-sw-2 172.28.0.191


a-sw-3 172.28.0.192


a-sw-4 172.28.0.193


a-sw-5 172.28.0.194

Administtation switch in rack 6 a-sw-6 172.28.0.195


a-sw-7 172.28.0.196


a-sw-8 172.28.0.197


a-sw-9 172.28.0.198


a-sw-10 172.28.0.199


a-sw-11 172.28.1.190


a-sw-12 172.28.1.191

Aggregation Administration Switch in rack 2 (first aggregation rack)

aggr-a-sw-1 172.28.1.248

Lower Interconnect switch in rack 1

i-sw-1 172.28.0.170

Upper Interconnect switch in rack 1

i-sw-2 172.28.0.180


i-sw-3 172.28.0.171


i-sw-4 172.28.0.181


i-sw-5 172.28.0.172


i-sw-6 172.28.0.182





i-sw-7 172.28.0.173


i-sw-8 172.28.0.183


i-sw-9 172.28.0.174


i-sw-10 172.28.0.184


i-sw-11 172.28.0.175


i-sw-12 172.28.0.185


i-sw-13 172.28.0.176


i-sw-14 172.28.0.186


i-sw-15 172.28.0.177


i-sw-16 172.28.0.187


i-sw-17 172.28.0.178


i-sw-18 172.28.0.188


i-sw-19 172.28.0.179


i-sw-20 172.28.0.189


i-sw-21 172.28.1.170


i-sw-22 172.28.1.180


i-sw-23 172.28.1.171


i-sw-24 172.28.1.181

Aggregation Interconnect Switch 1 in AGGREG

aggr-sw-1 172.28.0.248

Aggregation Interconnect Switch 2 in AGGREG

aggr-sw-2 172.28.0.249

Primary Master Server mdw 172.28.0.250 172.28.4.250 172.28.8.250 172.28.12.250

Standby Master Server smdw 172.28.0.251 172.28.4.251 172.28.8.251 172.28.12.251


NIC 1 IPhost-cm

IC 1host-1

IC 2host-2




**Hadoop Worker nodes are numbered 1-320. In order to accomodate the required number of hosts, the third IP address octet is incremented by 1 and the fourth octet restarts at 1 when the node number reaches 161. For example, the host hdw160-sp uses a third octet of 2 and a fourth octet of 160 - host hdw-161-sp uses a third octet of 3 and a fourth octet of 1.

GPDB Segment Server sdw# 172.28.0.# 172.28.4.# 172.28.8.# 172.28.12.#

DIA Server etl# 172.28.0.20# 172.28.4.20# 172.28.8.20# 172.28.12.20#

Hadoop Master Node 1-8 hdm1

hdm2

hdm3

hdm4

hdm5

hdm6

hdm7

hdm8

172.28.1.250

172.28.1.251

172.28.1.252

172.28.1.253

172.28.2.250

172.28.2.251

172.28.2.252

172.28.2.253

172.28.5.250

172.28.5.251

172.28.5.252

172.28.5.253

172.28.6.250

172.28.6.251

172.28.6.252

172.28.6.253

172.28.9.250

172.28.9.251

172.28.9.252

172.28.9.253

172.28.10.250

172.28.10.251

172.28.10.252

172.28.10.253

172.28.13.250

172.28.13.251

172.28.13.252

172.28.13.253

172.28.14.250

172.28.14.251

172.28.14.252

172.28.14.253

Hadoop Worker Node 1-160 hdw1-160 172.28.2.# 172.28.6.# 172.28.10.# 172.28.14.#

Hadoop Worker Node 161-320

hdw161-320 172.28.3.#

# = node number minus 160. Example: hdw162-sp = 172.28.3.2**

172.28.7.#

# = node number minus 160. Example:

hdw162 -cm= 172.28.7.2**

172.28.11.#

# = node number minus 160. Example: hdw162-1 = 172.28.11.2**

172.28.15.#

# = node number minus 160. Example: hdw162-2 = 172.28.15.2**

Hadoop Hbase Node 1-9 hbw1-9 172.28.0.161 - 172.28.0.169

172.28.4.161 - 172.28.4.169

172.28.8.161 - 172.28.8.169

172.28.12.161 - 172.28.12.169

Hadoop Hbase Node 10 hbw10 172.28.0.200 172.28.4.200 172.28.8.200 172.28.12.200


172.28.4.217 - 172.28.4.229

172.28.8.217 - 172.28.8.229

172.28.12.217 - 172.28.12.229


172.28.5.161 - 172.28.5.169

172.28.9.161 - 172.28.9.169

172.28.13.161 - 172.28.13.169



172.28.5.217 - 172.28.5.229

172.28.9.217 - 172.28.9.229

172.28.13.217 - 172.28.13.229


172.28.6.161 - 172.28.6.169

172.28.10.161 - 172.28.10.169

172.28.14.161 - 172.28.14-169



172.28.6.217 - 172.28.6.229

172.28.10.217 - 172.28.10.229

172.28.14.217 - 172.28.14.229


172.28.7.161 - 172.28.7.169

172.28.11.161 - 172.28.11.169

172.28.15.161 - 172.28.15.169



172.28.7.217 - 172.28.7.229

172.28.11.217 - 172.28.11.229

172.28.15.217 - 172.28.15.229


NIC 1 IPhost-cm

IC 1host-1

IC 2host-2




To see a complete list of IP addresses and hostnames, view the /etc/hosts file.

Multiple Rack - Aggregation Cabling

Servers in a multi-rack configuration communicate through the aggregation switches located in the AGGREG rack. The following diagram and table show the proper ports to cable:

FROM (ports 20-23)TO - Aggregation Switch Ports6 Rack or Smaller Config

TO - Aggregation Switch Ports7 Rack or Larger Config

Rack 1 0 - 3 1-4

Rack 2 8 - 11 6-9

Rack 3 16 - 19 11-14

Rack 4 4 - 7 16-19

Rack 5 12 - 15 21-24

Rack 6 20 - 23 26-29

Rack 7 n/a 31-34

Rack 8 n/a 36-39

Rack 9 n/a 41-44

Rack 10 n/a 46-49

Rack 11 n/a 51-54

Rack 12 n/a 56-59

Table B.5 Interconnect Switch to Aggregation Switch Cabling

Figure B.10 1 to 6 Rack Interconnect Switch to Aggregation Switch Cabling




Figure B.11 7+ Rack Aggregation Switch Cabling




Table B.6 DCA Multi-Rack Cable Kits

Kit NamePart Number

Quantity

Component Description

DCA1-CBL5

100-652-574 1 BRCD 10GBE TWINAX COPPER 5M 8-PACK

038-003-476 1 25’ CAT6 Ethernet Cable

DCA1-CBL10

100-652-576 2 BRCD 10GBE SFP+ SHORT WAVE 8-PACK

038-003-733 8 10m LC to LC Optical 50 Micron MM Cable Assemblies


DCA1-CBL30

100-652-576 2 BRCD 10GBE SFP+ SHORT WAVE 8-PACK

038-003-740 8 30m LC to LC Optical 50 Micron MM Cable Assemblies


Multiple Rack Cabling Information

Table B.7 Cabling Kits for a 6 Rack or smaller cluster

FROM AGGREG (Rack 2) TO Cable Kit

Rack 1 - SYSRACK DCA1-CBL5

Rack 2 - AGGREG DCA1-CBL5

Rack 3 - 1st EXPAND DCA1-CBL5

Rack 4 - 2nd EXPAND DCA1-CBL10

Rack 5 - 3rd EXPAND DCA1-CBL10

Rack 6 - 4th EXPAND DCA1-CBL10




Table B.8 Cabling Kits for a 12 Rack of smaller cluster

FROM AGGREG TO Cable Kit

Rack 2 Rack 1 - SYSRACK DCA1-CBL5

Rack 2 - AGGREG 1 DCA1-CBL5











Rack 7 Rack 1 - SYSRACK DCA1-CBL30















Configuration FilesConfiguration files are text files that contain the hostnames of servers for available configurations such as quarter, half or full rack. The file used will be based on the desired function. Refer to the table below for a description of each configuration and host file:

File Description

gpexpand_map Expansion MAP file used during GPDB upgrades to reallocate primary and mirror instances on the new hardware.

gpinitsystem_map MAP file used during installation of GPDB blocks to assign primary and mirror segments to each server.

hostfile Contains one hostname per server for ALL servers in the system. Includes GPDB, DIA and HD (if present)

hostfile_segments Contains the hostnames of the segment servers of all GPDB blocks

hostfile_gpdb_ic1 Contains the hostnames of the subnet 1 interface for GPDB servers.

hostfile_gpdb_ic2 Contains the hostnames of the subnet 2 interface for GPDB servers.

hostfile_dia Contains the hostnames of the DIA servers.

hostfile_dia_ic1 Contains the hostnames of the subnet 1 interface for DIA servers.

hostfile_dia_ic2 Contains the hostnames of the subnet 2 interface for DIA servers.

hostfile_hadoop Contains the hostnames of the Hadoop servers.

hostfile_hdm Contains the hostnames of all Hadoop Master servers.

hostfile_hdw Contains the hostnames of all Hadoop Worker servers.

hostfile_hbw Contains the hostnames of all Hadoop HBase servers.

hostfile_hadoop_ic1 Contains hostnames of the subnet 1 interface for Hadoop servers.

hostfile_hadoop_ic2 Contains hostnames of the subnet 2 interface for Hadoop servers.

hostfile_exkeys Contains the hostnames of all interfaces of all server blocks in the system.

Table B.9 Hostfiles created by the DCA Setup utility

Configuration Files 66



Default Passwords

Component User Password

Dell R510 and R610 Master Servers

iDRAC root user calvin

root changeme

gpadmin changeme

Brocade 8000 / VDX Interconnect / Aggregation Switch admin changeme

Allied Telesis Administration Switch

manager friend

The following table lists default passwords for all the components in a DCA

Default Passwords 67


DCA Expansion Guide – Appendix C: Working with the Integrated Dell Remote Access Controller (iDRAC)

C. Working with the Integrated Dell Remote

Access Controller (iDRAC)The Integrated Dell Remote Access Controller (iDRAC) is a built-in interface in Dell servers that provides out-of-band system management facilities. The controller has its own processor, memory, battery, network connection, and access to the system bus. Key features include power management, virtual media access and remote console capabilities, all available through a supported web browser. This gives system administrators the ability to manage a machine as if they were sitting at the local console. The following sections are included:

• Default iDRAC IP Addresses for the DCA

• Setting the iDRAC IP

• Accessing a Server via its iDRAC Interface

• Create SSH Tunnel to the iDRAC Web Interface

Default iDRAC IP Addresses for the DCAReplacement servers from manufacturing have the iDRAC IP Address setup differently if they are an R510 or R610:

• Dell R510 Segment, HD or DIA server: DHCP or 192.168.1.100

• Dell R610 Master server: 192.168.1.120

Setting the iDRAC IPThe IP Address needs to be set to the correct value if it is a GPDB, DIA or HD server. The server power cables should be connected, but not powered on.


2. On your Windows laptop, open the Control Panel.

3. Double-click on Network Connections to open the Network Connections window.

4. Right-click on Local Area Connection and select Properties.

5. Select Internet Protocol (TCP/IP) and click Properties.

6. Enter 192.168.1.200 as the IP address and 255.255.255.0 as the Subnet mask, and click OK.

7. Open a web browser and navigate to the new server by its iDRAC IP Address:

Default iDRAC IP Addresses for the DCA 68



• If the server is a Dell R610, navigate to its factory assigned, static IP address: https://192.168.1.120

• If the server is a Dell R510, the IP address will be set by DHCP on the Primary Master server. To find the IP address:# service dhcpd restart

# tail /var/lib/dhcpd/dhcpd.leases

Navigate to this IP address, for example: https://172.28.4.197

8. Accept the security warnings and login to the iDRAC web console as the user root with password calvin.

9. Click Remote Access on the left menu. When the page loads, click the Network/Security tab.

Figure C.1 iDRAC Remote Access Page

10. Scroll down to the IPv4 Settings section. Uncheck the DHCP Enable setting.

Setting the iDRAC IP 69



11. Set the IP address to the correct address for the new server. Refer to “Network Hostname and IP Configuration” on page 92 for the iDRAC IP address of each server. For example, for sdw3 the IP address is 172.28.0.3. The gateway is 0.0.0.0 and the netmask is 255.255.248.0 for all servers.

Figure C.2 iDRAC IPv4 Settings Section

12. Click Apply at the bottom of the page. After a short wait (less than 5 minutes), the iDRAC interface will stop responding and the current session ends. The server can now be accessed remotely using its newly assigned iDRAC IP address.

Accessing a Server via its iDRAC Interface

1. Connect your laptop to the Greenplum DCA and ssh to the Primary Master Server as the user root. See Appendix B, “Connecting Your Laptop to the DCA Admin Network” for detailed connection instructions.

2. Connect to the segment you want to configure using its iDRAC IP address. For example, to connect to sdw9:

# ssh 172.28.0.9

For a list of IP Addresses, refer to “Network and Cabling Configurations” on page 86.

Accessing a Server via its iDRAC Interface 70



3. When prompted, enter the iDRAC root password (default password is calvin).

4. Once you are connected to the iDRAC interface, you will see the admin1> prompt. Enter the command you want to execute. For example:

To start a powered off server:

admin1> start /system1

To stop a powered on server:

admin1> stop /system1

To start a power cycle a server:

admin1> reset /system1

To connect to the server’s serial console:

admin1> console com2

To exit the console and return to the admin1> prompt:

CTRL + \ (CTRL and backslash keyboard keys)

Create SSH Tunnel to the iDRAC Web InterfaceThis procedure details how to use Putty to create a tunnel to allow access to any host in the DCA through a single connection to the Primary or Standby Master server. The following scenarios represent use cases for this procedure:

• You are connected to the customer network locally and have access to the DCA through its external interface.

• You are connected to the customer network remotely (Webex or ESRS) and have access to the DCA through its external interface.

Do not use this procedure if you can connect to the DCA through the red service cable. Also, you cannot use this procedure to access to the Dell iDRAC web interface on the server through which you are connected. For example, if you are connected to the Primary Master server, you cannot access the Dell iDRAC web interface on the Primary Master server; You can access the Dell iDRAC on the Standby Master server.

1. Open the Putty client and establish an SSH connection to the Primary or Standby Master server.

Create SSH Tunnel to the iDRAC Web Interface 71



2. Click the icon in the top-left side of the Putty window and select Change Settings.

Figure C.3 Putty Client — Change Settings

3. In the Putty Reconfiguration window, go to Connection > SSH > Tunnels.

Figure C.4 Putty Client — SSH Tunnel Configuration 1

4. Enter the following information then click Add:

• Source port: 443

• Destination: 172.28.0.#:443where # is the server number. For example, the destination IP address for segment host sdw7 would be 172.28.0.7.

5. Enter the following information then click Add:




• Source port: 5900

• Destination: 172.28.0.#:5900where # is the server number. For example, the destination IP address for segment host sdw7 would be 172.28.0.7.

Figure C.5 Putty Client — SSH Tunnel Configuration 2

6. Click Apply. An SSH tunnel is now available to the specified server.

7. To access the Dell iDRAC web interface, open a web browser and navigate to:

https://localhost:443

Figure C.6 iDRAC Login Window






DCA Expansion Guide – Appendix D: Linux and vi Command Reference

D. Linux and vi Command Reference

This appendix is a quick reference for basic Red Hat linux commands, Greenplum-specific linux commands, and for common Vi text editor commands.

Common Linux CommandsThe following is a linux command line reference for common operations.

Table D.1 Common Linux commands

Linux command Description

Moving Around

/ refers to the root directory

.. refers to the parent directory

Up/down arrows repeats the last (up arrow) or next (down arrow) command you typed

pwd displays the current directory

cd name changes to the named directory

cd returns you to your home directory

Basic Commands

ls lists the contents of the current directory

ls dir_name lists the content of the named directory

ls -l lists the content of the named directory in long format; this includes file permissions, ownership information, and file size

ls -a lists all the files in the named directory including files that start with a period (“.)”

cat filename prints the content of the named file to the screen, one page at a time

more filename prints the content of the named file to the screen, with scrolling and search facilities

cp source destination

copies the source file to the named destination

for example: cp /misc/temp .copies a file called temp located in the misc directory, to the current directory (“.”)

mv source destination

moves the source file or directory to the named destination

for example: mv /misc/temp .this moves a file called temp located in the misc directory, to the current directory (“.”)

rm filename deletes (removes) the named file

mkdir dir_name creates a new directory

Common Linux Commands 75



rmdir dir_name removes the specified directory (directory must be empty)

source source path information

su assume the super user (root) identity

tar untars a tape archived and compressed file

unzip extracts compressed files from a ZIP archive

grep string filename

prints all the lines in a file that contain the specified string

su temporarily become the superuser - useful for system administration tasks

passwd allows you to change the password used to access your user account. You are prompted to enter your current password, then enter a new one.

who displays a list of users currently logged onto this computer

Getting Help

man command displays a (manual page (man) about the specified command, possible options and switches, and more detailed information about using that command

Shutting down and rebooting a Linux machine

/sbin/shutdown - r now

reboots the machine immediately

/sbin/shutdown - h now

shuts down the machine immediately

Greenplum Linux Commands

gpcheck verifies and validates Greenplum Database platform settings

gpexpand expands an existing Greenplum Database across new hosts in the array

gpinitsystem initializes a Greenplum Database system by using configuration parameters specified in the gp_init_config file

gpinitstandby adds and/or initializes a standby master host for a Greenplum Database system

gpseginstall installs the Greenplum Database software on multiple hosts

gpscp copies files between multiple hosts at the same time

gpssh-exkeys provides ssh access to multiple hosts at the same time

gpstate verifies the DCA master server status

Table D.1 Common Linux commands

Linux command Description

Common Linux Commands 76



vi Quick ReferenceThe following is a quick reference for the vi editor.

Table D.2 Common vi commands

vi command meaning

Inserting/Deleting TextTo leave insert mode press the [ESC] key.

a append text, after the cursor

i insert text, before the cursor

R enter overtype mode

x delete character

dd delete current line

Moving Cursor

h, [BACKSPACE] left one character

l, [SPACE] right one character

w forward one word

b back one word

e end of word

j down one line

k up one line

?pattern search backward for pattern

/pattern search forward for pattern

n repeat last search

N repeat last search in the opposite direction

Saving File and Exiting

:wq save file and quit

:q! force quit the editor, do not save changes

vi Quick Reference 77


DCA Expansion Guide – Appendix E: Enable Secured Mode

E. Enable Secured Mode

By default, the DCA ships in a non-hardened state. At the time of installation or upgrade, secured mode can be enabled. Secured mode is based on the Department of Defense DSA STIG guidelines. This chapter is OPTIONAL and should only be done if required by the customer for security reasons. The following sections are included:

• Enable Secured Mode on the DCA

• Revert to Unsecured Mode on the DCA

Enable Secured Mode on the DCAPerform the following steps to enable secured mode on the DCA. This procedure will cause temporary unavailability for the Greenplum Database.

1. Log into the DCA master server as the user gpadmin.

2. Stop the Greenplum Database if it is running:

$ gpstop -af

3. Switch to the user root:

$ su -

4. Stop the health monitoring service:


5. Run the secure mode upgrade utility. The host file specified should contain the name of ALL hosts in the system - this file should already be present if the dca_setup utility was run as part of installation.


# /opt/dca/sbin/dca_upgrade-1.2.0.0s.bin -u -f hostfile

6. The script will upgrade software on each server - the Primary Master server will be rebooted last, requiring you to log back in. Wait 15 minutes for the upgrade process to complete before attempting to log in the Primary Master Server.

7. Log back into the Primary Master server and verify the script completed successfully


# /opt/dca/sbin/dca_upgrade-1.2.0.0s.bin -s -f hostfile

8. Start the Greenplum Database:

# su - gpadmin

$ gpstart -a

9. Confirm the appliance version file reads 1.2.0.0s:

# cat /etc/gpdb-appliance-version

Enable Secured Mode on the DCA 78


DCA Expansion Guide – Appendix E: Enable Secured Mode

10. Verify FIPS-compliant mode of SSH is enabled, a value of 1 should be returned:

# cat /proc/sys.crypto/fips_enabled

Revert to Unsecured Mode on the DCAPerform the following steps to revert a system that is in secured mode to an unsecured state. This procedure will cause temporary unavailability for the Greenplum Database.

1. Log into the DCA master server as the user gpadmin.

2. Stop the Greenplum Database if it is running:

$ gpstop -af

3. Switch to the user root.

$ su -

4. Stop the healthmon daemon


5. Run the secure mode upgrade utility with revert option. The host file specified should contain the name of ALL hosts in the system.


# /opt/dca/sbin/dca_upgrade-1.2.0.0s.bin -r -f hostfile

6. The script will revert to the unsecured settings on each server - the Primary Master server will be rebooted last, requiring you log back in. Wait 15 minutes for the revert process to complete before attempting to log in the Primary Master Server.

7. Log back into the Primary Master Server and verify the script completed successfully


# /opt/dca/sbin/dca_upgrade-1.2.0.0s.bin -s -f hostfile

8. Start the healthmon daemon and the Greenplum Database

# dca_healthmon_ctl -e

# su - gpadmin

$ gpstart -a

9. Confirm the appliance version file reads 1.2.0.0:

# cat /etc/gpdb-appliance-version

10. Verify FIPS-compliant mode of SSH is disabled, a value of 0 should be returned:

# cat /proc/sys.crypto/fips_enabled

Revert to Unsecured Mode on the DCA 79


DCA Expansion Guide – Brocade VDX-6720 configuration

F. Brocade VDX-6720 configurationThis appendix contains configuration information Brocade VDX-6720 aggregation switches in a 12 rack DCA.

At the switch(config)# prompt, enter the following commands to configure the switch ports. Change the values in BOLD based on the location of the switch you are configuring.

diag post enabledpod 0/1 reserve!dpod 0/2 reserve!dpod 0/3 reserve!dpod 0/4 reserve!dpod 0/5!dpod 0/6 reserve!dpod 0/7 reserve!dpod 0/8 reserve!dpod 0/9!dpod 0/10!dpod 0/11 reserve!dpod 0/12 reserve!dpod 0/13 reserve!dpod 0/14 reserve!dpod 0/15!dpod 0/16 reserve

Running H/F 1 80



!dpod 0/17 reserve!dpod 0/18 reserve!dpod 0/19!dpod 0/20!dpod 0/21 reserve!dpod 0/22 reserve!dpod 0/23 reserve!dpod 0/24 reserve!dpod 0/25!dpod 0/26!dpod 0/27 reserve!dpod 0/28 reserve!dpod 0/29 reserve!dpod 0/30!dpod 0/31!dpod 0/32!dpod 0/33!dpod 0/34!dpod 0/35!dpod 0/36!dpod 0/37!dpod 0/38

81



!dpod 0/39!dpod 0/40!dpod 0/41!dpod 0/42!dpod 0/43!dpod 0/44!dpod 0/45!dpod 0/46!dpod 0/47!dpod 0/48!dpod 0/49!dpod 0/50!dpod 0/51!dpod 0/52!dpod 0/53!dpod 0/54!dpod 0/55!dpod 0/56!dpod 0/57!dpod 0/58!dpod 0/59!dpod 0/60!logging rbridge-id 1 raslog console INFO!logging auditlog class SECURITYlogging auditlog class CONFIGURATIONlogging auditlog class FIRMWAREswitch-attributes 1 chassis-name aggr-sw-1

82



host-name aggr-sw-1!support rbridge-id 1 ffdc!snmp-server contact "Field Support."snmp-server location "End User Premise."snmp-server sys-descr "Brocade VDX Switch."snmp-server community ConvergedNetworksnmp-server community OrigEquipMfrsnmp-server community "Secret C0de"system-monitor fan threshold marginal-threshold 1 down-threshold 2system-monitor fan alert state removed action raslogsystem-monitor power threshold marginal-threshold 1 down-threshold 2system-monitor power alert state removed action raslogsystem-monitor temp threshold marginal-threshold 1 down-threshold 2system-monitor compact-flash threshold marginal-threshold 1 down-threshold 0line vty exec-timeout 10!aaa authentication login localrole name admin desc Administratorrole name user desc Userservice password-encryptionusername admin password "O0cH0j8nlcFNrXNZpM1CHg==\n" encryption-level 7 role admin desc Administratorusername user password "O0cH0j8nlcFNrXNZpM1CHg==\n" encryption-level 7 role user desc Usercee-map default precedence 1 priority-group-table 1 weight 40 pfc on priority-group-table 15.0 pfc off priority-group-table 15.1 pfc off priority-group-table 15.2 pfc off priority-group-table 15.3 pfc off priority-group-table 15.4 pfc off priority-group-table 15.5 pfc off priority-group-table 15.6 pfc off priority-group-table 15.7 pfc off priority-group-table 2 weight 60 pfc off priority-table 2 2 2 1 2 2 2 15.0!interface Vlan 1 no shutdown!interface Vlan 199 description vlan_199 no shutdown!

83



protocol spanning-tree rstp bridge-priority 8192!protocol lldp advertise dcbx-tlv!no vlan dot1q tag nativeport-profile default vlan-profile switchport switchport mode trunk switchport trunk allowed vlan all switchport trunk native-vlan 1 !!interface Management 1/0 no ip address dhcp ip address 172.28.0.249/21 ip gateway-address 172.28.0.1 ipv6 address "" no ipv6 address autoconfig!interface TenGigabitEthernet 0/1 no fabric isl enable no fabric trunk enable channel-group 1 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/2 no fabric isl enable no fabric trunk enable channel-group 1 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/3 no fabric isl enable no fabric trunk enable channel-group 1 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/4 no fabric isl enable no fabric trunk enable channel-group 1 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/5 no fabric isl enable

84



no fabric trunk enable shutdown!interface TenGigabitEthernet 0/6 no fabric isl enable no fabric trunk enable channel-group 7 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/7 no fabric isl enable no fabric trunk enable channel-group 7 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/8 no fabric isl enable no fabric trunk enable channel-group 7 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/9 no fabric isl enable no fabric trunk enable channel-group 7 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/10 no fabric isl enable no fabric trunk enable shutdown!interface TenGigabitEthernet 0/11 no fabric isl enable no fabric trunk enable channel-group 2 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/12 no fabric isl enable no fabric trunk enable channel-group 2 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/13 no fabric isl enable no fabric trunk enable

85



channel-group 2 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/14 no fabric isl enable no fabric trunk enable channel-group 2 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/15 no fabric isl enable no fabric trunk enable shutdown!interface TenGigabitEthernet 0/16 no fabric isl enable no fabric trunk enable channel-group 8 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/17 no fabric isl enable no fabric trunk enable channel-group 8 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/18 no fabric isl enable no fabric trunk enable channel-group 8 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/19 no fabric isl enable no fabric trunk enable channel-group 8 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/20 no fabric isl enable no fabric trunk enable shutdown!interface TenGigabitEthernet 0/21 no fabric isl enable no fabric trunk enable channel-group 3 mode active type brocade

86



lacp timeout short no shutdown!interface TenGigabitEthernet 0/22 no fabric isl enable no fabric trunk enable channel-group 3 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/23 no fabric isl enable no fabric trunk enable channel-group 3 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/24 no fabric isl enable no fabric trunk enable channel-group 3 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/25 no fabric isl enable no fabric trunk enable shutdown!interface TenGigabitEthernet 0/26 no fabric isl enable no fabric trunk enable channel-group 9 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/27 no fabric isl enable no fabric trunk enable channel-group 9 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/28 no fabric isl enable no fabric trunk enable channel-group 9 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/29 no fabric isl enable no fabric trunk enable

87



channel-group 9 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/30 no fabric isl enable no fabric trunk enable shutdown!interface TenGigabitEthernet 0/31 no fabric isl enable no fabric trunk enable channel-group 4 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/32 no fabric isl enable no fabric trunk enable channel-group 4 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/33 no fabric isl enable no fabric trunk enable channel-group 4 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/34 no fabric isl enable no fabric trunk enable channel-group 4 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/35 no fabric isl enable no fabric trunk enable shutdown!interface TenGigabitEthernet 0/36 no fabric isl enable no fabric trunk enable channel-group 10 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/37 no fabric isl enable no fabric trunk enable channel-group 10 mode active type brocade

88



lacp timeout short no shutdown!interface TenGigabitEthernet 0/38 no fabric isl enable no fabric trunk enable channel-group 10 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/39 no fabric isl enable no fabric trunk enable channel-group 10 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/40 no fabric isl enable no fabric trunk enable shutdown!interface TenGigabitEthernet 0/41 no fabric isl enable no fabric trunk enable channel-group 5 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/42 no fabric isl enable no fabric trunk enable channel-group 5 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/43 no fabric isl enable no fabric trunk enable channel-group 5 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/44 no fabric isl enable no fabric trunk enable channel-group 5 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/45 no fabric isl enable no fabric trunk enable

89



shutdown!interface TenGigabitEthernet 0/46 no fabric isl enable no fabric trunk enable channel-group 11 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/47 no fabric isl enable no fabric trunk enable channel-group 11 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/48 no fabric isl enable no fabric trunk enable channel-group 11 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/49 no fabric isl enable no fabric trunk enable channel-group 11 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/50 no fabric isl enable no fabric trunk enable shutdown!interface TenGigabitEthernet 0/51 no fabric isl enable no fabric trunk enable channel-group 6 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/52 no fabric isl enable no fabric trunk enable channel-group 6 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/53 no fabric isl enable no fabric trunk enable channel-group 6 mode active type brocade

90



lacp timeout short no shutdown!interface TenGigabitEthernet 0/54 no fabric isl enable no fabric trunk enable channel-group 6 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/55 no fabric isl enable no fabric trunk enable shutdown!interface TenGigabitEthernet 0/56 no fabric isl enable no fabric trunk enable channel-group 12 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/57 no fabric isl enable no fabric trunk enable channel-group 12 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/58 no fabric isl enable no fabric trunk enable channel-group 12 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/59 no fabric isl enable no fabric trunk enable channel-group 12 mode active type brocade lacp timeout short no shutdown!interface TenGigabitEthernet 0/60 no fabric isl enable no fabric trunk enable shutdown!interface Port-channel 1 description 1 switchport switchport mode trunk switchport trunk allowed vlan add 199

91



switchport trunk tag native-vlan no shutdown!interface Port-channel 2 description 2 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 3 description 3 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 4 description 4 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 5 description 5 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 6 description 6 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 7 description 7 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 8 description 8

92



switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 9 description 9 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 10 description 10 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 11 description 11 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!interface Port-channel 12 description 12 switchport switchport mode trunk switchport trunk allowed vlan add 199 switchport trunk tag native-vlan no shutdown!mac-address-table aging-time 0

93


Date post:	17-Aug-2020
Category:	Documents
Upload:	others
View:	15 times
Download:	0 times

· Table of Contents i DCA Expansion Guide – Contents DCA Expansion Guide - Contents Preface

Documents