Cisco HyperFlex Systems Administration Guide for Kubernetes, … · calico-node-6mc7b 2/2 Running 0...

Cisco HyperFlex Systems Administration Guide for Kubernetes, Release3.5First Published: 2018-10-15

Last Modified: 2018-12-07

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706USAhttp://www.cisco.comTel: 408 526-4000

800 553-NETS (6387)Fax: 408 527-0883

C H A P T E R 1Cisco HyperFlex Storage Integration forKubernetes

• Overview, on page 1• Components, on page 2

OverviewThe Cisco HyperFlex Storage Integration for Kubernetes allows HyperFlex to dynamically provide persistentstorage to Kubernetes Pods running on HyperFlex. The integration enables orchestration of the entire PersistentVolume object lifecycle to be offloaded and managed by HyperFlex, while ultimately being driven (initiated)by developers and users through standard Kubernetes Persistent Volume Claim objects. Developers and usersget the benefit of leveraging HyperFlex for their Kubernetes persistent storage needs with zero additionaladministration overhead from their perspective.

With the HyperFlex Storage Integration for Kubernetes, each Persistent Volume object in Kubernetes isrepresented by an iSCSI-based LUN residing on the HyperFlex storage subsystem. Each iSCSI LUN is

Cisco HyperFlex Systems Administration Guide for Kubernetes, Release 3.51

presented to the Kubernetes nodes (VMs) by the scvmclient service running on the local ESXi host where theVMs reside. The Kubernetes nodes (VMs) themselves are running the software iSCSI initiator service whichallows them to mount the iSCSI LUNs provided by the iSCSI target. Each ESXi host is configured with ahost-only vSwitch, where all iSCSI-based traffic between the iSCSI target (scvmclient service in ESXi) andthe iSCSI initiators (Kubernetes VMs running locally on that ESXi host) resides.

ComponentsThere are two components that make up the Cisco HyperFlex Storage Integration for Kubernetes. Bothcomponents work in tandem to dynamically provision storage in HyperFlex and, ultimately, provide thatstorage to the appropriate Kubernetes Pod as a Persistent Volume object.

• HyperFlex FlexVolume Plug-in

• HyperFlex Kubernetes Storage Provisioner

HyperFlex FlexVolume Plug-in

The HyperFlex FlexVolume Plug-in is a plug-in that is developed by Cisco Systems and leverages theKubernetes FlexVolume “out-of-tree” framework. The Hyperflex FlexVolume plugin manages connectionsto the HyperFlex cluster from the Kubernetes nodes andmakes storage volumes available to containers throughthe Kubernetes FlexVolume interface.

HyperFlex Kubernetes Storage Provisioner

The HyperFlex Kubernetes Storage Provisioner is a container that is developed by Cisco Systems. Thiscontainer is deployed within the target Kubernetes cluster and serves as the storage provisioning orchestrationpoint for Persistent Volumes fromHyperFlex. Developers submit their application storage requirements usingPersistent Volume Claims and reference a specific StorageClass associated with HyperFlex. Kubernetes passesthe required storage request to the HyperFlex Kubernetes Storage Provisioner configured in the StorageClass.


Cisco HyperFlex Storage Integration for KubernetesComponents

C H A P T E R 2Kubernetes Support in HyperFlex Connect

• Kubernetes Integration in HyperFlex Connect, on page 3• Preventing FlexVolume Traffic Disruption, on page 3• Enabling Kubernetes Integration , on page 4• Partially Enabled Cluster Status, on page 5

Kubernetes Integration in HyperFlex ConnectKubernetes support must be explicitly enabled within HyperFlex Connect in order to use the HyperFlexStorage Integration for Kubernetes, regardless of whether you are using the Cisco Container Platform (CCP)or RedHat OpenShift Container Platform (OCP). EnablingKubernetes support in HyperFlex Connect configuresthe underlying HyperFlex storage subsystem to support the iSCSI-based LUNs (which ultimately maps toPersistent Volume objects). In addition, enabling Kubernetes support configures the required ESXi networkingto support iSCSI traffic between the iSCSI target (scvmclient service in each ESXi host) and iSCSI initiators(Kubernetes VMs residing locally on each ESXi host).

The ESXi network configuration required to support the HyperFlex Storage Integration for Kubernetes willnot disrupt any previous or existing network configuration in place on each ESXi host. In other words, enablingKubernetes support in HyperFlex Connect will not disrupt any virtual machines or applications currentlyrunning on the HyperFlex system.

Note

Preventing FlexVolume Traffic DisruptionAfter you upgrade your Kubernetes cluster from an older HXDP release to 3.5(2a) or later, ensure that thefollowing configuration changes are completed on all the storage controller VM nodes. This will avoid incorrectFlexVolume configuration on Kubernetes nodes.

Step 1 Open the application.conf file located at /opt/springpath/storfs-mgmt/stMgr-1.0/conf. Searchfor iscsiTargetAddress. Modify the value for this parameter from 169.254.254.1 to 169.254.1.1.

Step 2 Open the application.conf file located at /opt/springpath/storfs-mgmt/hxSvcMgr-1.0/conf/.Search for istgtConfTargetAddress. Modify the value for this parameter from 169.254.254.1 to 169.254.1.1.


Step 3 Run the following commands on all the storage controller VM nodes:

# restart hxSvcMgr# restart stMgr

Enabling Kubernetes IntegrationThe following procedure details the steps that are required to enable Kubernetes support in HyperFlex Connect:

1. Navigate to the HyperFlex cluster by using a supported web browser (for example,https://<hyperflex_cluster_management_IP_address).

2. Log in to HyperFlex Connect using a VMware SSO account and password with administrative privileges(that is, [email protected]).

3. In the upper right-hand corner of HyperFlex Connect, click the Settingsmenu icon (represented by a Gearicon).Figure 1: HyperFlex Connect Settings Menu

4. Under Integrations, click Kubernetes.


Kubernetes Support in HyperFlex ConnectEnabling Kubernetes Integration

Figure 2: Selecting Integrations > Kubernetes

5. On the Enable Persistent Volumes for Kubernetes page, the Current Status: value is Disabled for anew cluster. ClickEnable to configure the HyperFlex cluster to support Persistent Volumes for Kubernetes.Figure 3: Enable Persistent Volumes for Kubernetes Page

Partially Enabled Cluster StatusThere are certain circumstances where a previously enabled cluster may show a current status of PartiallyEnabled. This status typically appears when one of the following scenarios occur:

• Expansion of the HyperFlex cluster

• Change to required ESXi networking for iSCSI


Kubernetes Support in HyperFlex ConnectPartially Enabled Cluster Status

In either of the preceding scenarios, perform the procedure outlined in Enabling Kubernetes Integration , onpage 4 to reenable Kubernetes support in HyperFlex Connect. HyperFlex will ensure that all hosts areproperly enabled and configured. After reenabling Kubernetes support, the current status should change toEnabled.


Kubernetes Support in HyperFlex ConnectPartially Enabled Cluster Status

C H A P T E R 3Configuring HyperFlex FlexVolume StorageIntegration for Cisco Container Platform

• Support Matrix for HX FlexVolume Integration with CCP, on page 7• Prerequisites, on page 8• Creating Cisco Container Platform Tenant Cluster, on page 8• Managing HyperFlex FlexVolume Plug-in, on page 8• Managing HyperFlex FlexVolume Provisioner, on page 10• Configuring Storage Classes, on page 11• Provisioning Persistent Volumes, on page 11

Support Matrix for HX FlexVolume Integration with CCPThe following table summarizes the Cisco Container Platform (CCP) software versions that are supportedwith each of the HX Data Platform software versions.

Table 1: Support Matrix for HX FlexVolume Integration with CCP

CCP Version3.0

CCP Versions2.1, 2.2

CCP Version2.0

CCP Version1.5

CCP Versions1.1, 1.2, 1.3, 1.4

CCP Version1.0

HX DataPlatformVersion

———SupportedSupportedSupported3.0(1a) orlater

—PlannedSupportedNotSupported

——3.5(1a) orlater

—PlannedPlanned———3.5(2a) orlater

PlannedPlanned————4.0(1a) orlater

PlannedPlanned————4.1(1a) orlater


PrerequisitesThe following prerequisites must be met prior to configuring HyperFlex FlexVolume Storage Integration forCisco Container Platform.

• Cisco HyperFlex cluster is installed and running 3.5(1a) or later.

• Cisco Container Platform Control Plane is installed and running 2.0 or later.

Creating Cisco Container Platform Tenant ClusterDuring the CCP tenant cluster creation workflow within the CCP control plane, you must set the HyperFlexLocal Network option in order to install and configure the HyperFlex FlexVolume Storage Integration forKubernetes. Once the option is selected, CCPwill automatically install both the HyperFlex FlexVolume Pluginand HyperFlex FlexVolume Provisioner as part of the tenant cluster deployment.

The following section is an abbreviated description of the CCP “Create Cluster” workflow which highlightstheHyperFlex Local Network option to install the HyperFlex FlexVolume Storage Integration for Kubernetes.For more details on creating a new CCP tenant cluster, please refer to the CCP documentation.

Step 1 Log into the CCP control plane UI.Step 2 On the Clusters page, click New Cluster.Step 3 On the Basic Information page, enter the appropriate information and click Next.Step 4 On the Provider Settings page, enter the appropriate information and ensure the HyperFlex Local Network option is

set to k8-priv-iscsvm-network in order to tell CCP to install and configure the HyperFlex FlexVolume plug-in.Step 5 Click Next.Step 6 On the Summary page, view the cluster information and click Submit.

The CCP control plane will then deploy the requested CCP tenant cluster, including the installation and configuration ofall required components of the HyperFlex FlexVolume Storage Integration for Kubernetes.

Managing HyperFlex FlexVolume Plug-in

Installing HyperFlex FlexVolume Plug-inIf the HyperFlex Local Network option is configured properly when deploying the CCP tenant cluster, theCCP control plane automatically installs and configures the HyperFlex FlexVolume plug-in on the CCP tenantcluster. No additional configuration is required.


Configuring HyperFlex FlexVolume Storage Integration for Cisco Container PlatformPrerequisites

https://www.cisco.com/c/en/us/support/cloud-systems-management/container-platform/tsd-products-support-series-home.html

Checking HyperFlex FlexVolume Plug-in Version

The following steps must be performed on one CCP tenant cluster node only.Important

Step 1 Log into one of the tenant Kubernetes cluster nodes using SSH. Use the username that corresponds to the SSH keyprovided during the New Cluster workflow when provisioning the CCP tenant cluster through CCP.

Step 2 Run the following command to change directories to the/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex-hxvolume/ directory.cd /usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume/

Example:

ccpuser@tc1-mastercf1ff968f8:~$ cd

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume/$

ccpuser@tc1-mastercf1ff968f8:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume$

Step 3 Run the hxvolume version command as the root user (with sudo) to view the HyperFlex FlexVolume plug-in version.sudo hxvolume version

Example:

ccpuser@tc1-mastercf1ff968f8:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume$

sudo ./hxvolume version

hxvolume version: 1.0.284.git.4022e8e

ccpuser@tc1-mastercf1ff968f8:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume

Upgrading HyperFlex FlexVolume Plug-inUpgrading of the HyperFlex FlexVolume Plug-in is performed as part of the CCP tenant cluster upgradeprocess. Follow the steps that are provided in the CCP documentation to upgrade the CCP tenant cluster. Thisprocess includes upgrading the HyperFlex FlexVolume Plug-in to the latest available version.

Modifying Configuration for Kubernetes VMsComplete the following task when existing HyperFlex clusters use IP addresses in the range, 169.154.1.0 to169.154.1.24 for ESXi. After a Kubernetes cluster operation, such as scale up or upgrade, this procedure mustbe repeated on the ALL new VMs.

After an upgrade to HXDP release 3.5(2a), run the following command on eachKubernetes VM. This commandwill find the parameter "targetIp" in the hxflexvolume.json file, and replace the value from"169.254.254.1" to "169.254.254.1".


Configuring HyperFlex FlexVolume Storage Integration for Cisco Container PlatformChecking HyperFlex FlexVolume Plug-in Version

The <ssh user> must match the SSH user that was specified during cluster creation.

The <private key file> must correspond to the public key that was specified during cluster creation.

Note

Managing HyperFlex FlexVolume Provisioner

Installing HyperFlex FlexVolume ProvisionerIf the HyperFlex Local Network option is configured properly when deploying the CCP tenant cluster, theCCP control plane automatically installs and configures the HyperFlex FlexVolume Provisioner on the CCPtenant cluster. No additional configuration is required.

Checking HyperFlex FlexVolume Provisioner Version

Step 1 Run the kubectl get pods -n kube-system command to get the complete name of the deployed HyperFlex FlexVolumeProvisioner pod.kubectl get pods -n kube-system

Example:

ccpuser@admin-host:~$ kubectl get pods -n kube-systemNAME READY STATUS RESTARTS AGEcalico-node-6mc7b 2/2 Running 0 7dcalico-node-tjks9 2/2 Running 0 7dcalico-node-z4png 2/2 Running 0 7dcalico-typha-7d48f84746-crrb2 1/1 Running 0 7dcalico-typha-7d48f84746-vt6gm 1/1 Running 0 7detcd-tc1-mastercf1ff968f8 1/1 Running 0 7dhx-provisioner-f98479996-k79v6 1/1 Running 0 6dkube-apiserver-tc1-mastercf1ff968f8 1/1 Running 0 7dkube-controller-manager-tc1-mastercf1ff968f8 1/1 Running 0 7dkube-dns-6c74cdd686-k877b 3/3 Running 0 7dkube-proxy-8s6j6 1/1 Running 0 7dkube-proxy-f2d2z 1/1 Running 0 7dkube-proxy-vfqjz 1/1 Running 0 7dkube-scheduler-tc1-mastercf1ff968f8 1/1 Running 0 7dtiller-deploy-5c567bd778-7xr6d 1/1 Running 0 7dccpuser@admin-host:~$

Step 2 Run the kubectl describe pods... command to get the complete details of the deployed HyperFlex FlexVolumeProvisioner pod. Look for the hx-provisioner container image name which includes the version as a tag (that is, afterthe colon in the container name).kubectl describe pods <pod_name> -n kube-system

Example:

ccpuser@admin-host:~$ kubectl describe pods hx-provisioner-f98479996-k79v6 -n kube-systemName: hx-provisioner-f98479996-k79v6Namespace: kube-system


Configuring HyperFlex FlexVolume Storage Integration for Cisco Container PlatformManaging HyperFlex FlexVolume Provisioner

Node: tc1-worker87d761f2d0/172.0.13.116Start Time: Fri, 14 Sep 2018 21:23:41 -0400Labels: app=hx-provisioner

pod-template-hash=954035552Annotations: cni.projectcalico.org/podIP=192.168.2.11/32Status: RunningIP: 192.168.2.11Controlled By: ReplicaSet/hx-provisioner-f98479996Containers:hx-provisioner:Container ID: docker://f5cc3d45480a7a706264b965cd71ee7af47680393101d507ce36826e4e4b384fImage: hx-provisioner:0.10.274.git.365b059eImage ID: docker://sha256:0184783ed8cd143b786ab77654a9a1ec693c6c005adb22a988f39e0538e1b822Port: 443/TCPHost Port: 0/TCPArgs:-hxapi-url=$(HX_API_URL)-hxapi-token-file=/secrets/hxapi/token-hxapi-hxclusteruuid=$(HX_CLUSTERUUID)

State: Running

ccpuser@admin-host:~$

Upgrading HyperFlex FlexVolume ProvisionerUpgrading of the HyperFlex FlexVolume Provisioner is performed as part of the CCP tenant cluster upgrade.Follow the steps that are provided in CCP documentation to upgrade the CCP tenant cluster. This processincludes upgrading the HyperFlex FlexVolume Provisioner to the latest available version.

Configuring Storage ClassesIf the HyperFlex Local Network option was set properly when deploying the CCP tenant cluster, the CCPcontrol plane automatically creates a StorageClass for HyperFlex on the CCP tenant cluster. By default, theHyperFlex StorageClass is not set as the default StorageClass in the CCP tenant cluster. In this case, by default,developers must explicitly specify HyperFlex as the StorageClass in Persistent Volume Claims to use theHyperFlex FlexVolume storage integration.

Use the kubectl get sc command to view the StorageClasses on the CCP tenant cluster.

ccpuser@admin-host:~$ kubectl get scNAME PROVISIONER AGEhyperflex hyperflex.io/hxvolume 4sstandard (default) kubernetes.io/vsphere-volume 4sccpuser@admin-host:~$

Provisioning Persistent Volumes

Step 1 Run the following commands to create a Persistent Volume Claim YAML file that includes a user- definedpersistent_volume_claim_name and size.


Configuring HyperFlex FlexVolume Storage Integration for Cisco Container PlatformUpgrading HyperFlex FlexVolume Provisioner

The storageClassName: hyperflex line is required in order to send the storage requests to the HyperFlex FlexVolumeProvisioner.

vi <path>/<filename>.yaml<insert the following>apiVersion: v1kind: PersistentVolumeClaimmetadata:name: <persistent_volume_claim_name>

spec:storageClassName: hyperflexaccessModes:- ReadWriteOnce

resources:requests:storage: <size>Gi

Example:

ccpuser@admin-host:~$ vi ~/hxkube/pvc.yamlapiVersion: v1kind: PersistentVolumeClaimmetadata:name: message-board-pvc

spec:storageClassName: hyperflexaccessModes:- ReadWriteOnce

resources:requests:storage: 100Gi

ccpuser@admin-host:~$

Step 2 Run the kubectl create command to submit the pvc.yaml file and create the Persistent Volume Claim object in the CCPtenant Kubernetes cluster. In parallel, as part of the operation, HyperFlex creates a Persistent Volume object to complementthe Persistent Volume Claim object and bind the two together in Kubernetes.kubectl create -f ~/hxkube/<pvc_name>.yaml

Example:

ccpuser@admin-host:~$ kubectl create -f ~/hxkube/pvc.yamlpersistentvolumeclaim/message-board-pvc createdccpuser@admin-host:~$

Step 3 Check the status of the Persistent Volume Claim object with the kubectl get pvc command to make sure it was createdsuccessfully and is bound to a Persistent Volume object.kubectl get pvc

Example:

ccpuser@admin-host:~$ kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEmessage-board-pvc Bound

Step 4 Deploy Kubernetes Pod using the kubectl create command while specifying the Persistent Volume Claim object inthe Pod YAML file.kubectl create -f <pod_yaml_file>

Example:


Configuring HyperFlex FlexVolume Storage Integration for Cisco Container PlatformProvisioning Persistent Volumes

apiVersion: v1kind: Podmetadata:

name: message-boardlabels:

app: message-boardname: message-board

namespace: defaultspec:

containers:- name: message-boardimage: michzimm/message_board:version1volumeMounts:- name: demovolume1mountPath: /sqldb

ports:- containerPort: 5000

volumes:- name: demovolume1persistentVolumeClaim:

Example:

ccpuser@admin-host:~$ kubectl create -f ./message-board.yamlpod/message-board createdccpuser@admin-host:~$

Step 5 Check the status of the deployed Pod to ensure it is running.kubectl get pods

Example:

ccpuser@admin-host:~$ kubectl get podsNAME READY STATUS RESTARTS AGEmessage-board 1/1 Running 0 35mccpuser@admin-host:~$





C H A P T E R 4Configuring HyperFlex FlexVolume StorageIntegration for RedHat OpenShift ContainerPlatform

• Support Matrix for HX FlexVolume Integration with OCP, on page 15• Prerequisites, on page 16• Setting Up an Administrator Host, on page 16• Command Execution, on page 17• Deploying RedHat OpenShift Container Platform, on page 17• Distributing HyperFlex FlexVolume Software, on page 18• Managing HyperFlex FlexVolume Plug-in, on page 23• Managing HyperFlex FlexVolume Provisioner , on page 31• Configuring Storage Classes, on page 35• Provisioning Persistent Volumes, on page 36

Support Matrix for HX FlexVolume Integration with OCPThe following table summarizes the Red Hat OpenShift Container Platform (OCP) software versions that aresupported with each of the HX Data Platform software versions.

Table 2: Support Matrix for HX FlexVolume Integration with Red Hat OCP

Red Hat OCPVersion 3.13







————NotSupported

NotSupported

3.0(1a) orlater

———SupportedSupported—3.5(1a) orlater

———PlannedPlanned—3.5(2a) orlater









——PlannedPlanned——4.0(1a) orlater

TBDTBDPlannedPlanned——4.1(1a) orlater

PrerequisitesThe following prerequisites must be met before configuring HyperFlex FlexVolume Storage Integration forRedHat OpenShift Container Platform.

• Cisco HyperFlex cluster is installed and running 3.5(1a) or later

• RedHat OpenShift Container Platform installed and running version 3.9 or later

• Downloaded the latest HyperFlex Kubernetes bundle (zip) file from the HyperFlex HX Data Platformsection of Cisco Software Downloads.

Setting Up an Administrator HostIn the context of this document, the administrator host refers to a Linux-based host used for remotelyadministering the OpenShift cluster. This document does not dictate which Linux distribution should be usedfor the administrator host operating system, however some commands may vary slightly based on thedistribution that is used. The administrator host may be a newly deployed host, or it can also be an existinghost in the environment. The server used for the automated installation of the OpenShift cluster using Ansible,also known as the “bastion” node, makes a good candidate to use as the administrator host as it typicallyalready meets most of the required prerequisites, such as OpenShift node connectivity, password-less SSHaccess, and so on. The examples in the following sections use the “bastion” node as the administrator host.

Perform the following steps on the administrator host.Important

Step 1 Ensure the Kubernetes command-line toolset oc is installed and configured to manage the target OpenShift cluster. If thetoolset is not installed, you can find the procedure based on Linux distribution here:https://kubernetes.io/docs/tasks/tools/install-oc/#install-oc.

Step 2 Ensure an SSH (public and private) keypair has been generated. The SSH keypair is used to manage the remote OpenShiftcluster. Ensure that password-less SSH authentication works between the administrator host and all OpenShift clusternodes.

Step 3 Download the latest HyperFlex Kubernetes bundle (zip) file from the HyperFlex HX Data Platform section of CiscoSoftware Downloads.. Transfer the HyperFlex Kubernetes bundle (zip) file to the administrator host using any preferredmethod, such as scp. The remainder of this document assumes the HyperFlex Kubernetes bundle (zip) file has been copiedto the following directory path ~/hxkube on the administrator host.


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformPrerequisites

https://software.cisco.com/download/home/286305544/type/286305994/release/1.8%25281c%2529

https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl



By default, the ~/hxkube directory does not exist and will need to be created.Note

Step 4 Unzip the HyperFlex Kubernetes bundle (zip) file to the ~/hxkube directory on the administrator host. You may need toinstall the unzip package using a package manager (for example, yum or apt-get) based on the administrator host’sLinux distribution.

Command ExecutionThe sections in this chapter are related to OpenShift and require that some commands be repeated acrossmultiple nodes in the OpenShift cluster. You may manually run each command on all required OpenShiftnodes, however this is highly repetitive. It is recommended to leverage a “while” loop in order to iteratethrough a list of all OpenShift nodes and execute the required commands.

Example: Using a “while” loop and a text file containing a list of OpenShift nodes

Create a file containing the IP addresses or hostnames of all OpenShift nodes.administrator-host:~/hxkube$ vi ./ocp_nodes.txtocp-masterocp-infra1ocp-infra2ocp-node1ocp-node2

Iterate through the list of OpenShift nodes and run the command on each node.administrator-host:~/hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n <ocpuser>@$host <command>; \done

administrator-host:~/hxkube$

Hostnames or IP addresses can be used for each OpenShift node in the text file for the “while” loop.Note

Pay close attention to which commands should be run on which nodes, not all commands are run on all nodes.Note

Deploying RedHat OpenShift Container PlatformRedHat provides Ansible playbooks as a standard mechanism for automating the installation of a RedHatOpenShift Container Platform cluster. Extensive documentation and information can be found on the RedHatwebsite for installing and configuring an OpenShift cluster using Ansible. The subsequent sections assume arunning instance of RedHat OpenShift Container Platform exists or has been installed using standard RedHatmethods and best practices, as found on the RedHat website.


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformCommand Execution

Add an additional virtual machine network interface for each OpenShift node and attach it to thek8-priv-iscsivm-networkVMware port-group. This interface is required to use the HyperFlex FlexVolumeStorage Integration for OpenShift.

Important

The additional interface can be added during OpenShift node deployment or can be added after deploymentby editing the VMware virtual machine settings, as long as the additional interface exists prior to movingforward with the HyperFlex FlexVolume Storage Integration for OpenShift installation. There is no need toconfigure the added interface within the Operating System; this action will be done as part of the HyperFlexFlexVolume Storage Integration for OpenShift installation.

Note

Distributing HyperFlex FlexVolume SoftwareIn order to properly install and configure the HyperFlex FlexVolume Storage Integration for OpenShift,distribute the HyperFlex Kubernetes bundle (zip) file across all OpenShift cluster nodes. The following stepsdetail the process for distributing the HyperFlex Kubernetes bundle (zip) file to the appropriate hosts.

Step 1 Run the following command to create a directory named hxkube on each OpenShift cluster node.

Scope:

Run on all OpenShift nodes.

Command:mkdir ~/hxkube

Example:administrator-host:~/hxkube$cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n @$host sudo mkdir ~/hxkube; \

doneadministrator-host:~/hxkube$

Step 2 Copy the HX Kubernetes archive file to the ~/hxkube directory on each OpenShift cluster node.

Scope:

Run on administrator-host, copying to all OpenShift nodes.

Command:scp ~/hxkube/HX-Kubernetes-X.X.XXX.XXX.XXXXXXXX.XXXX.zip <ocpuser>@<remote-host>:<path>

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do scp ~/hxkube/HX-Kubernetes-1.0.284.git.4022e8ec.hx35.zip $host:~/hxkube; \done

HX-Kubernetes-1.0.284.git.4022e8ec.hx35.zip100% 37MB 74.7MB/s 00:00


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformDistributing HyperFlex FlexVolume Software





administrator-host:hxkube$

Step 3 Install the unzip package on each OpenShift cluster node using the yum package manager.

Scope:


Commands:sudo yum install -y unzip

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo yum install -y unzip; \done

Loaded plugins: product-id, search-disabled-repos, subscription-managerResolving Dependencies--> Running transaction check---> Package unzip.x86_64 0:6.0-19.el7 will be installed--> Finished Dependency Resolution

Dependencies Resolved

================================================================================Package Arch Version Repository Size================================================================================Installing:unzip x86_64 6.0-19.el7 rhel-7-server-rpms 170 k

Transaction Summary================================================================================Install 1 Package

Total download size: 170 kInstalled size: 365 kDownloading packages:Running transaction checkRunning transaction testTransaction test succeededRunning transactionInstalling : unzip-6.0-19.el7.x86_64 1/1Verifying : unzip-6.0-19.el7.x86_64 1/1

Installed:unzip.x86_64 0:6.0-19.el7

Complete!Loaded plugins: product-id, search-disabled-repos, subscription-managerResolving Dependencies--> Running transaction check---> Package unzip.x86_64 0:6.0-19.el7 will be installed--> Finished Dependency Resolution




























Complete!administrator-host:hxkube$

Step 4 Unzip the Kubernetes bundle (zip) file on each OpenShift cluster node.

Scope:




Command:sudo unzip ~/hxkube/HX-Kubernetes-X.X.XXX.XXX.XXXXXXXX.XXXX.zip -d ~/hxkube

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo unzip ~/hxkube/HX-Kubernetes-1.0.284.git.4022e8ec.hx35.zip -d ~/hxkube; \done

Archive: /root/hxkube/HX-Kubernetes-1.0.284.git.4022e8ec.hx35.zipextracting: /root/hxkube/RELEASE.txtextracting: /root/hxkube/hx-provisioner-setupextracting: /root/hxkube/hx-provisioner.tar.gzextracting: /root/hxkube/hxkube-collect-logsextracting: /root/hxkube/hxprovisioner-deploy.yamlextracting: /root/hxkube/hxvolumeextracting: /root/hxkube/hxvolume-plugin-1.0.284.git.4022e8ec.hx35-1.x86_64.rpmextracting: /root/hxkube/hxvolume-plugin_1.0.284.git.4022e8ec.hx35_amd64.debextracting: /root/hxkube/istgttoolArchive: /root/hxkube/HX-Kubernetes-1.0.284.git.4022e8ec.hx35.zipextracting: /root/hxkube/RELEASE.txtextracting: /root/hxkube/hx-provisioner-setupextracting: /root/hxkube/hx-provisioner.tar.gzextracting: /root/hxkube/hxkube-collect-logsextracting: /root/hxkube/hxprovisioner-deploy.yamlextracting: /root/hxkube/hxvolumeextracting: /root/hxkube/hxvolume-plugin-1.0.284.git.4022e8ec.hx35-1.x86_64.rpmextracting: /root/hxkube/hxvolume-plugin_1.0.284.git.4022e8ec.hx35_amd64.debextracting: /root/hxkube/istgttoolArchive: /root/hxkube/HX-Kubernetes-1.0.284.git.4022e8ec.hx35.zipextracting: /root/hxkube/RELEASE.txtextracting: /root/hxkube/hx-provisioner-setupextracting: /root/hxkube/hx-provisioner.tar.gzextracting: /root/hxkube/hxkube-collect-logsextracting: /root/hxkube/hxprovisioner-deploy.yamlextracting: /root/hxkube/hxvolumeextracting: /root/hxkube/hxvolume-plugin-1.0.284.git.4022e8ec.hx35-1.x86_64.rpmextracting: /root/hxkube/hxvolume-plugin_1.0.284.git.4022e8ec.hx35_amd64.debextracting: /root/hxkube/istgttoolArchive: /root/hxkube/HX-Kubernetes-1.0.284.git.4022e8ec.hx35.zipextracting: /root/hxkube/RELEASE.txtextracting: /root/hxkube/hx-provisioner-setupextracting: /root/hxkube/hx-provisioner.tar.gzextracting: /root/hxkube/hxkube-collect-logsextracting: /root/hxkube/hxprovisioner-deploy.yamlextracting: /root/hxkube/hxvolumeextracting: /root/hxkube/hxvolume-plugin-1.0.284.git.4022e8ec.hx35-1.x86_64.rpmextracting: /root/hxkube/hxvolume-plugin_1.0.284.git.4022e8ec.hx35_amd64.debextracting: /root/hxkube/istgttoolArchive: /root/hxkube/HX-Kubernetes-1.0.284.git.4022e8ec.hx35.zipextracting: /root/hxkube/RELEASE.txtextracting: /root/hxkube/hx-provisioner-setupextracting: /root/hxkube/hx-provisioner.tar.gzextracting: /root/hxkube/hxkube-collect-logsextracting: /root/hxkube/hxprovisioner-deploy.yamlextracting: /root/hxkube/hxvolumeextracting: /root/hxkube/hxvolume-plugin-1.0.284.git.4022e8ec.hx35-1.x86_64.rpmextracting: /root/hxkube/hxvolume-plugin_1.0.284.git.4022e8ec.hx35_amd64.debextracting: /root/hxkube/istgttooladministrator-host:hxkube$



Managing HyperFlex FlexVolume Plug-in

Installing HyperFlex FlexVolume Plug-in

Step 1 The Ansible installation playbooks for OpenShift should by default install the iscsi-initiator-utils package.Verify that the package is installed.

Scope:


Command:sudo yum list installed iscsi-initiator-utils

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo yum list installed iscsi-initiator-utils; \done

Loaded plugins: product-id, search-disabled-repos, subscription-managerInstalled Packagesiscsi-initiator-utils.x86_64 6.2.0.874-10.el7 @rhel-7-server-rpmsLoaded plugins: product-id, search-disabled-repos, subscription-managerInstalled Packagesiscsi-initiator-utils.x86_64 6.2.0.874-10.el7 @rhel-7-server-rpmsLoaded plugins: product-id, search-disabled-repos, subscription-managerInstalled Packagesiscsi-initiator-utils.x86_64 6.2.0.874-10.el7 @rhel-7-server-rpmsLoaded plugins: product-id, search-disabled-repos, subscription-managerInstalled Packagesiscsi-initiator-utils.x86_64 6.2.0.874-10.el7 @rhel-7-server-rpmsLoaded plugins: product-id, search-disabled-repos, subscription-managerInstalled Packagesiscsi-initiator-utils.x86_64 6.2.0.874-10.el7 @rhel-7-server-rpmsadministrator-host:hxkube$

Step 2 If, for some reason, the iscsi-initiator-utils package is not installed, install it using the yum packagemanager.

Scope:


Command:sudo yum install -y iscsi-initiator-utils

Example:administrator-host:~/hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo yum install -y iscsi-initiator-utils; \done


Step 3 Install the avahi-autoipd package using the yum package manager.


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformManaging HyperFlex FlexVolume Plug-in

Scope:


Command:sudo yum install -y avahi-autoipd

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo yum install -y avahi-autoipd; \done

Loaded plugins: product-id, search-disabled-repos, subscription-managerResolving Dependencies--> Running transaction check---> Package avahi-autoipd.x86_64 0:0.6.31-19.el7 will be installed--> Finished Dependency Resolution


================================================================================Package Arch Version Repository Size================================================================================Installing:avahi-autoipd x86_64 0.6.31-19.el7 rhel-7-server-rpms 40 k


Total download size: 40 kInstalled size: 44 kDownloading packages:Running transaction checkRunning transaction testTransaction test succeededRunning transactionInstalling : avahi-autoipd-0.6.31-19.el7.x86_64 1/1Verifying : avahi-autoipd-0.6.31-19.el7.x86_64 1/1

Installed:avahi-autoipd.x86_64 0:0.6.31-19.el7

Complete!Loaded plugins: product-id, search-disabled-repos, subscription-managerResolving Dependencies--> Running transaction check---> Package avahi-autoipd.x86_64 0:0.6.31-19.el7 will be installed--> Finished Dependency Resolution




Total download size: 40 k


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformInstalling HyperFlex FlexVolume Plug-in

Installed size: 44 kDownloading packages:Running transaction checkRunning transaction testTransaction test succeededRunning transactionInstalling : avahi-autoipd-0.6.31-19.el7.x86_64 1/1Verifying : avahi-autoipd-0.6.31-19.el7.x86_64 1/1












Total download size: 40 k



Installed size: 44 kDownloading packages:Running transaction checkRunning transaction testTransaction test succeededRunning transactionInstalling : avahi-autoipd-0.6.31-19.el7.x86_64 1/1Verifying : avahi-autoipd-0.6.31-19.el7.x86_64 1/1








Complete!administrator-host:hxkube

Step 4 In the Deploying RedHat OpenShift Container Platform section, you were instructed to add an additional virtual machinenetwork interface to each OpenShift node. The interface name of the added virtual machine network interface is nowrequired in order to proceed with the installation.

Use the ifconfig -a command to find the name of the interface on one of the OpenShift nodes. The interface shouldnot have an IP address assigned and it is recommended to cross reference the MAC address with VMware vCenter inorder to ensure the correct interface. In this particular environment the added interface is named ens224.

The ifconfig -a command only needs to be run on a single OpenShift node.Note

Scope:

Run on a single OpenShift node.

Command:



sudo ifconfig -a

Example:administrator-host:hxkube$ ssh ocp-master sudo ifconfig -adocker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500

inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0ether 02:42:e8:b1:c0:1e txqueuelen 0 (Ethernet)RX packets 0 bytes 0 (0.0 B)RX errors 0 dropped 0 overruns 0 frame 0TX packets 0 bytes 0 (0.0 B)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500inet 10.2.17.150 netmask 255.255.255.0 broadcast 10.2.17.255ether 00:50:56:98:ad:b3 txqueuelen 1000 (Ethernet)RX packets 799305 bytes 992373512 (946.4 MiB)RX errors 0 dropped 73 overruns 0 frame 0TX packets 275804 bytes 142731217 (136.1 MiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500ether 00:50:56:98:05:fd txqueuelen 1000 (Ethernet)RX packets 654 bytes 217464 (212.3 KiB)RX errors 0 dropped 2 overruns 0 frame 0TX packets 629 bytes 215118 (210.0 KiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536inet 127.0.0.1 netmask 255.0.0.0loop txqueuelen 1000 (Local Loopback)RX packets 3120877 bytes 1376730600 (1.2 GiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 3120877 bytes 1376730600 (1.2 GiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0administrator-host:hxkube$

Step 5 Ensure there is no network configuration file (/etc/sysconfig/network-scripts/ifcfg-<interface_name>) for theadditional virtual machine network interface that was added for the HyperFlex Storage Integration for OpenShift.

Scope:


Command:sudo rm /etc/sysconfig/network-scripts/ifcfg-<interface_name>

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo rm /etc/sysconfig/network-scripts/ifcfg-ens224; \done


Step 6 Restart the Network Manager service.

Scope:


Command:sudo systemctl restart NetworkManager.service



Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo systemctl restart NetworkManager.service; \done

Step 7 Restart the Atomic OpenShift Node service.

Scope:


Command:sudo systemctl restart atomic-openshift-node.service

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo systemctl restart atomic-openshift-node.service; \done

Step 8 Install the hxvolume-plugin RPM package to update the existing HyperFlex FlexVolume plug-in on each OpenShiftcluster node.

Scope:


Command:sudo rpm -i ~/hxkube/hxvolume-plugin_X.X.XXX.XXX.XXXXXXXX.XXXX.rpm

Example:administrator-host:~/hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo rpm -i ~/hxkube/hxvolume-plugin-1.0.284.git.4022e8ec.hx35-1.x86_64.rpm; \done


Step 9 Edit the /etc/kubernetes/hxflexvolume.json configuration file to change the target IP address from169.254.1.1 to 169.254.254.1.

Command:sudo sed -i -e s/169.254.1.1/169.254.254.1/ /etc/kubernetes/hxflexvolume.json.example

Example:administrator-host:~/hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \

do ssh -n $host sudo sed -i -e s/169.254.1.1/169.254.254.1//etc/kubernetes/hxflexvolume.json.example; \

done

Step 10 Create the HyperFlex FlexVolume Plugin configuration file. The file will be copied from an example file found in the/etc/kubernetes/ directory.

Scope:


Command:sudo cp /etc/kubernetes/hxflexvolume.json.example /etc/kubernetes/hxflexvolume.json



Example:administrator-host:~/hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo cp /etc/kubernetes/hxflexvolume.json.example /etc/kubernetes/hxflexvolume.json;\done


Step 11 Edit the /etc/kubernetes/hxflexvolume.json configuration file and update with the appropriate name forthe interface to be used for the HyperFlex Storage Integration for OpenShift. In the example in this document theinterface is named ens224.

Scope:


Command:sudo sed -i -e s/ens[[:digit:]+]/<interface_name>/ /etc/kubernetes/hxflexvolume.json

Example:administrator-host:hxkube$cat ~/hxkube/ocp_nodes.txt | while read host; \dossh -n $host sudo sed -i -e s/ens[[:digit:]+]/ens224/ /etc/kubernetes/hxflexvolume.json;\done


Step 12 Initialize the FlexVolume plugin.

Scope:


Command:sudo /usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume/hxvolume init

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo /usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume/hxvolumeinit; \done

{"capabilities":{"attach":false},"status":"Success"}{"capabilities":{"attach":false},"status":"Success"}{"capabilities":{"attach":false},"status":"Success"}{"capabilities":{"attach":false},"status":"Success"}{"capabilities":{"attach":false},"status":"Success"}administrator-host:hxkube$

Step 13 Restart the Atomic Openshift Node service.

Scope:



Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo systemctl restart atomic-openshift-node.service; \



done


Checking HyperFlex FlexVolume Plug-in Version

The following steps must be performed on one OpenShift node only.Important

Step 1 Log into one of the OpenShift cluster nodes using SSH.Step 2 Change directories to the

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex-hxvolume/ directory.

Scope:


Commandcd /usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume/

Example:ocpuser@openshift-master:~$ cd /usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume/ocpuser@openshift-master:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume$

Step 3 Run the hxvolume version command as the root user (with sudo) to view the HyperFlex FlexVolume plug-in version.

Scope:


Command:sudo hxvolume version

Example:ocpuser@openshift-master:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume$sudo ./hxvolume versionhxvolume version: 1.0.284.git.4022e8ec.hx35ocpuser@openshift-master:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hyperflex~hxvolume$

Updating HyperFlex FlexVolume Plug-inThe following steps must be performed from the administrator host.

Step 1 Install the hxvolume-pluginDebian package to update the existing HyperFlex FlexVolume plug-in on each OpenShiftcluster node.

Scope:


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformChecking HyperFlex FlexVolume Plug-in Version


Command:sudo rpm -i ~/hxkube/hxvolume-plugin_X.X.XXX.XXX.XXXXXXXX.XXXX.rpm

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo rpm -i ~/hxkube/hxvolume-plugin-1.0.284.git.4022e8ec.hx35-1.x86_64.rpm; \done


Step 2 Restart the Atomic OpenShift Node service.

Scope:



Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo systemctl restart atomic-openshift-node.service; \done


Managing HyperFlex FlexVolume Provisioner

Installing HyperFlex FlexVolume Provisioner

Step 1 Load the HyperFlex FlexVolume Provisioner docker image on each OpenShift cluster node.

Scope:


Commandsudo docker image load –-input ~/hxkube/hx-provisioner.tar.gz

Exampleadministrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo docker image load -i ~/hxkube/hx-provisioner.tar.gz; \done

Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35administrator-host:hxkube$


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformManaging HyperFlex FlexVolume Provisioner

Step 2 Run thehx-provisioner-setup script to generate the required YAML file for deploying the HyperFlex Provisionerpod on the OpenShift cluster. Provide the following information as parameters when running thehx-provisioner-setup script.

Parameters:

• -cluster-name—Name of the OpenShift cluster (must be unique across the HyperFlex cluster).

• -url—URL to reach the HyperFlex API. This URL is equivalent to thehttps://<hyperFlex_cluster_management_IP_address.

• -username—The username that is used to authenticate to the HyperFlex cluster. Typically, a vCenter SSO accountsuch as [email protected].

• -password—Name of the resulting output file generated by the hx-provisioner-setup script.

Scope:

Run on administrator host.

Commands:hx-provisioner-setup -cluster-name <ocp_cluster_name> -urlhttps://<hx_cluster_mgmt_ip> -username<hx_cluster_username> -output <output_file>.yml

Example:administrator-host:hxkube$ ~/hxkube/hx-provisioner-setup -cluster-name tc1 -url https://172.0.13.32-username [email protected] -output tc1.yml

password for [[email protected]] at [https://172.0.13.32]: <password>wrote config to tc1.yml


Step 3 Ensure the output file was created successfully. In this example, the output file was named tcl.yml.

Command:ls ~/hxkube

Example:administrator-host:hxkube$ ls -l ~/hxkube

total 76376-rwxr-xr-x. 1 root root 371 Jan 1 2008 hxkube-collect-logs-rw-r--r--. 1 root root 39078464 Nov 6 16:05 HX-Kubernetes-1.0.284.git.4022e8ec.hx35.zip-rw-r--r--. 1 root root 2374 Jan 1 2008 hxprovisioner-deploy.yaml-rwxr-xr-x. 1 root root 8193670 Jan 1 2008 hx-provisioner-setup-rw-r--r--. 1 root root 7024699 Jan 1 2008 hx-provisioner.tar.gz-rw-------. 1 root root 2356 Nov 16 13:26 hx-provisioner.yaml-rwxr-xr-x. 1 root root 4540037 Jan 1 2008 hxvolume-rw-r--r--. 1 root root 8807897 Jan 1 2008 hxvolume-plugin-1.0.284.git.4022e8ec.hx35-1.x86_64.rpm-rw-r--r--. 1 root root 8726200 Jan 1 2008 hxvolume-plugin_1.0.284.git.4022e8ec.hx35_amd64.deb-rw-r--r--. 1 root root 1780928 Jan 1 2008 istgttool-rw-r--r--. 1 root root 53 Nov 18 21:57 ocp_nodes.txt-rw-r--r--. 1 root root 1146 Jan 1 2008 RELEASE.txt-rw-------. 1 root root 2356 Nov 27 11:31 tc1.ymladministrator-host:hxkube$

Step 4 Run the oc create -f <file> command to deploy the HyperFlex Provisioner pod.


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformInstalling HyperFlex FlexVolume Provisioner

This procedure schedules the HyperFlex Provisioner pod to run on one of the OpenShift cluster nodes nodes.While there is no technical reason for it, if you require to run the HyperFlex Provisioner pod on a specificOpenShift cluster node (perhaps the infra nodes or even the master node), edit the <output> file created bythe hx-provisioner-setup script to include a Toleration to overcome to include a Toleration toovercome any configured taints. For more information seehttps://kubernetes.io/docs/concepts/configuration/taint-and-toleration/.

Note

Scope:


Command:oc create -f ~/hxkube/<file>

Example:administrator-host:hxkube$ oc create -f ~/hxkube/tc1.yml

secret "hxprovisioner" createdconfigmap "hxprovisioner-config" createdserviceaccount "hxprovisioner" createdclusterrolebinding.rbac.authorization.k8s.io "hxprovisioner-binding" createddeployment.apps "hx-provisioner" createdadministrator-host:hxkube$

Step 5 Verify that the HyperFlex Provisioner pod is running.

Scope:


Command:oc get pods -n kube-system

Example:administrator-host:hxkube$ oc get pods -n kube-system

NAME READY STATUS RESTARTS AGEhx-provisioner-cdf64fc5f-ghvrw 1/1 Running 0 10smaster-api-ocp-master 1/1 Running 0 18hmaster-controllers-ocp-master 1/1 Running 0 18hmaster-etcd-ocp-master 1/1 Running 0 18hadministrator-host:hxkube$

Checking HyperFlex FlexVolume Provisioner Version

Step 1 Run et pods -n kube-system command to get the complete name of the deployed HyperFlex FlexVolume Provisionerpod.

Scope:


Command:

oc get pods -n kube-system


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformChecking HyperFlex FlexVolume Provisioner Version

https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

Example:administrator-host:hxkube$ oc get pods -n kube-system

NAME READY STATUS RESTARTS AGEhx-provisioner-cdf64fc5f-ghvrw 1/1 Running 0 <invalid>master-api-ocp-master 1/1 Running 0 18hmaster-controllers-ocp-master 1/1 Running 0 18hmaster-etcd-ocp-master 1/1 Running 0 18hadministrator-host:hxkube$

Step 2 Run describe pods command to get the complete details of the deployed HyperFlex FlexVolume Provisioner pod. Lookfor the hx-provisioner container image name which includes the version as a tag (that is, after the colon in the containername).

Scope:


Command:

oc describe pods <pod_name> -n kube-system

Example:administrator-host:hxkube$ oc describe pods hx-provisioner-cdf64fc5f-ghvrw -n kube-system

Name: hx-provisioner-cdf64fc5f-ghvrwNamespace: kube-systemNode: ocp-node2/10.2.17.154Start Time: Tue, 27 Nov 2018 11:43:55 -0500Labels: app=hx-provisioner

pod-template-hash=789209719Annotations: <none>Status: RunningIP: 10.128.2.2Controlled By: ReplicaSet/hx-provisioner-cdf64fc5fContainers:hx-provisioner:Container ID: docker://b5cf5c79a9fec30221ec1f27c09fe51c138a371e474854577f1743baad343f45Image: hx-provisioner:1.0.284.git.4022e8ec.hx35Image ID: docker://sha256:e275592fd478b06b4e60c40eb42fc2321a15546d6e4eb9a8b7aee818073852b0Port: 443/TCPHost Port: 0/TCPArgs:-hxapi-url=$(HX_API_URL)-hxapi-token-file=/secrets/hxapi/token-hxapi-hxclusteruuid=$(HX_CLUSTERUUID)

State: Runningadministrator-host:hxkube$

Updating HyperFlex FlexVolume Provisioner

Step 1 Download the latest HX Kubernetes release package from Cisco Software Downloads.Step 2 Follow the steps in Distributing HyperFlex FlexVolume Software to copy the latest HX Kubernetes release package to

each OpenShift node. Unzip the file on each OpenShift node.

Scope:


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformUpdating HyperFlex FlexVolume Provisioner



Command:sudo docker image load –-input ~/hxkube/hx-provisioner.tar.gz

Example:administrator-host:hxkube$ cat ~/hxkube/ocp_nodes.txt | while read host; \do ssh -n $host sudo docker image load -i ~/hxkube/hx-provisioner.tar.gz; \done

Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35administrator-host:hxkube$

Step 3 Note the full name of the newly loaded Docker image from the output of the previous command.

Example:Loaded image: hx-provisioner:1.0.284.git.4022e8ec.hx35

Step 4 Run the oc set image command to update the HyperFlex FlexVolume Provisioner container deployment to use theupdated container image.

Scope:


Command:oc set image pod/<pod_name> hx-provisioner=<new_image_name>

Example:administrator-host:hxkube$ oc set image pod/hx-provisioner-cdf64fc5f-ghvrwhx-provisioner=hx-provisioner:1.0.284.git.4022e8ec.hx35 -n kube-system


Configuring Storage Classes

Step 1 Create a file called hx-storageclass.yml in the ~/hxkube directory

Scope:


Command:sudo touch ~/hxkube/hx-storageclass.yml

Example:administrator-host:hxkube$ touch ~/hxkube/hx-storageclass.yml



Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformConfiguring Storage Classes

Step 2 Edit the hx-storageclass.yml file using your choice of file editor (for example, vi.) and insert the following text.Be sure to copy the text exactly as shown below, including indentations. Save the file once complete.kind: StorageClassapiVersion: storage.k8s.io/v1metadata:name: hyperflexannotations:storageclass.kubernetes.io/is-default-class: "true"

provisioner: hyperflex.io/hxvolume

If you do not wish for the hyperflex storage class to be the default storage class, you can remove the followingtwo lines from the above hx-storageclass.yml file before creating the storage class:annotations:

storageclass.kubernetes.io/is-default-class: "true"

Note

Step 3 Use the oc create -f command to create the hyperflex storage class in the OpenShift cluster.

Scope:


Command:oc create -f <file>

Example:administrator-host:hxkube$ oc create -f ./hx-storageclass.yml

storageclass.storage.k8s.io "hyperflex" createdadministrator-host:hxkube$

Step 4 Use the get sc command to view the new hyperflex storage classes on the OpenShift cluster.

Scope:


Command:oc get sc

Example:

administrator-host:hxkube$ oc get sc

NAME PROVISIONER AGEhyperflex (default) hyperflex.io/hxvolume 23sadministrator-host:hxkube$

Provisioning Persistent VolumesAt this point, the HyperFlex Storage Integration for OpenShift has been fully deployed and can now beleveraged to provide persistent storage to OpenShift workloads. Developers and users can now simply submitPersistent Volume Claim requests, and if the hyperflex storage class is configured as the default storage class,the requested storage will be automatically provisioned by HyperFlex and provided to the OpenShift


Configuring HyperFlex FlexVolume Storage Integration for RedHat OpenShift Container PlatformProvisioning Persistent Volumes

environment. This results in a new Persistent Volume Claim bound to a new Persistent Volume object as well.The Persistent Volume Claim can then be used when deploying workloads within OpenShift.

When creating a Persistent Volume Claim request, the storageClassName:hyperflexline is required only ifyou decide not to create the hyperflex storage class as the default storage class. If the hyperflex storage classis the default storage class, you can remove that line from any Persistent Volume Claim requests.

Note

The following steps provide a sample workflow of deploying a simple “Cisco Message Board” applicationusing persistent storage from HyperFlex.

Step 1 Create a new project (namespace) for the sample application.

Scope:


Command:oc create namespace <name>

Example:administrator-host:hxkube$ oc create namespace message-board

namespace "message-board" createdadministrator-host:hxkube$

Step 2 Create a file named pvc.yaml in the ~/hxkube directory.

Scope:


Command:touch ~/hxkube/pvc.yml

Example:administrator-host:hxkube$ touch ~/hxkube/pvc.yml


Step 3 Edit the pvc.yaml file and insert the following text. Save the file once complete.

Example:apiVersion: v1kind: PersistentVolumeClaim metadata:name: message-board-pvcnamespace: message-boardspec:storageClassName: hyperflexaccessModes:

-ReadWriteOnceresources:requests:

storage: 100Gi



Step 4 Run the oc create <file> command to submit the pvc.yaml file and create the Persistent Volume Claim object inthe OpenShift cluster. In parallel, as part of the operation, HyperFlex will create a Persistent Volume object to complementthe Persistent Volume Claim object and bind the two together in OpenShift.

Scope:


Command:oc create -f ~/hxkube/pvc.yaml

Example:aadministrator-host:hxkube$ oc create -f ~/hxkube/pvc.yml

persistentvolumeclaim "message-board-pvc" createdadministrator-host:hxkube$

Step 5 Check the status of the Persistent Volume Claim object with the oc get pvc command to make sure it was createdsuccessfully and is “Bound” to a Persistent Volume object.

Scope:


Command:oc get pvc

Example:administrator-host:hxkube$ oc get pvc

NAME STATUS VOLUMECAPACITY ACCESS MODES STORAGECLASS AGEmessage-board-pvc Bound hx-default-message-board-pvc-c54defc5-f26b-11e8-8aff-00505698adb3100Gi RWO,ROX hyperflex <invalid>administrator-host:hxkube$

Step 6 Now that the Persistent Volume Claim and the supporting Persistent Volume (from HyperFlex) have been successfullycreated, you can deploy the Message Board Pod. Start the process by creating a file called message-board.yml in the~/hxkube directory.

Scope:


Command:touch ~/hxkube/message-board.yml

Example:administrator-host:hxkube$ touch ~/hxkube/message-board.yml


Step 7 Edit the message-board.yml file and insert the following text. Save the file once complete.

For purposes of this example, the Pod definition also includes a simple Service definition so the applicationcan be reached outside the OpenShift cluster.

Note

The Persistent Volume Claim message-board-pvc is referenced in the Pod definition.Note



Example:apiVersion: v1kind: Podmetadata:

name: message-boardlabels:

app: message-boardname: message-board

namespace: message-boardspec:

containers:- name: message-boardimage: michzimm/message_board:version1volumeMounts:- name: demovolume1mountPath: /sqldb

ports:- containerPort: 5000

volumes:- name: demovolume1persistentVolumeClaim:claimName: message-board-pvc

---apiVersion: v1kind: Servicemetadata:name: message-boardlabels:name: message-board

namespace: defaultspec:type: NodePortports:- port: 5000nodePort: 30002

selector:name: message-board

Step 8 By default, OpenShift runs containers as the default user within a project (namespace). In order to successfully deploythe Message Board Pod, it is required that you add the privileged security context constraint (scc) to the default userin the message-board project (namespace).

This step is not recommended in production as this presents a security risk.Important

Scope:


Command:oc adm policy add-scc-to-user privileged -z default -n <namespace>

Example:administrator-host:hxkube$ oc adm policy add-scc-to-user privileged -z default -n message-board

scc "privileged" added to: ["system:serviceaccount:message-board:default"]administrator-host:hxkube$

Step 9 Use the oc create -f <file> command to deploy the Message Board Pod definition.

Scope:




Command:oc create -f <file>

Example:administrator-host:hxkube$ oc create -f ~/hxkube/message-board.yml

pod "message-board" createdservice "message-board" createdadministrator-host:hxkube$

Step 10 Use the oc get pods command to check the status of the deployed Pod and ensure it is running.

Scope:


Command:oc get pods -n <namespace>

Example:administrator-host:hxkube$ oc get pods -n message-board

NAME READY STATUS RESTARTS AGEmessage-board 1/1 Running 0 2madministrator-host:hxkube$



Date post:	30-Dec-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Cisco HyperFlex Systems Administration Guide for Kubernetes, … · calico-node-6mc7b 2/2 Running 0...

Documents