NetApp OnCommand Workflow Automation – Useful Workflows … · 2018-09-02 · NetApp® OnCommand...

NetApp® OnCommand® Workflow Automation – Useful Workflows Cookbook Silverton Consulting, Inc. StorInt™ Briefing

Revision 1.0 – June 2014

NETAPP ONCOMMAND WFA – USEFUL WORKFLOWS COOKBOOK PAGE 2 OF 13 REVISION 1.0 – JUNE 2014

TWITTER.COM/RAYLUCCHESI | RAYONSTORAGE.COM | +1-‐720-‐221-‐7270 | SILVERTONCONSULTING.COM

© 2014 SILVERTON CONSULTING, INC. ALL RIGHTS RESERVED

Introduction Storage administration has changed significantly over the years. First there were storage system op-‐panels where administrators had to sit or stand at a machine typing commands and pushing buttons on the storage array. Then command line interfaces (CLIs) came along where storage administrators could stay at their desks and define aggregates, volumes and file systems/logical unit numbers (LUNs) by typing requests at command prompts. Next, web-‐based GUIs allowed administrators to define and update storage objects by selecting from multiple screen or dropdown menu options. Recently, Representational State Transfer (REST) interfaces and APIs have emerged that can be used to perform all of these tasks in an automated fashion. Then a little while back, NetApp’s OnCommand software suite offered an entirely new capability called Workflow Automation (WFA), which provides an almost visual, more intuitive, web-‐based approach to automating storage administration tasks without having to use to RESTful APIs. NetApp® OnCommand Workflow Automation® overview Workflow Automation (WFA) is an OnCommand product for creating consistent, reliable storage services. Workflow Automation supplies NetApp storage administrative and operational commands together with a development environment that can be used to define new combinations of storage commands and other functionality. It also provides an environment for executing a string of these storage activities while prompting WFA users to supply mandatory or optional parameters for these services. Essentially, OnCommand Workflow Automation consists of three separate environments: the WFA Designer Portal used by administrators to design, develop and debug new workflows; the WFA Execution Portal used by authorized users to execute workflows and interact with workflows to supply mandatory and optional parameters; and the WFA Administration Portal used to monitor workflow execution and establish connections between WFA workflows and external databases,

automation frameworks and other data sources, as well as to establish user authorization to access and use WFA workflows. In addition, a web services interface can be used to invoke workflows from external portals and other data center orchestration solutions. OnCommand Workflow Automation makes these encapsulated services or scripts available to any authorized user. When an authorized user logs




into the WFA Execution Portal, that user is presented with a matrix of workflows (sort of like a screen full of apps on a smart phone) that the user is authorized to execute. In essence, users and other administrators can be given access to workflows to potentially define their own file systems/LUNs, volumes and aggregates. The options used for these storage entities are specified by the workflow in combination with selected user input. The WFA workflow does all the hard work to define the storage and link it up with the user’s application environment. To execute a storage service request, users only need to click on a workflow and answer a few questions. OnCommand Workflow Automation is a powerful tool that enables users to define, alter and manage NetApp storage without ever having to interact with storage or server administrators. Many storage vendors are hoping to make this level of automation directly available to authorized users by providing RESTful interfaces. However, not all data centers have the needed expertise to code to storage system REST APIs. In contrast, NetApp OnCommand Workflow Automation addresses this challenge by hiding much of the complexity of storage administration. All users or administrators need to do is to fill in the storage names, networking addresses, mandatory storage attributes, etc., that apply to their data center environment in order to execute automated delivery of storage services. Equally important, Workflow Automation administrators specify information about the NetApp storage configuration, the virtualization environment and linkages to applications and orchestration external functionality. OnCommand Unified Manager can be used to supply some of this information automatically. Other storage, servers and application configuration information can be imported into OnCommand Workflow Automation using standardized WFA facilities, and linkages to external software functionality can be established. WFA workflows can then take advantage of all of these objects, entities and external capabilities to encapsulate storage administrative activities into a standard set of scripted actions that almost anyone can use.

Automation of storage service delivery These days’ data centers are moving applications and other functionality to the cloud at a rapidly increasing pace. Software as a Service (SaaS), Storage as a Service (STaaS), Infrastructure as a Service (IaaS), etc., are becoming ever easier and cheaper to use. But there are some important considerations to moving IT services to the cloud, especially for larger enterprises and can include increased security exposure, the increased possibility of missing SLAs and potential increases in IT expenses. Some but not all of these concerns can be mitigated with appropriate due diligence and additional contractual constraints. In contrast, NetApp OnCommand Workflow Automation provides many of the benefits of cloud services without the challenges associated with using Anything-‐as-‐a-‐Service (XaaS) solutions. WFA can supply cloud-‐like ease of use inside the data center. With WFA, users can define their own storage by simply clicking on a workflow, and the rest takes place behind the scenes, as with cloud




services. Thus, WFA can supply the ease of use of cloud services while taking advantage of the existing investments in security controls, expense controls and high-‐availability capabilities already present in your current data center. Furthermore, WFA can turn storage into a self-‐service environment by making storage services directly available to authorized users and not having to rely on storage administrators. Self-‐service customers can immediately use workflows to define new storage, retire old storage or even migrate storage from one storage system to another. WFA also makes it much easier to standardize storage processes. Naming conventions and other storage standards can be built into workflows, making them more easily enforced or changed as needed. For example, if you use non-‐deduplicated storage and later decide that deduplication is needed, then a simple change to a workflow template or two can make all new storage provisioning use deduplication. Finally, encapsulation of storage definition, retirement, migration, etc., into workflows enables these activities to automatically interface with configuration management database (CMDB) functionality and automation frameworks from Microsoft, VMware and others. WFA provides some packages with a built-‐in capability to integrate workflows with these external services. In the past, administrators had to log in to other CMDB services or automation frameworks or manually modify Excel spreadsheets, which meant the potential for error was significant. But with WFA, changes to NetApp storage automatically update external services as well, keeping functionality and frameworks current and connected.

OnCommand Workflow Automation workflows After describing some of what can be accomplished with workflows above, we will next discuss what a workflow consists of, where workflow information can be found, what workflow variables can be used and some of the other entities which can be manipulated by workflows. Accordingly, workflows are a scripted series of execution steps that perform various storage configuration, CMDB or external functionality using commands or primitives supplied by NetApp, supplied by the WFA web community or developed by the data center. These scripts can combine multiple workflow primitives with various control options. For example, this capability can be used to select which primitive executes next and indicate how many times each primitive should be executed. Workflow primitives can supply default options for storage configuration activities or ask users to specify options before they can proceed. Once defined, workflows are provided to different users or administrators in the data center through the WFA Execution portal. For example, application owners could be authorized to use workflows that define file systems, LUNs or other storage entities. Application or server administrators could be given access to these and other workflows that allow them to link applications to storage, create aggregates or decommission storage. Storage administrators could




be given access to all of these workflows and others that allow them to define new storage clusters. Workflow administrators, in turn, can create new workflows, link workflows to orchestration packages or authorize customers and administrators to use workflows. Within the WFA Designer portal discussed previously, administrators can create their own execution primitives to retrieve data from external information resources, externally validate some information, execute external scripts or combine one or more primitives into one workflow. Workflows depend on primitives or commands that can execute storage commands. These can include NetApp CLI commands such as creating Network File System (NFS) exports, adding quality of service (QoS) services to volumes and unprovisioning aggregates to free storage. NetApp WFA uses Microsoft Windows® PowerShell or Perl® Programming Language for workflow execution. In addition, workflows can retrieve or output information to data sources or databases. As discussed previously, OnCommand Unified Manager provides one important data source for most workflows. Unified Manager auto-‐discovers NetApp storage and can import this configuration information directly into a WFA data source. Once available in a data source, workflows can readily reference or access this information. Accordingly, entities or other objects manipulated by or used in workflow commands can be script variables created by the script itself or search objects where WFA searches for information in its data sources. These objects or entities can be passed from one command to another and consequently referenced throughout the execution of a workflow. In contrast to data sources that supply internal information, WFA schemes are used to define external environment attributes. For instance, a “vc” scheme would be used to describe a virtual machine environment, such as virtual machines, hosts and data stores. Templates are used as a blueprint for object definition in order to specify the types of storage efficiency characteristics used for aggregate creation. Finally, WFA categories are used to assign user groups to authorized workflows on the WFA Execution portal. Furthermore, WFA commands can also take advantage of powerful tools such as regular expressions, filters and finders to extract or manipulate information from data sources, schemes, or templates. Regular expressions can be used to help define and validate naming conventions or any other workflow text fields. Filters are SQL-‐based queries to WFA databases or data sources that can return lists of objects or entities that satisfy some particular selection criteria for further workflow command processing. Finders are combinations of one or more filters that can be used to select the one entity or object that will be used for follow-‐on workflow execution. WFA can take advantage of specialized functions written in MVFLEX Expression Language (MVEL) in order to reuse logic and functionality that has already been developed. Workflows can also pause while executing external Powershell or Perl scripts, which can be especially useful when performing functions needed to connect storage to other applications.




To modify existing workflows or create new ones, it helps to have a general familiarity with NetApp storage CLI and with WFA schemes, data sources and execution control capabilities. The OnCommand Workflow Automation 2.2 Developers Guide1 provides more information about how to develop workflows. Comprehensive storage service workflows that have been previously developed are generally supplied as packages, which can contain documentation, data sources, schemes, templates and one or more workflows or external scripts. These workflow packages are available from the NetApp WFA Community website,2 which will be discussed later.

OnCommand Workflow Automation use cases To give the user more of an understanding for what can be accomplished with Workflow Automation we will discuss some typical use cases of OnCommand WFA. There are more than 50 out-‐of-‐the-‐box WFA workflows available from NetApp, including the following:

• Create clustered Data ONTAP® (cDOT) SnapMirror™ relationships, NFS volume(s), Qtree Common Internet File System (CIFS) share(s), volume with QoS policy groups, VMware data stores, and a basic volume;

• Move or remove a cDOT volume; and • Create and configure a NAS Storage Virtual Machine or a storage area network (SAN)

Storage Virtual Machine. Similar sets of workflows exist for 7-‐mode storage, along with specific workflows for SnapVault®, SnapMirror and Infinite Volume support. All of these workflows come standard with OnCommand WFA. We have selected four specific WFA workflow packages that have been developed by subject matter experts or NetApp personnel to be described in detail because of their broad applicability and as good examples of what can be done with workflows. The four workflows include:

• Workflows for clustered Data ONTAP® (cDOT) transition • Workflows for orchestrating cDOT for Oracle® • Workflows for Storage-‐as-‐a-‐Service (STaaS) and private cloud • Workflows for Software Defined Data Center (SDDC)

1 Documentation available at Support.NetApp.com 2 Available at https://communities.netapp.com/community/products-‐and-‐solutions/data-‐storage-‐software/storage-‐management-‐software/oncommand-‐workflow-‐automation.




Workflows for cDOT transitions This is an advanced package of workflows for use by NetApp services and partners that helps support the transition from 7-‐mode to clustered-‐mode LUNs. One workflow moves an offline 7-‐mode volume LUN to clustered-‐mode. Another group of workflows migrates an online 7-‐mode volume LUN, with a short offline outage at the end to reinstate the LUN in clustered-‐mode. The overall processing is similar in either case. These workflows use SnapMirror and a 7-‐mode staging array for the migration. The two essential workflows for an online migration of a 7-‐mode LUN to cDOT are as follows:

1. 7toC LUN Migrate Baseline workflow takes the 7-‐mode source volume LUN and establishes a Qtree SnapMirror (QSM) relationship to another 7-‐mode staging volume LUN. It also creates the clustered-‐mode target volume and LUN and establishes a Transition Data Protection (TDP) SnapMirror relationship between the staging volume and the clustered-‐mode volume. Finally, it converts the 7-‐mode LUN to files, then copies and converts the files to the clustered-‐mode LUN.

2. 7toC LUN Migrate Update + Cutover workflow completes the migration process while the 7-‐mode source LUN is offline to the host by resyncing the QSM relationship between the 7-‐mode source volume LUN and the 7-‐mode staging volume LUN, breaking the QSM relationship, creating the file version of the 7-‐mode LUN, updating the TDP version of the file copy of the LUN on the clustered-‐mode staging volume, breaking the TDP relationship, deleting the 7-‐mode staging LUN and cloning the 7-‐mode staging LUN from the file copy version to the clustered-‐mode target LUN. After completing this workflow, customers can bring the new clustered-‐mode LUN online to the host.

Other optional workflows can be used to just perform updates of the QSM to TDP cascade (7toC LUN Migrate Update); just resync the TDP SnapMirrors (7toC LUN Migrate Resync); or just clean up the staging volumes, QSM and TDP relationships (7toC LUN Migrate Cleanup). These three workflows could be used to test the transition process before finally performing the cutover. There is also a “rapid” version of the cutover workflow (7toC LUN Migrate Update + Cutover – Rapid) that performs the final cutover faster, though it does not preserve the geometry of the expanded LUNs and has a few other restrictions. The offline 7toC LUN Offline Migrate workflow performs all of the above into one workflow. Most of the workflow inputs provide information for the source array volume/LUN, the staging array volume/LUN and the target clustered-‐mode volume/LUN. Some optional selections enable users to preserve LUN geometry, preserve LUN serial number, Remap LUN, use Vfiler QSM, etc. For more information, download the 7toC Migrate package from the WFA Community website.3

3 Available at https://communities.netapp.com/ Under documents: Direct 7mode and cDOT Script Data Sources.




Workflows for orchestrating cDOT for Oracle® application environments This Pirate Pack for Oracle Applications package of workflows helps administrators and Oracle application owners create and remove new Oracle application storage environments using NetApp clustered storage. To run a “gold” Oracle database environment, CONTROL, TEMP, DATA, REDO and ARCHIVE storage volumes need to be created, along with a VMware® vSphere® vApp™. All of these volumes can be created, cloned or removed using the workflows in this package. The package contains four workflows: Create a new Oracle gold environment – vCloud® clustered ONTAP; Add Snap Creator™ backup capability to Oracle Volumes; Clone and configure Oracle environment – vCloud clustered ONTAP; and Remove application volumes and empty storage virtual machine (SVM), formerly vServer. All four workflows assume the use of VMware vCloud and clustered storage. It may make sense to examine the process involved in creating a workflow for this package. For example, the steps in the “create a new Oracle gold environment” workflow are as follows:

1. Create vApp – Using operator input and the predefined vApp template, the workflow creates an Oracle vApp that will be used to connect to the storage environment.

2. Connect vApp to the network – Using the information supplied, the workflow connects the vApp to data center networking for further processing.

3. Start vApp – The vApp begins execution so that the rest of the workflow can execute with it in operation.

4. Create and configure SVM – A new NetApp SVM is created to host the application storage. An NFS export is also created to provide external access to the storage, which will be linked to the vApp.

5. Create volumes – All of the volumes for the Oracle environment are created in the SVM. 6. Create logical interfaces – Logical interfaces are created for data and management host

access to the SVM. 7. Set up DNS entries – DNS entries are established for the new vApp. 8. Configure vApp via a custom script – A custom Perl script is executed to configure vApp

access to storage and mount the new storage to the host. 9. Restart vApp – The vApp is restarted to bring up the new Oracle environment.

At this point, the new gold Oracle environment has been created and is ready for use. Most of the workflows have similar input, which identifies the NetApp storage cluster and storage SVM to be used; provides storage networking configuration information; identifies the Oracle application environment; defines the size of the TMP, BIN, DATA, ARCHIVE and REDO volumes; and defines supply counts for the DATA, ARCHIVE and REDO volumes. Once all of this information is supplied or selected using templates and other data sources, the workflows are ready to execute.




The Pirate Pack for Oracle Applications package can be downloaded from the WFA Community website.4 A video showing more detailed workflows in operation is also available from NetApp.5 Workflows for STaaS and private cloud STaaS has many meanings, but in general it attempts to bring the ease of installation, deployment and use of cloud services to storage administration. The STaaS package supplies a set of standard workflows that anyone can use to automate NetApp storage services for multiple tiers of service. For example:

• Gold storage services would be assigned to high-‐end storage systems (FAS6290, FAS8060 or larger), use solid state drive (SSD) only or Flash Pool™ aggregates, be locally Snapshot® every hour, be remotely SnapMirrored every hour, be thickly provisioned and have no deduplication or compression.

• Silver storage services would be assigned to mid-‐range storage systems (FAS3250, FAS8020 or other mid-‐tier storage), use SAS disk storage, be locally Snapshot every two hours, be remotely SnapMirrored every four hours, be thickly provisioned and be deduplicated but not compressed.

• Bronze storage services would be assigned to lower-‐end storage systems (FAS32xx, FAS22xx, Data ONTAP Edge or other SMB storage), use SAS or SATA disk, be locally Snapshot every four hours, be remotely SnapMirrored once per day, be thinly provisioned and be deduplicated but not compressed.

The STaaS package makes use of the MySQL Scheme table that relates data center storage domains to physical storage components both locally and remotely and defines authorized consumers of those services. After the STaaS scheme table structures are restored (copied) into the WFA Designer environment, the Storage Service Catalog package workflows can be imported. These workflows can provision/unprovision NFS exports, CIFS shares and Fibre Channel/Internet Small Computer System Interface (FC/iSCSI) LUNs, as well as create and remove service catalog objects such as consumers, storage domains and aggregates. The package contains 26 distinct workflows that do the following:

• Provision or Unprovision NFS Exports, CIFS shares and LUNs (6 workflows); • Create or Destroy (Snapshot/SnapMirror) schedules, storage domains, provisioning

policies, local protection policies, remote protection policies, storage services and consumers (14 workflows); and

• View consumers, provisioning policies, storage domains, storage services and storage objects by consumer (5 workflows).

4 Available at https://communities.netapp.com/docs/DOC-‐30718. 5 Available at https://communities.netapp.com/videos/3536.




Another workflow is provided to create a set of predefined protection schedules that can be used as defaults or used to create new schedules. A workflow administrator/designer can use the above service catalog workflows to construct a STaaS environment by creating storage domains, one or more provisioning policies, schedules, local and remote protection policies, storage services and consumers. Consumers or storage administrators can then use the additional workflows to provision or unprovision their storage services.

Most of the workflow inputs are contained in the MySQL data source described above. The workflows discussed here are documented in a NetApp OnCommand Workflow Automation Workflows for the Storage as a Service Technical Report.6 The Storage Service Catalog package of workflows can be downloaded from the WFA Community website.7 The package contains workflows, sample MySQL data source schemes, external Perl scripts and more, which can be used to construct the STaaS. Workflows for Software Defined Data Center (SDDC) The OnCommand WFA package for VMware vCenter Orchestrator can be used to help operate the SDDC. This package will take advantage of VMware vCloud Suite components, vCloud Automation Center (vCAC) and vCenter Orchestrator (vCO), NetApp WFA and NetApp OnCommand Unified Manager. This package integrates vCAC with NetApp WFA so that vCloud administrators can use NetApp workflows directly to provision, migrate and decommission storage; set up new virtualization environments; and set up storage for new applications.

6 Available at http://www.netapp.com/au/system/pdf-‐reader.aspx?pdfuri=tcm:28-‐120286-‐16&m=tr-‐4272.pdf. 7 Available at https://communities.netapp.com/docs/DOC-‐32623.




The workflows depend on three data sources: a vCenter data source, a vCloud data source and an OnCommand Unified Manager data source. All three data sources periodically query their services for information and, as such, require login credentials, host IP addresses and information regarding how often they should be updated. The data sources will be used to populate schemes for vc, vCloud and cm_storage. These schemes will be filtered to supply objects and other entities for the package’s workflows. After establishing the above data sources, users must provide credentials to gain WFA access to vCenter and vCloud Director and then configure vCO. vCO will be the place where vSphere infrastructure and other VMware automation processes execute and integrate directly with NetApp OnCommand WFA. vCO has its own workflows and will call OnCommand WFA to automate NetApp storage activities. vCO must support the vCO HTTP-‐REST plug-‐in that talks directly with NetApp WFA. vCAC is a centralized operational portal that enables users and administrators to request services in vCloud. vCAC will call vCO and use vCO workflows to perform orchestration tasks inside the vCloud environment. The WFA workflows described below assume vCAC is properly connected to vCO, vCenter server, VMware vCloud server and NetApp WFA. The NetApp WFA vCenter Orchestrator package consists of vCO workflows, vCO actions and configuration workflows. Essentially, these workflows link vCO to NetApp WFA so that WFA workflows can be executed and monitored to completion. This package includes the following:

• Connect and Disconnect NetApp WFA to vCO; • Verify WFA workflow inputs, Run WFA workflow, Get WFA job execution details, Wait

for WFA job to complete and Return WFA workflow outputs; and • Generate WFA e-‐mail message and Find WFA workflow ID.




The package supplies the three data sources with linkage information; supplies information to set up and connect vCAC, vCO and vCloud Director; and establishes the vCO REST connection to OnCommand WFA. Most of the other workflow input parameters come from the three data sources. These vCenter Orchestration workflows are described in more detail in a technical report on Automating and Orchestrating the SDDC-‐38, a video9 and the OnCommand WFA package for VMware vCenter Orchestrator, which includes actual workflows, data sources and other information that can be downloaded from the WFA Community website.10 NetApp OnCommand WFA Community website11 All of the above workflows and much more information can be found at the NetApp OnCommand WFA Community which provides downloads, video tutorials and documentation for all the WFA workflows described above as well as other workflows that cover integration with BMC Software, HP Operations Orchestrator and a number of other external software solutions. Further, on the Community portal NetApp and external subject matter experts can provide prompt answers to questions on such WFA topics as how to use the packages, how to develop new workflows and how to share external scripts. To learn more about WFA, examples of WFA commands, filters, finders, functions, templates and workflows please refer to the WFA Community website.

8 Available at http://www.netapp.com/us/media/tr-‐4217.pdf. 9 Available at http://www.youtube.com/watch?v=Kz8oZ16K48c. 10 Available at https://communities.netapp.com/docs/DOC-‐32111. 11 Available at https://communities.netapp.com/community/products-‐and-‐solutions/data-‐storage-‐software/storage-‐management-‐software/oncommand-‐workflow-‐automation




Summary OnCommand WFA enables storage architects, administrators and solution designers to encapsulate and script standard, everyday NetApp storage activities in a way that non-‐storage administrators can perform with the click of a button. Not all data centers may wish to design, develop and debug their own WFA workflows, but just about anyone can download the packages identified here from the WFA Community and use them to support NetApp storage activities. With WFA, NetApp has taken the next step in storage service automation by providing an almost visual scripting solution to automate these storage activities. Using WFA workflows, NetApp storage administration can be accomplished with minimal effort and storage expertise. WFA makes using NetApp storage as simple as possible for application, server and storage administrators.

Silverton Consulting, Inc., is a U.S.-based Storage, Strategy & Systems consulting firm offering products and services to the data storage community.

Date post:	09-Apr-2020
Category:	Documents
Upload:	others
View:	31 times
Download:	0 times

NetApp OnCommand Workflow Automation – Useful Workflows … · 2018-09-02 · NetApp® OnCommand...

Documents