HPE Converged Solution 750 · The HPE Converged Solution 750 described in this guide is designed to...

HPE Converged Solution 750 HPE Synergy Gen10 VMware 6.0 and 6.5 Design Guide

HPE Converged Solution 750

http://www.hpe.com/


Contents Executive summary ................................................................................................................................................................................................................................................................................................................................ 3 Introduction ................................................................................................................................................................................................................................................................................................................................................... 4 HPE Converged Solution 750 high-level design principles .............................................................................................................................................................................................................................. 4

Storage ........................................................................................................................................................................................................................................................................................................................................................ 4 Network ...................................................................................................................................................................................................................................................................................................................................................... 4 Computing ............................................................................................................................................................................................................................................................................................................................................... 5 Rack infrastructure ........................................................................................................................................................................................................................................................................................................................... 6

HPE Converged Solution 750 HPE Synergy Gen10 VMware 6.0 and 6.5 Solution overview ......................................................................................................................................... 7 Design principles .................................................................................................................................................................................................................................................................................................................................. 10

Fibre Channel based Storage Attached Networking (SAN) Topology......................................................................................................................................................................................... 10 iSCSI based Storage Attached Networking (SAN) Topology ................................................................................................................................................................................................................ 15 VMware vSAN based Storage Topology ................................................................................................................................................................................................................................................................. 16

Solution Compute / workload servers component ................................................................................................................................................................................................................................................ 18 HPE Synergy 12000 Frame, Management, and Fabric ............................................................................................................................................................................................................................. 19 HPE Synergy Gen10 Compute Module .................................................................................................................................................................................................................................................................... 23 Out of Frame Management servers – HPE ProLiant DL360 Gen10 Servers ........................................................................................................................................................................ 24

Ethernet and Storage Networking ....................................................................................................................................................................................................................................................................................... 26 Multiple redundant powered switch chassis ......................................................................................................................................................................................................................................................... 26 General Ethernet Network Topology Design ....................................................................................................................................................................................................................................................... 26 Ethernet topology when using Fibre Channel attached storage ....................................................................................................................................................................................................... 26 Fibre Channel Storage Attached Network (SAN) Topology Design .............................................................................................................................................................................................. 27 Ethernet topology when using HPE Nimble iSCSI storage ..................................................................................................................................................................................................................... 32 Ethernet topology when using VMware vSAN ................................................................................................................................................................................................................................................... 33 Compute Networking – HPE, Arista, and Cisco Nexus switches ........................................................................................................................................................................................................ 34 Cisco Nexus 9300 Series Switches ............................................................................................................................................................................................................................................................................... 38 Out of band Management Networking – HPE, Arista, and Cisco Nexus switches ............................................................................................................................................................. 38 Storage Networking – HPE StoreFabric B-Series Fibre Channel Switch .................................................................................................................................................................................... 39 Network sizing and planning .............................................................................................................................................................................................................................................................................................. 40 Verified, allowed, and required component design criteria ..................................................................................................................................................................................................................... 42

Storage .......................................................................................................................................................................................................................................................................................................................................................... 45 HPE 3PAR StoreServ Storage ........................................................................................................................................................................................................................................................................................... 45 HPE Nimble Storage ................................................................................................................................................................................................................................................................................................................... 47 HPE InfoSight for HPE Nimble Storage and 3PAR ........................................................................................................................................................................................................................................ 51 HPE Synergy D3940 Storage Module ....................................................................................................................................................................................................................................................................... 52 Verified, allowed, and required component design criteria ..................................................................................................................................................................................................................... 52

Hypervisor .................................................................................................................................................................................................................................................................................................................................................. 54 VMware vSphere 6.5 .................................................................................................................................................................................................................................................................................................................. 54


VMware vSAN 6.6 ......................................................................................................................................................................................................................................................................................................................... 55 VMware vSphere sizing and solution requirements ...................................................................................................................................................................................................................................... 56

Management software ..................................................................................................................................................................................................................................................................................................................... 59 HPE OneView ................................................................................................................................................................................................................................................................................................................................... 60 HPE OneView Global Dashboard ................................................................................................................................................................................................................................................................................... 63 VMware vCenter Server ........................................................................................................................................................................................................................................................................................................... 63 VMware vRealize Operations Manager ..................................................................................................................................................................................................................................................................... 64 HPE StoreOnce Recovery Manager Central for VMware ......................................................................................................................................................................................................................... 65

Operational Support from HPE Pointnext .................................................................................................................................................................................................................................................................... 65 HPE Proactive Care ..................................................................................................................................................................................................................................................................................................................... 65 Additional service options for HPE Converged Solution 750 .............................................................................................................................................................................................................. 66 Support considerations for customer owned hardware ............................................................................................................................................................................................................................. 66 Support considerations for Cisco switches ............................................................................................................................................................................................................................................................ 66 Support considerations for Converged Solution 750 with HPE Nimble Storage ............................................................................................................................................................... 66 HPE Connected Products ...................................................................................................................................................................................................................................................................................................... 67

Licensing considerations ............................................................................................................................................................................................................................................................................................................... 67 VMware licenses ............................................................................................................................................................................................................................................................................................................................. 67

Solution verification ........................................................................................................................................................................................................................................................................................................................... 67 Appendix A: Bill of materials ...................................................................................................................................................................................................................................................................................................... 68

Design 1: HPE Synergy Gen10 with Fibre Channel based HPE Nimble Storage ............................................................................................................................................................... 68 Design 2: HPE Synergy Gen10 with HPE 3PAR storage ......................................................................................................................................................................................................................... 72 Design 3: HPE Synergy Gen10 with HPE 3PAR storage, a 16Gb FC topology, and out of frame management servers ............................................................. 76 Design 4: HPE Synergy Gen10 with iSCSI Channel based HPE Nimble Storage ............................................................................................................................................................... 81 Design 5: HPE Synergy Gen10 with HPE Synergy D3940 with VMware vSAN ................................................................................................................................................................. 83

Resources and additional links ................................................................................................................................................................................................................................................................................................ 88

HPE Converged Solution 750 Page 3

Executive summary HPE Converged System 750 and HPE Converged Architecture 750 (combined products referred to as HPE Converged Solutions 750) delivers a scalable, composable, converged infrastructure platform designed, tested, and documented by Hewlett Packard Enterprise, to address the business requirements, workloads, and applications required by our customers. The HPE Converged Solution 750 incorporates a wide range of technologies and products into a portfolio of repeatable, scalable, composable, and supportable solutions supported by Hewlett Packard Enterprise.

This document describes the HPE Converged Solution 750 solution architecture, which combines HPE industry-leading HPE Synergy Gen10 compute modules with HPE 3PAR StoreServ and HPE Nimble Storage as well as multiple switch vendors (Hewlett Packard Enterprise, Arista, and Cisco) to reliably deploy and run a VMware® vSphere based virtualization foundation. Customers can leverage this foundation to support a wide variety of enterprise workloads, such as:

• Data center server consolidation and/or Cloud solutions.

• Business-critical applications, such as Oracle, Microsoft®, and SAP databases and applications.

• Virtual Desktop Infrastructure (VDI) solutions, such as Citrix® and VMware Horizon.

• Workforce-enablement applications, such as Microsoft Exchange Server, SharePoint Server, and Skype for Business Server.

The HPE Converged Solution 750 described in this guide is designed to provide a robust, fault tolerant, scalable, high performance, and high availability solution. Extensive lab testing was conducted to validate that this solution meets the above criteria. The HPE Converged Solution 750 solution allows customers to purchase and deploy their desired configuration with the confidence and support that Hewlett Packard Enterprise’s HPE Converged Solution 750 is an excellent foundation for a wide variety of IT tasks within the enterprise.

Target audience: The target audience for this design guide is Hewlett Packard Enterprise and Hewlett Packard Enterprise Partner solution engineers, Hewlett Packard Enterprise distributors and value-added resellers, and customer strategic decision makers. Readers can use this document to achieve the following goals:

• Gain insight into the value proposition for the HPE Converged Solution 750 solution.

• Better understand HPE Converged Solution 750 component requirements.

• Better understand the recommended software and features that are part of the HPE Converged Solution 750 solution.

• Leverage design guidance to architect an HPE Converged Solution 750 to fit a particular set of business cases.

• Better understand the design considerations related to fault tolerance, performance, and scalability when architecting the solution.

The HPE Converged Solution 750 solution is intended for midsize businesses, large enterprises, and IT service providers who are looking for and understand the value from the combination of consolidation, efficiency, and consistency enabled by the solution.


Introduction One of the biggest challenges within the IT industry is to provide a wide variety of software services with the appropriate service levels and performance for the applications and services needed by their consumers. New workloads and business demands are forcing customers to re-evaluate the way they buy and manage infrastructure. Rather than do-it-yourself, or “integrated systems” assembled from components provided by multiple vendors, administrators now want pre-engineered systems. They need repeatable and flexible building blocks fine-tuned to handle unpredictable workloads. These systems must deliver fast time-to-value, and include software-defined management of servers, storage, and networking which automates the data center and streamlines routine tasks.

Hewlett Packard Enterprise helps customers realize these requirements through HPE Synergy and its composable infrastructure. Composable infrastructure treats each element and resource of IT as a service, and composes them in real time to meet demand. Composable infrastructure, sometimes referred to as "infrastructure as code", is a software-defined solution that goes beyond simply converging or hyper converging hardware, compute, and storage into a single integrated unit. A composable system virtualizes the entire IT infrastructure: it treats physical compute, storage, and network devices as services, and manages all IT through a single application. This virtualization eliminates the need to configure hardware to support specific applications and allows the infrastructure to be managed by software command. Composable infrastructures create pools of resources that are automatically composed in near real time to meet compute needs.

The HPE Converged Solution 750 extends the value of HPE Synergy by including options for HPE 3PAR StoreServ and HPE Nimble Storage arrays, HPE FlexFabric, Arista, and Cisco Nexus Series network switches, and a validated management software stack to provide a highly available, scalable, and high-performance virtualization platform from a single vendor, Hewlett Packard Enterprise. These components are deployed according to Hewlett Packard Enterprise best practices, providing a prescriptively defined foundation onto which IT workloads can be deployed.

HPE Converged Solution 750 is a solution template whose components have been prevalidated together. The template indicates which families of hardware and software to deploy, how to connect and configure them and how to maximize operational efficiency in using them. The HPE Converged Solution 750 simplifies and accelerates deployment with a prescribed and validated deployment guide that produces predictable results. It also reduces the risk of failure due to the lack of knowledge of the hardware and software interdependencies.

HPE Converged Solution 750 high-level design principles This section lays out the high-level design principles for an HPE Converged Solution 750. It also describes how the solution achieves the goals of manageability, security, performance, and integration in the larger data center environment. Each Verified Architecture developed on the principles of an HPE Converged Solution 750 provides more details on these high-level principles in their respective design guides.

Storage All storage devices verified and allowed for use in an HPE Converged Solution 750 environment must have features that provide resiliency against hardware and cabling failures. To mitigate single points of failure at the hardware layer, the storage system must be designed with redundant components, including powered by two (1+1) redundant power supplies as well as a mechanism to ensure data integrity during the event of a power failure. At a minimum, the two storage controller nodes can work independent of each other during a single storage controller failure.

From a host standpoint, connections to the storage systems continue uninterrupted with all I/O routed through a different port on the storage system if there is a path failure. The storage system must also be upgradable in an online fashion and provide flexibility to meet the performance and capacity needs of different workloads the customer may place on the HPE Converged Solution 750.

Network All switches verified and allowed for use in an HPE Converged Solution 750 environment must have features that provide resiliency against hardware and cabling failures.

Network switches are optional in the HPE Converged Solution 750. However, if not using a verified or allowed set of network switches, the network infrastructure that the HPE Converged Solution 750 connects to must have the similar characteristic as described in the following section to prevent a single point of failure.

Multiple redundant powered switch chassis If a SAN Fabric is used with the HPE Converged Solution 750, the solution requires a minimum of two independently and redundantly powered switches for Ethernet networking and SAN networking. The architecture of an HPE Converged Solution 750 requires the connection of all


devices in the solution to a minimum of two network switches. For Ethernet networking, it is required to leverage link aggregation and/or operating system–based load balancing or failover features. For storage, it is required to have a minimum of two independent fabrics so the failure of one fabric does not affect the other. Using a minimum of two switches safeguards against data traffic interruption if there is a switch hardware or operating system–level failure.

In addition to redundant power supplies, fans must be redundant as well in Ethernet switches and SAN switches to prevent switch failure caused by power supply failure or fan failure.

Multichassis link aggregation The HPE Converged Solution 750 solution requires multichassis link aggregation at the compute network level, which allows multiple switch chassis to behave like a single network device when interacting with connected devices. If there is a switch failure, the multichassis link aggregation feature enables multichassis link aggregation groups for all components to interact with the network without interruption.

To reduce the likelihood of split-brain scenarios, a minimum of two links is required between switch groups that participate in multi-chassis link aggregation and to improve interchassis communication. Multichassis link aggregation is required between the switches and servers in an HPE Converged Solution 750 configuration. It is also required for connectivity into the data center network.

Out-of-band management It is required to have all the components in the solution have an out-of-band management capability. Examples of out-of-band connectivity are HPE Integrated Lights-Out interface, HPE Synergy Frame Link Module, network, and SAN switch management ports, and so on.

Computing Although the storage and network components of this solution are highly redundant and support failover scenarios, an individual server does not. Many typical applications do not provide protection against server failure. Therefore, server failures can create outages that impact business operations. Hypervisor features help mitigate this risk, but server failures should be avoided whenever possible. Following the guidance described in the following section will help to mitigate that as much as possible

Redundant power supplies All HPE ProLiant rack-mounted servers in this solution must be connected to at least two fully redundant power supplies. All HPE Synergy 12000 Frames must be fully populated with power supplies (6) and fans (10) that are designed to accommodate the configuration of the servers in the solution under full load and support N+N power-supply redundancy. The frame must be able to lose half of its power supplies and still be able to power all compute modules in the chassis.

Redundant network connectivity Each HPE ProLiant rack-mounted server and HPE Synergy 12000 Frame Fabric Module must be connected to at least two of the Ethernet network switches included in this solution. Two ports are required at a minimum for each component to allow the following operations:

• Continued server operation during the failure of a switch, network cable, or server/HPE Synergy Fabric Ethernet port.

• Continued operation during a firmware update to the network switches in the solution.

For Fibre Channel SAN networking, each HPE ProLiant rack-mounted server must be connected to at least two separate fabrics included in this solution. For an HPE Synergy fabric module, two connections from each module are required to go to the same SAN fabric.

Two ports are required at a minimum per component to allow the following operations:

• Continued server operation during the failure of a switch, fibre cable, or server fibre port.

• Continued operation during a firmware update to the SAN fabrics in the solution.

For Direct Attached storage networking, each HPE ProLiant rack-mounted server and HPE Synergy fabric module must be connected to at least two separate controllers in the same array. Two ports are required at a minimum per component to allow the following continued operations:

• Server operation during a controller, fibre cable or fibre port failure.

• Continued operations during a firmware update.


CPU and memory requirements Each computing and management component is required to use multiple processors. Using multiple processors is not only beneficial for raw CPU performance and capacity which is critical and a limiting factor in a virtualized environment, but also to enable full functionality in the servers themselves such as full memory capacity and IO expansion slots.

Computing options The requirements for redundant power supplies and redundant network connectivity provide a high degree of resiliency and performance at the server layer. Because this solution is workload independent, it does not require that specific bandwidth or performance requirements be met. If additional performance is required or additional fault tolerance is expected, then additional Ethernet or SAN networking adapters may be added to the solution to achieve the performance or fault tolerance needed.

Rack infrastructure When planning the rack infrastructure (rack and power), it is important to take into account the serviceability of the solution that is developed in order to ensure that all components in the solution can be replaced in the event of a failure without taking down the solution. Using the right rack and power design is critical.

The architecture requires PDUs that are easily replaced without having to take the rack down, for example to remove a zero U PDU. It is also required that the power be designed in a side A/B architecture to ensure that if 50% of the power is lost in the rack, the components within it stay functioning and available.

Vertical PDUs or one (1) U mounted PDUs are the preferred PDU deployment technique to meet this requirement. It also allows for all the components in the back of the rack such as HPE 3PAR StoreServ PDUs and Controller, as well as the HPE Synergy 12000 frame components (FLM, Fan, and Power), to be removed without having to remove power cables and power extension bars for example.

The architecture also requires that rack adhere to the all the component requirements contained in it. Examples of the criteria that needs to be met for all components are:

• Front and rear door adequate open area to ensure adequate airflow.

• Adequate front door, rear door and rack side clearances for cabling and serviceability.

• Adequate Rail to Rail Width.

• The rack must be able to accept the adjustable rack rails that come with components and have standard square or round-hole mounts.

All HPE Converged Solution 750 configurations have been verified using the HPE 42U 600mm x 1200mm Enterprise G2 Racks. Third party racks are allowed in the HPE Converged Architecture 750 as long as they meet the following criteria:

• Front and rear doors: A minimum of 65 percent open area must be provided to ensure adequate airflow.

• Front door: The clearance from face of rack to inside of the front door must be a minimum of 77 mm (3 in).

• Rear door: The clearance between the rear of the enclosure and the rear rack door must be a minimum of 177.80 mm (7.00 in) to accommodate system cabling.

• Side: The clearance between the installed rack component and the side panels of the rack must be a minimum of 70 mm (2.75 in).

• Rail to Rail Width: 483 mm (19 in).

• Depth: Maximum clearance between front and rear RETMA rails is 876.00 mm (34.5 in). Minimum clearance for square-hole and round-hole racks is 654.00 mm (25.75 in).

• The rack must be able to accept the adjustable rack rails that are shipped with each frame:

• Minimum rail length: 654.00 mm (25.75 in).

• Maximum rail length: 876.00 mm (34.50 in).

See the HPE Synergy 12000 Frame Site Planning Guide for more information.


HPE Converged Solution 750 HPE Synergy Gen10 VMware 6.0 and 6.5 Solution overview The HPE Converged Solution 750 solution is architected to address concerns of resiliency, fault tolerance, and high availability without impacting the availability of workloads running on the solution. As a core competency of the architecture, no single point of failure can exist in the solution that would affect the workloads running in the environment:

• At the network layer, the solution implements multi-chassis link aggregation groups (MLAG) for the network interfaces of all the components.

• Storage networking, whether it is Direct Attached or Fabric Attached, is designed with redundant paths from multiple storage controllers to each device in the solution.

• All components are configured with an N+N power design to be sure that the solution can withstand losing one side of a data center power feed.

• All management and production workloads reside on shared storage with two or more controllers working in a high availability design.

These features enable the following environmental components to continue operating during maintenance activities on the solution or during a failure scenario of components without causing workloads to fail:

• Network interfaces

• Power distribution units

• Cables

• Switches

• Controllers

The HPE Converged Solution 750 solution offers resiliency, fault tolerance, and high availability within the physical servers and the hypervisor within the computing environment. The HPE Synergy infrastructure and HPE ProLiant rack-mount servers have redundant power supplies to protect against partial power loss or power supply failure, redundant connections to both the network and SAN, and out-of-band management capabilities to enable remote server diagnostics and troubleshooting. The out-of-band management capabilities significantly improve the ability of the administrator to diagnose and recover a server after failure.

As a core component of the computing infrastructure, the HPE Synergy 12000 Frames are uniquely architected as Composable Infrastructure (CI) to match the powerful 'infrastructure-as-code' capabilities of the Hewlett Packard Enterprise intelligent software architecture. Flexible access to compute, storage, and fabric resources allows for current use and repurposing. Linking multiple HPE Synergy Frames efficiently scales the infrastructure with a dedicated single view of the entire management network. Creating multiple composable domains in the infrastructure can efficiently deliver available resources to the business. HPE Synergy Frames reduce complexity by using intelligent auto-discovery to find all available resources to accelerate workload deployments. This drives IT efficiency as the business grows and delivers balanced performance across resources to increase solution effectiveness.

Leveraging VMware vSphere on the computing environment, you can dynamically migrate virtual machines (VMs) across virtualization hosts, which enables the same kind of zero-downtime hardware maintenance that is provided by the storage system. This capability is further expanded with VMware vCenter support for HA clustering and Proactive HA functionality. This support enables you to automatically reboot or migrate VMs to another host if a host failure or a predicted failure event occurs. This version of the HPE Converged Solution supports both VMware vSphere 6.5 as well as 6.0. The use of VMware vSAN and/or HPE Synergy Image Streamer in combination with iSCSI based storage are only supported with VMware vSphere 6.5.

The compute and management VMs can be stored on several storage technologies. For a low cost, resilient, and composable storage option, the HPE Synergy D3940 Storage Module leveraging VMware vSAN technology can be used. For customer looking for the most flexible unified enterprise class storage technology without sacrificing the composability that HPE Synergy provides, HPE 3PAR StoreServ can be used. For simple to use storage with predictive and cloud-enabled flash storage, HPE Nimble Storage may be the best fit.

• HPE 3PAR StoreServ Storage encompasses features that further resiliency and high availability, such as persistent technologies, redundant components, and a Mesh-Active backplane technology. The HPE 3PAR Operating System Suite that comes with HPE 3PAR StoreServ Storage arrays includes everything required to get started. The suite includes HPE 3PAR Adaptive Flash Cache, a true extension of DRAM cache that offers lower latency for read-intensive workloads. HPE 3PAR delivers the performance and flexibility required to accelerate new application deployment, server virtualization, the cloud, new service delivery models, data growth, and whatever else the future might hold - effortlessly.


• The HPE Nimble Storage Predictive Flash is the only storage platform that optimizes across performance, capacity, data protection, and reliability within a dramatically smaller footprint. Adaptive Flash is built upon Cache Accelerated Sequential Layout (CASL™) architecture, and HPE InfoSight™, the cloud-connected management system of HPE Nimble Storage. CASL scales performance and capacity seamlessly and independently. HPE InfoSight leverages the power of deep data analytics to provide customers with precise guidance on the optimal approach to scaling flash, CPU, and capacity around changing application needs, while ensuring peak storage health.

• The HPE Synergy D3940 Storage Module coupled with VMware vSAN provides the flexibility of thinly provisioning software defined storage volumes, while still disaggregating storage resources, allowing compute and storage to grow independently. Leveraging the HPE Synergy allows iSCSI storage to be shared on a low latency server to server connection, including between HPE Synergy frames. HPE Synergy helps enterprises run vSAN in a single infrastructure, addressing the needs of both traditional and emerging business applications.

The HPE Converged Solution 750 topology provides full end-to-end redundancy for all hardware and data links. You can add storage (both capacity and additional arrays), compute, and networking to the scalability limits described in this document.

The architecture of the HPE Converged Solution 750 is based on best practices for all components that are involved. The requirements that are listed in this section are allowed for use in the HPE Converged Solution 750 solution. The requirements have been validated to mitigate the worries of any unknown interdependencies or issues with other components in the hardware and software stack.

Every HPE Converged Solution 750 deployment contains the following components:

• Ethernet switches: HPE FlexFabric, Arista, or Cisco Nexus 10/40GbE series network switches.

• Optional Fibre Channel Storage network switches: HPE B-series SAN switches (required when using Fibre Channel based HPE Nimble Storage or HPE Virtual Connect SE 16Gb FC Module for Synergy).

• Storage: HPE 3PAR StoreServ Storage leveraging a direct or fabric attached Fibre Channel topology, HPE Nimble storage leveraging an iSCSI or fabric attached Fibre Channel topology, and/or HPE Synergy D3940 with VMware vSAN.

• Composable infrastructure: Industry-leading HPE Synergy 480 Gen10 Compute Modules, HPE Synergy Virtual Connect and optional HPE Synergy 12G SAS Connection module paired with the D3940 Storage module. Optional HPE ProLiant Gen10 rack-mount servers for dedicated out of frame solution management.

• Hypervisor: VMware vSphere 6.5 U2 and/or VMware vSphere 6.0 U3.


Figure 1. High-level HPE Converged Solution 750 with VMware


Design principles The HPE Converged System 750 and HPE Converged Architecture 750 (combined products referred to as HPE Converged Solutions 750) are a flexible architecture that can suit various customer and workload requirements a customer requires in their environment. The following design examples are a few of hundreds that are possibilities with the HPE Converged Solution 750 and demonstrate the design flexibility.

Fibre Channel based Storage Attached Networking (SAN) Topology The HPE Converged Solution 750 supports 8/16Gb fabric attached Fibre Channel topology utilizing HPE B-Series SAN switches with HPE 3PAR and HPE Nimble Storage. It also supports direct attached HPE 3PAR StoreServ Storage (connected directly to rack servers and HPE Synergy frames).

Design 1: HPE Synergy Gen10 with Fibre Channel based HPE Nimble Storage This design is suitable for customers who need a high-performance and low cost all flash system.

Compute

• 3 x HPE Synergy 12000 Frame

– 2 x HPE Synergy Composer

– Optional 2 x HPE Synergy Image Streamer

– 2 x HPE Virtual Connect SE 40Gb F8 Module for HPE Synergy

– 4 x HPE Synergy 20G Interconnect Link Module

• HPE Synergy 480 Gen10 Compute Modules

– 2 x HPE Synergy 480 Gen10 Compute Modules for dedicated management hosts

• HPE Synergy 3820C 10/20Gb Converged Network Adapter

– 16 x HPE Synergy 480 Gen10 Compute Modules for VMware compute hosts


Storage

• HPE Nimble Storage All Flash AF40 array

– 8/16Gb FC ports

Networking

• HPE FlexFabric 5940 2-slot 2QSFP+ switch with one (1) x 8-port QSFP+ (Compute Traffic)

– Pair of switches in an IRF configuration

• HPE StoreFabric SN6600B Fibre Channel Switch

– Pair of switches in an FC SAN best practice configuration


Figure 2. HPE Converged Solution 750 with HPE Nimble Storage

For more information on this design example, refer to the HPE Converged Solution 750 Synergy Gen10 with HPE Nimble Fibre Channel Storage VMware 6.5 Deployment Guide.

Design 2: HPE Synergy Gen10 with HPE 3PAR storage This design is suitable for customers who need a balanced performance system but do not want to sacrifice any resiliency and or HA capabilities. In this configuration, three HPE Synergy 480 Gen10 Compute Modules are used to leverage VMware vCenter HA to have a fault tolerant VMware vCenter instance.


Compute







– 3 x Synergy 480 Gen10 Compute Modules for dedicated management hosts (bay 1 in each frame)


– 15 x Synergy 480 Gen10 Compute Modules for VMware compute hosts


Storage

• HPE 3PAR StoreServ 8450 2 Node Storage

– 1 x HPE 3PAR 8000 4-port 16Gb Fibre Channel Host Bus Adapter per node

– All-Inclusive Single-System Software

Networking

• Arista 7160-32CQ Switch (Compute Traffic)

– Pair of switches in an Arista MLAG configuration



– Not required if using HPE 3PAR direct attach (FlatSAN) topologies


Figure 3. HPE Converged Solution 750 with HPE 3PAR StoreServ storage

For more information on this design example, refer to the HPE Converged Solution 750 Synergy Gen10 with 3PAR Storage VMware 6.5 Deployment Guide.

Design 3: HPE Synergy Gen10 with HPE 3PAR storage, a 16Gb FC topology, and out of frame management servers This design is suitable for customers who need a dedicated 16Gb FC topology for the best storage networking performance with the option for a dedicated out of frame management servers to use all the HPE Synergy compute modules for VMware compute hosts.


Compute






– 6 x HPE Virtual Connect SE 16Gb FC Module for HPE Synergy




• HPE Synergy 3830C 16G Fibre Channel Host Bus Adapter

• HPE ProLiant DL360 Gen10 Servers

– Minimum two (2) required to host management software for solution, three (3) required for VMware vCenter HA

– FlexLOM: HPE FlexFabric 10Gb 2-port 534FLR-SFP+ Adapter

– HPE StoreFabric SN1100Q 16Gb Dual Port Fibre Channel Host Bus Adapter

– Optional: HPE Ethernet 10Gb 2-port 530SFP Adapter (for VMware vCenter HA)

Storage

• HPE 3PAR StoreServ 9450 Storage

– 2 x HPE 3PAR 9000 4-port 16Gb Fibre Channel Host Bus Adapter per node

– All-Inclusive Single-System Software

Networking

• Arista 7050SX2-72Q Switch (Compute Traffic)





Figure 4. HPE Converged Solution 750 with end to end 16Gb FC and dedicated out-of-band management

For more information on this design example, refer to the HPE Converged Solution 750 Synergy Gen10 with 3PAR Storage VMware 6.5 Deployment Guide.

iSCSI based Storage Attached Networking (SAN) Topology The HPE Converged Solution 750 supports 10GbE fabric attached iSCSI topology utilizing HPE 5940 series switches with HPE Nimble Storage.

Design 4: HPE Synergy Gen10 with iSCSI based HPE Nimble Storage This design is suitable for customers who need a high-performance iSCSI based system at a low entry cost.

Compute

• HPE Synergy 12000 Frame




– 2 x HPE Synergy 480 Gen10 Compute Modules for dedicated management hosts





Storage

• HPE Nimble Storage HF40 array

– 10GbE iSCSI ports

Networking

• HPE FlexFabric 5940 2-slot 2QSFP+ switch with one (1) x 8-port QSFP+ (Compute and iSCSI Traffic) and one (1) x 24-port SFP+ 2QSFP+ module (Nimble connectivity) in each switch

– Pair of switches in an IRF configuration

– iSCSI Fabric are configured in a side A side B configuration to prevent iSCSI traffic to pass over the IRF/MLAG connections

Figure 5. HPE Converged Solution 750 with HPE Nimble Storage

For more information on this design example, refer to the HPE Converged Solution 750 Synergy Gen10 with HPE Nimble iSCSI Storage VMware 6.5 Deployment Guide.

VMware vSAN based Storage Topology The HPE Converged Solution 750 supports VMware vSAN deployment using HPE Synergy D3940.

Design 5: HPE Synergy Gen10 with Synergy D3940 and VMware vSAN This design is suitable for customers who need a fully Composable Synergy solution ready for a VMware Cloud Foundation deployment.


Compute






– 6 x HPE Synergy 12Gb SAS Connection Module


– 3 x HPE Synergy 480 Gen10 Compute Modules for dedicated management hosts (bay 1 in each frame)


• HPE Smart Array P416ie-m SR Gen10 12G SAS Mezzanine Controller



• HPE Smart Array P416ie-m SR Gen10 12G SAS Mezzanine Controller

Storage

• 6 x HPE Synergy D3940 Storage Modules with HPE Synergy D3940 Redundant I/O Adapter (2 per frame) Each D3940 containing:

– 8 x HPE 400GB SAS 12G Write Intensive SFF SC DS SSD (2 Per HPE Synergy 480 Gen10 Compute Module)

– 24 x HPE 1.6TB SAS MU SFF SC DS SSD (6 Per HPE Synergy 480 Gen10 Compute Module)

Networking

• Arista 7050SX2-72Q Switch (Compute Traffic)



Figure 6. HPE Converged Solution 750 with HPE Synergy D3940 and VMware vSAN

For more information on this design example, refer to the HPE Converged Solution 750 Synergy Gen10 with HPE Synergy D3940 and VMware vSAN Deployment Guide.

Solution Compute / workload servers component The HPE Converged Solution 750 enables customers and their partners a high level of flexibility in the components they choose as well as how they are implemented. While Hewlett Packard Enterprise validates countless configurations and options in an HPE Converged Solution 750, not everything is tested or meets the architectural requirements to be included an HPE Converged Solution 750. Hewlett Packard Enterprise has implemented the following terminology to help readers of this document:

Verified A component that has been verified is a component that is shown in an HPE Converged Solution Deployment guide and has gone through vigorous testing against the solution software and hardware stacks. A verified component may also be a component that is of the same architecture and runs the same software/firmware as a component that has been shown in an HPE Converged Solution Deployment guide. A perfect example is with HPE 3PAR StoreServ 8000 Series arrays. An HPE Converged Solution Deployment guide may not show a particular array in the HPE 3PAR StoreServ 8000 Series that you may want to use, but since all HPE 3PAR StoreServ Array families are architected the same and run the same software and firmware, it is also verified.

Required A required component is a verified component that is required to be in the solution. For example, all compute modules in an HPE Converged Solution are required to have two (2) verified CPUs.


Allowed An allowed component is a component that has not been tested in the HPE Converged Solution. However, Hewlett Packard Enterprise feels confident that because of a component that was verified in the solution, it will function properly in the solution or it is not a component that would affect the functionality of the solution. For example, because the HPE 3PAR StoreServ 9000 Series array has been verified, the HPE 3PAR StoreServ 20000 is allowed as both are based on the same architecture, but scale and perform differently. Another example is any of the “HPE Unique Options” listed in the HPE ProLiant DL360 Gen10 Server QuickSpecs. The options are allowed since they do not affect the functionality of the server in the HPE Converged Solution.

Not Allowed A feature that is not allowed is a component that cannot easily be mapped to a component that is verified to allow it to be verified or allowed in the solution. For example, a NIC from one vendor may be verified, but a NIC from another might not be as they have different architectures and drivers that have not been tested in the HPE Converged Solution.

HPE Synergy 12000 Frame, Management, and Fabric HPE Synergy is a single infrastructure of physical and virtual pools of compute, storage, and fabric resources, and a single management interface that allows IT to instantly assemble and re-assemble resources in any configuration. As the foundation for the HPE New Style of Business infrastructure, HPE Synergy eliminates hardware and operational complexity so IT can precisely deliver infrastructure to applications faster and with greater flexibility.

The HPE Synergy 12000 Frame is the base for an HPE Synergy intelligent infrastructure with embedded management and scalable links that expands to meet business demand. The Frame is the base infrastructure that pools resources of compute, storage, fabric, cooling, power, and scalability. IT can manage, assemble, and scale resources on demand by using the HPE Synergy Frame with an embedded management solution combining the HPE Synergy Composer and Frame Link Modules. The HPE Synergy Frame is designed to meet current needs and future needs with continuing enhancements to compute and fabric bandwidths, including photonics-ready capabilities.

Figure 7. HPE Synergy 12000 Frame

An HPE Synergy 12000 Frame holds up to 12 half-height or 6 full-height Compute Modules and the zone designs allow space for double wide half height and full height Compute and/or Storage devices. Mixing allowed in designated areas. It also includes the flexibility for up to six (6) Interconnect bays for full redundancy of (3) fabrics.

Essential to the functionality of the HPE Synergy Ecosystem is the HPE Synergy Composer, a management appliance with HPE OneView embedded. The appliance plugs directly into the Frame to manage all HPE Synergy resources intelligently and seamlessly. The HPE Synergy Composer appliance is a single point of management for single or scaled frames, which is ideal for on-demand composability. It also extends its management capabilities outside of the frame to components such as HPE 3PAR StoreServ Storage and HPE B-Series SAN fabric. This extension brings composability into those pieces of infrastructure as well. Management of all frame resources is completed through HPE OneView profiles


and templates and Compute, Memory, Storage, and Fabrics within a Frame or across multiple connected Frames are Auto-Discovery upon insertion.

Extending the capabilities of the HPE Synergy Composer, the HPE Synergy Image Streamer is a new approach to deployment and updates for composable infrastructure. HPE Composer captures the physical state of the server in the server profile. HPE Synergy Image Streamer enhances this server profile (and its desired configuration) by capturing your golden image as the ‘deployed software state’ in the form of bootable image volumes. These enhanced server profile and bootable OS images are software structures (‘infrastructure as code’) with images and personalities for true stateless operation. The bootable OS images are stored on redundant HPE Synergy Image Streamer appliances, and they are available for fast implementation onto multiple compute nodes at any time. This enables bare-metal compute modules to boot directly into a running OS and multiple compute nodes to be quickly updated.

HPE Synergy Image Streamer implements ‘infrastructure as code’ to manage physical servers with flexibility like virtual machines to reduce complexity, provide consistency across infrastructure, and save significant time and resources.

Figure 8. HPE Synergy and stateless computing with HPE Synergy Image Streamer

In the HPE Converged Solution 750, it is required to leverage a pair of HPE Virtual Connect SE 40Gb F8 Module for HPE Synergy per logical frame (up to three frames wired together). The HPE Virtual Connect SE 40Gb F8 Module uses a master/satellite architecture to consolidate data center network connections, reduce hardware and scales network bandwidth across multiple HPE Synergy Frames.

The HPE VC SE 40Gb F8 modules eliminate up to 95% of network sprawl at the compute module edge with one device that converges traffic inside frames and directly connects to external LANs. Each redundant pair of Virtual Connect modules provide four adjustable downlinks connections (three Ethernet and one Fibre Channel) on each compute module. A Fibre Channel Upgrade license must be purchased separately to enable the Fibre Channel functionality. Up to six uplinks using QSFP+ interfaces are available for connection to upstream Ethernet switches. Up to 24 uplinks are available for connection to upstream Ethernet and Fibre Channel are available with splitter cables. The HPE VC SE 40Gb F8 Modules reduce the number of components required compared to traditional and other converged network solutions by eliminating the need for separate Ethernet and Fibre Channel switches and cables. Also, Virtual Connect wire-once connection management is built-in enabling compute modules adds, moves, and replacement in minutes instead of days or weeks. The Master/Satellite disaggregated architecture removes fixed of ratios of interconnects in every frame and allows extending networking resources pool for Virtual Connect to satellite frames.


Figure 9. HPE Virtual Connect SE 40Gb F8 Module for HPE Synergy

The HPE VC SE 40Gb F8 modules, master module, contains intelligent networking capabilities that extend connectivity to frames equipped with satellite modules. The HPE Synergy 20Gb Interconnect Link Module, satellite module, is designed for Composable Infrastructure. Based on disaggregated, rack-scale design, it uses a master/satellite architecture to consolidate data center network connections, reduce hardware and scale network bandwidth across multiple HPE Synergy Frames. This module extends HPE composable fabric up to two additional satellite frames with 20Gb connectivity to HPE Synergy Compute Modules. It also replaces fixed ratio of interconnects in each frame by extending the fluid pool of network resources without network oversubscription between frames.

Figure 10. Three Frame Master/Satellite wiring example

For the best-in-class enterprise data center performance environments, the HPE VC SE 16Gb FC module is the best choice for operating systems and applications demanding low-latency high I/O performance, for example, SSD storage. Much like the HPE Virtual Connect SE 40Gb F8 Module for HPE Synergy, it simplifies the process of connecting servers to Fibre Channel networks by reducing cables and your SAN switch management


domain. It also allows you to change servers in just minutes. No other server enclosure technology (iSCSI, Ethernet, or FCoE) on the market can guarantee up to 384Gb/s of bandwidth. The unique QSFP ports not only significantly simplify cabling infrastructure, reduces cabling from four to one, but also helps reduce power 2.5x attributed to optics transceivers. The integrated HPE Synergy design frees up rack space, reduces power and cooling requirements.

Figure 11. HPE Virtual Connect SE 16Gb FC Module for HPE Synergy

For environments looking to leverage direct attached storage, the HPE Synergy 12Gb SAS Connection Module connects Synergy Compute Modules to in-frame storage resources. In conjunction with the Synergy D3940 Storage Module, the Synergy SAS Connection Module connects composable direct attached storage for up to 10 compute modules in a single frame.

The HPE Synergy 12Gb SAS Connection Module dynamically creates virtual JBODs for any compute module in the HPE Synergy 12000 Frame, remaps connections, or changes personality, all done with stored profiles. The Connection Module provides in-frame connectivity between Synergy compute modules and up to up to five storage modules. Two Connection Modules can be added to each HPE Synergy 1200 Frame providing redundant paths to the storage modules.

Verified, allowed, and required component design criteria Every HPE Synergy Gen10 Converged Solution 750 has been verified with and requires:

• One (1) to 21 HPE Synergy 12000 Frames, each equipped with:

– Six (6) HPE 2650W Performance Hot Plug Titanium Plus power supplies

– Ten (10) Hot-Plug Fans

• Each HPE Synergy Frame management ring must contain the following:

– Two (2) HPE Synergy Composer

• Each Logical enclosure must contain the following:

– Two (2) HPE Virtual Connect SE 40Gb F8 Module for HPE Synergy

– When a logical enclosure contains, two (2) or three (3) frames, two (2) HPE Synergy 20Gb Interconnect Link Module must be added per additional frame.

• The following optional components have been verified to add to the HPE Synergy 12000 Frame in this architecture

– Two (2) HPE Synergy Image Streamers per Logical Enclosure

Note At this time, the use of HPE Synergy Image Streamer and an iSCSI based HPE Nimble array connected to the same logical enclosure is not supported with VMware vSphere 6.0.

– Two (2) or four (4) additional HPE Virtual Connect SE 40Gb F8 Module for HPE Synergy

– Two (2) or four (4) HPE Virtual Connect SE 16Gb FC Module for HPE Synergy

– Two (2) HPE Synergy 12Gb SAS Connection Modules


Note Additional fabric modules must be added in pairs and there are four (4) additional fabric slots available total.

Also note each compute module in the frame must have the appropriate verified/allowed compute I/O adapter in it to take advantage of the fabric module inserted into the frame.

Any supported cables and transceiver listed in the QuickSpecs of the interconnect module are supported in the HPE Converged Solution 750.

HPE Synergy Gen10 Compute Module The HPE Synergy Gen10 Compute Modules that are used in the HPE Converged Solution 750 solution deliver industry-leading performance, which has increased over previous generations. Embedded management enhancements include integrated life cycle automation capabilities that are enabled by innovations including the following:

• HPE Intelligent Provisioning for easy system setup.

• Active health checks for agentless hardware monitoring and alerting.

• HPE Smart Update for automated firmware and system software maintenance.

HPE Synergy Gen10 Compute Modules and the HPE Converged Solution 750 also use HPE OneView to automate key management processes, including the physical development, configuration, and problem management of the system.

Hewlett Packard Enterprise engineered the HPE Synergy Gen10 Compute Modules to get the most out of the latest advances in processors, memory, networking technology, and management solutions through the following features:

• HPE Synergy 480 Gen10 Compute Modules:

– Two socket Intel® Xeon® Processor Scalable Family from 4 - 28 Cores; 85W - 205W; 1.7 - 3.6 GHz.

– 24 DDR4 DIMM sockets (12 per processor) for up to 1.5 TB of HPE SmartMemory and a maximum memory speed up to 2666MT/s.

– Up to three (3) I/O expansion mezzanine connectors.

• Onboard storage with flexible HPE Smart Array controller options.

• HPE FlexibleLOM technology with 20Gb adapters.

• Unified Extensible Firmware Interface (UEFI) for full configuration and management flexibility.

• HPE Integrated Lights-Out 5 (iLO 5).

HPE Converged Solution 750 provides the flexibility to configure each HPE Synergy 480 Gen10 Compute Module with different memory, CPU, bootable media, and I/O configurations.

Verified, allowed, and required component design criteria HPE Converged Solution 750 provides the flexibility to configure each HPE Synergy 480 Gen10 Compute Module with different memory, CPU, bootable media and I/O configurations. Certain options are either verified or allowed, and some are required per this HPE Converged Solution 750 Design Guide.

Every HPE Converged Solution 750 configuration using the HPE Synergy Gen10 Compute Module has been verified with and requires:

• Two (2) to 252 HPE Synergy Gen10 Compute Modules per HPE Synergy Management Ring, each equipped with:

– Two (2) Intel® Xeon® Processor Scalable Family from 8 - 28 Cores.

– One (1) HPE Synergy 3820C 10/20Gb Converged Network Adapter.

– Boot from SAN or two (2) SFF HDD/SSD supported by the server with one of the following controllers:

• HPE Smart Array P204i-c SR Gen10 (four Internal Lanes/1GB Cache) 12G SAS Modular Controller.


• HPE Smart Array P416ie-m SR Gen10 (eight Int 8 External Lanes/2GB Cache) 12G SAS Mezzanine Controller (Required when a server is connecting to a HPE Synergy D3940 Storage Module).

– The following boot methods have been verified:

• Boot from direct attached storage (RAID 1 required).

• Boot from SAN (3PAR only).

• Stateless boot using HPE Synergy Image Streamer.


• The following optional components have been verified to add to the HPE Synergy Gen10 Compute Module:

– Additional HPE Synergy 3820C 10/20Gb Converged Network Adapter.

– HPE Synergy 3830C 16Gb Fibre Channel Host Bus Adapter.

• When using an HPE Synergy Compute Module as a Converged Solution 750 Management Server, the following requirements must be met and be the same across all management nodes:

– Two (2) Intel® Xeon® Processor Scalable Family from 14 - 28 Cores with a minimum frequency of 2.0GHz. The recommended CPU Choice is the Intel® Xeon® Gold 6132 Processor.

– A minimum of 192GB of memory. Please ensure that you populate the DIMM slots in a balanced configuration. For example 16GB x 12DIMMs is an optimal configuration as it places a DIMM in each channel.

• https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=a00017079enw

– Servers must have a local disk capacity of 600GB (configured in RAID 1) or more and cannot boot from SAN.

– Although a minimum of two (2) HPE Synergy 480 Gen10 Compute Modules are required to host the management infrastructure, a minimum of three (3) are required if you wish to leverage VMware vCenter HA and/or VMware vSAN.

• The following optional components can be added to the HPE Synergy Gen10 Compute Module but have not been verified in the HPE Converged Solution 750:

– HPE Synergy 3530C 16Gb Fibre Channel Host Bus Adapter.

Note The embedded HPE Smart Array S100i SR SW RAID controller is not supported by HPE OneView or VMware vSphere. You are required to use a Smart array listed as verified or allowed in this section for local boot. HPE Synergy 480 Gen10 Compute Module Premium Backplane model is preferred choice for option flexibility with regards to supporting a single Smart Array controller connectivity to the HPE Synergy D3940 and component module drive bays.

Out of Frame Management servers – HPE ProLiant DL360 Gen10 Servers The HPE Converged Solution 750 solution provides the option for redundant HPE ProLiant DL360 Gen10 Servers as management compute infrastructure in lieu of dedicating HPE Synergy Gen10 Compute Modules. These servers run the entire HPE Converged Solution 750 solution management stack and contain many of the same features as HPE Synergy Gen10 Compute Modules.

Hewlett Packard Enterprise engineered the HPE ProLiant DL360 Gen10 Servers to get the most out of the latest advances in processors, memory, networking technology, and new management solutions. The HPE Converged Solution 750 leverages the following features:

• Two socket Intel® Xeon® Processor Scalable Family from 4 - 28 Cores; 85W - 205W; 1.8 - 3.6 GHz

https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=a00017079enw


• 24 DDR4 DIMM sockets (12 per processor) for up to 3TB of HPE SmartMemory and a maximum memory speed up to 2666MT/s

• Connectors for up to three PCI-E expansion slots (two x 16, one x 8)

• Onboard storage (up to 10 SFF SAS/SSD drives) with flexible HPE Smart Array controller options

• Embedded 4x1GbE NIC

• HPE FlexibleLOM technology with 10Gb adapters

• Unified Extensible Firmware Interface (UEFI) for full configuration and management flexibility

• HPE Integrated Lights-Out 5 (iLO 5)

Verified, allowed, and required component design criteria HPE Converged Solution 750 provides the flexibility to configure each HPE ProLiant DL360 Gen10 Server with different memory, CPU, bootable media, and IO configurations. Configurations (CPU, memory, and I/O) must be identical across each of the management hosts. Certain options are either verified or allowed, and some are required per this HPE Converged Solution 750 design guide.

Every HPE Converged Solution 750 configuration using the HPE ProLiant DL360 Gen10 Servers for management compute infrastructure has been verified with and requires:

Note Although a minimum of two (2) HPE ProLiant DL360 Gen10 Servers are required to host the management infrastructure, a minimum of three (3) are required if you wish to leverage VMware vCenter HA and/or VMware vSAN.

It is verified to use the HPE ProLiant DL380 Gen10 server as an alternative to the HPE ProLiant DL360 Gen10 server. The same required/verified/allowed list below still applies)

• Two (2) HPE ProLiant DL3x0 Gen10 Servers, each equipped with:

– Two (2) Intel® Xeon® Processor Scalable Family from 14 - 28 Cores with a minimum frequency of 2.0GHz. The recommended CPU Choice is the Intel® Xeon® Gold 6142 Processor.

– A minimum of 192GB of memory. Please ensure that you populate the DIMM slots in a balanced configuration. For example 16GB x 12DIMMs is an optimal configuration as it places a DIMM in each channel.

• https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=a00017079enw

– One (1) HPE FlexFabric 10Gb 2-port 534FLR-SFP+ Adapter.

– One (1) HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter.

– One (1) HPE StoreFabric SN1100Q 16Gb Dual Port Fibre Channel Host Bus Adapter (When using Fibre Channel based storage).

– Two (2) to Ten (10) SFF HDD/SSD supported by the server with the HPE Smart Array P408i-a SR Gen10 12G SAS Modular.

– Two (2) HPE 800W or greater Flex Slot Hot Plug Power Supplies.

– HPE OneView Advanced license (HPE iLO Advanced license is also required, either bundled or separate from the HPE OneView Advanced license).

• The following optional components have been verified to add to the HPE ProLiant DL3x0 Gen10 Servers for management compute infrastructure:

– Ethernet and Converged Network Adapters:

• HPE StoreFabric CN1100R Dual Port Converged Network Adapter.

• HPE Ethernet 10Gb 2-port 530SFP Adapter (Required when using VMware vCenter HA or dedicated iSCSI links and 534FLR is used).

• HPE Ethernet 10/25G 2-port 640SFP28 Adapter (Required when using VMware vCenter HA or dedicated iSCSI links and 640FLR-SFP28 is used).

https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=a00017079enw


– Fibre Channel Adapters:

• Additional HPE StoreFabric SN1100Q 16Gb Dual Port Fibre Channel Host Bus Adapter.

• The following optional components are allowed to be added to the HPE ProLiant DL3x0 Gen10 servers but have not been verified in the HPE Converged Solution 750:

– HPE Unique Options listed in the HPE ProLiant DL360 or DL380 Gen10 Servers QuickSpecs

Ethernet and Storage Networking All switches verified and allowed for use in an HPE Converged Solution 750 environment have features that provide resiliency against hardware and cabling failures.

Ethernet Switches are optional in the HPE Converged Solution 750. For larger configurations or configurations that require the use of ToR switches before connecting to an aggregation layer, the following section shows the characteristics required for a ToR switch and the verified and allowed switches by Hewlett Packard Enterprise.

Multiple redundant powered switch chassis If an FC SAN Fabric is used when switches are integrated with the HPE Converged Solution 750, the solution requires a minimum of two independently and redundantly powered switches for compute Ethernet networking and Fibre Channel (FC) SAN networking.

In addition to redundant power supplies, fans must be redundant as well in Ethernet switches and SAN switches to prevent switch failure caused by power supply failure or fan failure.

General Ethernet Network Topology Design The HPE Converged Solution 750 solution requires link aggregation (LAG) and when possible multi-chassis link aggregation (MLAG) at the compute network level. MLAG enables the multiple switch chassis to behave like a single network device when interacting with connected devices. The MLAG feature enables MLAG groups for all components to interact with the network without interruption if there is a switch failure.

To facilitate MLAG, connectivity is required between switches that have ports belonging to the same IRF port group (HPE Networking) or port channel (Arista and Cisco vPC). You must have a minimum of two links between switch groups that participate in MLAG to reduce the likelihood of split-brain scenarios and to improve inter-chassis communication.

Hewlett Packard Enterprise highly recommends implementing MLAG failure detection mechanism. For Hewlett Packard Enterprise, IRF is known as Multiple Active Detection (MAD) for Cisco, it is the Peer-KeepAlive link, and for Arista MLAG dual-primary. You can leverage several methods of MAD and Peer-KeepAlive; see the switch documentation for more information and implement the best option for your customer.

Ethernet topology when using Fibre Channel attached storage It is required in the HPE Converged Architecture 750 to configure end-to-end link aggregation when using a Fibre Channel based configuring. This configuration includes MLAG configured between the Ethernet switches and any HPE Synergy frames and (if included in the solution) HPE ProLiant rack servers. Also using HPE Synergy Link Aggregation groups, configuring switch assisted load balancing on the VMware ESXi standard and distributed switches as well. All components in this topology must be configured and support load balancing across multiple links in an MLAG group to increase the total throughput that is available between network-connected devices.

It is also required that MLAG or some Layer 3 multi-path routing protocol (ECMP, VxLAN, and so on) be implemented for connectivity into the data center network.

The same high-level illustration as shown in Figure 12 applies to Arista and Cisco networking as well.


Figure 12. Ethernet configurations using Multi-chassis link aggregation groups in a segregated network design

Fibre Channel Storage Attached Network (SAN) Topology Design For SAN fabric networking, if used, each HPE ProLiant rack-mounted server must be connected to at least two of the SAN switches in two separate fabrics included in this solution. For an HPE Synergy fabric module, a minimum of two connections from each module are required to go to the same SAN fabric. Both ports from the odd-numbered interconnect must go to one SAN fabric switch and both ports from the even numbered interconnect must go to another SAN fabric switch included in the solution. Hewlett Packard Enterprise does not support having one HPE Synergy fabric module go to separate fabrics. However, Hewlett Packard Enterprise does support having one HPE Synergy fabric module go to multiple switches in the same SAN fabric. See Figure 13 for a visualization of this SAN network topology. At a minimum, you must have two ports per component to enable continued server operation during the failure of a switch, fibre cable, or server fibre port and to enable continued operation during a firmware update to the SAN fabrics in the solution.


Figure 13. SAN network Topology with HPE Nimble Storage example

Zoning is required in the SAN switches. Hewlett Packard Enterprise recommends and has verified in the HPE Converged Solution one (host) to many (storage ports). Currently zoning in a HPE Nimble-based solution is done manually. Example is a zone construct for SAN A with one host to an HPE Nimble array with four (4) host ports (2 ports from different PCI cards in each node) connected to SAN A.

• Host is sms01

• HPE Nimble Storage is sstor01, controller A is abbreviated to CA and Controller B is abbreviated to CB.

ssan01:FID128:admin> alicreate sms01_P1,"10:00:c2:31:06:a0:00:04" ssan01:FID128:admin> alicreate sstor01_CA_fc2a,"56:c9:ce:90:1c:9b:7e:01" ssan01:FID128:admin> alicreate sstor01_CA_fc3a,"56:c9:ce:90:1c:9b:7e:05" ssan01:FID128:admin> alicreate sstor01_CB_fc2a,"56:c9:ce:90:1c:9b:7e:09" ssan01:FID128:admin> alicreate sstor01_CB_fc3a,"56:c9:ce:90:1c:9b:7e:0d" ssan01:FID128:admin> zonecreate sms01_P1_sstor01, "sms01_P1; sstor01_CA_fc2a; sstor01_CB_fc2a; sstor01_CA_fc3a; sstor01_CB_fc3a"

When using HPE 3PAR StoreServ arrays, zoning is completed via HPE OneView and its integration with HPE Brocade Network Advisor software in a fabric attached topology. It is important to properly configure HPE OneView Storage port groups so that every pair of peer persistence ports is in the same Port Group. Below is an example of a 4 Node HPE 3PAR 8000 array.


Table 1. Port group mapping table for a 4 node HPE 3PAR StoreServ Storage 8000 array in Fabric Topology

HPE 3PAR StoreServ Port Group

0:0:1 PG_001_101

0:0:2 PG_002_102

0:2:1 PG_021_121

0:2:2 PG_022_122

0:2:3 PG_023_123

0:2:4 PG_024_124

1:0:1 PG_001_101

1:0:2 PG_002_102

1:2:1 PG_021_121

1:2:2 PG_022_122

1:2:3 PG_023_123

1:2:4 PG_024_124

2:0:1 PG_201_301

2:0:2 PG_202_302

2:2:1 PG_221_321

2:2:2 PG_222_322

2:2:3 PG_223_323

2:2:4 PG_224_324

3:0:1 PG_201_301

3:0:2 PG_202_302

3:2:1 PG_221_321

3:2:2 PG_222_322

3:2:3 PG_223_323

3:2:4 PG_224_324

For Direct Attached networking to HPE 3PAR StoreServ Storage, each HPE ProLiant rack-mounted server and HPE Synergy fabric module must be connected to at least two separate HPE 3PAR controllers in the same array. You must have two ports per component at a minimum to enable the following operations:

• Continued server operation during a controller, fibre cable or fibre port failure.

• Continued operations during a firmware update to an HPE 3PAR storage array.

When using Direct Attached networking, HPE OneView Port Group configuring is automatically configured and no further configuration is needed.


Figure 14. Direct Attached Topology example

The following two tables demonstrate connectivity from an HPE 3PAR StoreServ two and four node array in a Direct Attached topology.

Table 2. Port group mapping table for a two (2) node HPE 3PAR StoreServ Storage 8000 connecting to a single Frame HPE Synergy Configuration

HPE 3PAR StoreServ Port Number

Device Port

0:0:1 HPE Synergy Master Frame 1 ICM 3 Q5:1




0:2:3 (Optional) HPE ProLiant Gen10 Rack server – 1 PCI HBA Port 1





Device Port






Table 3. Port group mapping table for a 4 node HPE 3PAR StoreServ Storage 8000 connecting to a 3 Frame HPE Synergy Configuration


Device Port

N0:2:3 (Optional) HPE ProLiant Gen10 Rack server – 1 PCI HBA Port 1






N0:0:1 HPE Synergy Master Frame 1 ICM 3 Q5:1

















Ethernet topology when using HPE Nimble iSCSI storage Similar to a FC based topology, it is required in the HPE Converged Architecture 750 to configure end-to-end link aggregation for Ethernet traffic. However for the iSCSI traffic itself, the network must be architected to ensure that the traffic:

• Is lossless (via dedicated iSCSI links, QOS policies and/or Priority flow control).

• That all iSCSI traffic does not cross over the HPE Synergy Virtual Connect MLAG links or network switch IRF/Peer links.

In Figure 15, iSCSI VLANs/SANs, represented in orange for SAN A and purple for SAN B, are separate VLANs on each MLAG member switch and are not included on each respective peer. Ethernet networking still participates in the end-to-end link aggregation topology shown in the FC configuration, with a separate uplink sets into the HPE Synergy Logical Enclosure configuration for the iSCSI traffic.

If external management servers are used and they use the same ports for both Ethernet and iSCSI traffic, priority flow control must be configured to ensure that the iSCSI traffic get priority over normal Ethernet traffic.

At this time, support is not available today to utilize the iSCSI offload functionality on HPE Flex Fabric and HPE Synergy Converged Network adapters with HPE Nimble Storage. In order to establish iSCSI communication, the creation of two VMkernel ports are required from each network adapter. Then the two VMkernel ports will be added to the configured VMware software based iSCSI adapter.

The same high-level illustration applies to Arista and Cisco networking as well.

Figure 15. Ethernet configurations in an iSCSI based converged network design



Ethernet topology when using VMware vSAN Similar to a FC based topology, it is required in the HPE Converged Architecture 750 to configure end-to-end link aggregation for Ethernet traffic. However, as with iSCSI, VMware vSAN traffic must be architected to ensure that the traffic:

• Is lossless (via dedicated VMware vSAN links).

• does not cross over the HPE Synergy Virtual Connect MLAG links or network switch IRF/Peer links.

In Figure 16, the VLANs for VMware vSAN, represented in orange for SAN A and purple for SAN B, are separate VLANs on each MLAG member switch and are not included on each respective peer. Ethernet networking still participates in the end-to-end link aggregation topology shown in the FC configuration, with separate uplink sets into the HPE Synergy Logical Enclosure configuration for the iSCSI traffic. In Figure 16, the VLANs for VMware vSAN, represented in orange for SAN A and purple for SAN B, are separate VLANs on each MLAG member switch and are not included on each respective peer. Ethernet networking still participates in the end-to-end link aggregation topology shown in the FC configuration, with separate uplink sets into the HPE Synergy Logical Enclosure configuration for the iSCSI traffic.

In order to establish vSAN communication, the creation of two VMkernel ports are required from each network adapter (both in HPE Synergy Compute modules and external HPE ProLiant DL management servers if used). The two VMkernel ports will be added to the configured VMware software based iSCSI adapter.

The same high-level illustration applies to Arista and Cisco networking as well. In the following example Bridge-Agg 112 and 113 are only needed if you are extending your VMware vSAN clusters beyond a single HPE Synergy Logical Enclosure.


Figure 16. Ethernet configurations in a VMware vSAN based converged network design

Compute Networking – HPE, Arista, and Cisco Nexus switches Compute network connectivity in this solution is provided by a redundant set of 10/25/40/100GbE network switches. Because Hewlett Packard Enterprise understands customers want choices in networking vendors, especially for Top of Rack products, select HPEN, Arista, and Cisco Nexus switch models have been tested with the HPE Converged Solution 750.

Each switch is configured to meet the following network design goals:

• Multichassis link aggregation. Any given compute node has at least one connection to each switch in a pair of switches. The switch pair is interconnected with two or more links as well. The switches are configured in a vendor-specific way to provide chassis-level fault tolerance; if one switch goes down, traffic proceeds uninterrupted.

• Link Aggregation Control Protocol (LACP) and NIC Teaming. LACP and/or VMware NIC teaming technologies are configured on all compute nodes and infrastructure components to reduce the impact of a cable failure and network switch failure or misconfiguration.

• VLANs (802.1q). Use the 802.1q networking standard to partition the physical network into multiple VLANs. This configuration allows traffic separation for infrastructure management, back-end storage, and application access from multiple users.

HPE FlexFabric 5940 Series Switches The HPE FlexFabric 5940 Switch Series is a family of high-performance and low-latency 10GbE and 40GbE top-of-rack (ToR) data center switches. The HPE FlexFabric 5940 Switch includes 100G uplink technology which is part of the HPE FlexFabric data center solution and is a cornerstone of the HPE FlexNetwork architecture. The 5940 Switch is suited for deployment at the aggregation or server access layer of large


enterprise data centers or at the core layer of medium-sized enterprises. It is optimized for high-performance server connectivity, convergence of Ethernet and storage traffic, and virtual environments.

Figure 17. HPEN FlexFabric 5940 4-Slot switch

Key features include:

• VXLAN Layer 2 and Layer 3 gateway support.

• EVPN, OpenFlow, and SDN automate manual tasks and speed service delivery.

• High-density 10GbE, 40GbE with 40Gb or 100Gb uplink for spine and leaf deployments.

• IPv6 support in ToR with full L2/L3 features.

• High-performance switching Cut-through and non-blocking architecture delivers low latency (~1 microsecond for 10GbE) for very demanding enterprise applications.

The HPE FlexFabric 5940 Switch Series enables customers to scale their server-edge 10/40/100GbE ToR deployments with the following ports delivered in a 1RU design:

• High-density 48 x 10GbE (SFP or Base-T) with 6 x 40GbE ports.

• 48 x 10GbE (SFP+ or BASE-T) with 6 x 40 or 100GbE ports.

• 32 x 40GbE ports.

The HPE FlexFabric 5940 Switch Series is available with a 2-slot modular version and a 4-slot modular version. Each modular slot supports a variety of modules with various combinations of 10GbE (SFP+ or Base-T) 40GbE (QSFP+) and 100Gbe (QSFP28) ports.

HPE Intelligent Resilient Fabric

HPE Intelligent Resilient Fabric (IRF) overcomes the limitations of legacy spanning tree designs by providing rapid failover for delay-sensitive, mission-critical applications, and dramatically improving network usage and performance in the network core. With HPE IRF, you can manage up to nine switches as a single logical device with one IP address.

For decades, enterprises invested in network infrastructure capacity only to discover that most of the acquired bandwidth must be blocked. Companies using STP are forced to design their networks to adapt to the needs of this aging protocol. With HPE IRF, the network administrator can build STP-free networks that use all the available bandwidth. In essence, enterprises are able to take advantage of the investments that they have made in networking capacity.

In addition to freeing strained networking bandwidth, HPE IRF adds link aggregation capability to links that connect physical devices. Link aggregation dramatically enhances network reliability and makes aggregate network bandwidth available to devices in the IRF domain. The result is a network that is ready to support the most demanding business applications. Hewlett Packard Enterprise publicly demonstrated the superior capabilities of IRF by showing that workload mobility can be improved by more than 80% when VMware vSphere vMotion is used.


Arista 7160 Series Switches The Arista 7160 Series are key components of the Arista 7000 Series portfolio of data center switches. Highly dynamic cloud data center networks continue to expand bringing with them ever increasing bandwidth demands, accelerating the need for dense 25 and 100 Gigabit Ethernet switching in both leaf and spine tiers of modern networks. The Arista 7160 Series are purpose built fixed configuration 10/25GbE and 100GbE systems with wire speed layer 2 and layer 3 features, built for the highest performance environments, and the largest scale data centers. They combine scalable L2 and L3 resources and high density with a highly programmable and customizable switch architecture.

Figure 18. Arista 7160

The Arista 7160 can be deployed in a wide range of open networking solutions including large scale layer 2 and layer 3 cloud designs, overlay networks, virtualized or traditional enterprise data center networks.

The Arista 7160-48YC6 offers 48 wire speed 10/25G ports with six 100G QSFP (that allows for a choice of 6x 100GbE, 24x 25GbE, 6x 40GbE, or 24x 10GbE) for up to 72 total 25G or 10G ports at wire speed performance with a 24MB of buffer. This allows for easy migration from 1/10G to 1/10/25G using a familiar SFP connection.

The Arista 7160-32CQ offers 32 wire speed 40/100G ports with flexible interface combinations of up to 32x 100G/40G, 128x 25/10G, 64x 50G

• Multi-chassis link aggregation. Any given compute node has at least one connection to each switch in a pair of switches. The switch pair is also interconnected with two or more links. The switches are configured in a vendor-specific manner to provide chassis-level fault tolerance; if one switch goes down, traffic proceeds uninterrupted.

• Link Aggregation Control Protocol (LACP) and NIC teaming. All compute nodes and infrastructure components have multiple network connections that LACP and/or VMware NIC teaming technologies configure automatically to reduce the opportunity for human error.

• VLANs (802.1q). You can partition the physical network into multiple VLANs by using the 802.1q networking standard. This configuration allows traffic separation for infrastructure management, back-end storage, and application access from multiple users.

• Scalable designs with up to a 128-way ECMP provides flexibility and balances traffic evenly across the largest leaf-spine designs.

• Support for standards based IEEE 25GbE with mix and match support for both 10G and 25G for simple and cost effective migration.

• AlgoMatch to match business intent to network policy for flow matching, access control and telemetry.

• Flexible forwarding profiles for optimal allocation of L2 and L3 forwarding table resources in multiple use cases.

• VXLAN gateway, bridging and routing with VMTracer features to enable next generation data center designs.

AlgoMatch (TM)

AlgoMatch is a unique Arista innovation for modern cloud networks, combining both software and hardware to enable more flexible and scalable solutions for access control, policy based forwarding and network telemetry. By combining general purpose memory with advanced software algorithms, AlgoMatch delivers higher scale, performance and efficiency with lower power and is more cost effective than traditional solutions.


AlgoMatch provides a more efficient packet matching algorithm that in turn enables flow matching for access control, policy and visibility. The net benefits are a high performance policy engine with both increased functionality and scale in a cost and power efficient solution.

• AlgoMatch enables IPv4 and IPv6 access control at the same scale.

• L4 rule ranges are programmed efficiently without expansion or reduced capacity.

• Multiple actions can be performed on a single packet or flow.

• User defined filters allow flexible packet classification based on offsets for custom actions.

• Supports rich policy with consistent semantics that would exhaust classical resources.

Arista 7050SX2 Series Switches Arista 7050SX2 10/40GbE network switches offer native support for VMware, virtualization, and hundreds of Linux® applications that are designed to meet the stringent management, power, and cooling requirements of today’s most demanding data centers. The Arista Extensible Operating System (EOS) power the switches. The Arista EOS is a modern network operating system designed to address the needs of data center networks. If the need for scalability requires a network overlay, the switches are also capable of working with VXLAN and controllers such as VMware NSX.

Figure 19. Arista 7050SX2-72Q

The Arista 7050SX2-72Q (48 SFP+ and 6 QSFP+ ports) and 7050SX2-128 (96 SFP+ and 8 QSFP+ ports) provide high-density platforms and deliver line rate VXLAN routing in a single pass.

The Arista 7050SX2-72Q combine 48 host facing SFP+ interfaces with a choice of 10/40GbE options. The QSFP+ interfaces increase the system density up to 72 10 Gigabit ports or 48 10GbE and 6 40GbE ports. The combinations allow for design flexibility.

For higher density requirements, the Arista 7050SX2-128 offers 96 10GbE SFP+ interfaces and 8 40GbE QSFP+ interfaces in two RU. All systems are ideal for top of rack or end of row with consistent performance, and high availability.

The Arista 7050QX2-32S can be deployed as an all 40GbE system or as a combination of 10GbE leaf ports and 40GbE spine ports within the same chassis. Power efficient, compact footprint, flexible table sizes, and EOS innovations like SSU and MLAG make the 7050QX2-32S ideal for leaf and spine as well as scale-out ECMP designs. The 7050QX-32S offers four SFP+ ports that allow for 1/10G connections increasing the deployment options.

• Multi-chassis link aggregation. Any given compute node has at least one connection to each switch in a pair of switches. The switch pair is also interconnected with two or more links. The switches are configured in a vendor-specific manner to provide chassis-level fault tolerance; if one switch goes down, traffic proceeds uninterrupted.

• Link Aggregation Control Protocol (LACP) and NIC teaming. All compute nodes and infrastructure components have multiple network connections that LACP and/or VMware NIC teaming technologies configure automatically to reduce the opportunity for human error.

• VLANs (802.1q). You can partition the physical network into multiple VLANs by using the 802.1q networking standard. This configuration allows traffic separation for infrastructure management, back-end storage, and application access from multiple users.

• Maximum flexibility for scale-out network designs.

• The Arista 7050SX2 is designed to support scale-out network designs that enable solutions to start small and evolve over time; it includes enhancements that enable flexible scale-out designs:

• 64-way equal-cost multi-path (ECMP) and 64-way MLAG to balance traffic evenly across large scale 2 tier leaf-spine designs.

• Flexible allocation of L2 and L3 forwarding table resources.

• VXLAN routing, bridging, and gateway for seamless physical and virtual networking.


Arista Peer Link The Arista Peer Link overcomes the limitations of legacy-spanning tree designs by providing rapid failover for delay-sensitive, mission-critical applications. It also dramatically improves network usage and performance in the network core.

For decades, enterprises invested in network infrastructure capacity only to discover that most of the acquired bandwidth must be blocked. Companies using Spanning Tree Protocol (STP) are forced to design their networks to adapt to the needs of this aging protocol. With Arista Peer Link, the network administrator can build STP-free networks that use all the available bandwidth. In essence, enterprises are able to take advantage of the investments that they have made in networking capacity.

In addition to freeing strained networking bandwidth, Arista Peer Link adds link multi-chassis aggregation capability (MLAG). MLAG dramatically enhances network reliability and makes aggregate network bandwidth available to peer devices. The result is a network that is ready to support the most demanding business applications.

For more information, see the Arista Peer Link MLAG Overview at https://eos.arista.com/mlag-basic-configuration/.

Advanced Event Management (AEM) Simplifying the overall operations, AEM provides the tools to customize alerts and actions. AEM is a powerful and flexible set of tools to automate tasks and customize the behavior of EOS and the operation of the overall data center switching infrastructure. AEM allows operators to fully use the intelligence within EOS to respond to real-time events, automate routine tasks, and automate actions based on changing network conditions.

• Arista Data Analyzer (DANZ), sFlow, and multi-port mirroring to detect micro-burst congestion and provide network-wide visibility and monitoring to fully use the intelligence within EOS to respond to real-time events, automate routine tasks, and automate actions based on changing network conditions.

• Precise data analysis.

Arista Latency Analyzer (LANZ) is an integrated feature of EOS. LANZ provides precise real-time monitoring of micro-burst and congestion events before they impact applications. It identifies the sources and capture affected traffic for analysis.

For information on the Arista 7050SX switch, refer to the Arista website: https://www.arista.com/en/products/7050x-series

Cisco Nexus 9300 Series Switches For information on the Cisco Nexus 93180YC-FX switch, refer to the Cisco website: https://www.cisco.com/c/en/us/products/switches/nexus-93180yc-fx-switch/index.html

For information on the Cisco Nexus 9396PX switch, refer to the Cisco website: http://www.cisco.com/c/en/us/products/switches/nexus-9396px-switch/index.html

Out of band Management Networking – HPE, Arista, and Cisco Nexus switches All components, HPE Synergy Frame Link Modules, HPE ProLiant DL iLO 5 ports, SAN switch management ports, HPE 3PAR StoreServ Storage management ports, and so on, must be accessible through the same management network as the rest of the solution management stack (HPE OneView, VMware, and so on). Several pieces of the software stack must be able to communicate with these out-of-band management components for the solution to function properly.

Although it is not necessary to purchase dedicated out-of-band management switches for this solution, if an existing out-of-band network infrastructure is not available, you do not have enough open ports on your out-of-band network, or want a dedicated out-of-band management network for this solution, the following switches have been verified on an HPE Converged Solution 750:

• HPE FlexFabric 5900AF 48G 4XG 2QSFP+

– For information on the HPE FlexFabric 5900AF 48G 4XG 2QSFP+ switch, refer to the Hewlett Packard Enterprise website: https://www.hpe.com/us/en/product-catalog/networking/networking-switches/pip.hpe-flexfabric-5900af-48g-4xg-2qsfpplus-switch.5354511.html.

• HPE FlexFabric 5700 32XGT 48G 2QSFP+

– For information on the HPE FlexFabric 5900AF 48G 4XG 2QSFP+ switch, refer to the Hewlett Packard Enterprise website: https://www.hpe.com/us/en/product-catalog/networking/networking-switches/pip.hpe-flexfabric-5700-32xgt-8xg-2qsfpplus-switch.6638107.html.

https://www.arista.com/en/products/7050x-series

https://www.cisco.com/c/en/us/products/switches/nexus-93180yc-fx-switch/index.html

http://www.cisco.com/c/en/us/products/switches/nexus-9396px-switch/index.html

https://www.hpe.com/us/en/product-catalog/networking/networking-switches/pip.hpe-flexfabric-5900af-48g-4xg-2qsfpplus-switch.5354511.html

https://www.hpe.com/us/en/product-catalog/networking/networking-switches/pip.hpe-flexfabric-5900af-48g-4xg-2qsfpplus-switch.5354511.html

https://www.hpe.com/us/en/product-catalog/networking/networking-switches/pip.hpe-flexfabric-5700-32xgt-8xg-2qsfpplus-switch.6638107.html

https://www.hpe.com/us/en/product-catalog/networking/networking-switches/pip.hpe-flexfabric-5700-32xgt-8xg-2qsfpplus-switch.6638107.html


• Arista 7010T 48T 4SFP+

– For information on the Arista 7010T switch, refer to the Arista website: https://www.arista.com/en/products/7010-series.

• Cisco Nexus 3048TP

– For information on the Cisco Nexus 3048TP, refer to the Cisco website: https://www.cisco.com/c/en/us/products/switches/nexus-3048-switch/index.html.

Note that while the 10/40GbE network switches have the option for 1Gb RJ45 SFP transceivers, the cost per port might not make using that approach cost-effective and those transceivers do not communicate with any component that is 10/100Mb, which components in the solution might need.

Storage Networking – HPE StoreFabric B-Series Fibre Channel Switch When using a fabric attached Fibre Channel topology, this solution supports a redundant set of 16Gb or 32Gb capable HPE StoreFabric B-Series Fibre Channel switches.

HPE StoreFabric SN6600B Fibre Channel Switch The HPE SN6600B Fibre Channel Switch meets the demands of hyper-scale virtualization, larger cloud infrastructures, and growing flash-based storage environments by delivering market-leading 32Gb Fibre Channel technology and capabilities. It provides a high-density building block for increased scalability, designed to support growth, demanding workloads, and data center consolidation in small to large-scale enterprise infrastructures. It is built for maximum flexibility, scalability, and ease of use. Organizations can scale from 24 to 64 ports with 48 SFP+ and 4 QSFP ports, all in an efficient 1U package. It also provides a simplified deployment process and a point-and-click user interface-making it both powerful and easy to use. With the SN6600B Switch organizations gain the best of both worlds: high-performance access to industry-leading storage technology and "pay-as-you-grow" scalability to support an evolving storage environment.

Figure 20. HPE StoreFabric SN6600B Fibre Channel Switch

Key Features and Benefits

• Delivers 32 Gbps performance with up to 64 ports in an energy-efficient, 1U form factor, providing maximum flexibility for diverse deployment and cooling strategies.

• Features Ports on Demand (PoD) capabilities for fast, easy, and cost-effective scaling from 24 to 48 ports in 12-port increments along with a 16-port QSFP PoD for full 64-port support.

• HPE StoreFabric SN6600B is available in both bundled (32Gb SFP+ included) and unbundled variants providing flexibility to customers.

• Support high-density server virtualization, cloud architectures, and flash-based storage environments.

• Provides proactive, non-intrusive, real-time monitoring and alerting of storage IO health and performance with IO Insight, the industry's first integrated network sensors.

• Increases resiliency by automatically discovering and recovering from device or network errors.

• Simplifies troubleshooting with real-time and historical visibility in a single dashboard.

• Provides a flexible, simple, and easy-to-use SAN solution with industry-leading technology.

• Supports highly virtualized, flash storage with multi-tenancy and nonstop operations.

• Offers best-in-class port density and scalability for midrange enterprise SAN switches, along with redundant, hot-pluggable components and nondisruptive software upgrades.

https://www.arista.com/en/products/7010-series

https://www.cisco.com/c/en/us/products/switches/nexus-3048-switch/index.html

https://www.cisco.com/c/en/us/products/switches/nexus-3048-switch/index.html


• In-flight compression included providing efficient link utilization.

• Yields exceptional price/performance value, exceeding comparable Ethernet storage-based alternatives.

Network sizing and planning When planning or growing the HPE Converged Solution 750, make sure that you take port counts into account to be sure that you have the proper Ethernet and SAN networking. As you grow and need to add new switches, switches always must be expanded in pairs to provide redundancy throughout the infrastructure.

When planning connectivity for your HPEN IRF link connections, a minimum of two (2) x 100GbE, two (2) 40GbE or four (4) 10GbE connections must be used for each switch. It is possible to place up to eight (8) switches of the same family in the same IRF domain in the HPE Converged Solution 750. When having more than two switch members in the IRF domain, it is required to configure the IRF in a loop topology to prevent a single switch failure from bringing down the fabric. For larger configurations, it is recommended to implement a spine/leaf configuration. You must use MLAG to connect to the spine switches in the solution through a minimum of four (4) 40Gbe or 100GbE ports (two per switch).

When planning connectivity for your Arista Peer link or Cisco vPC Peer link connections, a minimum of two (2) x 100GbE, two (2) 40GbE or four (4) 10GbE connections must be used. You can add switches (in pairs) and put them into a new MLAG domain with redundant links back to the first set of Arista switches in the solution. For larger configurations, it is recommended to implement a spine/leaf configuration. You must use MLAG to connect the additional pair of switches to the primary or spine switches in the solution through a minimum of four (4) 40Gbe or 100GbE ports (two per switch).

Below is an example configuration of an Ethernet based spine/leaf design using Arista 7160 series switches. There are four sets of Arista 7160-48CY6 switches (leaf switches), each with several HPE Synergy Frames connected to them, connected to a pair of Arista 7160-32CQ switches (spine switches) via eight 100GbE Links. The interconnectivity between leaf and spine could be one of several layer 2 or layer 3 multi-pathing technologies.

Figure 21. Example of an Arista based spine/leaf topology

If using a fabric attached Fibre Channel topology, minimum of two (2) SAN switches are required. If more than two switches are needed in the solution, switches of the same vendor/type must be added in pairs so there is always an even number of switches. It is important to ensure that you select the correct model switch for your future growth plans. Select a bigger switch with only the ports you need licensed to start if you plan on expanding your HPE Converged Solution 750 in the future.


Figure 22. Example of a four SAN switch topology in a HPE Converged Solution

If you are planning to expand the solution past the capacity of four SAN switches, there are several SAN topologies that could be leveraged. An edge-core or an edge-core-edge topology is recommended if all HPE Synergy Compute modules need to be able to talk to all storage arrays. When planning connectivity for your Inter Switch Links (ISL) connections, a minimum of two (2) x 16/32/128GB connections must be used for each switch. The actual number of connections needed will depend on the over subscription ratio your solution can accept. • Edge-core, separates the storage and servers. Compute connects to the SAN fabric via edge switches, where storage is connected directly to

the core switches, resulting in a maximum of 1 hop between fabric switches to connect servers to storage. This approach provides an ease of management and moderate scalability. (Figure 23)

Figure 23. Example of edge–to-core SAN switch topology in a HPE Converged Solution.

• Edge-core-edge, has both storage and servers on edge. Compute and storage connects to the SAN fabric via edge switches, resulting in a maximum of 2 hop between fabric switches to connect servers to storage. This approach provides an ease of management and is much more scalable. (Figure 24)


Figure 24. Example of edge-core-edge SAN switch topology in a HPE Converged Solution.

For more information see the SAN Design and Best Practices white paper from Brocade (http://www.brocade.com/content/dam/common/documents/content-types/whitepaper/brocade-san-design-best-practices-wp.pdf).

Verified, allowed, and required component design criteria HPE Converged Solution 750 provides the flexibility to choose from several networking vendors and connectivity options. Certain options are either verified or allowed, and some are required per this HPE Converged Solution 750 Design Guide.

Every HPE Converged Solution 750 configuration requires a base configuration of two 10GbE or greater Ethernet network switches for compute level traffic. As the solution scales, compute switches can be added in pairs to accommodate the additional network connectivity.

Note: All switches must have all fan and power slots populated. See the QuickSpecs or datasheets for each switch type for supported options.

Connections to the HPE network adapters and HPE Synergy fabric modules must be cables and transceivers supported in the QuickSpecs of the network adapter or HPE Synergy fabric module.

Table 4. HPEN networking 10/40GbE Verified and Allowed switches. HPEN Verified or Allowed Required Components Optional components

HPEN FlexFabric 5940 48SFP+ 6QSFP+ Verified • 2 x X711 (port to power) or X712

(power to port) fan trays

• 2 x 650W power supplies

• All modules, Ethernet transceivers, and cable options supported by switch

• Active Copper or Optical Direct attached cables are not supported

• Note FC SFP+ transceivers and use of FCoE in the converged switch are not tested and thus not verified or allowed in the HPE Converged Solution

.

HPEN FlexFabric 5940 48SFP+ 6QSFP28 Verified

HPEN FlexFabric 5940 32QSFP+ Verified

HPEN FlexFabric 5940 2-slot 2QSFP+ Verified

HPEN FlexFabric 5940 4-slot Verified • 2 x 5930-4slot(port to power) or (power to port) fan trays

• 4x 650W power supplies

http://www.brocade.com/content/dam/common/documents/content-types/whitepaper/brocade-san-design-best-practices-wp.pdf


Table 5. Arista networking 10/25/40/100GbE Verified and Allowed switches.

HPEN Verified or Allowed Required Components Optional components

Arista 7160 48SFP28 Verified

• N/A, all Fans and power supplies come with switch

• All licenses, Ethernet transceivers, and cable options supported by switch


Arista 7160 32QSFP28 Verified

Arista 7050X2 48SFP+ 6QSFP+ Verified

Arista 7050X2 96SFP+ 8QSFP+ Verified

Arista 7050X 48SFP+ 4QSFP+ Verified



Arista 7050X2 32QSFP+ Verified

Arista 7050X 32QSFP+ Verified

Arista 7050X3 32QSFP28 Allowed

Arista 7050X3 48SFP28 12QSFP28 Allowed

Table 6. Cisco networking 10/40GbE Verified and Allowed switches.


Cisco Nexus 9396PX Verified • Fully populated FAN tray (required

by Cisco)

• 2 x power supplies (required by Cisco)

• All modules, licenses, Ethernet transceivers, and cable options supported by switch


Cisco Nexus 9372PX Verified

Cisco Nexus 9332PQ Allowed

Cisco Nexus 93180CY-EX Verified

The following are the DAC cables tested between the NIC and HPE Synergy fabric Modules in the associated Deployment Guides.

Note: Ethernet Optical Transceivers are supported for “in solution” connectivity where a DAC cable is not long enough to reach a switch or component in the solution or switch interoperability is a challenge, for example with Cisco Networking. HPE X240 have been verified with all verified switches in the configuration and are fully supported by both HPEN and Arista networking switches. The DAC cables will function properly in a Cisco Nexus switch, but are reported as unsupported by the switch.

Note AOC cables are not allowed or verified and should not be used for switch to server connectivity unless specifically called out in both the NIC/HPE Synergy fabric module and the network switch QuickSpecs and/or datasheet.

Table 7. Verified 10/40GbE Direct Attached Copper (DAC) cables for Switch to server connectivity

Verified 10/40GbE Direct Attached Copper (DAC) cables for Switch to server connectivity

HPE FlexNetwork X240 40G QSFP+ QSFP+ 1, 3 and 5m Direct Attach Copper Cable

HPE FlexNetwork X240 40G QSFP+ to 4x10G SFP+ 1, 3, and 5m Direct Attach Copper Splitter Cable

Cisco 40GBASE-CR4 QSFP+ to four 10GBASE-CU SFP+ direct attached breakout cable assembly, 1, 3, and 5m passive*

Arista QSFP+ 4x10G SFP+ 0.5, 1, 2, 3 and 5m Direct Attach Copper Cable*

HPE FlexNetwork X240 10G SFP+ to SFP+ 0.65, 1.2, 3, 5 and 7m Direct Attach Copper Cable

* Cisco and Arista QSFP+ to 4x10G cables are only supported in vendor switch for 10GbE connectivity to server. It is not supported from the QSFP+ port in server/HPE Synergy fabric to the switch.


Please take note of the following when selecting the 10/40GbE cables you want to use:

• For links between Arista switches or Cisco switches, always use vendor-specific Arista or Cisco cables.

• For QSFP+ to QSFP+ or SFP+ to SFP+ connections from Arista or Cisco switches to HPE Synergy Virtual Connect modules or HPE ProLiant servers, use the verified HPE FlexNetwork Cables in Table 7. It is not supported to use third party cables to connect to HPE Synergy Virtual Connect modules unless they are specified in the HPE Synergy Virtual Connect module QuickSpecs.

– HPE FlexNetwork X240 DAC cables are fully supported by Arista unless using a 4 x 10GbE cables and inserting the 40GbE end into the Arista switch, that configuration is not supported. It is supported to connect the 10GbE ends of a 4 x 10GbE cable into a Arista switch.

• When using 40Gb QSFP to 4 x 10 SFP cables for Synergy Virtual Connect Modules to ToR switch, use HPE FlexNetwork X240 splitter DAC cables listed in in Table 7.

– Cisco and Arista QSFP+ to 4x10G cables are only supported in vendor switch for 10GbE connectivity to server. It is not supported from the QSFP+ port in server/HPE Synergy fabric to the switch.

• At this time it has not been verified to connect to HPE Nimble controllers with DAC cables when using iSCSI. It has been verified to use optical transceivers which will help to avoid interoperability challenges. It is allowed and supported by HPE Nimble arrays to use DAC cables that meet the following specification:

– Passive 10GbE Direct Attached twinaxial cable less than 10 meters are supported

• All cables verified in the tables above are passive.

• Active Direct Attached twinaxial are not supported for switch to HPE Synergy and/or HPE ProLiant DL connectivity.

Table 8. Verified 25GbE Direct Attached Copper (DAC) cables for Switch to HPE ProLiant DL connectivity

Verified 25GbE Direct Attached Copper (DAC) cables for Switch to server connectivity

Arista QSFP28 4x25G SFP+ 1, 2, 3, and 5m Direct Attach Copper Splitter Cable

Arista 25G SFP28 to SFP28 1, 2, 3, and 5m Direct Attach Copper Cable

HPE X240 25G SFP28 to SFP28 1 and 3m Direct Attach Copper Cable

25GbE SFP28 cables and 100Gb QSFP28 to 4 x 25GbE SFP28 are only verified and supported when using HPE ProLiant DL based servers with the HPE Ethernet 10/25G 2-port 640FLR and or 640SFP28 network cards. At this time, the only 25GbE switch that has been verified is the Arista 7160.

Although it is allowed and fully supported to use an existing 1GbE network infrastructure, Hewlett Packard Enterprise recommends implementing a dedicated 1GbE out of band management (oobm) network for the solution.

Table 9. HPEN networking 10/100/1000MB Verified and Allowed switches. HPEN Verified or Allowed Required Components Optional components

HPE FlexFabric 5900AF 48G 4XG 2QSFP+ Switch Verified • 2 x port to power or power to port fan trays

• 2 x 650W power supplies

• All Ethernet transceivers, and cable options supported by switch. HPE FlexFabric 5700 48G 4XG 2QSFP+ Switch Verified

Table 10. Arista networking 10/100/1000MB Verified and Allowed switches. HPEN Verified or Allowed Required Components Optional components

Arista 7010T 48T 4SFP+ Verified

• N/A, all fans and power supplies come with switch

• All licenses, Ethernet transceivers, and cable options supported by switch.

Arista 7020R 48T 6SFP+ Allowed

Arista 7020RA 48T 6SFP+ Allowed

Table 11. Cisco networking 10/100/1000MB Verified and Allowed switches.



Cisco Nexus 3048TP Verified • 2 x 400W power supplies • All licenses, Ethernet transceivers,

and cable options supported by switch.

If using a fabric attached Fibre Channel topology, certain options are either verified or allowed, and some are required per this HPE Converged Solution 750 Design Guide. The HPE Converged Solution 750 requires a minimum of two (2) SAN switches. If more than two (2) switches are needed in the solution, switches of the same vendor/type must be added in pairs so there is always an even number of switches.

NOTE: Ensure that you select the correct model switch for your future growth plans. Select a bigger switch with only the ports you need licensed to start if you plan on expanding your HPE Converged Solution 750 in the future.

Note: HPE SmartSAN is not supported as HPE OneView manages the HPE 3PAR StoreServ storage and SAN fabric.

Table 12. HPE StoreFabric B-Series Fibre Channel verified and allowed switches. HPEN Verified or Allowed Required Components Optional components

HPE SN3000B Fibre Channel Switch Verified • 2 x HPE SN3000B Power Supplies

• All licenses, 8/16/32Gb Fibre Channel transceivers, and cable options supported by switch.

• 10Gb transceivers are not supported

HPE SN6000B Fibre Channel Switch Verified

• All power supplies and fans included



HPE StoreFabric SN8000B 4-slot Allowed

HPE StoreFabric SN8600B 4-slot Allowed

HPE StoreFabric SN8000B 8-slot Allowed • Four power supplies for N+N redundancy HPE StoreFabric SN8600B 8-slot Allowed

Storage All storage devices verified and allowed for use in an HPE Converged Solution 750 environment must have features that provide resiliency against hardware and cabling failures. They must also be upgradable online and provide flexibility to meet the performance and capacity needs of different workloads that the customer might put on the HPE Converged Solution 750.

HPE 3PAR StoreServ Storage Hewlett Packard Enterprise leveraged HPE 3PAR StoreServ Storage to meet the architectural guidelines of an HPE Converged Solution 750 to eliminate any single point of failure (hardware or software) in the system. To mitigate single points of failure at the hardware layer, the system is designed with redundant components, including redundant power domains. At a minimum, there are two controller nodes and two copies of the HPE 3PAR Operating System, even in the smallest system configuration.

To further the resiliency of HPE 3PAR StoreServ Storage, nodes are each powered by two (1+1) redundant power supplies and backed up by a string of two batteries. Each battery has sufficient capacity to power the controller nodes long enough to save all necessary data in memory into the local physical drive. Although many architectures use “cache batteries,” they are not suitable for extended downtimes that are usually associated with natural disasters and unforeseen catastrophes.

To further mitigate downtime due to component failure, HPE 3PAR Persistent Cache is a resiliency feature built into the HPE 3PAR Operating System that enables graceful handling of an unplanned controller failure or planned maintenance of a controller node. This feature eliminates the substantial performance penalties that are associated with traditional modular arrays and the cache “write-through” mode they have to enter under certain conditions. For the HPE 3PAR StoreServ 8000 4-node array, any node that loses its adjacent node can dynamically form a mirrored cache relationship with another storage controller node.

HPE 3PAR Persistent Ports is another persistent technology that enables a nondisruptive environment (from the host multipathing point of view) where host-based multipathing software is not required to maintain server connectivity in the event of a node down or link down condition


on any SAN fabric. This applies to firmware upgrades, node failures, and node ports that are taken offline either administratively, or as the result of a hardware failure in the SAN fabric that results in the storage array losing physical connectivity to the fabric.

From a host standpoint, connections to HPE 3PAR StoreServ Storage systems continue uninterrupted with all I/O being routed through a different port on the array. This helps you achieve an uninterrupted service level for applications running on HPE 3PAR StoreServ Storage systems.

Host Server

0:1:1

1:1:1

1:1:1

0:1:1

Prim

ary

HPE

3PAR

Por

t

SAN Fabric

HPE 3PAR Node 1HPE 3PAR Node 0

Port Identity: NativeGuest

Backup HPE 3PAR Port

Figure 25. Logical diagram of a Persistent Ports configuration

Although these storage devices have been tested with a particular firmware version, an HPE Converged Solution 750 configuration is fully supported by any subsequent version of the firmware unless specifically stated otherwise on HPE SPOCK, the HPE OneView Support Matrix, VMware Compatibility Guide, or the HPE Converged Solution 750 Deployment Guide.

At this time, only Fibre Channel connectivity through 16Gb ports is permitted.

Storage sizing HPE 3PAR StoreServ Storage has several sizing options and features. Customers can choose models by matching hardware characteristics to their business needs.

Storage provisioning Provisioning a HPE 3PAR StoreServ storage requires understanding of what the customer workload is and the characteristics of data that will stored on the array. In alignment with the HPE 3PAR Storage best practices, the HPE Converged Solutions 750 has recommends all common provisioning groups (CPG) created by default are protected by a RAID 6. This protection level applies for both all flash and hybrid arrays with and all storage tiers.

In order to provide a different service and protection level than the production volumes, a separate CPG is created for the management volumes. The default recommendation is that all productions volumes are thinly provisioned. Depending on the array being used, the workload and data characteristics, it might be possible to gain better storage efficiencies by enabling dedupe and/or compressed. Some customers with high level of duplicate or repeating data may benefit from using a dedupe-enabled volumes and see increased efficiencies with compression enabled. Work with a HPE 3PAR Certified Storage specialist to see what is best for your deployment.


HPE 3PAR StoreServ arrays provides the option of multi-tier storage. The choice between a 3-tier storage array and a 2-tier storage array is highly dependent on customer’s requirements in terms of capacity, response time and data lifecycle. There are some cases where more CPGs maybe required. For example, in the case of using thin virtual volumes in Adaptive Optimization, using virtual domains, and when capacity reporting is required per customer/application CPGs. A single copy CPG is created by default to minimize the number of CPGs created, and it is associated with every CPG to allow the use of array-based snapshots. However, when snapshots are heavily used and the snapshots data is kept in a different tier than the source data, HPE recommends using a copy CPG that matches the production CPG performance characteristics to maximize performance. When booting from a compute server from the 3PAR array (Boot from SAN (BfS)), a separate boot volume is created for each server for the sole purpose as the boot disk for that host. They each boot LUN must be exported as LUN ID 0 as recommended by 3PAR best practices.

For further information regarding 3PAR best practices, refer to HPE 3PAR StoreServ Storage A reference and best practice guide for HPE 3PAR StoreServ is available at the following link Storage https://h20195.www2.hpe.com/v2/GetPDF.aspx/4AA4-4524ENW.pdf

Storage cluster performance In any deployment of HPE 3PAR StoreServ Storage and in an HPE Converged Solution, a configuration of one HPE 3PAR StoreServ Storage system with at least two nodes is necessary. A wide variety of controller options is available, depending on business requirements. A remote office might require a smaller footprint and run lighter workloads; therefore, an 8200 model might be a good fit for the environment. For large data centers that require a substantial amount of horsepower or capacity, an 8440 model might be a better fit. For environments looking to deploy a VDI workload or having a workload optimized for an all-flash array, an 8450 or 9450 model might be best suited. Table 13 lists the hardware specifications for the appropriate controller models for the 8000/9000 series.

Table 13. Controller hardware specifications

Controller Model

Maximum Raw HA Capacity

Maximum Number of Disks

Maximum number of Solid State Disks

Processor Type

Maximum Onboard FC Ports

Number of PCI-E Expansion Slots

8200 750TiB 240 120 2 x 6-core 2.2GHz

12 (16Gb) 2

8400 2400TiB 576 240 2-4 x 6-core 2.2GHz

24 (16Gb) 2-4

8440 3000TiB 960 480 2-4 x 10-core 2.4GHz

24 (16Gb) 2-4

8450 1843TiB N/A 480 2-4 x 10-core 2.4GHz

24 (16Gb) 2-4

9450 6000Tib N/A 576 4-8 x 10-core 2.4GHz

80 (16Gb) 10-20

Designing storage involves striking a balance between the throughput and response time needed to support the expected load on the storage subsystems. The following steps review the process of determining the I/O performance and throughput requirements for a customer’s business needs.

1. Determine the storage performance (IOPs) and throughput (GB/s) required for business need.

2. Determine the disk type (SSD, FC, NL), the capacity required for growth, and the protection level of RAID configuration suitable for your mix loads.

3. Use the number of IOPs that are required to support the workload to calculate the distribution of disks and the number of controller nodes.

HPE Nimble Storage HPE Nimble Storage enables enterprise IT organizations to implement a single architectural approach to dynamically cater to the needs of varying workloads, driven by the power of predictive analytics. Predictive Flash arrays are the only storage platform that optimizes across performance, capacity, data protection, and reliability within a dramatically smaller footprint.

HPE Nimble Storage arrays are built upon HPE NimbleOS and HPE InfoSight™, the company’s cloud-connected predictive analytics and management system. HPE NimbleOS scales performance and capacity seamlessly and independently. HPE InfoSight leverages the power of deep data analytics to provide customers with precise guidance on the optimal approach to scaling flash, CPU, and capacity around changing application needs, while ensuring peak storage health.


The HPE Nimble Storage All Flash Arrays The HPE Nimble Storage All Flash (AF) series array family starts at the entry level with the AF20 and AF20Q, then expands up to the AF40, AF60, and then finally, at the high end, to the AF80. All flash arrays can be nondisruptively upgraded from the entry level, all the way to the high-end array model.

Table 14. HPE Nimble AF hardware specifications

Controller Model Maximum Raw Capacity

Maximum Usable Capacity1

Effective Capacity2

Max Expansion Shelves

Maximum IO (iSCSI/FC) Ports

Network Adapter Slots

AF20Q 46TB 25TB 128TB 1 16 4

AF20 46TB 33TB 168TB 1 16 4

AF40 184TB 136TB 682TB 1 24 6

AF60 553TB 407TB 2037TB 2 24 6

AF80 1106TB 815TB 4075TB 2 24 6

Scale-out 4xAF80 4423TB 3260TB 16303TB 8 96 24

Each 4U chassis supports 48 SSD drives with additional flash capacity available through expansion shelves.

For additional information, refer to: https://www.hpe.com/us/en/product-catalog/storage/disk-storage/pip.nimble-storage-all-flash-arrays.1009969160.html

.

Front View

Back View

Figure 26. The HPE Nimble Storage All Flash Array Family

HPE Nimble Storage Adaptive Flash Arrays The HF series array family starts at the entry level with the HF20H, HF20, and HF20C, then expands up to the HF40, and then finally, at the high end, to the HF60. Adaptive flash arrays can be nondisruptively upgraded from the entry level, all the way to the high-end array model.

Table 15. HPE Nimble HF hardware specifications


Maximum Flash Capacity


Effective Capacity4




HF20 210TB 23TB 169TB 845TB 6 16 4

HF20H 211TB 15TB 164TB 821TB 6 16 4

1 Usable Capacity is calculated based on RAID level Triple+ Parity 2 Assumes data reduction of five to one from deduplication and compression. Actual data reduction can be more or less depending on workload. 3 Usable Capacity is calculated based on RAID level Triple+ Parity 4 Assumes data reduction of five to one from deduplication and compression. Actual data reduction can be more or less depending on workload.

https://www.hpe.com/us/en/product-catalog/storage/disk-storage/pip.nimble-storage-all-flash-arrays.1009969160.html

https://www.hpe.com/us/en/product-catalog/storage/disk-storage/pip.nimble-storage-all-flash-arrays.1009969160.html



Maximum Flash Capacity


Effective Capacity4




HF20C 1260TB 23TB 1016TB 2032TB 6 24 6

HF40 504TB 23TB 406TB 2030TB 6 24 6

HF60 1260TB 23TB 1016TB 5080TB 6 24 6

Scale-out 4xHF60 5040TB 92TB 4065TB 20324TB 24 96 24

Each 4U chassis supports 21 HDDs with 6 SSDs (ratio of HDD to SSD is configurable), with additional HDD and flash capacity available through expansion shelves. The array has 24 drive bays. A single HDD occupies a single drive bay while the flash drives use the dual drive carrier. The dual drive carrier allows up to two flash drives to be installed in a single bay (each flash drive can be individually removed).

For additional information, refer to: https://www.nimblestorage.com/technology-products/adaptive-flash-arrays.

Front View

Back View

Figure 27. HPE Nimble Adaptive Flash Array

HPE NimbleOS Architecture The HPE Nimble Storage operating system, HPE NimbleOS is based on its patented CASL™ architecture. CASL is a flash friendly architecture that is pointer-based and natively handles variable block sizes, which minimizes write amplification and write wear on SSDs. HPE NimbleOS and HPE InfoSight™ form the foundation of the Predictive Flash platform, which allows for the dynamic and intelligent deployment of storage resources to meet the growing demands of business-critical applications.

In-Line Compression HPE NimbleOS uses fast, in-line compression for variable application block sizes to decrease the footprint of inbound write data by as much as 75 percent. Once there are enough variable-sized blocks to form a full write stripe, HPE NimbleOS writes the data to SSD (note that, the write is acknowledged once placed in NVDIMM). All reads will come from SSD and as a result will be very fast. Written data is protected with triple-parity RAID.

In-Line Deduplication HPE Nimble Storage all flash and adaptive flash arrays include in-line data deduplication in addition to in-line compression. The combination of in-line deduplication and in-line compression delivers the most comprehensive data reduction capability that allows HPE NimbleOS to minimize the data footprint, maximize usable space, and greatly minimize write amplification.

Thin-Provisioning and Efficient Capacity Utilization Capacity is consumed as data gets written. HPE NimbleOS efficiently reclaims free space on an ongoing basis, preserving write performance with higher levels of capacity utilization. This avoids fragmentation issues that hamper other architectures.

Acknowledging Writes at Memory Speed Once writes are placed in NVDIMM (made persistent and mirrored to the passive partner controller), they are acknowledged back to the host and sent to SSD at a later time (generally when there is a full stripe to be written). As a result, writes to an HPE Nimble Storage array are acknowledged at memory speeds.

Maximizing Flash Write Cycles By sequencing random write data, HPE NimbleOS sends full stripes of data to SSD. By compressing and deduplicating the data in-line (versus post-process), the data footprint on SSD as well as the block changes are kept to an absolute minimum. Additionally, the data being sent to disk is of variable block size, which means HPE NimbleOS does not have to break up data into smaller, fixed-sized chunks, to be placed on SSD. As a

https://www.nimblestorage.com/technology-products/adaptive-flash-arrays


result, data is efficiently sent to SSD. Efficient data chunk transfer allows HPE Nimble Storage arrays to maximize the deployable life of a flash drive by minimizing write wear on the flash cells. This is demonstrated by the use of the TLC SSDs in HPE Nimble Storage arrays, which are covered by a seven-year warranty.

Read Performance HPE NimbleOS and all flash arrays deliver submillisecond read latency and high throughput across a wide variety of demanding enterprise applications as all reads come from SSD.

Efficient, Fully Integrated Data Protection All-inclusive snapshot-based data protection is built into the Adaptive Flash platform. Snapshots and production data reside on the same array, eliminating the inefficiencies inherent to running primary and backup storage silos. And, HPE InfoSight ensures that customers’ data protection strategies work as expected through intuitive dashboards and proactive notifications in case of potential issues.

SmartSnap: Thin, Redirect-on Write Snapshots HPE Nimble Storage snapshots are point-in-time copies capturing just changed data, allowing months of frequent snapshots to be easily stored on a single array. Data can be instantly restored, as snapshots reside on the same array as primary data.

SmartReplicate: Efficient Replication Only compressed, changed data blocks are sent over the network for simple and WAN-efficient disaster recovery.

Zero-Copy Clones HPE Nimble Storage snapshots allow fully functioning copies, or clones of volumes, to be quickly created. Instant clones deliver the same performance and functionality as the source volume, an advantage for virtualization, VDI, and development/test workloads.

Application-Consistent Snapshots HPE Nimble Storage enables instant application/VM-consistent backups using the VSS framework and VMware integration, using application templates with pretuned storage parameters.

SmartSecure: Flexible Data Encryption HPE NimbleOS enables individual volume level encryption with little to no performance impact. Encrypted volumes can be replicated to another HPE Nimble Storage target, and data can be securely shredded at the volume level of granularity.

REST API HPE NimbleOS has a RESTful API that allows for powerful, secure, and scriptable management of storage objects. Using this API, an administrator can use the scripting or orchestration tool of their choice to interact with infrastructure components in a well-defined, repeatable manner.

Scale-to-Fit The HPE Nimble Storage Predictive Flash platform provides for application growth by scaling performance, capacity, or both – efficiently and non-disruptively. With HPE Nimble Storage scale-to-fit, organizations can:

• Scale capacity by adding expansion shelves. For example, the AF40 array can support an effective capacity of up to 682TB of flash capacity through the addition of expansion shelves while the HF40 array can support an effective capacity of up to 2PB

• Scale up performance by upgrading compute for greater throughput and IOPS

• Scale capacity and performance together by clustering any combination of HPE Nimble Storage all flash or adaptive flash arrays – see next section for the HPE Nimble Storage scale-out capabilities

Scale-Out HPE Nimble Storage arrays feature a scale-out architecture that makes it easy to scale capacity and performance beyond the physical limitations of a single array. With HPE Nimble Storage scale-out, organizations can:

• Group up to four HPE Nimble Storage arrays (any model or series) for higher scalability and performance.

• One management console to administer all storage hardware resources in the group as a single entity.

• Dynamic load balancing across arrays in a group to eliminate performance hot spots.

• Multi-array data striping, enabling any application to fully leverage the collective hardware resources of the scale-out group.


• Flexible configuration of any combination of HPE Nimble Storage arrays in the group, maximizing storage ROI.

• Seamless reconfiguration and hardware refreshes, without downtime.

HPE InfoSight for HPE Nimble Storage and 3PAR Businesses today cannot afford any disruptions or delays to their applications. But the complexity of infrastructure causes an “App-Data Gap” that impacts businesses and wastes time. HPE is working to minimize disruptions and lost time through the AI and predictive analytics capabilities of HPE InfoSight, reducing the time spent troubleshooting issues by up to 85%1.1

Every second, HPE InfoSight analyzes millions of sensors across the globally connected installed base and uses this data to predict and prevent problems from storage to applications. HPE InfoSight ensures reliable performance and application uptime, helping to deliver greater than 99.9999% of guaranteed availability. 1

HPE InfoSight lays the groundwork for an autonomous data center with the new AI recommendation engine. Building on its predictive capabilities, HPE InfoSight now pre-emptively advises IT how to avoid issues, improve performance, and optimize available resources. The recommendations are based on advanced machine learning that leverages almost a decade of data science expertise and rich telemetry collected from more than 10,000 HPE Nimble Storage customers. HPE InfoSight continuously learns, making customers’ systems smarter and more reliable. Third-party research has recently shown that HPE InfoSight lowers storage IT OpEx by 79 percent.

Using systems modeling, predictive algorithms, and statistical analysis, InfoSight solves storage administrators’ most difficult problems. HPE InfoSight also ensures that storage resources are dynamically and intelligently deployed to satisfy the changing needs of business-critical applications, a key facet of the HPE Nimble Storage Predictive Flash platform. At the heart of HPE InfoSight is a powerful engine comprised of deep data analytics applied to telemetry data gathered from HPE Nimble Storage arrays deployed across the globe. More than 30 million sensor values are collected per day per HPE Nimble Storage array. HPE InfoSight automatically collects and analyzes storage performance every four hours using predictive analytics to correlate vast amounts of data to known issues to identify and fix problems before they become outages.

1 Source: Based on actual customer data collected by the HPE Nimble Storage Support organization. See also https://www.hpe.com/h20195/v2/Getdocument.aspx?docname=a00018503ENW The HPE InfoSight Engine transforms the millions of gathered data points into actionable information that allows customers to realize significant operational efficiency through:

• Maintaining optimal storage performance.

• Projecting storage capacity needs.

• Proactively monitoring storage health and getting granular alerts.

• Proactively diagnoses and automatically resolves complex issues, freeing up IT resources for value-creating projects.

• Ensures a reliable data protection strategy with detailed alerts and monitoring.

• Expertly guides storage resource planning, determining the optimal approach to scaling cache, IOPS to meet changing SLAs.

• Identifies latency and performance bottlenecks through the entire virtualization stack.

For more information, refer to: https://www.hpe.com/us/en/storage/infosight.html.

Hewlett Packard Enterprise has demonstrated the unique capabilities of HPE InfoSight in HPE Nimble Storage environments, where today 86% of issues are resolved automatically before the customer knows that a problem exists. By extending HPE InfoSight to HPE 3PAR, both HPE Nimble Storage and HPE 3PAR customers can now realize the significant benefits of HPE InfoSight.

This first release of HPE InfoSight for HPE 3PAR makes available the following capabilities:

• Cross-stack analytics. For HPE 3PAR customers running the latest release of the HPE 3PAR operating system (3.3.1 or later), HPE InfoSight will also provide IT the ability to resolve performance problems and pinpoint the root cause of issues between the storage and host virtual machines (VMs). It also provides visibility to locate “noisy neighbor” VMs.

• Global visibility. Through a new cloud portal that combines HPE Infosight and HPE StoreFront Remote, all current HPE 3PAR customers with systems that are remotely connected will see detailed performance trending, capacity predictions, health checks, and best practice information across all their HPE 3PAR arrays.

https://www.hpe.com/us/en/storage/infosight.html


• Foundation to enable predictive support. Analytics and automation infrastructure are now in place that in the future will be used to detect anomalies, predict complex problems, and route cases directly to Level 3 support.

HPE Synergy D3940 Storage Module The HPE Synergy D3940 Storage Module is a directed attached storage module with 40 Small Form Factor (SFF) drive bays designed for use in HPE Synergy 12000 Frames. Through the HPE Synergy 12Gb SAS Connection Module it provides composable storage for up to 10 compute modules in a single frame. In a HPE Converged Solution 750, the HPE Synergy D3940 storage is optimized for use as a software-defined storage using VMware vSAN.

Note: The HPE Converged Solution 750 architecture only supports the use of VMware vSAN on VMware vSphere 6.5 based clusters.

Figure 28. HPE Synergy D3940 Storage Module

HPE Synergy storage enables a variety of workloads by permitting multiple drive types to be configured in the same storage module. HPE Synergy D3940 Storage Modules support a family of 12G SAS or 6G SATA HDD and SSD Smart Drives. Storage modules connect to compute modules within a frame through the HPE Synergy 12Gb SAS Connection Module. Any number of drive bays in a storage module can be composed with any compute module containing n HPE Smart Array controller connected to the SAS fabric, allowing for efficient utilization of available drives. HPE Synergy storage can scale to 200 SFF drives with five storage modules in a single HPE Synergy 12000 Frame. Adding a second I/O adapter and second SAS connection module provides a redundant path to SAS drives inside the Storage Module, ensuring high availability. The modular design of the HPE Synergy D3940 Storage Module allows it to slide out from the frame to service drives or I/O adapters without interrupting operation of other drives within the module. Additionally, the HPE Synergy D3940 Storage Module is optimized for solid state drives using a high performance SAS connection with sixteen 12Gb/s SAS lanes. This allows a single HPE Synergy storage module to deliver as much as eight (8) times the bandwidth of other JBOD options reaching up to 2M IOPs.

Verified, allowed, and required component design criteria HPE Converged Solution 750 provides the flexibility of multiple storage options. External storage connectivity is supported via Fibre Channel on both the HPE 3PAR StoreServ and HPE Nimble Storage as well as iSCSI for HPE Nimble Storage. For in-frame software defined storage, the use of HPE Synergy D3940 with VMware vSAN is also available. It is required to host all management production virtual machine data on shared storage (HPE 3PAR StoreServ, HPE Nimble Storage and/or HPE Synergy D3940 with VMware vSAN) in this architecture.


To support multipathing with FC and iSCSI, all compute and management hosts must have two or more ports available from which you can reach the storage array. This means that multiple paths connect each server with the storage arrays hosting its virtual machine files. Each storage controller and server must be connected to two SAN fabrics. This method provides increased redundancy to make sure that the paths from the host to its storage LUNs are always available

HPE 3PAR StoreServ Storage For every HPE Converged Solution 750 configuration using the HPE 3PAR StoreServ storage, the following been verified and are required:

Note Although the HPE 3PAR StoreServ Storage supports file-based protocols such as NFS and SMB and block protocols such as iSCSI, those protocols have not been verified or allowed for use for VMware datastores in the HPE Converged Solution 750 at this time.

• One or more HPE 3PAR StoreServ 8000, 9000 or 20000 R2 Series arrays:

– All-Inclusive Single-System Software (Note: HPE Smart SAN is not supported).

– For all HPE 3PAR StoreServ 8000 series controllers, it is required to include a 4-port 16Gb Fibre Channel Adapter or a 4-port 16Gb Fibre Channel/10Gb NIC Combo Adapter.

• Note the use of RCIP or File Persona is allowed but has not been verified in this architecture at this time.

– For all HPE 3PAR StoreServ 9000 and 20000 R2 series controllers, it is required to include at least two (2) 4-port 16Gb Fibre Channel Adapters for host connectivity and at least two(2) 4-port 12Gb SAS Host Bus Adapters for drive connectivity.

– All SSD and HDD options that are supported by the array.

• The following optional components have been verified to be added to HPE 3PAR StoreServ storage:

– All other licenses associated with the HPE 3PAR storage of your choice.

– When using a physical HPE 3PAR StoreServ Service Processor, service processor with redundant PDUs must be used.

• A HPE 3PAR StoreServ Redundant Power Supply Service Processor is recommended in all configurations, although it is verified to use a HPE 3PAR StoreServ Virtual Service Processor as well.

• Please check the compatibility of the HPE 3PAR StoreServ Virtual Service Processor with the VMware version you plan to run on your management nodes. At the time that this document was written, the HPE 3PAR StoreServ Virtual Service Processor version 4.4.x which is required to be used with HPE 3PAR StoreServ 3.2.2 (used on the HPE 3PAR StoreServ 8000 series) is not certified with VMware 6.5. However HPE 3PAR StoreServ Virtual Service Processor version 5.x which is required to be used with HPE 3PAR StoreServ 3.3.1 (used on the HPE 3PAR StoreServ 8000 series) is certified with VMware 6.5.

• The following optional components are allowed to be added to the HPE 3PAR StoreServ storage but have not been verified:

– HPE 3PAR StoreServ File Controllers.

– For the HPE 3PAR StoreServ 9000 and 20000 R2 series controllers, one (1) or more 2-port 10Gb Ethernet Host Bus Adapter (for RCIP or File Persona).

HPE Nimble Storage For every HPE Converged Solution 750 configuration using HPE Nimble Storage, the following been verified and are required:

Note Although HPE Nimble Storage support both Fibre Channel and iSCSI, it is not supported to use both protocols on the same array at the same time at this time. It is supported to use two different arrays, each running a different protocol.

Also, at this time, the use of HPE Synergy Image Streamer and an iSCSI based HPE Nimble array connected to the same logical enclosure is not supported with VMware vSphere 6.0.


• One or more HPE Nimble Storage Adaptive Flash and/or All Flash arrays:

– If using the Fibre Channel protocol for storage connectivity, at least one (1) 2-port or 4-port 16Gb FC adapter is required per controller.

– If using the iSCSI protocol for storage connectivity, at least one (1) 2-port or 4-port 10Gb Ethernet adapter is required per controller.

– All SSD and HDD options that are supported by the array.

• The following optional components have been verified to add to the HPE Nimble Storage:

– It is allowed to add one (1) 2-port or 4-port port Ethernet adapter for remote replication when using either iSCSI or Fibre Channel connectivity.

HPE Synergy D3940 Storage For every HPE Converged Solution 750 configuration using HPE Synergy D3940 Storage modules with VMware vSAN, the following has been verified and are required:

• One or more HPE Synergy D3940 Storage modules:

– It is supported to put up to five (5) HPE Synergy D3940 Storage modules per frame.

– Each frame looking to use VMware vSAN must contain at least one HPE Synergy D3940 Storage module and two (2) HPE Synergy 12Gb SAS Connection Modules.

– To map storage from the HPE Synergy D3940 Storage to a HPE Synergy compute module, the compute module must include a HPE Smart Array P416ie-m SR Gen10 controller.

– Disk configuration must match those of a certified option in the VMware vSAN ready node list and/or VMware vSAN HCL. For more information review the VMware blog "What You Can (and Cannot) Change in a vSAN Ready Node ™ " section "allowable changes in Storage Sub-Systems of vSAN ReadyNode BOM".

The VMware Compatibility Guide listing for HPE vSAN Ready Node models can be found here: https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan&details=1&vsan_type=vsanreadynode&vsan_partner=515&keyword="Synergy Gen10"&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc.

Note: The HPE Converged Solution 750 architecture only supports the use of VMware vSAN on VMware vSphere 6.5 based clusters.

Hypervisor VMware vSphere 6.5 VMware vSphere is an optimal virtualization platform and enabler for cloud computing architectures. VMware vSphere enables IT to meet service level agreements (SLAs) for the most demanding business-critical applications at the lowest total cost of ownership (TCO). VMware vSphere delivers control over IT resources with the highest efficiency and choice in the industry, as shown in the following list.

VMware vSphere virtualization solutions provide for:

• Consolidation. VMware virtualization allows multiple application servers to be consolidated onto one physical server, with little or no decrease in the overall performance. This consolidation helps to minimize or eliminate underused server hardware, software, and infrastructure.

• Manageability. The live migration of virtual machines from server to server and the associated storage is performed with no downtime using VMware vSphere vMotion, which simplifies common operations, such as hardware maintenance and VMware vSphere Storage vMotion.

• Availability. You can enable high availability to reduce unplanned downtime and enable higher service levels for applications. VMware vSphere High Availability (HA) ensures that, in the event of an unplanned hardware failure, the affected virtual machines are automatically restarted on another host in a VMware cluster. vSphere 6.5 also allows creating dependency chains using VM-to-VM rules. These dependency rules are enforced when vSphere HA is used to restart VMs from failed hosts. This dependency availability is great for multi-tier applications that do not recover successfully unless they are restarted in a particular order.

https://blogs.vmware.com/virtualblocks/2017/03/14/can-cannot-change-vsan-readynode/

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan&details=1&vsan_type=vsanreadynode&vsan_partner=515&keyword=%22Synergy%20Gen10%22&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc.%20

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan&details=1&vsan_type=vsanreadynode&vsan_partner=515&keyword=%22Synergy%20Gen10%22&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc.%20


VMware vCenter High Availability was introduced in vSphere 6.5 and exclusively available for the vCenter Server Appliance (VCSA). When vCenter HA is enabled, a three-node vCenter Server cluster (Active, Passive, and Witness nodes) is deployed. vCenter HA provides a Recovery Time Objective (RTO) of about 5 minutes for vCenter Server greatly reducing the impact of host, hardware, and application failures with automatic failover between the Active and Passive nodes.

• Automation. VMware automated load balancing takes advantage of vMotion and Storage vMotion to migrate virtual machines among a set of VMware ESXi hosts. VMware vSphere Distributed Resource Scheduler (DRS) and VMware vSphere Storage DRS enable automatic resource relocation and optimization decisions for virtual machines and storage.

• Provisioning. VMware virtualization encapsulates an application into an image that can be duplicated or moved, greatly reducing the cost of application provisioning and deployment.

Note The HPE Converged Solution 750 architecture is validated using vSphere 6.5 and 6.0. Not all features in VMware vSphere 6.5 are supported in 6.0.

VMware vSphere has a range of features at different licensing levels. This solution supports the use of any of these license levels, but it is designed to use the features offered by the vSphere with Operations Management (vSOM) enterprise plus level license. This license includes vSphere Enterprise and vRealize Operations Manager (vROps) standard to strike a balance between cost and functionality. Optionally, customers can upgrade vRealize Operations Manager to the advanced or enterprise level.

VMware vSAN 6.6 VMware vSAN is a distributed layer of software that runs natively as a part of the ESXi hypervisor. vSAN aggregates local or direct-attached capacity devices via a HPE Synergy D3940 Storage Module of a host cluster and creates a single storage pool shared across all hosts in the vSAN cluster.

HPE Synergy is a perfect choice for VMware vSAN. A core value prop of VMware vSAN is that it aggregates all local storage disks – both SSD and HDD – available on the hosts (compute modules) into a single datastore shared by all compute modules. With traditional rack servers, compute and storage do not generally scale independently, but with HPE Synergy, you can scale storage and compute separately. This makes it a perfect fit for vSAN solutions allowing for storage capacity growth without requiring additional compute resources to host them.

VMware vSAN is licensed separately from VMware vSphere or vSOM. HPE recommends VMware vSAN Enterprise license edition for use within the solution. For more information on vSAN licensing, refer to the VMware vSAN licensing guide https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsan/vmware-vsan-66-licensing-guide.pdf

Key Benefits:

• Disaggregated storage and compute

– HPE Synergy with VMware vSAN provides the flexibility of thinly provisioning software defined storage volumes, independent of compute resources.

• High-Speed interconnect between frames

– HPE Synergy provides a flat east-west Ethernet topology. This allows iSCSI storage to be shared on a low latency server to server connection, including between HPE Synergy Frames.

• Single infrastructure for any workload

– HPE Synergy helps enterprises run vSAN in a single infrastructure, addressing the needs of both traditional and emerging business applications.

Note The HPE Converged Solution 750 architecture only supports the use of VMware vSAN on VMware vSphere 6.5 based clusters.

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsan/vmware-vsan-66-licensing-guide.pdf


The following considerations must be addressed when using VMware vSAN in a HPE Converged Solution 750:

• When using VMware vSAN, all hardware used for a vSAN deployment must be on the VMware HCL. All I/O Controllers, HDD and SSD must be on the vSAN HCL. VMware and HPE cannot provide support for environments running on uncertified hardware.

• A minimum of three ESXi hosts contributing local storage in each VMware vSAN cluster managed by VMware vCenter.

– All hosts must be managed by vCenter Server and configured as a vSAN cluster member.

– Hosts in a vSAN cluster may not participate in any other cluster.

• vSAN introduces no more than 10% of CPU overhead per host and requires a min of 32Gb of memory to accommodate for the maximum number of disk and disk groups. Ensure this guideline is taken into account when sizing your servers.

• Each host must have a minimum of a two physical 10GB or greater Ethernet NICs available for vSAN use.

– The NIC can be shared with other traffic if using a HPE DL rack based management server.

– When using HPE Synergy compute modules for management servers, FlexNIC "b" of the base network adapter is required to be used with a minimum of 8GB of bandwidth allocated to it.

– All vSAN hosts in the same cluster must be on the same layer 2 network for the vSAN traffic.

• For the caching storage tier, at least 2 SAS/SATA Solid State Drive (SSD) are required.

– The cache tier must provide at least 10 percent of the anticipated storage consumed on the capacity devices, not including replicas such as mirrors.

• For virtual machine data storage: Whether SSD or HDD based capacity tier, cluster configuration must have at least 2 SAS/SATA SSDs or HDDs.

• For HPE Synergy based compute modules participating in a VMware vSAN cluster, a HPE Smart Array P416ie-m SR Gen10 12G SAS Mezzanine Controller is required and must be configured in HBA mode to all disks on the HPE Synergy D3940 Storage Module that will be participating in the vSAN storage cluster.

– Local boot devices to the compute module must still be configured in RAID 1 and therefore cannot be serviced by the HPE Smart Array P416ie-m SR controller utilized for the vSAN storage cluster. Hewlett Packard Enterprise recommends utilizing the HPE Smart Array P204i-c SR controller for the local boot device.

– Flash based boot devices (USB, SSD, M.2) are not supported at this time in the HPE Converged Solution 750.

For additional information on the requirements of VMware vSAN independent of the HPE Converged Solution 750, please refer to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.virtualsan.doc/GUID-4B738A10-4506-4D70-8339-28D8C8331A15.html.

VMware vSphere sizing and solution requirements Each server in this solution runs VMware vSphere as the host hypervisor. The hypervisor is protected by the resiliency features built into the server as described the Computing section of this document. Additional configuration is required to protect the VMs within the hypervisor from failures within the physical server section of this document. Additional configuration is required to protect the VMs within the hypervisor from failures within the physical server.

Number of physical Ethernet network adapters per VMware vSwitch Within vSphere, multiple virtual switches (vSwitches) can exist. Each vSwitch can have multiple virtual networks and multiple VMkernel ports that act as virtual IP addresses for the vSphere host operating system. Each vSwitch must be associated with at least two physical adapters on the server that go to at least two different switches in a multi-chassis link aggregation configuration.

On the compute modules, multiple vSwitches are leveraged because of the HPE FlexNIC capabilities of the 20GbE HPE FlexFabric LOM adapters and HPE FlexFabric Synergy fabric. The NIC teaming feature in vSphere achieves both load balancing and high availability by actively controlling VM traffic. In the event of a NIC failure, traffic from VMs is migrated to the remaining active ports. Although this process might cause the VMs to operate in a degraded state, it provides an opportunity for the administrator or the system to migrate VMs to other hosts and resolve any underlying physical issues.

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.virtualsan.doc/GUID-4B738A10-4506-4D70-8339-28D8C8331A15.html

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.virtualsan.doc/GUID-4B738A10-4506-4D70-8339-28D8C8331A15.html


For more information about VMware NIC teaming, see the VMware Knowledge Base article NIC Teaming in ESXi and ESX at http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004088.

Segregated management, migration, high availability, and production networks It is a best practice for any virtualization environment to have separate networks/VLANs for management, migration (VMware vMotion), and high availability (VMware Fault Tolerance) traffic. The main reason for this architecture is to be sure that one network does not consume the network resources all at once on a server. Having all your networks on one virtual switch can cause VMs to operate less effectively or, in some cases, prevent VMs from operating entirely.

Within the HPE Converged Solution 750 architecture, all HPE Synergy compute modules must leverage HPE FlexFabric adapters to enable the creation of up to eight separate FlexNICs per physical port. The advantage of creating separate FlexNICs is that each FlexNIC can be assigned a preferred (guaranteed) amount of speed (as defined by the HPE OneView administrator) and be allowed to burst up to the full 20GbE if the network has available bandwidth that the other HPE FlexNICs are not using.

The result is that a HPE Synergy based HPE Converged Solution 750 can be architected with separate vSwitches for management, VMware vMotion, and fault tolerance; it can also support Fibre Channel or iSCSI based storage in a highly available fashion by using a single dual-port HPE FlexFabric network adapter.

Data locations VMware vSphere does not prevent an administrator from putting VM disks on storage that is not highly available. In the event of a server failure, VMs might become unrecoverable if they are not placed on highly available storage. To eliminate local disks or other storage options that are not highly available as points of failure, this solution requires that all virtual machine data be stored only on shared storage hosted on a highly available storage array.

For the management servers in the solution, the option to use VMware vSAN on the DL-based management servers is supported for configurations that require a dedicated storage solution for the solution management software stack but not the price tag of an external storage solution.

VMware High Availability and Distributed Resource Scheduler Although hypervisor-level high availability does not protect against server failure, it does mitigate the impact of such failures by allowing automatic recoverability. VMware vSphere High Availability automatically restarts a VM on a different server when it detects a server failure. To function properly, the VM data must reside on shared-storage datastores. This feature requires a minimum of two servers, as recommended in the HPE Converged Solution 750 solution.

In addition, the VMware vSphere Distributed Resource Scheduler (DRS) provides additional ease of management and VM protection by automatically distributing VMs across pooled server resources according to server load and capacity. This feature allows the administrator to provision new VMs without having to manually load-balance and risk-impact the performance of other VMs in the pool. Figure 29 illustrates how vSphere DRS redistributes VMs across ESXi servers when a server failure occurs.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004088


Figure 29. vSphere DRS

For more information about these vSphere features and benefits, see VMware vSphere High Availability at vmware.com/products/vsphere/features/high-availability and the VMware vSphere Distributed Resource Scheduler at vmware.com/products/vsphere/features/drs-dpm.

VMware vSphere and vCenter configuration limits When determining the size and the number of VMware ESXi hosts for your configuration, it is important to understand what the maximum configurations for the hypervisor are. Multiple vCenter environments increase complexity and might be better suited to having multiple HPE Converged Solution 750 instances in place.

Each individual VMware ESXi host and VMware vCenter environment as a whole has limits on the number of physical and virtual components that it can support. The following table lists some of the most relevant limits to this solution. The rest can be found in VMware’s Configuration Maximums for vSphere 6.5 (https://www.vmware.com/pdf/vsphere6/r65/vsphere-65-configuration-maximums.pdf) and vSphere 6.0 (https://www.vmware.com/pdf/vsphere6/r60/vsphere-60-configuration-maximums.pdf). If using VMware VSAN for storage, you must take note of the configuration maximums in the VMware vSAN configuration maximums section of this document as those values will override the maximums in this section.

Table 16. VMware configuration maximums

Resource vSphere 6.0 vSphere 6.5

RAM per host 6TB 12TB

VMs per host 1024 1024 (200 when using VSAN)

Virtual CPUs per host 4096 4096

Virtual CPUs per core 32 32

vSwitch VM network ports per host 4096 4096

LUNs per host 256 512

LUN size 64TB 64TB

Number of paths to a LUN 32 32

Number of total paths to a server 1024 2048

Physical 10GbE ports 8 (Emulex and QLogic), 4 (Mellanox) 8 (Emulex and QLogic), 4 (Mellanox)

http://www.vmware.com/products/vsphere/features/high-availability

http://www.vmware.com/products/vsphere/features/drs-dpm

https://www.vmware.com/pdf/vsphere6/r65/vsphere-65-configuration-maximums.pdf



Resource vSphere 6.0 vSphere 6.5

Hosts per HA cluster 64 64

Hosts per vCenter server 1,000 2,000

Powered-on VMs per vCenter server 10,000 25,000

VMware vSAN configuration maximums Each individual VMware ESXi host and VMware vCenter environment as a whole has limits on the number of physical and virtual components that it can support. The following table lists some of the most relevant limits to this solution. The rest can be found in VMware’s Configuration Maximums and vSphere 6.5 (https://www.vmware.com/pdf/vsphere6/r65/vsphere-65-configuration-maximums.pdf).

Table 17. VMware vSAN configuration maximums

Resource vSphere 6.5

Virtual SAN disk groups per host 5

Spinning disks in all diskgroups per host 35

Cache tier maximum devices per host 5

Capacity tier maximum devices per diskgroup 7

Capacity tier maximum devices 35

Number of Virtual SAN hosts in a cluster 64

Number of datastores per cluster 1

Virtual machines per host 200

Virtual machines per cluster 6000

Management software HPE Converged Solution 750 provides holistic management for both virtual and physical environments.

The default configuration of the HPE Converged Solution 750 solution includes two HPE Synergy 480 Gen10 Compute Modules or two HPE ProLiant DL360 Gen10 servers running VMware vSphere 6.0 or 6.5, deployed with a number of VMs. These VMs provide the management connectivity and software required to administer and manage the solution:

• HPE Synergy Composer, which is powered by HPE OneView

• HPE OneView (virtual appliance for HPE ProLiant Servers)

• VMware vCenter Server Appliance (virtual appliance)

• VMware vRealize Operations Manager (virtual appliance)

– HPE OneView for VMware vRealize Operations

• HPE OneView for VMware vCenter (virtual appliance)

• Solution management VM (Microsoft Windows Server® 2012 R2)

– Solution media server

– HPE B-series SAN Network Advisor Software

– HPE 3PAR Command Line Interface

– HPE Nimble Windows Toolkit



• HPE 3PAR StoreServ Management Console (virtual appliance)

• HPE Recovery Manager Central for VMware (virtual appliance)

• HPE 3PAR Virtual Service Processor (optional virtual appliance)

• HPE Nimble Connection Manager for VMware

• HPE Nimble Storage vCenter Plugins

• HPE OneView Global Dashboard (optional virtual appliance)

HPE OneView HPE Converged Solution 750 uses HPE OneView, a comprehensive single platform designed from the ground up for converged infrastructure management. An integrated platform increases the productivity of every member of the internal IT team across servers, storage, and networking. By streamlining processes, incorporating best practices, and creating a new holistic way to work, HPE OneView provides organizations with a more efficient way to work. It is designed for open integration with existing tools and processes to extend these efficiencies.

HPE OneView is instrumental for the deployment and management of Hewlett Packard Enterprise servers and frame networking in the HPE Converged Solution 750 solution. It collapses infrastructure management tools into a single resource-oriented architecture that provides direct access to the HPE OneView managed logical and physical resources of the solution. Logical resources include server profiles and server profile templates; frames and frame groups; and logical interconnects and logical interconnect groups. Physical resources include server hardware, HPE Synergy compute modules and rack servers; networking interconnects; and computing resources.

Note In the current release of the HPE Converged Solution 750 solution, HPE OneView does not manage the verified Hewlett Packard Enterprise, Arista, or Cisco network switches.

The HPE OneView converged infrastructure platform offers a uniform way for administrators to interact with resources by providing a RESTful API foundation. This integrated resource model removes the need for the administrator to enter and maintain the same configuration data more than once and keep all versions up to date. It encapsulates and abstracts many underlying tools behind the integrated resource model, so the administrator can operate with new levels of simplicity, speed, and agility to provision, monitor, and maintain the solution. The integrated resource model is critical for diagnosing problems or determining the risk of making a change because it gives administrators the ability to see affected resources and how they are interconnected before making the change.

HPE OneView also streamlines the process of bringing the frames, interconnects, and server hardware under management. When a supported device is added to the solution, HPE OneView automatically detects all the hardware and prepares it for monitoring and management. In HPE OneView, server hardware either has a server profile and is allocated and fully configured or has no server profile and is available as raw hardware in a pool while it awaits a new configuration. This approach supports the most dynamic reconfiguration of hardware possible while preserving the simplicity of provisioning a new server profile with the same settings as existing ones. You can successfully deploy server profiles to the allocated hardware based on information about the server hardware type and frame group.

Figure 30 shows the high-level HPE OneView architecture.


Figure 30. HPE OneView architecture

Designed for automation A substantial portion of the work of operations teams consists of routine tasks related to infrastructure life cycle management including designing, provisioning, monitoring, and updating configurations. HPE OneView is designed to automate day-to-day responsibilities by simplifying time-consuming tasks which leads to increased productivity and reduced operational costs. HPE OneView is an automated infrastructure provider under any environment (including VMware, Microsoft, Red Hat®, and OpenStack®) that supports traditional, virtualized, and cloud workloads.

Enhanced user experience The simple, efficient, and consistent user interface of HPE OneView is designed to enhance the interaction between IT staff members and match existing work practices in the data center. It features new capabilities inspired by web technology and rethinks them for the data center. You also have the choice of using programmatic interfaces that are based on RESTful APIs.

The HPE OneView user interface design builds functionality around administrators’ work practices and puts resources in the menu while fully embracing the web experience. You can right-click any of the links in the HPE OneView dashboard to open them in a new tab, copy and paste, browse bookmarks, easily email links to colleagues, and print diagrams and data. The user interface has search capabilities, newsfeeds, and other functions that deliver a web experience. The HPE OneView user interface works on desktops, tablets, and mobile devices.


Figure 31. HPE OneView user interface

Software-Defined resources HPE OneView includes software-defined resources (templates, server profiles, frames, and frame groups) that provide an innovative way to manage the entire data center. These logical constructs enable administrators to specify the desired configuration of the environment and allow HPE OneView to automate the process of implementing that configuration.

Groups and templates are used to define configurations that are specific to the environment, such as configurations for vSphere virtual hosts. These resources provide flexibility to simplify changes across the data center and control change management. By using these reusable logical constructs, administrators can capture the best practices of experts across a wide variety of disciplines, including networking, storage, hardware configuration, and operating system build and configuration. HPE OneView keeps best-practice approaches intact as the environment grows while allowing customization to maintain ultimate control. This model facilitates faster provisioning, greater consistency, and fewer errors.

Server profiles and frame groups make it easier to prepare a bare metal server for operating system deployment because they contain the information necessary to configure the system firmware, BIOS settings, local storage settings, SAN settings, and network connectivity, thus defining the complete configuration. Template server profiles can be used to capture best practices once and then roll them out multiple times in an efficient and error-free way.

HPE OneView Sizing and solution requirements Every component in the solution must be managed or monitored by HPE OneView if HPE OneView supports that component. In planning the size of your solution, you must take the configuration maximums of HPE OneView into account. The HPE OneView 4.0 Support Matrix for HPE Synergy at https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00018562en_us contains all the configuration maximums; listed in the following table are the ones most prevalent to the HPE Converged Solution 750.

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00037745en_us


Table 18. Hardware configuration maximums

Resource Maximum

Frames 21

Storage Systems 40

Storage Volumes 10,000

Interconnects in all frames 126

Total number of servers 252

Table 19. Networking limits configuration maximums

Resource Maximum

Provisioned Ethernet networks 3966

Fibre Channel networks 255

Fibre Channel networks per interconnect 15

Fibre Channel networks per Logical interconnect

30

Networks per logical interconnect 4094

Networks per network set/physical downlink 162

Networks per uplink set 3966

Logical interconnect groups 126

Logical interconnects 126

HPE OneView Global Dashboard HPE OneView Global Dashboard provides a unified view of the health of Hewlett Packard Enterprise servers, profiles, enclosures and storage systems across multiple appliances for ease of management in the following capabilities:

• Provides a unified view across multiple versions of HPE OneView including: 2.0, 3.0, 3.1, and 4.0, and HPE Synergy Composer.

• Aggregates critical activates from multiple appliances into a single feed to quickly identify issues occurring on monitored hardware for prompt resolution.

• Generates reports for monitored assets to view inventory, including firmware versions, as well as compliance reports that allow you to verify that your equipment meets corporate standards.

• One click navigation with single sign-on, in context launch of managed resources.

VMware vCenter Server VMware vCenter Server offers centralized control and visibility at every level of the virtualized infrastructure. Its single console provides unified management of the entire vSphere environment, enabling resource management and optimization by giving administrators insight into the status and configuration of clusters, hosts, VMs, storage, guest operating systems, and other components to maximize efficiency.

A single vCenter 6.5 Server instance allows the VM administrator to manage up to 2,000 hosts and 25,000 VMs. For more information about vCenter Server, see the vCenter Server page at vmware.com/products/vcenter-server/.

http://www.vmware.com/products/vcenter-server/


HPE OneView for VMware vCenter To extend the capabilities of vCenter to the physical environment, HPE Converged Solution 750 includes HPE OneView for VMware vCenter. This Hewlett Packard Enterprise product integrates the manageability features of HPE Synergy, HPE BladeSystem, and HPE Synergy fabric with VMware vCenter Server and VMware vRealize Operations Manager to create a holistic, single pane of glass management experience.

HPE OneView for VMware vCenter enables administrators to gain a deeper control of the virtualized environment and reduces the time it takes to make changes, increase capacity, or manage planned and unplanned downtime. When used with the automation power of HPE OneView, best practices for HPE Converged Solution 750 can be defined once and reused many times to provision an entire cluster with computing and storage resources that can be fully configured in a few steps. New integrations with VMware vRealize Operations Manager are also available to deliver powerful analytics and deeper troubleshooting tools to VMware administrators.

The integration between Hewlett Packard Enterprise converged management and VMware vCenter management results in the following capabilities:

• Simplified administration with VMware console access to Hewlett Packard Enterprise infrastructure management.

• Automated responses to hardware events.

• Ability to take control by launching Hewlett Packard Enterprise management tools within the vCenter console.

• Ability to manage changes proactively with detailed relationship dashboards.

• Ability to leverage on-demand server and storage capacity.

• Stability and reliability with online firmware inventory and deployment.

• Deep analytics and troubleshooting through integrations with vRealize Operations Manager.

HPE Nimble Connection Manager HPE Nimble Connection Manager (NCM) manages connections from the host to volumes on Nimble systems. To simplify configuring multiple connections and Multipath Input/Output (MPIO), the HPE NimbleOS requires that only one IP address (the iSCSI discovery IP address) be advertised, instead of needing to advertise the full set of iSCSI network interfaces at the time of discovery.

This means that you do not need to manually make specific connections to the appropriate interfaces, or worry about how many connections there are to a volume. As connections are made to the same address (group target portal), the connections are redirected to the appropriate distribution of actual iSCSI network interfaces.

HPE Nimble Storage vCenter Plugin HPE Nimble Storage provides a plugin that works specifically with vCenter to manage datastores residing on HPE Nimble Storage arrays. The VMware thick client as well as the web-based vCenter client are both supported for use with the HPE Nimble Storage plugin.

The HPE Nimble Storage vCenter plugin allows the following capability directly from vCenter:

• Create, clone, grow, and edit datastores.

• Create, clone, and delete snapshots.

• Add HPE Nimble Storage-specific capabilities to vCenter server which can be used to create vCenter roles.

• Edit protection schedules.

VMware vRealize Operations Manager vRealize Operations Manager on HPE Converged Infrastructure combines a leading software and hardware virtualization platform with robust management capabilities. This combination enables you to gain operational insight into a vSphere environment while optimizing capacity. By providing detailed analysis of the virtual environment, the product enables you to reclaim unused capacity, right-size VMs, improve usage, and increase consolidation ratios. You can leverage vRealize Operations Manager on HPE Converged Infrastructure to obtain the following benefits:

• Improve application performance and business continuity. vRealize Operations Manager supports even the most resource-intensive applications to meet SLA and availability requirements. It reduces the cost and complexity of managing business continuity and disaster recovery with always-on IT capabilities and layered protection against service outages and data loss.


• Simplify IT operations at a lower cost. vRealize Operations Manager lowers operational overhead and simplifies the management of development, quality assurance, and production IT environments that are large and geographically distributed.

• Gain operational insight. vRealize Operations Manager enables IT to respond to performance trends proactively. It collects and analyzes performance data, correlates abnormalities, and identifies the root cause of building performance issues.

• Optimize capacity. vRealize Operations Manager makes it easier to manage and optimize capacity planning. Unused VMs can be reclaimed, and under provisioned VMs can be right-sized, to help make the most efficient use of resources.

HPE OneView integration with VMware vRealize Operations Manager HPE OneView integration with VMware vRealize Operations Manager simplifies administration by merging the Hewlett Packard Enterprise and VMware management ecosystems. It accomplished this by:

• Providing HPE OneView Server and infrastructure topology, status, and events to the vRealize Operations Manager environment.

• Correlating and managing relationships between the physical and virtual infrastructure.

• Providing key operational metrics for HPE Server/HPE Synergy for improved trend and root-cause/impact analyses.

• Proactively avoiding performance issues and providing deep insights across a list of monitored HPE components.

• Providing topology resource relationships between HPE OneView objects and virtual machines and application objects.

• Delivering end-to-end analyses of all types of machine-generated log data to reduce troubleshooting times and improve mean-time-to-resolution.

HPE StoreOnce Recovery Manager Central for VMware HPE StoreOnce Recovery Manager Central for VMware enables VMware administrators to create hundreds of VMware-consistent snapshots and initiate rapid online recovery directly from within the VMware vCenter Server Management Console. You can back up to an HPE StoreOnce directly by using the HPE StoreOnce RMC Express Protect feature. The backups are self-contained volumes that can be restored back to the original or different HPE 3PAR StoreServ Storage in the event of a disaster.

Operational Support from HPE Pointnext HPE Pointnext offers support options for the various types of the Converged Solution 750. The following sections describe the different product types and their minimum support requirements as well as other enhanced support options.

The Converged Architecture 750 is available in two models with specific support experiences for each product type:

• HPE Converged Architecture 750 for HPE Synergy Gen10 with Solution Support Tracking (R0H83A) requires HPE Proactive Care 24X7 with a 4 hour response service level.

• HPE Converged Architecture 750 for HPE Synergy Gen10 without Solution Support Tracking (Q8B00A) has no minimum support service level. Any of the available support options listed below may be applied to this product. It is important to note that this version of the Converged Architecture 750 is not eligible for the Solution Level support experience.

The HPE Converged System 750 is available in two models and share the same minimum support level.

• HPE Converged System 750 3.x for HPE Synergy VMware Kit Tracking (Q8A80A) and

• HPE Converged System 750 3.x for HPE Synergy Foundation Tracking (Q8A78A) require a minimum of Proactive Care 24X7 4 hour response service level.

HPE Proactive Care Advanced and DataCenter Care may be applied to the Converged Solution 750 portfolio as they exceed the minimum requirements specified above.

Contact your HPE Pointnext representative to discuss how to meet your specific business needs and service availability.

HPE Proactive Care As the minimum support level for the CA750 with Solution Level Support as well as the CS750, HPE Proactive Care Service can simplify IT operations, save time and costs, and maintain availability and performance. As a result, IT staff is freed up to focus on more value-added, strategic activities. HPE Proactive Care provides an enhanced call experience with advanced technical expertise and start-to-finish call management to solve problems quickly and effectively. In addition, HPE Proactive Care can leverage products connected to Hewlett Packard


Enterprise to provide proactive services and help avoid problems and reduce outages. HPE Proactive Care provides support that covers servers, operating systems, hypervisors, storage, storage area networks (SANs), and network components.

HPE Proactive Care includes onsite hardware repair if it is required to resolve an issue. You can choose from a range of hardware reactive support levels to meet your business and operational needs. Hardware reactive support choices include the following:

• HPE Proactive Care with 24x7x4 hour response (minimum support level).

• HPE Proactive Care with 6 hour call to repair commitment.

The HPE Proactive Care portfolio also offers hardware Defective Media Retention (DMR) and Comprehensive Defection Material Retention (CDMR) as additional core features if you need to retain possession of the hard drive or other components due to legal or security requirements.

Additional service options for HPE Converged Solution 750 HPE Proactive Care Advanced Service is designed for customers who want assigned resources for personalized technical assistance to help meet IT and business goals. HPE Proactive Care Advanced builds on HPE Proactive Care and gives you an assigned, local account support manager to share best practices, flexible access to specialist skills, and critical event management if there is a complex incident or system outage.

HPE Datacenter Care Service is Hewlett Packard Enterprise’s most comprehensive support solution, featuring a personalized assigned account team, a global enhanced call experience, access to highly skilled specialists, and many options to tailor the service to meet a wide range of data center support requirements. Datacenter Care is a structured framework of repeatable, tested, and globally available service “building blocks”, which enables you to deploy, operate, and evolve your data centers wherever you are on your IT journey.

With HPE Flexible Capacity Service, Hewlett Packard Enterprise delivers a unique way to consume IT based on a “pay per use” model for on-premises capacity with rapid scalability. Hewlett Packard Enterprise forms a partnership with you, where together we ensure that you have capacity to support your business needs.

HPE Foundation Care Service is composed of comprehensive hardware and software services aimed to help increase the availability of IT infrastructure. For hardware products, the service includes remote diagnosis and support, as well as onsite hardware repair if it is required to resolve an issue. For software products, Hewlett Packard Enterprise provides remote technical support and access to software updates and patches. A variety of reactive support levels are available to meet your business and operational needs. HPE Foundation Care Service is allowed in the following instances:

• For the HPE Converged Architecture 750 for HPE Synergy Gen10 without Solution Support Tracking (Q8B00A).

• When quoted in conjunction with a DataCenter Care service offer.

HPE Pointnext Support Credits offer flexible services and technical skills to meet changing IT demands through a menu of services that provide additional resources and specialist skills to help maintain IT peak performance. Support Credits are offered as annual credits, enabling you to plan your budget while proactively responding to dynamic business needs.

HPE Education Services helps you become proficient in managing the day-to-day system administration tasks of your environment through HPE Training Credits.

Support considerations for customer owned hardware If any customer-owned hardware is used in the Converged Architecture 750 configuration, channel partners and customers are advised to contact their HPE Pointnext representatives to determine the eligibility of that equipment to be added or transferred to the Converged Architecture 750 support contract.

Support considerations for Cisco switches Channel partners and customers are advised to purchase support directly from Cisco for any Cisco equipment used in their solution.

Support considerations for Converged Solution 750 with HPE Nimble Storage Converged Solution 750 with HPE Nimble Storage customers will have a separate support contract to cover the Nimble storage device but are encouraged to contact the Converged Architecture 750 with Solution Support or Converged Solution 750 Solution Support team for all issues. The HPE support team will assist customers in troubleshooting their Converged System or Converged Architecture system to include assisting in engaging the HPE Nimble support team if needed. Converged System 750 and or Converged Architecture 750 with Solution level support customers are free to contact the HPE Nimble Storage support team first for any problems with their Converged Solution 750 with HPE Nimble Storage. The HPE Nimble Storage support team is adept at isolating and resolving problems across the Converged Solution 750 configuration.


The HPE Nimble Storage support team will engage or transfer cases to other HPE Pointnext component support resources as appropriate for problem resolution.

HPE Connected Products Connecting products to Hewlett Packard Enterprise unlocks the benefits of your technology investment. Customers who connect support automation enabled products receive 24x7 monitoring, prefailure alerts, automatic call logging, and automatic parts dispatch. HPE Proactive Care Service and HPE Datacenter Care Service customers also benefit from proactive activities to help prevent issues and increase optimization. These benefits are available with HPE server, storage, and networking products, and are securely connected to Hewlett Packard Enterprise. Due to the flexibility of the CS750 and CS750 design, some technical components may be monitored by technologies unique to a specific product class. Examples of the different monitoring solutions include, IRS, OneView, Stats, and InfoSight.

Licensing considerations If you elect not to purchase VMware or Microsoft licenses through Hewlett Packard Enterprise, you must provide your own Enterprise License Agreement (ELA) licenses for each VMware component when your HPE Converged Solution 750 solution is being installed.

VMware licenses Customers have the option to include VMware licenses in their purchase or utilize their own VMware Enterprise Licenses Agreement (ELA) in HPE Converged Solution for VMware. VMware support is available for purchase from HPE Pointnext in either licensing scenario. Your HPE Pointnext representative can assist you if an ELA is being used. When VMware licenses are purchased from Hewlett Packard Enterprise, Hewlett Packard Enterprise will handle all VMware support issues, from beginning to end.

For more information about VMware vSphere licensing, see VMware vSphere: Licensing, Pricing, and Packaging white paper at vmware.com/files/pdf/vsphere_pricing.pdf.

Solution verification To determine if the HPE Converged Solution 750 solution can serve as a foundational infrastructure for general-purpose workloads, Hewlett Packard Enterprise validated the architecture with a range of tests to check the high-availability features of the environment. Through these tests, Hewlett Packard Enterprise has verified that zero data loss occurred in designated test scenarios and no service interruption occurred in each test case, except for an unexpected host failure, which resulted in a rapid reboot of the affected VMs.

http://www.vmware.com/files/pdf/vsphere_pricing.pdf


Appendix A: Bill of materials Design 1: HPE Synergy Gen10 with Fibre Channel based HPE Nimble Storage The HPE Converged Solution 750 solution used the following bill of materials (BOM) for testing and configuration of the Hewlett Packard Enterprise-verified configuration used to develop this design guide.

Note Part numbers are at time of testing and subject to change. The bill of materials does not include complete support options or other rack and power requirements. If you have questions regarding ordering, consult with your Hewlett Packard Enterprise Reseller or Hewlett Packard Enterprise Sales Representative for more details at hpe.com/us/en/services/consulting.html.

Table 1a. Solution SKU

QTY Part Number Description

1 R0H83A HPE CA750 Synergy Gen10 w/Sol Supp Trk

Table 1b. Rack infrastructure

Qty Part Number Description

1 P9K40A HPE 42U 600x1200 Ent G2 Shock Rack

1 P9L16A HPE G2 Rack 42U 1200mm Side Panel Kit

4 P9S15A HPE G2 Metered/Switched 4.9kVA/L6-30P 24A/208V Outlets (20) C13, (4) C19/Vertical NA/JP PDU

4 P9R51A HPE G2 Metered Modular 4.9kVA/L6-30P 24A/208V Outlets (6) IEC C19/1U Horizontal NA/JP PDU

4 C7535A HPE Ethernet 7ft CAT5e RJ45 M/M Cable

Table 1c. HPE Synergy Multi Frame Master 1

Qty Part Number Description Comments

1 797740-B21 HPE Synergy 12000 CTO Frame 1xFLM 10x Fan

1 794502-B23 HPE VC SE 40Gb F8 Module

1 779218-B21 HPE Synergy 20Gb Interconnect Link Mod

1 798096-B21 HPE Synergy 12000F 6x 2650W AC Ti FIO PS

1 804353-B21 HPE Synergy Composer

1 804942-B21 HPE Synergy Frame Link Module

1 804943-B21 HPE Synergy 12000 Frame 4x Lift Handle

1 804938-B21 HPE Synergy 12000 Frame Rack Rail Option

2 817040-B21 HPE Synergy 40/4x10GbE/4x8GbFC QSFP XCVR

1 N3R43A HPE Synergy 8Gb FC License Upgrade

4 804101-B21 HPE Synergy Interconnect Link 3m AOC

2 817040-B21 HPE Synergy 40GbE/4x10GbE/4x8Gb FC QSFP+ Transceiver

1 838327-B21 HPE Synergy Dual 10GBASE-T QSFP+ 30m RJ45 Transceiver Only needed if using HPE Synergy Image Streamer

3 861412-B21 HPE Synergy Frame Link Module CAT6A 1.2m Cable Reduce to 1 if not using HPE Synergy Image Streamer

http://www.hpe.com/us/en/services/consulting.html



1 720199-B21 HPE BladeSystem c-Class 40G QSFP+ to QSFP+ 3m DAC Cable

2 K2Q46A HPE MPO to 4 x LC 5m Cable

Table 1d. HPE Synergy Multi Frame Master 2







1 804937-B21 HPE Synergy Image Streamer Optional












Table 1e. HPE Synergy Multi Frame Satellite 1







1 804938-B21 HPE Synergy Frame Rack Rail Kit




1 861413-B21 HPE Synergy Frame Link Module CAT6A 3m Cable


Table 1f. Management servers (HPE Synergy 480 Gen10 Compute Modules)


2 871940-B21 HPE Synergy 480 Gen10 Configure-to-order Compute Module

2 872138-L21 HPE Synergy 480/660 Gen10 Xeon-G 6142 FIO Kit

2 872138-B21 HPE Synergy 480/660 Gen10 Xeon-G 6142 Kit

32 815098-B21 HPE 16GB 1Rx4 PC4-2666V-R Smart Kit

4 872477-B21 HPE 600GB SAS 12G Enterprise 10K SFF SC DS HDD

2 P01367-B21 HPE 96W Smart Storage Battery with 260mm Cbl

2 804428-B21 HPE Smart Array P204i-c SR Gen10 16G SAS Modular Ctrlr

2 777430-B21 HPE Synergy 3820C 10/20Gb CNA

Table 1g. Compute modules (HPE Synergy 480 Gen10 Compute Modules)






32 872475-B21 HPE 300GB SAS 10K SFF SC DS HDD Not required if using HPE Synergy Image Streamer

16 P01367-B21 HPE 96W Smart Storage Battery with 260mm Cbl Not required if using HPE Synergy Image Streamer

16 871573-B21 HPE Smart Array P204i-c SR Gen10 16G SAS Modular Ctrlr Not required if using HPE Synergy Image Streamer


Table 1h. HPE Nimble Storage AF40


1 Q8H41A HPE Nimble Storage AF40 All Flash Dual Controller 10GBASE-T 2-port Configure-to-order Base Array

1 Q8C23B HPE Nimble Storage 2x10GbE 4-port and 4x16Gb Fibre Channel 4-port FIO Adapter Kit

1 Q8G61B HPE Nimble Storage AF5000/7000/9000 All Flash Array 24x1.92TB Flash Bundle

1 Q8G27B HPE Nimble Storage NOS Default Software

2 Q8J27A HPE Nimble Storage C13 to C14 250V 13Amp 1m PDU Base Array Power Cord

Table 1i. Common Ethernet cabling (adjust lengths to meet the distance requirements of customer provided 1Gbe network infrastructure)


12 C7536A HPE Ethernet Cable 14ft CAT5 RJ45 M/M

Table 1j. HPE FF 5940 48SFP+ 6QSFP+ Switch



2 JH397A HPE FF 5940 2-slot Switch

4 JG552A HPE X711 Frt(prt) Bck(pwr) HV Fan Tray

4 JC680A HPE 58x0AF 650W AC Power Supply

4 JC680A B2B INCLUDED: Jumper Cable - NA/JP/TW

2 JH183A HPE 5930 8-port QSFP+ Module

2 JG326A HPE X240 40G QSFP+ QSFP+ 1m DAC Cable


Table 1k. Arista 7050SX2-72Q


2 JH943A Arista 7050X2 48SFP+ 6QSFP+ BF AC Switch

2 JH772A Arista 40G QSFP+ QSFP+ 0.5m DAC Cable

4 JG330A HPE X240 QSFP+ 4x10G SFP+ 3m DAC Cable

Table 1l. Cisco Nexus 9396PX Note Cisco Part Numbers and must be purchased from Cisco or a Cisco Authorized Reseller


2 N9K-C9396PX Cisco Nexus 9300 48p 1/10G SFP+ & additional uplink module req.

2 N9K-C9300-ACK Cisco Nexus 9300 Accessory Kit

2 N9K-C9300-RMK Cisco Nexus 9300 Rack Mount Kit

6 N9K-C9300-FAN2-B Cisco Nexus 9300 Fan 2, Port-side Exhaust

4 N9K-PAC-650W-B Cisco Nexus 9300 650W AC PS, Port-side Exhaust

4 CAB-C13-C14-2M Power Cord Jumper, C13-C14 Connectors, 2 Meter Length

2 N9K-M12PQ Cisco ACI Uplink Module for Nexus 9300, 12p 40G QSFP

2 QSFP-H40G-CU1M Cisco 40GBASE-CR4 QSFP+ direct-attach copper cable, 1 meter passive

4 JG330A (HPE Part Number) HPE X240 QSFP+ 4x10G SFP+ 3m DAC Cable

Table 1m. HPE SN6600B SAN switches


2 Q0U54A HPE SN6600B 32Gb 48/24 FC Switch

48 QK724A HPE B-series 16Gb SFP+SW XCVR

32 QK733A HPE Premier Flex LC/LC OM4 2f 2m Cbl


Design 2: HPE Synergy Gen10 with HPE 3PAR storage This design is suitable for customers who need a balanced performance system but do not want to sacrifice any resiliency and or HA capabilities. In this configuration, three HPE Synergy 480 Gen10 Compute Modules are used to leverage VMware vCenter HA to have a fault tolerant VMware vCenter instance.

















































































30 872475-B21 HPE 300GB SAS 10K SFF SC DS HDD Not required if using HPE Synergy Image Streamer or booting from SAN

15 P01367-B21 HPE 96W Smart Storage Battery with 260mm Cbl Not required if using HPE Synergy Image Streamer or booting from SAN

15 871573-B21 HPE Smart Array P204i-c SR Gen10 16G SAS Modular Ctrlr Not required if using HPE Synergy Image Streamer or booting from SAN


Table 2h. HPE 3PAR StoreServ 8450 4 Node Storage






1 H6Z23B HPE 3PAR 8450 4N+SW Storage Base

48 K2P91B HPE 3PAR 8000 3.84TB+SW SFF SSD

4 H6Z00A HPE 3PAR StoreServ 8000 4-port 16Gb Fibre Channel Adapter

6 H6Z26A HPE 3PAR StoreServ 8000 SFF(2.5in) SAS Drive Enclosure

1 K2R29A HPE 3PAR StoreServ RPS Service Processor

1 L7F20AAE HPE 3PAR All-in S-sys SW Current Media














Table 2k. Arista 7160-32QSFP28 Switch


2 JH946A Arista 7160 32QSFP28 BF AC Switch





















Design 3: HPE Synergy Gen10 with HPE 3PAR storage, a 16Gb FC topology, and out of frame management servers This design is suitable for customers who need a dedicated 16Gb FC topology for the best storage networking performance with the option for a dedicated out of frame management servers in order to use all the HPE Synergy compute modules for VMware compute hosts.

















2 779227-B21 HPE Virtual Connect SE 16Gb Fibre Channel Module for HPE Synergy

8 QK724A HPE B-series 16Gb SFP+ Short Wave Transceiver



















2 779227-B21 HPE Virtual Connect SE 16Gb Fibre Channel Module for Synergy



















2 779227-B21 HPE Virtual Connect SE 16Gb Fibre Channel Module for Synergy











Table 3f. Management servers (HPE ProLiant DL360 Gen10 Servers) Configured for use with VMware vCenter HA


3 867959-B21 HPE ProLiant DL360 Gen10 8SFF Configure-to-order Server

3 860669-L21 HPE ProLiant DL360 Gen10 Intel Xeon-Gold 6142 FIO Kit

3 860669-B21 HPE ProLiant DL360 Gen10 Intel Xeon-Gold 6142 Kit


3 867966-B21 HPE ProLiant DL360 Gen10 2SFF SAS/SATA Backplane Kit


3 P01366-B21 HPE 96W Smart Storage Battery with 145mm Cable Kit


3 652503-B21 HPE Ethernet 10Gb 2-port 530SFP Adapter

3 P9D94A HPE StoreFabric SN1100Q 16Gb Dual Port FC Host Bus Adapter

3 700751-B21 HPE FlexFabric 10Gb 2-port 534FLR-SFP+ Adapter Only needed if using VMware HA

6 865414-B21 HPE 800W Flex Slot Platinum Hot Plug Low Halogen PS Kit

3 P8B31A HPE OV for DL 3y 24x7 FIO Phys 1 Svr Lic

3 867996-B21 HPE ProLiant DL360 Gen10 SFF System Insight Display Power Module Kit

Optional

3 873770-B21 HPE DL3XX Gen10 Rear Serial Cable and Enablement Kit Optional

3 874543-B21 HPE 1U Gen10 SFF Easy Install Rail Kit







72 872475-B21 HPE 300GB SAS 10K SFF SC DS HDD Not required if using HPE Synergy Image Streamer or booting from SAN

36 P01367-B21 HPE 96W Smart Storage Battery with 260mm Cbl Not required if using HPE Synergy Image Streamer or booting from SAN



36 871573-B21 HPE Smart Array P204i-c SR Gen10 16G SAS Modular Ctrlr Not required if using HPE Synergy Image Streamer or booting from SAN


36 777452-B21 HPE Synergy 3830C 16Gb Fibre Channel Host Bus Adapter

Table 3h. HPE 3PAR StoreServ 9450 2 Node Storage






1 Q0E92A HPE 3PAR 9450 2N+SW Storage Base

4 Q0E97A HPE 3PAR 9000 4pt 16Gb FC HBA

1 P9M30A HPE 3PAR Direct Connect Cabling Option

1 Q1H95A HPE 3PAR 1U Rack Accessories Kit

1 Q0F86A HPE 3PAR StoreServ RPS Service Processor

1 Q0E95A HPE 3PAR 9000 2U SFF Drive Enclosure

60 Q0F41A HPE 3PAR 9000 3.84TB+SW SFF SSD

8 716197-B21 HPE Ext 2.0m MiniSAS HD to MiniSAS HD Cbl

3 Q0E95A HPE 3PAR 9000 2U SFF Drive Enclosure

6 P9Q41A HPE G2 Basic 4.9kVA/(20) C13 NA/JP PDU

1 L7F19A HPE 3PAR All-in Sngl-sys SW Latest Media





Qty Part Number Description Comment










2 JG329A HPE X240 QSFP+ 4x10G SFP+ 1m DAC Cable

4 required if additional nics for VMware vCenter HA are on the order

Table 3k. Arista 7160 48SFP28 6SFFP28 Switch


2 JH948A Arista 7160 48SFP28 6QSFP28 BF AC Switch



6 JD096C HPE X240 10G SFP+ SFP+ 1.2m DAC Cable

12 required if additional nics for VMware vCenter HA are on the order







4 N9K-PAC-650W-B

Cisco Nexus 9300 650W AC PS, Port-side Exhaust

4 CAB-C13-C14-2M

Power Cord Jumper, C13-C14 Connectors, 2 Meter Length

2 N9K-M12PQ

Cisco ACI Uplink Module for Nexus 9300, 12p 40G QSFP

2 QSFP-H40G-CU1M

Cisco 40GBASE-CR4 QSFP+ direct-attach copper cable, 1 meter passive


6 JD096C HPE X240 10G SFP+ SFP+ 1.2m DAC Cable

12 required if additional nics for VMware ProActive HA are on the order








Design 4: HPE Synergy Gen10 with iSCSI Channel based HPE Nimble Storage The HPE Converged Solution 750 solution used the following bill of materials (BOM) for testing and configuration of the Hewlett Packard Enterprise-verified configuration used to develop this design guide.























1 861412-B21 HPE Synergy Frame Link Module CAT6A 1.2m Cable



Table 4d. Management servers (HPE Synergy 480 Gen10 Compute Modules)













Table 4e. Compute modules (HPE Synergy 480 Gen10 Compute Modules)






8 872475-B21 HPE 300GB SAS 10K SFF SC DS HDD




Table 4f. HPE Nimble Storage HF40


1 Q8H39A HPE Nimble Storage HF40 Adaptive Dual Controller 10GBASE-T 2-port Configure-to-order Base Array

1 Q8H45A HPE Nimble Storage HF40/60 Adaptive Array 42TB (21x2TB) FIO HDD Bundle

1 Q8B63B HPE Nimble Storage HF40 Adaptive Array R2 5.76TB (6x960GB) FIO Cache Bundle

1 Q8B88B HPE Nimble Storage 2x10GbE 2-port Adapter Kit

1 Q8G27B HPE Nimble Storage NOS Default Software

2 Q8J27A HPE Nimble Storage C13 to C14 250V 13Amp 1m PDU Base Array Power Cord

Table 4g. Common Ethernet cabling (adjust lengths to meet the distance requirements of customer provided 1Gbe network infrastructure)




Table 4h. HPE FF 5940 48SFP+ 6QSFP+ Switch







2 JH180A HPE 5930 24p SFP+ and 2p QSFP+ Module



8 JL437A HPE X130 10G SFP+ LC SR DC Transceiver

8 QK733A HPE Premier Flex LC/LC Multi-mode OM4 2 fiber 2m Cable

Design 5: HPE Synergy Gen10 with HPE Synergy D3940 with VMware vSAN This design is suitable for customers who need a balanced performance system but do not want to sacrifice any resiliency and or HA capabilities. In this configuration, three HPE Synergy 480 Gen10 Compute Modules are used to leverage VMware vCenter HA to have a fault tolerant VMware vCenter instance.



















2 755985-B21 HPE Synergy 12Gb SAS Connection Module with 12 Internal Ports














































3 871942-B21 HPE Synergy 480 Gen10 CTO Prem Cmpt Mdl






3 804428-B21 HPE Smart Array P416ie-m SR Gen10 Ctrlr

3 871573-B21 HPE SAS Cable for P416ie-m SR G10 Ctrlr




21 871942-B21 HPE Synergy 480 Gen10 CTO Prem Cmpt Mdl




42 872475-B21 HPE 300GB SAS 10K SFF SC DS HDD Not required if using HPE Synergy Image Streamer

21 P01367-B21 HPE 96W Smart Storage Battery with 260mm Cbl Not required if using HPE Synergy Image Streamer

21 871573-B21 HPE Smart Array P204i-c SR Gen10 16G SAS Modular Ctrlr Not required if using HPE Synergy Image Streamer

21 871573-B21 HPE SAS Cable for P416ie-m SR G10 Ctrlr



Table 5h. HPE Synergy D3940 Storage Modules (2 storage modules per HPE Synergy Frame)


6 757323-B21 HPE Synergy D3940 Redundant I/O Adapter

6 755984-B21 HPE Synergy D3940 12Gb SAS Drive Enclosure with 40 SFF (2.5in) Drive Bays

48 873351-B21 HPE 400GB SAS 12G Write Intensive SFF (2.5in) SC 3yr Wty Digitally Signed Firmware SSD

192 873365-B21 HPE 1.6TB SAS 12G Mixed Use SFF (2.5in) SC 3yr Wty Digitally Signed Firmware SSD













Table 5k. Arista 7050SX2-72Q


2 JH943A Arista 7050X2 48SFP+ 6QSFP+ BF AC Switch















HPE Converged Architecture 750 Page 88

Sign up for updates

© Copyright 2018 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.

Cisco and Nexus are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.

Microsoft, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. Red Hat is a registered trademark of Red Hat, Inc. in the United States and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. Cloud Foundry is a trademark and/or registered trademark of Pivotal Software, Inc. in the United States and/or other countries.

Arista Networks and EOS are trademarks of Arista Networks Inc in the U.S. and other countries. The OpenStack Word Mark and OpenStack Logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed, or sponsored by the OpenStack Foundation, or the OpenStack community.

P09407-002a , October 2018, Rev. 2

Resources and additional links HPE Reference Architectures hpe.com/info/ra

HPE Servers hpe.com/servers

HPE Storage hpe.com/storage

HPE Nimble Storage HPE Nimble Storage

HPE Networking hpe.com/networking

HPE Pointnext services hpe.com/pointnext

To help us improve our documents, provide feedback at hpe.com/contact/feedback.

http://www.hpe.com/info/getupdated

http://www.hpe.com/

http://www.hpe.com/info/ra

http://www.hpe.com/servers

http://www.hpe.com/storage

https://www.nimblestorage.com/

http://www.hpe.com/networking

http://www.hpe.com/contact/feedback

Date post:	18-Mar-2020
Category:	Documents
Upload:	others
View:	8 times
Download:	0 times

HPE Converged Solution 750 · The HPE Converged Solution 750 described in this guide is designed to...

Documents