Resolving Security and Throughput Challenges for High-Performance ComputingSARA identified the I/O virtualization support in Intel® Ethernet 10 Gigabit Converged Network Adapters as the ideal means of meeting both security and throughput requirements for its high-performance cloud computing network infrastructure.

To provide high-performance computing (HPC) resources for research, Netherlands-based SARA must maintain very high throughput for message-passing interface (MPI) workloads between virtual machines (VMs) in a cluster environment. InfiniBand*, the conventional interconnect choice for these types of clusters, did not support the required combination of security and performance in the cloud environment. SARA resolved that issue using Intel® Ethernet 10 Gigabit Converged Network Adapters.

ChAllENGE

• Provide a high-performance interconnect between VMs to support MPI workloads in a cloud infrastructure, while meeting the stringent security standards required to protect valuable intellectual property and other sensitive data.

• Overcome security limitations of basic InfiniBand in the cloud, without adding security layers such as iptables, ebtables, and VLAN tagging that impose severe TCP/IP overhead and reduce throughput to unacceptable levels, due to the limitations of IP over InfiniBand.

SOlutION

• Intel Ethernet 10 Gigabit Converged Network Adapters incorporate I/O virtualization with Intel® Virtualization Technology for Connectivity (Intel® VT-c) that enables lower CPU utilization and improved networking and I/O throughput.

• Data protection for the cloud is provided by PCI-SIG Single Root I/O Virtualization (SR-IOV), a key component of Intel VT-c, which helps ensure that data traffic within one VM cannot be accessed by other VMs, and other platform features such as Intel® Trusted Execution Technology (Intel® TXT), which ensures a trusted launch environment.

CASE StuDyIntel® Ethernet Converged Network AdaptersHigh-Performance Computing

high-Performance Cloud Computing for the Research CommunityAs part of its mission to support research in the Netherlands with advanced, integrated ICT infrastructure, services, and expertise, SARA extends access to its HPC-cloud environment to the broader community. The goal of this approach is to provide infrastructure that is functionally equivalent to the resources that researchers have access to otherwise, but which is many times more powerful.

SARA extends its HPC-cloud computing environment using an infrastructure-as-a-service (IaaS) model that supplies resources on demand to researchers who might otherwise not have access to HPC. Users have access to their own virtual private HPC cluster, which they can configure as needed, without affecting others who are also using the facilities. People from many different scientific domains have found their way to the facility, from life scientists performing large-scale DNA analysis to linguistics researchers analyzing text.

The SARA HPC-cloud environment’s self-service paradigm allows researchers to build their own virtual cluster, either from scratch or using existing templates. Users can pattern a virtual cluster on their own development environment, so the cluster can be functionally identical, preventing the effort and expense of rewriting software to work in the SARA cloud.

This approach can relatively easily overcome the limitations of legacy software, security concerns, or source code unavailability. A high degree of flexibility in the environment enables researchers to tailor it to their needs. Research efforts can also scale from the desktop to the cloud with minimal effort, and cloud users are offered an environment where they can run MPI-dependent jobs over more than 500 virtual compute nodes.

Building the Cloud Environment to Meet Performance and Security RequirementsThe core of SARA’s HPC-cloud includes 19 compute nodes based on Dell PowerEdge* R810 servers, each equipped with four eight-core Intel® Xeon® processors E7-4830. Virtualization is enabled using kernel virtual machine (KVM) in CentOS 6.2, with a maximum of 32 VMs per server (one per physical core, without over-commitment).

Four utility servers host monitoring services and user interfaces, and two file servers provide 400 terabytes of storage. Each server in the environment includes Ethernet ports:

• 10 Gigabit Ethernet (10GbE). Two dual-port Intel® Ethernet Converged Network Adapters X520-DA2 (four ports total).

• Gigabit Ethernet (GbE). Four onboard GbE ports used for out-of-band management network connectivity.

A non-blocking 10GbE interconnect network connects all the cluster nodes. Each node has a bonded network interface with at least four 10GbE ports connected to an Arista Networks 7504 Switch.

The Intel Ethernet Converged Network Adapters X520-DA2 enhance both throughput and security by means of single-root I/O virtualization (SR-IOV), a PCI-SIG standard and key component of Intel VT-c. SR-IOV allows a single physical port to be divided into multiple virtual ports, or “virtual functions,” each of which can be assigned to a specific VM. This mechanism reduces processor overhead, increases overall system-level performance, provides a dedicated and isolated communication channel, and dramatically improves throughput.

Resolving Security and Throughput Challenges for High-Performance Computing

Providing High-Performance Computing Resources to the Dutch Research Community

SARA supports research in the

Netherlands with an advanced,

integrated information and

communications technology

(ICT) infrastructure. In support

of that mission, the organization

provides services in the areas of

high-performance computing,

data storage, visualization,

networking, cloud, and e-Science

support. SARA supports

academic research as well

as industry research in close

cooperation with universities

and research institutions in

the Netherlands and abroad.

SARA seeks to provide a

comprehensive “one-stop shop”

that delivers reliability, quality,

safety, and flexibility through

expertise and innovation.

2

SR-IOV ensures that each VM can access only the physical and virtual ports it has been assigned to. This mechanism effectively isolates the data in each VM from the others, helping secure the cloud environment. SR-IOV is complemented by other platform-level security mechanisms for the cloud, which also contribute to data protection.

Notably, Intel TXT verifies that only trusted software is present prior to launch of the OS or hypervisor, helping circumvent rootkits and other malware attacks. Remote attestation provides for the sharing of measured launch environment credentials across the cloud environment among local or remote users and systems. These mechanisms not only protect data but also help meet security guidelines put in place by research organizations and commercial entities, as well as regulations put in place by government agencies.

Successful Implementation for Performance, Simplicity, and Data ProtectionThe conventional approach of using InfiniBand as an interconnect fabric for MPI workloads had proven to be insufficiently secure for the needs of the SARA HPC-cloud environment, and adding sufficient security superstructure on top of InfiniBand was untenable because of the added overhead of IP over InfiniBand. Using Intel Ethernet 10 Gigabit Converged Network Adapters, on the other hand, offers a simplified, enhanced environment and reduced costs.

• Robust I/O virtualization. SR-IOV increases throughput by enabling direct I/O connectivity to VMs, bypassing software-based traffic mechanisms, and by isolating data traffic between VMs, for added security.

• Common fabric. Ethernet traffic allows for a relatively simple switching environment using the standards-based Arista Networks 7504 Switch in place at SARA, as opposed to the more elaborate switching infrastructure that would be required with InfiniBand.

• Simpler management. Configuration and management is easier than with InfiniBand, because existing TCP/IP-based tools “just work,” using approaches and expertise already in place. The simpler environment also reduces the likelihood of human error, which helps enhance reliability.

Uniquely among the alternatives, Intel Ethernet Converged Network Adapters with SR-IOV enable throughput of nearly 10 gigabits per second for traffic virtualized using KVM, with the data protection characteristics demanded by SARA’s research infrastructure. Notably, the SARA system engineers conducted trials with network adapters from a number of providers that claimed full SR-IOV functionality but found that Intel Ethernet Converged Network Adapters alone actually fulfilled that claim.

ConclusionSARA relies on Intel Ethernet 10 Gigabit Converged Network Adapters to help fulfill its core mission of providing world-class, high-performance cloud computing resources for research. This approach has afforded SARA advantages in terms of performance, security, and cost-effectiveness, based on capabilities not available from InfiniBand, including adapter teaming, SR-IOV, and the low cost and flexibility of Ethernet-based storage (including iSCSI, Fibre Channel over Ethernet, and NFS).

Looking ahead, the SARA architects see additional opportunities from Intel Ethernet Converged Network Adapters, including the implementation of advanced quality-of-service functionality to support robust service-level agreements with users. As a significant part of the basis for continued low-cost innovation for performance and security in the cloud, Intel Ethernet Converged Network Adapters help SARA enable the next generation of high-performance cloud computing for advanced research.

Resolving Security and Throughput Challenges for High-Performance Computing

CROSS-hyPERvISOR I/O vIRtuAlIzAtION

All Intel® Ethernet 10 Gigabit Converged Network Adapters include Intel® Virtualization Technology for Connectivity (Intel® VT-c), which includes both single-root I/O virtualization (SR-IOV) and virtual machine device queues (VMDq). These technologies help reduce I/O bottlenecks and improve overall server performance by offloading functionality to the Intel® Ethernet Controller, providing intelligent traffic-handling offload suitable for use with all major hypervisors.

3

For more about SARA, see www.sara.nl

For more about Intel Ethernet, see www.intel.com/go/ethernet

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.

Intelmaymakechangestospecificationsandproductdescriptionsatanytime,withoutnotice.Designersmustnotrelyontheabsenceorcharacteristicsofanyfeaturesorinstructionsmarked“reserved”or“undefined.”Intelreservestheseforfuturedefinitionandshallhavenoresponsibilitywhatsoeverforconflictsorincompatibilitiesarisingfromfuturechangestothem.Theinformationhereissubjecttochangewithoutnotice.Donotfinalizeadesignwiththisinformation.Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownaserratawhichmaycausetheproducttodeviatefrompublishedspecifications.Currentcharacterizederrataareavailableonrequest.ContactyourlocalIntelsalesofficeoryourdistributortoobtainthelatestspecificationsandbeforeplacingyourproductorder.Copiesofdocumentswhichhaveanordernumberandarereferencedinthisdocument,orotherIntelliterature,maybeobtainedbycalling1-800-548-4725,orbyvisitingIntel’sWebSitewww.intel.com.

SoftwareandworkloadsusedinperformancetestsmayhavebeenoptimizedforperformanceonlyonIntelmicroprocessors.Performancetests,suchasSYSmark*andMobileMark*,aremeasuredusingspecificcomputersystems,components,software,operationsandfunctions.Anychangetoanyofthosefactorsmaycausetheresultstovary.Youshouldconsultotherinformationandperformanceteststoassistyouinfullyevaluatingyourcontemplatedpurchases,includingtheperformanceofthatproductwhencombinedwithotherproducts.Formoreinformationgotowww.intel.com/performance.

*Othernamesandbrandsmaybeclaimedasthepropertyofothers. Copyright©2012IntelCorporation.Allrightsreserved.Intel,theIntellogo,andXeonaretrademarksofIntelCorporationintheU.S.andothercountries. 0712/BY/MESH/PDF 326880-001US

SoluTIoN ProVIDED By: