How HPC Transforms the Corporate Information Technology Ecosystem

Imagination at work

HPC in EnterpriseHow HPC transforms the Corporate Information Technology EcosystemThomas Leung – Sept 10, 2015

make happen

© 2015 General Electric Company - All rights reserved

Who am I?

'No one ever changed the world without first being labeled a 'troublemaker‘ ~ Author

Unknown


Agenda

What is Industrial HPC

Technology, design and

architecture

Position of HPC at enterprise

HPC as a service

Challenges and future focus

*IDC HPC and ROI Pilot Study 10/2013


Average of

revenue per

HPC dollar is

$462.4*

What is industrial HPC and difference to Research HPC?


Industrial HPC (aka Commercial / Enterprise HPC)

• Commodity hardware

• Small – large systems

• Co-located data center

• Resource management based on business priorities between business units. (long

run time.)

• Computational process is part of the design workflow.

• High availability and stability with 24 /7 operation. Business critical processes /

projects support

• Multiple years planning with yearly funding / commitment. (in most cases)

• Multiple source of funding with multiple plans/ directions.

• Highly integrated with enterprise IT systems / internal infrastructure.

• Legacy code and latest code support


System is busy all the time….

Industrial HPC systems operates 24 x 7 with over 95% utilization.


HPC expansion & requirements

HPC systems grow all the time with requirements of both types of HPC.

Research HPC

requirements

Enterprise IT

Requirements

Business Requirements

Industrial HPC

requirements


Technology, Design, Architecture

Bleeding edge technology focus on scale and speed. HPC system will see more “defects” in future.

ResearchHPC


Pull and Push force – on design decision

New technology adoption and wide range of applications support made HPC system design harder.

Vendors: I have the

latest and greatest

technology. Trust me!

It works.

Users: Why aren’t

we using the latest

greatest

technology?

Cloud provider:

Cloud can do

everything now. Finance:

Faster,

More,

Cheaper are

always

better!

Business Leaders:

It is stable now, I

don’t want

anything new and

risky.

Engineer: I have

to support codes

developed 20

years ago.

Managers:

Downtime is not

acceptable.


HPC changes enterprise cyber security

More collaborators, high bandwidth, low latency connections leads to new challenges.

HPC systems

Enterprise

Defense

Global Users

Machines

National Labs

Enterprise

Network

Enterpris

e Network


HPC systems

HPC is moving Enterprise IT

Networking Storage Infrastructure

/Cloud

Application

Mostly 10Gb NAS , Iscsi, FC

storage

Co-located data center

with low power

requirement

Java, .net

10Gb / 40Gb

Ethernet only

Everything is

backup

Virtual machine Max 244GB ram

(EC2)

1 - 10Gb internet

connection

TB – PB storage VM hypervisor controls

resource

No MPI

programming, Java

IaaS only IaaS only PaaS only SaaS only

HPC

40Gb / 100Gb

Ethernet with FDR /

ERR Infiniband

Some workflow

need TB ram

PB storage with fast

IO

Fortran,

Multi thread with

MPI Programming

Resource Mgr

controls

PaaS / SaaS

Enterprise IT


Marriage between Big Data and HPC……

Distributed storage and processing framework in a stable & scalable environment.

Big Data system makes “data” more accessible to HPC. It enhances the simulation efficiency and effectiveness.

Big DataHPC


Position of HPC in Enterprise

ERP / Enterprise

systems

HPC

inside

Past: ERP / Enterprise systems are

the center of the business workflow /

decisions.

HPC is just one of the engineering

tool kits.

Future: Enterprise Hadoop platform

becomes the “CENTER” of all decisions

and workflow.


Understand availability and redundancy of HPC

Single Cray system outage

Recover 100TB data from Tape

Fixing incorrect job submission

Fixing Job submission errors

HPC issues associate with system down

time

Not a lot of system issues can be fixed in less than 1

min.

This is the area of improvement!


Redundancy improves availability?


Please help me solve these problem.


Our Future……


HPC team evolving

HPCers are hard to get. Talent search for HPC positions are harder.

system administrators, storage administrator, database administrators, software

developers, network administrators, solution architect, technologist, business

analyst, project managers, researcher, computer scientist, leaders.Modern HPC

team

HPC team is usually a group of

geeks…..

Traditional

HPC team

Future key requirements:

• Big Data knowledge & hands on experience – able to build production hadoop system and support

operation

• Programming – manage systems with provided API

• Application support – Optimization, New platform adoption and migration

• Data science expertise, business expertise

• Experience with large systems – Really large systems

• Industry specific domain knowledge – Enhance partnership with users and business.


Focus areas for industrial HPC

High availability

• Establish automated

failover capability

• Enhance Job level

recovery after system

failure

• Don’t forget all

surrounding IT services.

24 x 7 service

• 24 x 7 system

monitoring

• 24 x 7 user support

• Full debug and

compiling environment

available

Multiple systems strategy vs

single system strategy

• Easy for compute resource

but hard for storage

solutions.

• Multiple location planning

Automation

• HPC service shall be

automated as Web

service

• API driven management

activity / functions will

increase


Reduce cost

• Software License

cost

• Storage cost

• Platform cost

• $ per CPU cores

Call for actions

Resource Manager

Big Data

VDI

HPCCloud

Storage


Q&A


Backup


Research HPC

• Commodity hardware

• Huge Systems

• Dedicated Data Center

• Resource management based on fair share between users /

programs. (relatively short run time.)

• Handle computational process / workflow only.

• Extended MTBF with parts fail all the time.

• Multiple years planning with one-time funding / commitment.

• Centralized source of funding – University / Government.

• Independent from IT systems / infrastructure.

• latest code support


Size of HPC systems

Small

~100 cores

Eg: Workgroup

<$100k

Medium

~500 cores

Eg:

Departmental

$100k - $250k

Large

~1000 cores

Eg:

Divisional

$250k - $500k

X- Large

>2000 cores

Eg:

Supercomputer

>$500k

XX – Large

>10000 cores

Commercial HPC

Research HPC

Research HPC covers National Lab, Government Agency

and universities

Industrial, Finance, Oil &Gas, Manufacturing companies


Image citation / source CDN Locations | EdgeCast, www.edgecast.com

Gauging Skills | Autism After 16www.autismafter16.com

Service Availability: I'll See Your Five 9s and Raise You One More ...aits.org

MySQL Master/Slave Management with mysql-master-ha | CRGNetwww.cirgan.net

Illustration Depicting Two Triangular Warning Road Signs With A ...www.shutterstock.com

Review: Thomas Pynchon's “Bleeding Edge” | The American Readertheamericanreader.com

Can Crowd Testing replace an Internal Quality Team? | 99testsblog.99tests.com

VIDEO: Find the balance between push and pull can help your ...www.teamsupport.com

FCoE Adoption Curve - Wikibonwikibon.org

Online Customers are Like Toddlers, They Don't Like Getting ...www.contenthook.com

Stkit MOC] Iron Man Hulkbuster statue: A LEGO® creation by St Kit ...www.mocpages.com

LEGO Ideas - Iron Man Hulkbuster armorideas.lego.com

How does an ERP system work? - Quorawww.quora.com

hadoops-role-in-enterprise-architecture-7-638.jpg?cb=1364477105www.slideshare.net

Get Free or Cheap Legal Fees Help in the USwww.grantfundingexpert.org

Migration | QualITasqualitas.co.id

Respect for system administratorsgeekshumor.com

Sincerely Stephanie: The Digit-al Dozen September Spectacular: Geekswww.sincerelystephanienails.com

Huntington's Disease Youth Organization - Young adults - Ask a ...en.hdyo.org

ABigFave | Flickr - Photo Sharing!www.flickr.com

Peak Hour Protests Immobilise Bangalore - The New Indian Expresswww.newindianexpress.com

Competitive Cost Benchmarking Software for Manufacturerswww.mtisystems.com

Zenith Software announces India Expansion PlansDATAQUESTwww.dqindia.com


http://www.edgecast.com/network/map/

http://www.edgecast.com/

http://www.autismafter16.com/article/11-19-2013/gauging-skills

http://aits.org/2014/11/Service-Availability-I-ll-See-Your-Five-9s-and-Raise-You-One-More/

http://www.cirgan.net/mysql-high-availability-with-mysql-master-ha/

http://www.shutterstock.com/pic-100236938/stock-photo-illustration-depicting-two-triangular-warning-road-signs-with-a-redundancy-concept-white.html

http://theamericanreader.com/review-thomas-pynchons-bleeding-edge/

http://blog.99tests.com/can-crowd-testing-replace-an-internal-quality-team/

http://www.teamsupport.com/blog/video-find-the-balance-between-push-and-pull-c

http://wikibon.org/wiki/v/FCoE_Adoption_Curve

http://www.contenthook.com/online-customers-are-like-toddlers-they-dont-like-getting-pushed/

http://www.mocpages.com/moc.php/385897

https://ideas.lego.com/projects/52310

http://www.quora.com/How-does-an-ERP-system-work

http://www.slideshare.net/Hadoop_Summit/hadoop-summit-keynote-shaun-connolly

http://www.grantfundingexpert.org/get-free-or-cheap-legal-help-in-the-us/

https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&ved=0CAUQjhxqFQoTCKyr2PGO6ccCFQPyPgodoSsC8w&url=http://qualitas.co.id/?page_id%3D29&bvm=bv.102022582,d.cWw&psig=AFQjCNGwHUmxCyJJllB7SjlvBd1SaZ7kPw&ust=1441859361050072

http://qualitas.co.id/?page_id=29

http://geekshumor.com/respect-for-system-administrators/

http://www.sincerelystephanienails.com/2013/09/the-digit-al-dozen-september_9.html

http://en.hdyo.org/you/questions

https://www.flickr.com/photos/a4gpa/482617677

http://www.newindianexpress.com/cities/bengaluru/Peak-Hour-Protests-Immobilise-Bangalore/2014/01/28/article2023880.ece

http://www.mtisystems.com/cost-benchmarking-software.html

http://www.dqindia.com/zenith-software-announces-india-expansion-plans/

Date post:	24-Jan-2018
Category:	Technology
Upload:	inside-bigdatacom
View:	772 times
Download:	0 times