MOSSCon 2013, Cisco Open Source talk

Cisco Public 1© 2013 Cisco and/or its affiliates. All rights reserved.

Open Source for CiscoHigh PerformanceComputingDr. Jeffrey M. Squyres

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 2

My talk today

1. Who am I?

2. Cisco and Open Source

3. My Open Source work at Cisco

Who am I?

Insert photo here


Who am I?

Me

Technical Lead at Cisco Systems

Server division, VIC group


Who am I?

I am not in marketing


Who am I?

I cannot fix

your Linksys

router for you(perhaps you should try

DD-WRT)


Who am I?

I write code

Lots of code

All day

Every day

Awwww… yeah


I work in Open Source


I work in these Open Source projects


I work in Open Source

Open MPI

Hardware Locality (hwloc)

OpenFabrics

Linux kernel

Vast majorityof my work

is here

I’ve mademinor contributions

to these other 3


My background story…


My background: research and academia

Undergrad, grad Post doc


My background: research and academia

LAM/MPI

I inherited this

I founded this


Open MPIPACX-MPI

LAM/MPI

LA-MPI

FT-MPI

Sun CT 6

Project foundedin 2003,

merging multiple open sourceMPI projects


My own naievete:This is how I thought it would be

PACX-MPI

LAM/MPI

LA-MPI

FT-MPI

Sun CT 6

Me


…but this is what actually happened

PACX-MPI

LAM/MPI

LA-MPI

FT-MPI

Sun CT 6

Me


…and then this happened

PACX-MPI

LAM/MPI

LA-MPI

FT-MPI

Sun CT 6


…and then this happened

Us


What I learned

Differences=

Good


When you write code no one else will see…

You writeto your own

level ofexpectations


When your code is reviewed by your peers…

You writeto theirlevel of

expectations


When your code is reviewed by everyone…

You writeyour best code

Insert photo here

Cisco and Open Source


Open Source around Cisco

More than just the 4 projectsI participate in

Much


Open Source around Cisco

More than just the 4 projectsI participate in

Much


Latest major open source initiative

After the clouds part,

you are left with…


Latest major open source initiative

Major contributionto


OpenDaylight overview:SDN for everyone


Pop quiz, hotshot

?Why does Cisco

do Open Source?


Pop quiz, hotshot

Why does Ciscodo Open Source?

IT MAKES GOOD BUSINESS SENSE

• Stand on the shoulders of giants• Become part of the community• Contribute to tools / ecosystem

that we all use• Sell more products


1. Stand on the shoulders of giants


1. Stand on the shoulders of giants

Weare

elevated

Weelevateothers


2. Become part of the community

Circle oftrust

you



Circle oftrust

This is whereyou need

to be



Insert FOSS project

name hereThis is where

you needto be


3. Contribute

Pretend this isa really gross

picture of a leech


3. Contribute


picture of a leech


3. Contribute


picture of a leech

Just say noto leeches!


3. Contribute

Let’s be clear here…

FOSS is for everyone

EVERYONE(that’s kinda the point, right?)


3. Contribute

Let’s be clear here…

FOSS is for everyone

EVERYONE(that’s kinda the point, right?)

So how can

there be leeches?


3. Contribute

I don’t contribute to everypiece of FOSS I use

Do you?

Of course not


3. Contribute

Those who can,should


3. Contribute

Big companies can contribute


3. Contribute

Individuals can contribute


3. Contribute

Small organizations can contribute


3. Contribute



3. Contribute


Submit a patch

Write a new pluginTest test test test testSubmit a GOOD bug report

Write documentation

Answer q

uestions

Suggest feature

s Evangelize

Send cash / beer

Use the software

Make schwag

Write a review

Review code

Obtain / provide funding


4. Sell more products

A giant



You, standingon the giant’sshoulders…


4. Sell more products…in thecircle

of trust


4. Sell more products …contributingto the community


Cisco UCSblade server


Cisco Nexus7000 router

Insert photo here

My Open Source work at Cisco


Some background first…


High Performance Computing (HPC)

Using supercomputers to solvereal world problems that are

TOO BIGfor laptops, desktops, or individuals servers


Generally speaking…

Supercomputer=

(Many) Racks of (commodity)high-end servers

(this is one definition; there are others)


A typical supercomputer

Rack of36 1U

servers


How does that work?

Computational problem

Input Output

Take your computational problem…


How does that work?

…and split it up!


Input Output


How does that work?


Input Output

Distribute the input dataacross a bunch of servers


How does that work?

Input Output

Use the network between serversto communicate / coordinate


How does that work?

Input Output

Use the network between serversto communicate / coordinate


How does that work?

Message Passing Interface (MPI)middleware is used for this communication

Input Output


Why go to so much trouble?


One processor

hour

1 processor = …a long time…


Why go to so much trouble?


One processor

hour

One processor

hour

One processor

hour

21 processors = ~1 hour (!)Disclaimer: scaling is rarely perfect

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour


Coordination requires communication

This communicationmay happen a LOT

It therefore needsto be FAST


Fast communication

Sourceserver

Destinationserver

Network


Let’s break it down

Source server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Destination server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Ethernet switch

Port A

Port B


Source server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Destination server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Ethernet switch

Port A

Port B


200 nanoseconds

299,792,458 m/s c

~8 microseconds(modern hardware) ~8-40 microseconds

Total:~17 – 81

microseconds

~40 microseconds(older hardware)


Really? Do we care about 17-81μs?

YES


Really? Do we care about 17-81μs?

• Intel Core i7 E5-2690 with turbo boost (3.5-3.8Ghz)“Sandy Bridge” 22nm processor

• LinX v0.6.4 (Linpack v10.3.4.007) benchmarkMeasures floating point operations per second

• 81.34 GflopsThat’s 81,340,000,000 floating point operations per second

17μs = 137,757,800 floating point operations

81μs = 656,375,400 floating point operations

Conclusion: yes, we absolutely care about 17-81μs!

Source: http://www.anandtech.com/show/4503/sandy-bridge-memory-scaling-choosing-the-best-ddr3


Network latency matters

HPC apps can do a LOT of computationduring network communication

Latency


How can we reduce network latency?

Hardware is faster than software.

The sooner software canhand off to hardware, the better.


Source server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Destination server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Ethernet switch

Port A

Port B


200 nanoseconds

299,792,458 m/s 299,792,458 m/s




Source server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Destination server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Ethernet switch

Port A

Port B


200 nanoseconds

299,792,458 m/s 299,792,458 m/s



Can’t do much about the speed of light


Source server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Destination server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Ethernet switch

Port A

Port B


200 nanoseconds

299,792,458 m/s 299,792,458 m/s




Fastest Ethernet switches today are about 200ns(they’ll probably get a little faster over time)


Source server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Destination server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Ethernet switch

Port A

Port B


200 nanoseconds

299,792,458 m/s 299,792,458 m/s




Fastest Ethernet switches today are about 200ns(they’ll probably get a little faster over time)

8-40us is, by far, the biggest chunk of timeReduce this!


Source server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Reducing server-side latency

What if we can skip some of these layers?

Who needs TCP? Raw L2 Ethernet frames, baby!

Who needs the operating system driver?


Source server

HPC application

Operating system

MPI middleware

TCP stack

NIC driver

NIC hardware

Reducing server-side latency

What if we can skip some of these layers?

Who needs TCP? Raw L2 Ethernet frames, baby!

Who needs the operating system driver?

Let MPI talk directly to the NIC hardware


Hardware usually only seen by the OS

Linux userspaceapplication

Linux kernel

Cisco VIChardware

Can I seethe hardware?

Please?


Hardware usually only seen by the OS


Linux kernel

Cisco VIChardware

No.


…except in special cases


Linux kernel

Cisco VIChardware

Can I seethe OpenFabrics

hardware?




Linux kernel

Cisco VIChardware

Sure!




Linux kernel

Cisco VIChardware

Can I seethe OpenFabrics

hardware?Yay!


OpenFabrics Linux support

• Coalition of network vendors

• Successfully upstreamed “OS bypass for networking” into Linux

• http://www.openfabrics.org

http://www.openfabrics.org/


Source server

HPC application

Linux

MPI middleware

VIC driver

Cisco VIC

Enabling direct MPI communication

Our project: enabling this MPI direct-to-hardware communication on Cisco serverswith the Cisco Virtual Interface Card (VIC)

in Linux.

Everything above the firmware will beopen source.


Kernel

Cisco VIC hardware

TCP / IP stack

Cisco VIC driver

Normal TCP software architecture

UserspaceUserspace sockets library

MPI library

Application


Kernel

Userspace verbs library

Cisco VIC hardware

Cisco USNIC software

MPI library

Userspace

Verbs IB core

Cisco USNIC driver

Bootstrappingand setup

Send and receivefast path

Application

Cisco codehere


…and the 4th project




• Query your server’stopology

• NUMA nodesIncluding memory

• Processor sockets

• L3, L2, L1 cachesInstruction and data

• Cores

• Hyperthreads

• PCI devices


hwloc CLI

• Output formats supported:PDF, JPG, PNG, TIFF, FIG, …

Text (for console windows)

Curses

XML

• Great for feeding into scripts!


hwloc-bind: replaces numactl

• hwloc-bind socket:0.core:2 commandBind command to core 2 on socket 0

• hwloc-bind –getPrint a bitmap of your current bindings

• hwloc-bind --get | hwloc-calc -p -H socket.corePrint something more readable than a bitmap

• hwloc-bind --get | hwloc-calc -p -H socket.core.puEven show the hardware threads

• hwloc-ps [-a]Show where processes are bound


hwloc C API

• Get a tree data structure representing the topology

• Many API calls for manipulating / traversing the tree

• Typical actions:Get, set processor and memory bindings

React to cache sizes

• …everything you can do in the CLI, and more


Why is hwloc useful?

• Verify the internal topology of your serverHow much memory do you have?

Where is that memory?

What processor(s) are local to that memory?

How big are your L1, L2, L3 caches?

• Verify your internal PCI devicesDistinguish ethX devices from each other


Why is hwloc useful?

• Bind services to specific coresEnsure related services are on the same NUMA node

Put non-essential services on core 0 (e.g., NTP)

• Bind server-related servicesApache, Bind, NFS, …etc.

Increase performance by not letting them migrate

Keeps memory local, less inter-NUMA-node traffic

NTP etc.

Apache

NFS


Cisco Public 9797© 2013 Cisco and/or its affiliates. All rights reserved.

“Open source is good.Open source works.

Thank you.

Date post:	12-May-2015
Category:	Technology
Upload:	jeff-squyres
View:	1,184 times
Download:	2 times

MOSSCon 2013, Cisco Open Source talk

Technology