www.egi.eu EGI-Engage isco-fundedbytheHorizon2020FrameworkProgrammeoftheEuropeanUnionundergrantnumber654142
LeveragingGPGPUComputinginGridandCloudEnvironments:
FirstResultsfromtheMobrain CC
AlexandreBonvin
24/8/16
Mainactivities
• Task1:Usersupportandtraining 9PM• Task2:Cryo-EMinthecloud:bringingcloudstothedata 23.4PM• Task3:Increasing thethroughputefficiencyofWeNMR 0PM
portalsviaDIRAC4EGI• Task4:CloudVMsforstructuralbiology 0PM• Task5:GPUportalsforbiomolecularsimulations 14PM• Task6:Integratingthemicro(WeNMR/INSTRUCT) and
macroscopic (NeuGRID4you)VRCs 11PM
• TOTALfundedeffort 57.4PM
34/8/16
mobrain.egi.eu§ Bringsthemicro(WeNMR)andmacro
(N4U)worldstogetherintoonecompetencecenterunderEGIEngage:
§ Withactivitiestoward:§ Integratingthecommunities§ Makingbestuseofcloudresources§ Bringingdatatothecloud(cryo-EM)§ ExploitingGPGPUresources
§ Whilemaintainingthequalityofourcurrentservices!
44/8/16
WeNMRVRC (December2015)
• Oneofthelargest(#users)VOinlifesciences• >720VOregisteredusers(36%outsideEU)• >2250VRCmembers(>60%outsideEU)• ~41sitesfor>142000CPUcoresviaEGIinfrastructure• User-friendlyaccesstoGridviawebportals
www.wenmr.eu
NMRSAXS
Aworldwidee-Infrastructure forNMRandstructuralbiology
54/8/16
SustainedgrowthoftheWeNMRVRC
0
250
500
750
1000
1250
1500
1750
2000
2250
2011
-01
2011
-03
2011
-05
2011
-07
2011
-09
2011
-11
2012
-01
2012
-03
2012
-05
2012
-07
2012
-09
2012
-11
2013
-01
2013
-03
2013
-05
2013
-07
2013
-09
2013
-11
2014
-01
2014
-03
2014
-05
2014
-07
2014
-09
2014
-11
2015
-01
2015
-03
2015
-05
2015
-07
2015
-09
2015
-11
End of WeNMR
EU funding
64/8/16
Sustained#ofjobs
End of WeNMR
EU funding
74/8/16
• SupportforMoBrain/West-Lifeactivites
• 75MCPUhours,50TBstoragefrom7sites– INFN-PADOVA(Italy)– RAL-LCG2(UK)– TW-NCHC (Taiwan)
– SURFsara (TheNetherlands)– NCG-INGRID-PT (Portugal)– NIKHEF(TheNetherlands)– CESNET-MetaCloud (Czech
Republic)
NewSLAagreement
84/8/16
z
x
y
6D cross-correlation
search
Automatic rigid body fitting of biomolecular structures into cryo-EM densities
Exhaustive 6D search of 3 translational and 3 rotational degrees of freedom
Calculate cross-correlation at every scanned position
Python package
Simple command-line program, able to run using single/multiple CPUs or GPU
ExploringGPGPUresources:PowerFit
vanZundertandBonvin.AIMSBiophysics 2,73-87(2015)www.github.com/haddocking/powerfit
94/8/16
Fast Fourier Transform for fast translational
scans
GPU acceleration
Optimised rotation sets
Resampling and trimming target
z
x
y
Speedingupthesearch
104/8/16
Required input:• PDB file• Cryo-EM density• Resolution
Optional:• Rotational sampling interval• Laplace filter• Core-weighted correlation• Number of solutions to file• Number of processors to use• Offload to GPU
PowerFitP
rotocol
114/8/16
ExploringGPGPUresources:DisVis
• Pythonpackagetovisualizeandquantifytheaccessibleinteractionspaceofdistancerestrainedbinarybiomolecularcomplexes.
• Simplecommand-lineprogram,abletorunusingsingle/multipleCPUsorGPU
vanZundertand Bonvin.Bioinformatics.31,3222-3224(2015)www.github.com/haddocking/disvis
124/8/16
PerformanceofCPUvsGPUinhouseresources
• CPU:AMDOpteron6344usingFFTW3• GPU:NVIDIAGeForceGTX680usingclFFT
PowerFit
DisVisSystem Number of
complexessampled
Time CPU TimeGPU
Speedup
RNA-polymerase II 19 × 109 19h44m 56m 21x
PRE5-PUP2 7× 109 7h 12m 15m 29x
System Mapsize(voxels)
Rotationssampled
TimeCPU
TimeGPU
Speedup
GroEL-GroES 90x72x72 70728 1h29m 4m9s 21x
RsgA into ribosome 72x80x72 70728 1h16m 4m2s 19x
134/8/16
PerformanceofCPUvsGPUinhouseresources
PowerFit
DisVis
144/8/16
RequirementsforPowerFit andDisVis
• Basic:– Python2.7– NumPy 1.8+– SciPy– GCC(oranotherC-compiler)
• OptionalforfasterCPUversion:– FFTW3– pyFFTW
• OptionalforGPUversion:– OpenCL1.1+– pyopencl– cIFFT– gpyfft
Solutionforgridandcloudcomputing:
Docker containersbuiltwithproperlibrariesandopencl support:
Basedockerfile withopencl:https://github.com/indigo-dc/docker-opencl
Dockerfile forDisVis:https://github.com/indigo-dc/docker-disvis
154/8/16
AccessingGPGPUresourcesviagrid
• TestbedatCIRMMPinFlorence• Newjdl requirementsforGPUs:
executable = "disvis.sh";inputSandbox = { ... };...GPUNumber=2;
• DirectsubmissiontoFlorencetestbedCE:glite-ce-job-submit -o jobid.txt -a \-r cegpu.cerm.unifi.it:8443/cream-pbs-batch disvis.jdl
164/8/16
AccessingGPGPUresourcesviagrid
• Exec script with docker command to access the GPU:docker run --device=/dev/nvidia0:/dev/nvidia0 \
--device=/dev/nvidia1:/dev/nvidia1 \
--device=/dev/nvidiactl:/dev/nvidiactl \
-v $WDIR:/home opencl_disvis /bin/sh \
-c 'export LD_LIBRARY_PATH=/usr/local/lib64; \
disvis /home/O14250.pdb /home/Q9UT97.pdb /home/restrain
• The docker container was built directly on the site with cuda5.5 and is thus site-specific (hardware dependencies)
174/8/16
AccessingGPGPUresourcesviacloud
thankstoMarioDavid
• Build a opencl docker container with the proper driver for the graphic card (i.e. site specific) (cuda 7.x)
• https://github.com/indigo-dc/docker-opencl– Dockerfile based on ubuntu_nvidia_opencl + disvis
184/8/16
Baremetal vsgridvscloud
ID Type GPU #CoresCPUtype Mem(GB)
B-K20 Baremetal TeslaK20 24HT(12real) Intel(R)Xeon(R)[email protected] 32
B-K40 Baremetal TeslaK40 48HT(24real) Intel(R)Xeon(R)[email protected] 512
D-K20 Docker onK20 TeslaK20 24 Intel(R)Xeon(R)[email protected] 32
K-K40 KVMonK40 TeslaK40 24 Intel(R)Xeon(R)[email protected] 32
Case MachineTimeGPU(sec)
TimeCPU1core CPU1/GPU
B-K40 Baremetal 674 7928 11.8
K-K40 KVM 671 7996 11.9
B-K20 Baremetal 830 11839 14.3
D- K20 Docker 837 11926 14.3
Nolossofperformance
CourtesyofM
arioDavid
INDIGO
<- Grid
<- Cloud
194/8/16
Conclusions
• GPGPU-enabledsoftwaresuccessfullyportedtobothgridandcloudenvironmentsusingdocker containers
• Thedocker containersaresite-specific(becauseofthedriverdependencies)
• SuccessfulgLite-basedsubmissiontotestbed
• Webportalindevelopment
• MoreGPGPUworkdone(refertoAntonioRosato’spresentationofWednesday)
204/8/16
Conclusions
www.egi.eu
Thankyouforyourattention.
Questions?
EGI-Engage isco-fundedbytheHorizon2020FrameworkProgrammeoftheEuropeanUnionundergrantnumber654142
Withthanksto:- MarcoVerlato (MoBrain/INDIGO)- AntonioRosato (MoBrain/INDIGO)- ZeynepKurkcuoglu (INDIGO)- MarioDavid(INDIGO)
