HelixNebulaScienceCloud6th LSDMASymposium

KIT

29th August2017João Fernandes

CERNJoao.Fernandes <at>cern.ch

HelixNebula– TheScienceCloudwithGrantAgreement687614isaPre-CommercialProcurementActionfundedbyH2020FrameworkProgramme

Outline

Pre-CommercialProcurementHybridCloudChallengesFunctionalRequirementsContractorsDesignsTestingFutureAdoptersNextSteps

9/4/17

HNSciCloud JointPre-CommercialProcurementProcurers:CERN,CNRS,DESY,EMBL-EBI,ESRF,IFAE,INFN,KIT,STFC,SURFSaraExperts:Trust-IT&EGI.eu

Thegroupofprocurershavecommitted• Procurementfunds• Manpowerfortesting/evaluation• Use-caseswithapplications&data• In-houseITresources

Resultingserviceswillbemadeavailabletoend-usersfrommanyresearchcommunities

Co-fundedviaH2020GrantAgreement687614

Totalprocurementbudget>5M€

9/4/17

HNSciCloud projectphases

Preparation

•Analysisofrequirements,currentmarketoffersandrelevantstandards

•Buildstakeholdergroup•Developtendermaterial

Implementation andsharing

Jan’16 Dec’18

Eachstepiscompetitive - onlycontractorsthatsuccessfullycompletethepreviousstepcanbidinthenext

5

4Designs 3Prototypes 2Pilots

Call-offFeb’17

Call-offDec’17

TenderJul’16

PhasesofthetenderaredefinedbytheHorizon2020Pre-CommercialProcurementfinancialinstrument

9/4/17

WhatwillbeprocuredAhybridcloudplatformfortheEuropeanresearchcommunity

6

HNSciCloudPCP

Source:C

loud

Compu

tingforG

ovies,DL

TSolutio

ns,

DavidBlankenh

orn,VanRistau

andCa

ronBe

esley

CombiningservicesattheIaaSleveltosupportscienceworkflows

TheR&DservicestobedevelopedaretobeintegratedwithResourcesindatacentresoperatedbythebuyersgroupGEANTnetwork

9/4/17

ChallengesInnovativeIaaSlevelservicesintegratedwithprocurersin-houseresourcesandpublice-infrastructureaspartofahybridcloudtosupportarangeofscientificworkloads

ComputeandStoragesupportarangeofvirtualmachineandcontainerconfigurationsincludingHPCworkingwithdatasetsinthepetabyterange

NetworkConnectivityandFederatedIdentityManagementprovidehigh-endnetworkcapacityviaGEANTforthewholeplatformwithcommonidentityandaccessmanagement

ServicePaymentModelsexplorearangeofpurchasingoptionstodeterminethosemostappropriateforthescientificapplicationworkloadstobedeployed

79/4/17

ThePre-CommercialProcurementprocess

04/09/17

BaselineofServicesRequiredIaaS

HTCVMconfigurations4-32vCPUs,2-8GBRAM

HPCaaS (lessthan5%ofthetotal)64vCPUsandmore,64GBRAM

DataManagementGlobalnamespaceregardlessofthelocationPOSIXsupportAutomateddatareplication

Network(Contractor’sDCstoGEANTVRF)Prototype:10Gbps,Pilot:40Gbps

IAMSupportforeduGAIN &ELIXIRFederatedAAI

APISupportTerraform,ObjectStores,COEs(Kubernetes,Docker),Monitoring,etc.04/09/17

9/4/17

04/09/17

04/09/17

04/09/17

HNSciCloudPrototypeActivityReport Unclassified 14

OVERVIEWOFPROTOTYPECLOUDPLATFORM

Integrated

Realistic

MinimallyIntrusive

Scalable

Managed

Sustainable

Functionalcomponentstodelivercompletedataanalysisplatform…

HNSciCloudPrototypeActivityReport Unclassified 15

OVERVIEWOFPROTOTYPESOLUTIONFunctionalcomponentstodelivercompletedataanalysisplatform…

HNSciCloudPrototypeActivityReport Unclassified 16

OVERVIEWOFPROTOTYPESOLUTIONFunctionalcomponentstodelivercompletedataanalysisplatform…

HNSciCloudPrototypeActivityReport Unclassified 17

OVERVIEWOFPROTOTYPESOLUTIONFunctionalcomponentstodelivercompletedataanalysisplatform…

HNSciCloudPrototypeActivityReport Unclassified 18

OVERVIEWOFPROTOTYPESOLUTIONFunctionalcomponentstodelivercompletedataanalysisplatform…

Martin Gasthuber | Next Steps | 03/04/2017 | Page 19

Test Phases & Resource Ramp-Up

TP1 TP2 TP3 TP42000C200TB

100C 10TB

3500C350TB

TestSuiteCERN• Edugain AAI• CERNBenchmarkSuite• S3Endpoints• PerfSONAR Network• VMProvisioningandPersonalisation• GridJobsinDocker• Dockerized deploymentofEOS+CERNBOX+SWANCNRS• IAASaccessviaEduGain andlocalaccountsDESY• HDF5_IOEMBL• CLOUD_BENCHMARK• ELIXIR_AAI• data_transparencyESRF• FDMNES

IFAE• monitoring-accounting• imagecreationandcontextualizationINFN• BelleII_SIM• COMPASS/TGEANT_IO• INFN/GPFS_AFM• DynfarmKIT• static-batch-extension• dynamic-batch-extension• AFM-deployment-testSTFC• batch-system-extension• jobs-on-kubernetesSURFsara• BBMRi_data• dCache_io• kubertest

Moredetails:https://cernbox.cern.ch/index.php/s/zCd8KRtBUfH9yjF

9/4/17

Specsvs.Tests

Criteria 3.2 - Standard IaaS Service CERN,KIT,EMBL

Criteria 3.3 - Storage CERN, SURFsara,INFN,DESY

Criteria 3.4 - Transparent Data Access DESY

Criteria 3.5 - Data Ingress/Egress DESY

Criteria 3.6 - Data Location EMBL

Criteria 3.7 - Container Support DESY

Criteria 3.8 - Networking ALL

Criteria 3.9 - Identity and Access Management CERN,EMBL,KIT

Criteria 3.10 - Service Capacity KIT

Criteria 3.11 - Service Availability ALL

Criteria 3.12 - Service Elasticity KIT

Criteria 3.14 -Service Reporting & Usage Monitoring IFAE

Criteria 3.15 - Cloud Service Support ALL

Criteria 3.16 - HPCaaS ESRF

ActivitiesinTestPhase1(TP1):April-June2017

EstablishingNetworkConnectivityviaGEANT/NRENsFederatedIdentityviaeduGAIN &ELIXIR

FineTuningTenancydefinitionandisolationrequirementsTokenandaccountterminationpolicies

Creatingtenantsandaccesses04/09/17

ActivitiesinTestPhase2(TP2):June-September2017Benchmarking

KV,Dirac,WhetstonePhoronix

ValidationscriptsofVMflavours

NetworktestsCommissioning

10GbpstesttoabaremetalperfSONAR serveratContractorsDCw/10GbpsNICRegulartestasthelinkgetsused

Throughput,Latency,PacketLoss

FederatedAAIAuthC attemptsinseveralscenariosCompliantattributereleaseAuthZ testsatTenancy,sub-Tenancylevels

ObjectStoresfunctionaltestsS3operationsPUT,GET(AuthC,Single/Multiplechunks,prefixmatching)

NextSteps20172nd progressreview4th-5th September

Scale-upIaaSresourcesMoretesting

3rd progressreview9th-10th October(DESY)LivewebcastdemonstrationsFeaturefreezeScale-upIaaSresources(totalling3.5kcores350TBstorage)Finalizethetests

Endofprototypephaseassessment11th -13th December

Selectionof2/3prototypesforpilotphase

04/09/17

PilotPhase2018Kick-offevent6th-8th Feb2018(INFNBologna)

Progressivelyscale-upIaaSresourcestotalling20kcores2PBstorage

FocusonPerformance,Scalability,Portability,Security

Deploymentofapplications

Assessprocurement/paymentmodels

04/09/17

Growingthebuyersgroup

Privatesector

Gov.agencies

Additionalresearchorgs.

Procuringbeneficiariesof

ECproject

Theinitialgroupofbuyersisthesetofresearchorgs.thatcommittedtheirresourcesatthestartoftheprojectandbecamebeneficiariesoftheECproject

Goalistoprogressivelygrowthebuyersgroupbyincludingmorepubliclyfundedresearchorganisations:Early adoptergroup

Draftdocumentdescribesbenefitsforthestakeholders,procurementprocess,stepsfor implementation

CERN,DESY,EMBL-EBI,ESRF,KIT,CNRS,INFN,PIC-IFAE,SURFsara,STFC

adoptergroup

Summary3prototypes(IBM,T-SystemsandRHEA)arebeingdeployedandtestedbytheBuyersGroup

Testingandscalingwillcontinueuntiltheendof2017

2/3prototypeswillbeselectedforlarge-scalepilotsin2018

TheEarlyadoptergroupwillallowmoreresearchorganisationstoprocuretheresultingservices

04/09/17

ThankYou!

04/09/17