Pre-Commercial Procurement proposal - HNSciCloud
13 May 2015 Bob Jones, CERN
Pre-Commercial Procurement proposal
• A Pre-Commercial Procurement proposal was submitted to the H2020 ICT 8a call in April 2015 for innovative IaaS cloud services
• The proposal is currently under evaluation by the EC• Official feedback expected in September• These slides give an overview of the proposal
The Helix Nebula Initiative
Strategic Plan
Establish multi-tenant, multi-provider cloud infrastructure
Identify and adopt policies for trust, security and privacy
Create governance structure
Define funding schemes
To support the computing capacity needs for the LHC
experiments
Setting up a new service to simplify analysis of large genomes, for a
deeper insight into evolution and biodiversity
To create an Earth Observation
platform, focusing on earthquake and
volcano research
To improve the speed and quality of research for finding
surrogate biomarkers based on brain
images
Adopters
Suppliers
Additional Users:
3 Bob Jones, CERN
• H2020 Coordination and Support Action
• Total budget: 500 K€• Duration: 18 months • Start date: 1 Oct. 2014• Partners:
4
Example HNX procurementCERN price enquiry for a small fraction of the resources necessary to run the ATLAS experiment simulation software
Several valid responses were received and Atos (data centre in Tenerife, Spain) was awarded the contract accessed via the SlipStream broker and provisioning engine (from SixSq SME) and delivered over the GÉANT network1.2 million CPU hours over 6 weeks producing 11.5 million GEANT4 tt̅� events with a high level of efficiency for CPU intensive workload that confirmed
the use of commercial IaaS is technically feasible for simulation workloadsvariations in RESTful APIs can be overcome with reasonable technical effortindependent monitoring of the quality of service and resources consumed is essential
Bob Jones, CERN5
• Hybrid – link public research organisations, e-Infrastructures & commercial cloud services• Use GEANT network to link Research Infrastructures, repositories
(EUDAT, OpenAIRE), EGI, PRACE etc. to commodity commercial cloud services (multiple providers)
• A cornerstone of the Open Science Commons*
• Trust - Researchers keep control of the cloud and their data• Guarantee a copy of all the data is kept on public resources• Ensure long-term preservation of the data• Insulate users from changes of service supplier and technology
• Economy - Must be cheaper than the ‘build our own’ approach• Avoid separate ‘silos’ for each Research Infrastructure/Community• Profit from the economies of scale in commercial data centres
* http://go.egi.eu/osc
Towards the EuropeanOpen Science Cloud
http://dx.doi.org/10.5281/zenodo.16140
PCP PPI
7
Why PCP?Commercial IaaS exists but not certified, integrated with public e-infrastructures, offering std interfaces with suitable SLA and contractual terms & conditions.
PPIPotential follow-on project if this PCP project is successful
Proposed Joint Pre-Commercial ProcurementThe group of buyers have committed• >1.6M€ of funds• Manpower• Applications & Data• In-house IT resources
To procure innovative cloud services
Integrated into a hybrid cloud model:• Commercial cloud services• e-Infrastructures
• GEANT network• eduGAIN Fed. Id mgmt.• EGI Fed Cloud
• Potentially host data services from EUDAT, INDIGO-Datacloud, etc.
• In-house IT resources
Made available to end-users including BBMI, DARIAH, ELIXIR, EISCAT_3D, EPOS, INSTRUCT, LifeWatch, LHC, etc. Bob Jones, CERN 8
Preparation
• Analysis of requirements, current market offers and relevant standards
• Build stakeholder group
• Develop tender material
Implementation Sharing
• Best practises• Recommendations• Training
Oct’16
PCP project phases
Launch of tender Pilots tested & assessed
6 months 18 months 6 months
Jan’16
Jan’18
Jun’18
All procuring organisations must sign letter to re-confirm commitment of funds
DesignPrototy
pe
Pilot
PCP project implementation phase
Design Prototype
Pilot
Bidders provide written design reportincluding architecture and technical design of components together with unit costs
Contractors build prototypes including all components and make it accessible to experts from buyers group that perform functionality and security tests
Contractors deploy expanded prototypes that are made available firstly to experts who perform scalability tests and then to end-users
Bidders have 3 months to respond to bid
select ≥3 designs taking into account tender criteria
select ≥2 prototypes taking into account feedback from experts
Evaluation committee includes representatives from all buyers
Multiple iterations with feedback to suppliers in prototype and pilot phases
PCP project implementation phase3 phases:• Solution design
• ≥3 designs• 3 months • 15% tender budget
• Prototype development• ≥3 prototypes• 6 months• 25% tender budget
• Deployment of limited scale prototypes• ≥2 pilot deployments • 6 months• 60% tender budget
Design15%
Prototype25%Pilot
60%
Total procurement funds available: ~5M€Each phase is competitive. Bids evaluated/tested against published criteria
Project organisation
WP1: Consortium Management
WP2: Preparation of procurement &Tendering
WP3: Design Platform
WP4: Prototype Platform
WP5: Pilot Platform
WP6: Pilots Evaluation & Recommendation
WP7: Stakeholder Engagement, Dissemination, Impact & Exploitation
CERN
CERN CNRS DESY INFN EGI.eu
TRUST-IT
The EC funding covers the procurement costs, project and procurement process administration as well as work package leadershipAll the buyers provide technical effort to evaluate designs/prototypes/pilots
User GroupsThe cloud resources procured will be made available to user groups during the pilot phase
• LHC experiments via WLCG• Procured resources will count against the buyers’ pledges (during pilot phase)• CERN will provide the interface via Tier-0 (OpenStack in tender spec.)
• ELIXIR• Managed by EMBL-EBI via the ELIXIR Compute platform
• Other research communities via EGI Fed Cloud• Request OCCI interface in tender spec.
• Local users at each buyers site• Each buyer is responsible for integration• Request web GUI interface in tender spec.
• Sites can also use OpenStack or OCCI interface as well
Each buyer decides what fraction of their procured resources is made available to each user group but cannot assign only to their local users
Collectively the users will form a user group with a role in the project to define requirements and provide feedback on pilot deployments
BBMRIDARIAHEISCAT_3DEPOSINSTRUCTLifeWatchLong-tail
Next Steps• CERN will perform another limited scale procurement in
preparation for the PCP• More details at the 6th Helix Nebula General Assembly @ CERN
• 24-25 June restricted to NDA signatories https://indico.cern.ch/event/369721/
• 26 June open day http://indico.cern.ch/e/OpenDay
• Assuming the PCP proposal is funded• September-December 2015:
• Approval of changes to CERN procurement rules to support EC co-funded PCP
• Prepare EC contract and consortium agreement• Make more detailed planning of tasks• Start work on tender material• Investigate to support data-intensive workloads
• Project start January 2016