Date post: | 11-May-2015 |
Category: |
Technology |
Upload: | stefan-freitag |
View: | 882 times |
Download: | 0 times |
D-Grid Infrastructure
Stefan FreitagDortmund University of Technology
6th International Workshop on Grid Computing for Complex Problems
Bratislava, November 2010
Outline
● Projects & Infrastructure● Lessons learned & Future Challenges
2004 2007 20102009 20112008 2012
Gap project
20062005
Integration project
Service layer
Academic projects
Public/ private projects
D-Grid GmbH
Working Groups
Pre-project Phase 1
Phase 2
Phase 3
D-Grid Projects
Gap projects
Service layerprojects
Commercial communityprojects
Academic communityprojects
Integration project
Project structure of D-Grid
Representation (?)
Academic service and resource providers
Academic research communities
D – Grid GmbH
Industrial service and resource providers
Organizational issues
Legal issuesInternational
issues
Public Relation
GAUSS-Allianz WissGrid
Representation structure in D-Grid
Academic projects
Astronomy
Climate Research
High Energy Physics
Medical Science
Humanities
Aircraft Industry
Financial Business
Automobile Industry
Building & Construction Industry
Geospatial Information Systems
Medical Services
Meteorology for regenerative energy industry
Commercial Projects
Commercial Projects
Logistics
Media Technology
Chemical Industry
Numerical Simulation
Medical Engineering
Plasma Technology
Dynamic Service Value Networks
Service Projects
Backup & Recovery
Sustainable Grid Infrastructure
SLA4D-Grid
Networks of Knowledge in the Grid
Gap & Service Projects
VO management
D-MON
DGSI
GapSLC
GIDS
D-Grid Infrastructure
Situation in 2005
High energy physicists● advanced knowledge (CERN experiments)● gLite middleware
Other communities● not really any prior knowledge
How to make all communities happy?● Let them choose!● Community requirements still unclear● Offer a set of commonly used middlewares
Leere Fusszeile
Supported Middlewares
Compute middleware● gLite● UNICORE● Globus Toolkit
Storage middleware● dCache● OGSA-DAI
Leere Fusszeile
Software Stack @ D-Grid Resource
Leere Fusszeile
D-Grid Reference System
● Template for a D-Grid compute and storage resource
● Targeted at small and medium resource centers
● Technical documentation & software packages available to system administrators
● http://dgiref.d-grid.de/wiki/Introduction
● Validation of packages and documentation by installation “from scratch”
● Release Cycle: six months
● Reference system is only a recommendation
● Support available
D-Grid Resources
● > 30.000 CPU cores
● > 5 PByte of storage
● In 2009 >= 140 Mio. CPUh consumed by users
● Accounting ● DGAS/ manually
Usage of D-Grid Resources
Towards Sustainability
Technology is not really a problem!
Focus on existing concepts
● Support for different Grid middlewares
● Virtualization
Bridging the gap to services and applications
● Workflow engines
● Commercial software
Addressing new requirements
● Dynamic virtual organizations
Cooperation between different projects
● Scheduling
Heterogeneous Provider Community
Some large hardware providers play a vital role in D-Grid
● Development of a business model for basic services
Some providers used D-Grid to acquire new hardware
● Investments to support a transitional phase to Grid systems
No known concepts to provide IaaS type of services
● Obstacle for commercial users
● Competition to cloud computing?
There are not enough users!
Prominent figures of the various disciplines have not yet joined● Imbalance between computer science and application science
Most users are not willing to provide the required additional effort. ● Potential lack of focus in the application area
Industrial users are pushed off by bureaucratic obstacles● Use of D-Grid hardware for commercial purpose
Grid computing is a solution, isn’t it?
Funding agencies cannot be wrong!
● More than 100 Mio € in D-Grid
Industry cannot be wrong!
● Enterprise Grids
● Cloud Computing
Science cannot be wrong!
● LHC production Grid
Due to the weather conditions... Cloud Computing!
Photo by: Paolo Malgari
Cloud Computing in D-Grid ?
Situation in 2009● Utilization of compute resources ≥ 140 Mio CPUh
● only around 50% of the available CPUh
½ of D-Grid compute resources idling● Turn them off → GreenIT
● Not really acceptable for a sustainable operation
→ Introduce compute cloud interface as new pillar
Why ?
● Grid is too complex and hence not attractive to SMEs
Attractivity of Cloud Interface I
Cloud● Register once via web. Specify name, credit card
information,... (few minutes)
Grid● Application for X.509 certificate (1-2 days)
Contact Certification authority
Contact Registration authority (f2f required)● Application for VO membership (hours - days)● Updating of the authorization information on the Grid
resources (hours - days)
Attractivity of Cloud Interface II
Cloud Computing in D-Grid
● Extension of the software stack
● What to add?● Platform
virtualization● Compute cloud
middleware
D-Grid Cloud Prototype
Located at Dortmund University of Technology
Services● Compute middlewares
– gLite 3.2
– UNICORE 6
– Globus Toolkit 4.0.8
● Storage middlewares
● dCache 1.9.5
● OGSA-DAI WSRF 2.2
Hardware● 2048 CPU cores, 108 TByte storage
All services deployed in Xen virtual machines
OpenNebula
Statement on homepage● Standard-based open-source toolkit to build private, public and
hybrid clouds. [...] provides flexible architecture, interfaces and components that fit into any existing data center.
Supports Xen, KVM and VMware
Interfaces with Amazon EC2 and ElasticHosts
Other users● CERN (cloud infrastructure consisting of 3,200 cores)
● NIKHEF, Netherlands
Prototyp Setup
Open Issues
Technical level● User management● Authorization● Accounting and Billing● Monitoring● Information system
→ Interfaces to all D-Grid core services
User Management
D-Grid● VO central user management
● Resources query VOMRS servers and generate a mapping
● VO member → local user ID
● Attribute-based authorization
Cloud (OpenNebula)● Local user management,
● No Attribute-based authorization
Can the concept of VOs and Cloud middleware go together?
Lessons learned
Reference System Acceptance
Only a few sites ● Follow the recommended setup● Use the technical documentation
Why should a resource install all middlewares?● System administrators are not qualified to support all
middlewares ● Lack of interest at resource provider level
Migration/ Upgrade path unspecified
Legal Issues – Dual Use
Dual-use technology: a technology which can be used for both peaceful and military aims.
Most industrial countries have export controls on certain types of designated dual-use technologies […] These controls restrict the export of certain commodities and technologies without the permission of the government.
(Wikipedia)
● Is Grid a dual-use technology?
● Export embargo for dual-use technologies to Iran
● Technical solution for a legal requirement
Legal Issues – Liability
With the development of business models, the liability of single VO members and of the whole VO is of interest
What is the legal status of a virtual organization?● Company constituted under Civil law
→ Joint and several liability● In joint and several agreements, the liability for default is
enforceable against all of the signatories as a group[...], or against any one of them as an individual at the choice of the enforcing party (www.businessdictionary.com)
Can business models and virtual organizations co-exist?
Avoid Parallel Universes
Before D-Grid German HEPs used WLCG resources
In 2010 German HEPs still use WLCG resources
● and not a single one uses D-Grid resources
Why?
● Accessibility to compute resources, data and experiment software is limited by the boundaries of a virtual organization
● In WLCG VOs for ALICE, ATLAS, CMS and LHCb were already existing
● In D-Grid a HEP-CG VO was established with the aim to absorb all HEPs independent from their origin
● Even if HEPs joined the HEP-CG VO, they were not able to access the experiment software
Future Challenges
Future Challenge: D-Grid/ NGI-DE
EGI was still EGEE as D-Grid started
With the end of EGEE-III, NGI-DE was established
Since then D-Grid and NGI-DE co-exist in Germany
Merge to only one initiative
● Avoid competition
● Combine endeavors for creating a sustainable infrastructure
● Removal of duplicity of service (GRRS/ GOC DB)
● Integration of D-Grid services (D-Mon)
● Operations concept (1:1 mapping, pool accounts)
New Service: D-Mon
Summary
Projects in D-Grid
D-Grid Infrastructure● Software stack, resources● Prototypic integration of a Cloud middleware
Lessons learned
Legal issues & Future challenges
Thank you for your attention!
If you found this talk interesting, slides are available at
http://www.slideshare.net/stefanfreitag