Date post: | 18-Dec-2015 |
Category: |
Documents |
View: | 215 times |
Download: | 0 times |
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 1
Organized by
PBSGridWorks, AltairPresentation
by
S.THAMRAI SELVI
Professor and Head
Department of Information Technology
MIT Campus, Anna University
Chromepet, Chennai
on 31.08.2008
Virtualization and Grid @
GRID 08, Bangalore
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 2
Agenda
Introduction Virtualization Trust management in Grid Semantic discovery of Resources in Grid Conclusion
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 3
The laws influencing Information Technology
The laws that are generally accepted as governing the spread of technology:
1. Moore's Law
2. Gilder's Law
3. Metcalfe's Law
4. Disk Law
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 4
IT Architecture Trends
Resource Sharing
Fle
xib
ility
Range of solutions
Infrastructure Consolidation
VirtualComputing
Service OrientedArchitecture
Web Services
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 5
Service Oriented Architecture modelService Oriented Architecture model
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 6
XML Web Services FoundationSimple, Open, Broad Industry SupportPublish, Find, Use Services:Service Descriptions :
UDDIWSDL
Service Interactions: SOAP
Universal Data Format: XML
Ubiquitous Communications: Internet(HTTP)
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 7
Web 2.0: Evolution Towards a Read/Write Platform
Web 1.0(1993-2003)
Pretty much HTML pages viewed through a browser
Web 2.0(2003- beyond)
Web pages, plus a lot of other “content” shared over the web, with more interactivity; more like
an application than a “page”
“Read” Mode “Write” & Contribute
“Page” Primary Unit of content
“Post / record”
“static” State “dynamic”
Web browser Viewed through…
Browsers, RSS Readers, anything
“Client Server” Architecture “Web Services”
Web Coders Content Created by…
Everyone
“geeks” Domain of… “mass amatuerization”
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 8
Core concepts used in Web 2.0 AJAX- Asynchronous JavaScript + XML (User
interface/Portal) JSON - JavaScript Object Notation (SOAP) REST Representational State Transfer
(WSDL). REST proposes to use Internet as programming platform with only HTTP
Widgets, Gadgets, Start page for Portal
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 9Source: http://blogs.zdnet.com/Hinchcliffe/
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 10
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 11
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 12
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 13
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 14
Today’s IT architecture is complex and unmanageable…
access tier
web tier
application tier
database tier
edge routers
routingswitches
authentication, DNS,intrusion detect, VPN
web cache1st level firewall
2nd level firewall
load balancingswitches
web servers
web page storage(NAS)
databaseSQL servers
storage areanetwork(SAN)
applicationservers
files(NAS)
switches
switches
internetinternetinternetinternet
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 15
Putting together network, storage and computing speed
…rates of growth of digital power
1,000,000,000,000
100,000,000,000
1970
Moore/Transistors
Gilder/Bandwidth
Metcalf/NetworkNodes
10,000,000,000
1,000,000,000
100,000,000
10,000,000
1,000,000
100,000
10,000
1,000
100
101
1975 1980 1985 1990 1995 2000 2005 2010
2,300 6,000 29,000 275,000 1.2 mil 5.5 mil 42 mil 252 mil 1.344 bil
50 50 56 1,544 45,000 145,000 10 mil 2.43 bil 200.49 bil
4 111 200 10,000 300,000 1 mil 140 mil 3.5 bil 300 bil
Moore’s Law. Transistors on a single
chip doubles approximately every 18–24 months.
Gilder’s Law. Aggregate bandwidth
triples approximately every year.
Metcalfe’s Law. The value of a network
may grow exponentially with the number of participants(community Law).
Disk Law Storage doubles every 12
months
Source: Cambridge Energy Resource Associates 10616-17
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 16
The Need for New Technology
“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 17
Technologies influencing a new era in computing…
Microprocessor Technology Optical Networking Technology Storage Technology Wireless Technology Sensor Technology Global Internet Infrastructure WWW and Web Services (SOA) Open Source Movements
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 18
Grid
A kind of open standard distributed
infrastructure that enables flexible,
secure, coordinated resource sharing
among dynamic collections of trusted
resources belonging to diverse
organizations across the globe
ensuring user’s Quality of Service
requirements.
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 19
Grid enables …
Virtual Organization Dynamic Resource sharing
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 20
Building Blocks Network Computing Nodes Storage
PARAM PADMA
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 21
Context
Middleware
Map to
resources
jobs / legacy code /binary executables
Resources
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 22
Overall GRID Architecture
Application
Collective
Resource
Connectivity
Fabric
Application
Transport
InternetLink
GRIDInternet
Source: The Anatomy of the GRID, Foster, Kesselman and Teucke
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 23
Protocols, Services,and APIs Occur at Each Level
Languages/Frameworks
Fabric Layer
Applications
Local Access APIs and Protocols
Collective Service APIs and SDKs
Collective ServicesCollective Service Protocols
Resource APIs and SDKs
Resource ServicesResource Service Protocols
Connectivity APIs
Connectivity Protocols
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 24
Three Generations of Grid
Source: Charlie Catlett
StandardizationStandardization is key for third-generation grids! is key for third-generation grids!StandardizationStandardization is key for third-generation grids! is key for third-generation grids!
• Local “metacomputers“– Distributed file systems– Site-wide single sign-on
• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept
• Local “metacomputers“– Distributed file systems– Site-wide single sign-on
• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept
• Utilize software services and communications protocols developed by grid projects:
– Condor, Globus, UNICORE, Legion, etc.
• Need significant customization to deliver complete solution• Interoperability is still very difficult!
• Utilize software services and communications protocols developed by grid projects:
– Condor, Globus, UNICORE, Legion, etc.
• Need significant customization to deliver complete solution• Interoperability is still very difficult!
• Common interface specifications support interoperability of discrete, independently developed services
• Competition and interoperability among applications, toolkits, and implementations of key services
• Common interface specifications support interoperability of discrete, independently developed services
• Competition and interoperability among applications, toolkits, and implementations of key services
We are here!
We are here!
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 25
Grids – Where to ?
Grid 1.0 – concerned with the virtualization, aggregation and sharing or compute resources
Grid 2.0 – focused on the virtualization, aggregation and sharing of all compute, storage, network and data resources
The promise is that in Grid 2.0 the resources will be easier to define, test, install, transport and adjust on demand.
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 26
Grid 2.0 EmergingGrid 2.0 Emerging
Grid 1.0Grid 1.0Compute Intensive Compute Intensive Cycle AggregationCycle Aggregation
SOASOASoftware Services Software Services with SLA & QoS with SLA & QoS
MetricsMetrics
Virtualization Virtualization Consolidation of Consolidation of
ResourcesResources
Grid 2.0*Grid 2.0* Virtualized Compute, Virtualized Compute,
Storage, Network, DataStorage, Network, Data
Service OrientedService Oriented
Policy Driven AutomationPolicy Driven Automation
Distributed across Distributed across firewallsfirewalls
Parallel, stateless, Parallel, stateless, stateful and transactional stateful and transactional appsapps
*The 451 Group: 'grid 2.0' is focused on the virtualization, aggregation and sharing of all *The 451 Group: 'grid 2.0' is focused on the virtualization, aggregation and sharing of all compute, storage, network and data resources. It is both Service-oriented and automated.compute, storage, network and data resources. It is both Service-oriented and automated.
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 27
Grid 3.0?
(Grid) Middleware
User Applications
Resources
Semantic Grid We
b 2
.0
Not just stitching Web and Grid together (Semantic
Web and Web 2.0 ARE the Web) The Grid is part of the
Web. The Web is a platform for the Grid.
A rich and relevant experience for users and developers.
Smashing and mashing.
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 28
Platform – Infrastructure Rapid Application development Agile, Loosely coupled Designed for Reuse
People - Social Keep it Simple Users count
Grid 3.0? The Participatory Grid.
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 29
NextGeneration
Grids
• En
d-us
er e
mpo
wer
men
t
• Li
fe-s
uppo
rt to
busi
ness
proc
esse
s
• Societal behaviour (millions of self-organising nodes)• Computational semantics, ontologies, meta-descriptions• Pervasive virtual organisations
• Continuously changing requirem
ents
• Grid services developm
ent
environments
Virtualization
End-
User
Vision
Software Vision
Architectural Vision
Sim
plifi
cation
Abstraction
Next Generation Grid(s) – 3-fold Vision
Expert Group Reports:
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 30
Virtualization
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 31
Virtualization
Virtualization is a framework or methodology of dividing the resources of a computer into multiple execution environments, by applying one or more concepts or technologies such as hardware and software partitioning, time-sharing, partial or complete machine simulation, emulation, quality of service (Source: Wiki)
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 32
Benefits from Virtualization Load balancing
Create VM for new work loads
Increased Resource utilization Create VM in the under utilized resources
Reusable One VM created can be reused somewhere
else
Isolation Destruction of one VM doesn’t affect the other
VM
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 33
Continued… High Availability
Reduction in downtime
Improved ROI More production with less Investment,
operational and maintenance
Reduced TCO Eliminate the need of new Infrasturctures such
as Server
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 34
Continued… Live migration
Support suspend resume and migration of created VMs
Persistent and Non Persistent Support for keep/drop the changes made in
VM images
Undoable and Append Allows undo and append options in VM
imagaes
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 35
“If you want people to use your system then make it as easy as possible to use.”Werner Vogels, Amazon CTO
Simple to use is not the same as simpleYes, Grids are doing complicated hard stuff. But why make the developer and user suffer so? Also, Trust is another major issue
Motivation for our Research
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 36
CARE Resource Broker (CRB ) A Grid Meta Scheduler
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 37
User PoolRequest Handler
Job Pool
RESOURCE MANAGER
Information System
Physical Machine Manager
Virtual Machine Manager
Transporter ExecutorAggregator
CONTROLLER
Job Tracker
Host IdentifierHost Pool
Scheduler
DB
Grid Resources
Virtual Cluster Service
CLI or Job Submission Portal
CARE Resource Broker
GridFTP FTP Pre-WS GRAM WS GRAMMDS4GT Components
Drivers
CR
B w
ith
Vir
tuali
zati
on
su
pp
ort
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 38
CRBRequest Handler: Gets the Job request, create Job
instance and store it in the Job Pool.
Controller: Initiates the Aggregator to collect the
information about all available resources and updates the Host object and Host Pool object.
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 39
Continued …
Scheduler:
Gets the unordered resource list and orders the resources based on their trust value and return back to Controller.
Virtual Cluster Service:
The controller gives the resource list to the VCS in order to create virtual cluster.
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 40
Continued…
Transporter
Initializes the RFT and transfer the input and executables into selected host
Executor:
Initializes the GRAM to submit the job into local scheduler.
Aggregator
Initializes MDS to get the resource information in the Grid
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 41
Trust Management System
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 42
We define Trust…
The degree of belief in the resource
provider’s competence to complete user’s
task dependably, securely and reliably in a
specific context at a given time
usersAgent / Resource Broker
Resources
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 43
Trust Integration
Trust MetricIdentification
Trust ValueCalculation
Trust MetricEvaluation
Trust ValueUpdation
TMSTMS
• Identify suitable parameters with which the respective trust can be defined
• Apply suitable methodology to determine the value of those metrics
• Determine the overall trust value using the values for various trust metrics obtained
• Integrate the trust mechanism in the resource broker / Scheduler to find out the most trustworthy resource provider for successful job execution/task completion
• Modify or update the value of trust periodically for each resource provider
Trust Management Life Cycle
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 44
User’sTrust
Broker’sTrust
InformationTrust
EquipmentProvision
Trust
ServiceProvision
Trust
VariousTrusts
Describes relying party’s trust in a service provider. The trustor trusts the trustee to provide a service that does not involve access to the trustor’s resource
•It describes trust in principals for the purpose of accessing resources owned by the relying party. A trustor trusts a trustee to use resources that he owns or controls. •It measures whether a resource provided by the resource provider is trustworthy.• The QoS is the main concern in this case
• It is the belief that information provided by the Information provider is reliable and accurate.
• It is a measure of belief that a resource broker has discovered a trustworthy resource
Types of trust
• Measures whether a resource provider is willing to offer his services to the user.• The previous behaviour / payment record may be considered for this trust
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 45
EquipmentProvision
Trust
Resource Registration Metrics
Resource Performance Metrics
User Feedback Metrics
Dependency Metrics
These metrics reflect reputation of the resource in the user community
These metrics reflect the reputation of the organization . It is used to identify initial trust value of the resource provider
These metrics reflect the throughput of the resources and their QoS
Availability, Bandwidth, Latency, Estimated Execution
Time, Actual Execution time, Success, Failure
Recommendation, Reputation
Government / Private, Registration Number
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 46
TrustUpdation
TrustCalculation
TrustMetrics Evaluation
User Feedback
Resource Registration
ResourcePerformance
Underlying Middleware, Resources
User
Resource
Provider
Block Diagram of Trust Management System
Trust Metrics IdentificationDetermines values of metrics using different methodologies
Determines Overall trust value by using suitable mathematical model
Periodically updates the trust value and stores in the database
Database
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 47
Conventional Gridway Flow Trust Enabled Gridway Flow
Job Submit
<job template>
Gathers AvailableResource
MatchesAgainst JobReq
R1R2 R3
Invokes Scheduling Operation
Performs Matchmaking
Selects and submits
Job Submit
<job template>
Gathers AvailableResource
MatchesAgainst JobReq
R1R2 R3
Invokes Scheduling Operation
Performs Matchmaking
Selects and submits
TMSInvokes TMS
TrustDB
Selects Most TrustedResource
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 48
PATENT Pending Filed Application for patent (Patent
application No.593/CHE/2007 A) Title of the Invention: Trust
Resource Broker International Classification:G06F
17/30 Patent has been published in
Journal No. 16/2007 dated 20/04/2007.
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 49
Integrated with Gridway Meta Scheduler
http://www.gridway.org/doku.php?id=related:related
Developed by other projects Trust Management
Description - This project provides GridWay with a mechanism to evalute the trust of the computational resources. The Trust module consists of modified host attribute parsers and match-making algorithms. Additionally the Trust module needs to modify the information providers of Grid resources.
Contact - This component has been developed by the team of Prof Dr.S.Thamarai Selvi ([email protected]) at Madras Institute of Technology
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 50
Semantic Component
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 51
Semantic Description and Discovery of
Grid Resources and its integration using
Gridbus Broker Semantic Component enables semantic
description and discovery of grid resources.
Protégé-OWL APIs are used to create
knowledge base of grid resources using
a resource ontology template.
MDS is used to aggregate resource
information
We use Algernon Inference Engine to
discover suitable resource
This module has been successfully integrated
With Garuda, a National Grid Computing
Initiative
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 52
Deployment in Garuda Grid
Semantic component has been deployed in Garuda Grid
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 53
Sample Screenshots
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 54
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 55
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 56
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 57
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 58
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 59
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 60
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 61
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 62
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 63
Contact
E-Mail: [email protected]
Web site: www.annauniv.edu/care
31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna
University 64