Date post: | 08-Jul-2015 |
Category: |
Technology |
Upload: | ow2-consortium |
View: | 285 times |
Download: | 1 times |
Project Overview
and perspectives
Andrea Manieri
Engineering Ingegneria Informatica SpA
Former VENUS-C Project Director
www.venus-c.eu
22
What scientists want
ONEAccount.Connect.
Interop.Toolkit
VMsRepo
CDMIProxy
PMES-COMPSs
PMES-GW
AzureAccount.Connect.
Account.And
Billing
EMOTIVEAccount.Connect.
TRE.
VENUS-C API
3
User Community in VENUS-C
3
>=5
3-4
1-2
4
Open Call Mgmt
• 60 proposals received
• 2 parallel evaluations
• 15 selected as pilots
• 5 more experiments
• 15 contracts subject to Italian Laws
• Monitoring of activities
– 7 internal monthly progress reports
– Monitoring Effort consumption
– Four deliverables per pilot
– Final Demo (available from web portal)
Conceptual Architecture
5
Client app / Portal
Execution Environment
Re
sea
rch
er
Traffic RedundancyElimination
Accounting, Billing and Monitoring
Job Management
SDK
Data
Management
SDK
Programming
Model
Enactment
Cloud Data
Management
Interface
Cloud
Infrastructure
Management
Storage
ServicesLocal
Storage
Venus-C API andComponent Interaction
Accounting, Billing and Monitoring
Execution Environment
HD
Local
Storage
Re
sea
rch
er
Storage
Services
Client app / Portal
Data
Management
SDK
Usage Tracker
Cloud Data
Management
Interface
TRE
Programming
Model
Enactment
Job Management
SDK
Cloud
Infrastructure
Management
Accounting
PortalConnector
SOAP WS–*
Sec. Scaling
Sec. Notification
Sec. Job Mngt.
Application Code
VENUS-C Services & SDKs
Accounting Storage Service
SNIA CDMI
OCCI
OGF
RUS/UROGF
BES /JSDL
6
Platform Release
• Rapid release cycles:– Incremental releases (V1.0 V1.5, V2.0)
– Central public download site: http://resources.venus-c.eu
– Feedback infrastructure http://venuscfeedback.codeplex.com
• Common Open Source license (Apache 2.0)
7
Usage of the subsystems:
Different Execution Models
Cloud-sideCloud-side
8
Submission
services
GW /
COMPSs
Submission
services
GW /
COMPSs
Cloud
Storage
Processin
g Workers
Processin
g WorkersAccounting
service
Accounting
service
CDMI
Service
CDMI
Service
Client-side Client-side
Stand-alone client
or ASP
Stand-alone client
or ASP
Local
Storage
Data Mgnt
CDMI Client
Data Mgnt
CDMI Client
Worker
for
Enactor
Worker
for
Enactor
Processing
Processing
Processing
splitter reducer
• High-Throughput Computing through
individual jobs executed by multiple
users or Parameter Sweep.
• Data flow, through the execution of
jobs that have different stages
(communicated by files), which may
have different concurrency level.
• Coordination of jobs expressed as
fine-grain workflows orchestrated
by an enactor.Staging and Scaling
up / down resources
when needed, job
grouping.
Staging and Scaling
up / down resources
when needed, job
grouping.
Agnostic
Interface to
data.
Agnostic
Interface to
data.
Synchronization
of jobs
Synchronization
of jobs
Execution Models
9
T5.2 Building T5.2 Building
Information Information
ManagementManagement
T5.5 BioinformaticsT5.5 Bioinformatics
T5.7 Drug T5.7 Drug
DiscoveryDiscovery
T5.4 Civil T5.4 Civil
Protection and Protection and
EmergenciesEmergencies
T5.3 Data for T5.3 Data for
Science Science ––
AquaMapsAquaMaps
T5.1 T5.1 Structural Structural
Analysis for Civil Analysis for Civil
EngineeringEngineering
Success of the user
community
• The 7 scenarios will be demonstrated during the review.
• A VENUS-C pilots showcase event
celebrated in Pisa 27 June.
• 15 prototypes addressing new
requirements
– Matlab front-end1, Use of “R”,
Customised VMs2 and integration of
desktop computing3.
– 6 deployed in Linux, 8 in Azure and 1 in both.
• Many interesting examples of how Clouds
and VENUS-C help them solve their
problems.
10
Earthquake Propagation
Simulation Portal
11
• Developed by the
Aristotle University of
Thessaloniki.
• When an earthquake
happens, it can be
used to simulate the
propagation of seismic
waves and its impact.
• Automatically it captures
the data from the seismic
registers offering in
nearly real-time, information about the areas affected.
• It will be ineffective to have a cluster of 100 nodes
dedicated for the processing of events that occur rarely.
Virtual Docking in a mixed Volunteer and cloud computing
infrastructure
12
• One of the most computationally
intensive tasks in Drug Design is the
identification of ligands reacting against
specific targets.
• Hundreds of thousands of molecules in
different configurations must be tested.
• Virtual docking is feasible through
widely common tools such as Autodock.
• The University of Westminster, partner of EDGI,
had a tool developed for voluntary computing that
benefits from the elastic provisioning of cloud
resources from VENUS-C to guarantee throughput.
• In this case, 180K molecules were analysed in
VENUS-C with respect to the 38 family of the
manosidase, selecting 9 candidates.
Cloud for Radiotheraphy planning
brings a new exploitation model
13
• Radiotherapy planning based in
Monte-Carlo methods is a highly
accurate model for the estimation
of the doses
– Especially in IMRT.
• Well-known problem in research,
already adapted to Grids and clusters.
– However, research resources cannot be used for exploitation and daily practice in hospital environments.
– Moreover, usage ratios makes the infrastructure costs for this purpose unaffordable.
• By the use of VENUS-C, CESGA is validating the feasibility of
exploiting this result.
Benefits from the VENUS-C
Platform
• Scenarios and pilots concur on: Reduced response time;
Increased problem size and Improved Business Opportunities
– Speed-ups of up to 94x with 100 cores.
– Data increase of up to 25x.
– Users are interested in two different offering models
• “free”, accessing a reduced pool of local resources.
• “subscribers only” accessing a larger amount of resources from public Clouds.
• Finally, 4 Pilots have found that
VENUS-C enabled them to open
new research lines
– Social trends analysis through
cloud computing.
– A repository of ICU vital signs for
studying early predictors.
14
Pilots
Scen.
Conclusion
• Users carried out an in-depth evaluation of the VENUS-C Platform.
– More than 1.5 Million of CPU hours in total (more than 1.3 Billion SPecInt2k
hours in EGI terms), 30 TB of cumulative data stored and 80 TB of data transfers.
• Application developers give a good score to the subsystems released
in April (from 3.92 to 4.37)
– User requirements completeness, ease of applications adaptation and
interoperability got the higher marks.
• The cooperation among the user community, developers and
infrastructure providers were very fruitful.
• VENUS-C demonstrates that public cloud infrastructures (i.e.
Windows Azure) are practical for scientific research, and that the use
of VENUS-C subsystems improves user experience.
– Open-source, private infrastructures have also been tested, with similar
conclusions regarding user experience.
15
Thanks!
• www.venus-c.eu
• Acknowledgment:
– Goetz Brasche, European Microsoft Innovation
Center,
– Ignacio Blanquer, Univ. Politecnica de Valencia
16