Date post: | 12-Apr-2019 |
Category: |
Documents |
Upload: | hoangkhanh |
View: | 214 times |
Download: | 0 times |
Basics of Cloud Computing – Lecture 7
More AWS and
Research at Mobile & Cloud LabResearch at Mobile & Cloud Lab
Satish Srirama
Outline
• More Amazon Web Services
• Cloud based Research @ Mobile & Cloud Lab
25/03/2014 2/38Satish Srirama
Cloud Providers and Services – we
already discussed• Amazon Web Services
– Amazon EC2
– Amazon S3
– Amazon EBS
– Amazon Elastic Load Balancing
– Amazon Auto Scale
– Amazon CloudWatch
• Eucalyptus• Eucalyptus
• OpenStack
• SciCloud
• Management providers– ElasticFox
– RightScale
• PaaS– Google AppEngine
– Windows Azure
25/03/2014 3/38Satish Srirama
AWS we discuss
• AWS Management Console
• AWS Identity and Access Management
• AWS Elastic Beanstalk
• AWS CloudFormation• AWS CloudFormation
• Amazon Simple Workflow Service
• Amazon Elastic MapReduce
25/03/2014 5/38Satish Srirama
AWS Management Console
• Hope some of you have started using Amazon accounts
• You can manage your complete Amazon account with management console (Similar to Hybridfox)– AMI Management– AMI Management
– Instance Management
– Security Group Management
– Elastic IP Management
– Elastic Block Store
– Key Pair management etc.
• Have different panes for different services
25/03/2014 6/38Satish Srirama
AWS Identity and Access Management
(IAM)
• How can an enterprise or group of people use a single credit card?
• Manage IAM users
– Create new users and manage them– Create new users and manage them
– Create groups
• Manage permissions
– Creating policies
• Manage credentials
– Create and assign temporary security credentials
25/03/2014 8/38Satish Srirama
IAM policy
• Example policy giving access to complete EC2
http://aws.amazon.com/iam/
25/03/2014 9/38Satish Srirama
AWS Elastic Beanstalk
• Enables to easily deploy and manage applications in the
AWS cloud
– Simply upload a bundle of the applications built using .NET, PHP
and Java technologies
• Automatically handles the deployment details of capacity • Automatically handles the deployment details of capacity
provisioning, load balancing, auto-scaling, and
application health monitoring
• Something similar to PaaS
• One retains full control over the AWS resources powering
the application
– You can access the underlying resources at any time
25/03/2014 10/38Satish Srirama
AWS Elastic Beanstalk
• AWS EB is built using familiar software stacks such as
the Apache HTTP Server for PHP, IIS 7.5 for .NET, and
Apache Tomcat for Java
• There is no additional charge for Elastic Beanstalk
– Only the underlying AWS resources (e.g. Amazon EC2,
Amazon S3) are charged
• Leverages AWS services such as Amazon EC2, S3, SNS,
ELB, and Auto Scaling to deliver the same highly
reliable, scalable, and cost-effective infrastructure
http://aws.amazon.com/elasticbeanstalk
25/03/2014 11/38Satish Srirama
AWS CloudFormation
• Provides an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion
• It is based on templates model– Templates describe the AWS resources, the associated
dependencies, and runtime parameters to run an app. dependencies, and runtime parameters to run an app.
– The templates describe stacks, which are set of software and hardware resources.
– Something similar to CloudML and RightScale server templates
• Hides several details– How the AWS services need to be provisioned
– Subtleties of how to make those dependencies work.
25/03/2014 12/38Satish Srirama
AWS CloudFormation
• Amazon provides several pre-built templates to start common apps as:
– WordPress (blog)
– LAMP stack
– Gollum (wiki used by GitHub)– Gollum (wiki used by GitHub)
– …
• There is no additional charge for AWS CloudFormation. You pay for AWS resources (e.g. EC2 instances, Elastic Load Balancers, etc.) http://aws.amazon.com/cloudformation/
25/03/2014 13/38Satish Srirama
Amazon Simple Workflow Service
• A workflow service for building scalable, resilient
applications
• Reliably coordinates all of the processing steps within
applications
– such as business processes, sophisticated data analytics – such as business processes, sophisticated data analytics
applications, or managing cloud infrastructure services
• Manages task execution dependencies, scheduling, and
concurrency
• Provides simple API calls from code written in any language
• Capable to run on EC2 instances, or any of the customer’s
machines located anywhere in the world
25/03/2014 14/38Satish Srirama
Amazon Simple Workflow Service
• Maintains application state
• Tracks workflow executions and logs their
progress
• Holds and dispatches tasks• Holds and dispatches tasks
• Controls which tasks each of the application
hosts will be assigned to execute
• http://aws.amazon.com/swf/
25/03/2014 15/38Satish Srirama
Amazon Elastic MapReduce
• Web interface and command-line tools for running Hadoop jobs on EC2
• Data stored in Amazon S3
• Monitors job and shuts machines after use• Monitors job and shuts machines after use
• Running a job
– Upload job jar & input data to S3
– Create the cluster
– Create a Job Flow as steps
– Wait for the completion and examine the results
25/03/2014 16/38Satish Srirama
http://aws.amazon.com/elasticmapreduce/
Other interesting AWS
• Amazon Relational Database Service
– Provides access to the capabilities of familiar
database engines
– MySQL, Oracle or Microsoft SQL Server– MySQL, Oracle or Microsoft SQL Server
• NoSQL databases
– Simple DB
– DynamoDB
25/03/2014 17/38Satish Srirama
Scientific Computing on the Cloud
• Public clouds provide very convenient access to computing resources
– On-demand and in real-time
– As long as you can afford them– As long as you can afford them
• High performance computing (HPC) on cloud
– Virtualization and communication latencies are major hindrances [Srirama et al, SPJ 2011; Batrashev et al, HPCS 2011]
• Things have improved significantly over the years
– Research at scale
25/03/2014 19/38Satish Srirama
Adapting Computing Problems to
Cloud• Reducing the algorithms to cloud computing frameworks
like MapReduce [Srirama et al, FGCS 2012]
• Designed a classification on how the algorithms can be adapted to MR– Algorithm � single MapReduce job
• Monte Carlo, RSA breaking• Monte Carlo, RSA breaking
– Algorithm � n MapReduce jobs• CLARA (Clustering), Matrix Multiplication
– Each iteration in algorithm � single MapReduce job• PAM (Clustering)
– Each iteration in algorithm � n MapReduce jobs • Conjugate Gradient
• Applicable especially for Hadoop MapReduce
25/03/2014 20/38Satish Srirama
Issues with Hadoop MapReduce
• It is designed and suitable for:
– Data processing tasks
– Embarrassingly parallel tasks
• Has serious issues with iterative algorithms
– Long „start up“ and „clean up“ times ~17 seconds
– No way to keep important data in memory between MapReduce job executions
– At each iteration, all data is read again from HDFS and written back there at the end
– Results in a significant overhead in every iteration
25/03/2014 21/38Satish Srirama
Alternative Approaches
• Restructuring algorithms into non-iterative versions – CLARA instead of PAM [Jakovits & Srirama, Nordicloud 2013]
• Alternative MapReduce implementations that are designed to handle iterative algorithms designed to handle iterative algorithms – E.g. Twister [Jakovits et al, ParCo 2011], HaLoop, Spark
• Alternative distributed computing models– Bulk Synchronous Parallel model [Valiant, 1990] [Jakovits et al,
HPCS 2013]
– Building a fault-tolerant BSP framework (NEWT) [Kromonov et al, UCC 2013]
25/03/2014 22/38Satish Srirama
Remodeling Enterprise Applications for
the Cloud
• Remodeling workflow based applications for the cloud
– To reduce communication latencies among the components
Intuition: Reduce inter-node communication and – Intuition: Reduce inter-node communication and to increase the intra-node communication
• LP based mathematical models to find ideal deployment configuration [Paniagua et al, iiWAS 2011]
– Based on the loads and regions
25/03/2014 23/38Satish Srirama
Mobile Applications
• One can do interesting things on mobiles directly
– Today’s mobiles are far more capable
– Location-based services (LBSs), mobile social networking, mobile commerce, context-aware services etc.
• It is also possible to make the mobile a service provider• It is also possible to make the mobile a service provider
– Mobile web service provisioning [Srirama et al, ICIW 2006; Srirama
and Paniagua, MS 2013]
– Challenges in security, scalability, discovery and middleware are studied [Srirama, PhD 2008]
– Mobile Social Network in Proximity [Chang et al, ICSOC 2012; PMC
2013]
25/03/2014 25/38Satish Srirama
Mobile Cloud Applications
• Bring the cloud infrastructure to the proximity
of the mobile user
• Mobile has significant advantage by going
cloud-awarecloud-aware
– Increased data storage capacity
– Availability of unlimited processing power
– PC-like functionality for mobile applications
– Extended battery life
25/03/2014 26/38Satish Srirama
Mobile Cloud Binding Models
Task Delegation Code Offloading
[Flores & Srirama, JSS 2013]25/03/2014 27/38Satish Srirama
Mobile Cloud Middleware
[Srirama and Paniagua, MS 2013]
[Flores et al, MoMM 2011; Flores and Srirama, JSS 2013]
[Warren et al, IEEE PC 2014]
25/03/2014 28/38Satish Srirama
CroudSTag – Scenario
• CroudSTag takes the pictures/videos from the cloud and tries to recognize people
– Pictures/Videos are actually taken by the phone
– Processes the videos
– Recognizes people using facial recognition technologies– Recognizes people using facial recognition technologies
• Reports the user a list of people recognized in the pictures
• The user decides whether to add them or not to the social group
• The people selected by the user receive a message in facebook inviting them to join the social group
25/03/2014 29/38Satish Srirama
CroudSTag [Srirama et al, PCS 2011;
SOCA 2012]
• Cloud services used
– Media storage on
Amazon S3
– Processing videos on Facial Recognition
Process
Taking picture/video
using the camera
Selecting CloudMain Menu
Send Asynchronous
Notification and Results
Storage Services
1.
2.
3.
Login
– Processing videos on
Elastic MapReduce
– face.com to recognize
people on facebook
– Starting social group
on facebookSend invitation to the
social group
Selecting people
Selecting Cloud
Authentication
Start Process
4.
5.
6.
7.
8.
9.
Send next
invitation
Back to Menu
25/03/2014 30/38Satish Srirama
Code Offloading
• Studied extensively by community [MAUI, Cloudlets etc.]
• Is Mobile Cloud taking full advantage of Cloud Computing?– Parallelization and elasticity are not exploited
• Offloading from a different perspective• Offloading from a different perspective– “Offloading is a global learning process rather than just a
local decision process“ [Flores and Srirama, MCS 2013]
• How it can learn?– Analysis of code offloading traces which are generated by
the massive amount of devices that connect to cloud
“EMCO: Evidence-based mobile code offloading“
25/03/2014 31/38Satish Srirama
Process-intensive Tasks on Cloud
• Media processing
– CroudSTag demonstrates image and video
processing
• Sensor data analysis• Sensor data analysis
– Human activity recognition [Srirama et al, NGMAST 2011]
– Context aware gaming
– MapReduce based sensor data analysis [Paniagua et al,
MobiWIS 2012]
25/03/2014 33/38Satish Srirama
Data Analytics on the Cloud
• Cloud scale data storage solutions
• Cloud scale data analytics
– Pig & Hive
• NoSQL• NoSQL
• Implementing graph algorithms on graph
databases
Large-scale Data Processing on the Cloud -
MTAT.08.036 (Fall 2014)
25/03/2014 34/38Satish Srirama
This week in lab
• Advanced Google AppEngine
– You will try accessing DB
25/03/2014 36/38Satish Srirama
Next Week
• Summarize what we have learnt
• How to prepare for the examination
25/03/2014 37/38Satish Srirama
References
• Check Amazon videos and webinars at http://aws.amazon.com/resources/webinars/
• List of Publications - Satish Narayana Srirama - http://math.ut.ee/~srirama/publications.html
• [Warren et al, IEEE PC 2014] I. Warren, A. Meads, S. N. Srirama, T. Weerasinghe, C. Paniagua: Push Notification Mechanisms for Pervasive Smartphone Applications, IEEE Pervasive Computing, ISSN: 1536-1268, PC-2012-12-0133.R1 (Accepted for publication).
• [Flores and Srirama, JSS 2013] H. Flores, S. N. Srirama: Mobile Cloud Middleware, Journal of Systems and Software, ISSN: 0164-1212. Elsevier. DOI: 10.1016/j.jss.2013.09.012 (In print).
• [Chang et al, PMC 2013] C. Chang, S. N. Srirama, S. Ling: Towards an Adaptive Mediation Framework for Mobile Social Network in Proximity, Pervasive and Mobile Computing Journal, MUCS Fast track, ISSN: 1574-1192. Elsevier. DOI: 10.1016/j.pmcj.2013.02.004 (In print)
• [Kromonov et al, UCC 2013] I. Kromonov, P. Jakovits, S. N. Srirama: NEWT - A fault tolerant BSP framework on Hadoop YARN, 6th IEEE/ACM International Conference on Utility and Cloud Computing (UCC 2013), December 9-12, 2013, pp. 309-310. IEEE.
• [Jakovits and Srirama, Nordicloud 2013] P. Jakovits, S. N. Srirama: Clustering on the Cloud: Reducing CLARA to MapReduce, 2nd Nordic Symposium on Cloud Computing & Internet Technologies (NordiCloud 2013), September 02-03, 2013, pp. 64-71. ACM.
• [Jakovits et al, HPCS 2013] P. Jakovits, S. N. Srirama, I. Kromonov: Viability of the Bulk Synchronous Parallel Model for Science on Cloud, The 2013 (11th) International Conference on High Performance Computing & Simulation (HPCS 2013), July 01-05, 2013, pp. 41-48. IEEE.
• [Jakovits et al, HPCS 2013] P. Jakovits, S. N. Srirama, I. Kromonov: Viability of the Bulk Synchronous Parallel Model for Science on Cloud, The 2013 (11th) International Conference on High Performance Computing & Simulation (HPCS 2013), July 01-05, 2013, pp. 41-48. IEEE.
• [Srirama and Paniagua, MS 2013] S. N. Srirama, C. Paniagua: Mobile Web Service Provisioning and Discovery in Android Days, The 2013 IEEE International Conference on Mobile Services (MS 2013), June 27 - July 02, 2013, pp. 15-22. IEEE.
• [Flores and Srirama, MCS 2013] H. Flores, S. N. Srirama: Adaptive Code Offloading for Mobile Cloud Applications: Exploiting Fuzzy Sets and Evidence-based Learning, The Fourth ACM Workshop on Mobile Cloud Computing and Services (MCS 2013) @ The 11th International Conference on Mobile Systems, Applications and Services (MobiSys 2013), June 25-28, 2013, pp. 9-16. ACM.
• [Srirama et al, SOCA 2012] S. N. Srirama, C. Paniagua, H. Flores: Social Group Formation with Mobile Cloud Services, Service Oriented Computing and Applications Journal, ISSN: 1863-2386, 6(4):351-362, 2012. Springer. DOI: 10.1007/s11761-012-0111-5.
• [Srirama et al, FGCS 2012] S. N. Srirama, P. Jakovits, E. Vainikko: Adapting Scientific Computing Problems to Clouds using MapReduce, Future Generation Computer Systems Journal, 28(1):184-192, 2012. Elsevier press. DOI 10.1016/j.future.2011.05.025.
• [Chang et al, ICSOC 2012] C. Chang, S. N. Srirama, S. Ling: An Adaptive Mediation Framework for Mobile P2P Social Content Sharing, 10th International Conference on Service Oriented Computing (ICSOC 2012), November 12-16, 2012, pp. 374-388. Springer LNCS.
• [Paniagua et al, MobiWIS 2012] C. Paniagua, H. Flores, S. N. Srirama: Mobile Sensor Data Classification for Human Activity Recognition using MapReduce on Cloud, The 9th International Conference on Mobile Web Information Systems (MobiWIS 2012), August 27-29, 2012, v. 10 of ProcediaComputer Science, pp. 585-592. Elsevier.
• [Srirama et al, SPJ 2011] S. N. Srirama, O. Batrashev, P. Jakovits, E. Vainikko: Scalability of Parallel Scientific Applications on the Cloud, Scientific Programming Journal, Special Issue on Science-driven Cloud Computing, 19(2-3):91-105, 2011. IOS Press. DOI 10.3233/SPR-2011-0320.
25/03/2014 Satish Srirama 38/38
References - continued
• [Flores et al, MoMM 2011] H. Flores, S. N. Srirama, C. Paniagua: A Generic Middleware Framework for Handling Process Intensive Hybrid Cloud Services from Mobiles, The 9th International Conference on Advances in Mobile Computing & Multimedia (MoMM-2011), December 5-7, 2011, pp. 87-95. ACM.
• [Paniagua et al, iiWAS 2011] C. Paniagua, S. N. Srirama, H. Flores: Bakabs: Managing Load of Cloud-based Web Applications from Mobiles, The 13th International Conference on Information Integration and Web-based Applications & Services (iiWAS-2011), December 5-7, 2011, pp. 489-495. ACM.
• [Srirama et al, PCS 2011] S. N. Srirama, C. Paniagua, H. Flores: CroudSTag: Social Group Formation with Facial Recognition and Mobile Cloud Services, The 8th International Conference on Mobile Web Information Systems (MobiWIS 2011), September 19-21, 2011, v. 5 of Procedia Computer Science, pp. 633-640. Elsevier. doi: 10.1016/j.procs.2011.07.082.
• [Srirama et al, NGMAST 2011] S. N. Srirama, H. Flores, C. Paniagua: Zompopo: Mobile Calendar Prediction based on Human Activities Recognition using the Accelerometer and Cloud Services, 5th International Conference on Next Generation Mobile Applications, Services and Technologies (NGMAST 2011), September 14-16, 2011, pp. 63-69. IEEE.
• [Jakovits et al, ParCo 2011] P. Jakovits, S. N. Srirama, E. Vainikko: MapReduce for Scientific Computing - Viability for non-embarrassingly parallel algorithms, The 14th International Parallel Computing conference (ParCo 2011), August 30-September 2, 2011.
• [Batrashev et al, HPCS 2011] O. Batrashev, S. N. Srirama, E. Vainikko: Benchmarking DOUG on the Cloud, The 2011 International Conference on High • [Batrashev et al, HPCS 2011] O. Batrashev, S. N. Srirama, E. Vainikko: Benchmarking DOUG on the Cloud, The 2011 International Conference on High Performance Computing & Simulation (HPCS 2011), July 4-8, 2011, pp. 677-685. IEEE.
• [MAUI] E. Cuervo, A. Balasubramanian, D.-k. Cho, A. Wolman, S. Saroiu, R. Chandra, and P. Bahl: MAUI: making smartphones last longer with code offload, in Proceedings of the 8th international conference on Mobile systems, applications, and services. ACM, 2010, pp. 49–62.
• [Cloudlets] M. Satyanarayanan, P. Bahl, R. Caceres, and N. Davies: The case for vm-based cloudlets in mobile computing, Pervasive Computing, IEEE, vol. 8, no. 4, pp. 14–23, 2009.
• [Srirama, PhD 2008] S. N. Srirama: Mobile Hosts in Enterprise Service Integration, PhD thesis, RWTH Aachen University, September, 2008.
• [Srirama et al, ICIW 2006] S. N. Srirama, M. Jarke, W. Prinz: Mobile Web Service Provisioning, Proceedings of the Advanced International Conference on Telecommunications and International Conference on Internet and Web Applications and Services (AICT-ICIW 2006), February 23-25, 2006, pp. 120-125. IEEE Computer Society Press.
• [Valiant, 1990] L. G. Valiant: A bridging model for parallel computation, Commun. ACM, vol. 33, no. 8, pp. 103–111, Aug. 1990.
25/03/2014 Satish Srirama 39/38