J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Chapter 9 Cloud-Enabled Climate@Home

Jing Li, Zhenlong Li, Min Sun, Qunying Huang, Kai Liu, and Myra Bambacus

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Learning Objectives

• Climate@Home: background and challenges• Deploy the Climate@Home system onto the cloud• Run the system onto the cloud• Performance and cost analysis

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Learning Materials•Online Videos:

o Chapter_9_Video_1.wmvo Chapter_9_Video_2.wmvo Chapter_9_Video_3.wmvo Chapter_9_Video_4.wmvo Chapter_9_Video_5.wmvo Chapter_9_Video_6.mp4o Chapter_9_Video_7.mp4o Chapter_9_Video_8.mp4

•Scripts, Files and others:o Chapter_9_Section9.2.1.1.zipo Chapter_9_Section9.2.1.2.zip

3

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Learning Modules

1. Climate@Home: background and challenges

2. Climate@Home Portal cloud deployment

3. Climate@Home Boinc cloud deployment

4. System demonstration

5. Conclusion and discussions

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Climate@Home: Background

Climate modeling Data management and analysis Citizen computing

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Climate@Home: the model

• Model E (Schmidt et al., 2006): NASA Goddard Institute for Space

Studies (GISS)• Coupled atmosphere-ocean models• Long term• Global coverage • Multidimensional • Monthly output• Chosen as the climate model for the Climate@Home project• URL: http://www.giss.nasa.gov/tools/modelE/

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Climate@Home: the components

• Spatial web portal• integrate all front-end components with back-end

functions• geovisual analysis of the climate data and visual

management of contributed computing resources (Sun et al., 2012)

• access these components in a secured manner• The BOINC server

• dispatch ModelE simulation tasks• collect model outputs • coordinat with the BOINC clients

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Climate@Home: challenges

scalable and elastic data warehouse

distributed data communications and concurrent access

reliable and robust centralized management

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Learning Modules

1. Climate@Home: background and challenges

2. Climate@Home Portal cloud deployment

3. Climate@Home Boinc cloud deployment

4. System demonstration

5. Conclusion and discussions

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Cloud deployment: spatial web portal

2. Launch an instance

3. Create an EBS volume

1. Authorize network access

8. Restore the database

7. Transfer the code/data of the web portal and visualization portlet

4. Attach the EBS volume to the instance

5. Install MySQL, Tomcat and Apache2 software package

9. Start the Apache2 and Tomcat server and configure the web portal

6. Mount the EBS Volume to the Data/Log directory of MySQL

10. Configure the load balance/scalability

11. Build a new AMI from the running instance

Steps 3,4,6,10 are optional, which are demonstrated in Chapter 8.

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 1 and Step 2

• Step 1. Authorize network access-- Three ports should be opened: Port 22, Port 80, and Port 8080.

• Step 2. Launch an instance—A public AMI with Ubuntu 12.04 as OS is selected as the base image to create an instance for further customization.

• After launching the instance, log on to the instance from a Windows machine using putty.

2. Launch an instance

3. Create an EBS volume

1. Authorize network access

8. Restore the database

7. Transfer the code/data of the web portal and visualization portlet

4. Attach the EBS volume to the instance

5. Install MySQL, Tomcat and Apache2 software package

9. Start the Apache2 and Tomcat server andconfigure the web portal

6. Mount the EBS Volume to the Data/Log directory of MySQL

10. Configure the load balance/scalability

11. Build a new AMI from the running instance

Video: Chapter_9_Video_1.wmv

Files/Tools used in this deployment: Chapter_9_Section9.2.1.1.zip

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 3, 4 and 6

• Steps 3, 4, and 6. Configure EBS—Steps 3, 4, and 6 are not described in this chapter. The detailed configurations have been discussed in Chapter 8, Section 8.2.2.1.

2. Launch an instance

3. Create an EBS volume

1. Authorize network access

8. Restore the database

7. Transfer the code/data of the web portal and visualization portlet

4. Attach the EBS volume to the instance

5. Install MySQL, Tomcat and Apache2 software package

9. Start the Apache2 and Tomcat server andconfigure the web portal

6. Mount the EBS Volume to the Data/Log directory of MySQL

10. Configure the load balance/scalability

11. Build a new AMI from the running instance

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 5Install software packages

• In this step, Apache2 server, Tomcat server, and MySQL are installed and configured as required by the spatial Web portal.

2. Launch an instance

3. Create an EBS volume

1. Authorize network access

8. Restore the database

7. Transfer the code/data of the web portal and visualization portlet

4. Attach the EBS volume to the instance

5. Install MySQL, Tomcat and Apache2 software package

9. Start the Apache2 and Tomcat server andconfigure the web portal

6. Mount the EBS Volume to the Data/Log directory of MySQL

10. Configure the load balance/scalability

11. Build a new AMI from the running instance

Video: Chapter_9_Video_2.wmv

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 7Transfer files through WinSCP

• Drupal source codes are moved to the web folder of Apache HTTP server.

• The geovisual analytical portlets are moved to the web folder of the Tomcat server.

• Database files should be moved to a secured folder in the instance for later restoration.

2. Launch an instance

3. Create an EBS volume

1. Authorize network access

8. Restore the database

7. Transfer the code/data of the web portal and visualization portlet

4. Attach the EBS volume to the instance

5. Install MySQL, Tomcat and Apache2 software package

9. Start the Apache2 and Tomcat server andconfigure the web portal

6. Mount the EBS Volume to the Data/Log directory of MySQL

10. Configure the load balance/scalability

11. Build a new AMI from the running instance

Video: Chapter_9_Video_3.wmv

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 8. Restore the database

root@ip-10-189-149-104:/mnt$ mysql –u username –p #log in the mysql database using rootmysql>create user ‘climateuser’@localhost identified by ‘password’ #create a user account (e.g. climateuser) for the portalmysql>create database climatehome #the name of the climate@home drupal databasemysql> grant all privileges on climatehome .* to ‘climateuser’@localhost identified by ‘password’ #grant permission to the databasemysql>flush privilegesmysql>exit #quit mysqlroot@ip-10-189-149-104:/mnt$ mysql -u username -p climatehome< climatehome.sql #import the climatehome database climatehome.sql on the cloud server

2. Launch an instance

3. Create an EBS volume

1. Authorize network access

8. Restore the database

7. Transfer the code/data of the web portal and visualization portlet

4. Attach the EBS volume to the instance

5. Install MySQL, Tomcat and Apache2 software package

9. Start the Apache2 and Tomcat server andconfigure the web portal

6. Mount the EBS Volume to the Data/Log directory of MySQL

10. Configure the load balance/scalability

11. Build a new AMI from the running instance

The Portal database and Boinc server database are restored in this step.

Command to restore a MySQL database

Video: Chapter_9_Video_4.wmv

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 9. Configure the spatial web portal and visualization portlet

2. Launch an instance

3. Create an EBS volume

1. Authorize network access

8. Restore the database

7. Transfer the code/data of the web portal and visualization portlet

4. Attach the EBS volume to the instance

5. Install MySQL, Tomcat and Apache2 software package

9. Start the Apache2 and Tomcat server andconfigure the web portal

6. Mount the EBS Volume to the Data/Log directory of MySQL

10. Configure the load balance/scalability

11. Build a new AMI from the running instance

• Start the Apache2 and Tomcat server and configure the web portal. The following commands can be used to start the servers on the Ubutun instance.

root@ip-10-189-149-104:/$ sudo service apache2 start root@ip-10-189-149-104:/Tomcat/bin/$ ./startup.sh

• Once the two servers are started, the web portal can be accessed from http://Your_VM_Public_DNS(IP)/climate@home/. Please refer to the video for the detailed steps.

Portal login information:User name: adminPassword: cisc2345!@#$GD

Video: Chapter_9_Video_5.wmv

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 10 and 11

• Step 10 is an optional step to make the system more flexible and scalable. The configuration is introduced in Section 8.2.2.2 and 8.2.2.3.

• The last step is to create a new AMI based on the running instance of the Climate@Home portal. Chapter 4, Section 4.3.1 describes the creation of an AMI. The image can be reused to create multiple virtual instances.

2. Launch an instance

3. Create an EBS volume

1. Authorize network access

8. Restore the database

7. Transfer the code/data of the web portal and visualization portlet

4. Attach the EBS volume to the instance

5. Install MySQL, Tomcat and Apache2 software package

9. Start the Apache2 and Tomcat server andconfigure the web portal

6. Mount the EBS Volume to the Data/Log directory of MySQL

10. Configure the load balance/scalability

11. Build a new AMI from the running instance

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Learning Modules

1. Climate@Home: background and challenges

2. Climate@Home Portal cloud deployment

3. Climate@Home Boinc cloud deployment

4. System demonstration

5. Conclusion and discussions

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Cloud deployment: the BOINC server

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 1 and 2Authorize network access andlaunch an instance

• Step 1. Authorize network access—Two ports should be opened: Port 22, Port 80. Port 22 is used to access the instance through the Secure Shell (SSH). Port 80 is for HTTP.

• Step 2. Launch an instance—A public AMI with Ubuntu 12.04 as the OS is selected as the base image for creating an instance for further customization.

Video: Chapter_9_Video_6.mp4

Files/Tools used in this deployment: Chapter_9_Section9.2.1.2.zip

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 3, 4 and 6

• The Climate@Home Boinc server uses distributed FTPS server to store the uploaded results. Hence, EBS is optional for the Boinc server. The detailed configurations have been discussed in Chapter 8, Section 8.2.2.1.

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 5 and 7Install software packages and Boinc server

• Step 5. Install software packages—Several software packages are needed for the BOINC to function, including MySQL, Apache HTTP server, Python, PHP5, and others.

• Step 7. Install BOINC server software.

Video: Chapter_9_Video_6.mp4

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 8 Create a BOINC project and configure BOINC server

Upon finishing the installation of the BOINC server, a BOINC test project should be created, which distributes the ModelE applications to computing devices contributed by citizens.

Video: Chapter_9_Video_7.mp4

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 9,10 and 11

• Step 9. Upload the ModelE VirtualBox image and ModelE configuration files to the instance. The ModelE VirtualBox Image is put into the application folders of the BOINC server. The ModelE configuration files are put into the download folder of the “climateathome” project created in Step 8.

• Step 10. Configure the load balance or scalability—This is an optional step. The configuration is introduced in Section 8.2.2.2 and Section 8.2.2.3

• Step 11. Create work units and start the project—Consumers need to create input and output templates to start the project.

Video: Chapter_9_Video_8.mp4

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Step 12Create an AMI from the running instance

• This step creates a new AMI based on the running instance of the BOINC server for reuses. Please see Chapter 4, Section 4.3 for creating an AMI.

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Learning Modules

1. Climate@Home: background and challenges

2. Climate@Home Portal cloud deployment

3. Climate@Home Boinc cloud deployment

4. System demonstration

5. Conclusion and discussions

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

The home page of the Climate@Home spatial web portal

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Seasonal mean analysis using model outputs

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Visualizing volunteered computing nodes with Google Map (Li et al., 2013)

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Learning Modules

1. Climate@Home: background and challenges

2. Climate@Home Portal cloud deployment

3. Climate@Home Boinc cloud deployment

4. System demonstration

5. Conclusion and discussions

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Cloud services in the Climate@Home system

Computing services

Configuration Network Costs with AWS Purpose

Cloud data storage

10 TB N/A $0.080 /GB-month(Amazon S3); $0.095/GB-month (Amazon EBS)

Archive the model outputs and visualization results

Cloud web server

8-core 2.33GHz;16G RAM

1G $0.299/hour (rent one year) $0.236 (rent three year);$0.5/hour (hourly base)

Host the spatial web portal

BONIC server 8-core 2.33GHz;16G RAM

1G $0.299/hour (rent one year); $0.236 (rent three year);$0.5/hour (hourly base)

Serve as the grid engine machine

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

• Deploying the spatial web portal to the cloud service handles

concurrent access to the online geovisual analytical portlet• Deploying the BONIC server in a cloud environment ensures

the robustness, high performance, security and reliability of

managing model simulations.

Conclusions

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

Discussion Questions

1. Using the Climate@Home system as an example, list the major system components required for supporting climate modeling and analysis in a distributed environment.

2. Using the development of Climate@Home system as an example, explain why cloud computing techniques are essential in supporting climate modeling and analysis in a distributed environment.

3. What are the major steps of deploying the spatial web portal onto the cloud services? In the process of deploying the spatial web portal, what are the special considerations related to network configurations?

4. What are the major steps of deploying the BONIC server onto the cloud services? What are the special considerations compared to the general application deployment in the Chapter 5?

5. What are the hardware configuration requirements of a virtual machine serving as a BONIC server?

6. What are the usages of an Elastic Block Store (EBS) in the cloud-enabled Climate@Home system?

7. List the cloud services used in the deployment of the Climate@Home system.8. What are the remaining issues for deploying the Climate@Home components onto t

he cloud?

J. Li, Z. Li, M. Sun, Q. Huang, K. Liu, M. Bambacus, Cloud-enabling Climate@home, 2013. In Spatial Cloud Computing: a practical approach, edited by C. Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 145-162.

References

• Li, Z., C. Yang, M. Sun et al. 2013. A High Performance Web-Based System for Analyzing and Visualizing Spatiotemporal Data for Climate Studies. In Web and Wireless Geographical Information Systems, pp. 190-198. Springer Berlin Heidelberg, 2013.

• Sun, M., J. Li, C. Yang et al. 2012. A Web-Based Geovisual Analytical System for Climate Studies. Future Internet 4, no. 4 : 1069-1085.

• Schmidt, G.A., Ruedy, R., Hansen, J.E. et al. 2006. Present-Day Atmospheric Simulations Using GISS ModelE: Comparison to In Situ, Satellite, and Reanalysis Data. Journal of Climate 19, 153–192.