Post on 14-Jun-2020
transcript
Giove Filippo, Longoni Davide {filippo.giove,davide.longoni}@mail.polimi.it
In cloud computing terminology, the phrase "cloud portability" means the ability to move applications and its associated data between one cloud provider and another, or between public and private cloud environments
Ideally, enterprises should be able to take applications and data in or out of a cloud as business requires without having to rewrite the application or transform the data.
But the real scenario is quite different… ◦ Nowadays if a developer wants to move an
application from one cloud to another one he must rewrites /reengineers most of the code.
To solve the cloud portability problem there are two possibilities:
◦ Cloud Providers decide to standardized the access
of cloud services Standard APIs
Or... ◦ Open Source Communities develop adapters to
create a common interface.
Standards bodies, industry consortiums and vendor partnerships have begun developing technology to enable interoperability among cloud providers.
The IEEE has formed a working group to develop a set of interoperability standards: ◦ IEEE P2301 Draft Guide for Cloud Portability and
Interoperability Profiles.
VMware and Google have partnered to create cloud portability tools for Java developers, called Spring Source Tool Suite.
So using Eclipse based Spring Source Tool Suite, developers can build Spring applications in a familiar and productive way and have the flexibility to choose to deploy their app in: ◦ current private VMware vSphere environment ◦ in Vmware vCloud partner ◦ or directly to GAE.
The Simple Cloud API brings cloud technologies to PHP and the PHPilosophy to the cloud, starting with common interfaces for four cloud
application services:
File Storage Services Document Storage Services Queue Services Infrastructure
You can start writing scalable, highly available,
and resilient cloud applications that are portable across all major cloud vendors today.
What does the Simple Cloud API do?
◦ It offers one API to interface different kinds of services.
What are the advantages for the developers?
◦ They only have to write their application using this API and they can deploy across these services with just configuration changes.
Zend has invited the open source community and software vendors of all sizes to participate. ◦ IBM, Microsoft, Rackspace, Nirvanix, and GoGrid have already joined
the project as contributors.
There are several ways of writing code to work with cloud services. Precisely there are 4 possible levels of API:
◦ Level 1 (Wire): this type of APIs use directly APIs of the service with a specific protocol (eg: REST or SOAP)
◦ Level 2 (Language-specific): this type of APIs uses a specific function of a language to access protocol.
Example: a REST request
listFolder.php?SessionToken=8da051b0a60f4c22a&folderPath=/cs1&pageNumber=1&pageSize=5
Example: a PHP request of REST type
file_get_contents('ListFolder.php?SessionToken=8da051b0a60f4c22a...');
◦ Level 3 (Service-specific): this type of APIs uses higher-level objects that encapsulate cloud services. Working at this level, you have no idea whether the underlying service is SOAP or REST; you simply invoke the service.
◦ Level 4 (Service-neutral): this type of APIs provides a set of high-level classes that let you write one piece of code that can work with multiple services and providers
Example: A request of adding a msg inside an Azure’s Queue
AzureQueueManager.putMessage(String nameQueue, String msg, Integer timeToLive);
Example: A request of adding a msg inside a Queue
QueueManager.putMessage(String nameQueue, String msg, Integer timeToLive);
Our work is quite similar to SimpleCloud, but we are focusing on the application portability between two of the most popular and used PaaS: ◦ Azure MS
◦ GoogleAppEngine
Our goal is to provide a set of high level APIs that let JAVA developers to write applications that can be deployed (with NO changes) on both these two Paas providers.
Actually java applications are not natively handled by Azure, so you have to provide ◦ your own applications server (tomcat, glassfish etc..) inside the
package ◦ the startup script , that will run the server with your app inside ◦ the jdk
So you have to upload the jdk, the server binaries and your application code every time you want to redeploy a new package.
Or you can store them in blob storage and, in a startup script, copy them to local storage and start tomcat
A java app is not deployed as a WebRole but as a Worker Role where you add your own server.
WebRoles are only for .NET and PHP applications.
Azure
STANDARD INTERFACE
Cloud Library
Azure API
Google API
Application
configuration.xml
<<using>>
+
Every call of a service is handled by object called MF (Manager Factory).
MF is built using the Abstract Factory Pattern.
It performs the following operations: ◦ It has to read from the configuration file which type
of cloud is used. (only for the first call) ◦ Instantiates a concrete, customized MF for the
specific chosen vendor. (only for the first call) ◦ (the customized MF) has to remap the high level call
to the corresponding Vendor specific call.
Configuration.xml
On which cloud??
On which cloud?
<<instance>>
<<instance>>
[Azure]
[GAE]
The configuration file is a simple xml file which contains important info required to deploy the application.
The essential information which this file must contain is about the chosen vendor.
<configurations>
<cloud>
<type name=“cloud.vendor.name” value=“&vendor” />
……
</cloud>
</configurations>
NoSQL is a class of database management system identified by its non-adherence to the widely-used RDBMS model:
◦ It does not use SQL as its query language ◦ It may not give full ACID guarantees ◦ It has a distributed, fault-tolerant architecture
Both Azure and AppEngine provide this kind of service.
We decide to use JPA (Java Persistence API) technique
to access to this service because:
◦ It models your data objects as Java classes ◦ It’s easier to move your data from one data storage
solutions to another one.
We use and extend an implementation of an open source JPA (jpa4azure) for Azure.
It simplifies Azure Table persistence, using Java objects as interface to Azure Table data. ◦ jpa4azure implements a subset of the JPA 2.0 specification for
Java.
It allows you to automate: ◦ Table creation from entity model
◦ Querying entities
The query operation, initially was allowed only using the simply find (Class type, key) operation, after our extension are available also the following methods:
◦ CreateQuery(String query)
Query q=cem.createQuery("SELECT u FROM UserProfile u");
q.getResultList();
◦ CreateNamedQuery(StringQueryName) (with or without parameters)
with parameters….
Query q2=aem.createNamedQuery(“UserProfileGender”);
q2.setParameter(“sex”, true);
q2. getResultList();
And without….
Query q3= aem.createNamedQuery(“MaleUserProfile”);
q3. getResultList();
This is the structure of the AzureEntityManager. Actually only the most important and used methods, like persist, find, remove, createQuery,
createNamedQuery, are implemented.
The App Engine Java SDK includes the experimental version 2.0 of the DataNucleus plugin for App Engine.
JPA exposes a standard interface for interacting with relational databases, but the App Engine datastore is not a relational database.
As a result, there are features of JPA that App Engine simply cannot support. ◦ e.g. it’s not possible to send a query to the datastore
which contains the logical operator OR.
This is the structure of the GoogleEntityManager.
The basic EM’s methods of Azure and GAE are exactly the same, so it was relatively easy to create an interface that standardize them.
The problem for this service is the extension of the Azure’s JPA, which lacked some basic methods of JPA implementation.
In the following slide are presented the structure (very similar to the two just seen) and a practical example of its use.
All the methods return cloud-independent objects. It means that these methods remap the type of the returning object (that are usually cloud-dependable) in a common independent type.
@Entity public class UserProfile implements Serializable {
@Id private String email; private String firstName; private String lastName; private Integer age; private Boolean gender; public UserProfile(String email, String firstName, String lastName, Integer age, Boolean gender) { … }
}
@Entity @NamedQueries({ @NamedQuery(name=“MaleUserProfile", query="SELECT up FROM UserProfile up WHERE gender=true;"), @NamedQuery(name="UserProfileGender", query="SELECT up FROM UserProfile up WHERE gender=:sex; ")})
… UserProfile user = new UserProfile(“a.b@ddd.it”, “Pinco”, “Pallino”, 25, true); CloudEntityManager cem= MF.getFactory().getEMF().createCloudEntityManager(); //PERSISTENT OPERATION
cem.persist(user); //FIND OPERATION
cem.find(UserProfile.class, a.b@ddd.it); //QUERY OPERATION Query q=cem.createQuery("SELECT u FROM UserProfile u");
q.getResultList(); //NAMED QUERY OPERATION (with parameters) Query q2=cem.createNamedQuery(“UserProfileGender”);
q2.setParameter(“sex”, true); q2. getResultList();
//NAMED QUERY OPERATION (without paramaters) Query q3= cem.createNamedQuery(“MaleUserProfile”);
q3.getResultList();
This type of service is present in both clouds, but its logical implementation inside them is quite different.
In Azure the Queue service stores messages that may be read by any client who has access to the storage account.
Queue service is a great way to send messages between front-end roles and worker roles.
A queue can contain an unlimited number of messages, each of which can be up to
64 KB in size.
Messages are generally added to the end of the queue and retrieved from the front of the queue, although first in first out (FIFO) behavior is not guaranteed.
private static final QueueManagerFactory aqmf = (AzureQueueManagerFactory)Persistence.createQueueManagerFactory(PersistentUnitString); AzureQueueManager aqm=aqmf.createQueueManager(); //CREATE QUEUE OPERATION aqm.createQueue(String nameQueue);
//PUT OPERATION
aqm.putMessage(String nameQueue, String message, Integer timeToLiveInterval);
//GET MESSAGES OPERATION
AzureQueueMessageCollection msgs=
(AzureQueueMessageCollection)aqm.getMessages(String QueueName,
Integer NumOfMsgToRetrieve,
Integer visibilityTimeout);
//DELETE MESSAGE OPERATION
aqm.deleteMessage(String queueName, String messageId, String popReceipt);
//DELETE QUEUE OPERATION
aqm.deleteQueue(String queueName);
//CLEAR MESSAGES OPERATION
aqm.clearMessages(String queueName);
Default 7 days
Time in which messages are not
visible (30 sec default)
It is a token needed to perform any action on the
message while it is invisible in the queue.
With the Task Queue API, applications can perform work initiated by a user request, outside of that request.
If an app needs to execute some background work, it can use the Task Queue API to organize that work into small, discrete units, called tasks.
A task is a unit of work to be performed by the application and it is an object of the TaskOptions class.
Each Task object contains an endpoint ◦ with a request handler for the task and an optional data payload
that parameterizes the task.
AppEngine provides two different queue configurations: ◦ Push queues process tasks based on the processing rate
configured in the queue definition. Push queues are the default.
◦ Pull queues allow you to design your own system to consume App Engine tasks.
This task consumer leases a specific number of tasks for a specific duration, then processes and deletes them before the lease ends.
We decide to use Push Queues.
The QueueFactory is used to create queue. We use Push Queues, so with this type of queue, you simply configure a queue and add tasks to it. App Engine handles the rest.
//CREATE QUEUE OPERATION
Queue queue = QueueFactory.getQueue(String QueueName);
//ADD TASK OPERATION (the “equivalent” of a put message)
queue.add(withUrl("/computeSimilarity").param("user", email).method(Method.POST));
//DELETE MESSAGE OPERATION
queue.deleteTask(String taskName)
This operation delete task if the task is not already processed, only when using pull queues, your
application needs to delete tasks after processing.
//PURGE OPERATION
queue.purge();
We decide to use an approach similar of that used by GAE. ◦ So we decide to recreate the scenario of execution of tasks of
GAE also for Azure.
Practically for Azure we put a structured message inside AzureQueues, in which are written the URL of the task, the parameters and other info useful for the Task execution.
But in Azure there is not a Background Worker that, in an automatic way, reads, parses the message and executes the call to the task…
You can write a dedicate Worker Role to do these operations
(but more Rolesmore $$).
An alternative solution is the creation of a dedicate thread inside the same Role.
+ I/O interaction
Java App+ Jdk+server
Consumer
Java App+ Jdk+server
I/O interaction
Internal Thread
IW 1)getMessagge()
APPLICATION 3) Call the task
2) Parse the msg
4)Delete msg
0)putMessagge()
For the moment we decide to use the second solution.
Our thread (called Internal Worker) periodically performs these operations.
CloudQueue q = MF.getFactory().getQMF().getQueue(StringQueueName);
//OPERATION TO STRUCTURE THE MESSAGE
TaskOptions t = new TaskOptions(); t.setMethod(TaskOptions.POST); t.setParameters("user", email); String serverPath = request.getScheme() + "://" + request.getServerName() + ":" + request.getServerPort() + request.getContextPath(); t.setHostUri(serverPath); t.setServletUri("/computeSimilarity"); t.setTaskName(email); //ADD TASK TO THE QUEUE
q.add(t);
CloudQueue queue=MF.getFactory().getQMF().getQueue(queueName);
while (true) {
AzureQueueMessage msg = queue.getMessage();
TaskOptions info= parserMessage(msg.getMessageText());
callTheTask(info); //this method creates an HTTP request to
//the endpoint relative to the task specified //in the message
queue.deleteMessage(msg);
Thread.sleep(RestingTime);
}
The Blob service stores text and binary data that are much larger than the size allowed for objects in the Table/Datastore service.
Blobs are useful for serving large files, such as video or image files, and for allowing users to upload large data files.
Both Azure and GAE use REST API for the Blob service that provides a way to work through HTTP operations.
The Azure Blob service offers the following three resources:
◦ storage account
◦ containers
◦ blobs
Within your storage account, containers provide a way to organize sets of blobs.
You can store text and binary data in either of two types of blob:
◦ Block blobs, which are optimized for streaming.
◦ Page blobs, which are optimized for random read/write operations and which provide the ability to write to a range of bytes in a blob.
DirectConnectToken token= new DirectConnectToken(String account,String key);
AzureBlobManager abm= new AzureBlobManager(token);
//CREATE CONTAINER
abm.createContainer(containerName, null, ContainerAccess.BLOB);
MetadataCollection mc = new MetadataCollection();
String contentType = MimetypesFileTypeMap.getDefaultFileTypeMap().getContentType(String blobName);
mc.add(Metadata.buildValid("blobName", blobName));
mc.add(Metadata.buildValid("size", Integer.toString(file.length)));
mc.add(Metadata.buildValid("contentType", contentType));
BlobData bd = new OctetBlobData(file);
MetadataCollection
//UPLOAD BLOB
abm.putBlockBlob(containerName, blobName, null, mc, bd);
//DOWNLOAD A BLOB
AzureBlob ab = abm.getBlockBlob(containerName, blobName);
//DELETE A BLOB
abm.deleteBlob(containerName, blobName, null);
//LIST ALL BLOBS IN A GIVEN CONTAINER
AzureBlobCollection abc = abm.listAllBlobs(containerName);
LeaseId
In GAE, blobs are created by uploading a file through an HTTP request. ◦ Typically, your applications will do this by presenting to the
user a form with a file upload field.
When the form is submitted, the Blobstore creates a blob from the file's contents and returns an opaque reference to the blob, called blob key, which you can later use to serve the blob.
Each blob has a corresponding blob info record, stored in the datastore, that provides details about the blob: ◦ Its blobKey, creation time, file name, size and content type.
JSP page
upload Servlet
Blobstore
Blobstore Service
Datastore
1.You perform the submit action…
2a. GAE stores the Blob inside the BlobStore
2b. GAE stores the Blob info inside Datastore
3. GAE passes the blobKey to a dedicate handler
4.Redirect to next page, eventually passing to it the blobKey….
But we don’t like this way to upload a blob, because is very different respect the Azure’s one.
GAE support also an experimental way to create blobs, providing a file-like APIs that you can use to read and write to blobs.
Regarding JSP/Servlet, there is no standard way to manage the file fields when you submit a form ◦ We solve this lack using the library named
“CommonsFileUpload”.
JSP page
upload Servlet
Blobstore
File Service
Datastore
1.You perform the submit action…
4a. GAE stores the Blob inside the BlobStore
2b. GAE stores the Blob info inside Datastore
3. Write the file to upload in an AppEngine Blob
Next page…
2. Extract all the content of the form fields
Standard GAE Blobs APIs
Experimental APIs (We use these)
//DOWNLOAD A BLOB (FINDING IT BY ITS NAME)
BlobInfoFactory bif = new BlobInfoFactory(datastore);
Iterator<BlobInfo>iter=bif.queryBlobInfos();
BlobInfo bi = null;
String blobKey = null;
while (iter.hasNext()){
bi=iter.next();
if (bi.getFilename().compareTo(fileName)==0)
{ blobKey=bi.getBlobKey();
break;}
}
BlobstoreInputStream(blobKey); // Now we can read the content of the blob by this stream
//DELETE A BLOB
blobstoreService.delete(bi.getBlobKey());
CloudBlobManagerFactory cbmf = MF.getFactory().getBMF(); CloudBlobManager cbm = cbmf.createCloudBlobManager(); //UPLOAD A BLOB cbm.uploadBlob(buffer,Blobname); //SERVE A BLOB CloudDownloadBlob* cdb = cbm.downloadBlob(fileName); InputStream streamDown = cdb.getFileStream(); //DELETE A BLOB cdb.deleteBlob(fileName);
* CloudDownloadBlob is an object which contains: • blobName • contentType • fileStream • blobSize • blobKey;
Many custom-developed applications need to send email, whether it is a part of the registration process, a way of notifying users when important events occur or something else.
This service allows to send mail through the cloud application.
Windows Azure does not currently provide such a capability.
There are some patterns for enabling emailing capabilities for applications deployed to the Windows Azure platform: ◦ Using a custom on-premise Email Forwarder Service
◦ Using Email Server's Web Services APIs
◦ Using a third part SMTP Service
We use a third part SMTP Service. ◦ For setting the server and the mail account which
we want to use, we have to specify them by using the configuration file.
<configurations>
<cloud>
<type name=“cloud.vendor.name” value=“&vendor” />
<mailService>
<property name=“mail.serverSmtp.name” value=“&smtpServer” />
<property name=“mail.serverSmtp.port” value=“&port” />
<property name=“mail.account.name” value=“&mailAddress” />
<property name=“mail.account.key” value=“&mailPassword” />
</mailService>
</cloud>
</configurations>
The Mail service Java API supports the javax.mail interface for sending email messages: ◦ All of the JavaMail classes you need are included within the App
Engine SDK.
When creating a JavaMail Session, you do not need to provide any SMTP server configuration: ◦ App Engine will always use the Mail service for sending messages.
For security purposes, the sender address of a message must be the email address of an administrator for the application or any valid email receiving address for the app.
String messageBody = "Body of the message";
CloudMessage msgToSend= new CloudMessage(email, "subject",messageBody);
MF.getFactory().getMailManager().sendMail(msgToSend);
MiC is a simple application that we have done to test our works.
Practically it allows to connect people who share same
interests.
Initially an user has to response same questions about general purpose topics.
Then the application computes for him his most similar users (in according with the similarity between his answers and the answer of other users).
Finally MiC sends back to him a mail with his best contacts.
MiC
Cloud Environment
1a) sends his profile
1b) sends his picture
2a) saves profile
2b) saves picture
5b) adds request of computation
6) gets request
8) sends result
3) questions
4) answers
Mic User
7) computes request
5a) saves answers
Upload a java application on Azure (video)
Upload a java application on GAE
Next possible steps are:
◦ Standardization of Cache Service
◦ Standardization of the Datastore APIs
(for direct access without Jpa)
◦ Creation of an Eclipse plugin that automate the creation of environment to develop your GAE/Azure app using our standard APIs.
App Engine (Java Overview) https://developers.google.com/appengine/docs/java/overview
Azure (for Java Developers) http://www.windowsazure.com/it-it/develop/java/
Jpa for Azure
http://jpa4azure.codeplex.com/
GAE/VMware Partnership http://www.vmware.com/company/news/releases/vmware-google.html
SimpleCloud http://simplecloud.org/
<body> <form action="<%= BlobstoreService.createUploadUrl("/upload") %>" method="post" enctype="multipart/form-data"> <input type="file" name="myFile"> <input type="submit" value="Submit"> </form> </body>
<body> <form action=“NextServlet" method="post" enctype="multipart/form-data"> <input type="file" name="myFile"> <input type="submit" value="Submit"> </form> </body>
//WRITING THE FILE TO UPLOAD IN AN APPENGINE BLOB
String contentType = mftm.getContentType(fileName);
AppEngineFile blobToUpload = fileService.createNewBlobFile(contentType, fileName);
boolean lock = true;
FileWriteChannel wc = fileService.openWriteChannel(blobToUpload, lock);
blobToUpload = new AppEngineFile(blobToUpload.getFullPath());
wc.write(ByteBuffer.wrap(buffer));
wc.closeFinally();
Map<String, BlobKey> blobs = BlobstoreService.getUploadedBlobs(req); BlobKey blobKey = blobs.get("myFile"); res.sendRedirect("/serve?blob-key=" + blobKey.getKeyString());