Art by Andrew Fryer

Dan Cohen Microsoft Consulting Services Microsoft Israel email: danco@microsoft.com Team blog: http://blogs.microsoft.co.il/blogs/mcs My blog: http://dannycohen.info

Agenda

Developer tools

Windows Azure

Compute

Storage

SQL Azure

Windows Azure platform AppFabric

Windows Azure Interoperability

Monitoring and Debugging

Microsoft Codename “Dallas”

Operating System

App/Web Server

Frameworks

Your Application

OS Services

Virtualized Instance

Hardware

Windows Azure Platform

.NET based

Application

Platform as a

Service

(PaaS) Windows Azure OS

IIS / WAS

.NET Framework

Lots of stuff…

Custom Hyper-V

Hardware

hardware

software

facilities

power/cooling

IT labor

support

network

security

maintenance

management tools

disaster recovery

backup

Acquisition cost is 10% of IT Spend

Operating cost is 90% of IT Spend

Source: IDC

Windows Azure Platform

Windows Azure Compute

Demo: Create an account / project

Windows Azure, On One Slide

The “cloud” is

1000s of geo-distributed servers & networking

Windows Azure is analogous to

An operating system for the cloud

The “operating system by analogy” provides

Application hosting

Automated service management

Upload your code and “service model”; press START

Durable storage at massive scale

Windows Azure, In One Picture

Desktop

VS Tools

WA SDK

Sto

rage

Clu

ste

r

MSFT Datacenters

Business Portal

Developer Portal

Com

pu

te C

luste

r

… …

Service Management Service

REST

REST …

Cloud VM

Runtime

API

VM Setup

User

Code

Azure Service Architectures

A service consists of

An isolation boundary

A set of component roles each with endpoints

Numbered, identical instances of each role

Azure Service Architectures A role instance is a set of:

Code

Configuration

local data

deployed in a dedicated VM

Web Role: The role is hosted on IIS

Worker Role: The role is an executable create your own web server

host a database

host Java VM, Ruby etc.)

Worker

Role

Worker

Role

Web

Role

Azure Service Architectures

A service is reachable externally via a single VIP for load-balanced traffic to multiple endpoints

Endpoints are reachable internally over TCP, HTTP Runtime API exposes & updates IP/Port values

LB

Windows Azure Compute

Demo: “Hello Cloud” with Visual Studio 2008/ 2010

Sample: Notification Of New Instances

Sample: Inter-role communication

2 VM Roles (currently) available: Web role

External synchronous HTTP/S communication

Worker role External HTTP/S / TCP communication

Mental model: “it‟s like a Windows Service”

Windows Azure Operating System Automatic /manual upgrades

See: http://msdn.microsoft.com/en-us/library/ee924680.aspx

Storage Local storage is volatile (!)

4 standard sizes Small -> Extra Large

See: http://msdn.microsoft.com/en-us/library/ee814754.aspx

Blobs

Drives

Tables Provide structured storage. A Table is a set of entities, which contain a set of properties

Queues

Blob Container Account

cohowinery

images

PIC01.JPG

PIC02.JPG

videos VID1.AVI

http://<account>.blob.core.windows.net/<container>/<blobname>

PutBlob

GetBlob

DeleteBlob CopyBlob SnapshotBlob LeaseBlob

Cache-Control Content-Encoding Content-Type

CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudBlobClient blobClient = new CloudBlobClient( account.BlobEndpoint, account.Credentials); // Create Container CloudBlobContainer cloudContainer = blobClient.GetContainerReference(containerName); bool hasCreated = cloudContainer.CreateIfNotExist(); // Access Blob in the Container CloudBlob cloudBlob = cloudContainer.GetBlobReference(blobName); //BlobRequestOptions has retry policy, timeout etc. BlobRequestOptions options = new BlobRequestOptions(); //Upload the local file to Blob service cloudBlob.UploadFile(uploadFileName, options); //Download to local file name cloudBlob.DownloadToFile(downloadFileName, options);

blocks

pages

10 GB Movie

Windows

Azure Storage

Blo

ck

Id

1

Blo

ck

Id

2

Blo

ck Id

3

Blo

ck

Id

N

blobName = “blob.wmv”; PutBlock(blobName, blockId1, block1Bits); PutBlock(blobName, blockId2, block2Bits); ………… PutBlock(blobName, blockIdN, blockNBits); PutBlockList(blobName, blockId1, blockId2…,blockIdN);

blob.wmv blob.wmv

PutPage 512, 2048PutPage 0, 1024ClearPage 512, 1536PutPage 2048,2560

GetPageRange 0, 4096

0,512 1536,2560

GetBlob 1000, 2048

1536,2048

0

10 GB

10 G

B A

dd

ress S

pace

512

1024

1536

2048

2560

ServicePointManager.DefaultConnectionLimit

ParallelOperationThreadCount CloudBlobClient

BlobRequestOptions

Timeout

Blob Operations

Copy, Snapshot, and Lease work for both types

Ways of Accessing and Serving Blob Content

Content Delivery Network access

Shared Access Signatures (Signed URLs)

Custom Domain Names

Blobs

Drives

Tables Provide structured storage. A Table is a set of entities, which contain a set of properties

Queues

Windows Azure

Blob Service

DemoBlob

Local Cache

Application

Lease

Drive X:

//Create Local Storage resource and initialize the local cache for drives CloudDrive.InitializeCache(localCacheDir, cacheSizeInMB); CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); //Create a cloud drive (PageBlob) CloudDrive drive = account.CreateCloudDrive(pageBlobUri); drive.Create(1000 /* sizeInMB */); //Mount the network attached drive on the local file system string pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None); //Use NTFS APIs to Read/Write files to drive … //Snapshot drive while mounted to create backups Uri snapshotUri = drive.Snapshot(); //Unmount the drive drive.Unmount();

Blobs

Drives

Tables Provide structured storage. A Table is a set of entities, which contain a set of properties

Queues

Entity Table Account

cohowinery

customers

Name =…

Email = …

Name =…

Email = …

winephotos

Photo ID =…

Date =…

Photo ID =…

Date =…

PartitionKey

(Category)

RowKey

(Title)

Timestamp ReleaseDate

Action Fast & Furious … 2009

Action The Bourne Ultimatum … 2007

… … … …

Animation Open Season 2 … 2009

Animation The Ant Bully … 2006

PartitionKey

(Category)

RowKey

(Title)

Timestamp ReleaseDate

Comedy Office Space … 1999

… … … …

SciFi X-Men Origins: Wolverine … 2009

… … … …

War Defiance … 2008

PartitionKey

(Category)

RowKey

(Title)

Timestamp ReleaseDate

Action Fast & Furious … 2009

Action The Bourne Ultimatum … 2007

… … … …

Animation Open Season 2 … 2009

Animation The Ant Bully … 2006

Comedy Office Space … 1999

… … … …

SciFi X-Men Origins: Wolverine … 2009

… … … …

War Defiance … 2008

Server B Table = Movies

[Comedy - MaxKey)

Server A Table = Movies

[MinKey - Comedy)

Server A Table = Movies

[DataServiceKey("PartitionKey", "RowKey")] public class Movie { /// Movie Category is the partition key public string PartitionKey { get; set; } /// Movie Title is the row key public string RowKey { get; set; } public DateTime Timestamp { get; set; } public int ReleaseYear { get; set; } public double Rating { get; set; } public string Language { get; set; } public bool Favorite { get; set; } }

CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudTableClient tableClient = new CloudTableClient( account.TableEndpoint, account.Credentials); // Create Movie Table string tableName = “Movies“; tableClient.CreateTableIfNotExist(tableName);

TableServiceContext context = tableClient.GetDataServiceContext(); // Add movie context.AddObject(tableName, new Movie("Action", “White Water Rapids Survival")); context.SaveChangesWithRetries(); // Query movie var q = (from movie in context.CreateQuery<Movie>(tableName) where movie.PartitionKey == "Action" && movie.Rating > 4.0 select movie).AsTableServiceQuery<Movie>(); foreach (Movie movieToUpdate in q) { movieToUpdate.Favorite = true; context.UpdateObject(movieToUpdate); } context.SaveChangesWithRetries( ); SaveChangesOptions.Batch

Scale

Queries

Entity Group Transactions

Avoid “Append only” write patterns based on PartitionKey values

Avoid using monotonically increasing suffix with a constant prefix

Example: using only the current timestamp as PartitionKey

If needed, add varying prefix to PartitionKey

Restructure your schema if required

Concatenate different keys to form appropriate index

Most Optimal: PartitionKey == “SciFi” and RowKey == “Star Wars”

Scans: Expect continuation tokens PartitionKey == “SciFi” and “Sphere” ≤ RowKey ≤ “Star Wars”

“Action” ≤ PartitionKey ≤ “Thriller”

PartitionKey == “Action” || PartitionKey == “Thriller” - currently scans entire table

“Cars” ≤ RowKey ≤ “Star Wars” - scans entire table

ServicePointManager.DefaultConnectionLimit = X;

SaveChangesWithRetries

AsTableServiceQuery

Conflict NotFound

http://dannycohen.info/2010/04/21/azure-storage-asynchronous-delete

Blobs

Drives

Tables Provide structured storage. A Table is a set of entities, which contain a set of properties

Queues

Message Queue Account

order processing

customer ID order ID http://…

customer ID order ID http://…

cohowinery

Azure Queue

Input Queue

(Work Items)

Azure Queue

Input Queue

(Work Items)

CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudQueueClient queueClient = new CloudQueueClient( account.QueueEndpoint, account.Credentials); //Create Queue CloudQueue queue = queueClient.GetQueueReference(queueName); queue.CreateIfNotExist(); //Add Message CloudQueueMessage message = new CloudQueueMessage(“some content"); queue.AddMessage(message); //Get Message message = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/); // Process Message within the Invisibility Timeout

//Delete Message queue.DeleteMessage(message);

DequeueCount

Windows Azure Storage Explorers (April 2010)

Windows Azure Storage Explorer Block Blob Page Blob Tables Queues Free

Azure Blob Client X Y

Azure Blob Compressor Enables compressing blobs for upload and download

X Y

Azure Blob Explorer X Y

Azure Storage Explorer X X X Y

Azure Storage Simple Viewer X X X Y

Cerebrata Cloud Storage Studio X X X X Y/N

Cloud Berry Explorer X X Y

Clumsy Leaf Azure Explorer

Visual studio plug-in

X X X X Y

Factonomy Azure Utility X Y

Gladinet Cloud Desktop X N

MyAzureStorage.com A portal to access blobs, tables and queues

X X X X Y

Space Block X Y

Windows Azure Management Tool X X X X Y

http://blogs.msdn.com/b/windowsazurestorage/archive/2010/04/17/windows-azure-storage-explorers.aspx

Cerebrata Cloud Storage Studio

http://www.cerebrata.com

Azure MMC Snap-in

http://code.msdn.microsoft.com/windowsazuremmc

Cerebrata – Azure Diagnostics Manager http://www.cerebrata.com/Products/AzureDiagnosticsManager/Default.aspx

Business Analytics

Reporting

Windows Azure Compute

Demo: Create a SQL Azure database

SQL Azure Network Topology Application

Internet

Azure

Cloud

LB

TDS (tcp)

TDS (tcp)

TDS (tcp)

Applications use standard SQL

client libraries: ODBC,

ADO.Net, PHP, …

Load balancer forwards „sticky‟

sessions to TDS protocol tier

Security Boundary

SQL SQL SQL SQL SQL SQL

Gateway Gateway Gateway Gateway Gateway Gateway

Scalability and Availability: Fabric, Failover, Replication, and Load balancing

Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to backend

SQL Azure Network Topology Application

Internet

Azure

Cloud

LB

TDS (tcp)

TDS (tcp)

TDS (tcp)

Applications use standard SQL

client libraries: ODBC,

ADO.Net, PHP, …

Load balancer forwards „sticky‟

sessions to TDS protocol tier

Security Boundary

SQL SQL SQL SQL SQL SQL

Gateway Gateway Gateway Gateway Gateway Gateway

Scalability and Availability: Fabric, Failover, Replication, and Load balancing

Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to backend

Application Topologies SQL Azure access from within

MS Datacenter (Azure compute) SQL Azure Access from outside

MS Datacenter (On-premises)

SQL Azure Access from within and outside MS

Datacenter (On-premises & Azure Compute)

Application/

Browser

SOAP/REST

HTTP/S

Astoria/REST - EDM

HTTP/S

App Code (ASP.NET)

App Code (ASP.NET)

T-SQL (TDS)

SQL Azure Windows

Azure

Code Near

App code/ Tools

T-S

QL

(T

DS

)

SQL Azure

Windows

Azure

Code Far Hybrid

SQ

L A

zu

re

Da

ta S

yn

c

Windows

Azure

SQL Azure

App code/ Tools

App Code (ASP.NET)

App Code (ASP.NET)

T-SQL (TDS)

SQL Azure Cluster

Node 55

DB3

Node 14 Node 19 Node 21 Node 99 Node 2 Node 76

Shards: Managing Lots of Data

App

Server

Master

DB DB1 DB2

Node 33

DB3 DB4 DB5 DB6

SQL Azure Cluster

Node 14 Node 19 Node 21 Node 99 Node 2 Node 76

Copies: Managing Lots of Reads

App

Server

Master

DB

DB

C1

DB

C2

Node 33

DB

C3

DB

C4

DB

C5

DB

C6

LB

What are your application‟s requirements? Storage and Transactional throughput

Storage Requirements Low High

Tra

nsa

ction

al R

eq

uire

me

nts

L

ow

H

igh

• Single Database

• No Partitioning

• Partitioned Data

• Partitioning Based on

Application

Requirements (Storage)

• Partitioned Data

• Partitioning based on

Application Requirements

(IOPS)

• Partitioned Data

• Partitioning based on

Application Requirements

(IOPS, Storage or both)

SQL Azure Tips

Developing on a local SQL Express instance has some advantages

Easy to get started, you already know how to do it!

Full fidelity with the designer and debugging tools

Reduces latency when using local Azure development tools

Reduces bandwidth and databases costs for development

Some caveats

Remember to alter your VS build settings to switch the connection string when deploying

Use tools (like SQLAzureMW) to keep you within the supported SQL Azure features

Always test in SQL Azure before deploying to production

SQL Azure Migration Wizard

http://sqlazuremw.codeplex.com

SQL Azure Tips SQL Azure connection strings follow normal SQL syntax

Applications connect directly to a database “Initial Catalog = <db>” in connection string

No support for context switching (no USE <db>)

Some commands must be in their own batch Create/Alter/Drop Database & Create/Alter/Drop Login, & Create/Alter USER with FOR/FROM LOGIN

Encryption security Set Encrypt = True, only SSL connections are supported

TrustServerCertificate = False, avoid Man-In-The-Middle-Attack!

Format of username for authentication: ADO.Net: Data Source=server.database.windows.net; User ID=user@server;Password=password;...

Setup your firewall rules first!

SQL Azure Tips

SQL Connections: Retry on failure

Connections can drop for variety of reasons Idleness

Transient (network) errors

Intentional throttling

First step: reconnect immediately Handles idleness- and transient-disconnects

Gateway handles connection retry for app Connections attempted for ~30s before failure

What to do on connection failure? Wait (10 seconds), then retry

Change your workload if throttled

SQL Connection Retry Pattern

while (true)

{

using (SqlConnection connection = new SqlConnection(connStr))

{

try

{

connection.Open();

using (SqlCommand cmd = connection.CreateCommand())

{

cmd.CommandText = @"SetBCPJobStartTime";

cmd.CommandType = CommandType.StoredProcedure;

cmd.Parameters.Add(new SqlParameter(@"@BCPJobId", BCPJobId));

cmd.ExecuteNonQuery();

}

}

catch (Exception exc)

{

// deal with error

}

}

// more stuff

// …..

}

SQL Azure Labs Data Sync

SQL Azure Labs Data Sync

SQL Azure Labs Data Sync

Windows Azure Platform

Service Remoting

Control Web services through the Internet using Service Bus

Sender solicits information from listeners

Service Bus

Access Control

Eventing

Notify remote parties of events

Sender transmits information to listeners

Listeners may handle events in different ways

Service Bus

Access Control

Tunneling

Traverse network borders without opening firewalls

Use an HTTP channel to mimic needed protocol

Service Bus

Access Control

Azure AppFabric scenario #1

Scenario: How do we connect Org. A to Org. B ?

Employees of Org. B wish to be informed of events from org. A.

Organization A

Internal

Application

Organization B

Internal

Application

Azure AppFabric scenario

Scenario: How do we connect Org. A to Org. B ?

Option #1 – On-premise integration

BizTalk + SQL + Windows + firewall licenses ?

Hardware, electricity, hosting ?

Maintenance, personnel, support services ?

Organization A

Internal

Application

Organization B

Internal

Application

Fire

wall

Fire

wall

BizTalk

+

SQL

Server

+

Windows

Servers

Azure motivation scenario

Scenario: How do we connect Org. A to Org. B ?

Option #2 – Windows Azure AppFabric integration

Cost: 3.99$ per connection / month (volume discount)

Data transfers: $0.10 in / $0.15 out per GB

Organization A

Internal

Application

Organization B

Internal

Application

http://www.microsoft.com/windowsazure/pricing

Service Bus

Access

Control

Question:

Is BizTalk Server Dead ?

In a word: No.

In two words: Hell, no!

Business Partner

CRM

HR

E-Commerce

ERP

Business

Partner

Integration Server

(EAI/B2B)

Enterprise Service Bus

(ESB)

Windows Azure

Platform

Cloud-enabled SOA platform

AppFabric

Service Bus

AppFabric

Service Bus

Device

On-Premise

Cloud

Enabling hybrid applications with AppFabric

Request

processing

and

scheduling

Website

Front End

Real

Estate

agent

laptop/

device

CRM System

Device

On-Premise

Cloud

Enabling hybrid applications with AppFabric

Request

processing

and

scheduling

Website

Front End

Real

Estate

agent

laptop/

device CRM System

Cloud

On-Premise

Enabling hybrid applications with AppFabric

Request

processing

and

scheduling

Website

Front End

Service Bus

Access Control

Cloud

On-Premise

Enabling hybrid applications with AppFabric

Request

processing

and

scheduling

Website

Front End

Service Bus

Access Control

Cloud

On-Premise

Enabling hybrid applications with AppFabric

Request

processing

and

scheduling

Website

Front

End

Service Bus

Access

Control

Rules OUT claims

IN claims

Cloud

On-Premise

Enabling hybrid applications with AppFabric

Website

Front

End

Service Bus

Access

Control

Request

processing

and

scheduling

two-way

one-way

multicast

multiplex

pub/sub

buffered

Communication

Patterns

Demo: Simple Publish/Subscribe & Multicast

Enabling hybrid applications with AppFabric

http://idesign.net

On-Premise

Site B

On-Premise

Site A

Service Bus

Access Control

Demo: Distributed Cloud based app with Windows Azure & AppFabric

Enabling hybrid applications with AppFabric

Windows Azure

LB

Web

Role

Worker

Role Azure

Queue

Windows Azure Interoperability

http://www.microsoft.com/WindowsAzure/interop

Hosting non-MS applications Mental Model:

Windows OS

Non-admin user

USB drive with “runme.bat”

Can your app run that way?

Use Worker Role

Configure Endpoints as necessary

Call Process.Start(…)

Continue monitoring from the Worker Role

Windows Azure Storage 4 Java

Windows Azure Storage

Blobs, tables, queues, drives

Highly scalable

REST interface

Java storage library (windowsazure4j.org)

Built by Soyatec

Resources

Building Java Applications with Windows Azure http://microsoftpdc.com/Sessions/SVC50

Developing PHP & MySQL Applications with Windows Azure http://microsoftpdc.com/Sessions/SVC51

Ruby on Rails on Windows Azure: http://blogs.msdn.com/b/simondavies/archive/2009/11/25/running-ruby-on-rails-on-windows-azure.aspx

Sample App: http://rubyonrails.cloudapp.net/

Downloads: http://windowsazure.com/interoperability

Thank you!