Wenming Ye

Sr. Research Program Manager

Microsoft Research Connections

Twitter: @wenmingye

http://www.windowsazure.com/

en-us/develop/nodejs/how-to-

guides/command-line-tools/

Gallery Images Available

MicrosoftWindows Server 2008 R2

SQL Server Eval 2012

Windows Server 2012

Biztalk Server 2013 Beta

Open SourceOpenSUSE 12.2

CentOS 6.3

Ubuntu 12.04/12.10

SUSE Linux Enterprise Server 11 SP2

VM with persistent drive

VM with persistent drive

VM with persistent drive

Server Rack 1 Server Rack 2

Blobs, Disks, Tables and Queues

8.5 trillion stored objects

900K request/sec on average (2.3+ trillion per month)

# Create containerfrom azure.storage import BlobServiceblob_service = BlobService(account_name, account_key)blob_service.create_container('taskcontainer')

# Uploadfrom azure.storage import BlobServiceblob_service = BlobService(account_name, account_key)blob_service.put_blob('taskcontainer', 'task1', file('task1-upload.txt').read(), 'BlockBlob')

#Downloadfrom azure.storage import BlobServiceblob_service = BlobService(account_name, account_key)blob = blob_service.get_blob('taskcontainer', 'task1')

Data centers

Account

Container Blobs

Table Entities

Queue Messages

https://<account>.blob.core.windows.net/<container>

https://<account>.table.core.windows.net/<table>

https://<account>.queue.core.windows.net/<queue>

Design Goals

• “Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM Symposium on Operating System Principals (SOSP), Oct. 2011

Storage Stamp

LB

Storage

Location

Service

Access blob storage via the URL: http://<account>.blob.core.windows.net/

Data access

Partition Layer

Front-Ends

DFS Layer

Intra-stamp replication

Storage Stamp

LB

Partition Layer

Front-Ends

DFS Layer

Intra-stamp replication

Inter-stamp (Geo) replication

Index

Partition Layer

Partition Layer

Partition Layer

• Does not move data around, only reassigns what part of the index a partition server is responsible for

Partition Layer

Index

Partition Layer

• “Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM Symposium on Operating System Principals (SOSP), Oct. 2011

and Queues (NEW)

Europe

West

North

Europe

Geo-replication

South

Central

US

North

Central

US

Geo-replication

East AsiaSouth

East Asia

Geo-replication

West US East US

Geo-replication

East USWest US

Azure

DNShttp://account.blob.core.windows.net/

DNS lookup

Data access

Hostname IP Address

account.blob.core.windows.net West US

Failover

Update DNS

East US

Geo-replication

Windows

Azure

Storage

180

182

184

186

188

190

192

194

196

198

200

660000

665000

670000

675000

680000

685000

690000

695000

700000

Average of TransactionCount

Average of TPS

0

50

100

150

200

250

300

350

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

6/2

4/2

013

6/2

4/2

013 0

:03

6/2

4/2

013 0

:06

6/2

4/2

013 0

:09

6/2

4/2

013 0

:12

6/2

4/2

013 0

:15

6/2

4/2

013 0

:18

6/2

4/2

013 0

:21

6/2

4/2

013 0

:24

6/2

4/2

013 0

:27

6/2

4/2

013 0

:30

6/2

4/2

013 0

:33

6/2

4/2

013 0

:36

6/2

4/2

013 0

:39

6/2

4/2

013 0

:42

6/2

4/2

013 0

:45

6/2

4/2

013 0

:48

6/2

4/2

013 0

:51

6/2

4/2

013 0

:54

6/2

4/2

013 0

:57

6/2

4/2

013 1

:00

Average of TransactionCount

Average of TPS

J S O N

http://www.nuget.org/packages/WindowsAzure.Storage

XL VM Uploading 512, 256MB Blobs (Total upload size = 128GB)

• C=1, P=1 => Averaged ~ 13. 2 MB/s

• C=1, P=30 => Averaged ~ 50.72 MB/s

• C=30, P=1 => Averaged ~ 96.64 MB/s

• Single TCP connection is bound by TCP

• rate control & RTT

• P=30 vs. C=30: Test completed almost

• twice as fast!

• Single Blob is bound by the limits of a

• single partition

• Accessing multiple blobs concurrently

• scales

P=1,

C=1

P=30, C

=1 P=1…

0

2000

4000

6000

8000

10000

Tim

e (

s)

• XL VM Downloading 50, 256MB Blobs (Total download size = 12.5GB)

• C=1, P=1 => Averaged ~ 96 MB/s

• C=30, P=1 => Averaged ~ 130 MB/s

0

20

40

60

80

100

120

140

C=1, P=1 C=30, P=1Tim

e (

s)

Internet of thingsAudio / Video

Log Files

Text/Image

Social Sentiment

Data Market Feeds

eGov Feeds

Weather

Wikis / Blogs

Click StreamSensors / RFID / Devices

Spatial & GPS Coordinates

WEB 2.0Mobile

Advertising CollaborationeCommerce

Digital Marketing

Search Marketing

Web Logs

Recommendations

ERP / CRM

Sales Pipeline

Payables

Payroll

Inventory

Contacts

Deal Tracking

Terabytes

(10E12)

Gigabytes

(10E9)

Exabytes

(10E18)

Petabytes

(10E15)

Velocity - Variety - variability

Vo

lum

e

1980

190,000$

2010

0.07$

1990

9,000$2000

15$Storage/GB

ERP / CRM WEB

2.0

Internet of things

Big Data, BIG OPPORTUNITY

Big Data is a top priority for institutions

49% CEOs and CIOs are planning big data projects

Software Growth

1.82.5

3.44.6

0

5

2012 2013 2014 2015

Bil

lio

ns

$ 34% compound

annual growth

rate2

Services Growth

2.73.9

5.16.5

0

5

10

2012 2013 2014 2015

Bil

lio

ns

$ 39% compound

annual growth

rate2

1. McKinsey&Company, McKinsey Global Survey Results, Minding Your Digital Business, 2012

2. IDC Market Analysis, Worldwide Big Data Technology and Services 2012–2015 Forecast , 2012

How do I optimize my services

based on patterns of weather,

traffic. How do I build a

recommendation engine?

What’s the social sentiment

of my product?How do I better predict

future outcomes?

Distributed Storage

(HDFS)

Query

(Hive)

Distributed Processing

(MapReduce)

OD

BC

Legend

Red = Core

Hadoop

Blue = Data

processing

Purple =

Microsoft

integration

points and

value adds

Orange = Data

Movement

Green =

Packages

Front

endFront

end

Stream

Layer

Partition

Layer

Name Node

de

Data Node Data Node

Front end

HDFS API

DFS (1 Data Node per Worker Role)

and Compute ClusterAzure Storage (ASV)

…

Azure Blob Storage

Hive, Pig, Mahout, Cascading, Scalding, Scoobi,

Pegasus…

C#, F# Map/Reduce, LINQ to Hive, .NET

management clients

JavaScript Map/Reduce, Browser hosted console,

Node.js management clients

PowerShell, Cross Platform CLI tools

http://www.google.com/imgres?imgurl=http://richfrombechtle.files.wordpress.com/2008/10/vs2010archexplorer.jpg&imgrefurl=http://richfrombechtle.wordpress.com/2008/10/13/&usg=__qPArABkba3JddWP-O2AT7MRoU1s=&h=500&w=749&sz=95&hl=en&start=3&zoom=1&itbs=1&tbnid=mMsoPo--rTSTfM:&tbnh=94&tbnw=141&prev=/images?q=visual+Studio+Application&hl=en&sa=X&tbs=isch:1&prmd=ivns&ei=TGhwTYmPNsKTtwflnuiTDw

Deploying and Interacting With HDInsight Service

demo

Batch Processing Interactive analysis Stream

processing

Query runtime Minutes to hours Milliseconds to minutes Never-ending

Data volume TBs to PBs GBs to PBs Continuous stream

Programming model MapReduce Queries DAG

Users Developers Analysts and developers Developers

Originating project Google MapReduce Google Dremel Twitter Storm

Open source project Hadoop / Spark Drill / Shark /Impala

Hbase / Cassandra

Storm / Apache S4 /Kafka

http://www.windowsazure.com/en-us/develop/net/

http://blogs.msdn.com/b/windowsazurestorage/

http://blogs.msdn.com/b/windowsazurestorage/archive/2011/11/20/windows-azure-storage-a-highly-available-cloud-storage-service-with-strong-consistency.aspx

Windows Azure Python SDKWindows AzureHow to use Service Management from Pythonhttp://www.windowsazure.com/en-us/manage/linux/other-resources/command-line-tools/http://research.microsoft.com/en-us/projects/azure/