Windows AzureData StorageAnton Boyko.NET Developer
Windows Azure StorageStorage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service uses
Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet
Windows Azure Storage AccountUser specified globally unique account name
North Central US Northern Europe
Western Europe East Asia
South East Asia
US Europe Asia
Can choose geo-location to host storage account:
South Central US
West US East US
Storage in the Development FabricProvides a local “Mock” storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or above
http://msdn.microsoft.com/en-us/gg433135
Storage SecurityWindows Azure Storage provides simple security for calls to storage serviceHTTPS endpoint
Two 512bit symmetric keys per storage accountCan be regenerated independently
More granular security via Shared Access Signatures
Windows Azure Storage Abstractions
TablesStructured storage. A table is a set of entities; an entity is
a set of properties.
QueuesReliable storage and delivery of messages for an application.
BlobsSimple named files along with metadata for the file.
DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.
Blob Storage
Blob Storage Concepts
BlobContainerAccount
http://<account>.blob.core.windows.net/<container>/<blobname>
Pages/ Blocks
contoso
PIC01.JPGBlock/Page
Block/Page
PIC02.JPGimages
VID1.AVIvideos
Blob Details
Main Web Service
Operations
PutBlobGetBlobDeleteBlobCopyBlobSnapshotBlob LeaseBlob
Blob Details
Associate Metadata with Blob
Standard HTTP metadata/headers (Cache-Control, Content-Encoding, Content-Type, etc)
Metadata is <name, value> pairs, up to 8KB per blob
Either as part of PutBlob or independently
Blob Details
Blob always accessed by
name
Can include ‘/‘ or other delimeter in name e.g. /<container>/myblobs/blob.jpg
GET http://.../products?comp=list&prefix=Tents&delimiter=/
<Blob>Tents/PalaceTent.wmv</Blob><Blob>Tents/ShedTent.wmv</Blob>
Enumerating Blobs
GET Blob operation takes parametersPrefixDelimiterInclude= (snapshots, metadata etc…)
http://adventureworks.blob.core.windows.net/ Products/Bikes/SuperDuperCycle.jpg Products/Bikes/FastBike.jpg Products/Canoes/Whitewater.jpg Products/Canoes/Flatwater.jpg Products/Canoes/Hybrid.jpg Products/Tents/PalaceTent.jpg Products/Tents/ShedTent.jpg
http://.../products?comp=list&prefix=Canoes&maxresults=2
&marker=MarkerValue
<Blob>Canoes/Hybrid.jpg</Blob>
Pagination
Large lists of Blobs can
be paginatedEither set maxresults or;Exceed default value for maxresults (5000)
http://.../products?comp=list&prefix=Canoes&maxresults=2
<Blob>Canoes/Whitewater.jpg</Blob><Blob>Canoes/Flatwater.jpg</Blob><NextMarker>MarkerValue</NextMarker>
Two Types of Blobs Under the Hood
Block BlobTargeted at streaming workloads
Each blob consists of a sequence of blocksEach block is identified by a Block ID
Size limit 200GB per blob
Optimistic Concurrency via Etags
Page BlobTargeted at random read/write workloads
Each blob consists of an array of pages Each page is identified by its offset from the start of the blob
Size limit 1TB per blob
Optimistic or Pessimistic (locking) concurrency via leases
TheBlob.wmv
Uploading a Block Blob
Uploading a large blob
10 GB Movie
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,
blockId1,…,blockIdN);
TheBlob.wmv
BenefitEfficient continuation and retryParallel and out of order upload of blocks
THE BLOB
Windows AzureStorage
Page Blob – Random Read/Write
Create MyBlobSpecify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them
Fixed Page Size = 512 bytesRandom Access Operations
PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)
GetPageRange[0, 4096) returns valid data ranges:
[0,512) , [1536,2560)GetBlob[1000, 2048) returns
All 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)
0
10 GB
5121024153620482560
10 GB Address Space
Shared Access Signatures
Fine grain access rights to blobs and containersSign URL with storage key – permit elevated rightsRevocationUse short time periods and re-issueUse container level policy that can be deleted
Content Delivery Network (CDN)
High-bandwidth global blob content delivery24 locations globally (US, Europe, Asia, Australia and South America), and growingSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Blob service URL vs. CDN URL:Windows Azure Blob URL: http://images.blob.core.windows.net/Windows Azure CDN URL: http://<id>.vo.msecnd.net/ Custom Domain Name for CDN: http://cdn.contoso.com/
pic1.jpg
Windows Azure CDN
To Enable CDN:Register for CDN via Dev PortalSet container images to public
pic1.jpg
GEThttp://guid01.vo.msecnd.net/images/pic.1jpg
http://sally.blob.core.windows.net/images/pic1.jpg
http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/
pic1.jpg
404
TTL Content Delivery Network
Windows Azure Blob Service
EdgeLocation
EdgeLocation
EdgeLocation
Table Storage
Table Storage Concepts
EntityTableAccount
contoso
Name =…Email = …
Name =…EMailAdd=
customers
Photo ID =…Date =…
photos
Photo ID =…Date =…
Table Details
InsertUpdate Merge – Partial updateReplace – Update entire entityDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction
Create, Query, DeleteTables can have metadata
Not an RDBMS! Table
Entities
Entity Properties
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic ConcurrencyExposed as an HTTP Etag
No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double
No Fixed Schema
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
FAV SPORT
Canoeing
Querying
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
?$filter=Last eq ‘Wegner’
Query Operators(Table Service Support)• From• Where• Take
(the value specified for the Take operator must be less than or equal to 1000)
• First• FirstOrDefault• Select
(projection is supported)
more details http://msdn.microsoft.com/en-us/library/windowsazure/dd135725.aspx
Purpose of the PartitionKey
Entity LocalityEntities in the same partition will be stored together
Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction
Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE) TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
Bikes Quick Cycle 200 Deluxe … 2007
… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006PARTITIONKEY(CATEGORY)
ROWKEY(TITLE) TIMESTAMP MODELYEAR
Rafts 14ft Super Tourer … 1999
… … … …
Skis Fabrikam Back Trackers … 2009
… … … …Tents Super Palace … 2008
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE) TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
Bikes Quick Cycle 200 Deluxe … 2007
… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006Rafts 14ft Super Tourer … 1999… … … …
Skis Fabrikam Back Trackers … 2009
… … … …Tents Super Palace … 2008
Partitions and Partition Ranges
Server ATable = Products
Server BTable = Products
[Canoes - MaxKey)
Server ATable = Products
[MinKey - Canoes)
SQL Azure
A Server Is Not A Machine
SQL Server A Machine
SQL Azure Database
ServerA TDS
Endpoint
How It WorksArchitectureClient Layer - Used by application to communicate directly with SQL Database.Services Layer – Gateway between Client layer and Platform layer.Platform Layer – Includes physical servicers and services that support the Services layer.Infrastructure Layer – IT administration of the physical HW and OS.
Client Layer
PHP WCFSQL App and Tools
ODBC ADO .NET
Tabular Data Stream (TDS)
Service Layer
TDS + SSL Endpoint
Create Database…Use Familiar TechnologiesTransact-SQLLanguages
.NET Framework (C#, Visual Basic, F#) via ADO.NETC / C++ via ODBCJava via Microsoft JDBC providerPHP via Microsoft PHP provider
FrameworksOData, Entity Framework, WCF Data Services, NHibernate
ToolsSQL Server Management Studio (2008 R2 and later)SQL Server command-line utilities (SQLCMD, BCP)CA Erwin® Data ModelerEmbarcadero Technologies DBArtisan®
SQL Server ComparisonFocus on logical vs. physical administrationDatabase and log files automatically placedThree high-availability replicas maintained for every databaseTables require a clustered indexMaximum database size is 150 GBUnsupported SQL Server FeaturesUse command, distributed transactions, distributed viewsService BrokerCommon Language Runtime (CLR)SQL Agent
Services Layer
Internet
SQL Database Firewall
SQL Database FirewallSecuring your dataIP Address-based access control for SQL DatabaseRules can be defined at the server and databaseNo IP authorized by defaultConfigurable using the SQL Database Portal and REST APIOption to disable/enable access from applications hosted in Windows Azure
SQL FederationDatabase ScalabilityScale to hundreds of nodes via database shardingMulti-tenancy via flexible repartitioningOnline split operations to minimize downtimeAutomatic data discovery regardless of changes in how data is partitioned
Q & A