Tony PetrossianMicrosoft CorporationTony.Petrossian@microsoft.com

SQL Azure

Best Practices

AgendaService Recap“Server” Management Connectivity and TracingUsing multiple databasesMigrating data and schema in and out of SQL Azure

SQL Azure Recap

Relational database serviceBuilt on SQL Server technology foundationHighly scaledHighly secureDatabase “as a Service” – beyond hosting

Customer Value PropsSelf-provisioning and capacity on demandSymmetry w/ on-premises database platformAutomatic high-availability and fault-toleranceAutomated DB maintenance (infrastructure)Simple, flexible pricing – “pay as you grow”

SQL Azure Topology

TDS Gateway

Front-end Node

Protocol Parser

Gateway LogicTDS

Session 1

Front-end Node

Protocol Parser

Gateway Logic

Master Cluster

Master Node

Data Node Components

Partition Manager

Master Node

Data Node Components

Partition Manager

1

2

3

5

7

8

9

Machine 5

SQL Instance

SQL DB

DB5 DB3DB2

Scalability and Availability: Fabric, Failover, Replication, and Load balancing

Machine 6

SQL Instance

SQL DB

Master1Meta 1 DB2 DB3

Machine 4

SQL Instance

SQL DB

DB2 DB3 DB4

Machine 7

SQL Instance

SQL DB

DB4 DB5

Scalability and Availability: Fabric, Failover, Replication, and Load balancing

46

Master1

SQL Azure Connection

Applications connect directly to a database“Initial Catalog = <db>” in connection stringNo support for context switching (no USE <db>)

SQL Azure connection strings follow SQL syntaxExcept for an unusual username format

Format of username for authentication:ADO.Net:Data Source=server.database.windows.net;User ID=user@server;Password=password;...

ODBC:Driver={SQL Server Native Client 10.0}; Server=server.database.windows.net; Uid=user@server;Pwd=password;...

Recommendations

Server ManagementWhere should I create my server?

Co-locate server with Windows Azure app to reduce latency

When should I create a new server?Trade off between geo/admin/billing

Managing ServerThrough the Portal

Add/Drop serverEstablish admin credentialsView usage reportsNetwork access configuration

Through the Master DatabaseFine-tune firewall settings through codeUser loginsUsage and metrics reporting (billing)Create/Drop databases

Admin roles permissionsCREATE/DROP database CREATE/DROP/ALTER loginGRANT/REVOKE rightsModifying server firewall settings

Portal

Master DB

User DB

User DB

User DB

HTTP

TDS

SQL Azure Server

Server: Network Access Control

Each server defines a set of firewall rulesDetermines access policy based on client IPBy default, there is NO ACCESS to server

Controlled using TSQL API against Master DB: sys.firewall_rules, sys.sp_set_firewall_rule, sys.sp_delete_firewall_rule

Portal UXID Name Start IP End IP Create Modify

1 Office 12.1.2.0 12.1.2.255 2010-03-18 …

2010-03-18 …

2 Home 12.2.2.5 12.2.2.5 2010-03-19 …

2010-03-19 …

3 All 0.0.0.0 255.255.255.255

2010-03-19 …

2010-03-19…

Server: Billing and Reporting

sys.bandwidth_usage: usage in KBTime Databas

eDirection

Class Period Quantity

2009-09-17 19:00

TPCH Egress Internal

Peak 55598

2009-09-17 19:00

TPCH Ingress Internal

Peak 76026

… … … … … …

Time SKU Quantity

2009-09-17 19:00

Web 1

2009-09-17 19:00

Business 10

… … …

sys.database_usage: instance count by type per day

Application Design Topics

Most-applicable SQL Best PracticesConnection PoolingQuery ParameterizationBatching

Scaling with data and loadShardingBuilding copies

Deploying and uploading dataBulk copy, BCP.EXESSISMicrosoft Sync Framework Power Pack for SQL Azure

Use Pooled Connections

// When pooling, use connection and return immediately// Do not hold for a long time – pool ensures fast turnaround// on second useusing (SqlConnection conn = new SqlConnection(…)){ conn.Open(); using (SqlCommand cmd = conn.CreateCommand()) { cmd.CommandText = …; … }}using (SqlConnection conn = new SqlConnection(…)){ conn.Open(); …

Increases efficiency by removing re-login

Connections: Retry on failureConnections can drop for variety of reasons

Idleness for more than 30-minutesTransient (network) errorsIntentional throttling

First step: reconnect immediatelyHandles idleness- and transient-disconnects

The SQL Azure Gateway handles connection retry for appConnections attempted for ~30s before failure

What to do on connection failure?Wait (10 seconds), then retry Check the error messages and follow recommendation

Server health can be checked via PortalTSQL APIs will come in later releases

Errors to Catch

Error Severity Description (message text)

40197 17 The service has encountered an error processing your request. Please try again. Error code %d.

40501 20 The service is currently busy. Retry the request after 10 seconds. Code: %d.

40544 20 The database has reached its size quota. Partition or delete data, drop indexes, or consult the documentation for possible resolutions.

40549 20 Session is terminated because you have a long-running transaction. Try shortening your transaction.

40550 20 The session has been terminated because it has acquired too many locks. Try reading or modifying fewer rows in a single transaction.

40551 20 The session has been terminated because of excessive TEMPDB usage. Try modifying your query to reduce the temporary table space usage.

40552 20 The session has been terminated because of excessive transaction log space usage. Try modifying fewer rows in a single transaction.

40553 20 The session has been terminated because of excessive memory usage. Try modifying your query to process fewer rows.

40613 17 Database '%.*ls' on server '%.*ls' is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of '%.*ls'

Connection Patternwhile (true){ using (SqlConnection connection = new SqlConnection(connStr)) { try { connection.Open(); using (SqlCommand cmd = connection.CreateCommand()) { cmd.CommandText = @"SetBCPJobStartTime"; cmd.CommandType = CommandType.StoredProcedure; cmd.Parameters.Add(new SqlParameter(@"@BCPJobId", BCPJobId)); cmd.ExecuteNonQuery(); } } catch (SQLException Sqexc) { // deal with error } catch (Exception exc) { // deal with error } } // more stuff // …..}

Keep it inside the main loop

Tracing Connectivity Problems

Each session assigned a unique ‘sessionId’Tracks session state and service errors

Retrievable from CONTEXT_INFO()Save this with each new connection

If you need support, support personnel will greatly appreciate that value

Tracing Helper Pattern

Guid? sessionId = null;using (SqlConnection conn = new SqlConnection(…)){ // Grab sessionId from new connection using (SqlCommand cmd = conn.CreateCommand()) { conn.Open(); cmd.CommandText = "SELECT CONVERT(NVARCHAR(36), CONTEXT_INFO())"; sessionId = new Guid(cmd.ExecuteScalar().ToString()); }

// continue processing ...}

Maximize your Performance

Batching: push logic to ServerUse stored procedures and batchingLimit number of round trips to server

Example: batch 10 statements vs. 10 round-trips

Parameterized queriesParameterize queries (limits compiles)Declare all parameters, type and lengthLack of parameter size leads to cache bloat

Declare Parameter Lengths!// Length inferred: leads to cache bloat cmd.CommandText = "SELECT c1 FROM dbo.tbl WHERE c2 = @1";cmd.Parameters.Add("@1", SqlDbType.NVarChar).Value = "1";…cmd.Parameters.Add("@1", SqlDbType.NVarChar).Value = "22";

// Length supplied: no cache bloat cmd.CommandText = "SELECT c1 FROM dbo.tbl WHERE c2 = @1";cmd.Parameters.Add("@1", SqlDbType.NVarChar, 128).Value = "1";

(@1 nvarchar(1)) SELECT c1 FROM dbo.tbl WHERE c2 = @1(@1 nvarchar(2)) SELECT c1 FROM dbo.tbl WHERE c2 = @1

(@1 nvarchar(128)) SELECT c1 FROM dbo.tbl WHERE c2 = @1

Maximize elasticity benefits

SQL Azure balances databases across machines

Divide your data into smaller chunksMakes for better load-balancingEnsures we can place your data on the most-appropriate servers

Using shorter transactionsEnsures we can respond to issues fasterAvoid impacting others sharing the same box

Thus avoiding being throttled by SQL Azure

Multi-Database Case Study

Provider of vehicle valuation data> 13 million visitors to their site per month2.5 GB Database SizeData refreshed every weekReplicate data across 6 databases for increased perfIncrease/Decrease database count based on demand

Kelly Blue Book http://www.kbb.com/

Copies: Managing Lots of Read Access

SQL Azure ClusterNode

14Node

19Node

21Node

99 Node 2Node

76

App

Server

Master DB

DBC1

DBC2

Node 33

DBC3

DBC4

DBC5

DBC6

LB

SQL Azure

Castellan

Castellan Venue

DB

Castellan Venue DB’s

Venue 1Partition(

s)

Castellan Venue

DB

Castellan Venue DB’s

Venue 2 Partition(

s)

Castellan Venue

DB

Castellan Venue DB’s

Venue N Partition(

s)

One application DB, many venue DB’s – each partitioned in to many parts (40+)

...

Azure Roles

http:// TicketDirect .* Dynamic Worker(tasks

uploaded as blobs)

Partitioner Worker

Azure StorageQueues for communication between clients and roles

-- - --- - -

-- - --- - -

Tables to record server & partition information

Blobs to store web and worker role resources

Client Applications

Castellan.old (VB6)

Castellan.Azure• Box Office sales• Ticket Printing • System Administration • Venue/Event Management• Partitioning

.Net Service Bus

WCF

On PremiseSQL Server

Castellan Venue

Distributed Cache Worker

MemCache

Sharding Example: TicketDirect Architecture

Shards: Managing Lots of Transactions

SQL Azure ClusterNode

14Node

19Node

21Node

99 Node 2Node

76

App

Server

Master DB

DB1

DB2

Node 33

DB3

DB4

DB5

DB6

Migrating Databases to SQL Azure

Managing the Schema

Moving between SQL Server & SQL AzureUse SSMS script generation toolSQL Azure Migration Wizard

http://sqlazuremw.codeplex.com

Use Data-Tier ApplicationVS2010, SQL Server 2008 R2, SQL Azure SU2 (April 2010)

Build your own scriptsWatch out for unsupported T-SQLDon’t depend on instance or server level objects

Getting Data In and Out

SQL Azure supports standard SQL data import and export patterns

Use bulk loading patterns where possibleBCP – console .EXE bulk load/export toolSSIS – SQL integration serverBulk APIs in ODBC and ADO.Net

SQL Azure supports data synchronizationWith on-premises DBs and client stores

Always good advice:Break batches up into smaller, consumable chunksAdd retry and tracing logic to ensure robust resume in face of failures

Data Import: ADO.Net Bulk Copy API// Bulk data importusing (SqlBulkCopy bulk = new SqlBulkCopy(new SqlConnection(conn)) { DestinationTableName = "dbo.data", BatchSize = 2000, // Transaction size (length) BulkCopyTimeout = 10000, // Transaction timeout NotifyAfter = 1000, // Progress callback }){ bulk.SqlRowsCopied += new SqlRowsCopiedEventHandler( myProgressCallback); bulk.WriteToServer(sourceDataReader);}

Data Export/Import: BCP.EXE

// BCP exampleSET SRV=somesrv.database.windows.netSET LOGIN=mylohin@somesrvSET PW=somethingSET S_DR=C:\flatsSET DB=TPCH

bcp %DB%.dbo.supplier in %S_DR%\supplier.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.nation in %S_DR%\nation.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.region in %S_DR%\region.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.customer in %S_DR%\customer.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.part in %S_DR%\part.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|“

bcp %DB%.dbo.supplier out %S_DR%\supplier.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.nation out %S_DR%\nation.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.region out %S_DR%\region.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.customer out %S_DR%\customer.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.part out %S_DR%\part.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"

Summary

Many SQL Server patterns apply to SQL AzureUse SQL best practices wherever possible

Patterns discussed:Connectivity (to database, not server)Tracing and supportBatching, Pooling and ParameterizationGetting data in and out

Additional Resources

Azure Prices and Packages:http://www.microsoft.com/WindowsAzure/offers/

SQL Azure Information:http://www.microsoft.com/windowsazure/sqlazure/

MSDN Resourceshttp://www.microsoft.com/windowsazure/sqlazure/

Thank you for your Attention!For more Information please contact

Tony PetrossianPrincipal Program ManagerSQL Azure Development

Microsoft Corporationtonypet@microsoft.com