Page 1

Queue Storage

Jeff Chu | 2010-6-24

Page 2

Agenda• Queue Storage Overview

• Programming Queue Storage

• Queue Storage tips

• Lab Time

Page 3

Queue Storage Overview

MessageQueueAccount

order processing

customer ID order ID http://…

customer ID order ID http://…

cohowinery

Page 4

Queue Storage Overview• Data Structure…

– FIFO (First-in, First-out)– FCFS (First-come, First-serve)

• Unlimited message stores– Message size limits: 8KB– Base on byte count, not text length.

• Inter-role communication– Web to Worker communication– Worker to Web communication

Page 5

Queue Storage Overview

Azure Queue

Input Queue (Work Items)

Page 6

QUEUE’S RELIABLE DELIVERYGuarantee delivery/processing of messages (two-step consumption)

1. Worker Dequeues message and it is marked as Invisible for a specified “Invisibility Time”

2. Worker Deletes message when finished processing it

If Worker role crashes, message becomes visible after the invisibility time for another Worker to process

Azure QueueInput Queue (Work Items)

Page 7

21

21

11

11

C1C1

C2C2

Removing Poison Messages

11

11

21

21

3340

40

Producers Consumers

P2P2

P1P1

30

30

2. GetMessage(Q, 30 s) msg 2

1. GetMessage(Q, 30 s) msg 1

11

11

21

21

7

10

10

20

20

Page 8

C1C1

C2C2

Removing Poison Messages

340

40

Producers Consumers

P2P2

P1P1

11

11

21

21

2. GetMessage(Q, 30 s) msg 23. C2 consumed msg 24. DeleteMessage(Q, msg 2)7. GetMessage(Q, 30 s) msg 1

1. GetMessage(Q, 30 s) msg 15. C1 crashed

11

11

21

21

6. msg1 visible 30 s after Dequeue30

30

8

12

1211

1112

12

Page 9

C1C1

C2C2

Removing Poison Messages

340

40

Producers Consumers

P2P2

P1P1

12

12

2. Dequeue(Q, 30 sec) msg 23. C2 consumed msg 24. Delete(Q, msg 2)7. Dequeue(Q, 30 sec) msg 18. C2 crashed

1. Dequeue(Q, 30 sec) msg 15. C1 crashed10. C1 restarted11. Dequeue(Q, 30 sec) msg 112. DequeueCount > 213. Delete (Q, msg1)

12

12

6. msg1 visible 30s after Dequeue9. msg1 visible 30s after Dequeue

30

30

9

13

1312

1213

13

Page 10

Programming Queue Storage• Get Client

– CloudQueue.CreateIfNotExist()

• Create Message– CloudQueueMessage class

– CloudQueue.AddMessage()

• Get Message– Get CloudQueueMessage, and change message’s property to invisible.

– GetMessages() accept max 32 messages.

• Peek Message– Get CloudQueueMessage, but DON’T change invisible property.

• Delete Message– CloudQueue.DeleteMessage()

– CloudQueue.Clear ()

Page 11

Programming Queue Storage• Get queue count

– RetrieveApproximentMessageCount() : must call this to update active message count.

– ApproximentMessageCount Property

• CloudQueueMessage class– Message can archive any string or data, but length cannot over 8KB.

– DequeueCount

• Times– ExpirationTime: Time of message will be deleted by Queue Storage.

– InsertionTime: Time of message inserted into queue.

– NextVisibleTime: Time of next visible time.

• Metadata– Queue Container support metadata (like BLOB storage).

Page 12

Programming Queue StorageCloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");CloudQueueClient queueClient = new CloudQueueClient( account.QueueEndpoint, account.Credentials);

//Create QueueCloudQueue queue = queueClient.GetQueueReference(queueName);queue.CreateIfNotExist();

//Add MessageCloudQueueMessage message = new CloudQueueMessage(“some content");queue.AddMessage(message);

//Get Messagemessage = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/);

// Process Message within the Invisibility Timeout

//Delete Messagequeue.DeleteMessage(message);

Page 13

Queue Storage tips• Messages can be up to 8KB

– Use blob to store large messages, and store blob ref in message

• A message may be processed more than once– Make message processing idempotent

• Work should be repeatable and can be done multiple times

• Assume messages put into queue can be processed in any order

• For higher throughput– Batch multiple work items into a single message or into a blob– Use multiple queues

• Use DequeueCount to remove poison messages– Enforce threshold on message’s dequeue count

• Monitor message count to dynamically increase/reduce workers

Page 14

Queue Storage tips• If your message must over 8KB…

– Store your message into BLOB storage and pass the reference to queue.

• Producer and Consumer Problem– Delete after data consumed.– Pass reference to consumer to process

finalized data.

Page 15

Show time for you!

Page 16

References• Windows Azure SDK – Table Storage

– http://msdn.microsoft.com/en-us/library/dd179423.aspx• Tech.Ed 2010 North America Session

– COS310: Windows Azure Storage Deep Dive– http://www.msteched.com/2010/NorthAmerica/COS310

• 邊做邊學 Windows Azure: Table, BLOB and Queue – http://msdn.microsoft.com/zh-tw/azure/ff477598.aspx

• 認識 Table Storage– http://www.dotblogs.com.tw/regionbbs/archive/

2010/03/01/vs2010.and.azure.platform.understanding.table.storage.service.aspx