+ All Categories
Home > Software > AMQP 1.0 introduction

AMQP 1.0 introduction

Date post: 15-Apr-2017
Category:
Upload: clemens-vasters
View: 1,109 times
Download: 0 times
Share this document with a friend
38
Introduction to AMQP 1.0 AMQP Foundations emens Vasters chitect, Messaging, Microsoft Corporation [email protected] © 2015 Microsoft Corpo All content in the presentation is available for reuse CC-BY 4.0 https://creativecommons.org/licenses/by
Transcript
Page 1: AMQP 1.0 introduction

Introduction to AMQP 1.0

AMQP Foundations

Clemens VastersArchitect, Messaging, Microsoft [email protected]

© 2015 Microsoft CorporationAll content in the presentation is available for reuse under

CC-BY 4.0 https://creativecommons.org/licenses/by/4.0/

Page 2: AMQP 1.0 introduction

What is AMQP 1.0?

• Secure, compact, symmetric, multiplexed, reliable, binary transfer protocol to move messages between applications• International ISO/IEC 19464:2014 standard• Layered model

• Transport and connection security protocol • Frame transfer protocol• Message transfer protocol

• Not tied to any particular message source or target model or topology. Supports classic message brokers, modern hyper-scale messaging infrastructure, and peer-to-peer exchange

Page 3: AMQP 1.0 introduction

Container

Incoming Connection

Endpoint

Container

Frame Transfer Protocol Overview

• Containers (Apps) connect to other containers• Connections manage transfer capacity (frame size, channel count)• Bidirectional sessions formed over pairs of unidirectional channels• Sessions allow for multiplexed conversations between containers• Connections and sessions are ephemeral

ConnectionCH0

CH0CH1

CH1Outgoing ConnectionEndpoint

Session Endpoint

Session Endpoin

t

SessionCH2

CH2

Page 4: AMQP 1.0 introduction

Container

Hierarchical

NodeTopology

• Container• Communicating app

• Node• Addressable entity within the

app• Nodes can be organized in any

way the application sees fit; flat, hierarchy, or graph• Node can be a queue or topic or

relay or event store or …• Link• Links connect via paths to

nodes

Nodes

a

b

c

d

b1b2b3b4

/

FlatNode

Topology

a

b

c

d

Session

Endpoint

Link Endpoint

a

Link Endpoint

c

Session

Endpoint

Link Endpoint

/a

Link Endpoint

/b/b4

Containers and Nodes

Page 5: AMQP 1.0 introduction

ContainerContainer

Message Transfer Protocol Overview

• Unidirectional, named links are formed over sessions• Links manage flow individually through link credits• Links can outlive collapsed connection/sessions and can be

recovered

Session Endpoint

Session EndpointSession

Link Endpoint

Link Endpoint

downstream_link_name

Link Endpoint

Link Endpointupstream_link_name

MNode Node

Page 6: AMQP 1.0 introduction

AMQP 1.0 Core Protocol Elements

Connections, Security, Sessions, and Links

Page 7: AMQP 1.0 introduction

Connection

• Layered over network stream-transport (usually TCP; SCTP, Pipes are suitable)• Connection provides a reliably ordered

sequence of frames w/ negotiated maximum frame size• Frames flow over a negotiated number

of multiplexed, unidirectional channels• Explicit idle-timeout management• Connection preamble indicates

protocol version. Integration with transport-level security; TLS and SASL security can be negotiated inline.

Incoming Connection

Endpoint ConnectionCH0

CH0CH1

CH1

CH2CH2

Outgoing ConnectionEndpoint

OPEN(container-id=0, hostname=ep.example.com, …)

OPEN(container-id=0, max-frame-size=262144, …)

CLOSE()

CLOSE()

Page 8: AMQP 1.0 introduction

TLS WS

Connection-Level Security Layers• Transport-Level Security (TLS/“SSL”)

1. Single-port model for AMQP and secure AMQP• Sender indicates desire for TLS session with a preamble

(protocol id 2) first, then AMQP version negotiation occurs. TLS negotiation occurs inline

2. “Pure TLS” model for AMQPS on dedicated port 5671• Sender opens TLS session to server on port 5671 without

upgrade3. WebSockets tunnel on port 443

• Sender creates WebSocket with TLS and overlays AMQP [AMQP 1.0 over WS spec draft]

• SASL Authentication• Optional. Negotiated with a preamble (protocol id 3)

after initial TCP version negotiation and before AMQP version negotiation

• Supports any SASL authentication model• SASL External permits binding TLS client

authentication context to the AMQP connection

AMQP TLS

TLS AMQP

AMQP

1

2

3

ConnectionTLS Connection

TLS

SASL

Page 9: AMQP 1.0 introduction

Session• Binds two unidirectional

channels to form a bidirectional, sequential conversation• Provides a window-based flow

control model • Number of total transfer frames

that sender and receiver can hold in buffers at any given time

• Decouples credit-based flow control driven by API and capacity management requirements of the underlying platform

• A connection can support multiple concurrent sessions

Session Endpoint

Session Endpoint

Session

CH3CH7

[CH3] BEGIN( remote-channel=null, …)

[CH7] BEGIN( remote-channel=3, …)

END()

END()

Page 10: AMQP 1.0 introduction

Link

• Named, unidirectional transfer route for messages from “source” to “target” node• A session can accommodate any

number of concurrent links• Links in either direction can be

initiated by either peer.• Links can be recovered on a

different connection/session when the previous broke• Credit-based flow control to

model flow management for various API shapes

ATTACH(name=N, handle=1, role=sender, source=A, target=B, …)

ATTACH(name=N, handle=2, role=receiver, source=A, target=B, …)

Link Endpoint

Link Endpoint

Session Endpoint

Session EndpointSession

link_name

DETACH()

DETACH()

Page 11: AMQP 1.0 introduction

Message Transfers

Page 12: AMQP 1.0 introduction

Message Transfers – Settlement

• Messages are transferred fire-and-forget (“settled”) or requiring explicit settlement, allowing for various delivery state handling and delivery assurance models• At-Most-Once / Fire-And-Forget

• TRANSFER(settled=true)

• At-Least-Once• TRANSFER(delivery_tag=DT, settled=false, state=S_0)

• DISPOSITION(delivery_tag=DT, settled=true, state=T_0)

Sender

Receiver

Session Endpoint

Session Endpoint

Sessionlink_name

Cr

TRANSFER(…, settled=false, …)

DISPOSITION(…, settled=true, …)

Page 13: AMQP 1.0 introduction

Message Transfers – Delivery State

• Intermediary States (transfer ongoing)• received – used for link recovery negotiation,

indicating that an unsettled message had been received

• Terminal States (transfer is done)• accepted – the message has been accepted

by the receiver (OK)• rejected – the message has been rejected by

the receiver (with error/diagnostics details)• released – the message has been abandoned

by the receiver and should be redelivered• modified – the message has been released

(see above) and should be modified at the source as indicated in the details

Sender

Receiver

Session Endpoint

Session Endpoint

Sessionlink_name

Cr

TRANSFER(settled=false, state=null, …)

DISPOSITION(settled=true, state=accepted, …)

Page 14: AMQP 1.0 introduction

Message Transfer – Recovering Links

• Links can be recovered from previous, broken or abandoned connections/sessions• Message delivery states are

exchanged as the link is re-attached by the peers• Transfers are subsequently

recovered to resend, reconcile state, or continue settlement • TRANSFER(…,resume=true, …)

Sender

Receiver

Session Endpoint

Session Endpoint

Sessionlink_name

Cr

ATTACH(role=sender, src=A, trg=B, unsettled = { 1-> null, 2-> null, 3-> accepted, 4-> null }, …)

ATTACH(role=sender, src=A, trg=B, unsettled = { 1-> released, 2-> null, 3-> accepted }, …)

Page 15: AMQP 1.0 introduction

Modeling Bi-Di and Request/Response

• For a request/response relationship, the client C initiates two links with the target node T• [CT] role=sender• [TC] role=receiver

• The client’s response node can be scoped to the conversation or global. Referenced in request.reply-to• Correlation via reply.correlation-

id set to request.correlation-id (or request.message-id if absent)

ATTACH(role=sender, source=C, target=T, …)

ATTACH(role=receiver, src=C, trg=T)

Session Endpoint

Session Endpoint

Session

request_link

reply_linkLEP

LEP

LEP

LEP

TC

ATTACH(role=receiver, source=T, target=C, …)

ATTACH(role=sender, src=T, trg=C)

Page 16: AMQP 1.0 introduction

Transfers – Flow Control

Page 17: AMQP 1.0 introduction

AMQP Body Frame TypesType Connect

ionSession Link Description

OPEN Handle Connection openBEGIN Inspect Handle Session beginATTACH Inspect Handle Link attachFLOW Inspect Handle Update flow control stateTRANSFER Inspect Handle Transfer messageDISPOSITION Inspect Handle Update transfer stateDETACH Inspect Handle Link detachEND Inspect Handle Session endCLOSE Handle Connection close

✔✔✔

✔✔✔

Already discussed

Page 18: AMQP 1.0 introduction

• Each session endpoint has an incoming and outgoing transfer window defining an upper bound for messages in transit

• Window size is expressed in frame count; maximum frame size and window size gate the memory capacity required for a session

• Transfers decrement the remaining window sizes. An outgoing window size of <=0 suspends transfers. Window sizes are updated/reset with FLOW

• Each link has an associated link-credit, which is the number of messages the receiver is currently willing to accept

• Messages can flow when the remaining link-credit is greater than zero. Only the receiver can manipulate the link credit value.

• Transfers decrement the remaining link credit. A link credit of <=0 suspends transfers. Link credit is updated with FLOW

Session Flow Control Platform Resource Management

Link Flow ControlApplication-Level Message Flow

Management

Session Endpoint

Session Endpoint

Session

CH3CH7

Out

In

In

Out

Link Endpoint

Link Endpoi

nt

Session Endpoint

Session Endpoint

Session

link_nameCr

Page 19: AMQP 1.0 introduction

Wait! Why two flow control models?• Session flow control protects the platform• Guards high-scale and hyper-scale platform implementations

from overcommitting transfer resources to sessions; shields from unexpected transfer bursts• Allows for session-level capacity management and throughput

throttling• Link flow control protects the application and supports

API gestures• Guards an application node from having to accept more

messages than it can currently handle. • Prevents concurrent, multiplexed session/link traffic from

being backed up behind messages that cannot currently be handled• Enables on-demand receive, pre-fetch receive, and just-in-

time receive (“push”) operations

Page 20: AMQP 1.0 introduction

Some API PatternsImperative: receiver.Receive()

Sender

Receiver

Session Endpoint

Session Endpoint

Sessionlink_name

Cr

TRANSFER()

FLOW(link-credit=1)

Reactive: observer.Subscribe(callback)

Sender

Observer

Session Endpoint

Session Endpoint

Sessionlink_name

Cr

TRANSFER()

FLOW(link-credit={concur-ops-limit})

Throttled: queueClient.Send()

Sender Queue

Session Endpoint

Session Endpoint

Sessionlink_name

Cr

FLOW(link-credit={throttle-delta})

Prefetch: receiver.SetPrefetch(10)

Sender

Receiver

Session Endpoint

Session Endpoint

Sessionlink_name

Cr

TRANSFER()

FLOW(link-credit=10)

TRANSFER()

Credit limited by concurrent operations

that the reactive system can perform

Credit limited by how many messages the application wants to keep available for immediate

processing

Credit limited by how many messages the receiving queue can store vs.

sender’s throughput capacity

Page 21: AMQP 1.0 introduction

Application Timeout/Cancellation/Pause

“drain” allows any immediately pending transfers to complete and then promptly

reduces the link credit to zero.

Sender

Receiver

Session Endpoint

Session Endpoint

Sessionlink_name

Cr

FLOW(link-credit=1)

…. time passes …FLOW(drain=true)

Setting the link-credit to zero immediately stops the link. Pending messages are no

longer transferred. Resume with increasing credit.

Sender

Receiver

Session Endpoint

Session Endpoint

Sessionlink_name

Cr

FLOW(link-credit=1)

…. time passes …FLOW(link-credit=0)

Page 22: AMQP 1.0 introduction

AMQP on the WirePrimitive Types

Page 23: AMQP 1.0 introduction

The AMQP Type System• AMQP has its own, portable, language-independent type

system• Used for all core AMQP protocol elements• Optimized for fast and flexible processing and compact

representation• Defines a broad range of primitive and composite types• Permits descriptive references to external type systems• Can be used to express and encode message payloads

• Supports schema-bound and schema-free encoding• Schema-bound (like Avro/ProtoBuf/Thrift)

• All fixed protocol elements• Only transmit data and rely on out-of-band-shared metadata for interpretation.

• Schema-free (like JSON/MsgPack) • Option for application metadata and payloads

Page 24: AMQP 1.0 introduction

Schema Example – Composite Type

• AMQP Schema notation is XML• DTD included in the core spec

• Type classes• primitive – built-in types

• Includes array, list, and map• composite – multi-value type

• Inherits from “list” primitive• restricted – restriction of existing type

• Restricts source type value space• Archetypes

• provides – categorization of types• Descriptor

• Unique string and numerical descriptors• Composite: Fields

• Zero or more fields

<type name="properties" class="composite" source="list" provides="section"> <descriptor name="amqp:properties:list" code="0x00000000:0x00000073"/> <field name="message-id" type="*" requires="message-id"/>

<field name="user-id" type="binary"/> <field name="to" type="*" requires="address"/> <field name="subject" type="string"/> <field name="reply-to" type="*" requires="address"/> <field name="correlation-id" type="*" requires="message-id"/> <field name="content-type" type="symbol"/> <field name="content-encoding" type="symbol"/> <field name="absolute-expiry-time" type="timestamp"/> <field name="creation-time" type="timestamp"/> <field name="group-id" type="string"/> <field name="group-sequence" type="sequence-no"/> <field name="reply-to-group-id" type="string"/></type>

properties – composite type schema

Page 25: AMQP 1.0 introduction

Schema Example – Restricted Type

• Restricted types inherit source type• here: ubyte, unsigned byte

• Type may constrain the value space• Explicit permitted values• Named choices act as alias

• May directly alias source• May override source

descriptor

<type name="sender-settle-mode" class="restricted" source="ubyte"> <choice name="unsettled" value="0"/> <choice name="settled" value="1"/> <choice name="mixed" value="2"/></type>

sender-settle-mode – restricted type schema

<type name="seconds" class="restricted" source="uint"/>

seconds – restricted type schema

<type name="delivery-annotations" class="restricted“ source="annotations" provides="section"> <descriptor name="amqp:delivery-annotations:map“ code="0x00000000:0x00000071"/> </type>

delivery-annotations – restricted type schema

Page 26: AMQP 1.0 introduction

Data on the wire

• All data elements are prefixed with a “constructor” indicating type• Built-in, primitive types use a single octet (0xA1 designates str8-utf) • Composite and restricted types use a symbolic or numeric (ulong) descriptor

0xA1

“Constructor”

0x1E

Type-specific preamble

“Hello Glorious Messaging World”

Type-specific data

str8-utf str8-utf length

str8-utf data

0x00

Constructor

descriptor follows

0xA3

sym8

0x11

sym8 length“example:book:lis

t” 0x0Csym8 value – type name primitive

Descriptor

… contents …

Page 27: AMQP 1.0 introduction

Fixed Size Primitives

• Fixed size primitives are organized by encoding size• null, Boolean true and false,

and zero-valued uint and ulong can be expressed with a constructor (no data byte)• uint and ulong valued < 256

encodes into 1 data byte• int and long valued from

-128 to 127 encodes into 1 data byte

0 1 2 4 8 16null 0x40boolean 0x41: true

0x42: false

0x56

ubyte 0x50ushort 0x6

0uint 0x43: v=0 0x52: v<256 0x7

0ulong 0x44: v=0 0x53: v<256 0x8

0byte 0x51short 0x6

1int 0x54: -

128<v<127long 0x55: -

128<v<1270x81

float 0x72

double 0x82

decimal32

0x74

decimal64

0x84

decimal128

0x94

char 0x73

timestamp

0x83

uuid 0x98

data bytes occupied by type

Page 28: AMQP 1.0 introduction

Variable Size Primitives

• Variable size primitive encodings have a 1 or 4 byte preamble. Which encoding is used depends on the actual data size. • UTF-8 strings, symbols, and binary data of up to 255 bytes has a

single byte preamble, otherwise a 4 byte preamble is used

0xA1 0x1E

1 byte

...

up to 255 byte

str8-utf str8-utf length

str8-utf data

0xB1 0x00000100

4 byte

...

up to 232-1 byte

str32-utf str32-utf length

str32-utf data

1 4binary – vbin8 0xA0

binary – vbin32 0xB0

string – str8-utf 0xA1

string – str32-utf 0xB1

symbol – sym8 0xA3

symbol – sym32 0xB3

preamble size

Page 29: AMQP 1.0 introduction

0xC0 0x03

list element count

0x50 0x02

[0] ubyte 2

0x61 0x07DF

[1] short 2015

0xA1 0x03

[2] string “Hi!”

“Hi!”

List – Polymorphic Sequence of Elements

0xD0 0x04

mapelement

count(K/V pairs) * 2

0xA1 0x04

[K0] string “Key1”

“Key1”

Map – Polymorphic Sequence of Key/Value Pairs

0xA1 0x04

[V0] string “Val1”

“Val1” Key2 Val2

[K1] [V1]

0xE0 0x05

array element count

0x07DF

[0]

Array – Monomorphic Sequence of Elements

0x61

element ctor

0x07DF 0x07DF 0x07DF 0x07DF

[1] [2] [3] [4]short

Page 30: AMQP 1.0 introduction

AMQP on the WireComposite Types and Messages

Page 31: AMQP 1.0 introduction

AMQP Messages

• The standard AMQP message format shown here is replaceable/extensible. • Message format kind and version is

selected on TRANSFER frame. Default format is zero.

• Bare Message core is immutable from sender to ultimate receiver through all intermediaries• Annotated Message frame may be

altered by intermediaries• All sections defined as composite

types (archetype “section”)

headertransfer headers

delivery-annotations

custom delivery metadatamessage-annotations

infrastructure-specific properties

propertiesstandard message

propertiesapplication-properties

application-defined metadata

Message Body(explained later)

footercustom metadata about the annotated msg (e.g.

signature)

AnnotatedMessage

BareMessage

0..1

0..1

0..1

0..1

0..1

0..1

1

Page 32: AMQP 1.0 introduction

Composite Type Encoding

• Prefixed with type descriptor • Encoded as list• List contains elements in schema

order• Trailing elements can be omitted<type name="properties" class="composite" source="list" provides="section"> <descriptor name="amqp:properties:list" code="0x00000000:0x00000073"/> <field name="message-id" type="*" requires="message-id"/> <field name="user-id" type="binary"/> <field name="to" type="*" requires="address"/> <field name="subject" type="string"/> <field name="reply-to" type="*" requires="address"/> <field name="correlation-id" type="*" requires="message-id"/> <field name="content-type" type="symbol"/> <field name="content-encoding" type="symbol"/> <field name="absolute-expiry-time" type="timestamp"/> <field name="creation-time" type="timestamp"/> <field name="group-id" type="string"/> <field name="group-sequence" type="sequence-no"/> <field name="reply-to-group-id" type="string"/></type>

properties – composite type schema

0xC0 0x04

list element count

0xA1 0x04 “A4D5”

0xA1 0x08 “clemensv”

0xA1 0x02 “q1”

0xA1 0x05 “hello”

0x00

descriptor

0x53

ulong8

0x73

code

Page 33: AMQP 1.0 introduction

The AMQP message body consists of one of the following three choices: • One or more data sections

• Binary data up to remaining maximum frame size • Hints to interpret this data can be carried in the

standard content-type and content-encoding properties.

• JSON, MsgPack, Avro, Thrift, XML, etc. payloads• One or more amqp-sequence sections

• A sequence is a list of AMQP encoded values • A single amqp-value section

• A single AMQP value (including null)• Can be a composite type value

dataraw binary data

amqp-sequencesequence of data elements

1..*

or

amqp-valuesingle data value

or

1..*

1

Application Data – AMQP Message Body

Page 34: AMQP 1.0 introduction

{ “books” : [ { “title” : “Normal Accidents”, “authors” : [“Charles Perrow”], “isbn” : “978-0-691-00412-9” }, { “title” : “Rockets and People, V3”, “authors” : [“Boris Chertok”, “Asif Siddiqi”], “isbn” : “978-0-160-81733-5” } ]}

JSON to AMQP - Schemaless

map 2str8-utf 5 “books” list 2 map 6 str8-utf 5 “title” strf8-utf 16 “Normal Accidents”, str8-utf 7 “authors” array 1 str8-utf 14 “Charles Perrow” str8-utf 4 “isbn” strf8-utf 16 “978-0-691-00412-9” map 6 str8-utf 5 “title” strf8-utf “Rockets and People, V3” str8-utf 7 “authors” array 2 str8-utf 13 “Boris Chertok”, 12 “Asif Siddiqi” str8-utf 4 “isbn” str8-utf 16 “978-0-160-81733-5” Identical in shape, easy from/to mapping

nearly identical in concrete footprint, but AMQP with deeper type system

0xD0 0xC0 0xA1 0xE0

Page 35: AMQP 1.0 introduction

{ “books” : [ { “title” : “Normal Accidents”, “authors” : [“Charles Perrow”], “isbn” : “978-0-691-00412-9” }, { “title” : “Rockets and People, V3”, “authors” : [“Boris Chertok”, “Asif Siddiqi”], “isbn” : “978-0-160-81733-5” } ]}

JSON to AMQP – Using Schema

map 2str8-utf 5 “books” list 2 desc ulong 0x0000000300000002 list strf8-utf 16 “Normal Accidents”, array 1 str8-utf 14 “Charles Perrow” strf8-utf 16 “978-0-691-00412-9” desc ulong 0x0000000300000002 list strf8-utf “Rockets and People, V3” array 2 str8-utf 13 “Boris Chertok”, 12 “Asif Siddiqi” str8-utf 16 “978-0-160-81733-5”

AMQP more compact as descriptive metadata is externalized into schema

AMQP can still be parsed without the external schema present

<type class="composite" name="book" label="example composite type"> <descriptor name="example:book:list" code="0x00000003:0x00000002"/> <field name="title" type="string" mandatory="true" /> <field name="authors" type="string" multiple="true"/> <field name="isbn" type="string"/></type>

Page 36: AMQP 1.0 introduction

AMQP Frames• Frame header (8 byte)

• Overall frame size (ulong)• Data offset (count of 4-byte words, byte)• Frame type indicator (byte)• Channel identifier (ushort)

• Performative• Composite type describing the AMQP

operation• Open, Begin, Attach, Flow, Transfer,

Disposition, Detach, End, Close• Payload

• Performative-dependent payload, immediately following the performative.

• Currently only defined to be used for Transfer’

frame headersize, 4 bytesoffset, 1 bytetype, 1 byte (AMQP 0x00 / SASL 0x01)channel-id (2 byte)

performative (@ size-offset*4)Open, Begin, Attach, Flow,

Transfer, Disposition, Detach, End, Close

payload

AMQP Message (w/ Transfer)

Page 37: AMQP 1.0 introduction

Wire Footprint?• Minimal AMQP transfer

footprint for a message containing the string “Hello!”• Just 35 bytes transfer

overhead. • plus IP, TCP, TLS (~100

byte)

Frame Header ..... 8 bytesTransfer Performative 5b link-handle (1b) 1+4b delivery-id (1b) 1+4b delivery-tag (2+1b) 2+4b message-format 1b (theoretical min 11 bytes) .... 22 bytesAMQP Message amqp-value 3b “Hello!” 2+6b .... 11 bytes ============= 41 bytes

Page 38: AMQP 1.0 introduction

... there’s moreTransactionsAddressing

Management Claims Based Security

... next time ...


Recommended