Atomic Enterprise SOA Patternslogosworld.com/docs/soaindia/soaindia2008/Angeli... · recurring...

1

The

Pag

e 4

Atomic Enterprise SOA Patterns

Elementary Middleware Communication Patterns for a Successful SOA Implementation

Logosworld Technology Research, Luxemburg

nLeague Technologies Alpharetta, Georgia, USA

nLeague Technology Labs, HITEX, Hyderabad, India

Building SOA City SOA Blue Books Series Atomic SOA Patterns Axel Angeli Lynton Grice Srinath Nagabhirava Sreekanth Kanthamneni Varun Reddy

2

II

The

Pag

e 4

The Page 4

F:\BLUEBOOK\Dokumente\mySOA\SOAcity\SOAbook\Chapters\SOApatterns\SOAcity_Atomic_SOA_India_2008.docx Axel Angeli, Page 2/80; 11/22/2008 Last

saved by Axel Angeli © 2008 Logosworld.com All rights reserved.

2

Logosworld.com

1 The Page 4

The internet is a big computing cloud and all software components are merely

players.

The very simple concept of SOA is to cater for an architecture that makes the communication between alien entities as easy as possible.

We all are brought up with what our parents used to call “Good Manners”. Manners are nothing

more than a small set of rigid patterns in behaviour that allow a smooth and gentle communication

between humans in a social society. Similarly, every professional working in the field of networked

communication with middleware and SOA needs to aware that there are a limited number of ever-

recurring patterns when programs communicate with each other. While implementing these

patterns you will receive a high performing, easy to maintain SOA for your enterprise.

This book will be indispensable for the IT professionals charged with the privilege of being allowed

to promote SOA for their enterprise. The patterns herein are derived from real life experiences and

In this book, you will read about:

How to enforce separation of semantic and physical transport layers

Pig-Runs and Crash-test-Dummies for the Test-First paradigm

Most popular filter and pipe patterns

Efficient parallel execution with Fan-Out and Merge Patterns

Emulation of Event Driven Architecture on a classical synchronous middleware

Building a small Content Management System from patterns

Hints to implement patterns on top of SAP Process Infrastructure

There is course material available for self and classroom training including PowerPoint slides and

ready to use sample implementations for SAP PI and IntelliBO.

About the Author

Leading SOA Evangelist Axel Angeli is founder and director of 1984 established

Logosworld.com, a specialist firm to give advice to management in IT strategy.

During his long career, Axel Angeli worked with micro processors as well as with

mainframes and robots. He travelled the world implementing all sort of software

thereof over 20 years as project manager and development mentor in the SAP

arena. Through his writing, speaking and consulting he helps now major enterprises finding the right

and efficient way to achieve a stable and rock-solid SOA.

[email protected]

3

II

The

Pag

e 4

The Page 4



3

Logosworld.com

Table of Contents

1 The Page 4 ............................................................................................................................ 2

2 Introduction to Atomic SOA Patterns .................................................................................... 3

2.1 Mind the 5 Ws............................................................................................................................... 4

2.1.1 Where does the message come from? ......................................................................................... 4

2.1.2 Who is listening to the message? ................................................................................................. 4

2.1.3 What path the message goes? ..................................................................................................... 5

2.1.4 How is the message transported from sender to receiver? ......................................................... 5

2.1.5 Why do we communicate this message? ..................................................................................... 5

2.2 Collaboration and SOA .................................................................................................................. 6

2.2.1 Understanding SOA by learning from other sciences ................................................................... 6

3 Introduction to Patterns ....................................................................................................... 9

3.1 What are communication Patterns ............................................................................................ 10

3.2 Why do we need patterns .......................................................................................................... 11

3.2.1 Patterns reduce complexity ........................................................................................................ 11

3.2.2 Patterns save money .................................................................................................................. 11

3.2.3 Patterns enhance quality ............................................................................................................ 11

3.2.4 Patterns make better programmers ........................................................................................... 11

4 Communication Categories ................................................................................................. 13

4.1 Synchronous Dialog: Request & Reply ........................................................................................ 15

4.1.1 Sub pattern: Request and Acknowledge .................................................................................... 16

4.2 Asynchronous Exclamation: Fire & Forget .................................................................................. 17

4.3 Anonymous communication: Broadcast ..................................................................................... 18

4.3.1 Sub pattern: Trigger .................................................................................................................... 18

4.3.2 Parallizing (Spawn and Collect) ................................................................................................... 19

5 Canonical Communication .................................................................................................. 20

5.1 Communication Layers ............................................................................................................... 21

5.2 Layer stacks ................................................................................................................................. 22

5.2.1 Traditional OSI Seven-Layer ........................................................................................................ 22

5.2.2 SOA Communication Layers ........................................................................................................ 22

5.3 Middleware Layers ..................................................................................................................... 24

5.3.1 Middleware separates data layer and transport layer ............................................................... 24

6 On Envelopes and Payloads and Canonical Messaging ......................................................... 26

6.1 Canonical Formats in Middleware .............................................................................................. 27

6.1.1 Unified transport format ............................................................................................................ 27

6.1.2 Message Components ................................................................................................................ 27

7 Filters and Pipes ................................................................................................................. 29

7.1 Pass Filter (Pipe).......................................................................................................................... 30

7.1.1 Pattern Characteristics ............................................................................................................... 30

7.1.2 Variations and Sub-Patterns ....................................................................................................... 31

4

II

Intr

od

uct

ion

to

Ato

mic

SO

A

Pat

tern

s

Introduction to Atomic SOA Patterns



4

Logosworld.com

2 Introduction to Atomic SOA Patterns

“A designer knows he has achieved perfection not when there is nothing left

to add, but when there is nothing left to take away.” Antoine de Saint

Exupéry, Pilot and Philosopher

Patterns are quite in fashion in computer sciences. Communication in a society follows a certain

number of patterns and they are indeed nothing else than an abstract categorization of the

potentially unlimited individual communication instances according a number of easily

identifiable criteria. They are the atoms of communication. We shall deal with atomic patterns,

with the simplest elements that can be used to build complex compounds in order to model a real

life business process.

Logosworld.com

SOA City

Logosworld.com

Atomic SOA Patterns

Elementary Middleware Communication Patterns for a Succesful SOA

Implementation

There are several approaches to pattern based design.

Unfortunately, the idea of patterns made its way into

computing through the sciences as a more or less

theoretical exercise. This is indeed a little bit of a pity; since

the publications about patterns are either abstract

theoretical models or they are examples of implementation

of certain tools.

5

II

Intr

od

uct

ion

to

Ato

mic

SO

A

Pat

tern

s




5

Logosworld.com

2.1 Mind the 5 Ws We have chosen here a pragmatic approach to pattern

design. We call it “Visual Approach”, since we start every

pattern by designing the communication in a visual sketch.

We wanted to see:

Where does the message come from?

Who is listening to the message?

What path the message goes?

And then we can add the remaining questions:

How is the message transported from sender to receiver?

Why do we communicate this message?

Mind the 5 W! The five questions can be easily

remembered, since they reflect the classical 5 W of rhetoric;

they are the five questions you always need to ask before

you start sketching a plan.

2.1.1 Where does the message come from? We need the answer to this question, since the channels

that we need to open for the communication need to be

prepared correctly. In addition the sender may use a

completely different medium as we need it to process in our

own ecosystem. E.g. the message arrives from an RFID

device or is a video-stream. Those input streams need first

be converted to our internal XML format.

2.1.2 Who is listening to the message? Mind here, that we do not ask “where does the message go

to”. Indeed the process should be completely agnostic with

respect to the receiver unless the receiver media is explicit

part of the desired process. If we say: “convert the data

stream to PDF.” Then it should be of no interest if the

stream is later stored to a file or send via email. If the

process is designed to store data to a file, then the process

may of course code explicit information for the file adapter,

accepting that the process is then private to the file storage

process.

In the context of Event Driven Architecture the principle

that the process should be developed in a way that it does

not depend on a special receiver gets fairly crucial.

Process should be agnostic to the receiver; the listeners are of interest for the configuration process

6

II

Intr

od

uct

ion

to

Ato

mic

SO

A

Pat

tern

s




6

Logosworld.com

2.1.3 What path the message goes? In fact when asking which path the message should go, it

means the process logic. And we call it What Path to keep

you in the visual context, try to visualize the flow instead of

just coding it.

2.1.4 How is the message transported from sender to receiver? When you save a file you have the file content but also the

file name. When you send a message to an email you have

the email text but also the email sender and receiver and

maybe a subject line and information about the encoding of

the text body.

In any case, we send a message and some context

information. Every message lives in a context. If it is juggled

from one port to the next it is often necessary to provide

some of the context information

2.1.5 Why do we communicate this message? Knowing about the reasons why a communication takes

place can be important context information for the

communication framework.

7

II

Intr

od

uct

ion

to

Ato

mic

SO

A

Pat

tern

s




7

Logosworld.com

2.2 Collaboration and SOA

The internet is a big computing cloud and all software components are merely

players.

The very simple concept of SOA is to cater for an architecture that makes the communication between alien entities as easy as possible.

SOA is about making communication between disparate

entities possible and by entities we mean people, lines of

business, computers connected via the internet and other

items that are able to send and/or receive messages. It is

about sharing tasks and duties and offering above average

skills to others to use and refrain from reinventing a wheel

that can be acquired from other parties easily. Sounds very

much like a social city life? It is about sharing tools and

services and about anticipating where one single citizen may

assist others for the better of the whole city.

The essential wording in the initial paragraph is: easily. SOA

is not about introducing something that would not already

exist. SOA is about making best use of what exists already.

In this sense SOA does not merely promote reuse; its inner

heart is about reusing the existing possibilities in a way that

they can act as individual, self-contained components but

collaborate to form an aggregate compound that by itself

appears as new organism which as a whole is better than

the mere sum of its components.

2.2.1 Understanding SOA by learning from other sciences

If our world of IT were as integrated as we expect our

business process to be, this book is simply unnecessary.

Everything - I repeat: everything - what is described in this

book has been repeatedly taught and described in many

other areas of engineering and craftsmanship.

The core paradigm of SOA is communication in the sense of

exchanging messages between senders and receivers that

are housed in different places of the internet and

communicate on an ad hoc basis.

The general terms we use for that are:

Senders and receivers are located in different physical

places and do not necessarily share common access

The path to SOA is to make SOA easy

Disparate components

8

II

Intr

od

uct

ion

to

Ato

mic

SO

A

Pat

tern

s




8

Logosworld.com

technology.

The communication takes place on an ad hoc basis. A

connection between sender and receiver is only established

when it is needed. This differs widely from classical

communication in EAI where you would configure a

communication channel permanently.

Many people talk about SOA as it were a new discipline that

the gods had cast on earth from heaven. Nothing could be

more wrong than that. SOA is nothing more than applying

communication theory and the concepts of social sciences

on the society of components that automatically emotes

when you build a network of software components and

computers.

Communication is ubiquitous as such. Humans

communicate. Animals communicate. Every closed system

with several entities communicates: plants with plants, lakes

with the river. There is communication every time when two

systems exchange information and this is always the case

when two systems interdependent.

SOA is about integration and this means exchanging

messages. There are already two science areas that take

care of message exchange:

Communication Theory

This one mainly takes care of the semantic level of

communication and the impact of communication on sender

and receiver. Communication is not restricted to electronics

but also refers to exchange of information and intelligence

in general, including human communication.

Communications engineering

This science looks on the technical and physical

requirements to guarantee a reliable exchange of messages

between partners.

Everything that comes now to speech in the context of SOA

has been talked about previously in the computer science

classrooms and research labs. In fact every old-fashioned

mainframe is a SOA.

I

Loosely Coupling

SOA learns from traditional communication theory

Communication is ubiquitous and the patterns are the same in IT and real life

SOA patterns are based on classical communication theory

9

II

Intr

od

uct

ion

to

Ato

mic

SO

A

Pat

tern

s




9

Logosworld.com

What is the lesson we learn from this? Quite simple: IT

needs to refrain from adopting an ignorant and arrogant

position in believing that communication and anything that

comes in the context of SOA is a new invention of IT. Neither

are the theories new in IT nor are they a new paradigm for

mankind.

10

II

Intr

od

uct

ion

to

Pat

tern

s

Introduction to Patterns



10

Logosworld.com

3 Introduction to Patterns When we analyze carefully any technical equipment – this

includes software algorithm – we soon realize that they all

follow a small number of abstract design patterns.

So it is with many conversations: After a while of vain argument you realize

that you are not talking of the same think. (André Gide)

11

II

Intr

od

uct

ion

to

Pat

tern

s




11

Logosworld.com

3.1 What are communication Patterns Communication patterns are abstract description of the communication behavior during the

exchange of messages between a sender and a receiver. Due to identification of the underlying

patterns, we gain the necessary tool to reduce any communication to a controlled aggregate made

from simple atomic components.

Let us recall quickly the concept through an example:

You receive an email and print it out on a printer

You request a page in your browser and save the data to a local file

Your ERP system produces a report and emails it off to a colleague

All those above examples share a common abstract pattern: they receive data from one port and

deliver the content to another port. The delivery format of the data is different in all cases, but the

process always the same:

Get the data from one location,

Unpack the content

Pack the content in the target format

Deliver the content to the target

12

II

Intr

od

uct

ion

to

Pat

tern

s




12

Logosworld.com

3.2 Why do we need patterns Patterns serve a strong case: they allow us better understanding the behavior of communication.

They abstract the potentially unlimited number of communication scenarios and give us the

possibility to make use a pre-fabricated solutions that are only decorated and fine-tuned for an

individual purpose.

Traditionally most developers are tempted to code end-to-end. They do this, because they think that

they need to catch all possible real events within their programs. If we imagine a program that wants

to convert a data stream into a PDF file format, we used to have a large number of flows.

Data can be delivered in a file on the file system or via FTP, via HTTP, via email etc. The result can

also be sent out via the same channels. If we had four channel types, we would already end up in

implementing 4 times 4 = 16 sub programs that basically do the same and only differ in the way how

they deal with the data sources. This can become quite tedious if the data sources get more

complex, e.g. a data base system or an ERP system like SAP:

3.2.1 Patterns reduce complexity

When we reduce a program or scenario down to a number of well defined patterns, we immediately

reduce the number of possible communication pairs.

In fact we see only one single pattern in the current example case.

Receive -> Transform -> Send.

The data conversion will be outsourced to special adapters, which deal with the conversion. For

every format we would therefore need only one pair of dapaters: one for inbound and one for

outbound streams. The data processing is then limited to one single algorithm for any of the use

cases.

3.2.2 Patterns save money

The economical win of using patterns can then easily be calculated: instead of 16 individual

programs we now have 8 + 1 = 9 program components.

3.2.3 Patterns enhance quality

In addition every component can be realized and tested independently and reused by future

applications without touching the existing ones. This enhances quality. The smaller a component is,

the easier it can be tested for flaws and the quicker an eventual malfunction can be corrected. The

more other programs use a components, the more stable our component will become over time,

since using a component means additional testing in real situations with the chance to detect hidden

errors, that have not been thought of before and hence nobody built a test for it.

13

II

Intr

od

uct

ion

to

Pat

tern

s




13

Logosworld.com

3.2.4 Patterns make better programmers

Developers that are forced to map their design to a predefined set of abstract patterns immediately

get a better understanding of the task for themselves. And as soon as they have identified a pattern

they can make use of similar solutions done by other programmers before. Ideally the pattern helps

them finding an already existing component they can simply reuse.

14

II

Co

mm

un

icat

ion

Cat

ego

ries

Communication Categories



14

Logosworld.com

4 Communication Categories Communication patterns are generally classified in two main categories, depending on whether the

speaker expects an answer or is satisfied without a dedicated audience.

We all know people who like to talk endlessly without a break and actually never verify if there is

anybody listening. While such a guy can be well a party killer it teaches us – the observer – much

about communication and also about the effects.

There are two possibilities with our speaker. The speaker may expect the audience to listen and has

simply no concept how he can check whether the information he passes has been correctly

understood. In technical terms we say, that the speaker does expect a dialogue but does not expect

acknowledge. If the speaker is the news presenter in TV, however, the whole concept is a one way

street: we call this “fire & forget”.

Logosworld.com

SOA City

Logosworld.com


Saturd

ay, 15

No

vemb

er 20

08 11

Axel A

ngeli -

(c) 20

08

logo

swo

rld.co

m -

Bu

ildin

g the SO

A C

ity

Request & Response

Fire & Forget

Broadcast

15

II

Co

mm

un

icat

ion

Cat

ego

ries




15

Logosworld.com

Logosworld.com

SOA City


Request and Response

HTTP, FTP

Fire and Forget

Maildrop, Triggers

SMTP

Broadcast

MQ, Database, website

Saturd

ay, 15

No

vemb

er 20

08 12

Axel A

ngeli -

(c) 20

08

logo

swo

rld.co

m -

Bu

ildin

g the SO

A C

ity

16

II

Co

mm

un

icat

ion

Cat

ego

ries




16

Logosworld.com

4.1 Synchronous Dialog: Request & Reply

This is the category where the sender sends out a message and waits for a response. The typical use

case for that is the web browser dialog via the HTTP protocol.

You send an URL request from your browser to the web

server and then the browser waits until a response is sent

back. The response should come in any case, whether the

requested URL exists or not. In that case the server needs to

respond with a qualifying error message.

Figure 1: Synchronous Request & Response

Logosworld.com

SOA City

Request and Response

• HTTP, FTPSatu

rday, 1

5 N

ovem

ber 2

00

8 13

Axel A

ngeli -

(c) 20

08

logo

swo

rld.co

m -

Bu

ildin

g the SO

A C

ity

Client Server

Request

Response

The critical thing with synchronous process is always the

exception handling. There are several error situations.

Interestingly enough we have to see that most protocols for

request and reply (like HTTP) are designed for good weather

only. Mostly they define how the communication should

take place but make no explicit statements how certain

error patterns need to be handled.

The classical error situation is losing the communication

during the process, i.e. that the response does not reach the

requester. If the server dies during the communication the

browser would effectively wait forever for its response.

17

II

Co

mm

un

icat

ion

Cat

ego

ries




17

Logosworld.com

There needs to be a process that defines the reaction of the

browser when the response takes too long time.

On the other hand, it is also possible that the server is fine

with its response but the requesting client died meanwhile.

In that case the response would have lost the receiver.

Without precautions it may result in having more and more

dying communication threads.

4.1.1 Sub pattern: Request and Acknowledge A common sub pattern is that the response is not the

requested information but only a statement that the

request has been received. There are several different ways

how this comes real: the acknowledgement is sent and then

the communication is stopped with the idea that the client

will go fetching the response at a later time independent of

the current communication.

The acknowledgement may also lead in another request

that then waits for the result. We use this model to allow a

better error handling. Imagine the submission of payment

credentials to a payment site. Processing the credit card

may take some while and is a risky process. The

acknowledgement of submission is a quick thing and tells

the client that the request is received and under process

now. Even if credit card processing aborts for technical

reasons it is no problem. The processing will take place in

background as part of error recovery, while in a simple

direct communication we would have lost the whole

transaction leading in probably having to tell the customer

to re-submit the whole purchase.

18

II

Co

mm

un

icat

ion

Cat

ego

ries




18

Logosworld.com

4.2 Asynchronous Exclamation: Fire & Forget The fire and forget category combines all patterns where

the sender delegates the message delivery to the next

communication layer. Those patterns will not listen to a

response, and nor will the active communication process

check whether the message arrives at the designated

receiver.

The monitoring and supervision of the message delivery is

the duty of the middleware infrastructure. This is best

compared with a postal mail service; you drop a letter in the

mail drop and rely on the postman and the authorities

behind that the letter is correctly delivered. On the contrary

the postal authorities will also supervise that no illicit

content – think of poisoned letters or package bombs – are

doing harm to anybody.

Figure 2: Fire & Forget

Logosworld.com

SOA City

Fire and Forget

• Sender does not accept a response• Mail drop; Message Queue

• SMTP

• Infrastructure can supervise the delivery

Saturd

ay, 15

No

vemb

er 20

08 14

Axel A

ngeli -

(c) 20

08

logo

swo

rld.co

m -

Bu

ildin

g the SO

A C

ity

SenderServer

Fire &Forget

19

II

Co

mm

un

icat

ion

Cat

ego

ries




19

Logosworld.com

4.3 Anonymous communication: Broadcast The fire and forget category combines all patterns where

the sender delegates the message delivery to the next

communication layer. Those patterns will not listen to a

response, not will the active communication process check

whether the message arrives at the designated receiver.

The monitoring and supervision of the message delivery is

the duty of the middleware infrastructure. This is best

compared with a postal mail service; you drop a letter in the

mail drop and rely on the postman and the authorities

behind that the letter is correctly delivered. On the contrary

the postal authorities will also supervise that no illicit

content – think of poisoned letters or package bombs – are

doing harm to anybody.

Figure 3: Broadcast

Logosworld.com

SOA City

Broadcast

• One sender; unknown number of receivers• Website

• Message Queue, Database

• Event Triggers

• Sender Does not know who is listeningSatu

rday, 1

5 N

ovem

ber 2

00

8 15

Axel A

ngeli -

(c) 20

08

logo

swo

rld.co

m -

Bu

ildin

g the SO

A C

ity

Sender

Publish

4.3.1 Sub pattern: Trigger

Triggers are a common representation of broad-casting. The

sender raises an event and this event is made public to a

variety of candidates. In the simplest case the event is made

20

II

Co

mm

un

icat

ion

Cat

ego

ries




20

Logosworld.com

known to everybody and any matching candidate will be

able to pick up the event for processing.

4.3.2 Parallizing (Spawn and Collect)

In the category broadcast you will also find the task of

parallizing a process flow.

21

II

Can

on

ical

Co

mm

un

icat

ion

Canonical Communication



21

Logosworld.com

5 Canonical Communication

Exchanging content from one port to another allows for infinite variations. In order to reduce the

complexity through a combinatory explosion of internal exchange formats, it is a wise decision to

limit you to using a very small number of internal canonical exchange formats.

Think of sending over a simple text message like „The quick

brown fox“. You may decide exchanging just the plain

format or you pack the message in an envelope, allowing

you to add additional context information like the legacy,

the encoding and the desired receiver. As an example you

may wrap this now in an XML package:

<message> <head></head> <body>The quick brown fox</body> </message>

In practice it is a necessity that the canonical exchange format that you decide to use as your

standard is comprehensive. Inflation of exchange format is normally not the bad will or negligence of

the developers but simple a case of need, when the suggested norm is incomplete and therefore

useless for a specific use case.

How would a good canonical format look like? Sending raw messages is certainly not a good idea,

since it simply does not allow immediate sending additional information along with the message. As

a consequence we need a more complex structure that embeds the message along with a number of

additional information. The example XML envelope shown above is a starting point. Since you may

put whatever you want in the <head> section, the format is already useful in many cases. However,

many a middleware – e.g. SAP Process Infrastructure is not very tactile when it comes to use XML

tags that have not been previously defined. It is therefore a good idea to add optional but well

defined parameters to the header section of the envelope.

Since we have an aversion in re-inventing the wheel, we have look a bit left and right and decided to

base the definition of our format on the grounds of the Java Messaging Service API (JMS). Of course

we liked to make it slightly better but at least we kept the information level in sync with the popular

JMS. The format is shown in the attachment of this document.

22

II

Can

on

ical

Co

mm

un

icat

ion




22

Logosworld.com

5.1 Communication Layers

23

II

Can

on

ical

Co

mm

un

icat

ion




23

Logosworld.com

5.2 Layer stacks

When we speak of communication we can find that it is handy to separate the individual logical

stages according their purpose.

There is of course the message content; if the message is

exchanged it needs to be wrapped in a kind of envelope that

allows passing information to the communication agents

and participants. Once we have defined an envelope we can

exchange the information in the very same canonical format

across all physical transportation paths.

5.2.1 Traditional OSI Seven-Layer The OSI Seven-Layer-Model is the common structuring of

networking communication. The design was mainly

developed for the physical network design to make a clear

point.

Figure 4: The OSI Seven-Layer Model

Layer Sample Description

1. Application Content That is the data format that is finally presented to the application; mind that in case of a browser the application is the “person” who looks on the screen.

2. Presentation HTML, XML, JSON

Here a universal data format is transformed into a format suitable for presentation

3. Session Envelopes HTTP, XMLRPC, FTP, RPC

This protocol layer is completely agnostic towards the semantic content of the data

4. Logical Delivery TCP, UDP This layer is responsible to collect and re-compile all the low-level data packages and guarantee that data is delivered as a complete entity.

5. Routing IP, IPX This layer is responsible to bring the data package from one endpoint to another

6. Data Linkage Plain, SSL, Ethernet, Tokenring

This is the layer where the network needs to authenticate itself and it is assured that the right physical peers communicate. In addition data compression and redundancy checks take place.

7. Physical . WLAN, 3G This layer is the real hardware, where the bits and bytes flow;

24

II

Can

on

ical

Co

mm

un

icat

ion




24

Logosworld.com

5.2.2 SOA Communication Layers For communication in SOA we slightly remodelled the OSI

model to match different context and terminology.

Figure 5: SOA Application Layer Model

Logosworld.com

SOA City

SOA Application Layer Model (non-OSI)

• Canonical Content representationContent

• HTML, XML, JSON, ASCIIEnvelope

• HTTP, FTP, WebDAV, SteganographieTransport

• TCP, UDP, IPXDelivery

• IPData Linkage

• Ethernet, Token-RingNetwork

• WLAN, Cable, Sound, File-SystemPhysical

Saturd

ay, 15

No

vemb

er 20

08 7

Axel A

ngeli -

(c) 20

08

logo

swo

rld.co

m -

Bu

ildin

g the SO

A C

ity

25

II

Can

on

ical

Co

mm

un

icat

ion




25

Logosworld.com

5.3 Middleware Layers

Middleware makes the distinction between data layers and transport layers. The data layer is the

semantic content of the messages, while the transport layer caters for packing the data into

universal format that allows shuffling the data from one process step to another.

5.3.1 Middleware separates data layer and transport layer Let us do a fine example. We want to design a process that

takes data from an HTTP source and delivers the data

onward to a SOAP target. Designing this process will end up

in designing an abstract process that does map a payload

from input to output. This process will be designed

completely on the logical layer. For the logical process HTTP

or SOAP are not in the picture for now.

So how do we make the link to HTTP or SOAP now? This is

handled in another orbit: the transport layer. All data will be

packed in an envelope that matches the requirements to

the next receiver of the message. We are dealing with two

layers:

the transport layer

the business flow layer

We first design the abstract flow without taking care of the adapters

26

II

Can

on

ical

Co

mm

un

icat

ion




26

Logosworld.com

Figure 6: Data Layer strictly separate from transport layer

Logosworld.com

SOA City

Data Layer versus Transport layer

In Process Out

Saturd

ay, 15

No

vemb

er 20

08 23

Axel A

ngeli -

(c) 20

08

logo

swo

rld.co

m -

Bu

ildin

g the SO

A C

ity

HTTP XML SOAP

Data layer

Transport Layer

The transport layer takes care of transforming the

envelopes from inbound format -> to intermediate format -

> to outbound format, in your example and in the case of

SAP PI: HTML -> XML.pi.sap -> SOAP

27

II

On

En

velo

pes

an

d P

aylo

ads

and

C

ano

nic

al M

essa

gin

g

On Envelopes and Payloads and Canonical Messaging



27

Logosworld.com

6 On Envelopes and Payloads and Canonical Messaging

The achievement of middleware communication is to define certain rules how messages are

exchanged. In a more mnemonic way you can easily compare middleware with defining a common

language and communication structure.

In classical ERP communication we have an anarchistic way

to exchange information. IF you wanted to export an invoice

then every system has an own way to format the invoice;

fields are named different, columns are arranged in strange

ways; others cannot even export an invoice in simple format

and deliver print formatting along with it all. Some system

write ASCII file, others a binary format. Some files have the

invoice plus the receiver information all in once; others

separate the payload and the extra information in two files.

28

II

On

En

velo

pes

an

d P

aylo

ads

and

C

ano

nic

al M

essa

gin

g




28

Logosworld.com

6.1 Canonical Formats in Middleware

Middleware does some cleanup of the formatting anarchy by reducing the possible communication

means to a small number of canonical standards.

6.1.1 Unified transport format First of all it defines an internal transport format which is

used uniquely to exchange data within the steps of the

middleware. It has been become a habit to use XML for that

purpose, since XML is somehow human readable, fairly

popular and allows a simple way to arrange data

hierarchically.

In order to allow using the internal format all incoming data

is first converted in an inbound adapter from the external

format into the internal format. It is the whole rule in

middleware that an adapter must never do any

manipulation to the data payload and never lose any

information delivered with the envelope of the incoming

data.

We call the rule holy, since it is unfortunately the bad habit

of developers to scram all code steps in one single program,

and the adapter is always pointed out as the most

convenient place for it. E.g. read the file, change the

Unicode characters to ASCII, remove all line that appear to

be unnecessary for the moment and toss the data in a

database. If the later steps need one of the deleted lines,

the programming needs to return to the adapter and make

changes there.

6.1.2 Message Components Messages are structured in three shelled components: The

payload, the envelope and the transport protocol.

Understanding these three concepts is the essential key to

understand the pattern nature in messaging.

6.1.2.1 Payload The payload is the pure information. In an email it is the

email text only; sender or receiver address or the subject

line are not part of the payload, since they are additional

information like a packaging slip.

Middleware defines a unique internal transport format

Adapters convert inbound format in to internal format without changing the payload

The payload is the information that the message is subject to deliver.

29

II

On

En

velo

pes

an

d P

aylo

ads

and

C

ano

nic

al M

essa

gin

g




29

Logosworld.com

6.1.2.2 Envelope The envelope is a wrapper around the message to allow for

adding pragmas and directives to the message handling

agent how to treat the message.

Such directives may be the receiver address of the message

(e.g. email address), a subject line to describe the message

content for monitoring tolls or what efforts the middleware

should make to guarantee delivery and whom to notify at

certain incidents.

6.1.2.3 Transport Protocol The transport protocol is the syntactical media used to

transport and store the message during its expected life-

time.

There are two terms that appear in the middleware context

all along: Envelope and payload.

The envelope is a wrapper around the message to allow for adding pragmas and directives

30

II

Filt

ers

and

Pip

es

Filters and Pipes



30

Logosworld.com

7 Filters and Pipes

Filters and pipes are the elementary communication patterns. Their purpose is simply to receive a

message from one place, do something with the message and hand over the result to the next

communication partner.

The brain is a filter. You receive sound via your ear, get the

sound transformed in electrical signals; the signals are piped

to your brain where the message modulated on the signal is

processed and filtered; finally the resulting electrical pattern

is converted back to sound via our vocal chord.

There is no difference in meaning between pipe and filter;

we use both terms depending on whether we want to

emphasize their role as transport media or as data

manipulator. Traditionally DOS preferred the expression

filter while the UNIX world favoured speaking of pipe.

A brain is also a filter

Pipe and filter mean the same

31

II

Filt

ers

an

d P

ipes

Filters and Pipes



31

Logosworld.com

7.1 Pass Filter (Pipe)

A filter in its general meaning is any algorithm that can investigate a message and decide upon an action triggered based on the content. The filter may manipulate the message or its envelope itself or initiate some side activities without tampering with the message. A synonym for filter is the name pipe.

Figure 7: A simple pass filter

Logosworld.com

SOA City

Atom: Pass Filter

Message is handled and filtered by an algorithm during pass through

Sender

Adapte

r

Adapte

r

FilterReceiver

7.1.1 Pattern Characteristics A pass filter is a process that evaluates and transforms a

message according a defined algorithm. Although this

pattern is more than a filter in the narrow meaning, the

name is traditionally used for any simple algorithm that

pipes data from an inbound stream to an outbound stream.

The name stems from the common usage of pipes to block,

reject, redirect or replace the data stream at discretion.

In brief: A filter receives a message, does something with the data and sends the message further on.

The name pass filter is taken from electronics where such

components are used to filter the extreme trembling heights

or basses of a sound stream.

32

II

Filt

ers

and

Pip

es

Filters and Pipes



32

Logosworld.com

A very special form of a pass filter is the pass-through filter,

where the data is forwarded to another party without

making any manipulation to the content.

Inbound-Adapter receives the data

Filter is applied

Outbound-Adapter issues the data to a target

The filter is an essential component of any BPM process.

The simplest form of a filter is drive-through pattern, which

hands over a data stream from one port to another. A pipe

connects two ports directly with each other and is hence the

underlying abstract for any synchronous process.

7.1.2 Variations and Sub-Patterns

There are several sub-patterns of the filter pattern. The sub-patterns differ in their usage, not from the way how they work. They all have in common that data is washed in the inner process of the filter.

7.1.3 Data Mapping: Changing the content In this pattern the data content of the message is

manipulated. The classical form here is a simple data

mapping or a transformation algorithm where parts of the

message are transformed into another format. A data

mapping matches input data to a predefined destination

structure.

Mapping is one of the most critical application patterns in

middleware design and the main cost factor in the whole

SOA arena. The better the mapping engine, the better the

more efficient the whole SOA process will turn out to be.

We shall therefore dedicate a special chapter to the

mapping subject.

33

II

Filt

ers

and

Pip

es

Filters and Pipes



33

Logosworld.com

Figure 8: Data mapping as a filter variation

Logosworld.com

SOA City

Pass Filter Variation: Mapping

Message payload is changed by the filter

Sender

Adapte

r

Adapte

r

ProcessReceiver

Fox

Wolf

Hound

Hyane

Tiger

Dogs

Cats

7.1.4 Data Transformation: Changing the envelope Data transformation means changing the structure how the

data is represented; contrary to a mapping that would

change the data content embedded in an envelope. The

classical example is changing a plain text data table where

fields are delimited with commas into an XML format.

34

II

Filt

ers

and

Pip

es

Filters and Pipes



34

Logosworld.com

Figure 9: Data transformations change envelope

During a transformation the content of the message payload

should remain unchanged. Changing content payload and

transforming the envelope format means simply mixing

operations on two communication layer in a single step. In a

consequence we will end up in difficulties during

maintenance and add another potential error leavel in form

of possible conflicts caused by the mixing and matching of

the layers.

35

II

Filt

ers

and

Pip

es

Filters and Pipes



35

Logosworld.com

7.2 Null-Filter (A-0-B) (Drive Through, Synchronous Forwarding)

A special case of a pass filter is a drive-through pattern, often known as Null Filter. The purpose for

such a filter is forwarding a message between endpoints without changing them. Such a pattern is

a basic necessity as a bridge between different media.

Figure 10: Drive-Through Pattern

Logosworld.com

SOA City

Null Filter

A special case of a pass filter that would not change the message‘s payload Usages: Protocol converter

Bridge between media

Decouple sender and receiver to avoid impact

Adapte

r

Adapte

r

Null-Filter

36

II

Filt

ers

and

Pip

es

Filters and Pipes



36

Logosworld.com

7.2.1 Pattern Characteristics

This is the most elementary pattern pattern that you find in

a middleware. The works are done mainly by the adapter.

Inbound-Adapter receives the data

Null-Filter is activated

Outbound-Adapter issues the data to a target

The drive-through is the underlying pattern for any

synchronous process. The message itself is not changed but

the envelope and transport protocol normally changes in

between.

7.2.2 Use cases

The drive-through pattern is usually used to exchange data

between different media. Like shuffling data from a file to a

database or receive a file via FTP and send it over to a

programming API like SAP RFC.

The pattern is also a critical component in any testing

scenario to verify if the data flows smoothly through a

process chain. It is used to test the connectors between two

components and to test the communication between a

newly defined endpoint against a well known existing

receiver or sender.

7.2.3 Null Filter Examples

All the examples share the same integration scenario, which

in fact does nothing then let the message pass through. All

variations take a data stream from a source location and

hand over the message to a target location.

The work is done in the adapters that may need additional

meta-information in order to do its job. For example a file

adapter needs to know the name and path of the file to

fetch. This information may or may not be part of the

message. How the meta-information is retrieved is normally

handled by a dedicated service-call.

Here are some examples of basic null pass filters:

All the variations can be reduced to a simple case.

Protocol conversion

Basic testing

37

II

Filt

ers

and

Pip

es

Filters and Pipes



37

Logosworld.com

7.2.4 Null-Pass-Filter (Pipe)

A filter that receives data and forwards the payload without

making any changes is called a Null-Pass-Filter or Simple

Pipe.

On a first thought a filter seems to be quite useless that

does nothing than pushing data onwards without changing

them. However, that filter turns out to be one of the most

important patterns in communication.

There are indeed many use cases for a null pass filter, even

if data I not changed within the pipe:

7.2.4.1 Probing and logging

A very fine use case for null pass filters is the ability to

intercept the data and write it to a log file. Usually the

sender and receiver protocol are identical in these cases.

7.2.4.2 Place holder

An even more simple variation of the logging feature is acting as a place holder for a filter that needs

to be inserted or implemented later. The place holder gives the left and right peers a proper

connection point. When at a later time a more complex process needs to be inserted in this place, it

will be completely transparent to the neighboring ports.

7.2.4.3 Firewall: Intercepting malicious or unwanted data

A firewall is also nothing than a null pass filter. Its only purpose is to intercept and block the routing

of malicious data streams. Since such a filter is an aggregate on its own it will allow implementing

additional actions like cleaning malicious streams – imagine removing attachments in emails -,

alerting an administrator or consulting another virus checker before ultimately rejecting the stream.

7.2.4.4 Content Based Routing

One of the finest duties of a null pass filter is the

determination of the receiver of a message based on the

message content. Imagine receiving a hospital patient file

for pregnancy. Normally the file would be forwarded to the

gynecologist; however, if the person is a juvenile it might be

worth consulting a pediatric specialist first and hence the

file needs to be forwarded to the pediatric department

rather than the gynecology.

38

II

Filt

ers

and

Pip

es

Filters and Pipes



38

Logosworld.com

Figure 11: Content based routing

Logosworld.com

SOA City

Pa

tie

nt

Fil

e

Pass Filter: Content based routing

Receiver depends on message content

Sender

Ad

apte

r

Ad

apte

r

ProcessReceiver

Age=45

Female

Age=12

Male

Pregnant

Gynecologist

Psychiatrist

Pediatric

12yr old girl

35yr old lady

A man

7.2.4.5 Protocol Conversion (Adapter to adapter)

Although the pipe does not change the messages’ payload,

it may be connected to different data streams protocols. For

instance, data may be:

Received via HTTP and stored to a file

Read from a file and forwarded to SAP-RFC

Received as HTTP and forwarded as FTP

Received internally as HTTP and forwarded to an external HTTP address

In any of these example cases, the data is taken from one

data source and delivered to a target data source that uses a

different protocol. In a SOA environment such a null pass

filter proxy frees the developer of applications from being

obliged to care of different protocols. A programmer will

only write to a stream, while the pipe will take care of

finding the right algorithm to right the output stream.

7.2.5 Protocol Conversion Examples

The standard task of a null filter is building the connector

between two data streams of different type, mostly with the

39

II

Filt

ers

and

Pip

es

Filters and Pipes



39

Logosworld.com

purpose to adopt the instream protocol with the protocol

requested by the following processing step.

7.2.5.1 File-to-File

A file is picked from one location and sent over to another

file location. The in-between process does nothing with the

data. The only characteristic that is specific to the scenario

here is the necessity to control the adapters. In this case

both the inbound and the outbound adapter need a

filename as meta-instructions.

7.2.5.2 HTTP-to-File

The version would receive data in the body of a web request

with the instruction to store it to a file. The difficulty here is

to determine the name of the file to store the data in. It

might be encrypted in the data,and sent in the envelope as

meta-instruction.

Target filename can be derived from the file content

Target filename is sent in the envelope

Data is stored by an automatically generated filename

File to HTTP (Fetch the file via Web Service)

The process fetches the file and returns the content to an HTTP. This variation is special in the sense,

that the requester and receiver of the information are the same request as it is the normal case with

a web service call. Here again, the difficulty is to retrieve the name of the file that needs to be

fetched from the incoming HTTP request. The name might be part of the incoming Http request or it

might be derived by applying some logic.

The actual file retrieval or file storage parts are identical for any of these variations. However,

determination of the additional meta information may be the tricky part.

7.2.5.3 File-to-HTTP

Getting file content by filename is one of the most simple

practices. But in addition to it, SOA middleware provides

the most precious services of fetching the content of a file

from some place in the file system and return the data in

universal HTTP format.

The example will fetch a message from a file and send the

message over to an HTTP URL endpoint. There are two

versions to test this scenario. The adapter might poll a

special location for new files and trigger the process when it

finds new data or an external program may kick of the

40

II

Filt

ers

and

Pip

es

Filters and Pipes



40

Logosworld.com

process, which then will fetch the file and forward it to

HTTP.

Poll file location

If file is found, kick of process

Process will activate the file adapter to read the file

The file content is handed over to the outbound HTTP adapter

The HTTP adapter will then call an URI with the message in the body

7.2.5.4 SAP-RFC to HTTP

This scenario will allow SAP to issue an RFC call whose

parameters are then transformed into a format that can be

sent over via HTTP.

7.2.6 Use cases

Protocol conversion

The drive-through pattern is usually used to exchange data

between different media. Like shuffling data from a file to a

database or receive a file via FTP and send it over to a

programming API like SAP RFC.

Data mapping and conversion

In a variation, data conversion and data mapping is added to

the process. In this case the messages are transformed

between reception and issue.

Pass Probe

That is an electronic wedge that allows redirecting or simply

wire-tapping the communication between two process

steps.

Basic testing

The pattern is also a critical component in any testing

scenario to verify if the data flows smoothly through a

process chain. It is used to test the connectors between two

components and to intercept (probe) messages within a

flow without changing the flow itself.

7.2.7 Example

7.2.7.1 Wire-Tap a Message Flow

The example will be plugged in between two process steps

and log every message that flows through it a log file.

41

II

Filt

ers

and

Pip

es

Filters and Pipes



41

Logosworld.com

7.2.8 Example: Pass Filter in XI The simple theory of a pass filter is not necessarily so simple

enough to understand in a real-world middleware.

42

II

Filt

ers

and

Pip

es

Filters and Pipes



42

Logosworld.com

7.3 Proxy Pattern Did you ever try applying for a visa for a country with strict

visa rules? Then you know that it can be a lengthy and time-

consuming process. You will have to queue in the embassy’s

visa counter, turn in your application and then wait, wait,

wait. But there are companies who have specialized in

queuing up for you. They take your application and go to the

embassy and when they got the visa they will return the

documents to you. There are several advantages. In the

meantime you can do another job. But in addition the visa

service is much more familiar with the whole processing. It

may check your documents for completeness and formal

mistakes before they even report in the embassy. Such a

service that does a certain task on your behalf is called a

proxy service; we apply such a proxy service as an atomic

standard pattern to our SOA scenarios.

Figure 12: Using a proxy to encapsulate asynchronous processing

Logosworld.com

SOA City

Atom: Proxy - Asynchronous Capsule

Services are activated asynchronously only• Proxy provides emulation of synchronous processing

Adapte

r

Adapte

r

Requester Collector

Service

43

II

Filt

ers

and

Pip

es

Filters and Pipes



43

Logosworld.com

7.4 Request&Reply Pattern A request & reply pattern is the basic form of synchronous communication. Initially it was invented

to allow retrieval of smaller bits of information from a remote data location like the content of a web

page via the URL..

Figure 13: Request&Reply Pattern

Logosworld.com

SOA City

Atom: Request Reply Pattern

Request-reply patterns are filter where sender and receiver are the same entity

Adapte

r

Service

The initial idea of a request & reply patterns is having asynchronous communication between the

requester and the sender. However a synchronous communication is not a desired architecture

when you communicate within a universe where you can neither predict the amount of

simultaneous requests via the internet nor be sure who the recipient is and if it will then wait for a

proper response.

Most Request&Reply implementations are in fact asynchronous patterns that only emulate a

synchronous behavior. This is done by keeping the communication channel alive while the actual

process is submitted in a parallel task.

44

II

Filt

ers

and

Pip

es

Filters and Pipes



44

Logosworld.com

7.5 Load Balancing

Figure 14: Messages are queued and dispatched to next free process

Logosworld.com

SOA City

Dispatcher

Atom: Load Balancing

Receive a message and assign to the next available resource to process from

Ad

apte

r

Ad

apte

rA

dap

ter

Ad

apte

r

Process

Process

Process

Ad

apte

r

SenderMQ-

InboxProcess Receiver

45

II

Filt

ers

and

Pip

es

Filters and Pipes



45

Logosworld.com

46

II

Co

mm

un

icat

ion

Co

ntr

ol a

nd

Saf

ety

Communication Control and Safety



46

Logosworld.com

8 Communication Control and Safety

47

II

Co

mm

un

icat

ion

Co

ntr

ol a

nd

Saf

ety




47

Logosworld.com

8.1 Throttling Throttling plays an important role in messaging. A high

performance middleware can simply overflow a receiver

when it sends out messages faster than it can be processed

at its endpoint. While the middleware vendors always

proudly announce their new world records in the number of

messages processed per second, the real problems lies in

throttling the throughput.

8.1.1 Throttling Strategies There are several strategies for throttling messages.

Figure 15: Messages are usually fetched but may delegate fetching to a dispatcher

Logosworld.com

SOA City

Passive Dispatching

Receiver peeks messages

Active Dispatching

Messages are forwarded

Active and Passive Throttling

Messages are normally fetched by a free handler If processor is unable to poll, MQ may forward messages

QueueJanitor

MQ

Process

Process

Process

QueueJanitor

MQ

Process

Process

Process

The decision of choosing the most appropriate strategy one

depends on the circumstances and the capability of the

communication parties. In any case the middleware needs

to be capable to allow choosing between the strategies via

configuration.

It is self-understood that it should be possible to switch the

choices on the fly to get them effective immediately without

interrupting communication or allowing side-effects on

Middleware needs to cater for all strategies

Switching strategy takes place at configuration time

48

II

Co

mm

un

icat

ion

Co

ntr

ol a

nd

Saf

ety




48

Logosworld.com

other messaging channels.

We make a difference between

active throttling and

passive throttling.-

Active throttling means that the middleware reduces the

speed with which it sends out the messages according a

given parameter or by synchronizing the sending with a

heartbeat signal given by the receiver.

8.1.1.1 Passive Load Throttling via Message Queue Passive throttling will simply deliver the messages to a local

message queue and allow the receiver picking the right

message.

Figure 16: Messages are queued and then dispatched to free processes

Logosworld.com

SOA City

Atom: Load Throttling

Messages are stored in an MQ and then dispatched to processes that signal availability

Ad

apte

r

QueueJanitor

Ad

apte

rA

dap

ter

MQ

Process

Process

Process

Ad

apte

r

SenderMQ-

InboxProcess Receiver

Active Throttling

Passive Throttling

49

II

Co

mm

un

icat

ion

Co

ntr

ol a

nd

Saf

ety




49

Logosworld.com

8.1.2 Use cases

Throttling is used always when we know that the receiving

endpoint may not process messages at lightning speed. In

such a case, we add a throttle simply with the intent to send

forward messages at the heartbeat of the receiving party.

Allow messages sent at heartbeat of receiver

50

II

Co

mm

un

icat

ion

Co

ntr

ol a

nd

Saf

ety




50

Logosworld.com

8.2 Galvanic Decoupling A common cause of error in communication arises from having a direct physical connection between

a sender and a receiver. A galvanic decoupling frees the sender from the necessity to provide

precautions for unexpected communication errors, while the receiver is always talking to a defined

well-behaving sender instrument.

Figure 17: Processes can be galvanically decoupled through a MQ

Logosworld.com

SOA City

Atom: Galvanic Decoupling (Q&Forward)

The process will deploy the response into a message queue for next process to pick up

MQ Process

Process

MQ

SenderMQ-

InboxProcess

MQ-Outbox

NextProcess

51

II

Co

mm

un

icat

ion

Co

ntr

ol a

nd

Saf

ety




51

Logosworld.com

8.2.1 Use cases

Galvanic decoupling avoids the effect that a failed

communication blocks the sender channel.

The basic cause is normally triggered by the fact that either

sending or receiving data source is not yet ready for the

communication. An example may be that the network

connection to the receiving endpoint is temporarily

unavailable. Normally the sender needs to have treatment

for this error. Imagine that the receiver requires a re-

authentication when the network is up again; in such a

situation it happens that the sender will keep sending since

it cannot verify the state, while the receiver denies

accepting the data. This results then in a dead-lock situation,

since the sender will never end sending and the receiver will

never receive anything..

Avoid blocking a communication channel

52

II

On

e to

Man

y C

om

mu

nic

atio

n

One to Many Communication



52

Logosworld.com

9 One to Many Communication

53

II

On

e to

Man

y C

om

mu

nic

atio

n




53

Logosworld.com

9.1 Fanout Alternate names: Sprints; Parallization; Spawn

Figure 18: Fanning out messages to several subscribed processes

Logosworld.com

SOA City

Atom: Fanout

A fan-out distributes a message to list of registered subscribers

Ad

apte

r

Ad

apte

r

ProcessSubscription

Handler

Ad

apte

rA

dap

ter

Sender ProcessDetermineSubscriber

Fan-OutReceivers

Receiver

54

II

On

e to

Man

y C

om

mu

nic

atio

n




54

Logosworld.com

9.1.1 Use cases

The classical use case of spawning or fanning out messages

is a newsletter. Whenever a message arrives it shall be

distributed to a controlled list of previously defined

subscribers.

One of the challenging tasks in modern computing is the art

of parallizing certain tasks. The Fanout pattern is used to tell

several processors to processes the same message.

Why on earth would we want to get the same message

processed by several tasks? Well, one very common use is

simulation as it is used in engineering or in sales planning.

In engineering, we often define different parameterization

of the same model. In order to see how the variations

behave we would send the same message to different

servers that are accordingly configured. We usually call this

type of comparison a benchmark.

A very clever use case of a Fanout parallization is plying the

idea of simulation to several SAP QA systems. For that

purpose we would set up several identical instances of a SAP

system, each of them with some controlled changes in

configuration. Then we create test samples that are fanned

out to the individual instances and the results are

compared.

Another very cool usage is simulating the effect of changes

in a marketing strategy to the sales and wins prognoses. For

that purpose we define several scenarios with our

expectations how price, customer behaviour and global

economical climate interact. Then we send out millions of

messages that reflect a real life behavior of people

requesting the services and watch how the concurring

systems behave.

We shall deal with this special variation of the Fanout in another chapter.

9.1.2 Enhancements: Serialization

There are situations where you have one computer that

hosts several of the destination programs. In such a case,

fanning out would not necessarily be clever. However, from

a logical point of view this is not the problem of the sender

of the message to care for the performance problem.

Mailing Lists

Parallization

Simulation

Benchmarks

Test SAP Customizing

Sales simulation

Spawning may overload the target

55

II

On

e to

Man

y C

om

mu

nic

atio

n




55

Logosworld.com

For these cases we may advice the Fanout algorithm to

submit the parallel messages in small hives or one after the

other. This will delegate the task of load balancing to the

middleware and allow a parametrizing of the behavior.

Such a strategy has much influence in the overall

maintenance strategy. Due to delegating the execution plan

to the middleware it can be uniformly monitored in a

central place, even if the managed subsystems run

completely different software.

Let middleware orchestrate load balancing

Allows central monitoring and strategy instatiation

56

II

On

e to

Man

y C

om

mu

nic

atio

n




56

Logosworld.com

9.2 Fanout and Resynchronize (Spawn and Collect)

Fan-out and recollect is a synonym for parallizing tasks and wait for the set of tasks to end.

Figure 19: The ESB will synchronize the responses of the fanned out processes

Logosworld.com

SOA City

Atom: Fanout and Collect

This pattern fans out the messages and then waits for all tasks to terminate

Ad

apte

rDistributor

Ad

apte

rA

dap

ter

SynchronizerA

dap

ter

Ad

apte

rA

dap

ter

SenderDistri-bute

Fan-Out Re-Sync Receiver

57

II

On

e to

Man

y C

om

mu

nic

atio

n




57

Logosworld.com

9.2.1 Use cases

The classical use case of spawning and resynchronizing is

when parallizing works that indeed can run and execute

independently. Imagine that you need information from

several web services.

9.2.2 Example

9.2.2.1 Calculate independent sub sections

An example may be a sales order where you need to check

both for availability and pricing.Instead of executing those

tasks sequentially you may as well submit the works in

parallel tasks and wait for all of the tasks to finish. Once all

results are collected the next step of the work flow can

execute.

Parallel execution of tasks

Check availability and pricing in parallel

58

II

On

e to

Man

y C

om

mu

nic

atio

n




58

Logosworld.com

9.3 Campaigning

“Campaigning” makes a message public for a certain period of time. This allows posting

information for a wider range of potential receivers without being obliged to inform them

explicitly and expressively.

Figure 20: Messages are made persistent for certain period

Logosworld.com

SOA City

Atom: Campaigning

Message will be made public in a queue• A watchdog will remove the queue after expiration

QueueJanitor

MQ

Process Process

WatchDog

Sender MQ-Inbox Published Withdraw

Process

59

II

On

e to

Man

y C

om

mu

nic

atio

n




59

Logosworld.com


9.3.2 Use cases

A common use of this pattern is displaying information to

the ESB that has a natural expiration date like momentary

stock exchange or currency conversion rates. The message

will be stored in the MQ and removed when the requested

retention period has elapsed.

A very common use case is displaying a request to the public

that cannot be fulfilled by a single task but requires a shared

effort. Think of a plant production order to produce a high

amount of some special hardware, let us say 10 millions

screws. There may be ten production lines that are able to

manufacture them but such a machine will not be able to

issue more than 1000 screws at a time. In such a case, the

message will be the production order and every time a

machine has free capacity, it will read the message and

report the quantity it is going to produce.

Every time when the campaigned message or a related one

is accessed, the ESB will consult a service that decides if the

message should expire or persist.

9.3.2.1 Currency Data

There are two services. One service will regularly post

messages that contain the topical currency exchange rate of

a currency and the validity period of the information.

Another service will display the current exchange rate for a

requested currency.

9.3.2.2 Production Order

This example will work with four services.

Deploy a production order to the message queue

Read the production order from the queue

Deliver production output quantity to the queue

A monitor service that is called every time when an involved message is

accessed

The message holding the production order will be held in

the queue until the monitor service decides to remove the

message.

Data with retention Period, like Exchange Rates

Conditional Events

60

II

On

e to

Man

y C

om

mu

nic

atio

n




60

Logosworld.com

9.3.3 Example

61

II

On

e to

Man

y C

om

mu

nic

atio

n




61

Logosworld.com

9.4 Sort & Merge

Sorting and Merging is one of the master pieces of any programming environment. It is yet

surprising how this area is always neglected in professional development frameworks. Hardly does

any middleware inherently take care of bringing a collection of messages in a correct order or

how several collections are combined into one serialized processing step.

Figure 21: Merging Message Queues

Logosworld.com

SOA City

Atom: Sorting & Merging Queues

Messages need to be sorted

Queues need to be merged I to one queue

QueueMerger

MQ

ProcessA

dap

ter

SenderMQ-

InboxesMerger Process Receiver

MQ

MQ

9.4.1 Example Q from our R&D department made an interesting new improvement for our production line. In order

to better track the goods moving through the exit gate he installed nice RFID reader at the gates that

try registering every commodity moving through the gate. Unfortunately RFID technology is by far

not that perfect as their promoters want to make us believe. Therefore Q had to install in addition a

more reliable light barrier, which is far more reliable in recognizing that something crossed the gate

but unfortunately is unable to tell the identity of the traversing good. So the next fix was adding two

IP cameras that record the movements at the gate.

We now ended up in having three data sources. All the

three data sources are happy to deliver their messages in

small files to a dedicated directory. The RFDID reader reads

62

II

On

e to

Man

y C

om

mu

nic

atio

n




62

Logosworld.com

every read tag to a file in the RFID directory; the light barrier

makes a mark in another directory and the snapshots taken

are also streamed to another directory.

Let us do an analysis of what we have so far: Effectively we

have three file based message queues. Every message has a

unique message DI (the filename) and a unique timestamp

(the file date) in addition to the message body stored in the

file content.

We now ended up in having three data sources. All the

three data sources are happy to deliver their messages in

small files to a dedicated directory. The RFDID reader reads

every read tag to a file in the RFID directory; the light barrier

makes a mark in another directory and the snapshots taken

are also streamed to a private directory.

Let us do an analysis of what we have so far: Effectively we

have three file based message queues. Every message has a

unique message DI (the filename) and a unique timestamp

(the file date) in addition to the message body stored in the

file content. Unfortunately the three queues are running

independently in parallel.

What we now require is a program that matches the light

barrier trigger, the RFID tag and the camera snapshot so

that we shall store all three information as only single triple

in a database row like in this example:

Row Timestamp RFID Image

1 20081026123401 R6754183475

2 20081026123723 R6754183489

63

II

On

e to

Man

y C

om

mu

nic

atio

n




63

Logosworld.com

3 20081026123759 R6754183493

The RFID queue delivers the read RFID tag and the time stamp for it.

Row Timestamp RFID

1 20081026123401 R6754183475

2 20081026123723 R6754183489

3 20081026123759 R6754183493

The video image queue delivers the picture and the file date as timestamp.

Row Timestamp Image

1 20081026123401

2 20081026123723

3 20081026123759

The light barrier effectively delivers nothing but a time stamp.

Row Timestamp

1 20081026123401

2 20081026123723

3 20081026123759

RFID-Queue

Image-Queue

Light Barrier Queue

64

II

On

e to

Man

y C

om

mu

nic

atio

n




64

Logosworld.com

9.4.2 Use cases

The use cases are many folds, since sorting and merging is a

basic algorithm of any kind of process flow. The main

purpose is consolidating and cleansing of multiple input

queues.

9.4.3 Enhancements

Merging queues in practice is more complicated as it first appears. In our example of matching the

RFID tag and the video snapshots we cannot assume that the timestamps in both cases will match

100%. We would rather need to apply a hysteresis to cope with the fuzzy uncertainty of the data and

the timestamps.

The real RFID queue looks then rather like in the example below where the RFID reader fires

permanently and only one of the tome stamps should be taken for matching.

Row Timestamp RFID

1 20081026123401 R6754183475 2 20081026123723 R6754183489 3 20081026123724 R6754183489

4 20081026123725 R6754183489

5 20081026123756 R6754183493 6 20081026123757 R6754183493

7 20081026123759 R6754183493

Real RFID-Queue

65

II

On

e to

Man

y C

om

mu

nic

atio

n




65

Logosworld.com

9.5 Bucketing

Bucketing names a pattern that requires a flow to wait for a certain number of messages to arrive:

until the bucket is full. A simple count is, however, not comprehensive enough. In real world, we

need to care for irregularities, like the case that the bucket will never fill up.

Figure 22: Waiting for the message bucket to fill with tie breaking

Logosworld.com

SOA City

Atom: Bucketing

• Process is halted until a number of messages arrive

• Lock prevention needs to be configurable• E.g Timeout, trigger message („joker“)

Process

SenderMQ-

InboxesWait for

fillProcess Receiver

9.5.1 Use cases

The classical use case is an attempt to be more economical

in triggering a certain task when there is a run-in overhead.

Imagine the case that the message is a pick list for the shop

floor for some goods that require the use of a fork lift or a

crane. It is now not very economical if you request the fork

lift for every small parcel. It is better to wait until there are

enough parcels that shall be delivered to the same target

address to make a full fork lift.

A more complicated use case is a situation where you want

to receive many redundant messages and want to continue

with some statistical processing. For calculating the mean,

Wait until a full load is present

Wait for a minimal number for statistical calculations

66

II

On

e to

Man

y C

om

mu

nic

atio

n




66

Logosworld.com

median, maximum or minimum you need to wait for a

nominal number of messages to arrive.

Imagine that you want to make a promotional offer for a

discounted city travel tour to a European city. Your options

to choose from are Paris, Berlin, Copenhagen and Rome.

You now set up a web site that allows your clients to vote

for one of the cities. You now may wait for a minimum

number of votes to arrive and then continue posting that

promotional offer for the city hitting the most votes.

Another use case may be, that you want to wait for a while

because you want to select in the later step the best offer

found in the messages. Imagine the case of requesting

quotations from several hundred suppliers. You then decide

that you want to wait until 50% of all possible vendors have

replied.

When working with machine data interfaces we often

encounter – like in the RFID case – that we receive

thousands of messages of the same kind. Here we do not

actually expect the bucket to fill up; this would indeed be

the error case rather than the success situation. Instead, we

wait until a certain message arrives, like one that indicates

that the state of the machine changed or the quality of the

processed goods drops. However, if the state change does

not happen in time or if we feel that after 100000 messages

the queue fills up, we raise a process that will collect the

message garbage appropriately.

Voting

9.5.2 What to do if the bucket never fills up?

There may be well situations that we expect 100 messages

to arrive but the messages never arrive in time. For that

purpose we need some tie breaking algorithm that allows us

preemptively launching the process, even if the number of

messages to fill the bucket has not been reached.

The mandatory defaulting mechanism for that are timer

watchdogs that unlock the wait state and continue with the

current bucket load.

Waiting for best-can-offer

Garbage collect

67

II

On

e to

Man

y C

om

mu

nic

atio

n




67

Logosworld.com

Another mandatory tie breaker is reacting on an unlock

message, a message that works like a joker or a master key.

In an elaborate variant the joker message may also carry

additional directives, e.g. a command that advices the

bucket to be tossed instead of processed or the process is

triggered, while an alert message is sent out to some

predefined subscriber.

68

II

On

e to

Man

y C

om

mu

nic

atio

n




68

Logosworld.com

9.6 Quality of service: Guaranteeing Delivery

A quality of service is mainly a watch dog functionality that tells the processing agent how to

handle possible incidents during message delivery.

69

II

On

e to

Man

y C

om

mu

nic

atio

n




69

Logosworld.com


There are many cases where a service sends out messages

and needs to trust that the message is processed by the

designated receiver. If the receiver is not accessible or fails

to process the message the sender would need to take

precautions how to react on the malfunction.

Delegation of the message delivery failure handling to the

middleware makes processing easy for the sender. Instead

of writing extensive code for error handling, the sender

service will only describe in the message envelope the

requested quality of service

Try delivery and forget

Try delivery and reroute to equivalent (usually an email alert to an admin)

Try deliver until success or expiration

Alarm when delivery fails

.

9.6.2 Use cases

Classical use is when sending messages over unsecured or

unstable lines. It is a basic functionality of every ESB to

guarantee a delivery even if the transmission failed.

9.6.3 Example

Imagine sending data via a slow telefon line to a mobile

device on Mount Everest. The line may be so wacky that it

cannot stand transmission of several mega byte on one

shot.A possible solution is splitting the file into small pieces

and reassemble them on the receiver end. It should not be

the duty of an application to cut the message into chunks

but rather ESB functionality to select an appropriate

protocol to transmit the data in little units.

Another example is sending delicate data via an unprotected

line. A guaranteed delivery could also require that the data

is transferred in a way that it cannot easily be hijacked or

spied out by a malicious application.

Controlling unstable or unsecure lines

Sending data to Mount Everest

Encrypting data

70

II

On

e to

Man

y C

om

mu

nic

atio

n




70

Logosworld.com

9.7 Lookups Lookups are used to find additional values as a function of

known values of the message. It is a simple passfilter that

manipulates the streamed message by replacing or adding

values.

Figure 23: Looking up values

9.7.1 Pattern Characteristic

Exception handling is the challenge of the lookup pattern.

Following situations need treatment:

No lookup value found

Loopup service temporarily unavailable

Possible actions are:

Constant default value

Calculated default value

Parking the message for later retry

Parking the message and request lookup completion

Parking the message and trigger a workflow requesting human interaction

9.7.2 Use cases

9.7.3 Exception handling

Exception handling is the challenge of the lookup pattern.

Following situations need treatment:

No lookup value found

Loopup service temporarily unavailable

Possible actions are:

Constant default value

Calculated default value

Parking the message for later retry

Parking the message and request lookup completion

Parking the message and trigger a workflow requesting human interaction

71

II

On

e to

Man

y C

om

mu

nic

atio

n




71

Logosworld.com

72

II

On

e to

Man

y C

om

mu

nic

atio

n




72

Logosworld.com

9.8 Acknowledge

An acknowledge pattern handles reporting feedback to the sender. The sender may wish what

acknowledgement it requires.

Figure 24: Acknowledgement

73

II

On

e to

Man

y C

om

mu

nic

atio

n




73

Logosworld.com

9.8.1 Pattern Characteristic

Possible acknowledgements may be like seen in SAP IDoc usage.

message sent (idoc=03)

message delivered (idoc=04)

message accepted (idoc=06)

message reached point of no return (idoc=09)

message finally processed successfully (idoc=12)

message finally failed (idoc=11)

message temporarily parked for technical reasons

message temporarily parked for user interaction

9.8.2 Use cases

Reporting back the status of a trasnsmission and application processing.

74

II

Co

mp

osi

te P

atte

rns

Composite Patterns



74

Logosworld.com

10 Composite Patterns

75

II

Solu

tio

n P

atte

rns

for

Erro

r R

eso

luti

on

Solution Patterns for Error Resolution



75

Logosworld.com

11 Solution Patterns for Error Resolution

76

II

Solu

tio

n P

atte

rns

for

Erro

r R

eso

luti

on




76

Logosworld.com

11.1 Handling incomplete messages We are facing the following challenge: There is a message in form of an SAP IDoc sent over from an

external data source. The IDoc shall be converted and its data stored to a local database of a

Manufacturing Execution System (MES) using ODBC. Unfortunately the IDoc is not fully complete

and before pushing the data to the MES the delivered material numbers need to be completed with

some internal part numbers. The internal part numbers are, however, available via a service call to

the central ERP. The sender of the IDoc also asked to get an ALEAUD receipt when the Idoc has been

received and another one, when the IDoc has been successfully stored to the database.

11.1.1 Preparing a Solution

11.1.1.1 Inventory of the Requirements

The sender requests a “fire-and-forget” scenario from the receiver of the

IDoc. After sending the data the sender expects that any error handling is

done by the receiver.

Consequence: The data needs to be secured after receiving it from the

external source in case that the subsequent processing is

Solution: The received IDoc needs to be staged to a local staging area

before it is processed. This is usually achieved by receiving the incoming

message directly to a message queue.

The received data is incomplete but the missing information can be

acquired by making a lookup call to a service delivered by the central SAP

ERP

Solution: We shall define an intermediate step that reads the message

from the staging area, makes the service call and flushes the enhanced

message back to the staging area.

Addition: Making a lookup call is a standard atomic pattern. Input and

output message are normally homomorph, this means they share the

same data type. The atomic pattern has standard decorations to give

directions how to behave in case of failure, like default behavior in case of

no matching values, non availability of the service.

11.1.2 Example

An IDoc

o ORDER01 leaves SAP and is destined for SAP XI

o and eventually a DB table

o all handled ASYNC at first

One extra field is needed in the DB that is NOT part of the IDoc

o and the receiving partner refuses to extend the IDoc

o so an RFC has been made.I know they are stupid but this is what

they want and I’m sure there would be other scenarios similar to

this somewhere where ASYNC and SYNC are mixed.

Note as well that an ALEAUD IDoc must go back to SAP only after the data is safely in the DB table.

77

II

Solu

tio

n P

atte

rns

for

Erro

r R

eso

luti

on




77

Logosworld.com

When the IDoc reaches XI a BPM is triggered …..

When the IDoc is received by XI it will be immediately stored to a staging area. There after an event

is raised to tell the processing step to act.

° The RFC is called and the data retrieved….but what happens if the SYNC RFC fails because of

network connectivity etc?

The next step will actually make an RFC call and map the missing values.

The RFC may fail for several and different reasons.

The RFC destination might be temporarily unavailable fails

In this case the message should be parked and the BPM will pause until

the RFC is available again

The RFC might be successful but would not return a value

Here business logic needs to decide how to cope with it. Standard action

patterns are

Alert a responsible to add the missing data

The data may be added either to the IDoc; in this case the workflow

continues after the value has been provided

The data may be added to the RFC source, in which case the RFC needs to

be executed again.

Do the BPM loop until it is successful before moving on?

Yes, but you may set a timeout or additional conditions when you

deliberately abort

Now that I have the extra field I can MAP the IDoc + extra field into the

target JDBC message structure and send it ASYNC to the DB.

So now the message in SXMB_MONI has a successful message BUT the

message has not left the CHANNEL yet. So I can send back an ALEUAD

message at the same time I send the message to the JDBC channel but I

cannot guarantee that it reached the DB?

Normally you send a first ALEAUD with status 06 after the message has

been staged, meaning that it has been accepted but not yet processed.

After the message has been delivered to the database you may send

another ALEAUD with status 09 saying that the message has been

completely processed.

Then you can try reading the data back from the database. If it was

successful you send the final ALEAUD with status=12 to signal that the

process has been completed and verified.

78

II

Solu

tio

n P

atte

rns

for

Erro

r R

eso

luti

on




78

Logosworld.com

79

II

Tab

le o

f Fi

gure

Table of Figure



79

Logosworld.com

12 Table of Figure Figure 1: Synchronous Request & Response ........................................................................................ 16

Figure 2: Fire & Forget .......................................................................................................................... 18

Figure 3: Broadcast ............................................................................................................................... 19

Figure 4: The OSI Seven-Layer Model ................................................................................................... 23

Figure 5: SOA Application Layer Model ................................................................................................ 24

Figure 6: Data Layer strictly separate from transport layer ................................................................. 26

Figure 7: A simple pass filter ................................................................................................................. 31

Figure 8: Data mapping as a filter variation .......................................................................................... 33

Figure 9: Data transformations change envelope ................................................................................ 34

Figure 10: Drive-Through Pattern ......................................................................................................... 35

Figure 11: Content based routing ......................................................................................................... 38

Figure 12: Using a proxy to encapsulate asynchronous processing ..................................................... 42

Figure 13: Request&Reply Pattern ........................................................................................................ 43

Figure 14: Messages are queued and dispatched to next free process ............................................... 44

Figure 15: Messages are usually fetched but may delegate fetching to a dispatcher ......................... 47

Figure 16: Messages are queued and then dispatched to free processes ........................................... 48

Figure 17: Processes can be galvanically decoupled through a MQ ..................................................... 50

Figure 18: Fanning out messages to several subscribed processes ...................................................... 53

Figure 19: The ESB will synchronize the responses of the fanned out processes ................................ 56

Figure 20: Messages are made persistent for certain period ............................................................... 58

Figure 21: Merging Message Queues ................................................................................................... 61

Figure 22: Waiting for the message bucket to fill with tie breaking ..................................................... 65

Figure 24: Looking up values ................................................................................................................. 70

Figure 25: Acknowledgement ............................................................................................................... 72

80

II

Ind

ex

Index



80

Logosworld.com

13 Index benchmark, 53 Canonical Format, 27 Filter, 30, 31, 34, 35, 36, 40 Filters, 29 heartbeat, 48

pass filter, 30 Pass Filter. See Filter Pipe, 30, 36 Pipes, 29 simulation, 53

Date post:	19-Aug-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Atomic Enterprise SOA Patternslogosworld.com/docs/soaindia/soaindia2008/Angeli... · recurring...

Documents