116
| Date post: | 30-Nov-2014 |
| Category: |
Documents |
| Upload: | ganesh-prasad |
| View: | 6,005 times |
| Download: | 5 times |
“Slicing the Gordian Knot of SOA Governance”
Version 0.99.1, November 2012
© Ganesh Prasad
This work is licensed under Creative Commons Attribution-No Derivs 3.0 Australia
http://creativecommons.org/licenses/by-nd/3.0/au/
Confessions of a back-to-basics SOA architect
Table of ContentsSynopsis.................................................................................................................................6
Part I – Why The Industry Is Doing SOA Governance All Wrong..........................................7
Timidity in Asserting the All-Encompassing Scope of SOA...............................................7
Confusion between Governance and Management..........................................................9
Obesity of the Governance Function...............................................................................10
Part II – Elements of a Common Sense Approach..............................................................11
Recognising “Services” as a Weasel Word.....................................................................11
Back to First Principles and the Notion of “Dependencies”.............................................12
Why are Dependencies so Important?............................................................................14
Case Study 1 – Take it or Leave it: Monopsony and Choice......................................15
Case Study 2 – Culture Shock....................................................................................16
Case Study 3 – The Weakest Link of a Supply Chain................................................17
Case Study 4 – Locked Out: Key Person Risk............................................................17
Case Study 5 – The Stress and Strain of an Engineer's Life......................................18
Case Study 6 – It Goes Without Saying......................................................................19
Case Study 7 – “ASSUME” Makes an ASS of U and ME...........................................20
Case Study 8 – What Makes a Good Technology Platform?......................................21
An Architectural Framework to Analyse Dependencies..................................................24
BAIT as Layers of Dependencies................................................................................24
TOGAF Artifacts as Dependency Relationships.........................................................25
Part III – A Practical Guide To Governing And Managing Dependencies............................27
Agencies and Vehicles.....................................................................................................27
Key Roles....................................................................................................................27
Key Bodies...................................................................................................................30
How Many Committees?.........................................................................................32
Functions and Processes................................................................................................35
One-time Processes....................................................................................................35
Initial Dependency Review......................................................................................37
Recurring Processes...................................................................................................37
Periodic Dependency Review ................................................................................37
Remediation Program Review................................................................................37
Remediation Program Proposal..............................................................................39
BAU Program Management....................................................................................39
New Initiative Appraisal ..........................................................................................39
Processes in Steady State..........................................................................................43
Governance and Management Check-lists.....................................................................44
Classification of Dependencies...................................................................................44
The List of Check-Lists................................................................................................45
A Basic Governance Check-list...................................................................................45
A Basic Management Check-list..................................................................................46
Fundamental Enterprise Dependencies......................................................................47
Enterprise Dependency Check-lists........................................................................48
Business Layer Dependencies....................................................................................51
Modelling the Business with Domain-Driven Design..............................................51
Realism around Reuse............................................................................................52
Business Layer Dependency Check-lists................................................................55
Application Layer Dependencies.................................................................................58
Cohesion and Coupling...........................................................................................59
A Fun Exercise in Applying the High Cohesion Principle........................................62
Criteria for High Cohesion.......................................................................................66
Process, Product and Service.................................................................................68
Application Layer Dependency Check-lists.............................................................70
Information (Data) Layer Dependencies.....................................................................73
Data on the Outside versus Data on the Inside......................................................73
Aspects of Low Coupling.........................................................................................74
The Domain-specific Data Dictionary......................................................................75
The Internal Data Model (“Data on the Inside”)......................................................76
The Interface Data Model (“Data on the Outside”).................................................77
Message Data “on the wire”....................................................................................78
Information (Data) Layer Dependency Check-lists.................................................81
Technology Layer Dependencies................................................................................85
Implementing Products...........................................................................................86
Implementing Services............................................................................................86
Implementing the Nouns and Verbs of the Interface Data Model...........................87
The Three Core SOA Technology Components......................................................88
How Not to Use a Broker........................................................................................93
Implementing the Adverbs of an Operation............................................................94
Technology Layer Dependency Check-lists............................................................96
Bringing about Desired Behaviour – Velvet Glove or Iron Hand?...................................99
Summary and Conclusions................................................................................................101
Contributions of this White Paper..................................................................................102
Defining Terms...........................................................................................................102
Restoring Potential....................................................................................................102
Identifying Gaps.........................................................................................................102
Simplifying Tasks.......................................................................................................103
Doubling the Pay-off..................................................................................................103
Potential Criticism of This Approach..............................................................................104
The Weight of Tradition.............................................................................................104
Making Mountains out of Molehills............................................................................104
Drawing a Long Bow.................................................................................................105
A Bridge Too Far........................................................................................................105
About the Author................................................................................................................106
Acknowledgements............................................................................................................106
Appendix A – SOA Governance and Management – An Issue of Definition.....................107
Appendix B – Lessons from Cadet Camp (or Why SOA is Like a Snakepit).....................109
Appendix C – Core Entities and Dependencies in the TOGAF 9 Model...........................111
Appendix D – Artifacts In the TOGAF 9 Model..................................................................112
Appendix E – References..................................................................................................115
Synopsis
“SOA Governance” as practised in the industry today suffers from three major problems:
1. “SOA Governance” is wrongly understood to be the governance of SOA as an IT
function, rather than the application of SOA principles to the governance of all levels of
the enterprise. Most of the potential benefits of SOA as an organising principle for the
enterprise therefore go unrealised because of this overly narrow, technology-focused
interpretation.
2. There is widespread confusion between the terms “governance” and “management”
even among SOA practitioners. Many of the processes associated with “SOA
Governance” are in fact routine management functions, and true governance tasks
requiring direction are often neglected, resulting in costly yet preventable mistakes.
3. The processes associated with “SOA Governance” (which are more often
management than governance) are needlessly heavyweight. Given the overly limited
scope of “SOA Governance” as mentioned in Point 1 above, these processes soak up
a disproportionate share of an organisation's resources without corresponding benefit.
This white paper aims to do the following:
1. Redefine both Governance and Management in simple and unambiguous terms so
that there is no confusion between the two functions and both can be performed
effectively.
2. Raise the scope of SOA Governance and SOA Management beyond their current
narrow technology focus so that SOA thinking can be applied at all levels to improve
the agility, cost and risk profile of an organisation.
3. Recommend a comprehensive yet lightweight approach involving a few core roles and
bodies, a minimal set of processes and a manageable set of checklists to enable both
SOA Governance and SOA Management to be performed cost-effectively.
6
Part I – Why The Industry Is Doing SOA Governance All Wrong
Timidity in Asserting the All-Encompassing Scope of SOA1
“SOA Governance” does not mean the governance of SOA, any more than “scientific
thinking” means “thinking about science”.
We know of course that “scientific thinking” means a different way of thinking about
everything, i.e., adopting a rigorous, analytical, evidence-based approach to understanding
every aspect of the universe without exception. Scientific thinking is about applying
science to thinking, not the other way around.
In exactly analogous fashion, “SOA Governance” is about applying SOA thinking to
governance, not about applying governance to SOA.
So how does one “think SOA”?
We need to understand that SOA is an organising principle that impacts every aspect of
the enterprise. It is not a set of technology products or even an approach to deploying
technology components. The word “technology” refers to the implementation of business
logic2, and most “SOA Governance” activities in organisations that would describe
themselves as “doing SOA” relate to implementation-related activities such as setting
development and environment standards, reviewing the design of SOAP-based web
services and Business Process Management (BPM), controlling versions of services,
establishing policies around the use of an Enterprise Service Bus (ESB), using a
registry/repository to centralise information about services, etc. All of these are in fact
routine management activities, with a narrow focus on technology to boot. These cannot
be called SOA Governance at all!
The definition of SOA we propose is “the science of analysing and managing
dependencies between systems”. Systems need not be computer systems, and neither
are dependencies restricted to technology. Dependencies exist at any level of business,
human relations or technology, as we will show using diverse examples. The term
“managing dependencies” as used above is shorthand for “eliminating needless
dependencies and formalising legitimate dependencies into readily understood contracts”.
1 Service-Oriented Architecture
2 “Technology” may conjure up visions of advanced computer systems, but even a manual ledger to record
transactions is technology. Indeed, it would seem like high technology to a race without paper!
7
In a previous white paper published under the aegis of WSO2 (“Practical SOA for the
Solution Architect”3), we showed how a solution design that is tightly coupled at the data
layer can completely negate the benefits of expensively procured SOA technology (e.g.,
ESBs, registries, etc.) Such situations are unfortunately quite common because
practitioners often take a technology-only approach to SOA and do not see the
dependencies that exist between systems at different levels. The fault lies not with SOA
itself but in our misunderstanding of SOA as being limited to technology. We need to start
seeing SOA as a way of thinking about dependencies, not just within the technology realm
or even at the level of data, but across the board. Unless we as an industry adopt
“Dependency-Oriented Thinking”, the returns on our SOA investments will remain
anaemic4.
It would seem that industry analysts and prominent vendors have knowingly or otherwise
misled us for over a decade with the conceptual model illustrated below, and this view has
stood in the way of our ability to realise the full benefit of SOA. Every expert unfailingly
issues the standard disclaimer that SOA is not about technology, but the opposite
message gets dog-whistled through the emphasis on products to manage web services,
and ultimately prevails. In a later section, we hazard an explanation for this.
Fig. 1 – The limited (and limiting) view of SOA Governance as a subset of IT Governance
(itself a subset of Corporate Governance) – a view endorsed by industry thought leaders
such as the Burton Group5 and IBM6.
3 Downloadable from http://bit.ly/stbfiA
4 In retrospect, DOT may have been a better term than SOA.
5 The Burton Group's Research Director speaks on SOA and Governance: http://bit.ly/HFdF6t
6 IBM's definition of SOA Governance: http://ibm.co/qm46
8
Corporate Resources
IT Resources
SOAResources
Corporate Governance
IT Governance
SOA Governance
To repeat, SOA is not a subset of IT. The benefits of SOA thinking when applied more
broadly are:
1. An improvement in business agility because of minimal dependencies between
systems and consequently minimal “friction” that can impede change
2. Sustainably lower operating costs for the same reasons
3. A significant reduction in operational risk because of better-understood
dependencies and fewer surprises
Clearly, we have lost out on some major benefits by defining SOA as narrowly as we have.
Confusion between Governance and Management
“Governance” is an over-used (and abused) term. One of the unfortunate side-effects of
the collapse of corporations like Enron and WorldCom in 2001-2002 was the sudden
popularity of the term “governance” in popular discourse. The word “governance” has
acquired such a cachet today that it often tends to be used just for effect, even when what
is meant is just plain old “management”. So let's set these terms apart right away.
Governance is ensuring that the right things are done.
Management is ensuring that things are done right.
In spirit, governance is about the ends, or the “What”; management is about the means, or
the “How”. This is a fundamental distinction that is key to correctly implementing what we
recommend in this white paper, hence it is worth spending some effort to understand this
thoroughly.
An analogy with the functions of a company's board of directors and its executive
management will make the distinction between corporate governance and corporate
management more concrete. A decision on whether the company should enter a new
market is a decision for the board, because it pertains to the fundamentals of what the
company wants to be and whether or not the move is in the best interests of its
shareholders. In other words, this is about doing the right thing (i.e., governance).
Once the decision is taken to enter a new market, executive management is responsible
for retooling the resources of the company to enable it to compete effectively in that
9
market. This is about doing things right (i.e., management).
Governance and management decisions apply at lower levels of the organisation as well.
Projects often have steering committees as well as working groups. Steering committees
are comprised of key stakeholders and they tend to make governance (i.e., “what”)
decisions – objectives, scope, success criteria, etc. They also monitor adherence to these
parameters. Working groups are teams of hands-on people assigned to the project. They
make the day-to-day management (i.e., “how”) decisions that will enable them to solve
routine problems and achieve the objectives that have been set by the steering committee.
The steering committee is only concerned with the ends and not the means. The working
group is responsible for the means.
In general, one can tell whether a given decision is about governance or about
management by thinking about how those decisions could be judged in hindsight. If the
verdict is likely to be “right” or “wrong”, then it's a governance decision. If the assessment
is likely to be one of a spectrum (e.g., “excellent”, “good”, “fair” or “poor”), then it's a
management decision.
Obesity of the Governance Function
Organisations that have embraced the requirement for “SOA Governance” generally have
committees, processes and tools to bring discipline to the way services are designed, built,
deployed and managed. After all, this is how the governance of SOA is commonly
interpreted. Some organisations have an “Integration Centre of Competence” that is
tasked with defining standards and controlling the introduction of new services into the
ecosystem.
While these are sensible measures, they impose a rather high overhead especially given
their limited technology-only focus. A centralised approving authority is an overworked
bottleneck, and so agility and operational cost are the first casualties. Many organisations
consequently see very modest improvements in their operational cost and efficiency as a
result of moving to SOA. The economies and dis-economies largely neutralise each other,
and the heavyweight governance processes shoulder a large portion of the blame.
If projects in your organisation are constantly trying to find short-cuts around a centralised
authority and a large part of this latter group's efforts are around reigning in such rogue
projects, it could be a sign that the organisation is chafing under heavyweight governance.
10
Part II – Elements of a Common Sense Approach
Recognising “Services” as a Weasel Word
No matter how often we parrot the mantra “SOA is not about technology”, we end up with a
technology-only view of SOA while we're looking elsewhere. How does this happen?
Our opinion is that there is a slippery slope that we embark on once we agree to talk about
“services”. From “services” to “web services”, and from web services to a technology-only
view of SOA and SOA Governance is then just a natural progression, as illustrated below:
There is a place for the concept of “services” in what we lightly call “Service-Oriented”
Architecture. But the wide-angle lens of dependencies shows us that there's much more to
SOA. We need to derive our view of “services” much more systematically and rigorously
than we have been used to doing. So let's look at this process more closely.
11
“SOA is not about technology”
“Service-Oriented Architecture is a view of the organisation as a collection of reusable services”
“We should expose all business functionality through Web Services”
“It's good practice to proxy all Web Services through an ESB and use a registry/repository to manage them, especially as their numbers start to grow.”
“We need some governance around the use of all this technology”
“SOA assets are owned by IT, so SOA Governance is an IT function”
Fig. 2 – The slippery slope that leads to the view of SOA as a part of IT
Back to First Principles and the Notion of “Dependencies”
The four most important principles of SOA are dependencies, dependencies,
dependencies and dependencies7. We're not being entirely flippant in saying this, because
there are four distinct “layers” in an organisation where dependencies need to be
managed. These layers, as we will explain shortly with the help of a formal framework, are
Business, Applications, Information (Data) and Technology8.
Fig. 3 – The True Scope and Concerns of SOA
- More than just a few Web Services managed by a section of IT!
We believe that for an organisation to be effective in achieving its goals, an acceptance of
this dependency-based view of SOA is essential. Any discussion of SOA Governance and
SOA Management has to start from this basis.
Since SOA is all about the management of dependencies, SOA Governance should be
seen as deciding what dependencies are legitimate and SOA Management as deciding
how to manage dependencies. Both these aspects are important, so even though SOA
7 To paraphrase the 3 rules of real estate: “Location, location, location”.
8 This is commonly referred to as the “BAIT Model”.
12
SOA Principles
(Eliminate needless dependencies between systems,formalise legitimate dependencies into well-understood “contracts”)
Business Layer
Application Layer
Information (Data) Layer
Technology Layer
Manage dependencies
Manage dependencies
Manage dependencies
Manage dependencies
Manage dependencies
Manage dependencies
Manage dependencies
Governance seems to have all the mind-share, this white paper will talk about SOA
Management9 with equal emphasis. Ironically, traditional approaches to “SOA
Governance” are usually about SOA Management (i.e., the “How”).
So here are our simple and readily-understandable definitions:
• SOA Governance is determining what dependencies are legitimate at every layer of
the organisation and identifying what existing dependencies fall outside this set.
• SOA Management deals with how to remediate illegitimate dependencies at every
layer of the organisation, how to formally document and communicate legitimate
dependencies and how to prevent recurring violations.
We will break these down into specific tasks in Part III, but for now, the following high-level
illustration will summarise how we arrive at these definitions.
Fig. 4 – SOA Governance and SOA Management at a glance
This is the core philosophy behind our approach to SOA Governance and SOA
Management in this white paper.
9 In the literature, we often come across the term Service Management. This term derives from the view of
SOA as a subset of IT rather than as a set of organising principles for the entire enterprise. Our term
“SOA Management” is more comprehensive.
13
SOA(The science of analysing and managing dependencies)
Governance(What are the right
things to do?)
Management(How do we do
things right?)
SOA GovernanceWhat dependencies are legitimate?
What existing dependencies fall outside this set?
SOA ManagementHow do we remediate illegitimate dependencies?How do we formalise legitimate dependencies?
How do we prevent recurring violations?
Why are Dependencies so Important?
We believe that a deep understanding of dependencies makes all the difference between
a successful organisation and one that is less so. Sun Tzu said10 many centuries ago,
“Know thyself, know thy enemy – a thousand battles, a thousand victories.” By “knowing”
oneself and one's enemy, he was really talking about dependencies, because strengths as
well as weaknesses are dependencies. What is it that makes you and the other guy tick?
Fast cavalry? Long range artillery? What are the things that can trip you up? Lack of
supplies (logistics)? Troops weakened by dysentery? What buttons can be pushed to
make you or the other guy behave in a predictable way? Siege of a prestigious fort?
Abduction of a corps commander's son? All of these are dependencies. All of these relate
to “knowing thyself” or “knowing thy enemy”.
In that classic marketing strategy book of the '80s, “Marketing Warfare”, authors Al Ries
and Jack Trout offer this nugget of wisdom to their clients who are looking for the most
effective way to compete against their rivals - “Don't attack a weakness that is a weakness.
Attack the weakness inherent in a strength.” Their idea is that weaknesses that are purely
weaknesses can be fixed, but weaknesses that are an inherent part of a competitor's
strength cannot be fixed without compromising that strength. As an example, they suggest
that any company trying to compete against a giant like Campbell Soup should attack not
the price of the soup (since a larger company can easily drop its prices long enough to
drive a smaller competitor out of business), but something peripheral like the metal cans in
which the giant corporation packages its products. The enormous investment in metal can
packaging is a weakness inherent in the strength of Campbell Soup's scale of operations,
and this investment cannot easily be thrown away. A competitor positioning themselves as
using either environmentally friendlier packaging or packaging that is less suspect from a
health perspective would have a better chance than one competing on price.
In other words, Trout and Ries suggest attacking dependencies that a competitor cannot
easily rid themselves of. That's how important dependencies are to the life-and-death
struggle of companies in the marketplace.
To further reinforce the importance of dependencies to an organisation, we provide a few
real-life case studies11 which clearly demonstrate two things – one, that dependencies are
10 The Art of War
11 The examples are authentic but have been anonymised, not so much to protect the guilty as to protect
the authors and their associates from accusations of breach of confidentiality.
14
not restricted to information technology, and two, that a lack of understanding of
dependencies can cost an organisation dearly.
Case Study 1 – Take it or Leave it: Monopsony12 and Choice
A medium-sized commercial organisation in a developing country was experiencing strong
growth, and began hiring more staff to meet the demands of its expanding business. One
of its hires was a seasoned manager from another industry, and his eyes immediately
picked up an aspect of his new employer's business that its owners had been oblivious to.
He was shocked to discover that although the volume of growth was strong, profits were
not growing at all. On the contrary, many of the projects were actually losing money. It took
him very little time to home in on the problem. Almost all of the company's business was
coming from government contracts, which had been its traditional mainstay. Government
contracts had two features that were both very unfavourable to suppliers. One was the
policy of always awarding contracts to the lowest bidder, which squeezed margins to the
bone. The second was the one-sided payment policy, under which the bulk of the payment
for a project would only be released on completion of the work, with a significant amount
withheld for a 12 month warranty period, leaving the supplier with a huge financing gap
while the project was being executed and potential liability for months after completion.
The newly hired manager immediately began to implement a strategy to diversify the
company's client base. As he moved to other markets, he was able to trade in on the
company's reputation and credibility to command higher premiums from clients in the
private sector, generally bypassing the tender process altogether. He also concentrated on
short delivery cycles and 100% payment on completion. In four years, 30% of the
company's business was from the non-government segment. More dramatically, 85% of its
profits came from this segment too.
Moral of the story: The company had a legitimate dependency on its reputation for quality
and reliability because it was impossible to win contracts without a good track record.
However, it had developed an unnecessary dependency on government contracts through
sheer history or habit. The new manager, coming from outside the company, had no such
mental blinkers and could see this dependency that the owners weren't even aware of. He
then leveraged the company's legitimate dependency as an explicit marketing tool to win
12 http://en.wikipedia.org/wiki/Monopsony
15
new business, simultaneously reducing its unnecessary dependency on government
contracts with their unfavourable commercial terms. The results were spectacular.
Case Study 2 – Culture Shock
When a new General Manager took over at a company specialising in turnkey engineering
projects, he was immediately struck by the lack of standardised process in his new
organisation. No two projects, however similar, followed the same process. Project
execution was ad hoc and freewheeling, with lower level managers and engineers making
informal decisions in oral negotiations with their counterparts in their client and supplier
organisations. True, the business seemed to be running smoothly, but the lack of process
seemed to be a ticking time-bomb.
The GM came down hard on this laidback culture. He began to insist on standardised
processes and timetables for all projects. Instead of informal agreements between
operating staff, the company began to send out formal communications of intent, spelling
out what was going to be done and when, and what the other party was expected to do.
Suppliers were kept on a tight leash and offered little room to negotiate on terms.
Within months, dramatic results began to be seen. Project schedules began to slip. Clients
began to complain loudly. Suppliers were less accommodating when plans had to change.
Things became bad so rapidly that the manager was fired and replaced with one of his
senior direct reports who was familiar with the old way of doing things. It took many more
months to bring the situation back to normal.
It turned out that in the turnkey engineering business, every client and every site is
different and requires a different approach. Flexibility is key, because there are lots of
unknowns and unexpected developments during the course of a project's execution. Two
important aspects of managing these changes are sufficient autonomy for local staff and
maintenance of good working relationships between key people, on the client side as well
as on the supplier side. A one-size-fits-all approach to process does not work. Neither
does a rigid and formal approach to managing external relationships.
Moral of the story: The company had a unavoidable dependency on local and individual
conditions for every project. The impacts of this dependency could only be mitigated
through flexibility, and this was achieved through a conscious culture of autonomy to staff
“at the coalface”, mutual respect and adjustment between business partners, and regular
16
and informal communication between key people. The new general manager wrongly saw
the informal agreements between people as a failure to formalise dependencies into
contracts, and he then tried to replace these with more standardised processes and rigid
contractual agreements, but these measures removed the cushion that the company had
developed to shield it from its fundamental dependency on local conditions.
This case study provides a useful contrast with the previous one because what looks like
an unnecessary dependency may not in fact be so. It takes astute observation to
understand dependencies and to leverage or manage the legitimate ones while eliminating
those that are truly unnecessary.
Case Study 3 – The Weakest Link of a Supply Chain
An Australian construction company based in Sydney fabricated aluminium and timber
windows for the building industry. The company was careful to hedge against currency
fluctuations as well as the price of aluminium. But no such arrangements were in place for
timber which was sourced from a supplier based in the neighbouring state of Victoria.
What the construction company didn't realise was that Chile and Malaysia are the two
major sources of timber in the world, and that the Victorian supplier imported timber
exclusively from Chile.
In February 2010, a major earthquake in Chile disrupted the supply of timber, resulting in
worldwide timber shortages and a steep rise in prices. Since the supplier could no longer
source timber in a timely or cost-effective manner, the Windows fabricator in turn could not
deliver windows to its builders, resulting in delays, losses and unhappy customers.
Moral of the story: If the construction company had looked beyond its immediate
Australian supplier and studied the entire supply chain for its products, it would have
realised the critical dependency it had on Chilean timber. Once formally recognised, it
could then have done a number of things to mitigate that dependency, such as hedge
against the price of timber, force its supplier to hold sufficient inventory, or diversify its
supplies across Chile and Malaysia. Since it neglected to understand and mitigate the
effects of this dependency, it paid a heavy price.
Case Study 4 – Locked Out: Key Person Risk
A small value-added, niche-market IT distributor had a sales team of 6 to 7 people headed
17
by a national manager. The company had recently changed owners, and the new owner
had not had enough time to familiarise himself with the company's customers.
The national manager was suddenly poached by a competitor. He not only took his entire
sales team with him, but also destroyed all records at his old company before leaving,
wiping computers clean and removing all business documents.
The owner was faced with the closure of his new business, since he had no records of any
of his customers. Although his national manager's betrayal could have been criminally
prosecuted, he did not have the time or the resources to pursue litigation.
This story did however end happily. The owner put together a fresh team under a new and
dynamic national manager, and they painstakingly won back all their customers over the
next five years. The predatory competitor went bust.
Moral of the story: The business had a critical dependency on its customer relationships,
physically manifested as its database of customers and records of interactions with them.
This dependency was not recognised and formalised as a contract, e.g., a customer
database that was regularly backed up and protected against loss or damage. The owner's
failure to formalise and manage this dependency cost him heavily.
Case Study 5 – The Stress and Strain of an Engineer's Life
A company supplying and installing power generators in buildings encountered a problem
that had not been anticipated or designed for.
A generator installed in the basement of a twelve storey building was hooked up to an
exhaust pipe to transport hot fumes to the outside through a shaft in the core of the
building. The metal exhaust pipe ran horizontally for 6 metres along the ceiling of the
basement and then turned upwards in a 90-degree L-joint to run a further 40 metres to the
top of the building through the central shaft. This was the first time the company's
engineers had deployed a generator in this configuration, since all their previous
engagements involved ground-level generators with short exhaust pipes.
The engineers noticed a problem after installation. Whenever the generator was in
operation and began to push out hot exhaust fumes, the 40 metre column would expand
with the heat, pushing down on the L-joint and forcing the horizontal section to arch
downwards almost half a metre. The pipe would return to its position once the generator
18
stopped and the pipe cooled, but this repeated movement due to expansion and
contraction was a certain recipe for failure of the pipe due to fatigue.
Ultimately, the engineers came up with a solution to replace the rigid L-joint with a much
looser (though still airtight) bell-shaped container into which both the horizontal and
vertical sections of the pipe could extend. A system of rollers allowed both sections of the
pipe to expand smoothly into the “bell” and contract equally smoothly out of it, with neither
section's movement impacting the other. Although the company had to spend extra time
and money to implement this solution (which they could have built into the initial design
had they anticipated the problem), they managed to avert a future, potentially more
expensive accident.
Moral of the story: The two sections of the generator's exhaust pipe had an unnecessary
dependency on each other's thermal expansion due to the rigid L-joint between them.
Once this dependency was eliminated through a more loosely-coupled expansion
mechanism, the problem disappeared.
Case Study 6 – It Goes Without Saying
At a large bank in the Middle East, a sincere and hard-working British expat Business
Analyst was put in charge of a major branch automation project. The bank's tellers were
using mainframe “green screen” terminals, and these were overdue for replacement with a
more modern interface.
The BA worked hard for many months, interviewing bank staff and management, gathering
requirements, creating questionnaires for vendors, issuing RFIs and RFPs, conducting
evaluations and negotiations, etc., and finally selected a vendor that seemed superior in
every respect. Although he made presentations from time to time to management to keep
them informed of his progress, he was doing everything virtually single-handed.
Finally, after about a year of hard work, and just before the contract was to be awarded to
the successful vendor, the BA organised a demonstration of the branch automation system
for the bank's senior management and branch heads.
Partway through the impressive demonstration, one of the managers asked, “Does the
system support Arabic?”
The BA was stunned. He turned to the vendor representative, who looked embarrassed.
19
No, the system did not support Arabic. The commotion and head-shaking that greeted this
admission made it abundantly clear that the deal was dead. The poor BA was later spotted
sitting at his desk with his head in his hands.
The search for a branch automation system had to resume almost from scratch, at a huge
cost in time and money.
Moral of the story: Although the BA had taken great pains to gather detailed requirements
from a wide cross-section of bank staff, Arabic language support was considered to be
such an obvious requirement that no one had actually spelt it out. Being a recently-arrived
expat, the BA didn't consider it either. And so the most obvious and important requirement
for the system went unmet and ruined a year's worth of work. In other words, a legitimate
dependency was not formalised in the form of a contract (i.e., the requirements document).
Its absence was therefore not noticed until too late and resulted in a nasty surprise.
Case Study 7 – “ASSUME” Makes an ASS of U and ME
A multinational systems integrator was engaged to deliver a comprehensive Corporate
Internet Banking solution to a major Australian bank. Most of the functionality of the
application would leverage the bank's mainframe-based product systems, and these would
be exposed as services through middleware. What remained to be sourced was a suitable
web-friendly front-end product with the required rich user interface. The SI evaluated a few
packaged products and selected one for its client.
The overall project was a multi-million dollar undertaking with about a hundred people
working on the program. There were business analysts, project managers and a large
number of technical staff working on the mainframe and middleware, along with a smaller
team of front-end designers and developers.
It was known that the selected product had been originally designed to work against data
in a relational database, but the product vendor's architects had assured the SI that it was
relatively easy to graft the front-end onto the middleware services that were being built.
A few months into the project, the SI's front-end designers got their first look at the source
code of the front-end product, and one of them noticed references to a “Row ID” sprinkled
throughout the code. The vendor's front-end developers innocently explained that this
referred to the row number in the database. The SI team was aghast. “But there's no
database! You know this has to work against services talking to a mainframe!”
20
A meeting of the SI's senior architects and project managers was hastily called. The
vendor's senior architects were summoned from overseas. Not having a detailed
knowledge of the code, they had not realised that the relational database assumption
would be so deeply rooted in their product's design, and they estimated that it would take a
further 3 months to fix. All knew that a 3 month delay was unacceptable to the client.
Ultimately, the vendor relationship was terminated, the SI's project manager was fired, and
the SI barely salvaged the contract after demonstrating a custom-built working prototype to
the client. The fiasco threatened to derail a prestigious project and came perilously close
to rupturing a highly visible, multi-million dollar relationship between a multinational
systems integrator and one of its biggest Australian clients.
Moral of the story: The front-end application had hard-coded its dependency on a
relational database into every element of its user interface. The vendors' architects had not
adequately documented how this dependency should have been isolated, and those who
documented the implementation had not adequately recorded the extent of the
dependency. Either of these “contracts” would have alerted people to the extent of the
dependency. No wonder even the vendors' architects were blind-sided.
Case Study 8 – What Makes a Good Technology Platform?
To round out this section, let's end with a generic example from the realm of technology –
the notion of a “platform”.
A platform is a technology environment that provides all the necessary run-time support for
applications. These could be virtual machines, interpreters, dynamically linkable libraries,
etc. Let's look at five platforms that have been popular over the last decade and a half.
i. The Windows operating system is a platform for native Windows applications. While
its ubiquity was attractive to Independent Software Vendors (ISVs), there were a
few problems with this platform. One was the shift from 16-bit to 32-bit around 1995,
which rendered quite a few third-party applications unable to run on the newer
version. Another persistent problem was known as “DLL hell”, because of the
dependencies of applications on versions of dynamically linked libraries. An
upgrade to an underlying system library could and often did break dozens of
third-party apps. A third issue was that Microsoft exploited “undocumented Windows
21
APIs” in its own applications that rival app vendors could not use, and so it was not
a level playing field.
ii. Linux (like any Unix variant) has the notion of symbolic links that i t uses to
advantage to manage its dynamically linked libraries (called “shared object files”).
Shared object files have major version numbers and minor version numbers.
Changes in minor version numbers represent changes that do not break the API.
Changes in major version numbers represent changes that do break the API.
Symbolic links are used to hide changes in minor version numbers but not in major
version numbers, so applications that declare their dependencies in terms of the
symbolic links will be shielded from irrelevant changes but rightly impacted when
the API changes. Also, old and new versions of files can simultaneously exist to
support old and new versions of applications.
Shared Object
version
Symbolic link Description
libxxx.so.2.1 libxxx.so.2 Original version that applications are dependent on.
libxxx.so.2.2 libxxx.so.2 Change in shared object's minor version number is not
reflected in the symbolic link, so dependent applications
are (correctly) not impacted. Both shared objects can
exist simultaneously, but only one is “active”.
libxxx.so.3.1 libxxx.so.3 Change in shared object's major version number is
reflected in the symbolic link, so dependent applications
are impacted, as they should be. Both symbolic links
can exist simultaneously, supporting old and new
versions of applications.
Table 1 – Linux and the use of symbolic links to abstract insignificant version changes
iii. The Java Virtual Machine is a platform that runs Java bytecode. The Java language
has undergone numerous version revisions, but the JVM has hardly changed in 15
years. Compiled bytecode from 15 years ago runs on the JVM just as well as
compiled bytecode from the latest version of Java.
iv. The browser is a platform too. In the mid- to late-nineties, developers were
discouraged from writing applications that relied on Javascript running on the
browser, because there were at least two broad variants of browser platforms –
22
Netscape's and Microsoft's – and even the versions of each browser implemented
different versions of the HTML and JavaScript standards (if there were standards at
all). It is only since very recently that the browser can be relied upon as a platform
that will run applications in a predictable and consistent way, independent of
browser vendor.
v. Apple's products have a reputation for ease of use, but they are also highly 'closed'.
Standard input/output mechanisms like USB ports are not supported, and very little
can be achieved by a device without integration to iTunes, iCloud or both. For users
who don't mind the restrictions that the Apple platform places, everything “just
works”. The competing Android platform is more open, but also a bit more chaotic in
terms of multiple versions and multiple implementations by device vendors.
Moral of the story: For a platform to be considered a good one, applications must, above
all, be able to rely on consistent and stable behaviour. Irrelevant changes must not be
allowed to impact applications. There must be considerable latitude in the definition of
what constitutes a change, so that the bulk of changes within the platform can be
“absorbed” without exposing applications to them. The five examples above showed
various approaches to the construction of a stable platform, with varying degrees of
success. We can see that the notion of “dependencies” underlies all of them.
Summary
We have now seen several dramatic examples of how unappreciated dependencies
caused, or threatened to cause, problems of reduced agility, increased cost and increased
risk to organisations. Not all of these have to do with technology, much less with “SOA
technology”, and the choice of examples has been deliberate in that regard. Yet the failure
to understand the potential impact of dependencies is the common thread in all these
stories. A basic level of dependency-oriented analysis would have highlighted the costs
and risks in each of these cases, saving the concerned organisations significant amounts
of time, money and risk.
Having therefore established the crucial importance of dependencies to an organisation's
viability and success, let us see how we can approach the analysis of dependencies in a
systematic way. This is what SOA Governance and SOA Management are really about.
23
An Architectural Framework to Analyse Dependencies
We need a framework to analyse dependencies before we determine how to govern and
manage them. Rather than reinvent the wheel, let's look at some existing and
well-understood models to see if they suit our requirements.
BAIT as Layers of Dependencies
The BAIT model is a popular architectural framework that decomposes an organisation
into four layers – Business, Applications, Information (Data) and Technology.
BAIT has the right idea about layers, but it is traditionally focused on identifying the entities
that exist at each layer and the relationships between those entities. There is no strong
emphasis on the aspect of dependencies, even when dealing with relationships. When we
adopt the BAIT model to aid us in SOA Governance and Management, we have to adapt it
to focus on dependencies, as follows:
In other words, we prefer to think of BAIT's four layers as representing four “I”s, referring to
the (1) Intent, (2) Internal Cohesion (grouping of business functions into highly cohesive
“applications”), (3) Interfaces/Integration between logical functions with low coupling, and
(4) physical Implementation of all the components of the logical business organisation.
The emphasis on cohesion and coupling is particularly relevant, as we will see in later
sections.
24
Business-LevelDependencies
Domain Dependencies(High Cohesion Principle)
Interface Dependencies(Low Coupling Principle)
Technology Dependencies
The BAIT Model applied to SOA concerns
Business Entities
Applications and Systems
Domain Data Models
Technology Standards,Software and Hardware
Traditional concerns ofthe BAIT Model
Business Layer(Intent)
Application Layer(Internal Cohesion)
Information (Data) Layer(Interfaces/Integration)
Technology Layer(Implementation)
Fig. 5 – Adapting the BAIT Model to address SOA concerns
TOGAF13 Artifacts as Dependency Relationships
The TOGAF framework of Enterprise Architecture builds on the BAIT model and describes
generic entities in each layer along with their inter-relationships. This is extremely valuable
to us when embarking on SOA Governance and SOA Management because the hard work
of understanding the components of an organisation and how they fit together has already
been done.
However, just like with BAIT itself, TOGAF as it stands is not a perfect framework that can
be used without adaptation to our treatment of SOA Governance and SOA Management. It
does not inherently support the dependency-oriented organisational transformation that we
would like to model and plan for. SOA is a single isolated chapter in The Open Group's
780-page TOGAF 9 book, which betrays a traditionalist view of SOA as a subset of IT
rather than as the science of analysing and managing dependencies across the board.
However, nothing prevents us from using a dependency lens to view the detailed set of
entity relationships identified by TOGAF and derive a more applicable model for our
requirements. We'll show how to do that in Part III of this document.
TOGAF defines some core concepts at the four layers and lists out some important and
standardised documents (“viewpoints” in the TOGAF terminology) that provide useful
information about these concepts and their interactions. TOGAF uses the term “catalog” to
refer to any list. Similarly, it uses the term “matrix” for tables that show the relationship
between exactly two concepts, and “diagram” for any depiction of how more than two
concepts are related.
If we interpret “relationships” as “dependencies”, matrices become two-entity
dependencies and diagrams become multi-entity dependencies. The set of
TOGAF-defined matrices and diagrams at each of the four BAIT layers then gives us a
ready-made check-list of the dependencies we need to be mindful of within those layers as
well as between layers.
13 The Open Group Architecture Framework: http://www.opengroup.org/togaf/
25
(a) (b) (c)
Fig. 6 – (a) a TOGAF “catalog”, or list of entities; (b) a TOGAF “matrix”, or table of two-entity relationships; (c) a TOGAF “diagram”, or a depiction of multi-entity
relationships.
We consciously adapt TOGAF's set of artifacts to align them with a SOA view of the
enterprise because TOGAF is not inherently dependency-oriented. In other words, the
entity dependencies we study will correspond to our SOA version of the BAIT Model rather
than the traditional one.
We can now begin to see why the traditional view of “SOA Governance” is so limited.
Clearly, our entire approach needs nothing less than an overhaul. It's time to look at how
SOA Governance and SOA Management should in fact be done.
26
Business-LevelDependencies
Domain Dependencies(High Cohesion Principle)
Interface Dependencies(Low Coupling Principle)
Technology Dependencies
SOA Version of BAIT Model
Business Layer(Intent)
Application Layer(Internal Cohesion)
Information (Data) Layer(Interfaces/Integration)
Technology Layer(Implementation)
Core entities,Two-entity
dependencies and Multi-entity
dependencies
TOGAF Artifacts
Fig. 7 – TOGAF artifacts classified by BAIT level
SOA Governance(“What dependencies are
legitimate?”)
SOA Management(“How do we manage
dependencies?”)
Business
Application
Information (Data)
Technology
Development and environment standards, reuse metrics, Web Service version management, repository structures to store WSDL, schema and policy files, etc.
Fig. 8 – Traditional “SOA Governance” concerns are more correctly SOA Management, and a subset at that. Little effort is focused on the governance question, “What
dependencies are legitimate?”
Traditional“SOA Governance”
concerns
Part III – A Practical Guide To Governing And Managing
Dependencies
In the previous two parts of this white paper, we critiqued the industry's approach to SOA
Governance and suggested how we could do it better by considering the elements of a
more rational approach based on the analysis of dependencies.
In this part of the white paper, we will show how to put these elements together into a
lightweight method. We'll recommend some key roles and bodies that will be responsible
for the functions of SOA Governance and SOA Management, the minimal set of processes
they need to carry out, and a manageable set of check-lists that they should use for these
purposes.
Agencies and Vehicles
Key Roles
We believe that the primary agency to apply dependency-oriented thinking at every level of
an organisation is the Enterprise Architecture function. In organisations with an Enterprise
Architecture group, there are usually architects with specialised skills who are tasked with
defining the business architecture, application architecture, data architecture and
technology architecture, so the BAIT Model is already an operational tool. These
specialised architects are the people who need to spearhead a new way of approaching
SOA Governance and SOA Management.
The role of Enterprise Architecture under this model is to analyse the four layers of the
organisation specifically from a dependency focus, first to determine the “what” (the
legitimate dependencies that should exist at each level as well as the unnecessary ones
that do currently exist) and then to guide the “how” (the programs of work required to align
the organisation to the set of legitimate dependencies that were identified).
One of the suggestions we have for the Enterprise Architecture function to assist them in
transitioning to this world-view is to include professionals from certain other disciplines,
even if on a part-time basis. Project Managers, Risk Managers and Contract Lawyers are
all used to looking for dependencies in their own areas, and they can inject some of the
fresh blood that the traditional Enterprise Architecture group needs to make them effective
in a SOA sense.
27
In organisations with a reasonably effective architecture function, here's how business
intent is transformed into working systems today:
Fig. 9 – Key roles and their traditional concerns
Here's the subtle but important change we would like to see:
Fig. 10 – Key roles and their recommended concerns
It's not as if architects don't already think about dependencies. It's just that dependencies
need to be promoted to be a primary, top-of-mind item in all discussions and decisions,
28
especially where the business and IT are involved. Every individual who is part of the
decision chain from business objective to implementation, no matter what their level,
needs to be sensitised to the importance of dependencies. That's what it means to
inculcate “SOA Thinking” within an organisation, and architects need to don the mantle of
evangelists to spread this philosophy.
The perspectives of project managers, risk managers and contract lawyers could be very
useful additions to their skill set, since these give them a “fresh pair of eyes” with which to
recognise the diverse set of dependencies impacting the enterprise.
SOA Governance and SOA Management become far easier to execute when the
organisational culture understands the importance of dependencies.
29
Key Bodies
While the Enterprise Architecture function is key to our recommended approach, SOA
Governance and SOA Management require the active participation of many different roles
and functions to be effective. The business and IT are crucial stakeholders, as we saw. It is
essential to involve multiple roles in the SOA Governance and SOA Management
functions, and this calls for new organisational structures. We are all too aware of the
reputation of committees as inefficient bureaucracies14, but the multi-disciplinary nature of
the tasks at hand, together with the fact that these are ongoing responsibilities, dictate the
need for one or more standing bodies rather than ad hoc teams. Broadly, two groups of
people are required to carry out the required functions:
• a “Dependency Governance Committee”
• a “Dependency Management Committee”
Now this is a functional classification, and these two logical groups could be realised as
several physical committees, e.g., one or more at each level of the BAIT model plus one at
the Enterprise level. Some committees could also take responsibility for more than one
BAIT layer, as long as they are clear which layer they are dealing with at any given time.
The decision on how many physical committees to have is entirely up to the organisation
in question, but we would strongly recommend against combining governance and
management functions within any single committee. That would defeat one of our primary
goals, which is to perform both functions effectively and without confusion.
Additionally, although it's the committees' functions that are important and not their names,
we would recommend that the term “dependency” be explicitly used in their names in
order to inculcate a dependency focus within every person in the organisation. We've lost
a decade's worth of benefits thanks to poor naming (“Service-Oriented” as opposed to
“Dependency-Oriented”), so let's call a spade a spade this time, however corny it may
sound. A committee should either be a “Dependency Governance Committee” or a
“Dependency Management Committee”, with prefixes as appropriate to the BAIT level
and/or business unit.
14 Two of our personal favourites are “A camel is a horse designed by a committee” and “A committee is a
group of people who individually can do nothing, but who get together to decide that nothing can be
done.”
30
In a nutshell,
• The Dependency Governance Committee is responsible for defining which of the
dependencies at the level they are responsible for are legitimate and which are not.
They are also responsible for prioritising the programs of work to remediate
dependencies.
• The Dependency Management Committee is responsible for costing, planning and
initiating the programs of work that will bring and keep the organisation in line with the
model validated by the Dependency Governance Committee and according to the
priority decided by that committee.
The Dependency Governance Committee should have a fair representation of
Enterprise Architects, with specialists in each of the BAIT layers. At least some of the
members of the Dependency Governance Committee should have a background in one of
the non-IT professions listed earlier15, because these professionals are trained to look for
dependencies. Depending on the layer of the BAIT model that the committee is
responsible for, there should be representation from senior executives, business
managers, functional heads and technology specialists. This may mean separate
committees, or a single committee with some fixed members and a number of others who
attend only the sessions that are relevant to them.
Importantly, the Dependency Governance Committee must have authority and access to
funding to be able to approve programs of work, otherwise the organisation will be unable
to remediate the dependencies identified as invalid. One of the reasons why we expended
so much ink in Part II to present 8 separate case studies was to hammer home the idea
that attention to dependencies saves real money. Business unit heads should therefore not
be churlish about funding dependency remediation tasks, and their representation on
Governance committees serves more than just a decorative function!
The Dependency Management Committee should comprise people skilled in costing and
planning, typically project managers and functional heads. They may need representation
from Enterprise Architecture to keep them focused on the dependency aspect while they
attend to their more familiar roles of costing and planning programs of work. Again, there
will be a need to have different people engaged for different layers of the BAIT model, so
there could be considerable variety in the structure and/or composition of the
committee(s).
15 Project managers, risk managers and contract lawyers.
31
How Many Committees?
The dependency governance and management functions can be performed by just two
committees in a small organisation, since they can do justice to the complexity of the entire
organisation themselves. As organisations become larger, the committees will need to
become more specialised, as the following examples illustrate.
32
Dependency Governance Committee
Dependency Management Committee
Fig. 11 – Committees for a small organisation
Enterprise Dependency Governance Committee
Enterprise Dependency Management Committee
Business Dependency Governance Committee
Application & Information Dependency Governance
Committee
Technology Dependency Governance Committee
Business Dependency Management Committee
Application & Information Dependency Management
Committee
Technology Dependency Management Committee
Fig. 12 – Committees for a medium-sized organisation
Enterprise Dependency Governance Committee
Enterprise Dependency Management Committee
Business Dependency Governance Committee
Information Dependency Governance Committee
Technology Dependency Governance Committee
Business Dependency Management Committee
Information Dependency Management Committee
Technology Dependency Management Committee
Application Dependency Governance Committee
Application Dependency Management Committee
Business Units /
Geographies
Fig. 13 – Committees for a large organisation
If this set of committees is beginning to look too bloated, our audacious claim is that these
are the only governance and management structures that an organisation will ever need,
so this could in fact represent a drastic simplification to the plethora of management
bodies that currently exist in a given organisation.
Let's look at a few special cases to support this claim.
Security dependencies: Security is an important aspect of business operations, and
information security traditionally tends to be treated as an IT concern. However, security
concerns are relevant at different levels of the BAIT model. Perimeter security, for
example, may be relevant only at the level of technology, since it is completely opaque to
higher layers. Data Leakage Protection applies at all levels from the business layer down,
since it is impacted by business processes, application design, data interchange as well as
technology-related aspects like USB device control and disk encryption. PCI-DSS16
compliance requirements may or may not impact business processes but will certainly
impact application design, data storage and encryption/tokenisation technology. The
dependency governance and management committees at these various levels will address
and manage the security concerns listed above, as well as any others that arise from time
to time.
Non-technology domains that are not line-of-business, such as Human Resources,
Accounting, Finance or Legal, also fit into this dependency-oriented model.
Human Resources dependencies: Clearly, an organisation that employs many more
contractors than permanent employees has a very different dependency profile than one
with the opposite composition. The contractor-based organisation can downsize more
readily in an economic downturn and re-hire when required, but it has a more tenuous hold
on IP due to the less “sticky” nature of employee knowledge. The contractor-based
organisation will have to invest more heavily into knowledge-management systems to
mitigate against their risk of losing critical business and operations knowledge. The
organisation with a higher proportion of permanent employees will have to be more
cautious about hiring in an economic upturn because of their larger financial exposure
arising from retrenchment benefits they may have to pay out when downsizing. A business
dependency governance committee and business dependency management committee
focused on the HR business unit will be able to formalise these concerns and evolve
mitigating strategies.
16 Payment Card Industry Data Security Standard
33
Finance dependencies: An organisation that is funded mainly by debt has a different
dependency profile to one funded mainly by equity. A debtor and a creditor represent
dependencies of different kinds, and every financial instrument that is issued or purchased
imposes a different flavour of dependency on the enterprise. Financial controllers make
decisions based on such dependencies all the time, whether or not they use the word
“dependency”. Regardless, the business dependency governance and management
committees for the Finance business unit will standardise these functions in a consistent
corporate style.
In short, these varied examples support our claim that SOA, being all about dependencies,
is not about technology but is an organising principle for the enterprise and lends itself to a
simple and consistent form of governance and management. Form follows function, and
the structure of the required governance and management committees follows the
discipline of dependency-oriented thinking.
[It may be argued that the bulk of a modern enterprise's business logic tends to be coded
into software-based systems with very little residual manual processing, so “it's all IT
anyway”. Let us refute this argument with a simple analogy. Just because all legal
documents in a country are in the English language does not imply that a mastery of
English is all that is required to understand or to draft legal documents. One needs to
understand law. In similar fashion, even if every piece of business logic in an organisation
is implemented within a technology platform, understanding how to govern and manage
technology alone is not sufficient. The principles behind how business logic and business
data are identified and organised need to be understood, and the analysis of
dependencies is a crucial tool to achieve this understanding.]
34
Functions and Processes
Broadly speaking, the processes required for SOA Governance and SOA Management are
either one-time or recurring.
To keep an otherwise dry topic light and readable, we will illustrate what occurs during
each of these processes through imaginary conversations between various parties. The
players in all these scenarios are the following:
G – Dependency Governance Committee
M – Dependency Management Committee
B – Any Business Unit
One-time Processes
When an organisation embarks on doing SOA Governance and Management the way this
white paper recommends, there will be a one-time activity (which could of course be
broken up into phases for manageability) on the part of the Dependency Governance
Committee to do two things:
1. Agree on the set of approved dependencies at each level of the organisation as
suggested by the BAIT model;
2. Identify the dependencies that actually exist, specifically the ones that fall outside the
set of approved dependencies above.
The latter list (the set of actual dependencies that are not approved) is the input to the
Dependency Management Committee to plan a program of work to remediate.
35
36
Management FunctionsGovernance Functions
One-TimeDependency
Review
NewInitiativeProposal
PeriodicDependency
Review
RemediationProgramReview
RemediationProgramProposal
BAUProgram
Management
NewInitiativeReview
Publication:List of Approved Dependencies
List of Dependencies to be remediated
Update:List of Approved Dependencies
Update: List of Approved Dependencies
List of Dependencies to be remediated
List of programs, grouped and costed
Initiative Details
Non-approval:List of Dependencies in violation
Approval
Approval:Prioritised list of programs, with funding
Report:Dependencies remediated
Reviewcycle
BusinessProjectcycle
Need not align to BusinessProject cycle
Fig. 14 – Dependency Governance and Management Processes
Dependency-related Process Existing Business Process
BusinessDriver
(Objective)
Business Requirements
Initial Dependency Review
Scenario 1:
G: “We've done a review and decided that the only legitimate dependencies are these.
There are many actual dependencies we've found in our systems that are not in this
approved set. They need to be eliminated.”
M: “OK, we'll come back with a proposal for a program of work to remediate them.”
Recurring Processes
There are recurring processes within both the governance and management functions.
Recurring governance processes are reviews to re-validate the set of legitimate
dependencies at any level and to identify afresh the set of existing dependencies that fall
outside of this approved set. The latter set (which is expected to shrink with each iteration)
is then an input to management committees to initiate programs of work to remediate.
Prioritisation and approval of these proposed programs of work is also a governance
function.
Periodic Dependency Review
Scenario 1:
G: “Given the changing nature of our business since the last review, some new
dependencies are now deemed legitimate and some old ones no longer are.”
M: “OK, we'll make a note of the new set of approved dependencies and come back to you
with a proposal for a program of work to eliminate the ones that are no longer approved.”
Scenario 2:
G: “Some new dependencies that are not in the approved list seem to have “snuck into”
our systems since our last review. These need to be eliminated ASAP.”
M: “We don't understand how that could have happened. We'll come back to you with a
proposal for a program of work to remediate them.”
Remediation Program Review
M: “We've put together a proposal for a set of programs that will remediate the unapproved
37
dependencies that you identified. We've worked out the costs and benefits of each of
them. Please tell us how you would like to prioritise them for execution, and please
approve funding for the ones you want done in the current period.”
G: “We think you should initiate the following subset of programs in the current reporting
period and defer the remaining for a future period. The funding for the programs in the
current period is hereby approved.”
M: “We'll initiate these programs right away.”
38
Recurring management processes are intended to identify programs of work to remediate
the invalid dependencies identified by the governance committee(s) and to manage these
programs once approved by them.
Remediation Program Proposal
Scenario 1:
M (internal conversation): “Given the set of existing dependencies that were identified by
the Dependency Governance Committee as not being on the approved list, let's work out
the costs and benefits of eliminating them, and group these tasks into cohesive programs
of work for them to prioritise and approve.”
BAU Program Management
Once a Dependency Remediation Program is approved, it is managed in exactly identical
fashion to regular business projects. These are not to be treated as projects of secondary
importance or nice-to-haves because of a mistaken characterisation of SOA as technology.
On the contrary, the involvement of business and technology representatives and proper
positioning of the benefits of these initiatives by Enterprise Architecture should ensure that
they are taken equally seriously as programs initiated by the business. After all, unfixed
dependencies mean significant amounts of real money, as we have seen earlier.
The business initiatives detailed in the next section also feed into Business-As Usual
(BAU) program management alongside dependency remediation programs.
The standard discipline of conducting Post-Implementation Reviews (PIRs) to evaluate the
success of a program should be conducted for dependency remediation initiatives as well.
Granted, the benefits are not always immediate but recurring, but it should be possible to
assess if the dependencies slated to be eliminated were in fact removed, and a fresh
assessment of the costs and benefits of the initiative should be conducted to report back to
the Dependency Governance Committee.
New Initiative Appraisal
An event that could trigger a fresh governance activity is when there is a business driver
(objective) necessitating a program of work, the proposal is put forward, and the
governance committee evaluates if this introduces any fresh dependencies that are not on
the approved list. (Of course, any additional dependencies may be justified, in which case
39
the approved list is updated, or may even make some existing dependencies unnecessary,
which again results in updates to the approved list). The proposal may need to be
amended to avoid introducing dependencies that are not approved.
A new initiative could be on the business side (e.g., entering a new market, introducing a
new product, acquiring a company, outsourcing or re-insourcing a business function,
changing a business process, signing up a new partnership, etc.)
The initiative could also be on the technology side (e.g., buying and installing a new
product system, developing a new system in-house, outsourcing or re-insourcing a
technology function, changing the technology implementation of a function or process,
decommissioning a system or application, undertaking a turnkey project, etc.)
All of these changes have dependency implications that have to be reviewed before the
initiatives concerned can be approved. A large part of the technical and business “debt”
that is incurred by organisations is due to a failure to assess new initiatives for their
dependency implications.
The following process is meant to plug that loophole.
Scenario 1:
B: “Here's our proposal for a new project.”
G: “It appears that your project is about to eliminate some dependencies. That's good.
Project approved.”
B: “Thanks, we'll initiate the project.”
M: “We'll make a note that the set of approved (and existing) dependencies will shrink after
this project.”
Scenario 2:
B: “Here's our proposal for a new project.”
G: “Your project introduces a new set of dependencies, but they do seem to be valid, so
we'll expand the set of approved dependencies. Project approved.”
B: “Thanks, we'll initiate the project.”
M: “We'll make a note of the expanded set of approved dependencies.”
40
Scenario 3:
B: “Here's our proposal for a new project.”
G: “Your project will introduce new dependencies that are not valid. We're afraid we can't
approve it. Go back and re-think your processes and/or designs so that they stay within
the set of approved dependencies.”
B: “OK, we'll get back to you with a reworked proposal.”
M: (“Nothing for us to do.”)
Incorporating a Dependency Focus into New Initiative Appraisal
The business case for any program of work should include its impact on dependencies
(positive or negative), in addition to the standard calculations of cost-to-implement and
business benefit. Dependencies carry a hidden yet heavy cost in terms of business agility,
operating cost and operational risk, so we would like to have these made explicit and
brought to the notice of approving authorities. Introducing dependencies (legitimate or not)
incurs cost and risk, and removing them has the opposite effect. This is where the work
done by governance committees in identifying dependencies will form a useful input. The
following page has a simplified form that illustrates this.
41
New Initiative Appraisal Form
Cost Business Benefit
(3 Yrs)
Dependency impacts (+/-)
Description of proposed project CapEx OpEx
(3 Yrs)
Business agility
(3 Yrs)
Operating costs
(3 Yrs)
Operational risk
assessment
Remove replication of customer
address data by consolidating it
within system X where it belongs
(200,000) (25,000)
(25,000)
(25,000)
90,000
90,000
90,000
50,000
50,000
50,000
80,000
80,000
80,000
Changes from
medium to low
Approved by:
Approval date:
Fig. 15 – An indicative format for a “New Initiative Appraisal Form”
The form above illustrates how the hidden benefits from the removal of dependencies (or conversely, the hidden costs of introducing
fresh dependencies), once quantified, could sway the business case for a project either way. In this example, the business case for a
proposed project doesn't by itself stack up because the total business benefit over a 3 year ROI horizon ($270,000) is less than the cost
incurred ($275,000). However, the benefit goes up enormously once the improvements to the organisation's agility, cost and risk profile
are taken into account. A dependency focus therefore drives behaviour that is desirable over the longer term.
This is the kind of change we would like to see in the way organisations evaluate programs of work. It represents a “least-dependency”
model of doing business which keeps them lean and low-risk.
Processes in Steady State
The recurring processes around dependency review followed by remediation are expected
to wind down as the culture of dependency-sensitivity begins to take hold and the number
of exceptions correspondingly reduces.
In steady state, the only processes that we are likely to see are New Initiative Proposal,
New Initiative Review and BAU Program Management, since these processes will
subsume the dependency-related activities that are the sole focus of the other processes.
Periodic Dependency Reviews will still take place, but with any luck, no deviations will
have occurred and hence no remediation programs will need to be initiated.
43
Management FunctionsGovernance Functions
NewInitiativeProposal
BAUProgram
Management
NewInitiativeReview
Update: List of Approved Dependencies
Initiative Details
Non-approval:List of Dependencies in violation
Approval
BusinessProjectcycle
Fig. 16 – Ideal Dependency Governance and Management Processes in “Steady State”
Dependency-related Process Existing Business Process
BusinessDriver
(Objective)
Business Requirements
Governance and Management Check-lists
With our governance/management bodies and processes in place, it's time to look at what
they will actually do. Here is a recommended approach based on a detailed analysis of
organisational layers along with a sample check-list of questions for the committees to ask.
Classification of Dependencies
Consider a set of dependencies between entities within a BAIT layer (or across layers).
These dependencies may be further categorised into (say) two broad groups because they
are themselves related in some way. A governance or management committee may then
assign responsibility for these two dependency groups to two separate teams.
44
Entity 2Entity 3Entity 4dependency 2dependency 3
Entity 1
Entity 5
dependency 1
dependency 4
Fig. 17 – An unclassified set of dependencies
Entity 2Entity 3Entity 4
Dependency Group 1(covers dependencies 1 and 2)
dependency 2dependency 3
Dependency Group 2(covers dependencies 3 and 4)
Entity 1
Entity 5
dependency 1
dependency 4
Fig. 18 – Dependency Groups containing cohesive sets of dependencies
The Dependency Governance Committee(s) and Dependency Management Committee(s)
may want to organise themselves into specialised teams to study and monitor smaller,
cohesive sets of dependencies. We will call this intermediate level of granularity (i.e.,
somewhere between a single dependency and all the dependencies that may exist at a
BAIT layer) a “Dependency Group”. Although not identified in TOGAF, a Dependency
Group helps to provide a “system” view of related dependencies.
In the following sections, we will document not just the set of dependencies suggested by
TOGAF at each BAIT layer, but also our suggested classification of these dependencies
into Dependency Groups that can be assigned to specialised teams as their responsibility.
The List of Check-Lists
Our check-lists will be numbered according to the following scheme:
G(overnance) M(anagement)
Basic G-00 M-00
E(nterprise) EG-00 EM-00
B(usiness) BG-00 BM-00
A(pplication) AG-00 AM-00
I(nformation) IG-00 IM-00
T(echnology) TG-00 TM-00
A Basic Governance Check-list
Regardless of the level of the BAIT model at which governance is sought to be applied,
there are some standard questions that will need to be asked:
G-01 What are the core entities involved?
G-02 What are the legitimate dependencies between them?
G-03 What dependencies can we identify in the current situation that fall outside this
set of legitimate dependencies?
There will of course be more specialised governance questions depending on the BAIT
layer concerned, and we will cover them under the appropriate sections.
45
A Basic Management Check-list
In similar fashion, there are some standard management questions that need to be asked
at every level of the BAIT model:
M-01 How do we express the dependencies between entities as suitable contracts?
M-02 How will we enforce contracts?
M-03 How will we monitor adherence to contracts?
M-04 How do we re-engineer our existing systems to align them with the target model?
Every BAIT layer will add its own specialised management questions to this list, and we
will examine each under its corresponding section.
When an organisation's governance and management committees are staffed with
appropriate experts and when they adapt to the dependency-oriented way of thinking, they
will be able to formulate more relevant questions to add to these check-lists.
46
Fundamental Enterprise Dependencies
Before we even go into the four BAIT layers, we need to understand some fundamental
dependencies at the overall organisational level as illustrated below. The reason why the
“enterprise” has been split from even the business layer is that most organisations are
highly silo-ed at the business unit level (from immediately below the CEO). Hence
enterprise concerns need to be addressed at a level higher than those of business units.
The Dependency Governance Committee at this level is necessarily high-powered. It
should consist of the Head of Architecture and the CIO engaging with the CEO and
executive management. This is the only body that can authoritatively decide on the
legitimate dependencies at the enterprise level.
47
ExternalEntity
ArchitecturalPrinciple
BusinessPrinciple
EnterpriseStakeholder
Core Roles
Value Chain
Policy Framework
external dependencies
responsibility assignment
governing principles
Goal
Statement of Purpose
vision and missionalignment
Fig. 19 – Dependencies and Dependency Groups at the Enterprise level (TOGAF)
The Dependency Governance Committee should review these dependencies on a periodic
basis, perhaps half-yearly. Any deviations in practice should be referred to a Dependency
Management Committee which may consist of the executive management and their direct
reports. The Dependency Management Committee will be responsible for bringing the
organisation back into alignment with the approved set of Enterprise dependencies.
Enterprise Dependency Check-lists
The four broad Dependency Groups at the fundamental Enterprise level are as below, and
different skill groups may be engaged to deal with each:
1. The Statement of Purpose associates the enterprise with its goals through vision and
mission statements. The vision is the enterprise's opinion of what its Utopia would look
like. The mission defines the role the enterprise asserts for itself to bring about that vision
of Utopia.
Sample Governance Questions:
EG-01 What is the Enterprise's concept of Utopia?
EG-02 What is the Enterprise's role in bringing about that vision of Utopia?
Sample Management Questions:
EM-01 How do we justify our concept of Utopia?
EM-02 How do we communicate our concept of Utopia?
EM-03 How do we execute our self-appointed role? (The answer to this question creates
the foundation of the business architecture.)
2. The Value Chain describes the dependencies between the enterprise and important
external entities.
Sample Governance Questions:
EG-03 What are the dependencies that external entities have on the enterprise? (This is
the reverse of the basic governance question of what dependencies the
enterprise has on external entities.)
48
EG-04 What alternative external entities can feasibly replace our current ones without
impact?
EG-05 What abstract external entities do we have a dependency on (e.g., the economy,
the regulatory environment)?
EG-06 What external entities may also be considered partially internal (e.g., head office,
subsidiaries)?
EG-07 What time-bound contractual obligations currently exist (that can be
re-negotiated at an appropriate juncture)?
Sample Management Questions:
EM-04 How do we reduce the leverage that external entities have over us?
EM-05 How do we increase the leverage that we have over external entities?
3. Core Roles describe how responsibilities are assigned to stakeholders within the
enterprise, usually through a RACI/RASCI17 model.
Sample Governance Questions:
EG-08 What gaps currently exist in the set of stakeholders in terms of responsibility
assignments?
EG-09 What overlaps currently exist in the set of stakeholders in terms of responsibility
assignments?
EG-10 What set of stakeholders (in terms of responsibility assignments) best covers the
requirement, including concerns like key person risk, dual controls, oversight,
succession planning, etc.?
EG-11 What is the alignment between stakeholders and enterprise goals (when viewed
through the prism of responsibility assignment)?
17 RASCI – R(esponsible), A(ccountable), S(upports)/S(ponsors), C(onsulted), I(nformed). This is a model of
responsibility assignment that usually takes the form of a matrix and visually displays the coverage of
responsibilities across roles. There are many variants of this.
49
Sample Management Questions:
EM-06 How do we describe stakeholder responsibilities for the purposes of hiring as
well as performance measurement?
EM-07 How do we measure the performance of stakeholders towards the achievement
of goals?
EM-08 How do we detect and compensate for failures to adhere to assigned
responsibilities?
4. The Policy Framework defines a set of principles or values that will guide how the
enterprise will set about meeting its goals. These cover both business principles (e.g., “we
will never create a conflict between our distributor network and our direct sales channels”)
and architectural principles (e.g., “Our policy towards acquiring new capability is to reuse
before we buy, and to buy before we build”).
Sample Governance Questions:
EG-12 What is the priority to be applied to policies in case of a conflict?
EG-13 What specific stakeholder roles will be responsible for formulating and
maintaining policies of various kinds?
Sample Management Questions:
EM-09 How do we measure and monitor adherence to policies?
EM-10 How do we determine if a policy is outdated or not applicable in a given
situation?
EM-11 How do we intimate the responsible stakeholders of a possible need to modify
policies when the situation demands?
The fundamental enterprise dependencies need to be studied with great care by executive
management (aided by the CIO and the Head of Architecture). These dependencies and
the way the enterprise decides to manage them will determine the shape of the business
layer as well as all lower layers.
50
Business Layer Dependencies
The Business Layer is really where the shape of the organisation is decided, in a manner
of speaking. Here is where the business intent in terms of the vision and mission
statements is analysed for its implications in terms of the functions, processes and process
steps (operations) that will need to be implemented.
Once the Business Architecture is done, the rest is a matter of placing related entities
together, housing them somewhere and executing codified logic. That's really all there is to
it, although much of the Sturm und Drang in any typical organisation seems to happen at
these lower layers.
Modelling the Business with Domain-Driven Design
Our recommended way to approach the business, application, information and technology
layers is to use the method popularised by Eric Evans – Domain-Driven Design (DDD),
an all-encompassing methodology that is a joint venture between business and
technology.
DDD introduces another aspect that traditional SOA practice doesn't seem to consider, –
the need to develop a “ubiquitous language” to describe the domain, and to unite both
business and technology realms through the use of this ubiquitous language. At one
stroke, the ubiquitous language defines both the high-level data model and high-level
process model for all the business functions of the enterprise. Using a quick first pass of
DDD, it is possible to make reasonably insightful decisions on the right entities of each
type that should exist at the business level.
The logical progression in modelling the business is as follows:
Vision => Mission => Business Functions => Processes => Process Steps (Operations)
The types of entities at the business layer and their relationships are illustrated in the
figure above. DDD helps to determine the appropriate instances of each type.
51
Realism around Reuse
One of the much-ballyhooed benefits of SOA is the notion of reuse, although you will
notice that this white paper has been very understated about it. True, we have talked about
improved agility and reduced operating cost, both of which are well-served if business
logic can be reused rather than reinvented. But reuse itself is not a first-class benefit. It is
a means to an end rather than an end in itself.
We do need to understand the nature of reuse and the limits to which we can reasonably
expect to reuse operations, because traditional SOA practice has created unrealistic
expectations about reuse that are often belied.
The Business Layer is where the potential for reuse of operations first suggests itself. The
determination of the right processes to support defined business functions, and the further
decomposition of processes into steps or operations, will often yield sets of operations that
are remarkably similar in their function. Common sense would suggest reusing such
operations rather than creating multiple versions of essentially the same operation.
However, this suggestion is based on very high-level contours of the operations in
question. The potential for reuse needs to be assessed more critically at the Application
Layer and Information (Data) Layer before it can be turned into implementation reality at
the Technology Layer.
The reason for our seeming pessimism is that operations that appear the same may not in
fact be reusable in different contexts. It is not the operation itself that determines the
feasibility of reuse. We will see that the internal and interface data models create a specific
context for an operation. Unless the various contexts for an operation can be readily
abstracted through mechanisms like the judicious use of common super-types, reuse will
not be feasible. So although we have seen a fair degree of functional reuse in our own
experience, it is best not to burden an initiative with overly high expectations. Reuse is not
a guaranteeable metric.
The following diagrams illustrate this situation.
52
Fig. 20 – An analysis of the operations (process steps) that follow from the identification
of the organisation's vision, mission and business functions in the Business Layer may
suggest candidates for reuse. However, this is only a tentative identification.
Fig. 21 – After analysis at the Application and Information (Data) Layers, some of these
operations may be confirmed to be reusable, while others turn out to be too different in
their contexts to be reused, in spite of superficial functional similarities
53
The fundamental business layer dependencies to be understood are illustrated below.
54
Goal
Organisational Unit
Business Function
Process
Process Step(Operation)
Data
Technology Component
Location
Role Actor
Event
Business Footprint
Organisation Decomposition
Use Cases and Process Flows
vision
mission
placement
identity
behaviour
logical realisation
coordination
data life-cycle
implementation
triggers
interaction
Fig. 22 – Dependencies and Dependency Groups at the Business Layer (TOGAF)
Business Layer Dependency Check-lists
There are three broad Dependency Groups that emerge at the business layer, and they
are discussed below.
1. The Business Footprint is a related set of entity dependencies that acts as a
traceability tool to map how the enterprise achieves its goals through every layer from the
business down to technology. It describes the responsibility of each organisational unit
towards achieving the vision of the enterprise, and the organisational unit's business
functions that are derived from its vision statement. It also follows the business functions
through to the level of processes and the logical steps (operations) that these processes
coordinate. [At the lowest levels, a fully fleshed-out business footprint diagram shows what
data elements are involved in an operation and what technology components implement
these functions and process/store this data. However, at the business layer, this level of
detail is neither available nor necessary to consider. Work needs to be done at the data
and technology layers to flesh out these aspects of the business footprint diagram.]
The Business Footprint Diagram in the TOGAF model is one of the most important
descriptions of how the organisation functions. It is usually not a single physical diagram
but a logical one spread across multiple physical ones, and requires a multi-disciplinary
team across multiple business units to fully document.
Although the creation of the Business Footprint Diagram is very tedious and difficult, there
are rich rewards to be enjoyed once it is created. This comes closest to a “control panel” or
“dashboard” for the enterprise, because it is possible to drill up or down to any level to
understand the implications of a change or disruption to all related systems. It is also the
foundation for the application, information and technology layer models.
Sample Governance Questions:
BG-01 What is the extent of functional duplication at each level?
BG-02 What is the abstraction hierarchy for entities that describes how a single logical
function gets specialised into multiple concrete variants? (e.g., an abstract
Customer entity that has Corporate Customer and Retail Customer as its
subtypes forms the basis for postulating common functions such as Registering a
New Customer, which will necessarily have specialised variants for its concrete
subtypes.)
55
BG-03 What are the factors that drive differentiation between the multiple concrete
variants of a single logical function?
Sample Management Questions:
BM-01 How do we plan programs of work at every level to align our current footprint to
the ideal one? (This is a high-level question that will spawn more detailed
questions as one drills further down.)
2. Organisational Decomposition shows how organisational units are distributed in
various geographical locations. It decouples identity from behaviour through its definition of
actors and roles, and shows how roles relate to business functions as well as to
organisational units and locations.
Sample Governance Questions:
BG-04 What is the organisation structure that best reflects the functional make-up of the
organisation? What level of centralisation/federation is appropriate?
BG-05 What constraints govern the placement of roles at particular locations?
BG-06 What roles required for business functions require to be within the related
business unit and what roles require to be outside of it?
Sample Management Questions:
BM-02 How should resources be assigned to locations (i.e., relocation/rotation policy)?
3. Use Cases and Process Flows describe how processes and services interact with
roles as well as with events.
Sample Governance Questions:
BG-07 What are the most common event triggers and role interactions?
BG-08 What exception use cases and process flows are important?
56
Sample Management Questions:
BM-03 How do we best cater to high volume interactions?
BM-04 How do we deal with time-critical events?
BM-05 How do we deal with exceptions?
57
Application Layer Dependencies
The business layer that we just considered identifies the business intent – the enterprise's
vision and mission – and the resulting high-level business functions, their constituent
processes, and the individual steps (operations) that make up these processes. But
processes and steps of processes don't float disembodied in an organisational function
“soup”. They are usually grouped by functional domain, and in the Technology Layer, each
group is housed in an appropriate “container” under the care of suitable custodians. To
complicate matters, the “containers” of these groups of operations have life-cycles and
dependencies of their own which demand to be managed.
One of the impediments to truly understanding the Application and Information (Data)
Layers of an organisation is our tendency to be distracted by how business logic and data
are implemented. In other words, the Technology Layer crowds out the Application and
Information (Data) Layers in our minds. What is known as the “As-Is” architecture of an
organisation also tends to fuse these three important layers into monolithic
technology-based entities, because these are what we can see and touch, so to speak.
To be successful in SOA thinking, we need to suspend for a while our awareness of the
physical landscape of hardware and software, i.e., how business logic and business data
are implemented, and step back to first see how they should be organised for the optimal
functioning of the enterprise. With this understanding, we can then begin to see our “As-Is”
landscape with fresh eyes and gain insights into the dependencies that exist at the
Application, Information (Data) and Technology layers.
Our focus at the Application and Information (Data) Layers will be on how to organise
business logic and business data, respectively. Our focus when we get to the Technology
Layer will then be on how to implement the logic and data that have been so organised.
Funnily enough, looking at the Technology Layer and the way business logic and data are
implemented gives us insights into how they are typically organised at these higher layers.
A generic name for a group of related operations housed in a container is “application”, but
this is a relatively abstract term. There are two concrete types of application that we
usually need to deal with. They are “products” and “services”.
While products and services, being ways to implement business logic, are Technology
Layer entities, their distinctive approaches to organising business logic begin at the
Application Layer.
58
Products are containers that implement pre-defined processes and their constituent steps
or operations, encapsulating them from external view except through native interfaces.
Products can be thought of as “black box” applications.
Services are containers for groups of operations that are individually exposed to external
systems (as opposed to being encapsulated and only accessible as part of pre-defined
processes). Services can be considered “glass box” applications.
We saw during our discussion of the business layer that DDD's ubiquitous language
defines both the high-level data model and high-level process model for all the business
functions of the enterprise. The decisions on how operations should “clump” together to
form products and services, and how these should then interact with each other to
exchange information are the key organising decisions at the Application and Information
(Data) Layers.
Cohesion and Coupling
Our recommended approach to making these organising decisions is to apply the
dependency principles of “cohesion” and “coupling”. These terms were first introduced in
“Structured Design”, a 1974 paper by WP Stevens, GJ Meyers and LL Constantine.
“Cohesion refers to how closely all the routines in a class or all the code in a routine
support a central purpose” - Steve McConnell
59
Service
Ope
ratio
n
Ope
ratio
n
Ope
ratio
n
Ope
ratio
n
Ope
ratio
n
Ope
ratio
n
Product
Fig. 23 – In a Service, all constituent operations are individually visible and available to compose into external processes. In a Product, processes are internal, “baked in” and
not changeable, and not all operations are externally visible. Some Products may consciously expose a number of their constituent operations to enable new, external
processes to be built from them, and these are more “service-oriented”.
External process External process
In other words, things that “belong” together should go together. When related functions
(to use the term loosely) are found together, they are said to be highly cohesive. High
cohesion is a desirable feature, but as we will see, there are different criteria that are
simultaneously possible that could result in how functions are grouped.
At the Technology Level, the cohesion of related functions may not be as high as it should
be. This is either because products are often purchased off the shelf, and their functional
boundaries do not cleanly mesh together within the organisation's ecosystem, or because
in-house built products and services were not architected with a consistent vision of
functional boundaries and data models. The gap between the ideal and the actual is what
needs to be remediated at the Technology Layer, but any such remediation requires an
understanding of what belongs together at the Application Layer. That's why the
Application Layer is so important. A well-designed Application Layer that is used to make
decisions on Technology artifacts is like a system of satellites providing GPS navigation to
cars at the street level. Without such guidance, we can get hopelessly lost in the maze and
chaos of the corporate IT network.
The governance function at the Application Layer deals with identifying the deviations in
the As-Is architecture from an ideal cohesion-based application grouping. The
management function at the Application Layer deals with formulating plans to remediate
these deviations, and these plans then translate into appropriate pieces of work at the
Technology Layer.
There is a concept related to “cohesion” called “coupling”, which we will look at briefly now
but cover in detail in the next section.
"If two modules communicate, they should exchange as little information as
possible" - Bertrand Meyer
The perceptive will note that this is a prescription for minimal Data Layer dependencies
between two systems. When that is ensured, the systems are said to have low coupling.
As we know from our extensive case studies on dependencies, low coupling is a highly
desirable feature, just like high cohesion, and the two concepts are complementary. That's
why the design of the Application Layer and Information (Data) Layer often go together.
Design at the Application Layer is guided by the high cohesion principle. Design at the
Information (Data) Layer, especially the design of data exposed between applications
through interfaces, is guided by the low coupling principle.
60
This section will mainly focus on application design, and hence will leverage the “high
cohesion” principle. The next section will focus on information (data) design, and will
therefore leverage the “low coupling” principle.
To internalise the concept of cohesion, let's do a fun exercise. We have a map showing a
set of islands in the Pacific Ocean, and there's nothing to distinguish them from each other
at first glance. The objective is to group them into distinct collections such that each
collection of islands has something in common that sets it apart from the others.
Try it yourself first before flipping forward to see some possible answers.
61
A Fun Exercise in Applying the High Cohesion Principle
Fig. 24 – Group these Pacific islands into appropriate collections!
Fig. 25 – An arbitrary, random and whimsical attempt to group these islands with no rigorous basis at all
Fig. 26 – A cultural basis for grouping the islands, resulting in groups that are each internally cohesive and distinct from the others.
Source: http://en.wikipedia.org/wiki/File:Pacific_Culture_Areas.jpg
Fig. 27 – A possible geopolitical basis for grouping islands (circa 2030?). Such a classification could coexist with the cultural one.
The key takeaway from this exercise is that the criteria for cohesion are critically important
in determining the groups that subsequently emerge. If we apply a cultural criterion, then
we end up with the Polynesian, Melanesian and Micronesian island groups. If we apply a
hypothetical geopolitical criterion, we end up with spheres of influence of Pacific powers.
Criteria for High Cohesion
Returning to SOA and to the point where the business layer leaves off, a key question is
what criteria must be used to evaluate the natural cohesion of operations.
There are at least two ways in which operations can be grouped, but before we discuss
what these are, let's look at an analogous example to ease our understanding.
A matrix organisation groups people in two different ways at the same time. The functional
or “straight line” reporting groups define their job roles, and this is one criterion for
cohesion. But on projects, multi-disciplinary teams need to be put together, with one or a
few individuals from each of these functional groups. Project teams are united for a period
of time in their pursuit of a common goal, and team members have “dotted lines” reporting
to the project owner until the project completes, the team is disbanded and members are
reassigned to other projects. Hence project goal is a second criterion for cohesion.
66
Person
Project Mgrs
Person
Person
Person
Person
Architects
Person
Person
Person
Person
Developers
Person
Person
Person
Person
Testers
Person
Person
Person
Project 1
Project 2
Project 3
Project 4
Functional groupGoal-oriented team
Fig. 28 – A matrix organisation groups people simultaneously in two different ways
In very similar fashion, the atomic “operations” that are the functional units of a business
can be put together in at least two ways. The business layer has already applied a
goal-oriented cohesion criterion to group operations into processes. But another criterion
for cohesion is the “domain”, or subject area, to which the operations pertain. Grouping
operations according to a domain-oriented cohesion criterion creates “applications”. The
determination of the “right” domains for a business is by no means always obvious. Does a
“handle customer payment” operation belong to the “Customer” domain or to the
“Payment” domain? Either classification may work. As we saw with the islands example,
the decision on what the right criteria (i.e., domains) are is a crucial governance question
at the application layer.
So to recapitulate, when operations are grouped together by goal, they form Processes.
When operations are grouped together by “domain”, they form Applications. Applications
that hide the way operations are put together internally are called Products. Applications
that expose individual operations are called Services. And this tells us the true place of the
Service concept within Service-Oriented Architecture, - a static way to group externally
exposed operations based on their functional domain.
67
Operation
CustomerDomain
Operation
Operation
Operation
Operation
ProductDomain
Operation
Operation
Operation
Operation
XYZDomain
Operation
Operation
Operation
Operation
PQRDomain
Operation
Operation
Operation
BusinessProcess 1
Business Process 2
Business Process 3
Business Process 4
Application (Product or Service)
Process
Fig. 29 – Operations can also be grouped simultaneously in two different ways
Process, Product and Service
Here's a more detailed diagram that shows a Process, a Product and a Service together
with their related components. We also get a sneak preview into elements of the
Information (Data) Layer because business logic and business data cannot be isolated.
In all cases, there is an internal data model that is manipulated by operations. When
operations form part of Services, they can be accessed from external systems, and as part
of this interaction, data is exchanged between the external systems and the operations.
This is the interface data model. Both the interface data model and the internal data model
tend to conform to the same “data dictionary”, that of the domain to which the operations
68
PublicProcess
Internal Data Model
Public Process Step
(Operation)
manipulate
Process
Interface Data Model
Internal Data Model
Internal Process Step(Operation)
InternalProcess
orchestrate
exposeexternally
manipulate
orchestrate
chain/nest
nest
proxy, aggregate
Service(“Glass box” Application)
Product(“Black box” Application)
Native UserInterface
exposeexternally
exposeinternally
Domain Data Dictionary
conform to
conform to
loosely conform to
Fig. 30 – Process, Product, Service and related Entities
belong, but there is no tight relationship between them. We will see why when we study
the Information (Data) Layer in greater detail.
In contrast, when operations form part of Products (as is the case with off-the-shelf
software packages), many business processes come pre-built as part of the product's
functionality. The internal data model may not strictly conform to any domain data
dictionary that the procuring enterprise recognises. Also, since the operations within the
Product may not be accessible individually from the outside, there may only be some
visual means of interacting with external parties, and there is no formal interface data
model to speak of. Some products may expose a subset of their operations to external
systems the way Services do, so this is a hybrid model between Service and Product.
In other words, the main difference between a Product and a Service is in the extent of
visibility and composability of its constituent operations.
The “internal data model” and the “interface data model”, as well as their relationship to a
domain-specific “data dictionary”, are topics that we will cover in the next section on the
Data Layer.
With this lengthy conceptual introduction, it is time to look at the fundamental
dependencies that exist at the Application Layer.
69
Application Layer Dependency Check-lists
The application layer dependencies as derived from TOGAF are illustrated below. Note
that the TOGAF model of the application layer includes physical application components,
i.e., the implementation of business logic or what is properly the Technology Layer.
There are two broad Dependency Groups that are apparent at the application layer.
1. The Functional Model picks up the business footprint model from the business layer
and applies domain cohesion to determine which business functions belong together as
70
Organisational Unit
Business Function
Process
Process Step (Operation)
Location
Role
Actor
Logical AppComponent(Product or
Service)
logical interaction
Physical AppComponent(Software
Implementation)
physical communication,
manageability
distribution
ownership, usage
usage
decomposition
realisation
Gaps and overlaps, migration plan
ownership, usage
realisation
responsibility(functional domains)
Functional Model
Application Portfolio(strictly speaking a Technology Layer Dependency Group)
Fig. 31 – Dependencies and Dependency Groups at the Application Layer (TOGAF)
logical application components (products and services). It then defines the ownership and
usage relationship of these logical application components to actors/roles as well as to
organisational units. Logical application components may also interact and have
dependencies on one another.
Sample Governance Questions:
AG-01 What criteria must be used to evaluate the natural cohesion of operations (i.e.,
responsibility for business functions)?
AG-02 (Hence) What are the appropriate functional domains into which the enterprise's
process steps (operations) should be classified?
AG-03 What is the right mapping that should exist between business functions, logical
applications (products/services) and implementations? What is the mapping that
currently exists?
AG-04 What other dependencies govern applications (other than domain-based
cohesion)?
AG-05 What interaction models are appropriate between applications? (Real-time,
asynchronous, batch, etc., also invocation versus event-based mechanisms)
AG-06 What are the operations that need to be exposed (through services) to enable
processes to be composed from them?
AG-07 What are the dependencies that exist between the applications we have defined?
(These should be minimal, and expressible in terms of exposed operations
(services) alone.)
Sample Management Questions:
AM-01 How can we ensure optimal coverage of the required business functions using
our applications (products and services)? (Future state Application Layer
architecture)
AM-02 How best can we procure, develop, test, deploy and maintain our products and
services? (Roadmap)
2. The Application Portfolio shows the “physical” implementations of products and
systems that exist (or should exist) in the enterprise. As mentioned, this is strictly speaking
71
a Technology Layer Dependency Group, but it may make logistical sense to have the
same governance and management committees look at both the logical cohesion of
business operations as well as their ultimate physical implementation. Physical application
components are a realisation of their logical counterparts and execute processes and their
constituent steps. Physical application components are distributed geographically among
various locations and are used by various actors and roles.
Sample Governance Questions:
AG-08 What product implementations exist in our environment? What is missing or
superfluous (gaps and overlaps)?
AG-09 What service implementations exist in our environment? What is missing or
superfluous (gaps and overlaps)?
AG-10 What is our policy around new functionality – buy or build? (The answer comes
from the enterprise dependency governance committee.)
Sample Management Questions:
AM-03 How do we migrate from the current implementation landscape to the target
state?
We now understand that operations can be grouped in at least two different ways (i.e., by
goal and by domain) to form Processes and Applications, respectively, and that the two
domain-based groupings (Product and Service) pertain to encapsulated groups of
operations and to externally visible operations, respectively.
Both Products and Services manipulate data conforming to an internal data model.
Services additionally expose an external data model through its operations. The external
and internal data models are loosely correlated, and a domain-specific “data dictionary”
may be postulated to provide a common foundation to all Products and Service operations
within a domain.
We are now ready to move down to the Information (Data) Layer and study the related
concepts of “Data on the Inside” and “Data on the Outside”.
72
Information (Data) Layer Dependencies
Over the last two sections, we have analysed not only the business intent and all the
processes and operations that arise as a consequence, but also the set of dependencies
characterised as cohesion that suggested how best to clump operations into “applications”.
We now need to understand how these applications will interact, specifically how they will
exchange information. And this brings us to the dependency principle called coupling:
“If two modules communicate, they should exchange as little information as
possible” - Bertrand Meyer
At the outset, let us point out that while TOGAF goes into a fair amount of detail at the
Information (Data) layer to analyse entities and their relationships, it does not distinguish
between “Data on the Outside” and “Data on the Inside”18. This distinction, made most
eloquently by Pat Helland in his seminal paper19, is crucial from a coupling perspective.
Data on the Outside versus Data on the Inside
Applications develop dependencies on the data that they can “see”. “Data on the Inside” is
well encapsulated inside an application and the application's own dependency on it has
limited impact, because a change to an application's internal data does not impact other
applications. In contrast, “Data on the Outside” always has a dependency implication,
because it is shared or exchanged between applications, all of which then develop a
dependency on it. In other words, a change to shared data impacts more than one
application. Since our approach is dependency-focused, we need to split TOGAF's generic
“Data Entity” into internal and external flavours and deal with them separately.
18 Another fallout of TOGAF's view of SOA as a subset of technology and not as a pervasive organising
principle.
19 “Data on the Outside vs. Data on the Inside”, Pat Helland (http://bit.ly/RHj7dp)
73
Application (Service/Product) BApplication (Service/Product) A
Operation a1 Operation b1Internal Data
Model (A)Internal Data
Model (B)
Interface Data Model
Fig. 32 – The Internal Data Model of each application corresponds to “Data on the Inside”, and the Interface Data Model that they share corresponds to “Data on the Outside”.
Aspects of Low Coupling
There are two levels of dependency that we must concern ourselves with at the
Information (Data) Layer. One is the dependency that applications develop on each other,
or more correctly, the dependency they develop on their shared interface data model. The
other is the dependency between any given application's internal data model and an
interface data model that it shares with other applications.
It's very important to recognise that the dependency of an application on any data that it
“sees” can be quite significant, with negative impacts to agility, cost and risk. So our
strategies to reduce dependency should be as follows:
1. To reduce the dependency of an application on an interface data model, we follow a
minimalist strategy. In other words, if an operation needs two data elements in order
to do its job, then we need to ensure that no more than those two data elements are
in the interface data model. The smaller the “radar cross-section” of an operation,
the better.
2. To reduce the dependency of an application's internal data model on its interface
data model and vice-versa, we employ a mediating strategy. In other words, we
make both these models rely on a third model rather than on each other. This point
is worth emphasising because too many system designs assume that the internal
data model and the interface data model are one and the same! One can imagine
the kinds of dependencies that this assumption can lead to.
74
Application (Service/Product) BApplication (Service/Product) A
Operation a1 Operation b1Internal Data
Model (A)Internal Data
Model (B)
Interface Data Model
Fig. 33 – Two levels of dependency at the Information (Data) Layer
Dependency between applications through their Interface Data Model
Dependency between an application's Internal Data Model and its Interface Data Model
Dependency between an application's Internal Data Model and its Interface Data Model
1
2 2
A good way to recognise when this is happening is to ask ourselves if external systems are
likely to be impacted when an application's internal data model changes. If they are, then it
means the application's internal data model is tightly coupled to its interface data model
and the link needs to be broken. We need to introduce a mediating entity.
The mediating entity for “Data on the Inside” and “Data on the Outside” is a common “data
dictionary”, so the two aren't entirely unrelated. That's why we don't say the two models
are decoupled, but merely loosely-coupled. The data dictionary is their only link.
The Domain-specific Data Dictionary
Every domain has a data dictionary, i.e., a model that describes the data of that domain in
a comprehensive, consistent and definitive way. The loose coupling between a domain's
internal data model and its interface data model stems from the fact that both of these are
based on the same data dictionary and share some data structures, but nothing more20.
20 This approach is an application of the Venetian Blind pattern (http://bit.ly/UXFqL2), with types defined in
the data dictionary and elements defined in the internal and interface data models.
75
Application (Service/Product) BApplication (Service/Product) A
Operation a1 Operation b1Internal Data
Model (A)Internal Data
Model (B)
Internal Data Model (A)
Fig. 34 – What if an application's internal and interface data models are the same?
Consequently, dependency between Application B and Application A is extremely high, and the integration between them is brittle
Dependency between Application A's Internal Data Model and its Interface Data Model is total, because they are one and the same!
Fig. 35 – The data dictionary for a domain (type hierarchy and composition)
A data dictionary defines types. There are two fundamental types – entities and value
objects21. Entities are persistent and have identifiers. Value objects are anonymous. They
have structure but no identity. They are only persisted when they form part of an entity.
A data dictionary is restricted to defining only type hierarchies and composition
relationships. The reasons for this will become clearer when we look at how the data
dictionary is used by an application's internal data model and its interface data model.
[In terms of the Technology Layer implementation of the data dictionary, this may just be a
human-readable document that is used to guide the design of the other two data models. It
could also take the form of an XML schema document with type definitions,]
The Internal Data Model (“Data on the Inside”)
The internal data model for an application leverages its domain-specific data dictionary
and models the various relationships between entities. Some specialised inheritance
relationships peculiar to the application may also be modelled.
[In terms of its Technology Layer implementation, the internal data model could be
object-oriented or relational (the most common), but its elements should conform to the
domain data dictionary.]
21 As follows from the Domain-Driven Design approach.
76
Fig. 36 – Internal Data Model (entity association, aggregation and instantiation)
The internal data model of an application is the one that defines how entities hang together
and what their relationships are. “Black box” applications (Products) work on an internal
data model that may or may not relate to a domain's data dictionary, because they could
very likely be off-the-shelf software packages that come with their own view of the domain.
However, “glass box” applications (Services) built in-house by an organisation would
benefit from using an agreed data dictionary for the domain when they manipulate data.
The Interface Data Model (“Data on the Outside”)
The interface data model is a different beast altogether from the internal data model. It is
not a “model” of a system in the same way that the internal data model is. It is just a loose
collection of data elements that needs to form part of a message passed between
applications. To use an analogy from C++, the internal data model is like a set of classes;
the interface data model is like a set of structs. An important constraint to keep in mind is
that interfaces need to balance the reality of change with the need for stability. We will
elaborate on this shortly.
Another important point to remember is that the name of an operation is part of its
interface data model, because a change of name is immediately visible to other systems.
[At the Technology Layer, the interface data model tends to be implemented either as an
XML schema or JSON schema document.]
77
Fig. 37 – Interface Data Model (type aggregation)
Importantly, the interface data model must not contain entity types, even though many of
the attributes of entities may be found within a message. This is because it needs to be
made crystal clear that what is being passed between applications is not a reference to a
“live” entity instance but merely an immutable representation of it. Any identifier that
accompanies the entity representation should be a specially-created “external identifier”
and not the internal identifier of the entity within the application. Applications should use
the mapping between external and internal entity identifiers to determine which entity
instance a given representation pertains to. This association of identifiers is a key aspect
of loose coupling at the Data Layer, because any visibility of internal identifiers externally
creates a dependency that should not exist.
The interface data model defines messages that flow between applications when
operations are invoked, so the main element is a message type. Various types of
messages can be defined, and each of these is defined as an aggregation of other value
types. Why is the value type a super-type rather than a sub-type? The dictionary defines
both super-types and sub-types, doesn't it? Wouldn't a sub-type be a better fit? Let's see.
Message Data “on the wire”
Consider the data that applications actually exchange “over the wire” when they interact.
For this, we need to skip ahead for a while to the Technology Layer to look at an
implementation, e.g., an XML document. An XML document is expected to conform to an
XML schema that describes it., but how well (i.e., how tightly) should the schema describe
the document?
78
Let's postulate a message data model that precisely describes a message, e.g., an XML
schema that is highly tailored to the XML document that goes over the wire. What is the
difference between this message data model and the interface data model? Shouldn't they
be exactly the same?
The answer is no, and the reason relates to the point we made earlier - interfaces need to
balance the reality of change with the need for stability. There is a crucial element to
message design that leverages the data dictionary to provide a sufficiently low level of
coupling between applications, but not too low.
The message data model for any message document exchanged between applications
should look like this.
Fig. 38 – Message Data Model (instantiation of value subtypes)
79
The interface data model is shown circled in red, and there are one or more levels of
subtypes between the interface data model and the actual (instance) elements of the
message document.
Recall that the data dictionary defines the type hierarchy (supertypes and subtypes) for all
the data elements in the domain but does not define the elements themselves. This
approach allows the binding of types to message elements to be deferred to as late a
stage as possible.
One may expect the message data model to define the interface, since this exactly
validates a message document. But there are two drawbacks in doing this.
1. Specialised subtypes, while providing tighter definition, are more sensitive to change.
The degree of dependency between applications is higher. Any change to a subtype will
necessitate a change to the interface data model and impact other applications. Having to
support multiple concurrent versions of an interface data model is then an unavoidable
consequence of two simultaneous requirements – for change and for stability.
2. Besides, any two minor variants of a similar interface data model will require two
distinct definitions, even if they share common super-types.
In other words, high levels of version churn and proliferation of services are the natural
consequence of tight coupling to an overly detailed interface definition. This might sound
familiar to large “SOA shops”!
We can balance the need for change with the need for stability by defining interfaces in
terms of super-types. This gives us the ability to describe an interaction meaningfully but
also withstand minor changes to the data that is exchanged, which is why the interface
data model shown earlier used super-types for values rather than specialised subtypes.
The message document therefore conforms to the interface data model without being
tightly bound to any one specialised interface.
With this conceptual framework as a background to help make sense of the Information
(Data) Layer, let us now turn to look at the dependencies that exist here, and derive some
governance and management check-lists from our analysis.
80
Information (Data) Layer Dependency Check-lists
The core TOGAF-derived Information (Data) Layer dependencies are illustrated below,
with our refinement of the data model into internal and external data entities linked to a
shared data dictionary.
81
Organisational Unit
Business Function
ProcessProcess Step
(Operation)
InterfaceData Entity
InternalData Entity
Logical AppComponent
Physical AppComponent
exposed data
consumed data
Entity-Relationship Models; Class Hierarchies; Data Structures, Life-cycles and Events; Master-Replica Relationships;
Business Value of Data; Security Classification, etc.
Data Ownership and Dissemination
External Data Dependencies
Internal Data Dependencies
inputs and outputs
physical data model
logicaldata model
mission
realisation
coordination
responsibility (functional domains)
Data DictionaryElement
conformance
conformance
Fig. 39 – Dependencies and Dependency Groups at the Information (Data) Layer (TOGAF)
Three Dependency Groups can be seen at the information (data) layer.
1. Internal Data Dependencies are perhaps less important in the larger scheme of things.
The diagram lists the various types of dependencies that should be analysed.
Sample Governance Questions:
IG-01 What is the minimal data dictionary for each functional domain, as suggested by
the various internal data models of applications in that domain?
IG-02 What is the life-cycle of data items? What core events impact these life-cycles?
IG-03 What data items are mastered and where? What data items are replicated and
where?
IG-04 What metadata applies to data items (business value, security classification,
etc.)?
Sample Management Questions:
IM-01 How do we remediate the internal data models of applications to conform to the
domain's data dictionary? (In many cases, remediation is neither feasible nor
necessary. It may suffice to document the logical mapping between these models
along with exceptions and inconsistencies to aid in maintenance and functional
enhancement.)
IM-02 How do we ensure that data is kept current and relevant?
IM-03 How do we implement Master Data Management (MDM)?
IM-04 How do we leverage metadata when managing data?
2. External Data Dependencies are more critical. “Interface data entities” are loosely
associated with “internal data entities” through the application components that realise
services. Interface data entities are exposed as inputs and outputs of operations.
82
Sample Governance Questions:
IG-05 What (minimal) data dependencies should exist between groups of functional
capabilities? (This is probably one of the most important SOA questions of all, in
spite of its innocuous appearance, and therefore the one on which the
Information (Data) Governance Committee should spend a disproportionate
amount of time.)
IG-06 What is the minimal data dictionary for each functional domain, as suggested by
the various interface data models of applications in that domain?
IG-07 What are the appropriate data structures underlying interface data? (e.g., type
hierarchies for data exposed through service interfaces)
IG-08 What is our version support policy when dealing with changes to interface data
models?
Sample Management Questions:
IM-05 How do we remediate the interface data models of applications to conform to the
domain's data dictionary?
IM-06 How do we communicate the interface data model of an application to others?
How do we communicate changes to it?
IM-07 How do we deal with changes to the data dictionary that impact the interface data
models of applications? How do we deal with changes to the interface data
models of applications that necessitate changes to the data dictionary?
IM-08 How do we format data?
IM-09 How do we perform data validations (e.g., coarse-grained schema validation
against base types, fine-grained validation within implementations)?
3. Data Ownership and Dissemination traces how data entities are related to
organisational units and processes through business functions and application
components.
Sample Governance Questions:
IG-09 What business unit and business function owns a domain's data dictionary?
83
IG-10 What business unit and business function is responsible for initiating and
managing changes to the data dictionary?
Sample Management Questions:
IM-10 How do owners of a domain's data dictionary interact with other stakeholders?
IM-11 How are change requests communicated to the owners of a domain's data
dictionary? How are changes communicated back to all stakeholders?
IM-12 How are the semantics of a domain's data dictionary explained to owners of other
domains? (This is crucial when brokering or transforming messages.)
84
Technology Layer Dependencies
It has been a long road to finally arrive at what the impatient would consider to be the only
SOA layer that really matters – the Technology Layer! But hopefully, our discussions in the
previous sections have illustrated the importance of the higher layers of the enterprise.
Technology is of course important, because business logic is only useful when it is
implemented, but technology has its place in the overall scheme of things. We need to
determine how business logic and business data are logically organised before we can
start to make decisions on how they must be housed and/or implemented.
The following diagram shows how the business logic and data dictated by the enterprise
vision and mission are organised. Note that while Products are largely “black box”
applications that hide all data, operations and processes behind a native interface, the
more “service-oriented” among them could also expose some operations and related
interface data to be consumed and orchestrated by external processes. Services, on the
other hand, do not hide but explicitly make operations visible to external processes to
orchestrate, and (ideally) make a clear distinction between “Data on the Outside” (interface
data) and “Data on the Inside” (internal data).
85
ServiceProduct
Operation(s)
Native Interface
Internal Data
Interface Data
Operation(s)
Internal Data
Interface DataInterface Data
Internal Process External Process
Two types of Application
Not the
same
Usually the same
Fig. 40 – Business logic and data organised into applications and ready for implementation
Implementing Products
Implementing “black box” applications is relatively easy, and organisations have been
doing this for decades. Indeed, this is the traditional way that organisations have
implemented business logic. The only decisions here have been “buy or build?”.
In recent times, there has been a push from consumers of applications to have more of the
business logic inside a Product exposed so that it can participate in business processes
defined externally. That's why many products today feature “service APIs”. Obviously, even
with a service interface, a product's operations cannot readily be “consumed” because the
interface data model of those operations is not likely to conform to the organisation's
domain data dictionary, and considerable effort is often required to bring these in line with
the other operations of the enterprise.
Products, being monolithic and standalone applications, are boring. Worse, they're out of
style in our collaborative world! The more interesting aspect of our discussion, therefore, is
how to implement Services. With an application landscape composed mainly of Services,
the implementation of business logic should become like the assembly of LegoTM blocks.
Implementing Services
This is what we're interested in doing:
Hence this is what we want to see:
But this is often what we have:
86
Process
Service
Interface Data
Native Interface
Product
A process orchestrates multiple operations
A service exposes one or more operations
Legacy systems expose non-standard interfaces
Our challenge is therefore around building new Services and converting Products to make
them look like Services, so that they can all be composed into Processes that implement
the business functions of the organisation and help it achieve its mission and vision.
Implementing the Nouns and Verbs of the Interface Data Model
An operation can be loosely described as some actor performing some action on some
resource, using some contextual data. The Interface Data Model is the complete
description of how the operation is invoked by the actor.
At the Technology (with a capital 'T') Layer, this relatively abstract description needs to be
translated into a working implementation. The exact flavour that this implementation takes
will depend upon the technology (with a small 't'). The two common technologies used to
build Services are SOAP-based Web Services22 and REST.
When working with the SOAP approach, the verb (the action being performed) and one of
the nouns (the resource being manipulated) are usually merged into the name of the
operation (e.g., “getCustomerDetails”). The associated data that is passed and returned is
encoded in XML documents (which conform to the Interface Data Model in terms of
supertypes, as we saw earlier).
When working with REST, the resource needs to be identified with a suitable URI (e.g.,
“/customers/{id}”) and the verb is a standard one (i.e., “GET”). Associated data is passed in
the body of the HTTP request and response.
Regardless of the flavour of technology chosen, there are some standard ways to
approach the implementation exercise. The thrust of this approach is to be able to treat all
Applications as Services (even “black box” ones), and to be able to consume their
operations by orchestrating them within Processes.
22 Be aware that technologies introduce fresh dependencies of their own in the course of implementing a set
of operations. In the SOAP world, WSDL is a prime culprit. The dependencies that a WSDL file has on
service and operation definitions, and the dependencies of these, in turn, on input and output (XML)
schema files is the cause of much brittleness. A minor change to a single element of a schema file
pertaining to a single operation can ripple through and change the version of a WSDL file, impacting
many otherwise unrelated operations, services and client systems. REST introduces far fewer
dependencies of its own, with the binding to HTTP being the major one.
87
The Three Core SOA Technology Components
In a nutshell, there are just three core technology components that we need to learn about,
which will help us in our quest to turn all the applications in the enterprise into Lego TM
blocks and compose them into processes. They are the Service Container, the Broker and
the Process Coordinator.
Service Container: When business logic does not already exist and is not available off the
shelf, it has to be developed in-house and must be hosted in a way that makes it easy for
other applications and systems to invoke its functionality. That calls for a Service
Container.
We depict a Service Container visually as follows:
How do we use a Service Container? Here's how.
The figure above shows that custom functionality, when hosted within a Service Container,
is exposed as a potentially reusable service to other components in the system.
Service Containers in practice may be simple web servers exposing SOAP or REST
interfaces.
88
Fig. 41 – Notation for a Service Container
Custom logicService interface
Service Container
Service Consumer
Fig. 42 – Hosting business logic in a Service Container and exposing it through a “service interface”
Broker: The Service Container is a useful component when we need to host a particular
piece of functionality that we had to develop ourselves. In practice, we do not generally
develop complete, end-to-end solutions from the ground up. The solution we put together
will usually be able to exploit pieces of logic or data that already exist somewhere in the
organisation, within some legacy system or application, i.e., a Product. We will invariably
come up against some complications that prevent us from readily accessing and using
such components.
• Perhaps the legacy component that hosts our required logic speaks a specialised or
proprietary protocol. To use an electrical analogy, we need an adapter to let us plug
into this differently-shaped socket. An adapter can translate proprietary protocols into
more open ones.
• Perhaps the data that this legacy component provides is not in exactly the right format
that we can readily use. Again, to use an electrical analogy, we need a transformer to
step its voltage up or down. Transformers can enrich data and massage it into a form
that is more readily understood and usable. Usually, that means an Interface Data
Model that conforms to a domain's Data Dictionary.
• And perhaps there are reasons why we do not want our solution to directly access this
legacy functional component. We may want to hide, proxy or throttle access to it. An
extension block or power board lets electrical components tap into power without
directly accessing a wall socket. In the SOA context, we call such a component a
mediator.
There is something in common between the functions of an adapter, a transformer and a
mediator, and a single component can usually perform all three functions at once. We call
this combined component a Broker, and depict it visually as follows.
In practice, the role of a Broker is performed by an Enterprise Service Bus in the case of
SOAP-based web services. REST services can usually adapt to different kinds of service
consumers through content negotiation and thereby act as their own Brokers. In case
external Broker components are required for REST services, these are usually web proxy
89
Fig. 43 – Notation for a Broker
servers.
This is how we use a Broker.
90
Service Consumer
Legacy component
Proprietary interface“Adapter”
logic
“Transformer” logicService interface
(“Mediator” function)
Broker
Fig. 44 – Mediating access to a legacy “black box” component with a Broker, adapting its native interface and transforming data to create a more standardised interface data model
Process Coordinator: LegoTM blocks are great for building solutions quickly, and a Broker
can even tie together services in a rudimentary way, but sometimes we don't know the
exact steps of the logic in advance. When we need to pull components together
dynamically, based on a status that can only be evaluated at run-time, we need a Process
Coordinator. This is how we visually depict a Process Coordinator.
Here's how a Process Coordinator is used.
In a SOAP environment, Process Coordinators are BPEL engines. The REST process
model relies on choreography, not orchestration, hence may not need this component.
91
Fig. 45 – Notation for a Process Coordinator
Service Consumer
Process Coordinator
Service interface for the process
Operations (Process steps)Process logic
Fig. 46 – Using a Process Coordinator to pull individual operations into a step-by-step process that implements a business function
We can readily imagine situations requiring a combination of these three component types.
Service Consumer
Process Coordinator
Service interface for the process
Legacy componentBroker
Service Container
Operations (Process steps)
Fig. 47 – A Process Coordinator orchestrates two operations, one of which is directly hosted within a Service Container and the other is actually implemented within a legacy “black box” application but made to appear like a standard operation through the use of a Broker
How Not to Use a Broker
The right tool for the job: Brokers are very powerful, but they must not be used in place
of a Service Container or a Process Coordinator. Such incorrect use of a Broker outside of
its normal role as a mediator introduces fresh dependencies with impacts to stability and
performance23. Designers and developers should be equally proficient in the use of all
three core SOA components to avoid the temptation of “hacking” the Broker to perform
functions it is not meant to do.
Not a hub-and-spokes solution: Perhaps because of the prohibitive cost of many
commercially available Brokers, there is a tendency to install just one instance of this
component for the entire organisation and push all traffic through it24. The obvious
implications of such a centralised, hub-and-spokes architecture relate to performance (i.e.,
scalability of throughput) and to availability (because of its emergence as a single point of
failure).
However, the Broker is inherently federated! Its mediation, transformation and adapter
functions are best performed closest to the endpoints where functionality is either exposed
or consumed. This decentralised or “federated” architecture is illustrated below.
23 This is commonly referred to as ESB-itis.
24 This is another common symptom of ESB-itis.
93
ProcessCoordinator
ServiceContainer
Broker
LegacyService
Provider
LegacyService
Provider
LegacyService
Consumer
BrokerBroker
ServiceContainer
Fig. 48 – The Broker Deployment Architecture – Naturally Federated, not Hub-and Spokes
Implementing the Adverbs of an Operation
We may seem to have glossed over some important aspects of implementing business
logic, and these pertain to what are known as “non-functional” requirements. In other
words, “how” something is done is as important as “what” is done. Our functional
definitions of processes and operations deal exclusively with the “what”, but the “how”
aspect isn't often stated as explicitly. Nevertheless, we need to deal with these “adverbs” in
our implementation just as we do with the “nouns” and “verbs”.
Here's a quick overview of some of the more important ones. They will feature in the
dependency check-lists at the end of this section.
Asynchronous interaction:
SOAP-based Web Services are based on the notion of messaging, and it is important not
to be trapped into the older and deprecated RPC style. Asynchronous capability is inherent
in the SOAP model.
REST appears to be intractably synchronous because of HTTP's blocking model of
client/server interaction. However, this is only true of the “transport” side of HTTP. On the
logical “message” side, asynchronous messaging is relatively easy to model using the
status code “202 Accepted”, which is an acknowledgement rather than a response. The
response can be returned at a later time either through polling or a callback mechanism
set up at the time of the original request.
Security:
There are many aspects to security, such as confidentiality (message encryption), trust
(authentication of the source), authorisation (access control), etc.
SOAP uses special headers to ensure all of these, and there are standards such as
WS-Trust, WS-SecureConversation and WS-Security to provide a measure of security
around interactions. It is also possible to specify security requirements declaratively using
WS-SecurityPolicy.
REST's security model for confidentiality is relatively clunky (TLS/SSL), in that it is not
routable or capable of partially encrypting message payloads, but it is simple. For
authentication and authorisation, the newer web-based security standards such as OpenID
Connect and OAuth 2.0 are elegant mechanisms that can be layered over the actual
services themselves in an unobtrusive way.
94
Reliable Delivery:
SOAP follows the TCP model of reliability in its WS-ReliableMessaging protocol. The
mechanism of message identifiers, acknowledgements, timeouts and retries is exactly
analogous to TCP's sliding window protocol.
REST-based systems often adopt a different model of reliability based on idempotence.
The REST verbs (with the exception of POST) are all idempotent, which means they can
be retried without danger of (repeated) side-effects. Even POST can be made idempotent
if a one-time string is sent along with a request and the server checks for a duplicate
before acting on the message.
Availability:
Services are meant to be stateless, and data is either transient (part of the message) or
persistent, never session-based. This allows for redundant servers to be used to host
services, and this model works for both SOAP and REST. A load-balancer in front of these
servers can ensure availability of the service as well as performance.
Processes are a different kettle of fish. They are stateful by definition, and a WS-BPEL
process server in the SOAP world needs special measures to ensure availability. There
are many “session clustering” or “memory grid” style products to provide a measure of
availability to a cluster of process servers.
Performance:
Latency and throughput are common measures of performance. Improvement of latency
requires detailed analysis and tuning that are specific to the nature of the operation, data,
network characteristics, etc. Throughput is relatively more standard in terms of available
mechanisms. Load-balancing a farm of servers hosting operations is a fairly standard way
to scale performance horizontally. Both SOAP and REST can exploit such mechanisms.
To improve the performance of process servers, it may be required to host high-overhead
processes on their own dedicated nodes.
95
The fundamental technology layer dependencies to be understood are illustrated below.
Technology Layer Dependency Check-lists
1. The Technology Portfolio traces how logical application components such as
processes, applications and operations and logical data elements such as elements of the
internal data model are ultimately implemented in technology through physical application
components. This can be used for a gap analysis of target state and current state.
Sample Governance Questions:
TG-01 What data elements are co-located with applications in a way that does not
permit logical separation?
TG-02 What non-functional dependencies impact the deployment of applications on
physical servers? (scalability, fault-tolerance, etc.)
TG-03 What technologies are best suited to implement various kinds of business logic
and data (e.g., rules engines for rules, graph databases for highly complex and
arbitrary relationships, etc.)
TG-04 Who can access business logic as implemented? (Identity and Access
Management)
96
Physical AppComponent
PhysicalLocation
Internal Data Element
Technology Distribution
Technology Portfolio
implementation
deployment
Technology Dependencies
constraints
Fig. 49 – Dependencies and Dependency Groups at the Technology Layer (TOGAF)
Logical AppComponent
Sample Management Questions:
TM-01 How do we implement platform capabilities (specific products and technologies)?
TM-02 How do we decommission redundant or obsolete platforms?
2. Technology Distribution shows how various technologies are deployed at different
locations.
Sample Governance Questions:
TG-05 What is the level of location-dependency of every physical application
component? (Can everything be moved to the cloud?)
TG-06 What is the level of location-dependency of every element of internal data? (e.g.,
Legislative concerns over moving customer data offshore, etc.)
TG-07 What is the optimal number of installations of each physical application
component, versus the number we actually have? (licensing considerations)
Sample Management Questions:
TM-03 How do we optimally distribute applications to where they are needed?
TM-04 How do we consolidate and migrate applications to the cloud?
TM-05 How do we track all of the technology artifacts that are deployed? (Registries,
respositories, etc.)
3. Technology Dependencies enumerate the various dependencies that exist between
technology components themselves.
Sample Governance Questions:
TG-08 What dependencies should we legitimately have on platform capabilities, versus
actuals? (Storage capacity, latency, bandwidth, availability, scalability, etc.)
TG-09 What market-related dependencies do technology components impose on us?
(Single dominant vendor, vendor without staying power, etc.)
TG-10 What technology platforms afford us the most flexibility and choice?
97
Sample Management Questions:
TM-06 How do we diversify our sources while keeping the number of platforms
manageably low?
TM-07 How do we protect data against physical corruption, accidental loss and data
leakage?
TM-08 How do we monitor the performance and other real-time characteristics of our
technology deployment? (Business Activity Monitoring, etc.)
98
Bringing about Desired Behaviour – Velvet Glove or Iron Hand?
Making sure we're doing the right thing (governance) and making sure we do things right
(management) both pertain to bringing about desired behaviour.
In general, there are two ways to bring about desired behaviour, one obtrusive and the
other unobtrusive. The unobtrusive way is often better. The fable of how the sun
succeeded in getting a man to remove his coat where the wind failed, and the saying “You
can catch more flies with honey than with vinegar” are about this very idea.
Consider a door that must be pushed, not pulled, to open. We have often seen examples
of such doors with big bold “PUSH” signs on them. But we also know that lots of people
unthinkingly try and pull them by mistake anyway. Wouldn't it be far better to design a door
so that it would be physically impossible to pull? This is indeed part of the thinking behind
ergonomic design. We can now see doors that have no handles but only metal plates that
“call out” to an approaching person to push, not pull. These are called "affordances". With
an intuitive affordance, there is no need for a sign telling people to "PUSH". It is physically
impossible for a person to do anything but.
Another example is from the requirement to control speeding on the roads. There is an
obtrusive way to control speeding, which is by putting up clearly visible speed limit signs
and enforcing compliance with those limits. Such enforcement may be by installing speed
cameras and/or conducting periodic targetted campaigns with mobile police squads
equipped with radar guns.
There is however another way to control speeding that may be more effective. Clearly
visible speed humps force motorists to slow down, because the price of non-compliance is
a very uncomfortable ride and possibly a large repair bill from a mechanic. There is no
need for elaborate monitoring and policing, and the expected behaviour is ensured
anyway.
We could call these contrasting approaches the “iron hand” and the “velvet glove”.
We are fans of the more unobtrusive “velvet glove” approach because we consider “iron
hand” style policing to be clunky, expensive and error-prone. However, we recognise that
the latter approach will sometimes be necessary. Our preferred approach to SOA
Governance and Management is therefore “Unobtrusive and intrinsic wherever possible;
obtrusive and extrinsic wherever necessary”.
99
Example: Speed limits and policing versus speed humps
Enforcing desired behaviour
(External to design, requires active policing)
Ensuring desired behaviour
(Inherent in design, no policing required)
Fig. 50 – The Iron Hand and the Velvet Glove of speed reduction
However, the design of systems that are unobtrusive is orthogonal to our discussion of
SOA governance and SOA management. An organisation would certainly benefit by
developing “velvet glove” mechanisms for governance and management, whereby
everyone in the organisation is almost unconsciously shepherded into following the correct
and expected behaviour, but this is an advanced idea that we will not cover further in this
document.
It's certainly worth considering when an organisation designs its systems, though.
100
Summary and Conclusions
Service-Oriented Architecture (SOA) should really be thought of as Dependency-Oriented
Thinking, since vitally important dependencies exist at every layer of the enterprise, and
these have significant impact on an organisation's agility, cost and risk profiles.
Determining “what” dependencies are legitimate, and “how” these dependencies must be
managed, should therefore be the goals of SOA Governance and SOA Management,
respectively.
The model of SOA Governance and SOA Management proposed in this white paper
covers all of the above aspects. Although comprehensive, it is nowhere near as complex
as current industry recommended best practice would have us believe. In this white paper,
all we have done is specify the core entities that exist at each layer of the enterprise, and
the types of dependencies that can occur between them. It is up to the individual
organisation to work out the dependencies that are appropriate to its business, and to
ensure that these are the only ones that in fact exist.
Two logical committees are recommended to oversee the Governance and Management
of the enterprise, respectively, through the application of SOA principles. These two logical
committees may translate to a number of physical committees, depending on the size and
complexity of the organisation. Each committee may in turn set up more specialised teams
to oversee groups of related dependencies.
The sample dependency check-lists in Part III can help these committees and their
specialised teams get started with their tasks. These are based on the BAIT and TOGAF
models of the enterprise, although re-purposed with a dependency focus to serve as a
suite of tools for SOA governance and management.
We hope this comprehensive yet lightweight framework enables organisations to
implement SOA Governance and SOA Management quickly and cost-effectively.
[Designing systems that are unobtrusive and which ensure rather than enforce correct
behaviour is an intellectual challenge of a higher order, and enterprising organisations may
want to experiment with such “velvet glove” approaches.]
101
Contributions of this White Paper
We're tempted to call our method the Dependency-Oriented Governance and
Management Approach, but the resulting acronym runs counter to its pragmatic
philosophy!
We believe there are several tangible and immediate benefits to anyone who seriously
studies this white paper.
Defining Terms
After more than a decade of familiarity with the following terms, it's disappointing that the
industry still hasn't been able to agree on simple definitions for them. We went back to first
principles to see if we could do better.
SOA: The science of analysing and managing dependencies.
Governance: Ensuring that the right things are done. (The ends, or the “What”.)
Management: Ensuring that things are done right. (The means, or the “How”.)
SOA Governance: Determining what dependencies are legitimate and what existing
dependencies fall outside this set.
SOA Management: Dealing with how to remediate illegitimate dependencies, how to
formalise legitimate ones and how to prevent the recurrence of violations.
Restoring Potential
SOA has been a Cinderella languishing in a technology cellar. This paper crowns it the
governing principle for the enterprise. We show how to govern with SOA, rather than how
to govern SOA.
Enterprise Architecture is often criticised as an ivory tower function. This approach gives it
a change agent's role that will impact an organisation's agility, cost and risk in an
immediately measurable way.
Identifying Gaps
Enterprise Architecture needs a practical focus to demonstrate value. The focus on
dependencies is the missing piece that this paper highlights.
102
We have also highlighted the need to inject professional skills from traditionally non-IT
backgrounds into SOA Governance and SOA Management bodies to aid in identifying and
remediating dependencies.
Simplifying Tasks
For all its ambitious scope, the approach outlined here is actually quite simple and
minimal. We have provided a listing of the core roles, bodies, functions, processes and
check-lists that are required to implement it..
However, if all you do is add the processes described here on top of the ones you already
use, you will end up with anything but a lightweight approach. What we are audaciously
recommending here is that you replace your existing control systems with these
processes, fleshing them out only to the extent necessary to meet your business
objectives.
Doubling the Pay-off
The point of this entire white paper is that adopting dependency-orientation as your
enterprise's primary organising principle is not only lightweight in itself, but also trims your
operating model overall. It's a double-benefit.
103
Potential Criticism of This Approach
While we are certain we have contributed something of value to the industry with this white
paper, we don't believe it will be received without controversy. Initial reviews by peers
leads us to believe there could be several angles to the opposition it is likely to provoke.
The Weight of Tradition
The first and most obvious source of resistance is from the traditionalist SOA practitioner
camp that continues to believe in SOA Governance as a set of processes that bring
discipline to the way “SOA technology” is used within an organisation.
A good thumb rule we would suggest if you ever come across a book/article or an expert
who hews to the traditionalist view of SOA Governance is to check if they are explicitly
repudiating this white paper. If they make no reference to this document or have never
heard of it, we humbly suggest that our view trumps theirs, since we have taken the pains
to rationally rebut the establishment view but they have not reciprocated the courtesy. Of
course, if someone from this camp has specific issues with the approach in this white
paper and is able to make a reasoned argument against it, then by all means listen to
them and use your best judgement.
Making Mountains out of Molehills
The second argument against our approach would probably come from those who see our
distinction between Governance and Management as splitting hairs. This argument may
state that the functions of Governance and Management as defined here are both to be
legitimately considered “Governance”, so the industry is not so badly off-track after all.
Our answer is that the “What” and “How” aspects of a decision always go hand in hand,
and both are important. Hence our insistence on making a distinction between the two
does not reduce the effectiveness of these functions, only the confusion between their
concerns. We don't believe both can be bundled together under the label of “Governance”.
On the contrary, it is this conflation that has stifled the real governance function.
In both these arguments, you should view this paper's approach as a revolution against
the establishment line, and if you happen to come across a dissenting view, recognise the
difference between a mere old-schooler and a genuine counter-revolutionary.
104
Drawing a Long Bow
A third argument is possibly that this approach overstates the case for
dependency-orientation as a comprehensive approach to managing a business, and that a
complete overhaul of the governance and management processes of an organisation to
focus on dependencies is difficult to justify and may even be counterproductive.
Our response is that all effort needs to be justified on the basis of costs and benefits. We
have devoted a fair proportion of this document to detailing true life case studies that
illustrate the high costs entailed by dependencies, no matter what kind of business we deal
with or the level of abstraction we examine. There is serious money to be saved by
eliminating dependencies, if organisations will only examine themselves seriously from this
angle. Unfortunately, no major consultancy organisation, industry analyst, B-school guru or
process tool vendor has yet latched onto dependency-orientation as a buzzword, so this is
not a fashionable thing to do. We flatter ourselves with the possibility that this white paper
could just spark such a trend.
A Bridge Too Far
Finally, there is the “you can't get there from here” argument. The pain here is the
wrenching change in focus that the entire organisation will have to undergo. It is a massive
re-education exercise with the inevitable missteps, periods of backsliding and moments of
self-doubt that will accompany the effort. First-movers will have an additional disadvantage
in that they will have no benchmark to follow and will have to summon up the will to stay
the course.
But like with any attempt at self-improvement, moving to dependency-orientation is
another instance of “no pain, no gain”. Perhaps the approach can be piloted within a
smaller business unit and rolled out more widely once its benefits are demonstrated.
105
About the Author
Ganesh Prasad is a Sydney-based architect with over 25 years of experience in the IT
industry and over 10 years as an architect responsible for “Enterprise Shared Services” at
diverse employers such as a leading Australian bank, insurance company and telecom
corporation. He is TOGAF 9 certified and is intimately familiar with warts-and-all SOA as it
is practised in large organisations. He has also studied Risk Management as part of a
degree in Applied Finance.
Ganesh can be contacted at [email protected].
Acknowledgements
Thanks to (…) for reviewing this white paper.
Thanks also to Sushil Gajwani and Ravish Juneja for contributing many of the case studies
cited in this document.
106
Appendix A – SOA Governance and Management – An Issue of
Definition
If you do a search on the term “ungulate”, you are probably looking for what is called an
intensional definition (“Ungulates are mammals that use the tips of their toes, usually
hoofed, to sustain their whole body weight while moving”). You would probably not be
satisfied with a purely extensional definition (“Ungulates are the horse, zebra, donkey,
cattle/bison, rhinoceros, camel, hippopotamus, tapir, goat, pig, sheep, giraffe, okapi,
moose, elk, deer, antelope, and gazelle”). The extensional definition can be used as a set
of examples to support the intensional one, but isn't very useful on its own.
In our own research into the terms “Governance”, “Corporate Governance”, “IT
Governance” and “SOA Governance”, we have been surprised and disappointed to find a
plethora of extensional definitions (e.g., “the set of policies, processes, controls and
metrics”) rather than a simple and readily understandable intensional one (i.e., what
governance actually is). Such extensional definitions tend to lose the reader at the very
outset. Without a clear communication of what governance is, any list of terms carries an
implicit expectation of rote learning rather than understanding.
Sometimes, extensional definitions take the form of a diagrammatic chart that lays out
components in a colourful, eye-catching way (such as the ISO 38500 definition of IT
Governance), but even this friendlier format fails to convey what IT Governance really is.
Even on the rare occasion when we stumbled upon an intensional definition, it has proven
too fuzzy to be meaningful, e.g., Gartner's definition of SOA Governance as “ensuring and
validating that assets and artifacts within the architecture are acting as expected and
maintaining a certain level of quality”. Does that result in an epiphany on the part of the
reader? We suspect not. In the same vein, the Burton Group defines governance as “the
processes that an enterprise puts in place to ensure that things are done [...] in
accordance with best practices, architectural principles, government regulations, laws, and
other determining factors.” This is an extensional definition (a list of things to consider)
wrapped up inside an intensional one (a promising definition of governance as a set of
processes), with the net result that things are no clearer.
107
Many of the definitions of SOA governance are just plain wrong, in our opinion, and these
errors stem from a fundamental misunderstanding of the scope of SOA. Does SOA
Governance refer to the governance of “SOA assets”, or is it a unique philosophy of
governance itself? We believe it's the latter, but the industry seems to think it's the former.
In sum, all of the definitions we have come across have been unsatisfying. We have been
left to wonder – What is SOA Governance? It seemed rather arrogant to try and
second-guess an entire industry, but someone had to bell the cat, hence this white paper.
Here, we provide simple (and intensional) definitions of both “governance” and
“management” that address the confusion between the two terms, with enough extensional
definitions as examples to fix these concepts firmly in one's mind. The approach we
describe is based on these fundamental definitions.
108
Appendix B – Lessons from Cadet Camp (or Why SOA is Like a Snakepit)
At a high school cadet camp, the author remembers how an instructor walked past a group
of cadets pitching their tent and reminded them to dig a “snakepit” that would keep
scorpions and small snakes from entering the tent. One of the cadets dutifully went off and
dug a “snakepit” to one side of the tent, as the plan below illustrates.
There was a fair amount of merriment when the instructor came by again and saw what
had been done. The actual concept of a snakepit, as the cadets then learnt, was as
illustrated below.
109
Tent
“Snakepit”
Tent
“Snakepit” (trench)
Raised “bund”
A snakepit must surround the tent entirely in order to be effective. Digging a pit to one side
of the tent achieves nothing. Snakes and scorpions will not obediently fall into it.
In similar fashion, SOA is most effective when its principles are applied across the board.
It's not a narrow sub-domain of IT that has to be put into its own little sandpit and managed
there. It informs the way the enterprise is run, no less. It's amazing how many industry
luminaries fall prey to this limiting delusion.
At the risk of belabouring the point, let us illustrate it graphically below.
110
The Enterprise
Pervasive SOA principles
SOA “sandpit”
Appendix C – Core Entities and Dependencies in the TOGAF 9 Model
Core Entities
Catalogs list Hierarchies of Core Entities. Matrices and Diagrams represent Two-Entity
and Multi-Entity Dependencies
111
Appendix D – Artifacts In the TOGAF 9 Model
Fundamental Enterprise Entities and Dependencies25
Principles Catalog
Stakeholder Map Matrix
Value Chain Diagram
Business Layer Dependencies
Business Interaction Matrix (Organisations and Actors)
Actor/Role Matrix
Business Footprint Diagram
Business Service/Information Diagram
Functional Decomposition Diagram
Goal/Objective/Service Diagram
Business Use-Case Diagram
Organisation Decomposition Diagram (Actors, Roles and Location)
Process Flow Diagram
Event Diagram (Events and Processes)
Application Layer Dependencies
System/Organisation Matrix
Role/System Matrix
25 TOGAF does not identify the “Driver/Goal/Objective Catalog” as part of the Preliminary or Architecture
Vision phases (which correspond to the fundamental Enterprise level) but as part of the Business
Architecture phase. However, our experience with enterprise utilities/shared services tells us that Goals
apply at the overall enterprise level as well as at the level of business unit silos below it, and the dichotomy is
often critical. That's why we include it in our set of Enterprise Dependencies as well as Business Layer
Dependencies..
112
Application Interaction Matrix
System/Function Matrix
Application Communication Diagram
Application and User Location Diagram
System Use-Case Diagram
Enterprise Manageability Diagram
Process/System Realisation Diagram
Application Migration Diagram
Software Distribution Diagram
Software Engineering Diagram
Information (Data) Layer Dependencies
Data Entity/Business Function Matrix
Business Service/Information Matrix
System/Data Matrix
Class Diagram
Data Dissemination Diagram
Data Life-Cycle Diagram
Data Security Diagram
Data Migration Diagram
Class Hierarchy Diagram
Technology Layer Dependencies
System/Technology Matrix
Environments and Locations Diagram
Platform Decomposition Diagram
Processing Diagram
113
Networked Computing and Hardware Diagram
Communications Engineering Diagram
114
Appendix E – References
1. Wikipedia: http://bit.ly/HyPBBE
2. Dave Oliver's blog entry: http://bit.ly/H5mzY2
(A lone blogger gets it right where giants like IBM and the Burton Group fail. He doesn't go
far enough, however, preferring to restrict the scope of SOA to just IT.)
3. IBM's entry: http://ibm.co/qm46
4. Towards a reference model for SOA Governance: http://bit.ly/H4yjKP
5. Forrester SOA Value Assessment: http://bit.ly/H4ynu4
6. SOA Software's reference model: http://bit.ly/HvuRYq
(Two fundamental layers; an application and messaging services layer, and an
infrastructure services layer.)
7. Burton Group on SOA and Governance: http://bit.ly/HFdF6t
(Extensional definition: What is a governance program? Policies, Processes, Metrics,
Organisation)
Confusion on slides 2 and 5 (doing the right thing versus doing things right)
8. The reincarnation of SOA: http://bit.ly/9aowov
9. Resurrecting SOA: http://bit.ly/aQMCRk
10. You can't buy governance: http://bit.ly/HaRxkx
(confusion on pages 2 and 5 – doing the right thing versus doing things right)
IT governance:
11. Confuse, scare and sell: http://bit.ly/c8ZD
12. ISO 38500: http://bit.ly/j3Ym4H
(Extensional definition in the form of a chart)
13. CIO.com article: http://bit.ly/pz2lr1
14. Simplicable model: http://bit.ly/Hb5NcM
Corporate Governance:
15. Wikipedia: http://bit.ly/XsnAz
115
16. ASX: http://bit.ly/HlhO6F
(Partly intensional, partly extensional, useful principles)
17. SearchFinancialSecurity: http://bit.ly/e9x20x
(Extensional definition)
18. Auditor's perspective: http://bit.ly/pkwHKh
(Extensional definition: Corporate governance is the set of processes, customs, policies,
laws, management practices and institutions affecting the way an entity is controlled and
managed.)
19. Data on the Outside vs. Data on the Inside: http://bit.ly/RHj7dp
(Seminal paper by Pat Helland)
116
