ProcessPerformance Zone
SOA01 1-2
Getting Close to the Customer The Rôle of Decision
Automation
Ian GrahamPrincipal Consultant and Technical Director
TriReme International Ltd.
Trireme International Ltd. © MMVII
© MMVII TriReme International Ltd SOA01 1-3
The future of IT Our architecture is complex and
messy
All this BPM and SOA nonsense is just more hype
Monitoring comes to me in reports
Data are stored in multiple databases and hard to integrate
Modelling is too expensive
Application backlog: don’t ask!
Our Web 2 initiatives are quite separate from core systems
Etc.
orHalf full?
Half empty?
We are moving forward in meeting business needs
Most business functions are provided as services
Many services are orchestrated by our process models and the processes are tracked and constantly improved
All our core static data are integrated and available for analysis
We welcome changes to requirements because they are so easy to implement
Etc.
© MMVII TriReme International Ltd SOA01 1-4
Why systems projects so often fail
lack of user involvement
no clear statement of requirements
no project ownership
no clear vision and objectives
lack of planning
59% of USA projects are cancelledor overrun budgets
Standish Group, 1995, 1997, … 2004,...
The need: agility, business focus.
© MMVII TriReme International Ltd SOA01 1-5
Enterprise Decision Management (EDM) Manifesto
Operational decisions matter
Customers judge you by your decisions: speed, quality, logic, etc. The effect is cumulative over many small decisions
Many decisions can be automated
This can help with consistency, precision, speed, labour costs, … … and agility Mature BRMS and analytics technology exists
Control decisions to get competitive edge
Capture and preserve staff expertise (rules) Learn from actual sales experience (mine your data) Give the customer more control (Web 2.0?)) Put the business experts in control (rules, SOA) Update the rules quickly (rule authoring) Provide explanation/audit (rules engine)
© MMVII TriReme International Ltd SOA01 1-6
The problem
How to ensure that staff can make small but business critical decisions that
are precisely correct. meet the customer’s expectations in terms of speed. are customized to the circumstances of the customer. are customized to the needs of the business. are consistent. meet regulatory requirements. do not entail big labour cost overheads (more skilled staff, etc.)
How to ensure that the business rules that are used to reach decisions
can be changed easily and quickly. can be put directly into the hands of the business people.
© MMVII TriReme International Ltd SOA01 1-7
Conventional approaches...
…separate operational systems from decision support systems
Operational systems manipulate data to control and support business functions
Decision support systems transform and enhance data for the benefit of knowledge workers. These workers must then analyze the results and submit change requests to IT to get any requisite changes actioned.
It is a slow and unpredictable process
Process improvement only takes place at a coarse (or strategic) granularity
The millions of customer-facing micro-decisions embedded in operational systems cannot be upgraded.
© MMVII TriReme International Ltd SOA01 1-8
Kinds of decision
Decisions that might be candidates for automation:
Operational High volume High value Subject to regulator’s or management audit
And those that might not:
Strategic One-off Low volume and low value Highly judgmental or intuitive Socially or politically sensitive
© MMVII TriReme International Ltd SOA01 1-9
Operational decision automation technologies
Business Intelligence (BI) and performance monitoring, BAM (Business Activity Monitoring)
Data mining and analytics
Data warehousing
BPM
SOA
Business rules
© MMVII TriReme International Ltd SOA01 1-10
BI and BAM
Business Intelligence (BI) and
Collects significant aggregate data Management reports Management dashboards Usually not done in real time Manual feedback to BPM
Performance monitoring
As BI but includes forecasting and planning
Business Activity Monitoring (BAM )
Real time data collection Often part of BPMS suites (e.g. Pegarules) Management dashboards (usually requiring human attention) A few BAM products can alert apps, services or processes
© MMVII TriReme International Ltd SOA01 1-11
Data warehousing
Extracts aggregate and (sometimes) transactional data from operational systems
Usually not in real time (overnight?)
Requires complex ‘data cleaning’ processes
Provides a single location for data mining and analytics
Good basis for initial EDM projects
© MMVII TriReme International Ltd SOA01 1-12
Data mining and analytics
Data mining (aka machine learning)
Genetic algorithms Algorithmic methods
These (and GAs) extract latent relationships, often directly in the form of business rules
Neural nets Hybrid approaches
Analytic models
Statistics (chiefly regression and clustering) Optimization (linear and dynamic programming) E.g. What attributes of lenders make most them likely to default? What cross-selling strategy works best for over 50s? What mix of contract options gives the highest margin?All these techniques can draw conclusions based on actual data
(and the assumptions of the models).
© MMVII TriReme International Ltd SOA01 1-13
BPM – UML activity diagrams
Prepareorder
Assign itemto delivery
Assign goodsto warehouse
Pick deliveryto satisfy order
Assign stockto delivery
Order prepared
Order satisfied?Y
N &priorityorder
Mark orderas satisfied
Reorderstock
Y
Reorderneeded?
Goodsarrive
Neworder
Allorders
satisfied
Deliveryto pick?
YN
Goodsremaining?
Y
SALES GOODS INWARD
swimlanes
© MMVII TriReme International Ltd SOA01 1-14
BPM – Use case decomposition with sequence rules
enter deal
EnterDetails checkLimits commit deal sendConfirms adjustPositions
checkDealerLmtcheckCptyLmts
checkGlobalLmt ConfToCpty ConfToSettlement
precedes
Sequence diagrams impose design decisions. Therefore...
© MMVII TriReme International Ltd SOA01 1-15
A BPMN Process
© MMVII TriReme International Ltd SOA01 1-16
The BPM life cycle
analysis
design
implementation
enactment
evaluation
monitoring
requirements
process model
infrastructure
case data
case data
requirements
Process improvement does not take place in real time
SOA01 1-17
SOA technology
© MMVII TriReme International Ltd SOA01 1-18
What is service oriented architecture?
A software architectural style that focuses on the use of services to support business requirements.
In an SOA, resources are made available to other participants in the network as independent services that are accessed in a standardized way.
Many definitions of SOA identify the use of web services (using SOAP and WSDL) in its implementation, however it is possible to implement SOA using any service-based technology.
Though built on similar principles, SOA is not the same as Web services. SOA is independent of any specific technologies.
© MMVII TriReme International Ltd SOA01 1-19
What is service oriented architecture?
Services are loosely coupled; i.e., the service interface is independent of the implementation. There can be unintended consequences of coupling (good and bad ones).
Each SOA service has a quality of service (QoS) associated with it. Example QoS elements are security requirements, such as authentication and authorization, reliable messaging, and policies regarding who can invoke services. SLAs++
Application developers or system integrators can build applications by composing one or more services without knowing the services' underlying implementations.
For example, a service can be implemented either in .NET or J2EE, and the application consuming the service can be on a different platform or language.
© MMVII TriReme International Ltd SOA01 1-20
Composing loosely coupled services
6 widgets @ 6p1 6mm toggle @ £1.20Total cost £1.56 +VAT
In stock: All items
What do Ineed to makeup a floggle?
Bill of materialsservice
Stock controlservices
Pricingservice
VAT rulesservice
External pricing feed(Honest John’s prices?)
© MMVII TriReme International Ltd SOA01 1-21
Business drivers for SOA High project failure rate
Rapidly evolving technology and platforms
The need for agility
High maintenance costs Rapidly evolving or poorly understood requirements (requirements creep) The need to innovate new business processes The need to conform to rapid changes in regulation
The need for greater business focus
The lunatics are running the asylum! The need for a common language. Involving the business in service definition
and delivery Focus on the real customers for competitive edge
Economies of scale and reuse
Exploiting and integrating the legacy (EAI) Sharing common services (consistency)
© MMVII TriReme International Ltd SOA01 1-22
Getting SOA right
implementation based services support this interface
Service Oriented Architecture support this interface
systemuserreal user
© MMVII TriReme International Ltd SOA01 1-23
BPM and SOA
Business Process Models
Application & decision services
Operational systems/applications
Orchestration/choreography layer
Business services
© MMVII TriReme International Ltd SOA01 1-24
Services for the business
SOA (if done properly) –
leads to interfaces about the business, not about supporting the user interface
beware of just wrapping existing interfaces — too low level must understand the business, not just the computer system
provides services for people to do tasks that deliver them value
employees customers suppliers regulators
They should understand the system in their own terms — not the software developers’ argot
© MMVII TriReme International Ltd SOA01 1-25
Standards
Web services
XML, WDSL, UDDI, SOAP, … WS* – security, reliable messaging, … , etc.
Business Process Modelling
BPEL, BPMN, UML
Business rules
OWL, SVBR, UML, …
Vertical industry standards
E.g. Insurance (ACORD), Oil & Gas, … Business rules ‘knowledge packs’
SOA01 1-26
Business rules technology
© MMVII TriReme International Ltd SOA01 1-27
A business rule is ...
… a compact, atomic, well-formed, declarative statement about an aspect of a business that can be expressed in terms that can be directly related to the business and its collaborators, using simple unambiguous language that is accessible to all interested parties: business owner, business and product analyst, technical architect, customer, and so on. This simple language may include domain-specific jargon. Business rules are always interpreted against a defined domain ontology.
There are several sources of business rules• Users’ knowledge• Documentation• Use case goals• Type models• Data mining
© MMVII TriReme International Ltd SOA01 1-28
The architecture of a business rules service
Knowledge Base
Inference Engine
Symbol manipulation, computation, etc.
Rules, facts, objects, associations, etc.
Chaining, learning, conflict resolution, etc.
© MMVII TriReme International Ltd SOA01 1-29
Techniques for representing rules
Rules A loan may be approved if
the status of the customer is highand the loan is less than 2000
unless the customer has a low rating Rulesets
Decision trees and decision tables Equivalent to rulesets
Type models & semantic networks (ontologies)
© MMVII TriReme International Ltd SOA01 1-30
What are BRMSs? Using BRMSs together with better requirements engineering and business
modelling within the context of SOA should decrease development costs and dramatically shorten development and maintenance cycles.
Business rules management systems separate the rules from data and control logic and maintain them in a repository.
Rules are grouped into rulesets and inference over and within rulesets is both possible and transparent.
BRMSs have applications across all industries and many types of business problem.
In the context of SOA and EDM, the main impact of BRMSs is that process and decision logic can change much more rapidly.
Rules (rulesets) should be regarded as services.
And the business OWNS the rules.
© MMVII TriReme International Ltd SOA01 1-31
The architecture of a BRMS
Rule authoring service
Rule engine(decision service)
Repository
Businessservices
Infrastructural services,including persistence service
(not rule-based)Database
© MMVII TriReme International Ltd SOA01 1-32
Business drivers for BRMS Current software development practice inhibits rapid delivery; modest
changes to existing systems can take too long.
Competitive pressure means that policy and rules must be amenable to rapid change.
Personalizing services, content and interaction styles.
In regulated industries, rules for governance and regulation will change outside the control of the organization.
Sarbanes-Oxley, Basel II, etc. mean that business processes and rules must be visible.
Business rules and processes can be shared by applications across the whole enterprise using multiple channels, thereby encouraging consistent practices.
© MMVII TriReme International Ltd SOA01 1-33
Benefits of BRMSs
Better alignment and understanding between business and IT
Creation of a common language Representation of the ‘implemented’ knowledge is more
understandable to the business
Improved agility
Faster development (particularly of changes) Faster maintenance, which particularly relevant in service oriented
architectures, where the maintenance of a rules component is addressed outside of the wider IT maintenance context
Clearer auditability of business rule execution
Greater consistency across the enterprise
the same rule executed in different applications One point of change management
© MMVII TriReme International Ltd SOA01 1-34
Products JBoss Rules (DROOLS – BRE only) – free?
Jess – BRE only, Oracle’s rule engine now
PegaRules – BPM focused
Versata – database update focus
Corticon – not rete-based
ILOG JRules (and the .NET product) Suits Java culture (you have to write your own business friendly language)
Haley Authority (“natural language” authoring) Suits environments where business people and business analysts may
create and maintain some rules
Blaze Advisor Suits conventional IT culture, business users to not author rules but user
management applications can be written
© MMVII TriReme International Ltd SOA01 1-35
A good BRMS should... allow business analysts to create and modify the rules
use a fully-featured repository
support backward chaining
allow the rule engine to be a component or service within larger applications
allow applications to be deployed in a service oriented architecture
focus on business rules management (as opposed to just workflow) problems
provide good report generation facilities
provide evidence of successful commercial applications
be compatible with a component-based or service-oriented architecture
offer commercial-standard professional support
SOA01 1-36
Patterns
© MMVII TriReme International Ltd SOA01 1-37
Patterns Standard solutions to commonly encountered problems
E.g. ‘linked list’, ‘recursive descent’, ‘observer’, ‘ten minute rule’. Named problems and solutions make it easier to think about and talk about models, designs or processes Standard solutions make it easier to anticipate details of designs; e.g. when reading code
Benefits
Easier skills transfer — in digestible chunks Reduces multiplicity of ways problems are solved; hence:
Easier to discuss and document designs
Easier to read, check, maintain code — fewer surprises
Make your own for your project
job of the chief architect to establish global ways of doing things project architecture = patterns used on project = ‘how we do things around here’
Originally the notion of a building architect, Christopher Alexander
Actually, Victorian builders
© MMVII TriReme International Ltd SOA01 1-38
Patterns
Alexander – built environment
pattern language
GoF – software design
P4 business process patterns (Aalst, et al., 2003)
pattern catalogues
PoV/POSA – software and system design
pattern system
ADAPTOR, Borschers, Coplien&Harrison, Wu, RulePatterns,…
– software, modelling & process pattern languages
© MMVII TriReme International Ltd SOA01 1-39
Patterns Alexandrian pattern format:
Name, star rating, AKAs Sensitizing image Context Problem Forces at work, known uses, examples, discussion Solution Resultant context
There are other formats, notably for more technical patterns.
Choose a format and stick to it.
© MMVII TriReme International Ltd SOA01 1-40
Processes as pattern languages (RulePatterns)1. Establish the
business objectives
3. Establishthe use cases
2. Businessprocess model
7. Timeboxes9. Automate
testing
10. UsabilityTesting
8. Gradualstiffening
6. User-centredservice structure
4. Build a typemodel (ontology)
5. Discoverbusiness rules
13. Ask the business
11. Association loopsconceal rules
12. Write theconstraints as rules
14. Assign rulesto components
16. Policy blackboard17. Store rules in a
repository18. Encapsulate
a reference
19. Determinesecurity model
21. Followstandards
22. Determine ownership& permissions
23. Define a rule-writing style
24. Write theconsequent first
15. Base errormessages on rules
UI Patterns
Rule ObjectPatterns
20. Separatevolatile rules
Concrete &Terminal
Abstractpattern
ConcretePattern
Key
ExternalPatterns
SOA01 1-41
Adopting EDM
© MMVII TriReme International Ltd SOA01 1-42
EDM process and architecture
Datawarehouse
Operational systems
Externalfeeds
Analyticmodels
Rulerepository
Ruleauthoringsoftware
Businessexpert
Customer
Decision service
Application Ruleengine
© MMVII TriReme International Ltd SOA01 1-43
High ROI accretes to systems that...
Have a large number of rules
Have rules that change often
Have rules that involve domain expertise
Have users that are prepared to maintain the rules
Have rules that are complex or interact in complex ways
Store a large amount of data about customers, business processes, etc.
Have clearly defined business processes
Require audit (regulatory controls)
© MMVII TriReme International Ltd SOA01 1-44
Adoption strategies
Change management. Securing management ‘buy in’.
Manageable but visible project(s) Define the long-term architecture Define the (local) domain model Start with BRMS introduction and service and decision definition Introduce BPM Quick win(s) Add in analytics and optimization Manage expectations (EDM is not just for Christmas) Rebuild trust Be agile (learn how to descope rather than run late) Publish the method and architecture (Consider patterns for this) Education and training (how to talk English!) Consider fully adaptive control
© MMVII TriReme International Ltd SOA01 1-45
Skills required
Business skills
Communication skills
Running workshops and requirements engineering
Modelling, modelling, modelling. (BPM, use cases and types, ontology, business rules, maths & stats)
Web services
Product-specific skills
Business Rules Management Systems
Agile process management
SOA services portfolio governance
© MMVII TriReme International Ltd SOA01 1-46
References and further reading
James Taylor and Neil Raden (2007) Smart (Enough) Systems: How to Deliver Competitive Advantage by Automating the Decisions Hidden in Your Business, Addison-Wesley
Project manager’s guide to these ideas
Ian Graham (2006) Business Rules Management and Service Oriented Architecture, Wiley
More technical introduction to the business rules approach
Thomas Davenport and Jeanne Harris (2007) Competing on Analytics, Harvard Business School Press
Management level introduction to predictive analytics
SOA01 1-47
© 2007 Trireme International Ltd
Questions?
TriRemewww.trireme.com
V1.0
ProcessPerformance Zone