Date post: | 25-Jul-2015 |
Category: |
Technology |
Upload: | wso2 |
View: | 380 times |
Download: | 2 times |
Exploring Data Integration Capabilities of the WSO2 Platform
June 2015
Jagath AriyarathneTechnical Lead
Ravi UndupitiyaSenior Software Engineer
About the Presenters๏ Jagath Ariyarathne - [email protected]
Jagath is a Technical Lead at WSO2. Prior to joining WSO2, he worked for DirectFN (Pvt) Ltd, where he was involved in the design and development of DirectFN Financial Data dissemination back-end system. His main focus was on real-time data distribution with low latency, high performance and bandwidth friendly solutions. He holds a BSc First Class degree in Computer Science and Engineering from University of Moratuwa, Sri Lanka.
๏ Ravi Undupitiya - [email protected]
Ravi is a senior software engineer in the integration technology team where he mainly focuses on the WSO2 Enterprise Service Bus. Prior to joining WSO2 he interned at the Intelligent Agents and Multimedia Research Group at the University of Southampton where he worked on a smart grid project which was aimed at helping people make smarter decisions when consuming energy. Additionally he also dived into the world of Natural Language Processing and carried out an evaluation of several text classification algorithms.
*
Agenda๏ Background to Data Integration
๏ SOA based approach to Data Integration
๏ WSO2 Data Services Server
๏ WSO2 Enterprise Service Bus
๏Demo
๏ Security ๏ Transactions
*
Why we need Data Integration?๏ Data is an important asset.
๏ Found in various sources and often siloed within applications.
๏We must leverage this data.
๏ Data consumers on various devices and platforms. ๏Security and privacy.
๏ Needs to be flexible for change.
๏ Legacy system support.
*
Data Integration Solutions ๏ Ad-hoc approach to data integration not good!
๏ Unmaintainable and messy.
๏ Not flexible enough for changes.
๏ Need data source specific implementations
๏SOA approach better!
๏ Data exposed as a services.
๏ Applications consume the data through services.
.NET J2EE C++
Data Sources
.NET J2EE C++
Data Sources
*
Data Services on WSO2 Platform๏ WSO2 Data Services Server (DSS)
๏ Allows rapid development of data services.
๏ Data services are decoupled from the infrastructure where it is stored.
๏ Wizard based UI to generate data service description language.
๏Data service description language
๏ Maps service requests to queries operating on data store.
๏ Maps query results to XML schema definitions.
๏ Supports validation, transformation, eventing etc…
DSS Key Features - Any Data Store
DSS Key Features - Data Federation
DSS Key Features - Server Side Batch Processing
DSS Key Features - Client Side Batch Processing
DSS Key Features - Array/UDT/Complex Types
DSS Key Features - Distributed Transactions
DSS Key Features - Notifications๏Event sources, subscribers, topics๏Allows a subscriber to be notified of a specific activity in a data service
๏Event Triggers๏ Input Event Trigger๏Output Event Trigger
DSS Key Features - REST Support
DSS Key Features - Caching Support
DSS Key Features - Security Support
DSS Key Features - Transformation
*
Demo 1๏Exposing MySQL data table as a service.
๏ SOAP
๏ RESTful
*
Data Integration Solutions ๏ Ad-hoc approach to data integration not good!
๏ Unmaintainable and messy.
๏ Not flexible enough for changes.
๏ Need data source specific implementations
๏SOA approach better!
๏ Data exposed as a services.
๏ Applications consume the data through services.
.NET J2EE C++
Data Sources
.NET J2EE C++
Data Sources
*
Layered Approach๏SOA alone is not enough. Point-to-Point service integration. ๏Enterprise Service Bus for a cleaner layered approach.
Point-to-Point Service Integration (Spaghetti Integration) ESB in Action
What is an ESB? ๏An ESB is a middleware solution that enables interoperability, among heterogeneous environments using a service oriented model. ๏Stateless and Seamless Integration๏Standard protocols/formats like SOAP, REST, JSON ๏Transports like HTTP/S, JMS, TCP, VFS (and many more)
The WSO2 ESB๏Lightweight and high performance๏Feature rich and standards compliant๏ SOAP and WS-* standards๏REST support๏Domain specific transport support (FIX, HL7)๏User friendly and highly extensible ๏Configuration driven! ๏100% free and open source
WSO2 ESB๏Apache Synapse ๏A lightweight and open source ESB implementation from ASF๏Makes up the mediation engine of WSO2 ESB๏Multithreaded and asynchronous message processing๏Based on a number of well known open source projects๏Apache Axis2 ๏Apache HTTP Components
WSO2 ESB Fundamentals๏ Routing๏ Filtering๏ Transformation๏ Protocol Switching๏ Load Balancing ๏ Quality of Service
More Features๏100% EIP support๏Connectors๏Message Stores/Processors๏JSON Support
๏ JMX based monitoring and control๏ Statistics collection๏ Priority based message mediation๏ XSLT, XPath, XQuery, Smooks, JSONPath๏ Caching, Throttling๏ Scripting languages (JavaScript, Groovy)๏ JDBC connectivity๏ Drools๏ Clustering
Functional ComponentsMediator Sequence Endpoints Local Entry Proxy Service REST API Message Stores/Processors Templates Connectors Tasks Priority Executors Registry
Functional Components๏Each functional component serves a specific purpose
๏Functional components can be mixed and matched to implement various integration scenarios and patterns
๏Configuring WSO2 ESB for a given scenario requires: ๏ Identifying the right set of components๏Putting them together in the most optimal manner
ESB and DSS
๏ Why not access data stores directly from ESB?
๏ ESB has rudimentary DB support๏DBLookup Mediator๏DBReport Mediator
๏ ESB’s function is to ensure low latency with high throughput for stateless transactions. Data intensive processing done on ESB could hinder ESB performance.
๏ Separate JVM/servers to handle data intensive tasks.
๏ Can deploy DSS features on top of ESB (thanks to WSO2 Carbon platform).
Layered Approach
๏ Why access DSS through ESB?
๏ Perfectly ok to access DSS data services directly.
๏ Handling DSS invocations through ESB provides a layer of abstraction over data layer for integration.๏ Interception point๏ Service Chaining๏ Parallel processing๏ Load balancing ๏ Simple transformations๏Handling Multiple protocols
๏ A merger causes an organization to consolidate their account databases.
๏ Leverage WSO2 DSS to expose data from multiple data sources as services.
๏Leverage WSO2 ESB to provide a secured uniform interface and aggregation.
Demo 2 Use Case
MySQL Database
Excel Data Sheet
*
Demo 2Aggregate data from multiple WSO2 DSS Data Services using WSO2 ESB
Excel Data Service
RDBMS Data Service
Excel File
MySQL DB
Data Services ProxyClient Service
● Aggregate Mediator● Clone Mediator● Header Mediator
Request
Response
Multiple Sources
Demo 3 Use Case
๏ A fund transfer between two accounts need to occur in a transacted manner.
๏ Leverage WSO2 DSS for distributed transaction using boxcarring feature.
๏ Leverage WSO2 ESB for service chaining.
*
Demo 3Boxcarring sample with WSO2 DSS and ESB
begin_boxcar
execution 1
execution 2
………
execution 10
end_boxcar
begin_boxcar
execution 1
execution 2
abort_boxcar
execution 10
end_boxcar
*
Demo 3 cont
Bank Data Service
Bank DB (Book Store’s)
Payment Services ProxyBook Store
Application
● Customer’s Account Number● Book Store’s Account Number● Payment Amount
Bank DB (Customer’s)
● Customer’s Account Number● Payment Amount
● Book Store’s Account Number● Payment Amount
begin_boxcar
end_boxcar
credit_bookstore
debit_customer
Service Chaining
Where to begin..
๏ WSO2 ESB Documentation ๏ https://docs.wso2.org/display/ESB481/WSO2+Enterprise+Service+Bus+
Documentation
๏ WSO2 DSS Documentation๏ https://docs.wso2.com/display/DSS322/WSO2+Data+Services+Server+D
ocumentation
Further Reading๏Master Data Management using WSO2 Platform ๏ http://wso2.com/library/webinars/2013/10/mdm-with-wso2-enterprise
-middleware-platform/
Contact us !