Real-time Big Data in Financial Services
Perry Krug – Couchbase Principal Solutions ArchitectBrian Hopkins – Forrester Research Analyst
1
Source: “Connected Cows?” Joseph Sirosh, Microsoft at Strata San Jose 2015
Image source: Siloam Springs Veterinary Clinic (http://siloamspringsvet.com/)
The tale of the connected cow
© 2015 Forrester Research, Inc. Reproduction Prohibited 4
Big data promises insights discovery
65% of firms will be
using big data
analytics to optimize
digital experiences.
56% of firms will
have implemented
Hadoop.
38% of firms will
have spent more
than $10 million on
data and analytics.
Base: 3,005 business and technology decision-makers Source: Forrester’s Global Business Technographics® Data And Analytics Survey, 2015
© 2015 Forrester Research, Inc. Reproduction Prohibited 5
The age of the customer is the driving force
More than 50% of
technology project
spending will focus on
age of the customer
objectives in 2015.
© 2015 Forrester Research, Inc. Reproduction Prohibited 6
The data hub circa 2012
› Source: June 12, 2013, “Deliver On Big Data Potential With A Hub-And-Spoke Architecture” Forrester report
© 2015 Forrester Research, Inc. Reproduction Prohibited 7
The data hub circa 2015
› Source: March 25, 2015, “Boost Your Business Insights By Converging Big Data And BI” Forrester report
© 2015 Forrester Research, Inc. Reproduction Prohibited 8
Data is a means; effective action is the end
Right Data
Effective Actions
Digital Insights
X Volume and variety are bottlenecks
X Testing and finding the right
ones is time consuming
X Deploying into software is painful
© 2015 Forrester Research, Inc. Reproduction Prohibited 9
You will need a digital insights architecture
Insights-driven
applications
• Sales and marketing
• Operations
• Product design
• . . . lots more
Insight
Fabric
• Services framework
• Federation and virtualization
• Big data management
• Cloud services
Data feed
management
Execution
engines
Insight
discovery and
implementation
tools
Collaboration
and governance
tools
Source: April 27, 2015, “Digital Insight Is The New Currency Of Business” Forrester report
© 2015 Forrester Research, Inc. Reproduction Prohibited 10
Financial services insight trends and challenges
Trends
› Age of the customer and
regulations forcing change
› Engagement data piles onto
tons of SoR data
› From EDW to discovery and
text analytics
› Re-platforming for real time
Challenges
› Real time customer
engagement
› Real time fraud loss prevention
› Improving the cost for legacy
systems performance
©2014 Couchbase Inc.
24x365
What makes Couchbase unique?
12
Performance & scalability leader
Sub millisecond latency with high throughput; memory-centric architecture
Multi-purpose
Simplified administration
Easy to deploy & manage; integrated Admin Console, single-click cluster expansion & rebalance
Cache, key value store, document database, and local/mobile database in single platform
Always-on availability
Data replication across nodes, clusters, and data centers
Enterprises choose Couchbase for several key advantages
©2014 Couchbase Inc.
Couchbase provides a complete Data Management solution
13
High availability cache
Key-value store
Document database
Embedded database
Sync management
Multi-purpose capabilities support a broad range of apps and use cases
©2014 Couchbase Inc.
Security with Couchbase
Administrative Security: Admin user (full access) vs. Read-only user (monitoring, developer access) SSL encryption to REST API and Web UI HTTP access log
Data Security: Applications connect via SASL with single user/pass Data-at-Rest encryption via partnership with Vormetric SSL encryption for over-the-wire
Coming in 4.0: LDAP/Kerberos integration Extensive administrative action auditing
©2014 Couchbase Inc.
Big Data = Operational + Analytic (NoSQL + Hadoop)
15
Online
Web/Mobile/IoT apps
Millions of customers/consumers
Offline
Analytics apps
Hundreds of business analysts
©2014 Couchbase Inc.
Couchbase + Big Data
New Data Stream
Merged View
All DataPrecompute
Views (Map Reduce)
Process Stream
Incremental Views
Partial Aggregate
Partial Aggregate
Partial Aggregate
Real-Time Data
BatchRecompute
Batch Views
Real-Time Views
Real-TimeIncrement
Merge
Batch Layer
Serving Layer
Speed Layer
Couchbase HadoopConnector
©2014 Couchbase Inc.
Real Time Big Data
18
ObjectiveDrive revenue, customer satisfaction, and operational efficiency by leveraging insights from big data analytics in real time
Business requirements
Manage massive data volumes at high speed
Store and manage numerous and changing data types
Export/import data to/from analytics platformsThe Couchbase Solution
Push-button scalability – fast, easy and inexpensive to scale to any size
Integrated cache – enables fast performance and high throughput
Flexible JSON data model – easily adapts new data types and attributes on the fly
Real time Hadoop integration via in-memory streaming – easily export data and import analytics results
Technical requirements
Scalability and throughput
Data model flexibility
Integrate with Hadoop
©2014 Couchbase Inc.
User Activity Tracking and real-time analyticsObjectives & Challenges
Provide business users with real time reports and visualizations of user interaction data
Collect web and mobile clickstream in real time
Integrate with other big data technologies (Hadoop and Storm)
Provide views of data across multiple dimensions (e.g., time, location, browser and device types)
19
130m+ active accounts, in 190+ countries, 25 currencies
10TB data
1B documents
Solution
Deploy Couchbase Server to capture, store, and process real time web data
Ingests data (via Storm) from multiple inputs, including mobile, web, and other services, storing data as JSON documents
Integrates with Hadoop to pass data for additional offline analytics
Generates views for business users in under 1 minute, based on 10-minute data collection intervals
The Couchbase AdvantageReal time performance, easy integration with Storm and Hadoop
©2014 Couchbase Inc.
User Activity Tracking and real-time analytics
20
Couchbase Solution Couchbase Server deployed to capture, store, and process real time
web data Ingests data (via Storm) from multiple inputs, including mobile, web,
and other services, storing data as JSON documents Integrates with Hadoop to pass data for additional offline analytics
Results Consistent low latency (sub 10-msec response) High availability enabled by distributed caching and XDCR Views for business users are generated in under 1 minute, based on
10-minute data collection intervals
©2014 Couchbase Inc.
Fraud Detection
24
ObjectiveIncrease profitability, reduce risk, and comply with regulations in real-time by analyzing user, transaction, and contextual data
Business requirements
Update frequently changing data and data types -customer data ,account data, detection rules
Provide real time responsiveness
Process very high volume of interactions
The Couchbase Solution
Flexible JSON data model – easily adapts new data types and attributes on the fly
Integrated cache – provides real time responsiveness and high throughput processing
Push-button scalability – fast and easy to meet growth requirements
Technical requirements
Data model flexibility
Low latency
High throughput
©2014 Couchbase Inc.
Fraud Detection with FICOObjective & Challenges
Capture transactions, store card / account profiles, customer profiles & user defined variables with sub-msec latency and high throughput
Growing number of accounts, cards and customers means more data needs to be tracked and faster latencies are required
Relational systems unable to scale to the required throughput
HA / DR solutions not streamlined – need custom development
25
Falcon
#1 Fraud Detection platform in the world
65% of worlds credit / debit cards scored by Falcon
Solution
Use Couchbase as the “profiling store” and replace relational database
Each Falcon customer has 100’s of millions of card and / or account profiles that can easily be stored and updated based on consumer’s real time activity
Neural networking algorithms run on Couchbase and access data as key-value pairs. Memory-first architecture allows <1ms responses.
Complete HA / DR solution delivers 24x365 application uptime
The Couchbase AdvantageMemory-first architecture means high throughput, all with click-button scalability
©2014 Couchbase Inc.
Fraud Detection with Couchbase at Wells Fargo
26
Couchbase Solution Use Couchbase as the “profiling store” and replace relational database Each Falcon customer has 100’s of millions of card and / or account
profiles that can easily be stored and updated based on consumer’s real time activity
Results Complete HA / DR solution delivers 24x365 application uptime Memory-first architecture allows <1ms responses. Neural networking algorithms run on Couchbase and access data as
key-value pairs
©2014 Couchbase Inc.
Improving Legacy Systems through caching
30
The problemProvide low latency and high throughput access to a variety of data types. Alleviate load on backend systems.
Application Requirements
Lots of users accessing different datasets
Data in varying formats: HTML, JSON, protobuf
High read performance
Uptime critical
Legacy systems are hard and expensive to scale
The Couchbase Solution Based on memcached = fast!
Replicated and persistent with auto-failover
Fully distributed and clustered with “push button” scaling: easy, inexpensive
Support for binary and JSON data types
Challenges with other caching technologies
Complicated to setup and monitor
Not persistent
Restriction of supported data types
Not truly distributed or clustered (i.e. ehcache)
Poor performance under load
©2014 Couchbase Inc.
Caching @ ExperianThe problem
Massive and spikey user traffic to small bits of data supporting web experience
Mainframes are expensive to scale
New applications constantly coming online
Need to handle spikes in traffic
31
Experian.com
Provide fast access to credit information
The solution
Deploy Couchbase Server as standardized distributed caching layer
Compatible with memcached, highly optimized for latency and throughput
Shared nothing, replicated and persistent for reliability
Support for JSON as well as any binary data type
Shared-nothing, replicated and persistent architecture
The Couchbase AdvantageMassive speed and scale that’s easy to manage
©2014 Couchbase Inc.
Caching @ Experian
32
User Requests
Cache Misses and Write Requests
Mainframe
ApplicationLayer
CouchbaseDistributed Cache
Read-Write Requests