Date post: | 22-Jan-2018 |
Category: |
Technology |
Upload: | daniel-cohen |
View: | 105 times |
Download: | 3 times |
Building Scalable, Real Time Applications for Financial Services
Simon Webster SVP & General Manager of EMEA
© DataStax, All Rights Reserved.
Feedback• Please use the Event App to give us your
feedback • Everyone who submits feedback will be entered
in the prize draw for a UE BOOM 2 by Ultimate Ears Bluetooth Waterproof Portable Speaker
• Winner will be notified via email
3
DataStax Enterprise Foundations for FinanceDaniel Cohen Solutions Engineer
© DataStax, All Rights Reserved.
But Enough About Me…• Solutions Engineer at DataStax • LA ➜ SF ➜ NYC ➜ SF ➜ London • Previously at JP Morgan in London • Morgan Stanley, UBS, Merrill Lynch…
5
© DataStax, All Rights Reserved.
But Enough About Me…• Solutions Engineer at DataStax • LA ➜ SF ➜ NYC ➜ SF ➜ London • Previously at JP Morgan in London • Morgan Stanley, UBS, Merrill Lynch…
5
© DataStax, All Rights Reserved.
But Enough About Me…• Solutions Engineer at DataStax • LA ➜ SF ➜ NYC ➜ SF ➜ London • Previously at JP Morgan in London • Morgan Stanley, UBS, Merrill Lynch…
5
Sandy
7
Sandy
8
Sandy
© DataStax, All Rights Reserved.
DataStax Enterprise
9
The database for cloud applications
© DataStax, All Rights Reserved.
1 Introductions
2 Technology Overview
3 Architectures
4 Use Cases
5 Questions?
10
© DataStax, All Rights Reserved.
What is Apache Cassandra?• A distributed NoSQL database – Google BigTable ✖ Amazon Dynamo
• Scale out with linear performance – Just add nodes
• Continuously available – Disaster avoidance, not disaster recovery
• Run on commodity hardware – In the cloud, on premise, or hybrid
11
© DataStax, All Rights Reserved.
Distributed Architecture• Fully distributed – Data spread over multiple nodes – All nodes participate in a cluster – Configurable data replication
• Masterless – All nodes are equal – Read from or write to any node
• No Single Point of Failure
12
Node 11st copy
Node 4
Node 5Node 2
2nd copy
Node 33rd copy
© DataStax, All Rights Reserved.
Scale Out Linearly• Need more storage? – Add more nodes.
• Need greater throughput? – Add more nodes.
• Predictable, linear performance gains
13
© DataStax, All Rights Reserved.
Scale Out Linearly• Need more storage? – Add more nodes.
• Need greater throughput? – Add more nodes.
• Predictable, linear performance gains
13
© DataStax, All Rights Reserved.
Scale Out Linearly• Need more storage? – Add more nodes.
• Need greater throughput? – Add more nodes.
• Predictable, linear performance gains
13
© DataStax, All Rights Reserved.
Linear Scale Out at Netflix
14
Source: The Netflix Tech Blog http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
© DataStax, All Rights Reserved.
A Quick Vocabulary Lesson
15
© DataStax, All Rights Reserved.
A Quick Vocabulary Lesson
15
Replication Factor – RF• Specifies how many copies of a datum C* distributes across nodes • Highly configurable for SLA & network topology (racks, data centers)
CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'CPH1' : 3, 'CPH2' : 3};
© DataStax, All Rights Reserved.
A Quick Vocabulary Lesson
15
Replication Factor – RF• Specifies how many copies of a datum C* distributes across nodes • Highly configurable for SLA & network topology (racks, data centers)
CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'CPH1' : 3, 'CPH2' : 3};
• Specifies how synchronized a row of data must be across C* nodes • Tunable consistency: client application decides CL for each operation
… ONE … LOCAL_QUORUM … QUORUM … EACH_QUORUM …
Consistency Level – CL
© DataStax, All Rights Reserved.
A Quick Vocabulary Lesson
15
Replication Factor – RF• Specifies how many copies of a datum C* distributes across nodes • Highly configurable for SLA & network topology (racks, data centers)
CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'CPH1' : 3, 'CPH2' : 3};
• Specifies how synchronized a row of data must be across C* nodes • Tunable consistency: client application decides CL for each operation
… ONE … LOCAL_QUORUM … QUORUM … EACH_QUORUM …
Consistency Level – CL
© DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
16
Node 2 2nd copy
Node 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
16
Node 2 2nd copy
Node 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
16
Write Consistency Level = QUORUM Replication Factor = 3 Node 2
2nd copy
Node 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
16
Parallel Write
Write Consistency Level = QUORUM Replication Factor = 3 Node 2
2nd copy
Node 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
16
Parallel Write
Write Consistency Level = QUORUM Replication Factor = 3
5 μs ack
Node 2 2nd copy
Node 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
16
Parallel Write
Write Consistency Level = QUORUM Replication Factor = 3
5 μs ack
12 μs ack
Node 2 2nd copy
Node 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
16
Parallel Write
Write Consistency Level = QUORUM Replication Factor = 3
5 μs ack
12 μs ack12 μs ack
Node 2 2nd copy
Node 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
16
Parallel Write
Write Consistency Level = QUORUM Replication Factor = 3
5 μs ack
12 μs ack
500 μs ack
12 μs ack
Node 2 2nd copy
Node 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
16
Parallel Write
Write Consistency Level = QUORUM Replication Factor = 3
5 μs ack
12 μs ack
500 μs ack
12 μs ack
Node 2 2nd copy
Node 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Read Consistency Level = QUORUM Replication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Read Consistency Level = QUORUM Replication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Parallel Read
Read Consistency Level = QUORUM Replication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Parallel Read
Read Consistency Level = QUORUM Replication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Parallel Read
Read Consistency Level = QUORUM Replication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Parallel Read
Read Consistency Level = QUORUM Replication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Parallel Read
Read Consistency Level = QUORUM Replication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Parallel Read
Read Consistency Level = QUORUM Replication Factor = 3
Hints
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Continuous Availability
17
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Parallel Read
Read Consistency Level = QUORUM Replication Factor = 3
Hints
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copy
Node 4
Node 5
© DataStax, All Rights Reserved.
Multiple Data Centers
18
Node 11st copy
Node 4
Node 5 Node 22nd copy
Node 33rd copy
Node 11st copy
Node 4
Node 5 Node 22nd copy
Node 3
London DCNew York DC
© DataStax, All Rights Reserved.
Disaster Avoidance
19
New York
London
Singapore
© DataStax, All Rights Reserved.
Disaster Avoidance
19
New York
London
Singapore
© DataStax, All Rights Reserved.
Disaster Avoidance
19
New York
London
Singapore
© DataStax, All Rights Reserved.
Disaster Avoidance
19
New York
London
Singapore
© DataStax, All Rights Reserved.
Mixed Workload DSE Cluster
Cassandra Only DC
Transactions
Workload Isolation
20
Cassandra+ Spark DCAnalytics
Analytical Application
Operational Application
© DataStax, All Rights Reserved.
• Syntax similar to RDBMS SQL • Create objects via DDL – e.g. CREATE…
• INSERT, UPDATE, DELETE • GRANT, REVOKE • SELECT…WHERE • Command line and GUI tools
CQL: Cassandra Query Language
21
CQL Example
CREATE TABLE market_prices ( symbol TEXT, date TIMESTAMP, price DECIMAL, side INT, PRIMARY KEY (symbol, date)) WITH CLUSTERING ORDER BY (date DESC);
© DataStax, All Rights Reserved.
Drivers & Connectors• Certified by DataStax: – Java, C# – Node.js, Python – Ruby, C / C++ – PHP, ODBC – Connector for Apache Spark
• Many community drivers – Go, Clojure – Erlang, Rust
22
© DataStax, All Rights Reserved.
1 Introductions
2 Technology Overview
3 Architectures
4 Use Cases
5 Questions?
23
© DataStax, All Rights Reserved.
Application Tier Resilience• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra – Sessions replicate naturally – Recover on disaster / failure
• UI ➜ intelligently stateful – Active failover to app server
24
AppServerCache
C* C*
AppServer Cache
DC LDN1 DC LDN2
© DataStax, All Rights Reserved.
Application Tier Resilience• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra – Sessions replicate naturally – Recover on disaster / failure
• UI ➜ intelligently stateful – Active failover to app server
24
AppServerCache
C* C*
AppServer Cache
DC LDN1 DC LDN2
© DataStax, All Rights Reserved.
Application Tier Resilience• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra – Sessions replicate naturally – Recover on disaster / failure
• UI ➜ intelligently stateful – Active failover to app server
24
AppServerCache
C* C*
AppServer Cache
DC LDN1 DC LDN2
© DataStax, All Rights Reserved.
Application Tier Resilience• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra – Sessions replicate naturally – Recover on disaster / failure
• UI ➜ intelligently stateful – Active failover to app server
24
AppServerCache
C* C*
AppServer Cache
session
DC LDN1 DC LDN2
© DataStax, All Rights Reserved.
Application Tier Resilience• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra – Sessions replicate naturally – Recover on disaster / failure
• UI ➜ intelligently stateful – Active failover to app server
24
C* C*
AppServer Cache
session
DC LDN1 DC LDN2
© DataStax, All Rights Reserved.
Application Tier Resilience• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra – Sessions replicate naturally – Recover on disaster / failure
• UI ➜ intelligently stateful – Active failover to app server
24
C* C*
AppServer Cache
session
DC LDN1 DC LDN2
© DataStax, All Rights Reserved.
Application Tier Resilience• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra – Sessions replicate naturally – Recover on disaster / failure
• UI ➜ intelligently stateful – Active failover to app server
24
C* C*
AppServer Cache
session
DC LDN1 DC LDN2
© DataStax, All Rights Reserved.
…A Resilient Enterprise Architecture
DC LDN2
App App
App App
DC NY1
App App
App App
DC LDN2
C*
DC LDN1
C*
DC NY2
C*DC NY1
C*
25
© DataStax, All Rights Reserved.
…A Resilient Enterprise Architecture
DC LDN2
App App
App App
DC NY1
App App
App App
DC LDN2
C*
DC LDN1
C*
DC NY2
C*DC NY1
C*
25
© DataStax, All Rights Reserved.
…A Resilient Enterprise Architecture
DC LDN2
App App
App App
DC NY1
App App
App App
DC LDN2
C*
DC LDN1
C*
DC NY2
C*DC NY1
C*
25
© DataStax, All Rights Reserved.
…A Resilient Enterprise Architecture
DC LDN2
App App
App App
DC NY1
App App
App App
DC LDN2
C*
DC LDN1
C*
DC NY2
C*DC NY1
C*
25
© DataStax, All Rights Reserved.
…A Resilient Enterprise Architecture
DC LDN2
App App
App App
DC NY1
App App
App App
DC LDN2
C*
DC LDN1
C*
DC NY2
C*DC NY1
C*
25
© DataStax, All Rights Reserved.
…A Resilient Enterprise Architecture
DC LDN2
App App
App App
DC NY1
App App
App App
DC LDN2
C*
DC LDN1
C*
DC NY2
C*DC NY1
C*
25
© DataStax, All Rights Reserved.
…A Resilient Enterprise Architecture
DC LDN2
App App
App App
DC NY1
App App
App App
DC LDN2
C*
DC LDN1
C*
DC NY2
C*DC NY1
C*
25
❓
© DataStax, All Rights Reserved.
…A Resilient Enterprise Architecture
DC LDN2
App App
App App
DC NY1
App App
App App
DC LDN2
C*
DC LDN1
C*
DC NY2
C*DC NY1
C*
25
❌
© DataStax, All Rights Reserved.
…A Resilient Enterprise Architecture
DC LDN2
App App
App App
DC NY1
App App
App App
DC LDN2
C*
DC LDN1
C*
DC NY2
C*DC NY1
C*
25
❌
© DataStax, All Rights Reserved.
Unify & Scale Legacy Infrastructure
26
…USA Equities
UK FX
UK Bonds
Global Users
Legacy Systems
USA FX
DataStax Enterprise ClusterC*
User Interface / Application Services
© DataStax, All Rights Reserved.
Microservices on Legacy
27
Users µServices
DC NY1A E
F T
DC LDN1A B
E F
Messages
DC NY1
DC LDN1
DC NY1
DC LDN1
C*
C*
USA Equities
DSE
UK FX
Legacy
© DataStax, All Rights Reserved.
DSE Real-time Analytics Reference Architecture
HTTP Application Message Queue
Streaming Analytics
Batch Analytics
Real-time
28
© DataStax, All Rights Reserved.
1 Introductions
2 Technology Overview
3 Architectures
4 Use Cases
5 Questions?
29
© DataStax, All Rights Reserved.
DataStax Use Cases• Customer 360° • Master data management • Customer profile management • Authentication & identity management • Product personalization • Anti-fraud & money laundering • Payments & transactions • Risk reporting / capital adequacy • Market data capture / replay
30
© DataStax, All Rights Reserved.
Customer 360° / User Profile• Consolidate data, reduce complexity • Alleviate strain on legacy systems • Integrated data sovereignty • Eliminate downtime • Recommend higher margin products • Customer personalization • Audit and regulatory compliance • Full text indexing and document search
31
“DataStax has allowed us to break the complexities of our legacy systems, empowering us to leverage data to innovate and make more informed decisions so we can provide a truly personalized and premium experience to our customers.”
© DataStax, All Rights Reserved.
Payments & Transactions• Transaction history tagging, search and
budgeting, cash flow prediction • Product recommendation based on
purchasing history • Data safe against loss of node, rack, or
data center • Shed load from legacy systems to reduce
cost and provide better user experience
“We had reached the limits of our scalability. We wanted to remove all single points of failure. We had to be active-active. That's why we chose Cassandra.”
32
© DataStax, All Rights Reserved.
Regulatory Compliance“The platform enables traders to capture vast amounts of versioned market data which is then used for risk management purposes in real time.”
33
• Capture and analyze every trade, every position for every trader
• Real-time risk analysis • Capital adequacy • MiFID II • Globally replicated, 100% uptime • Low latency, high ingest
© DataStax, All Rights Reserved.
“DataStax Enterprise gives us the power to change information in our supporting systems into actionable insights… we can not only scale but give our users a real-time, engaging customer experience.”
• Financial product recommendation • Provide the “small bank” feel through
personalization • Real-time recommendation using transaction
data, search, and browsing history • Measure engagement and satisfaction to
prevent flight • Website and mobile metrics • Prevent customer churn
Recommendations & Engagement
34
© DataStax, All Rights Reserved.
Market Tick & Time Series Data• Capture high fidelity time series
information at high ingest rates and low latencies
• 100% uptime even in the event of data center loss
35
“We primarily use Cassandra as a distributed data store for Financial Instrument and Time-Series data, where the ‘master’ data stores cannot scale globally and provide the high availability needed to support a global application with heavy user demand.”
© DataStax, All Rights Reserved.
Mission Critical Applications
36
“DSE is a perfect match… easing availability challenges by being active-active and
having an always-on architecture”
Source: Exploiting Hotel Cassandra ➜ http://bit.ly/1mf0KRE