Date post: | 20-Aug-2015 |
Category: |
Technology |
Upload: | couchbase |
View: | 1,525 times |
Download: | 0 times |
Technical Evangelist
twi0er: @scalabl3email: [email protected]
Jasdeep Jaitla
Couchbase 105: XDCR and elasEcsearch
Replication Use Cases
• Unidirectional - Create Backup Cluster- Create Geographic Cluster, Chained Propagation- Create Development/Test Cluster with Active Data- Integrate with Elastic Search- Integrate with Custom Consumer
• Bi-Directional - Populate Multiple Clusters with Data Modified in each Cluster
Active-Active Replication• XDCR configured Per Bucket • Within Bucket, Each Partition (vbucket) Replicated Directly - Data is replicated from Active Partition -> Active Partition- Each Partition (1024) has a separate append only file!
- XDCR process:๏ Queues documents for replication๏ Transmits Documents via HTTP in batches๏ Keeps track of Checkpoint Intervals if Replication Streams are
Disconnected๏ Upon disconnection between clusters, automatically handles
reconnection attempts
8091
8092
8091
8092
XDCR Ports-Connections
MAPMAP
HTTP
Retrieve Target Cluster Map & Cluster Map
HTTP
Data Replica4on
16 GB RAM
512
Partitions
6 IO Workers
16 GB RAM
6 IO Workers512
Partitions
8 GB RAM
512
Partitions
3 IO Workers
8 GB RAM
3 IO Workers512
Partitions
Replication Follows Cluster Map
MAP
MAP
MAP
MAP
16 GB RAM
512
Partitions
6 IO Workers
16 GB RAM
6 IO Workers512
Partitions
8 GB RAM
512
Partitions
3 IO Workers
8 GB RAM
3 IO Workers512
Partitions
Replication Follows Cluster Map
MAP
MAP
MAP
MAP
16 GB RAM
512
Partitions
6 IO Workers
16 GB RAM
6 IO Workers512
Partitions
8 GB RAM
512
Partitions
3 IO Workers
8 GB RAM
3 IO Workers512
Partitions
Replication Follows Cluster Map
MAP
MAP
MAP
MAP
16 GB RAM
512
Partitions
6 IO Workers
16 GB RAM
6 IO Workers512
Partitions
8 GB RAM
512
Partitions
3 IO Workers
8 GB RAM
3 IO Workers512
Partitions
Replication Follows Cluster Map
MAP
MAP
MAP
MAP
Different Configura4on OK
8 GB RAM
512
Partitions
3 IO Workers256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
8 GB RAM
3 IO Workers512
Partitions
Cluster Sizes Don’t Have to Match
8 GB RAM
512
Partitions
3 IO Workers256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
8 GB RAM
3 IO Workers512
Partitions
Cluster Sizes Don’t Have to Match
Smaller Cluster
Cluster Sizes Don’t Have to Match
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
8 GB RAM
512
Partitions
3 IO Workers
8 GB RAM
3 IO Workers512
Partitions
Cluster Sizes Don’t Have to Match
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
256
8 GB RAM
Partitions
3 IO Workers
8 GB RAM
512
Partitions
3 IO Workers
8 GB RAM
3 IO Workers512
Partitions
Larger Cluster
Chaining XDCR
Partitions
1023
3210
102210211020
Partitions
1023
3210
102210211020
Partitions
1023
3210
102210211020
Chaining XDCR
Partitions
1023
3210
102210211020
Partitions
1023
3210
102210211020
Partitions
1023
3210
102210211020
All Writes
Chaining XDCR
Partitions
1023
3210
102210211020
Partitions
1023
3210
102210211020
Partitions
1023
3210
102210211020
All Writes
Unidirec4onal XDCR
Chaining XDCR
Partitions
1023
3210
102210211020
Partitions
1023
3210
102210211020
Partitions
1023
3210
102210211020
All Writes
Unidirec4onal XDCR 1.5
Intra-Cluster Node
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
Storage to XDCR Replication
Remote Cluster
Intra-Cluster Node
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
Storage to XDCR Replication
storage ops
Remote Cluster
Intra-Cluster Node
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
Storage to XDCR Replication
storage ops
Remote Cluster
Intra-Cluster Node
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
Storage to XDCR Replication
storage ops
Remote Cluster
Intra-Cluster Node
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
XDCR: Eventually Consistent
Remote Cluster
Intra-Cluster Node
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
XDCR: Eventually Consistent
storage ops
Remote Cluster
Time 1
Intra-Cluster Node
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
XDCR: Eventually Consistent
storage ops
Remote Cluster
Time 1
Time 2
Intra-Cluster Node
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
XDCR: Eventually Consistent
storage ops
Remote Cluster
Time 1
Time 2
Time 3
Bi-Directional Conflicts
Cluster 2
App Servers
Cluster 1
App Servers
What happens when you write to the same key in two clusters?
Bi-Directional Conflicts
Cluster 2
App Servers
Cluster 1
App Servers
replace(“key”, data)replace(“key”, data)
What happens when you write to the same key in two clusters?
Bi-Directional Conflicts
Cluster 2
App Servers
Cluster 1
App Servers
replace(“key”, data)replace(“key”, data)
What happens when you write to the same key in two clusters?
(conflict)
Conflict Resolution Rules• Completely Automatic, Currently Not Customizable
!
- Rule 1: Revision #/Count๏ Whichever cluster has higher revision number will replicate to other cluster๏ If document is mutated in cluster with lower revision number, the document
will not be put in replication queue
!
- Rule 2: Metadata๏ If Revision Count equal, use CAS, Flags, and TTL๏ Basically Random
!
• It is not Time/NTP based
Bi-Directional Caution!
Avoid (as in don’t) update the same document in multiple clusters with bi-directional XDCR, as it may not have expected behavior due to
the conflict resolution rules.!
Separate Key spaces (use DC prefix) so that you avoid conflicts on individual documents.
Recommended
DC1::user:a9838-‐s92-‐s00
DC2::user:293ba-‐293-‐922
The Document Life• Storage operation (set/add/replace/incr/decr)
• Stored in RAM Cache
๏ Can be Read by get operation
• Added to Disk Write Queue/Replication Queue
• Pushed to Replica Node RAM Cache (and DWQ)
๏ Exists after Failover
• Persisted to Active Partition Disk
• Added to XDCR Replication Queue
• Indexed by View Indexers (Design Document)
๏ Can be Queried in View
• Pushed to XDCR Remote Cluster RAM Cache
๏ Can be Read on Remote Cluster get operation
• Indexed by Elastic Search
๏ Can be Queried in Elastic Search
Replica Couchbase Intra-Cluster Machine
Architecture
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
Replica Couchbase Intra-Cluster Machine
Architecture
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
storage ops
Replica Couchbase Intra-Cluster Machine
Architecture
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
storage ops
Replica Couchbase Intra-Cluster Machine
Architecture
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
storage ops
Replica Couchbase Intra-Cluster Machine
Architecture
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
storage ops
Replica Couchbase Intra-Cluster Machine
Architecture
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
storage ops
Replica Couchbase Intra-Cluster Machine
Architecture
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
View Engine
Indexers
XDCR Replication Queue
Application Server
storage ops