Post on 22-Aug-2015
transcript
What’s New in CouchbaseTop 10 Capabilities in 3.0
Dipti BorkarSr. Director | WW Solution Engineering
©2014 Couchbase Inc.
Couchbase provides a complete Data Management solution
3
High availability
cache
Key-value store
Document
database
Embedded database
Sync management
Multi-purpose capabilities support a broad range of apps and use cases
Enterprises often start with cache, then broaden usage to other apps and use cases
©2014 Couchbase Inc.
What makes Couchbase unique?
4
Performance & scalability
leaderSub millisecond latency with high throughput; memory-centric architecture
Multi-purpose
Simplified administrationEasy to deploy & manage; integrated Admin Console, single-click cluster expansion & rebalance
Cache, key value store, document database, and local/mobile database in single platform
Always-on availability
Data replication across nodes, clusters, and data centers
Enterprises choose Couchbase for several key advantages
24x365
©2014 Couchbase Inc.
Journey so far…
Couchbase Server v1.8High Scale Key/Value Database
Couchbase Server v2.1 Improved XDCRAdvanced Storage Engine
Couchbase Server v2.2Enhanced Security HA/DR
Couchbase Server v2.5Rack/Zone Awareness Connection Management
Couchbase Mobile 1.0Offline Data Availability
Auto Synchronization
Couchbase Server v3.0Mission Critical Scale & Perf
Simplified AdministrationImproved HA/DR & Security
2010
Couchbase Server 4.0With N1QLSQL for Documents
Couchbase Server v2.0Document ExtensionsNew XDCR
©2014 Couchbase Inc.
DEVELOPER EMPOWERMENT
MISSION CRITICAL SCALE
ULTRA HIGH AVAILABILITY
SECURE COMMUNICATION
SIMPLIFIED ADMINISTRATION
• Fully Integrated JSON Programmability
• Native Async and Reactive Programmability
• New Integration with Popular Frameworks
• N1QL Preview Support in SDKs
• Optimized Massive Databases with Tunable Memory
• Auto-Tuning Thread Pool for Extreme IO Throughput
• Faster Replication with DCP
• Improved Durability
• Faster view building
• Low Latency, In-Memory XDCR
• Delta Node Recovery
• Faster Rebalance
• Faster Node Restart with Efficient Warm-up
• New Incremental Backup option
• Encryption for Data Access
• Encryption for Administrative Access
• XDCR Pause and Resume
• Improved Resource Governance
• Graceful Failover
• Cluster-wide Diagnostics
What’s new in Couchbase Server v3.0
©2014 Couchbase Inc.
New DCP (Database Change Protocol) protocol for cluster-wide communication Eliminate Disk-IO from the replication access path High-Performance, Stream-based, In-memory Ordered based on global Sequence Numbers per partition Snapshot-able Restart-able
• Powers Node to Node Replication• Powers View • Powers XDCR
Database Change Protocol
This is non-trivial stuff!!!
9©2014 Couchbase Inc.
33 2
XDCR after Write
Managed Cache
Dis
k Q
ueue
Disk
Replication Queue
App Server
Couchbase Server Node
Doc 1
Doc 1
XDCR Queue
Doc 1
Doc 1
(New in 3.0) Memory-to-Memory Replication to remote cluster
Memory-to-Memory Replication to other node
10©2014 Couchbase Inc.
33 2
View processing after write
Managed Cache
Dis
k Q
ueue
Disk
Replication Queue
App Server
Couchbase Server Node
Doc 1
Doc 1
To other node
View engine Doc 1
Doc 1
©2014 Couchbase Inc.
Optimized Memory Usage with Metadata Ejection Policy Better optimization of memory for massive databases Enable efficient management of rarely accessed data set Cache only keys and data for the working set & eject all
historic data
Tunable Memory - Optimization for Massive Databases
©2014 Couchbase, Inc. ©2014 Couchbase Inc.
Each hash table item consists of {key, metadata, value} Metadata memory overhead is 40 bytes at least
A Key in the Hash Table
13
Key Metadata Blob pointer
Blob value
Expiration time
CAS identifier
Sequence number (DCP)
Revision number (XDCR)
Lock expiry (GetLocked API)
Flag, NRU, …
Hash Table Item
Pointer to next item
©2014 Couchbase, Inc. ©2014 Couchbase Inc.
Application’s entire key space is maintained in the hash table
Highly cache-oriented architecture
Item pager ejects only an item’s value from the hash table
Value-Only Ejection
14
Key: “foo” Metadata Blob pointer
Blob valuePointer to next item
Hash Table ItemStorage Engine
Batch Reader
Get(“foo”)
read_value(“foo”)
©2014 Couchbase, Inc. ©2014 Couchbase Inc.
Application’s entire key space doesn’t need to be loaded in cache
Reduce the memory overhead significantly in heavy DGM (Disk Greater than Memory) cases
Item pager ejects an item’s key and metadata along with its value
Full Metadata Ejection
15
Key: “foo” Metadata Blob pointer
Blob valuePointer to next item
Hash Table ItemStorage Engine
Batch Reader
Get(“foo”)
read_meta_value(“foo”)
©2014 Couchbase Inc.
Tunable Memory - Optimization for Massive Databases
100sx Reduction in Metadata Memory Consumption from 2.5 to 3.0
Note: The graph represents characteristics under data mutations. ~50M docs with value size ~0.5KB
v.3.0 – Large DB with Hot Working-set
3 GB Consumed for Metadata in RAM
2.5.1 or earlier – Real-time Latency
80 MB Consumed for Metadata in RAM
©2014 Couchbase Inc.
Delta Node Recovery Shorter time to add nodes back into the cluster! Bring back a server incrementally into the cluster without
having to rebuild the full node.
Incremental Node Catch-up with Delta Node Recovery
100sx Reduction in Time to Re-Add Node from 2.5 to 3.0Note: The absolute performance improvement depend on data size and mutation count that needs to be caught up
©2014 Couchbase Inc.
Efficient Auto-Tuning Engine Detect and allocate threads based on HW resources Pool threads for best resource utilization across buckets Improved latency across the board:
Faster ReadsFaster RebalanceFaster Node reactivation Faster Durability with Writes & PersistTo (more
details on this on later)…and many more
Auto Tuning Thread Pool
©2014 Couchbase, Inc. ©2014 Couchbase Inc.
Inside data manager
21
Append-only B-Tree Storage Engine
Engine APIs(get, set, del, add, append, DCP,
…)
PartitionHash Table
(active)
PartitionHash Table
(replica)
PartitionHash Table
(active)…
Checkpoints
Checkpoints
Checkpoints
ReaderThreads
Non-IO Thread
s
DataReplicato
r
I/O Completion
Notifier
Aux-IOThreads
FlushersData
Backfill
User Configured Replica Count = 1
Batch Readers
WriterThreads
…
Item Pager
Expiry Pager
Checkpoint Manager
Shared Thread Pool
©2014 Couchbase Inc.
Efficient warm-up Parallelization on cache warm-up Further warm-up time improvements under metadata ejection
Up to 60% Faster Warm-up between 3.0 and 2.5.1 Up to 8x Faster Warm-up with Full Ejection in 3.0
*Note: Absolute latency depend on disk IO latency and bandwidth
Faster Node Re-Activation
©2014 Couchbase Inc.
Priority Setting on Buckets Ability to assign high and low priority to buckets High priority buckets get higher IO priority and more
throughput Low priority buckets can still be processed without
starvation
Improved Resource Governance with Bucket Priority
©2014 Couchbase Inc.
Faster Durability with Replication
Improved latency on ReplicateTo DCP Improves replication speed and durability Better protects your data!
Up to 150x Improvement on ReplicateTo latency from 2.5 to 3.0
*Note: Absolute latency depend on network bandwidth and latency
*95th percentile ReplicateTo=1 latency (ms), 1 bucket x 200M x 1KB, 250 mutations/sec/node
Faster Replication within the cluster & across clusters
0
200
400
600
2.5.1 3.0
95 Percentile Repli-cateTo=1 Latency*
Tim
e (
ms)
©2014 Couchbase Inc.
Improved Protections for Your Data
Drastically Lower XDCR Latency In memory change detection with DCP
No longer require persisting to disk
Up to 4x better on XDCR latency between clusters between 3.0 & 2.5.1
*Note: Absolute latency depend on WAN latency and bandwidth
Lower Latency across clusters
0
100
200
300
400
500
600
2.5.1 3.0
90th Percentile. XDCR Latency
Tim
e (
ms)
90th percentile replication lag (ms), 5 -> 5 UniDir, 2 buckets x 500M x 1KB, 10K SETs/sec, LAN
©2014 Couchbase Inc.
More up-to-date views Mutation show up at the
indexes faster with the new DCP protocol
No longer require persisting to disk for change detection
Improved consistency with views (stale=false behavior)
Ports to C in view engine
Very fast index building
05000
10000150002000025000300003500040000
2.5.1 3.0
Indexing Under Load
Tim
e (
ms)
Up to 50x Faster Indexing from 2.5 to 3.0*Note: Absolute latency depend on disk IO speed
*95th percentile Indexing latency (ms), 1 bucket x 20M x 2KB, non-DGM, 1 view, 250 mutations/sec/node, 400 queries/sec
©2014 Couchbase Inc.
Minimize time and resources during backups
Efficient Recovery with Incremental Backup & Restore
• Back up only the data updated since the last backup
• Differential Backups• Cumulative Backups
©2014 Couchbase Inc.
XDCR Pause and Resume Want to suspend replication? No longer need to drop
replications Ability to pause XDCR traffic anytime Resume and incrementally catch-up from where you left
off
Efficient Maintenance with XDCR
34©2014 Couchbase Inc.
Connection Management
Optimized Connection Management
Cluster map / topology information transmitted via 11210 port rather than 8091
Much more scalable and better performance during topology changes
35©2014 Couchbase Inc.
Connection Management
Increased connection limits
Data port : 11210 max_conns_on_port_11210: 30000Internal Cluster communication : 11209 max_conns_on_port_11209: 5000
©2014 Couchbase Inc.
New option to gracefully failover a node. Failover with a graceful handoff May take longer to failover but prevents potential for any
data loss
Node Maintenance
©2014 Couchbase Inc.
Componentized libraries to simplify use cases• Couchbase JVM Core - Reusable Foundation
• Handles core IO, topology changes. Reactive and asynchronous.• Java Client Library
• Native JSON handling built-in• Provides synchronous and asynchronous interface• Preview of N1QL language integrated query
What’s New in SDK 2.0
©2014 Couchbase Inc.
Integrated native JSON data type programmability
• Simplified Development • Native JSON handling built-in• Preview of N1QL language integrated query
• Performance & Scale• Improved connection management
• Native .Net: • Simplified Configuration Management with Web.config & App.config
What’s New in SDK 2.0
©2014 Couchbase Inc.
Node.js, PHP• Native JSON handling built-in• Core IO enhancements through updated C libcouchbase• Better connection management
Python, Ruby• Core IO enhancements through updated C libcouchbase• Better connection management
What’s New in SDK 2.0
Couchbase Roadmap2015 – Couchbase Server 4.0
POWERFUL QUERYPERFORMANCE &
SCALEHIGH
AVAILABILITYSECURITY
COMPLIANCEDEVELOPER
EMPOWERMENT
• N1QL: SQL for Documents
• Secondary indexing for interactive queries
• Improved Spatial Queries
• Multi Dimensional Scaling
• Scalability Enhancements for Massive Database Deployments
• Improve Performance for Full Eviction – Bloom Filters
• Filtering Replication for Efficient Geo Data Distribution with XDCR
• Auditing Framework and Admin Auditing
• External Account Management System Integration for the Enterprise (LDAP)
• Enhanced Consistency Options
• Legacy Data interfaces ODBC/JDBC
©2014 Couchbase Inc.
Introduction to N1QL – SQL for Documents Next generation, NoSQL query language
SQL-like : SELECT * FROM WHERE/LIKE/JOIN/GROUP/etc, CREATE INDEX Extended for JSON to support nested and hierarchical data structures Support for views and newly-developed secondary indexes Query (DQL), Manipulation (DML), Description (DDL) ODBC/JDBC drivers in development
Built into Couchbase Server: Single installation package Multi-threaded, stateless query and indexing components Leverages high-performance, high-scale Couchbase buckets
Coming in 2015, preview at query.couchbase.com
©2014 Couchbase Inc.
N1QL Architecture
Single node installation, services defined dynamically
Query service access Index and Data to formulate response
All queries and direct access is topology aware and dynamically scalable