Cassandra and Clojure

transcript

Nick Bailey

OpsCenter Architect

Who am I?• OpsCenter Architect

• Monitoring/management tool for Cassandra

• Organizer of Austin Cassandra Users• http://www.meetup.com/Austin-Cassandra-Users/

• Third Thursday each month. Come join!

• Working with Cassandra for 4 years

Cassandra - An introduction

Cassandra - Intro

• Based on Amazon Dynamo and Google BigTable papers

• Shared nothing

• Distributed

• Predictable scaling

Dynamo

BigTable

Cassandra - Architecture

Cassandra - Cluster Architecture

• All nodes participate in a cluster

• Shared nothing

• Add or remove as needed

• More capacity? Add a server

Cassandra - Data Distribution

• Each node owns 1 or more “tokens”

• Each piece of data has a “partition key”

• Partition key is hashed to determine token

• Hashes:

• Murmur3 (default)

• Md5

Cassandra - Replication

• Client writes to any node

• Node coordinates with replicas

• Data replicated in parallel

• Replication factor (RF): How many copies of your data?

Cassandra - Failure Modes

• Consistency level

• How many nodes?

• ONE/QUORUM/ALL

Cassandra - Geographically Distributed

• Client writes local

• Data syncs across WAN

• Replication Factor per DC

• Consistency Level

• LOCAL_QUORUM

Datacenter East Datacenter West

Data Modeling - Concepts

CQL• Cassandra Query Language

• SQL-like

• Not Relational

Terminology• Keyspace

• Table (Column Family)

• Row

• Column

• Partition Key

• Clustering Key

Data Typescqlsh:clojure_cassandra_demo> help types

CQL types recognized by this version of cqlsh:

ascii bigint blob boolean counter decimal double float inet int list map set text timestamp timeuuid uuid varchar varint

Advanced Concepts• Lightweight Transactions

• Atomic Batches

• User Defined Types (coming soon)

Data Modeling - An Example

Approaching Data Modeling• Model your queries, not your data

• Generally, optimize for reads

• Denormalize!

• Iterate!

Basic Last.fm Clone• See songs that user X has listened to recently

• See user X’s favorite songs in a specific month

• See who has recently listened to artist Y

• See artist Y’s most popular songs in a specific week

• One of the most common patterns/data models

• Time series

• Immutable (good fit for Clojure!)

SELECT song, artist, played_at FROM user_history WHERE username = ‘nickmbailey’ORDER BY played_at DESC;

• Partition key = ‘username’

• Clustering key = ‘played_at’

CREATE TABLE user_history ( username text, played_at timestamp, album text, artist text, song text, PRIMARY KEY (username, played_at)) WITH CLUSTERING ORDER BY (played_at DESC)

• This table has a “bad” partition key

CREATE TABLE user_history ( username text, played_at timestamp, album text, artist text, song text, PRIMARY KEY (username, played_at)) WITH CLUSTERING ORDER BY (played_at DESC)

• Much better partition key

CREATE TABLE user_history ( username text, year_and_month text, played_at timestamp, album text, artist text, song text, PRIMARY KEY ((username, year_and_month), played_at)) WITH CLUSTERING ORDER BY (played_at DESC)

cqlsh:clojure_cassandra_demo> select * from user_history limit 5;

Partition Key - unordered Clustering Key - Ordered

Basic Last.fm Clone• See user X’s favorite songs in a specific month

SELECT song, artist, play_count FROM user_history WHERE username = ‘nickmbailey’ AND month = ‘July’ORDER BY play_count DESC;

• Partition key = ‘username’, ‘month’

• Clustering key = ‘play_count’?

• Counters are a special case

Counters• Counter can not be part of the PRIMARY KEY

• No ordering based on counter value

• All non counter columns must be part of the PRIMARY KEY

• Limitations due to the storage format

CREATE TABLE user_song_counts ( username text, year_and_month text, artist text, song text, play_count counter, PRIMARY KEY ((username, year_and_month), artist, song))

• Results unordered• Client will have to do the sorting

cqlsh:clojure_cassandra_demo> select * from user_song_counts where username = 'nickmbailey' and year_and_month = '2014-07';

Basic Last.fm Clone• See who has recently listened to artist Y

CREATE TABLE artist_history ( artist text, year_and_week text, played_at timestamp, album text, song text, username text, PRIMARY KEY ((artist, year_and_week), played_at)) WITH CLUSTERING ORDER BY (played_at DESC)

Basic Last.fm Clone• See artist Y’s most popular songs in a specific week

CREATE TABLE artist_song_counts ( artist text, year_and_week text, album text, song text, play_count counter, PRIMARY KEY ((artist, year_and_week), album, song))

Cassandra from Clojure

Building Blocks

• Java Driver

• Hayt

Java Driver

• Fully featured

• Connection pooling

• Failover policies

• Retry policies

• Sync and Async interfaces

• Exposes client metrics

• https://github.com/datastax/java-driver

• CQL DSL

• Similar to Korma

• Solely for building CQL strings

• https://github.com/mpenet/hayt

(select :foo (where { :bar 1

:baz 2)})

(->raw (select :foo (where {:bar 1 :baz 2)}))> "SELECT * FROM foo WHERE bar = 1 AND baz = 2;"

Clients

• Alia

• https://github.com/mpenet/alia

• Cassaforte

• https://github.com/clojurewerkz/cassaforte

• Both built on Java Driver and Hayt

• Not particularly different

Alia vs. Cassaforte

Cassaforte(let [conn (cc/connect ["127.0.0.1"])] (cql/create-keyspace conn "cassaforte_keyspace" (with {:replication {:class "SimpleStrategy" :replication_factor 1 }})))

Alia(def cluster (alia/cluster {:contact-points ["localhost"]}))(def session (alia/connect cluster))(alia/execute session

(create-keyspace :alia (if-exists false) (with {:replication {:class "SimpleStrategy" :replication_factor 1}})))

Learn by Example - Alia

Cluster Object

• Entry point

• Configures relevant client options

• :contact-points

• :load-balancing-policy

• :reconnection-policy

• :retry-policy

• and more!

(def cluster (alia/cluster {:contact-points ["localhost"]}))

Session Object

• A Session is associated with a keyspace

• Allows interacting with multiple keyspaces

(def cluster (alia/cluster {:contact-points [“localhost"]}))(def session (alia/connect cluster))(def session (alia/connect cluster) :my_keyspace)

Querying

• Multiple ways to query

• alia/execute

• Synchronous, block on result

• alia/execute-async

• Returns a Lamina result-channel (basically, a promise)

• Optional success/error callbacks

• alia/execute-chan

• Returns a core.async channel

• We won’t dive in to core.async now

Prepared Statements

• Statements can be prepared server side

• Better performance for common queries

(def prepared-statement (alia/prepare session "select * from users where user_name=?;"))

What else?

• See github and docs

• https://github.com/mpenet/alia

• http://mpenet.github.io/alia/qbits.alia.html

• https://github.com/nickmbailey/clojure-cassandra-demo

• Built with

• CCM - https://github.com/pcmanus/ccm

• Alia - https://github.com/mpenet/alia

• ring - https://github.com/ring-clojure/ring

• compojure - https://github.com/weavejester/compojure

• hiccup - https://github.com/weavejester/hiccup

• least - https://github.com/Raynes/least

MoreCassandra: http://cassandra.apache.org

DataStax Drivers: https://github.com/datastax

Documentation: http://www.datastax.com/docs

Getting Started: http://www.datastax.com/documentation/gettingstarted/index.html

Developer Blog: http://www.datastax.com/dev/blog

Cassandra Community Site: http://planetcassandra.org

Download: http://planetcassandra.org/Download/DataStaxCommunityEdition

Webinars: http://planetcassandra.org/Learn/CassandraCommunityWebinars

Cassandra Summit Talks: http://planetcassandra.org/Learn/CassandraSummit

Cassandra and Clojure

Software