Writing a cassandra Client with perl

transcript

Writing a Cassandra client in Perl

Stefano Rodighierostefano.rodighiero@gmail.com

@larsen

• High Write Throughput

• Large Tables

• Reports

• Analytics

Cassandra

Bigtable: A distributed storage system for structured data, 2006

Dynamo: Amazon's highly available keyvalue

store, 2007

Dynamo partitioning and

replication

ColumnFamily data model similar

to Bigtable's

High availability

Incremental scalability

Eventually consistent

Tunable tradeoffs between consistency

and latency

Minimal administration

(not quite so)

No SPOFp2p distribution model

Writing

Writinginsert(key, {column_name => value})

Writing

17% 17%R

Writing

Writing a Cassandra client in Perl

Talking with a server

Cassandra::CassandraClient

Thrift::BinaryProtocol

Thrift::FramedTransport

Thrift::Socket

The Transport layer provides a simple abstraction for reading/writing from/to the network. This enables Thrift to decouple the underlying transport from the rest of the system (serialization/deserialization, for instance).

Thrift::BinaryProtocolThe Protocol abstraction defines a mechanism to map in-memory data structures to a wire-format. In other words, a protocol specifies how datatypes use the underlying Transport to encode/decode themselves.

Cassandra::CassandraClient"High level" interface: login, reading & writing data, …

1) Install Cassandra2) Install Thrift

thrift -‐gen perl cassandra.thrift

thriftservice Cassandra { # auth methods void login(1: required AuthenticationRequest auth_request) i throws (1:AuthenticationException authnx, 2:AuthorizationException authzx), …

Perl (generated code)package Cassandra::CassandraClient;…

sub login{ my $self = shift; my $auth_request = shift;

$self-‐>send_login($auth_request); $self-‐>recv_login();}

Perl (generated code)package Cassandra::CassandraClient;…

sub login{ my $self = shift; my $auth_request = shift;

$self-‐>send_login($auth_request); $self-‐>recv_login();}

thrift

Perl (generated code)

struct AuthenticationRequest { 1: required map<string, string> credentials}

package Cassandra::AuthenticationRequest;use base qw(Class::Accessor);Cassandra::AuthenticationRequest-‐>mk_accessors( qw( credentials ) );

sub new { …}

my $ks = 'TestIPW';

my $socket = Thrift::Socket-‐>new($server_name, $server_port);my $transport = Thrift::FramedTransport-‐>new($socket, 1024, 1024);my $protocol = Thrift::BinaryProtocol-‐>new($transport);$client = Cassandra::CassandraClient-‐>new( $protocol );

eval { $transport-‐>open; my $auth = Cassandra::AuthenticationRequest-‐>new( { credentials => {} } ); $client-‐>login($auth);

$client-‐>set_keyspace($ks);};if ( $@ ) { die Dumper( $@ );}

my $ks = 'TestIPW';

Thrift::Socket

my $ks = 'TestIPW';

Reading

thriftlist<ColumnOrSuperColumn> get_slice( 1:required binary key, 2:required ColumnParent column_parent, 3:required SlicePredicate predicate, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)

Perl (generated code)package Cassandra::CassandraClient;…sub get_slice{ my $self = shift; my $key = shift; my $column_parent = shift; my $predicate = shift; my $consistency_level = shift;

$self-‐>send_get_slice($key, $column_parent, $predicate, $consistency_level); return $self-‐>recv_get_slice();}

thrift

struct ColumnParent { 3: required string column_family, 4: optional binary super_column,}

package Cassandra::ColumnParent;use base qw(Class::Accessor);Cassandra::ColumnParent-‐>mk_accessors( qw( column_family super_column ) );

thrift

struct SlicePredicate { 1: optional list<binary> column_names, 2: optional SliceRange slice_range,}

package Cassandra::SlicePredicate;use base qw(Class::Accessor);Cassandra::SlicePredicate-‐>mk_accessors( qw( column_names slice_range ) );

thrift

struct SliceRange { 1: required binary start, 2: required binary finish, 3: required bool reversed=0, 4: required i32 count=100,}

package Cassandra::SliceRange;use base qw(Class::Accessor);Cassandra::SliceRange-‐>mk_accessors( qw( start finish reversed count ) );

thriftenum ConsistencyLevel { ONE = 1, QUORUM = 2, LOCAL_QUORUM = 3, EACH_QUORUM = 4, ALL = 5, ANY = 6, TWO = 7, THREE = 8,}

package Cassandra::ConsistencyLevel;use constant ONE => 1;use constant QUORUM => 2;use constant LOCAL_QUORUM => 3;use constant EACH_QUORUM => 4;use constant ALL => 5;use constant ANY => 6;use constant TWO => 7;use constant THREE => 8;

my $cf = '20121011.larsen.tweets';

my $columnParent = Cassandra::ColumnParent-‐>new({column_family => $cf});my $sliceRange = Cassandra::SliceRange-‐>new({ start => '', finish => '' });

my $predicate = Cassandra::SlicePredicate-‐>new({ slice_range => $sliceRange });

$res = $client-‐>get_slice( $key, $columnParent, $predicate, Cassandra::ConsistencyLevel::ONE);

thrift

struct SliceRange { 1: required binary start, 2: required binary finish, 3: required bool reversed=0, 4: required i32 count=100,}

Writing

thrift

void insert(1:required binary key,· 2:required ColumnParent column_parent, 3:required Column column, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)

sub insert{ my $self = shift; my $key = shift; my $column_parent = shift; my $column = shift; my $consistency_level = shift;

$self-‐>send_insert( $key, $column_parent, $column, $consistency_level); $self-‐>recv_insert();}

thrift

void insert(1:required binary key,· 2:required ColumnParent column_parent, 3:required Column column, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)

sub insert{ my $self = shift; my $key = shift; my $column_parent = shift; my $column = shift; my $consistency_level = shift;

$self-‐>send_insert( $key, $column_parent, $column, $consistency_level); $self-‐>recv_insert();}

thrift

package Cassandra::Column;use base qw(Class::Accessor);Cassandra::Column-‐>mk_accessors( qw( name value timestamp ttl ) );

struct Column { 1: required binary name, 2: optional binary value, 3: optional i64 timestamp, 4: optional i32 ttl,}

eval { my $cf = 'TestCF'; my $key = 'bar';

my $columnParent = Cassandra::ColumnParent-‐>new({ column_family => $cf });

my $ts = sprintf "%.6f", Time::HiRes::time; $ts =~ s/\.//; say $ts;

my $column = Cassandra::Column-‐>new({ name => 'test', value => 'a test value', timestamp => $ts });

$client-‐>insert( $key, $columnParent, $column, Cassandra::ConsistencyLevel::ONE);};if ( $@ ) { die Dumper( $@ );}

eval { my $cf = 'TestCF'; my $key = 'bar';

my $columnParent = Cassandra::ColumnParent-‐>new({ column_family => $cf });

my $ts = sprintf "%.6f", Time::HiRes::time; $ts =~ s/\.//; say $ts;

my $column = Cassandra::Column-‐>new({ name => 'test', value => 'a test value', timestamp => $ts });

$client-‐>insert( $key, $columnParent, $column, Cassandra::ConsistencyLevel::ONE);};if ( $@ ) { die Dumper( $@ );}

thrift

struct Column { 1: required binary name, 2: optional binary value, 3: optional i64 timestamp, 4: optional i32 ttl,}

Net::Cassandra ✘Net::Cassandra::Easy ✘

Cassandra::Lite ✘AnyEvent::Cassandra ✘

Questions?

Grazie! ☺Stefano Rodighierostefano.rodighiero@gmail.com

@larsen

Writing a cassandra Client with perl

Documents