Apache cassandra en production - devoxx 2017

#DevoxxFR

Apache Cassandra

En production

#DevoxxFR @jxerome @alexanderDeja

Qui sommes-nous ?

Jérôme MainaudArchitecte Java @Ippon Technologiescertifié Datastax Enterprise Solution Architect@jxerome

Alexander DejanovskiConsultant Apache Cassandra @The Last PickleDatastax MVP for Apache Cassandra / Big Data Hebdo@alexanderDeja


Qui êtes-vous ?

Levez la main si vous utilisez déjàApache Cassandra en production


Qui êtes-vous ?

Levez la main si vous utilisez déjàApache Cassandra en production

...depuis plus de 3 mois


Qui êtes-vous ?

Levez la main si vous allez bientôt utiliser Apache Cassandra en production


Qui êtes-vous ?

Levez la main si vous n’avez jamais utilisé Apache Cassandra


Agenda

- Rappels- Modélisation- Hardware- Versions de C*- Configuration et tuning GC- Sécurité- Ajout / retrait de noeuds- Multi DC- Compaction

- Purge- Upgrade- Réparation- Backup- Monitoring- Outillage- Problèmes courants- Bonus


Déconseillé aux personnes sensibles

AvertissementDes scènes d'incident de production peuvent

heurter la sensibilité d'un responsable de production


Rappels

Donné à la fondation Apache en 2008 par FacebookUtilisé massivement par Apple, Netflix, Spotify, Instagram, etc...Scalabilité horizontale linéaireCodé en Java


Rappels - Partitionnement et réplication

Sans VNodes



Avec VNodes



CREATE TABLE myks.etape( id_flux varchar, id_etape varchar, date_heure_debut timestamp, date_heure_fin timestamp, status_etape int PRIMARY KEY (id_flux, id_etape, date_heure_debut));



CREATE TABLE myks.etape( id_flux varchar, id_etape varchar, date_heure_debut timestamp, date_heure_fin timestamp, status_etape int PRIMARY KEY (id_flux, id_etape, date_heure_debut));


Rappels - Niveau de cohérence (Consistency level)

ONE / LOCAL_ONE >> Cohérence in-fineQUORUM / LOCAL_QUORUM / EACH_QUORUM >> Cohérence forteALL >> Personne n’utilise ça…ANY >> Les gens qui n’aiment pas leur travail l’utilisent...

14


Rappels - Write path


Rappels - Flush


Rappels - Read Path


Rappel - Read Path (complet)

Memtable

SSTables

LectureRéponse

Row Cache(si actif)

Mémoire

Disque

BloomFilter

BloomFilter

BloomFilter

KeyCache

PartitionSummary

PartitionIndex

On heap Off heap

Trouvé


Des questions ?


Modélisation


Hardware

8 à 16 coeurs16 Go mini - 32 à 64 Go en sweet spot

SSD de préférence 1 à 4 To (2 To utiles max)


Hardware

AWSI2.2xlarge (8c+61Go) / 4xlarge (16c+122Go) (SSD)

I3.2xlarge (8c+61Go) / 4xlarge (16c+122Go) (NVMe)M4.2xlarge (8c+32Go) / 4xlarge (16c+64Go) (EBS)EBS gp2 : 900Go Commit Log + 3.5To Data


Hardware

OVHServeurs dédiés

EG-32 (8 t – 32 Go – 3 * 600 Go HDD RAID 0)EG-64 (12 t – 64 Go – 4 * 800 Go SSD RAID 0)

EG-16 (8 t – 16 Go – 2 * 450 Go NVMe)


Hardware

Un SAN avec Cassandra ?


Hardware - Pourquoi pas de SAN ?

SPOFRéplication redondanteProblèmes de perfs (flush, compaction, repair, hints, etc…)http://www.datastax.com/dev/blog/impact-of-shared-storage-on-apache-cassandra


Des questions ?


Configuration - quelle version de C* utiliser ?

2.1.17 ?2.2.9 ?3.0.12 ?3.10 ?


Configuration - cassandra.yaml

SeedsThread pools : concurrent_reads / concurrent_writesTimeoutscompaction_throughput_mb_per_secconcurrent_compactors


Configuration - cassandra.yaml

data directoriescommit logphi_convict_thresholdtrickle_fsync


Configuration - Tuning GC

Heap > 8 Go = OKCMS/ParNew : 16 Go heap + 6 Go de new gen > ça passe graaaave!Logs GC + observation + tuning G1 > 24 Go (risque d’OOM en dessous)Un bon tuning GC change tout !!


Configuration - VNodes

256 vnodes ?Rallonge : repair, bootstrap, décommissions, rebuild16-32 vnodes : sweet spotC* 3.0.X+ : allocate_tokens_for_keyspacehttps://www.datastax.com/dev/blog/token-allocation-algorithm


Des questions ?


Sécurité

Authentification : Activez là !system_auth : RF 3 à 5superuser cassandra : CL.QUORUMAutres users : CL.LOCAL_ONEUsers deprecated en 3.0, rôles en remplacement


Sécurité

Interfaces réseau : pas 0.0.0.0 ! listen_/rpc_address : IP privéebroadcast_address : IP publique en multi-regionSécurisation JMX : IP privée et/ou authentification (cassandra-env.sh)Chiffrement : internode / inter-rack / inter-dc et node <> client Securing Cassandra for compliance (or paranoia) : https://www.slideshare.net/zznate/hardening-cassandra-q22016


Des questions ?


Ajout/Retrait de noeuds

Stratégies de placementSimpleStrategy / NetworkTopologyStrategy

SnitchGossipingPropertyFileSnitch : Racks / DCEc2Snitch et Ec2MultiRegionSnitch : Spécifique AWSNombre de racks proportionnel au RF !


Ajout/Retrait de noeuds

Bootstrap replace_address / replace_address_first_bootRetrait : nodetool decommission / removenode / assassinate


Des questions ?


Multi DC

Cas d’utilisationLoad balancing + Disaster recoveryCo-localisation géographique avec les clientsSéparation opérationnel et analytique


Multi DC

DC avec cohérence forte - CL.QUORUM / CL.EACH_QUORUMBasse latence entre DCEt au moins 3 DCEt pas de DC analytique


Multi DC

DC avec cohérence “in fine” - CL.ONE / CL.LOCAL_*Haute latence entre DCOu 2 DC uniquementOu au moins 1 DC analytiquePas de failover automatique entre DC !!!


Multi DC

Ajout d’un DCKeyspaces (sauf system*) en NetworkTopologyStrategyDCAwareRoundRobinPolicy !!auto_bootstrap: false ?Démarrage des nouveaux noeudsAjout des répliques sur le nouveau DC (ALTER KEYSPACE)nodetool rebuild (sur tous les noeuds du nouveau DC)


Multi DC

Retrait d’un DCDCAwareRoundRobinPolicy >> Arrêt des accès au DCRetrait des répliques sur le DC (ALTER KEYSPACE)Arrêt des noeudsnodetool decommission


Des questions ?


Compaction

Périodiquement les SSTables sont fusionnées en une table plus grande

Objectifs:Réduire le nombre de fichiers à lireÉliminer les données : - périmées

- supprimées- mises à jour


Compaction

SizeTieredCompactionStrategyLeveledCompactionStrategy

DateTieredCompactionStrategyTimeWindowCompactionStrategy


Compaction - TWCS

CREATE TABLE twcs.twcs ( id int, value int, text_value text, PRIMARY KEY (id, value)) WITH CLUSTERING ORDER BY (value ASC) AND compaction = {

'compaction_window_size': '1', 'compaction_window_unit': 'DAYS''class': 'TimeWindowCompactionStrategy'

}


Compaction - TWCS


Compaction - TWCS


Compaction - TWCS


Compaction

Quel workload pour la TWCS ?


Compaction

Vous ne devriez jamais lancer de compaction majeure......quelle que soit votre stratégie de compaction


Des questions ?


Purge

Tombstones DELETETTLCollectionsPrepared StatementsMise à jour d'une table avec vue matérialisée


Purge

Impacts des tombstones


Purge

Conditions de purge des tombstonesgc_grace_secondsZombie records


Purge

CREATE TABLE ze_ks.ze_table ( id int, value int, text_value text, PRIMARY KEY (id, value)) WITH CLUSTERING ORDER BY (value ASC) AND compaction = {

'class': 'SizeTieredCompactionStrategy','unchecked_tombstone_compaction': 'true','tombstone_threshold': 0.2,'tombstone_compaction_interval': 86400

}


Purge

Badass ?

Deletion Compaction Strategy - ProtectWise



Upgrade

Préparer son upgrade


Upgrade

Effectuer son upgradeEn cours de procédure, ne jamais : ● Ajouter un nouveau noeud● Lancer de réparation● Modifier le schéma de donnée


Upgrade

Et si ça se passe mal ?


Des questions ?


Réparation


Réparation

Calcul des arbres de Merkle

32k feuilles max.


Réparation

Over-repair (nodetool repair)


Réparation

nodetool repair -pr


Réparation

OverstreamingSubrange repair

Reapergithub.com/thelastpickle/reaper

cassandra_range_repair github.com/BrianGallew/cassandra_range_repair


Réparation

ReaperSubrangeBackpressureUI / Rest / Command line(Auto-) Scheduler


Réparation

Réparation incrémentale (défaut C* 2.2+)


Réparation

Réparation incrémentale : ça a l’air génial !


Réparation

Réparation incrémentale, les problèmes…Anticompaction

Pas de sub-range et ne pas lancer avec “-pr”Exécutions concurrentes

2 pools de SSTables non “compactables”Bugs : CASSANDRA-9143 / CASSANDRA-11696 / CASSANDRA-13153


Réparation

CASSANDRA-12580Fix merkle tree size calculation

2.2.9 / 3.0.10 / 3.10


Des questions ?


Backup

Pourquoi faire un backup alors quej’ai déjà plein de répliques ?

dsh -a -c “rm -Rf /var/lib/cassandra”


Backup

Risques Panne d'un nœud ⇒ RéplicationPanne d'un datacenter ⇒ Multi DatacenterErreur conduisant à la suppression de données

⇒ Sauvegardes


Backup

Backup automatique sur truncate / dropauto_snapshot: true


Backup

4 principales techniques de backup :snapshots programmés incremental_backup: true (cassandra.yaml)tablesnap / tableslurp / tablechop snapshot EBS (AWS)


Backup

Restauration de backupArrêt du noeud(Vider les commit logs — selon le cas)Copie des fichiers dans leur emplacement d’origineRelance du noeud

Faut-il monter le backup ou remplacer le nœud (bootstrap) ?


Des questions ?


Monitoring

On premisesGrafana, Graphite, Prometheus.io, ELK

CloudDataDog


Monitoring

Construire ses dashboardsOverview (global et par DC/noeud)

Read path (global, par DC/noeud, par keyspace/table)

Write path (global, par DC/noeud, par keyspace/table)

...


Monitoring - overview

83


Monitoring - Overview

Read/Write latencies : p95 (+ p75/p99) o.a.c.m.ClientRequest.*.Latency.95p

Read/Write rate : 1m o.a.c.m.ClientRequest.*.Latency.1MinuteRate

Dropped Messages o.a.c.m.DroppedMessages.*.Dropped.1MinuteRate

Disk space collectd

Load average / CPU (+GC) collectd

Network collectd

(+ Max partition size o.a.c.m.ColumnFamily(.Keyspace.Table).MaxRowSize )


Des questions ?


Outillage

nodetoolinfostatus cfstatstpstatsnetstats


Outillage

nodetool cfhistograms ks table Percentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 4.00 20.50 20924.30 24601 21575% 60.00 24.60 43388.63 105778 92495% 72.00 42.51 107964.79 545791 572298% 86.00 61.21 129557.75 1131752 988799% 86.00 88.15 155469.30 1629722 14237Min 0.00 3.31 51.01 104 0Max 86.00 962624.93 962624.93 20924300 219342


Outillage

nodetoolstop [COMPACTION | CLEANUP | SCRUB | INDEX_BUILD ]getendpoints ks table partition_keygetsstables ks table partition_keycompactionstats -H getcompactionthroughput / setcompactionthroughput [value in MB/s]toppartitions ks table duration_in_ms


Outillage

sstablemetadata[a.dejanovski@ip-10-1-1-197 table-290fbca624193f6b9758aa765e75c988]$ /opt/tlp/sstablemetadata ks-table-ka-1788-Data.dbSSTable: ./tmo_my_account-events-ka-1788Partitioner: org.apache.cassandra.dht.Murmur3PartitionerBloom Filter FP chance: 0.010000Minimum timestamp: 1489622340515999Maximum timestamp: 1489708671766000SSTable max local deletion time: 1490918271Compression ratio: 0.15311473937163064Estimated droppable tombstones: 0.8747949761639867SSTable Level: 0Repaired at: 0ReplayPosition(segmentId=-1, position=0)Estimated tombstone drop times:%n1489622986: 4995531489624730: 4648071489626655: 4340191489628549: 4159821489630350: 3670931489632089: 3901021489633658: 2999731489635072: 14519001489636682: 13822861489638346: 1242136…


Outillage

sstablemetadata[a.dejanovski@ip-10-1-1-197 table-290fbca624193f6b9758aa765e75c988]$ /opt/tlp/sstablemetadata ks-table-ka-1788-Data.dbSSTable: ./tmo_my_account-events-ka-1788Partitioner: org.apache.cassandra.dht.Murmur3PartitionerBloom Filter FP chance: 0.010000

Minimum timestamp: 1489622340515999Maximum timestamp: 1489708671766000SSTable max local deletion time: 1490918271Compression ratio: 0.15311473937163064Estimated droppable tombstones: 0.8747949761639867SSTable Level: 0Repaired at: 0ReplayPosition(segmentId=-1, position=0)Estimated tombstone drop times:%n1489622986: 4995531489624730: 4648071489626655: 4340191489628549: 4159821489630350: 3670931489632089: 3901021489633658: 2999731489635072: 14519001489636682: 13822861489638346: 1242136…


Outillage

sstablemetadata[a.dejanovski@ip-10-1-1-197 table-290fbca624193f6b9758aa765e75c988]$ /opt/tlp/sstablemetadata ks-table-ka-1788-Data.dbSSTable: ./tmo_my_account-events-ka-1788Partitioner: org.apache.cassandra.dht.Murmur3PartitionerBloom Filter FP chance: 0.010000Minimum timestamp: 1489622340515999Maximum timestamp: 1489708671766000

SSTable max local deletion time: 1490918271Compression ratio: 0.15311473937163064Estimated droppable tombstones: 0.8747949761639867SSTable Level: 0Repaired at: 0ReplayPosition(segmentId=-1, position=0)Estimated tombstone drop times:%n1489622986: 4995531489624730: 4648071489626655: 4340191489628549: 4159821489630350: 3670931489632089: 3901021489633658: 2999731489635072: 14519001489636682: 13822861489638346: 1242136…


Outillage

sstablemetadata[a.dejanovski@ip-10-1-1-197 table-290fbca624193f6b9758aa765e75c988]$ /opt/tlp/sstablemetadata ks-table-ka-1788-Data.dbSSTable: ./tmo_my_account-events-ka-1788Partitioner: org.apache.cassandra.dht.Murmur3PartitionerBloom Filter FP chance: 0.010000Minimum timestamp: 1489622340515999Maximum timestamp: 1489708671766000SSTable max local deletion time: 1490918271

Compression ratio: 0.15311473937163064Estimated droppable tombstones: 0.8747949761639867SSTable Level: 0Repaired at: 0ReplayPosition(segmentId=-1, position=0)Estimated tombstone drop times:%n1489622986: 4995531489624730: 4648071489626655: 4340191489628549: 4159821489630350: 3670931489632089: 3901021489633658: 2999731489635072: 14519001489636682: 13822861489638346: 1242136…


Outillage

sstablemetadata[a.dejanovski@ip-10-1-1-197 table-290fbca624193f6b9758aa765e75c988]$ /opt/tlp/sstablemetadata ks-table-ka-1788-Data.dbSSTable: ./tmo_my_account-events-ka-1788Partitioner: org.apache.cassandra.dht.Murmur3PartitionerBloom Filter FP chance: 0.010000Minimum timestamp: 1489622340515999Maximum timestamp: 1489708671766000SSTable max local deletion time: 1490918271Compression ratio: 0.15311473937163064

Estimated droppable tombstones: 0.8747949761639867SSTable Level: 0Repaired at: 0ReplayPosition(segmentId=-1, position=0)Estimated tombstone drop times:%n1489622986: 4995531489624730: 4648071489626655: 4340191489628549: 4159821489630350: 3670931489632089: 3901021489633658: 2999731489635072: 14519001489636682: 13822861489638346: 1242136…


Outillage

sstablemetadata[a.dejanovski@ip-10-1-1-197 table-290fbca624193f6b9758aa765e75c988]$ /opt/tlp/sstablemetadata ks-table-ka-1788-Data.dbSSTable: ./tmo_my_account-events-ka-1788Partitioner: org.apache.cassandra.dht.Murmur3PartitionerBloom Filter FP chance: 0.010000Minimum timestamp: 1489622340515999Maximum timestamp: 1489708671766000SSTable max local deletion time: 1490918271Compression ratio: 0.15311473937163064Estimated droppable tombstones: 0.8747949761639867SSTable Level: 0

Repaired at: 0ReplayPosition(segmentId=-1, position=0)Estimated tombstone drop times:%n1489622986: 4995531489624730: 4648071489626655: 4340191489628549: 4159821489630350: 3670931489632089: 3901021489633658: 2999731489635072: 14519001489636682: 13822861489638346: 1242136…


Outillage

cassandra-stressTest de charge sur votre modèle de donnéeshttp://thelastpickle.com/blog


Outillage

sstableloaderChargement de SSTables dans un cluster en activitéRestauration de backupGère une topologie différente


cqlshcqlsh> expand oncqlsh> select * from reaper_db.repair_run;

@ Row 1--------------------+---------------------------------------------------- id | 3 cause | no cause specified cluster_name | c2116 creation_time | 2017-03-31 16:44:45+0000 end_time | 2017-03-31 16:44:48+0000 intensity | 0.9 last_event | no events owner | test pause_time | 2017-03-31 16:45:53+0000 repair_parallelism | dc_parallel repair_unit_id | 3 segment_count | 201 start_time | 2017-03-31 16:44:48+0000 state | PAUSED

Outillage


Outillage

cqlshcqlsh> tracing oncqlsh> select * from reaper_db.repair_run;Tracing session: d37dd960-1850-11e7-910c-753d3b81463a

activity | timestamp | source | source_elapsed------------------------------------------------------------------------------------------------------------------------+----------------------------+-----------+---------------- Execute CQL3 query | 2017-04-03 11:35:08.280000 | 127.0.0.1 | 0 Parsing select * from reaper_db.repair_run; [SharedPool-Worker-1] | 2017-04-03 11:35:08.288000 | 127.0.0.1 | 5575 Preparing statement [SharedPool-Worker-1] | 2017-04-03 11:35:08.289000 | 127.0.0.1 | 6439 Computing ranges to query [SharedPool-Worker-1] | 2017-04-03 11:35:08.290000 | 127.0.0.1 | 6580 Submitting range requests on 3 ranges with a concurrency of 1 (248.1231 rows per range expected) [SharedPool-Worker-1] | 2017-04-03 11:35:08.290000 | 127.0.0.1 | 6801 Enqueuing request to /127.0.0.2 [SharedPool-Worker-1] | 2017-04-03 11:35:08.290000 | 127.0.0.1 | 6980 Submitted 1 concurrent range requests covering 1 ranges [SharedPool-Worker-1] | 2017-04-03 11:35:08.291000 | 127.0.0.1 | 7003 Sending PAGED_RANGE message to /127.0.0.2 [MessagingService-Outgoing-/127.0.0.2] | 2017-04-03 11:35:08.291000 | 127.0.0.1 | 7749 PAGED_RANGE message received from /127.0.0.1 [MessagingService-Incoming-/127.0.0.1] | 2017-04-03 11:35:08.292000 | 127.0.0.2 | 190 REQUEST_RESPONSE message received from /127.0.0.2 [SharedPool-Worker-1] | 2017-04-03 11:35:08.294000 | 127.0.0.2 | 14778 Read 0 live and 0 tombstone cells [SharedPool-Worker-3] | 2017-04-03 11:35:08.295000 | 127.0.0.2 | 15474 Read 1 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-04-03 11:35:08.295000 | 127.0.0.2 | 4190 Sending PAGED_RANGE message to /127.0.0.3 [SharedPool-Worker-1] | 2017-04-03 11:35:08.296000 | 127.0.0.2 | 15670 Enqueuing response to /127.0.0.1 [SharedPool-Worker-1] | 2017-04-03 11:35:08.296000 | 127.0.0.2 | 4400 Sending REQUEST_RESPONSE message to /127.0.0.1 [MessagingService-Outgoing-/127.0.0.1] | 2017-04-03 11:35:08.296000 | 127.0.0.2 | 4759 Submitted 2 concurrent range requests covering 2 ranges [SharedPool-Worker-1] | 2017-04-03 11:35:08.297000 | 127.0.0.1 | 15680 Executing seq scan across 1 sstables for (max(3074457345618258602), min(-9223372036854775808)] [SharedPool-Worker-4] | 2017-04-03 11:35:08.299000 | 127.0.0.1 | 15884 Read 0 live and 0 tombstone cells [SharedPool-Worker-4] | 2017-04-03 11:35:08.299000 | 127.0.0.1 | 16152 Scanned 1 rows and matched 1 [SharedPool-Worker-4] | 2017-04-03 11:35:08.299000 | 127.0.0.1 | 16209 PAGED_RANGE message received from /127.0.0.1 [MessagingService-Incoming-/127.0.0.1] | 2017-04-03 11:35:08.302000 | 127.0.0.3 | 262 REQUEST_RESPONSE message received from /127.0.0.3 [SharedPool-Worker-3] | 2017-04-03 11:35:08.304000 | 127.0.0.3 | 24514 Read 1 live and 0 tombstone cells [SharedPool-Worker-3] | 2017-04-03 11:35:08.304000 | 127.0.0.3 | 5939 Scanned 1 rows and matched 1 [SharedPool-Worker-3] | 2017-04-03 11:35:08.305000 | 127.0.0.3 | 24671 Enqueuing response to /127.0.0.1 [SharedPool-Worker-3] | 2017-04-03 11:35:08.305000 | 127.0.0.3 | 6044 Sending REQUEST_RESPONSE message to /127.0.0.1 [MessagingService-Outgoing-/127.0.0.1] | 2017-04-03 11:35:08.305000 | 127.0.0.3 | 6534 Request complete | 2017-04-03 11:35:08.304794 | 127.0.0.1 | 24794


Outillage

ccm : créez des clusters complets sur votre portable ccm create mon_cluster_229 -v 2.2.9 -n 3:3

ccm node1 cqlsh

ccm node1 nodetool status

ccm start/stop

ccm switch un_autre_cluster


Outillage

Instaclustr SSTable tools https://github.com/instaclustr/cassandra-sstable-tools

ic-sstables : Formatage tableau et lisible de sstablemetadataic-pstats : Top partitions selon fichier Index.dbic-cfstats : partitions les plus grosses, ayant le plus de tombstones, dans bcp de sstables, etc… (compaction)ic-purge : Plus grosses partitions supprimables (compaction)


JMX

cassandra-env.shif [ "$LOCAL_JMX" = "yes" ]; then

JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT -XX:+DisableExplicitGC"

else

JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"

JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"

JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"

JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"

JVM_OPTS="$JVM_OPTS

-Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"

Port 7199 par défaut, accessible en local uniquement, sans authentification


JMX


JMX


JMX


JMX

jmxterm : JMX en ligne de commandehttp://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar

java -jar jmxterm-1.0-alpha-4-uber.jar --url node_ip_adress:7199 >> REPLouecho “[command]” | java -jar jmxterm-1.0-alpha-4-uber.jar --url node_ip_adress:7199 >> One liner


JMX - jmxterm

$>bean org.apache.cassandra.db:type=BatchlogManager

#bean is set to org.apache.cassandra.db:type=BatchlogManager$>info

#mbean = org.apache.cassandra.db:type=BatchlogManager

#class name = org.apache.cassandra.batchlog.BatchlogManager

# attributes

%0 - TotalBatchesReplayed (long, r)

# operations

%0 - int countAllBatches()

%1 - void forceBatchlogReplay()

$>get TotalBatchesReplayed

#mbean = org.apache.cassandra.db:type=BatchlogManager:

TotalBatchesReplayed = 0;

$>run countAllBatches

#Appelle l’opération countAllBatches de mbean org.apache.cassandra.db:type=BatchlogManager

#Retourne :

0


JMX - jmxterm


#bean is set to org.apache.cassandra.db:type=BatchlogManager

$>info



# attributes


# operations







#calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager

#operation returns:

0


JMX - jmxterm



$>info



# attributes


# operations








#Retourne :

0


JMX - jmxterm



$>info



# attributes


# operations








#Retourne :

0


JMX

https://fr.slideshare.net/zznate/advanced-apache-cassandra-operations-with-jmx


Des questions ?


Problèmes courants

Partitions trop grandesBatchs multi-partitions / multi-tablesGC mal tunéLecture de tombstonesTrop de SSTables accédées par readTrop de SSTables (tout court…)Compaction overload / Retard de compactionEspace disqueConflits de schéma


Bonus

Fonctionnement de la clause LIMIT

SELECT * FROM table WHERE id = ?LIMIT 10


Bonus

Severity score (dynamic snitch) + nodetool disablethrift/binary


Bonus

Changement de stratégie de compaction via JMXecho "set -b org.apache.cassandra.db:columnfamily=table,keyspace=ks,type=ColumnFamilies

CompactionParametersJson

{\"tombstone_compaction_interval\":\"86400\",\"compaction_window_size\":\"1\",\"compaction_window_

unit\":\"DAYS\",\"tombstone_threshold\":\"0.20\",\"unchecked_tombstone_compaction\":\"true\",\"cla

ss\":\"com.jeffjirsa.cassandra.db.compaction.TimeWindowCompactionStrategy\"}"

| java -jar jmxterm-1.0-alpha-4-uber.jar --url 127.0.0.1:7199


Date post:	21-Apr-2017
Category:	Data & Analytics
Upload:	alexander-dejanovski
View:	455 times
Download:	1 times

Apache cassandra en production - devoxx 2017

Data & Analytics