• Lucene/Solr committer. Work for Lucidworks; focus on hardening SolrCloud, devops, big data architecture / deployments

• Operated smallish cluster in AWS for Dachis Group (1.5 years ago, 18 shards ~900M docs)

• Solr Scale Toolkit: Fabric/boto framework for deploying and managing clusters in EC2

• Co-author of Solr In Action with Trey Grainger

About Me

Agenda

1. Quick review of the SolrCloud architecture

2. Indexing & Query performance tests

3. Solr Scale Toolkit (quick overview)

4. Q & A

Solr in the wild …

https://twitter.com/bretthoerner/status/476830302430437376

SolrCloud distilled

Subset of optional features in Solr to enable and

simplify horizontal scaling a search index using

sharding and replication.

Goalsperformance, scalability, high-availability,

simplicity, elasticity, and

community-driven!

Collection == distributed index

A collection is a distributed index defined by:• named configuration stored in ZooKeeper

• number of shards: documents are distributed across N partitions of the index

• document routing strategy: how documents get assigned to shards

• replication factor: how many copies of each document in the collection

Collections API:

curl "http://localhost:8983/solr/admin/collections?

action=CREATE&name=logstash4solr&replicationFactor=2&

numShards=2&collection.configName=logs"

SolrCloud High-level Architecture

ZooKeeper• Is a very good thing ... clusters are a zoo!• Centralized configuration management• Cluster state management• Leader election (shard leader and overseer)• Overseer distributed work queue• Live Nodes

• Ephemeral znodes used to signal a server is gone

• Needs at least 3 nodes for quorum in production

ZooKeeper: State Management

• Keep track of live nodes /live_nodes znode• ephemeral nodes

• ZooKeeper client timeout

• Collection metadata and replica state in /clusterstate.json

• Every Solr node has watchers for /live_nodes and /clusterstate.json

• Leader election• ZooKeeper sequence number on ephemeral znodes

Scalability Highlights

• No split-brain problems (b/c of ZooKeeper)• All nodes in cluster perform indexing and execute queries; no

master node• Distributed indexing: No SPoF, high throughput via direct updates

to leaders, automated failover to new leader• Distributed queries: Add replicas to scale-out qps; parallelize

complex query computations; fault-tolerance• Indexing / queries continue so long as there is 1 healthy replica per

shard

Cluster sizing

How many servers do I need to index X docs?

... shards ... ?

... replicas ... ?

I need N queries per second over M docs, how many servers do I need?

It depends!

Testing Methodology• Transparent repeatable results

• Ideally hoping for something owned by the community

• Synthetic docs ~ 1K each on disk, mix of field types• Data set created using code borrowed from PigMix

• English text fields generated using a Zipfian distribution

• Java 1.7u67, Amazon Linux, r3.2xlarge nodes• enhanced networking enabled, placement group, same AZ

• Stock Solr (cloud) 4.10• Using custom GC tuning parameters and auto-commit settings

• Use Elastic MapReduce to generate indexing load• As many nodes as I need to drive Solr!

Cluster Size # of Shards # of Replicas Reducers Time (secs) Docs / sec

10 10 1 48 1762 73,780

10 10 2 34 3727 34,881

10 20 1 48 1282 101,404

10 20 2 34 3207 40,536

10 30 1 72 1070 121,495

10 30 2 60 3159 41,152

15 15 1 60 1106 117,541

15 15 2 42 2465 52,738

15 30 1 60 827 157,195

15 30 2 42 2129 61,062

Indexing Performance

Visualize Server Performance

Direct Updates to Leaders

Replication

Indexing Performance Lessons• Solr has no built-in throttling support – will accept work until it falls over; need to

build this into your indexing application logic• Oversharding helps parallelize indexing work and gives you an easy way to add

more hardware to your cluster• GC tuning is critical (more below)• Auto-hard commit to keep transaction logs manageable• Auto soft-commit to see docs as they are indexed• Replication is expensive! (more work needed here)

GC Tuning• Stop-the-world GC pauses can lead to ZooKeeper session expiration (which is bad)• More JVMs with smaller heap sizes are better! (12-16GB max per JVM ~ less if you

can)• MMapDirectory relies on sufficient memory available to the OS cache (off-heap)• GC activity during Solr indexing is stable and generally doesn’t cause any stop-

the-world collections … queries are a different story• Enable verbose GC logging (even in prod) so you can troubleshoot issues:-verbose:gc –Xloggc:gc.log -XX:+PrintHeapAtGC -XX:

+PrintGCDetails \-XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps \-XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime \-XX:+PrintGCApplicationConcurrentTime

GC Flags I use with Solr-Xss256k \

-XX:+UseConcMarkSweepGC -XX:+UseParNewGC \-XX:MaxTenuringThreshold=8 -XX:NewRatio=3 \

-XX:CMSInitiatingOccupancyFraction=40 \-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \

-XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 \-XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=12m \

-XX:CMSFullGCsBeforeCompaction=1 \-XX:+UseCMSInitiatingOccupancyOnly \

-XX:CMSTriggerPermRatio=80 \-XX:CMSMaxAbortablePrecleanTime=6000 \

-XX:+CMSParallelRemarkEnabled \-XX:+ParallelRefProcEnabled \

-XX:+UseLargePages -XX:+AggressiveOpts

Sizing GC Spaces

http://kumarsoablog.blogspot.com/2013/02/jvm-parameter-survivorratio_7.html

Query Performance• Still a work in progress!

• Sustained QPS & Execution time of 99th Percentile (coda hale metrics is good for this)

• Stable: ~5,000 QPS / 99th at 300ms while indexing ~10,000 docs / sec

• Using the TermsComponent to build queries based on the terms in each field.

• Harder to accurately simulate user queries over synthetic data• Need mix of faceting, paging, sorting, grouping, boolean clauses, range queries, boosting,

filters (some cached, some not), etc ...

• Does the randomness in your test queries model (expected) user behavior?

• Start with one server (1 shard) to determine baseline query performance.• Look for inefficiencies in your schema and other config settings

Query Performance, cont.

• Higher risk of full GC pauses (facets, filters, sorting)

• Use optimized data structures (DocValues) for facet / sort fields, Trie-based numeric fields for range queries, facet.method=enum for low cardinality fields

• Check sizing of caches, esp. filterCache in solrconfig.xml

• Add more replicas; load-balance; Solr can set HTTP headers to work with caching proxies like Squid

• -Dhttp.maxConnections=## (default = 5, increase to accommodate more threads sending queries)

• Avoid increasing ZooKeeper client timeout ~ 15000 (15 seconds) is about right

• Don’t just keep throwing more memory at Java! –Xmx128G

Call me maybe - Jepsen

• Solr tests being developed by Lucene/Solr committer Shalin Mangar (@shalinmanger)

• Prototype in place:• No ack’d writes were lost!• No un-ack’d writes succeeded

See: https://github.com/LucidWorks/jepsen/tree/solr-jepsen

https://github.com/aphyr/jepsen

Solr Scale Toolkit

• Open source: https://github.com/LucidWorks/solr-scale-tk

• Fabric (Python) toolset for deploying and managing SolrCloud clusters in the cloud

• Code to support benchmark tests (Pig script for data generation / indexing, JMeter samplers)

• EC2 for now, more cloud providers coming soon via Apache libcloud

• Contributors welcome!

• More info: http://searchhub.org/2014/06/03/introducing-the-solr-scale-toolkit/

Provisioning cluster nodes

• Custom built AMI (one for PV instances and one for HVM instances) – Amazon Linux

• Block device mapping• dedicated disk per Solr node

• Launch and then poll status until they are live • verify SSH connectivity

• Tag each instance with a cluster ID and username

fab new_ec2_instances:test1,n=3,instance_type=m3.xlarge

Deploy ZooKeeper ensemble

• Two options:• provision 1 to N nodes when you launch Solr cluster

• use existing named ensemble

• Fabric command simply creates the myid files and zoo.cfg file for the ensemble

• and some cron scripts for managing snapshots

• Basic health checking of ZooKeeper status:echo srvr | nc localhost 2181

fab new_zk_ensemble:zk1,n=3

Deploy SolrCloud clusterfab new_solrcloud:test1,zk=zk1,nodesPerHost=2

• Uses bin/solr in Solr 4.10 to control Solr nodes• Set system props: jetty.port, host, zkHost, JVM opts• One or more Solr nodes per machine• JVM mem opts dependent on instance type and # of Solr

nodes per instance• Optionally configure log4j.properties to append messages

to Rabbitmq for SiLK integration

Automate day-to-day cluster management tasks• Deploy a configuration directory to ZooKeeper• Create a new collection• Attach a local JConsole/VisualVM to a remote JVM• Rolling restart (with Overseer awareness)• Build Solr locally and patch remote

• Use a relay server to scp the JARs to Amazon network once and then scp them to other nodes from within the network

• Put/get files• Grep over all log files (across the cluster)

Wrap-up and Q & A

• LucidWorks: http://www.lucidworks.com -- We’re hiring!

• Solr Scale Toolkit: https://github.com/LucidWorks/solr-scale-tk

• SiLK: http://www.lucidworks.com/lucidworks-silk/

• Solr In Action: http://www.manning.com/grainger/

• Connect: @thelabdude / tim.potter@lucidworks.com