varnish cache

VARNISH-CACHE *Lessons Learnt from Performance Testing

* https://www.varnish-cache.orgTuesday, 25 October 11

WHY? FAST!

If your throughput is currently measured in the hundreds per second, think of varnish in the thousands per second.

Theoretical limits in hundreds of thousands per second.*

* http://kristianlyng.wordpress.com/2010/10/23/275k-req/Tuesday, 25 October 11

GET OUT OF JAIL. FREE.

Crappy cache control?

Expensive CDNs? 20/GB?

Not enough hardware?

Backend struggling?

Tuesday, 25 October 11

IT WONT

Do HTTPS... terminate SSL with something else, hopefully your load balancers.


INSTALL THE LATEST

RHEL / CentOS

Debian

rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-release-3.0-1.noarch.rpmyum install -y varnish

apt-get install varnish


USE 64BIT*

Its all about virtual memory: Things like stack size suddenly matter on 32bit. If you must use Varnish on 32-bit, youre somewhat on your own. However, try to fit it within 2GB. I wouldnt recommend a cache larger than 1GB, and no more than a few hundred threads (Why are you on 32bit again?)

* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/Tuesday, 25 October 11

SYSTEM CONFIG

The minimum.

Do not set tw_reuse to 1 (sysctl). It will cause problems for NAT based clients.

Enable keepalives on load balancers.

echo 'fs.file-max=1000000' >> /etc/sysctl.confecho 'net.ipv4.ip_local_port_range = 1024 65535' >> /etc/sysctl.confsysctl -pecho '* soft nofile 1000000' >> /etc/security/limits.confecho '* hard nofile 1000000' >> /etc/security/limits.confecho 'session required pam_limits.so' >> /etc/pam.d/loginulimit -n 1000000ulimit -n -H


THREADS

Adjust default thread* settings in /etc/sysconfig/varnish

thread_pool_add_delay=2thread_pools = thread_pool_min = thread_pool_max = 4000session_linger = 50sess_workspace =

* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/

The only thing that made a real difference while tuning Varnish was the number of threads. And making sure it actually caches. Beyond that, it really doesnt matter much. Our defaults are good.


STORAGE

Memory is fast, duh.

Got enough RAM to store the cache?Yes! then use -s mallocUnsure? then use -s file


VCL

Varnish Configuration Language

Looks like C and compiled into C but ...~ has regex tilda operatorAssignments use set keyword.Only has conditional if tests.Has a bunch of variables and some built in functions.

Defined by backends, probes, ACLs and subroutines which hook into the varnish workflow.


VCL

Default VCL is defined in/etc/varnish/default.vcl

Then create your custom VCL in -f /var/custom.vclso varnish can fall through to the default.vclCan also include "/var/other.vcl";

Keep it simple within the VCL workflow*Do most of the work in vcl_recv and vcl_fetch

* https://www.varnish-cache.org/trac/wiki/VCLExampleDefaultTuesday, 25 October 11

LOOKUP AND MISS

request received from client

hash request

lookup object and miss

fetch response from backend

deliver to client

recv

hash

pass

pipe

misshit

fetch

deliver


LOOKUP AND HIT


hash request

lookup object and hit

deliver to client

recv

hash

pass

pipe

misshit

fetch

deliver


PASS AND DELIVER


pass through to backend

fetch response

deliver to client

recv

hash

pass

pipe

misshit

fetch

deliver


PIPE


pipe to backend until client or backend closes

recv

hash

pass

pipe

misshit

fetch

deliver


PROBES

Define common health checks for multiple backends.

window is how many of the latest polls are probed for health.

threshold is how many polls must succeed to be healthy.

probe health {.url = "/healthCheck?onFailure=true";.expected_response = 200; .interval= 15s; .timeout = 5s;.window= 4; .threshold = 4; }


BACKENDS

Define backend host, port and health checks inline.

Or named.backend mysite {.host = "www.example.com";.port = "http"; .probe = "health";}

{ .backend = { .host = "10.10.172.1"; .port = "8001"; .probe = health; } }


DIRECTORS

Group backends together using different algorithms. Random Client (based on identity) Hash (based on url) Round robin Fallback


YMMV

I found it better to balance in front of Varnish using a hardware load balancer ...

Random seemed a little too random using Varnish directors however was probably an artefact of the load generation.


DESIGN DECISIONS

Share nothing. Easy to scale horizontally.Great if your site is not transactional.Backend session affinity via LB set cookie.

LB

app

varnishvarnishvarnish

appapp

Set-Cookie=BackendID


DESIGN DECISIONS

Share everything. Varnish loves RAM, scale vertically.Varnish cache becomes the shared source of truth.

LB

app

varnish

appapp


SHARE NOTHING. FALLBACK.

LB

app

varnishvarnishvarnish

appapp

Set-Cookie=VarnishIDdirector varnishA fallback {{ .backend = { .host = "a";} } { .backend = { .host = "b";} } { .backend = { .host = "c";} } }

director varnishB fallback {{ .backend = { .host = "b";} } { .backend = { .host = "a";} } { .backend = { .host = "c";} } }

director varnishC fallback {{ .backend = { .host = "c";} } { .backend = { .host = "b";} } { .backend = { .host = "a";} } }


TERMINATE SSL.

LB varnish appSet-Cookie=VarnishIDPort=https:80:443

if (req.http.Port ~ "https") { set req.backend = appa_80;} else {set req.backend = appa_443; }

Get the LB to set a Port header. Useful when user switches from HTTP to HTTPS so varnish knows which backend port to route to.


VCL TRICKS

Rewrite headers to/from production.com.au to testdomain.com

vcl_recv

vcl_deliver

set req.http.Host = regsuball(req.http.Host, "testdomain\.com", "com\.au");set req.http.Referer = regsuball(req.http.Host, "testdomain\.com", "com\.au");

set resp.http.Host = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com");set resp.http.Referer = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com");set resp.http.Location = regsuball(resp.http.Location, "\.com\.au", "\.testdomain\.com");set resp.http.Set-Cookie = regsuball(resp.http.Set-Cookie, "Domain=.*;", "");


VCL TRICKS

Add some debug headers for hit/miss in firebug in vcl_deliverif (obj.hits > 0) {set resp.http.X-Varnish-Hit = "++";set resp.http.X-Varnish-Hits = obj.hits;set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");} else {set resp.http.X-Varnish-Hit = "--";set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");}


VCL TRICKS

Normalize the Accept-Encoding headers to reduce cache usage in vcl_recv

if (req.http.Accept-Encoding) {if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {remove req.http.Accept-Encoding;} elsif (req.http.Accept-Encoding ~ "gzip") {set req.http.Accept-Encoding = "gzip";} elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate";} else {remove req.http.Accept-Encoding;}}


VCL TRICKS

Remove Google Analytics cookies in vcl_recv

Remember, varnish will pass any request with cookies by default. You will need to over-ride this behaviour if your web app uses cookies.

Allow lookup of requests with basic auth (in test) or cookies in vcl_recv but beware of creating a per-user-session store.

set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");

if (req.http.Authorization || req.http.Cookie) { return (lookup); }


VCL TRICKS

Pass on the client IP to the backend servers using the X-Forwarded-For header in vcl_recv

remove req.http.X-Forwarded-For;setreq.http.X-Forwarded-For = client.ip;


VCL TRICKS

Let varnish keep objects for 2 hours beyond their TTL in vcl_fetch

Let varnish serve stale objects up to 30 secs old for healthy servers, otherwise 2 hours in vcl_recv

Can also use saint mode to disable backends for a period of time, otherwise if all backends unavailable, use grace.

set beresp.grace = 2h;

if (! req.backend.healthy) {set req.grace = 2h;} else {set req.grace = 30s;}


VCL TRICKS

Force a grace on error. Restart keyword can be called any where and will return request all the way back to vcl_recvsub vcl_error {if (req.restarts == 0) {set req.http.X-Serve-Graced = "1";return(restart);}}


TTL

Varnish will obey cache control headers from backendCache-Control: s-maxageCache-Control: max-age

Can also override TTL control in VCL, but best do it from the backendset obj.ttl = x


REDUCING CACHE SIZE

Normalize / sanitize request headers

Set Content-Length on backends

Serve (static) digital assets from VFS cache

Pipe large objects like videos


MONITORING VARNISH

Handy command line toolsvarnishstatvarnishlogvarnishtopvarnishadm


VARNISHSTAT

varnishstatHit rate % (x100)Average per secondAverage/sec since bootSum total

Filteredvarnishstat -f \ client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_fail


VARNISHSTAT

Log all statistics to disk ...

Then graph as you see fit ...

#!/bin/bashlogdir=/var/log/perf/`hostname`/latestmkdir -p $logdirwhile truedovarnishstat -x >> "$logdir/varnishstat.xml"sleep 60done


VARNISHLOG

varnishlog debugging (-d to dump shared memory) 12 BackendOpen b xx_80[0] x.x.x.x 45546 x.x.x.x 80 12 TxRequest b GET 12 TxURL b / 12 TxProtocol b HTTP/1.1 12 TxHeader b Host: 12 TxHeader b Referer: 12 TxHeader b Cookie: 12 TxHeader b X-Forwarded-For: 1.2.3.4 12 TxHeader b X-Varnish: 2092287889 12 TxHeader b Accept-Encoding: gzip 12 RxProtocol b HTTP/1.1 12 RxStatus b 401 12 RxResponse b Authorization Required 12 RxHeader b Date: Mon, 24 Oct 2011 05:46:18 GMT 12 RxHeader b WWW-Authenticate: Basic realm="XX Authorised Personel Only" 12 RxHeader b Vary: Accept-Encoding 12 RxHeader b Content-Encoding: gzip 12 RxHeader b Content-Length: 336 12 RxHeader b Connection: close 12 RxHeader b Content-Type: text/html; charset=iso-8859-1 12 Fetch_Body b 4 0 1 12 Length b 336 12 BackendClose b xx_80[0]


VARNISHTOP

Monitor top URLs hitting backendvarnishtop -i txurl

Monitor top received URLsvarnishtop -i rxurl


VARNISHADM

varnishadm for administration# varnishadmCLI connected to 127.0.0.1 6082200 -----------------------------Varnish Cache CLI 1.0-----------------------------Linux,2.6.32-131.12.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbitType 'help' for command list.Type 'quit' to close CLI session.varnish> help200 help [command]ping [timestamp]auth responsequitbannerstatusstartstopvcl.load vcl.inline vcl.use vcl.discard vcl.listvcl.show param.show [-l] []param.set panic.showpanic.clearstorage.listban.url ban [&& ]...ban.list


MORE HELP?

Talk to the devs, read their blogs.http://kristianlyng.wordpress.comhttp://ingvar.blog.redpill-linpro.com/

Get some online training.https://www.varnish-software.com/products-services/training/varnish-administration-course

Consider the Varnish Moral License.http://phk.freebsd.dk/VML/


Date post:	29-Sep-2015
Category:	Documents
Upload:	khang-nguyen
View:	42 times
Download:	1 times

varnish cache

Documents