Dyn Roadshow: Cricket Liu & Cory von Wallenstein On Scalability & Availability

Scalability and Availabilityin the Real World

Cupertino, CA – October 2, 2013

Cory von WallensteinChief Technologist,

Dyn [email protected]

@cvwdyn

Cricket Liu, Chief Infrastructure Officer,

[email protected]

@cricketondns

http://www.linkedin.com/company/dyn

http://www.facebook.com/Dyn

http://twitter.com/dyninc

http://www.youtube.com/dyninc

Pg. 2 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns

What do we care about?

• Achieving high(er) availability

• Resilience in disaster (DDoS)

• Flexibility to change infrastructure without

downtime

• Ability to expand infrastructure beyond current 4

walls

• And of course, performance!






How can we do it?

• Know Thy Enemy: DDoS• Understanding DNS-based DDoS, and what you can do

• The Iovation Technical Story• Going from one datacenter to five

• How Dyn Helps• Anycast DNS and DDoS resilience• Global load balancing & traffic management





4 | © 2013 Infoblox Inc. All Rights Reserved. 4 | © 2013 Infoblox Inc. All Rights Reserved.

Cricket Liu

DNS-based DDoS Attacks

4


What You’ll Learn (or Your Money Back!)• What is a DNS-based DDoS Attack?• Why should I worry?• What should I worry about?• How can I defend myself?

5


DDoS and DNS• DDoS attacks are twice the threat to DNS

�DDoS attacks target name servers�DDoS attacks use name servers


DDoS Attacks Target Name Servers• Authoritative name servers are obviously a

critical resource

�Without them, your customers can’t get to your web site, send you email

• Authoritative name servers are easy to find

–dig ns company.example.

• Recent attack against a Prolexic customer: 167 Gbps


And DDoS Attacks Use Name Servers• Why?

�Because name servers make surprisingly good amplifiers

This one goes to

eleven…


DDoS Illustrated

Open recursive name servers

Evil resolver Target

Responseto spoofedaddress

Spoofedquery


% dig any isc.org. +dnssec

; <<>> DiG 9.8.3-P1 <<>> any isc.org. +dnssec;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57121;; flags: qr rd ra ad; QUERY: 1, ANSWER: 29, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:; EDNS: version: 0, flags: do; udp: 4096;; QUESTION SECTION:;isc.org. IN ANY

;; ANSWER SECTION:isc.org. 7200 IN RRSIG SPF 5 2 7200 20130719232951 20130619232951 50012 isc.org. Q8n5F9ZucnRaYw762EghVeq9NLLFN4tuAvJZTue/spQJUnRKcM5WuwR4 F8FuEh55EbIs5YxnrG2LbDmEJDOBh0aER+lE6Ts8TdCyZoTVylSf0kmr tmzf0r80Q5xBOdPMfsSARNxWrFDQr03r69IU0Lsp4EbneiM6wIiI7oyJ bz0=isc.org. 7200 IN SPF "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all"isc.org. 3600 IN RRSIG NSEC 5 2 3600 20130719232951 20130619232951 50012 isc.org. r9HtzBqbh52z37xEleIZfNY6gK7SU/6BvlQiSmv8d5bGjyW21vW1zT4N +nUXcd2TJCIJqYRMveZttOom4PgR/6HNq06vS67fn+9YlB/PtWbtvoh/ X1fAU107U+5u7s5EATiGKLcY/7hxPT6UcJd7RvInCyG8BrnxegilRqxG qq4=isc.org. 3600 IN NSEC _adsp._domainkey.isc.org. A NS SOA MX TXT AAAA NAPTR RRSIG NSEC DNSKEY SPFisc.org. 7200 IN RRSIG NAPTR 5 2 7200 20130719232951 20130619232951 50012 isc.org. Sv3chyUtJk8h6G4x/GXAtnV/owBxIsnRKV+FFJBdAyI0BJjwaIW8lCVE 5ntEfn/CbuyAj/nhEUZ7pwhIAKiY8sApkNwnRAlUFB2kJDxKZwyQ2F5R Bas4BbauN/yIyrEeQupIafsc88B7Hy3dl2GJKifPxocUJDvGQTWm8tsU mWk=isc.org. 7200 IN NAPTR 20 0 "S" "SIP+D2U" "" _sip._udp.isc.org.isc.org. 60 IN RRSIG AAAA 5 2 60 20130719232951 20130619232951 50012 isc.org. jPhFaMBwgJckbh6F27bYrr+28xUvurUlE0g75EsRLpAZ55b3di0F3jdX fmpBd/1YoBOR7UcOdSg7Uq596kewRFOeNGILPJW8V69Hb1CLL/JLnUOp x5hX8y9mduN0INm3wvyImH/GB6NHJ0/RKkEh3hHFVXgXTl9z284HHrkH Ba0=isc.org. 60 IN AAAA 2001:4f8:0:2::69isc.org. 7200 IN RRSIG TXT 5 2 7200 20130719232951 20130619232951 50012 isc.org. Bl7lhqWAPJcSB6lFlITQ8AB74bxxHJ6Pm02pKh9dtDvOQn/0FFPT6Y5U YsqQCbyfZZPH8cVEH5+VFrE76cWH4WoOzz7urd9DrjGh+o41pu2ersPn C8dp7cY81O6s9v66y8pb0CISYDAAhzdIi5Sasx4nKtPXZXlXjWJTWRZj 5r0=isc.org. 7200 IN TXT "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all"isc.org. 7200 IN TXT "$Id: isc.org,v 1.1824 2013-06-18 00:33:44 bind Exp $"isc.org. 7200 IN RRSIG MX 5 2 7200 20130719232951 20130619232951 50012 isc.org. PQrHeMs1C/vuOeklOHA92Ls1mzRgJrE7SY8Yg2gk5IGylMmHKH68gaaa rhdhG6tpmA8X20mMUwFP2YIPkMEecjRatDoSzKctH1YaPwRJJl3QLG4z Fiy0NSGS/qaHHCdoMiQ64KNm88p3xK2vsFa27WyI8pjSXqWOcbvPLvW8 FqU=isc.org. 7200 IN MX 10 mx.pao1.isc.org.isc.org. 60 IN RRSIG A 5 2 60 20130719232951 20130619232951 50012 isc.org. Tu753SDWWqAlfFQSzqJ0vEFF9cweMkvHC2MSK7VU1pntWcdUngwXBgLO DtPJLds7nZ1eUyVgos+WlsWtENw8PMRrYuNlwRxW9PRWpT8jIZTN1ieh fDOu/Y9JNeCqauE54eMfMluc+GH3R2Lh06513yaZB0G/Zn4dSJF6E0XI rCs=isc.org. 60 IN A 149.20.64.69isc.org. 7200 IN RRSIG SOA 5 2 7200 20130719232951 20130619232951 50012 isc.org. UtICg7Is/C+8NHjYoN79iuI+tgc/Wn1AaqTBkcdGn2NY6XL5KEY5iwdo TZiN8VHyLObwBtwyn3W4tMRH0ETEf6SzSETnvFYf6NbRkrz4snIcvBIa Vj9HhJ3UwYqfOpJCA7EWxb1QvvVhdDYidm60WBEiohMDoVHZdJ5Ol4DK VHo=isc.org. 7200 IN SOA ns-int.isc.org. hostmaster.isc.org. 2013062000 7200 3600 24796800 3600isc.org. 83390 IN RRSIG DS 7 2 86400 20130708155016 20130617145016 2373 org. OFtHIU34tL9lYvSoe7uLlQCyvHOrY5ldFbK+WM48av0FScRCqEWyjXYg 0vEpojvzR6CPxJ/Lh41HFlCb3ZevRn8ETykiNEgGwViFznPhBsrz0gdT ONmJMHAQgmVt8Lar0GwsjjjI2J6k5gCTwzQyZjkI31V6RiRNoKe8M1iA k3w=isc.org. 83390 IN DS 12892 5 1 982113D08B4C6A1D9F6AEE1E2237AEF69F3F9759isc.org. 83390 IN DS 12892 5 2 F1E184C0E1D615D20EB3C223ACED3B03C773DD952D5F0EB5C777586D E18DA6B5isc.org. 7200 IN RRSIG DNSKEY 5 2 7200 20130719230127 20130619230127 12892 isc.org. RMMZLopr6bX0u3MureNVdNPGUjtv1V6fFxyXVAlD9EOLRz9ND0fFoKr4 YnX2W70i2llvlg1uA0vMUUeUKaEM8RtR5olCChNBSLIurU/SwzsjKDG3 jfovHzwhEOF18Na9Fzd701jkn3q3rqqXsMSUFRA5MOiIfPBSplzlqtLS fJ2rF0MHgZzy5lzmsVNX2FPcbWG5lf+p3doxoGkLrYaBYCBCMVKZNw9f QFTRgvju2shpfNUodq7Jur958lmTbPV/BG8xQ2tFSUuJnVojIJQpD3Kz v6EnnjPDKP2djNS8fr3xsc4KxZPfHQ1MUGCJBROVGaxxpGP4TglG15XD WJjfJQ==isc.org. 7200 IN RRSIG DNSKEY 5 2 7200 20130719230127 20130619230127 50012 isc.org. TfbYfiP8bq6k89EudcS69xRB7DDuWhEmedUdq30/DNmWi1omAfNz1lrC iXL8OQHvO88YG0p0IuPrpQqYZMw7FYxVe913KydhlbozR83T6nLdpHwZ /TeYTm9zrGWDubbhlFW2OP/cgETIbcj7w3flFs4MNlkIu4ur38ALWoaZ Zdo=isc.org. 7200 IN DNSKEY 257 3 5 BEAAAAOhHQDBrhQbtphgq2wQUpEQ5t4DtUHxoMVFu2hWLDMvoOMRXjGr hhCeFvAZih7yJHf8ZGfW6hd38hXG/xylYCO6Krpbdojwx8YMXLA5/kA+ u50WIL8ZR1R6KTbsYVMf/Qx5RiNbPClw+vT+U8eXEJmO20jIS1ULgqy3 47cBB1zMnnz/4LJpA0da9CbKj3A254T515sNIMcwsB8/2+2E63/zZrQz Bkj0BrN/9Bexjpiks3jRhZatEsXn3dTy47R09Uix5WcJt+xzqZ7+ysyL KOOedS39Z7SDmsn2eA0FKtQpwA6LXeG2w+jxmw3oA8lVUgEf/rzeC/bB yBNsO70aEFTdisc.org. 7200 IN DNSKEY 256 3 5 BQEAAAABwuHz9Cem0BJ0JQTO7C/a3McR6hMaufljs1dfG/inaJpYv7vH XTrAOm/MeKp+/x6eT4QLru0KoZkvZJnqTI8JyaFTw2OM/ItBfh/hL2lm Cft2O7n3MfeqYtvjPnY7dWghYW4sVfH7VVEGm958o9nfi79532Qeklxh x8pXWdeAaRU=isc.org. 7200 IN RRSIG NS 5 2 7200 20130719232951 20130619232951 50012 isc.org. YaKIWDJdbioSHJ7XBShYxVvvSFHn4cFJLfbW+fUjtXTRRF+ezR2B0FXI wd1ItCOya2k//JGkQ9dxQmM9+lgIwrBUJLi4QuR5uVTAhbPLyZAqoCvW adNa2qmQQeubOpalMYRjqVI8Pf42D6Rcq0FQvXJDKLv4LEKmYygti2XG vso=isc.org. 7200 IN NS sfba.sns-pb.isc.org.isc.org. 7200 IN NS ns.isc.afilias-nst.info.isc.org. 7200 IN NS ord.sns-pb.isc.org.isc.org. 7200 IN NS ams.sns-pb.isc.org.

;; Query time: 37 msec;; SERVER: 10.102.3.10#53(10.102.3.10);; WHEN: Thu Jun 20 15:55:59 2013;; MSG SIZE rcvd: 3284

Amplification: They Go Past Eleven…

Query for isc.org/ANY36 bytes sent, 3284

bytes received~91x amplification!


A Little Math• Say each bot has a measly 1 Mbps connection to

the Internet�It can send 1Mbps/36B =~ 28K qps�That generates 28K * 3284B =~ 736 Mbps

• So 14 bots =~ 10 Gbps


The Scourge of the Open Recursor• Open recursors are like the AK-47s the Soviets left

all over the world, just waiting to be used for no good

But just how common are they?

12

=

33 million resolvers


Why Should I Worry?• More bad news about DDoS attacks

�Average attack bandwidth up 718% to 48 Gbps from Q4 2012 to Q1 2013

�Average attack packet rate now 32.4 Mpps�Average attack duration up 7% to 34.5 hours�6.97% of attacks were DNS-based

- An increase of over 200% in the last year

13

*Source: Prolexic Quartlerly Global DDoS Attack Report, Q1 2013


What Can I Do to Protect Myself?1. Overprovision2. Use anycast3. Screen traffic to your name servers4. Monitor traffic to your name servers

14


Overprovision• (Yes, I know, it seems primitive)• Overprovisioning is one of the simplest ways to

resist a DDoS attack�Run authoritative name servers with more capacity than you

need�Run a widely distributed set of authoritative name servers�Augment your authoritative name servers with cloud-based

secondary name servers- Make sure the provider uses anycast

15


Anycast• Anycast allows multiple, distributed name servers

to share a single virtual IP address• Each name server advertises a route to that

address to its neighbors• Queries sent to that address are routed to the

closest name server instance


Anycast in Action

Router 2

Router 4Router 3

Router 1

Server instance A

Server instance B

Client

DNS query to 10.0.0.1

Routing table from Router 1:

Destination Mask Next-Hop Distance 192.168.0.0 /29 127.0.0.1 0 10.0.0.1 /32 192.168.0.1 1 10.0.0.1 /32 192.168.0.2 2

192.168.0.1

192.168.0.2

10.0.0.1

10.0.0.1


Anycast in Action

Router 2

Router 4Router 3

Router 1

Server instance A

Server instance B

Client

Routing table from Router 1:

Destination Mask Next-Hop Distance 192.168.0.0 /29 127.0.0.1 0 10.0.0.1 /32 192.168.0.1 1 10.0.0.1 /32 192.168.0.2 2

192.168.0.1

192.168.0.2

10.0.0.1

10.0.0.1


How Does Anycast Address DDoS Attacks?• From any one location on the Internet, you can

only see (and hence attack) a single member of an anycast group at once

• If you succeed in taking out that replica, routing will shift traffic to another

�The first replica will probablyrecover

�It’s like Whac-A-Mole


Screen Traffic to Your Name Servers• Take advantage of any anti-DDoS features built into

devices on the path between your name servers and the Internet, such as�Internet firewalls�Load balancers

• For example�SYN flood mitigation, such as rate limiting SYN frames�Router traffic shaping of UDP

20


Monitor Traffic to Your Name Servers• Monitor traffic to your name servers, including

�Aggregate query rate�Top queriers

21


Monitoring Aggregate Query Rate

22


Setting an Alert on Aggregate Query Rate

23


Monitoring Top Clients

24


Don’t Be a Part of the Problem1. Use ingress filtering2. Apply ACLs to your recursive name servers3. Rate-limit traffic or responses from your name

servers

25


Rate-limit Traffic from Your Name Servers• If you can, rate-limit traffic from your name servers

�Using Response Rate Limiting, for example- A patch to BIND 9 by Paul Vixie and Vernon Schryver- Applies to authoritative name servers used in DDoS attacks

against others- Prevents these name servers from sending the same response to

the same client too frequently- Implemented in

– NSD (3.2.15)– Knot (1.2-RC3)– As patches to BIND 9.8 and later

- See www.redbarn.org/dns/ratelimits

26


How RRL Works

isc.org/ANY[3335 byte response]

tokenbucket


Thank you!

28


Going from one datacenter to five:

The Iovation Story





30© 2012 iovation Inc.

What iovation Does

Recognize devices connecting to websites

Understands how these devices are related to each other

Block devices that are known to be associated with fraud or abuse and flag anomalies


Driving Factors• Successful Product

But ran into capacity limits scaling vertically Vertically scaling became cost prohibitive Unable to add features due to compute capacity limits

• Availability As we entered new markets, customers demanded higher

levels of availability Taking monthly downtimes for maintenance and code

upgrades no longer possible

• Disaster Preparedness We were operating out of a single datacenter which

represented risk to the business


Design Criteria• Scalability

System must be able to scale horizontally by adding more nodes

• Availability Code deployments do not require taking any real time

services down Real time services must continue functioning through the

loss of a datacenter PLUS the simultaneous loss of a single server in another datacenter

Serving datacenters must be geographically disparate

• Financial Must be based on commodity x86 hardware, running on

open source software, without depending on SAN’s


Service Oriented Architecture

Consumer Facing

Subscriber Facing

Internal Service

Real-time

Asynchronous

Web Service APIs

Device Recognition

Service

Association & Reputation

Service

Business Rules Service

Admin Console UI Reporting

Message Bus

Web Device Print Distribution

GeoService

VelocityService

Analytics

WWW Internet

iovationsubscribers

consumers


Datacenter Types• Primary Data Processing

Real-time customer requests are sent to these facilities Every piece of the real-time system is N+1 redundant

• Data Storage Each datacenter that is designated for data storage has one

copy of all key data elements Storage nodes do not have storage level redundancy (the

redundancy is across-datacenters)

• Content Delivery These datacenters deliver content to our customers end users

computers Are N+1 redundant such that individual failures do not cause

the loss of the entire node


Network Design

BB1

AMS

MIA

10g

20g

10gInternet

consumers

Content Delivery

#3

Data StoragePDX

MIAData StorageSEA

iovationsubscribers

Data Storage

Content Delivery

Data Processing

Subscriber Queries

Private Network

Content Downloads


Portland to Seattle

http://www.zayo.com/sites/default/files/images/Zayo-US-Network-EXTERNAL-11-1-2012.kmz


How does Iovation use Dyn?

•API Interface• Active/Active between two sites

•Admin Console• Active/Active between two sites

•Content Distribution• GSLB among four sites






• DNS cache poisoning, DNSSEC and general DNS security






Anatomy of a HTTP connection






How does DNS loadbalancing work?






Simple active/passive example

• Primary location assumes 100% traffic

• In event of disaster, swing 100% of traffic to

a standby location• Could be a “we’ll be back soon” or “status” page• Could be a backup copy of your app

• We call this Active Failover





















Active/Active Load Balancing

(Global Server Load Balancing, GSLB)

(Hot/Hot Load Balancing)

(High Availability Load Balancing)






Traffic management with

Dyn Traffic Director











Dyn Traffic Director

• Fast Anycast network enables low TTLs

• Monitor endpoints for health

• Globally load balance among 7 regions

• Use Anycast to gauge “where is the user?”
















Favor performance over network topology?

• Real-time monitoring of endpoints

• Always serve the fastest endpoint for each

user, regardless of network topology

• That’s real-time traffic management with

Dyn’s Traffic Director





















Favor geopolitical boundaries above all?

• Per query lookup on source address

• Geopolitical IP mapping database

• State by state and country by country

granularity

• That’s geo traffic management with Dyn’s

Traffic Director











Traffic Management Recap

• Active/Passive with health checks• Dyn Active Failover

• Active/Active with health checks• 7 global regions by network topology -> Dyn Traffic

Director• Add in real-time latency measurement -> Dyn Traffic

Director with real-time traffic management• Add in geopolitical granularity -> Dyn Traffic Director

with geo traffic management











Dyn Delivers Internet Performance

•Traffic management and managed

DNS

•Message management and email

delivery

•Remote access and domain services





Pg. 60 Presentation Title (edit from Slide Master) @twitterhandle

Scalability and Availabilityin the Real World

Cory von WallensteinChief Technologist,

Dyn [email protected]

@cvwdyn

Thank You!

Cricket Liu, Chief Infrastructure Officer,

[email protected]

@cricketondns





Date post:	22-Oct-2014
Category:	Technology
View:	678 times
Download:	3 times

Dyn Roadshow: Cricket Liu & Cory von Wallenstein On Scalability & Availability

Technology