Bots for MeetUp[1] (Read-Only) Web... · ©2016"Imperva,"Inc."All"rights"reserved....

© 2016 Imperva, Inc. All rights reserved.

Understanding Web Bots and How They Hurt Your Business

Ofer GayerProduct Manager

May 12, 2016

@ZigZag_IL


Overview

• An overview of Bot technology• How bots are used for Hacking and Denial of Service Attacks• The Impact of Content Scraping on Websites• Suggestions for Bot detection and Mitigation

2


Speaker Bio for Nabeel Saeed

• Background– 5+ years experience with web application securityand SaaS security solutions

– Held product marketing roles at Imperva, Incapsula, Vertical Systems, etc.

• Contact:• Email: [email protected]

3


Speaker Bio for Ofer Gayer

• Background– 10+ years experience with web application security and SaaS security solutions

– Recovering security researcher– Built and maintained Incapsula’s DDoS Mitigation logic.– Giraffe impersonator.– Knows his bots.

• Product Manager• Favorite Bot: “BOT for JCE” - I appreciate the honesty.• Email: [email protected]• Twitter: @ZigZag_IL4


What is an Internet Bot?

• A bot is a software program that runs automated tasks over the internet

• They typically perform simple, repetitive tasks

• Are able to operate at a higher rate of speed than humans can achieve

5


Popular Legitimate Uses for Web Bots

Bots tend to visit websites in regular cycles performing tasks like• Search Engine Crawling– Google– Bing– Yandex– Baidu

• Website Health Monitoring • Fetching Web Content• Web vulnerability Scanning• Operating APIs (Application Programming Interfaces)

6


Automated Clients are Almost Half of Web Traffic

Over 66% of all bot traffic is malicious.

7


The Impact of Bots on Website Security

• Good Bots• Search Engine Crawling

• Website Health

Monitoring

• Fetching Content

• Powering APIs

• Automation

• Vulnerability Scanning

• Bad Bots• DDoS

• Site Scraping

• Comment Spam

• SEO Spam

• Competitive Analysis

• Vulnerability scanning

8


Evolution of Bots

• Bots are increasingly able to imitate browser and human behavior to bypass security solutions

• Browser-based bots which live inside of infected browsers are becoming more sophisticated

30%Bots that accept Javascript and cookies

9


DDoS Attack Landscape Trends

2x

Seen every day in 2015

>100Gbps >50Mpps

10

The number of DDoS attacks in 2015 vs. 2014


Imposter Google Bots are on the Rise

11

Googlebot visits websites an averageof 187 times per day.

On average 7 of them are fake.


Imposter Google Bots are on the Rise

Google Imposter Bots by Activity Type

12


How Bots are Used for Hacking2

13


Bots and Automated Indiscriminate Attacks

• Bot traffic behind more than 90% of all security events, including:– SQLi / XSS / RFI– High profile CVEs - Heartbleed, Shellshock, TimThumb, Magento Shoplift– Login dictionary attacks– Spam – Comment Spam, Referer Spam

• Bots scan millions of sites– Dorks– Shodan– Crawling

• Why it matters– A real threat to small and medium-sized websites

• Compromised for their resources• Often targeted as part of massive campaigns

14


Bots and Comment Spam

• What is Comment Spam– Posts in comment sections on websites allegedly linking to:

• “Male enhancement pills”• Streams of popular TV shows• Insurance, car rental• Designer clothing, etc.

• How bots are involved– Bots are used to automatically find victim sites and insert spam posts

• Why it matters– Comment spam is frequently responsible for

• Degrades UX and reputation• Lower website conversions (links usually exit your site)• Malware distribution (infecting your visitors)

15


SEO Referral Spam

What is it?1. Semalt is a Ukrainian search engine optimization (SEO) “company”2. They used malware to hijack computers and create a giant botnet3. This Botnet visits sites across the internet with fake referral sources

What damage could this cause your website?• Spamming Google Analytics• Long term SEO Damage to your website’s rankings• Complete search engine result page blacklisting and removal

16


SEO Referral Spam

17


Bots for Distributed Denial Of Service (DDoS) Attacks

• DDoS attack are attacks where many infected computers band together to attack a single target

• These attacks exhaust network connections and server resources causing website outages

18


How DDoS Attacks Impact Site Availability

• DDoS attacks make your website completely inaccessible

• If website availability is important to you, then DDoS protection should be too• Any application without a DDoS mitigation strategy is at risk

LegitimateTraffic

Your Site

Your Internet

ConnectionYour ISP

DDoS Bots

19


Bots as Website Reconnaissance

• Website Vulnerability Scanners – Powered by bots at scale– Crawl websites searching for security flaws– Provide operators with a list of technologies used by websites (can later be used when a new CVE appears)

List of

Vulnerabilities

20


Websites Have Many Vulnerabilities

96% of web applications have vulnerabilities

13% of websites can be compromised automatically

21

96%WEBAPP

Sources: Cenzic, Inc. – Feb. 2014, Incapsula, Inc. –2013

13%


Bots for Scraping

• Site scraping bot visit a website to copy or steal content• Database scraping bots enter all possible parameters into an application to retrieve content from a database

• Site scraping can lead to IP theft or competitive disadvantage

Bot

Your Content

Your Site Your DB

22


Identifying and Mitigating Bots3

23


Inspecting Website Traffic for Bots

• Static approach:– Structure of web requests– Header information– Visitor browser agent info– Reputation

• Progressive challenge approach– Cookies– JS– CAPTCHA

• Behavioral approach– Order and frequency of requests– Interaction between clients and servers– Javascript Injection to actively classify clients

24


Identify and Block Bad Bots

• Implement a solution which can:– Block bad bots automatically– Allow access for good bots

• Bot mitigation solutions– Standalone service– Homebrew rule-set– Part of a technology used in a WAF

25


Website Security and Performance in Minutes with a Simple DNS Change

By routing website traffic through the Incapsula network, malicious traffic is blocked, and legitimate traffic is accelerated.

For a Free Trial of Incapsula visit us atwww.Incapsula.com

Incapsula Network Your WebsiteLegitimate Traffic

26


Want to Learn More?

View the Global Bot Traffic Report 2015

visit http://bit.ly/2015BotTrafficReport

Global Bot Traffic ReportA statistical study of the bot traffic landscape

2015

27


Questions?

28

© 2016 Imperva, Inc. All rights reserved.30

Date post:	14-Aug-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Bots for MeetUp[1] (Read-Only) Web... · ©2016"Imperva,"Inc."All"rights"reserved....

Documents