main.pdf

A PROJECT REPORT ON

A HYBRID MODEL TO DETECT PHISHING SITES USING

CLUSTERING AND BAYESIAN APPROACH

SUBMITTED TO UNIVERSITY OF PUNE,

IN THE PARTIAL FULFILMENT OF THE REQUIREMENTS FOR

AWARD OF BACHELORS

OF

BACHELOR OF ENGINEERING (COMPUTER ENGINEERING)

BHUSHAN DHAMDHERE

ROHIT CHINCHWADE

KAUSHAL DHONDE

SWAPNIL MEHETRE

Under the Guidance of

PROF. RAHUL PATIL

DEPARTMENT OF COMPUTER ENGINEERING,

PIMPRI CHINCHWAD COLLEGE OF ENGINEERING,PUNE

DEPARTMENT OF COMPUTER ENGINEERING

PIMPRI CHINCHWAD COLLEGE OF ENGINEERING,PUNE

CERTIFICATE

This is to certify The Final Year Project report entitled

A HYBRID MODEL TO DETECT PHISHING SITES USING CLUSTERING AND

BAYESIAN APPROACH

is a record of bonafide work for Project carried out by and submitted by

BHUSHAN DHAMDHERE

ROHIT CHINCHWADE

KAUSHAL DHONDE

SWAPNIL MEHETRE

Under Guidance Of

Prof. Rahul Patil,

in partial fulfillment of the requirements for the

award of Degree of Bachelors in Computer Engineering of University of Pune.

(PROF. RAHUL PATIL) (PROF. DR. J. S. UMALE)

Project Guide Head, Computer Engineering

Examination Approval Sheet

The Project Report entitled

A HYBRID MODEL TO DETECT PHISHING SITES USING CLUSTERING AND

BAYESIAN APPROACH

By

Bhushan Dhamdhere

Rohit Chinchwade

Kaushal Dhonde

Swapnil Mehetre

is approved for Project, B.E Computer Engineering, University of Pune

at

Pimpri Chinchwad College of Engineering

Examiners :

External Examiner :

Internal Examiner :

Date :

Acknowledgments

We express our sincere thanks to our Guide Prof. Rahul Patil, for his constant encourage-

ment and support throughout our project, especially for the useful suggestions given during the

course of project and having laid down the foundation for the success of this work.

We would also like to thank our Project Coordinator Mrs. Deepa Abin, for her assistance,

genuine support and guidance from early stages of the project. We would like to thank Prof.

Dr. J. S. Umale, Head of Computer Department for his unwavering support during the entire

course of this project work. We are very grateful to our Principal Dr. A. M. Fulambarkar for

providing us with an environment to complete our project successfully. We also thank all the

staff members of our college and technicians for their help in making this project a success. We

also thank all the web committees for enriching us with their immense knowledge. Finally, we

take this opportunity to extend our deep appreciation to our family and friends, for all that they

meant to us during the crucial times of the completion of our project.

Bhushan Dhamdhere

Rohit Chinchwade

Kaushal Dhonde

Swapnil Mehetre

Contents

List of Figures viii

List of Tables x

Abstract xi

1 Introduction 1

1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Brief Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.4 Applying Software Engineering approach . . . . . . . . . . . . . . . . . . . . 3

2 Literature Survey 5

3 Software Requirements Specifications 11

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1.2 Intended audience and reading suggestions . . . . . . . . . . . . . . . 12

3.1.3 Project Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.1.4 Design and Implementation Constraints . . . . . . . . . . . . . . . . . 12

3.1.5 Assumptions and Dependencies . . . . . . . . . . . . . . . . . . . . . 13

3.2 System Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2.1 System Feature 1: String Searching . . . . . . . . . . . . . . . . . . . 14

3.2.2 System Feature 2: String Tokenization . . . . . . . . . . . . . . . . . . 14

3.2.3 System Feature 3: K-Means Clustering . . . . . . . . . . . . . . . . . 15

3.2.4 System Feature 4: DOM Tree Parsing . . . . . . . . . . . . . . . . . . 16

v

3.2.5 System Feature 5: Naive Bayes Classifier . . . . . . . . . . . . . . . . 16

3.3 External Interface Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3.1 User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3.2 Hardware Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3.3 Software Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3.4 Communication Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 20

3.4 Non-Functional Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.4.1 Performance Requirements . . . . . . . . . . . . . . . . . . . . . . . . 21

3.4.2 Safety Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.4.3 Security Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.4.4 Software Quality Attributes . . . . . . . . . . . . . . . . . . . . . . . 21

3.5 Analysis Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.5.1 Data Flow Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.5.2 Class Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.5.3 State-Transition Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.6 System Implementation Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.6.1 Cost Estimation Model . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.6.2 Gantt Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4 System Design 28

4.1 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.2 UML Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5 Technical Specification 42

5.1 Technology used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6 Schedule, Estimate and Team Structure 45

6.1 Project Estimate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

6.2 Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

6.3 Team Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

7 Software Implementation 49

7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

7.2 Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

vi

7.3 Important Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

7.4 Business Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

8 Software Testing 61

8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

8.2 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

8.3 Snapshot of GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

9 Results 66

9.1 Accuracy of Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

9.2 Project Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

10 Deployment and Maintenance 70

10.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

11 Appendix A: Glossary 74

12 Appendix B: Semester I Assignments 75

vii

List of Figures

2.1 Total reported attacks per month for 1 year[7] . . . . . . . . . . . . . . . . . . 7

2.2 Major attacked countries by volume of attack[7] . . . . . . . . . . . . . . . . . 7

2.3 Major attacked countries by Brands attacked[7] . . . . . . . . . . . . . . . . . 8

3.1 Level 1 DFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.2 Level 2 DFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.3 Class Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.4 State-Transition Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.5 Cocomo-II Embedded Project Model . . . . . . . . . . . . . . . . . . . . . . . 25

3.6 Gantt Chart Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1 System Architecture Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.2 Feature Extraction Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.3 K-Means Clustering Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.4 Naive Bayes Classifier Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.5 Use Case Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.6 Class Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.7 Sequence Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.8 State-Transition Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.9 Collaboration Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.10 Package Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.11 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.12 Component Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.13 Deployment Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

6.1 Cocomo-II Embedded Project Model . . . . . . . . . . . . . . . . . . . . . . . 45

viii

7.1 Sample DOM Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7.2 DOM Tree constructed in PROJECT . . . . . . . . . . . . . . . . . . . . . . . 52

8.1 Test Cases for Project Main Modules . . . . . . . . . . . . . . . . . . . . . . . 62

8.2 Main Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

8.3 Manual Entry Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

8.4 Manual Entry Form Empty . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

8.5 Prediction Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

8.6 Load Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

9.1 Accuracy Testing graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

9.2 New site feature extraction in progress . . . . . . . . . . . . . . . . . . . . . . 67

9.3 Prediction Results of Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

10.1 JDK Step 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

10.2 JDK Step 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

10.3 JDK Step 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

10.4 JDK Step 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

ix

List of Tables

7.1 Sample Dataset for K-Means Clustering . . . . . . . . . . . . . . . . . . . . . 54

7.2 Initial Cluster Centroid values . . . . . . . . . . . . . . . . . . . . . . . . . . 54

7.3 Dataset after clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.4 Final Cluster Centroid values . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.5 Sample Training data set of Classifier . . . . . . . . . . . . . . . . . . . . . . 57

7.6 New Unknown site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

7.7 Probability for feature set to be Original . . . . . . . . . . . . . . . . . . . . . 58

7.8 Probability for feature set to be Phish . . . . . . . . . . . . . . . . . . . . . . . 58

x

Abstract

As the Electronic Commerce and On-line Trade expand, phishing has already become one

of the several forms of network crimes. Our project model presents an automatic approach

for intelligent phishing web detection based on learning from a large number of legitimate and

phishing webs. As given a web, its Uniform Resource Locator (URL) features are first analyzed,

and then classified by K-Means Clustering. When the webs legality is still suspicious, its web

page is parsed into a document object model tree, and then classified by Naive Bayes Classifier

(NB). Experimental results show that our approach can achieve the high detection accuracy, the

lower detection time and performance with a small sample of the classification model training

set.

A novel framework using a Bayesian approach for content-based phishing web page detection

is presented. Our model takes into account textual and visual contents to measure the similarity

between the protected web page and suspicious web pages. A text classifier and an algorithm

fusing the results from classifiers are introduced.

Chapter 1

Introduction

1.1 Overview

One of the most dangerous attacks in the todays internet trend are happening in the form of

phishing sites. The major attacks are done to retrieve the personal information of the users from

the banking sectors.

Phishing is the act of acquiring electronic information such as Username, Password, and

Credit-Cards Information by masquerading as trustworthy authority. This information may be

used to retrieve some information by logging into the system with these username and password

or performing some transaction with the use of username, password and credit card information

retrieved from this phishing.

Phishing can be of many types but nowadays the very usual way of phishing is through the

E-Mail or creating the Web-Sites of brands (like ICICI Bank, SBI Bank, www.faceboook.com,

etc.) which looks very alike with their legitimate sites and asking users to enter their username

password or any such personal information.

Phishing sites are the major attacks by which most of internet users are being fooled by the

phisher. The replicas of the legitimate sites are created and users are directed to that web site

by luring some offers to it. There are certain standards which are given by W3C (World Wide

Web Consortium), based on these standards we are choosing some features which can easily

describe the difference between legit site and phish site.

1

CHAPTER 1. INTRODUCTION

Phisher is the community of hackers which creates the replicas of the legitimate web sites

to retrieve users personal information such as passwords, credit card number, and financial

transaction information. As per the survey done by RSA Fraud Surveyor, the Phishing attacks

have been raised by 2% since the last December 2012 to January 2013.

The W3C has set some standards that are followed by most of the legit sites but a phisher

may not care to follow these standards as this site is intended to catch many fish in very small

amount of time and bait. There are certain characteristics of the URLs and source code of the

Phishing site based on which we can guess the site is fake or not.

To detect and prevent the attacks from such phishing sites various preventive strategies are

employed by anti-phishing service providers like Google Toolbar, an Anti-Virus service provider.

These are the most common in the anti-phishing service providers. These service providers are

creating and maintaining the databases of blacklisted sites. Some of the anti-phishing organiza-

tions are available like www.phishtank.com who maintains the blacklist of the reported phishing

sites and their current status if they are still online or not.

The phisher are creating sites at such a rate that there always will be some period in what

the site is not reported as phish, in that case these techniques of maintaining online blacklist

repositories fails. The major drawback or setback we have seen in this method is like the

normal user will not always be taking caution about the phishing site, he may get tricked by

overall look of site like legitimate site and it may happen like the site is not yet verified by the

service providers and hence is not blocked.

1.2 Brief Description

We are proposing the system which will detect the phishing sites based on training models

provided after studying the results from various phishing sites. We have proposed an approach

where we will determine the site is phishing or not based on URL and HTML features of the

website. We will first retrieve the URL features from the URL of the website such as follows:

IP as URL

Dots in URL

2 Dept. of Comp. Engg. PCCOE Pune-44.


Slashes in URL

Suspicious Characters in URL

After retrieving these features of URL we will download the source code of URL webpage

and parse using HTML DOM Parser to get more HTML features from the website as follows:

Null Anchors count in URL

Foreign Anchors count in URL

HTTPS /SSL /TSL certificate validity check

1.3 Problem Definition

The aim of our project is towards the detection of phishing web pages by selecting textual

and visual contents of the Web-Site such as URL features and Anchor tag features from visual

contents of web pages, we are applying string parsing algorithm on textual features and using

DOM tree of the web-sites visual content to analyze further features which may contribute to

the prediction of the result more efficiently.

The model which we are proposing uses the textual features from Web-Site such as: no. of

slashes in URL, no. of dots in the URL; these features are used to put the Web-Site in the cluster

of the database using K-Means Clustering algorithm.

If the model still lies in the Suspicious Cluster, more visual features are extracted by down-

loading the Web-Site and applying DOM Tree Parsing then extracting features we require like

HTTPS:// or SSL certified, No. of Foreign anchor tags, No. of Null anchor tags. Then we

are applying Naive Bayes Classifier which will be predicting the result thus results are more

correctly predicted.

1.4 Applying Software Engineering approach

New Advances in internet technology and the rapid growth of networks in quality and quan-

tity has introduced new applications and concerns in Internet Banking and industry. The unique



requirements and constraints associated with Internet Securities have brought new challenges

to software development for such environments, as it demands extensive improvements to tra-

ditional Anti-Phishing systems development methodologies in order to fulfill the special needs

of this field.

We examine the challenges of developing software for personal system connected to internet,

starting by reviewing website characteristics and investigating the status Anti-Phishing software

development methods. It has been shown that Agile methodologies are appropriate methods for

the development of such systems; based on this assumption, we identify specific requirements

for a Internet Security software development methodology, based on which a new agile method

is engineered using the Hybrid Methodology Design approach.


Chapter 2

Literature Survey

The literature survey of the anti-phishing has been done for this model and following are the

conclusive records of the literature survey.

The model is being surveyed with respect to following points:

1. Existing Model

2. Current Phishing Status

3. Existing documentation for the proposed model which is being referred for the current

project.

Existing Models

1. Plug-in for Browsers

The browser plug-ins which are used (for Mozilla Firefox, Google Chrome) to

detect the site is phishing site or not. The working of the browser is like whenever

you enter the URL in the browsers address bar, the browser will just copy the URL

and the plug-in will send the URL to the online repository of the browser and the

browser will search for the entries for that URL and if there are no entries it will not

raise the alarm even if the site is Phishing site[8].

If the site is not present in the repository of the browser it will not raise the alarm

and user will continue to the Web-Site because the plug-in is showing this site is not

malicious.

5

CHAPTER 2. LITERATURE SURVEY

It may not be possible for the online repository to maintain the record of each and

every site because there is a very large no. of Web-Sites launched every day.

2. Anti-Viruses having Internet Phishing Security.

Anti-Virus works very similar to the browser plug-in it also catches the URL from

the browser and checks into its own repository which may be updated at the client

site on daily basis.

Here the anti-virus service provider is making the surveys and it checks the sites

on regular basis and updates the database if the phishing site is found then the

database is updated at the client end which prevents the attacks more efficiently

than only depending upon the plug-ins of the browser[8].

The question remains same for the new web sites which has not yet being iden-

tified by the anti-virus service provider. There is no any protection for the user and

user relies on the anti-virus service provider that the site may be being tested by the

anti-virus. The models which are used to detect the Phishing attack uses only the

URL features to predict the site is malicious or not even they are using the Visual

features of the sites very low amount of features are used to predict and machine

learning approach is not yet being used to detect the phishing sites[4].

Current Phishing Status

Looking at the First fortnight report by Anti-Phishing Organization (www.antiphishing.org)

and RSA Online Fraud Attacks Surveys few major points:

Phishing attacks has been increased by 2% since December 2012.

India is having 4% of global attacks by volume of attack.

India is being targeted 4% of global attacks by volume of brands attacked.



Figure 2.1: Total reported attacks per month for 1 year[7]

In January, RSA identified 30,151 attacks launched worldwide, a 2% increase in attack vol-

ume from December. Considering historical data, the overall trend in attack numbers in an

annual view shows slightly lower volume of attacks through the first quarter of the year.

Figure 2.2: Major attacked countries by volume of attack[7]

The U.S. was targeted by phishing attacks most in January, with 57% of total phishing vol-

ume. The UK endured 10%, followed by India and Canada both on 4% of attack volume

respectively.



Figure 2.3: Major attacked countries by Brands attacked[7]

Brands in US were most targeted in January; 30% of Phishing attacks were targeting US or-

ganizations followed by UK representing 11% of worldwide brands attacked by Phishers. Other

nations whose brands were most targeted includes India, Italy, Australia, France and Brazil.

Supporting papers

A Layout-Similarity-Based Approach for Detecting Phishing Pages-Angelo P.

E.Rosiello, Engin Kirda, Christopher Kruegel, Fabrizio Ferrandi, Politecnico di Mi-

lano

In this paper, an extension of our system (called DOM-Anti-Phish) that mitigates

the shortcomings of our previous system. In particular, our novel approach lever-

ages layout similarity information to distinguish between malicious and benign web

pages. This makes it possible to reduce the involvement of the user and signifi-

cantly reduces the false alarm rate. Our experimental evaluation demonstrates that

our solution is feasible in practice.



We are referring the use of DOM Tree for the feature extraction process and the

Visual features of the Web-Pages.

Textual and Visual Content-Based Anti-Phishing: A Bayesian Approach-IEEE

Transactions October- 2011 Haijun Zhang, Gang Liu, Tommy W. S. Chow, Senior

Member, IEEE, and Wenyin Liu, Senior Member, IEEE

A novel framework using a Bayesian approach for content-based phishing web

page detection is presented. Our model takes into account textual and visual con-

tents to measure he similarity between the protected web page and suspicious web

pages. A text classifier, an image classifier, and an algorithm fusing the results from

classifiers are introduced. An outstanding feature of this paper is the exploration of

a Bayesian model to estimate the matching threshold. This is required in the classi-

fier for determining the class of the web page and identifying whether the web page

is phishing or not. In the text classifier, the Naive Bayes rule is used to calculate

the probability that a web page is phishing. In the image classifier, the earth movers

distance is employed to measure the visual similarity, and our Bayesian model is

designed to determine the threshold. In the data fusion algorithm, the Bayes the-

ory is used to synthesize the classification results from textual and visual content.

The effectiveness of our proposed approach was examined in a large-scale data set

collected from real phishing cases. Experimental results demonstrated that the text

classifier and the image classifier we designed deliver promising results, the fusion

algorithm outperforms either of the individual classifiers, and our model can be

adapted to different phishing cases.

We are referring the use of Naive Bayes Classifier for the detection of the mali-

cious Web-Pages.

An Efficient Approach to Detecting Phishing Web-Xiaoqing GU, Hongyuan

WANG, Tongguang NI



This paper presents an automatic approach for intelligent phishing web detection

based on learning from a large number of legitimate and phishing webs. As given a

web, its Uniform Resource Locator (URL) features are first analyzed, and then clas-

sified by Naive Bayesian(NB)classifier. When the webs legality is still suspicious,

its web page is parsed into a document object model tree, and then classified by Sup-

port Vector Machine (SVM) classifier. Experimental results show that our approach

can achieve the high detection accuracy, the lower detection time and performance

with a small sample of the classification model training set.

This paper refers to the use of textual features of the URL which can be used for

the detection of the fraud Web-Pages.


Chapter 3

Software Requirements Specifications

3.1 Introduction

3.1.1 Purpose

Our project aims towards the detection of phishing web pages by selecting textual and visual

contents of the Web-Site such as URL features and Anchor tag features from visual contents of

web pages, we are applying string parsing algorithm on textual features and using DOM tree of

the web-sites visual content to analyze further features which may contribute to the prediction

of the result more efficiently.

The model which we are proposing uses the textual feature from Web-Site such as: no. of

slashes in URL, no. of dots in the URL; these features are used to put the Web-Site in the cluster

of the database using K-Means Clustering algorithm[8].

If the model still lies in the Suspicious Cluster more visual features are extracted by down-

loading the Web-Site and applying DOM Tree Parsing then extracting features we require like

HTTPS:// or SSL certified, No. of Foreign anchor tags, No. of Null anchor tags. Then we

are applying Naive Bayes Classifier which will be predicting the result thus results are more

correctly predicted[3][2].

11

CHAPTER 3. SOFTWARE REQUIREMENTS SPECIFICATIONS

3.1.2 Intended audience and reading suggestions

This SRS is intended for the reading of Project Developing team, Project Analysis team,

Project Head, Users and other managing committee. This project SRS is following IEEE stan-

dard format in IEEE Standards 830-1998.

The readers of this SRS are advised to go through indexed points in order to access this SRS

more efficiently.

3.1.3 Project Scope

Phishing frequently impacts users privacy and safety. Internet Service Providers (ISPs) are

facing a huge problem in the Internet community from phishers and hackers. The scope of this

project revolves around the identification, reduction and elimination of phishing activities and

protection of users from phishing artists.

The Software will be detecting the web sites if they are malicious (Phishing) based on strong

features using clustering and if it is not able to detect the result, the software will use the Naive

Bayes Classifier Prediction which will give result based on probabilistic model.

In this model a fast and accurate approach is proposed to detect phishing web. Our approach

determines whether a web page is a phishing web or a legitimate one, based on its URL and

web page features, and is merely a combination of NB and K-Means. The K-Means classifier

used to detect the URL is that K-Means is a rapid detection method for classification and URL

features can be easily acquired. If the K-Means classifier cannot judge the given webs legality

definitely, the NB classifier is used to detect it based on its web page features.

3.1.4 Design and Implementation Constraints

Java Technology to be used

The Java technology enables portability and scalability of the software hence Java plat-

form is to be used. Most of the techniques used in the processing of the data are already

implemented in the Java hence reducing the efforts of programming.



HTTP communication protocol to be used

The software is using the internet access to download the Web Page for the textual

feature extraction if required for prediction hence the HTTP standard protocols is being

used for online data downloading.

Serialization of databases required

Serialization is the process by which the application can send program objects through

a stream, which can be a file stream or a network stream. Sending objects through a

stream will allow developers to create solutions that were not available until now.

Strong Database Requirement

The System will using the existing database entries to predict the result of the current

data set. Thus the system requires strong database of VALID as well as INVALID Phish

Entries without which it is very hard to produce the output for the Naive Bayes Classifier.

3.1.5 Assumptions and Dependencies

The Input Database is assumed to be correct.

The Database which we will be used for the initial entries of the training of the system

is assumed to be the correct input for the system. The URL which is selected as the

fake or phishing Web Pages must be the originally declared as phishing Web Page and

vice-versa.

The training data set is taken from the online repositories like www.phishtank.com from

where the known valid phish Web Pages can be retrieved and some legitimate web pages

directly taken from Google search tool.



3.2 System Features

3.2.1 System Feature 1: String Searching

String searching algorithms, sometimes called string matching algorithms, are an important

class of string algorithms that try to find a place where one or several strings (also called pat-

terns) are found within a larger string or text[1].

The String Parsing of the URL is to be done for extraction of feature from the URL and

creating the data set of the textual contents.

This Feature will create following set of database:

Total Number of slashes in URL.

Total Number of dots in URL.

Total Number of suspicious characters in URL.

URL as IP Address.

3.2.2 System Feature 2: String Tokenization

The system is accepting the CSV input in which all the entries for the given data set URL are

enclosed within single string and are separated by the commas.

This type of input cannot be directly transformed to the data set entry; we first need to format

that string according to the data set requirements. Hence the string tokenization is required to

accept the CSV from the User and store into database.

This Feature will work like following example:

CSV INPUT: http://www.my.input.com,0,3,0,0,2,1



This will produce following data set:

URL : http://www.my.input.com

Number of Slashes : 0

Number of Dots : 3

Suspicious Characters : 0

SSL Certificate : 3

Foreign Anchors : 2

Null Anchors : 1

3.2.3 System Feature 3: K-Means Clustering

The K-Means Clustering algorithm is used for clustering of the Strong Features of the system

which will be directly giving results in two clusters for a site is More Suspicious and Less

Suspicious.

K-Means clustering is applied onto the feature which may have discrete values in it, such as

count of suspicious characters, slashes, null anchors, foreign anchors and dots. These discrete

values are converted into the form of 0 and 1. 0 for less suspicious values and 1 for more

suspicious values are used. The feature will be providing the result in two clusters based on all

above mentioned features[5]:

This feature is of High priority and preliminary Data Mining will give the better performance

of system. Risk can be there as if the result is unpredictable one.

This feature will take the data set prepared by the String Searching feature of the system and

will apply K-Means Clustering Algorithm for Data Mining over the system.

arg maxSk

i=1

xjSi ||xi-i||2

Where i is the mean of points Si.



The feature will be providing the result in two forms based on only considerably strong

features of the Web-Site whose result is to be declared:

More Suspicious: If the values of the feature are very much larger then it has more suspicions

of being phishing site.

Less Suspicious: If the values of the feature are considerably low then they may not be treated

as the phishing site.

3.2.4 System Feature 4: DOM Tree Parsing

HTML Parser is a Java library used to parse HTML in either a linear or nested fashion.

Primarily used for transformation or extraction, it features filters, visitors, custom tags and easy

to use JavaBeans. It is a fast, robust and well tested package[6].

If the result of the K-Means lies in region Suspicious Region we need to extract visual features

of the URL this requires to download and parse the URL using DOM Tree.

This Parsing will help to identify the following data set:

SSL Certificate

NULL Anchor Tags

Foreign Anchor Tags

3.2.5 System Feature 5: Naive Bayes Classifier

Naive Bayes Classifier is the strong predictor algorithm which we will be using in this par-

ticular module but using it only if site is not predicted using the Clustering because of the cost

of execution of the algorithm.

This feature is of Medium priority and used for secondary Data Mining which will not give

the better performance of system but the accuracy of prediction can be achieved. The risk factor

in Clustering can be lowered using the Naive Bayes Classifier.



The Naive Bayes Classifier will be using the data set prepared by both String Searching and

the DOM Tree HTML Parsing to predict the output hence the results will be near to accurate[3].

Following is the formula to calculate the results:

Vnb = argmaxj P (j)P (ai|j)

Generally estimate P (ai|j) using m estimates.

P (ai|j) = nc+mpn+m

where,

n = the number training examples for which = j .

nc = the number of examples for which = j and a = ai.

p = a priori estimate for P (ai|j).

m = the equivalent sample size.

The feature will be providing the result in two forms based on all the features taken into data

set of the Web-Site whose result is to be declared:

Phishing Site: If the site is resulting into Valid Phish.

Legitimate Site: If the site is resulting into Invalid Phish.

3.3 External Interface Requirements

3.3.1 User Interfaces

The User of the system will be interacting with the system by using following functionality

provided:

1. Manage Dataset



This feature enables users to add or delete more records into the dataset.

2. Upload CSV

If user wishes to get the ready dataset from another site or another computer then he

may upload any CSV file that has compatible format of dataset.

3. Apply Clustering

To determine the new centroid after adding records into the dataset user can use this

features.

4. Prediction Model

To determine whether a site is phishing site or not user can use make use of main

feature of the project.

5. Save Database

After changing the database user needs to save the new database, for this user can rely

on this feature of the project.

3.3.2 Hardware Interfaces

Operating System:Windows Platform

Hardware:IntelrCore 2 Duoror better

Internet Connection

3.3.3 Software Interfaces

Java SDK:1.7 or better

Database System:My SQL

Libraries



DOM Tree Parser

SAX Parser

Database

Serialized Database

Operating System

Windows XP or better

Data Set[8]

URL

The input URL for which the detection is to be done by system.

IP as URL

Getting URL name will cost the phisher to buy space on some web-hosting site.

The phisher may ignore this and use the IP address itself as the URL.Legitimate

sites will always have some URL name.

Suspicious Characters

The total count of the characters which are not included A-Z and 0-9 in the URL.

The phisher may use tricky characters to look like the legitimate site and the stan-

dard procedure is not to include any other characters than A-Z and 0-9 for easy

remembering for users.

Phisher may trick the User by inserting any of

& % - _ @

to look like the web site as legitimate site.

Number of Slashes



Total number of the Slashes occurred in the URL. The URL should not contain

more number of slashes. If it contains more than five slashes then the URL is con-

sidered to be a phishing URL.

Number of Dots

Total number of the Slashes occurred in the URL. The dots may provide the in-

formation regarding the total number of sub-domains used by the URL. More the

sub-domains used more the suspicious site.

Number of Dots

1. NULL Anchor

A null anchor is an anchor that points to nowhere. The more nil anchors a

page has, the more suspicious it becomes.

2. Foreign Anchor

An anchor tag contains href attribute whose value is an URL to which the

page is linked with. If the domain name in the URL is not similar to the domain

in page URL then it is called as foreign anchor.

HTTPS-SSL Certificate

Most of the legitimate sites are using SSL certificate for online identity. SSL

certificate is provided by trusted authority and need to be updated by some time

period.

Phisher cannot get the SSL certificate by providing fake identity and will not

manage to update the certificate.

3.3.4 Communication Interfaces

Standard HTTP COMMUNICATION interface required for internet connection.



3.4 Non-Functional Requirements

3.4.1 Performance Requirements

The product must use the Clustering as the preliminary function to detect the phishing site if

that module is not able to determine then and then only go for Naive Bayes Classifier. This will

increase the performance of system as the Naive Bayes is Complex Algorithm for prediction

and K-Means Clustering is the easy method for Data Mining.

3.4.2 Safety Requirements

The safety of the system can be achieved by providing an authenticated login to the system

and limited privileges to the end users of the system to make the changes into the databases.

Safety of the system is achieved by providing backup of the data contained into system so that

even if the system crashes down during the working all the data would remain safe and data loss

would not take place.

3.4.3 Security Requirements

The system which is to be developed is provided with authentication (i.e., username and

password) so that other workers who should not be granted access to system are restricted.

This also helps us to keep the database secure from various actions to alter the data by an

unauthorized user.

3.4.4 Software Quality Attributes

1. We are not depending on only single Data Mining method thus we are ensuring reliability

of the software in case of failure of primary module, also correctness about the output can

be stated.

2. Most of the components can be used as cross platform so we can state the robustness of

the system.

3. Scalability of the software can be considered as the SQA as the Java components are to

be used, the java components can be modified and more packages classes can be added

into system to extend its features.



4. Portability can be achieved while using the java platform components as the java can be

easily available in any system it is open source and easy to install and use.

3.5 Analysis Models

3.5.1 Data Flow Diagrams

Figure 3.1: Level 1 DFD

Figure 3.2: Level 2 DFD



3.5.2 Class Diagram

Figure 3.3: Class Diagram



3.5.3 State-Transition Diagram

Figure 3.4: State-Transition Diagram

3.6 System Implementation Plan

3.6.1 Cost Estimation Model

Basic COCOMO computes software development effort (and cost) as a function of program

size. Program size is expressed in estimated thousands of source lines of code (SLOC)[1].

COCOMO applies to three classes of software projects:

Organic Projects-small teams with good experience working with less than rigid

requirements.



Semi-Detached Projects-medium teams with mixed experience working with a mix of

rigid and less than rigid requirements.

Embedded Projects- developed within a set of tight constraints. It is also combination

of organic and semi-detached projects.(hardware, software, operational, ...)

The basic COCOMO equations take the form:

1. Efforts Applied (E) = ab(KLOC)bb[person-months]

2. Development Time (D) = cb(E)db[months]

3. People Required (P ) = E/D [count]

Where, KLOC is the estimated number of delivered lines (expressed in thousands) of code

for project. The coefficients ab, bb, cb and db are given in the following table:

Organic 2.4 1.05 2.5 0.38

Semi-Detached 3.0 1.12 2.5 0.35

Embedded 3.6 1.20 2.5 0.32

Basic COCOMO is good for quick estimate of software costs. However it does not account

for differences in hardware constraints, personnel quality and experience, use of modern tools

and techniques, and so on.

Figure 3.5: Cocomo-II Embedded Project Model



3.6.2 Gantt Chart

Gantt charts illustrate the start and finish dates of the terminal elements and summary ele-

ments of a project. Terminal elements and summary elements comprise the work breakdown

structure of the project. Some Gantt charts also show the dependency (i.e. precedence network)

relationships between activities. Gantt charts can be used to show current schedule status using

percent-complete shadings and a vertical TODAY[1].



Figure 3.6: Gantt Chart Model


Chapter 4

System Design

4.1 System Architecture

Figure 4.1: System Architecture Model

Above figure explains the architecture for the system which contains the major components

and their connectors along with the topology among the components.

The System is having 3 major modules divided into:

28

CHAPTER 4. SYSTEM DESIGN

1. Feature Extraction

This feature will extract the features of the URL required to identify the Phish site.

This includes various methods which are explained in next section.

2. Apply Clustering Algorithm

The database clustering is to be done by using K-Means Clustering which will help to

produce the results at very early stage and using very small amount of data set from the

features extracted by previous methods.

3. Apply Naive Bayes Classifier

Naive Bayes Classifier is only used when system has plotted current data set in suspi-

cious cluster using K-Means Clustering. NB then use all the features and compare them

with existing data set finally producing a prediction result about the site is VALID or

INVALID Phish.

Feature Extraction

The URL is provided as the input to the system and system needs to apply some methods to

fetch the features from that URL. Feature includes Visual and Textual features.

The Feature extraction process will involve two measure algorithms to extract the features

from the URL which are String Searching Algorithm and DOM Tree Parsing Algorithm.

String Searching Algorithm will be used to determine the textual features of the web site

URL. DOM Tree Parser will be used to parse the HTML source code of Web-Page and extract

required features from the DOM Tree.



Figure 4.2: Feature Extraction Model

Clustering Algorithm

The Data set prepared by using feature extraction process is used in Data Mining Algorithm

of K-Means Clustering where three clusters of the system are created VALID Phish, INVALID

Phish, and suspicious Cluster.

According to threshold value of the Data set it is inserted into the cluster if the site is showing

high threshold value then it should go into VALID phish where it can be declared as Phishing

Web Page.

If value of data set is very low than threshold value, the web page lies into INVALID Phish

cluster where it is declared as the Legitimate Web page.

If value of data set is near to the threshold value, the web page lies into suspicious cluster

where another method of classification is applied to predict the result.



Figure 4.3: K-Means Clustering Model

Naive Bayes Classifier

Once the site is inserted into the Suspicious cluster of database the Naive Bayes Classifier

is applied onto that data set where the data set is compared with respect to existing data set in

database and the results produced if site is VALID phish or INVALID phish and accordingly it

is shifted from suspicious cluster to applicable cluster.



Figure 4.4: Naive Bayes Classifier Model

4.2 UML Diagrams

Use Case Diagram

A use case diagram at its simplest is a representation of a users interaction with the system

and depicting the specifications of a use case. A use case itself might drill into a lot of detail

about every possibility; a use-case diagram can help provide a higher-level view of the system.

For our system only actor applicable is the User itself he can perform the tasks such as logging

into system and accessing application to provide input to the system. Other tasks are included

into the accessing the application itself such as Enter URL, Enter CSV File, Access Database,

Apply System Functionality.



Figure 4.5: Use Case Diagram



Class Diagram

There are 5 classes which can be identified based on the features and functions of the respec-

tive class.

Single user class is identified and the user is able to access the system for login, log-out,

manage the databases, view output results etc. This class is having one to one association with

Application class

Other two classes based on the basic functionality are K-Means and NaiveBayes which are

performing computations and providing the results for the system.

Main Application class is the parent class of all the other classes and it consists all the func-

tionality control of the application, these other classes are called using Application class.

Figure 4.6: Class Diagram



Sequence Diagram

The sequence diagram provides the flow messages with respect to the time. In the given

system only Log-in and Log-out Stimulus are having synchronous messages of Authentication

and Confirmation messages respectively.

All the other stimulus are asynchronous in nature as the system is performing its action and

leaving the data set at its place so no return call for the stimulus is being used for this purpose

of messages.

Figure 4.7: Sequence Diagram



State-Transition Diagram

State transition diagram provides various phase the software or application will go throughout

its life cycle.

Here application being developed goes through various phases of activities which are going

to be performed one after another.

Figure 4.8: State-Transition Diagram

Collaboration Diagram

Communication diagrams show a lot of the same information as sequence diagrams, but

because of how the information is presented, some of it is easier to find in one diagram than

the other. Communication diagrams show which elements each one interacts with better, but

sequence diagrams show the order in which the interactions take place more clearly.



In order to maintain the ordering of messages in such a free-form diagram, messages are

labeled with a chronological number and placed near the link the message is sent over. Reading

a communication diagram involves starting at message 1.0, and following the messages from

object to object.

For the given system there are no sub messages to communicate amongst the objects only the

messages are communicated through one object to another irrespective of return call for that

message.

Figure 4.9: Collaboration Diagram



Package Diagram

Package diagrams can use packages that represent the different layers of a software system

to illustrate the layered architecture of a software system. The dependencies between these

packages can be adorned with labels / stereotypes to indicate the communication mechanism

between the layers.

Package diagram used for this application contains mainly the packages from Java platform as

the development platform is java platform and most of the functions are derived from the inbuilt

packages from the java technology hence main Java package includes various sub packages as

AWT, JPCAP and many more.

Figure 4.10: Package Diagram



Activity Diagram

Here the prediction using Naive Bayes and prediction using K-Means can be performed in

parallel and all the other activities are needed to be performed in serial.

The activities are mostly system controlled hence swim-lane is not required to be shown.

Also very few activities are branching and conditional activities as log-on and log-out.

Figure 4.11: Activity Diagram



Component Diagram

When using a component diagram to show the internal structure of a component, the provided

and required interfaces of the encompassing component can delegate to the corresponding in-

terfaces of the contained components.

Major components that can be distinguished based on the functionality of the system are

given in above diagram from our system. A Java collections component includes the packages

and classes which will be used as it is from the java software development kit. Process Builders

is the component which enable system to download the web site and URL.

Serialization component shows the database to be used and types of data sets including the

data set members etc. Naive Bayes Collection is the whole new component which is not directly

available in the system and includes the data mining techniques to predict the output of the

system.

Figure 4.12: Component Diagram



Deployment Diagram

For our system there is no hardware node needed to be attached hence only software deploy-

ment is viewed in above diagram. Here most of the nodes are the packages of the java and

remaining are the modules which are needed to be connected with one or more modules of the

project.

Figure 4.13: Deployment Diagram


Chapter 5

Technical Specification

5.1 Technology used

Java Platform

Java is a set of several computer software products and specifications from Sun Microsystems

(which has since merged with Oracle Corporation), that together provide a system for develop-

ing application software and deploying it in a cross-platform computing environment. Java is

used in a wide variety of computing platforms from embedded devices and mobile phones on

the low end, to enterprise servers and supercomputers on the high end. While less common,

Java applet are sometimes used to provide improved and secure functions while browsing the

World Wide Web on desktop computers.

Writing in the Java programming language is the primary way to produce code that will be

deployed as Java byte code. There are, however, byte code compilers available for other lan-

guages such as Ada, JavaScript, Python, and Ruby. Several new languages have been designed

to run natively on the Java Virtual Machine (JVM), such as Scala, Clojure and Groovy. Java syn-

tax borrows heavily from C and C++, but object-oriented features are modelled after Smalltalk

and Objective-C.[9] Java eliminates certain low-level constructs such as pointers and has a very

simple memory model where every object is allocated on the heap and all variables of object

types are references. Memory management is handled through integrated automatic garbage

collection performed by the JVM.

42

CHAPTER 5. TECHNICAL SPECIFICATION

Clustering

Clustering, in the context of databases, refers to the ability of several servers or instances

to connect to a single database. An instance is the collection of memory and processes that

interacts with a database, which is the set of physical files that actually store data.

Clustering takes different forms, depending on how the data is stored and allocated re-

sources. The first type is known as the shared-nothing architecture. In this clustering mode,

each node/server is fully independent, so there is no single point of contention. An example

of this would be when a company has multiple data centers for a single website. With many

servers across the globe, no single server is a master. Shared-nothing is also known as database

sharding.

Classification

Classification consists of predicting a certain outcome based on a given input. In order to

predict the outcome, the algorithm processes a training set containing a set of attributes and the

respective outcome, usually called goal or prediction attribute. The algorithm tries to discover

relationships between the attributes that would make it possible to predict the outcome. Next

the algorithm is given a data set not seen before, called prediction set, which contains the same

set of attributes, except for the prediction attribute not yet known. The algorithm analyses the

input and produces a prediction. The prediction accuracy defines how good the algorithm is.


CHAPTER 5. TECHNICAL SPECIFICATION


Chapter 6

Schedule, Estimate and Team Structure

6.1 Project Estimate

Project is not requiring any new hardware components so there is very less financial require-

ment for the project.

The project is required to be the cost estimation for the man power to be allocated and used

efficiently. We have followed COCOMO II model with moderate constraints to allocate the man

power for the project to be completed on or before the deadline.

Figure 6.1: Cocomo-II Embedded Project Model

45

CHAPTER 6. SCHEDULE, ESTIMATE AND TEAM STRUCTURE

6.2 Schedule

The project is to be completed on or before the deadlines provided hence a strong project

planning was required, the use of Gantt chart has increased the efficiency for keeping the project

on track. With the help of the Gantt chart we could track the project flow and corrective actions

were taken in order to follow the deadlines strictly.

Because of following the deadlines strictly, the project is completed before deadlines pro-

vided hence we could thoroughly test the project modules and most of the small defects found

were scanned and removed immediately.

6.3 Team Structure

The team required team for the development of this project is 4.5 persons as per the estimation

of the COCOMO-II model based on working hours and average lines of code to be carried out.

The team structure we have decided has four developing members and one guide member.

The guide member played major role by keeping the project flow as per schedule and solving

the major error and obstacles that affected the project development schedule.

The other four members are working as a team of developers and testers. Team lead was

given to member no.1. The role of member 1 was to work on the initial system designing and

developing the system. The member 2 had worked into resource gathering and literature survey

as well as developing the source codes of system. Team member 3 had been allocated the role of

tester whose job was to thoroughly test system with respect to test cases written by developers.



Last team member worked as scribe of the team and has done all the documentation during the

development of the system.


Chapter 7

Software Implementation

7.1 Introduction

We have conducted a survey for the current status and the patterns in the phishing techniques

used by the phishers. We found a trend of patterns that can be found in the phishing sites such

as the phishers uses some characters which are very identical with the alphabets in English

language as @ looks identical to character a etc.

This series of pattern and many more are traceable from the URL of the site. Some phishers

uses the source codes of the original sites and performs minimal changes into code this results

into the visual look very identical with the original sites. But this results into anchor tags of the

site to open another domain. This can also be traced and both of the textual and HTML features

can be used to find out whether a site is original or the phishing site.

We can use classification technique to predict the result if the site is original or the phishing

site.

7.2 Databases

As system is implemented in JAVA language and also with the help of the system procedure

call interface there is no overhead of database.

49

CHAPTER 7. SOFTWARE IMPLEMENTATION

All the records are stored into the single table; attributes of the table are as follows:

URL contains the name of URL of Record.

IP as URL Boolean value

Dots in URL Numerical Value

Slashes in URL Numerical Value

Suspicious Characters in URL Numerical Value

HTTPS / SSL / TSL Boolean Value

Foreign Anchors Numerical Value

Null Anchors Numerical Value

Serialized database is used hence there is no requirement to use any other database manage-

ment tool to store the database.

7.3 Important Modules

There are four major modules:

Feature Extraction

Apply Clustering

Apply Classifier

Detailed description of each module is given below.

Feature Extraction

Feature Extraction process is the initial stage in the project to create database or to find the

site is phishing site or not. It requires two methods to create a single dataset of features in the

feature extraction process. These processes are as follows:

1. String Parsing

2. DOM Tree Parsing



String Parsing is applied on the string input URL given by the user itself. In java string

searching is made easy by providing the in built package for string operations. We just need to

give inputs as what to search and input string.

DOM Tree Parsing is much harder job than the string searching as we need to create our

source code string and parse it. In java no readily available packages are present to parse the

DOM Tree. Hence we need to create a vector and insert each tag found in the HTML source

code of the URL single tag at time and again with the help of the DFS parsing we are adding

these nodes in the list to display.

Figure 7.1: Sample DOM Tree



Figure 7.2: DOM Tree constructed in PROJECT

Apply Clustering

Clustering, in the context of databases, refers to the ability of several servers or instances

to connect to a single database. An instance is the collection of memory and processes that

interacts with a database, which is the set of physical files that actually store data.

Clustering takes different forms, depending on how the data is stored and allocated re-

sources. The first type is known as the shared-nothing architecture. In this clustering mode,

each node/server is fully independent, so there is no single point of contention. An example

of this would be when a company has multiple data centers for a single website. With many

servers across the globe, no single server is a master. Shared-nothing is also known as database

sharding.

Contrast this with shared-disk architecture, in which all data is stored centrally and then

accessed via instances stored on different servers or nodes.



The distinction between the two types has become blurred recently with the introduction

of grid computing or distributed caching. In this setup, data is still centrally managed but

controlled by a powerful virtual server that is comprised of many servers that work together as

one.

In this case the database clustering is applied to reduce the complexities of the stored values.

Simply two clusters are created for each discrete feature of the dataset. Then the lowest value

and highest values of that respective feature set are taken and considered as the initial centroid

to start the algorithm.

K-Means Clustering is used because it is unsupervised algorithm and provides faster results

as compared to the other clustering algorithms.

Following is the example of the execution of K-Means Clustering for this project.

We take a dataset and find the minimal and maximal value present in each of the feature used

for K-Means Clustering.

CLUSTER DOTS SLASH S.CHAR N.ANCHR F.ANCHR

A 2 0 1 10 11

B 2 0 3 1 2

C 3 0 2 0 0

D 6 9 8 0 0

E 1 1 0 1 9

F 2 3 3 0 1

G 10 2 2 0 0




H 8 11 3 0 3

I 2 4 6 0 0

J 6 8 1 7 1

Table 7.1: Sample Dataset for K-Means Clustering

Then we are calculating the distance of new dataset item one feature at time by comparing

current value with the high centroid and low centroid of that feature. The value lies nearer to

any cluster centroid is labeled with that centroid cluster number and the centroid is recalculated

by taking mean of all values present in that cluster.


Less=0 1 0 0 0 0

More=1 10 11 8 10 11

Table 7.2: Initial Cluster Centroid values

This algorithm is unsupervised algorithm which means it should terminate itself after some

condition is satisfied. In this case the algorithm comes to halt when there is no movement in

centroid is observed.

CLUSTER DOTS LBL SLASH LBL S.CH LBL N.AN LBL F.AN LBL

A 2 0 0 0 1 0 10 1 11 1

B 2 0 0 0 3 0 1 0 2 0

C 3 0 0 0 2 0 0 0 0 0

D 6 1 9 1 8 1 0 0 0 0

E 1 0 1 1 0 0 1 0 9 1



CLUSTER DOTS LBL SLASH LBL S.CH LBL N.AN LBL F.AN LBL

F 2 0 3 0 3 0 0 0 1 0

G 10 1 2 0 2 0 0 0 0 0

H 8 1 11 1 3 0 0 0 3 0

I 2 0 6 0 6 1 0 0 0 0

J 6 1 8 1 1 0 7 1 1 0

Table 7.3: Dataset after clustering

Then all the calculated centroid is declared as the final centroid for that feature in the database.


Less=0 2 1.5 1.85 0.25 0.825

More=1 7.5 7.25 7 8.5 10

Table 7.4: Final Cluster Centroid values

Apply Classifier

Classification consists of predicting a certain outcome based on a given input. In order to

predict the outcome, the algorithm processes a training set containing a set of attributes and the

respective outcome, usually called goal or prediction attribute. The algorithm tries to discover

relationships between the attributes that would make it possible to predict the outcome. Next

the algorithm is given a data set not seen before, called prediction set, which contains the same

set of attributes, except for the prediction attribute not yet known. The algorithm analyses the

input and produces a prediction. The prediction accuracy defines how good the algorithm is.

In simple terms, a naive Bayes classifier assumes that the value of a particular feature is

unrelated to the presence or absence of any other feature, given the class variable. For example,

a fruit may be considered to be an apple if it is red, round, and about 3 in diameter. A naive

Bayes classifier considers each of these features to contribute independently to the probability

that this fruit is an apple, regardless of the presence or absence of the other features.



For some types of probability models, naive Bayes classifiers can be trained very efficiently

in a supervised learning setting. In many practical applications, parameter estimation for naive

Bayes models uses the method of maximum likelihood; in other words, one can work with the

naive Bayes model without accepting Bayesian probability or using any Bayesian methods.

Abstractly, the probability model for a classifier is a conditional model

p(C|F1, . . . , Fn)

Over a dependent class variable with a small number of outcomes or classes, conditional on

several feature variables through . The problem is that if the number of features is large or when

a feature can take on a large number of values, then basing such a model on probability tables

is infeasible. We therefore reformulate the model to make it more tractable.

Using Bayes theorem, this can be written

p(C|F1, . . . , Fn) = p(C)p(F1,...,Fn|C)p(F1...,Fn)

In plain English, using Bayesian Probability terminology, the above equation can be written

as:

posterior = priorlikelihoodevidence

Or more simplified formula is as given below

p(fsi|Cj) = nc+mpn+m

where,

n = the number training examples for which = j .

nc = the number of examples for which = j and a = ai.

p = a priori estimate for P (ai|j).

m = the equivalent sample size.

Suppose we have taken a training dataset of 10 websites on which K-Means Clustering is

already applied. This training dataset is given as input for the Naive Bayes Classifier.



URL IP DTS LB SLS LB SCH LB NAC LB FAC LB SSL PSH

A 0 2 0 0 0 1 0 10 1 11 1 1 1

B 0 2 0 0 0 3 0 1 0 2 0 0 0

C 1 3 0 0 0 2 0 0 0 0 0 0 0

D 0 6 1 9 1 8 1 0 0 0 0 1 1

E 0 1 0 1 0 0 0 1 0 9 1 0 0

F 0 2 0 3 0 3 0 0 0 1 0 1 0

G 0 10 1 2 0 2 0 0 0 0 0 1 1

H 0 8 1 11 1 3 0 0 0 3 0 0 1

I 0 2 0 4 0 6 1 0 0 0 0 1 0

J 1 6 1 8 1 1 1 7 1 1 0 0 1

Table 7.5: Sample Training data set of Classifier

We have taken a new site whose results of valid or invalid phish are not known to us as

follows:

URL IP DTS LB SLS LB SCH LB NAC LB FAC LB SSL PSH

X 0 4 0 1 0 11 1 0 0 5 1 0 ?

Table 7.6: New Unknown site

By using formula p(fsi|Cj) = nc+mpn+m we can evaluate probability of each feature contributingto the final probability.

We can calculate final probability of data set of legit site as follows:



X N NC M P PROB

IP 8 4 2 0.5 0.5

DOTS 6 5 2 0.5 0.75

SLASHES 7 4 2 0.5 0.56

S.CHARS 2 1 2 0.5 0.5

N.ANCHR 8 5 2 0.5 0.6

F.ANCHR 8 4 2 0.5 0.5

SSL 5 3 2 0.5 0.57143

0.017857

Table 7.7: Probability for feature set to be Original

X N NC M P PROB

IP 8 4 2 0.5 0.5

DOTS 6 1 2 0.5 0.25

SLASHES 7 3 2 0.5 0.44

S.CHARS 2 1 2 0.5 0.5

N.ANCHR 8 3 2 0.5 0.4

F.ANCHR 8 4 2 0.5 0.5

SSL 5 2 2 0.5 0.4285

0.00238

Table 7.8: Probability for feature set to be Phish

After calculating the final probabilities of feature set to be valid or invalid phish we need

to compare the results, result with maximum probabilities are declared as the final prediction

results.

Hence we can say that current record is not a phishing site as 0.017857 > 0.002381.

7.4 Business Logic

After analyzing the project approach and studying the research papers for the given system

we have decided to follow the Waterfall Model for the Software Development Life Cycle in this



project development process. This approach consists of the following steps that are followed in

order to achieve the goal:

1. Requirement Specification resulting into the project requirement documentation.

2. Design resulting into the Software Architecture.

3. Construction resulting into actual writing codes and developing software.

4. Integration resulting into combining all the modules of the project and finalizing the

development phase.

5. Testing and Debugging gives the defect free software.

6. Installing resulting into the providing the software to end user.

Thus the waterfall model maintains that one should move to next phase only when previous

phase are verified and reviewed.


Chapter 8

Software Testing

8.1 Introduction

In data mining, data scientists use algorithms to identify previously unrecognized patterns and

trends hidden within vast amounts of structured and unstructured information. These patterns

are used to create predictive models that try to forecast future behaviour.

These models have many practical business applications they help banks decide which cus-

tomers to approve for loans, and marketers use them to determine which leads to target with

campaigns.

But extracting real meaning from data can be challenging. Bad data, flawed processes and the

misinterpretation of results can yield false positives and negatives, which can lead to inaccurate

conclusions and ill-advised business decisions.

Thorough testing is needed to be done before handover of the software to the end user as

the user may rely on the predictions made by the software to take some major decisions for his

business requirements.

61

CHAPTER 8. SOFTWARE TESTING

8.2 Test Cases

Figure 8.1: Test Cases for Project Main Modules

8.3 Snapshot of GUI

Figure 8.2: Main Form



Figure 8.3: Manual Entry Form

Figure 8.4: Manual Entry Form Empty



Figure 8.5: Prediction Model

Figure 8.6: Load Form


Chapter 9

Results

9.1 Accuracy of Result

Accuracy testing is done to measure up to what level the software may be trusted in order to

make decisions based on the predictions of the project.

We have taken a total of 100 sites to build the training model of the database which is used

to predict the result using classifier constructed in this project. The sites are taken as the 50

- 50 division, 50 sites are taken as known phishing sites from the http://www.phishtank.com/

site which stores the phishing site reported by the users and declares it database to be used by

the other software companies. Another 50 sites are the known legit sites which are taken from

official page links.

Then we have taken 20 sites for which results were not known to the system as the input

of the classifier. The sites were classified with the 85% of accuracy. Another 20 sites were

introduced to the software as input and they were also classifier with correct output of 83.33%

accuracy.

66

CHAPTER 9. RESULTS

Figure 9.1: Accuracy Testing graph

9.2 Project Results

Figure 9.2: New site feature extraction in progress


CHAPTER 9. RESULTS

Figure 9.3: Prediction Results of Site


CHAPTER 9. RESULTS


Chapter 10

Deployment and Maintenance

10.1 Installation

Java Standard Edition JDK 7 Installation

The JDK 7.2 can be downloaded from this website:

http://www.oracle.com/technetwork/java/javase/downloads/index.html

Click the Download JDK button in the Java Platform Standard Edition section. Make sure

you download the JDK and not the JRE.

Figure 10.1: JDK Step 1

70

CHAPTER 10. DEPLOYMENT AND MAINTENANCE

Then, select the installation file for your platform: If your system is 32-bit, select the jdk-

7u2windows-i586.exe. If your system is 64-bit, select the jdk-7u2-windows-x64.exe. You can

find out what type of system you have by going to Start, Control Panel, System, and look at the

information listed under System type.


Once you have obtained the installation file, double-click it to begin the installation process.

This process will lead you through the following series of windows:

Setup Click Next.

Custom Setup You do not need to make any changes to the default setting. Just verify the

installation directory,

Click Next.

Progress Wait next window to open.

Destination Folder You do not need to make any changes to the default setting. Just verify the

installation directory,



Click Next.

Progress Wait process to end.

Complete Click Finish to complete. A browser window may open that asks you to register the

software. You may do so, or just close it without registration.


The documentation can be downloaded from the same website as the JDK:

http://www.oracle.com/technetwork/java/javase/downloads/index.html

This time, scroll down, and click the Download button in the Java SE 7 Documentation

section of the Additional Resources box.


Chapter 11

Appendix A: Glossary

NB: Naive Bayes Classifier, a mathematical from the Bayesian Approach used to produce the

results based on existing evidences.

CSV: Comma Separated Values, a terminology used in databases referring to the string which

includes all the table column entries from the current database.

DOM Tree: Document Object Model, is an internal representation used by browsers to repre-

sent a web page.

IDS: Intrusion Detection System, a system which will work as background process for detec-

tion of web pages in real time.

JDK: Java Development Kit is the set of standard libraries provided by JAVA which are re-

quired to develop the basic block of java project.

URL: Uniform Resource Locator, the name of website by which it is known in Computer

Networks.

SSL: Secure Socket Layer is cryptographic protocol that is designed to provide communication

security over the Internet.

74

Chapter 12

Appendix B: Semester I Assignments

Assignment No. 1

Modules of the project development: Mathematical model for project

This hybrid model is adapting the divide and conquer strategy as we are dividing the problem

into smaller two problems and solving them individually to solve the given problem. Here Clus-

tering and Bayesian Classifier Approach are two different methods applied on separate parts to

solve the problem by dividing into two small problems.

S {DS,FS, FE,L, URL,K MEANSpred, NBmodel, NBpred}where,

DS = Data Set for given Model.

FE = Feature Extraction Procedure to produce FS.

K MEANSpred = K-Means Clustering Prediction.NBmodel = Naive Bayes Classifier Training Model.

NBpred = Naive Bayes Classifier Prediction.

FS FE(URL)where,

FS = Feature set for the given Model.

FE = Feature Extraction Procedure to produce FS.

URL = URL input to system.

75

CHAPTER 12. APPENDIX B: SEMESTER I ASSIGNMENTS

fs1, fs2, fs3....fsn DSL1, L2, L3 L

fsi FE(URL)where,

fsi = Current Feature Set.

Li K MEANSpred(DS,L)

KMEANSpred(fs1 ,fs2 ,fs3 ....fsn ) = argminsLSi=1

fsjLi ||fsj i ||2

where,

i =the mean of pointsSi.

NBmodel NBpred(DS,L)fsi FE(URL)Li NBpred(NBmodel, fsi)

NBpred(C|fs1, fs2, fs3....fsn) = p(C)p(fs1,fs2,fs3....fsn|C)p(fs1,fs2,fs3....fsn)

where,

C =dependant class variable.

p =probability.

The K-Means Clustering is NP-Hard problem.

The Naive Bayes Classifier is P-Complete problem and we can solve the complete polyno-

mial for the given problem for naive bayes classifier.



Assignment No. 2

Algorithmic strategies used in project: Algorithms K-Means Clustering

and Naive Bayes Classifier

K-Means Clustering Simulation

Following given is the sample Data Set and its evaluation based on K-Means Clustering.

For this example we have used K = 3 as the cluster size.

WebSite IP DOTS SLASHES SUS.CHAR REMARK

A 0 2 2 1

B 0 1 1 2

C 0 2 3 4

D 0 5 4 9

E 1 3 5 10

F 1 4 7 20

G 1 5 9 4

H 1 8 13 15

I 1 9 9 16

For the above given Data Sets, applying K-Means for K = 3 we are forming 3 clusters with

following initial centroid.

Cluster Centroid1 Centroid2 Centroid3 Centroid4

Cluster1 0 1 1 5

Cluster2 1 5 5 10

Cluster3 2 10 10 20




A 0 2 2 1 1

B 0 1 1 2 1

C 0 2 3 4 1

D 0 5 4 9 1

E 1 3 5 10 2

F 1 4 7 20 2

G 1 5 9 4 2

H 1 8 13 15 3

I 1 9 9 16 3

After 1st Iteration.


Cluster1 0 2.5 2 4

Cluster2 1 4 7 11.33

Cluster3 2 8.5 11 15.5


A 0 2 2 1 1

B 0 1 1 2 1

C 0 2 3 4 1

D 0 5 4 9 2

E 1 3 5 10 2

F 1 4 7 20 2

G 1 5 9 4 2

H 1 8 13 15 3

I 1 9 9 16 3

Here we can see that Feature Set D first included into 1st cluster and after er-arranging cen-

troid Feature Set D is included into 3rd cluster.

After few iterations centroid becomes stable as follow:




Cluster1 0 1.67 2 2.34

Cluster2 0.75 4.25 6.25 10.75

Cluster3 2 8.5 11 15.5

Naive Bayes Classifier Simulation

To evaluate the results using Naive Bayes Classifier we can use following formula:

P(ai|j) = nc+mpn+m

WebSite IP DOTS SLASHES SUS.CHAR SSL FrA NlA CLUSTER

A 0 2 2 1 0 1 0 1

B 0 1 1 2 0 2 1 1

C 0 2 3 4 0 1 0 1

D 0 5 4 9 1 5 2 1

E 1 3 5 10 7 0 3

F 1 4 7 20 0 1 5 1

G 1 5 9 4 1 7 4 3

H 1 8 13 15 1 5 7 3

I 1 9 9 16 1 9 8 3

The Feature Set for which cluster is to be decided :

WebSite IP DOTS SLASHES SUS.CHAR SSL FrA NlA CLUSTER

J 1 5 9 10 1 7 2

Here we want to classify our new Feature Set which is not listed above and is unique for the

given model. We need to calculate probabilities:

P(1|1), P(5|1), P(9|1), P(10|1), P(1|1), P(7|1), P(2|1)

P(1|3), P(5|3), P(9|3), P(10|3), P(1|3), P(7|3), P(2|3)

P(ai|j) = nc+mpn+m



For Calculation P(J|3)

n nc m p P (ai|j)IP=1 5 4 7 0.5 0.625

DOTS=5 2 1 7 0.5 0.500

SLASHES=9 2 2 7 0.5 0.611

SUS.CHAR=10 1 1 7 0.5 0.563

SSL=1 5 4 7 0.5 0.625

FrA=7 2 2 7 0.5 0.611

NlA=1 1 0 7 0.5 0.438

For Calculation P(J|1)

n nc m p P (ai|j)IP=1 5 1 7 0.5 0.375

DOTS=5 2 1 7 0.5 0.500

SLASHES=9 2 0 7 0.5 0.389

SUS.CHAR=10 1 0 7 0.5 0.438

SSL=1 5 1 7 0.5 0.375

FrA=7 2 0 7 0.5 0.389

NlA=1 1 1 7 0.5 0.563

P(1|1), P(5|1), P(9|1), P(10|1), P(1|1), P(7|1), P(2|1)

= 0.375 0.500 0.389 0.438 0.375 0.389 0.563 = 0.002617

P(1|3), P(5|3), P(9|3), P(10|3), P(1|3), P(7|3), P(2|3)

= 0.625 0.500 0.611 0.563 0.625 0.611 0.438 = 0.017950

Here 0.017950 > 0.002617, hence our Feature Set gets classified as VALID PHISH.



Assignment No. 3

Study of various options available to implement the project modules and

why then given options are chosen?

Our project aims towards detecting a web page is a Valid Phish, Invalid Phish. We use

K-means algorithm to cluster data set. For this purpose we used machine learning technique

Naive Bayes Classifier to identify the most important features that differentiate Phishing Site

from Legitimate Site.

Why use Data Mining?

Two major reasons to use data mining :

1. The amount of data is very large and useful information

Date post:	13-Sep-2015
Category:	Documents
Upload:	bhushandhamdhere01
View:	213 times
Download:	0 times

main.pdf

Documents