Web Usage MiningPart-1
Web Usage MiningIt’s main goal is to:Discover usage patterns from web data
in order to understand and better serve the needs of web based applications
Web Usage MiningWeb usage mining consists of three
phases Preprocessing Pattern discovery Pattern analysis
Generated by users’ interaction with the Web, data sources include:
web-server access logs proxy-server logs browser logs user profiles registration data user sessions and transactions cookies user queries bookmark data mouse clicks and scrolls
Web-Usage Mining
Web-Log ProcessingA server log:
set of files consisting of the details of an activity performed by a server
files are automatically created and maintained by the server
The World Wide Web Consortium (W3C) has specified a standard format for web-server log files
There are other proprietary formats for web-server logs.
Most web logs contain: IP address of the client making the request date and time of the request URL of the requested page number of bytes sent to serve the request user agent (such as a web browser or web crawler) referrer (the URL that triggered the request)
Logs can all be stored in one file A better alternative is to separate:
access log error log referrer log
Web-Log Processing
Format of Web Logs
Common log format(http://www. W3.org/Daemon/User/Config/Logging.html#common-logfile-format)
Examples of Common Log Format140.14.6.11 - pawan [06/Sep/2001:10:46:07 -0300] "GET /s.htm HTTP/1.0"
200 2267
140.14.7.18 - raj [06/Sep/2001:11:23:53 -0300] "POST /s.cgi HTTP/1.0" 200 499
GET request that retrieves a file s.htm POST request sends data to a program s.cgi Fields:
client machine’s IP address (140.14.6.11) RFC 1413 identity of the client is missing (-) Date and time Request Error code Number of bytes transferred
Examples of Common Log Format
An example of a log file in extended format
Format of Web Logs#Version: version of the extended log file format
used#Fields: fields recorded in the log#Software: software that generated the log#Start-Date: date and time at which the log was
started#End-Date: date and time at which the log was
finished#Date: date and time at which the entry was
added#Remark: Comments that are ignored by
analysis tools
The directives #Version and #Fields are mandatory and must appear before all the entries
Each field in the #Fields directive can be specified in one of the following ways: an identifier; e.g., time an identifier with a prefix separated by a
hyphen; e.g., cs-method a prefix following a header in parentheses;
e.g., sc(Content-type)
Format of Web Logs
No prefixes for date, time, time-taken, bytes, cached
Prefixes for ip, dns, status, comment, method, uri, uri-stem, uri-query, host
Prefixes can be:cs client to serversc server to clientsr server to remote server (this prefix is used by proxies)rs remote server to server (this prefix is used by proxies)x application-specific identifier
Format of Web Logs
Analyzing Web logs
Analyzing Web Logs
General Summary from Analog
Analyzing Web Logs
Monthly report from Analog
Analyzing Web Logs
Daily summary from Analog
Hourly summary from Analog
Analyzing Web Logs
Organization report from Analog
Organization report from Analog
Search-word report from Analog
Operation-system report from Analog
Status-code report from Analog
File size report from Analog
File type report from Analog
Directory report from Analog
FRequest report from Analog
Analysis of Clickstream: Studying Navigation Paths
Clickstream using Pathalizer with seven link specification
Analysis of Clickstream: Studying
Navigation Paths
Analysis of Clickstream: Studying
Navigation Paths
Clickstream using Pathalizer with twenty link specification
Visualizing Individual User Sessions
A brief on-campus session identified by StatViz that browses the bulletin board
Visualizing Individual User Sessions
A brief off-campus session identified by StatViz with three distinct activities
Visualizing Individual User Sessions
A long on-campus session identified by StatViz with multiple activities
Caution in Interpreting Web-Access Logs Requests may not always reach the server as they
may be served from a proxy server’s cache You do not really know:
Identity of readers Number of visitors Number of visits User’s navigation path through the site Entry point and referral How users left the site or where they went next How long people spent reading each page How long people spent on the site
Turner (2004)I’ve presented a somewhat negative view here, emphasizing what you can’t find out. Web statistics are still informative: it’s just important not to slip from “this page has received 30,000 requests” to “30,000 people have read this page.” In some sense these problems are not really new to the web—they are present just as much in print media too. For example, you only know how many magazines you’ve sold, not how many people have read them. In print media we have learnt to live with these issues, using the data which are available, and it would be better if we did on the Web too, rather than making up spurious numbers.