Post on 07-Aug-2020
transcript
INDIAN INSTITUTE OF TECHNOLOGY, BOMBAY
Ekalavya Summer Internship, 2018
Event Logging System for Collaborative
Communities
Divyansh Sharma Kartik Verma Rahul Raj Vishrut Jetly
NIT-Hamirpur NIT-Hamirpur NIT-Patna DA-IICT
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 1 / 28
Outline
1 Introduction
2 Motivation
3 Technologies Used
4 Work Flow
5 Capturing theEvents
6 Creating the Log
7 Storing Logs inElasticsearch
8 Event Log RESTAPI
9 User Analytics
10 Dockerization andSource Code
11 Selenium Testing
12 Problems Solved
13 Future Scope
14 References
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 2 / 28
Introduction
Introduction
What is Event Logging ?
Logging or storing the activities done by user of the system e.g.login, view a community, view article, edit article etc.
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 3 / 28
Motivation
Motivation
The event logs will be used to create a view history for everyuser which will help in training the Recommendation System.
Providing analytics to the users like
Top communities and articlesTrending communities and articles
which will assist in users’decision making.
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 4 / 28
Technologies Used
Technologies Used
Django Middleware
ELK
C3.js
Django REST framework
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 5 / 28
Technologies Used
Technologies Used
Django Middleware
Every activity that gen-erates a http request tothe server first passesthrough these middle-wares which attach andextract some informa-tion from the requeste.g. session id, authen-tication tokens etc.
C3.js
C3.js is used to im-plement high quality,interactive charts forvisualisation. Chartsbuilt using this librarycan be rendered in realtime.
Django RESTFramework
Django REST frame-work is a powerfuland flexible toolkit forbuilding Web APIs. Itsa way to expose partsof the application tothird-parties
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 6 / 28
Technologies Used
Technologies Used
ELK
ELK is acronym for three open source projects
Elasticsearch
Logstash
Kibana
Logstash [3] receives the data from the files or through the httprequests, applies filters, and then Elasticsearch [1] stores the logs,which can be later retrieved using Elasticsearch API.Kibana [2] is used for visualisations and analytics.
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 7 / 28
Work Flow
Work Flow Diagram
Figure: Work Flow Diagram
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 8 / 28
Capturing the Events
Capturing the Events
Figure: Django Middleware
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 9 / 28
Capturing the Events
Capturing the Events
The request passes through all the middlewares and finallymoves through the eventlog middleware.
The eventlog middleware [4] captures all the information fromthe request.
Benefit of Middleware
Middleware keeps the Eventlog system as a completely separatemodule or app.
It can be reused in other django projects.
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 10 / 28
Creating the Log
Creating the Log
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 11 / 28
Creating the Log
Important Points
The size of the bucket holding logs is not fixed.
The function which processes the dictionary present in bucketruns in a separate daemon thread.
Logs can be stored either in file or in Elasticsearch but here weare showing it only with Elasticsearch because storing logs in afile are not useful.
Logs are passed to ELK in JSON format.
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 12 / 28
Creating the Log
Log Structure
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 13 / 28
Storing Logs in Elasticsearch
Storing Logs in Elasticsearch
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 14 / 28
Storing Logs in Elasticsearch
Important Points
Elasticsearch, Logstash and Kibana runs on separate ports insidea docker.
The name of the index which stores the logs is logs but can bechanged in settings.
Kibana is available only for the Super Admin users.
Elasticsearch uses DSL query syntax built over Apache luceneQuery Syntax parser for searching.
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 15 / 28
Event Log REST API
Event Log REST API
Event log System provides a feature rich REST based API whichis a wrapper over the Elasticsearch search API.
This API is used by Recommendation System and for UserAnalytics
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 16 / 28
Event Log REST API
Features of REST API
The eventlog REST API is a feature rich API. The notable ones are:
Pagination
Sorting
Custom Fields
Filters based searching
Time Range Based searching
Aggregations or group by
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 17 / 28
User Analytics
User Analytics
User Analytics provides an overview of the activities of the user.
Some of the Analytics added are:
Trending articles in a community or group.
Most viewed articles in community.
User’s most viewed articles.
User’s own articles in various states.
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 18 / 28
User Analytics
Some Snapshots
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 19 / 28
User Analytics
Some Snapshots
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 20 / 28
User Analytics
Some Snapshots
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 21 / 28
Dockerization and Source Code
Dockerization and Source Code
Components Dockerized:
Elasticsearch
Logstash
Kibana
Source Code Link:https://github.com/fresearchgroup/Collaboration-System/tree/eventlogs
ELK Docker Link:https://github.com/fresearchgroup/Collaboration-System-Event-Logs
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 22 / 28
Selenium Testing
Selenium Testing
Verified all events are logged.
Verified the log structure.
Tested the Event Log REST API.
Selenium Test Code Link:https://github.com/fresearchgroup/Collaboration-System-Selenium/tree/eventlogs
This Testing was done by the Notification System Team,Fundamental Research Group
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 23 / 28
Problems Solved
Problems Solved
Added the Event Logging System without affecting the existingsystem in any way using middleware.
Added a System which can be easily modified to add new eventsfor capture.
Provided a feature rich API for use by recommendation systemand for user analytics.
Added User Analytics to the current System.
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 24 / 28
Future Scope
Future Scope
Adding browser events i.e. events that are generated in frontend, for instance
Window resize eventsScroll eventsThese events may provide some useful information about humanbehaviour
Adding Video related events
Adding more User Analytics
Add real time analytics whenever logs are updated
Storing logs in cloud storage.
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 25 / 28
References
References I
[1] elastic. Elasticsearch Reference and Docs. url:https://www.elastic.co/guide/en/elasticsearch/
reference/6.3/index.html (visited on 06/02/2018).
[2] elastic. Kibana Reference. url: https://www.elastic.co/guide/en/kibana/current/getting-
started.html (visited on 06/04/2018).
[3] elastic. Logstash Reference. url: https://www.elastic.co/guide/en/logstash/6.2/index.html
(visited on 06/03/2018).
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 26 / 28
References
References II
[4] Vitor Freitas. How to Create a Custom Django Middleware.url: https://simpleisbetterthancomplex.com/tutorial/2016/07/18/how-to-create-a-custom-django-
middleware.html (visited on 05/20/2018).
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 27 / 28
References
Thank You
Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 28 / 28