Date post: | 28-Jan-2018 |
Category: |
Technology |
Upload: | sanil-mhatre |
View: | 65 times |
Download: | 3 times |
Qualitative Text AnalysisUsing Azure and Power BI
Sanil Mhatre@sqlsuperguru
https://sqlwithsanil.wordpress.com/https://github.com/SQLSuperGuru
About me
• Senior Developer – Data Analytics @ WWT
• DBA, Developer, Architect, Amateur Data Scientist
• Oracle, SQL Server, My SQL, DB2, Mongo, Vertica
• Cloud, Big Data & Hadoop, Data Visualization
• Speaking, Blogging & volunteering
Agenda
• From Qualitative to Quantitative
• Employee Survey for Acme Corp.
• AI + Cognitive Services – Text Analytics APIs
• Power BI
• ETL with Power BI & Azure
• Visual Analytics
• Dashboards & Power BI Service
• Gaining useful Insights
• Q&A
Qualitative vs Quantitative Data & Analysis• Qualitative
• Data consists of Words & Narratives• Analysis includes Key Words, Themes & Concepts• Understand perceptions, look for themes• Examples : Transcripts, Comments, Free form text responses to surveys
• Quantitative• Data is numeric• Summarize/Aggregate, slice & dice• Use statistical techniques to draw conclusions• Examples : Anything that can be counted and measured
• Number of responses to a survey, Number of participants in a study• Shoe size, temperature
Combining Techniques
• Perform Qualitative Analysis on Text
• Translate the text to some Numeric measure (sentiment score)
• Perform Quantitative Analysis on the score
• Descriptive Statistics• Range, Min, Max, Frequency
• Measures of Central Tendency• Mean – The Average
• Median – The middle score after values are numerically ordered
• Mode – The most frequently occurring value
• More Insights by combining Qualitative & Quantitative Analysis
Use Case
• Employee Survey at Acme Corp.
• 2 Questions• 1. Is there anything unique or unusual about this company that makes it a
great place to work?
• 2. If you could change one thing about this company to make it a better place to work, what would it be?
• Free form Text• 252 replies
• 995 lines
• 12,368 words
• Lets take a look at the Excel document
AI + Cognitive Services
• Microsoft AI + Cognitive Services • A collection of machine learning & AI algorithms in the Azure cloud
• Artificial Intelligence• Machine Learning services
• Azure bot service
• Azure Batch AI
• Cognitive Services• Vision
• Speech
• Language (Text Analytics is a part of this group)
• Knowledge
• Search
Text Analytics API
• Cloud based service - natural language processing over raw text
• 3 types of analysis
https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/overview
Take it for a spin
• Try out the Text Analytics API - https://azure.microsoft.com/en-
us/services/cognitive-services/text-analytics/
• Create Cognitive Services API Account in Azure
• Demo
• Location – Not available everywhere
• Pricing – Free Tier
• Endpoints
• Access keys
Power BI Desktop
• Desktop Product for Data Analytics & Visualization
• Connects to variety of Data sources
• ETL – Prepare and Model Data
• Advanced Analytics
• Interactive reports and visualizations
• Share
• Quick tour
Power BI Desktop in action
• Demo
• Import the Excel file
• Filter the responses to remove NULL
• See the Relationships Pane
• Create Custom Functions
• Invoke the Custom Functions to add columns for KeyPhrases & SentimentScore
• Adjust Data Source Permission settings if needed
• Round the score to 2 decimals
• Create Bins for Sentiment Score
Visualizations in Power BI Desktop
• Demo
• Summary Page
• Text box
• Cards for # of Questions & # Responses
• Word Cloud (Qualitative)
• Details Page (Descriptive statistics)
• Slicer for filtering by Question
• Score Gauge
• Box & Whiskers Plot – Mean, Median, Max, Min
• Histogram – Frequency
• Responses Page (Curated Raw text)
• Slice by question & Response Score Bin
• See all the associated response texts
Power BI Service
• Share this report with .pbxi file
• Publish to Power BI service
• Dashboards, Reports, workbooks & datasets
• Organized into workspaces
• Dashboard
• A canvas that uses visualizations to tell a story
• Tiles and Pages can be pinned from reports to one or more dashboards
• More information about Dashboards : https://docs.microsoft.com/en-us/power-bi/service-dashboards
• Demo – Build the Dashboard
• Insights (The bulb on your Data sets)
• Natural Language Queries (Ask) for Dashboards
Gaining Useful Insights
• How many employees responded
• Word Cloud – Trends/what are people talking about ?
• Descriptive statistics on Sentiment Scores• What is the overall perception/feeling• Range, Mean, Median, Outliers• Distribution – Responses seem polarizing• How do these numbers compare between the 2 questions
• See the raw response text, sliced by• Question• Response Score Bin
• Share, Collaborate
Tableau vs Power BI
• Power BI• Easier ETL and invoking APIs
• Easier to visualizations
• Tiles are easier to move and size to page/canvas grid
• Slices and Page/Report level filters can create some confusion
• Tableau• Creating some visualizations like Gauge or WordCloud requires several
manual steps
• Filters are easier to work with at both worksheet and Dashboard level
• Moving & sizing various worksheets on a Dashboard manually can get tedious
Other Analytics Tools
• Vertica • Distributed columnar Database & Big Data/Advanced Analytics Platform• Pulse Package for Text Analysis functions• Easy to use SQL Interface• Expensive
• R• Popular for data analytics• Extensive libraries and capabilities• Easy to use , powerful , creative• Requires familiarity with R programming language• Free Desktop• Limited by memory of the machine• Demo
Resources• Qualitative or Quantitative Data - https://cyfar.org/qualitative-or-quantitative-data
• Quantitative Data Analysis - https://cyfar.org/analysis-quantitative-data
• Azure Cognitive Services Text Analytics - https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/overview
• Power BI Desktop - https://powerbi.microsoft.com/en-us/desktop/
• Text Analytics with Power BI - https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/tutorials/tutorial-power-bi-key-phrases
• Power BI Service - https://docs.microsoft.com/en-us/power-bi/service-basic-concepts
• Dashboards in Power BI Service - https://docs.microsoft.com/en-us/power-bi/service-dashboards
• Tableau - https://www.tableau.com/
• Vertica Pulse -
https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/Pulse/PulseCookbook.htm%3FTocPath%3DVertica%2520Pulse%7CPulse%2520Cookbook%7C_____0
• R Tidytext package - https://cran.r-project.org/web/packages/tidytext/vignettes/tidytext.html
• R syuzhet package - https://cran.r-project.org/web/packages/syuzhet/vignettes/syuzhet-vignette.html
• My GitHub for the R code use in Demos - https://github.com/SQLSuperGuru
Thank youQ&A
Sanil Mhatre@sqlsuperguru
https://sqlwithsanil.wordpress.com/https://github.com/SQLSuperGuru