An Approach for Ex-Post-Facto Analysis of KnowledgeGraph-Driven Chatbots – the DBpedia Chatbot
Rricha Jalota, Priyansh Trivedi, Gaurav Maheshwari, Axel-Cyrille Ngonga Ngomo, RicardoUsbeck
November 20, 2019
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 1 / 17
Introduction
Figure: U.S. Chatbot Market by Vertical, 2014 - 2025 (USD Million) 1
1Source: https://www.grandviewresearch.com/industry-analysis/chatbot-marketJalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 2 / 17
IntroductionKnowledge Graphs and Knowledge Graph-Driven Systems
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 3 / 17
BackgroundKnowledge Graph-Driven Chatbot: The DBpedia Chatbot
Deployed in August 2017
Purpose2
- Answer factual questions- Answer questions related to DBpedia- Expose the research work being done in DBpedia as product features- Casual conversation/banter
Hybrid Chatbot - domain-specific information (DBpedia-centric FAQs) +domain-agnostic factual questions (using DBpedia KG)
2Source: https://wiki.dbpedia.org/blog/meet-dbpedia-chatbotJalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 4 / 17
BackgroundKnowledge Graph-Driven Chatbot: The DBpedia Chatbot
Deployed in August 2017
Purpose2
- Answer factual questions- Answer questions related to DBpedia- Expose the research work being done in DBpedia as product features- Casual conversation/banter
Hybrid Chatbot - domain-specific information (DBpedia-centric FAQs) +domain-agnostic factual questions (using DBpedia KG)
2Source: https://wiki.dbpedia.org/blog/meet-dbpedia-chatbotJalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 4 / 17
BackgroundCase Study: The DBpedia Chatbot
Total: 9084 users, 90,800 interactions
Table: Feedback Statistics
Feedback-asked 28953Feedback-received 7561Negative-feedback 4155
Figure: Architecture of the DBpedia Chatbot
Check http://chat.dbpedia.orghttps://github.com/dbpedia/chatbot
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 5 / 17
Objective of the Ex-Post-Facto Analysis
Understand the nature of user-requests
- query-patterns- user-intentions
Examine whether the chatbot can serve its purpose – satisfy user-requests
Get insights about the conversation flow to improve the chatbot’s architecture
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 6 / 17
Approach Overview
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 7 / 17
ApproachRequest Analysis - Intent Analysis
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 8 / 17
ApproachRequest Analysis - Intent Analysis
Figure: Visualization of clusters obtained via HDBSCAN on sentence embeddings. Each cluster consistsof at least 25 samples. The top 10 clusters out of a total of 33 have been labeled with their top terms.
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 9 / 17
ApproachRequest Analysis - Complexity of utterances
Complex Query
Example: Can you give me the names of women born in the Country during the 19th century?
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 10 / 17
ApproachRequest Analysis - Miscellaneous Analysis
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 11 / 17
Response Analysis
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 12 / 17
Response AnalysisEntity Types in Utterances prior to Negative Feedback
Figure: Entity type distribution from 1000manually annotated failed utterances.
Table: spaCy-NER and DBpedia Spotlightaccuracy for detecting person and locationmentions.
System Person Location
spaCy-NER 41.3% 42.2%DBpedia Spotlight 69.2% 46.1%
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 13 / 17
Conversation Analysis
Figure: Topics as identified by DBpedia Spotlight
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 14 / 17
Implications for DBpedia Chatbot
Adding support for multilingualism
Smart Suggestions
Detecting implicit feedback andout-of-scope queries
Knowledge-based QA
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 15 / 17
Implications for Knowledge-driven Chatbots
Multilingual Support
Guide User Input
Guiding User Expectations
Adding explainability
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 16 / 17
That’s all Folks!
Get in touch:
Rricha JalotaData Science Group, Paderborn University
[email protected]/dice-group/DBpedia-Chatlog-Analysis
Follow us on Twitter: @DiceUPB, @FraunhoferIAIS,@RrichaJalota
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 17 / 17
