Post on 10-Feb-2016
description
transcript
SemTechText Analytics
EvaluationTom Reamy
Chief Knowledge ArchitectKAPS Group
Knowledge Architecture Professional Serviceshttp://www.kapsgroup.com
2
Agenda Text Analytics Features, Varieties, Vendors Evaluation Process
– Start with Self-Knowledge – Text Analytics Team – Features and Capabilities – Filter
Proof of Concept/Pilot– Themes and Issues– Case Study
Conclusion
3
KAPS Group: General Knowledge Architecture Professional Services Virtual Company: Network of consultants – 8-10 Partners – SAS, SAP, FAST, Smart Logic, Concept Searching, etc. Consulting, Strategy, Knowledge architecture audit Services:
– Taxonomy/Text Analytics development, consulting, customization– Technology Consulting – Search, CMS, Portals, etc.– Evaluation of Enterprise Search, Text Analytics– Metadata standards and implementation– Knowledge Management: Collaboration, Expertise, e-learning– Applied Theory – Faceted taxonomies, complexity theory, natural
categories
4
Introduction to Text AnalyticsText Analytics Features Noun Phrase Extraction (Entity, Concept, Events, etc.)
– Catalogs with variants, rule based dynamic– Multiple types, custom classes – entities, concepts, events– Feeds facets
Summarization– Customizable rules, map to different content
Fact Extraction– Relationships of entities – people-organizations-activities– Ontologies – triples, RDF, etc.
Sentiment Analysis– Statistical, rules – full categorization set of operators
5
Introduction to Text AnalyticsText Analytics Features Auto-categorization
– Training sets – Bayesian, Vector space– Terms – literal strings, stemming, dictionary of related terms– Rules – simple – position in text (Title, body, url)– Semantic Network – Predefined relationships, sets of rules– Boolean– Full search syntax – AND, OR, NOT– Advanced – NEAR (#), PARAGRAPH, SENTENCE
This is the most difficult to develop Build on a Taxonomy Combine with Extraction
– If any of list of entities and other words
6
7
8
9
10
11
12
Varieties of Taxonomy/ Text Analytics Software Taxonomy Management
– Synaptica, SchemaLogic Full Platform
– SAS-Teragram, SAP-Inxight, Smart Logic, Data Harmony, Concept Searching, Expert System, IBM, GATE
Content Management – embedded Embedded – Search
– FAST, Autonomy, Endeca, Exalead, etc. Specialty
– Sentiment Analysis , VOC – Lexalytics, Attensity / Reports– Ontology – extraction, plus ontology
Evaluating Taxonomy/Text Analytics Software Start with Self Knowledge Strategic and Business Context Info Problems – what, how severe Strategic Questions – why, what value from the taxonomy/text
analytics, how are you going to use it Formal Process - KA audit – content, users, technology, business
and information behaviors, applications - Or informal for smaller organization, application specific initiatives
Text Analytics Strategy/Model – forms, technology, people– Existing taxonomic resources, software
Need this foundation to evaluate and to develop
13
Evaluating Taxonomy/Text Analytics Software Start with Self Knowledge
Do you need it – and what blend if so? Taxonomy Management Full Functionality
– Multiple taxonomies, languages, authors-editors Technology Environment – Text Mining, ECM, Enterprise Search
– Where is it embedded, integration issues Publishing Process – where and how is metadata being added –
now and projected future– Can it utilize auto-categorization, entity extraction, summarization
Applications – text mining, BI, CI, Social Media, Mobile?
14
Design of the Text Analytics Selection Team
Traditional Candidates - IT Experience with large software purchases
– Search/Categorization is unlike other software Experience with needs assessments
– Need more – know what questions to ask, knowledge audit Objective criteria
– Looking where there is light? Asking IT to select text analytics software is like asking a
construction company to select the design of your house. They have the budget
– OK, they can play.
15
Design of the Text Analytics Selection Team
Traditional Candidates - Business Owners Understand the business
– But don’t understand information behavior Focus on business value, not technology
– Focus on semantics is needed Asking business owners to select text analytics software is
like asking a restaurant owner to do the cooking They can get executive sponsorship, support, and budget.
– OK, they can play
16
Design of the Text Analytics Selection Team
Traditional Candidates - Library Understand information structure
– But not how it is used in the business Experts in search experience and categorization
– Suitable for experts, not regular users Asking librarians to select text analytics software is like asking an
accountant to establish your financial strategy Experience with variety of search engines, taxonomy software,
integration issues– OK, they can play
17
Design of the Text Analytics Selection Team
Interdisciplinary Team, headed by Information Professionals Relative Contributions
– IT – Set necessary conditions, support tests– Business – provide input into requirements, support project– Library – provide input into requirements, add understanding
of search semantics and functionality Much more likely to make a good decision Create the foundation for implementation
18
Evaluating Text Analytics Software – Process
Start with Self Knowledge Eliminate the unfit
– Filter One- Ask Experts - reputation, research – Gartner, etc.• Market strength of vendor, platforms, etc.
– Filter Two - Feature scorecard – minimum, must have, filter – Filter Three – Technology Filter – match to your overall scope
and capabilities – Filter not a focus– Filter Four – Focus Group one day visit – 3-4 vendors
Deep pilot (2) / POC – advanced, integration, semantics Focus on working relationship with vendor.
19
Initial Evaluation Example Outcomes
Filter One:– Company A, B – sentiment analysis focus, weak categorization– Company C – Lack of full suite of text analytics– Company D – business concerns, support– Open Source – license issues– Ontology Vendors – missing categorization capabilities
4 Demos– Saw a variety of different approaches, but – Company X – lacking sentiment analysis, require 2 vendors– Company Y – lack of language support, development cost
20
Evaluating Taxonomy SoftwarePOC - Approach Quality of results is the essential factor 6 weeks POC – bake off / or short pilot Real life scenarios, categorization with your content Preparation:
– Preliminary analysis of content and users information needs– Set up software in lab – relatively easy– Train taxonomist(s) on software(s)– Develop taxonomy if none available
Six week POC – 3 rounds of development, test, refine / Not OOB Need SME’s as test evaluators – also to do an initial
categorization of content
21
Evaluating Taxonomy SoftwarePOC – Initial Design Majority of time is on auto-categorization Need to balance uniformity of results with vendor unique
capabilities – have to determine at POC time Risks – getting software installed and working, getting the right
content, initial categorization of content Elements:
– Content– Search terms / search scenarios– Training sets– Test sets of content
Development Team – expert consultants plus internal taxonomists, technical
22
Evaluating Taxonomy SoftwarePOC – Range of Evaluations Basic – Can this stuff work at all? Auto-categorization to existing taxonomy – variety of content Clustering – automatic node generation Summarization Entity extraction – build a number of catalogs – design which
ones based on projected needs – example privacy info (SS#, phone, etc.)
Entity example –people, organization, methods, etc. Evaluate usability in action by taxonomists Integration – with ontologies Output – XML, API’s
23
Evaluating Text Analytics SoftwarePOC - Issues Quality of content – range of issues – spelling to size to ? Quality of initial human categorization Normalize among different test evaluators Quality of taxonomists – experience with text analytics software
and/or experience with content and information needs and behaviors
Quality of taxonomy– General issues – structure (too flat or too deep)– Overlapping categories – Differences in use – browse, index, categorize
Categorization essential issue is complexity of language Entity Extraction essential issue is scale and disambiguation
24
Evaluating Text Analytics Software Risks CIO/CTO Problem –This is not a regular software process Language is messy not just complex
– 30% accuracy isn’t 30% done – could be 90% Variability of human categorization / expression
– Even professional writers – journalists examples Categorization is iterative, not “the program works”
– Need realistic budget and flexible project plan Anyone can do categorization
– Librarians often overdo, SME’s often get lost (keywords) Meta-language issues – understanding the results
– Need to educate IT and business in their language
25
Case Study: Telecom Service
Company History, Reputation Full Platform –Categorization,
Extraction, Sentiment Integration – java, API-SDK,
Linux Multiple languages Scale – millions of docs a day Total Cost of Ownership Ease of Development - new Vendor Relationship – OEM,
etc.
Expert Systems IBM SAS Smart Logic
Option – Multiple vendors – Sentiment & Platform
26
POC Design Discussion: Evaluation Criteria
Basic Test Design – categorize test set– Score – by file name, human testers
Categorization– Accuracy Level – 80-90%– Effort Level per accuracy level
Sentiment Analysis– Accuracy Level – 80-90%– Effort Level per accuracy level
Quantify development time – main elements Comparison of two vendors – how score?
– Combination of scores and report
27
Text Analytics POC OutcomesCategorization Results
SAS IBM
Recall-Motivation 92.6 90.7
Recall-Actions 93.8 88.3
Precision – Mot. 84.3
Precision-Act 100
Uncategorized 87.5
Raw Precision 73 46
28
Text Analytics POC OutcomesVendor Comparisons
Categorization Results – both good, edge to SAS on precision– Use of Relevancy to set thresholds
Development Environment– IBM as toolkit provides more flexibility but it also increases
development effort Methodology – IBM enforces good method, but takes more time
– SAS can be used in exactly the same way SAS has a much more complete set of operators – NOT, DIST,
START
29
Text Analytics POC OutcomesVendor Comparisons - Functionality
Sentiment Analysis – SAS has workbench, IBM would require more development– SAS also has statistical modeling capabilities
Entity and Fact extraction – seems basically the same– SAS and use operators for improved disambiguation –
Summarization – SAS has built-in– IBM could develop using categorization rules – but not clear that
would be as effective without operators Conclusion: Both can do the job, edge to SAS
30
Conclusion
Start with self-knowledge – what will you use it for?– Current Environment – technology, information
Basic Features are only filters, not scores Integration – need an integrated team (IT, Business, KA)
– For evaluation and development POC – your content, real world scenarios – not scores Foundation for development, experience with software
– Development is better, faster, cheaper Categorization is essential, time consuming Next: Text Analytics + Semantic Web + Ontology
– Integration of Data and Text Mining– Mutual Enrichment – smarter data, richer analytics
31
Questions? Tom Reamy
tomr@kapsgroup.comKAPS Group
Knowledge Architecture Professional Serviceshttp://www.kapsgroup.com