Date post: | 05-Dec-2014 |
Category: |
Data & Analytics |
Upload: | ibm-sverige |
View: | 287 times |
Download: | 2 times |
© 2013 IBM Corporation
A New Era of Smart
Moving from Descriptive to Cognitive Analytics on your Big Data Projects
Date: October 7, 2014
Gene VilleneuveDirector & European Sales Leader Predictive & Business Intelligence
© 2013 IBM Corporation2
A New Era of Smart
Agenda
Introduction and some clarification regarding terminology
The evolution of analyticsDescriptive Predictive Prescriptive Cognitive
Analytics in the Context of Big Data
Big Data & Analytics Reference Model
Sample projects and customer case studies illustrating the evolution of analytics
Current research & development areas
© 2013 IBM Corporation3
A New Era of Smart
INTRODUCTION & TERMINOLOGY
© 2013 IBM Corporation4
A New Era of Smart
Analytics: a Business Imperative across Industries
LOB buyers are driving new demand for industry solutions
The new era of computing enables new analytic methods
At the pointof impact
Big Data and
Analytics
Big Data and
Analytics
All perspectivesAll perspectives
All decisionsAll decisions
All informationAll information
All peopleAll people
Programmatic
SearchDeterministicEnterprise dataMachine languageSimple outputs
Cognitive
Discovery Probabilistic Big Data Natural language Intelligent options
* Source: IBM Market Development & Insight – GMV 1H2013
© 2013 IBM Corporation5
A New Era of Smart
The Evolution of Analytics
CognitiveAnalytics
PredictiveAnalytics
PrescriptiveAnalytics
DescriptiveAnalytics
Descriptive “After-the-facts”
analytics by analyzing historical data
Provides clarity as to where an enterprise or an organization stands related to defined business measures
Applied to all LoB for fact finding, visualization of success and failure
Cognitive Pertaining to the
mental processes of perception, memory, judgment, learning, and reasoning
Range of different analytical strategies that are used to learn about certain types of business related functions
Natural language processing
Predictive Leverages data
mining, statistics and ML algorithms, etc. to analyze current and historical data to predict future events and business outcome.
Discovers patterns derived from historical and transactional data to optimize business measures
Prescriptive Synthesizes big data,
mathematical and computational sciences, and business rules to suggest decision options
Takes advantage of a future opportunity or mitigate a future risk and shows the implication of each decision option
© 2013 IBM Corporation6
A New Era of Smart
What could happen in the future?
Information Layer
How is data managed and stored?
How can everyonebe more right…….more often?
How can everyonebe more right…….more often?
Descriptive
What has already happened?
Predictive
Prescriptive
How can we achieve the best outcome?
Cognitive
How can we learn dynamically?
Bus
ines
s V
alue
Bus
ines
s V
alue
Reasoning Learning Natural Language
Reasoning Learning Natural Language
Alerts & Drill Down Ad hoc Reports Standard Reports
Alerts & Drill Down Ad hoc Reports Standard Reports
Big Data Platforms Content Management RDBMS and Integration
Big Data Platforms Content Management RDBMS and Integration
Machine learning Forecasting Statistical Analysis
Machine learning Forecasting Statistical Analysis
Optimization Rules Constraints
Optimization Rules Constraints
IBM Big Data & AnalyticsIBM Big Data & Analytics
The Scope of Advanced Analytics
• IBM analytics breadth covers the full spectrum of decisions
• IBM is the undisputed leader in advanced analytics
© 2013 IBM Corporation7
A New Era of Smart
Accelerating the Client’s Journey to Cognitive
Natural, Intuitive or Automated Interaction
Co
nte
xt S
pec
ific
Usa
ge Opportunities to infuse cognition and
collaboration in existing solutions and products for differentiation
The Analytics
Contin
uum
INFORMATIO
N FOUNDATION
DESCRIPTIVE
PREDIC
TIVE
PRESCRIPTIVE
C
OGNITIVE Win on
Innovation
Compete on time to business value – through context specific data, methods, workflow.
Reasoning
Learning
Natural Language
Optimization
Rules
Predictive Modeling
Forecasting
Statistical Analysis
Alerts
Drilldown Query
Ad-hoc Reports
Standard Reports
Big Data Platforms
ECM
Information Integration
RDBMS
© 2013 IBM Corporation8
A New Era of Smart
Analytics: a Business Imperative across Industries
Clients realize value through solutions
* Source: IBM Market Development & Insight – GMV 1H2013
IBM Predictive Maintenance & QualityImproves productivity, prevents downtime and reduces costs
IBM Predictive Maintenance & QualityImproves productivity, prevents downtime and reduces costs
IBM Credit Risk ManagementDerive competitive advantage from risk management processes
IBM Credit Risk ManagementDerive competitive advantage from risk management processes
IBM Enterprise Marketing ManagementDiscover and react in real time to how consumers are interacting
IBM Enterprise Marketing ManagementDiscover and react in real time to how consumers are interacting
IBM Social Media AnalyticsUncover customer sentiment, predict behavior, improve marketing
IBM Social Media AnalyticsUncover customer sentiment, predict behavior, improve marketing
IBM Watson Engagement AdvisorTransforms client experience with deep personalized Q&A
IBM Watson Engagement AdvisorTransforms client experience with deep personalized Q&A
© 2013 IBM Corporation9
A New Era of Smart
IBM’s Portfolio delivers Business Value
Business value from automation of routine decisions, to transformative new usages of data
Line of Business Leaders
Industry Solutions Integrated by Design
Big Data & Analytics
Mobile Social
Cloud
Client-Driven Capabilities and
Platforms
Market-Growth Initiatives
Big Data Infrastructure
Predictive Prescriptive Cognitive
Sup
port
ed b
y IB
M e
xper
tise
thr
ough
BA
O s
ervi
ces
Smarter Commerce
Smarter Workforce
Smarter Cities
CPO CMO CIOCFOCHRO CRO Mayors
Smarter Analytics
Cloud
© 2013 IBM Corporation10
A New Era of Smart
1.3M IOPS Scalability99.997% Availability 0 Incidents, Vulnerability
Pow
er S
yste
ms
Des
ign
Pow
er S
yste
ms
Des
ign
Open & flexible infrastructure - Available on premise or through the CloudOpen & flexible infrastructure - Available on premise or through the Cloud
Industry SolutionsIndustry Solutions Cognitive ComputingCognitive ComputingBusiness & Predictive AnalyticsBusiness & Predictive Analytics
Pow
er S
olu
tion
sP
ower
Sol
utio
ns
IBM WatsonIBM Watson
1,000+ Concurrent Queries
Parallel processing
Large-scale memory processing
Massive IO bandwidth
Stream Computing
Real-time Analytics
Natural Language Learning
Continuous data load
Power Systems enables next Generation Big Data and Analytics Applications
© 2013 IBM Corporation11
A New Era of Smart
ANALYTICS IN THE CONTEXT OF BIG DATA
© 2013 IBM Corporation12
A New Era of Smart
Data Content>80%<20%
Data Content>80%<20%
Analytics in the Context of Big Data - The Big Data Analytics Challenge
Requiring overcoming the high volume, real-time, and unstructured nature of social media and Enterprise data streams
From noisy data to trustworthy insights
Understand jargon and acronyms, eliminate spam
Heterogeneous data
Combine, correlate information over 100’s of sources (sites, forums, message boards, newswires…)
Timely Decision making Make decisions in near real-time over 10K+ messages/second
Growing volume of data
Social media or other media source data
Extract concepts from several 100M messages/day
100M+ active users per source
VeracityVeracity
VarietyVariety
VelocityVelocity
VolumeVolume
Data Volume
360-degree Profiles• Micro-segmentation• Predict Behavior
360-degree Profiles• Micro-segmentation• Predict Behavior
Manual Interaction• Polling & ExtrapolationManual Interaction• Polling & Extrapolation
Listening and Monitoring• Sentiment, Buzz• Key influencers
Listening and Monitoring• Sentiment, Buzz• Key influencers
Ana
lytic
s C
ompl
exity
Learning, NLP, Discovery• Auditory & visual processing • Logic & reasoning • Improve interventions
Learning, NLP, Discovery• Auditory & visual processing • Logic & reasoning • Improve interventions
© 2013 IBM Corporation13
A New Era of Smart
Analytics in the Context of Big Data - Key Drivers for Cognitive Analytics
The need for cognitive analytics is driven by the confluence of SoLoMo (Social, Local, Mobile), Big Data, and Cloud
VeracityVeracity VarietyVariety
VelocityVelocity VolumeVolume
Cognitive Systems
© 2013 IBM Corporation14
A New Era of Smart
Analytics in the Context of Big Data - Veracity / Trust / Sentiment
Addressing the information trustworthiness of social media data
Some dimensions of trustworthiness /
Trustworthiness Sentiment
– Jokes– Prosody– Sarcasm– Seriousness– Emotion– Mood– Ambiguity– Humor– Dialect– Social factors …– Social media languages– Context– etc.
VeracityVeracity
InformationProvenance
AuthorClassification
IntegrityAssumption
ContentAnalysis
RelevanceDetermination
UsageIntention
InformationProvenance
AuthorClassification
IntegrityAssumption
ContentAnalysis
RelevanceDetermination
UsageIntention
© 2013 IBM Corporation15
A New Era of Smart
Analytics in the Context of Big Data DeepQA: The Architecture underlying Watson Generates many hypotheses, collects wide range of evidence, balances the combined
confidences of >100 different analytics that analyze the evidence from different dimensions
Answer Scoring
Models
Answer & Confidence
Evidence Sources
Models
Models
Models
Models
ModelsPrimarySearch
CandidateAnswer
Generation
HypothesisGeneration
Hypothesis and Evidence Scoring
Final Confidence Merging & Ranking
Synthesis
Answer Sources
Question & Topic Analysis
EvidenceRetrieval
Deep Evidence Scoring
Learned Modelshelp combine and
weigh the Evidence
HypothesisGeneration
Hypothesis and Evidence Scoring
QuestionDecomposition
Each year the EU selects capitals of culture; one of the 2010 cities was this Turkish “meeting place of cultures”
© 2013 IBM Corporation16
A New Era of Smart
99%60%10%
Understands natural language and human speech
Adapts and Learns from user selections and responses
Generates and evaluates
hypothesis for better outcomes
3
2
1
…built on a massively parallel probabilistic evidence-based
architecture optimized for Linux on POWER7+
Analytics in the Context of Big Data - Watson drives optimized outcomes
© 2013 IBM Corporation17
A New Era of Smart
BIG DATA ANALYTICS REFERENCE MODEL
© 2013 IBM Corporation18
A New Era of Smart
Big Data & Analytics Platform
An innovative, foundational big data platform can help tackle big data’s four V’s (volume, variety, velocity and veracity) with an integrated set of big data technologies to address the business pain, reduce time and cost, and provide quicker return on investment
More cost-effectively analyze petabytes of structured and unstructured formation
Analyze streaming data and large data bursts for near-real-time insights
Access deep insight with advanced in-database analytics and operational analytics
Big data platform
Data warehouseApache Hadoop system Stream computing
Data Media Content Machine Social
Systems management Application development Discovery
Information integration and governance
© 2013 IBM Corporation19
A New Era of Smart
Infrastructure Services
Dat
a Tr
ansf
orm
ation
& In
tegr
ation
Lay
er
Het
erog
eneo
us D
ata
Sour
ces
Visu
aliz
ation
&Re
porti
ng L
ayer
Data Persistency Layer
Business Analytics &Applications Layer
Infrastructure Services
Dat
a Tr
ansf
orm
ation
& In
tegr
ation
Lay
er
Het
erog
eneo
us D
ata
Sour
ces
Visu
aliz
ation
&Re
porti
ng L
ayer
Data Persistency Layer
Business Analytics &Applications Layer
Big Data Analytics Reference Model - Key Capabilities
Components to build a trusted information integration layer with ETL, data quality, real-time data processing, federation, metadata mgmt, …
Comprehensive Big Data advanced analytics layer with applications & research assets on heterogeneous source data
Traditional reporting and BI analytics, with visualization & exploration of heterogeneous data
Traditional DW system (SOR, ODS, marts) with MDM system, DW appliances, and augmented with Hadoop platform
Common infrastructure services, such as systems management, security, backup, information governance, …
Heterogeneous data landscape including existing data stored in BSS systems, from the network, external, customer touch points
© 2013 IBM Corporation22
A New Era of Smart
SAMPLE PROJECTS AND CUSTOMER CASE STUDIES ILLUSTRATING THE EVOLUTION OF ANALYTICS
(IN THE CONTEXT OF BIG DATA)
CognitiveAnalytics
PredictiveAnalytics
PrescriptiveAnalytics
DescriptiveAnalytics
© 2013 IBM Corporation24
A New Era of Smart
Predictive AnalyticsDemographics Enrichment for unknown Subscribers
Gain analytical insight for pre-paid demographics Understand post-paid subscribers
– Using post-paid demographics data (age, gender, income, …)– Gaining insight: propensity/predictive modeling, micro-segmentation,
clustering, sentiment analytics, … from appl usage data, web browsing, CDR, social media
Understand pre-paid subscribers– Gaining insight: propensity/predictive modeling, micro-segmentation,
clustering, sentiment analytics, …– Demographics data isn't available or not sufficiently trustworthy
Correlate post- with pre-paid subscribers and map demographics– Correlate post- with pre-paid segments, clusters, behavior, interest, …– Map known demographics for post-paid to corresponding pre-paid
subscribers
Required Data Sources Voice & data CDR (MSISDN & Usage) Behavioral data:
– Web browsing & search (internal and external), user agent: browser, appl and/or device that made request, content type: type of data sent/downloaded
Public sources (will be used, not required from CSP):– Wikipedia– IMDB http://www.imdb.com/– Open Directory Project (ODP)
Subscriber reference data (e.g. from CRM or EDW)
PredictiveAnalytics
PredictiveAnalytics
© 2013 IBM Corporation25
A New Era of Smart
Predictive AnalyticsDemographics Enrichment for unknown Subscribers
CSP
DATA SOURCESCSP & other
Voice & data CDR (MSISDN & Usage)
MSP (MSISDN & URL) Behavioral data (e.g. blogs, use of
mobile apps, Web browsing & Web search )
Public sources (e.g. ODP) Metadata, e.g. time, size, … CRM or EDW
Data understandingData transformationData preparation
IBM BigInsights Admin Customer Modeler Admin(Predictive Analytics)
IBM Singapore
Data anonymization Data provisioning
CorrelationPredictive modelingPropensity modelingMicro-segmentationClusteringSentiment
Analytical insightVisualizationConsumption by Advertisement
PRODUCTS & Tools
BigInsights (incl. BigSheets, SystemT, HDFS, Jaql, …)
Customer Modeler SPSS Modeler NLP DB2
SaaS
PredictiveAnalytics
PredictiveAnalytics
© 2013 IBM Corporation26
A New Era of Smart
HDFSAnalytical Model
(pre-paid)
DB2Predictive Model
(for pre-paid)
HDFSAnalytical Model
(post-paid)
Predictive AnalyticsDemographics Enrichment for unknown Subscribers
Post-paid CSP
Data Sources:Voice/Data CDRs Behavioral Data
• InfoSphere BigInsights• Customer Modeler• SPSS / DB2 / NLP
GTS SmartCloud Enterprise
Analysis/InsightPre-paid:
• Age• Gender• Income
TransformationAnonymization
(to be validated)
Source DataTransformation
Pre-paid CSP
Data Sources:Voice/Data CDRs Behavioral Data
Public Sources(not from CSP):
WikipediaIMDBODP
Used for gainingAnalytical Insight
Post-paid CSP
Data Sources:Subscriber
Demographics
Visualization
Used for buildingPredictive Model
PredictiveAnalytics
PredictiveAnalytics
© 2013 IBM Corporation27
A New Era of Smart
XO Communications takes control of customer satisfaction
142 percent reductionin revenue erosion for customers at most risk of churning
$10 million+ savings/yearfrom increased retention and reduced customer service costs
5 months to achieve full return on investment
Solution components
The transformation: XO Communications had already taken the first steps in identifying customer retention risks through analytics; now it wanted to seize the opportunity to put these insights into action more effectively. By using IBM® SPSS® solutions to hone its predictive models, the company built a richer, more up-to-date picture of its client base and began delivering this data to a greater range of employees.
“We are only just starting to realize the true potential that IBM analytics holds across the business.”
— Bill Helmrath, Director of Business Intelligence, XO Communications • IBM® SPSS® Analytics Catalyst• IBM SPSS Modeler• IBM SPSS Modeler Server• IBM SPSS Statistics• IBM InfoSphere® BigInsights™
YTP03235-USEN-00
© 2013 IBM Corporation28
A New Era of Smart
IBM® AIX® IBM Cognos® Business Intelligence IBM DB2® IBM InfoSphere® Warehouse IBM PowerHA® IBM PowerVM® IBM SPSS® IBM Tivoli® Storage Manager and
System Automation for Multi-Platforms
IBM WebSphere® Application Server IBM Power® 770
Fiserv cuts IT costs while enhancing analytics capabilities with software and infrastructure from IBM
$8 million savedin IT costs over a five-year period
90% reductionin the number of midrange servers under management
Boosts availabilityand improves the agility of service delivery
Solution Components
Business Challenge: Fiserv was seeking new ways to attract, retain and grow profitable customer relationships while helping its clients compete with newer and larger banks. Leveraging predictive analytics applications proved key to this goal, but Fiserv realised that it also needed a more agile, available and scalable IT infrastructure to support its new capabilities.
The Solution: IBM information management and predictive analytic solutions enable Fiserv to transform billions of raw transactions into actionable insights that help small and midsize banks better target offers and maximize their marketing dollars. The use of cloud technologies to consolidate and virtualize servers helps reduce costs and accelerate time-to-market.
“We have estimated a five-year-cumulative run rate reduction of about $8 million with the server consolidation and virtualization project.”
—Leroy Hill, Manager, Midrange Engineering, Fiserv
© 2013 IBM Corporation29
A New Era of Smart
Cognitive AnalyticsHalalan 2013 Social Media Tracking
BUZZ – candidates, topics, personalities, broadcasters
– How much / What is being said about the candidates (ongoing and for key “events” like debates, advertisements, etc.), different shows, news anchors.
– How does this change over time, what is trending.
SENTIMENT – popular opinion– What do voters like or dislike about the candidates,
the parties, campaigns, constituents, etc.– How does this sentiment break down by the
different groups (voters, political affiliation, news professionals, demographics, affinity groups, etc.)
– Understand brand sentiment, i.e., whether ABS-CBN is being perceived as unbiased and trusted. How are the different news personalities being perceived: credible, neutral, fair?
INTENT – action– What is the intent to act (support / vote) for each
candidate.– What election outcomes can be predicted (shifts in
candidate sentiment, voter intent, etc.)
CognitiveAnalytics
CognitiveAnalytics
© 2013 IBM Corporation30
A New Era of Smart
CURRENT RESEARCH & DEVELOPMENT AREAS
(just a few examples)
© 2013 IBM Corporation31
A New Era of Smart
Cognitive Analytics: Technical Capabilities requiredWatson Solutions – Build on repeatable Assets
Watson forHealthcare
Watson forFinancial Services
Watson forClient Engagement
Watson for Industry
Sol
utio
ns Sample Advisor Solutions Sample Advisor Solutions Sample Advisor Solutions
Utilization
Oncology
Research
Care Mgt.
Banking
Financial Markets
Insurance Call Center
Knowledge
Help Desk
Technical
NLP & MachineLearning
Data Analytics Cloud Mobile Workload OptimizedSystems
100111001
10010010010
1000101100101
10001010010
00110101
Cap
abili
ties
ASK Services DISCOVER Services DECISION Services
Pla
tfor
m
Content Tooling Methods Algorithms APIs
Ready Build Teach RunFull Lifecycle
© 2013 IBM Corporation32
A New Era of Smart
Massive Scale SNA (X-RIME) over BigInsightsCurrent Research Area Project Overview
– X-RIME is a library that consists of MapReduce programs, which are used to do raw data pre-processing, transformation, SNA metrics and structures calculation, and graph / network visualization
– Based on IBM InfoSphere BigInsights (Hadoop) – Goes beyond SPSS SNA for churn propensity modeling
Reference– Commercial Solution: China Mobile enterprise blog analysis solution– ARL MSA on Power Benchmarking: Pageranking 390 millions of nodes
on 10-nodes power7 cluster (2 hours per iteration)– Integrated to SystemG as GraphBase– Open Source X-RIME on SourceForge
Selected X-RIME SNA Algorithms
– Vertex degrees (in/out/both/average/max)
– Weekly connected components
– Bi-connected components– Breadth first search (BFS) – K-core– Maximal clique– Community detection
based on label propagation
– Community detection based on scored label propagation
– Community detection based on propinquity
– Modularity evaluation– Hyperlink induced topic
search (HITS)– Pagerank– Minimal spanning tree
(MST)– Ego-centric network– Vertex clustering
coefficient– Edge clustering coefficient
HDFSHDFS
Graph Data Model (Object)Graph Data Model (Object)
Message Passing FrameworkMessage Passing Framework
MapReduceMapReduce
SNA library SNA library
X-RIME Architecture
© 2013 IBM Corporation
A New Era of Smart
Thank you