Diploma of Advanced Studies

Outline MD Classification MDSSL Application to SA Conclusions Future Work

DEA - Diploma of Advanced StudiesSemi-supervised Learning of Multi-dimensional Class Bayesian

Network Classifiers: Application to Sentiment Analysis

Jonathan Ortigosa-Hernandez

advised by

Jose A. Lozano and Inaki Inza

Intelligent Systems GroupComputer Science and Artificial Intelligence Department

University of the Basque Country

November 4th, 2010

Jonathan Ortigosa-Hernandez DEA - Diploma of Advanced Studies


Curriculum Vitae

Undergraduate Education2003-2008: Masters of Science Degree in ComputerEngineering, University of the Basque Country.2007-2008: Bachelor of Science Degree in Informatics,Coventry University.

Postgraduate Education2008-Present: PhD Student, ISG Group, University of theBasque Country (Four-Year MEC-FPU Grant).Doctorate Program: Probabilistic Graphical Models forArtificial Intelligence and Data Mining.Doctorate Lectures:

Fundamentals of Probabilistic Graphical ModelsInference in PGMsLearning PGMsBioinformatic Applications of PGMsScientific Research MethodologyStatistical and Computational Basis for PGMs



Research Interest

MethodologyMulti-dimensional ClassificationMulti-dimensional Class Bayesian Network ClassifiersSemi-supervised Learning

ApplicationsOpinion Mining and Sentiment AnalysisAffect Analysis



1 Multi-dimensional Supervised Classification

2 Multi-dimensional Semi-supervised Learning

3 Application to Sentiment Analysis

4 Conclusions

5 Current and Future Work



Supervised Classification

It consists of building a classifier Ψ from a given labelledtraining dataset D, by using an induction algorithm A(A(D) = Ψ),

X1 X2 ... Xn C

x(1)1 x

(1)2 ... x

(1)n c (1)

x(2)1 x

(2)2 ... x

(2)n c (2)

... ... ... ... ...

x(N)1 x

(N)2 ... x

(N)n c (N)

in order to predict the value of a class variable C for any newunlabelled instance x (Ψ(x) = c).



Uni-dimensional and Multi-dimensional Classification

Uni-dimensional classification tries to predict a single classvariable based on a dataset composed of a set of labelledexamples.

(Uni-dimensional Class) Bayesian Network Classifiers(Larranaga et al, 2005).

Multi-dimensional classification is the generalisation of thesingle-class classification task to the simultaneous predictionof a set of class variables.

Multi-dimensional Class Bayesian Network Classifiers (v.d.Gaag and d. Waal, 2006).Do not confuse with multi-class and multi-label classification.



Multi-dimensional Supervised Learning

A typical supervised training dataset

X1 X2 ... Xn C1 C2 ... Cm

x(1)1 x

(1)2 ... x

(1)n c

(1)1 c

(1)2 ... c

(1)m

x(2)1 x

(2)2 ... x

(2)n c

(2)1 c

(2)2 ... c

(2)m

... ... ... ... ... ... ... ...

x(N)1 x

(N)2 ... x

(N)n c

(N)1 c

(N)2 ... c

(N)m

Each instance of the dataset contains both the values of theattributes and m labels which characterise the attributes.



Bayesian Network Classifiers

X1 X2 X3 X4 X5 X6

C

Figure: A (uni-dimensional) naive Bayes structure.



Multi-dimensional Class Bayesian Network Classifiers(MDBNC)

X1 X2 X3 X4 X5 X6

C1 C2 C3

Figure: A multi-dimensional naive Bayes structure.



MDBNC Structure

X1 X2 X3 X4 X5

C1 C2 C3

X1 X2 X3 X4 X5

C1 C2 C3

X1 X2 X3 X4 X5

(a) Complete graph

(c) Class subgraph

(b) Feature selection subgraph

(d) Feature subgraph

C1 C2 C3

Figure: A MDNBC structure and its division.



Sub-families of MDBNC

(a) Multi-dimensional naive Bayes

(c) Multi-dimensional J/K dependence Bayesian (2/3)

(b) Multi-dimensional tree-augmented network

!"

!#

!$

!%

!& !'

!(

!)

*# *$ *%

!+ !",

*"

!" !# !$ !% !& !' !( !) !* !"+

," ,# ,$

!" !# !$ !% !&

'" '#

Figure: Different subfamilies of MDBNC.



Sub-families of MDBNC - MDnB

Multi-dimensional naive Bayes (MDnB)

!" !# !$ !% !& !' !( !) !* !"+

," ,# ,$The class and featuresubgraphs are empty.

Each class variable isparent of all thefeatures.




Multi-dimensional naive Bayes (MDnB)

!" !# !$ !% !& !' !( !) !* !"+

," ,# ,$ It has a fixed structure.

Thus, it has nostructural learning (v.d.Gaag and d. Waal,2006).




Multi-dimensional tree-augmented network classifier (MDTAN)

!" !# !$ !% !&

'" '#

The class and featuresubgraphs are trees.




Multi-dimensional tree-augmented network classifier (MDTAN)

!" !# !$ !% !&

'" '#

A wrapper structurallearning algorithm isproposed in (v.d. Gaagand d. Waal, 2006).




Multi-dimensional J/K dependence Bayesian classifier (MD J/K )

!"

!#

!$

!%

!& !'

!(

!)

*# *$ *%

!+ !",

*"

The class subgraph is aJ-dependence graph.

The feature subgraph isa K -dependence graph.




Multi-dimensional J/K dependence Bayesian classifier (MD J/K )

!"

!#

!$

!%

!& !'

!(

!)

*# *$ *%

!+ !",

*"

There was not a specificstructural learningalgorithm.

So, we proposed alearning algorithm in(Ortigosa-Hernandez etal, 2010).



A supervised method to learn a MD J/K structure(Ortigosa-Hernandez et al, 2010)

Step 0 - Initialisation

Establish the maximumnumber of parents inboth class and featuresubgraphs, i.e. J = 2and K = 2.

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4




Step 1 - Learn the structure between the class variables (Ac )

Calculate the mutualinformation MI (Ci ,Cj )for each pair of classvariables.

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





Calculate the p-values(significance of eachmutual information)using independence test.

C1 C2 C3

C4 0.36 0.57 0.01C3 0.27 0.63C2 0.06

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





Remove the p-valuesgreater than thethreshold α = 0.1.

C1 C2 C3

C4 0.36 0.57 0.01C3 0.27 0.63C2 0.06

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





From the lowest value,start adding arcs to thegraph fulfilling theconditions of no cyclesand no more thanJ-parents per classvariable.

C1 C2 C3

C4 x x 0.01C3 x xC2 0.06

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





From the lowest value,start adding arcs to thegraph fulfilling theconditions of no cyclesand no more thanJ-parents per classvariable.

C1 C2 C3

C4 x x xC3 x xC2 0.06

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4




Step 2 - Learn the structure between the class variables and thefeatures (ACF )

Calculate the mutualinformation MI (Ci ,Xj )for each pair Ci and Xj . X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





Calculate the p-value ofeach mutualinformation.

C1 C2 C3 C4

X1 0.64 0.00 0.77 0.98X2 0.82 0.03 0.11 0.37X3 0.00 0.06 0.00 0.01X4 0.68 0.09 0.00 0.55X5 0.81 0.12 0.81 0.65X6 0.57 0.24 0.00 0.00X7 0.25 0.26 0.00 0.00X8 0.32 0.15 0.00 0.44

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





Remove the p-valuesgreater than thethreshold α = 0.1.

C1 C2 C3 C4

X1 0.64 0.00 0.77 0.98X2 0.82 0.03 0.11 0.37X3 0.00 0.06 0.00 0.01X4 0.68 0.09 0.00 0.55X5 0.81 0.12 0.81 0.65X6 0.57 0.24 0.00 0.00X7 0.25 0.26 0.00 0.00X8 0.32 0.15 0.00 0.44

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





Add all the arcs to thestructure.

C1 C2 C3 C4

X1 x 0.00 x xX2 x 0.03 x xX3 0.00 0.06 0.00 0.01X4 x 0.09 0.00 xX5 x x x xX6 x x 0.00 0.00X7 x x 0.00 0.00X8 x x 0.00 x

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4




Step 3 - Learn the structure between the features(AF )

Calculate theconditional mutualinformationMI (Xi ,Xj ||Pac(Xj )).

Calculate thep-values.

Remove thep-values greaterthan the thresholdα = 0.1.

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4




Step 3 - Learn the structure between the features(AF )

Add arcs between thefeatures fulfilling theconditions of no cyclesbetween the featuresand no more thanK -parents per feature.

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4



Major Problem of Supervised Learning

However, in many real world problems, obtaining data isrelatively easy, while labelling is difficult, expensive or laborintensive (usually done by an external mechanism, e.g. humanbeings).

This problem is accentuated when using multiple targetvariables.

DESIRE: Learning algorithms able to incorporate a largenumber of unlabelled data with a small number of labeleddata when learning competitive classifiers.



Multi-dimensional Semi-supervised Learning

A typical semi-supervised training dataset

X1 X2 ... Xn C1 C2 ... Cm

x(1)1 x

(1)2 ... x

(1)n c

(1)1 c

(1)2 ... c

(1)m

x(2)1 x

(2)2 ... x

(2)n c

(2)1 c

(2)2 ... c

(2)m

... ... ... ... ... ... ... ...

x(L)1 x

(L)2 ... x

(L)n c

(L)1 c

(L)2 ... c

(L)m

x(L+1)1 x

(L+1)2 ... x

(L+1)n ? ? ... ?

x(L+2)1 x

(L+2)2 ... x

(L+2)n ? ? ... ?

... ... ... ... ... ... ... ...

x(N)1 x

(N)2 ... x

(N)n ? ? ... ?

Semi-supervised Learning fulfils this desire.



The Expectation-Maximisation Algorithm

The EM algorithm (Dempster et al, 1977)

Learn an initial model.Repeat until convergence:(a) Expectation step: Using the current model, estimate themissing values of the data.(b) Maximisation step: Using the whole data and the previousestimations, learn a new current model.

Any MDBNC learning algorithm can be used as model in thisalgorithm.



Artificial Experimentation

20 different mult-dimensional datasets are sampled.

Training subset: 10, 000 instances (L:100/U:10, 000)Test subset: 5, 000

Feat. Class V.

Num. 5 to 20 2 to 4Card. 2 to 4 2 to 3

Learning algorithms:

4 uni-dimensional (nB, TAN, 2-DB and 3-DB)4 multi-dimensional (MDnB, MDTAN, MD 2/2 and MD 2/3)

Scenario: Supervised / Semi-supervised



Artificial Experimentation

Numerical results can be found in:

http://www.sc.ehu.es/ccwbayes/members/jonathan/home/News_and_Notables/Entries/2010/11/30_

Artificial_Experiments_2010.html

(Semi-supervised) Multi-dimensional algorithms

V(Supervised) Multi-dimensional algorithms

VUni-dimensional algorithms


http://www.sc.ehu.es/ccwbayes/members/jonathan/home/News_and_Notables/Entries/2010/11/30_Artificial_Experiments_2010.html

http://www.sc.ehu.es/ccwbayes/members/jonathan/home/News_and_Notables/Entries/2010/11/30_Artificial_Experiments_2010.html


Application to Sentiment Analysis

Sentiment Analysis (AKA Opinion Mining) is thecomputational study of opinions, sentiments and emotionsexpressed in text (Liu, 2010).

When treating Sentiment Analysis as a classification problem,several different (but related) problems appear. For example:

1 Subjectivity Classification. Its aim is to classify a text assubjective or objective.

2 Sentiment Classification. It classifies an opinionated text asexpressing a positive, neutral, or negative opinion.



Motivation for Using Semi-supervised Learning ofMulti-dimensional Classifiers

1 Up to now, these subproblems have been studied in isolationdespite of being closely related. So, probably it would behelpful to use multi-dimensional classifiers.

2 Obtaining enough labeled examples for a classifier may becostly and time consuming. This motivates us to deal withunlabelled examples in a semi-supervised framework.



Hypothesis Formulated

First Hypothesis

The explicit use of the relationships between different classvariables can be beneficial to improve their recognition rates.

Second Hypothesis

Multi-dimensional techniques can work with unlabelled data inorder to improve the classification rates in Sentiment Analysis.



Properties of the Dataset

Collected by Socialware Company S.A., from the ASOMOservice of mobilised opinion analysis.It consists of 2, 542 Spanish reviews extracted from a blog:

150 documents have been labeled in isolation by an expert.2, 392 posts are left unlabelled.

Figure: The ASOMO corpus.



Properties of Each Document

Each document is represented as:

14 features

Obtained by using an open source morphological analyser(Carreras et al, 2006).Each feature provide different information related topart-of-speech (POS).Eg. First Persons, Agreement Expressions, Imperatives,Prediction Verbs (future), Questions, Positive Adjectives, etc.Represented as a real number between 0 and 1.

3 class variables

Will to Influence: {declarative sentence, soft WI, medium WI,strong WI}Sentiment: {very negative, negative, neutral, positive, verypositive}Subjectivity: {Yes (subjective), No (objective)}



Experiment 1 - Set Up I

First Hypothesis


The ASOMO corpus has been used to learn:

3 (uni-dimensional) naive Bayes classifiers, one per each classvariable.A (uni-dimensional) naive Bayes classifier with a compoundclass variable.A multi-dimensional naive Bayes classifier.



Experiment 1 - Set Up II

First Hypothesis


Features from ASOMO dataset are discretised into 3 valuesusing equal frequency.

In addition to the ASOMO feature set, two state-of-the-artfeature sets are used:

UnigramsUnigrams + Bigrams

Results averaged over 5 × 5 fold cross validation.



Experiment 1- JOINT Accuracies

Figure: JOINT accuracies on ASOMO corpus using three differentfeature sets in both uni and multi-dimensional scenarios (5× 5cv)



Experiment 1 - Computation Time

Figure: Computational times of the learning algorithms using differentfeature sets.



Experiment 2 - Set Up

Second Hypothesis

Multi-dimensional techniques can work with unlabelled data inorder to improve the classification rates in Sentiment Analysis.

The ASOMO dataset has been used to learn:

3 (uni-dimensional) Bayesian network classifiers: nB, TAN and2DB.5 MDBNC: MDnB, MDTAN, MD 2/2, MD 2/3 and MD 2/4.

In both Supervised and Semi-supervised (EM algorithm)learning frameworks.

Features from ASOMO dataset are discretised into 3 valuesusing equal frequency.

Results averaged over 5 × 5 fold cross validation.



Experiment 2 - JOINT Accuracy

5

10

15

20

25

nB TAN 2DB MDnB MDTAN MD 2/2 MD 2/3 MD 2/4

Figure: JOINT accuracies on ASOMO dataset in the supervised andsemi-supervised learning frameworks.



Experiment 2 - Will to Influence

30

40

50

60

70


Figure: Accuracies for the Will to Influence class variable on ASOMOdataset in the supervised and semi-supervised learning frameworks.



Experiment 2 - Sentiment Polarity

20

25

30

35

40


Figure: Accuracies for the Sentiment Polarity class variable on ASOMOdataset in the supervised and semi-supervised learning frameworks.



Experiment 2 - Subjectivity

50

60

70

80

90


Figure: Accuracies for the Subjectivity class variable on ASOMO datasetin the supervised and semi-supervised learning frameworks.



Conclusions I - Methodology

Multi-dimensional classification and semi-supervised learningare two different branches of machine learning.

With this research, we have established a bridge betweenthem showing that:

Uni-dimensional approaches cannot capture the real nature ofmulti-dimensional problems.More accurate classifiers can be found using themulti-dimensional learning approaches.The use of large amounts of unlabelled data can be beneficialto improve recognition rates.



Conclusions II - Application

With respect to the Sentiment Analysis application, we haveproposed a novel perspective to solve the problem.Experimental results demonstrate that the use ofmulti-dimensional classification, as well as the use ofunlabelled data, can lead us to more accurate classifiers.



Conclusions III - Publications

Publication

J. Ortigosa-Hernandez, J.D. Rodriguez, L. Alzate, I. Inza, J.A.Lozano. (2010). A Semi-supervised Approach to Multi-dimensionalClassification with Application to Sentiment Analysis. CEDI 2010,V Simposio de Teoria y Aplicaciones de Mineria de Datos(TAMIDA2010), Valencia, Spain.

We are writing the final draft of a paper for the Special Issue onData Mining Applications and Case Studies at Neurocomputing.



Short Term Future Work

Title: Application to Affect AnalysisDescription: Use the methodology proposed in this presentation to dealwith the problem of Affect Analysis.Collaboration: Socialware S.A.

Motivation (Abbasi et al., 2008)

Affect Analysis is concerned with the analysis of text containing emotionsand it tries to extract a large number of potential emotions, e.g.happiness, sadness, anger, hate, violence, excitement, etc.



Short Term Future Work

We want to take a step forward in this problem taking advantageof the potential possibilities of the MDBNC to model complexrelationships between the class variables.

(a) Plutchik’s affect model (b) Chromatic affect model

Figure: Psychological Human Affection models considered for this project.



Long Term Future Work

Wrapper Structural Search Algorithm.

Adapt other semi-supervised learning approaches to themulti-dimensional classification domain, e.g. Co-training,Active Learning, ...

The scalability of MDBNC is a problem that has to be studied(high dimensionality of multi-label problems).

“As discussed in the literature, currently there is no coherentstrategy for handling unlabelled data, so some creativity must beexercised.” (Cohen’s thesis, 2003)



Questions

THANK [email protected]



References

Abbasi, A. and Chen, H. and Thoms, S. and Fu, T. (2008). Affect Analysis of Web Forums and Blogs usingCorrelation Ensembles. IEEE Transactions on Knowledge and Data Engineering, Vol. 20(9), pp. 1168–1180.

Carreras X., Chao I., Padro L. and Padro M. (2006). An Open-Source Suite of Language Analyzers. In Proceedingsof the 4th Int. Conference on Language Resources and Evaluation, Vol. 10, pp. 239–342.

Cozman, F. and Cohen, I. (2006). Risk of Semi-Supervised Learning. In: Chapelle, O. Scholkopf, B. and Zien, A.Semi-Supervised Learning. The MIT Press. pp 57-72.

Dempster A., Laird N. and Rubin D. (1977). Maximum Likelihood from Incomplete Data via the EM Algorithm.Journal of the Royal Statistical Society. Series B, 39(1): 1–38.

Friedman, N. (1998.) The Bayesian Structural EM algorithm. In Proc. 14th Conf. on Uncertainty in ArtificialIntelligence. Morgan Kaufmann, San Francisco, CA, pp. 129–138.

van der Gaag L. and d. Waal P. (2006). Multi-dimensional Bayesian Classifiers. In Proceedings of the ThirdEuropean Workshop in Probabilistic Graphical Models, pages 107–114.

Larranaga, P., Lozano, J.A., Pena, J.M. and Inza, I. (2005). Special Issue on Probabilistic Graphical Models forClassification. Machine Learning, 59(3).

Liu B. (2010). Sentiment Analysis and Subjectivity. In: Indurkhya N. and Damerau F.J. Handbook of NaturalLanguage Processing, Chapman & Hall, 2nd Ed.

Ortigosa-Hernandez J., Rodriguez J.D., Alzate L., Inza I. and Lozano J.A. (2010). A Semi-supervised Approach toMulti-dimensional Classification with Application to Sentiment Analysis. In Proc. of the V Simposio de Teoria yAplicaciones de Mineria de Datos (TAMIDA2010), CEDI 2010, Valencia, Spain.

Rodriguez, J.D. and Lozano, J.A. (2008). Multi-objective learning of multi-dimensional Bayesian classifiers. InProceedings of the Eighth International Conference on Hybrid Intelligent Systems, HIS 2008, Barcelona, Spain. pp.501-506.


Date post:	03-Jul-2015
Category:	Documents
Upload:	jonathan-ortigosa-hernandez
View:	368 times
Download:	0 times

Diploma of Advanced Studies

Documents