Extracting Key-Substring-Group Features for Text Classification

Dell Zhang and Wee Sun Lee

KDD2006

The Context

Text Classification via Machine Learning (ML)

L Classifier U

Learning Predicting

TrainingDocuments

TestDocuments

Text Data

to_be_or_not_to_be…

To be, or not to be

…

tobeor

betonot

…

Some Applications

Non-Topical Text Classification Text Genre Classification

Paper? Poem? Prose? Text Authorship Classification

Washington? Adams? Jefferson?

How to exploit sub-word/super-word information?

Some Applications

Asian-Language Text Classification

How to avoid the problem of word-segmentation?

Some Applications

Spam Filtering

How to handle non-alphabetical characters etc.?

(Pampapathi et al., 2006)

Some Applications

Desktop Text Classification

How to deal with different types of files?

Learning Algorithms

Generative Naïve Bayes, Rocchio, …

Discriminative Support Vector Machine (SVM) , AdaBoost, …

For word-based text classification, discriminative methods are often superior to generative methods.

How about string-based text classification?

String-Based Text Classification Generative

Markov Chain Models (char-level) fixed order: n-gram, … variable order: PST, PPM, …

Discriminative SVM with string kernel (= taking all substrings as

features implicitly through the “kernel trick”) limitations: (1) ridge problem; (2) feature redundancy; (3)

feature selection/weighting and advanced kernels.

generative

discriminative

word-based string-based

?

The Problem

The Difficulty

The number of substrings: O(n2)

5 + 9 = 14 characters

15 + 45 = 60 substrings

d1: to_be

d2: not_to_be

Our Idea

The substrings could be partitioned into statistical equivalence groups

toto_to_bto_be

d1: to_be

d2: not_to_be

otot_ot_tot_toot_to_ot_to_bot_to_be

……

n o t _ t o _ b e d2

to _ b e

_ t o _ b e d2

o_ b e

t _ t o _ b e d2

_b e

t o _ b e d2

b e

e

a suffix tree node=

a substring group

Suffix Tree d1

d2

d1

d2

d1

d2

d1

d2

d1

d2

Substring-Groups

The substrings in an equivalence group have exactly identical distribution over the corpus, therefore such a substring-group could be taken in whole as a single feature to be used by a statistical machine learning algorithm for text classification.

Substring-Groups

The number of substring-groups: O(n) n trivial substring-groups

leaf nodes frequency = 1 not so useful to learning

at most n-1 non-trivial substring-groups internal (non-root) nodes frequency > 1 to be selected as features

Key-Substring-Groups

Select the key (salient) substring-groups by -l the minimum frequency

freq(SGv) -h the maximum frequency

freq(SGv) -b the minimum number of branches

children_num(v) -p the maximum parent-child conditional probability

freq(SGv) / freq(SGp(v)) -q the maximum suffix-link conditional probability

freq(SGv) / freq(SGs(v))

Suffix Link

“c1 c2 …ck ” “c2 …ck ” v s(v) s(v) root

Feature Extraction Algorithm

Input a set of documents the parameters

Output the key-substring-groups for each document

Time Complexity: O(n) Trick

make use of suffix links to traverse the tree

Feature Extraction Algorithm

construct the (generalized) suffix tree T

using Ukkonen’s algorithm;

count frequencies recursively;

select features recursively;

accumulate features recursively;

for each document d {

match d to T and get to the node v;

while v is not the root {

output the features associated with v;

move v to the next node via the suffix link of v;

}

}

Experiments

Parameter Tuning the number of features the cross-validation performance

Feature Weighting TFxIDF (with l2 normalization)

Learning Algorithm LibSVM linear kernel

English Text Topic Classification Dataset

Reuters-21578 Top10 (ApteMod) The home-ground of word-based text classification

Classes (1) earn; (2) acq; (3) money-fx; (4) grain; (5) crude; (6) trade;

(7) interest; (8) ship; (9) wheat; (10) corn.

Parameters -l 80 -h 8000 -b 8 -p 0.8 -q 0.8

Features 9*1013 6,055 (extracted in < 30 seconds)

English Text Topic Classification

The distribution of substring-groups ~ Zip’s law (power law)

English Text Topic Classification

The performance of linear kernel SVM with key-substring-group features on the Reuters-21578 top10 dataset.

English Text Topic Classification

Comparing the experimental results of our proposed approach and some representative existing approaches.

English Text Topic Classification

The influence of feature extraction parameters to the number of features and the text classification performance.

Chinese Text Topic Classification Dataset

TREC-5 People’s Daily News Classes

(1) Politics, Law and Society; (2) Literature and Arts; (3) Education, Science and Culture; (4) Sports; (5) Theory and Academy; (6) Economics.

Parameters -l 20 -h 8000 -b 8 -p 0.8 -q 0.8

Chinese Text Topic Classification Performance (miF)

SVM + word segmentation: 82.0% (He et al., 2000; He et al., 2003)

char-level n-gram language model: 86.7% (Peng et al. 2004)

SVM with key-substring-group features: 87.3%

Greek Text Authorship Classification Dataset

(Stamatatos et al., 2000) Classes

(1) S. Alaxiotis; (2) G. Babiniotis; (3) G. Dertilis; (4) C. Kiosse; (5) A. Liakos; (6) D. Maronitis; (7) M. Ploritis; (8) T. Tasios; (9) K. Tsoukalas; (10) G. Vokos.

Greek Text Authorship Classification Performance (accuracy)

deep natural language processing: 72% (Stamatatos et al., 2000)

char-level n-gram language model: 90% (Peng et al. 2004)

SVM with key-substring-group features: 92%

Greek Text Genre Classification Dataset

(Stamatatos et al., 2000) Classes

(1) press editorial; (2) press reportage; (3) academic prose; (4) official documents; (5) literature; (6) recipes; (7) curriculum vitae; (8) interviews; (9) planned speeches; (10) broadcast news.

Greek Text Genre Classification Performance (accuracy)

deep natural language processing: 82% (Stamatatos et al., 2000)

char-level n-gram language model: 86% (Peng et al. 2004)

SVM with key-substring-group features: 94%

Conclusion

We propose the concept of key-substring-group features and a linear-time (suffix tree based) algorithm to

extract them We show that

our method works well for some text classification tasks

clustering etc.?gene/protein sequence data?

?