• Text MiningText MiningText ClassificationText ClassificationText ClusteringText Clustering

2004. 11. 2004. 11.

ContentsContents

IntroductionIntroduction

Related TechnologiesRelated Technologies Feature selectionFeature selection Text classificationText classification Text clusteringText clustering

Introduction (1/2)Introduction (1/2)

Text classification (categorization)Text classification (categorization) Sorting new items into existing Sorting new items into existing

structuresstructures general topic hierarchiesgeneral topic hierarchies email foldersemail folders general file systemgeneral file system

Information filtering/pushInformation filtering/push Mail filtering(spam vs.not)Mail filtering(spam vs.not) Customized Push service Customized Push service

Categorization

Document Categorization

Clustering(topic discovery)

Document Clustering

Introduction (2/2)Introduction (2/2)

Difference with data miningDifference with data mining Analyze both raw data and textual Analyze both raw data and textual

information at the same timeinformation at the same time Require complicated feature selection Require complicated feature selection

technologiestechnologies May include linguistic, lexical, and May include linguistic, lexical, and

contextual techniquescontextual techniques

Classifier

unknowndocuments

sampledocuments

1. learning1. learning 2. classification2. classification

A. fun B. business C.private

Text classification Text classification 예예 : e-: e-mailmail

ProcessProcess Construction of vocabularyConstruction of vocabulary

optionaloptional ExtractionExtraction

Keep incoming documents in the systemKeep incoming documents in the system ParsingParsing

StemmingStemming Vector model, bag-of-wordsVector model, bag-of-words

Feature selection (reduction)Feature selection (reduction) LearningLearning

Off-line process: Build model parametersOff-line process: Build model parameters CategorizationCategorization

On-line processOn-line process Re-learningRe-learning

On-line processOn-line process

ExtractionExtraction

DatabasesDatabases DocumentsDocuments

Incoming/Training/Categorized documentsIncoming/Training/Categorized documents DictionaryDictionary

StopwordsStopwords 조사조사 , , 어미어미

……

StemmingStemming

Table LookupTable Lookup 검색어와 관련된 모든 어간을 테이블 기록검색어와 관련된 모든 어간을 테이블 기록

N-gram stemmerN-gram stemmer 접사 제거접사 제거

어근 추출을 위해 접두사어근 추출을 위해 접두사 , , 접미사접미사 , , 어미 등을 제거어미 등을 제거 Porter Porter 알고리즘알고리즘

‘‘ies’ ies’ ‘y’ ‘y’ ‘‘es’ es’ ‘s’ ‘s’ ‘‘s’ s’ NULL NULL

ExtractionExtraction

색인어색인어 주로 명사주로 명사 (( 구구 )) 그외 형용사그외 형용사 (( 구구 ), ), 동사동사 (( 구구 ))

색인방법색인방법 통계적 기법통계적 기법

단어 출현 통계량 단어 출현 통계량 (term frequency) (term frequency) 사용사용 언어학적 기법언어학적 기법

형태소 분석형태소 분석 , , 구문 분석구문 분석

ExtractionExtraction

한글 문서의 특징한글 문서의 특징 띄어쓰기가 자유로움띄어쓰기가 자유로움

복합명사 분해 문제복합명사 분해 문제 대학생선교회 대학생선교회 대학 대학 ++ 생선생선 ++ 교회 교회 or or 대학생대학생 ++ 선교회선교회

용언의 변화용언의 변화 , , 축약축약 음절 분석 필요음절 분석 필요

불용어 처리불용어 처리 맞춤법 처리 맞춤법 처리

색인어로 적당한 한글의 격틀색인어로 적당한 한글의 격틀 명사명사 : ex) : ex) 정보 정보 명사 명사 + + 명사명사 : ex) : ex) 정보검색정보검색 명사 명사 + + 조사 조사 + + 명사명사 : ex) : ex) 정보의 검색정보의 검색

Feature Selection Feature Selection (reduction)(reduction)

: Curse of dimensionality: Curse of dimensionality Removal of stopwordsRemoval of stopwords Feature SelectionFeature Selection

Zipf’s LawZipf’s Law DF (document frequency)-basedDF (document frequency)-based xx22 Statistics-based Statistics-based Mutual InformationMutual Information ……

Feature Selection Feature Selection (reduction)(reduction)

: Curse of dimensionality: Curse of dimensionality StopwordsStopwords

Feature Selection Feature Selection (reduction)(reduction)

: Curse of dimensionality: Curse of dimensionality Zipf’s LawZipf’s Law

Feature Selection Feature Selection (reduction)(reduction)

: Curse of dimensionality: Curse of dimensionality xx22 statistics-based statistics-based

)()()()(

)(),(2

srsqqprp

qrpsntc

CC /C/C

tt pp rr

/t/t qq ss

Parsing: representing Parsing: representing documentsdocuments

Vector Representation - term frequency - document frequency - weights

Classification ModelClassification Model: machine learning approach: machine learning approach

Learner

Classifier

ObservedTraining

documents

Unknown documents

Model(hypothesis)Parameters

Categorizeddocuments

Classification ModelClassification Model: machine learning approach: machine learning approach

Naïve Bayesian ClassificationNaïve Bayesian Classification

Nearest Neighbor ClassificationNearest Neighbor Classification

dwccccwqcqclassifier )Pr()|Pr(maxarg)|Pr(maxarg)(

q

Classification Classification 예예

환자본인의 유전자를 이용 , 배아를 만든 후 이를 이용해 실험실에서 건강한 세포를 배양시켜 환자에 다시 주입하는 이른바 치료복제법이 실험을 통해 입증되기는 이번이 세계최초라고 연구진은 주장했는데 이 방법은 주입된 세포에 대한 인체의 거부 반응이 없어 그동안 의학계의 관심을 끌어왔다

환자 본인 환자본인 유전자 이용 배아 이용 실험실 건강 세포 배양 환자 주입 치료복제법 실험 입증 이번 세계 최초 세계최초 연구진 주장 방법 주입 세포 인체 거부 반응 의학계 관심

수의학 0.191149 의학 , 생명공학 , 약학 0.134847치의학 0.114641 생물 , 미생물 0.109833성 0.099062 질병 , 증상 , 죽음 0.084554. . .

Learning the text Learning the text classifierclassifier

Before system startsBefore system starts Define category (class, topic) Define category (class, topic) Learning representative documents for Learning representative documents for

each defined categoryeach defined category During system operationDuring system operation

Incremental learning for each classifierIncremental learning for each classifier Define new categories by clustering Define new categories by clustering

uncategorized documentsuncategorized documents

Machine Learning based approach

(Basic architecture)

Machine Learning MethodsMachine Learning Methods Similarity-basedSimilarity-based

K-Nearest NeighborK-Nearest Neighbor Decision TreesDecision Trees Statistical Learning: Statistical Learning:

Naïve BayesNaïve Bayes Bayes NetsBayes Nets

Support Vector MachinesSupport Vector Machines Artificial Neural NetworksArtificial Neural Networks . . . . . . OthersOthers

Hierarchical classificationHierarchical classification Expectation-Maximization techniqueExpectation-Maximization technique Variants of BoostingVariants of Boosting Active learningActive learning

Naïve Bayes Text ClassifierNaïve Bayes Text Classifier

||

1)|Pr()Pr(maxarg

)|Pr()Pr(maxarg

)Pr(

)|Pr()Pr(maxarg

)|Pr(maxarg)(

i

j

j

j

jNB

d

k jikjCc

jijCc

i

jijCc

ijCci

cwc

cdc

d

cdc

dcdf

||

)|Pr()Pr(

||

1^

tl

D

i ijjc

D

dcc

tl

j

Vw c

ikc

jikcwVwtf

wtfcw

j

j

jik||)(

1)()|Pr(|

^

Classification model of NB classifiers

- Class prior estimate - Word probability estimate

1,0)|Pr( ij dc

Class prior estimate Word probability estimate

Uses of Clustering in IRUses of Clustering in IR Clustering as RepresentationClustering as Representation ( (abstractionabstraction))

Clustering is unsupervised learningClustering is unsupervised learning of the underlying structure, classes of the underlying structure, classes

Clustering can be used to transform representationsClustering can be used to transform representations documents are represented by class membership as well as individual termsdocuments are represented by class membership as well as individual terms

Can be viewed as Can be viewed as dimensionality reductiondimensionality reduction especially term clustering (e.g., word variant clusters)especially term clustering (e.g., word variant clusters)

Clustering for BrowsingClustering for Browsing Clustering has been proposed as a technique for organizing documents Clustering has been proposed as a technique for organizing documents

for browsing, interaction and visualizationfor browsing, interaction and visualization constructing hypertext constructing hypertext clustering the results of searches clustering the results of searches iterative clustering of the collection (e.g, Scatter/Gather) iterative clustering of the collection (e.g, Scatter/Gather) clustering the webclustering the web

Also has been used to group terms for browsingAlso has been used to group terms for browsing automatic thesauri automatic thesauri topic summariestopic summaries

Clustering for topic discoveryClustering for topic discovery

IntroductionIntroduction

Text clusteringText clustering Summarization of large text dataSummarization of large text data Discovering new categoriesDiscovering new categories

Abstraction of a set of Abstraction of a set of documentsdocuments

Document within a cluster “relevant”

Information Retrieval Information Retrieval (browsing)(browsing)

Clustering of Query Results Clustering of Query Results (Scatter/Gather)(Scatter/Gather)

Scatter&Gather

Clustering for Topic Clustering for Topic discovery discovery (Evolution of topic (Evolution of topic

hierarchy)hierarchy)

“Movie &Film”

. . .

A

“Movie &Film”

“Plays”

“Film Festivals”

. . .

“ScreenPlays”

“Movie” “Genres”

“Film Festival”

. . .

“Horror”“ScienceFiction”

BReorganizationNew topic discovery

Concept drift

Change of

viewpoint

Clustering AlgorithmClustering Algorithm

Two general methodologiesTwo general methodologies HierarchicalHierarchical

pairs of items or clusters are successively linked pairs of items or clusters are successively linked to produce larger clusters (to produce larger clusters (agglomerativeagglomerative))

or start with the whole set as a cluster and or start with the whole set as a cluster and successively divide sets into smaller partitions successively divide sets into smaller partitions ((divisivedivisive))

Non-hierarchicalNon-hierarchical - divide a set of N items - divide a set of N items into M clusters (top-down)into M clusters (top-down)

GraphGraph partitioningpartitioning

Clusters

Supervised Clustering (for Topic Discovery)

Clustering

Document

Collection

A’ B’ C’ D’ E’

Human Knowledge

Topics (categories)