level-2 PTree, hdfP?? (Hi Doc Feq): pred=NOTpure0 applied to tfP1
8 1 38 1 3
8 1 30 0 0 . . .
1 1 2 . . . df count
<--dfP3
<--dfP0
doc1 doc2 doc3
1 1 2 . . . hdfPdoc1 doc2 doc3
Preface pTree 1 1 1 0 0 0 0
LastChpt pTree 0 0 0 0 0 1 0
Refrncs pTree 0 0 0 0 0 0 1Any of these masks can be ANDed into the Pt= , d= pTrees before
they are concatenated as above (or repetitions of the mask can be ANDED after they are concatenated).
I have put together a pBase of 75 Mother Goose Rhymes or Stories. Created a pBase of the 15 documents with 30 words (Universal Document Length, UDL) using as vocabulary, all white-space separated strings.
Latent semantic indexing (LSI) is indexing and retrieval that uses Singular value decomposition for patterns in terms and concepts in text. LSI is based on the principle that words that are used in the same contexts tend to have similar meanings. LSI feature: ability to extract conceptual content of a body of text by establishing associations between terms that occur in similar contexts. [1]LSI overcomes synonymy, polysemy which cause mismatches in info retrieval [3] and cause Boolean keyword queries to mess up.LSI performs autodoc categorization (assignment of docs to predefined categories based on similarity to conceptual content of the categories.[5] LSI uses example docs for conceptual basis categories - concepts are compared to the concepts contained in the example items, and a category (or
categories) is assigned to the docs based on similarities between concepts they contain and the concepts contained in example docs.
Mathematics of LSI (linear algebra techniques to learn the conceptual correlations in a collection of text). Construct a weighted term-document matrix, do Singular Value Decomposition on it. Use that to identify the concepts contained in the text.Term Document Matrix, A: Each (of m) terms represented by a row, each (of n) doc is rep'ed by a column, with each matrix cell, a ij, initially
representing number of times the associated term appears in the indicated document, tf ij. This matrix is usually large and very sparse.Once a term-document matrix is constructed, local and global weighting functions can be applied to it to condition the data. local: [13] Binary if
term exists in the doc TermFrequency; global weighting functions: Binary Normal GfIdf, Idf EntropyMathematics of LSI (linear algebra techniques to learn the conceptual correlations in a collection of text). Construct a weighted term-document matrix, do Singular Value Decomposition on it. Use that to identify the concepts contained in the text.Term Document Matrix, A: Each (of m) term represented by a row, each (of n) doc is rep'ed by a column, with each matrix cell, a ij, initially
representing number of times the associated term appears in the indicated document, tf ij. This matrix is usually large and very sparse.SVD basically reduces the dimensionality of the matrix to a tractable size by finding the singular values. It involves matrix operations and may not be amenable to pTree operations (i.e. horizontal methods are highly developed and my be best.We should study it though to see if we can identify a pTree based breakthrough for creating the reduction that SVD achieves.Is a new SVD program run required for every new query? or is it a one time thing? If it is one-time, there is probably little advantage in
searching for pTree speedups?If and when it is not a one-time application to the original data, pTree speedups my hold promise.Even if it is one-time, we might take the point of view that we do the SVD reduction (using standard horizontal methods) and then covert the
result to vertical pTrees for the data mining (which would be done over and over again). That pTree-ization of the end result of the SVD reduction could be organized as in the previous slides.
Here is a good paper on the subject of LSI and SVD: http://www.cob.unt.edu/itds/faculty/evengelopoulos/dsci5910/LSA_Deerwester1990.pdf
Thoughts for the future:
I am now convinced we can do LSI using pTree processing. The heart of LSI is SVD. The heart of SVD is Gaussian Elimination (which is adding a constant times a matrix row to another row - which we can do with pTrees).
We will talk more about this next Saturday and during the week.
SVD: Let X be the t by d TermFrequency (tf) matrix. It can be decomposed as T0S0D0T where T and D have ortho-normal
columns and S has only the singular values on its diagonal in descending order. Remove from T0,S0,D0, row-col of all but highest k singular values, giving T,S,D. X ~= X^ ≡ TSDT (X^ is the rank=k matrix closest to X).
We have reduced the dimension from rank(X) to k and we note, X^X^T = TS2TT and X^TX^ = DS2DT
There are three sorts of comparisons of interest: Comparing1. terms (how similar are terms, i and j?) (comparing rows)2. documents (how similar are documents i and j?) (comparing documents)3. terms and documents (how associated are term i and doc j?) (examining individual cells)
Comparing terms (how similar are terms, i and j?) (comparing rows)Dot product between two rows of X^ reflects their similarity (similar occurrence pattern across the documents).X^X^T is the square t x t symmetric matrix containing all these dot products. X^X^T = TS2TT This means the ij cell in X^X^T is the dot prod of i and j rows of TS (rows TS can be considered coords of terms).
Comparing documents (how similar are documents, i and j?) (comparing columns)Dot product of two columns of X^ reflects their similarity (extent to which two documents have a similar profile of terms).X^TX^ is the square d x d symmetric matrix containing all these dot products. X^TX^ = DS2DT This means the ij cell in X^TX^ is the dot prod of i and j columns of DS (considered coords of documents).
Comparing a term and a document (how associated are term i and document j?) (analyzing cell i,j of X^)Since X^ = TSDT cell ij is the dot product of the ith row of TS½ and the jth column of DS½
c1 Human machine interface for Lab ABC computer appsc2 A survey of user opinion of comp system response timec3 The EPS user interface management systemc4 System and human system engineering testing of EPSc5 Relation of user-perceived response time to error measmnt
m1 The generation of random, binary, unordered treesm2 The intersection graph of paths in treesm3 Graph minors IV: Widths of trees and well-quasi-orderingm4 Graph minors: A survey
X = T0S0D0T T0 D0 col-orthonormal. Approx X keeping only 1st 2 singular values and corresp cols of T,D
which are coords used to position terms and docs in 2D rep above. In this reduced model: X ~ X^ = TSDT
1.73 (m1-q)^2 1 0 1 0 0 0 0 2.00 (m2-q)^2 1 0 1 0 0 0 0 2.24 (m3-q)^2 1 0 1 0 0 0 0 2.24 (m4-q)^2 1 0 1 0 0 0 0What this tells us is that c1 is closests to q in the full space and thatthe other c documents are no closer than the m documents.
Therefore q would probably be classified as c (one voter in the 1.5nbhd) but not clearly. This shows the need for SVD or Oblique FAUST!
