Vector Space Model unique words selected as dimensions

Vector Space Model

The basic idea is to represent each document as a vector of certain weighted word frequencies. In order to do so, the following parsing and extraction steps are needed.

Ignoring case, extract all unique words from the entire set of documents.
Eliminate non-content-bearing ``stopwords'' such as ``a'', ``and'', ``the'', etc. For sample lists of stopwords, see [#!frakes:baeza-yates!#, Chapter 7].
For each document, count the number of occurrences of each word.
Using heuristic or information-theoretic criteria, eliminate non-content-bearing ``high-frequency'' and ``low-frequency'' words [#!salton:book!#].
After the above elimination, suppose unique words remain. Assign a unique identifier between and to each remaining word, and a unique identifier between and to each document.

The above steps outline a simple preprocessing scheme. In addition, one may extract word phrases such as ``New York,'' and one may reduce each word to its ``root'' or ``stem'', thus eliminating plurals, tenses, prefixes, and suffixes [#!frakes:baeza-yates!#, Chapter 8].

The above preprocessing yields the number of occurrences of word in document , say, $f_{ji}$ , and the number of documents which contain the word , say, . Using these counts, we can represent the -th document as a -dimensional vector $\mbox{$\mathbf{x}$}$ as follows. For $1 \leq j \leq w$ , set the -th component of $\mbox{$\mathbf{x}$}$ , to be the product of three terms

$\displaystyle x_{ji} = t_{ji} \cdot g_j \cdot s_i,$

where $t_{ji}$ is the term weighting component and depends only on $f_{ji}$ , while

is the global weighting component and depends on

, and

is the normalization component for $\mbox{$\mathbf{x}$}$

. Intuitively, $t_{ji}$ captures the relative importance of a word in a document, while $g_{j}$ captures the overall importance of a word in the entire set of documents. The objective of such weighting schemes is to enhance discrimination between various document vectors for better retrieval effectiveness [#!salton:buckley!#].

There are many schemes for selecting the term, global, and normalization components, see [#!kolda:thesis!#] for various possibilities. In this paper we use the popular ${\sf tfn}$ scheme known as normalized term frequency-inverse document frequency. This scheme uses $t_{ji} = f_{ji}$ , $g_j = \log ( d / d_j)$ and $s_i = \left( \sum_{j=1}^w (t_{ji}g_j )^2 \right)^{-1/2}$ . Note that this normalization implies that $\Vert$ $\mbox{$\mathbf{x}$}$ $_i \Vert = 1$ , i.e., each document vector lies on the surface of the unit sphere in . Intuitively, the effect of normalization is to retain only the proportion of words occurring in a document. This ensures that documents dealing with the same subject matter (that is, using similar words), but differing in length lead to similar document vectors.