New methodologies for improving the effectiveness of text document clustering

Abstract

Document categorization is a big challenge in information retrieval and text mining because the amount of document collection has been increasing, day by day. This thesis presents five proposed new methodologies for improving the effectiveness of text document clustering. Among the five methodologies, two for document representation models, one for the efficient centroid selection and two for similarity measures are proposed. Document representation plays a vital role in the processing of text document collection. The traditional document representation method, vector space model, uses a bag-of-words for representing a document vector by its term (or word) frequencies. In the traditional Term Frequency Inverse Document Frequency (TF-IDF) vector space model, the term weighting scheme measures the importance of a term in a document collection. This thesis puts forth two novel weighting schemes for vector space models to represent text documents, namely ii) Term-weighting scheme for document representation based on Term Frequency-Ranking of Term Frequency (TF-RTF) and ii).Term-weighting scheme for document representation based on Term Frequency-Ranking of Fuzzy logic with Semantic relationship of Terms (TF-RFST). The ranking of each term in a document is based on its frequency in the document. This ranking of terms provides its priority in the document and uses these priorities for document representation in TF-RTF document representation model. In the TFRFST document representation model, each term is represented based on its frequency and the frequency of semantic related terms for that term. Hence, the ranking of each term is based on the combined frequencies of the term and its semantic related terms with a specific weighting scheme. The fuzzy-C-Mean clustering algorithm has been used to find the semantic related term groups of each document in the document collection. newline