Improved weightage approaches for data dependency and classification in web documents

Abstract

In recent years, World Wide Web (WWW) has become only the largest and most important source for all kind of information. Web mining is one of the applications in data mining that is widely spread in various fields like science, business, medical, engineering, banking, etc. Hence, many researchers are working and developing various algorithms and techniques to deal with different issues in web mining. However, there are some challenging tasks in the field of web mining to extract the optimal and valuable information from the web. Term weighting plays an important role in retrieving the documents from the web. So, several term weighting methods have been proposed for assigning weight to the documents based on the occurrences of the terms. Term Frequency-Inverse Document Frequency (TF-IDF) is the most frequently used term weighting method in the field of Information Retrieval. The main objective of this thesis is to implement an efficient and effective term weighting method based on classical TF-IDF method for text classification. To improve the retrieval accuracy of the web documents, this thesis investigates two different term weighting methods namely Improved TF-IDF (ImpTF-IDF) and Co-Term Frequency (CTF) method. ImpTF-IDF is a single term based term weighting method which is an extension of classical TF formula. The idea behind this term weighting method is that the average frequency of a term in a collection of documents is considered for assigning weight to that term. The ratio between the lengths of the document to the total number of distinct terms in a corpus is found for normalization. newline