A study on data level solutions for class imbalance problem in big data

Abstract

This research work addresses the class imbalance problem and its solutions in the context of big data. Initially, efforts are made to understand the complex and cosmic nature of big data. Subsequently, the current state of research in the field of class imbalance problem in context of big data is analysed. A detailed study and comparative analysis between two types of solutions for this problem namely, data level and algorithmic level, is performed. After perceiving their better performance, data level solutions and their different types are further explored. It was found that clustering based approaches are still not well recognized and are in their native state in this domain. Therefore, clustering based methodologies are uncov ered more in this research work. New clustering based hybrid methodologies are proposed i and further their performance is compared with standard machine learning approaches and conventional methods. Since data level techniques focus on under or over sampling of imbal anced data, therefore, partitioning based clustering methods are acquired in current research work. The research work is carried out on two types of dataset, normal imbalanced and big imbalanced datasets, both acquired from UCI repository. The datasets are pre-processed using Apache Hive to create their imbalanced versions for further experimentation