Efficient machine learning models for handling class imbalance concept drift and dimensionality reduction in big data

Abstract

Big data revolution in various domains offers unprecedented newlineopportunities for insight and innovation. The key challenges that affect data newlinestorage and analysis are concept drift, Dimensionality Reduction, and Class newlineimbalance problems, which are significant and critically evaluated in this newlineresearch work. newlineThe accurate prediction of extreme weather occurrences is significantly newlinehindered by the class imbalance problem. Handling these issues with traditional newlinemachine learning models is found to be difficult. This laid the path to newlinedeveloping a hybrid of the AutoEncoder (AE), Synthetic Minority class newlineOversampling (SMOTE), and Extreme Learning Machine (ELM) newline(AE_SMOTE_ELM) method to perform binary classification on four newlineimbalanced climate data sets. The proposed method also mitigates the struggle newlineto learn from imbalanced datasets, as they tend to prioritize the majority class, newlineneglecting minority classes, thereby improving the predictive performance of newlinemodels on imbalanced data. This model, when compared with other state-ofthe- newlineart deep learning methods, shows better accuracy and efficiency of 90.4% newlineand 91.76%, respectively. newlineConventional machine learning methods encounter difficulties such as newlineincreased computational complexity, overfitting, and trouble interpreting the newlineresults when dealing with high-dimensional datasets. To overcome this issue, newlinethe proposed Dimensionally Reduced Optimal Skyline Evaluation (DROSE) newlinealgorithm improves stability and performance in a high-dimensional newlineenvironment. The k-dominant optimal skylines are detected with k-means newlineclustering of the DROSE newline