Assessment of computational cost in big data by implementing mapreduce strategy

Abstract

Big data is a key component of most contemporary technologies, newlineincluding social media, smart cities, and the internet of things (IoT). Class newlineoverlap and class imbalance are two data issues that arise when large data is newlineused in practical applications. Most conventional classifiers are trapped in the newlinelocal optimum problem when working with huge datasets. As a result, newlineresearch into novel approaches to handling massive data volumes is required. newlineThe issue has been addressed with a number of methods. The fast expansion newlineof data sources poses a challenge to the continued usefulness of many newlineestablished techniques. Class imbalance concerns have shown considerable newlinepromise for methods like oversampling and under-sampling. newline The Synthetic Minority Oversampling Technique (SMOTE), which newlinegenerates synthetic samples for the minority class in constructing a balanced newlinedataset, has produced the greatest results of any of these strategies. The newlineproblem is that their practical application is limited to situations where there newlineare tens of thousands or fewer of each. A parallel mode method combining newlineSMOTE and the MapReduce strategy was put forth in this study to address newlinethe aforementioned issue. This method divides the algorithm s operation newlineamong a number of processing nodes. The first step is to divide the data into newlinevarious blocks using a mapping function. Each mapping block is then newlinesubjected to a pre-processing step that uses a hybrid SMOTE method to solve newlinethe class unbalanced problem. A decision tree model would be built for each newlinemap block. The decision tree building components would then be integrated newlineto produce a categorization model. newline