Predictive Modeling Techniques Using Big Data

Abstract

newline ABSTRACT newlineInitial predictive analytics was based upon statistical methods but in recent times machine learning has taken over. Here the system tries to envisage a complex mathematical function depending upon large number of variables. Keeping in view the inherent structures of these systems, one can infer that this strategy can only be applicable in case of the environment, which are fundamentally governed by the mathematical functions. But in the natural environment, most of the times the things are not purely based upon mathematical functions and there is some contribution of natural and human elements as well. Therefore we are of the view that pure machine learning methods are not adequate for designing the predictive systems. They have to be augmented with some other mechanisms to take care of human and natural elements. The work proposed in this thesis is an effort in this direction. newlineThe domain undertaken for the purpose of predictive analytics in the proposed work is price prediction in the Indian Stock Market. The reason for choosing this domain are: availability of data in public domain, ease of verification of input data, ease of verification of results, major characteristics of Big Data are complied. newlineWhile taking the inference from the historical data, the working pattern of all the papers is somewhat similar to that of the time series prediction wherein the basic underline philosophy is that the trend will continue. However, the stock market prediction is not a matter of mere time series trend. Moreover, prediction accuracy has not been that good in any case as it varies from 60% to 85%. An error to the tune of 15% to 20% is quite huge and can lead to mega loss in the financial market. All these conventional machine learning mechanisms suffer from the usual drawbacks of opacity and overfitting. Moreover, the random fluctuations in the stock price data which is a very common element, in the stock prices, is a big hindrance to the proper convergence. Most of the papers go for the few promine