A Novel Approach for Low Latency Processing in Stream Data

Abstract

Stream data has been defined as the data that is generated continuously from newlinethe different data sources and has no discrete beginning or end. Processing the newlinestream data is a part of data analytics that aims at querying the continuously newlinearriving data and extracting meaningful information from the stream. Earlier newlineprocessing of stream data was using a batch analytics approach. However, realworld newlineapplication demands analytics be done instantaneously along with the newlineconsideration of dependent data. The dependent stream data usually arrive from newlinethe distributed environment, hence any delay in incoming data from different newlinedistributed sources will affect the prediction. So, there is a need to design the newlinestream processing system, which can deal with the latency in the arrival of newlinedependent data from distributed sources. The state-of-the-art stream processing newlinesystems deal with latency, which relies on a static threshold value. However, the newlinechoice of a threshold value based on a trial and error approach may impact the newlineprediction of stream processing. The experiments with the different threshold newlinevalues are presented to analyze the performance of the prediction task. The newlinecomparative analysis based on the experiment demonstrates the need for an newlineoptimum threshold value for stream processing. newlineAn effective statistical approach is proposed to overcome the problem of the newlinetraditional threshold-based approach. The proposed system uses the statistical newlinetechnique called Hazard Rate to handle the data latency. The proposed approach newlinecan forecast the probability of latency based on the input data distribution. The newlineapplications considered in this work are: Air Pollution Prediction with its affected newlinedependent parameters called PM2.5 and PM10 and Stock Market Prediction which newlineis affected by various dependent parameters like USD, OIL price, and Gold price. newlineThe proposed work is evaluated using Normalized Root Mean Square Error newline(NRMSE), which shows a significant improvement in the result when considering newlinevi newlinethe affecting parameters. However, the pe