Scaling real time processing using in memory computing for big data

Abstract

Big Data emerged as a field of research after data mining. Big Data has three basic properties: volume, velocity, and variety. Data Stream is a stream of data generated or passed through various sources. Data Stream when combined with velocity of big data becomes Elephant flows. The streaming data can be stored in data lakes for a limited time, after which data overflows. The mentioned limitation motivated us to use in-memory computing technique. The research deals with: (i) the optimization techniques of big data volume and velocity, (ii) the visualization techniques of big data volume and velocity, (iii) the resource aware parallel computing techniques of big data volume and velocity, and (iv) the framework for high performance computing of big data volume and velocity. newlineThe principle of parallelism is employed to accelerate stream data computing. A framework, Mille Cheval Framework, is proposed and tested. Mille Cheval Framework is a GPU based in-memory High-Performance-Computing framework for accelerated processing of big-data streams. French words Mille and Cheval translates to Thousand Horses in English language and Sahastra Ashwa in Hindi language, respectively. Streams are temporally ordered, rapidly changing, ample in volume, and infinite in nature. It is nearly impossible to store the entire data stream due to its large volume and high velocity. GPU based High-Performance Computing (HPC) framework is proposed for accelerated processing of big-data streams using the in-memory data structure. We have implemented three parallel algorithms to prove the viability of the framework. The contributions of Mille Cheval are: (i) the viability of streaming on accelerators to increase throughput, (ii) carefully chosen hash algorithms to achieve low collision rate and high randomness, and (iii) memory sketches for approximation. The objective is to leverage the power of a single node using in-memory computing and hybrid computing.