A Novel Framework for Analysis of Big Data

Abstract

The world is already into the information age. The huge growth of digital data has overwhelmed the traditional systems and approaches. Big data is touching almost all aspects of our life and the data-driven discovery approach is an emerging paradigm for computing. The ever-growing data provides a tidal wave of opportunities and challenges in terms of data capture, storage, manipulation, management, analysis, knowledge extraction, security, privacy, and visualization. Though the promise of big data seems to be genuine, still a wide gap exists between its potential and realization. In this era of digitization, a huge amount of data being generated has resulted in an exponential growth of widespread cyber threats. Moreover, the ever-evolving threat landscape and rapidly growing network environments are offering additional ways for the attackers to break in. This scenario has overwhelmed the existing traditional solutions and rendered them outdated to handle such attacks. To encounter the real-world cybersecurity challenges, the security researchers are putting a lot of efforts on technologies stemming from areas like big data, and artificial intelligence to extract powerful insights. Malware is one of the most critical and challenging security threats in the Internet world. It is growing exponentially in terms of volume, variety and velocity, and thus overwhelms the traditional approaches employed for malware detection and classification. Moreover, with the advent of Internet of Things, there is a huge growth in the volume of digital devices and in such scenario, malicious binaries are bound to grow even faster making it a big data problem. The main aim of this research is to explore the various tools and techniques of big data processing and analysis, and propose a framework for analyzing big data to generate the actionable insights or intelligence. A case study of malware analysis and detection has been used in the research.