An Optimization Technique to Integrate Structured and Unstructured Data in the Big Data Analytics

Abstract

Big data explore new opportunities to modern era for discovering new information and knowledge for better understanding and rapid decision making. Big data refers to the bang of existing information. The big data movements are obsessed by the very large amounts of high-dimensional or unstructured data which is continuously generated and stored with a much cheaper cost for further use. There is massive increase in the data size in every sector. The term big data became very popular now days due to large amount of information s are added every second. With the increase of data, the efficient and critical analysis of data is required for its better understanding and decision making. In rapid growing industries lots of data is generating from different-different sources but to get proper meaningful information there is a demand of market to take all the generated data on one place. Researcher focuses on these types of issues and designs a solution architecture with the help of open-source architecture to integrate data from heterogeneous data sources on centralized repository. The researcher used data modelling before integrate to central repository, the schema registry use to maintain the data model of structured data. To model the unstructured data, the researcher developed a custom parser using ruby programming language. The focus on this study is sources of data may be any type any technology and target the researcher use elastic search as centralize repository. The architecture will work as plug in and play, means it will support any type of sources of input. As per background studies there is solution available in one architecture that integrate and analyze the structure and unstructured data with open-source tools.An architecture is designed by using open sources tools and techniques.Top of the integrated data, the researcher developed an analytics approach using the java programming language.Using proposed architecture data analytics has been significantly improved over the other contemporary approaches.