Hadoop File System Approach for Managing Big Data

Abstract

Globally the dependency on Information Systems has increased due to which creation of Data has also increased in different sectors. One of the surveys says that we create more than 2.5 quintillion bytes of data daily. Out of which, 90% of data has been created in the last two years only, and as per the IBM survey, 80% of this data is unstructured. This collected unstructured data can comprise data used by sensors to gather climate information, information posted on different social media sites, volumes of videos and pictures uploaded on the web, records based on purchases transactions and biometric information collected in recent UIDAI project etc. newlineGovernments in developed and developing nations are investing considerable funds in developing useful tools and better technologies that could help to manage highly complex data. These complex data sets when analyzed and visualized, reveals eye-opening facts and help organizations to improve their performance. This process ultimately enhances efficiency and improves decision making. However, all this is possible only if the complex, unstructured data is stored and analyzed along with appropriate clustering or classification of data under relevant groups. Big Data has opened multiple avenues for this problem. However, Big Data has various issues that are required to be dealt with. This can range from the classification process to searching techniques, not to forget the security mechanism adopted for the Big Data. My focus is on various techniques which can be implemented in Hadoop File System to manage Big Data using various cloud service providers. I have studied two cases, one DigiLocker System and another UIDAI System. I have discussed the security issues and proposed security algorithms to give security solutions to Hadoop based UIDAI system. I have also suggested Use cases to solve security issues that might exist in DigiLocker system. newlineOrganizations have started realizing the importance of analyzing Data to track and support decisions they might take for their str