Investigation on indexing algorithms for big data retrieval

Abstract

Big Data and its implications have received their own recognition in many aspects of which healthcare system emerges as one of the promising sectors The healthcare and biomedical sciences have rapidly become dataintensive as investigators are generating and using large complex high dimensional and diverse domain specific datasets Due to the existence of diversified data formats huge volume and associated uncertainty that exist among the sources of Big Data the task of data retrieval from huge sources plays a vital role Data retrieval is the process of using a query to extract data from the huge sources of data particularly a large database Indexing is considered as one of the important aspect of retrieval system Indexing structures are data structures aiming to reduce comparisons and consequently reduce the search time The proposed approach uses Hash based indexing schemes which vary according to the respective hashing function trying to reduce the search process and minimize the retrieval time Hashing is chosen because it outperforms trees when input is large for example in billions Two popular Big Data platforms such as MongoDB a NoSQL document database and Hadoop a distributed computing framework are used for data storage and fast processing times In the first phase of the work the most used indexes such as B Tree indexing and the Hash indexing are introduced for handling the Big Data In this work B Tree indexing method and the Hash algorithm that is written using java programming language are analyzed in the MongoDB database The time complexity of Hashing Algorithm is O 1 whereas the time complexity for B Tree is Olog n however when the number of records increases there is a gradual increase in the execution time for the B tree indexing. Hashing is efficient when there are more records say in billions whereas B tree works fine for limited records in distributed sharded and unsharded databases