Improving data deduplication performance in cloud storage

Abstract

In the digital era, the volume of data growth is increased newlineenormously, and International Data Corporation (IDC) predicts the data newlinegrowth which would exponentially increase from 33 to 173 Zettabyte (ZB) newlinefrom 2018 to 2025. The storage and transfer of data becomes a great newlinechallenge in the digital world. IDC predicts that three-quarters of storage data newlineare considered as duplicate. This results show that the data occupies large newlinestorage space and utilize more bandwidth in the cloud environment. This issue newlinehas been solved by introducing a new technique called Data Deduplication . newlineThis is one of the important storage optimization and data compression newlineapproach, which helps to reduce the redundant data in the cloud storage. Data newlinededuplication are processed into five stages. They are Chunking, Hashing, newlineIndexing, Compression and Storage Management. This research work focuses newlinemainly on the chunking and indexing phase. Chunk level deduplication plays newlinea major role in identifying duplicates in the cloud storage. Various challenges newlinethat prevails in the deduplication approach are deduplication performance, newlineChunk size variance, Computational overhead, processing and chunk time, newlineand throughput .In chunking phase, this thesis propose a novel hybrid chunking newlinealgorithm to resolve these problems that will enhance the deduplication newlineperformance in cloud storage. The proposed algorithms, namely, Smart newlineChunker (SC), Optimus Prime Chunking (OPC) in Content Defined Chunking newline(CDC), helps to break the chunks using prime numbers. The SC algorithm newlineoperates in hybrid approaches, namely file-level and content defined levels. newlineThe first level works on less than 2 Kilobytes (KB) file size, and the newlineremaining file follow with the second level. The next algorithm, OPC, breaks newlinethe chunks without applying hash and sliding window for computation. newlineResults obtained through these algorithms minimizes the computational newlineoverhead and processing time. Then, the constant average chunk size newlinedistributes the equal chunk size variance in cloud storage. Implementation