Scalable Metadata Storage and Retrieval Techniques for Very Large Distributed Storage Systems

Abstract

The rapid proliferation of global digital data has reached unprecedented levels, with a projected increase to a staggering 175 ZB by 2025, according to forecasts by the International Data Corporation (IDC). This exponential data growth presents immense challenges for storage and processing capabilities. Over the past three decades, parallel and distributed techniques have emerged as viable solutions to address the storage and retrieval demands of this ever-expanding data landscape. As digital data surges, large-scale storage systems generate massive volumes of metadata. Metadata is a crucial component of storage and file systems that contains vital information about organizational structure, block identification numbers, physical block locations, and access permissions. Effectively managing metadata of exabyte or zettabyte storage capacities has become critical for parallel and distributed storage techniques. This research delves into the challenges posed by the continuous growth of global digital data. It explores the development of parallel and distributed metadata management techniques for efficient storage and retrieval. It investigates the significance of metadata in storage systems and underscores the importance of scalable distributed metadata management to handle these colossal data volumes. The provisioning of an efficient ultra-large scalable distributed storage system for expanding cloud applications has been a challenging endeavour for both academia and industry. In such an ultra-large-scale storage system, data are distributed across multiple storage nodes to enhance performance, scalability, and availability. Access to this distributed data is facilitated through metadata, maintained by multiple metadata servers. The metadata carries crucial information about the physical address of data and access privileges. The efficiency of a storage system heavily relies on effective metadata management. This research presents a comprehensive systematic literature analysis of metadata management techniques