Enhancement in Query Optimization in Big Data Based on Various Optimization Techniques

Abstract

Today, any application must be able to store and retrieve data within a certain time limit. As a result, a well-designed query allows the user to acquire results in the timeframe requested while also establishing confidence in the relevant application. This thesis presented an improved query optimization procedure in Big Data (BD) using the ACOGA algorithm with HDFS map-reduce to overcome the complexity of query optimization. The proposed methodology is divided into two phases: BD arrangement and query optimization. The input data is pre-processed in the first phase by utilizing the SHA-512 method to get the hash value (HV) and the HDFS map-reduce function to remove repeated data. After that, characteristics including a closed frequent pattern, support, and confidence are calculated. The entropy calculation is then used to determine the support and confidence. The relevant information is grouped using the entropy-based Normalized K-Means (NKM) technique. The BD queries are collected in the second phase, and the same features are extracted. The ACO-GA technique is then used to find the optimal query for execution. Finally, similarity evaluation is carried out. The results of the experiments show that the algorithm outperformed existing algorithms.Big Data is becoming a hot research topic in the digital age, especially with the rise of data during the last decade. In a large data scenario, intellectual approaches of query optimization play a critical role in data retrieval. To provide efficient and cost-effective solutions for big data query optimization, many cloud-based distributed data processing platforms have been developed. However, due to a lack of evaluation of energy issues and query characteristics, the majority of solutions result in excessive energy consumption and low accuracy. This thesis presents an effective query optimization using the and#120590;-ANFIS load balancer and the CaM-BWO optimizer to address this issue. Big Data arrangement and query optimization are the two