Complex Query JOIN Optimization in Parallel Distributed Environment

Abstract

Query optimization is a deep rooted area of research. This research area has gained prominence due to increasing demand of complex queries and large databases. Among the different query operators of SELECT-PROJECT-JOIN (SPJ) queries, JOIN is the predominant operator. Usually it is the key operator in query optimization as it gives the final result in JOIN operation on multiple relations or tables. newlineAs a complex query is composed with many JOIN operations and JOIN operation itself is time consuming, we need to minimize the time required for JOIN operation. Our main research focus is to reduce the execution time and network cost (in terms of amount of data transfer on network) of JOIN operation of a complex query on large databases. newlineOur research used parallel distributed environment for query optimization. The elementary JOIN types considered for research are SEMI JOIN, BLOOM JOIN, POSITIONAL ENCODED REDUCTION FILTER (PERF) JOIN and DISTINCT ENCODED REDUCTION FILTER (DERF) JOIN. There are certain issues to be solved in above JOIN operations such as transmission of uncompressed data, collision problem, transmission of duplicate values, extra time to remove duplicate values etc. Our research focused on these research issues of above mentioned JOINs. newlineIn distributed environment, different machines have different binary data representations such as little endian or big endian. So while transferring data in compressed format such as byte array, the machines may read data differently. It is called as binary data incompatibility problem. While transmitting compressed data on network, there is need to provide security newlineto that network data. So security and incompatibility of binary data representation are other issues we considered during our research. newline