An enhanced big data processing model for drill down search engine with content mining crawler

Abstract

Big Data is the current key topic in today s emerging technology. Right now, the top most popular search engines like Google, Yahoo are offering rank based search features in which the end users need to page navigate to find the exact search results. There is no Drill down (multi-level) search feature. Also the search results of existing search engines are not organized in a category based view. Thus conventional Search Engines lack Category feature as well as drill down search feature. To find the exact search results among the vast paginated search results, Drill down search would be beneficial for the end user. Category based search feature with higher level of drill down search leads to get accurate search results. newlineExisting Search Engine providers (Google, Yahoo) are implementing their own custom file system in order to store the crawler big data efficiently. Google developed its own file system called Google File System (GFS). These search engine providers store the big data in fixed-size chunks of 64 megabytes. These files are extremely rarely overwritten or shrunk and are usually appended to or read. When going to implement a drill down search engine with category based search results view, it is not possible to have a fixed chunk of files. Implementing drill down search with category based search view increases the number and size of the big data file system during the crawling process. This will lead to big problems for search engine organizations about how to store their huge amount of information and indexing such data. Traditional disk storage analytics may not be able to handle such large quantities of big data. New storage technologies are not in the near future. Next generation search engines would be category based and offer multi-level search capability. The purpose of this dissertation is to implement a big data storage reduction file system model that supports Drill down search feature with category based search results view using content mining crawler. Developing a good file system model fo