Design of a hidden web crawler based search engine

Abstract

The World Wide Web is a global information medium of interlinked hypertext documents accessed via computers connected to the internet. Most of the users rely on traditional search engines to search the information on the web. These search engines deal with the Surface Web which is a set of Web pages directly accessible through hyperlinks and ignores a large part of the Web called Hidden Web which is hidden to present-day search engines. It lies behind search forms and this part of the web containing an almost endless amount of sources providing high quality information stored in specialized databases, can be found in the depths of the WWW. World Wide Web (WWW) is broadly divided into two categories: and#61472;The surface web contains 1% of information content of the web. Search engine crawl along the web to extract and index text from HTML documents on the websites, then make this information searchable through keywords. and#61472;The hidden web contains 99% of information content of the web. Most of this information is contained in the databases and is not indexed by search engines. This means if we are searching for information from surface web only, we search through only 1% of WWW and miss 99% of it whereas 95% of hidden web is free publicly accessible information. As the Hidden web information that is hidden behind the search query forms can only be accessed by interacting with these forms, development of automated system that interacts with the search forms and extracts the hidden web content would be of great value to human users. Today, the web is crowded with home-pages and sites that sell various types of products. Since the companies selling same type of products are not at all interested to publish the products of their competitors on their site, it would be nice if there is a free web service which collaborate the marketing of the products of the competitor web sites. Infact all the information is available on the internet but buried behind search interfaces and stored inside the databases.