Please use this identifier to cite or link to this item:
Title: Performance Optimization of Web crawler
Researcher: Goel K.
Guide(s): Prasad J.S., Hilal S.
Keywords: Engineering and Technology,Computer Science,Computer Science Information Systems
University: MVN University,Palwal
Completed Date: 2018
Abstract: newlineOptimizing performance of crawler is the requirement in the area of crawling and searching. newlineAs web contains a huge volume of information and it is a challenging task to extract information, there is a need of the user interface for a user to extract information from the web. A Search engine is the user interface to extract information from web and crawler is a tool used by search engine to create a database. Search results generated by the search engine are derived from the results given by crawler. If crawler provides better results, it will give relevant results in searching also. Finding useful information from the web has inherent issues of page freshness, crawling multimedia contents, and duplicate contents. Crawling and indexing similar contents and URLs implies wastage of resources. Crawler gives duplicate results because of the bad crawling algorithm, poor quality ranking algorithm. newlineThis thesis contributes to the area of optimizing crawler s performance by removing duplicate URLs. Removing duplicate URLs at crawling level improves crawler s efficiency in terms of time and space also. newlineSix popular search engines are at first analyzed for identifying the presence of redundancy in the content over 44 categories of user search interest. Further, the new algorithm based on the URL normalization in query parameter and categorization is developed. To test the effectiveness of the proposed algorithm, a proposed crawler has been developed. To compare and analyze the results another crawler i.e. base crawler based on breadth-first search has been developed. The results of proposed crawler are compared with results of the base crawler, and encouraging performance improvement in terms of crawling time, space, search engine execution time and reduction in the number of duplicates has been observed. The percentage improvement of crawling time between base crawler and proposed crawler varies from 0.086% to 17.44%. The proposed algorithm crawls a URL in a particular category which yields more relevant results
Appears in Departments:Computer Science Engineering

Files in This Item:
File Description SizeFormat 
01_title page.pdfAttached File17.85 kBAdobe PDFView/Open
02_certificate.pdf76.72 kBAdobe PDFView/Open
03_abstract.pdf81.47 kBAdobe PDFView/Open
04_acknowledgement.pdf81.79 kBAdobe PDFView/Open
05_tableofcontents.pdf113.62 kBAdobe PDFView/Open
06_listof tables.pdf10.82 kBAdobe PDFView/Open
07_listoffigures.pdf15.44 kBAdobe PDFView/Open
08_listofabbreviations.pdf7.58 kBAdobe PDFView/Open
09_chapter1.pdf126.71 kBAdobe PDFView/Open
10_chapter2.pdf557.33 kBAdobe PDFView/Open
11_chapter3.pdf282.58 kBAdobe PDFView/Open
12_chapter4.pdf868.59 kBAdobe PDFView/Open
13_chapter5.pdf344.54 kBAdobe PDFView/Open
14_chapter6.pdf209.6 kBAdobe PDFView/Open
15_chapter7.pdf9.49 kBAdobe PDFView/Open
17_references.pdf181.26 kBAdobe PDFView/Open
18_listofpublications.pdf177.4 kBAdobe PDFView/Open
19_antiplagiarismcertificate.pdf83.56 kBAdobe PDFView/Open

Items in Shodhganga are protected by copyright, with all rights reserved, unless otherwise indicated.

Altmetric Badge: