Web Based Content Extraction and Retrieval Using Natural Language

Abstract

The Internet has become a key communication and information medium for various organizations. It is a medium for accessing a great variety of information stored in different parts of the world. In recent years, the growth of the World Wide Web exceeded all expectations. Today, there are several billions of HTML documents, pictures and other multimedia files available via Internet and the number is still rising. Mining the Web data is one of the most challenging tasks for data mining and data management scholars because there are huge heterogeneous, less structured data available on the Web and can easily get overwhelmed with data. To solve the above mentioned problems in first phase presents, a discrete framework for running NLP tasks in a parallel fashion and crawling web documents. NLP is one of the significant features which can be utilized for text explanation and first feature extraction from request area with high computational supplies; therefore, these responsibilities can have advantage over similar architectures. Also HTML parser is used for cleaning and produces well-formed XTML. In this work Map reducer is used for extracting information from the web. However HTML parser has a few significant drawbacks, such as its static nature, its inability to render content in an aesthetically pleasing way, its well-known compatibility issues and its overall complexity. To overcome these issues DOM tree parser is used in second phase, DOM (Document Object Model) is the application programming interface of HTML and XML.In which the web page is parsed into a DOM tree .In this method, using two indicators to describe characteristics of web pages: text density and hyperlink density and fuzzy c means clustering used for measuring the similarity of contents. Also POS tagging is used to know the meaning of the content .Based on the tagging , here extracting the content of web page by using which map reducer.