Abstract

The web is a huge and highly dynamic environment which is growing exponentially in content and developing fast in structure. No search engine can cover the whole web, but it has to focus on the most valuable pages for crawling. Many methods have been developed based on link and text analysis for retrieving the pages. In this paper, an algorithm based on link, text, logarithmic distance and probabilistic measure is presented to find the relevancy of the web pages. Here, the most relevant pages are retrieved. It has been proved experimentally that this method provides more number of relevant pages.

References

• G. Almpanidis, C. Kotropoulos, I. Pitas, September 2007. Combining text and link analysis for focused crawling—An application for vertical search engines. Information Systems, Vol 32, No: 6, pp: 886-908.
• Zhumin Chen; Jun Ma; Jingsheng Lei; Bo Yuan; Li Lian, Aug 24-27, 2007. An Improved Shark-Search Algorithm Based on Multi-information. Fourth International Conference on Fuzzy Systems and Knowledge Discovery, pp: 659 – 658.
• J. Jayanthi, Dr. K. S. Jayakumar, January 2011. An Integrated Page Ranking Algorithm for Personalized Web Search, International Journal of Computer Applications, Vol 12 – No. 11.
• Sotiris Batsakis, Euripides G. M. Petrakis, Evangelos Milios, Improving the Performance of Focused Web Crawlers, Data & Knowledge Engineering, Vol: 68, No: 10, pp: 1001-1013, October 2009.
• Blaž Novak, Survey of focused web crawling algorithms, in Proceedings of SIKDD, pp. 55-58, 2004.
• Shalin shah, Spe 2006. Implementing an Effective Web Crawler.
• Yang Yongsheng, Wang Hui , Implementation of Focused crawler, Journal of computers Vol. 6, No: 1, January 2011.
• Pant, G. , Srinivasan, P. , Menczer, F. , Crawling the Web. Web Dynamics: Adapting to Change in Content, Size, Topology and Use, edited by M. Levene and A. Poulovassilis, Springer- verlog, pp: 153-178, November 2004.
• Debashis Hati and Amritesh Kumar, An Approach for Identifying URLs Based on Division Score and Link Score in Focused Crawler, International Journal of Computer Applications, Vol. 2, no. 3, May 2010.
• A. Rungsawang, N. Angkawattanawit, Learnable topic-specific web crawler. Journal of Network and Computer Applications, Issue no:28,page no:97-114,2005
• Sandeep Sharma, Mr. Ravinder Kumar, Web-Crawling Approaches in Search Engines, June 2008.
• Michael Hersovici, Michal Jacovi, Yoelle S. Maarek, Dan Pelleg, Menachem Shtalhaim and Sigalit Ur, The shark-search algorithm. An application: tailored Web site mapping, in Proceedings of the Seventh International World Wide Web Conference on Computer Networks and ISDN Systems, Vol. 30, no. 1-7, pp. 317-326, April 1998.
• Brin, S. , & Page, L. The anatomy of a large-scale hyper textual web search engine. In Proceedings of the seventh international conference on World Wide Web (WWW), pp: 107–117, 1998.
• P. De Bra, G-J Houben, Y. Kornatzky, and R. Post, Information Retrieval in Distributed Hypertexts, in the Proceedings of RIAO'94, Intelligent Multimedia, Information Retrieval Systems and Management, New York, NY, 1994.
• Lili Yan, Wencai Du, Yingbin Wei, and Henian Chen, A Novel Heuristic Search Algorithm Based on Hyperlink and Relevance Strategy for Web Search, Advances in Intelligent and Soft Computing Volume 149, 2012, pp 97-102.
• S. Chakrabarti, M. van den Berg, and B. Dom, Focused Crawling: A New Approach for Topic-Specific Resource Discovery, In Proc. 8th WWW, 1999.
• M. Najork and J. L. Wiener. Breadth-first crawling yields high-quality pages, In Proceedings of the Tenth Conference on WorldWideWeb, Hong Kong, Elsevier Science, May 2001, pp. 114–118.
• Aggarwal C. Garawi F. Yu P. Intelligent crawling on the world wide web with arbitrary predicates, In: Proceedings of the 10th International World Wide Web Conference. Hongkong: 2001. p. 96-105.
• A. Rungsawang, N. Angkawattanawit, Learnable Crawling: An Efficient Approach to Topic-specific Web Resource Discovery, 2005.
• R. Steele, Techniques for specialized search engines, Directory of Open Access Journals, 2001.