International Journal of Computer Applications
Foundation of Computer Science (FCS), NY, USA
|
Volume 1 - Issue 1 |
Published: February 2010 |
Authors: Niraj Singhal, Ashutosh Dixit, Dr. A. K. Sharma |
![]() |
Niraj Singhal, Ashutosh Dixit, Dr. A. K. Sharma . Design of a Priority Based Frequency Regulated Incremental Crawler. International Journal of Computer Applications. 1, 1 (February 2010), 42-47. DOI=10.5120/23-131
@article{ 10.5120/23-131, author = { Niraj Singhal,Ashutosh Dixit,Dr. A. K. Sharma }, title = { Design of a Priority Based Frequency Regulated Incremental Crawler }, journal = { International Journal of Computer Applications }, year = { 2010 }, volume = { 1 }, number = { 1 }, pages = { 42-47 }, doi = { 10.5120/23-131 }, publisher = { Foundation of Computer Science (FCS), NY, USA } }
%0 Journal Article %D 2010 %A Niraj Singhal %A Ashutosh Dixit %A Dr. A. K. Sharma %T Design of a Priority Based Frequency Regulated Incremental Crawler%T %J International Journal of Computer Applications %V 1 %N 1 %P 42-47 %R 10.5120/23-131 %I Foundation of Computer Science (FCS), NY, USA
The World Wide Web is a huge source of hyperlinked information contained in hypertext documents. Search engines use web crawlers to collect these documents from web for the purpose of storage and indexing. However, many of these documents contain dynamic information which gets changed on daily, weekly, monthly or yearly basis and hence we need to refresh the search engine side storage so that latest information is made available to the user. An incremental crawler visits the web repeatedly after a specific interval for updating its collection. In this paper to regulate the revisiting frequency a novel mechanism and a novel architecture for incremental crawler is being proposed.