REDUCING DISTRIBUTED URLS CRAWLING TIME : A COMPARISON OF GUIDS AND IDS

Shakir, I. and Shibghatullah, Abdul Samad and Hussin, Burairah and Gede , Pramudya Ananta and Shafei, Suhailan (2014) REDUCING DISTRIBUTED URLS CRAWLING TIME : A COMPARISON OF GUIDS AND IDS. Journal of Theoretical and Applied Information Technology. pp. 1-8. ISSN 1992-8645

Preview

PDF GUIDs.pdf - Published Version Download (689kB)

Abstract

Web crawler visits websites for the purpose of indexing. The dynamic nature of today’s web makes the crawling process harder than before as web contents are continuously updated. In addition, crawling speed is important considering tsunami of big data that need to be indexed among competitive search engines. This research project is aimed to provide survey of current problems in distributed web crawlers. It then investigate the best crawling speed between dynamic globally unique identifiers (GUIDs) and the traditional static identifiers (IDs). Experiment are done by implementing Arachnot.net web crawlers to index up to 20000 locally generated URLs using both techniques. The results shown that URLs crawling time can be reduced up to 7% by using GUIDs technique instead of using IDs.

Item Type:	Article
Subjects:	T Technology > T Technology (General)
Divisions:	Faculty of Information and Communication Technology > Department of Industrial Computing
Depositing User:	Dr Abdul Samad Shibghatullah
Date Deposited:	20 Nov 2014 20:49
Last Modified:	28 May 2015 04:30
URI:	http://eprints.utem.edu.my/id/eprint/13289
Statistic Details:	View Download Statistic

Actions (login required)

View Item