An Improved Focused Web Crawler based on Hybrid Similarity

doi:10.23940/ijpe.19.10.p10.26452656

Abstract

Abstract:

Web crawler is an efficient strategy for downloading data automatically from the Internet. Focused web crawler is a special kind of web crawler that is responsible for getting certain information from webpages and making them available to users. The most important problem of focused web crawler is to confirm the similarity between the target webpages and the topics. Therefore, this paper proposes an improved focused web crawler algorithm, whose similarity calculating methods derive from three aspects: anchor text, content, and structure of the webpages. This improved algorithm is called hybrid similarity. If the anchor text similarity is bigger than the threshold, the target webpages are downloaded directly; otherwise, the target webpages' similarity is analyzed by using the TF-Gini feature weighting algorithm and the improved cosine similarity algorithm. The experimental results in this paper have proven that the hybrid similarity algorithm is more effective than the traditional algorithm. The precision increases by nearly 10% compared with the traditional algorithm.

Key words: focused web crawler, TF-Gini, similarity, hybrid similarity

Songtao Shang, Huaiguang Wu, and Jiangtao Ma. An Improved Focused Web Crawler based on Hybrid Similarity [J]. Int J Performability Eng, 2019, 15(10): 2645-2656.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References 24.

1.	CNNIC, “The 43th China Internet Network Development Report,” (http://www.cnnic.net.cn/hlwfzyj/hlwxzbg/hlwtjbg/, accessed 2019
2.	J. Yue and Q. Liu. “Survey on Topic-Focused Crawlers,” Computer Engineer and Science, Vol. 37, No. 2, pp. 231-237, 2015
3.	H. Dong and F. K. Hussain, “Self-Adaptive Semantic Focused Crawler for Mining Services Information Discovery,” IEEE Transactions on Industrial Informatics, Vol. 10, No. 2, pp. 1616-1626, 2014
4.	T. J.Kim and H. J. Kim, “Machine Learning-based Topical Web Crawler: An Ensemble Approach Incorporating Me-ta-Features,” Journal of Engineering and Applied Sciences, Vol. 12, No. 8, pp. 4651-4656, 2017
5.	S. Batsakis, E. Petrakis,E. Milios, “Improving the Performance of Focused Web Crawler,” Data and Knowledge Engineering, Vol. 68, No. 10, pp. 1001-1013, 2009
6.	D. Bra and R. Post, “Information Retrieval in the World-Wide Web: Making Client-based Searching Feasible,” Computer Networks and Isdn System, Vol. 27, No. 2, pp. 183-192, 1994
7.	M. Hersovici, M. Jacovi, Y. Maarek, D. Pelleg,M. Shtalhaim, “The Shark-Search Algorithm. An Application: Tailored Web Site Mapping,” in Proceeding of the International Conference on World Wide Web, Vol.30, No. 1-7, pp. 317-326, 1998
8.	L. Zhang, Y. Qi,H. Jiang, “Application of Improved Shark-Search Algorithm in Web Crawler,” Computer Technology and Development, Vol. 27, No.8, pp. 192-194+199, 2017
9.	B. Wang, J. Wang, X. Sun,N. Wang, “An Improved Shark-Search Algorithm for Agriculture Web Search Engine,”Chemical Engineering Transactions, Vol. 51, pp. 709-804, 2016
10.	Q. Liu, B. Xiang, N. Yuan, E. Chen, H. Xiong, Y. Zheng, et al., “An Influence Propagation View of PageRank,” ACM Transactions on Knowledge Discovery from Data, Vol. 11, No. 3, pp. 1-30, 2017
11.	Y. Zhou, X. Chen,W. Wen, “Research on Focused Crawler based on Bayes Classifier,” Application Research of Computers, Vol. 26, No. 9, pp.3415-3420+3439, 2009
12.	I. S.Altingovde and O. Ulusoy, “Exploiting Interclass Rules for Focused Crawling,” IEEE Intelligent System, Vol. 29, No. 6, pp. 66-73, 2004
13.	G. Pant and P. Srinivasan, “Link Context in Classifier-Guided Topical Crawlers,” IEEE Transaction on Knowledge and Data Engineering, Vol. 18, No. 1, pp. 107-122, 2006
14.	J. Li and K. Yamaguchi, “Focused Crawling by Exploiting Anchor Text using Decision Tree,” inProceeding of the 14th International World Wide Web, pp. 1190-1191, 2005
15.	G. Pant and P. Srinivasan, “Learning to Crawl: Comparing Classification Schemes,” ACM Transactions on International Systems, Vol. 23, No. 4, pp. 430-462, 2005
16.	Q. Zhang and D. Haglin, “Semantic Similarity Between Ontologies at Different Scales,” IEEE/CAA Journal of Automatic SINICA, Vol. 3, No. 2, pp. 132-140, 2016
17.	H. Yan, F. Li,Y. Zhou, “Calculation of Sentence Similarity based on HowNet,” Computer Technology and Development, Vol. 25, No. 11, pp. 53-57, 2015
18.	P. Resnik, “Using Information Content to Evaluate Semantic Similarity in a Taxonomy,” inProceeding of the 14th International Joint Conference on Artificial Intelligence, pp. 39-41, 1995
19.	S. Sohangir and D. Wang, “Improved Sqrt-Cosine Similarity Measurement,” Journal of Big Data, Vol. 4, No. 25, pp. 1-13, 2017
20.	S. A.Fawaz and A. Dia, “Toward an Enhanced Arabic Text Classification using Cosine Similarity and Latent Semantic,” Journal of King Saud University: Computer and in-Formation Sciences, Vol. 29, No. 2, pp. 189-195, 2017
21.	K. Chen, Z. Zhang, J. Long,H. Zhang, “Turning from TF-IDF to TF-IGM for Term weighting in Text Classification,”Expert Systems with Applications, Vol. 66, pp. 245-260, 2016
22.	L. Rutkowski, L. Pietruczuk, P. Duda,M. Jaworski, “Decision Trees for Mining Data Streams based on the McDiarmid's Bound,” IEEE Transactions on Knowledge and Data Engineering, Vol. 25, No. 6, pp. 1272-1279, 2013
23.	C. J. Rijsbergen, “Information Retrieval,” Butterworths, London, 1979
24.	M. Ohasaki, P. Wang, K. Matsuda, S. Katagiri, H Watanabe,A. Ralescu, “Confusion-Matrix based Kernel Logistic Regression for Imbalanced Data Classification,” IEEE Transactions on Knowledge and Data Engineering, Vol. 29, No. 9, pp. 1806-1819, 2017

[1]	Ashima Arya and Sanjay Kumar Malik. Software Fault Prediction using K-Mean-Based Machine Learning Approach [J]. Int J Performability Eng, 2023, 19(2): 133-143.
[2]	Weihan Yang, Feng Qu, Fei He, and Wei He. Hierarchical 2D/3D Alignment Method based on Enhanced DRR and Gradient Direction Weighted Histogram [J]. Int J Performability Eng, 2023, 19(1): 1-9.
[3]	Assia Brek and Zizette Boufaida. Enhancing Information Extraction Process in Job Recommendation using Semantic Technology [J]. Int J Performability Eng, 2022, 18(5): 369-379.
[4]	Divya Mohan and Latha Ravindran Nair. Probit Regressive Tversky Indexed Rocchio Convolutive Deep Neural Learning for Legal Document Data Analytics [J]. Int J Performability Eng, 2021, 17(10): 837-847.
[5]	Zhaoyu Shou, Yiru Wen, Pan Chen, and Huibing Zhang. Personalized Knowledge Map Recommendations based on Interactive Behavior Preferences [J]. Int J Performability Eng, 2021, 17(1): 36-49.
[6]	Xueyao Gao, Xinran Dong, and Chunxiang Zhang. Model Similarity Calculation based on Self-Adaptive Global Best Harmony Search Algorithm [J]. Int J Performability Eng, 2020, 16(7): 1019-1026.
[7]	Tao Yan, In-Ho Ra, Hui Wen, Hang Xu, and Linyun Huang. Bit Allocation Algorithm based on SSIM for 3D Video Coding [J]. Int J Performability Eng, 2019, 15(7): 1813-1821.
[8]	Peng Chen, Guoyou Shi, Shuang Liu, and Miao Gao. Collision Avoidance Situation Matching with Vessel Maneuvering Actions Identification from Vessel Trajectories [J]. Int J Performability Eng, 2019, 15(6): 1499-1507.
[9]	Xiao Chen, Li Han, Meng Leng, and Xiao Pan. Similarity based on the Importance of Common Features in Random Forest [J]. Int J Performability Eng, 2019, 15(4): 1171-1180.
[10]	Tao Yan and In-Ho Ra. Improved Bit Allocation Algorithm for Multiview High Efficiency Video Coding [J]. Int J Performability Eng, 2019, 15(3): 842-849.
[11]	Shengjie Zhao and Qianyun Jiang. Short Text Classification based on Feature Extension using Information in Images [J]. Int J Performability Eng, 2019, 15(2): 667-675.
[12]	Wu Wen, Xiaobo Xue, Ya Li, Peng Gu, and Jianfeng Xu. Code Similarity Detection using AST and Textual Information [J]. Int J Performability Eng, 2019, 15(10): 2683-2691.
[13]	Songtao Shang, Yong Gan, and Huaiguang Wu. An Improved Text Sentiment Analysis Algorithm based on TF-Gini [J]. Int J Performability Eng, 2018, 14(9): 2008-2014.