Username   Password       Forgot your password?  Forgot your username? 

Performance Evaluation of Recommender Systems

Volume 13, Number 8, December 2017, pp. 1246-1256
DOI: 10.23940/ijpe.17.08.p7.12461256

Mingang Chena,b, Pan Liuc

aShanghai Key Laboratory of Computer Software Testing and Evaluating, Shanghai 201112, China
bShanghai Development Center of Computer Software Technology, Shanghai 201112, China
cShanghai Business School, Shanghai 201112, China

(Submitted on October 29, 2017; Revised on November 17, 2017; Accepted on December 1, 2017)


Recommender systems play an important role in e-commerce. This paper discusses three classical methods - offline analytics, user study, and online experiment - to evaluate the performance of recommender systems and also analyzes their application scenarios. Some performance evaluation metrics of recommender systems are reviewed and summarized from four perspectives (machine learning, information retrieval, human-computer interaction and software engineering) combined with the above three evaluation methods. These evaluation methods and evaluation metrics summarized in the paper provide the designers with guidance for the comprehensive evaluation and selection of recommended algorithms.


References: 32

      1. G. Adomavicius and A. Tuzhilin, “Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions,” IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 6, pp. 734-749, 2015
      2. T. Bogers and A. V. Bosch, “Recommending scientific articles using citeulike,” in Proceedings of the 2008 ACM conference on Recommender systems, pp. 287-290, Lausanne, Switzerland, October 2008
      3. R. Burke, B. Mobasher B, C. Williams and R. Bhaumik, “Classification features for attack detection in collaborative recommender systems,” in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 542-547, Philadelphia, PA, USA, August 2006
      4. Ò. Celma and P. Herrera, “A new approach to evaluating novel recommendations,” in Proceedings of the 2008 ACM conference on Recommender systems, pp.179-186, Lausanne, Switzerland, October 2008
      5. M. Deshpande and G.Karypis, “Item-based top-n recommendation algorithms,” ACM Transactions on Information Systems (TOIS), vol. 22, no. 1, pp. 143-177, 2003
      6. M. D. Ekstrand, J. T. Riedl and J. A. Konstan, “Collaborative Filtering Recommender Systems,” Foundations and Trends® in Human–Computer Interaction, vol. 4, no. 2, pp. 81-173, 2011
      7. T. Fawcett, “An introduction to ROC analysis,” Pattern recognition letters, vol. 27, no. 8, pp. 861-874, Amsterdam, The Netherlands, July 2006
      8. A. Gunawardana and G. Shani, “A survey of accuracy evaluation metrics of recommendation tasks," The Journal of Machine Learning Research, vol.10, pp. 2935-2962, 2009
      9. C. Hayes and P. Cunningham, “An on-line evaluation framework for recommender systems,” In Workshop on Personalization and Recommendation in E-Commerce, Malaga, Spain, May 2002
      10. J. Herlocker, J. A. Konstan and J. Riedl, “An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms,” Information retrieval, vol. 5, no. 4, pp. 287-310, 2002
      11. J. L. Herlocker, J. A. Konstan and L. G. Terveen, “Evaluating collaborative filtering recommender systems,” ACM Transactions on Information Systems (TOIS), vol. 22, no. 1, pp. 5-53, 2004
      12. K. Järvelin and J. Kekäläinen, “Cumulated gain-based evaluation of IR techniques,” ACM Transactions on Information Systems, vol. 20, no. 4, pp. 422-446, 2002
      13. P. B. Kantor, “Recommender systems handbook,” Springer, Berlin, Germany, 2011
      14. A. B. Kouki, “Recommender System Performance Evaluation and Prediction: An Information Retrieval Perspective,” Universidad Autónoma de Madrid, 2012
      15. X. N. Lam, T. Vu, T.D. Le, and A.D. Duong, “Addressing cold-start problem in recommendation systems,” in Proceedings of the 2nd international conference on Ubiquitous information management and communication, pp. 208-211, Suwon, Korea, January 2008
      16. N. Lathia, S. Hailes and L. Capra, “Evaluating collaborative filtering over time,” in Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pp. 41-42, Boston, USA, July 2009
      17. G. Linden, B. Smith, and J. York, “Amazon. com recommendations: Item-to-item collaborative filtering,” IEEE Internet Computing, vol. 7, no. 1, pp. 76-80, 2003
      18. P. Lops, M. de Gemmis and G Semeraro, “Content-based recommender systems: State of the art and trends,” Recommender Systems Handbook, Springer US, pp. 73-105, 2011
      19. T. Mahmood and F. Ricci, “Learning and adaptivity in interactive recommender systems,” in Proceedings of the ninth international conference on Electronic commerce, pp. 75-84, Minneapolis, MN, USA, August 2007
      20. S. M. McNee, J. Riedl and J. A. Konstan, “Being accurate is not enough: how accuracy metrics have hurt recommender systems,” in Proceedings of CHI'06 extended abstracts on Human factors in computing systems. ACM, pp. 1097-1101, Montréal, Québec, Canada, April 2006
      21. B. Mobasher, R. Burke, R. Bhaumik and C. Williams, “Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness,” ACM Transactions on Internet Technology (TOIT), vol. 7, no. 4, pp. 23-38, 2007
      22. F. Mourão, L. Rocha and J. A. Konstan, “Exploiting non-content preference attributes through hybrid recommendation method,” in Proceedings of the 7th ACM conference on Recommender systems, pp. 177-184, Hong Kong, China, October 2013
      23. P. Pu and L. Chen, “Trust building with explanation interfaces,” in Proceedings of the 11th international conference on Intelligent user interfaces, pp. 93-100, Sydney, Australia, January 2006
      24. J. Reilly, J. Zhang, L. McGinty, P. Pu and B. Smyth, “Evaluating compound critiquing recommenders: a real-user study,” in Proceedings of the 8th ACM conference on Electronic commerce. ACM, pp.114-123, San Diego, California, USA, June 2007
      25. A. Said, A. Bellogın, A. D. Vries and B. Kille, “Information Retrieval and User-Centric Recommender System Evaluation,” in Proceedings of the 21st Conference on User Modeling, Adaptation, and Personalization, Rome, Italy, June 2013
      26. G. Shani and A. Gunawardana, “Evaluating recommendation systems,” Recommender systems handbook. Springer US, pp. 257-297, 2011
      27. K. Swearingen and R. Sinha, “Beyond algorithms: An HCI perspective on recommender systems,” ACM SIGIR 2001 Workshop on Recommender Systems, New Orleans, USA, November 2001
      28. H. Wang, N. Wang and D. Y. Yeung, “Collaborative deep learning for recommender systems,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1235-1244, Sydney, NSW, Australia, August 2015
      29. B. Yang, T. Mei, X. S. Hua, L. Yang, S. Yang and M. Li, “Online video recommendation based on multimodal fusion and relevance feedback,” in Proceedings of the 6th ACM international conference on Image and video retrieval, pp. 73-80, 2007
      30. M. Zhang and N. Hurley, “Avoiding monotony: improving the diversity of recommendation lists,” in Proceedings of the 2008 ACM conference on Recommender systems, pp. 123-130, Lausanne, Switzerland, October 2008
      31. Y. C. Zhang, D. Ó. Séaghdha, D. Quercia and T. Jambor, “Auralist: introducing serendipity into music recommendation,” in Proceedings of the fifth ACM international conference on Web search and data mining, pp.13-22, Seattle, Washington, USA, February 2012
      32. P. Zigoris and Y. Zhang, “Bayesian adaptive user profiling with explicit & implicit feedback,” in Proceedings of the 15th ACM international conference on Information and knowledge management, pp. 397-404, Arlington, Virginia, USA, November 2006


          Click here to download the paper.

          Please note : You will need Adobe Acrobat viewer to view the full articles.Get Free Adobe Reader

          This site uses encryption for transmitting your passwords.