Evaluation of recommender systems: A multi-criteria decision making approach

Document Type: Research Paper


1 Faculty of Management, University of Tehran

2 Department of Business Administration, Technical University of Ostrava

3 School of Engineering, University of Tehran

4 University of Tehran


The evaluation and selection of recommender systems is a difficult decision making process. This difficulty is partially due to the large diversity of published evaluation criteria in addition to lack of standardized methods of evaluation. As such, a systematic methodology is needed that explicitly considers multiple, possibly conflicting metrics and assists decision makers to evaluate and find the best recommender system among a given set of alternatives. This paper introduces Multi-Criteria Decision Making (MCDM) approach for evaluation of recommender systems. In particular, this paper proposes the use of Data Envelopment Analysis (DEA) approach, as a sub-category of MCDM, in order to solve this problem. Various DEA models are introduced and their applicability are illustrated. A real case of evaluation of recommender systems is used to demonstrate the approach.


Main Subjects

Article Title [Persian]

ارزیابی سیستم های پیشنهاددهنده با بهره گیری از تکنیک های تصمیم گیری چند معیاره

Authors [Persian]

  • بابک سهرابی 1
  • مهدی طلوع 2
  • علی معینی 3
  • سروش نالچیگر 4
1 دانشکدة مدیریت دانشگاه تهران، ایران
2 گروه مدیریت بازرگانی، دانشگاه فنی استراوا، جمهوری چک
3 دانشکدة فنی مهندسی، دانشگاه تهران، ایران
4 دانشکدة مدیریت دانشگاه تهران، ایران
Abstract [Persian]

ارزیابی و انتخاب سیستم‌های پیشنهاددهنده فرایند تصمیم‌گیری پیچیده و دشواری است. بخشی از این پیچیدگی به دلیل وجود معیارهای ارزیابی متنوع و متعددی است که در این حوزه وجود دارد. به‌علاوه، فقدان روشی استاندارد در ارزیابی این سیستم‌ها، پیچیدگی ارزیابی را بیشتر می‌کند. برای حل این مسئله، نیاز به روش‌شناسی سیستماتیک است که معیارهای چندگانه و در صورت لزوم متضاد را در نظر داشته باشد و به تصمیم‌گیرندگان این امکان را بدهد که بهترین سیستم پیشنهاددهنده را از بین مجموعه‌ای از گزینه‌ها انتخاب کند. این مقاله، تکنیک‌های تصمیم‌گیری چند معیاره را برای حل این مسئله معرفی می‌کند و به‌طور مشخص از مدل‌های تحلیل پوششی داده‌ها، به‌عنوان زیرشاخه‌ای از تکنیک‌های تصمیم‌گیری چند معیاره استفاده می‌کند. این مقاله کاربرد مدل‌های مختلفی از تحلیل پوششی داده‌ها را برای ارزیابی سیستم‌های پیشنهاددهنده نشان می‌دهد. داده‌های مورد نیاز، از مطالعة موردی واقعی موجود در متون موضوع استخراج شده است.

Keywords [Persian]

  • ارزیابی
  • تحلیل پوششی داده‌ها
  • تصمیم‌گیری چندمعیاره
  • سیستم‌های پیشنهاددهنده
Adomavicius, G. & Tuzhilin, A. (2005). “Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions”. Knowledge and Data Engineering, IEEE Transactions on, 17(6), 734-749.
Amin, G. R. (2009). “Comments on finding the most efficient DMUs in DEA: An improved integrated model”. Computers & Industrial Engineering, 56(4), 1701-1702.
Amin, G. R. & Toloo, M. (2007). “Finding the most efficient DMUs in DEA: An improved integrated model”. Computers & Industrial Engineering, 52(1), 71-77.
Balabanović, M. & Shoham, Y. (1997). “Fab: content-based, collaborative recommendation”. Communications of the ACM, 40(3), 66-72.
Banker, R. D.; Charnes, A. & Cooper, W. W. (1984). “Some models for estimating technical and scale inefficiencies in data envelopment analysis”. Management science, 30(9), 1078-1092.
Bobadilla, J.; Ortega, F.; Hernando, A. & Gutiérrez, A. (2013). “Recommender systems survey”. Knowledge-Based Systems, 46, 109-132.
Breese, J. S.; Heckerman, D. & Kadie, C. (1998). “Empirical analysis of predictive algorithms for collaborative filtering”. In Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence (pp. 43-52). Morgan Kaufmann Publishers Inc.
Charnes, A.; Cooper, W. W. & Rhodes, E. (1978). “Measuring the efficiency of decision making units”. European journal of operational research, 2(6), 429-444.
Chen, M. C. (2007). “Ranking discovered rules from data mining with multiple criteria by data envelopment analysis”. Expert Systems with Applications, 33(4), 1110-1116.
Cook, W. D. & Seiford, L. M. (2009). “Data envelopment analysis (DEA)–Thirty years on”. European Journal of Operational Research, 192(1), 1-17.
Doyle, J. & Green, R. (1993). “Data envelopment analysis and multiple criteria decision making”. Omega, 21(6), 713-715.
Ertay, T.; Ruan, D. & Tuzkaya, U. R. (2006). “Integrating data envelopment analysis and analytic hierarchy for the facility layout design in manufacturing systems”. Information Sciences, 176(3), 237-262.
Fouss, F. & Saerens, M. (2008). “Evaluating performance of recommender systems: An experimental comparison”. In Web Intelligence and 

Intelligent Agent Technology, 2008. WI-IAT'08. IEEE/WIC/ACM International Conference on (Vol. 1, pp. 735-738). IEEE.
Fouss, F.; Pirotte, A.; Renders, J. M. & Saerens, M. (2007). “Random-walk computation of similarities between nodes of a graph with application to collaborative recommendation”. Knowledge and Data Engineering, IEEE Transactions on, 19(3), 355-369.
Gattoufi, S.; Oral, M. & Reisman, A. (2004). “A taxonomy for data envelopment analysis”. Socio-Economic Planning Sciences, 38(2), 141-158.
Gunawardana, A. & Shani, G. (2009). “A survey of accuracy evaluation metrics of recommendation tasks”. The Journal of Machine Learning Research, 10, 2935-2962.
Herlocker, J. L.; Konstan, J. A.; Terveen, L. G. & Riedl, J. T. (2004). “Evaluating collaborative filtering recommender systems”. ACM Transactions on Information Systems (TOIS), 22(1), 5-53.
Hernández del Olmo, F. & Gaudioso, E. (2008). “Evaluation of recommender systems: A new approach”. Expert Systems with Applications, 35(3), 790-804.
Konstan, J. A. & Riedl, J. (2012). “Recommender systems: from algorithms to user experience”. User Modeling and User-Adapted Interaction, 22(1-2), 101-123.
Liu, J. S.; Lu, L. Y.; Lu, W. M. & Lin, B. J. (2013). “Data envelopment analysis 1978–2010: A citation-based literature survey”. Omega, 41(1), 3-15.
Lootsma, F. A. (Ed.). (1999). Multi-criteria decision analysis via ratio and difference judgment. Kluwer Academic Publishers, Applied Optimization Series (Vol. 29). Springer.
Mannino, M.; Hong, S. N. & Choi, I. J. (2008). “Efficiency evaluation of data warehouse operations”. Decision Support Systems, 44(4), 883-898.
McLaughlin, M. R. & Herlocker, J. L. (2004). “A collaborative filtering algorithm and evaluation metric that accurately model the user experience”. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 329-336). ACM.
McNee, S. M.; Riedl, J. & Konstan, J. A. (2006). “Being accurate is not enough: how accuracy metrics have hurt recommender systems”. In CHI'06 extended abstracts on Human factors in computing systems (pp. 1097-1101). ACM.
Paradi, J. C. & Zhu, H. (2013). “A survey on bank branch efficiency and 

performance research with data envelopment analysis”. Omega, 41(1), 61-79.
Pazzani, M. J. & Billsus, D. (2007). Content-based recommendation systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web Methods and Strategies of Web Personalization, vol. 4321, pp. 325–341. Springer, Berlin/Heidelberg.
Resnick, P.; Iacovou, N.; Suchak, M.; Bergstrom, P. & Riedl, J. (1994). “GroupLens: an open architecture for collaborative filtering of netnews”. In Proceedings of the 1994 ACM conference on Computer supported cooperative work (pp. 175-186). ACM.
Said, A.; Tikk, D.; Shi, Y.; Larson, M.; Stumpf, K. & Cremonesi, P. (2012). Recommender systems evaluation: A 3D benchmark. In ACM RecSys 2012 Workshop on Recommendation Utility Evaluation: Beyond RMSE, Dublin, Ireland (pp. 21-23).
Sowlati, T.; Paradi, J. C. & Suld, C. (2005). “Information systems project prioritization using data envelopment analysis”. Mathematical and Computer Modelling, 41(11), 1279-1298.
Toloo, M. (2012). “On finding the most BCC-efficient DMU: A new integrated MIP–DEA model”. Applied Mathematical Modelling, 36(11), 5515-5520.
Toloo, M. & Nalchigar, S. (2009). “A new integrated DEA model for finding most BCC-efficient DMU”. Applied Mathematical Modelling, 33(1), 597-604.
Toloo, M. & Nalchigar, S. (2011). “A new DEA method for supplier selection in presence of both cardinal and ordinal data”. Expert Systems with Applications, 38(12), 14726-14731.
Toloo, M.; Sohrabi, B. & Nalchigar, S. (2009). “A new method for ranking discovered rules from data mining by DEA”. Expert Systems with Applications, 36(4), 8503-8508.
Wu, W.; He, L. & Yang, J. (2012). “Evaluating recommender systems”. In Digital Information Management (ICDIM), 2012 Seventh International Conference on (pp. 56-61). IEEE.
Zaier, Z.; Godin, R. & Faucher, L. (2008). “Evaluating recommender systems”. In Automated solutions for Cross Media Content and Multi-channel Distribution, 2008. AXMEDIS'08. International Conference on (pp. 211-217). IEEE.
Zeleny, M. (1982). Multiple Criteria Decision Making (Vol. 25). J. L. Cochrane (Ed.). New York: McGraw-Hill.