Evaluation of recommender systems: A multi-criteria decision making approach

Document Type : Research Paper

Authors

1 Faculty of Management, University of Tehran

2 Department of Business Administration, Technical University of Ostrava

3 School of Engineering, University of Tehran

4 University of Tehran

Abstract

The evaluation and selection of recommender systems is a difficult decision making process. This difficulty is partially due to the large diversity of published evaluation criteria in addition to lack of standardized methods of evaluation. As such, a systematic methodology is needed that explicitly considers multiple, possibly conflicting metrics and assists decision makers to evaluate and find the best recommender system among a given set of alternatives. This paper introduces Multi-Criteria Decision Making (MCDM) approach for evaluation of recommender systems. In particular, this paper proposes the use of Data Envelopment Analysis (DEA) approach, as a sub-category of MCDM, in order to solve this problem. Various DEA models are introduced and their applicability are illustrated. A real case of evaluation of recommender systems is used to demonstrate the approach.

Keywords

Main Subjects

References

Adomavicius, G. & Tuzhilin, A. (2005). “Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions”. Knowledge and Data Engineering, IEEE Transactions on, 17(6), 734-749.
Amin, G. R. (2009). “Comments on finding the most efficient DMUs in DEA: An improved integrated model”. Computers & Industrial Engineering, 56(4), 1701-1702.
Amin, G. R. & Toloo, M. (2007). “Finding the most efficient DMUs in DEA: An improved integrated model”. Computers & Industrial Engineering, 52(1), 71-77.
Balabanović, M. & Shoham, Y. (1997). “Fab: content-based, collaborative recommendation”. Communications of the ACM, 40(3), 66-72.
Banker, R. D.; Charnes, A. & Cooper, W. W. (1984). “Some models for estimating technical and scale inefficiencies in data envelopment analysis”. Management science, 30(9), 1078-1092.
Bobadilla, J.; Ortega, F.; Hernando, A. & Gutiérrez, A. (2013). “Recommender systems survey”. Knowledge-Based Systems, 46, 109-132.
Breese, J. S.; Heckerman, D. & Kadie, C. (1998). “Empirical analysis of predictive algorithms for collaborative filtering”. In Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence (pp. 43-52). Morgan Kaufmann Publishers Inc.
Charnes, A.; Cooper, W. W. & Rhodes, E. (1978). “Measuring the efficiency of decision making units”. European journal of operational research, 2(6), 429-444.
Chen, M. C. (2007). “Ranking discovered rules from data mining with multiple criteria by data envelopment analysis”. Expert Systems with Applications, 33(4), 1110-1116.
Cook, W. D. & Seiford, L. M. (2009). “Data envelopment analysis (DEA)–Thirty years on”. European Journal of Operational Research, 192(1), 1-17.
Doyle, J. & Green, R. (1993). “Data envelopment analysis and multiple criteria decision making”. Omega, 21(6), 713-715.
Ertay, T.; Ruan, D. & Tuzkaya, U. R. (2006). “Integrating data envelopment analysis and analytic hierarchy for the facility layout design in manufacturing systems”. Information Sciences, 176(3), 237-262.
Fouss, F. & Saerens, M. (2008). “Evaluating performance of recommender systems: An experimental comparison”. In Web Intelligence and 
Intelligent Agent Technology, 2008. WI-IAT'08. IEEE/WIC/ACM International Conference on (Vol. 1, pp. 735-738). IEEE.
Fouss, F.; Pirotte, A.; Renders, J. M. & Saerens, M. (2007). “Random-walk computation of similarities between nodes of a graph with application to collaborative recommendation”. Knowledge and Data Engineering, IEEE Transactions on, 19(3), 355-369.
Gattoufi, S.; Oral, M. & Reisman, A. (2004). “A taxonomy for data envelopment analysis”. Socio-Economic Planning Sciences, 38(2), 141-158.
Gunawardana, A. & Shani, G. (2009). “A survey of accuracy evaluation metrics of recommendation tasks”. The Journal of Machine Learning Research, 10, 2935-2962.
Herlocker, J. L.; Konstan, J. A.; Terveen, L. G. & Riedl, J. T. (2004). “Evaluating collaborative filtering recommender systems”. ACM Transactions on Information Systems (TOIS), 22(1), 5-53.
Hernández del Olmo, F. & Gaudioso, E. (2008). “Evaluation of recommender systems: A new approach”. Expert Systems with Applications, 35(3), 790-804.
Konstan, J. A. & Riedl, J. (2012). “Recommender systems: from algorithms to user experience”. User Modeling and User-Adapted Interaction, 22(1-2), 101-123.
Liu, J. S.; Lu, L. Y.; Lu, W. M. & Lin, B. J. (2013). “Data envelopment analysis 1978–2010: A citation-based literature survey”. Omega, 41(1), 3-15.
Lootsma, F. A. (Ed.). (1999). Multi-criteria decision analysis via ratio and difference judgment. Kluwer Academic Publishers, Applied Optimization Series (Vol. 29). Springer.
Mannino, M.; Hong, S. N. & Choi, I. J. (2008). “Efficiency evaluation of data warehouse operations”. Decision Support Systems, 44(4), 883-898.
McLaughlin, M. R. & Herlocker, J. L. (2004). “A collaborative filtering algorithm and evaluation metric that accurately model the user experience”. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 329-336). ACM.
McNee, S. M.; Riedl, J. & Konstan, J. A. (2006). “Being accurate is not enough: how accuracy metrics have hurt recommender systems”. In CHI'06 extended abstracts on Human factors in computing systems (pp. 1097-1101). ACM.
Paradi, J. C. & Zhu, H. (2013). “A survey on bank branch efficiency and 
performance research with data envelopment analysis”. Omega, 41(1), 61-79.
Pazzani, M. J. & Billsus, D. (2007). Content-based recommendation systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web Methods and Strategies of Web Personalization, vol. 4321, pp. 325–341. Springer, Berlin/Heidelberg.
Resnick, P.; Iacovou, N.; Suchak, M.; Bergstrom, P. & Riedl, J. (1994). “GroupLens: an open architecture for collaborative filtering of netnews”. In Proceedings of the 1994 ACM conference on Computer supported cooperative work (pp. 175-186). ACM.
Said, A.; Tikk, D.; Shi, Y.; Larson, M.; Stumpf, K. & Cremonesi, P. (2012). Recommender systems evaluation: A 3D benchmark. In ACM RecSys 2012 Workshop on Recommendation Utility Evaluation: Beyond RMSE, Dublin, Ireland (pp. 21-23).
Sowlati, T.; Paradi, J. C. & Suld, C. (2005). “Information systems project prioritization using data envelopment analysis”. Mathematical and Computer Modelling, 41(11), 1279-1298.
Toloo, M. (2012). “On finding the most BCC-efficient DMU: A new integrated MIP–DEA model”. Applied Mathematical Modelling, 36(11), 5515-5520.
Toloo, M. & Nalchigar, S. (2009). “A new integrated DEA model for finding most BCC-efficient DMU”. Applied Mathematical Modelling, 33(1), 597-604.
Toloo, M. & Nalchigar, S. (2011). “A new DEA method for supplier selection in presence of both cardinal and ordinal data”. Expert Systems with Applications, 38(12), 14726-14731.
Toloo, M.; Sohrabi, B. & Nalchigar, S. (2009). “A new method for ranking discovered rules from data mining by DEA”. Expert Systems with Applications, 36(4), 8503-8508.
Wu, W.; He, L. & Yang, J. (2012). “Evaluating recommender systems”. In Digital Information Management (ICDIM), 2012 Seventh International Conference on (pp. 56-61). IEEE.
Zaier, Z.; Godin, R. & Faucher, L. (2008). “Evaluating recommender systems”. In Automated solutions for Cross Media Content and Multi-channel Distribution, 2008. AXMEDIS'08. International Conference on (pp. 211-217). IEEE.
Zeleny, M. (1982). Multiple Criteria Decision Making (Vol. 25). J. L. Cochrane (Ed.). New York: McGraw-Hill.
Interdisciplinary Journal of Management Studies (Formerly known as Iranian Journal of Management Studies)
Volume 8, Issue 4 - Serial Number 4
October
October 2015
Pages 589-605

Files

Share

How to cite

Statistics