Layered evaluation in recommender systems: A retrospective assessment

Abstract

Evaluation of recommender systems has only lately started to become more systematic, since the emphasis has long been on the experimental evaluation of algorithmic performance. Recent studies have proposed adopting a layered evaluation approach, according to which recommender systems may be decomposed into several components, evaluating each of them separately. Nevertheless, there are still no evaluation studies of recommender systems that apply a layered evaluation framework to explore how all the different components or layers of such a system may be assessed. This paper introduces layered evaluation and examines how a previously proposed layered evaluation framework for adaptive systems can be applied in the case of recommender systems. It presents the possible adaptation of this layered framework that may fit the interaction components of recommender systems. Then, it focuses on a specific recommender system and carries out a retrospective analysis of its past evaluation results under the new prism that the layered evaluation approach brings. Our analysis indicates that implementing a layeredbased evaluation of recommender systems has the potential to facilitate a more detailed and informed evaluation of such systems, allowing researchers and developers to better understand how to improve them.