Various libraries have been released to support the development of recommender systems for some time, but it is only relatively recently that larger scale, open-source platforms have become readily available. In the context of such platforms, evaluation tools are important both to verify and validate baseline platform functionality, as well as to provide support for testing new techniques and approaches developed on top of the platform. We have adopted Apache Mahout as an enabling platform for our research and have faced both of these issues in employing it as part of our work in collaborative ltering. This paper presents a case study of evaluation focusing on accuracy and coverage evaluation metrics in Apache Mahout, a recent platform tool that provides support for recommender system application development. As part of this case study, we developed a new metric combining accuracy and coverage in order to evaluate functional changes made to Mahout's collaborative ltering algorithms.
H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval{Information ltering
Selecting a foundational platform is an important step in developing recommender systems for personal, research, or commercial purposes. This can be done in many di erent ways: the platform may be developed from the ground up, an existing recommender engine may be contracted (e.g., Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
OracleAS Personalization1), code libraries can be adapted, or a platform may be selected and tailored to suit (e.g., LensKit2, MymediaLite3, Apache Mahout4, etc.). In some cases, a combination of these approaches will be employed.
For many projects, and particularly in the research context, the ideal situation is to nd an open-source platform with many active contributors that provides a rich and varied set of recommender system functions that meets all or most of the baseline development requirements. Short of nding this ideal solution, some minor customization to an already existing system may be the best approach to meet the speci c development requirements. Various libraries have been released to support the development of recommender systems for some time, but it is only relatively recently that larger scale, open-source platforms have become readily available. In the context of such platforms, evaluation tools are important both to verify and validate baseline platform functionality, as well as to provide support for testing new techniques and approaches developed on top of the platform. We have adopted Apache Mahout as an enabling platform for our research and have faced both of these issues in employing it as part of our work in collaborative ltering recommenders.
This paper presents a case study of evaluation for
recommender systems in Apache Mahout, focusing on metrics
for accuracy and coverage. We have developed functional
changes to the baseline Mahout collaborative ltering
algorithms to meet our research purposes, and this paper
examines evaluation both from the standpoint of tools for baseline
platform functionality, as well as for enhancements and new
functionality. The objective of this case study is to evaluate
these functional changes made to the platform by comparing
the baseline collaborative ltering algorithms to the changed
algorithms using well known measures of accuracy and
coverage [
Revisiting evaluation in the context of recommender
platforms has received recent attention in the thorough
evaluation of the LensKit platform using previously tested
collaborative ltering algorithms and metrics, as reported in [
Recommender system research has been primarily
concerned with improving recommendation accuracy [
To support our research in collaborative ltering, several recommender system platforms were surveyed, including LensKit, easyrec5, and MymediaLite. We selected Mahout because it provides many of the desired characteristics required for a recommender development workbench platform. Mahout is a production-level, open-source, system and consists of a wide range of applications that are useful for a recommender system developer: collaborative ltering algorithms, data clustering, and data classi cation. Mahout is also highly scalable and is able to support distributed processing of large data sets across clusters of computers using Hadoop6. Mahout recommenders support various similarity and neighborhood formation calculations, recommendation prediction algorithms include user-based, item-based, SlopeOne and Singular Value Decomposition (SVD), and it also incorporates Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) evaluation methods. Mahout is readily extensible and provides a wide range of Java classes for customization. As an open-source project, the Mahout developer/contributor community is very active; the Mahout wiki also provides a list of developers and a list of websites that have implemented Mahout7. 3.1
Although Mahout is rich in documentation, there are implementation details on how Mahout works that could only be understood by looking at the source code. Thus, for clarity in evaluation, we needed to verify the implementation of baseline platform functionality. The following describes some of these details for Mahout 0.4 `out-of-the-box':
Similarity Weighting: Mahout implements the classic
Pearson Correlation as described in [
result = -1.0 + scaleFactor * (1.0 + result); else
result = 1.0 - scaleFactor * (1.0 - result); where count is the number of co-rated items between two users, num is the number of items in the dataset, and result is the calculated Pearson Correlation coe cient.
User-Based Prediction Algorithm: Mahout implements a
Weighted Average prediction method similar to the approach
described in [
Item-Based Prediction Algorithm: Mahout implements a
Weighted Average prediction method. This approach is
similar to the algorithm in [
Accuracy Evaluation calculation: Mahout executes the
recommender system evaluator speci ed at run time (MAE
or RMSE) and implements traditional techniques found in
[
M AE =
Pn i=1 j ActualRatingi n
P redictedRatingi j (1) where n is the total number of ratings predicted in the test run. 3.2
Through personal email communication with one of the Mahout developers, we were informed that Mahout intended to provide basic rating prediction and similarity weighting capabilities for its recommenders and that it would be up to developers to provide more elaborate approaches. Several changes were made to the prediction algorithms and the similarity weighting techniques for both the user-based and item-based recommenders in order to meet our speci c requirements and to match the best practices found in the literature, as follows:
Similarity weighting: De ned as Signi cance Weighting in
[
User-user mean-centered prediction: After identifying a
neighborhood of similar users, a prediction, as documented
in [
Pv V j simu;v j rv) (2) where V is the set of k similar users who have rated item i, rv;i is the rating of those users who have rated item i, ru is the average rating for the target user u over all rated items, rv is the average rating for user v over all co-rated items, and simu;v is the Pearson correlation coe cient.
Item-item mean-centered prediction: A prediction, as
documented in [
Pj Nu(i) j simi;j j rj ) (3) where Nu(i) is the set of items rated by user u most similar to item i, ru;j is u's rating of item j, rj is the average rating for item j over all users who rated item j, ri is the average rating for target item i, and simi;j is the Pearson correlation coe cient.
Item-item similarity thresholding: This method was added
to Mahout and used in conjunction with the item-item
meancentered prediction described above. Similarity
thresholding, as described in [
Coverage and combined accuracy/coverage metric: As
suggested in [
Coverage =
T otal#RatingsCalculated T otal#RatingsRequested (4) Code changes were also made to calculate a combined accuracy and coverage metric as de ned in Section 4.
The metrics selected for this case study, accuracy and
coverage, were chosen because they are fundamental to the
utility of a recommender system [
What constitutes `good' accuracy or coverage, however,
has not been well de ned in the literature: studies such
as [
Accuracyi Coveragei ; (5) where i indicates the ith trial in an evaluation experiment.
The AC Measure simply adjusts (upward) the Accuracy according to the level of Coverage metrics found in an experimental trial and is agnostic to the accuracy metric used, e.g., MAE or RMSE. Using a family of curves for the Mean Absolute Error (MAE) accuracy metric, Figure 1 illustrates the relationship between accuracy, coverage, and the AC Measure. As an example, following the \M AE : 0:5'' curve we see that at 100% coverage, the AC Measure is 0.5, and at 10% coverage, the AC Measure has increased to 5. The intuition behind this metric is that when the recommender system is able to provide predictions for a high percentage of items in the dataset, the accuracy metric more closely indicates the level of system performance; conversely, when the coverage is low, the accuracy metric is \penalized" and is adjusted upwards. We believe that the major bene t of the AC Measure is that it formulates a solution for addressing the trade-o between accuracy and coverage and can be used to create a ranked list of results (low to high) from multiple experimental trials to nd the best (lowest) AC Measure for each set of test conditions. The simpli ed visualization of the combined AC Measure shown in Figure 1 is an additional bene t. For our evaluation purposes, the use of a combined metric was ideal in addressing the inherent trade-o s between accuracy and coverage, especially in the cases where accuracy is found to be high when coverage is low; we posit that the AC Measure will also be useful for other researchers performing evaluations using accuracy and coverage.
The objective of this case study was to understand Mahout's baseline collaborative ltering algorithms and evaluate functional changes made to the platform using accuracy and coverage metrics. The main intent of making functional changes to Mahout recommender algorithms was to bring the Mahout algorithms in line with best practices found in the literature. Therefore, the overall hypothesis to be tested in this case study was that the modi ed algorithms improve Mahout's `out-of-the-box' prediction accuracy for both userbased and item-based recommenders while maintaining reasonable coverage. 5.1
The data used in this study were the MovieLens datasets downloaded from GroupLens Research8: the 100K dataset with 100,000 ratings for 1,682 movies and 943 users (referred to as ML100K in this study) and the 10M dataset with 10,000,000 ratings for 10,681 movies and 69,878 users (referred to as ML10M in this study). Ratings provided in these datasets consist of integer values between 1 (did not like) to 5 (liked very much).
For User-based (see x3.1), Mahout uses Pearson Correlation similarity (with and without similarity weighting), Neighborhood formation (similarity thresholding or kNN), and Weighted Average prediction. This was tested against a modi ed algorithm (see x3.2) consisting of Pearson Correlation similarity (with and without similarity weighting), Neighborhood formation (similarity thresholding or kNN), and Mean-centered prediction. For Item-based (see x3.1), Mahout uses Pearson Correlation similarity (with and without similarity weighting), no Neighborhood formation, and Weighted Average prediction. This was tested against a modi ed algorithm (see x3.2) consisting of Pearson Correlation similarity (with and without similarity weighting), Neighborhood formation (similarity thresholding), and Meancentered prediction. 5.1.1
Test Cases
In order to test the overall hypothesis, the following test cases were developed and executed for both user-based and item-based recommenders using the ML100K and ML10M datasets: 1. Mahout Prediction, No weighting 2. Mahout Prediction, Mahout weighted 3. Mahout Prediction, Signi cance weighted 4. Mean-Centered Prediction, No weighting 5. Mean-Centered Prediction, Mahout weighted 6. Mean-Centered Prediction, Signi cance weighted 5.1.2
Accuracy and Coverage Metrics
We used Mahout's MAE evaluator to measure the accuracy of the rating predictions. For prediction coverage, we used dataset training data to estimate the rating predictions for the test set; the random sample of user-item pairs in our testing was 30K pairs for ML100K and 25K pairs for ML10M (see x3.2). AC Measures were calculated for all test cases. 5.1.3
Dataset Partitioning
The Mahout evaluator creates holdout 9 partitions according to a set of run-time parameters. For the tests using the
ML100K dataset, the training set was 70% of the data, the test set was 30% of the data, and 100% of the user data was used; a total 30K rating predictions from 943 users were requested for each test set. For the tests using the ML10M dataset, the training set was 95% of the data, the test set was 5% of the data, and 5% of the user data was used; a total 25K rating predictions from 3180 users were requested for each test set. 5.1.4
Test Variations
Various similarity thresholds and kNN neighborhood sizes were executed for each test case in order to understand and evaluate the corresponding behavior of the recommenders. For User-based recommender testing, similarity thresholds of 0.0, 0.1, 0.3, 0.5, and 0.7 and kNN neighborhood sizes of 600, 400, 200, 100, 50, 20, 10, 5, and 2 were tested. For Item-based recommender testing, in addition to using no similarity thresholding, similarity thresholds of 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, and 0.7 were tested. 6. 6.1
Figures 2 and 3 show the results of test cases 1 through 6 for user and item-based algorithms, respectively10. The key results of the experiment, for both user-based and itembased algorithms unless otherwise noted, were as follows: 1. MAE for mean-centered prediction with signi cance weighting is a signi cant improvement (p<0.01) over MAE for Mahout prediction, regardless of weighting, across similarity thresholds (except item-based at similarity threshold of 0.7) and kNN neighborhood sizes (except user-based at kNN of 2, not shown).
2. Mahout similarity weighting does not signi cantly improve (p<0.01) Mahout prediction MAE over prediction with no similarity weighting (except Mahout prediction for userbased and item-based at a similarity threshold of 0.4, not shown). This would indicate that Mahout similarity weighting is not very e ective as a weighting technique, especially as compared to signi cance weighting. 6.2
The results and trend lines for the ML100K experiment are similar to ML10M. The key results, for both user-based and item-based algorithms unless otherwise noted, were: 1. MAE for mean-centered prediction with signi cance weighting is a signi cant improvement (p<0.01) over MAE for Mahout prediction, regardless of weighting, across similarity thresholds and kNN neighborhood sizes (except userbased at kNN of 400).
2. Mahout similarity weighting does not signi cantly improve (p<0.01) Mahout prediction MAE over prediction with training set and a test set, and the partitioning is performed by randomly selecting some ratings from all, or some, of the users. The selected ratings constitute the test set, while the remaining ones are the training set. 10The following curves are superimposed over each other because the values are very similar: MAE results for meancentered prediction (no weighting and Mahout weighted), MAE results for Mahout prediction (No weighting and Mahout weighted), Coverage results for Mahout prediction and mean-centered prediction (No weighting and Mahout weighted), Coverage results for Mahout prediction and mean-centered prediction (both Signi cance weighted).
As hypothesized, results for both of the ML100K and ML10M experiments show signi cant improvements in MAE using the mean-centered prediction algorithm with signi cance weighting compared to the Mahout baseline prediction algorithm. However, when coverage is considered, the \best" MAE results may need a second look. Can an MAE of 0.5 or less be considered \good" when the associated coverage is in the single digits? In this case, the recommender system may only be able to provide recommendations to a very small subset of its users and is a situation that must be avoided by system operators. To help address the accuracy vs. coverage trade-o , combined measures such as the AC Measure (Section 4), can help by considering both accuracy and coverage simultaneously. For the ML10M experiment, we determined that the lowest MAE for the Userbased algorithm using mean-centered prediction with signi cance weighting was 0.578 at a similarity threshold of 0.7 and coverage of 0.833%; the AC Measure for this result is calculated as 69.42. Similarly, the lowest MAE for the Item-based algorithm using mean-centered prediction with signi cance weighting was 0.371 at a similarity threshold of 0.7 and coverage of 1.02%; the AC Measure for this result is calculated as 36.32. In each of these cases, the exceedingly high values for the AC Measure indicate that these results are not very desirable in a recommender system.
Figures 4 and 5 show the AC Measure results for user and item-based algorithms using ML10M, respectively. Rather than show all 30 results for each algorithm (5 similarity thresholds x 2 prediction methods x 3 weighting types), we show only the results with calculated AC Measure values less than 1.0; therefore, the lowest MAE results reported above for user-based and item-based algorithms are clearly beyond the range of this chart. We found that the best combined accuracy/coverage results were found at higher levels of coverage and lower levels of similarity threshold, i.e., the best (lowest) AC Measure for user-based was 0.688 at a similarity threshold of 0.1 and for item-based was 0.665 at a similarity threshold of 0.0, both using mean-centered prediction and signi cance weighting. We can also see that, with few exceptions, mean-centered prediction is improved over the Mahout prediction for the same similarity weighting and similarity threshold. We observed similar results using the ML100K dataset where the best (lowest) AC Measure for user-based was 0.765 and for item-based was 0.746, both at a similarity threshold of 0.0 and both using meancentered prediction and signi cance weighting. These results demonstrate that the \best" MAE may not always be the lowest MAE, especially when coverage is also considered; furthermore, recommender system settings such as similarity weighting and neighborhood size also need to be considered during system evaluation.
Other observations of our experiments that match results
reported in [
Our case study of Mahout as a recommender system platform highlights evaluation considerations for developers and also shows how straightforward functional enhancements improves the performance of the baseline platform. We evaluated our changes against current Mahout functionality using accuracy and coverage metrics not only to assess baseline results, but also to provide a view of the trade-o s between accuracy and coverage resulting from using di erent recommender algorithms. We reported cases where the lowest MAE accuracy results were not necessarily always the `best' when coverage results were also considered, and we instrumented Mahout for a combined accuracy and coverage metric (AC Measure) to evaluate these trade-o s more directly. We believe that this case study will provide useful guidance in using Mahout as a recommender platform, and that our combined measure will prove useful in evaluating algorithm changes for the inherent trade-o s between accuracy and coverage.