Introduction

Matrix factorization-based Recommender System (RecSys) algorithms are specialized for predicting missing entries, e.g., ratings, in sparsely filled user-item matrices. Many often-used benchmark datasets exist [1,2], representing such a RecSys task. Some of these datasets include side information, also called features, which are not used by the RecSys algorithms mentioned above. Instead, the data is directly reduced to a sparse user-item matrix, ignoring side information. In contrast, Machine Learning (ML) algorithms are broad in their applications and usually profit from the availability of additional, meaningful features [3,4,5]. As an extension, Automated Machine Learning (AutoML) techniques further increase ML performance through automated algorithm selection and hyperparameter optimization. Furthermore, the same tasks RecSys algorithms intend to solve can generally be solved by (Auto)ML algorithms. Recent advances in RecSys have led to more sophisticated model-based algorithms, with the likes of Factorization Machines [6] and especially the latest Deep Neural Networks (DNN), that can incorporate side information and are more similar to their ML relatives [7,8,9,10,11]. However, the performance gap between (Auto)ML and RecSys has not been explicitly researched.

Feature engineering is a broad topic that is well documented and researched due to its positive influences on ML [12,13,5,14,15]. It summarizes many feature-processing techniques that mainly intend to increase prediction performance. Today, such feature engineering techniques are standard for many ML pipelines. Among those techniques is the curation of features by selection and extraction. Additional real-world features and features extracted from existing data often benefit a model's performance. In this context, comparing the impact of feature quantity on the performance of RecSys and ML algorithms provides a meaningful indicator for the effects of feature engineering. However, we could not find previous comparative studies on the effect of feature quantity on RecSys algorithms.

Due to the aforementioned positive effects of feature engineering techniques in ML algorithms, we hypothesized that the same effects could also be shown for RecSys algorithms. The following two research questions arise since a performance comparison between (Auto)ML and RecSys regarding feature quantity is absent in the literature. In a RecSys problem setting, how does feature quantity impact the performance:

• of ML, AutoML, and RecSys algorithms in general?

• of RecSys algorithms compared to (Auto)ML algorithms?

We explore these questions through a case study on the Movielens-100K [16] dataset. The code that produced the results reported in this paper is available on our GitHub repository1 .

Method

We evaluated 19 algorithms from nine libraries (Table 1) on six feature sets generated from the Movielens-100K [16] dataset (Table 2).

Table 1

An overview of the evaluated algorithms and their category. We evaluated the algorithms as implemented in their respective library. Importantly, it is shown whether the algorithm can incorporate side information and how their hyperparameters were tuned. MyFM [26] Bayesian Factorization Machine X -Movielens-100K [16] is one of the regularly used explicit feedback algorithm performance evaluation datasets in the RecSys community [27,28,29,30]. The full dataset consists of a table with user IDs and item IDs and their observed ratings, where each rating has a timestamp. Additionally, each user ID and item ID contains a set of features specific to them. The user features are their age, gender, occupation, and (North American) ZIP code. The item features are their movie genre, title, release date, and IMDb URL. We solve the prediction of ratings as a regression task and measure and compare the performance of a given algorithm through the Root Mean Squared Error.

To analyze the impact of the number of features on algorithms, we cut and/or enriched the features of the original dataset and finally grouped them, resulting in six separate feature sets (Table 2). The default feature set contains most of the basic features of the original dataset. The idea is to use as many original features as possible that require little to no further processing. For this reason, the item's title, IMDb URL, and the user's ZIP codes were removed.

From the observed user-item-ratings relation, many statistical features can be engineered. They can also be calculated separately in terms of the users and items. We calculated the following nine statistical features regarding the users and items: mean, median, mode, minimum, maximum, standard deviation, kurtosis, and skew. This provides a total of 18 additional features. Notably, these features were only calculated on the training set after splitting the data into a separate training and test set to avoid leaking information about the training set into the test set. These engineered features should provide helpful additional information to the algorithms at hand.

Finally, additional real-world data points can be added to the list of available features. For this, we chose the median and mean household income and population of a period as close as possible to the release date of the original dataset. A set of these data points grouped by ZIP codes from the US from 2006 to 2010 [31] is the earliest publicly available data directly related to the user features.

From various combinations of the feature sets mentioned above, we created six combinations to perform the experiments on. An overview of the sets and their names that we refer to from here on is listed in Table 2.

Table 2

An overview of the evaluated feature sets that we generated from the base Movielens-100K [16] dataset either by cutting or enriching features. The columns denote the contained features in the named feature sets shown in the rows and provide the total number of features. We applied additional processing steps to some of the features to make them suitable for the evaluation, sometimes depending on the dataset or automatically as an algorithm requires. Generally, we tried to stay as close to the original features as possible, making changes only where sensible or necessary. As a result, we did not treat the user ID and item ID as categorical features where applicable. The movie genre is provided as a categorical feature by default, which we did not change. However, the user's occupation is not provided as a categorical feature, so we transformed it into one. Movie release dates are provided in a date format, and we converted them to a signed UNIX timestamp representation. The user's age and zip code are special cases. As provided in the original set, we treated the age as an integer for the 'basic' sets. We removed the ZIP code because it contains some non-numerical entries and because we did not intend to filter ratings in these sets. For the 'feature-expansion' sets, we divided the age by 18 to create five age categories and then treated the age as a categorical feature. We had to keep the ZIP code to add the mean and median household income and population features. Finally, we removed entries with ZIP codes that were not contained in the additional feature sets, which incurs a loss in observed ratings of 7.05%. The remaining ZIP codes range from '00000' to '99999'. To use them as a feature, we selected only the first digit of each ZIP code and then transformed it into a categorical feature. We chose to apply this processing step because the first digit in the ZIP code has a geographical meaning and, therefore, serves as an estimation for the residential area of each user.

To have an equal ground for algorithm comparisons, we applied some constraints. We performed all experiments on implementations in publicly available libraries to increase accessibility and reproducibility. The Movielens-100K [16] dataset contains explicit ratings as integers ranging from one to five. Therefore, we only chose algorithms that can take explicit ratings as input and predict ratings in that same format. We evaluated the algorithms using five-fold cross-validation. Since one of the research questions is about a comparison of the performance of algorithms against each other, we performed hyperparameter tuning on algorithms that do not default to a tuned parameter setup for the Movielens-100K [16] dataset. Depending on the algorithm, we manually tuned the hyperparameters with a random search or using SMAC3 [18], an all-purpose hyperparameter optimization tool. We set the time budget of AutoML tools to one hour for each fold. Table 1 lists all libraries and algorithms and their categories, whether they use side information, and how their hyperparameters were tuned.

Results

Figure 1 aggregates the results gathered during the experiments and clearly shows that a higher feature quantity generally results in a lower Root Mean Square Error (RMSE). In particular, when comparing evaluations of the highest quantity of features with the lowest, the RMSE is 10% lower for ML, 4% lower for AutoML, 1% lower for model-based RecSys, and 6% in total.

Figure 1:

This plot shows the RMSE performance of the algorithm categories as denoted in Table 1 on the feature sets denoted in Table 2. Each line plots the average RMSE of an algorithm category evaluated on each feature set represented by their number of features. The vertical lines and labels on the x-axis denote the collected data points. The first three data points do not include statistical features, while the final three do. The plot shows that more features result in higher performance in most cases. The evaluation shows that AutoML outperforms the traditionally strong matrix factorization-based RecSys contenders in most of the evaluated feature sets. When only the 'basic' feature sets are included, the RMSE is 1% lower. Notably, however, there is an increase in RMSE between the feature sets with eleven features which is 'feature-expansion-no-stats' and 20 features which is 'stripped-with-stats'. In these special cases, feature quantity alone can not significantly improve the algorithm's performance. A likely reason is the feature importance seen in Figure 2, which shows that the feature set with the higher feature quantity is missing essential features like the rating timestamp or item genre. The ordered Gini feature importance in percent. We evaluated these with a Random Forest Regressor that we fitted on the training data of the largest feature set. The chart shows the impact of each feature on the trained model. These important values are not necessarily true for the evaluated algorithms but provide a good estimation nonetheless. The feature names denote if they are an item or user feature. Statistical

Gini importance in percent

Figure 3 provides a detailed overview of all experiments. It shows that the performance difference and ranking in terms of the feature sets for RecSys and AutoML algorithms enormously varies by the algorithm, which is in contrast to the ML algorithms, where the ranking is mostly the same. Notably, the cross-validation procedure averages the results of multiple evaluations on an algorithm, and the figures show these aggregations. However, the evaluations of an algorithm had only marginal performance differences (<2%). Therefore, the reported averaged results shown here are relatively stable. Furthermore, we observe that the performance of the different feature sets is divided into two groups for AutoML. One of the groups contains the 'stripped' feature sets for which the search could not find a proper result even when provided with statistical features. In the other group, all other feature sets are tightly packed together, with a slight lead for the most extensive feature set, indicating the preference for a good combination and a higher quantity of real-world and statistical features.

Regarding the model-based RecSys algorithms, the DNN approaches that are Wide & Deep and Deep Interest Network do not show a clear trend toward more favorable features. However, the Bayesian Factorization Machine is one of the most exciting results. Its performance increases clearly with feature quantity. Though a RecSys algorithm by nature, it could also be used to solve more general ML problems by design. It exhibits a mix of the behavior of ML and AutoML algorithms in its results, which can be seen in Figure 3. The most significant difference is that its performance far exceeds that of any other algorithm compared to any feature set, making it a prime example of the capability of additional features for ML and RecSys.

As expected, ML algorithms consistently demonstrated improved performance with an increased number of features. This shows that the provided features are of good enough quality and distribution. The feature sets have an almost equal ranking order across all ML algorithms. 1 on the feature sets listed in Table 2. The bar chart is grouped by algorithm, and each group's bars are ordered by RMSE ascending. A lower RMSE is better. Algorithms that can not use additional features perform the same on all feature sets. For these only, the 'basic' feature set is plotted. These results are aggregated over algorithm categories in Figure 1 Overall, the results of our study indicate a clear preference for using more features. The important caveat to this is the feature type and quality. As reported, the relation between feature quantity and algorithm performance is not monotonic for AutoML and RecSys due to feature importance. The evaluated model-based RecSys algorithms performed best in this study, but our evaluation shows how even this gap can be closed by introducing additional features.