As learning robots and smart devices become common household occurrences, their users will be required to invest more time to train them on the details speci c to their household and lifestyle. This burden of personalization may eventually become a roadblock to the adoption of smart devices and robots. We are interested in reducing the burden of personalization by leveraging learned information from other households. However, machine learning methods incorporating such data will require smart recontextualizations that can map the preferences from a collection of similar users onto the user's own household space. We present several collaborative ltering based methods to solve the problem of a robot organizing the items in a kitchen for their user: a traditional collaborative ltering method based on prior work incorporating user's item{item distance ratings, and a context-aware collaborative ltering method, which enables direct learning of item{location ratings. We present results on user-annotated kitchens.
As learning robots become common household items,
people will be required to train their robot on the details speci c
to their household and lifestyle. In the context of
household automation, smart devices like the Nest thermostat
can learn user's preferred heating and cooling schedules from
tracked manual inputs [
Copyright remains with the authors and/or original copyright holders the user for training. As this burden of personalization becomes more commonplace in home automation and robotics, users may nd themselves investing more and more time into training their smart devices and robots. We are interested in reducing the burden of personalization by leveraging learned information from other households.
The challenge lies in the inability to directly transfer learned information from one household to another, due to uniqueness of household designs, schedule, demographics and composition of the household members, and the combined preferences of all users interacting with the system. Therefore, it is important to identify certain contextual facets of the learning problem to work on.
We consider one such personalization problem in the area of household robotics: the problem of a robot tasked with learning the organization of items in a user's kitchen like putting away groceries or the items from a dishwasher. Asking for the location of every item would be time consuming| possibly more so than having the person put away the items themselves. Therefore, we were motivated to design an algorithm that leverages the organization of other users' kitchens to help predict the user's organization of items.
In the context of videos, products, and news articles,
collaborative ltering techniques used for recommender
systems have been developed to transfer the experience of other
users to help identify items a target user would nd
desirable. Typically, these techniques predict ratings of
useritem pairs, that is, how would a given user rate an item
based on their past ratings [
Context-aware collaborative ltering, for example as
presented by Panniello et al. [
One of our proposed methods employs a context-based
recommender system, factorization machines (FMs) [
We present data collected from participants on Amazon Mechanical Turk who annotated kitchens with locations of items. We describe several methods for enabling a learning robot to predict item{location ratings for novel users from this data. We also discuss how these ratings can be used in an interactive system that attempts to nd a trade-o between asking the user for the correct input and making a wrong choice. We demonstrate results on a collection of user-annotated, simulated kitchen examples.
Researchers have long argued that user interfaces can
bene t from learning from their users [
Home organization is a good example of a task that both
exhibits strong user preferences and is highly desirable to
automate. Several researchers found that cleaning and
organization are the top two tasks users most want robots to
do [
Several researchers in the robotics community have begun
to look at placing and organizing items. Cha et al. [
However, in collaborative ltering, SVM methods are often disregarded due to their poor performance on sparse datasets. We present as a baseline a Support Vector Regression method that struggles on the full data set when considering individual locations, but performs better only over item and location features. However, our proposed solutions still outperform this SVR method.
Fisher et al. [
The work by Abdo et al. [
Our proposed solutions predict a user's preferred locations for items they want organized. It consists of two components, a rating prediction collaborative lter and an algorithm for producing the optimal location given a set of location ratings produced from the collaborative lter.
There are two solutions we present here. The rst builds on previous work by using a collaborative lter to predict a user's item{item ratings { whether the two items should be placed together or not. Predicting the correct location is a matter of choosing the location which contains the item with lowest distance rating to the item being placed for the evaluated user. The second uses a context-aware collaborative lter to directly predict a user's item{location ratings. The predicted location is just the location with the minimum item{location rating for the user.
We envision this work as a component in a robot's back and forth interactions with their users. To minimize the number of interactions required at the risk of misplacing objects, we also present a method for tuning the predictive success of the location-prediction system by allowing the system to choose to place based on its predicted value or ask the user for the actual location. 4.
Recommender systems are concerned with the problem of predicting user{item ratings. That is, given user u, what rating r 2 R would they assign to the item i? For the problem of household organization, we modify the problem so that the system makes predictions on item{item ratings or item{location ratings. This modi cation makes our problem a variant of the classical recommender problem in that users provide ratings on either item{item pairs or item{location pairs.
Collaborative ltering is a category of methods for
recommender systems that seek to predict ratings for items novel
to a user from ratings of the items from similar users. The
typical input is a rating matrix R 2 RM N , where rows are
associated with the items, and the columns are associated
with users. Factorization methods attempt to generate a
lower dimensional representation to improve generalization.
As shown in Abdo et al. [
Compared to classical recommender systems, context-aware recommender systems assume that some additional information exists that relates to the user's choices of ratings. For example, a user may choose a location based on the other items already placed there or based on the location's salient features. We include information about the locations into the context-based recommender, but our chosen contextaware recommender also learns automatically information related to item{item collocation to inform its predictions. 4.2
Rendle et al. [
n n y^(x) := w0 + X wixi + X i=1 i=1 j=i+1 n X (vi vj) xixj; (2) where x is the input variable vector, w0 is the global bias, w is the weights over x. (vi vj) is the dot product of vi and vj and models the interaction of the i-th and j-th variable, where vi=j 2 V is a factorized representation of the weighting of xi=j .
Instead of estimating an individual weight parameter (wi;j 2 R) for higher-order interactions (d 2), the factorization of v enables modeling even under sparsity. Here, w0, w and V 2 Rn k are the parameters to be estimated.
Context does not need to be explicitly represented because it can be learned through the factorization of vi . Though we show that the item{item collocation has very powerful explanatory power, we do not need to represent that information in the context variables. Instead, we include users, items, locations and features, and the optimization of the model will seek to explain these ratings appropriately.
Though the model presented in equation 2 suggests a solution would require a run time of O(kn2) where k is the number of dimensions and n is the number of context variables, they show this equation can be reformulated into a linear solution with runtime complexity O(kn). Indeed, for models with higher degrees of interactions, it can still be refactored into a linear time solution.
One di culty of choosing a correct location for an item is that many locations in a kitchen are interchangeable and are justi able placement locations ignoring the current organization. A user's preferred organization therefore requires the robot to identify the most likely location from existing item placements and physical suitability of the location for the item.
For our work, we chose to code ratings as 0:0 being a positive example and 1:0 as a negative example, though the reverse { 0:0 for negative and 1:0 for positive { is equally valid. For item{location pairs, a rating, r(i; l) = 0:0 indicated the item is in the preferred location and 1:0 indicated it is not. For item{item pairs, a rating, r(i; i0) = 0:0, indicated the items should be located together. Rating predictions produced by the collaborative ltering methods are real valued approximately in the range of [0; 1]. 5.1
In our rst approach, item{item ratings r(i; i0) are learned
from training input via a context-unaware collaborative
ltering technique [
Formally, for items i; i0; j; j0 2 I = fi1; i2; ; img, and location l 2 L = fl1; l2; ; lng where i 6= i0 and j 6= j0, n is the number of locations and m is the number of items, we want to nd the item{location rating Ritem{item summarized from item{item ratings r(i; i0). (4) (5) (6) Ritem{item(i; l) = (mini02l r(i; i0) if 9i0 2 l; m12 Pj2I Pj02I r(j; j0) otherwise. (3)
We then select the location with the lowest distance rating. We denote the event of item i being placed in location l as i 2 l and write the selected location as ^l = argmin Ritem{item(i; l):
l 5.2
We use the factorization machine's model to predict item{ location ratings. Each row in the input matrix includes context variables for the user, item, location, and a location category. For each provided example of an item{location pairing in the input data, we assign a distance rating of 0. We produce n 1 other ratings of value 1 for this same item in all the other locations in the kitchen. Each location variable is encoded uniquely for the user, even if in experiments di erent users annotated the same kitchens.
In Equation 5, we show the context variable vector we used in our dataset. Each context variable is assigned a value of 1 when active and 0 otherwise. In our notation, u is a user, ii is an item, lj is location, and (l) are the feature values over the active location variable l: x = fu1; ; uU ; i1; ; im; l1; ; ln; (l)g:
Because the model directly learns item{location ratings Ritem{location(i; l) = r(i; l), we can nd the location with the lowest distance rating: ^l = argmin Ritem{location(i; l): l
Both techniques, predicting via item{item pairs and item{ location pairs, provide ratings for an item in a location. We can combine the two values in a number of ways. We chose harmonic mean to capture information provided by both ratings without being unduly in uenced by the relative o set biases between the two types of ratings. ^l = argmin l 2:0Ritem{location(i; l) Ritem{item(i; l) : Ritem{location(i; l) + Ritem{item(i; l) (7)
To create a data set with maximum opportunities for generalizing between users, we were interested in nding a collection of items that many people would be familiar with and would have in their households. 6.1
To create our item set, we began by pulling a list of
recommended cooking items from `How to Cook Everything' by
Bittman [
We randomly split the selected items into surveys of 10 items, and one survey of 8 item and asked participants on Amazon Mechanical Turk if they had any of the items in their household kitchen. For each survey, we collected 20 responses. Participants were allowed to answer as many di erent surveys as they chose (min: 1, max: 30, mean: 10.5). No one participant completed all the surveys.
From the full set, we selected the items at least 85% of respondents indicated were in their households, producing a list of 111 items to be used for location annotations.
We were provided four kitchen layouts from the authors of the CMU Kitchen Dataset. We labeled each drawer, cabinet, refrigerator, freezer and counter with a unique numerical identi er. Figure 6.2 shows one of the labeled kitchen images we provided for a survey. We created four di erent surveys, each with a di erent labeled kitchen. We asked participants to assign locations in their labeled kitchen for all 111 items. Participants were asked not to respond to more than one survey. For the participants who took multiple surveys, we only kept responses to their rst. We received 25 responses to each survey. We removed three responses from two groups due to duplication of users. Ten participants from the previous task also completed this one.
To summarize patterns in how objects were placed closely to one another, we built the hierarchical clustering of item{ item pair distances shown in Figure 1. Over all users, and for each item{item pair, we computed the fraction of times the items were not placed together. Groups of items with distances less than or equal to 4 are given unique colors in the diagram. The gure illustrates that items fall into clear categories in their placement around the kitchen. For example, `a glass cup', `a large glass', `a cup', `a mug', and `a large co ee mug' were commonly placed together.
From the data presented in Section 6.2, we created four sets of training and testing data by assigning data collected for three kitchens to training and the fourth to testing. Even though many people annotated the same kitchen, we used a variable for a location unique to each user. For each item and location in a kitchen, we created a collocation data set by indicating for each item{location pair in the original data set whether it was found with one of the other 111 items. That is, we changed the data lines from rating{item{location to rating{item{item{location. We set the associated rating to 0:0 if the two items were found together in the location and 1:0 otherwise.
For the item{location dataset, probes were drawn from the testing data. For each user in the testing data, we selected a random set of items and moved the ratings of that user for those items to the training dataset. We used the same sets of items for the collocation dataset and created a set of probes for all item pairings in this set of items. 7.1
We trained the factorization-machine model (FM) using
libFM [
For the data-driven factorization method (DD), we trained
the collaborative lter on the collocation training dataset
without the additional probes. The probes were then used
to update the model through the new-user update method
presented by Abdo et al. [
We included a support vector regression (SVR) model
to predict ratings from the dataset. The SVR model was
trained on the item{location data set. We used the
implementation provided in scikit-learn [
We trained both SVR models and one FM model with features. For both, we included features about the locations describing the location type: drawer, low cabinet, high cabinet, counter, refrigerator, freezer. Additionally, for the SVR models, we also included item features related to the use of the item: edible, drinkable, food storage, etc.
For the combined model, we used the FM model trained with features and the DD method discussed above. 7.2
Figure 2 presents the predictive success of the non-probe items in the testing dataset for two FM models, a datadriven factorization model over item collocation data (DD), a model combining the FM model and DD model (DD FM Combined), and two SVR models. We measured predictive success as the fraction of times the model predicted the correct location a user chose to place an item.
For the two FM models, we present a model trained as we described in previous sections (FM), and also a model trained on these variables but without location features (FM, no features). For the two SVR methods, we show the results of one model trained using location variables (SVR) and one without (SVR, no location). We show results for di erent fractions of items used as probes, speci cally 0, 41 , 12 and 43 , corresponding to 0, 28, 56, and 83 probes out of the 111 total items.
As seen in this graph, the proposed methods perform quite strongly for this di cult task. Even with no information about a current user (0 probes), the FM and DD FM combined model can correctly predict a signi cant fraction of locations for that user. The FM, no features and the DD methods perform strongly when probes are available, but reduce to random guessing when 0 probes are used. This is due to the limitations of collaborative ltering without additional contextual information. Without any information from the user, the base performance is generally quite poor.
The FM models both performed strongly over a wide range of probes. Both of the FM models were trained without explicit collocation data, yet both performed as well as the data-driven model, implying that the FM model is correctly learning the importance of item{item pairings for prediction. Indeed, the success of the DD method to predict locations when probes were available illustrates the importance of the inherent item{item a nities presented earlier.
The DD FM Combined model outperformed the FM models and the DD model by themselves when probes are provided. It combined the learning of the FM model on location features with the high accuracy of item{item ratings provided by the DD model.
Both SVR methods performed worse than the proposed solutions regardless of the number of probes presented. The SVR method trained with locations performed especially poorly. The locations were uniquely encoded for each individual user, which created a highly sparse input that posed di culty to this method. By excluding locations, the SVR no locations method performed better because this input is signi cantly less sparse. Regardless, the collaborative ltering methods strongly outperformed the SVR methods. 7.3
An important design consideration for an interactive system is the trade-o between placing an item incorrectly and asking the user for the correct placement. We observed that each algorithm places higher con dent answers closer to the ranges of their output. Therefore, we can enable systems to ask questions for uncertain items by requiring a threshold for ratings.
Figure 4 plots the success of each algorithm against the fraction of items in the testing dataset that do not meet the threshold for placement with 50% of probes already placed. We normalized the predictions of each algorithm between 0 and 1 to simplify the comparison.
Data points at a fraction of 0 correspond to data presented in Figure 2 for 50% probes. The DD FM Combined method remains among the strongest overall. Data for fractions above 0.85 were not included because there are too few data points for accurate predictive success.
In this work, we presented several methods for reducing the burden of personalization that comes from users supplying a learning system household organization preferences. We collected data about the existence, pairing and location of items in participants' kitchens. We found that people have a lot of items in common but also many unique items not found in other kitchens. A system that recognizes a location suitable for an item could easily still fail to match the user's preference for its location. In response, we presented a solution that incorporates a user's preferences but enables an autonomous system to make use of perceivable features of these locations when learning.
We showed that a context-based collaborative ltering model called factorization machines is well suited to predict item{location ratings. Choosing the appropriate location is a simple task of nding the location with the best rating. For interactive systems, we show the value in trading o asking for a location versus placing it incorrectly.
Though household robotics entails a sizable burden of personalization, this issue also plagues many other modern smart-household devices like thermostats and lighting. Machine-learning methods that use data from other households in learning a user's preferences will require smart recontextualizations that can map the preferences drawn from a collection of di erent users onto the user's own household space. We envision these topics as bright areas of future research.