RankFromSets
In building a recommender system to help editors sift through
many documents, it is motivating to highlight the trade-off in
user privacy intrinsic to recommender systems. A machine
learning model must exploit information about a user.
However, the incentive structures of operating a recommender
system within a business can influence decisions around privacy
and transparency [
recommending-interesting-writing/
2https://github.com/the-browser/
recommending-interesting-writing
on online advertising may engender recommender systems
that upweight attention-grabbing content and hence time spent
looking at ads. Such content might maximize a user’s time
spent with a service over time at the expense of long-term
user experience or consent. In comparison, privacy-preserving
and open source tools such as the Signal encrypted messaging
service3 may provide improved user experience in terms of
privacy-preserving, transparent, and explainable algorithms
and visual interfaces [
We build an end-to-end recommender system visual
interface to address two aims: (1) to aid editors at The
Browser in their decision-making task, and give them
control through an explanation-aware interface, and (2) to release
a lightweight, performant, open-source visual interface
framework for explanation-aware recommender systems for
document recommendation. In an offline evaluation, we show that
the recommendation model we use for the visual interface
outperforms BERT, a competitive document classification model.
In a qualitative study, the control and explanations provided
by the visual interface help editors in their decision-making
and help find bugs in the recommendation model.
RECOMMENDATION MODEL
RANKFROMSETS (RFS) is the recommendation model that
powers the visual interface; the main part of the pipeline
illustrated in Figure 1. RFS scales to large numbers of articles, and
can maximize the evaluation metric of recall [
negative examples (articles seen but not selected by the editors). Further, as our goal is to build an explanation-aware visual interface that can also serve to control recommendations, and RFS is fast, interpretable, and simple to integrate into a user interface as we describe later.
RFS is a recommendation model defined by a binary classifier. For a user u and item m with attributes xm (the set of unique words in an article), RFS is described by the probability of yum = 1 (user u consuming item m):
p(yum = 1 j u; m) = s ( f (u; xm)) ;
where s is the sigmoid function. To parameterize the binary
classifier in RFS, we use an inner product architecture:
In this architecture, the user embedding qu includes a
dimension that is fixed to unity. Word embeddings b j (including a
bias dimension for every word) and the publication
embedding are fit with maximum likelihood estimation, and negative
examples are sampled uniformly at random to balance positive
examples [
VISUAL INTERFACE The visual interface is designed with RFS as the backend recommendation model. We describe how the inner product architecture for RFS enables a visual interface that is interpretable to provide explanations for why an item is recommended, and enables control so users can filter recommendations to help with decision-making.
Explanation-aware recommendation The user embedding qu and word embeddings b j in Equation (1) can be used to interpret a recommendation. The logit for a given document with a set of words xm is the sum of per-word logits, which are computed as the inner product of the user embedding and word embedding. The per-word contribution of a word in a document to the logit that determines the document’s ranking in a list of recommendations is wu j = qu>b j : (2) This weight wu j helps explain why a document was recommended, using information about both the user u and the word j. In the visual interface, words in a document are first sorted by their contributions to a document’s logit wu j, and the top words are displayed. Similarly, words that lower a document’s ranking are also displayed, to inform a user of which words detract from the recommendation of a document.
Interface for controlling recommendations In a
decisionmaking task, a user such as an editor for The Browser may
wish to filter recommendations according to topics such as
crime, technology, or business. The recommendations output
by RFS can be controlled, by altering the per-word
contributions in Equation (2) according to whether a word is topical.
This is accomplished by first calculating words related to a
topic word using pre-trained word embeddings from BERT [
Data collection and preprocessing For positive examples,
we use the historical set of articles curated by editors at The
Browser. We augment the training data with articles selected
by the editors of other curation services, and treat all
positivelylabeled examples curated by editors as data from a single user
due to a paucity of data. We use articles from news websites as
examples with negative labels, and collect additional articles
with negative labels from websites most-featured by the editors
to mimic the editorial process of reading a large swath of
articles in a feed and distilling an article list to a select few.
For preprocessing the data we use the tokenizer released by
Devlin et al. [
Experimental setup: RankFromSets We cross-validate
using the RMSProp optimizer [
Qualitative evaluation In a user study, editors at The Browser provided feedback that they used the visual interface to choose articles, and found this to be an improved workflow. The control over recommendations, and explanation-aware visual interface provided by RFS helped elicit bugs in data collection (such as foreign language sources, or fiction writing) and provides an enjoyable user experience.
DEPLOYMENT The visual interface is deployed on Github Pages, with the backend, RFS, deployed as a microservice on Amazon Web Services Lambda. Equation (3) is cheap to compute, so the lambda function is a short python script that requires numpy as a dependency, compared to BERT which would require a hosted GPU solution. RFS recommends recent articles from the editors’ reading list of feeds. As a proof of concept, we include a tab for coronavirus-related articles that users can search through using the sliders and Equation (3). DISCUSSION We built a visual interface for a recommender system powered by RFS, a flexible recommendation model. Empirically, we demonstrated that RFS outperforms BERT in an offline evaluation, while being orders of magnitude faster during training and recommendation. By deploying RFS to AWS Lambda and hosting the visual interface on Github Pages, we demonstrated a fully open-source pipeline for creating an explanation-aware, controllable visual interface for document recommendation for editorial decision-making. Future work includes studying whether the transparency and control provided by open-source recommendation systems can improve user experience and inform users as to how recommendation models influence attention online.
Acknowledgments The authors are grateful to Christian Bjartli for help with data collection.