=Paper=
{{Paper
|id=Vol-2554/preface
|storemode=property
|title=None
|pdfUrl=https://ceur-ws.org/Vol-2554/preface.pdf
|volume=Vol-2554
}}
==None==
https://ceur-ws.org/Vol-2554/preface.pdf
7th International Workshop on News Recommendation and
Analytics (INRA 2019)
in conjunction with 13th ACM Conference on Recommender Systems (RecSys 2019), 16-20 September,
Copenhagen, Denmark
Özlem Özgöbek Benjamin Kille
Norwegian University of Science and Technology Institute of Technology Berlin
Trondheim, Norway Berlin, Germany
ozlem.ozgobek@ntnu.no benjamin.kille@tu-berlin.de
Jon Atle Gulla Andreas Lommatzsch
Norwegian University of Science and Technology Institute of Technology Berlin
Trondheim, Norway Berlin, Germany
jon.atle.gulla@ntnu.no andreas.lommatzsch@dai-labor.de
1 INTRODUCTION 2 WORKSHOP DETAILS
The 7th International Workshop on News Recommendation 2.1 Keynote Speech
and Analytics (INRA 2019)1 is held in conjunction with in
Democracy, Diversity and Design - Sharing experi-
conjunction with 13th ACM Conference on Recommender
ences from an interdisciplinary project
Systems (RecSys 2019), 16-20 September, Copenhagen, Den-
mark. This workshop aims to bring researchers, media com-
Dr. Natali Helberger, University of Amsterdam
panies, and practitioners together, in order to exchange ideas
Abstract: News-recommender systems, which automati-
about how to create and maintain a trusted and sustainable
cally select the content of newsletters, personalized news apps
environment for digital news production and consumption.
or social-media news feeds are playing an increasingly critical
This version of INRA workshop series includes a keynote
role in helping users to filter and sort information. And as
speaker and 10 peer reviewed papers, where each paper have
such are fulfilling a crucial role in democratic society. Data
been reviewed by at least two program committee members.
analytics and recommender systems are going to be more and
INRA 2019 have received 16 submissions and has an accep-
more pivotal in deciding what kind of news the public does
tance rate of 62.5%.
and does not see. Depending on their design, recommenders
can either unlock the diversity of online information for their
In INRA 2019, thinking of creating a more interactive
or lock them into so-called filter bubbles. The challenge for
workshop setting, we have introduced a poster session. All
the development of diversity-sensitive recommenders is defin-
the accepted papers’ authors have given the chance to display
ing what diversity in recommendations actually means. Often
their works as a poster during the workshop. More than half
conceptualised as a measure of variance or even serendipity,
of the authors had a poster and we have observed interactions
diversity is an inherently normative concept, deeply rooted in
between the authors and the participants during the half an
democratic theory and our ideas of what it means to live in
hour break. During the call for papers of INRA 2019, we
a democratic society. Funded by the SIDN fonds, a team of
have provided the researchers access to several data sets
legal scholars, communication and computer scientists from
and an evaluation platform for news recommendations, in
the University of Amsterdam and RTL have worked on a
case they would like to test their systems by using them [6].
project that translates insights from democratic theory into
Unfortunately, we have not received any submissions using
concrete metrics that can help to assess the performance of
these data sets and platforms.
news recommenders. Condensing a concept that is as vague
In this year’s edition, we mainly focus on three categories:
and colourful as diversity into a number of concrete metrics
News recommendation, news analytics, and ethical aspects
is not a trivial task. In my keynote I would like to present
of news recommendation. More information can be found in
some of our work, and draw some lessons for future work on
[6].
’diversity by design’.
2.2 Accepted Papers
∙ Public Service Media, Diversity and Algorith-
mic Recommendation: Tensions between Edi-
1
http://research.idi.ntnu.no/inra/2019 torial Principles and Algorithms in European
INRA’19, September, 2019, Copenhagen, Denmark Özlem Özgöbek, Benjamin Kille, Jon Atle Gulla, and Andreas Lommatzsch
PSM Organizations, Jannick Kirk Sørensen the user, a setting named session-based recommenda-
Abstract: Public Service Media (PSM) websites are tion. Another particularity of the news domain is that
an interesting case for the implementation of recom- constantly fresh articles are published, which should be
mender systems for media personalization, as the PSM immediately considered for recommendation. To deal
organizations need to balance the optimization of expo- with this item cold-start problem, it is important to
sure with traditional but ill-defined PSM policy goals consider the actual content of items when recommend-
such as fairness, viewpoint diversity and transparency. ing. Hybrid approaches are therefore often considered
Furthermore, the mathematical logic of recommender as the method of choice in such settings. In this work,
system needs to be adapted to the legacy broadcasting we analyze the importance of considering content infor-
scheduling and publishing strategies and procedures. mation in a hybrid neural news recommender system.
Finally, as the PSM organizations step into new territo- We contrast content-aware and content-agnostic tech-
ries, domestication and adaption of the recommender niques and also explore the effects of using different
system technologies must take place while PSM organi- content encodings. Experiments on two public datasets
zations try to embrace the new knowledge and new pro- confirm the importance of adopting a hybrid approach.
fessions associated with recommender systems. Based Furthermore, we show that the choice of the content
on 25 in-depth interviews conducted from December encoding can have an impact on the resulting perfor-
2016 to April 2019, this paper presents a cross Euro- mance.
pean analysis of the implementation of recommender ∙ Defining a Meaningful Baseline for News Rec-
systems in nine European public service media orga- ommender Systems, Benjamin Kille and An-
nizations from eight countries. The findings indicate dreas Lommatzsch
that PSM organizations, although viewing personalisa- Abstract: Evaluation protocols for news recommender
tion as competitive necessity, approach recommenda- systems typically involve comparing the performance
tion systems with hesitation in order to maintain core of methods to a baseline. The difference in performance
PSM-values in the online environment. Furthermore, ought to tell us what benefit we can expect from using
although the collaborative filtering chosen by the PSM a more sophisticated method. Ultimately, there is a
organizations indicate a user-centered approach, cura- trade-off between performance and effort in implement-
tion systems on top of recommender systems re-install ing and maintaining a system. This work explores what
a broadcaster-centric approach. baselines have been used, what criteria baselines must
∙ Semi-supervised sentiment analysis for under- fulfil, and evaluates a variety of baselines in a news
resourced languages with a sentiment lexicon, recommender evaluation setting with multiple pub-
Peng Liu, Cristina Marco and Jon Atle Gulla lishers. We find that circular buffers and trend-based
Abstract: This paper presents the results of using predictions score highly, need little effort to implement,
semi-supervised sentiment analysis on an under-resourced and require no additional data. Besides, we observe
language such as Norwegian. To perform these experi- variations among publishers, suggesting that not all
ments, two external resources have been used: an avail- baselines are equally competitive in different circum-
able training corpus containing Norwegian reviews from stances.
major newspaper sources (NoRec), and a newly created ∙ On-the-Fly News Recommendation Using Se-
general sentiment lexicon for Norwegian. The results quential Patterns, Mozhgan Karimi, Boris Cule
of our experiments show that the performance im- and Bart Goethals
proves significantly when the sentiment lexicon is used. Abstract: The news recommendation problem poses
Besides, the best results are obtained using Support a number of specific challenges that established recom-
Vector Machines (SVM) as the machine learning algo- mendation techniques, successful in other settings, do
rithm used for training with an AUC score of around not tackle adequately. For example, unlike in other do-
92%. An alternative statistical measure was used for mains, the relevance of news articles drops significantly
evaluation, Area Under ROC Curve (AUC), in order to over time, and the order in which users visit news ar-
deal with the highly imbalanced nature of the dataset. ticles matters greatly. Furthermore, in the context of
∙ On the Importance of News Content Repre- breaking news, user interests can change rapidly, and
sentation in Hybrid Neural Session-based Rec- there is a need to generate recommendations on-the-fly,
ommender Systems, Gabriel De Souza P. Mor- taking into account recently published articles and the
eira, Dietmar Jannach and Adilson Marques latest trends among users’ preferences. To address these
Da Cunha issues, we use a form of sequential pattern mining to
Abstract: News recommender systems are designed generate up-to-date news recommendations on a click-
to surface relevant information for online readers by by-click basis. In this approach, patterns are mined
personalizing their user experiences. A particular prob- incrementally from the incoming clickstream so that
lem in that context is that online readers are often new items and trends are considered. Our experimental
anonymous, which means that this personalization can
only be based on the last few recorded interactions with
7th International Workshop on News Recommendation and Analytics (INRA 2019) INRA’19, September, 2019, Copenhagen, Denmark
evaluation demonstrates that our method compares fa- longer sufficient for building modern recommender sys-
vorably with existing techniques and outperforms them tems in domains such as online news services, mainly
on a variety of metrics. due to the high dynamics of environment changes and
∙ Giveme5W1H: A Universal System for Extract- necessity to operate on a large scale with high data
ing Main Events from News Articles, Felix Ham- sparsity. The ability to balance exploration with ex-
borg, Corinna Breitinger and Bela Gipp ploitation makes the multi-armed bandits an efficient
Abstract: Event extraction from news articles is a alternative to the conventional methods, and a robust
commonly required prerequisite for various tasks, such user segmentation plays a crucial role in providing the
as article summarization, article clustering, and news context for such online recommendation algorithms.
aggregation. Due to the lack of universally applica- In this work, we present an unsupervised and trend-
ble and publicly available methods tailored to news responsive method for segmenting users according to
datasets, many researchers redundantly implement their semantic interests, which has been integrated with
event extraction methods for their own projects. The a real-world system for large-scale news recommenda-
journalistic 5W1H questions are capable of describ- tions. The results of an online A/B test show significant
ing the main event of an article, i.e., by answering improvements compared to a global-optimization algo-
who did what, when, where, why, and how. We pro- rithm on several services with different characteristics.
vide an in-depth description of an improved version Based on the experimental results as well as the explo-
of Giveme5W1H, a system that uses syntactic and ration of segments descriptions and trend dynamics,
domain-specific rules to automatically extract the rel- we propose extensions to this approach that address
evant phrases from English news articles to provide particular real-world challenges for different use-cases.
answers to these 5W1H questions. Given the answers Moreover, we describe a method of generating traceable
to these questions, the system determines an article’s publishing insights facilitating the creation of content
main event. In an expert evaluation with three assessors that serves the diversity of all users needs.
and 120 articles, we determined an overall precision ∙ Enriched Network Embeddings for News Rec-
of p=0.73, and p=0.82 for answering the first four W ommendation, Janu Verma
questions, which alone can sufficiently summarize the Abstract: News aggregators collects content from vari-
main event reported on in a news article. We recently ous sources and presents them in one website or mobile
made our sys tem publicly available, and it remains the application for easy access. A key challenge for the
only universal open source 5W1H extractor capable news applications is to help users discover relevant
of being applied to a wide range of use cases in news articles. Both the user experience and the key metrics
analysis. depend on the high-quality personalized recommen-
∙ Recommendation systems for news articles at dations. However, building a news recommendation
the BBC, Maria Panteli, Alessandro Piscopo, presents a set of challenges due the large number of
Adam Harland, Jonathan Tutcher and Felix Mer- articles being published every hour, the surge and de-
cer Moss cline in the popularity of news, and critical nature of
Abstract: Personalised user experiences have improved recency etc. In this paper, we present a graph-based
engagement in many industry applications. When it news recommendation model which is deployed on a
comes to news recommendations, and especially for a real-world news application. Our system is a hybrid
public service broadcaster like the BBC, recommenda- of collaborative-filtering and the content-based filter-
tion systems need to be in line with the editorial policy ing. We enrich the user-article interaction graph by
and the business values of the organisation. In this adding new nodes corresponding to the named entities
paper we describe how we develop recommendation extracted from the contents of the articles. The random
systems for news articles at the BBC. We present three walk based graph embeddings are used to learn latent
models and describe how they compare with baseline representation for users, articles and named entities in
approaches such as random and popularity. We also the same space. We evaluate the learned embeddings
discuss the metrics we use, the unique challenges we via a multi-class classification of news articles into high-
face and the considerations needed to ensure the recom- level categories. We propose a recommendation system
mendations we generate uphold the trust and quality based on the binary classification problem which takes
standards of the BBC. as input a combination of the user, item and entity
∙ Trend-responsive user segmentation enabling embeddings and computes the probability of the user
traceable publishing insights. A case study of clicking on the article. We perform experiments to show
a real-world large-scale news recommendation the superiority of our model to the previous system.
system, Joanna Misztal-Radecka, Dominik Rusiecki, ∙ Leveraging Emotion Features in News Recom-
Michal Żmuda and Artur Bujak mendations, Nastaran Babanejad, Ameeta Agrawal,
Abstract: The traditional offline approaches are no Heidar Davoudi, Aijun An and Manos Papage-
lis
INRA’19, September, 2019, Copenhagen, Denmark Özlem Özgöbek, Benjamin Kille, Jon Atle Gulla, and Andreas Lommatzsch
Abstract: Online news reading has become very pop- ∙ 4th International Workshop on News Recommenda-
ular as the web provides access to news articles from tion and Analytics (INRA) 2016 5 held in conjunction
millions of sources around the world. As a specific ap- with 24th Conference on User Modeling, Adaptation
plication domain, news recommender systems aim to and Personalization (UMAP 2016), Halifax, Canada.
give the most relevant news article recommendations Acceptance rate is 75%.[2]
to users according to their personal interests and pref- ∙ 5th International Workshop on News Recommendation
erences. Recently, a family of models has emerged that and Analytics (INRA) 2017 6 held in conjunction with
aims to improve recommendations by adapting to the IEEE/WIC/ACM International Conference on Web In-
contextual situation of users. These models provide the telligence (WI), 23-26 August 2017, Leipzig, Germany.
premise of being more accurate as they are tailored to Acceptance rate is 70% [1]
satisfy the continuously changing needs of users. How- ∙ 6th International Workshop on News Recommenda-
ever, little attention has been paid to the emotional tion and Analytics (INRA 2018) 7 held in conjunction
context and its potential on improving the accuracy with CIKM 2018. 22-26 October 2018, Turin, Italy.
of news recommendations. The main objective of this Acceptance rate is 75%. [5]
paper is to investigate whether, how and to what ex-
tent emotion features can improve recommendations. 3 ORGANIZATION
Towards that end, we derive a large number of emo-
3.1 Workshop Chairs
tion features that can be attributed to both items and
users in the domain of news. Then, we devise state- ∙ Özlem Özgöbek, Department of Computer and Infor-
of-the-art emotion-aware recommendation models by mation Science, Norwegian University of Science and
systematically leveraging these features. We conducted Technology (NTNU), Norway
a thorough experimental evaluation on a real dataset ∙ Benjamin Kille, Institute of Technology Berlin, Ger-
coming from news domain. Our results demonstrate many
that the proposed models outperform state-of-the-art ∙ Jon Atle Gulla, Department of Computer and Infor-
non-emotion-based recommendation models. Our study mation Science, Norwegian University of Science and
provides evidence of the usefulness of the emotion fea- Technology (NTNU), Norway
tures at large, as well as the feasibility of our approach ∙ Andreas Lommatzsch, Institute of Technology Berlin,
on incorporating them to existing models to improve Germany
recommendations.
3.2 Program Committee Members
2.3 Previous Workshops ∙ Alejandro Bellogin, Universidad Autónoma de Madrid
7th International Workshop on News Recommendation and (UAM), Spain
Analytics (INRA 2019) is based on the following previous ∙ Andreas Lommatzsch, Technische Universität Berlin,
workshops: Germany
∙ Asbjørn Følstad, SINTEF, Norway
∙ International News Recommender Systems Workshop ∙ Benjamin Kille, Technische Universität Berlin, Ger-
and Challenge (NRS) 2 held in conjunction with the many
7th ACM Recommender Systems Conference in 2013. ∙ Cristina Marco, Amazon Alexa, Turin, Italy
This workshop had a minimal scope, which restricted ∙ Frank Hopfgartner, Information School of University
the number of submissions and led to an acceptance of Sheffield, UK
rate of 75%. ∙ Lemei Zhang, Norwegian University of Science and
∙ International Workshop on News Recommendation and Technology, Norway
Analytics (NRA) 2014 3 held in conjunction with 22nd ∙ Mozhgan Karimi, University of Antwerp, Belgium
Conference on User Modelling, Adaptation and Per- ∙ Özlem Özgöbek, Norwegian University of Science and
sonalization (UMAP) in 2014. In this workshop, we Technology, Norway
have expanded the scope with news analytics, which is ∙ Peng Liu, Norwegian University of Science and Tech-
closely linked with news recommendation. This expan- nology, Norway
sion of the scope led to more submissions and a 50% ∙ Shumpei Okura, Yahoo! Reserach Japan
acceptance rate. [3]
∙ 3rd International Workshop on News Recommenda- REFERENCES
tion and Analytics (INRA) 2015 4 held in conjunction [1] 2017. WI ’17: Proceedings of the International Conference on
with ACM RecSys 2015 Conference in September 2015, Web Intelligence, SESSION: INRA. ACM, New York, NY, USA.
Vienna, Austria. Acceptance rate is 66%. [4] [2] Jon Atle Gulla, Luc Martens, Özlem Özgöbek, Nafiseh Shabib,
and Toon De Pessemier. 2016. 4nd International Workshop on
2 5
http://recsys.acm.org/recsys13/nrs http://research.idi.ntnu.no/inra/2016
3 6
http://research.idi.ntnu.no/nra2014 http://research.idi.ntnu.no/inra/2017
4 7
http://research.idi.ntnu.no/inra/2015 http://research.idi.ntnu.no/inra
7th International Workshop on News Recommendation and Analytics (INRA 2019) INRA’19, September, 2019, Copenhagen, Denmark
NewsRecommendation and Analytics (INRA2016). http://ceur- [5] Özlem Özgöbek, Benjamin Kille, Jon Atle Gulla, Martha Lar-
ws.org/Vol-1618/INRA preface.pdf son, and Andreas Lommatzsch. 2018. 6th International Work-
[3] Jon Atle Gulla, Ville Ollikainen, Özlem Özgöbek, and Nafiseh shop on News Recommendation and Analytics (INRA 2018). In
Shabib. 2014. 2nd International Workshop on NewsRecommen- Proceedings of the CIKM 2018 Workshops co-located with 27th
dation and Analytics (NRA2014). http://ceur-ws.org/Vol-1181/ ACM International Conference on Information and Knowledge
nra2014 preface.pdf Management (CIKM 2018), Torino, Italy, October 22, 2018.
[4] Jon Atle Gulla, Bei Yu, Özlem Özgöbek, and Nafiseh Shabib. 2015. http://ceur-ws.org/Vol-2482/paper10.pdf
3rd International Workshop on News Recommendation and Ana- [6] Özlem Özgöbek, Benjamin Kille, Jon Atle Gulla, and Andreas
lytics (INRA 2015). In Proceedings of the 9th ACM Conference Lommatzsch. 2019. The 7th International Workshop on News
on Recommender Systems, RecSys 2015, Vienna, Austria, Sep- Recommendation and Analytics (INRA 2019). In Proceedings of
tember 16-20, 2015. 345–346. https://dl.acm.org/citation.cfm? the 13th ACM Conference on Recommender Systems (RecSys
id=2798721 ’19). ACM, New York, NY, USA, 558–559. https://doi.org/10.
1145/3298689.3346972