The evolution of online platforms over the past decades has radically transformed the way people discover music, and thanks to social media and music streaming services nowadays listeners have access to an ever-increasing amount of tracks and artists. Within these platforms, one of the goals of recommender systems is to help users discover music without making them feel overwhelmed while exploring the huge catalogues available. However, these systems have come under scrutiny from the scientific community, policy-makers, and civil society due to their potential negative societal impact, notably with regard to issues of fairness, non-discrimination, inclusion and diversity. Algorithmic auditing has emerged as a tool to analyse the problematic behaviours exhibited by recommenders, and to ofer remedies that can limit their negative impact. In this position paper, we advocate for the involvement of end-users in the auditing process, which can contribute to the recognition, analysis, and mitigation of problematic behaviours which may arise while discovering music. Highlighting how recommenders, by influencing the discoverability of music, may impact listeners' exposure to culturally diverse content, we seek to address the challenges posed by music recommender systems problematic behaviours, ultimately with the goal of fostering a more inclusive and diverse environment for music discovery within the digital landscape.
The music sector is one among the Cultural and Creative Sectors (CCSs) with the largest audience reach,
and an essential component of cultural diversity. It has the power to bring positive changes in society
and people’s well-being, whilst at the same time generating billions of revenue. Among the challenges
this sector is facing [
In these platforms, recommender systems help listeners discover new music within the huge
catalogues available. At the same time, these systems benefit the artists, who on online platforms seek a
connection with the most appropriate audience. However, the growing enthusiasm accompanying the
rise of online platforms and recommender systems has been followed by increasing concerns about
emerging problematic behaviours and their impact on society. In particular, the negative impact on the
exercise of fundamental rights, such as the right to non-discrimination and respect for cultural diversity,
is of particular relevance for the CCSs [
In what follows, we motivate our position by briefly examining background research on related topics. We then continue by discussing the research objectives and questions we consider as relevant, for then describing the methodology we designed to tackle the presented challenges. Finally, we conclude by highlighting the potential impact that such research activities could have in the future.
Our research roadmap builds upon the idea of end-user [
Another pillar of our research is the concept of music discoverability, which can be defined as “[...]
its availability online and its ability to be found among a wide range of other content, particularly by
someone who was not specifically looking for it" [
In the music sector, the recent report on the impact of algorithmically driven recommendations on
music consumption and production commissioned by the UK Centre for Data Ethics and Innovation
[
Nonetheless, it has been shown how users perceive diversity as a motivating factor for discovering
music in the long term [
We identify three specific research objectives (ROs) and corresponding research questions (RQs) on which the investigation should focus on:
RO1. Algorithmic Auditing as a source of evidence: Initially, it is mandatory to examine instances of recommender systems’ problematic behaviours in online platforms not limited to the music ifeld, understanding how the lessons learned can be applied to the music sector. The related research question (RQ1) is: What is the potential of auditing music recommender systems, and what obstacles must be overcome? We hypothesise that through the identification of a feasible number of instances of problematic behaviours, and by documenting the lessons learned and their applicability to the music ifeld, we can provide evidence of how recommender systems may be audited.
RO2. Towards a framework for auditing discoverability: Second, it is needed to develop a tool for end-user auditing, focusing on the online platforms where recommender systems mediate the discoverability of music content. The related research question (RQ2) is: How can we audit recommender systems’ mechanisms afecting music discoverability? By evaluating the functionality and efectiveness of the developed tool in identifying and auditing recommender system mechanisms, we plan to verify, through testing and validation processes, its ability to accurately audit problematic behaviours.
RO3. Facilitating the discoverability of culturally diverse music: Lastly, it should be considered
the use of audits as a tool for remediation, proposing solutions to facilitate the algorithmic-mediated
discoverability of culturally diverse music, where needed. The associated research question (RQ3) is the
following: How can we design recommender systems for promoting cultural diversity, by limiting their
problematic behaviours? We intend to verify the efectiveness of the proposed solutions by comparing
the empirical evidence gathered with the state of the art in algorithmic auditing [
The methodology we propose is aligned with the aforementioned research objectives and research questions. We designed a two-phase mixed-method exploratory study using the outcomes of the initial qualitative phase to facilitate the development of the subsequent quantitative phase. Figure 1 depicts an overview of the proposed methodology, and the methods required to reach the aforementioned ROs are described hereafter.
RO1 will involve the gathering of evidence on how algorithmic auditing can be applied to music recommender systems. We will start by reviewing existing cases of algorithmic auditing of recommender systems where problematic behaviours have been reported. This review will provide realworld scenarios that will be afterwards discussed with end-users, by means of think-aloud interviews. During the interviews, we will discuss existing cases of problematic behaviours of recommender systems, concerning topics such as bias, diversity, fairness and non-discrimination. Then, we will look together with the participants for new cases, performing open-ended tasks and asking them to think aloud to provide insights. Finally, we will invite them to look for cases of problematic behaviours in their everyday life while discovering music on online platforms, by taking part in a diary study. They will be asked to record such cases and provide an explanation of why they thought it would be problematic.
In this qualitative phase, recruiting participants for think-aloud interviews and the diary study might be challenging. Moreover, ensuring diverse and engaged participants is crucial for obtaining comprehensive insights. In order to tackle these issues, we will i) clearly communicate the purpose and benefits of participation, ii) target a diverse group of listeners, and iii) use incentives to motivate engagement.
RO2 will focus on the design and test of an algorithmic auditing tool for assessing the
problematic behaviours which recommender systems may present whilst discovering music.
The cases of problematic behaviour related to music discovery in online platforms submitted by
participants during the diary study will be discussed in a workshop. Participants will work together
to understand the identified cases by asking and answering questions about the reported issue. The
analysis of the workshop’s insights will serve as a basis for the next stage, wherein we will make use
of crowd-based user-centred design techniques [
In order to conduct a productive workshop where participants collaboratively discuss and understand identified cases of problematic behaviour, we will prepare well-structured materials, including clear case descriptions and discussion prompts. We will facilitate the workshop efectively, encouraging open communication, and moderating discussions to ensure productive collaboration. Moreover, evaluating the usability, user experience, and acceptance of the auditing tool requires careful planning to capture users’ genuine interactions and perceptions.
RO3 will focus on the analysis and interpretation of the previously generated audit reports in order to inform policy-makers about the status of the discoverability of culturally diverse music in online platforms. Finally, we will review the audit reports generated throughout the research by the study participants using both qualitative methods (grounded theory) and quantitative analysis (text mining, statistical analysis). The end goal of the process will be to consolidate the accumulated knowledge in a policy brief.
Integrating qualitative methods and quantitative analysis can be complex and requires a coherent approach. We will develop a systematic process to combine insights, considering how qualitative ifndings can inform the formulation of quantitative research questions and vice versa. Furthermore, creating a comprehensive policy brief that efectively communicates the accumulated knowledge might be challenging.
A final remark needs to be made on the recruitment of participants. Indeed, based on ongoing
empirical and theoretical analysis, it has been shown that women, LGBTQIA+ communities, and ethnic
minorities often experience more impactful negative efects from problematic algorithmic behaviours
[
During the aforementioned empirical inquiries, we will actively seek to balance the sample group if we notice an overrepresentation of any other dominant group. While collecting data, we will only gather information relevant to the research, even if in certain cases we may include questions to ensure diversity and equal representation in the sample. This will prevent from relying solely on data from a homogenous group, ensuring a more comprehensive understanding of the issues. Nevertheless, working with a diverse group of participants may present challenges such as communication barriers, varying levels of technological literacy, and cultural diferences. To address these, it will be pivotal to provide clear instructions, ofer multilingual support when needed, and ensure accessibility for all participants. Additionally, fostering cultural sensitivity and establishing guidelines for conflict resolution can help create a respectful and inclusive environment, enabling efective collaboration among participants.
The multifaceted impact of the planned research will extend beyond its immediate duration, fostering positive changes in scientific discourse, policy-making, and social awareness, all while catalysing advancements in the understanding and utilisation of algorithmic auditing. The impact, extending beyond its immediate scope and duration, can be categorised into the following dimensions:
Scientific impacts. The approach to algorithmic auditing developed in this research activities will significantly contribute to the ongoing discourse concerning the inclusion of end-users in the identification of problematic algorithmic behaviours and the proposition of potential solutions. While human-centred design has long been a cornerstone in technology development, its application to address ambiguous mechanisms behind these technologies has received limited attention. We will fill this gap, sparking a new wave of exploration in this field. Scholars working in the fields of Human-Computer Interaction and Recommender Systems will mostly benefit from the research outcomes.
Economic and political impacts. The described strategies to advance the state of the art of
recommender systems’ auditing will hold particular significance within recent legislative frameworks,
e.g., the EU initiative on cultural diversity and the conditions for authors in the European music
streaming market [
Social and cultural impacts. The promotion of a culturally diverse landscape in the music sector stands as a core objective for various initiatives advocated by several institutions. In this context, the contributions made by the presented research will be instrumental in realising this goal. The involvement of users in the critical analysis of the technology that underpins music discoverability aligns with this objective. By showing the inner workings of recommender systems, the research will empower users to enhance their algorithmic literacy further promoting informed digital engagement.
The research roadmap described in this position paper is part of the European Commission’s funded
project Algorithmic Auditing for Music Discovery (AA4MD), funded under the Marie
SkłodowskaCurie Actions (MSCA) (Grant agreement ID: 101148443) [