The BASE prototype aims to improve user awareness of biases in search engine results. It utilises existing resources and NLP tools to identify biases in news articles. It incorporates bias visualisation features to inform users of biases in each news article and at the search results level. It also incorporates results reranking features to allow users to retrieve diferent sets of results based on their search preferences. Preliminary evaluation results suggest the prototype achieves a positive usability score (64.3 out of 100) and has a potential for increasing user awareness of biases, with the reranking features rated more useful than the bias visualisation features.
Increasingly, it becomes obvious that news search engines may include biases in their search
results [
Previous studies have proposed a number of visualisations to increase user awareness of biases.
News aggregators, such as AllSides [
In this paper, we introduce a novel prototype of a search engine interface designed to increase users’ awareness of multiple types of biases in the results. The prototype also aims to provide the ability to users to easily access diferent facets of the results. Instead of developing new methods for measuring biases, the prototype makes use of available resources and techniques to inform users of possible biases in the results. This means that such system can be made usable in the near future to support users in their information seeking tasks. An initial evaluation of how users respond to these visualisations are also provided in this study. This work provides a valuable contribution in understanding how bias-aware news search engines should be designed.
To identify specific features to include in the design, we conducted three user studies on designing bias-aware search engines using a participatory approach. These resulted in eight designs that incorporated two diferent approaches: i) bias visualisation approach, for informing users of possible biases in the results, and ii) results-reranking approach, which allows users to access diferent results by modifying (the ranking of) the results. We invited 132 participants to evaluate these eight designs. The findings suggest that users would like i) to see information on diferent types of biases in search results, ii) the ability to retrieve a diferent set of results using their preferred aspect, and iii) to have both approaches in search engines.
We incorporated findings from these studies into the design of BASE.1 The prototype provides both bias visualisation and results-reranking features. As proof-of-concept, we selected four aspects to be included in the re-ranking features: political bias, geographical locations of the publishers, geographical focus of the articles, and the entity focus of the articles. More aspects may be integrated in the next future if methods to measure them become available.
When users access the prototype, they are asked to enter a query (e.g., “coronavirus”) to start searching. Once the query is submitted, the system will display the search results (Figure 1), showing a list of relevant articles in the left panel. In addition to the articles, the prototype shows two bias visualisation features. The first feature provides bias information at the article level (shown in the left panel as diferent icons on the right side of each article). Each icon represents diferent types of biases. When a user hovers on an icon (e.g., the scale), it provides 1https://cycat.group.shef.ac.uk/prototype/BASE/ information on the type of biases and the specific biased aspect of the article (e.g., “Political bias: left-center”). The second feature provides bias information at the results level (shown in the right panel) in the form of bar charts and choropleth maps. These visualisations show the distribution of political biases of the publishers, geographical location of the publishers, geographical focus of the articles and the entity focus of articles in the search results (see Figure 3).
These visualisations also incorporated results-reranking features. By clicking a specific aspect in the figures, users can easily obtain a new set of results containing articles only from the specified political bias, country, or entity. E.g., clicking “Left-center” on the political bias bar chart will retrieve only articles from news publishers identified to have a “left-center” bias. Similarly, by clicking on “Australia” in the “geographical focus of the articles” map, users will be able to view only those articles reporting COVID-19 in Australia. We describe the methods to identify and visualise these biases in Section 2.2.
This section describes the information processing workflow of the search engine (illustrated in Figure 2) and outlines the methods used to measure and visualise these biases.
We limited our index on news articles related to the COVID-19 pandemic. For this, we
used the most popular queries for this topic according to Google Trends in February 2021. We
retrieved 100 news articles per query returned by Google News using the Zenserp API [
For each article, we carried out two processes. Firstly, we extracted the URL of the publishers for the news articles, e.g. bbc.co.uk (BBC), or ft.com (Financial Times). These URLs were then used to identify the political bias of the publishers and the location of the publishers. Secondly, we crawled the content of the articles and removed the boilerplates. These contents were processed using a named entity recogniser to identify the geographical focus of the article and the entity focus of the article. We describe these processes in more detail below.
Political bias. We utilised an external resource, Media Bias/Fact Check (MBFC) [
Geographical location of the publishers. We determined the location of the publishers by analysing the sufix of the URL (e.g., “ bbc.co.uk” is based in the UK, “abc.net.au” is based in Australia). When this information was not available, we used ‘whois’ command to identify the country where the domain is registered. Similarly, the publisher location was extracted for each article, and was aggregated for all the search results. This information is displayed using a choropleth map (see Figure 3b).
Geographical focus of the articles. We used Scrapy [
(b) Geographical location of the publishers (c) Geographical focus of articles (d) Entity focus of articles article. The most frequent country is selected as the geographical focus of the article. Similarly to the locations of the publishers, this information is also aggregated at the results level and is visualised using a choropleth map (see Figure 3c).
Entity focus of the articles. We used spaCy [
Due to the amount of processing required, the bias identification task was performed ofline.
Once completed, the bias information (and focus) was stored in an index, together with each
article’s information (e.g., URL, title, snippets, etc.). When users submit a query to the BASE
system, the articles are retrieved and displayed on the graphical user interface. The interface
is developed using PHP, and the visualisations (bar chart and choropleth map) are developed
using Plotly Javascript open source graphing library [
A preliminary evaluation study involving 21 participants – 47.62% BSc, 33.33% MSc and 4.76%
PhD students, and 14.29% non-students; 38.1% males and 61.9% females; ranging from 18 to over
40 years old; from Cyprus (42.9%), Greece (47.6%), France (4.8%) and Italy (4.8%) – suggested
that this prototype achieves a moderately positive usability score (64.3 out of 100 using the
System Usability Scale) [
A further evaluation study involving 60 MSc students – 55% males, 43.33% females, 1.67% preferred not to say; 92% between 21-25 years old, and the remaining 26 and older; majority (88.33%) from China, and the rest from other Asian countries and Slovakia – suggested that they found the reranking results features to be the most useful (4.08 out of 5). Bias information at the results level were found to be more useful (4.02) than those at the article level (3.75), due to the dificulties to understand the meaning of bias icons for each article (left panel). Participants liked the distribution of biases in the search results (right panel). They also liked the ability to click on the bar chart or maps to easily retrieve results from each aspect. Further feedback from users suggested that users need more clarity, especially how biases were calculated. Others also suggest that the design should be more inclusive, as the “left” and “right” aspect for political ideologies are not necessarily the same nor a familiar concept for users from other countries.
We realise that bias identification is a challenge on its own and may contain its own subjectivities and biases. We reduced this risk by selecting trustworthy resources (MBFC) and focusing on biases that can easily be determined (e.g., locations). MBFC, however, does not have an extensive coverage, especially for non-English news sites. Moreover, the named entity recogniser does not map any cities or towns towards the relevant country counts. It also selects the most frequent entities without taking the query context into account (e.g., that Pfizer and Moderna are relevant entities for “covid vaccine” query, but Reuters is not). More sophisticated methods, therefore, will need to be implemented to accurately identify biases in news search results.
Despite these limitations, the BASE prototype illustrates how biases in search results could be communicated to the users. The prototype incorporates bias visualisation and results-reranking features to inform users of the existing biases and support them in their search tasks. We utilised available resources and NLP tools to identify biases in search results. Our initial evaluation shows that the prototype has potentials for increasing transparency of search results. Future work will investigate ways to improve these features and to reduce the complexity of the system.
We would like to thank the reviewers for their constructive feedback on this paper. This project is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreements No. 810105 (CyCAT).