Measuring Gender Stereotype Reinforcement in Information Retrieval Systems*

Abstract

Can we measure the tendency of an Information Retrieval (IR) system to reinforce gender stereotypes in its users? In this abstract, we define the construct of Gender Stereotype Reinforcement (GSR) in the context of IR and propose a measure for it based on Word Embeddings. We briefly discuss the validity of our measure and summarize our experiments on diferent families of IR systems. IIR 2021 - 11th Italian Information Retrieval Workshop, September 13-15, 2021, Bari, Italy " [e-mail] (A. Fabris); [e-mail] (A. Purpura); [e-mail] (G. Silvello); [e-mail] (G. A. Susto) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CPWrEooUrckResehdoinpgs 2IhStTpN:/c1e6u1r3-w-h0s.o7r3g e CprEeUseRnWtboirnkasrhyofpraPmroincgeeodfinggesnd(CerEiUsRa-WcoSn.soerqgu)ence of this fact and a clear limitation of our work. This is a common weakness for work in this space; addressing it is far from trivial. * Extended abstract of Fabris et al. [4].

Keywords

• eol>Fairness
• Gender Stereotypes
• Information Retrieval
• Search Engines
• Word Embeddings

1. Introduction

Search Engines (SEs) increasingly act as the gatekeepers of information. Their role in information access is undisputed, with a user base exceeding 90% of all people connected to the internet [ 1 ]. SEs inevitably influence users, helping them map concepts and link entities across queries and documents. For this reason, they can play an important role in countering or reinforcing stereotypical associations [ 2 ].

Stereotypes are generalised beliefs about groups of individuals, held widely in a population of interest. They arise from a co-occurrence of features, such as membership to a group and display of certain traits and roles. The extent to which an individual believes a stereotypical trait to be common in a given group is often measured through an association test between groups and traits [ 3 ].

Male and female are highly salient categories in human cognition, available from an early age for stereotypical associations.2 As a result, western societies maintain a wide range of gender stereotypes, relating e.g. to professions, career, competence, care, predisposition for science and mathematics. The same stereotypes are also found in artifacts and technology produced by the same societies. For example, the search results of popular image SEs were found to contain gender-stereotypical associations [ 5 ] and to influence users’ cognition accordingly [ 6 ]. Only recently, novel approaches to measure gender bias in text-based SEs have been proposed [ 4, 7 ]. -0.4

-0.2 0.2 g(w) 0.4 x him man jobascg_amesrncecieeyrnce commaruftansmioinly jobs_f woman her (a) Gender direction In this work we provide an overview of the Gender Stereotype Reinforcement (GSR) construct and measure of Fabris et al. [ 4 ].

2. GSR: Construct, Measure and Validity

In the context of IR, we define GSR as a SE’s tendency to reinforce (or counter) genderstereotypical associations in its users. Direct measurement of this construct would require impractical longitudinal user studies of counterfactual nature. Fabris et al. [ 4 ] propose a computational approach, based on Word Embeddings (WEs).

Indeed, WEs have been found to reliably encode several gender stereotypes [ 9, 10 ], typically along a single direction of the embedded space. More precisely, Bolukbasi et al. [ 9 ] show how to isolate a problematic direction, called gender subspace, where gender-related concepts are clustered in accordance with gender stereotypes. To illustrate this concept, Figure 1a depicts the gender direction of Word2vec embeddings [ 11 ] along the axis. A sample word, = beauty, is projected onto the gender direction, where it is closer to intrinsically female words (her, woman) and to stereotypically female concepts than to their male counterparts.

Let us indicate by () = ( · )/(||||) the function associating a word with its normalized scalar projection on the gender direction . By extension, (· ) maps a query into the average projection of its words (). Let us call () the genderedness of . Moreover, we apply function (· ) to the ranked list of documents ℒ returned by an IR system in response to query . We indicate it by (ℒ) and define it as the average projection of words in ranked documents , weighted according to the rank of each document in ℒ. In symbols (ℒ) = ∑︀∈ℒ · (), with computed according to a DCG-like logarithmic discount [ 12 ].3 Given a set of queries and a collection of documents available for retrieval, we define the GSR of an IR system over (, ) in terms of the correlation between the genderedness of queries in (), ∈ and the genderedness of ranked lists of documents (ℒ) produced in response. More precisely, GSR is defined as (, ) = 1

1 ∑︁(() − )((ℒ) − ℒ), 2() =1 (1) where , ℒ represent the average genderedness of queries and ranked lists, while 2 () is a scaling factor to go from correlation to slope coeficient. Informally, Equation 1 captures the agreement between the language of queries and documents along stereotypically gendered lines, induced by an IR system .

A thorough assessment of the suitability of this equation to measure the GSR construct is an important and complex endeavour undertaken in [ 4 ]. Here we show the precision of the projection function (· ) in finding interesting queries for the study of gender stereotypes in SEs. Figure 1b (1c) shows the ten queries with lowest (highest) genderedness () in the Robust04 collection [ 8 ], which are the most associated with women (men) according to (). Indeed these are gendered queries, ranging from intrinsically gendered (mentioning women), to biologically gendered (mentioning menopause), to stereotypically gendered (with quilts and child among words in stereotypically female queries, dangerous and heroic in stereotypically male queries).

3. Experiments and Discussion

Our experiments on the Robut04 collection [ 8 ], omitted here for brevity, compare IR ranking algorithms from diferent families. We consider lexical models (e.g. BM25 - [ 13 ]), semantic models (e.g. w2v add - [ 14 ]) and neural architectures (e.g. MatchPyramid - [15]). We find that semantic models, based on biased WEs, are most prone to reinforcing gender stereotypes, while neural systems based on the same word representations can mitigate this efect. Indeed neural models exhibit low GSR, comparable to that of lexical systems such as BM25. Moreover, we test the reliability of these conclusions by measuring GSR according to two diferent sets of WEs (Word2Vec [ 11 ] and fastText [16]), finding strong agreement between the two. Finally, we assessed the impact of debiasing WEs [ 9 ] on downstream IR tasks. By measuring system performance and GSR both before and after debiasing, we find this approach to be superficial and insuficient to reduce the tendency of an IR system to reinforce gender stereotypes.

3To be precise, (ℒ) should be query-dependent [ 4 ]. Here we neglect this aspect.

Acknowledgments

Part of this work was supported by MIUR (Italian Minister for Education) under the initiative "Departments of Excellence" (Law 232/2016). [15] L. Pang, Y. Lan, J. Guo, J. Xu, S. Wan, X. Cheng, Text matching as image recognition, in:

Proc. of AAAI 2016, AAAI Press, 2016, p. 2793–2799. [16] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, Bag of tricks for eficient text classification, in: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, ACL, 2017, pp. 427–431.

References

[1] K. Purcell , J. Brenner , L. Rainie , Search engine use 2012 , 2012 . URL: https: //www.pewresearch.org/internet/wp-content/uploads/sites/9/media/Files/Reports/ 2012/PIP_Search_Engine_Use_ 2012 .pdf.

[2] S. U. Noble , Algorithms of oppression: How search engines reinforce racism , NYU Press, 2018 .

[3] A. G. Greenwald , D. E. McGhee , J. L. Schwartz , Measuring individual diferences in implicit cognition: the implicit association test ., Journal of personality and social psychology 74 ( 1998 ) 1464 - 1480 .

[4] A. Fabris , A. Purpura , G. Silvello, G. A. Susto , Gender stereotype reinforcement: Measuring the gender bias conveyed by ranking algorithms , Information Processing & Management 57 ( 2020 ) 102377 .

[5] J. Otterbacher , J. Bates , P. Clough , Competent men and warm women: Gender stereotypes and backlash in image search results , in: Proc. of CHI 2017 , ACM, 2017 , p. 6620 - 6631 .

[6] M. Kay , C. Matuszek , S. A. Munson , Unequal representation and gender stereotypes in image search results for occupations , in: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM , 2015 , pp. 3819 - 3828 .

[7] N. Rekabsaz , M. Schedl , Do neural ranking models intensify gender bias? , in: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '20, Association for Computing Machinery, New York, NY, USA, 2020 , p. 2065 - 2068 .

[8] D. Harman , The darpa tipster project , SIGIR Forum 26 ( 1992 ) 26 - 28 .

[9] T. Bolukbasi , K.-W. Chang , J. Y. Zou , V. Saligrama , A. T. Kalai , Man is to computer programmer as woman is to homemaker? debiasing word embeddings , in: Advances in neural information processing systems , 2016 , pp. 4349 - 4357 .

[10] A. Caliskan , J. J. Bryson , A. Narayanan , Semantics derived automatically from language corpora contain human-like biases , Science 356 ( 2017 ) 183 - 186 .

[11] T. Mikolov , I. Sutskever, K. Chen , G. S. Corrado , J. Dean , Distributed representations of words and phrases and their compositionality , in: Advances in neural information processing systems , 2013 , pp. 3111 - 3119 .

[12] K. Järvelin , J. Kekäläinen , Cumulated gain-based evaluation of ir techniques , ACM Transactions on Information Systems 20 ( 2002 ) 422 - 446 .

[13] S. E. Robertson , U. Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, Foundations and Trends in Information Retrieval (FnTIR) 3 ( 2009 ) 333 - 389 .

[14] I. Vulić , M.-F. Moens , Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings , in: Proc. of SIGIR 2015 , ACM, 2015 , p. 363 - 372 .