It is well established in psychology that framing of content afects the behavior of people. This efect is, however, only sparsely explored in information-seeking and retrieval behavior. In the present work, we consider the diversity of consumed content and repetition patterns regarding their framing. We conduct a framing analysis in the MIcrosoft News Dataset (MIND) comprising textual content and user interaction behaviors. By extracting the frames of the item sequences, we uncover a tendency of users to consume similar framed news repeatedly when sticking to the same type of content. Consequently, framing biases are important to consider in information systems. We hope that our work inspires future research on corresponding debiasing methods.
regarding consumed categories. Similarly, we can extract the framing based on the content
and assign it to the items, which results in sequences of consumed frames. We consider such
sequences to uncover biased behavioral patterns regarding the framing. For frame extraction,
we use the FrameFinder library [
We find that frame consumption depends on the consumed categories, the types of frames and the information system itself. In particular, users repeatedly consume the same frames when sticking to the same category, which could be counteracted by the information system. Overall, the consumption behavior is more balanced concerning moral frames compared to semantic frames, whereas media frames depend on the categories the most.
In sum, our main contributions are: 1. We connect two separate strands of research in computer science (i.e., computational framing analysis and biases in information systems) that are both rooted in psychology. 2. We introduce an approach to analyze biased behavior patterns based on sequences of consumed frames. 3. We provide empirical evidence of behavioral biases due to framing on a well-established recommendation dataset.
To the best of our knowledge, we are the first to directly investigate this link between the framing of content and the consumption behavior of users. For reproducibility reasons, we additionally open source the code (also containing the supplementary materials referenced in the paper), as well as the framing dataset used for our study1.
Framing Theory: Framing has long been considered as a fractured paradigm in literature [
Recently, a vast amount of research uses computational methods for framing detection on
wide range of frames, e.g., war [
Dataset: https://zenodo.org/records/10509498 [
economic sports
N38046 Mohamed Sanu Can't Contain Excitement Over
Trade To Patriots
On Twitter
tv N44402
quality sports N57768
Recommendations uses the latter to extract the framings present in news articles, as it also employs Transformer models [25] to extract frame representation in an unsupervised manner.
Biased Behavior Patterns: Cognitive psychology plays a vital role in information systems,
which also provides the inspiration for various recommendation approaches [26]. As an example,
a cognitive model of human memory (ACT-R) can predict music genre preferences [27].
Moreover, it has been shown that the cognitive-inspired ACT-R model also efectively predicts music
relistening behavior [28], while also increasing the diversity of genres [29]. The relistening
behavior is a type of repeat consumption, defined as “the act of consuming an enjoyable
stimulus that one has already consumed in full in the past” in psychology [30]. Such biased
repetition patterns have been found in a variety of domains and platforms, such as on Wikipedia,
Google Maps, and YouTube [
In the present work, we investigate biased behavior patterns in news consumption sequences due to framing concerning both repeat consumption and viewpoint diversity with frame labels.
In an information system, a set of users interacts with a set of items . Each user ∈ has a consumption history , which consists of a sequence of ℎ consumed items ∈ by the user. For simplicity, we consider all user histories from the same specified time, thus omitting an additional time index (,,1 = ,1). To access the information, a user might be presented with a list of potential items ∈ given by the function ℛ(, ). The function takes as input the Notation (· ) ℎ, , , ,
Description # in MIND-small set of users, represented by their user IDs: ∈ | | = 50, 000 set of items, represented by their item IDs: ∈ || = 51, 282 set of user histories, = ⋃︀∈ || = 49, 108 set of impression logs from the function ℛ(, ) || = 156, 965 set of click logs from the function (, ) || = 236, 344 set of label spaces; ∈ ⋃︀∈1,2,... ; # category labels: || = 17 mapping function for label space from a set of mapping functions (· ) ∈ lengths of specific item set (i.e., logs of history, impression, and click, respectively) item lookup from logs (i.e., , , and ) with providing required indices user and a specific time . After evaluating the potential items in ℛ(, ), a user then interacts (i.e., consumes) one (or more) items of the list of potential items ∈ ℛ(, ). This interaction can be formalized by the function (, ). The number of interacted items is denoted by , which is = 1 in most cases (i.e., where we can omit the positional index: (, ) = {,}). The three described equations are thus given by (a summary of the main symbols is in Table 1): = [,1, ,2, . . . , ,ℎ ] ℛ(, ) = [,,1, ,,2, . . . , ,, ]
(, ) = {,,1, ,,2, . . . , ,, }
Each item contains some content and can additionally be assigned some metadata, such as labels. For instance, we can assign a category label to each item based on its content () = , where (· ) is the mapping function from the content to the label space from a list of potential categories ∈ . Note that the system can have multiple label spaces = 1, 2, . . ., each with their corresponding mapping function. Consequently, we can transform the previous equations to the label space for analysis, as shown in Figure 1: = [ (,1), (,2), . . . , (,ℎ )] = [,1, , ,2, , . . . , ,ℎ , ] ℛ (, ) = [ (,,1), (,,2), . . . , (,, )] = [,,1, , ,,2, , . . . , ,,ℎ , ] (, ) = { (,,1), (,,2), . . . , (,, )} = {,,1, , ,,2, , . . . , ,,ℎ , }
We employ a two-step approach to identify biased behavior patterns regarding framing in the
MIND dataset [
The MIND dataset [
We use a two-step procedure to construct the label sequences. First, we use metadata assigned
to the items to construct sequences of categories. Second, we extract framing representations
from the textual data (specifically the titles, as the short text is partially incomplete). Here,
we employ the FrameFinder library [
Media Frames: For the media frames: we use the facebook/bart-large-mnli model for zeroshot learning [34, 35, 36] with label definitions from the media frame corpus [ 37]. This model transforms the textual data to label probability scores, where we take the label with the maximum score. It is thus similar to the categories, but the labels are computed automatically rather than assigned manually. The set of 15 labels comprises: ’morality’, ’economic’, ’quality’, ’capacity’, ’crime’, ’security’,’health’, ’political’, ’public’, ’other’, ’cultural’, ’fairness’, ’policy’, ’legality,’, and ’external’.
Moral Frames: We use the sentence-transformers/all-mpnet-base-v2 encoder model [38, 39] to extract the moral frames with the definitions derived from the moral foundation theory [ 40, 41]. Here, the textual data is transformed into alignment scores, which can be positive or negative, as each dimension is formed by an antagonistic label pair. Therefore, we take the maximum absolute value with a corresponding label (i.e., positive or negative, depending on the original sign). This forms a set of 10 labels: ’authority’, ’cheating’, ’subversion’, ’degredation’, ’harm’, ’fairness’, ’care’, ’betrayal’, ’loyalty’, and ’sanctity’.
Semantic Frames: The model Iseratho/model_parse_xfm_bart_base-v0_1_0, which is a copy on Hugging Face of an AMRLib2 model. The model is based on BART using abstract meaning
Name Specific All same Alternating All diferent Encased Random
Example sequence [, , , , , ] 0.2 0.5 0.4 if || ≥ 6 [, , . . . , ] 1.0 1.0 0.0 [, , , , . . . , , ] 0.0 0.5 1/(|| − 1) [, , , . . . , ] 0.0 0.0 1.0 [, , , . . . , , ] →∞ = 1 0.5 1/(|| − 1) [(), (), →∞E[] →∞E[] →∞E[] . . . , ()] = 1/|| = 1/|| = 1 0.6 1˙ 0.0 0.5 →∞ = 1 →∞ = 0 →∞E[] = 1− (1/||) representations [42, 34] that transforms texts to semantic graphs comprising semantic frames3. From the semantic graphs, we extract the most pronounced frames. Due to the large size of the label space, we only consider frames that appear at least 200 times at the root (i.e., the most pronounced position). The resulting set contains 23 frames: ’say-01’, ’possible-01’, ’report-01’, ’cause-01’, ’die-01’, ’find-01’, ’have-degree-91’, ’watch-01’, ’contrast-01’, ’get-01’, ’arrest-01’, ’belocated-at-91’, ’charge-05’, ’open-01’, ’show-01’, ’kill-01’, ’have-03’, ’reveal-01’, ’recommend-01’, ’announce-01’, ’want-01’, ’close-01’, and ’win-01’. We then use the first frame of the set in the serialized form of the graph. If none of the frames are present, we insert a special ’other’ frame (similar to how the media frames have an ’other’ label) instead.
For the behavior analysis, we use two metrics each (one coarse- and one fine-grained) as a proxy
to measure repeat consumption behavior and viewpoint diversity, respectively. All metrics are
normalized to fall in the range of [
The metrics are defined to work on arbitrary sequences containing categorical data. In the most basic case, we evaluate the sequence of a user’s history of a particular label space, i.e., = . For simplicity, we omit the details of the indices besides the positional index (i.e., ,ℎ , ] becomes [1, 2, . . . , ]). Besides, we use 1 as the indicator [,1, , ,2, , . . . , function, which returns 1 if the is true and 0 otherwise.
Direct Repetition Ratio (DRR) measures the ratio of sequential item pairs having the same labels.
− 1 () = 1/( − 1) ∑︁ 1=+1 =1 (1) 3The representation also contains additional data beyond the scope of this work. The list of frames is available at: https://propbank.github.io/v3.4.0/frames/
When considering the example sequences in Table 2, we observe the in the specific example sequence one out of five sequential pairs is a direct repetition (i.e., 1/5). Note that higher order patterns (e.g., alternating sequences) do not impact the value. Therefore, singular outliers (e.g., in the encased sequence) will only marginally afect the value. The convergence behavior of random sequences depends on the size of the label space.
Reciprocal Repeat Distance (RRdist) measures the average distance between neighboring repetitions (i.e., same labels while every label between them is diferent) and is normalized by the reciprocal value. Therefore, it can be seen as a sort of probability score that labels are repeated.
() = ∑︀=−11 ∑︀
=2 1<,=∧̸=,∀,<< ∑︀=−11 ∑︀
=2( − )1<,=∧̸=,∀,<<
Concerning the specific example in Table 2, has a distances of one and two, while has a distance of three, which results in an average distance of two (i.e., reciprocal value of 0.5). Note that metric capture higher order patterns, such as both the alternating and encased sequence having a distance of 0.5. In the former case, the distance is always two, while in the latter case, − 3 times a distance of one and one time a distance of − 1 resulting of − 2 repetition − 2 events (i.e., (− 3)* 1+1* (− 1) ). Similar to DDR, the limit of a random sequence approaches the reciprocal value of the label space.
Uniqueness Index (Uniq) determines how much of unique labels are present compared to the theoretical maximum. The maximum depends on the sequence length and label space and is bounded by whichever is smaller. Therefore, if | | < , then the maximum is reached when all labels are present, whereas if < | |, the maximum is reached when all labels are diferent . () =
|{}| − 1 (||, | |) − 1 (2) (3) (4)
In Table 2, the specific sequence is (3 − 1)/(6 − 1) as three of potentially six labels are present. If all items are the same, then the minimum of zero is reached (which is why one is deducted from both the enumerator and denominator). The value tends towards one for long random sequences. Therefore, the metric is a form of coverage on the sequence level rather than system level.
Distribution Imbalance (Gini) uses the Gini index, which considers the probabilities of label occurrence. Therefore, uniform distribution lead to higher values than skewed distributions. () = 1 − ∑︁ ()2, = ∈
1 ∑︁ 1 || =1 =
The specific example of Table 2 is thus the result of 1 − ((1/6)2 + (2/6)2 + (3/6)2). Gini is 0 with all same sequence, has 0.5 with two labels equally distributed (e.g., alternating sequence), and tends towards 1 as long sequence of all diferent labels. Similar to DDR, singular outliers do 10 1 100 100 100 100 (a) DRR ECDF Plot (b) RRdist ECDF (c) Uniq ECDF (d) Gini ECDF not significantly afect the outcome on long sequences (e.g., consider sequences, the value depends on the size of the label space. encased). For long random
We want to answer the following research questions by analyzing their corresponding label sequences (denoted by →): RQ1: How is the repeat consumption behavior and viewpoint diversity of frames compared to categories? → = : the set of user histories; also used for comparison in RQ2 and RQ3. RQ2: What is the interplay between frames and categories?
Whether more of the same frames are consumed in per-category sub-sequences?
→ / : the subsets from the user histories per category
RQ3: What are the efects of framing with regard to (a) retrieved, i.e., with impressions,
(→ ⊕ : the user history enhanced with a single impression)
and (b) consumed, i.e., with clicked, content
(→ ⊕ : the user history enhanced with a single click)?
RQ1: Comparison of Framing Behavior
Concerning the user history , we observe that categories and media frames are closely related
(Table 3 and Figure 2), which can be the result of the set of media frames being defined in terms
of topics (for which they were already criticized [
User History (RQ1, | | = 49, 108, = 18.85)
Per Category (RQ2, |/ | = 687, 054, / = 6.84)
With Impressions (RQ3a, |⊕ | = 5, 723, 002, ⊕ = 37.26) Categories ()
In sum, the repeat consumption and viewpoint diversity are frame-specific. Moral frames appear to be consumed in a more balanced way compared to semantic frames. Furthermore, categories and media frames seem to be closely related in terms of consumption behavior. Therefore, we investigate this relation in RQ2.
RQ2: Relation between Categories and Framing All three types of frames are correlated with the categories on all four metrics (plots are provided in the code repository). The consideration of the subsequences per category (/ ) leads to statistically significant changes in all metrics and frames (Table 3). Specifically, the repeat consumption always increases (both and ), while always decreases. In fact, this results in the highest (bold in semantic frames) and second highest (underlined in media frames) values overall in terms of repeat consumption and similarly the lowest and second lowest for . Therefore, the consumption behavior appears less balanced when considering individual categories. In other words, a balanced consumption behavior regarding framing appears to be partially the result of a more diverse set of categories consumed. Interestingly, the , while still afected, does not show such a tendency. Moreover, it even increases for media and semantic frames, thus indicating a still broad range of frames in these shorter sequences.
Overall, we can conclude that categories play a vital role in the consumption behavior of frames, as the same frames are consumed even more repeatedly. As information systems are also prone to narrow the content shown to users [43], e.g., by repeatedly recommending similar items in terms of categories, we investigate these efects more closely in RQ3. RQ3: Framing Efects in Information Systems To start, we investigate whether shown and click items are a mere repetition of the last item’s label in the user history (i.e., whether increases in Table 3). Apparently, the last category is not used to determine the shown items, while the users themselves, more often than not, stick to the same category. Here, user intent might play a role (see [44] for an example of intent modeling in sequential recommendation), which is beyond the scope of the current study. Nevertheless, the system seems to repeat the media and semantic frames, which also afects the user click behavior. The efect is more pronounced in ⊕ compared to ⊕ , which might indicate that the system is the source of the bias rather than the users themselves. Interestingly, moral frames do not seem that afected (no statistically significant change of < 0.0005) and stay low (being the lowest values of overall). In comparison, decreases for both sets of sequences, while viewpoint diversity tends to increase. This efect is most pronounced regarding the moral frames, especially on the click behavior. In general, the click behavior is more afected regarding , , and . One outlier here is the uniqueness of media frames, which decreases and is more pronounced in the impressions rather than click behavior.
The results suggest that, although information systems tend to promote sticking to the same type of content, the efects on consumption behavior might be a net positive, as users could be supported in balancing their media consumption. Please note that the current study cannot deduce long-term efects and therefore urges for future work.
In the present, study we relate the framing of content to consumption behavior in information systems. Herein, we investigate the repeat consumption behavior and viewpoint diversity for three types of frames (i.e., media, moral, and semantic frames). Our findings suggest the relation to behavior is diferent per frame type, with media frames closely following categories. The repetition of frames also increases when investigating the categories separately, whereas the diversity tends to increase due to the efects of information systems.
Our study has broad implications for the design of information systems, as it suggests considering user behavior within particular types of content rather than diversifying through recommending a broad spectrum of types.
Limitations. Our study has two main limitations. First, the scope of the study is narrow, as we consider only a single dataset in the news domain, which was designed for recommendation research, with three specific models. Second, we performed a simplified analysis for better comparison, which omitted fine-grained details in content (e.g., the graph structure of semantic frames), metrics (e.g., the influence due to number of labels), and behavior (e.g., user intent).
Future Work. We hope our work sparks interest in considering framing as a form of bias in information systems. Most of all, we call for the development of debiasing methods concerning user behavior due to framing. Specifically, we see personalized user interfaces that support a balanced consumption diet, e.g., through transparency, as a promising research direction for future work. polarization in social media: Method and application to tweets on 21 mass shootings, arXiv preprint arXiv:1904.01596 (2019). [18] M. Reiter-Haas, S. Kopeinik, E. Lex, Studying moral-based diferences in the framing of political tweets, arXiv preprint arXiv:2103.11853 (2021). [19] C. Shurafa, K. Darwish, W. Zaghouani, Political framing: Us covid19 blame game, in:
International Conference on Social Informatics, Springer, 2020, pp. 333–351. [20] J. Piskorski, N. Stefanovitch, G. Da San Martino, P. Nakov, Semeval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup, in: Proceedings of the the 17th International Workshop on Semantic Evaluation (SemEval-2023), 2023, pp. 2343–2361. [21] Q. Liao, M. Lai, P. Nakov, Marseclipse at semeval-2023 task 3: Multi-lingual and multi-label framing detection with contrastive learning, arXiv preprint arXiv:2304.14339 (2023). [22] M. Reiter-Haas, A. Ertl, K. Innerebner, E. Lex, mcpt at semeval-2023 task 3: Multilingual label-aware contrastive pre-training of transformers for few-and zero-shot framing detection, arXiv preprint arXiv:2303.09901 (2023). [23] B. Wu, O. Razuvayevskaya, F. Heppell, J. A. Leite, C. Scarton, K. Bontcheva, X. Song, Shefieldveraai at semeval-2023 task 3: Mono and multilingual approaches for news genre, topic and persuasion technique classification, in: Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), 2023, pp. 1995–2008. [24] V. Bhatia, V. P. Akavoor, S. Paik, L. Guo, M. Jalal, A. Smith, D. A. Tofu, E. E. Halim, Y. Sun, M. Betke, et al., Openframing: open-sourced tool for computational framing analysis of multilingual data, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2021, pp. 242–250. [25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). [26] E. Lex, D. Kowald, P. Seitlinger, T. N. T. Tran, A. Felfernig, M. Schedl, et al., Psychologyinformed recommender systems, Foundations and Trends® in Information Retrieval 15 (2021) 134–242. [27] D. Kowald, M. Reiter-Haas, S. Kopeinik, M. Schedl, E. Lex, Transparent music preference modeling and recommendation with a model of human memory theory (2024). [28] M. Reiter-Haas, E. Parada-Cabaleiro, M. Schedl, E. Motamedi, M. Tkalcic, E. Lex, Predicting music relistening behavior using the act-r framework, in: Proceedings of the 15th ACM Conference on Recommender Systems, 2021, pp. 702–707. [29] M. Moscati, C. Wallmann, M. Reiter-Haas, D. Kowald, E. Lex, M. Schedl, Integrating the actr framework with collaborative filtering for explainable sequential music recommendation, in: Proceedings of the 17th ACM Conference on Recommender Systems, 2023, pp. 840–847. [30] E. O’Brien, A mind stretched: The psychology of repeat consumption, Consumer Psychology Review 4 (2021) 42–58. [31] N. Tintarev, E. Sullivan, D. Guldin, S. Qiu, D. Odjik, Same, same, but diferent: algorithmic diversification of viewpoints in news, in: Adjunct publication of the 26th conference on user modeling, adaptation and personalization, 2018, pp. 7–13. [32] C. Baden, N. Springer, Conceptualizing viewpoint diversity in news discourse, Journalism 18 (2017) 176–194. [33] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38–45. [34] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, arXiv preprint arXiv:1910.13461 (2019). [35] A. Williams, N. Nangia, S. Bowman, A broad-coverage challenge corpus for sentence understanding through inference, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018, pp. 1112–1122. [36] W. Yin, J. Hay, D. Roth, Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach, in: 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Association for Computational Linguistics, 2019, pp. 3914–3923. [37] D. Card, A. Boydstun, J. H. Gross, P. Resnik, N. A. Smith, The media frames corpus: Annotations of frames across issues, in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2015, pp. 438–444. [38] K. Song, X. Tan, T. Qin, J. Lu, T.-Y. Liu, Mpnet: Masked and permuted pre-training for language understanding, Advances in Neural Information Processing Systems 33 (2020) 16857–16867. [39] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), Association for Computational Linguistics, 2019, pp. 3982—-3992. [40] J. Frimer, J. Haidt, J. Graham, M. Dehghani, R. Boghrati, Moral foundations dictionaries for linguistic analyses, 2.0, Unpublished Manuscript. Retrieved from: www. jeremyfrimer. com/uploads/2/1/2/7/21278832/summary. pdf (2017). [41] J. Graham, J. Haidt, B. A. Nosek, Liberals and conservatives rely on diferent sets of moral foundations., Journal of personality and social psychology 96 (2009) 1029. [42] L. Banarescu, C. Bonial, S. Cai, M. Georgescu, K. Grifitt, U. Hermjakob, K. Knight, P. Koehn, M. Palmer, N. Schneider, Abstract meaning representation for sembanking, in: Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, 2013, pp. 178–186. [43] T. T. Nguyen, P.-M. Hui, F. M. Harper, L. Terveen, J. A. Konstan, Exploring the filter bubble: the efect of using recommender systems on content diversity, in: Proceedings of the 23rd international conference on World wide web, 2014, pp. 677–686. [44] Y. Chen, Z. Liu, J. Li, J. McAuley, C. Xiong, Intent contrastive learning for sequential recommendation, in: Proceedings of the ACM Web Conference 2022, 2022, pp. 2172–2182.