Using memes is a popular method of online communication, including both lighthearted and dark humor as well as polarizing subjects. But what makes a meme o ensive? Does this depend on a meme's content or format? Does a meme o ensiveness vary across audiences? In order to answer these questions, we recruit 316 participants to score two sets of meme variants in terms of their o ensiveness. We augment this with interviews to extract additional insights.
An Internet meme (or just meme) is a multimodal medium of communicating a certain message using the juxtaposition of textual and visual formats. This commonly takes the form of a brief comment over an image referencing pop culture, politics, or social issues. An example of a meme is shown in Figure 1.
Memes have been very widely used in social media of various types and in
innumerable languages [
Anecdotally, memes are quite e cient at delivering a complex message in a succinct way. Generally, this is due to the mix of text and visual content (in image or video format), which allows for almost immediate portrayal of context and illustration of contrast. This also allows low-e ort embedding into trending phenomena, as well as crossing cultural and language barriers.
The signi cant role of memes in modern online culture and society in general
is well documented [
We carry out a randomized controlled trial to answer the above questions. We recruit a total of 316 participants to review two di erent groups of potentially o ensive memes, and ask them to quantitatively score their reaction towards them. We also invite them to provide qualitative feedback. Consequently, we are able to draw some fundamental insights into the nature of memes. This is the rst study of its kind to try to isolate, using mixed methods analysis, the characteristics of a meme that makes it palatable or o ensive to di erent audiences.
In the remainder of this paper we will be using politically incorrect and potentially in ammatory language as and when context dictates. Section 2 brie y surveys related work. Section 3 describes our methodology for collecting memes, recruiting participants, and designing the survey. Section 4 presents the results of our study. Section 5 re ects on the results and discusses their wider implications. Section 6 concludes and outlines future work. 2
Memes play a signi cant role in modern online culture [
Various works have been published on the spread of memes in online
media. Shifman [
Yoon [
Despite all of this interest, however, our work is the rst (trial or observational) to dive into why some memes could be more o ensive than others. 3
We utilized a set of `real' memes collected from Reddit, and complemented them with synthesized ones in order to vary a certain feature of the meme whilst controlling the others. We designed a survey to collect data on perceived emotions from a large number of user of di erent backgrounds. 3.1
We collected a set of memes from the /r/ImGoingToHellForThis subreddit where o ensive content is typically posted. The memes, collected in NovemberDecember 2019, covered a variety of sensitive subjects such as politics, religion, and race. We attempted to select a variety of meme formats to appeal to di erent audiences. However, we aimed not to choose memes that are highly topical in order to minimize the e ect of context or trendiness on meme perception.
We focus in this study on the two main features that constitute a meme: image format and textual message. In order to identify the degree to which each can play on perception of o ense, we vary one while keeping the other the same as in the original meme collected from Reddit. We used MemeGenerator.com to alter either the format or text to include in the survey variants. When text is changed, a message from another /r/ImGoingToHellForThis meme is used. Examples are shown in Figures 2{3, with the full list included in Table 1. All memes could be accessed at https://github.com/kieranhill97/Memeology/.
7 Crying woman 8 Troll quotes
10 I am speed 11 Idadmonn'tvewgaenttabalneys
Theme Social niceties Atomic bombings
Meme I'm kind of retarded Pro gamer move
Theme Web culture Murder, Gaming In delity, Domestic Keep your secrets Abandonment abuse Sexuality Race
Starter packs
Assassination (Lincoln) Poverty Suicide Disability (child) Troll quotes I worry about you sometimes, Candice I am speed I don't want any damn vegetables
Assassination (Kennedy) Poverty Race Disability (sibling) Atomic bombings, Race Gender, Trump 12 Plants vs Zombies Atomic bombings Rice Krispies 13 She can't do that Gender, Religion 14 Roll safe Death 15 tWhahnat2's4 funnier Nazism 16 Careful, he's a hero Gender 17 Daily struggle Slavery 18 aWcoamt an yelling at Gender 19 Nut button Terrorism 20 Surprised Pikachu Terrorism She can't do that When Your
Money Went tWhahnat2's4 funnier 9/11 Careful, he's a hero Race, Murder Daily struggle Murder, Cancer Who would win Highway exit turn Surprised Pikachu
Gender Terrorism Abuse We used a survey questionnaire as the tool to collect input speci cally due to its inherent nature of disconnecting the participant from the researcher and also from other participants (as in a focus group). The aim was to help the participants answer freely without feeling implicit pressure to select a politically correct answer when faced with a polarizing topic.
The survey had 2 variants, each with 20 memes. The size was calibrated to sustain participant interest throughout the survey and elicit earnest responses. Upon start, each participant was assigned one of the 2 sets based on a simple round robin protocol. The corresponding questions in each survey matched each other in either visual format or textual content. Variation was made between sets to identify the characteristic of the meme that has potential to cause more o ense. Each question asked the participants how o ended the meme made them feel on a 5-point Likert scale between `Not o ensive at all' and `Extremely offensive'. Subjects were informed at the beginning of the survey that the study was intended to o end and that the memes do not represent the views of the researchers. The survey was IRB approved.
At the end of the survey, participants were asked to indicate if they are willing to answer follow-up open-ended questions. 3.3
Between February 23rd and 26th 2020, di erent social media (namely Reddit, Facebook, and Twitter) were used to spread the survey in an attempt to make the group of participants as diverse as possible in terms of online culture. We also recruited locally on our university campus. Demographic information was collected for summary statistics but all responses were anonymized.
Participation was limited to a single response linked to a Google account in order to prevent multiple responses from skewing the data. This restriction limited participation to those with a Google account and were willing to use it to sign in. There were 550 visits to the survey landing page, 320 of which completed the survey (a response rate of 57.5%). 4 results had to be disregarded due to incomplete data, resulting in 316 valid responses. Of these, 37 participants provided further input in the form of answers to our open-ended questions. 4
In this section we present and examine the results of the study. 4.1
First, we inspect how our round robin assignment of survey a ected the
participant demographics of each survey set. This resulted in fairly equivalent sets, as
is depicted in Figures 4{5. The only observable disparity is in the frequency of
meme usage, but the di erence is not statistically signi cant.
To systematically identify the di erence in scores between corresponding
questions of the two surveys (shown in Figure 6), we used the t -test and
MannWhitney-Wilcoxon methods. Both are non-parametric and produce statistically
signi cant results [
Participant response seems to be associated with the message of a meme more than with its visual aspects. This can be particularly observed in Q7, Q8, Q11,
Q18 and Q19 where the message was constant while the meme changed. Exceptions are Q5, Q9, and Q14, which can be attributed to slightly awkward meme use when the same message was retained without adjustment while the meme changed. In e ect, the framing in most cases seems less e ective than the content.
Similarly, this is also observed in instances where the meme is reused with a di erent message (such as Q3, Q13, and Q17) where it can be signi cantly more or less o ensive depending on the content. Interestingly, though, 86% of those interviewed after the survey provided input that agreed with the statement \the format of a meme can change its meaning", which is contrary to what the data shows us. 4.4
The theme of tragic events, especially large-scale ones (e.g., the atomic bombings of Hiroshima and Nagasaki) are consistently received as being more o ensive. This is evident from the signi cant changes in Q2, Q6, Q12, and from the lack thereof in Q15 where both themes were large-scale tragic events (The Holocaust and 9/11, respectively). Moreover, combining more than one o ensive theme tends to garner higher scores signaling higher level of perceived o ense. Examples include Q3 and Q12. An exception is Q17 where the theme of slavery was more o ensive than murder and cancer combined.
Some themes were deemed to be less o ensive than expected, such as religion. However, we avoid to draw any conclusion on this as we did not collect nor control for participants' religious and spiritual beliefs. 4.5
We observed some associations between participant demographics and response to memes. Although no extremely strong correlations were detected, a few deserve remark (Figure 8). Most obvious, perhaps, is that age is a fairly good indicator of the frequency of meme use. A stronger correlation, however, exists between a participant's gender3 and how easily o ended they are by the memes. Moreover, gender is also moderately correlated to varying Likert scores using median absolute deviation, a robust measure of variability. Essentially, female participants were more likely to use the full range of the Likert scale over the various survey questions, while many male participants tended to score using only part of the scale, e.g., between 1 and 3. Non-binary gender participants showed no distinguishable tendency. Elsewhere, participants who signi ed that they are easily o ended did indeed score higher and with more variability. Finally, participant location showed no signi cant correlation with scores. 5
We now discuss the implications and limitations of the above results, and comment on threats to validity. 5.1
Based on our ndings in Section 4.3, the automated detection of potential o ense
and, similarly, misinformation in memes might be easier than anticipated as
it is more dependent on the textual rather than visual elements of a meme.
Fortunately, textual analysis methods are more advanced and readily available
3 After removing those who withheld their gender (7), but keeping non-binary answers.
than visual ones. Thus, we propose the development of systems to predict the
e ect of a meme in terms of its o ense as an important future direction. This
would be useful for detecting misinformation as well as commercial uses such
as advertising campaigns. Most e orts to date on automated analysis of memes
have focused on distribution, identi cation and clustering [
Some attribute the success of memes to its reliance on humor as a tool, which
allows its creator to `soften the blow' of the contained message [
This inherent property has the potential to be exploited to mask the o ensive disposition of the message, or equally political critique or misrepresented information within. In other words, using memes makes it slightly more easier to `swallow the pill' of a message that is otherwise less palatable due to its o ensive or misleading character.
However, there seems to be a limit to this capacity to subvert. Our results are encouraging as they indicate that it is more di cult to dress a overtly o ensive message with the humorous visual elements of a meme. More work is needed to investigate if this also applies to other emotions that a meme can trigger, and the potential to carry misinformation. 5.3
In our study, we do not aim to establish any causal relationships hence threats to internal validity are minimal. Survey results are susceptible to be swayed by any biases that the participants already have. This aspect is di cult to control in an anonymous online survey without an additional step to calibrate perspectives, and by signi cantly increasing sample size. Although we comment on the correlations we observe, we are cautious not to generalize in order to mitigate threats to external validity.
There are some threats to construct validity due to the nature of how memes are created, and we did notice this e ect with a few memes (see Section 4). Furthermore, memes seldom live in isolation and are typically part of a wider discussion that is context- and community-speci c. As such, there will always be a degree of threat to construct validity with synthesizing memes purely for research purposes. 6
Through a randomized controlled trial, we studied how di erent memes are received as being o ensive or otherwise. We established an understanding of the importance of a meme's content as opposed to just its visual and cultural aspects. We also uncovered some relationships between user demographics and how likely they are to being o ended by a meme. As such, our work helps address a gap in terms of how sensitive or polarizing messages could be interpreted by di erent audiences. This knowledge is important not just in understanding how the potential of propagating certain messages through the medium of memes, but also how controversial opinions (due to political, racial, religious, etc. perspectives) can have varied e ects. We also discussed the degree to which a meme could be used as a delivery vehicle to normalize what is arguably controversial, extreme, or misleading. However, this area requires more research which is a future target of ours.
There are numerous other directions for future work. Further investigation could be made into the relationship between the timeliness of a meme and its o ense e ect. How this may vary across topics, online communities, and social platforms are all interesting question. Another interesting direction is to investigate the possible dependency of the `memorability' of a meme and its inherent properties.