Deepfakes videos are becoming increasingly believable, and their pervasiveness poses critical ethical and technical concerns regarding the ability of humans to detect them. It is currently unknown to what extent our human preferences and prejudices impact human deepfake detection ability. The initial phase of our project presents a survey experiment (phase 1 of our study surveyed 1,000 participants) where people are exposed to short video clips, not knowing the content might be fake. Survey participants are sampled through a Qualtrics survey panel to match the demographics of U.S. social media users in 2021 [1]. Participants are subsequently asked to guess the demographics (e.g., age, gender) of the persona of the video and whether each video watched is real or a deepfake. We measure the accuracy rate at which survey participants of diferent demographic backgrounds are duped and by what types of self-similar or self-dissimilar deepfake personas. Our project explores four primary questions. (Q1) Are humans better at correctly classifying deepfake videos if primed about deepfake content before exposure? (Q2) self-similarity bias: Are there categories of humans better at detecting a deepfake video if the persona in the video matches their own identity? (Q3) self-dissimilarity bias: Are there categories of humans better at detecting a deepfake video if the persona in the video is diferent from their own identity? (Q4) Prior knowledge bias: Are human viewers better at detecting a deepfake video if they know more about deepfakes or use social media more frequently? There is a growing body of work on the distributed threats in online social networks: from leaky data [2] and group privacy concerns [3] to hate speech [4], misinformation [5] and detection of computer-generated content such as deepfakes [6]. We tackle this last example by exposing human subjects to real and deepfake video clips to study the relationship between human demographics and the perceived demographics of personas depicted in video content. To determine the variables that influence whether a participant's guess about the status of a video (real or fake) is correct, we ran a Matthew's Correlation Coeficient (MCC). MCC is typically used for classification models to test the classifier's performance. Here we are treating human participant subgroups as classifiers and are measuring their performance with MCC. Given the counts of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), the MCC is defined as
( × )− ( × ) √( + )( + )( + )( + )
Out of the 1,866 total videos watched (half of the videos shown to our participants are
deepfakes), 36% were deepfakes that duped our participants. The overall accuracy rate of our
participants was 50% (where accuracy = (TP+TN)/(TP+FP+FN+TN) compared to 66% in previous
work with primed subjects [
The overarching takeaway of the survey can be summarised as follows. (Q1) If not primed, humans are not particularly accurate at detecting deepfakes. (Q2-Q3) Accuracy varies by demographics, and younger participants were much more accurate overall, but older demographics see a dramatic increase in accuracy when classifying videos that they identified as self-similar. (Q4) Accuracy increases with frequent social media usage, perhaps explaining our results from the previous question.
In essence, our results simply suggest that diferent age groups performed diferently when
classifying a video subject as either a real human or as a deepfake persona. This, in turn,
implies that populations of social media users have a heterogeneous susceptibility to video
misinformation. To explore the impacts of these results, we integrate some of our findings
into a mathematical model to understand how deepfakes spread on social networks with a
diverse population. The model uses a network with a heterogeneous degree distribution and a
structure inspired by the mixed-membership stochastic block model with modules like echo
chambers and bridge nodes with diverse neighborhoods. We track individuals based on their
demographics, here just abstract classes 1 or 2 that could represent younger and older people.
We also track their state: duped (or infectious) and non-duped (or susceptible). We also track
the demographics of their neighbor (tagged degrees and ℓ) to know their role in the network.
Individuals get duped by their duped neighbor at rate dependent on their demographic class ,
and duped individuals can get corrected by their susceptible neighbors at rate . The dynamics
can be tracked using a heterogeneous mean-field framework [
where mean-field quantities like 12 are calculated as ∑︀, 2,/ ∑︀, (2, + 2,) and represent the probability of an infected ( ) or susceptible () neighbor given the demographics involved (indices {, } ∈ {1, 2}).
For a given specific set of parameters, we can use our model to ask who is afected by misinformation based on their place in the network (bridge or modular nodes) as well as their demographics (type). We show two diferent model runs in Fig. 2(a and c). Overall, we find that in populations with heterogeneity in susceptibility varying from small (20% variation in susceptibility) to large (100% variation), more susceptible nodes benefit greatly from being bridges between communities. In a nutshell, diverse friends can correct each other’s blind spots.
Altogether, lifelikeness may not be an objective measure we can apply to all humans and lifelike personas in the same way. Humans hold a host of biases, and recognition of lifelikeness may depend on the biases and prior experiences of the viewer as well as on their social network.
Future work will incorporate the second phase of our study and model. We hope this study
is a step towards understanding the impacts of social biases in an emerging societal problem
with many multilevel interdependencies. We hope it will contribute to the timely literature
on the interplay between human biases and machine-generated content. As deepfakes begin
to deceive viewers at greater rates, it becomes increasingly necessary to understand who gets
duped by deepfakes and how our biases and those of our social circle impact our interaction
with video content. Deepfakes also call into question numerous ethical issues such as the
power of video evidence in legal frameworks [