The article presents the results of predicting user engagement with in-video ads using physiological sensor signals. Specifically, we examine pupil response and heart rate as possible predictors of user engagement. To this end, we conducted an experiment with 33 young participants (age M = 21.70, SD = 2.36; female = 68%) in which their psychometric (engagement score) and physiological responses (pupil dilation and heart rate) to four in-video ads were recorded. The ground truth for the ad engagement was collected using the User Engagement Scale Short Form (UES-SF), a standardized psychometric instrument for measuring user engagement. The UES-SF dimensions Aesthetic Appeal (AE) and Perceived Usability (PU) were used to calculate the combined User Engagement Score. Several machine learning classifiers were evaluated that used heart rate and pupil response as predictors of engagement. The best overall results were obtained by Random Forest Classifier ('weighted' F1 score = .76, Precision=.84, Recall=.95), Logistic Regression and Support Vector Classifier (the latter two with the same scores: 'weighted' F1
Advances in technology and digital media advertising have enabled new approaches to
measuring consumer engagement and exposure to online marketing [
The goal of the research presented here is to assess heart rate and pupil dilation as predictors
of user engagement with in-video advertising used by online streaming services. The ground
truth for the ad engagement was collected using the User Engagement Scale-Short Form
(UESSF) [
The contribution of the presented study is to examine the potential of physiological cues as a
measure of advertising exposure. It is well known that advertising evokes emotional arousal and
triggers cognitive processes in consumers [
In the following, we briefly present related work. Next, we present the experimental design and procedure for collecting user responses and the selection of materials and tools. The statistical analysis and the evaluation of machine learning models are presented in the Results section. The article concludes with a summary of the main results and possible directions for future work.
Contemporary advertising strategies and services require rapid and accurate insight into consumer engagement and exposure to media content. Fast and eficient measurement methods with little interference with the observed subject are preferred. We hypothesize that measuring physiological signals known to be associated with emotional arousal and cognitive processes could provide a good basis for unobtrusive measures of engagement and exposure to media advertising.
Richardson et al. examined engagement with video and audio narration using wrist sensors
that measure heart rate variability, electrodermal activity, and body temperature [
Recent research has shown that pupil dilation can be used to assess cognitive processes in
participating subjects. P. van der Wel [
Much of the evidence cited above suggests that among physiological measures, pupil dilation and heart rate are good choices for detecting fluctuations in emotional arousal, mental activity, and cognitive processes. In the presented study, we focus on the putative efects of the two on advertiser engagement and media use.
A pre-selection of 12 videos and 12 ads was taken from the online streaming service YouTube. These materials were selected in collaboration with three marketing experts from The Nielsen Company to address diferent levels of engagement. All materials were in English and aired in the United States.
A crowdsourcing study with Clickworker1 was conducted to determine engagement with YouTube’s 12 videos and ads. The ads were inserted into the videos at random positions, with combinations of diferent engagement levels. The goal of these combinations was to simulate the experience of ad-supported video streaming. Engagement was measured on a 5-point scale (How engaging is the ad?: none-medium-strong-very strong). Engagement scores were collected from study participants (N=360, age=18-24).
Based on the results of the crowdsourcing study, diferent video and in-video ad combinations were created based on the engagement scores. The final selection for the experiment was made in collaboration with Nielsen media experts. For the experiment, four combinations of ads and videos were created based on the engagement level and brand awareness criteria for the ads (2x2: known vs. unknown brand with higher and lower engagement scores). The number of combinations was limited to make the experiment feasible (see 3.2). The following four ads were used: Dior Joy Perfume2, Coca Cola3, Little Baby’s Ice Cream4, Waring Ice Cream Maker5.
Ground truth for user engagement was measured using the User Engagement Scale-Short Form
(UES-SF) [
Several physiological sensor signals were collected from the participants: eye-tracking data with the Tobii Pro Glasses 2 eyetracker (pupil dilation, saccades, fixations) and heart rate and electrodermal activity (EDA) with Empatica 3. In this article, we report pupil dilation and heart rate.
An experimental design was used in which all participants watched all four ads within their assigned shufle set. This was done to control for potential carryover efects from the preceding combination to the next, as the engagement response elicited by the preceding sequence could influence the participant’s response in the next sequence. Four sets were created, yielding four combinations (Set1: Ad1, Ad2, Ad3, Ad4; Set2: Ad1, Ad3, Ad2, Ad4; Set3: Ad4, Ad2, Ad3, Ad1; Set4: Ad4, Ad3, Ad2, Ad1). Note that the number of combinations was limited to four sets to keep the duration of the experiment per user still workable, but shufle ads so that no ad has the same preceding ad more than once.
Next, the four sets were randomly and evenly assigned (taking into account age and gender) to participants (N=44, age=18-24), with each participant being assigned only one set. Within each set, the four combinations of video ads were interspersed with a 2-minute break to further isolate possible carryover efects of the preceding combination to the next and to give participants a break.
The experiment was conducted in a controlled environment (the Lucami laboratory at the Faculty of Electrical Engineering) to ensure a uniformly lit and quiet environment. First, informed consent and demographic data were collected from each participant. The participants were familiarized with the goal of the experiment (to collect physiological data and engagement scores for the ads) and given time to familiarize themselves with the procedure. They then sat down on a sofa and watched each video sequence on TV. After viewing each ad, they were asked to rate their engagement with the ad (on a laptop) using UES-SF. Participants also recorded other aspects related to advertising exposure, including afective state (valence and arousal), brand familiarity and recall, and purchase intention. The average duration of the experiment was 45 minutes. The experiment was completed by 44 participants. Due to incomplete responses and errors in sensor measurement, a final sample of 33 participants (age M=21.70, SD =2.36; female=68%) was used for further analysis.
Within-subject normalization of pupil response data was performed to account for individual
diferences in pupil dilation between the participants. Pupil dilation and heart rate data were
analyzed directly; no additional features were extracted. For each individual participant, the
average pupil dilation across all four ads was calculated and then subtracted from the mean
values of pupil size for each ad. In this way, only the relative diferences in pupil dilation per ad
were recorded. Heart rate data were averaged, and median heart rates per ad per participant
were calculated. Outliers (outside the threshold SD =2.5) for both pupil dilation and heart rate
were imputed with the medians per participant and per ad. Next, both heart rate and pupil
dilation were normalized using MinMaxScaler (sklearn), heart rate data to a range [
Statistical analyses (ANOVA and pairwise t-tests) and classification using machine learning were performed in Python with the Scipy, Pinguoin, open-cv, and sklearn libraries. Visualization of the data was done using the matplotlib and seaborn libraries. The boxplots in Figures 1, 2, and 3 represent the minimum, first quartile, median, third quartile, and maximum of the visualized data.
The dimensions Aesthetic Appeal (AE) and Perceived Usability (PU) were selected for modeling user engagement. For each participant and for each dimension, the average scores were first calculated from the average scores of the respective items of the dimensions. Figure 1 shows the distribution of AE and PU across the ads. We can observe a strong negative correlation between the two dimensions (r(117)=-.67, p <.001).
AdID = 1
AdID = 2
AdID = 3
AdID = 4 Dimension
AA
PU Dimension
AA
PU Dimension
AA
PU Dimension
Next, both dimensions were summed and averaged to produce a combined engagement score, which was later used in machine learning. Statistical analysis examined diferences in participant engagement between the four ads. Summary statistics are provided in Table 1. The Kruskal-Wallis test revealed no significant diferences in engagement scores between the four ads (H=6.83, p=.078).
In our earlier work [15] we had reported preliminary summary statistics for each physiological dimension. The Shapiro-Wilk test showed that the data for both dimensions were normally distributed. A Pearson correlation coeficient was calculated to assess the relationship between [2.43 – 2.80] [2.75 – 3.04] [2.50 – 2.82] [2.69 – 3.00] 0.49 0.39 0.41 0.43 0.09 0.07 0.08 0.08
Mean pupil dilation and heart rate, and a weak negative correlation was found between the two sensor signals (r(117)=-.17, p=.08).
One-way ANOVAs were performed to determine whether the median of pupil dilation and heart rate (Figure 2) was the same between the four ads (Ad1-Ad4). For the heart rate (F=2.80, p=.043; M=.4429, SD =.1736, SE =.02, 95%CI=[.41, .47]), multiple comparison t-tests (Bonferronicorrected) revealed no significant diference between the ads. For pupil dilation (F=76.44, p <.001; M=-.037, SD =.28, SE =.026, 95%CI=[-.089, .014]), several significant diferences were found, as also visible in Figure 2(b). Multiple pairwise t- tests (with Bonferroni correction) showed significant diferences in the increase in pupil dilation between the following ads: Ad1 and Ad2 (T=-7.74, p <.001), Ad1 and Ad3 (T=-9.69, p <.001), Ad1 and Ad4 (T=-13.70, p <.001), Ad2 and Ad4 (T=-6.32, p <.001), and Ad3 and Ad4 (T=-6.88, p <.001). No significant diference in pupil dilation was observed between Ad2 and Ad3.
A moderate negative correlation was found between the engagement dimensions AE and PU and pupil dilation (AE and pupil dilation: r(117)=-.39, 95%CI=[.23, .53], p <.001, power=.99; PU and pupil dilation: r=-.36, 95%CI=[-.51, -.19], p <.001, Hedges’ g=.98). No significant correlations were found between heart rate and the dimensions AE and PU.
Some studies have reported that pupil dilation can be afected by both light and contrast [ 16]. Also in our case, the diferences in pupil dilation between the ads could be due to the diferences in brightness and contrast between the ads and not to the content of the ads.
To this end, we analyzed the brightness and contrast characteristics of the ads. The overall average brightness (median) and contrast (standard deviation) of an ad were calculated from the averages of the 1-second frames (first converted to grayscale) for each in-video ad. A strong positive correlation was found between brightness and pupil dilation, r(117)=-.67, p <.001. Although the results show a significant diference in overall pupil dilation, the brightness of the ad may not be the most important predictor of pupil size. As Figure 2(b) shows, the average brightness is diferent for all but Ad1 and Ad2. The connected red line shows the average brightness value for each ad, which is given here along with the standard deviation (average contrast). For example, while there is a significant diference in pupil dilation between Ad1 and Ad2, there is no diference between the brightness levels of the two ads (Ad1: M=.26, SD =.37; Ad2: M=.26, SD =.26). On the other hand, while there is not much diference in pupil dilation between Ad2 and Ad3, there is a significant diference in brightness (Ad2: M=.26, SD =.26; Ad3: M=.48, SD =1.8). While the diference in brightness between Ad4 (M=.6, SD =1.9) and Ad3 (M=.48, SD =1.8) is smaller than between Ad2 and Ad3, the pupil dilation is significantly larger for Ad4 compared to all the other ads. From these results, it appears that there is no independent efect of brightness on pupil dilation.
Several classifiers were used without optimizing the parameters of the models: Logistic Regression, Support Vector Classifier, Decision Tree, Random Forest, Gaussian Naive Bayes, and AdaBoost. Features ’Age’, ’Gender, ’Heart Rate, and ’Pupil Dilation’ were used as predictors, while ’Engagement Score’ was used as a binary target. The decision to model engagement as a binary score was due to the bimodal distribution of engagement scores. The midpoint of the 5-point scale was set as the threshold for the binary engagement classes: lower engagement class < 2.5 vs. higher engagement class > =2.5. Because the target distribution has a relatively large class imbalance (only 22% of the target represents the ’lower engagement’ class) and the data set is small, repeated stratified cross-validation (folds=10, repeats=5, fixed number of seeds) was used to preserve the distribution of samples for each target class. Consequently, model performance was evaluated primarily using F1 scores and ”weighted” F1 to account for class imbalance, as these provide more reliable evaluation metrics when data are imbalanced. Table 2 shows the evaluation metrics for each model. Logistic Regression, Support Vector Classifier, and Random Forest Classifier achieved the highest overall scores.
For the feature evaluation, the permutation importance metric was used because of the imbalaned class distribution of the target variable. Another advantage of the permutation-based estimation is the ability to assess whether individual feature is useful as a predictor for the test set. Permutation importance was calculated based on the Random Forest Classifier. Figure 3 shows the permutation importance of the predictors for the training set (a) and the test set (b). We can see how the importance of each predictor changes.
The evaluation metrics for several classifiers provide good results and show potential for predicting ad engagement based on pupil dilation and heart rate. However, the presented models still need to be evaluated on a larger number of ads to reliably assess their robustness and predictive power. An interesting finding of the presented study is the strong negative correlation between the dimensions AE and PU, which is consistent throughout all the ads. These diferences are partially lost in the combined engagement score. In our future work, we will evaluate both dimensions separately.
The results on the efects of brightness on pupil dilation are inconclusive, as no independent efect of brightness on pupil dilation was found across the ads. Further research is needed to evaluate the efects of brightness, again with a larger sample of ads and specifically in the context of the efects of ad exposure. in Public Health 5 (2017). URL: https://www.frontiersin.org/articles/10.3389/fpubh.2017. 00258. doi:10.3389/fpubh.2017.00258. [15] T. Dobovšek, Š. Bernik, G. Strle, Pupil dilation and heart rate as responses to ad exposure, in: Proceedings of the MEi: CogSci Conference, volume 16, 2022. URL: https://journals.phl. univie.ac.at/meicogsci/article/view/400. [16] J. Beatty, B. Lucero-Wagoner, The pupillary system, 2000.