Multisensory integration (MSI) refers to the brain's ability to synthesize information from different sensory modalities, facilitating a coherent perception of the environment [1]. This process is critical for navigating daily life [2], impacting attention [3], decision-making [4], and motor coordination [5], [6]. Traditionally, MSI has been studied through the lens of external sensory inputs, such as visual and auditory stimuli, but emerging evidence suggests that internal states-such as arousal, emotion, and motivation-also play a crucial role in shaping how multisensory information is processed [7], [8], [9].
Arousal, in particular, has been identified as a key internal factor influencing perception, with heightened states of arousal potentially enhancing or disrupting the sensory inputs depending on context [10]. However, the specific mechanisms underlying the interaction between internal arousal and external sensory stimuli remain unclear. This study aims to explore how internal arousal, induced through visual cues, influences MSI performance in the Sound-Induced Flash Illusion (SIFI), a widely used paradigm for studying audiovisual integration [11], [12]. In addition to investigating the role of arousal, we assessed individual differences in sensory sensitivity using the Highly Sensitive Person (HSP) scale [13], which may mediate the effects of internal states on MSI. zahra.azizi@dcu.ie (Z. Azizi); jason.chan@ucc.ie (J. Chan); tomas.ward@dcu.ie (T. Ward); a.setti@ucc.ie (A. Setti) 0000-0002-5181-4958 (Z. Azizi); 0000-0002-4663-5779 (J. Chan); 0000-0002-6173-6607 (T. Ward); 0000-0002-9741-2559 (A. Setti) MSI is crucial for many aspects of daily life, including understanding speech in noisy environments, coordinating motor actions while driving, and maintaining balance while walking. This research not only enhances our understanding of MSI but also holds significant clinical potential. For individuals with MSI deficits, such as those with Parkinson's disease, it could guide the development of therapies to improve motor coordination, cognitive function, and overall quality of life. Additionally, insights into age-related changes in MSI could support innovations like fall prevention systems and cognitive training programs for older adults. Beyond healthcare, the findings extend to mental health and technology, offering applications such as enhancing virtual reality systems through improved multisensory feedback integration.
Our study involved 23 participants, recruited through Prolific, an online platform commonly used for academic research that allows for access to a diverse pool of participants [14]. Informed consent was obtained from all participants prior to the start of the experiment. The experiment was coded using PsychoPy, an open-source software for running behavioral science experiments [15]. The data collection was conducted online, utilizing Pavlovia (Open Science Tools, Nottingham, UK, https://pavlovia.org/)-a platform that enables remote experiments designed in PsychoPy to be deployed and run in browsers. Additionally, Qualtrics was used to administer the survey component of the experiment (Qualtrics, Provo, UT, https://www.qualtrics.com), ensuring ease of access and user-friendly interaction for participants. Ethical approval was granted by the Ethics Committee at the School of Applied Psychology, University College Cork. All data were securely stored and handled according to ethical standards and protocols.
In this arousal-cued SIFI task, participants were exposed to either a disgust or neutral face before each trial, aiming to manipulate their arousal levels. Each trial began with a briefly presented face stimulus, followed by a fixation cross. Participants then viewed a visual flash (a white circle) presented below the fixation cross, accompanied by an auditory beep played through participants' speakers. The trials could be congruent, where the visual and auditory stimuli matched (1F1B or 2F2B), or illusory (2B1F), in which a beep was presented without a corresponding flash. In the 2B1F condition, the beep could occur either before or after the visual flash (Figure 1). The stimulus onset asynchronies (SOAs) in these trials were varied and could be −200 ms, −150 ms, −50 ms, 50 ms, 150 ms, or 200 ms. Unisensory trials (0B2F), where only flashes were presented without auditory beeps, were presented at a single SOA of 50 ms. Participants were asked to report how many flashes they perceived and to rate their confidence in their responses. The experiment consisted of 288 trials, balanced between neutral and disgust face cues, SOA, and congruent/illusory conditions, with five different faces used for each emotional condition. On each trial, SIFI tasks were preceded by either a disgust or a neutral face. In the SIFI task, during a trial, the visual stimulus (white circle) was presented below the fixation cross while the pure tone auditory beep was presented via played through participants' speakers. Trials were either congruent or illusory trials. Congruent trials could either be a single flash-beep pair (1F1B) or two sequential flash-beep pairs (2F2B) that were separated by a variable SOA (right panel). During the illusory trials, a flash-beep pair was first presented followed by a second auditory beep at some SOA (left panel). Regardless of trial type, participants were asked to respond as to how many flashes they perceived during the trial and how confident they are about their response.
To empirically validate the effectiveness of our image stimuli, participants completed a valence and arousal rating task at the end of the main experiment. During this task, participants viewed the same images used in the primary experiment and rated each image based on its emotional valence (how pleasant or unpleasant they felt) and arousal (how emotionally stimulating they found it). For these ratings, we utilized The Affective Slider [16], which is a digital tool designed to measure both valence and arousal on continuous scales, allowing for a more nuanced and precise assessment of the participants' emotional responses to the stimuli. In our implementation, participants rated each image on continuous scales from 1 to 9, where 1 represented the least pleasant or arousing, and 9 represented the most. This validation step was beneficial to ensure that the images elicited the intended emotional responses for the study.
The Highly Sensitive Person (HSP) Scale is utilized to assess participants' sensory processing sensitivity (SPS). The scale consists of 27 self-report items rated on a Likert scale ranging from 1 (not at all) to 7 (extremely). These items reflect various aspects of heightened sensitivity, including sensitivity to external stimuli (e.g., loud noises, bright lights), emotional reactivity, and depth of processing. Participants' scores on the HSP Scale were calculated by summing the responses, with higher scores indicating greater sensitivity. In this study, the scale was administered.
We used the R statistical programming environment, version 4.2.0 (R Core Team, 2023) for all analyses. We conducted a regression analysis using R to examine the relationship between confidence/performance and HSP scores. The model was fitted using the lm() function for linear regression, and diagnostic tests were performed to assess key model assumptions, including normality, homoscedasticity, and multicollinearity.
Out of 59 initial HSP responses, 38 were deemed acceptable. Similarly, out of 37 Pavlovia responses, 27 were acceptable. Ultimately, a total of 23 participants provided valid data for both HSP and Pavlovia responses, which were included in the final analysis as the preliminary findings. Among these participants, 9 are older than 53, while the remaining participants are aged between 18 and 52.
The results from the arousal-cued and nonarousal-cue SIFI task are presented in Figure 2. As shown in panel (a), the probability of correct responses increased with the SOA, with both disgust and neutral face conditions exhibiting similar trends. Notably, the highest accuracy was observed at positive SOAs, indicating that participants were more likely to accurately perceive the visual stimuli when presented with a delayed auditory cue. Panel (b) illustrates participants' confidence ratings, which also varied with SOA. Confidence levels were highest at the positive SOAs, while a dip was observed at the 50 ms SOA for both image types. Overall, these findings suggest that the type of image presented before the trial did not significantly influence either the accuracy of responses or the confidence ratings, highlighting the robustness of the SIFI effect across different emotional cues. Participants' ratings from the valence and arousal tasks indicated a clear differentiation between the emotional impact of disgust and neutral images. For valence ratings, disgust images were perceived as more unpleasant, with a mean of 2.73 (SD = 1.19), while neutral images were rated as significantly more pleasant, with a mean valence of 4.75 (SD = 1.00). In terms of arousal, disgust images elicited higher emotional stimulation, with a mean of 5.29 (SD = 1.92), compared to neutral images, which had a mean arousal rating of 3.66 (SD = 1.69).
In Figure 4 illustrates the relationship between sensory processing sensitivity (HSP score) and two dependent variables: (a) the probability of correct responses and (b) confidence ratings. In panel (a), there is no significant relationship between HSP scores and the probability of correct responses in the SIFI task, as indicated by the relatively flat regression line (Figure 4, left; β = 0.03, p= 0.14). This suggests that higher sensory processing sensitivity does not substantially influence participants' accuracy in this task. In panel (b), a slight negative correlation is observed between HSP scores and confidence ratings. As HSP scores increase, participants tend to report lower confidence in their responses (Figure 4, right; β = -0.17, p<0.001). However, the confidence intervals (shaded area) suggest that the relationship may not be strong. Together, these results indicate that while HSP may affect subjective confidence in sensory perception, it does not significantly impact objective task performance.
The current study aimed to investigate the influence of internal states, specifically arousal, on MSI using SIFI. While external stimuli, such as audiovisual cues, are well-established in MSI research, the role of internal states like arousal remains less understood. In this study, we hypothesized that internal arousal, induced by emotionally charged images, would affect participants' ability to integrate sensory information. However, our preliminary findings did not show significant differences in response accuracy or confidence between arousal-cued and neutral images, suggesting that the chosen method for inducing arousal may not have been effective in this context. Although we specifically used disgust-inducing images-known to provoke strong emotional responses and higher arousal levels compared to other negative stimuli like fear and sadness [17], [18]-this did not lead to the expected effects. One possible explanation for these results is the nature of the emotional cues used in the experiment. While emotionally charged images can elicit some degree of arousal [19], they may not evoke a strong enough physiological response to influence MSI meaningfully. The impact of arousal on MSI and perception performance can vary greatly depending on both the context and individual differences [20]. Building on previous research that has demonstrated the pronounced impact of stronger arousal inducers, such as physical stressors, on cognitive and perceptual processes [21], we suggest that future studies should explore more robust methods for arousal induction to further elucidate these effects.
Interestingly, our results highlighted the role of individual differences in sensory sensitivity, measured by the HSP scale. Participants with higher sensory sensitivity exhibited enhanced internal judgment (i.e. confidence), which suggests that individual differences in sensory processing styles may modulate the effects of internal states on MSI. This finding aligns with previous research showing that sensory sensitivity is associated with heightened responsiveness to both internal and external stimuli [22]. For highly sensitive individuals, brief shifts in internal arousal might be sufficient to alter their perception and integration of multisensory inputs, even if these shifts are not detectable on a group level.
In our study, methodological design choices aimed to balance experimental rigor with accessibility for an online participant pool. Platforms like Prolific and Pavlovia enabled diverse recruitment and seamless deployment but introduced variability in environmental conditions, such as audio quality and participant attentiveness. Additionally, the systematic variation of SOAs provided valuable insights into temporal windows of multisensory integration [12]. However, realworld sensory processing often involves more asynchronous and complex inputs, suggesting the need for follow-up studies in naturalistic environments. While the inclusion of the HSP scale offered a layer of individualized analysis, its reliance on self-reports highlights the importance of incorporating complementary physiological measures, such as galvanic skin response (GSR), to enrich our understanding. Acknowledging these constraints and their implications reinforces the transparency of our approach and suggests avenues for enhancing future research. Specifically, given the limitations of using emotionally charged images, we propose adopting physical methods, such as the cold-water pressor technique in future studies as a more reliable method of inducing arousal [23]. The cold-water pressor is known to induce more significant physiological changes, including heightened heart rate and GSR [24], which can be more easily measured and correlated with changes in MSI. By employing this method alongside physiological monitoring, we expect to better capture the dynamic relationship between internal arousal and multisensory integration. Additionally, the present study is limited by its relatively small sample size (n = 23), which reduces the statistical power and generalizability of our findings. While the use of an online platform for recruitment allowed us to access a diverse pool of participants, this sample may not fully capture the variability present in the general population, particularly across different age groups, cultural backgrounds, or sensory sensitivities. To address this limitation, in future research, we will focus on increasing the sample size to enhance the reliability and applicability of the results. By addressing these limitations, we aim to enhance the technical rigor of the research and expand its implications for understanding MSI in both typical and clinical populations.
The implications of this research extend beyond theoretical understanding and encompass practical applications in both clinical and non-clinical settings. For instance, individuals with Parkinson's disease often struggle with MSI [25], [26], impairing their ability to navigate environments safely and increasing the risk of falls [27]. Understanding how internal states, such as arousal, interact with MSI in these populations could inform interventions aimed at improving sensory integration and enhancing motor coordination and safety. Additionally, the findings offer broader insights into general cognitive and perceptual processes, shedding light on how individuals process stimuli in everyday settings such as workplaces, classrooms, or high-stress environments like driving.
This research is also relevant to aging populations, providing valuable understanding of agerelated changes in MSI and informing the development of tools like fall prevention systems or cognitive training programs to enhance safety and quality of life. Furthermore, the link between arousal and sensory processing has potential applications in mental health, particularly for conditions like anxiety, where sensory overload is a significant factor. Beyond healthcare, these insights can drive advancements in human-computer interaction, including virtual and augmented reality systems, where effective multisensory feedback integration is crucial for creating immersive, tailored experiences. Collectively, these applications highlight the study's broad impact across diverse fields.
In conclusion, while our preliminary findings did not support the hypothesis that arousal-cued images influence MSI, the role of individual differences in sensory sensitivity suggests that internal states might modulate sensory integration in specific subgroups. Future research employing more robust arousal induction methods and physiological monitoring will be crucial to furthering our understanding of how internal and external inputs interact in MSI. These insights may contribute to the development of therapeutic strategies for individuals with multisensory deficits, ultimately enhancing their quality of life and reducing risks associated with impaired sensory processing.
Zahra Azizi is supported under the European Union's Horizon 2020 research and innovation programme through the Marie Skłodowska-Curie grant agreement No. 101034252.
The authors have no competing interests to declare that are relevant to the content of this article.