Multisensory integration, the brain's ability to combine information from different sensory modalities, is influenced by both external stimuli and internal states, such as arousal, motivation, and emotion. This study explores the balance between internal and external inputs and their effect on multisensory integration, with a focus on arousal. We hypothesize that multisensory integration abilities vary depending on both internal states and external inputs. In our experiment, we tested 23 participants using the Sound-Induced Flash Illusion (external input) and arousal-cued images (internal arousal). Sensory sensitivity was also assessed using the Highly Sensitive Person (HSP) scale. Preliminary findings revealed no significant differences in response accuracy or confidence when participants were presented with cue-induced arousal versus neutral images across different age groups. However, greater sensory sensitivity was linked to enhanced internal judgment. These results suggest that while arousal-cued images may not be the most effective method for inducing internal arousal in this context, individual differences play a role in how internal states affect multisensory integration. To further explore this, we plan to implement the cold-water pressor technique, along with physiological monitoring (GSR and heart rate), to more effectively induce and control internal arousal. By exploring how internal states influence multisensory integration, our findings could inform strategies to improve sensory processing and behavioral outcomes in individuals with deficits, such as those with Parkinson's disease or age-related challenges. These interventions may enhance motor coordination, cognitive function, and quality of life while also providing insights for broader applications in mental health and technology.
Multisensory integration (MSI) refers to the brain’s ability to synthesize information from different
sensory modalities, facilitating a coherent perception of the environment [
Arousal, in particular, has been identified as a key internal factor influencing perception, with
heightened states of arousal potentially enhancing or disrupting the sensory inputs depending on
context [
MSI is crucial for many aspects of daily life, including understanding speech in noisy environments, coordinating motor actions while driving, and maintaining balance while walking. This research not only enhances our understanding of MSI but also holds significant clinical potential. For individuals with MSI deficits, such as those with Parkinson's disease, it could guide the development of therapies to improve motor coordination, cognitive function, and overall quality of life. Additionally, insights into age-related changes in MSI could support innovations like fall prevention systems and cognitive training programs for older adults. Beyond healthcare, the findings extend to mental health and technology, offering applications such as enhancing virtual reality systems through improved multisensory feedback integration.
Our study involved 23 participants, recruited through Prolific, an online platform commonly used for academic research that allows for access to a diverse pool of participants [14]. Informed consent was obtained from all participants prior to the start of the experiment. The experiment was coded using PsychoPy, an open-source software for running behavioral science experiments [15]. The data collection was conducted online, utilizing Pavlovia (Open Science Tools, Nottingham, UK, https://pavlovia.org/)—a platform that enables remote experiments designed in PsychoPy to be deployed and run in browsers. Additionally, Qualtrics was used to administer the survey component of the experiment (Qualtrics, Provo, UT, https://www.qualtrics.com), ensuring ease of access and user-friendly interaction for participants. Ethical approval was granted by the Ethics Committee at the School of Applied Psychology, University College Cork. All data were securely stored and handled according to ethical standards and protocols.
In this arousal-cued SIFI task, participants were exposed to either a disgust or neutral face before each trial, aiming to manipulate their arousal levels. Each trial began with a briefly presented face stimulus, followed by a fixation cross. Participants then viewed a visual flash (a white circle) presented below the fixation cross, accompanied by an auditory beep played through participants’ speakers. The trials could be congruent, where the visual and auditory stimuli matched (1F1B or 2F2B), or illusory (2B1F), in which a beep was presented without a corresponding flash. In the 2B1F condition, the beep could occur either before or after the visual flash (Figure 1). The stimulus onset asynchronies (SOAs) in these trials were varied and could be −200 ms, −150 ms, −50 ms, 50 ms, 150 ms, or 200 ms. Unisensory trials (0B2F), where only flashes were presented without auditory beeps, were presented at a single SOA of 50 ms. Participants were asked to report how many flashes they perceived and to rate their confidence in their responses. The experiment consisted of 288 trials, balanced between neutral and disgust face cues, SOA, and congruent/illusory conditions, with five different faces used for each emotional condition.
To empirically validate the effectiveness of our image stimuli, participants completed a valence and arousal rating task at the end of the main experiment. During this task, participants viewed the same images used in the primary experiment and rated each image based on its emotional valence (how pleasant or unpleasant they felt) and arousal (how emotionally stimulating they found it). For these ratings, we utilized The Affective Slider [16], which is a digital tool designed to measure both valence and arousal on continuous scales, allowing for a more nuanced and precise assessment of the participants' emotional responses to the stimuli. In our implementation, participants rated each image on continuous scales from 1 to 9, where 1 represented the least pleasant or arousing, and 9 represented the most. This validation step was beneficial to ensure that the images elicited the intended emotional responses for the study. 2.3. HSP The Highly Sensitive Person (HSP) Scale is utilized to assess participants' sensory processing sensitivity (SPS). The scale consists of 27 self-report items rated on a Likert scale ranging from 1 (not at all) to 7 (extremely). These items reflect various aspects of heightened sensitivity, including sensitivity to external stimuli (e.g., loud noises, bright lights), emotional reactivity, and depth of processing. Participants’ scores on the HSP Scale were calculated by summing the responses, with higher scores indicating greater sensitivity. In this study, the scale was administered.
We used the R statistical programming environment, version 4.2.0 (R Core Team, 2023) for all analyses. We conducted a regression analysis using R to examine the relationship between confidence/performance and HSP scores. The model was fitted using the lm() function for linear regression, and diagnostic tests were performed to assess key model assumptions, including normality, homoscedasticity, and multicollinearity.
Out of 59 initial HSP responses, 38 were deemed acceptable. Similarly, out of 37 Pavlovia responses, 27 were acceptable. Ultimately, a total of 23 participants provided valid data for both HSP and Pavlovia responses, which were included in the final analysis as the preliminary findings. Among these participants, 9 are older than 53, while the remaining participants are aged between 18 and 52.
The results from the arousal-cued and nonarousal-cue SIFI task are presented in Figure 2. As shown in panel (a), the probability of correct responses increased with the SOA, with both disgust and neutral face conditions exhibiting similar trends. Notably, the highest accuracy was observed at positive SOAs, indicating that participants were more likely to accurately perceive the visual stimuli when presented with a delayed auditory cue. Panel (b) illustrates participants' confidence ratings, which also varied with SOA. Confidence levels were highest at the positive SOAs, while a dip was observed at the 50 ms SOA for both image types. Overall, these findings suggest that the type of image presented before the trial did not significantly influence either the accuracy of responses or the confidence ratings, highlighting the robustness of the SIFI effect across different emotional cues. The results in Figure 3 demonstrate the effect of SOA on both performance and confidence, compared across age groups. Panel (a) shows the probability of correct responses (%), which increases with positive SOA values in all age groups, with younger participants generally achieving higher accuracy across SOA values. Performance peaks around SOA = 200 ms for all groups. In panel (b), confidence ratings also increase with positive SOA, particularly in the younger groups, who consistently report higher confidence than the older groups, especially at SOA = 200 ms. Error bars indicate standard error of the mean.
Participants' ratings from the valence and arousal tasks indicated a clear differentiation between the emotional impact of disgust and neutral images. For valence ratings, disgust images were perceived as more unpleasant, with a mean of 2.73 (SD = 1.19), while neutral images were rated as significantly more pleasant, with a mean valence of 4.75 (SD = 1.00). In terms of arousal, disgust images elicited higher emotional stimulation, with a mean of 5.29 (SD = 1.92), compared to neutral images, which had a mean arousal rating of 3.66 (SD = 1.69).
In Figure 4 illustrates the relationship between sensory processing sensitivity (HSP score) and two dependent variables: (a) the probability of correct responses and (b) confidence ratings. In panel (a), there is no significant relationship between HSP scores and the probability of correct responses in the SIFI task, as indicated by the relatively flat regression line (Figure 4, left; β = 0.03, p= 0.14). This suggests that higher sensory processing sensitivity does not substantially influence participants' accuracy in this task. In panel (b), a slight negative correlation is observed between HSP scores and confidence ratings. As HSP scores increase, participants tend to report lower confidence in their responses (Figure 4, right; β = -0.17, p<0.001). However, the confidence intervals (shaded area) suggest that the relationship may not be strong. Together, these results indicate that while HSP may affect subjective confidence in sensory perception, it does not significantly impact objective task performance.
The current study aimed to investigate the influence of internal states, specifically arousal, on MSI using SIFI. While external stimuli, such as audiovisual cues, are well-established in MSI research, the role of internal states like arousal remains less understood. In this study, we hypothesized that internal arousal, induced by emotionally charged images, would affect participants' ability to integrate sensory information. However, our preliminary findings did not show significant differences in response accuracy or confidence between arousal-cued and neutral images, suggesting that the chosen method for inducing arousal may not have been effective in this context. Although we specifically used disgust-inducing images—known to provoke strong emotional responses and higher arousal levels compared to other negative stimuli like fear and sadness [17],[18]—this did not lead to the expected effects. One possible explanation for these results is the nature of the emotional cues used in the experiment. While emotionally charged images can elicit some degree of arousal [19], they may not evoke a strong enough physiological response to influence MSI meaningfully. The impact of arousal on MSI and perception performance can vary greatly depending on both the context and individual differences [20]. Building on previous research that has demonstrated the pronounced impact of stronger arousal inducers, such as physical stressors, on cognitive and perceptual processes [21], we suggest that future studies should explore more robust methods for arousal induction to further elucidate these effects.
Interestingly, our results highlighted the role of individual differences in sensory sensitivity, measured by the HSP scale. Participants with higher sensory sensitivity exhibited enhanced internal judgment (i.e. confidence), which suggests that individual differences in sensory processing styles may modulate the effects of internal states on MSI. This finding aligns with previous research showing that sensory sensitivity is associated with heightened responsiveness to both internal and external stimuli [22]. For highly sensitive individuals, brief shifts in internal arousal might be sufficient to alter their perception and integration of multisensory inputs, even if these shifts are not detectable on a group level.
In our study, methodological design choices aimed to balance experimental rigor with accessibility for an online participant pool. Platforms like Prolific and Pavlovia enabled diverse recruitment and seamless deployment but introduced variability in environmental conditions, such as audio quality and participant attentiveness. Additionally, the systematic variation of SOAs provided valuable insights into temporal windows of multisensory integration [12]. However, realworld sensory processing often involves more asynchronous and complex inputs, suggesting the need for follow-up studies in naturalistic environments. While the inclusion of the HSP scale offered a layer of individualized analysis, its reliance on self-reports highlights the importance of incorporating complementary physiological measures, such as galvanic skin response (GSR), to enrich our understanding. Acknowledging these constraints and their implications reinforces the transparency of our approach and suggests avenues for enhancing future research. Specifically, given the limitations of using emotionally charged images, we propose adopting physical methods, such as the cold-water pressor technique in future studies as a more reliable method of inducing arousal [23]. The cold-water pressor is known to induce more significant physiological changes, including heightened heart rate and GSR [24], which can be more easily measured and correlated with changes in MSI. By employing this method alongside physiological monitoring, we expect to better capture the dynamic relationship between internal arousal and multisensory integration. Additionally, the present study is limited by its relatively small sample size (n = 23), which reduces the statistical power and generalizability of our findings. While the use of an online platform for recruitment allowed us to access a diverse pool of participants, this sample may not fully capture the variability present in the general population, particularly across different age groups, cultural backgrounds, or sensory sensitivities. To address this limitation, in future research, we will focus on increasing the sample size to enhance the reliability and applicability of the results. By addressing these limitations, we aim to enhance the technical rigor of the research and expand its implications for understanding MSI in both typical and clinical populations.
The implications of this research extend beyond theoretical understanding and encompass practical applications in both clinical and non-clinical settings. For instance, individuals with Parkinson's disease often struggle with MSI [25], [26], impairing their ability to navigate environments safely and increasing the risk of falls [27]. Understanding how internal states, such as arousal, interact with MSI in these populations could inform interventions aimed at improving sensory integration and enhancing motor coordination and safety. Additionally, the findings offer broader insights into general cognitive and perceptual processes, shedding light on how individuals process stimuli in everyday settings such as workplaces, classrooms, or high-stress environments like driving.
This research is also relevant to aging populations, providing valuable understanding of agerelated changes in MSI and informing the development of tools like fall prevention systems or cognitive training programs to enhance safety and quality of life. Furthermore, the link between arousal and sensory processing has potential applications in mental health, particularly for conditions like anxiety, where sensory overload is a significant factor. Beyond healthcare, these insights can drive advancements in human-computer interaction, including virtual and augmented reality systems, where effective multisensory feedback integration is crucial for creating immersive, tailored experiences. Collectively, these applications highlight the study's broad impact across diverse fields.
In conclusion, while our preliminary findings did not support the hypothesis that arousal-cued images influence MSI, the role of individual differences in sensory sensitivity suggests that internal states might modulate sensory integration in specific subgroups. Future research employing more robust arousal induction methods and physiological monitoring will be crucial to furthering our understanding of how internal and external inputs interact in MSI. These insights may contribute to the development of therapeutic strategies for individuals with multisensory deficits, ultimately enhancing their quality of life and reducing risks associated with impaired sensory processing.
Zahra Azizi is supported under the European Union’s Horizon 2020 research and innovation programme through the Marie Skłodowska-Curie grant agreement No. 101034252. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. [11] L. Shams, Y. Kamitani, and S. Shimojo, ‘Visual illusion induced by sound’, Cognitive brain research, vol. 14, no. 1, pp. 147–152, 2002. [12] R. J. Hirst, D. P. McGovern, A. Setti, L. Shams, and F. N. Newell, ‘What you see is what you hear: Twenty years of research using the Sound-Induced Flash Illusion’, Neurosci Biobehav Rev, vol. 118, pp. 759–774, 2020. [13] E. N. Aron and A. Aron, ‘Sensory-processing sensitivity and its relation to introversion and emotionality.’, J Pers Soc Psychol, vol. 73, no. 2, p. 345, 1997. [14] S. Palan and C. Schitter, ‘Prolific. ac—A subject pool for online experiments’, J Behav Exp
Finance, vol. 17, pp. 22–27, 2018. [15] J. W. Peirce, ‘PsychoPy—psychophysics software in Python’, J Neurosci Methods, vol. 162, no.
1–2, pp. 8–13, 2007. [16] A. Betella and P. F. M. J. Verschure, ‘The affective slider: A digital self-assessment scale for the measurement of human emotions’, PLoS One, vol. 11, no. 2, p. e0148037, 2016. [17] CJ. Croucher, AJ. Calder, C. Ramponi, PJ. Barnard, FC. Murphy, ‘Disgust enhances the recollection of negative emotional images’, PloS one Nov 16;6(11):e26571, 2011. [18] H. Revers, K. Van Deun, W. Strijbosch, J. Vroomen, M. Bastiaansen. ‘Decoding the neural responses to experiencing disgust and sadness’, Brain Research Oct 15;1793:148034, 2022. [19] A. N. Zsidó, ‘The effect of emotional arousal on visual attentional performance: a systematic review’, Psychol Res, vol. 88, no. 1, pp. 1–24, 2024. [20] K. P. Körding, U. Beierholm, W. J. Ma, S. Quartz, J. B. Tenenbaum, and L. Shams, ‘Causal inference in multisensory perception’, PLoS One, vol. 2, no. 9, p. e943, 2007. [21] E. S. Abdelall et al., ‘The interaction between physical and psychosocial stressors’, Front Behav
Neurosci, vol. 14, p. 63, 2020. [22] J. Ward, ‘Individual differences in sensory sensitivity: A synthesizing framework and evidence from normal variation and developmental conditions’, Cogn Neurosci, vol. 10, no. 3, pp. 139– 157, 2019. [23] DU. Silverthorn, J. Michael, ‘Cold Stress and the cold pressor test’, Advances in physiology education, 2013. [24] S. Ghiasi, A. Greco, R. Barbieri, EP. Scilingo, G. Valenza, ‘Assessing autonomic function from electrodermal activity and heart rate variability during cold-pressor test and emotional challenge’, Scientific reports, Mar 25;10(1):5406, 2020. [25] Z. Rostami, M. Salari, S. Mahdavi, M. Etemadifar. ‘Abnormal multisensory temporal discrimination in Parkinson’s disease’, Brain Research. Jul 1;1834:148901, 2024. [26] C. Ding, CJ. Palmer, J. Hohwy, GJ. Youssef, B. Paton, N. Tsuchiya, JC. Stout, D. Thyagarajan, ‘Parkinson's disease alters multisensory perception: Insights from the Rubber Hand Illusion’, Neuropsychologia, Mar 1;97:38-45, 2017. [27] LK. Allison, T. Kiemel, JJ. Jeka, ‘Multisensory reweighting of vision and touch is intact in healthy and fall-prone older adults’, Experimental brain research, Nov;175:342-52, 2006.