=Paper= {{Paper |id=Vol-3910/aics2024_p80 |storemode=property |title=Investigating the Impact of Internal Inputs on Mmultisensory Integration: A ‎Study of External ‎Multisensory Inputs and Internal Arousal |pdfUrl=https://ceur-ws.org/Vol-3910/aics2024_p80.pdf |volume=Vol-3910 |authors=Zahra Azizi,Jason Chan,Tomas Ward,Annalisa Setti }} ==Investigating the Impact of Internal Inputs on Mmultisensory Integration: A ‎Study of External ‎Multisensory Inputs and Internal Arousal== https://ceur-ws.org/Vol-3910/aics2024_p80.pdf

Zahra Azizia,*, Jason Chanb, Tomas Warda, and Annalisa Settib
a
Insight SFI Research Centre for Data Analytics, School of Computing, Dublin City University, Ireland
b
School of Applied Psychology, University College Cork, Cork, Ireland

Abstract
Multisensory integration, the brain's ability to combine information from different sensory modalities, is
influenced by both external stimuli and internal states, such as arousal, motivation, and emotion. This study
explores the balance between internal and external inputs and their effect on multisensory integration, with
a focus on arousal. We hypothesize that multisensory integration abilities vary depending on both internal
states and external inputs. In our experiment, we tested 23 participants using the Sound-Induced Flash
Illusion (external input) and arousal-cued images (internal arousal). Sensory sensitivity was also assessed
using the Highly Sensitive Person (HSP) scale. Preliminary findings revealed no significant differences in
response accuracy or confidence when participants were presented with cue-induced arousal versus neutral
images across different age groups. However, greater sensory sensitivity was linked to enhanced internal
judgment. These results suggest that while arousal-cued images may not be the most effective method for
inducing internal arousal in this context, individual differences play a role in how internal states affect
multisensory integration. To further explore this, we plan to implement the cold-water pressor technique,
along with physiological monitoring (GSR and heart rate), to more effectively induce and control internal
arousal. By exploring how internal states influence multisensory integration, our findings could inform
strategies to improve sensory processing and behavioral outcomes in individuals with deficits, such as those
with Parkinson's disease or age-related challenges. These interventions may enhance motor coordination,
cognitive function, and quality of life while also providing insights for broader applications in mental health
and technology.

Keywords
Multisensory integration, Arousal-cued images, Sensory sensitivity, Sound-Induced Flash Illusion†

1. Introduction
Multisensory integration (MSI) refers to the brain’s ability to synthesize information from different
sensory modalities, facilitating a coherent perception of the environment [1]. This process is critical
for navigating daily life [2], impacting attention [3], decision-making [4], and motor coordination
[5], [6]. Traditionally, MSI has been studied through the lens of external sensory inputs, such as
visual and auditory stimuli, but emerging evidence suggests that internal states—such as arousal,
emotion, and motivation—also play a crucial role in shaping how multisensory information is
processed [7], [8], [9].
Arousal, in particular, has been identified as a key internal factor influencing perception, with
heightened states of arousal potentially enhancing or disrupting the sensory inputs depending on
context [10]. However, the specific mechanisms underlying the interaction between internal arousal
and external sensory stimuli remain unclear. This study aims to explore how internal arousal,
induced through visual cues, influences MSI performance in the Sound-Induced Flash Illusion (SIFI),
a widely used paradigm for studying audiovisual integration [11], [12]. In addition to investigating
the role of arousal, we assessed individual differences in sensory sensitivity using the Highly
Sensitive Person (HSP) scale [13], which may mediate the effects of internal states on MSI.

AICS’24: 32nd Irish Conference on Artificial Intelligence and Cognitive Science, December 09—10, 2024, Dublin, Ireland
*
Corresponding Author
zahra.azizi@dcu.ie (Z. Azizi); jason.chan@ucc.ie (J. Chan); tomas.ward@dcu.ie (T. Ward); a.setti@ucc.ie (A. Setti)
0000-0002-5181-4958 (Z. Azizi); 0000-0002-4663-5779 (J. Chan); 0000-0002-6173-6607 (T. Ward); 0000-0002-9741-2559
(A. Setti)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
MSI is crucial for many aspects of daily life, including understanding speech in noisy
environments, coordinating motor actions while driving, and maintaining balance while walking.
This research not only enhances our understanding of MSI but also holds significant clinical
potential. For individuals with MSI deficits, such as those with Parkinson's disease, it could guide the
development of therapies to improve motor coordination, cognitive function, and overall quality of
life. Additionally, insights into age-related changes in MSI could support innovations like fall
prevention systems and cognitive training programs for older adults. Beyond healthcare, the findings
extend to mental health and technology, offering applications such as enhancing virtual reality
systems through improved multisensory feedback integration.
2. Methods
2.1. Participants

Our study involved 23 participants, recruited through Prolific, an online platform commonly used
for academic research that allows for access to a diverse pool of participants [14]. Informed consent
was obtained from all participants prior to the start of the experiment. The experiment was coded
using PsychoPy, an open-source software for running behavioral science experiments [15]. The data
collection was conducted online, utilizing Pavlovia (Open Science Tools, Nottingham, UK,
https://pavlovia.org/)—a platform that enables remote experiments designed in PsychoPy to be
deployed and run in browsers. Additionally, Qualtrics was used to administer the survey component
of the experiment (Qualtrics, Provo, UT, https://www.qualtrics.com), ensuring ease of access and
user-friendly interaction for participants. Ethical approval was granted by the Ethics Committee at
the School of Applied Psychology, University College Cork. All data were securely stored and
handled according to ethical standards and protocols.

2.2. Sound-Induced Flash Illusion

In this arousal-cued SIFI task, participants were exposed to either a disgust or neutral face before
each trial, aiming to manipulate their arousal levels. Each trial began with a briefly presented face
stimulus, followed by a fixation cross. Participants then viewed a visual flash (a white circle)
presented below the fixation cross, accompanied by an auditory beep played through participants’
speakers. The trials could be congruent, where the visual and auditory stimuli matched (1F1B or
2F2B), or illusory (2B1F), in which a beep was presented without a corresponding flash. In the 2B1F
condition, the beep could occur either before or after the visual flash (Figure 1). The stimulus onset
asynchronies (SOAs) in these trials were varied and could be −200 ms, −150 ms, −50 ms, 50 ms, 150
ms, or 200 ms. Unisensory trials (0B2F), where only flashes were presented without auditory beeps,
were presented at a single SOA of 50 ms. Participants were asked to report how many flashes they
perceived and to rate their confidence in their responses. The experiment consisted of 288 trials,
balanced between neutral and disgust face cues, SOA, and congruent/illusory conditions, with five
different faces used for each emotional condition.
Figure 1: Trial schematic illustrating the arousal-cued SIFI task, in which an unexpected, disgusted
face increased arousal just prior to a sound-induced flash illusion judgment and confidence rating.
On each trial, SIFI tasks were preceded by either a disgust or a neutral face. In the SIFI task, during
a trial, the visual stimulus (white circle) was presented below the fixation cross while the pure tone
auditory beep was presented via played through participants’ speakers. Trials were either congruent
or illusory trials. Congruent trials could either be a single flash-beep pair (1F1B) or two sequential
flash-beep pairs (2F2B) that were separated by a variable SOA (right panel). During the illusory trials,
a flash-beep pair was first presented followed by a second auditory beep at some SOA (left panel).
Regardless of trial type, participants were asked to respond as to how many flashes they perceived
during the trial and how confident they are about their response.

To empirically validate the effectiveness of our image stimuli, participants completed a valence and
arousal rating task at the end of the main experiment. During this task, participants viewed the same
images used in the primary experiment and rated each image based on its emotional valence (how
pleasant or unpleasant they felt) and arousal (how emotionally stimulating they found it). For these
ratings, we utilized The Affective Slider [16], which is a digital tool designed to measure both valence
and arousal on continuous scales, allowing for a more nuanced and precise assessment of the
participants' emotional responses to the stimuli. In our implementation, participants rated each
image on continuous scales from 1 to 9, where 1 represented the least pleasant or arousing, and 9
represented the most. This validation step was beneficial to ensure that the images elicited the
intended emotional responses for the study.

2.3. HSP

The Highly Sensitive Person (HSP) Scale is utilized to assess participants' sensory processing
sensitivity (SPS). The scale consists of 27 self-report items rated on a Likert scale ranging from 1 (not
at all) to 7 (extremely). These items reflect various aspects of heightened sensitivity, including
sensitivity to external stimuli (e.g., loud noises, bright lights), emotional reactivity, and depth of
processing. Participants’ scores on the HSP Scale were calculated by summing the responses, with
higher scores indicating greater sensitivity. In this study, the scale was administered.
2.4. Statistical analysis

We used the R statistical programming environment, version 4.2.0 (R Core Team, 2023) for all
analyses. We conducted a regression analysis using R to examine the relationship between
confidence/performance and HSP scores. The model was fitted using the lm() function for linear
regression, and diagnostic tests were performed to assess key model assumptions, including
normality, homoscedasticity, and multicollinearity.

3. Results
Out of 59 initial HSP responses, 38 were deemed acceptable. Similarly, out of 37 Pavlovia responses,
27 were acceptable. Ultimately, a total of 23 participants provided valid data for both HSP and
Pavlovia responses, which were included in the final analysis as the preliminary findings. Among
these participants, 9 are older than 53, while the remaining participants are aged between 18 and 52.
The results from the arousal-cued and nonarousal-cue SIFI task are presented in Figure 2. As
shown in panel (a), the probability of correct responses increased with the SOA, with both disgust
and neutral face conditions exhibiting similar trends. Notably, the highest accuracy was observed at
positive SOAs, indicating that participants were more likely to accurately perceive the visual stimuli
when presented with a delayed auditory cue. Panel (b) illustrates participants' confidence ratings,
which also varied with SOA. Confidence levels were highest at the positive SOAs, while a dip was
observed at the 50 ms SOA for both image types. Overall, these findings suggest that the type of
image presented before the trial did not significantly influence either the accuracy of responses or
the confidence ratings, highlighting the robustness of the SIFI effect across different emotional cues.

Figure 2: Performance and confidence as a function of SOA for two image types (disgust and neutral
faces). (a) The probability of correct responses (%) across different SOA values for disgust faces (blue)
and neutral faces (orange). Accuracy increases with positive SOA for both image types, with minimal
differences between disgust and neutral faces. (b) Confidence ratings (arbitrary units, a.u.) for the
same SOA values. Confidence increases with positive SOA, showing similar trends for both image
types, with a notable rise at SOA = 200 ms. Error bars represent standard error of the mean (SEM).

The results in Figure 3 demonstrate the effect of SOA on both performance and confidence, compared
across age groups. Panel (a) shows the probability of correct responses (%), which increases with
positive SOA values in all age groups, with younger participants generally achieving higher accuracy
across SOA values. Performance peaks around SOA = 200 ms for all groups. In panel (b), confidence
ratings also increase with positive SOA, particularly in the younger groups, who consistently report
higher confidence than the older groups, especially at SOA = 200 ms. Error bars indicate standard
error of the mean.
Figure 3: Performance and confidence as a function of SOA for different age groups. (a) The
probability of correct responses across different SOA values for participants in different age groups.
All groups show improved accuracy with increasing SOA, with younger participants generally
outperforming older ones. (b) Confidence ratings for the same SOA values. Confidence increases
with positive SOA, with younger participants consistently reporting higher confidence, particularly
at SOA = 200 ms. Error bars represent standard error of the mean (SEM).

Participants' ratings from the valence and arousal tasks indicated a clear differentiation between the
emotional impact of disgust and neutral images. For valence ratings, disgust images were perceived
as more unpleasant, with a mean of 2.73 (SD = 1.19), while neutral images were rated as significantly
more pleasant, with a mean valence of 4.75 (SD = 1.00). In terms of arousal, disgust images elicited
higher emotional stimulation, with a mean of 5.29 (SD = 1.92), compared to neutral images, which
had a mean arousal rating of 3.66 (SD = 1.69).
In Figure 4 illustrates the relationship between sensory processing sensitivity (HSP score) and
two dependent variables: (a) the probability of correct responses and (b) confidence ratings. In panel
(a), there is no significant relationship between HSP scores and the probability of correct responses
in the SIFI task, as indicated by the relatively flat regression line (Figure 4, left; β = 0.03, p= 0.14).
This suggests that higher sensory processing sensitivity does not substantially influence participants'
accuracy in this task. In panel (b), a slight negative correlation is observed between HSP scores and
confidence ratings. As HSP scores increase, participants tend to report lower confidence in their
responses (Figure 4, right; β = -0.17, p<0.001). However, the confidence intervals (shaded area)
suggest that the relationship may not be strong. Together, these results indicate that while HSP may
affect subjective confidence in sensory perception, it does not significantly impact objective task
performance.

Figure 4: The relationship between sensory processing sensitivity (HSP scores) and two key metrics
in our SIFI task: (a) the probability of correct responses and (b) confidence ratings.
4. Discussion
The current study aimed to investigate the influence of internal states, specifically arousal, on MSI
using SIFI. While external stimuli, such as audiovisual cues, are well-established in MSI research, the
role of internal states like arousal remains less understood. In this study, we hypothesized that
internal arousal, induced by emotionally charged images, would affect participants' ability to
integrate sensory information. However, our preliminary findings did not show significant
differences in response accuracy or confidence between arousal-cued and neutral images, suggesting
that the chosen method for inducing arousal may not have been effective in this context. Although
we specifically used disgust-inducing images—known to provoke strong emotional responses and
higher arousal levels compared to other negative stimuli like fear and sadness [17],[18]—this did not
lead to the expected effects. One possible explanation for these results is the nature of the emotional
cues used in the experiment. While emotionally charged images can elicit some degree of arousal
[19], they may not evoke a strong enough physiological response to influence MSI meaningfully.
The impact of arousal on MSI and perception performance can vary greatly depending on both the
context and individual differences [20]. Building on previous research that has demonstrated the
pronounced impact of stronger arousal inducers, such as physical stressors, on cognitive and
perceptual processes [21], we suggest that future studies should explore more robust methods for
arousal induction to further elucidate these effects.
Interestingly, our results highlighted the role of individual differences in sensory sensitivity,
measured by the HSP scale. Participants with higher sensory sensitivity exhibited enhanced internal
judgment (i.e. confidence), which suggests that individual differences in sensory processing styles
may modulate the effects of internal states on MSI. This finding aligns with previous research
showing that sensory sensitivity is associated with heightened responsiveness to both internal and
external stimuli [22]. For highly sensitive individuals, brief shifts in internal arousal might be
sufficient to alter their perception and integration of multisensory inputs, even if these shifts are not
detectable on a group level.
In our study, methodological design choices aimed to balance experimental rigor with
accessibility for an online participant pool. Platforms like Prolific and Pavlovia enabled diverse
recruitment and seamless deployment but introduced variability in environmental conditions, such
as audio quality and participant attentiveness. Additionally, the systematic variation of SOAs
provided valuable insights into temporal windows of multisensory integration [12]. However, real-
world sensory processing often involves more asynchronous and complex inputs, suggesting the
need for follow-up studies in naturalistic environments. While the inclusion of the HSP scale offered
a layer of individualized analysis, its reliance on self-reports highlights the importance of
incorporating complementary physiological measures, such as galvanic skin response (GSR), to
enrich our understanding. Acknowledging these constraints and their implications reinforces the
transparency of our approach and suggests avenues for enhancing future research. Specifically, given
the limitations of using emotionally charged images, we propose adopting physical methods, such
as the cold-water pressor technique in future studies as a more reliable method of inducing arousal
[23]. The cold-water pressor is known to induce more significant physiological changes, including
heightened heart rate and GSR [24], which can be more easily measured and correlated with changes
in MSI. By employing this method alongside physiological monitoring, we expect to better capture
the dynamic relationship between internal arousal and multisensory integration. Additionally, the
present study is limited by its relatively small sample size (n = 23), which reduces the statistical
power and generalizability of our findings. While the use of an online platform for recruitment
allowed us to access a diverse pool of participants, this sample may not fully capture the variability
present in the general population, particularly across different age groups, cultural backgrounds, or
sensory sensitivities. To address this limitation, in future research, we will focus on increasing the
sample size to enhance the reliability and applicability of the results. By addressing these limitations,
we aim to enhance the technical rigor of the research and expand its implications for understanding
MSI in both typical and clinical populations.
The implications of this research extend beyond theoretical understanding and encompass
practical applications in both clinical and non-clinical settings. For instance, individuals with
Parkinson's disease often struggle with MSI [25], [26], impairing their ability to navigate
environments safely and increasing the risk of falls [27]. Understanding how internal states, such as
arousal, interact with MSI in these populations could inform interventions aimed at improving
sensory integration and enhancing motor coordination and safety. Additionally, the findings offer
broader insights into general cognitive and perceptual processes, shedding light on how individuals
process stimuli in everyday settings such as workplaces, classrooms, or high-stress environments
like driving.
This research is also relevant to aging populations, providing valuable understanding of age-
related changes in MSI and informing the development of tools like fall prevention systems or
cognitive training programs to enhance safety and quality of life. Furthermore, the link between
arousal and sensory processing has potential applications in mental health, particularly for
conditions like anxiety, where sensory overload is a significant factor. Beyond healthcare, these
insights can drive advancements in human-computer interaction, including virtual and augmented
reality systems, where effective multisensory feedback integration is crucial for creating immersive,
tailored experiences. Collectively, these applications highlight the study's broad impact across
diverse fields.
5. Conclusion
In conclusion, while our preliminary findings did not support the hypothesis that arousal-cued
images influence MSI, the role of individual differences in sensory sensitivity suggests that internal
states might modulate sensory integration in specific subgroups. Future research employing more
robust arousal induction methods and physiological monitoring will be crucial to furthering our
understanding of how internal and external inputs interact in MSI. These insights may contribute to
the development of therapeutic strategies for individuals with multisensory deficits, ultimately
enhancing their quality of life and reducing risks associated with impaired sensory processing.

6. Acknowledgements
Zahra Azizi is supported under the European Union’s Horizon 2020 research and innovation
programme through the Marie Skłodowska-Curie grant agreement No. 101034252.

Disclosure of Interests. The authors have no competing interests to declare that are relevant to
the content of this article.

References
[1] M. T. Wallace, T. G. Woynaroski, and R. A. Stevenson, ‘Multisensory integration as a window
into orderly and disrupted cognition and communication’, Annu Rev Psychol, vol. 71, no. 1, pp.
193–219, 2020.
[2] J. L. Mozolic, C. E. Hugenschmidt, A. M. Peiffer, and P. J. Laurienti, ‘Modality-specific selective
attention attenuates multisensory integration’, Exp Brain Res, vol. 184, pp. 39–52, 2008.
[3] C. Spence and C. Frings, ‘Multisensory feature integration in (and out) of the focus of spatial
attention’, Atten Percept Psychophys, vol. 82, no. 1, pp. 363–376, 2020.
[4] C. R. Fetsch and U. Noppeney, ‘How the brain controls decision making in a multisensory
world’, 2023, The Royal Society.
[5] M. Tagliabue and J. McIntyre, ‘A modular theory of multisensory integration for motor control’,
Front Comput Neurosci, vol. 8, p. 1, 2014.
[6] C. Roy, S. Dalla Bella, S. Pla, and J. Lagarde, ‘Multisensory integration and behavioral stability’,
Psychol Res, vol. 85, no. 2, pp. 879–886, 2021.
[7] Q. Ren, A. C. Marshall, J. Kaiser, and S. Schütz-Bosbach, ‘Multisensory integration of anticipated
cardiac signals with visual targets affects their detection among multiple visual stimuli’,
Neuroimage, vol. 262, p. 119549, 2022.
[8] M. Saltafossi, A. Zaccaro, M. G. Perrucci, F. Ferri, and M. Costantini, ‘The impact of cardiac
phases on multisensory integration’, Biol Psychol, vol. 182, p. 108642, 2023.
[9] M. A. Khan, S. Abbas, A. Raza, F. Khan, and T. Whangbo, ‘Emotion Based Signal Enhancement
Through Multisensory Integration Using Machine Learning.’, Computers, Materials & Continua,
vol. 71, no. 3, 2022.
[10] M. Mather and M. R. Sutherland, ‘Arousal-biased competition in perception and memory’,
Perspectives on psychological science, vol. 6, no. 2, pp. 114–133, 2011.
[11] L. Shams, Y. Kamitani, and S. Shimojo, ‘Visual illusion induced by sound’, Cognitive brain
research, vol. 14, no. 1, pp. 147–152, 2002.
[12] R. J. Hirst, D. P. McGovern, A. Setti, L. Shams, and F. N. Newell, ‘What you see is what you hear:
Twenty years of research using the Sound-Induced Flash Illusion’, Neurosci Biobehav Rev, vol.
118, pp. 759–774, 2020.
[13] E. N. Aron and A. Aron, ‘Sensory-processing sensitivity and its relation to introversion and
emotionality.’, J Pers Soc Psychol, vol. 73, no. 2, p. 345, 1997.
[14] S. Palan and C. Schitter, ‘Prolific. ac—A subject pool for online experiments’, J Behav Exp
Finance, vol. 17, pp. 22–27, 2018.
[15] J. W. Peirce, ‘PsychoPy—psychophysics software in Python’, J Neurosci Methods, vol. 162, no.
1–2, pp. 8–13, 2007.
[16] A. Betella and P. F. M. J. Verschure, ‘The affective slider: A digital self-assessment scale for the
measurement of human emotions’, PLoS One, vol. 11, no. 2, p. e0148037, 2016.
[17] CJ. Croucher, AJ. Calder, C. Ramponi, PJ. Barnard, FC. Murphy, ‘Disgust enhances the
recollection of negative emotional images’, PloS one Nov 16;6(11):e26571, 2011.
[18] H. Revers, K. Van Deun, W. Strijbosch, J. Vroomen, M. Bastiaansen. ‘Decoding the neural
responses to experiencing disgust and sadness’, Brain Research Oct 15;1793:148034, 2022.
[19] A. N. Zsidó, ‘The effect of emotional arousal on visual attentional performance: a systematic
review’, Psychol Res, vol. 88, no. 1, pp. 1–24, 2024.
[20] K. P. Körding, U. Beierholm, W. J. Ma, S. Quartz, J. B. Tenenbaum, and L. Shams, ‘Causal
inference in multisensory perception’, PLoS One, vol. 2, no. 9, p. e943, 2007.
[21] E. S. Abdelall et al., ‘The interaction between physical and psychosocial stressors’, Front Behav
Neurosci, vol. 14, p. 63, 2020.
[22] J. Ward, ‘Individual differences in sensory sensitivity: A synthesizing framework and evidence
from normal variation and developmental conditions’, Cogn Neurosci, vol. 10, no. 3, pp. 139–
157, 2019.
[23] DU. Silverthorn, J. Michael, ‘Cold Stress and the cold pressor test’, Advances in physiology
education, 2013.
[24] S. Ghiasi, A. Greco, R. Barbieri, EP. Scilingo, G. Valenza, ‘Assessing autonomic function from
electrodermal activity and heart rate variability during cold-pressor test and emotional
challenge’, Scientific reports, Mar 25;10(1):5406, 2020.
[25] Z. Rostami, M. Salari, S. Mahdavi, M. Etemadifar. ‘Abnormal multisensory temporal
discrimination in Parkinson’s disease’, Brain Research. Jul 1;1834:148901, 2024.
[26] C. Ding, CJ. Palmer, J. Hohwy, GJ. Youssef, B. Paton, N. Tsuchiya, JC. Stout, D. Thyagarajan,
‘Parkinson's disease alters multisensory perception: Insights from the Rubber Hand Illusion’,
Neuropsychologia, Mar 1;97:38-45, 2017.
[27] LK. Allison, T. Kiemel, JJ. Jeka, ‘Multisensory reweighting of vision and touch is intact in healthy
and fall-prone older adults’, Experimental brain research, Nov;175:342-52, 2006.