We present recent and emerging advances in computational cognitive vision addressing artificial visual and spatial intelligence at the interface of (spatial) language, (spatial) logic and (spatial) cognition research. With a primary focus on explainable sensemaking of dynamic visuospatial imagery, we highlight the (systematic and modular) integration of methods from knowledge representation and reasoning, computer vision, spatial informatics, and computational cognitive modelling. A key emphasis here is on generalised (declarative) neurosymbolic reasoning & learning about space, motion, actions, and events relevant to embodied multimodal interaction under ecologically valid naturalistic settings in everyday life. Practically, this translates to general-purpose mechanisms for computational visual commonsense encompassing capabilities such as (neurosymbolic) semantic question-answering, relational spatio-temporal learning, visual abduction etc. The presented work is motivated by and demonstrated in the applied backdrop of areas as diverse as autonomous driving, cognitive robotics, design of digital visuoauditory media, and behavioural visual perception research in cognitive psychology and neuroscience. More broadly, our emerging work is driven by an interdisciplinary research mindset addressing human-centred responsible AI through a methodological confluence of AI, Vision, Psychology, and (human-factors centred) Interaction Design.
Multimodality in embodied interaction is an inherent aspect of human activity, be it in social,
professional, or everyday mundane contexts. Next-generation human-centred AI technologies, operating in
such contextualised everyday settings, will require an inherent foundational capacity to “make sense”
of —e.g., perceive, understand, explain, anticipate— everyday, naturalistic interactional multimodality.
This would be essential towards successfully achieving technology mediated (“human-in-the-loop” )
collaborative assistance, as well as ensuring compliance with emerging human-centred ethical and legal
requirements, performance benchmarks, and inclusive usability expectations. It is therefore crucial
that the foundational building blocks of such next-generation systems be semantically aligned with the
descriptive, analytical, and explanatory characteristics and complexity of human task conceptualisation,
performance benchmarks, and usability expectations. Against this backdrop, we define artificial visual
intelligence [
Within the scope of artificial visual intelligence are a wide-spectrum of high-level human-centred sensemaking capabilities. These capabilities encompass operational functions such as: • Visuospatial conception formation, commonsense/qualitative generalisation, analogical inference; • Hypothetical reasoning, argumentation, explanation, counterfactual reasoning; • Event based episodic maintenance & retrieval for perceptual narrativisation.
The afore enumeration is by no means exhaustive: in essence, in scope of artificial visual intelligence are diverse high-level cognitive visuospatial sensemaking capabilities —be it mundane, analytical, or creative— that humans acquire developmentally or through specialised training, and are routinely adept at performing seamlessly in their everyday life and work (e.g., driving a vehicle, tracking moving objects, navigating a crowded urban environment, engaging in sports, interpreting subtle cues in everyday people-communication from visual / gestural and auditory signals).
Our central focus is on the development of general, domain-independent methods that may be
seamlessly integrated as part of hybrid computational cognitive system, or even within computational
cognitive models / cognitive architectures [
Learning about Space, Motion, and Inter(A)ction In the present status quo, our research in (computational) neurosymbolic visual commonsense categorically addresses three key questions:
I. What kind of (relational) abstraction mechanisms are needed to computationally “make-sense” of
embodied multimodal interaction ?
II. How can (and why should) abstraction mechanisms (such as in I) be founded on behaviourally
established cognitive human- factors emanating from naturalistic empirical observation in
realworld applied contexts?
III. How to articulate behaviourally established abstraction mechanisms, preferences (etc) as
formal declarative models suited for computational modelling aimed at operational“sensemaking”
(encompassing capabilities such as abduction, relational learning, counterfactual inference) ?
Present work is particularly aimed at developing general methods for the semantic interpretation
of (multimodal) dynamic visuospatial imagery with an emphasis on the ability to neurosymbolically
perform abstraction, reasoning, and learning with cognitively rooted structured characterisations of
commonsense knowledge pertaining to space and motion. Here, we specifically emphasise:
• General foundational commonsense abstractions of space, time, and motion needed for
representation mediated (grounded) reasoning and learning with dynamic visuospatial stimuli (e.g.,
emanating from multimodal human behavioural signals in modalities such as RGB(D), video,
audio, eye-tracking and possibly even bio signals [
The vision that drives our scientific methodology is: » To shape the nature and character of (machine-based) artificial visual intelligence with respect to human-centred cognitive considerations, demonstrating an exemplar for developing, applying, and disseminating such methods in socio-technologically relevant application areas where: (a) embodied (multimodal) human interaction is inherent; (b) human-in-the-loop collaborative work is of the essence; and (c) normative ethico-legal compliance based on regulatory requirement and human-factors driven inclusive or universal design criteria is to be ensured.
Towards realising this vision, we adopt an interdisciplinary approach –at the confluence of Cognition, AI, Interaction, and Design– which we deem necessary to better appreciate the complexity and spectrum of varied human-centred challenges for the design and (usable) implementation of (explainable) artificial visual intelligence solutions in diverse human-system interaction contexts.
One of the key technical driving forces in our work is that of “representation mediated multimodal
sensemaking”. In essence, we consider (neurosymbolic) representation mediated grounding as being
significant in semiotic construction, e.g., enabling high-level meaning-making. This view stems from the
long-established value of “grounding” in Artificial Intelligence and related disciplines [
In this invited position statement, we have attempted to summarise our mindset and ongoing work in the CoDesign Lab towards: » Establishing a human-centric foundation and roadmap for the development of neurosymbolically grounded inference about embodied multimodal interaction as identifiable in a range of real-world application contexts.
The summary is not meant to be a comprehensive literature review; this may be obtained through the
cited works. For key technical details and to obtain a summary of open directions, we direct interested
readers to select publications: a compact starting point may be obtained via the comprehensive summary
in [