A central challenge in explainable AI, particularly in the visual domain, is producing explanations grounded in human-understandable concepts. To tackle this, we introduce OCEAN (Object-Centric Explananda via Agent Negotiation), a novel, inherently interpretable framework built on object-centric representations and a transparent multi-agent reasoning process. The game-theoretic reasoning process drives agents to agree on coherent and discriminative evidence, resulting in a faithful and interpretable decision-making process. We train OCEAN end-to-end and benchmark it against standard visual classifiers and popular post-hoc explanation tools like Grad-CAM and LIME across two diagnostic multi-object datasets. Our results demonstrate competitive performance with respect to state-of-the-art black-box models with a faithful reasoning process, which was reflected by our user study, where participants consistently rated OCEAN's explanations as more intuitive and trustworthy.
Recent developments in visual models have led to highly
accurate image classifiers, but their “black-box” nature present
challenges in trust, accountability, and human-aligned
reasoning. At the crux of explainable AI (XAI) lies the
fundamental goal of clearing the opacity of machine
learning models, making them more intelligible to human users.
This need is particularly acute in high-stakes applications,
such as medical imaging [
To navigate this hurdle, we draw inspiration from recent
work on ante-hoc explainability, where the model
architecture is carefully designed to expose internal reasoning [
In this work, we present OCEAN, an inherently transparent and human understandable framework for visual classification. OCEAN casts image classification as a neurosymbolic collaborative game between agents who must iteratively select visual symbols towards a shared classification in the Consensus Game.
OCEAN is informed by recent trends in neuro-symbolic
reasoning [
Together, these contributions demonstrate the impact of integrating object-centric learning with agent-based collaboration in building more interpretable, faithful, and humanaligned visual classifiers.
Debate Games Recent research has explored using
structured agent interactions to enhance model explainability
through debate or dialogue, simulating a conversation
between agents. Visual Debates [
In parallel, the Consensus Game [
Explainability methods Without any need to modify
the underlying architecture of pre-trained models, post-hoc
methods are able to generate explanations by diagnosing
the model’s decision. These methods are widely used due
to their flexibility, as they allow explainability to be applied
to highly performant models without any alterations.
Examples include Grad-CAM [
Conversely, ante-hoc methods aim to embed
explainability directly into a model’s decision-making process, making
the model inherently interpretable. These methods seek
to avoid the pitfalls of post-hoc explanations by producing
faithful explanations during the model’s inference time and
aligning with the model’s reasoning. Thus, the
explanations generated from the model stem from fully transparent
predictions. This makes these models very suitable for
highstakes applications where trust, accountability, and
transparency are essential. Designing such models requires a
rethink of the structure of making predictions, often achieved
by introducing architectural constraints, interpretable
feature transforms, or modularising parts of the model.
Examples include ProtoPNet [
Neuro-Symbolic Learning is a vast area of research based
on combining neural elements with symbolic reasoning to
improve interpretability and generalisation in deep learning.
In vision, frameworks like NeSy-XIL [
Notations We begin by introducing the notation used throughout the paper. Each input images ∈ R× × is processed by the Slot Attention module, which outputs a fixed number of slot representations. Each slot is a vector of dimension , resulting in a set of slot encodings = {1, . . . , } for each image.
While the number of players can be changed, we
present our Consensus Game module as consisting of
two players, 1 and 2, who take turns presenting a
sequence of arguments, 1 = {11, 21, ..., 1} and 2 =
{1, 22, ..., 2}, before ultimately making their
respec2
tive final claims, 1 and 2. Players collaboratively select
the most salient slots as arguments to support their claims,
which are then aggregated to produce the final classification.
Finally, both players are equipped with a shared utility ,
representing the efectiveness of their choices of arguments.
Similar to [
⟨, { 1, 2}, {1, 2}, {1, 2}, ⟩.
The sequence of arguments put forward in the game supports the final claims made by both players. An argument for player at stage ∈ {1, .., } composes of two elements: a slot selection and a claim , such as in the following:
= (, ).
Attention (SA) [
Agent modelling Player ’s policy : ℋ → maps the current game history ℋ, comprising of all arguments put forward, at time step to the next action , which is an argument at turn of the game. Here, we emphasise the distinction between time step and turn . Multiple players can share the same turn number , but the time step always increases uniquely with each action.
Each player in the Consensus Game operates as a neural agent composed of several cooperating components, as seen in Figure 2. The architecture is designed to process partial observations (slots) sequentially, maintain memory of prior selections, and make decisions that improve classification accuracy over time. The player is implemented as a composition of six main components: a recurrent state encoder, an index embedder, a modulator, a policy network, and a classifier. These components are trained jointly to optimise the performance objective of succinct informative selections and accurate classifications.
To reason over sequences of slot selections, each player
maintains an internal memory of the selection history ℋ
using a Recurrent Neural Network (RNN), capturing the
evolving state of the game from their perspective. At turn
, player observes a selected slot with embedding
and a index embedding (adapted from transformers [
These embeddings are summed and passed through the player-specific modulator network ℳ, a lightweight multilayer perceptron (MLP) that reparameterises the input into a more informative representation: = ℳ( + ) ℎ = ℛ(, ℎ− 1)
This representation is then fed into the RNN ℛ of player , which updates its hidden state ℎ at time step :
To enable the policy network to make informed decisions from the very beginning of the game, we initialise the RNN hidden state ℎ0 with knowledge of the entire slot pool. This is done by sequentially processing all available slot embeddings through the RNN prior to the start of the game. However, this initialisation incorporates full information, which is incompatible with our intention with the classifier, where predictions should be based solely on cumulative evidence. To resolve this, we maintain two separate RNN hidden states per player throughout the game. The first, denoted ℎΠ,, is used by the policy network and is initialised with all slots, providing a strong prior for action selection. The second, ℎ,, is used by the classifier and is initialised to zero, updating only as new slots are selected during gameplay. This separation ensures that the classifiers operate under partial information constraints, while the policy network benefits from full-slot context for strategic selections. The two RNNs may share architecture and parameters, but they operate over disjoint information flows: ℎΠ, = ℛΠ,(, ℎΠ− ,1) ℎ, = ℛ,(, ℎ−,1) (Policy path) (Classifier path)
The player’s policy network is an MLP that produces a distribution over available actions at each time step. It receives the hidden state ℎ as input and produces a logit vector , which is passed through a softmax to define a stochastic policy:
= Π (ℎΠ,)
Central to each player is a classifier that integrates the selected slot features to predict the true class of the input instance. Each player maintains its own classifier instance, which receives the cumulative set of slot embeddings chosen up to the current time step. The classifier produces a distribution over class labels via a softmax layer, and the confidence assigned to the true class label is used to compute both sparse and dense reward signals, depending on the training regime. The classifier thus serves both as a target for individual inference and as a feedback channel to guide each player’s learning.
Reasoning via consensus game The players as described above participate in the Consensus Game, where they aim to learn optimal strategies for correct, efective and concise reasoning for classification. Our Consensus Game module enables any number of agents to iteratively select slot information with the goal of jointly identifying the correct class label. This interaction is framed as a self-play reinforcement learning game, where each agent must communicate selectively to reach a consensus. The module can be seen in two complementary ways: as an ante-hoc visual explanation mechanism that reveals how decisions emerge through inter-agent reasoning, or as a novel explainable classification approach leveraging multi-agent reasoning. Following that, we developed an environment (Figure 3) for facilitating transparent player interactions. During training, it provides reward feedback to the players.
For reward feedback, we have a shared dense reward function Γ based on player confidence toward the true label. This reward structure encourages both agreement and correctness, rewarding agents only when consensus aligns with the ground truth, and penalising both blind agreement and disagreement.
At every turn of the game of length , each player outputs a probability distribution over class labels. Let denote the probability assigned to the ground-truth class by player at turn . The reward for that turn is computed as the gain in confidence relative to a baseline value 0, which can be initialised as zero or the uniform probability ||− 1. The per-turn reward is thus: = − 0
This formulation incentivises agents to select informative slots that gradually build up confidence in the correct class, even before reaching the final prediction step. We also experimented with an alternative reward scheme based on relative changes in confidence between players, defined as: = − −− 1
This approach also improved learning eficiency but introduced a risk of reward hacking, where agents could suppress early confidence to artificially inflate perceived improvement later. This undermined the integrity of the learning signal. As a result, we adopted the more stable and interpretable absolute-confidence-based dense reward formulation. The shared utility function Γ is then defined as the expected reward over the joint policy of both agents: Γ( 1, 2) = E1,2∼ ( 1, 2) [︀ Γ(1, 2, )]︀ , where 1 and 2 denote the policies of players 1 and 2, respectively. During training, the agents are optimised jointly to maximise Γ, promoting cooperative behaviour that balances consensus with predictive accuracy.
In the Consensus Game, each player seeks to maximise their expected utility by learning a policy that governs their slot selection and final claim. Since both agents share a reward function Γ and are penalised for disagreement or incorrect consensus, their optimal strategy is cooperative.
Let denote the policy of player . Each policy maps a game state history to a distribution over available actions. The goal of each player is to learn a policy * that maximises their expected reward under the joint policy = ( 1, 2): * = arg max E [Γ]
The model is trained end-to-end through a combination of reinforcement learning and supervised learning objectives. Each epoch consists of multiple consensus games played over batches of input instances. For every batch, slot embeddings and class labels are extracted and passed to the game environment, which orchestrates the game dynamics such as the starting player, turn order, and game termination. To mitigate ordering bias, the starting player is alternated across batches. During training, the number of turns per game is fixed, simplifying reward attribution and ensuring consistent episode lengths. During inference, the Consensus Game operates very similarly to training mode depending on the end condition. Instead of running for a ifxed number of turns, the game terminates once a certain condition is satisfied or until a maximum number of turns. This setup enables the agents to reach a dynamic consensus based on selected factors, rather than being constrained to a fixed interaction length. These end conditions are based on one of or some of the following: repetition, confidence and consensus, as elaborated in Section 4.
Maximisation [
A B C A B C A B C A B C based on the current slots, with upstream Slot Attention module frozen, which is equivalent to the training protocol of the pipelined version of our architecture. The alternating updates decouple the learning dynamics of the two components and reduce interference.
A practical setup would also warm up the Slot Attention module by training it without the downstream Consensus Game. This allows for efective learning in the Consensus Game when training does begin for the agents following the warm up. Alternatively, in the interest of time, we typically load in pre-trained weights.
Following EM training, the slot reconstructions look identical to the ones obtained from the original pipeline setup, suggesting that the added reward signal did not compromise the perceptual quality of the extracted slots. By framing the optimisation steps as an EM loop, we decouple the pressure from the combined loss from Consensus Game, while allowing some reward signal to flow back to the slot autoencoder. We also observed incremental improvements in accuracy and quality of explanations as a result of E2E learning.
Datasets Suitable datasets must feature multiple objects with compositional variability, where the images exhibit diverse combinations of object attributes such as colour, shape and position. Importantly, the images should be labelled based on logical rule-based conditions such as “a red square left of a green circle”. These requirements ensures that the task has a well-defined reasoning component. Our experimental setup utilises two datasets that fit the criteria: CLEVR-Hans and a relabelled version of Multi-dSprites. We focus our experimental eforts towards the CLEVR-Hans dataset, particularly due to the non-confoundedness of its test split, although we also present accuracy metrics for the confounded validation split. CLEVR-Hans also provides versions with 3 labels (CLEVR-Hans3) and with 7 labels (CLEVR-Hans7).
Baselines As the focus of this research is explainability, we will be evaluating both the predictive performance and the explanation quality of our framework. We used the standard feature extractor + multi-layer perceptron (MLP) framework for our two prediction baselines. Typical models use a Convolutional Neural Network (CNN) for feature extraction, which informs our first baseline. The other swaps out the CNN for SA with slot size 128, where classification is based on slots. Baseline results were collected using a standard ResNet18 implementation trained on images of resolution 224 × 224, and a Slot-based classifier evaluated at resolutions of 64 × 64 and 128 × 128 across all datasets (except for Multi-dSprites on the 128 × 128 configuration). The baselines serve as performance anchors, particularly in scenarios where full image observability and unconstrained inference are available. The prediction accuracies are detailed in Table 3.
Conversely, to test our framework’s transparency and reasoning quality, we compare with standard post-hoc mechanisms. Namely, we apply Grad-CAM, LIME and Gradient
CH7 MdS CH7 MdS CH7 MdS CH7 92 100 85 81 100 100 65 100 73 96 SHAP to both prediction baselines. Explanations generated from these setups would be used to compare with our framework, presented in a survey.
CLEVR-Hans7 (CF) CLEVR-Hans7 (NCF) CLEVR-Hans3 (CF) CLEVR-Hans3 (NCF)
Multi-dSprites
ResNet
224 90.32% 83.54% 93.11% 60.93% 83.20%
64 86.70% 86.66% 86.62% 85.60% 92.72%
Slot MLP
128 85.34% 86.39% 87.90% 88.76% —
Config A
64 77.64% 70.15% 78.50% 52.58% 79.76% Configurations To evaluate the impact of diferent gameplay dynamics and learning settings, we trained three representative configurations of the Consensus Game module. These configurations are denoted as Config A, B and C. They are summarised as follows: • Config A: We constrain to 64 × 64 images for lower resolution slots in the Consensus Game, while fixing to a consensus-based end condition. Here, games conclude when players arrive to the same claim while having a confidence that exceeds the threshold of 70% confidence. • Config B: We augment the end condition of the 64 × 64 variant to trigger on repeated selections by any player instead (repetition-based). To encourage this behaviour and end games early, we reward repeated good selections, which reinforce ideas and the shared prediction. • Config C: We repeat Config A for the 128 × 128 variant of the Slot Attention module, with some modifications in the network widths to cater the increase in data.
Evaluation Metrics For evaluating the performance of our method, we measure and compare various properties of the framework, including slot reconstruction loss, consensus rate, game length, slot uniqueness, and empty slot usage. Formally: (i) slot reconstruction loss is the mean squared error of the reconstructed image and the original image, (ii) consensus rate measures the frequency of the agents aligning on the same final claim, (iii) game length is the number of arguments presented by all players during the game, (iv) slot uniqueness represents the rate at which the players select a unique slot, and (v) empty slot usage is the percentage of empty slots being presented; “empty” meaning slots that do not map to objects in the image. We identify empty slots using a heuristic based on the attention values of the corresponding slots.
Results We utilise three representative configurations of our model, denoted as Config A, B, and C. We report results (Table 1), including accuracy and F1-Score, on three synthetic multi-object datasets, CLEVR-Hans3, CLEVR-Hans7, and Multi-dSprites. This evaluation will help quantify how well our framework performs classification while maintaining the structural constraints needed for interpretability and partial observability of the entire input. In contrast, our framework operates under stricter constraints, where agents make predictions on partial observations through sequential slot selection, and are jointly optimised for both prediction and interpretability. We observed that the configurations with the “Consensus” end condition outperformed Config B with its repetition-based end condition. The drop in performance can be attributed to the agents adopting repetition-based strategies that are less aligned with the goal of achieving consensus. Interestingly, Config B achieves comparable performance on CLEVR-Hans3 with the other configurations, likely due to the less complex combinations of objects that are needed for a conclusive classification. We also consider the slot reconstructions, ensuring that interpretability and object-centricity were not compromised as a result of E2E training.
While these results are competitive wrt baselines in terms of classification accuracy, it is important to contextualise the numbers. Our model is not solely optimised for accuracy, but it is explicitly designed to support transparent reasoning chains aligned with human interpretability, as demonstrated in Figure 5. However, our results for the non-confounded split for CLEVR-Hans7 show promise when compared with the metrics of the confounded split, displaying only a small drop in performance. We leave further investigation on the robustness and generalizability aspects to future work.
Generally, we attribute the performance hit to partial observability of the input image, where our framework makes classifications on the slots that the players select. However, the poor predictive performance for the non-confounded cases likely stems from overfitting and an underlying confusion between objects of the same shape, regardless of the other properties. A contributing factor is the entangled slot representations, which group correlated features like size, colour and shape during training, making it hard for our framework to decouple those attributes when these correlations no longer hold. This entanglement makes it dificult for the players to reason over fine-grained details that are needed to perform well for the non-confounded test set. This behaviour reflects the model’s tendency to overfit to prominent but irrelevant features rather than selecting holistically over the entire set of objects in the scene, which is seen in (a), (b), and (d) of Figure 5.
Survey As the main investigation was on the explainability and interpretability of our framework, we conducted a survey asking participants to qualitatively assess five explanation methods: LIME, Grad-CAM, Grad-CAM on Slot Attention, Gradient SHAP and our OCEAN framework. GradCAM was the only explanation baseline built on top the slot-based classifier chosen for this survey. This is due to the explanations for LIME and Gradient SHAP being indistinguishable from their CNN-based counterpart.
Participants were tasked with guessing the predicted class of the model based on the explanations alone. Explanations generated from most explanation mechanisms rely on the input image for highlighting regions of the image. Thus, the survey includes six statements that the participants rate using a Likert scale. They score each statement from 1 (Strongly Disagree) to 5 (Strongly Agree): • It is clear which parts of the image is the explanation. • The explanation was easy to interpret without prior technical knowledge. • The explanation would help me identify errors or biases in the model’s prediction. • The explanation only focused on relevant parts/objects of the image. • The explanation helped me understand why the model predicted the class.
• I am satisfied with the quality of the explanation.
Our survey results (Table 2) reveal distinct diferences in the interpretability and user satisfaction among the five explanation methods evaluated. For each question, our framework leads in the average score, indicating that our framework provided good explanations for the input images. While the results are favourable, we should cautiously interpret their implications. In the context of synthetic, multi-object datasets, our framework performs well due to our object-centric encoder and sequential reasoning module, making our generations easier for users to interpret. As such, the generalisability of our framework to real-world image datasets should be investigated to determine whether the interpretability advantages of our framework persist for more complex or noisy images.
Through this work, we gained insight into the delicate
tradeofs between performance, transparency, and
interpretability. Faithful explanations require an architecture where
they can be derived naturally from the decision-making
process. Our findings show that multi-agent interaction,
when structured carefully, provides a promising scafold
for such reasoning. An immediate improvement of this
work would be upgrading the encoder component for a
more scalable Slot Attention variant that uses vision
transformer architectures, such as in [22] and [
Collectively, these challenges point toward the importance of future research into more scalable object-centric models, principled design for interpretable systems, and robust evaluation frameworks for explainability. We learned that explanations are not merely outputs to be generated after the fact, but processes that should be embedded into opaque systems to ensure transparency and trust. This shift in perspective is shared across the industry with broader implications for AI trustworthiness, particularly in high-stakes domains like medical diagnosis or autonomous decisionmaking. As a result, we are heading straight on to a world where understanding why a model makes a decision is just as critical as the decision itself.
Acknowledgments This research was partially supported by the ERC under the EU’s Horizon 2020 research and innovation programme (grant no. 101020934), and by EPSRC postdoctoral prize fellowship.
Generative-AI Declaration Authors have employed the use of Generative AI tools, such as ChatGPT, to refine/rephrase certain parts of the paper, while maintaining the conceptual originality of the work. IEEE Signal Processing Magazine 13 (1996) 47–60. doi:10.1109/79.543975. [22] M. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow, T. Xiao, C.-J. Simon-Gabriel, T. He, Z. Zhang, B. Schölkopf, T. Brox, F. Locatello, Bridging the gap to real-world object-centric learning, in: The Eleventh International Conference on Learning Representations, 2023. URL: https://openreview.net/forum?id=b9tUk-f_ aG. [23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: European conference on computer vision, Springer, 2014, pp. 740–755. [24] Y. Choi, Y. Uh, J. Yoo, J.-W. Ha, Stargan v2: Diverse image synthesis for multiple domains, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8185–8194. doi:10.1109/ CVPR42600.2020.00821.