Interpretable machine learning seeks to enhance transparency in model decision-making, particularly in highstakes applications. Concept bottleneck models (CBMs) improve interpretability by using human-defined concepts as intermediate representations. Yet, they often depend on extensive manual annotations and may fail to capture relevant features beyond predefined concepts. We propose an iterative communication emergence framework for interpretable machine learning that integrates a concept bottleneck model with data-driven discovery of latent features. Our approach employs a sender-receiver architecture, where the sender encodes raw inputs into discrete latent signals refined via reinforcement learning, and the receiver uses these latent concepts to predict outcomes. Latent representations are aligned post hoc with human-observable concepts, which are automatically generated by a language model and validated statistically, enabling transparent explanations while reducing reliance on manual annotations. Experiments on a cat breed classification task demonstrate that our framework maintains high predictive performance while progressively refining interpretable concept representations. Results suggest that emergent latent concepts can meaningfully align with human-understandable attributes, facilitating more lfexible and scalable interpretability in deep learning models.
Interpretable machine learning is essential for high-stakes decision-making, where understanding model
behavior is critical. Rather than relying on post-hoc explanations of black-box models, researchers
advocate for inherently interpretable models [
Concept bottleneck models provide a structured approach to interpretability by introducing an
intermediate layer of human-defined concepts, enabling direct user intervention. These models achieve
competitive performance while enhancing transparency by allowing users to modify concept values.
However, CBMs are constrained by their reliance on manual concept annotations, which can be costly or
infeasible in specific domains. Additionally, they often underperform compared to unconstrained models,
limiting their practical adoption [
To address these limitations, post-hoc concept bottleneck models (PCBMs) have been proposed to
transform pre-trained neural networks into CBMs without sacrificing performance. PCBMs facilitate
external concept integration and enable eficient model editing to mitigate dataset biases [
CEUR
ceur-ws.org
and a reinforcement learning loop iteratively refines the concept representations.
promising approach involves automatically discovering latent concepts from model representations,
reducing reliance on predefined annotations while maintaining interpretability and narrowing the
accuracy gap between interpretable and black-box models [
Recent advancements leverage LLMs to generate human-readable concepts automatically. For
example, the language in a bottle (LaBo) framework utilizes GPT-3 to replace manual concept annotations
with natural language descriptions. However, this approach introduces challenges such as assessing the
validity of generated concepts and mitigating information leakage [
We propose an iterative language emergence framework within a concept bottleneck model pipeline to
achieve interpretable and scalable AI. The framework consists of two cooperative agents—a sender and a
receiver—that communicate through discrete signals of latent concepts. The sender converts raw inputs
into binary latent concept vectors, while the receiver, implemented as a simple linear classifier, maps
these vectors directly to predicted labels. A reinforcement learning-based feedback loop using proximal
policy optimization (PPO) [
We address an image classification task with training dataset {( () , () )} =1 where () is an image, () is the associated class label, and denotes the training dataset size. In addition, we have access to a set of human-observable concepts (HOCs) that can provide interpretable annotations (e.g., “slender body” or “no visible whiskers”). Let = { 1, … , } be the set of natural language descriptions of the HOCs, and () = { () 1 , … , () } be binary values indicating the presence or absence of the -th HOC in the -th image. We assume that LLMs can be leveraged to automatically generate candidate HOCs and () from class descriptions, domain-specific glossaries, or other textual corpora by prompting with questions such as “What are the visual features of ?’’ (where is the class label) and “does the image () have the visual feature ?”, respectively. In contrast to traditional CBMs, this assumption does not require concept generation by human domain experts. We emphasize that, however, the available HOCs are not necessarily efective features for the classification task.
Our goal is to develop a model that classifies images accurately and ofers transparent reasoning based
on a combination of discrete interpretable concepts. The discrete binary representation of HOCs (i.e.,
hard concepts) enhances interpretability and prevents information leakage [
The sender is a feedforward neural network (FNN) parameterized by , which processes raw input features () (e.g., an image of a cat) and outputs a vector of discrete latent concepts, () = ( 1() , … , () ).
These latent concepts represent task-relevant patterns that emerge from data. To enforce discreteness,
the continuous outputs of the FNN are thresholded using a predefined threshold :
() = {
1, if (
0, otherwise,
() ∣ () ) > ,
where (
() ∣ () ) is the predicted probability that concept
() is active for a given input () . This
binarization yields a vector (e.g., () = [
The set of discrete latent concept symbols forms a communication protocol between the agents. Each can be seen as a “word” in the emergent communication that the sender uses to describe the input’s relevant properties.
The receiver agent , parameterized by , is a simple linear classifier that takes the sender’s message () as input and outputs a prediction ̂ () of the final class label (e.g., the breed of a cat, such as “Persian”). The receiver is essentially the label predictor in the CBM pipeline: it maps from latent concepts to the target output. Because we ensure that any information used by the receiver to make the prediction must pass through the bottleneck of discrete concepts (intended to be human-understandable), its decision process is more transparent than a direct end-to-end model.
We optimize the sender’s policy to communicate useful concepts using reinforcement learning (RL). The sender is updated with a policy gradient method – specifically, proximal policy optimization (PPO) – to maximize the expected reward. The reward for the sender’s action (i.e., message () ) is computed after the receiver predicts a label ̂ () as follows: = { + ⋅ max( )̂, − ⋅ max( )̂, if arg max( )̂ = , otherwise, where max( )̂ represents the maximum logit for the predicted class, is a positive scaling factor for correct predictions, and controls the penalty for incorrect classifications. At each training iteration, we sample a batch of inputs, let the sender produce messages , and let the receiver predict labels .̂ Then, the receiver’s parameter is trained via supervised learning using cross-entropy loss, and the sender’s parameter is updated by the PPO algorithm. This training process can be seen as the agents jointly evolving a more efective shared communication protocol. (1) (2)
the ℎ concept to a HOC.
While the latent concepts emerge autonomously during training, we align them with HOCs in a post hoc fashion using statistical validation. The proposed algorithm consists of the -th cycle consists of three steps: (1) training the receiver for ℎ iterative cycles, where using supervised learning and updating the sender via PPO during each epoch, (2) choosing the best parameter pair ( () , () ) among epochs based on validation accuracy, (3) establishing a new alignment mapping from a latent and {
In the step (3), the proposed method generates the latent concept vector () from the sender () for each image () in the training set. For each pair of latent concept and HOC dimension HOC , we construct a contingency table of co-occurrence statistics between the presence vectors { (1), … , ( ) } (1), … , ( ) } and apply Fisher’s exact test to calculate the -value of the dependence. We take the pair (, ) such that the -value is lowest and establish a mapping from the latent concept to HOC if it indicates a statistical significance (i.e., < 0.1 ).
Once a mapping from to HOC is established, we substitute the value of with the corresponding
HOC annotation HOC in all subsequent processes without modifying the model’s internal computation
or predictions. For example, consider a scenario where the agents are trained to classify images of cats,
and 2 is aligned with the HOC1 “long whiskers” at the end of the first cycle. If the sender outputs
() = [
The alignment mapping from latent concepts to HOCs is cumulative; once a latent concept is aligned
to a HOC, that mapping is retained in all subsequent cycles. Suppose that 5 is aligned with HOC8 “flat
nose” in the next cycle of the example above. If the sender outputs () = [
Over successive iterations, this cycle progressively refines both the predictive power and interpretability of the model. Latent concepts that prove useful for the task and show consistent alignment with HOCs become more semantically meaningful, while unaligned concepts may remain the model’s newly discovered factors as novel concepts, which can be studied further by experts or potentially added to the set of HOCs for future iterations. Importantly, the sender’s parameters remain persistent across iterations, enabling stable concept refinement, while the alignment mapping evolves to capture emerging associations. This design allows the model to maintain high task accuracy while ofering increasingly transparent, concept-based explanations for its decisions.
We evaluate our framework on a subset of the Oxford-IIIT-Pet dataset [
Our experiments follow the iterative training procedure described in Section 2. The sender and receiver are trained over five iterative cycles ( = 5), each consisting of 30 epochs ( ℎ = 30), with a batch size of 32. PPO settings include n_steps = 64, a batch size of 16, and a learning rate of 3 × 10−4. All other hyperparameters follow the default values in stable-baselines3 (v2.5.0).
The alignment between latent concepts and HOCs is updated at the end of each iterative cycle using Fisher’s exact test, applying a significance threshold of < 0.1 . Model performance is evaluated via classification accuracy and cross-entropy loss on both validation and test sets. We also provide qualitative analysis by inspecting the translated, human-readable concept vectors.
To evaluate the framework’s full potential under controlled conditions, we isolate the model’s behavior from noise introduced by LLMs. While candidate HOCs are automatically generated via a large language model, human involvement is restricted to annotating the presence or absence (0/1) of each predefined concept. This preserves the automation of concept discovery while ensuring label quality.
For comparison, we trained a traditional CBM using the same label predictor architecture as our proposed method. To evaluate the efect of concept supervision, we trained CBMs using 5, 10, and 15 randomly selected HOCs. Each model was trained for 150 (= × ℎ ) epochs using a combined loss function: binary cross-entropy for HOC prediction and cross-entropy for label prediction. We repeated each configuration across five random seeds and report the mean test accuracy.
We report results from a single trial of the proposed iterative framework, which progressively aligned latent concepts with HOCs over five training iterations. In each iteration, one statistically significant alignment was identified via Fisher’s exact test ( < 0.1 )—e.g., 15 with HOC21 (slender body) in Iteration 1 and 16 with HOC18 (long, tapered tail) in Iteration 5. Full alignments and metrics are shown in Table 1. These results indicate that the framework refines latent structures over time, aligning emergent concepts with HOCs while maintaining high predictive accuracy. Unlike traditional CBMs that rely on predefined concepts, our method identifies meaningful associations post hoc, enabling more flexible and scalable interpretability.
Validation accuracy increased from 85% to 97.5%, and test accuracy from 75% to 100% over iterations. Validation loss declined from 0.8490 in iteration 1 to 0.3675 in iteration 5, with minor fluctuations across epochs. These metrics indicate improved performance over time, although gains were not strictly monotonic. This suggests a potential relationship between concept alignment and model confidence. As training progressed, more stable predictions were observed—likely enabled by PPO-based senderreceiver optimization, which allowed latent representations to evolve without causing abrupt changes in downstream accuracy. However, whether such stability generalizes to datasets with more complex feature distributions remains an open question.
To further examine the relationship between latent units and human-interpretable concepts, we visualize sample groupings in Figure 2. Subfigure 2a compares latent concept 5 with HOC24, and (a) 5 and HOC24 (Slim oval paws).
(b) 17 and HOC16 (no visible whiskers).
Subfigure 2b compares 17 with HOC16. In each panel, the top two rows are grouped by HOC values, and the bottom two by latent activations. For 5, some visual consistency is observable with the paw shape associated with HOC24. For 17, activated samples tend to belong to a consistent breed (e.g., Russian Blue), though the whisker attribute associated with HOC16 is not always present. We discuss this type of partial alignment and its implications in Section 4. These examples reflect the post hoc nature of alignment in our framework—some latent features align well with visual attributes, while others only partially capture the associated HOC. This underscores the need for more robust validation methods to assess the semantic coherence of emergent representations.
The traditional CBM achieved a mean test accuracy of 71.5% when trained with 5 randomly selected HOCs. Increasing the number of HOCs to 10 and 15 improved the mean test accuracy to 96.5% and 99.5%, respectively. These results indicate that strong performance in the traditional CBM setting is dependent on supervision from a suficiently large and informative concept set. In contrast, our framework achieved 100% test accuracy by the final iteration while relying on only 5 HOC alignments selected through iterative discovery. Performance trends are illustrated in Figure 3. This highlights its ability to dynamically refine concept representations based on data, without requiring full concept annotations. The progressive nature of alignment, combined with strong classification performance, suggests potential advantages in generalization and interpretability when compared to fixed, manually labeled bottlenecks.
While the iterative communication-emergence CBM framework shows promise, it has limitations in concept coverage and consistency. In our experiments, only five latent concepts aligned confidently with HOCs, leaving many unmapped due to weak associations or missing human-defined attributes. Future work could cluster activations and analyze them via language models or human feedback to expand or refine the HOC space. Some unmapped concepts may reflect novel patterns, but not all are necessarily interpretable.
We did not explicitly assess mapping consistency across training runs or datasets. Although aligned
concepts remained fixed once identified during our single trial, their stability under diferent data
distributions or longer training remains unverified. Initial assignments may vary due to dataset
properties or learning dynamics, potentially leading to concept drift. Addressing this may require more
lfexible alignment strategies and evaluations across multiple runs [
Limitations also arise in the scope of interpretability. Our evaluation relies on statistical correlations with predefined HOCs, confirming whether a latent factor rediscovers a known concept, but not how it is internally represented. The fixed binarization threshold ( = 0.5 ) may also afect which concepts align but was not evaluated. Additionally, the sender is optimized for prediction accuracy, not concept-level supervision, allowing spurious correlations to persist. Although aligned variables are filtered before prediction, the sender may still generate them, with no mechanism to discourage reliance on noninterpretable factors. Future work could examine the threshold’s efect, use visualization or attribution methods to interpret latent factors, and incorporate causal or adversarial regularization to mitigate spurious patterns. Expert review may help define new HOCs, and applying the framework to diverse datasets will be essential to assess generalizability. The traditional CBM was not further fine-tuned. While the comparison still highlights diferences in reliance on supervision, future work should explore whether tuning CBM hyperparameters improves performance. In our PPO reward, we use the maximum predicted logit as a confidence measure. While this supports interpretability, exploring alternatives such as maximum softmax probability or output entropy remains a promising direction.
A key limitation is that the current alignment strategy permits only many-to-one mappings from latent concepts to HOCs, without supporting many-to-many relationships. In practice, multiple latent concepts may redundantly align with the same HOC, while a single latent concept may also encode multiple abstract or overlapping attributes. Post hoc alignment does not influence training in our current setup. Incorporating soft alignment rewards could encourage interpretability as associations emerge, but must be balanced against model flexibility to avoid the rigidity of traditional CBMs.
This work presents an iterative communication emergence framework for interpretable machine learning, integrating reinforcement learning into a CBM to enable emergent concept discovery. Unlike traditional CBMs that rely on predefined annotations, our approach allows the model to autonomously align latent representations with human-understandable attributes through iterative refinement.
Experiments on cat breed classification demonstrate that interpretability improves progressively over training—latent concepts become increasingly aligned with observable features—while predictive accuracy also improves. The model achieved perfect test performance by the final iteration, despite relying on only five post hoc-aligned HOCs, without direct concept supervision. Aligned concepts remained stable once discovered, suggesting robustness in alignment.
While our framework enables latent concepts to align with human-interpretable attributes, it has limitations. Many concepts remain unmapped, and the current many-to-one alignment may oversimplify complex or overlapping patterns. Since alignment occurs post hoc, it does not guide training. Future work could incorporate soft alignment objectives, adaptive mappings, and visualization techniques to better integrate interpretability into the learning process. Ultimately, bridging emergent representations with human semantics ofers a path toward more transparent and trustworthy AI systems. The author(s) used GPT-4o and Grammarly for grammar and minor wording improvements and reviewed and edited the content as needed. The author(s) take full responsibility for the publication’s content.
This work was partly supported by JST CREST grant number JPMJCR22M2.