Autonomous vehicles and intelligent transport systems depend on advanced computer vision technologies, with object detection being a critical task. This enables vehicles to recognize and respond to surrounding objects effectively, with region proposal identifying potential object locations early in the detection process, crucial for timely responses in autonomous driving [1]. Traditional techniques like Non-Maximum Suppression (NMS) often struggle to balance precision and recall, especially in dynamic environments. Solovyev et al. [2] introduced Weighted Boxes Fusion (WBF), using confidence scores to average overlapping bounding boxes from multiple detection models, demonstrating superior performance over conventional methods.
The integration of Multiagent Systems (MAS) into object detection workflows offers new perspectives to address traditional challenges [3,4]. MAS provide dynamic and adaptable decision-making capabilities, enhancing autonomous vehicles' ability to handle complex, unpredictable road conditions. MAS support distributed and adaptive processing [5], complementing modern GPU-based computer vision. By distributing tasks across agents, MAS enhances system flexibility and resilience, especially in dynamic environments like autonomous driving or video surveillance [6,7,8]. Each agent manages a subset of tasks, improving resilience to errors [9,10].
MAS can adjust strategies based on scenarios [11,12], adapting parameters for bounding box fusion based on context, scene complexity, or environmental changes [13]. Agents operate independently on different hardware, optimizing processing power and allowing system scalability [14]. Local decisions are combined through a global process, enhancing accuracy [15]. MAS can continually learn from 13th International Workshop on Agents in Traffic and Transportation (ATT 2024) held in conjunction with ECAI 2024 alaa.daoud@insa-rouen.fr (A. Daoud); corentin.bunel@insa-rouen.fr (C. Bunel); maxime.gueriau@insa-rouen.fr (M. Guériau) 0000-0002-3640-327X (A. Daoud); 0000-0001-6637-9795 (C. Bunel); 0000-0002-8742-6623 (M. Guériau) their environment and from the interactions between agents [16]. This potential for adaptive learning motivates the agentification approach, as it opens the possibility for future enhancements. By achieving an agentified method, we can later integrate learning capabilities to further improve adaptability and performance in evolving object detection scenarios. Agent-based approaches are well-suited for integrating diverse models and data sources [17], which is essential for the ensemble approaches used in WBF where predictions from different models are combined.
Agentifying output refinement methods such as NMS or WBF involves assigning individual agents to handle specific bounding boxes, enabling dynamic adjustment based on individual box characteristics. This approach addresses real-time processing requirements and improves scalability and fault tolerance by decentralizing the decision-making process [18,19,20]. In this work, we aim to design and implement a proof-of-concept system integrating MASs into the process of improving bounding boxes in object detection. We will develop two behaviors for the bounding box agents: one replicating the state-ofthe-art Weighted Boxes Fusion (WBF) method in a decentralized manner, and the other introducing competitive behavior where agents interact based on Intersection over Union (IoU) and confidence values. Finally, we will deploy the system and assess its performance using the COCO dataset, testing various levels of competition and cooperation between agents. The remainder of this paper is structured as follows: Section 2 presents the related work in object detection, multiagent systems, and their integration. Section 3 details the system architecture and design principles of the AWBF method. Section 4 describes the implementation of the proof-of-concept system and the development of agent behaviors. Section 5 discusses the experimental setup and evaluation using the COCO dataset. Section 6 presents the results and analysis of the experimental evaluation. Section 7 concludes the paper with a summary of findings and future work directions.
Object detection is a fundamental task in computer vision, critical for intelligent transportation systems (ITS) applications such as autonomous driving, traffic monitoring, and surveillance. The integration of MASs into object detection workflows offers significant potential to enhance system efficiency, robustness, and adaptability. This section reviews recent advancements in object detection techniques relevant to the ITS, with a focus on bounding box fusion and the role of MASs. Wang et al. (2019) presented the Multi-Stage Complementary Fusion (MCF3D) network, an end-toend architecture for 3D object detection that integrates LiDAR and RGB data. This network employs attention mechanisms and prior knowledge to achieve state-of-the-art results, enhancing the detection accuracy necessary for autonomous driving applications [21]. Qian et al. (2020) proposed an improved object detection method for remote sensing images, incorporating a novel bounding box regression loss and a multi-level features fusion module. This method enhances the precision of object localization, which is crucial for applications such as traffic monitoring and vehicle detection [22]. Solovyev et al. (2021) introduced the Weighted Boxes Fusion (WBF) method, which averages overlapping bounding boxes from multiple detection models using confidence scores. This approach demonstrated superior performance over traditional techniques, highlighting the effectiveness of fusion methods in improving object detection accuracy [2]. This method is particularly relevant for ITS applications where robust and accurate object detection is paramount for safety and efficiency.
Zhang and Wu (2022) proposed a multi-view feature adaptive fusion framework that enhances 3D object detection by optimizing depth feature fusion and loss function design. This approach improves the regression accuracy of bounding boxes, which is essential for ITS applications where precise object localization is critical [23]. Liu et al. (2023) developed the Fusion network by Box Matching (FBMNet) for multi-modal 3D detection. This method aligns features at the bounding box level, providing stability in challenging scenarios such as asynchronous sensors and misaligned sensor placements, common issues in ITS applications [24].
Introducing MAS to object detection and computer vision systems is not a new idea. For example Choksuriwong et al. (2005) developed a MAS for image understanding that localizes and recognizes objects using a distributed system implemented on a cluster computer. This approach leverages invariant features and supervised classification to improve object recognition accuracy, which is vital for traffic monitoring systems [25]. However, the application of MAS in these areas has decreased recently with the advancements in machine learning techniques and their improved performance in handling object detection tasks. Despite this shift, some researchers have continued to explore the potential of MAS in object detection through various approaches.
Jiang et al. ( 2019) proposed a multi-agent deep reinforcement learning (MADRL) approach for multi-object tracking, using YOLO V3 for object detection and Independent Q-Learners (IQL) for policy learning. This method achieves better performance in precision, accuracy, and robustness compared to other state-of-the-art methods, which is particularly beneficial for real-time traffic monitoring and surveillance [26].
Fekir and Benamrane (2015) introduced a MAS for boundary detection and object tracking using active contours and multi-resolution treatment. This system improves object boundary detection and tracking through cooperative agent strategies, enhancing the accuracy and efficiency of ITS applications such as vehicle and pedestrian tracking [27].
Vincent et al. ( 2022) described a MAS using stereovision for perception, enabling agents to collaborate and enhance scene understanding through graph matching algorithms. This approach addresses challenges in correspondence identification and non-covisibility, critical for ITS applications such as multi-vehicle coordination and traffic management [28].
Mahmoudi et al. ( 2013) utilized a MAS for object recognition in complex urban areas, leveraging WorldView-2 satellite imagery and digital surface models. This system improves object recognition accuracy through knowledge-based reasoning and cooperative agent capabilities, essential for urban traffic monitoring and smart city applications [29].
In light of the existing work, our proposal aims to integrate the strengths of both bounding box fusion techniques and MASs to develop a more robust and efficient object detection framework tailored for ITS applications. Our approach leverages the distributed processing capabilities of MASs to enhance the accuracy and scalability of bounding box fusion methods. By incorporating advanced fusion techniques and adaptive agent strategies, our system aims to address the limitations of existing methods, such as handling dynamic environments and improving detection precision. Our contributions include:
1. A multi-agent based framework for bounding box improvement that dynamically assigns agents to handle specific bounding boxes.
Maximum Suppression (NMS), to enhance detection accuracy in various ITS scenarios. 3. Implementation of adaptive agent strategies / behaviors that allow the switch between cooperation and competition dynamically, ensuring robust performance in real-world ITS applications.
To the best of our knowledge, we are among the first to propose integrating MAS into specific computer vision sub-tasks such as bounding box filtering and fusion. This approach aims to exploit the advantages of MAS to enhance the accuracy, efficiency, and adaptability of object detection systems in ITS applications.
The agentified Weighted Boxes Fusion (WBF) system integrates multiple agents, each handling individual bounding boxes from various detection models. This Multiagent System (MAS) enhances the efficiency and accuracy of bounding box fusion through distributed processing and specialized agent roles. A central blackboard mechanism facilitates information sharing and coordination.
MAS offers decentralized decision-making and dynamic adaptability, enhancing resilience and flexibility in handling varied scenarios [30]. The blackboard acts as a global communication hub, simplifying data interactions and providing a robust framework for synchronized information exchange among agents [31]. Specific agent roles, from bounding box processing to model-specific adaptations, optimize performance and accuracy by leveraging domain-specific knowledge and algorithms [32]. Feedback mechanisms enable dynamic adaptation, allowing agents to adjust strategies based on performance and data input changes, maintaining high accuracy in dynamic environments [33].
The system includes various agents with specific responsibilities:
• Bounding-Box Agents: Handle individual bounding boxes, analyze, and propose fusions with overlapping boxes. • Model-specific Agents: Manage bounding boxes from specific detection models. Can be seen as interfaces between the MAS and CV models. Each agent extracts bounding box proposals from its respective model to ensure compatibility and apply model-specific behaviors and adjustments. • Coordinator Agents: Oversee the fusion process, resolve conflicts between bounding-box agents, and make final decisions on merged bounding boxes. • Data Processing Agents: Optionally handle image preprocessing and result postprocessing.
The blackboard serves as a shared information space for communication and data exchange:
• Data Repository: Central storage for bounding box data, including coordinates, confidence scores, and model origins. • Communication Medium: Allows agents to read and write data, maintaining system modularity and scalability. • Coordination Facilitator: Coordinates actions among agents, especially in resolving fusion conflicts.
The workflow involves:
Our implementation is developed in Python, utilizing the existing WBF codebase to maintain consistency in data processing. By forking the original WBF repository, we leverage developed libraries, utilities, and functions, ensuring the use of the exact same logic in data processing. This allowed us to focus on integrating MAS features without reinventing the core bounding box fusion logic. We built an ad-hoc MAS framework tailored to our requirements. The agents interact via a shared blackboard for communication, and the system supports both centralized and decentralized processing. Following the system architecture described in the previous section, one can implement diverse behaviors and a variety of solution method logics by only changing the decision logic of the bounding box agent and adjusting the coordination mechanism. Model-specific Agents interact with existing object detection models (e.g., YOLO, Faster R-CNN) to receive and process bounding boxes. Model-specific agents convert detection outputs into a standard format used by the system.
Main implementation challenges included managing computation time, communication overhead, and integrating the MAS with existing computer vision models. Future improvements will focus on developing variety of agent behaviors with optimized parameters for computational and accuracy performance, enhancing the system's scalability, robustness, and adaptability, exploring further integration with advanced machine learning models and real-world deployment scenarios.
We developed two distinct agent behaviors to demonstrate the versatility and potential of MAS in object detection. The first behavior replicates the Weighted Boxes Fusion (WBF) in a decentralized manner, while the second introduces a competitive interaction among agents.
This behavior replicates the state-of-the-art WBF method in a decentralized manner. Each agent processes bounding boxes independently and posts results to a shared blackboard (see Algorithm 1), improving system resilience. The agent determines overlapping boxes as candidates for fusion by calculating the Intersection over Union (IoU). Boxes are considered for fusion if their IoU exceeds a certain threshold. The IoU calculation is given by:
Algorithm 1 Decentralized WBF Algorithm (AWBF) -BoundingBox Agent behavior Fuse 𝐴 𝑖 and 𝐵 𝑗 using weighted average 14:
end if 15: end for 16: 𝐴 𝑖 posts result to the blackboard 17: Output: Final bounding boxes
In this behavior, agents compete based on a new metric that we introduce as Intersection over Box area(IoB). The IoBs for two boxes, 𝐴 and 𝐵, are calculated separately as:
Attacking or cooperating with other agents depending on calculated strengths. The strength of an attack of 𝐴 on 𝐵 and defense of 𝐵 against 𝐴 are defined by:
The decision rule is based on the difference between attack and defense strengths and a decision threshold
To illustrate the AWBF and competitive behavior in action, we consider the detection of bicycles in image "138639" from COCO dataset using two ad-hoc models. (see Figure 2) The bounding boxes from the two models are as follows:
• For competitive behavior, the attack and defence strengths rely on the calculation of the Intersection over Box values:
Given (𝑇 = 0.3), the agent 𝐵1 wins and 𝐵2 is removed as (Result > 𝑇 ). Increasing the 𝑇 value to 4, the conflict result will fall into the cooperation range and thus we return back to AWBF.
To evaluate our methods, we conducted extensive tests using the COCO dataset. Our primary objective was to demonstrate the proof of concept without optimizing parameters or model weights beyond the default settings provided by the WBF code. Therefore, our results focus on comparing performance metrics rather than optimizing for maximum accuracy.
The evaluation metrics used were those recommended and specified by COCO dataset. Namely, Average Precision and Average Recall:
-Average Precision (AP) Reveals the model's ability to make accurate positive predictions. It is calculated at different Intersection over Union (IoU) thresholds.
• AP@[IoU=0.50:0.95]: This is the average AP over ten IoU thresholds (0.50 to 0.95 with a step size of 0.05). • AP@0.50: This is the AP at an IoU threshold of 0.5.
• AP@0.75: This is the AP at an IoU threshold of 0.75.
• AP[small]: AP for small objects (area < 32 pixels).
• AP[medium]: AP for medium sized objects (32 ≤ area ≤ 96 pixels).
• AP[large]: AP for large objects ( area ≥ 96 pixels).
-Average Recall (AR) Measuring the sensitivity by focusing on the model's ability to correctly identify positive samples from the entire pool of positive instances.
• AR@[IoU=0.50:0.95]: This is the average recall over ten IoU thresholds (0.50 to 0.95 with a step size of 0.05). • AR@0.50: This is the average recall at an IoU threshold of 0.5.
• AR@0.75: This is the average recall at an IoU threshold of 0.75.
• AR[small]: AR for small objects.
• AR[medium]: AR for medium objects.
• AR[large]: AR for large objects.
The results from test runs over the entire dataset are shown in the Table 1. Notably, the results demonstrate that AWBF outperforms individual models whose outputs were used in the fusion process. Although our results did not surpass those of the centralized WBF, they were mostly comparable. Specifically, our approach performed better than WBF on AP-small and AR@10 at an IoU of 0.5.
Avg. Recall @[0.5 , 0.95] @[0.5] @[0.75] Small Medium Large @[0. Wen running experiments on subsets of the COCO dataset with different sizes, we observed that the centralized WBF method performs better with larger datasets but shows reduced efficiency on smaller datasets (see Figure 3). This can be explained by several factors:
• Law of Large Numbers: As the dataset size increases, the averaging process tends to smooth out random errors and fluctuations, leading to improved performance for the centralized WBF method. • Error Compensation: With more data points, errors in individual detections can compensate for each other, leading to more accurate fusion results. • Increased Data Redundancy: Larger datasets contain more redundant information, reinforcing correct detections and diluting the impact of incorrect ones.
The AWBF method exhibited more robust and stable performance across varying dataset sizes as Figure 3 shows, which can be attributed to the distributed processing and the redundancy: Each agent processes bounding boxes independently and in parallel, reducing the impact of individual errors and improving overall robustness. Also, Each agent's localized decision-making can lead to better performance, especially in smaller datasets where individual detections have a higher impact.
We also evaluated the competitive behavior using the default parameters. While the initial results did not match the quality of WBF, they demonstrated the potential for diverse agent behaviors. By adjusting the value of T, which controls the level of cooperativeness (1 -competitiveness). We conducted multiple tests on a subset of 500 COCO images, varying the competitiveness level. We observed an interesting trend, as the results showed that increasing competitiveness improved precision (see Figure 4). AP increased with higher competitiveness, likely because competition removed lower-scoring boxes, reducing false positives and improving precision. Recall remained stable as even with fewer boxes, sufficient accurate boxes were retained.
To summarize, theses evaluations demonstrated the flexibility and potential of integrating MAS into object detection workflows. While the competitive agent behavior requires further optimization, the initial results validate our approach and open avenues for more sophisticated multi-agent behaviors in future work.
In this work, we presented a proof-of-concept implementation integrating MAS into object detection workflows, specifically focusing on improving bounding box predictions, an essential component of autonomous vehicle perception systems. By leveraging the decentralized processing capabilities of MAS, we demonstrated two distinct agent behaviors: Decentralized (Agentified) Weighted Boxes Fusion and Competitive Interaction. Our experimental evaluation using the COCO dataset showed that while the decentralized WBF approach performed comparably to the centralized WBF, the competitive behavior illustrated the potential for further optimization and innovation in agent-based object detection systems. The results indicate that MAS can offer robust and adaptable solutions for object detection tasks, particularly in dynamic and complex environments like AV perception and intelligent transportation systems. Future work will focus on refining agent behaviors, enhancing system scalability, and integrating more advanced machine learning models to further improve performance and adaptability for AV applications.
This work is funded by the French National Research Agency as part of the MultiTrans project under reference ANR-21-CE23-0032.