Object detection is a pivotal task in computer vision, with applications spanning from autonomous driving to surveillance. Traditionally, methods like Non-Maximum Suppression (NMS) and its variants have been used to refine object detection outputs. Fusing predictions from diferent object detection models using confidence scores to average overlapping bounding boxes from multiple detection models has demonstrated superior performance over conventional methods. In this work, we employ multiple agents, each responsible for handling individual bounding boxes, to generate an improved fused prediction. This agent-based adaptation aims to leverage decentralized processing to potentially increase the system's eficiency and adaptability across various object detection scenarios, particularly in autonomous vehicle (AV) perception systems. We develop two distinct behaviors for the bounding box agents: one replicating the state-of-the-art Weighted Boxes Fusion (WBF) method in a decentralized manner, and the other introducing competitive behavior where agents interact based on Intersection over Union (IoU) and confidence values. We evaluate the performance of our approach using the COCO dataset, demonstrating the flexibility and potential of integrating MAS into object detection workflows including those for AV perception systems.
Autonomous vehicles and intelligent transport systems depend on advanced computer vision
technologies, with object detection being a critical task. This enables vehicles to recognize and respond
to surrounding objects efectively, with region proposal identifying potential object locations early in
the detection process, crucial for timely responses in autonomous driving [
The integration of Multiagent Systems (MAS) into object detection workflows ofers new perspectives
to address traditional challenges [
MAS can adjust strategies based on scenarios [
Agentifying output refinement methods such as NMS or WBF involves assigning individual agents to handle specific bounding boxes, enabling dynamic adjustment based on individual box characteristics. This approach addresses real-time processing requirements and improves scalability and fault tolerance by decentralizing the decision-making process [18, 19, 20]. In this work, we aim to design and implement a proof-of-concept system integrating MASs into the process of improving bounding boxes in object detection. We will develop two behaviors for the bounding box agents: one replicating the state-ofthe-art Weighted Boxes Fusion (WBF) method in a decentralized manner, and the other introducing competitive behavior where agents interact based on Intersection over Union (IoU) and confidence values. Finally, we will deploy the system and assess its performance using the COCO dataset, testing various levels of competition and cooperation between agents. The remainder of this paper is structured as follows: Section 2 presents the related work in object detection, multiagent systems, and their integration. Section 3 details the system architecture and design principles of the AWBF method. Section 4 describes the implementation of the proof-of-concept system and the development of agent behaviors. Section 5 discusses the experimental setup and evaluation using the COCO dataset. Section 6 presents the results and analysis of the experimental evaluation. Section 7 concludes the paper with a summary of findings and future work directions.
Object detection is a fundamental task in computer vision, critical for intelligent transportation systems (ITS) applications such as autonomous driving, trafic monitoring, and surveillance. The integration of MASs into object detection workflows ofers significant potential to enhance system eficiency, robustness, and adaptability. This section reviews recent advancements in object detection techniques relevant to the ITS, with a focus on bounding box fusion and the role of MASs.
Wang et al. (2019) presented the Multi-Stage Complementary Fusion (MCF3D) network, an end-toend architecture for 3D object detection that integrates LiDAR and RGB data. This network employs attention mechanisms and prior knowledge to achieve state-of-the-art results, enhancing the detection accuracy necessary for autonomous driving applications [21].
Qian et al. (2020) proposed an improved object detection method for remote sensing images, incorporating a novel bounding box regression loss and a multi-level features fusion module. This method enhances the precision of object localization, which is crucial for applications such as trafic monitoring and vehicle detection [22].
Solovyev et al. (2021) introduced the Weighted Boxes Fusion (WBF) method, which averages
overlapping bounding boxes from multiple detection models using confidence scores. This approach
demonstrated superior performance over traditional techniques, highlighting the efectiveness of fusion
methods in improving object detection accuracy [
Zhang and Wu (2022) proposed a multi-view feature adaptive fusion framework that enhances 3D object detection by optimizing depth feature fusion and loss function design. This approach improves the regression accuracy of bounding boxes, which is essential for ITS applications where precise object localization is critical [23].
Liu et al. (2023) developed the Fusion network by Box Matching (FBMNet) for multi-modal 3D detection. This method aligns features at the bounding box level, providing stability in challenging scenarios such as asynchronous sensors and misaligned sensor placements, common issues in ITS applications [24].
Introducing MAS to object detection and computer vision systems is not a new idea. For example Choksuriwong et al. (2005) developed a MAS for image understanding that localizes and recognizes objects using a distributed system implemented on a cluster computer. This approach leverages invariant features and supervised classification to improve object recognition accuracy, which is vital for trafic monitoring systems [25]. However, the application of MAS in these areas has decreased recently with the advancements in machine learning techniques and their improved performance in handling object detection tasks. Despite this shift, some researchers have continued to explore the potential of MAS in object detection through various approaches.
Jiang et al. (2019) proposed a multi-agent deep reinforcement learning (MADRL) approach for multi-object tracking, using YOLO V3 for object detection and Independent Q-Learners (IQL) for policy learning. This method achieves better performance in precision, accuracy, and robustness compared to other state-of-the-art methods, which is particularly beneficial for real-time trafic monitoring and surveillance [26].
Fekir and Benamrane (2015) introduced a MAS for boundary detection and object tracking using active contours and multi-resolution treatment. This system improves object boundary detection and tracking through cooperative agent strategies, enhancing the accuracy and eficiency of ITS applications such as vehicle and pedestrian tracking [27].
Vincent et al. (2022) described a MAS using stereovision for perception, enabling agents to collaborate and enhance scene understanding through graph matching algorithms. This approach addresses challenges in correspondence identification and non-covisibility, critical for ITS applications such as multi-vehicle coordination and trafic management [28].
Mahmoudi et al. (2013) utilized a MAS for object recognition in complex urban areas, leveraging WorldView-2 satellite imagery and digital surface models. This system improves object recognition accuracy through knowledge-based reasoning and cooperative agent capabilities, essential for urban trafic monitoring and smart city applications [29].
In light of the existing work, our proposal aims to integrate the strengths of both bounding box fusion techniques and MASs to develop a more robust and eficient object detection framework tailored for ITS applications. Our approach leverages the distributed processing capabilities of MASs to enhance the accuracy and scalability of bounding box fusion methods. By incorporating advanced fusion techniques and adaptive agent strategies, our system aims to address the limitations of existing methods, such as handling dynamic environments and improving detection precision. Our contributions include: 1. A multi-agent based framework for bounding box improvement that dynamically assigns agents to handle specific bounding boxes. 2. Integration of advanced fusion techniques, such as Weighted Box Fusion (WBF) and Non
Maximum Suppression (NMS), to enhance detection accuracy in various ITS scenarios. 3. Implementation of adaptive agent strategies / behaviors that allow the switch between cooperation and competition dynamically, ensuring robust performance in real-world ITS applications.
To the best of our knowledge, we are among the first to propose integrating MAS into specific computer vision sub-tasks such as bounding box filtering and fusion. This approach aims to exploit the advantages of MAS to enhance the accuracy, eficiency, and adaptability of object detection systems in ITS applications.
The agentified Weighted Boxes Fusion (WBF) system integrates multiple agents, each handling individual bounding boxes from various detection models. This Multiagent System (MAS) enhances the eficiency and accuracy of bounding box fusion through distributed processing and specialized agent roles. A central blackboard mechanism facilitates information sharing and coordination.
MAS ofers decentralized decision-making and dynamic adaptability, enhancing resilience and flexibility in handling varied scenarios [30].
Blackboard Model 1 Model 2 Model 3
Model1 specific agent Model2 specific agent Model3 specific agent
Coordinator Agent BB Agent1 (bus)
BB Agent 3 (car) BB Agent 2 (bus)
BB Agent 4 (people)
BB Agent 5 (people) DATA Processing Agent
BBB1B2
BB3
BB4 BB5
The blackboard acts as a global communication hub, simplifying data interactions and providing a robust framework for synchronized information exchange among agents [31]. Specific agent roles, from bounding box processing to model-specific adaptations, optimize performance and accuracy by leveraging domain-specific knowledge and algorithms [ 32]. Feedback mechanisms enable dynamic adaptation, allowing agents to adjust strategies based on performance and data input changes, maintaining high accuracy in dynamic environments [33].
The system includes various agents with specific responsibilities: • Bounding-Box Agents: Handle individual bounding boxes, analyze, and propose fusions with overlapping boxes. • Model-specific Agents: Manage bounding boxes from specific detection models. Can be seen as interfaces between the MAS and CV models. Each agent extracts bounding box proposals from its respective model to ensure compatibility and apply model-specific behaviors and adjustments. • Coordinator Agents: Oversee the fusion process, resolve conflicts between bounding-box agents, and make final decisions on merged bounding boxes.
• Data Processing Agents: Optionally handle image preprocessing and result postprocessing.
The blackboard serves as a shared information space for communication and data exchange: • Data Repository: Central storage for bounding box data, including coordinates, confidence scores, and model origins. • Communication Medium: Allows agents to read and write data, maintaining system modularity and scalability. • Coordination Facilitator: Coordinates actions among agents, especially in resolving fusion conflicts.
The workflow involves: 1. Data Input and Distribution: Model-specific Agents extract bounding boxes from diferent models and transfer them to Data processing agents who prepare and distribute bounding boxes to bounding box agents. 2. Bounding Box Analysis and Posting: Bounding box agents analyze and post findings to the blackboard, proposing fusions. 3. Review and Fusion: Coordinator agents review and finalize fusion decisions, consulting modelspecific agents as needed. 4. Final Processing and Output: Data processing agents optimize the fused bounding boxes for downstream applications. 5. Feedback and Adaptation: The system adapts to changes by updating agent strategies or parameters based on performance metrics.
Our implementation is developed in Python, utilizing the existing WBF codebase to maintain consistency in data processing. By forking the original WBF repository, we leverage developed libraries, utilities, and functions, ensuring the use of the exact same logic in data processing. This allowed us to focus on integrating MAS features without reinventing the core bounding box fusion logic. We built an ad-hoc MAS framework tailored to our requirements. The agents interact via a shared blackboard for communication, and the system supports both centralized and decentralized processing. Following the system architecture described in the previous section, one can implement diverse behaviors and a variety of solution method logics by only changing the decision logic of the bounding box agent and adjusting the coordination mechanism.
Model-specific Agents interact with existing object detection models (e.g., YOLO, Faster R-CNN) to receive and process bounding boxes. Model-specific agents convert detection outputs into a standard format used by the system.
Main implementation challenges included managing computation time, communication overhead, and integrating the MAS with existing computer vision models. Future improvements will focus on developing variety of agent behaviors with optimized parameters for computational and accuracy performance, enhancing the system’s scalability, robustness, and adaptability, exploring further integration with advanced machine learning models and real-world deployment scenarios.
We developed two distinct agent behaviors to demonstrate the versatility and potential of MAS in object detection. The first behavior replicates the Weighted Boxes Fusion (WBF) in a decentralized manner, while the second introduces a competitive interaction among agents.
This behavior replicates the state-of-the-art WBF method in a decentralized manner. Each agent processes bounding boxes independently and posts results to a shared blackboard (see Algorithm 1), improving system resilience. The agent determines overlapping boxes as candidates for fusion by calculating the Intersection over Union (IoU). Boxes are considered for fusion if their IoU exceeds a certain threshold. The IoU calculation is given by: (1, 2) = (1 ∩ 2) (1 ∪ 2) Algorithm 1 Decentralized WBF Algorithm (AWBF) - BoundingBox Agent behavior 1: Input: Bounding boxes , confidence scores , labels 2: Read , , from the blackboard 3: Determine overlapping boxes as candidates for fusion 4: Filter candidates using IoU metric 5: Apply WBF on the final set of candidates 6: Post fused boxes to the blackboard 7: Output: Fused bounding boxes Algorithm 2 Competitive Interaction Algorithm - BoundingBox Agent behavior
In this behavior, agents compete based on a new metric that we introduce as Intersection over Box area(IoB). The IoBs for two boxes, and , are calculated separately as: | = ( ∩ ) () , | = ( ∩ ) () Attacking or cooperating with other agents depending on calculated strengths. The strength of an attack of on and defense of against are defined by:
attack(, ) = confidence × |, defense(, ) = confidence × | The decision rule is based on the diference between attack and defense strengths and a decision threshold ( ): Result(, ) = attack(, ) − defense(, )
⎧⎪Result(, ) > : wins and is removed winner determination ⎨Result(, ) < − : wins and is removed
⎪⎩otherwise : and fuse using WBF The least case represent the area where agents can cooperate as their strengths are close. Threshold can determine the level of cooperativeness, and thus the value (1 − ) refers to the competitiveness level. ( = 1) indicates full cooperativeness settings, reverting to AWBF. Contrarily, ( = 0) indicates full competitiveness unless attack and defense strengths are equal.
To illustrate the AWBF and competitive behavior in action, we consider the detection of bicycles in image "138639" from COCO dataset using two ad-hoc models. (see Figure 2) (a) Model 1 Proposals (b) Model 2 Proposals (c) Combined Proposals (d) AWBF Result (e) Competitive Results The bounding boxes from the two models are as follows: • Model 1: {‘box’: [0.192, 0.752, 0.312, 0.873], ‘score’: 0.9, ‘label’: 2} • Model 2: {‘box’: [0.203, 0.756, 0.314, 0.875], ‘score’: 0.5, ‘label’: 2} The Intersection over Union for the bounding boxes is: IoU = aarreeaaooffouvneirolanp ≈ 0.85 If we apply the WBF method:
WBFbox = Given ( = 0.3), the agent 1 wins and 2 is removed as (Result > ). Increasing the value to 4, the conflict result will fall into the cooperation range and thus we return back to AWBF.
To evaluate our methods, we conducted extensive tests using the COCO dataset. Our primary objective was to demonstrate the proof of concept without optimizing parameters or model weights beyond the default settings provided by the WBF code. Therefore, our results focus on comparing performance metrics rather than optimizing for maximum accuracy.
The evaluation metrics used were those recommended and specified by COCO dataset. Namely, Average Precision and Average Recall: - Average Precision (AP) Reveals the model’s ability to make accurate positive predictions. It is calculated at diferent Intersection over Union (IoU) thresholds.
• AP@[IoU=0.50:0.95]: This is the average AP over ten IoU thresholds (0.50 to 0.95 with a step size of 0.05). • AP[small]: AP for small objects (area < 32 pixels). • AP[large]: AP for large objects ( area ≥ • AP[medium]: AP for medium sized objects (32 ≤ 96 pixels).
area ≤ 96 pixels). - Average Recall (AR)
Measuring the sensitivity by focusing on the model’s ability to correctly identify positive samples from the entire pool of positive instances.
• AR@[IoU=0.50:0.95]: This is the average recall over ten IoU thresholds (0.50 to 0.95 with a step size of 0.05). • AR@0.50: This is the average recall at an IoU threshold of 0.5. • AR@0.75: This is the average recall at an IoU threshold of 0.75. • AR[small]: AR for small objects. • AR[medium]: AR for medium objects.
• AR[large]: AR for large objects.
The results from test runs over the entire dataset are shown in the Table 1. Notably, the results demonstrate that AWBF outperforms individual models whose outputs were used in the fusion process. Although our results did not surpass those of the centralized WBF, they were mostly comparable.
Specifically, our approach performed better than WBF on AP-small and AR@10 at an IoU of 0.5.
Medium
Small
Medium Benchmarking on COCO dataset
Wen running experiments on subsets of the COCO dataset with diferent sizes, we observed that the centralized WBF method performs better with larger datasets but shows reduced eficiency on smaller datasets (see Figure 3). This can be explained by several factors: • Law of Large Numbers: As the dataset size increases, the averaging process tends to smooth out random errors and fluctuations, leading to improved performance for the centralized WBF method. • Error Compensation: With more data points, errors in individual detections can compensate for each other, leading to more accurate fusion results. • Increased Data Redundancy: Larger datasets contain more redundant information, reinforcing correct detections and diluting the impact of incorrect ones.
AVG PRECISION @ IOU [ 0.5 , 0,95]
ALL AREA SIZES
AVG PRECISION @ IOU = 0.5
ALL AREA SIZES
AVG PRECISION @ IOU = 0.75
ALL AREA SIZES 0 100 200 NB3E0X0AMPLES 400 500 600 0 100 200 NB EXAMPLES 400 500 600 300 0 100 200 NB EXAMPLES 400 500 600
300 400 500 600 0 100 200 NB E3X0A0MPLES 400 500 600 AVG PRECISION @ IOU [ 0.5 , 0,95]
LARGE BOXES AVG RECALL @ IOU = 0.5
ALL AREA SIZES
AVG RECALL @ IOU =0,75
ALL AREA SIZES 0 100 200 NB3E0X0AMPLES 400 500 600 0 100 200 NB EXAMPLES 400 500 600
300 AVG RECALL @ IOU [ 0.5 , 0,95]
MEDIUM AREA BOXES
AVG RECALL @ IOU [ 0.5 , 0,95]
LARGE BOXES 1 1 0,8 0,8 0,6 0,6 0,4 0,4 0,2 0,2 0 0 100 200 NB3E0X0AMPLES 400 500 600 0 0 100 200 300
NB EXAMPLES AVG PRECISION @ IOU [ 0.5 , 0,95] AVG PRECISION @ IOU [ 0.5 , 0,95]
SMALL BOXES MEDIUM AREA BOXES 1 0,8 0,6 0,4 0,2 0 1 0,8 0,6 0,4 0,2 0
AVG RECALL @ IOU [ 0.5 , 0,95]
ALL AREA SIZES 1 0,8 0,6 0,4 0,2 0 0 100 200 300 400 500 600
AVG RECALL @ IOU [ 0.5 , 0,95]
SMALL BOXES 1 0,8 0,6 0,4 0,2 0 1 0,8 0,6 0,4 0,2 0 1 0,8 0,6 0,4 0,2 0 1 0,8 0,6 0,4 0,2 0 1 0,8 0,6 0,4 0,2 0 1 0,8 0,6 0,4 0,2 0 1 0,8 0,6 0,4 0,2 0 0 100 200 NB EXAMPLES 400 500 600 300 0 100 200 NB EXAMPLES 400 500 600 300 0 100 200 NB EXAMPLES 400 500 600
300 AWBF
WBF
The AWBF method exhibited more robust and stable performance across varying dataset sizes as Figure 3 shows, which can be attributed to the distributed processing and the redundancy: Each agent processes bounding boxes independently and in parallel, reducing the impact of individual errors and improving overall robustness. Also, Each agent’s localized decision-making can lead to better performance, especially in smaller datasets where individual detections have a higher impact.
We also evaluated the competitive behavior using the default parameters. While the initial results did not match the quality of WBF, they demonstrated the potential for diverse agent behaviors. By adjusting the value of T, which controls the level of cooperativeness (1 - competitiveness). We conducted multiple tests on a subset of 500 COCO images, varying the competitiveness level. We observed an interesting trend, as the results showed that increasing competitiveness improved precision (see Figure 4).
AP increased with higher competitiveness, likely because competition removed lower-scoring boxes, reducing false positives and improving precision. Recall remained stable as even with fewer boxes, suficient accurate boxes were retained.
To summarize, theses evaluations demonstrated the flexibility and potential of integrating MAS into object detection workflows. While the competitive agent behavior requires further optimization, the initial results validate our approach and open avenues for more sophisticated multi-agent behaviors in future work.
In this work, we presented a proof-of-concept implementation integrating MAS into object detection workflows, specifically focusing on improving bounding box predictions, an essential component of autonomous vehicle perception systems. By leveraging the decentralized processing capabilities of MAS, we demonstrated two distinct agent behaviors: Decentralized (Agentified) Weighted Boxes Fusion and Competitive Interaction. Our experimental evaluation using the COCO dataset showed that while the decentralized WBF approach performed comparably to the centralized WBF, the competitive behavior illustrated the potential for further optimization and innovation in agent-based object detection systems. The results indicate that MAS can ofer robust and adaptable solutions for object detection tasks, particularly in dynamic and complex environments like AV perception and intelligent transportation systems. Future work will focus on refining agent behaviors, enhancing system scalability, and integrating more advanced machine learning models to further improve performance and adaptability for AV applications.
This work is funded by the French National Research Agency as part of the MultiTrans project under reference ANR-21-CE23-0032.