Incident commanders of fire brigades still need to invest significant manual and mental efort to assess dynamic ifre scenes or accidents, coordinate personnel, make critical decisions, and manage risks under extreme conditions. We propose Far, a concept for an interactive firefighting assistant robot that leverages advanced sensing and real-time data analysis to support incident commanders. Besides the overall concept, we outline key use cases such as hazard detection, personnel tracking, automated decision support, and adaptive team coordination with the goal of identifying challenges in efective human-robot interaction. With this concept paper, we want to spark a discussion on firefighter-centric robotic assistance, emphasizing usability, trust, and seamless integration to improve emergency response.
This chapter gives an overview of related work in the field of assistants and robots for fire departments and in dangerous contexts. Sensors and related tasks, such as search and rescue, will also be discussed.
CEUR Workshop
ISSN1613-0073
There are numerous areas where robots are already being used to support firefighters. Bogu[e1] provides an overview of robots currently available on the market: mainly extinguishing robots for various applications, but also special robots equipped with chainsaws.
Dhiman et al.[
In the field of human search and rescue (SAR) are a large number of diferent robots for a wide range
of requirements [
In addition to these forms of robots, there are also exploration robots that are primarily based on
swarm intelligence [
Artificial intelligence can be used in many areas of fire protection. Hodges et al. [
Abdelrahman[
In addition to robots, there are many other IT systems designed to support the work of firefighters.
Scholz et al. [19], for example, have investigated communication during firefighting operations and developed concepts on how this can be improved. One broad field is flying firefighting drones, which currently operate mainly under human control. Boonyard et [a2l0.] have looked at the user needs for these systems. Another area is the supervision of firefighters. The focus here is on ensuring physical integrity and not on evaluating work results. Parker et a[2l.1] have developed portable devices to measure this in rural areas.
In our own research group, we developed and maintain Apps that support firefighters during incidents. The app comprises diferent features such as displaying a crowd sourced map of water collection points (hydrants, lakes, ...), looking up car rescue data sheets based on a car’s license plate, hazmat data sheets, and monitoring firefighters when wearing air breathing masks. The Apps are actively deployed for all ifre brigades in the German states of Saxony, Thuringia, and soon Saxony-Anhalt.
There is a wide range of diferent robots and systems that could theoretically support firefighters in their daily work. These include, in particular, extinguishing robots for a wide variety of operations. So far, many of these devices are rather single-purpoFsea.r instead aims at fusing the data of various systems with the goal to assist the incident commander by providing a better overview of the scene and by improving the decision-making process.
While the previously mentioned advances in robotics and drone technology mostly happened in research so far, such technology is expected to also arrive in actual fire departments in the near future. If we assume that fire departments will continue to equip themselves with state-of-the-art technology in the future, we can expect that multiple departments will bring a variety of automated systems from diferent manufacturers to an operation. This raises the question of how to extract the greatest possible benefit from these costly systems. How can a symbiosis of heterogeneous systems unfold its potential in a hectic emergency situation?
A key challenge is ensuring that automation supports, rather than overwhelms human firefighters. We assume that human expertise remains essential, as the task performance (human vs. automation/AI vs. collaboration) may difer from task to task [22]. For instance, automated systems may excel in tasks like thermal imaging and hazardous material detection, while human judgment is crucial for complex decision-making.
We propose a hybrid firefighting team where firefighters and machines collaborate eficiently. In the course of collaboration, it must be ensured that humans always remain in control and are not overwhelmed by the operation of the systems. To achieve this, we envision a superior system called ifrefighting assistant robot ( Far) that aggregates and processes digital information from any source, such as drone video feeds, sensor data or radio communication. The goal is to create an extended situation awareness and assist the incident commander to maximize operational eficiency and safety. AI-driven analyses will interpret available data in real-time, enabling intelligent decision support without burdening firefighters with system management.
While Far is currently imagined as ahumanoid robot located near the commander, we also acknowledge the potential of alternative implementations, such awsearable systems (e.g., AR glasses) or as asmart virtual agent, which may prove more practical and acceptable. Future work will compare these formats in terms of usability and efectiveness.
Far is envisioned to support in the areas that an incident commander commonly has to perform: • Real Time Situation Detection: What is the current situation about, which are the most urgent tasks? Supported through a multitude of sensor devices, robots, and droneFsa,r is able to create a real-time digital twin of the incident site. • Adaptive Risk Estimation: Where is the highest risk potential? What could go wrong? • Tactical Operation Strategies: Which actions should be performed to improve the current situation? • Assistance in Performing the Tasks: Support the incident commander or firemen in carrying out certain tasks in the physical world.
Common practise is to follow the sense-plan-act paradigm, in which sensors (sense) provide the data, a planning module (plan) determines the next action and finally an actuation module (act) carries out the physical action. The following is an overview of how the sense-plan-act paradigm can be equipped with useful components in the field. 3.2.1. Sensing Possible sensors include microphones for monitoring the commander’s environment and direct conversations, digital radio receivers to monitor remote communication, as well as various camera systems, including those with optical zoom, 360° capture, and thermal imaging capabilities. LiDAR-based sensors (Light Detection and Ranging) precisely capture distances and spatial structures. Hyperspectral cameras can capture light in many narrow wavelength ranges to enable detailed material analysis. UV cameras in the ultraviolet spectrum ofer additional possibilities for surface analysis.
These sensors can be permanently mounted in the fire truck or be available as mobile variants on robots or drones. Multiple mobile sensors can be placed at strategically advantageous locations, similar to how multiple drones can simultaneously provide video streams. Additionally, all firefighters could be equipped with portable cameras on their uniforms or helmets.
All incoming information from the sensing devices is bundled to create a digital twin of the incident scene. A multimodal sensor fusion AI could take over the analysis and evaluation of this digital twin. Based on the trained models, for example by observing similar operations, the operational situation can be evaluated and a plan or forecast of the next actions can be determined.
The execution of tasks within the physical domain remains the purview of firefighters. However, there is a growing trend of automation, with firefighting robots and drones becoming increasingly prevalent. These robots and drones could facilitate tasks such as material transportation and reconnaissance of hazardous environments that are currently inaccessible to humans. For instance, a firefighting drone could assist by executing the initial attack on small, emerging fires in high-rise areas. In addition, the majority of actuators are equipped with sensors that extend the sensing part dynamically.
At the scene of an incident, units from various fire departments may come together. Each unit is equipped diferently – some specialize in specific types of accidents, while others are generally equipped for firefighting and technical assistance. We assume that these units are equipped with state-of-the-art technology, which means that a wide range of initially undefined technologies converge at the scene. Depending on the type of incident and location, diferent units are deployed – and they may arrive at diferent times: units may arrive one after another, and additional units may be requested. The systems brought to the scene therefore do not follow a uniform standard but instead have a dynamic character.
With the Sense-Plan-Act architecture, it is possible to ensure that theFar operates efectively even under these conditions. The technologies of the arriving units dynamically expand the sensing capabilities.
The firefighter acts based on his abilities, knowledge, experience, values, and courage, while the system operates according to its learned AI-based model. This inevitably leads to conflicts. Even if the system’s assessment results in better situational awareness than that of the human, there is still a risk of conflict. The human may not understand why the system arrives at certain conclusions. Such conflicts lead to reduced trust, negative emotions, a sense of helplessness, and a lack of acceptance. In the following, we present a concept for mitigating conflicts by explaining our user interface.
To put the ideas into practice, we consider the following example scenario: A highway accident involving a truck that carries hazardous materials. The incident commander of the fire department is the first to arrive at the scene, accompanied byFar. The situation is hectic and confusing, which is why the incident commander decides to be supported by theFar system upon arrival as much as possible.
The user interface of the system accompanies the user in the same order as the core functions of the system are defined. Since these build on each other, a wrong situation monitoring would also lead to wrong subsequent assessments such as risk estimation.
Suggestions of recognized sub-incidents are displayed on a screen in a priority-sorted list. This list can change dynamically during the operation. Around the monitor are few, but physically well-perceivable hardware buttons, which can be easily and clearly operated even with firefighter gloves. Available buttons include a joystick for navigating the list and marking an entry, a confirmation button for accepting the case, a rejection button for discarding the case, and a restart button.
Instead of using a screen with hardware buttons, a more compact variant in the sense of a smart virtual agent with gesture control or similar concepts would also be conceivable. It is recommended that further work be initiated to discuss these options and compare them with the hardware variant in ifrefighting operations.
Finally, the simple but particularly important function of the restart button should be outlined: When pressing the restart button, all previously captured input data from the sensors is discarded. This means that conversations or radio messages before pressing the button are not incorporated into the AI model of multi-modal sensor fusion. This function is particularly important in exercises where various operational scenarios are played out, or when the system fails. The interface concept has been modeled on recommendation systems in aviation that alert pilots in high-pressure situations and provide suggestions for action.
Similar to situation monitoring, there is a list of hazards sorted according to their potential probability of occurrence and severity of impact. Hazards that the commander does not consider relevant in the current operational situation can be manually deselected and thus temporarily deactivated. A hazard can be selected to obtain background or real-time information on this hazard. In the case of contextrelated hazards, there is an explanation as to why the model classifies them as a hazard. An example would be: “Hazard: further accidents due to moving trafic, as there have been 2 rear-end collisions in comparable accidents on this section of the highway in the last 3 months.” In the case of sensor-based risks, visualization of the sensors detecting the risk is conceivable. An example of this would be:D“anger: Possible leakage of hazardous substances at this location of the vehicle.” The visualization shows a video stream based on the evaluation of the zoom camera and hyperspectral camera at the expected point of danger.
Based on the previously identified situations and risks, targeted solution strategies can be provided to the user, supporting the eficient management of the current situation.
When using this feature, it is assumed that the user seeks inspiration and wishes to explore the approach proposed byFar for the current situation. In this context, the user is expected to allocate more time for reflection and adjustment compared to other core functions. This additional time is necessary because an AI-proposed operational strategy may initially diverge from the conventional perspective of the incident commander and thus necessitate a discussion. For example, while the incident commander might favor the strategy c“ontain leaking hazardous material with a mobile collection basin” the AI might suggest an alternative such ass“ealing the damaged area of the tank.” In such a case, the commander could object: “We do not have the materials to seal this area.” The system might then respond: “The RLF 2000 vehicle is stocked with wooden wedges for sealing. My analysis shows that a wedge with a 4 cm diameter would be best suited.”
This example clearly demonstrates the added value of a voice-based interaction.
It is essential to evaluate how acceptable and efective it is for the system to mimic human conflict behavior and act as an interactive partner — through the use of persuasion techniques, assertiveness, or negotiation skills to establish a cooperative dialogue with the user.
This approach also raises important questions regarding social-psychological influencing factors and the distribution of roles within the hybrid team, particularly concerning who takes the lead in each situation. What remains crucial is that the system does not make autonomous decisions but serves as an advisory assistant — comparable to an experienced firefighter who, after decades of service, contributes his extensive knowledge in an advisory capacity and is open to discussion, but not in an authoritarian manner.
From the solution strategy, individual sub tasks might be derivable that can be transferred to autonomous robots, such as blocking of a highway in the described sample scenario. A separate task management view would be useful for such tasks.
As we are at the very beginning of conceptualizing and developing ouFrar system, we are still in the process of reviewing related work and projects, assessing user needs and developing the overall goals of our envisioned system. Having collected experiences with our existing Apps for firefighters, we have established a network with local fire academies and the responsible authorities that command and equip fire brigades. As next steps, our aim is to create a representative selection of sample scenarios where ourFar system is expected to support incident commanders the most. In collaboration with the diferent fire academies, we then want to deep dive into selected incident scenarios with the goal of understanding the particular needs in these scenarios, identifying the potential for improvements using Far, and designing and evaluating a first Far prototype.
Our proposed concept aims to enable human-AI collaboration between fire fighters/incident commanders and an assisting robot at the incident site. By integrating and fusing the data of a diverse set of sensors and automated systems, we expect to create a digital twin of the incident scene which then serves as a basis for AI-based tactical recommendations to the commander to improve the overall performance and response at the incident scene.
During the preparation of this work, the authors used ChatGPT and DeepL for text translations, grammar and spelling checks, drafting content (initial versions of individual paragraphs), paraphrasing and rewording. After using these tools/services, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. 2019.8666471. [19] M. Scholz, D. Gordon, L. Ramirez, S. Sigg, T. Dyrks, M. Beigl, A concept for support of fireifghter frontline communication, Future Internet 5 (2013) 113–127. URL: https://www.mdpi.com/ 1999-5903/5/2/113. doi:10.3390/fi5020113. [20] C. Boonyard, C. Joufrais, J. R. Cauchard, A. Brock, Firefighting with Drone Assistance: User Needs and Design Considerations for Thailand, in: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, ACM, Yokohama, Japan, 2025. URLh:ttps://enac.hal. science/hal-04963852. doi:10.1145/3706598.3714172. [21] R. Parker, A. Vitalis, R. Walker, D. Riley, H. G. Pearce, Measuring wildland fire fighter performance with wearable technology, Applied Ergonomics 59 (2017) 34–44. URL:https://www.sciencedirect. com/science/article/pii/S0003687016301715. doi:https://doi.org/10.1016/j.apergo.2016.08. 018. [22] M. Vaccaro, A. Almaatouq, T. Malone, When combinations of humans and AI are useful: A systematic review and meta-analysis, Nature Human Behaviour 8 (2024) 2293–2303. d1o0i:.1038/ s41562-024-02024-1.