The water environment in which unmanned surface vehicles (USVs) navigate presents many unique challenges. One of these is the risk of encountering obstacles that are (partially) submerged and therefore poorly visible. Therefore, their extent cannot be determined directly from available above-water sensor data. On the other hand, it is well known that human skippers are able to safely navigate boats around obstacles even without underwater sensors and only with the help of their expertise. In this paper, we describe initial work on extending the USV obstacle detection to include such functionality using a compositional model. To learn to hallucinate the extent of obstacles with a minimum of learning efort, we exploit the nature of obstacles (people in kayaks, canoes, and on paddleboards) that are visible most of the time, but not always. We evaluate the impact of such hallucinations on USV safety and maneuverability, and suggest additional cases where such hallucinations can be used to improve USV safety.
Unmanned surface vehicles (USVs) are increasingly recognized as a valuable tool for a variety of applications, including military, environmental, and commercial purposes. These autonomous craft are capable of operating in dificult or hazardous environments, making them ideal for tasks that would be too risky for humans.
On the other hand, one of the envisioned benefits of
USVs is the ability to gather data and perform tasks for
extended periods of time without the need for human
intervention. This would allow them to cover large areas
and collect a large amount of data that can then be used
for a variety of purposes. USVs equipped with sensors
and cameras could be used, for example, to monitor and
map the marine environment, track wildlife [
This paper is organized as follows: Following related
work, we define the problem we want to use to
demonstrate the capabilities of our method. We then introduce
the basic concepts of compositional models and describe
our use case and evaluation method. In the experimental
part, we present our own dataset used in our experiments
and its properties, followed by the evaluation setup
focusing on the USV navigation. Finally, we discuss the results
and further applications of the presented approach.
2. Related work
branches of obstacle detection have been improved. On
the one hand, researchers have adapted or retrained
general object detectors for marine environments [
The other branch of obstacle detection is semantic
segmentation. Several methods have adapted general
segmentation methods to the marine environment [
The method presented in this paper operates at a
higher level of reasoning and aims to use assumptions
that reasonably hold in water-bound environments. It
relies on existing but imperfect methods for obstacle
detection (in this paper we use Yolov7 [
The computer vision aspect of marine environment
interpretation has been approached in several ways so
far: Some authors have acquired datasets to facilitate
domain transfer for Deep Learning and further investigate
the specific problems in the maritime domain [
Several USV architectures with diferent sensors have
been presented to solve problems such as poor lighting • A method for improving the safety of USV and
conditions and the need for absolute distance measure- its environment by improving the estimation of
ments [
Han et al. [
Several datasets have also been published, some of floating objects in situations where a person was detected
which will be used as learning data for Deep Learning- above the water surface, but the corresponding boat was not
based methods and others as benchmarks for existing detected. Such cases often occur when boats are of a
simmethods. One such dataset, SMD, was proposed by ilar color to the surrounding water, partially submerged
Prasad et al. [
In the past, obstacle detection was performed directly for additional training.
by estimating salient regions [
This problem has the following interesting properties: • Solid physical foundation. People cannot walk or sit on the water. There must be some kind of highly buoyant device present to support their weight. • No opportunity to introduce gross errors with false detections. False positives only restrict the possibilities for the USV to advance, and our experiments were designed to check for that efect. • No manual annotations are needed, since we can obtain ground truth using the object detector (Yolov7) and therefore obtain plenty of data to train the higher-reasoning model.
In computer vision, a composition refers to the arrangement of visual elements in an image. These visual elements are called parts and can be low level primitives (e.g. edges, corners) or high-level objects themselves (e.g. cap, a label and recognizable shaped bottom on a bottle of soft drink), as shown in Fig. 2. Parts can be compositions themselves, yielding a hierarchical compositional model.
The compositional model, as shown in Fig. 2 is not particularly useful, as it is completely rigid. In practice, geometrical parameters of the parts are modelled as random vectors. In Figure 3 we show a hierarchical, compositional model of a 3-part Coke bottle under the assumption that the probability distribution of -th part position ( , ) relative to the center (origin) of the -th composition is Gaussian: 1–9 Our method is heavily influenced by the work of Koporec et al. [22], that uses hierarchical compositional models to detect objects’ visible parts even when large parts of objects are occluded, and allows collection of ex- and covariance matrix Σ . The parameters of the pert knowledge from a small number of targeted human Gaussian distribution are obtained by learning on a sufannotations. In our work we use a highly simplified im- ifciently large set of train data, from which vectors X plementation of Human-Centered Deep Compositional are extracted. (HCDC) model [22].
ways: • 1. Robust, explainable detection of partially occluded objects, where the object (composition) is detected even if not all its parts are visible. • 2. Explanation (hallucination) of the missing part.
This is the functionality we use in the presented work.
where subscripts and denote left-top or right4.2. Model of a person on a boat bottom point of the boat bounding box, respectively, and denotes the scale index. Therefore, a total parameter Human-Centered Deep Compositional (HCDC) set of our 2D model consists of 2 Gaussian means and model [22] operates on parts that are itself deep 2 2D Gaussian covariance matrices. detections (detections, obtained by convolutional neural network models, CNNs). This makes the model 4.3. Training the compositional model explainable, as the parts are already categorized into human-understandable categories. Our training does not require any manual annotations.
We follow this example and use the detections Due to pretty good (but not perfect) performance of the provided by an obstacle detector pretrained on MS chosen detector (Yolov7 detects about 95% boats and even COCO [23]. We only retained the pertinent detection higher percentage of persons) we use those cases where classes: person, boat and surfboard. Additionally, we both the boat and the person on it were detected, to treated the classes boat and surfboard as the same se- establish a model that can reasonably predict the position mantic entity (referred to as boat in the remainder of the and size of a boat in absence of detections. text), since both of those classes almost always appear Although we assume Gaussian model for probability simultaneously with the class person. The compositional distributions and , we estimate each separate model that we use is shown in Fig. 4. distribution using expectation maximization (EM) algorithm with 2-component Gaussian Mixture Model (GMM) and retain the larger of the two components as either Person usingo2r-comp.oOneunrtpGreMliMmirneasruylttseisntimngorheaascrceuvreaatelefitdtitnhgat of Gaussian model to the data, collecting the outliers in the significantly smaller component. d =
︀] X = [︀ X = [︀
︀] X ∼ ( , Σ) X ∼ ( , Σ) 1–9 (2) ΣL L R Boat
ΣR
separate Gaussian models for upper-left and bottom-right corners of the boat bounding box, and that for each of scales.
To hallucinate the most likely bounding box of the (undetected) boat, we examine the bounding box of the detected person, calculate its centroid and diagonal , calculate the scale index and look up the relevant Gaussian models and obtained during the training. The hallucinated bounding box points of a boat are determined at displacements and at which and have their maximum values. Note that and are relative to the person’s centroid point.
To compare performance of object detectors, a generic
approach by counting false positives and false negatives,
with respect to some minimum intersection over union
(IoU) value is often used. However, when evaluating
the detectors in with actual application in mind, it is
often the case, that not all errors are equally important
or relevant. For example, USV benchmark [
Note that a crucial safety issue here is that the USV by projected bounding boxes. may navigate in the areas that actually contain part of the boat. Fig. 5 shows the situation with multiple detections This evaluation protocol does not assume or require and corresponding hallucinations. The aim of the USV is complex obstacle avoidance maneuvers, and is not sensito proceed in the forward direction, but it has to avoid ob- tive to vertical displacement of bounding boxes. stacles. Therefore, it can proceed only through navigable channels, marked with arrows in Fig. 5. To ensure safety, 6. Experiments navigable channels cannot contain any part of the boat at any distance, and the problem can be compressed to one-dimensional representation along the horizontal () axis. However, if the hallucinations are too wide, there may not be any navigable channel left in front of the boat.
Therefore, we define the following two metrics:
We recorded several hours of video on the Ljubljanica river (sessions denoted LJU1, LJU2, and LJU3) in diferent weather conditions, on Lake Bled (denoted BLE1), and on the Adriatic Sea (near the coast, in several areas between Koper and Portorož), denoted ADR1. In each case, we hired human workers who served as obstacles • One-dimensional IoU value (referred to as IoU- in boats, kayaks, canoes and on paddleboards. The data 1D), calculated from the projections of actual contains about 10 obstacles in the near vicinity of the (ground truth) bounding boxes and hallucinated recording boat, captured in diferent configurations and bounding boxes, both projected downwards onto from diferent angles relative to the position of the sun (so the horizontal axis (evaluation line in Fig. 5). This challenging backlit scenes were also captured). Videos value should be as high as possible. were recorded at 10 frames per second using Stereolabs • One-dimensional coverage (referred to as Cov- ZED 3D stereo camera1, mounted between 1-1.5 meters 1D) of the horizontal axis (evaluation line) with above the water surface (diferent watercraft were used projection of both ground truth bounding boxes at diferent locations). In this experiment we only use the and hallucinated bounding boxes. If the coverage of hallucinations becomes too high, then USV 1https://www.stereolabs.com left RGB images, the right RGB image and depth were not used in any way.
The training data was constructed by first obtaining predictions for all the relevant classes using Yolov7. The compositions were then constructed from cases where there was overlap between detections of class person and either of the classes boat or surfboard.
Analysis of the detections provide some insight into the problem of ”invisible” boats and paddle boards, as shown in Table 1.
Session (dataset) person only (%) person+boat (%)
LJU1 0.04 0.96
LJU2 0.05 0.95
LJU3 0.05 0.95
BLE1 0.03 0.97
ADR1 0.05 0.95
We decided to use session BLE1 for training of the Gaussian distributions and , as it featured boats of varying shapes and sizes. The training time using precalculated Yolov7 detections was negligible.
Free from requirements for manual annotation, we were able to run the evaluation of our method on all images from our dataset, For evaluation, we used only the detections of people with corresponding boats. Boat detections, obtained via Yolov7, were considered ground truth, against which the hallucinations, obtained using our compositional model, were tested. Person detections without corresponding boats were not used, as these had no usable ground truth. Table 2 shows the results.
Analysing the results, we can see that there is good overlap between ground truth detections and hallucinations, with IoU-1D ranging from 0.465 to 0.605 for the same dataset on which the model was trained. Note that IoU-1D of 0.5 means that the middle half of bounding box projections overlap, while the 1/4 at each edge is non-overlapping.
Coverage of hallucinations is not as high as coverage of detections, and, most surprisingly, coverage of pure person detections (e.g. in absence of any detected boats) is not much lower than the coverage of hallucinations. We examined the reason behind this and found that the increase is not as high as expected due to obstacles which are further away and have disproportionately wide person detection bounding boxes, and due to diferences in the set of boats used for training and testing (note the highest increase in Cov-1D from person detection to hallucination when the training set BLE1 was tested). Figure 6 shows an image where the result of our method is poor.
This paper presents a preliminary research on use of hallucinations, provided by compositional models, in water-borne obstacle detection and avoidance. The experimental design in this work has been subject to many constraints, most notably the absence of proper ground truth annotations. These issues will be addressed in further work, towards a general framework to hallucinate obstacles that are not directly observed by the sensors.
Since using an obstacle detector precludes us from detecting unknown objects, combining their results with either semantic segmentation or another method of anomaly detection or a diferent sensor modality (such as LIDAR) might help in producing a more general hazard detection system that will perform hazard detection from multimodal cues.
A state of the art in experimental autonomous road vehicles relies heavily on multimodal sensor setup, with sensors like LIDAR and RADAR [24, 25], which bear no resemblance to human sensing. Therefore, an argument could be made that instead of hallucinating the obstacles and trying to emulate the skipper, one could detect the hidden obstacles using proper underwater sensor setup. Session (dataset) IoU-1D Ground truth Cov-1D Hallucination Cov-1D Person detection Cov-1D
LJU1 0.465 0.13 0.074 0.067
In practice, this results in fragile setup due to water tur- leaves in the fall), so avoiding it at all times is not an opbidity – USVs are expected to navigate safely even in tion. However, debris may accumulate in shallow water water that is dirty or muddy. areas (it may not be debris, but aquatic plants sticking
Note also that a paddleboard, as shown in Fig. 1, is a out of the shallow water). So, if we encounter debris farvery thin object at the boundary between air and water, ther from shore, it is not a cause for concern as it is most which is not comparable to the situations encountered in likely floating. However, if it is found near land features autonomous driving (on the road), so it is unlikely that (e.g., trees, mud), then it usually means that the area is additional (underwater) sensors will reliably detect it. In dangerous, shallow, and not navigable. To detect this fact, some watercraft may be completely submerged at case, we might model the shallow, non-navigable area as times, as can be seen in Fig. 7 which shows a fast-moving a composition of debris and other land-based features. athlete in a kayak. As it can be seen in top right image in Fig. 8, it is sometimes dificult to determine whether the situation is a hazard or not. The labeling of such situations cannot be done by (untrained) labelers, but must be defined by experienced skippers working in cooperation with computer vision engineers. These compositions and their parameters must be defined by hand for a small number of available cases. The HCDC approach [22] has shown that this is indeed possible for common, well known food items. In this case, it will be used to insert concentrated expert knowledge into the compositional hazard detection model.
Missing detections of boats and paddleboards are immediately available in our waterborne datasets. However, there are other scenarios where such an approach would be useful, but for which there is currently insuficient data to train the models. The main reason for this is that these scenarios are to some extent hazardous to the USV and represent edge cases in USV deployment. In Figure 8, we present a common scenario that we have encountered several times, but for which we currently do not have enough data to properly test, let alone train. Plant debris is common in continental waters and usually safe to traverse. Often it covers the entire navigable area (e.g.,