Object multi-tracking has been a relevant topic for diferent applications, such as surveillance, mobility, and ambient intelligence. It is particularly challenging when considering open spaces, like Smart Cities, which demand multi-camera solutions with issues like re-identification. In this paper, we describe a framework aiming to provide multi-tracking of people throughout a university campus as part of a larger project (Lab4USpaces) to develop a Smart Campus initiative. Several object detection models and real-time tracking open-source algorithms were compared. The project contemplates a set of low-cost video cameras covering most of the campus, with or without overlapping. After researching diferent alternatives, the proposed framework uses the YOLOv7 tiny model for object detection, BoT-Sort for multiple object tracking, and Deep Person Reid for re-identification. We also faced challenges concerning the privacy and security of campus users. The multi-tracking system complies with current regulations since no personal identification is ever performed, and no images are stored for longer than necessary for object detection and re-identification. Besides describing the first prototype, this paper discusses some validation tests and describes some potential uses.
In the context of an undergoing research project named Lab4U&Spaces1 – aiming at exploring
innovative technologies to raise the quality of life at the university campus – the work described
here is focused on the campus’ users management and mobility dimension. Using this platform,
students can, for example, avoid a place with a more extensive flow of users when scheduling a
joint activity. On the other hand, campus managers can quickly locate areas of more significant
influx, properly understand this dynamic, and prepare more appropriate responses to avoid it, if
recommended. The need to prevent excessive exposure to UV rays due to carelessness by users
or even reduce contact to prevent viral dissemination (as happened in the recent pandemic
situation caused by COVID-19) are other examples of important campus management objectives
that would benefit from this platform. Using video-based techniques for this purpose, indoor
and outdoor, is not usually identified as a possible solution for economic reasons [
Most of the techniques used in this paper are more frequently detailed in video surveillance
or Computer Vision applications. Concerning university campuses and the project’s context,
those techniques should be used and framed by specific requirements. When researching
related projects, we searched within both domains, but emphasised application rather than
the algorithm’s development themselves. In [
In [
The general system architecture for the Lab4USpaces platform is divided into four layers, as shown in Figure 1. The physical layer includes the tracking component, which is located where all sensors and actuators are placed. IP cameras capture video and send it to an edge server for processing, including configuration management and object geolocation. The network layer enables wireless communication between the sensor subsystems and the middleware. The integration layer includes an Identity and Access Manager module for device authentication, a Message Broker for communication organisation, a Temporal Database for data storage, and the Home Assistant platform as the Hub. The application layer uses the collected data for analysis, visualisation, and decision-support applications. The data tracking subsystem will be explained in detail later.
The object detection module must be able to identify correctly all people in crowded scenarios
using low-cost video cameras. Such scenarios pose challenges like occlusion and clustering,
hindering precision and recognition [
There are two types of trackers: ofline (previous and subsequent frames are available to create more accurate predictions), and online (work on the fly). In this project, we need a real-time online tracking system. Multiple problems can occur, like occlusions, initialisation and termination of tracks, people with similar appearances, and interaction between multiple objects. Occlusion can cause identity switches and fragmentation of trajectories, which should be avoided in our project. A common way to benchmark object tracking algorithms is to use the MOTChallenge, a standardised evaluation framework for multiple object tracking (MOT). It contains two datasets with indoor and outdoor videos.
We evaluated four MOT algorithms: SORT [
This section presents the results of object detection, tracking, and re-identification when applied to images captured in the Lab4U&Spaces project, using two video cameras, one inside and the other outside a building, in connected spaces without scene overlapping. We also describe the configuration settings of the global system and demonstrate the final results on campus.
Concerning object detection, Figures 3a and 3b display the outcomes obtained from both cameras. The inside camera covers a wider area with extreme resolution variation due to near and far objects, with no constraints imposed on the minimum object size. It is noticeable that some people in the most distant zone, on the right side of the figure, are not detected. Even though, the selected technique performs better than all other solutions in terms of response time and computing resources required. The rate of false negatives in figure 3b is approximately 64% – despite not being optimal, it is acceptable.
Concerning the Single Camera Multiple Object Tracking adopted technique, Figures 3c and 3d illustrate the result of the BoT-SORT algorithm in sequential scenes captured from both cameras. The average time required for detecting and tracking people was between 142.8ms (7 (a) outside the building
(b) inside the building
(c) outside the building
(d) inside the building
FPS) and 263ms (3.8 FPS). These values are consistent with the recommended frame rate for
this application class, as suggested in [
Concerning the re-identification operation, figure 4a shows a person crossing the delimiter zone line in the outside scene, using the label "cam2_1" (meaning camera 2 and the person ID 1); Figure 4b shows the same person entering the inside camera’s view area a few seconds later and being labelled "cam1_22", meaning it was not yet re-identified; finally, in figure 4c, it is visible that the ID was redefined, to the one previously assigned by camera 2 (about two seconds after being initially detected). In the small dataset used, out of four possible re-identifications, three of them were correctly performed suggesting an eficacy of 75% – but particularly in this case, more experiments are required to validate this result.
(a) subject ID at outside (b) subject ID at inside (c) subject ID re-assigned
After completing the main loop shown in Figure 2, the detection and tracking data is stored in the Hub. The collected data can be utilised in several applications, such as the one depicted in Figure 5, which shows in real-time or using recorded data the density of people in diferent campus spaces through colour and bubble size codes – this example uses only one camera, for illustration purposes. The left image displays the complete campus map, whereas the right image focuses on a particular corridor where the indoor camera was installed. The tracking module needs additional configuration details to function properly throughout the entire campus, allowing to characterise the space and optimise computational and storage resources. A web application was created to manage configuration data. The main items include Zone, defined by multiple polygons in each camera’s field of view, allowing for the definition of zones of interest where specific views or details should be highlighted; Line Intersection Zone, which delimits the boundary between zones in a camera’s scene where particular operations like re-identification or people counting should be applied; Black Area, used to remove unwanted areas from a camera’s view where no person can be found, or when two cameras overlap to prevent resource wastage; and Global Coordinates, necessary to track and re-identify individuals throughout the campus (it involves mapping each camera’s field of view to the global campus map and define a scale, angle, and ofset to transform the tracking data into campus’ coordinates.
This paper describes a framework for tracking people on a university campus as part of the Lab4U&Spaces project, which aims to develop a platform for exploring smart campus technologies. We evaluated various technologies and selected the ones that best suits the project requirements, which included low computational resources, energy constraints, and opensource solutions. Privacy is another fundamental requirement we guarantee by not storing any image for consultation or beyond the time strictly required in the re-identification function. We conducted experiments at the prototype level to validate all operations and found that the framework is viable. We also discussed the potential use of this technology at the campus level. However, to determine the framework’s actual usefulness, it needs testing with more than two cameras and evaluation of the behaviour of thousands of daily campus visitors. As future work, we start designing applications that exploit all available data to transform campus management and life into more intelligent activities.
Lab4U&Spaces – Living Lab of Interactive Urban Space Solution, Ref. NORTE-01-0145-FEDER000072, financed by community funds (FEDER), through Norte 2020. 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), 2018, pp. 368–371.