We present 2SDS (Scene Separation and Data Selection algorithm), a temporal segmentation algorithm used in real-time video stream interpretation. It complements CNN-based models to make use of temporal information in videos. 2SDS can detect the change between scenes in a video stream by com-paring the image diference between two frames. It separates a video into segments (scenes), and by combining itself with a CNN model, 2SDS can select the optimal result for each scene. In this paper, we will be discussing some basic methods and concepts behind 2SDS, as well as presenting some preliminary experiment results regarding 2SDS. During these experiments, 2SDS has achieved an overall accuracy of over 90
Image recognition models have gone increasingly
accurate in the past few years, yet video semantics tasks are
still challenging. A detailed comprehension on video
stream could play a significant part in video accessibility
[
In the processing of video stream, a 2D CNN can be implementation that complements the CNN model to
extended into 3D CNN by adding a temporal dimension solve the continuity issue. This implementation should
[
The problem is that 2D CNNs only recognise a video
as discrete images, rather than a continuous stream of 1.2. Related Work
images. This poses some issues. For example, a CNN
model could not resolve the motion of a person (e.g., SlowFast Networks. The SlowFast Networks use a
twowalking, dancing) be-cause the person is stationary in pathway architecture for video recognition, the slow
every frame, and this will cause the loss of significant pathway (low frame rate) is used to capture spatial
seinformation in video analysis. So, we need to devise an mantics, and the fast pathway (high frame rate) is used to
capture temporal semantics like motions in a relatively
ifne temporal resolution [
STRL’22: First International Workshop on Spatio-Temporal Reasoning and Learning, July 24, 2022, Vienna, Austria * Corresponding author. 1.3. Our Work † These authors contributed equally. " sc20yx2@leeds.ac.uk (Y. Xin); sc20zz2@leeds.ac.uk (Z. Zhou); What we have achieved is to devise the temporal segmensc202yx@leeds.ac.uk (Y. Xia) tation algorithm, 2SDS, which stands for “Scene Separa0000-0002-9732-2414 (Y. Xin); 0000-0003-2613-7569 (Z. Zhou); tion and Data Selection algorithm”. It can slice the video 0000-0002-1185-2722 (Y. Xia)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License stream into segments on the temporal axis, so it can be
inCPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) terpreted using 2D CNN models while preserving critical
information on the temporal dimension. By combining
2SDS with a CNN model (Fig. 2), this implementation is
similar to the SlowFast Networks on splitting the input
into two pathways, in which the 2SDS is similar to the
fast pathway of the SlowFast Networks, except we do not
introduce another neural network, but we replace the
network with the faster 2SDS, which guarantees even
better temporal resolution.
2SDS stands for “Scene Separation and Data Selection
algorithm”. It works as a temporal segmentation and
result selection algorithm to complement CNN-based
models. It contains a two-part procedure of separating
the video stream into segments and selecting a
represen2. Motivation: Why Not Neural tative recognition result from the CNN model for output.
2SDS utilises the diference hash (dHash) method [
RNN-based models like LSTM [
However, RNNs do have the advantage of acting upon lowing sections) from the whole data segment of a scene temporal information, especially for models like LSTM. (referred as candidate in the following sections). So, we need to help the CNN-based models to preserve temporal information, and that is where we introduce our temporal segmentation algorithm, 2SDS. 3.1. Scene Separation: based on dHash
By adding the 2SDS algorithm, alongside a CNN model, The scene separation procedure of 2SDS (Fig. 3) is based we were able to achieve RNN-like results. In the mean- on an improved version of the dHash algorithm, which time, by avoiding the introduction of a neural network, is originally used to judge the similarity of two images. this implementation is also faster than the CNN + RNN By applying the scene separation procedure, the tempoarchitecture or the CNN + CNN architecture. ral information can be preserved by the sequencing of the separated scenes. The exact workflow of the scene separation process is discussed extensively below. whole image (this is also the reason why the original im- ously used in the data smoothing procedure, which is age is down sampled into an 8 by 9 sub-image rather than also a Weighted Average Pooling operation. This procean 8 by 8 sub-image, because the 8 by 8 image will face dure will select the recognition result from a frame whose some inconvenience when converting into a hexadecimal feature intensity is the closest to the weighted average hash value).
value of all the candidates (feature intensity refers to Calculating the Hamming distance. By calculating the number of diferent classes of objects in a particular the Hamming distance between the hash values of two frame). adjacent frames, we can judge whether the two frames
scene. This particular recognition result will be used to represent the whole scene it is located in, and through the help of NLP and other models, this can even be used to output the natural language interpretation of this video scene.
could only provide some preliminary and experimental 1 usage of the 2SDS algorithm .
We choose YOLOv5s as our image recognition CNN
for this experiment, and we have built an experimental
dataset on video object detection using selected YouTube
videos in the YouTube-VOS dataset [
We are most interested in how 2SDS will perform in scene separation (temporal segmentation) tasks. We classified the testing videos into 3 classes: interviews, vibrant, and hybrid.
The interviews are usually straight forward and easier to undergo scene separation tasks. Vibrant videos are the more dificult ones due to their fast-moving images and transition efects that might seem deceptive to 2SDS. The hybrid video sits in between the first two types, they have some features of the interview videos, as well as features from the vibrant videos, their dificulty should sit in the middle.
We conducted 3 separate experiments using interview videos (Table 1). The total amount of scenes in these 3 experiments is 82. The overall accuracy of 2SDS during these experiments is 90.10%. There are 2 cases where we ifnd the 2SDS algorithm actually over-judged the transition between two scenes. This is potentially a sensitivity issue posed by the hard coded threshold during scene separation.
on scene separation (temporal segmentation) tasks, more detailed experiments are still needed to be conducted.
Vibrant 1 Vibrant 2
We conducted 2 separate experiments using vibrant videos (Table 2). The total amount of scenes in these 2 experiments is 51. The overall accuracy of 2SDS during the two experiments is 54.90%. The accuracy in vibrant videos is substantially lower than interview videos for the 2SDS is unable to separate two fast-moving scenes efectively. It is important to notice that we used harsh videos like sport videos and dynamic advertisement videos in this experiment, so the performance of the 2SDS is expected to be much lower comparing to the previous experiment. This is the biggest limitation of 2SDS, but this issue is addressable with future improvements of the algorithm.
We conducted one experiment using a long hybrid video (Table 3). The total amount of scenes in this experiment is 106. The overall accuracy of 2SDS is 99.06%. Theoretically, the result of this experiment should sit between the previous two tests, however, an anomaly has arisen most likely due to the lack of samples. A more detailed experiment should be conducted to further determine the accuracy of 2SDS on hybrid videos.
can potentially play a huge role in video interpretation.
Previously dificult and untouchable problems like
continuous gesture recognition and scene recognition are being
cracked using the CNN-based spatio-temporal reasoning
model [
Our work has only utilised the temporal information
in video stream, our future work can make use of graphs,
and spatially model a frame into a graph, with the
objects as the vertices and the spatial relations between
the objects as the edges, like the MST-GNN [
This future work would bring immense potential with the use of spatial information, which will add a whole other dimension of usable information that can beneift video analysis with richer semantics and the ability of grouping fast-moving frames, bringing video interpretation models yet another step closer to how human perceive visual information.
ing temporal segmentation methods, we devised 2SDS, a temporal segmentation algorithm that can be used alongside CNNs to complement for the lack of temporal information handling ability of the CNN-based models. We gave, yet another powerful tool that CNN models can utilise, the ability to take advantage of the inherent temporal aspect of videos. Video stream analysis is a completely diferent task compared to image recognition, and we are finally seeing some evidence that we can still use 2D CNNs to interpret video information.
The 2SDS algorithm utilise a refined diference hash value method and a novel data smoothing and data selection technique to crack the temporal segmentation problem. Although there are still drawbacks with fastmoving frames in vibrant videos, the 2SDS algorithm has already done a great job at separating relatively simple and stationary scenes in videos, and it gets the job done at a respectful speed, which will allow the 2SDS to get a finer temporal resolution compared with neural networks.
For future work, some improvements on 2SDS (e.g., adding graphs to model spatial relations) can potentially boost the algorithm’s performance on fast-moving scenes and smooth transitions.