Eficient multimedia event processing is a key enabler for real-time and complex decision making in streaming media. The need for expressive queries to detect high-level human-understandable spatial and temporal events in multimedia streams is inevitable due to the explosive growth of multimedia data in smart cities and internet. The recent work in stream reasoning, event processing and visual reasoning inspires the integration of visual and commonsense reasoning in multimedia event processing, which would improve and enhance multimedia event processing in terms of expressivity of event rules and queries. This can be achieved through careful integration of knowledge about entities, relations and rules from rich knowledge bases via reasoning over multimedia streams within an event processing engine. The prospects of neuro-symbolic visual reasoning within multimedia event processing are promising, however, there are several associated challenges that are highlighted in this paper.
Internet of multimedia things (IoMT), data
analytics and artificial intelligence are
continuously improving smart cities and urban
environments with their ever-increasing
applications ranging from trafic management
to public safety. As middleware between
internet of things and real-time applications,
complex event processing (CEP) systems
process structured data streams from multiple
producers and detect complex events queried
by subscribers in real-time. The enormous
increase in image and video content
surveillance cameras and other sources in IoMT
applications posed several challenges in
realtime processing of multimedia events, which
motivated researchers in this area to
extend the existing CEP engines and to devise
new CEP frameworks to support
unstructured multimedia streams. Over the past few
years, several eforts have been made to
mitigate the challenges in multimedia event
processing by developing techniques for
extension of existing CEP engines for
multimedia events [
ing data is conventionally modelled
accord2. Background ing to RDF [
This section presents a review of the re- representation and tracking of entities and
cent work in stream reasoning, multimedia relations [
CEP engines inherently lacked the support
for unstructured multimedia events, which
was mitigated by a generalized approach for
handling multimedia events as native events
in CEP engines as presented in [
lations. Table 1 presents some knowledge events will be performed on the objects,
rulebases publicly available for visual reasoning. based relations and relations extracted
usWan et al. [
3.1. Prospects 1. Suitable representation for reasoning It is crucial to select a generalized and The current multimedia event representation scalable model to represent events and efecmethods use knowledge graph to represent tively perform automated reasoning to derive the detected objects, their attributes and re- more meaningful and expressive spatiotemlations among the objects in video streams. poral events.
Pre-defined spatial-temporal rules are used 2. Expressive query definition and
to form relations among the objects. How- matching Providing a generic and
humanever, the complex relations that exist among friendly format to subscribers for writing
real-world objects also depend on seman- expressive and high-level queries would
retic facts and situational variables that can quire new constructs. Matching queries with
not be explicitly specified for every possi- the low-level events and relations along with
ble event as rules. The statistical reason- reasoning via knowledge bases requires
efing methods and knowledge bases discussed ifcient retrieval within the complex event
in Section 2 have great potential to com- matcher. Real-world complex events can
plement the rule-based relation formation share similar patterns, occur as a cluster of
in multimedia event processing by inject- similar events or occur in a hierarchical
maning some semantic knowledge and reason- ner, which requires generalized, adaptive and
ing to extract more semantically meaning- scalable spatiotemporal constructs to query
ful relations among objects. This advance- such events.
ment will allow subscribers to define abstract 3. Labeling and training samples of
vior high-level human-understandable event sual relations There can be a large numbers
query rules that can be decomposed into of objects and possible relations among them
spatial and temporal patterns. The spatio- in images, which can result in a large
numtemporal matching of the queried high-level ber of categories of relations. It is dificult
been explored much, which is crucial for
spabalanced categories of relations in the
traintiotemporal event processing.
ing data. For example, Visual Genome [
person, human, man) are not always the same. Similarly, knowledge bases have diferent labels for the same entity, diferent names for the same attribute (e.g. birthPlace and placeOfBirth) or relation (e.g. ’at left’ and ’to left of ’). This can cause inconsistency or redundancy while integrating relations from the knowledge bases. It is important to select the knowledge base and dataset that are consistent and suitable for the combined use of both object detection and visual reasoning. 5. Supporting rare or unseen visual relations Apart from the common relations, very rare or unseen relations among objects also appear in certain scenes. It is nearly impossible to collect suficient training samples for all possible seen and unseen relations.
Handling such relations while evaluating the models is also a challenge. 6. Temporal processing of objects and relations The recent methods on this subject address complex inference tasks by decomposing images or scenes into objects and visual relations among the objects. The temporal events and temporal tracking of the detected objects and predicted relations has not
support of the Science Foundation Ireland