While the concept of digital twin, and the use of semantic technologies, is already established in the eld of IoT, and while the co-existence of such other sensor devices with IP-enabled video cameras is common, this combination, in so-called Video IoT, has yet to see widespread adoption of these technologies. In this paper we discuss our `in use' context in managing all of these devices, and our experience in building a prototypical implementation of digital twins using semantic technologies.
The introduction of the IP-enabled digital video camera, commonly called IP
camera, by Axis Communications in 1996 [
In particular, modern VMSs provide multiple user interfaces which may be `desktop' (installed), which allows capabilities like driving a video wall, or Web or mobile based | native mobile clients often providing more advanced user interactions, including comprehensive provision for searching over existing video recordings. Furthermore, modern IP cameras allow a high degree of softwarebased con guration, both with analogues to traditional cameras | i.e. aperture, shutter speed, etc. | and digital speci c video streams characteristics | such as resolution and codec / compression targets. In fact a single camera might produce multiple streams with di erent settings for this latter class. Furthermore, for an entire class of cameras, commonly called `PTZ cameras', there is continuous operational control of motorised pan and tilt, as well as zoom. These settings motivate having a database of device management settings, as well as a media database, containing the actual recordings. ? Supported by the EU Horizon 2020 project 'SAFECARE: Integrated cyber-physical security for health services', grant agreement number 787002.
Video Wall IP-based Video
Cameras
Desktop Client
Search
Alarms
Video Monitoring
Video Review Camera Drivers
Video Management Server Recording Module Media DB Video Analytics
Video Monitoring
Video Review
Mobile Client
Search
Alarms Integration
Plug-ins Search
Module Device Mgt.
DB Events, Rules
& Alarms
The next common feature of modern VMSs is some provision for rules and events via which they are triggered. An in-built source of events might be motion detection within a video stream, so that the rule can trigger recording of the stream; in this way storage is not wasted on video in which no motion occurs. Some more sophisticated level of video analytics might also provide events that trigger rules: for instance, license plate recognition might trigger the recording of vehicles. External devices might also trigger events via integration plug-ins.
One of the most successful commercial VMS o erings is XProtect [
XProtect has adopted the open source framework GStreamer [
One of the key device integrations in XProtect is with access control systems, which control physical access to areas of buildings. Figure 2 illustrates in more detail how an access control system may be used in a door. What might colloquially be called a `lock', or more properly a `strike', controls electronically when a door can be opened. A door controller unit contains the logic to control the strike, and this is connected to a reader, which may simply be an RFID card reader or PIN pad or, commonly, a combination thereof. When a card is used to identify a permitted person, the door controller can operate the strike to allow the door to be opened. REX
In some con gurations, a reader is used on both sides of the door. In others, a simple REX (request-exit) button is used, in which the operator does not need to identify themselves with a card or PIN, as exit is allowed for all. In more sophisticated set-ups the door might be equipped to automatically open, or `speed gates' might be employed, which have both physical and digital means to prevent | or at least provide an alarm in the case of | `tailgating', that is a second unauthorised person gaining access by following closely behind an authorised user.
In this paper, we will sketch a combination of the above technologies in which access control on a simple door is combined with video analytics, speci cally object detection and tracking, to provide a means for tailgating detection. Further, a knowledge graph that contains information on the layout of a hospital building and the devices deployed in it, based on a set of ontologies that also allow the representation of events based thereon in dynamic scenarios, will be shown to allow inference of the vulnerabilities when such situations occur. An integration with re sensors will also be included, which show escalation of the scenario of attack.
These scenarios are de ned by the collaborative project SAFECARE1, specifically by three hospital consortium partners, the Assistance Publique | H^opitaux de Marseille (coordinator of the project), the Academisch Medisch Centrum in Amsterdam and ASLTO5 (Azienda Sanitaria Locale - Chieri - Carmagnola Moncalieri - Nichelino) | hospitals in the Piedmont region of Italy. While trials of the project's results are planned with all three hospital partners, an interim test platform will be housed at the Instituto Superior de Engenharia do Porto, as illustrated in Figure 3.
This part of the test platform for the SAFECARE project is built in Unreal
Engine [
In the remainder of this paper we will rst introduce the ontologies used to build this system, and then sketch the cloud-based architecture via which the system was created. 2
In order to locate devices within a building, simulated or real, it is natural that we introduce a building ontology, which is shown in Figure 6. While this ontology introduces concepts of Building and Floor, further structure is left to the concept Location, which is de ned in the Grid Ontology, shown in Figure 7; this concept is subclassed to de ne Zones and Intersections. In a building context, Zones describe areas such as rooms and hallways, while Intersections are those junctures between Zones, which may be controlled for instance by physical access control.
It is worthy of note that the Grid Ontology was not de ned for this buildingbased use case, but originated in the description of tra c networks, where Intersections are literal tra c intersections, and these are controlled by tra c lights.
In a similar way, the IoT Ontology shown in Figure 8 is intended to represent a whole array of sensor-type devices, which in another context might be tra c sensors like inductive loops. Two characteristics of the IoT ontology are of particular relevance. First, those subset of devices modelled by the Camera class have a FieldOfView property, which is illustrated by four yellow triangles on the map in Figure 5, representing the eld captured by the cameras in the XProtect Smart Client on the left. Second, while all devices have a locatedAt relationship with locations, AccessController s have a second relationship named controlsAccessTo; this is shown for select parts on the oor plan in Figure 10. leadsTo leadsTo leadsTo leadsTo Hallway 1
Door 1
Hallway 2 locatedAt leadsTo leadsTo
Door 2 leadsTo
Door 3 Fig. 10. Graph Representing Segment of Floor Plan leadsTo leadsTo leadsTo leadsTo
Observation Room 1
Emergency Room 1
Finally, the Events Ontology, shown in Figure 9 is used to model the dynamics of this system, as follows: { AccessControlEvent is used to represent the operation of the access control reader | this carries a property to convey whether the authorisation was successful; { AccessChangeEvent is used to represent the operation of the door sensor associated with the access controller, or of the gate in the case of speed gates; { DoorTransitionEvent is used to represent the event that a human actually transits through a door, and is produced using video analytics (though in the current prototype this is simulated by having an extension to Unreal Engine that produces this event when a person model passes through a door); { FireDetectionEvent is used to represent the operation of the re detection sensors; { Alarms are produced by event processing logic, described in the following section, as relate the events that led to the alarm.
All instances of these event types carry a timestamp, and can be related to locations, sensors and actors. 3
The software architecture used for this system in shown in Figure 11. This is cloud-based, built on Amazon Web Services, and uses the following services in particular: { AWS EKS | the Elastic Kubernetes Service | allows the orchestration of containers of custom code, for instance deployed using Docker containers; { AWS MSK | the Managed Streaming in Kafka service | allows horizontally scalable publish-subscribe messaging of payloads such as JSON messages, via Apache Kafka; { AWS AppSync | a managed service based on Apache Apollo that allows the formation of GraphQL-based APIs, including the ability to subscribe to receive noti cations according to a graph pattern; { AWS Neptune | a managed graph database service, which supports the W3C RDF and SPARQL standards.
Using these technologies together with the ontologies described in the
previous section, we can construct this system according to a uniform graph-based
data model. In particular, JSON-LD payloads are used on Kafka via which the
sensors, in the prototype proxied via the virtual hospital in Unreal Engine,
transmit their changes. Clients can both subscribe to hear particular messages, for
instance ltering by sensor or event type, using GraphQL, a combination of
technologies that is explored for instance in [
The complex event processing involved in inferring tailgating from constituent events is illustrated in Figure 12. In particular the sequence of a single authorisation in the form of an AccessControlEvent, followed by two DoorTransitionEvents, without an interleaved AccessControlEvent leads to Events Processing to raise a TailgatingAlarm and publishing this back to the Kafka topic. true ac1 cam1 cam1 ac1 cam1 AccessControlEvent
DoorTransitionEvent
DoorTransitionEvent
TailgateAlarm AWS MSK (Kafka)
target Door1 target
target Door1
Door1
target Door1
By querying the oor plan graph, according to the dynamic information on state, we can consequently determine a reachability graph for the tailgater. Similarly, since the state of doors change in the event of a re alarm, allowing free access due to the potential need to escape res, this reachability result can be updated in the event of re, as shown (reachability indicated in red) in Figure 13.
We have shown how an ontology-based approach allows both static information on the situation of security devices within a hospital environment, and the dynamics of these devices, to be modelled. We have shown further that an architecture based on modern, horizontally-scalable cloud technologies can be used to realise a useful implementation. We have illustrated the use of digital twins for both access control devices and re sensors for simulation within this framework.
Work is already underway in ontology-based representation of the full set of camera, and other device, con guration based on the XProtect Management Server. In future work we will make the virtual cameras true digital twins, where these can be used as proxies for real hardware cameras.
The approach here would be greatly aided by the convergence of GraphQL and RDF/SPARQL, and we aim to contribute to this e ort. Similarly the use of Apache Kafka together with JSON-LD, we feel, is an important enabling approach for scaling the promising approach of digital twins in an IoT context where network connectivity is neither constant nor reliable.
The demonstration is available at https://youtu.be/UYH YI 31v4?t=906, but the intention is also to give this demonstration at ESWC2020, both during the workshop and at the demo session, for which a shorter paper has been submitted.
The authors would like to acknowledge the work of Dennis J rgensen at Milestone Systems in assembling the complete virtual hospital model and developing its virtual devices, as well as John Madsen and Tong Su at Milestone Systems for their work on ontologies and graph data. We would also like to thank the sta of Data Language for their collaboration on this project, and particularly their insights into the use of GraphQL and cloud architecture.
The virtual hospital made use of: { 'Modular Hospital', a 3D model developed by Aleksandr Zhdanov, and purchased via Unreal Marketplace. { 3D models for cameras, re sensors and access control panels purchased via SketchFab, with the following credits: https://sketchfab.com/Terizmeh https://sketchfab.com/gleb tihon https://sketchfab.com/michael.diaz029