Cyber-physical systems (CPS) are at the outset of completely changing how society interacts with the physical world around it. These systems measure different features across the physical environment (e.g., the location of an agent) and enable computational models that interact with a cyber-core (i.e., the computing and communications backbone of the CPS) and with their corresponding physical environment to provide some desired benefit or utility. In most cases, sensors provide the cyber-core with the primary mechanism for recognizing events or changes in the physical environment. The actions and interactions between human or physical autonomous agents and the cyber-core are captured through sensors. Consider some real-world applications where the activities of human and physical autonomous agents are identified by sensors coupled to a cyber-physical system:
• In the National Football League (NFL), each football player and stadium is equipped with RFID sensors and receivers permitting the league to track fine-grained location data for each play. In this case, the Internet-of-Things (IoT) and CPS has been incorporated into operations and management of professional sports venues [2]. • In the recreational Unmanned Aerial Vehicle (UAV) market, some manufacturers have equipped their drones with ADS-B sensors, which is a type of sensor for aerial cooperative collision detection and avoidance [3]. ADS-B is an element of the Federal Aviation Administration (FAA)'s Next Generation Air Transportation System (NextGen), which has been described as a airborne network instantiation of a cyber-physical system [1]. These examples also illustrate a specific characteristic present in some cyber-physical systems where competition or cooperation exist between human or physical autonomous agents. Subsequently, this paper reflects an interest in these type of cyber-physical systems. This research effort is narrowly focused on identifying higher-level abstractions such as strategies used by agents from observations derived from sensors in the CPS. Further, we focus on an approach for the representation and modeling of competitive actions and interactions of agents in cyber-physical systems.
The high-level methodology for this research activity has been decomposed into five components. First, an exemplar for our experiment is identified. Gridiron Canadian and American Football is distinguished as an elaborate, competitive game activity that involves multiple agents, which are organized into two teams for the purpose of executing a series of offensive/defensive advances intended to score points and win the game. This scenario is selected to match the CPS exemplar identified in [2]. A NFL play-by-play data set that offers a likeness or model of the kind of RFID-derived data we would expect in [2] is acquired for the experiment. Second, the information within the data set is conceptualized based on the domain knowledge of Gridiron Football and modeled for its semantic relationships using event semantic abstractions. The result is the contributed Gridiron Football Ontology, which is a conceptual vocabulary of American and Canadian football, in the namespace http://www.ncsu.org/kr/fball. Third, the data set is extracted from its previous form, transformed into the Resource Description Framework (RDF) metadata model (including serialization) according to the fball ontology and loaded into a public-accessible SPARQL Protocol and RDF Query Language (SPARQL) Endpoint and RDF Store. Fifth, we integrate or point a public-accessible SPARQL Endpoint Explorer for Expressive Question Answering to the Gridiron Football Event Endpoint in order to interrogate the nearly 40 million triples generated for indications of tactics or strategies being employed during game events. Further details are provided in each respective section of the paper.
Our research effort was presented with a considerable challenge pertaining to the acquisition of rich context data that would comprise human or agent activities, and that would simulate the type of data we would expect to acquire from the scenario identified in [2]. Prior to recent introduction of RFID sensors into NFL game stadiums [2], the data capture of movements of players on a football game field demanded a human-in-the-loop to observe the execution of plays across the field and to sequentially report the specifics of each event as illustrated in figure 1. During games, this reporting data is manually generated by a distributed network of league statisticians, and immediately propagated from individual stadiums to hundreds of websites in a span of seconds [2]. In turn, the data set used in the experiment is provided by a sports analysis firm that uses web extraction and other techniques to generate a structured data set from disparate web post left by these league statisticians. The data set contains NFL play-by-play events from 2000 to 2013. Further, we assert that this data is characteristic of the type of complex yet coarse-or fine-grained event data that can be expected from observations of human or autonomous physical agent's activities as they engaged in competition within a cyber-physical system.
An effort was made to grasp an understanding of the physical environment in which our football player agents operate. Domain knowledge of concepts such as game field, players, game timeline, driving the ball, alternating possession of the ball, kickoffs, free kicks, scoring and penalties associated with Gridiron Football was acquired primarily through a literature review of popular publications such as Football's Matchup Zone Coverages [4], The Art of Place-Kicking and Punting [5], Defending the Spread Offense [6], Offensive Football Strategies [7] and Winning Football [8]. Familiarity of these concepts was used to formulate ontological abstractions designed to codify types of events with associative kinds of entity-attribute-models with agent, temporal, spatial, factor and product classes and relationships in the Gridiron Football Ontology.
The contributed Gridiron Football Ontology expresses a formal representation of knowledge within the Gridiron (or American and Canadian) Football domain via a set of concepts and inter-concept relationships. The ontology is engineered according to a methodology that partitions the ontology into two levels: theory ontology (or upper ontology) and domain ontology (or lower ontology). The theory ontology is abstract and compact. It focuses on concepts such as time, space, goals, etc. The domain ontology provides a formal description of the classes (i.e., concepts) and relationships between classes that exist in a domain. In this manner, the Gridiron Football Ontology is conceived as a two-level ontology with an upper level that abstracts a football game as a sequence of organized spatio-temporal events and a lower level that provides a concrete specification of the ontology components (i.e., individuals, classes, attributes, relations, etc.) associated with Gridiron Football. By concrete specification, we mean that the ontology is machine-readable and understandable. The result is that RDF Schema (RDFS) Language and Web Ontology Language (OWL)-based ontological software components were developed for all items (370+ unique key-value pairs) in the NFL Play-by-Play data set. Ontological modeling and engineering efforts were assisted using several Semantic Web tools to include CMAP Knowledge Modeling Environment [9] and Stanford Protege [10].
At the center of the upper ontology is the notion of an event, and we reuse the existing Event Ontology developed at Queen Mary, University of London [11]. The ontology is partitioned into a set of classes and properties identified as: event:Event, which is an arbitrary classification of a space/time region, which may have participating agents, passive factors, products, and a location); event:sub_event, which provides a mechanism to partition a complex event; event:agent, which relates an event to an active agent such as a person; event:time, which relates an event to a time object such as a duration of time. event:place, which relates an event to a spatial object; event:product, which relates an event to something produced during an event; and event:factor, which relates an event to something that contributes to its result such as a cause.
In figure 2, domain-specific classes and properties are created through subsumption of the event:Event class to model the many type of events associated with Gridiron Football. For example, fball:GameEvent class conceptualizes a football game whereas fball:SeasonEvent represents a football season. In the case of the NFL, a season event is comprised of 16 football games over a duration of 17 weeks. The fball:hasFootballSubEvent property is devised through subsumption of the event:hasSubEvent property and used to partition a Season event and a Game event into a collection of games and collection of plays respectively. The foaf:Agent class and its sub-classes, foaf:Person and foaf:Group are extended to model the various types of football playing positions (e.g., fball:LeftTackle), the membership of particular playing positions to certain football sub-groups (e.g., a Left Tackle has membership of the fball:OffensiveLineUnit). The fball:Player is established to represent the idea of a generic football player that has datatype properties (i.e., attributes) such as fball:hasFullName, fball:hasPrimaryPosition, fball:hasHeight, fball:has40YardDashTime and fball:hasBenchPressWeight. The fball:hasPlayer, fball:hasUnit and fball:hasTeam properties are created by means of subsumption of the event:hasAgent property to relate football persons and groups to kinds of football events. For example, the fball:hasPlayer property is used to establish a relation between the fball:FumbleEvent and one of three player roles expected or required during a fumble type of event: fball:Fumbler, fball:ForcingPlayer and fball:RecoveringPlayer. Domain-specific classes and properties are devised by way of subsumption of the event:Product class to model the many types of outcomes that result from different types of football events associated with Gridiron Football. For example, fball:KickoffOutcome is related to fball:KickoffEvent using the fball:hasOutcome. Though not shown, the event:Product and event:hasProduct are subsumed to generate the various types of football outcomes and fball:hasOutcome property respectively. In addition, domain-specific classes and properties are devised by way of subsumption of the event:Factor class to model the many types of factors that affect different types of football events associated with Gridiron Football. For example, fball:WeatherFactor and fball:FieldConditionFactor are related to fball:GameEvent using the fball:hasEventFactor. The event:Factor and event:hasFactor are subsumed to generate the various types of football factors and fball:hasEventFactor property respectively.
In addition, portions of the Time [OWL-TIME] [12] and The WGS84 Geo Positioning Ontology [Basic Geo] [13] vocabularies are used. The primary classes and properties include: time:TemporalEntity, which is a parent class that relates temporal information to an event; time:Interval, which is a subclass of TemporalEntity and temporal things with extent that have interior points; and geo:SpatialThing, which is a parent class that relates spatial information to an event. For brevity, the time and spatial ontological components used in the fball ontology are not described; however, spatial ontological components are illustrated in figure 3.
This work involved creating efficient RDF representations for the NFL Play-by-Play data set using the Gridiron Football Ontology. The effort experimented with reasoning software for the purpose of providing an inference reasoning capability to the project's software components for deriving new RDF triples (knowledge) from the instances generated directly from input data and its related ontology. This activity involved software development using the Apache Jena Open Source Java Framework for Semantic Web Development [14] to covert the elaborate NFL play-by-play data set, which is comprised of more than a half million game events and nearly 8500 unique agents, to a semantic graph containing 44,676,644 RDF triples.
Specifically, this effort consisted of the deployment a SPARQL End-Point web server and RDF-based triple store using the Apache Fuseki/TDB suite [15]. Ingest of triples into the triple store are primarily made by scripts or the upload feature in the Apache Fuseki web client. Queries are made through also made through the SPARQL interface within the Fuseki client as well as the Sparklis: a SPARQL Endpoint Explorer for Expressive Question Answering [16] web service. In Figure 4, an illustration of SPARKLIS being used to interrogate the generated NFL Football triples is given.
In this section, we apply our approach of ontology-based activity recognition to the Gridiron Football domain and try to show how event semantics may be used to help identify the base offensive scheme being used by a particular team during a football game. In Gridiron Football, an offensive scheme can be thought of as an offensive strategic system that a team uses to counter his opponent's defensive attack. Here, we show multiple components that may be decomposed from a team's overall offensive scheme according to [17]: In our experiment, we attempt to identify these components of an offensive scheme. This is at least partially achieved by integrating a natural language-to-RDF query engine to our fball project's SPARQL-endpoint/RDF triple store. The integration of these two technologies allowed our research team to interrogate the fball event knowledgebase for a collection of propositions (i.e., statements that are either true or false) that may be supported by the facts represented in the triplestore knowledgebase. Principally, a collection or sequence of statements would be derived to simulate a strategy-recognition pattern that matches against the RDF-encoded triples in the store and provide evidence of certain offensive scheme components. Identifying certain offensive components such as Pass Component (Huddle): No Huddle and Passing Component (Tempo): Hurry-Up Tempo was fairly simple and straight forward to accomplish. This ease to identify a particular offensive component was due primarily to completeness of data and that offensive component being directly available within the fball knowledgebase to directly support that question. In other cases, even with the detection of an offensive component or combination of offensive components it was not sufficient for identifying a more complex offensive strategy. To illustrate that point, here we focus on a particular type of offensive strategy called a West Coast Offense strategy. As background domain knowledge, we offer football passing theory that describes the West Coast Offense as the concept of using short passes to replace some of the running attacks [18]. Moreover, the short pass receiver is expected to run for good yardage after the completion. Therefore, a West Coast Offense strategy is a composite strategy of at least two of the components identified in the previous enumeration: the Passing Component (Setup Mode) and Passing Component (Length of Passes). In figure 7, we show an example of the type of natural language query used by our researchers for detecting pass completions by a particular team during a single football season where the receiver or pass target caught the pass within 5 yards of the goal line and net gained 15 yards or greater after the pass completion. It follows that this particular team, which is the 2013 Miami Dolphins, coached by Joe Philbin and offensively coordinated by Mike Sherman, was known for using a West Coast Offensive strategy during the 2013 game season [19]. In figure 8, we show the results of the query. Our researchers asked the following question: "In the time duration of a football game or season is this team using a west Coast Offense strategy as part of its offensive scheme? It follows that the research team proposed the following hypothesis, which is two-fold:
• In regard to the Passing Component (Setup Mode), if the team demonstrates a higher percentage of Pass to Setup the Run attempts (i.e., passing attempts) than a team that demonstrates a higher percentage of Run to Setup the Pass attempts (i.e., rushing attempts); and • in regard to the Passing Component (Length of Passes), if a team demonstrates a higher percentage of Short to Intermediate pass attempts as compared to the percentage of Vertical Intermediate to Deep Passing Game passes. In the evaluation of our detection pattern for the use of a West Coast Offense strategy in the base offensive scheme for a particular team, we were not able to identify teams that were definitely employing this strategy. In figure 5, we illustrate the average percentage of short-to-intermediate pass attempts and intermediate-to-long pass attempts that were executed by NFL teams that were known to use a West Coast Offense strategy as part of their base offensive scheme during the 2013 game season. In figure 6, the same statistics are illustrated; however, in this case we show the statistics of teams that were not known to use a West Coast Offense strategy during the 2013 Season. A quick visual examination of these bar charts show that there is not any major difference in the percentage of short-to long-distance pass attempts between the two category of teams (i.e., those known to use a West Coast Offense and those that do not). In addition to the statistics on Passing Component (Length of Passes), statistics related to the Passing Component (Setup Mode) also did not show a major difference between teams known to use a West Coast Offense versus those that are not known to use that type of strategy.
A number of insights were made as a result of our research effort. First, we believe that the complexity and Fig. 7. An illustration of a natural language query using the SPARKLIS engine/interface that models the pattern of West Coast Offense scheme . fine-granularity of the real-world, NFL Play-by-Play data set provided our team with an intermediate step for evaluating the application of semantic-based event models to the observations of human or autonomous agents engaged in competition at the scale we would expect within a cyber-physical system. As stated previously, NFL football players and stadiums are equipped with RFID sensors and we assert that the type of sensor data that we expect to be derived by an NFL Stadium Venue's cyber-physical system (CPS) will be similar to the data used by our research team. This type of activity data provided the research team with a unique challenge in effort to properly reflect in each RDF statement the appropriate semantic using the Gridiron Football Ontology. We expect that the modeling of other applications or domains involving human or autonomous agents competing or cooperating within a cyber-physical system would present a similar challenge.
The primary strength of the ontology-based activity (strategy) recognition approach is the relative simplicity and straightforwardness involved in incorporating domain knowledge and heuristics into the recognition models. The use of event semantics was especially beneficial in this regard. For instance, the The Event Ontology provides the ability to model and interrogate the NFL event knowledgebase based on five dimensions: event type, time, location, factor and outcome. Additionally, the upper ontology also provided an abstraction (i.e., event:hasSubEvent) for describing an event that is composed of other events. This abstraction allowed the research team identify what type of football sub-events were associated with a particular play event, drive event, game event or season event. For example, a particular play event may be comprised of a penalty event and a pass event.
Though challenging, the research team determined that developing semantic queries that can detect certain strategies being used by NFL competing agents is possible. Initially, the research team exclusively relied on the development of SPARQL queries. Thereafter, the team learned that the guidance of an expressive Natural Language-to-RDF query builder such as SPARKLIS is useful for formulating straightforward queries for answering particularly complex hypotheses. The weakness of solely using the ontology-based strategy recognition approach is the lack of learning ability in terms of identifying patterns that can identify certain complex strategies. In this case, machine learning techniques for performing statistical and probabilistic reasoning may have been useful. However, the logical model approach (i.e., ontology-based activity recognition) can certainly play a dominant role when it is integrated along with techniques for learning patterns as well as dealing with the inability of the logical model to represent fuzziness and uncertainty. We offer that our approach and contribution is an intermediate step that can be further extended to include using an instance of an event ontology as a seed ontology for statistical and probabilistic strategy recognition. In some cases, this seed ontology may be used to develop a more comprehensive ontology using ontology learning techniques.
In this paper, an approach is given for capability that uses event semantics to represent the temporal, spatial, factor, and outcome characteristics of events generated from the observations of agents engaged in a competitive activity between each other. Further, we have described the likeness of the data set used for this experiment with the kind of data set we would expect to be generated from the type of "cyber-physical game" scenario identified in [2]. The approach extends existing modular vocabularies and is based in the specification of event semantics for our contributed Gridiron Football Ontology using Resource Description Framework Schema (RDFS) language and Ontology Web Language (OWL). Our future work has already begun and includes: extending the event ontology to an applicable data set for autonomous physical agents cooperating or competing within a cyber-physical system. Moreover, our effort seeks to integrate aspects of game theory analysis with the ontology-based strategy recognition approach to account for concepts such as payoffs and tensions between the different strategies that may be used by agents.
The authors would like to acknowledge the Sparklis research activity under Dr. Sebastien Ferre supported at IRISA, Universite de Remes, Rennes cedex, France. The authors would also like to acknowledge Arm Chair Analysis as the source of the NFL data set used in this research project.