-The Federal Aviation Administration (FAA) is observing an increasing number of incidents involving recreational drones, and imagining a future where every drone will be equipped with an Automatic Dependent Surveillance-Broadcast (ADS-B) transponder that communicates and cooperates with the FAA's Next Generation (NextGen) Aviation Cyber-Physical System in order to help mitigate aerial collision risk [1]. This exemplar application involves human or autonomous agents interacting within some sort of cyber-physical system where competition or cooperation between cyber-physical elements exist. We anticipate that the use of higher-level abstractions will be required for modeling human or autonomous agent's interactions within these type of systems in order to make sense of the observations derived from sensor-data. In this paper, we articulate an approach that uses event semantics to represent the temporal, spatial, factor, and outcome features of activities generated by competing or cooperating agents functioning within a cyber-physical environment. We use those semantics, along with observations of activity, to model higher-level activity abstractions and to help perform strategy recognition from a concrete, competition-oriented scenario reflected in a real-world, game data set comprised of more than a half million events involving nearly 8500 unique agents. The strength of the approach is grounded in a specification of event semantics for our concrete multi-agent, competitive game ontology using Resource Description Framework Schema (RDFS) and Ontology Web Language (OWL). By leveraging these Semantic Web languages, we anticipate that the use of event semantics to describe cooperative or competitive agent interactions within cyber-physical systems will become more predominant in the future. Index Terms-Agent-Based Model, Human Agent, Autonomous Agent, Unmanned Aerial Vehicle, Gridiron Football, Semantic Web
Cyber-physical systems (CPS) are at the outset of
completely changing how society interacts with the physical world
around it. These systems measure different features across
the physical environment (e.g., the location of an agent) and
enable computational models that interact with a cyber-core
(i.e., the computing and communications backbone of the CPS)
and with their corresponding physical environment to provide
some desired benefit or utility. In most cases, sensors provide
the cyber-core with the primary mechanism for recognizing
events or changes in the physical environment. The actions and
interactions between human or physical autonomous agents
and the cyber-core are captured through sensors. Consider
some real-world applications where the activities of human and
physical autonomous agents are identified by sensors coupled
to a cyber-physical system:
• In the National Football League (NFL), each football
player and stadium is equipped with RFID sensors and
receivers permitting the league to track fine-grained
location data for each play. In this case, the Internet-of-Things
(IoT) and CPS has been incorporated into operations and
management of professional sports venues [
These examples also illustrate a specific characteristic present in some cyber-physical systems where competition or cooperation exist between human or physical autonomous agents. Subsequently, this paper reflects an interest in these type of cyber-physical systems. This research effort is narrowly focused on identifying higher-level abstractions such as strategies used by agents from observations derived from sensors in the CPS. Further, we focus on an approach for the representation and modeling of competitive actions and interactions of agents in cyber-physical systems.
The high-level methodology for this research activity has
been decomposed into five components. First, an exemplar for
our experiment is identified. Gridiron Canadian and American
Football is distinguished as an elaborate, competitive game
activity that involves multiple agents, which are organized
into two teams for the purpose of executing a series of
offensive/defensive advances intended to score points and win
the game. This scenario is selected to match the CPS exemplar
identified in [
III. PROPOSED SOLUTION
Our research effort was presented with a considerable
challenge pertaining to the acquisition of rich context data
that would comprise human or agent activities, and that would
simulate the type of data we would expect to acquire from the
scenario identified in [
Prior to recent introduction of RFID sensors into NFL game
stadiums [
An effort was made to grasp an understanding of the
physical environment in which our football player agents operate.
Domain knowledge of concepts such as game field, players,
game timeline, driving the ball, alternating possession of the
ball, kickoffs, free kicks, scoring and penalties associated
with Gridiron Football was acquired primarily through a
literature review of popular publications such as Football’s
Matchup Zone Coverages [
The contributed Gridiron Football Ontology expresses a
formal representation of knowledge within the Gridiron (or
American and Canadian) Football domain via a set of concepts
and inter-concept relationships. The ontology is engineered
according to a methodology that partitions the ontology into
two levels: theory ontology (or upper ontology) and domain
ontology (or lower ontology). The theory ontology is abstract
and compact. It focuses on concepts such as time, space, goals,
etc. The domain ontology provides a formal description of
the classes (i.e., concepts) and relationships between classes
that exist in a domain. In this manner, the Gridiron Football
Ontology is conceived as a two-level ontology with an upper
level that abstracts a football game as a sequence of
organized spatio-temporal events and a lower level that provides
a concrete specification of the ontology components (i.e.,
individuals, classes, attributes, relations, etc.) associated with
Gridiron Football. By concrete specification, we mean that the
ontology is machine-readable and understandable. The result
is that RDF Schema (RDFS) Language and Web Ontology
Language (OWL)-based ontological software components were
developed for all items (370+ unique key-value pairs) in
the NFL Play-by-Play data set. Ontological modeling and
engineering efforts were assisted using several Semantic Web
tools to include CMAP Knowledge Modeling Environment [
At the center of the upper ontology is the notion of an
event, and we reuse the existing Event Ontology developed
at Queen Mary, University of London [
In figure 2, domain-specific classes and properties are created through subsumption of the event:Event class to model the many type of events associated with Gridiron Football. For example, fball:GameEvent class conceptualizes a football game whereas fball:SeasonEvent represents a football season. In the case of the NFL, a season event is comprised of 16 football games over a duration of 17 weeks. The fball:hasFootballSubEvent property is devised through subsumption of the event:hasSubEvent property and used to partition a Season event and a Game event into a collection of games and collection of plays respectively. The foaf:Agent class and its sub-classes, foaf:Person and foaf:Group are extended to model the various types of football playing positions (e.g., fball:LeftTackle), the membership of particular playing positions to certain football sub-groups (e.g., a Left Tackle has membership of the fball:OffensiveLineUnit). The fball:Player is established to represent the idea of a generic football player that has datatype properties (i.e., attributes) such as fball:hasFullName, fball:hasPrimaryPosition, fball:hasHeight, fball:has40YardDashTime and fball:hasBenchPressWeight. The fball:hasPlayer, fball:hasUnit and fball:hasTeam properties are created by means of subsumption of the event:hasAgent property to relate football persons and groups to kinds of football events. For example, the fball:hasPlayer property is used to establish a relation between the fball:FumbleEvent and one of three player roles expected or required during a fumble type of event: fball:Fumbler, fball:ForcingPlayer and fball:RecoveringPlayer.
Domain-specific classes and properties are devised by way of subsumption of the event:Product class to model the many types of outcomes that result from different types of football events associated with Gridiron Football. For example, fball:KickoffOutcome is related to fball:KickoffEvent using the fball:hasOutcome. Though not shown, the event:Product and event:hasProduct are subsumed to generate the various types of football outcomes and fball:hasOutcome property respectively. In addition, domain-specific classes and properties are devised by way of subsumption of the event:Factor class to model the many types of factors that affect different types of football events associated with Gridiron Football. For example, fball:WeatherFactor and fball:FieldConditionFactor are related to fball:GameEvent using the fball:hasEventFactor. The event:Factor and event:hasFactor are subsumed to generate the various types of football factors and fball:hasEventFactor property respectively.
In addition, portions of the Time [OWL-TIME] [
This work involved creating efficient RDF representations
for the NFL Play-by-Play data set using the Gridiron Football
Ontology. The effort experimented with reasoning software
for the purpose of providing an inference reasoning capability
to the project’s software components for deriving new RDF
triples (knowledge) from the instances generated directly from
input data and its related ontology. This activity involved
software development using the Apache Jena Open Source
Java Framework for Semantic Web Development [
Specifically, this effort consisted of the deployment a
SPARQL End-Point web server and RDF-based triple store
using the Apache Fuseki/TDB suite [
IV. ACTIVITY AND STRATEGY RECOGNITION
In this section, we apply our approach of ontology-based
activity recognition to the Gridiron Football domain and try
to show how event semantics may be used to help identify the
base offensive scheme being used by a particular team during
a football game. In Gridiron Football, an offensive scheme
can be thought of as an offensive strategic system that a team
uses to counter his opponent’s defensive attack. Here, we show
multiple components that may be decomposed from a team’s
overall offensive scheme according to [
Up Tempo • Passing Component (Huddle): Normal Huddle and No
Huddle • Passing Component (Length of Passes): Short to Intermediate and Vertical Intermediate to Deep Passing Game • Passing Component (Quarterback Position): Under Center and Pistol Depth, Shotgun Depth • Passing Component (Route Assignments): Route Tree Assignments (Air Coryell), Group Assignments (ErhardtPerkins) • Passing Component (How, Where, When): Predetermined Pass to Spot Before Break, Predetermined Pass to a Person after the Break and Option Pass to a Person after the Break
In our experiment, we attempt to identify these components
of an offensive scheme. This is at least partially achieved by
integrating a natural language-to-RDF query engine to our
fball project’s SPARQL-endpoint/RDF triple store. The
integration of these two technologies allowed our research team
to interrogate the fball event knowledgebase for a collection
of propositions (i.e., statements that are either true or false)
that may be supported by the facts represented in the
triplestore knowledgebase. Principally, a collection or sequence of
statements would be derived to simulate a strategy-recognition
pattern that matches against the RDF-encoded triples in the
store and provide evidence of certain offensive scheme
components. Identifying certain offensive components such as Pass
Component (Huddle): No Huddle and Passing Component
(Tempo): Hurry-Up Tempo was fairly simple and straight
forward to accomplish. This ease to identify a particular offensive
component was due primarily to completeness of data and
that offensive component being directly available within the
fball knowledgebase to directly support that question. In other
cases, even with the detection of an offensive component or
combination of offensive components it was not sufficient for
identifying a more complex offensive strategy. To illustrate that
point, here we focus on a particular type of offensive strategy
called a West Coast Offense strategy. As background domain
knowledge, we offer football passing theory that describes
the West Coast Offense as the concept of using short passes
to replace some of the running attacks [
In figure 7, we show an example of the type of natural
language query used by our researchers for detecting pass
completions by a particular team during a single football
season where the receiver or pass target caught the pass within
5 yards of the goal line and net gained 15 yards or greater
after the pass completion. It follows that this particular team,
which is the 2013 Miami Dolphins, coached by Joe Philbin
and offensively coordinated by Mike Sherman, was known for
using a West Coast Offensive strategy during the 2013 game
season [
Our researchers asked the following question: "In the time duration of a football game or season is this team using a west Coast Offense strategy as part of its offensive scheme? It follows that the research team proposed the following hypothesis, which is two-fold: • In regard to the Passing Component (Setup Mode), if the team demonstrates a higher percentage of Pass to Setup the Run attempts (i.e., passing attempts) than a team that demonstrates a higher percentage of Run to Setup the Pass attempts (i.e., rushing attempts); and • in regard to the Passing Component (Length of Passes), if a team demonstrates a higher percentage of Short to Intermediate pass attempts as compared to the percentage of Vertical Intermediate to Deep Passing Game passes.
In the evaluation of our detection pattern for the use of a West Coast Offense strategy in the base offensive scheme for a particular team, we were not able to identify teams that were definitely employing this strategy. In figure 5, we illustrate the average percentage of short-to-intermediate pass attempts and intermediate-to-long pass attempts that were executed by NFL teams that were known to use a West Coast Offense strategy as part of their base offensive scheme during the 2013 game season. In figure 6, the same statistics are illustrated; however, in this case we show the statistics of teams that were not known to use a West Coast Offense strategy during the 2013 Season. A quick visual examination of these bar charts show that there is not any major difference in the percentage of short- to long-distance pass attempts between the two category of teams (i.e., those known to use a West Coast Offense and those that do not). In addition to the statistics on Passing Component (Length of Passes), statistics related to the Passing Component (Setup Mode) also did not show a major difference between teams known to use a West Coast Offense versus those that are not known to use that type of strategy.
V. PRELIMINARY RESULTS & DISCUSSION
A number of insights were made as a result of our research effort. First, we believe that the complexity and fine-granularity of the real-world, NFL Play-by-Play data set provided our team with an intermediate step for evaluating the application of semantic-based event models to the observations of human or autonomous agents engaged in competition at the scale we would expect within a cyber-physical system. As stated previously, NFL football players and stadiums are equipped with RFID sensors and we assert that the type of sensor data that we expect to be derived by an NFL Stadium Venue’s cyber-physical system (CPS) will be similar to the data used by our research team. This type of activity data provided the research team with a unique challenge in effort to properly reflect in each RDF statement the appropriate semantic using the Gridiron Football Ontology. We expect that the modeling of other applications or domains involving human or autonomous agents competing or cooperating within a cyber-physical system would present a similar challenge.
The primary strength of the ontology-based activity (strategy) recognition approach is the relative simplicity and straightforwardness involved in incorporating domain knowledge and heuristics into the recognition models. The use of event semantics was especially beneficial in this regard. For instance, the The Event Ontology provides the ability to model and interrogate the NFL event knowledgebase based on five dimensions: event type, time, location, factor and outcome. Additionally, the upper ontology also provided an abstraction (i.e., event:hasSubEvent) for describing an event that is composed of other events. This abstraction allowed the research team identify what type of football sub-events were associated with a particular play event, drive event, game event or season event. For example, a particular play event may be comprised of a penalty event and a pass event.
Though challenging, the research team determined that developing semantic queries that can detect certain strategies being used by NFL competing agents is possible. Initially, the research team exclusively relied on the development of SPARQL queries. Thereafter, the team learned that the guidance of an expressive Natural Language-to-RDF query builder such as SPARKLIS is useful for formulating straightforward queries for answering particularly complex hypotheses. The weakness of solely using the ontology-based strategy recognition approach is the lack of learning ability in terms of identifying patterns that can identify certain complex strategies. In this case, machine learning techniques for performing statistical and probabilistic reasoning may have been useful. However, the logical model approach (i.e., ontology-based activity recognition) can certainly play a dominant role when it is integrated along with techniques for learning patterns as well as dealing with the inability of the logical model to represent fuzziness and uncertainty. We offer that our approach and contribution is an intermediate step that can be further extended to include using an instance of an event ontology as a seed ontology for statistical and probabilistic strategy recognition. In some cases, this seed ontology may be used to develop a more comprehensive ontology using ontology learning techniques.
In this paper, an approach is given for capability that uses
event semantics to represent the temporal, spatial, factor, and
outcome characteristics of events generated from the
observations of agents engaged in a competitive activity between each
other. Further, we have described the likeness of the data set
used for this experiment with the kind of data set we would
expect to be generated from the type of "cyber-physical game"
scenario identified in [
The authors would like to acknowledge the Sparklis research activity under Dr. Sebastien Ferre supported at IRISA, Universite de Remes, Rennes cedex, France. The authors would also like to acknowledge Arm Chair Analysis as the source of the NFL data set used in this research project.