Knowledge representation Aandbsretarsoancintg (KR&R) systems are widely employed for the representation of abstract knowlmet, an(dKiRti&sRex)pescytsetdetmhast athree dweisdigenlyateedmepffleocytsedwiflolrtatkhee place wrheepnrtehseeancttaiotnioinsexoefcautbesdt.rHaocwtekvnero, wemlebdogdeie.dAagcetniotsn need admdiotidoenlasl kanroewluedsugealalbyourtehporwesethnetiartbioodnys shoofulsdtabtee moved ttoraancshiiteivoentsh:eiArgcotaiolsnwsitchaonut bcaeuspinegrfuonrwmaendtedifsidaell performtahnecier cbooupdlyedswhiothulsdubb-seymmboovliecddattoa, aancdhitehveye sthhaereir Reseavracthe aCnedntaecrad(Semonidcepruforrpsocsheusn.gsbereich) 1320 “EASE - Ev-
edge. Action models are usually representations of state tran
Knowledge representation and reasoning sitions: Actions can be performed if all pre-conditions are effects. The proposed action representation is based on force pre-conditions are met, and it is expected that dynamic events that occur when an embodied agent interacts with itsthweordlde.sWigneastheodw ehowectpsattwerinlsl otfakfoercpeleavceentswchaenn be usedt htoe daeficntieosnemisanteicxsecouftaecdti.onHvoerwbes.vRero,boetmsubsoedoiuerd their experience through the knowledge service OPENEASE. goals without causing unwanted side e ects.
on force dynIanmtricodevuecnttisonthat occur when an The cogenmitiboondiseydsteamgenotf inhtuemraacntss awlliothwsitsuswotorlda.cWcoemplish masnhipouwlahtioown ptaastktservnesryofcfoomrcpeeetevnetnlyts. cTahnisbies upsoesdsible throutgoh dthee noergasnemizaatniotincsofoafctaicotnisonin vteerrmbss. oRf omboottison phases, aunsde tohurorumgho dtheel tporedaiccqtuioinreoefpeisffoedcitcs mthaetmaocrtiieosns might cauwsheiicnhtaermestoofrifeosrcoef etvheenirtsptehraftomrmigahntcoeccouur.pled
In thiswwiothrk,suwbe-sinyvmesbtoigliactedaantaa,ctaionnd mthoedyelsphoasrteultahteedirin human pseyxcpheorlioegnyc,eantdhmroaukgehusethoef it kinnoanwalerdtigfieciaslesryvsitceem. The modeolpweansEpAroSpoEs.ed by Flannagan et al. (2006). Actions are decomposed into motion phases with different subgoals. The subgoals are force dynamic events that also generate
distinctive sensory feedback in the nervous system. lematic because events may be monitored in the physics ensible through the organization of actions in terms of gine of virtual worlds, or observed by some agent. This is, motion phases, and through the prediction of e ects for example, that the hand gets into contact with the milk packageTbheeforeresegarrcahsprienpgorittedfroinmthtihseptaapbelreh,aosr btheeant sthuepppoarctekd- by age ltohoesGesercmoanntaRcetsteoartchheFsouupnpdoarttioinngDsFuGrf,aacsepwarhteonf Cthoellaabgoernattive perfoRremsesaarcrhetCraecnttienrg(mSoontdioernfoarfstcehrutnhgesbmerilekichh)as1b32e0en\EgrAaSsEpe-d.Everyday Activity Science and Engineering", University of Bremen ⇤ T(hhtetpr:e/s/ewarwcwh.ereapseo-rctrecd.oirng/t)h.is paper has been supported by Intelligence (www.aaai.org). All rights reserved. erydaIyn:AGc.tiSvtiteyinSbcaiueenrc,eAa.nFderErenignin(eeedrsi.n):g”P,rUocneievdeirnsgitsyooffthBere1m1tehnInReach Push Retract contact+ contactthat actions might cause in terms of force events that
One of the main reasons for investigating action models might occur. from human psychology in robotics is that action models in
In this work, we investigate an action model posAI, such as PDDL (Ghallab et al. 1998), usually do not have atnulaaptperdopinriahteumlevaenl opfsyacbhsotrlaocgtyio, nafnodr rmobaoktes.uIsnepoafrtiictuilnar, aacntioanrtmiocdiaellssyinstAe mIo.fTtehneambsotrdaeclt wawasayprfroopmosbeoddbyymFoltaionn-s annadgaonnlyetcoanl.ce[6n]t.raAtectoionnrseparreesednetcinogmapcotisoend pinret-oamndotpioonstcpohnadsietisonwsi,tahndiseeqrueenntcesus.bIgnotealllsi.geTnht eemsubbogdoieadlsagaerentfsonreced tdoybnraidmgiec tehveegnatps tbheatwt eaelnsothgeesneerreaptreesdeinsttainticotnisvewsitehnsmoirsysifnegedinbfaocrkmiantiotnheanndertvhoeuasctsuyaslteemxe.cution of an action in the physical world. Bridging this gap is non trivial and a prob
Intentions of others can not be monitored directly. lem which is widely unsolved on the abstract level (i.e., by Monitoring force events, on the other hand, is at least re-usable general knowledge). It is further expected that conless problematic because events may be monitored in ditions and effects of actions are pre-defined – a hard to meet the physics engine of virtual worlds, or observed by requirement with the diversity of effects actions may cause isnotmhee pahgyesnict.alTwhoirsld.
is, for example, that the hand gets into contact with the milk package before grasping it
The central question for successful embodied action exefcruotmionthies htaobwlea,goernttshsahtouthlde mpaocvkeatgheeilroboosedsiecsotnotaacchtieve to ctehretasinupefpfoecrttsinwghsiluerafavcoeidwinhgeunnwthaentaegdesnitdep-eerfffeocrtms.sTahirsei-s, ftorraecxtianmgpmleo,thioown aafrtoebrotthsehomuilldkmhoavsebietsenargmrassupcehdt.hat the panOcankee mofixthceonmtaaininedrienastohnesbofottrlei nitvehsotlidgsatiisnpgouarcetdionon tmopodofeltshefropmanchaukmeamnakpesry,cahnodlofgoyrmins aropbaontciacskeiswtihthat1a0cc-m dtiiaomn emteor.dIenlsthine aAreI,a soufchAIasthPerDeDarLe [o7n]l,yufseuwallayppdrooancohtes that address this problem despite the semantic nature of this have an appropriate level of abstraction for robots. In reasoning problem. particular, action models in AI often abstract away from body motions and only concentrate on representing action pre- and post-conditions, and sequences. Intelligent embodied agents need to bridge the gap between these representations with missing information and the actual execution of an action in the physical world. Bridging this gap is non trivial and a problem which is widely unsolved on the abstract level (i.e., by re-usable general knowledge). It is further expected that conditions and e ects of actions are pre-de ned { a hard to meet requirement with the diversity of e ects actions may cause in the physical world.
The central question for successful embodied action execution is how agents should move their bodies to achieve certain e ects while avoiding unwanted sidee ects. This is, for example, how a robot should move its arm such that the pancake mix contained in the bottle it holds is poured on top of the pancake maker, and forms a pancake with 10cm diameter. In the area of AI there are only few approaches that address this problem despite the semantic nature of this reasoning problem.
One of the peculiarities of our KR&R system is that
it runs inside the perception-action loop of a robotic
agent. Symbols correspond to data structures of the
robot control system, and as such they have a rather
simple grounding. The representations in our system
are inspired by the role that episodic memories play in
the acquisition of generalized knowledge in the human
memory system [
The proposed representation of episodic memories consists of two parts. One part stores experiences and events as symbolic data. Those events and experiences can be e.g. perceived objects, or performed actions, their duration, and possible failures. The second part stores sensor data from the robot in a database. We de ne this unstructured data as sub-symbolic data. In the rst section, we will describe the symbolic knowledge representation. An overview about the subsymbolic data will be given afterwards. Then, we will show how those memories can be used to improve the robot's action models by getting insights about manipulation activities. This will be achieved by using a combination of query answering and visual analytic tools.
Our KR&R system is made available as part of the
knowledge web service openEASE 1 [
We use the openEASE platform for storing and managing the episodic memories represented with our model. It also allows to ask queries about it, such as how the robot was moving when an action was performed, and to visualize snapshots of the activity with visual annotations. Figure 1 shows such an example where the robot was closing a drawer in a kitchen environment. The action is properly segmented into the di erent motion phases, which is also visible in the Figure. The vision is to collect a large data set of episodic memories, and utilize them for learning tasks to gain a better understanding about manipulation activities.
There are several projects with e orts to provide
symbolic knowledge about manipulation activities to
robots. The most notable one is the IEEE-RAS
working group ORA (Ontologies for Robotics and
Automation) [
Another related branch of research is task and
motion planning. In this work, we present an action model
that can be used to yield higher level activities from
observations of force events. Such force events can
also be detected through haptic feedback, and be used
to minimize uncertainty during manipulation
activity planning [
Another aspect is that our system can yield partial
boundaries of motion phases given some observation.
Motion segmentation methods typically apply some
form of clustering to build stochastic representations of
primitive motions and motion sequences. These
methods include self-similarity [
This section introduces an action model for robots inspired by the Flanagan model. The basis of it are force events that occur when an agent moves its body, and the di erent motion phases of actions. Our ontology is organized along these areas. It has 4 levels: Force events, situations, motion phases, and intentional activities. In addition, we use rules to declare identity constraints. In this section we provide a description of how this information is organized and represented.
In this work, we build upon the KnowRob ontology, and (manually) extend it with concepts of our action model such as ForceEvent and PouringMotion. We have chosen KnowRob because it provides the necessary infrastructure for interfacing with robot control systems, and to record episodic memories from task execution. It de nes concepts such as Event and Situation, and also speci c ones to describe e.g. robots and their parts.
At the lowest level of our action representation there are events that physical objects cause in a (simulated) physical world. They are described independently from intentions. This is to allow detecting them fully automated, without taking into account previous events and higher level knowledge about task or embodiment.
PhysicalEvent v Event is the most general concept in this ontology. It implies that physical events occur at a particular time instant (derived from Event ), and that at least one object is involved. Involved means that one of their physical properties is salient during the event. This is the case if the object involved is created or destroyed, touched or untouched, transformed into something else, etc.
The most essential events are the contact events (ContactEvent) that occur whenever an object moves in the world such that it touches (contact+ v involved) another object within a spatial region (contactRegion).
The property contact+ is further decomposed into functional properties contact+1 and contact+2 denoting the two salient objects during the contact event (the two objects can be randomly assigned). The objects remain touched until they separate again which is indicated by a LeavingContactEvent. The contact is either caused by an agent moving objects into contact, or through a physical process such as gravity, for example, pulling an object such that it falls onto the oor.
Creation (CreationEvent ) and destruction (DestructionEvent ) events are also distinctive subgoals of activities that we use for activity representation at a higher level of our ontology (e.g., cutting a bread creates a slice of bread).
The last category of physical events we consider in the scope of this work are uid ow events (FluidFlowEvent ). These are events in which some liquid or gaseous substance moves, for example, milk owing from a package to a glass, or water owing in a river. Such events may be intended as in \pouring milk in a glass", or unintended as in \spilling milk on the oor during navigating". The primary involved object is the liquid or gaseous substance, linked to the event via the functional property uid v involved.
At the next level of our ontology there are situations during which force events occurred (ForceSituation v Situation). Force events occur at time instants, for example, in the moment the hand touches some object, and when it leaves contact again. We use such temporal patterns of force events to expand them to distinctive situations.
Sub-events are linked to situations via the inverse functional event object property. With inverse functional we imply that each event can only be the subevent of a single situation. For detecting situations, we use two dedicated events: One indicating the start and the other indicating the end of the situation. These are represented using the functional properties starter v event for the event starting the situation, and stopper v event for the one stopping it.
Surely, starter event should occur before stopper
event. Situations during which the object is not in
contact could else be classi ed as contact situations.
We use predicates from Allen's interval algebra [
The starter event of contact situations is the contact event and the stopper event is the leaving contact event. Both have exactly the same involved objects. We represent this type of information using identity constraint rules using a property chain starting from the starter event via involved objects, stopper event, and back to the starter event.
Fluid ow situations are a bit di erent because there are no distinct starter and stopper event types. At some time instant the rst and at a later time the last uid ow event of a situation occurs. However, not every sequence of uid ow events referring to the same uid makes a situation. If the container is put aside for a while, for example, one would rather say that the situation ended then, and that a new situation starts when the container is used later on. This can be enforced by asserting that, during uid ow situations, the container may only be salient for uid ow events.
Motions can be detected by monitoring the joint conguration of an agent. Movements are either re exive or intentional. But at this level of our ontology, without knowing intentions of agents, we can not distinguish between re exive and intentional motions and represent motions solely in terms of expected events and body parts used.
The di erent body parts are de ned in the KnowRob ontology. Here, we de ne a general \body part moved" concept for each of these body parts. We de ne the functional relation partMoved to represent which body part moved during a motion, and restrict the range of this property to the corresponding body part type. For ArmMovement 's, for example, we assert: 8partM oved:Arm and = 1partM oved:Arm. Force events salient for a motion are denoted by the inverse functional event relation. Temporal ordering constraints are asserted by temporal properties before, after, and during.
Here, we only investigate arm movements. Hand movements are also represented, but only at a coarse level using a boolean state: Opened or closed. We also ignore gaze motions in this work. However, it would be interesting to look into gaze contact events and to compare gaze patterns for di erent expert levels in future work. Arm movements are fundamental for object manipulation. The repertoire of di erent arm motions of humans is rich: reaching, lifting, throwing, cutting, pouring, etc. Some of which have distinct patterns of force dynamic events, such as cutting, that we use for representing them.
We use force events as delimiters of motion phases. In particular important are contact situations between body parts and other objects. Motions during which lifetime the contact between body part and object is continuously salient are called carrying motions (CarryingMotion). The body part in contact with the object must be part of the body part (denoted by partOf ) which is moved during the motion. This is to allow, for example, that the contact occurs between hand and tool while the body part referred to by the motion is the arm (which in turn has a hand part).
Objects held by agents may also touch other objects or liquids during the motion, causing distinct force events during that interaction. We use this pattern of force events for the representation of tool motions. A cutting motion, for example, is a carrying motion, performed with a cutting tool, during which some object was cut into pieces. Cutting events may also be destruction events in case the object cut into pieces entirely disappeared. We further assert that the tool used in the cutting event (cutter ) is also salient during the carrying situation.
Another challenging manipulation task is pouring. It can be performed in many ways, and on many different expert levels. The motion pro les of di erent expert levels are drastically di erent, but they all generate uid ow events when particles are leaving the source container. We represent pouring motions as contact situations with a subgoal which is a uid ow situation. First, we state that pouring motions are carrying situations where a container that contains some uid is a salient object, and that at least one uid ow situation is a subgoal of this situation. We further state that the uid transported in uid ow events of subgoals is exactly the uid inside of (contains) the contacted container.
At the highest level of our ontology there are activities composed of motions with expected event patterns. At this level of the ontology, the intention of agents is implied by action concepts. The standard example quoted in the work of Flanagan et al. is a fetch-andplace activity. During fetch-and-place tasks, there is a contact situation between agent and fetched object, and also distinct events indicating that the carried object rst leaves contact to a supporting surface, and later gets into contact with a supporting surface again.
We state that fetch-and-place activities have a submotion which is a carrying motion. And that there are two additional force events linked to the action via the subevent relation. We further state that there is a subevent in which the carried object looses contact to a supporting surface.
At this level, we can distinguish between colliding, supporting, and intentionally touching. Unexpected contacts during an activity are classi ed as collisions. This makes it very easy to detect them. With expected we mean that the activity concept asserts their occurrence during the activity.
We use the same scheme to distinguish between pouring and spilling: Pouring actions have intended uid ow subgoals while spillage events are exactly the unintended uid ow events occurring during an action. More concretely, pouring actions have a target location where the uid should be poured into or onto. We classify all uid ow events where the uid is transported to somewhere else then the target location as spillage events.
Experience data captures low-level information about experienced activities represented as time series data streams. This data has often no or only unfeasible lossless representation as facts in a knowledge base. To make this data knowledgable, procedural hooks are dened in the ontology to compute relations from the experience data, and to embed this information in logicbased reasoning.
The data is stored in a NoSQL database using JSON documents. Each individual type of data is stored in a collection named according to the type of data stored in it. When imported, the knowledge system stores the data in a MongoDB 2 server, for which the knowledge system implements a client for querying the data during question answering.
A robotic system typically has many mobile components arranged in a kinematic chain. Each component in a kinematic chain has an associated named coordinate frame such as world frame, base frame, gripper frame, head frame, etc. 6 DOF relative poses are assigned to frames. These are usually updated with about 10 Hz during movements, and expressed relative to the parent in the kinematic chain to avoid updates when only the parent frame moves. The transformation tree is rooted in the dedicated world frame node (also often called map frame).
The data is used by our knowledge system to answer questions such as: \Where was the base relative to the object, 5 seconds ago".
The knowledge represented in acquired experiences is very comprehensive. It not only contains narrations of activities but also raw experience data. Competent robot behavior needs both: Experience data encodes particularities of motions such as forces and velocities, and the narrative is required to make sense of the data at higher cognitive levels.
Here, we provide reasoning examples with our action representation. We rst describe how activities can be obtained from force events, and also how an agent can make sense of action concepts. We nally outline some analytical reasoning tasks that can be performed on episodic memories.
In virtual worlds, force dynamic events can be monitored perfectly. These can be asserted to the knowledge base as they occur. Given the occurrence of force events, we can infer new knowledge using descriptions from higher levels of our ontology. In the rst step, the events are expanded to situations. The situations are then re ned to motions with distinct force event patterns. Finally, high level activities are detected based on patterns of force events and motions.
The expansion process exploits representations of situation concepts to identify events that determine the situation. Situations are determined by so called starter and stopper events. The events are processed from earliest to latest. A situation symbol is created when a starter event was detected, and a triple that speci es the starter relation is asserted. The procedure stores a list of situations without stopper events. For each new event, this list is rst iterated to test whether the event is a stopper event of the situation, and a triple that speci es the stopper relation is asserted if this is the case. Finally, it is also tested if new events are sub-event of one of the situations without stopper.
We assume that arm motions are only segmented by zero velocity segmentation in advance. We use force events as delimiters for coarse-grained segmentation. We think that this segmentation is su cient because it captures the force events which are the essential subgoals of manipulation activities. Here, we only consider arm motions. For each situation during which an arm motion occurred, we iterate through the di erent subclasses of ArmMotion which are also contact situations, and we test if classifying the situation with that type would yield a contradiction. The motion type is asserted if this is not the case. The motion classes are disjoint such that situations can only be classi ed as being instance of one of the motion classes.
Motions and force events are then used as building blocks for activities. Activities can be parsed using rules that detect temporal patterns of events and motions that are distinctive for them. Force events and motions that are subgoals of activities are denoted by the subevent and submotion properties. Patterns with partial ordering constraints can be inferred from this model. The output of the parser is an ontology, describing instances of detected actions. Here, we provide one hand-written rule that is used to detect pick-andplace activities shown in Algorithm 1.
Algorithm 1 Detect Pick-and-Place 1: procedure detect-pick-and-place 2: CarryingMotion(?s); contact+S (?s; ?obj), 3: LeavingContactEvent(?ev1), 4: loose-support-event(?s; ?ev1; ?obj), 5: ContactEvent(?ev2), 6: gain-support-event(?s; ?ev2; ?obj), 7: before(?ev1; ?ev2). 8: procedure loose-support-event(?s,?ev,?obj) 9: contact-(?ev; ?obj); contact-(?ev; ?t), 10: SupportingSurface(?t), 11: stopper(?s; ?x); before(?ev; ?x). 12: procedure gain-support-event(?s,?ev,?obj) 13: contact+(?ev; ?obj); contact+(?ev; ?t), 14: SupportingSurface(?t), 15: starter(?s; ?x); after(?ev; ?x).
Our ultimate goal is to enhance the performance of robots by supplying them with knowledge about everyday activities, and in particular with high-level stories about what happened combined with experience data. In this section, we provide a description of how robots may use the information represented in episodic memories.
A typical query rst asks for a particular semantic action that ful lls certain constraints such as being successful, being performed by a particular agent, etc. The inferred action symbol is bound to a variable which is used as index to sub-symbolic data in the experience part of episodic memories. This is done to access data slices corresponding to the semantic activity for which the symbol was inferred earlier. An example of such a query is shown in the following: e n t i t y ( Act , [ an , a c t i o n , [ type , putting down ] ] ) , o c c u r s ( Act , [ , End ] ) , h o l d s ( pose ( pr2 : ' p r 2 b a s e l i n k ' , Pose ) , End ) . Which corresponds to the question \Where did the robot stand at the end of put-down actions?".
Based on our model, we can also ask questions
about the goals of an action, for example, \What
motion phases are the subgoals of an action". For our
introductory example of a robot closing a drawer (see
Figure 1), the motion phases can be queried with a
query such as:
e n t i t y ( Act , [ an , a c t i o n , [ type , c l o s i n g a d r a w e r ] ,
[ part moved , [ an , o b j e c t , [ b a s e l i n k , HandBase ] ] ] ] ) ,
f i n d a l l (M, e n t i t y ( Act , [ sub motion , M] ) , Motions ) .
For a more detailed description of the question
answering system used here, please consult the system paper
written by Beetz et al. [
Episodic memories are very comprehensive and additional tools for inspection are required. For illustration, we pick one simple pick-and-place task performed by a robot and show how our visual analytics tools are used to get insights about manipulation activities and reasoning processes. Our goal is to provide tools for gathering data for learning algorithms, and to learn about the requirements for robots performing everyday activities. Clustering methods may be used, for example, to group actions based on their parameterizations, and to identify e.g. what kind of actions require two arms to be performed successfully, or what kind of actions require additional tools. Di erent components of our analytics framework will be described below.
Cognition-enabled plan frameworks, such as CRAM
[
With our visual analytics framework we want to go beyond showing just hierarchies and statistics. Each visual component is linked to the knowledge base which allows us to perform queries on the displayed data. To be speci c, the nodes in the action hierarchy can be selected by the user, and the user can ask queries about them such as getting the error type of an unsuccessful task, the time duration, etc. In addition, trajectories during actions can be queried and visualized. Having the experience data linked to the narrative of an activity further allows to correlate success of an action with e.g. the goal pose relative to the base.
For every episodic memory we can request a cooccurrence matrix between actions and errors which occurred during an activity. Figure 2 shows an error matrix for a pick and place activity. The rows and columns can be sorted by frequency to get quickly an overview which actions failed the most or which error type occurs the most. Referring to Figure 2, the matrix shows that most failed action was MovingToLocation due to collision.
We are also using the error matrix to extract action preconditions which were not considered during plan design. Currently we are extracting the preconditions manually. In the future we are planning to automatize this extraction so the robot can extend its action model by itself.
The matrix is also linked to the knowledge base, this allows us to query detailed information about the errors. For instance, for perception errors we can query which objects could not be perceived. Those queries can give us an overview e.g. for which objects the perception system might need to be improved.
MovingToLocation
BaseMovement
LookingAt VisualPerception
LookingFor
Reaching
MovingToOperate PickingUpAnObject
Retracting PuttingDown
LiftingAnArm OpeningAGripper LoweringAnArm
Pulling FetchAndDeliver
AcquireGrasp ClosingAGripper SettingAGripper Cognition-enabled plans require a signi cant amount of reasoning. We provide multiple visualization tools available to get insights about reasoning processes. Figure 3 shows a co-occurrence matrix with the action types (rows) and the reasoning questions (columns) which are asked during a pick and place action. This matrix gives an overview which reasoning tasks were performed the most and which tasks required the most reasoning. In our example, a signi cant amount of spatial and perception reasoning tasks were performed.
Our analytics framework serves additional statistics, such as depicted in Figure 4. The left pie chart shows the ratio between the frequency of reasoning tasks compared to actions. A high number of reasoning tasks indicates the robot performed a very abstract plan since it required a lot of reasoning to be able to execute it. The right pie chart in Figure 4 depicts an overall time usage between reasoning and action execution. Note that even though the general amount of reasoning tasks is signi cantly higher than the number of actions, the action execution requires the most time. This insight gives us the the opportunity to let the robot do more expensive reasoning in the future without extending the overall experiment runtime because we could run the reasoning in parallel during the action execution.
In this paper, we have introduced an approach for representing episodic memories of embodied agents performing manipulation tasks. The action model is inspired by a model from human psychology. Its representations are based on force dynamic events which are used to de ne semantics of action verbs. We have 86 14
Action Action 98 2
Reasoning shown that patterns of force events can be used to detect intentions, and what actions an embodied agent performed. The action model is coupled with experience data that stores control level information. We believe that collections of episodic memories are key for understanding how experiential knowledge about manipulation tasks can be generalized.