- Industrial robotics is a complex domain, not easily amenable to formalization using semantic technologies. It involves such disparate aspects of the real world as geometry, dynamics, constraint-satisfaction, planning and scheduling, real-time control, robot-robot and human-robot communication and, finally, intentions of the robot user. To represent so different kinds of knowledge is a challenge and the research on combining those topics is only in its infancy. This paper describes our attempts to combine descriptions of robot tasks using natural language together with their realizations using robot hardware involving force sensing, ultimately leading to a potential of learning new robot skills employing force-based assembly. We believe it is a novel approach opening possibilities of semantic anchoring for learning from demonstration.
Recent developments in robotics, artificial intelligence and cognitive science lead to bold predictions about the soonto-come robotization of all aspects of human life. Robots will help the elderly, perform mundane jobs no one wants, drive our cars, fill our refrigerators when needed, tirelessly rehabilitate patients in need of physical exercise, fight our wars, become our sex partners, etc., etc. Some draw even the conclusion that robots will take over Earth and will turn humans into obsolete pets.
However, when observing the development of the robotics field, we can realize that this perspective is rather far, far away. Service robots are clumsy and unskilled, no one trusts a robotized car, and production still relies on simple manipulators programmed in a classical manner by skilled engineers. Any attempt to instruct a robot to perform a concrete manufacturing task consists of person-weeks of work of skilled system integrator engineers, who take into account the geometrical layout of the workcell, all the objects involved, involving their geometry, physical properties and, last but not least, purpose of the task. It is the implicit knowledge that needs to be transferred into robot code, which makes this task so complex.
Use of semantic technologies is advocated since at least a decade. Unfortunately, industrial robotics is a complex domain, not easily amenable to formalization. It involves such disparate aspects of the real world as geometry, dynamics (including forceful interaction with the work objects), constraints, planning, scheduling, optimization, realtime control, robot-robot and human-robot communication and, finally, intentions of the robot user. To represent so different kinds of knowledge is a challenge and the research on combining those topics is only in its infancy.
In particular, we have devoted last years to understand and describe robotic assembly, including force-based operations (snap, drill, press, etc.), using machine-readable formalism expressing the semantics of possible robot actions. Without it there will be no possibility to create a meaningful reasoning leading from the task specification (what needs to be manufactured) to task synthesis (how can this be achieved using the available robot skills) and robust execution by the synthesized code for a particular architecture. Not to mention swift error handling in case of unexpected problems, portability of a robot skill from one robot to another, and learning of new skills.
In previous research, we have focused our attention on two
areas: interaction between the user and the robotic system,
preferably on the user’s conditions, e.g., using natural
language [
So far these kinds of systems are developed only in
research laboratories. Our own research is done in the context
of several EU projects, in particular ROSETTA, PRACE and
SMEROBOTICS, aiming at developing intelligent interactive
systems suitable for inexperienced users, such as SMEs.
Before they reach the factory floor though, they need to
be filled with sufficient production knowledge so that they
become useful. Knowledge acquisition is a bottleneck in
developing practical systems, as it can only happen while
the system is used, but it won’t be useful before it is done:
a classical chicken and egg problem. Therefore the only
viable solution is a learning system, capable of sharing
its experiences by storing them in a (possibly cloud-based)
knowledge base [
The work on standardization of robotics domain is already
quite well advanced. There exist ontologies for specific
domains, like service robotics or surgical robotics, and a
core ontology for robotics and automation (CORA) recently
standardized by IEEE [
Learning from demonstration leads to new problems in semantic anchoring of robot actions, as there is no obvious, apparent meaning in robot movements. Semantics may be either guessed, derived by inductive reasoning, or attributed post factum by humans via some form of annotating. In particular, force-based assembly is problematic as quite often the difference between success and failure depends on a particular profile of the force signal. So far, this issue has been approached using sensor fusion techniques, without direct support from semantics. Our work attempts to remedy this situation, introducing natural language into the picture and letting assembly to be not only detected via sensor readings, but also by being simultaneously told about.
In the domain of service robotics, there are some
interesting frameworks for representation of household tasks
and environments. KnowRob [
We are interested in integrating low-level statistical task
representations taken from demonstrations. Such tasks can
be represented by a trajectory or force profile. The
trajectories can be extracted from the demonstration by first
applying segmentation algorithms and then parameterizing
each segment as a trajectory. Niekum et al. [
One way to annotate objects and actions is to describe
them using natural language. Matuszek et al. [
Please note that our understanding of the term multimodal
semantics differs from the one quite commonly encountered
in literature, see e.g. [
The focus of current work is to semantically annotate task demonstrations to enable reuse and reasoning. This involves annotating log data with quantities, units, and task states. The logs can then be used to identify force/torque and position constraints and application specific parameter values (positions, velocity, stiffness, etc.). The demonstrations are use to segment the task in different sub-skills and extract parameters for each skill. One approach is to describe the trajectory of the sub-skills using DMPs and then parameterize the primitives and describe them with for example skill type, preconditions, and postconditions.
As an example, when demonstrating picking and placing of an object, the task can be segmented into different sub-skills. First the robot approaches the object, opens the gripper, moves into pick position, closes the gripper, retracts from the surface moves to the place position (perhaps using via positions as well), positions the object correctly, releases it and retracts. Each segment can be described using a trajectory (e.g., a DMP) in some reference frame together with a gripper state. Multiple demonstrations can be used for each sub-skill in order to detect which for example relevant reference frame and allowed gripping poses. To enable reuse, we are working on annotating the segments with initial allowed start positions and gripper state, a skill type and postconditions. This will allow the planner to add required actions before or after the skill and add errorhandling procedures to the task (e.g., if the robot drops the object when transporting it, the object should be localized and picked up again). The skill also has to be parameterized so that it can be initialized correctly, for example, specifying controller, reference frames, velocity values.
Another example is force-controlled assembly. The force
data is not used for sensor fusion, it is used to control
the motions of the robot and signals failure or success
of the assembly. In a snapfit assembly skill, where a two
plastic pieces, a switch and a box shown in Fig. 1, are
”snapped” together, the force signature indicates whether the
snap occurred or not. In previous work [
In another assembly, a rectangular metal plate (a shield can) is inserted on a printed circuit board (PCB). The PCB is attached in a fixture, which is attached to a force sensor. The assembly starts by tilting the shield can above the PCB (see Fig. 5), moving down until the corner touches the board. Then, the robot attaches one corner of the plate to a corner on the PCB and rotates the plate into place. The rotation is first carried out around the xy-plane of the PCB until either the long or the short side of the rectangle touches the PCB, then the last side has to be rotated into place. That is, if the longer side of the rectangle is parallel with the x-axis and the rotation around a xy-vector from the initial tilted position will align it with the PCB, the execution will branch into a rotation around the x-axis until the short side is aligned with the PCB. Otherwise, the rotation will align the short side first, as seen in Fig. 6.
To lower the threshold for the user, we want to use natural
language dialogues to describe the demonstration and extend
the task. Together with the parameterized demonstrations,
this will allow the user to use high-level structures such as
loops and if-then-else statements, which are easily described
using language but tedious or difficult to describe using
demonstrations only. In our current system, the user can
instruct the robot using unstructured text or dictate the task
using Google dictation tools. An example instruction is
displayed in Fig. 7. All parameters have default values, which
makes the high-level nominal task easy and fast to generate
from text. The programming interface use language specific
statistical tools to extract the semantics of the sentences,
then a rule-based mapping to robot skills and world objects.
At the moment, English and Swedish [
The immediate future work involves investigating how to teach pre-and postconditions for skills learned from demonstration, to enable online reasoning. These conditions need to be anchored in sensor readings. Inductive inference is one possibility; another is to use mixed-initiative dialogue with the user, asking for guidance or confirmation, yet another is to introduce some annotation tool to be used simultaneously with the learning procedure.
It is desirable to have natural language support on all levels in the system. At the moment, we only support task instruction, but we also want to be able to describe the world and connect the perceived objects and situations to (new) semantic symbols. E.g., saying ”This is a nut” after teaching the camera system to recognize an object, or describing a pallet as ”empty”. At the moment, the robot is a passive participant in the dialogue, only reacting on commands from the human. When interacting with non-expert users, the robot should ask questions and come with suggestions on what to do.
The next step is to introduce the possibility of extending the robot knowledge by adding new concepts to the semantic hierarchy. This is a more complex task than the previous one, as it involves inducing relations with existing concepts and proper placing of the new symbol in the IsA hierarchy.
Yet another interesting problem is to reason about “synonyms” among robot programs, i.e. syntactically different structures or programs leading to the same effect. A simple example is a “localize and pick” task that may use different kinds of sensors to localize an object, while the goal (of picking the object from its current location) is achieved irrespectively of which concrete sensor is used. How to teach the system that two skills are equivalent in such (or some other) sense? What needs to be told? What kind of reasoning performed?
Representing knowledge about industrial processes involving semantically-capable robots is a challenge leading to fascinating questions. We are quite sure we will have a lot to do in years to come.
The research leading to these results has received partial funding from the European Union’s seventh framework program under grant agreement No. 287787 (project SMErobotics) and from the European Union’s H2020 program under grant agreement No. 644938 (project SARAFun).