Industrial Priorities for Cognitive Robotics David Vernon† Markus Vincze Carnegie Mellon University Africa Technische Universität Wien Rwanda Austria Email: vernon@cmu.edu Email: vincze@acin.tuwien.ac.at Abstract—We present the results of a survey of industrial three parameters: safety for people, safety for equipment, and developers to determine what they and their customers require safety of the robot system. The rules may change depending from a cognitive robot. These are cast as a series of eleven on the environment and a cognitive robot will not exceed functional abilities: the limits of safe operation. The limits may be application 1) Safe, reliable, transparent operation. 2) High-level instruction and context-aware task execution. specific, e.g., the robot should not deviate further than a given 3) Knowledge acquisition and generalization. specification/distance/etc. A cognitive robot will use this type 4) Adaptive planning. of knowledge to act responsibly and will ask for assistance 5) Personalized interaction. when necessary (e.g. before it encounters difficulties). In 6) Self-assessment. particular, in emergency situations, the robot will stop all tasks 7) Learning from demonstration. 8) Evaluating the safety of actions. to follow some emergency procedure. Ideally, if the user is 9) Development and self-optimization. deliberately trying to misuse the robot, e.g. programming it to 10) Knowledge transfer. assist with some unethical task, a cognitive robot will cease 11) Communicating intentions and collaborative action. operation. I. I NDUSTRIAL R EQUIREMENTS B. High-level instruction and context-aware task execution While cognitive robotics is still an evolving discipline and much research remains to be done, we nevertheless need to Cognitive robots will be given tasks using high-level in- have a clear idea of what cognitive robots will be able to structions and they will factor in contextual constraints that do if they are to be useful to industrial developers and end are specific to the application scenario when carrying out users. The RockEU2 project canvassed the views of thirteen these tasks, determining for themselves the priority of possible developers to find out what they and their customers want. actions in case of competing or conflicting requirements. The results of this survey follow, cast as a series of eleven Goals and tasks will be expressed using high-level instruc- functional abilities. tions that will exploit the robots contextual knowledge of the task. This will allow the robot to pre-select the information A. Safe, reliable, transparent operation that is important to effectively carry out the task. The goals Cognitive robots will be able to operate reliably and safely will reflect the users perspective. This means that all skills around humans and they will be able to explain the decisions which implicitly define the goals are tightly linked to real- they make, the actions they have taken, and the actions they are world needs and to the solution of specific problems, e.g., “get about to take. A cognitive robot will help people and prioritize me a hammer”. The following guidelines will apply. their safety. Only reliable behaviour will build trust. It will • Instructions will use natural language and gestures to explain decisions, i.e. why it acted the way it did. This is specify the goals. essential if the human is to develop a sense of trust in the • Natural language will be relatively abstract but will be robot. grounded in the codified organisational rules, regulations, A cognitive robot will have limited autonomy to set interme- and behavioural guidelines that apply to a given ap- diate goals to when carrying out tasks set by users. However, plication environment. This grounding means that each in all cases it defers to the users preferences, apart from abstract instruction is heavily loaded with constraints some exceptional circumstances, e.g. people with dementia which should make it easier for the robot to understand can interact in unpredictable ways and the robot will be able and perform the task effectively. to recognize these situations and adapt in some appropriate • The goals should be specified in a formalised and struc- manner. tured way, where the designer defines them well and can The freedom to act autonomously will have formal bound- verify them. For example, teach the robot the environment aries and the rules of engagement will be set on the basis of it is working in, follow a described route to reach each of † Much of the work described in this paper was conducted while the author the target locations and reach these positions to carry out was at the University of Skövde, Sweden. This research was funded by the the task. These clearly-specified tasks are tightly coupled European Commission under grant agreement No: 688441, RockEU2. with risks and costs, e.g. of incorrect execution. Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 6 • It should be possible for the robot to be given goals in C. Knowledge acquisition and generalization non-specific terms (e.g. assist in alleviating the symptoms Cognitive robots will continuously acquire new knowledge of dementia), guidelines on acceptable behaviour (or and generalize that knowledge so that they can undertake new action policies), and relevant constraints, leaving it to the tasks by generating novel action policies based on their history robot to identify the sub-goals that are needed to achieve of decisions. This will allow the rigor and level of detail with these ultimate goals. which a human expresses the task specification to be relaxed • A cognitive robot will learn ways of measuring the on future occasions. success of outcomes for the objectives that have been A cognitive robot will build and exploit experience so set, e.g., creating a metric such as the owners satisfaction that its decisions incorporate current and long term data. related not only to the directly specified objective but also For example, route planning in a factory, hospital, or hotel the manner in which the job was done). It should be learn should take into account the history of rooms and previous from these metrics. paths taken, or it might take another look to overcome high A cognitive robot will consider the contextual constraints uncertainty. In general, the robot will overcome uncertainty in that are specific to the application scenario. It will determine a principled manner. the priority of potential actions, e.g., in case of competing or A cognitive robot will generalize knowledge to new task by conflicting needs. understanding the context of a novel task and extrapolating For example, the robot might know the procedure to be from previous experience. For example, a care-giving robot followed but the locations to be visited or the objects to be will reuse knowledge of a rehabilitation exercise, customizing manipulated need to be specified (or vice versa). For example, it to another person. A welding robot will weld a new instance when an automated harvester encounters a bale of straw, it of a family of parts. In general, a cognitive robot will extract can deal with it as an obstacle or something to be harvested, useful meaning from an interaction for a future and more depending on the current task. For example, the robot might general use, with the same or another user. This may extend engage in spoken interaction with older adults until the goal is to learn cultural preferences and social norms. communicated unambiguously, using context to disambiguate For example, in a domestic environment, a cognitive robot the message and allow for the difficulties in dealing with will learn how to do simple household tasks, e.g. how to grasp different accents, imprecise speech, and poor articulation. different objects and them bring to a person that wants them. This will be continuously extended, allowing the robot to do A cognitive robot will know what is normal, i.e. expected, more complex things, including cooking. behaviour (possibly based on documented rules or practices) and it will be able to detect anomalous behaviour and then D. Adaptive planning take appropriate action. Cognitive robots will be able to anticipate events and The following guidelines will apply. prepare for them in advance. They will be able to cope • It will be possible to pre-load knowledge about the with unforeseen situations, recognizing and handling errors, robots purpose and its operating environment, including gracefully and effectively. This will also allow them to handle any rules or constraints that apply to behaviour in that flexible objects or living creatures. environment. A cognitive robot will be able to recognize that circum- • It will be possible to utilize domain-specific skill pools stances have changed to avoid situations where progress is (e.g. from shared databases) so that the robot is pre- impossible. It will also be able to recognize errors and recover. configured to accomplish basic tasks without having to This may include retrying with a slightly different strategy. resort to learning or development. The learning process will be fast, ideally learning from each • The robot will continually improve its skills (within limits error. of the goals and safety, see above) and share these with A cognitive robot will be able to learn how to handle other robots. errors, how to react to situations where, e.g., a human is doing • The robot might assist the user by proposing goals from something unexpected or parts are located in an unexpected what it understood and the user makes the final selection. place. A cognitive robot will be able to anticipate events and The level of detail in the description required by a cognitive compensate for future conditions. For example, an automated robot will decrease over time as the robot gains experience, combine harvester will be able to apply a pre-emptive increase in the same way as someone new on the job is given very of power to compensate for the demands caused when an area explicit instructions at first and less explicit instructions later of high yield is encountered. on. One should need to demonstrate only the novel parts of A cognitive robot will be able to learn about the environ- the task, e.g., pouring liquid in a container, but not the entire ment it is in and modify the its current information accord- process. ingly. That is, it will adapt to changes in the environment, It will be possible to instruct the robot off-line if there is no verifying that the environment matches with what is known, access to the physical site; e.g., using a simulation tool, with or there is a change and updates. This may require an update the robot then being deployed in the real scenario. of the task but only after asking the user. Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 7 A cognitive robot will be able to manipulate flexible or live If a cognitive robot is asked to perform a certain task, it objects, e.g. living creatures such as laboratory mice. To do will be able to say whether it can do it or not. It will detect so means that the robot must be able to construct a model of when something is not working and will be able to ask for their behaviour and adapt its actions as required, continually help. refining the model. A cognitive robot will assess the quality of its decisions and apply some level of discrimination in the task at hand, E. Personalized interaction e.g. being selective in its choice of fruit to harvest. Cognitive robots will personalize their interactions with humans, adapting their behaviour and interaction policy to G. Learning from demonstration the users preferences, needs, and emotional or psychological state. This personalization will include an understanding of Cognitive robots will be able to learn new actions from the person’s preferences for the degree of force used when demonstration by humans and they will be able to link interacting with the robot. A cognitive robot will be able to this learned knowledge to previously acquired knowledge of adapt its behaviour and interaction policy to accommodate related tasks and entities. the user’s preferences, needs, and emotional state. It will Instructions will be communicated by demonstration, learn the personal preferences of the person with whom it through examples, including showing the robot the final re- is interacting. For example, an autonomous car will learn the sults, with the robot being able to merge prior know-how and preferred driving style of the owner and adopt that style to knowledge with learning by demonstration. Some of this prior engender trust. knowledge should be extracted from codified organisational A cognitive robot will understand nuances in tone to learn a rules, regulations, and behavioural guidelines. person’s voice, detecting signs of stress so that it can react to it The situation is analogous to training an intern or an and review what it is doing. In the particular case of interaction apprentice: a trainer might ask“Has someone shown you how with older adults, the robot will be able to understand gestures to do this? No? Okay, Ill show you how to do three, then you to help disambiguate words. do 100 to practice (and to throw away afterwards). If you get A cognitive robot will able to extrapolate what has been stuck on one, call me, and Ill show you how to solve that taught to other situations. For example, it might remember problem”. that the user has certain preferences (e.g. to be served tea in A cognitive robot will learn and adapt the parameters to the morning) and the robot will remember that preference. achieve the task. Today in the assembly of components, often However, the robot will not allow these learned preferences to robot assembly is not robotized because it requires too much over-ride critical actions policies. engineering and it is too difficult for robots because it is based In cases where showing the robot what to do involves on traditional programming, tuning and frequent re-tuning of physical contact between the user and the robot, the robot parameters. will be able to learn the dynamics of the user, i.e. his or her Teaching will exploit natural language, gaze and pointing personal preferred use of forces when interacting with objects gestures, and by showing the robot what to do and helping it in the environment. when necessary. A cognitive robot will be able to the psychological state of Actions will be expressed in high-level abstract terms, like a user, e.g. based on the facial expressions, gestures, actions, a recipe, ideally by talking to it. For example, “go to hall 5 movements. Based on this, it will be able to determine what from hall 2 and pick up the hammer” or “open the valve”. they need by cross-referencing that with knowledge of the When being taught, the robot should be anticipating what persons history. you are trying to teach it so that it predicts what you want it A cognitive robot will be able to make decisions from a to do and then tries to do it effectively. large body of observed data, thereby assisting people who It will be possible to provide direct support for the robot, typically make decisions based on learned heuristic knowledge switching fluidly between full autonomy, partial autonomy, or but without a quantitative basis for this decision-making. For manual control. example, there is a need to provide farmers with a fact- based quantitative decision-making framework. A cognitive H. Evaluating the safety of actions robot or machine would observe the physical environment and the farmer and provide a sound bases for making improved When they learn a new action, cognitive robots will take decisions. steps to verify the safety of carrying out this action. If a robot learns new action, it will be difficult to certify the new action. F. Self-assessment The process of generating a new action will involve interaction Cognitive robots will be able to reason about their own capa- with the world and that may already be harmful. So, when bilities, being able to determine whether they can accomplish learning a new action, there needs to be a step to verify the a given task. If they detect something is not working, they will safety of carrying out this action. For example, showing a new be able to ask for help. They will be able to assess the quality action plus defining safety and success such that the robot can of their decisions. check if it achieved success. Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 8 I. Development and self-optimization intelligence, autonomous systems, and cybernetics, among Cognitive robots will develop and self-optimize, learning in others — that can be deployed to satisfy these requirements an open-ended manner from their own actions and those of in practical robots. It remains to complete this exercise. others (humans or other robots), continually improving their abilities. A cognitive robot will be able to use what it has learned to determine possible ways to improve its performance, e.g. through internal simulation at times when the robot is not working on a given task. It will also be able to learn from its mistakes, e.g., breaking china but learning from the effect of the action. A cognitive robot will learn to optimize the actions it performs (e.g. doing something faster) within the certified limits of safety and without increasing the risk of failure and associated costs. J. Knowledge transfer Cognitive robots will be able to transfer knowledge to other robots, even those having a different physical, kinematic, and dynamic configurations and they will be able to operate seamlessly in an environment that is configured as an internet of things (IoT). A cognitive robot will be a crucial component of cyber- physical systems where the robot can be used, for example, as a way of collecting data from large experiments. K. Communicating intentions and collaborative action Cognitive robots will be able to communicate their inten- tions to people around them and, vice versa, they will be able to infer the intention of others, i.e. understanding what someone is doing and anticipating what they are about to do. Ultimately, Cognitive robots will be able to collaborate with people on some joint task with a minimal amount of instruction. The need for people around a cognitive robot to be able to anticipate the robots actions is important because, if cognitive robots are to be deployed successfully, people need to believe the robot is trustworthy. A cognitive robot will be able to interact with people, collaborating with them on some joint task. This implies that the robot has an ability to understand what the person is doing and infer their intentions. II. C ONCLUSION Establishing functional requirements is an essential pre- requisite to developing useful systems. This is as true of cognitive robotics as it is for any other domain of information and communication technology. However, the effort to give robots a capacity for cognition is made more difficult by the fact that cognitive science, as a discipline in its own right, does not yet have many established normative models that lend themselves to realization in well-engineered systems. The goal of the work described in this short paper is to re- assert the priority of user requirements in the specification of cognitive robot systems. The motivation underpinning this goal is that, having identified these requirements, we can then proceed to determine the scientific and technological tools and techniques — drawn from the disciplines of artificial Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 9