In this paper, we investigate the use of OpenAI ChatGPT to improve the natural language understanding of an End-User Development environment, called CAPIRCI, supporting users neither expert in computer programming nor expert in robotics to create programs for a collaborative robot. The integration of ChatGPT in CAPIRCI is studied to be transparent for the user, who will be allowed to check and correct that program description generated by ChatGPT, by interacting with an intuitive block-based interface, according to a Human-Centered Artificial Intelligence design approach.
Collaborative robots (also known as cobots) are recently being deployed to automate tasks in
several domains — from ofice work to manufacturing, from logistics to healthcare. They can
operate in the same space of human workers and collaborate with them to achieve shared goals.
Collaborative robots contribute to create the conditions for increasing production flexibility,
being more afordable, compact and easy-to-use than traditional industrial robots [
In [6, 7], we proposed CAPIRCI (Chat And Program Industrial Robots through Convenient Interaction), an End-User Development (EUD) [8, 9, 10] environment providing an intuitive and
https://daniela-fogli.unibs.it (D. Fogli) natural way to program pick-and-place tasks for a collaborative robot. Specifically, CAPIRCI encompasses a hybrid interaction style that merges two paradigms: natural language interaction with a chat-based interface, and visual interaction with a block-based graphic interface.
The final aim of CAPIRCI is not simply demonstrating that the proposed interaction style for EUD of robot tasks is feasible and easier than block-based programming alone, as shown in the experiment carried out in [7], but also promoting a way to foster gradual learning of robot programming on behalf of human workers, thus facilitating a smooth acquisition of computational fluency.
However, one of the main weaknesses of the CAPIRCI prototype presented in [7] was the computational power of the chat-based interface. Natural language processing (NLP) was implemented exploiting Python libraries, like the Standard CoreNLP and the NLTK (Natural Language ToolKit) package, and was based on the recognition of a finite set of noun domaindependent phrases. In this paper, we explore the use of OpenAI ChatGPT1 to improve the natural language understanding of our chat-based interface and show how ChatGPT could be of help in the case of robot programming.
The EUD feature of CAPIRCI is composed of 1) a chat environment where the system and the user interact through a simple guided natural language dialogue that leads to the definition of tasks for a collaborative robot, and 2) a visual environment where tasks can be composed through direct manipulation of specific types of blocks.
A robot task may include the specification of the objects to pick up, the actions that must be performed on the objects, and the locations where the objects must be put. Pick-and-place tasks may include repetitions if more than one object must be manipulated or the action must be carried out on the object several times. Conditional termination of tasks may depend on specific events. For instance, considering the scenario of flask manipulation in an analytical laboratory, Figure 1 shows a user-system dialogue in the chat environment aimed at defining a task for picking up 10 flasks and put them in a container after that each flask has been rotated twice, and where no specific event determines the end of task execution. The interaction with the chat always ends with a question to the user asking whether they would like to see the program in the visual environment. If the user answers afirmatively, they will have the possibility to verify the program created by means of the natural language dialogue in the visual environment, and possibly modify or extend it through drag-and-drop of predefined controls and functional blocks. Figure 2 shows the main parts of the visual environment. We denoted with (A) the part of the graphic interface where the user can find the libraries of components useful for task programming (Tasks — since tasks previously developed can be saved and re-used —, Controls, Events, Actions, Objects, and Locations). The library Controls is currently selected in the figure, and the blocks corresponding to the available control statements are shown at the right of the libraries; the user can select and drag-and-drop one of them in the working area denoted with (B) in the figure. In this working area, there is the visualization of the task just created with the chat. The two programming modes may accommodate diferent users’ attitudes: users 1https://openai.com/blog/chatgpt/ with advanced computational fluency may prefer interacting only with the visual environment, which supports the creation of complex tasks including nested loops and conditional statements; users with more limited computational fluency, instead, may create a first draft of the robot program with the chat, which appears more intuitive and easy to use, and then use the visual environment to refine and complete the draft program.
Each task created either with the chat environment or the visual environment is saved as an XML file in the Tasks library for future modifications or re-use in other robot tasks. The XML description of a task represents a formal specification of a robot program. To run the program on a real robot (e.g., the COBOTTA robot2 by DENSO WAVE Ltd.), a parser analyzes the XML document to identify the elements (objects to be manipulated, actions to be executed, locations where to put the objects, loops to be performed, etc.) and generates the corresponding Python code. This code is interpreted at run time, along with a few proprietary libraries, to perform the task on the real robot.
Analysing the features of CAPIRCI, one might argue that its chat environment could provide more powerful features, in order to allow all users to create complex tasks in a natural way, without resorting to the graphic interface. However, the visual representation of the program permits to check its correctness and completeness in an immediate way. Alternatively, the chat could generate directly the executable code corresponding to the robot task, but most of end users not expert in computer programming would be unable to understand and check it.
The idea of providing a natural language interface for robot programming has been investigated in literature for several years (e.g., [11, 12, 13]) and is currently gaining momentum thanks to the recent introduction of OpenAI ChatGPT, a pre-trained generative text model, 2https://www.densorobotics-europe.com/product-overview/products/colla borative-robots/cobotta which provides impressive interaction capabilities. For instance, in [14], the authors investigate ChatGPT as a potential versatile tool for robot programming, by first creating a high-level function library and then allowing ChatGPT to parse user requests and convert them to a logical sequence of function calls. However, this work assigns to the user a significant and critical role underlining that “the user stays on the loop to evaluate code output by ChatGPT, either through direct analysis or through simulation, and provides feedback to ChatGPT on the quality and safety of the output code” [14]. According to this approach, after a few iterations of user-ChatGPT dialogue, the final code can be deployed on the robot. The assumption here is that the user is able to understand the generated code, to assess its correctness, and to suggest possible modifications, if needed.
In our approach, we would like to keep a natural language dialogue to generate a first draft of the program for the robot and provide the user with the possibility of visualizing this draft program in a graphic interface, so to modify it, if necessary, in an intuitive manner, that is, with drag-and-drop and direct operation on the graphic visualization of the draft program. This is even more important when created programs must be executed on a robot, as the safety of the human and the environment, as well as of the robot itself, is at stake.
To this end, our idea is integrating ChatGPT features in CAPIRCI, by substituting the NLP Python libraries exploited in the first prototype and using ChatGPT to generate the XML intermediate description of robot tasks.
The basic idea for integrating the powerful NLP capabilities of ChatGPT in CAPIRCI was creating a new software layer able to acquire the user’s request in natural language defining a specific task for a collaborative robot and formulate the correct question to ChatGPT to obtain the XML file describing the robot task. The XML file must be syntactically structured according to the rules specified in CAPIRCI for its correct interpretation both by the parser functionalities that generate the graphic representation of tasks and by those ones able to generate the Python code for run time execution.
Preliminary tests have been performed with ChatGPT to assess its determinism in generating an answer to a request: when the input is partially specified or incomplete, thus not adequately limiting the solution space, ChatGPT generates several diferent answers, often completely wrong or not suitable to the request. Therefore, our layer should have provided ChatGPT not only with the user’s sentence and the generic request of generating an XML file describing a robot task, but also with a set of additional specifications to make the system converge toward a unique and correct solution.
After these preliminary tests, we derived that our layer must provide rigorous instructions to ChatGPT about the structure of the XML files. To this end, we decided to provide the system with a dataset of XML files created in previous experiments with CAPIRCI as training examples for learning the correct structure of an XML file describing a robot task, that is, learning the XML Schema Definition (XSD). Also in this case, since the dataset did not cover a complete suite of cases, the output files generated by ChatGPT resulted to be very diferent one another (non determinism) and sometimes inconsistent.
A third step was creating a request to ChatGPT that combines i) the XSD obtained in the previous step that was able to validate the highest percentage of XML documents, with ii) a set of constraints (still expressed in natural language) that better clarify the roles of tags and attributes to be used for generating the XML files describing the robot tasks. A convergent behavior and correct XML documents were obtained in this case, allowing us to delineate a new architecture of CAPIRCI, which integrates a new NLP layer with an adapter able to invoke ChatGPT APIs for user’s request interpretation (see Figure 3).
Figure 4 shows an example of request by the adapter to ChatGPT to interpret the user’s sentence “Pick up ten flasks, rotate each flask twice, and put them in the container” (corresponding to the dialogue with our previous chat shown in Figure 1). The adapter functionality is fundamental to limit the creativity and non determinism of ChatGPT.
With this approach the natural language interaction may become even more natural than that ofered by the original chat-based interface of CAPIRCI. In fact, users can express their requests with unique and articulated sentences without being involved in a rigid exchange of speech turns for gathering all information necessary to generate a complete task description.
In this paper, we have explored how to exploit the NLP capabilities of OpenAI ChatGPT to generate robot programs. Diferently from existing proposals (e.g., [ 14]), our aim was not to generate the final code for the collaborative robot starting from the user’s request, but rather to obtain an intermediate representation of the program that could be processed in our EUD environment CAPIRCI to provide a user, neither expert in programming nor in robotics, with the possibility of verifying the correctness of the generated program.
In this way, we allow the user to keep on expressing their requests for robot programming using an intuitive interaction through the formulation of sentences in natural language based on the concepts of the domain. On the other hand, we provide the user with the block-based representation of the robot program corresponding to their request, as was originally in CAPIRCI [6, 7], in order to assess its correctness and completeness, and tune it as needed, by simply manipulating the visual blocks.
In synthesis, the idea is to use the Artificial Intelligence features of ChatGPT as services for advanced natural language understanding, leaving the control on the final robot program description to the human, in a way that the human can find simple and intuitive. Indeed, the target users of CAPIRCI are not software programmers, but workers that, at some time during the usage of a collaborative robot, may need to program a new robot task to accomplish a specific work. In this regard, CAPIRCI can become a EUD environment with a high degree of EUDability [15] also for workers with a low level of computational fluency. In addition, computational fluency may be nurtured by the system helping users to gradually acquire the concepts of abstraction, decomposition, algorithm design, generalization, and evaluation.
The approach presented in this paper is very preliminary and has several limitations. The most severe one is that only the generation of XML descriptions of small and simple tasks for a restricted domain has been tested. Thus, further experimentation and tuning of the system are needed. Moreover, another type of system extension (still a EUD activity) was implemented in the original version of CAPIRCI [7]: it supported users to enrich the natural language and graphic blocks by defining new objects, actions, and locations to be used in the description of robot tasks; this was performed through the interaction with the graphical interface of CAPIRCI and image processing algorithms. We plan to integrate in the future the use of ChatGPT also for this EUD activity.