While in the manufacturing industry robots do the majority of the assembly tasks, robotics process automation, where software robots are taking over repetitive tasks from humans have been introduced only recently. Many routine tasks continue to be executed without adequate assistance from tools that would be in reach of the current technical capabilities of AI. Using the example of taking meeting minutes, the paper presents some intermediate results of the capabilities and problems of currently available natural language processing systems to automatically record meeting minutes. It further highlights the potential of optimizing the allocation of tasks between humans and machines to take the particular strengths and weaknesses of both into account. In order to combine the functionality of supervised and unsupervised machine learning with rule-based AI or traditionally programmed software components, the capabilities of AI-based system actors need to be incorporated into the system design process as early as possible. Treating AI as actors enables a more effective allocation of tasks upfront, which makes it easier to come up with a hybrid workplace scenario where AI can support humans in doing their work more efficiently.
Most physical goods such as cars are predominantly manufactured by industrial robots. The international federation of robotics states that the worldwide robot density is currently 74 robot units per 10’000 employees (IFR, 2018). The allocation of this workforce to blue-collar and white-collar workers is quite different. The automation of tasks from blue-collar workers in manufacturing sites is still much more common than the automation for document centric tasks of white-collar workers in enterprise backoffices. The potential of machine learning and knowledge engineering is huge and a broad variety of different products and services enabled by AI are on the market to support humans with office tasks. The solutions are mostly isolated silo-type of applications and far away from being seamlessly integrated into the white-collar business processes. However, in some business domains such as banking or insurance companies, new technologies such as robotics process automation or the usage of customer facing chat bots gained momentum.
Robotics process automation is increasingly used to automate tedious repetitive tasks where humans currently transfer information from one application front end to the other. Especially in cases, where the backend integration of two systems is too cumbersome or takes too much implementation effort, RPA can be setup to quickly take over. Although, the complexity and stability of the resulting IT software architectures might be questioned, rule-based AI and RPA is increasingly used for process automation. In consequence, there is a fortunate side effect from regulatory requirements (e.g. implementing the four eyes principle, clarify responsibilities in case of failures) resulting in an increasing pressure to establish explicit guidelines for the supervision and governance of AI-driven processes in enterprises and the collaboration between humans and software robots in general.
Although the current potential of AI in general and NLP
(natural language processing) in particular would allow for
more human centricity, we still largely rely on constraints
from the past
Creating meeting minutes consumes time, money and
resources. An average Fortune 500 company with around
50000 employees spends 5 Million USD annually on the
creation of meeting minutes
Meetings are very important planning and coordination vehicles for organizations in order to mutually communicate and track status information to meet the overall business goals. Meetings that bind n resources simultaneously are ntimes more expensive than individual work. In many cases, at least some of the participants consider it as waste of time, especially if the meetings are too long or unstructured. On the other hand, structured meetings are real time-savers if the right people meet at the right time for the right outcome, and helps to reach a certain objective. In these cases, meetings are very valuable for organizations and need appropriate documentation for later recall and information retrieval. The sooner meeting minutes are available with the relevant information, decisions and action items, the better participants and absent co-workers can start working on the action items. Thus, early distribution of the meeting minutes can boost productivity.
However, recording meeting minutes is very challenging during the meetings as well as time-consuming for later rework and consolidation. While recording, it is difficult to follow the conversation flow because participants speak at a greater pace than it is possible to take notes. Moreover, the minute taker is absorbed and cannot actively participate in the meeting. All of these aspects tend to lead to inaccurate, incomplete and inconsistent meeting minutes. That said, automated recording and writing of meeting minutes would be of great value to enterprises and leads to the following research questions: • RQ1: What problems can be identified in creating meeting minutes? • RQ2: What are the requirements for an information system to create meeting minutes? • RQ3: To what extent can a speech recognition system support the speech to text transcription so that it still contains the relevant information from the meeting? • RQ4: To what extent does a speech recognizer separate multiple voices in a meeting? • RQ5: To what extent does a particular NLP component extract information from the speech-to-text transcript?
As a preliminary result from the master thesis
It not very surprising, that it is currently not possible to generate meeting minutes solely based on AI. However, the task could be decomposed into subtasks that can be allocated to humans and machines in a cobot-like scenario. Both parties could solve those parts of the problem where they are most capable. In the case of taking meeting minutes, humans that manually take notes struggle with the speed of typing the sentences while AI based STT conversion would outperform humans by far. On the other hand, speaker recognition is not a problem for humans while AI components are still inaccurate and fail in our preferred application scenario.
Why current applications are not designed using the potential of both? How could novel hybrid approaches be stimulated upfront? In many cases, requirements engineering and high-level specification of system design start with a UML use case diagrams where all actors and use cases are identified. Even in this early design phase, AI needs to be taken into account. Adding additional AI-system actors in UML use case diagrams would best represent the active role that AI can have and be comparable to the role of humans, such as shown in figure 1.
In consequence, additional AI based system actors lead to additional swim lanes in UML activity diagrams, such as shown in figure 2. This allows allocating the different activities explicitly to the different swim lanes and afore-mentioned AI-system actors are responsible to execute their activities. By designing systems in such a way, the mutual collaboration is stimulated and the “AI-first” scenario, which is said to be the successors of “cloud-first” and “mobile-first” application architectures, comes into play in a hybrid workplace with an AI-augmented system design upfront. In such a hybrid system design, one can better focus on the different strengths and weaknesses of human and AI actors in order to improve the effectiveness of current IT applications.
In most scenarios, where AI tools are used today, the
functionality is embedded in silo-type of applications. Data
scientists are using the functionality provided by the
graphical user interface (GUI) of their business intelligence
tools. In the case of solving data classification problems,
experts train the classifiers with the help of tools, where data
sets are imported in order to train and apply the models.
Knowledge engineers are manually building up ontologies
and rule-based semantic models for specific business
domains, which are executed in their workbenches. Such
scenarios are time consuming and resource intensive and it
would be a relief to delegate the knowledge base
construction process (KBC) to a deep learning system
Although many AI services are at hand that can be called by appropriate APIs, they have to be considered as “blackbox” logic and are not suitable to combine with company internal business logic, traditional software components (SC) and software design (SD) in cases where the data should stay on premises. It should be the goal to facilitate a seamless way of combining the different components more interactively, and to design more hybrid systems, where the strengths and weaknesses of humans and AI can be allocated more effectively.
In the case of creating meeting minutes, one should
construct a GUI, which makes it possible to enable AI
actors. For cobots in the manufacturing industry, different
methods have been developed to plan the sharing of tasks
The number of RPAs, digital agents and software cobots is still way behind the number of physical robots and cobots in the manufacturing industry. Nevertheless, the role of AI in the digitalization process is increasing and the different business scenarios, where AI algorithms reach from cancer diagnostics in health care over fraud detection in banking to voice-based digital assistants in the consuming sector. However, AI algorithms are rarely seen as active participants in most use cases. The focus of the system design should be leveraged from specifying the requirements of the components at design-time and include the specification of the business scenario at run-time, and how humans can delegate tasks to AI actors. Furthermore, not only the ability to act but also the ability to learn are new features of software components which definitely have an impact on the design of the different run-time business scenarios. Treating AI as system actors can be visualized at the early design phase with UML use case diagrams and changes the way how HCI is perceived. An AI-augmented system design is also visible in UML activity diagrams, where the activities are allocated according to the strengths and weaknesses of the different actors.
In many cases, the current distribution of tasks between humans and AI actors can be optimized, as can be seen in the example of taking meeting minutes. Although an application that autonomously can take accurate meeting minutes is out of reach with the current technology at hand, a hybrid scenario with a GUI that supports the collaboration with a software cobot, tedious tasks such as STT transcription, which is one of the mature components of NLP, could easily be delegated to an AI actor. Even more, the interactions on the GUI could be used as supervised learning for speaker recognition and built into the application itself. Rule-based systems could be established that help to distinguish the different formats of meeting minutes, that could be necessary for different meeting types such as decision meetings, brainstorming sessions or even specialized meetings for a scrum team. In cases where the quality of the speaker recognition is currently insufficient, the system could learn it over time if appropriate AI actors are incorporated at design-time in order to learn during runtime.
Compared to blue-collar workers, where the majority of repetitive manufacturing tasks are delegated to robots, many white-collar workers still lack appropriate tool support where at least some of the tasks can be delegated to machines and a more optimized cooperative workplace with humans and AI enabled system actors can be designed. As demonstrated with the example of taking meeting minutes, humans have difficulties with the speed of the utterance while currently available AI based STT transcription systems are much more accurate. On the other hand, speech separation is easy for humans, but none of the two speech recognizers came even close to the actual speech segments, which confirms the well-known “cocktail party” problem. In conclusion, more efficient systems could be designed that take the particular strengths and weaknesses of humans and AI-based software components into account. By starting the system design with use cases where the different AI based components are taken into account as system actors, the capabilities of rule-based AI as well as machine learning can be taken into account explicitly and up-front. Later during the systems design, the activities can be allocated to the most appropriate system actors and the lanes, and the activity diagram can be used as design methodology to optimize the interactive workplace of humans and cobots.
In some application areas such as autonomous driving cars, AI has already the role of an assistive technology. Although AI has the potential to replace human drivers in the long run, the current application scenario is hybrid and consists of a mutual collaboration between human and AI actors. Other than the Turing test, where the focus is towards building a machine that cannot be distinguished from a human being, the goal should be to build cobots with capabilities that can complement those of humans. Building systems that combine the capabilities of traditional software components, knowledge engineering and machine-learning components more seamlessly can help reshaping the traditional HCI in a way where humans can benefit the most. Humans can focus on valuable activities they can accomplish better than machines and benefit from delegating tedious tasks to machines where they are superior.