We present an approach to facilitate the design of Assisting Conversational Agents dedicated to the Function of Assistance to ordinary people interacting in Natural Language with assisting agents on Web 2.0 pages. For each new assisted RIA, an experiment is carried out to collect a specific corpus of textual questions. It is then analyzed to exhibit the specific linguistic entities of the application required by the generic skeleton of the agent. Using a real large-scale example of a cooperative music prototyping application, we study the feasibility and evaluate the cost-effectiveness of this approach.
1.1
Day by day, the Web 2.0 based on Rich Internet Applications (RIA), is growing as an
environment for communication, providing interaction facilities that support the
growth of virtual communities. Thus, the Web has become a rich and ideal way to
study the new so-called Distributed Collective Practices (DCP) [
Along with the rapid evolution of the Internet, the population of computer applications has evolved from the relatively small number of expert or corporate users to the huge amount of general public users who are mainly novices. This phenomenon is accentuated in the Distributed Collective Practices and it has been observed in the CODES environment. This is the reason why, we think the Function of Assistance, dedicated to novice users, is a key issue of the future Web 2.0.
Assisting users of computer applications has been a challenging task since personal
computers have started to be available to a broader audience of non-expert users at the
beginning of the 1980s. As a mere transposition of a paper-based documentation into
an electronic version has been shown to be not very efficient [
From a strictly computational point of view, the main qualities of a help system are
the precision and the completeness of its informational content. But as far as general
public users are concerned, it has been shown that ease of use is by far the primary
factor (otherwise the help system is merely left aside) since according to the
‘motivational paradox’ [
Key Issue: a Corpus-Based Approach to the Function of Assistance
A typical ACA is built upon a Natural Language processing chain (NLP-chain). There
are two main approaches to the design of dialogue-oriented NLP-chains:
a) Human/machine dialogue stand-alone systems, like Allen’s TRAINS [
In the context of Web 2.0 applications and services, where the cost of the assisting agent has to be small compared to the cost of the application itself, the three advantages mentioned above appear crucial. This is the reason why we have favored this approach to develop a generic assisting agent i.e. an agent that is supposed: a) to be easily ‘pluggable’ into new Web 2.0 applications; and b) to provide a NLP-chain skeleton quick and easy to adapt to the new application.
Therefore, the key issue to the filling of the NLP-chain skeleton is the elicitation of the specific linguistic entities that appear to be actually occurring in the users’ textual requests. For this purpose, we have been relying on a corpus collected on a group of novice users carrying out CODES basic tasks during an experimental session.
In Section 2.1, we present the architecture of our ACA and its NLP-chain skeleton. The rest of the paper is then dedicated to the experiment carried out with a large-scale Web 2.0 application (the CODES framework) in order to assess a) the integration of the agent into CODES; b) the feasibility of the corpus-based phase of adaptation; and c) the global cost-effectiveness. 2 2.1
In order to study the Function of Assistance in the context of RIA, we have developed
an experimental toolkit called DIVA, freely available for research and education
purposes at [
User « User’s textual request » . . .
NLP-chain Formalization phase
Lemmatization
Word sense association Interpretation phase
Semantic space rules 1 Semantic space rules …
Semantic space rules n « Textual answers »,
Gestural answers,
GUI events.
Assisting
Agent
Rule triggers Heuristics i reaction
Actions Session, Agent, User ruupnotnime models updates
User’s . .e.vents Browsing
Application GUI
Runtime Static definition as XML resources
Dynamic updating on user’s events
Application Agent
User
Session Symbolic Models
Following the guidelines stated in section 1.2, the NLP-chain of the DIVA toolkit is
based on a typical chatbot approach but with a much more sophisticated structure as
shown in figure 1 (middle–top):
1) The formalization phase: based on two sets of filtering rules applied sequentially:
- Syntactical level: first a typical chunking phase is applied, then words’ flexions are
transformed into their corresponding lemmas (root words);
- Word-sense association level: lemmas are then transformed into semantic
socalled synsets, as done in WordNet [
Given the proposed structure of the DIVA NLP-chain, the key point is now to evaluate its quick and easy adaptation to existing Web applications and services. This task is carried out in two steps: a) exhibiting the linguistic entities expressed and/or referred to by the users; and b) writing the formalization and interpretation rules according to these entities. In a first stage, several small Web pages (also available on the DIVA homepage) were developed to test this approach, with success in terms of cost-effectiveness. However, this work had to be improved on two main points: 1) By evaluating the procedure to a real large-size application (large-scale problem); 2) By defining a more systematic procedure to exhibit the application’s entities.
The CODES framework is a large-size Web 2.0 application dedicated to collaborative music prototyping on the Internet involving general public users, and hence, a good candidate for an experiment involving a large-scale problem. This experiment was carried out in two main steps: - The first step was to integrate the DIVA toolkit into the CODES framework (this was done quite easily, in less than a week); - The second step was to define an experimental protocol: a) to collect a corpus of assistance in the CODES environment, and b) to exhibit the linguistic entities occurring in the corpus (which was more time-consuming – see table 4).
The experimental protocol is described in the next section and the systematic procedure to exhibit the application’s entities is detailed with its results in section 3. 2.3
Objective. This experiment aims at collecting a small corpus of Natural Language Requests of Assistance uttered by novice users while interacting with the CODES framework to perform simple tasks. This corpus is meant to be manually analyzed and annotated to exhibit the entities required to build rules of the ACA (cf. section 3). Subjects. Experiment sessions involved 12 subjects (4 male / 8 female) between 23 and 35 years old, most of them being MSc and PhD students (from very diverse domains: architecture, law, cooking, acrobatic dance, urbanism, history, psychology and physics). To keep the homogeneity of the sample group and hence of the collected corpus, no subject had any background knowledge neither in computer science (beyond surfing the web) nor in music (theory or practice) and can thus be considered as novices in both domains.
Conditions. In this experiment, all subjects were told to ‘think aloud’ [
During this experiment, all subjects asked for assistance from the CODES expert. Hence, they were given the best possible help information and consequently all subjects achieved successfully all the tasks. It resulted in the constitution of a corpus of 115 Natural Language requests, acquired over a month. They have been collected in Portuguese, Spanish and English and all translated into English for further processing. Table 2 displays an excerpt of 30 oral utterances, transcribed off-session from the audio data. Using the recorded sessions, utterances have been associated with additional contextual information, thus enabling the analyst to resolve anaphora (e.g. ‘it’ pronoun ― lines 1 , 5, 6, 9…) and indexical items (e.g. ‘this’― line 15, 18). Even if few utterances aren’t assistance requests about the structure or the functioning of the application (e.g. line 20 expresses a subject preference), most actually are. 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 We have observed that assistance was often provided by answering a single question, the CODES expert’s answer being enough for the subject to be able to continue the task at hand. So each assistance turn corresponds to an independent pair question/answer and not to a real dialog (in the worst case, there sometimes was a ‘rephrasing dialog’ where some questions were asked twice or three times with some linguistic variations).
Subject Loquacity. The general data from the assistance turns related to the five tasks is represented in figure 4, with subjects sorted from the most loquacious (i.e. help-seeking) to the least loquacious. We can easily note that for each task, in the group of subjects, there is a large inter-subjects variation. It is also shown in figure 5a, with a range factor of 4 between Julia and Julieta. This may be proportional to the familiarity with others Web-based applications, not necessarily related to music. The Novice Effect. In figure 5b, we can see that whereas the five tasks were defined with increasing complexity, the total amount of turns indicates that while the subjects progress from task 1 to task 5 they tend to require less assistance. This is emphasized by the turn count of task 1 exhibiting a peak of assistance turns. If we analyze the interrelationships among the tasks, we can suppose that some questions related to one action performed, in relation to a given task, were not repeated when a similar action was performed in another task execution. This clearly shows a so-called ‘novice effect’ where assistance is mostly needed when the users enter the application, and tend to decrease even when the users are involved in more complex tasks. Once collected, the corpus was transcribed manually into a formal form. Table 3 presents some examples of transcriptions of users’ questions into formal requests. Note that deictic items (e.g. ‘it’, ‘there’ while pointing with the finger on screen) or session anaphoric items (e.g. ‘come back’ which requires registering the previous states) were registered by the WoZ operator so that they could be filled here within brackets (left column in table 3), making it possible for the transcription to be accurately completed. Can I put it back to the library? [it = SP] Can I try all the styles? How can I stop it? [it = SP] How one can do to put the sound pattern there [there = EA] How can I take off the broom? How can I deactivate the broom? How can I play the whole sequence? How can I come back? [previous = EA] What do you mean with 3 sound patterns? Should I listen to all of them? [all of them = SL] Should I drag and drop? [to put them=SP in the EA] Where is the editing area? Are the musical styles: rock, funk, jazz..? Etc.
CANI(Move(SP, SL)) CANI(Try(MUSIC) HOWTO(Stop(SP)) HOWTO(Put(SP, EA)) HOWTO(Quit(Broom)) HOWTO(Quit(Broom)) HOWTO(Play(MP)) HOWTO(Comeback(EA)) ASK(Meaning(SP)) SHOULDI(Listen(SL)) SHOULDI(Move(SP, EA)) ASK(Location(EA)) CHECK(Listof(MUSIC)) … We were able to exhibit three main types of entities in the subjects’ utterances, as described in the following list: Type Speech Acts Predicates
Description They express the mental position taken by the speaker about the propositional content of the utterance.
They are mainly linked to action verbs identifying the operations in the CODES GUI.
They are the elements categorized by the users and targeted through the referential expressions.
Speech Acts. For simplicity, we limited the speech acts to five subclasses1. Their definition, limited here to a shortened gloss and one example, is as follows: Speech Act HOWTO ASK CHECK SHOULDI CANI
Definition and example
The user asks how to achieve some function
e.g. “How can I listen to the sound pattern?”
The user asks the value of an attribute
e.g. “what do you mean with 3 sound patterns?”
The user checks for the confirmation of a proposition
e.g. “This does not correspond to this?”
The user asks for a suggestion so as to proceed
e.g. “Should I record before?”
The user ask for the possibility to perform an action
e.g. “Can I put them in any level I want?”
Number
5 classes
28 classes
10 classes
% corpus
21 %
20 %
18 %
14 %
11 %
Predicates. They were manually exhibited and classed into semantic classes, like in
Wordnet synsets [
Delete Meaning Possibility Etc.
Gloss Action of playing a sample of music.
Action of listening to a sample of music.
Action of moving an item into a place (e.g. by drag & drop actions).
Action of deleting an item (often represented by an icon).
The sense/purpose attached to a GUI object, a function name etc.
Logical predicate indicating that its argument can be activated.
… The distribution of the most occurring predicates is given in the figure 6.
References. Referential objects are ‘things’ that the users categorize ‘in their mind’ and refer to in their linguistic expressions. They can be actual GUI objects (buttons, browser, window…), screen areas and panels, icons representing application items like the musical prototypes (MP) or the sound patterns (SP); other significant elements referenced in the corpus are the “broom” button, numbers, and applicationspecific concepts like musical-styles or musical-instruments (for simplicity, they are all collected in a single class called MUSIC). Their notation, definition and corpus distribution is given in the following list: 3.4
As stated before, this experiment was done with two objectives in mind: a) to demonstrate that the structure of the resource definition files, associated with a particular application, could be revealed from the analysis of the data; b) that the task of analyzing and programming the resources files would be cost-effective. Section 3.3 has attempted to bring an answer to the first point so we can now add some qualitative remarks about the cost-effectiveness.
As shown in table 4, the total experiment was achieved in about a month time. Globally, even if not completely cost-effective, this can be considered encouraging compared to the development duration of classical dialogue systems which often amounts in months, even in years. One can see that the part dedicated to the experiment with the subjects took more than half the time, even with a small number of subjects. This is certainly the weak point of this approach which relies on ordinary people availability and WoZ techniques. In the future, an effort should be made to adapt this phase with automated Web-based procedures, using virtual characters instead of WoZ operators. This will allow to reach a large population of potential users and to automate the transcription phase. However, it will then be difficult to keep performing manually the annotation and exhibition steps, thus leading to a tradeoff between the loss of accuracy and the larger coverage of the linguistic phenomena. 4
We have presented an approach to facilitate the design of Assisting Conversational Agents dedicated to the Function of Assistance to ordinary people interacting in Natural Language with assisting agents on Web 2.0 pages. In this particular context, the key issue is the feasibility and the cost effectiveness of the adaptation of the assisting agent to new assisted Rich Internet Applications. Our approach is based on a) the collection of a specific corpus of textual requests with novice subjects asking help while performing predefined tasks; and b) the elicitation of the specific linguistic entities through an analysis of the collected data. 2 Actually, step 6 which is related to the integration of the exhibited entities within the agent rules is not detailed in this paper which focuses on the linguistic part.
In this paper, we have described an example of such an experiment with a real, large-scale, typical Web 2.0 application which shows that it is feasible and relatively cost effective to carry out the proposed procedure. The concluding discussion points some weak points among them the extent of the WoZ experiments and of the manual analysis: this will prompt future effort on the automation of this part of the procedure.