=Paper=
{{Paper
|id=Vol-2390/PaperD2
|storemode=property
|title=Chatbots' Greetings to Human-Computer Communication
|pdfUrl=https://ceur-ws.org/Vol-2390/PaperD2.pdf
|volume=Vol-2390
|authors=Maria João Pereira,Pedro Fialho,Luísa Coheur,Ricardo Ribeiro
}}
==Chatbots' Greetings to Human-Computer Communication==
https://ceur-ws.org/Vol-2390/PaperD2.pdf
Chatbots’ Greetings to Human-Computer Communication
Maria João Pereira1 , Luı́sa Coheur1,2 , Pedro Fialho1 , Ricardo Ribeiro1,3
1
INESC-ID Lisboa, 2 IST, Universidade de Lisboa, 3 Instituto Universitário de Lisboa (ISCTE-IUL)
Rua Alvez Redol, 9, 1000-029 Lisboa, Portugal
mjoao.madeiras@gmail.com, luisa.coheur@inesc-id.pt, peter.fialho@gmail.com, ricardo.ribeiro@inesc-id.pt
Abstract
In the last years, chatbots have gained new attention, due to the interest showed by widely known personalities and companies. The
concept is broad, and, in this paper we target the work developed by the (old) community that is typically associated with chatbot’s
competitions. In our opinion, they contribute with very interesting know-how, but specially with large-scale corpora, gathered by
interactions with real people, an invaluable resource considering the renewed interest in Deep Nets.
Keywords: natural language interfaces, agent-based interaction, intelligent agents, interaction design
1. Introduction of references to web pages in this paper) makes it difficult
Chatbots are currently a hot-topic, both for industry and to uncover the technology and the real possibilities behind
academia (Dale, 2016; Følstad and Brandtzæg, 2017). them. In this paper, we unveil the main contributions of
There are many platforms to help developing such systems, this community, as we believe that this line of work can
and the number of new chatbots continues to increase at bring important insights to the human-machine communi-
a dizzying pace. Pandorabots hosting service1 declares to cation field, as some of them contribute with large amounts
have more than 225,000 botmasters (people in charge of of data gathered during their interactions with the crowd,
creating/maintaining a chatbot), which have built more than which could be used by current data-driven chatbots (e.g.,
280,000 chatbots, resulting in more than 3 billion interac- (Li et al., 2016; Vinyals and Le, 2015)). As we will see,
tions (numbers collected in July 2017). On the academia these chatbots range from “simpler” ones, based on pre-
side, since 2016, at least four workshops were dedicated to written pattern-matching templates, exploiting large stores
chatbots (defined as non goal-oriented dialogue systems), of prepared small talk responses, to more complex architec-
which have been co-located with well-stablished confer- tures, based on some sort of learning process. Finally, we
ences2 ; also, several works point how chatbots could be will see that concepts/tricks introduced by some chatbots
used to in learning environments (e.g., (Bibauw et al., 2019) often result in a more solid contribution to the “illusion of
and (Fialho et al., 2013)). intelligence” than the involved models.4 This document is
Although the current definition of chatbot is broader that organised as follows: in Section 2. we present a brief histor-
the one we use in this paper3 , we will use the word “chat- ical overview, in Section 3. we discuss chatbot’s platforms
bot” to name the old school chatbots, typically associated and how to enrich them, and, in Section 4., we summarise
with chatbot’s competitions. the main “tricks” towards the “illusion of intelligence”. Fi-
We focus on chatbots that freely engage conversation about nally, in Section 5., we present some conclusions and point
any subject (the non goal-oriented feature), making them to future challenges.
“entertaining in a large variety of conversational topic set-
tings” (Schumaker et al., 2007). However, these are also 2. Historical overview
systems that “seek to mimic conversation rather than un-
derstand it”, that is, there is no real intention of making In this section we make a brief review of these chatbots’
them “intelligent”, as the main goal of their developers is history, moving from the first chatbots to the ones with
to make these chatbots effective in their simulation of in- which we interact nowadays.
telligence. Some of these chatbots were developed and
tailored with the goal of participating in chatbot’s compe- 2.1. Early days
titions (in fact, the term chatbot was coined in (Mauldin, Although the term chatbot was not invented by that
1994) to name the systems that have the goal of passing the time, the first chatbot came to public in 1966 under
Turing Test (Turing, 1950)), and, due to that, some have the appearance of a Rogerian psychotherapist called
gained visibility. The lack of full descriptions and papers Eliza (Weizenbaum, 1966). Eliza was a program devel-
about these chatbots (which explains the abnormal number oped by Joseph Weizenbaum that was able to establish a
conversation with human beings, simulating it was one too.
1
www.pandorabots.com Eliza’s conversational model was based in the rephrasing
2
workshop.colips.org/wochat/ of input sentences, when these matched a set of pre-defined
3
Many terms are used as synonyms of chatbot, as for instance
dialogue system, avatar, intellectual agents, and virtual person.
4
A list of more than 160 terms used as synonyms of chatbot can be An extended version of this paper can be found in https:
found in www.chatbots.org/synonyms/. //arxiv.org/abs/1609.06479.
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 61
rules. For instance, consider the following rule5 constituted summarised, namely: (1) admitting ignorance; (2) chang-
by a regular expression (match) and an answer (answer): ing the conversation topic; and, (3) introducing small sto-
ries about the Mafia throughout the conversation. These
three tricks are (respectively) illustrated in the following
match: * you are * answers given by Parry:
answer: What makes you think I am (2)?
Parry: I don’t get you.
Example 1. ...
Parry: Let’s talk about something else.
...
In this rule, if the match part coincides with the input (* Parry: I know the mob controls the big
is the wildcard and matches every sequence of words), the rackets.
text associated with the answer part will be returned, be-
ing the variable (2) replaced by the sequence from the input Example 3.
captured by the second wildcard. The following dialogue
(Example 2) illustrates an application of this rule. Notice After Colby gathered transcripts of interviews between psy-
that some internal processing needs to be done, so that the chiatrists, patients and his program, he presented the results
sequence captured by (2) entitled to your opinion is modi- to another group of psychiatrists. He asked this group if
fied into entitled to my opinion. they could guess in what transcripts the interviewed was a
human and in which ones it was a program. The psychia-
trist could not do better than randomly guessing.
user: You are entitled to your opinion. It is possible to conclude from these results that the emo-
Eliza: What makes you think I am entitled
to my opinion? tional side can be easier to imitate than the intellectual
one (Kuipers et al., 1976). However, one of the main crit-
Example 2. icisms Parry received was of not being more than an illu-
sion, incapable of modelling a real person (Colby, 1974).
Eliza completely exceeded the expectations, given that
many people, when interacting with it, believed they were 2.2. The chatbots’ competitions
talking with another human (this outcome is currently Moving back to 1950, Alan Turing questioned “can ma-
called the “Eliza effect”). Without having any intention chines think?” (Turing, 1950), and proposed a way of test-
of modelling the human cognitive process and despite its ing it: the imitation game (now known as the Turing Test).
simplicity, Eliza showed how a program impersonating a The original imitation game is played by a man, a woman
specific professional role can cause a huge impression by and an interrogator whose objective is to guess the sex of
the mere illusion of understanding. Weizenbaum was taken the players. Turing proposed substituting one of the players
aback by some aspects of this success (Hutchens, 1997). by a machine and playing the same game. In this version,
What shocked him most was the fact that people actually if the interrogator wrongly identifies who is the human it
believed that the program understood their problems6 . Per- means that the machine “can think”.
ceiving Eliza as a threat, Weizenbaum wrote “Computer Based on (their own interpretation of) the Turing Test, chat-
Power and Human Reason” (Kuipers et al., 1976) with the bots’ competitions keep appearing. Chatterbox Challenge8 ,
aim of attacking the Artificial Intelligence (AI) field and or, more recently, the Chatbot Battles9 , are examples of
educating uninformed persons about computers. such competitions, although the most widely known is the
Nowadays, Eliza is still one of the most widely known Loebner prize10 , where participants are challenged with a
programs in AI and is at the base of a great number of simplified version of the total Turing Test (Powers, 1998).
chatbots, including Parry, its “successor”. Following a This prize is due to Hugh Loebner, who offered a reward
very similar architecture to that of Eliza, Parry appeared in to the first person whose program could pass the proposed
1971 by the hands of Kenneth Colby, simulating a paranoid test. The first Loebner Prize Contest took place in 1991,
mental patient (Saygin et al., 2000). An interesting com- at Boston’s Computer Museum (Epstein, 1992), and, since
parison between Parry and Eliza was made by Güzeldere then, the competition has been held annually in the quest of
and Franchi7 : “Parry’s strategy is somewhat the reverse finding the “thinking computer”.
of Eliza’s”, as one simulates the doctor, distant and with- As some chatbots, competing for the Loebner prize, are in-
out personality traces, and the other a paranoid patient deed capable of managing a conversation, keeping it con-
which states its anxieties. Differently from Eliza, Parry sistent, at least for a while, every year the most human-like
has knowledge of the conversation and it also some sort computer is distinguished with a prize. However, since the
of “state of mind”. The combination of these two factors first edition of the Loebner prize, in 1991, until now, no one
affects the output as it becomes a function not only of the won it. Nevertheless, in another Turing Test organised in
input, but also of Parry’s beliefs, desires, and intentions. 2014 by the U.K.’s University of Reading, a chatbot simu-
In (Mauldin, 1994) a few tricks to which Parry resorts are lating a 13-year-old boy, named Eugene Goostman, created
5 8
Inspired from Eliza’s implementation in web.archive.org/web/20150905221931/http://www.
search.cpan.org/˜jnolan/Chatbot-Eliza-1.04/ chatterboxchallenge.com/
6 9
www.alicebot.org/articles/wallace/eliza.html www.chatbotbattles.com
7 10
www.stanford.edu/group/SHR/4-2/text/dialogues.html www.loebner.net/Prizef/loebner-prize.html
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 62
by Vladimir Veselov and his team, convinced 33% of the modify, develop and deploy. Therefore, anyone, even non-
human judges that it was human. computer-experts, can make use of it (Wallace et al., 2007),
This event brought to the spotlight the old question of AI as no prior knowledge about AIML is required.
and generated (again) much controversy. In fact, many peo- ChatScript14 , the scripting language and open-source en-
ple consider that there was a misunderstanding of Turing’s gine, should also be addressed, as is at the basis of Suzette
intentions in the different implementations of the Turing (2010 Loebner Prize winner), Rosette (2011 Loebner Prize
test, as deep models of thinking were a presupposition un- winner), Angela (2nd in 2012 Loebner Prize), and the previ-
derlying Turing’s imitation game. Following this, even if a ously referred Rose (2014 Loebner Prize winner). It comes
chatbot was good enough to deceive the jury, it would not with useful features, including an ontology of nouns, verbs,
pass the Turing Test in Turing’s sense, as it does not have adjectives and adverbs, and offers a scripting language
a cognition model behind it. Another important criticism is (inspired by the Scone project, a knowledge-base system
stressed by Levesque (Levesque, 2014). For this author, AI developed to support human-like common-sense reason-
is the science that studies “intelligent behaviour in compu- ing and the understanding of human language (Fahlman,
tational terms”, and the ability to be evasive, although in- 2011)). According to Bruce Wilcox, its creator, ChatScript
teresting, may not show real intelligence. A computer pro- settles several AIML problems, such as not being reader
gram should be able to demonstrate its intelligence with- friendly. In fact, as AIML is based on recursive self-
out the need for being deceptive. In this sense, Levesque modifying input, it is harder to debug and maintain. A de-
et al. (Levesque et al., 2012) further explore this idea by tailed comparison between ChatScript and AIML capabili-
conceiving a reading comprehension test based on binary ties was made available by Wilcox, as a motivation for the
choice questions with specific properties that make them development of a new (his own) chatbot platform.15
less prone to approaches based on deception. Apart from
the numerous controversies regarding the Turing Test, the 3.2. Building chatbots by chatting
fact is that all these competitions strongly contributed to the Another approach to develop chatbots’ knowledge sources,
main advances in the field, and the most popular chatbots which avoids handcrafted rules, is based on chatting and
are the ones that were/are present in these competitions. learning from the resulting chats. Systems like the already
3. Building chatbots mentioned Jabberwacky (and Cleverbot) learn by keeping
never seen user interactions and posing them later to other
Behind each chatbot there is a development platform. users. The acquired answers are then considered suitable
These are typically based on a scripting language that al- answers for these interactions. That is, they learn to talk
lows the botmaster to handcraft its knowledge base, as well by talking, by relying on what has been said before by
as an engine capable of mapping the user’s utterances into users and mimicking them. The user’s intelligence becomes
the most appropriate answer. “borrowed intelligence” as, instead of being wasted, it in-
3.1. Scripting languages/platforms corporates a loop: what is said is kept (along with the in-
formation of when it was said) and in the future that knowl-
An impressive collection of Elizas can be currently found
edge may be exposed to another user. The given replies
in the web. For instance, Chatbot-Eliza11 is an implementa-
are then saved as new responses that the system can give
tion in Perl that can be used to build other chatbots. Knowl-
in the future. It is only possible to give a brief overview of
edge is coded as a set of rules that are triggered when
Jabberwacky’s or Cleverbot learning mechanisms as their
matched against the user’s input. Some of the available
architecture is not available to the public. The only dis-
programs offer features such as a certain capability to mem-
closed aspect is that the AI model is not one of the usually
orise information, adding synonyms or ranking keywords.
found in other systems, but a “layered set of heuristics that
The most popular language to build chatbots is probably the
produce results through analyses of conversational context
“Artificial Intelligence Markup Language”, widely known
and positive feedback”16 .
as AIML,12 a derivative of XML that includes specific tags.
As usual, knowledge is coded as a set of rules that will Another example of a chatbot that learns is Robby Garner’s
match the user input, associated with templates, the genera- “Functional Response Emulation Device” (Fred), the an-
tors of the output. The large usage of AIML can be justified cestor of Albert One, the winner of 1998 and 1999 Loeb-
by the following facts: besides its detailed specification, its ner Prize. Fred was a computer program that learned from
community allows anyone to obtain, for free, interpreters of other people’s conversations in order to make its own con-
AIML in almost all coding languages, from Java (program versations (Caputo et al., 1997). Fred began with a library
D) to C/C++ (program C) or even Lisp (program Z); the of basic responses, so that it could interact with users, and
set of AIML files that constitute the contents of A.l.i.c.e.’s from then on, it learned new phrases with users willing to
brain can also be freely obtained13 . All the pandorabots are teach it17 . Although such an (unsupervised) learning may
based on AIML, more specifically in AIML 2.0. This spe- lead to unexpected and undesirable results, with the Internet
cific release is usually characterised as being very easy to growth and the possibility of having many people talking
11 14
search.cpan.org/˜jnolan/Chatbot-Eliza-1.04/Chatbot/ sourceforge.net/projects/chatscript/
15
Eliza.pm This comparison can be found in gamasutra.com/blogs/
12
www.alicebot.org/aiml.html BruceWilcox/20120104/9179/.
13 16
code.google.com/p/aiml-en-us-foundation-alice/ www.icogno.com/a_very_personal_entertainment.html
17
downloads/list www.simonlaven.com/fred.htm
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 63
with the chatbots, one may foresee that these will quickly restricted the conversation by giving Joe a “fairly narrow
evolve. worldview”. Another trick was to use Joe’s eminent dis-
missal to introduce some stories revolving around it, which
4. The illusion of intelligence and/or the art would, at the same time, provide a way of directing the con-
of scripting versation, the topic of the next section.
Creating chatbots goes beyond writing good programs and
developing algorithms, as in order to create a chatbot, more 4.2. Directing a conversation
than being a programmer, the botmaster must be an author. Personality can justify some appropriate answers, but the
Juergen Pirner, creator of the 2003 Loebner prize winner best way to deal with unexpected interactions is to avoid
Jabberwock18 , emphasises the scripting process behind a them. Thus, being able to direct the conversation is a trick
chatbot, stating that in the presence of possible failures, the used by many chatbots, including the simple forms used by
one at fault is not the engine but its author. Eliza, where the usage of questions incited the user partic-
Since making a chatbot involves preparing it to the impos- ipation and made him/her keep the conversation with little
sible mission of giving a plausible answer to all possible contribution from the program.
interactions, the botmasters usually take advantage of sev- Converse (Batacharia et al., 1999), created by David Levy,
eral tricks to simulate understanding and intelligence. For was the 1997 winner of the Loebner competition and did
instance, Pirner describes basic techniques of scripted di- extremely well by using the clever trick of controlling a
alogs like “having a set of responses for each scripted di- conversation. Although directing a conversation by “talk-
alog sequence” and “ending those same responses with a ing a lot about a predefined topic” was already used (Saygin
clue, a funny remark or a wordplay”. With Eliza, we learnt et al., 2000), Converse’s performance convinced a judge for
that including the user’s strings in its answers helps main- the first five minutes that he was really human: after greet-
taining an illusion of understanding (Mauldin, 1994). Other ing the judge, Catherine (Converse’s character) asked the
approaches focus on trying to guess what the user might say interrogator about something that had passed on the news
or forcing him/her to say something expected. the previous day and then kept talking about it, as can be
seen in the transcripts21 . David Levy’s won again the Loeb-
4.1. Giving the bot a personality ner prize in 2009 with Do-Much-More22 , but this time the
Whereas personality has been a subject of study among the system was more flexible in the range of topics and re-
agent’s community, deeply exploited in all its complexity, sponses it covered.
the concept is kept as simple as possible within chatbots.
As we have seen, what is common is the association of 4.3. Paying attention to small talk
an a priori “personality” to a chatbot, which can justify Small talk, or phatic communication (Malinowski, 1923),
some answers that otherwise would be considered inappro- is another hot topic in chatbots. It can be seen as a “neutral,
priate. For instance, Rogerian mode of Eliza covers for its non-task-oriented conversation about safe topics, where no
answers, as it leads to a conversation where the program specific goals needs to be achieved” (Endrass et al., 2011).
never contradicts itself, never makes affirmations, and is Small talk can be used for two main purposes (Schneider,
free to know nothing or little about the real world without 1988): establish a social relation by building rapport and
being suspicious. The same happens with Parry: being a avoiding (embarrassing) silence. As stated in (Bickmore
paranoid mental patient its changes in subject or incongru- and Cassell, 1999), chatbots have been making use of the
ous answers are considered satisfactory and hide its absence small talk mechanism. For instance, Epstein, an American
of understanding. The aforementioned Eugene Goostman psychologist, professor, author, and journalist, went to an
also follows along these lines. Veselov explains his reason- online dating service, and believed for several months that
ing for such a character: “a 13 years old is not too old to a chatbot, met in the dating service, was a “slim, attractive
know everything and not too young to know nothing”19 . brunette” (Epstein, 2007). In brief, small talk is a constant
Thomas Whalen, winner of 1994 Loebner prize, took this in all chatbots programs, used in non-sequiturs or canned
a step further with Joe, the janitor. Whalen’s decision was responses. It fosters the idea of understanding and eases
related to the fact that contrary to previous editions of Loeb- cooperation, facilitating human-like interactions by gaining
ner competitions, where the conversation was restricted to user’s trust and developing a social relationship (Bickmore
a topic, in 1995 the judges could pose any question. Hence, and Cassell, 2000).
Whalen decided that the best approach to deal with a non-
topic situation, would be to present a system that “would 4.4. Failing like a human
not simply try to answer questions, but would try to incor- After introducing the imitation game, Turing presented an
porate a personality, a personal history, and a unique view example (Example 4) of a possible conversation one could
of the world”20 . And so Joe was born. Joe was a night- have with a machine (Turing, 1950). Observing this ex-
worker janitor in the verge of being fired. He was only ample, besides the delay in providing the response, we can
“marginally literate”, and he did not read books, newspa- easily see that the answer is wrong. As Wallace wrote23 ,
pers, or watch television. These premises by themselves “we tend to think of a computer’s replies ought to be fast,
accurate, concise and above all truthful”. However, human
18
www.chatbots.org/developer/juergen_pirner/
19 21
www.huffingtonpost.com/2012/06/27/eugene-goostman- www.loebner.net/Prizef/converse.txt
22
2012-turing-test-winner_n_1630412.html www.worldsbestchatbot.com/
20 23
hps.elte.hu/˜gk/Loebner/story95.htm www.alicebot.org/anatomy.html
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 64
communication is not like that, containing errors, misun- Springer International Series in Engineering and Com-
derstandings, disfluencies, rephrases, etc. puter Science, pages 205–215. Springer.
Bibauw, S., François, T., and Desmet, P. (2019). Dis-
Human: Add 34957 to 70764. cussing with a computer to practice a foreign lan-
(after pause of about 30 seconds)
Machine: 105621. guage: research synthesis and conceptual framework
of dialogue-based call. Computer Assisted Language
Example 4. Learning, 0(0):1–51.
Bickmore, T. and Cassell, J. (1999). Small talk and con-
This is something that earlier chatbot’s writers already had
versational storytelling in embodied conversational inter-
in mind, as some already cared about simulated typing. For
face agents. In Proc. of the AAAI 1999 Fall Symposium
instance, Julia (Mauldin, 1994) simulated human typing by
on Narrative Intelligence, pages 87–92. AAAI Press.
including delays and leaving some errors. Simulated typing
also proves to be useful in decreasing mistakes by slowing Bickmore, T. and Cassell, J. (2000). How about this
down the interaction: Philip Maymin, a Loebner contestant Weather? Social Dialogue with Embodied Conversa-
in 1995, slowed the typing speed of his program to the point tional Agents. In Socially Intelligent Agents: The Hu-
that a judge was not able to pose more than one or two man in the Loop, pages 4–8. AAAI Press.
questions (Hutchens, 1997). Caputo, L., Garner, R., and Nathan, P. X. (1997). FRED,
Milton and Barry: the evolution of intelligent agents for
5. Conclusions and future challenges the Web. In F. C. Morabito, editor, Advances in Intelli-
The number of chatbots that can be found in the web in- gent Systems, pages 400–407. IOS Press.
creases every day. Besides tools and corpora, the chatbots’ Colby, K. M. (1974). Ten criticisms of PARRY. SIGART
community has important know-how, which should not Newsletter, pages 5–9.
be neglected by researchers targeting advances in human- Dale, R. (2016). The return of the chatbots. Natural Lan-
machine communication. Therefore, we present a brief his- guage Engineering, 22(5):811–817.
torical overview of chatbots and describe main resources Endrass, B., Rehm, M., and André, E. (2011). Plan-
and ideas. Furthermore, we highlight some chatbots, rele- ning Small Talk behavior with cultural influences for
vant because they introduce new paradigms and/or won the multiagent systems. Computer Speech & Language,
Loebner prize. However, it should be clear that these are 25(2):158–174.
only the tip of the iceberg of the panoply of chatbots that Epstein, R. (1992). The Quest for the Thinking Computer.
currently exist. AI Magazine, pages 81–95.
We have seen that AIML and, more recently, Chatscript Epstein, R. (2007). From Russia, with Love. How I got
are widely used languages that allow to code the chatbots’ fooled (and somewhat humiliated) by a computer. Scien-
knowledge sources, and that even in chatbots that imple- tific American Mind.
ment learning strategies, scripting is still at their core. We Fahlman, S. E. (2011). Using Scone’s Multiple-Context
have also seen that a personality capable of justifying some Mechanism to Emulate Human-Like Reasoning. In Ad-
of the chatbot’s answers, the capacity of directing a conver- vances in Cognitive Systems: Papers from the 2011 AAAI
sation and producing small talk, and the idea of failing like Fall Symposium, pages 98–105. AAAI Press.
a human are some of the chatbots’ features that give the il- Fialho, P., Coheur, L., Curto, S., Cláudio, P., Ângela Costa,
lusion of intelligence. We have also grasped that to create Abad, A., Meinedo, H., and Trancoso, I. (2013). Meet
a chatbot, one “only” needs to think about a character and Edgar, a tutoring agent at Monserrate. In Proc. of the
enrich its knowledge bases with possible interactions. Even 51st Annual Meeting of the ACL: System Demonstra-
better, that work does not need to be done from scratch as tions, pages 61–66.
many platforms already provide pre-defined interactions,
Følstad, A. and Brandtzæg, P. B. (2017). Chatbots and the
which can be adapted according to the chatbot character.
New World of HCI. Interactions, 24(4):38–42.
And this is the main richness of the chatbot’s community:
Hutchens, J. L. (1997). How to Pass the Turing Test by
the immense amount of collected interactions, where the
Cheating. Technical report, Univ. of Western Australia.
majority of them represent real human requests. All this
data (after some validation) could be used to train current Kuipers, B., McCarthy, J., and Weizenbaum, J. (1976).
end-to-end data-driven systems. A major future challenge Computer power and human reason. SIGART Bull.,
is to be able to automatically use all this information to pages 4–13.
build a credible chatbot. How to avoid contradictory an- Levesque, H., Davis, E., and Morgenstern, L. (2012). The
swers? How to choose appropriated answers considering a Winograd Schema Challenge. In Proc. of the Thirteenth
chatbot’s character? And if we move to other sources of di- International Conf. on Principles of Knowledge Repre-
alogues, like the ones from books, theatre plays or movies sentation and Reasoning, pages 552–561. AAAI.
subtitles, will we be able, one day, to integrate all that in- Levesque, H. J. (2014). On our best behaviour. Artificial
formation simulating real human dialogues? Intelligence, 212:27–35.
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B.
6. Bibliographical References (2016). A diversity-promoting objective function for
Batacharia, B., Levy, D., Catizone, R., Krotov, A., and neural conversation models. In Proceedings of the 2016
Wilks, Y. (1999). Converse: a conversational com- Conference of the North American Chapter of the Associ-
panion. In Machine Conversations, volume 511 of The ation for Computational Linguistics: Human Language
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 65
Technologies, pages 110–119. Association for Computa-
tional Linguistics.
Malinowski, B., (1923). The Meaning of Meaning, chapter
The Problem of Meaning in Primitive Socities, page 38.
Harcourt Brace Jovanovich, Inc.
Mauldin, M. L. (1994). ChatterBots, TinyMuds, and the
Turing test: entering the Loebner Prize competition. In
Proc. of the 12th National Conference on Artificial Intel-
ligence (vol. 1), AAAI ’94, pages 16–21. AAAI Press.
Powers, D. M. W. (1998). The total Turing test and the
Loebner prize. In Proc. of the Joint Conf. on New Meth-
ods in Language Processing and Comp. Natural Lan-
guage Learning, NeMLaP3/CoNLL ’98, pages 279–280.
Saygin, A. P., Cicekli, I., and Akman, V. (2000). Turing
test: 50 years later. Minds and Machines, 10:2000.
Schneider, K. (1988). Small Talk: Analyzing Phatic Dis-
course. Sprachwissenschaftliche Reihe. Hitzeroth.
Schumaker, R. P., Ginsburg, M., Chen, H., and Liu,
Y. (2007). An evaluation of the chat and knowl-
edge delivery components of a low-level dialog system:
The AZ-ALICE experiment. Decision Support Systems,
42(4):2236–2246.
Turing, A. M. (1950). Computing Machinery and Intelli-
gence. Mind, 59:433–460.
Vinyals, O. and Le, Q. (2015). A neural conversational
model. arXiv preprint arXiv:1506.05869.
Wallace, R., Tomabechi, H., and Aimless, D. (2007). Chat-
terbots Go Native: Considerations for an eco-system fos-
tering the development of artificial life forms in a hu-
man world. http://www.pandorabots.com/pandora/pics/
chatterbotsgonative.doc.
Weizenbaum, J. (1966). ELIZA – a computer program for
the study of natural language communication between
man and machine. Comm. of the ACM, 9:36–45.
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 66