When trying to identify the author of a book, a paper, or a letter, the object is to detect a style that distinguishes one author from another. With recent developments in artificial intelligence, chat bots sometimes play the role of the text authors. The focus of this study is to investigate the change in chat bot linguistic style over time and its effect on authorship attribution. The study shows that chat bots did show a behavioral drift in their style. Results from this study imply that any non-zero change in lingual style results in difficulty for our chat bot identification process.
Biometric identification technologies are not limited
to fingerprints. Behavioral traits associated with each
human provide a way to identify the person by a biometric
profile. Behavioral biometrics provides an advantage over
traditional biometrics in that they can be collected
unbeknownst to the user under investigation
Authorship identification is a research field interested
in finding traits, which can identify the original author of
the document. Two main subfields of authorship
identification are: (a) Authorship recognition, when there
is more than one author claiming a document, and the task
is to identify the correct author based on the study of style
and other author-specific features. (b) Authorship
verification, where the task is to verify that an author of a
document is the correct author based on that author’s
profile and the study of the document
When examining people, a major challenge is that the
writing style of the writer might evolve and develop with
time, a concept known as behavioral drift
Chat bots are computer programs mainly used in
applications such as online help, e-commerce, customer
services, call centers, and internet gaming
Chat bots are typically perceived as engaging software
entities, which humans may communicate with, attempting
to fool the human into thinking that he or she is talking to
another human. Some chat bots use Natural Language
Processing Systems (NLPS) when replying to a statement,
while majority of other bots are scanning for keywords
within the input and pull a reply with the most matching
keywords
B.
The ongoing threats by criminal individuals have migrated from actual physical threats and violence to another dimension, the Cyber World. Criminals try to steal others information and identity by any means. Researchers are following up and doing more work trying to prevent any criminal activities, whether it is identity theft or even terrorist threats.
II.
Data was downloaded from the Loebner prize website
III.
The collected data had to be preprocessed by deleting unnecessary labels like the chat bot name, and time-date of conversation (Fig. 1). A Perl script was used to clean the files and split each chat into two text files, one for the chat bot under study, the other for the human judge. The judge part was ignored, and only the chat bot text was analyzed.
Eleven chat bots were used in the initial experiments:
Alice
Process Document
Normalize
Validation Store
Word list
In Fig. 3 we use the saved word list and model as input for the testing stage, and the output will give us the percentage prediction of the tested files.
Get Word list
Process Document
Normalize
Get
Model
The data was tested using two different saved models, one with a complete set of chat bots (eleven bots) in the training stage, and the second model was built with training using only the three chat bots under study.
When performing the experiments, the model output is confidence values, in which, values reflecting how confident we are that this chat bot is identified correctly. Chat bot with highest confidence value (printed in boldface in all tables) is the predicted bot according to the model. Table 1 shows how much confidence we have in our tested data for Alice’s text files in different years, when using eleven chat bots for training.
Fig 5. Identification percentage over different years using only the three chat bots under study for training.
The initial experiments conducted on the collected data did show a variation between chat bots, which is expected. It is not expected that all chat bots will act the same way, since they have different creators and different algorithms.
Some chat bots are more intelligent than others; the Loebner contest aims to contrast such differences. Alice bot showed some consistency over the years under study, but in 2005 Alice’s style was not as recognizable as in other years. While Jabberwacky performed well for all years when training with just three bots and was not identified in 2001 when the training set contained all eleven chat bots for training, Jabberwacky gave us a 40% correct prediction in 2005. Jabberwock, the third chat bot under study here, was the least consistent compared to all other bots, and gave 0% correct prediction in 2001 and 2004, and 91% for 2011, which may indicate that Jabberwock’s vocabulary did improve in a way that gave him his own style.
With three chat bot training models, Jabberwacky was identified 100% correctly over all years. Alice did well for all years except for 2005, and Jabberwock was not identified at all in 2001 and 2004.
With these initial experiments, we can state that some chat bots do change their style, most probably depending on the intelligent algorithms used in initializing conversations. Other chat bots do have a steady style and do not change over time.
More data is required to get reliable results; we only managed to obtain data from the Loebner prize competition, which in some cases was just one 4KB text file. With sufficient data, results should be more representative and accurate.
Additional research on these chat bots will be conducted, and more work on trying to find specific features to identify the chat bots will be continued. This is a burgeoning research area and still much work need to be done.