=Paper=
{{Paper
|id=Vol-2916/paper_14
|storemode=property
|title=Impossibility of Unambiguous Communication as a Source of Failure in AI Systems
|pdfUrl=https://ceur-ws.org/Vol-2916/paper_14.pdf
|volume=Vol-2916
|authors=William Howe,Roman Yampolskiy
|dblpUrl=https://dblp.org/rec/conf/ijcai/HoweY21
}}
==Impossibility of Unambiguous Communication as a Source of Failure in AI Systems==
https://ceur-ws.org/Vol-2916/paper_14.pdf
Impossibility of Unambiguous Communication as a Source of Failure in AI
Systems
William J. Howe1 , Roman V. Yampolskiy2
1
Johns Hopkins University
2
University of Louisville
whowe1@jhu.edu, roman.yampolskiy@louisville.edu
Abstract 2 Phonology
Ambiguity is pervasive at multiple levels of linguis- Computational phonology is a core component of speech-
tic analysis effectively making unambiguous com- based NLP systems. The ultimate goal of automatic speech
munication impossible. As a consequence, natural recognition is to take an acoustic waveform as input and de-
language processing systems without true natural code it into a string of words as text [Jurafsky, 2000]. The
language understanding can be easily ”fooled” by field which for several years was dominated by the Gaus-
ambiguity, but crucially, AI also may use ambiguity sian Mixture Model - Hidden Markov Model (GMM-HMM)
to fool its users. Ambiguity impedes communica- framework has now made significant advancements using
tion among humans, and thus also has the potential deep neural network (DNN) architectures to enable technolo-
to be a source of failure in AI systems. gies like Siri, Alexa, and Google Assistant [Yu and Deng,
2016]. In particular recurrent neural networks which cap-
1 ture the “dynamic temporal behavior” of sequence data that
DNN-HMM architectures do not capture, have proven very
effective [Yu and Deng, 2016]. Despite these advances, au-
1 Introduction tomatic speech recognition (ASR) still performs poorly with
The human language faculty allows any given speaker to far field microphones, noisy conditions, accented speech, and
”make infinite use of finite means” [Chomsky, 2006]. This is multitalker speech [Yu and Deng, 2016]. To see why ambi-
to say that the set of all possible sentences is infinite while guity poses such a problem for these models, we’ll consider
the set of words which make them up is finite. However, a architecture which uses some statistical technique to rec-
ambiguity – the existence of more than one interpretation of ognize speech units along with some language model over
an expression, is rampant in natural language [Wasow et al., some dictionary to find the highest probability sequence of
2005]. It is not clear why ambiguity exists at all in natural speech units [Jurafsky, 2000]. It is clear that because such a
language. Given that it impedes communication, one might model is probabilistic, it lacks true natural language under-
assume languages would evolve to avoid it, yet this is not ob- standing – this means the model can fail when faced with a
served [Wasow et al., 2005]. One explanation is that mapping speech waveform that might be unlikely or low probability.
a word to multiple meanings saves memory. Another account It may favor the more likely incorrect output over the less
asserts that ambiguity is a consequence of a human bias to- likely yet correct target output. Because humans possess lin-
ward shorter morphemes [Wasow et al., 2005]. Yet another guistic creativity – the ability to produce never before seen
account construes ambiguity as a product of optimization to- utterances which a model might consider highly improbable,
wards efficiency (principle of least-effort) over the course of current ASR systems have an inherent deficit. One way to
language evolution. On this view, ambiguity is the price paid remedy this is to filter out hypotheses that don’t make sense
for a least effort language [Solé and Seoane, 2015]. In this with, “[a] speech recognition system augmented with Com-
paper, we won’t seek to explain the root cause of ambiguity, monsense Knowledge [that] can spot its own nonsensical er-
but rather to show how it can pose a problem for AI systems. rors, and proactively correct them” [Lieberman et al., 2005;
First we’ll identify types of ambiguity which occur at the lev- Liu et al., 2016]. Nevertheless, brittle ASR systems, “may
els of phonology, syntax, and semantics, noting how mod- misinterpret commands due to coarticulation, segmentation,
ern natural language processing (NLP) systems disambiguate homophones, or double meanings in the human language”
ambiguous input. Finally, we’ll consider how more advanced [Yampolskiy, 2016].
AI could exploit ambiguity and how bad actors might utilize
2.1 Homophones
such systems to their ends.
Homophones – sets of words which sound the same but have
1
Copyright © 2021 for this paper by its authors. Use permitted different meanings, are a classic case of phonological ambi-
under Creative Commons License Attribution 4.0 International (CC guity. The following data present utterances which could be
BY 4.0). misinterpreted by an ASR system but which are easily disam-
biguated by humans provided some context [Forsberg, 2003]. information leakage, and “as a stepping stone to further at-
tacks” [Carlini et al., 2016]. Though these exploits don’t nec-
essarily target natural language ambiguity they serve to show
(1) a. the tail of the dog that current ASR systems are vulnerable to a range of attacks.
b. the tale of the dog
(2) a. the sail of the boat 3 Syntax
b. the sale of the boat
Syntax determines how words are organized into phrases
The pairs in (1) and (2) are phonologically and syntactically and sentences [Carnie, 2012]. Historically, syntax has been
identical, yet convey distinct meanings. With sufficient con- processed with computational models including context-free
text, an ASR with a good enough language model would eas- grammars, lexicalized grammars, feature structures, parsing
ily disambiguate tail/tale shown below: algorithms, and HMM part-of-speech taggers. Parsing a sen-
tence into constituency or dependency tree structure is use-
(3) a. the tail of the dog was wagging ful for downstream NLP tasks. The same is true for part-of-
b. the tale of the dog was told speech tagging [Jurafsky, 2000]. Deep learning using ANNs
c. the tail/tale of the dog was long has achieved state of the art performance on syntax-related
tasks, though ANNs still do not match human level perfor-
However, sufficient context is not always provided as shown mance on phenomena like filler-gap dependencies [Linzen
in c). Thus, carefully chosen homophones could be used to and Baroni, 2020]. Here, we’ll examine several characteri-
intentionally fool an ASR system. zations of ambiguity at the level of syntax.
2.2 Continuous Speech 3.1 Structural Ambiguity
Continuous speech is very different from written lan- Structural ambiguity occurs when more than one underly-
guage. Spoken language introduces word boundary ambigu- ing structure exists for a single sentence with the structures
ity, speaker variability, and a different vocabulary. It is com- having different meanings. The term structure is used here
mon for the spoken register of a language to be different from because sentences with this type of ambiguity are usually
the more formal written register [Forsberg, 2003]. All these disambiguated by distinguishing between two different con-
factors contribute to the difficulty of ASR and introduce the stituency trees.
possibility of ambiguity when decoding.
Global Ambiguity
(4) a. How to wreck a nice beach you sing calm in- Global ambiguity is ambiguity that persists after a sentence
cense. has been fully parsed. In this case a sentence in and of itself
b. How to recognize speech using common sense. contains more than one structural interpretation. Consider the
(5) a. I want to experience youth in Asia. following data:
b. I want to experience euthanasia.
(6) a. Eliminate [NP the target] [PP with a bomb.]
These constructed yet plausible examples show that it may b. Eliminate [NP the target [PP with a bomb.]]
be possible to generate adversarial examples to fool an ASR
Here, the NP (noun phrase) has two interpretations; one
system. Once again, one would expect an effective language
where the PP (prepositional phrase) is contained within the
model to be successful at disambiguating these examples, but,
NP and the other where it is not. The former refers to an in-
as we will note below, there is evidence that fooling an ASR
dividual carrying a bomb, while the latter refers to the action
system is even more easily achieved by simply perturbing the
of bombing someone.
input waveform.
Local Ambiguity
2.3 Fooling Automatic Speech Recognition Local ambiguity, unlike global ambiguity, is resolved upon
Systems complete parsing of a sentence. The canonical case of local
We’ve shown that it ought to be possible to fool ASR systems ambiguity is the garden path sentence. Consider the follow-
using phonological ambiguities which are common in natu- ing data [Ferreira and Henderson, 1991]:
ral language. This involves carefully crafting utterances with (7) a. Because Bill drinks wine ...
homophones or with word boundary ambiguity. However, it b. Because Bill drinks wine beer is never kept in the
is possible to exploit such systems without leveraging natural house.
language ambiguity. It has been shown that adversarial ex- c. Because Bill drinks wine is never kept in the
amples can be created by applying perturbations to an input house.
waveform such that the waveform is nearly indistinguishable
from the unperturbed input. Even more worrying is fact that As seen in (7a), Because Bill drinks wine is ambiguous:
this perturbed input can generate any desired output phrase wine could take on a direct object semantic role as in (7b) or
[Carlini and Wagner, 2018]. The researchers have also shown it could take on a subject semantic role as in (7c). Assum-
that hidden voice commands, unintelligible inputs used to at- ing that a human parser employs the principle of late closure,
tack ASR systems, can be used to cause denial of service, preferring to attach new material into the phrase or clause
currently open rather than create new clauses or constituents, derived from the use of syntactic heuristics which quickly
(7b) is easier to parse for a human than (7c). In general for break down when confronted with more complex examples
humans, garden-path recovery is thought to involve reanalysis [McCoy et al., 2019]. This is a major problem for narrow AI.
of the sentence by reassigning the thematic roles of a misana- For advanced AI, the ability to toy with the very meaning of
lyzed phrase [Ferreira and Henderson, 1991]. Regardless, the language would have wide ranging consequences from sow-
ambiguity and added parsing difficulty of garden-path sen- ing disinformation to generating ambiguous legal documents
tences could be a source of failure in NLP and AI systems. or tweets.
3.2 Formal Language Ambiguity 4.1 Scope Ambiguity
A context-free grammar is a grammar whose rules all follow Here, the scope of a syntactic constituent is ambiguous. The
the form A → Ψ where A is a non-terminal symbol and Ψ is following data further elucidates scopal ambiguity [Wasow et
any string, even the empty string, from the union of the termi- al., 2005]:
nal and non-terminal alphabets [Partee et al., 2012]. Consider
the following context-free grammar: (8) No student solved exactly two problems.
S → (A B) | (C D) In (8) either, “there was no student who solved exactly two
problems”, or “there were exactly two problems that no stu-
A→U C→U dent solved” [Wasow et al., 2005]. Either interpretation is
B→V D→V valid depending on the location of constituents in the under-
U →a V →b lying sentence structure which determines their scope (this is
sometimes referred to as LF, logical form). For this reason,
Even in this simple context-free grammar, the string ab can scope ambiguity lies at the syntax-semantics interface [An-
either be generated using the rule S → A B or the rule derson, 2004].
S → C D. Thus, there exists more than one parse tree
structure for the same surface string representation and this 4.2 Lexical Ambiguity
constitutes one characterization of syntactic ambiguity. One This type of ambiguity deals with the meanings of words.
notable technique for disambiguating context-free grammars When a word has more than one distinct meaning it is said to
is the PCFG (probabilistic context-free grammar) which as- have lexical ambiguity. We’ll highlight examples of lexical
signs probabilities to rules in a CFG (different weights for ambiguity and examine current NLP approaches to address-
the two S rules above, for example) [Jurafsky, 2000]. As in ing it.
the discussion on phonology, probabilistic language models
may serve to make NLP systems more “natural” (more simi- Contranyms
lar to human language) yet this may not give models the capa- In the case of contranyms, a word has two different meanings
bility to reason about more complex ambiguities. The same which are antonyms [Jackson, 2018]:
applies for neural network based language models such as
GPT-2 [Radford et al., 2019] and BERT [Devlin et al., 2018] (9) hold up
which can be thought of as massive context-free grammars a. to support
with extremely well fine-tuned probabilistic weights. b. to hinder
(10) dust
3.3 Security of Language Models
a. add fine particles
In addition to the advances made in downstream NLP b. remove fine particles
tasks by means of language model pretraining and fine-
(11) left
tuning, recently, neural network language models have been
shown to perform well as knowledge bases [Petroni et al., a. departed
2019]. Specifically, BERT (Bidirectional Encoder Represen- b. remaining
tations from Transformers) has been shown to contain re- Word Sense Disambiguation
lational knowledge competitive with traditional knowledge
The most salient case of lexical ambiguity is known as pol-
base methods and to perform well on open-domain question
ysemy in which one word has more than one distinct mean-
answering [Petroni et al., 2019]. If neural network language
ing. The work bank can refer to a bank account, to a river
models become widely adopted as knowledge bases, this ne-
bank, or as a verb, to moving on an incline. There is a long
cessitates the question, Is the private information encoded in
history of computational techniques for word sense disam-
a language model secure? Though it does not relate to ambi-
biguation from dictionary based methods to semantic simi-
guity, there is work showing that privacy can be preserved in
larity metrics [Yarowsky, 1995; Banerjee and Pedersen, 2002;
such models using encryption [Ryffel et al., 2018].
Navigli, 2009; Resnik, 1999].
4 Semantics 4.3 Winograd Schema
Semantics, the meaning of words and sentences, is of consid- A winograd schema is a pair of sentences that differ in only
erable interest in NLP. However, much of the perceived se- two words and contain a referential ambiguity that is resolved
mantic knowledge encoded in current NLP systems is instead in “opposite directions” in the two sentences. The Winograd
Schema Challenge presents such a pair as an alternative to proaches nonetheless can taut that it is their models which
the Turing Test since a successful agent must have some level have achieved such success on natural language tasks and
of natural language understanding to solve the challenge and benchmarks. The debate between deep learning approaches
cannot depend on statistical patterns [Levesque et al., 2012]. and symbolic approaches is not yet resolved. An interesting
Though the Winograd Schema is technically a referential am- area is neural-symbolic computation which seeks to marry
biguity, its difficulty is rooted in machines’ lack of common- neural network models with symbolic approaches [Smolen-
sense knowledge so we’ve placed it in the semantics section. sky et al., 2016; Garcez et al., 2015].
(12) The trophy doesn’t fit in the brown suitcase because
it’s too (big/small). What is too (big/small)? 6 Inevitability of Ambiguity
a. the trophy Pragmatics, the area of linguistics which focuses on the
b. the suitcase co-operative assumptions of communication, arguably bears
(13) Joan made sure to thank Susan for all the help she its own ambiguities such as irony and sarcasm [Wilson,
had (given/received). Who had (given/received) the 2006]. There are existing computational approaches for deal-
help? ing with various other discourse ambiguities [Macagno and
a. Joan Bigi, 2018; Ammicht et al., 2001]. On the basis of discourse
b. Susan analysis [Blum-Kulka and Weizman, 1988] argue that “com-
munication is inherently ambiguous”. Intuitively, we endorse
In (12) there are two sentences that can be generated based this view since wholly unambiguous communication seems
on the choice of big or small which have two different an- to be impossible using an inherently ambiguous natural lan-
swers. Answering correctly requires natural language under- guage. On the basis of the above examples, it is clear that
standing and reasoning. The dataset WINOGRANDE showed there are some ambiguities that even humans cannot easily
that although models performed well (90% accuracy) on ex- disambiguate. These effects are only multiplied when one
isting Winograd datasets, this was likely due to algorith- considers the ambiguity in pragmatics which might cause one
mic bias. Producing adversarial Winograd Schema exam- to “question the validity of co-operative assumptions” such as
ples by means of a debiasing algorithm allowed the authors to the Gricean maxims [Blum-Kulka and Weizman, 1988]. Here
achieve state of the art performance on these existing Wino- we’ll seek to discuss this claim with more mathematical rigor.
grad benchmarks showing their technique to be a powerful We’ll represent a discourse with a finite-state discrete time
example of transfer learning [Sakaguchi et al., 2019]. Markov Chain with two states (Figure 1). The chain is in state
0 when ambiguity is introduced into a discourse and in state
1 when there is no natural language ambiguity. This two state
5 Criticism of Deep Learning Approaches chain is a positive recurrent irreducible Markov Chain [Ross
The approaches for dealing with natural language and thus et al., 1996]. Each new utterance in a discourse is represented
in turn ambiguity discussed above are largely engineering by a transition in the chain. This chain is a good model since a
approaches. These include things like, deep learning, fine- conversation can stay in the unambiguous state with positive
tuning of language models, building more robust models that probability. The conversation can move from a unambiguous
generalize better, and improving state of the art performance state to an ambiguous state with positive probability – this is
using adversarial examples [Jia and Liang, 2017; Subrama- when ambiguity is introduced into the discourse. The con-
nian et al., 2017; Wu et al., 2018; Wallace et al., 2019; versation can remain ambiguous (with positive probability)
Gong et al., 2018]. Although these engineering approaches by what [Blum-Kulka and Weizman, 1988] cite as indetermi-
have achieved state of the art performance in many areas, nate ambiguity in which “the speaker does not commit him-
there is a sense that they lack true natural language under- self to an intended meaning” and the indeterminacy is “left
standing as alluded to in several of the above types of ambi- unattended to by both participants”. The ambiguity can also
guity. [Marcus, 2020] describes deep learning based models be resolved (with positive probability) through clarification.
as, “data hungry, shallow, brittle, and limited in their ability All entries in the transitive matrix are positive probabilities.
to generalize” advocating instead for symbolic approaches in- Thus, for any finite discourse there is positive probability that
corporating insights from cognitive science. Character based there is no ambiguity. This is achieved by simply taking the
translation models break down under the introduction of noise transition from state 1 to state 1 at every point in the conver-
(letter swap errors) proving such NMT (neural machine trans- sation.
lation) models to be extremely brittle [Belinkov and Bisk, However, if we consider an infinite number of transitions
2017]. [Bender and Koller, 2020] argues that current ap- which may be a good model for a work of fiction, a long
proaches cannot learn form from meaning and thus will not speech, or an extended conversation over hours or days, in
achieve natural language understanding (NLU). It has been the limit, the chain will enter state 0 (the ambiguity state)
noted even that algorithmic approaches to anaphor resolution with probability 1 and return to the state in a finite number of
may never achieve complete success since, in principle the transitions on average [Ross et al., 1996]. This follows from
interpretation of a well-formed sentence like, Who wants the the fact that the chain is irreducible and positive recurrent.
first one? is free in the absence of sufficient context and ap- If we then consider the totality of human natural language
plication of constraints (i.e one could refer to anything) [Hen- generation as an infinite sequence of transitions through the
driks and De Hoop, 2001]. Advocates of deep learning ap- chain, it is clear that ambiguity is inevitable as long as we en-
1−α
α
β
0 1 α 1−α
P =
1−β β
1−β
Figure 1: State 0 denotes ambiguous discourse while state 1 denotes unambiguous discourse. P is the 2 state transition matrix [Ross et al.,
1996]
.
dorse the premise that there is positive probability of generat- continuous speech could be used to give a command with un-
ing ambiguous language at any step in the sequence. In order desired effects. A benign waveform command could be per-
to achieve unambiguous communication one would have to turbed to cause system failure, and open the door to further
ensure that at any step in this infinite sequence there is zero hacks and exploits. An input command like, Eliminate the
probability of generating an ambiguity – we claim that this is target with the bomb. could have unintended consequences
computationally intractable if one is using an inherently am- depending on the interpretation of its structural ambiguity
biguous natural language. which could be particularly dangerous in military and AI
This result, the impossibility of unambiguous communi- weaponry scenarios. Clearly, serious thought should be put
cation, is in accordance with two existing impossibility re- into designing systems that intelligently deal with ambiguous
sults in AI safety. The first, unpredictability of AI, states input. Just as Google auto-completes search results, an AI
that, “ it is impossible to precisely and consistently predict might be designed to answer and react to a query as quickly
what specific actions a smarter-than-human intelligent sys- as possible – this could result in failure on garden path sen-
tem will take to achieve its objectives” [Yampolskiy, 2019b]. tences. The AI might be tricked into the wrong interpretation
The second result, incomprehensibility of AI, shows that “ad- by using a late closure parsing technique.
vanced AIs would not be able to accurately explain some of Although machine translation is largely dominated by se-
their decisions and for the decisions they could explain people quence to sequence methods, a dictionary based translation
would not understand some of those explanations” [Yampol- system could fail due to cross-linguistic ambiguity. In a
skiy, 2019a]. This is true for opaque, black box NLP sys- case reminiscent of the movie Arrival, there is a character
tems discussed in Section 7.1. [Doran et al., 2017]. However, in Chinese which can mean instrument or weapon in En-
we’ll show in Section 7.2 that the impossibility of unambigu- glish.2 A mistake in translation could be dangerous in this
ous communication also contributes to unexplainability and case. There are plenty of other cases of lexical ambiguity
unpredictability for advanced AI . that could present a dangerous situation, particularly in a mil-
itary context. The word execute is a contranym – it can refer
7 Ambiguity as a Source of Failure to the execution (start) of a program (“execute the firmware
update”), or the execution (end) of a person (“execute the ad-
Pioneer in the field of machine translation, Yehoshua Bar-
versary”)3 . These failure modes are a result of the opaque,
Hillel claimed “FAHQT [fully automatic high quality transla-
black box nature of contemporary narrow AI systems without
tion] is out of the question for the foreseeable future because
natural language understanding. The ambiguity induced fail-
of the existence of a large number of sentences the determi-
ure modes discussed here are examples of by mistake, post-
nation of whose meaning, unambiguous for a human reader,
deployment AI hazards according to Yampolskiy’s taxonomy
is beyond the reach of machines” citing machines’ lack of [Yampolskiy, 2016].
commonsense knowledge [Bar-hillel, 1964]. While more ad-
vanced NLP systems may achieve higher accuracy on am- 7.2 AI Exploiting Ambiguity
biguous language tasks, the main thrust of his argument still
stands, even today. On top of that, there are examples of am- Ambiguity has been identified as a source of miscommunica-
biguity given above that, without sufficient context, even hu- tion in air traffic control [Mcmillan, 1998]. Ambiguity is such
mans cannot correctly interpret. Thus, it is clear that these a problem even for humans that some have attempted to build
weaknesses inherent in natural language could be exploited to controlled natural languages by restricting language use to a
fool an NLP system. Additionally, the reverse could be pos- wholly unambiguous subset of an existing natural language
sible – AI could exploit natural language ambiguity to fool its 2
According to Google Translate.
human users. 3
A reviewer notes that the event predicate execute is not strictly
synonymous with a start event predicate and thus this example does
7.1 AI Fooled by Ambiguity not constitute a true contranym. However, since performing some
AI without true natural language understanding can easily be action entails starting to perform the action, the example still makes
tricked by many of the above examples. Homophones and sense and is thus useful for explanatory purposes
Narrow AI/NLP System Human-Level AI
-tricked using adversarial examples
-built from end-to-end systems
-uses probabilistic or statistical methods -generates news headlines with
natural language understanding ambiguity
with lack of NLU
-brittle; may not generalize to other
dialects/accents -writes ambiguous tweets
-fails when presented with ambiguities below automatic ambiguity detection
-deploys attacks on narrow
AI systems (generating
adversarial speech waveforms)
-writes legal documents, software
code, contracts, with ambiguity
-used by bad actors to exploit
ambiguity
Figure 2: A narrow AI system is vulnerable to attack and may fail on ambiguous input. A more advanced system is then able to exploit
ambiguous language to generate misleading headlines or tweets.
[Fuchs et al., 2008]. Such a language could be used to enable ing public confusion and disinformation. Here, the metric
precise and unambiguous specification of rules and guidelines (ambiguity detection module) is used to “distort and corrupt
for organizations and software specifications. There are ex- the social processes it is intended to monitor” [Manheim and
isting attempts to detect ambiguity in requirements specifica- Garrabrant, 2018]. The unpredictability of such a system is
tions for software [Kiyavitskaya et al., 2008]. An AI capable a major risk. According to Yampolskiy’s Taxonomy of Path-
of exploiting ambiguity could reek havoc in these areas. ways to Dangerous AI, the use of such a system by a bad
In monetary theory, the observation that, “Any observed actor is an on purpose, post-deployment AI hazard [Yampol-
statistical regularity will tend to collapse once pressure is skiy, 2016].
placed upon it for control purposes” has been termed Good-
hart’s Law [Goodhart, 1983]. The observation has been re- 7.3 Safety Risk
stated as: “When a measure becomes a target, it ceases to be
Natural language ambiguity appears to introduce risk into
a good measure” [Hoskin, 1996]. Without making any claims
three types of systems. First, there are the uninterpretable
about the monetary theories underlying these observations,
NLP systems that largely dominate today’s state-of-the-art
we present a generalization of Goodhart’s Law for AI sys-
NLP leaderboards. These models may be trained end-to-end
tems: A tool for recognizing ambiguity in natural language,
on a specific narrow task like speech recognition. Due to the
once applied to a sufficiently intelligent AI will cease to be
brittleness of these models, the inevitable errors they make
effective and could be exploited. This is to say that any at-
on ambiguous input as a result of their normal operation con-
tempt to recognize ambiguity as in [Sproat and Santen, 1998;
stitute a safety risk in critical systems. In a poorly designed
Chantree et al., 2006] could be used by that AI to create a
voice activated self-driving car you might give the command,
dataset which upon self-training will allow the AI to generate
”Drive me up to Oxridge” which could be erroneously rec-
ambiguous language. This would allow the AI to write le-
ognized as ”Drive me off a bridge.” Of course this example
gal documents, software, news headlines, and tweets riddled
is only a safety risk if the natural language command is able
with ambiguity.4 This could have far reaching implications,
to override the car’s control system which is programmed to
including the potential for widespread disinformation cam-
drive only on drive-able space.
paigns and the disruption of systems discussed in the previ-
ous paragraph. Based on an existing categorization of Good- Secondly, there are interpretable NLP systems which
hart’s Law variants by [Manheim and Garrabrant, 2018], the nonetheless lack human-level AI capability and which in turn
dynamic illustrated here may well be considered an “adver- can pose an AI safety risk. Even if a system is able to iden-
sarial misalignment Goodhart [variant]” in which “The agent tify an interpretation of an ambiguity as more plausible (ei-
applies selection pressure knowing the regulator will apply ther statistically or using rule-based knowledge) there is no
different selection pressure on the basis of the metric”. In guarantee that in a given situation the maximum likelihood
this case, a bad actor, or the AI itself (subject to a utility solution is the correct one. For example, communication is
function perhaps) may use ambiguity detection not to notify increasingly being augmented with AI generated smart re-
users of the ambiguity, but to generate ambiguities, spark- ply suggestions. It could be the case that two business part-
ners routinely agree on contracts such that the AI’s training
4
Interestingly, an advanced AI would also be capable of attack- data is strongly biased towards replying yes to new contracts
ing narrow AI systems with some of the exploits on the left hand between them. Eventually though, there will be a contract
side of the Figure 2 such as the hidden voice command exploit. which one partner doesn’t agree on – but if the business is us-
ing an AI bot or AI negotiator, it will accept this bad contract References
due to its training data. [Ammicht et al., 2001] Egbert Ammicht, Alexandros
Finally there is the risk for human-level AI which can man- Potamianos, and Eric Fosler-Lussier. Ambiguity repre-
ufacture ambiguous legal reports, contracts, or software re- sentation and resolution in spoken dialogue systems. In
quirements to achieve unforseen objectives. Seventh European Conference on Speech Communication
and Technology, 2001.
7.4 Potential Solutions
[Anderson, 2004] Catherine Anderson. The structure and
For uninterpretable AI systems, there simply is no good way real-time comprehension of quantifier scope ambiguity.
to deploy these systems for critical applications without a PhD thesis, Northwestern University Evanston, IL, 2004.
way to quantify safety risk. Taking self-driving cars as an
[Banerjee and Pedersen, 2002] Satanjeev Banerjee and Ted
example, it is certainly possible to measure safety by track-
ing human driver disengagements of the AI control system. Pedersen. An adapted lesk algorithm for word sense dis-
The system can be deemed safe when these engagements are ambiguation using wordnet. In International conference
statistically unlikely. For AI systems where performance is on intelligent text processing and computational linguis-
not so easily quantified, there can be no safety guarantees. tics, pages 136–145. Springer, 2002.
Even then, a statistically safe system could still fail due to [Bar-hillel, 1964] Yehoshua Bar-hillel. A demonstration of
low probability events not captured in the system’s training the nonfeasibility of fully automatic high quality machine
data. It is not as obvious how statistical safety could be estab- translation, 1964.
lished for an NLP system. [Belinkov and Bisk, 2017] Yonatan Belinkov and Yonatan
For interpretable systems, human-in-the-loop and human- Bisk. Synthetic and natural noise both break neural ma-
AI teaming may solve ambiguity related AI risk provided that chine translation. arXiv preprint arXiv:1711.02173, 2017.
an AI system can quantify its uncertainty about a prediction
or detect ambiguous input to the system. Then in the case of [Bender and Koller, 2020] Emily M Bender and Alexander
ambiguity, the system can simply offload its decision making Koller. Climbing towards nlu: On meaning, form, and
to the human. understanding in the age of data. In Proc. of ACL, 2020.
One useful tool for natural language ambiguity might be [Blum-Kulka and Weizman, 1988] Shoshana Blum-Kulka
come from the idea of prefix codes which convert an alphabet and Elda Weizman. The inevitability of misunderstand-
into a unique set of binary ”codes” which can be concatenated ings: Discourse ambiguities. Text-Interdisciplinary
to form messages. These messages are ”uniquely decoded” Journal for the Study of Discourse, 8(3):219–242, 1988.
such that there is no ambiguity [Le Boudec et al., 2015]. This
[Carlini and Wagner, 2018] Nicholas Carlini and David
type of unambiguous communication might be usefully ex-
tended into NLP systems in order to achieve unambiguous Wagner. Audio adversarial examples: Targeted attacks
decoding, though it would require a more complex coding on speech-to-text. In 2018 IEEE Security and Privacy
scheme. Workshops (SPW), pages 1–7. IEEE, 2018.
[Carlini et al., 2016] Nicholas Carlini, Pratyush Mishra,
Tavish Vaidya, Yuankai Zhang, Micah Sherr, Clay Shields,
8 Conclusion David Wagner, and Wenchao Zhou. Hidden voice
We’ve detailed a non-exhaustive set of cases of ambiguity commands. In 25th {USENIX} Security Symposium
in natural language at the levels of phonology, syntax, se- ({USENIX} Security 16), pages 513–530, 2016.
mantics, and pragmatics along with contemporary NLP and [Carnie, 2012] Andrew Carnie. Syntax: A generative intro-
AI approaches to handling them. These areas taken together duction, volume 18. John Wiley & Sons, 2012.
give of good sense of the breadth of natural language ambigu-
ity. Furthermore, we claim that ambiguous communication is [Chantree et al., 2006] Francis Chantree, Bashar Nuseibeh,
inevitable when using an inherently ambiguous communica- Anne De Roeck, and Alistair Willis. Identifying nocu-
tion system. Despite the success of techniques including neu- ous ambiguities in natural language requirements. In 14th
ral networks, deep learning, adversarial examples, and bet- IEEE International Requirements Engineering Conference
ter language models (these are not mutually exclusive), many (RE’06), pages 59–68. IEEE, 2006.
worry that end-to-end systems only learn “surface” represen- [Chomsky, 2006] Noam Chomsky. Language and mind.
tations and don’t have true natural language understanding. Cambridge University Press, 2006.
We noted several areas where natural language ambiguity
[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken-
might lead to a failure mode in AI systems with deleterious
effects. The security risk is twofold – one set of concerns ton Lee, and Kristina Toutanova. Bert: Pre-training of
exists for narrow AI systems while a different set exists for deep bidirectional transformers for language understand-
human-level AI. Finally, we formulated a generalized Good- ing. arXiv preprint arXiv:1810.04805, 2018.
hart’s Law to express the idea that techniques allowing AI to [Doran et al., 2017] Derek Doran, Sarah Schulz, and Tarek R
identify, recognize, and detect ambiguity, might be reversible Besold. What does explainable ai really mean? a
in that they could be exploited to generate ambiguous lan- new conceptualization of perspectives. arXiv preprint
guage and “fool” their human users. arXiv:1710.00794, 2017.
[Ferreira and Henderson, 1991] Fernanda Ferreira and of the 10th international conference on Intelligent user
John M Henderson. Recovery from misanalyses of interfaces, pages 278–280, 2005.
garden-path sentences. Journal of Memory and Language, [Linzen and Baroni, 2020] Tal Linzen and Marco Baroni.
30(6):725–745, 1991. Syntactic structure from deep learning. arXiv preprint
[Forsberg, 2003] Markus Forsberg. Why is speech recogni- arXiv:2004.10827, 2020.
tion difficult. Chalmers University of Technology, 2003. [Liu et al., 2016] Quan Liu, Hui Jiang, Zhen-Hua Ling, Xi-
[Fuchs et al., 2008] Norbert E Fuchs, Kaarel Kaljurand, and aodan Zhu, Si Wei, and Yu Hu. Commonsense knowledge
Tobias Kuhn. Attempto controlled english for knowl- enhanced embeddings for solving pronoun disambiguation
edge representation. In Reasoning Web, pages 104–124. problems in winograd schema challenge. arXiv preprint
Springer, 2008. arXiv:1611.04146, 2016.
[Garcez et al., 2015] Artur d’Avila Garcez, Tarek R Besold, [Macagno and Bigi, 2018] Fabrizio Macagno and Sarah
Luc De Raedt, Peter Földiak, Pascal Hitzler, Thomas Bigi. Types of dialogue and pragmatic ambiguity. In Argu-
Icard, Kai-Uwe Kühnberger, Luis C Lamb, Risto Miikku- mentation and Language—Linguistic, Cognitive and Dis-
lainen, and Daniel L Silver. Neural-symbolic learning and cursive Explorations, pages 191–218. Springer, 2018.
reasoning: contributions and challenges. In 2015 AAAI [Manheim and Garrabrant, 2018] David Manheim and Scott
Spring Symposium Series, 2015.
Garrabrant. Categorizing variants of goodhart’s law.
[Gong et al., 2018] Zhitao Gong, Wenlu Wang, Bo Li, Dawn CoRR, abs/1803.04585, 2018.
Song, and Wei-Shinn Ku. Adversarial texts with gradient [Marcus, 2020] Gary Marcus. The next decade in ai: four
methods. arXiv preprint arXiv:1801.07175, 2018.
steps towards robust artificial intelligence. arXiv preprint
[Goodhart, 1983] Charles Albert Eric Goodhart. Monetary arXiv:2002.06177, 2020.
theory and practice: The UK-experience. Macmillan In-
[McCoy et al., 2019] Tom McCoy, Ellie Pavlick, and Tal
ternational Higher Education, 1983.
Linzen. Right for the wrong reasons: Diagnosing syntactic
[Hendriks and De Hoop, 2001] Petra Hendriks and Helen heuristics in natural language inference. In Proceedings of
De Hoop. Optimality theoretic semantics. Linguistics and the 57th Annual Meeting of the Association for Computa-
philosophy, 24(1):1–32, 2001. tional Linguistics, pages 3428–3448, 2019.
[Hoskin, 1996] Keith Hoskin. The ‘awful idea of account- [Mcmillan, 1998] David Mcmillan. “. . . say again?. . . ” mis-
ability’: inscribing people into the measurement of ob- communications in air traffic control, 1998.
jects. Accountability: Power, ethos and the technologies
[Navigli, 2009] Roberto Navigli. Word sense disambigua-
of managing, 265, 1996.
tion: A survey. ACM computing surveys (CSUR), 41(2):1–
[Jackson, 2018] Elizabeth Jackson. Words of many mean- 69, 2009.
ings. Schwa: Language and Linguistics, page 1, 2018.
[Partee et al., 2012] Barbara BH Partee, Alice G ter Meulen,
[Jia and Liang, 2017] Robin Jia and Percy Liang. Adversar- and Robert Wall. Mathematical methods in linguistics,
ial examples for evaluating reading comprehension sys- volume 30. Springer Science & Business Media, 2012.
tems. In Proceedings of the 2017 Conference on Empirical
Methods in Natural Language Processing, pages 2021– [Petroni et al., 2019] Fabio Petroni, Tim Rocktäschel, Se-
2031, 2017. bastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang
Wu, and Alexander Miller. Language models as knowl-
[Jurafsky, 2000] Dan Jurafsky. Speech & language process- edge bases? In Proceedings of the 2019 Conference on
ing. Pearson Education India, 2000. Empirical Methods in Natural Language Processing and
[Kiyavitskaya et al., 2008] Nadzeya Kiyavitskaya, Nicola the 9th International Joint Conference on Natural Lan-
Zeni, Luisa Mich, and Daniel M Berry. Requirements for guage Processing (EMNLP-IJCNLP), pages 2463–2473,
tools for ambiguity identification and measurement in nat- 2019.
ural language requirements specifications. Requirements [Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon
engineering, 13(3):207–239, 2008. Child, David Luan, Dario Amodei, and Ilya Sutskever.
[Le Boudec et al., 2015] Jean-Yves Le Boudec, Patrick Thi- Language models are unsupervised multitask learners.
ran, and Rüdiger Urbanke. Introduction aux sciences de OpenAI blog, 1(8):9, 2019.
l’information: entropie, compression, chiffrement et cor- [Resnik, 1999] Philip Resnik. Semantic similarity in a tax-
rection d’erreurs. PPUR Presses polytechniques, 2015. onomy: An information-based measure and its application
[Levesque et al., 2012] Hector Levesque, Ernest Davis, and to problems of ambiguity in natural language. Journal of
Leora Morgenstern. The winograd schema challenge. In artificial intelligence research, 11:95–130, 1999.
Thirteenth International Conference on the Principles of [Ross et al., 1996] Sheldon M Ross, John J Kelly, Roger J
Knowledge Representation and Reasoning, 2012. Sullivan, William James Perry, Donald Mercer, Ruth M
[Lieberman et al., 2005] Henry Lieberman, Alexander Davis, Thomas Dell Washburn, Earl V Sager, Joseph B
Faaborg, Waseem Daher, and José Espinosa. How to Boyce, and Vincent L Bristow. Stochastic processes, vol-
wreck a nice beach you sing calm incense. In Proceedings ume 2. Wiley New York, 1996.
[Ryffel et al., 2018] Theo Ryffel, Andrew Trask, Morten
Dahl, Bobby Wagner, Jason Mancuso, Daniel Rueck-
ert, and Jonathan Passerat-Palmbach. A generic frame-
work for privacy preserving deep learning. arXiv preprint
arXiv:1811.04017, 2018.
[Sakaguchi et al., 2019] Keisuke Sakaguchi, Ronan Le Bras,
Chandra Bhagavatula, and Yejin Choi. Winogrande: An
adversarial winograd schema challenge at scale. arXiv
preprint arXiv:1907.10641, 2019.
[Smolensky et al., 2016] Paul Smolensky, Moontae Lee, Xi-
aodong He, Wen-tau Yih, Jianfeng Gao, and Li Deng. Ba-
sic reasoning with tensor product representations. arXiv
preprint arXiv:1601.02745, 2016.
[Solé and Seoane, 2015] Ricard V Solé and Luı́s F Seoane.
Ambiguity in language networks. The Linguistic Review,
32(1):5–35, 2015.
[Sproat and Santen, 1998] Richard Sproat and Jan PH van
Santen. Automatic ambiguity detection. In Fifth Interna-
tional Conference on Spoken Language Processing, 1998.
[Subramanian et al., 2017] Sandeep Subramanian, Sai Ra-
jeswar, Francis Dutil, Christopher Pal, and Aaron
Courville. Adversarial generation of natural language.
In Proceedings of the 2nd Workshop on Representation
Learning for NLP, pages 241–251, 2017.
[Wallace et al., 2019] Eric Wallace, Pedro Rodriguez, Shi
Feng, Ikuya Yamada, and Jordan Boyd-Graber. Trick me if
you can: Human-in-the-loop generation of adversarial ex-
amples for question answering. Transactions of the Asso-
ciation for Computational Linguistics, 7:387–401, 2019.
[Wasow et al., 2005] Thomas Wasow, Amy Perfors, and
David Beaver. The puzzle of ambiguity. Morphology and
the web of grammar: Essays in memory of Steven G. La-
pointe, pages 265–282, 2005.
[Wilson, 2006] Deirdre Wilson. The pragmatics of verbal
irony: Echo or pretence? Lingua, 116(10):1722–1743,
2006.
[Wu et al., 2018] Lijun Wu, Yingce Xia, Fei Tian, Li Zhao,
Tao Qin, Jianhuang Lai, and Tie-Yan Liu. Adversarial neu-
ral machine translation. In Asian Conference on Machine
Learning, pages 534–549, 2018.
[Yampolskiy, 2016] Roman V Yampolskiy. Taxonomy of
pathways to dangerous artificial intelligence. In Work-
shops at the Thirtieth AAAI Conference on Artificial In-
telligence, 2016.
[Yampolskiy, 2019a] Roman V Yampolskiy. Unexplainabil-
ity and incomprehensibility of artificial intelligence. arXiv
preprint arXiv:1907.03869, 2019.
[Yampolskiy, 2019b] Roman V Yampolskiy. Unpredictabil-
ity of ai. arXiv preprint arXiv:1905.13053, 2019.
[Yarowsky, 1995] David Yarowsky. Unsupervised word
sense disambiguation rivaling supervised methods. In 33rd
annual meeting of the association for computational lin-
guistics, pages 189–196, 1995.
[Yu and Deng, 2016] Dong Yu and Li Deng. Automatic
Speech Recognition. Springer, 2016.