Artificial Intelligence (AI) algorithms permeate many of the systems and devices we interact with in everyday life. Since its conception as a field, a wide variety of these intelligent algorithms have sought to simulate or even surpass human cognitive abilities and behavior. However, there is as yet no widely accepted definition of what intelligence in machines means, nor of human intelligence. The primary goal of this paper is to propose an intelligence vocabulary or catalog in the quest for the boundaries that shape the current discourse of the experts on intelligence. The idea is to provide and inform researchers, practitioners, journalists, and policymakers, among many others, with a terminology that can be used when defining intelligence. Considering these challenges, we analyze the data from hundreds of experts around the world, provided when they were asked to give both their opinions on existing definitions of intelligence in the scientific literature and their own suggested definitions. All of the answers that were gathered (despite being subjective views) were evaluated using stateof-the-art text analytics processing and AI mechanisms. This ensures objectivity and allows us to extend (or even reproduce) the study in the future. Ultimately, our work contributes to strengthen the bridge between human and automatic reasoning: we examine experts' opinions on definitions of human and machine intelligence with both manual and automatic methods. Some linguistic tasks like normalizing and clustering the opinions are performed automatically, thereby the twofold goal being to find an overview and to enable a drill-down to the most interesting answers. These individual artefacts are then interpreted manually. We hope that the proposed intelligence vocabulary will not only contribute to defining (machine) intelligence better but also to an understanding of both the current views of experts on intelligence and intelligence itself. This would help to frame a common language around AI, which has unfortunately been absent thus far. In the future, extending the procedure presented in this paper might lead to an interdisciplinary machine-assisted method for extracting knowledge from subjective opinions.
Over the last 100 or so years, the concept of intelligence has been defined on
numerous different occasions and in different fields, both formally and informally. There are
many informal definitions of both human and machine or artificial intelligence (AI).
For example, Legg and Hutter [
Several works have tried to characterize and structure the field of AI. For example,
researchers at Elsevier [
This was not the first time that the evolution of AI as a discipline had been the
focus of research. In [
However, although there has been a recent and growing interest in characterizing, structuring and tracking the evolution of AI in particular and of intelligent systems in
general, there has been very little research on finding a common language around intelligence and AI. The two-fold purpose of this paper is therefore to use AI to understand intelligence and to help to frame a common language around defining it. To reach these goals, we apply various AI techniques to extract the most representative verbs, nouns, adjectives, and adjective-noun phrases contained in experts’ opinions that might indicate cognitive and behavioral abilities (or capabilities) that could be used when defining intelligence. 1.1
Our research starts with the analysis of two data sources: respondents’ written opinions
to the AGISI research survey “Defining (machine) Intelligence” [
The diversity of opinions that were collected reflects the current vastly mixed landscape of research, development and theory in AI. Most of the respondents to the survey were AI experts with several years of experience in the field, and an overwhelming majority were explicitly invited to participate in the survey due to their research in AI or intelligence-related areas. They originated from 57 different countries and more than 184 institutions around the world. They worked mainly in academia (N =441, 79.3%) and industry (N =114, 20.5%), and had primary roles as researchers (N =424, 76.3%) and/or educators (N =193, 34.7%). This was the first time that such a contemporary collection of opinions on and new suggested definitions of human and machine intelligence had been gathered.
A thorough analysis of this information could have a significant impact on the AI field for several reasons. Firstly, to obtain clarity around defining intelligence (and AI as a field), we must consider experts’ opinions and the current debate on AI. Definitions of intelligence (and especially machine intelligence) should not rely solely on published content, since the scientific literature does not fully reflect the evolving dynamics of the field. Shaping the boundaries of the current scientific discourse is necessary, and thus is important for the advancement of the field.
Secondly, there is no consensus definition of AI, let alone of intelligence; this is
primarily because the former has evolved with a fluid definition due to the varied human
conceptions of the latter [
Thirdly, it matters how and by whom the AI field is defined [
Due to the overwhelming response rate to the survey, a manual analysis of the data was difficult. We therefore decided to apply AI techniques ourselves, and to use text mining to analyze the results. The following sections describe in more detail how these techniques were applied to extract and analyze the experts’ opinions on and their new suggested definitions of intelligence, as introduced above. As a result of this work, an intelligence catalog that includes cognitive and behavioral capabilities is proposed that can be considered when defining intelligence, and this forms the main contribution of this work. With this, we hope to contribute to a common language around both intelligence and AI. 2
Two corpora were created: the first consisted of 4,041 experts’ opinions, of which 2,424 were on definitions of machine intelligence (MI) and 1,617 on definitions of human intelligence (HI) from the literature; the second consisted of 338 new, suggested definitions of intelligence, of which 213 were suggested for machine intelligence and 125 for human intelligence. The easiest way to look at the data in these corpora was by applying simple text analytics techniques for gathering information about the number of words that are used, together with their frequency of occurrence, the number of sentences and so on. The corpus of opinions, for example, contains more than 71,000 words. This work would be very exhausting to do manually, and the separation of verbs, nouns, and other parts of the speech into unigrams, bigrams, etc. for later processing would be very difficult and time-consuming. Other techniques and algorithms should therefore be used instead.
Our pre-processing of the data from the corpora consisted of the following phases and steps (see Figure 1):
Data preparation: First, the raw data from the survey was converted into a format ready for further processing. Then, all personal information about the survey participants (like name, age, ethnicity, institution, email, etc.), available in separate fields, was removed.
Cleaning: The data was cleaned. For example, opinions containing a URL as only content were deleted. Some intelligence-specific words might be losing their significance if they are not spell checked or have grammatical errors in their occurrences. Thus, a spelling and grammar check was carried out.
Natural Language Processing (NLP): Part-of-speech (POS) tagging was performed. This allows for the identification of verbs, adjectives, nouns, etc. in a sentence according to their meaning and context. Other techniques that were applied in this phase included tokenization and lemmatization. Stop words and unimportant characters like non-printable characters and emojis were also removed.
Analysis: This phase comprises the following steps: i) Statistical analysis: Word clouds, heat maps and histograms were generated, among other graphics, together with some general statistics. This allowed for an initial understanding of the data and the subsequent inclusion of other analyses; ii) Unsupervised learning: We used a slight modification of topic modeling, aggregating the content written by individual authors and using these documents as basis. We refer to these as data-driven persona models; iii) Semantic analysis: A word embedding was created and trained to convert words to vectors. This is useful for finding semantic relations between definitions of intelligence, respondents’ opinions, etc. We started with a simple bag-of-words model and refined it using TF-IDF to penalize frequent words with little distinction quality. To obtain a better understanding of the semantics, we analyzed both unigrams and bigrams.
Our analysis pipeline depended mainly on open source software. Data preparation
was carried out with Pandas (https://pandas.pydata.org), and for cleaning we used GNU
Aspell (http://aspell.net). NLP was carried out using spaCy (https://spacy.io) and we
chose scikit-learn (https://scikit-learn.org) for machine learning and topic modeling.
Semantics were extracted using word2vec [
Our initial approach to interpreting the data consisted of the analysis of 18 word clouds
created in the final phase of the pre-processing. They included the most common verbs,
nouns, adjectives, and adjective-noun combinations used in the experts’ opinions on
definitions of human and machine intelligence from the literature, the most common
verbs used in opinions about the three most agreed upon definitions of human
intelligence (Gottfredson’s [
Intelligent capabilities are mostly described in ostensive and operational ways, i.e. by
exemplifying and using the cognitive and behavioral characteristics of humans, agents,
or intelligent systems in general that denote or are important in achieving intelligence
[
A first analysis of these verb word clouds indicates the presence of two types of verbs used by experts in their arguments for or against existing definitions and in their suggested definitions: not only verbs that exemplify intelligent capabilities, such as think, anticipate, and feel, but also those that complement respondents’ written speech when justifying their opinions, such as define, agree, and include. The former are important in constructing a catalog of intelligence. Furthermore, the verbs used in the suggested definitions shape the current discourse on intelligence, whilst those used when criticizing definitions from the literature may refer to issues that are no longer relevant or that should not be considered when defining intelligence.
It is worthwhile to analyze in depth those definitions for which the level of
agreement was positive, and thus less polarized. Although comments expressing positive
agreement were many fewer than those expressing disagreement [
A closer look at the verbs used in the comments on the most agreed-upon definitions shows a similar trend. Again, two types of verbs can clearly be identified. The process of constructing the catalog of intelligence should identify only those verbs that denote cognitive and behavioral capabilities. This is not a task that can be automated easily, however, and depends heavily on the context. There are sometimes verbs that could be of both types, such as define: compare “The goal is to define intelligence easily” to “An agent can define its own goals.” In the latter example, there is a cognitive capability that could denote intelligence. This would mean that the data pre-processing and application of NLP techniques in conjunction with visualization using word clouds can provide “the first filter,” but the remainder would mostly depend on human work. 3.2
Other words besides verbs can indicate cognitive and behavioral capabilities that denote intelligence, for example adjective and nouns. We analyzed the most commonly used adjectives and their combinations with nouns for all of the experts’ comments on definitions of intelligence from the literature and for their suggested definitions in the corpora. Again, the most frequently used words included the two types discussed above: those that could be used when constructing the catalog of intelligence, and those used to complement the respondents’ written speech. For example, the adjectives cognitive, rational, and adaptive may be related to properties of intelligence, but vague, fuzzy, and restrictive may form part of a negative opinion on a concrete definition that is being criticized. A property of intelligence should not be vague, although a definition may be vague. This distinction is more evident in the adjective-noun phrases, since bigrams give more information about the context than unigrams. Furthermore, adjectives and nouns arise in respondents’ opinions for or against definitions from the literature that are used to qualify those definitions (for example good, narrow, general definition) and that are not used at all in the respondents’ new, suggested definitions of intelligence. Thus, the parts of speech from the latter can better contribute to a catalog of intelligence. 3.3
Figure 3 shows a heat map of the similarities between the opinions given by respondents to the definitions of human (top left side) and machine (bottom right side) intelligence, in descending order of similarity in each category. The higher the similarity values (i.e. closer to 1), the more similar the level of agreement between two definitions. The similarity values were calculated using the cosine similarity measure of summed TFIDF vectors. These vectors give information on the frequency of the words used by respondents in their opinions against their importance with respect to the other opinions in the corpus. No other similarity measures were considered at this stage; further work could include an extended comparison to other methods.
There is a remarkable low similarity in the opinions on definitions of machine
intelligence and those given for human intelligence. The similarities between the arguments
that were given to justify the level of agreement with definitions of machine intelligence
(lower right-hand side) are much more evident (have higher values) than when they are
compared to human intelligence (lower values in the top right and lower left quadrants).
Correspondingly, the similarities between the opinions given for definitions of human
intelligence (top left-hand side) are similar to each other but not to the opinions given
for machine intelligence (except for HI8 and MI6). Once again, this confirms the
findings in [
Figure 4 shows a heat map of the similarities between the experts’ opinions and their
new, suggested definitions of machine and human intelligence; the higher the similarity
values, the more similar the terminology used in the opinions on the definitions in the
literature to the new, suggested definitions. As can be interpreted from the figure, the
respondents tended to (re-)define both machine and human intelligence using much of the
same terminology (and thus intelligent capabilities) that was used when commenting on
definitions of machine intelligence from the literature (see the higher similarity values
on the left-hand side). However, the vocabulary used in their opinions on the definitions
of human intelligence was much less often used in their suggestions for new
definitions (see lower similarity values on the right-hand side). In other words, the suggested
definitions of both machine and human intelligence seem to require much of the same
vocabulary that was previously used to subjectively evaluate definitions of machine
intelligence from the research literature. This may be another reason for supporting two
separate definitions of intelligence. We note that 41.9% of the respondents to the
research survey on defining intelligence supported the need for only one definition, in
contrast to 48.2% who preferred two separate definitions, as concluded in [
Interestingly, the new, suggested definitions of human intelligence used a vocabulary that was very similar to that used to comment on the definitions of machine intelligence that were not the most agreed upon. Three of the four highest similarity values (see the darker colors at the top left-hand side of Figure 4) corresponded to opinions given on the definitions of machine intelligence that were not the most agreed upon.6 The question arose as to whether the ratings given by experts, i.e. their level of agreement with the definitions of intelligence from the literature, were correlated with the number of words they wrote in their opinions. However, these were not correlated, i.e. a significant relationship between the length of the opinions and the level of agreement was not observed. Due to space constraints, detailed results are not include here. The interested reader can receive more information about the concrete calculations upon request.
Does the vocabulary used in the opinions depend on respondents’ years of experience in the field? To answer this question, we again aggregated the TF-IDF vectors, but this time for respondents with similar numbers of years of experience. It turns out that novices with less than five years of experience and experts used a similar vocabulary.7 The same is true for respondents with a considerable amount of experience. However,
the similarity values decreased with the level of experience in the field. One possible explanation is that greater expertise not only means a deepened knowledge of AI but also a wider knowledge of other fields, which may enrich the experts’ point of view. This may empower the use of a more sophisticated, distinctive or selective vocabulary that occasionally also contributes to the discourse with new terminology. We can surmise that experts not only have knowledge that goes beyond others’ grasp, but that they also use it actively to produce new information, in this case opinions on definitions of intelligence from the scientific literature.
The intelligence catalog was created with the most representative verbs, nouns,
adjectives, and adjective-noun phrases (i.e. bigrams) that were extracted after applying text
mining and machine learning techniques, and the analysis introduced above. They were
manually filtered out from the top 100 occurrences of each word type after considering
the techniques presented in this paper. These extracted words help us to distinguish the
boundaries of the discourse around definitions of intelligence from the scientific
literature and to shape the current experts’ view around how intelligence should be defined
instead. Figure 5 shows the main categories and subcategories of the intelligence
catalog and Table 1 shows a list with the most common words that were extracted. For
example, the top five verbs that are related to the internal processing of the intelligent
entity are learn, solve, achieve, adapt, and understand, in this order. The more
general categories are a suggestion from the authors and relate to their expert knowledge
in the intelligent agents and multi-agent systems domains (i.e. they are not extracted
automatically nor any ontology engineering is used, yet).
Verbs related to perception (with at least five occurrences):
perceive [
The findings presented in this paper clearly suggest that the current discourse on intelligence (and on AI) requires a deeper analysis of how intelligence is defined. This should be the first step in understanding intelligence. The aim of the present study was therefore to use AI in helping to frame a common language around AI and intelligence by analyzing thousands of experts’ opinions on definitions from the scientific literature and their new, suggested definitions of intelligence. As a result, an intelligence vocabulary is suggested that could be used when defining intelligence.
The possible uses of such a vocabulary are many. For example, it could be used by lecturers when introducing the concept of (machine) intelligence in their courses; or by researchers when defining the goals of their artificial intelligence-related research; or by journalists when reporting scientific results in the neurosciences, psychology, or AI fields, to name a few; or by policy makers, lawyers, and ethicists when delineating recommendations and regulations on how to design, deploy, and use intelligent systems, because for regulating something, that something must be well-defined. In short, we are of the opinion that the intelligence vocabulary could be essential when developing a common language around AI and intelligence.
A wider discussion remains open, however, and it includes questions such as: to what degree can the resulting capabilities be considered properties of intelligence? Which capabilities or parts of the intelligence catalog are related to which specific areas from the AI field? Is there any distinction between capabilities that relate only to humans or only to machine or non-human intelligence? The intelligence vocabulary offers one of several initial approaches to these questions, but further research is needed before we can achieve a common language around AI and a satisfactory understanding of what intelligence is.