This paper4 presents some preliminary results after having analyzed the metadata of 146,585 and the full text of 9,879 relevant AI-related papers. Other datasets collected from different sources were prepared for further analysis, too. The idea is to investigate the use of capabilities that denote intelligence in the AI scientific discourse with the goal to track the evolution of the narrative around intelligence and its manifestations in humans and machines over time. Intelligence capabilities and related properties that are used when defining intelligence were extracted from previous work in this area in the form of a catalog and spanning over different fields, AI included. Although still an elusive, difficult to define concept, analyzing and understanding the discourse around intelligence may shape both the way we use it and how intelligent artifacts are and will continue to be developed in the future.
To the best of our knowledge, there is no known study that tracks the historical development of the terminology used in intelligence research, nor how it has affected both the evolution of machine or artificial intelligence (AI) and the language used by the machine intelligence community to define it.
The reasons are many and have been accentuated along the years: the absence of a common language around AI is a missing piece for a wide-ranging understanding of what intelligence means; there is no broadly accepted definition of machine intelligence; the field is multidisciplinary and almost each researcher defines intelligence through the lens of their discipline; and the exacerbated media AI hype from the last few years attempts against more understanding, fuels division (not only) inside the AI community, and deviates investments to the areas where only rapid gains can be attained, among other reasons.
But that situation must change and that is the main objective of the research project introduced in this paper. The research is based on an exhaustive analysis of more than 1,6 million of scientific papers. The central goal driving the research lies in discovering of how the vocabulary on intelligence, in particular, and on AI, in general, have shaped the discourse of the scientific community.
This would have several positive implications not only for the AI community, but also for those stakeholders outside it. For example, journalists and PR people, politicians, regulators, AI novices, and researchers from other fields, to name just some of the stakeholders that have been using more and more the AI vocabulary in their work, would be aware of the preferred scientific terminology by the AI community, would be able to study its history in a more supporting way, and would be exposed to a common language that facilitates understanding and dialogue across disciplines. 1.1
Although there has been a recent and growing interest in characterizing, structuring, and tracking the evolution of AI in particular and of intelligent systems in general, there has been very little research on finding a common language around both intelligence and AI, and no research at all, to our best knowledge, on tracking the evolution of definitions of intelligence in the scientific literature. This is why the research presented here may be the first of its kind.
Already in [
The central research goal of the project is to study the shifting focus of intelligence
research. This goal is subdivided into the following sub-goals:
– Sub-goal 1: To study how the vocabulary on intelligence has been created and used
by the research community along the years. This sub-goal can be accomplished by
analyzing how the concepts from the intelligence catalog that was created in [
This paper contributes to most of the research sub-goals mentioned above. 2
The research project consists of the following phases:
1. Data collection and preparation: Gradual, automated download of metadata to
create a new corpus of scientific literature on intelligence and AI:
(a) All papers published in the Computer Science category “artificial intelligence”
(cs.ai) of the arXiv e-prints.
(b) All papers published in other Computer Science categories of the arXiv e-prints
that might be closely related.
(c) Other AI-related papers and scientific literature (depending on availability), as
studied in [
(d) Other sources like leading conferences and workshops on AI, for instance.
2. Data pre-processing and analysis using machine learning techniques like natural
language processing (NLP), as well as statistical analysis. The intelligence catalog
available at https://tinyurl.com/intelligenceCtlg, as well as other results from [
It is expected that the phases involving the collection of the data and its pre-processing might likely consume a huge amount of time compared to other phases, as it is usually the case in related work.
Two main sources for tracking the use of intelligence capabilities in the scientific literature were considered for analysis.
The first source includes attributes that contributors to two symposia on intelligence
used in their definitions. These were compiled by Sternberg and Detterman in [
The second literature source includes words extracted from a corpus of definitions
and experts’ comments on defining intelligence that conform the Intelligence Catalog or
vocabulary presented in [
The vocabulary used to define intelligence in the first literature source might be biased, however: it considers contributions by only 39 psychologists in total that come only from US and Europe. Yet, the vocabulary from the second literature source is much broader and may counter the bias: a total of 567 individuals coming from more than 57 countries and from more than 15 different fields participated. Furthermore, more than 343 definitions of intelligence, both of human and machine (or artificial) intelligence, were examined. 3.1
Collecting, converting, and preparing the data made it to be the most consuming tasks
of the whole project. In particular, the data was collected in the following way:
– The datasets D01, D02, and D03 were provided by Fernando Mart´ınez-Plumed,
the leading author in [
Dataset Source
No. of papers Time period Related work Relevant data 41,000 – The dataset D05 contains the metadata of papers published in the arXiv category cs.ai, available at https://github.com/karenhao/techreview arxiv scrape. It might be a subset of D04; thus, we considered repetitions of single papers for deletion. – The dataset D06 contains the metadata of all arXiv papers published in all Computer Science categories (40 in total) through the end of 2018; thus, the datasets D04 and D05 might be subsets of it. It should be noted that papers uploaded to arXiv can be submitted to more than one category. Thus, it was important to check for duplicates in the dataset D06 since it is formed by putting together 40 different categories (or files with metadata) from Computer Science. – The dataset D07 contains only the titles of all arXiv papers published in all categories (175 in total) through the end of 2018. It helps in identifying how some concepts from the intelligence catalog have been used in other fields (e.g. in Astrophysics, Quantitative Biology, Economics, and so on), at least in their titles. – The dataset D08 contains the metadata of all papers published in the ECAI proceedings from 2000 to 2016, as it can be found in http://frontiersinai.com/. – The dataset D09 was constructed after downloading the metadata of the ICML papers from dblp (Digital Bibliography and Library Project). This information can be found at urlhttps://dblp.org/db/conf/icml/index. – Finally, the dataset D10 was constructed after downloaded the IJCAI papers from the website of the conference (see https://www.ijcai.org/Proceedings/). This dataset contains the whole text of the papers published in the IJCAI proceedings until the year 2020 (all as PDF files).
Most of these datasets have been already considered in some detail in the first phase of the research project; see the next sections for more. 4
Python was used for processing and analyzing the data.6 Among the libraries used are tz and PyMuPDF, which are used to read the text from the PDF files (in the case of the IJCAI papers), and Matplotlib and NumPy, which are used to generate different diagrams. Pandas is used for reading the Excel files, data cleansing, data formatting and analysis, for it provides several useful operations for one- and multidimensional arrays.
Figure 1 illustrates the major phases and data flow that constitute our pipeline when processing the datasets.
The configurator reads the configuration files, one with the column names of the
relevant metadata, one with the keywords or intelligence capabilities to be searched
for.7 Then, the configurator matches that information with the data contained in the
datasets and prepares it in order to be processed by the other components of the
analysis tool. The datasets are then cleansed and irrelevant metadata and content are deleted.
Other common NLP pre-processing tasks are performed as well, like lower-casing the
keywords, deleting emoticons and other especial characters, and converting the data
to CSV data files for later processing, among others tasks. The remaining content is
formatted to a convenient representation. The analysis then starts by considering the
year of publication of the different scientific papers and the frequency of their
publication. Moreover, the occurrences of keywords are counted thereby avoiding occasional
repetitions from the same entry. Percentages relative to the number of papers per year
6 Both the source code of the Python scripts and the data that the authors of this paper collected
are available upon request. They will be made available for others to download and work with.
7 In what follows, we refer to keywords to denote the intelligence capabilities from the
intelligence catalog presented in [
The corresponding analysis tool is much simpler for the case of the IJCAI papers which content is available. It consists of only two components: the PdfMiner and the analyzer. The former extracts the text from the PDF files and calculates the frequency of the keywords that are of interest. The latter reads the processed data, generates suitable diagrams, and saves the results in a specified folder. Also convenient NLP tasks of the pre-processing are performed here. It is worth mentioning that reading the PDF files with the appropriate functions from the PyMuPDF library is much faster (about 0.001 seconds per PDF file) than doing it with other libraries (like textract, Apache Tika or PyPDF2). 5
More than 100 diagrams were produced for visualization and analysis. Some of them will be commented in what follows, mainly in a descriptive way.
For all the considered datasets, the number of papers published per year has increased dramatically in the last decade (more exactly: since 2010), almost showing an exponential growth.
A first analysis of the frequency which with the intelligence capabilities and related attributes from the intelligence catalog appear in the datasets D01 to D10 helped to distinguish three different stages:
2. AAAI, IJCAI, ECAI and arXiv>cs>ai: Recent past. 3. ICML: Machine Learning in focus.
Other words, however, are getting more importance as time pass (e.g. adapt, learn,
predict) and in so doing, they are documenting the increasing momentum that the AI
field has been experiencing in the last decades. It is not surprising that some of them are
even considered to be essential capabilities of machine intelligence, like in the widely
accepted definition of intelligence by Wang: “The essence of intelligence is the
principle of adapting to the environment while working with insufficient knowledge and
resources. Accordingly, an intelligent system should rely on finite processing capacity,
work in real time, open to unexpected tasks, and learn from experience” [
The arXiv papers go back only to 1993 and the collected metadata from AAAI and IJCAI goes back only to 1997. The AI field is much older, though.
The ECAI proceedings are a very good source of information and the conference takes place since 1974, but only a few proceedings are available online (from the year 2000 onward, in a two-year frequency).
Furthermore, there are many other AI conferences and workshops that take place every year, some of them with a long tradition inside the AI field, that are not considered in this research project. For example, NeuriIPS electronic proceedings are available online for the time period 1987-2018, but this is a conference with focus on specific areas of Machine Learning that does not cover other important AI sub-fields. Nevertheless, it will be considered in further work. Data has been also collected and is available for similar conferences, but the time periods are very limited or the proceedings are not available fro download. The code for scraping the data might be available, too. Some of these conferences have a narrower focus, though.
Another possible limitation of this work and the ongoing bigger research project that includes it, as it was rightly pointed out by one of the anonymous reviewers,8 is that we don’t take into account how the English language nor how some words’ meanings, especially in the field and sub-fields of AI, have evolved over time. However, we are of the opinion that our work would contribute to reaching that goal in a better way, especially when considering a semantic analysis (topic of current work), because it would be easier to detect in which contexts which intelligent capabilities and narratives have been used by the scientific community. 7
Although preliminary, the data collection, preparation, and analysis, together with the
first results presented in this paper, set the basis for a much broader analysis of how
the terminology is used to define intelligence has evolved over time. More work in this
regard is needed, however. For example, semantic aspects that allow for distinguishing
specific contexts on one side, and the relative importance of the words used to define
intelligence on the other side, would help in framing a more accurate use of the
terminology on intelligence over time. This is the topic of ongoing work. A similar analysis
was already initiated in [
Finally, there is an extensive body of work on intelligence outside the AI community (for example in Psychology and Neuroscience, to name two of the most prominent fields in that research area) that must also be considered in future work. Although the vocabulary on intelligence used in these fields does not suffer from the same lack of consensus the AI community cannot put aside, acknowledging their lessons learned and the good practices would benefit both the AI field and the discourse on intelligence inside and outside it.
In order to accomplish these new goals, it would first be needed to harvest and process the new data, i.e. to collect, analyse, and pre-process, for example, already published papers and their metadata in the fields of Psychology, Neuroscience, and the like. These tasks should not be underestimated; they are actually the most time consuming tasks in any machine learning project.