=Paper=
{{Paper
|id=Vol-1964/CS1
|storemode=property
|title=Automated IQ Estimation from Writing Samples
|pdfUrl=https://ceur-ws.org/Vol-1964/CS1.pdf
|volume=Vol-1964
|authors=Austin Hendrix,Roman Yampolskiy
|dblpUrl=https://dblp.org/rec/conf/maics/HendrixY17
}}
==Automated IQ Estimation from Writing Samples==
https://ceur-ws.org/Vol-1964/CS1.pdf
Austin Hendrix and Roman Yampolskiy MAICS 2017 pp. 3–7
Automated IQ Estimation from Writing Samples
Austin Hendrix, Roman Yampolskiy
University of Louisville
Louisville, KY 40208
austin.hendrix@louisville.edu, roman.yampolskiy@louisville.edu
Abstract demonstrates certain writing trends are independent of the
authors’ language and are therefore likely stronger
The primary focus of this research is to introduce a candidates for comparing authors that write in different
method of measuring an individual’s IQ by analyzing languages.
the vocabulary in said individual’s writing. In this
paper, we show that the ratio of SAT words in a As there is no true scientific measurement that is
dataset of writing samples is roughly normally currently used to quantify someone’s intelligence, many
distributed, though with an obvious left skew. We go different measurements have been used. Intelligence tests
on to show a method that can be used to calculate an have often been a common way to determine an individual’s
individual’s IQ with this ratio and provide samples intelligence relative to others. There have been many
with measured accuracy. The conclusion suggests negative and controversial opinions on these tests, yet
ways to increase accuracy in order to further develop experts still agree on their overall usefulness (Snyderman,
the research along with applications of doing so. Rothman; 1989). Further studies have shown that a standard
intelligence test provides the best single, reliable predicator
1 Introduction of academic aptitude (Bullerdieck, 1985). One popular
example of standard intelligence tests measures an
Stylometry is the statistical analysis of differences in individual’s Intelligence Quotient (IQ). The assumption
literature between authors (Franking, 1988). As early as behind this system of measurement is that if a large sample
1880, the study of stylometry has been used as a method of of IQs are mapped together, the distribution will be normal.
authorship identification on disputed texts. With the It has been shown that there are issues with the structure and
development of computers and automation techniques, quality of the standard IQ test (Lawler, 1977). Still, the IQ
sylometric analysis has become easier. An early example of test can be a useful way for individuals to compare
software defined stylometry was used to identify the author intelligence. For this paper, we will act under the
of the disputed papers amongst the “Federalist Papers” assumption that an individual’s IQ score relates directly to
(Tweedie, Singh, Holmes; 1996). This work demonstrated their true intelligence level.
that stylometric analysis using automation is, at least in this
application, able to draw similar conclusions about the This preliminary research project is focused on
authorship of these papers as previous work on the subject. exploring whether an individual’s IQ can be determined by
In recent years, stylometry has taken on a broad range of using software defined stylometry. The novelty of this
applications. More specifically, stylometry has been used in process is that it is not centered around author identification.
the identification of chat bots (Ali, Hindi, Yampolskiy; Instead, stylometry will be used to determine the relative
2011). Further research was done to show that when a chat writing quality of a known author. The process will involve
bot changes behavior over time, the stylometry approach analyzing an attribute of a known author’s writing to
becomes more difficult (Ali, Schaeffer, Yampolskiy; 2012). determine said author’s IQ. There are multiple attributes of
In addition, it has been demonstrated that stylometric author writing that are potential candidates for this application. For
identification processes can be used on a single author that the beginning of this research, we will focus on the
is capable of writing in multiple languages. (Ali, individual in question’s vocabulary. Other research has been
Yampolskiy, 2014). This is significant in that it done to discuss other attributes with possible merit. These
3
Automated IQ Estimation from Writing Samples pp. 3–7
attributes include, but are not limited to, word-length, Now that we have a clearly defined a process for calculating
syllables, sentence-length, and distribution of parts of the CWR of a sample, we need to execute this software on a
speech (Holmes, 1994). large dataset. An ideal dataset would consist of writing
samples by many randomly selected individuals. Along with
2 Collegiate Word Ratio this, each writing sample would represent each individual’s
average writing ability. As such a dataset was not available
To determine an individual’s IQ based on their vocabulary, to the authors of this project, another source had to be
a quantitative way to measure the quality of their vocabulary found.
is necessary. For the purposes of this project, we will define
a “Collegiate Word” as a word the SAT considers a part of The Common Crawl is a corpus containing raw
strong vocabulary usage.1 The College Word Ratio (CWR), web page data, extracted metadata and text extractions.2 The
which we will refer to through this paper, is therefore text extractions from this corpus contain the raw text taken
defined as: directly from websites. We are acting under the assumptions
that the text extractions are all written by humans and likely
Collegiate Word Ratio = Collegiate Word Count contain that individual’s average writing. To help increase
/ Total Word Count the accuracy of results under this assumption, only samples
with more than 100 words were used. After collecting a
The CWR of each sample will be measured by software and
large number of samples from the Common Crawl corpus,
then compared to the rest of the samples to determine its
each sample’s CWR was stored and mapped onto a
relative quality by use of a distribution. A pseudo-code for
distribution (Figure 2). The distribution is fairly normal,
calculating the CWR of a sample is shown in Figure 1.
though there is a slight skew to the left. This implies that on
for SampleWord in Sample: a large number of samples, the distribution of CWR is fairly
for CollegiateWord in CollegiateWordList: normal and resembles the distribution of IQs.
if SampleWord == CollegiateWord:
CollegiateWordCount++
CollegiateWordRatio = (CollegiateWordCount /
SampleWordCount)
Figure 2: Calculating CWR Pseudo-code
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2
Collegiate Word Ratio
Figure 1: Collegiate Word Ratio Distribution
1
The full list of words used for this project can be found at
2
www.freevocabulary.com. https://aws.amazon.com/public-datasets/common-crawl/
4
Austin Hendrix and Roman Yampolskiy MAICS 2017 pp. 3–7
55 65 75 85 95 105 115 125 135 145 155
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18
CWR Curve IQ Curve
Figure 3: CWR VS IQ Distribution
3 Determining IQ from CWR mean of both distributions. For the IQ curve, these are fixed
values. The mean IQ value of all individuals is said to be
We have shown the distribution of CWR is fairly normal, 100 and the standard deviation of all IQ values is said to be
and now we will demonstrate the process of using CWR to 15. For our data set, the mean CWR is 0.074759005 and the
calculate an individual’s IQ. A graph showing these two standard deviation is 0.031552108.
distributions overlaid is located below (Figure 3).
Using these values and an induvial data point’s
The IQ curve shown is the ideal expected IQ CWR, a corresponding IQ score can be calculated.
distribution. It is perfectly normal with a mean of 100. The Performing this calculation involves finding the z-score of
CWR distribution, though skewed slightly left, is mapped the data point. This is done by the following:
very closely to the IQ distribution for the second and third
positive standard deviation from the mean. For the purposes Z-Score = (CWR Data Point – CWR Mean) /
of this analysis, we will assume that this indicates the CWR CWR Standard Deviation
in this area will map onto its corresponding IQ. This will
result in a certain amount of error when calculating IQ from
CWR. Nevertheless, the distributions are close enough that
the process should give a good estimation of an individual’s
IQ.
To begin the process of transferring between the
two curves, we need to know the standard deviation and
Sample World Sample Collegiate Word Sample Expected Measured %
Length Count CWR IQ IQ Error
752 94 0.1250 153 123.88 19.03
412 51 0.1238 130 123.31 5.15
136 22 0.1618 141 141.36 0.26
3279 433 0.1321 129 127.24 1.36
Table 1: Predicted VS Measured IQ
5
Automated IQ Estimation from Writing Samples pp. 3–7
This z-score represents the number of standard deviations, This research paper is intended to be purely
positive or negative, that the data point is away from the preliminary and simply introduce the concept and one
mean. Since we know the standard deviation and mean of all possible implementation of using an individual’s vocabulary
IQ scores, the corresponding IQ can be calculated as to determine IQ levels. To further develop this research, the
follows: authors suggest a larger dataset be used to create a more
accurate distribution. In addition, a more reliable dataset is
Corresponding IQ = (Z-Score * IQ Standard Deviation) necessary to test the accuracy of these methods. For the
+ IQ Mean strongest possible results, self-reported IQ scores should not
4 Testing IQ Estimation Software be used. Ideally, the next stage in research will include an
IQ test along with a specific writing prompt on which to run
Now that a sample of writing can be used to determine the our software. Lastly, there is likely merit in exploring the
IQ of an individual from their CWR, we must determine if analysis of the other attributes of writing that are mentioned
the IQ is accurate. The process of doing this is at the introduction to this piece. It is possible that one or all
straightforward, though difficult to accomplish. For it to be of these attributes may provide a better avenue for
reliably said that CWR can be used to calculate an calculating an individual’s intelligence level.
individual’s IQ, we must find multiple individuals with a
known IQ and access to writing that is their own. The The ability to analyze the intelligence of
pseudo-code for the software used to map the CWR of individuals is a very useful tool. It has been shown in
samples on to a corresponding IQ is shown in Figure 4. previous research that numerous factors influence whether
an intellectually gifted child will ultimately lead a
Using social media contacts, we located several successful life (Tomlinson-Keasey, Little; 1990). Earlier
individuals willing to give their IQ and a sample of their identification of these children, through application of this
research, has the potential to allow these children to be
Sample_Z_Score = (CWR_Sample – CWR_Mean) / guided down a positive path that will lead to their personal
CWR_Standard_Deviation success. In addition, this research could play a role in
Sample_IQ = (Sample_Z_Score *
evaluating the abilities of persons currently prominent in the
IQ_Standard_Deviation) + IQ_Mean
political and scientific realms. Nevertheless, further research
must be done in this area of study before anything truly
Figure 4: Calculating IQ Pseudo-code conclusive can be said.
writing for the purposes of testing our software. It should 6 References
be noted that there is no external verification that these
individuals gave an accurate IQ, but these samples are a Ali, Nawaf; Hindi, Musa; Yampolskiy, Roman. (2011).
good starting point for testing the reliability of this software. Evaluation of authorship attribution software on a
The data collected from these samples is shown in Table 1. Chat bot corpus. Information, Communication and
Regardless of the large error in the first sample, the Automation Technologies (ICAT), 2011 XXIII
accuracy of the rest of the samples provide support for this International Symposium on, IEEE.
approach for calculating IQ.
Ali, Nawaf; Schaeffer, Derek; Yampolskiy, Roman. (2012).
5 Conclusions and Future Work Linguistic Profiling and Behavioral Drift in Chat
Bots. MAICS, 27-30.
Though our first sample produced a result with a moderate
error, there still seems to be merit to looking further into this Ali, Nawaf; Yampolskiy, Roman. (2014). BLN-Gram-TF-
methodology. It should be noted that the samples used were ITF as a Language Independent Feature for
approximately 2 standard deviations above the mean. Authorship Identification and Paragraph Similarity.
Further sampling should include data on both ends of the 9th Cyber and Information Science research
curve. There may not ultimately be a cause effect Conference, Oak Ridge, Tennessee.
relationship between intelligence level and vocabulary
Bullerdieck, K. Kelly McK. (1985). Considerations in
usage, but this research does indicate the two are correlated.
Defining the Gifted.
The normality of the distribution of CWR may be
http://journals.sagepub.com/doi/abs/10.1177/10762
significant in other applications, and should be noted
1758500800607
regardless of the final merits of this approach to calculating
intelligence.
6
Austin Hendrix and Roman Yampolskiy MAICS 2017 pp. 3–7
Franking, Holly. (1988). Stylometry: A statistical method Snyderman, M., & Rothman, S. (1987). Survey of expert
for determining authorship, textual integrity, and opinion on intelligence and aptitude testing.
chronology. University of Kansas, ProQuest American Psychologist, 42(2), 137-144.
Dissertations Publishing.
Tomlinson-Keasey, Carol; Little, Todd D. (1990).
Holmes, D. (1994). Authorship Attribution. Computers and Predicting educational attainment, occupational
the Humanities, 28(2), 87-106. achievement, intellectual skill, and personal
http://www.jstor.org/stable/30200315 adjustment among gifted men and women. Journal
of Educational Psychology, vol. 82(3), pp. 442-
Lawler, James. (1977). IQ: Biological Fact or 455.
Methodological Construct? Science & Society, vol.
41, no. 2, pp. 208–218. Tweedie, F., Singh, S., & Holmes, D. (1996). Neural
www.jstor.org/stable/40402014. Network Applications in Stylometry: The
"Federalist Papers" Computers and the Humanities,
30(1), 1-10. Retrieved from
http://www.jstor.org/stable/30204514
7