Manual Curation: Cyc and Freebase
For manually created knowledge graphs, we have to estimate the e ort of
providing the statements directly.
Cyc [
6
] is one of the earliest general purpose knowledge graphs, and, at the
same time, the one for which the development e ort is known. At a 2017
conference, Douglas Lenat, the inventor of Cyc, denoted the cost of creation of Cyc
at $120M.1 In the same presentation, Lenat states that Cyc consists of 21M
assertions, which makes a cost of $5.71 per statement. As a footnote, the
development time of 1,000 person years boils down to 9.5 minutes per assertion.
1 http://www.ttivanguard.com/conference/Napa2017/4-Lenat.pdf
Freebase has been collaboratively created by volunteers [
1
], and hence, its
development e ort is more di cult to assess. To assess the time for curating
the statements in Freebase, we follow the assumption that adding a statement
should be approximately as much e ort as adding a sentence to Wikipedia.2
In [
4
], the time of creating the English language Wikipedia up to April 2011
has been estimated to a total of 41M working hours. At that time, Wikipedia
contained 3.6M pages,3 at an average of 36.4 sentences each [
10
]. This boils down
to 18.7 minutes per sentence.4 Since the majority of Wikipedians is US-based,5
we use the US federal minimum wage of $7.25 per hour6 as an estimate for labor
cost, leading us to the cost of $2.25 per sentence.
Therefore, we also assume a cost of $2.25 per statement in Freebase. This
is less than half of the price of a statement in Cyc { which is reasonable since
Cyc was created by experts, while Freebase was created by laymen users.
In total, given that the last version of Freebase contains 3B facts [
9
], the cost
of creating Freebase totals to $6.75B.
1.2
Automatic Creation: DBpedia, YAGO, and NELL
The estimation of e ort for creating a knowledge graph automatically is di erent.
We consider the software used for creating the knowledge graph and estimate
its development e ort based on the lines of code (LOC).7 We follow the ndings
in [
2
], stating that in a software development project, an average of 37 LOC are
produced by hour.8 Furthermore, since YAGO has been developed by a German
research institute and the majority of developers of DBpedia is also based in
German research institutions, we use the researcher salaries proposed by the
German national research funding agency DFG9 for our cost estimates.
DBpedia is created from Wikipedia dumps by running the DBpedia
Extraction Framework10, which uses mappings to a central ontology11 for creating the
knowledge graph. They account for 4.9M and 2.2M LOC, respectively. Using the
numbers above, this leads to a total development cost of $5.1M. Given the 400M
statements in the English language DBpedia [
11
], this boils down to 1.85g per
statement.12 Comparing this to the $2.25 per statement for manual curation,
the automation leads to savings by a factor of around 100.
2 Disclaimer: this is a debatable assumption, and it is by far not the only one in this
paper.
3 https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
4 This number may seem high, but it includes revisions and, since the measurement
is based on the length of edit sessions, even research for facts to a certain extent.
5 https://en.wikipedia.org/wiki/Wikipedia:Wikipedians
6 https://www.dol.gov/whd/minimumwage.htm
7 Determined using GitHub SLOC https://github.com/martianyi/github-sloc
8 The authors measured the total software development cost, not only the coding.
9 http://www.dfg.de/formulare/60_12/60_12_en.pdf
10 https://github.com/dbpedia/extraction-framework
11 https://github.com/dbpedia/mappings-tracker
12 We do not include the cost of creating Wikipedia in the rst place. Otherwise,
assuming that each statement in DBpedia has its root in one infobox entry (which
YAGO is a knowledge graph which combines information extracted from
Wikipedia with the ontology WordNet [
7
]. For a fair comparison, the development
cost of WordNet is included. The YAGO codebase13 has 1.6M LOC (including
rules to map infoboxes to an ontology), which makes a total of $1.6M.
WordNet itself consists of 117k synsets,14 each including a gloss, and we estimate the
cost of de ning such a synset roughly the same as producing a Wikipedia page,
i.e., $10M on top. Given that YAGO has 1.4B statements [
11
], this totals to
0.83g per statement. Compared to the manual curation, automation leads to
savings by a factor of around 250 here.
NELL is a system that learns patterns for relation extraction [
8
]. Its core
technology encompasses 103k LOC,15 which accounts for an estimated development
cost of $109k. Furthermore, 1,467 statements are manually validated per month.
Assuming that manually validating a statement costs as much as creating it, this
accounts for another $376k, i.e., a total development cost of $485k. Given the
size of NELL, this totals to 14.25g per statement, i.e., a savings factor of 16
compared to manual curation.
2
Towards new Evaluation Metrics
Introducing cost as a measure for knowledge graph creation can also pave the
way for other kinds of evaluation. For example, a new method for adding missing
knowledge to a knowledge graph [
9
] can be inspected by cost: e.g., an approach
developed by one person over half a year should add signi cantly more than
2,800 statements, which, according to the numbers used in this paper, would
be the amount of triples that person would produce manually in that time.
Furthermore, approaches that propose the creation a custom knowledge graph
for improving the performance of a speci c task can estimate the cost of that
improvement more e ciently.
Another interesting consideration is the relation between development e ort
and data quality. In gure 1, we graphed the error rate of the knowledge graphs
discussed in this paper against the cost per triple. While the general trend that
can be observed is that triples created at higher expenses also have a higher
likelihood of being correct, NELL is an outlier here, depicting a much worse
relation between accuracy and cost.
3
Concluding Remarks
In this paper, we have shown estimates for the cost of the creation of popular
knowledge graphs, an aspect of knowledge graph creation that is currently
underrepresented in the literature. We have quanti ed the gain of automatic over
we could understand as a "`sentence"') in Wikipedia, the cost would always be higher
than that of manual curation.
13 https://github.com/yago-naga/yago3
14 https://wordnet.princeton.edu/
15 https://groups.google.com/forum/#!topic/cmunell/aAZVG9zVwSU
0.1
0
0
NELL
YAGO
0.01
0.1
Cost (USD/Triple)
1
Freebase Cyc
manual curation (i.e., 2-3 orders of magnitude), and proposed using cost for the
de nition of new evaluation metrics, e.g., trading o cost for accuracy.
That being said, we are aware that many of the assumptions and
approximations we took for computing those estimates are questionable (e.g., we did
not consider the cost of third party software libraries used by the approaches,
or the infrastructure cost), and one could have used other numbers in most of
the cases. Moreover, the cost of providing the knowledge graphs is currently not
considered. Nevertheless, we are con dent that shedding light at the cost aspect
of knowledge graph creation is valuable.