Wikidata is a large collaborative knowledge graph, containing multilingual data for hundreds of languages across millions of entities (items) using thousands of properties. To explore research questions regarding the general state of data on Wikidata (e.g. multilingual research about potential biases or missing data in certain languages), a comprehensive data pro le of Wikidata is rst needed. This paper presents a data pro ling framework that parses the entire Wikidata data dumps to produce a series of granular descriptive statistics that summarize the full extent of data on Wikidata. This method addresses the limitations of using existing SPARQL API querying methods. The output from the data pro ling framework is presented through a series of interactive dashboards using Google Data Studio. Future work using output from the data pro ling is also proposed.
What languages are the most widely used for labels, aliases, and descriptions in Wikidata? Which ones are the least? What properties are most frequently used in statements, in quali ers, and in references? How many statements state that there is \some value" or de nitely \no value"? What kinds of subject matter topics are prevalent in Wikidata? These are some of the questions that motivate the data pro ling framework and the resulting dashboards that are described in this paper.
Since its launch in 2012, Wikidata is a widely used collaborative knowledge
graphs (KG) with data populated in hundreds of languages [
Commons License Attribution 4.0 International (CC BY 4.0). with labels, descriptions, and aliases in various languages in addition to the many properties that link it to other items and literal values.1
As KGs like Wikidata continue to grow in volume and usage, it is important to explore Wikidata's data coverage across international languages and communities. This can be seen in initiatives such as the \Whose Knowledge?" Wikimedia campaign2 and the \Reimagining Wikidata from the margins"3 initiative ahead of WikidataCon 2021. This paper presents ongoing research that seeks to answer questions on the distribution of multilingual data on Wikidata and eventually pose recommendations for data augmentation and enrichment of the knowledge graph. 2
To our knowledge, the most recent comprehensive research study on the
multilingual data on Wikidata was done in 2017 by Ka ee et al. [
In addition, many interactive data visualization tools and dashboards have
been created to describe certain aspects of the multilingual information on
Wikidata, with each tool uniquely tailored for a speci c objective.4 An all-purpose
1 https://www.wikidata.org/wiki/Q6053150
2 https://meta.wikimedia.org/wiki/Whose_Knowledge
3 https://www.wikidata.org/wiki/Wikidata:Reimagining_Wikidata_from_the_
margins
4 https://www.wikidata.org/wiki/Wikidata:Tools/Visualize_data
tool that supports all possible features to explore Wikidata exhaustively is not
yet extant, nor proposed as the scope of this paper. For instance, the Wikidata
Languages Landscape5 dashboard provides average descriptive statistics on
language data in the Wikidata ontology and for a few select items (or entities).
ProWD6 o ers interactive dashboards that display the level of \completeness"
of Wikidata items at a class-level, but does not look into the multilingual data
of those items [
Conventional methods of obtaining descriptive statistics from Wikidata rely on writing SPARQL queries to the Wikidata API endpoint or Wikidata Query Service.8 However, queries on the scale required for comprehensive data pro ling described in this paper are not possible as the API times out due to the computational complexity of even optimized queries.
For reference, the following SPARQL query that gets all scholarly articles (Q13442814) that have a missing French label (https://w.wiki/4C8H) fails with a \Query timeout limit reached" timeout error. On the other hand, the same query that gets all doctoral thesis items (Q187685) with a missing French label (https://w.wiki/4C8K) successfully completes. This discrepancy may be explained by the signi cantly larger amount of scholarly article entities compared to doctoral theses. Using the output from the WikiMetaData Pro ler as of the June 30, 2021 data dump, it was found that scholarly articles (Q13442814) made up over 34 million items or 38% of all items that have a value using the P31 \instance of" property. While the possible reason for the timeout errors may have been uncovered, this does not address the root issue that SPARQL queries fail for large expansive queries that may very well be the raison d'^etre for their use.
An alternative is to run a local machine, or cloud computing setup to ingest the entire Wikidata data dumps and run queries. However, with these alternatives a consideration that must be kept in mind is the cost of the hardware and software setups. The large size of the data dumps makes it necessary for a large disk and memory size to be used in order for a local solution be viable. The June 30, 2021 gzipped data dump was >100GB in its compressed form, and the latest October 6, 2021 data dump is over 106 GBs 9. A documented example where the entire Wikidata data dumps were loaded into Apache Jena, an open-source 5 https://meta.wikimedia.org/wiki/Wikidata_Languages_Landscape 6 https://prowd.id/#about 7 https://wdprop.toolforge.org/wdprop.html 8 https://query.wikidata.org/ 9 https://dumps.wikimedia.org/wikidatawiki/entities/ triplestore, used Intel Xeon CPUs with 32 cores and 128 GBs of RAM and almost 700 GB of disk space.10. As this was done in 2019, more recent data dumps would require commensurately higher technical requirements owing to the larger data dump size.
Plugging the aforementioned requirements into the pricing calculators11 of popular cloud service providers yields a monthly cost in the range of hundreds of US dollars per week. As experiments and queries are run on these instances over weeks and months, the costs would also scale accordingly. As a a result, this would limit the availability of these tools to researchers and institutions with such resources. 3.2
Regular data dumps of Wikidata are provided by the Wikimedia Foundation online in various formats. The June 30, 2021 Wikidata JSON dump (>100GB in its compressed form) was pro led using the data pro ling framework, which is released as open-source code on GitHub.12 More recent data dumps may also be downloaded and processed through the same pipeline in order to reproduce the results using the code on GitHub.
The code operates in the following manner. Within the nested JSON
structure of the data dump, each line contains all of the key: value (or property:
object ) pairs for a single Wikidata item, which correspond to the property and
object values in a (subject; property; object) triple. As the entirety of the key:
value data is parsed, certain prede ned elds and statistics of interest
(languages (labels, aliases, descriptions), properties, classes of items, quali ers, and
references) are captured in memory, updated in memory, and eventually
written to output les. More precisely, the data that was maintained in memory is
rst formed into Python pandas DataFrames and then written to tab-separated
value (TSV) output les [4,?]. These output les were uploaded to Google Data
Studio (GDS) and presented through a series of data visualizations and
interactive dashboards.13 To monitor the pro ling progress, the tqdm progress bars
were used [
The data pro ler is able to run on widely available consumer hardware setups, such as commercially sold laptops and desktops. Most computations were run using Jupyter notebooks on two local machines: (1) running Ubuntu 18.04, with 8 CPU cores and 16 GB memory, and (2) running macOS with 6 CPU cores and 16 GB memory. The Google Colaboratory notebooks, each with 2 CPU cores 10 (non-English:) https://muncca.com/2019/02/14/wikidata-import-in-apachejena/ 11 Virtual machines at https://azure.microsoft.com/en-us/pricing/calculator/ and EC2 instances at https://calculator.aws/#/createCalculator/EC2 12 https://github.com/nchah/wikidata-profiler 13 https://datastudio.google.com/reporting/7f6f76eb-c24f-4a1d-b7a086fc4c2a55c4
From the interactive Google Data Studio dashboards, the following results that are relevant to language locales are described in further detail. The full set of interactive dashboards allows for additional ndings to be made across di erent aspects of Wikidata.
Distribution of multilingual data, properties, classes of items, quali ers, and references. A selection of the Google Data Studio dashboards is shown in Figure 2. In the full GDS dashboard, descriptive statistics are shown for the distribution of di erent language data used in labels, aliases, and descriptions for items and properties (part of the ontology). To discover what kinds of entities exist in Wikidata, a view into the di erent classes of items using the P31 property (\instance of") is also presented. The distributions of properties that are used for statements, quali ers, and references are also shown.
Interactive comparison of the distribution of multilingual data. It is also possible to use the dashboards to compare the distribution of values across language variants or dialects. For instance, in Figure 3, the various dialects or variations of a larger language are compared for Chinese (zh) and English (en) dialects. These dialects are shown by a \-" (dash) and additional language code. Across items, the Chinese (zh) dialects are almost evenly distributed for aliases and descriptions, with a slight di erence in the even distribution for labels. This is unlike the distribution of signi cantly fewer localized values in the dialects under English (en): British English (en-gb), Canadian English (enca), and American English (en-us).
Dominant languages continue to stay largely dominant, with a few
upstart changes. Among the various dashboards, a number of general ndings
on the languages is described. According to the data pro ling work done as of
2017, prominent languages for label data included English (en) at 11.04%, Dutch
(nl) at 6.47%, French (fr) at 6.02%, German (de) at 5.08%, and Spanish (es) at
4.07% (see Figure 1, from earlier in this paper) [
While the Wikidata data dumps are comprehensive, they do not capture other important aspects of the Wikidata system. The myriad of users and bots that contribute information to Wikidata are not tracked in the data dumps. As such, data such as the provenance of Wikidata contributions, the speed and frequency of edits (and reversions), and the general degree of activity of users are not easily obtained from a dedicated data dump. Furthermore, the active discussions and coordination that take place on the \Talk" pages of Wikidata pages are not captured.
This kind of provenance data would have a potential application in determining, for example, the upstart rise of Wikidata labels, descriptions, and aliases in Asturian (ast) and other previously underrepresented languages (Irish (ga) and Bengali (bn)). Were bots or automatic processes responsible for the massive in ux of data? Or, were there grassroots community movements, hackathons, or other local initiatives that brought together people to add data to the languages? 6
This paper presented early work on a data pro ling framework and pipeline to parse the entire Wikidata data dumps and visualize the output in interactive Google Data Studio reports. In a forthcoming paper, the output data from this data pro ling framework was used to generate heatmaps that depict the distribution of multilingual data (labels, aliases, and descriptions) across machine learned and human annotated topical domains in Wikidata. These heatmaps are then useful for determining where multilingual data is concentrated and sparsely populated.
Future work will use the output from the data pro ling framework for research questions on the current state of Wikidata (e.g. with questions on the prevalence of language data for various classes of items, and questions on the frequency of \high quality" properties that use quali ers and references) and machine learning approaches to programmatically recommend data enrichment regimes (e.g. using knowledge graph embeddings for targeted link prediction in certain under-served languages). Acknowledgements This research was supported by funding from the Natural Sciences and Engineering Research Council of Canada (NSERC).