The main promise of the digital humanities is the ability to perform scholar studies at a much broader scale, and in a much more reusable fashion. The key enabler for such studies is the availability of su ciently well described data. For the eld of socio-economic history, data usually comes in a tabular form. Existing e orts to curate and publish datasets take a top-down approach and are focused on large collections. This paper presents QBer and the underlying structured data hub, which address the long tail of research data by catering for the needs of individual scholars. QBer allows researchers to publish their (small) datasets, link them to existing vocabularies and other datasets, and thereby contribute to a growing collection of interlinked datasets. We present QBer, and evaluate our rst results by showing how our system facilitates two use cases in socio-economic history.
In a 2014 article in CACM, [
This paper presents QBer and the underlying CLARIAH Structured Data Hub (CSDH),8 whose aim is to address the limitations of current data-publishing practice in the digital humanities, and socio-economic history in particular. The CSDH integrates a selection of large datasets from this domain, while QBer is a user-facing web application that allows individual researchers to upload, convert and link `clean' data to existing datasets and vocabularies in the hub without compromising the detail and heterogeneity of the original data (see Section 2). Under the hood, we convert all data to RDF, but QBer does not bother scholars with technical aspects. An inspector-view displays the result of the mappings { a growing network of interconnected datasets { in a visually appealing manner (See Figure 1). The visualization is just one of the incentives we are developing. The most important incentive will be the ability to allow for transposing research queries across datasets, and the ability to perform crossdataset querying. Section 4 describes two use-cases that evaluate the ability of QBer and the CSDH to ful ll that promise. We rst discuss related work in Section 2 and describe the QBer and CSDH systems in Section 3. 2
Historical data comprises text, audiovisual content or { in our case { data in
the more traditional sense: structured data in tabular form. Preparing
historical data for computational analysis takes considerable expertise and e ort. As
a result, digital data curation e orts are organized (and funded) in a top-down
fashion, and focus on the enrichment of individual datasets and collections of
sufcient importance and size. Examples are the North Atlantic Population Project
(NAPP) [
For this last reason, it is di cult for individual researchers to make their
data available in a sustainable way [
In socio-economic history, a central challenge is to query data combined from
multiple tabular sources: spreadsheets, databases and CSV les. The multiple
bene ts of Linked Data as a data integration method [
An important question then is: how are these mapping les created? Work in
ontology and vocabulary alignment, as in the OAEI,16 or identity reconciliation,
aim to perform automatic alignments. Given the often very speci c (historic)
meaning of terms in our datasets, these techniques are likely to be error-prone,
hard to optimize (given the heterogeneity of our data) and unacceptable to
scholars. Interactive alignment tools, such as Amalgame [
10See http://easy.dans.knaw.nl 11See http://dataverse.harvard.edu and http://dataverse.nl 12See http:// gshare.com 13See http://datadryad.org 14For a comprehensive list, see e.g. https://github.com/timrdf/csv2rdf4lodautomation/wiki and http://www.w3.org/wiki/ConverterToRdf 15https://www.w3.org/2001/sw/wiki/TopBraid 16The Ontology Alignment Evaluation Initiative, see oaei.ontologymatching.org/ 17https://www.w3.org/2001/sw/wiki/Anzo 18https://www.sysmo-db.org/right eld
To create a viable research ecosystem for Linked Humanites Data of all sizes, we need to combine expert knowledge with automated Linked Data generation. It should be easy and pro table for individual researchers to enrich and publish their data via our platform. To achieve the rst goal, we developed QBer;19 an interactive tool that allows non-technical scholars to convert their data to RDF, to map the `variables' (column names) and values in tabular les to Linked Data concept schemes, and to publish their data on the structured data hub. What sets QBer apart is that all Linked Data remains under the hood. To achieve the second goal, we build in direct feedback (reuse of existing content, visualizations, etc.) on top of the CSDH and demonstrate the research bene ts of contributing data to it (see Section 4). We illustrate QBer by means of a walkthrough of the typical usage of the tool, and then summarize its connection with the CSDH.
Using QBer consists of interacting with three main views: the welcome screen, the mapping screen, and the inspector. In the welcome screen, users rst authenticate with OAuth compatible services (e.g. Google accounts), and then select a raw dataset to work with. Datasets can be selected directly from the CSDH, or imported from a Dataverse collection by providing a DOI.
Once a dataset is loaded, QBer displays the mapping screen (Figure 1). This screen is divided into the variables sidebar (left) and the variable panel (right). The sidebar allows the user to search and select a variable (i.e. column) from the dataset. Once the user clicks on one variable, the variable panel will show that 19See https://github.com/CLARIAH/qber variable's details: the variable category, the variable metadata, and the value frequency table.
We distinguish between three variable categories : coded, identi er and other.
Values for coded variables are mapped to corresponding concepts (skos:Concept)
within a skos:ConceptScheme, which establishes all possible values the variable
can take. If the variable is of type identi er, its values are mapped to dataset
speci c minted URIs. Finally, the values of variables of type other are mapped
to literals instead of URIs. The `Community' button gives access to all known
prede ned datacube dimensions. These come from LSD Dimensions, an index
of dimensions used in Data Structure De nitions of RDF Data Cubes on the
Web [
The variable metadata panel can be used to change the label and the description of the variable. If the variable has been speci ed to be \coded" in the previous pane, it can be linked to existing code lists curated by the Linked Open Data community. QBer provides access to all concept schemes from the Linked Data cache,20 and the CSDH itself. If the variable is of type \other", this panel lets users de ne their own transformation function for literals.
The frequency table panel has three purposes. First, it allows for quick inspection of the distribution of all values of the selected variable, by displaying their frequency. Second, if the variable type is \coded", it lets the user map the default minted URI for the chosen value to any skos:Concept within the selected skos:ConceptScheme in the variable metadata panel. QBer also has a batch mapping mode that prompts the user to map all values of the variable interactively. Third, the panel shows the current mappings for values of the selected variable.
Mappings can be materialized in two ways. Users can click on Save in the
navigation bar, which stores the current mapping status of all variables in their
local cache. Clicking on Submit sends the mappings to the CSDH API, which
integrates them with other datasets in the hub. Under the hood, the data is
converted to a Nanopublication[
The inspector, shown at the bottom right of Figure 1, allows users to explore the contents of the CSDH. The visualization shows a graph of nodes and edges, with di erent icons representing di erent node types. User nodes represent users that have submitted data to the hub, according to their provided OAuth identities in the welcome screen. Dataset nodes represent Data Struc20See http://lod.openlinksw.com we aim to extend this with schemes from the LODLaundromat, http://www.lodlaundromat.org. ture De nitions21 (DSD) submitted by these users. Dimension nodes represent dimensions (i.e. variables, columns in the raw data) within those DSD. Dimensions that are externally de ned (e.g. by SDMX or some other external party) and are thus not directly used in datasets, are represented as cloud-icons. Users can interact with the inspector in two ways: hovering on nodes displays their properties; and dragging them moves the graph elements for better layout.
QBer and the Inspector work on top of the CSDH API,22 which carries out the backend functionality. This includes converting, storing, and managing POST and GET requests over datasets. The CSDH API functionality will be extended to cover standard data search, browsing and querying tasks. All data is currently stored in an OpenLink Virtuoso CE triplestore,23 but since CSDH communicates through the standard SPARQL protocols it is not tied to one triple store vendor. 4
In this section, we evaluate the results of our approach by means of two use cases in socio-economic history research. The rst use case investigates the question as to whether the CSDH indeed allows research to be carried out on a broader scale. In this case, we transpose a query that was built to answer a research question aimed at a dataset of one country to a dataset that describes another country. The second use case investigates the question as to whether our system facilitates the work ow of a typical individual researcher. 4.1
Use Case 1: Born Under a Bad Sign
Economic and social history takes questions and methods from the social
sciences to the historical record. An important line of research focuses on the
determinants of historical inequality. One hypothesis here is that prenatal [
Most studies on the impact of early life conditions are case studies of single
countries. Therefore, the extent to which results can be generalized { their
external validity { is di cult to establish (e.g., di ering impact of early life conditions
in rich and poor countries). Moreover, historical data is often idiosyncratic. This
21According to [
22https://github.com/CLARIAH/wp4-csdh-api 23See http://www.openlinksw.com means that dataset-speci c characteristics such as sampling and variable coding schemes might in uence the results (see Section 2).
In this use case, we explore the relation between economic conditions in
individuals' birth year and occupational status in the historical census records of
Canada and Sweden in 1891. In many cases it would be necessary to link the
two census datasets so that they can be queried in the same way. Here, however,
we use two harmonized datasets from the North Atlantic Population Project
(NAPP) [
In general terms, the data requirements are typical of recent trends in large database usage in economic and social history: 1) the primary unit of analysis is the individual (microdata); 2) a large number of observations is analyzed; 3) multiple micro-datasets are analyzed; 4) microlevel observations are linked to macro-level data through the dimensions time and geographical area; 5) qualitative data is encoded to extract more information from it.
Current Work ow. The traditional work ow to do this could include the following steps. First, the researcher has to nd and download the datasets from multiple repositories. The datasets, which come in various formats, then have to be opened, and, if necessary, the variables have to be renamed, cleaned, and re-encoded to be able to join them with other datasets. We can rely on previous cleaning and harmonization e orts of the NAPP project, but in many other situations the researcher would have to do this manually. Finally, the joined data has to be saved in a format that can be used by a statistical program.
New Work ow. Using QBer and the CSDH, the work ow is as follows. Linkeddata tools are used to discover data on the hub. In our case, a linked data browser26 and exploratory SPARQL queries were used. Note that to discover datasets and especially linked datasets on the CSDH, it is necessary that someone uploaded the datasets and created the links in the rst place, for example by linking datasets to a common vocabulary. While it is unavoidable that someone has to do this at some point, the idea behind the hub is that if it is done once, the results can be re-used by other researchers.
Next, queries were built and stored on GitHub. The resultsets that these queries produce against the data hub are used to create the dataset that is to be analyzed. grlc, a tool we developed for creating Linked Data APIs using SPARQL queries in GitHub repositories, was helpful in exploring the data on 24HISCO: Historical International Standard Classi cation of Occupations. 25https://github.com/rlzijdeman/o-clack and http://www.camsis.stir.ac.uk/ hiscam/
26https://github.com/Data2Semantics/brwsr .398 ● ● ● ● ●
Canada
● ● ● ●
●
●● ● ●● il()scghoam .......934369893239390400402 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● il()csahogm .......943363994320989300240 ● ●●●●● ● ●● ●●●● ●●● ●●● ● ●● ● ●●●● ● ●● ● ●● ● ● ● ●●● ●● ● ● ● ● ● ●
the hub and executing the eventual query [
Observations. While more sophisticated models are required to disentangle
cohort, period and age e ect [
The creation and use of this dataset is exemplary of the work ow of small to medium quantitative historical research projects in the sense that it relies on multiple data sources that need to be connected in order to answer the research questions. We brie y discuss this work ow, and then show the impact that QBer and the CSDH have.
27https://github.com/CLARIAH/grlc 28In Dutch, a \dwarsligger" can mean both a railroad tie, and an obstructive person.
Current Work ow. Zandhuis' current work ow is very similar to the one reported in the rst use case. He rst digitized the main dataset on the strike behaviour of employees at the maintenance workshop of the railway company (N = 1163). Next, he gathered data from multiple sources in which these employees also appear, adding individual characteristics that explain strike behaviour. For example, he derived family situations from the Dutch civil registers, and the economic position from tax registers, resulting in a separate dataset per source. Next, he inserted these datasets into a SQL database. In order to derive a concise subset to analyze his research questions, using e.g. QGIS, Gephi or R, he wrote SQL queries to extract the relevant information. These queries are usually added as an appendix to his research papers.
New Work ow. In collaboration with Zandhuis, we revisited this work ow using QBer. Zandhuis, as most historians, uses spreadsheets to enter data, and uses a speci c layout to enhance the speed and quality of data entry. The rst step was to convert the data to a collection of .csv les. This is just a temporary limitation, as the CSDH is not necessarily restricted to CSV les. It uses the Python Pandas library29 for loading tabular les into a data frame.
The second step involves visiting each data le in turn, and linking the data to
vocabularies and through them to other datasources. Data about the past often
comes with a wide variety of potential values for a single variable. Religion, for
example, can have dozens of di erent labels as new religions came about and
old religions disappeared. As described in Section 3, QBer provides access to
a large range of such classi cations, basically all those available in the Linked
Data cloud and the CSDH. For example, QBer provides all occupation concepts
from the HISCO classi cation used in the rst use case [
The third step was then to query the datasets in order to retrieve the subset
of data needed for analysis. As in the rst use case, we design SPARQL queries
that, when stored on GitHub, can be directly executed through the grlc API.
This makes replication of research much easier: rather than including the query
as an appendix of a research paper, the query is now a rst order citizen and can
29See http://pandas.pydata.org
even be applied to other datasets that use the same mappings. Again, through
the API, these queries can easily be accessed from within R, in order to perform
statistical analysis. Indeed, the grlc API is convenient, but it is a lot to ask
non-computer science researchers to design SPARQL queries. However, as we
progress, we expect to be able to identify a collection of standard SPARQL
query templates that we can expose in this manner (see also [
To illustrate this, consider that since the Dwarsliggers collection contains multiple datasets on the same individuals at the same point in time, there are multiple observations of the same characteristics (e.g. age, gender, occupation, religion). However, the sources di er in accuracy. For example, measuring marital status is one of the key aims of the civil registry, while personnel les may contain information on marital status, but it is not of a key concern for a company to get this measurement right. By having all datasets mapped to vocabularies through QBer and having the queries stored in GitHub and executed by grlc, each query can readily be repeated using di erent sources on the same variables. This is useful as a robustness check of the analysis or even be used in what historians refer to as a 'source criticism' (a re ection of the quality and usefulness of a source). This, again, is similar to the rst use case, but it emphasizes an additional role for the queries as so-called 'edit rules'.
Observations. To conclude, this use case shows that the QBer tool and related infrastructure provides detailed insight in how the data is organized, linked and analyzed. Furthermore, the data can be queried live. This ensures reusable research activities ; not just reusable data. 5
The preceding sections presented QBer and the CSDH to address the limitations of existing digital humanities data curation projects in facilitating 1) the long tail of research data and 2) research at a broader scale, enabling cross-dataset querying and reuse of queries. We argued that existing Linked Data publishing and mapping tools do not meet the needs of scholars that are not technologically versed (or interested).
QBer and the CSDH enable individual scholars to publish and use their data in a exible manner. QBer allows researchers to publish their (small) datasets, link them to existing vocabularies and other datasets, and thereby contribute to a growing collection of interlinked datasets hosted by the CSDH. The CSDH o ers services for inspecting data, and (in combination with grlc) reusable querying across multiple datasets. We illustrated these features by means of two use cases. The rst shows the ability of the Linked Data paradigm used in the CSDH to signi cantly lower the e ort needed to do comparative research (even when the data was published as part of the same larger standardization e ort). The second use case shows how publishing data through QBer allows individual researchers to have more grip on their data, to be more explicit regarding data interpretation (coding) and, via the CSDH, to be able to answer more questions for free (e.g. the mapping through HISCO to HISCAM).
Of course, there still is room for expansion. To ensure uniqueness of identiers, historical 'codes' need to be mapped to URIs. This is technically trivial, but historians are not used to these lengthy identi ers in their statistical analyses. Secondly, formulating research questions as queries requires an understanding of the structure of the data. Given the large numbers of triples involved, this can be di cult. As said above, standard APIs based on SPARQL query templates should solve some of this problem, but o ering a user-friendly data inspection tool is high on our list. SPARQL templates allow us to solve another issue: allowing for free-form querying can have a detrimental e ect on the performance of the CSDH. The use of templates enables more e cient use of caching strategies.
But even without such improvements, we believe that the use cases show that QBer and the CSDH already broaden the scope of supported work ows and data in our ecosystem, and bring the bene ts of Linked Data and the Semantic Web at the ngertips of humanities scholars.
Acknowledgements This work was funded by the CLARIAH project of the Dutch Science Foundation (NWO) and the Dutch national programme COMMIT.