This paper describes the history, policy, semantics, and uses of the HathiTrust Research Center Extracted Features dataset, an open-access representation of the 17+ million volume HathiTrust Digital Library, including a major current e昀ort to extend computational access in a variety of more 昀氀exible and easily implemented ways, including a modern API supporting customizable visualizations and analyses. HathiTrust1, founded in 2008, is a not-for-pro昀椀t collaborative initiative of academic and research libraries that seeks to preserve and make accessible for reading and computation the combined corpora of digitized objects from the HathiTrust consortium's membership. As of this writing, that corpus, known as the HathiTrust Digital Library, has now grown to contain more than 6.1 billion digitized pages of content comprising almost 17.7 million volumes of text representing nearly 9 million unique works (including more than 8.4 million books and 470 thousand serials titles). The work described below addressed two core research problems. First, the Extracted Features (EF) dataset provides access to the HathiTrust corpus in a non-consumptive manner
that allows computational analysis of a representation of the entire corpus while respecting copyright and other restrictions on that content. The TORCHLITE project addresses a second research problem by providing e昀케cient, programmatic access to the Extracted Features dataset. Previously, general users were required to use the rsync utility to access the EF dataset. The TORCHLITE project provides more 昀氀exible, e昀케cient, and programmatic access to the EF dataset through modern web-based APIs. The EF dataset and the TORCHLITE APIs are open and available to all users regardless of membership or a昀케liation with the HathiTrust Digital Library.
One of the core missions of HathiTrust is to provide access to its collections. Towards this end, in 2011, HathiTrust established the HathiTrust Research Cent2e r(HTRC), with one of its primary objectives to solve the problem of computational access to the massive HathiTrust corpus. One key service through which HTRC supports this access is the periodic publication of Extracted Features (EF) datasets, comprised of quanti昀椀ed features derived from the full-text corpus using computational analysis. The EF datasets serve researchers in a range of 昀椀elds (including the humanities, social sciences, data science, and machine learning) with a foundational, structured dataset that may be used to develop analytics algorithms for computational exploration of the HathiTrust corpus. The EF datasets may be downloaded or accessed through our new TORCHLITE EF API, without restriction, as easily parsed 昀椀les encoded as “JSON for Linked Data” (JSON-LD). Through these EF datasets, researchers have access to a snapshot of the entire HathiTrust corpus, in a format that provides useful, inferential 昀椀ngerprints of a set of texts without giving away the original copyrighted work, a mode of access known as “non-consumptive research.” Non-consumptive research is computational analysis performed on text without substantial portions of the text displayed or read to understand its expressive content.3
The initial releas4eof EF, versioned as 0.2 1[] [
The unigram tokens in this dataset were parsed from HathiTrust’s OCR text 昀椀les using the OpenNLP6 library. In addition to parsing the tokens, the OpenNLP library was used to characterize each token’s part of speech (POS). The primary language used on each page analyzed was also calculated using Shuyo Nakatani’s Language Detection libra7rAy. highly structured JSON schema was developed that allowed for the presentation of all the tokens, their parts of speech, and the number of their occurrences, separated into the header, body, and footer sections of each page. Each JSON 昀椀le encoded the features extracted from exactly one volume in the HathiTrust corpus.
Importantly, an unusual implementation feature of the common so昀琀ware libraries used for parsing JSON documents was exploited to provide a more e昀케cient means of processing the EF data encapsulated within the individual EF 昀椀les. In particular, rather than follow the typical"key":"value" pairing de昀椀ned by the JSON standard, 昀椀elds containing the tokens, POS tags, and their respective counts (i.e., the actual textual-statistical data) take advantage of the fact that JSON parsing libraries have no semantic sense of the concept of a "key". Rather than treating each "key" as a label, these libraries parse it as an arbitrary string8. This helpful semantic ignorance provides an important simplifying pattern for using JSON to communicate data, rather than following a verbose pattern such as "key":{"key":"value","key":"value","key":"value"} — that rigorously conforms to the JSON speci昀椀cation, EF 昀椀les employ an ”o昀-formulary” approach using a much less verbose pattern of "value":{"value":"value"}. Thus, the all-important statistical part of the dataset follows the simpli昀椀ed pattern"token":{"POS":"count"}.
This non-standard approach for thfeeatures section of each EF 昀椀le reduces both 昀椀le size (by approximately half) and data complexity ( “昀氀attening” the 昀椀le structure and bringing the all-important linguistic and statistical data one level “higher” in the index and parsing trees); it also makes possible the exploitation of e昀케ciencies already built into existing JSON parsing libraries9. While these gains may seem trivial in most common contexts (involving hundreds or thousands of 昀椀les), the current EF release weighs in at 17.1 million 昀椀les and 4.2 TB (compressed), so our estimated 50% gains in storage, data transfer, and processing seem to us more than substantial.
This approach, however, requires data consumers to familiarize themselves with the EF schema and its non-standard usage to be aware of 昀椀elds for which standard key-value pairs are intended, and those for which value-value pairs are used. As long as these di昀erences in semantics are taken into account (and they should be fairly obvious and visible to users), the same JSON parsing and processing libraries can be for both data patterns; users simply need 6http://opennlp.apache.org 7https://code.google.com/p/language-detection/ 8Note that the rest of the JSON 昀椀le — that is, everything except word and POS counts — useskey:value pairs in a more customary fashion to denote header, metadata, and statistical information about the dataset. 9These estimates are based on a simple comparison of a single random value expressed in the two encodings, requiring about 50 characters in the standard encoding, and about 25 using our approach. Even in this more streamlined encoding, the features section of a single EF 昀椀le, representing a randomly selected 220-page volume, is about 850,000 characters, as compared with themetadata section of only about 2,000 characters, even in the more verbose standard syntax. to treat the outputs di昀erently. As we discuss in the section describing the EF 2.0 dataset (below), these semantic di昀erences also have consequences for the dataset’s compliance with the Linked Data standard.
In addition to the token data for a particular volume, each EF 昀椀le also included a basic set of bibliographic metadata for that volume (using thkeey-value pair pattern) describing the following information: • volume title • primary author • volume imprint information (a single, unparsed 昀椀eld that included publisher, place of publication, year of publication) • the OCLC10 identi昀椀er associated with the volume’s catalog record • the primary language of the volume (as identi昀椀ed by human catalogers) • the URL of the volume’s HathiTrust catalog record • the URL of the digitized volume’s persistent HathiTrust identi昀椀er
With each release of the EF dataset, we have considered separating the JSON standardcompliantmetadata section from the non-compliant but still fully functiofneaaltures section; this consideration was especially intense with the current release (described below) in which we 昀椀rst created Linked Data (expressing volume and 昀椀le metadata as fully compliant JSON-LD). So far, we have made the decision not to do this in favor of o昀ering a “one-stopshopping” single 昀椀le per volume, which contains all of the data and metadata that a researcher needs to carry out analytic tasks. Maintaining our original approach also facilitates some backward compatibility with both previous EF versions and existing tools built speci昀椀cally for this data.
The subsequent EF releases (designated as versions 1.0 and 1.151) [
In addition to vastly increasing the size and coverage of the dataset, the metadata section of each EF 昀椀le was also signi昀椀cantly expanded with more complete and granular information from the MARC records describing its volumes. In addition to the basic catalog 昀椀elds listed above, volume metadata also now included: 10OCLC, https://www.oclc.org,/ formerly the Online Computer Library Center, is perhaps best known as the compiler and maintainer of WorldCahtt,tps://www.worldcat.org./ 11https://wiki.htrc.illinois.edu/pages/viewpage.action?pageId=37322778 • publication place, parsed out from the imprint statement • publisher, parsed out from the imprint statement • publication (copyright) year, parsed out from the imprint statement • cataloger-determined genre • cataloger-determined resource type • additional creator names (e.g., additional authors, editors, illustrators, etc.) • copyright status as determined by HathiTrust (i.e., whether the volume is in the public domain in particular jurisdictions) • additional bibliographic identi昀椀ers such as ISBN, ISSN, LCCN (Library of Congress Catalog Number), and call numbers assigned by the volume’s owning institution(s)
One of the original goals of creating a new EF 2.0 schema and datas1e2t[
In addition to this move to Linked Data, EF 2.0 represents the latest set of bootstrapping features computationally mined from the HathiTrust corpus. Since the release of EF 1.0/1.5, the dataset has grown in size to just over 2.9 trillion unigrams parsed from over 6.2 billion pages contained in just over 17.1 million volumes. Like the leap from EF 0.2 to EF 1.0/1.5, the new EF 2.0 dataset further expands, while also re昀椀ning, the volume-level metadata provided within each EF 昀椀le. In particular, the initial development of the JSON schema underlying the EF 2.0 model was undertaken in collaboration with the JSTOR and Portico proj1e3catnsd while not all of the innovations developed during this collaboration were implemented into EF 2.0 in the end, the switch from MARC metadata as the basis for volume-level metadata in EF 昀椀les to the more re昀椀ned, linked-data-compliant BIBFRAME14 records made it possible, for instance, to link EF 昀椀les describing di昀erent instances of the same work to one another.
In the following sections we delve more deeply into how the EF 2.0 dataset di昀ers from older EF datasets, and how it continues in their tradition. 2.3.1. EF 2.0 Metadata The primary innovations for the metadata section of EF 2.0 昀椀les involve the implementation of linked data practices. First and foremost, a subtle change in the 昀椀le format from JSON to 12EF 2.0 is the current version as of the time of writingh:ttps://wiki.htrc.illinois.edu/pages/viewpage.actnio?pageI d=79069329 13Both JSTOR, https://www.jstor.org,/ and Portico,https://www.portico.org,/are projects under the nonpro昀椀t
ITHAKA umbrellah,ttps://www.ithaka.org/. 14https://www.loc.gov/bibframe./ JSON-LD was made. Unlike ordinary JSON 昀椀les which use arbitrary vocabulary structures to manage their key labels, JSON-LD employs a type of schema document known ascaontext document to establish the semantics for each JSON-LD 昀椀le. Each context 昀椀le works like an XML schema, de昀椀ning the semantics for each key label or linking its semantics to an existing ontology or metadata vocabulary standard. To 昀椀ll this requirement, HTRC developed a robust context document15 with two primary purposes: 昀椀rst, to linked the semantics of various EF key labels to terms in the Schema.org vocabular16y;and second, to formally name extensions to the Schema.org vocabulary that were created explicitly for use with EF data 昀椀les.
This change from ”ordinary” metadata to linked data a昀orded the opportunity for the HTRC
to exploit previous work transforming the HathiTrust catalog of MARC metadata records into
linked data records conforming to the Library of Congress’s (LoC) BIBFRAME metadata
ontology [
In addition to the named entity reconciliation work undertaken to create richer linked data, the EF 2.0 metadata section was further expanded to incorporate additional metadata to support research in the humanities and a variety of other 昀椀elds. These newly added metadata details include: • Additional links to brief and long metadata records in the HathiTrust catalog. • Finer-grained information regarding journals and monographic series, including which issue and volume numbers make up a particular digitized volum18e). • Plain-text classi昀椀cation descriptions for Library of Congress call numbers, when they are included in the original records, mapped directly from the LoC’s documenta1t9ion. 2.3.2. EF 2.0 Data The JSON-LD approach works wonderfully in the metadata section of EF 昀椀les by enabling a variety of linked data a昀ordances, and by forcing a distinction between the metadata describing the EF 昀椀les themselves, the metadata describing the volume from which the extracted features were derived, and the metadata describing the features themselves (e.g., token and part-ofspeech counts. However, the stricter rules of JSON-LD work less well for the data section of the document. Fortunately, the realities of processing JSON and JSON-LD documents allow any 15https://worksets.htrc.illinois.edu/context/ef_context.jsonld 16https://schema.org/ 17https://www.loc.gov/bibframe/ 18The semantics here are subtly di昀erent from ”imprint,” which contains this information in human-readable format.
The 昀椀ner grained ”issue” and ”volume” keys make this data machine-readable. 19https://id.loc.gov/authorities/classification.html so昀琀ware capable of parsing a JSON document to be used to parse the EF JSON-LD document as well, by simply ignoring the ”LD” semantics that might be used by Semantic Web tools. Applying an EF 2.0 昀椀le’s context document to the whole of the 昀椀le will end up corrupting the data due to the e昀케ciency-making measures designed into the 0.2 and 1.0/1.5 versions of the dataset’s schemas. However, it is not the case that the context document need be applied to the whole of each EF 2.0 昀椀le (or to any of it at all).
Context documents exist to enforce document semantics by mapping the keys to predicates in speci昀椀ed ontologies, like Schema.org or BIBFRAME, and the various data values to objects and subjects that are focal points and data values within the scope of those ontologies. However, in most cases it will not be necessary to apply the context document to the EF 2.0 昀椀le itself. If one chooses to apply a context document, for instance, because one wants to add volumelevel descriptive metadata to a triple store, this can be simply done by separating the metadata section from the data section and then applying the context document to just the metadata section of the 昀椀le to produce linked data triples in JSON format, which most triple store so昀琀ware can consume directly. Applying the context document to the data section of the EF 昀椀le will cause corruption of the data within the section because applying the context document will attempt to transform all of the ”key” values into URLS, even where the pattern used for the data is ”value-value” instead of ”key-value.”
With the exception of the above caveats, no signi昀椀cant changes were made to the data part of the EF schema (though some typographical errors in key labels that had persisted since the 0.2 version were corrected). Instead, the primary change to the data section in EF 2.0 concerns how the HTRC generates the extracted features data. Instead of employing the OpenNLP library as its primary means for parsing the OCR text 昀椀les, we switched to the Stanford NLP libra2r0y to parse and count the unigram tokens, and to apply POS tags to them. This change in NLP libraries necessitated a change in the tagsets being used for part-of-speech tags — from the OpenNLP POS tagset to the Stanford NLP tagset. The rest of the data supplied as part of EF datasets, including line counts, empty line counts, sentence counts, tokens, token POS tags, and associated counts, all remained consistent with the EF 2.0 release.
We conclude with a description of a major new HTRC e昀ort intended to make its rich EF data more accessible, more usable, and more readily available to researchers and librarians: Tools for Open Research and Computation with HathiTrust: Leveraging Intelligent Text Extraction (TORCHLITE), generously funded by the U.S. National Endowment for the Humanities in 2021.21
The TORCHLITE project is intended to enhance HTRC’s current data delivery infrastructure with a new, non-relational (“NoSQL”) database customized to host the EF dataset and a robust and well-documented API for accessing the data within it. The API allows for retrieving highly targeted subsets of the EF dataset, down to the level of individual volumes or pages, if desired. For example, using a HathiTrust volume ID, one can retrieve the complete EF data for the 20https://stanfordnlp.github.io/stanfordnlp/models.html 21Grant no. XXX corresponding volume, and with a HathiTrust volume ID and a sequential page number, one could retrieve the EF data for a speci昀椀c page. These subsets of the EF dataset can be gathered into user-de昀椀ned ”worksets,” which are simple lists of HathiTrust volume IDs (optionally with speci昀椀c page numbers).
Not only does the API enable the retrieval of all the EF data for a particular workset of volumes (or pages); it also enables retrieval of speci昀椀c, granular subsets of the EF data for that set of volumes, including volume-level metadata elements (e.g., title, publisher, publication date, genre, page count, etc.); page-level metadata such as algorithmically-determined page language; and page-level token counts, parts of speech, and line and sentence counts.
This API is intended to feed lightweight analytical tools (“widgets”) that can be incorporated into any modern website — for example, that of a library catalog or digital scholarly publication. These widgets can also be combined into a “dashboard,” together data selection and cleaning tools, statistical information about the workset that is represented in the various widgets (e.g., total word count, most frequent words, vocabulary density), word frequency visualizations (e.g., word clouds, charts comparing frequencies of user-selected words across volumes), and comparison of multiple volumes in that workset. The dashboard and widgets are built with modern, open data visualization libraries (e.g., chart.js, D3.js, etc.). A mockup of the TORCHLITE dashboard with two sample widgets is shown in Figure 1.
The TORCHLITE dashboard will be part of the HTRC Analytics site hatttps://analytic s.hathitrust.org,/ accounts for which are freely available to anyone from an educational or research institution, regardless of HathiTrust membership status, but the API is also open to others to access EF data for use in their research and tool development. The data and metadata selection and cleaning parameters, together with whatever statistical protocols are employed in a particular widget, can also be downloaded as an independent Jupyter notebook for further customization and manipulation by researchers.
Before the TORCHLITE project, EF data was accessed either through the standarrdsync 昀椀le transfer too2l2 or through a custom Python library called the HTRC Feature Rea2d3erW. hile these modes of access are serviceable and support individual researchers who want to download and process the data, they require of those researchers a certain amount of programming skills, and do not integrate well with the modern Web environment or current development trends, which assume simple API-based data access.
As an initial step towards building a data dashboard that would allow users to analyze and visualize subsets of the HathiTrust collection, HTRC created a Mongo2D4 Bto store a copy of the EF data and developed a simple API for accessing the data. 22https://wiki.htrc.illinois.edu/display/COM/Downloading+Extracted+Features 23https://github.com/htrc/htrc-feature-reader 24https://www.mongodb.com
The TORCHLITE API is a RESTful API with a few simple calls for creating and retrieving worksets and retrieving EF data. For instance, the following call will retrieve the volume metadata for a single volume: https://tools.htrc.illinois.edu/ef-api/volumes/{cleanhtid}/metadata. By replacing{clean-htid} with the HathiTrust volume ID for a volume of poetry by Victorian poet Algernon Charles Swinburne, we can view the volume metadata for a copy of Swinburne’sPoems and Ballads (1866): https://tools.htrc.illinois.edu/ef-api/vol umes/uc2.ark+=13960=t17m0815m/metadata. Below is the metadata retrieved in response to the above call: 1 { 2 3 4 5 6 7 "code": 200, "data": { "htid": "uc2.ark:/13960/t17m0815m", "metadata": {
"schemaVersion": "https://schemas.hathitrust.org/EF_Schema_MetadataSubSchema_v_3 .0", "id": "http://hdl.handle.net/2027/uc2.ark:/13960/t17m0815m",
A subset of 昀椀elds may be retrieved with thefields query parameter, e.g.:https://tools.htrc .illinois.edu/ef-api/volumes/uc2.ark+=13960=t17m058m1 /metadata?fields=metadata.pubDate ,metadata.pubPlace. This API call retrieves the following data: "code": 200, "data": { "metadata": { "pubDate": 1866, "pubPlace": { "id": "http://id.loc.gov/vocabulary/countries/enk", 8 9 10 11 12 } 13 } "type": "http://id.loc.gov/ontologies/bibframe/Place", "name": "England"
This API call retrieves all of the EF data for a single volume: https://tools.htrc.illinois.edu/ef-api/volumes/{clean-htid}
As in the previous example, replacing{clean-htid} with the HathiTrust volume ID for Swinburne’sPoems and Ballads, we can retrieve the EF data for the entire volume of poetry: https://tools.htrc.illinois.edu/ef-api/volumes/uacr2k.+=13960=t17m0815m. The output includes tokens and token counts for every page in the volume, and is thus too long to include here, but the data may be viewed by following the above link. As noted above, a subset of metadata and data 昀椀elds may be retrieved using thefields query parameter.
Additional calls are documented in the complete API documentationhtattps://htrc.stoplig ht.io/docs/torchlite./While not yet widely promoted, the TORCHLITE API is now open and available to users and developers. In May 2024, we will hold a developer-focused “Hackathon” event to encourage the adoption of the API by digital humanities researchers and developers and others applying text-mining methods to their research. Also in 2024, we will release the public version of the Dashboard, which is currently still in development and testing.
One particularly exciting feature of TORCHLITE’s API-based architecture is the new ability to combine EF data with non-HTRC data, retrieved through other APIs, into richer visualizations and other types of analysis. The widget shown on the le昀琀 side of Figure 1, for example, combines publication dates and author names from a workset of Extracted Features (all retrieved via the TORCHLITE API), with geographic and biographic data about those authors (retrieved from Wikidata via its API), to map the birthplaces of authors represented in the workset.
As with past versions of the Extracted Features dataset, future iterations will continue to include new volumes as they are continually added to the HathiTrust corpus and incorporate new statistical measures, data features, and functionality based on the state of the art in cultural analytics and linked data research.
For the more immediate future, the TORCHLITE project is re昀椀ning its existing suite of widgets and expanding it to include new types of analysis and visualization. Since an important goal of the project is to allow for community-conceived, community-created, and even externally-hosted widgets, the project also has plans to host a workshop and hackathon to attract scholars and developers from the HathiTrust user community interested in experimenting with the vastly improved access to the rich open data that HTRC has made available from the HathiTrust Digital Library.