Digital tools have played an important role in Digital Humanities (DH) since its beginnings. Accordingly, a lot of research has been dedicated to the documentation of tools as well as to the analysis of their impact from an epistemological perspective. In this paper we propose a binary and a multi-class classi昀椀cation approach to detect and classify tools. The approach builds on state-of-the-art neural language models. We test our model on two di昀erent corpora and report the results for di昀erent parameter con昀椀gurations in two consecutive experiments. In the end, we demonstrate how the models can be used for actual tool detection and tool classi昀椀cation tasks in a large corpus of DH journals.
in DH. These include tutorials such as the renowPnroegdramming Historian site1, but also tool
directories, such as thDeigital Methods Initiative2 or theSSH Open Marketplace3. The oldest
and most extensive tool directory is TAPToeRxt(Analysis Portal for Research). TAPoR started
out as a database for collecting primarily text analysis tools and recently has integrated another
tool directory named DiRDTi(gital Research Tools), which also contains tools that go beyond
the text modality13[
In this paper we propose a binary and a multi-class classi昀椀cation approach based on neural language models to detect and classify tools. For the detection, the binary classi昀椀er checks if sequence has tool or sequence has no tool, while for the multi-class task the model tries to assign one of seven TaDiRAH categories to a sequence that contains a tool. Taking up the perspective of scientometrics, we believe that searching for tools in existing DH publications is a good approximation of the most important tools actually used in DH research. While TAPoR may be categorized as a more qualitative, crowdsourcing-like approach, where DH scholars actively submit their tool candidates, our approach takes a more empirical, corpusbased route. In this current paper we present experiments that are based on a corpus of three established DH journals. However, we plan to use the approach on further DH publications such as DH abstracts or journals articles from neighboring disciplines such as information science or computational linguist2ic0s, 4[].
For the identi昀椀cation of so昀琀ware tools in academic publications, two main approaches can
be identi昀椀ed: (a) dictionary-based approaches and (b) machine learning approaches. Most
of the existing lexicon-based approaches use the TAPoR site as a basic dictionary to detect
tools mentioned in DH abstrac1t,s1[
In this paper, we present a new model based on the tansformer architecture RoB1E8R]Ttao[ reliably detect tools in DH publications and beyond (will be referrbeidntaroyamsodel). Furthermore, we enable the classi昀椀cation of tools into basic usage categaonraielyss(is, capturing, creation, etc.; will be referred tomauslti-class model), something that none of the aforementioned approaches has considered so far.
In this study, we will use two datasets for building and evaluating our models. The 昀椀rst corpus (cor-1) consists of 3,737 English-language publications from three DH jourCnoamlspu(ters and the Humanities, Digital Humanities Quarterly, Literary and Linguistic Computing/Digital Scholarship in the Humanities) published between 1966 and 2020. We use cor-1 as our main data input for the training and evaluation of the model. In addition, we use another existing corpus (cor-2) that has been used by26[] to train a machine-learning approach for extracting so昀琀ware mentions from scholarly articles. The corpus comprises approx. 1.9 million sentences from ADHO DH conference abstracts (2015 and 2020) and PLOS ONE papers from Linguistics and Sociology5and is used as a model-external data source to test the generalization capabilities of our model.
cor-1: This dataset was compiled by the authors and contains no annotations. To generate training, testing, and validation data from cor-1 we need to identify passages with tool mentions. A manual extraction of text passages in our data is not feasible given the amount of text documents available. To obtain high-quality training data, we 昀椀rst searched for particularly popular tools by a string based search. Popular tools were identi昀椀ed by going through existing lists and tutorials that document DH tools (see1T)aabnldeselecting only those tools that were mentioned in at least two di昀erent lists. From these tools, we then removed those that exhibited a high degree of ambiguity to further improve quality. This reduced lexicon, with a total of 246 tools, served as the starting point for creating the training data. In the next step, all papers in the corpus were tokenized and searched with the tool lexicon. Whenever a tool was found successfully, the sequence was transferred to the training dataset as a text snippet with 15 tokens before and 15 tokens a昀琀er the tool mention as a sequence-label pair.
Example: [analysis of the sample tweets . Students evaluated each of the default analysis tools in]V[oyant][, and were given examples of how to use the visualizations , statistics , and] 5The dataset is available frohmttps://gitlab.gwdg.de/sshoc/data-ingestion/-/treest/mera/repositories/extrac t.ion Note that cor-2 is already split into sentences which is di昀erent to the format we used in cor-1.
Such a sequence is annotated with the label tool. Then, the set of tool sequences was supplemented by randomly chosen no_tool examples. Those were gathered from evenly distributed samples from the entire corpus and serve as negative examples of tool occurrences. For the multi-class training dataset, we used the available categories which TAPoR provides for the majority of the 246 tools and mapped them to their corresponding TaDi2R]AanHn[otations in our training set. For those tools that were still lacking a category annotation, we provided a manual TaDiRAH annotation. Since TAPoR sometimes assigns multiple categories for single tools, we selected the – in our opinion – best-昀椀tting category to prevent our classi昀椀er from having to deal with multi-label situati6ons.
cor-2 is used to test the generalization capabilities of our model and to identify external examples which can be used to enrich the training data. In a second experiment, we will investigate how the model performance is in昀氀uenced by this alteration. Each line in cor-2 contains a sentence and an annotation that documents whether there are tool mentions in a particular section of the sentence or not. If any tool occurrences are annotated, we take over the sentence with the label tool. If there are no tool annotations, we label the sentence as no_tool. This data set therefore o昀ers the opportunity for us to de昀椀ne an external gold standard.
So昀琀ware and tool entity recognition can be seen as a text classi昀椀cation problem. Our approach is a supervised machine learning method that consists of a binary and a multi-class classi昀椀cation task. The basic idea of the approach is that the mention of a tool in a paper happens in a speci昀椀c linguistic context, which can be mapped and classi昀椀ed in the form of a sequence embedding using a Transformer-based language model. On the one hand, we try to determine whether a tool is in a sequence (binary task) and on the other hand, we try to assign a tool usage category to a sequence with a tool named in it (multi-class task). 6All the tools and their TaDiRAH categories can be foundhhtetrpes:://git.informatik.uni-leipzig.de/computationa l-humanities/tools-in-dh/tool-classifier/-/blob/main/resources/tool_lists/tadirah_taxonomy.json
We used a uniform framework for all classi昀椀cation tasks. The superior performance of
Transformer-based language models in the context of text classi昀椀cations was one of the main
decisions for our technical implementation. We experimented on the Transformer-based model
RoBERTa-Base as the main input for the classi昀椀cation componen1t8][. For e昀케ciency reasons,
distilRoBERTa-Base was used as a ”case-sensitive” and ”knowledge distilled” variant, as it
produced comparably good results to the RoBERTa-Base variant, but with considerably less
computation e昀ort. In detail, the solution consists of a Transformer-based encoder which uses the
classi昀椀cation token ([CLS]) of the model directly, a linear layer, and an output layer with
So昀琀max activation. For each instance, the encoder converts each token into an embedding vector
and adds a special classi昀椀cation token at the beginning of the sequence. The 昀椀nal hidden state
corresponding to this token is used as the aggregate sequence representation for classi昀椀cation
tasks [
The experiment design in our study consists of several parts that build on each other. In the 昀椀rst step, we use only the training data from cor-1. Following this step, we assess automatically classi昀椀ed false positive (FP) examples from cor-2 to qualitatively investigate errors in the model. In a second experiment, we enrich our training data with the identi昀椀ed FP-examples from cor-2 and re-evaluate the approach.
Experiment 1:In our 昀椀rst experiment, the Transformer model is 昀椀ne-tuned on the training data. Instead of using a static train, test and validation split, the datasets for each of the two classi昀椀cation tasks were split into 昀椀ve training and test sets according to the principle of a 5-fold cross-validation. This was done using the K-fold component in the SciKitLearn framework7. Finally, the dataset consists of 18,898 sequence-label pairs divided into 5 folds. For each round of cross-validation, four folds were used for training and the 昀椀琀h fold was used as a test dataset for validation. The experiment tests performance individually for the binary and multi-class cases on cor-1. We report on the average performances for each split of cor-1 in the evaluation section. We also evaluate the binary case of experiment 1 twice. On cor-2, we apply the binary model and use the existing annotations of this dataset to evaluate whether our classi昀椀er can correctly determine the existence of a tool. Qualitative Model AssessmentI:n a second step, we use this model to identify additional tool mentions from cor-1 and cor-2 which are not yet annotated. In detail, we look at where 7https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html the model makes mistakes and which text components are relevant for the classi昀椀cation. This allows us to better understand what the model learns and what consequences our modeling decisions have.
Experiment 2:In the second experiment, we will expand the training data set. Based on the evaluations from experiment 1 and the qualitative tests, we repeat the evaluation again under di昀erent conditions. For this purpose, we will include the false positive examples from cor-2 into the training data.
In this section we present the evaluation of the classi昀椀cation results with the values precision, recall and F1. Furthermore, we will investigate the multi-class case with micro and macro averaged F1 values. In the 昀椀rst experiment, we will document the binary and the multi-class case. In the qualitative evaluation, we will show some examples and explain the decisions of the classi昀椀er. Based on these observations we adapt the model and repeat the evaluation.
As mentioned before, we based the evaluation in our 昀椀rst experiment on a 5-fold crossvalidation restricted to cor-1. Based on the averaged metrics, the result of the binary classi昀椀cation is very good on the corpus internal evaluation. The detailed results are shown2.in Table For the second case of multi-class classi昀椀cation, Ta3blsehows the results. With an accuracy of 0.974 the model shows excellent performance within cor-1. Although the recall is below 0.9 in two categories, the overall precision is very high.
Based on these promising results, we wanted to test how the model behaves when
transferred to another text type. Since we have annotations in cor-2 to indicate whether a sentence
contains a tool, we can evaluate the binary approach very easily outside the training data. The
results are documented in Tab2l.eAlthough we have a very high recall in the classi昀椀cation of
tools, the precision here is very low. This can only be due to false positive classi昀椀cations made
in cor-2. For this reason, we will look at the classi昀椀cation decisions in some more detail in the
next step and take a closer look at the false positive decisions found in cor-2. We looked at the
models output using thCeaptum-based Transformers Interpret library [
Example C
Four of the cafeteria diet food items were administered per day and variety of diet was maintained by alternating food items daily . taken from the output of Transformers Interpret. The separation of the tokens was determined directly from the Transformer, which uses subword tokens. This results in fragmented single words in some representations (sDeei昀 – erences in example A).
The FP predicted in cor-2 are almost all descriptions of a methodological approach. In these examples, one can relatively well recognize a certain pattern of syntactic components. In all examples, and also in many FP classi昀椀cations from cor-2, there are constructions of nouns and proper nouns followed by verbs that indicate a methodological activity. In our examples, we 椀昀nd were examined using, were considered signi昀椀cant if , was realized simply by, be opened or closed by, were administered per and was maintained by. This surface form is very similar to the constructions used in the naming of tools. Actually, we could just insert a tool at the end of many of the phrases and the sentence would make perfect syntactical sense. We believe that all of these phrases are standardized language constructs to document a procedural approach. Since the naming of tools typically belongs to the methodical part of the text, their descriptions and what was done with them strongly resemble these passages.
The lesson to be learned here, is that our classi昀椀er learns more about syntactical patterns or sentence constructions than about the semantic representations of the entity references they contain. We conclude that cor-2 by containing a broader scienti昀椀c domain, there are more such constructs in the data that would not meet the de昀椀nition of a tool when annotated. While examples A-C demonstrate how syntactic constructions in昀氀uence the classi昀椀cation model in a way it is likely to produce false positives, there are also many examples for words that have higher weights and that actually imply some realistic tool context, for minesatsaunrec,e(arithmetic) mean, assess, score oranalyses.
A昀琀er showing that the false positive classi昀椀cations come from a somewhat more restrictive de昀椀nition of what is and is not a tool, we can still evaluate whether the results can be improved by expanding the training data set with the false positive classi昀椀cations from cor-2. We retrained the model with 500 additional data points and evaluated it again on the cor-2. The results show a boost in quality, which also shows that our model is easy to adapt and can be adapted to other de昀椀nitions of tool mentions.
The experiments with extended training samples show that we can easily extend our training data with another dataset and not lose any quality on the cor-1. On the contrary, we gain almost 0.08 points on the cor-2 and the precision has increased signi昀椀cantly. Although both datasets were created and annotated with a slightly di昀erent idea of what a tool is, we were able to create a reliable classi昀椀er. We assume that the loss of quality with cor-2 is also due to the fact that we do not have sequences, but sentences, which can sometimes be very short.
In this section we demonstrate how our classi昀椀er can be used to detect and classify tools in cor-1 – a corpus with three popular DH journals. With the detection of tools in scienti昀椀c papers we can learn more about methodological standards and so昀琀ware packages that are being used within a certain discipline, such as the DH. Furthermore, the categorization of tool occurrences is useful to understand typical scholarly activities of a discipline and to investigate which activities have been digitally enhanced so far.
As was argued in the beginning, the TAPoR 3.0 tool repository includes a great number of tools already. However, using our binary classi昀椀er on cor-1, we found numerous examples of tools that are currently not included in TAPoR 3.0. Our approach produces sequences of length 31 tokens and labels those as tool or no_tool. To be able to extract the actual tool entities from these sequences, we used the Stanford Stanza POS tagg2e2r] a[nd generated a list of all the proper nouns contained in the sequences. Besides tool entities, these also include place and person names, organizations and institutions, etc. We had a look at any proper noun with a document frequency> 3 and removed any non-tool entities from the list. This way we were able to detect around 100 tools that are not yet reported in TA8PoR 3.0. 8Note that we also identi昀椀ed around 50 tool candidates that need some more close reading to verify if they are actual tools or some other named entity.
In the following we present the main types of tools (manually assigned by us) that were identi昀椀ed by our classi昀椀er.9 • markup technologies (TEI, XML, SGML, HTML, XSLT, XPointer, etc.) • metadata standards (Dublin Core, CIDOC, MARC, etc.) • programming languages (Java, BASIC, Pascal, COBUILD, PASCAL, PHP, etc.) • authoring tools (HyperCard, Photoshop, Storyspace, PageMaker, etc.) • operating systems (DOS, UNIX, Linux, Windows, etc.) • web browsers (Netscape, Lynx, Internet Explorer, etc.) • web services (Google, Google Books, Google Scholar, Google Earth, Google Maps, Facebook, YouTube, Wikipedia, Yahoo, Flickr, Gopher, Internet Archive, etc.) • NLP (Word Net, TreeTagger, etc.) • crowdsourcing (Mechanical Turk, Transcribe Bentham, etc.) • corpora and databases (Perseus, Project Gutenberg, EEBO, Europeana, HathiTrust, Pro
Quest, JSTOR, etc.) • statistics (SPSS, Zeta, etc.) • word processors (Microso昀琀 Word, WordPerfect, WordStar, WordNet, Wordstar, MS Word, etc.) • infrastructures (CLARIN, DARIAH, etc.) • hardware (Kinect, iPad, etc.)
One reason for thesneew tool discoveries could well be our very broad de昀椀nitiodnigoitfal tools, which might be somewhat narrower in TAPoR 3.0. For future analyses of tools in DH, and also for a further 昀椀ne-tuning of the classi昀椀er, we will need to discuss what should be counted as a tool and what not. While we beliemvaerkup technologies and programming languages could be classi昀椀ed as ”tools for making” (as opposed to tools for exploring and think3in],gw)[eb services and corpora might be harder to justify as actual tools.
Our multi-class model was able to classify the tool sequences into six of the seven main TaDIRaH categories for tool types. ”Interpreting” was not included in the model, as we did not have enough examples in our training data. Interestingly, tools for ”Analysis” are by far the most frequent tools, followed by tools for ”Dissemination” and ”Enrichment” (se5e).Table
To see how these categories have evolved through time we have plotted the frequencies diachronically (see Figur1e). While we can observe a general diachronic upwards trend for most of the categories, it is noticeable that tools for analysis are not only the most frequent tool category in total, but that they are also more or less consistently mentioned in DH publications throughout the whole time span – with a general upwards trend, similar to the other categories. Tools for analysis include examples from many di昀erent areas, inclundeitnwgork 9For a list of all tool entitieshtseteps://docs.google.com/spreadsheets/d/1ZMV3kFAF2mooclIlvW_0gJYSh68FlE_j gGhyeT-VVqc/edit?usp=sharing analysis (Gephi, Cytoscape, NodeGoat, networkXn)a,tural language processing (NLTK, Standford POS Tagger, Gensim),text analysis (Voyant, AntConc)g,eo information systems (ArcGIS, Lea昀氀et), music analysis (music21) and many more. This observed dominance of tools for analysis is also interesting for future work, as we believe this is the category of tools that has the most potential for epistemological shaping of scholarly research processes, which means we will have a closer look at this category in follow-up studies.
In this paper we have presented a Transformer-based classi昀椀er for the detection and classi昀椀cation of digital tools. The task of tool detection worked very well (F1 = 0.978) for a large corpus of DH journals, which also was the basis of the training data. Tool detection results for another corpus that includes ADHO abstracts and PLOS ONE articles from Sociology and Linguistics also worked well, with an F1 score of 0.69 and an increase to 0.77 a昀琀er 昀椀ne-tuning the model with 500 false positive examples from the novel corpus. It can be concluded that the model yields good results in the domain of DH journal publications already, and can also be successfully adopted to other domains.
We plan to use this model for follow-up studies in which we want to investigate the actual tool results and their epistemological implications for DH scholars in a more detailed way. First, we plan to enhance our current DH corpus by adding some more recent journals such as ”Journal of Cultural Analyt1i0cso”r ”International Journal of Digital Human1i1t.ieWs”e also plan to add abstracts from past DH conferences, which are available from the ”Index of Digital Humanities Conferenc1e2s.”We will evaluate the performance of our model on such an enlarged corpus and 昀椀ne-tune it with further training data if necessary. As was mentioned in the previous section, we will also have to agree upon a working de昀椀nition of atcotolusa[lsee 3] as opposed to digital resources, services, infrastructures, etc. We certainly plan to correlate the tools and tool categories with further metadata such as geolocation of authors, gender of authors, disciplinary background of authors, keywords and LDA topics. By means of such large-scale analyses we hope to be able to contribute to the historiography of DH through the lens of what has been called ”tool scienc2e4”].[