Direct and Indirect Annotation with Generative AI: A Case Study into Finding Animals and Plants in Historical Text Arjan van Dalfsen1,∗ , Folgert Karsdorp2 , Ayoub Bagheri3 , Dieuwertje Mentink1 , Thirza van Engelen1 and Els Stronks1 1 Department of Language, Literature and Communication, Utrecht University, Trans 10, Utrecht, 3512 JK, The Netherlands 2 KNAW Meertens Instituut, Oudezijds Achterburgwal 185, 1012 DK Amsterdam, The Netherlands 3 Department of Methods and Statistics, Utrecht University, Padualaan 14, 3584 CH, Utrecht, The Netherlands Abstract This study explores the use of generative AI (GenAI) for annotation in the humanities, comparing direct and indirect annotation approaches with human annotations. Direct annotation involves using GenAI to annotate the entire corpus, while indirect annotation uses GenAI to create training data for a special- ized model. The research investigates zero-shot and few-shot methods for direct annotation, alongside an indirect approach incorporating active learning, few-shotting, and k-NN example retrieval. The task focuses on identifying words (also referred to as entities) related to plants and animals in Early Modern Dutch texts. Results show that indirect annotation outperforms zero-shot direct annotation in mimick- ing human annotations. However, with just a few examples, direct annotation catches up, achieving similar performance to indirect annotation. Analysis of confusion matrices reveals that GenAI annota- tors make similar types of mistakes, such as confusing parts and products or failing to identify entities, which are broader than those made by humans. Manual error analysis indicates that each annotation method (human, direct, and indirect) has some unique errors. Given the limited scale of this study, it is worthwhile to further explore the relative affordances of direct and indirect GenAI annotation methods. Keywords large language models, natural language processing, historical text, token classification, environmental humanities 1. Introduction The introduction of advanced generative AI (GenAI) models has sparked interest among hu- manities scholars in leveraging these tools to extract structured information from texts [3, 23, 2, 12, 25, 18, 19, 4, 5]. So far, the use of GenAI in the humanities has primarily involved “direct CHR 2024: Computational Humanities Research Conference, December 4–6, 2024, Aarhus, Denmark ∗ Corresponding author. £ j.a.vandalfsen@uu.nl (A. v. Dalfsen); folgert.karsdorp@meertens.knaw.nl (F. Karsdorp); a.bagheri@uu.nl (A. Bagheri); d.l.mentink@students.uu.nl (D. Mentink); t.w.e.vanengelen@students.uu.nl (T. v. Engelen); e.stronks@uu.nl (E. Stronks) ç https://www.karsdorp.io/ (F. Karsdorp); https://ayoubbagheri.nl/ (A. Bagheri) ȉ 0000-0002-4209-4063 (A. v. Dalfsen); 0000-0002-5958-0551 (F. Karsdorp); 0000-0001-6366-2173 (A. Bagheri); 0000-0001-9741-7264 (E. Stronks) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1053 CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings annotation”, where GenAI analyzes a corpus without further interference. This approach has shown promise, potentially “supercharging the humanities” [11]. Researchers in Natural Language Processing (NLP) have proposed an alternative “indirect annotation” framework. This two-step process involves GenAI generating training data, which is then used to train a specialized model. This approach offers potential cost and performance advantages over direct annotation [31, 32]. However, indirect annotation’s effectiveness has primarily been demonstrated on languages well-represented in GenAI training data, raising questions about its applicability to texts from smaller languages or historical variants often encountered in humanities research. As a first exploration into its usability in the humanities, our study tests GenAI as an indirect annotator for nature-entities in historical Dutch texts. We employ the LLMaAA (Large Lan- guage Models as Active Annotators) framework [32], which combines few-shotting, k-Nearest Neighbors (k-NN) example retrieval, and active learning. Our research compares the perfor- mance of indirect annotation, i.e. an LLMaAA-derived model, against both human annotations and direct annotation with GenAI. We find that our proposed method of indirect GenAI annota- tion performs better than fully-unsupervised direct GenAI annotation. However, we also find that providing direct annotation with demonstrations (i.e., examples of annotations) results in similar performance. Moreover, our study reveals that humans, direct GenAI annotators, and indirect GenAI annotators each have unique weaknesses and strengths. This study is structured as follows: We first examine the broader context of using GenAI for annotation in humanities research. We then provide an overview of current research on direct and indirect annotators in NLP. Subsequently, we introduce our specific use-case: identifying animals and plants in historical texts. Finally, we detail our methodology for comparing the performances of human annotation, direct GenAI annotation, and indirect GenAI annotation. 2. Related Work 2.1. GenAI annotations in humanities In the humanities, research on GenAI annotation has primarily focused on direct annotation experiments. Studies have compared GenAI methods with traditional approaches and human annotators across various tasks, including sentiment analysis [2, 25, 5], topic detection [18], and text classification [19]. Findings generally suggest that while GenAI often outperforms dictionary-based methods, it typically falls short of specialized models. However, Karjus [12] reports human-level annotations by GenAI across diverse tasks and languages, proposing a machine-assisted mixed methods approach. These studies underscore the potential of GenAI in humanities research, while also highlighting the need to explore both direct and indirect annotation approaches to fully leverage its capabilities. 2.2. GenAI as direct annotators in NLP Direct GenAI annotation involves prompting GenAI to annotate a dataset for immediate use. Studies assessing this approach have found that while GenAI generally lags behind state-of-the- art models [9, 16, 24, 33], it often equals or outperforms crowd-workers [30, 33, 10]. Challenges 1054 in direct GenAI annotation include difÏculties with long-tail target types, irrelevant context, and specific tasks like sequence tagging [16, 24]. These limitations have led to the exploration of indirect annotation methods, which aim to address these shortcomings by integrating GenAI in a more targeted manner. 2.3. GenAI as indirect annotator in NLP In what we coin the indirect GenAI annotation framework, GenAI is not employed to perform the entire annotation task on a given dataset. Instead, it is used to annotate a specific subset of the dataset. This annotated subset is then subjected to further fine tuning by another model, such as a BERT model. Wang et al. [31] found models trained on GenAI-annotated data equal to human-annotated models and outperforming direct use of GenAI. Ding et al. [7] largely echo this but also high- light a practical problem when it comes to textual analysis: GenAI is good at finding entities, but oftentimes struggles with defining the boundaries of these entities. Li et al. [20] pro- pose a CoAnnotating framework, in which GenAI output-uncertainty is measured and annota- tions with the highest uncertainty (i.e., a lack of result robustness when confronted with small prompt perturbations) are sent to a human annotator. While they report promising results, there is the disadvantage of higher costs. With Large Language Models as Active Annotators (LLMaAA) by Zhang et al. [32], the idea is to use active annotation to make the downstream specific model better. It includes: • Few-shotting: putting exemplary annotations in the prompt for the GenAI (this helps GenAI to annotate [21]); • k-NN example retrieval: sequence embeddings of the text to annotate and the examples are used to select the examples closest to the new text for few-shotting; • Training cycles: doing step-by-step training, where first a specific model is trained, new data is annotated by the GenAI, and the specific model is trained again; • Active learning: selecting examples for indirect annotation for which the current model struggle; • Automatic reweighting: assigning learnable weights to the annotated training sam- ples [27] (this makes it possible to reduce the impact of noisy labeling by GenAI). The authors test their method for NER and Information Extraction (modern Chinese and mod- ern English) and find that the resulting model strongly outperforms zero-shot direct annotation with GenAI. In comparison to few-shot GenAI annotation (with k-NN optimized examples), LLMaAA shows a marginal performance advantage, besides the obvious advantages when it comes to robustness, costs, and speed, making it promising for humanities research. 2.4. Plants and animals In this study, we research the detection of plants and animals in historical texts, which can be seen as the traditional NLP task of token (in sequence) classification or NER. Roughly starting with publications as Man and the Natural World [29] and The Animal Estate [28], humanities’ scholarly interest in nature has skyrocketed. This is only natural, considering a widely shared 1055 sense of humanity being in environmental, ecological, and climate crises. For cultural histori- ans, the main question has been how humans observed, interpreted, and represented and thus perceived nature [22]. Research on this topic has virtually exclusively been done qualitatively. Although this approach provides illuminative insights, a qualitative approach will always work with a relatively narrow scope because of the sheer size of the historical record, thus resulting in a less precise large-scale overview of the studied phenomenon. It has, therefore, great po- tential to complement qualitative studies with quantitative research. 3. Methods In this section, we describe the methodology employed to compare direct and indirect GenAI annotation strategies for identifying plants and animals in historical Dutch texts. First we ex- plain the data parsing used to prepare our dataset. Following this, we describe the annotation procedure undertaken by human annotators. Then, we address the dataset creation. Subse- quently, we describe the token classification. After this, we describe our prompts. Then, we document the training process of the indirect annotation models. Finally, we outline the ways in which we compare the annotation approaches. 3.1. Data parsing Our study used the Digitale Bibliotheek voor de Nederlandse Letteren (DBNL) [6], comprising about 1,500 diverse Dutch texts. After preprocessing, the corpus yielded approximately 7 mil- lion unique sentences [17, 8]. 3.2. Annotation procedure Two texts from the 1750s were selected for manual annotation by two Early Modern Dutch literature experts. 200 sentences, parsed to have a minimal length of 10 and a maximal length of 100 words, were annotated using the INCEpTION tool [15] (cf. Fig. 1), following iteratively developed guidelines (Appendix A). The annotation schema tagged entities on three levels: Category (Plants/Animals), Type (Organism/Part/Product/Collective), and Usage (Literal/Sym- bolical/Petrified). For example, in “The bear grabbed an apple with its claw”, “bear” would be tagged as Animals-Organisms-Literal. 3.3. Task description The annotated sentences were split into demonstration, validation, and test sets. We concep- tualized the detection of animals and plants as a token classification task. Prompts for both direct and indirect annotators included the annotation schema, with technical details omitted to improve performance (full prompts in Appendix B, model settings in Appendix C). 3.4. Training indirect annotation models For indirect annotation, we adapted the LLMaAA framework by Zhang et al. [32], integrat- ing it with Huggingface and OpenAI ecosystems. We used GysBERT [1] for historical Dutch, 1056 Figure 1: The annotation interface of the INCEpTION tool, which was used for conducting the human annotations. applied k-NN few-shot selection (with the paraphrase-multilingual-mpnet-base-v2 sentence transformer [26]), and confidence-based active learning. Automatic reweighting was not in- cluded. GPT4o served as the LLM backbone. To address the scarcity of plant and animal enti- ties, we employed a pre-filtering strategy using GPT-3.5. The specialized model underwent 10 training rounds with 10 epochs each, adding 50 example sentences per round (25 pre-filtered sentences + 25 sentences with lowest confidence). This process was repeated for two datasets, using five distinct random seeds, resulting in 10 indirect models. It is important to point out that the indirect annotators have not seen any of the human annotations in their training regime. However, the 500 sentences they are trained on are an- notated with the help of the human-annotated demonstration set and the performance of the model is determined by its score on a human-annotated validation set. 3.5. Comparing annotation strategies All of the analyses below were done on the annotations of the various strategies on the held-out test set. Note that the sentences in this set came from the same documents as the demonstration and validation. To assess the inter-annotator agreement among human annotators, direct GenAI, and indi- rect GenAI approaches, we conducted inter-annotator agreement analysis, confusion matrix analyses, and performed a manual error analysis. We compare each human annotator’s results against each automatic system’s output: 1. Human Annotations • Human1: Annotations from the first human annotator • Human2: Annotations from the second human annotator 1057 2. Direct GenAI Annotations • Direct zero-shot: Zero-shot direct annotation (without examples) • Direct few-shot 1: Few-shot direct annotation using examples from Human1 • Direct few-shot 2: few-shot direct annotation using examples from Human2 3. Indirect GenAI Annotations • Indirect1: Indirect annotation using examples from Human1 • Indirect2: Indirect annotation using examples from Human2 For the inter-annotator agreement, positives-only weighted F1 is used as a metric. The positives-only weighted F1 is the weighted average of all harmonious means of precision and recall of labeled entities. Thus, words that were not labeled as referring to a plants or animals related word, which is by far the most common category in this situation, are disregarded. For all GenAI annotations (direct and indirect), predictions were done five times. The average and standard deviation of the inter-annotator agreements were calculated from these iterations. The observed low variance across models and approaches suggests that these results are likely stable, despite the relatively small number of iterations. The confusion matrices were made by comparing the human annotations to single instances of the other strategies. It should be noted that Human1 had labeled one example as ”None- label” (a token was tagged but no label was chosen), which was removed later for the process of making the confusion matrices. In addition to the confusion matrix analysis, we performed a manual error analysis on the annotations for the held-out test set. 4. Results 4.1. Inter-annotator agreement Table 1 shows the inter-annotator agreement using positives-only weighted F1 as a metric. Key findings include: 1. Annotators of the same type resemble each other best (e.g., Human1 is closest to Hu- man2). 2. Zero-shot direct annotation demonstrates lower internal coherence (F1 = 0.74) compared to few-shot direct (F1 = 0.9 and 0.88) and indirect annotation (F1 = 0.81 and 0.86). 3. Direct zero-shot annotation consistently underperforms, while annotations from the other human annotator achieve the highest agreement. 4. Few-shot direct and indirect annotations perform similarly, falling between zero-shot and human performance. 5. GenAI models don’t simply mimic the specific human annotator they were trained on, but generalize from the examples. These results suggest that indirect annotation and few-shot direct annotation are more reli- able methods for replicating human-like annotations compared to zero-shot approaches. The choice between these methods may depend on factors beyond performance, such as ease of implementation or specific task requirements. 1058 Table 1 Inter-annotator agreement (positives-only weighted F1): The first two columns of the table display the annotations treated as a reference point, the so-called “gold labels”. The corresponding rows indicate the level of agreement of the other strategies with these gold labels. Method human1 human2 direct zero-shot direct few-shot1 direct few-shot2 indirect1 indirect2 human1 1.0000 human2 0.8332 1.0000 direct zero-shot 0.5456 0.5206 0.7352 ± 0.0629 ± 0.0856 ± 0.1572 direct few-shot1 0.5939 0.5842 0.6258 0.9039 ± 0.0256 ± 0.0341 ± 0.0869 ± 0.0523 direct few-shot2 0.5738 0.6002 0.6090 0.8323 0.8822 ± 0.0212 ± 0.0249 ± 0.0756 ± 0.0290 ± 0.0650 indirect1 0.5481 0.5489 0.4828 0.6927 0.6805 0.8099 ± 0.0524 ± 0.0330 ± 0.0718 ± 0.0419 ± 0.0428 ± 0.1163 indirect2 0.5753 0.5785 0.5099 0.7218 0.7149 0.7976 0.8555 ± 0.0277 ± 0.0139 ± 0.0746 ± 0.0328 ± 0.0400 ± 0.0589 ± 0.0773 4.2. Confusion matrices While F1 scores provide an overall measure of performance, they do not offer insights into how well the annotation methods perform for individual labels. To gain a more nuanced un- derstanding of the agreement (or lack thereof) between the annotations, we turn to confusion matrices (Fig. 2), which provide insights into individual label performance across annotation methods: 1. Human2’s annotations most closely align with Human1’s. 2. Zero-shot direct annotation shows low recall, with many entities remaining unlabeled. 3. All GenAI annotators struggle with both precision and recall: • Precision errors: confusion between “Animals Parts Literal” and “Animals Products Literal”. • Recall errors: suggesting “No Label” for entities labeled by Human1. 4. GenAI methods, including indirect approach, produce labels not found by human anno- tators, demonstrating higher label diversity. These patterns highlight the strengths and weaknesses of each annotation method, emphasiz- ing the need for careful selection and potential combination of approaches in annotation tasks. 4.3. Manual Error Analysis Our manual examination of the annotations (Appendix D) reveals distinct error patterns across different annotation strategies. First, human annotators occasionally overlook words in a sen- tence. For instance, in sentence 10, Human1 tagged the word vleesch (meat) only twice out of its three occurrences. Such errors likely stem from simple oversight rather than misunder- standing (although misunderstandings occur too). Second, few-shot direct annotation strate- gies sometimes struggle with entity aggregation. A notable example is sentence 41, where 1059 (a) Human1 vs. Human2 (b) Human1 vs. Direct Zero-Shot (c) Human1 vs. Direct Few-Shot1 (d) Human1 vs. Direct Few-Shot2 (e) Human1 vs. Indirect1 (f) Human1 vs. Indirect2 Figure 2: Confusion matrices comparing Human1 annotations with those from Human2 and different GenAI strategies. The red box indicates animals parts and animal products, which often proves to be a hard category for the annotators to decide on. Although all GenAI annotations were done five fold, only one of these instances is used for the confusion matrix. 1060 “nek van het Varken” (neck of the pig) is incorrectly labeled as a single entity, instead of rec- ognizing “neck” and “pig” as separate entities (with distinct labels “Animals Products Literal” and “Animals Organisms Literal”, respectively). Importantly, these errors are incidental, not systematic, suggesting they are unlikely to be consistently repeated. Third and finally, while indirect annotation models are (once trained) deterministic, and therefore not susceptible to in- cidental mistakes, they can produce counter-intuitive systematic errors. A revealing example is sentence 31, where “salt” is misclassified as “Plant Product Literal”. This error likely stems from the proximity of salt to spices like pepper and nutmeg in the transformer model’s vector space. 5. Discussion This study compared direct and indirect annotation with GenAI in a humanities context, fo- cusing on identifying plant and animal-related words in Early Modern Dutch texts. While we studied a specific case, we believe that this method can be used for a wide range of applica- tions. Our findings reveal both the potential and limitations of various annotation strategies for historical humanities studies. Indirect annotation demonstrates clear advantages over fully-unsupervised zero-shot direct annotation, particularly in terms of recall. However, few-shot direct annotation achieves com- parable performance to indirect annotation, suggesting that both approaches have merit in different contexts. Based on these results, we advise against using zero-shot direct annota- tions for historical humanities research. Its significantly lower recall compared to the alterna- tives means that many relevant entities are likely to be missed, potentially skewing research outcomes. The choice between few-shot direct annotation and indirect annotation is less clear- cut, as both display similar F1 scores. Here, time, cost, and technical considerations should be considered. The unique error patterns suggest two important points. First, it is crucial to investigate shortcomings of chosen methods on a micro-level to be aware of specific pitfalls. Second, there’s potential for stacking annotation methods: human, direct, and indirect annotation can be applied to the same texts, after which points of contention can be analyzed. In this way, they may bundle their strengths and cover each other’s weaknesses. Regarding the generalizability of this explorative study, several points should be noted. To- ken labeling is a specific task, and the behavior of direct and indirect GenAI annotators may differ for tasks of another nature. The prompts used for direct annotation have not been system- atically tested, and it’s possible that especially zero-shot direct annotation would have better results with more guidance regarding the output format. The held-out test set was small and from the same document (i.e., not the same data) as the training data, which might have influ- enced the results. The training of the indirect annotation model has been done with just 500 examples, a typically low number for fine-tuning its underlying transformer model. Addition- ally, during the training of the indirect model, automatic reweighting was not applied (as we deemed its effects in the LLMaAA paper to be marginal), but integrating it might approve the model. 1061 6. Conclusion Despite these limitations, this study shows potential for applying GenAI as indirect annota- tors in humanities research. However, there are are notable differences to other annotation strategies. Future research should address questions about indirect GenAI annotation’s perfor- mance on other tasks (e.g., text classification), the impact of prompt optimizing frameworks (e.g., DSPy [13, 14]), and the potential of combining human and GenAI annotations to check on each other. Author Contributions Conceptualization: Arjan van Dalfsen, Folgert Karsdorp, Ayoub Bagheri, Els Stronks; Data Cu- ration: Thirza van Engelen, Dieuwertje Mentink; Investigation: Arjan van Dalfsen; Method- ology: Arjan van Dalfsen; Writing - Original Draft: Arjan van Dalfsen; Writing - Review & Editing: Folgert Karsdorp, Ayoub Bagheri, Els Stronks; Visualization: Arjan van Dalfsen; Su- pervision: Folgert Karsdorp, Ayoub Bagheri, Els Stronks. Acknowledgments This research would not have been possible without the financial support from Utrecht Uni- versity AI Labs, the Meertens Instituut, and the Utrecht University focus area Advanced Data Science. Their generous contributions provided the necessary resources to conduct this study. Additionally, we would like to extend our gratitude to SURF for providing cloud computing services, which were instrumental in the analysis and processing of our data. References [1] E. M. Arevalo and L. Fonteyn. “Non-Parametric Word Sense Disambiguation for Histor- ical Languages”. In: Proceedings of the 2nd International Workshop on Natural Language Processing for Digital Humanities. Taipei, Taiwan, 2022. url: https://aclanthology.org/20 22.nlp4dh-1.16. [2] J. Borst, J. Klähn, and M. Burghardt. “Death of the Dictionary?– The Rise of Zero- Shot Sentiment Classification”. In: Computational Humanities Research Conference (CHR). Paris, France, 2023, pp. 303–319. url: https://ceur-ws.org/Vol-3558/paper3130.pdf. [3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-Shot Learners. arXiv preprint https://arxiv.org/abs/2 005.14165. 2020. doi: 10.48550/arXiv.2005.14165. 1062 [4] Y. Chen, S. Li, Y. Li, and M. Atari. Surveying the Dead Minds: Historical-Psychological Text Analysis with Contextualized Construct Representation (CCR) for Classical Chinese. arXiv preprint https://arxiv.org/abs/2403.00509. 2024. doi: https://doi.org/10.48550/arXiv.240 3.00509. [5] T. Dejaeghere, P. Singh, E. Lefever, and J. Birkholz. “Exploring Aspect-Based Senti- ment Analysis Methodologies for Literary-Historical Research Purposes”. In: Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) LREC-COLING-2024. Torino, Italia: ELRA and ICCL, 2024. url: https://acla nthology.org/2024.lt4hala-1.16. [6] Digitale Bibliotheek voor de Nederlandse Letteren (DBNL). Collectie publiek domein. htt ps://www.dbnl.org/letterkunde/pd/index.php. 2023. [7] B. Ding, C. Qin, L. Liu, Y. K. Chia, B. Li, S. Joty, and L. Bing. “Is GPT-3 a Good Data An- notator?” In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada, 2023. doi: 10.18653/v1/2023.acl-lo ng.626. [8] M. van Gompel. python-ucto [computer software]. https://languagemachines.github.io/u cto/. 2023. [9] R. Han, T. Peng, C. Yang, B. Wang, L. Liu, and X. Wan. Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors. arXiv preprint https://arxiv.org/abs/2305.14450. 2023. doi: https://doi.org/10.48550/arXiv.230 5.01445. [10] X. He, Z. Lin, Y. Gong, A.-L. Jin, H. Zhang, C. Lin, J. Jiao, S. M. Yiu, N. Duan, and W. Chen. AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators. arXiv preprint https://arxiv.org/abs/2303.16854. 2024. doi: https://doi.org/10.48550/arXiv.230 3.16854. [11] A. Karjus. Large language models to supercharge humanities and cultural analytics re- search. Poster presentation at CHR2023 https://2023.computational-humanities-researc h.org/programme/. 2023. [12] A. Karjus. Machine-assisted mixed methods: augmenting humanities and social sciences with artificial intelligence. arXiv preprint https://arxiv.org/abs/2309.14379. 2023. doi: https://doi.org/10.48550/arXiv.2309.14379. [13] O. Khattab, K. Santhanam, X. L. Li, D. Hall, percy Liang, C. Potts, and M. Zaharia. Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge- Intensive NLP. arXiv preprint https://arxiv.org/abs/2212.14024. 2022. doi: https://do i.org/10.48550/arXiv.2212.14024. [14] O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv preprint https://ar xiv.org/abs/2310.03714. 2023. doi: https://doi.org/10.48550/arXiv.2310.03714. 1063 [15] J.-C. Klie, M. Bugert, B. Boullosa, R. E. de Castilho, and I. Gurevych. “The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation”. In: Pro- ceedings of the 27th International Conference on Computational Linguistics: System Demon- strations. Santa Fe, New Mexico, 2018. url: https://aclanthology.org/C18-2002. [16] J. Kocoń, I. Cichecki, O. Kaszyca, M. Kochanek, D. Szydło, J. Baran, J. Bielaniewicz, M. Gruza, A. Janz, K. Kanclerz, A. Kocoń, B. Koptyra, W. Mieleszczenko-Kowszewicz, P. Miłkowski, M. Oleksy, M. Piasecki, Ł. Radliński, K. Wojtasik, S. Woźniak, and P. Kazienko. “ChatGPT: Jack of all trades, master of none”. In: Information Fusion 99 (2023), p. 101861. doi: https://doi.org/10.1016/j.inffus.2023.101861. [17] Koninklijke Bibliotheek. Over ons - Diensten DBNL. https://www.kb.nl/over-ons/dienst en/dbnl. 2024. [18] A. Kosar, G. D. Pauw, and W. Daelemans. “Comparative Evaluation of Topic Detection: Humans vs. LLMs”. In: Computational Linguistics in the Netherlands Journal 13 (2024), pp. 91–120. url: https://www.clinjournal.org/clinj/article/view/173. [19] L. D. Langhe, A. Maladry, B. Vanroy, L. D. Bruyne, P. SIngh, E. Lefever, and O. D. Clercq. “Benchmarking Zero-Shot Text Classification for Dutch”. In: Computational Linguistics in the Netherlands Journal 13 (2024), pp. 63–90. url: https://clinjournal.org/clinj/article /view/172. [20] M. Li, T. Shi, C. Ziems, M.-Y. Kan, N. Chen, Z. Liu, and D. Yang. “CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore, 2023. doi: 10.18653/v1/2023.emnlp-main.92. [21] J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen. “What Makes Good In-Context Examples for GPT-3?” In: Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. Dublin, Ireland and Online, 2022. doi: 10.18653/v1/2022.deelio-1.10. [22] L. Molle. “Inleiding - Een geschiedenis van mensen en (andere) dieren”. In: Tijdschrift voor Geschiedenis 125 (2012), pp. 464–475. doi: 10.5117/tvgesch2012.4.moll. [23] OpenAI. Introducing ChatGPT. https://openai.com/blog/chatgpt. 2022. [24] C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang. Is ChatGPT a General- Purpose Natural Language Processing Task Solver? arXiv preprint https://arxiv.org/abs/2 302.06476. 2023. doi: https://doi.org/10.48550/arXiv.2302.06476. [25] S. Rebora, M. Lehmann, A. Heumann, W. Ding, and G. Lauer. “Comparing ChatGPT to Human Raters and Sentiment Analysis Tools for German Children’s Literature”. In: Com- putational Humanities Research Conference (CHR). Paris, France, 2023. url: https://ceur- ws.org/Vol-3558/paper3340.pdf. [26] N. Reimers and I. Gurevych. “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”. In: Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing. Hong Kong, 2019, pp. 3982–3992. doi: https://doi.org/10.48550 /arXiv.1908.10084. 1064 [27] M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to Reweight Examples for Robust Deep Learning. arXiv preprint https://arxiv.org/abs/1803.09050. 2019. doi: https://doi.org/10 .48550/arXiv.1803.09050. [28] H. Ritvo. The Animal Estate: The English and Other Creatures in the Victorian Age. New ed. Cambridge, MA: Harvard University Press, 1989. [29] K. Thomas. Man and the Natural World: Changing Attitudes in England 1500-1800. New edition. London, UK: Penguin Books Ltd, 1991. [30] P. Törnberg. ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning. arXiv preprint https://arxiv.org/abs/2304.065 88. 2023. doi: https://doi.org/10.48550/arXiv.2304.06588. [31] S. Wang, Y. Liu, Y. Xu, C. Zhu, and M. Zeng. “Want To Reduce Labeling Cost? GPT-3 Can Help”. In: Findings of the Association for Computational Linguistics: EMNLP 2021. Punta Cana, Dominican Republic, 2021. doi: 10.18653/v1/2021.findings-emnlp.354. [32] R. Zhang, Y. Li, Y. Ma, M. Zhou, and L. Zou. “LLMaAA: Making Large Language Models as Active Annotators”. In: Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore, 2023. doi: 10.18653/v1/2023.findings-emnlp.872. [33] C. Ziems, W. Held, O. Shaikh, J. Chen, Z. Zhang, and D. Yang. Can Large Language Models Transform Computational Social Science? arXiv preprint https://arxiv.org/abs/2305.03514. 2024. doi: https://doi.org/10.48550/arXiv.2305.03514. 7. Appendices A. Annotation Guidelines A.1. Annotation Schema Category • Animals: A living thing that can move around to search for food. It usually has ways to see, hear, smell, taste, and feel the world around it. • Plants: A living thing that usually stays in one place. It creates its own food using sunlight, water, and air. Type • Organisms: A whole, living animal or plant. Think of it like one complete cat, or one whole oak tree. • Parts: A piece of an animal or plant. Things like a bird’s wing, a flower petal, or a bear’s claw. • Products: Something we get from a plant or animal that we use. Only first-order products count (i.e. it’s the first “product” that comes from the plant/animal, not a product of an earlier product). Examples are milk from a cow, honey from bees, or apples from a tree. 1065 • Collective: Something is collective if the word refers to a heterogenous multitude of plants/animals. Nature explicitly and inherently is a prominent part of, but it is not 100% clear what kinds of nature. If the collective might belong to both categories (you choose the best or least-wrong category). Examples are: weide, grastapijt, bos, woud, vee, kudde. Usage • Literal: When the word means exactly the animal, plant, part, or product itself. If you envision the text, you should see it. (”The bear ate a fish.”) • Symbolical: When the word is used as a symbol or metaphor, representing something else. If you envision the text, you should not see it. (”His heart was as cold as a snake.”) Pictures are symbolic. Nicknames are probably symbolical. • Petrified: if the plants/animals word is the name of something or someone. A.2. Technicalities General rule Textual context is always dominant in annotating. Discontinuous annotations Sometimes an annotation is discontinuous, meaning that there are words between the parts to be annotated. An example is: “esschen- en pijnhout”. Here, “esschen-” and “hout” should be annotated. This can be done by annotating both (here, meaning that you should also make a separate annotation for hout!) and then defining a relationship. Step-by-step guide: 1. Annotate “esschen-” and “pijnhout” and “hout”. 2. Select the first part (“esschen-“). 3. Right click the second part (“hout”). 4. Click “Link to” and select “discontinuous entity”. Part – Whole constructions Sometimes part-whole constructions occur, e.g. “de wortel van de brem”. Here, it is important to look at the parts that are separately referential (wortel and brem, here). If the text has “bremwortel” there is just one separate referential entity. Syntactic head Concerning compound words, we annotate based on the syntactic head. You can find the syntactic head by doing a reference test: to what pard of the compound can you refer? (“hazenpad”: not annotated; “padhaas”: annotated). In Dutch, the syntactic head is normally on the right side of the word. Co-references Co-references to entities are not tagged. (in “de wolf is blij, hij eet graag haas”, “hij” should not be annotated). Likewise, words that in a specific context refer to plants/animals should not be annotated, unless the plant/animal aspect is inherent. “Veulen” and “kalf” are names for young animals and should be annotated, however, “jong”, “wijfje”, “mannetje”, “wederhelft”, “lichaam”, are not. Adjectives As a general rule, adjectives are not annotated. There are a few exceptions: 1. If the adjective is part of the name of a plant/animal, it should be annotated (e.g. “blauwe” in “blauwe vinvis” and “kruipende” in “kruipende boterbloem”). 2. Sometimes a word looks like an adjective, but it is used as a substantive. In that case, annotate it. 1066 Foreign languages When plants/animals/nature-locations are in a non-Dutch language, they should still be annotated. There are two exceptions: 1. If the whole text is in a different language, it should not be annotated; 2. If the entities name is in a non-Latin script (e.g., Arabic, Greek, Hebrew), it shouldn’t be annotated. B. Used Prompts A ChatPromptTemplate is used. Therefore, ”chat messages” are provided between parentheses; inside the parentheses the ”sender” of the message and the message itself are divided with a comma. Parts in italics dependent on the texts that is annotated. Here, it is only indicated that these parts exist. B.1. Pre-filtering prompt ( System, You are a helpful assistant. You'll get a historical Dutch text. It's your task to tell whether (non-human) animals or plants are directly present in this text. You do this by reasoning step by step, and then end by completing: 'I deem the statement that literal animals are present in this text to be:' with True or False. I know you can do it! ), ( User, \textit{Text to pre-filter} ) B.2. Few-shot direct annotation prompt ( System, You are a highly intelligent and accurate nature domain information extraction system. I'll provide a small text, written in historical Dutch. Your task is to recognize and extract all entities related to plants or animals. If you have found anything that falls into that category, you should annotate it on three levels: 1. Category; 2. Type; 3. Usage. For Category, there are two possibilities: Plants and Animals. * Animals: A living thing that can move around to search for food. It usually has ways to see, hear, smell, taste, and feel the world around it. * Plants: A living thing that usually stays in one place. It creates its own 1067 food using sunlight, water, and air. For Type, there are four possibilities: Organisms, Parts, Products, Collective. * Organisms: A whole, living animal or plant. Think of it like one complete cat, or one whole oak tree. * Parts: A piece of an animal or plant. Things like a bird's wing, a flower petal, or a bear's claw. * Products: Something we get from a plant or animal that we use. Only first-order products count (i.e. it's the first 'product' that comes from the plant/animal, not a product of an earlier product). Examples are milk from a cow, honey from bees, or apples from a tree. * Collective: Something is collective if the word refers to a heterogeneous multitude of plants/animals. Nature explicitly and inherently is a prominent part of, but it is not 100\% clear what kinds of nature. If the collective might belong to both categories (you choose the best or least-wrong category). Examples are: weide, grastapijt, bos, woud, vee, kudde. For Usage there are three possibilities: Literal, Symbolical, Petrified. * Literal: When the word means exactly the animal, plant, part, or product itself. If you envision the text, you should see it. ('The bear ate a fish.') * Symbolical: When the word is used as a symbol or metaphor, representing something else. If you envision the text, you should not see it. ('His heart was as cold as a snake.') Pictures are symbolic. Nicknames are probably symbolical. * Petrified: if the plants/animals word is the name of something or someone. To summarize, you should detect all plant and animal related words and tag them according to this schema. So, for each found entity you annotate its category (Plant/Animal), its Type (Organisms/Parts/Products/Collective), and its Usage (Literal/Symbolical/Petrified). It is extremely important that you work precise. Therefore, you should explain step by step why you make a choice. Also extremely important: the annotation you do should be in the form of a list with dictionaries. You should also do an explanation, but your ultimate annotation should be in that format. So you should always have output like this: 1068 [{"span": span, "type": Category-Type-Usage}, ...] Very important: if you don't find any entities, your annotation should be an empty dictionary in a list: [{}] otherwise the postprocess script will get in trouble. Good luck, I count on you! ) ( System, The span must be exactly the same as in the original text, including white spaces. ) ( User, Here are some examples: \textit{ Example1, Example2, Example3, Example4, Example 5} Please now annotate the following input: Input: \textit{Text to annotate.} ) B.3. Zero-shot direct annotation prompt Same prompt as above, but without the examples. C. Model Settings OpenAI API parameters temperature = 1; top_p = 1; frequency_penalty = 0; pres- ence_penalty = 0; gpt-4-o version: gpt-4o-2024-05-13. gpt-3.5-turbo version: gpt-3.5-turbo-0125. ‘2023-03-15-preview’ GysBERT parameters archictecture: BertForTokenClassification; optimizer: Adam; learn- ing_rate: 2e-5. D. Held-out Test Set 1. Doch in kommerlyke tyden word dit kruid, een weinig geroost, door de menschen ten spyze gebruikt. 2. Al de gedroogde visch, die zich toen op het eiland bevond, werd daar van geheel zwart en onbruikbaar, ja in de twe naastvolgende jaren werden door die assche, of veeleer door 1069 de ’er mede vermengde scherpachtige rotsbrokjes of zand, gelyk boven by den brand op Jan Mayen eiland aangemerkt is, zo verre het 3. Als men het Varken, in ’t midden aan weeder zyden van de rugge-graad, doorgesne- den heeft, zo laat men ieder helft, even onder de schouwder nog eens doorsnyden in de breedte. 4. Het gerookt vleesch moet ook acht dagen in het zout leggen, en dan in zakken genaait in de rook gehangen worden, en moet drie of wel vier maanden rooken. 5. Zouten van Spek, Hammen en Ossen-Vleesch, hoe daar mede te handelen. 6. Dan legt men alles aldus in de kuip om in order te gebruiken: 1. de 6 klapstukken van de buik onder in, want ze konnen het langste duuren: 2. de staartstukken: 3. de schouwderbladeren: 4. de twee borststukken: 5. de twee beste ribben: 6. de vier an- dere ribben: de twee ongeschikte ribben die by de schouwders zitten: 7. de huspot zo men wil boven op; maar men moet zorg dragen dat de stukken wel vast in malkanderen sluiten, en de openingen moeten met zout gevuld worden, en wat zout ’er boven op, en eerst onder op den bodem gespreid; ook moet de kuip eerst schoon uitgebroeid en met kruidnagels gedroogt worden. 7. Dit alles te zaamen in een groote pan of styfsel-kom of hakkebord gedaan, en 6 tinne kommetjes met Osse-vleesch-nat of ander vleesch-nat, of warm water daar op gegooten en digt toegedekt en altemets eens omgeroert. en zo een nagt over, op de warme plaat laaten staan weken; en dan stopt men ze gelyk Leverbeulingen; dog om dat de gort sterk zwelt maar half vol, en dan zynze half vol als men ze plat duuwt: Als ze gestopt zyn laat men ze zeer zagt kooken dat het water maar even beweegt omtrent een half uurtje, en men prikt ze ondertussen met een doorntje om niet te barsten en uittekooken, en dan zyn ze heel goed. 4. 8. En schoon veele staande houden dat het eeten van dit vleesch geen quaad aan de men- schen doedt, zo zyn fatsoendelyke lieden nogtans beschroomd om het te gebruiken: om dit met zeekerheid te weeten zo kan men daar deeze proeve van neemen. 9. §. XXXI. De Koemelk word tot artzeny gebruikt. De Melk is de voornaamste artzeny der Yslanders, en word daarom ook, zodra zy van de koe koomt, door gene anderen, dan alleen kranken, genoten. 10. Die het beter willen maken, en ’er de middelen toe hebben, kopen een weinig zout, sny- den, als het ge-slagt dier noch onafgehakt hangt, op drie of vier plaatsen een diepe snede in het vleesch, en doen in iedere opening een kleine hand vol zout, zich verbeeldende, dat het dus zelf, zo veel nodig is, door het gantsche beest trekt, en het vleesch, wanneer ’er vervolgens wind en rook by koomt, zeer wel bewaard word Op de beide gezegde wyzen handelen de ingezetenen ook met het schapenvleesch, als zy het voor hun huisgezin slagten. 11. Zeeusche Pens en Hoofdvleesch, hoe men die maaken zal. 12. Ossen en Koeyen vallen niet groter dan het kleinst geestvee in Duitsland; hebben, gelyk bereids gezegt is, gene Hoornen, en genieten alleen het voorrecht, door de huis lieden in den winter mede onder ’t dak genomen en met het zo kommerlyk gewonnen hooy, of, by mangel van het zelve, met het gedroogd zeegewas Zeenestel spaarzaam gevoed te worden. 1070 13. Men stopt de beulingen maar half vol om dat die anders te ligt uitkooken of barsten; en men bind ze met een touwtje onder en boven toe, en dan wordenze op een schootel plat nedergelegt, tot dat ze gekookt worden: Voor al moet men niet vergeeten genoeg vet daar in te doen, want anders zyn de Leverbeulingen te droog. 14. weshalven de boeren ’er aldaar meer acht op geven. Dezen jagen alleen de Hamels in ’t gebergte; doch houden de Oyen zo veel by huis, als doenlyk is. 15. Als men zo veel moeiten niet doen wil om Rolpens en Hoofd-vleesch te maaken, zo snyd men de pens in stukken, en men kookt het met de kop tot dat alles gaar is, en dan legt men het vleesch met de pens door een, met wat zout en heele peper, in den azyn, in een keulse aarde pot, is heel goed om met appelen des winters gebakken te eeten. 17. 16. De Boter kaarnen de meesten voor en na zo hairig, als zy uit ongereinigde melk in een zamengenaaide schapenvacht gemolken is, en leggen dezelve dus op; weshalven een vreemdeling die Boter niet ligtelyk door de keel zoude konnen krygen. 17. Dan neemt men een groote vleesch keetel en men hangt ze vol regen water over het vuur, en als het water kookt doet men de beulingen daar in, dat die regt uit en niet op malkanderen leggen, daarom mag men niet meer als anderhalf douzyn beulingen te gelyk kooken; en ze moeten heel zagtjes kooken, omtrent een half uur lang. 18. Afhakken van ’t vleesch in de Slacht-tyd, en hoe men de stukken best en ten meesten voordeelen zal gebruiken, en hoe men verder met alles in de Slacht-tyd, moet handelen. 1. 19. Neemt by de 20 ponden, gehakt redelyk vet, varkens vleesch, anderhalf loot of twee loot nootemuscaten; twee loot nagelen; twee loot zwarte peeper, dit alles ter deegen fyn gestooten zynde, zo roert men het onder anderhalf vierendeel zout, en men kneed het door het gekapte Varkensvleesch heen; en men laat het zo een nacht met een schoone doek bedekt staan doortrekken. 20. Hunne vellen vallen in den winter, als zy het meeste en vastste hair hebben, het best; weshalven de Yslanders dezelve dan naarstig vangen, en wel, uit aangebore afschuuw van schietgeweer, met uitgezette netten of vangyzers, die gelyk een kleermakersschaar gevormt, en met een dood lam ten lokaas voorzien zyn. 21. geweld der uitbrekende en uitgezette lucht een groot gedeelte van den berg, ’t geen te zwaar was, om opgeligt te worden, op zyde en niet slegts een gantsche myl wegs langs het eiland tot aan het strand, maar zelfs noch een myl verr’ in zee voortgeschoven, en aldaar neder gezet wierd, alwaar het, onaangezien de diepte, in den beginne wel 60 vademen boven het water uitstak, en aldaar merendeels noch staat e. 22. Neemt voor het vleesch, het geen men daar in legt, het vleesch van de schouwder van een Os dat het malste is; of anders een van de platte billen. 23. Ja zy zyn het zelven, die gemeenlyk het begin der aardbranden veroorzaken. 24. §. XXXIV. Hebben geen Zwynen, maar wel Honden en Katten. 25. Doch wat de eigentlyke en natuurlyke oorzaak dezer zeldzaamheid zyn mag, is niet zeer ligt te beseffen w. 26. Reusel, hoe men die wel zal smelten. 27. Van harde of Coraalachtige Zeegewassen wist myn berichter te zeggen, dat enigen van dezelven op de gronden gevonden wierden; doch konde hen niet noemen of beschryven, nadien hy, volgens zyne eigen belydenis, ’er nooit naar gezien had. 1071 28. Dezen zyn de Snoriper op de lappische Alpen, die zich a steeds op het land houden, meer lopen dan vliegen, en mitsdien niet bezwaarlyk te vangen zyn. 29. Men moet zich verwonderen, wat zy konnen uitstaan; doch zy worden wel degelyk door de ongemakken verhard, nadien zy jaar uit jaar in in het open veld onder den bloten Hemel blyven, en ’s winters onder de sneeuw zowel, als ’s zomers, hun voeder zelven moeten zoeken, waar toe zy alleen de weldaad van de natuur genieten, dat zy met byzon- dere styve, lange en dikke hairen, allermeest tegen den wintertyd, bedekt zyn. 30. Vervol-gens bragt men het zieke volk aan land, ’t geen, ofschoon het, behalven enig Lep- elblad, niet als Zuring in warme Melk en een weinig Schapenvleesch nuttigde, nochtans velen binnen acht en de anderen binnen veertien dagen zo fris en gezond werden, dat zy huppelden en sprongen, en in minder dan vier weken na hun komst weder scheep gaan, zelven hun anker lichten, en die lange en bezwaarlyke reize voorts vrolyk voleinden konden. 31. Het vleesch snyd men eerst aan stukken als Ossekarbenaden; en dan snyd men het aan lange reepen omtrent een vinger dik en vierkant; men snyd het vet ook aan zulke langw- erpige stukken; en dan bestrooid men de pens met wat geprepareerd zout en kruit, gelyk ik boven gezegt heb. 32. Men neemt 3 loot bruine peper, en een halfvierendeel nagelen; dit te zaamen eerst fyn gestoten en in een aarde schootel gedaan, en een hand vol gedroogde Saly, die men op den haart wat te droogen legt en die klein gewreven is, en een hand vol of vier zout daar onder geroert, tot men denkt dat men genoeg zal hebben; want den een doet het wel wat hartiger dan den ander. 33. Men behoefd ’er geen Sukade nog Amandelen in te doen als men niet wil, en is evenwel goed maar zo lekker niet. 34. De Harsten laat men een dag of vyf in het zout leggen en men moet ze niet te groot laaten hakken, om dat ze anders te ongeschikt zyn, en ieder een doet dit naa de groote van zyn huisgezin, ook zyn die Harsten dus zeer goed om in den Oven gezet en gebraaden te worden. 35. Het vleesch om in de Kuip in te zouten, daar neemt men toe de zes klapstukken van de buyk, de twee staartstukken, de schouwderbladeren, de twee borststukken, de vier andere ribben, als men de twee beste ribben wil in de rook hangen, anders kan men ook de Paterstukken inzouten, en dan nog de twee ongeschikte ribben die by de schouders zitten; en men laat die stukken groot of klein hakken naa dat men het wil hebben en het huisgezin groot is. 36. Mitsdien ziet men zelden op Ysland andere, dan uitgebrande bergen, aan en om welke men bequaam de werkingen en overgebleven tekenen van een vorigen brand bespeuren kan. 37. Buiten dien tyd leggen de inwoonders, nadien de Vossen de schapen zeer schadelyk zyn, kraanogen (nuces vomicae) in honig geweekt, die zy, anders niets zoets te eten bekomende, zeer begerig inzwelgen. 38. Neemt 4 kop Gort schoon afgewasschen: 4 pond korenten die wel verlezen en schoon gewassen zyn: 8 loot gestoote kaneel: 1 loot gestoote nagelen: 3 loot gestoote notemus- caaten; 1/2 pond poeijer-zuiker: 1 pond gepelde amandelen in stukjes gesneden: 6 sukade 1072 schellen aan stukjes gesneden: Een hand vol zout: 10 pond of daar omtrent Osse-niervet aan dobbelsteentjes gesneden. 39. Het zoude gezwellen verwekken, en, als men ’er veel van eet, sterk openende zyn. 40. de Ravens verjagen; doch het Lam, vermits het, zyn voeder niet konnende zoeken, elendig omkomen moet, slagten, en het het zachte vel afstropen, ’t geen de peltery geeft, die in Denmarken en Holstein onder den naam van Schmaaskin of Schmaasken x verkogt en zeer veel door lieden van een middelbaar vermogen gedragen word. 41. Neemt een van de grootste Kalfskoppen, en reinigt die, en wascht ze vier of vyfmalen ter degen schoon af, en laatze een nacht in schoon regen water staan te trekken, dat ’er de slym en het bloedige wel schoon af is, en hangt de kop met schoon regen-water over het vuur; en doet ook in de keetel, de nek van het Varken, en de twee ooren met wat veel zwoort dat ’er genoeg is om het vleesch in het hakkebord van booven en onderen te bedekken; en als men te veel zwoort en niet genoeg vleesch heeft, zo doet men ’er wel een of twee van de vleesigste stukken van het varken by, en men laat het te zaamen een uur of drie kooken, na dat men het alvorens wel schoon geschuimt heeft, en het moet zeer gaar zyn tot dat het vleesch van de beenen af valt, en dan schept men het uit op een aarde vergiettest of doorslag. 42. Hunne manier, om het Rundvee te slagten, heeft ook iets byzonders, Zy kollen het niet voor den kop, menende, dat daar door het bloed in ’t vleesch stremt, en mitsdien niet lopen kan; maar steken het een dun penmes diep in den nek, waar door het ter aarde valt; als dan trekken zy de poten gezwind met strikken zamen, en openen de keel, op dat al het bloed zoude uitvlieten Het ingewand word door de Yslanders allereerst, zonder veel te reinigen, genuttigt, en het dier zelf afgehakt. 43. Neemt de Lever van het Varken en wascht die schoon, en laat die op een aarde schotel leggen; doet daar zo raauw de vellen en spieren met een mes ter degen schoon uit, en doet het in een schoon tobbetje. 44. Voor een geheele pens heeft men omtrent 20 pond vleesch noodig, behalven het vet dat men daar by gebruikt, dat nog omtrent 10 ponden is. 45. Laat dan een ketel of twee met regenwater kooken en laat het Koud worden; en als het koud is neemt dan schaars drie kommetjes van dat water tegen ruim een kommetje wyn azyn, en mengt dat te zaamen onder malkanderen zo veel tot dat de pens, als he daar over gegooten is, kan onderleggen, en zet ze dan zo open weg daar ze niet te vogtig staan, is heel goed om met appelen gebakken, of gestooft met wyn te eeten. 10. 46. Saucysen of Worst van Varkenvleesch, hoe men die maaken zal. 47. De stukken worden niet met zout gewreven, maar slegts twemaal door zeewater gehaalt, en dan in de lucht, op dat zy winddroog zouden worden, en vervolgens in hunne hutten over hunne haardsteden gehangen, om dezelve te roken, en te meer te doen drogen Dus behandelen zy hun geslagt half verrot en half stinkend vleesch, tot zy het voorts opeten. 48. Het vleesch om in de rook te hangen daar toe neemt men de Paterstukken, de twee andere platte billen; en de twee beste ribben; en de spieren, die achter tusschen de beenen van de ribben inzitten, moeten daar schoon uitgedaan worden, om dat daar door ligt verderf ontstaan kan. 1073 49. Dan doet men twee geraspte nootemuscaten, met wat gestoote foelie en met wat zout daar in, en men hakt het te zaamen onder een tot het redelyk klein, maar niet al te klein is. 50. Alsdan begeeft een harder zich met de afgerichte honden op een heuvel, en geeft met zyn hoorn een teken, waarop de honden zich verdelen, en de Schapen van alle kanten uit de klippen en wildernissen in een zekere omtuining of staketzel dryven, ’t geen vooraan wyd uitgezet is; doch, op dat zy niet zouden konnen ontvluchten, naar achter allengs enger word. E. Cost In humanities, costs are often an important consideration. For all strategies, there’s the cost of establishing annotation guidelines and making something of a test set. After that point: • Human annotation costs €0.7 per sentence; • Direct annotation costs €0.007 per sentence (for GPT-4o, directly via OpenAI), with zero- shotting being slightly cheaper due to omitting examples in the prompt; • Indirect annotation costs €4.50 to train the model, and nothing per sentence. It is important to emphasize that costs per strategy are likely to change over time since GenAI models are getting cheaper and that human annotations might differ significantly per country or institution. Also, multiple factors should be considered, such as available hardware and environmental effects. F. Online Resources Code and data used in this study can be found here: • Data: https://www.dbnl.org/letterkunde/pd/index.php, • Code and annotations GitHub Repository. 1074