Text recognition technologies increase access to global archives and make possible their computational study using techniques such as Named Entity Recognition (NER). In this paper, we present an approach to extracting a variety of named entities (NE) in unstructured historical datasets from open digital collections dealing with a space of informal British empire: the Persian Gulf region. The sources are largely concerned with people, places and tribes as well as economic and diplomatic transactions in the region. Since models in state-of-the-art NER systems function with limited tag sets and are generally trained on English-language media, they struggle to capture entities of interest to the historian and do not perform well with entities transliterated from other languages. We build custom spaCy-based NER models trained on domain-specific annotated datasets. We also extend the set of named entity labels provided by spaCy and focus on detecting entities of non-Western origin, particularly from Arabic and Farsi. We test and compare performance of the blank, pre-trained and merged spaCy-based models, suggesting further improvements. Our study makes an intervention into thinking beyond Western notions of the entity in digital historical research by creating more inclusive models using non-metropolitan corpora in English.
With the increase in digitization and transcription of historical archives, Named Entity
Recognition (NER) is often regarded as an important step in text processing, ensuring scaled access to
layers of information found in text, such as names of people, places or currencies [
In this paper we focus on early stage research into detecting NE in colonial historical
documents, in particular from samples of the ledgers of the British Political Residency (BPR) in the
Persian Gulf [
NER with Historical Collections. Despite the marginal position of historical datasets in
general NER research, the demand for systems for use in archival scenarios has steadily
increased [
Using spaCy for Custom NER with Historical Documents. NE processing eforts are
led by supervised machine learning and deep neural networks [
We used two large entity-rich text collections from the era of British informal empire in the region: excerpts from the BPR ledgers held in the India Ofice Records (IOR/R/15) as well as a 45000-word entry on Iraq from Lorimer’s Gazetteer. Since we were keen on testing NER both on diferent textual genres and formats, we used a combination of unstructured textual data types: corrected OCR of typeset materials, uncorrected OCR of typewritten documents and ground truth used to train Handwritten Text Recognition (HTR) models.
The Handwritten And Typewritten Bushire Political Residency Ledgers. The BPR
ledgers were not all originally written in English; instead, they are a combination of English
originals and translations made from incoming Arabic and Farsi documents. From image
lifes available at the Qatar Digital Library (QDL) we created a generic HTR model in
Transkribus for early handwritten volumes in the collection. Whereas the oficial letters exhibit
a formulaic consistency of style, evolving capitalization norms and tendencies to Anglicize
foreign names over the course of the nineteenth century further complicate the entity space
in the corpus. A companion dataset to the handwritten corpus was created by collecting
samples of OCR’d typewritten documents from twentieth-century BPR ledgers when the use of
typewriters had become standard (from 1932-1939 and 1940-1948). Preference for document
selection was based on a cursory glance at the number of entities on the letters and page layouts.
Lorimer’s Gazetteer . The last document used in our NER experiment is an excerpt of the
45000-word chapter on Iraq from Lorimer’s Gazetteer. The Gazetteer was a compendium
resulting from “gathering and processing information” about the Gulf region over the years 1904-1908
[
The documents described above are especially germane to the study of the modern history of
the Gulf states. For pre-oil Gulf studies, NER shows promise in identifying people for building
correspondence networks, as well as studying the movement of ideas, goods in the region. These
elements can be correlated, in turn, with the growth of colonial authority and the emergence of
Orientalist transliteration norms aimed at “de-Anglicization of indigenous names” [
Annotation workflow . For training, development and testing, we selected samples from the BPR ledgers and Lorimer’s Gazetteer totaling 6586 words. We allocated 82.5% of data for training and development and 17.5% for testing. The training data was further split into 60-line segments for streamlined group annotation. We used an open-source, full-stack web app (NER-Annotator) with custom entity tags and exported the annotated data in the requisite training format for spaCy3. To ensure that NER-annotator system requirements were met across machines and multiple annotators, the application was hosted on a virtual machine.
Tag selection and customization. Our system includes pre-trained models for detecting
pre-established named entities, such as “GPE” for “countries, cities, states,” or “QUANTITY” for
“measurements, as of weight or distance”. We maintained a selection of spaCy default tags both
to track their recognition and to mitigate the efects of catastrophic forgetting problem [
System architecture. Our system is based on spaCy and consists of several deep learning
models using the transition-based framework of [
Model Name
Description SM and LG: Pre-trained, en_core_web_sm (SM) and en_core_web_lg (LG) are English models prenon-fine-tuned trained on Ontonotes 5, Wordnet 3, etc. difering in size, with pre-trained components for tokenization, POS tagging, dependency parsing, NER, etc.
BLK-F: Fine-tuned Blank
No pre-trained components for tagging, parsing, NER. Converts tokens to vector embeddings (‘tok2vec’) and entity detecting components. Finetuned by its components trained with entity names, see section 4.
DEF-F: Pre-trained, fine- Based on spaCy’s SM model. It is fine-tuned by its components (except tuned with default ‘ner’ for ‘ner’) re-trained with our custom annotated data. UPD-F: Pre-trained fine- Based on spaCy’s SM model. Fine-tuned by its ‘ner’ component re-trained tuned with updated ‘ner’ with custom annotated data.
REP-F: Pre-trained, fine- Based on spaCy’s SM model. Fine-tuned by its ‘ner’ component replaced
tuned with replaced ‘ner’ with a new ‘ner’ component trained on custom annotated data.
DOB-F: Fine-tuned double- Double-ner models: DOB-BLK (BLK-F, SM), DOB-DEF (DEF-F, SM),
DOBner REP (REP-F, SM), DOB-UPD (UPD-F, SM).
custom NER pipeline, we apply transfer learning and use spaCy’s English pre-trained and
blank models to build, fine-tune, and evaluate our custom models. Moreover, we consider the
efects of the ‘catastrophic interference (forgetting)’ problem [
We apply non-fine-tuned, pre-trained SM and LG models on gold test data, discovering that the SM model works better. We then use SM as a baseline to build and fine-tune our custom models, named here DEF-F, UPD-F, REP-F and DOB-F. Also, we build and fine-tune a blank model, BLK-F, using it as another basis to build the abovementioned custom models. In the DEF-F model, we train only non-‘ner’ components to evaluate the potential of fine-tuning, parsing and tagging components. In the UPD-F model (in contrast to BLK-F), we update the ‘ner’ component of a pre-trained SM model taking advantage of other pre-trained non-‘ner’ components (tagging, parsing, etc.) for the additional layer of information and potential performance improvement. Moreover, the ‘ner’ component of a pre-trained model can already recognize some entities, which is beneficial if there are overlaps between its built-in entities and our entities in our training data. To mitigate efects of catastrophic forgetting, we build a REP-F model with a new ifne-tuned ‘ner’ component that overwrites the SM’s ‘ner’ component and retains other pretrained features. An alternative approach to this problem involves building a double-ner model with two separate ‘ner’ components. In this approach, ‘ner’ components of both fine-tuned models (DEF-F, BLK-F, REP-F, UPD-F) and non-fine-tuned SM co-exist in one pipeline.
Resampling training data. In our annotations, some entity examples and labels are
underrepresented, resulting in a class imbalance problem with potential negative efects on model
performance [
In this section we look at the performance of diferent models on our diferent datasets.
Performance of NER systems is traditionally evaluated in terms of f1-score, precision and recall
[
We evaluated spaCy’s pre-trained SM and LG models without customization. For all datasets,
SM outperformed LG possibly due to the overbearing efects of the latter’s embedded feature
vectors [
As for BLK-F, it performed best on samples from the Bushire dataset, which contains the majority of tagged entities and worst on Lorimer’s Gazetteer, a dataset relatively underrepresented in training. This tendency is expected since the blank model has neither a built-in ‘ner’ component nor entity classes and it depends highly on training. Similarly, the blank model has high entity f-scores for frequently tags (i.e. GPE, PERSON, VESSEL) and low scores for less common (i.e. QUANTITY, TRIBE) labels. Notably, the model can successfully classify non-Western entities “Baghdad” or “Basrah Wilayat” as GPE, but it does not detect entities of CARDINAL, NORP types from built-in models. A conclusion we can make from this is that depending on the end goal, sometimes such limitations can be mitigated by more training, resulting in a comparatively high total weighted f-score for the model.
Generally, for samples from datasets underrepresented in training, such as Lorimer’s Gazetteer,
custom pre-trained models with both replaced and updated ‘ner’ components show slightly, but
not significantly, better performance. These cases are the ‘catastrophic forgetting’ problem in
action. Although a nfie-tuned pre-trained model is originally based on SM with a built-in ‘ner’
component, after re-training or replacing this ‘ner’ component, entities that were previously
recognized (e.g. NORP) are no longer recognized or ‘forgotten’ [
Our experiments reveal that for detecting entities from datasets underrepresented in training,
combining a custom ‘ner’ component with spaCy’s built-in ‘ner’ component without direct
overwriting is a better option. Due to the limited training of this scenario, leveraging pre-trained
features benetfis performance. However, when applied to datasets well-represented in training,
double-ner models can cause model underfitting [
Evaluation scores in table 3 confirm this observation across all datasets. Notably, for onecomponent custom models where the ‘ner’ component is mostly assembled during training, precision is higher than recall. This implies that one-component custom models are most likely to classify any model-recognized entities correctly–particularly useful for our task of detecting entities of non-Western and historical origin. These models might not detect, however, all possible true entities. By contrast, double-ner models have higher recall than precision; they detect all possible true entities more successfully, since the double-ner models leverage already built-in ‘ner’ components with a more extensive set of tags. However, double-ner models are less likely to classify detected entities correctly (e.g. the label TITLE as WORK_OF_ART in ifg 4). For our specific project needs of detecting and correctly classifying custom entities, custom models perform generally better. However, our experiments indicate that potential exists to create a more generalized NER system that performs well across all entities, not only those built-in within spaCy’s list, but also custom entities. This process would require further ifne-tuning double-ner and enriching both training and testing datasets.
In this paper, we developed an extensive blueprint for a custom NER system that outperforms state-of-the-art built-in spaCy NER models in detecting and classifying NE of a non-Western linguistic origin. The system includes custom NER models based on a combination of i) built-in and ii) fine-tuned custom ‘ner’ components. For our initial goal of detecting non-Western, historical entities, fine-tuned, one-component models perform best. However, it is still possible to create a generalized custom NER system, extending the NER system to a larger and more diverse set of historical entities and documents. We plan to apply the NER models to a corpus of HTR-generated text, rather than the HTR ground truth used for the current project.
We plan to preprocess datasets to account for historical norms and HTR artefacts in view of identifying diachronic developments in the transliterated entities across the corpus. Moreover, the NER system trained on such a rich dataset of entities has the additional value of being applicable to other historical texts of similar style and origin.