Historical administrative corpora are 昀椀lled with jargon and formulaic expressions that were used consistently across many documents. Governmental decisions, notarial deeds and o昀케cial charters o昀琀en contain 昀椀xed expressions to ensure that the same legal aspects in di昀erent documents had the same interpretation. Such formulaic expressions can be used to identify speci昀椀c elements of a document. For instance, a deed has di昀erent formulas to indicate whether it concerns the sale of property or the transferal of rights. In this paper we explore formulas as a methodological devise to structure the text of an administrative corpus and make the information contained in it better accessible. We use a datadriven method to detect potential formulaic expressions in historical corpora, that can deal with spelling variation and change and recognition errors introduced in the digitisation process. We apply this exploratory technique on a corpus of almost 300,000 eighteenth-century resolutions of the States General of the Dutch Republic and 昀椀nd many formulaic expressions that capture relationships between the political actors involved and the decisions that were made. A 昀椀rst analysis suggests that many formulas can be used to add metadata to individual resolutions on various elements of the proposals and decisions that are part of each resolution.
The Resolutions of the States General of the Dutch Republic (1576-1796) is a digitised archive containing an estimated 1 million decisions made by the States General (SG) during their daily meetings. It is 昀椀lled with administrative jargon and formulaic expressions that were used consistently, tens of thousands of times across a 220 year period, in resolutions with a very 昀椀xed structure. These formulaic expressions were used to signal speci昀椀c elements in the text, so that anyone relying on the resolutions for their day-to-day work could easily 昀椀nd back requests, decisions and agreements by looking for these 昀椀xed phrasings, which also made sure that similar decisions and agreements had similar interpretations.
In this paper we explore formulas as a methodological device to structure the text of the archive and make the information contained in it better accessible. The secretaries of the meetings used a 昀椀xed structure and 昀椀xed expressions to signal the opening of a new resolution, that always started with a proposal submitted to the SG. Each resolution ends with a decision paragraph, which also starts with a formulaic expression, followed by the details of what was agreed upon and what should happen next. For instance, to signal the SG reached an agreement on what should be done in response to a proposition, they used the formula ‘Waer op gedelibereert zijnde, is goetgevonden ende verstaen, ...’ (ENO:n which has been agreed and understood .... A number of examples of this phrase are shown in Figur1e). This phrase recurs tens of thousands of times in the resolutions and signals that an agreement and decision were reached that are detailed in the following paragraph.
The formulas thus not only help us structure the material, but also to add metadata to the individual resolutions and make them better accessible for analysis.
Our experience in working with other historical collections prompted a set of questions: Are such formulaic expressions used in other administrative corpora? And what other domains and document genres contain formulaic expressions? From an information access perspective, is the textual repetition of an administrative corpus like the Resolutions di昀erent from textual repetition in corpora in other domains?
To get a better idea of the relevance of formulas, we 昀椀rst compare repetitive text characteristics in corpora of di昀erent domains, to establish whether administrative texts like the resolutions are of a di昀erent quality from other types of text. The second topic of this paper is the methodology we use for algorithmically identifying potential formulas in the resolutions, and a discussion of their use. In this paper we con昀椀ne ourselves to these points, but we realise that formulas have relevance for a number of wider humanities research questions. We discuss these at the end in Section6.
Our work touches on three strands of research: 1) formulaic language use, 2) text reuse detection and 3) dealing with variation-rich text.
The use of formulaic expressions is mostly studied in the 昀椀elds of linguistic4s7,[26] and
language learning 1[
We found two studies that investigated the use of formulas in text corpora. Karsdorp identi昀椀ed and classi昀椀ed formulaic opening and closing expressions of Dutch folk tales. Repetitive patterns in the 昀椀rst and last 昀椀ve words of folk tales are detected and are found to be predictive of the genre of a folk tale. That is, the opening formula o昀琀en signals that something is a joke, saga or fairy tale. In the resolutions, the opening formulas are similarly indicative of what kind of proposal (petition, report, declaration, etc.) is discussed in the resoluti3o7n].m[ anually annotated the use of formulaic expressions in seventeenth and eighteenth century Dutch letters and found that more experienced writers tend to use more 昀椀xed expressions. They suggest that this indicates that formulas are partly used to reduce cognitive e昀ort.
Text reuse detection has been studied extensively in the context of plagiarism detection. The
annual PAN competitions, starting in 2009, have been a main driver for developing algorithms
for plagiarism detection and text reuse detection29[
Most research on text reuse detection focuses on modern texts, o昀琀en digital born and using
modern language, which has rules for spelling and syntax. Detecting text reuse becomes more
complicated for long serial archives covering historic documents from an extensive historical
period in which the language used had no consistent spelling, spelling changed over time, and
the digitisation of those documents introduces text recognition erro4r5s, [
In addition to plagiarism detection, textual repetition has been studied extensively in the
context of text alignment, collation and comparison10[
One of the big challenges of text analysis on corpora of historical texts is that they are rich in spelling variation. Many historical languages had no standard spelling and changed in spelling over time. Moreover, many texts extracted from digitised documents contain text recognition errors. These issues together lead to possibly many di昀erent spellings of the same word or phrase.
This challenge can to some extent be addressed by normalising the spelling. This maps
spelling variants of words to a standard or ‘normal’ spelling of the word. VAR5,D126][ is
a lexicon-based technique that was originally developed for historical English but ported to
di昀erent historical languages. TICCL [
Since we are using the same corpus as 2[
We narrow down our object of research with a more precise but still pre-theoretical de昀椀nition of formulaic expressions and their context. A literature search has not given us any de昀椀nition of formulaic expressions beyond the notions olefxical bundles and idiomatic expressions in common language use 7[, 26]. Lexical bundles are o昀琀en noun phrases and examples of domain speci昀椀c terminology. We take a information theoretical perspective, and need a de昀椀nition that is applicable to di昀erent corpora and that helps to identify formulaic expressions from large amounts of text. Given the nature of the corpus of resolutions and the corpus-speci昀椀city of its formulaic expressions, we need a de昀椀nition that takes into account that formulas tend to be longer phrases, though not necessarily complete clausal units, that can incorporate and give context to variable elements like names of persons, organisations and locations or dates.
We de昀椀ne formulaic expression as a multi-word phrase (anextended lexical unit) that is reused o昀琀en across documents in a collection , with minimal word variation, but with potentially high variation in spelling.1 They may contain variable elements, spans consisting of e.g. entity names or dates. In the resolutions, a phrase might express that a certain type of proposal was submitted by a person, whose name is a variable element in the formula. As far as we can tell, this de昀椀nition captures the formulas found by Rutten and Wal as well. What constitutes a formulaic expression might di昀er across domains, genres or corpora. We will discuss this further at the end of this paper in Section6, but note here that this de昀椀nition does not yet give us proper criteria for deciding what is a formula and what is not. Therefore, our research design is exploratory, rather than descriptive or explanato4r,ypp[.91-92]. It serves to give us a better understanding of the phenomenon of formulaic expressions and how we can develop methods to study them, rather than to o昀er a precise description of how they are used or an explanation of how they emerge or evolve.
The study of formulaic language use has identi昀椀ed a number of reasons for speakers and authors to use formulaic expression4s2[]. Within the domain of legal and administrative texts, the most relevant are precision of communicated information (e.g. “Cleared for takeo昀” signals permission to enter a runway and commence takeo昀), and signalling the structure of discourse (e.g. “on the other hand” signals an opposition)1[3, p.46].
The formulaic expressions that are used over and over in many historical legal and administrative documents, serve as precise referents to the information that the document is to communicate. They prevent di昀erences in interpretation, but also structure the information of the document. For instance, in a corpus of proclamations, a standard opening phrase signals where each proclamation starts. Notarial deeds o昀琀en have template text to indicate the role of the actors involved in the deed and that the contract is a legally binding agreement between the actors. In this way, formulaic expressions let us detect this structure.
We compare the corpus of resolutions against a set of historical and modern text corpora from various domains, to get insight in how textually repetitive it is and whether that is related to the domain and genre of administrative and legal texts.
We use the following document collections to study the amount of textual repetition (T1a)b.le There are 昀椀ve corpora consisting of mostly administrative documents: • The Resolutions corpus contains 286,871 printed resolutions of the SG in the period 17051796, with a Character Error Rate (CER) of 3%. • The Notarial Deeds of the city archive of Amsterdam contain legal transactions. We expect deeds to have at least a few formulaic expressions stating the nature of the transaction, the parties involved, and that it has been signed o昀 by a notary. We estimate the CER to be in around 10%. 1The latter aspect is part of the de昀椀nition to account for historical variations in spelling of what is semantically or pragmatically the same phrase. • The collection oDfutch medieval charters contains manual transcriptions of handwritten charters, which we expect to contain mostly formal language with low CER. • The Mandate and Police books from the city state of Bern cover the same period (and more) and also contain administrative and legal documents where we expect formal language use, but in a di昀erent language (German) and with a higher rate of text recognition errors (CER of around 20%). The two types of books represent di昀erent administrative subgenres so we treat them as separate corpora.
We compare these against 昀椀ve corpora with mostly free-form text from various domains, including two with historic Dutch language and three corpora with modern Dutch: • The general missives of the Verenigde Oostindische Compagnie (VOC, Dutch East India Company) consist of long business correspondences between the o昀케ces of the VOC in Asia and Amsterdam, which we expect are less formal and more free-form than the resolutions. These are handwritten documents, for which we could not obtain accurate CER information, but we estimate it to be in the range of 15-20%. • The Dutch Newspaper corpus of the National Library of the Netherlands contains over 700,000 articles from dozens of Dutch language newspapers in the 18th century. This corpus was OCR’ed around 2006 and has a CER of 15-20%. The articles are free-form and cover many topics, so we expect low repetition. • Dutch Wikipedia consisting of articles that are in principle free-form, but occasionally, entire databases of e.g. sports clubs, television shows or plant species are algorithmically turned into a set of article stubs using a template article. Although later manual edits of a template-based article tend to transform the template text to more free-form prose, some template phrasings may remain. • Dutch novels, a set of 10,921 recently published Dutch novels (text extracted from epubs).
We expect these to be free-form with little repetition. • Book reviews is a set of 472,810 online Dutch book reviews9[, 21] from seven di昀erent reviewing platforms. Reviews are also free-form, although book reviews represent a very narrow domain with potentially many stock phrases, so we expect some form of repetition.
A Vocabulary Growth Curve [
The vocabulary growth curves are shown in Figu2refor frequencies of word n-grams with
Ā ∈ [
For single words, the modern Dutch corpora have higher curves, meaning they have relatively more terms that occur more frequently, than the historic corpora based on algorithmic text recognition. This is not surprising, given the spelling variation and recognition errors in historic corpora. Both phenomena increase the size of the vocabulary and thereby lead to more mass at to the hapax legomena. Novels and reviews have the highest curves. We speculate that novels tend to use mostly common vocabulary to be easy to read by a large audience, while book reviews are a speci昀椀c domain and genre, so use a relatively narrow vocabulary. Wikipedia contains articles about a huge range of topics, so it is understandable that it has a longer tail of hapax legomena. The medieval charters use a very limited vocabulary and have a lot of term repetition. As it is based on manual transcription, we assume the rate of recognition errors to be much lower than for the corpora based on OCR or HTR. The low/high CER distinction corresponds to a clear di昀erence, with all low CER corpora having much higher curves.
For word 3-grams, the resolutions and Bern police and mandate books have curves that fall o昀 very little, signalling that, although they have many single term hapax legomena, they have relatively many word 3-grams that occur more than once, compared to the other corpora. The curve for online book reviews overtakes the resolutions curve a昀琀er around 500,000 tokens, suggesting indeed that additional reviews introduce relatively few new 3-grams and that reviews are therefore relatively similar to each other. For word 5-grams, the top curves are those of the charters, Bern police and mandate books, the resolutions, notarial deeds and the book reviews. With the exception of the latter, they are all in the domains of legal and administrative texts.
These curves support our intuition that texts in the legal domain have relatively many repeated phrases, despite the spelling variation and character recognition mistakes.
How frequent should a phrase be to be considered a formulaic expression? There can be formulaic expressions that are borrowed from other domains or genres, that are used with low frequency in the collection in which formulaic expressions are analysed. We leave these out of the scope of this paper, as we want to focus on expressions that are frequent enough that they can be used as metadata that cover most of the resolutions. Identifying borrowed formulas requires knowledge or analysis of external resources.
We want to 昀椀nd word sequences that occur frequently. The simplest way would be to count the frequencies of word n-grams for some range oĀ,f similar to the word n-gram analysis of Section 3.2. However, the number of n-gram types grows rapidly Āasincreases, and the vast majority of these occur only once or twice. The corpus of resolutions has 735,919 distinct words, so the number of word 1-gram types is the same, but foĀr= 4, the number of n-gram types is 23,985,191. We exploit the fact that frequently occurring phrases can only consist of words that individually occur at least as frequently as the phrases themselves. That is, phrases that occur 100 times in the collection must consist of words that occur at least 100 times. To 椀昀nd phrases that occur at least ĂℎĄ ą Ą ÿă Ā= 100, we can exclude all candidate phrases that contain words with a corpus frequency below this phrase frequency threshold. Furthermore, the words within a phrase also co-occur with each other at least 100 times within a window that is equal to the word length of the phrase.
With these observations in mind, we developed a naive algorithm for identifying candidate formulaic expressions. In the pre-processing phrase, candidate phrases of 昀椀xed length are detected, a昀琀er which their contexts of preceding and following words are clustered and analysed to extend the partial formulas and identify their start and end boundaries. The parameterisation we arrive at is ad hoc and speci昀椀c this the corpus of resolutions, since we have no precise de昀椀nition yet of what makes a phrase formulaic in a particular context. The goal is to explore the corpus with a pre-theoretical notion of what we are looking for.
Concretely, the pre-processing phase consist of the following steps: 1. Tokenise each resolution into sentences and sentences into words. 2. Iterate over the corpus and count frequencies of individual words 3. Iterate over the corpus a second time, and replace each word with a variable to<kVeAnR> if it either has 1) a term frequencyĆ Ąÿ Ą ă( ) < ĂℎĄ ą Ą ÿă Ā, or 2) a co-occurrence frequency āā Ą ă(, ) < ĂℎĄ ą Ą ÿă Āwith at least one of its remaining neighbouring = 5 words ∈ ( − , + ) on either side. 4. Slide a 5-word window over each individual sentence, extract the 5-word window as a phrase if it contains no variable token, and count the frequency of each extracted phrase. The process is demonstrated in detail in AppendiAx.
In the extension phase, we reduce the set of candidate phrases to a set of formulas in two steps. First, we use fuzzy string matching to 昀椀nd clusters of candidate phrases that are spelling variations of each other. In the second step, we gather the contexts around each occurrence of a cluster of phrases, and count how o昀琀en the phrases are preceded and followed by the same sequence of words.
Many of the common word 5-grams are spelling variants of each other. We cluster them by indexing these word 5-grams as vectors of character 1-skip-2-grams. That is, we consider not only 2 adjacent characters, but also pairs of characters that are separated by another character.
Starting from the most frequent word 5-gram phrases, we query the index to 昀椀nd candidate variants using cosine similarity. Further details are provided in AppenBd. ix
Next, we build frequency lists of the 8 words preceding and following the 昀椀xed length phrase
and use transition probabilities to identify extensions that have a probability close to 1 of
preceding or following the phrase. This is inspired by probabilistic language models based on
Hidden Markov Models3[
Because the preceding (pre昀椀x) and following (post昀椀x) contexts can include clusters of spelling variants as well, we use the same fuzzy matching algorithm as used for clustering the phrases. We then split all 8-word contexts into sequences of words and calculate transition probabilities for pre昀椀x and post昀椀x contexts separately, starting from the 昀椀xed length phrase to the word immediately preceding or following it, and from that word to the next word, etc. Words that occur in multiple pre昀椀x or post昀椀x contexts thereby have a higher transition probability. Words that have a probability below 0.1 are considered to be not part of the formulaic expression and are replaced by <aVAR> token. Once all transition probabilities have been computed, we traverse the transition model starting from the 昀椀xed length phrase and consider preceding words part of the formulaic expression if the probability is above 0.9. Once the probability drops below 0.9 but is still above 0.1, we assume to have reached a common context of the formula that is not part of the formula itself. We repeat the same process for the post-phrase context, again, computing transition probabilities starting from the 昀椀xed phrase.
This process is described in more detail in AppendBi x.
We start with the 23,141 candidate phrases that we got from using a minimum frequency threshold of 100. Clustering variant phrases reduces this to 12,880 clusters. The most frequent phrase is considered the representative variant. Extending these phrases with preceding and following words with a transition probability above 0.9 results in 11,497 candidate formulas and 52,348 common extensions (with cumulative transition probabiliti0e.s1 ≤ ĂĆĄ Āą < 0.9).
The 10 most frequent phrases are shown in Tabl2e, together with their corpus frequencies and the formulas that were derived from analysing their contexts. The phrase ‘<START> ontfangen een missive van’ is the most frequent 昀椀xed-length phrase that is also the most frequent formula (there is no extension that has a transition probability above 0.9). A less frequent and partially overlapping phrase is ‘ontfangen een missive van den’, which in the extension step is extended to ‘<START> ontfangen een missive van den’, that is, the ‘<START>’ token is added to it. The 昀椀rst two formulas therefore also partially overlap, but the second is an extension of the 昀椀rst. The word ‘den’ (EN: the) is the most common continuation of the 昀椀rst formula, but other common continuations are names of persons, so the second formula is less ‘formulaic’ than the 昀椀rst. Phrases 3 and 4 lead to the exact same formula, as do phrases 5, 7, 8 and 9. Phrase 10 is also a partial overlap with these phrases, but because it includes the word ‘dat’ (tEhNat:) at the end — which is again a common follow up, but not the only one — it is not extended to the same formula.
We can further cluster these formulas by re-categorising the less frequent formulas that are extended or reduced versions of more frequent formulas as commexotnensions. If we perform this re-categorisation, the list of 11,497 formulas is reduced 7,153 formulas. Eyeballing the list of remaining formulas reveals there are still many spelling variants in the list. This shows that spelling variant is a challenging problem that needs to be analysed in more detail.
Almost 58% of the candidate formulas are longer than 5 words. Of course, our choice to start from candidate phrases of 5 words ensures no formulas shorter than 5 words are found, but the fact that more than half are extended shows that formulaic expressions in the resolutions are long syntactic units consisting of more than compound nouns and noun phrases.
The 10 formulas resulting from clustering the most frequent formulas are shown in T3a.ble The last column describes how we can use these formulas to identify meaningful elements in the running text, and how they help in classifying these elements. It is worth noticing that many formulas precede of follow a named entity, suggesting that formulas were frequently used to assert the relationship of that entity to the proposition or decision. Several formulas also contain verbs h(as been read, has been agreed and understood, to report back) that signal speci昀椀c actions. In future work, we will analyse the types of relationships and actions that these formulas express. waar op gedelibereert zynde is goetgevonden en verstaan which, upon deliberation, has been agreed and understood en van alles alhier ter vergaderinge rapport te doen <END> and to report back on everything, here in the meeting en andere haar hoogh mogende gedeputeerden tot de and other high and mighty deputies of de heeren gedeputeerden van de
the gentlemen deputies of the in handen van de heeren
in the hands of the gentlemen haar hoogh mogende resolutie van den resolution of her high and mighty of the aan het hof van sijne
at the court of his 10
Is ter vergaderinge gelesen de requeste van
Has been read in this meeting, the petition of Applying the approach to other corpora is elaborated in AppenEd.ix
Signal this is the start of a resolution and the start of the proposal paragraph, the proposal document type is missive start of the decision paragraph decision to start a investigative committee that will report back at a later date, signal that there is a future resolution related to this one name preceding this formula is a deputy, name following this formula is an institution, the deputy is a representative of the institution the name following this formula is a province or institution decision that the matter is handed to a committee to investigate, the name(s) following this formula are the members of this committee what follows is a date of a previous resolution, the previous resolution is related to this resolution the name preceding this formula is a representative of the court of the name following this formula this is the start of a resolution and the start of the proposal paragraph, the proposal document type is missive
The detection approach above is exploratory, as we have no precise de昀椀nition of what a formulaic expression is. We need clear criteria to determine if a phrase is formulaic before we can precisely de昀椀ne the task of formula detection and quantitatively evaluate methods designed to perform that task. For a proper evaluation of the detection method the entire corpus needs to be annotated with all formulas. We could reduce the problem by focusing on a small sample of resolutions and annotate anything that we think is a formula, but we would still need clear criteria to decide what is a formula and what is not. Another alternative is to use simulation and generate text and insert arti昀椀cially generated formulaic expressions to fully control the characteristics of formula, including their length, variation and frequency of occurrence.
Intuitively, a quantitative evaluation should consider precision and recall of detecting formulaic expressions, with two di昀erent measures of recall. One is the fraction of di昀erent types of formulaic expressions that are identi昀椀ed (type-based recall), and the other is the fraction of occurrences of formulaic expressions that are detected (token-based recall).
Formulas and their use have relevance to a number of both methodological and humanities research questions. Because formulaic expressions and their use for text structuring are understudied, for a large part we can only raise these research questions.
Our 昀椀ndings of the use of repetitive phrases in various corpora suggests that repetitive language use di昀ers strongly across domains and genres, with texts in administrative domains containing more repetitive phrasing. It is not clear whether this is true for all or just a speci昀椀c type of administrative texts. Our 昀椀ndings with detecting formulas suggest that this type of serial government sources may contain many formulaic expressions that can be exploited to structure texts and extract information. Further research has to shed light on the extent to which this also holds for other administrative sources and how the composition and diversity of collections relates to the statistical properties of formulas. A second type of research questions centres on the use of formulas. We do not know whether comparable administrative archives from di昀erent periods have the same rate of repetitive phrases and formulas. We observed that formulas emerged, changed and disappeared over time, but not at what rate and what the causes were. We do not know if there was an increase in adoption of formulaic expressions in the resolutions. There could have been changes in legal or administrative customs, more general language and cultural changes or perhaps even in昀氀uences of speci昀椀c scribes. The switch to the use of printing would lead us to assume that there was less variation in the formulas used, but it is too early to test this assumption. It is also an open question where the formulas originate, and whether formulas are reused across (administrative) domains. A formula borrowed from another domain might not be used o昀琀en in a corpus, in which case the method and de昀椀nition we developed in this paper do not su昀케ce. So further discussion is needed on what constitutes a formulaic expression, and what its generic and context-speci昀椀c elements are. As for the content of formulas, we can only make some remarks about those that occur in the resolutions. We have been able to spot a great number of formulas but it is hard to give a precise de昀椀nition of what a formula is, or to establish if we can discern constituent elements and if it makes sense to divide up formulas in these constituent parts. In this paper we describe a methodology in development, that iteratively gathers formulas from our corpus. This works well for identifying the most frequently used formulas and their variations. Further steps have to make clear what the optimal rate of formula detection is, how to categorise the di昀erent formulas, and if they are su昀케cient, to identify and localise di昀erent logical elements in the text and whether this works for all resolutions. A further point of discussion is how much of the methodology can be used for other corpora. We believe that the general methodology is suitable for comparable text corpora.
This work is part of the REPUBLIC project (2019-2024), a Research Infrastructure project funded by the Dutch Research Council (NWO, Grant number 175.217.024).
D. S. Carvalho, V. D. Tran, K. Van Tran, V. D. Lai, and M.-L. Nguyen. “Lexical to DiscourseLevel Corpus Modeling for Legal Question Answering”.TInen:th International Workshop on Juris-Informatics (JURISIN). 2016. [28] [42]
A. Wray. Formulaic language and the lexicon. Eric, 2002.
We demonstrate the processing steps to identify candidate partial formulas using the following example sentences:
Ontfangen een Missive van het Collegie ter Admiraliteyt in Zeelandt, geschreven te Middelburgh den negentienden deser loopende maandt, houdende, in gevolge en tot voldoeninge van haar Hoogh Mogende Resolutie van den vyfden der voorlede maandt, der zelver advis op het verzoeck van Burgermeesters en Scheepenen van het hooge en laage Zas van Gent.
With word tokenisation, we add<START> and <END> tokens so that for words at the start or end of a sentence, this boundary is included in its context. Certain phrases only appear at or near the start or end of a sentence, and adding boundary tokens allows us to keep track of such cases.
A昀琀er 昀椀ltering out low frequency words and words that do not meet the co-occurrence frequency threshold, we end up with the following list: [‘<START>’, ’ontfangen’, ’een’, ’missive’, ’van’, ’het’, ’collegie’, ’ter’, ’admiraliteyt’, ’in’, ‘<VAR>’, ’geschreven’, ’te’, ‘<VAR>’, ’den’, ‘<VAR>’, ’deser’, ’loopende’, ’maandt’, ’houdende’, ’in’, ‘gevolge’, ’en’, ’tot’, ’voldoeninge’, ’van’, ’haar’, ’hoogh’, ’mogende’, ’resolutie’, ’van’, ’den’,<‘VAR>’, ’der’, ’voorlede’, ’maandt’, ’der’, ’zelver’, ’advis’, ’op’, ’het’, ‘<VAR>’, ’van’, ‘<VAR>’, ’en’, ‘<VAR>’, ’van’, ’het’, ‘<VAR>’, ’en’, ‘<VAR>’, ‘<VAR>’, ’van’, ‘<VAR>’, ‘<END>’]
The example above shows that most of the words in the sentence occur frequently and cooccur with each other frequently. This results in a large number of candidate phrases. In the resulting sentence, we extract candidate phraseĂs( , +5by identifying sequences of word tokens (i.e. non-variable tokens) of length|Ă| = 5 and count their frequency over the entire corpus.
As little is known in advance about the relationship between frequencies of words, word co-occurrence and formulas, we have few meaningful clues for setting a minimum phrase frequency. The corpus of resolutions has 286,871 resolutions and over 58 million words, but we have no reliable estimates of the total number of formulaic expressions that are used and how o昀琀en they occur. In identifying the start of proposition paragraphs of resolution2s,2][ use a list of 32 formulas, most of which are between 5 and 10 words long. But we do not know if these are all the formulas used for openings of proposition and decision paragraph. We also do not know which other elements of a resolution are expressed in formulas.
To get some insight, we experimented with minimum frequencies of di昀erent orders of magnitude, ranging from 1 to 10,000. The results are shown in Tab4l,ewith for each frequency threshold, the size of the vocabulary (number of distinct words), the number of distinct cooccurring word-pairs within a 5 word window, the number of distinct 5-word phrases (phrase types) a昀琀er pre-processing, the total number of phrases (phrase tokens) and the number of phrases that meet the frequency threshold.
Frequency thresholds of 1 and 10 lead to large vocabularies and huge numbers of phrases. If they are to be made useful as metadata labels or boundary signals, we need to analyse each of them in their context manually, so these thresholds result in an unmanageable number of phrases. On the other extreme, a threshold of 10,000 leads to a very small vocabulary of 605 highly common words and only 140 highly frequent phrases. The most common phrase is <START> Ontfangen een Missive van’ (EN: <START> Received a missive of), which is also the start of the example sentence above, with a frequency of 138,682. Note that there are variant spellings of this phrase that are not included in the count. With such a high frequency, it is clear that this phrase is part of a 昀椀xed formula, but because we made 昀椀xed-length phrases, it is not clear whether this the entire formula.
It is quite likely that some formulas contain words with a frequency below 10,000, and because we know so little yet about the usage of formulas, it is better to be conservative and choose a lower threshold. For the rest of this paper, we pick a minimum frequency threshold ĂℎĄ ą Ą ÿă Ā = 100. This gives us a large set of 23,141 frequent phrases that are candidate formulaic expression or part of them.
The process of transforming the set of candidate phrases to a set of formulas has two main steps. First, we use fuzzy string matching to 昀椀nd clusters of candidate phrases that are spelling variations of each other. In the second step, we gather the contexts around each occurrence of a cluster of phrases, and count how o昀琀en the phrases are preceded and followed by the same sequence of words.
We index each word 5-gram phrase as a vector of character 1-skip-2-grams. That is, we consider not only 2 adjacent characters, but also pairs of characters that are separated by another character. The reason to include a single skip is that spelling variants o昀琀en have di昀erences in characters in the middle of a word, which results in multiple ngram mismatches and thus few matches when no skips are used.
Starting from the most frequent word 5-gram phrases, we query the index to 昀椀nd candidate variants using cosine similarity. Further details are provided in AppenBd. iWxe limit the candidate set to phrases that di昀er in length with the query phrase by at most 2 characters, based on the assumption that much longer or shorter phrases are unlikely to be spelling variants. We 椀昀lter the candidate phrases by checking that the words in each position 1..5 of the candidate phrase di昀er in length no more than 2 characters with their aligned words in the query phrase. This avoids matching a query phrase ‘op gedelibereert zynde is goetgevonden’ with its partial overlap ‘gedelibereert zynde is goetgevonden en’. The latter is typically the next 5-word phrase following the former, but because the query phrase has a short 昀椀rst word and the candidate phrase has a short last word, their ngram similarity is high. Using the length restriction on aligned words 昀椀lters out such erroneous matches.
In the extension step, we build frequency lists of the 8 words preceding and following the 昀椀xed
length phrase and use transition probabilities to identify extensions that have a probability
close to 1 of preceding or following the phrase. This is a similar approach to probabilistic
language modelling1[
This models the prediction of the next word only the current word. A natural extension is to model the prediction based on all words preceding it: ( | ) = ( , )
( ) ( | 1, 2, ..., ) = ( 1, 2, ..., , ) ( 1, 2, ..., ) (1) (2)
For extending phrases, we use this model, where the probability(ĂℎĄ) of a phrase ĂℎĄ consisting of words , ..., is ( , ..., ). The transition probability from a phrasĂeℎĄ to a (ĂℎĄ, ) ( ,ĂℎĄ) word is (ĂℎĄ) and from a word to ĂℎĄ is (ĂℎĄ) .
We split all 8-word contexts into sequences of words and calculate transition probabilities for pre昀椀x and post昀椀x contexts separately, starting from the 昀椀xed length phrase to the word immediately preceding or following it, and from that word to the next word, etc. Words that occur in multiple pre昀椀x or post昀椀x contexts thereby have a higher transition probability. Words that have a probability below 0.1 are considered to be not part of the formulaic expression and are replaced by a<VAR> token. Once all transition probabilities have been computed, we traverse the transition model starting from the 昀椀xed length phrase and consider preceding words part of the formulaic expression if the probability is above 0.9. Once the probability drops below 0.9 but is still above 0.1, we assume to have reached a common context of the formula that is not part of the formula itself. We repeat the same process for the post-phrase context, again, computing transition probabilities starting from the 昀椀xed phrase.
We extend the 昀椀xed length phrase with up to 8 words preceding it and up to 8 words following it, which means we can identify formulas o8f + 5 + 8 = 21 words in a single pass. If the full 8-word path preceding or following the phrase has a cumulative transition probability close to 1, we repeat this extension process with the extended phrase to identify the boundary of the formula.
An example of extending the phrase ’gevolge en tot voldoeninge van’ using transition probabilities is shown in Figure3. On the le昀琀 side, the pre昀椀x context is shown, with the word ’in’ being the only word directly preceding the partial phrase. This means that ’gevolge en tot voldoeninge van’ is not the full formulaic expression, but that ’in’ is also part of it. The expression ’in gevolge en tot voldoeninge van’ is also a syntactically more comprehensible phrase, meaning in consequence and ful昀椀lment of . There are multiple possible words preceding it. Two common ones are the verbs ’hebbende’ (EN:having), which precedes the phrase ’in gevolge en tot voldoeninge van’ in 38% of the occurrences of the phrase, and ’houdende’ (EmNa:intaining) which precedes the phrase in 47% of its occurrences. The remaining occurrences of the phrase are preceded by a variety of other words. Each of these two verbs have multiple possible pre昀椀xes, but are themselves not part of the formula according to the de昀椀nition above, because their transition cumulative transition probabiliti1e.0s ∗(0.47 = 0.47 and 1.0∗0.38 = 0.38 respectively) do not meet the threshold o0f.9.
On the right side, the post昀椀x context is shown, with two variants of continuations, neither of which is part of the formulaic expression itself. One common continuation is ’der selver resolutie’ and the other is ’haar hoog mogende resolutie’. Note that although neither path to the word ’resolutie’ has itself a cumulative transition probability above00. 3.99(∗ 0.99 ∗ 0.99 = 0.38 for the former and0.59 ∗ 0.99 ∗ 0.99 ∗ 0.96 = 0.56 for the latter), their combined probabilities add up to 0.94. This meets the 0.9 probability threshold, thereby being an example of a formulaic expression that can have a variable middle part. In some cases that variable part is ’der selver’ and in others it is ’haar hoogh mogende’.
The 昀椀nal formula is determined by extending the 昀椀xed length phrase with preceding and following words that have a cumulative probability of at least 0.9. In the case of the phrase ’gevolge en tot voldoeninge van’, we extend with the pre昀椀x ’in’ to ’in gevolge en tot voldoeninge van’
One of the big hurdles is spelling change. Finding phrases that are orthographically similar to each other is not hard, but phrases o昀琀en contain highly frequent, short function words that require only few character edits to transform one function word into another. This makes it di昀케cult to distinguish cases where two function words are variant spellings of each other from cases where they represent di昀erent words and therefore signal that these phrases have di昀erent meanings.
We experimented with both classic Word2Vec2[
One of the characteristic of resolutions that we can study with our list of formulas is the fraction of a resolutions text is made up of formulaic expressions, and which part is not. There are many very short resolutions based on a received missive (starting with the formula ‘Ontfangen een Missive van’) that merely states who wrote the missive, when and where they wrote it, but that do not contain any proposal or request that the SG had to make a decision on. Such resolutions end with the formula ‘Waar op geen resolutie is gevallen.’ (EONn: which no decision was made.
We therefore expect there to be a relationship between the length of a resolution and the amount of non-formulaic content. Longer resolution provide more detail of the proposition or of the decision or both. We assume that these details are only given when they are deemed relevant and necessary. The details vary across resolutions, therefore lead to less formulaic text.
The relationship between the length of resolutions and the fraction of words that are part of formulaic expressions is shown in Figur5e. There is a clear relationship: short resolutions tend to have a larger fraction of formulaic content. As resolutions get longer, a larger fraction of the words they contain are below one of the two frequency thresholds.
To check if this approach generalises to other corpora, we use the same detection process on the corpora of Notarial Deeds, Bern Manate books and t he Dutch Wikipedia.
Because of the high Character Error Rate (CE∼R0.2) and the much smaller size of the corpus (3.8 million words compared to 58 million of the Resolutions), we used word 4-grams instead of 5-grams. The procedure found a handful of candidates, two of which could be extended to • The most common phrase is ’Schultheiß und Rath der Statt Bern’, which occur 907 times in 505 di昀erent spellings. It refers to the head o昀케cial and the council of the city of Bern. • The second most frequently identi昀椀ed phrase is ’An alle Deütsch und Weltsche’, which occurs 559 times in 443 di昀erent spellings, and is extended to the formula ‘An alle Deütsch und Weltsche Herren Amtleüth’ (ENT:o all German and X gentlemen o昀케cials ). This is a formula to signal that the following statute pertains to the o昀케cials of both the French and German speaking parts of the city state Bern, and therefore signals the start of a statute.
The notarial deeds corpus also has a high CER of 15-20%. The most frequently found phrase are: • ‘als getuijgen hier overgestaen’ (ENs:tanding here as witnesses: this is part of a formulaic phrase in the opening paragraph of a notarial deed to indicate who act as witnesses in formalising the transaction. This is useful in identifying the starting paragraphs of deeds that are spread across pages. • ‘H Schaef N P’: the name one of the Amsterdam notaries, who is the o昀케cial responsible for ensuring the transaction is legal.
• ‘J de Winter N P’: the name of another Amsterdam notary.
These phrases have similar potential in identifying meaningful structural elements in the running text, such as where deeds start and end, and where certain elements of deeds are located within their text.