Historians base their inquiries on the sources that are available to them. However, not all sources that are relevant to the historian's inquiry may have survived the test of time. Consequently, the resulting data can be biased in unknown ways, possibly skewing analyses. This paper deals with the Dutch East India Company its digitized ledgers of contracts. We apply an unseen species model, a method from ecology, to estimate theactual number of unique seafarers contracted. We 昀椀nd that the lower bound of actual seafarers is much higher than what the remaining contracts indicate: at least, thirty-six percent of the seafarers is unknown. Moreover, we 昀椀nd that even in periods when few records survived, we can still credibly estimate a lower bound on the unique number of seafarers.
Historians can only rely on the archival records that have survived the test of time. That a
substantial share of historical records has not survived may be due to natural causes, such
as 昀椀res, decisions on the level of archival policy making, but also content production
biases [17]. For instance, whether or not particular sources were retained can depend on
socioeconomical factors [
As historians are working with data that is hampered by many possible types of bias, they
need to critically evaluate to what extent the remaining data is representative of the collection
or historical period from which its stems1[4]. Put di昀erently, historians need to re昀氀ect on how
transmitted archival records relate to the actual historical past. Especially now when historical
records are rapidly and continuously being digitized, we have to be even more cautious of
potential biases in archives. The speed with which we can analyze data combined with the
increased distance between the researcher and the source material makes it easier to overlook
how bias impacts the historical inferences we make [
Yet, at the same time, the fact that data is digitized also facilitates the use of statistical and
computational methods which help chart and possibly deal with the blind spots of the data.
Studies have already applied statistical methods to expose how bias can lead to overestimating
the impact of historical events [
In this paper, we focus on estimating the number of unique seafarers employed by the Dutch
East India Company (Verenigde Oost-Indisch Compagnie, VOC). The company kept detailed
personnel administration records (pay ledgers), which have been digitized in the VOCOP datase1t.
While the dataset, which has been used extensively by historians and social scientists in the
context of 昀椀nancial [
For the estimation, we draw on unseen species models 7[, 6], which aim to estimate the
number of unique species living in a given environment. Beyond ecology, these models have
been successfully applied to a wide array of cases, ranging from estimating the number of
classes of stone tools in archaeology 1[2], the number of bugs in so昀琀ware code [
This paper uses two di昀erent data sources: VOC: CAREERS (VOCCAR) and Dutch Asiatic Shipping in the 17th and 18th centuries (DAS).
VOCCAR is an enriched version of the VOCOP dataset, which contains digitized pay ledgers
of the VOC. [
We only focus on the records that have been clustered, resulting in 546,973 records ( ), of which 460,274 are unique seafarers ( ). Figure 1 provides an overview of the clustered records in the VOCCAR dataset. We see a sharp increase in records during the 1t7h century as well as noticeable gaps in the data for the 18th century.
DAS provides an overview of the number of ships that sailed out from the Dutch republi3c.
This data is almost complete, with only a few voyages missing from the data4. We learn from DAS that the VOC sailed out 4,352 times between 1633 and 1795. For about 91 percent of these voyages, DAS provides information on the number of people that boarded the vessel. However, because this data contains some noticeable outliers, we decided to calculate the mean voyagers per ship for periods of twenty-昀椀ve years rather than imputing merely the missing values. Next, for each period, we multiplied this mean by the number of voyages, including those that lack information on the number of voyagers. This provides us with an estimation of the total number of records: 952,1457. 3http://resources.huygens.knaw.nl/das/EnglishIntro 4The data quality is discussed here:http://resources.huygens.knaw.nl/retroboeken/das/ 5We use DAS to calculate the total number of records rather than VOCCAR because DAS is much more complete.
To estimate the number of unique seafarers of the VOC, we employ an unseen species model. The model was originally developed in ecology, where researchers are o昀琀en confronted with incomplete data as a result of undersampling. Due to such data incompleteness, it is possible that important statistics such as biodiversity are estimated to be much lower than they actually are. To combat such estimation biases, it is an important research question in ecology how the resulting di昀erence between the number of observed and the true number of unique species can be reliably estimated. A potential solution is given by the Chao1 estimator, developed by Anne Chao [7, 6].
The Chao1 estimator is a non-parametric unseen species model that estimates a universally valid lower bound on the number of unseen entities (e.g., seafarers; call that 0), based on entities that have been observed once or twice (call those 1 and 2). Theoretically, we can calculate the number of unseen entities by taking the product of the average relative frequency of unseen entities ( 0) and the number of unseen entities ( 0), divided by 0. However, 0 cannot be calculated directly. What we do know is that the average relative frequency of unseen entities ( 0) is probably lower than that of entities occurring once (1), i.e., 0 ≤ 1. It then follows that 000 must be at least equal to or greater than 010 (hence, 0 is a lower bound). That latter expression is computable and can be rewritten into the Chao1 estimator [cf8.]: ̂ = where refers to the observed number of unique entities, to the sum of their occurrences, and ̂ to the bias-corrected lower bound. It is important to note that when 0 ≈ 1, that is, when unseen entities have approximately the same average relative frequency as entities occurring once, Chao1 becomes an unbiased point estimator 8[].
The Chao1 estimator was developed assuming that samples are formed with replacement.
This means that during each sampling moment, the same individuals can be observed multiple
times. It also means that observations are independent of each other, and that the observation
of one individual does not a昀ect the observation of the next. In other words, the covariation
between successive observations is zero. Thus, sampling with replacement essentially assumes
an in昀椀nite population. For example, a snippet of text can be seen as a sample of an author’s
in昀椀nite stream of words. And if we apply the Chao1 estimator to this snippet, a lower bound
on the vocabulary of the author is also exactly what is estimated 1[
We can also think of the snippet as a sample of the 昀椀nite space of the snippet’s encompassing book. Treating the snippet as such would imply that the sample was createdwithout replacement. In such samples, observations are not independent, nor is the covariation between successive observations zero. Crucially, however, because of its assumption that samples are formed with replacement and are thus drawn from an in昀椀nite population, the Chao1 estimator does not estimate the number of unique words in the book encompassing the snippet. Thus, even though we know a given sample to come from a 昀椀nite population, Chao1 always treats it as coming from an in昀椀nite one.
The VOC records of this study should be conceptualized as samples created without replacement. There has been a 昀椀nite population of seafarers with the VOC of which the records show a sample without replacement.6 The problem, however, is that when we apply the Chao1 estimator to this sample, we do not obtain an estimate of the number of unique individuals in the total, 昀椀nite population, but rather that of a potential population of seafarers, which is not what we are a昀琀er. To estimate the number of unique individuals in the 昀椀nite population of employees of the VOC, we employ a modi昀椀ed Chao1 estimator developed by Chao and Lin for samples without replacement [9]. This modi昀椀ed estimator assumes we know the size of the total population, and thus know the ratio of the observed sample size to the total population: ̂wor = + , (2) (3) where = /( − 1)and = /(1 − .)Note that when approaches zero, Eq. 2 reduces to the standard Chao1 estimator in Eq.1. We refer to the modi昀椀ed estimator as Chao1 wor. Con昀椀dence intervals for Chao1wor can be computed based on the variance estimator 9[]: var( ̂wor) = 0̂ +
Based on the total number of records we derived from DAS (̂ = 952, 147 , see above), we calculate the sample fraction by dividing by ̂ . For the complete dataset ≈ 0.57. Note that ̂ ≠ ̂ , since individuals may have been shipped out multiple times.
At least thirty-six percent of the seafarers is unknown Based on the observed abundances, i.e. how many times each unique individual was s“ighted” in the data, and the sample fraction estimated from ̂ , we calculate with Chao1wor the lower bound on the number of actual unique individuals in the VOC population (̂ ) to be 716,818 (95%CI: 715,439 to 718,203). This number suggests that we should account for a survival rate of / ̂ ≈ 64%, or conversely, that of the original VOC population, at least 36% of the individuals is unknown. The loss of records impacted the number of unique individuals in the records To get a better understanding of the coverage of the data across time, we applied the same approach to successive periods of twenty-昀椀ve years. For each period, we calculated the mean number of voyagers on a journey and multiplied this with the total number of journeys in that period, thus estimating the actual number of seafarers (see Table 1). Figure 2 displays the observed number of unique seafarers against the estimated number ̂ over time. The gray overlay 6This 昀椀nite population can be constrained by many di昀erent things, about which we can now only speculate: the total number of ships, skills required to be enlisted, etc. Chao1wor for time spans of 25 years. represents the 95% con昀椀dence intervals of the estimates. The plot shows that, especially in the 17th century, the lack of data has led to a severe underestimation of the number of unique individuals (ranging from 45% in 1683–1708 to 96% in 1633–1658). With an average of≈ 23%, the gap between the observed and the actual number of unique individuals is smaller in the 18th century but still considerable. Overview of data for twenty-five-year periods. The last period only spans twelve years.
period just below 1 million. The same is true for the shorter periods of 25 years. Here too, Chao1 systematically produces impossible estimates. As we explained above, the Chao1 estimate might be considered thepotential rather than the actual number of seafarers that could have worked for the VOC. By contrast, the estimate of the Chao1wor estimator is compatible with the upper limit of ̂ and thus supports our approach of conceptualizing these sightings as samples without replacement. More generally, these results emphasize the need to understand the sampling process underlying the data, and to exercise caution when applying the estimators. When data are sampled without replacement, but the estimator assumes otherwise, Chao1 is not guaranteed to produce a lower bound, which puts any reliable interpretation of the results into question.
This short paper is the 昀椀rst to quantify the scale and extent of the assumed data loss and lack of representativeness of the archives of the the Dutch East India Company (the VOC). We applied the Chao1wor estimator to a database of employees of the VOC, and found that we can make credible predictions on the lower bound of the number of unique seafarers that have been employed by the company. Moreover, even when relatively small fractions of the records have survived, the estimates appear to be robust. For the entire archival period, we estimate that at least forty percent of unique seafarers are not recorded in the archives. Put di昀erently, the actual number of unique seafarers was much higher than the surviving records indicate. Moreover, the estimated increase in the number of unique seafarers in the 17th century is not as steep in actuality as the empirical, observed records suggest. Finally from the 1t8h century onward, the di昀erence between the observed and the actual number of unique seafarers is smaller but still considerable. More generally, our results show how unseen species models from ecology can be used to obtain a clearer perspective on the parts of historical archives that are lost.
This paper adds to a series of recent studies exploring the applicability of unseen species
models to cultural data [