Research in automatic map processing is largely focused on homogeneous corpora or even individual maps, leading to inflexible models. Based on two new corpora, the first one centered on maps of Paris and the second one gathering maps of cities from all over the world, we present a method for computing the figurative diversity of cartographic collections. In a second step, we discuss the actual opportunities for CNN-based semantic segmentation of historical city maps. Through several experiments, we analyze the impact of figurative and cultural diversity on the segmentation performance. Finally, we highlight the potential for large-scale and generic algorithms. Training data and code of the described algorithms are made open-source and published with this article.
representational flexibility [
Classical segmentation algorithms are based on the specific knowledge of a map collection.
Despite color appearing relatively late in map printing processes [
Beyond pure graphical marks, some researches focus on detecting morphological features,
such as lines [
More recently, CNNs opened up new perspectives for the resolution of semantic segmentation
problems [
On the other hand, [
In this research, we seek to better understand the impact of figuration and figurative diversity on the learning of neural networks, in the context of the semantic segmentation of historical city maps. Thus, our research questions are the following: 1) How to measure the figurative diversity in a map corpus? 2) How robust are CNNs when facing high figurative diversity?
To answer these research questions, we first present a method to operationalize the cartographic figuration and measure the figurative diversity of a map corpus. We demonstrate the significant variability of our data, in comparison with other map corpora commonly found in the literature. Then, we propose an efective processing pipeline involving pre-segmentation of the map frame and semantic segmentation of the map itself. We highlight the potentials and the limitations of neural networks for solving generic semantic segmentation problems. Finally, we conduct a set of experiments that allow to investigate the learning mechanisms deployed by neural networks and to explore the design space of map semantic segmentation. Specifically, we seek to qualify the impact of learning transfer on each class. We then aim to evaluate the perspectives for an educated corpus constitution, through examination of cross-cultural performance and confidence prediction. Finally, we challenge the importance of graphical cues, in comparison to non-graphical concepts.
To create an experimental context that challenges the genericity of the semantic segmentation, we contrast two corpora that present a diferent kind of variability. The first corpus gathers 330 maps of Paris from the collections of the Bibliothèque nationale de France (BnF) and the Bibliothèque historique de la Ville de Paris. Most of the maps were published between 1800 and 1950 in the Paris region, and their scale is generally comprised between 1:25’000 and 1:2’000. The second corpus gathers 256 maps of cities from all over the world, including numerous reference maps. They come from 32 diferent collections, the main ones being: the BnF, the Library of Congress, the Harvard Library, the David Rumsey Collection, the University of Bordeaux, the British Library, the Boston Public Library and the Institut Cartogràfic i Geològic de Catalunya. In total, the corpus represents 182 diferent cities in 90 countries. The distribution across the various regions is balanced, except for Oceania (8 maps only). The regions with the most maps are Eastern Europe & Central Asia (34), Western Europe (34), and East Asia (30). Conversely, the regions accounting for the fewest maps are Oceania, Subsaharian Africa (15), and Middle East (17). The urban form of each map in the World corpus was manually classified into three categories: regular (95 maps), irregular (68), or mixed (93). Most maps were published between 1720 and 1950. However, due to historical reasons, most non-Western publications occurred after 1800.
Both corpora present specific difficulties. For the Paris corpus (Fig. 1), one of the most complex issue to apprehend is the information cluttering or overlay. In particular, the superimposition of information concerning mobility, such as road or underground network (Fig. 1 A.2-3), but also on the water system, the catacombs, or more simply on administrative divisions (Fig. 1 B.3-4). For Paris, this intensive use of the city map as a tool for planning urban works was likely caused by the lack of a proper cadastre before the late 19th century. Low-contrast may also make the map images difficult to read (Fig. 1 B.3, C.4). For the World
E
Figure 2: Map samples from the World corpus.
the frame, i.e. to non-geographic content of the document; and the map content, i.e. all the
geographic content that do not belong to the road network. In the extended ontology, the
map content was subdivided into 3 additional classes: the blocks, the water, and the non-built,
which in fact includes all non-aquatic unbuilt land, except the road network, i.e. wasteland,
meadows, crops, forests, but also parks, or inner courtyards. The Tab. 1 summarizes the
distribution of classes in the diferent corpora and sets. The training patches are open-source
and freely available online [
As in [
To make cartographic figuration measurable, we extracted 3 sets of descriptors related to color,
texture, and orientation. The color and texture features are based on previous works by Uhl
et al. [
These visual features are computed, using 50x50 sub-patches, for each image of both corpora,
as well as for the USGS [
On the other hand, a 32-bins histogram is computed on the distribution of each feature.
As the distributions can be multimodal, due to repetitive homogeneous figuration, the modes
are extracted by smoothing the histogram with a Savitzky-Golay filter [
For the semantic segmentation, we are using a CNN with UNet architecture [
The second pretraining is done in a crossed way, Paris being pretrained on the World, and the World being pretrained on Paris, as described in the following subsection. The CNN is then trained successively during 150 epochs on the Paris (2+1 and 4+1) datasets and on the World (2+1 and 4+1) datasets. ResNet101 is used as encoder with LeakyReLU as activation.
Three metrics are used to quantify performance: intersection over union (IoU), precision, and recall. In a second step, the confusion matrices between the diferent classes are computed. They are normalized regarding the proportion of pixels belonging to the class, according to the ground-truth.
In a first experiment, the CNN with ResNet50 as encoder was trained during 100 epochs on the Paris 5-classes-dataset, respectively on the World 5-classes-dataset. After the first training, the network was re-trained on the Paris set, using the weights trained in the previous step on the World corpus as initialization. Reciprocally, the World set was re-trained by initializing this time the weights on the Paris corpus. 2.3.2. Analysis of cross-cultural performance and dataset design To estimate the bias of the validation set, and to investigate cross-cultural performance, an 8fold cross validation was performed on the World 4+1 dataset. Each time, the CNN was trained during 100 epochs as described in the subsection on the performance of semantic segmentation.
In this experiment, we attempted to create an estimator of confidence at the patch scale. First, a 10-fold cross validation was performed on the Paris 3-classes-dataset in order to estimate the error on each patch of the training set. A simple ResNet50 encoder was used and trained during 60 epochs each time. The output predictions from the kth-fold are compared with the ground truth, and an accuracy map, in which the pixels take the value 1 if the prediction is correct and 0 if the prediction is wrong, is created.
A second network, identical to the first one, is then trained on 300 pairs of images and accuracy maps, and validated on another 30 pairs. Instead of segmenting, the aim of this network is to predict the accuracy map corresponding to the input image. The output of the CNN prediction is classified using a global threshold, which is set to meet the mean accuracy of the training set. The confidence index is defined as the patch accuracy, and the reference as the accuracy previously measured by k-fold cross validation. In order to evaluate the confidence prediction performance, an 8-fold cross validation was performed on the World 4+1 dataset. Each time, the CNN was trained during 100 epochs with the same parameters as in 2.2.
GRAY
BINARY
TEXTURELESS BIN.
This experiment aims to determine what role color, texture, and morphology take in the
CNN performance, in contrast with non-graphical concepts. For this purpose, the images of
the training and validation sets are subjected to 4 diferent treatments (Fig. 4): reference,
gray, binary, and textureless binary. For the reference, the images are not modified in any
way. The second treatment is the gray treatment. In the latter, the RGB color channels of the
image are transformed into a grayscale. For the third treatment, the images are transformed
into a grayscale, then binarized [
The median overall κ was computed on each dataset (Tab. 2). The result is very close for the two studied corpora, while the Napoleonic cadaster and the ICDAR dataset are already further away. The USGS map is in a diferent order of magnitude.
Fig. 5 and Tab. 3 are the aggregated representations of the correlations of the figurative features between and within both corpora. The frame class seems to be represented very similarly in both corpora (ρ = 0.96, Fig. 5), while the non-built class is the most distant (ρ = 0.81, Fig. 5). In the Paris corpus, the blocks class seems to stand out clearly from the other classes (ρ¯Blocks = 0.775, Tab. 3, and Fig. 5), while in the World corpus, it is rather the road network that stands out (ρ¯RoadNetwork = 0.888, Tab. 3, and Fig. 5). In general, however, all classes are more distinct in the Paris corpus (ρ¯Mean = 0.842 for Paris, ρ¯Mean = 0.917 for the World, Tab. 3). k oadR treowN e m a r F s i r a P n o i t a l e r r o c s s a l c r e t n I s i r a P d l r o W n o i t a l e r r o c s s a l c s u p r o c r e t n I d l r o W n o i t a l e r r o c s s a l c r e t n I Frame Blocks
Water Non-built
The results of the third experiment are summarized in Tab. 4 and Fig. 6. Some example prediction outputs can be observed in Fig. 7. The clearest finding is the consistent drop in performance, when increasing the number of classes from 2+1 to 4+1. The mean IoU (mIoU) on the 4+1 classes problems is not sufficient for reliable map segmentation. However, the mIoU is good on both World and Paris 2+1 corpora. The drop is more noticeable for precision than recall. The second clear diference occurs between the Paris and the World corpora, the first performing better. Again, the disparity is mostly due to a low recall. It is worth noticing that the top-50% of the World corpus performs very similarly to Paris average sample.
For the Parisian corpus, most confusion occurs as non-built is predicted as blocks, or to a lesser extent, as blocks are predicted as non-built, or as non-built is predicted as blocks. As it is also noticeable in Tab. 4, non-built is by far the class performing worse in the 4+1-classes problem. However, the water is the class sufering from the lowest precision score. In the 2+1classes problem, the blocks are sometimes classified as road network, which heavily impacts the precision of the road network class. Metric IoU Precision Recall n o i t c i d e r P t li u b n o N r e t a W k oadR treowN AVERAGE Paris 2+1
AVERAGE World 2+1
TOP50% World 2+1
For the World corpus, water on the contrary benefits from a relatively high precision but a very low recall. It is heavily confused with non-built and blocks. Non-built is still sometimes predicted as blocks, but for this dataset, the contrary is more frequent. Both blocks and road network classes sufer from a low precision.
As one can see in Fig. 8, the performance in class segmentation shows significant disparities between both datasets. If, in general, the Parisian corpus achieves much higher performances, the World corpus seems to be better at recognizing the non-built lands. The transfer learning is quite successful for the World corpus, when pretrained on Paris. The water (pvalue = 0.0063 < 0.05) in particular seems to be better recognized. The transfer learning from the World corpus to Paris is a bit less successful. However, the water class also demonstrates a significant improvement (0.0076). Overall, the improvement is trending for the World corpus, when pretrained on Paris (0.062).
Frame
Road network World wrt. Paris
Paris pretrained on World wrt. Paris
World pretrained on Paris wrt. World 3.3.2. Analysis of cross-cultural performances and dataset design In total, the mIoU over the 8 experiments is 0.6112, which is noticeably better than the 0.5595 score obtained in the previous section for the same 4+1 World set. That means that the average performance is slightly better than the performance observed on the validation set. The mIoU can also be computed on each patch separately, which corresponds to an average of 0.5424. Fig. 9, shows a few examples of the high disparity between the top-50% and the bottom-50% of the World corpus, which was already noticed in the results of the previous experiment.
Maps that have been published by a Western country score 0.5608, while other maps score 0.4911. The region of the city represented is also impactful, with Subsaharian (0.6322) and North African (0.6280) cities scoring best, followed closely by Eastern Europe and Central Asia (0.5931), South America (0.5913), Western Europe (0.5891), and North America (0.5711). At the end of the line are the South Asian cities (0.3985). In the middle, one would find the Middle East (0.5048), East Asia (0.4896), Oceania (0.4685), and Central America (0.4635). The urban form also has a clear impact, as cities with a more regular (0.5453) or mixed (0.5535) urban form score better than cities with an irregular (0.5088) urban form. This performance drop is especially noticeable on the blocks class for regular (0.3548), mixed (0.3027), and irregular (0.2179) urban form.
The average mIoU over the 10-folds is 0.6993. For this third setting, the correlation between the obtained confidence index and the reference is 0.571 ( pvalue = 1.2 ∗ 10−3) on the validation set, which represents an intermediate to high dependency. SUCCESSFUL
SUCCESSFUL
UNSUCCESSFUL E G A M I
The removal of color had noticeably almost no impact on performance. The median loss is only 1.3% with regard to the reference mIoU. The binarization of the values resulted in a drop of 7.2%. Finally, the disappearance of colors and textures led to a 10.4% decrease in performance. This experiment thus shows that even when most graphical cues are removed, most of the performance is conserved, and therefore that neural networks may also heavily rely on more abstract reasonings for image segmentation.
As measured through the operationalization of the figuration, both corpora, Paris and the World, present a much greater figurative diversity than the other datasets used in the literature (Tab. 2). This is good news and validates the interest of the studied datasets. The USGS is massively less diverse than the other datasets, which is logical since it is a digital-born map. The representation of the diferent elements is therefore perfectly codified and reproduced. The ICDAR21 corpus also shows relatively little diversity. It is composed of plates published in diferent years, but is still based on a single printed collection, and thus on a unified grammar. The Napoleonic cadaster for its part is famous for the high level of formalization of cartographic grammar. However, its manual execution explains a certain residual diversity. Finally, the two studied corpora show a similar level of figurative diversity, although Paris stems from a single cultural pool. This is consistent with the samples taken from this corpus, which show a great variety of grammars, a high density of information, and above all a high level of technicality, which also allows for an important figurative diversity.
Regarding the performances of semantic segmentation, while the Paris corpus shows excellent results which already indicate a potential for production (mIoU 0.8905, Tab. 4), the results of the World corpus seem globally still perfectible (mIoU 0.8055). However, the best half (top50%) of the World corpus presents performances very similar to the Paris corpus and may thus also present an important automation potential (see also Fig. 9). The question therefore lies in identifying this outperforming half in the large map collections, to open automatic vectorization perspectives. As we demonstrated in the experiment on confidence prediction, the estimation of a confidence index is possible and could probably solve this problem by identifying promising maps beforehand. However, we consider that further research is still needed to maximize the reliability of such an index and that other paradigms should be explored.
The identification of the most promising maps can also be based on the results of crosscultural validation. Indeed, we observed that maps with regular urban forms, like most colonial cities, or mixed forms, like most European capitals, are better segmented on average. This is consistent with the results on the importance of non-graphical cues for learning, which emphasize the crucial importance of non-visual features, such as morphology, topology, and semantic hierarchy, in CNN performance. In general, we notice that areas urbanized more recently, such as Africa, perform well. This is consistent with the findings on urban form. The poorer performance of South and East Asian cities can be explained on the one hand by atypical visual elements, from a Western point of view, by the irregularity of Indian and Islamic urban forms, as well as, in some cases, by the poor state of preservation of the documents. The little performance of the Oceanian maps could be explained by the relatively low representation of the region (around 3%) in the sample. The cross-cultural validation also highlights the greater ease of segmenting maps published in the West, although, as mentioned earlier, nonWestern maps are on average more recent. These cultural biases tend to argue for a diferential treatment of non-Western maps (e.g. Fig. 2 A.3-4, C.1-2).
The detailed analysis of the results of semantic segmentation brings many additional and interesting insights. First, we notice the particular case of the water class. This class was the least represented in the training corpora (Tab. 1), in particular for Paris, where it represents less than 3% of the surface. The proportion lies close to the road network for the World (12%). However, that class difers little from the other classes, figuratively (r = 0.94, Tab. 3). This explains its poor final performance ( IoUP aris = 0.47, IoUW orld = 0.33, Tab. 4). The performance is higher for Paris. However, water seems to be mostly recognized by elimination of other classes, which is expressed in poor precision (0.49), compared to recall (0.88). Moreover, the water class is vulnerable to overfitting on the blue color, as demonstrated by the particular case of the blueprint (Fig. 9), while the results on the importance of graphical cues for learning show on the contrary that most of the performance relies on more abstract features. These elements point to an imbalance of the water class in the original sample. This imbalance is not specific to our corpus, since it is the result of geographical constraints. However, our research provides a first solution to this limitation. Indeed, the experiment on the impact of learning transfer on each class demonstrated that the water class could benefit from a significant transfer (Fig. 8, +18% for Paris, +36% for the World). The constitution of large and diversified corpora intended for pretraining therefore seems to be justified by this example.
The second noteworthy result is that the non-built class scores much better for the World (0.6528), compared to Paris (0.4235), even though the World generally scores lower. For the World, however, the figuration of this class is not really outstanding (in average rpearson = 0.924). In reality, this class benefits (and sufers) from a catch-all efect. Indeed, a third (0.33) of the water surfaces and nearly a quarter (0.23) of the blocks surfaces are wrongly classified as non-built, while the non-built class itself does not confuse these classes (0.03, respectively 0.12). This catch-all efect results in a relatively high recall, compared to Paris, while the impact on precision remains little, as this class represents a large area (0.359) of the dataset. As it can be seen in Fig. 6, this catch-all non-built class is the main reason for the underperformance of the World dataset. Solutions to this problem might include separating this class into two smaller and figuratively more specific classes.
For Paris, this same non-built class obtains lower results, though quite close to those obtained for water, for example. This class is less of a catch-all, as the balance between confusing and being confused is shifted. In particular, the non-built is abundantly confused with blocks (0.33), even though the figuration between those two is the least correlated (0.7). Conversely, blocks are also sometimes confused with the non-built (0.17). This confusion is by far the most important reason for this poor performance. Several explanations are possible. Notably, in Paris, this class mainly describes parks, as well as courtyards. Compared to the World, the topology of the non-built class is less outstanding. Some parks adopt forms that are quite similar to building blocks, especially in Paris. Moreover, the spatial juxtaposition of inner courtyards and buildings promotes confusion, especially in this direction, since an unrecognized inner courtyard will be classified as a non-built confused with blocks. The Parisian corpus also sufers from poor learning transfer from the World, which may be caused by a relatively large figuration distance between Paris and the World for this class ( rpearson = 0.81). This seems obvious, considering that the elements represented in one and the other corpus are also semantically quite diferent, as explained.
The arguments discussed above do not pretend to be exhaustive because neural networks have many as yet unknown springs. However, they do shed some light on a few points. We consider that the framework of the maps is ideal for this discussion on the ability of neural networks to combine figurative and topological cues and to open up avenues of understanding on their performances. Indeed, maps have a ”textbook” figuration, with basic textures, such as hatching, some colors, and partly geometric morphology, squares, rectangles, trapezoids. They are therefore easier to characterize from this point of view. These conditions are met in very few fields of application of deep learning. In addition, for semantic segmentation, the ability to visualize the results and understand the errors is particularly useful for interpretation. This field therefore brings together all the elements that can help to better understand the performance of neural networks.
Developing a generic pipeline for processing historical maps will be an important milestone for massively extracting information from this rich family of cultural heritage documents. In this research, we made progress in understanding the figurative variance of cartographic corpora, and established a congruent metric to measure the figurative diversity of a collection based on the acuity of the multimodal distribution of graphical features. Through several experiments, we have shown that neural networks are extremely robust in the face of figurative diversity, even if some map grammars seem more difficult to segment. This high performance is mostly due to the fact that neural networks can integrate highly abstract reasoning, such as morphology, topology, and semantic hierarchy, to supplement figurative features, without excluding the latter. This work, and its conclusions, paves the way for the generic segmentation of historical maps, highlighting the weaknesses of learning processes and outlining potential levers for action.
The authors declare to have no conflicts of interest. We would like to thank our former collaborators, Raphaël Barman and Nils Hamel, for their support on this project. [37] S. Muhs, H. Herold, G. Meinel, D. Burghardt, and O. Kretschmer. “Automatic delineation of built-up area at urban block level from topographic maps”. In: Computers, Environment and Urban Systems 58 (2016), pp. 71–84. doi: 10.1016/j.compenvurbsys. 2016.04.001.