<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Towards a Network of Sixteenth Century Book Illustrations</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Germaine Götzelmann</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Karlsruhe Institute of Technology (KIT) Karlsruhe</institution>
          ,
          <country country="DE">Germany</country>
        </aff>
      </contrib-group>
      <fpage>38</fpage>
      <lpage>59</lpage>
      <abstract>
        <p>The following article investigates the reuse of book illustrations in German printed books of the sixteenth century. We utilize a workflow involving data and metadata retrieval, illustration segmentation, and reverse image search to detect identical and nearly identical illustrations in digitized books. The results of these steps are modeled as graph data, which enables us to filter, assess, and categorize them with the aid of graph measures and graph visualization. Drawing on a data sample of books published in Frankfurt am Main, we demonstrate how illustration reuse links the books in question in a complex and interconnected network, which can in turn be analysed and exploited to identify the specific patterns and characteristic features of this Early Modern cultural practice.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>block books formed part of a fixed page layout that combined text and images
in a permanent arrangement. With the advent of letterpress printing,
illustrations could be placed in multiple positions and on multiple pages within
the same publication. They could also easily be transferred to other books,
which in many cases altered their context (and meaning) completely. While
individual cases of re-purposed illustrations have been studied extensively,
the wider phenomenon of illustration reuse in the sixteenth century remains
elusive. This article seeks to showcase how a graph-based perspective can be
employed in a data-driven approach to obtain a much-needed overview of
this practice.</p>
      <p>If we consider the research topic from the perspective of graphs or
networks, we can untangle the following layers of networks:
1. Network(s) of printers and publishers
2. Network(s) of printing materials (wood blocks)
3. Network(s) of books
4. Network(s) of woodcut illustrations</p>
      <p>Network 1 is a social, economic, and legal network based mainly on
printers’ guilds and permissions for setting up a print shop in a city. Printers and
publishers were connected by family ties through marriage and inheritance,
but also via joint ventures and the purchase of printing materials from other
printers, occasionally in the form of bankruptcy assets.</p>
      <p>
        Printing blocks were valuable items that were sometimes bought in large
quantities and varieties (i.e. by Christian Egenolf in the aftermath of
Heinrich Steiner’s ban
        <xref ref-type="bibr" rid="ref3">kruptcy in Augsburg (Künast, 2013</xref>
        )). Printers focusing
on the production of illustrated/ornamental books had a vested interest in
assembling a large pool of printing blocks for reuse in various contexts. This
circulation of printing materials constitutes network 2.
      </p>
      <p>Books produced by a network of printers and publishers are interlinked
in multiple ways, including re-prints (authorized or unauthorized),
intertextuality, and compilation (network 3). As far as book illustrations are
concerned, these multifarious entanglements engendered phenomena such as
repeated use of the same printing block, recutting of wood blocks, and
adaptation of certain elements and motifs from the said illustrations. In this
article, we focus specifically on the reuse and recutting of wood block
illustrations (network 4).</p>
      <p>
        All of these networks are of potential interest to scholars working in the
historical disciplines as well as in the fields of book and literary studies, as
they can be linked to questions about the social, political, and economic
conditions under which illustrated books were created and produced in a newly
emerging context of printing, anonymous readers, book fairs, and ‘mass
production.’ Typically, knowledge about such networks is either the result of
tedious and time-consuming individual research and thereby limited in scope
(e.g. to one specific topic, type of illustration or genre, printer or
blockcutter, etc.),1 or it is based on quantitative analysis and therefore concerned
with the ‘big picture.’ This article assumes the latter approach. In order to
analyse illustration reuse in a quantitative way, we have established a
worklfow that exploits metadata and image data for the creation of an illustration
reuse graph. Our workflow consists of seven steps:
1. Metadata retrieval and corpus definition
2. Data retrieval
3. Image segmentation/illustration classification
4. Set-up of an image search engine
5. Creation of search result graphs
6. Categorization of search result graphs
7. Creation of illustration reuse graphs
VD 16, the register of printed works of the sixteenth century published in
German-speaking countries,2 is a collection of over 100,000 metadata
records that provides access to a comprehensive and detailed national
bibliography of German prints from the period in question. This extensive data
pool enables data-driven approaches on a large scale and is the starting point
for step 1 of our workflow. Over 68,000 of the records in question link to one
or more digitization(s) of the respective books. The data is retrieved (step 2)
through the collection of digitized books from library pages and especially
via IIIF APIs
        <xref ref-type="bibr" rid="ref5">(Snydman et al., 2015)</xref>
        . Step 3 addresses the task of
segmenting every digitized book page in a dataset, so that illustrations can be
identified and described by means of a selector shaped as a polygon or rectangle.
While we use a very lightweight segmentation algorithm based on
morphological image analysis with OpenCV, any segmentation/classification approach
will work as long as it results in a collection of images and segmented regions
that are either rectangular or can be translated into bounding boxes.3 At the
end of step 3 of the proposed workflow, each illustration (ideally) has been
assigned a segmentation region by the illustration segmentation algorithm.
For the illustration search, we use the open source VGG Image Search
Engine (VISE) in version 2.0.1-beta 1.4
      </p>
      <p>1As an example, see Bodo Gotzkowsky’s seminal studies of the uses and reuses of the
works of block-cutter Hans Brosamer.</p>
      <p>
        2http://www.gateway-bayern.de/index_vd16.html
3For the basic principles of semi-automatic Layout Analysis and Region EXtraction
(LAREX), see
        <xref ref-type="bibr" rid="ref4">Reul et al. (2017)</xref>
        .
      </p>
      <p>4https://gitlab.com/vgg/vise/-/tree/vise-2.x.y. For a detailed description of the
applica</p>
      <p>For the purposes of this article, we will mostly leave steps 1–4 aside, and
focus instead on the graph-related steps 5–7. Steps 1 and 2 are briefly described
in relation to a specific application in subsection 2.1. Step 5 begins with the
task of querying every segmented illustration region in the image search
engine and retrieving all matches for the said region of interest, which results
in a number of query results that is identical with the number of images that
have been fed into the search engine. These query results are then refined by
removing noise and irrelevant results, and the data is put into the format of
a search result graph. The procedures and results of this step are described
in detail in section 1. While this step of the workflow already limits the
result data, further manual work may still be necessary to bring the results in
line with a specific research question. We outline this process with regard to
sixteenth century book data in subsection 2.2. As a final step, we merge the
collected results into a network of illustration reuse, fully enabling our
dataset to be used for social network visualization and analysis (subsection 2.3).
1</p>
    </sec>
    <sec id="sec-2">
      <title>Search Result Graphs</title>
      <p>Search results in general are lists created by a search engine as an answer to a
specific user input. A visual search is a type of search query that receives
image information as input and outputs other image information as search
results. Search results are usually ranked in descending relevance to the search
query. In our visual search with VISE, the user input is a selected book page
and a defined Region of Interest (ROI) on the said page, both of which are
derived from the image segmentation in step 3 of our workflow (see Figure 1).
The search result is a list of other book pages with matching image features.
The ranking is based on the number of relevant matching key features in
close proximity to each other (inliers) in a method called bag of visual words.5</p>
      <p>
        Upon retrieving the complete results of a full n × n image search for n
input images, we are faced with two problems: first, the image search is a black
box, which means we can never be exactly sure how well it is dealing with
a specific dataset until we have a look at the results; and second, while the
ranking ensures that the most relevant search results come first, it does not
prevent the return of potential matches that have virtually no resemblance to
the input ROI. A fixed value score threshold for the number of feature inliers
can be used, but given that the number of inliers is highly dependent on the
number of key features in both the query image and in the matching image,
this is not really a feasible solution for achieving a sharp limitation of results.
Were we to use those search results without further filtering, we would either
tion of VISE on incunabula data from the fifteenth century, see also
        <xref ref-type="bibr" rid="ref8">Zisserman et al. (2020)</xref>
        .
5For technical details regarding this approach, see
        <xref ref-type="bibr" rid="ref1">Arandjelovic (2013)</xref>
        .
lose a significant amount of valuable results by being too restrictive, or we
would end up with a large number of book pages that technically match one
another, while in fact having no meaningful connection at all. Therefore,
it is necessary to limit the result sets provided by the image search
according to various parameters. The main parameters that can be used for this
operation are relative inlier score, transformation matrix properties, and the
Jaccard coeficient between search result and segmented region.
(a) Page 1 (2 regions)
(b) Page 2 (2 regions)
The relative inlier score takes into account the maximum possible number of
matching key features that can be found in a given query region. This is easy
to determine, since every query region finds itself as the best match. Based
on this maximum score, we can compute the ratio between current score and
maximum score for every search result. Empirically, cutting of the results
at a threshold of about 0.02 for that ratio removes a reasonable amount of
irrelevant matches (for details, see Table 1), making the score parameter the
most simple and robust cutting parameter.
1.1.2
      </p>
      <sec id="sec-2-1">
        <title>Transformation Matrix Thresholds</title>
        <p>The relation between a query region and a result region is given by the
values of a 3 × 3 transformation matrix. It describes the transformation of a
given rectangle into a polygon by translation in x and y dimension, scaling in
x and y dimension, rotation, and skewing. Translation only describes a shift
between image positions and cannot be used for cutting results. While the
digitized data of book pages can be quite heterogeneous and of varying
quality, we can assume that we always have a picture of a more or less flat page
positioned parallel to the image frame, with the camera or scanner pointing
directly at it. While warped pages or slightly rotated books do eefct the
outcome, we can rule out the rotation of the camera as a factor in most of the
illustrations that concern us. This vastly limits the possibilities of plausible
skewing between the query region and the result region, and enables us to
assess the decomposed transformation matrix in itself without the need to
account for the specifics of a given query: skewing is limited to values close to
0. Rotation is expected in ranges around 0, (−)90 and (−)180 degrees,
taking into account that a woodcut might have been printed on a book page in
a vertical position instead of a horizontal one, or that a digitized page might
not have been rotated properly. Scaling is unlimited, but the ratio between
x and y scaling is not. If the scaling ratio diefrs too much from a 1:1 ratio,
the transformation must be considered as distorted, eliminating it from the
list of possible results.
1.1.3</p>
      </sec>
      <sec id="sec-2-2">
        <title>Jaccard Coeficient Threshold</title>
        <p>While the query regions are defined as bounding box rectangles for an
illustration segment, the resulting regions of the image search are parallelogram
regions. For a concise result of the whole image search process, we need to
match the resulting regions with the ones we have provided as query regions.
For that purpose, we calculate the intersection over union value (IoU), also
called Jaccard coeficient. 6 This matching is necessary if there are any pages
in the corpus that contain more than one illustration region.</p>
        <p>Tests have shown that once one of the applied filter parameters indicates
an invalid match, it is very unlikely for results with a lower search result rank
to be valid. Therefore, the list of results can be cut once an invalid parameter
threshold is met.
1.2</p>
      </sec>
      <sec id="sec-2-3">
        <title>Graph Creation</title>
        <p>Graph creation can be explained by means of two sample pages to which the
image search workflow is applied. On sample page 1 (p1), two illustrations
were segmented successfully (indicated by blue bounding boxes in Figure 1):
6The Jaccard similarity coeficient is often used in machine learning to determine the
quality of classification – especially object detection – compared to a ground truth. Given
a classified region and a region for comparison, the similarity between the two regions is
computed by calculating the intersection of both areas and dividing it by the union of both
areas. Identical regions result in a value of 1, disjoint regions in a value of 0.
one depicting a blue elephant (p1r1), the other a large three-leaf clover (p1r2).
Likewise, on sample page 2 (p2), two illustrations were segmented: a gray
mouse (p2r1), and a gray elephant (p2r2) that is similar to the blue elephant
on p1, except for minor changes to its contours and a slight rotation to the
left. A third illustration, depicting a three-leaf clover identical in shape and
color to the one on page one was not segmented, because the leaf was too
small (due to a size threshold for illustration areas used in the segmentation
step).</p>
        <p>When the image search is applied to the four segmented illustration
regions, the following occurs:
(a) Matching of p1r1</p>
        <p>(b) Matching of p2r2
(c) Matching of p1r2</p>
        <p>(d) Matching of p2r1
• p1r1 (blue elephant) finds a match on p2 ( Figure 2a). This match has
a Jaccard similarity of 0 with p2r1 (no intersection) and a Jaccard
similarity close to 1 with p2r2. Therefore, p1r1 and p2r2 are considered a
match.
• p2r2 (gray elephant), on the other hand, finds a match on p1 (
Figure 2b). This match has a Jaccard similarity of 0 with p1r2 (no
intersection) and a Jaccard similarity close to 1 with p1r1. Therefore, p2r2
and p1r1 are considered a match, making the match reciprocal.
• p1r2 (three-leaf clover) finds a match on p2 ( Figure 2c). This match
has a Jaccard similarity of 0 with p2r1 and a Jaccard similarity of 0 with
p2r2. Therefore, no valid match is detected.
• p2r1 (mouse) does not find match on any of the available pages (
Figure 2d).
It is possible to create a search result graph directly from the above results.
This search result graph is a directed graph where the nodes are segmented
illustration regions and the edges connect two segmented regions if they are
considered a match. Such a graph can be drawn by following a few general
rules (the resulting graph can be found in Figure 3):
1. A segmented region pnrm finds an image search match on po. po
contains k segmented regions (k &gt; 0). The Jaccard similarity to one or
more of those k regions is above an IoU threshold (min 0). Region
porp with the best IoU value is considered a match. A directed edge
(pnrm, porp) is drawn.
2. A segmented region pnrm finds an image search match on po. po
contains k segmented regions (k &gt;= 0). No Jaccard similarity to those k
regions is above an IoU threshold (min 0). No edge is drawn.
3. Optional: All non-reciprocal edges are removed.7
p1r1</p>
        <p>1
p1r2
3
p2r1</p>
        <p>2
p2r2</p>
        <p>4
Graph creation and result filtering can be done in any order. Both
approaches have their own distinct advantages. Filtering first reduces the
amount of data to be processed in the graph creation step tremendously.8
For large real world datasets with over 50,000 images, filtering reduced the
number of results to be processed by over 90% (Table 1). This approach is
7Keeping only reciprocal edges simplifies calculation of graph measures, because the
graph can then be treated as undirected. It also has the potential to significantly enhance
the precision of the result, but may in turn have a detrimental eefct on the recall due to
valid results being discarded. Experiments have shown that it is not unlikely for valuable
matches to be discarded if they are borderline cases for the image search (coloring, skewing,
etc.), so this option should be handled with care.</p>
        <p>8With the exception of filtering by Jaccard coeficient threshold, since calculation of the
intersection over union value is mandatory in the graph creation step.
especially advisable if speed is of the essence, or if system resources are
limited. Creating the graph first, before refining the result by filtering it with
the same thresholds and by removing any edges from the graph, slows down
the process. However, it also makes full use of the graph approach for the
assessment of the filter threshold parametrization. The characteristics of the
created graph provide valuable information on the quality of the search
results that cannot be taken into account when filtering the search result data
directly.</p>
        <p>The ideal search result graph consists of n fully connected components.
A fully connected component is a subgraph where each node is connected
by an edge with every other node within it. A fully connected subgraph with
k nodes perfectly describes one matching illustration with k corresponding
segmentation regions on up to k book pages (see Figure 4). In reality, the
result usually diefrs to a certain degree from the ideal graph form due to
errors both in image segmentation and image search. In theory, we would
like to calculate the diefrence between the ideal graph and the real graph
as a measure of quality, but besides specified ground-truth data, we know
neither the correct number of subgraphs, nor their individual ideal sizes for a
given result graph. When it comes to evaluating real world data, however, we
can still assess some graph measures in order to estimate whether the graph
is reasonably close to an ideal graph structure, even though such proximity
does not in itself guarantee an ideal or even passable result.
1.3.1</p>
      </sec>
      <sec id="sec-2-4">
        <title>Assessment by Graph Measures</title>
        <p>Graph measures for quality assessment must be divided into global measures
calculated for the whole graph and local measures calculated for individual
subgraphs or nodes. Given the sheer size of search result graphs that contain
millions of nodes, global graph measures tend to be dificult to calculate
efifciently. Some basic filtering is mandatory before starting to look into the
ifner grained assessment, because without it, it might not even be possible
to partition the overall graph into subgraphs. Once basic filtering is done, it
becomes significantly easier to apply local graph measures on the subgraphs.</p>
      </sec>
      <sec id="sec-2-5">
        <title>Global Measures</title>
        <p>In the following section, we demonstrate the global assessment of a basic
graph filtering sequence using a string of (conservatively chosen) thresholds.
We start with an unfiltered graph and apply thresholds in sequence, so every
new filter is applied to the output of the previous filtering. Initially, a
relative score filter of 0.01 is applied to the unfiltered graph due to the high
number of nodes it contains. This is done to circumvent excessive memory
consumption in generic graph analysis tools like Gephi. Whenever the edge
ifltering leaves isolated nodes with no further connections, those nodes are
also removed. The following filter chain is then applied in sequence:
filtering edges by relative score threshold, then by rotation angle dif mod 90,
followed by ratio threshold of scaleX and scaleY, and finally by IoU threshold
of the matching regions. For the filtered graphs, we calculate the following
global graph measures: number of nodes, number of edges, number of
connected components, and the global avg. clustering coeficient for directed
graphs9. The results can be found in Table 1.</p>
        <p>The clustering coeficient of a graph is a measure of how densely the
subgroups of a graph are connected. The closer a graph is to a graph
consisting of fully connected components as subgraphs, the higher the clustering
coeficient. Therefore, the avg. clustering coeficient of a graph is a good
global measure with which to assess the distance to an ideal graph structure.
Since the image search produces a lot of ‘noisy’ and meaningless results, we
expect to be able to filter a significant amount of edges from the initial graph.
The number of filtered nodes is dependent on the specific dataset. Ideally, if
every segmented illustration had at least one match in the dataset, no nodes
would have to be removed in the filtering process. In reality, however, we
encounter a varying number of segmented illustrations which are unique
within the dataset. These are created in the initial, unfiltered graph and are
9We use Patrick McSweeney’s implementation in Gephi. For details on the implemented
algorithm, see https://github.com/gephi/gephi/wiki/Average-Clustering-Coefficient.
subsequently removed. The number of filtered nodes therefore can only be
assessed with a deeper knowledge of the dataset. Table 1 shows that the
number of edges is reduced tremendously over the course of filtering. Over 90%
are filtered on the score threshold, while the avg. clustering coeficient also
increases. This shows that the relative score filter is crucial to the creation of
a meaningful graph. As expected, the number of connected components
increases as the edge filtering successively breaks up larger subgraphs into
smaller, more precise subgraphs.</p>
        <p>|Nodes|
|Edges|
Conn. Comp.</p>
        <p>Cluster. Coef.</p>
        <p>Edge filter %</p>
        <p>Edge filter
per step %
Node filter %</p>
      </sec>
      <sec id="sec-2-6">
        <title>Local Measures</title>
        <p>Iterating over the subgraphs of the search result graph, we can supplement
those global measures with more detailed local measures. In practice, even
if the graph is not completely broken up into perfect subgraphs, individual
components never contain more than a few thousand nodes and, on average,
considerably less. We can identify fully connected components by two local
measures: either by their density, or by the size of their maximum clique.
The maximum clique of a graph is the largest fully connected subgraph of a
given graph, so if we want to have an ideal search result subgraph of n nodes,
the maximum clique size is also n. The clique problem is an np-complete
problem and therefore impossible to solve in polynomial time on arbitrary
graphs, but it can nevertheless be computed for our specific graph shape of
small subgraphs. Calculating the fraction of the maximum clique size and
the number of nodes in a subgraph gives us a measure of closeness between
the subgraph and an ideal – i.e. fully connected – one. A similar measure
is provided by graph density. Graph density calculates the ratio of edges in
a graph with respect to the number of edges possible. Due to the
complexity of this operation, some implementations of the maximum clique size are
only provided for undirected graphs10, which is why we must treat our graph
as such. The simplified result might not always represent the true shape of
a subgraph, but using a simpler density measure means that it can be
computed for either directed or undirected graphs.</p>
        <p>A third measure of interest is the calculation of a minimum dominating
set in a subgraph. A dominating set of nodes in a graph is a set of nodes from
which all other nodes in the graph can be reached by traversing a single edge.
In a minimum dominating set, the number of nodes in that set is as small as
possible for the given graph. The set itself does not have to be unambiguous
– multiple minimum dominating sets may exist in one graph. In an ideal,
fully connected graph, the size of the minimum dominating set is one, and
every node of the graph qualifies as a dominating set, because every other
node can be reached directly. Again, the minimum dominating set size can
be computed for both the directed or undirected variant of our subgraphs,
providing diefrent measures. Density and minimum dominating set size are
not directly correlated, since even a very sparse graph can have a close to ideal
dominating set size (if we have a graph of n nodes (with a large n), where one
node is connected to all other nodes, which in turn are not connected among
each other at all, we have a sparse graph and a minimum dominating set size
of 1). The dominating set size is especially interesting as a complementary
graph measure because it gives us a much more detailed idea of a specific
graph shape. It is also useful for identifying the most ‘characteristic’ nodes
in a search result, since it contains the nodes that are most important for a
graph’s connectivity.
1.3.2</p>
      </sec>
      <sec id="sec-2-7">
        <title>Visual Assessment</title>
        <p>In addition to allowing the calculation of graph measures for quality
assessment, arranging the search results in a graph structure enables the user of
the workflow to employ generic graph visualization tools for visual
assessment. It is a fast, flexible, and easy way to identify unusual graph
characteristics (subgraphs with an extraordinarily large number of nodes, or subgraphs
that diefr from the ideal fully connected component structure). Moreover,
metadata information can be used for graph partitioning/node coloring, so
that, for example, nodes can be colored by book, by place of publication, by
printer, etc. This can help to identify underlying problems and may be
useful for understanding deviations from the ‘ideal’ structure explained above.
In Figure 5 a subgraph is shown with a (directed) density of 0.384. Diefring
significantly from a fully connected component structure, the graph is
separ10See, for example, the Python module networkx: https://networkx.github.io/documentation/
stable/reference/algorithms/generated/networkx.algorithms.clique.graph_clique_number.html.
ated quite clearly into two strongly interconnected areas with relatively weak
links between the two. We examine two nodes from the minimum
dominating set in the graph, one from the left area and one from the right. The
segmented illustration regions can be found in Figure 6. Both feature the
same image of a man boarding a ship, so they are overall correctly placed in
the same subgraph. Upon closer examination, however, it becomes clear that
the prints are not identical: one woodcut is a recut of the other and exhibits
slight variations in details like the facial features of the figures. 11 While the
recut is similar enough to the original to trigger a match in the image
similarity search, the diefrences are visible in the shape of the resulting subgraph,
which in turns divides the graph into two quite separate clusters. This is a
case where the graph structure clearly reflects an important aspect of historic
production processes. Outliers such as these cannot satisfactorily be
evaluated by simple graph measures, but are quick and easy to assess with the help
of visual analysis.
2</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>Application to Sixteenth Century Book Illustrations</title>
      <p>The workflow described above can be used to research networks of sixteenth
century books and their illustrations. In this section, we will define a sample
dataset and further elaborate on its characteristics as well as workflow steps
6 and 7, which lead us from a search result graph to a network of book
illustrations.
2.1</p>
      <sec id="sec-3-1">
        <title>Fine-Tuning the Dataset: Illustrated Books from Frankfurt am</title>
      </sec>
      <sec id="sec-3-2">
        <title>Main</title>
        <p>Taking the VD16 register as our starting point, we can use the
bibliographical metadata to select a specific dataset to which our workflow can then
be applied. In addition to bibliographical data concerning a work’s author,
title, printer, or place and year of publication, the VD16 also collects more
material-oriented information, such as book size and ornamentation
(“Buchschmuck”). The data concerning ornamentation is organized as follows:
TE (Titeleinfassung – title frame), TH (Titelholzschnitt – title woodcut),
TK (Titelkupferstich – engraving on the title page) for ornamental elements
on the title page; H (Holzschnitt – woodcut), K (Kupferstich – engraving)
11See, for example, the man on the bow of the ship (especially his hat and face) and the
beards/faces of the men on the right, as well as other details.</p>
        <p>12Digitized by Staatsbibliothek Berlin, see http://resolver.staatsbibliothek-berlin.de/
SBB0001C26800000000.</p>
        <p>13Digitized by Bayerische Staatsbibliothek, see http://mdz-nbn-resolving.de/urn:nbn:de:bvb:
12-bsb00070159-3.</p>
        <p>(a) Woodcut on VD16 H 2662, p. 11312</p>
        <p>
          (b) Woodcut on VD16 H 3866, p. 6213
and RL (Randleiste – ornamental frame) for decorative elements within the
book; and D (Druckermarke – printer’s mark) for the insignia of printers
and publishers. While engravings do not play a significant role as an
illustration technique in the sixteenth century, over 26,000 books are tagged with
H for woodcuts and over 23,000 with TH for title woodcuts. This
information plays a critical role in our data selection, since it allows us to limit our
focus on books containing woodcuts or title woodcuts. Frankfurt am Main,
which became important to the printed book trade following the
establishment of Christian Egenolf’s print shop in the 1530s
          <xref ref-type="bibr" rid="ref2">( Kulturvereinigung
Hadamar, 2002)</xref>
          , is among the top 10 printing centers listed in the VD16.
It is in fact the city where the most books featuring woodcuts were printed
(close to 2,600), followed by Wittenberg and Strasbourg. This is especially
impressive since Frankfurt only turned into a major hub of the emerging
printing industry in the second half of the sixteenth century, while
Wittenberg and Strasbourg had played an important role more or less from the
beginning of the century. Frankfurt’s prolific output and its many active
printers and printing dynasties make the city an ideal candidate to study the reuse
of illustrations in a quantitative way. The VD16 contains 1886 entries with
‘Frankfurt/Main’ as their place of publication which are both labelled as
containing woodcuts and linked with a digitized reproduction, thereby enabling
them to be used in an image search workflow.
        </p>
        <p>Since nearly a quarter of the books with woodcuts from Frankfurt are
not linked to a digitized version in VD16, our application does not claim
to provide a complete result for the corpus in question. The main goal of
the following example is to show if and how the results of network creation
are meaningful and coherent. At this point, it is to be understood as a study
of technical feasibility rather than as an accurate visualization of historical
practices. It does, however, provide a basis for further inquiries along the
same lines. We will present the results of our approach for around one
thousand digitized books in subsection 2.3, which yielded around 53,700 pages
that were segmented as containing one or more illustration region(s) in
worklfow step 3, and were subsequently ingested into a VISE search engine in
workflow step 4. The resulting search result graph consists of around 9,100
subgraphs.
2.2</p>
      </sec>
      <sec id="sec-3-3">
        <title>Fine-Tuning the Definition: Woodcuts vs. Illustrations</title>
        <p>
          It became clear both during data retrieval and after initial use of the image
search that the VD16 metadata is not detailed enough to meet the
requirements of our specific research question. In VD16, the label ‘H’ is assigned to
all non-textual elements in a book outside of the title page that are produced
by inserting a wood block into the page layout for printing.14 This includes
all kinds of decorative elements such as ornaments and initials, as well as
images, such as botanical illustrations, mathematical figures, etc. While this
categorization appears quite clear-cut, the term ‘illustration’ itself is actually
rather fuzzy. Narrowly defined, illustrations are
all pictures in a book that are meant to accompany a literary text to
clarify content and plot for the reader, to strengthen the text’s
significance and impact. Portraits, factual drawings
[Sachzeichnungen], depictions of cityscapes, technical representations,
botanical images etc. are to be called figures.
          <xref ref-type="bibr" rid="ref7">( Wendland, 1991)</xref>
          15
Our goal is to find a middle ground. We are interested in all images that
support, clarify or accompany a text. Whenever such an image is reused, some
kind of knowledge transfer takes place, which is precisely what makes the
reuse and recontextualization of such materials so interesting. Yet our
experience has shown that if initials and ornamentation are included, the
majority of books printed in Frankfurt are somehow connected with one another,
making the resulting network incredibly large, noisy, and outright unusable
for further analysis. Therefore, the decision was made to leave purely
decorative elements as well as printer’s marks aside, while botanical and medical
illustrations, cityscapes, and other explanatory figures are included. There
are, of course, borderline cases such as historiated or inhabited initials (i.e.
decorative letters that contain identifiable figures or scenes from a story).
        </p>
        <p>However, regardless of the specific definition of illustration being used,
an image similarity search cannot be aligned with such fuzzy, a priori
criteria with acceptable precision. It is a clear advantage of the graph approach
that the visual assessment described in subsubsection 1.3.2 is not only useful
to assess the graph quality, but can also be utilized to further specify the data
of interest for a given research question. To do so, a lightweight graphical
user interface has been implemented to show the subgraphs of the created
search result graph in a tabular view (see Figure 7). This enables a researcher
to quickly assign tags to subgraphs such as ‘initial,’ ‘printer’s mark,’
‘decoration,’ etc. Multiple tags chosen from a simple predefined list of terms (to
limit erroneous user input) can be assigned to one subgraph. While this
system is currently only used to define the subset of data for a specific
research interest, it can of course be expanded to meet the demands of various
annotation tasks. The list of terms can easily be expanded into more
com14The only exception are printer’s marks, which are labelled separately.
15Author’s translation.
plex controlled vocabularies for the purposes of classification and analysis,
e.g. through the addition of Icon Class16 information to illustrations.
This approach allowed 77 diefrent printer’s marks from our dataset to be
identified, 17 and 185 subgraphs to be tagged as initials.18 Combining
automatic segmentation and search with manual categorization narrows the
figure of several thousand book pages down to a manageable number of
subgraphs that can then be used to accommodate more finely tuned research
requirements. This versatility makes our approach suitable for a wide range
of research topics: not only does it allow for manual categorization on large
datasets, but it also makes it possible to conduct complex algorithmic
assessment or analysis on one representative image per subgraph instead of all
images in the larger dataset. To ensure representativity, graph measures can
again be utilized, for example, by choosing one image per subgraph from the
respective minimum dominating set.</p>
      </sec>
      <sec id="sec-3-4">
        <title>Fine-Tuning the Representation: Illustration Reuse Graph</title>
        <p>
          Following the creation and optional categorization of the search result graph,
the next step is to transform the subgraphs that identify identical
illustra16http://www.iconclass.nl/home
17The identification of common printer’s marks was greatly aided by the collection of
          <xref ref-type="bibr" rid="ref6">Wendland (1984)</xref>
          .
        </p>
        <p>18The total number of initials would be significantly higher, but the workflow frequently
(and erroneously) puts diefrent initials in the same subgraphs due to their high structural
similarity and their low number of distinct visual features.
tions into a network of books connected by identical illustrations. The
result of this step is a graph in which each node represents one book in our
dataset. An edge is drawn between node a and node b if they share at least
one illustration. The edge is weighted by the number of shared illustrations
between the books, so that if one illustration is found in book a and book b,
the edge weight is 1, whereas if 42 illustrations are found in both book a and
b, the edge weight is 42. The input that is required for such a node merging
process to transform a search result graph into a network of illustration
reuse is shown in Figure 8. Meanwhile, the merged output appears in Figure 9.
Isolated book nodes with no connections to other books are removed after
merging.
b1
b3</p>
        <p>It is possible to apply graph layout algorithms on the created network that
take into account the weighted edges between the books. Thus, two books
are attracted to each other if they are connected with a (highly) weighted edge
and repel each other if no edge exists between them. This way, books with
similar illustrations cluster together in subgroups. Bibliographical metadata
from VD16 can then be used to color nodes and edges by partitioning. For
example, book nodes can be colored according to the respective printers
involved, in order to show how illustrations circulated within one print shop,
or were transferred to other businesses. Edges can be colored according to
the diefrence of publication dates between books, ranging from a ‘hot’ color
for illustrations that are reused within a short period of time (within a few
years or even in the same year) to a ‘cold’ color for those that are reused over
a longer temporal duration. Figure 10 shows an illustration reuse network
for Frankfurt am Main before and after application of a graph layout and
coloring of nodes and edges by metadata. It contains 735 books connected
by at least one shared illustration. The data has been processed and manually
categorized according to the steps described above. The network depicts a
high amount of illustration reuse both within and between particular print
shops. Node degrees (and therefore illustration reuses involving individual
books) range from 1 to 148 (the latter being VD16 P 3553, a Naturalis
historia), the time span between the first and the last use of the same illustration
varies between 0 and 67 years (the latter showing the reuse of an illustration
of medical blood-letting). The type of clustering that occurs here is
indicative of a ‘functional’ use of illustrations: it is hardly surprising that botanical
or medical books are far more likely to share illustrations with each other
than literary texts or religious treatises. It is still evident, however, that the
functional clusters are not completely separated, but remain connected by
singular instances of reuse that transcend categorizations by functionality
or ‘genre.’ Clearly, further exploring the illustrations in question and the
books (re)using them would be a worthwhile pursuit, and herein lies one of
the main advantages of a quantitative approach like the one proposed here:
far from being limited to showing the connections between books by one
specific printer, dealing with one topic, or belonging to one genre, it is
capable of detecting weaker links between the clusters which are much harder
to discover by traditional means.</p>
        <p>We will now examine one of the subgroups in which literary works such as
Fortunatus, Herzog Ernst, Thüring von Ringoltingen’s Melusine, Elisabeth
von Nassau-Saarbrücken’s Hug Schapler, and other well-known and widely
disseminated narrative texts are clustered together. Historically, this book
cluster began to emerge with the activities of Weigand Han in Frankfurt,
who illustrated multiple print runs of Fortunatus with a collection of
woodcuts produced by Hans Brosamer, which were then quickly reused in
conjunction with other narrative texts in the 1550s. The network as it stood at
the end of the first half of the sixteenth century is represented in Figure 11a.
At this point, significant reuse of the illustrations in question had already
taken place, albeit only within the ‘genre’ of entertaining narrative texts, and
only in Weigand Han’s own print shop (Han mostly printed under his own
name, but on one occasion used the imprint of ‘heirs of Hermann
Gülfferich’). In the second half of the sixteenth century, however, the reuse of
illustrations became more diverse, and went beyond the group of narrative
texts for which the images had initially been created. For example, a generic
(a) Random layout
(b) Force Atlas Layout (sigmajs)
illustration of a grifin saw frequent reuse as a title woodcut for a group of
cautionary ‘devil’s books’ (‘Teufelsbücher’). Generic social scenes, such as an
image of people sitting at a table talking, or of two knights riding in a
tournament, were also recontextualized, as were images of torture and execution.
Entry J 627 in the VD 16 register plays an extraordinary role in linking the
cluster of narrative texts to another group of books. Printed in 1580 by
Weigand Han’s son, Hartmann Han, it is a German version of John Mandeville’s
Travels, a fictional travelogue detailing the author figure’s journey through
the Middle and Far East. The text exhibits a fascinating mixture of literary
story-telling and biblical references, and the book’s illustrations follow the
same pattern, with pictures that typically accompanied literary texts being
intermingled with images otherwise used to illustrate religious texts.</p>
        <p>Our case study shows that all four network types outlined in the
beginning play a significant role in the reuse of book illustrations in the sixteenth
century; a practice characterized by a multitude of social, economic, and
material connections between the various actors and publications involved.
Using the approach outlined here, all of these links can be explored
simultaneously.
3</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>Conclusion</title>
      <p>As we have demonstrated, graph technologies can be employed at every step
of the way from ranked visual search results to a fully-fledged network of
illustration reuse. All individual parts of the data workflow can be assessed and
analysed by means of graph measures and graph visualization. Our method
allows us to apply a quantitative approach to large-scale datasets, while at the
same time enabling us to take full advantage of the benefits of visual search.
(a) Illustration reuse up to 1559
The application of our workflow to digitized books from the sixteenth
century and their bibliographical metadata has shown that it delivers viable and
useful results with a high degree of versatility and transparency. Even if some
of the links between books and printers from this time period are bound to
go undetected as long as gaps remain in the digitized data, our approach is
an expedient step towards a more complete picture of the historical practice
of illustration reuse. The method we propose allows researchers to tap into
a wealth of cultural heritage data that has yet to be explored with the aid of
big data, and helps bridge the gap between generic image similarity search
and highly individualized research interests.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <string-name>
            <surname>Arandjelovic</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2013</year>
          ).
          <article-title>Advancing Large Scale Object Retrieval</article-title>
          .
          <source>PhD thesis</source>
          , University of Oxford, http://ethos.bl.uk/OrderDetails.do?uin=uk.bl. ethos.
          <volume>640168</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          <string-name>
            <given-names>Kulturvereinigung</given-names>
            <surname>Hadamar</surname>
          </string-name>
          , editor (
          <year>2002</year>
          ).
          <source>Christian Egenolf</source>
          <volume>1502</volume>
          -1555:
          <article-title>Ein Frankfurter Meister des frühen Buchdrucks aus Hadamar</article-title>
          . Glaukos, Limburg.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          <string-name>
            <surname>Künast</surname>
            ,
            <given-names>H.-J.</given-names>
          </string-name>
          (
          <year>2013</year>
          ).
          <article-title>»Getruckt zu Augspurg«</article-title>
          . De Gruyter, Berlin/Boston, MA, DOI: 10.1515/9783110935080.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          <string-name>
            <surname>Reul</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Springmann</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Puppe</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          (
          <year>2017</year>
          ).
          <article-title>LAREX - A semiautomatic open-source Tool for Layout Analysis and Region Extraction on Early Printed Books</article-title>
          . arXiv,
          <volume>1701</volume>
          .07396, https://arxiv.org/abs/1701. 07396.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <string-name>
            <surname>Snydman</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sanderson</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Cramer</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          (
          <year>2015</year>
          ).
          <article-title>The International Image Interoperability Framework (IIIF): A Community &amp; Technology Approach for Web-Based Images</article-title>
          .
          <source>In Archiving 2015: Final Program and Proceedings</source>
          , pages
          <fpage>16</fpage>
          -
          <lpage>21</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          <string-name>
            <surname>Wendland</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>1984</year>
          ). Signete. Deutsche Drucker- und
          <string-name>
            <surname>Verlegerzeichen</surname>
          </string-name>
          1457-
          <fpage>1600</fpage>
          . Schlütersche Verlagsanstalt und Druckerei, Hannover.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          <string-name>
            <surname>Wendland</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>1991</year>
          ).
          <source>Illustration. In Lexikon des gesamten Buchwesens</source>
          , volume
          <volume>3</volume>
          .
          <string-name>
            <surname>Hiersemann</surname>
          </string-name>
          , Stuttgart.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <string-name>
            <surname>Zisserman</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dutta</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Malaspina</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Dondi</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          (
          <year>2020</year>
          ).
          <article-title>The Use and Reuse of Printed Illustrations in 15th-Century Venetian Editions</article-title>
          . In Dondi, C., editor,
          <source>Printing R-Evolution and Society 1450-1500</source>
          , pages
          <fpage>839</fpage>
          -
          <lpage>869</lpage>
          . Edizioni Ca' Foscari, DOI: 10.30687/
          <fpage>978</fpage>
          -88-6969-332-8/030.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>