The advent of large-scale citation services has greatly impacted the retrieval of scienti c information for several domains of research. The Humanities have largely remained outside of this shift despite their increasing reliance on digital means for information seeking. Given that publications in the Humanities probably have a longer than average life-span, mainly due to the importance of monographs in the eld, we propose to use domain-speci c reference monographs to bootstrap the enrichment of library catalogs with citation data. We exemplify our approach using a corpus of reference monographs on the history of Venice and extracting the network of publications they refer to. Preliminary results show that on average only 7% of extracted references are made to publications already within such corpus, therefore suggesting that reference monographs are e ective hubs for the retrieval of further resources within the domain.
The Humanities are the Cinderella of sciences with respect to citation-driven
information retrieval. The lack of citation data not only prevents the quantitative
analysis of the eld's communication practices (
There are several reasons for this state of a airs, yet the lack of citation data
is the most renown problem, lamented several times over (5; 8; 14). For these
and other reasons the use of citations as a means to evaluate research in the
Humanities has also been questioned (
Giovanni Colavizza, Matteo Romanello, and Frederic Kaplan
We sketch here the main steps of our approach, as illustrated in Fig. 1. The
corpus of reference works is rst selected, then digitized and OCRed. The manual
annotation of a subset of references is then followed by their automatic extraction
over the whole corpus. Finally, the look-up module nds matches into the library
catalog, and connects paired resources (cited with citing monographs, or
cocited monographs). Eventually, a second look-up module is introduced in order
to evaluate the cohesiveness of the selected corpus, de ned as the fraction of
references made to resources within the corpus over the total extracted resources.
1. all the resources in the consultation shelves devoted to the History of Venice;
2. all the resources under subject History of Venice (e.g. Dewey code 945.31);
3. expand by keyword search over the title (e.g. using words as \Venice" in
multiple languages) and by using scholarly bibliographies such as Zordan's
repertory (
The outcome is a set of 1904 monographs and 10 journals. The number of monographs with a list of references is 836 (201 in consultation, cat. 1), 44% of the corpus of monographs, equally distributed over time as shown in Fig. 1. Of these, 701 (184 cat. 1) have structured lists of references, as opposed to end notes, which have been manually individuated and will be used in what follows.
The second step in our pipeline is the classi cation of reference styles and the manual annotation of a sub-set of references for each individuated class. A reference style is a speci c combination of elements in a reference, such as author and title, encoded in a prede ned way (e.g. using quotations for the title). We grouped styles in classes and families. For example: \De Virine, Theodore Low. Notable Printers of Italy during the Fifteenth Century. New York: The Grolier Club, 1910."
Is a reference presenting the author's surname, then name separated by comma, title, place of publication, publisher and date. The punctuation and capital letters in use are particularly relevant. A di erent class stems from the elimination of at maximum one element, or one change in encoding. E.g. removing the publisher would create a new class of the same family. A di erent family is identi ed by at least two removals or additions of elements, and/or sensible changes in the encoding of the same information. For example: \De Virine, T. L. Notable Printers of Italy during the Fifteenth Century. The Grolier Club, 1910."
Would stem a di erent class in a separate family as the author's name is now abbreviated and the publisher has been dropped. Classes and their families are useful as a feature for parsing since the references from a speci c publication all belong to a unique class/family combination.
In total we individuated 33 classes and 6 families.
Manual annotation was then done over a set of references for each class.3 Annotations are distinguished into two categories: generic and speci c. A generic annotation distinguishes the completeness of a reference (if full or abbreviated) and the type of referred object (if a monograph or a contribution, such as a journal article). Speci c annotations identify the components of generic categories. Examples of speci c annotation tags are: \author", \title", \publisher".
Approximately 27% of the 701 monographs have been annotated, 2 pages of references each on average. As a consequence, circa 3.8% of all available pages with references have been annotated. We total 49580 annotations, of which 8646 are generic (i.e. full references) and 40934 speci c (i.e. their components). 2.2 The following component of the pipeline is a parser and reference extraction module, which performs two tasks: 1. Reference parsing: given a text stream of lists of references, parse the text to assign the most likely speci c tag to each token. 2. Reference extraction and categorization: given a stream of tokens with speci c tags, decide where a reference begins and ends, and assign a generic category to the reference (\monograph", \abbreviated" reference and \contribution"). 3 Using the Brat annotation environment available at http://brat.nlplab.org/.
Both parsers use Conditional Random Fields with the same set of features|
except for speci c tags resulting from task 1 that are used in task 2|a technique
commonly adopted for similar tasks, introduced by (
Extracted references need to be disambiguated in order to be used in the catalog. This task is performed by a look-up system that tries to match the components of the extracted reference against a bibliographic database, e.g. a library catalog.
Given the nature of the data at hand, such look-up system had to: have a good coverage of the domain; have the ability to work with a limited set of metadata elds as input; and have a degree of tolerance for errors from OCR.
The solution we implemented builds upon the uno cial API of the electronic library catalog of the Italian library system (SBN).4 This API provides a good coverage of the publications within our monographs, which can be easily explained in light of the focus of our materials on the history of Venice. 3
We present results for the two main components of our pipeline: the module
for reference parsing and extraction, and the look-up. We also brie y discuss
the cohesiveness of our corpus, and its usefulness as a hub for nding further
resources within the domain.
3.1
This module performs the following: given a stream of text likely to contain a
list of references, it initially tags every token with speci c tags. A second model
then parses the text again in order to attribute generic and begin-end tags at
the same time. Eventually, all individuated references for each monograph are
exported for the look-up module. We do all our implementation in Python, using
the CRFSuite (
For both models we began by keeping a validation set of 25% of references on a side, on which we base our nal evaluations. We then experimented with cross-validation on the remaining 75%, in order to nd the best parameters and combinations of training approaches. We tested: 1- reducing the features by removing the token and its lowercase version, plus all su xes and pre xes; 2- removing references to primary sources; 3- training separate models for each family of reference styles; 4- splitting the training data in di erent sizes (sets of references to parse contiguously); and 5- changing the order of the parsing tasks. Test 2 was positive and kept, test 4 gave us a windows of slices of text containing 5 references as optimal for splitting annotated pages for training. Tests 1 and 5 slightly reduced performance, while test 3 produced over tted models, probably because of the lack of su cient and balanced annotated data for every family.
Once the tasks were con gured, we searched the parameter space for the best con guration of our CRFs. Using a quasi-Newton gradient descent method (L-BFGS), we have two main parameters: c1 for L1 and c2 for L2 regularizations respectively. Good parameters were found to be: { Model 1, c1: 0.0289; c2: 0.0546. { Model 2, c1: 1.53; c2: 0.002.
Intuitively, model 2 bene ts from sparse regularization much more than model 1. The result is a set of 181699 references, 8632 of which were part of the golden set and 173067 were newly parsed and extracted.
A 5-fold validation over the whole dataset gives a at and weighted F1-score of 0.77 and 0.85 for task 1 and 2 respectively, while validation scores on the validation set are summarized in tables 1 and 2, which should be read along with confusion matrices in Figure 3.
0) null 0.679 0.553 0.609 1) pagination 0.900 0.905 0.902 2) publisher 0.780 0.688 0.731 3) author 0.847 0.862 0.855 4) title 0.839 0.911 0.873 5) publication number-year 0.772 0.835 0.802 6) publication place 0.860 0.873 0.867 7) year 0.882 0.880 0.881 avg / total 0.805 0.812 0.806 Table 1: Extraction results for task 1: parsing.
For Model 1 the main source of errors are null tokens (without tag). Several tags have been removed due to being either under-represented or too varied to be properly captured. This explains the di culty of the parser to properly t the null tag. Model 2 instead behaves consistently with the availability of data, meaning the abbreviated references are not as well captured as monographs and contributions. It is nevertheless important to note that begin tags mostly get mistaken for other begin tags, and the same for inside and end tags, all of which are weak errors.
8 3.2 The catalog look-up attempts to match the metadata elds of the input extracted reference against the bibliographic metadata accessible via the SBN API. Potential disambiguation candidates are retrieved from the API by performing a search across the whole catalog by using the title of the publication as the search key.
Once some candidates are found, the look-up further expands its search trying to match the input reference against the metadata corresponding elds of each returned catalog record. The number of successfully matched elds is then used to calculate a con dence score, ranging from 0 to 1, which allows us to rank the disambiguation candidates. The con dence score is especially useful in those cases where there are several correct matches for a given input reference. This is often the case given that multiple editions of a given work correspond to multiple records in the catalog.
Several factors needed to be taken into account when performing matching on metadata elds: 1. the title eld of catalog records may contain other details in addition to the title (e.g. the editor of a collective volume). To overcome this issue we check whether each token in the title of the reference is also contained in the title of the disambiguation candidate. 2. Names of authors are usually abbreviated in our references, whereas they are given in their full form in catalog records. Therefore, when matching the \author" we strip name initials and try to match on family names.
We carried out a preliminary evaluation of the accuracy of the catalog look-up in order to assess the feasibility of our approach and identify problem areas.
We took a random sample of 2000 references out of the total 181699, equally distributed between manually annotated and automatically extracted references. For each reference we veri ed whether the look-up result with highest con dence score constitutes a correct disambiguation of the input reference. We considered a result correct also when a di erent edition of the same work was returned.
The evaluation showed the following results: { in 41.7% of the cases the look-up does not return any candidate result. Given that the title is used as a search key this issue may be due to at least four di erent reasons: 1. the title contains an OCR error that prevents it from matching against the catalog (e.g. \La pittura e la scultura veronese dal secolo Vili al secolo XIII" where \VIII" was wrongly recognized as \Vili"); 2. the title is wrongly segmented or even absent due to a parsing error; 3. the cited publication is not contained in the catalog; 4. some words in the title are spelled di erently in the reference and in the catalog record (e.g. \Una famiglia veneziana dal X al XIII secolo" as opposed to \una famiglia veneziana dal 10. al 13. secolo"). { in the remaining cases (58.3%), for 72.3% of the references the rst disambiguation candidate was correct.
While the precision showed by the look-up is encouraging, improving its recall constitutes a crucial area for further work. A qualitative evaluation of the pipeline at the end of the look-up is currently undergoing in order to individuate all classes of errors and then quantify them. 3.3
An important assumption on which our approach rests is the structural hub
role of the selected corpus within the citation network of the domain at hand.
If the selected corpus is not su ciently spanning outside of itself, at the same
time being well-connected internally (presenting a giant component), then we
might nd it not e ective in order to connect di erent research areas within
the same domain of study. Our assumption was not immediately supported by
previous work, which in general highlighted great variability in citation patterns
among di erent disciplines in the Humanities. Co-citation structures by domain
and by research themes can both be found (see e.g. (
We adapted the lookup module presented in section 3.2 in order to look references up within the corpus itself. The details of this adaptation are beyond the scope of this paper, su ce to say we tuned it to maximize precision in order to avoid miss-matches. The adapted look-up module has been manually evaluated on a small set of 500 extracted references, resulting in a precision score of nearly 1.00 and a recall score above 0.95.6
As a reminder, the extracted references of 701 or 37% monographs (of which 184 or 9.7% in consultation, cat. 1) have been matched against the whole corpus of 1904 (100%) selected monographs. We considered only extracted references to monographs, which are 96607 (over the total of 181699).
Firstly, we investigated the cohesiveness of our corpus, de ned as the proportion of references inside of the corpus itself, over the extracted total. Results are summarized in Table 3. Overall, only 7% of the extracted references are to monographs within the corpus, slightly more for the monographs in consultation (8%).
Consultation, cat. 1 0.0802 1861(21337) Without cat. 1 0.0669 5398(75270) All set 0.0699 7259(96607)
Table 3: Citation span of the elected corpus: most of the references are to the outside. 6 These high scores should not be taken as nal: the evaluation was carried out on extracted references, which have errors from previous steps. The evaluation of the whole pipeline from the beginning to the end is still ongoing.
Giovanni Colavizza, Matteo Romanello, and Frederic Kaplan
Secondly, we investigated the connectedness of the corpus itself by the extracted references. This is equivalent to the proportion of monographs from the corpus which are in the giant component of the co-citation network resulting from the look-up procedure. The giant component is well-individuated and comprises circa 59% of the corpus. The coverage drops to 32.5% using only the 184 monographs in consultation.
These preliminary results suggest that most of the selected corpus could be useful as a collection of hubs pointing to the relevant literature in the domain, also being strongly connected internally. 4
We proposed to use a selection of domain-speci c reference monographs in order to enrich Humanities library catalogs with citation data. Our main goal is to allow users to rapidly nd the most relevant publications on a topic of interest. Our contribution in this respect is twofold. Firstly, we developed a robust pipeline for the extraction and disambiguation of references contained within publications from the Humanities, evaluating it on a dataset from the domain of the History of Venice. This system constitutes the rst necessary step towards the envisaged enrichment of library catalogs. Secondly, we brie y investigated the citation structure of the same dataset in order to assess how e ective it may be in serving as a hub to access the domain literature. We found that only 7% of the references made from such corpus are to monographs already within the corpus, suggesting that a wide span over the literature might be achieved from a limited set of selected reference works.
The work presented in this paper constitutes but one aspect of our project. Other aspects that are currently being developed are 1) the extraction of references from footnotes contained in journal articles and 2) the extraction of references to primary sources, such as archival documents, that are often found in publications on the history of Venice. The latter, in particular, will allow us to transform those references into links to archival information systems.
We thank Martina Babetto and Silvia Ferronato for the annotation work. The Library of the Ca' Foscari University of Venice for collaborating with bibliographical resources and logistics support. This project is funded by the Swiss National Fund under Division II, project number 205121 159961.
Giovanni Colavizza, Matteo Romanello, and Frederic Kaplan