This paper presents an architecture for historical archives maintenance based on Open Linked Data technologies and open source distributed development model and tools. The proposed architecture is being implemented for the archives of the Center for Teaching and Research in the Social Sciences and Contemporary History of Brazil (CPDOC) from Getulio Vargas Foundation (FGV).
amount of textual, iconographic and audiovisual documents; (2) updating the dictionary DHBB; and (3) prospecting innovative technologies that enable new uses for CPDOCs collections.
The advances in technology o er new modes of dealing with digital contents and CPDOC is working to make all data available in a more intelligent/semantic way in the near future, o ering swift access to its archives. In collaboration with the FGV School of Applied Mathematics (EMAp), CPDOC is working on a project that aims to enhance access to documents and historical records by means of data-mining tools, semantic technologies and signal processing. At the moment, two applications are being explored: (1) face detection and identi cation in photographs, and (2) voice recognition in the sound and audiovisual archives of oral history interviews. Soon it will be easier to identify people in the historical images, and link them to the entries in CPDOC archives. Additionally, voice recognition will help locate speci c words and phrases in audiovisual sources based on their alignment with transcription { a tool that is well-developed for English recordings but not for Portuguese. Both processes are based on machine learning and natural language processing, since the computer must be taught to recognize and identify faces and words.
CPDOC also wants its data to constitute a large knowledge base, accessible using the standards of semantic computing. Despite having become a reference in the eld of organization of collections, CPDOC currently do not adopt any metadata standards nor any open data model for them. Trends for data sharing and interoperability of digital collections pose a challenge to the institution to remain innovative in its mission of e ciently providing historical data. It is time to adjust CPDOC's methodology to new paradigms.
In Brazilian scenario many public data is available for free, but very few
are in open format following the semantic web accepted standards. Examples in
this direction are the Governo Aberto SP [
In this sense, we present hereby a research project that re ects a change in the
way CPDOC deals with archives maintenance and di usion. The project is an
ongoing initiative to build a model of data organization and storage that ensures
easy access, interoperability and reuse by service providers. The project proposal
is inspired by: (1) Open Linked Data Initiative principles [
The project started with an initiative of creating a linked open data version
of CPDOC's archives and a prototype with a simple and intuitive web interface
for browsing and searching the archives was developed. The uses of Linked Open
Data concept are conformed to the three laws rst published by David Eaves [
Among the project objectives we emphasize the construction of a RDF [
This paper re ects a real e ort grounded in research experience to keep CPDOC as a reference institution in the eld of historic preservation and documentation in Brazil. 2
2.1
This system is composed by personal les from people who in uenced the political and social scenario of our country. These historical documents, in textual or audiovisual form, in form of handwritten and printed texts, diaries, letters, photographs, speeches or memos, represent much more than private memories: they are the registry of a whole collective memory.
Currently, more than 200 personal archives from presidents, ministers, military personal and other Brazil's important public gures compose the CPDOC's collections. Together, they comprise nearly 1.8 million documents or 5.2 millions pages. From this, nearly 700 thousands pages are in digital format and the expectance is to digitize all collections in the next few years. The collection entries metadata are stored in an information system called Acessus. It can be accessed through the institution's intranet for data maintenance or by internet for simple data query. Currently, allowed queries are essentially syntactic, i.e., restricted to keywords searches linked to speci c database elds de ned in an ad hoc manner. For those documents that are already digitized, two digital le versions were generated: one in high resolution aiming long-term preservation and another in low resolution for web delivery. High resolution les are stored in a storage system with disk redundancy and restricted access, while low resolution les are stored in a le server 4 (Figure 1). 2.2
The CPDOC collection of Oral History hosts currently more than 6.000 hours of recording, corresponding to nearly 2.000 interviews. More than 90% of it, video or audio, are in digital format. For the time being, two kinds of queries are available in the database: query by subject and query by interviewed. Each interview record holds a brief technical information and a summary with descriptions of the interview themes in the order they appear in the record. Almost 80% of the interviews are transcribed, and to access the audio/video content the user is requested to come personally to CPDOC.
Currently, CPDOC is analyzing better ways of making this data available online, considering di erent aspects such as the best format, use policies, access control and copyrights.
As in the case of Acessus, the database actually stores only the metadata about the interviews, while digitized recorded audios and videos are stored as digital les in the le servers. 2.3
The Brazilian Historical-Biographic Dictionary (DHBB) is certainly one of the main research sources for contemporary Brazilian politicians and themes. It contains more than 7.500 entries of biographic and thematic nature, i.e., people, institutions, organizations and events records carefully selected using criteria that
measure the relevance of those to the political history for the given period. The entries are written evenly, avoiding ideological or personal judgments. CPDOC researchers carefully revise all entries to ensure the accuracy of the information and a common style criteria.
The DHBB's database stores few metadata concerning each entry, and the query is limited to keywords within the title or text. 3
In this section we summarize the main problems identi ed in CPDOC's current infrastructure and daily working environment.
As described in Section 2, CPDOC's archives are maintained by three
different information systems based on traditional relational data models. This
infrastructure is hard to maintain, improve and re ne, and the information is
not found or accessed by standard search engines for two reasons mainly: (1) an
entry page does not exist until it is created dynamically by an speci c query;
(2) users are required to login in order to make queries or access the digital
les. Service providers do not access data directly and therefore cannot provide
specialized services using it. Users themselves are not able to expand the queries
over the collections, being limited to the available user interfaces in the website.
Thereupon, data of CPDOC's collections is currently limited to what is called
\Deep Web" [
The maintenance of current di erent information systems is very problematic. It is expensive, time demanding and ine ective. Improvements are hard to implement and therefore innovative initiatives are usually postponed. A relational database system is not easily modi ed, because relational data models must be de ned a priori, i.e., before the data acquisition's stage. Moreover, changes in the database usually require changes in system interfaces and reports. The whole work ow is expensive, time consuming and demands di erent professionals with di erent skills from interface developers to database administrators.
Concerning terminology, CPDOC's collections do not follow any metadata standards, which hinders considerably the interoperability with other digital sources. Besides, the available queries usually face idiosyncratic indexing problems with low rates of recall and precision. These problems are basically linked to the ad hoc indexing strategy adopted earlier to de ne database tables and elds.
Finally, data storage is also an issue. Digitized Acessus's documents and Oral History's interviews are not stored in a single place, but scattered in di erent le servers. The CPDOC database only stores the metadata and le paths to the le servers, making it very di cult to ensure consistency between les, metadata information and access control policies. 4
As discussed in Section 3, relational databases are often hard to maintain and share. Also, the idea of having in-house developed and closed source information systems is being increasingly replaced by the concept of open source systems. In such systems the responsibility of updating and creating new features is not sustained by a single institution but usually by a whole community that share knowledge and interests with associates. In this way the system is kept upto-date, accessible and improving much faster due to the increased number of contributors. Such systems are usually compatible with standards so as to ensure they can be widely used.
Our objective is to propose the use of modern tools so CPDOC can improve the way they maintain, store and share their rich historical data. The proposal focuses on open source systems and a lightweight, shared way of dealing with data. Concretely, we propose the substitution of the three CPDOC systems by the technologies described as follows.
The Acessus data model comprises personal archives that contains one or more series (which can contain also other series in a strati ed hierarchy) of digitalized documents or photos. The PHO system data model is basically a set of interviews grouped according to some de ned criteria within the context given by funded projects. For instance, a political event could originate a project which involve interviewing many important people taking part on the event.
Therefore, Acessus and PHO systems can be basically understood as systems
responsible for maintaining collections of documents organized in a hierarchical
structure. In this way, one can assume that any digital repository management
system (DRMS) have all the required functionalities. Besides, DRMS usually
have desirable features that are not present in Acessus or PHO, such as: (1) data
model based on standard vocabularies like Dublin Core [
In our proposal the data and les from Acessus and PHO systems are planned to be stored in an open source institutional repository software such as Dspace 5 or Fedora Commons Framework 6. In this article we assume the adoption of Dspace with no prejudice of theoretical modeling.
As to DHBB, its relational model can be summarized to a couple of tables that store metadata about the dictionary entries (stored in a single text eld of a given table). The actual dictionary entries are created and edited in text editors outside the system and imported to it only after being created and revised.
The nature of its data suggests that DHBB entries could be easily maintained as text les using a lightweight human-readable markup syntax. The les would
be organized in an intuitive directory structure and kept under version control for coordinated and distributed maintenance. The use of text les 7 is justi ed by a couple of reasons. They are: easy to maintain using any text editor allowing the user to adopt the preferred text editor (tool independent); conform to longterm standards by being software and platform independent; easy to be kept under version control by any modern version control system 8 since they are comparable (line by line); and e cient to store information 9.
The use of a version control system will improve the current work ow of DHBB reviewers and coordinators, since presently there is no aid system for this task, basically performed using Microsoft Word text les and emails. The adoption of such tool will allow le exchanges to be recorded and the process controlled without the need of sophisticated work ow systems, following the methodology developed by open sources communities for open source software maintenance. For instance, Git 10 is specially suited to ensure data consistency and keeps track of changes, authorship and provenance.
Many of the ideas here proposed were already implemented as a proof of concept to evaluate the viability of such environment in CPDOC. Figure 2 illustrates the necessary steps to fully implement our proposal. In the following text we brie y describe each step.
Step (1) is implemented and the relational database was exported to RDF [
9 A text le of a DHBB entry has usually 20% the size of a le DOCX (Microsoft
Word) for the same entry.
10 http://git-scm.com.
model based on RDF. The mapping from the relational model to RDF model
was already de ned using the standard translation from relational to RDF model
sketched out in [
Step (2) is planned and represents a re nement of the graph data model
produced in step (1). The idea is to produce a data model based on standard
vocabularies like Dublin Core [
Step (3) is already implemented and deploys a text le for each DHBB entry.
Each text le holds the entry text and metadata 11. The les use YAML [
Note that step (3) was actually implemented to use the initial version of the RDF produced in step (1). The code can be easily adapted to use the nal RDF model produced by step (2).
In the planned step (4) the digital les and their metadata will be imported into a DRMS. This step is much more easily implemented using the RDF produced in step (2) than having to access the original database. It is only necessary to decide which repository management system will be adopted.
The proposed work ow architecture is presented in Figure 3. Recall that
one of the main goals is to make CPDOC archive collections available as open
linked data. This can be accomplished by providing data as RDF/OWL les for
download and a SPARQL Endpoint [
Finally, it is vital to stress the main contrast between the new architecture and the current one. In the current CPDOC architecture the data is stored in relational databases and maintained by information systems. This means that 11 It is out of this article scope to present the nal format of these les. 12 In this application we used Jekyll, http://jekyllrb.com, but any other static site generator could be used. 13 http://lucene.apache.org/solr/ any data modi cation or insertion is available in real time for CPDOC website users. However, this architecture has a lot of drawbacks as mentioned in Section 3, and also the nature of CPDOC data does not require continuous updates, which means that the cost of this synchronous modus operandi is not needed. Usually, CPDOC teams work on projects basis and therefore new collections, documents and metadata revisions are not very often released.
The results obtained so far encouraged us to propose a complete data model aligned with open linked data vocabularies, presented in detail in next section. 5
More than improving the current infrastructure for storing and accessing the CPDOC's data, we would like to exploit the semantic possibilities of such rich source of knowledge. One of the ways to do that is to embed knowledge from other sources by creating links within the available data. Since much of the data is related to people and resources with historical relevance, or historical events, some available ontologies and vocabularies can be used in this task.
The personal nature of the data allows us to use projects that are already
well developed for describing relationships and bonds between people, such as
FOAF [
The RDF graph model enables also the merging of data content naturally. The DBpedia project, for instance, allows users to query relationships and properties associated with Wikipedia resources, and users can link other datasets to the DBpedia dataset in order to create a big and linked knowledge knowledge base. CPDOC could link their data to DBpedia and then make their own data available for a bigger audience.
In the same direction, the use of lexical databases, such as the WordNet [
Figure 5 shows how PHO model can be re ned. The new model uses standard vocabularies and ontologies, making the whole model much more understandable and interoperable. In the Figure 5, prov:Activity was duplicated only for a better representation. The pre xes in the names indicate the vocabularies and ontologies used: prov, skos, dcterms, dc, geo, and bio. We also de ned a CPDOC ontology that declares its own classes and speci c ontology links, such as the one that states that a foaf:Agent is a prov:Agent. In this model, we see that some classes can be subclasses of standard classes (e.g. Interview), while some classes can be replaced by standard classes (e.g. LOCALIDADE).
In this paper we presented a new architecture for CPDOC archives creation and maintenance. It is based on open linked data concepts and open source methodologies and tools. We believe that even though CPDOC users would need to be trained to use the proposed tools such as text editors, version control software and command line scripts; this architecture would give more control and easiness for data maintenance. Moreover, the architecture allows knowledge to be easily merged to collections data without the dependency of database refactoring. This means that CPDOC team will be much less dependent from FGV's Information Management and Software Development Sta .
Many proposals of research concerning the use of lexical resources for
reasoning in Portuguese using the data available in CPDOC are being carried out so
as to improve the structure and quality of the DHBB entries. Moreover, the
automatic extension of the mapping proposed in Section 5 can be de ned following
ideas of [
Finally, we aim to engage a wider community and an open-source
development process in order to make the project sustainable. As suggested by one of
the reviewers, we must also learn from experiences of projects like Europeana 14
and German National Digital Library [