The ingestion and digital storage of historical records has had a profound impact on scholarly practices. Yet in order for digitized archives to be used to full advantage, it is imperative that they are searchable and organized in a coherent and consistent way. These requirements are particularly evident in the case of historical records pertaining to realworld individuals: personal data is likely to be intricate and may originate from disparate sources, whose rules of data collection and data storage vary greatly. While this makes graph-oriented databases a very attractive option for storing historical records due to their emphasis on attributes and relationships, document-oriented databases may oefr similar advantages in terms of flexibility and precision of record storage. In the present paper, both kinds of database are analyzed and compared in terms of the ease of ingestion and accuracy of record presentation within the database. The historical data used was gathered from a number of diverse collections of historical records referring to persons of Irish descent who served in European armies or studied at European universities between the sixteenth and the nineteenth century.
The data hosted on the Virtual Research Environment (VRE) of the Irish in Europe Project was collected between 2001 and 2008 by researchers at the Universities of Leuven, Oxford, Toulouse, Dublin (Trinity College), Madrid (Complutense), and Maynooth. Their work was part of a coordinated eofrt to capture and host biographical material, held mainly in European archival repositories, on Irish soldiers, merchants, and clerics in Europe and the European empires in the early modern period (1550–1800). In total, basic biographical records on about 15,000 individuals were identified and harvested for hosting in a virtual research environment that would facilitate advanced querying, graphical representation, and mapping. The assembled material provides us with access to information about a specific migrant population and its evolution over a significant time scale. More importantly, within their digital environment, these sources help to deepen our understanding of early modern migrant populations in general, while serving as a template for the reconstitution of parallel migrant populations. They also open up, for the first time, the possibility of quantitative, as well as qualitative, comparative studies.
One of the most crucial issues facing the digital humanities, and the Irish
in Europe Project VRE specifically, is the ingestion of ‘unstructured’ data
into a digital database and the manipulation thereof. By its very nature,
historical data can consist of uncertain and/or divergent primary source
material, which is why systems centered on modeling and representing
personal historical data are often extensive, complex, and heterogeneous
NoSQL – commonly referred to as ‘Not Only SQL’ (with SQL standing
for Structured Query Language) – databases have been developed as a
response to the limitations of existing relational database management
systems (RDBMS). While traditional RDBMS are capable of large-scale data
management for structured data, NoSQL databases adeptly manage large
amounts of structured, unstructured, semi-structured, and hybrid data at
reduced complexity and cost
NoSQL databases vary greatly in how they store and manage data. One of
the primary distinctions is that in relational models the schema is extremely
rigid and possible relationships are fixed in advance. In graph databases,
the information is stored in a schemaless format, (key-value pairs), which
allows several related values to be stored at the same node
One of the immediate incentives for historians and digital archivists who
wish to digitize historical data is that the archives become more accessible
instead of remaining in storage with no public access. The immense changes
taking place in the past number of years have begun to reshape digital
curation and digital historiography, as detailed extensively by Sabharwal
In particular, this means detailing relationships between the information chunks of a record, as well as delineating common relationships between records in a precise manner. This is a significant challenge when compiling a database of historical data, as it is easy to convolute multiple records from diefring datasets if they are not correctly represented. What results is an interpolation problem, as it must be determined how best to reproduce intricate, heterogeneous data without loss of complexity, making an analysis of integrity constraint support vital for digital archivists.
The data being examined in this study originally formed part of a now
obsolete and no longer publicly accessible VRE website where each dataset was
modeled, and each record stored, as an intricate XML file. XML databases
could not be considered due to the volume of data, as the syntactical
redundancy of XML would have resulted in a prohibitively large transport and
storage eofrt. Moreover, the records collected have been archived in multiple
diefrent languages, which means that it is entirely possible for a military
record in the French collection, for example, to have details listed in English,
Irish (Gaelic), Latin, and French. Given that any or all of the names Sean De
Paor, John Power, or Jean LePoer could be cataloged in a record for the same
individual, querying a database for precisely one of the above would not
return results for the others. This is why a database that permits regex (short
for regular expression) queries, which allow the fields being searched to be
matched to a query pattern, is an essential requirement. XML databases are
limited in this regard, as nested tags are very dificult to manage with
regular expressions
Integrity constraints are rules that restrict the information that may be
present in the database. They play a major role in maintaining the precise
structure of a record or set of records. The constraints in SQL databases can
be divided into two main components: entity integrity, which tests the
validity of the data by providing primary keys; and referential integrity, which
adds meaningful structure to the data by using foreign keys to tie relations
together
When processing large amounts of real-world data, entities may be
represented in a variety of formats, such as JSON (JavaScript Object Notation) or
relational records, and representations may contain redundant or
inconsistent information
In order to accurately compare the performance of MongoDB against Neo4J, the research has been divided into three distinct categories: • to observe and determine the dificulty, or lack thereof, of ingesting a record into both types of database, for an individual record and a set of records • to examine the structure of the data in both sets of software, placing particular emphasis on the precision of conversion from the XML record to its reproduction within the database and the lack of dificulty of merging records between datasets • to analyze the complexity at which an individual record can be extracted from both types of database The databases were evaluated according to both shared and unique features. 1.7
The data being examined and archived comes from a variety of sources. It is diverse, real world data that is both complex and heterogeneous. The library containing the data consists of five primary components: • Brockliss & Ferté; this data was gathered by historians Laurence Brockliss and Patrick Ferté in collaboration with institutions in Oxford and Toulouse, and details Irish students that studied in Irish colleges in Europe in the 1700s and 1800s • Spain; records of Irish soldiers serving in Spain during the seventeenth and eighteenth century, collected by Óscar Recio Morales of the Complutense Institute for International Studies. • France; records of Irish regiments in France in the 1700s and 1800s, collected by Colm Ó Conaill of Trinity College Dublin • King’s Inns; records of Irish students who attended King’s Inns in the pre-Cromwellian era, archived by Brid McGrath of Trinity College Dublin Each dataset contains intricate and varied records, which can make it dificult to establish a comprehensive pattern. On the other hand, as mentioned above, the records are currently stored in XML files, and the fact that each class of data has already been resolved into elements and attributes is of great help when it comes to examining the structure of an individual record. 2 2.1
Neo4j and MongoDB are compared in terms of both speed of processing and the formation of the syntax necessary to create a query. As stated above, the records that are being processed are complex and involve nested data, which makes it advisable to create an encoding of each data library before they are absorbed by the software. The model is based on the structure of the XML ifle of each collection, and therefore no assumptions were made in transforming the records to the model produced. A parent-child model of the Brockliss & Fertè dataset is shown in Figure 1. 2.2
MongoDB (as well as Neo4j) does not use Structured Query Language
(SQL) to interact with the database, but it is compatible with a number of
languages, including Go, C++, and Python. MongoDB’s primary querying
language is JavaScript.1 The documents are hierarchical tree data structures
which can consist of maps, collections, and primitive values. MongoDB uses
the following hierarchy: database, collection, and document (Mahajan et al.,
1When commands are composed in this language, it creates a JSON document, BSON
(Binary JSON) object, or sub-documents, which are the primary components of collections
in the database
In MongoDB, records are stored in collections. These are analogous to
tables in relational databases, although the implementation and operational
concept of MongoDB and RDBMS is diefrent. Unlike in SQL databases,
the schema of a table does not have to be determined before inserting data.
For MongoDB, it is not imperative for all records within a collection to
have the same schema, which makes it possible to change the framework
2ACID stands for atomicity, which means that a database transaction must be entirely
ifnished, or it will not commit; consistency, which means that a database must remain
consistent before, during, and after the transaction occurs; isolation, which means that when
multiple transactions are executed simultaneously, transactions are processed exclusively
and consecutively – data from one transaction cannot be transferred to another when the
transaction has not been completed; and durability, which indicates that once database
transactions are registered, events are recorded to a permanent medium which will not be
modified outside a transaction.
of a schema within a collection by adding, removing, or updating fields
Boyle, I.C.P., p. 200."}}) The fields of the schema can be then adjusted for diefrent records within a collection (Figure 2).
Graph databases, such as Neo4j, provide the most sophisticated and
evolved method of data modeling, making it easy to update the schema
according to the user’s needs (
Although graph databases are constantly under development in order to
increase their stability and expand their range of features, they have not yet
reached the maturity level of other data management solutions, such as
relational databases. Nonetheless, a number of mechanisms exist which can
help to increase the applicability of graph database technology in a real-world
context. In some instances, traditional solutions developed for relational
databases can be reworked and adapted to the context of graphs, integrity
constraints being a case in point: (Šestak et al., 2021) for example, much of
the integrity constraint support available for Neo4j has been written into the
Cypher query language in a similar style to SQL (e.g. UNIQUE or NOT NULL)
In order to directly ingest documents into the MongoDB database, the mongoimport command can be used. This utility allows data to be imported from JSON, CSV, or TSV files. It is not necessary to specify a collection when importing data into the database; however, a collection will be created upon the addition of the records. Like other NoSQL databases, MongoDB has a dynamic schema design, allowing the documents in a collection to have varied fields and structures.
While Neo4j oefrs an option to eefctively import large datasets in
several diefrent formats, it recommends the LOADCSV command as the optimal
method. This facilitates the conversion from relational or other type
databases to a graph database format
Within the MongoDB database, each document is stored in the JSON
format. As JSON documents support embedded fields and nested data,
related data can be included within the document instead of having to be
stored externally within the collection. Embedded fields act as placeholders
that can be added to text fields to dynamically display entity-specific content.
Each JSON field consists of unordered key-value pairs, a form of NoSQL
database that has become increasingly prevalent in recent years, with each
pair stored in a key-based lookup structure
As mentioned previously, Neo4j has native graph storage, which means
that each part of the graph data model is stored separately. There are diefrent
notions as to what makes up the key components of a graph database, one
of them being the property graph model, which is schemaless and allows
the user to represent the data close to a real-world conception. The records
in a Neo4j graph database are structured with each entity as a node, which
are linked through directed connections named relationships. Nodes and
relationships refer to their attributes as properties
A fundamental challenge in Neo4j data modeling is classifying a
categorical variable as a property, label, or node. A categorical variable may be
described as having two or more distinct categories with no intrinsic
ordering
When searching for nodes that share a common property, or if the
cardinality of the categorical variables is inordinately high, modeling each variable
as a separate node can be the most eficient choice. Retrievability of nodes
has been the subject of extensive research for a number of decades
[:HAS]->(o:Origin)[:HAS]->(r:Residence)[:HAS]->(d:diocese)<-[:HAS]-(r:Residence) <-[:HAS]-(o:Origin)<-[:HAS]<-(j:PersonalDetails) <-[:HAS]-(other:Person)
We can see how cumbersome this syntax can be, as the levels of the graph descend from the root. Moreover, if data is too densely connected, it can result in the creation of supernodes, i.e. vertices with a disproportionately high number of relationships, which will have a negative impact on any queries that attempt to access them.
Therefore, our approach to modeling the data in Neo4j had to be tailored
to the questions the user will need to ask of the database, and an adaptable
attitude is required. Unlike traditional RDBMS, there is no significant
decrease in performance for highly connected data ingested, stored, and
navigated in Neo4j – as can be noted from Figure 3, it has no dificulty in
representing structured, semi-structured, or unstructured data (
Both types of database support a Python driver.4 Although both types of database support several diefrent methods of bulk import, some of which have been outlined in the previous chapter, this approach was deemed the fastest and most objective way of conducting a fair comparison test for multiple data ingestion and retrieval. 4.2
Retrieving data from the database in MongoDB is also done via JavaScript queries, using a simple command.
This can then be appended depending on the specificity of the results that need to be recovered. Each record in a collection will have a unique ID, which can also be accessed when the ‘find’ command is used with a single criterion.
The node always has its own variable name, an alias that is assigned by the user the first time the node is referenced within a query. This makes the retrieval of individual nodes a less complex process. However, returning an entire record is a more formidable task in Neo4j. Graph databases, on the other 4PyMongo is the recommended method of interacting with MongoDB from Python, while the oficial Python driver for Neo4j is ‘neo4j-python-driver.’ hand, are well suited for ontology-oriented data, representing a record as a set of concepts and relationships. These entities are all dispersed and stored separately, meaning that their retrieval requires a more explicit approach. 4.3
Due to the static nature of the data, updating the records would be an infrequent operation, but it is still prudent to examine the intricacy of such an undertaking. In MongoDB, the command is straightforward: db.brocklissferte.update_one( {'id' : origin.get('id') }, {'$set': {"occupationalHistory":
({"occupation": "Cleric", "role": "Priest"})}})
Indexing is paramount in MongoDB, and the ID is required in order to retrieve and update the record. Updating an individual record in Neo4j is a similarly easy task:
The situation becomes more complicated when making changes to the structure of an individual record. Although MongoDB is commonly referred to as ‘schemaless’ data, it is important to contemplate how data is stored in order to optimize database performance. If we consider the Spanish Military parent-child model, it can be observed that the framework of the record is as follows: <employment> <employer> <address>
<country>"Spain"</country> </address> </employer> </employment> <employment> <employer>
<address> In some of the records, an extra set of parameters is present, recording the details of military inspection: <country>"Spain"</country> </address> <date>1748</date> <record field="age">21</record> <record field="height">5.2</record> <record field="eyes">Black</record> <record field="hair">Black</record> </employer> </employment> In order to accurately represent this structure across multiple records in MongoDB, it is necessary to invoke the $addFields(aggregation) function, which appends extra fields into each document.
The format of a Neo4j update query is quite diefrent. A node can be created and then accessed and added to directly using the SET clause.
This is an important issue when attempting to accurately represent this data in the database: the complexity of the data is such that it is necessary to append and update the data encoding habitually in order to ensure accurate reproduction. All told, both of the command syntaxes in question provide a distinct advantage over schema migration in SQL databases, as they are less complex and more intuitive.
When considering multiple collections, as is being done in this instance,
one feature that needs to be discussed is merging multiple records on a
particular field or attribute. In database technology, these are known as
aggregate functions. The dominant aggregation framework of MongoDB uses a
pipeline concept, as it provides eficient data aggregation using native
operations. A pipeline is an array consisting of distinct operators which modifies
a collection (or sub-collection) in stages. As the collection passes through
each stage, certain operators modify the collection documents according to
various techniques
[ "$employer", "$$address" ] } } }, { "$lookup": { "from": Innstudents, "let": { "address": "$address" }, "pipeline": [ { "$match": { "$expr": { "$in": ["$place", "$$address" ] } } } ], "as": "address" }} ], "as": "address"}}])
An updated parent-child model of the Brockliss & Fertè records may be observed in Figure 4. This syntax, while looking quite intricate upon initial observation, is of great help in the consolidation of nested data, as it covers the range of each record to find and merge the relevant field. In this instance, it provides an assortment of records to the user, giving access to information regarding relevant benefactors and their level of influence in a particular area. Meanwhile, $lookup is a pipeline operator that allows the user to perform a left join to combine two collections in the same database. A left join will contain all documents from the collection referenced first, as well as the matching documents from the second collection. All unmatched documents from the second collection will be omitted. The aggregation functions in Neo4j are comparable to those in relational databases with a syntax similar to SQL. The amount of matching rows can be queried using the count(*). The collect() command returns a list of heterogeneous elements in a single list.
One major advantage NoSQL databases have over relational databases is
the increase in scalability. A highly scalable database is one which can
increase its workload and throughput when additional resources are added to
it. Scalability may be supported by the following approaches: horizontal
scaling, the process of adding more hardware to the system; and vertical
scaling, which increases the memory of the existing server. Although our data
is static, and is infrequently increased or altered, it is worth considering this
aspect to fully understand the capabilities of our database. MongoDB
supports horizontal scaling through sharding; a method for distributing data
across several servers
The comparison between both types of non-relational database yielded a number of illuminating observations and results. The primary objective was to ingest the data with as little loss of data integrity as possible. As the complexity of the data was illustrated using a parent-child relationship model, it made the composing of queries and interactions with both types of database a less arduous task. The syntax of creating this structure with both MongoDB and Neo4j was exacting. However, the fact that a Cypher command could be automatically generated in Neo4j upon the creation of a graph data model, as opposed to being painstakingly produced for each individual dataset by the user, gives graph databases a distinct advantage. 5.2
It is worth noting the contrast between the syntax of retrieving a nested data object from both types of database, especially with regards to the complexity of each query. From traversing the parent-child model in Figure 1, it can be observed that the entity ‘country’ is a child of the entity ‘residence,’ which is a child of the entity ‘origin,’ which is a child of the entity ‘personal details.’ The parent entity is described as ‘person.’ To correctly traverse the data in order to retrieve this entity in MongoDB, each property has to be labeled in the command.
This results in the following record being retrieved: {'name': {'forename': 'Thomas', 'surname': 'Williams'}, '_id': ObjectId('5e6396629578d4b117ea55fa'), 'origin': {'residence': 'province': 'Connaught', 'country': 'Ireland', 'diocese': 'Wexford'}}, 'id': 128016, 'gender': 'male', 'education': {'qualification': {'achieved': 'yes', 'date': 1747-10-22 08:47:32', 'course': 'Law'}, 'address': {'town': 'Brussels', 'country': 'Belgium'}, 'institution': 'University Of Brussels'}}
This is a more complex syntax than the Neo4j variant. As can be observed in Figure 3, a graph data model which is representative of the structure of the record within the database, the entity ‘country’ is stored as a property of the node address. This node can be directly queried by the user with the following cipher query:
The contrast in complexity between both commands emphasizes that graph-based data models provide a much more eefctive and adept method of data traversal when interacting with the database.
As stated earlier, Neo4j provides a graphical environment which allows the user to observe and examine patterns in the data. This can aid historians in gathering knowledge or forming an impression of a particular aspect of the dataset. To give an example: one of the smaller datasets being ingested in this study is the King’s Inns dataset, which provides information regarding Irish students who attended the Inns of Court in London in the pre-Cromwellian era, 1603–1633. The records are divided into four datasets, one for each of the four preliminary colleges that students could attend to gain admittance to King’s Inns in Dublin, namely Gray’s Inn, Lincoln’s Inn, Inner Temple, and Middle Temple. Each dataset has approximately 250 records. While the datasets may have slightly diefrent relationships, they all possess the structure of the property graph model shown in Figure 3.
All Irish students that attended the institution during this period were obliged to be sponsored either by their family or by an afluent member of their community. Therefore, by analyzing each dataset by area as it is absorbed into Neo4j, a pattern emerges of the most wealthy areas during this time period. The Lincoln’s Inn dataset linked students to 25 diefrent counties represented by the blue nodes, with the more prominent areas having more student nodes linked to them. This can be observed in Figure 5. When the fourth dataset was added to the database a diefrent visualization can be seen in Figure 6.
After absorbing all four datasets, 399 nodes and 359 relationships are
present in the database. Clear areas of prominence have emerged, allowing
the user to gain insight into areas of influence during this period:
NoSQL databases have grown in popularity over the last decade, with
MongoDB emerging as the forerunner for large-scale data management and
processing. The advantage of adopting a collection-based approach to data
administration is that each set of key-value pairs can be accessed in a flexible
manner. This allows a collection to be composed of similar but diversified
documents, and makes the storage and retrieval of semi-structured records
more convenient for the user. Another benefit is that it is possible to
interact with the MongoDB engine using the JavaScript programming language,
rather than having to learn and master SQL. However, this can be regarded
as a mixed blessing: SQL is a standardized programming language, designed
for processing data stored in relational database management systems.
Technically, it is less complicated to connect diefrent relational DBMS than it is
to connect relational systems with NoSQL databases
Throughout this paper, it can be observed that the graph database Neo4j
provides an elegant alternative to other NoSQL databases for users
confronted with highly complex relationships and entities. As well as providing a
platform which makes the absorption of records into the database
significantly easier, it also maintains uniformity between the structure of an
individual record and the architecture of the data within the database. With a
graph database schema, schema-instance consistency is required (
It cannot be concluded from this that traditional relational database
management systems are ideal for the digitization of all types of historical source
material. When confronted with complex and diverse real-world data,
relational systems suefr from the fact that they cannot capture their inherent
graph structure