Following the steps of previous projects such as EuDML, Mathdoc is launching its Digital Mathematics Library. Based on a reliable infrastructure made for Numdam, learning from previous projects, and relying on a network of institutions we trust, we aim to push the ball further for accessing mathematical content online. We focus for a start on the aggregation part, aiming to reach a critical mass of mathematical content by harvesting various sources: OJS instances, preprint repositories, and locals DMLs. We thus build a database of mathematical documents, linking back to the source's website for accessing content.
Basically the goal is to make a big part of the mathematical corpus available from the same place, with the best
possible metadata to facilitate searching, and interoperate with relevant infrastructures. The system is based on
• an OAI-PMH harvester to gather metadata;
• a periodic task orchestrator;
• the new Numdam platform [
As the project started a few months ago, we intend to first reach a critical mass in indexed content with a highly usable interface, so the focus is currently on the aggregation part. Later on, we will take an incremental approach to improve our DML services.
Compared to mini-DML, which was a proof-of-concept, we try to use much more detailed metadata in order to offer a better user experience, we also started to harvest from much more sources, and to adapt to more common metadata schemas. Compared to EuDML, we use more or less the same metadata, we do not have yet an API, we won’t try to revive some of its features. The main benefit of our work is to move ahead: an entirely new technology behind, a worldwide scope. Our main target audience is the working mathematician, always struggling to find a source for published references they gather from database searches, citations or colleagues’ hints. We try to make it easy to search a still highly fragmented, heterogeneous corpus. 2.1
The first thing to do for building a DML is to choose where the data comes from. There are quite a lot of available resources on the internet, and thus we must choose on which criterion we base our choice. We decided to stand on EuDML’s shoulders, which means we intend to aggregate from local DMLs that ensure • quality of the mathematical content; • long-term reliability (well-maintained systems with persistent URLs); • a usable OAI-PMH server delivering quality metadata (JATS/BITS if possible, or at least fine-grain enough to enable a decent browsing of collections).
We thus rely on a network of institutions we trust, with a common goal of archiving and broadcasting mathematical content, with sustainability rather than profit in mind.
Following on mini-DML, we decided to also include preprint servers such as arXiv or HAL, because they provide open access to a huge quantity of useful mathematics. It will be possible to filter search results so as to exclude preprints, when the user is looking for formally published material only1.
In order to maximize the number of sources, we also started to ingest content from isolated journals published with Open Journal System (OJS), when we believe that they are backed up by a trusted institution, such as a learned society or a University library. OJS instances are now shipped with an OAI server, and a JATS plugin is available, so a lot of quality items are available through this method. The challenge here will be to draw an inventory of all relevant OJS instances, and to select which are eligible with our criteria.
EuDML has done a great job of ingesting data sources with still no support of OAI protocol to this day, thus we use EuDML’s OAI server to retrieve some of its content, in order to take advantage of the former project rather than spending time reproducing what has been done. We avoid it when it’s possible though, because we want regular updates from our chosen sources, and EuDML is currently stalled. We intend to harvest anew currently frozen sources when possible.
When we have defined the source, the next step is to import the data in our database. The goal here is not to store locally a copy of the full text of articles, but to facilitate the search and link to the source for accessing the content. We build thus more a catalog than a physical library, making the choice of the source more important as we rely on it to provide content.
Our first goal is thus to break the (quite artificial) “EU” barrier in EuDML. In this first round, we leave out the more fancy stuff as we focus on making more content visible. 2.2
As outlined above, we select sources that support the OAI-PMH protocol [
As said above, this is the first step for importing data, we are prepared to support more formats and other protocols over time.
Once harvested as or converted to JATS and BITS, we clean somehow the metadata and ingest it in our
platform, which is based on the one described in [
An other big advantage of the OAI Protocol is the possibility to choose in which date interval we want to import the data, thus making the import of new data easy, and avoiding the cost of ingesting the same data again and again. Moreover, it allows us to set up automated regular update over all of our sources, bringing new content automatically on MDML. Although scheduled, this never really happened with EuDML harvester.
We also set up a log system to keep track of import, storing raw data and the source, making it possible to understand why it crashed, and how to fix it.
When we have stored the data, the next step is to present the data back to the users, and make it possible to browse the digital library, in the same way a user can wander in a physical library and browse printed volumes. Searching among digital items can be done in different manner: searching authors, keywords, title, equations or browsing specific journals or books. To be able to propose an fielded search, we need to have thorough metadata. However, the most important thing is to have a lot of items available, if we really want added value.
The implementation of searching in MDML is based on the Numdam platform, thus benefiting from an already proven tool. The searching engine is getting better over time because its core is common for Numdam and all journals managed by Centre Mersenne.
As the MDML project is tightly linked with the Numdam platform written in Python, it seemed obvious to use Python as well for MDML. To harvest and retrieve the XML from OAI sources, we use the great Sickle plugin4. We store different information about the source to harvest: OAI server url, OAI set, XML format, if it is a one shot ingestion or not (i.e. needs update or not), the type of provider and the last harvest date. Moreover, we also store what kind of processing we will do. The tricky part is that almost every source of data is different in some way, regardless of format used (JATS, Dublin Core, etc). Even if it’s a minor difference, we need to have a system with a common processing and specify only the small part specific to each source. Thus we made a sort of multiplexer of XML parsers, depending on OAI server. Each source parser inherits from the common XML parser, and we override what’s different.
Then we feed it to the Numdam based platform, with an additional layer to store OAI metadata such as OAI id and OAI source.
A recurrent task has been set up in the background with Celery5, to check regularly for new data. As said earlier, OAI-PMH allows us to specify a date interval for harvesting, so the last harvest date is stored for every source of MDML, and we update it at each new harvest.
Detailed logs are stored if any of the items harvested failed to be ingested by the platform, including raw XML and source information, then allowing us to enhance our import tasks quickly, and to do the import again. 3.2
The website is based on Django, same as Numdam and sites managed by Centre Mersenne. There is of course an additional layer as well to serve items on MDML website, but the core is common and can benefit from the work done by Mathdoc on Numdam. For instance, the platform natively supports a dual TeX/MathML description for mathematical content, with mathjax on board in order to present it correctly in most situations. The end goal here is a one click access to the article on its source website. 4
Based on the experience of Mathdoc and its various projects in the area of mathematical documents and
metadata, and all previous DML projects, the ambition is that the Mathdoc DML be a significant step forward in
terms of content covered towards the Global DML [
Figure 2: Searching for items in MDML
Figure 3: One article’s details on MDML