The rapid growth of modern information technologies influences all aspects of human life. Companies all over the world are adopting new approaches to solve business problems, such as diverse automation, by using information technologies. Automation substitutes routine human work and noticeably increases eficiency. This research examines diferent approaches to document automation. Basic concepts of document processing using XML and existing solutions have been reviewed and a library based on LibreOfice UNO API has been designed and implemented. The library contains diferent helpers, wrappers, and processing tools to create an additional layer of abstraction. Moreover, the library is aimed at simplifying processing, working, and converting documents, which might considerably optimize a process of creating document reports generators.
A significant amount of organizations, companies, and educational institutions deal frequently
with diferent document-related processes. Eventually, the growth of a company causes a demand
on optimizing processes. Documentation generators are one of the earliest and substantial
stages of business processes automation [
According to McKinsey Global Institute [
Hospitals, as well as other organizations, work with an immense amount of documents.
According to Steve Wilson’s paper on Electronic Health Reporter website [
Whilst the described problem seems completely explored, it is not exactly so. Many existing implementations are proprietary, that is to say you could not obtain their source code easily. This leads to the fact that it is hardly possible to launch software locally for your company or set it up preferably, for instance, choose a web-server or database. Moreover, the assortment of the supported documents is usually meager and often includes only Microsoft Ofice formats. Another hot topic is privacy. If processed documents contain users’ sensitive or corporate data, you could not trust proprietary cloud services you are not able to control. Moreover, it could be simply considered illegal in some countries to transfer personal data to 3rd parties servers. Consequently, it is critical for document automation systems to allow users to have control over their data. The aforesaid leads us to the reasons why we decided to develop our own document management system as an attempt to solve the mentioned problems.
A review of scientific literature [
Let’s take a look at the proprietary document processing systems.
Hypatos [
DocuPhase [
Docupilot [
Contactbook [
Now let’s take a look at the open-source applications.
One of these is Docassemble [
M2Doc [
Summarizing this section, the reviewed systems are competitive and powerful tools. But, they have the following disadvantages: 1. Static patterns. Most tools use only one proposed pattern for fields filling. It means that only system prefix and sufix ought be used in templates. For instance, with the prefix {{ and the sufix }}, field definition would look like {{field}}. 2. Solely Microsoft Word formats support. Most mentioned systems don’t support LibreOfifce file format or other similar formats. However Microsoft products usage is not always possible or acceptable by some companies. 3. No internal converters. Sometimes it is needed to convert a document into diferent format than docx or pdf.
Thus, it was decided to design our own system for documents processing that would satisfy our needs.
Document management system needs a core document processing tool. There are a few diferent approaches in Microsoft Ofice and LibreOfice documents processing. We will overview the most popular: XML processing and frameworks.
Microsoft Ofice and LibreOfice documents are basically archives with all content inside. Most of the files inside are XML files. They represent document’s structures, styles, metadata, settings, and other configurations.
Microsoft Ofice documents (doc and docx) have their own XML-based file format developed by Microsoft, which is called Ofice Open XML (OOXML). Its structure is shown on the figure 1.
The actual content of the document is stored in the word folder in the document.xml file.
LibreOfice documents also have their own XML-based format called Open Document Format (ODF) also known as OpenDocument [15]. ODF is developed by The Document Foundation. The structure of the document is shown on the figure 2.
In this case, the actual content of the document is stored in the content.xml file. Depicted structures may vary depending on the complexity of a document. In comparison to docx document, which has three folders files hierarchy, an odt document has a similar structure but contains additional folders such as configurations.
In the case of document generating on the basis of a template with custom keywords, the keywords might be split by ofice software into diferent tags. Therefore, this approach needs additional validation and handling of the keywords parts to merge them together.
To inquire the issue let us look at a simple document that contains the following text: 1 ${KeyWord}${KeyWord2} and ${KeyWord3} ${KeyWord4} some text.
${ and } statements indicate the beginning and the ending of a keyword. All the paragraphs have the same style family, namely Calibri 11 pt. However, things appear to be more surprising in the content file. Figure 3 shows the XML representation of the first keyword.
Microsoft Word splits text into diferent w:r elements called runs. Inside each run, we can see a w:r tag that represents a text element. So one keyword in this example has 3 diferent runs with diferent parts of the keyword. The second keyword is shown on figure 4.
In this case, we have four diferent runs. The number of runs depends on the length of the keyword and diferent special symbols. The same issue may be found in LibreOfice documents.
For the LibreOfice document, we will use the same font family and font size. Right after document creation, we get the solid not split paragraphs. The XML representation of the text is shown on the figure 5.
A problem may appear after editing the document with LibreOfice editor. Let’s change the KeyWord2 keyword to KeyWord_New. The result of this replacement is depicted on the figure 6.
As a result of a slight document editing, the XML changed significantly. New elements were added and the keyword split into 2 parts, even though the keyword still has the same style. At ifrst glance, it may seem that the problem is in using the underscore character. However, to dispel this assumption, we will return the original value to the keyword. The result is shown on the figure 7.
Even after original value recovery, we still have the XML code which is diferent from the initial one. Moreover, two extra text:span elements appeared. In the case of the LibreOfice documents, text:span elements may be added as a consequence of updating or text changing within the document.
Another approach is using LibreOfice UNO API. LibreOfice provides Universal Network Objects, which allows using this API in diferent programming languages, such as C++, Java, Python, Perl, C#, JavaScript, and many others. This API supports working with diferent formats, originally LibreOfice applications, but partly including support of Microsoft Ofice applications. As a matter of fact, LibreOfice UNO API is almost completely compatible with OpenOfice.
LibreOfice has a Frame-Controller-Model paradigm (FCM) that is similar to the Model-ViewController paradigm (MVC) [16]. The model contains the document data and methods to change them. The controller views the status of the documents and manipulates screen presentations. The frame contains the controller and knows which windows are being used. This approach allows interacting easily with the application’s GUI and its functionality.
LibreOfice UNO API is extremely functional and useful in document manipulation. However, API documentation is bulky and might be time-consuming to read [17]. Due to this fact, we decided to develop a library as a layer over the LibreOfice UNO API.
Returning to the split issue in XML documents, LibreOfice UNO API allows one to use GUI and work with text in a simpler manner. It handles text as though it had been edited by user. In addition, in comparison with the XML approach, this API provides access to styles and other functionality, like pictures, converters and other GUI functions.
We have chosen the Java programming language to work with the LibreOfice UNO API. Our library provides an abstraction to process documents easier in comparison with UNO API, and it does not require knowledge of the LibreOfice UNO API. As a part of this library, we have implemented classes for XML manipulations. In more detail, this library will be discussed in the next section.
The easier document processing approach is XML processing. It allows developers to implement a simple keywords replacement. On the other hand, LibreOfice UNO API provides a rich set of functionality for document manipulation. Nevertheless, it does not nullify the usefulness of the XML approach. A combination of two diferent approaches allows choosing developers which one is the most appropriate for their application. Usually, one ought to use two diferent libraries or frameworks to implement it, but our library provides a simple interfaces to interact with both solutions simultaneously.
Our library’s purpose is to simplify access to the documents and their handling by providing an additional abstraction. The library has been implemented using the Java programming language. The source code may be found at [18].
The XML approach is quite simple to use. The main class is OdtDocumentPatternsAdjust. It has two constructors. The first one is empty, and the second one with a Pattern parameter. The Pattern class is a JavaBean class with two fields, the start of the pattern and the end of the pattern.
The OdtDocumentPatternsAdjust class implements the DocumentPatternsAdjust interface which has two methods for adjusting the XML content. The methods are the following: String adjustPatterns(File archive)
The actual processor of the XML content is the OdtXmlPatternAdjustProcessor class. It contains diferent methods for content processing, most of which are private. One of the public method is processXml. The algorithm of XML content processing is the following: 1. Get the position of the start and the end of the pattern. 2. Set ofset to the position of the start of the pattern. 3. Get text before the next tag. It is needed to get the part of the pattern before there will be the next tag like w:r or text:span. 4. Move ofset by adding the length of the found part of the pattern. 5. Look for the next possible part of the pattern meanwhile skipping tags without actual text inside. 6. When the next part of the pattern is found, get the text. At this step, the text will be extracted and inserted into the beginning of the pattern in the XML content. 7. Check whether the ofset is less than the position of the end of the pattern; if it is, then repeat every action starting with step 5, otherwise the next step. 8. If the next part of the pattern could be found, repeat every action starting with step 1, and add the earlier found pattern into the ArrayList, otherwise return the list of the pattern.
The LibreOfice UNO API part is larger and ofers richer functionality. There are a few essential classes. First of all, consider the DocumentManagerProvider class. This class is a Factory and provides the implementation of corresponding DocumentManager depending on file extension. This class contains one static method called createDocumentManager and has the following signature: DocumentManager createDocumentManager(File file)
DocumentManager is an interface that provides an ability to open a passed document. It has the following method: Document openDocument(File file);
The openDocument method returns a Document instance, which is also an interface. This approach allows avoiding specific implementations for a developer. The Document class contains the following set of methods: void saveDocument(File file); void saveDocument(String filepath); void saveDocument(); void saveDocumentAs(File file, DocumentConvertTypes convertTo); void saveDocumentAs(String filepath, DocumentConvertTypes convertTo); void saveDocumentAs(DocumentConvertTypes convertTo); void replace(String search, String replace); void close();
These methods allow converting documents to any supported format and replacing a particular value in a document. The LibreOfice UNO API supports a considerable amount of formats to convert. All of them are described in Apache OpenOfice Wiki [ 19]. Partly, those types had been moved to our library and stored in an enum called DocumentConvertTypes. We decided to use enums to simplify the usage of constants that can be used as properties. In comparison with final static variables, enums make it easier to specify what should be passed there.
To provide more functionality, a lower abstraction layer is available. The LibreOfficeUnoManager contains most of the implemented methods in the Document class. This class provides basic methods to interact with documents without direct work with UNO API. As return values, it uses API’s objects, so it may be considered an additional functionality layer.
There are small utility classes which might help in working with documents. Nevertheless, developers will rarely use them because most of the LibreOfficeUnoManager methods already have been optimized with the use of those utility classes. Let us look into two useful classes. The OdtDocumentProperties provides a wrapper for the PropertyValue class to simplify working with document properties. The OdtFilePathHandler helps to convert the initial File class into an understandable LibreOfice UNO API string. The reason for OdtFilePathHandler class existence is that LibreOfice UNO API works with Uniform Resource Identifier (URI). This means that the file path should be started with the file:/// prefix and all backslash characters should be replaced with the slash character.
The next example demonstrates a basic usage of our library to replace keywords in the document and convert it into an appropriate format: File file = ResourcesManager.getResourceFile("Document.odt"); DocumentManager documentManager =
DocumentManagerProvider.createDocumentManager(file); Document document = documentManager.openDocument(file); document.replace("{Search}", "Value"); document.saveDocumentAs(new File( "C:/Users/hp/IdeaProjects/XmlDocumentProcessing/File.docx"), DocumentConvertTypes.MS_WORD_2007_XML);
In order to work with text, we implemented a few specific classes. The LibreOfficeUnoManager class supports working with text using the findAllAsText method. The method’s signature is the following: public List<Text> findAllAsText(String search);
This method returns a list of Text classes. The Text class supports text editing, creating cursor, getting all paragraphs, setting font weight, and paragraph adjustment. The list of the methods is shown on the Figure 8.
An example of getting a text and performing some basic operations is shown below. Text allDocumentText = libreOfficeUnoManager.findAllAsText("and").get(0); allDocumentText.createCursor().gotoStartOfTheSentence(true); allDocumentText.setCenteredAdjustment();
The Cursor class is basically usual graphic cursor. In order to move through the text, LibreOfice UNO API implements cursor as a main mechanism for this purpose. But, considering the fact of complexity of some original UNO API methods, we have implemented a simplified wrapper class. The list of its methods is shown on the figure 9.
The names of most methods, such as gotoNextSentence, are intuitively recognizable. Every type of goto moving has two diferent implementations. One does not have parameters and another one has a Boolean parameter. The Boolean parameter is used for telling the LibreOfice UNO API, whether should we stop and select current word or go to next one. Methods without parameters basically just use methods with parameters by passing false to them.
Also, to implement a more convenient way of Cursor class methods usage, goto methods take advantage of Builder design pattern. The example of such use is shown below. allDocumentText.createCursor() .gotoStartOfTheSentence() .gotoNextSentence() .gotoNextWord() .gotoPreviousWord();
It is impossible to predict diferent components usage due to LibreOfice UNO API complexity and massiveness. So, to simplify it for developers, all the classes contain corresponding methods which return the original LibreOfice UNO API objects. For instance, the Cursor class has getTextCursor, which returns a XTextCursor object.
In order to demonstrate the developed library usage, we implemented a cloud-based system which aims to automate document flow.
The application is divided onto frontend and backend parts. The development stack is shown below: • Server development stack: Spring (Spring boot, Spring Security, Spring WebFlux, Spring JPA), jjwt (Java JWT: JSON Web Token for Java and Android), Connector/J (Mysql Java Connector). • Client development stack: Vue.js 3 (Vue Cli, Vue Router, Vuex, Vue i18n, Vue Class Component, Vue FontAwesome, SFC, Element Plus), Typescript, Javascript, Babel, Webpack.
The backend has microservice architecture. In order to minimize the application load, we have implemented 3 diferent microservices: 1. Microservice for login and token generation.
2. Microservice for document processing (storage and document management). 3. Microservice for generating documents according to the data.
As a matter of application security and microservice communication, we have used the JWT token as the most eligible.
For signing up and signing in into the application, the login page may be used (figure 10).
After this procedure, the user goes to the main page for handling documents, which is called Document Management (figure 11).
This page contains all the document information. To create a document, one should push a side bar button which leads to a document adding page (figure 12). The index of the documents is shown as a list, and each item has two diferent buttons: 1. Generate Form. This button is responsible for form generating. These forms may be used as data origins for producing documents from templates. 2. Delete. Remove the entry.
Furthermore, our system supports custom template patterns, which means that documents may contain any kind of keyword distinguishers.
The forms are common way of document generating from an uploaded template. All created forms are displayed and might be changed in the Form page (figure 13).
The actual form page, which may be accessed by using the View Form button, contains all of the extracted from the template document keywords. The example of a form is shown on the ifgure 14.
Considering the fact that key words are not always named human-friendly, it is also possible to change their display name using the Edit button on the table. After submitting a form, the user automatically receives the document.
At the moment, the following features have been implemented:
We are planning to implement a cloud-based interface for working with documents without coding. In addition, we have intention to provide the richest functionality for working with LibreOfice UNO API.
Document processing may be complicated and confusing. The XML processing is more complicated and limited. The reason is that handling raw XML is dificult, especially when a document is massive.
LibreOfice UNO API is one of the richest open-source APIs for processing documents. It provides the necessary functionality to edit and process documents. In comparison with XML processing, this approach is more advantageous. Moreover, the LibreOfice UNO API solves the keyword splitting issue, or to be more precise, allows avoiding it.
The developed library allows one to handle both of the described processing approaches. It is easier to combine them regardless of whether you only need to process patterns or additionally edit the inner structure of the document. Looking at the future, we are planning to complete the development of this library. Converters of the library are useful tools because there are not many solutions that could manage all the major formats, such as doc, docx, odt, html, and others. As an application of this library, we are currently working on creating a cloud-based document management system that will be able to help in storing, handling, and processing documents. It is going to be discussed in the further reports. [15] Wikipedia, Opendocument, 2021. URL: https://en.wikipedia.org/w/index.php?title=
OpenDocument&oldid=1025760709. [16] Apache, Frame-controller-model paradigm in apache openofice, 2021. URL: https://wiki. openofice.org/wiki/Documentation/DevGuide/OficeDev/Frame-Controller-Model_ Paradigm_in_OpenOfice.org. [17] A. Davison, Java libreofice programming, 2021. URL: https://fivedots.coe.psu.ac.th/~ad/ jlop/. [18] CodePsi, GitHub - CodePsi/Lycorse-DPL: Lycorse Document Processing Library, 2021.
URL: https://github.com/CodePsi/Lycorse-DPL. [19] Apache, Framework/article/filter/filterlist ooo 3 0, 2021. URL: https://wiki.openofice.org/ wiki/Framework/Article/Filter/FilterList_OOo_3_0.