This article describes the methods of cross-language migration from the PHP programming language to the Python programming language on the example of modernization of the search module of the system for complex analysis of poetic texts. The choice of language was determined by the need of integration of the module in the system, and by the fact that Python has a large number of open libraries that make it possible to work with machine learning technologies, what is an advantage in the intensive data processing. The main problems of cross-language migration are considered and the methods to solve these problems are proposed. The proposed solutions concern the data security, the separation of module representation from the data model, the work with database via declarative mapping. In the process of cross-language migration, the search module was rewritten and all problems were solved, what allowed to integrate it into the information system for complex analysis of poetic texts.
In the practice of the development of the information systems, there are a number of problems associated with the need of the modernization of the source code of an application that is already in active usage. This can be caused, for example, by the migration of an information system into a new programming language. It is necessary to correctly reproduce the work of all parts of the old code so that there is no noticeable difference in the code execution time for the end user.
This article describes the features of migration of a software product that works with intensive data from the PHP programming language to Python.
The Python programming language, together with the Flask framework, is used to
develop the applications in various scientific fields without any restrictions, for
example, the creation of the framework for Covid-19 diagnostics [
Let's describe the technology for solving this problem on the example of one of the modules of the information system for the automated complex analysis of poetic texts.
The data on poetic texts which are extracted in the process of their multilevel
analysis (phonetic, metro-rhythmic, strophic, lexical, etc. characteristics) in accordance with
the approach described in [
In accordance with the modern realities of the software development, there was a need to modernize this module, which works with the corpuses of poetic texts of large volume.
To achieve this goal, the following tasks were set: ─ to investigate the correctness of the considered software; ─ to detect the problems that exist in the initial code; ─ to substantiate the choice between updating and rewriting of the code; ─ to substantiate the choice of the final programming language; ─ to define the architecture of the application; ─ to develop a new version of the software product. 2
The purpose of the development of the information system for the automated complex
analysis of poetic texts [
The described information system (IS) is a complex of separate modules, each of which is an independent software product that performs a specific task and, at the same time, is integrated into the system.
One of the modules of the system, which is responsible for processing of big data in
the database, was written using the PHP programming language, while the Bootstrap
framework was used to organize the user interface. The code provided the solution to
all given tasks, namely, the interaction with the database and the generation of the user
interface. PostgreSQL was chosen as the database management system. This relational
DBMS, although not traditional when it is used in conjunction with PHP, has the ability
to analyze the main linguistic objects (token/word/phrase/sentence/text), what is in
demand when processing text information [
However, the further development of the system is carried out in the more universal programming language Python 3, what leads to the problem of the integration of the modules written in PHP with the new modules.
During the analysis of the initial code, the following problems were found: ─ the complexity of the code, which causes a lot of internal dependencies, what leads to low scalability; ─ the inseparability of the visual representation of data and the data model; ─ critically low level of the security of IS.
Thus, despite the fact that the modules written in PHP are functional, they need to
be improved: if the system is functioning, but it is difficult to maintain it or expand it,
then, according to general theoretical guidelines [
There are two approaches to improving the initial code: the refactoring and the complete rewriting.
The refactoring is the improvement of the initial code without changing the result of
its execution [
A complete rewriting of the code has the same effect as refactoring, but it is usually used either when the adding of new functionality within the old concept is unnecessarily time-consuming (for example, due to incompatible architectures), or when there is a need of the migration to another programming language. In addition, a complete rewriting carries the risk of adding the new errors that may not be obvious.
Also, regardless of the chosen approach, it is necessary to test the resulting product during the development process, what allows to find the errors in the operation of the modernized code and to solve them quickly, without having to return to the problem code after a while.
To choose between two different approaches, it is necessary to analyze the initial code of the application for problems and their severity. After that, it is possible to make a conclusion about the need for refactoring or rewriting of the code.
In addition, if there is a need to choose between different programming languages, the differences between the application deployment in probable programming languages should be determined. 2.2
Let's analyze the emerging problems in the initial code.
The user interface of the module is written in HTML using the Bootstrap framework, which includes CSS and the JavaScript programming language, while some of the search fields contain dynamic parameters that the interface requests from the database during the initialization.
Let's consider the part of the code under discussion that is responsible for the dynamic construction of the "Author" field. The program code presented below makes a request to the database, and, returning the values, generates a field of the select type using them, which is then displayed as a drop-down list of authors. <div class="form-group"> <label class="col-sm-2 form-check-label">
<input type="checkbox" id='authorCheckBox' name='authorCheckBox' class="form-check-input" onclick='activateAuthorsFieid()'>Автор</label>
<select class="selectpicker" multiple id='authorsList' name='authorsList[]' data-live-search="true" data-max-options="10" data-actions-box="true" data-width="auto" title="Ничего не выбрано..." data-size="5"> <?php
$select_authors = "SELECT \"LASTNAME\", \"FIRSTNAME\", \"MIDDLENAME\" FROM \"AUTHOR\";"; $authors = pg_query($select_athors); while ($authors_list = pg_fetch_array($authors)) {
$author_string = $authors_list[0]."
".mb_substr($authors_list[
echo '<option>'.$author_string.'</option>';
} ?> </select><br><small class="col-sm-6 form-text textmuted">Выберите автора из списка или воспользуйтесь поиском</small> </div>
This example shows that the work with the data is not separated from the formation of the interface. In particular, if there is a need to call the same query in another part of the system, for example, to further access to this data, it should be copied. Thus, the procedural architecture shown in the example leads to code redundancy, and, as a result, to the possible errors. 2.3
Another problem concerns the analysis of the level of data protection.
The analysis of the protection level of the search module for poetic texts consists of the following actions: ─ the analysis of the initial code of the application; ─ the analysis of compiled SQL queries for the vulnerabilities such as SQL injection; ─ the implementation of an SQL injection attack.
The SQL injection is an attack for which an attacker inserts the deleterious code into the text of a database request, what makes it possible to execute an unauthorized request. In this way, a hacker can get stored data or cause damage to the database.
The main reasons for the possibility of SQL injection are the incorrect processing of
special characters and data types, as well as the dynamic query construction [
There are many ways to implement such attacks, as well as to protect against them. One of the main methods of leveling and preventing the potential threats is to check the correctness of the data which is entered from the client to the server. This method is implemented through the special characters escaping, the strict data type conversion, and the prepared or parameterized queries.
However, during the analysis of the application source code, it was found that there is no verification of the correctness of the entered data, requests are made dynamically, and the data is transmitted to the DBMS in the form in which they were sent by the user, without any processing of special characters or restrictions on data types.
To confirm the presence of the vulnerability, the attack was made by injecting "1 OR 1=1", entered in the filter field by metrorhythmic statistics, regardless of the selected metric. If the application works correctly, such a string must lead to no search results for all metrics, but instead, the entire corpus of poetic texts available in the database is displayed. Accordingly, the SQL injection attack is successful, and the user may receive incorrect data. 2.4
Today, the Python programming language is more popular than the PHP programming
language. This is indicated by the TIOBE index, where Python ranks second and PHP
ranks ninth [
Both languages offer a large number of frameworks for development, what allows to achieve fast, secure and well-scalable applications. However, Python has a large number of open libraries that let to work with machine learning technologies in a web environment, what is an advantage when working with big data.
Python is a language with strong implicit dynamic typing, and PHP is a language
with weak implicit dynamic typing. [
Since Python is focused not only on Web development, for the interaction of a
Python program with a server, it is required to use the WSGI interaction standard [
PHP was not created as a programming language and was originally a scripting language embedded in html-pages, so the separation of data from the interface in PHP is a non-trivial task. In turn, Python is a general-purpose programming language, so the separation of data and interface is achieved by simple methods. 3
Taking into account the above, the decision about the rewriting of the module in Python 3 was made.
The rewriting of the module separates the data model from the module representation, and also allows to solve the scalability problem and to use the data model in other tasks. The module retains its isolation, what improves the possibility of its usage in the further development of the system.
As the basis of the system, the Flask framework [
The strong side of the Python language is the number of libraries and tools available
for it for both data processing and web programming, what distinguishes it from other
languages, such as PHP. The strong side of the Flask web framework can be called its
easiness, since it requires a small amount of code to get both a web application and a
simple web-API from the functions and modules which are written in Python. [
The database that the described module works with is described in detail in the article
[
Let's consider the ER diagram of the poems database [
By initializing all the entities which are presented in the diagram as data models in the SQLAlchemy ORM, the universal tool for working with the database is get. This will allow to use the data from the database not only for search, but also for other purposes that may be required for the implementation of certain functions of the system. 3.2
the SQLAlchemy Object-oriented Paradigm used in the Flask
Since the tables in the database already exist, the task is to declare each table as a class
in accordance with the SQLAlchemy syntax [
The complexity of this declarative mapping is that for each author represented in the database, a set of tables is created according to its unique identifier, where the identifier serves as a postfix. In this case, the tables with the same structure store the data from different authors. The description of each such table as a class separately from each other is not an effective solution, because there is a code redundancy. It was decided to create a function with a numeric parameter that returns a class object depending on the specified parameter – the unique identifier of the author. This solution allows to avoid a redundancy, since there is no need to add a description of a new class to the database when a new table is being created, and it is possible to create the queries for each author dynamically without the need to modify the initial code of the program.
After declaring the classes, the tables can be used to display the information in the user interface, as well as to search the tables by the specified parameters. 3.3
Since the Flask software environment uses the Jinja2 template engine as a tool, the interface templates should be prepared. All dynamic elements in the template use the rules adopted in Jinja2, and the static part is designed to match all the rules for creating the web documents.
During the template design process, at the analysis of the previous version of the interface, it was found that parts of the interface can be divided into separate templates. The separate documents were created, representing various parts that are common to most pages: the set of meta tags, two navigation panels (a general and for DB queries), and the error panel. If necessary, these templates are embedded in other html documents. In particular, the set of meta tags and the general navigation panel are embedded in any html-documents, while the navigation panel for database queries is embedded in the template for searching the database, the template for viewing saved user queries, and the template for viewing all the works available in the database. In addition, the pages for user registration and authorization were rewritten. 3.4
The Flask software environment uses the concept of "blueprints" to create the various components of the application. The blueprints are used to expand the application, and they can also be applied to divide the code into logically related sets of operations.
The operations in the described module can be divided into two sets: the functions for working with the data of the corpus in the database and the functions for working with the user's personal data. Based on this, the blueprints 'main' and 'auth' are written, respectively.
The blueprint “main” is a set of the following operations: ─ poems_list – the output of all the poetic texts available in the database; ─ index – the output of the main page of the interface; ─ search – the output of the form used to make a query in the database for data sampling from the corpus; ─ search_result – the output of query results to the database; ─ save_search – the saving a specific DB query for the user; ─ saved_search – the output of all saved queries for the user.
The blueprint "auth" combines the following operations: ─ signup_post – the creation of a user, ─ login_post – the authorization, ─ signup – the view of the user creation page, ─ login – the view of the authorization page, ─ signout – the exit application.
The save_search and saved_search operations are only available for authorized users, just like in the old version of the application. The difference is that when the code was rewritten, the Flask Login library was used, which automates the process of creating of a user session, and in the PHP version of the application, the session was created manually. At the attempt to open the pages corresponding to these operations without authorization, the user sees an error message and a request to log in.
The search_result operation has been split into the separate functions, since the processing of a user-filled form and the building of a query are different operations. The function of the building of a query to the database is also divided into component parts, depending on which part of a query is being built, since the determination of the columns of the table from which the selection is made and the determination of the data filtering parameters can also be understood as two different actions. 3.5
The end result of cross-language migration of the application is a code which is completely separate from the presentation of html pages, and which dynamically builds the queries to the database depending on the parameters set by the user. The application structure has been rebuilt (Figure 2). 4
The technological features of cross-language migration between PHP and Python were identified on the base on the solved tasks.
Compared to the PHP language, Python does not allow does not allow to embed the code in html pages, so if there are similar pages in the project, it becomes necessary to isolate the logic inherent in the embedded code. The setting up of the correct and secure database queries is done in simpler ways than in PHP, where it may be non-trivial. The Python scripts and the related frameworks may not be supported by default on the using web server, and if there is no opportunity to use another web server, it is necessary to make additional settings.
There also may be the cases where the different data types are mixed in the initial code of a PHP application and the implicit but valid conversions occur. In such situations, when an analog of the application in Python is being written, it is necessary to have a clear idea of what data is used in a particular situation, and to apply a secure data type conversion. This imposes the certain limitations related to error processing: it is necessary to convert all the data sent by the user.
Acknowledgement. The study was carried with the support of the Russian Science Foundation (project No. 19-18-00466).