-The paper considers approaches to the improvement of one of the steps of the algorithm used for the automated determination of rhyme in poetic texts. The automated rhyme detection tool is one of the modules of the system of complex analysis of poetic texts. In the current module implementation, the rhyme search and definition subtask are solved by finding words with consonant endings using the A. A. Zaliznyak Grammar Dictionary of the Russian Language and the basic rules of phonetic analysis. Alternative solutions to the search problem in the dictionary of words with consonant endings are proposed. The results obtained will allow us to conclude that the current implementation is optimistic and the methods used can be finalized to solve the problems of determining the rhyme of a poetic text.
I. INTRODUCTION
One of the tasks of the automated complex analysis of
poetic texts [
The problem solution of automated analysis of poetic texts requires the adaptation to various languages. The different approaches are caused by both the specifics of the language (in particular, the features of the construction of poetic texts) and the tools used by researchers. The toolkit, in turn, depends on the goals set by the researchers (for example, to obtain the confirmation of any regularity in the structure of the poetic text), and, to some extent, on technologies that were relevant at the time of the study.
As for the linguistic versatility of instruments, it is
impossible to develop a system for automated analysis of
meter and rhythm, designed for a wide range of languages.
Moreover, the insoluble task is to develop a metrorhythmic
analysis system suitable for at least a group of related
languages — each language requires the development of its
own approaches that take into account its structure. The
authors of the study [
The problem of analyzing the metrorhythm of poetic texts for each language (or a group of the similar languages) is obtained differently. Next, we will consider some of the projects of the authors who solve the indicated problem for different languages.
D. Fusi in studies [
The author [
B. Navarro proposed a tool that studies the metrics of
Spanish sonnets and performs semantic analysis of poems [
The system [
text preprocessing (conversion to XML format); extract words from a poem; finding a stressed syllable; division into syllables; phonetic transcription forming (using an independent dictionary); selection of transcription options for each poem (determination of the rhythmic scheme); an attempt to determine the metric of a poem; search for matching metrics based on the most appropriate rhythmic scheme; splitting a work into syllables according to metric.
The results of the analysis [
M. Agirrezabal et al. [
R. Ibrahim and P. Plecháč [
A number of works are known devoted to the analysis of
versification for Arabic and similar languages. A. Kurt and
M. Kara [
For the Russian language, a number of solutions during
metrorhythmic analysis problem are also proposed. In study
[
The authors2 developed an open network resource, which
is represented by the components: the problem-oriented
“Poetology Thesaurus” and the “Block of Analysis and
Specification” of the text objects. In the “analysis and
specification” block, two sets of tasks are identified [
Among the tools that execute metrorhythmic analysis, web resources 3 , 4 are of interest. The first of them, Rifmoved.ru, is positioned as an supporting tool for the analysis of poetic text, which determines the stanza and the forms (sonnet, sextine). The algorithms were developed on the basis of the author’s concept of program poetry analysis by V. Onufriev, however, a theoretical description of these algorithms was not found in scientific sources. The authors of the resource indicate that the work of the algorithms is designed to analyze poetic texts written in traditional forms, classical stanzas and sizes. This fact greatly limits the usage of the tool for large corpuses of texts of poets who are not related to classical literature.
The second resource, the Neogranka.ru, obviously, is an amateur web portal for determining the poetic size, generating new poems and selecting rhymes. When a user tries to determine the verse size, the service clarifies all controversial situations (accentuation options), what takes a lot of user time. There is also no theoretical description of the algorithms used in available sources.
It is important to note that almost all the algorithms
mentioned above are aimed to study relatively small text
corps covering the work of one or more authors, therefore,
the speed of rhyme determination algorithms is not a critical
parameter. However, in the research conducted at the Federal
Research Center for Information and Computational
Technologies (FRC ICT), it is planned to study the
interdependence of the phonometric and lexical-thematic
levels of poetic texts with the aim of identifying and
measuring the relationships of semantic associations
described on the basis of semantic fields with poetic sizes;
the so-called textures that take into account the construction
and metrorhythmics (a detailed statement of the problem
described in [
A web application has been developed at FRC ICT 5 ,
which is used to analyze the structural level of
Russianlanguage poetic texts. The algorithms are described in [
To identify the phonetically matching endings, the data
about endings from article [
The purpose of the study is to find for alternative approaches to search for rhyming lines in the database and conduct a series of experiments to find out the most effective method for usage in the algorithm. The proposed solution: 1) To build a table with inverted rows sorted lexicographically.
2) To separate all words into sections by ranges after endings. In other words, the store endings (inverted) separately from the word (as metadata).
3) To add the trigram symbolic indexes to the original table with all the words.
4) To perform an experiment with the aim of find rhyming words using sections (search only endings) and trigram indexes.
5) To compare the performance of a section search option using indexes or a combination of these options.
To test the hypothesis about the effectiveness of application of the trigram symbolic indexes, it was decided to conduct an additional experiment to measure the performance of SQL queries using the indexes in the search module of the complex analysis of poetic texts. This module solves the problem of searching for low-level characteristics (for example, metrorhythmic statistics) and high-level characteristics (for example, genre-style affiliation). When a search query is done, SQL queries to the database are generated, some of which include a search by values of the varchar and text type. The execution time of such queries can be reduced by using the symbolic indexes. 5 Analysis of poetic texts online. http://poem.ict.nsc.ru/ 6 Big Rhyme Dictionary. http://rifmovnik.ru/docs.htm
One of the options for the search implementation is the partition of the source table with words into sections. The version of the PostgreSQL database deployed on the FRC ICT server supports simple partitioning: the splitting of one large logical table into several small physical sections7. The benefits of the usage of sections:
When queries or updates access a large percentage of a single partition, the performance can be improved by taking advantage of sequential scan of that partition instead of using an index and random access reads scattered across the whole table.
The bulk loads and deletes can be accomplished by adding or removing partitions, if that requirement is planned into the partitioning design. ALTER TABLE NO INHERIT and DROP TABLE are both far faster than a bulk operation. These commands also entirely avoid the VACUUM overhead caused by a bulk
Seldom-used data can be migrated to cheaper and slower storage media.
PostgreSQL supports partitioning via range partitioning (for example, one might partition by date ranges) and list partitioning − the table is partitioned by explicitly listing which key values appear in each partition. In this study, the list partitioning is used, where the ends of the dictionary words are indicated as key values.
To create a list of sections in form of tables, a Python script is used that operates by the following algorithm: 1) The request to a table with words. 2) The selection of the N-last characters from the word. 3) The formation of an array of all dictionary endings. 4) The counting and sorting the usage of each ending in descending order.
5) The separating of M-first endings from the array, on the basis of which the sectioning will be performed.
As the last N characters, four characters are taken, this value can be changed in the future. To build a sorted dictionary, we use the collections module of the Counter library8. The result is a dictionary of the following structure:
Counter({'НОГО': 86077, 'ЕЙСЯ': 76978, 'ВШЕЙ': 76400, 'ИМСЯ': 62934, 'ШЕГО': 61719, 'ГОСЯ': 57630, 'ИХСЯ': 57617, 'НОМУ': 57354, 'ВШИМ': 57282 ...})
In the received dictionary, the key is the desired ending (last N characters), and the value is the number of occurrences of this ending. It was decided to isolate the values of endings with coefficients included in the 90th percentile from the created dictionary. These endings were used to create the sections.
The process of the creation of partitioned tables includes the following steps: 7 PostgreSQL: Documentation 9.4: Partitioning. https://www.postgresql.org/docs/9.4/ddl-partitioning.html 8 Collections — Container datatypes. https://docs.python.org/3.7/library/collections.html
1) To create a parent table whose properties inherit all
the child tables (sections). The parent table is a table with
words from the dictionary of A. A. Zaliznyak [
d) word type;
As an additional column, the endings (last N characters) of each word in inverted order are added.
2) To create the child tables with inheritance of parent structure. In these tables there will be no additional columns except legacy ones. All child tables will be called the sections.
3) To add the restrictions to the section tables that define the valid key values for each section. The restrictions do not overlap — no key values apply to several sections at once.
4) To create a key column index for each section. In this study, the indexes were created for the “word” column.
5) To define a trigger to redirect data added to the main table to the corresponding section. Created trigger is work out when SQL command INSERT is run.
The created trigger launches a function that adds values to the corresponding section (table). Fragment of the function:
words_with_reversed_endings_insert_function()
IF (NEW.ending = 'нии') THEN
The creation of tables, indexes, trigger and function is performed through a Python script in an automated mode. The manual adjustment of table and index names is required, since transliterated ending names were used for naming — some cases required the manual intervention (transliterate9 is used). These cases include, for example, the coincidence of names during transliteration of the endings “ЕМСЯ” and “ЁМСЯ”.
In the context of a PostgreSQL database, a trigram is a group of three consecutive characters. We can measure the similarity of the two lines by counting the number of matching trigrams. This simple idea turns out to be very effective for measuring the similarity of words in many 9 Transliterate – PyPi. https://pypi.org/project/transliterate/ natural languages, as well as for solving related problems, such as, for example, fuzzy search (search by similarity).
PostgreSQL supports two types of text indexes10: GIN
(Generalized Inverted Index) and GIST (Generalized Search
Tree), which provide a work with symbol trigrams, what is
prerequisite for using GIN, which operates the lexemes by
defaults. Despite the fact that the GIN by description is very
similar to the experiment with inverted strings, GIST also has
a basis for application: its tree-like structure increases the
completeness of search results by including inaccurate hits,
which is quite suitable for the rhyme search task, since the
table from the work of V.M. Zhirmunsky [
As part of the search module for a comprehensive analysis of poetic texts, it is possible to add text indexes to solve the following problems: search by accentuation mask; search by words from the name and text.
The corpus of Pushkin’s works was loaded to the database; the main tables with the texts of works and their metadata contain data with a volume of more than 700 rows. The search query includes not only the direct solution of the above problems, but is also adapted for the user to understand: the response array includes additional entries, such as the author’s full name and title of the poem, i.e. the request is composite. During the experiment, the query runtime of processing additional parts of the request are not taken into account.
It is supposed to conduct the following experiments with the search for rhymes in corpuses of the PostgreSQL database:
1) To search for the desired ending among the section names: SELECT * from pg_catalog.pg_tables where %section name conditions%. It is worth noting that only in this experiment the previously inverted lines described above are used.
2) To search the endings by the incomplete match of LIKE on a table without indexes.
3) To search the endings by the incomplete match of LIKE on the table with the constructed GIN index by symbol trigrams: CREATE INDEX trgm_idx ON test_trgm USING
4) To search the endings by the incomplete match of LIKE on the table with the constructed GIST index by symbol trigrams: CREATE INDEX trgm_idx ON test_trgm
For conducting the experiment, the smallest possible sample of 100 examples of endings was taken; 80% of the sample consisted of randomly selected the most frequently used endings (the first 500 one), the remaining 20% were examples from the following 100 used endings (also randomly selected). The time spent on experiments were measured for each of the options (1)–(4). During each experiment, the characteristics are received (the abbreviations are indicated in brackets): 10 Postgres Pro Standard. https://postgrespro.ru/docs/postgrespro/9.5/textsearch-indexes average SQL query runtime (avg); 50th percentile (median); 90th percentile (90 perc); 95th percentile (95 perc); 98th percentile (98 perc).
The results are shown in table I.
The least time-consuming option turned out to be (1), suggesting a search among section names. This indicator is partly conditioned by those endings for which the sections were not created — in such cases, the cost of executing the SQL query was negligible. Search results without indexes and searches using the GIST index differ slightly from each other, what indicates the inappropriateness of using the GIST index to solve the research problem.
Satisfactory results showed the usage of the GIN index to search for incomplete matches (3).
An additional experiment on measuring the time which is spent for searching by the accentuation mask or by words from the poems consists in the formation of search queries and comparison their effectiveness. Trigram symbolic indexes GIST and GIN affected in the query are added separately to the text fields of the tables, namely the fields “Mask of accentuation” and “Text of the poem”; at the first stage of the experiment, the query runtime without indexes was measured. Types of executed requests: without indexes; using GIST index (the operator class gist_trgm_ops was used); using GIN index (the operator class gin_trgm_ops was used).
A fragment of a typical SQL query for which runtime was measured:
a
LEFT JOIN a."ID" as AUTHOR_ID, a."LASTNAME", a."FIRSTNAME", a."MIDDLENAME", p."ID" as POEM_ID, p."NAME" as POEM_NAME, m."ACCENTUATION_MASK" FROM "AUTHOR" "POEM" p ON p."AUTHOR_ID" = a."ID"
The query runtime was measured with different variants of the search conditions (for example, a search for a different number of words); the result was an average score of 10 queries with GIST and GIN indexes. The results of the experiment are shown in table II.
The results of an additional experiment showed an increase in the time for processing queries for text values used in the SQL query. Such an increase can be caused either by insufficient test data, or by the inefficiency of the applied indexes within the framework of the problem being solved.
The usage of PostgreSQL built-in database tools has long been limited by search engines in their modern understanding, the results were returned on request in a natural language using a DBMS (Database Management System). For the task of rhyme search, the program performance is not a determining factor. In the present work, the most prospective approaches were shown, as well as the examples on how to significantly speed up the algorithm using simple steps, what allows other researchers to apply these approaches as part of their research without requiring expert knowledge of the PostgreSQL database. In addition, the interface to access the DBMS does not change (except for the manual construction of a table with inverted rows), what is convenient for developers who integrate the text analysis systems with the PostgreSQL database.
The study was carried with the support of the Russian Science Foundation (project No. 19-18-00466).