Let's consider the architectural patterns of CLS design based on supporting the life cycle of the ML model for monitoring/managing the pipeline (information flow) of content (Fig. 1) [1]. The standard content processing pipeline implements an iterative process consisting of the stages of creating and deploying the machine learning (ML) process [2][3][4][5]. The process of monitoring/managing the content pipeline should still consist of additional stages to improve the quality/efficiency/efficiency of NLP problem solving [6][7][8][9]. At the construction stage, raw integrated content is filtered from noise/duplicates and formatted into a suitable form for further processing/management, conducting experiments on it, transferring it to ML models for classification/clustering/prediction/evaluation, etc. [10][11][12]. At the stage of content analysis and support, the content is deployed to determine the best ML model for making assessments/forecasts that directly affect the regular user and target audience.
Based on Feedback and model output, the target audience interacts with CLS, which facilitates the adaptation of the selected learning model. Five stages of related processes define the basic architectural principles for building a typical CLS. The processes of monitoring, processing and managing content are interaction, formatting/filtering, NLP, machine learning [13][14][15] and data accumulation in DS. For content analysis and support processes, respectively, these are feature analysis, deployment, prediction, interpretation, and content/result presentation. At the interaction stage, a set of rules for integrating content from multiple reliable sources at certain time intervals is necessary. Also, in parallel, a set of rules for checking the data entered by the CLS user is required as a preliminary stage for the formatting/filtering stage according to a collection of rules preset by the moderator and content from DS [16][17][18][19][20][21]. The next stage of NLP is a preparatory intermediate stage for machine learning and data accumulation. The machine learning stage can take various forms from SQL queries to various software modules. The support process is easier to implement than the management stage, provided that the latter is implemented correctly, especially during NLP analysis, in which additional lexical resources and artefacts (dictionaries, translators, regular expressions, etc.) are created, on which the effectiveness of CLS functioning directly depends (Fig. 2) [1][2][3]. The process of transition from raw text to a developed machine-learning model consists of a sequence of additional content transformations. First, the input textual content is transformed into an input corpus as a collection of texts, accumulated and stored in the DS. The incoming content is further grouped, filtered, formatted, linguistically processed, marked, normalized and converted into vectors for further processing. In the final transformation, the model/models (Fig. 3) are trained on the vector corpus, and a generalized presentation of the original content is created for further use in solving a specific NLP problem [1][2][3][4][5][6]. An ML-based CLS architecture with accelerated or even automatic model generation should support and optimize content transformation with ease of testing and tuning. The process of generating an optimal ML model is a complex cyclic algorithm, the main stages of which are the formation of a collection of features, model selection, and hyperparameter adjustment. After each iteration, the results are evaluated to determine the optimal collection of features, models, and parameters for solving a specific NLP problem with the appropriate input data [1][2][3][4][5][6]. The process of generating an optimal machine learning model
Analysis of signs and parameters
Model repository
Testing of the ML model According to [1][2][3][4][5][6], there are 3 main areas of statistical ML: a class of models, a form of a model, and a trained model. The class of models defines the relationship between the variables and the formed goal (for example, a linear model, a recurrent neural network, etc.). A model form is a specific component of a model: a collection of features, an algorithm, or a collection of hyperparameters. A trained model is a form of model that is trained on a specific data set and adapted to make predictions. CLSs consist of many trained models built during their selection, which creates and evaluates model shapes.
Any natural language text is initially a collection of non-random unstructured data as input content to CLS. But usually, the text is formed based on certain linguistic rules for the possibility of understanding these data. The purpose of the integration module is to transform this collection of non-random unstructured data into structured/semistructured fields (records) or markup for convenient interpretation by CLS modules. ML methods (for example, learning by a teacher) allow you to train (and retrain) statistical models as the language changes during NLP processes. By generating ML models on context-sensitive corpora, CLSs can apply narrow semantic values to improve accuracy without the need for additional interpretation.
Formally, the ML model of the Ukrainian language has to supplement the input incomplete phrase with missing words/phrases that are most likely to complete the content of the statement according to the previous text (context analysis for further guessing/predicting the meaning). Usually, a competently and correctly constructed text is predictable based on its coherence. Calculation of the entropy (degree of uncertainty/unpredictability) of the probability distribution of the model of the Ukrainian language measures the degree of predictability of the text. Thus, unfinished phrases Київстолиця... [Kyyiv -stolytsya...] (Kyiv -the capital...) and сонце сходить на... [sontse skhodytʹ na...] (the sun rises on...) have low entropy and statistical speech models are highly likely to guess the continuation of України [Ukrayiny] (Ukraine) and сході [skhodi] (the east), respectively. And expressions with high entropy like ми йдемо в гості до... [my ydemo v hosti do...] (we go to visit...) and я зустрів сьогодні... [ya zustriv sʹohodni...] (I met today...) offer many continuation options (parents, friends, neighbours, colleagues are equally likely without analyzing the previous context). Speech models can make inferences or identify connections between lexemes. Formally, the model uses context to identify a narrow decision space from a set of a small number of options. The application of statistical ML methods (with and without a teacher) allows the generation of speech models for extracting meaning from texts to support its predictability. First, the characteristic features of the content are identified to predict the goal. Textual data provides many opportunities to extract surface features based on parsing and breaking up sentences/utterances/phrases (e.g. bag of words), as well as based on extracted morphological/syntactic/semantic features. Special attention is paid to linguistic/ contextual/ structural features.
1. An example of the analysis of a linguistic feature can be the identification of the predominant gender in a fragment of the news text (the role of gender) in different contexts [1] to identify gender biases regarding the subject of publications. In the gender analysis of the text, words in the feminine and masculine gender are used to form a frequency assessment of gender characteristics, i.e.
𝑆𝑖𝑛𝑔 𝐺𝑆 =< 𝑋 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 , 𝑊 𝑀𝑎𝑙𝑒 , 𝑊 𝐹𝑒𝑚𝑎𝑙𝑒 , 𝑊 𝑈𝑛𝑘𝑛𝑜𝑤𝑛 , 𝑊 𝐵𝑜𝑡ℎ , 𝑓 𝑔𝑒𝑛𝑑𝑒𝑡𝑖𝑧𝑒 , >,
(1) where 𝑋 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 is analyzed sentence/expression; 𝑊 𝑀𝑎𝑙𝑒 is a set of words with the sign of a man; 𝑊 𝐹𝑒𝑚𝑎𝑙𝑒 is a set of words with the attribute woman; 𝑊 𝑈𝑛𝑘𝑛𝑜𝑤𝑛 is a set of words with an unknown gender sign; 𝑊 𝐵𝑜𝑡ℎ is a set of words with the sign of a man and a woman; 𝑓 𝑔𝑒𝑛𝑑𝑒𝑡𝑖𝑧𝑒 is the operator for identifying the gender class of a sentence.
𝑆𝑖𝑛𝑔 𝐺𝑆 = 𝑓 𝑔𝑒𝑛𝑑𝑒𝑡𝑖𝑧𝑒 (𝑋 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 , 𝑊 𝑀𝑎𝑙𝑒 , 𝑊 𝐹𝑒𝑚𝑎𝑙𝑒 , 𝑊 𝑈𝑛𝑘𝑛𝑜𝑤𝑛 , 𝑊 𝐵𝑜𝑡ℎ ), 𝑆𝑖𝑛𝑔 𝐺𝑆 = { 𝑁 𝑀𝑎𝑙𝑒 > 0, 𝑁 𝐹𝑒𝑚𝑎𝑙𝑒 = 0 → 𝑚𝑎𝑙𝑒 𝑁 𝑀𝑎𝑙𝑒 = 0, 𝑁 𝐹𝑒𝑚𝑎𝑙𝑒 > 0 → 𝑓𝑒𝑚𝑎𝑙𝑒 𝑁 𝑀𝑎𝑙𝑒 > 0, 𝑁 𝐹𝑒𝑚𝑎𝑙𝑒 > 0 → 𝑏𝑜𝑡ℎ 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 (2) where 𝑁 𝑀𝑎𝑙𝑒 is the number of words with the sign of a man in the analyzed sentence 𝑋 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 ; 𝑁 𝐹𝑒𝑚𝑎𝑙𝑒 is the number of words with the sign female in the analyzed sentence 𝑋 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 .
It is also necessary to determine the frequency of words, gender signs and sentences in the entire publication:
where 𝑋 𝑇𝑒𝑥𝑡 is analyzed publication text; 𝑆 𝑁𝐺 is a set of numbers of 𝑋 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 sentences of the analyzed text 𝑋 𝑇𝑒𝑥𝑡 marked by gender; 𝑁 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 is the number of sentences in the analyzed text 𝑋 𝑇𝑒𝑥𝑡 ; 𝑊 𝑁𝐺 is the set of the number of words of each gender characteristic for each marked sentence 𝑋 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 ; 𝑓 𝑐𝑜𝑢𝑛𝑡𝑔𝑒𝑛𝑑𝑒𝑟 is an operator of identification and classification/marking of all sentences of the analyzed text 𝑋 𝑇𝑒𝑥𝑡 by gender based on 𝑓 𝑔𝑒𝑛𝑑𝑒𝑡𝑖𝑧𝑒 .
For gender identification, it is necessary to parse the original text of publications with the subsequent marking of sentences and words based on the NLTK library:
where 𝑁 𝑤𝑜𝑟𝑑 is the number of words in the analysed text 𝑋 𝑇𝑒𝑥𝑡 ; 𝑁 𝐺𝑒𝑛𝑑𝑒𝑟 is the number of classifications by gender (in this particular case -4); 𝑊 𝑁𝐺 𝑘 is the number of words in sentences of a certain gender sign; 𝑆 𝑁𝐺 is the set of the number of sentences in the analyzed text of a certain gender sign; 𝑝𝑐𝑒𝑛𝑡 𝑘 is the percentage of publication text belonging to a certain gender sign; 𝑆𝑖𝑛𝑔 𝐺𝑆 𝑘 is a specific gender sign; 𝑁 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 𝑘 is the number of sentences in the analyzed text of a specific gender sign; 𝑆 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒 is a set of sentences identified by parsing in the analysed text 𝑋 𝑇𝑒𝑥𝑡 ; 𝑊 𝑊𝑜𝑟𝑑 is a collection of sets of words identified by parsing in each sentence of the analyzed text 𝑋 𝑇𝑒𝑥𝑡 ; 𝑊 𝑊𝑜𝑟𝑑 is the set of all words of the text 𝑋 𝑇𝑒𝑥𝑡 ; 𝑡𝑜𝑡𝑎𝑙 is the number of all words in the analysed text 𝑋 𝑇𝑒𝑥𝑡 . Such a deterministic mechanism demonstrates how the content/frequency of use of words/phrases (especially stereotypical ones) affects the predictability of the content according to the previous context (the gender sign is built directly into the Ukrainian language -every noun has a gender). But speech signs are not always decisive, for example, plural and time are used to analyse language/processes/actions/events in time.
2. An example of the analysis of a contextual feature can be the analysis of moods or sentiment analysis of a text (emotional colouring when discussing a specific topic by a relevant group of people). Usually used in complex analysis of feedback from users, for example, e-commerce, the polarity of messages or reactions to events/phenomena, in social networks or in political/economic discussions/forums, etc. In superficial sentiment analysis, the mechanism of gender classification (positive/negative/neutral coloured word) is usually used. For example, for positive -чудовий [chudovyy] (wonderful), прекрасний [chudovyy] (beautiful), правдивий [pravdyvyy,] (true), negative-лінивий [linyvyy] (lazy), поганий [pohanyy] (bad), дратівливий [drativlyvyy] (annoying), and neutral -білий [bilyy] (white), сонячний [sonyachnyy] (sunny), космічний [sonyachnyy] (cosmic). But the mood is not a feature of the language and depends on the meaning of the words/phrases according to the surrounding context of the text, for example, the word кумедний [kumednyy] (funny) has several interpretations of conveying the mood, in particular, positive -смішний клоун [smishnyy kloun] (funny clown), negative -кумедний одяг [kumednyy odyah] (funny clothes), and neutral -кумедний кіт [kumednyy kit] (funny cat) or кумедна іграшка [kumedna ihrashka] (funny toy). The word гострий [hostryy] (sharp) from the word перець [peretsʹ] (pepper) or ніж [nizh] (knife) has a positive meaning when buying, but from the word біль [bilʹ] (pain) and ніж [nizh] (knife) in a criminal case, it has a negative meaning. Also, negation turns the meaning of a positive text with positive words into a negative one and vice versa, for example, ми дуже багато очікували від відпочинку на морі сонячними гарними днями, але обіцяна курортна база відпочинку все спаскудила [my duzhe bahato ochikuvaly vid vidpochynku na mori sonyachnymy harnymy dnyamy, ale obitsyana kurortna baza vidpochynku vse spaskudyla] (we expected a lot from a vacation at the sea on sunny, beautiful days, but the promised holiday resort spoiled everything) (one negative word спаскудила [spaskudyla] (spoiled) all the previous positive ones) or дощ, прохолода та вітер не стали перепонами гарно відпочити в чудовій компанії [doshch, prokholoda ta viter ne staly pereponamy harno vidpochyty v chudoviy kompaniyi doshch, prokholoda ta viter ne staly pereponamy harno vidpochyty v chudoviy kompaniyi] (rain, coolness and the wind did not become an obstacle to a good rest in a wonderful company). Only thanks to machine learning in such cases it is possible to get the predictability of the text and reveal the emotional colouring according to the context. An a priori deterministic/structural approach loses the flexibility of context and meaning, so most speech models take into account the location of words in context, using ML methods for prediction. The main method of developing simple speech models is the bag of words as the frequency of co-occurrence of words in a narrow, limited context (Fig. 4). Such evaluation helps to determine the probability neighbourhood and to determine their meaning from small fragments of text. Next, using statistical inference methods, word order can be predicted. This is quite simple for English texts where words are not inflected. For Ukrainian language tests, it is better to use not a bag of words, but a bag of word bases. For example, for 12-word combinations as a 3-gram (36 words) without taking into account declension, we will get a matrix of size 2020, and with consideration of declension, gender and person (analysis of only the bases of words) -1515. Moreover, for the Ukrainian language, the location of bases in the 3-gram is usually not important and often has an unambiguous probability of compatibility in terms of content, for example, інформаційний ресурс (інформ ресурс) [informatsiynyy resurs (inform resurs)] (information resource (inform resource)) and ресурс інформації (ресурс інформ) [resurs informatsiyi (resurs inform)] (information resource (inform resource)). The bag-of-words/stems model is also extended by analyzing the co-occurrence of stable phrases and fragments of expressions that are of great importance for identifying the meaning of the text. The expressions зелений край скатертини (межа) [zelenyy kray skatertyny (mezha)] (green edge of the tablecloth (border)) and зелений край батьківщини (місцевість) [zelenyy kray batʹkivshchyny (mistsevistʹ)] (green edge of the homeland (locality)) in the form of a 3gram carry a different meaning. That is, there are several interpretations only for the word edge (the boundary of an object, a piece, the end of an action/state, a special area, a place of residence, an administrative-territorial unit). Statistical analysis of n-grams makes it possible to distinguish patterns of context. Speech models based on the analysis of n-gram contexts require the ability to explore the relationship of text to some target variable. The application of the analysis of linguistic and contextual features contributes to the formation of the general predictability of the text. However, their identification and further use require the ability to parse/identify the linguistic units of the language.
3. An example of the analysis of a structural feature can be the construction of an ontology for the implementation of IIS. Along with linguistic and contextual features, it is then necessary to identify and process high-level language units to define a vocabulary of operations for the text corpus. Different units of language are processed at different levels, and the correct implementation of NLP methods based on ML is important for the operational and correct identification of the linguistic context (sem relationship structure). Based on a typical pattern of utterances (statement or simple phrase) in the form of the subject verb object object definition (subject predicate appendix) construct ontologies that define specific relationships between entities. They make it possible to solve the problem of the lack of a mandatory order of words in a Ukrainian sentence to identify its semantics. It is advisable to use it for tasks where it is necessary to constantly process large volumes of text data and there is long-term resource support for the project. Semantic analysis consists not only in identifying the content of the text but also in generating data structures to which logical reasoning can be applied. Thematic Meaning Representations (TMR) are used to encode sentences in the form of predicate structures based on first-order logic or lambda calculus (λ-calculus). Network/graph structures are used to encode interactions of predicates of relevant text features. Then a traversal is implemented to analyze the centrality of terms or subjects and the reasons for the relationships between elements. Graph analysis is usually not a complete SEM (semantical analysis), but helps to form part of important logical decisions or conclusions. Semantics, syntax and morphology allow you to add data to simple text strings with linguistic meaning and generate new meaningful text content. Nowadays, natural language is one of the most commonly used forms of content. Its analysis makes it possible to increase the usefulness of data applications and make them an integral part of everyday life. Scalable analysis and machine learning of text primarily require up-to-date knowledge and text corpora of the relevant SA. For example, in the field of finance, CLS needs to identify financial terms, stock abbreviations and company names. Therefore, documents in the SA corpus must contain these entities. That is, the development of any CLS begins with obtaining textual data of the appropriate type and forming a corpus with structural and contextual features of SA.
For the GA of text strings, it is best to use regular expressions (RE) as algebraic notations for the features of a set of character strings. Commonly used in the development/maintenance of each type of computer language (programming, communication protocols, data markup, specification, and design), the operation of text editors, and word processing software, especially with IIS templated or SA text corpora collections. Identification/search of a fragment/string by pattern in a sequence of character strings is implemented to find all matches or the first one. The templates use special characters [, ], ^, \, -, ?, *, +, ., $, |, (, ), _, {, }, etc., including /, but the latter is not RE, but its boundaries The simplest RE is a tuple of simple characters (Table 1) to recognize the first or all pattern-like occurrences of character sequences. [13][14][15]; simply to denote carriages ^ (RE [16][17]. Question mark special character ? for RErules 20-21 allows you to mark optional characters in the searched string. This is useful in cases where there may be both present/absent characters in a certain sequence that do not resolve [].
In [] -you can indicate the absence of a specific symbol from the range of possible ones, but do not describe the absence of any symbol at all, indistinguishable from ?. The dot special character . for RE-rules 22-23 allows you to mark the location of any symbol in the sequence of the analyzed string. If the special character ? there is the absence or presence of one symbol, then we can submit the doubling of the symbol through the special symbol * (RE 26-29), which means the absence of a specific symbol or RE before * in the RE or its arbitrary number in consecutively placed in the recognized line, i.e. the result can be a line without this symbol. Therefore, to find at least one symbol from a possible sequence of the same two -for example RE 29, and for two different ones -30. The + special character for RE-rules 30-31 allows you to mark one or more cases immediately preceding the /RE symbol. {} (RE 32) is used to indicate the exact quantity (for example, exactly 2 times). The dot special character. often used together with the special character * to indicate any string of characters (RE 33).
An anchor is a special symbol (for example, a double sign ^ or a dollar sign $) specifying the location of the RE in the character string. In some cases, the caret ^ marks the beginning of a line (RE 34). The dollar sign $ recognizes the end of a line (RE 35-36). The backslash \ allows you to recognize special characters in the character string of the input test (RE 37-38). The anchors \b and \B identify the presence and absence of word boundaries, respectively (RE 39-42). A word is any tuple of numbers, underscores or letters (without special characters).
To organize the selection of alternatives between, for example, synonyms, the disjunction operation based on the special symbol | (RE 43-46). The combination of special characters | inside () allows you to arrange disjunction recognition only for a specific pattern, taking into account different inflexions/prefixes (RE 44). Special characters () are used to organize counters of type * (RE 46). The difference is that * is used for one character, not a whole sequence.
For complex disjunctive RE operators, when grouping from different special symbols, the concept of priority is used (Table 2): () *, +, ?, {} string, ^, $ | from the highest to the lowest (delimitation by the symbol ) (). Greedy RE patterns of the type /[a-ya]*/ recognize zero or more letters and no matches, expanding the identification to cover as many strings as they can. Non-greedy RE based on *? and +? find the smallest possible text. RE of the type /˽*/ is used to indicate the absence or presence of a certain number of spaces, since there can always be additional spaces around. There are aliases for general ranges that can be used primarily to preserve grapheme type (Table 3). Correctly constructed REs avoid errors of assumption (overrecognition) and negation (accidentally missed). Reducing the overall error rate for GA implies two antagonistic conditions for generating a collection of REs increasing recall (minimizing false ignores) and increasing precision (minimizing false recognitions). RE /{9}/ is the recognition of exactly 9 cases of the previous symbol/expression, RE /а.{3}я/ -sequences v, RE /{3,12}/ -from 3 to 12 of the previous symbol/expression, RE /(5,)/ is at least 5 occurrences of the preceding character/expression, and RE /(,13)/ is up to 13 occurrences of the preceding character/expression. The special character s before RE allow you to replace the expression with a pattern. The special character \k indicates the location of the character/phrase/expression as a duplicate of the first element in the capture group, i.e. the pattern in (), where k is the number of brackets or capture groups. Thus, special characters () have a double function in RE: to group conditions and to determine the order of application of operators. For grouping, without fixing the received template in the register, the RE of the form (?: template) is used as a group that does not capture the expression. When applying RE, the rank of use in the queue is determined. An RE of type (?: template) is a positive statement (RE 23).
The (?=pattern) operator is positive when identifying a zero-width pattern, i.e. the match pointer is not advanced. The (?!pattern) operator is positive if the pattern does not match, is zero-width, and the cursor does not advance. Negative statements are usually used in the analysis of a complex content model when a special case needs to be removed (RE 24).
Grapheme analysis is the preliminary processing and transformation of the text into a certain marked and compressed format for the following NLP processes (Fig. 5): extracting content extracting paragraphs extracting sentences within a paragraph extracting tokens within a sentence marking tokens with tags for MA as part-of-speech marking.
Sentence At the first stages of the integration of content from various sources, it is necessary to implement the processes of filtering, access and calculation of text sizes based on the application of the standard API of pre-grapheme processing of the division of documents through the execution of the following sequence of NLTK methods:
and if necessary, additional methods, such as adding tags or parsing sentences, converting annotated text into tree-like data structures, or extracting individual XML elements. To identify and extract the main content from an information resource with an undefined structure and high variability of documents from different sources, 𝑓 ℎ𝑡𝑚𝑙 () based on the Python readability-lxml library is used, which removes all anomalous artefacts, leaving only the text. When processing HTML text, 𝑓 ℎ𝑡𝑚𝑙 () uses a collection of formal REs to identify and remove navigation menus, declarations, script tags, and CSS, then creates a new content object model tree, extracts the text from the source tree, and embeds it into the newly created tree.
Vectorization, feature extraction, and ML tasks rely heavily on CLS's ability to efficiently break down textual content into its constituent components while preserving the original structure. The accuracy and sensitivity of ML models depend on the efficiency of identifying the connections of tokens with the corresponding context in the text. Paragraphs contain complete ideas of context and are the structural unit of content. Based on NLTK, the 𝑓 𝑝𝑎𝑡𝑎𝑠 () operator is implemented as a paragraph generator, which is defined as blocks of text separated by two newline characters. The 𝑓 𝑝𝑎𝑡𝑎𝑠 () the operator scans all files and passes each HTML text to the RE constructor, indicating that parsing of the HTML markup should be done through the lxml HTMLparser. The resulting object maintains a tree structure that can be navigated using native HTML tags and elements.
If paragraphs are structural units of content, then sentences are semantic units. As a paragraph expressing a single idea, a sentence contains a complete thought that the author has formulated and expressed in many words. Grapheme segmentation is the division of text into sentences for further processing by marking words with parts of speech in MA. The operator 𝑓 𝑠𝑒𝑛𝑡𝑠 (), calling 𝑓 𝑝𝑎𝑡𝑎𝑠 () and returning an iterator (generator), sorts all sentences from all paragraphs.
The 𝑓 𝑠𝑒𝑛𝑡𝑠 () operator bypasses all paragraphs selected by the 𝑓 𝑝𝑎𝑡𝑎𝑠 () operator and uses the 𝑓 𝑤𝑜𝑟𝑑𝑠 () operator to perform the actual grapheme segmentation. Internally, the 𝑓 𝑡𝑜𝑘𝑒𝑛𝑠 () operator uses 𝑓 𝑚𝑎𝑟𝑘 (), a model pre-trained with RE recognition/identification rules for various kinds of tokens, punctuation marks, abbreviations, geographical names, abbreviations, and other marks that serve as sentence start/end or tab marks. Punctuation marks do not always have an unambiguous interpretation, for example, or are a sign of the end of a sentence, but they are also present in dates, abbreviations, abbreviations, ellipses, etc. Determining sentence boundaries is not always an easy task. Punctuation is crucial for identifying word boundaries (commas, spaces, colons) and for identifying certain aspects of meaning (question marks, exclamation marks, quotation marks). For some tasks, such as tagging parts of speech, and analyzing or synthesizing speech, it is sometimes necessary to treat punctuation marks as if they were separate words. When analyzing speech, punctuation marks replace pauses, accents, and changes in intonation dynamics. Lexemization is the process of obtaining lexemes (syntactically encoded strings of symbols) and for its implementation, the operator 𝑓 𝑤𝑜𝑟𝑑𝑠 () based on RE is used, which is selected through 𝑓 𝑚𝑎𝑟𝑘 () markers for spaces and punctuation marks and returns a list of alphabetic and non-alphabetic characters. Like delimiting sentences, lexeme recognition is not always an easy task: the presence of punctuation marks in a lexeme, punctuation marks as independent lexemes, lexemes with and without hyphens, and lexemes as shortened forms of words (one or more words). Different marker selection tools are chosen for these cases. Any statement is a speech correlate of a sentence. The presence of lexemes of the dysfluency type (loss of speech speed, for example, a longer pause when thinking) carries not so much a semantic load as an emotional one. Exclamations such as мммм, ох, ах [mmmm, ohh, ah], etc. are fillers or filled pauses and are also emotionally coloured, but not semantically coloured. An unfinished word with further repetition and its ending or simply with repetition is a fragment that does not carry a semantic load, but only an emotional one. Therefore, when conducting PHA, depending on the goal of solving a specific problem through CLS, it is important to take into account (mark accordingly) or ignore some types of punctuation (ellipsis, exclamation points, etc.), dysfluencies, double fragments, exclamations, etc. If CLS is just a transcription of speech, then such phonemes should be ignored to avoid loss of speech rate. But they make it possible to determine the psychological state of the speaker and his emotional state, to identify the peculiarity of the speaker's authorial speech when the tone of the voice changes, they are relevant in predicting the future word, because they signal that the speaker is restarting the statement/idea, and therefore, for speech recognition, ordinary tokens are considered as phonemes. Marking a lexeme as a lemma (a set of lexical forms having the same base, the same main part of speech and the same word content) or as a word form (a fully inflected or derived form of a word) is a significant difference for conducting the next stage of MA as lemmatization or stemming, i.e. identification of word bases. For many NLP tasks in the English language, it is enough to mark the corresponding lexemes as word forms, but for the Ukrainian language -no, it is still necessary to identify the bases of the words (for example, based on the analysis of inflexion according to the tree of endings).
There are two ways to identify words with punctuation ignored -token recognition as types (the number of different words |V| in the set of words of the corpus, i.e. the cardinality of the alphabet of the corpus, where an element of the alphabet/dictionary is a unique word) and tokens (the total number N of words of the analyzed corpus), i.e. |V| N. The largest Google N-grams corpus contains 13 million types among those displayed 40, so the true number is much larger.
The ratio between the number of types |V| and the number of tokens N is called Herdan's law (Herdan, 1960) or Heaps' law (Heaps, 1978): |𝑉| = 𝑘𝑁 𝑥 , where 𝑘 and 𝑥 are positive constants for 0 < 𝑥 < 1. The value of x depends on the size of the corpus and the genre, for large corpora x varies within [0.67; 0.75], when the size of the dictionary for the text grows much faster than the square root of the length of its words. Another measure of the number of words in a language is the number of lemmas rather than word types (for example, the Oxford English Dictionary has over 615,000 entries).
Morphology identifies the shape of things, and in textual analysis, the shape of individual words/tokens. Lexemes are both words and punctuation marks, allowing you to conduct the next SYA (syntactic analysis) more clearly. Word structure helps determine plural, gender, tense, person, declension, etc. MA is a difficult task, as most languages have many exceptions to the rules and special cases. The main task of MA is to identify parts of words to assign them to certain classes (tags) of parts of speech. For example, sometimes it is important to understand whether a noun is singular or plural, or is a proper name. It is also often necessary to know whether the verb has an indefinite form, past tense, or is an adjective. The resulting parts of speech are then used to generate larger structures (fragments/phrases), or whole word trees, which are then used to build semantic reasoning data structures. After GA (grapheme analysis), we have access to tokens in sentences in paragraphs of integrated content texts, which makes it possible to apply MA to mark words from the collection of tokens with parts of speech (e.g., verbs, nouns, prepositions, adjectives) that indicate the role of the word in the context of the sentence. In the Ukrainian language, the same word can usually take on different roles, depending on the inflexions. Part-of-speech tagging based on MA rules consists of adding a corresponding tag to each word from a collection of tokens that contains information about the definition of the word and its role in the current context. MA rules are used for the development of modules/subsystems for keyword identification, text classification (Fig. 4.6), machine translation, and error correction, as well as for human psychological analysis, semantic analysis, etc. When identifying words for further classification, the rub_id attribute describes the rubric to which a specific keyword belongs (Table 4). The flag of the attribute defines the properties of this keyword (the part of the language to which it belongs). In thematic dictionaries, each word has its property, for example, a b c d o -different types of nouns, A -verbs, V -adjectives (Fig. 7). To compare the complexity in thematic dictionaries (23 rules in total), each English word also has a property, for example, the numbers 1-23 are the numbers of rules of the PFX type (prefixes, rules 1-7) and SFX (suffixes and endings, rules 8-23) and describe some nouns for English words (Fig. 8). For example, PFX-type rules describe the modification of some nouns for English words The letters e and y near the suffixes are decision markers. A file of affixes (parts of words that attach to the root and bring grammatical or wordforming meaning, elements of word formation, for example, prefix, suffix, postfix, inflexion) has the *.aff file type and may contain additional attributes -the rules of reduction to the base of the word (Fig. 9). The notation SET is usually used to identify the sequence of parts of affixes and directories. REP forms a lookup table to correct multiple characters for words. TRY identifies sequences to replace. SFX and PFX identify the types of suffixes and prefixes that are marked by word affixes. The flag of the flag attribute determines the type of word, the mask of the mask attribute shows the ending identification rule, the value of the find attribute is the ending of the word in the nominative case, the value of the repl attribute is the ending of the word in the nonnominative case. Exceptions to the rules are given in square brackets. For example, the first line (ordering 26) describes a specific example of recognizing nouns of group a with the alternation of -і -о and the inflection -ін of the nominative case in the instrumental case (inflection -оном), and the next entry (ordering 27) is the same nouns, but in the local case (inflection -оні), but does not recognize other rules of that group or other groups in the dative case -inflections -онові and -ону (Fig. 10). The third record (ordering 28) already recognizes nouns with alternation -і -о with inflections v of the nominative case in the dative case -inflection -огу, but does not recognize other rules of the same group and (rules 29-31 do this, respectively): -огові (Д.М.), -огом (О.), -озі (М.). The presence of a ratio of words blocked by the moderator (Fig. 12), in particular those that cannot be key, allows to reduce the amount of verification during text classification (Fig. 13). To identify keywords, it is important to correctly recognize adjectives in any case, gender and number (Fig. 14). c. Group V for nouns marked with the flag as p: masculine (m.) and feminine (f.) patronymics of the singular (s.) and plural (m.) of male names. For further SYA, it is appropriate to recognize the verbs correctly (Fig. 15). Let us describe in more detail each marked class of the set of noun recognition rules, indicating their total number N (Table 5). In total, about 1,300 rules for processing suffixes and endings are used for MA Ukrainian-language nouns, taking into account the alternation of letters. It is quite difficult to generate terminal chains in English (but MA rules are much less, not in Ukrainian), because the presence of articles and the connection of groups of nouns with each other with the corresponding preposition makes the tree longer and wider. The generation of terminal chains in the Ukrainian language is complicated by cases and generic differences in inflexions of the term used in the context. To identify keywords, it is not enough to recognize nouns (about 1300 RE-rules), it is also necessary to identify adjectives -a total of 99 RE-rules for Ukrainian texts (Table 4.6-Table 4.7). For correct SYA and SEM, including ontology construction, it is necessary to recognize verbs based on more than 800 RE rules.
CLS marks the words of the input text as parts of speech (clarifies after GA the tagged/marked lexemes as words) based on RE-rules and analysis of inflexions as singular nouns of the corresponding gender and case, plural nouns of the corresponding case, adjectives, adverbs, verbs, personal pronouns, etc. (each with a collection of features).
The MA module returns a collection of paragraph lists, each of which is a list of sentences, which are lists of tokens, including words marked by parts of speech. Periodic interim analysis of the input/integrated textual content allows to assess how the thematic corpus changes over time. In the process of analysis, we will count the number of paragraphs, sentences and words, and also save each unique lexeme in an additional intermediate dictionary. If the lexeme/word did not exist in the dictionary of lexemes/word bases, we mark it as new and store it in the intermediate dictionary for analysis by the moderator. We count the number of content and categories in the corpus of incoming text content and form a dictionary with a statistical summary of the corpus, which contains: the total number of integrated content and categories; the total number of paragraphs, sentences and words; the number of unique tokens; lexical diversity as the ratio of the number of unique lexemes to their total number; the average number of paragraphs in the content; average number of sentences per paragraph; total processing time.
Since the corpus grows as new data is collected, pre-processed and compressed, the MA method will allow us to calculate these features and analyze their dynamics of change. It is an important content monitoring tool to identify possible problems in CLS, for example, in an ML model, a significant change in lexical diversity and the number of paragraphs per content affects the quality of the model. That is, the MA method and GA methods, in addition to the identification of tokens and direct marking of words by parts of speech, are used to collect additional information when determining the amount of changes in the corpus to timely start further vectorization and restructuring of the ML model. The main stage of the MA method is the identification of the bases of words (stemming) without taking into account inflexions (suffixes and endings) and in some cases -prefixes. According to the content of the inflexions, a part of the language is identified as a word (Fig. 16).
For the next SYA, this is not enough (to mark the word only as a part of speech), it is still necessary to determine, for example, gender/distinctiveness, etc., for a noun/adjective. The classic Porter stemmer algorithm works by sequentially cutting off endings and suffixes. For English-language texts, this is not a problem, as there are very few inflexions. For Ukrainian words, a modified (extended) algorithm of Porter's stemmer should be applied with a check of both additional inflexions depending on the part of the language (according to the tree of endings), as well as the obtained word bases with a dictionary of bases to identify the existing word (Fig. 17).
Algorithm The increase in volume of MA RE-rules increases in a geometric progression the load on CLS only due to the recognition of inflexions and the bases of word forms. For Englishlanguage texts, the complexity is less due to several parameters, for example, for nouns 2 cases -2 inflexions in the plural (s|es). For the German language, the complexity increases -4 cases (but inflexions almost do not change, only articles change), phrases with 2 words are written together, etc. In the Ukrainian language, there are 7 cases of nouns, each of which changes its inflexion depending on the gender and plural/singular, and some words have different endings in some cases (for example, for втручання [vtruchannya] (intervention) in the local case, there are two options -втручанню, втручанні), in addition, there is often alternation of letters.
Therefore, for Ukrainian words, Porter's simple classic stemming algorithm is not suitable (reducing the word to the base root by cutting off inflexions). It is better to combine such an algorithm with a search/check of the obtained intermediate results with a tree of inflexions (so as not to go through all possible inflexions) and with the content of thematic dictionaries of bases with a set of RE-rules for the identification of features (classification by parts of speech). Only for text rubrication based on word identification, it is enough to conduct MA only for some noun groups (adjectives with nouns and nouns with nouns) without analyzing words of other parts of speech (recognition by the tree of inflexions -not an adjective and not a noun -ignore, in addition, the key ones should be sometimes there can be 1 preposition next to and only between nouns. It is enough to identify the bases of nouns/adjectives/abbreviations in the text and analyze their probability of clustering in different parts of the content relative to the total volume.
The classic stemming algorithm -Porter's Stemmer -does not use dictionaries of word bases but only applies a set of RE-rules for cutting off inflexions in sequence according to the specifics of a specific language. The algorithm works with individual words without analyzing and taking into account the context. Linguistic features such as features of word formation (prefix, suffix, etc.) and parts of speech (noun, verb, etc.) are not taken into account. The basis is the following techniques for words:
cutting off the inflexion from the analyzed word (for Ukrainian words, it can be implemented with the obtained bases and inflexions check with analogues in DB). the word has an invariable inflexion (the condition is impossible for most Ukrainian words, but it is possible to identify particles, conjunctions, prepositions, some nouns of foreign origin, abbreviations, etc.).
changes inflexion in declension due to dropping/alternating letters. the change of word inflexion and word formation corresponds to a specific RE-rule, for example, when forming words from some verb groups:
(ов)*ува(ти|нню|нням|нні|ння|ли|ло|ла|вшись|вши|в|вся|всь|лися|лись|тися|тись) [(ov)*uva(ty|nnyu|nnyam|nni|nnya|ly|lo|la|vshysʹ|vshy|v|vsya|vsʹ|lysya|lysʹ|tysya|tysʹ)].
changing the inflexion of the word as an exception to the RE rules.
the ending of the word coincides with the envelope RE-rule of identification of inflexion, but the word itself has no inflexion: вітер [viter] (wind), but відер [vider] (bucket). most short words are invariable (stop word dictionary is sufficient).
Such techniques significantly complicate the stemming algorithm of Ukrainian words. Therefore, first, widespread inflections are analyzed, for example, for 1 letter ц (34), щ (110), ф (214), б (281), п (341), ж (353), з (581), г (636), л (754), с (914), ч (959), д (1038), н (2531), р (2709) or 1-4 letters (Table 2.2). Inflexions 5 (for example, max(йтесь)=6837, max(ванням)=4656) are significantly less among keywords, therefore, for the speed/efficiency of the solution in some CLS NLP tasks, they are ignored, but for SYA/ SEM will not allow this. Many NLP tasks do not require full implementation of all NLP processes from grapheme to pragmatic analyses. For example, to identify keywords, it is enough to provide a grapheme and morphological analysis (algorithm 4.2). But before almost any NLP process, the text must be normalized. sequentially marking each sequence of non-alphabetic characters as tokens and recognizing alphabetic sequences between spaces and other special characters (eg numbers and punctuation) according to RE rules as token words to form a list 𝑆 of identified alphabetic tokens as words 𝑤 𝑖 .
Step 1.3. Sort the list 𝑆𝑆 𝐴 identified tokens 𝑤 𝑖 alphabetically, counting occurrences of identical chains and forming an alphabetic-frequency dictionary 𝐷 𝑎 , the record of which is in the form of the number of occurrences -a word. Step 1.4. Transferring all letters of the upper register to the lower register and recalculating occurrences of word-tokens in the alphabetic-frequency dictionary 𝐷 𝐴 𝐷 𝑎 .
Step 1.5. Sort and save the dictionary 𝐷 𝑎 𝐷 𝑁 of identified 𝑤 𝑖 words by decreasing the frequency of appearance (in Germanic languages, the top will be articles, pronouns, adjectives and conjunctions, and in Slavic languages, most words with the same base and different inflexions will occupy different lines of the list, which significantly distorts the picture of the real distribution of words in texts). Stage 2. Segmentation/tokenization of words of the analyzed text content.
Step 2.1. Word segmentation based on dictionaries, metrics such as the probability of an error in a word, and statistical sequence models pre-trained from segmented text corpora (between spaces, punctuation, etc.).
Step 2.2. Tokenization based on RE-rules of marked tokens of the sequence type of non-alphabetic characters as tokens (dates, prices, URLs, hashtags, e-mail addresses, etc.), punctuation (as the end of a sentence or the boundary of a subordinate clause), mixed tokens of alphabetic-nonalphabetic characters (abbreviations, complex hyphenated words, with an apostrophe, etc.), lines with uppercase characters (such as the beginning of a sentence, geographical names, proper names, abbreviations) and their normalization if necessary (for example, к.т.н. ктн (PhD) as a separate word-token or ML як машинне навчання [mashynne navchannya] (machine learning)).
Step 2.3. Analysis of tokens with uppercase characters (except when only the first letters are capitalized) for labelling based on the RE-rules of finite automata or as an abbreviation or emotion transfer.
Step 2.4. Marking of unidentified 𝐷 𝑥 tokens and ambiguities (e.g. apostrophe as part of a word, etc.). Stage 3. Lemmatization of a set of recognized and labelled alphabetic tokens of the text as lemmas, identified as words of the analyzed text.
Step 3.1. Normalization of tokens based on the identification of affixes from the termination tree as stenocardia of marked token-words (reducing the word to its initial form based on RE-rules MA for identification roots and affixes through Algorithm 1 of Porter's modified stemmer), i.e. determination of whether the analyzed tokens have the same root and differ only in inflexion with sequential identification of the part of the language of the analyzed words with subsequent marking of them as lemmas with all accompanying linguistic features. Step 3.2. Regrouping and recalculation of word frequencies in the alphabetic-frequency dictionary 𝐷 𝑁 𝐷 𝑙 taking into account the normalized words in step 3.1. Stage 4. Additional analysis of unidentified tokens 𝐷 𝑥 by iteratively combining frequent character/string pairs within token words (for example, whether tokens between spaces or other punctuation marks контент-аналіз [kontent-analiz] (content-analysis), Web-сайт
or Web-resource [Web-resource] (Web-resource) are one word, or two) through bit-pair encoding, or BPE based on text compression for further possible identification of words, their labelling and normalization.
Step 4.1. Formation of a set of symbols equal to the collection of properties with 𝐷 𝑥 . К We present each word as a sequence of characters plus a special character at the end of the word or a special character, such as a dash, within a token (for example, контент-, Web-, контент-or Web-). We denote 𝑖 = 0. Step 4.9. We denote 𝑖 = 𝑖 + 1. If ℎ > 0, 𝐷 𝑥 for 𝑛 𝑙 > 1 and in 𝐷′ 𝑥 is at least 1 marked 𝑏 𝑖 , then go to step 4.4, otherwise (ℎ = 0 and 𝐷 𝑥 = or 𝐷 𝑥 at 𝑛 𝑙 = 1 or 𝑠 𝑘 𝑥 no non-unique pair 𝑠 𝑗 𝑥 for formation 𝑎 𝑖 = (𝑠 𝑘 𝑥 , 𝑠 𝑗 𝑥 )) -until step 5.
Stage 5. Segmentation of sentences in the analysed content.
The process of lexical analysis of the Ukrainian-language text 𝐶′ consists in parsing, segmentation and tokenization of each sentence separately, which is characterized not by a strict order of words, but at the same time by a constant arrangement of individual linguistic units. In a complete simple Ukrainian sentence with direct word order, the structural scheme is conditionally fixed. The main lexical categories of the corresponding sentence are noun and verb groups. Type 0 grammar according to N. Chomsky's classification is not appropriate for such sentences due to the complexity of implementation. With contextdependent grammar, specific restrictions are applied, in particular, to the structure of a Ukrainian-language sentence with some set of variations. Based on the syntactic rules of generating Ukrainian-language sentences with partial word order (for example, there is no strict order for the subject and predicate in the sentence, but the adjective is usually before the noun or another adjective, if it is not a poetic passage, also the lexical units of the noun group are placed around the subject, etc.), we derive the lexical scheme for the noun group 𝑆 ̃ based on regular expressions:
where 𝐴 = 𝑎 1 𝑎 2 𝑎 3 … 𝑎 𝑁−1 𝑎 𝑁 is a sequence of adjectives, and the entry [𝐴]{0, 𝑛} is a selection from 0 to 𝑛 adjectives from 𝑎 1 𝑎 2 𝑎 3 … 𝑎 𝑁−1 𝑎 𝑁 , at 𝑛𝑁; 𝑆 = 𝑠 1 𝑠 2 𝑠 3 … 𝑠 𝑀−1 𝑠 𝑀 is a sequence of nouns, and the entry [𝑆]{1, 𝑚} is a selection from 1 to 𝑚 nouns from 𝑠 1 𝑠 2 𝑠 3 … 𝑠 𝑀−1 𝑠 𝑀 , at 𝑚𝑀; 𝑃 = 𝑝 1 𝑝 2 𝑝 3 … 𝑝 𝐾−1 𝑝 𝐾 is a sequence of pronouns, and the entry [𝑃] is the choice of 1 pronoun from 𝑝 1 𝑝 2 𝑝 3 … 𝑝 𝐾−1 𝑝 𝐾 ; record (𝑥|𝑦) is a choice of either 𝑥, or 𝑦; the values of 𝑎 𝑖 and 𝑠 𝑗 agree in gender, number and case. Accordingly, for the verb group, the lexical scheme based on RE-expressions:
where 𝑉 = 𝑣 1 𝑣 2 𝑣 3 … 𝑣 𝑁−1 𝑣 𝑁 is a sequence of verbs, and the entry [𝑉]{1, 𝑛} is a choice from 1 to 𝑛 verbs from 𝑣 1 𝑣 2 𝑣 3 … 𝑣 𝑁−1 𝑣 𝑁 , at 𝑛𝑁; 𝑆′ ̃= 𝑆 ̃1𝑆 ̃2𝑆 ̃3 … 𝑆 ̃𝑀−1 𝑆 ̃𝑀 is a sequence of noun groups, and the entry [𝑆′ ̃]{0, 𝑚} is a choice from 0 to 𝑚 noun groups from 𝑆 ̃1𝑆 ̃2𝑆 ̃3 … 𝑆 ̃𝑀−1 𝑆 ̃𝑀, at 𝑚𝑀; entry (𝑥|𝑦) is choice of either 𝑥, or 𝑦; agreement between 𝑣 𝑖 and 𝑆 ̃𝑗 is carried out by person, gender and number. The lexical scheme of a Ukrainian sentence based on REexpressions:
where 𝑉′ ̃= 𝑉 ̃1𝑉 ̃2𝑉 ̃3 … 𝑉 ̃𝑁−1 𝑉 ̃𝑁 is a sequence of verb groups, and the entry [𝑉′ ̃]{0,1} is a selection from 0 to 1 verb groups with 𝑉 ̃1𝑉 ̃2𝑉 ̃3 … 𝑉 ̃𝑁−1 𝑉 ̃𝑁 with the presence of a predicate; 𝑆′ ̃= 𝑆 ̃1𝑆 ̃2𝑆 ̃3 … 𝑆 ̃𝑀−1 𝑆 ̃𝑀 is a sequence of noun groups, and the entry [𝑆′ ̃]{0,1} is a selection from 0 to 1 noun groups from 𝑆 ̃1𝑆 ̃2𝑆 ̃3 … 𝑆 ̃𝑀−1 𝑆 ̃𝑀 with the presence of a subject; record (𝑥|𝑦) is a choice of 𝑥 or 𝑦; agreement between 𝑉 ̃𝑖 and 𝑆 ̃𝑗 is carried out by person, gender and number.
The main lexical features of the verb group are tense, number, person. For comparison, the lexical scheme of the noun group based on the RE-expression for an English-language sentence:
(10) The lexical scheme of the English verb group based on the RE-expression:
) Lexical scheme for an English-language sentence based on the RE-expression:
The agreement of cases between the lexical units of the Ukrainian-language sentence affects the further syntactic and semantic analysis of the content:
where 𝑥 𝑖 , 𝑥 𝑖 ′ , 𝑞 are the main lexical units; 𝑅, 𝑌 𝑖 are auxiliary lexical units; 𝑅 is the initial symbol as an indicator of the type of sentence chain generation.
Stages of lexical formation of a chain of tokens
An example of lexical generation of the type {𝑥𝑞𝑥 ′ }: Саша
,поет 𝑑 , … respectively, where𝑥 (𝑎𝑏𝑐𝑑. . . ) is a sequence of proper names, 𝑥 ′ (𝑎 ′ 𝑏 ′ 𝑐 ′ 𝑑 ′ . . . ) is a sequence of professions agreed with proper names; 𝑞 is a dash. Any verb has the ability to act as a complement: моя дитина вподобала книгочитання [moya dytyna vpodobala knyhochytannya] (My child liked reading books). This process can theoretically be repeated an unlimited number of times: він книгочитанняцікаводумає про книгочитанняцікавість [vin knyhochytannyatsikavodumaye pro knyhochytannyatsikavistʹ] (it is interesting to read books, thinks about reading books, interesting), i.e.
) is generated by a grammar of 6 rules:
Such grammar do not provide, for example, a natural description for the so-called nonproject constructions with breaks, crossing ( . . . . , . . . . ) or framing (
. ) directions of syntactic dependence (Fig. 19).
Ukrainian. Наша мова, як і будь-яка інша, посідає унікальне місце.
A theorem is stated which describes the properties of this function.
German. ... die Tatsache, daß die Menschen die Fähigkeit besitzen, Verhältnisse der objektiven Realität in Aussagen wiederzuspiegeln.
Francian. 3. Sequential subordination (Fig. 20):
досить повiльно рухлива черепаха or очень быстро бегущий олень. Only with the correct identification and recognition of non-project constructions can a grammatical and syntactic analysis of Ukrainian sentences be carried out to build dependency trees of the components of these sentences.
The syntax is a set of relational rules for the formation of sentences/phrases, usually defined by the grammar. Sentences are linguistic units of language for generating meaning and encoding information. The purpose of SYA is to demonstrate meaningful relationships between words based on the division of a sentence into parts, or between tokens in a treelike structure 𝐶′ . Syntax is a necessary basis for reasoning about a system of concepts or semantics because it is an important tool for determining the degree to which words influence each other in the generation of phrases. For example, SYA identifies the prepositional phrase в потяг There are cases in the textual content when not only the right but also the left sequential subordination has an unlimited depth of derivation, for example, due to subordinate clauses with the operative word which, what, when, etc. (тваринка, яку врятувала Софія [tvarynka, yaku vryatuvala Sofiya] -the animal that Sofia saved). Fig. 23 illustrates a phrase with a depth of 22 and is completely grammatically correct (as is its Ukrainian version). Moreover, nothing prevents you from continuing the phrase to the left на волю в обійми зеленої пахучої трави [na volyu v obiymy zelenoyi pakhuchoyi travy] (freely into the embrace of green, fragrant grass). The Ukrainian language allows you to generate phrases with an unlimited number of sequentially subordinating from left to right constructions of the type 𝑌 1 𝑌 2 . . . 𝑌 𝑖 . .. (unlimited right subordination), and at the same time, unlimited left subordination is possible in each of the constructions 𝑋 𝑖 -a sequence of chains . . . 𝑌 𝑖𝑗 . . . 𝑌 𝑖3 𝑌 𝑖2 𝑌 𝑖1 ; however, within the sequence 𝑌 𝑖𝑗 further unlimited expansion is impossible. According to the rules of the Ukrainian language 𝑌 𝑖 are interpreted as simple sentences, each of which is an additional determiner to the previous one, and 𝑌 𝑖𝑗 are interpreted as prepositive adjective inflexions.
The grammar 𝐺 ′ = ⟨𝐷 ′ , 𝐷 1 ′ , 𝐼 ′ , 𝑅 ′ ⟩ has a basic dictionary 𝐷 ′ = 𝑁 1 , 𝑁 2 , . . . , 𝑁 𝑛 symbols and rules of the form 𝑅 ′ = {𝑌 → 𝑍𝑁 𝑖 , 𝑋 → 𝑁 𝑖 }, where 𝑌𝐷 1 ′ and 𝑍𝐷 1 ′ . Each of 𝑁 𝑖 corresponds to some regular grammar 𝐺 𝑖 ′ = ⟨𝐷, 𝐷 1 𝑖 , 𝑁 𝑖 , 𝑅 𝑖 ⟩, where 𝐷 is the main dictionary 𝐺 𝑖 ′ , 𝐷 1 𝑖 is the auxiliary dictionary for 𝐷 1 𝑖 𝐷 ′ = 𝑁 𝑖 and 𝐷 1 𝑖 𝐷 1 ′ = 𝑁 𝑖 ; 𝑁 𝑖 is the initial symbol; scheme rules of the form 𝑅 𝑖 = {𝐶 → 𝑒𝐸, 𝐶 → 𝑐} (heading Latin characters are non-terminal, and line characters are terminal). The non-terminal dictionaries of the grammar 𝐺 𝑖 ′ are pairwise disjoint. Association:
where the main dictionary 𝐷 in all 𝐺 𝑖 ′ , and the auxiliary additional dictionary and scheme:
The grammar 𝐺 is special and equivalent to an automatic one, for example: To analyze the syntactic structure of a sentence is to identify the order of words depending on the syntactic structure and relationships, which is determined necessarily according to the analysis of neighbours and something derived/secondary. It is advisable to modify the grammar so that both parts of the predicate (Fig. 24) are trees of syntactic relations. Lines with subscripts describe syntactic relations of various types; symbols 𝐴, 𝐵, 𝐶, . .. are syntactic categories. As a result, the syntactic structures (rather than phrases) of the language are obtained as part of the generative grammar. Another part of this grammar is the calculation in the Ukrainian language -with mandatory consideration of the logical derivation of linear sequences of words, solving the problem of discontinuous constituents.
Semantic analysis consists not only in identifying the content of the text but also in generating data structures to which logical reasoning can be applied. Thematic Meaning Representations (TMR) are used to encode sentences in the form of predicate structures based on first-order logic or lambda calculus (λ-calculus). Network/graph structures are used to encode interactions of predicates of relevant text features. Then a traversal is implemented to analyze the centrality of terms or subjects and the reasons for the relationships between elements.
Analysis of graphs, including ontology О, is usually not a complete SEM, but helps to form part of important logical decisions/conclusions based on the taxonomy of concepts 𝑋: 𝑂: 𝑈𝐿𝑆𝑅 𝐶𝑜𝑛𝑐𝑒𝑝𝑡𝑠.
(18) The result of SEM based on the ontological model of the rules of the syntax of the Ukrainian language О are weighted oriented graphs of the semantics of the text:
where 𝑅𝑒𝑙𝑎𝑡𝑖𝑜𝑛𝑠ℎ𝑖𝑝𝑠 is a tuple of relationships between SA concepts of the Ukrainian language; 𝐶𝑜𝑛𝑐𝑒𝑝𝑡𝑠 is a tuple of SA concepts describing the rules of the Ukrainian language; 𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛𝑠 is a tuple of functions for the interpretation of concepts/rules of the Ukrainian language.
The taxonomy of concepts sets the syntax of the language as the root concept of the ontology:
The optimal definition of the tuple of relations between these concepts and the tuple of the rules of the Ukrainian language, formalized by the descriptive logic of DL, will allow effective processing of Ukrainian texts: 𝐶𝑜𝑛𝑐𝑒𝑝𝑡𝑠 =< 𝑅 𝑀𝑟𝑝 , 𝑅 𝑃𝑛𝑐 , 𝑅 𝑆𝑡𝑟 , 𝑅 𝑆𝑛𝑡 , 𝑅 𝑆𝑚𝑛 >, (21) where tuples of concepts of morphology 𝑅 𝑀𝑟𝑝 , punctuation 𝑅 𝑃𝑛𝑐 , structure 𝑅 𝑆𝑡𝑟 , syntax 𝑅 𝑆𝑛𝑡 (Fig. 25) and semantics 𝑅 𝑆𝑚𝑛 .
In SEM, to identify the set of semes of the corresponding text and their relationship, first, based on the results of SYA, a semantic graph of the relations of linguistic units is built, taking into account the parts of the language of words:
where 𝐶 𝑊𝑟𝑑𝐶𝑚𝑏 is a tuple of word formation concepts; 𝐶 𝑆𝑛𝑡𝐶𝑚𝑏 is a tuple of sentence generation concepts in the Ukrainian language (Fig. 26).
Tuple 𝐶 𝑊𝑟𝑑𝐶𝑚𝑏 according to the rules of the Ukrainian language syntax (Fig. 26): Similarly, tuples are formed to identify the members of the sentence 𝑆𝑔𝑛 𝑆𝑛𝑀𝑏 𝑆𝑛𝑡 (Fig. 28-Fig. 29) and the complex sentence 𝑆𝑔𝑛 𝐶𝑙𝑆𝑡 𝐼𝐼𝐼 (Fig. 30). The process of extracting data from the Ukrainian-language text based on the syntax ontology allows you to supplement the conceptual weighting graphs of the content.
Pragmatics examines the dependence of meaning on the context of the textual content of the author and takes into account his prior knowledge, intentions, purpose, etc., in contrast to semantics, which analyzes the meaning itself depending on the results of GA, MA, LA and SYA within a particular text. Pragmatics is a continuation of SEM, taking into account the peculiarities of the context of the analysed text, taking into account the ambiguity of the statements of the analyzed text, based on the analysis of the features of the author's statements in previous similar texts, based on the time, place, method, purpose and other circumstances of the conversation.
In PA, when resolving the ambiguity of the author's speech in a specific analyzed text, taking into account the features of the author's speech in previous similar speeches, it is best to use word prediction models, for example, N-grammatical Language Models (LM). Each speaker, as a person with a unique life experience, has not only his dictionary of thematic words but also a unique handwriting of the use of these words and their sequence in a certain context of the relevant thematic direction. In the expression «лінгвістична система опрацьовує …» [linhvistychna systema opratsʹovuye …] (the linguistic system processes ...) the next word depends not only on the context but also on the so-called speech handwriting of the author of the text: текст, контент, текстовий контент, вхідні дані, вхідну інформацію, інтегровані дані, авторський контент, публікації [tekst, kontent, tekstovyy kontent, vkhidni dani, vkhidnu informatsiyu, intehrovani dani, avtorsʹkyy kontent, publikatsiyi] (text, content, text content, input data, input information, integrated data, author content, publications), etc. The phrase «включіть свою виконану лабораторну роботу ...» [vklyuchitʹ svoyu vykonanu laboratornu robotu ...] (include your completed lab work...) as opposed to «додайте свою виконану лабораторну роботу ...» [dodayte svoyu vykonanu laboratornu robotu ...] (add your completed lab work...) has a broader meaning and depends significantly not only on the context but also on the speaker (include can mean like download the developed software on the computer or in the sense of adding it as an item to some list, etc.). Dialogue participants intuitively understand the content based on their experience of communicating with the author of the phrase. Pragmatic analysis requires the introduction of models that determine the probability for each subsequent word. They are also intended for assigning the probability of the target utterance for correct machine translation, identification/correction of grammatical and stylistic errors, and handwriting or language recognition. Each language has special statistical parameters, and the analysis of the probability of the appearance of only letters and their combinations as N-grams of the corresponding language makes it possible to identify the language itself or the style of the author (Fig. 31 -with greater probability, the author of the benchmark wrote Excerpt 1).
Figure 31: Probability of appearance of letters in the standard and analyzed passages For Ukrainian texts, the statistical parameters of styles are the probabilities of vowels, consonants, and gaps between words, as well as soft and sonorous groups of consonants. Probability is also important for enhancing communication. Physicist Stephen Hawking used simple movements to select words from a menu for speech synthesis. For such IS, it is appropriate to use word prediction to generate suggestions for a list of likely words for the menu. One of the most widespread and easiest to implement for English-language texts is LM -N-gram, which assigns probabilities to sentences or sequences of words. For Ukrainian-language texts, it is better to apply such LM to the sequence of word bases without taking into account inflexions (otherwise incorrect PA results will be obtained) to calculate 𝑃(𝑏|𝑎) is the probability of the appearance of the base of the word 𝑏 after the sequence of bases 𝑎. Taking into account words in N-grams of LM in Ukrainian-language texts is appropriate for identifying grammatical errors. 𝑃(систем|комп ′ ютер лінгвіст), 𝑃(системи|комп′ютерні лінгвістичні), 𝑃(систему|комп ′ ютерну лінгвістичну).
(37)
One of the best ways to calculate such a probability is to conduct a statistical analysis on large corpora of texts of the relevant author or relevant thematic direction from reliable Internet sources.: .
(
This gives a probabilistic result for a certain period because the language is creative, not homogeneous, and the vocabulary is updated and constantly develops both in general and for a specific speaker -the author of the text. To analyze the corresponding random linguistic event 𝐴 𝑖 = комп ′ ют, 𝑃(𝐴 𝑖 ) is found to calculate the probability of the appearance of a certain sequence of linguistic events based on the chain rule or the general product rule (chain rule of probability): .
(41)
The chain rule reflects the relationship between the overall probability of the appearance of a specific sequence of bases and the conditional probability of the appearance of a word base by specific previous word bases in this sequence. Taking into account the entire dynamics of the occurrence of all word bases in the text to sequences of other word bases is a redundant/inefficient process due to the variability of language/speech over time. Prediction of the 2-gram model consists of approximating the dynamics of the appearance of only the last few bases of words in a given sequence:
For example, for three sentences of the mini-corpus (conditionally, the <p> </p> tags are the boundaries of one sentence), we will calculate the Markov assumption of the 2-gram occurrence of word bases: With each subsequent multiplication, the probability decreases. Applying the logarithm of probabilities (log probabilities) will allow you to operate with not-so-small values for calculating accuracy. (48)
The resulting matrices will in most cases be sparse. Phrase and different variations (plural/singular and cases) система електронної контент-комерції [systema elektronnoyi kontent-komertsiyi] (electronic content commerce system):
𝑃(систем електрон контент комерц) = = 𝑃(електрон|систем)𝑃(контент|електрон) 𝑃(комерц|контент) = =0,1240,810,179=0,01797876.
The general architecture of computer linguistic systems is developed based on the main processes of processing information resources such as integration, maintenance and content management, as well as using methods of intellectual and linguistic analysis of text flow using machine learning technology. The IT of intellectual analysis of the text flow based on the processing of information resources has been improved, which made it possible to adapt the generally typical structure of content integration, management and support modules to solve various NLP problems and increase the efficiency of CLS functioning by 6-9%. This became possible thanks to the combination of linguistic analysis methods adapted to the Ukrainian language, improved IT processing of information resources, ML and a set of metrics for evaluating the effectiveness of CLS functioning. The main principle of building such CLS is modularity, which facilitates their construction according to the requirements for the availability of appropriate processes for solving a specific NLP problem. The main NLP methods based on regular expression matching with patterns in grapheme and morphological analyses of Ukrainian-language texts are described. NLP methods based on pattern-matching regular expressions have been improved, which made it possible to adapt methods of text tokenization and normalization by cascades of simple substitutions of regular expressions and finite state machines. The main valid operations of regular expressions are defined as union and disjunction of symbols/strings/expressions, number and precedence operators, as well as anchors as special symbols for identifying the presence/absence of symbols in RE. The main stages of tokenization and normalization of the Ukrainian text by cascades of simple substitutions of regular expressions and finite state machines are defined. The MA method of the Ukrainian-language text based on word segmentation and normalization, sentence segmentation and modified Porter's stemming algorithm was improved as an effective means of identifying lem affixes for the possibility of marking the analysed word, which made it possible to increase the accuracy of keyword searches by 9%. Algorithms for word segmentation and normalization, sentence segmentation, and Porter's modified stemming are implemented and described as an effective way of identifying lem affixes for the possibility of marking the analysed word. Unlike the classic Porter algorithm (it does not have high accuracy even for English-
language texts), the modified one is adapted specifically for the Ukrainian language and gives an accurate result in 85-93% of cases, depending on the quality, style, genre of the text and, accordingly, the content of CLS dictionaries. The algorithm for the minimum editorial distance of lines of Ukrainian texts is described as the minimum number of operations necessary to transform one into another.