Phrasal patterns, such as 'in this paper we propose', are often used in scientific papers. These are called formulaic expressions (FEs) and constitute sentential communicative functions (CFs) that convey how a sentence should be read by the readers. FEs are useful for scientific paper analyses and academic writing assistance, but FE extraction methods have thus far not been investigated in detail. In this paper, we propose a sentence-level FE extraction method in which the CFs are taken into account. The proposed method is compared to existing methods to demonstrate that it is better at CForiented FEs.
In scientific papers, the authors often use several fixed phrasal patterns that are specific to the genre, such as ‘in this paper, we propose’. These patterns are called formulaic expressions (FEs) or formulaic sequences. FEs convey the intentions of the authors to the readers, i.e., the manner in which a sentence should be understood. This characteristic of the FE is called communicative function (CF). For example, the phrase ‘in this paper, we propose’ conveys the CF of the sentence meaning ‘showing the aim of the paper’. FEs are useful for understanding the composition of a scientific paper and are helpful in writing the paper.
A few studies have been reported on addressing the extraction of FEs and subsequent assignment of CF labels to them (Cortes 2013; Mizumoto, Hamatani, and Imao 2017). However, these works have not rigorously investigated whether the extracted FEs convey the CFs of a sentence. Extracting word n-grams with frequency thresholds has been reported in several studies, although frequent FEs do not always convey the sentential CFs. Machine-learning approaches have hitherto been scarcely adopted because of the dearth of sufficient FE-annotated resources.
In this paper, we propose a new sentence-level FE extraction method and compare it to several existing methods. We assume that a single FE is extracted from each sentence because it conveys the entirety of the CF of that sentence. The proposed method consists of two steps. First, the named and scientific entities are removed from the sentence. Second, two types of n-grams are extracted from the sentence.
Then, the extracted FEs are evaluated based on whether they convey the sentential CFs. The results of manual evaluations show that the proposed method can extract more FEs representing the CFs of sentences than existing methods.
Considering the compilation of a list of FEs, which will be a possible application of the FE extraction, removing noisy FEs and enhancing precision is important. Thus, we test how effective filtering FEs based on the number of occurrence of an FE is, and show that it improves precision much. 2
We used a CF-labelled sentence datasets that were made from scientific papers of four disciplines: computational linguistics (CL), chemistry (chem), oncology (onc), and psychology (psy). Each discipline consists of four sections; introduction, methods, results, and discussion; thus, 16 datasets were used (combination of four disciplines and four sections). The numbers of sentences and words in these datasets are listed in Table 1. Compared with some of the existing studies, in which the sizes of the corpora were around 2 million (Simpson-Vlach and Ellis 2010) or 8 million (Mizumoto, Hamatani, and Imao 2017) words, we determined that the datasets are sufficient.
3
3.1
Two main approaches were considered here for
extracting the FEs: corpus- and sentence-level approaches. In the
corpus-level approach, the FEs are extracted from the entire
corpus, whereas in the sentence-level approach, a single FE
is extracted from each sentence (Figure 1). The corpus-level
approach may cause problems with deciding the FE size and
overlap between FEs (Iwatsuki
Frequent N -grams Word n-grams were extracted from the dataset, and depending on the frequency-based threshold, the infrequent FEs were removed. Although various studies have used different lengths and frequency thresholds for the n-grams, we extracted FEs whose lengths were three words or greater, and followed the method in Cortes (2013) for the frequency thresholds: 20 per million words (pmw) for four-word or shorter n-grams, 10 pmw for fiveword phrases, 8 for six- and seven-word phrases, and 6 pmw for phrases longer than seven words.
Word embeddings play an important role in natural language processing. 0 0 1 1 1 1 1 0 0 0 0: non-formulaic 1: formulaic
play an important role in
Lattice FS This approach was originally proposed by
Frequency-Based Filtering Based on the frequency, each word of a sentence is labelled as either formulaic or nonformulaic. Non-formulaic words are removed, and the remaining words are regarded as the FE. We used two frequency thresholds, namely 1/50,000 and 1/100,000 words. LDA-Based Filtering Liu et al. (2016) proposed utilising latent Dirichlet allocations (LDA) because they assumed that topic-specific words do not comprise FEs. Thus, each word of a sentence was judged as either topic-specific or topicindependent based on the following criterion:
P(w) = 1 max pw(i) P pw(i) ; where pw(i) is the probability of the word w in a topic i. If P (w) is greater than the threshold, w is formulaic. We use P (w) > 0:65 and 10 topics, which was reported optimal. Proposed Method The proposed method comprises two steps: (1) removing named and scientific entities and (2) extracting longest word n-grams (Figure 2). The first step was based on the idea that the named and scientific entities, including places, organisations, materials, and methods, such as ‘Helsinki’ and ‘word embeddings’, do not constitute FEs. In the second step, dependency parsing was applied to the sentences to determine their roots. After removing the named and scientific entities, two types of word n-grams were labelled as formulaic: 1. the longest word n-gram satisfying a frequency threshold; 2. the longest word n-gram that contains a root of the sentence and satisfies the frequency threshold.
If multiple FEs of the same lengths were found, the most frequent one was prioritised.
We focused on the longest word sequences because Cortes (2013) observed that lengthy FEs, such as ‘the rest of the paper is organized as follows’ existed. Additionally, we assumed that in several cases, sentential CFs were realised around the root of the sentence, so that two types of n-grams should be extracted. Specifically, n-grams whose lengths were less than three words were ignored because such FEs would be too short. The remaining words in the sentence after n-gram extraction were removed. The frequency threshold was thus set to 3 to collect the maximum number of FEs. The entity removal was conducted with ScispaCy (Neumann et al. 2019).
In the example in Figure 2, the root word is ‘show’. The longest n-gram satisfying the threshold and containing the root would thus be ‘the results show that’, while ‘is significantly better than’ would be another n-gram that does not contain the root. There could also be cases where these two types of FEs overlap or be the same.
The results show that the BERT classifier is significantly better than the SVM classifier.
Entity removal The results show that the BERT classifier is significantly better than the SVM classifier. longest n-gram with root
longest n-gram n-gram extraction
The results show that * is significantly better than For compiling a list of FEs, which is one of the applications of the FE extraction, it is not always necessary to use all these FEs extracted from every sentence. It is more important to discard non-FEs. Because the word sequences that occur only once or twice are not formulaic, filtering FEs based on the number of the occurrence is effective. Therefore, we set thresholds of the number of FE occurrence in the dataset, and removed FEs not satisfying the thresholds. 4
We randomly chose 100 sentences from the sentence dataset to evaluate the FE extraction. For the sentence-level methods, a single FE was extracted from each sentence. For the corpus-level methods, the FEs and sentences were not clearly connected. Thus, we randomly selected a single FE from the set of extracted FEs for each sentence.
The evaluations were then conducted manually. Three annotators were asked to check if the FEs extracted with each method had the same CFs as the sentences from which they were extracted and if these were reusable when writing scientific papers. The FEs were presented to the annotators simultaneously, and the method that was applied to the FE was not disclosed. A total of 100 combinations of sentences and FEs were randomly selected for evaluations.
The results of the evaluations are shown in Table 2, and the proposed method is observed to show clear advantage over other baselines in the FE extraction.
Frequent n-grams Lattice FS Frequency-based (1/50,000) Frequency-based (1/100,000) LDA-based Proposed (Step 1) Proposed (Step 2) Proposed (Step 1+2) 2/3 0.30 0.07 0.04 0.05 0.08 0.13 0.54 0.58 3/3 0.09 0.03 0.02 0.02 0.03 0.05 0.28 0.39 0.36 0.30 -0.36 -0.39 -0.20 -0.27 0.23 0.44
Table 3 shows the thresholds of the number of occurrence of FEs and scores. From the table, it can be seen that if FEs occurring less than three times in a corpus are ignored, the precision would change from 0.39 (39/100) to 0.49 (24/53). It should be noted that the recall cannot be calculated because there are no available FE-annotated resources.
We analysed the errors (FEs that 1/3 or less annotators judged as correct) in the proposed method. The errors in the entity recognition (step 1) accounts for approximately 60% of all the errors. They can be classified into two types: (1) entities are not removed and (2) formulaic words are removed as entities though they are not entities. Most of the errors were the type (2).
Table 4 lists the examples of this error. From this table, it can be seen that formulaic words such as ‘table’ and ‘investigated’, which are indispensable for representing the CFs, were removed. When formulaic words are removed at this stage, meaningful n-grams are not to be extracted in the step 2. This results infer that entity recognition is crucial to the proposed method and should be improved much. Another type of errors is the errors in the n-gram extraction (step 2). In the proposed method, we extracted two different n-grams: the longest n-gram containing the sentential root and the longest n-gram that does not necessarily contain the root, both of which satisfied the threshold of the number of occurrence in the corpora.
The majority of this error is that the extracted two ngrams are the same but do not contain CF-realising part. Table 5 lists the examples of this error. The span error occurred in the second example. Since ‘both plasma and urine’ is content part, the FE should not include ‘both’. The other examples missed the CF-realising part. In the first example, ‘a common approach’ is important to the introduction to the methodology. In the third example, detail number was extracted. It should be noted that the numbers sometimes constitute an FE because in some disciplines, there exist very fixed numbers, such as ‘a p value less than 0.05 was considered significant’. In the fourth example, the FE missed ‘as proposed by’ to show the method was used in past work. In the last example, the controversy is represented by ‘has been challenged’, which was not extracted.
The last example also shows that n-grams that contain the sentential root do not always convey the CF. It is true that the that clause conveys the CF showing controversy within the field, but the phrase in the main clause ‘it should be noted that’ may have a different CF. This is a limitation when a sentence is regarded as a unit of a CF because a long sentence may have more than one CF. However, it is difficult to determine the length that constitutes the unit of a CF.
Full sentence From this table, we observe that the topics learned by our method are better in coherence than those learned from the baseline methods, which again demonstrates the effectiveness of our model.
Although the cellular uptake efficiency could be improved by adjusting the size and the sequence of DNPs in the previous study, it has not been investigated whether the DNPs can also be used in the in vivo environment rich in nucleases.
Sentence without entities from this * we observe that the topics learned by our * are better in * than those learned from the * which again demonstrates the * of our although the * could be improved by adjusting the * and the * of * in the previous * it has not been * whether the * can also be used in the * rich in
A common approach used to assign structure to language is to use a probabilistic grammar where each elementary rule or production is associated with a probability.
For example, shared specific genomic aberrations were observed in both plasma and urine cfDNAs at loci of PTEN, TMPRSS2 and AR (Figure 1 and [CITATION] ).
Rs679620 was also associated with increased OA risk in dominant (“TC-TT”, OR = 2.03, 95% CI: 1.03-4.01, P = 0.038) and overdominant model analyses (“TC”, OR = 2.04, 95% CI: 1.05-3.96, P = 0.033).
The smoothness value used for the AlphaSim calculation was based on the smoothness of the residual image of the statistical analysis as proposed by [CITATION] .
However, it should be noted that the biological involvement of many of these targets in HBD-3 activities has been challenged in recent years [[CITATION] ].
FE is to use a were both p 0038 and
observed in was based on the however it should be noted that the The existing FE extraction methods have different drawbacks. Table 7 lists the number of FEs extracted with the sentence-level methods after removing infrequent FEs occurring less than three times in the corpus. Compared to the proposed method, these methods extracted smaller numbers of FEs because most of these FEs rarely occur in the corpus. An example of sentence-level extraction is illustrated in Figure 3. The existing methods do not remove the nonformulaic words sufficiently here because the focus is only on a single word, and words such as ‘in’ or ‘results’ do not always constitute the FE.
The corpus-level methods are different in this regard. The numbers of extracted FEs are 23,847 (frequent n-gram) and 2,480,935 (Lattice FS). The frequent n-gram method extracts a smaller number of FEs because of the frequency Original sentence Frequency (1/50,000) Frequency (1/100,000)
LDA-based
In order to avoid over fitting, PA with PCA was chosen for this study. in order to avoid over fitting pa with * was chosen for this study in order to avoid over fitting pa with pca was chosen for this study in order to avoid over fitting * with * chosen for this study Proposed
in order to avoid * was chosen for this thresholds. Further, it achieves a relatively good quality score, which is still lower than that of the proposed method (Table 2). The Lattice FS extracts too many FEs, which can deteriorate the quality of the FEs.
6
In this paper, we proposed a new sentence-level FE extraction method to realise CF-oriented analysis. We compared CF Showing limitation or lack of past work Comments on the findings Showing explanation or definition of terms or notations Unexpected outcome Describing interesting or surprising results Summary of the results Comparison of the results Showing the limitation of the research Showing the characteristics of samples or data Showing reasons why a method was adopted or rejected Showing methodology used in past work Suggestion of hypothesis Showing the outline of the paper Showing the aim of the paper Suggestion of future work Explanation for findings Showing criteria for selection Showing the main problem in the field 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the proposed method to four existing methods, and our manual evaluations showed that the proposed method extracted CF-realising FEs better than these other methods. Although FE extraction has not been discussed in detail thus far in reported literature, we showed the existence of a more robust method than just extracting frequent n-grams, as adopted in the past studies. The FEs extracted with the proposed method are provided at our website2 for utilisation in various tasks, such as information extraction and computer-based academic writing assistance.
This work was supported by JSPS KAKENHI Grant Numbers 19J12466 and 18H03297. Cortes, V. 2013. The purpose of this study is to: Connecting lexical bundles and moves in research article introductions. Journal of English for Academic Purposes 12(1): 33–43. doi:10.1016/j.jeap. 2012.11.002.
Iwatsuki, K.;