Benchmarking Natural Language Processing Algorithms for Patent Summarization

Abstract

The number of patent applications is enormous, and patent documents are long and complex. Methods for automatically obtaining the most salient information in a short text would thus be useful for patent professionals and other practitioners. However, patent summarization is currently under-researched; moreover, the proposed methods are dificult to compare directly as they are generally tested on diferent datasets. In this paper, we benchmark several extractive, abstractive, and hybrid summarization methods on the BigPatent dataset, compare automatic metrics and show qualitative insights.

Keywords

• eol>Summarization
• Patents
• Natural language processing
• Natural language generation

1. Introduction

proaches in the patent domain, specifically on the BigPatent [ 4 ] dataset. The dataset is popular in the NLP community, as patents present several challenges in terms of abstractivity, length, and language, among others; moreover, while not exempt from design issues, it is also one of the few patent benchmarks that allow for a direct comparison between approaches. We evaluate extractive, abstractive, and hybrid methods; we also explore transferring summarization methods from the scientific paper domain [ 5 ] with limited success. For each method, we discuss strengths and limitations, provide standard summarization metrics and qualitative insights.

Patents protect inventions that their holders consider important enough to take legal action to obtain the monopoly in using, making, and selling them — and thus, profit from their wit. Thus, they help in valuing intellectual work. At the same time, inventors must disclose the invention and its characteristics in detail to file a patent application: thus, patents are intended to benefit society and help new knowledge spread — correcting the tendency to keep valuable technical details secret. Patents, however, are dificult to process: the number of patent applications is enormous and patent documents are long and hard to read, rich in technical and legal language.

To this end, tools that automatically extract or gener- 2. Previous work ate summaries from patent documents can be particularly valuable in helping patent agents, R&D groups, and other 2.1. Automatic text summarization professionals; using summaries instead of the whole doc- Methods for text summarization are generally classified ument can also improve the performance of automatic into extractive, abstractive, and hybrid ones. processes, as shown in other domains [ 1, 2 ]. In extractive text summarization, a subset of sentences

In the general domain, summarization tools and from the source document is chosen as the most represenmethodologies have shown promising results; applica- tative, and the final summary is a simple concatenation of tions to the patent domain are, however, still relatively such sentences. Methods can be graph-based [ 6, 7, 8 ], rely limited. Moreover, while previous work has explored on token frequency [ 9 ], or on learned intrinsic features methods for automatically generating patent summaries, [ 10, 11, 12 ]. these methods are hard to compare, as no generally ac- In contrast, abstractive text summarization aims at cepted benchmarks exist; thus, conclusions on the pros generating a new piece of text based on the source, simand cons of each approach are hard to make. Even the ilar to what a person would do, and can contain novel most recent abstractive dataset presents important limi- vocabulary or expressions. Sequence-to-sequence modtations and issues that might make direct comparisons els [ 13, 14, 15, 16, 17 ] are popular for this task, with meaningless [ 3 ]. transformer-based ones being particularly performative

To partially fill this gap, we benchmark existing ap- [18, 19, 20, 21]. Finally, hybrid methods try to fuse both PatentSemTech’23: 4th Workshop on Patent Text Mining and Semantic approaches, for example, by extracting and rewriting Technologies, July 27th, 2023, Taipei, Taiwan sentences [22].

Extractive models are generally simpler than abstrac* Corresponding author. $ [e-mail] (S. Casola); [e-mail] (A. Lavelli)

© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License tive ones and require fewer computational resources and CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) data; however, summaries have to contain complete sentences from the source, which often contain both central and peripheral information. Moreover, the final summary is a simple concatenation of sentences, with pending references and no discourse structure. Abstractive summaries are more similar to those written by humans.

Information can be easily condensed and the generated text is much more natural and easier to read. However, abstractive models might produce non-factual information, i.e. include statements that are not in the source or that directly contradict them. See, e.g., [23] for a comprehensive survey of summarization techniques.

2.2. Patent summarization

Summary

Source # docs # tokens (avg) # sents (avg) sent len (avg) # tokens (avg) # sents (avg) sent length (avg) compression ratio 258,935 121.0 3.6 43.4 4,893.6 161.2 31.3 45.8

Many traditional approaches for patent summarization have been extractive. The document is often segmented tains the full Description with all its subsections, in the into sentences or fragments [24] and preprocessed (e.g., original casing. We will use this version in this paper. to keep specific parts of speech only [ 25, 26]); features However, we notice two main limitations. First, patents can then be extracted. General-domain ones include key- lack section headers due to the performed preprocessing; words [27], title words, cue words, and position. An thus, any structural information is lost. Second, the input anthology for technical terms might also be used [26, 28]. often contains the author’s Summary of the Invention, Domain-specific approaches [ 24] are often linguistically- which significantly simplifies the task. motivated. Once extracted, features are used to score the To solve both issues, we download the raw data and (i) sentence relevance in the summary either heuristically apply all original preprocessing steps, excluding remov[27] or in a data-driven way [24, 29, 30, 31]. Alternative ing the subsection headers and newlines, and (ii) remove approaches use the patent discourse structure, which the Summary of the Invention section by heuristically they prune [32]. matching headers. Table 1 contains some metrics on our

Recently, [ 4 ] introduced the BigPatent dataset, whose version of the dataset. associated task is that of summarizing the patent’s Detailed Description into its Abstract. As authors show, patents’ Abstracts are highly abstractive — with relevant 4. Evaluation Protocol content spread throughout the input — and have many novel n-grams. The dataset has been used as a testbed Evaluating patent summarization results is challenging. for general-purpose systems [19, 33, 34, 21, 35], given On the one hand, automatic text simplification outputs the high abstractivity of its targets and the length of its (and Natural Language Generation outputs in general) inputs. are dificult to evaluate automatically, and the problem

For an overview of patent summarization approaches, is considered open [37, 38]. see [36]. While automatic metrics such as ROUGE [39] exist, they have known limitations. In the patent domain in particular, some previous work [40, 31] has anecdotally 3. Dataset questioned the metric validity (and its correlation to expert’s opinion and practical utility), even if no quantitaWe use the G (Physics) subsection of the BigPatent dataset tive studies in the patent domain have been performed, [ 4 ]. to the best of our knowledge. More complex metrics, e.g.,

The dataset is associated with the task of generating model-based methods [41, 42], should be fine-tuned with the patent Abstract from its Description. We are aware domain-specific data. of the practical imitations of this setting, as the Abstract On the other hans, human evaluation is not easier. contains superficial and general information, but still In fact, it is particularly hard in the patent domain for consider experimenting on the dataset useful given its two main reasons: a) the best way to evaluate a summapopularity in the Natural Language Processing commu- rization output is to read the whole source document. nity. However, patents are extremely long and hard to read; b)

The dataset exists in two versions [ 3 ]. The original ver- patent documents and Abstracts are extremely complex sion text is uncased and tokenized, and its input typically and should be evaluated by legal and technical experts, contains the Detailed Description only (i.e., a subsection but hiring such experts is very expensive and unpractical of the Description section). The alternative version con- in most scenarios.

Aware of these limitations, we will use two main evaluation methods: • Automatic evaluation: we will select hyperparameters and automatically evaluate outputs using ROUGE [39]. We also experimented with factuality-related metrics, e.g., QAEval [41]; however, they do not seem to adapt well to the patent domain and should be fine-tuned. • Qualitative evaluation: we report a preliminary qualitative evaluation of a subset of candidate summaries. We will consider the patent fluency, consistency, and similarity to the Abstract.

5. Extractive methods

5.1. Graph-based systems

The core idea of graph-based methods is to represent the original document as a graph having sentences as nodes and their similarity as edges, and then extract the most central sentences only. 5.1.1. TextRank [ 7 ] TextRank uses the number of shared words among two sentences, normalized by the length of the sentences as its similarity metrics. Edges in the complete graph are then pruned using a threshold, and the most central sentences according to PageRank [43] are extracted. We used the summa1 implementation. In this implementation, the user chooses the target summary length in terms of tokens, and the number of sentences that best approximate that number is extracted. We cross-validated the number of tokens and left any other parameters at their default values. Some sample outputs are in Table ??.

Set Val Val Val Val Val Val Test ROUGE-1 26.03 34.72 37.48 37.76 36.92 35.62 37.76

ROUGE-2 8.12 10.93 12.02 12.40 12.46 12.36 12.46

Qualitative assessment The outputs obtained using the two algorithms are relatively similar. We notice that the sentence tokenization is not always perfect: for example, the extracted summary of patent US-2005152022-A1 5.1.2. LexRank [ 6 ] contains the sentence "The mixed color display [...] by the LexRank is similar in nature to TextRank, but it uses type of processes described in the aforementioned U.S. Pat. the cosine similarity of their Term Frequency–Inverse No.", where the patent number has been incorrectly conDocument Frequency (TF-IDF) representation as its simi- sidered as a stand-alone sentence. This is in accordance larity metrics. We used the sumy implementation2. We with previous work [44, 45], which showed that generalvalidated the number of extracted sentences per patent domain Natural Language Processing resources tend to and left any other parameters at their default value. The have suboptimal performance in the patent domain and algorithms are unsupervised and can easily be used even should be adapted. for very long documents with no modifications. We also Moreover, sentences naturally contain references to tried to perform experiments with PacSum [ 8 ] but found other parts of the original text3 e.g., "as described bethe algorithm extremely computationally demanding in low" in US.2005152011-A1 or "according to claim 1" in our use case. US-9478115-B2.

We also notice that all the extracted sentences tend to be extremely long and naturally contain core and peripheral information (e.g., included in parenthesis). These 1https://summanlp.github.io/textrank/ 2https://github.com/miso-belica/sumy 3Sentences also tend to contain numerical references to the figures, which are lost. #S 1 2 3 4 5 6 7 8 5

ROUGE-1 20.09 28.51 32.37 33.93 34.28 34.00 33.30 32.44 34.26

Automatic evaluation Table 4 shows the ROUGE scores. LSA tends to perform worse than the graph-based algorithms. In contrast to the graph-based methods, it tends to work best when extracting several short sentences.

Qualitative assessment Even with the known limitations of extractive systems (references, structure, senare known limitations of naive extractive models and are tences needing compression, etc.), some reasonable convery common problems of our extracted summaries. Ex- tent selection is performed. For example, they often extracted sentences do not seem too similar to each other, tract the sentence that describes the invention’s nature, which is sometimes described as a limitation of graph- as in “The present invention is based on the object to probased systems. vide an operator system for a machine, which is ergonomic

Even with their limitations, the algorithms seem to per- with regard to the handling thereof and ofers suficient form reasonable content selection (with TextRank being work protection." for US-9478115-B2 or “The present insuperior to LexRank also from a qualitative perspective); vention relates to computer security and, more particularly, when compared to their references, the extracted sum- to an eficient method of screening untrusted digital files." maries often contain most of their core elements and, in for US-9208317-B2. Sentences are generally shorter than many cases, are very similar to the reference in terms those extracted by graph-based systems. of content. This is evident in some specific cases (e.g., [31] noticed LSA showed a better quality when compatent US-9478115-B2 and US-2003016244-A1) and is in- pared to TextRank in the generation of patent titles. Our teresting, considering the algorithm is unsupervised. results do not confirm this finding for Abstract gener

If we assume the final target of the extracted sum- ation from the Description as measured automatically; maries is human readers, the lack of discourse structure qualitatively, the results are relatively diferent and might and the length of the extracted sentences might make be used for diferent purposes. the outputs too hard to understand. It might, however, be possible to use the outputs in an ad hoc interface, e.g., where core sentences are highlighted. 6. Abstractive methods

5.2. Latent Semantic Analysis

Latent Semantic Analysis [ 46 ] aims at exploiting the latent semantic structure of the document and extracts sentences that best represent the most important latent topics. The algorithm decomposes the term-sentence matrix constructed from the source document using SVD [ 47 ]. The × terms-by-sentence matrix is thus decomposed as = Σ . Thus, the original matrix is decomposed into a matrix of term distributions over latent topics, a diagonal matrix of topic importance (the singular values), and a matrix of topic distributions across sentences. For each of the most salient latent topics (i.e., those corresponding to the largest singular values), the sentence with the largest index value is included in the summary [ 48, 10 ]. We use the sumy implementation, validate the number of sentences, and leave all other parameters at their default values. Some sample outputs are in Table ??.

We use BART [18], a sequence-to-sequence system, as a baseline for abstractive summarization. We fine-tune a BART-base model (∼ 140 million parameters) on the BigPatent/G datasets. We train using the Hugging Face library with early stopping on the evaluation loss (patience: 5) and the following hyperparameters: max target length: 250; number of beams: 5; evaluation steps: 10k; max steps: 500M. We leave all other parameters at their default values. Some sample outputs are in Table ??. Automatic evaluation Table 5 shows the results in terms of ROUGE. As expected, the results improve over all extractive systems, with an increase of almost 5 ROUGE-2 points over the best extractive system. Qualitative assessment Qualitatively, we notice that summaries are generally grammatical, with very rare local problems. Text is coherent and much easier to read and understand than those composed through extracted

Coverage (avg)

Density (avg)

BART

95.75 11.84 sentences. In all cases, summaries seem adequate and Thus, in this section, we explore a hybrid approach. We convey the main points of their gold standard counter- first select important sentences using an unsupervised parts. graph-based algorithm and then rewrite the content us

However, we noticed that the generated summaries ing an abstractive system. Specifically, we use TextRank are largely extractive, with no or few modifications to as it performed best among the considered extractive sentences in the source. In the following example, the models. We considered three extracted lengths: 1000, extractive fragments in the summary generated for patent 500, and 250 tokens. Then, we train a BART system to US-2005152022-A1 and its source (Background of the rephrase the selected sentences to generate the target Invention subsection) are underlined. summary: we use the selected sentences as the input and

More specifically, in one aspect this invention relates the original gold standard as the target and fine-tuned to electro-optic displays with simplified backplanes, and the model. Some sample outputs are in Table ??. methods for driving such displays. In another aspect, this invention relates to electro-optic displays in which mul- Automatic evaluation Table 7 reports the ROUGE tiple types of electro-optic units are used to improve the scores. Extracting 1000 tokens through TextRank and colors available from the displays. The present invention then rephrasing the summary using BART results in the is especially, though not exclusively, intended for use in highest ROUGE, surpassing the vanilla BART approach electrophoretic displays. on all metrics. The obtained metrics are the highest

While some deletion is performed, most text is directly among all the extractive and abstractive models we conextracted from the source. To quantify how extractive sidered. the generated summaries are with respect to the source, Note that, even for the approaches where a smaller we compute the coverage and the density of the gen- number of tokens is extracted, relatively good perforerated summaries, following [ 49 ], which we report in mances are obtained. Extracting 500 tokens results in Table 6. The extractive fragment coverage measures the scores only marginally worse than those obtained by a proportion of tokens in the summary that is part of an BART model fed with the first 1024 subtokens. While reextractive fragment; it roughly measures how much a sults obtained by extracting 250 tokens only score worse summary vocabulary is derivative of a text. The den- in terms of ROUGE, the rewriting component is crucial. sity also takes into account the length of the extractive In fact, an improvement of 5 ROUGE-1, 3.3 ROUGE-2, and fragments: the higher the density, the better a summary 5.3 ROUGE-L points is observed over the results obtained can be described as a series of extractions. We notice using TextRank only. that the generated summaries tend to have much longer abstractive fragments with respect to the gold standard.

Qualitative assessment The outputs obtained with this approach are fluent, and relatively similar to those obtained through the vanilla BART. The coverage and density (Table 6) also show a marginally lower extractivity of the generated summaries.

7.2. DANCER

An alternative approach to deal with high document length is to exploit the document structure. To summarize scientific documents, for example, [ 5 ], proposed to deal with diferent sections independently; however, no experiments were performed in the patent domain.

7. Hybrid methods

7.1. Extractive to abstractive: select and rephrase

Results in the previous sections show graph-based extractive methods tend to be able to select central content but lack any discourse structure. Using BART solved some of these issues, but the model can only summarize the ifrst part of the patent document, as its input length is limited to 1024 subtokens. Here, we explore if adapting this method to the patent domain can be useful.

Specifically, we perform the steps described in the following.

• Dividing and normalizing subsections: To divide the Description text into subsections, we use simple regular expressions, exploiting the fact that section headers lines include fully cased tokens only. Patent headers can follow diferent conventions4. Thus, we normalize the headers through a simple keyword-matching algorithm into nine classes. The classes are shown in Table 8. Subsections that did not match with any of the keywords were left in a default category and ignored. • Alignment between abstract sentences and subsections: Following [ 5 ], we use ROUGE-L [39] to align sentences in the abstract to patent subsections. Specifically, for each sentence in the Abstract, we compute its ROUGE-L recall with all individual paragraphs in all subsections; we then align the sentence with the subsection containing the paragraph with the maximum score5. Figure 3 shows the percentage of subsections that, when present, align with at least one sentence in the patent’s Abstract. • Using paired elements as training data: Following the previous steps, each Abstract sentence is aligned with a Description subsection. Thus, for each (Description, Abstract) pair, we created (Subsection, Abstract sentence(s)) pairs, where is the number of unique subsections that are aligned with at least one sentence in the Abstract. If multiple sentences align with the same patent subsection, the target contains all the aligned sentences in their original order. We then trained a BART-base model [18] using the subsection as input and the aligned sentence(s) as target; we set the maximum generated length to 250, the number of beams to 5, and left all other hyperparameters at their default values. We trained with early stopping on the validation set. Table 9 reports the metrics obtained by the model on the sentence generation step. We also experimented with prepending the subsection type (as a special token) to its text but with no improvement. • Inference: At inference, we obtain the final summary by concatenating the sentences generated from the individual subsections. Patent structure is less coherent than that of papers; in fact, not 4For example, subsections with similar content can be named Fields, Field, Field Of The Invention, etc. 5We retrieve the subsection containing most of the sentence content, regardless of any possible additional text (that the summarization model will learn to filter out).

all subsections appear in all patents. We thus consider several strategies for subsection selection: (i) Pre-selection: We heuristically pre-select subsections based on their role6 and fed them to the trained model in their original order. We then concatenated the results. (ii) Generate from M subsections: We retrieve all subsections in the patent and sort them according to how likely they are to be aligned in the whole dataset (Figure 3). We generate from the first M most commonly aligned subsection, where M goes from 1 to the total number of subsections in the patent. The final summary is a concatenation of the generated sentences. (iii) Generate from all subsections in the patent: we use all subsections in their original order and concatenate the results. • Second abstractive step: The final abstract obtained as a concatenation of sentences lacks any discourse structure and might not be coherent; in particular, we notice that it often contains repeated information. Thus, we explore if performing a second abstractive step can improve performance. To this end, we train a second BART model that, given the output of the previous step (i.e., the summary as a concatenation of sentences), is trained to paraphrase it to be more similar to the target Abstract.

Some sample outputs (before performing the second abstractive step) are in Table ??.

Automatic evaluation Table 10 reports the results on the validation set. We report results obtained by generating from pre-selection, using the best-aligned section only (as a baseline), the best result with a varying number of sections (and Figure 2 shows ROUGE-L as a function of 6We selected the subsections of type FIELD, BACKGROUND, EMBODIMENTS, OBJECTIVE, DESCRIPTION [ name=plotLeft, scale=0.45, tick label style=font=, ylabel near ticks, xlabel style=yshift=2.2ex, xticklabel style=rotate=90, title=Train aligned sections, title style=yshift=-1.5ex,, symbolic x coords=BACKGROUND, DESCRIPTION,

DRAWINGS, EMBODIMENTS, FIELD, OBJECTIVE,

REFERENCES, RELATED ART, xtick=data, ] [ybar] coordinates (BACKGROUND, 48.66) (DESCRIPTION, 87.02) (DRAWINGS, 1.76) (EMBODIMENTS, 83.49) (FIELD, 22.51) (OBJECTIVE, 2.14) (REFERENCES, 2.02) (RELATED ART, 33.65) ; [ at=(plotLeft.right of south east), anchor=left of south west, scale=0.45, tick label style=font=, ylabel near ticks, xlabel style=yshift=2.2ex, xticklabel style=rotate=90, title=Val aligned sections, title style=yshift=-1.5ex,, symbolic x coords=BACKGROUND, DESCRIPTION, DRAWINGS,

EMBODIMENTS, FIELD, OBJECTIVE, REFERENCES,

RELATED ART, xtick=data, ] [ybar] coordinates (BACKGROUND, 49.31) (DESCRIPTION, 86.67) (DRAWINGS, 1.86) (EMBODIMENTS, 83.59) (FIELD, 23.04) (OBJECTIVE, 29.21) (REFERENCES, 1.79) (RELATED ART, 34.74) ; subsections are very similar and describe what the invention is and its goal. While the second abstractive step helps limit repetition, the resulting output is often short and contains too little information compared to the gold standard. We noticed a number of issues that could make the transfer from the scientific publications to the patent domain unsuccessful: • Less predictable structure and session headers: Scientific papers have a very coherent structure as they tend to roughly follow a fixed schema (e.g., Introduction, Previous Work, Method, Conclusions), with each section having a clear fixed role. While, on a superficial level, patent documents have a similar structure with sections and subsections, they are less coherent. As Table 8 shows, the subsections of the Description tend to vary. Moreover, the role of each subsection is less determined. • Less compositional Abstracts: An analysis of the Abstracts’ compositionality shows that many of the sentences in the Abstract align with the same patent subsections. Figure 3 represents the number of unique sentences to which each Abstract aligns. Note most patent Abstracts only align to one or two diferent subsections. Moreover, a qualitative analysis of the Abstract shows that while paper abstracts tend to follow a fixed structure (first describing the background, then the goal and methods, then the results and conclusions), patent Abstracts seem to lack the compositional nature of scientific papers. The lack of a ifxed flow in the Abstract might also explain the relatively low results obtained by the abstractive model when generating the Abstract sentence(s) from the original subsections. As the alignment is more random, finding a pattern and correctly generating the aligned sentences is more challenging.

8. Conclusions

In this paper, we have benchmarked several extractive, abQualitative analysis Inspecting the outputs, we no- stractive, and hybrid methods on the BigPatent/G dataset. ticed that many of the sentences generated from various

Among extractive systems, we found that graph-based ones seem appropriate for content selection and perform relatively well in metrics and outputs. However, the extracted outputs are subject to all the limitations of extractive summarization, with dangling references being particularly common. The length of the sentences, the dangling references, and the lack of discourse structure make the outputs challenging to process for humans and possibly machines.

Among the abstractive approaches, we have analyzed BART and have found that it performs best in automatic metrics compared to extractive algorithms. We have also found that the produced outputs are, in fact, not very abstractive with respect to the input, with long chunks of texts identical to input passages; the model seems, however, very good in removing non-central content from the single sentences, which extractive systems are natively unable to do. In future work, we plan to explore more powerful abstractive models, including those in the GPT family [ 50, 51, 52 ].

We have considered a simple select-and-rewrite approach, which obtained the best automatic metrics. We have also tried to adapt DANCER, initially designed for scientific articles, to the patent domain. However, we have found that patents are more variable in the sections they contain and in the sections’ content itself, and their Abstracts tend to be less compositional than those of papers. Thus, the approach was not particularly successful when transferring to the patent domain.

Our setting, however, has several limitations. First, the BigPatent dataset has known issues, and the Abstract is not regarded as the best target for summarization in the patent community, as it contains superficial information rather than the core invention nature. Second, we did not have the opportunity to collaborate with legal and technical experts to evaluate our outputs.

We believe that future work on patent summarization should tackle a number of open problems. First, we hope that this work will motivate the creation of better benchmarks, which can be shared among researchers and practitioners interested in patent summarization. Second, we hope that the design of such a benchmark can be made in conjunction with patent experts and industrial practitioners to ensure that it can be practically useful; while it is likely not practical to ask experts to write goldstandard summaries, there is space for improvement in the current setting. Third, the validity of the standard evaluation metrics in the patent domains should be measured based on experts’ evaluation of the outputs. Finally, the factual accuracy of abstractive methods — which is particularly important in a legal and technical domain —should be better investigated.

Acknowledgments

We acknowledge the support of the PNRR project FAIR — Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU 10.1016/j.ipm.2016.07.002. Methods in Natural Language Processing (EMNLP), [25] A. Trappey, C. Trappey, B. H. Kao, Automated Association for Computational Linguistics, Online, Patent Document Summarization for R&D Intellec- 2020, pp. 9308–9319. URL: https://www.aclweb.org/ tual Property Management, 2006 10th International anthology/2020.emnlp-main.748. doi:10.18653/ Conference on Computer Supported Cooperative v1/2020.emnlp-main.748.

Work in Design (2006) 1–6. [34] J. He, W. Kryściński, B. McCann, N. Rajani, C. Xiong, [26] A. J. C. Trappey, C. V. Trappey, C.-Y. Wu, A CTRLsum: Towards Generic Controllable Text SumSemantic Based Approach for Automatic Patent marization, arXiv preprint arXiv:2012.04281 (2020). Document Summarization, in: R. Curran, S.-Y. [35] M. Guo, J. Ainslie, D. Uthus, S. Ontanon, J. Ni, Y.Chou, A. Trappey (Eds.), Collaborative Product and H. Sung, Y. Yang, LongT5: Eficient text-to-text Service Life Cycle Management for a Sustainable transformer for long sequences, in: Findings of the World, Springer London, London, 2008, pp. 485– Association for Computational Linguistics: NAACL 494. 2022, Association for Computational Linguistics, [27] Y.-H. Tseng, C.-J. Lin, Y.-I. Lin, Text mining Seattle, United States, 2022, pp. 724–736. URL: https: techniques for patent analysis, Information Pro- //aclanthology.org/2022.findings-naacl.55. cessing & Management 43 (2007) 1216 – 1247. [36] S. Casola, A. Lavelli, Summarization, simpliURL: http://www.sciencedirect.com/science/article/ ifcation, and generation: The case of patents, pii/S0306457306002020. doi:https://doi.org/ Expert Systems with Applications 205 (2022) 10.1016/j.ipm.2006.11.011, patent Process- 117627. URL: https://www.sciencedirect.com/ ing. science/article/pii/S0957417422009356. doi:https: [28] A. Trappey, C. Trappey, C.-Y. Wu, Automatic patent //doi.org/10.1016/j.eswa.2022.117627. document summarization for collaborative knowl- [37] A. Celikyilmaz, E. Clark, J. Gao, Evaluation of Text edge systems and services, Journal of Systems Generation: A Survey, 2020. arXiv:2006.14799. Science and Systems Engineering 18 (2009) 71–94. [38] E. Lloret, L. Plaza, A. Aker, The Challenging doi:10.1007/s11518-009-5100-7. Task of Summary Evaluation: An Overview, Lang. [29] K. Girthana, S. Swamynathan, Query Oriented Resour. Eval. 52 (2018) 101–148. URL: https:// Extractive-Abstractive Summarization System (QE- doi.org/10.1007/s10579-017-9399-2. doi:10.1007/ ASS), in: Proceedings of the ACM India Joint Inter- s10579-017-9399-2. national Conference on Data Science and Manage- [39] C.-Y. Lin, ROUGE: a Package for Automatic Evalument of Data, CoDS-COMAD ’19, Association for ation of Summaries, in: Workshop on Text SummaComputing Machinery, New York, NY, USA, 2019, rization Branches Out, Post-Conference Workshop p. 301–305. URL: https://doi.org/10.1145/3297001. of ACL 2004, Barcelona, Spain, 2004, pp. 74–81. 3297046. doi:10.1145/3297001.3297046. [40] J.-S. Lee, Controlling Patent Text Generation by [30] K. Girthana, S. Swamynathan, Query-Oriented Structural Metadata, Association for Computing Patent Document Summarization System (QPSS), Machinery, New York, NY, USA, 2020, p. 3241–3244. in: M. Pant, T. K. Sharma, O. P. Verma, R. Singla, URL: https://doi.org/10.1145/3340531.3418503. A. Sikander (Eds.), Soft Computing: Theories and [41] A. Wang, K. Cho, M. Lewis, Asking and AnswerApplications, Springer Singapore, Singapore, 2020, ing Questions to Evaluate the Factual Consistency pp. 237–246. of Summaries, in: Proceedings of the 58th An[31] C. M. de Souza, M. E. Santos, M. R. G. Meireles, nual Meeting of the Association for Computational P. E. M. Almeida, Using Summarization Techniques Linguistics, Association for Computational Linguison Patent Database Through Computational Intelli- tics, Online, 2020, pp. 5008–5020. URL: https://www. gence, in: P. Moura Oliveira, P. Novais, L. P. Reis aclweb.org/anthology/2020.acl-main.450. doi:10. (Eds.), Progress in Artificial Intelligence, Springer 18653/v1/2020.acl-main.450.

International Publishing, 2019, pp. 508–519. [42] A. Pu, H. W. Chung, A. P. Parikh, S. Gehrmann, [32] N. Bouayad-Agha, G. Casamayor, G. Ferraro, T. Sellam, Learning compact metrics for MT, in: S. Mille, V. Vidal, L. Wanner, Improving the compre- Proceedings of EMNLP, 2021. hension of legal documentation: The case of patent [43] L. Page, S. Brin, R. Motwani, T. Winograd, The claims, in: Proceedings of the International Con- PageRank citation ranking : Bringing order to the ference on Artificial Intelligence and Law, 2009, pp. web, in: WWW 1999, 1999.

78–87. doi:10.1145/1568234.1568244. [44] A. Burga, J. Codina, G. Ferraro, H. Saggion, L. Wan[33] J. Pilault, R. Li, S. Subramanian, C. Pal, On Ex- ner, The challenge of syntactic dependency parsing tractive and Abstractive Neural Document Summa- adaptation for the patent domain, in: ESSLLI-13 rization with Transformer Language Models, in: workshop on extrinsic parse improvement., 2013. Proceedings of the 2020 Conference on Empirical [45] L. Andersson, M. Lupu, A. Hanbury, Domain

References

[1] I. Mani , D. House , G. Klein , L. Hirschman , T. Firmin , B. Sundheim , The TIPSTER SUMMAC text summarization evaluation, in: Ninth Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics , Bergen, Norway, 1999 , pp. 77 - 85 . URL: https://aclanthology.org/E99-1011.

[2] T. Sakai , K. Sparck-Jones , Generic summaries for indexing in information retrieval , in: Proceedings of the 24th Annual International ACM simplificationR Conference on Research and Development in Information Retrieval , SIGIR '01, Association for Computing Machinery, New York, NY, USA, 2001 , p. 190 - 198 . URL: https://doi.org/10.1145/ 383952.383987. doi: 10 .1145/383952.383987.

[3] S. Casola , A. Lavelli , H. Saggion , What's in a (dataset's) name? The case of BigPatent , in: Proceedings of the 2nd Workshop on Natural Language Generation , Evaluation, and Metrics (GEM), Association for Computational Linguistics , Abu Dhabi, United Arab Emirates (Hybrid) , 2022 , pp. 399 - 404 . URL: https://aclanthology.org/ 2022 .gem- 1 . 34 .

[4] E. Sharma , C. Li , L. Wang, BIGPATENT: A LargeScale Dataset for Abstractive and Coherent Summarization, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Florence, Italy, 2019 , pp. 2204 - 2213 . URL: https://www.aclweb.org/anthology/P19-1212. doi: 10 .18653/v1/ P19 -1212.

[5] A. Gidiotis , G. Tsoumakas , A divide-and-conquer approach to the summarization of long documents , IEEE/ACM Transactions on Audio, Speech, and Language Processing 28 ( 2020 ) 3029 - 3040 . doi: 10 . 1109/TASLP. 2020 . 3037401 .

[6] G. Erkan , D. R. Radev , LexRank: Graph-Based Lexical Centrality as Salience in Text Summarization , J. Artif . Int. Res . 22 ( 2004 ) 457 - 479 .

[7] R. Mihalcea , P. Tarau, TextRank: Bringing Order into Text, in: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, Barcelona, Spain, 2004 , pp. 404 - 411 . URL: https://www.aclweb.org/anthology/W04-3252.

[8] H. Zheng , M. Lapata , Sentence centrality revisited for unsupervised summarization, in: Proceedings of the 57th Annual Meeting of the Associa- Association for Computational Linguistics (Volume tion for Computational Linguistics, Association for 1: Long Papers), Association for Computational LinComputational Linguistics , Florence, Italy, 2019 , pp. guistics , Vancouver, Canada, 2017 , pp. 1073 - 1083 . 6236 - 6247 . URL: https://aclanthology.org/P19-1628. URL: https://www.aclweb.org/anthology/P17-1099. doi: 10 .18653/v1/ P19 -1628. doi: 10 .18653/v1/ P17 -1099.

[9] A. Nenkova , L. Vanderwende , The impact of fre- [18] M. Lewis , Y. Liu , N. Goyal , M. Ghazvininejad, quency on summarization, Microsoft Research , Red- A. Mohamed , O. Levy , V. Stoyanov , L. Zettlemond, Washington, Tech. Rep. MSR-TR- 2005 101 moyer, BART: Denoising sequence-to-sequence ( 2005 ). pre-training for natural language generation , trans-

[10] J. Steinberger , K. Jezek , et al., Using latent seman- lation, and comprehension, in: Proceedings of the tic analysis in text summarization and summary 58th Annual Meeting of the Association for Comevaluation , Proc. ISIM 4 ( 2004 ) 8 . putational Linguistics, Association for Computa-

[11] Y. Liu , M. Lapata , Text summarization with pre- tional Linguistics , Online, 2020 , pp. 7871 - 7880 . URL: trained encoders, ArXiv abs/ 1908 .08345 ( 2019 ). https://aclanthology.org/ 2020 .acl-main. 703 . doi:10.

[12] M. Zhong , P. Liu, Y. Chen , D. Wang , X. Qiu , 18653 /v1/ 2020 .acl-main.703. X. Huang , Extractive summarization as text [19] J. Zhang , Y. Zhao , M. Saleh , P. Liu, PEGASUS: matching, in: Proceedings of the 58th An- Pre-training with Extracted Gap-sentences for Abnual Meeting of the Association for Computa- stractive Summarization , in: H. D. III , A. Singh tional Linguistics, Association for Computational (Eds.), Proceedings of the 37th International ConferLinguistics, Online , 2020 , pp. 6197 - 6208 . URL: ence on Machine Learning , volume 119 of Proceedhttps://aclanthology.org/ 2020 .acl-main. 552 . doi:10. ings of Machine Learning Research, PMLR, 2020 , pp. 18653 /v1/ 2020 .acl-main. 552 . 11328 - 11339 . URL: http://proceedings.mlr.press/

[13] I. Sutskever , O. Vinyals , Q. V. Le , Sequence to Se- v119/zhang20ae.html. quence Learning with Neural Networks , in: Pro- [20] I. Beltagy , M. E. Peters , A. Cohan , Longformer: The ceedings of the 27th International Conference on long-document transformer , ArXiv abs/2004.05150 Neural Information Processing Systems - Volume ( 2020 ). 2, NIPS'14, MIT Press, Cambridge, MA, USA, 2014 , [21] M. Zaheer , G. Guruganesh, K. A. Dubey , J. Ainslie, p. 3104 - 3112 . C. Alberti, S. Ontanon , P. Pham , A. Ravula , Q. Wang ,

[14] A. M. Rush , S. Chopra , J. Weston , A neural attention L. Yang , A. Ahmed , Big Bird: Transformers for model for abstractive sentence summarization , in: Longer Sequences, in: H. Larochelle , M. Ranzato , Proceedings of the 2015 Conference on Empirical R . Hadsell, M. F. Balcan , H. Lin (Eds.), Advances in Methods in Natural Language Processing, Associa- Neural Information Processing Systems , volume 33 , tion for Computational Linguistics, Lisbon, Portu- Curran Associates, Inc., 2020 , pp. 17283 - 17297 . URL: gal, 2015 , pp. 379 - 389 . URL: https://aclanthology. https://proceedings.neurips.cc/paper/2020/file/ org/D15-1044. doi: 10 .18653/v1/ D15 -1044. c8512d142a2d849725f31a9a7a361ab9 -Paper.pdf.

[15] S. Chopra , M. Auli , A. M. Rush , Abstractive sen- [22] S. Huang , R. Wang , Q. Xie , L. Li , Y. Liu , An tence summarization with attentive recurrent neu- extraction-abstraction hybrid approach for long ral networks , in: Proceedings of the 2016 Con - document summarization, in: 2019 6th Internaference of the North American Chapter of the As- tional Conference on Behavioral, Economic and sociation for Computational Linguistics: Human Socio-Cultural Computing (BESC ), 2019 , pp. 1 - 6 . Language Technologies , Association for Compu- doi:10.1109/BESC48373. 2019 . 8962979 . tational Linguistics, San Diego, California, 2016 , [23] W. S. El-Kassas , C. R. Salama , A. A. Rafea , pp. 93 - 98 . URL: https://aclanthology.org/N16-1012 . H. K. Mohamed , Automatic text summadoi: 10 .18653/v1/ N16 -1012. rization: A comprehensive survey , Expert

[16] R. Nallapati , B. Zhou , C. dos Santos, Ç. Gulçehre, Systems with Applications 165 ( 2021 ) 113679 . B. Xiang , Abstractive text summarization using URL: https://www .sciencedirect.com/science/ sequence-to -sequence RNNs and beyond , in: Pro- article/pii/S0957417420305030. doi:https: ceedings of The 20th SIGNLL Conference on Com- //doi.org/10.1016/j.eswa. 2020 . 113679. putational Natural Language Learning , Association [24] J. Codina-Filbà , N. Bouayad-Agha , A . Burga, for Computational Linguistics, Berlin, Germany, G. Casamayor, S. Mille , A. Müller , H. Sag2016, pp. 280 - 290 . URL: https://aclanthology.org/ gion, L. Wanner, Using genre-specific feaK16-1028 . doi: 10 .18653/v1/ K16 -1028. tures for patent summaries , Information Pro-

[17] A. See , P. J. Liu , C. D. Manning , Get to the point: cessing & Management 53 ( 2017 ) 151 - 174 . Summarization with pointer-generator networks , URL: http://www.sciencedirect.com/science/article/ in : Proceedings of the 55th Annual Meeting of the pii/S0306457316302825 . doi:https://doi.org/ Adaptation of General Natural Language Processing Tools for a Patent Claim Visualization System , in: M. Lupu , E. Kanoulas , F. Loizides (Eds.), Multidisciplinary Information Retrieval , Springer Berlin Heidelberg, Berlin, Heidelberg, 2013 , pp. 70 - 82 .

[46] S. Deerwester , S. T. Dumais , G. W. Furnas , T. K. Landauer , R. A. Harshman , Indexing by latent semantic analysis , Journal of the Association for Information Science and Technology 41 ( 1990 ) 391 - 407 . doi: 10 .1002/(SICI) 1097 - 4571 ( 199009 ) 41: 6 < 391 : :AID-ASI1>3.0 .CO; 2 - 9 .

[47] V. Klema , A. Laub , The singular value decomposition: Its computation and some applications , IEEE Transactions on Automatic Control 25 ( 1980 ) 164 - 176 . doi: 10 .1109/TAC. 1980 . 1102314 .

[48] Y. Gong , X. Liu , Generic text summarization using relevance measure and latent semantic analysis , in: Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , 2001 .

[49] M. Grusky , M. Naaman , Y. Artzi , Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies , in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 ( Long Papers) , Association for Computational Linguistics , New Orleans, Louisiana, 2018 , pp. 708 - 719 . URL: https://aclanthology.org/N18-1065. doi: 10 .18653/ v1/ N18 -1065.

[50] A. Radford , K. Narasimhan , T. Salimans , I. Sutskever , Improving language understanding by generative pre-training , Technical Report , 2018 .

[51] A. Radford , J. Wu , R. Child , D. Luan , D. Amodei , I. Sutskever , Language Models are Unsupervised Multitask Learners , Technical Report , 2019 .

[52] T. B. Brown , B. Mann , N. Ryder , M. Subbiah , J. Kaplan , P. Dhariwal , A. Neelakantan , P. Shyam , G. Sastry , A. Askell , S. Agarwal , A. Herbert-Voss , G. Krueger, T. Henighan , R. Child , A. Ramesh , D. M. Ziegler , J. Wu , C. Winter , C. Hesse , M. Chen , E. Sigler, M. Litwin , S. Gray , B. Chess , J. Clark , C. Berner , S. McCandlish , A. Radford , I. Sutskever , D. Amodei , Language Models are Few-Shot Learners , in: H. Larochelle , M. Ranzato , R. Hadsell , M. Balcan , H. Lin (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 , NeurIPS 2020 , December 6- 12 , 2020 , virtual, 2020 . URL: https://proceedings.neurips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract. html.