Researchers in areas such as translation and summarization need to compare their results to a wide range of published baselines that commonly use diferent evaluation methods. We aim to enable an easy comparison by presenting TextGen-Benchmarch, an open-sourced tool1 for streamlining the generation and evaluation of text. Text generation methods and evaluation metrics can easily be added to TextGen-Benchmarch, and its pipeline results in a more eficient comparison between methods as users can supply corpora, systems, and evaluation techniques and receive comparison reports in easy to analyze tabular and graphic formats.
An in-depth evaluation and a fair comparison to the
current literature are crucial parts in the development of
Machine Learning (ML) systems. In addition to
modelspecific investigations, this evaluation process typically
includes automated metrics that allow predictions to be
compared to those of other approaches. However, subtle
diferences in output formatting or evaluation metrics
can lead to drastically diferent reported results [
In the case of (conditional) text generation problems,
put and subject to constraints defined by the task, for
example, the length. Depending on the task, there are
various metrics that can be applied for the evaluation,
such as ROUGE [
Some tools encapsulate diferent metrics into a single library so that users can evaluate their hypotheses against references using a shared interface.While these tools suc
gold folder contains files with line separated references. 2 Samples are read in using Python’s file-stream which ensures minimal memory usage. The references can be stored as either plain-text or as a JSON file to enable multiple references. Here, each line should be formatted as follows: cessfully enable the evaluation of a specific system, they are limited to a single system at a time. Therefore, each user is required to develop their own comparison.
Some libraries are also restricted in the compatible
input formats. For example, the COCO (Common Objects
in Context) Caption evaluation library [
It has support for BLEU, METEOR, ROUGE-L, CIDEr, and { ” r e f e r e n c e s ” :
SPICE. The evaluation library enables users of COCO [ ” r e f 1 ” , ” r e f 2 ” , ” r e f 3 ” ]
caption to streamline the evaluation of their results but is }
limited to COCO-compatible input objects as the library TextGen-Benchmarch loads samples from the datasets
was intended to be used in the context of the MS-COCO specified in the configuration file. It parses the files and
Evaluation Server [
Other libraries can compare models with diferent in- The Text-Generator returns model-generated text, which put formats, but only for limited tasks. For example, is then stored in the file system to be used during evalSpark provides ML Pipelines 1, a high-level API for their uation. Users may provide their own generated text in data handlers. At the end of a pipeline, users may pass conjunction with model-generated texts or skip text gentheir results to evaluators which are designed for clas- eration entirely by turning of the generation in consification and regression models, and do not serve text ifguration. Text generation is also skipped if TextGengeneration models. Benchmarch infers that a given dataset has already been processed with the model and is cached on the file system. 3. System Overview If the evaluation is enabled, the user and model-generated text are evaluated against the reference texts. TextGenBenchmarch currently supports ROUGE, METEOR, NIST, and BLEU scores. We provide additional details on how the library interfaces with data in Section 4.
The TextGen-Benchmarch framework is built in Python and provides a pipeline as illustrated in Figure 1. Before starting, TextGen-Benchmarch parses a configuration file that contains (1) the paths to datasets, (2) the systems, and (3) the metrics to be used. It additionally allows for descriptors for the text format. For example, if sentences are surrounded by tags that should be ignored during evaluation, it can be specified here. Any specified dataset must contain two sub-folders samples and gold. The
TextGen-Benchmarch is designed to make it as easy as possible for users to add and remove text generators and metrics. TextGen-Benchmarch interfaces with two 1https://spark.apache.org/docs/latest/mllib-evaluationmetrics.html 2Python natively supports file-stream with line separated files which is why it is a formatting requirement.
CNN-DM DUC-2004
Arxiv
METEOR Metric Score for each System Sorted by Corpus smmrRE sumyEdmundson sumyEdmundsonLocation sumyEdmundsonCue sumyEdmundsonKey sumyEdmundsonTitle sumyTextRank sumyLuhn sumyLSA sumyLexRank sumySumBasic sumyRandom 0.0 2.5 5.0 7.5 15.0 17.5 20.0 10.0 12.5 METEOR Score library files – one for metrics and one for text generators. Additions can be added to these two libraries.
The text generator library provides a single public method with two inputs: a targeted text generator and text. The targeted text-generator is called and it returns the resulting text. A user can add additional models by adding a method for their model that takes in text as input and returns a generated text as output.
On load of the library class, information related to the format of samples is saved. This information includes separators for tokenized sentences and a Boolean that indicates whether the text is tokenized. Custom methods must use this information to decide how to preprocess the input text before passing it into the flow of their added model. Some text generators require sentences to be pre-tokenized whereas other text generators have custom tokenizers and expect raw text. For additional convenience, we provide an interface for a tokenizer and a detokenizer with the library.
Adding metrics follows a similar process to to one outlined for text generators. The metric library provides a method with a single input: a custom Summary Reader Object (SRO). The SRO has two public methods: r e a d O n e and r e a d A l l . When r e a d O n e is called a tuple of the form ( p r e d i c t i o n , r e f e r e n c e s ) . When r e a d A l l is called, a list of all tuples is returned, where corresponds to the number of generated texts.
The r e a d O n e and r e a d A l l methods are abstractions for Python’s file-stream reader.
TextGen-Benchmarch provides the following report types.
• CSV: Fixed Metric generates a separate CSV for each metric. Each row is a diferent model. Each column represents a corpus. To assist with comparisons of the same evaluation metric and set of summarizers but against diferent corpora. • CSV: Fixed corpus generates a single report for one corpus. Each row represents a model and each column a metric. This assists with comparisons on the same corpus with a single set of models but against diferent metrics. • Horizontal Barchart: Fixed Metric. Grouped by the corpus, this shows scores on the X-axis, sorted by average metric score across corpora. This visualization helps draw comparisons between models across diferent corpora.
We demonstrate the usage of TextGen-Benchmarch
using the extractive summarization problem. An extractive
summary is defined as a subset of sentences from a
number of documents (either one or many) that efectively
summarizes the message of the input. Typical metrics
for this task include ROUGE and METEOR. We present a
comparison of popular non-parametric extractive
summarizers on the DUC 2004 [
For ArXiv and CNN-DM, we used 1,000 samples of the test-set for demonstration purposes. Thus, the results should not be interpreted as oficial scores. They do, however, highlight some interesting variation between the performance of the summarizers in the diferent metrics.
Figure 2 shows the METEOR scores. The order corresponds, from top to bottom, to a summarizer’s rank when comparing the average score across all corpora. Here, smmrRE ranks first and sumyRandom comes in last.