This paper explores the concept of dynamically generating definitions using a deep-learning model. We do this by creating a dataset that contains definition entries and contexts associated with each definition. We then fine-tune a GPT-2 based model on the dataset to allow the model to generate contextual definitions. We evaluate our model with human raters by generating definitions using two context types: short-form (the word used in a sentence) and long-form (the word used in a sentence along with the prior and following sentences). Results indicate that the model performed significantly better when generating definitions using short-form contexts. Additionally, we evaluate our model against human-generated definitions. The results show promise for the model, showing that the model was able to match human-level fluency. However, while it was able to reach human-level accuracy in some instances, it failed in others.
Prior studies have suggested that 95% of words in a text must be within a reader's
vocabulary for adequate reading comprehension to occur [10, 15]. This presents a problem
for academic texts, which often use low-frequency words to describe key concepts
related to their field. For readers to comprehend these concepts, they must first acquire
the field-specific vocabulary. One of the classical ways to help with this problem is to
have a glossary near the end of the textbook or a list of key terms at the beginning of
the chapter, allowing the reader to have a relatively easy way to familiarize themselves
with words that they are unfamiliar with and reference these definitions during reading.
This classical approach in paper-based textbooks has a few notable issues: (
Intelligent systems have made the process of finding a definition much easier. Rather
than turning pages or firin1g up a search engine, readers can now click on a piece of
text, and a pop-up will appear displaying the definition for the selected term [5, 7].
These definitions are typically acquired by querying a definition database. While this
is a good solution, it does come with some issues. (
This paper explores the concept of using a deep-learning model to generate dynamic,
contextual definitions by paying attention to the surrounding context of the word. We
do this by creating a new dataset consisting of words, a definition for each word, and a
list of contexts associated with each definition. We then fine-tune a GPT-2 based model
on this dataset, resulting in a model capable of autoregressively generating a definition
for any English word with only the word and a context as inputs. We assess the model
using human evaluation with three research questions in mind: (
We answer these questions by analyzing how human raters rate machine-generated and human-generated definitions for terms from 5 college-level textbooks in terms of accuracy and grammatical fluency.
A key constraint before beginning data collection was to find sources where word entries had both a definition and a context that matched that definition. A matching definition and contextual pair are important since sampling random sentences/paragraphs the term is within and attempting to find the most appropriate definition could lead to errors. We did not impose strict types of contexts. Contexts could be words in example sentences, words in example paragraphs, hypernyms, synonyms, hyponyms, etc. The primary goal was to give the model enough of a contextual clue to internally disambiguate the sense of the word and generate a definition that fit that sense.
With this in mind, we collected training data from the following sources: (
Data from each source was then combined into a JSON file that totaled approximately 300MB when compressed. All entries from each source were kept should they contain enough information about the word, as further discussed in section 2.2. This includes word entries that had the same (or paraphrased slightly) sense definition in multiple sources. No merging was performed to ensure that all possible senses and contexts were represented in the dataset. However, it may also deter the model discussed in section 2.3 from memorizing definitions due to slight definition variations. 2.2
Many definitions within the training dataset did not contain enough information about the word it was describing to be helpful to readers unfamiliar with the word. An example of this can be seen in Figure 1A above. Here the definition of the word countries is "plural of country." While this definition is not wrong, it does not contain enough information for someone who does not have prior knowledge of the root word "country." We attempt to fix this issue by identifying the referenced word and appending the referenced word's definition to the end of the original definition.
The referenced word was found using regular expressions, word frequency, and parts-of-speech tags. Regular expressions were used to find key phrases (e.g., plural of) that regularly pointed to the referenced word (e.g., the original word with grammatical suffixes removed). We additionally found that many definitions contained less than four words and heavily relied upon a key noun/adjective to convey the word's meaning. This was appropriate from our perspective when the referenced word was frequent and wellknown to most English speakers. However, there were also cases where the referenced word, such as displayed in Figure 1B, was infrequent and perhaps not well-known enough to strengthen the reader's mental representation of the original word. In cases such as these, we also attempted to expand the definition to something more useful.
To expand the definitions, we identify the referenced word and search the training dataset for all entries of that reference word. We then use SentenceTransformer (RoBERTa-large variant) to embed each found definition into sentence vectors, along with the base term's contexts [16, 18]. We then compute cosine similarity to compare each definition entry for the referenced word with the base term's contexts. The definition with the highest cosine similarity score was then appended to the end of the definition, as displayed in Figure 1. If the referenced word was not within the dataset, we query Wikipedia using wptools API [24]. If the word is found, we append the first sentence to the end of the definition. If not, the word and definition were removed from the dataset.
After removing terms with low information, the total dataset contained 254k unique terms, 512k definitions, and 2.66M contexts.
Before training, we begin by making the data more machine-readable by placing each word, definition, and context from the dataset into the format shown in Figure 2 for each context entry. This format contains two special tokens: <CONTEXT> and <DEFINITION>. These tokens were used to make it easy for the model to determine where the context ended and where definition generation should begin.
We then trained a fine-tuned version of GPT-2 called WikiMorph on the dataset [9, 22]. WikiMorph uses the large variant of GPT-2 made available by Hugging Face to break down words into morphological compounds and definitions associated with each compound [20]. It was chosen for this work because it contains some ability to generate definitions for words and sub-words. Therefore, likely giving the model a good initialization point. We then fine-tune this model on the dataset for 1 epoch. We stopped at only a single epoch because there were no improvements in validation loss or ROUGE scores on the validation data [17]. 2.4
The model is used by placing the word and a context into the format displayed in Figure 2 and omitting everything following the special <DEFINITION> token. We then feed this text into the model, which uses its tokenizer to encode the text into tensors. The model then begins to autoregressively generate a definition by referencing the given word, the given context, and all prior output tokens until it reaches the end token designated as "<|endoftext|>". We then use regular expressions to find all text between the <DEFINITION> and <|endoftext|> token to display to the user.
5
To evaluate the model, we began by collecting sample texts and definitions from 5
different university-level textbooks. These were from the following subjects: Anatomy
and Physiology [6], American Government [1], Astronomy [2], Chemistry [4], and
Psychology [14]. We then searched these textbooks for paragraphs containing each of the
key terms within its glossary. These paragraphs were then divided into sentences using
Spacy's sentence segmentation model [19]. We then randomly chose 50 terms from
each textbook and generated definitions using two different context types. (
Each survey began by displaying a multiple-choice question asking participants to
choose which of the above subjects they felt most knowledgeable in. Their selection
would determine what set of generated definitions they would assess throughout the
rest of the survey. (e.g., if they chose psychology, they would evaluate the 50 generated
definitions from the psychology textbook). For each generated definition, we asked two
questions: (
We additionally created a third survey containing human-generated definitions to compare our model against. Human-generated definitions were collected from the training dataset described in Section 2. We searched the training dataset for all definition entries for each term used in the prior two surveys. We then selected the most appropriate definition using SBERT sentence vectors. Each definition for the term was embedded into a sentence vector. We then computed cosine similarity between each of ject. these definitions from the training dataset with the definition from the textbook. The definition from the training dataset with the highest similarity value was displayed to the user in the same format shown in Figure 4. If a term happened to not be within the training dataset, it was dropped from the survey entirely. This lowered the number of questions for the human-generated definitions from 50 to 40-45, depending on the sub
For each survey, we implemented 3-4 control questions. These control questions
replaced the generated definition displayed in Figure 4 with a random definition for a
different term from the subject's textbook. Participants were required to assign an
accuracy value below 50 on the slider question for over half of the control questions to be
included in the results. (i.e., to be considered reliable raters). Below 50 was the
threshold selected due to 50 being the starting position for the slider. (i.e., the participants had
to actively move the slider towards "no" and indicate inaccuracy).
The only imposed requirement was that participants were required to be from
Englishspeaking countries. This led to a total of 194 participants. Of which, 53 were excluded
from the results for two reasons: (
To answer our research questions, we ran mixed-effects beta regressions for accuracy
and fluency using all three survey conditions and random intercepts for definitions and
participants. Each model was initially fit with the interaction between context and topic,
but if the interaction was not significant, it was dropped, and the model refit. For
accuracy, we ran an ANOVA and found a significant main effect of context, χ2(
To answer our second research question to see how the model performs relative to
human-generated definitions, we conducted an additional post hoc comparison using
Tukey’s HSD, which revealed that human-generated definitions (M = 78.57, SE = 0.70)
performed significantly better than definitions generated from shorter contexts (M =
62.60, SE = 0.875), t(5316) = 4.11, p < .001, as well as those generated from longer
contexts ( = 53.70, = 0.886), t(5316)=2.38, p < .001. Further examination of
the accuracy density plot seen in Figure 5 sheds some light on this result. It shows that
the model was able to perform admirably in many situations, with a median rating equal
to 75. However, it also had an abundance of error cases where ratings were below 25.
The exact reasons for these error cases require further investigation. Two possible
culprits include: (
To answer our third research question to see if the model performed better on some topics relative to others, we did not find significant differences between textbook subjects. Though, a non-significant trend suggests that, while human-generated definitions did equally well across topics, the model might perform better on some topics relative to others. In particular, the average meaning ratings for definitions generated from shorter contexts were approximately equal to those for human-generated definitions on the topic of Anatomy and Physiology. This trend is potentially worthy of further analysis in subsequent studies examining a wider variety of textbook sources and topics.
We performed an identical ANOVA analysis for fluency and found no significant main effects of context, topic, or interaction. As shown in the fluency density plot in Figure 5, the model's performance was excellent for fluency relative to human-generated definitions. Some of this could be due to the data found within our dataset being slightly less fluent than a typical human definition due to some entries coming from dirty sources or the definition expansion method discussed in Section 2.2. However, even with this considered, the model appeared to perform exceptionally well with mean values effectively greater than or equal to 80 for both short and long-form context surveys. 5
This work presents a deep-learning model capable of dynamically generating definitions based solely on the surrounding context. We examined the model's ability to generate definitions using two context types: short and long-form. Short-form contexts significantly outperformed long-form contexts in human-rated accuracy but fell short of human-generated definitions on this metric. In contrast, short-form and long-form conditions were indistinguishable from human-generated definitions in terms of fluency, displaying some promise for the model.
This material is based upon work supported by the National Science Foundation (1918751, 1934745) and the Institute of Education Sciences (R305A190448).