In this paper we describe our early work with determining domainspeci c di erences of polysemous words. The goal is to determine whether a word has di erent meanings in two di erent text corpora. Previous work tried to identify such words by training word embeddings speci c for those domains and then comparing the resulting vectors. Our use case, however, does not contain enough text to train word embeddings. Thus, we present a novel method that is able to make use of context information and determine domain-speci c di erences of words without generating word embeddings.
Copyright c 2019 by the paper's authors. Copying permitted for private and academic purposes.
In other cases it could lead to miscommunication during requirement elicitation [FDG17], or low accuracy for requirement tracing [WNLN18].
This problem can be avoided when using domain-speci c texts for training, which match the domain of the requirements, as that would generate a more speci c representation of problematic words. Yet, in certain cases it is unrealistic to train word embeddings for niche (sub)domains (in which requirements are often described, as they are highly speci c), since collecting enough suitable text material is often infeasible. For example, the word server, which would be part of the computer science domain, can even have two meanings in this domain. First, the server can be a service on the internet that provides a response to user requests. Second, the server can be a physical device composed of processor units and other hardware. And even though, both instances of the word can refer to the same object, both aspects will have di erent types of requirements that have to be solved by di erent people. Segregating those two meanings with speci c word embeddings means collecting and curating enough texts containing di erent interpretations of this word. We can assume that the related projects mentioned above are all part of the technical or computer science domain. However, the computer science domain is too broad to allow for a precise interpretation of requirements. In our case, one requirement set describes infrastructure to distribute information in areas without internet connection, while the second set describes applications that are built upon this infrastructure. Curating a text corpus to train representative word embeddings for those two cases would require laborious work from experts.
We propose a new method, which is able to calculate the domain- and subdomain-speci c di erences of words without using di erent word embeddings. This provides a more exible and application-friendly solution. The remainder of the paper is structured as follows. Section 2 provides a brief overview of the related work, as well as an explanation of the previous research that is used for our method. In section 3, we present our proposed algorithm. The algorithm is veri ed in section 4. In section 5, we describe the advantages and shortcomings of our method and how it ts into our application. Section 6 concludes the paper with a summary and a short outlook into future work. 2
Today, sophisticated word embedding algorithms, like word2vec [MCCD13], GloVe [PSM14] and fastText [BGJM16], are able to place words in a vector space that contains semantic information about the words. Similar words will be placed close to each other and form groups of words. Therefore, the distance between two words in this space can describe how semantically similar they are. For example, synonyms of a certain word will be placed close to each other, while non-related words will have a higher distance. Additionally, the words have a xed dimensionality (generally orders of magnitude lower than the number of di erent words in the corpus). Previous work in requirement analysis discusses the possibility of integrating Word Embeddings [FDE+17] or integrates them in classi cation approaches [WV16]. Additionally, Lucassen et al. also use the approach for clustering requirements [LDvdWB16].
Looking closer at word2vec, the models are generated by reading large sources of text, such as Wikipedia, and trying to determine the word positions of word x by looking at words used around x. In short, the model is trained by looking at n words before and after x to estimate x (other methods like the Skip-Gram Model [MSC+13], GloVe [PSM14] or fastText [BGJM16] prove to be more precise, but would be too extensive in this report). Thus the position of x in the vector space is determined by the words often used with x. The assumption is, that similar words are used with the same surrounding words. Because of this training, we assume that the model trained with an appropriate data set contains ner nuances of words like server, as the surrounding words will catch the di erent meanings and contexts.
When trying to determine the polysemy of words in requirement sets, di erent methods were proposed [FDG17], [WNLN18], [FEG18]. In the rst method, domain-speci c word embeddings of two di erent domains are merged into one corpus. Certain hand chosen words are marked with a su x to distinguish the domains. All instances of chosen word a will be replaced with adomain1 in the texts from the rst domain and with adomain2 in the second domain. The word a will receive two di erent vector representations, one for each domain, which can be compared. The higher the resulting distance, the more di erent are the interpretations of the word in the domains. However, this means that controversial words need to be known beforehand, as they have to be marked manually. Another method [FEG18] tries to identify polysemous words by looking at a words and its neighbors in a certain domain and then comparing it to the neighbors of the same word in another domain. This means that no preprocessing of words is needed, but it requires one completely trained word embedding for each domain that is involved.
In [DvdSL18], Dalpiaz et al. describe a method to determine how similar words are used in a set of user stories. The goal is to improve the quality of the data set by ensuring only clearly de ned terms are used and they are independent from each other. For example, the words broadcast and transmit can both be used similarly in the computer science domain. User stories for one project, generated from di erent sources, can contain both words with identical meanings. Using both words however, may cause problems, as the precise nature of user stories, as well as requirements, would suggest that when both words are used, they mean di erent things. Otherwise only one of those words would be needed. To determine whether words a and b might potentially be interchangeable, the word embedding distance of both words is considered, as well as the contextual distance. For this, the usage of words a and b in the data set are determined by looking at the other words in the same sentence. The more the words in the sentences di er, the higher the resulting contextual di erence is. Following formula was used: Given two terms t1 and t2: ambigt1;t2 = 2 simt1;t2 + simct1;t2 ; 3 with simt1;t2 being the similarity between t1 and t2 and simct1;t2 being the context similarity. For the context similarity, all sentences that contain t1 are collected and compared against the sentences that contain t2. If those collections use similar words, the simc of t1 and t2 is high. 3
As described above, word embeddings, as well as simc as presented by Dalpiaz et al. [DvdSL18], provide a higher distance between words that have di ering contexts. In word embeddings this happens during training, which results in vector representations that are further apart. This means, using simc, in comparison to calculating the vector distance on the same data set as the word embedding was trained upon, would give di erent values, but the ordering and relative values will strongly correlate. However, if simc is used on texts that di er from the trained word embedding, it indicates whether the word was used di erently in this speci c text. Furthermore, it is also possible to use simc on one word across di erent texts. For example, using simc, two texts from di erent domains will result in a lower similarity if the word is used di erently in those domains.
The semantic folding theory [DSW15] is another method to place words in a semantic vector space [DvdSL18]. In our approach we use word embeddings, but the theoretical idea behind it and the work of Dalpiaz et al. applies to both methods.
In [DvdSL18], no exact method is given on how to calculate simc (a method provided by cortical.io1 is used in the paper), so we devised our own formulation for word embeddings. Let D1 and D2 be text corpora from two di erent domains. Given a term t 2 D1 \ D2, its contexts c1 D1 and c2 D2 consist of all words from sentences that contain t. Furthermore, let vw be the vector representation of a term w in a given embedding. We calculate the context similarity of t as simct =
center(c1) center(c2) kcenter(c1)k kcenter(c2)k ; where center(c) = 1
X IDFD(w) vw: jcj w2c In other words, we calculate the cosine similarity of the mean vectors of all words from c1 and c2. This method of determining a sentence representation was previously shown to be e ective for capturing and distinguishing semantic meaning [BCM+14]. In addition to this, each word w is weighted by its inverse document frequency IDFD(w) from the corpus D the context was taken from.
Even though this method requires some word embedding to be trained, it is not necessary to use a word embedding trained on either D1 or D2, as the speci c nuances of the word in those domains are represented by the formula. 4
To evaluate the described approach, we tested it on corpora that were retrieved analogously to the work of Ferrari et al. [FDG17]. We crawled topic pages of the English Wikipedia to retrieve pages from six categories: Computer Science (CS), Electronic Engineering (EEN), Mechanical Engineering (MEN), Literature (LIT), Medicine (MED) and Sports (SP). We also trained a word2vec embedding on a dump of the English Wikipedia and the created Computer Science corpus using the implementation from Gensim [RS10]. Then we calculated the similarity 1https://www.cortical.io/compare-text.html scores of common nouns that are mentioned in [FDG17] using our method. More speci cally, we used the CS corpus as D1 and each one the other fetched corpora as D2. Additionally, we reimplemented the approach proposed in [FDG17] to be more exible in comparing our results with those reported by Ferrari et al. The goal in this chapter is to verify if, using the same conditions as in the work provided by Ferrari et al., our method provides a comparable ordering of terms with respect to their similarity value. This would suggest that the two approaches provide the same results. 4.1
The retrieval of the six corpora was done using a Wikipedia API for Python2 by fetching pages of a category and subsequently the subcategories, until a total of 10,000 pages was reached or no all pages were found. The raw texts of every page were concatenated to create a corpus for the given category. Apart from lowercasing the text and removing all stopwords and punctuation (except for sentence boundaries), no further preprocessing was applied to the corpora. The overlap between the corpora is on average lower than 0.1% and thus negligible. 4.2
word2vec training Using a publicly available dump of the English Wikipedia, we trained a word2vec model on the raw text that was preprocessed like the corpora described above. This resulted in a total of 74,526 unique terms. We chose a dimension of 200 and 15 epochs to train the model. Other parameters were the default values used in the implementation of Gensim [RS10]. We also trained a similar word embedding on the 10,000-page corpus for the topic "Computer Science". 4.3
We reimplemented the approach proposed in [FDG17] with two important di erences. First, we used the corpora described above, which means no lemmatization was used. On top of that, we used the 100 most common nouns for which Ferrari et al. reported similarity values, instead of determining the most frequent nouns ourselves. 4.4
To calculate the scores of the 100 nouns, we used the formula described in chapter 3. The scores represent the calculated similarity score between the CS corpus and each of the other corpora. Table 1 shows the ten terms with highest and lowest similarity scores for each of the ve corpora EEN, LIT, MEN, MED and SP, compared to CS. It is important to note, that there is no clear threshold when words are used ambiguously or not. For our examples in chapter 5 we used relative values and the rankings of the words as suggestions which words seem to be used polysemous and thus are problematic.
For example the results show that the term le is used in similar contexts in the CS and the EEN corpus, whereas the similarity value is much lower for non-technical categories (LIT, MED, SP). This may point to the fact that EEN and MEN refer to les of a le system in a computer, contrary to physical documents. Likewise, the term web has higher similarity values in LIT, MED, SP than in the more technical categories (EEN, MEN), which could be interpreted as web being a more nuanced term in technical areas compared to non-technical descriptions. Furthermore, the word software is in the ten most similar words across all corpora. 4.5
In a second step, we compared the values yielded by our reimplementation of the method proposed by Ferrari et al. with the values they reported in [FDG17] and our results using di erent embeddings. More precisely, we calculated the Spearman rank correlation comparing the reimplementation with the results found in [FDG17] (Table 2a) and our results on the CS-embedding (Table 2b), the Wikipedia embedding without using IDFweighting (Table 2c) and with the weighting enabled (Table 2d). The rank correlation coe cient denotes the strength and direction of the monotonic relationship, values > 0 representing a monotonous increasing and values < 0 a monotonous decreasing relationship. The p-value represents the probability of the correlation being non-existent ( = 0), therefore p < 0:05 is commonly considered a desired result.
Values in Table 2a show a moderate but signi cant correlation between the original values and our reimplementation. Di erences in the calculated values are presumably caused by not applying lemmatization while 2https://pypi.org/project/Wikipedia-API/ Electr. Engineering science 0.9954
code 0.9899 security 0.9898 memory 0.9874
le 0.9873 language 0.9870 algorithm 0.9867 database* 0.9864 software 0.9859 user 0.9858 . .
.
see 0.9619 structure 0.9617 type 0.9605 input 0.9581 reference 0.9559
source 0.9529 technology 0.9465 game 0.9458
eld 0.9447 non 0.9238 generating our corpora as well as by fetching pages from Wikipedia in a di erent order, so that our corpora might considerably di er from the ones created by another fetching method. Also, as seen in [CDLRL18], our choice of word2vec hyperparameters may have in uenced the training signi cantly, resulting in dissimilar inter-corpora embeddings.
Column b shows that using an embedding only trained on the CS corpus leads to weak and non-signi cant correlations compared to the results of Ferrari et al., so the quality of the results might be sub-par, seemingly because the embedding is unable to map semantic di erences su ciently. In contrast, correlations shown in column c and d are signi cant, which underlines the importance of using a general-purpose word embedding. Furthermore, the IDF-weighting improved the correlation across all corpora, as can be seen in column d. 4.6
To examine the stability of our solution regarding the size of the corpora, we repeated the experiment using a limit of 100,000 pages (and a maximum subcategory crawling depth of 5) and compared the results with our ndings above. Table 3 show the resulting values. We also determined the correlation between this data and the results from Table 1 (Table 4a), the results of our reimplementation on the smaller corpora (Table 4b) and the bigger corpora (Table 4c).
While Table 4 shows that the overall trend of the results is similar to that of Table 1, the correlation is much weaker for corpora SP and MEN than for MED. For example the term text was ranked as one of the most similar words in MEN for the smaller corpus size, while it is one of the least similar for the 100,000 page corpus. This could be caused by the smaller corpora representing the categories only partially and using more Wikipedia pages led to a more general representation of the individual topics. Interestingly, our results on the EEN corpus are notably di erent from the ones our reimplementation produced (Table 4b-c). For EEN, a weaker correlation was determined, whereas a strong monotonic relationship was found for the other corpora, especially with the results of our reimplementation on the bigger corpora. These results can suggest the instability of word embeddings in general or word embeddings with smaller corpus size. In conclusion, this creates the need for a more thorough investigation in the future. 5
Comparing the method from [FDG17] to the method presented in this work, the new method proves to be more exible and needs less initial e ort. This stems mainly from the fact that no separate word embeddings need to be trained for speci c domains. The results in chapter 4 suggest that the best results can be achieved with a general purpose word embedding as a basis. Thus, in an application it would be su cient to provide the two texts from di erent domains or sources and the domain-speci c di erences can be calculated. Notable are the following advantages: 1. Since there is no need for word embeddings, the di erences can be calculated even on small data sets that would not provide enough data for own word embeddings. This means di erences in niche domains,
subdomains as well as in di erent requirements sets can be compared to each other. 2. The word di erences can be calculated for every word. In [FDG17] the words have to be marked before the word embeddings are calculated. This means that either the most frequent words are taken (as is done in the paper), or they have to be marked manually. However, this would require previous knowledge about which words may have di erent interpretations in those domains. Additionally, the more words are marked, the more tokens will be di erent between the domains. Thus, less words will be common during training and the domains will completely separate from each other, resulting in less expressive distances. This is not desirable.
One disadvantage we found while comparing the two methods is the calculation time. Determining the distance of a word in two domains in [FDG17] requires two look-ups in the word embedding model and a simple distance calculation between two vectors. The presented method, however, needs to determine the whole context of the word in both domains, meaning all instances of the word have to be found and the context words in the sentence have to be collected. This then requires multiple look-ups in the model, the calculation of the average vector and a distance calculation. This results in approximately 120 ms per call when comparing two 10,000-page corpora from above, and approximately 1800 ms for the bigger 100,000-page corpora, but only about 4 ms in requirement texts with around 800 requirements. In comparison, the method devised in [FDG17] is able to calculate the distance in less than a millisecond.
Turning to our previously described example, we compared the distances of words in four di erent requirement sets. As all those requirement sets were related to each other, but discussed di erent parts of connected projects, there should be multiple words that are shared between those sets, while also containing some di erences in their usage. The notation in Table 5 shows the results between those four data sets. P1 describes the overarching military project that will connect multiple sub projects, like P2 and P3. The requirements in P2 describe the infrastructure for a mobile network that has to be able to connect multiple devices in areas without internet connection. P3 describes devices that will work in the infrastructure provided by P2. Additionally, P30 is a small part in P3 that was focused separately from the rest of P3.
The results show that in most comparisons the word service has one of the lowest similarity scores. Upon further investigation of the requirement sets, some di erences can be seen. In P1 the word is mainly used on which services should actually be used and supported for the project, while P2 describes which features the services shall integrate. The requirements for P3, however, specify which features for the user shall be provided and how they should be able to use them. This indicates that P1 and P2 are emphasizing the technical aspects and details of the services more, while in P3 the user interaction aspect of services are emphasized. In P2, the system uses the services, while in P3 humans use the services. The word clients (which in this context was used as a term for a computer device) had a high di erence between P3 and P 0, as one part was focused on what 3 features shall be provided by the client, while the other part speci ed details as the maximal weight or battery runtime. In one case the client was used as a pure hardware product, while the other part speci es the software part of the client. The same semantic di erence was found with the word plattform which had a low similarity between P2 and P3.
However, the results provided in this experiment may have some shortcomings. In section 4 we were able to show that our method provides the same results as a previously established method on text corpora that are big in comparison to this experiment. There was no qualitative study on how good the results are on smaller texts, or whether smaller texts even contain enough information to show discrepancies in word usage.
In this paper we discussed the potential problems that may arise from polysemous words in requirements. Those di erent interpretations of a word can lead to confusion when working with requirement sets. Di erent methods were suggested that solve this problem by comparing domain speci c word embeddings. However, it is not always possible to train word embeddings due to text limitations. The method proposed in this work is able to calculate domain speci c di erences by comparing the two di erent text corpora without training word embeddings. This results in a more exible method that needs less initial work. We verify our work by comparing it to a previously established method in this eld and prove that our method, albeit a bit slower in calculation, provides similar results. In our experiment we show the usefulness of our method, as it is able to calculate word di erences even on requirement sets that contain less than 1000 sentences. In the future we would like to conduct a qualitative analysis of the usefulness of our results. This means the method needs to be integrated into a tool for users to interact with. Additionally we want to see how users can implement this information to nd di erences in the interpretation of polysemous words. [BCM+14]
Carmen Banea, Di Chen, Rada Mihalcea, Claire Cardie, and Janyce Wiebe. Simcompass: Using deep learning word embeddings to assess cross-level similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 560{565. Association for Computational Linguistics, 2014.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606, 2016.
Hugo Caselles-Dupre, Florian Lesaint, and Jimena Royo-Letelier. Word2vec applied to recommendation: Hyperparameters matter. In Proceedings of the 12th ACM Conference on Recommender Systems, RecSys '18, pages 352{356, New York, NY, USA, 2018. ACM.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391{407, 1990.
Francisco De Sousa Webber. Semantic folding theory and its application in semantic ngerprinting. arXiv preprint arXiv:1511.08855, 2015.
Fabiano Dalpiaz, Ivor van der Schalk, and Garm Lucassen. Pinpointing ambiguity and incompleteness in requirements engineering via information visualization and nlp. In International Working Conference on Requirements Engineering: Foundation for Software Quality, pages 119{135. Springer, 2018.
Alessio Ferrari, Felice Dell'Orletta, Andrea Esuli, Vincenzo Gervasi, and Stefania Gnesi. Natural language requirements processing: A 4d vision. IEEE Software, 34(6):28{35, 2017.
A. Ferrari, B. Donati, and S. Gnesi. Detecting domain-speci c ambiguities: An nlp approach based on wikipedia crawling and word embeddings. In 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW), pages 393{399, Sep. 2017.
Alessio Ferrari, Andrea Esuli, and Stefania Gnesi. Identi cation of cross-domain ambiguity with language models. In 2018 5th International Workshop on Arti cial Intelligence for Requirements Engineering (AIRE), pages 31{38. IEEE, 2018. [LDvdWB16] Garm Lucassen, Fabiano Dalpiaz, Jan Martijn EM van der Werf, and Sjaak Brinkkemper. Visualizing user story requirements at multiple granularity levels via semantic relatedness. In International Conference on Conceptual Modeling, pages 463{478. Springer, 2016.
Tomas Mikolov, Kai Chen, Greg Corrado, and Je rey Dean. E cient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. [PSM14] [WNLN18]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Je Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111{3119, 2013.