This paper is focused on semantic relations between Czech words. Knowledge of these relations is crucial in many research fields such as information retrieval, machine translation or document clustering. We obtained these relations from newspaper articles. With the help of LSA1, HAL2 and COALS3 algorithms, many semantic spaces were generated. Experiments were conducted on various settings of parameters and on different ways of corpus preprocessing. The preprocessing included lemmatization and an attempt to use only ”open class” words. The computed relations between words were evaluated using the Czech equivalent of the Rubenstein-Goodenough test. The results of our experiments can serve as the clue whether the algorithms (LSA, HAL and COALS) originally developed for English can be also used for Czech texts.
great motivation for us was also the S-Space package [
The rest of the paper is organized as follows. The following section deals with related works. The next section describes the way we created semantic spaces for Cˇ TK4 corpora. Our experiments and evaluations using the RG benchmark are presented in section 4. In the last section we summarize our experiments and present our future work. 2
The principles of LSA can be found in [
We also come from a paper written by Paliwoda [
Different methods which judge how some words are related exploit lexical
databases such as WordNet [
The final form of semantic space is firstly defined by the quality of the corpus
used [
Corpus and corpus preprocessing
The Cˇ TK 1999 corpus, which consists of newspaper articles, was used for our
experiments. The Cˇ TK corpus is one of the largest Czech corpuses we work with
in our department. For lemmatization, Hajic’s tagger for the Czech language
was used [
There was no further preprocessing of input texts performed. Finally, 4 different input files7 for the S-Space package were used. The first input file contained plain texts of the Cˇ TK corpora. The second one contained plain text without stopwords. Pronouns, prepositions, conjunctions, particles, interjections and punctuation8 were considered as stopwords in our experiments. That means that removing stopwords from the text in our paper is the same as keeping only open class words in the text. The third file contained lemmatized texts of the Cˇ TK corpora. And the last file contained lemmatized Cˇ TK corpora without stopwords. Statistics on the texts of the corpus are depicted in Table 1, statistics on texts without stopwords are depicted in Table 2 respectively. The LSA principle differs essentially from HAL and COALS. While HAL and COALS are window-based, LSA deals with passages of texts. The passage of text is presented by a whole text of any article of the Cˇ TK corpus in our case.
distinct document.
for the LSA algorithm.
Both LSA and COALS exploit untrivial mathematical operation SVD9 while
HAL does not. COALS simply combines some HAL and LSA principles [
The S-Space package provides default settings for its algorithms. The settings are based on previous research. The default settings of parameters are depicted in Table 3. We tried to change some values of parameters because of the Czech language of our texts. The Czech language differs from English especially in the number of forms for one word and in word order, which is as not strictly fixed as in English. Therefore, there are more different terms10 for Czech language texts. Since the algorithms are sensitive to the term occurrence, this is one of the reasons11 we tried to remove low occurring words.
Another parameter we observed is HAL’s window size. It was expected that the more terms for the Czech language meant the smaller window size would be more appropriate.
The last parameters we changed from defaults were HAL and COALS
retained columns’ counts. We reduced the dimensionality of spaces in this way by
setting the reduction property to the values adopted from [
The following chapter describes the origination of the Czech equivalent of the RG test. The next chapter comprises our results on this test for many generated semantic spaces. 4.1
Rubenstein-Goodenough test The RG test comprises pairs of nouns with corresponding values from 0 to 4 indicating how much words in pairs are related. The powers of relations were judged by 51 humans in 1965. There were 65 word pairs in the original English RG test.
The translation of the original English RG test into Czech was performed by
a Czech native speaker. The article by Pilot [
After our translation of the RG test into Czech, 62 pairs are left. We had to remove the ”midday-noon”, ”cock-rooster” and ”grin-smile” pairs because we couldn’t find any appropriate and different translations for both words of these pairs in Czech. Our Czech RG test12 was evaluated by 24 Czech native speakers with differing education, age and sex. Pearson’s correlation between Czech and English evaluators is 0.94.
A particular word we removed from our test before comparing it with semantic spaces is ”crane”. The Czech translation of this word has 3 different meanings. Furthermore, only one of these meanings was commonly known by the people who participated in our test. Therefore, another 3 pairs disappeared: ”bird-crane”, ”crane-implement” and ”crane-rooster”. A similar ambiguate word is the Czech translation of ”mound”, which was also used in a different meaning in the corpus. We removed it with these 4 pairs: ”hill-mound”, ”cemeterymound”, ”mound-shore”, ”mound-stove”. In the end, 55 word pairs were left in our test.
Another issue we had to face was the low occurrence of the RG test’s words in our corpus. Therefore, we tried to remove the least frequent words of the RG test in sequence and the pairs which they appear in as a consequence. In the end it turned out that especially this step showed us that the relations obtained from SSpace algorithms correlate with human judgments quite well. To evaluate which of the semantic spaces best fits with human judgments the standard Pearson’s correlation coefficient was used. 4.2
Experiments and results We created many semantic spaces with the LSA, HAL and COALS algorithms. Cosine similarity was used to evaluate whether two words are related in semantic spaces. Other similarity metrics did not work well.
The obtained results for the different semantic spaces are depicted in Table 4 for plain texts of the Cˇ TK corpora and in Table 5 for lemmatized texts. The 12 Available at http://home.zcu.cz/~lkrcmar/RG/RG-ENxCZ.pdf best 2 scores in Table 4 and the best 3 scores in Table 5 are highlighted for each tested set of pairs in our RG test. LSA m2 0,26 0,25 0,27 0,33 0,35 0,36 0,24 N LSA 0,28 0,28 0,29 0,33 0,33 0,33 0,16 N LSA m2 0,27 0,26 0,29 0,33 0,30 0,32 0,11 HAL m2 0,20 0,19 0,24 0,28 0,25 0,24 0,14 HAL m2 s1 0,12 0,11 0,18 0,19 0,14 0,06 0,04 HAL m2 s2 0,17 0,18 0,25 0,30 0,25 0,18 0,15 N HAL m2 0,36 0,38 0,39 0,43 0,43 0,44 0,44 N HAL m2 s1 0,39 0,41 0,43 0,47 0,46 0,48 0,53 N HAL m2 s2 0,40 0,42 0,44 0,48 0,48 0,49 0,53 COALS m2 0,43 0,45 0,48 0,52 0,54 0,57 0,62 COALS m2 d2 0,28 0,30 0,30 0,35 0,38 0,39 0,42 COALS m2 d4 0,17 0,18 0,18 0,19 0,21 0,27 0,32 N COALS m2 0,42 0,43 0,46 0,50 0,53 0,54 0,59 N COALS m2 d2 0,31 0,27 0,25 0,35 0,31 0,23 0,34
N COALS m2 d4 0,43 0,44 0,45 0,50 0,51 0,51 0,57
It turned out we do not have to take into account words which occur only once in our corpora. It saves computing time without a negative impact on the results. This is the reason most of our semantic spaces are computed omitting words which occur only once.
The effect of omitting stopwords is very small for the LSA and the COALS
algorithms. However, the HAL algorithm scores are affected a lot (compare HAL
and N HAL in Tables 4 and 5). This difference in results can be caused by the
fact that LSA does not use any window and works with whole texts. The COALS
algorithm may profit from using the correlation principle[
The scores in our tables show that especially the COALS method is very successful. The best scores are achieved by COALS for the plain texts, and COALS scores for the lemmatized texts are also among the best ones (compare Table 4 and 5).
The HAL method is also very successful. Furthermore, the best score of 0.72 is obtained using the HAL method on lemmatized data without stopwords (see Table 5). It turns out that HAL even outperforms COALS when only pairs containing only very common words are left. On the other hand, this shows the strength of COALS when also considering low occurring words in our corpora.
It turned out that the LSA algorithm is not as effective as the other
algorithms in our experiments. Our hypothesis is that scores of LSA would be better
when experimenting with larger corpora such as Rohde [
Our results differ from scores of tests evaluated on English corpora and
performed by Rohde [
The last Figure 2 in our paper compares human and HAL judgments about the relatedness of 14 pairs containing the most common words from the RG word list in the Cˇ TK corpora. The English equivalents to the Czech word pairs are listed in Table 6. We can notice the pairs which spoil the scores of the tested algorithms in the graph. The graph also shows the difference in human and machine judgments. The pair ”automobile-car” is less related than ”food-fruit” for the algorithms than for humans. On the other hand, the words of the pair ”coast-shore” are more related for our algorithms than for humans. 10 14 19 24 27 29 32 35 37 44 51
Count of omitted pairs
Our experiments showed that HAL and COALS algorithms performed well and better than LSA on the Czech corpora. Our hypothesis based on our results is 0,80 0,70 0,60 n 0,50 o i t lae 0,40 r r Co 0,30 0,20 0,10 0,00 N_COALS_m2 N_HAL_m2_s1 LSA 3.5 3 sse 2.5 tedn 2 a leR 1.5 0.5 1 0 HAL Human
Pairs containing the most common words in the ČTK corpora u´stav - ovoce asylum - fruit bratr - chlapec brother - lad ovoce - pec fruit - furnace j´ızda - plavba journey - voyage pobˇreˇz´ı - les coast - forest j´ıdlo - ovoce food - fruit u´smˇev - chlapec grin - lad auto - j´ızda car - journey pobˇreˇz´ı - kopec coast - hill pobˇreˇz´ı - beh coast - shore u´stav - hˇrbitov asylum - cemetery kluk - chlapec boy - lad bˇreh - plavba shore - voyage automobil - auto automobile - car that COALS semantic spaces are more accurate for low occurring words, while semantic spaces generated by HAL are more accurate for pairs of words with higher occurrence. Our experiments show that the lemmatization of corpora is the appropriate approach to improve the scores of algorithms. Furthermore, the best scores of correlation were achieved when only the ”open class” words were used.
It turned out that the translation of the original English RG test was not so appropriate for our Czech corpora while it contains words which are not so common in the corpora. However, we believe that when the pairs containing low occurring words were removed, the applicability of the test was improved. The evidence for this is a discovered dependency between the scores of tested algorithms on omitting pairs with low occurring words in them.
We believe that semantic spaces are applicable for the query expansion task which we will focus on in our future work. Apart from this, we are attempting to get some larger Czech corpora for our experiments. We also plan to continue testing the HAL and COALS algorithms, which performed well during our experiments.
The work reported in this paper was supported by the Advanced Computer and Information Systems project no. SGS-2010-028. The access to the MetaCentrum supercomputing facilities provided under the research intent MSM6383917201 is also highly appreciated. Finally, we would like to thank the Czech News Agency for providing text corpora.