=Paper=
{{Paper
|id=Vol-1589/MultiLingMine3
|storemode=property
|title=Profile-based Translation in Multilingual Expertise Retrieval
|pdfUrl=https://ceur-ws.org/Vol-1589/MultiLingMine3.pdf
|volume=Vol-1589
|authors=Hossein Nasr Esfahani,Javid Dadashkarimi,Azadeh Shakery
|dblpUrl=https://dblp.org/rec/conf/ecir/EsfahaniDS16
}}
==Profile-based Translation in Multilingual Expertise Retrieval==
https://ceur-ws.org/Vol-1589/MultiLingMine3.pdf
Profile-based Translation in Multilingual
Expertise Retrieval
Hossein Nasr Esfahani, Javid Dadashkarimi, and Azadeh Shakery
{h_nasr,dadashkarimi,shakery}@ut.ac.ir
School of ECE, College of Engineering, University of Tehran, Iran
Abstract. In the current multilingual environment of the web, authors
contribute through a variety of languages. Therefor retrieving a number
of specialists, who have publications in different languages, in response
to a user-specified query is a challenging task. In this paper we try to
answer the following questions: (1) How does eliminating the documents
of the authors written in languages other than the query language affect
the performance of a multilingual expertise retrieval (MLER) system?
(2) Are the profiles of the multilingual experts helpful to improve the
quality of the document translation task? (3) What constitutes a good
profile and how should it be used to improve the quality of translation?
In this paper we show that authors’ documents are usually related top-
ically in different languages. Interestingly, it has been shown that such
multilingual contributions can help us to construct profile-based trans-
lation models in order to improve the quality of document translation.
We further provide an effective profile-based translation model based on
topicality of translations in other publications of the authors. Experi-
mental results on a MLER collection reveal that the proposed method
provides significant improvements compared to the baselines.
Keywords: Expert retrieval, multilingual information retrieval, profiles.
1 Introduction
Expert retrieval has achieved growing attention during the past decade. Users in
the web aim at retrieving a number of specialists in specific areas [3]. A couple
of methods have been introduced for this purpose; retrieving the experts based
on their profiles (the candidate-based model), and retrieving the experts based
on their published contributions (the document-based model) [3]. The latter
approach is usually opted in the literature due to its better performance and its
robustness to free parameters [1].
Since there exist a lot of authors who contribute through a variety of lan-
guages, using documents written in other languages than the query should in-
tuitively be able to improve the performance of the expertise retrieval system.
However scoring documents in such a multilingual environment is challenging.
Multilingual information retrieval (MLIR) is a well-known research problem and
Profile-based Translation in Multilingual Expertise Retrieval 27
has been extensively studied in the literature [10]. There are two options for
scoring documents written in languages other than the language of the query;
translating the query into all the languages of the documents, or representing
all the documents in the language of the query. In MLIR it has been shown
that the second approach outperforms the first one in the language modeling
framework [9]. In the current paper we are going to cast such an approach to
multilingual expert retrieval (MLER). Indeed, our new problem is to retrieve
experts who are contributing in multiple languages.
In this research we choose the document translation approach for our prob-
lem. It is noteworthy that no translated document in the traditional sense is
produced, but rather a multilingual representation of the underlying original
document that is suitable for retrieval, but not for consumption by a reader, is
constructed.
Furthermore, proper weighting of translations has always had a major effect
on MLIR performance. Therefore improving the translation model based on user
profile can supposedly lead to better MLER performance.
We are trying to answer the following research questions in this paper:
1. How does eliminating the documents of the authors written in languages
other than the query language affect the performance of an MLER system?
2. Are the profiles of the multilingual experts helpful to improve the quality of
the document translation task?
3. What constitutes a good profile and how should it be used to improve the
quality of translation?
Our findings in this paper reveal that multilingual profiles of the experts are
useful resources for extraction of expert-centric translation models. To this aim
we propose two profile-based translation models using (1) maximum likelihood
estimation (PBML), and (2) topicality of the terms (PBT). Indeed translations
are chosen based on their contributions in the target language documents of
an expert. Our experimental results on a multilingual collection of researchers,
specialists, and employees at Tuilberg University [5] reveal that the proposed
method achieves better performance on a variety of query topics, particularly in
ambiguous ones.
In Section 2 we provide brief history of studies in the literature of MLER and
MLIR. In Section 3 the proposed profile-based document translation method is
introduced. In Section 4 we provide experimental results of the proposed method
and several baselines and then we conclude the paper in Section 5.
2 Previous Work
There have been multiple attempts in the expert finding literature. Most of the
research studies aim at retrieving a number of experts in response to a query [4].
Usually a couple of models are employed in an expert retrieval system; candidate-
based model and document-based model. Although the former model takes ad-
vantage of lower costs in terms of space by providing brief representations for
the experts, the latter one achieves better results in some collections [1]. A num-
ber of frameworks have been proposed for this aim; model-based frameworks
28 Authors Suppressed Due to Excessive Length
based on statistical language modeling, and frameworks based on topic model-
ing [2,8]. Balog et al. proposed a language modeling framework in which they
first retrieve a number of documents in response to a query and then rank the
documents based on their likelihood to the user-specified query. After employ-
ing an aggregation module, experts are ranked based on their contributions in
the retrieved documents. Theoretically in such a module, there are two factors
affecting the retrieval performance; the query likelihood of the documents of
the experts, and the prior knowledge about the documents. In the lack of prior
knowledge about documents, the documents of an expert are assumed to have
uniform distribution. Deng et al. introduced a citation-based model to improve
the accuracy of the knowledge about the documents [8]. Nevertheless, the former
approach due to its simplicity and its promising results is a popular one in the
literature.
In the current multilingual environment of the web, experts are contributing
in a variety of languages. In such an environment, a reliable strategy should
be employed to bridge the gap between the languages [10,14,7]. A couple of
methods for acheiving this goal are proposed; posing either multiple translated
queries to the system or retrieving multiple translated documents in response to a
query [10]. Although the former method demands an effective rank-aggregation
strategy [12], the latter one achieves promising performance in the language
modeling framework [10]. These approaches in MLIR can also be adapted to
MLER.
3 Profile-based Document Translation
In this section we introduce the proposed expert finding system. The system is
going to be used in a multilingual environment to retrieve a number of experts
in response to a user-specified query. In this environment the documents of the
experts are not necessarily represented in the language of the query.
In MLIR two major approaches are used to overcome this issue. the first
approach translates the query into all the languages of the documents and then
executes multiple retrieval processes and finally aggregates the results; the second
approach represents the documents in all the languages that the query can be
posed in and then executes a single retrieval process. Since superiority of the
latter approach compared to the former one has been shown in the literature
[10], the strategy of the proposed framework lies also on the same road.
To this aim, we use the documents in the profile of an expert to disambiguate
translations of terms in the document. Our assumption is that an expert usually
publishes articles in one area. So we expect to be able to estimate a robust
translation model using the documents of an expert from other languages. In
Section 3.1 we delve into the problem by introducing a novel method to build
a profile for each expert to improve the translation disambiguation quality, in
Section 3.2 we use the proposed profiles to disambiguate translations, and in
Section 3.3 we explain the whole expertise retrieval process.
3.1 Building Profiles for Translation Disambiguation
The main goal of the proposed PDT framework is to use local information of the
experts’ documents to improve the quality of translations. In order to intuitively
Profile-based Translation in Multilingual Expertise Retrieval 29
explain the key idea, consider the following example: suppose an expert has 2
document sets D1 and D2 in languages l1 and l2 respectively and we want to
translate term ws from one of the documents of D1 to language l2 . If ws has two
translations wt1 and wt2 , we investigate how these translations are contributing
in D2 documents. The higher the contribution of a translation in D2 , the more
likely it is to be the correct translation of ws . To this end we first construct
multiple term distributions in different languages for each expert. We explore
two methods to compute the contribution of each term: maximum likelihood
and topicality.
Maximum Likelihood Estimation of Contribution of Each Term: In
this method we assume that the terms that are more frequent in each expert’s
documents are more contributing to the whole profile, so we estimate the con-
tribution of each term in a set of documents D as follows:
P
d∈D c(w; d)
C(w|D) = P (1)
d∈D |d|
In Equation 1, C(w|D) indicates the contribution of term w to document set
D, c(w; d) indicates the number of occurrences of term w in document d and
N (d) is the number of terms in document d.
Topicality Estimation of the Contribution of Each Term: We can use
topicality of each term as the measure of contribution of that term to a document
set. Zhai Lafferty in [15] proposed an EM based method to compute topicality
of terms for pseudo-relevance feedback. We use a similar method: let θelki be
the estimated profile model of expert ei in language lk based on the relevant
document set Delki = {d1 , d2 , .., dn }. According to Zhai & Lafferty we also set λ
to some constant to estimate θelki . Similar to the model-based PRF we estimate
the model with an expectation maximization (EM) method:
(n)
(1 − λ)pλ (w|θelki )
t(n) (w; lk ) = (n)
(2)
(1 − λ)pλ (w|θelki ) + λp(w|C lk )
Pn (n)
(n+1) lk j=1 c(w; dj )t (w; lk )
pλ (w|θei ) = P Pn 0 ; d )t(n) (w 0 ; l )
(3)
w0 j=1 c(w j k
in which lk is the k-th language of the expert ei . λ indicating amount of back-
ground noise when generating documents dj . The obtained language model for
expert ei in language lk is based on topicality of the words. If a word frequently
occurrs in the publications of the expert and also if it is a non-common term
through the collection C lk , it will get a high weight in the profile θelki . Our main
contribution is to use the language models of the experts in different languages
to construct a robust translation model for document translation. Therefore
contribution of each term in document set Dlk would be:
C(w|Delki ) = pλ (w|θelki ) (4)
30 Authors Suppressed Due to Excessive Length
Query (Nl)
3
n ei
p(w|T=1)
e1 En-Nl Nl En-Nl
e2 t1 t2 wi
e3
5 2
4
ER
n ei
PBT: Profile-based
en-1 Translation
en En Nl
1
Pseudo
di d1 d 2 dj Comparable Docs
Fig. 1: The proposed expert retrieval framework in multilingual environments.
3.2 Document Translation Based on Cross-lingual Profiles
In this section we introduce the proposed document translation method based
on the constructed profiles for each expert. Our goal is to construct translation
models for the experts and then to build multilingual documents for them. The
translation model for expert ei is computed as follows:
C(wtj |Delti )
p(wtj |ws ; ei ) ≈ P lt
(5)
j 0 C(wtj 0 |Dei )
in which wT = {wt1 , wt2 , .., wtm } is the set of translation candidates for term ws
from the dictionary. Translations are in language lt and since we have document
translation, wt is in the source language ls .
Combining with Other Translation Models: As shown in the cross-lingual
information retrieval (CLIR) literature, combining different translation tech-
niques can be useful to obtain a robust translation model [13]. In the proposed
framework we also use a general probabilistic dictionary and aim at adapting it
to the domain of each expert. We exploit a simple linear interpolation technique:
pα (wtj |ws ; ei ) = αp(wtj |ws ; θpar ) + (1 − α)p(wtj |ws ; ei ) (6)
where p(wtj |ws ; θpar ) is the translation probability of ws to wtj regarding the
model obtained from a probabilistic dictionary, and α is a controlling constant.
3.3 The Proposed Expert Retrieval Process
Figure 1 shows the whole process of the proposed expert retrieval system. As
shown in the figure, in the first step documents whose languages are different
Profile-based Translation in Multilingual Expertise Retrieval 31
from the query are translated using the PDT framework. This translation tech-
nique is based on Rahimi et al. [10] in which all the translations are considered
in the retrieval process. Indeed documents are scored based on their relevance
to the query. The relevance is computed based on pα (wtj |ws ; ei ) obtained in
Equation 6. Finally experts are scored based on a document-based model:
X
p(q|ei ) = p(q|d)p(d|ei ) (7)
d
For simplicity we estimate p(d|ei ) with a uniform distribution over all the
publications of ei . Moreover we estimate p(q|d) as follow:
Y
p(q|d) = p(w|θd ) (8)
w∈q
Similar to [10] we compute p(w|θd ) in a multilingual environment as follows:
p(w|θd ; ei ) = λpml (w|θd ; ei ) + (1 − λ)p0 (w|C) (9)
in which:
P
0 d∈C cp (w, d) cp (w, d)
p (w|C) = P , pml (w|θd ; ei ) = ,
N d∈C |d| N |d|
X
cp (w, d) = p(w|u; θelki )c(u, d). (10)
u∈d
and N is the number of languages in the collection.
Time Complexity: Although document translation could be time consuming,
and profiled based translation exacerbates the problem, but it is worth men-
tioning that we only translate the terms which are likely to be translated to a
query term. Furthermore the EM process is to be computed once per expert
and could be done offline, hence this process is totally practical. Nevertheless,
the translation model for each expert must be updated when a new document
is inserted.
4 Experiments
In this section we provide experimental results of the proposed PDT framework
and a number of baselines on a multilingual expert retrieval collection.
4.1 Experimental Setups
We used the bilingual TU expert collection [5] in our experiments. This collection
contains a number of documents written by scientists, researchers, and support
staff from Tilburg University The collection is provided in an English-Dutch
environment. Table 1 shows some statistics one the dataset and Figure 2 shows
the contribution of each expert on the set. As shown in Figure 2, experts have
enough documents in both languages which makes the dataset suitable for our
tests.
32 Authors Suppressed Due to Excessive Length
100 PBT EN Queries
0.25
PBML EN Queries
PBT NL Queries
80
# NL Documents
PBML NL Queries
60 0.2
MAP
40
0.15
20
0
0 50 100 0 0.5 1
# EN Documents α
Fig. 2: Distribution of number of docu- Fig. 3: Sensitivity of the interpolation
ments for each expert. framework to α
ID collection Queries #queries #experts #docs µd #qrels
Researchers, Scientists, and EN 1,673 893 16,237 1,336 3,936
TU
Employees at Tuilberg University NL 2,470 881 20,356 1,204 4,868
Table 1: Collection Statistics. µd is the average document length.
Parameter Settings: In all experiments, the Jelinek-Mercer smoothing pa-
rameter λ is set to the typical value of 0.9. All free parameters, particularly the
constant controlling values of the linear interpolations, are set using 2-fold cross
validation over the collection. The noise constant in the EM algorithm is set to
0.7 according to [15].
Evaluation Metrics: We evaluate all the methods based on Mean Average Pre-
cision (MAP) of all the retrieved experts as the main evaluation metric. We also
report the precision of the top 5 (P@5) and top 10 (P@10) retrieved documents.
Statistical differences between the performance of the proposed PDT method
and all the baselines are also computed based on two-tailed paired t-test with
95% confidence level on the main evaluation metric [11]. We also provide ro-
bustness index (RI) [6] for the last set of our experiments for all the competitive
−N−
baselines computed as N+|Q| where |Q| is the number of queries in the collec-
tion. N+ shows the number of queries we have improvements by the proposed
method and N− shows the number of queries in which we have performed worse.
Indeed, RI represents the robustness of the method among the query topics.
4.2 Results and Discussions
In this section we report the experimental results of the proposed method and
some MLER and CLER baselines. The baselines include MLER based on docu-
ment translation using (1) top-ranked translations in a probabilistic dictionary
Profile-based Translation in Multilingual Expertise Retrieval 33
English (EN) Dutch (NL)
TOP-1 MT PAR PBT EN-EN TOP-1 MT PAR PBT NL-NL
1 012∗
MAP 0.2898 0.2740 0.2898 0.2911 0.2633 MAP 0.2656 0.2458 0.2668 0.2674 0.2504
P@5 0.1782 0.1637 0.1782 0.1787 0.1723 P@5 0.1559 0.1392 0.1568 0.1571 0.1474
P@10 0.1208 0.1244 0.1208 0.1212 0.1164 P@10 0.1007 0.0981 0.1016 0.1016 0.0942
Table 2: Using different translation methods for multilingual expert retrieval.
Indicators 0/1/2 denote statistical differences between TOP-1/MT/PAR with
confidence of 95%. ∗ shows the confidence is above 90%.
(TOP-1)1 , (2) document translation based on machine translation (MT), (3)
weighted translation provided by a probabilistic dictionary (PAR), (4) mono-
lingual retrieval by eliminating documents in out-of-the-context languages (the
EN-EN run or the NL-NL one), (5-6) profile-based document translation where
profiles are computed w.r.t maximum likelihood (PBML) and topicality (PBT).
Table 2 shows all the results. As shown in the table, all the MLER base-
lines outperform the simple mono-lingual one. This demonstrates that all the
publications of an author, either those in the language of the query or those in
other languages, are helpful in our retrieval performance. Although the proposed
PBT method outperforms all the baselines in terms of MAP, P@5, and P@10,
the improvements in English queries are marginal. The reason for marginal im-
provements in this dataset goes back to the high performance of the monolingual
results. As shown in the table the results of the mono-lingual runs are competi-
tive to the MLER ones (90.45% and 93.64% of PBT in EN-EN and NL-NL runs
respectively).
We did further experiments to directly study the effect of the proposed
profile-based document translation method. We opted CLER instead of MLER
for this purpose. In Table 3 experimental results of a number of CLER runs are
provided. These experiments are done only on the documents which are in out-
of-the-context languages. To shed light on the effectiveness of the profile-based
translation model, we experiment on a subset of the queries which are ambiguous.
A query is considered to be ambiguous if at least one of its terms is ambiguous.
A term wt is ambiguous if there exists a term ws such that p(wt |ws ) > 0 and
there exist at least 2 term wt0 which p(wt0 |ws ) > δ, where δ is a constant value
(empirically we set δ = 0.2). As shown in the table, PBT outperforms all the
TOP-1, PAR, and PBML baselines in all the evaluation metrics. In the Dutch
queries improvements in terms of MAP are also robust (0.2215 out of [−1, 1]).
Figure 3 shows the sensitivity of the interpolation framework to α (see Equa-
tion 6). As shown in the figure, although the proposed PBT takes advantage
of the interpolation approach in both English and Dutch queries, the overall
changes are very robust to the parameter. Nevertheless, the results of the PAR
baseline without any interpolation with the profile-based translation model drop
considerably in Dutch.
1
We have used a probabilistic dictionary provided by the Google machine translator.
34 Authors Suppressed Due to Excessive Length
English (EN) Dutch (NL)
TOP-1 PAR PBML PBT TOP-1 PAR PBML PBT
MAP 0.1945 0.1955 0.1949 0.2026012 MAP 0.1221 0.1341 0.1455 0.145801
P@5 0.1195 0.1208 0.1229 0.1275 P@5 0.0712 0.0829 0.0883 0.093325
P@10 0.087 0.0867 0.0885 0.09015 P@10 0.0585 0.0647 0.0669 0.0691
RI - -0.1566 -0.0482 0.0172 RI - 0.0196 0.1538 0.2215
Table 3: Experimental results for different translation methods for cross-lingual
expert retrieval over ambiguous queries.
To sum up our findings we answer the following research questions:
1. How does eliminating the documents of the authors written in languages
other than the query language affect the performance of an MLER system?
Regarding the competitive mono-lingual results in Table 2 in the TU dataset
we can claim that the authors repeat majority of their contributions through
languages and so their publications in only one language are almost good but
not complete indicators of their expertise. However this kind of conclusion
is not valid in real-world data and sometimes authors contribute mainly in
a language other than the language of the query.
2. Are the profiles of the multilingual experts helpful to improve the quality
of the document translation task? When we want to translate a document
of an expert, documents of the expert written in the target language help
us to find topical terms. Since correct translations are more likely to be the
topical ones we expect to reach a better translation (see Figure 3).
3. What constitutes a good profile and how should it be used to improve the
quality of translation? According to Table 3 the proposed PBT method out-
performs PBML. This shows that topicality of translations instead of their
simple maximum likelihood probabilities are helpful for the document trans-
lation task. Further results reveal that interpolating the topical probabilities
with values from parallel dictionaries are also useful.
5 Conclusion and Future Work
In this paper we elaborate on the subject of MLER by introducing a novel
profile-based document translation method. We have set a number of research
questions to this aim and our findings supported the following views: (1) Accord-
ing to our observations, although authors contribute almost similarly in multiple
languages, considering all the contributions in different languages can be helpful
for expertise retrieval system. Since authors usually repeat their contributions
through languages, eliminating documents in out-of-the-context languages does
not harm the retrieval performance considerably. (2) Document translation in
MLER takes advantage of profile-based translation models. The profile of each
expert helps us to opt for topical translations which usually contributes to cor-
rect translations. Experimental results on the TU dataset, demonstrate that the
proposed profile-based translation approach outperforms a variety of baselines.
Profile-based Translation in Multilingual Expertise Retrieval 35
An interesting future work of this paper is dynamically learning the interpola-
tion weight between topical probabilities and values from dictionaries based on
generality of words. Constructing profiles for a number of expert clusters and
employing them in the document translation process will be another future work
for this paper.
References
1. Balog, K., Azzopardi, L., de Rijke, M.: Formal models for expert finding in en-
terprise corpora. In: Proceedings of the 29th Annual International ACM SIGIR
Conference on Research and Development in Inf. Ret. pp. 43–50. ACM (2006)
2. Balog, K., Azzopardi, L., de Rijke, M.: A language modeling framework for expert
finding. Inf. Proc. & Man. 45(1), 1–19 (2009)
3. Balog, K., Fang, Y., de Rijke, M., Serdyukov, P., Si, L.: Expertise retrieval. Found.
Trends Inf. Retr. 6, 127–256 (Feb 2012)
4. Balog, K., Fang, Y., de Rijke, M., Serdyukov, P., Si, L.: Expertise retrieval. Foun-
dations and Trends in If. Ret. 6(2–3), 127–256 (2012)
5. Berendsen, R., Rijke, M., Balog, K., Bogers, T., Bosch, A.: On the assessment of
expertise profiles. Journal of the American Society for Inf. Sci. and Tec. 64(10),
2024–2044 (2013)
6. Collins-Thompson, K.: Reducing the risk of query expansion via robust constrained
optimization. In: Proceedings of the 18th ACM Conference on Inf. and Know.
Manag. pp. 837–846. ACM (2009)
7. Dadashkarimi, J., Shakery, A., Faili, H.: A Probabilistic Translation Method for
Dictionary-based Cross-lingual Information Retrieval in Agglutinative Languages.
In: Conference of Computational Linguistic (2014)
8. Deng, H., King, I., Lyu, M.R.: Formal models for expert finding on dblp bibliogra-
phy data. In: Data Mining, 2008. ICDM’08. Eighth IEEE International Conference
on. pp. 163–172. IEEE (2008)
9. Nie, J.Y.: Cross-language information retrieval. Synthesis Lectures on Human Lan-
guage Technologies 3(1), 1–125 (2010)
10. Rahimi, R., Shakery, A., King, I.: Multilingual information retrieval in the language
modeling framework. Inf. Ret. Journal 18(3), 246–281 (2015)
11. Sanderson, M., Zobel, J.: Information retrieval system evaluation: Effort, sensitiv-
ity, and reliability. In: Proceedings of the 28th Annual International ACM SIGIR
Conference on Res. and Dev. in Inf. Ret. pp. 162–169. ACM (2005)
12. Tabrizi, S.A., Dadashkarimi, J., Dehghani, M., Esfahani, H.N., Shakery, A.: Re-
visiting optimal rank aggregation: A dynamic programming approach. In: Interna-
tional Conference on the Theo. of Inf. Ret., ICTIR, September (2015)
13. Türe, F., Lin, J.J., Oard, D.W.: Combining statistical translation techniques for
cross-language information retrieval. In: COLING. pp. 2685–2702 (2012)
14. Vulic, I., Smet, W.D., Tang, J., Moens, M.: Probabilistic topic modeling in mul-
tilingual settings: An overview of its methodology and applications. Inf. Process.
Manage. 51(1), 111–147 (2015)
15. Zhai, C., Lafferty, J.: Model-based feedback in the language modeling approach
to information retrieval. In: Proceedings of the Tenth International Conference on
Inf. and Kno. Man. pp. 403–410. ACM (2001)