Extended Abstract. A system that recognises cross-lingual plagiarism needs to establish – among other things – whether two pieces of text written in different languages are equivalent to each other. Potthast et al. (2010) give a thorough overview of this challenging task. While the Joint Research Centre (JRC) is not specifically concerned with plagiarism, it has been working for many years on developing other cross-lingual functionalities that may well be useful for the plagiarism detection task, i.e. (a) cross-lingual document similarity calculation, (b) subject domain profiling of documents in many different languages according to the same multilingual subject domain categorisation scheme, and (c) the recognition of name spelling variants for the same entity, both within the same language and across different languages and scripts. The speaker will explain the algorithms behind these software tools and he will present a number of freely available language resources that can be used to develop software with cross-lingual functionality.
The JRC’s motivation to work on multilingual and cross-lingual language
technology applications was to offer readers and news analysts access to up to
150,000 online media articles per day in about fifty languages that are gathered and
processed by the fully automatic Europe Media Monitor (EMM) family of
applications
The name spelling variant recognition is performed daily for all newly identified
entities in 20 languages, by first normalising the names to a simplified consonant
signature and by then applying a string similarity measure between all new names and
all previously known names in the JRC’s name database. While other string distance
similarity algorithms rely on learning equivalence rules from bilingual name lists
The method to find related news clusters across the twenty NewsExplorer
languages relies on simple cosine similarity calculations between four
languageindependent cluster representations: (a) a ranked list of EuroVoc subject domain
codes; (b) a frequency list of the named entity mentions found in the clusters; (c) a
frequency list of geo-references found and (d) a log-likelihood-weighted list of words
found in the cluster. The first three representations are language-independent
representations, i.e. they are numerical representations of normalised subject domains,
persons, organisations and locations. Unlike the common cross-lingual text similarity
measures, which are either based on translation and then using monolingual similarity
measures or on bilingually trained vector space representations such as LSA
A number of further multilingual linguistic resources are available from http://langtech.jrc.ec.europa.eu/JRC_Resources.html.
Relevant references
Steinberger Ralf, Bruno Pouliquen, Mijail Kabadjov & Erik van der Goot (2011).
JRC-Names: A freely available, highly multilingual named entity resource. Proceedings of the 8th International Conference Recent Advances in Natural Language Processing (RANLP'2011), pp. 104-110. Hissar, Bulgaria, 12-14 September 2011.
Steinberger Ralf, Mohamed Ebrahim & Marco Turchi (2012). JRC EuroVoc Indexer JEX - A freely available multi-label categorisation tool. Proceedings of the 8th international conference on Language Resources and Evaluation (LREC'2012), pp. 798-805, Istanbul, 21-27 May 2012.
Vinokourov Alexei, John Shawe-Taylor, Nello Cristianini (2002). Inferring a semantic representation of text via cross-language correlation analysis . Advances of Neural Information Processing Systems 15.