[Context & Motivation] In the SUPERSEDE project, methods and tools have been developed to collect and analyze user feedback, to identify relevant information for deciding which are the most important requirements to be considered for the next release of a product. [Question/problem] Even if the project proposal was to analyze feedback in the English language only, later it emerged that there was a need to analyze multi-lingual (German, English) feedback. [Principal ideas] We considered two di erent solutions: 1) translating user feedback from German to English, and processing it with the techniques developed for the English language; 2) exploiting Natural Language Processing (NLP) techniques for German to analyze directly the feedback in German. [Contribution] In this short report we describe this project experience, summarizing main commonalities and di erences between the aforementioned solutions.
Copyright c 2018 by the paper's authors. Copying permitted for private and academic purposes.
1Project started in May 2015 and will end in April 2018. Website: www.supersede.eu textual feedback from the industrial case study translated from German to English; (ii) building analysis tools directly for German textual feedback.
The implementations of both solutions rest on a similar process that includes the following steps: (1) dataset preparation, where manual annotation of feedback messages by type and sentiment is performed by a domain expert. Type includes the following labels: Bug Report, Feature Request, Enhancement Request, and Other, while sentiment is labeled as negative, neutral, and positive; (2) pre-processing, where uninformative tokens are removed; (3) feature extraction, where di erent linguistic properties and sentiment are extracted; (4) feedback classi cation, where machine-learning techniques are employed to train a classi er on a (portion of the) dataset.
Among the main di erences in the implementations of the two solutions are: (a) an additional activity for the dataset preparation step was requested for the rst solution, that is the feedback was translated from German to English by a domain expert in SEnerCon; (b) in the feature extraction, di erent type of features were extracted in the two solutions, in particular for the rst solution combinations of the speech-acts used in the messages were extracted, by applying a novel technique that was developed for English text [MKP17]. Moreover, since feedback data were scarcely available at the beginning of the project, we have used openly available datasets that closely mimic the characteristics of the feedback data we analyze. In particular, we used user feedback from the issue tracking system of the OpenO ce Writer application, which were available in English. Since the second solution was implemented later in the project, we used directly the dataset of feedback in German from the SEnerCon use case which were collected during the second year of the project.
Applying the two approaches, we were able to obtain reasonable results, considering the fact that the datasets available were very much limited in size. In particular, for the rst solution (translating to English), the dataset from SEnerCon was composed of 575 messages translated to English from German and annotated by domain experts. On this dataset, we obtained classi cation accuracy of 83%. Similar results were also achieved for sentiment. On the other hand, for the second solution (directly analyzing feedback in German), the dataset was composed of 600 messages in German annotated by domain experts. The accuracy of the analysis was 59:20% for classi cation and 65:81% for sentiment. It is important to note here that the underlying machine learning techniques applied in the two approaches are also di erent. However, in both cases the size of the dataset is quite small. Hence, when in the future when more user feedback data becomes available, the accuracy of the trained models is expected to improve.
In conclusion, the decision regarding the two approaches depends, among other things, on availability of resources and the intended application of the tool. If the required expertise, domain and language knowledge are available in house at the time of the development of the analysis tools and potentially in future use of the tool, then implementing the analysis tools to work directly on the feedback messages in the original language (e.g., German) is the optimal choice. Otherwise it is useful to consider the application scenario of the analysis tool as well. If models are built once from the dataset and then used afterwards without the need for continuous update, then adopting the option of translating to English may be considered.
Acknowledgement This work is a result of the SUPERSEDE project, funded by the H2020 EU Framework Programme under agreement number 644018. We also thank the Future Media group of FBK for their contribution. [MKP17]