In this work, we focus on three different NLP tasks: image captioning, machine translation, and sentiment analysis. We reimplement successful approaches of other authors and adapt them to the Czech language. We provide end-to-end architectures that achieve state-ofthe-art results on all of the tasks within a single sequence learning toolkit. The trained models are available both for download as well as in an online demo.
Traditionally, solving tasks such as machine translation or sentiment analysis required complex processing pipelines consisting of tools which transformed one explicit representation of the data into another, with the structure of the internal representations defined by the system designer. In machine translation, we would devise explicit word alignment links, extract phrase tables, train a language model, etc.; in sentiment analysis, we could label the data with part-of-speech tags, decode their syntactic structure, and/or assign them with semantic labels. All of these more-or-less linguistically motivated internal representations are not inherently required to produce the desired output, but have been devised as clever and useful ways to break down the large and hard task into smaller and manageable substeps.
With the advent of end-to-end training of deep neural networks (DNN), the need for most of this has been eliminated. In the end-to-end learning paradigm, there is only one model, directly trained to produce the desired outputs from the inputs, without any explicit intermediate representations. The system designer now only has to design a rather generic architecture of the system. It mostly does not enforce any complex explicit representations and processing steps, but rather offers opportunities for the DNN to devise its own notion of intermediate representations and processing steps through training.
This also means that similar architectures can be used to solve very different tasks. Rather than by the nature of the task itself, the structure of the DNN to use is mostly determined by the structure of the input and output – e.g. image inputs are processed by two-dimensional convolutions, while text inputs are processed by one-dimensional convolutions, recurrent units, and/or attentions; classification can produce its output in one step, while text generation is better done iteratively using recurrent decoders; etc.
Thanks to that, a single general framework can be used to solve many different tasks. One just needs to transform the inputs and outputs into a suitable format, define an adequate network structure, and let the system train for a few weeks.
Sadly, the burden of hyperparameter tuning has not been alleviated by DNNs, but rather made worse by the computational costliness of the training. However, with a bit of experience, one is often able to propose a suitable architecture and hyperparameter values at the first attempt, already achieving very competitive results even without any further tuning. 1.1
Most of the papers in the field only evaluate their
setups on English datasets. In our work, we try to rectify
this shortcoming by reimplementing existing
state-of-theart approaches in the Neural Monkey framework [
Neural Monkey is an open-source toolkit for
sequenceto-sequence learning, implemented in the TensorFlow
library [
We decided to focus on three rather varied tasks – sentiment analysis, machine translation, and image captioning. For each of the tasks, we reimplemented one or more existing state-of-the-art architectures within Neural Monkey and trained it on available datasets. Our evaluations show that we manage to reach or surpass state-of-the-art results for all the three tasks.
As we wish to encourage other NLP researchers to focus on Czech language, we make sure that our source codes, our configuration files and our trained models are all freely available to anyone interested to use them, to study them and to build upon them. With our work, we hope to establish well-performing approaches for Czech NLP, as well as to allow e.g. investigation of the internals of the trained models to try to decipher in what ways language seems to be implicitly captured in them. Moreover, we have created a simple web-based demo that allows anyone to easily apply our models to any input data, intended to popularize deep learning and its applications for the Czech audience.
The goal of the sentiment analysis tasks is to decide
whether a text expresses positive, neutral or negative
judgment on its topic, sometimes also a degree of the positivity
or negativity.
2.1
We use a state-of-the-art architecture by Lin et al. [
For English, Lin at el. [
After replicating the results on the English dataset, we
evaluated the same approach on a Czech dataset. We also
experimented with architectures based on processing the
input with a convolutional network or a recurrent network
followed by max-pooling in time [
All models use embeddings of size 300 and a classifier with 100 hidden units. In the experiments with CNN, we used kernels of size 3, 4 and 5 with output dimension 100. The LSTM network used 300 hidden units in both directions. The self-attentive layer used 10 heads and a hidden layer of 300 dimensions. 2.2
The largest existing Czech dataset for sentiment
analysis is the CSFD CZ dataset [
The textual reviews are on average 60 tokens long, and bear a rating of 0 to 6 stars, which the authors of the dataset mapped into three classes: negative (0-2 stars), neutral (34 stars), and positive (5-6 stars). The three classes are represented rather uniformly, each being assigned to 32%34% reviews.
We split off 2,000 reviews for validation and another 2,000 for testing (there is no official split of the dataset), retaining the nearly uniform distribution of the classes, as well as other characteristics of the dataset such as average review length.
We use tokenization from the Moses MT toolkit [
Setup
Most frequent class
Maxpool on embeddings
CNN + maxpool
SAN on embeddings
SAN on LSTM
Lenc+ [
Similarly to our approach, Lenc and Hercig [
To the best of our knowledge, the best result on this
dataset has been reported for the “ME + sspace + Dir”
setup of Brychcín+ [
Machine translation (MT) is one of the most well-studied problems from NLP. In general, the goal of MT is given a sentence in a source language, generate a sentence in a target language which as similar meaning as possible to the source sentence. 3.1
We use our implementation of the self-attentive
architecture called the Tranformer [
The architecture uses the encoder-decoder scheme [
In the self-attentive layers, we use the state as a query to
an attention over the remaining states of the layer and
output a weighted combination of the states. This is always
followed by a feed-forward layer. All layers are
normalized [
The decoder uses also attention to the encoder after each self-attentive layer. The decoder is autoregressive. In every time step, a new word is generated and the stack of all the layers applied on the text generated so far, including the newly generated word.
We use hyper-parameters and training strategy proposed
by Popel and Bojar [
We use CzEng 1.75 [
The model is validated on WMT13 test set and
evaluated on WMT17 [
We evaluate the model on the WMT17 test set [
The quantitative results are in Table 3, examples of the outputs in Table 2. As far as we know, this is the best publicly reported MT system for English-to-Czech translation.
Our best performing model was obtained by training for 8 days on 8 GPUs.
In image captioning, the task is to provide a short textual description of a given image – i.e., the input for the task
is an image (a two-dimensional matrix of bits, where each bit is represented by the values of its red, green and blue channel), and the output is a caption (a sequence of words).
As the Czech image captioning dataset is very new, we
believe to be the first ones to train models for the image
captioning task for Czech.
4.1
We re-implement an attentive architecture by Xu et al.
[
Image features are extracted with Resnet50 v2 (8 × 8
× 2048) [
The model is optimized using the Adam optimizer [
At the inference time, we use a beam search of width 5
with length penalty 1.0 [
The dataset uses 29,000 images for training, 1,014 for validation and 1,000 for testing. Unlike the original Flickr30k dataset which contains 5 independent descriptions for each image, we only have one Czech sentence for each image.
This means we can have only have one reference sentence for the evaluation which makes the evaluation less robust than in case of English. Image captioning is usually evaluated using metrics originally developed for machine translation.
There is only one reference in the dataset, while the
standard is to evaluate with BLEU [
The next chance won’t come until winter.
Další prˇíležitost prˇijde až v zimeˇ.
Další šance prˇijde až v zimeˇ.
All private correspondence and images should remain private.
Veškerá soukromá korespondence a obrazy by meˇly z˚ustat soukromé.
Všechna soukromá korespondence a všechny soukromé obrázky by soukromé meˇly z˚ustat.
A group of people are standing in front of a building.
A group of people wearing snowshoes, and dressed for winter hiking, is standing in front of a building that looks like it’s made of blocks of ice.
The people are quietly listening while the story of the ice cabin was explained to them.
A group of people standing in front of an igloo.
Several students waiting outside an igloo.
as more references are not available, we use the chrF3
metric [
We believe our work to be the first to perform image captioning in Czech language. As can be seen in Table 4, the standard evaluation shows rather low scores for Czech.
However, when investigating the data, we found the produced image labels to be usually correct, even if rather simple and generic. See Figure 1 for an example of an input image together with its captions produced by our system. 5
We implemented and trained models for English-to-Czech machine translation, sentiment analysis of Czech texts, and image captioning in Czech within Neural Monkey, using approaches reported to be state-of-the-art for other languages (typically English). We gathered and standardized existing datasets, adapted the Neural Monkey toolkit where necessary, and trained and tuned the tools. Our evaluation shows that the resulting tools reach or surpass stateof-the-art for all three tasks. Both the source codes and the trained models are available online under free licences.67 The tools are also available as an online demo.8
As a future work, we plan to add more tasks, especially text summarization.
7http://hdl.handle.net/11234/1-2839