We present a general framework, RatVec, for learning vector representations of non-numeric entities based on domain-speci c similarity functions interpreted as rational kernels. We show competitive performance using k -nearest neighbors in the protein family classi cation task and in Dutch spelling correction. To promote re-usability and extensibility, we have made our code and pre-trained models available at https://github.com/ratvec.
The success of distributed vector representations in natural language processing
(NLP) tasks has motivated their use for other areas such as biological sequence
analysis [
Copyright c 2019 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
Our approach extends our previous work on KPCA embeddings, where we studied
vector representations for words and DNA sequences via KPCA replacing the dot
product by a speci c similarity function [
Let X be a dataset consisting of n elements, to which we will further refer as the full vocabulary ; and k a rational kernel (a particular similarity function). Computing a kernel matrix K from X may be computationally prohibitive for large datasets due to its time and space complexity (O(n2)). Hence, we select only the m elements considered "most representative" (in a domain-speci c formulation) from X to construct our representative vocabulary V and compute a kernel matrix KV from all element pairs from V . After centering and diagonalizing KV, we construct our projection matrix PV . This is required to generate m-dimensional representations for the full vocabulary. In a last step, we assign the rst d m components of each computed vector to each full vocabulary element. There are the resulting d-dimensional vector representations obtained with our approach. When k is suitable for the task, d being of a lower order of magnitude than m can lead to optimal results, as we will see in Section 4.1. The choice of m can be adjusted to t the computational resources of the user.
The produced representations tend to cluster naturally according to the domain-speci c concept of similarity applied by the selected rational kernel, as we can see in Fig. 1. This has two main advantages: 1. A simple k NN classi er (eventually 1NN) can su ce for classi cation tasks. 2. Learning the vector representations and k NN constitute an explainable classi cation pipeline: an entity is assigned to a particular class because it is similar to the labeled entities used to train the classi er.
Protein Family Classi cation
In this task, we classify protein sequences to their corresponding families with the
same Swiss-Prot dataset as in [
We produced protein representations with 25 dimensions applying our
approach with bigram and trigram similarity and trained a nearest neighbor
classi er (1NN). We generated our representative vocabulary by taking the shortest
sequence of the 1,000 most frequent protein families. We evaluated this pipeline
in the same setting as [
The results from Table 1 show that our approach with the bigram similarity reaches the same accuracy as the baseline and or even outperforms it when we restrict the dataset to the rst 3,000 or 4,000 most frequent protein families. Considering that the reported BioVec representations are four times longer than ours, our approach to encode proteins seems to be more e cient.
In contrast to [
6 https://github.com/kyu999/biovec
We also evaluated our approach by participating in the CLIN28 shared task,
where our system obtained the best F1 score among the competing teams [
Spell checkers involve two steps: misspelling detection and correction. The
latter generally requires ranking a set of correction candidates. Generating them
implies nding all valid words di ering from the detected misspelling less than a
determined edit distance, which is mostly set to 1 for real-world applications to
limit computation time [
In our RatVec framework, our full vocabulary is wdutch, a word list from the
OpenTaal project, from which the 3000 most frequent words form our
representative vocabulary. The applied rational kernel is the composition of the bigram
similarity [
Once a misspelling is detected, we compute its RatVec representation and search for the closest precomputed vector. Its related word (its nearest neighbor) is our correction to the misspelling. Formally, this is equivalent to training a 1NN classi er where each valid word is assigned a di erent label.
Our RatVec framework is only relevant during the correction phase for spelling mistakes that are related to the word form. Hence, we restrict our analysis to the correction results on the ve error categories where the word form is relevant, namely those displayed in Table 3. Although the baseline system Valkuil achieves a better average accuracy than our approach for the analyzed misspellings, RatVec outperforms Valkuil in some categories where it completely fails (redundant punctuation errors) or where it cannot be even evaluated because it failed to detect any error (capitalization and archaic spelling errors). From these results, we interpret that our word vectors encode word forms in a suitable way so that similar words can be retrieved. 5
We showed that our approach involving rational kernels on KPCA for vector space embeddings provides rich representations for two di erent kinds of en
tities: proteins and words. In the rst application, we presented how to learn protein vectors from amino acid sequences and how to apply them to predict their protein family. In the second, we learned word representations that encode word form information so that they can be applied to correct misspellings once these are detected. In both tasks, our results are comparable to state-of-the-art approaches. Thanks to the simplicity of k NN classi ers and the interpretable similarity concept we apply to generate the vector representations, our approach may be advantageous for some real-world applications compared to more complex machine learning models such as deep neural networks.
In future work, we will explore possibilities of learning optimal similarity metrics (modeled as transducers) that, incorporated in our presented approach, solve a particular task.