=Paper=
{{Paper
|id=Vol-1679/paper2
|storemode=property
|title=Estimating Party-user Similarity in Voting Advice Applications using Hidden Markov Models
|pdfUrl=https://ceur-ws.org/Vol-1679/paper2.pdf
|volume=Vol-1679
|authors=Marilena Agathokleous,Nicolas Tsapatsoulis,Constantinos Djouvas
|dblpUrl=https://dblp.org/rec/conf/recsys/AgathokleousTD16
}}
==Estimating Party-user Similarity in Voting Advice Applications using Hidden Markov Models==
https://ceur-ws.org/Vol-1679/paper2.pdf
Estimating party-user similarity in Voting Advice
Applications using Hidden Markov Models
Marilena Agathokleous Nicolas Tsapatsoulis Constantinos Djouvas
Cyprus Univ. of Technology Cyprus Univ. of Technology Cyprus Univ. of Technology
30, Arch. Kyprianos str. 30, Arch. Kyprianos str. 30, Arch. Kyprianos str.
CY-3036, Limassol, Cyprus CY-3036, Limassol, Cyprus CY-3036, Limassol, Cyprus
mi.agathokleous@edu.cut.ac.cy nicolas.tsapatsoulis@cut.ac.cy costas.tziouvas@cut.ac.cy
ABSTRACT 1. INTRODUCTION
Voting Advice Applications (VAAs) are Web tools that in- Citizens, partly because of their lack of knowledge on the
form citizens about the political stances of parties (and/or political issues, tend to avoid the democratic decision mak-
candidates) that participate in upcoming elections. The tra- ing process contributing in low voter turnout that affects
ditional process that they follow is to call the users and the most advanced democracies. Ladner and Pianzola [18]
the parties to state their position in a set of policy state- specifically mentioned Switzerland, where the voter turnout
ments, usually grouped into meaningful categories (e.g., ex- does not exceed 50% by 1975. E-democracy tools and ser-
ternal policy, economy, society, etc). Having the aforemen- vices can be used to inform people about the political stances
tioned information, VAA can provide recommendation to of the parties (and/or candidates) who take part in the up-
users regarding the proximity/distance that a user has to coming elections, aiming at increasing citizen participation
each participating party. A social recommendation approach and promoting direct involvement in political activities [22].
of VAAs (so-called SVAAs) calculates the closeness between Voting Advice Applications (VAAs) are specifically designed
each party’s devoted users and the current user and ranks e-democracy tools that further serve this purpose [17, 26].
parties according the estimated ‘party users’ - user similar- They have been applied to facilitate citizens’ decision mak-
ity. In our paper we stand on this approach and we assume ing process by matching their political stances with those of
that ‘typical’ voters of particular parties can be character- parties and/or candidates. Findings have shown that VAAs’
ized by answer patterns (sequences of choices for all policy recommendations affect the decision making process of a sig-
statements included in the VAA) and that the answer choice nificant part of voters, especially those who are undecided or
in each policy statement can be ‘predicted’ from previous belong to specific categories, such as people under 34 years
answer choices. Thus, we resort to Hidden Markov Models old and/or first time voters [9, 26].
(HMMs), which are proved to be effective machine learning Recommender Systems (RSs) are software tools and tech-
tools for sequential and correlated data. Based on the prin- niques, which recommend products or services to users, in
ciples of collaborative filtering we try to model ‘party users’ an effort to help them decide what they really need from
using HMMs and then exploit these models to recommend the sheer volume of data that many modern online applica-
each VAA user the party whose model best fits their answer tions manage [14, 24]. Although the recommender systems
pattern. For our experiments we use three datasets based are strongly affiliated with the field of e-marketing, several
on the 2014 elections to the European Parliament1 . other application areas were also emerged. Recently, sev-
eral researchers used recommender systems for e-elections
in e-government to inform citizens about candidates and
CCS Concepts enhance their participation in democratic processes [7, 28],
•Computing methodologies → Machine learning; while Katakis et al. [15] introduced SVAAs (Social Voting
Advice Applications), an extended form of VAAs that is
based on the principles of collaborative filtering.
Keywords VAAs ask users and parties to fill a specific questionnaire
Hidden Markov Models; Voting Advice Applications; collab- that contains a number of policy statements, which are se-
orative filtering; expectation maximization; recommender lected according to issues that concern the nation in time
systems of elections and represent important political, economic and
social issues [15, 19]. Figure 1 shows an example of such
1
http://www.euvox2014.eu/ a policy statement along with the set of possible answers a
user can select. The recommendation process that a VAA
traditionally follows contains two main steps: first, it calcu-
lates the similarity scores utilizing the user’s and the par-
Permission to make digital or hard copies of all or part of this work for personal or ties’ and/or candidates’ answers in the policy statements
classroom use is granted without fee provided that copies are not made or distributed and then, the VAA ranks the parties according to party-
for profit or commercial advantage and that copies bear this notice and the full cita- user ‘similarity’. Figure 2 presents an example taken from
tion on the first page. Copyrights for components of this work owned by others than
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- the German VAA of the elections to the European parlia-
publish, to post on servers or to redistribute to lists, requires prior specific permission ment in 2014.
and/or a fee. Request permissions from permissions@acm.org. Researchers from different research fields deal with many
Copyright remains with the authors and/or original copyright holders
Figure 2: Party ranking based on party-user similar-
Figure 1: A question that was included in EUVox ity as computed in traditional VAAs (EUVox 2014,
2014 along with the given set of answer options. Germany data).
aspects of VAAs [25]. Some of them investigate whether
VAAs urge citizens to vote and whether recommendations
made by these systems affect the final vote decision [9, 26].
Other researchers are interested in the design of VAAs deal-
ing with practical issues such as the derivation of optimal
party-user similarity estimation methods that accurately pre-
dict users’ voting intention [20, 21, 29]. We note here that
the estimation of similarity between users based on their
choices from a set of products is a core problem in Recom-
mender Systems as well.
Recently Katakis et al. [15] coined the term ‘Social VAAs’
(SVAAs) in an effort to describe VAAs, whose recommenda-
tion is based on the collaborative filtering philosophy that is
widely used in RSs [12, 13]. SVAAs in addition to parties’
answers to the policy statements, they also utilize models Figure 3: The supplementary questions as they ap-
that capture the behavior - in respect to the policy state- pear in EUVox 2014.
ments - of each party voters. Thus, a social VAA has the
same policy questionnaire with the traditional VAA but also
party voters models created by estimating the joint probabil- corresponds to a specific party. In essence, what is accom-
ity of answer patterns and vote intention of each user. Vote plished with machine learning is to model each party cord-
intention is an opt in question which is included in VAAs as ing to its supporters’ answer patterns to policy statements.
one of the supplementary questions. An example of supple- Thus, if a user is classified into a party, it is more likely
mentary questions included in VAAs is shown in Figure 3, this user has the same political positions with people who
where the vote intention question is the second one. are already classified to the same party. Katakis et al. [15]
In SVAAs users are classified into groups according to resorted to clustering and classification approaches for gen-
their voting intention, i.e., party or candidate choice, and erating vote advice in SVAAs and they showed that party
then models are created for each party to ’show’ the com- voter modeling based on data mining classifiers and Support
mon way, if any, in which party supporters fill the online Vector Machines, achieve the best performance.
questionnaire producing their own answer pattern. Then, Tsapatsoulis and Mendez [30] dealt with building party
the SVAA recommends new user with the party or the can- voter models for SVAAs based on the probability to vote
didate whose users’ model matches better their answer pat- each one of parties participating in the German elections in
terns. Figure 4 presents an example of the matching scores 2013. They compared a Mahalanobis Classifier, a Weighted
presented to a user based on the SVAA philosophy. SVAAs Mahalanobis Classifier and function approximation approaches,
proved to make better voting predictions than the tradi- and they concluded that there is no much gain when using
tional matching schemes between users’ and parties’ pro- the probability to vote instead of the vote intention. They
files [1]. In addition, as recorded by users’ feedback through also noticed that non-linear party modeling techniques, such
the emoticons shown in the right part of Figures 2 and 4, as neural network based ones, outperform the linear meth-
SVAA recommendation surpasses VAA recommendation in ods like Mahalanobis.
terms of users satisfaction [6]. Tsapatsoulis et al. [29] in an effort to provide practical
In order to tackle the recommendation problem of SVAAs, design guidelines for SVAAs dealt with the problem of find-
machine learning techniques [2] can be used to indicate the ing the minimum number of VAA users required to build
likelihood that a user belongs into a class, where each class effective party’s voter models. They limited their analysis
vantage compared to other machine learning methods: they
can capture the correlation between answers in different pol-
icy statements.
In short, the purpose of our paper is to introduce an SVAA
method for similarity matching between parties and users
based on HMMs and investigate its performance based on
the accuracy of predicting their voting intention. We show
that, even if the order in which the questions are answered
in a VAA does not really matter, the HMM classifier per-
forms quite well in estimating vote intention of unseen users.
Nevertheless, the HMMs’ performance relies on the smooth
distribution of samples per party and on the consistency
between the answers of the users, who are classified as be-
longing to these parties. Therefore in the cases where these
conditions are not met, the results may not be satisfactory;
in such case datasets used for training should be cleaned
Figure 4: Party ranking based on matching scores using outlier and/or rogue detection techniques [5].
between party models and user’s answer pattern. To the best of our knowledge this is the first time HMMs
are used to compute party-user similarity either in VAAs
or in SVAAs. For our experiments we use three datasets
derived from EUVox 2014. EUVox is an online application
to the Mahalanobis Classifier for minimize the factors in-
that was sponsored by the Open Society Initiative for Eu-
fluencing their research questions. They found that, as the
rope (European Elections 2014) and the Directorate-General
number of parties modeled is increased the performance of
for Communication of the European Parliament (area of
recommendation is decreased. In addition they showed that
effective party voter models can be built based on a rather internet-based activities/online media âĂŞ 2014). It pur-
small number of user profiles. pose was to help voters to have quick access to informa-
In this work we adopt the social approach of VAAs and we tion related to the political positions of the parties partici-
investigate the application of Hidden Markov Model (HMM) pated in the 2014 elections to the European Parliament (see
classifiers for party-user similarity estimation in an effort more information at http://www.euvox2014.eu/). The cho-
to improve the effectiveness of social vote recommendation. sen datasets differ in size, in the number of parties par-
HMM classifiers provide a way to apply machine learning to ticipating in the elections and in the population’s distribu-
data represented as a sequence of correlated observations [2]. tion percentage among the various parties. An important,
In VAAs the order in which policy statements are dis- possible, contribution to researchers belonging to the Rec-
played to users is not important; however, policy state- ommender Systems community is that the corresponding
ments are usually correlated and grouped into categories datasets, as well as many other VAA datasets, are freely
(e.g., external policy, economy, society, etc). Thus, opting available through the Preference Matcher Website2 . One of
from the various answer choices in each policy statement the aims of the current work is to mobilize researchers of
is related with selections in previous and subsequent policy RSs to investigate the performance of their techniques on
statements. Given that the order of policy statements is VAA data.
kept fixed within each VAA one can assume that (a) answer
patterns, that are sequences of choices for all policy state- 2. PROBLEM FORMULATION
ments included in the VAA that characterize ‘typical’ voters The basic aim of a traditional VAA is to recommend par-
of particular parties can be found, and (b) the answer choice ties to users. In such a case there is a set of N users
in each policy statement can be ‘predicted’ from previous X = {x~1 , x~2 , . . . , x~N }, a set of U policy statements Q =
answer choices. When users answer the questions, they are {q1 , q2 , . . . , qU }, and a set of D political parties or candi-
incrementally producing a sequence of symbols. Whenever dates P = {p~1 , p~2 , . . . , p~D }. Each user ~xj ∈ X and each
a process includes a sequence of dependent observations, political party p~i ∈ P , has answered each policy question
HMM classifiers can be used to model input sequences as qk ∈ Q.
generated by a parametric random process. This is our ba- Based on their answers, every political party or user can
sic rationale for employing HMMs for obtaining similarity be represented in a vector space model:
matching between parties and users for SVAAs.
We assume that VAA users, who support the same party, xj = {x(j,1) , x(j.2) , . . . , x(j,k) , . . . , x(j,U ) }
~ (1)
produce similar sequences of symbols, i.e., answer patterns.
Thus, HMM classifiers can be used to predict and identify
the ‘path’ that users, who support the same party, follow to p~i = {p(i,1) , p(i,2) , . . . , p(i,k) , . . . , p(i,U ) } (2)
answer the online questionnaire, and to create simple and
where x(j,k) , p(i,k) ∈ L are the answers of the j-th user
compact models for each party, so as to be able to clas-
and i-th party, respectively, to the k-th question. The vec-
sify new users into the most probable party class. Although
tors ~
xj and p~i are, usually, named user and party profiles
there is enough evidence about the appropriateness of HMM
respectively.
classifiers for SVAA recommendation, they have not been
A typical set of answers is a 6-point Likert scale: L ={1
applied so far. This is probably due to the fact that there
(Completely disagree), 2 (Disagree), 3 (Neither agree nor
are simpler machine learning techniques that can be used in-
2
stead. However, we strongly believe that HMMs have an ad- http://www.preferencematcher.org/?page id=18
disagree), 4 (Agree), 5 (Completely agree), 6 (No opinion)}. discrete time instant, the system switches from one state to
In several cases, and in the majority of SVAA methods pro- another, while an observation is produced by the probability
posed so far, the sixth point is not taken into consideration distribution according to the current state [16]. In an HMM,
since it does not correspond to a particular stance and is usu- the states are not observable, i.e., they are ‘hidden’, but an
ally replaced with the third point, i.e., with ‘neither agree observation is generated as a probabilistic function of the
nor disagree’. In this work we decided to keep the sixth state, when the system visits the state [2].
point as a distinct emission symbol (see also Section 3) in An HMM is described by three parameters: λ = (A, B, π),
order to avoid a common criticism by political scientists who which can be estimated based on specialized Expectation
strongly argue about the difference between these two cate- Maximization (EM) techniques, such as the Viterbi or the
gories [3]. As a result the set L, in the context of this study, Baum-Welch algorithm. The parameters are calculated through
becomes: L ={1,2,3,4,5,6}. Figure 1 shows an example of several training iterations, by using the entire training data
the way the policy statements in the EUVox 2014 appear set at each time, until an objective function is maximized.
and how the answer options are presented to VAA users. To avoid knowledge corruption, the data should be storage
The VAA recommendation task tries to approximate the in memory and be trained from the start at each iteration,
unknown relevance h(j, i) of user j to party i given the user’s a costly and time consuming process. Therefore in real life,
answers x~j and then to suggest a ranking of political parties the datasets used for training HMMs are often small and
based on user-party similarity. In machine learning terms, this might significantly reduce their performance since the
the task is to approximate the hidden function h(j, i) with a effectiveness of HMMs depend heavily on the availability of
function ĥ : RU × RU → R, where ĥ(~ xj , p
~i ) is the estimation a sufficient quantity of representative training data to cal-
of the relevance of user j with political party i. Typically culate the model parameters [16].
ĥ(~ ~) ∈ [0, 1]. In each case, the top suggestion pjq for user
x, p As already stated, in this work we try to optimize SVAA
j should be: recommendation with the aid of a Hidden Markov Model
classifier. This is, probably, the first time the HMMs are
used in SVAAs and one of the very few times used in Rec-
pjq = argmax(ĥ(x~j , p~i )) (3) ommender System applications in general. A possible ex-
| {z }
i planation is the fact that within a VAA, and in many RSs,
In many VAAs, the users are asked to answer a number of the observations corresponding to user (answer) choices are
supplementary questions in addition to the U policy state- not time dependent. However, as we already mentioned, in
ments. One of these supplementary (opt in) questions is the VAAs user answer choices can be considered as a sequence
vote intention of user i.e., which party the user intends to of correlated observations while HMM states could corre-
vote in the upcoming election. An example of the type of spond to the set of permissible answer options (‘Completely
supplementary questions and how they appear in the EUVox disagree’, ‘Disagree’, ‘Neither agree nor disagree’, ‘Agree’,
2014 is shown in Figure 3. ‘Completely agree’). Under these circumstances the HMMs
The main idea behind the SVAA is to use the vote inten- can be applied to VAA, as we have a sufficient number of
tion variable yj and model each party’s voters using statisti- states and a fairly rich set of data.
cal or machine learning approaches. Thus, for every party i
a model M ~ i is created using as training examples the subset
3. METHODOLOGY
Ti of user profiles who expressed voting intention for party i,
An HMM is characterized by [2]:
that is Ti = [~xj |yj =i]. Then, these models can be exploited
to provide a recommendation based on collaborative filter- • A set of W discrete states S = S1 , S2 , S3 , ..., SW , with
ing [11] that takes advantage of a VAA’s voter community. G = g1 g2 ...gT to be the state sequence (i.e., if we have
In this case the top recommendation pjq for user j is given gt = Si that means at time t the system is in state Si ).
by:
• A set of E observations V = v1 , v2 , v3 , ..., vE , with
~ i ))
pjq = argmax(ĥ(x~j , M (4) O = O1 O2 ...OT to be the sequence of observations
| {z } corresponding to states G.
i
In this work we use Hidden Markov Models to create the • A state transition matrix A, that shows the probability
party-voter models M~ i (see Section 3). Thus, Eq. 4 be- of going from state Si to state Sj : A ≡ [aij ] where
comes: aij ≡ P (gt+1 = Sj |gt = Si ).
• An observation emission matrix B, that describes the
pjq = argmax(ĥ(V j , λi )) (5) probability of observing ve in state Sj : B ≡ [bj (e)]
| {z }
i where bj (e) ≡ P (Ot = ve |gt = Sj ).
j
where V is the set of observations corresponding to user • The probability distribution of being in the first state
profile x~j and λi is the party-voters model for party i created of a sequence: π ≡ [πi ] where πi ≡ P (g1 = Si ).
using HMM training. The solution of Eq. 5 is obtained with
the aid of Viterbi algorithm as usually happens in HMM In our implementation we consider HMMs with three states,
classifiers [2]. i.e., W = 3, S = {S1 , S2 , S3 }, labeled as S1 : ‘Negative’, S2 :
An HMM is a double stochastic process that models data ‘Neutral’, and S3 : ‘Positive’ corresponding to answer choices
evolving in time. It is defined by a latent Markov chain, S1 : (Completely disagree, Disagree), S2 : (Neither agree nor
which consists of a finite number of states, and a number of disagree, I have no opinion), and S3 : (Agree, Completely
observation probability distributions for each state. At each agree) that could be given in the U policy statements of the
VAA questionnaire. Furthermore, there are six possible ob- samples per party which is not the case in the Bulgarian and
servations V = {v1 , v2 , v3 , v4 , v5 , v6 }, where v1 : ‘Completely Czech datasets (see Figure 5). These differences helped us
disagree’, v2 : ‘Disagree’ v3 : ‘Neither agree nor disagree’, v4 : to examine the behavior of HMMs when there is no sufficient
‘I have no opinion’, v5 : ‘Agree’, and v6 : ‘Completely agree’. number of data points per party and when the number of
Every state sequence G has length equal to the number of samples varies among parties.
policy statements, i.e., T = U = 30 while the mapping from In order to measure the performance of voting prediction
a user profile x~j (see also Eq. 1) to an emission sequence using HMMs, we took into consideration only the users who
V j = {v1j , v2j , v3j , ..., vE
j
} is obtained as follows: expressed a voting intention for a specific party. Therefore,
the questionnaires of the users, who did not answer the sup-
plementary question on voting intention, or answered either
vqj = x(j,q) + |L| · (q − 1) (6)
‘not decided yet’ or ‘I will not vote’ were exempted. In
where x(j,q) is the answer choice of user j to policy state- all three datasets approximately 40% of the VAA users ex-
ment q (q = 1, ..., E), L is the set of answer options (see pressed voting intention for a specific party. The main char-
also Section 2) and |L| is its cardinality, i.e., the number of acteristics of the used datasets are summarized in Table 1.
answer options in the policy statements. Thus, in our case
|L| = 6. 4.2 Results and Discussion
As an example consider that a VAA user selected ‘Com-
Experiments were designed to investigate the performance
pletely Disagree’ in the 1st policy statement; then, according
of social voting recommendation using HMMs for estimat-
to Eq. 6 the recorded observation in the 1st place of the se-
ing party-user similarity. For the evaluation we divided
quential answers of the voter would be: 1 + 6 ∗ (1 − 1) = 1;
the users of dataset into a training and a test set [8]. A
whereas if the answer choice in the 23rd policy statement
HMM is built against the training set Tr = {(~ xj , yj )|j =
was ‘I agree’, then the observation 4 + 6 ∗ (23 − 1) = 136
1...Nl , yj 6= ∅} consisting of the profile vectors ~ xj corre-
would be registered in the 23rd place of the V j sequence.
sponding to user answers to the online questionnaire along
An HMM is fully described by three parameters: λ =
with the user’s expressed vote intention yj . Evaluation of
(A, B, π). In the framework of this work we consider that
the trained HMMs on unseen data was facilitated using the
each party voters can be modeled by an HMM λi since the
test set Te = {(~ xt , y t ) ∈
xt , yt )|(~ / Tr , t = 1...Nt , yt 6= ∅} which
way VAA users respond to the first policy statement differs
is a set of profiles and voting intention pairs (~ xt , yt ) not used
among supporters of different parties reflecting into different
in the training set.
πi , the same holds for any other policy statement reflecting
In order to perform our experiments we resorted to Mat-
in different Bi , while the way answer choices are given in
lab’s HMM toolbox. This toolbox was built by Kevin Mur-
two consecutive policy statements also varies among differ-
phy and it uses the Baum-Welch (BW) algorithm for esti-
ent party supporters reflecting into different Ai .
mating the parameters of HMMs with discrete outputs [23].
We created an HMM λi = (Ai , Bi , πi ), for every party in-
4. EXPERIMENTAL RESULTS cluded in each one of the datasets. Thus, we ended up with
seven HMMs for the Danish and Bulgarian datasets and
4.1 Datasets ten models for the Czech dataset. After training the party
As in the majority of VAA and SVAA methods, in this models using the training set Tr the test set Te was used to
work we set the performance criterion to be the accuracy classify unseen users, expressed through their profiles, into
of predicting a user’s vote intention. This also aligns with the party in which the user most likely belongs to, i.e., the
the approach followed in Recommender Systems where the user’s answer pattern most accurately fits i-th party’s model.
criterion is the accuracy of predicting users’ ratings. Thus, In the end, to examine the voting prediction performance of
we carried out experiments to measure the performance of HMMs, the real voting intention of each user in the testing
voting prediction by applying the HMM classifier on three set was compared to the predicted voting intention, that is
EUVox datasets derived from Denmark, Bulgaria and Czech the party id of the party in which they were classified. At the
Republic. EUVox is an EU-wide voting advice application end an overall score of how well the algorithm performed was
that was utilized during the 2014 European Parliament elec- calculated using the Precision, Recall and F-measure scores
tions. Its questionnaire consists of 30 policy statements and and then a total weighted average was estimated [29].
it is based on European-wide issues, issues that are salient In Tables 2-4 we can see the results for each party of the
for voters in a particular region, and country-specific issues. three datasets while Table 5 shows the total weighted av-
The policy statements are clustered into three groups; to erages for Precision, Recall and F-measures, and the Mean
those that refer to European Union issues, to those dealing Average Precision (MAP) for each dataset. The aggregate
with economy, and to those related to societal issues. results of HMMs obtained in the Danish and Czech datasets
The three datasets were chosen such as to differ in size. are better than the ones obtained in the Bulgarian dataset,
The number of samples of the Bulgarian dataset is quite but without a marked difference. The HMM classifier achieved
small; approximately 2800 entries were correct and also con- a similar overall prediction performance for the Danish and
tained a voting intention answer. The Czech dataset is ap- Czech datasets, with the former to be slightly better.
proximately 5 times larger than the Bulgarian while the In the Danish dataset the smooth distribution of samples
Danish dataset is the largest; it contains almost 4 times per party (see Figure 5(a)) along with the homogeneity of
more samples than the Czech dataset. In addition the num- answer patterns among the supporters of the same party re-
ber of parties participating in the elections varies among flects in quite smooth performance across parties as it can
the selected datasets while the same holds for the popula- be seen in Table 2. However, the prediction performance
tion distribution among the various parties. The Danish for the sixth party, which holds the majority of the users,
dataset is characterized by a rather smooth distribution of exceeds the performance of the others. The third and the
Table 1: Datasets’ characteristics
# samples # samples in # samples in # parties
Dataset (Questionnaires) the training set the test set modeled
Danish 53284 31970 21314 7
Bulgarian 2755 1653 1102 7
Czech 15278 9167 6111 10
Figure 5: Distribution of samples per party in the training set for (a) Danish dataset, (b) Bulgarian dataset,
(c) Czech dataset
Table 2: HMMs performance per party in the Dan- Table 3: HMMs performance per party in the Czech
ish dataset dataset
Party Id Recall Precision F-measure Party Id Recall Precision F-measure
1 0.2915 0.4925 0.3663 1 0.4778 0.4687 0.4732
2 0.5103 0.4763 0.4927 2 0.2597 0.3262 0.2892
3 0.7948 0.2014 0.3213 3 0.5411 0.2970 0.3835
4 0.3955 0.5779 0.4696 4 0.6953 0.5115 0.5894
5 0.1735 0.6101 0.2702 5 0.2192 0.2874 0.2487
6 0.6132 0.7837 0.6880 6 0.4246 0.1836 0.2563
7 0.5419 0.5483 0.5451 7 0.2889 0.7027 0.4094
8 0.4941 0.5476 0.5194
9 0.2281 0.4171 0.2949
fifth parties have the same number of users and the smallest 10 0.6980 0.7183 0.7080
distribution of samples in the training set. Consequently,
the HMMs for these parties achieved the worst performance
exhibiting high variance between recall and prediction which of parties with fewer samples, such as the third and sixth,
reflected in low F-score. Even so, the results for the third whose HMMs performed better than parties with more sam-
party were better than the results for the fifth party. This ples such as the second, fifth and ninth party. By carefully
shows that the users in the third party depict higher con- examining these cases in Table 3 we see that the low number
sistency on their answers and thus the HMM for this party of samples reflects in unbalanced recall and precision scores,
was more effective compared to that of the fifth party. which in turn lead to low F-scores. The poor performance
The vote prediction performance of HMMs for the Czech for the other parties is possibly due to the non-homogeneity
dataset, shown in Table 3, varies significantly among parties. of user profiles which leads to low scores in both recall and
Once again the HMMs for the parties with the higher num- precision. Non-homogeneity within party supporters occurs
ber of supporters, i.e., the tenth and fourth (see Figure 5(c)) for various reasons, such as different political background
give the best scores. The relatively low performance in vote and different view for the various categories of policy state-
prediction for the supporters of small parties is mainly due ments. For instance, the supporters of the same party might
to insufficient number of samples. However, there are cases have a common view on economy but totally different in EU
Table 4: HMMs performance per party in the Bul- Table 5: The aggregate results of HMMs
garian dataset Dataset Recall Precision F-measure MAP
Party Id Recall Precision F-measure Danish 0.4747 0.5647 0.5158 0.6772
1 0.3426 0.5114 0.4103 Bulgarian 0.4174 0.4933 0.4522 0.6246
2 0.2832 0.3478 0.3122 Czech 0.4934 0.5062 0.4997 0.6683
3 0.1316 0.3571 0.1923
4 0.6000 0.4636 0.5231
5 0.5706 0.6884 0.6240 tion predictions remains high, which is in agreement with
6 0.3409 0.1786 0.2344 the results reported by Tsapatsoulis et al. [29].
7 0.3333 0.0955 0.1485
5. CONCLUSION
In this work we use HMM classifier in order to improve
policy issues. As we explain later in the Conclusion sec- the effectiveness of social voting recommendation feature of
tion, within party clusters can be investigated separately by VAAs. We based on the idea that while the users are an-
modeling data from each specific cluster through a Gaus- swering the VAA policy statements they are incrementally
sian distribution and then generating mixture of Gaussians producing sequences of observations, i.e., answer patterns,
taking into account the ratio of each source [4, 27]. It is that might characterize ‘typical’ voters of particular parties.
known that whenever the distributed data are asymmetric Thus, the ability of HMMs to capture correlations in sym-
and multi-modal, a mixture of Gaussians can be used to bol sequences would be beneficial. The performance of the
model them [10]. proposed technique was evaluated based on the well known
The results for the Bulgarian dataset, shown in Table 4, Recall, Precision and F-score metrics. We observed that,
are more difficult to interpret. The HMM for the first party, even if the order in which policy statements are displayed
i.e., the one with the majority of supporters, achieved a in VAAs does not actually matter, the HMMs perform very
moderate performance, while the HMM for the fourth party, well in estimating the vote intention of users taking into
which was trained on less than 100 samples, presents the account the intra-sequence correlations. This is not a sur-
second best performance. Once more, the observation made prise as the SVAAs are based on party-voters models and
in previous datasets is confirmed: HMMs can be effectively HMM classifier creates simple and compact models by uti-
trained even with very small training samples when these lizing the ‘path’ that users of the same party create when
samples form a single cluster in the U-dimensional hyper- answering the online questionnaire. Also, the policy state-
space, where U is the number of policy statements. Never- ments in VAAs are grouped together according to the issue
theless, if the number of samples is adequate but there is category that they represent. The statements that refer on
no or low coherence between the profiles of party-supporters the same subject are correlated and are answered similarly
then the results tend to be poor. by the users. Therefore, answering paths are depended and
The overall performance of HMMs in predicting vote in- next answers depend on previous answers of same category.
tention in SVAAs is quite satisfactory as it can be seen in By finding the conditional probability in which a statement
Table 5. Thus, the use of HMMs, which make use of the is given according to category path already occurred, the
conditional probabilities of the VAA user answers, seems to HMMs can effectively provide vote recommendation.
be working. This was expected since the policy statements From our experiments we noticed that the prediction per-
in VAA questionnaires are usually correlated and grouped formance of HMMs depends on the consistency between
into categories representing specific political issues. There- the answers of the users in each party and the distribution
fore, answers to next policy statement can be ‘predicted’ of samples per party. Parties with the majority of users
from previous answers. Also the policy statements are an- achieved the best performance in the Danish and Czech
swered with a specific display order, from the first to last datasets. In the case of Bulgarian dataset, the HMM for
one, and is kept constant for a specific VAA creating se- the party with the highest percentage of samples presented
quences of symbols; people who support the same party are moderate results, while the HMM for the fourth party with
likely to create similar sequences, since they usually share very few users (less than 100) achieved the second best per-
same political opinions. Thus, an HMM classifier, by uti- formance. This lead us to the observation that in some cases
lizing answer patterns of users supporting the same party, the party-supporters profiles create a multi-modal clustering
is able to create simple and compact models that perform in the policy statements hyperspace (due to different politi-
quite well in terms of prediction scores. cal backgrounds and different views in the various categories
By applying HMMs to VAAs we realized that HMM clas- of policy statements). In such cases the use of mixture of
sifier performance is closest to Mahalanobis classifier behav- Gaussians [10] or different clustering techniques could be
ior in other VAAs, while it surpasses the performance of beneficial. In the near future we plan to tackle this problem
other machine learning algorithms, which were applied in by using per party and per category of policy statements
the past to model user-party similarities (see Agathokleous HMMs. Thus, a combination of HMMs for party-supporters
et al. [1], Katakis et al. [15], Tsapatsoulis and Mendez [30], modeling will be pursued to account for the multi-modal
Tsapatsoulis et al. [29]). We noticed, however, that imper- distribution of VAA user profiles within the same party.
fect modeling happens either due to insufficient number of
samples of a party or because of the inconsistency among
users classified to the same party. Nevertheless, the non-
6. REFERENCES
accurate results for small parties do not critically affect the [1] M. Agathokleous, N. Tsapatsoulis, and I. Katakis. On
design of social recommendation, i.e., the overall vote inten- the quantification of missing value impact on voting
advice applications. In Engineering Applications of Cybernetics, IEEE Transactions on, 44(7):1039–1052,
Neural Networks, pages 496–505. Springer, 2013. 2014.
[2] E. Alpaydin. Introduction to machine learning. MIT [16] W. Khreich, E. Granger, A. Miri, and R. Sabourin. A
press, 2014. survey of techniques for incremental learning of hmm
[3] A. Baka, L. Lia Figgou, and V. Triga. ‘neither agree, parameters. Information Sciences, 197:105–130, 2012.
nor disagree’: a critical analysis of the middle answer [17] A. Ladner, J. Fivaz, and J. Pianzola. Voting advice
category in voting advice applications. International applications and party choice: evidence from
Journal of Electronic Governance, 5:244–263, 2012. smartvote users in switzerland. International Journal
[4] S. Dasgupta. Learning mixtures of gaussians. In of Electronic Governance, 5(3-4):367–387, 2012.
Foundations of Computer Science, 1999. 40th Annual [18] A. Ladner and J. Pianzola. Do voting advice
Symposium on, pages 634–644. IEEE, 1999. applications have an effect on electoral participation
[5] C. Djouvas, K. Gemenis, and F. Mendez. Weeding out and voter turnout? evidence from the 2007 swiss
the rogues: How to identify them and why it matters federal elections. In Electronic participation, pages
for vaa-generated datasets. In Proceedings of the 2014 211–224. Springer, 2010.
European Consortium for Political Research General [19] J. Lefevere and S. Walgrave. A perfect match? the
Conference, pages 1–7. ECPR, 2014. impact of statement selection on voting advice
[6] C. Djouvas and N. Tsapatsoulis. A view behind the applications’ ability to match voters and parties.
scene: Data structures and software architecture of a Electoral Studies, 36:252–262, 2014.
vaa. In Semantic and Social Media Adaptation and [20] T. Louwerse and M. Rosema. The design effects of
Personalization (SMAP), 2014 9th International voting advice applications: Comparing methods of
Workshop on, pages 136–141. IEEE, 2014. calculating matches. Acta politica, 49(3):286–312,
[7] K. Dyczkowski and A. Stachowiak. A recommender 2014.
system with uncertainty on the example of political [21] F. Mendez. Modelling proximity and directional logic
elections. In Advances in Computational Intelligence, in vaas. Paper presented at ECPR, 5:7, 2014.
pages 441–449. Springer, 2012. [22] M. E. Milakovich. The internet and increased citizen
[8] M. D. Ekstrand, J. T. Riedl, and J. A. Konstan. participation in government. JeDEM-eJournal of
Collaborative filtering recommender systems. eDemocracy and Open Government, 2(1):1–9, 2010.
Foundations and Trends in Human-Computer [23] K. Murphy. Hidden markov model (hmm) toolbox for
Interaction, 4(2):81–173, 2011. matlab. online at http://www. ai. mit. edu/˜
[9] J. Fivaz, J. Pianzola, and A. Ladner. More than toys: murphyk/Software/HMM/hmm. html, 1998.
a first assessment of voting advice applications’ impact [24] F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor.
on the electoral decision of voters. Technical Recommender Systems Handbook. Springer, 2011.
Report 48, National Centre of Competence in [25] M. Rosema, J. Anderson, and S. Walgrave. The
Research (NCCR): Challenges to Democracy in the design, purpose, and effects of voting advice
21st Century, 10 2010. applications. Electoral studies, 36:240–243, 2014.
[10] M. Gales and S. Young. The application of hidden [26] O. Ruusuvirta and M. Rosema. Do online vote
markov models in speech recognition. Foundations and selectors influence electoral participation and the
trends in signal processing, 1(3):195–304, 2008. direction of the vote. In ECPR general conference,
[11] J. L. Herlocker, J. A. Konstan, and J. Riedl. pages 13–12, 2009.
Explaining collaborative filtering recommendations. In [27] Y. Song, L. Zhang, and C. L. Giles. Automatic tag
Proceedings of the 2000 ACM conference on Computer recommendation algorithms for social recommender
supported cooperative work, pages 241–250. ACM, systems. ACM Transactions on the Web (TWEB),
2000. 5(4), 2011.
[12] J. L. Herlocker, J. A. Konstan, L. G. Terveen, and [28] L. Terán, A. Ladner, J. Fivaz, and S. Gerber. Using a
J. T. Riedl. Evaluating collaborative filtering fuzzy-based cluster algorithm for recommending
recommender systems. ACM Transactions on candidates in e-elections. Fuzzy Methods for Customer
Information Systems (TOIS), 22(1):5–53, 2004. Relationship Management and Marketing, 2012.
[13] M. Jamali and M. Ester. A matrix factorization [29] N. Tsapatsoulis, M. Agathokleous, C. Djouvas, and
technique with trust propagation for recommendation F. Mendez. On the design of social voting
in social networks. In Proceedings of the fourth ACM recommendation applications. International Journal
conference on Recommender systems, pages 135–142. on Artificial Intelligence Tools, 24(3), 2015.
ACM, 2010. [30] N. Tsapatsoulis and F. Mendez. Social vote
[14] D. Jannach, M. Zanker, A. Felfernig, and G. Friedrich. recommendation: Building party models using the
Recommender systems: an introduction. Cambridge probability to vote feedback of vaa users. In Semantic
University Press, 2010. and Social Media Adaptation and Personalization
[15] I. Katakis, N. Tsapatsoulis, F. Mendez, V. Triga, and (SMAP), 2014 9th International Workshop on, pages
C. Djouvas. Social voting advice applications – 124–129. IEEE, 2014.
definitions, challenges, datasets and evaluation.