<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>How MPs Use Variation in Lower-Level Institutional Design as an Electoral Responsiveness Mechanism</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Responsiveness Mechanism</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Martin Søyland</string-name>
          <email>martin.soyland@stv.uio.no</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>University of Oslo</institution>
          ,
          <addr-line>Gullhaug torg 1, 0484 OSLO</addr-line>
          ,
          <country country="NO">Norway</country>
        </aff>
      </contrib-group>
      <fpage>101</fpage>
      <lpage>117</lpage>
      <abstract>
        <p>Diferent parliamentary activities allow Members of Parliament (MPs) varying amounts of autonomy. Previous studies have shown that, in parliamentary systems with strong parties and party-centered electoral rules, MPs have limited room for crossing the party line in the legislature both in voting and speech. Further, party-centered systems limit MP's ability to address electoral concerns of their constituency; they are less responsive. In this paper, I combine these findings by showing that even within system variation in party control over institutions afects the levels of responsiveness in parliamentary questions. By linking MP's constituency mentions with diferent types of questions, my results show that the institutional design in the Norwegian Storting afects the level of MP constituency signaling. Specifically, I show that questions with low levels of party control and public attention (written questions and question time) give MPs far more opportunity to raise constituency specific issues than the more party controlled activities (interpellations and question hours). Consequently, I argue that responsiveness does not disappear in party-centered systems; it is located at lower-level institutions. Particularly, some types of questions, where shirking from the party line is less consequential and the party organizations have less control over its members, allow for constituency signaling.</p>
      </abstract>
      <kwd-group>
        <kwd>parliamentary institutions</kwd>
        <kwd>parliamentary questions</kwd>
        <kwd>responsiveness</kwd>
        <kwd>named entity recognition</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>Parliamentary questions ofer a tool for Members of Parliament (MPs) and their political parties
to monitor the government in a wide range of parliamentary democracies across the world.
However, parliamentary questions are also used as a tool for MPs to express their opinions
on policy (position signalling), or even as a means to communicate directly with the voters in
one’s constituency. For example, during an interpellation on transport services for people with
disabilities in the Norwegian parliament (Stortinget ) on the 20th of May 2014, MP Eirin Sundt
(Labor Party) stated that:
https://martigso.github.io/ (M. Søyland)</p>
      <p>© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
We are talking about parents, mothers and fathers, who want to participate in their
children’s activities, for instance football tournaments. You can not get from Ålgård
to Kverneland, there is no bus.1</p>
      <p>
        The geographical mentions of Ålgård and Kverneland are interesting in this speech, because
most MPs (or Norwegians in general) will probably not know these two places unless they are
from the area, and certainly not that “football tournament” in this context very specifically
refers to a yearly football tournament for kids in Kverneland (Grønt Gras). So, why does the
MP use a context most of the audience in the national assembly will not recognize? Numerous
previous studies have covered MPs’ propensity to highlight problems within the constituency
they were elected from, and Eirin Sundt was elected from Rogaland whichis the electoral district
for both locations mentioned in the speech. Some studies have also shown that political systems
with strong parties strongly dampens the ability to voice constituency concerns [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Seeing as
Norway has a party party-centered proportional representation electoral system, this should
make speeches as highlighted above rare – in contrast to more candidate-centered systems (UK,
US, etc) or even mixed-member systems (Germany, Italy, etc).However, yet other studies have
shown that the institutional setting of diferent parliamentary activities can dictate how free
MPs are in that specific activity. For example, MPs can have a hard time crossing their party in
voting [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], but have more freedom in plenary debates [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] and diferent types of parliamentary
questions [
        <xref ref-type="bibr" rid="ref4 ref5">4, 5</xref>
        ]. Some studies have even shown that the institutional design of diferent types of
questions are important for MP behavior [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. Still, there has not been given as much attention to
the combination of institutional design and constituency signaling in parliamentary questions.
      </p>
      <p>In this paper, I utilize diferent question types in the Norwegian parliament to show that
lower-level institutional variations matter for the amount of constituency signaling MPs will
engage in. In contrast to higher-level institutional designs, such as electoral systems, lower-level
institutional designs are defined as the rules of an institution that have little or no influence on
incentives and behavior beyond the institution itself. Parliamentary questions are particularly
well-suited for the task at hand. First, the institutional design of question types varies within
systems. Thus, party control will also vary between the diferent types of questions. Second,
MPs use a substantial amount of time on asking the questions, and questions are regarded by
the vast majority – at least in the Norwegian legislature – to serve as an important tool for
setting important issues on the agenda and as a control mechanism towards the government [7,
pp. 452-453].</p>
      <p>I utilize a unique dataset of parliamentary questions and answers coupled with named
entity recognition of MP constituency mentions, to show that the relationship between party
control and lower-level institutional variation matters for how much MPs focus on their own
constituency in the party-centered Norwegian parliamentary system. The variation in diferent
types of parliamentary questions afect how much MPs focus on their constituency. When the
party has less control over question types, MPs tend to do more signaling. More specifically, I
ifnd that the more party controlled and formally restricted legislative activities (interpellations
and question hours) contain substantially less amounts of constituency mentions than less party
controlled settings (written questions and question time). MPs refer to their constituency far
1Translated from Vi snakker her om foreldre, altså mødre og fedre, som ønsker å delta i sine barns aktiviteter, som
f.eks. en fotballturnering. Du kan ikke komme deg fra Ålgård til Kverneland, det går ikke buss.
less in the question hour and interpellations than they do in written questions and question
time. Further, the limited time frame and increased media attention of question hours seems to
give parties even more incentive to control their MPs and push the frontbenchers to the pulpit.</p>
      <p>Consequently, my findings give important insights on MP behavior in parliamentary
democracies; even though the system under scrutiny here is party-centered, the results show that there
is room for MPs to have substantial focus on constituency issues under the right circumstances.
Thus, this paper corroborates and supplements previous findings of studies on variation in
institutional design and the efects of party control on MP behavior in parliamentary
democracies – the diferent types of parliamentary questions do influence what MPs want to and are
allowed to talk about. In sum, both institutional design at the lower-level and, in extension,
party control within these institutions are essential to the level of responsiveness in legislative
activities.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Parliamentary Activities</title>
      <p>Institutional design varies significantly between democracies. At the higher level institutions,
electoral systems difer between those more party-centered and those more candidate-centered.
Some electoral system designs have strict procedures for assigning the prime minister and
cabinet, whereas others have less formalized procedures; even appointments of public administration
personnel can vary between US style spoils system and merit systems.</p>
      <p>At the lower-level institutions, there are all the more diferences – even between parliamentary
democracies and diferent levels of government. From various parliamentary procedures,
through rules for roll-call votes, to seating arrangements, parliamentary systems operate with
diferent (and mostly exclusive) sets of rules. In this section, I will discuss the role of questions
in parliamentary systems and explore some of the most important previous findings with regard
to parliamentary questions and their efect on MP behavior. Further, I outline the important
institutional designs of the Norwegian case both at the higher and lower-level controls, with
emphasis on the design of parliamentary questions.</p>
      <sec id="sec-2-1">
        <title>2.1. Parliamentary Questions</title>
        <p>
          Questions are a part of the institutional tool kit for controlling the executive branch in most
democratic legislatures. Parliamentary questions are non-legislative in nature because they do
not directly produce policy [8, pp. 348], but they nevertheless serve an important purpose as a
monitoring mechanism to scrutinize government policy and an instrument for agenda-setting
[6, pp. 382]. It is also stressed in the literature on parliamentary questions that this is not a
costless efort [ 4, pp. 263]; parliamentarians invest time into formulating questions and cabinet
members usually invest time into answering them. A common criticism of this notion is that
MPs have writers prepare their questions and speeches. However, there are no indications in the
Norwegian case that this is a valid concern. Indeed, if MPs did have writers, it is highly unlikely
that these would act on behalf of the constituency the MP was elected from at the cost of the
party. Accordingly, the results of my analysis would be skewed towards the null-hypotheses,
which is far better than the opposite. The public awareness of the content in these types of
parliamentary activities, nevertheless, seems to be low, as shown by Soontjens [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ]. They also
demonstrate that MPs overestimate voter awareness of questions in parliament.
        </p>
        <p>Another concern with using questions as a source of data is the possibility that questions
can be used by MPs to “waste” time for the governing parties – either by submitting a lot of
questions to increase the workload for the minister or ask unneccessary questions (rethorical).
This is, of course, a plausible, but pessimistic, assumption to make. It is plausible because it
would be naive to assume this never happens, but pessimistic about parliamentary democracy
to assume all questions are asked with this motivation. Nevertheless, the formal restrictions of
the diferent types of questions (there is no opportunity to filibuster), discussed below, will limit
how much time MPs can waste. Further, the remaining types of time wasting are impossible to
control for (unless MPs admit they do this through a survey or similar) and it is not clear how it
would afect the results of my analysis. Consequently, this paper will assume that parliamentary
questions are sincerely used for monitoring, agenda-setting, or control.</p>
        <p>
          The study of parliamentary questions and responsiveness has had a somewhat split focus
between studying the efects of institutional arrangements on the one hand, and responsiveness
in specific types of questions on the other hand. Rozenberg and Martin [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ] provide a thorough
mapping of the diferent types of parliamentary questions over various parliamentary systems
and investigate the consequences of institutional design on how often MPs use diferent types of
questions. They argue that oral question hours and written questions serve diferent functions.
Oral questions are more focused on “general policy issues whereas written questions tend to
address more specific and detailed information requests” [ 10, pp. 395]. Further, written questions
are less visible to the public than question hours. Various empirical studies on parliamentary
questions have also revealed several interesting insights on how the incentives for participating
in diferent types of these non-legislative activities alter MP behavior.
        </p>
        <p>
          For example, Proksch and Slapin [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ] show how national opposition Members of the European
Parliament (MEPs) ask more written questions to Commissioners, arguing that such questions
are regarded as an important oversight tool for the opposition. Although this study only utilizes
written questions, it “highlights an oversight mechanism previous overlooked in by the EU
oversight literature” [11, pp. 73]. Further, Kellermann [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ] shows that British MPs ask more
questions when their previous electoral victory was smaller. He attributes this to MPs that
are more electorally vulnerable want to send stronger signals of efort more than constituency
specific issues. And, Whitaker and Martin [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ] find that opposition parties strategically focus
on asking questions about policies that might uncover intra-coalition tensions in Britain. But
parliamentary questions can also be used by coalition partners to control that their collaboration
partners do not drift [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ].
        </p>
        <p>
          Even more relevant to the question of this paper, Martin [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ] suggests a framework for
utilizing parliamentary questions to measure constituency focus through an analysis of how
Irish MPs convey responsiveness in written questions. Using data on hand-coded constituency
focus in written questions, Martin [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ] not only shows that MPs in the Single Transferable
Vote (STV) electoral system of Ireland use written questions as a means to ask constituency
specific questions, but also that geographic distance from the center (Dublin) is important for
MP behavior in these questions. Adjacent to the Martin [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ] study, Russo [15, pp. 299] finds
that there is variation in written question constituency focus within the Italian parliament,
despite the lack of electoral incentive to cultivate personal vote-seeking. Russo [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ] argues that,
in contrast to American studies, looking for constituency focus in roll call votes is fruitless in
parliamentary systems. MPs in parliamentary systems tend to vote in line with their party.
Therefore, parliamentary questions ofer some advantages: they give MPs opportunities to
defend territorial interests that might cross the party line and tabling a question is a more costly
task than the act of giving a yes/no vote [15, pp. 292].
        </p>
        <p>
          Rasch [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ] explores this from a slightly diferent angle. Looking at the small institutional
diferences between question types in the Norwegian parliament, he shows that the time
constraints between question time and question hours produce large diferences in which and
how often MPs participate. On the one hand, question hour has developed into more of an
activity for frontbenchers because the limited time frame (one hour) gives parties more control
over the time allocation [6, pp. 391]. On the other hand, backbenchers are far more active in
question time, which lasts longer and is more accessible.
        </p>
        <p>These studies have shown both that the institutional design parliamentary questions is
important for MP behavior and that constituency focus is prevalent, even when the electoral gain
incentives are small.My approach aims to combine these two branches in the literature:
unveilhow constituency focus varies over diferent types of questions, often with small institutional
design diferences.</p>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. Stortinget</title>
        <p>
          The particular case of this paper, Norway, has a fairly standard setup with regards to
parliamentary activities, with some notable exceptions. The Norwegian electoral system is a closed
list proportional representation system, with elections every four years, and the parliament is
ifxed for four parliamentary sessions between each election (no snap elections). The session
lenght does not vary more than a couple of days. Further, parties having a strong position
throughout the system; who gets to run for election and their list placement is gated by the party
organizations. This does, of course, afect the day to day work in parliament. MPs that want to
keep their seat have to follow the party line. As voters vote for parties and not individuals, the
electoral gain from crossing the party line will generally be smaller than in candidate-centered
systems. For further discussion on the Norwegian electoral system, see Cox et al. [
          <xref ref-type="bibr" rid="ref16">16</xref>
          ].
        </p>
        <p>
          As the main aim of this paper is to explore the relationship between responsiveness and
institutional design, it is also important to highlight some of the lower-level institutional
arrangements in the Norwegian parliamentary system; specifically how day to day work in
parliament is controlled by various formal and informal rules. Here, I will discuss the formal
rules of interpellations, question hours, question time, and written questions, with emphasis
on how free the MPs are to raise constituency specific concerns within these settings (see also
Rasch [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ]). There are, of course, also opportunities for MPs to voice constituency concerns in
other types of parliamentary activities, such as plenary debates or even voting (voting against
party because of constituency concerns). However, this paper will focus exclusively on the
question types discussed in order to keep the institutional setting of the data as equal as possible.
        </p>
        <p>For the four types of questions allowed in Stortinget, there are a fair amount of similarities in
the rules guiding the questions and answers. First, the Presidency can dismiss questions that
do not fall under the jurisdiction of the cabinet. This is a limitation for raising constituency
concerns by itself, because a lot of constituency specific issues are under the jurisdiction of the
municipalities. The questions, nevertheless, give opportunities for the opposition (or in some
cases MPs from cabinet parties) to ask the sitting cabinet about their policies. However, the MPs
can circumvent this restriction by asking broader questions but refer to their constituency as
an example. Second, all MPs have equal access to the diferent types of questions. This means
that the party organizations do not have any formal power in these institutions; if they want to
influence what issues are taken up in the questions, they will have to do so informally. Third,
cabinet members can always refuse to answer a parliamentary question, but might be required
to justify why. Finally, the Presidency can reject the question if it goes against the etiquette of
the parliament (foul language, inappropriate questions, etc).</p>
        <p>Interpellations are regarded as the formally most strict type of questions in Stortinget. In
addition to the limitations discussed above, interpellations can not be about legislation or
questions already active in the parliament, which will enable MPs to use this question type as
an agenda-setting arena. The interpellation has to be held within one month after it was asked.
This gives the relevant cabinet ministers a lot of time to prepare for the question, even though
they can refuse to answer and even provide justification for not answering. The time frame
for interpellations are 10 minutes for the interpellant and minister to put forward/answer the
question and 3 minutes to close the discussion for both at the end. The particular thing for
interpellations is that the floor is also opened for the other MPs to debate after the introductions.
Each interpellation can last for up to one and a half hours, giving MPs quite a bit of room to
voice their opinion.</p>
        <p>Question hours are less strict than interpellations, but also these kinds of questions have
some formal limitations. The questions in the question hour are meant to be short, and each MP
can only ask one question per week. Cabinet members do not get to prepare for the questions
before question hours. The questioner can ask follow up questions in the question hour, and
the minister can answer these. The floor is then opened up for other MPs to ask follow-up
questions. Time wise, each question in the question hour is restricted to six minutes. This gives
question hours a feeling of debate battles between the opposition and cabinet, which is also the
reason for why these types of questions are more popular to follow by the media and public
than other forms of parliamentary debates and questions (even though they do not produce any
policy) [6, pp. 384]. The President decides when to stop the question hour.</p>
        <p>
          Question time is less strict in terms of time limits. Here, questions are restricted to a maximum
of seven to eight minutes. Question time is similar to question hour, but the floor is not opened
for debate after the questioner and cabinet member have had their say, but follow up questions
by the questioning MP are allowed. Cabinet members are, however, prepared for the questions
in the question time; MPs have to submit their question almost one week beforehand. The
less spontaneous nature of question time also makes this less interesting for the frontbenchers
in the Storting. The question time used to be extremely time consuming because MPs used it
“for partisan or self-serving electoral purposes” [6, pp. 387], and studies showed that electoral
incentives were important for why MPs used the institution extensively [
          <xref ref-type="bibr" rid="ref17">17</xref>
          ].
        </p>
        <p>The solution to limit the time pressure produced by question time was to introduce written
questions in the mid 1990s. Written questions are diferent from the other question types in
that they are asked and answered in written form and require no face-to-face dialogue. Further,
MPs can ask two written questions per week, which is very lenient in comparison to the other
question types. Because the question time, question hour and interpellations are presented in
the plenary, this in itself limits the number of questions that can be asked. Written questions are
required to be very short (with an optional elaboration of max one page), and cabinet members
have six days to answer or justify why they do not want to answer a written question.
18
16
14
12
s
n
o
its10
e
u
q
#8
6
4
2</p>
        <p>Figure 1 shows the number of questions MPs on average have engaged in over the diferent
question types throughout the period covered in this paper.2 What sticks out first is the sharp
and steady increase in the amount of written questions. In the earlier sessions, individuals
on average submitted between 2 and 6 questions per session. Whereas in the later sessions,
they submitted up to almost 18 written questions. Seeing as written questions were introduced
to dampen the time pressure of question time, it is interesting that, although participation
in question time is getting steadily lower over the period, the decrease does not mirror the
increase in written questions. MPs participated just under 12 times per session in question
time over the first half of the period, but the decrease in amounts of this question type only
slowly decreased, reaching an average just above 8 questions in the latter part of the period.
As for interpellations, this is consistently the question type utilized least frequently, averaging
between 2 and 4 questions per MP per session throughout most of the period. Last, question
hour has seen a slight increase in usage, going from an average of around 6 questions per MP
in the early periods to around 8 in the later periods.</p>
        <p>In sum, the diferent types of parliamentary questions have unique sets of formal rules guiding
the MPs. Although all questions are restricted to stay within policy areas under the jurisdiction
of the cabinet, other factors such as preparation time and the form of debate difer substantially.</p>
        <p>2Minister participation is excluded in this plot because ministers will naturally participate more often in
questions; they are fewer in number and at least one will always have to participate.
The expectations for the analysis are listed in Table 1.</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Methods and Data</title>
      <p>In this section, I describe the methods and data used for this paper. I start by giving a short
rundown of the data, before I proceed by outlining the dependent, main independent, and
control variables used in the analyses. Finally, I detail the choice of the undersampled binary
logistic regression model for the analyses.
3.1. Data
The main part of the data for this paper is built on all parliamentary questions, their answers,
and following debates in the period from 1998 to 2021. As outlined above, the institutional
rules across questions vary, which also means that what actors are allowed to participate in the
diferent types vary.</p>
      <p>
        The parliamentary questions were gathered from the Storting’s API (data.stortinget.no), using
the stortingscrape package for R [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]. Even though this makes for a more seamless gathering of
the data, the transcripts from the plenary are largely not standardized and include a fair amount
of errors that were somewhat time-consuming to rectify.
      </p>
      <p>In total, the data consist of 155098 questions produced by Norwegian MPs in the
19982021 period. For the main analysis, however, I exclude questions held by MPs from parties not
represented in all periods (Kystpartiet, Miljøpartiet De Grønne, Rødt, and Tverrpolitisk Folkevalgte).
Additionally, some variables have missing values because of some inconsistencies in the API,
giving a total of 129348 observations in the full model of the analysis (see the online appendix).</p>
      <sec id="sec-3-1">
        <title>3.2. Dependent Variable</title>
        <p>The dependent variable for the main analysis is whether MPs or ministers mention a place
within their constituency in the question, answer, or following debate. Because the paper has
its main focus on exploring whether constituency signalling is used under diferent institutional
settings, rather than the amount of constituency signalling in these settings, I have opted to
use a binary variable. More specifically, the variable is coded as 1 if one or more places were
mentioned and 0 if not.</p>
        <p>Although this coding seems uncomplicated at first sight, operationalizing this variable is not
trivial. There are several ways to identify constituency signaling, or localism, in parliaments.
Martin [14, pp. 476] lists six ways for identifying localism in questions:
1. Did the MP mention her constituency directly?
2. Did the MP mention a location within the constituency?
3. Did the MP mention a case or individual attached to the constituency?
4. Did the MP mention a building or structure within the constituency?
5. Did the MP mention an organization or business geographic arrangements within the
constituency?
6. Did the MP mention an event that occurred within the constituency?</p>
        <p>There are, however, methodological challenges in actually identifying constituency mentions
through these six ways. For example, identifying direct constituency mentions seems
straightforward. But in the Norwegian case specifically, the constituencies can be defined by MP’s
electoral districts, which are counties. However, electoral districts are quite large and MPs do
not necessarily have strong ties to their entire electoral district. They will, most likely, have
strong connections to specific locations within the constituency, such as hometowns, childhood
homes, and so on, which taps into the second type of constituency signalling listed above.
But, reliably identifying places within constituencies can be dificult for a variety of reasons:
there are an almost endless amount of place names, the names often overlap between electoral
districts, and their type difers substantially (mountains, towns, lakes, etc). Similar problems
occur with the remaining four types of localism identifiers. In general terms, a lot of these
indicators can measure localism, but they can also measure other things. Specific organizations,
for example, might be mentioned by all MPs, regardless of constituency. In that case, counting
only mentions of the organization by the MPs from the constituency where that organization is
located would bias the measure, because MPs could mention the organization based on other
factors than constituency specific concerns.</p>
        <p>
          My approach is limited to the second point on the list: whether the MP mentions a location
within the constituency. As discussed, this approach is not flawless, but I try to limit the amount
of false positives as much as possible. The first step of my approach is to automatically extract
named entities from each text in the corpus using spacyr, which is a spaCy wrapper for R [
          <xref ref-type="bibr" rid="ref19 ref20">19, 20</xref>
          ].
Named entities are, in short, named real-world objects, such as persons, geographical locations,
titles, and so on. After the texts are tagged with spaCy, I subset only the named entities tagged
as locations, either in single token entities (Ytterøy) or multi-token entities (Wessels plass). The
entities are then matched with geographical data from Kartverket (Norwegian map data agency).
These data lists all place names in Norway with an indicator for county and municipality for
each place. Seeing as MPs are elected on the county level, if the named entities from a speech
match one or more places in the map data for the MPs constituency (county), my dependent
variable gets a value of 1, and 0 if not.
        </p>
        <p>Figure 2 shows the average amount of constituency mentions for each question per 1000
words of that question over question types and sessions within the range of the data. Written
questions are relatively stable, although with a slight downwards going trend, at between
0.50 and 0.34 average mentions throughout the period. Question hours have low amounts of
mentions in the first half of the period, but has a moderate increase in the latter part of the
period, peaking at 0.35 mentions per 1000 words on average. Question time shows the highest
amount of variation between sessions (0.40 to 0.71), but also higher amounts of mentions over
all periods. The amount only drops below 0.50 mentions in six of the 23 sessions covered
Question time</p>
        <p>Interpellations
Written questions</p>
        <p>Question hour</p>
        <p>Question time
here. Interpellations show very low and stable numbers of constituency mentions on average,
spanning between 0.14 and 0.28 mentions.</p>
        <p>This seems unsurprising, as we expect interpellations and question hour questions to generate
less constituency focus than written questions and question time. Interpellations are on average,
however, substantially longer than other types of questions.</p>
      </sec>
      <sec id="sec-3-2">
        <title>3.3. Independent Variables</title>
        <p>The main independent variable of interest for this paper is the institutional setting of the texts.
The variable is split into the four institutional settings discussed above: 1) written questions
(reference category), 2) question hour, 3) question time, and 4) interpellations.</p>
        <p>However, I expect the role of the participant – MP or minister – to alter the efect of
institutional variation. More precisely, on the one hand, ministers are expected to mention
constituency very rarely in all types of questions, as they seldom have jurisdiction over local
matters (that is usually the responsibility of municipalities) and the minister does not set the
agenda for the question – they only respond to the MP. MPs, on the other hand, are expected
to utilize the variation in institutions over the diferent types of questions to mention their
constituency when possible.</p>
        <p>Importantly, in order to make sure that the efect of setting can be attributed to personal
MP preference, I include a control for MP party alignment. This variable is constructed by
calculating the cosine similarity between the text produced by an MP in plenary debates during
a parliamentary period with the text produced by all other MPs of the same party in the same
period. Thus, the measure gives a similarity score between an MP’s plenary debates and fellow
party members in parliament. The resulting score will be higher when the MP is closer to the
party average and lower when the MP is further away from the average. There are some missing
values for the similarity measure – see Table 2 – because not all MPs participate enough in
debates to calculate the cosine score.</p>
        <p>
          Next, I also include personal characteristics for each MP in the analyses. Gender and age
have been shown to drive behavioral diferences in a variety of studies on parliaments (for
example see the Bäck and Debus [
          <xref ref-type="bibr" rid="ref21">21</xref>
          ] study on the efect of gender on debate participation).
The general findings in this literature is that men speak more than women and that higher age
leads to more participation. Additionally, I include the number of words in a given text in the
regressions. As mentioned above, the longer a text is, the more room an MP or minister has for
mentioning her constituency within the text.
        </p>
        <p>As for further controls, I use fixed efects for the party of the MP and parliamentary period
in order to isolate variation over time and parties: new MPs are elected after elections, which
might drive debates in more or less constituency focused directions, and some parties have local
matters as more salient issues than others. Last, I control for whether the text was produced in an
election year or not. If electoral concerns are the main reason for signaling to the constituency,
we would expect there to be a spike in mentions whenever there is an election coming up.</p>
        <p>Table 2 shows descriptive statistics for the full dataset over the variables used in the analyses.
Note that the mean for the binary variables are the proportion of positive (1) outcomes on that
variable. For example, there are constituency mentions in 9% (0.09 ∗ 100) of the cases in the
data.</p>
      </sec>
      <sec id="sec-3-3">
        <title>3.4. Models</title>
        <p>
          With a binary variable, showing whether MPs mention their constituency or not, logistic
regression is the best fit. Because the dependent variable is skewed towards zero – there are
a lot more texts without constituency mentions than with – I use an undersample simulation
approach through the ROSE package for R in the main analysis [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ]. In short, this approach
entails balancing the data (undersampling) by randomly removing majority class units. I also do
the undersampling over 1000 iterations in order to make sure the results are stable (see Menardi
and Torelli [
          <xref ref-type="bibr" rid="ref23">23</xref>
          ] for a thorough discussion).
        </p>
        <p>Further, I run a series of additional robustness models, shown in the online appendix.3 First,
I run a model with the same model specification as the main analysis, but using the full dataset
for the estimation. Second, I estimate a model restricting the data to only texts between 50 and
500 tokens long. Third, I run a model excluding mentions of the four biggest cities in Norway –
Oslo (capital), Bergen, Trondheim and Stavanger – from the dependent variable. This is done
because all MPs mention these cities often, regardless of their constituency, which could inflate
the efect for MPs actually from the constituencies where these cities are located (since I do not
count mentions for MPs not in the constituency). The results of these robustness models and
how they difer from the main analyses is discussed more in detail in the online appendix, but
the general take is that the general findings of the main analyses also holds in these alternative
models.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Analyses</title>
      <p>In this section, I present the results for answering the question outlined earlier: Does lower
party control in various institutional settings increase MP autonomy? The first analysis shows
that MPs do indeed utilize the liberty of decreased party control in certain settings, namely in
written questions and question time, to raise constituency concerns to the government.</p>
      <sec id="sec-4-1">
        <title>4.1. Constituency Mentions</title>
        <p>For the results of the main model, shown in Figure 3. The figure shows the mean logit coeficients
(points) over all undersampled simulations with the 5% to 95% quantiles over the simulations
as confidence bands. We can immidiately observe that constituency focus is higher, when all
else is equal, in written questions and question time, than in question hour and interpellations
(reference category) for MPs (reference category). Also, notice that there is a large negative
efect for going from MPs to ministers; the expected probability for ministers mentioning their
constituency is substantially lower than for MPs in all types of questions. This is in line with
expectations. The interaction between question type and institutional role will be discussed in
detail below.</p>
        <p>As for the remaining controls, I will discuss these individually, holding all other variables
constant at 0. First, ideological distance has a relatively strong negative efect ( −0.781). This
means that the further an MP (or minister) is from the party mean in the cosine measure, the
more likely that MP is to mention her constituency in speeches. Because the cosine measure has
a range of 0.119 to 0.910 and a mean of 0.660 in the data, the efect is less strong than it might
appear at first glance. Going from the lowest cosine value to the highest, decreases the expected
3The online appendix can be found at: https://github.com/martigso/party_control_responsiveness
Question hour
Question time</p>
        <p>Interpellations
Institutional role (minister)
Interpellations * minister
Question hour * minister</p>
        <p>Question time * minister
Ideological distance (cosine)</p>
        <p>Gender (male)</p>
        <p>Age
Word count</p>
        <p>FrP</p>
        <p>H
KrF
Sp
SV</p>
        <p>V
Election year (yes)
2001−2005
2005−2009
2009−2013
2013−2017
2017−2021</p>
        <p>Constant
−1.2
−1.0
−0.8
−0.6</p>
        <p>−0.4 −0.2 0.0 0.2 0.4</p>
        <p>Coef. mean
N varies from 23176 to 23671 observations between iterations.
probability of mentioning one’s constituency from 52.1% to 36.9%4. This is still a substantial
efect and in line with the expectations for the variable in question.</p>
        <p>Next, gender has a weak positive efect. Using the same formula as above, the model estimates
a probability for women to mention their constituency 54.4% of the time, whereas men are
expected to do this in 56.9% of their speeches. Further, the efect of age is close to 0, but has a
wider range in the data (about 19 to 77 years old). Nevertheless, the efect is weak: going from
a 40 year old MP to a 60 year old MP only increases the probability of constituency mentions by
4.3 percentage points (63.4% and 67.7%).
∗ 100 = 52.1% and 1+((0(.01.7167−6−0.07.8718∗10∗.09.1901)0))</p>
        <p>The word count variable efect is positive, as expected, and moderately strong (due to the
wide range of the variable). A speech with 87 words (25 quantile) has an expected probability of
57.8% for an MP mentioning her constituency and a speech with 323 words (75 quantile) 66.5%,
which is a 8.7 percentage point increase.</p>
        <p>As for the party fixed efects, the magnitudes are relatively small. Unsurprisingly, the Center
Party (Sp) has the highest expected probability of mentioning a constituency. An MP from
the Center Party is expected to have a 11.3 percentage point higher probability than an MP
from the Christian Democrats (KrF), which is the party with lowest estimated probability for a
positive dependent variable, of using constituency signaling in their speeches (57.9% for the
Center Party and 46.6% for the Christian Democrats).</p>
        <p>Election year also has a positive, albeit very weak, efect. In a non-election year any given
speech is estimated to have a constituency mention probability of 54.4%, whereas the same
probability in an election year is 55.7% – a meager 1.3 percentage point increase. The same
goes for the period fixed efects: there is some variation, but the diferences are relatively small.
The biggest gap in expected constituency mention is between 1997-2001 (reference category) to
2009-2013 is 3.0 percentage points.</p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Question Types</title>
        <p>Looking more carefully on the efect magnitude of the main independent variables, Figure 4
highlights the expected probabilities for MPs and ministers mentioning their own constituency
in the diferent types of questions.</p>
        <p>The figure clearly shows that there are large amounts of diferences between ministers and
MPs in how often they mention their constituency over all types of questions. Interestingly, the
variation between question types for ministers is also substantially smaller than for MPs: holding
all other variables constant, a minister will be expected on average to mention her constituency
about 35% of the time in question time and written questions, 28.5% in interpellations, and 24.4%
in the question hour. The uncertainty bands for these estimates are also mostly overlapping,
except for the diference between question hour and question time together with written
questions. The point estimate diferences are, nevertheless, ordered as expected for ministers.</p>
        <p>For MPs, the diferences in expected constituency mentions between question types is larger,
although the ordering of the point estimates remains the same. Question time is the most likely
place to see a constituency mentioned among MPs, clocking in at an average of 68.1%. Also for
MPs, overlapping the confidence band with question time, written questions have an expected
probability for mentioning constituencies of 64.7%. Further, in question hours, MPs are expected
to mention their constituency far less often at a probability of 41.9%, and between this estimate
and the two others, interpellations give an average probability of 51.5%</p>
        <p>In substance, these findings are, uncertainty aside, along with expectations throughout. The
question types that are less party controlled for all MPs (also backbenchers), see the most
amount of constituency signaling. On the one hand, the gated form in question hours leads to
this being the least constituency focused question type, both for ministers and MPs. On the
other hand, the more accessible question time and written questions are the most constituency
focused types of questions. It is also unsurprising that question time and written questions are
so similar in efect size, seeing as written questions were designed to ofload the pressure on
0.7
0.6
question time. Further, the variation across question types is much higher for MPs than for
ministers. In sum, the analysis shows that when the party leadership has an opportunity or
incentive to take control over an institution, constituency focus diminished greatly.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Discussion</title>
      <p>The content of questions are an underutilized source of data in the literature on parliaments
and the behavior of the actors within these systems. Not only are parliamentary questions
an important arena in the day-to-day work of parliament, but the diferent types of questions
also ofer unique opportunities for investigating behavioral diferences caused by institutional
design.</p>
      <p>In this paper, I have shown how party control and lower-level institutional settings matter for
how much MPs focus on their own constituency in a party-centered parliamentary system. By
utilizing the variation in diferent types of parliamentary questions in the Norwegian parliament,
I have demonstrated that MPs focus on their constituency more often when the party has less
control over their agenda. Particularly, my results show that the more party controlled and
formally restricted legislative activities, such as interpellations and question hours, contain
substantially less amounts of localism than more party controlled settings, such as written
questions or question time; MPs refer to their constituency far less in question hours and
interpellations than they do in written questions and question time. Further, the limited time
frame and increased attention given by the media to oral question hours seems to attract the
party to control what their MPs talk about even more than the other types of legislative activities.
An alternative explanation for why question time and written questions see more signaling
is that these types of questions are closed of for other MPs to participate in. It is a strict
dialogue between the MP asking the question and the minister answering. Thus, the MP asking
a question has all the agenda-setting power in the dialogue, which could make it easier to steer
the debate towards that MPs constituency concerns.</p>
      <p>Nevertheless, even in a party-centered electoral system, I show that there is room for MPs
to have substantial focus on their constituency under the right circumstances. Building on
this point, this paper corroborates and supplements findings in studies on institutions within
parliamentary democracies; the diferent types of parliamentary questions do influence what
MPs want to and are allowed to talk about. Constituency signaling does not disappear when
going from a candidate-centered to a party-centered system; it is just located in diferent places.</p>
    </sec>
    <sec id="sec-6">
      <title>Acknowledgments</title>
      <p>Earlier versions of this paper has been presented at research seminars at the University of Oslo.
I thank the audiences for valuable and constructive comments. I also thank the reviewers and
participants of the Digital Humanities in the Nordic and Baltic Countries 6th Conference for
helpful feedback. Special thanks to Bjørn Høyland, Zoltán Fazekas, Jon H. Fiva, and Bjørn Erik
Rasch for useful comments and suggestions on previous iterations of the paper.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>A.</given-names>
            <surname>André</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Depauw</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. S.</given-names>
            <surname>Shugart</surname>
          </string-name>
          , Oxford Handbook of Legislative Studies, Oxford University Press,
          <year>2014</year>
          , pp.
          <fpage>231</fpage>
          -
          <lpage>249</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>H.</given-names>
            <surname>Rosenthal</surname>
          </string-name>
          , E. Voeten,
          <source>Analyzing Roll Calls with Perfect Spatial Voting: France</source>
          <year>1946</year>
          -1958,
          <source>American Journal of Political Science</source>
          <volume>48</volume>
          (
          <year>2004</year>
          )
          <fpage>620</fpage>
          -
          <lpage>632</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>S.-O.</given-names>
            <surname>Proksch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. B.</given-names>
            <surname>Slapin</surname>
          </string-name>
          ,
          <source>The Politics of Parliamanetary Debate: Parties, Rebels and Representation</source>
          , Cambridge University Press,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>S.</given-names>
            <surname>Martin</surname>
          </string-name>
          ,
          <article-title>Parliamentary Questions, the Behaviour of Legislators, and the Function of Legislatures: An Introduction</article-title>
          ,
          <source>The Journal of Legislative Studies</source>
          <volume>17</volume>
          (
          <year>2011</year>
          )
          <fpage>259</fpage>
          -
          <lpage>270</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>R.</given-names>
            <surname>Whitaker</surname>
          </string-name>
          , S. Martin,
          <article-title>Divide to conquer? strategic parliamentary opposition and coalition government</article-title>
          , Party
          <string-name>
            <surname>Politics</surname>
          </string-name>
          (
          <year>2021</year>
          ).
          <article-title>a r X i v : h t t p s : / / d o i</article-title>
          .
          <source>o r g / 1 0 . 1 1</source>
          <volume>7 7 / 1 3 5 4 0 6 8 8 2 1 1 0 4 2 8 5 9</volume>
          , early access.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>B. E.</given-names>
            <surname>Rasch</surname>
          </string-name>
          ,
          <article-title>Behavioural Consequences of Restrictions on Plenary Access: Parliamentary Questions in the Norwegian Storting</article-title>
          ,
          <source>The Journal of Legislative Studies</source>
          <volume>17</volume>
          (
          <year>2011</year>
          )
          <fpage>382</fpage>
          -
          <lpage>393</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>B. E.</given-names>
            <surname>Rasch</surname>
          </string-name>
          , Opposisjonen, in: H.
          <string-name>
            <surname>M. Narud</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          <string-name>
            <surname>Heidar</surname>
          </string-name>
          , T. Grønlie (Eds.),
          <source>Stortingets historie 1964-2014</source>
          , Fagbokforlaget,
          <year>2014</year>
          , pp.
          <fpage>439</fpage>
          -
          <lpage>454</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>C.</given-names>
            <surname>Green-Pedersen</surname>
          </string-name>
          ,
          <article-title>Bringing Parties Into Parliament: The Development of Parliamentary Activities in Western Europe</article-title>
          ,
          <source>Party Politics</source>
          <volume>16</volume>
          (
          <year>2010</year>
          )
          <fpage>947</fpage>
          -
          <lpage>369</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>K.</given-names>
            <surname>Soontjens</surname>
          </string-name>
          , The Awareness Paradox:
          <article-title>(Why) Politicians Overestimate Citizens Awareness of Parliamentary Questions</article-title>
          and
          <string-name>
            <given-names>Party</given-names>
            <surname>Initiatives</surname>
          </string-name>
          ,
          <source>Representation</source>
          <volume>57</volume>
          (
          <year>2021</year>
          )
          <fpage>75</fpage>
          -
          <lpage>94</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>O.</given-names>
            <surname>Rozenberg</surname>
          </string-name>
          , S. Martin, Questioning parliamentary questions,
          <source>The Journal of Legislative Studies</source>
          <volume>17</volume>
          (
          <year>2011</year>
          )
          <fpage>394</fpage>
          -
          <lpage>404</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>S.-O.</given-names>
            <surname>Proksch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. B.</given-names>
            <surname>Slapin</surname>
          </string-name>
          ,
          <article-title>Parliamentary questions and oversight in the european union</article-title>
          ,
          <source>European Journal of Political Research</source>
          <volume>50</volume>
          (
          <year>2010</year>
          )
          <fpage>53</fpage>
          -
          <lpage>79</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>M.</given-names>
            <surname>Kellermann</surname>
          </string-name>
          , Electoral Vulnerability, Constituency Focus, and
          <article-title>Parliamentary Questions in the House of Commons</article-title>
          ,
          <source>The British Journal of Politics and International Relations</source>
          <volume>18</volume>
          (
          <year>2016</year>
          )
          <fpage>90</fpage>
          -
          <lpage>106</lpage>
          . URL: https://doi.org/10.1111/
          <fpage>1467</fpage>
          -
          <lpage>856X</lpage>
          .
          <fpage>12075</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>D.</given-names>
            <surname>Höhmann</surname>
          </string-name>
          , U. Sieberer,
          <article-title>Parliamentary questions as a control mechanism in coalition governments</article-title>
          ,
          <source>West European Politics</source>
          <volume>43</volume>
          (
          <year>2020</year>
          )
          <fpage>225</fpage>
          -
          <lpage>249</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>S.</given-names>
            <surname>Martin</surname>
          </string-name>
          ,
          <article-title>Using Parliamentary Questions to Measure Constituency Focus: An Application to the Irish Case</article-title>
          ,
          <source>Political Studies</source>
          <volume>59</volume>
          (
          <year>2011</year>
          )
          <fpage>472</fpage>
          -
          <lpage>488</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>F.</given-names>
            <surname>Russo</surname>
          </string-name>
          ,
          <article-title>The Constituency as a Focus of Representation: Studying the Italian Case through the Analysis of Parliamentary Questions</article-title>
          ,
          <source>The Journal of Legislative Studies</source>
          <volume>17</volume>
          (
          <year>2011</year>
          )
          <fpage>290</fpage>
          -
          <lpage>301</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>G. W.</given-names>
            <surname>Cox</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. H.</given-names>
            <surname>Fiva</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. M.</given-names>
            <surname>Smith</surname>
          </string-name>
          , Parties, Legislators, and
          <source>the Origins of Proportional Representation, Comparative Political Studies</source>
          <volume>52</volume>
          (
          <year>2019</year>
          )
          <fpage>102</fpage>
          -
          <lpage>133</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>B. E.</given-names>
            <surname>Rasch</surname>
          </string-name>
          ,
          <article-title>Question Time in the Norwegian Storting, in: Parliamentary control in the Nordic countries: forms of questioning and behavioural trends</article-title>
          ,
          <source>The Finnish Political Science Association</source>
          ,
          <year>1994</year>
          , pp.
          <fpage>247</fpage>
          -
          <lpage>275</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>M.</given-names>
            <surname>Søyland</surname>
          </string-name>
          ,
          <article-title>stortingscrape: An R package for accessing data from the Norwegian parliament</article-title>
          ,
          <year>2021</year>
          . URL: https://github.com/martigso/stortingscrape.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>K.</given-names>
            <surname>Benoit</surname>
          </string-name>
          ,
          <string-name>
            <surname>A</surname>
          </string-name>
          . Matsuo, spacyr: Wrapper to the 'spaCy' 'NLP' Library,
          <year>2020</year>
          . URL: https: //CRAN.R-project.org/package=spacyr,
          <source>r package version 1.2.1.</source>
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>M.</given-names>
            <surname>Honnibal</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. Montani</surname>
          </string-name>
          , spaCy 2:
          <article-title>Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing, 2017</article-title>
          . To appear.
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>H.</given-names>
            <surname>Bäck</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Debus</surname>
          </string-name>
          , Political Parties, Parliaments and
          <string-name>
            <given-names>Legislative</given-names>
            <surname>Speechmaking</surname>
          </string-name>
          , Palgrave,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>N.</given-names>
            <surname>Lunardon</surname>
          </string-name>
          , G. Menardi,
          <string-name>
            <surname>N.</surname>
          </string-name>
          <article-title>Torelli, ROSE: a Package for Binary Imbalanced Learning</article-title>
          ,
          <source>R Journal</source>
          <volume>6</volume>
          (
          <year>2014</year>
          )
          <fpage>82</fpage>
          -
          <lpage>92</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>G.</given-names>
            <surname>Menardi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Torelli</surname>
          </string-name>
          ,
          <article-title>Training and assessing classification rules with imbalanced data, Data mining and knowledge discovery 28 (</article-title>
          <year>2014</year>
          )
          <fpage>92</fpage>
          -
          <lpage>122</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>