<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>A Database Framework for Probabilistic Preferences</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Batya Kenig</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Benny Kimelfeld</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Haoyue Ping</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Julia Stoyanovich</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Drexel University</institution>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Technion</institution>
          ,
          <country country="IL">Israel</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>? This work was supported in part by ISF Grant No. 1295/15, BSF Grant No. 2014391 and by the Taub Foundation. ?? This work was supported in part by NSF Grants No. 1464327 and 1539856, and BSF Grant No. 2014391.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>
        Preferences are statements about the relative quality or desirability of items.
Ever larger amounts of preference information are being collected and analyzed
in a variety of domains, including recommendation systems [
        <xref ref-type="bibr" rid="ref16 ref18 ref2">2, 16, 18</xref>
        ], polling
and election analysis [
        <xref ref-type="bibr" rid="ref15 ref3 ref6 ref7">3, 6, 7, 15</xref>
        ], and bioinformatics [
        <xref ref-type="bibr" rid="ref1 ref11 ref19">1, 11, 19</xref>
        ].
      </p>
      <p>
        Preferences are often inferred from indirect input (e.g., a ranked list may be
inferred from individual choices), and are therefore uncertain in nature. This
motivates a rich body of work on uncertain preference models in the statistics
literature [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]. More recently, the machine learning community has been
developing methods for e ective modeling and e cient inference over preferences, with
the Mallows model [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] receiving particular attention [
        <xref ref-type="bibr" rid="ref12 ref17 ref4 ref5">4, 5, 12, 17</xref>
        ].
      </p>
      <p>
        In this paper, we take the position that preference modeling and analysis
should be accommodated within a general-purpose probabilistic database
framework. Our framework is based on a deterministic concept that we proposed in
a past vision paper [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. In the present work we focus on handing uncertain
preferences, and develop a representation of preferences within a probabilistic
preference database, or PPD for short.
      </p>
      <p>This paper is an abbreviated version of our PODS 2017 paper, where an
interested reader can nd additional details about the formalism and proposed
algorithmic solutions.
A preference schema S is a relational schema with some relation symbols marked
as preference symbols (and others as ordinary symbols ). Figure 1 gives an
example of a preference database instance, with the ordinary symbols Candidates
and Voters, and the preference symbol Polls.</p>
      <p>An instance over a preference symbol (such as Polls in Figure 1) represents
a collection of preferences among a set of items, where each such preference is</p>
      <p>Candidates (o)
cand party sex edu
Trump R M BS
Clinton D F JD
Sanders D M BS
Rubio R M JD</p>
      <p>Voters (o)
voter edu sex age
Ann BS F 25
Bob BS M 35
Cat MS F 40</p>
      <p>Dave MS M 45
A MAL-instance over Polls (p)
voter date Preference model MAL( ; )
Ann Oct-5 hClinton; Sanders; Rubio; Trumpi; 0:3
Bob Oct-5 hTrump; Rubio; Sanders; Clintoni; 0:3
Polls (p)
voter date lcand rcand
Ann Oct-5 Sanders Clinton
Ann Oct-5 Sanders Rubio
Ann Oct-5 Sanders Trump
Ann Oct-5 Clinton Rubio
Ann Oct-5 Clinton Trump
Ann Oct-5 Rubio Trump
Bob Oct-5 Sanders Rubio
Bob Oct-5 Sanders Clinton
Bob Oct-5 Sanders Trump
Bob Oct-5 Rubio Clinton
Bob Oct-5 Rubio Trump</p>
      <p>Bob Oct-5 Clinton Trump
itself a binary relation called a session. A a binary relation over a set I =
f 1; : : : ; ng of items is a (strict) partial order if it is irre exive and transitive. A
linear (or total ) order is a partial order where every two items are comparable.
By a slight abuse of notation, we often identify a linear order 1 n with
the sequence h 1; : : : ; ni, and we call it a ranking.</p>
      <p>Example 1. Our running example is on individual preferences among the set of
US presidential candidates I = fClinton; Rubio; Sanders; Trumpg. The ranking
= hClinton; Rubio; Sanders; Trumpi is an example ranking over I. tu</p>
      <p>A preference relation instantiates a special relation symbol with a signature of
the form ( ; Al; Ar), where is the session signature, and Al and Ar are the
lefthand-side (lhs) attribute and right-hand-side (rhs) attribute, respectively. We
use semicolon (;) to distinguish between the di erent parts and write ( ; Al; Ar).
Example 2. We use the preference signature (voter; date; lcand; rcand) in our
running example. Here the components , Al and Ar are (voter; date), lcand, and
rcand, respectively. The table Polls of Figure 1 is an instance of this preference
signature that contains two sessions. The session (Ann; Oct-5) is associated with
the ranking hSanders; Clinton; Rubio; Trumpi. The tuple (Ann; Oct-5; Sanders;
Clinton) denotes that in the session of the voter Ann on October 5th, the
candidate Sanders is preferred to the candidate Clinton.
tu</p>
      <p>We now make the knowledge about voters' opinions probabilistic, interpreting
the preference database of Figure 1 as one possible world of a probabilistic
preference database. A probabilistic preference database (abbrv. PPD) over the
preference schema S is a probability space over preference databases over S.
A PPD can be represented by explicitly specifying the entire sample space;
however, we wish to allow for standard compact representations of preferences.</p>
      <p>
        A probabilistic preference model is a ( nite and typically compact)
representation M of a probability space over partial orders over a nite set of items;
we denote this probability space by JM K. A model family is a collection M of
probabilistic preference models. As prominent examples, we de ne two model
families: RIM is the family of RIM [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] models RIM( ; ), and MAL is the
family of Mallows [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] models MAL( ; ).
      </p>
      <p>
        A Mallows model [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] MAL( ; ) is parameterized by a reference ranking
= h 1; : : : ; mi and a dispersion parameter 2 (0; 1]. The model assigns a
non-zero probability to every ranking : The higher the Kendal's tau distance [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]
of is from , the lower its probability under the model. Lower values of
concentrate most of the probability mass around , while = 1 corresponds to
the uniform probability distribution over the rankings. Doignon [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] showed that
MAL( ; ) can be represented as the insertion model RIM( ; ).
      </p>
      <p>In the PPD representations we explore, termed RIM-PPD, each session is
associated with the parameters of a RIM model. A RIM-PPD represents a
probability space over preference databases, where a possible world is obtained by
independently sampling a preference from the model of each session. Figure 1
gives an example of a MAL-instance over the p-symbol Polls that associates
each session in Polls with a Mallows model. It is straight-forward to extend this
representation to a mixture of Mallows, by associating each session with k
components MAL1( 1; 1); : : : ; MALk( k; k), with the corresponding probabilities
p1; : : : ; pk. A possible world would then be obtained by rst sampling
component MALi( i; i) with probability pi independently for each session, and then
sampling a preference from MALi( i; i).
3</p>
    </sec>
    <sec id="sec-2">
      <title>Query Evaluation over PPDs</title>
      <p>
        We adopt the semantics of probabilistic databases [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ] for query evaluation.
Speci cally, let S be a schema, let Q be a query, and let D = ( ; ) be a PPD.
A possible answer for Q is a tuple a over sig (Q) such that a 2 Q(D) for some
sample D of D. We denote by PosAns(Q; D) the set of all possible answers. The
con dence of a possible answer a 2 PosAns(Q; D), denoted confQ(D; a), is the
probability of having a as an answer when querying a sample of D. If E is an
MPPD for some model class M, then evaluating Q on E is the task of computing
the following ( nite) set: Q(E) = f(a; confQ(JEK; a)) j a 2 PosAns(JEK)g.
      </p>
      <p>We study the data complexity of evaluating Conjunctive Queries (CQs) over
RIM-PPDs. We focus on CQs to which we refer as itemwise. Intuitively, these
are CQs where items are connected only through preferences. We show a natural
fragment of CQs where the itemwise CQs are precisely the CQs in which query
evaluation can be done in polynomial time. In the fragment we consider, we
prove that every query that is not itemwise is actually #P-hard, and therefore,
we establish a dichotomy in complexity.</p>
      <p>
        Let S be a preference schema, and let Q be a CQ over S. An atomic formula
of Q is called a p-atom if it is over a p-symbol, and an o-atom if it is over an
o-symbol. Let P (s1; : : : ; sk; tl; tr) be p-atom of Q. Each term si for i = 1; : : : ; k
is said to occur in a session position, and each of tl and tr is said to occur in
an item position. A session variable of Q is a variable that occurs in a session
position, and an item variable of Q is a variable that occurs in an item position.
We say that Q is sessionwise if all p-atoms of Q refer to the same session; that
is, if P (s1; : : : ; sk; tl; tr) and P 0(s01; : : : ; s0l; t0l; t0r) are p-atoms of Q, then P = P 0
and (s1; : : : ; sk) = (s01; : : : ; s0l). We say that Q is itemwise if Q is sessionwise,
and the joins between item variables occur only inside the p-atoms, or through
session variables. Put di erently, in an itemwise CQ with a constant session,
the o-atoms state properties of individual items but do not draw connections
between the items. In [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] we de ne this property more formally, by means of
the Gaifman graph of the CQ.
      </p>
      <p>Example 3. Consider the following Boolean CQs. The query Q1 asks whether
there is a voter with a BS degree who prefers a male Democratic candidate to a
female Democratic candidate.</p>
      <p>Q1()</p>
      <p>P (v; ;l; r); V (v; BS; ; ); C(l; D; M; ); C(r; D; F; )
The query Q2 asks whether there is a voter who prefers a male candidate to a
female candidate such that both candidates are of the same political party.</p>
      <p>Q2()</p>
      <p>P ( ; ;l; r); C(l; p; M; ); C(r; p; F; )
The query Q3 asks whether there is a voter who prefers a female candidate to
both Trump and Sanders.</p>
      <p>Q3()</p>
      <p>P (v; d; l; Trump); P (v; d; l; Sanders); C(l; ;F; )
All of these CQs are sessionwise. Indeed, Q1 and Q2 involve a single p-atom
(hence, they are sessionwise by de nition), and in Q3 both atoms have (v; d) in
their session parts. CQs Q1 and Q3 are itemwise, while Q2 is not itemwise.
tu</p>
      <p>
        In [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] we prove the following theorem, which states that every itemwise
Boolean CQ can be evaluated in polynomial time, under data complexity.
Theorem 1. Let S be a preference schema, and let Q be a Boolean CQ over S.
If Q is itemwise, then Q can be evaluated in polynomial time on RIM-PPDs.
      </p>
      <p>We also prove that the class itemwise CQs are precisely the tractable ones
(among the queries in the class), under conventional complexity assumptions. In
other words, every Boolean CQ (in the class) that is not itemwise is necessarily
hard to evaluate, and therefore, we establish a dichotomy.</p>
      <p>Theorem 2. Let S be a preference schema, and let Q be a Boolean CQ over S
such that Q has no self joins and Q has a single p-atom. If Q is not itemwise,
then the evaluation of Q on RIM-PPDs over S is FP#P-hard.</p>
      <p>
        In [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] we give is a polynomial-time algorithm for evaluating itemwise CQs.
Interestingly, such CQs translate into a natural (and novel) inference problem
over RIM. In this problem, every item is associated with one or more labels
(e.g., \democratic" party or \comedy" genre), and the goal is to compute the
probability that a graph pattern (or equivalently a partial order) over these
labels matches the random ranking.
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <given-names>S.</given-names>
            <surname>Aerts</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Lambrechts</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Maity</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. V.</given-names>
            <surname>Loo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Coessens</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F. D.</given-names>
            <surname>Smet</surname>
          </string-name>
          , L.-
          <string-name>
            <surname>C. Tranchevent</surname>
            ,
            <given-names>B. D.</given-names>
          </string-name>
          <string-name>
            <surname>Moor</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Marynen</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          <string-name>
            <surname>Hassan</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Carmeliet</surname>
            , and
            <given-names>Y.</given-names>
          </string-name>
          <string-name>
            <surname>Moreau</surname>
          </string-name>
          .
          <article-title>Gene prioritization through genomic data fusion</article-title>
          .
          <source>Nature Biotechnology</source>
          ,
          <volume>24</volume>
          (
          <issue>5</issue>
          ):
          <volume>537</volume>
          {
          <fpage>544</fpage>
          ,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <given-names>S.</given-names>
            <surname>Balakrishnan</surname>
          </string-name>
          and
          <string-name>
            <given-names>S.</given-names>
            <surname>Chopra</surname>
          </string-name>
          .
          <article-title>Two of a kind or the ratings game? adaptive pairwise preferences and latent factor models</article-title>
          .
          <source>Frontiers of Computer Science</source>
          ,
          <volume>6</volume>
          (
          <issue>2</issue>
          ):
          <volume>197</volume>
          {
          <fpage>208</fpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <given-names>P.</given-names>
            <surname>Diaconis</surname>
          </string-name>
          .
          <article-title>A generalization of spectral analysis with applications to ranked data</article-title>
          .
          <source>Annals of Statistics</source>
          ,
          <volume>17</volume>
          (
          <issue>3</issue>
          ):
          <volume>949</volume>
          {
          <fpage>979</fpage>
          ,
          <year>1989</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <given-names>W.</given-names>
            <surname>Ding</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Ishwar</surname>
          </string-name>
          , and
          <string-name>
            <given-names>V.</given-names>
            <surname>Saligrama</surname>
          </string-name>
          .
          <article-title>Learning mixed membership mallows models from pairwise comparisons</article-title>
          .
          <source>CoRR, abs/1504.00757</source>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <given-names>J.-P.</given-names>
            <surname>Doignon</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Pekec</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M.</given-names>
            <surname>Regenwetter</surname>
          </string-name>
          .
          <article-title>The repeated insertion model for rankings: Missing link between two subset choice models</article-title>
          .
          <source>Psychometrika</source>
          ,
          <volume>69</volume>
          (
          <issue>1</issue>
          ):
          <volume>33</volume>
          {
          <fpage>54</fpage>
          ,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <given-names>I. C.</given-names>
            <surname>Gormley</surname>
          </string-name>
          and
          <string-name>
            <given-names>T. B.</given-names>
            <surname>Murphy</surname>
          </string-name>
          .
          <article-title>A latent space model for rank data</article-title>
          .
          <source>In ICML</source>
          ,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <given-names>I. C.</given-names>
            <surname>Gormley</surname>
          </string-name>
          and
          <string-name>
            <given-names>T. B.</given-names>
            <surname>Murphy</surname>
          </string-name>
          .
          <article-title>A mixture of experts model for rank data with applications in election studies</article-title>
          .
          <source>The Annals of Applied Statistics</source>
          ,
          <volume>2</volume>
          (
          <issue>4</issue>
          ):
          <volume>1452</volume>
          {
          <fpage>1477</fpage>
          , 12
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <given-names>M.</given-names>
            <surname>Jacob</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Kimelfeld</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Stoyanovich</surname>
          </string-name>
          .
          <article-title>A system for management and analysis of preference data</article-title>
          .
          <source>PVLDB</source>
          ,
          <volume>7</volume>
          (
          <issue>12</issue>
          ):
          <volume>1255</volume>
          {
          <fpage>1258</fpage>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <given-names>M. G.</given-names>
            <surname>Kendall</surname>
          </string-name>
          .
          <article-title>A new measure of rank correlation</article-title>
          .
          <source>Biometrika</source>
          ,
          <volume>30</volume>
          (
          <issue>1</issue>
          /2):
          <volume>81</volume>
          {
          <fpage>93</fpage>
          ,
          <year>1938</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <given-names>B.</given-names>
            <surname>Kenig</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Kimelfeld</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Ping</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Stoyanovich</surname>
          </string-name>
          .
          <article-title>Querying probabilistic preferences in databases</article-title>
          .
          <source>In PODS</source>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <given-names>R.</given-names>
            <surname>Kolde</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Laur</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Adler</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Vilo</surname>
          </string-name>
          .
          <article-title>Robust rank aggregation for gene list integration and meta-analysis</article-title>
          .
          <source>Bioinformatics</source>
          ,
          <volume>28</volume>
          (
          <issue>4</issue>
          ):
          <volume>573</volume>
          {
          <fpage>580</fpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <given-names>T.</given-names>
            <surname>Lu</surname>
          </string-name>
          and
          <string-name>
            <given-names>C.</given-names>
            <surname>Boutilier</surname>
          </string-name>
          .
          <article-title>E ective sampling and learning for mallows models with pairwise-preference data</article-title>
          .
          <source>J. Mach. Learn. Res.</source>
          ,
          <volume>15</volume>
          (
          <issue>1</issue>
          ):
          <volume>3783</volume>
          {
          <fpage>3829</fpage>
          ,
          <string-name>
            <surname>Jan</surname>
          </string-name>
          .
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <given-names>C. L.</given-names>
            <surname>Mallows</surname>
          </string-name>
          .
          <article-title>Non-null ranking models</article-title>
          .
          <source>i. Biometrika</source>
          ,
          <volume>44</volume>
          (
          <issue>1-2</issue>
          ):
          <volume>114</volume>
          {
          <fpage>130</fpage>
          ,
          <year>June 1957</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <given-names>J. I.</given-names>
            <surname>Marden</surname>
          </string-name>
          .
          <article-title>Analyzing and Modeling Rank Data</article-title>
          .
          <source>Chapman &amp; Hall</source>
          ,
          <year>1995</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <given-names>G.</given-names>
            <surname>McElroy</surname>
          </string-name>
          and
          <string-name>
            <given-names>M.</given-names>
            <surname>Marsh</surname>
          </string-name>
          .
          <article-title>Candidate gender and voter choice: Analysis from a multimember preferential voting system</article-title>
          .
          <source>Political Research Quarterly</source>
          ,
          <volume>63</volume>
          (
          <issue>4</issue>
          ):pp.
          <volume>822</volume>
          {
          <issue>833</issue>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>A. D. Sarma</surname>
            ,
            <given-names>A. D.</given-names>
          </string-name>
          <string-name>
            <surname>Sarma</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Gollapudi</surname>
            , and
            <given-names>R.</given-names>
          </string-name>
          <string-name>
            <surname>Panigrahy</surname>
          </string-name>
          .
          <article-title>Ranking mechanisms in twitter-like forums</article-title>
          .
          <source>In WSDM</source>
          , pages
          <volume>21</volume>
          {
          <fpage>30</fpage>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>J. Stoyanovich</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Ilijasic</surname>
            , and
            <given-names>H.</given-names>
          </string-name>
          <string-name>
            <surname>Ping</surname>
          </string-name>
          .
          <article-title>Workload-driven learning of mallows mixtures with pairwise preference data</article-title>
          .
          <source>In WebDB, page 8</source>
          ,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>J. Stoyanovich</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Jacob</surname>
            , and
            <given-names>X.</given-names>
          </string-name>
          <string-name>
            <surname>Gong</surname>
          </string-name>
          .
          <article-title>Analyzing crowd rankings</article-title>
          .
          <source>In WebDB</source>
          , pages
          <volume>41</volume>
          {
          <fpage>47</fpage>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>J. M. Stuart</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          <string-name>
            <surname>Segal</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Koller</surname>
            , and
            <given-names>S. K.</given-names>
          </string-name>
          <string-name>
            <surname>Kim</surname>
          </string-name>
          .
          <article-title>A gene-coexpression network for global discovery of conserved genetic modules</article-title>
          .
          <source>Science</source>
          ,
          <volume>302</volume>
          :
          <fpage>249</fpage>
          {
          <fpage>255</fpage>
          ,
          <year>2003</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <given-names>D.</given-names>
            <surname>Suciu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Olteanu</surname>
          </string-name>
          , C. Re, and
          <string-name>
            <given-names>C.</given-names>
            <surname>Koch</surname>
          </string-name>
          . Probabilistic Databases.
          <source>Synthesis Lectures on Data Management</source>
          . Morgan &amp; Claypool Publishers,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>