<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>A Case Study in Fitting Area-Proportional Euler Diagrams with Ellipses using eulerr</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Johan Larsson</string-name>
          <email>johanlarsson@outlook.com</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Peter Gustafsson</string-name>
          <email>peter.gustafsson@stat.lu.se</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Department of Statistics, School of Economics and Management, Lund University</institution>
          ,
          <addr-line>Lund</addr-line>
          ,
          <country country="SE">Sweden</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2018</year>
      </pub-date>
      <fpage>84</fpage>
      <lpage>91</lpage>
      <abstract>
        <p>Euler diagrams are common and user-friendly visualizations for set relationships. Most Euler diagrams use circles, but circles do not always yield accurate diagrams. A promising alternative is ellipses, which, in theory, enable accurate diagrams for a wider range of input. Elliptical diagrams, however, have not yet been implemented for more than three sets or three-set diagrams where there are disjoint or subset relationships. The aim of this paper is to present eulerr: a software package for elliptical Euler diagrams for, in theory, any number of sets. It ts Euler diagrams using numerical optimization and exact-area algorithms through a two-step procedure, rst generating an initial layout using pairwise relationships and then nalizing this layout using all set relationships.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Background</title>
      <p>
        The Euler diagram, rst described by Leonard Euler in 1802 [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ], is a generalization
of the popular Venn diagram. Venn and Euler diagrams both visualize set
relationships by mapping areas in the diagram to relationships in the data. They
di er, however, in that Venn diagrams require all intersections to be present|
even if they are empty|whilst Euler diagrams do not, which means that Euler
diagrams lend themselves well to be area-proportional.
      </p>
      <p>
        Euler diagrams may be fashioned out of any closed shape, and have been
implemented for triangles [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], rectangles [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], ellipses [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], smooth curves [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ],
polygons [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], and circles [
        <xref ref-type="bibr" rid="ref2 ref5">5, 2</xref>
        ]. The latter are most common, and for good reason,
being that they are easiest to interpret [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. Circles, however, sometimes cannot
be used to produce accurate area-proportional diagrams.
      </p>
      <p>
        With four or more sets that all intersect, for instance, exact Euler diagrams
are impossible with circles, given that we require 24 1 = 15 intersections but
with four circles can yield no more than 13 [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. A solution to this problem is
o ered with ellipses, which may intersect in up to four, rather than two, points,
consequently yielding the necessary 15 unique areas. Elliptical Euler diagrams
were rst introduced with eulerAPE [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], which, however, only supports three
sets and prohibits empty intersections.
      </p>
      <p>
        Fitting elliptical or circular Euler diagrams must be done numerically even in
the two-set case [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] where the separation required by the circles has no closed-form
solution. Many algorithms accomplish this in two steps, rst nding a coarse
starting layout that is nalized in a second, more thorough, algorithm. For the
initial layout, the aforementioned eulerAPE package [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ], for instance, uses an
algorithm that tries to minimize the error in the three-way intersection by
arranging circles representing the sets. The venneuler package [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ], meanwhile, uses
multi-dimensional scaling (MDS). The javascript package venn.js [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] combines
a constrained version of the MDS algorithm from venneuler with a greedy
algorithm.
      </p>
      <p>
        In the nal layout, we need to compute the areas of the overlaps in the
diagram. Frederickson [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] (venn.js) and Micallef and Rodgers [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] (eulerAPE) have
developed exact-area algorithms for circles and ellipses respectively|although the
latter, as we previously covered, restricts itself to three intersecting ellipses. The
parameters of the circles or ellipses are then optimized numerically to minimize
a loss measure, which vary depending on implementation.
      </p>
      <p>
        The R-package eulerr was created as part of a bachelor's thesis [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] and
is the rst package to support Euler diagrams for, in theory, any number of
ellipses, regardless of subset and disjoint intersections. In this paper, we aim to
demonstrate the package through a series of well-known examples from previous
literature on the subject as well as a simulation study for the three-set case.
2
      </p>
    </sec>
    <sec id="sec-2">
      <title>Method</title>
      <p>eulerr allows input in the form of disjoint subsets, unions and identities, a matrix
of binary or boolean indices, a list of sample spaces, or a two- or three-way table.
The Euler diagram is t in two steps: rst, an initial layout is formed with circles
using only the sets' pairwise relationships. Second, this layout is ne-tuned taking
all intersections into consideration.
2.1</p>
      <sec id="sec-2-1">
        <title>Initial layout</title>
        <p>
          For our initial layout, we adopt a constrained version of multi-dimensional
scaling (MDS) that is used in venn.js [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ], which in turn is a modi cation of an
algorithm from venneuler [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ].
        </p>
        <p>We begin by placing circles representing each set uniformly at random in a
square space with area Pin=1 ri2 , where ri is the radius of the ith circle. The
circles are initialized so that their areas are proportional to the size of their
respective sets. The algorithm then moves the circles so that the separation
between each pair of circles matches their respective sets' intersection. If the two
sets are disjoint, however, the algorithm is indi erent to the relative locations of
those circles as long as they do not intersect. The equivalent applies to subset
sets: as long as the circle representing the smaller set remains within the larger
circle, their locations are free to vary. In all other cases, the loss function (1) is
the residual sums of squares of the separation of circles, d, required to obtain
accurate pairwise overlaps and the actual distance in the layout.
if disjoint
if subset
otherwise,
(1)
L(h; k) =</p>
        <p>X
where h and k are the x and y coordinates of the centers of the circles respectively.
The analytic gradient follows naturally after some algebra.</p>
        <p>
          We optimize (1) using the nonlinear optimizer nlm() from the R core
package stats [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ], which uses a set of quasi-Newton algorithms for unconstrained
minimization [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ].
The initial layout (section 2.1) will sometimes turn up perfect diagrams1, but only
reliably so when the diagram is accurately determined by its pairwise intersections.
In the nal layout, we cover the remaining cases by taking each intersection into
account. We now also extend ourselves to ellipses.
        </p>
        <p>
          To nd the overlap areas, we rst locate all the points of intersection of the
ellipses [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ], after which we can establish the overlap areas of the ellipses. We
are interested only in the points that are contained within all of these ellipses,
which together form a shape consisting of a convex polygon, the sides of which
are made up of straight lines between consecutive points, and a set of elliptical
arcs|one for each pair of points (Fig. 1).
        </p>
        <p>
          It is trivial to nd the area of the polygon section since it is always convex [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ].
And because each elliptical segment is formed from the arcs that connect successive
points, it is also straightforward to establish the segments' areas [
          <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
          ].
        </p>
        <p>
          Having computed the areas of all the intersections we now decompose them into
disjoint areas, wherein each area uniquely represents a subset of the input. This is
the form we need for our nal optimization. We feed the initial layout computed
in section 2.1 to the optimizer|once again we employ nlm() from stats but now
1 By perfect, we refer to solutions with diagError &lt; 10 6 (see equation (3)).
also provide the option to use ellipses rather than circles, allowing the \circles"
to rotate and the relation between the semiaxes to vary, altogether rendering
ve parameters to optimize per set and ellipse (or three if we restrict ourselves
to circles). For each iteration of the optimizer, the areas of all intersections are
analyzed and a measure of loss returned. The loss we use is the same as that in
venneuler [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ], namely stress, de ned as
stress =
        </p>
        <p>Pin=1(Ai</p>
        <p>!i)2
Pin=1 Ai2
with
=</p>
        <p>Pin=1 Ai!i
Pin=1 !i2 ;
where Ai is the area representing !i, the size of the ith disjoint intersection, and
n the number of intersections in the set con guration.</p>
        <p>
          As an additional option, the user may activate a last optimization step2 that
uses a Generalized Simulated Annealing optimizer [
          <xref ref-type="bibr" rid="ref17">17</xref>
          ].
        </p>
        <p>
          To measure the goodness of t of the resulting diagram, we adopt two widely
used measures: the previously covered stress [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ] (2) and diagError [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ],
diagError =
        </p>
        <p>max
i=1;2;:::;n Pin=1 !i
!i</p>
        <p>Ai
Pin=1 Ai
:
(2)
(3)
The complete algorithm is provided in Algorithm 1.</p>
      </sec>
      <sec id="sec-2-2">
        <title>2.3 Implementation</title>
        <p>
          eulerr is primarily written in R but its backbone is implemented in C++ and make
heavy use of the linear algebra library Armadillo [
          <xref ref-type="bibr" rid="ref18">18</xref>
          ] through Rcpp [
          <xref ref-type="bibr" rid="ref19">19</xref>
          ] and
RcppArmadillo [
          <xref ref-type="bibr" rid="ref20">20</xref>
          ]. The package is compatible with all major operating systems
(Linux, OS X, and Windows) and is featured on the Comprehensive R Archive
Network (CRAN) [
          <xref ref-type="bibr" rid="ref21">21</xref>
          ]. It is installed by calling install.packages("eulerr")
within R. The source code and development version are hosted at https://
github.com/jolars/eulerr. In addition, we have developed a shiny [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ] web
application for eulerr, available at http://eulerr.co.
3
        </p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>Results</title>
      <p>In this section, we will study set con gurations|and the diagrams t to them|
from previous papers featuring software for Euler diagrams. The packages we
will study are eulerr 4.1.0, eulerAPE 3.0.0, and venneuler 1.1-0.</p>
      <p>
        We begin with a set relationship from Wilkinson [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ],
wilkinson &lt;- c("A" = 4, "B" = 6, "C" = 3, "D" = 2, "E" = 7, "F" = 3,
"A&amp;B" = 2, "A&amp;F" = 2, "B&amp;C" = 2, "B&amp;D" = 1,
"B&amp;F" = 2, "C&amp;D" = 1, "D&amp;E" = 1, "E&amp;F" = 1,
"A&amp;B&amp;F" = 1, "B&amp;C&amp;D" = 1)
2 By default, this last-ditch optimizer kicks in for three-set combinations where the
diagError of the solution surpasses 0.001.
Algorithm 1. The eulerr algorithm for elliptical diagrams.
      </p>
      <p>Data: N sets, n intersections, !: the required disjoint areas for an optimal
diagram, Fi: the size of the ith set, : a prede ned tolerance threshold.
Result: N ellipses with parameters = fh; k; a; b; g.
for i 1 to N do</p>
      <p>Ai !i
ri pAi=
xi; yi</p>
      <p>U 0; qPiN=1 ri2
foreach i &lt; j N do
nd dij that minimizes Oij (Fi \ Fj) 2, where Oij is the overlap of the
circles representing Fi and Fj
for i 1 to 10 do obtain h(i), k(i) by minimizing (1) using a local optimizer
j i 2 f1; 10g that minimizes (1)
obtain nal by minimizing (2) using a local optimizer with h = h(j), k = k(j),
a = 0, b = 0, = 0 as starting values
if diagError( nal) &gt; then
obtain last-ditch using a global optimizer
if diagError( nal) &gt; diagError( last-ditch) then return last-ditch
else return nal
else
return</p>
      <p>nal
speci ed as disjoint subsets using the &amp;-operator such that "A&amp;B", for instance,
are the items unique to the intersection between A and B. We t this speci cation
with venneuler and eulerr, in the latter case with both circles and ellipses,
using euler(): the workhorse of eulerr.
f1 &lt;- venneuler::venneuler(wilkinson)
f2 &lt;- euler(wilkinson)
f3 &lt;- euler(wilkinson, shape = "ellipse")
# fit with venneuler
# fit with eulerr (circles)
# fit with eulerr (ellipses)
eulerr manages to t this con guration perfectly using ellipses in addition to
producing a marginally better circular diagram. The stress values are 0.007,
0.004, and 4:687 10 12 for venneuler, eulerr (with circles), and eulerr (with
ellipses) respectively.</p>
      <p>Diagrams in eulerr are plotted using plot(), which allows considerable
customization of the resulting diagram. In the following code, we plot the circular
diagram using the default options and the elliptical one with a few modi
cations (Fig. 2).
plot(f2)
plot(f3,
fills = rainbow(6, s = 0.5, v = 1), # change fills
edges = list(lex = 3, col = "white"), # white, broader edges
labels = list(font = 3)) # italic labels</p>
      <p>B
F</p>
      <p>D
A</p>
      <p>
        C
A
B
C
Micallef and Rodgers feature a diagram from Lenz et al. [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ] that they remodeled
using eulerAPE [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. We will do the same here, using eulerr, and compare the
results of the two packages. The data from the study|as disjoint subsets|is
lenz &lt;- c("A" = 0.36, "B" = 0.03, "C" = 0,
      </p>
      <p>
        "A&amp;B" = 0.41, "A&amp;C" = 0.04, "B&amp;C" = 0, "A&amp;B&amp;C" = 0.11)
Because eulerAPE cannot t set con gurations with empty intersections, the
authors used 0.00001 as a proxy for ;. Using eulerr, however, we can t the
diagram using the original data.
plot(euler(lenz, shape = "ellipse"), legend = TRUE) # add a legend
The ts from both packages are exact (Fig. 3). Although we instructed eulerr
to allow ellipses in the t, the algorithm stuck to circles, which, given that the
t is exact, is the appropriate choice since circles are easier to interpret [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ].
eulerAPE, in contrast, did not. It tries to keep the three shapes intersecting,
albeit marginally, which cannot be done with circles if the layout is to be exact.
      </p>
      <p>E</p>
      <p>B</p>
      <p>Finally, we provide a benchmark of the accuracy of venneuler, eulerAPE,
and eulerr in reproducing 1,000 random three-set combinations sampled from
U (10 6; 1) (Fig. 4). We use three-set combinations with all intersections present
in order to enable all tested packages to t the diagrams.</p>
      <p>In terms of both stress and diagError, the elliptical diagrams from eulerr
and eulerAPE perform the best, with the former coming out marginally ahead
with median stress and diagError at 3:123 10 13 and 1:638 10 7 respectively,
whilst the equivalent gures for eulerAPE are 7:834 10 12 and 8:219 10 7.</p>
      <p>For the circular diagrams, eulerr achieves the lowest median stress at 0.022,
followed by venneuler and eulerAPE at 0.035 and 0.08 respectively. In terms
of diagError, venneuler performs best followed by eulerr and eulerAPE with
respective median diagErrors of 0.048, 0.055, and 0.067.</p>
      <p>stress
0.0
0.2
0.6</p>
      <p>0.8
0.4
diagError
eulerAPE (ellipses)
eulerAPE (circles)</p>
      <p>venneuler
eulerr (ellipses)
eulerr (circles)
0.0
0.2
0.4
0.6
0.8
In this paper, we have presented an R-based software package, eulerr, for
generating elliptical Euler diagrams for any number of sets. We have examined
its performance for set relationships from previous publications of software for
Euler diagrams and shown that eulerr performs adequately for our examples
and for random three-set combinations. In general, we have also shown that
elliptical Euler diagrams have the potential to outperform circular diagrams. The
reason for this is simple: elliptical Euler diagrams feature two additional degrees
of freedom for each shape in the diagram, provided by stretch and rotation.</p>
      <p>
        eulerr is the rst software to feature area-proportional elliptical Euler
diagrams for more than three sets. The only other software for elliptical Euler
diagrams, eulerAPE, is restricted to three sets. This limitation is discussed by
the authors of the package, who argue that Euler diagrams with more than three
sets often lack well-formed solutions and that their complexity make
implementations di cult [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. Whilst it is true that inputs with more than three sets do
not always reduce to adequate Euler diagrams, it is our stance that those that
do warrant a solution to nd them.
      </p>
      <p>The results of this paper are limited to a few cases and it is not known
whether they generalize to other set combinations. This is a topic for future
research in the eld, which should examine di erent software for Euler diagrams
in large-scale simulation studies.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Euler</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>Letters of Euler to a German princess, on di erent subjects in physics and philosophy</article-title>
          .
          <source>Murray and Highley</source>
          (
          <year>1802</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Swinton</surname>
          </string-name>
          , J.:
          <article-title>Vennerable: Venn and Euler area-proportional diagrams</article-title>
          .
          <source>(2011) R package version 3.1.0</source>
          .9000.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Micallef</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rodgers</surname>
            ,
            <given-names>P.:</given-names>
          </string-name>
          <article-title>eulerAPE: drawing area-proportional 3-Venn diagrams using ellipses</article-title>
          .
          <source>PLOS ONE 9</source>
          (
          <issue>7</issue>
          )
          <issue>(</issue>
          <year>July 2014</year>
          ) e101717
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Micallef</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rodgers</surname>
            ,
            <given-names>P.:</given-names>
          </string-name>
          <article-title>eulerForce: force-directed layout for Euler diagrams</article-title>
          .
          <source>Journal of Visual Languages and Computing</source>
          <volume>25</volume>
          (
          <issue>6</issue>
          ) (
          <year>December 2014</year>
          )
          <volume>924</volume>
          {
          <fpage>934</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Wilkinson</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>Exact and approximate area-proportional circular Venn and Euler diagrams</article-title>
          .
          <source>IEEE Transactions on Visualization and Computer Graphics</source>
          <volume>18</volume>
          (
          <issue>2</issue>
          ) (
          <year>February 2012</year>
          )
          <volume>321</volume>
          {
          <fpage>331</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Blake</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>The impact of graphical choices on the perception of Euler diagrams</article-title>
          .
          <source>Ph.D. dissertation</source>
          , Brighton University, Brighton,
          <string-name>
            <surname>UK</surname>
          </string-name>
          (
          <year>February 2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Chow</surname>
            ,
            <given-names>S.C.</given-names>
          </string-name>
          :
          <article-title>Generating and drawing area-proportional Euler and Venn diagrams</article-title>
          .
          <source>Ph.D. dissertation</source>
          , University of Victoria, Victoria,
          <string-name>
            <surname>BC</surname>
          </string-name>
          , Canada (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Micallef</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>Visualizing set relations and cardinalities using Venn and Euler diagrams</article-title>
          .
          <source>Ph.D. dissertation</source>
          , University of Kent (
          <year>September 2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Frederickson</surname>
          </string-name>
          , B.:
          <article-title>venn.js: area proportional Venn and Euler diagrams in JavaScript (November 2016) original</article-title>
          -date:
          <fpage>2013</fpage>
          -05-09.
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Larsson</surname>
          </string-name>
          , J.:
          <article-title>eulerr: Area-proportional Euler diagrams with ellipses (</article-title>
          <year>2018</year>
          ) Available at: http://lup.lub.lu.se/student-papers/record/8934042.
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <given-names>R</given-names>
            <surname>Core Team: R: A Language</surname>
          </string-name>
          and
          <article-title>Environment for Statistical Computing</article-title>
          . R Foundation for Statistical Computing, Vienna, Austria. (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Schnabel</surname>
            ,
            <given-names>R.B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Koonatz</surname>
            ,
            <given-names>J.E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Weiss</surname>
            ,
            <given-names>B.E.</given-names>
          </string-name>
          :
          <article-title>A modular system of algorithms for unconstrained minimization</article-title>
          .
          <source>ACM Trans Math Softw</source>
          <volume>11</volume>
          (
          <issue>4</issue>
          ) (
          <year>December 1985</year>
          )
          <volume>419</volume>
          {
          <fpage>440</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Richter-Gebert</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          :
          <source>Perspectives on Projective Geometry: A Guided Tour Through Real and Complex Geometry. 1 edn</source>
          . Springer, Berlin, Germany (
          <year>February 2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Finley</surname>
            ,
            <given-names>D.R.</given-names>
          </string-name>
          :
          <article-title>Ultra-easy algorithm with C code sample</article-title>
          . http://alienryder ex.com/polygon area/ (
          <year>December 2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Eberly</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>The area of intersecting ellipses</article-title>
          (
          <year>November 2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Micallef</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rodgers</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Computing the Region Areas of Euler Diagrams Drawn with Three Ellipses</article-title>
          . In Burton, J.,
          <string-name>
            <surname>Stapleton</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Klein</surname>
          </string-name>
          , K., eds.
          <source>: CEUR Workshop Proceedings</source>
          . Volume
          <volume>1244</volume>
          ., Melbourne,
          <source>Australia (July</source>
          <year>2014</year>
          )
          <volume>1</volume>
          {
          <fpage>15</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Xiang</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gubian</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Suomela</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hoeng</surname>
          </string-name>
          , J.:
          <article-title>Generalized simulated annealing for global optimization: the GenSA package</article-title>
          .
          <source>The R Journal</source>
          <volume>5</volume>
          (
          <issue>1</issue>
          ) (
          <year>June 2013</year>
          )
          <volume>13</volume>
          {
          <fpage>28</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Sanderson</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Curtin</surname>
          </string-name>
          , R.:
          <article-title>Armadillo: a template-based C++ library for linear algebra</article-title>
          .
          <source>The Journal of Open Source Software</source>
          <volume>1</volume>
          (
          <issue>2</issue>
          ) (
          <year>2016</year>
          )
          <fpage>26</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Eddelbuettel</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Francois</surname>
          </string-name>
          , R.: Rcpp:
          <string-name>
            <surname>Seamless R and C+</surname>
          </string-name>
          <article-title>+ integration</article-title>
          .
          <source>Journal of Statistical Software</source>
          <volume>40</volume>
          (
          <issue>8</issue>
          ) (
          <year>2011</year>
          )
          <volume>1</volume>
          {
          <fpage>18</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Eddelbuettel</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sanderson</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>RcppArmadillo: accelerating R with highperformance C++ linear algebra</article-title>
          .
          <source>Computational Statistics and Data Analysis 71 (March</source>
          <year>2014</year>
          )
          <volume>1054</volume>
          {
          <fpage>1063</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <given-names>R</given-names>
            <surname>Core Team: The Comprehensive R Archive Network</surname>
          </string-name>
          (
          <year>November 2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Chang</surname>
          </string-name>
          , W., Cheng, J.,
          <string-name>
            <surname>Allaire</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Xie</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>McPherson</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <article-title>: shiny: Web Application Framework for</article-title>
          R. (
          <year>2017</year>
          )
          <article-title>R package version 1.0.5</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Lenz</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fornoni</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Chronic kidney disease care delivered by US family medicine and internal medicine trainees: results from an online survey</article-title>
          .
          <source>BMC medicine 4 (December</source>
          <year>2006</year>
          )
          <fpage>30</fpage>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>