<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>ProvenanceMatrix: A Visualization Tool for Multi-Taxonomy Alignments</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Tuan Dang</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Nico Franz</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Bertram Luda¨scher</string-name>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Angus Graeme Forbes</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Arizona State University</institution>
          ,
          <addr-line>Tempe, AZ</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>University of Illinois at Chicago</institution>
          ,
          <addr-line>Chicago, IL</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>University of Illinois at Urbana-Champaign</institution>
          ,
          <addr-line>IL</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
      </contrib-group>
      <fpage>13</fpage>
      <lpage>24</lpage>
      <abstract>
        <p>Visualizing and analyzing the relationships between taxonomic entities represented in multiple input classifications is both challenging and required due to recurrent new discoveries and inferences of taxa and their phylogenetic relationships. Despite the availability of numerous visualization techniques, the large size of hierarchical classifications and complex relations between taxonomic entities generated during a multi-taxonomy alignment process requires new visualizations. This paper introduces ProvenanceMatrix, a novel tool allowing end users (taxonomists, ecologists, phylogeneticists) to explore and comprehend the outcomes of taxonomic alignments. We illustrate the use of ProvenanceMatrix through examples using taxonomic classifications of various sizes, from a few to hundreds of taxonomic entities and hundreds of thousands of relationships.</p>
      </abstract>
      <kwd-group>
        <kwd>Taxonomic classification</kwd>
        <kwd>multi-taxonomy alignment</kwd>
        <kwd>phylogenetic relationship</kwd>
        <kwd>matrix representation</kwd>
        <kwd>glyph-based visualization</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        Visualization tools developed for the field of biological taxonomy (herein broadly
defined to include phylogenetics) may focus on representing the information content of
one comprehensive classification, or provide visual information on the relationships
between taxonomic entities represented in multiple, alternative classifications [
        <xref ref-type="bibr" rid="ref11 ref9">9, 11</xref>
        ].
The latter visualization services are useful in particular for illustrating important
similarities and differences in taxonomic perspective, which may be empirically rooted in
the discovery of new taxonomic entities, new evidence of phylogenetic relationship, or
in the differential sampling and weighting of phylogenetic evidence [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]. Such
multitaxonomy comparisons can be viewed as a solution to the challenge of representing
taxonomic provenance [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ], i.e., linking a taxonomy T1 to another (pre- or
postceding) taxonomy T2. To achieve this, taxonomic concepts endorsed by each alternative
classification are individuated using taxonomic concept labels with the syntax:
taxonomic name sec. (according to) taxonomic source [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. Linkage of same-sourced
concepts via parent-child (is-a) relationships permits the assembly of multiple independent
classifications, and therefore presents new opportunities for inferring and visualizing
taxonomic provenance across multiple classifications.
      </p>
      <p>
        Here were describe ProvenanceMatrix, a novel tool for visualizing some of the
knowledge products of EULER/X, a multi-taxonomy alignment toolkit [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. EULER/X is
a logic-based reasoning software capable of aligning (or “merging”) two or more
taxonomic concept hierarchies, using different underlying inference mechanisms, in
particular, answer sets [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] and qualitative reasoning using RCC-5 constraints [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]. The
reasoning process models taxonomies T1 and T2 as sets of is-a constraints, together with a
set A of expert-asserted input articulations that relate concepts in T1 with those in T2,
typically at the leaf level. Using RCC-5 (Region Connection Calculus) relations, the
expert can express through the articulations in A which relation holds between a concept
T1.X and a concept T2.Y , i.e., equals, includes, is included in, overlaps, or disjoint.
If the precise relationship is not known, then one of the non-elementary 25 = 32
disjunctive combinations of the 5 base relations can be used to express this uncertainty
[
        <xref ref-type="bibr" rid="ref17">17</xref>
        ].
      </p>
      <p>
        The toolkit workflow iteratively guides the expert user towards identifying sets of
input articulations that are both logically consistent and sufficiently specific to yield only
a limited number of consistent alignments [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. An important product of the alignment
process is the set of Maximally Informative Relations (MIR): for any pair (C1, C2) of
concepts from T1, T2, the MIR of (C1, C2) is the unique relation in the powerset lattice
R32 over the RCC-5 base relations which implies all other relations that hold between
C1 and C2, given T1, T2 and A.
      </p>
      <p>
        The MIR play a critical role in generating the set of consistent alignments
(“possible worlds”), in diagnosing undesired ambiguities in the input or output articulations,
and generally in understanding the toolkit reasoning outcomes. Visualization tools are
important in this context because the number of MIR for two taxonomies with m and n
concepts, respectively, is m × n. For instance, the alignment use case of Primates sec.
Groves (1993; T1) and Primates sec. Groves (2005; T2) contains 317 × 483 taxonomic
concepts and hence 153,111 MIR relations [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]. Displaying the MIR in list format is
not an effective method for exploration. Instead users need dynamically rendered,
scalable visualization solutions to navigate the large and semantically complex reasoning
outcomes and adjust the input accordingly to achieve the desired alignments.
      </p>
      <p>
        Key visualization challenges for multi-taxonomy alignment outcomes include the
following scenarios. Frequently the alternative taxonomies have unequal sets of
leaflevel children. For instance, recently published taxonomies may include new
specieslevel concepts for which there are no corresponding entities in preceding
classifications [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. Additionally, the visualization must display large numbers of data points
(&gt; 150,000 in the medium-sized Primate use case), where each point can be constituted
by any subset of RCC-5 articulations in the R32 lattice. In order to empirically assess
the reasoner-inferred articulations, users may also need to access taxonomic provenance
information such as feature-based diagnoses, illustrations, and other taxonomic
information.
      </p>
      <p>Using ProvenanceMatrix, we can visualize alignments of large taxonomies with up
to hundreds of concepts. Our technique uses matrix representation and glyphs in each
cell to highlight RCC-5 articulation sets and alignments. In Section 4, we demonstrate
how our technique effectively facilitates the exploration of multi-taxonomy alignments
with varying sizes and levels of alignment ambiguity.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Related Work</title>
      <p>
        An overview of the EULER/X multi-taxonomy alignment approach is provided in [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
Fig. 1 shows the current visualizations of two related concept taxonomies, plus
articulations among the respectively entailed taxonomic concepts. The aim is to visualize
the input taxonomies T1 and T2 and the resulting merged visualizations (rendered with
GraphViz [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]). In the figure, “==” means equals, “ &lt;” means is included in, “ &gt;” means
includes, “ &gt;&lt;” means overlaps, and “ |” means disjoint. The final product is a merged
taxonomy (as depicted in Fig. 1(b)) that represents the concept-level similarities and
differences among the aligned input trees. However, current GraphViz visualizations are
not interactive and do not facilitate efficient exploration of ambiguous (under-specified)
articulations which generate multiple possible world solutions. Resolving ambiguity is
a critical aspect of the alignment process.
      </p>
      <p>
        Tanglegrams are widely used in biology, for instance to represent the inferred
evolutionary histories of rooted phylogenetic networks [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ] and to highlight common
structures as well as differences in multiple DNA sequences [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]. A tanglegram draws two
rooted trees with the leaves opposing each other and uses auxiliary lines to connect
matching taxonomic entities at the leaf-level. These auxiliary lines can be rendered in
different styles or colors to encode different types of relationships (e.g., host-parasite
associations).
      </p>
      <p>
        The Concept Relationship Editor [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] extends the alignment process to support
assertions of relationships between taxonomic classifications at all levels of each aligned
hierarchy. Concept Relationship Editor adopts a space-filling adjacency layout which
allows users to expand multiple lists of taxonomic concepts with common parents. The
lens mode and scroll mode are two different ways to navigate across the hierarchy of
either classification while ensuring that the text strings in focus remain legible. Lines
are used to connect the related taxa with symbols at either end to indicate the
relationship type. Similar to tanglegrams, this technique can introduce visual clutter due to edge
crossings as the number of taxa increases.
      </p>
      <p>
        An alternative visualization approach utilizes
icicle tree representations [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]. The RCC-5
relationships are colored bands to connect pairs of
taxonomic concepts. Neighboring bands of the
same color are bundles that reduce cognitive load.
      </p>
      <p>Spaces between concepts of one taxonomy may
be used to better align the two trees and reduce
crossed bans. In addition, nodes may be
colorcoded to indicate what percentage of a node’s
descendants are congruent or not. Figure 2 shows an
example of the icicle tree representation. In the
diagram shown, purple means equals or congruent Fig. 2. An example of the icicle tree
(==), black means is included in or subset (&lt;), representation and colored bands to
blue means overlaps (&gt;&lt;). However, this tech- highlight articulations between pairs of
nique is only suitable for smaller numbers of con- tMaxcoGnuofmfin)ic concepts. (credit Michael
cepts or aggregate views of large classifications.</p>
      <p>When we try this technique on a large number of
taxonomic concepts, and especially when
multiple articulations between paired concepts must be displayed, the visualization becomes
cluttered due to band crossings.
3</p>
    </sec>
    <sec id="sec-3">
      <title>Design Motivation</title>
      <p>In this section, we review the primary challenges inherent in displaying RCC-5-based,
multi-taxonomy alignments. Addressing these challenges has motivated us to create a
new visualization technique that better supports the visual exploration tasks relevant to
such taxonomic reasoning products.</p>
      <p>The EULER/X input (constraint) and output (alignment) visualizations as depicted
in Figure 1 present slightly different sets of challenges. They are currently produced by
toolkit-native stylesheets that translate the user input and reasoner output into
GraphVizcompatible data files. While there is some limited flexibility in tweaking the GraphViz
output using EULER/X stylesheet options1, the ranked graph layout computed by
GraphViz may not reflect the user’s intuitions regarding the spatial arrangement of concepts
and relationships. For example, the ordering of children of a parent concept computed
by GraphViz is different from the order in which they appear in the source publications
for that taxonomy. This can create unintuitive experiences for the user.</p>
      <p>Smaller scale visualization enhancement goals is improving usability for
annotating/editing the GraphViz output data files. Larger scale goals entail acquiring the ability
1 The stylesheet options result in different GraphViz
“ constraint=false” ignores certain edges for layout purposes.
attribute
settings,</p>
      <p>e.g.,
to export/edit EULER/X visualizations in other (phylo-visualizing) platforms (however,
a related challenge is that the most popular programs may not support EULER/X
semantics which mandate the use of taxonomic concept labels, parent/child relationship
[same taxonomy], RCC-5 relationships [across taxonomies], and merge concepts labels
[AB, Ab, aB]).</p>
      <p>Here, we provide an overview of some of the main visualization tasks for visualizing
related concept taxonomies (hierarchies), as well as the relations between the concepts
in the input taxonomies. Given two or more taxonomies:</p>
      <p>T1. Focus on specific articulations.</p>
      <p>T2. Provide different ways to organize hierarchies. This helps to compare structure
of the input taxonomies.</p>
      <p>T3. Highlight taxonomic concepts in one classification that stand in various specific
and incongruent relations to concepts in the other classification.</p>
      <p>T4. Find related concepts and subtrees of one taxonomic classification to the other
taxonomic classification.</p>
      <p>T5. Display details on demand. In particular, users want to be able to overlay
distinctions between user-provided and toolkit-inferred articulations (i.e., articulation
source), and display additional domain-relevant information (such as characters,
images) when mousing over a concept label.</p>
      <p>T6. Collapse and expand a subtree to simplify or fully explore a branch. This feature
is particularly useful when dealing with large taxonomies.
4</p>
    </sec>
    <sec id="sec-4">
      <title>Methods</title>
      <p>
        Matrix representation is a useful tool for visualizing networks in many application
domains, such as protein-protein biological interactions [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] and social networks [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. This
technique is superior to using node-link diagrams when the networks are dense, given
that edge-crossings are the main limitation of node-link diagrams in visualizing these
networks. A drawback of matrix representation is the inability to represent the flow of
the networks [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. However, since network flow is irrelevant in multi-taxonomy
alignments, we found matrix representation to be best suited for visualizing the data products
discussed above. Moreover, matrix representation enables the display of multiple
(disjunct) relationships that may exist between a pair of elements from both dimensions in
a matrix [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
      </p>
      <p>
        Figure 3 shows an example of ProvenanceMatrix for the Perelleschus
classifications [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. Each side of the matrix displays an input taxonomy. The arcs are used to
indicate hierarchical information, directed from parent to subordinate child concepts.
MatLink [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] also uses arcs to indicate relationships but only considers undirected
networks. Moreover, the taxonomic concept labels are also indented appropriately to
highlight hierarchical arrangement of each input classification. In each cell of the matrix we
use circular sectors, divided similarly into a pie-chart, to indicate the articulations that
hold true between two taxonomic concepts, where each sector (pie-slice) in the circle
is given a color to consistently indicate the articulation type. The more pie-slices are
shown, the less we know about the pair of concepts. Thus, a “full circle” (with all 5
pieslices) means we know nothing about a relation. These “full circle” can act as “alerts”
to the user that the alignment is problematic (too ambiguous). Conversely, a single slice
is the best case, specifying a unique (fully specified) relationship between two concepts.
Color legend is depicted on the right of Figure 3. For example, green represents equals
and blue represents includes. We use the same color coding for articulations in the the
rest of this paper. Users can enable or disable an articulation type as desired (T1).
      </p>
      <p>ProvenanceMatrix supports three ways
of ordering taxonomic concepts,
designed to highlight different aspects of
the input hierarchies as well as their
RCC-5 articulations. (1) Ordering the
matrix with respect to the structure of the
input trees. Figure 4 shows
ProvenanceMatrix with different orderings of
taxonomic concepts (T2). (1.1.) Breadth-first
ordering in Figure 4(a) lists all sibling
together before diving into their
respective child-level concepts. (1.2.) Depth- Fig. 3. An example showing the use of
Provefirst ordering in Figure 4(b) lists the chil- nanceMatrix for the Perelleschus classifications.
dren right after each taxonomic concept.</p>
      <p>
        The hierarchy is more readable in this
ordering since there are no crossing arcs in the same taxonomic classification. To avoid the
overlapping between arcs and glyphs in the matrix, we can replace arcs by straight lines
connecting parent to child concepts. (2) In Figure 4(c), we order the taxonomic
concepts based on the similarity of their articulation sets. (The details of how we compute
similarity and the ordering algorithm are described in [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].) This ordering brings
concepts with multiple alignments to the top left corner of the matrix; these multiple
alignments generate the 160 possible worlds in the taxonomy alignment of Gymnospermae
sec. Weakley (2010) versus RAB (Radford, Ahles and Bell) (1968) [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. The example
shows ambiguities in the multi-taxonomy alignment which our visualization software
can readily identify and isolate to facilitate user-mediated diagnosis and resolution of
such ambiguities. In addition, congruent relations (in green) are pushed further to the
bottom right of the matrix.
      </p>
      <p>Due to the discovery and/or inclusion of new taxonomic entities in the later (2010)
classification, the alternative taxonomies have unequal sets of leaf-level children. In
other words, recently published taxonomies may include new species-level concepts
for which there are no corresponding entities in preceding classifications. Accordingly,
in ProvenanceMatrix, we classify taxonomic concepts into three different categories
(T3):
–
–</p>
      <p>Neither the concept nor any of its children of one taxonomy have congruent
relationships with entities in the other taxonomy. In other words, a (set of) concept(s)
has no match whatsoever (“bad apples”). Such concepts are highlighted in red in
Figure 5.</p>
      <p>A parent-level concept is incongruent but entails one or more congruent child-level
concepts. In other words, the higher-level concepts is a unique conglomerate of
variously congruent subentities, some of which have matching entities in the other
taxonomy. Such parent-level concepts are the dark green entities in Figure 5.
– A concept has at least one congruent relationship with a concept in the other
taxonomy. Such concepts are highlighted in green.</p>
      <p>In Figure 5, we also show brushing and linking to highlight the corresponding
subtrees of the aligned taxonomic classifications ( T4). An associated subtree is discovered
based on the presence of congruent relationships which are connected by green lines. In
this example, the associated subtree (on the left) of Pinus sec. 2010/1968 (in the box)
is discovered in light of its aligned children, not the selected (higher-level) taxonomic
concept itself. Notice that half of the children (in red) of Pinus sec. Weakley (2010)
have no congruent match in the RAB (1968) classification.</p>
      <p>Additional information and sample images (e.g., from Wikipedia pages of which
may entail taxonomic concept information) can be displayed on demand when mousing
over a taxonomic concept label (T5). Moreover, users can request to overlay the source
of articulations (i.e., user input, reasoner inference). Figure 6 shows an example of
overlaying such articulation sources in a non-domain demonstration alignment of U.S.
regional classifications from National the Diversity Council and Big Data Hubs,
respectively. In particular, black cells indicate user input whereas light blue and pink cells are
deduced and inferred articulations. Notice that articulation types (circular sectors) are
still visible in each cell.</p>
      <p>
        ProvenanceMatrix offers two ways navigate and comprehend larger classifications
of hundreds of taxonomic concepts: (a) lensing and (b) collapsing a subtree of the input
hierarchies (T6). Figure 7 shows a use case of aligning the Primates sec. Groves (1993)
and sec. Groves (2005) that contains 317*483 taxonomic concepts and hence 153,111
MIR [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]. Figure 7(a) shows lensing on a sub-section of the matrix, where only the
concept labels (about 20 labels) in the lensing area are printed out. Figure 7(b) shows
collapsing of a section of the input hierarchies. A plus sign appears in front of those
taxonomic concept labels which are collapsed. ProvenanceMatrix also provides
searching capability. When users input a concept name into a textbox, ProveanceMatrix only
expands the subtree of the search concept and collapses other irrelevant subtrees. At the
same time, only related concepts in the other taxonomic classification are expanded.
      </p>
      <p>The ProvenanceMatrix application, source code, and an accompanying video
tutorial are available online via our project repository.2
5</p>
    </sec>
    <sec id="sec-5">
      <title>Expert User Feedback</title>
      <p>
        The herein provided use cases were provided by EULER/X user and co-author NMF,
whose feedback has driven the optimization of the new visualizations.
ProvenanceMatrix confers two immediate and new visualization services:
2 https://github.com/CreativeCodingLab/ProvenanceMatrix.
Fig. 7. Visualizing the alignment of two Primates classifications containing 317*483 taxonomic
concepts and 153,111 MIR [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]: (a) lensing on area of interest in the matrix (b) collapsing
subhierarchies.
(1) In cases where certain concept-to-concept articulations are ambiguous (RCC-5
disjunctions) in the output, the corresponding concepts can be spatially aggregated and
thus identified very easily by the user. This can lead to an accelerated understanding
and subsequent removal of the ambiguity issues. Without the visualization, one has to
instead “comb through” a spreadsheet that may contain many thousands of rows of
data. We have succeeded in scaling ProvenanceMatrix to this level, even with 153,111
articulations in the Primates sec. 2005/1993 use case [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ].
      </p>
      <p>(2) We can show “information expression” that is newly acquired through the E
ULER/X toolkit reasoning process. For instance, in the Primates use case the expert user
provided 402 pairwise input articulations. The reasoning process produces from this
153,111 pairwise MIR relations, i.e., about 380 times as many articulations are
logically implied by the input but were not explicitly stated therein. The differential levels
of information expression before and after the reasoning process are correspondingly
visualized with ProvenanceMatrix through two matrix versions, and thus show the
powers of the reasoning approach.</p>
      <p>In summary, ProvenanceMatrix provides speedy and interactive identification of
ambiguous input and newly inferred output information, presenting a major
improvement over existing visualizations.
6</p>
    </sec>
    <sec id="sec-6">
      <title>Conclusion and Future Work</title>
      <p>This paper introduces a novel technique, ProvenanceMatrix, for visualizing the
products of a multi-taxonomy alignments generated with the reasoning toolkit EULER/X.
Using ProvenanceMatrix, users (taxonomists, ecologists, phylogeneticists) can
visualize alignments of large taxonomies with up to hundreds of input concepts. Glyphs in
each cell highlight RCC-5 articulations for a pair of taxonomic concepts.
ProvenanceMatrix supports a range of desirable user interactions, such as filtering the matrix by
articulations, ordering taxonomic entities with respect to the structure of the input
hierarchies, brushing and linking concepts, and collapsing/expanding sub-hierarchies. We
have demonstrated how our application effectively facilitates the exploration of
multitaxonomy alignments with different levels of alignment ambiguity and varying sizes,
from a few to hundreds of taxonomic entities (and hundreds of thousands of
relationships). This technique can be extended to visualize more than two taxonomic
classifications – a feature in development for the corresponding reasoning toolkit. In particular,
we can have multiple input classifications aligned by rows and columns, where each
pair of taxonomic classifications forms a new ProvenanceMatrix. In other words, we
can create a matrix of ProvenanceMatrix matrices, where each cell contains a matrix
(similar to the idea of a scatterplot matrix). Future work will investigate this strategy to
enable multi-dimensional alignments.</p>
    </sec>
    <sec id="sec-7">
      <title>Acknowledgements</title>
      <p>This work was funded by the DARPA Big Mechanism Program under ARO contract
WF911NF-14-1-0395, and in part by the National Science Foundation through NSF
DEB-1155984, DBI-1342595, NSF IIS-118088, and DBI-1147273.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <given-names>M.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Franz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bowers</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Luda</surname>
          </string-name>
          <article-title>¨scher. Euler/x: A toolkit for logic-based taxonomy integration</article-title>
          .
          <source>CoRR, abs/1402</source>
          .
          <year>1992</year>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <given-names>M.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Franz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bowers</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Luda</surname>
          </string-name>
          <article-title>¨scher. A hybrid diagnosis approach combining black-box and white-box reasoning</article-title>
          . In A. Bikakis,
          <string-name>
            <given-names>P.</given-names>
            <surname>Fodor</surname>
          </string-name>
          , and D. Roman, editors,
          <source>Rules on the Web. From Theory to Applications</source>
          , volume
          <volume>8620</volume>
          of Lecture Notes in Computer Science, pages
          <fpage>127</fpage>
          -
          <lpage>141</lpage>
          . Springer International Publishing,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <given-names>P.</given-names>
            <surname>Craig</surname>
          </string-name>
          and
          <string-name>
            <given-names>J.</given-names>
            <surname>Kennedy</surname>
          </string-name>
          .
          <article-title>Concept relationship editor: a visual interface to support the assertion of synonymy relationships between taxonomic classifications</article-title>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <given-names>T. N.</given-names>
            <surname>Dang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Murray</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Aurisano</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A. G.</given-names>
            <surname>Forbes. ReactionFlow:</surname>
          </string-name>
          <article-title>Visualizing relationships between proteins and complexes in biological pathways</article-title>
          .
          <source>BMC Proceedings</source>
          ,
          <volume>9</volume>
          (
          <issue>6</issue>
          ):
          <fpage>S6</fpage>
          ,
          <year>August 2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <given-names>T. N.</given-names>
            <surname>Dang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Murray</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A. G.</given-names>
            <surname>Forbes. PathwayMatrix:</surname>
          </string-name>
          <article-title>Visualizing binary relationships between proteins in biological pathways</article-title>
          .
          <source>BMC Proceedings</source>
          ,
          <volume>9</volume>
          (
          <issue>6</issue>
          ):
          <fpage>S3</fpage>
          ,
          <year>August 2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <given-names>J.</given-names>
            <surname>Ellson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E. R.</given-names>
            <surname>Gansner</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Koutsofios</surname>
          </string-name>
          , S. C. North, and
          <string-name>
            <given-names>G.</given-names>
            <surname>Woodhull. Graphviz - Open Source Graph Drawing Tools</surname>
          </string-name>
          .
          <source>Graph Drawing</source>
          , pages
          <fpage>483</fpage>
          -
          <lpage>484</lpage>
          ,
          <year>2001</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <given-names>N.</given-names>
            <surname>Elmqvist</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.-N.</given-names>
            <surname>Do</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Goodell</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Henry</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Fekete</surname>
          </string-name>
          . Zame:
          <article-title>Interactive large-scale graph visualization</article-title>
          .
          <source>In Visualization Symposium</source>
          ,
          <year>2008</year>
          . PacificVIS '08. IEEE Pacific , pages
          <fpage>215</fpage>
          -
          <lpage>222</lpage>
          ,
          <year>March 2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <given-names>N.</given-names>
            <surname>Franz</surname>
          </string-name>
          and
          <string-name>
            <given-names>R.</given-names>
            <surname>Peet</surname>
          </string-name>
          .
          <article-title>Perspectives: towards a language for mapping relationships among taxonomic concepts</article-title>
          .
          <source>Systematics and Biodiversity</source>
          ,
          <volume>7</volume>
          (
          <issue>1</issue>
          ):
          <fpage>5</fpage>
          -
          <lpage>20</lpage>
          ,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <given-names>N. M.</given-names>
            <surname>Franz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Kianmajd</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bowers</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Luda</surname>
          </string-name>
          <article-title>¨scher. Reasoning over taxonomic change: Exploring alignments for the Perelleschus use case</article-title>
          .
          <source>PLoS ONE</source>
          ,
          <volume>10</volume>
          (
          <issue>2</issue>
          ):e0118247,
          <year>02 2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>N. M. Franz</surname>
            ,
            <given-names>R. K.</given-names>
          </string-name>
          <string-name>
            <surname>Peet</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A. S.</given-names>
            <surname>Weakley</surname>
          </string-name>
          .
          <article-title>On the use of taxonomic concepts in support of biodiversity research and taxonomy</article-title>
          . Systematics Association Special Volume,
          <volume>76</volume>
          :
          <fpage>63</fpage>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>N. M. Franz</surname>
            ,
            <given-names>N. M.</given-names>
          </string-name>
          <string-name>
            <surname>Pier</surname>
            ,
            <given-names>D. M.</given-names>
          </string-name>
          <string-name>
            <surname>Reeder</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Chen</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Yu</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Kianmajd</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Bowers</surname>
            , and
            <given-names>B. Luda¨scher. Taxonomic</given-names>
          </string-name>
          <string-name>
            <surname>Provenance: Two Influential Primate Classifications Logically Aligned. ArXiv</surname>
          </string-name>
          e-prints,
          <source>Dec</source>
          .
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>M.</surname>
          </string-name>
          <article-title>Gelfond</article-title>
          .
          <source>In Handbook of Knowledge Representation, chapter Answer Sets. Elsevier Science</source>
          ,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <given-names>N.</given-names>
            <surname>Henry</surname>
          </string-name>
          and
          <string-name>
            <given-names>J.-D.</given-names>
            <surname>Fekete</surname>
          </string-name>
          . Matlink:
          <article-title>Enhanced matrix visualization for analyzing social networks</article-title>
          . In C. Baranauskas,
          <string-name>
            <given-names>P.</given-names>
            <surname>Palanque</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Abascal</surname>
          </string-name>
          , and S. Barbosa, editors,
          <source>HumanComputer Interaction INTERACT</source>
          <year>2007</year>
          , volume
          <volume>4663</volume>
          of Lecture Notes in Computer Science, pages
          <fpage>288</fpage>
          -
          <lpage>302</lpage>
          . Springer Berlin Heidelberg,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>J. B. Kruskal</surname>
            and
            <given-names>J. M.</given-names>
          </string-name>
          <string-name>
            <surname>Landwehr</surname>
          </string-name>
          .
          <article-title>Icicle plots: Better displays for hierarchical clustering</article-title>
          .
          <source>The American Statistician</source>
          ,
          <volume>37</volume>
          (
          <issue>2</issue>
          ):
          <fpage>162</fpage>
          -
          <lpage>168</lpage>
          ,
          <year>1983</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <given-names>D. A.</given-names>
            <surname>Randell</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Cui</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A. G.</given-names>
            <surname>Cohn</surname>
          </string-name>
          .
          <article-title>A spatial logic based on regions and connection</article-title>
          .
          <source>In Proceedings of the 3rd international conference on Knowledge Representation and Reasoning</source>
          ,
          <year>1992</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>C. Scornavacca</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          <string-name>
            <surname>Zickmann</surname>
            , and
            <given-names>D. H.</given-names>
          </string-name>
          <string-name>
            <surname>Huson</surname>
          </string-name>
          .
          <article-title>Tanglegrams for rooted phylogenetic trees and networks</article-title>
          .
          <source>Bioinformatics</source>
          ,
          <volume>27</volume>
          (
          <issue>13</issue>
          ):
          <fpage>i248</fpage>
          -
          <lpage>i256</lpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <given-names>D.</given-names>
            <surname>Thau</surname>
          </string-name>
          .
          <article-title>Reasoning about taxonomies and articulations</article-title>
          .
          <source>In Proceedings of the 2008 EDBT Ph. D. workshop</source>
          , pages
          <fpage>11</fpage>
          -
          <lpage>19</lpage>
          . ACM,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <given-names>W. N. W.</given-names>
            <surname>Zainon</surname>
          </string-name>
          and
          <string-name>
            <given-names>P.</given-names>
            <surname>Calder</surname>
          </string-name>
          .
          <article-title>Visualising phylogenetic trees</article-title>
          .
          <source>In Proceedings of the 7th Australasian User Interface Conference - Volume 50, AUIC '06</source>
          , pages
          <fpage>145</fpage>
          -
          <lpage>152</lpage>
          , Darlinghurst, Australia, Australia,
          <year>2006</year>
          . Australian Computer Society, Inc.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>