<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Extending a dashboard meta-model to account for users' characteristics and goals for enhancing personalization</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>GRIAL Research Group</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Computer Sciences Department</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Research Institute for Educational</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Department of Mechanics, Computer Science and Aerospace Engineering. University of León.</institution>
          <addr-line>León</addr-line>
          ,
          <country country="ES">Spain</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Sciences, University of Salamanca</institution>
          ,
          <addr-line>Salamanca</addr-line>
          ,
          <country country="ES">Spain</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>VisUSAL Research Group. University of Salamanca.</institution>
          <addr-line>Salamanca</addr-line>
          ,
          <country country="ES">Spain</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2019</year>
      </pub-date>
      <fpage>0000</fpage>
      <lpage>0002</lpage>
      <abstract>
        <p>Information dashboards are useful tools for exploiting datasets and support decision-making processes. However, these tools are not trivial to design and build. Information dashboards not only involve a set of visualizations and handlers to manage the presented data, but also a set of users that will potentially benefit from the knowledge generated by interacting with the data. It is important to know and understand the requirements of the final users of a dashboard because they will influence the design processes. But several user profiles can be involved, making these processes even more complicated. This paper identifies and discusses why it is essential to include the final users when modeling a dashboard. Through meta-modeling, different characteristics of potential users are structured, thus obtaining a meta-model that dissects not only technical and functional features of a dashboard (from an abstract point of view) but also the different aspects of the final users that will make use of it. By identifying these user characteristics and by arranging them into a meta-model, software engineering paradigms such as model-driven development or software product lines can employ it as an input for generating concrete dashboard products. This approach could be useful for generating Learning Analytics dashboards that take into account the users' motivations, beliefs, and knowledge.</p>
      </abstract>
      <kwd-group>
        <kwd>Information Dashboards</kwd>
        <kwd>Meta-model</kwd>
        <kwd>Information Visualization</kwd>
        <kwd>User Model</kwd>
        <kwd>MDA</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>Information dashboards are compelling tools for generating knowledge and for
supporting data-driven decisions. These tools allow users to visually understand and
extract patterns from their datasets, fostering informed decision-making processes.</p>
      <p>However, dashboards are also sophisticated tools, both in terms of development and
use. First, the development of an information dashboard is not trivial; developers need
to detail and understand the goal of the dashboard, the domain in which it will be
framed, the information that will be presented, and, last but not least, the users that will
use the dashboard.</p>
      <p>
        Users, from an abstract point of view, are complex entities, with different
characteristics from one to the other, with different behaviors, beliefs, and goals [
        <xref ref-type="bibr" rid="ref1 ref2">1, 2</xref>
        ]. This
fact means that a specific dashboard configuration could be extremely beneficial for
one individual, but entirely useless for another, as it could not match his or her goals,
domain knowledge, visual literacy, and of course, his or her individual preferences.
      </p>
      <p>
        In existing literature about the process of designing a dashboard, several authors
point out the necessity of taking into account the problem to be solved through the
visual presentation of data [
        <xref ref-type="bibr" rid="ref3 ref4 ref5">3-5</xref>
        ]. However, the problem definition is tightly related to
the data domain and the user goals [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ], thus needing to address the problem particularly
in the target domain’s context, spoiling the opportunity of reusing components, hence
consuming time and resources.
      </p>
      <p>Generalizing these user dimensions can be useful to understand the problem's
domain better, to improve the dashboards' development processes, and to provide
personalized products that take into account individual requirements. That is the reason why
it is crucial to extract commonalities in user tasks and interactions, no matter the data
context or domain. In the end, the user behavior is based in primitive tasks (pan, zoom,
click, hover, etc.) that will provide them with outputs to reach their goals and to improve
insights delivery processes.</p>
      <p>
        Some software engineering paradigms can benefit from the abstraction of the
elements that compose a dashboard, users included. Such paradigms, like model-driven
development (MDD) [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] or software product lines (SPL) [
        <xref ref-type="bibr" rid="ref8 ref9">8, 9</xref>
        ] aim at decreasing
development time by leveraging the reuse of software components or by mapping
highlevel models to concrete models or code.
      </p>
      <p>In this paper, an extension of a previously developed dashboard meta-model
[1012] is presented. This extension takes into account different user dimensions that can
influence dashboard components, to establish a framework for generating personalized
dashboards that foster better user experience and insights delivery.</p>
      <p>
        Characterizing the user could lead to benefits in fields like Learning Analytics (LA),
where dashboards showing the users’ learning data could be counterproductive if
individual aspects are not addressed [
        <xref ref-type="bibr" rid="ref13 ref14">13, 14</xref>
        ].
      </p>
      <p>The remainder of this paper is organized as follows. Section 2 describes the
methodology followed to model the user from an abstract point of view using
meta-modeling. In Section 3, the obtained meta-model is provided and explained. In Section 4, the
meta-model is discussed, to finally close with Section 6, where conclusions and future
research lines are presented.</p>
    </sec>
    <sec id="sec-2">
      <title>Methodology</title>
      <p>
        The followed methodology employs a meta-model, an artifact from the model-driven
architecture paradigm [
        <xref ref-type="bibr" rid="ref15 ref7">7, 15</xref>
        ]. Meta-models are useful for capturing high level and
abstract concepts, and not only for understanding the problem's domain but also to
document and represent in a structured manner these concepts. Thus meta-modeling fosters
the development of general rules, constraints, structures, etc., for a set of related
problems by abstracting shared features and relations found in particular domain's instances.
      </p>
      <p>
        But why applying meta-modeling to the dashboards' domain? As introduced, this
domain is extraordinarily complex, because not only the technical features of a
dashboard should be identified and detailed, but also the final users’ characteristics that can
influence their experience with the dashboard. Through domain engineering [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]
processes, all these properties can be abstracted into a set of conceptual classes and
relations among them, obtaining a simplified representation of the problem’s domain.
      </p>
      <p>
        These abstract models can be mapped to concrete products, according to the OMG
four-layer meta-model architecture [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ]: meta-meta-model layer (M3), meta-model
layer (M2), user model layer (M1) and user object layer (M0). In this work, the
presented dashboard meta-model is an M2 model (an instantiation of the M3 layer, using
MOF language), which, in turn, can be instantiated to obtain dashboard instances.
3
      </p>
    </sec>
    <sec id="sec-3">
      <title>The meta-model</title>
      <p>In this section, the designed meta-model is presented. As introduced in the previous
section, the level of abstraction of the meta-model is high, to capture generic
commonalities among the potential objects. The main benefit from these levels of abstraction is
the achievement of a general model from which concrete models can be instantiated.</p>
      <p>The initial dashboard meta-model to be extended consist of five main classes, and
two specializations (showed in Fig. 1). This meta-model captures at a very high level
the different components of an information dashboard, as well as its generic layout,
which, in the end, is based on containers that can be either rows and columns. A user
could employ one or more dashboards, and a dashboard, through this approach, belong
to one user, because involving more users would introduce noise in the personalization
process of a dashboard.</p>
      <p>As can be seen in this simple meta-model, details are omitted. The User class
represents a high-level user, but none of his or her characteristics are represented nor
detailed. However, the user should be defined in terms of different significant and
influential aspects to support a personalized dashboard design, thus being necessary to
extend this meta-model with more elements regarding the users’ characteristics and goals,
as well as defining the relations of these aspects with the dashboard’s components.</p>
      <p>Given that, the extended dashboard meta-model is presented in Fig. 2. The diagram
represents the same dashboard structure as in Fig. 1, but in this case, the user has been
decomposed in terms of his or her goals and his or her characteristics.</p>
      <p>Firstly, a new concept arises; Goal. A user employing a dashboard must have at least
one goal, however implicit. Even users that want to explore data casually have a goal
(that is, exploring data itself). That is the reason for the "one or more" (1..*)
multiplicity. In turn, a goal can belong to any user, and users can share common and general
purposes, explaining the "zero or more" (*) multiplicity on that side of the relation.</p>
      <p>On the other hand, a goal can be broken down into individual and more specific
tasks. Simple goals can be accomplished by performing one task, e.g. if a concrete goal
is “to know which USA city has the largest number of inhabitants," a straightforward
yet necessary task could be “to sort USA cities by population number," meaning that
the dashboard components must support sorting capabilities.</p>
      <p>However, more complex goals might involve several specific and chained tasks such
as “to understand why there has been a business income loss within the last six months,"
which could involve applying different tasks to different dimensions of the data to reach
insights about the stated problem. That is the reason why the dashboard’s components
need to support the identified responsibilities to enable them.</p>
      <p>Finally, a user can have zero or more identified characteristics, given the fact that, at
a certain point, there could be no user data available of the possible dimensions. These
characteristics can belong to zero or more users, as different users can share general
characteristics. Characteristics can be of a different kind; preferences, disabilities,
knowledge about different domains, visualization literacy, and bias (action, perceptual,
or social bias). These characteristics can influence the dashboard’s components to adapt
them and, therefore, to match the identified user aspects.
4</p>
    </sec>
    <sec id="sec-4">
      <title>Discussion</title>
      <p>
        Including the user as an extremely significant element within the dashboard, domain is
crucial. The development processes of a visualizations and dashboards start with the
user (requirement elicitation) and end with the user (product refining) [
        <xref ref-type="bibr" rid="ref18 ref5">5, 18</xref>
        ], so not
only the technical features of a dashboard should be taken into account when
metamodeling these tools, as these features arise from the users' requirements and are
influenced by them [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ].
      </p>
      <p>
        The developed meta-model defines the users of a dashboard in terms of their goals
and their characteristics. The users' goals drive the whole dashboard design processes,
as it will influence user behavior when interacting with the dashboard's components
[
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. However, goals are not enough to define a dashboard’s configuration, it is
necessary to decompose these goals into primitive tasks that can be directly supported by the
dashboard's features (e.g., sort data, highlight data, annotate data, zoom, etc.) [
        <xref ref-type="bibr" rid="ref21 ref22 ref23">21-23</xref>
        ].
      </p>
      <p>High-level user goals and user characteristics would be mapped low-level
interactions in particular dashboard views presenting specific data dimensions to provide the
user with a dashboard that could fulfill their information needs.</p>
      <p>Once goals are addressed at high-level, the next phase is to take into account user
preferences (implicitly exposed in its purposes, like, for example, the data that the user
is interested in) as well as other characteristics, like the user’s knowledge level about
the data domain, the user’s visual literacy and the user’s potential biases. This process
would provide the most suitable view type by configuring recognizable visual marks or
visual metaphors, proper axes domains, preferred visual design, etc. Finally, user
disabilities, such as color blindness, hand tremors, etc., would refine the dashboards' visual
design and interaction methods by choosing right color palettes, mouse sensibility, etc.</p>
      <p>
        The listed characteristics are hugely significant as they play an essential role when
interpreting visualizations and reaching insights from them. For example, not being
familiar with a type a visualization can lead to confusion and could be error prone when
trying to reach insights [
        <xref ref-type="bibr" rid="ref24 ref25">24, 25</xref>
        ]. For these reasons, assessing visualization literacy is
currently an important research field [
        <xref ref-type="bibr" rid="ref26 ref27">26, 27</xref>
        ], to address beforehand the users'
visualization skills, delivering an understandable yet useful set of visualizations for them.
Also, the users' knowledge level about the data domain should be addressed in the same
manner; by providing views with right data dimensions and contextual information to
mitigate unawareness about the domain [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ].
      </p>
      <p>
        On the other hand, user bias is not only influenced by past visualization experiences,
but also by gender, age, race, etc. Why is it important to take this information into
account? It could be seen as irrelevant factors, but the truth is that, unconsciously, bias
could lead to valuable information loss [
        <xref ref-type="bibr" rid="ref1 ref2">1, 2</xref>
        ], that not only could undermine people but
could also lead to financial losses by not addressing final users' bias when analyzing
data [
        <xref ref-type="bibr" rid="ref28">28</xref>
        ]. But not only social biases (beliefs, expectations, etc.) are relevant within this
context; action and perceptual biases can be harmful as well [
        <xref ref-type="bibr" rid="ref29">29</xref>
        ]. It is crucial to model
dashboards taking into account these factors, because unintentionally, and from the
user’s point of view, he or she could ignore data that could lead to beneficial decisions,
thus being the insights reached half-truths.
      </p>
      <p>
        Using generalization for modeling the above characteristics support the inclusion of
new factors that might arise, allowing the meta-model’s evolution. These identified
factors can influence dashboards to match both explicit and implicit characteristics,
obtaining an effective and tailored visualization tool. However, there should also be room
for customizing the dashboard, as the user should also have the freedom to craft their
dashboards or to modify certain features. The main drawback of this approach is the
retrieval of all the presented user dimensions, not only because several factors are
involved, but because the information must be precise to map these characteristics into
proper dashboard components successfully. Questionnaires about the different
dimensions could be employed, like [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ] for visual literacy, or even automatic approaches
that measure these aspects through the analysis of users’ behavior [
        <xref ref-type="bibr" rid="ref30">30</xref>
        ].
      </p>
      <p>
        Understanding user necessities is essential in the dashboard domain, but especially
in some subdomains, such as LA dashboards. LA dashboards aim at visually assisting
users (teachers, students, etc.) through a “single display that aggregates different
indicators about learner(s), learning process(es) and learning context(s) into one or multiple
visualizations," as stated in [
        <xref ref-type="bibr" rid="ref31">31</xref>
        ]. Personalizing these displays can foster self-regulated
learning and academic achievement [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. The presented meta-model can support
personalization processes to achieve the mentioned benefits. Also, using this abstract
metamodel can leverage reusability not only at a component-development level but also at
design-level, by reusing knowledge.
5
      </p>
    </sec>
    <sec id="sec-5">
      <title>Conclusions</title>
      <p>In this paper, a dashboards meta-model extension is presented. The extension involves
the inclusion of the final users as the main element of a dashboard, given their influence
in the different design processes regarding the development of these tools. Different
perspectives of the user are identified and discussed, such as the user goals, preferences,
bias, disabilities, etc., to include them in the meta-model through high-level classes.</p>
      <p>The purpose of having a dashboards meta-model is to provide a framework for
instantiating any possible dashboard product, enabling personalization of individual
dashboards. This approach could be useful for tailoring LA dashboards, where the
necessities of each user can depend on their learning processes and motivations.</p>
      <p>Future research lines would involve refining the meta-model through the addition
of more specific properties, constraints, rules and the inclusion of design guidelines to
support the automatic generation of concrete dashboards by instantiating the
metamodel, and also designing questionnaires and methods to retrieve the presented user
characteristics to finally implement the meta-model and validate it through case studies.
Acknowledgements
This research work has been supported by the Spanish Ministry of Education and
Vocational Training under a FPU fellowship (FPU17/03276). This work has been partially
funded by the Spanish Government Ministry of Economy and Competitiveness
throughout the DEFINES project (Ref. TIN2016-80172-R), T-CUIDA project (Ref.
SA061P17), and the PROVIDEDH project, funded within the CHIST-ERA Programme
under the national grant agreement: PCIN-2017-064 (MINECO, Spain).</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Hullman</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Adar</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Shah</surname>
            ,
            <given-names>P.:</given-names>
          </string-name>
          <article-title>The impact of social information on visual judgments</article-title>
          .
          <source>In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems</source>
          , pp.
          <fpage>1461</fpage>
          -
          <lpage>1470</lpage>
          . ACM, (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Kim</surname>
            ,
            <given-names>Y.-S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Reinecke</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hullman</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          :
          <article-title>Data through others' eyes: The impact of visualizing others' expectations on visualization interpretation</article-title>
          .
          <source>IEEE transactions on visualization and computer graphics 24</source>
          ,
          <fpage>760</fpage>
          -
          <lpage>769</lpage>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Mazza</surname>
          </string-name>
          , R.: Introduction to information visualization. Springer Science &amp; Business
          <string-name>
            <surname>Media</surname>
          </string-name>
          (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Few</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>Information dashboard design</article-title>
          . (
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Berinato</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>Good charts: The HBR guide to making smarter, more persuasive data visualizations</article-title>
          . Harvard Business Review Press (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Colomo-Palacios</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Casado-Lumbreras</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Soto-Acosta</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Misra</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>García-Peñalvo</surname>
            ,
            <given-names>F.J.:</given-names>
          </string-name>
          <article-title>Analyzing human resource management practices within the GSD context</article-title>
          .
          <source>Journal of Global Information Technology Management</source>
          <volume>15</volume>
          ,
          <fpage>30</fpage>
          -
          <lpage>54</lpage>
          (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Kleppe</surname>
            ,
            <given-names>A.G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Warmer</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bast</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          : MDA Explained.
          <article-title>The Model Driven Architecture: Practice and Promise</article-title>
          .
          <string-name>
            <surname>Addison-Wesley Longman</surname>
          </string-name>
          Publishing Co., Inc., Boston, MA (
          <year>2003</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Clements</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Northrop</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>Software product lines</article-title>
          .
          <source>Addison-Wesley</source>
          (
          <year>2002</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Gomaa</surname>
          </string-name>
          , H.:
          <article-title>Designing Software Product Lines with UML: From Use Cases to Pattern-Based Software Architectures</article-title>
          .
          <source>Addison Wesley Longman Publishing Co., Inc</source>
          . (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <given-names>Vázquez</given-names>
            <surname>Ingelmo</surname>
          </string-name>
          ,
          <string-name>
            <surname>A.</surname>
          </string-name>
          : Aplicación de Ingeniería de Dominio para la generación de dashboards personalizados.
          <source>Grupo GRIAL</source>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Vázquez-Ingelmo</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>García-Peñalvo</surname>
            ,
            <given-names>F.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Therón</surname>
          </string-name>
          , R.:
          <article-title>Addressing Fine-Grained Variability in User-Centered Software Product Lines: A Case Study on Dashboards</article-title>
          .
          <source>In: World Conference on Information Systems and Technologies</source>
          , pp.
          <fpage>855</fpage>
          -
          <lpage>864</lpage>
          . Springer, (
          <year>2019</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Vázquez-Ingelmo</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>García-Peñalvo</surname>
            ,
            <given-names>F.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Therón</surname>
          </string-name>
          , R.:
          <article-title>Taking advantage of the software product line paradigm to generate customized user interfaces for decision-making processes: a case study on university employability</article-title>
          . PeerJ Computer Science (Article In Press)
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Roberts</surname>
            ,
            <given-names>L.D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Howell</surname>
            ,
            <given-names>J.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Seaman</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          :
          <article-title>Give me a customizable dashboard: Personalized learning analytics dashboards in higher education</article-title>
          .
          <source>Technology, Knowledge and Learning</source>
          <volume>22</volume>
          ,
          <fpage>317</fpage>
          -
          <lpage>333</lpage>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Jivet</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Scheffel</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Drachsler</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Specht</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Awareness is not enough: pitfalls of learning analytics dashboards in the educational practice</article-title>
          .
          <source>In: European Conference on Technology Enhanced Learning</source>
          , pp.
          <fpage>82</fpage>
          -
          <lpage>96</lpage>
          . Springer, (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Pleuss</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wollny</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Botterweck</surname>
          </string-name>
          , G.:
          <article-title>Model-driven development and evolution of customized user interfaces</article-title>
          .
          <source>In: Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems</source>
          , pp.
          <fpage>13</fpage>
          -
          <lpage>22</lpage>
          . ACM, (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Harsu</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>A survey on domain engineering</article-title>
          .
          <source>Citeseer</source>
          (
          <year>2002</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Álvarez</surname>
            ,
            <given-names>J.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Evans</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sammut</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Mapping between levels in the metamodel architecture</article-title>
          .
          <source>In: International Conference on the Unified Modeling Language</source>
          , pp.
          <fpage>34</fpage>
          -
          <lpage>46</lpage>
          . Springer, (
          <year>2001</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Plaisant</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>The challenge of information visualization evaluation</article-title>
          .
          <source>In: Proceedings of the working conference on Advanced visual interfaces</source>
          , pp.
          <fpage>109</fpage>
          -
          <lpage>116</lpage>
          . ACM, (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Sarikaya</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Correll</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bartram</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tory</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fisher</surname>
          </string-name>
          , D.:
          <source>What Do We Talk About When We Talk About Dashboards? IEEE Transactions on Visualization Computer Graphics</source>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Gotz</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wen</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          :
          <article-title>Behavior-driven visualization recommendation</article-title>
          .
          <source>In: Proceedings of the 14th international conference on Intelligent user interfaces</source>
          , pp.
          <fpage>315</fpage>
          -
          <lpage>324</lpage>
          . ACM, (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Amar</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Eagan</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Stasko</surname>
          </string-name>
          , J.:
          <article-title>Low-level components of analytic activity in information visualization</article-title>
          .
          <source>In: IEEE Symposium on Information Visualization</source>
          ,
          <year>2005</year>
          .
          <source>INFOVIS</source>
          <year>2005</year>
          ., pp.
          <fpage>111</fpage>
          -
          <lpage>117</lpage>
          . IEEE, (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Gotz</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zhou</surname>
            ,
            <given-names>M.X.</given-names>
          </string-name>
          :
          <article-title>Characterizing users' visual analytic activity for insight provenance</article-title>
          .
          <source>Information Visualization</source>
          <volume>8</volume>
          ,
          <fpage>42</fpage>
          -
          <lpage>55</lpage>
          (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Saket</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Endert</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Demiralp</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Task-based effectiveness of basic visualizations</article-title>
          .
          <source>IEEE transactions on visualization and computer graphics</source>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>Börner</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Maltese</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Balliet</surname>
            ,
            <given-names>R.N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Heimlich</surname>
          </string-name>
          , J.:
          <article-title>Investigating aspects of data visualization literacy using 20 information visualizations and 273 science museum visitors</article-title>
          .
          <source>Information Visualization</source>
          <volume>15</volume>
          ,
          <fpage>198</fpage>
          -
          <lpage>213</lpage>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Maltese</surname>
            ,
            <given-names>A.V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Harsh</surname>
            ,
            <given-names>J.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Svetina</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Data visualization literacy: Investigating data interpretation along the novice-expert continuum</article-title>
          .
          <source>Journal of College Science Teaching</source>
          <volume>45</volume>
          ,
          <fpage>84</fpage>
          -
          <lpage>90</lpage>
          (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <surname>Lee</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kim</surname>
          </string-name>
          , S.-H.,
          <string-name>
            <surname>Kwon</surname>
            ,
            <given-names>B.C.</given-names>
          </string-name>
          :
          <article-title>Vlat: Development of a visualization literacy assessment test</article-title>
          .
          <source>IEEE transactions on visualization and computer graphics 23</source>
          ,
          <fpage>551</fpage>
          -
          <lpage>560</lpage>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          27.
          <string-name>
            <surname>Boy</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rensink</surname>
            ,
            <given-names>R.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bertini</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fekete</surname>
          </string-name>
          , J.
          <string-name>
            <surname>-D.</surname>
          </string-name>
          :
          <article-title>A principled way of assessing visualization literacy</article-title>
          .
          <source>IEEE transactions on visualization and computer graphics 20</source>
          ,
          <fpage>1963</fpage>
          -
          <lpage>1972</lpage>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          28.
          <string-name>
            <surname>Perez</surname>
            ,
            <given-names>C.C.</given-names>
          </string-name>
          :
          <article-title>Invisible Women: Exposing Data Bias in a World Designed for Men</article-title>
          .
          <source>Random House</source>
          (
          <year>2019</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          29.
          <string-name>
            <surname>Valdez</surname>
            ,
            <given-names>A.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ziefle</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sedlmair</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>A framework for studying biases in visualization research</article-title>
          . (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          30.
          <string-name>
            <surname>Bedek</surname>
            ,
            <given-names>M.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nussbaumer</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Huszar</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Albert</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Methods for Discovering Cognitive Biases in a Visual Analytics Environment</article-title>
          .
          <source>Cognitive Biases in Visualizations</source>
          , pp.
          <fpage>61</fpage>
          -
          <lpage>73</lpage>
          . Springer (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          31.
          <string-name>
            <surname>Schwendimann</surname>
            ,
            <given-names>B.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rodriguez-Triana</surname>
            ,
            <given-names>M.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vozniuk</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Prieto</surname>
            ,
            <given-names>L.P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Boroujeni</surname>
            ,
            <given-names>M.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Holzer</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gillet</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dillenbourg</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Perceiving learning at a glance: A systematic literature review of learning dashboard research</article-title>
          .
          <source>IEEE Transactions on Learning Technologies</source>
          <volume>10</volume>
          ,
          <fpage>30</fpage>
          -
          <lpage>41</lpage>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>