We undertook a user study to evaluate whether the perceived utility of some common open data sets for family doctors would increase if they are rendered in interactive heat maps. We also investigated whether Parallel Coordinates (PC) are perceived as a convenient diagram to summarize multiple patients data; and whether making PCs interactive would increase their informativity. We interviewed 29 expert family doctors through a questionnaire to nd out that: interactive maps make health datasets be perceived as more useful, especially in regard to registers on exposure to carcinogens. PCs were found to be informative visualizations but making them interactive increases their perceived informativity. In light of this evidence, designing for better interactivity is worthy of further e orts in Human-Data Interaction research.
An evidence-based approach to IT design4 [
\evidence-based medicine" is not coincidental at all. Indeed, it constitutes a sort of natural extension of that methodological attitude to the IT-based tools that can have an impact on medical decision and care quality much similarly to the knowledge retained by doctors and nurses about clinical conditions and treatments.
As the name suggests, HDI lies at the intersection of the elds of
humancomputer interaction, data and information visualization and cognitive ergonomics.
While other elds, which focus on very large databases, data warehouses and
cooperative information systems, are concerned with the collection of increasingly
complex and vast datasets, on the improvement and management of their
information quality, HDI studies focus on how to improve the user experience of the
information services that can be enabled by these datasets, and on the
assessment of dimensions of this experience like informativity, intuitiveness, clarity,
interactivity, con dence in decision making [
In this paper we focus on health datasets and how these can positively
inform general practice of family doctors as consumers of the above datasets. In
particular, we will report the results of an exploratory user study that we aimed
at evaluating whether the perceived utility of some common open data sets and
visualization tool would increase if the data consumers were allowed to
interact with the data and explore them. This study is then to be related to similar
initiatives aimed at evaluating the value of data visualization tools in the
healthcare sector. To our knowledge the most relevant of the recent initiatives is the
\Visualizing Health" project5, developed by the University of Michigan and the
Robert Wood Johnson Foundation. In this project, which is subtitled \a scienti
cally vetted style guide for communicating health data", the involved researchers
compared tens of di erent styles by which to communicate risk-related
information in order to select the best ones for the general public according to a series
of empirical tests involving real users. The main nding, which we also back up,
is that when it comes to presenting health information, there is no single \best"
graphic, as this depends on the intended goal and, in a subtler way, on what
matters most to the intended users of the diagrams and infographics [
In this paper we focus on the context of use of health open data, from the
perspective of the health practitioners. Nowadays, an increasing number of data
are collected by health agencies and hospital institutions, often with the
necessary collaboration of the General Practitioners (GPs). Building health datasets,
keeping them up-to-date and making them open, freely accessible, and above all,
usable in everyday medical practice (i.e., the primary HDI concern) require skills
and resources, in particular the time of doctors and the money of citizens and
tax payers [
This study addresses two related research questions on open data in healthcare.
First: What health open data are perceived as more useful in general practice (if
any)? Second: Would visualizing these open data by means of interactive (that
is browsable, zoomable, etc.) maps make these data to be perceived as more
useful and usable? In addition, we also address the question whether diagrams
displaying parallel coordinates [
To these aims, the authors conceived a short questionnaire to be administered as a Computer-Aided Web Self-Interview (CAWI). This questionnaire consisted of four sections, which were to be rendered each in a di erent Web page: in the rst page, the platform was to present a list of 13 datasets that the authors had previously selected among the health open datasets made available on the DatiOpen.it6 platform, which is one of the biggest Italian open data portals collecting more than 2,000 thousand datasets. In particular the respondents were asked to choose at least one dataset (and at most three) that they could want to consult in an online and customized \visual dashboard" supporting their own professional practice.
In the second page we further selected 6 datasets for their suitability to be mapped into a georeferenced map like the one depicted in Figure 1.a, and asked the doctors to evaluate the degree of utility for their profession of each dataset on a six-value ordinal scale, from 1 (not useful at all) to 6 (very useful).
Then, we asked the doctors to imagine to be able to interact with the map, so that they could browse the map by dragging, and zooming in and out at the desired level of visualization (e.g., province, city, neighborhood if possible) and get a heat map like the one depicted in Figure 1.b. We then asked whether visualizing the above datasets on such a browsable map would make the data more useful or not (on a 5-value scale ranging from 1 \the data would be much less useful" to 5 \the data would be much more useful", passing through a middle value 3 \it wouldn't make almost any di erence").
In the third section, the questionnaire displayed a parallel coordinates diagram like the one depicted in Figure 2. We asked to indicate on a six-value ordinal scale the degree of informativity of such a way to represent patient data (from 1 not informative at all, to 6, very informative).
Then we asked to imagine to be able to interact with the diagram, so as: 1) to select one speci c line (that is a patient) and see that line highlighted while the other lines would fade slightly in the background; 2) to draw small rectangles above one or more vertical axis, in order to have the system highlight only the patient-lines intersecting the axes within that data ranges (see Figure 3). Similarly to the case of dataset utility, we asked whether the interactive diagram would be more or less informative than the static version of it.
Finally, the last page of the questionnaire contained two pro le-related questions: one regarding the work experience of the respondent; the other one regarding the perceived familiarity with IT.
We invited the potential respondents by sending an electronic mail to the family doctors registered to the NetAudit7 mailing list; this is a list of approximately 120 Italian GPs who are interested in medical audit and research initiatives regarding the assessment of performance in general practice and the continuous improvement of the related medical quality and outcome.
The questionnaire was left open for two weeks after the invitation had been sent to potential respondents. In this time lapse no reminder was sent. When we closed the survey, 29 GPs had accessed the questionnaire; 23 of them had completely lled in every item of the questionnaire in an average time of 5.9 minutes (SD=2.6 minutes). In this respondent sample, 52% have been family doctors for more than 30 years, 91% for more than 20 years; no respondent had less than 10 years of work experience.
The majority of the respondents (65%) claimed to be competent in Information Technology (IT) but neither experts nor enthusiasts. These latter ones were, respectively, approximately one third and one fth of the sample. We detected a statistically signi cant and moderately negative correlation between age and IT expertise (-.49, p=.018), as it could be expected.
As said in the previous section, respondents could choose the three most useful datasets to be visualized on a personal dashboard for their practice, out of a list of 13 data sets. The sets chosen more frequently by the respondents were: 1. Regional distribution of the percentage of elderly people8 involved in the program of Integrated Home Care (Assistenza Domiciliare Integrata ). 2. Number (and details) of the Rehabilitation facilities for the elderly (Istituti di Riabilitazione Extraospedaliera per Anziani ). 3. Inpatient care average length of stay.
These datasets were selected by approximately one third of the sample (38%, 33% and 30% respectively). The other datasets collected much fewer preferences, with \number of cases a ected by pneumococcal disease by year" that was not chosen by any respondent.
In regard to the utility of data sets to be displayed on a georeferenced map, we got statistical signi cance for two sets only, which were considered useful for general practice, namely: 1. Number of registers on exposure to carcinogens (.5 vs. .95, p=.000). 2. Hospitalization rates, by regime, patient genre and region (.26 vs. .74, p=.035).
For the other data sets we did not get statistical signi cance: however, the \number of medical prescriptions delivered" can be considered the least appreciated information set (.64 vs. .36).
We also asked whether putting all of the above datasets into an interactive map, instead of a static one, would change the perceived utility of the datasets themselves. The majority claimed mapping data would make a di erence, and that in so doing the datasets would be more useful (.83 vs. .17, p=.003, one third of the GPs even claiming that the data \would be much more useful"). The ranking of relative utility of mapping open data sets is reported in Table 1.
In regard to the last research question addressed in this short study, parallel coordinates diagrams (see Figures 2 and 3 were found to be an informative visualization to render patient data on multiple dimensions. The responses were positive for the majority (.64 vs. .46, p=.286). Furthermore, almost every GP involved in the user study claimed that making such a diagram interactive would make it even more informative (.96 vs. .04, p=.000, 60% \much more informative"). 4
The characteristics of the respondent sample, in terms of work experience and attitude towards Health IT suggest that the respondents can be considered representatives of a population of real domain experts and that the potential (positive) bias introduced by the IT enthusiasts should be low. Besides the results reported above, the questionnaire also allowed the respondents leave free text comments in order to suggest datasets and applications to be considered in further studies, give advice on potential improvements to the visual tools presented, and more generally share remarks with the research team.
Through these items, three respondents suggested to perform a similar
usability study on one of the most adopted GP governance tools in Italy, i.e.,
MilleGPG 9, by also noticing an almost total lack of this kind of studies, with
a few exceptions (e.g.,[
Moreover, we report two comments that suggested relevant improvements to the Parallel Coordinate (PC) dashboard. One GP suggested to implement PCs indicating explicitly central tendency parameters on their axes (wherever applicable), like means and medians, standard deviations and interquartile ranges, both at the level of the single GP patient set, and (if available) at the regional and national level. This doctor noticed how the bundle of lines, in virtue of their variable thickness at the intersection with the di erent coordinates, would nicely and visually represent the frequency distribution of the values within the subject population. Another GP suggested to allow the user to switch from a cross-sectional view (i.e., the default one) where each line represents the current conditions of a single patient, to a longitudinal view. In such a view, the current conditions would be represented as a highlighted line, while the bundle of the other lines would represent the di erent conditions that the same patient exhibited in the past, as these are recorded on the medical record of the GP or in the patient's health record.
We are aware of the limitations of this study. For instance, the fact that the involved doctors had to imagine the novel interaction, by looking at static pictures of how the visualization tools would change according to their actions. However, giving doctors real dashboards with which to interact and ask about their perceptions would have been problematic for the impossibility to give them a uniform training and to check remotely their interaction with the tool to evaluate. For this reason, we believe that studies like the present one, where interactivity is simulated in terms of its e ects, could be useful anyway in a preliminary phase of development, where to understand if endowing visual tools with some kind of interactivity would be worth the e ort.
In light of the evidence collected in this study, we can conclude that
designing for a better interactivity of open data platforms for general practice is
worth further e orts. From the HDI perspective, this conclusion was desirable;
although it might also seem easily predictable, nevertheless that conclusion could
not be taken for granted in a committed evidence-based approach to Health IT
design [
and design-oriented indications from both caregivers and patients to improve their experience in interacting with the next data visualization tools to come.