Understanding the provenance of clinical guidelines is important for both practitioners and researchers as it allows for deeper understanding of the provided recommendations and could potentially provide a basis for updating guidelines. Often such provenance is incomplete or unavailable. We describe a prototype of a multi-signal pipeline for reconstructing provenance and show preliminary results of reconstructing dependencies between documents in the context of clinical guidelines and associated documents.
'()*+,)%-.)+/+)+%% 8.()%49-9:+*9)6%
I.9;A)./%BF-% ! & " 01/.(%,21).1)% 013.*%4.-+15,% )67.4% 0-+;.%49-9:+*9)6% <2-+91=47.,9>,% 49-9:+*9)6% ?.)+/+)+% 49-9:+*9)6% B9-9:+*9)6% )A*.4A2:/4% <2-+91=47.,9>,%
>:).*91;% ! !#$%
" & $#"% 13 14 23 20 16 21 15 18 17 Cluster 1: Blood Cultures EvidenceQ||
The prototype combines several multi-modal similarity measures, in
particular text, image and metadata similarity, and aggregates them into a single
similarity score. The prototype performs the following tasks:
{ Gather all available versions and metadata of the les (e.g. authors, revisions,
timestamps) using the Dropbox Java API 1
{ Extract content (both text and images) and metadata using Apache Tika2.
{ Index the content of the les using Apache Lucene3 and LIRE [
The experimental setting consisted of a Dropbox folder containing the clinical guideline for febrile neutropenia, a set of publications referred to by the guideline and two Excel sheets that describe the references from the guideline for two research questions. The provenance of the les in the folder was manually annotated in PROV-DM, as shown in Fig. 2, in which each node represents a le and each edge a dependency of the origin le from the destination le. 1 https://www.dropbox.com/developers/reference/sdk 2 http://tika.apache.org/ 3 http://lucene.apache.org/ 4 https://github.com/lucmoreau/ProvToolbox 3 2 6 4 Cluster 1: Blood Cultures EvidenceQ|| The main document is the guideline, which has two versions in the Dropbox folder (light blue nodes in Fig. 2). All the publications are cited by the guideline and there are two Excel sheets that contain copy-paste text from the guideline. Each of these sheets details the references needed to answer a research question. In particular EvidenceQII (green nodes in Fig. 2) details the references for Question A, while EvidenceQX (blue node) focuses on Question B.
In the manual annotation, we considered citations as dependencies. Then we divided the publications in 3 clusters based on the citation network: 1) publications on blood cultures (red nodes); 2) publications on markers (purple nodes); 3) other publications (yellow nodes). 3
We ran our prototype in the previously described experimental setting with different sets of similarity measures and obtained several predictions of dependency graphs. One example can be seen in Fig. 3, in which we show the predicted dependency graph using all the implemented similarity metrics.
In order to evaluate the results we obtained, we compared the edges of
the original dependency graph and each dependency graph, predicted with our
method. The results are shown in Table 1, where the rows represent the
evaluation using di erent similarity measures. The rst row represents our baseline,
i.e. the approach described in [
We compared the di erent systems to see if there was a statistically signi cant di erence in the results. Using the T-test provided in the R statistical package, we obtained a small di erence between the baseline and system1 (p-value is 0.4216), while system2 and system3 are very di erent from the baseline (both have p-value 2.388e-06).
Similarity measures baseline: text system1: text, metadata system2: text, metadata, inverse lucene system3: text, metadata, inverse lucene, images
As we can see from Table 1, much of the structure of the original dependency graph is well-predicted. The Excel sheets depend on the guideline and the guideline is connected to all of the publications. The clusters of citations are quite recognizable.
Among the errors, some can be easily explained. For example, some papers are connected even when there is no citation, e.g. the newer clinical guideline is connected to its older version, but does not cite it. The two Excel sheets are connected because they have the same author and creation data (metadata similarity). There are some di culties in nding the appropriate temporal order, since some documents have a very di erent creation and publication date. Due to the temporal pruning that we perform, this means that several dependencies were discarded because of temporal inconsistency.
The issues with temporal ordering can be partially solved by retrieving bibliographic information on the publications. There are also other domain-speci c improvements than can be made, e.g. using the knowledge of citations. Moreover, there is the need for a better aggregation algorithm. Up to now we targeted high recall, in our future work we aim at re ning the predictions in terms of precision. Finally, we want to apply the technique on a much larger corpus of the biomedical papers and guidelines. Overall, we have shown that multimodal similarity combined with knowledge of the structure of provenance graphs is a good start towards reconstructing the provenance of clinical guidelines.
Acknowledgements This work was funded by Data2Semantics project in the Dutch national program COMMIT.