Clinical studies involve typically surveys of patients to gather data. Coaching aims at identifying the least engaged patients to enhance the response rate. To support the coaches we have developed a platform that enables ongoing data pipelines to elicit insights during the study from these patient-generated data. The most challenging aspect derives from the attempt of including the content of the individual responses. Exploiting this information occurs to the detriment of the possibility to generalize the results, preventing any kind of inference across diferent studies. The framework designs two AI solutions for the analysis of survey data. The first method is a forecast of the response rate. The second consists in an implementation of a descriptive module that generates an user-friendly interface displaying patterns in the response rate behaviour. The coach leverages this visualization tool especially if the predictive module falls short, taking advantage of a tidy view for optimizing the patient engagement strategies during the study. A first approach to the stated problem and an innovative employment of the tree technique are the main contributions of this paper.
For medical research, surveys are the standard to collect data from patients participating
to clinical studies such as trials or observational studies. A survey research is essentially a
collection of information from a sample of individuals through their responses to questions [
In this paper we design the implementation of a tool in a digital platform for gathering and processing survey data. This tool is developed to meet the specific needs of the coach, the enduser of the platform that is an health worker committed to patients monitoring. The coaching activity aims at increasing response rate by contacting the patients to maximize the response rate. The objective of the work is to elicit from survey data relevant insights for the coaching.
To achieve that, we propose a data-driven approach with the implementation of two independent AI solutions. The first functionality enables a forecast of the response rate levels allowing to straightforwardly identify the least engaged patients. Alternatively, the coach could resort to a second module that ofers a tidy view of the relevant data afecting mostly the response rate behaviour. The coach, a domain expert with no knowledge about data, identifies supported by an highly interpretable interface which patients to contact.
The novelty of the work consists in a new approach to increase the response rate by eliciting insights from survey data while the study is still running and by including the responses. Furthermore, to our knowledge there are no works which exploit the tree technique in such a manner.
The remainder of the paper is organized as follows: Section 2 describes the studies features and the relative implications for the analysis; Section 3 deals with the system managing the data survey platform; Sections 4 and 5 introduce the methods designed to solve the problem resorting to the gradient boosting algorithm for the predictive task and to an innovative implementation of the tree technique for the descriptive interface; Section 6 describes the results of the experiments and the challenges involved in the field-test of the method; in Section 7 are argued the main contributions; lastly, Section 8 draws the conclusions and links them to future improvements.
In the introduction are mentioned two features colliding with predictive task: dataset with a
limited sample size and a high number of explanatory variables. The large number of explanatory
variables derives from considering each question as an input feature in the framework and,
hence, the forecast could sufer from the curse of dimensionality. Whereas, we can identify at
least two main reasons that pose the framework in a scenario of narrow data: small surveys are
frequent and their processing occurs in the ongoing period. Typically, survey management is
expensive and it is common for small and medium research group to aim at keeping their size
as small as possible due to limited resources [
In the analyzed studies those aspects are actually present (i) restricted amount of patients (ranges from 30 to 100 patients) (ii) large amount of questions for each study (the count can reach 100 questions) (iii) time span of studies difers (lengths of the studies range from one month to one year).
Some may argue that combining studies could overcome the limited size of the dataset. However, the description of the studies suggests that they difer significantly preventing us to join them. Additionally, the lack of consistency in the questions across the studies precludes us from combining the studies and meanwhile exploiting the responses.
Patients PostgreSQL Gathering Extraction 1. Pre-processing 2. Processing End-users Visualization Prediction
This module organizes data so that the processing phase runs optimally; the actions performed
in the module pertain mostly to the structure or the organization of the dataset. Granularity
refers to the number of entities represented by a record in a dataset [
The predictive module gives in output the response rate values of the future cycle, fitted
with a Gradient Boosting algorithm. This particular model belongs to a highly efective and
extensively used family of machine-learning techniques [
The module contains an evaluation modality as well as the predictive one. In the field of predictive modeling, it is crucial to evaluate the dependability of the fitted values. The assessment of the model’s performance on the current data provides an indication of its ability to perform well in future predictions. The evaluation modality generates two distinct metrics the score and the progressive recall. The score evaluates the goodness of fit using a re-examined coeficient of determination, which is measured on a scale from -1 to 1. When the score is 0, the model performance is equivalent to the baseline model, and, hence, if the score is negative, the model performance falls shorter than fitting all values with the mean. The progressive recall measures the percentage of patients correctly identified by the predictive model as part of the least responding patients. 5. Descriptive Trees: a Method to Discriminate Patient
Behaviour
Alternatively to the predictive module, the coach could resort to a tree interface displaying
descriptive rules for discriminating patient behaviour based on their response rate. The task
consists in clustering patients not only by response rate, but also by the factors that have the
greatest impact on it. This visualization may suggest the coach which are the most efective
actions for increasing patient engagement. To perform this task a decision tree method is
adopted for two main reasons. Firstly, it can handle numerous explanatory variables and
perform the task of variable selection [
The work exploits the tree technique in an innovative way since we are not performing any predictive task. In particular, we do not split data in training and test set but we just run the model on the whole dataset. This choice follows the intuition that the method needs just describing/summarising data.
It was possible to process data indiferently either with a regression tree or with a decision tree. A decision tree is adopted since it requires to discretize the dependent variable into a qualitative one, making it more intuitive with the definition of two categories of patients. The mean of the response rate is the default choice to set the threshold to categorize the response rate. However the coach may customize the tree by setting a diferent threshold according to her budget. For what concerns (Figure 2) (i) the two patient categories are distinguished by diferent colours (i.e., the most engaged patients are the green) (ii) it is reported the text of the input feature that generates the split. 6. Experiments and Open Challenges of Descriptive Trees
Field-Test
The framework has been tested on 5 diferent studies. Initially, diferent set of explanatory variables are tested to evaluate how the predictor behaves. The results indicate that all available explanatory variables should be included, suggesting that the algorithm can efectively handle the curse of dimensionality. The metrics of each study are generated at an intermediate and a conclusive time point and collected in Table 1.
In general, the performance improves according to the length of the study and to its progress. This result suggests that the size of the study is a critical point and lack of data represents an issue.
The evaluation of the efectiveness of the descriptive tree method is yet to be tested through a ifeld-test in an ongoing study, and thus, there are no actual validations available. Due to the experimental nature of the method, a well-structured plan is required for the implementation of its field-test. The first challenge is the identification of appropriate metrics to measure the efectiveness of the method. One possible approach could involve soliciting feedback from Study Study1 Study1 Study2 Study2 Study3 Study3 Study4 Study4 Study5 Study5
Time Intermediate End Intermediate End Intermediate End Intermediate End Intermediate End coaches who have utilized this method, such as through a survey. Secondly, further improvements are necessary to ensure that the resulting clusters are meaningful. The meaningfulness of a cluster depends on the relevance of the insights it provides, where the explanatory variables involved in the splits can suggest the reasons for the cluster behavior and guide the appropriate actions. Thus, the inclusion of domain-specific knowledge or expert input could address the interpretability of the visualization. For instance, the coach could undertake a feature selection task by retaining the most interesting questions related to the treatment. Alternatively, the coach could re-weight the features to give more importance to the most promising ones. The third point, which is closely related to the second, is the balance between a user-friendly tool and its efectiveness. Simplifying the tool usability occurs to the detriment of its capability to provide relevant summaries. For example, achieving the desired level of usability requires careful consideration of the design parameters with particular regard to their number and complexity of the settings. The issue of balancing the size and the accuracy of the displayed tree is an additional example of this challenge. In general, a cost-complexity parameter controls this trade-of, which, in a predictive scenario, trees are pruned according to preventing the tree from overfitting the data. In this framework the parameter regulation depends on other criteria such as the screen size of the coach’s smart device or the lack of enough explanatory power of the data itself.
In the literature there is a plethora of research focusing on the processing phase when the studies
have been concluded yet [
This framework aims at eliciting insights from survey data to increase the response rate. In addition to its predictive functionality, the framework includes a descriptive module which ofers an alternative solution when the predictive task falls short. The descriptive module employs a highly interpretable tree visualization that groups patient based on their response rate behaviour. However, further improvements and a field-test are necessary to validate the efectiveness of the descriptive trees, given their experimental nature.
In the future, mapping at least the more general questions across diferent studies and standardizing the size and length of each study could enable the aggregation of survey data from multiple sources. This approach may facilitate the identification of patterns to infer across diferent studies.