UWM-Adverse Drug Events Corpus (UWM-ADEC) is an annotated corpus that has been developed from consumer drug review posts in social media. In this corpus, we identified four types of Adverse Drug Reactions (ADRs) including physiological, psychological, cognitive, and functional problems. Additionally, we mapped the ADRs to corresponding concepts in Unified medical language Systems (UMLS). The quality of the corpus was measured using well-defined guidelines, double coding, high inter-annotator agreement, and final reviews by pharmacists and clinical terminologists. This corpus is a valuable source for research in the area of text mining and machine learning for ADRs identifications from consumer health posts, specifically for psychiatric medications.
The lexicon-based approach for name entity recognition in the area of pharmacovigilance currently dominates other
methods of health entity extraction in consumer health posts. The lexicons have been mostly developed by
combining standard medical vocabularies including COSTART (that was developed by the FDA for coding
postmarketing ADR reports and was later replaced by MedDRA), the FDA Adverse Event Reporting System (FAERS),
MedEffect (Canadian Adverse Reaction and Medical Device Problem Reporting database), SIDER (which has been
developed based on resources published by public sources, mainly the FDA such as structured product labeling
(SPL)), the Drug Bank Database, and the European agency for the Evaluation of Medical Product (EMEA). The
lexicons were mainly built on clinical trial findings and clinicians’ reports, which often have low coverage of
colloquial expressions available in consumer health posts. To address this problem, pharmacovigilance studies have
used a few approaches mostly focused on augmentation of the standard medical lexicons by embedding Consumer
Health Vocabularies (CHV). CHV was developed mainly with the purpose of covering colloquial expression of
health professional vocabularies
Pharmacovigilance studies that focused on identifying ADRs from consumer healthcare forums mostly attributed systems errors to misspelling, colloquial expression of ADRs, use of non-standard terms, and high variability of semantic representations of a specific ADR in health posts. In addition, augmentation of the standard lexicons with CHV did not improve the system’s recall significantly, indicating that the CHV is not rich in colloquial expressions of ADRs. Therefore, there is a need for an annotated corpus that not only clarifies the text segments of health posts for the presence of specific information, such as ADRs, but also fills the gap between patient and clinician terminologies by mapping colloquial expressions to standard medical terminologies.
In line with this need,
The UWM-ADEC corpus is specifically developed for identifying ADRs associated with psychiatric medications. Although these medications have shown substantial evidence of effectiveness in treatment of mental illness such as depression and anxiety, they are associated with significant numbers of physiological, psychological, and cognitive ADRs unique to these types of medications. We built UWM-ADEC on drug reviews from “askapatient.com” for two classes of psychiatric medications including SSRI (Selective Serotonin Reuptake Inhibitor) and SNRI (Serotonin–norepinephrine reuptake inhibitor). In addition, we identified functional problems associated with drugs’ adverse effects, such as limitations in daily functioning and social activities from the drug reviews. Identifying druginduced functional problems was not previously identified in CADEC and the Twitter corpus. Functional problems can result in patient non-adherence behavior, and therefore may lead to an increased risk of illness relapse, emergency room visits and hospitalizations.
UWM-ADEC can be used for text mining systems and machine learning systems, specifically for psychiatric medication pharmocovigilance and hypothesis testing related to the impact of the ADRs on attitude, discontinuation, and other medical entities.
We examined data from an Online Message Board (OMB) “askapatient.com” that compiles uncensored user comments on the effects of taking different types of medication from people with a range of clinical diagnoses. In this OMB, patients can record their experience with a medication by filling out a form for a medication brand name. This form is composed of eight fields including rating, reason for prescription, side-effects, comments, gender, age, duration/dosage, and date of posting the review. Patients can rate their satisfaction with drugs ranging from 1 to 5, where 1 presents the least satisfaction and 5 presents the highest satisfaction. Patients are instructed to report drug ADRs in the side-effect field and the details of their experience in the comment field. However, patients were noted to report various aspects of their experiences, such as drug effectiveness or perceived distress due to ADRs, in both fields. Table 1 shows an example of posts for Cymbalta in “askapatient.com”. We used drug review posts in “askapatients.com” to collect information for four psychiatric medications: Sertraline (brand name: Zoloft) and Escitalopram (brand name: Lexapro) from Selective Serotonin Reuptake Inhibitor (SSRI) Class and venlafaxine (brand name: Effexor XR) and duloxetine (brand name: Cymbalta) from Serotonin Norepinephrine Reuptake Inhibitor (SNRI) Class. These four drugs have been primarily prescribed for depression and mood disorders. According to a dataset from Symphony Health Solutions, these medications had the highest prescription rates in 2012. Because this healthcare forum does not have application program interface (API), we designed a web-crawler to collect information from the OMB. Since there is an option for filtering drug reviews for a specific drug, we could collect the data without requiring further effort. All the data in askapatient.com is anonymous and publicly available. Therefore, we did not seek any IRB approval for the data collection phase.
We randomly selected 892 posts from a healthcare forum called “askapatient.com”. Table 2 shows demographic information of the whole sample. The gender proportion in the sample for female is significantly higher than male for both classes of drugs. Age range of the reviewer is 14-83 years old with the average of 37, and median of 35; implying that patients less than 40 are more likely to report their experience with drugs. Duration of drug usage ranged from 1 day to 20 years with an average of 18 months and median of 5 months, indicating that the duration of usage is highly skewed due to the effect of outliers. Posting reviews as soon as 1 day of treatment may indicate patients’ high concern for potential ADRs.
controlled vocabularies. In the next sections, we explain the annotation guidelines and the annotation process.
Guidelines for ADRs identification includes the ADR definitions and rules for proper identification of entities. Table 3 includes the entity definitions and the associated rules for identification with examples. The identification rules are related to patient certainty in linking ADRs with the drug, identifying patient subjective complaints and functional problems as ADRs, as well as excluding unnecessary context such as “similes” and “metaphors” from ADRs. Identifying patient subjective complaints are important because they may reflect subtle physiological, psychological, or cognitive ADRs associated with drugs. For example, “felt like I couldn't stop moving” reflects patient restlessness, which is a sign of akathisia. Identifying functional problems in drug review posts is also significant, not only for understanding how ADRs influence the normal daily activities of patients and their interpersonal relationships, but also for estimating the indirect cost associated with the ADRs. Collecting this information also enhances clinicians' abilities to predict the impact of ADRs on patient functionality, such as limitations of daily activities, social participation, and work performance. We further categorized identified ADRs as physiological (Phys), Psychological (Psycho), Cognitive (Cogn), and functional problem (FP). 1. Certainty: If patient is not confident 1. It caused hair loss and about the association between ADRs stomach bloating (ADR), and drug, the ADR was not extracted. however I am not sure that 2. Subjective complaints: If ADR is hair loss (not ADR) is expressed as subjective complaint, it because of the drug. should be extracted with the entire 2. “It certainly erased the necessary context. anxiety, but I hardly feel 3. Functional problems: if patient human anymore (ADR).
mentions their experiences with drugs 3. I would just stay around as functional problems, such as and do nothing all day problem with daily functioning and (ADR). social activities, it should be extracted 4. Very hard to take a deep and labeled as an ADR. breath (ADR) like 4. Excluding simile and metaphor: If someone is squeezing my patient used a simile or metaphor to lungs. (Smile –non provide information about his/her necessary) feelings towards ADRs, that simile or 5. The anxiety (ADR) was metaphor should not be extracted. debilitating. I also had 5. Duplicates: Duplicate ADRs in a severe headache (ADR), sentence should be independently but the anxiety (ADR) was extracted, that is, all the occurrences worse.
of the entities are identified. 6. Anxiety is now though the 6. Qualifiers: If an ADR were associated roof (severity) with qualifiers presenting severity or • Constant (persistency) persistency of it, it needs to be bad (severity) identified. headaches.
Four annotators participated in the process of identification and extraction of the three entities explained in Table 3. In the second step, the documents were divided into three sets and each set was reviewed by an annotator for entity identification. In order to calculate inter-annotator agreement, the entire dataset was reviewed by the fourth annotator. We did not extract general mentions of entities, such as “side-effects” in the sentences. For example, in these sentences, “I really suffered from side-effects,” side effects and was not extracted.
To calculate inter-annotator agreement, we used pair-wise agreement between the annotators using the following
formula
While sentence classification and entity identification in drug review posts have significant implications for automatic systems that focus on information retrieval, translating these entities to the language of health professionals fills the gap between layperson and professional expressions of medical entities, such as ADRs. This translation may benefit the generation and testing of medical hypotheses by providing unambiguous and standard information for statistical data collection and analysis.
This translation process (terminology mapping) typically involves identifying terms used by healthcare consumers
and mapping them to their equivalent concepts available in medical standard vocabularies. This process is also
referred to as normalization in other research
Due to the different ways in which symptoms, feelings, concerns etc. are described by lay persons and medical professionals, simple matching of words will sometimes fail to capture the synonymy in meaning. For example, the consumer term “feeling sick in my stomach” is equivalent to the medical term “nausea” but no words are shared. Therefore, proper mapping of consumer terms to the concepts in the UMLS must take into account both lexical and semantic matching. Since this process sometimes involves subjective judgment, to ensure consistency in mapping, we have drawn up mapping guidelines, which were iteratively updated. These guidelines were based on insights we gained by reviewing publications including clinical trial studies targeting ADRs of the drugs specified in this study and qualitative studies investigating the themes of patient experiences with the drugs. In these publications, ADRs are often grouped into three broad areas: physiological, psychological, and cognitive, an approach which we have also adopted in our study.
In some cases, the symptom mentioned by the patient is more fine-grained than the meaning of a UMLS concept, whose meaning is more general and broader in scope. In such cases, we label the map as a “specific-to-general” map. One example is the UMLS concept “executive dysfunction”. According to our research, executive dysfunction as a cognitive ADR is associated with inability to initiate and follow processes of completing a task, such as problems with initiating a task, problems with organizing a task, or problems with switching between tasks. So for a patient complaint of “cannot follow through on simple tasks”, we map it to “executive dysfunction” as a more general concept.
Three annotators with diverse backgrounds (pharmacist, physician, and health scientist) mapped the ADRs to proper UMLS concepts based on the guidelines of mapping that we developed for this study. Annotators used the UMLS Terminology Services, UTS browser (2017) for finding proper UMLS and SNOMD-CT concepts. Example of mapping the concepts to UMLS is shown in Table 4. Corpus Statistics 1 3 2 1 1 2 7 Sen_ID
“Zombie” like
Statistics for annotation from the normalization stage were also shown in Table 6. The final normalization set contains 695 concepts from UMLS concepts, from which 61% belong to the physiological category and only 5% of the concepts are related to the functional category. We also report the two most frequently mapped concepts with their frequencies across the corpus for each category of ADRs.
While normalization of consumer health posts has significant implications for understanding pharmacological aspects of medications, it is a subjective process. We attempted to address this by developing guidelines that include underlying concepts for both patient and professional expressions of entities. But, some expressions strongly related to the life context of patients. For example, we did not map “hardly feel human anymore” to any concepts due to uncertainty of the underlying concepts associated with it. It is not clear what the patient meant with this expression: is it about the patient feeling emotionally detached, having a problem in performing daily activities, or is it about feeling detached from his mind and his body (de-realization)? There were also some cases that, while the expression of an ADR is clear and can be translated to an equivalent medical concept, there are no UMLS concepts available for it. For example, brain zap, which is known as the professional term “brain shivers”, does not have any concept in UMLS.
The size of sample is limited to 892 posts for four psychiatric medications. While this sample size is a good representative of the four most common psychiatric medications, it may not be a good representative of other consumer posts in this forum or other healthcare forums. It is also possible that a specific group of patients tend to report their experiences with drugs in this forum leading to reporting bias.
Our corpus covers sentence labeling and entities identifications for two classes of psychiatric medications, SSRI and SNRI. While limiting the dataset to a specific set of drugs enabled us to have a better understanding of conceptual models associated with layperson and professional expressions of medical entities, it may not include rare ADRs related to other classes of psychiatric medications and medications for other diseases or disorders.
The focus of patients in review posts is mostly on the selected drug. Hence, it is not clear, whether the reported adverse effects are merely caused by the drug or it is the result of interaction of the drug with other potential drugs or herbal treatment that administered by patients. Moreover, some of the ADRs for psychiatric medications, such suicidal ideation or emergency visits can happen because of patient’s overdose, this information is not available in the review posts because of the nature of these reports.
Although patient self-reported experiences is a reliable source for evaluating pharmacological effects of medications, there is still the risk of inaccurate and false information. In addition, we only identified and extracted ADRs that patients directly associated with their medications, however, there is the possibility that patients misinterpreted the symptoms of their psychiatric condition as an ADR of their psychiatric medication.
Although the entire data set is double coded, there is still the possibility that annotators did not interpret a sentence correctly and therefore assign a wrong label to it. In addition, the span of the identified entities may include less or more information than necessary. These problems affect the performance of any machine learning system trained on this corpus to identify drug effectiveness, ADRs, and drug indications in consumer health posts.
We have created a corpus of ADRs with the purpose of improving recall and precision of automatic systems designed for identifying ADRs from social media. The source of this corpus is patient reviews of psychiatric drugs in a medical forum called askpatient.com. Sentences in review posts were annotated for the presence of ADRs and span of ADRs. This corpus can benefit researchers in several areas including 1) developing and evaluating systems that automatically identify ADRs from consumer health posts, 2) developing systems that automatically map free text to UMLS, 3) Creating a structured vocabulary of layperson expressions of adverse effects and indications which can be used in electric health records (EHR) for facilitating seamless information between patients and clinicians. This can be achieved by mapping information in personal health records (PHR) to EHR systems. We are in the process of annotating withdrawal symptoms, drug indications, and effectiveness from the consumer health reports. We are also mapping the entities to corresponding terms in SNOMED-CT terminology.