A comprehensive solution for psychological treatment and therapeutic path planning based on knowledge base and expertise sharing

Abstract

-The healthcare systems are nowadays going trough a broad standardization process so that the caregivers are guided in making decisions regarding the diagnosis and the therapeutic plannning. Differently from other fields of medicine, psychology does not base its protocol on drugs and prescription, and neither on standard surgical procedures. In this work we present a software solution to support psychologists during their decision making process, helping them both during the diagnosis and the treatment of psychological patients. The developed software is structured as a twofold application: it returns a set of clinical decision rule starting from the definitions contained on the Diagnostic and Statistical Manual of Mental Disorder, but joining the approach with a consensus-based clinical practice oriented approach. The first is realized by applying knowledge base to the research of well structured standard practices, while the second is obtained by means of expertise sharing from a network of psychologists. This latter characteristic not only allows us to share experiences and tips among expert operators, but also to integrate a scoring and evaluation system for to guide the choice of the user among comparable practices. The prototype has been tested and highly appreciated with an overall 96% of good evaluations. We believe that such a support could build the basis of an entire set of support software tools for medical and psychological diagnosis and treatment. Index Terms-component, formatting, style, styling, insert

I. INTRODUCTION

In the recent years the field of medicine and healthcare have gone trough a broad standardization process by means of therapeutic protocols and standard procedures to be applied by physicians, caregivers and healthcare operators in general. Obviously the introduction of standard protocols helps both the physician and the patients by means of a well scheduled and well experimented and refined therapeutic path.

While standardization comes with a price, since it results in a lack of customization for the developed therapy, it also presents great advantages in terms of comparability and results ©2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) testing among different patients. Moreover trough standardization the caregivers are guided in making decisions regarding the more appropriate therapeutic plan for a specific conditions, while the medical practices can be rationalized improving, in the end, the general outcome for the therapy at full advantage of the patient’s well being.

Unfortunately for several field of healthcare standardization is not easily feasible and sometimes even impossible due to the extreme variability of the human subjects and their afflictions. Differently from other fields of medicine, psychology does not base its protocol on drugs and prescription, and neither on standard surgical procedures. It follows that, while the diagnostics assessment of a psychological patient is based on well standardized tests and observations, the therapeutic plan, while grossly defined, must be adapted to the peculiar personality and characteristics of each patient. It follows that a psychological treatment and its therapeutic path plan must mainly rely on the therapist experience and knowledge of similar scenario.

On the contrary many other fields of medicine can rely on very effective clinical prediction rules in order to reduce the uncertainty inherent the medical practice by defining how to use clinical findings to make predictions [ 1 ]. Clinical prediction rules are derived from systematic clinical observations. They can help physicians identify patients who require diagnostic tests, treatment, or hospitalization [ 2 ].

We can define a clinical decision rule as a decision making tool that is derived from original research and incorporates variables from the history, physical examination, or simple tests [ 3 ]. In [ 4 ] the development process of clinical decision rules has been originally described.

On the other hand clinical decision rule must be based on evidences, when no evidence-based guideline exists, i.e. due to the extreme variability of a disease, then a consensus-based clinical practice guideline is the best option [ 5 ]. This latter is often used for psychological treatments planning, sometime also along with more orthodox clinical decision rules. Finally, it must be said that in certain cases it is uttermost difficult to draw methodology-proof clinical practice guidelines due to the extreme statistical and subjective variability of the matter at hand [ 6 ].

It follows that physicians, therapists, psychologists, and caregivers in general could obtain great advantages from specific support systems in order to be informed of the existing decision making rules. When such rules are not available it could be of great use to be made aware of the common clinical decision rules at hand. On the other hand, end especially in this latter scenario, extreme benefit is reached by communicating with other colleagues to share knowledge and feedback about their clinical practices.

In this work we present a software solution to support psychologists during their decision making process, helping them both during the diagnosis and the treatment of psychological patients. The developed software is structured as a twofold application: it returns a set of clinical decision rule starting from the definitions contained on the DSM-5, the Diagnostic and Statistical Manual of Mental Disorders [ 7 ], but joining the approach with a consensus-based clinical practice oriented approach. The first is realized by applying knowledge base to the research of well structured standard practices, while the second is obtained by means of expertise sharing from a network of psychologists. This latter characteristic not only allows us to share experiences and tips among expert operators, but also to integrate a scoring and evaluation system for to guide the choice of the user among comparable practices [ 8 ], [ 9 ].

The paper is organized as follows. After this brief introduction, in the following Section III the designed system is described in its constituent parts. In Section IV we will focus on the management of the cloud services giving further details on the resource allocation policies. Finally in Section V we will report a pilot case study and the obtained results. Finally in Section VI we will draw our conclusions.

II. RELATED WORKS

Decision making rules have been adopted since many years and with different purposes. E. g. in [ 10 ] descision making rules have been developed as a guide for hospitalization of patients presenting community-acquired pneumonia, while in [ 11 ]–[ 13 ] decision making rules are adopted to define when x-rays are needed in acute ankle injuries. In facts such a support tool is often used for trauma treatments and when diagnostical imagery is involved [ 14 ], [ 15 ].

There are many works in literature about the extraction and formulation of decision making rules. In [ 16 ] decision making rules have been extracted by means of a decision tree [ 17 ]–[ 20 ] for the diagnostic workup of patients with Meniere’s disease, vestibular schwannoma, traumatic vertigo, sudden deafness, benign paroxysmal positional vertigo, and vestibular neuritis. In [ 21 ] the authors present the results of a prospective, cross-sectional study involving patients with acute headache and demonstrate that their best bedside decision rule identified all cases of subarachnoid hemorrhage among emergency department patients presenting with new, isolated headaches. In [ 22 ] uses fuzzy decision-making rules adapted to classification problems by using the methodology of exploratory analysis followed by unification of particular decision rules into fuzzy groups. The fuzzyfication of such rules in facts can introduce an ‘useful randomness which is often the key requirement for a large point of view often required in differential diagnosis. In facts shown in [ 23 ] several ambiguous pathological conditions can lead to the possible diagnosis of suspected deep venous thrombosis, on the other hand in this case is the physicians’ judgement that takes over the decision rules. Moreover, while many authors tried to reach a universal model of diagnostic reasoning [ 24 ], ir is common knowledge that the physicians’ personal experiences, skills and diagnostic abilities hold a key role on the decision making process [ 25 ]. This latter is also often self-regulated by precise ad well as perfectible psychological mechanisms [ 26 ]. Individuals’ independent judgment as well as common and shared experiences are therefore the basis for a good diagnostic process [ 27 ], on the other hand such a process cannot neglect a minimum standardization requirement which decision-making rules help to fulfill. In facts it has been shown that diagnostic decisions can be generally improved when a decision-making rules are associated with a knowledge-based approach [ 28 ].

From this short survey of the literature it follows that while therapeutic path planning must be based on a set of codified rules, it also retain a paramount dependence from common knowledge. The first requirement can be fulfilled by using techniques such as knowledge base, while the second can be implemented by means of a customized expertise sharing support system. These two aspect have been integrated by the solution that will be explained in the following.

III. THE DEVELOPED SYSTEM

In Figure 1 a gross schema of the designed system is reported, this is composed by the following components:

I. Frontend:

Online interface

DNS handler II. Backend:

A. QoS handler

Local Search Visibility component (LSV) Secure Sockets Layer (SSL)

HTTP caching component B. Cloud VM

Apache service Local Storage Local cluster cache service Computing nodes (CN)

Storage Units (SU) C. Cloud Services

Job queque component QOS HANDLER CLOUD VM

The frontend of the system has been developed by means of the Angular JS [ 29 ], [ 30 ] framework in order to grant portability and compatibility with almost all the available hardware and software systems. In this manner there are no particular requirements to interface with the developed system, granted the ability to execute JavaScript on a browser-like application. Although a web browser would have sufficed to interface with the online service, we developed a simple adhoc application to oversimplify the interface. In this manner it is possible to avoid unnecessary distractions during the test execution. Finally a psychologist provided with the necessary credentials can log into the system to administer the test to a patient once such a test has been standardized and approved to be used. The frontend remote client only provides the interface for the final users.

In the following the backend is described. 43

HTTP caching Apache Local Cluster cache

Job Queque

DISTRIBUTED DB

LOG HANDLER

Local storage Slave

DB Main DB AMAZON S3

AWS IOT CORE

For distributed systems to properly react to peaks of requests, their adaptation activities would benefit from the estimation of the amount of requests. We implemented a a solution to adapt server-side resources on-the-fly. In order to ensure a minimum level of quality of service (QoS), even when sudden variations on the number of requests arise, a large number of (over-provisioned) resources is often used, hence incurring into relevant costs and wasted resources for a considerable time interval. We wanted to guarantee a minimum QoS level once the connection has been established by using resource allocation and adaption algorithms.

Moreover, in order to minimize bandwidth and resources usage we implemented an internal local search visibility component (LSV) and an integrated HTTP caching system. In this manner the internal research engine can spare the user a pedant manual search between the existent records. Moreover due to obvious privacy and security requirements we also implemented a Secure Sockets Layer (SLL) connection between the interface and the HTTP caching service. This latter maintains a record of headers and tags to index the knowledge base content and speed up the internal research and retrieval of the required information.

The virtual machine when allocated run an Apache servie daemon, in this manner a simple set of queries makes it possible for the user to interact with the system in order to select, extract and store data from and to the base, distributed on several storage units. Each VM retains on its local storage only a portion of the distributed databases, in fact the connection to such a VM depends on the required portion of data and the related operations. The data are then updated connecting with a a slave database which retain a potentially updated version of the data. Each VM keep trace of the required updates by means of a local cluster cache, this latter is responsible to signal the necessary global updates to the job queue scheduler. D. Cloud services

The cloud resources are allocated both for computational and provisional purposes. The details on the cloud policies are given in the following Section IV. The cloud service layer is responsible for the update and merge procedure between the temporary slave databases and the main database. The reason for this double database system is the time required for a global update and merge phase, since this latter is extremely expensive in terms of time and resources consumption. Therefore it is more suitable to live store temporary slave DB the updates, while delegate at scheduled time both the global merging and update, and the distribution of shadow temporary copies to the slave DBs.

The users sessions are logged and stored on a local storage represented by a replicated sql database. The process is handled by a LOG handler component.

IV. THE CLOUD ENVIRONMENT

For this work we took advantage of the Amazon Web Services (AWS) [ 31 ], and particularly on the AWS ECS and S3 service [ 32 ] (see Figure 2).

The resource request is provided to the cloud manager component which uses the Amazon AWS APIs to effectively request the allocation of new virtual machines. The cloud administration is up to the AWS IoT Core taking into consideration the AWS IoT rule component that determine the policies for the Amazon Kinesis Data Stream. The Amazon Kinesis Data Stream is a real-time streaming service that provides event-driven messaging and supports extended microservice architectures. This latter allows the processing requests trough the Amazon API Gateway once an admin has been logged and identified trough his credentials by the Amazon Lex component to access the Amazon S3 service.

In our system design also the database is distributed on the cloud and supported by the Amazon DynamoDB services that allows data flow by means of the Amazon DynamoDb Streams component. Data transactions and session state are encrypted at-rest and securely managed in the high-performance and scalable NoSQL datastore offered by DynamoDB. The Amazon DynamoDB Streams is also able to trigger an AWS Lambda function in order to send notifications, by means of the Amazon Pinpoint and Amazon Polly services.

An example of the system in action is provided in the following Section V.

V. A CASE STUDY

Let suppose that a psychologist wants to diagnose a patient with attention deficit hyperactivity disorder (ADHD): a mental disorder of the neurodevelopmental type [ 33 ]. When the psychologist searches the relative keywords on the system the ADHD diagnosis will be suggested and associated with the information contained within the DSM-5 diagnostic manual [ 7 ]: 1) Five or more symptoms of inattention and/or 5 symptoms of hyperactivity/impulsivity must have persisted for 6 months to a degree that is inconsistent with the developmental level and negatively impacts social and academic/occupational activities. Several symptoms (inattentive or hyperactive/impulsive) were present before the age of 12 years. 2) Several symptoms (inattentive or hyperactive/impulsive) must be present in 2 settings (eg, at home, school, or work; with friends or relatives; in other activities). 3) There is clear evidence that the symptoms interfere with or reduce the quality of social, academic, or occupational functioning. 4) Symptoms do not occur exclusively during the course of schizophrenia or another psychotic disorder, and are not better explained by another mental disorder (eg, mood disorder, anxiety disorder, dissociative disorder, personality disorder, substance intoxication, or withdrawal).

Since the system has been applied in Italy , given the known association between ADHD and impaired performance on neuropsychological tests due to the effects of ADHD symptoms on speed and performance on a non-verbal intellectual test [ 34 ], the interface also reports the official guidelines written by the Italian Society of Infantry and Adolescence NeuroPsychiatry and approved by many official associations and operator’s syndicates. Therefore it also reports that the following tests are commonly used: 1) Child Behavior CheckList (CBCL) [ 35 ] 2) Conners Rating Scales (CRS) [ 36 ] 3) Disruptive Behavior Disorder Rating Scale (DBD) [ 37 ] 4) ADHD Rating Scale IV [ 38 ] 5) SNAP-IV [ 39 ] 6) Diagnostic Interview for Children and Adolescents (DICA) [ 40 ] 7) Kiddie-Schedule for Affective Disorders and

Schizophrenia (K-SADS) [ 41 ] 8) Ravens Standard Progressive Matrices (RSPM) [ 42 ] While the system will shows these guidelines, it will also highlight a note written by a colleague psychologist that says: The DSM-5 manual extends from 7 years to 12 years the age limit for the comparisons of ADHD related symptoms.

Finally, while the system proposes an high scoring for the suggested procedures, it also shows tips and comments from other colleagues which commonly positively rated the following advice:

It can be helpful and good practice to associate the Ravens Standard Progressive Matrices with the Leiter International Performance Scale Revised (Leiter-R).

Then leading the user to use the Leiter-R test [ 43 ].

While the reported case is only an example, the system has been tested with the help of 25 psychologists that, after using the developed system, evaluated the overall performances and utility as an asset for their profession.Figure 3 shows the results of such a poll that indicated an high degree of appreciation for the developed solution with 96% of good evaluations among the overall received scores.

VI. CONCLUSION

In this work we developed a software solution to support psychologists during their decision making process, helping them both during the diagnosis and the treatment of psychological patients. The application offers a set of clinical decision rule starting from the definitions contained on the DSM-5 as well as the evaluations and advises of other colleagues. The prototype has been tested and highly appreciated with an overall 96% of good evaluations. We believe that such a support could build the basis of an entire set of support software tools for medical and psychological diagnosis and treatment.

ACKNOWLEDGMENT

This work has been supported by “Piano della Ricerca 2016/2018 - linea di intervento 2, University of Catania”.

References

[1] Brendan M Reilly and Arthur T Evans . Translating clinical research into clinical practice: impact of using prediction rules to make decisions . Annals of internal medicine , 144 ( 3 ): 201 - 209 , 2006 .

[2] John H Wasson , Harold C Sox, Raymond K Neff, and Lee Goldman . Clinical prediction rules: applications and methodological standards . New England Journal of Medicine , 313 ( 13 ): 793 - 799 , 1985 .

[3] Andreas Laupacis , Nandita Sekar , et al. Clinical prediction rules: a review and suggested modifications of methodological standards . Jama , 277 ( 6 ): 488 - 494 , 1997 .

[4] Alvan R Feinstein. Clinimetrics . Yale University Press, 1987 .

[5] Tetsuo Ashizawa , Cynthia Gagnon, William J Groh, Laurie Gutmann, Nicholas E Johnson, Giovanni Meola, Richard Moxley, Shree Pandya, Mark T Rogers, Ericka Simpson , et al. Consensus-based care recommendations for adults with myotonic dystrophy type 1 . Neurology: Clinical Practice , 8 ( 6 ): 507 - 520 , 2018 .

[6] Terrence M Shaneyfelt , Michael F Mayo-Smith , and Johann Rothwangl . Are guidelines following guidelines?: The methodological quality of clinical practice guidelines in the peer-reviewed medical literature . Jama , 281 ( 20 ): 1900 - 1905 , 1999 .

[7] American Psychiatric Association et al. Diagnostic and statistical manual of mental disorders, fifth edition (DSM-5®) . American Psychiatric Pub, 2013 .

[8] Kathleen N Lohr , Marilyn J Field , et al. Guidelines for clinical practice: from development to use . National Academies Press, 1992 .

[9] Michael D Cabana , Cynthia S Rand, Neil R Powe, Albert W Wu , Modena H Wilson, Paul-Andre C Abboud, and Haya R Rubin. Why don't physicians follow clinical practice guidelines?: A framework for improvement . Jama , 282 ( 15 ): 1458 - 1465 , 1999 .

[10] Michael J Fine , Thomas E Auble, Donald M Yealy, Barbara H Hanusa , Lisa A Weissfeld, Daniel E Singer, Christopher M Coley, Thomas J Marrie, and Wishwa N Kapoor . A prediction rule to identify low-risk patients with community-acquired pneumonia . New England journal of medicine , 336 ( 4 ): 243 - 250 , 1997 .

[11] Ian G Stiell , Gary H Greenberg, R Douglas McKnight , Rama C Nair , I McDowell, and James R Worthington. A study to develop clinical decision rules for the use of radiography in acute ankle injuries . Annals of emergency medicine , 21 ( 4 ): 384 - 390 , 1992 .

[12] Clement J McDonald and J Marc Overhage . Guidelines you can follow and can trust: an ideal and an example . Jama , 271 ( 11 ): 872 - 873 , 1994 .

[13] John H Wasson and Harold C Sox . Clinical prediction rules: have they come of age? Jama , 275 ( 8 ): 641 - 642 , 1996 .

[14] Cynthia Madden , Donald B Witzke, Arthur B Sanders, John Valente, and Mark Fritz . High-yield selection criteria for cranial computed tomography after acute trauma . Academic Emergency Medicine , 2 ( 4 ): 248 - 253 , 1995 .

[15] Ian D Graham , Ian G Stiell, Andreas Laupacis , Annette M O'Connor , and George A Wells. Emergency physicians' attitudes toward and use of clinical decision rules for radiography . Academic Emergency Medicine , 5 ( 2 ): 134 - 140 , 1998 .

[16] Erna Kentala , Ilmari Pyykko¨ , Kati Viikki , and Martti Juhola . Production of diagnostic rules from a neurotologic database with decision trees . Annals of Otology, Rhinology & Laryngology , 109 ( 2 ): 170 - 176 , 2000 .

[17] Wei-Yin Loh . Classification and regression trees . Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 1 ( 1 ): 14 - 23 , 2011 .

[18] Tin Kam Ho . Random decision forests . In Proceedings of 3rd international conference on document analysis and recognition , volume 1 , pages 278 - 282 . IEEE, 1995 .

[19] Simon Bernard , Laurent Heutte, and Sebastien Adam . On the selection of decision trees in random forests . In 2009 International Joint Conference on Neural Networks , pages 302 - 307 . IEEE, 2009 .

[20] Jehad Ali , Rehanullah Khan, Nasir Ahmad, and Imran Maqsood . Random forests and decision trees . International Journal of Computer Science Issues (IJCSI) , 9 ( 5 ): 272 , 2012 .

[21] David E Newman-Toker and Jonathan A Edlow. High-stakes diagnostic decision rules for serious disorders: the ottawa subarachnoid hemorrhage rule . JAMA , 310 ( 12 ): 1237 - 1239 , 2013 .

[22] NA Korenevskiy . Application of fuzzy logic for decision-making in medical expert systems . Biomedical Engineering , 49 ( 1 ): 46 - 49 , 2015 .

[23] Geert-Jan Geersing , Kristel J Janssen, Ruud Oudega, Henk van Weert, Henri Stoffers, Arno Hoes, Karel Moons, AMUSE Study Group, et al. Diagnostic classification in patients with suspected deep venous thrombosis: physicians' judgement or a decision rule? Br J Gen Pract , 60 ( 579 ): 742 - 748 , 2010 .

[24] Pat Croskerry . A universal model of diagnostic reasoning . Academic medicine, 84 ( 8 ): 1022 - 1028 , 2009 .

[25] Robert Wood and Albert Bandura . Impact of conceptions of ability on self-regulatory mechanisms and complex decision making . Journal of personality and social psychology , 56 ( 3 ): 407 , 1989 .

[26] John A Swets , Robyn M Dawes , and John Monahan . Psychological science can improve diagnostic decisions . Psychological science in the public interest , 1 ( 1 ): 1 - 26 , 2000 .

[27] Ralf HJM Kurvers , Stefan M Herzog, Ralph Hertwig , Jens Krause, Patricia A Carney, Andy Bogart, Giuseppe Argenziano, Iris Zalaudek, and Max Wolf . Boosting medical diagnostics by pooling independent judgments . Proceedings of the National Academy of Sciences , 113 ( 31 ): 8777 - 8782 , 2016 .

[28] Luiz Moutinho , Paulo Rita, and Shuliang Li . Strategic diagnostics and management decision making: a hybrid knowledge-based approach . Intelligent Systems in Accounting, Finance & Management: International Journal , 14 ( 3 ): 129 - 155 , 2006 .

[29] Brad Green and Shyam Seshadri. AngularJS. ” O'Reilly Media , Inc.”, 2013 .

[30] Peter Bacon Darwin and Pawel Kozlowski . AngularJS web application development . Packt Publ., 2013 .

[31] James Murty . Programming amazon web services: S3, EC2 , SQS , FPS, and SimpleDB. ” O'Reilly Media , Inc.”, 2008 .

[32] Zhiwu Xie , Yinlin Chen, Tingting Jiang, Julie Speer, Tyler Walters, Pablo A Tarazaga, and Mary Kasarda . On-demand big data analysis in digital repositories: A lightweight approach . In International Conference on Asian Digital Libraries , pages 274 - 277 . Springer, 2015 .

[33] Ariane Sroubek , Mary Kelly, and Xiaobo Li . Inattentiveness in attentiondeficit/hyperactivity disorder . Neuroscience bulletin , 29 ( 1 ): 103 - 110 , 2013 .

[34] Emily Goodwin , Gisli H Gudjonsson, Jon Fridrik Sigurdsson, and Susan Young. The impact of adhd symptoms on intelligence test achievement and speed of performance . Personality and Individual Differences , 50 ( 8 ): 1273 - 1277 , 2011 .

[35] Thomas M Achenbach and Craig Edelbrock . Child behavior checklist (cbcl ). 1983 .

[36] C Keith Conners . Conners 3 . MHS, 2008 .

[37] William E Pelham Jr , Elizabeth M Gnagy , Karen E Greenslade , and Richard Milich. Teacher ratings of dsm-iii-r symptoms for the disruptive behavior disorders . Journal of the American Academy of Child & Adolescent Psychiatry , 31 ( 2 ): 210 - 218 , 1992 .

[38] George J DuPaul , Thomas J Power, Arthur D Anastopoulos , and Robert Reid. ADHD Rating ScaleIV: Checklists, norms, and clinical interpretation . Guilford Press, 1998 .

[39] J Swanson , W Nolan , and WE Pelham . The snap-iv rating scale . Irvine, CA: University of California at Irvine, 1992 .

[40] Wendy Reich . Diagnostic interview for children and adolescents (dica) . Journal of the American Academy of Child & Adolescent Psychiatry , 39 ( 1 ): 59 - 66 , 2000 .

[41] Joan Kaufman , Boris Birmaher, David Brent, UMA Rao , Cynthia Flynn, Paula Moreci, Douglas Williamson , and Neal Ryan . Schedule for affective disorders and schizophrenia for school-age children-present and lifetime version (k-sads-pl): initial reliability and validity data . Journal of the American Academy of Child & Adolescent Psychiatry , 36 ( 7 ): 980 - 988 , 1997 .

[42] John Carlyle Raven et al. Raven's progressive matrices and vocabulary scales . Oxford pyschologists Press, 1998 .

[43] Gale H Roid and Lucy J Miller . Leiter international performance scalerevised (leiter-r) . Wood Dale , IL: Stoelting, 1997 .

[44] Nathan M Finnerty , Robert M Rodriguez , Christopher R Carpenter , Benjamin C Sun, Nik Theyyunni, Robert Ohle, Kenneth W Dodd, Elizabeth M Schoenfeld, Kendra D Elm , Jeffrey A Kline, et al. Clinical decision rules for diagnostic imaging in the emergency department: a research agenda . Academic Emergency Medicine , 22 ( 12 ): 1406 - 1416 , 2015 .

[45] Marshal F Folstein , Susan E Folstein , and Paul R McHugh. mini-mental state: a practical method for grading the cognitive state of patients for the clinician . Journal of psychiatric research , 12 ( 3 ): 189 - 198 , 1975 .

[46] Leonard R Derogatis and Rachael Unger . Symptom checklist-90-revised. The Corsini encyclopedia of psychology , pages 1 - 2 , 2010 .

[47] Starke Rosecrans Hathaway and John Charnley McKinley . Minnesota multiphasic personality inventory; manual, revised . 1951 .

[48] James N Butcher . Minnesota multiphasic personality inventory . The Corsini Encyclopedia of Psychology , pages 1 - 3 , 2010 .

[49] David Wechsler . Wechsler intelligence scale for children . 1949 .

[50] David Wechsler . Wechsler preschool and primary scale of intelligencefourth edition . The Psychological Corporation San Antonio, TX, 2012 .

[51] David Wechsler . Wechsler adult intelligence scale-fourth edition (waisiv) . San Antonio, TX: NCS Pearson, 22 : 498 , 2008 .

[52] William A Hunt. The future of diagnostic testing in clinical psychology . Journal of Clinical Psychology , 1946 .

[53] Max Gluckman . Clinical psychology: The study of personality and behavior . Routledge , 2017 .

[54] Andrew M Pomerantz. Clinical psychology: Science, practice, and culture . Sage Publications , 2016 .

[55] David Watson and Lee Anna Clark. Clinical diagnosis at the crossroads . Clinical Psychology: Science and Practice , 13 ( 3 ): 210 - 215 , 2006 .

[56] Donald A Rock , Charles E Werts, and Ronald L Flaugher. The use of analysis of covariance structures for comparing the psychometric properties of multiple variables across populations . Multivariate Behavioral Research , 13 ( 4 ): 403 - 418 , 1978 .

[57] Cyril Burt . The factorial analysis of qualitative data . British Journal of Statistical Psychology , 3 ( 3 ): 166 - 185 , 1950 .

[58] Ingwer Borg and Patrick Groenen . Modern multidimensional scaling: Theory and applications . Journal of Educational Measurement , 40 ( 3 ): 277 - 280 , 2003 .

[59] Leonard Kaufman and Peter J Rousseeuw . Finding groups in data: an introduction to cluster analysis , volume 344 . John Wiley & Sons, 2009 .