<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Personal Electronic Record Systems in Psychiatric Care; a Call for Integration</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Jakob Valen</string-name>
          <email>jakobv@ntnu.no</email>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Dept. of Psychology, Norwegian University of Science and Technology</institution>
          ,
          <addr-line>7491 Trondheim</addr-line>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>St Olavs University Hospital</institution>
          ,
          <addr-line>P O Box 3008 Lade, 7441 Trondheim</addr-line>
        </aff>
      </contrib-group>
      <abstract>
        <p>This paper proposes how research based developments are needed to use current computer technology to optimize the psychological/ psychiatric service. Research areas that can contribute to an optimized and integrated service within a personal electronic patient record are: intelligent assessment systems with feedback routines, matching and booking functionality, and evidence based intervention teaching for therapists. The proposed benefits of this research integration is a positive spiral of improved patient satisfaction, reduced cost of treatment, and a basis for better research. An integrative development program is proposed.</p>
      </abstract>
      <kwd-group>
        <kwd>psychotherapy outcome</kwd>
        <kwd>matching</kwd>
        <kwd>booking</kwd>
        <kwd>psychotherapy research</kwd>
        <kwd>cost efficiency</kwd>
        <kwd>feedback</kwd>
        <kwd>monitoring</kwd>
        <kwd>quality improvement</kwd>
        <kwd>information technology</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        Patients in need of psychological care are faced with a challenging maze of how to
find the best treatment for her/ his current problem. Knowledge about diagnostic
assessment[
        <xref ref-type="bibr" rid="ref1 ref2 ref3 ref4">1-4</xref>
        ], treatment availability and the best treatment[
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] for a given problem
is distributed to various degrees in the population of primary care providers as well as
in the patient population. When the patient does get assigned to a treatment, the
matching with the patient and the treatment may be based on personally acquired
network knowledge. The procedures to in depth assess the problem as well as the
systems to document treatment response[
        <xref ref-type="bibr" rid="ref6 ref7">6, 7</xref>
        ] vary largely between providers. Details
from the patients’ previous treatment history is difficult to document in detail and
often impossible. The lack of systematic assessment procedures can make it random
whether the patient is offered the optimally empirically supported treatment
ingredients for the specific problem. Only 8 to 10% of American psychologists uses
computer supported assessment[
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. Finally, data for research on the efficacy and
process on the patient is limited to the data collected from when treatment started at a
treatment provider. Consequently, data from a naturalistic longitudinally study of the
patient is only possible trough a relationship to a research project set up for such
studies. With the recent scandals from researchers’ manipulation of data in mind, the
need for transparency and solid data storage can not be over emphasized.
In sum, the field of psychiatric care has large potentials for improved efficiency by
implementing a personal electronic medical record (PEMR)[
        <xref ref-type="bibr" rid="ref10 ref8 ref9">8-10</xref>
        ] witch includes 1)
systems for assessment, 2) treatment matching, and 3) empirically based teaching
systems for provider. These fields will be discussed in further details in the following.
      </p>
    </sec>
    <sec id="sec-2">
      <title>1.1 Assessment and Monitoring</title>
      <p>The primary health provider does assess the patient by ICPC-2e International
Classification of Primary Care into approximately 30 different possible psychiatric
diagnoses (in the p and z domain) before referring to a specialist. Thus, the detailed
assessment of the patient has to be done by the psychologist/ psychiatrist. The first
sessions with the specialist will therefore be used to further diagnostic assessment.
Once the problem is focused, it may be necessary to refer the patient to another
therapist who specializes in treating the present problem. In the process time goes on
before the patient gets the treatment and the patient have little control and
responsibility during the process.</p>
      <p>
        When the patient finally gets to the right treatment, there is seldom routinely
monitoring of the treatment process. Consequently, there quality of treatment is
seldom documented, and there will not be routinely actions when the patient does not
respond to the treatment or when the patient has reached the goal of treatment. These
decisions are fully left to the clinical judgment, which is of low reliability without
objective decision support [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ].
      </p>
      <p>
        The missing possibility for the patient to fully or partly drive the assessment
process online also makes it impossible to enjoy the benefits of new technology where
treatment is computer assisted [
        <xref ref-type="bibr" rid="ref12 ref13 ref14 ref15">12-15</xref>
        ].
      </p>
      <p>Most of patients’ self reported information is used for research. This information is
often collected by a researcher and used to analyze groups of patients. Researchers
and their employers often consider data from assessment forms their property and will
not easily let other have access to their data. Valuable individual longitudinal
information will not be made available to the patient when going to a different site for
later treatment. Further, data that can document the effect of a treatment may
potentially be hidden or manipulated in the care of the treatment provider.</p>
      <p>In all, there is a large potential for alleviating patients’ suffering and improving
efficiency by letting patients do their own assessment in a PEMRS.</p>
    </sec>
    <sec id="sec-3">
      <title>1.2 Treatment Matching and Booking</title>
      <p>Patients are often referred by their primary physician to a local psychiatric clinic or a
list of private practitioners. The list of private practitioners contains is not dynamic
and will therefore not be updated on availability, or specialty accreditation. Therefore
the patient is often randomly assigned to a therapist. Consequently, valuable time may
be used in searching and waiting for a good matching specialist. Current internet
technology could make this process much faster and accurate.</p>
    </sec>
    <sec id="sec-4">
      <title>1.3 Provider Supervision</title>
      <p>
        Therapists, especially in private practice are often not under supervision. No therapist
has supervision based on routine reporting by the patient. They get counseling in the
areas they request. Consequently, the patient has no control of this process unless
letters of complaint are written; witch is less likely when a patient is under much
distress. A regular monitoring of patient response can alarm supervisors to cases
where the therapist does not handle the situation [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ].
      </p>
    </sec>
    <sec id="sec-5">
      <title>1.3.1 Teaching Systems for Treatment Providers</title>
      <p>The research based knowledge is not easily accessible for the clinical practitioner.
Research articles are time consuming reading. The computer technology has
availability to present audiovisual material simple and focused provided that systems
are prepared for collaboration with objective information on the patient. To the best of
my knowledge, there is no such system available. As a result, there is no way to
ensure that the practitioner is updated on the empirically validated intervention for the
present problem.</p>
      <sec id="sec-5-1">
        <title>2 Proposed Solutions</title>
        <p>In this section I will propose some areas where already established knowledge could
be integrated into a PEMRS to improve efficiency in the treatment of psychiatric care.</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>2.1 Assessment and Monitoring</title>
      <p>
        Since the 1860’s, psychometric research has developed systems for assessing patient
characteristics. The development of solid self report instruments is a long and
laborious process, including measuring the normal variation of specific symptoms.
These instruments provide invaluable clinical guidelines for the treatment provider.
However, top level self report systems to guide the patient down to the bottom level
electronic assessment form are yet to be ready for free use[
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Future research should
integrate current assessment knowledge into PEMR to deliver patients freedom to
assess their problems in a reliable and valid way, fully in control of their own process.
      </p>
      <p>
        Quality assurance [
        <xref ref-type="bibr" rid="ref16 ref17 ref18 ref19 ref20 ref21 ref22 ref23 ref6 ref7">6, 7, 16-23</xref>
        ] in measuring the treatment response should be a
natural procedure in all clinical settings. Monitoring of treatment response provides
the ability to present important clinical feedback[
        <xref ref-type="bibr" rid="ref16 ref21 ref22 ref24 ref25 ref26">16, 21, 22, 24-27</xref>
        ] to patients as well
as the therapist. The benefits of incorporating such procedures in PEMR are therefore
obvious. In a PEMRS, the patient is in control and drives the monitoring [
        <xref ref-type="bibr" rid="ref16 ref6">6, 16, 28,
29</xref>
        ] independent of provider. The monitoring may be an incentive from the provider.
This solves the problem of transparency of therapist and treatment site efficiency,
since the patient can submit her or his data to any data collection they may want to.
      </p>
    </sec>
    <sec id="sec-7">
      <title>2.2 Treatment Matching and Booking</title>
      <p>Once a patient has focused the problems into a diagnosis, a major set of problematic
symptoms or personally anchored problematic pattern; research based knowledge can
be used to match the right treatment [30, 31]. The clinical psychiatric research have
compiled lists of treatments that are evidence based or empirically supported [32] for
specific psychiatric diagnoses. There are emerging systems for certification of
therapists within specific treatment packages. A natural place to collect information
about therapist’s location, available sessions and accreditation status would be in a
PEMR site. Consequently, the PEMR site can include booking systems for direct
booking of a therapist.</p>
    </sec>
    <sec id="sec-8">
      <title>2.3 Provider Supervision</title>
      <p>Recent research has developed algorithms for identifying patients at risk for
treatment failures. Supervision of such cases can reduce the likelihood of failures.
Studies of the treatment process have given necessary and detailed insight of the most
effective change agent. Process data can be the patients own feedback at various
intervals in the treatment, observer data, or therapists’ judgments during the
treatment[33]. A PEMRS should therefore include procedures for suggesting
empirically supported methods to most efficiently alleviate the patients suffering.</p>
    </sec>
    <sec id="sec-9">
      <title>2.3.1 Teaching Systems for Treatment Providers</title>
      <p>Process research centers in Trondheim and Bergen are collaborating with the
American Psychological Association (APA) in developing an empirically based
fundament for future teaching videos to be delivered over the internet. The crucial
point for timing of learning intervention is just before the therapist sees his patient. If
the patient can drive the assessment procedures, then the therapist can focus on being
ready with the relevant tools for the current problem. A future incorporation of the
Norwegian-APA system can base the delivery of therapist demonstrations on the
current patient’s PEMRS assessment.</p>
      <sec id="sec-9-1">
        <title>3 Discussion</title>
        <p>A focus on PEMRS research has enormous potentials for improved efficiency in
psychiatric treatments, thus potentials for large national savings. The potentials of the
PEMRS to improve patients feeling of control of their own healing also have valuable
benefits in itself.</p>
        <p>However, the process of integrating the different fields of research within the
psychiatric field into a PEMRS produces many challenges. Firstly, researchers will
need to agree on a common goal. The principles of a PEMRS may challenge
fundamental philosophy in existing research. Some assessment systems are
commercially based on being paid by the treatment provider.</p>
        <p>Secondly, state funding is essential for the development of the psychiatric part of a
PEMRS. Without active participation of the funding part, short cuts into commercial
shallow systems may undermine the integrated potential of a PEMRS. Further the
participation of psychiatric professionals with research knowledge and insight into
computer systematization is an essential ingredient in the process. Such personnel is
limited and may hamper the development.</p>
        <p>Further developments in PEMRS should include the psychiatric field and work on
getting the necessary funding to get an optimal functioning.
27. Brown, G.S. and E.R. Jones, Implementation of a feedback system in a managed
care environment: What are patients teaching us? Journal Of Clinical Psychology,
2005. 61(2): p. 187-198.
28. Percevic, R., M.J. Lambert, and H. Kordy, What is the predictive value of
responses to psychotherapy for its future course? - Empirical explorations and
consequences for outcome monitoring. Psychotherapy Research, 2006. 16(3): p.
364-373.
29. Hannan, C., et al., A lab test and algorithms for identifying clients at risk for
treatment failure. Journal of Clinical Psychology, 2005. 61(2): p. 155-163.
30. Lambert, M.J. and D.E. Barley, Research summary on the therapeutic relationship
and psychotherapy outcome, in Norcross, John C. (Ed). (2002). Psychotherapy
relationships that work: Therapist contributions and responsiveness to patients.
(pp. 17 32). London, Oxford University Press. xii, 452 pp.SEE BOOK. 2002.
31. Beutler, L.E., C. Moleiro, and H. Talebi, How practitioners can systematically use
empirical evidence in treatment selection. Journal Of Clinical Psychology, 2002.
58(10): p. 1199-1212.
32. Leichsenring, V.F. and U. Ruger, Psychotherapy and Evidence Based Medicine
(EBM) - Randomized controlled vs. naturalistic studies: Is there only one gold
standard? Zeitschrift Fur Psychosomatische Medizin Und Psychotherapie, 2004.
50(2): p. 203-217.
33. APA-Division-of-Psychotherapy. Position Paper on Funding for Psychotherapy
Research. Position Paper on Funding for Psychotherapy Research 2006 March 7,
2006 [cited 2006 March 7, 2006]; Available from:
http://www.divisionofpsychotherapy.org/FundingPositionPaper.php.</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Jager</surname>
            ,
            <given-names>R.S.</given-names>
          </string-name>
          , Computer Diagnostics - A
          <string-name>
            <surname>Survey Practical Applications Of Computerized Assessment - Theoretical Principles</surname>
          </string-name>
          And
          <string-name>
            <surname>Perspectives. European Review Of Applied Psychology-Revue Europeenne De Psychologie Appliquee</surname>
          </string-name>
          ,
          <year>1991</year>
          .
          <volume>41</volume>
          (
          <issue>4</issue>
          ): p.
          <fpage>247</fpage>
          -
          <lpage>268</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Newman</surname>
            ,
            <given-names>M.G.</given-names>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Consoli</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.B.</given-names>
            <surname>Taylor</surname>
          </string-name>
          ,
          <article-title>Computers in assessment and cognitive behavioral treatment of clinical disorders: Anxiety as a case in point</article-title>
          .
          <source>Behavior Therapy</source>
          ,
          <year>1997</year>
          .
          <volume>28</volume>
          (
          <issue>2</issue>
          ): p.
          <fpage>211</fpage>
          -
          <lpage>235</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Cawthorpe</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <article-title>An evaluation of a computer-based psychiatric assessment: Evidence for expanded use</article-title>
          .
          <source>CyberPsychology and Behavior</source>
          ,
          <year>2001</year>
          .
          <volume>4</volume>
          (
          <issue>4</issue>
          ): p.
          <fpage>503</fpage>
          -
          <lpage>510</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Horesh</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <article-title>Self-report vs. computerized measures of impulsivity as a correlate of suicidal behavior</article-title>
          .
          <source>Crisis</source>
          ,
          <year>2001</year>
          .
          <volume>22</volume>
          (
          <issue>1</issue>
          ): p.
          <fpage>27</fpage>
          -
          <lpage>31</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Leichsenring</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <article-title>Empirically supported treatments - Scientific theory and methodology in controlled versus naturalistic studies</article-title>
          .
          <source>Zeitschrift Fur Klinische Psychologie Psychiatrie Und Psychotherapie</source>
          ,
          <year>2004</year>
          .
          <volume>52</volume>
          (
          <issue>3</issue>
          ): p.
          <fpage>209</fpage>
          -
          <lpage>222</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Percevic</surname>
            , R.,
            <given-names>M.J.</given-names>
          </string-name>
          <string-name>
            <surname>Lambert</surname>
            , and
            <given-names>H.</given-names>
          </string-name>
          <string-name>
            <surname>Kordy</surname>
          </string-name>
          ,
          <article-title>Computer-supported monitoring of patient treatment response</article-title>
          .
          <source>Journal Of Clinical Psychology</source>
          ,
          <year>2004</year>
          .
          <volume>60</volume>
          (
          <issue>3</issue>
          ): p.
          <fpage>285</fpage>
          -
          <lpage>299</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Lambert</surname>
            ,
            <given-names>M.J.</given-names>
          </string-name>
          , et al.,
          <article-title>Is it time for clinicians to routinely track patient outcome? A meta-analysis</article-title>
          .
          <source>Clinical Psychology-Science And Practice</source>
          ,
          <year>2003</year>
          .
          <volume>10</volume>
          (
          <issue>3</issue>
          ): p.
          <fpage>288</fpage>
          -
          <lpage>301</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Simons</surname>
          </string-name>
          , W.W.,
          <string-name>
            <surname>K.D. Mandl</surname>
            ,
            <given-names>and I.S.</given-names>
          </string-name>
          <string-name>
            <surname>Kohane</surname>
          </string-name>
          ,
          <article-title>The PING personally controlled electronic medical record system: Technical architecture</article-title>
          .
          <source>Journal Of The American Medical Informatics Association</source>
          ,
          <year>2005</year>
          .
          <volume>12</volume>
          (
          <issue>1</issue>
          ): p.
          <fpage>47</fpage>
          -
          <lpage>54</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Riva</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , et al.,
          <source>The Personal Internetworked Notary and Guardian</source>
          .
          <source>International Journal Of Medical Informatics</source>
          ,
          <year>2001</year>
          .
          <volume>62</volume>
          (
          <issue>1</issue>
          ): p.
          <fpage>27</fpage>
          -
          <lpage>40</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Szolovits</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          , et al.,
          <source>Guardian Angel: Patient-Centered Health Information Systems. Technical Report</source>
          <year>1994</year>
          . 604(TR-
          <volume>604</volume>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Lutz</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          , et al.,
          <article-title>Empirically and clinically useful decision making in psychotherapy: Differential predictions with treatment response models</article-title>
          .
          <source>Psychological Assessment</source>
          ,
          <year>2006</year>
          .
          <volume>18</volume>
          (
          <issue>2</issue>
          ): p.
          <fpage>133</fpage>
          -
          <lpage>141</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Klein</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <given-names>J.C.</given-names>
            <surname>Richards</surname>
          </string-name>
          , and
          <string-name>
            <given-names>D.W.</given-names>
            <surname>Austin</surname>
          </string-name>
          ,
          <article-title>Efficacy of internet therapy for panic disorder</article-title>
          .
          <source>Journal of Behavior Therapy and Experimental Psychiatry</source>
          ,
          <year>2006</year>
          .
          <volume>37</volume>
          (
          <issue>3</issue>
          ): p.
          <fpage>213</fpage>
          -
          <lpage>238</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Schneider</surname>
            ,
            <given-names>A.J.</given-names>
          </string-name>
          , et al.,
          <article-title>Internet-guided self-help with or without exposure therapy for phobic and panic disorders - A randomised controlled trial</article-title>
          .
          <source>Psychotherapy And Psychosomatics</source>
          ,
          <year>2005</year>
          .
          <volume>74</volume>
          (
          <issue>3</issue>
          ): p.
          <fpage>154</fpage>
          -
          <lpage>164</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Reng</surname>
            ,
            <given-names>C.M.</given-names>
          </string-name>
          , et al.,
          <article-title>Use of computer applications to support clinical processes. An electronic letter of discharge as resource for DRG-relevant coding</article-title>
          .
          <source>Medizinische Klinik</source>
          ,
          <year>2004</year>
          .
          <volume>99</volume>
          (
          <issue>9</issue>
          ): p.
          <fpage>548</fpage>
          -
          <lpage>+</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Cavanagh</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          and
          <string-name>
            <given-names>D.A.</given-names>
            <surname>Shapiro</surname>
          </string-name>
          ,
          <article-title>Computer treatment for common mental health problems</article-title>
          .
          <source>Journal Of Clinical Psychology</source>
          ,
          <year>2004</year>
          .
          <volume>60</volume>
          (
          <issue>3</issue>
          ): p.
          <fpage>239</fpage>
          -
          <lpage>251</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Lambert</surname>
            ,
            <given-names>M.J.</given-names>
          </string-name>
          , et al.,
          <article-title>Providing feedback to psychotherapists on their patients' progress: Clinical results and practice suggestions</article-title>
          .
          <source>Journal Of Clinical Psychology</source>
          ,
          <year>2005</year>
          .
          <volume>61</volume>
          (
          <issue>2</issue>
          ): p.
          <fpage>165</fpage>
          -
          <lpage>174</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Lutz</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          , et al.,
          <article-title>The Heidelberg Model of Active Internal Quality Assurance in inpatient psychotherapy</article-title>
          .
          <source>Psychotherapeut</source>
          ,
          <year>1996</year>
          .
          <volume>41</volume>
          (
          <issue>1</issue>
          ): p.
          <fpage>25</fpage>
          -
          <lpage>35</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Lambert</surname>
            ,
            <given-names>M.J. and E.J.</given-names>
          </string-name>
          <string-name>
            <surname>Hawkins</surname>
          </string-name>
          ,
          <article-title>Measuring outcome in professional practice: Considerations in selecting and using brief outcome instruments</article-title>
          .
          <source>Professional Psychology-Research And Practice</source>
          ,
          <year>2004</year>
          .
          <volume>35</volume>
          (
          <issue>5</issue>
          ): p.
          <fpage>492</fpage>
          -
          <lpage>499</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Lutz</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          ,
          <article-title>Patient-focused psychotherapy research and individual treatment progress as scientific groundwork for an empirically based clinical practice</article-title>
          .
          <source>Psychotherapy Research</source>
          ,
          <year>2002</year>
          .
          <volume>12</volume>
          (
          <issue>3</issue>
          ): p.
          <fpage>251</fpage>
          -
          <lpage>272</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Beutler</surname>
            ,
            <given-names>L.E.</given-names>
          </string-name>
          ,
          <article-title>Comparisons among quality assurance systems: From outcome assessment to clinical utility</article-title>
          .
          <source>Journal Of Consulting And Clinical Psychology</source>
          ,
          <year>2001</year>
          .
          <volume>69</volume>
          (
          <issue>2</issue>
          ): p.
          <fpage>197</fpage>
          -
          <lpage>204</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Lutz</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          , et al.,
          <article-title>Evaluation and feedback of psychotherapeutic progress in psychotherapy</article-title>
          .
          <source>Verhaltenstherapie</source>
          ,
          <year>2005</year>
          .
          <volume>15</volume>
          (
          <issue>3</issue>
          ): p.
          <fpage>168</fpage>
          -
          <lpage>175</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Lambert</surname>
            ,
            <given-names>M.J.</given-names>
          </string-name>
          ,
          <article-title>Emerging methods for providing clinicians with timely feedback on treatment effectiveness: An introduction</article-title>
          .
          <source>Journal of Clinical Psychology</source>
          ,
          <year>2005</year>
          .
          <volume>61</volume>
          (
          <issue>2</issue>
          ): p.
          <fpage>141</fpage>
          -
          <lpage>144</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Lutz</surname>
            , W.,
            <given-names>Z.</given-names>
          </string-name>
          <string-name>
            <surname>Martinovich</surname>
            , and
            <given-names>K.I.</given-names>
          </string-name>
          <string-name>
            <surname>Howard</surname>
          </string-name>
          ,
          <article-title>Prediction of the course of individual psychotherapy</article-title>
          .
          <source>Zeitschrift Fur Klinische Psychologie Und Psychotherapie</source>
          ,
          <year>2001</year>
          .
          <volume>30</volume>
          (
          <issue>2</issue>
          ): p.
          <fpage>104</fpage>
          -
          <lpage>113</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>Berking</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <given-names>U.</given-names>
            <surname>Orth</surname>
          </string-name>
          , and
          <string-name>
            <given-names>W.</given-names>
            <surname>Lutz</surname>
          </string-name>
          ,
          <article-title>How effective is systematic feedback of treatment progress to the therapist? An empirical study in a cognitivebebavioural-orieted impatient setting</article-title>
          .
          <source>Zeitschrift Fur Klinische Psychologie Und Psychotherapie</source>
          ,
          <year>2006</year>
          .
          <volume>35</volume>
          (
          <issue>1</issue>
          ): p.
          <fpage>21</fpage>
          -
          <lpage>29</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Harmon</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          , et al.,
          <article-title>Improving outcomes for poorly responding clients: The use of clinical support tools and feedback to clients</article-title>
          .
          <source>Journal of Clinical Psychology</source>
          ,
          <year>2005</year>
          .
          <volume>61</volume>
          (
          <issue>2</issue>
          ): p.
          <fpage>175</fpage>
          -
          <lpage>185</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <surname>Brown</surname>
          </string-name>
          , G.S., et al.,
          <article-title>Identifying highly effective psychotherapists in a managed care environment</article-title>
          .
          <source>American Journal Of Managed Care</source>
          ,
          <year>2005</year>
          .
          <volume>11</volume>
          (
          <issue>8</issue>
          ): p.
          <fpage>513</fpage>
          -
          <lpage>520</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>