Drs. Brookman-Frazee and Lau (R01MH112536-01 Fidelity measures are largely impractical for assessing psychotherapy quality in usual care (UC) because they: (1) often involve labor-intensive observational methods, (2) measure many psychotherapy features with unknown validity for predicting care outcomes, and (3) are almost exclusively EBP-specific, thereby limiting their utility in system reform efforts that involve implementing multiple EBPs to meet the range of patients presenting for services.1 Therapist reports of EBP delivery represent a pragmatic approach to QA. Despite concerns about the validity of therapist reports,2 emerging findings suggest that therapist reports of EBP delivery align with other treatment integrity measures when EBPs are delivered and predict variance in client-level outcomes, supporting their potential use as a QA strategy in the context of EBP implementation in UC.3,4 We propose to build upon our current 4KEEPS R01 study (R01MH100134) to develop a pragmatic QA of ongoing multiple EBP delivery in children?s mental health services. The purpose of this ongoing study is to characterize multiple EBP sustainment and identify organizational and therapist characteristics that predict sustainment outcomes within the nation?s largest public mental health system.5 Our community partner, the Los Angeles County Department of Mental Health (LACDMH), is the nation?s largest public mental health (MH) authority, and is a leader in implementing multiple EBPs in system driven reform. However, LACDMH leaders are struggling with lack of tools to feasibly assess ongoing quality of multiple EBPs. EBP concordant care, a primary sustainment outcome referring to the degree to which a therapist?s practice resembles the essential strategies one would expect within an EBP protocol for a given MH focus, offers a potential solution to this problem. We depart from protocol-specific fidelity measures that exclusively assess features of one specific treatment by assessing elements of sets or families of EBP?s for target MH conditions. We have developed the EBP Concordant Care Assessment (ECCA) to provide a common metric to assess therapist delivery of EBPs for major child MH targets. Item content was drawn from established tools used to characterize UC therapist practice6-8 and EBP developers? report of core strategies essential to treatment integrity. Our preliminary data support the promising validity of the innovative ECCA to assess therapy content and highlight essential refinement needed to improve assessment of therapeutic techniques. It is also essential to isolate which content and technique strategies function as quality indicators predictive of client outcomes. In the proposed study, we will use a community-partnered approach to build upon the initial ECCA (?ECCA-??) for pragmatic QA in UC. We will: (1) modify instrumentation to increase concordance between therapist report and gold standard observational ratings (?ECCA-??), (2) identify a subset of ECCA-? items that predict client-level outcomes, and (3) provide guidelines for administration and interpretation. The results will inform future research aimed to develop and test an ECCA Toolkit for use by agencies and systems to monitor and improve the quality of ongoing multiple EBP delivery.
Building upon the initial development of the EBP Concordant Care Assessment (ECCA) we will develop and validate a toolkit can be used in usual care settings to assess the quality of implementation of EBPs for major child MH targets (anxiety, depression, conduct problems, and trauma). The study aims are to optimize concordance of ECCA therapist reports with gold standard observational ratings, (2) identify a subset of quality indicator items that predict client-level outcomes, and (3) provide guidelines for administration and interpretation.
