The ultimate objective of the proposed study is to develop a pragmatic observational treatment integrity instrument designed to support quality control efforts in community settings that strikes a balance between practicality (short, easy to use) and validity evidence (sensitivity to change, criterion, predictive). Practical instruments tht have validity evidence for use in community settings are needed to support implementation research. This is because most existing instruments of constructs central to implementation research are impractical (too long, too difficult to use) and lack validity evidence for use in community settings (sensitivity to change, criterion, predictive). However, as few pragmatic instruments exist it is not yet known if it is even possible to develop instruments that are both practical and possess strong validity evidence. The proposed study addresses this measurement gap and, in so doing, evaluates a critical question pertaining to implementation science: Can pragmatic instruments that balance validity evidence and practicality be developed? The project will accomplish two primary aims: (a) Develop the Treatment Integrity Measure- Efficient (TIME) a short observational treatment integrity instrument; and (b) Optimize the practicality and validity evidence of the TIME Scale scores.
For Aim 1, the TIME will be developed from existing observational treatment integrity instruments that have strong evidence for their valid use in efficacy and effectiveness contexts. The TIME will have three scales that represent key treatment integrity domains relevant to quality control efforts: adherence, differentiation, and competence. The validated source instruments have been used to code sessions from an efficacy trial and 2 effectiveness trials (N = 144 youth participants; N = 1521 therapy sessions). Using Rasch measurement models we will use these data to develop a short observational instrument (~12 items) from the three validated source instruments containing 87 items that has a small number of items for each of the three TIME Scales (Adherence, Differentiation, Competence).
For Aim 2, the TIME will be used to code sessions from the 3 trials (1 efficacy trial and 2 effectiveness trials) previously coded with the three source observational treatment integrity instruments. Each session will be coded for 5, 10, 15, and 20 minutes with the TIME. For each of the three TIME Scales (Adherence, Differentiation, Competence), the previous archived scores from coding the full session will be used as benchmarks to determine the shortest observation length (5, 10, 15, 20 minutes) that retains preferred reliability and validity features (Aim 2). The end product, TIME, will be a pragmatic/practical observational treatment integrity instrument designed as a quality control tool with validity evidence that is ready to be field-tested in community settings by end-users to determine the instrument's utility for implementation science.
The aim of this study is to develop a practical, short, and easy to use observational treatment integrity instrument capable of assessing the extent to which a therapist delivers cognitive behavioral therapy for youth anxiety with integrity and skill. The development of this practical instrument will help support the evaluation, implementation, and sustainability of evidence-based treatments in community settings.
