Poor quality and impractical measurement has impeded the study of implementation barriers, facilitators and strategies necessary for promoting widespread delivery of evidence-based care and improving mental health. Measures derive research and policy/practice insights that guide subsequent decision making. However, for measures to advance science and inform practice they need to be both psychometrically strong and pragmatic. No studies have evaluated the psychometric properties of a comprehensive array of measures delineated in highly-cited, foundational implementation science frameworks such as the Consolidated Framework for Implementation Research (CFIR) 4 or the Implementation Outcomes Framework (IOF).5 Moreover, no research has attempted to advance pragmatic implementation science measures, and the only existing pragmatic criteria was generated in the absence of stakeholder involvement. Our long-term goal is to develop a comprehensive battery of reliable, valid, and pragmatic measures that researchers and stakeholders could use to advance the science and practice of implementation. The overarching objective of this application is to put forth a measurement-focused research agenda for implementation science (i.e., which constructs possess psychometrically strong and pragmatic measures, which require further development) as well as measures and methods to accomplish this work.
Three aims guide this proposal. One, establish a stakeholder-driven operationalization of pragmatic measures and develop reliable, valid rating criteria for assessing the construct. Our approach to generating pragmatic rating criteria employs mixed methods (literature review, qualitative interviews, and Q-sort and Delphi methods) to engage stakeholders. Two, develop reliable, valid, and pragmatic measures of accept- ability, appropriateness, and feasibility. We have vetted an effective development procedure to establish their reliability; substantive, discriminant content, structural, known-groups, and predictive validity; structural invariance; and sensitivity to change. Our approach to developing and testing new measures involves a series of laboratory studies to assess psychometric properties under controlled conditions and field studies to gauge psychometric and pragmatic performance under real world conditions. Three, identify CFIR and IOF-linked measures that demonstrate both psychometric and pragmatic strength. Our approach to assessing over 420 existing measures of 46 constructs builds upon our preliminary studies and accesses an established task force. The pragmatic measure rating criteria will be integrated into our established psychometric rating criteria rendering psychometric and pragmatic measure profiles. We predict that few existing measures are psycho- metrically sound and pragmatic, but that rigorous, replicable development and testing procedures will generate measures that possess these qualities. Once pragmatic, psychometrically strong measures are identified, the field can generate robust scientific knowledge about which implementation strategies work best, when, and for whom so that stakeholders can use evidence-based approaches for implementing evidence-based practices.

Public Health Relevance

This project will advance NIMH's mission of identifying 'effective implementation processes and mechanisms to increase the uptake of scientifically informed treatments' because it will yield critical information about psychometric and pragmatic qualities of a comprehensive array of existing measures used to evaluate implementation initiatives. Despite the potential for measures to derive research and policy/practice insights, th psychometric strength of existing measures is poorly understood and no one has yet to include stakeholders in determining what makes measures 'pragmatic'. This project will contribute the most comprehensive review of implementation measures to date, based on a stakeholder-driven 'pragmatic' measure rating criteria and an established psychometric rating system, as well as three new reliable, valid, and pragmatic measures of critical implementation outcomes, to facilitate the generation of robust scientific knowledge regarding implementation barriers, facilitators, and processes.

Agency
National Institute of Health (NIH)
Institute
National Institute of Mental Health (NIMH)
Type
Research Project (R01)
Project #
5R01MH106510-04
Application #
9245750
Study Section
Dissemination and Implementation Research in Health Study Section (DIRH)
Program Officer
Rupp, Agnes
Project Start
2016-08-23
Project End
2019-03-31
Budget Start
2017-04-01
Budget End
2019-03-31
Support Year
4
Fiscal Year
2017
Total Cost
Indirect Cost
Name
Kaiser Foundation Health Plan of Washington
Department
Type
DUNS #
078198520
City
Seattle
State
WA
Country
United States
Zip Code
98101
Lewis, Cara C; Stanick, Cameo; Lyon, Aaron et al. (2018) Proceedings of the Fourth Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2017: implementation mechanisms: what makes implementation work and why? part 1. Implement Sci 13:30
Stanick, Cameo F; Halko, Heather M; Dorsey, Caitlin N et al. (2018) Operationalizing the 'pragmatic' measures construct using a stakeholder feedback and a multi-method approach. BMC Health Serv Res 18:882
Lewis, Cara C; Mettert, Kayne D; Dorsey, Caitlin N et al. (2018) An updated protocol for a systematic review of implementation-related measures. Syst Rev 7:66
Birken, Sarah A; Rohweder, Catherine L; Powell, Byron J et al. (2018) T-CaST: an implementation theory comparison and selection tool. Implement Sci 13:143
Birken, Sarah A; Powell, Byron J; Shea, Christopher M et al. (2017) Criteria for selecting implementation science theories and frameworks: results from an international survey. Implement Sci 12:124
Powell, Byron J; Beidas, Rinad S; Lewis, Cara C et al. (2017) Methods to Improve the Selection and Tailoring of Implementation Strategies. J Behav Health Serv Res 44:177-194
Shea, Christopher M; Young, Tiffany L; Powell, Byron J et al. (2017) Researcher readiness for participating in community-engaged dissemination and implementation research: a conceptual framework of core competencies. Transl Behav Med 7:393-404
Weiner, Bryan J; Lewis, Cara C; Stanick, Cameo et al. (2017) Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 12:108
Powell, Byron J; Stanick, Cameo F; Halko, Heather M et al. (2017) Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping. Implement Sci 12:118
Lyon, Aaron R; Connors, Elizabeth; Jensen-Doss, Amanda et al. (2017) Intentional research design in implementation science: implications for the use of nomothetic and idiographic assessment. Transl Behav Med 7:567-580

Showing the most recent 10 out of 15 publications