Predicting Implementation from Organizational Readiness to Change Principal Investigators: Dean Blevins, PhD & Christian Helfrich, PhD, MPH Background: Initial states of organizational readiness-e.g., attitudes about change, leadership support-are critical to successfully implementing evidence-based practices. However, those who have assessed readiness have mostly relied on measures that have little or no published psychometric properties. Based on the widely adopted model of organizational change, Promoting Action on Research Implementation in Health Services (PARiHS), VA researchers have developed the Organizational Readiness to Change Assessment (ORCA). Preliminary reliability estimates and construct validity of the ORCA have been positive; however, it is unclear whether these results can be replicated or if the instrument possesses criterion and predictive validities. Objectives: Our immediate goal is to rigorously assess the psychometric properties of the ORCA by extending knowledge about the ORCA's : 1) inter-rater and internal consistency reliabilities; 2) content validity; and 3) criterion validity, including predictive, concurrent, convergent, and discriminant validities. Our secondary aims are to 4) assess the sensitivity of the ORCA to changes in readiness over time, and 5) explore options for a short form of the survey. Methods: This study will be include cross-sectional and longitudinal, secondary data from four partner projects. Each project tests an intervention to improve the implementation of an evidence-based practice. Each will field the ORCA (n = 255 respondents; 50 facilities), and assess the degree to which the practice was implemented. Three partner studies will also field measures of job satisfaction and organizational stress for convergent and discriminant validity analyses (n = 229 respondents; 32 facilities). Using data from all four partner projects, inter-rater reliability will be assessed with intra-class correlations. Internal-consistency reliability will be evaluated with Cronbach's alpha. Predictive and concurrent validities will be conducted using hierarchical linear modeling and multivariate regression, respectively. Predictive validity, the sample will be three partner projects with baseline measures for all three ORCA scales (185 respondents from 43 facilities); the outcome is an implementation effect size and the independent variables are the ORCA scales. For concurrent validity, the outcomes are the ORCA scales and the independent variable is the intervention vs. control arms across partner projects. Convergent and discriminant validities will use data from three partner projects and examine associations between measures of different aspects of job satisfaction (e.g., leadership, salary) and organizational stress. Content validity will be assessed using a modified Delphi technique conducted with an expert panel. The expert panel will rate the ORCA items for content validity and readability, and help develop a short form of the survey. Confirmatory factor analysis will also be used to assess the factor structure of the individual survey scales.
(Statement of Relevance) Health care providers repeatedly fail to implement evidence-based practices shown to improve quality and efficiency of care;there is often a lag of 15-20 years from the time a new practice has evidence of its effectiveness to the time it is widely implemented. Differences in organizational readiness, e.g., attitudes about change, leadership support, and slack resources, may help explain this failure. Better addressing organizational readiness as part of implementation planning may allow us to can improve the odds that a new practice will be effectively implemented. However, we currently lack reliable, valid measures of organizational readiness. Although many measures have been published, few have undergone rigorous validation, and of these, all have one or more critical problems. Researchers with the Ischemic Heart Disease Quality Enhancement Research Initiative (IHD QUERI) in Seattle have developed an instrument, the Organizational Readiness to Change Assessment (ORCA), as an implementation tool to assist with evidence-based practice change.
