Observational studies have to rely on computationally expensive, post-experimental balancing procedures, such as matching or propensity score adjustment, to address non-response bias when missingness is at random. If the missing mechanism is non-ignorable, then computations become even more complex, relying on modeling procedures of uncertain validity. Additional to being time-intensive and costly, these methods must be applied after data collection ends, when there is no further opportunity to rectify the underlying bias. Furthermore, these methods do not always yield the correct answer and sometimes introduce their own bias into the results. Our goal is to test three methods that, if successful, would avert or at least reduce non-response bias at the design stage. Project Objectives: (1) To conduct a 3X2X2 factorial, randomized, controlled trial to examine the impact of two different types of cover letter manipulations, one at 3 levels and one at two levels, and two different after the-fact payments on non-response bias in 480 male and 480 female Veterans from Operations Enduring Freedom, Iraqi Freedom, and/or New Dawn (OEF/OIF/OND) who are also applying for VA PTSD disability benefits. (2) To use the information gleaned to identify the best method to recruit the least biased panel of OEF/OIF/OND Veterans possible for long-term evaluation, to describe key characteristics of the targeted population, and to calculate preliminary effect sizes for power estimations for a future cohort study. Project Methods: Using Veterans Benefits Administration databases, we will randomly select a gender stratified sample of 480 male and 480 female OEF/OIF/OND Veterans (960 total) who are actively applying for PTSD disability benefits. Veterans will be mailed the same survey we anticipate using in our future cohort. Prior to survey we will create a rich sampling frame of known characteristics for these 960 Veterans. Besides gender, 5 other characteristics will be studied because respondent/non-respondent differences on these characteristics are likely to be informative (non-ignorable). These are: Veterans' combat exposure, exposure to military sexual trauma, age, race, and psychiatric illness severity. Post-survey, we will also assess Veterans' claim outcome (awarded/not awarded). Two components of the survey's cover letter will be varied and crossed: First, the survey's content will be described in one of three ways. Second, Veterans will be given one of two descriptions of how their name was obtained for the study. Third and last, Veterans will be randomized to receive either a $20 or $40 after-the-fact gift card for completing the survey. Our prior research, which meshes well with Leverage-Salience Theory, suggests that these cover letter elements and payment levels may induce informative non-response bias or non-ignorable unit missingness. The study's main outcome examines unit-level missingness across each study condition and tests whether the missing mechanism is completely at random, random after controlling for other covariates, or non-ignorable. Secondary outcomes examine missingness at the item level of the survey, changes in participants' sadness and tenseness pre- and post-survey, participants' impressions of the cover letter, and participants' satisfaction with the cover letter. For the last objective, the main outcomes are proportion of Veterans successfully recruited before their claim is decided, the time in days from Veterans' filing to receiving a claim decision, and survey response rate for the least biased method identified by Objective 1. Assuming an overall survey response rate of 50%, beta = 0.80, and two-tailed alpha = 0.05, the study is powered to identify a summary total deviation between respondents and the parent population (a marker of bias) as small as 0.13 per study condition and a difference in summary total deviations across study conditions of approximately 0.03, a small effect (such that Cohen's d = 0.25).
Current methods for addressing non-response bias in observational studies are often expensive and time consuming. Sometimes, they interject their own bias into the final results. Using a randomized controlled trial, our primary, immediate objective is to evaluate 3 methods to reduce non-response bias. If successful, these methods will prevent or reduce non-response bias at the design stage for very low cost. Our second objective examines these innovations? effects on likely recruitment and participant representativeness in a planned, future cohort of Veterans from Operations Enduring Freedom, Iraqi Freedom, and New Dawn (OEF/OIF/OND) who are currently applying for VA posttraumatic stress disorder (PTSD) disability benefits. Findings will be used to identify the best method to quickly recruit a large and as unbiased sample as possible, to describe the targeted population?s key characteristics, and to calculate preliminary effect sizes for power estimations for the larger study.
