Experimental research methods in field, clinical, and laboratory settings are the preponderant source of systematic knowledge about the effectiveness of treatment in the mental health field. The application of these methods, however, is quite varied in practice and often falls well short of textbook standards. While methodological and statistical theory provide insights into the vulnerabilities of experimental design when it varies from the ideal, it cannot tell us which potential biases and errors will prove most problematic in practice. For that we need empirical research on the influence that different method variations have upon the estimates of the treatment effects that result. The now extensive body of meta-analyses of treatment effectiveness studies includes much information regarding the effects of methodological variation and can be viewed as a form of such research. The study proposed here will extract and analyze the information found in over 100 meta-analyses of mental health treatment effectiveness research that bears on the role of method in shaping the treatment effect estimates. Additionally, to determine which relationships are distinctive to mental health research, Comparisons will be made with two other domains of treatment effectiveness research that are well represented in meta-analysis. The resulting database will be analyzed to (1) assess the proportion of total treatment effect estimate variance associated with differences in various aspects of design, measures, samples, and treatments across all the treatment areas represented, and (2) examine the characteristics of treatment areas and research circumstances that might account for differences in the amount of influence methodological practices have on treatment effect estimates.
| Wilson, D B; Lipsey, M W (2001) The role of method in treatment effectiveness research: evidence from meta-analysis. Psychol Methods 6:413-29 |
