Improving access to social services such as safe water and sanitation, education and health care depends on being able to learn what approaches work better than others, when and why. Household surveys of intended beneficiaries are an essential source of information about program effectiveness. Recent evidence, however, suggests that these survey data may be unreliable because program beneficiaries may change the way they report their behavior as a result of the program. For example, if a program expands access to safe drinking water, people living in areas covered by the program may become more likely to report that they are drinking safe water, even if they are not. Such reporting bias could distort estimates of the impact of social programs, or comparisons between different programs, resulting in misleading policy recommendations. This research measures the size and prevalence of reporting bias in the context of a recent experiment that investigates whether participation by intended beneficiaries in decisions made during program implementation improves program outcomes.
A recent randomized controlled trial in Bangladesh has provided the first experimental evidence that documents how beneficiary participation in decision making affects outcomes of a social program to provide access to safe drinking water. Preliminary results, based on reported changes in behavior, suggest that involving beneficiaries in decision making results in improved access to safe drinking water, but only when the decision-making structure incorporates protective measures to avoid the capture of program benefits by influential individuals or groups. Participation in decision making, however, may influence not only program outcomes, but also the way in which program beneficiaries report outcomes in household surveys.
The researchers directly observe the behvior and characteristics of users of the water sources installed, and compare these observations to water use reported in household surveys in order to determine the size and extent of reporting bias. In particular, the researchers measure how reporting bias varies depending on the degree of community participation in the decision-making process. Because the original project randomly assigned the decision-making structure, communities that implemented the program under different decision-making structures were comparable in terms of all other characteristics. This enables the researchers to measure the direct effect of community participation in program decision making on the size and extent of reporting bias.
The study will further understanding of the limitations of using self-reported survey data to measure the effectiveness of social programs. It will also make two further contributions: the first direct, experimental evidence on the connection between community participation in decision making and the likelihood of capture of program benefits by elite or influential community members. The research will provide experimental evidence on whether community participation in decision making improves project sustainability, as measured by maintenance of the installed water sources.
Improving access to social services such as safe water and sanitation, education and health care depends on being able to learn what approaches work better than others, when and why. Household surveys of intended beneficiaries are an essential source of information about program effectiveness. However, recent evidence suggests that this essential source may be unreliable because program beneficiaries may change the way they report their behavior as a result of the program. For example, if a program expands access to safe drinking water, people living in areas covered by the program may become more likely to report that they are drinking safe water, even if they are not. Such reporting bias could distort estimates of the impact of social programs, or comparisons between different programs, resulting in misleading policy recommendations. This study has investigated the size and prevalence of reporting bias in the context of a recent experiment that investigates whether participation by intended beneficiaries in decisions made during program implementation improves program outcomes. A recent randomized controlled trial in Bangladesh (NSF-0624256) has provided the first experimental evidence that documents how beneficiary participation in decision-making affects outcomes of a social program to provide access to safe drinking water. Preliminary results, based on reported changes in behavior, suggest that involving beneficiaries in decision-making results in improved access to safe drinking water, but only when the decision-making structure incorporates protective measures to avoid the capture of program benefits by influential individuals or groups. However, participation in decision-making may influence not only program outcomes, but also the way in which program beneficiaries report outcomes in household surveys. We have collected direct observations of actual numbers and characteristics of users of the water sources installed. The direct observations suggest that the actual number of users is smaller than the reported number of users (20 households per village versus an estimated 52 households per village, in villages in which a water source was installed and is still functional). Preliminary analysis rejects some alternative potential mechanical explanations for this difference e.g. time trends, infrequent users. However, we are still in the process of investigating other potential hypotheses to explain these differences, including whether this is explained by a general tendency to over-report use of safe drinking water sources. We are also investigating how this reporting bias varies depending on the degree of community participation in the decision-making process. Since we randomly assigned the decision-making structure to communities in the original experiment, communities in which we implemented the program under different decision-making structures were comparable in terms of all other characteristics. This enables us to measure the direct effect of community participation in program decision-making on the size and extent of reporting bias. Preliminary results suggest that participation in decision-making may induce differences in reporting behavior, but these results are still being verified. The results from this study help to understand the limitations of using self-reported survey data to measure the effectiveness of social programs. Future analysis will also provide the first direct, experimental evidence on the connection between community participation in decision-making and the likelihood of capture of program benefits by elite or influential community members. We will also provide the first experimental evidence on whether community participation in decision-making improves project sustainability, as measured by maintenance of the installed water sources.
