We propose to test and disseminate an innovative Ordinal Bayesian Instrument Development (OBID) method that seamlessly integrates expert and participant data, while using fewer subjects than classical approaches, to achieve a coherent and economical estimate of validity evidence for new instrument development. One contributor to delays in translating research evidence into practice is the amount of time it takes to develop valid and reliable psychometric instruments for measuring patient reported outcomes. Identifying sufficient numbers of participants, particularly when there are small populations or limited resources, often extends instrument development time. For example, for populations where there are small numbers or limited resources available, the development of instruments can take four years instead of two. Current accepted instrument validation methods analyze expert and participant data separately and consecutively, and require a large number of subjects. Content experts' evaluations of the extent to which items match the theoretical definition of the construct are first used to estimate content validity. Following that, the items re tested with participants to garner construct validity evidence (e.g., internal structure through factor analysis). Information from the expert data (content analysis) is not used in a factor analysis (or item response theory model) of the participants' data. In contrast, the proposed OBID method uses a framework grounded in long-standing and empirically verified Bayesian analyses where experts' data (prior distributions) are updated with participants' data (posterior distributions) to efficiently achieve a unified psychometric model. Building on our preliminary studies that used an approximation approach, the specific aims for this proposed study are to: 1) Test Ordinal Bayesian Instrument Development (OBID) by comparing its performance (i.e., stability and development time differences) to classical instrument development with exact estimation procedures, using simulation data; 2) Test OBID across a variety of patient and family caregiver populations; and 3) Disseminate BID & OBID software for evaluation by investigators in other research communities. We hypothesize OBID and classical instrument development will be equally efficient for large sample sizes, but that OBID will be more efficient than classical instrument development when only small numbers of participants or limited resources are available. Thus, OBID promises to be a new and expeditious method for instrument development, adding to our current measurement toolbox.
Health research can take several years to translate into important changes in health care due in part to the time it takes to develop and test valid and reliable instruments. Using statistical models derived from both experts and participants, our study expedites psychometric instrument development, in particular for studies from small populations (e.g. research on health disparities and on rare diseases) and/or studies with limited resources. A tested and disseminated Ordinal Bayesian Instrument Development (OBID) method will allow investigators to decrease the development time by reducing the number of participants needed by transferring the number of responses needed from participants to expert panels.
