We propose to test and disseminate an innovative Ordinal Bayesian Instrument Development (OBID) method that seamlessly integrates expert and participant data, while using fewer subjects than classical approaches, to achieve a coherent and economical estimate of validity evidence for new instrument development. One contributor to delays in translating research evidence into practice is the amount of time it takes to develop valid and reliable psychometric instruments for measuring patient reported outcomes. Identifying sufficient numbers of participants, particularly when there are small populations or limited resources, often extends instrument development time. For example, for populations where there are small numbers or limited resources available, the development of instruments can take four years instead of two. Current accepted instrument validation methods analyze expert and participant data separately and consecutively, and require a large number of subjects. Content experts'evaluations of the extent to which items match the theoretical definition of the construct are first used to estimate content validity. Following that, the items re tested with participants to garner construct validity evidence (e.g., internal structure through factor analysis). Information from the expert data (content analysis) is not used in a factor analysis (or item response theory model) of the participants'data. In contrast, the proposed OBID method uses a framework grounded in long-standing and empirically verified Bayesian analyses where experts'data (prior distributions) are updated with participants'data (posterior distributions) to efficiently achieve a unified psychometric model. Building on our preliminary studies that used an approximation approach, the specific aims for this proposed study are to: 1) Test Ordinal Bayesian Instrument Development (OBID) by comparing its performance (i.e., stability and development time differences) to classical instrument development with exact estimation procedures, using simulation data;2) Test OBID across a variety of patient and family caregiver populations;and 3) Disseminate BID &OBID software for evaluation by investigators in other research communities. We hypothesize OBID and classical instrument development will be equally efficient for large sample sizes, but that OBID will be more efficient than classical instrument development when only small numbers of participants or limited resources are available. Thus, OBID promises to be a new and expeditious method for instrument development, adding to our current measurement toolbox.

Public Health Relevance

Health research can take several years to translate into important changes in health care due in part to the time it takes to develop and test valid and reliable instruments. Using statistical models derived from both experts and participants, our study expedites psychometric instrument development, in particular for studies from small populations (e.g. research on health disparities and on rare diseases) and/or studies with limited resources. A tested and disseminated Ordinal Bayesian Instrument Development (OBID) method will allow investigators to decrease the development time by reducing the number of participants needed by transferring the number of responses needed from participants to expert panels.

National Institute of Health (NIH)
National Institute of Nursing Research (NINR)
Small Research Grants (R03)
Project #
Application #
Study Section
Biomedical Computing and Health Informatics Study Section (BCHI)
Program Officer
Marden, Susan F
Project Start
Project End
Budget Start
Budget End
Support Year
Fiscal Year
Total Cost
Indirect Cost
University of Kansas
Biostatistics & Other Math Sci
Schools of Medicine
Kansas City
United States
Zip Code
Bott, Marjorie; Karanevich, Alex G; Garrard, Lili et al. (2018) Confirmatory Factor Analysis Alternative: Free, Accessible CBID Software. West J Nurs Res 40:257-269
Gibbs, Heather D; Ellerbeck, Edward F; Gajewski, Byron et al. (2018) The Nutrition Literacy Assessment Instrument is a Valid and Reliable Measure of Nutrition Literacy in Adults with Chronic Disease. J Nutr Educ Behav 50:247-257.e1
Gibbs, Heather D; Camargo, Juliana M T B; Owens, Sarah et al. (2018) Measuring Nutrition Literacy in Spanish-Speaking Latinos: An Exploratory Validation Study. J Immigr Minor Health 20:1508-1515
Garrard, Lili; Price, Larry R; Bott, Marjorie J et al. (2016) A novel method for expediting the development of patient-reported outcome measures and an evaluation across several populations. Appl Psychol Meas 40:455-468
Garrard, Lili; Price, Larry R; Bott, Marjorie J et al. (2015) A novel method for expediting the development of patient-reported outcome measures and an evaluation of its performance via simulation. BMC Med Res Methodol 15:77
Gajewski, Byron; Price, Larry R; Bott, Marjorie (2015) Response to Sijtsma and van der Ark (2015): ""Conceptions of reliability revisited and practical recommendations"". Nurs Res 64:137-9