This engineering education research award will explore language differences on the engineering concept inventory for engineering statics. The Concept Assessment Tool for Statics (CATS) is an instrument developed to identify students? conceptual understanding of statics concepts. In pilot tests at University of Puerto Rico Mayaguez, results demonstrated very low performance (28%) of students on CATS as compared with national standards which ranged between 30% - 70%. Because this test is in English, which is not the first language of most University of Puerto Rico Mayaguez students, the researcher proposes to test the efficacy of CATS with bilingual students. To accomplish this goal the study will be guided by the following research question: Do mechanical and civil senior engineering students? from the University of Puerto Rico at Mayaguez exhibit language difficulties with CATS? A mixed methods design is proposed in which data will be collected using quantitative (questionnaires and/or closed-ended questions) and qualitative (open-ended questions using focus groups) instruments. Similarly, data will be analyzed following quantitative analyses, but also applying open coding techniques for qualitative purposes. The expected results of this study will be the development of a beta version of CATS, in Spanish, to be tested further for validity and reliability. This project advances knowledge and explores creativity because it will be the first time that: (1) 100% of the sample of an engineering concept research study will consist of bilingual students; (2) an engineering concept inventory is developed in Spanish; (3) such a study measures language differences on engineering concept inventories. The results of the work are expected to enhance student learning of difficult concepts in statics, and particularly to improve the success of Hispanic students.
Intellectual Merit: CATS is one of the most robust concept inventories developed in engineering. Since its development, it has been validated using multiple statistical analyses. This project allowed us for the first time to test a concept inventory for validity in a targeted population. Students at UPRM are very unique since they are bilingual students, and Spanish is their primary language. The protocol established includes an incorporation of qualitative research methods such as think-aloud and linguistic analysis. This methodology could be applied to other concept inventories or instruments that are used to measure conceptual understanding, such as pre and post-tests, course examinations, etc. Our findings indicate that conceptual errors committed by our students are very similar to those identified by the developers of CATS, but the graphical representation generated by students were different. Therefore, the distractors (wrong answers in the multiple choice questions) might not be effective with our student population. Due to sample size limitations, we did not found statistical difference in item performance between participants who responded the CATS (English version) and the CATS-S (Spanish version). We have the CATS-S available for faculty to use at UPRM and expect to continue performing statistical analyses from CTT, IRT and cognitive assessment. Since the quantitative approach was not successful, a linguistic analysis was conducted and revealed that common errors students made were not due to language limitations (remember these are Hispanic students) but to conceptual errors (misconceptions). Several publications have resulted directly or indirectly from this project throughout the awarded period. They include: Mazak, C.M., Font, C.B., Santiago-Román, A.I., "Effects of Language on CATS Performance", proceedings ASEE Annual Conference and Exposition. Indianapolis, IN, 2014. Denick, D., Santiago-Román, A.I., et.al., "Testing the diagnostic capability of the CATS with students think-alouds", proceedings ASEE Annual Conference and Exposition. Atlanta, GA, 2013. Denick, D., Santiago-Román, A.I., et.al, "Protocol studies of students’ conceptual reasoning", AERA Annual Conference, San Francisco, CA, 2013. Santiago-Román, A.I., Denick, D., et.al, "Validating of the diagnostic capabilities of concept inventories: Preliminary evidence from the Concept Assessment Tool for Statics", ASEE Annual Conference, San Antonio, TX, 2012 González, N., Marín, R., Nieves, L., Pacheco, P., Santiago-Román, A.I., "The Development of a Spanish Version of the Concept Assessment Tool for Statics", proceedings ASEE Annual Conference, San Antonio, TX, 2012 Santiago-Román, A.I., Quintero, P., Serrano, G., "Broadening Participation Activities at UPRM through the National Science Foundation BRIGE Program", proceedings ASEE Annual Conference, San Antonio, TX, 2012 Streveler, R.A., Miller, R.L., Santiago-Rom?n, A.I., et.al. (2011). Rigorous Methodology for Concept Inventory Development: Using the ‘Assessment Triangle’ to Develop and Test the Thermal and Transport Science Concept Inventory (TTCI). IJEE 27(5), 968-984. Other presentations include: Hernández, C., González, N., Cotto, A., Acevedo, I., Santiago-Román, A.I., BRIGE: Validating Concept Assessment Tool for Static items distractors among native-Spanish-speaking engineering students at University of Puerto Rico, Mayaguez. UPRM's SACNAS 2nd Annual Symposium: Science and Technology, Innovation through Research, April 2014. Santiago-Román, A.I., Mazak, C., et.al. Testing the Efficacy of Concept Inventories With Bilingual Engineering Students. AERA. Philadelphia, Pennsylvania, April 2014. Cotto, A., González, N., Santiago-Román, A.I.. Testing the Validity of Concept Inventories at UPRM: A Case Study with CATS. UPRM's Graduate Studies Open House, February 2013 Santiago-Román, A.I., Pomales, C., Cesaní, V., "An Outcome-Based curriculum design approach for an Industrial Engineering Program curriculum revision", ISERC conference, San Juan, PR, May 2012. Santiago-Román, A.I., "BRIGE: Testing the efficacy of concept inventories with bilingual students: The application of CATS at UPRM", NSF EEC PI Meeting, Washington D.C., March 2012 and NSF BRIGE PI Meeting, Washington D.C., August 2011. Broader Impacts: A total of ten undergraduate students and two graduate students (7 female) have been impacted directly with this grant. They had had the opportunity to learn and conduct research in engineering education, an area that is growing at UPRM. Four of the students were able to attend the ASEE Annual Conference between 2011-13 and presented a paper session or a poster. Also, those undergraduate students, who had a research experience, continued participating in undergraduate research in their "home departments" or continued graduate studies. One of the graduate students with research experience in this project continued working in the academia. Also, three courses have been created and delivered through the period of this grant. Both graduate and undergraduate students have enrolled in these courses. Some of the undergraduate students continued graduate studies or got involved in other research experiences. Similarly, graduate students developed topics for their thesis projects with an educational component. In summary, research and educational experiences provided by this award has served as a pipeline since students have been able to (1) complete their academic degrees, (2) continue being involved in research experiences, or (2) continued graduate studies. Finally, the CATS-S is available to be used at other Hispanic institutions in the US and other countries.
