The University of Maryland, Baltimore County (UMBC) and University of Illinois at Urbana-Champaign (UIUC) propose to evaluate two cybersecurity educational assessment tools: the Cybersecurity Concept Inventory (CCI) and the Cybersecurity Curriculum Assessment (CCA). The CCI assesses the quality of instruction of any first course in cybersecurity; the CCA assesses how well a college curriculum prepares graduates for a career in cybersecurity. The evaluation will consist of student interviews, expert reviews, and statistical analyses to examine the efficacy of these tools. In addition, the project will provide initial insights into the efficacy of educational approaches by comparing student performance relative to their instructors' teaching practices. At present, there is not a rigorous, research-based method for measuring the quality of cybersecurity instruction. Validated assessment tools such as the ones in development at UMBC are essential so that educators have trusted methods for discerning whether efforts to improve student preparation are successful. This validation study will complete an essential part of the broader Cybersecurity Assessment Tools project, which will provide rigorous, evidence-based instruments for assessing and evaluating educational practices.
To develop evidence for the validity of the CCI's and CCA's measurement of student conceptual understanding of cybersecurity, the project will complete six tasks for each assessment tool: (1) conduct at least thirty cognitive interviews of students thinking aloud while answering the draft questions, and use results to improve the validity of the questions; (2) collect and use feedback on the draft questions from at least twenty experts in cybersecurity or cybersecurity education; (3) administer the draft tool to at least 200 students and use psychometric methods of analysis on the results; (4) revise the draft questions in light of tasks 1-3; (5) administer the revised tool to at least 1,000 students and use psychometric analysis on the results; and (6) hold workshops at four cybersecurity education meetings to promote awareness of, and to seek additional feedback on the tools. By following well-established rigorous methods for developing CCIs, this project will help the cybersecurity educational community to develop and improve curricula and instructional methods and materials.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
