Concerns about both economic competitiveness and educational equity emphasize the need for the United States to broaden and diversify the pipeline of students prepared and motivated to pursue STEM college majors. An emerging strategy for addressing this need is large-scale implementation of inclusive STEM high schools. In this exploratory project, investigators from SRI International and George Washington University are laying the foundation for a rigorous quasi-experiment to test the effects of attending such a school using longitudinal student records, surveys, and interviews. The project's operational definition for an inclusive STEM high school (ISHS) is a school, school within a school, or school program that accepts students primarily on the basis of interest rather than aptitude or prior achievement and gives them the mathematics and science preparation they need to succeed in a STEM college major. ISHSs enroll students from groups underrepresented in STEM professions through an application process that does not require high test scores before high school entry. In contrast to selective STEM schools that admit gifted and talented students on the basis of entrance examination scores and thus select for perceived STEM aptitude. ISHSs have the more ambitious goal of developing STEM expertise.
To establish the feasibility of a large, multi-state investigation of the effectiveness of inclusive STEM schools at scale, researchers are: Developing a tentative taxonomy of ISHSs and exploring implications of ISHS heterogeneity for the research design; - Recruiting three school partners representing different ISHS approaches; - Using state data to identify a comparison school (without a particular focus on STEM) for each ISHS school partner and recruiting comparison school partners; - Developing School Leader and three student surveys (fall 9th-grade, spring 12th-grade, and spring post-graduation); - Collaborating with partner schools in design of data collection procedures, recruiting materials, and incentives; - Piloting the School Leader Survey and two student surveys (9th-grade fall survey and 12th-grade spring survey) in six partner schools; - Identifying and recruiting a larger sample of ISHSs and matched comparison schools for Year 2 data collection; - Administering surveys in 40 or more high schools; - Locating spring 2012 graduates of the three ISHS partner schools and pilot testing the post-graduation student survey with these students; and - Engaging an Advisory Board who will provide methodological expertise and advice.
Ultimately, by documenting survey response rates, student location rates, and rates for successful matching of student administrative and survey data, this feasibility work is demonstrating that it is possible to collect the kind of data that would enable a large-scale study to be launched with the necessary instruments and experience in hand. As evidenced by the recent call from the President's Council of Advisors in Science and Technology for 1,000 new STEM schools and the National Research Council's report entitled "Successful K-12 STEM Education: Identifying Effective Approaches in Science, Technology, Engineering, and Mathematics" that highlights various STEM schools, the proposed research is highly relevant to current policy initiatives and debates. Moreover, the research has the potential to promote diversity in the STEM pipeline by influencing policymakers in states and districts that have yet to implement ISHSs at scale.
Objectives This project’s core objectives were two-fold: (1) to develop advanced personalized learning technologies and (2) to study to what extent and under which conditions these technologies promote robust learning of science concepts in secondary science topics such as plate tectonics, weather and climate, and biological evolution. Major Outcomes An educational recommender system – CLICK2 – was created to assess learners’ essays and rich textual responses to science questions and automatically identify potential knowledge gaps and misconceptions. Interactive learning resources are strategically recommended by CLICK2 to help learners’ address these issues and develop robust understandings of core science concepts. Computer science research examined the degree to which state-of-the-art natural language processing algorithms could be used to automatically identify knowledge gaps and misconceptions and prioritize which ones to target for further instruction. Effective prioritization is both very important and very challenging. Studies of effective teaching and learning science theory highlight the importance of targeting instruction to enable students to develop deep understanding of core ideas. Such "robust" understandings are critical for supporting learners to make inferences and to generalize their knowledge to new domains. Prioritization is computationally challenging as it required fundamental advances in NLP algorithms in order to: (1) automatically extract and compare core concepts in learning resources and student writings to identify potential misconceptions and (2) sequence potential misconceptions in terms of instructional importance and recommend resources in a pedagogically useful order. Significant NLP algorithmic advances in both identifying core concepts and prioritizing misconceptions were made in this project. Our algorithm for identifying short textual similarity won the SemEval STS 2014 competition, a leading international NLP algorithm benchmarking contest. Our algorithm for prioritizing misconceptions ranked highly in a student misconception identification task at SemEval 2013. Learning sciences research informed the project’s theoretical and empirical approaches. Conceptual change theory informed the design of the CLICK2 user interface: its principals guided both how the interface highlighted potential gaps and misconceptions to the learner and how the learning resource recommendations were presented. A series of controlled learning studies were conducted to measure the influence of CLICK2 on learning outcomes. Results showed that CLICK2 had a significant positive impact on learners’ conceptual understandings and metacognitive skills. Intellectual Merit This project illustrates the powerful role learning sciences can play in fostering innovation in computer science. Our overall goal – to develop cost-effective personalized learning environments – led to fundamental advances in NLP algorithms and the design of recommendation system interfaces. The CLICK2 algorithms demonstrated very high accuracies in leading NLP competitions and have direct applications to many other problems in computer science, such as text summarization, unstructured data mining, dialog systems, and question answering. The CLICK2 interface advanced the state-of-the art in recommendation systems by demonstrating the utility of cognitively-informed theory for designing effective computer-human interaction models. Broader Impacts This project has contributed to advancing STEM learning for traditionally under-represented populations in two ways. First, CLICK2 is designed to tailor instruction to individual student needs. This capability is particularly critical for diverse students, who have widely varying background knowledge, life experiences, and linguistic skills. Scalable and effective online instructional methods that acknowledge and build on these differences are increasingly needed as our nation’s classrooms become more diverse and more opportunities for student learning move online. Second, this project contributed to the professional development of K12 STEM teachers, including those serving large numbers of diverse students, through their involvement in a series of participatory design workshops. In these workshops, teachers learned about advanced cyberlearning tools and learning sciences’ theories, and developed their capacity to serve as partners in educational technology design and research.
