To build high quality software, it is increasingly important for engineers to reason quantitatively about critical system properties, such as performance and reliability, as early as possible in the design process. Ideally, these properties are assessed before significant time and cost are expended on implementation. However, making useful predictions in early design stages is difficult at best due to the interplay among many relevant factors, such as complex properties of software components, effects of the firmware, and conflicts between desired system attributes. In this project, we focus on design-time evaluation of architectures with respect to one key attribute: reliability. The approach we pursue will enable an engineer to build a multi-faceted, hierarchical model of a system and assess its reliability in an incremental, scalable fashion. Moreover, we particularly focus on the area of embedded systems. Embedded systems present a rich target of opportunity for this work as (1) they demand close interplay of software and execution substrate; (2) they are often unencumbered with legacy concerns, allowing easier introduction and exploration of new techniques such as the one we propose; (3) they often have stringent and complex requirements, yet are seldom approached from a software architectural perspective. An extensive evaluation of our research focuses on two measures of interest: tractability (intended to address scalability issues existing in real, complex systems) and sensitivity (intended to address issues of confidence in our predictions under numerous design-time uncertainties).
To build high quality software, it is increasingly important for engineers to reason quantitatively about critical system properties, such as performance and reliability, as early as possible in the design process. Ideally, these properties are assessed before significant time and cost are expended on system implementation. However, making useful predictions in early design stages is difficult at best due to the interplay among many relevant factors, such as complex properties of the components comprising the software system, effects of the underlying computing infrastructure (which has its own software and hardware components), and conflicts between desired system attributes. In this project, we have focused on design-time evaluation of software systems' architectures with respect to one key attribute: reliability. The approach we pursued enables an engineer to build a multi-faceted, hierarchical model of a system very early in its development, and to assess the system's reliability in an incremental, scalable fashion. The particular focus of this project has been on the area of concurrent embedded systems (such as systems involving fleets of mobile robots). Embedded systems have presented a rich target of opportunity for this project as (1) they demand close interplay of software and its execution substrate; (2) they are often unencumbered with legacy concerns, allowing easier introduction and exploration of new techniques such as the one we developed; (3) they often have stringent and complex requirements, yet are seldom approached from a software architectural perspective. An extensive evaluation of our research has focused on two measures of interest: tractability and sensitivity. Tractabiliity is intended to address scalability issues inherent in real, complex systems. On the other hand, sensitivity is intended to address issues of confidence in our predictions under numerous design-time uncertainties. In particular, we have evaluated the tractability and sensitivity of our predictions with respect to "ground-truth" models that were obtained using existing techniques. These existing techniques rely on software system implementations (i.e., on system information our models do not require, and in fact assume is not available). Furthermore, the existing techniques result in much larger (and thus less tractable) models than those produced by our technique. At the same time, our results show that our technique's predictive power is within a few percent of the ground truth. As an example, the attached figure shows the comparison of our technique (denoted as "SHARP") to the ground truth (denoted as "Flat Model") in the context of a large building-security system. The broader impact of this work includes improving multi-disciplinary education of future systems engineers as well as the foundation for creating a new multi-disciplinary community that is significantly better equipped to deal with enormous difficulties of highly-complex, large-scale system design, analysis, and implementation as compared to existing (more traditional) communities.
