The Effects of Education on Students’ Perception of Modeling in Software Engineering Omar Badreddin Arnon Sturm Abdelwahab Hamou-Lhadj Northern Arizona University Ben-Gurion University of the Negev Concordia University Flagstaff, U.S.A Beer Sheva, Israel Montreal QC, Canada Omar.Badreddin@nau.edu sturm@bgu.ac.il abdelw@ece.concordia.ca Timothy Lethbridge Waylon Dixon Ryan Simmons University of Ottawa Northern Arizona University Northern Arizona University Ottawa ON, Canada Flagstaff, U.S.A Flagstaff, U.S.A tcl@eecs.uottawa.ca waylon.dixon@nau.edu rsimmons07@gmail.com Abstract— Models in software engineering bring significant by and large [5]. Petre [16] also provided evidence that the potential in improvements of productivity of engineers, and actual adoption of UML is quite low. improved quality of the artifacts they produce. Despite this The reasons behind such low adoption may be attributed to significant potential, modeling adoption in practice remains many factors. These include complexity of modeling tools, the rather low. Computer Science and software engineering curriculums may be one factor that causes this low adoption. lack of compatibility within different tools, the lack of In this study, we investigate the effects of education on integration of modeling tools within existing environments, and students’ perception of modeling. We conducted a survey in three the lack of education about the value of modeling tools and separate institutions, in Canada, Israel, and the U.S. The survey techniques [6]. covers various aspects of modeling and addresses students Our goal in this study is to investigate the effect of ranging from a first year in undergraduate studies until final education in software engineering and computer science on years in graduate studies. students’ perception of modeling. In particular, we want to The survey’s findings suggest that the perception of understand what effect, if any, education has on how students undergraduate students towards modeling declines as they perceive the value of models. progress in their studies. While graduate students tend to be more favorable of modeling, their perception also declines over We designed and distributed a survey covering many the years. The results also suggest that students prefer more aspects of modeling for students throughout the full academic modeling content to be integrated earlier in the curriculum. spectrum, from undergraduate to post graduate students. The survey is conducted in three separate universities to allow for Index Terms—Survey of Perceptions, Pedagogy, Modeling in different cultural and perspectives. Software Engineering, UML, Education. This paper is organized as follows. In Section II, we review the related work. In Section III, we introduce the study design. I. INTRODUCTION We then briefly introduce a background on the modeling- Model Driven Engineering promotes the use of models, related curriculum at the three participating institutions. The rather than code, for system development. Models can be easier results of the survey are presented in section V and further to understand [1][2], improve communications amongst discussed and analyzed the results in Section VI. Finally, In stakeholders [3], and help generate executable artifacts [4]. In Section VII we conclude and as set plans for future research addition, platform independent models can improve system directions. portability, and can facilitate migrating systems from legacy platforms [15]. II. RELATED WORK UML has emerged as the standard modeling language in The perception of UML by professional software engineers software engineering. In an empirical assessment of MDE in has been investigated, with mixed results. Ariadi and Chaudron industry, Hutchison et al. [10] mentioned that UML was used have surveyed 80 professional software engineers about their by 85 percent of the respondents. Petre [16] mentioned many perception of the value of UML in terms of productivity and studies that have indicated the UML is the de facto standard quality [7]. Despite the low adoption of UML, its value is modeling language or the “lingua franca” one. The standard is perceived very positively in design, analysis and managed by OMG, and supports many aspects of the life cycle implementation. Other such surveys have reported negative of software development, from requirements, specification, to perceptions of UML due to its complexity, incompleteness, and development and deployment. However, the adoption of finds that UML is perceived to be difficult to learn [13]. modeling in software engineering practice remains dismal. UML as the standard modeling language is increasingly Studies point to significantly low adoption of modeling in becoming integrated into academic curriculums for practice [2] and that open source projects remain code-centric undergraduate and graduate students. There has been a number of studies that reported on experiences on teaching UML [12], 39 as well as studies in innovative tools for UML education [11]. f. Models are useful for implementation and/or code In addition, a number of studies on the effectiveness of a generation specific teaching technique for software engineering students g. Models are useful for testing have been reported, such as case study and problem based h. Models are useful for maintenance approaches [8]. 2. Modeling Characteristics (CHR) There has been a number of studies on the comprehension a. Models are normally used just as drawings of specific modeling notation, such as the work of Glezer et al. b. Code is just a type of model [9] on the comprehension of UML Interaction diagrams. c. Models are precise (i.e., unambiguous) However, there has been very little known about students’ d. Models can be easily checked to find opportunities perception of modeling, modeling tools and the curriculum in for improvement terms of modeling coverage and depth. e. Models are more comprehensible than code f. In general, models are easy to understand III. STUDY DESIGN g. Models facilitate abstractions and comprehension A. Goal h. Textual models are easier to understand than The goal of this study is to uncover the perceived value and graphical models usefulness of models by undergraduate and graduate students i. Textual models are easier to construct than graphical in Computer Science and Software Engineering as they models progress in their studies. The focus in this study is not on the j. Models are implementation independent specific modeling language (e.g., UML or BPMN) but rather k. Models help provide flexibility during the on the applicability and usability of models in general. development process The research questions we were interested in are the l. Modeling is counterproductive since the models need following: to be changed all the time m. Models are usually abandoned after the code is • Do students perceive models useful? And in what context? written What are the reasons for that? 3. Implementation (IMP) • Do students think or wish to have a more substantial a. Modeling tools are not mature enough modeling education? b. Modeling tools are too complex and are difficult to • Does students’ perception over modeling evolve over their learn studies? c. It is not easy for developers to obtain modeling tools B. Intended Subjects that meet their needs 4. Modeling Education (EDU) The intended subjects of this study are undergraduate and a. Modeling should be taught before programming graduate students in system development related fields such as b. Modeling and programming should be taught at the software engineering, computer science, and information same time systems engineering. c. Modeling is not being taught sufficiently C. Administering the Survey d. Modeling should be integrated in most software The survey was filled by students either in classrooms, labs, engineering and computer science courses or online. Participation was both anonymous and voluntary. The survey was conducted in three institutions, Concordia These questions had Likert scale: Strongly Agree, Agree, University in Canada, Ben-Gurion University of the Negev in Neutral, Disagree, Strongly Disagree, and NA; Israel, and in the Northern Arizona University in the U.S. representing a scale from 5 to 1. The full list of questions as well as the raw data is included in the supplementary D. The Survey material. The survey consisted of two parts: demographic data and a reflection on modeling. IV. BACKGROUND ON THE INSTITUTIONS AND THEIR The demographic data part included questions regarding the MODELING COURSES university, the study program, the academic year, age, work In the following we elaborate on the education background experience (with ranges: 0-3, 4-7, 8-12, 13+), and the average of the participants by introducing the curriculum in each of the grade (with ranges: 65-75, 76-85, 86-90, 91+). institutes. The reflection part included the following questions: A. Concordia University 1. Applicability of Models (APP) a. Models are very useful Concordia University (CU) offers two related programs that b. Models are useful for documentation are both managed by Department of Computer Science and c. Models are useful for communication Software Engineering: Computer Science and Software d. Models are useful for representing requirements Engineering. The Computer Science program focuses primarily e. Models are useful for specification on the study and design of computer systems, such as design algorithms, languages, hardware architecture, systems software, and applications software and tools. Whereas, the 40 Software Engineering program, while built on the that cover software modeling. Other courses refer to fundamentals of computer science, is focused more on the information systems management such as production principles and practices of engineering to develop creative management, organizational culture, information retrieval and applications such as, computer games, web services, data mining, operational research etc. The second program is information security, and avionics. the Software Engineering, which is managed by the two Tables 1 and 2 present the courses in which modeling departments of Information System Engineering and Computer education takes place along with their modeling content, in Science. Graduates of that program serve mainly in both programs. Although two programs are administratively development positions. Table 4 presents the courses in which separate, in this survey we unified the results of the two modelling education takes place along with their modeling programs since the modeling content is largely similar. content. Other courses include computer science foundation such as principles of programming languages, automata, Table 1. Computer Science @ Concordia compilation, etc. Credit Sem Course Modeling Content (/90) Table 3. Information Systems Engineering @ BGU Includes basics of object- Fundamentals of Sem Course Scope Modeling Content 2 3 oriented programming, (/160) Programming essentially UML aspects. Introduction to Abstraction and modeling of Basics of UML, mainly class 3 System Hardware 3 1 Information Systems 3 system architecture. diagram Engineering Object-Oriented Essentially, a modeling course 5 Database Systems 3.5 Modeling DB using ERD 4 7 Programming 1 & 2 related to all UML aspects. Analysis and Design of Focus mainly of functional Introduction to 5 5 5 4 Modeling DB using ERD Information Systems modeling Database Applications Object-Oriented Essentially, a modeling course Using modeling construct to 6 3.5 Analysis and Design related to all UML aspects. teach CA content such as 6 Computer Architecture 3 Using models to implement content/data flow, shared 7-8 Capstone Project 8 and manage a whole project. memory models, etc. Modeling DB using ERD, Table 4. Software Engineering @ BGU 6-8 Databases 4 OOD, and ODL. Using modeling construct to Credit Sem Course Modeling Content Introduction to (/160) 6-8 4 teach other SE content such as Software Engineering Introduction to Basics of UML, mainly class design patterns and refactoring. 2 2.5 Software Engineering diagram Essentially, a modeling course 6-8 Database Design 4 3 Database Systems 3.5 Modeling DB using ERD related to all UML aspects. Computer Science Using models to implement and Essentially, a modeling course 7,8 6 Analysis and Design of Project 1 & 2 manage a whole project. 4 5 related to all UML aspects as Software System well as DFD. Table 2. Software Engineering @ Concordia Using modeling construct to Topics in Software teach other SE content such as Credit 5 4.5 Sem Course Modeling Content Engineering design patterns, and (/120) refactoring. Basic principles of SE with Software Using models to implement 6-8 Software Process 3 activities in software notations 6 Implementation 3 iteratively a small scale and documentations. Workshop application. Software Architecture Essentially, a modeling course 6-8 6 Using models to implement and and Design 1 & 2 related to all UML aspects. 7-8 Capstone Project 8 manage a whole project. Using modeling construct to User Interface teach other UID content such as C. University of Northern Arizona 6-8 3 Design usability engineering, user models, and notations. At Northern Arizona University (NAU), there are two Control Systems and Using modeling construct to related programs; the first is Computer Science where there is 6-8 3 teach CSA content such as emphasis on theoretical foundation of computer science Applications block diagrams. (Automata theory, Algorithms, etc.) and the second is Applied Software Engineering Using models to implement and 7 3.5 Computer Science where students are given the option to Team Design Project manage a software project. Capstone Software replace theory courses with more applied courses (such as Using models to implement and 8 Engineering Design 4 manage a whole project. mobile and web development courses). The Computer Science Project program at NAU is accredited under ABET [14]. Table 5 B. Ben-Gurion University of the Negev presents the courses where modeling is covered along with their content. In Ben-Gurion University of the Negev (BGU) there are two system engineering programs which are related to the goal Table 5. Computer Science @ NAU of the survey. The first is the Information System Engineering Credit Sem Course Modeling Content program in which the focus on data analysis, yet, graduates of (/~120) that program are expected to perform development activities as Introduction to 1 Computer Science I 4 Basic Class Diagrams well. The program is managed by the department of the (+lab) Information System Engineering. Table 3 presents the courses 41 Introduction to B. Reflection on Modeling 2 Computer Science II 4 Basic Class Diagrams (+lab) Figures 1-3 summarize the average results for each 5 Data Base Systems 3 ER Diagrams institution and give an overview of the results per institution. In Many UML notations are the following, we discuss these results. 6 Software Engineering 3 presented (class, state machines, In general, the perception of BGU’s students towards use cases) Requirements modeling is positive. In particular, they perceive modeling as a 7 Engineering (Capstone 2 Project-specific UML. useful means mainly for documentation and communication. I) One of the reasons for that limited usefulness might be the students’ perception of modeling characteristics. For example, V. RESULTS the students perceive models as drawings, they find it In this section, we present the summarized results for each counterproductive, and they find textual models (like code) institution. The complete raw data as well as summarized data easier to deal with. As for the training they receive, the are made publicly available1 to facilitate replication and students think that more training on modeling is required. validation of the results [17]. Overall, the perception of NAU’s student of modeling is generally positive. Graduate students seem to appreciate A. Demographics modeling for documentation and communication. But they do All in all we got 195 filled questionnaires for the three not find models to be that useful for representing requirements universities. Analyzing the profiles of the participating or for specification. Their perception of models tends to get students, as they appear in the following tables, we found out significantly lower when it comes to using models for testing that most of the participants have good grades and they have and maintenance. limited work experience (a fact that emphasizes that their NAU Undergraduate students seem to find models to be perception is mainly established by their education). more comprehensible than code. This could be interpreted by Table 6. Profile of the participating students the fact that undergraduates find code to be challenging and/or complex. Graduate students, on the other hand, do not find (a) Number of Responses models to be more comprehensible. Number of Responses CU students find modeling very useful and that it should be Institute integrated in the curriculum earlier (as shown in Figure 3). Y1 Y2 Y3 Y4 Grad NAU 5 10 26 8 2 They also believe that modeling is important for various BGU-SE 8 12 17 22 software engineering tasks, and not just used for drawing 25 diagrams. Concordia students also believe that textual BGU-ISE 3 3 23 12 modeling is not easier to understand and construct than CU 0 0 0 0 19 graphical modeling. We attribute this positive reaction to (b) Years of experience modeling of Concordia students mainly to the fact that they are Experience (years) graduate students. Many of them had taken some graduate Institute 0-3 4-7 8-12 13+ courses on modeling as well. It is also interesting to note that NAU 46 5 1 0 when asked whether modeling and programming should be BGU SE 47 0 1 0 taught at the same time, Concordia students seem less favorable BGU ISE 35 3 0 0 to this idea (average score is 3.4/5). This might indicate that CU 12 5 1 1 students wish to see more courses dedicated to using models as the main development artifacts. Courses that combine tightly (c) Average grades obtained in modeling-related courses both perspectives (code and models) seem to reiterate the Average Grade (%) traditional perception of modeling, which restricts models to Institute 65-75 76-85 86-90 91+ the design phase only. NAU 0 20 14 17 The survey’s results towards the educational section tend to BGU SE 6 25 9 6 have an upward trend that is more evident for the graduate BGU ISE 5 23 6 4 students. Students in general want more training in modeling CU 0 4 8 7 and feel that modeling should be taught at the same time as coding. Interestingly, graduate students tend to agree more. The response rates we had for the questionnaire are as This can be interpreted by the fact that graduates appreciate follows. For BGU-ISE and BGU-SE, as the survey was models more, and have more appreciation on the role of conducted on line, we got response rate of 13 percent. For modeling in Software Engineering, and therefore, are more BGU graduate students, for NAU, and CU we had a response positive regarding increasing the modeling portions in the rate of above 90 percent as the survey was conducted in class curriculum. as a paper questionnaire. 1 https://zenodo.org/record/20367?ln=en#.Veuv5dNViFI 42 Fig. 1. BGU Results Fig. 2. NAU Results Fig. 3. Concordia University Results 43 Q3. How do students perceive modeling tools? For this VI. CROSS-UNIVERSITY ANALYSIS question, we limit our data analysis for graduate students as A. Perception Trends undergrads may not have the sufficient maturity to understand Of particular interest to this study, is to investigate whether the distinction between the tools, and the approach. curriculums have positive, negative, or neutral effects on how Q4. How does the students’ perception change from their students perceive the value of modeling. We studied the undergrad education to their final years in graduate studies? perception trend of both undergraduates and graduates as follows. For undergraduates, we analyze the changes in perception from year to year, starting from year 1 to 4. For example, if students’ average perception of “Models Usefulness” in year 1 is 3.0/5.0 and in year 2 the average perception is 4.0/5.0, this implies that the perception has improved from year 1 to 2. For graduates, we analyze the differences in perception from undergraduates and graduate averages. We do this by taking the average of the entire data set for undergrads and subtract it from the average for graduates. This analysis is performed using only a subset of questions that reflect models usefulness and value. The subset includes the following questions as listed in Section III.D: 1.a through 1.h, 2.c through 2.g, 2.j, 2.k. 2.l. We also report on the analysis of the students’ perception of educating students on modeling using analysis of answers to questions 4.a through 4.d. We use the sign analysis technique as reported in [18]. We count the number of positives (indicating perception Fig. 4. Models for documentation versus implementation improvement) and negative (indicating perception decline). The results are summarized in the following table. For Q1, it is evident that students tend to find models more useful for documentation and communication, and less for Table 7. Sign analysis of the survey results other development tasks (Fig. 4). This preference in perception Usefulness Education was more evident in graduate students that undergraduate + - + - students, particularly for NAU and Concordia graduate students. NAU UG 0 14 0 5 For Q2, both graduates and undergraduates agree that more NAU G 5 13 3 2 should be taught about modeling (UG: 3.7, G: 4.0) and that BGU UG 6 10 0 4 modeling should be taught at the same time (UG: 3.2, G: 3.3), BGU G 3 12 1 3 and that modeling should be integrated in computer science and NAU Undergraduate students’ perception of modeling software engineering courses (UG: 3.8, G: 4.0). declines as they progress in their undergraduate education, For Q3, students did not find modeling tools to be more evident by 14 negatives, and 0 positives. This result is also complex, but rather, modeling tools were perceived to be more reflected in students’ perception of modeling education (0 difficult to obtain (Fig. 5). This is potentially due to the fact that positive, and 5 negatives). For NAU graduate students, the data students are introduced to coding much earlier than modeling. is more balanced, but remains overall negative. One possible They become more quickly familiar with the coding platforms explanation might be that students with low perception of than with the modeling tools. modeling do not enroll in graduate studies, or that perception of modeling is an indicator of academic success. The results for BGU are more balanced, but remain overall negative. The perception of usefulness and education among undergraduates and graduates tend to decline over the years of their education. B. Cross-University Trends For the cross-university analysis, we are interested in answering the following high-level questions. Q1. Do students perceive models to be more useful for documentation and communication, as opposed to software development activities (code generation, implementation and maintenance)? Q2. Do students in general think that more modeling need Fig. 5. Perceptions of Modeling Tools to be integrated into the curriculum? 44 For Q4, we found a consistent pattern of declining participated, and that participants were not selected from a perceptions emerging in both NAU and BGU undergrad and specific group or study year. grad students. In general, the perception declination was more prominent in the case of undergraduate than graduate students. This can be interpreted in a number of ways, 1) the VIII. CONCLUSION curriculum fails to highlight the value of modeling in software This paper reports on a survey that was conducted in three engineering or 2) students come to the program assuming independent institutions. The goal of the survey is to uncover unrealistically high value of modeling. During their education students’ perception of the value of modeling in software years, the curriculum does not improve on that initial development. Participants included students from perception. 3) For large software projects, students fail to undergraduate year one to last year in the graduate program. discover the value of modeling (usually, UML), and may be The results suggest that students’ perception of the value of relying exclusively on code. As a result, students may come to modeling declines as they progress in their education. This was the conclusion that modeling is not as useful as they may have true for both undergraduate and graduate students. These thought initially. The lacks of tools may also contribute to this. results warrant further investigation into why this is the case. As the advance in the programs, students usually want to build The reasons of the decline in perception may be attributed to interesting systems that run quickly so they can make wrong perceptions of modeling for young students, in adequate modifications and improve their functionality. The coverage of modeling topics, lacks of adequate modeling tools, unavailability of good tools make this difficult. This may be a or immaturity or unsuitability of the modeling techniques and factor that discourages students from adopting the modeling approaches for students’ projects. It is also possible that UML paradigm as advanced stages. This question may require further is simply not appropriate for defining and/or implementing the investigation to uncover exactly why the perception declines. problems and solutions they face. However, grads tend to perceive modeling more favorably The results suggest that graduate students on average than undergrads, especially for communication, documentation, appreciate modeling more than undergraduates. This could be and tool availability and readiness. This is with the exception attributed to the nature of tasks that graduate students perform of a few aspects of models suitability for software development that may be more suitable for modeling approaches. and testing. This may be interpreted by the nature of the work The results of the study calls for further investigation of the performed by the grads vs. undergrads. Grads may be using reasons of the relatively low perceptions of modeling models to abstract ideas and communicate early concepts. usefulness by students. This can be done by further correlations Models for such tasks may be more suitable than code. analysis and interviewing students about their perceptions and the reason for that. Furthermore, the results call for revisiting VII. THREATS TO VALIDITY modeling curriculum in order to introduce improvements and Threats to validity of this study are discussed in this to further recruit the students into the modeling era. section. REFERENCES A. Question Bias [1] Omar Badreddin, Andrew Forward, and Timothy C. Lethbridge. The majority of the questions in this study were presented "Model oriented programming: an empirical study of in the positive sense (i.e. models are useful). It is possible that comprehension." Procs of the 2012 Conference of the Center for negative questions may have a different impact on how Advanced Studies on Collaborative Research. IBM Corp., 2012. participants respond to questions. [2] Andrew Forward, Timothy C. Lethbridge, and Omar Badreddin. "Problems and opportunities for model-centric vs code-centric B. Profile of the Respondents development: A survey of software professionals." 5th The researchers in this study did not have control on Workshop" From code centric to model centric: Evaluating the selecting participants. It is possible those participants who effectiveness of MDD (C2M: EEMDD)", University Pierre & opted to complete the survey, or those who decided to complete Marie Curie, Paris. 2010. the survey after it had started, may have had different views on [3] Robert France, and Bernhard Rumpe. "Model-driven modeling than the general population. Our study collected development of complex software: A research roadmap." 2007 profiling data and as discussed in this paper, we attempted to Future of Software Engineering. IEEE Computer Society, 2007. analyze the paper while considering the collected profiling [4] Djedjiga Mouheb, Mourad Debbabi, Makan Pourzandi, Lingyu data. Wang, Mariam Nouh, Raha Ziarati, Dima Alhadidi, Chamseddine Talhi, Vitor Lima. "Unified Modeling Language." C. Different Modeling Teaching Approaches Aspect-Oriented Security Hardening of UML Design Models. Springer International Publishing, 2015. 11-22. The three participating institutions deployed different curriculum and different teaching styles. It is possible that the [5] Omar Badreddin, Timothy C. Lethbridge, and Maged Elassar. "Modeling Practices in Open Source Software." Open Source participating universities teaching of modeling may have Software: Quality Verification. Springer Berlin Heidelberg, influenced the views of the participants. This could in effect 2013. 127-139. mean that the participating universities are not a good [6] Alan Zeichick, "UML adoption making strong progress." representation of the general population. This external validity Software development times 15 (2004). threat was minimized by the fact that three different institutions 45 [7] Ariadi Nugroho , and Michel RV Chaudron. "A survey into the [12] Dirk Frosch-Wilke, "Using UML in software requirements rigor of UML use and its perceived impact on quality and analysis-Experiences from practical student project work." productivity." Procs of the 2nd ACM-IEEE int’l sym. on InSITE-Informing Science and IT Education Conference. 2003. Empirical software eng. and measurement. ACM, 2008. [13] Martin Grossman, Jay E. Aronson, and Richard V. McCarthy. [8] George G. Mitchell, and James Declan Delaney. "An assessment "Does UML make the grade? Insights from the software strategy to determine learning outcomes in a software development community." Information and Software engineering problem-based learning course." International J. of Technology 47.6 (2005): 383-397. Engineering Education 20.3 (2004): 494-502. [14] Richard M. Felder, and Rebecca Brent. "Designing and teaching [9] Chanan Glezer, Mark Last, Efrat Nachmany, Peretz Shoval. courses to satisfy the ABET engineering criteria." J of Eng. "Quality and comprehension of UML interaction diagrams-an Education-Washington- 92.1 (2003): 7-26. experimental comparison." Information and Software [15] João Paulo Almeida, Remco Dijkman, Marten Van Sinderen, Technology 47.10 (2005): 675-692. and Luís Ferreira Pires. "Platform-independent modelling in [10] John Hutchinson, Jon Whittle, Mark Rouncefield, and Steinar MDA: supporting abstract platforms." Model Driven Kristoffersen. Empirical assessment of MDE in industry. Architecture. Springer Berlin Heidelberg, 2005. 174-188. In Proceedings of the 33rd International Conference on [16] Marian, Petre. "UML in practice". In Proceedings of the Software Engineering (ICSE '11). ACM, New York, NY, USA, International Conference on Software Engineering (ICSE '13), 2011. 471-480 2013. IEEE Press, Piscataway, NJ, USA, 722-731. [11] Timothy C. Lethbridge, Gunter Mussbacher, Andrew Forward, [17] First International Workshop on Human Factors in Modeling and Omar Badreddin . "Teaching UML using umple: Applying (HuFaMo’15). Available: http://hufamo.compute.dtu.dk/. model-oriented programming in the classroom." Software [18] W. J. Dixon and A. M. Mood. "The statistical sign test". 1946. Engineering Education and Training (CSEE&T), 2011 24th IEEE-CS Conference on. IEEE, 2011. Journal of the American Statistical Association pp. 557-566. 46