A Socio-Technical Framework for Face-to-Face Teaching in Large Software Development Courses Marlo Haering Walid Maalej Department of Informatics Department of Informatics University of Hamburg University of Hamburg Hamburg, Germany Hamburg, Germany haering@informatik.uni-hamburg.de maalej@informatik.uni-hamburg.de Abstract—In face-to-face teaching, students work in pairs on programming exercises and present their solutions to tutors. This setting fosters social skills. Students benefit from immediate feedback loops and personalized explanations. However, with an increasing number of students, it becomes challenging to scale this approach to very large courses due to the logistic and organizational effort. In this paper, we first report on significant challenges that we identified while conducting face-to- face teaching in a software development course with more than 600 students and 50 tutors. Second, we introduce a preliminary Fig. 1. Laboratory room with pair programming students (yellow frame) and socio-technical framework for face-to-face teaching to facilitate student tutors (red frame). The tutors check the solutions of the students. logistical aspects, monitor the students exercise progress, and improve the students’ learning experience. Index Terms—face-to-face exercises, software development ed- ucation, learning analysis I. I NTRODUCTION Our Applied Software Technology group at the Univer- sity of Hamburg hosts the introductory programming course Software Development 1 (SE1). SE1 consists of two parts: a weekly 90-minute lecture and a weekly 3-hour face-to-face exercise session. The course is compulsory for all computer science majors. Students with other majors are also eligible to register for SE1 including mathematics, pedagogy, physics, or psychology. SE1 is an entry-level course in programming and requires no prior knowledge. We host SE1 annually in winter Fig. 2. Part of a student’s evaluation sheet: student details (top), attendance terms. table (top right), and one table per exercise sheet, each with four subtasks. The exercises are structured thematically in accordance with the lecture. At first, we introduce new concepts with examples in the lecture. In the subsequent practice week, the students questions. Tutors check off each passed subtask by signing work in pairs on an exercise sheet in the laboratory rooms as the student’s evaluation sheet, which is shown in the tables in shown in Figure 1. The programming exercises are inspired Figure 2. The tutor declines the solution if the presentation or by Barnes and Kölling [1] and typically require the students the solution is insufficient. In this case, the students have to to complete a well-defined implementation task in a prepared improve the shortcomings and present their corrected solution. project. Students have to have all exercises accepted to pass SE1. During this session, tutors (teaching assistants and student Course instructors prepare the teaching material, monitor the tutors) are present for two main purposes. On the one hand, students’ progress, and manage the logistic aspects. The course they support the students in case of questions or uncertainties evaluation showed that students appreciate this format. and on the other hand they approve the solutions of the Face-to-face exercises encourage the students to reproduce students. After the students completed a subtask, they have their just learned knowledge by explaining it with their own to present their solutions to a tutor. The tutor either accepts or words to a tutor. Thereby, tutors identify and correct mis- declines the solution. A tutor accepts the solution if it (1) understandings early and provide immediate feedback with solves the task, (2) the students present and explain their personalized explanations. Students have to program in pairs solution with their own words, and (3) answer subsequent as previous research has shown an improved code quality and ISEE 2019: 2nd Workshop on Innovative Software Engineering Education @ SE19, Stuttgart, Germany 3 better student grades [2], [3]. Furthermore, this fosters commu- spreads across seven rooms. Tutors change rooms to check nication and presentation skills as well as social interactions the occupancy in other rooms. Based on our experiences, we among students and with their tutors as we consider these im- found that a ratio of one tutor to seven students is sufficient. portant skills for prospective software developers. In contrast For tutoring, we usually require ~40 student tutors and ten to a solely technical based framework that checks students’ teaching assistants. Most of the tutors apply for more than task automatically, we retain the social interaction between one exercise session. students and tutors. Therefore, we label our framework “socio- The programming experience among the tutors is hetero- technical” to highlight that the operational face-to-face tutoring geneous. We observed that experienced tutors have higher stays an essential part in our framework [4]. demands on the solutions of the students. However, it is In recent years, the number of students has risen steadily challenging to maintain consistent acceptance criteria across and has now reached more than 600, which are mostly first all tutors for the evaluation. Therefore, we set up a two-hour semester students. Applying this approach to large software introductory meeting with all tutors before the semester starts. development classes is challenging because it does not scale Additionally, we provide a briefing session each week for the easily. We summarize the main challenges that we identified in tutors to collect feedback about the previous exercise week, Section II. In Section III, we introduce our preliminary socio- walk through the upcoming exercise sheet, and discuss the technical framework to facilitate the logistical workload, to acceptance criteria. enable learning analysis for course instructors, and to keep B. Alternating Teaching Staff the face-to-face component in the exercise sessions. The general aim is to maintain knowledge about the lo- II. C HALLENGES gistics independently of an individual course instructor so In this section, we describe the challenges that we identified that different employees can take over course management during our previous SE1 courses and partly outline how we activities at any time. This is particularly important as the currently cope with them. logistics contain fine-grained tasks with crucial deadlines and the staff is typically rotating after two years. Therefore, we A. Scalability and Logistics explicitly document single steps and their deadlines for each In recent years, the number of students in our SE1 course part. Always two course instructors take responsibility for the has steadily increased (~300 students in 2007-09, ~400 stu- SE1 exercises to ensure reliability. Furthermore, the fluctuation dents in 2010-12, ~500 students in 2012 and ~600 students among the tutors is high with ~60% new tutors each semester. in 2016). Additionally, computer science is becoming more Experienced tutors are beneficial to support new tutors and and more important across various disciplines. In particular, teaching assistants in many aspects of the practice. To mitigate SE1 is also eligible for non-computer scientists. We expect the knowledge loss, we require student tutors to follow along that the number of students will increase to maybe more with other tutors during the semester for three hours in total than 800 students in the next 2-3 years. A higher number to enable a knowledge transfer between experienced and new of students will make it challenging to keep conducting face- tutors. to-face exercises in SE1 as it requires more laboratory rooms C. Short-term Replacement and Rush Management and more tutors. The occupancy of each exercise session is inconsistent as The logistics of the SE1 exercises comprise various parts students visit other time slots when they could not finish the about tutor management, teaching material, and student com- exercises in time. Also, tutors are regularly absent due to ill- munication. For the preparation of the SE1 exercises, we ness. Due to these unexpected factors, single exercise sessions recruit student tutors in addition to teaching assistants. The exceed a proportional ratio between tutors and students. This number of required tutors scales directly with the number of leads to long waiting times for students until a tutor is available students. We contact students via different channels including for their requests. In this case, tutors request tutoring support other courses, mailing lists, or face-to-face. The students apply on demand from teaching assistants to overcome a temporary via an online form for a tutor position. We mainly select shortcoming. students based on their tutoring experience, recommendations As each of the student tutors have their course schedule, we by previous tutors, and prior grades. provide two redundant briefing sessions per week. The student Students register for one of eight different time slots during tutors have to attend one of these sessions to prepare for the the week for the face-to-face exercises. The preferences of upcoming exercise sheet. Two teaching assistants carry out two the students differ each semester. The course instructors try to separate briefing sessions for the student tutors so that they assign a sufficient number of tutors to each exercise session, can substitute each other. The teaching assistants prepare the depending on the availability of tutors and the number of regis- briefing session and discuss the current version of the exercise tered students. Furthermore, a teaching assistant is assigned for sheet. each exercise session for supervision and answering questions of the tutors. D. Continuous Progress Monitoring and Feedback on Course Most of the exercise sessions require multiple rooms, for To pass the SE1 exercises, students have to pass the instance, one of our sessions with more than 120 students requirements on each exercise sheet. During the semester ISEE 2019: 2nd Workshop on Innovative Software Engineering Education @ SE19, Stuttgart, Germany 4 the tutors continuously monitor how many students currently the number of predecessors in the queue and the number of comply with the conditions to pass the SE1 exercises. As the available tutors, the framework indicates an estimated waiting tutors check off the approvals on the evaluation sheet, it is time for the students until the next tutor will be available. currently time-consuming to analyze each sheet manually and to continuously monitor the students’ progress. B. Tutors App Moreover, course instructors collect feedback on the SE1 Tutors log into the framework via their smartphone. They exercise part via three different channels: (1) from the tutors have access to an extended version of the exercise sheet with in the briefing sessions on their impression about how the additional remarks, sample solutions, and sample questions to students cope with the exercises, (2) from students via an ask students during the evaluation. During the exercise session, anonymous online survey, (3) from students via an anonymous they process the queue of student requests. An available tutor evaluation of the whole course at the end of each semester. queries the next request from the student queue and visits Course instructors aggregate the feedback and formulate con- their workplace, which might be in a different room. The crete change requests to improve both the process as well as tutor has access to the evaluation sheets of the students and each weekly exercise sheet. The complexity of these changes sees comments by tutors on previous evaluations. The tutor varies from typos to replacing subtasks or source code projects. evaluates the solution and rates different categories as, for instance, presentation, code quality, and speech share. Thereby, E. Participants with Different Prior Knowledge we keep the operational face-to-face teaching by the tutors The students in SE1 are from diverse studies with different as a social component in our framework. Following tutors prior knowledge. Because the participants increasingly come utilize previous evaluation data and take previous comments from subjects outside of the computer science department and into account. For instance, if a tutor notices that one of the outside of the MIN faculty, we have to adapt to different prior students shows less participation in the presentations, the tutor knowledge levels. Therefore, in the lecture and especially in could directly address questions to that student. the exercises, we must be able to act in a target-group-oriented In our course, the attendance of the weekly exercise sessions and adaptive manner to reach all students. Consequently, it is mandatory. The framework automatically flags students as is a challenge to design the course for students without any “attended”, if a tutor checks a solution to a subtask. Otherwise, previous experience and at the same time motivate advanced students have to enqueue to contact a tutor for requesting programmers. In an ideal case, students could learn from each attendance manually. other. We observed that restrained students hesitate to ask tutors We still explore and experiment with different prior knowl- when they get stuck on an exercise. Our framework monitors edge combinations and other mechanisms to motivate experi- the progress of each student pair in the background and en- enced programmers and educate beginners. We observed two queues a request for slow progressing students automatically. different patterns when students, having an unequal level of Thereby, tutors intervene and offer support early when students prior knowledge work together. Some experienced students fall behind. allow their partners to contribute to the solution and they C. Course Instructors Dashboard actively explain concepts to their partners. Unfortunately, the other pattern is that experienced programmers finish the Course instructors access the framework via a web interface. exercises quickly alone, explain the solution to their partner, For the tutor recruitment, they set up the questions for the and finally get the solution checked by a tutor and leave early. application form, which is an integrated part of the framework. We highlight this as a negative example in the lecture and ask Course instructors monitor both the application process as well the tutors to spot this pattern. as the distribution of students in the exercise sessions during the registration period. Applicants fill out the form with their III. S OCIO - TECHNICAL T EACHING F RAMEWORK personal information, additional references, information about In this section, we introduce our preliminary socio-technical previous grades, as well as their availability and how many face-to-face teaching framework that partly solves the chal- exercise sessions they plan to supervise. The framework auto- lenges we described in the previous section. We describe the matically indicates unbalanced exercise sessions and notifies framework from the perspective of the three roles: students, the course instructors. Course instructors constantly monitor tutors, and course instructors. the ratios between tutors and students for each time slot and cap the participation limits for students to register for the A. Students Web Interface exercise sessions. Students access the framework via a web interface. They Based on the data, collected in the exercise sessions, course access the current state of their evaluation sheet, exercise instructors acquire an insight into the current status of the SE1 sheets, lecture slides, and further references. At the beginning exercise part. The dashboard for course instructors offers an of each exercise session, each student pair logs into the system insight into the following metrics: and enters their seat and pair programming partner. During the • Ratio of students compliant with conditions. The exercise session, students enqueue requests for the tutors, for students in SE1 require to fulfill the conditions of the instance, an evaluation of a subtask or a question. Based on SE1 exercises. Our framework shows the proportion ISEE 2019: 2nd Workshop on Innovative Software Engineering Education @ SE19, Stuttgart, Germany 5 of students who are currently compliant with the SE1 V. D ISCUSSION AND C ONCLUSION exercise conditions, how many are behind, and how many Face-to-face exercises entail a huge logistic effort, which dropped out. covers diverse aspects including tutor management, improving • Attendance. Course instructors get an overview of when teaching material, and student communication. We report on students attend the exercise sessions. How many students major challenges that we identified in hosting the exercises of have to catch up with exercise sheets when they were SE1, a beginner level software development course with more not able to complete the exercise sheet in one exercise than 600 students with different levels of prior knowledge from session? We further use this information to fine-tune the different majors. We conduct face-to-face exercises assisted by balancing of tutors among the time slots. 50 tutors who assist and evaluate the students and identified • Complexity of exercises. As tutors check each subtask challenges including, sustaining hidden knowledge in case of an exercise sheet individually, course instructors get an of alternating teaching staff, reacting adaptively when tutors insight into the time it takes students to complete each are overwhelmed by a high number of students, monitoring subtask. Based on this metric, course instructors scope the workload of students, and dealing with heterogeneous and fine-tune each subtask so that solving a complete participants. exercise sheet is manageable in a three hour exercise We suggest a preliminary socio-technical teaching frame- session. work which keeps the face-to-face tutoring as an essential so- • Pair programming partner matching. Course instruc- cial part. Tutors enter the evaluation of the students’ solutions tors get an overview about the pair programming partner in an app. This data enables course instructors to gain insight combinations. Which pair programming partner com- into the practical exercise sessions to further improve the binations are common? What is their level of prior- learning experience for students in subsequent SE1 courses. So knowledge? Course instructors use these insights to adjust far, we implemented a proof of concept, but in the future, the future pair programming partner matching. complete framework has to become robust, secure and tested • Quality assurance for tutors. Each evaluation of a extensively to be deployed as it processes sensitive student student solution is different. It is crucial that tutors have data. uniform requirements for approving a subtask. Course ACKNOWLEDGMENT instructors are interested in questions such as: How many evaluations does each tutor perform? How much time We thank Axel Schmolitzky, Heinz Züllighoven, Fredrik does a tutor spend on checking each subtask? What is the Winkler, and Guido Gryczan for implementing this face-to- accept and reject ratio for each tutor? Course instructors face teaching approach in SE1 at the University of Hamburg. can discuss significant evaluation differences in the tutor R EFERENCES briefing sessions. [1] D. J. Barnes and M. Klling, Java lernen mit BlueJ: Eine Einfhrung in die objektorientierte Programmierung. Pearson Deutschland GmbH, 2009. [2] C. McDowell, L. Werner, H. Bullock, J. Fernald, C. McDowell, L. Werner, IV. R ELATED W ORK H. Bullock, and J. Fernald, “The effects of pair-programming on perfor- mance in an introductory programming course,” ACM SIGCSE Bulletin, Other researchers discussed teaching approaches for teach- vol. 34, pp. 38–42, Feb. 2002. ing software development face-to-face to benefit from im- [3] N. Nagappan, L. Williams, L. Williams, M. Ferzli, E. Wiebe, K. Yang, C. Miller, and S. Balik, “Improving the CS1 Experience with Pair mediate feedback loops. Kothiyal et al. [5] applied Think- Programming,” in Proceedings of the 34th SIGCSE Technical Symposium Pair-Share in a large introductory programming course. They on Computer Science Education, SIGCSE ’03, (New York, NY, USA), showed that they obtained a sustained engagement of 83%. pp. 359–362, ACM, 2003. [4] G. Baxter and I. Sommerville, “Socio-technical systems: From design This lecture integrated two Think-Pair-Share phases while methods to systems engineering,” Interacting with computers, vol. 23, students could ask an instructor for help. Krusche et al. [6] no. 1, pp. 4–17, 2011. reported on their experiences with an interactive learning ap- [5] A. Kothiyal, R. Majumdar, S. Murthy, and S. Iyer, “Effect of Think-pair- share in a Large CS1 Class: 83% Sustained Engagement,” in Proceedings proach. They pause the lecture in between and conduct in-class of the Ninth Annual International ACM Conference on International exercises to minimize the delay between theoretical knowledge Computing Education Research, ICER ’13, (New York, NY, USA), and practical application. They further show in another study pp. 137–144, ACM, 2013. [6] S. Krusche, N. v. Frankenberg, and S. Afifi, “Experiences of a Software [7] that active participation in these exercises leads to a better Engineering Course based on Interactive Learning,” in SEUH, 2017. exam grade. Krusche and Seitz [8] developed an automatic [7] S. Krusche, A. Seitz, J. Brstler, and B. Bruegge, “Interactive Learning: assessment system for large computer science courses to cope Increasing Student Participation Through Shorter Exercise Cycles,” in Proceedings of the Nineteenth Australasian Computing Education Con- with the manual assessment effort. Our approach differs from ference, ACE ’17, (New York, NY, USA), pp. 17–26, ACM, 2017. this approach by adding a social component in which the [8] S. Krusche and A. Seitz, “ArTEMiS: An Automatic Assessment Man- students have to present their solutions to a tutor. Iacob and agement System for Interactive Learning,” in Proceedings of the 49th ACM Technical Symposium on Computer Science Education, SIGCSE Faily [9] redesigned their second-year undergraduate course ’18, (New York, NY, USA), pp. 284–289, ACM, 2018. on software engineering for a large cohort. They also pointed [9] C. Iacob and S. Faily, “Redesigning an Undergraduate Software Engineer- out main challenges when hosting this course and concluded ing Course for a Large Cohort,” in Proceedings of the 40th International Conference on Software Engineering: Software Engineering Education that scalable exercise sessions are needed to teach software and Training, ICSE-SEET ’18, (New York, NY, USA), pp. 163–171, engineering skills. ACM, 2018. ISEE 2019: 2nd Workshop on Innovative Software Engineering Education @ SE19, Stuttgart, Germany 6