This paper describes the lessons learned during the creation, development, and delivery of the rst MicroMasters program. The MITx MicroMasters credential in Supply Chain Management, consisting of 5 MOOCs and a comprehensive nal exam (CFx), was launched in the Fall of 2015. As of March 2019, 26 course runs and 4 CFx exams have been delivered to over 280,000 learners. A total of 1,592 learners have earned the MicroMasters credential so far. The two key learnings from this experience are that MicroMasters are neither simply a collection of individual MOOCs nor are they the same as a residential in-person program. MicroMasters have unique properties and requirements that should be considered in their design and delivery.
recognized by employers for its real job relevance. Once a MicroMasters
credential is obtained, students may apply to various Masters programs. If accepted,
students can get credit for their online courses and pursue an accelerated
graduate program (Masters Degree) on campus. This process is also called "inverted
admissions" - accepting students on the bases of demonstrated competence in a
domain, in place of standardized test scores of undergraduate grades [
There are currently 52 MicroMasters programs across almost two dozen
universities. Collectively, these MicroMasters teams have conducted over 750
individual course runs with average enrollment of 7,000 learners and a veri ed to
registered (VTR) ratio of 2.5%. A total of 3,287 learners have successfully earned
a MicroMasters credential so far [
The remainder of the paper is organized as follows. In Section 2, we provide an overview of the program. In Section 3, we review the four of the primary challenges we faced and lessons learned running the rst MicroMasters program. Finally, in Section 4, we discuss the implications of running this program and present our conclusions. 2 2.1
The MITx MicroMasters R in Supply Chain Management was announced in Fall 2015. The program has been created and designed by MIT the Center for Transportation and Logistics (MIT CTL) and delivered on the edX platform. This program enables a learner to earn a Masters Degree from MIT's Supply Chain Management (SCM) program in just 5 months at MIT. This was the rst time that Massive Open Online Course (MOOC) could be applied for academic credit at MIT.
The program consists of ve MOOCs covering all aspects of logistics and supply chain management. To nish the program, students must successfully complete the ve online courses and a proctored comprehensive nal exam. As of February 2019, 26 SCx courses have been completed and a total of 4 Comprehensive Final Exams have been o ered. A total of 1,592 learners have earned the MicroMasters credential program in Supply Chain Management with 40 earning either a Master of Engineering or Master of Applied Science (MASC) degree from MIT {with 36 additional credential holders currently taking this program.
A funnel of participation [
We analyze the demographics of these twenty-six SCx courses and we found that these courses attract a similar population of learners. Between 142 and 192 countries represented in each course, top ve countries are USA (22%), India (12%), Brazil (4%), Canada (3%), and Mexico (3%). The median age was 31 years old, 24% females, and the majority of learners held a college degree (51%), or a M.Sc. degree (36%). 2.2
The ve graduate-level courses as part of the MITx MicroMasters Program in SCM include supply chain analytics (SC0x), supply chain fundamentals (SC1x), supply chain design (SC2x), supply chain dynamics (SC3x), and supply chain technology and systems (SC4x). The comprehensive nal exam (CFx) consists of two exams. Both exams are time-constrained and proctored.
We have two types of learners in our courses: audit and veri ed. Audit learners enroll the course and get access to the contents for free. Veri ed learners pay a fee (currently US$ 200) that grants them a certi cate if they pass ( nal grade equal or above 60%). Since anyone can take the courses for free, learners who pay to become veri ed demonstrate an intention to complete the course. Surprisingly, we have observed that 30%, on average, of our veri ed learners drop out before the end of the course in 2018. Enrollment in SCx courses varies between 6,000 and 33,000 students. The veri ed to registered ratio per course (percentage of veri ed learners/total enrollment) varies from 6% to 15%.
Each SCx course is structured in 13 weeks: one intro-to-course week (week 0), 8 content-based weeks (weeks 1 to 4 and 7 to 10), two o -weeks (weeks 5 and 11), and two weeks for midterm (week 6) and nal (week 12) exams. Every course week is released on Wednesday at 15:00 UTC. See Table 1.
The concepts of every content-based week are explained in two lessons, each of them provides a series of lecture videos interspersed with short questions called quick questions. Following the lessons, a set of practice problems o ers learners the opportunity to reinforce what they just learned. These problem sets provide a space to practice and receive immediate feedback. The median number of videos per course is 178, 64 Practice Problems, 109 Quick Questions and 17 Graded Assignments.
The program o ers each course twice a year. The main di erence between consecutive runs of the same course relies on the graded pieces. Graded assignments are updated every new run, while midterm and nal exams are created from scratch. 2.3
To earn the MITx MicroMasters Credential in Supply Chain Management (SCM) learners need to pass as a veri ed learner each individual SCx courses in the MITx MicroMasters Program in SCM and a Comprehensive Final Exam. Only those who pass the ve SCx courses qualify to take the CFx proctored and nal exam.
The assessment policy of each individual SCx courses in the MITx MicroMasters Program in Supply Chain Management includes (see Table 1): { Weekly Graded Assignment. This piece represents 20% of the total grade.
Learners have two weeks to complete them and solutions and explanations are released after the due date. { Midterm Exam. We o er one midterm exam in Week 6. The weight of the midterm is 35% of the total grade. The exam is open during one week, but once learners start the exam, they only have a limited time to complete it (typically 4 hours). { Final Exam. We o er one nal exam, at the end of the course, in Week 12.
Final represents 45% of the total grades. It is also a timed exam (4 hours exam) accessible during one week.
Before taking any assessment problem, learners need to ll and sign the academic honor code in the SCx courses (set as a pre-requisite in the edX platform to make it mandatory). Overall pass bar for each SCx course to gain a veried course certi cate is 60%. The Comprehensive Final Exam (CFx) consists of two proctored and timed exams. Both exams must be completed within a period of 12 hours. The CFx focuses on assessing their analytical skills for the methods and techniques taught across the ve SCx courses. It consists of several problems, each containing di erent questions. Typically, a combination of quantitative questions (e.g., numerical problems) and qualitative problems (e.g., check boxes questions). Overall pass bar for the CFx is 60% 3
MicroMasters programs, we discovered, di er in some critical ways from both the traditional methods of in-residence or face to face teaching, as well as from newer techniques used in standalone MOOCs. MicroMasters programs are more than a collection of individual MOOCs and are not just as an online version of a residential program. Best, or at least most common, practices in each of these two environments do not always translate well to a MicroMasters. This section summarizes four of the primary challenges we faced and lessons learned through this journey. They are: Distinguishing between learning and assessment, Managing a longer learner lifecycle, Sequencing courses and concepts, and Serving two distinct and di erent objectives. 3.1
MOOCs are tremendous learning platforms where feedback is instantaneous and collaboration between learners is both enabled and encouraged. In the rst SCx courses, run in 2014 and 2015 before being incorporated into a MicroMaster, we focused solely on creating a collaborative learning environment. Unfortunately, MicroMasters programs, because they carry potential academic credit, require rigorous and accurate assessment. Unfortunately, there is a fundamental trade-o involved with providing an educational platform for both learning and assessment. A platform that is ideally suited for a pure learning experience will, in all likelihood, not be able to provide an accurate assessment, and vice versa. The edX platform was initially designed and implemented primarily as a learning platform where the assessment capabilities were not a high priority.
Unfortunately, features that encourage learning and collaboration between learners, can enable potential academic dishonesty and cheating. In order to better meet both objectives (learning and assessment) we classi ed all portions of the courses to 10 di erent levels ranging from level 1 (pure learning) to level 10 (pure assessment). Figure 2 illustrates the key di erences between the learning and assessment environments. The pure learning sections (videos, quick questions, practice problems) received no grade points but allowed unfettered access to solutions and collaboration. The pure assessment sections (mid-term and nal exams), on the other hand, never displayed solutions, allowed minimal chances, and strictly forbid cross-learner collaboration. We also used all available randomization tools within the platform for these sections and also developed several of our own. We actually had to modify the platform to restrict the display of correct/incorrect solution indicators (the green check and red X) to problems during exams.
The weekly Graded Assignments, however, were a problem in that they are neither pure assessment nor pure learning. We tried a number of di erent combinations of settings to try to maximize learning and minimize potential cheating. In the end, we opted to lean more towards learning in these sections and lowered the points assigned to them so that the mid-term and nal exams counted for 80 percent of all points (see Table 1). We considered making the weekly Graded Assignments completely ungraded, but believed that by forcing a learner do some work each week, they would not procrastinate until right before an exam to study the material. By giving some, albeit a small amount, of credit to the Graded Assignments we forced them to study the material over a longer sustained period which we believe improves learning and retention.
There is no single best way to design an online course for both learning and assessment. The bottom line is - you need to determine which portions of your course are there for learning and which are there for assessment. Using the di erent settings and parameters within the platform allows you to customize each portion to their primary objective. While the extremes (100 percent learning and 100 percent assessment) are easier to design, the middle ground sections require more trial and error to nd the right mix of learning and assessment.
Lessons from the rst MicroMasters Program Also, the clear and consistent communication of these rules to the learners is critical to the overall success. MOOCs were, by and large, designed as stand alone courses with the platforms designed to track and monitor learner performance only within that course. To earn a MicroMasters, however, learners must complete a sequence of courses. This raised the need for managing, monitoring, and encouraging learners between courses - a capability that the existing platforms do not have.
To ll this gap, we have created our own o -line tool that pulls data from the platform and allows us to track each learners journey. But, this is a hack. It relies on the somewhat lagged data from the platform and does not always sync completely with the platform.
It is important to manage learners between courses because that is where most of the drop outs occurred. Once engaged in a speci c course, learners receive positive reinforcement to increase retention. Between courses, however, learners also need to be encouraged and managed in order to them to continue on their MicroMasters journey. 3.3
In virtually all graduate level in-residence programs in Supply Chain Management, the rst semester will include a comprehensive quantitative methods course. The idea is to provide the student with all of the mathematical tools they will need in the following courses in the program as early as possible. This makes sense, since all of the students have gone through an admissions process ensuring their preparedness and dropping out of a program in the very rst semester tends to be quite low for in-residence programs.
Online courses are just the opposite. The learners are neither captive nor vetted ahead of time. The cost of dropping an online course is essentially zero - as opposed to an in-residence program that involves full semester tuition, lodging costs, and typically relocation. Online students are more likely to drop because the cost of dropping out is exceptionally low. Because anyone can enroll in a MicroMasters course, the level of preparedness will vary dramatically from learner to learner.
We learned this the hard way. Applying typical "in-residence" best practices, we created SC0x Supply Chain Analytics as the rst or entry course in the MicroMasters sequence. It consists of fairly intensive mathematical techniques in optimization, statistics, and probability - with only slight emphasis on the domain of Supply Chain Management. By placing our most demanding course up front, we discouraged those learners who needed more mathematical preparation (typically the older learners that MicroMasters are intended to target!) right o the bat. The result was that we had the highest attrition within SC0x compared to the other four courses.
To remedy this, we have recently converted the course from instructor paced to on-demand. We now encourage learners to start with SC1x Supply Chain Fundamentals - a much more domain oriented course - and to take SC0x in pieces as they nd they need to better understand the concepts. We never considered just removing the more rigorous mathematical concepts - just the timing when they were introduced. In retrospect, we should have adopted a "explain the math when it is rst being applied" approach across all of the courses in the sequence. However, doing this retroactively is not currently possible.
The key lesson we learned is to never treat the learners as captive - their cost of dropping out is almost zero. This means that the courses and the course teams need to constantly romance the learners (within and between the courses) in order to retain them. 3.4
MicroMasters programs have a bipolar mission - by design. On one hand, they need to have su cient content to justify a standalone certi cate recognized by industry. On the other hand, they need to provide a foundation to a more intensive Master's degree program. Those goals are not always aligned.
It is important that the MicroMasters is not considered just a "prep" course for a Masters degree. This not only limits its value to industry, it severely restrict the number of learners interested in completing it. On the other hand, the course cannot just be a "tour" of the domain without any analytical rigor. It has to provide su cient background for a learner to be successful if they do continue on in a graduate degree program. Also, universities will have speci c expectations of what needs to be covered within a graduate degree.
Lessons from the rst MicroMasters Program
In order to meet this challenge, we took a critical look at what is best taught online and what is best taught face to face. We actually ended up modifying the graduate degree program that the MicroMasters feeds in to so that the learning online through the MicroMasters is better aligned to the traditional rst semester. This was a surprise to us - the online courses in uenced what and how we taught in residence.
Four speci c changes to the in-residence courses are worth mentioning. First, as expected, we were able to re-use the videos to enable the ipping of the classroom where we could replace traditional lectures with more engaging case discussions, exercises, games, and simulations. Second, we split several of our full semester courses, such as Supply Chain Finance and Global Supply Chains, into two parts: basic concepts and applied applications. The basic concepts portion is identical to the material covered in the online material within the MicroMasters courses. The second half of the courses were then able to feature more advanced concepts. By splitting the courses in half we further modularized the learning. This allowed students to access the advanced concepts without having to repeat the redundant material. Third, we have modi ed our acceptance criteria for the traditional in-residence masters program to no longer require taking the GRE or GMAT tests. Instead, if a learner earns an 85 or higher in SC0x Supply Chain Analytics, we waive the testing requirement. We have found that while performance on the GRE/GMAT had no correlation to success in the program, performance in the online courses does. It is a better indicator of mastery of the techniques and concepts the students actually need in the program. Finally, we have incorporated some of the online feedback and testing tools within the in-residence classes. For example, we have nally stopped using bluebooks for exams where the student had to handwrite their solutions. We replaced them with more realistic problems using the edX platform. This not only dramatically sped up the grading process, it allowed us to let them use selected software packages that made the testing more relevant. Overall, our experience with the MicroMasters program improved the on campus educational experience. 4
Over the course of developing (and revising) ve courses and delivering 26 course runs, we have identi ed four key challenges that any team creating a MicroMasters should consider. The rst is to clearly distinguish between the learning and assessment environments within their courses. Unlike a standalone MOOC, assessment is just as important as learning. While some other MicroMasters programs have since moved to use a separate specialized platform just for assessment, we have not as we want to retain a common look and feel across both the learning and assessment environments.
The second challenge is learning to manage a longer learner lifecycle. Most attrition occurs between, not within, courses in a MicroMasters sequence. Be sure to have a way to monitor and manage your learners between the course runs to increase retention.
The third challenge is to match the sequencing of the courses and concepts to the of the courses. They are neither vetted nor captive - so the introduction of complex material needs to be paced out accordingly. We never recommend "dumbing down" any content, but to be more cognizant of the timing.
Finally, all MicroMasters, by de nition, will be bipolar in that they must meet two distinct and di erent objectives. Any MicroMasters team needs to consider both populations (those desiring a masters degree and those not) when designing their program. The in-residence program's curriculum might need to be modi ed as well as the online components.
The over-arching lesson we learned over the last three years is that a "MicroMasters program is more than the sum of its individual MOOCs".
M icroM asters > SU M (M OOCs) (1) And, it is not just an online version of an in-residence program. Because it has unique characteristics, it needs to be designed and delivered di erently. As next steps, we are currently designing strategies to increase learners engagement in the program, to reduce the program dropout rate, and nally to implement strategies that add value to the MicroMasters program.