A Glimpse on How MOOCs from IDB are Impacting Learners in Latin America Edgar González1, Antonio García2, Carlos Macher3 and Dou Zhang4 1 Knowledge and Learning Lead Specialist, IDBx Program Manager, Inter-American Development Bank (IDB), Washington, DC, USA 2 Operations Coordinator, IDBx Program, IDB, Washington, DC, USA 3 e-Learning Technologist and Data Czar, IDBx Program, IDB, Washington, DC, USA 4 Data Analyst, IDBx Program, IDB, Washington, DC, USA Abstract. The Inter-American Development Bank (IDB) is working to improve lives in Latin America and the Caribbean. IDB is offering Massive Open Online Courses (MOOCs) on key social and economic topics to scale up the capacity of development practitioners in the region. This paper presents preliminary findings of the first survey exploring how these MOOCs are impacting the academic, professional and social life of learners in Latin America. Enabling and restricting factors to use the knowledge gained in the MOOCs at the learners’ workplace were identified. Our findings show that MOOCs on social and economic topics offer valuable learning opportunities, in- spire learners and empower them to apply their knowledge to improve their lives, es- pecially in developing countries where access to formal education is more limited and where access to, use of, and skills in information and communication technology are more seldom. Keywords: IDB, MOOC, Development, Impact, Latin America 1 Introduction The Inter-American Development Bank (IDB) works to improve lives in Latin America and the Caribbean (LAC). The IDB is the leading source of development financing for this region and provides loans, grants, technical assistance; and conducts extensive re- search. As a multilateral development agency, the IDB maintains a strong commitment to achieving measurable results and the highest standards of increased integrity, trans- parency, and accountability. The IDB is also committed to sharing knowledge on economic and social develop- ment topics to strengthen the capacity of decision-makers and development actors in the region and to provide evidence of what works and what does not work in terms of development policies, programs and projects. This knowledge is the result of thousands of development operations implemented over the years, collected and distilled through project evaluations, research findings, publications, lessons learned, and case studies. In addition to the open educational resources, online tutor-led and face-to-face courses, the IDB is now offering a series of Massive Open Online Courses (MOOCs), available through IDBx, its training initiative in edX. Proceedings of the International Conference MOOC-MAKER 2017. 53 Antigua Guatemala, Guatemala, November 16-17, 2017. The objective of this paper is to discuss our preliminary findings on how IDBx MOOCs are impacting learners in Latin America. 2 The IDBx Program 2.1 MOOCs offered The IDBx Program started on September 30, 2014 with the launching of its first MOOC and has offered a total of 62 MOOCs (25 new courses and 37 re-runs) up to September 10, 2017. Those MOOCs have been offered in the four official lan- guages of the Inter-American Develop- ment Bank (currently 46 in Spanish, 8 in English, 6 in Portuguese, and 2 in French). Fig. 1. Distribution of enrolled learners The main subject matter areas in IDBx MOOCs include management for development results, project management, sustaina- ble development of cities, macroeconomics, public-private partnerships, urban devel- opment, pensions, management of water resources, trade agreements, early childhood development, education for climate change, agricultural policy food safety and climate change, and social development. 2.2 IDBx learners Up to September 10, 2017, 629,089 learners have enrolled in IDBx MOOCs. Out of the en- rolled learners, 301,837 (45% of enrolled learners) have actively participated in those courses, by accessing its learning resources like videos, readings, discussion forum, in- fographics, case studies, and assessments, among others. Of the active learners (partici- pants), 230,360 of them are Exploring Partic- ipants (37% of enrolled learners), whom took advantage up to 49% of the available learning resources, while 71,480 are Advanced Partic- ipants (11% of enrolled learners), whom took advantage of 50% or more. 56,808 partici- Fig. 2. Cumulative number of enrolled, pants (9% of enrollments) completed the participating, completed and certified course, this is, obtained a passing grade of 65 learners (in thousands) by Sep.10, 2017 or more points out of 100, and were eligible to obtain a certificate. 42,468 (7% of enrollments) obtained a certificate (a free honor code certificate until the end of 2015 or a verified certificate by verifying their identity and paying US$25 starting in 2016) [1] (Fig. 2). Proceedings of the International Conference MOOC-MAKER 2017. 54 Antigua Guatemala, Guatemala, November 16-17, 2017. Country of residency of enrolled learners. The number of countries and territories in which IDBx enrolled learners live range from 45 to 193 depending on the course. How- ever, 84.1% of all enrolled learners live in the 26 borrowing member countries of the IDB in LAC [1] (Fig. 1). Fig. 3 presents the top 10 countries of residency by the number of enrolled learners in all MOOCs [1], while Fig. 4 presents the top 10 countries of residency by the number of enrolled learners as a ratio of their Economically Active Population (EAP) per 100,000 inhabitants [1] [2]. Fig. 3. Top 10 countries by the number of Fig. 4. Top 10 countries by the number of enrolled learners (in thousands) enrolled learners as a ratio of EAP per 100,000 inhabitants Course completion per country and Human Development Index (HDI). The data shows that learners in the less developed countries of the 26 countries in the region [3] are taking more advantage of the MOOCs offered. Those countries have a higher ratio of learners who completed the course in relation with enrolled users when compared with countries with higher HDI [1][4] (Fig. 5). This suggests that MOOCs are more appreciated in those countries where learning opportunities are more limited. Fig. 5. Ratio of completed participants in relation to enrolled learners plotted by HDI of 26 IDB borrowing countries (the size of each circle represents the number of enrolled learners in that country by Sep. 10, 2017) Proceedings of the International Conference MOOC-MAKER 2017. 55 Antigua Guatemala, Guatemala, November 16-17, 2017. Age, gender, and level of education of enrolled learners. Out of all enrolled learners in IDBx, 52.1% are male, while 43.9% are female, 92.9% are within economically ac- tive age, 35.8% have a bachelor’s degree and 34.4% have master’s degree [1]. Fig. 6 and 7 presents more detail about their demographic characteristics. Fig. 6. Age group of enrolled learners Fig. 7. Education of enrolled learners Occupation of enrolled learners. In line with the intended target audience, most learn- ers enrolled in IDBx MOOCs are development practitioners working in the public sec- tor (33.2%), the private sector (22.3%), con- sultants and other independent occupations (8.8%), NGOs (8.6%), and international or- ganizations (3.2%) [5]. 21.4% of enrolled learners come from the academia. We have found that a high number of teachers are using our courses to improve their professional practice and that an increas- ing number of graduate and postgraduate students are interested about development Fig. 8. Occupation of enrolled learners issues [5] (Fig. 8). 2.3 Learners’ satisfaction with IDBx MOOCs The IDBx Program conducted Kirkpatrick level-1-surveys [6], also known as reaction or satisfaction surveys, weekly during course delivery and a final survey at the end of each course, to evaluate learners’ perception of the course, including the achievement of learning objectives (course and module objectives); the sufficiency, appropriateness, relevance and currency of course content; the quality of learning resources and activi- ties (videos, readings, interactive forum, formative and summative evaluations, etc.), in terms of their contribution to the learning process; as well as, the usefulness of each course in relation to the applicability of its contents to the learners’ context. Proceedings of the International Conference MOOC-MAKER 2017. 56 Antigua Guatemala, Guatemala, November 16-17, 2017. The response rate of the weekly surveys ranged from 14.9% to 17.2% of participants, while the response rate for the final survey ranged from 10.9% to 13.3% of participants. The IDBx MOOC Quality Index (MQI), the simple average of those indicators rated by learners in all courses delivered during 2014-2017, is 4.56 out of 5.0, which indicates a high level of learners’ satisfaction. As cultural context, relevance and practicality are critical for adult learning, IDBx MOOCs are designed and developed considering regional and cultural complexities, similarities, and differences among LAC countries, including offering the courses in their national language. This creates a positive sense of belonging and social identity for participants of the region, which is conducive to learning, in contrast with findings reported for MOOCs not tailored to the regional culture and offered only in English [6]. To promote use of knowledge, IDBx MOOCs intend to provide guidance, inspiration, tools, methodologies, case studies and examples relevant and useful to development practitioners in LAC countries, as well as, to put course participants in the mindset of seeing themselves as agents of change of their own reality (personal, organizational, institutional and social). 2.4 Perceived learners’ knowledge gain As part of the final survey in each MOOC, learners reported in a 1-10 scale their per- ception of their level of knowledge on the topics of the course before and after taking each course. Using this data, the IDBx program estimated the perceived learner’s knowledge gain, as the difference, measured in percentage points (PP), between levels of knowledge reported by those learners before and after the course. The average knowledge gain in all MOOCs delivered during 2014-2017 is 34.5 PP. 3 Impact of IDB MOOCs on Learners in Latin America 3.1 Characteristics of the first survey On July 2017, the IDBx Program sent its first Kirkpatrick level-3-survey [7] to 46,563 learners who accessed 30% or more of the available learning resources of 25 MOOCs in Spanish offered from 2014 to 2016. The goal of this survey was to collect data on how MOOCs were impacting the academic, professional and social life of learners in Latin America. 6,582 learners answered the survey (14.1% response rate). 3.2 Results of the survey Through a series of closed and open questions, the level-3-survey explored the impact of MOOCs on learners’ academic life, professional/work life, including impact in their workplace, as well as in their social life. This paper presents below the data collected through closed questions. Proceedings of the International Conference MOOC-MAKER 2017. 57 Antigua Guatemala, Guatemala, November 16-17, 2017. Impact on learners’ academic life. 74% of learners agreed that the MOOC helped them to decide what to study or research in the future, 64.9% reported that improved their academic performance, while for 61.6% of learners the course inspired them to initiate or reinitiate their studies [8] (Fig. 9). Fig. 9. Impact of IDBx MOOCs on learners’ academic life Impact on learners’ professional/work life. The most noticeable impacts reported were on improving knowledge and skills for their current job (93.2%), improving pro- fessional performance (91.7%), increasing leadership and influence in the workplace (78.7%) and in their professional career (67%), as well as improving their standing for a new job (59.5%) [8] (Fig. 10). Fig. 10. Impact of IDBx MOOCs on learners’ professional/work life Proceedings of the International Conference MOOC-MAKER 2017. 58 Antigua Guatemala, Guatemala, November 16-17, 2017. Impact on learners’ workplace. 87.61% of the learners who answered the survey re- ported using the knowledge gained in IDBx MOOCs to improve their workplace or organizational setting. When analyzing the knowledge use per country in relation with their respective Information and Communication Technology (ICT) Development In- dex (IDI), which integrates ICT readiness, use and capability [9], we observed that a higher percentage of learners used their knowledge in countries with lower IDI in rela- tion with learners in countries with higher IDI (Fig. 11). This suggests that learners in less developed countries were more eager to take advantage of their learning opportu- nities and use the knowledge gained to face the restrictions and challenges they have. Fig. 11. Knowledge use per country plotted by ICT Development Index (IDI) Of the learners who reported using their knowledge to improve their workplace or or- ganizational setting, 90.6% agreed that they were more productive/efficient, 88.3% ap- plied new/innovative methodologies or technologies in their work, 86.9% were able to advance existing or new initiatives, projects or programs, and 78.3% improved their leadership and influence at work [8] (Fig. 12). Fig. 12. Impact of IDBx MOOCs on learners’ workplace Proceedings of the International Conference MOOC-MAKER 2017. 59 Antigua Guatemala, Guatemala, November 16-17, 2017. Enabling factors to use knowledge in learners’ workplace. Participants reported that the main enabling factor was the tools and methodologies provided by the MOOC (94.7%), followed by the availability of time at work (83.6%), the flexibility of existing norms and procedures (62%), the support from supervisors and coworkers (53.8%), and the availability of financial resources (42.5%) [8] (Fig. 13). Fig. 13. Facilitating factors to use knowledge in learners’ workplace Restricting factors to use knowledge in learners’ workplace. For those learners who reported not being able to use their knowledge in their workplace, the main reported restricting factor was the rigidity of norms and procedures (36.3%), followed by the lack of financial resources (24.6%), lack of time at work (23.1%), lack of support from supervisors or coworkers (20.3%), and lack of tools/methodologies to use their knowledge (9.5%) [8] (Fig. 14). Fig. 14. Restricting factors to use knowledge in learners’ workplace Proceedings of the International Conference MOOC-MAKER 2017. 60 Antigua Guatemala, Guatemala, November 16-17, 2017. Impact on learners’ social life. 63% of learners reported that people in their commu- nities benefited from initiatives, projects or programs implemented by them, 57.7% re- ported that people in their families, workplaces or communities changed their behavior because of those initiatives, projects or programs, and 45.2% reported that family and friends also benefited from those initiatives, projects or programs implemented by learners [8] (Fig. 15). Fig. 15. Impact of IDBx MOOCs on learners’ social life Discussion and conclusions Further analysis of current data as well as additional surveys are necessary to better identify possible patterns. However, it is clear, even at this early stage, that MOOCs are positively impacting the academic, professional and social life of learners in Latin America. Despite many comments stating that MOOCs have not fulfilled their promise, our findings show that MOOCs covering social and economic topics offer valuable learning opportunities, inspire learners and empower them to apply their knowledge to improve their lives, especially in developing countries where access to formal education is more limited and where access to, use of, and skills in information and communication tech- nology are more seldom. Knowledge use was reported higher in less developed countries. One hypothesis to explain this could be that, although possibly counterintuitive, learners in these countries were more eager to take advantage of these learning opportunities. They cited in their responses that both time and flexibility were important factors in exploring such oppor- tunities to use their acquired knowledge. It could also be argued that learners in less developed countries have more room to be more productive and efficient, to apply new and innovative methodologies or technologies in their work, to advance existing or new initiatives, projects or programs, and to improve their leadership and influence at work. If this is the case, MOOCs could be considered as an effective open knowledge tool for leveling the plainfield amongst various LAC countries, strengthening organizations and promoting social and economic development. Proceedings of the International Conference MOOC-MAKER 2017. 61 Antigua Guatemala, Guatemala, November 16-17, 2017. In line with findings in previous research [10] [11], we observe that many learners in IDBx MOOCs stay engaged and committed to learning without taking assessments nor with the intention of completing the course or get certified. In fact, so far, we have not found any statistical difference between learners who were able, and not able to use their knowledge when it comes to their progress in the course (this is, exploring learn- ers, advanced learners, completing and certified learners) [1][5]. Assessing the impact of massive training initiatives is not easy, especially when learners display a wide spectrum of social and economic conditions that affect their access and capacity for acquisition of knowledge, and when there are many enabling and restricting factors to use the knowledge gained. This paper is just a first glimpse of how MOOCs are impacting learners in Latin America. Further research and data anal- ysis will bring a better understanding on this issue. Continued efforts from international organizations, governments and academic in- stitutions are still needed to maximize the potential of MOOCs as an essential tool for capacity building and socio-economic development in LAC countries. References 1. IDBx Data Engine 2017, last accessed 2017/09/10. (Figures 1, 2, 3, 4, 6 and 7). 2. World Bank, World Development Indicators, http://data.worldbank.org/data-catalog/world- development-indicators, last accessed 2017/08/14. (Figures 4 and 5). 3. Inter-American Development Bank, http://www.iadb.org/en/about-us/borrowing-member- countries,6005.html. 4. UNDP, Human Development Reports, http://hdr.undp.org/en/indicators/137506#, last ac- cessed 2017/09/14. (Figure 5, 11) [Note: HDI is composed of 3 dimensions: Life expectancy, education and per capita income]. 5. IDBx Initial Survey 2014-2017, last accessed 2017/09/11. (Figure 8). 6. Kizilcec, R., Saltarelli, A., Reich, J., Cohen, G.: Closing global achievement gaps in MOOCs. In Science 355 (6322), 251-252. [doi: 10.1126/science.aag2063] (2017) 7. Kirkpatrick, D. L. & Kirkpatrick, J. D., Evaluating Training Programs, Berret-Koehler Pub- lishers (1994). 8. IDBx Level 3 Survey in Spanish 2017, last accessed 2017/09/11. (Figures 9, 10, 11, 12, 13, 14 and 15). 9. International Telecommunication Union, Measuring the Information Society Report 2015, Place des Nations CH-1211, Geneva, Switzerland, ISBN 978-92-61-15791-3 [Note: IDI is composed by indicators for ICT access (fixed-telephone and mobile-cellular subscriptions per 100 inhabitants, international Internet bandwidth per internet user, % of households with a computer, % of households with Internet access), ICT use (% of individuals using Internet, fixed-broadband and active mobile broadband subscriptions per 100 inhabitants), and ICT skills (adult literacy rate, secondary and tertiary gross enrollment ratio). 10. DeBoer, J., Ho, A.D., Stump, G.S., & Breslow, L.: Changing “course”: Reconceptualizing educational variables for massive open online courses. Educational Researcher, 43(2), 74- 84 (2014). 11. Kizilcec, R. F., Piech, C., & Schneider, E.: Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. In Proceedings of the Third Interna- tional Conference on Learning Analytics and Knowledge, pp. 170-9. New York, NY, USA ACM. doi:10.1145/2460296.24603304 (2013). Proceedings of the International Conference MOOC-MAKER 2017. 62 Antigua Guatemala, Guatemala, November 16-17, 2017.