A Glimpse on How MOOCs from IDB are Impacting Learners in Latin America

Abstract

The Inter-American Development Bank (IDB) is working to improve lives in Latin America and the Caribbean. IDB is offering Massive Open Online Courses (MOOCs) on key social and economic topics to scale up the capacity of development practitioners in the region. This paper presents preliminary findings of the first survey exploring how these MOOCs are impacting the academic, professional and social life of learners in Latin America. Enabling and restricting factors to use the knowledge gained in the MOOCs at the learners' workplace were identified. Our findings show that MOOCs on social and economic topics offer valuable learning opportunities, inspire learners and empower them to apply their knowledge to improve their lives, especially in developing countries where access to formal education is more limited and where access to, use of, and skills in information and communication technology are more seldom.

Keywords

• IDB
• MOOC
• Development
• Impact
• Latin America

Introduction

The Inter-American Development Bank (IDB) works to improve lives in Latin America and the Caribbean (LAC). The IDB is the leading source of development financing for this region and provides loans, grants, technical assistance; and conducts extensive research. As a multilateral development agency, the IDB maintains a strong commitment to achieving measurable results and the highest standards of increased integrity, transparency, and accountability.

The IDB is also committed to sharing knowledge on economic and social development topics to strengthen the capacity of decision-makers and development actors in the region and to provide evidence of what works and what does not work in terms of development policies, programs and projects. This knowledge is the result of thousands of development operations implemented over the years, collected and distilled through project evaluations, research findings, publications, lessons learned, and case studies. In addition to the open educational resources, online tutor-led and face-to-face courses, the IDB is now offering a series of Massive Open Online Courses (MOOCs), available through IDBx, its training initiative in edX.

The objective of this paper is to discuss our preliminary findings on how IDBx MOOCs are impacting learners in Latin America. 2 2.1

The IDBx Program

MOOCs offered

The IDBx Program started on September

30, 2014 with the launching of its first MOOC and has offered a total of 62 MOOCs (25 new courses and 37 re-runs) up to September 10, 2017. Those MOOCs have been offered in the four official languages of the Inter-American Development Bank (currently 46 in Spanish, 8 in English, 6 in Portuguese, and 2 in French).

The main subject matter areas in IDBx

MOOCs include management for development results, project management, sustainable development of cities, macroeconomics, public-private partnerships, urban development, pensions, management of water resources, trade agreements, early childhood development, education for climate change, agricultural policy food safety and climate change, and social development. Up to September 10, 2017, 629,089 learners have enrolled in IDBx MOOCs. Out of the enrolled learners, 301,837 (45% of enrolled learners) have actively participated in those courses, by accessing its learning resources like videos, readings, discussion forum, infographics, case studies, and assessments, among others. Of the active learners (participants), 230,360 of them are Exploring Participants (37% of enrolled learners), whom took advantage up to 49% of the available learning resources, while 71,480 are Advanced Participants (11% of enrolled learners), whom took advantage of 50% or more. 56,808 partici- Fig. 2. Cumulative number of enrolled, pants (9% of enrollments) completed the participating, completed and certified course, this is, obtained a passing grade of 65 learners (in thousands) by Sep.10, 2017 or more points out of 100, and were eligible to obtain a certificate. 42,468 (7% of enrollments) obtained a certificate (a free honor code certificate until the end of 2015 or a verified certificate by verifying their identity and paying US$25 starting in 2016) [ 1 ] (Fig. 2). Country of residency of enrolled learners. The number of countries and territories in which IDBx enrolled learners live range from 45 to 193 depending on the course. However, 84.1% of all enrolled learners live in the 26 borrowing member countries of the IDB in LAC [ 1 ] (Fig. 1). Fig. 3 presents the top 10 countries of residency by the number of enrolled learners in all MOOCs [ 1 ], while Fig. 4 presents the top 10 countries of residency by the number of enrolled learners as a ratio of their Economically Active Population (EAP) per 100,000 inhabitants [ 1 ] [ 2 ].

Course completion per country and Human Development Index (HDI). The data

shows that learners in the less developed countries of the 26 countries in the region [ 3 ] are taking more advantage of the MOOCs offered. Those countries have a higher ratio of learners who completed the course in relation with enrolled users when compared with countries with higher HDI [ 1 ][ 4 ] (Fig. 5). This suggests that MOOCs are more appreciated in those countries where learning opportunities are more limited.

Age, gender, and level of education of enrolled learners. Out of all enrolled learners

in IDBx, 52.1% are male, while 43.9% are female, 92.9% are within economically active age, 35.8% have a bachelor’s degree and 34.4% have master’s degree [ 1 ]. Fig. 6 and 7 presents more detail about their demographic characteristics. Occupation of enrolled learners. In line with the intended target audience, most learners enrolled in IDBx MOOCs are development practitioners working in the public sector (33.2%), the private sector (22.3%), consultants and other independent occupations (8.8%), NGOs (8.6%), and international organizations (3.2%) [ 5 ].

21.4% of enrolled learners come from the academia. We have found that a high number of teachers are using our courses to improve their professional practice and that an increasing number of graduate and postgraduate students are interested about development issues [ 5 ] (Fig. 8). The IDBx Program conducted Kirkpatrick level-1-surveys [ 6 ], also known as reaction or satisfaction surveys, weekly during course delivery and a final survey at the end of each course, to evaluate learners’ perception of the course, including the achievement of learning objectives (course and module objectives); the sufficiency, appropriateness, relevance and currency of course content; the quality of learning resources and activities (videos, readings, interactive forum, formative and summative evaluations, etc.), in terms of their contribution to the learning process; as well as, the usefulness of each course in relation to the applicability of its contents to the learners’ context.

The response rate of the weekly surveys ranged from 14.9% to 17.2% of participants, while the response rate for the final survey ranged from 10.9% to 13.3% of participants.

The IDBx MOOC Quality Index (MQI), the simple average of those indicators rated by learners in all courses delivered during 2014-2017, is 4.56 out of 5.0, which indicates a high level of learners’ satisfaction.

As cultural context, relevance and practicality are critical for adult learning, IDBx MOOCs are designed and developed considering regional and cultural complexities, similarities, and differences among LAC countries, including offering the courses in their national language. This creates a positive sense of belonging and social identity for participants of the region, which is conducive to learning, in contrast with findings reported for MOOCs not tailored to the regional culture and offered only in English [ 6 ]. To promote use of knowledge, IDBx MOOCs intend to provide guidance, inspiration, tools, methodologies, case studies and examples relevant and useful to development practitioners in LAC countries, as well as, to put course participants in the mindset of seeing themselves as agents of change of their own reality (personal, organizational, institutional and social). 2.4

Perceived learners’ knowledge gain

As part of the final survey in each MOOC, learners reported in a 1-10 scale their perception of their level of knowledge on the topics of the course before and after taking each course. Using this data, the IDBx program estimated the perceived learner’s knowledge gain, as the difference, measured in percentage points (PP), between levels of knowledge reported by those learners before and after the course. The average knowledge gain in all MOOCs delivered during 2014-2017 is 34.5 PP. 3 3.1

Impact of IDB MOOCs on Learners in Latin America

Characteristics of the first survey

On July 2017, the IDBx Program sent its first Kirkpatrick level-3-survey [ 7 ] to 46,563 learners who accessed 30% or more of the available learning resources of 25 MOOCs in Spanish offered from 2014 to 2016. The goal of this survey was to collect data on how MOOCs were impacting the academic, professional and social life of learners in Latin America. 6,582 learners answered the survey (14.1% response rate). 3.2

Results of the survey

Through a series of closed and open questions, the level-3-survey explored the impact of MOOCs on learners’ academic life, professional/work life, including impact in their workplace, as well as in their social life. This paper presents below the data collected through closed questions. Impact on learners’ academic life. 74% of learners agreed that the MOOC helped them to decide what to study or research in the future, 64.9% reported that improved their academic performance, while for 61.6% of learners the course inspired them to initiate or reinitiate their studies [ 8 ] (Fig. 9).

Impact on learners’ professional/work life. The most noticeable impacts reported

were on improving knowledge and skills for their current job (93.2%), improving professional performance (91.7%), increasing leadership and influence in the workplace (78.7%) and in their professional career (67%), as well as improving their standing for a new job (59.5%) [ 8 ] (Fig. 10). Impact on learners’ workplace. 87.61% of the learners who answered the survey reported using the knowledge gained in IDBx MOOCs to improve their workplace or organizational setting. When analyzing the knowledge use per country in relation with their respective Information and Communication Technology (ICT) Development Index (IDI), which integrates ICT readiness, use and capability [ 9 ], we observed that a higher percentage of learners used their knowledge in countries with lower IDI in relation with learners in countries with higher IDI (Fig. 11). This suggests that learners in less developed countries were more eager to take advantage of their learning opportunities and use the knowledge gained to face the restrictions and challenges they have. Of the learners who reported using their knowledge to improve their workplace or organizational setting, 90.6% agreed that they were more productive/efficient, 88.3% applied new/innovative methodologies or technologies in their work, 86.9% were able to advance existing or new initiatives, projects or programs, and 78.3% improved their leadership and influence at work [ 8 ] (Fig. 12).

Enabling factors to use knowledge in learners’ workplace. Participants reported that

the main enabling factor was the tools and methodologies provided by the MOOC (94.7%), followed by the availability of time at work (83.6%), the flexibility of existing norms and procedures (62%), the support from supervisors and coworkers (53.8%), and the availability of financial resources (42.5%) [ 8 ] (Fig. 13).

Restricting factors to use knowledge in learners’ workplace. For those learners who

reported not being able to use their knowledge in their workplace, the main reported restricting factor was the rigidity of norms and procedures (36.3%), followed by the lack of financial resources (24.6%), lack of time at work (23.1%), lack of support from supervisors or coworkers (20.3%), and lack of tools/methodologies to use their knowledge (9.5%) [ 8 ] (Fig. 14). Impact on learners’ social life. 63% of learners reported that people in their communities benefited from initiatives, projects or programs implemented by them, 57.7% reported that people in their families, workplaces or communities changed their behavior because of those initiatives, projects or programs, and 45.2% reported that family and friends also benefited from those initiatives, projects or programs implemented by learners [ 8 ] (Fig. 15). Further analysis of current data as well as additional surveys are necessary to better identify possible patterns. However, it is clear, even at this early stage, that MOOCs are positively impacting the academic, professional and social life of learners in Latin America.

Despite many comments stating that MOOCs have not fulfilled their promise, our findings show that MOOCs covering social and economic topics offer valuable learning opportunities, inspire learners and empower them to apply their knowledge to improve their lives, especially in developing countries where access to formal education is more limited and where access to, use of, and skills in information and communication technology are more seldom.

Knowledge use was reported higher in less developed countries. One hypothesis to explain this could be that, although possibly counterintuitive, learners in these countries were more eager to take advantage of these learning opportunities. They cited in their responses that both time and flexibility were important factors in exploring such opportunities to use their acquired knowledge. It could also be argued that learners in less developed countries have more room to be more productive and efficient, to apply new and innovative methodologies or technologies in their work, to advance existing or new initiatives, projects or programs, and to improve their leadership and influence at work. If this is the case, MOOCs could be considered as an effective open knowledge tool for leveling the plainfield amongst various LAC countries, strengthening organizations and promoting social and economic development.

In line with findings in previous research [ 10 ] [ 11 ], we observe that many learners in IDBx MOOCs stay engaged and committed to learning without taking assessments nor with the intention of completing the course or get certified. In fact, so far, we have not found any statistical difference between learners who were able, and not able to use their knowledge when it comes to their progress in the course (this is, exploring learners, advanced learners, completing and certified learners) [ 1 ][ 5 ].

Assessing the impact of massive training initiatives is not easy, especially when learners display a wide spectrum of social and economic conditions that affect their access and capacity for acquisition of knowledge, and when there are many enabling and restricting factors to use the knowledge gained. This paper is just a first glimpse of how MOOCs are impacting learners in Latin America. Further research and data analysis will bring a better understanding on this issue.

Continued efforts from international organizations, governments and academic institutions are still needed to maximize the potential of MOOCs as an essential tool for capacity building and socio-economic development in LAC countries.

References

1. IDBx Data Engine 2017 , last accessed 2017 /09/10. ( Figures 1 , 2 , 3 , 4 , 6 and 7).

2. World Bank, World Development Indicators, http://data.worldbank.org/data-catalog/worlddevelopment-indicators, last accessed 2017 /08/14. ( Figures 4 and 5 ).

3. Inter-American Development Bank, http://www.iadb.org/en/about-us/borrowing-membercountries, 6005 .html.

4. UNDP, Human Development Reports, http://hdr.undp.org/en/indicators/137506#, last accessed 2017 /09/14. ( Figure 5 , 11) [Note: HDI is composed of 3 dimensions: Life expectancy, education and per capita income].

5. IDBx Initial Survey 2014-2017, last accessed 2017 /09/11. ( Figure 8 ).

6. Kizilcec , R. , Saltarelli , A. , Reich, J., Cohen, G. : Closing global achievement gaps in MOOCs . In Science 355 ( 6322 ), 251 - 252 . [doi: 10.1126/science.aag2063] ( 2017 )

7. Kirkpatrick , D. L. & Kirkpatrick , J. D. , Evaluating Training Programs, Berret-Koehler Publishers ( 1994 ).

8. IDBx Level 3 Survey in Spanish 2017 , last accessed 2017 /09/11. ( Figures 9 , 10 , 11 , 12 , 13 , 14 and 15).

9. International Telecommunication Union , Measuring the Information Society Report 2015 , Place des Nations CH- 1211 , Geneva, Switzerland, ISBN 978 -92-61-15791-3 [Note: IDI is composed by indicators for ICT access (fixed-telephone and mobile-cellular subscriptions per 100 inhabitants, international Internet bandwidth per internet user, % of households with a computer, % of households with Internet access), ICT use (% of individuals using Internet, fixed-broadband and active mobile broadband subscriptions per 100 inhabitants), and ICT skills (adult literacy rate, secondary and tertiary gross enrollment ratio).

10. DeBoer , J., Ho , A.D. , Stump , G.S. , & Breslow , L. : Changing “course”: Reconceptualizing educational variables for massive open online courses . Educational Researcher , 43 ( 2 ), 74 - 84 ( 2014 ).

11. Kizilcec , R. F. , Piech , C. , & Schneider , E. : Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses . In Proceedings of the Third International Conference on Learning Analytics and Knowledge , pp. 170 - 9 . New York, NY, USA ACM. doi: 10 .1145/2460296.24603304 ( 2013 ).