This paper provides a progress report on the first 18 months of Phase 1, the conceptualization phase, of the Learner Data Institute (LDI; www.learnerdatainstitute.org). LDI is currently in Phase 1, the conceptualization phase, to be followed by Phase 2, the institute or convergence phase. The current 2-year conceptualization phase has two major goals: (1) develop, implement, evaluate, and refine a framework for data-intensive science and engineering for the future institute, and (2) use the framework to provide prototype solutions, based on data, data science, and science convergence, to a number of core challenges in learning science and engineering. By targeting a critical mass of key challenges that are at a tipping point, LDI aims to start a chain reaction that will transform the whole learning ecosystem. We will emphasize here the key elements of the LDI science convergence framework that our team developed, implemented, and now is in the process of evaluating and refining. We highlight important outcomes of the convergence framework and related processes, including a 5-year plan for the institute phase and data-intensive prototype solutions to transform the learning ecosystem.
This paper provides a progress report on the first 18 months of the two-year conceptualization phase of the Learner Data Institute (LDI; www.learnerdatainstitute.org). The present work updates that of Rus et al. (2020), which provided an introduction to LDI and early activities and outcomes. We emphasize here the developments of the past 12 months (since the 2020 paper), focusing on the key elements of the science convergence framework, its development, implementation, evaluation, and refinement, and key outcomes such as the 5-year plan of the future institute and data-intensive prototype solutions to address key challenges in the learning ecosystem.
The LDI is a “frameworks” project funded by the United States’
National Science Foundation (NSF) under the Data-intensive
Research in Science and Engineering (DIRSE) program to make
the learning ecosystem more effective, efficient, engaging,
equitable, relevant, and affordable. It is part of the NSF’s
Harnessing the Data Revolution1 (HDR) Institutes effort. “HDR
Institutes… enable breakthroughs in science and engineering
through collaborative, co-designed programs to formulate
innovative data-intensive approaches to address critical national
challenges”
There is a twofold focus during the current 2-year conceptualization phase: (1) develop, implement, evaluate, and refine a framework for data-intensive science and engineering, and (2) use the framework to provide prototype solutions, based on data, data science, and science convergence, to a number of core challenges in learning science and engineering. The institute or convergence phase would build on results realized and insights gained from this conceptualization phase. By targeting a critical mass of key challenges that are at a tipping point (i.e., targeting challenges for which timely investment in data-intensive approaches has the maximum potential for a transformative effect), LDI will start a chain reaction that will transform the whole learning ecosystem, lifting it to a qualitatively higher state that is more effective, engaging, equitable, relevant, and affordable. Indeed, since the learning ecosystem is a complex web of interrelated elements, improvements in key aspects will percolate throughout the whole learning ecosystem.
LDI has brought together a team which currently consists of 60+ researchers, developers, and practitioners from three continents spanning many disciplines and backgrounds. Team members are
drawn from institution and organizations representing academia, government, and industry.
Together, we intend a rigorous test of the hypothesis that emerging learning ecologies that incorporate adaptive instructional systems (AISs) are capable of providing affordable, effective, efficient, equitable, and engaging individualized assistance for both learners and instructors, and that the characteristics, parameters, and impacts of these systems, for example, effectiveness (in terms of learning gains), can be improved over time given sufficient attention to evidence, captured as data, and expertise, provided by teams of interdisciplinary researchers like ours.
The idea that AISs and data science have the potential to radically
transform existing learning ecosystems is based on the following:
(1) evidence suggesting that individualized instruction is generally
more effective than traditional classroom instruction where
monitoring and tailored support to each individual learner is not
possible
The LDI is founded on the key observation that data science and AISs are a powerful mix with potentially transformative impact on the learning ecosystem.
Big educational data (edu-data) create tremendous opportunities to reveal facets along which learner experiences can be tailored or adapted in ways heretofore impossible. A particular learning environment may result in different learning outcomes for different (groups of) students because of students’ idiosyncratic prior knowledge, experience(s), interest(s) and motivation(s). A small minority of students, for example, that approach a problem in a unique way could be overlooked in a small dataset, but larger datasets give us the possibility to detect and account for individual differences in learning. To this end, our mission is to harness the data revolution to further our understanding of how people learn. AISs can monitor and scaffold learners at a fine level of granularity (e.g., capturing every single step during instructional activities) and with respect to many aspects of learning (e.g., cognitive, behavioral, affective, social, motivational facets of learning) at scale (i.e., for millions of learners and teachers and across many topics and domains) and across time periods (e.g., across gradelevels). Such rich data, when collected, can be characterized as deep (many data instances from millions of learners), wide (capturing many aspects of the learning process at a fine granularity level), and long (longitudinal, i.e., across time and grade levels). Such big edudata, together with advanced data science methods, are likely to offer insights about learning and instruction and lead to the development of effective and affordable instructional tools that were not possible before. This is promising enough to believe that the learning ecosystem is at a tipping-point to be transformed. Indeed, LDI is built on the belief that AISs constitute a necessary catalyst to enable the transformation of the learning ecosystem through harnessing the data revolution because, as noted earlier, AISs can monitor and scaffold learners at a very fine granularity level, at scale, and across time. It should be noted that much of education data, (e.g., currently collected by schools), relies on a set of predefined competencies or standards to monitor student progress. Such data only reveal what students know or mastered and what they don’t know (didn’t master yet), but such data often do not reveal much about the learning and instructional process. That is, much of the school data focus on “where the student is” but not what they do during instructional activities. Fundamentally, teachers and schools in general lack the capacity to monitor and store data about all students at every single step of the learning and instruction process. LDI will thus offer schools a new powerful framework to understand, monitor, and intervene at a fine-grain level with potentially transformative effects on the learning ecosystem.
A major goal of LDI conceptualization phase is to develop, implement, test, and refine a framework for data-intensive research in science and engineering enabling science convergence, aligning with the Growing Convergence Research (GCR) “big idea” identified by the National Science Foundation.
According to NSF, “convergence research is a means of solving
vexing research problems, in particular, complex problems
focusing on societal needs. It entails integrating knowledge,
methods, and expertise from different disciplines and forming
novel frameworks to catalyze scientific discovery and innovation."
Also, “convergence is a deeper, more intentional approach to the
integration of knowledge, techniques, and expertise from multiple
disciplines in order to address the most compelling scientific and
societal challenges”
NSF identifies Convergence Research as having two primary characteristics: “Research driven by a specific and compelling problem. Convergence Research is generally inspired by the need to address a specific challenge or opportunity, whether it arises from deep scientific questions or pressing societal needs.” “Deep integration across disciplines. As experts from different disciplines pursue common research challenges, their knowledge, theories, methods, data, research communities and languages become increasingly intermingled or integrated. New frameworks, paradigms or even disciplines can form sustained interactions across multiple communities” NSF(GCR, 2020).
LDI’s compelling problem is making the learning ecosystem more effective, engaging, equitable, efficient, relevant, and affordable. To foster deep integration across scientific disciplines, we have put in place a convergence framework, comprising a diverse team, organizational structures, processes, mechanisms, activities, and tools, meant to encourage broad participation, coordination, collaboration, and diffusion and integration of knowledge across disciplines.
LDI has intentionally sought, from its inception, to follow NSF’s
characterization of convergence research by “intentionally
bring[ing] together [from the inception] intellectually diverse
researchers and stakeholders to frame … research questions,
develop effective ways of communicating across disciplines and
sectors, adopt common frameworks for their solution, and, when
appropriate, develop a new scientific vocabulary.”
To make these intentions a reality, LDI’s leadership team and
participants have designed, prototyped, and tested a process and a
corresponding set of tools designed to transform what is currently
a loosely coupled group of research centers, AIS commercial
providers, and governments research labs engaged in similar but
disparate research and development efforts into a set of interacting
teams
The key elements of the LDI convergence framework are listed below.
An intellectually diverse team with stakeholder representation (researchers, developers, practitioners including school and teachers’ representatives)
Processes, mechanisms, and tools to nurture collaboration, broad participation, diffusion and integration of knowledge across disciplines, and coordination Resources, in terms of funding, student support, travel, and access to big edu-data and other cyber-infrastructure resources Incentives for team members to proactively and deeply engage in convergent activities and working towards accomplishing the goal/mission of the team which is to solve the compelling problem: o o o o o
Progress monitoring and refinement of the convergence framework Our framework will enable team members to develop a shared vision and language, which over time should lead to effective and meaningful cross-discipline, collaborations, i.e., science convergence. Such mutual sense- making, science convergence, and R&D efforts are likely to incubate solutions to complex problems to enable effective, efficient, engaging, equitable, and affordable learning experiences for everyone. We detail next the main components of our science convergence framework.
LDI’s mission is to harness the data revolution (HDR) to further our understanding of how people learn, how to improve adaptive instructional systems (AISs), and how to make emerging learning ecologies that include online and blended learning with AISs more effective, efficient, engaging, equitable, relevant, and affordable. Our vision is for LDI to: (i) serve as a hub to identify investment opportunities for data-intensive approaches to core learning science and engineering challenges to accelerate progress toward equitable learning and achievement in education; (ii) foster, support, and build a portfolio of inter-related, inter-disciplinary prototyping or “Scale-up Projects” to research, develop, and disseminate dataintensive solutions across multiple academic and non-academic communities that currently cannot easily communicate with each other, embodying a process of science convergence; (iii) bridge the HDR ecosystem with the educational data science and learning engineering community and the broader education world, and, in particular, serve as the education & training hub for the HDR ecosystem, assisting other teams with developing data science training platforms for their communities.
LDI will forge new HDR frontiers by: furthering our understanding of learning and instructional processes and environments; developing data science infrastructure for the education and the HDR ecosystem; improving AISs and scale them up both horizontally and vertically; advancing research at the human-technology frontier in future learning ecologies that involve AISs; transforming communities of practice (e.g., triggering a culture shift in teacher training programs); exploring how data science can address equity, ethics, diversity, and inclusion aspects of education.
LDI’s team evolved and grew from 45+ members
LDI is led by the Institute of Intelligent Systems at The University of Memphis and main corporate partner Carnegie Learning, developer of commercial-grade AISs serving over 500,000 students in 2,000+ school districts. The assembled team now spans 14 main organizations on 3 continents, including NSF-funded partners such as the Institute for Data, Econometrics, Algorithms, and Learning (IDEAL; NSF HDR TRIPODS project led by researchers at Northwestern University) and LearnSphere: Building a Scalable Infrastructure for Data-Driven Discovery and Innovation in Education (NSF DIBBs project; Carnegie Mellon University lead). In addition, partners include researchers, practitioners, and other stakeholders from the US Army’s Generalized Intelligent Framework for Tutoring project (Sottilare et al, 2016) and 6 additional corporate partners, 3 laboratory schools (The Early Learning & Research Center, Campus Elementary School, and University Middle School in Memphis, TN), 3 K-12 school districts - Shelby County Schools (Memphis, TN area; 200 schools, 100,000 students), Brockton Public Schools (Boston, MA area; 24 schools, 15,000 students), Val Verde Unified School District (Los Angeles, California area; 21 schools, 20,000 students), and one teacher training program at Christian Brothers University.
The team structure consists of a leadership team, domain-oriented Expert Panels, and task-oriented groups that in the conceptualization phase have driven prototyping projects for very concrete, well-defined tasks, hence called concrete tasks. The LDI Core Leadership Team is responsible for overseeing and coordinating LDI activities, making sure those activities align with the mission of the institute and offering necessary support for cohesiveness of activities. The Leadership Team consists of Lead Principal Investigator (PI) Dr. Vasile Rus, Carnegie Learning Principal Investigator Dr. Stephen Fancsali (co-PI), and co-PIs from University of Memphis: Dr. Dale Bowman, Dr. Philip Pavlik, and Dr. Deepak Venugopal. Project coordinator Jody Cockroft, Senior Research Scientist Dr. Donald Morrison, Dr. Arthur Graesser, a Professor Emeritus at The University of Memphis round out the Leadership Team.
LDI Expert Panels are homogeneous in terms of expertise in order to maximize intellectual coverage of particular research areas, as individual researchers are specialized in different subareas of a relatively broad area such as Data Science or Learning Science. Expert Panels were composed in this homogenous way to encourage meaningful discussions from the start leading to more efficient and engaging conversations early on, benefitting team building and engagement. Cross-domain interactions are more challenging. One major purpose of LDI is to engage our team members (including Expert Panels) in cross-domain interactions that develop shared sense making, a common language, and mission-driven culture over time.
The role of the Expert Panels is twofold: (1) to provide solid (breadth and depth) input from an area of expertise to all LDI efforts such as concrete prototyping tasks that are being carried out in the Phase 1 conceptualization and (2) to help shape the 5-year plan for Phase 2 by identifying opportunities for investment (i.e., promising developments in one area that could benefit the other areas or specific activities of the institute).
The following Expert Panels were initially formed: Data Science, K-12 Education, Learning Sciences, Learning Systems Engineering, Ethics & Equity, and Human-Technology Frontier. Expert Panel membership is flexible; LDI participants may belong to more than one Expert Panel but must be actively engaged in at least one. Expert Panels have co-leaders who are responsible for ensuring that the panels successfully reach milestones (e.g., reviewing concrete tasks).
Concrete tasks or “Scale-Up Projects” are prototyping endeavors led by individual researchers (see the section on Building Prototypes for Concrete Tasks later). Examples of concrete tasks include projects directed at scaling data-driven domain model refinement, using auto-encoders for student assessment, and datadriven instructional strategy discovery.
Our team includes representatives of various communities with an
invested interest in the learning ecosystem such as researchers,
developers, practitioners, government, policymakers, and funders.
Nevertheless, there are gaps in LDI’s expertise. For instance, we do
not currently have representatives from domains including
neuroscience, the law, and social and moral philosophy, primarily
due to Phase 2 budget constraints. We hope to account for such
expertise through ad-hoc engagement with appropriate experts
(e.g., reviewing and feedback from targeted experts in those areas).
While diverse opinions and perspectives are represented within the
team and make possible greater organizational learning and
synergy, interdisciplinary teams also deal with the pull of
competing loyalties and demands
A key element of the LDI convergence framework is a set of processes, mechanisms, and tools to foster collaboration, broad participation, diffusion and integration of knowledge across disciplines, and coordination.
LDI has implemented an iterative process of idea and solution
generation and refinement that includes internal (from other LDI
members) and external (paid, external ad-hoc reviewers) feedback
loops. Furthermore, we have set in place synchronous and
asynchronous, face-to-face and virtual coordination, collaboration,
and communication channels supported by adequate processes that
will facilitate exchange of ideas across disciplines. Processes that
enable broad participation and input from everyone were designed
and implemented, including the use of NGT
An abbreviated list of activities, tools, and structures LDI implemented to realize the above iterative idea and solution generation and broad and deep collaborations include: An iterative process of ideas and solution generation and refinement that includes internal (from other team members) and external (paid, external ad-hoc reviewers) feedback loops asynchronous and synchronous, face-to-face and virtual coordination, collaboration, and communication channels supported by adequate processes that will facilitate exchange of ideas across disciplines A federation of semi-autonomous groups (e.g., Expert Panels, concrete task teams) coordinated by a Leadership Team Regular virtual meetings of the Core Leadership Team (as the conceptualization phase has largely taken place during the global pandemic) Two full-team or “all-hands” virtual meetings each year Two workshops (in 2020 and 2021) at the International Conference on Educational Data Mining (to which this piece contributes) to engage with a broader international community of scholars Meetings at major conferences that our team members attend Quarterly updates and Requests-for-Comments from Expert Panels Mini-workshops in the form of full-day brainstorming sessions on a particular task Transformative app ideation at “all-hands” meetings Email, cloud-shared documents, wikis, Slack, and other collaboration tools for collaboratively drafting and refining ideas, solutions, and processes Software repository managed with the version control software, e.g., github or SVN Project management software to keep track of task progress and major milestone deadlines and deliverables
LDI participants have started to develop an emerging shared vocabulary and language, which enables more effective and efficient communication and collaboration across disciplines and which constitutes a key ingredient of convergence research. For instance, new vocabulary includes introducing many team members to the notion of convergence research, concrete tasks or “Scale-Up Projects,” “learner model,” “cloud continuum,” scalingup AISs “horizontally” and “vertically,” and AISs-teacher partnership models. The vocabulary is dynamic and evolving. For instance, we have been using the term “concrete task” to indicate prototyping tasks led by researchers in LDI Phase 1 which would result in some kind of data science prototype or deliverable (e.g., a significant dataset and/or peer-reviewed publications). In this work, we use the term “concrete task” and “Scale-Up Project” essentially interchangeably as the latter reflects our intent for each concrete task to scale up in some dimension in Phase 2.
Synchronous and asynchronous interactions and activities have enabled better communication and understanding of various domain-specific terms by team members with limited initial expertise or understanding of those terms (e.g., “model parameters” in machine learning/data science, “domain model” in learning engineering, or the meaning and importance of the socio-cultural aspects of human learning). We expect the development and emergence of a shared vocabulary and language to continue and stabilize over time.
Our strategy to accomplish the LDI mission of transforming the learning ecosystems, in a proposed 5-year institute, is to focus on a number of carefully selected research priorities, targeting key aspects of the learning ecosystem which we believe are at a “tipping point” (i.e., a point at which timely investment in data-intensive approaches focusing on those critical aspects has the maximum potential for a transformative effect).
The identified research priorities were the result of an intense science convergence process involving a number of activities (e.g., brainstorming sessions or “ideas labs” followed by iterative discussions for ranking and selection at “all-hands” virtual meetings, engagement with Expert Panels, etc.). Processes and activities engaged all LDI team members across many disciplines (e.g., educators, education researchers, computer scientists, statisticians, cognitive scientists), developers (Carnegie Learning, Age of Learning, Gooru), school districts (Shelby County Schools, Brockton Public Schools), as well as researchers from other projects funded by NSF (e.g., Northwestern’s TRIPODS Cohort II project: IDEAL - The Institute for Data, Econometrics, Algorithms, and Learning; CMU’s DIBBS LearnSphere: Building a Scalable Infrastructure for Data-Driven Discovery and Innovation in Education; and the University of Memphis NSF project: Advancing the Science of Learning Data Science with Adaptive Learning for Future Workforce Development). That is, the identified research priorities reflect our collective interdisciplinary wisdom that timely investment in data-intensive approaches will have the maximum potential for a transformative effect.The identified investment opportunities (or research priorities) constitute the central focus of the 5-year plan for the LDI. It should be noted that we also generated a 10-year plan such that the impacts of the LDI Institute will propagate and evolve beyond the lifetime of the award and beyond our own team thus acting as an agent of change for how research questions are conceived and addressed through interdisciplinary collaboration.
Identified key investment opportunity areas or thrusts include:
Learning Data – From Impoverished Datasets To Learning Data Convergence To Comprehensive Learner Models
Powerful, Scalable, and Accurate Data-intensive Solutions to Core Education Tasks
Frontier – Pushing For Wider Adoption and Integration Of AISs
Data. To enable data science, there must be data and in particular “big” education data (big edu-data). To this end, a key long term goal of LDI is learning data convergence, i.e., collecting and aligning (more) comprehensive data about the same learner(s) across skills, disciplines, and modalities (cognitive, meta-cognitive, emotional, motivational, behavioral, social) and across time (e.g., K-12 grade-levels), as well as data about the learning process and environment.
Prior efforts such as LearnSphere/DataShop have made progress
towards building data infrastructure and capacity in education
contexts, but slow data convergence is a critical issue that hinders
realizing the full potential of data and data science to transform the
learning ecosystem. For instance, the DataShop metric reports
show that most of the data is composed of datasets in the standard
DataShop format, of which there are about 3500
(https://pslcdatashop.web.cmu.edu/MetricsReport). While
accumulating this many datasets is no small feat, the average
number of observations per student is less than 400. A large number
of students, greater than 800,000, is spread across more than 3000
datasets, resulting in less than 260 students per dataset. Similary,
the recently released EduNet
Ideally, big edu-data would include data about millions of learners that are fine-grain (e.g., step/substep level information or detailed process data), rich (capturing cognitive, affective, motivational, behavioral, social, and epistemic facets of learning), and longitudinal (across many grades). That is, big edu-data should be deep (e.g., about many learners), wide (e.g., capture as many learning relevant aspects as possible), and long (being longitudinal, across many grades or even a learner’s lifetime). Convergence efforts will seek to “deepen” samples and “lengthen” timeframes of datasets that are (sometimes, but not always, already) “wide” in terms of features captured.
Using these concepts, our goal can be re-stated as enabling the collection of deep, wide, and long education data which could then be analyzed using emerging, state-of-the-art data science methods capable of learning patterns from such massive collections of data and also accounting for input from diverse domain experts with the ultimate goal of transforming the learning ecosystem. In order to fully harness the data revolution to transform the learning ecosystem we need: (1) improved, at-scale data collection and (near) real-time access to big edu-data (i.e., addressing the “impoverished datasets” challenge) in ways that account for security, privacy, and ownership and (2) infrastructure to process learner data at scale using distributed computing (e.g., leveraging the cloud-continuum), scalable algorithms, and richer/more powerful algorithms (e.g., emerging neuro-symbolic approaches). Indeed, access to data at scale is a more critical, upstream challenge that needs to be addressed first as before being able to process learning data, one must have access to the data and have permission to share it. LDI adopts the principle that data owners (e.g., learner/ parent/ guardian/ teacher/ school/ developer/ etc.) should be given a spectrum of options with respect to data sharing or, if deciding not to share, with respect to providing access to data. The spectrum of options should accommodate all attitudes that learners/learning data owners may have towards data ownership, security, and privacy. Indeed, access to learner data is a complex issue due to privacy, security, ownership, and regulatory concerns. We are aware that full data convergence would be hard to achieve for various reasons. However, our goal is to push the limits of what is possible, understand those limits, and act accordingly. Understanding the limits of data convergence will allow us to understand the limits of technology, what teachers can do to compensate for those limitations, and how to best orchestrate the learner-teacher-AISs partnership.
Our data convergence activity focuses on concrete examples from
math and computer science (STEAM+C) as well as literacy and
leverage prior efforts in the area of building data infrastructure and
capacity, contributing and expanding on those previous efforts to
move us closer to the goal of full data convergence. Specifically,
one major goals is to build a fine-grain, large, and diverse (deep,
wide, long) dataset that will enable LDI to explore the potential of
data science methods to better model learners and the learner
process. We announced and started the process of building
LearnerNet in Fall 2019 as part of LDI Phase 1 (see Rus, 2019 –
ADL Directors’ meeting talk). Indeed, we have called for the
development of LearnerNet
Investment Opportunity Area 2: Novel, Richer, More Powerful, Scalable, and Accurate Data-intensive Solutions to Core Education Tasks.
This investment opportunity area focuses on improving existing methods and models with respect to their scaling and extension using big edu-data and developing novel, richer, more powerful, scalable, and accurate computational models for a number of core educational tasks such as prediction and assessment of learner mastery of knowledge components (KCs; micro-competencies or skills), domain model refinement (i.e., improving models of what learners need to learn to acquire mastery of a domain), and inferring optimal strategies to coordinate the behavior of AISs for how and when to optimally implement guidance to promote student learning. The goal is to improve our understanding of how learners learn, improve the effectiveness and efficiency of AISs, make AISs more affordable and scalable horizontally (across topics and domains), and scale AISs vertically (offering training on higherlevel skills such as deep conceptual understanding and collaborative problem solving).
One major opportunity from a learning engineering perspective is the automation of the development and refinement of AISs and adaptive instructional content. Making progress towards automating the authoring of AISs should begin to enable better scalability across topics and domains (horizontal scalability), which currently is a major stumbling block for a wider adoption of such systems. Expert-driven approaches to developing domain models, learner models, and instructional strategies for new topics and domains are expensive, tedious, and time-consuming. Automated or semi-automated approaches to discovering domains models, inferring learner models, and discovering instructional strategies are much needed. For instance, we intend to use neuro-symbolic approaches to automatically extract from both structured, e.g., student performance data, and semi-structured data, i.e., text in textbooks, domain models.
A second major opportunity within this thrust involves AISs for
collaborative learning with intelligent discourse components.
Widely deployed, commercial AISs largely do not target advanced
topics such as collaborative problem solving. Collaborative work
and collaborative problem-solving skills are much needed in the
21st century
Language and discourse play a central role in learning
Language and discourse is not only essential for learning within
individuals but also learning in group contexts. Problems have
dramatically increased in complexity, requiring collaborative
problem solving by people with disparate expertise and
perspectives
Pushing For Wider Adoption and Integration Of AISs
This investment opportunity fosters a portfolio of efforts to push
for wider adoption and integration of AISs with school-based and
teacher-led learning activities at the Human-Technology Frontier,
one other of NSF’s ten Big Ideas for Future Investment.
Many teachers are overwhelmed by the many duties and tasks they
have to handle, resulting in burnout and reduced teacher job
satisfaction and retention rates
Models of Learner-Teacher-AISs Partnership. Finding the best
learner-teacher-AISs partnerships could have transformative
impact on the learning ecosystem such as freeing teachers from
certain duties that AISs can do in an autonomous manner thus
allowing them to focus on higher level tasks such as designing new
instructional materials or novel tailored interventions for students,
, motivational support, and other tasks for which AISs are not ideal
This better distribution of duties and coordination between teachers
and AISs should lead to a more effective, efficient, engaging, and
equitable learning ecosystem. We will study four levels of AISs
autonomy with respect to how teachers may use AISs (see later).
Detect and Mitigate Issues Related to Ethics, Equity, Inclusion, and
Diversity in Education. As a general principle, all LDI activities
will be informed and guided by our goal of using data science and
AISs to promote ethics and equity in education
Evaluation and analysis are key elements of the LDI convergence framework to both demonstrate its effectiveness and provide a way to identify opportunities for improvement and refinement. We focus on quantitative and qualitative metrics for LDI community building and engagement efforts, identifying investment opportunities priorities, and development and refinement of prototyping concrete task or Scale-Up Project activities. For quantitative metrics, to account for different perspectives, we will report how many experts and from how many different disciplines contribute to specific tasks (e.g., identification of data requirements for Investment Opportunity Area 1, above). For each expert, we can monitor their individual contributions in terms of content (e.g., word counts), comments, and revisions to others’ contributions (by using shared documents that track such metrics). More qualitatively, each member’s contributions will be assessed in terms of the depth of their contributions. A researcher might identify that a particular expert’s contribution initiated the development of a novel solution that could improve the detections of learners’ emotions in a classroom context.
Furthermore, we report the scientific and societal impact of the proposed convergence framework. Scientific impact can be reported in terms of the number of publications, presentations, tutorials, meetings, email exchanges and other forms of direct communication (among LDI members and the broader research community) as well as improvements of prototype solutions over existing solutions. Other scientific success measures can monitor longer term impact such as how many citations the products of this project generate and how many research groups integrate the proposed solutions (e.g., user adoption of analysis toolkits developed).
Societal impact can be assessed through impact on learners and teachers as well as impact on the learning ecosystem (e.g., in terms of how LDI efforts have made aspects of the learning ecosystem more effective, engaging, equitable, efficient, relevant, and affordable, as well as other outcomes such as transforming educators’ community of practice).
An important requirement for the evaluation process is documentation of the various elements of the convergence framework. For this purpose, for instance, all meetings of the leadership team were recorded (key metric: hours of meetings and interactions; volume of those interactions). Other processes and activities have been documented in various ways such as Google docs, meeting recording, and Slack asynchronous discussions. For instance, the convergence process implemented to generate the 5year institute plan has been well documented through other records such as spreadsheets used in NGT processes employed by the various Expert Panels to generate and rank ideas for investment opportunities to be included in the 5-year plan.
We will illustrate how we have been evaluating the effectiveness of convergence framework holistically as well as from the perspective of Expert Panels. For brevity, we illustrate the evaluation of the convergence process from the perspective of the Learning Engineering Expert Panel.
The LDI’s Learning Engineering Expert Panel comprised a diverse group of researchers and developers with vast experience in research and development of learning systems. The 10-member expert panel was drawn from the academe, government, and industry.
The Learning Engineering Expert Panel, like other LDI expert panels, engaged in two major activities that contribute to the LDI Phase 1 project:
Provide input to each of the concrete tasks (forward-looking “Scale-Up Projects”) addressing various challenges in the learning ecosystem with the goal of converging to solutions to those challenges that account for input from many domains. Identify, rank, and propose investment opportunities for the 5year plan of the convergence or institute phase (LDI Phase 2) The concrete task reviewing and feedback process involved significant expert time (see Table 2, which presents a summary of the quantitative evaluation of the initial cycle of the review and feedback process by the Learning Engineering Expert Panel). In addition to this quantitative summary of the convergence process related to concrete tasks, we also developed a 5-stage model to characterize the maturity of concrete tasks: (1) ideation or initial idea, (2) conceptualization and convergence of a data science solution with input from experts from many domains, (3) implementation & refinement, (4) product release (e.g., an emerging data science prototype or dataset release), (5) impact, in which the product from stage 4 is adopted by or integrated into external research projects or a learning environment, having some external impact on the research landscape or on the learning ecosystem. Work of LDI participants during the conceptualization phase has centered primarily on concrete tasks in the first four phases (ideation, conceptualization and convergence, and product release). Ideally, the transition from concrete task to “Scale-Up Projects” in LDI Phase 2 will reflect progression to later stages of this model.
The other major task of each Expert Panel was to identify investment opportunities for the 5-year plan of the LDI institute phase (Phase 2). Expert panels had the freedom to adopt different internal processes to identify investment opportunities.
Summary Word Count 9 (1 of 10 Expert Panel members left LDI after assignment to Expert Panel.) 7 / 9 (Two members were assigned reviews but did not submit any reviews.) 17 34 (17 task x 2 reviews/task) 3.3 (average over the 7 reviewers submitting at least one review; min: 2; max: 7) (34 x 2) + (7 x 2) = 82 hours of expert time (assuming 2 hours spent per concrete task review and 2 hours of Expert Panel meeting to summarize the reviews for each concrete task) 4.82 hours (82 total hours / 17 concrete tasks) 279 words per task (average); 4,749 total
This policy was adopted for two main reasons: (i) offer autonomy to each expert panel to self-organize and (ii) explore different collaboration processes in order to discover the best one (e.g., in terms of member engagement, effectiveness, and efficiency) or identify from each expert panel a set of best practices for later adoption. In the case of the Learning Engineering Expert Panel, investment opportunity ideas were solicited via e-mail from the Expert Panel by the Co-Leads. A brief summary of candidate opportunities is provided below:
Improving and scaling up AISs horizontally across topics and domains Scaling up AISs vertically targeting advanced skills such as collaborative problem solving and deep conceptual understanding of complex STEAM+C topics (More) Comprehensive learner models Pushing for wider adoption and integration of AISs in schoolbased and teacher-led instruction (Human-Tech Frontier)
Inclusive learning engineering R&D (ethics, equity, inclusion, and diversity) This list was further discussed and the initial investment opportunities were ranked by all expert panel members. A recommendation of the most important investment opportunities was put forward to the whole LDI team for further debate and refinement by other Expert Panels and paid, ad-hoc external reviewers and the public at large. Many of the proposed investment opportunities that originated in the Learning Engineering Expert Panel are part of the 5-year institute plan adopted by the broader LDI community.
Holistically, the LDI convergence framework can be evaluated in terms of the level of engagement of a diverse team of researchers, developers, practitioners, and other stakeholders as well as its key outcome, which is the 5-year plan for the institute or convergence phase which was described and submitted as a proposal to NSF. The level of engagement can be summarized briefly by noting that our 60+ strong team participated so far in 3 all-hands meeting each for about 20 hours (2.5 days) resulting in 60 x 20 = 1,200 expert hours of effort. Experts spent hundreds of additional hours spent in other meetings and other activities. Most meetings were recorded and transcribed. A more detailed, quantitative and qualitative analysis is being conducted right now, and the results will be widely disseminated.
We conclude this progress report by briefly presenting two emerging ideas from the collective work of the LDI during its conceptualization phase to date.
Our work so far also results in a number of policy recommendations:
Publicly funded education technologies similar to publicly funded education adopted in the 19th and 20th century. Learning data owners keep ownership of their data and have decision power with respect to where their data is stored, how the data is accessed, by whom and for what purposes, how their data is used, and if their data can be shared, with whom, and under what conditions and circumstances.
Learning data infrastructure is needed to enable responsible learning data collection, storage, access, sharing, and processing.
The need for a culture shift in teacher training programs and data literacy curriculum for future teachers.
Finding the best teacher/learner-AISs partnerships could have transformative impact on the learning ecosystem, potentially freeing teachers from certain duties that AISs can do in an autonomous manner and allowing teachers to focus on higher level tasks such as tailored, individualized interventions for students, motivational support, and other tasks for which AISs are not ideal. This better distribution of duties and coordination between teachers and AISs should lead to a more effective, efficient, engaging, and equitable learning ecosystem.
We defined and intend to study four levels of AISs’ “autonomy” with respect to how teachers can use such AISs: (1) fully autonomous – teachers need little (if any) training and have little (if any) involvement in “tuning” AISs, (2) minimal teacher involvement – teachers tune the parameters of the AISs with the help of the AISs developer at the beginning of the school year or semester (minimal teacher training with respect to the workings of the AISs), (3) average teacher involvement – teachers require training, and they work with the system on a weekly basis selecting instructional tasks and receiving information from the AISs, (4) teacher-driven – the teachers exerts full control of the AISs including overriding decisions the AISs may take or suggest, the teacher will interact almost daily with the AISs. There is in fact one other level (level 0) which are self-improving, fully autonomous AISs – they improve with experience with minimal or no developer intervention. While we will explore as resources permit the role of data science to enable such level 0, self-improving fully autonomous AISs, from a teacher and learner perspective they are similar to the fully autonomous level of AISs (level 1). We plan to study and understand the trade-offs in terms of teacher involvement in tuning AISs vs. levels of AIS autonomy. For instance, teachers may choose a fully autonomous mode of operation for an AIS meant for students working independently with the system afterschool as supplemental instruction, whereas for student interactions with the AIS during a class period (i.e., in a blended-learning environment), the same teacher may choose to control more the behavior of the AISs. Similarly, teachers may decide to use/download a pre-trained learner model and update it with data from her students, assuring data security and privacy and maintaining full ownership of the data. They may decide to share a sample of her own student data to benefit the pooled/pre-trained models that everyone can download as default.
LDI intends to serve as an agent of change for how research questions are conceived and addressed through interdisciplinary collaboration such that LDI’s impacts will propagate and evolve beyond the lifetime of the award.
More specifically, we have the explicit intent to start a culture shift in teacher training programs through two specific actions: (1) involve a few dozen teachers and pre-service teachers in our work in order to co-design solutions and account for their input and expose them to the potential of data science and AISs while also introducing them to science convergence approaches to address key challenges in education and (2) develop new curriculum recommendations for teacher training programs as well as accompanying training materials to build capacity for teachers and other stakeholders to adopt AISs and data science approaches, tools, and principles to improve learning and teaching. Wider adoption of advanced data-driven science and engineering approaches and tools such as AISs is still lacking for at least three reasons: (1) Data science and education technology training is often limited in teacher training programs. (2) The sophistication and complexity of AISs often entail a significant effort to train teachers to effectively use such advanced education technologies. (3) New approaches are often developed with a lack of substantive involvement of educators and schools.
Involving educators will help to ensure that new approaches based on data science to tackle various education challenges, nextgeneration AISs, and learning environments that include AISs, are designed to help eliminate biases and promote equity, inclusion, and diversity, offering high quality education opportunities for all learners. We will therefore push for schools, teacher training programs, and instructors to collaborate more with data science and educational technology researchers and developers to improve learning and instruction. To this end, in addition to substantive involvement of teachers and other stakeholders in LDI activities, we will explore avenues for delivering professional learning, including workshops for teachers, summer schools (e.g., by adding a track to CMU’s LearnSphere summer school) for pre-service teachers and Research Methods instructors in schools of education. We are an expanding community of practice and promote Scale-Up Projects that will ideally become bona fide research programs beyond the award period, securing their own funding as they make scientific progress. Furthermore, Scale-Up projects and research thrusts will ideally result in career-long efforts for some younger faculty members.
To sum up, our strong team of interdisciplinary experts, developers, and practitioners will work together during the 5-year LDI institute project to move current practices beyond the small-scale studies to bring the learning sciences into the era of big data and interdisciplinary science convergence. The impact of LDI will be felt far and wide, propagating and evolving beyond the lifetime of the award and beyond our own team, acting as an agent of change for how research questions are conceived and addressed through interdisciplinary, collaboration, and co-designed research and development. The proposed processes, methods, and studies pave the way for taking these outcomes to other domains.
The Learner Data Institute is sponsored by the National Science Foundation (NSF; award #1934745). The opinions, findings, and results are solely the authors’ and do not reflect those of NSF.