This paper provides a progress report on the first 18 months of the two-year conceptualization phase of the Learner Data Institute (LDI; www.learnerdatainstitute.org). The present work updates that of Rus et al. (2020), which provided an introduction to LDI and early activities and outcomes. We emphasize here the developments of the past 12 months (since the 2020 paper), focusing on the key elements of the science convergence framework, its development, implementation, evaluation, and refinement, and key outcomes such as the 5-year plan of the future institute and data-intensive prototype solutions to address key challenges in the learning ecosystem.
There is a twofold focus during the current 2-year conceptualization phase: (1) develop, implement, evaluate, and refine a framework for data-intensive science and engineering, and (2) use the framework to provide prototype solutions, based on data, data science, and science convergence, to a number of core challenges in learning science and engineering. The institute or convergence phase would build on results realized and insights gained from this conceptualization phase. By targeting a critical mass of key challenges that are at a tipping point (i.e., targeting challenges for which timely investment in data-intensive approaches has the maximum potential for a transformative effect), LDI will start a chain reaction that will transform the whole learning ecosystem, lifting it to a qualitatively higher state that is more effective, engaging, equitable, relevant, and affordable. Indeed, since the learning ecosystem is a complex web of interrelated elements, improvements in key aspects will percolate throughout the whole learning ecosystem.
LDI has brought together a team which currently consists of 60+ researchers, developers, and practitioners from three continents spanning many disciplines and backgrounds. Team members are drawn from institution and organizations representing academia, government, and industry.
Together, we intend a rigorous test of the hypothesis that emerging learning ecologies that incorporate adaptive instructional systems (AISs) are capable of providing affordable, effective, efficient, equitable, and engaging individualized assistance for both learners and instructors, and that the characteristics, parameters, and impacts of these systems, for example, effectiveness (in terms of learning gains), can be improved over time given sufficient attention to evidence, captured as data, and expertise, provided by teams of interdisciplinary researchers like ours.
The idea that AISs and data science have the potential to radically transform existing learning ecosystems is based on the following: (1) evidence suggesting that individualized instruction is generally more effective than traditional classroom instruction where monitoring and tailored support to each individual learner is not possible (Bloom, 1984;Chi, Roy, & Hausmann, 2008;Cohen, Kulik, & Kulik, 1982;VanLehn et al., 2007); (2) the capability of modern technologies to collect, store, and access vast and rich learner data; (3) incentive-based mechanisms to share goods such as education data using online market places (Hartline, 2012;Hartline et al., 2019) and secure and privacy preserving ways to access and process data based on differential privacy and multiparty computation (Dwork, 2008; Wang, Ranellucci, & Katz, 2017); ( 4) promising new advances in data science, including powerful machine learning and statistical methods such as deep neural networks, statistical relational learning, causal modelling, and probabilistic temporal graphs, for extracting useful knowledge from massive educational data sets (Spirtes, Glymour, & Scheines, 2001; LeCun, Bengio, & Hinton, 2015; Schmidhuber, 2015; Bach, Broecheler, Huang, & Getoor, 2017; Pearl & Mackenzie, 2018); and (5) recently available access to affordable, powerful, and scalable cloud-based computing resources for processing big data (Hellerstein et al., 2019;Atwal, 2020).
The LDI is founded on the key observation that data science and AISs are a powerful mix with potentially transformative impact on the learning ecosystem.
Big educational data (edu-data) create tremendous opportunities to reveal facets along which learner experiences can be tailored or adapted in ways heretofore impossible. A particular learning environment may result in different learning outcomes for different (groups of) students because of students' idiosyncratic prior knowledge, experience(s), interest(s) and motivation(s). A small minority of students, for example, that approach a problem in a unique way could be overlooked in a small dataset, but larger datasets give us the possibility to detect and account for individual differences in learning. To this end, our mission is to harness the data revolution to further our understanding of how people learn.
AISs can monitor and scaffold learners at a fine level of granularity (e.g., capturing every single step during instructional activities) and with respect to many aspects of learning (e.g., cognitive, behavioral, affective, social, motivational facets of learning) at scale (i.e., for millions of learners and teachers and across many topics and domains) and across time periods (e.g., across gradelevels). Such rich data, when collected, can be characterized as deep (many data instances from millions of learners), wide (capturing many aspects of the learning process at a fine granularity level), and long (longitudinal, i.e., across time and grade levels). Such big edu-data, together with advanced data science methods, are likely to offer insights about learning and instruction and lead to the development of effective and affordable instructional tools that were not possible before. This is promising enough to believe that the learning ecosystem is at a tipping-point to be transformed. Indeed, LDI is built on the belief that AISs constitute a necessary catalyst to enable the transformation of the learning ecosystem through harnessing the data revolution because, as noted earlier, AISs can monitor and scaffold learners at a very fine granularity level, at scale, and across time. It should be noted that much of education data, (e.g., currently collected by schools), relies on a set of predefined competencies or standards to monitor student progress. Such data only reveal what students know or mastered and what they don't know (didn't master yet), but such data often do not reveal much about the learning and instructional process. That is, much of the school data focus on "where the student is" but not what they do during instructional activities. Fundamentally, teachers and schools in general lack the capacity to monitor and store data about all students at every single step of the learning and instruction process. LDI will thus offer schools a new powerful framework to understand, monitor, and intervene at a fine-grain level with potentially transformative effects on the learning ecosystem.
A major goal of LDI conceptualization phase is to develop, implement, test, and refine a framework for data-intensive research in science and engineering enabling science convergence, aligning with the Growing Convergence Research (GCR) "big idea" identified by the National Science Foundation.
According to NSF, "convergence research is a means of solving vexing research problems, in particular, complex problems focusing on societal needs. It entails integrating knowledge, methods, and expertise from different disciplines and forming novel frameworks to catalyze scientific discovery and innovation." Also, "convergence is a deeper, more intentional approach to the integration of knowledge, techniques, and expertise from multiple disciplines in order to address the most compelling scientific and societal challenges" (NSF-GCR, 2020).
The team structure consists of a leadership team, domain-oriented Expert Panels, and task-oriented groups that in the conceptualization phase have driven prototyping projects for very concrete, well-defined tasks, hence called concrete tasks.
The LDI Core Leadership Team is responsible for overseeing and coordinating LDI activities, making sure those activities align with the mission of the institute and offering necessary support for cohesiveness of activities. Expert Panels were composed in this homogenous way to encourage meaningful discussions from the start leading to more efficient and engaging conversations early on, benefitting team building and engagement. Cross-domain interactions are more challenging. One major purpose of LDI is to engage our team members (including Expert Panels) in cross-domain interactions that develop shared sense making, a common language, and mission-driven culture over time.
The role of the Expert Panels is twofold: (1) to provide solid (breadth and depth) input from an area of expertise to all LDI efforts such as concrete prototyping tasks that are being carried out in the Phase 1 conceptualization and (2) to help shape the 5-year plan for Phase 2 by identifying opportunities for investment (i.e., promising developments in one area that could benefit the other areas or specific activities of the institute). Concrete tasks or "Scale-Up Projects" are prototyping endeavors led by individual researchers (see the section on Building Prototypes for Concrete Tasks later). Examples of concrete tasks include projects directed at scaling data-driven domain model refinement, using auto-encoders for student assessment, and datadriven instructional strategy discovery.
Our team includes representatives of various communities with an invested interest in the learning ecosystem such as researchers, developers, practitioners, government, policymakers, and funders. Nevertheless, there are gaps in LDI's expertise. For instance, we do not currently have representatives from domains including neuroscience, the law, and social and moral philosophy, primarily due to Phase 2 budget constraints. We hope to account for such expertise through ad-hoc engagement with appropriate experts (e.g., reviewing and feedback from targeted experts in those areas).
While diverse opinions and perspectives are represented within the team and make possible greater organizational learning and synergy, interdisciplinary teams also deal with the pull of competing loyalties and demands (Berry, 2011). Sense-making of the beliefs or actions of others (here, disparate experts) is a constant struggle in team environments (Guribye, Andressen, & Wasson, 2003), and this difficulty can be exacerbated by the greater intellectual diversity of the team. Shared goals and shared understandings are required, and negotiation of these common goals is an intrinsic part of the team-building process. Effective social relationships are a required constant for effective collaborative work, virtual or face to face, and it may occur more slowly at first (Vroman & Kovachich, 2002;Walther, 1995).
A key element of the LDI convergence framework is a set of processes, mechanisms, and tools to foster collaboration, broad participation, diffusion and integration of knowledge across disciplines, and coordination.
LDI has implemented an iterative process of idea and solution generation and refinement that includes internal (from other LDI members) and external (paid, external ad-hoc reviewers) feedback loops. Furthermore, we have set in place synchronous and asynchronous, face-to-face and virtual coordination, collaboration, and communication channels supported by adequate processes that will facilitate exchange of ideas across disciplines. Processes that enable broad participation and input from everyone were designed and implemented, including the use of NGT (Nominal Group Technique; Delbecq & Van de Ven, 1971) process for meetings to ensure everyone's voice is heard and accounted for. Other processes such as SWOT analysis (to identify strengths, weaknesses, opportunities, threats) and "pre-mortem" analysis (Klein, 2007) (i.e., identifying possible points of failure prospectively rather than retrospectively, by imagining a future situation in which a project has failed and considering how that imaginary failure might have occurred) were used as well.
Processes implemented were intended to grow science convergence among our large team of interdisciplinary experts. Within-and cross-domain interaction and collaboration processes were designed among subgroups of our team as well as all-team interactions and communications (e.g., whole-team meetings, mailing lists, website) in order to develop a common vision and language and to ensure cohesiveness and clarity with respect to the mission of the LDI, responsibility for various tasks, and engaging the community for assistance when needed.
An abbreviated list of activities, tools, and structures LDI implemented to realize the above iterative idea and solution generation and broad and deep collaborations include: An iterative process of ideas and solution generation and refinement that includes internal (from other team members) and external (paid, external ad-hoc reviewers) feedback loops asynchronous and synchronous, face-to-face and virtual coordination, collaboration, and communication channels supported by adequate processes that will facilitate exchange of ideas across disciplines A federation of semi-autonomous groups (e.g., Expert Panels, concrete task teams) coordinated by a Leadership Team Regular virtual meetings of the Core Leadership Team (as the conceptualization phase has largely taken place during the global pandemic) Two full-team or "all-hands" virtual meetings each year Two workshops (in 2020 and 2021) at the International Conference on Educational Data Mining (to which this piece contributes) to engage with a broader international community of scholars Meetings at major conferences that our team members attend Quarterly updates and Requests-for-Comments from Expert Panels Mini-workshops in the form of full-day brainstorming sessions on a particular task
Transformative app ideation at "all-hands" meetings Email, cloud-shared documents, wikis, Slack, and other collaboration tools for collaboratively drafting and refining ideas, solutions, and processes Software repository managed with the version control software, e.g., github or SVN Project management software to keep track of task progress and major milestone deadlines and deliverables
LDI participants have started to develop an emerging shared vocabulary and language, which enables more effective and efficient communication and collaboration across disciplines and which constitutes a key ingredient of convergence research. For instance, new vocabulary includes introducing many team members to the notion of convergence research, concrete tasks or "Scale-Up Projects," "learner model," "cloud continuum," scalingup AISs "horizontally" and "vertically," and AISs-teacher partnership models. The vocabulary is dynamic and evolving. For instance, we have been using the term "concrete task" to indicate prototyping tasks led by researchers in LDI Phase 1 which would result in some kind of data science prototype or deliverable (e.g., a significant dataset and/or peer-reviewed publications). In this work, we use the term "concrete task" and "Scale-Up Project" essentially interchangeably as the latter reflects our intent for each concrete task to scale up in some dimension in Phase 2.
Synchronous and asynchronous interactions and activities have enabled better communication and understanding of various domain-specific terms by team members with limited initial expertise or understanding of those terms (e.g., "model parameters" in machine learning/data science, "domain model" in learning engineering, or the meaning and importance of the socio-cultural aspects of human learning). We expect the development and emergence of a shared vocabulary and language to continue and stabilize over time.
Our strategy to accomplish the LDI mission of transforming the learning ecosystems, in a proposed 5-year institute, is to focus on a number of carefully selected research priorities, targeting key aspects of the learning ecosystem which we believe are at a "tipping point" (i.e., a point at which timely investment in data-intensive approaches focusing on those critical aspects has the maximum potential for a transformative effect).
The identified research priorities were the result of an intense science convergence process involving a number of activities (e.g., brainstorming sessions or "ideas labs" followed by iterative discussions for ranking and selection at "all-hands" virtual meetings, engagement with Expert Panels, etc. Prior efforts such as LearnSphere/DataShop have made progress towards building data infrastructure and capacity in education contexts, but slow data convergence is a critical issue that hinders realizing the full potential of data and data science to transform the learning ecosystem. For instance, the DataShop metric reports show that most of the data is composed of datasets in the standard DataShop format, of which there are about 3500 (https://pslcdatashop.web.cmu.edu/MetricsReport). While accumulating this many datasets is no small feat, the average number of observations per student is less than 400. A large number of students, greater than 800,000, is spread across more than 3000 datasets, resulting in less than 260 students per dataset. Similary, the recently released EduNet (Choi et al., 2020) contains data from 784,309 students preparing for the Test of English for International Communication at an average of 400.2 interactions per student. Despite progress in building edu-data repositories, there is an "impoverished datasets" challenge in education.
Ideally, big edu-data would include data about millions of learners that are fine-grain (e.g., step/substep level information or detailed process data), (capturing affective, motivational, behavioral, social, and epistemic facets of learning), and longitudinal (across many grades). That is, big edu-data should be deep (e.g., about many learners), wide (e.g., capture as many learning relevant aspects as possible), and long (being longitudinal, across many grades or even a learner's lifetime). Convergence efforts will seek to "deepen" samples and "lengthen" timeframes of datasets that are (sometimes, but not always, already) "wide" in terms of features captured.
Using these concepts, our goal can be re-stated as enabling the collection of deep, wide, and long education data which could then be analyzed using emerging, state-of-the-art data science methods capable of learning patterns from such massive collections of data and also accounting for input from diverse domain experts with the ultimate goal of transforming the learning ecosystem.
In order to fully harness the data revolution to transform the learning ecosystem we need: (1) improved, at-scale data collection and (near) real-time access to big edu-data (i.e., addressing the "impoverished datasets" challenge) in ways that account for security, privacy, and ownership and (2) infrastructure to process learner data at scale using distributed computing (e.g., leveraging the cloud-continuum), scalable algorithms, and richer/more powerful algorithms (e.g., emerging neuro-symbolic approaches). Indeed, access to data at scale is a more critical, upstream challenge that needs to be addressed first as before being able to process learning data, one must have access to the data and have permission to share it. LDI adopts the principle that data owners (e.g., learner/ parent/ guardian/ teacher/ school/ developer/ etc.) should be given a spectrum of options with respect to data sharing or, if deciding not to share, with respect to providing access to data. The spectrum of options should accommodate all attitudes that learners/learning data owners may have towards data ownership, security, and privacy. Indeed, access to learner data is a complex issue due to privacy, security, ownership, and regulatory concerns.
We are aware that full data convergence would be hard to achieve for various reasons. However, our goal is to push the limits of what is possible, understand those limits, and act accordingly. Understanding the limits of data convergence will allow us to understand the limits of technology, what teachers can do to compensate for those limitations, and how to best orchestrate the learner-teacher-AISs partnership.
Our data convergence activity focuses on concrete examples from math and computer science (STEAM+C) as well as literacy and leverage prior efforts in the area of building data infrastructure and capacity, contributing and expanding on those previous efforts to move us closer to the goal of full data convergence. Specifically, one major goals is to build a fine-grain, large, and diverse (deep, wide, long) dataset that will enable LDI to explore the potential of data science methods to better model learners and the learner process. We announced and started the process of building LearnerNet in Fall 2019 as part of LDI Phase 1 (see Rus, 2019 -ADL Directors' meeting talk). Indeed, we have called for the development of LearnerNet (Rus et al., 2020), an "ImageNet" (Su, Deng, & Fei-Fei, 2012) for learner modeling which could enable a transformation of our modelling and understanding of how learners learn, of how AISs can be made more capable of adapting to diverse learners, and fueling a better understanding of the learning ecosystem as a whole.
Investment Opportunity Area 2: Novel, Richer, More Powerful, Scalable, and Accurate Data-intensive Solutions to Core Education Tasks.
This investment opportunity area focuses on improving existing methods and models with respect to their scaling and extension using big edu-data and developing novel, richer, more powerful, scalable, and accurate computational models for a number of core educational tasks such as prediction and assessment of learner mastery of knowledge components (KCs; micro-competencies or skills), domain model refinement (i.e., improving models of what learners need to learn to acquire mastery of a domain), and inferring optimal strategies to coordinate the behavior of AISs for how and when to optimally implement guidance to promote student learning. The goal is to improve our understanding of how learners learn, improve the effectiveness and efficiency of AISs, make AISs more affordable and scalable horizontally (across topics and domains), and scale AISs vertically (offering training on higherlevel skills such as deep conceptual understanding and collaborative problem solving).
One major opportunity from a learning engineering perspective is the automation of the development and refinement of AISs and adaptive instructional content. Making progress towards automating the authoring of AISs should begin to enable better scalability across topics and domains (horizontal scalability), which currently is a major stumbling block for a wider adoption of such systems. Expert-driven approaches to developing domain models, learner models, and instructional strategies for new topics and domains are expensive, tedious, and time-consuming. Automated or semi-automated approaches to discovering domains models, inferring learner models, and discovering instructional strategies are much needed. For instance, we intend to use neuro-symbolic approaches to automatically extract from both structured, e.g., student performance data, and semi-structured data, i.e., text in textbooks, domain models.
A second major opportunity within this thrust involves AISs for collaborative learning with intelligent discourse components. Widely deployed, commercial AISs largely do not target advanced topics such as collaborative problem solving. Collaborative work and collaborative problem-solving skills are much needed in the 21st century (Autor, Levy, & Murnane, 2003;Carnevale & Smith, 2013), and learning activities fostering the acquisition of such skills must be adopted by learning ecologies of the future in order to make such ecologies more effective and equitable for all learners and more relevant to emerging needs and new realities. Our goal is to scale up AISs vertically, to offer training opportunities for such advanced skills. The strategy is to extend AISs such as those offered by Carnegie Learning and Age of Learning with language through discourse components.
Language and discourse play a central role in learning (Vygotsky, 1978), particularly for the acquisition of difficult topics that require deep comprehension, reasoning, problem solving, and collaboration that are required for higher paying jobs in the 21st century ( (Graesser, 2020), and formulating explanations of complex systems in science (Chi et al., 1989;Graesser, 2015), math (Fancsali et al., 2016), and computer code (Lasang et al., 2021). Language and discourse is not only essential for learning within individuals but also learning in group contexts. Problems have dramatically increased in complexity, requiring collaborative problem solving by people with disparate expertise and perspectives (Carnevale & Smith, 2013;Graesser et al., 2018;OECD, 2017).
This investment opportunity fosters a portfolio of efforts to push for wider adoption and integration of AISs with school-based and teacher-led learning activities at the Human-Technology Frontier, one other of NSF's ten Big Ideas for Future Investment.
Many teachers are overwhelmed by the many duties and tasks they have to handle, resulting in burnout and reduced teacher job satisfaction and retention rates (Grayson & Alvarez, 2007;Rhodes, Nevill, and Allan, 2004). To assist teachers, major goals and corresponding Scale-up Projects include: (1) to help teachers better understand the potential of using AISs and data science to transform education including their job performance and satisfaction; (2) to propose and investigate learner-teacher-AISs collaboration models and interfaces including the validation of a framework for learning experience design; and (3) to design and develop dashboards for teachers to learn from, interpret, and make decisions based upon fine-grained, comprehensive learning data. Helping teachers, parents, and other stakeholders understand the potential of data science and AISs is important for LDI's transforming communities of practice effort. To this end, we plan to develop new curricula for data literacy to be used by teacher training programs.
Finding the best learner-teacher-AISs partnerships could have transformative impact on the learning ecosystem such as freeing teachers from certain duties that AISs can do in an autonomous manner thus allowing them to focus on higher level tasks such as designing new instructional materials or novel tailored interventions for students, , motivational support, and other tasks for which AISs are not ideal This better distribution of duties and coordination between teachers and AISs should lead to a more effective, efficient, engaging, and equitable learning ecosystem. We will study four levels of AISs autonomy with respect to how teachers may use AISs (see later).
As a general principle, all LDI activities will be informed and guided by our goal of using data science and AISs to promote ethics and equity in education (Riddle et al., 2015; Corbett-Davies & Goel, 2018; Gardner, Brooks, & Baker, 2019). At the same time, the Ethics and Equity Expert Panel will review all LDI efforts to ensure ethics and equity aspects are properly addressed. Furthermore, our institute 5-year plan includes a set of activities focusing on ethics and equity which fall into three categories: (1) using data and data science to further our understanding of biases and achievement gaps in the learning ecosystem; (2) understanding and mitigating ethics and equity throughout the data lifecycle with a focus on algorithmic bias and developing tools to address these issues throughout the work of the LDI; and (3) increasing diversity and inclusion during collaborative learning activities.
Evaluation and analysis are key elements of the LDI convergence framework to both demonstrate its effectiveness and provide a way to identify opportunities for improvement and refinement. We focus on quantitative and qualitative metrics for LDI community building and engagement efforts, identifying investment opportunities priorities, and development and refinement of prototyping concrete task or Scale-Up Project activities. For quantitative metrics, to account for different perspectives, we will report how many experts and from how many different disciplines contribute to specific tasks (e.g., identification of data requirements for Investment Opportunity Area 1, above). For each expert, we can monitor their individual contributions in terms of content (e.g., word counts), comments, and revisions to others' contributions (by using shared documents that track such metrics). More qualitatively, each member's contributions will be assessed in terms of the depth of their contributions. A researcher might identify that a particular expert's contribution initiated the development of a novel solution that could improve the detections of learners' emotions in a classroom context. Furthermore, we report the scientific and societal impact of the proposed convergence framework. Scientific impact can be reported in terms of the number of publications, presentations, tutorials, meetings, email exchanges and other forms of direct communication (among LDI members and the broader research community) as well as improvements of prototype solutions over existing solutions. Other scientific success measures can monitor longer term impact such as how many citations the products of this project generate and how many research groups integrate the proposed solutions (e.g., user adoption of analysis toolkits developed).
Societal impact can be assessed through impact on learners and teachers as well as impact on the learning ecosystem (e.g., in terms of how LDI efforts have made aspects of the learning ecosystem more effective, engaging, equitable, efficient, relevant, and affordable, as well as other outcomes such as transforming educators' community of practice).
An important requirement for the evaluation process is documentation of the various elements of the convergence framework. For this purpose, for instance, all meetings of the leadership team were recorded (key metric: hours of meetings and interactions; volume of those interactions). Other processes and activities have been documented in various ways such as Google docs, meeting recording, and Slack asynchronous discussions. For instance, the convergence process implemented to generate the 5year institute plan has been well documented through other records such as spreadsheets used in NGT processes employed by the various Expert Panels to generate and rank ideas for investment opportunities to be included in the 5-year plan.
We will illustrate how we have been evaluating the effectiveness of convergence framework holistically as well as from the perspective of Expert Panels. For brevity, we illustrate the evaluation of the convergence process from the perspective of the Learning Engineering Expert Panel. The LDI's Learning Engineering Expert Panel comprised a diverse group of researchers and developers with vast experience in research and development of learning systems. The 10-member expert panel was drawn from the academe, government, and industry.
The Learning Engineering Expert Panel, like other LDI expert panels, engaged in two major activities that contribute to the LDI Phase 1 project:
-Provide input to each of the concrete tasks (forward-looking "Scale-Up Projects") addressing various challenges in the learning ecosystem with the goal of converging to solutions to those challenges that account for input from many domains.
-Identify, rank, and propose investment opportunities for the 5year plan of the convergence or institute phase (LDI Phase 2)
The concrete task reviewing and feedback process involved significant expert time (see Table 2, which presents a summary of the quantitative evaluation of the initial cycle of the review and feedback process by the Learning Engineering Expert Panel).
In addition to this quantitative summary of the convergence process related to concrete tasks, we also developed a 5-stage model to characterize the maturity of concrete tasks: (1) ideation or initial idea, (2) conceptualization and convergence of a data science solution with input from experts from many domains, (3) implementation & refinement, (4) product release (e.g., an emerging data science prototype or dataset release), ( 5) impact, in which the product from stage 4 is adopted by or integrated into external research projects or a learning environment, having some external impact on the research landscape or on the learning ecosystem. Work of LDI participants during the conceptualization phase has centered primarily on concrete tasks in the first four phases (ideation, conceptualization and convergence, and product release). Ideally, the transition from concrete task to "Scale-Up Projects" in LDI Phase 2 will reflect progression to later stages of this model. Holistically, the LDI convergence framework can be evaluated in terms of the level of engagement of a diverse team of researchers, developers, practitioners, and other stakeholders as well as its key outcome, which is the 5-year plan for the institute or convergence phase which was described and submitted as a proposal to NSF. The level of engagement can be summarized briefly by noting that our 60+ strong team participated so far in 3 all-hands meeting each for about 20 hours (2.5 days) resulting in 60 x 20 = 1,200 expert hours of effort. Experts spent hundreds of additional hours spent in other meetings and other activities. Most meetings were recorded and transcribed. A more detailed, quantitative and qualitative analysis is being conducted right now, and the results will be widely disseminated.
We conclude this progress report by briefly presenting two emerging ideas from the collective work of the LDI during its conceptualization phase to date.
Our work so far also results in a number of policy recommendations:
-Publicly funded education technologies similar to publicly funded education adopted in the 19 th and 20 th century.
-Learning data owners keep ownership of their data and have decision power with respect to where their data is stored, how the data is accessed, by whom and for what purposes, how their data is used, and if their data can be shared, with whom, and under what conditions and circumstances.
-Learning data infrastructure is needed to enable responsible learning data collection, storage, access, sharing, and processing.
-The need for a culture shift in teacher training programs and data literacy curriculum for future teachers.
Finding the best teacher/learner-AISs partnerships could have transformative impact on the learning ecosystem, potentially freeing teachers from certain duties that AISs can do in an autonomous manner and allowing teachers to focus on higher level tasks such as tailored, individualized interventions for students, motivational support, and other tasks for which AISs are not ideal. This better distribution of duties and coordination between teachers and AISs should lead to a more effective, efficient, engaging, and equitable learning ecosystem.
We defined and intend to study four levels of AISs' "autonomy" with respect to how teachers can use such AISs: (1) fully autonomousteachers need little (if any) training and have little (if any) involvement in "tuning" AISs, (2) minimal teacher involvementteachers tune the parameters of the AISs with the help of the AISs developer at the beginning of the school year or semester (minimal teacher training with respect to the workings of the AISs), (3) average teacher involvementteachers require training, and they work with the system on a weekly basis selecting instructional tasks and receiving information from the AISs, (4) teacher-driventhe teachers exerts full control of the AISs including overriding decisions the AISs may take or suggest, the teacher will interact almost daily with the AISs. There is in fact one other level (level 0) which are self-improving, fully autonomous AISsthey improve with experience with minimal or no developer intervention. While we will explore as resources permit the role of data science to enable such level 0, self-improving fully autonomous AISs, from a teacher and learner perspective they are similar to the fully autonomous level of AISs (level 1).
We plan to study and understand the trade-offs in terms of teacher involvement in tuning AISs vs. levels of AIS autonomy. For instance, teachers may choose a fully autonomous mode of operation for an AIS meant for students working independently with the system afterschool as supplemental instruction, whereas for student interactions with the AIS during a class period (i.e., in a blended-learning environment), the same teacher may choose to control more the behavior of the AISs. Similarly, teachers may decide to use/download a pre-trained learner model and update it with data from her students, assuring data security and privacy and maintaining full ownership of the data. They may decide to share a sample of her own student data to benefit the pooled/pre-trained models that everyone can download as default.
LDI intends to serve as an agent of change for how research questions are conceived and addressed through interdisciplinary collaboration such that LDI's impacts will propagate and evolve beyond the lifetime of the award.
More specifically, we have the explicit intent to start a culture shift in teacher training programs through two specific actions: (1) involve a few dozen teachers and pre-service teachers in our work in order to co-design solutions and account for their input and expose them to the potential of data science and AISs while also introducing them to science convergence approaches to address key challenges in education and (2) develop new curriculum recommendations for teacher training programs as well as accompanying training materials to build capacity for teachers and other stakeholders to adopt AISs and data science approaches, tools, and principles to improve learning and teaching.
Wider adoption of advanced data-driven science and engineering approaches and tools such as AISs is still lacking for at least three reasons: (1) Data science and education technology training is often limited in teacher training programs. (2) The sophistication and complexity of AISs often entail a significant effort to train teachers to effectively use such advanced education technologies. (3) New approaches are often developed with a lack of substantive involvement of educators and schools.
Involving educators will help to ensure that new approaches based on data science to tackle various education challenges, nextgeneration AISs, and learning environments that include AISs, are designed to help eliminate biases and promote equity, inclusion, and diversity, offering high quality education opportunities for all learners. We will therefore push for schools, teacher training programs, and instructors to collaborate more with data science and educational technology researchers and developers to improve learning and instruction. To this end, in addition to substantive involvement of teachers and other stakeholders in LDI activities, we will explore avenues for delivering professional learning, including workshops for teachers, summer schools (e.g., by adding a track to CMU's LearnSphere summer school) for pre-service teachers and Research Methods instructors in schools of education.
We are an expanding community of practice and promote Scale-Up Projects that will ideally become bona fide research programs beyond the award period, securing their own funding as they make scientific progress. Furthermore, Scale-Up projects and research thrusts will ideally result in career-long efforts for some younger faculty members.
To sum up, our strong team of interdisciplinary experts, developers, and practitioners will work together during the 5-year LDI institute project to move current practices beyond the small-scale studies to bring the learning sciences into the era of big data and interdisciplinary science convergence. The impact of LDI will be felt far and wide, propagating and evolving beyond the lifetime of the award and beyond our own team, acting as an agent of change for how research questions are conceived and addressed through interdisciplinary, collaboration, and co-designed research and development. The proposed processes, methods, and studies pave the way for taking these outcomes to other domains.
The Learner Data Institute is sponsored by the National Science Foundation (NSF; award #1934745). The opinions, findings, and results are solely the authors' and do not reflect those of NSF.
The LDI is a "frameworks" project funded by the United States' National Science Foundation (NSF) under the Data-intensive Research in Science and