Project-based learning (PBL) approaches are organized around open-ended challenges that require feedback and revisions to facilitate learning. However, feedback on projects provides a challenge as the varied states of projects, the equivocality of the process, and the varied solutions that students might create present scafolding challenges to meet the needs of the students. In this paper, we propose an ontology that characterizes activity flow and discourse use during a project. More specifically, we analyze weekly student reports in an interdisciplinary entrepreneurship class to derive a categorization system. The categories within the ontology, in combination, help define a “state” for the project. By identifying the “current state”, human (or intelligent) instructors can provide relevant and timely feedback that can help student teams navigate various procedural knowledge problems associated with project-based learning. This augments the educator's capacity to meet the specific needs of the student by providing the building blocks for solutions that automate and/or replicate their insightful guidance.
Project-based learning (PBL) presents challenges in recognizing when there is a need for
intervention. In a traditional learning setting, educators have clearly defined learning goals that
directly equate to assignments. This allows educators to recognize when attainment targets
are not being met leading to intervention. However, as written by Mergendoler et. al, the
most efective application of PBL allows students to make innocuous errors without immediate
correction [
Determining when any intervention is necessary becomes complicated in self-regulated
learning settings, such as project-based learning. A current approach to this problem involves
recognizing help-seeking behaviors. A model described by Aleven et. al, acknowledges when
help is necessary based on knowledge of positive help-seeking behaviors [
While recognizing help-seeking behaviors could be used as a notifier for a need for
intervention, it lacks the ability to provide personalized feedback. Personalized feedback has a significant
impact on the learning outcomes in a PBL setting. Korbi is an online learning platform that
includes the application of concepts in a project-based learning setting [
The use of computers to provide feedback (or scafolding) is not new. Various strategies
and customization within scafolding include feedback, question prompts, hints, and expert
modeling [
Ontologies, a set of concepts and categories in a subject area, are useful frameworks that facilitate
the identification and clarification of diferent concepts or systems across the same domain [
An additional aid in identifying learning and project development patterns is a standardized project artifact. Progress reports simulate an authentic medium and artifact used in the real world to navigate the risks and uncertainties associated with projects. This standardized artifact allows the instructor to assess what assumptions have yet to be de-risked, validated, or addressed. Knowing the current state of the assumptions facilitates the selection of the appropriate scafolding that provides enough support while still allowing students to learn for themselves. This avoids stunting the development of meta-skills and potential prompts of disengagement due to perceived power relationships associated with instructor expertise.
The “Assumptions” assessed are rooted in the types of risks and uncertainties that are common to commercialization projects but can largely be generalized to projects as summarized in Table ??. The table also includes keywords or concepts that indicate the action taken to address or navigate the risks and uncertainties of the project. It should be noted that these assumptions, though interconnected, are not necessarily sequential in nature. For example, the team may already have a solution in mind for the assumed problem and begin working on building a prototype prior to validating assumptions regarding unmet needs or receiving the funding needed to build a working prototype or production-ready design. The uncertainty and risk associated with these assumptions fluctuate and change such that even though progress may have been made in one area, new information in another area might trigger the need to start all over again. For example, they may discover a diferent unmet need that would require a diferent solution to be built causing them to abandon or modify the design of the prototype. Reported actions can also address multiple assumptions simultaneously. For example, when seeking to understand the unmet needs the same conversation might reveal the current state and their willingness to pay for alternative solutions, and the whole process of reaching out to speak with a client or customer might reduce uncertainty about how to find and attract additional customers or clients.
Our dataset is comprised of weekly team reports from nine teams enrolled in an entrepreneurship class. Dr. Haines, the class instructor, and a student volunteer served as subject-matter-experts and labeled the dataset using the categorization system described above. More specifically, each sentence was labeled as addressing one or more assumptions. A weekly team report contains three main sections: (1) Progress that they made in the last week, (2) Plans for next week, and (3) Roadblock or problems they are facing (see example in Figure 1).
Each sentence was labeled as addressing one or more assumptions. Each weekly report can, then, be used to reflect the team’s “state”, represented by the combination of assumptions of the sentences within. We annotated a total of 712 sentences from all teams’ reports. Of the 712 sentences, only 540 sentences were “unique.” We can deduce that the remaining sentences appeared in multiple reports (within the same team). This might have occurred because teams report the same sentence under “Progress” that they reported under “Plans” in a previous week’s report.
To visualize how a team’s plans outlined in their weekly reports culminated in reaching their original goals described in their MOKR, a scoping document written at the beginning of the year defining their mission-oriented objective and key results, we drew a flowchart for each team (see example in Figure 2). These flowcharts take the sentences from the “Progress” section of the weekly reports and organizes them by week number and relation to each other. For instance, in the top left corner of Figure 1, “Find 5 competitors to the planned VRSS service” from Week 2, has been placed directly prior to “Find specific details about 3 competitors” from Week 3.
These flowcharts allowed us to analyze how teams progressed within an assumption. When we mapped out our first team, we noticed that there seemed to be many ‘branches’. We were able to associate each of these ‘branches’ with our assumptions.
Associating sentences describing related events within a certain boundary, entrepreneurship in this case, is inherent to a human but can become complicated for a computer. Our assumptions will be an important factor for the implementation of computer-based scafolding. The assumptions provide context to phrases described in the weekly reports, so that a computer will be able to recognize the way that groups move through steps to reach their goal. This will allow the computer-based scafolding to provide multiple options specific to a group’s current position in their project. We already trained a base-line language model to predict the (sub-)set of assumptions that the team is working towards during each week based on this dataset. For our dataset of size 540, 378 sentences were used for training and 162 were used for testing. The maximum micro-averaged F1 score we obtained from the entire dataset is 0.72, with two assumptions getting an F1 score above 0.84. Given our dataset size, we believe we arrived at a good micro-averaged F1 score. We plan to include reports from 1184 weekly team reports from 107 teams/projects from the same course from previous semesters.
Also, by utilizing our assumptions, we could see that there was no obvious pattern through which teams moved through their projects. We established that the structure of the flowcharts were vastly diferent between groups reinforcing that there is no clear path that could be taken by each group allowing them to reach their individual goals outlined in their MOKR.
One can reliably use the characterization via “assumptions” to track the progression of a team, regardless of the domain or scope of the project. With that benefit, we, as instructors and “instrumentors’’ of PBL, can start to delve deeper into our sense-making process and as we aim to design and develop systems (including ones with intelligence and/or collaborative interfaces) to facilitate PBL. Here are some examples of such opportunities now available for exploration: 1. Building a database or digital library to learn from peers or past teams: By looking at the steps and/or progress that other teams are making or have made in the past, teams can learn the diferent tasks they might need to undergo to address an assumption. Furthermore, they can create a “priority queue’’ of sorts for these tasks to better estimate which set of sub-tasks to work on in sequence and/or in parallel based on the time it takes to complete them.
2. Track team progress (Passive): Instructors can now build a visualization (or similar) to observe, at a high level, how each team is progressing, and compare/contrast it against other teams in the same class or compare/contrast it against teams in the past (who have worked in similar domains or with similar objectives).
3. Track team progress II; (Active) Intervention and personalized feedback: In a similar manner, instructors can now build solutions (either based on heuristics or through learning) to identify when teams are stalled or are taking more time than commonly required to complete a set of (sub-)tasks to address an assumption. Going deeper, they can parse the specific plans made by teams to address problems they in a previous week (both being categorized under the same ontology class) and in combination with progress they have made in the past to identify patterns of problem solving. Having this at their disposal allows them to exercise judgement on when (and/or whether) to intervene and provide personalized feedback based on the circumstance.
All in all, with the development of the ontology instructors can build an information system that supports information exploration via user scenarios for student teams and instructors alike, through execution of services that run sophisticated models, built upon ever-growing database of project team artifacts that naturally gets bigger and better through use.
In this paper, we share a categorization system that captures the assumptions that student teams needs to address during their project lifecycle. This system is robust enough to be applied across diferent ideas, types, and domains. We apply this ontology to artifacts from teams in an technology commercialization course and can immediately observe time spent addressing each assumption, and track their progression from one “state” to the next. The current state model can be used as input into scafolding design and implementation so that students receive the right information at the right time to allow them to develop and progress.