=Paper=
{{Paper
|id=Vol-2128/designsci6
|storemode=property
|title=Adopting a Productive-Failure Instructional Design in Project-Based Learning to Improve Primary Students' Collaborative Problem Solving
|pdfUrl=https://ceur-ws.org/Vol-2128/designsci6.pdf
|volume=Vol-2128
|authors=Yanjie Song,Kan Man Lung
}}
==Adopting a Productive-Failure Instructional Design in Project-Based Learning to Improve Primary Students' Collaborative Problem Solving==
https://ceur-ws.org/Vol-2128/designsci6.pdf
Adopting a Productive-failure Instructional Design in Project-based
Learning to Improve Primary Students’ Collaborative Problem Solving
Yanjie SONG, The Education University of Hong Kong, Hong Kong, Email: ysong@eduhk.hk
Ka Man LUNG, Man Kiu Association Primary School, Hong Kong, Email: carmenlung@mkaps.edu.hk
Abstract: The paper reports on an empirical study on adopting an innovative pedagogical
design in project-based learning to improve upper primary students’ problem solving skills in
science learning. A Grade 6 class in a Hong Kong primary school was involved in this study.
The findings show that the students produced high group artifacts/projects in problem-solving
quality and were positive in facing challenges in their project-based learning process. This
indicates that project-based learning with PF instructional design can be an effective way to
develop students’ problem solving skills.
Introduction
In the digital age, problem solving is one of 21st Century skills critical for preparing students in a global
economy and society (Voogt & Roblin, 2012). This is in line with what is advocated in science education in
Hong Kong which comprises a core component of the primary school General Studies’ curriculum. Science
education promotes creativity through the problem-solving process in authentic learning environments
supported by digital technologies (The Education Bureau, 2011). However, in practice, science learning, in
many cases, is still largely disconnected from learners’ daily life and confined to textbook learning
(Anastopoulou et al. 2012). Students are passive knowledge receivers resulted from a lack of opportunities to be
question askers, inquiry method designers and action-takers. This empirical study attempts to explore an
innovative pedagogical design to improve upper primary students’ collaborative science learning competency.
Literature
Collaborative problem-solving in science through project-based learning
In science learning, the project-based learning approach aims to involve students in working at real-world
problems in small groups and striving for solution options where the teacher acts as a facilitator (Brundiers &
Wiek, 2013). Thus, problem solving competency is critical in carrying out the project. Its processes include:
exploring and understanding; representing and formulating; planning and executing; and monitoring and
reflecting (PISA, 2017, p. 9). In many cases, problem solving involves collaboration, especially in dealing with
complex tasks. Collaboration is defined as “the activity of working together towards a common goal” (Hesse,
Care, Buder, Sassenberg, & Griffin, 2015, p. 38). Collaborative problem solving requires social and cognitive
skills to develop shared understanding, take appropriate action and establish and maintain team organisation to
solve the problem (Dillenbourg, 1999). Guided inquiry is advocated in existing collaborative problem solving
for young learners, where scaffolding is provided when students encountered learning difficulties to avoid
failure in making the inquiries (e.g., Hakkarainen, 2003). However, Kapur (2015) posits that learners are
learning from failures. One concern of this study is: can young learners learn better without the scaffolding in
their inquiry process?
Productive failure instructional design
Productive failure (PF) is defined as “a learning design that affords students opportunities to generate solutions
to a novel problem that targets a concept they have not learned yet, followed by consolidation and knowledge
assembly where they learn the targeted concept” (Kapur, 2015, p. 52). PF instructional design involves two
phases: (1) students first engage in unguided problem solving activities to elicit their prior knowledge,
particularly the failure to solve the problem; and (2) students then use this information to consolidate and
aggregate new knowledge after the teacher helps solve misconceptions (Kapur, 2016). The failure stems from
the fact that learners are commonly unable to generate or discover the correct solution to the novel problem by
themselves; on the other hand, they are able to generate sub-optimal or even incorrect solutions to the problem,
the process can be productive in preparing them to learn better from the subsequent instruction that follows.
Indeed, in science learning, generating “wrong answers” may help to focus students’ attention on the
complexities and frustrations of a good investigation plan or design (Hodson, 2014).
This empirical study aims to examine the effectiveness of project-based learning with productive failure
(PF) instructional design in improving primary students’ collaborative problem solving skills in science learning.
The research question was: Was the project-based learning with PF instructional design effective in improving
students’ collaborative problem solving skills?
Methods
Research context and participants
This study was conducted in a Grade 6 class with 27 aged between 12 and 13 years’ old on the project of
“Plants and Environments” in a primary school. Students were classified into four groups. Each group had 6 or
7 members. The school lent an iPad and a laptop to each group. Students could use their own mobile devices as
well in their inquiry. Before this study, the students had conducted project-based learning supported by mobile
devices for two years, and were familiar with the apps of (1) a social network platform - Google Classroom for
groups to document their project-based learning process; (2) a built-in camera on mobile devices for picture
taking to collect data; and (3) an augmented reality (AR) App: MKAPS. The AR artifacts were created using the
school’s AR creation platform (http://mkaps.ar.myprint.asia). Students created a video clip first, and then
uploaded it to the AR creation platform to generate a picture. As long as students downloaded the app –
MKAPS from Apple Store, they could scan the picture to view the videos.
The project lasted for two weeks. The teacher had over six years’ teaching experience and had been
involved in the project-based learning programme initated by the school in the past two years. Before
conducting this study, the researcher conducted two 2-hour teacher professional development workshops on
social constructivist principles and pedagogical models, especially on inquiry/project-based learning and
productive failure instructional designs in the school. The researcher also co-designed the project of “Plants and
Environments” with the teacher after the two professional development workshops, adopting the PF
instructional design.
Project-based learning with PF instructional design on “Plants and Environments”
Because the projected lasted two weeks, the students were suggested choosing two kinds of Rhizome plants to
grow, and find out factors that influenced the growth of the plants they chose. In doing so, they needed to work
in groups to make their own plans to raise the plants. In order to understand better the factors that contributed to
the growth of the plant, they usually prepared two or three plants of the same kind to grow in different
conditions, and observed, documented and explained their process of growth.
The project-based learning with PF instructional design consisted of two stages (see Figure 1). Stage 1
was designed for unguided collaborative problem solving activities: (1) Explore and understand: group members
explored and gained some understanding of the plants and environments in a field trip to a school farm; (2)
Represent and formulate: group members discussed with each other after the field trip to reach a common goal
for doing the project; (3) Plan and execute: group members worked out plans for the project and executed them;
and (4) Monitor and reflect: group members monitored and reflected on the progress of the project to reach
better shared understandings. Stage 2 was designed for consolidating and aggregating new knowledge by
solving misconceptions and failures facilitated by the teacher.
Figure 1. Project-based learning with PF instructional design.
Students in groups explored factors that influenced plant growth in different conditions on their own
before learning new concepts followed by the teacher’s help in solving misconceptions, from which students
consolidated and aggregated the conceptual knowledge. Students in groups investigated problems related to
plants and their environments in across different spaces like home, farm, school and online with mobile devices.
Data collection and analysis
Data collection included student artifacts such as pictures or videos captured using mobile devices to document
the process of plant growth, searched and downloaded online resources, concept maps, notes related to the plans
of the group project, and group created project booklets as final project products. In addition, two focus group
post interviews were conducted in order to “hear the participants’ voice” through the interactions. The interview
items (5 items with follow-up questions) were related to the research aims about collaborative problem skills in
the project-based learning. The interview items were reviewed by an experienced researcher. Thus, the
interview instrument had its face validity. Each of the interviews consisted of three to four members, and lasted
about 35 minutes which was recorded. Further, students’ post-reflections were also collected after the project-
based learning. The reflections were in the forms of written text and video recordings in Chinese. All the data
were submitted to the Google Classroom, some of which were selected and incorporated into the final project
product - project booklet. The video recordings were embedded in the booklet in the form of AR picture with a
star logo, which could be scanned and accessed. The interviews were conducted in Cantonese. All the recorded
data were transcribed from Chinese into English. Finally, pre- and post-domain tests consisting of eight multiple
choice items (a, b and c choices) on conceptual knowledge related to plants and their environments were
conducted before and after the project-based learning. The items were constructed based on the science learning
curriculum.
Mixed data analysis methods were utilised. First, “process-oriented analysis” was adopted in a natural
context (Järvelä, Veermans, M., & Leinonen, 2008, p. 305) including on-task analysis and content analysis to
understand the process and outcomes of the students’ collaborative problem solving skills in the project-based
learning. In particular, process-oriented analysis using a multiple-methodological qualitative approach via
overlapping and interactive analysis of data between on-task analysis and content analysis offered a more
profound understanding of the cases (Song, 2016). On-task analysis in this study was conducted using the
framework of the “matrix of collaborative problem solving skills” (PISA, 2017) premised on the evidence-
centered design (ECD) framework (Mislevy & Haertel, 2006), in which “assessment is considered a process of
reasoning from imperfect evidence using claims and evidence to support the inferences being made about
student proficiency” (PISA, 2017, p. 26,).
In this study, the framework of the “matrix of collaborative problem solving skills” focuses on the
analysis of student activities in each of the project-based learning task components (vertical): “explore and
understand; represent and formulate; plan and execute; and monitor and reflect (PISA, 2017) together with the
collaborative components (horizontal) of “establish and maintain shared understanding, take appropriate action
to solve the problem, and establish and maintain team organization”. The score on each component was given
based on the detailed instructions in PISA (2017). Each task contained one or more items to be scored. Each
item was coded from zero to a number of categories (0, 1, …categories). The project-based learning task
components were coded as (A) Explore and understanding (20 scores); (B) Represent and formulate (25 scores);
(C) Plan and execute (30 scores); and (D) Monitor and reflect (25 scores). The collaborative learning
components were coded as (1) Establish and maintain shared understanding (45 scores); (2) Take appropriate
action to solve the problem (25 scores) and (3) Establish and maintain team organization (30 scores). The
collaborative problem solving competency of each group was the result of the matrix ABCD1, ABCD2, and
ABCD3. The weighting of the collaborative learning components (1) and (3) were higher because these
competencies were closely related to collaborative skills; and (2) was more related to problem-solving behavior
within a collaborative context (PISA, 2017).
Content analysis was used to analyze focus group interviews, student post-reflections and project
booklets using the matrix of collaborative problem solving as the coding framework. All the data analysis
results served for addressing the research question about the effectiveness of project-based learning with PF
instructional design. The inter-coder reliability regarding collaborative problem solving competency was
assessed by two independent researchers. The Pearson correlation between the two coders was .81. This
indicated that the data analysis results were reliable. Finally, descriptive quantitative analysis using SPSS
version 24 was conducted to analyze pre- and post-domain tests.
Research results and discussions
Research results
To address the research question: “Was the project-based learning with productive failure (PF) instructional
design effective in improving students’ collaborative problem solving skills?”, the results are presented in three
ways: First, overall results of collaborative problem solving skills were presented, followed by the results of
pre- and post-domain tests. Finally, the results of one group’s collaborative problem solving process were
selected in order to evaluate the groups’ collaborative problem solving process.
Results of collaborative problem solving competency
The final results of all the groups’ collaborative problem solving skills are presented in Table 1.
Table 1. The Results of collaborative problem solving skills in Class 1 and Class 2
Class Group *Matrix1 *Matrix2 *Matrix3 Total=100
1 G1 ABCD1=33 ABCD2=19 ABCD3=24 76
G2 ABCD1=28 ABCD2=19 ABCD3=21 68
G3 ABCD1=38 ABCD2=24 ABCD3=25 87
G4 ABCD1=37 ABCD2=23 ABCD3=23 83
* Matrix1 = The project-based task components A, B, C D versus collaborative learning component 1
Matrix2 = The project-based task components A, B, C D versus collaborative learning component 2
Matrix3 = The project-based task components A, B, C D versus collaborative learning component 3
Table 3 shows that in general the groups’ scores of collaborative problem solving skills were high. This
indicates that generally speaking, students performed well in the project-based learning with PF instructional
design.
Results of pre- and post-domain tests
A difference score for each student between post- and pre-domain tests was calculated. The mean difference of
the class was .82. This means that there was comparatively a large change in the students’ learning.
Results of a group’s collaborative problem solving process
In order to present collaborative problem solving process, one group with six members was chosen randomly.
The group chose shallots and mug beans to grow and explored the relationship between these plants and their
environments. We selected some of the artifacts created by the group as evidence to demonstrate their
collaborative problem solving process under the framework of matrix of collaborative problem solving skills.
“Explore and understand” and “Establish and maintain shared understanding” (A1)
At the collaborative “explore and understand” stage, the students took pictures during their visit to the school
farm in the field trip and made reflections after the visit. They paid more attention to the shallot which was the
focus of their study and collected detailed materials for their further exploration.
The group members’ reflection was quite detailed and in depth. They indicated that they got to know a
variety of plants. Two members reported that they learned new knowledge about plants like edible plants and
herbs. For example, Member 1 said, “Plants can help people in various ways. Except being used for food, Li
Shizhen used herbal medicine to treat the disease and save people’s lives in ancient times. Besides, plants can
convert carbon dioxide to oxygen in order to provide fresh air.”. In addition, the members learned how to plant
shallots outdoors and indoors. In particular, they took some pictures of shallots in an outdoor garden in order to
compare them with what they planted, and identified the differences between them. Thus, at this stage, members
reached common understanding of the plants and what topic of the project they planned to work on. They would
like to explore whether it was more effective to plant shallots in water than in soil, and what factors might
influence the growth of the shallot.
“Plan and execute” and “Take appropriate action to solve the problem” (C2)
In growing the plants, the group members planted shallots in three different living conditions and explored how
the environmental factors like sunlight and water influenced the plants’ growth. When facing with problems in
the project, they would like to look for materials and discuss with their group members to propose solutions
collaboratively. In addition, they collected detailed data in order to develop a comprehensive understanding of
their experiments (see Figure 2). They took down the data about temperature, humidity and luminosity in three
environments, and the growth height of shallots daily. They also summarized the status of the plant growth
daily. Moreover, they took notes on the pictures to illustrate the growth process of shallots clearly.
Figure 2. Detailed data records of 1G3.
“Monitor and reflect” and “Establish and maintain shared understanding” (D1)
In monitoring the growth of the shallots, the group members focused on comparing the growing situation in
three different living conditions: (a) air, water and sufficient sunlight; (b) air and water without sufficient
sunlight; and (c) only air, no water and sufficient sunlight. The experiment results (see Figure 3) show that only
the shallot in the first living condition (a) began to sprout and grow (indicated with an red arrow), while shallots
in other living conditions became dried (c) or the outer layer of the shallot started to fall over gradually (b).
The group members made three conclusions based on their experimental results, and by studying other
groups’ experimental results. Firstly, they believed that some kinds of plants could grow well in different living
conditions but others could not. They observed the growth situation of the lettuce in another group, and found
that it could grow well in the soil and in the water, while their shallot could not grow in water. Secondly, plants
could not grow well without photosynthesis. They observed that seedlings of their shallot in the first living
condition were green. Also they observed that the lettuce raised by another group in the environment with the
air, water and sufficient sunlight was green; by contrast, the lettuce in the environment without sufficient
sunlight was light green. Thirdly, water, the air and sufficient sunlight were essential for plant growth. They
compared shallots in two other living conditions, as well as the lettuce and radish leaves compared in same the
conditions by other groups, and found that none of them could grow healthily. They synthesized their findings
as well as the findings from other groups, and drew a comprehensive conclusion to solve their problems
successfully after the teacher’s facilitation at the end of the project.
a b c
Figure 3. a, b and c. Shallots in three different living conditions from left to right.
“Monitor and reflect” and “Establish and maintain team organization” (D3)
In the group reflections, the members made more detailed reflections after the project-based learning. They
expressed their views and shared them with other groups. They also made video clips and created the AR
artifacts using the AR app adopted in this project to reflect what new knowledge they learned in the project-
based learning. They reported that shallots could be planted in water. But shallots needed to be transplanted to
the soil because they needed to get nourishment from the soil. They also knew that different kinds of plants
needed to live in different environments. For example, mung beans could grow faster in an environment lacking
abundant sunlight, and shallots could not grow well in the water but lettuce could. Besides, they understood in
more depth about the importance of photosynthesis for healthy growth of plants. Most importantly, through this
collaborative project-based learning activity, they realized teamwork was vital for the success of their project.
Moreover, even though they failed to grow the shallot in water, they obtained ample evidence to solve their
problems consolidated by the teacher, and did not feel frustrated or disappointed about the failures. They
mentioned that the sharing of the group project work in the class and the teachers’ facilitation and instruction
was useful for them to solve their misconceptions and get deeper understanding of the concepts related to plants
and their environments. Thus, they reported that they did learn from failures.
The final project booklet created by the members was presented logically and scientifically from how to
explore and understand plants and their environments in a field trip to a school farm to what problems they
focused on, how to plan and work out the problems, then to making what they have learned (deep reflections) at
the end of the report (see Figure 4). What was most impressive about the group was that at the end of the project
booklet, they compared the features of the shallots growing outdoors that they observed on the farm with the
one they grew indoors in soil, and identified the differences between them. They also consolidated their findings
by observing other group’s project of growing lettuce indoors. This suggests that the group advanced their
conceptual knowledge about plants and their environments in the project-based learning. Throughout the
group’s booklet, the AR artifacts (video clips) that the members created were embedded in the booklet (see
Figure 4 with a star logo for an example). By doing so, the project booklet was augmented and enriched by
video clips. The members in the group were motivated to produce the best work they could and deepened their
knowledge. They reported that the artifacts, especially the project booklet they created made them feel strong
ownership of their own learning and would like to do more such kind of studies in future.
Indicating an AR
artifact embedded in
the booklet created
by the group.
Figure 4. Selected picture from the Project Booklet embedded with AR artifact
To conclude, the findings of this study indicate that productive-failure instructional design is conducive
to developing primary students’ collaborative solving skills in science learning.
References
Anastopoulou, S., Sharples, M., Ainsworth, S., Crook, C., O'Malley, C., & Wright, M. (2012). Creating personal
meaning through technology-supported science inquiry learning across formal and informal
settings. International Journal of Science Education, 34(2), 251-273.
Dillenbourg, P. (ed.) (1999), Collaborative learning: Cognitive and computational approaches, Advances in
Learning and Instruction Series, Elsevier Science, Inc, New York, NY.
Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A framework for teachable collaborative
problem solving skills. In Assessment and teaching of 21st century skills (pp. 37-56). Springer
Netherlands.
Hakkarainen, K. (2003). Progressive Inquiry in a computer-supported biology class. Journal of Research in
Science Teaching, 40(10), 1072-1088.
Hodson, D. (2014). Learning science, learning about science, doing science: Different goals demand different
learning methods. International Journal of Science Education, 36(15), 2534-2553.
Järvelä, S., Veermans, M., & Leinonen, P. (2008). Investigating student engagement in computer-supported
inquiry: A process-oriented analysis. Social Psychology of Education, 11(3), 299-322.
Kapur, M. (2015). The preparatory effects of problem solving versus problem posing on learning from
instruction. Learning and Instruction, 39, 23-31.
Kapur, M. (2016): Examining Productive failure, productive success, unproductive failure, and unproductive
success in Learning, Educational Psychologist, 51(2), 289-299.
Mislevy, R.J. and G. Haertel (2006), Implications of evidence-centered design for educational testing (Draft
PADI Technical Report 17), SRI International, Menlo Park, CA.
PISA 2015 collaborative problem-solving framework (2017). Retrieved from:
https://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Collaborative%20Problem%20Solvi
ng%20Framework%20.pdf
Song, Y. (2016). “We found the ‘black spots’ on campus on our own”: development of inquiry skills in primary
science learning with BYOD (Bring Your Own Device). Interactive Learning Environments, 24(2), 291-
305.
The Education Bureau (2011) http://www.edb.gov.hk/attachment/en/curriculum-development/cross-kla-studies
/gs-primary/gs_p_guide-eng_300dpi-final%20version.pdf
Voogt, J., & Roblin, N. P. (2012). A comparative analysis of international frameworks for 21st century
competences: Implications for national curriculum policies. Journal of Curriculum Studies, 44, 299–321.