In this article, we introduce and evaluate the concept of robosourcing for creating educational content. Robosourcing lies in the intersection of crowdsourcing and large language models, where requests to large language models replace some of the work traditionally performed by the crowd. Robosourcing includes a human-in-the-loop to provide priming (input) as well as to evaluate and potentially adjust the generated artefacts; these evaluations could also be used to improve the large language models. We explore the feasibility of robosourcing in the context of education by conducting an evaluation of robosourced programming exercises, generated using OpenAI Codex. Our results suggest that robosourcing could significantly reduce human efort in creating diverse educational content while maintaining quality similar to human-created content. Thus, we argue that robosourcing has the potential to alleviate known issues around learner motivation and content quality that have been shown to limit the benefits of learnersourcing in practice.
Learnersourcing is a broad term that is used to describe student-centered pedagogies that
involve learners in the creation and evaluation of educational resources. In contrast to more
traditional models of teaching, where expert instructors assume the responsibility for producing
the resources that are subsequently used by students, learnersourcing leverages the creativity
and energy of a cohort of students in order to produce large repositories of useful learning
content. Khosravi et al. describe learnersourcing as a form of crowdsourcing that mobilizes
students as “experts-in-training to contribute to teaching or learning while being engaged in a
meaningful learning experience themselves” [
As highlighted by the previous definition, learnersourcing activities ofer several benefits to students. Broadly speaking, these benefits relate to the two primary activities underlying learnersourcing – the generation of content, and the use of content generated by others. When producing novel learning content, students must engage with and understand the concepts they are targeting. The act of creating content leads to more robust recall of information when compared to passive engagement with content produced by others [2, 3]. Generating model answers or solutions as part of this process prompts self-explanation which is also known to be beneficial to learning [ 4, 5]. When using content produced by other learners, there are benefits relating to the ways in which that content is presented and the quantity of resources available. Learners appreciate the dificulties that their peers face and do not sufer from the phenomenon of expert blind spots which can make expert-generated resources challenging for novices to understand [6]. Learnersourced repositories also scale with the size of the cohort, and thus provide a wide variety of content suitable for the needs of individual learners [7, 8].
However, despite the potential benefits ofered by learnersourcing, in practice there are
several challenges to successful implementation. With respect to creating content, issues of
low motivation can prevent learners from properly engaging with the generative aspects of
learnersourcing. Indeed, prior research has shown that students tend to be much more inclined
to use and evaluate resources created by others than they are to create resources themselves
[9, 10, 11]. With respect to utilising the resources produced by other learners, one of the widely
cited issues with learnersourcing is the low quality of some of the content generated by novices
[12, 13, 14]. Low quality resources are of limited use for learning. Eforts to train novices to
produce higher quality resources can help, but these can be time consuming and limit the
scalability of learnersourcing in very large classes [
The recent emergence of large language models (LLMs) presents the possibility to scafold
learnersourcing activities in a way that may address, to some extent, the challenges relating to
low student motivation and low quality content. LLMs have proven to be remarkably adept
at generating realistic human-like content of various types including text, images and source
code. Widely known models such as GPT-3 [
In this work, we propose ‘robosourcing’ as a model to scafold the creation of educational
content. Robosourcing combines large language models and learnersourcing by utilizing the
large language models to facilitate the creation of content. A human (i.e. the learner) is involved
in the robosourcing process as the initiator providing input to the models (e.g. content and topic
priming) as well as evaluating and curating the content generated by the models. Robosourcing
shifts the primary focus of learners from content creation to content evaluation – thus more
accurately reflecting professional practice. We evaluate preliminary evidence 1 highlighting the
potential of robosourcing and argue that it may be an efective strategy for improving learner
productivity and the quality of the content they generate when engaging in learnersourcing
activities.
1Full evaluation to appear in [
A common type of learnersourcing task involves students creating practice questions which can
then be used for drill and practice learning. Across a wide range of educational contexts, drill
and practice activities are both popular and highly efective [
A primary challenge of supporting practice testing at scale is the human efort associated
with generating questions and associated solutions. Large repositories of questions that cover
all relevant concepts are required in order to support efective practice [
A number of automated approaches have been explored for the generation of questions that
could be used for practice testing. These approaches commonly involve the use of question
templates [
In certain domains, such as computer science and mathematics, specialized techniques can
be employed to generate relevant problems. For example, to generate code-tracing practice
questions for computing students, Thomas et al. use a stochastic tree-based generation algorithm
[
Recently, there has been great progress on generative NLP methods (such as OpenAI’s
GPT3 [
Both GPT-3 and Codex have been previously used to solve math and programming exercises
respectively. For math word problems, Cobbe et al. [
In this work, we explore the potential of such models for the scalable generation of practice questions as part of a learnersourcing activity. Learners can leverage the computational efort of large language models to produce content quickly, which they can then evaluate and modify before sharing with others. They can also provide basic priming information to the large language models, such as initial example problems and desired contextual information in the output. We use the term robosourcing to describe this new approach.
A schematic view of the robosourcing model is shown in Figure 1; first, a learner provides a priming exercise that includes a problem statement, a sample solution, and any expected themes and concepts (for programming exercises, automated tests can also be included). Once the priming exercise has been provided, the system automatically generates a pool of exercises for evaluation. This is followed by a filtering phase where the system first automatically filters generated exercises, after which the learner can perform additional filtering. The filtered exercises can then be edited, if necessary, and added to an exercise database. In a typical learnersourcing environment, students are involved in both creating and evaluating content. This robosourcing model leverages the power of large language models for content creation, shifting the primary role of the learner towards evaluation.
In the first stage of the model, a learner provides a priming exercise as input to a large language model. Consider the priming exercise for a programming problem, as shown below. In this case, the format of the priming exercise consists of a label (i.e. Exercise 1) followed by keywords for both the contextual themes and the programming-related concepts within exercise Problem statement
Sample solution Themes and concepts
exercises
RobSys creates
RobSys and human filters creating exercises, which the system then uses to create an exercise pool. The exercise pool is filtered initially by the system based on options given by the user, after which the user can also filter out exercises. The exercises that are not filtered out are added to an exercise database. the exercise (i.e. donuts, function, conditional), a natural language problem statement and a solution (in the form of a Python function). For space reasons, we omit a list of test cases but these can also be included for programming problems. The priming input ends with the explicit prompt for a new exercise to be generated (i.e. Exercise 2), along with the desired concepts and themes that it should target which are expressed as keywords (i.e. basketball, function, list and for loop).
"""Exercise 1 --Keywords-donuts function conditional --Problem statement-Write a function called donut_cost that takes three inputs: the price of a ˓→ donut, the number of donuts bought, and whether or not it is the weekend. ˓→ The function should return the total cost of the donuts. If it is the ˓→ weekend, the donuts cost twice as much. --Sample solution-def donut_cost(price, number, is_weekend): if is_weekend:
return price * number * 2 else:
return price * number """Exercise 2 --Keywords-basketball function list for loop --Problem statement-
If this input is provided, verbatim, to the Davinci-2 model from OpenAI Codex [
total = total + player[2] return total
In the second phase of the model, the learner then evaluates this output and makes a decision about whether it should be shared to a database of filtered problems, which can later be used for practice. Some of this filtering can be performed automatically – for example, in the case of programming problems, the generated code can be executed and tested against a suite of test cases (which can be also generated by the model, but were omitted for space reasons in this example). Any generated problems for which the code cannot be executed, or for which the test cases do not pass, can also be automatically filtered out. Finally, although in the example above we used the Davinci-2 model, the underlying generative model can be switched to something more appropriate for the problem type (such as GPT-3 or similar for contexts other than programming).
To assess the feasibility of robosourcing, we use OpenAI Codex to generate a corpus of practice
exercises suitable for university level introductory programming. The goal of this evaluation is
to determine whether the automatically generated exercises are of suficient quality that they
could be evaluated and modified with little efort by learners. For more details on the evaluation,
please see [
We generated a total of 240 programming exercises. These were a combination of the two programming exercises (see Appendix 6.1), a total of nine themes (and an extra for leaving out the contextual concept) and two programming-related concept sets (and an extra for leaving out the programming-related concepts). This resulted in 10 × 3 × 2 = 60 diferent combinations of inputs (themes × programming-related concept sets × exercise primings). In addition, we explored two values for Codex’s temperature parameter (0 and 0.75) and created two exercises for each parameter combination. In total, this led to a sample of 60 × 2 × 2 = 240 programming exercises. We conducted the evaluation qualitatively and quantitatively. For the qualitative evaluation, we assessed a random sample of 120 of these exercises. 3.1.2. Exercise evaluation For the qualitative part of the evaluation, we assessed the sensibleness, novelty, topicality, and readiness for use of the 120 randomly sampled exercises. For sensibleness, we inspect whether the problem statement contains a sensible and practical problem that students might be expected to solve. For novelty, we use Google search (on phrases contained within the problem statement) to see if we can find the exercise or a similar one online. For topicality of the exercises, we analyse whether the generated problem incorporates the provided theme and concepts from the required sets.
We also evaluate the readiness for use of the exercises that were deemed to be sensible. This category involved both a qualitative and a quantitative aspect. For the qualitative piece, we evaluate whether the generated sample solution matches the problem description. For the quantitative analysis, we examine the complete corpus of 240 programming exercises and assess three aspects. We explore; 1) can the sample solution be executed/run, 2) does the sample solution pass the automated tests that are generated, and 3) what is the statement coverage of the automated tests when the code runs. These analyses were conducted programmatically2.
The qualitative analysis was conducted by four researchers. Each researcher worked on a subsample of the exercises. The researchers assessed the items with Yes / No / Maybe statements, adding qualitative notes for the latter category. All the Maybe answers were then discussed by at least two researchers to determine the consensus label (Yes or No). We tallied the Yes / No answers to provide quantitative results. 2Analysis of statement coverage used Coverage.py version 6.3.2 (https://coverage.readthedocs.io/)
The statistics for sensibleness, novelty, readiness for use and topicality of the evaluated programming exercises are presented in Table 2. Out of the evaluated programming exercises, 75.0% were sensible, 81.8% were novel, and 76.7% had a matching sample solution. In addition, 79.2% of the exercises matched the priming theme, and more than three-quarters had the desired programming concepts present. The most common reason for a programming exercise not being sensible was the problem statement asking for the calculation of some sort of a value that required another value that was not available (e.g. amount of tax but missing price). In terms of the programming concepts, ‘arithmetic’ was missing relatively often despite the programs featuring a ‘+’-symbol; in these cases, ‘+’ was used for concatenating strings.
The statistics for the programmatic analysis of all 240 generated exercises are presented in
Table 3. Out of the 240 programming exercises, 203 had a sample solution (84.6%). From the 203
sample solutions, 182 (89.7%) could be executed (i.e. running the code did not produce any errors).
A total of 170 programming exercises had automated tests, while 165 programming exercises
had both a sample solution and automated tests. From these 165 programming exercises, 51 had
a sample solution that passed the automated tests. Out of the 51 programming exercises with
a working sample solution and automated tests, 48 exhibited 100% statement coverage. The
statement coverage averaged over all of the 51 programming exercises was 98.0%. We observed
that the most common issue preventing exercises from passing the tests was not related to the
code logic, but in how the outputs were handled. In those cases, the sample solution printed
a value, while the automated tests expected that the sample solution would return a value
(e.g. the tests called a function and expected that the function would return a value, but the
function printed a value). We note, of course, that a confusion between printing and returning
values is a commonly cited error made by novices [44, 45]. Another common issue was that the
tests expected specific numbers that were not possible with the inputs (e.g. checking whether a
program correctly extracted and returned a list of even values from a list received as a parameter,
a test provided the list [
Overall, most generated exercises satisfied most of the criteria. We noted that in multiple cases, only minor tweaks would have been necessary to transform failing tests into passing ones. In the cases where tests were missing, we could simply add the generated exercise to the initial priming and the tests would likely be generated on a “second” run (we tested this behavior directly when exploring the output). The results of this preliminary evaluation provide some support for our hypothesis that these automatically generated exercises could serve as useful starting materials for learners within a learnersourcing activity. Learners could quickly evaluate the potential of a given exercise, and generate new ones if necessary. With reasonably little efort, and certainly with less efort than producing an exercise from scratch, such exercises could be edited or modified and then contributed to a shared learning resource. 1Five of the exercises contained --Tests-- but not --Sample solution-- (needed for content extraction) 2The n out of N for test coverage is counted as the number of full coverage (100%) cases out of the number of all test suites that did not fail (i.e. when coverage can be computed)
Below, we show a verbatim example of a well formed programming exercise generated by OpenAI Codex that received a positive score on our evaluation criteria. It is generated using the “speeding” priming example exercise (in Appendix 6.1), theme “music”, and concept set of “class”, “list” and “conditional”.
These preliminary results demonstrate that robosourcing is indeed a feasible approach for the scalable generation of practice questions as part of a learnersourcing activity. The computational power of large language models (LLMs) can be utilised to generate artefacts that are of suficient quality that they can be reviewed quickly, thus shifting the focus of learners from content creation towards content evaluation. Robosourcing may help to address the issues relating to low learner motivation and low quality content which are known to negatively afect learnersourcing activities in practice.
We observed that approximately one third of the generated programming exercises would be directly usable for teaching, and certainly suitable as a starting point for learners to evaluate and modify. The robosourced exercises were predominantly novel as they could not be found via Google search, and the themes and domain-specific concepts can be easily adjusted to influence the output. At least from this initial exploration, LLMs such as OpenAI Codex appear to be useful tools for robosourcing a diverse and large pool of exercises. The possibility for learners themselves to influence and customize the content through appropriate choice of keywords may also increase motivation. Exploring student perceptions of the use of such models as part of a robosourcing activity would be a fascinating avenue for future work.
Although these results are promising, there is still need for caution. Prior work on LLMs has found that they can contain biases, for example, towards minorities [46, 47]. Indeed, we did observe that the exercises generated during our evaluation did seem to more commonly involve men in problem descriptions. However, we did not observe any ofensive content. A similar concern is that LLMs might leak personally identifiable information present in the training data [48], although we did not observe any such data in our evaluation. In addition, recent work in the programming education literature has found that LLMs can be used to solve introductory programming exercises [42]. Thus, familiarizing students with these technologies could increase their use for plagiarism – and it is potentially even more likely that exercises created by LLMs can be solved by LLMs compared to exercises created by teachers.
Overall, while our results suggest robosourcing is feasible already, the performance of the underlying generative models is almost certain to improve over time, and will thus likely also lead to improved performance when these models are used for robosourcing. For example, newer LLMs such as Gopher [49] by DeepMind outperform GPT-3 (which Codex is based on). Similarly, we used Codex without any fine-tuning: future work should explore fine-tuning LLMs specifically for robosourcing, which is likely to lead to further improvements in performance. Lastly, the LLMs used in robosourcing could themselves be improved by feedback from humanin-the-loop evaluations, increasing their performance for robosourcing tasks. [2] R. J. Crutcher, A. F. Healy, Cognitive operations and the generation efect., Journal of
Experimental Psychology: Learning, Memory, and Cognition 15 (1989) 669. [3] P. A. DeWinstanley, E. L. Bjork, Processing strategies and the generation efect: Implications for making a better reader, Memory & cognition 32 (2004) 945–955. [4] K. VanLehn, R. M. Jones, M. T. Chi, A model of the self-explanation efect, The journal of the learning sciences 2 (1992) 1–59. [5] M. T. Chi, M. Bassok, M. W. Lewis, P. Reimann, R. Glaser, Self-explanations: How students study and use examples in learning to solve problems, Cognitive science 13 (1989) 145–182. [6] P. J. Guo, J. M. Markel, X. Zhang, Learnersourcing at scale to overcome expert blind spots for introductory programming: A three-year deployment study on the python tutor website, in: Proceedings of the Seventh ACM Conference on Learning @ Scale, L@S ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 301–304. URL: https://doi.org/10.1145/3386527.3406733. doi:10.1145/3386527.3406733. [7] N. T. Hefernan, K. S. Ostrow, K. Kelly, D. Selent, E. G. Van Inwegen, X. Xiong, J. J. Williams, The future of adaptive learning: does the crowd hold the key?, International Journal of Artificial Intelligence in Education 26 (2016) 615–644. [8] E. L. Glassman, A. Lin, C. J. Cai, R. C. Miller, Learnersourcing personalized hints, in: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work; Social Computing, CSCW ’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 1626–1636. URL: https://doi.org/10.1145/2818048.2820011. doi:10.1145/2818048. 2820011. [9] A. Singh, C. Brooks, Y. Lin, W. Li, What’s in it for the learners? evidence from a randomized ifeld experiment on learnersourcing questions in a mooc, in: Proceedings of the Eighth ACM Conference on Learning @ Scale, L@S ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 221–233. URL: https://doi.org/10.1145/3430895.3460142. doi:10. 1145/3430895.3460142. [10] N. Pirttinen, J. Leinonen, Can students review their peers? comparison of peer and instructor reviews, in: Proceedings of the 27th ACM Conference on Innovation and Technology in Computer Science Education Vol 1, 2022, p. 12–18. doi:10.1145/3502718. 3524762. [11] P. Denny, F. McDonald, R. Empson, P. Kelly, A. Petersen, Empirical support for a causal relationship between gamification and learning outcomes, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–13. doi:10.1145/3173574.3173885. [12] X. Yang, X. Guo, S. Yu, Student-generated content in college teaching: content quality, behavioural pattern and learning performance, Journal of Computer Assisted Learning 32 (2016) 1–15. URL: https://onlinelibrary.wiley.com/ doi/abs/10.1111/jcal.12111. doi:https://doi.org/10.1111/jcal.12111. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/jcal.12111. [13] S. Abdi, H. Khosravi, S. Sadiq, G. Demartini, Evaluating the quality of learning resources: A learnersourcing approach, IEEE Transactions on Learning Technologies 14 (2021) 81–92. doi:10.1109/TLT.2021.3058644. [14] H. Purchase, J. Hamer, P. Denny, A. Luxton-Reilly, The quality of a peerwise mcq repository, in: Proceedings of the Twelfth Australasian Conference on Computing Education - Volume [42] J. Finnie-Ansley, P. Denny, B. A. Becker, A. Luxton-Reilly, J. Prather, The robots are coming: Exploring the implications of openai codex on introductory programming, in: Australasian Computing Education Conference, 2022, pp. 10–19. [43] A. Venables, G. Tan, R. Lister, A closer look at tracing, explaining and code writing skills in the novice programmer, in: Proceedings of the Fifth International Workshop on Computing Education Research Workshop, ICER ’09, Association for Computing Machinery, New York, NY, USA, 2009, p. 117–128. URL: https://doi.org/10.1145/1584322.1584336. doi:10. 1145/1584322.1584336. [44] A. Ettles, A. Luxton-Reilly, P. Denny, Common logic errors made by novice programmers, in: Proceedings of the 20th Australasian Computing Education Conference, ACE ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 83–89. URL: https: //doi.org/10.1145/3160489.3160493. doi:10.1145/3160489.3160493. [45] C. Izu, P. Dinh, Can novice programmers write c functions?, in: 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), 2018, pp. 965–970. doi:10.1109/TALE.2018.8615375. [46] I. Solaiman, C. Dennison, Process for adapting language models to society (palms) with values-targeted datasets, Advances in Neural Information Processing Systems 34 (2021). [47] A. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, D. Klein, Detoxifying language models risks marginalizing minority voices, arXiv preprint arXiv:2104.06390 (2021). [48] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al., Extracting training data from large language models, in: 30th USENIX Security Symposium (USENIX Security 21), 2021, pp. 2633–2650. [49] J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hofmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al., Scaling language models: Methods, analysis & insights from training gopher, arXiv preprint arXiv:2112.11446 (2021).
The following two snippets were used as the priming exercises for robosourcing programming exercises. The keywords encompass the themes and concepts, which are followed by the problem statement and the sample solution. The place marked with ‘(themes and concepts are entered here)’ is filled by the robosourcing system, while the used large language model generates content starting at ‘(generation starts here)’. """Exercise 1 --Keywords-currency class function parameters dictionary arithmetics --Problem statement-Write a class called Converter that is initialized with a dictionary of exchange rates for currencies against the USD, e.g. {"USD": 1, ˓→ "EUR": 0.9, "GBP": 0.75}.
The class should have a method called convert, which takes in three parameters: from_currency, to_currency, and amount.
The function should return the given amount converted from the first currency ˓→ (first parameter) to the second currency (second parameter) using the exchange rate dictionary ˓→ given in the class constructor. def convert(self, from_currency, to_currency, amount): amount_in_usd = amount / self.exchange_rates[from_currency] return amount_in_usd * self.exchange_rates[to_currency] --Tests-class TestConverter(unittest.TestCase): def test_converter(self): converter = Converter({"USD": 1, "EUR": 0.8}) self.assertEquals(converter.convert("USD", "EUR", 100), 80) def test_converter2(self): converter = Converter({"USD": 1, "EUR": 0.9, "GBP": 0.75, "SEK": 9.71}) self.assertEquals(converter.convert("USD", "USD", 100), 100) self.assertEquals(converter.convert("USD", "EUR", 100), 90) self.assertEquals(converter.convert("GBP", "EUR", 10), 12) self.assertEquals(converter.convert("EUR", "GBP", 10), 8.333333333333332) """Exercise 2 --Keywords-(themes and concepts are entered here) --Problem statement-(generation starts here)