=Paper=
{{Paper
|id=Vol-2384/paper12
|storemode=property
|title=Tobbits Calculation Workbook: An Offline-to-Online Intelligent Textbook
|pdfUrl=https://ceur-ws.org/Vol-2384/paper12.pdf
|volume=Vol-2384
|authors=Junchen Feng,Ke Li,Ming Li
|dblpUrl=https://dblp.org/rec/conf/aied/FengLL19
}}
==Tobbits Calculation Workbook: An Offline-to-Online Intelligent Textbook==
https://ceur-ws.org/Vol-2384/paper12.pdf
Tobbits Calculation Workbook: An Offline-to-Online
Intelligent Textbook
Junchen Feng1, Ke Li2, Ming Li3
1,2,3
17zuoye.com
junchen.feng@17zuoye.com
Abstract. This paper describes an offline paper workbook that offers online in-
telligent tutor service through OCR with a smart phone. Tobbits calculation
workbook is an example of such Offline-to-Online(O2O) intelligent workbook.
Learner practices mental calculation on the paper workbook, grades them by tak-
ing a photo of the booklet through smartphone app, and gains access to the diag-
nosis of their procedural misconceptions. Among the recipients of half a million
of paper workbooks, more than 30% of them used the online service.
Keywords: Intelligent Textbook, Offline-to-Online, Optical Character Recog-
nition, Procedural Misconception
1 Introduction
Mental Calculation is a key component in Chinese math education [1, 3] during the first
three years of the primary school. Students are expected to perform simple calculations
without pen and paper within the time limit. There are specialized workbooks with basic
arithmetic questions, designed to cultivate students’ mental math capability. Despite
the wide usage of these workbooks among Chinese families, it is a tedious job for par-
ents to check their children’s responses, as each page contains about 50 calculation
items. In addition, parents usually do not have the pedagogical know-how, or the pa-
tience, to diagnose their children's procedural misconceptions. There are popular video
clips recording a frustrating parent yelling at their wailing kid for not getting "3*4"
right, whose comment section is filled with parents suffering from the same fate.
Technology can save these parents from themselves. Optical Character Recognition
(OCR) powered by deep learning can check students’ response with 99% accuracy. A
learning analytics can reconstruct learner's procedural misconception to help the tutor
(usually the parent) better understand the problem. An interactive course can step in as
the pedagogical intervention. These three components can be integrated in a single
smartphone app. Tobbits Calculation Workbook introduced in this paper is such a
blended intelligent textbook. The workbook is a paper product, but the user can access
intelligent tutoring services through both mobile apps and WeChat mini programs.
Within the literature context of the intelligent textbook, this paper is unorthodox in
two ways: Its subject is a workbook, rather than a textbook. The workbook is paperback
rather than digital. The first difference can be bridged by equating the workbook to the
2
homework, or practice, portion of the textbook [5]. In our business scenario, the second
difference is a feature not a bug. Although adaptive hyperlink system [2, 6] or more
broadly digital course management [8] has made great strides and hold great promises
in the age of ubiquitous smart devices, pen and paper is still the dominant mode of
learning now and probably for the next decades. Especially in K-12 education, parents
have serious misgiving about leaving their kids alone a smartphone or a pad while
teachers maybe uneasy about handing their students each with a pad in the classroom.
Though the paper textbook is a natural reference point of efficacy [6] if e-book is
viewed as its substitutes, it may be productive to think paper-book and intelligent tutor
system as compliments. The blended approach introduced in this paper, human tutor,
whether parents or teachers, have strong control on the exposure to smart devices.
Another practical advantage of the blended approach is affordability. As students
need to use their digital textbooks or workbooks on a frequent basis, owning or having
easy access to smart devices is a requirement. While under the blended approach, stu-
dents can work on physical textbooks or workbooks, and borrow their parents’ devices
for grading and getting feedback. Given limited levels of household income in devel-
oping countries, greater affordability means that the blended approach is feasible to
scale up. For educational companies, it makes economic sense to distribute the paper
book at a reduced price, or even for free, to bring users onto the digital platform and
achieve economic return via premium services. Such business model is called Offline-
to-Online (O2O), which is proven to be effective in China but may be alien to readers
outside of China. Several leading educational technology companies have distributed
such intelligent workbooks, and collectively reached millions of users.
2 The Tobbits Calculation Workbook
2.1 The offline Workbook
The first version of Tobbits calculation workbook has four volumes, targeting elemen-
tary school students from grade one to four in spring semester. Pedagogical experts
select and sequence the exercise items to match what students will learn at school. Each
workbook consists of 60 pages, 30 items per page, covering around 150 knowledge
components. Students are expected to finish one page of exercises per school day
Fig. 1. Example Pages from Tobbits Calculation Workbook of Grade One
2.2 The Online Service
The online tutoring services for Tobbits calculation workbook can be accessed via a
WeChat mini-program. A WeChat mini-program supports features similar to a mobile
3
app, but it resides in the WeChat ecosystem instead of existing as an independent app
on the smartphone. The main function of the mini-program is to take a photo of the
paper workbook to initiate the automatic correction service. Users may accumulate
points by registering, using the automatic correction service, or recommending this
mini-program to their friends. Users can use these points to redeem tutoring courses
offered in the course center.
User Registration
Grade by Photo
Course Center
User Referral
Fig. 2. A Screenshot of the WeChat Mini-program Interface
After users take a picture of the filled-out worksheet, the photo will be sent to a
backend server for grading and diagnosis. Within a second, the user interface will dis-
play a graded worksheet (see Fig 2). the user may click red question marks to check
detected mistakes, along with diagnosis of procedure misconception (see Fig 3). Users
can choose to submit feedback in case the automatic correction service fails. A team of
product managers and algorithm experts review user feedback regularly to keep im-
proving the service.
Detected Mistakes
Right Response
4
Fig. 3. Segment of a Graded Worksheet
User Feedback
Diagnosis
Fig. 4. Example of Diagnosis Section
3 Technologies that Power the Workbook
3.1 The Automatic Grading
The auto-grading algorithm consists of two parts, an OCR algorithm to recognize stu-
dents’ handwriting and an algorithm to grade students’ response.
Off-the-shelf open-source OCR algorithms that recognize handwritten numbers are
typically trained by adults’ handwriting data. Therefore, they have low accuracy in rec-
ognizing children’s different handwriting styles. The new OCR model includes formula
detection and recognition. The detection model marks formulas from other irrelevant
texts and symbols with a rectangular box. Scale and aspect ratio variety is the most
difficult part to handle. A carefully designed data augment proves to be helpful for this
task. With 10000 samples as training data, the accuracy of the detection model can
reach 99.6% on normal images. The formula recognition is relatively complicated for
variety of students’ handwriting.
To develop a new OCR algorithm for primary school students, it requires new data
sets. Due to the lack of online data collection mechanism in the early stage, the training
of recognition model is divided into two steps: offline training and online iteration. At
the first step, we collected 10,000 of students’ workbook pictures for annotation, which
are as close as possible to online images. When the model converges, the accuracy of
the offline data we collected can reach more than 98%. When the model is online, the
accuracy of the model is not unexpectedly reduced to a lower level, about 90%, for
online images. It shows that students who decide to use online service generates a sub-
stantial different sample from those we collected offline. Despite the initial setback, As
online users use this feature and upload pictures, larger, more diverse data are collected
to iterate the OCR and grading algorithms. Within a month, the accuracy reaches 99%.
Similarly, the grading algorithm requires AI experts coding specific grading rules
for different types of equations as the cold start. Table 1 shows some selected identifi-
able question types. Within 2 months of the launch, the grading accuracy of common
question types increased to 99% accuracy.
5
Table 1. Selected Identifiable Question Types
Question Types Example
Basic arithmetic (Addition / Subtraction / multiplication / division) 13-4=;2*3 =
compare 13-4 __ 15-7
Unit calculation 1km = ( ) m
Percentage-to-number conversion 35% =
Complete 13- __ =9
Solve for variable 15+3x = 24, x= ( )
3.2 Procedural Misconceptions and Adaptive Tutor Service
Among all errors identified by the OCR algorithm, about 32% of the errors diagnosed
by pedagogical experts with a specific procedural misconception. The analytics team
groups these misconception diagnoses into 31 major categories, mostly covering single
addition, subtraction, multiplication and division of integer number, decimal number
and fraction. For example, multiplication of integer has 4 major categories of proce-
dural misconceptions. For each category, the pedagogical expert gives a description of
the wrong procedure, which may not be self-obvious. If a learner answers 23*15=138,
she is diagnosed with “vertical form misalignment” (115+23=138). The description of
the wrong procedural reads “The multiplication result of the digit of tens is mistakenly
aligned with that of the ones.”
Table 2. Major Procedural Misconceptions of Multiplication of Integers
Category Example
Insufficient Mastery of the Multiplication Table 4*4=36
Forget to Carry 23*15=335
Vertical Form Misalignment 23*15=138
Do not Understand Vertical Form 55*9=95;23*15=215
Mistreatment of 0 as the Last Digit 25*20=50
The analytics team turned the work pedagogical expert into a rule based expert sys-
tem, which is dubbed as the calculation bot. Following the previous example, if the text
contains the print of “23*15”, the calculation bot is invoked by the app and lists all
possible wrong responses, along with the diagnosis and pedagogical suggestion. If the
recognized user answer matches any of the listed answers, the app shows the corre-
sponding diagnosis on the screen, as in Fig 4. When a wrong response is associated
with multiple possible misconceptions, which is likely in the case of multiple steps of
calculation, the calculation bot randomly picks one of them. If the recognized answer
does not match any of the listed answers, the app leaves the tutor section blank.
As stated previously, the overall diagnosis coverage rate is 32%, but the coverage is
uneven among knowledge components. The bot is better at covering single step calcu-
lation of integers or decimals, with a coverage rate around 60%. The bot is relative
weak on fractions and multi-step calculations, with a coverage rate around 20%.
6
To gauge whether such light-weight tutor service is helpful to parents, we designed
a voting system of “helpful” or “not helpful” for users to voluntarily express their opin-
ion. To date, about 6000 user ratings are collected, with an overall “useful” rating of
52%. Table 3 shows the votes of major misconceptions associated with integer multi-
plication. Per manual checking, diagnosis that are voted as “not useful” has an accuracy
rate of 70%. Therefore, the label “not useful” may have ambiguous meanings for users:
the diagnosis is not useful either because it is wrong or because I know it already alt-
hough it is right.
Table 3. Feedback Votes of Procedural Misconceptions of Multiplication of Integers
Category Votes Useful Percentage
Insufficient Mastery of the Multiplication Table 197 51%
Forget to Carry 72 57%
Vertical Form Misalignment 6 50%
Do not Understand Vertical Form 28 57%
Mistreatment of 0 as the Last Digit 211 51%
4 Discussion
Since Jan of 2019, more than 500,000 of Tobbits calculation workbooks are distributed
to students and parents. Over 30% of the recipients became active users of the WeChat
mini-program. Over 20% of these active users recommended the WeChat mini-program
to their friends. About 10% of the users checked the diagnosis details. Although the
current version of the Tobbits workbook is not a run-away success, it proves that the
Offline-to-Online strategy does acquire users for the online intelligent tutor service
quickly and at scale.
Moreover, the technologies described in this paper do not limit to grading only the
proprietary workbook. In fact, it can be used to grade any calculation exercises as long
as the equation type is supported, which makes the O2O strategy extremely scalable. In
addition, on the native IOS or android app, the intelligent tutor service offers more than
just description of the procedural misconceptions but also interactive teaching courses,
adaptive practice items and reports of learner mastery. Within three months after the
intelligent tutoring services were added to the mobile app, more than three million users
have uploaded photos of calculation exercises for automated grading. Majority of these
photos are non-proprietary calculation workbooks distributed by traditional publishers.
5 References
1. Stigler, James W. "“Mental abacus”: The effect of abacus training on Chinese children's
mental calculation." Cognitive Psychology 16, no. 2, 145-176, (1984).
7
2. Brusilovsky, Peter, Elmar Schwarz, and Gerhard Weber. "ELM-ART: An intelligent tutor-
ing system on World Wide Web." In International conference on intelligent tutoring sys-
tems, pp. 261-269. Springer, Berlin, Heidelberg (1996).
3. Zhang, Houcan, and Yuren Zhou. "The teaching of mathematics in Chinese elementary
schools." International Journal of Psychology 38, no. 5, 286-298, (2003).
4. Xu, Tiansheng, and Jiong Zhang. "A Development Strategy of O2O Business in China."
In 2015 International Conference on Computer Science and Intelligent Communication. At-
lantis Press (2015).
5. Johnson, Benny G., Fred Phillips, and Linda G. Chase. "An intelligent tutoring system for
the accounting cycle: Enhancing textbook homework with artificial intelligence." Journal of
Accounting Education 27.1 (2009): 30-39.
6. Chaudhri, Vinay K., Adam Overholtzer, and Aaron Spaulding. "An intelligent textbook that
answers questions." International Conference on Knowledge Engineering and Knowledge
Management. Springer, Cham, 2014.
7. Brusilovsky, Peter, and Christoph Peylo. "Adaptive and intelligent web-based educational
systems." International Journal of Artificial Intelligence in Education (IJAIED) 13 (2003):
159-172.
8. Romero, Cristóbal, Sebastián Ventura, and Enrique García. "Data mining in course manage-
ment systems: Moodle case study and tutorial." Computers & Education 51.1 (2008): 368-
384.