=Paper=
{{Paper
|id=Vol-1746/paper-16
|storemode=property
|title=Student Assessment by Optimal Questionnaire Design
|pdfUrl=https://ceur-ws.org/Vol-1746/paper-16.pdf
|volume=Vol-1746
|authors=Melisa Aruci,Giuseppina Lotito,Giuseppe Pirlo
|dblpUrl=https://dblp.org/rec/conf/rtacsit/AruciLP16
}}
==Student Assessment by Optimal Questionnaire Design==
<pdf width="1500px">https://ceur-ws.org/Vol-1746/paper-16.pdf</pdf>
<pre>
                Student Assessment by Optimal Questionnaire Design

       Melisa Aruci                        Giuseppina Lotito                       Giuseppe Pirlo
Departmenti i Informatikës             “Gorjux-Tridente-Vivante”            Dipartimento di Informatica
 Fakulteti i Shkencave të                   70125 Bari, Italy                Univ. di Bari, 70125 Bari,
 Natyrës, Tiranë, Albania              {giuseppina.lotito@istruzio                     Italy
{melisa.aruci@gmail.com}                         ne.it}                     {giuseppe.pirlo@uniba.it}


                       Abstract                              The organization of the paper is the following. Section 2
                                                             presents the problem of item evaluation by IRT. The
    In this paper a new technique is presented for           problem of optimal questionnaire design is formally
    automatic design of optimal questionnaires.              described in Section 3. Section 4 presents the genetic
    The technique, that is based on the Item                 algorithm used for automatic questionnaire design.
    Response Theory, performs multiple-choice                Section 5 presents the experimental results. Section 6
    item selection by a Genetic Algorithm. The               reports the conclusion.
    experimental results demonstrate the validity
    of the proposed approach to adjust the                   2. Item Evaluation by IRT
    characteristics of the questionnaire to the
    abilities of the student class.                          IRT states that responses to a set of items can be
                                                             explained by the existence of one or more latent traits,
1. Introduction                                              named abilities [Van der Linen1997; Fraley2000]. A
Computer-based student assessment is now considered          main objective in item response modelling is to
a fundamental service of Learning Management                 characterize the relation between a latent trait, , and
Systems [Amelung2011; Dimauro2003; Romero2008;               the probability of item endorsement. This relation is
Greco2006b]. Although several types of computer-             typically referred to as the Item Characteristic Curve
based systems for student’s assessment have been             (ICC) and can be defined as the (nonlinear) regression
proposed so far Multiple-choice Item on-line                 line that represents the probability of endorsing an item
Questionnaires (MIQs) is the most diffuse approach           (or an item response category) as a function of the
[Lan2011; Romero2010] since they can be easily               underlying trait [Fraley2000]. For the purpose of this
integrated into computer-based assessment systems            work, the Two-Parameter Logistic Model (2PLM)
[Kuechler2003; Romero2009]. When a MIQ is                    [Birnbaum1968] is considered. In this case, given the
considered, students are asked to select the best            set of items T={t1, t2,…, tj…, tM}, the probability that
possible answer from the choices provided on a list          an individual with trait level i will endorse item tj is
[Kuechler2003]. Data from MIQs can be used for               defined as a function [Birnbaum1968]:
providing      personalized       learning   suggestions
[Chu2006], for the analysis of individual targets                                                 1
[Yamanishi2001], for discovering the individual needs                      Pj ( i )            j ( i   j )
                                                                                                                     (1)
of the students [Pechenizkiy2008], for discovering rule                                  1 e
patterns [Chen2009]. Unfortunately, the design of a          where j and j are the item discrimination parameter
questionnaire is a complex task that requires the            and the item difficulty parameter, respectively. The
selection of the set of items most advantageous for          difficulty parameter j represents the level of the latent
assessing the skill level of a student [Lan2011].            trait necessary for an individual to have a 50%
In this paper a new approach for optimal questionnaire       probability of endorsing the item; the item
design is proposed, based on the Item Response Theory        discrimination parameter j represents an item’s ability
(IRT). A questionnaire is considered as an entity that       to differentiate between people with contiguous trait
must be tailored according to the specific                   levels. Of course, items are not equally informative
characteristics of the group of students to be assessed.     across the entire range of the trait . In fact, an item
The proposed approach uses a two-steps strategy. In          yields the most information when i equals j.In the
the first step the system estimates item difficulty for a    IRT, an item is considered difficult if a high level of
given student class with specific abilities. In the second   ability or knowledge is required to answer it correctly.
step a Genetic Algorithm (GA) is used to determine the       Therefore, the difference Pj(max)-Pj(min) can be used
best set of items to be included in the questionnaire.
to estimate the extent to which item tj is effective to        [Michalewicz1996; Goldberg1989]. The genetic
assess students in the range [min, max]: the greater the     approach is based on the following phases
difference Pj(max)-Pj(min) the better the item tj.           [Baeck1996].        The        initial   –     population
                                                               Pop=Npopof random individuals was
                                                               created. In our tests Npop has been set to 20. since some
                                                               preliminary experiments have shown Npop=20 is a good
                                                               trade-off between convergence speed of the genetic
                                                               algorithm and its capability to escape from local
                                                               extrema. In our approach, each individual (that is a
                                                               MIQ) is represented by a vector kh h hj
                                                               h, where each gene hj was a Boolean value: hj=0
                                                               means that j-th item of T (i.e. the item tj) was not
                                                               included in MIQ; hj=1 means that j-th item of T (i.e.
   Figure 1. Item Effectiveness Estimation by ICCs             the item tj) was included in Q. Of course, since P items
                                                               must be included into the questionnaire Q, the
Figure 1 shows the ICCs of two items t1 and t2. In this        following normalization procedure was performed for
case, the results indicated that t1 is better than t2 for      each individual k. In particular, let be
assessing the students in the range [min, max], since        P’=h1+h2+…+hM, if P’>P then select randomly (P’-P)
P1(max)-P1(min) > P2(max)-P2(min).                         genes equal to 1 and set them to 0; if P’ < P then select
                                                               randomly (P-P’) genes equal to 0 and set them to 1.
3. A Theoretical Approach to Optimal                           Successively, the fitness function was computed for
MIQ Design by GA                                               each individual k of the population, according to eq.
In this paper the problem of optimal MIQ design is             (2).
considered as an optimization process in which - from          From the initial - population, the following four genetic
the set of M items T - the subset of N items (N<M)             operations were used to generate the new populations of
more suitable for investigating the latent abilities of the    individuals:
set of students belonging to the skill range [θmin, θmax] is
selected.                                                      i) Individual Selection. In the selection procedure
Formally, let T={t1, t2,…, tj…, tM} be the set of M            Npop/2 random pairs of individuals were selected for
items available, and S={s1, s2,…, si…, sN} the set of N        crossover, according to a roulette-wheel strategy. This
students under consideration, being θi the trait ability       associates a selection probability to each individual. The
level of the i-th student, i=1,2,…,N. The problem of           higher the fitness function of the individual, the higher
optimal MIQ design concerns the selection from T of            the selection probability [Baeck1996].
the subset Q={tip | p=1,2,…,P with (1ipM and ipiq
                                                               ii) Crossover. In our approach, a one-point crossover
for pq)}, which maximize the fitness function:                was used [Baeck1996]. In this case, for each pair of
                                                               individuals selected for crossover, a random integer
 F (Q)  P Q ( max )  P Q ( min )                           was chosen and the child individuals are
                                                         (2)   defined according to the following rule: 
where:                                                         ○        has=has and hbs=hbs , if s < ;
θmax=max{θii=1,..,N} and θmin=min{θii=1,..,N}.               ○        has=hbs and hbs=has , if s ≥ .

                                                               iii) Mutation. In this approach a uniform mutation
4. Optimal MIQ Design by GA                                    operator is considered. Let h h h be an
                                                               individual, the uniform mutation operator changed
A binary-coded genetic algorithm was considered is             (inverted) each gene of the individual according to a
used to solve the optimization problem in eq. (2), since       mutation probability, Mut_prob (Mut_prob=0.02 in our
genetic algorithms have potential for solving non-linear       tests). After mutation, the normalization procedure was
optimization problems, in which the analytical                 also applied to all individuals k, k=1,2,…,Pop, in order
expression of the object function is not known
to ensure that each questionnaire has a number of items          questionnaire FSM of M items (M=50 in our test) and
equal to P,                                                      other MIQ obtained by selecting the optimal subset T*P
                                                                 of P items out of M (P=10,15,20 in our test). In order to
iv) Elitist Strategy. From the Npop individuals generated        evaluate the effectiveness of the proposed approach, the
by the above-described operations, one individual was            ability estimated when using the optimal questionnaire
randomly removed and the individual with the                     T*P was compared with the average ability determined
maximum fitness in the previous population was added             when using the random-generated MIQs of P items,
to the current population [Baeck1996].                           where item selection was performed randomly. In
Operations (i),(ii),(iii),(iv) were then repeated until N iter   particular, each value TrndP is the average ability
successive populations of individuals were generated             calculated when taking into account 10 MIQs, each one
(Niter=50 in our tests). When the process stopped, the           realized by selecting P random items from FSM. In
optimal questionnaire was obtained by the best                   order to estimate the effectiveness of the MIQs for
individual of the last-generated population.                     student assessment we considered the following
                                                                 measures:
5. Experimental Results                                          • A_FSQ(i) the ability of the i-th student estimated
                                                                     through the Full Set questionnaire FSM of M items;
In order to evaluate the new technique for optimal               • A_T*P(i) the ability of the i-th student estimated
questionnaire design, a well-defined simulated dataset               through the optimal questionnaire T*P of P items;
was considered. First a set of MT*N random responses             • A_TrndP(i) the ability of the i-th student estimated by
simulating the answers of N of students to a set of MT               averaging the abilities determined through 10
items was generated automatically.                                   random-generated P items questionnaires.
The experiment included two steps: (1) the ability                  Hence the accuracy of T*P(i) and TrndP(i) to assess
estimation step; (2) the optimal test design step.               student ability was estimated, respectively, by the
                                                                 standard deviations:
1) In the ability estimation step the student models (i.e.
                                                                                                 A _ FSQ (i)  A _ T (i)
                                                                                                N
    the trait ability level of each student) were estimated.            SD( FSQ _ T * p )                                *
                                                                                                                               p
                                                                                                                                          2

    After data simulation, the ICC of each item was                                             i 1

    evaluated using the 2PLM model and the trait ability         and
    level of each student was computed. For the
                                                                                                 A _ FSQ (i)  A _ T                    
                                                                                                 N
                                                                                                                                          2
    purpose, the Marginal maximum likelihood                            SD( FSQ _ T rnd p )                             rnd
                                                                                                                               p   (i )
                                                                                                i 1
    estimation was considered, where the hidden student
    variables are chosen to maximize the likelihood of           Of course, the comparison between SD(FSQ_T*P) and
    the data, according to the approach proposed in the          SD(FSQ_TrndP) reported in Table I provides a useful
    literature [Bock and Aitkin 1981]. Finally, the skill        information about the capability of the proposed
    range of the set of students [θ min, θmax] was               approach in selecting optimal subsets of items for
    determined.                                                  questionnaire design, able to assess students more
2) In the optimal test design step the optimal MIQ was           precisely than using randomly selected items.
    designed for the specific set of students under
    consideration. In the test step, a new set of M items        Table I. Experimental Results
    named Full Set (FSM) was generated and the                                P SD(FSQ_T*P)                 SD(FSQ_TrndP)
    optimal questionnaire T*P was defined by                      Simulated    10             0.27                  0.62
                                                                    Data       15             0.21                  0.47
    automatically picking out the optimal subset of P
                                                                               20             0.13                  0.25
    items from FSM, for the given set of simulated
    students with a range equal to [θmin, θmax].
Table I shows the experimental results obtained with the         6. Conclusion
simulation procedure. In this case, we considered N=20           This paper presents a new technique for optimal
students and MT=100 items. Successively, the ability of          questionnaire design based on the IRT. The aim of this
each student was estimated according to the approach of          work is twofold. First, the problem of optimal
Bock and Aitkin [Bock and Aitkin 1981] and the skill             questionnaire design is considered as an optimization
range [θmin, θmax]=[2.20, 3.31] of the student set was           problem. Second, a genetic algorithm is proposed for
determined. The test step was carried out using the              optimal questionnaire design and its effectiveness is
demonstrated. The algorithm automatically selected the       [Greco2006b] Greco N., Impedovo D., Pirlo G.
best set of items for the specific range of ability of the      (2006b) “New Steps toward the Effective
students under consideration.                                   Evaluation     of    E-learning     Activities: A
The experimental results confirm the effectiveness of           Participant-Based Approach”, WSEAS Trans. Adv.
the new approach in adapting questionnaire to the               Eng. Educ, Issue 7, Vol. 3, pp. 662-666.
abilities of a given set of students.                        [Impedovo2006] Impedovo S., Lucchese M.G., Pirlo
                                                                G. (2006) “e-Examinations: an Advanced
References                                                      Methodology for Student’s tests on e-Learning
[Amelung2011] Amelung M., Krieger K., Rosner D.                 University   Courses”      ,     WSEAS        Trans.
   (2011) “E-Assessment as a Service”,    IEEE                  Adv. Eng. Educ., Issue 5, Vol. 3, pp. 361- 366.
   TLT, Vol. 4, No. 2, pp. 35-46.                            [Kuechler2003] Kuechler W. L., Simkin M. G. (2003)
[Baeck1996] Baeck T. (1996) Evolutionary Algorithms            ”How well do multiple choice tests evaluate
   in Theory and Practice: Evolution Strategies,               student understanding in computer programming
   Evolution Programming, Genetic Algorithms.                  classes?”,J.Inf.Syst.Educ.,Vol.14,No.4,pp.389–399.
   New York: Oxford Univ. Press.                             [Lan2011] Lan C. H., Graf S., Lai K. R., Kinshuk
[Birnbaum1968] Birnbaum A. (1968) “Some latent                  (2011) “Enrichment of Peer Assessment with Agent
trait models and their use in inferring an examinee's           Negotiation”, IEEE TLT, Vol. 4, No. 1, pp. 35-46.
     ability”, Statistical theories of mental test scores,   [Michalewicz1996] Michalewicz Z. (1996) Genetic
     F. M. Lord and M. R. Novick (eds.), Addison-               Algorithms + Data Structure=Evolution Programs,
     Wesley, pp. 397-472.                                       Springer Verlag, Berlin, Germany.
[Bock and Aitkin 1981] Bock,R. and Aitkin,M.                 [Pechenizkiy2008] Pechenizkiy M., Calders T.,
   (1981)    “Marginal    Maximum      Likelihood               Vasilyeva E., De Bra P. (2008) “Mining the student
   Estimation of Item Parameters: Applications of               assessment data: lessons drawn from a small scale
   an EM Algorithm”, Psychometrika, Vol. 46, pp.                case study”, International Conference on
   443-459.                                                     Educational Data Mining, Spain, pp. 187-191.
[Chen2009] Chen Y., Wenig C. (2009)”Mining fuzzy             [Romero2010] Romero C., Ventura S. (2010)
   association rules from questionnaire data”,                 “Educational Data Mining: A Review of the State-
   Knowledge-Based Systems Journal, Vol. 22,                   of-the-Art”, IEEE TSMC -Part C: Applications and
   No. 1, pp. 46-56.                                           Reviews, Vol. 40, No. 6, pp. 601–618.
[Chu2006] Chu H.C., Hwang G. J., Tseng J.C.R.,               [Romero2009] Romero C., Ventura S., De Bra P.
   Hwang G. H. (2006) “A computerized approach                  (2009)    “Using     Mobile     and Web-Based
   to diagnosing student learning problems in                   Computerized Tests to Evaluate University
   health education”, Asian Journal of Health and               Students”, Computer Applications in Engineering
   Information Sciences, Vol. No. 1, pp. 43-60.                 Education, Vol. 17, No. 4, 435-447.
[Dimauro2003] Dimauro G., Impedovo S., Pirlo G.              [Romero2008] Romero C., Ventura S., Salcines P. E.
   (2003) “Traditional learning toward on-line                  (2008) “Data mining in course management
   learning”, Proc.TEL’03, Italy, pp. 355-360.                  systems: Moodle case study and tutorial”,
[Fraley2000] Fraley R.C., Waller N. G., Brennan K. A.           Computers & Education, Vol.51, No.1,pp.368-384.
    (2000) “An Item Response Theory Analysis of              [Van der Linen1997] Van der Linen W.J., Hambleton
     Self-Report Measures of Adult Attachment”,                 R.K. (eds.) (1997) Handbook of modern item
    Journal of Personality and Social Psychology,               response theory, Springer, New York.
    Vol. 78, No. 2, pp. 350-365.                             [Yamanishi2001] Yamanishi K. and Li H. (2001)
[Goldberg1989] Goldberg D.E. (1989) Genetic                     “Mining from open answers in questionnaire data”,
   Algorithms in Search, Optimization and                       Proceedings of the Seventh ACM SIGKDD, pp.
   Machine Learning, Addison–Wesley, New York.                  443-449.

</pre>