In this paper a new technique is presented for automatic design of optimal questionnaires. The technique, that is based on the Item Response Theory, performs multiple-choice item selection by a Genetic Algorithm. The experimental results demonstrate the validity of the proposed approach to adjust the characteristics of the questionnaire to the abilities of the student class.
In this paper a new approach for optimal questionnaire design is proposed, based on the Item Response Theory (IRT). A questionnaire is considered as an entity that must be tailored according to the specific characteristics of the group of students to be assessed. The proposed approach uses a two-steps strategy. In the first step the system estimates item difficulty for a given student class with specific abilities. In the second step a Genetic Algorithm (GA) is used to determine the best set of items to be included in the questionnaire. The organization of the paper is the following. Section 2 presents the problem of item evaluation by IRT. The problem of optimal questionnaire design is formally described in Section 3. Section 4 presents the genetic algorithm used for automatic questionnaire design. Section 5 presents the experimental results. Section 6 reports the conclusion.
IRT states that responses to a set of items can be explained by the existence of one or more latent traits, named abilities [Van der Linen1997; Fraley2000]. A main objective in item response modelling is to characterize the relation between a latent trait, , and the probability of item endorsement. This relation is typically referred to as the Item Characteristic Curve (ICC) and can be defined as the (nonlinear) regression line that represents the probability of endorsing an item (or an item response category) as a function of the underlying trait [Fraley2000]. For the purpose of this work, the Two-Parameter Logistic Model (2PLM) [Birnbaum1968] is considered. In this case, given the set of items T={t1, t2,…, tj…, tM}, the probability that an individual with trait level i will endorse item tj is defined as a function [Birnbaum1968]:
Pj ( i )
1 1 e j (i j ) (1) where j and j are the item discrimination parameter and the item difficulty parameter, respectively. The difficulty parameter j represents the level of the latent trait necessary for an individual to have a 50% probability of endorsing the item; the item discrimination parameter j represents an item’s ability to differentiate between people with contiguous trait levels. Of course, items are not equally informative across the entire range of the trait . In fact, an item yields the most information when i equals j.In the IRT, an item is considered difficult if a high level of ability or knowledge is required to answer it correctly. Therefore, the difference Pj(max)-Pj(min) can be used to estimate the extent to which item tj is effective to assess students in the range [min, max]: the greater the difference Pj(max)-Pj(min) the better the item tj. In this paper the problem of optimal MIQ design is considered as an optimization process in which - from the set of M items T - the subset of N items (N<M) more suitable for investigating the latent abilities of the set of students belonging to the skill range [θmin, θmax] is selected.
Formally, let T={t1, t2,…, tj…, tM} be the set of M items available, and S={s1, s2,…, si…, sN} the set of N students under consideration, being θi the trait ability level of the i-th student, i=1,2,…,N. The problem of optimal MIQ design concerns the selection from T of the subset Q={tip | p=1,2,…,P with (1ipM and ipiq for pq)}, which maximize the fitness function:
F (Q) PQ ( max ) PQ ( min ) where: θmax=max{θii=1,..,N} and θmin=min{θii=1,..,N}. (2)
A binary-coded genetic algorithm was considered is used to solve the optimization problem in eq. (2), since genetic algorithms have potential for solving non-linear optimization problems, in which the analytical expression of the object function is not known [Michalewicz1996; Goldberg1989]. The genetic approach is based on the following phases [Baeck1996]. The initial – population Pop=Npopof random individuals was created. In our tests Npop has been set to 20. since some preliminary experiments have shown Npop=20 is a good trade-off between convergence speed of the genetic algorithm and its capability to escape from local extrema. In our approach, each individual (that is a MIQ) is represented by a vector kh h hj h, where each gene hj was a Boolean value: hj=0 means that j-th item of T (i.e. the item tj) was not included in MIQ; hj=1 means that j-th item of T (i.e. the item tj) was included in Q. Of course, since P items must be included into the questionnaire Q, the following normalization procedure was performed for each individual k. In particular, let be P’=h1+h2+…+hM, if P’>P then select randomly (P’-P) genes equal to 1 and set them to 0; if P’ < P then select randomly (P-P’) genes equal to 0 and set them to 1. Successively, the fitness function was computed for each individual k of the population, according to eq. (2).
From the initial - population, the following four genetic operations were used to generate the new populations of individuals: i) Individual Selection. In the selection procedure Npop/2 random pairs of individuals were selected for crossover, according to a roulette-wheel strategy. This associates a selection probability to each individual. The higher the fitness function of the individual, the higher the selection probability [Baeck1996]. ii) Crossover. In our approach, a one-point crossover was used [Baeck1996]. In this case, for each pair of individuals selected for crossover, a random integer was chosen and the child individuals are defined according to the following rule: ○ has=has and hbs=hbs , if s < ; ○ has=hbs and hbs=has , if s ≥ . iii) Mutation. In this approach a uniform mutation operator is considered. Let h h h be an individual, the uniform mutation operator changed (inverted) each gene of the individual according to a mutation probability, Mut_prob (Mut_prob=0.02 in our tests). After mutation, the normalization procedure was also applied to all individuals k, k=1,2,…,Pop, in order to ensure that each questionnaire has a number of items equal to P, iv) Elitist Strategy. From the Npop individuals generated by the above-described operations, one individual was randomly removed and the individual with the maximum fitness in the previous population was added to the current population [Baeck1996].
Operations (i),(ii),(iii),(iv) were then repeated until Niter successive populations of individuals were generated (Niter=50 in our tests). When the process stopped, the optimal questionnaire was obtained by the best individual of the last-generated population.
In order to evaluate the new technique for optimal questionnaire design, a well-defined simulated dataset was considered. First a set of MT*N random responses simulating the answers of N of students to a set of MT items was generated automatically.
The experiment included two steps: (1) the ability estimation step; (2) the optimal test design step. 1) In the ability estimation step the student models (i.e. the trait ability level of each student) were estimated. After data simulation, the ICC of each item was evaluated using the 2PLM model and the trait ability level of each student was computed. For the purpose, the Marginal maximum likelihood estimation was considered, where the hidden student variables are chosen to maximize the likelihood of the data, according to the approach proposed in the literature [Bock and Aitkin 1981]. Finally, the skill range of the set of students [θmin, θmax] was determined. 2) In the optimal test design step the optimal MIQ was designed for the specific set of students under consideration. In the test step, a new set of M items named Full Set (FSM) was generated and the optimal questionnaire T*P was defined by automatically picking out the optimal subset of P items from FSM, for the given set of simulated students with a range equal to [θmin, θmax].
Table I shows the experimental results obtained with the simulation procedure. In this case, we considered N=20 students and MT=100 items. Successively, the ability of each student was estimated according to the approach of Bock and Aitkin [Bock and Aitkin 1981] and the skill range [θmin, θmax]=[2.20, 3.31] of the student set was determined. The test step was carried out using the questionnaire FSM of M items (M=50 in our test) and other MIQ obtained by selecting the optimal subset T*P of P items out of M (P=10,15,20 in our test). In order to evaluate the effectiveness of the proposed approach, the ability estimated when using the optimal questionnaire T*P was compared with the average ability determined when using the random-generated MIQs of P items, where item selection was performed randomly. In particular, each value TrndP is the average ability calculated when taking into account 10 MIQs, each one realized by selecting P random items from FSM. In order to estimate the effectiveness of the MIQs for student assessment we considered the following measures: • A_FSQ(i) the ability of the i-th student estimated through the Full Set questionnaire FSM of M items; • A_T*P(i) the ability of the i-th student estimated through the optimal questionnaire T*P of P items; • A_TrndP(i) the ability of the i-th student estimated by averaging the abilities determined through 10 random-generated P items questionnaires.
Hence the accuracy of T*P(i) and TrndP(i) to assess student ability was estimated, respectively, by the standard deviations:
SD(FSQ _ T * p )
N A _ FSQ(i) A _ T * p (i)2 i1 and
N SD(FSQ _T rnd p ) A _ FSQ(i) A _ T rnd p (i)2
i1 Of course, the comparison between SD(FSQ_T*P) and SD(FSQ_TrndP) reported in Table I provides a useful information about the capability of the proposed approach in selecting optimal subsets of items for questionnaire design, able to assess students more precisely than using randomly selected items.
This paper presents a new technique for optimal questionnaire design based on the IRT. The aim of this work is twofold. First, the problem of optimal questionnaire design is considered as an optimization problem. Second, a genetic algorithm is proposed for optimal questionnaire design and its effectiveness is demonstrated. The algorithm automatically selected the best set of items for the specific range of ability of the students under consideration.
The experimental results confirm the effectiveness of the new approach in adapting questionnaire to the abilities of a given set of students.