=Paper=
{{Paper
|id=None
|storemode=property
|title=Automata and Petri Net Models for Visualizing and Analyzing Complex Questionnaires: A Case Study
|pdfUrl=https://ceur-ws.org/Vol-827/25_HeikoRolke_article.pdf
|volume=Vol-827
|dblpUrl=https://dblp.org/rec/conf/acsd/Rolke10
}}
==Automata and Petri Net Models for Visualizing and Analyzing Complex Questionnaires: A Case Study==
https://ceur-ws.org/Vol-827/25_HeikoRolke_article.pdf
Automata and Petri Net Models
for Visualizing and Analyzing Complex
Questionnaires
– A Case Study –
Heiko Rölke
Deutsches Institut für Internationale Pädagogische Forschung
(German Institute for International Educational Research)
Solmsstraße 75, 60486 Frankfurt, Germany
roelke@dipf.de
Abstract. Questionnaires for complex studies can grow to considerable
sizes. Several hundred questions are not uncommon. In addition, routings
are used to distinguish between question paths for different respondents.
This leads to the question of how to ensure validity and other important
properties.
We examine this question for a case with even more demanding side
conditions: An important part of the OECD study ”PIAAC” (Programme
for the International Assessment of Adult Competencies) is a background
questionnaire (BQ) containing more than 400 questions. This BQ has to
be adapted by all participating countries. Nevertheless, integrity of the
overall system has to be ensured.
Keywords: automata, Petri nets, questionnaires, analysis
1 Motivation and Overview
Large scale studies in psychology, sociology, and for many other purposes try to
find out characteristics of complete populations or at least of big parts of a pop-
ulation. International studies often aim at comparing the complete population
of one country to that of another country. The well-known PISA study of the
OECD, as an example, aims at comparing all students of age 15 worldwide. This
is done by examining representative samples in each country that participates
in PISA.
To be able to compare one first has to find out some background information
about the people that are compared. This is mainly done by asking those people
questions. Larger chunks of questions grouped together in order to find about
the background of the surveyed people are called background questionnaires, or
BQ in short.
Recent Advances in Petri Nets and Concurrency, S. Donatelli, J. Kleijn, R.J. Machado, J.M. Fernandes
(eds.), CEUR Workshop Proceedings, ISSN 1613-0073, Jan/2012, pp. 317–329.
318 Petri Nets & Concurrency Rölke
The author of this paper was involved in the definition, implementation,
national adaptation, and deployment of the BQ for the OECD PIAAC study.1
The OECD is the ”Organization for Economic Co-Operation and Development”,
see [5] for details. Among many other activities, the OECD is well-known for
organizing world-wide comparability studies, like the PISA study. PISA [6] is the
abbreviation of ”Programme for International Student Assessment”. The PIAAC
study, ”Programme for the International Assessment of Adult Competencies” can
be seen as an extension of the PISA study for adults. It aims at finding out about
skills needed by adults in order to be successful in everyday work life. See [7]
for details about the PIAAC study. PIAAC is carried out by 24 countries all
over the world (participating countries are located in North and South America,
Europe, Asia and Oceania).
1.1 Background Questionnaire Properties
There is no exact definition of what a BQ is. It is therefore not possible to exactly
determine properties that have to be valid for each and every BQ. Naively, it is
just a bunch of questions that an interviewer has to present to an interviewee. In
practice, in discussions with psychologists, sociologists, or other questionnaire
practitioners, certain universally agreed principles and best practices become
clear. From this starting point, desirable properties can be derived. Neverthe-
less, it is not possible in the moment to definitely define and answer all related
questions. We strive for more general validity, though.
A BQ usually has one single entry or starting point, the first item or ques-
tion. In practice, this is often a hidden item, where predefined data is imported.
An example: One often knows the name of the interviewee in advance. Within
the BQ, there may be many different paths through the question pool, often
depending on previously entered data or chosen randomly. An item that is in-
tended to be the last question of a BQ is called an end item. Again, this may be
a visible item (a question) or a hidden item not visible to the interviewer. While
there often only is one end item, for example thanking the interviewee for time
and patience or, more technically, exporting the assembled data, this is not a
standard requirement of a BQ.
Due to practical considerations, there often is the possibility to pause an
interview or to break it off. While pausing has no implications for the structure,
a break-off means that any item can be an end item or has a connection to an
end item.
The normal flow through a BQ should not result in a dead end. A dead end
is an item that was not considered to be an end item. Other requirements are
more on the semantic side. Each possible question sequence has to make sense
semantically. On the other hand, each desired or planned sequence has to be
possible, e.g. by entering appropriate answers.
1
The work was done in the international consortium responsible for implementing,
deploying, and analyzing the study, led by ETS [3] in Princeton, USA. Most of
the implementation work on the BQ was done by the CRP Henry Tudor [2] in
Luxembourg.
Visualizing and analyzing questionnaires Petri Nets & Concurrency – 319
1.2 Overview
The rest of the paper is structured as follows: In Section 2 we give an overview
on the BQ of the PIAAC study. We also give examples of the format in that
the BQ is defined. Following up on that we develop first simple models for the
PIAAC BQ in Section 3. The models are put into practice in Section 4. They
are used to gain quite some insight and find errors, but are not sufficiently
powerful to represent all important aspects of our application. So we carry on in
Section 5 with more powerful Petri net models that allow for more sophisticated
analysis. We conclude in Section 6 with an outlook on further work and possible
generalizations.
2 The PIAAC Background Questionnaire
The PIAAC study mainly consists of two important parts: a background ques-
tionnaire (BQ) and cognitive tests (cognitive items, CI). Both parts are embed-
ded into an overall workflow that controls all parts of the survey. This workflow
is implemented in the same way as the BQ.
The PIAAC BQ starts with general questions about the interviewee to find
out whether he is suited to take part in the survey or not. Afterwards questions
in different categories are asked, grouped together in blocks. Examples for such
blocks are questions about the educational background, skills needed in everyday
work, and questions about private life related to work skills. In order to shorten
the overall interview time, parts of the blocks are arranged in a rotated design
so that not all interviewees are asked the same questions. Another example of
inter-block routing is that certain blocks are not administered if the requirements
for asking these questions are not fulfilled, e.g. questions about current work in
case of an unemployed interviewee.
In addition to the inter-block routing, complex routing is used within blocks
to administer the right questions. A good example for such a routing are ques-
tions about the education of an interviewee: If an interviewee has never been to
an university it is useless to ask questions about academic degrees. Respective
questions should be skipped. Another typical situation includes loops: One might
be interested in the degree of skills related to foreign languages. To accommodate
speakers fluent in multiple languages, some kind of cycle or loop is needed.
The code example in Figure 1 shows an example of a questionnaire item with
a free text entry. The XML syntax is not important here.2 The item group that is
defined in the code snipped defines a single item, i.e. one question is administered.
The item has a unique identifier (ID), instruction text and answering possibilities.
In this case a free text entry of length 12.
The second code example in Figure 2 shows a routing with two possible tar-
gets. This is a hidden item, i.e. an item that is not displayed but used internally.
2
The XML syntax of the PIAAC BQ has been specially designed for this purpose.
At the time of writing of this paper only limited support like editors or visualizers
is available.
320 Petri Nets & Concurrency Rölke
-
Please enter the sampled person ID
Sampled Person ID:[FTE]
Fig. 1. BQ code example: free text entry