<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Digital Literacy in Lower Secondary Education { A First Evaluation of the Situation in Austria</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Digital Literacy</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Digital Education</string-name>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Johannes Kepler University</institution>
          ,
          <addr-line>Linz</addr-line>
          ,
          <country country="AT">Austria</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>As long ago as 1988, Austria introduced the subject \Computer Science" in grade 9. Quite a long time there was solely this one year of mandatory IT-education during school career. In 2011, all European countries had digital education policies in place, either as standalone policies or as part of a national ICT strategy. The strategic weight of these policies remained on nurturing students' digital competences, justi ed by future economic bene ts. For Austria, these changes have not resulted in further computer science education in school. When Austria implemented the new mandatory subject \Digital Education" in September 2018 for all students in lower secondary education, computer science education nally found its way into additional grades. The curriculum covers digital competences, media competences, as well as civic education. Schools can decide if they o er stand-alone subjects or if they implement the curriculum in an integrative way in several other subjects. However, schools still ght the problem who is teaching and how, because most schools just cannot install an extra subject, due to lack of teachers or lack of teaching hours available. This paper reports on the implementation of the subject \Digital Education" in Upper Austrian schools and takes a closer look at rst experiences by evaluating a survey answered by 117 teachers approximately two years after the implementation of the new subject.</p>
      </abstract>
      <kwd-group>
        <kwd>STEM tional Thinking</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        Computational thinking tools and methods found their way into multiple
scienti c elds, becoming widespread among scientists long ago [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]. Computational
thinking covers solving problems, designing systems, and understanding human
behavior by drawing on the concepts of computer science [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]. Furthermore,
computational thinking can be understood as the connective tissue that combines
computer science with many other disciplines [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. However, there is still a lack
of those concepts in educational context in Austria, even if the implementation
of computational thinking in schools is nothing new. In the 1960s, the American
computer scientist and university professor Alan Perlis insisted on the
introduction of \theory of computation" for all college students in the USA [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. As long
ago as 1988, Austria introduced the subject \Computer Science" in grade 9.
The curriculum includes the topics \Computer Science, People, and Society",
\IT Systems", \Applied Computer Science", \Computer Science in Practice".
So, in this curriculum there is no digital literacy ar all.
      </p>
      <p>
        Concerning the International Computer and Information Literacy Study (ICILS)
30% of the tested German students are \functional digital analphabets",
henceforth they only show rudimentary skills in working with new technologies and
digital devices (as Austria did not participate in the study, Germany was chosen
because both countries start their computer science education by age 10) [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ].
Consequently, educators have the responsibility to make computational thinking
available to school students as well [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. Furthermore, research shows that it is
better to implement computational thinking in other subjects than to teach it as
a stand-alone subject because it tends to be separated from real-world problems
[
        <xref ref-type="bibr" rid="ref15">15</xref>
        ].
      </p>
      <p>This paper describes related work as well as the implementation of the
curriculum \Digital Education" in Austria. In section 4 the methodology and results
of a study, conducted in secondary schools in Upper Austria, are characterized.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Related Work</title>
      <p>
        Regarding a nationwide promotion of media literacy, both the EU and UNESCO
act as driving institutions. Consequently, individual nations try to integrate those
recommendations into their own educational reforms and curricula. European
Union nations share lots of common characteristics in their implementation of
digital literacy in educational context [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. This can also be observed in computer
science education world-wide. Australia, USA, and Great Britain introduced an
early education in computer science, whereas both Germany and Austria can be
found at the back of the eld (see gure 1) [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ].
      </p>
      <p>Regarding a study from 2010 from the university in Dresden, 12 out of 16
states of Germany integrated media literacy or basic concepts of computer
science in their curricula. But otherwise there is no nationwide directive for teaching
computer science or digital education.</p>
      <p>In Switzerland a project called \Lehrplan 21" has been developed to
introduce the topic \Media and Computer Science" throughout the school career. The
project focuses on \Understanding Media &amp; Responsible Usage", \Basic
Computer Science Concepts and problem Solving", as well as \Applied Computer
Science".</p>
      <p>
        Shortly after providing each and every student at age 11 and 12 with BBC
micro:bit, England introduced a mandatory subject \Information and
Communication Technology" in 2014. Educational and teaching objectives concentrate
on \Computer Science", \Digital Literacy", and \Information Technology".
Fig. 1. Comparison of the starting age of computer science education between countries
[
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] (adapted by the authors) { AT: Austria; AU: Australia; CH: Switzerland; GER:
Germany; GB: Great Britain; PL: Poland; SK: Slovakia; USA: United States of America
      </p>
      <p>Slovakia installed the subject \Informatika" for all students from grade two
to eleven by focusing on computational thinking. Poland integrated the topics
\Understanding and Analysis of Problems" and \Programming and Problem
Solving by Using Computers and other Digital Devices" in their curriculum.</p>
      <p>As part of Barack Obama's \Computer Science for All" initiative in 2016,
the USA enhanced computer science education from kindergarten until grade
12.</p>
      <p>There has not been deeper research in the Austrian curriculum of
\Digital Education" due to the fact that it was installed in 2018. Furthermore, the
COVID-19 pandemic took other topics to the center of the stage, even if the
home-schooling situation forced students and teachers to deal with digital
devices.</p>
      <p>
        Furthermore, there are no long time studies concerning digital literacy skills
of students in Austria but Lazonder et al. [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] report on a three-year study that
investigated the development of children's digital literacy skills in the
Netherlands. One hundred fty-one students have been tested three times in yearly
intervals to gather information on how they collect, create, transform, and safely
use digital information. The study suggested that there is a linear increase in all
skills, but nonetheless natural development of digital literacy skills is slow.
Furthermore, they found out that the investigated students gained the most in the
competence \collecting information on the Internet", whereas the expertise in
creating digital products from scratch improves the least. However, the growth
of most skills seemed unrelated to socio-demographic characteristics [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ].
      </p>
      <p>
        Concerning teacher training at university level, a study of the Austrian
Computer Society (OCG { O sterreichische Computer Gesellschaft) shows that the
suggested 8 to 12 ECTS of digital competences cannot be reached at Austrian
universities { they could merely be counted with 4.7 and are characterised by
the usage of software applications [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. Nonetheless, each teacher has to
implement the \Digital Literacy" curriculum in Austria, even if there are few available
teacher training courses or university support. Still, Bratengeyer et al. show that
e-learning can be found at each Austrian university, though intensity and o ers
vary. Especially smaller institutions lack the necessary infrastructure as well as
technical support [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ].
3
      </p>
    </sec>
    <sec id="sec-3">
      <title>Digital Education in Austria</title>
      <p>
        In the last two decades the idea of life-long learning gained more and more
importance. According to the EU key competences, \Digital Literacy" is one of those
topics students should master when they nish school. In 2006 \digi.komp", a
model for digital competences, was introduced in Austria. Digi.komp4 describes
the model until grade 4, digi.komp8 states examples from grade 4 until grade 8,
digi.komp12 suggests competences for grades 9 - 12, and digi.kompP
characterizes the model for teachers [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
      </p>
      <p>
        In 2018 the Austrian government published a \master-plan for digitalization"
where three main elds of action are described. Field one concerns curricula and
their development, whereby digital content has to be integrated. Furthermore,
infrastructure of schools has to be improved, as for example 24% of the
primary schools have no WiFi installed in their building. Topic three covers teacher
training and teacher education, concentrating on ideas for the implementation
of digital literacy [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ].
      </p>
      <p>
        This master-plan also presented an 8-point-concept to foster digital education
that is outlined as the following [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]:
1. \Portal Digital School": should be a single point of entry and should unify
all necessary pedagogical and administrative applications.
2. Standardization of learning platforms
3. Teacher training concerning distance- and blended learning
4. Expansion of the platform \eduthek": this learning platform provides
additional exercises and has been further developed since the COVID-19
pandemic.
5. Development of veri ed learning-apps
6. Upgrading IT infrastructure
7. Supplying students with digital devices
8. Supplying teachers with digital devices
      </p>
      <p>
        Concerning this 8-point-plan, the Austrian government invested 200 million
Euros until 2022, to guarantee a consistent development [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ].
      </p>
      <p>The new subject \Digital Education" was installed in September 2018 in
lower secondary education (grades 5-8), implemented by two to four weekly
lessons. If schools want to implement more than 4 hours per week, they have the
nance it from the school budget { ministry will not pay for it. School
administration can decide if they o er stand-alone subjects or if they implement the
curriculum in an integrative way in several other subjects. The Austrian
government recommends the following implementation when \Digital Education" is
taught as a stand-alone subject (see gure 2).</p>
      <p>
        Of course it is also possible to introduce other concepts. For example it is
possible to start with 1 hour per week stand-alone in grade 5, then switch to
1 hour per week integrated in other subjects in years six to eight. Of course,
schools can also decide to install \Digital Education" in other subjects in year
5 using 1 hour per week and grades six and seven each with 0.5 hours per week
stand-alone. Many other options are possible and common [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ].
      </p>
      <p>
        \Digital Education" covers digital competences, media competences, as well
as civic education (see gure 3). According to the curriculum, those three topics
should not be taught separately but must be connected to other subject-speci c
elds. The BGBL (Bundesgesetzblatt fur die Republik O sterreich | federal law
gazette of Austria) states that the main aim is to develop students who deal
with media and technology to be more responsible and well-briefed [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. The
subject-speci c topics are described as the following [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]:
1. Social aspects of digitalization: re ecting the usage of digital devices in
everyday life as well as bene ts and ethical boundaries
2. Information, data, &amp; media: queries, evaluating sources, sharing information
3. Operating systems &amp; standard software: basic knowledge of operating
systems, text processing, presentation software, calculations
4. Media design: adopting, producing, and adapting media
5. Communication &amp; social media: di erent communication platforms, creating
digital identities, cloud-sharing
6. Data security &amp; privacy: securing devices as well as private data
7. Technical problem solving: solving basic IT problems
8. Computational thinking: working with algorithms, creative usage of
programming languages
Fig. 3. Overview of the Curriculum of \Digital Education" combining di erent topics
[
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] (adapted by the authors)
4
4.1
      </p>
    </sec>
    <sec id="sec-4">
      <title>Study</title>
      <sec id="sec-4-1">
        <title>Methodology</title>
        <p>The study concentrated on the implementation of the new curriculum
\Digital Education" that was introduced in September 2018. The following research
question lay out the basis for the survey: How do schools implement the new
curriculum? Stand-alone or integrated in other subjects?</p>
        <p>The survey was sent to all 133 local public schools whereas 117 teachers were
willing to complete the questionnaire. Basic questions formed the rst part of the
survey concerning gender, age group, years in service, school type and subjects
taught. The next section focused on the implementation of \Digital Education"
at schools where we concentrated on the following questions:
{ How is \Digital Education" implemented at your school? (stand-alone
subject/integrative in regular lessons/both)
{ If \Digital Education" is implemented as a stand-alone subject: who teaches
this subject? (computer science teachers/interested teachers/other)
{ If \Digital Education" is implemented integrative in regular lessons: who
is implementing the curriculum? (everyone has to implement the
curriculum/only those who are interested implement the curriculum/other)
{ If \Digital Education" is implemented integrative in regular lessons: How is
it checked whether material from the curriculum has been taught? (comment
in class register/with a list in the class/not at all/other)
{ Is there any revision (test, quiz, . . . ) of the students knowledge of the topics
of \Digital Education"? (yes/no/other)
{ If so, in which form? (open question)</p>
        <p>The third section was dedicated to the personal experiences of the teachers
and wanted to nd out if teachers have already been confronted with the new
curriculum. Furthermore, an opportunity was provided to add personal opinion:
{ Have you already given a lesson on the \Digital Education" curriculum?
(yes/no/other)
{ If yes: which content was covered? (open question)
{ If so, did you have any di culties implementing the content? (yes/no/other)
{ Which sources do you use to create \Digital Education" lessons?
(digicomp.at/saferinternet.at/easy4me.info/eeducation.at/book \Digitale
Grundbildung fur digi.komp8"/other)
{ How useful do you consider the subject \Digital Education" on a scale from
1 (very useful) to 5 (not useful at all)?
{ Further comments on the introduction of \Digital Education" (open
question)</p>
        <p>
          The question using a scale-rating was implemented applying a ve-point
Likert scale with the options \very useful, rather useful, partly useful, rather not
useful, not useful at all" [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ]. In this paper we concentrate on evaluating the
quantitative data of the survey, whereas the qualitative data are analyzed in
following papers.
4.2
        </p>
      </sec>
      <sec id="sec-4-2">
        <title>Results</title>
        <p>In total there were 117 answered questionnaires while 73 de ned themselves as
female, 43 as male, and 1 as diverse. Thirteen people were under 30 years old,
16 between 30 and 39 years, 12 between 40 and 49 years, 58 between 50 and
59, and 18 over 60 years old. Re ecting their years in school service 19 teachers
stated under 5 years, 10 teachers 5-10 years, 11 teachers 11-20 years, 21 teachers
21-30 years, and 56 teachers 30 or more years in service.</p>
        <p>The subjects taught are displayed in gure 4. Fifty-four stated they teach
mathematics, 42 teach foreign languages (English, Italian, Spanish, French, Latin,
Greek, or Russian), 36 computer science, 24 biology, 22 German, 21 physics, 21
physical education (PE), 17 history, 16 geography, 15 arts, 10 chemistry, 10
music, 7 religious education, 5 home economics, and 3 teach career guidance.
Fourteen named other subjects like ethics, handicraft, natural sciences, accounting,
nutrition, CAD, or else. Of course, it was also possible to choose more than one
subject, as in Austria most of the teachers cover two elds of expertise. In total
117 teachers picked 317 subjects, so on average one teacher covers approximately
2.7 subjects.</p>
        <p>The next section of the questionnaire concentrated on the realization of the
curriculum. The rst question explored how \Digital Education" is implemented
at school with 4 di erent options, whereas it was only possible to pick one
exclusively. 30 teachers (26%) said that a stand-alone subject was introduced,
25 (21%) stated they integrate the curriculum in other subjects, 53 (45%) choose
\mixed", and 9 (8%) picked \other"(see gure 5).</p>
        <p>51% stated that if the curriculum is implemented as a stand-alone subject,
it is taught by computer science teachers, whereas 37% said it is taught by
interested ones, and 11% stated both. Nine teachers (8%) described di erent
approaches and therefore picked the category \other". One possible alternative
was stated as doing workshops or blocked courses, whereas one teacher stated
that he/she does not even know if \Digital Education" is taught at school.</p>
        <p>
          When integrated in other subjects, 59% speci ed that each teacher has to
implement the new curriculum, while 41% said that only interested teachers
do that. To verify whether the curriculum has been taught in an integrative
way, 40 people stated that they use comments in the class register, 18 said
that they work with checklists of the topics of the curriculum or the like in
the classroom, and 30 claimed that there is no veri cation at all. Three teachers
picked \other", while they added the answers \competence catalogue", \checklist
in teachers o ce", and \teacher conferences". When looking at assessment, 47
teachers (40%) reported that there is some kind of veri cation of the topics of the
curriculum of \Digital Education", and 70 (60%) claimed that there is no test
at all. 42% out of these 47 teachers who said that there is a veri cation, stated
that they use tests, quizzes, or the like. The governmental tool \digicheck" [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ]
was chosen by 15%, 23% reported they use the ECDL (\European Certi cate of
Digital Literacy')', 8% picked the option \mixed", and 12% named other forms
of assessment.
        </p>
        <p>The following part concentrated on personal experience of teachers with the
implementation of \Digital Education". Concerning the question if teachers have
already held a lesson implementing the curriculum, 66 people (56%) answered
\yes", 46 (39%) \no", and 5 (4%) picked \other". Only 11% claimed that they
had problems teaching the new content, whereas 80% experienced no
complications at all (8% chose \other"). Regarding educational resources, the
questionnaire listed the most popular ones known of the eld in Austria. 25% use the
website \digicomp.at", 24% \saferinternet.at", 19% \easy4me.info", 17%
\eeducation.at", 12% the book \Digitale Grundbildung", and 4% other sources.</p>
        <p>The last question wanted to nd out how teachers would assess the new
subject concerning usefulness and was implemented using a ve-point
LikertScale. Seventy-one teachers (61%) rated the subject \very useful", 31 (26%)
\rather useful", 11 (9%) \partly useful", 2 (2%) \rather not useful", and 2 (2%)
\not useful at all" (see gure 6). The rating is in the upper section with an
arithmetic mean of approximately 1.57.
4.3</p>
      </sec>
      <sec id="sec-4-3">
        <title>Discussion</title>
        <p>Remarkably, the age group of 50-59 years appears most often in this survey
(49%). This mirrors the current situation in Austria, as the majority of teachers
are close to retirement (64% of the participants are 50 years or older). Henceforth,
approximately half of the teachers stated that they have 30 or more years in
service.</p>
        <p>An interesting fact is that most often computer science teachers are deployed
if \Digital Education" is taught as a stand-alone subject. That could be due to
the fact that the new curriculum has a close connection to the computer science
curriculum in Austria. Still, it would also be possible that teachers with other
subjects get involved.</p>
        <p>If taught in an integrative way in other subjects, 59% claimed that each
teacher has to implement the new curriculum, while 41% said that only interested
teachers do so. This is a very interesting nding because legally every teacher
has to implement the new curriculum. Still, approximately half of the teachers
do not know or at least do not consider that fact.</p>
        <p>Another interesting discovery is shown in the answers of the question \Did
you have any di culties implementing the content?". 80% of the respondents
claimed that they had no problems at all. This is the exact opposite of the
tenor in teacher society because most teachers state that they do not know how
to implement the curriculum and need further materials. However, when the
survey was conducted only 56% already held a lesson in \Digital Education".
Maybe those were the ones that had no di culties from the start and the rest
still deals with lack of teaching materials and sparse knowledge in the eld.
5</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>Conclusion and Outlook</title>
      <p>This paper focused on the implementation of the subject \Digital Education"
that was introduced in 2018 in Austria. A study was conducted to investigate how
schools cope with the launch of the new curriculum. Summarizing, the research
question could be answered. It was investigated how schools implement the new
curriculum { stand-alone or integrated in other subjects. 26% claimed that they
introduced a stand-alone subject, 21% said they integrate the curriculum in
other subjects, 45% picked \mixed", and 9 (8%) chose \other".</p>
      <p>In summary, further development-steps are:
{ analyzing the qualitative data of the study
{ expanding the study nationwide to get broader insight
{ development of an online teacher training course to support and encourage
teachers to install \Digital Education" in their daily lessons at school by
concentrating on integrative implementation with computational thinking
{ evaluating the implemented teacher training course</p>
      <p>Henceforth, there is still a lot of e ort that has to be put into the
implementation of the curriculum to gain further motivation of the teachers. Especially,
due to the COVID-19 pandemic, it would be interesting to nd out if the rating
of usefulness of the subject \Digital Education" increased.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1. BGBLA. Bundesgesetzblatt der Republik Osterreich:
          <fpage>71</fpage>
          .
          <string-name>
            <surname>Verordnung</surname>
          </string-name>
          ,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Brandhofer</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Baumgartner</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ebner</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , Koberer, N.,
          <string-name>
            <surname>Tru</surname>
            ltzschWijnen,
            <given-names>C.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Wiesner</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <article-title>Bildung im Zeitalter der Digitalisierung</article-title>
          .
          <source>Bildungsbericht</source>
          (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Bratengeyer</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Steinbacher</surname>
            ,
            <given-names>H.-P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Friesenbichler</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , Neubock,
          <string-name>
            <given-names>K.</given-names>
            ,
            <surname>Kopp</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            , Groblinger, O., and
            <surname>Ebner</surname>
          </string-name>
          ,
          <string-name>
            <surname>M.</surname>
          </string-name>
          <article-title>Die osterreichische Hochschul-ELearning-Landschaft</article-title>
          . Book On Demand,
          <year>April 2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Bundesministerium</surname>
            ,
            <given-names>B. Verbindliche</given-names>
          </string-name>
          <article-title>Ubung digitale grundbildung { umsetzung am schulstandort</article-title>
          ,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5. Bundesministerium Digitalisierung und Wirtschaftsstandort.
          <source>Digitales Kompetenzmodell fur Osterreich DigComp 2</source>
          .2 AT. Tech. rep.,
          <source>Bundesministerium Digitalisierung und Wirtschaftsstandort</source>
          ,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6. Bundesministerium fur Bildung, W. u. F.
          <source>Masterplan fur die Digitalisierung im Bildungswesen</source>
          ,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7. Bundesministerium fur Bildung, W. u. F. digicheck: Online Fragebogen zu den digitalen Kompetenzen,
          <year>2020</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Grandl</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Ebner</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Informatische</surname>
          </string-name>
          Grundbildung - ein
          <source>Landervergleich. Medienimpulse</source>
          <volume>55</volume>
          (
          <year>June 2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Guzdial</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <article-title>Education: Paving the way for computational thinking</article-title>
          ,
          <year>Aug 2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Hollweger</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Stumvoll</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Narosy</surname>
          </string-name>
          , T. Verbindliche Ubung Digitale Grundbildung, Praxishandbuch. Haider
          <string-name>
            <surname>Lehrmittelverlag</surname>
            <given-names>OG</given-names>
          </string-name>
          ,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Joshi</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kale</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chandel</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Pal</surname>
            ,
            <given-names>D. K.</given-names>
          </string-name>
          <article-title>Likert scale: Explored and explained</article-title>
          .
          <source>British Journal of Applied Science &amp; Technology 7</source>
          ,
          <issue>4</issue>
          (
          <year>2015</year>
          ),
          <fpage>396</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Lazonder</surname>
            ,
            <given-names>A. W.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Walraven</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gijlers</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Janssen</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          <article-title>Longitudinal assessment of digital literacy in children: Findings from a large dutch single-school study</article-title>
          .
          <source>Computers &amp; Education</source>
          <volume>143</volume>
          (
          <year>2020</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Lee</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Grover</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Martin</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pillai</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Malyn-Smith</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <article-title>Computational Thinking from a Disciplinary Perspective: Integrating Computational Thinking in</article-title>
          K-12
          <string-name>
            <surname>Science</surname>
          </string-name>
          , Technology, Engineering, and Mathematics Education.
          <source>Journal of Science Education and Technology</source>
          (
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Orton</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Weintrop</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Beheshti</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Horn</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Jona</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Wilensky</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          <article-title>Bringing Computational Thinking Into High School Mathematics and Science Classrooms</article-title>
          .
          <source>ICLS Proceedings</source>
          (
          <year>2016</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Weintrop</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Beheshti</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Horn</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Orton</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Jona</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Trouille</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Wilensky</surname>
          </string-name>
          , U. De ning Computational
          <source>Thinking for Mathematics and Science Classrooms. Journal of Science Education and Technology 25, 1 (Feb</source>
          <year>2016</year>
          ),
          <volume>127</volume>
          {
          <fpage>147</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Wing</surname>
            ,
            <given-names>J. M. Computational</given-names>
          </string-name>
          <string-name>
            <surname>Thinking</surname>
          </string-name>
          .
          <source>Communications of the ACM</source>
          (
          <year>2006</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Wolf</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Koppel</surname>
          </string-name>
          , I. Digitale Grundbildung:
          <article-title>Ziel oder Methode einer chancengleichen Teilhabe in einer mediatisierten Gesellschaft? Tech. rep</article-title>
          ., erwachsenenbildung.at,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>