In the paper, a way to improve the teaching quality by applying a resource-based approach is discussed. The solution includes a model of teaching quality, a method of student-centered teaching, and means for analyzing and managing the unobservable quality characteristics. We propose to consider the education process as a system with negative feedback, where the deviation of the values of the educational process characteristics from the normal ones diagnoses the type of the problem that interferes with the normal state of the educational process. The formalization of the unobservable quality characteristics of the educational process is carried out. The paper considers a means for measuring the values of these characteristics in different scales and provides examples of measuring. We also provide a formalized description of the teaching process and a set of questions for a student to evaluate teaching. An algorithm for the creation of a questionnaire, which covers the characteristics of various components of the teaching process, has been developed. Finally, we describe the task of informational diagnostics for the results of a student's evaluation and the catalog of teaching tactics as a means for systematic formation of quality-improving activities.
Teaching quality means different things for different people. For example, students expect high-quality content provided in a clear, friendly, and creative manner. Professors contribute to achieving the objectives and learning outcomes. Experts of methodological departments look for appropriate study materials, etc. In the paper, we restrict the set of stakeholders by the principal ones – students and professors.
For many years, students’ evaluation of teaching (SET) has been the most common
process employed at universities to evaluate faculty-teaching quality. However, many
problems – such as mental biases, participation rates, validity, reliability, and several
other related issues remain actual. For example, in [
In the paper, we do not focus our attention on the problems of SET. We focus
attention on the activities that enable us to discover the ways for improving teaching
performance with reliable results, which becomes possible with the use of particular
resources. As a resource, we understand any unit of practical knowledge required by a
teacher or academic manager in order to perform a particular task of quality
improvement. Resources include everything required for solving a challenge: models, methods,
and means [
Let us describe the resource-based approach for searching the ways for improving teaching performance. The remaining part of the paper is organized as follows. Section 2 describes the model of teaching quality with an emphasis on quality measurement problem. Section 3 presents the method of teaching quality improvement based on measuring different quality characteristics. Section 4 describes the formal presentation of the questionnaire structure to obtain full coverage of the teaching quality characteristics and involves the issue of cataloging the teaching tactics. 2
The idea of using feedback from participants to control the teaching quality is not new. Different kinds of learning processes involve various approaches, tools, and techniques that should interact harmoniously and effectively. The level of harmony determines the degree of participants’ satisfaction in the learning process. Anyway, the quality of feedback is determined by the quality of the survey. The construction of the survey depends on the model of the learning process. Since the educational process has a multi-criteria nature, it is quite difficult for description, which causes the complexity of building adequate models. Let us review some approaches for construction of such models.
The TALIS model assesses the teaching quality basing on the working condition of
teachers and the learning environment [
One of the well-known models – Assessing Quality Teaching Rubrics (AQTR) –
assesses the pre-service teachers’ quality teaching practices [
However, the assessment of the educational process by teachers and educational
institutions is one side of the coin. The other side is represented by students who can
express their degree of satisfaction with the learning process in which they participated.
The cumulative effects model (CEM) is developed to accomplish this. It assesses the
quality of the teacher’s preparation on the basis of students’ outcomes using survey
tools [
TEQAS model is a model in which quality is assessed via assessment of pedagogical
education [
To eliminate the shortcomings of the above models, the Teaching Maturity (TeaM)
Model was created [
Thus, some of the above models have demonstrated their applicability in practice, which means that finding ways to describe and model the educational process is an urgent and promising task. Correction of the shortcomings and limitations of the given models will allow the proposed resource-based approach to finding ways to increase the effectiveness of the educational process. 3
In [
The correlation between teaching quality and success level of students’ learning is beyond doubt. The better is the teaching excellence, the higher students’ grades should be. Therefore, the set of individual grades or integrated indexes calculated on the set should be considered as observed quality characteristics.
In addition, the effect of the professor’s behavior on students’ achievement is beyond doubt. The study process reflects the professor’s behavior, which means we should seek to explain changes in students’ progress by students’ experiences in the classroom or other study settings. The appropriate aspects of teaching quality are unobserved via learning. SET is traditionally used to evaluate such quality characteristics, more precisely, such aspects as: ─ teaching excellence and encouragement, which includes teaching skills, level of contact and involvement, the effectiveness of the course design, assessment and feedback in developing students’ knowledge and skills; ─ the learning environment, which includes the effectiveness of study resources in supporting students’ learning and the development of independent study skills; ─ students’ engagement and outcomes, which include self-evaluation of their own work and progress.
Quality characteristics should be unambiguous. To determine whether a response to a characteristic is satisfactory, it is necessary to provide the measurement of the characteristic. The purpose of measuring is to reduce subjectivity while monitoring activities and provide data for analysis.
To achieve the required level of quality characteristic, the professor can use the teaching tactics. A tactic is a decision of the teaching staff that influences the achievement of a response to the quality characteristic; tactics directly affect the students’ achievements. The focus of tactics is on a response to a single quality characteristic. However, tactics can refine other tactics.
Fig. 1 shows the relationship between the entities of the teaching quality model
graphically. Teaching quality characteristics reflect the factors that relate to teaching
quality requirements. As it was mentioned above, there are two types of characteristics:
observed and unobserved via learning. The quality characteristics provide the means
for measuring to determine whether the teaching quality is meeting the required quality
thresholds set by stakeholders. Teaching tactics describe how a given quality
characteristic can be achieved.
For the model, one of the core issues is the measure of unobserved characteristics. SET
instruments typically consist of a set of Likert scale survey questions [
However, the Likert scale in itself is categorical, and SET data cannot be evaluated
validly using parametric statistics. These categories differ in quality, not in quantity or
magnitude. In other words, the “interval distance” between the categories is undefined
[
The opponents of SET as measures of teaching effectiveness argue that SET has no
or only limited validity as a measure of professors’ teaching effectiveness [
Let us describe how to encapsulate SET into a student-centered educational process to improve the teaching quality.
The educational process aims to transform the learning goals into learning outcomes demonstrated by students. Learning outcomes are the nonempty set LO = {LO1, LO2, …, LOn}, which should be clearly and unambiguously formulated in the course syllabus. The teaching/learning process realizes the transformation.
The teaching process is based on three foundations: course content, delivery methods, and course materials. The learning process depends on students’ engagement and involvement. The concrete realization of the education process depends on its form (for example, full-time, part-time, e-learning). The process representation of education is shown in Fig. 2. As the elements of LO describe the knowledge or skills students should acquire, they focus on the context and potential applications of knowledge and skills. The assessments and evaluations are the models of LOi, which provide possibilities to define the level of students’ success. These assignments can take different forms, such as theoretical research, case study, solving practical problems, testing, preparing presentations. Due to the various types of assignments, they can be intended for individual or collective execution, have different time constraints, and allow or forbid the free choice of performing tools.
Traditionally, the course professor develops the assignments and their evaluation criteria according to the curricula. The evaluation scheme is specified in the syllabus. Assignment grading can be organized with varying degrees of automation. In the case of dual education, stakeholders can create and evaluate practical assignments as part of real-life, ongoing projects. Also, students can perform self-assessment, passing independent online testing on the course topics.
As it can be seen, despite а wide variety of assignment forms, they provide evaluation only for observed teaching quality characteristics (OTQC). However, as it was mentioned above, unobserved teaching quality characteristics (UTQC) have a significant impact on the educational process as well. Therefore, the set of teaching quality characteristics (TQC) should be considered as
TQC = OTQC ⋃ UTQC, (1) where each element of the set corresponds to one or more measurers.
The vital component of the educational process is feedback [
The educational process can be considered as a process with negative feedback. At the same time, deviations of the actual values of TQC elements (TQCfact) from the recommended ones (TQCrec) should lead to formation of quality improvement activities (QIA), which counteract the deviation to minimize it. In general, QIA can be represented as an integral indicator of the impact on the educational process in the framework of the chosen study program:
QIA = F (diff (TQCfact, TQCstd)) → min. (2) The influence of OTQC on the quality of the educational process is well understood: traditionally, assignments are evaluated on a quantitative scale using the existing recommendations. Therefore, we should study the set of UTQC elements, namely: ─ a set of unobserved teaching quality characteristics and their borderline values; ─ a set of possible measures, including scales and methods of assessment.
By this line of reasoning, the development and organization of quality improvement activities cause a need of solving then following essential tasks: ─ the selection of UTQC and the definition of their borderline values; ─ the creation of effective mechanisms for assessing actual UTQC and their deviations from the borderline; ─ the identification of the causes of deviations and analyzing the possibility of eliminating them within a particular educational process; ─ the formation of quality improvement activities, taking into account the type and degree of deviations for teaching quality improvement via implementing teaching tactics.
UTQC can be determined indirectly by SET, while students provide feedback on the course content, delivery methods, and course materials. Various techniques, such as a questionnaire (preferred), survey, focus group, could be used to obtain the values of UTQC characteristics. Usually the target properties of SET consist of ─ mindfulness: the awareness that course follows the syllabus, understanding of the course role in the curriculum, satisfaction with their own progress in the course, acceptance of the course structure, clear presentation of the course material that facilitates understanding, clearness of the connections between the topics in the course and with other courses of the curricula, understanding of the assessment methods; ─ adaptability: the correspondence of the course workload and requirements to the course level, the acceptability of presentation speed; ─ efficiency: following the schedule, the time loss caused by access to the course materials, the timeliness of the information about schedule changes, aiding students’ learning by teaching methods, creation a welcoming and inclusive learning environment, timeliness of recommended reading and instructions, providing confidence to do more advanced work in the subject, challenging and value of the course materials, willingness, and ability of the professor to answer questions clearly and thoroughly, returning assignments and exams on time, providing helpful feedback on time; ─ functionality: coherence of the requirements of the professor and teaching assistants, the impact of instructional materials on increasing students’ knowledge and skills in the subject, the availability of free and understandable reading in the library and electronic access, updating and accuracy of the online course platform; ─ interference: knowledge or skills on the course subject earned early, the regularity of class attendance, the effectiveness of organization and facilitation of learning activities by the professor; ─ activity: preparation for classes, dealing with efforts aimed at doing reading tasks and graded assignments.
When SET is over, it is advisable to analyze the characteristics of the evaluated foundations of the teaching process:
UTQC = {UTQCCC, UTQCDM, UTQCCM}, where UTQCCC, UTQCDM, and UTQCCM are, respectively, the sets of unobserved characteristics for the course content, delivery method, and course materials.
Formally, the unobserved characteristic utqc ϵ UTQC can be represented as: utqc = <Pr, TFnd, Measure>, where Pr is one of the measurable properties listed above, TFnd is the evaluated foundation of the teaching process (course content, delivery method, or course materials), measure defines the set of measures for Pr.
The informative scales for the properties listed above and some examples for them are shown in Table 1. The values of some properties can be measured by different scales. Thus, it is advisable to select the most appropriate scales for measuring the values of the property. The scale defines the method of value measuring and a corresponding method for statistical estimation.
When conducting SET, a list of questions should be compiled; each question has to correspond to one or more UTQC characteristics. The list of questions can be considered complete if it covers all valuable UTQC. The simplest way to achieve this is to solve the problem of seeking the shortest coverage on condition that the list of questions includes all valuable elements of UTQC. 5 5.1
Let us provide a formal description of the questionnaire as the most popular means of feedback in the framework of the educational process.
There are various criteria for classifying questionnaires; for example, the number of
respondents, location, and the delivery method [
Despite the simplicity and applicability of the questionnaire in all areas of life, there are still some issues concerning the survey organization. Some of them are listed below. 1. Authorization during the survey
The advantage of anonymous surveys is a free expression of opinions. The disadvantages include the possibility of multiple filling-in of the questionnaire by one person, the difficulty of maintaining anonymity in small groups, the challenge of controlling the completeness of the respondents’ group. The non-anonymous survey provides such an additional advantage as a correlation with the parameters of the student’s profile, in particular, academic performance, attendance, research work.
2. Time of the survey
The surveys are administered to students during or at the end of the course. The advantage of the first option is the ability to correct the teaching performance for the course presentation. The advantage of the second option is the lack of impact of the professor’s impression on the evaluation of a particular student.
3. The universality of the questionnaire
The advantage of using a standard questionnaire for all courses is the possibility to compare the results of the survey to create a rating of courses, programs, and professors. The advantage of creating a specific questionnaire for each course consists in taking into account the features of particular courses.
4. Assessment of a set of questions
It is necessary to consider whether the author of the questions is competent enough
from the professional expertise [
5. The visibility of the results
It is necessary to determine who can see and analyze the results of the survey: the course professor, the management staff, other teaching staff, students of the course, other stakeholders.
6. The complexity of the interpretation and decision-making for quality improvement activities in case of open-answer questions.
The problem of the questionnaire completeness remains an issue. As a possible solution, we propose a formal description of the correspondence between the teaching/learning process and the questionnaire.
The course teaching process is described as a set of foundations where CC is the course content, DM is the set of course delivery methods, and CM is the set of course materials.
The course content should be defined as a set; each element cciCC is defined as follows: where spi is the structural part (for example, lectures, visual presentation, command project), qhi is the credit hours.
The delivery methods should be defined as a set; each element dmiDM is defined as follows: where toi is the organizational component of the teaching process, and tpi is the description of the professor’s personality given by a set of characteristics tpiMTP ={Gender, Age, Experience,...}. The set MTP can vary depending on the purpose of the questionnaire.
The course materials are also represented as a set; each element cmiCM is defined as follows:
TF = {CC, DM, CM}, cci = <spi, qhi>, dmi = <toi, tpi>, (5) (6) (7) (9) (10) cmi = <spi, kcmi, vi, ami>, (8) where spi is the structural part which the material belongs to, kcmi describes the type of presentation (for example, a printed form, a video, a presentation), vi defines the volume of the course material, and ami defines the access method.
Each foundation of TF has its own set of characteristics to evaluate ССС NTQCCC, СDM NTQCDM, СCM NTQCCM, respectively.
Now the questionnaire for SET can be described as follows:
SETQ = <TF, QN, yqn>, where QN is the questionnaire used for the course TF in the yqn academic year.
QN is a set of questions, each of which is described as components:
qn = <qtxt, kqn, Cq’, ms>, where qtxt is the wording of the question text, kqn is the type of question, Cq’ is a set of characteristics affected by the question qn (Cq’СССUСDMUСCM), ms is a scale of measure.
The degree of completeness of the questionnaire QN is determined as the ratio: QN
QN ' q1 Cq CCC CDM CCM .
When QN=0, the questionnaire does not correspond to the course; when QN=1, the questionnaire covers all fundamental properties of teaching quality characteristics. The borderline value QN for a sufficient questionnaire could always be defined.
Let us describe the procedure of questionnaire development. 1. Form the set CCC as a collection of characteristics set for components of CC: CCC CT
iSP1 Cspi Cqhi , where CT is the set of characteristics for the course as a whole, SP is the set of structural parts of the course, Cspi is the set of characteristics for the ith structural part of the course, Cqhi is the set of characteristics for the credit hours of the ith structural part of the course.
2. Form the set CDM as a collection of characteristics set for components of DM: CDM Cto where T is the set of the course teachers (professor and teaching assistants), Cto is the set of characteristics for the organizational component of the teaching process as a whole, Ctpi is the set of characteristics for the ith course teacher.
3. Form the set CCM as a collection of characteristics set for components of CM: CCM
SP i1
CCMspi Ckcmi Cvi Cami . 4. Obtain the combined set C CCC CDM CCM . 5. Formulate the set of questions for сiС:
QNC qnC qtxtC , kqnC , ci , msC .
6. Define the number of questions nq characteristics with corresponding questions.
7. If CQ C , then calculate QN. ciCQ
QNci , where CQC is the set of 8. If QN QlimN , where QlimN is the borderline value, then return to step 5. 9. Find a sufficient set of questions. (11) (12) (13) (14) (15)
The proposed procedure depends on the issue of sufficiency of the question set. Let us suppose that the complexity of each question preparation is the same. The answering efforts for all possible questions of the questionnaire are also the same. Then the definition of the questions optimal set of questions can be considered as the unweighted problem of seeking the shortest coverage of the set of examined characteristics C with the smallest set of questions from the set QNC.
The set of characteristics of C can be considered as a reference set С = {C1, …,Cn}, n=|C|. There is a set QN consisting of m subsets QNi of the set C, where QNi.CqC, m QNi .Cq C . The shortest coverage QN*QN is defined as i1
QN* i1
QNi .Cq C, QNi QN*, QN* min . (16) This problem can be solved using the boundary-search algorithm on the concave set to obtain all unabundant coverages. As a result, the shortest coverage QN*containing the least number of questions is selected. 5.2
Because of the possibility to determine the borderline values for each quality
characteristic of the teaching process, processing of SET results to develop appropriate quality
improvement activities can be seen as the task of informational diagnostics. SET is a
rather complicated process; therefore, at the end of each iteration, it is necessary to fix
and determine the actual state based on the values of the characteristics [
Let us consider the process of the formation of diagnostic states (classes) when using various measures.
For the properties of the characteristics obtained by the measurers on the ordinal, interval, and dichotomous scales, the appropriate classification method exists. The number of classes is equal to the number of categories, the number of intervals, or two (true/false), respectively. When using a ratio scale, it is advisable to bring the values obtained on it to the interval scale. Diagnostics of SET results is performed using a hierarchical classification method, which is characterized by the sequential division of a set of objects (in our case, teaching process foundations) into smaller subsets (in our case, properties of teaching process foundations).
We consider four levels of classification: the first level represents teaching process foundations, the second level represents the properties of teaching process foundations, the third level represents measurement scales, the fourth level represents sets of possible values. Diagnostic states are the vertices of the fourth level.
During SET, each question uniquely identifies the vertex of the third level. Each answer to the question contributes to a fixed size (for example, a single one) to one of the vertices of the subordinated fourth level.
In the normal state of the teaching process, the vast majority of contributions should be concentrated at the vertices corresponding to the typical values of the measured properties of the studied characteristics. For vertices that do not correspond to the typical values, the threshold values can be determined, exceeding of which signals about the necessity of the additional analysis. Experts can evaluate these threshold values based on the possible number of such vertices: the more vertices, the lower the threshold. Cases of the threshold exceeding could be classified according to the conditions. ─ The threshold is exceeded for one vertex or adjacent vertices. The reason consists in the poor quality of the teaching process conditioned by the poor quality of the corresponding components of the teaching process. The required action is the formation of quality improvement activities within the potential of technological means of the resource-based approach. ─ The threshold is exceeded for several non-adjacent vertices (for example, for two located at opposite ends of the scale). The possible reasons are the violation of SET procedure or unaccounted teaching process characteristics that have a hidden effect.
The case requires additional research. The Catalog of Teaching Tactics Another means created in the framework of the resource-based approach is teaching tactics, which may be taken as fundamental or essential units of the professor’s behavior helpful in creating a suitable learning structure for the realization of the set teachinglearning objectives [19]. It is teaching methods, teaching techniques, teaching aid materials, and anything else helpful to them to realize their teaching objectives.
A teaching process usually bases on a collection of tactics. They have been used for years, so now they are well isolated, cataloged, and described. To support professors, we should accumulate the set of tactics formulated as “the diagnostic state – a tactic – a result” as the core of the recommendation system. Three reasons cause the following solution: 1. A professor can more easily assess the options for augmenting an existing process to achieve a target value of the quality characteristic by understanding the role of tactics. 2. Tactics give the professor insight into the properties of the resulting teaching activities. 3. By cataloging tactics, we provide a more systematic way of making a design of the teaching process within some limitations.
The tactics usually overlap, and the professor frequently has a choice among multiple tactics to improve a particular quality characteristic. The choice of which tactic to use depends on such factors as tradeoffs among other quality characteristics, and the implementation cost.
The work presents the research aimed at improving the teaching quality within the framework of the resource-based approach. The resource development reflects the model of multistage development, which means that each previous stage is the base for all the following ones. In our case, the development of the teaching quality model makes the first stage. The method formed at the second stage of resource development is built on the base of the model. The means created at the third stage are information technologies based on the model and encapsulated in the method.
At the first stage, we proposed a model for the teaching quality which reflects the semantic and hierarchical relationship between ─ observed characteristics that can be measured directly; ─ unobserved characteristics that can be evaluated by analyzing students’ feedback; ─ measures that determine the teaching process characteristics quantitatively; ─ teaching tactics that describe all kinds of activities for the teaching process improvement.
At the second stage, we proposed a method, which considers the teaching process as a teaching quality management system with negative feedback. Measures of observed and unobserved characteristics are compared with their threshold values. The difference between the threshold and obtained values becomes the base for choosing the appropriate teaching tactics. Each tactic describes the activities that lead to improving the particular quality characteristic and can affect three foundations of the teaching process, namely the course content, delivery methods, and course materials.
At the third stage, we proposed a formalization of the questionnaire structure. Such formalization supports covering and evaluating unobservable characteristics of the teaching quality. In addition, the issue of teaching tactics cataloging is brought forth. Such a catalog can support the systematic process of forming measures for teaching quality improvement. This issue is the object of further research.
The comparison of SET results for different courses could provide such additional benefits as: ─ determination of the appropriate sequence of courses in the curriculum from the students’ point of view; ─ adjustment of the combination of structural elements in the courses of the curriculum; ─ determination of the valid set of teaching aid materials and acceptable channels for their delivery.
The proposed resource for improving the teaching quality does not depend on the particular education form (such as full-time, part-time, e-learning). Comprehensive formalization could be realized at different automation levels of teaching quality improvement caused by conditions and context of implementation. Continuous monitoring gives the possibility to achieve and preserve the requested quality level of the teaching process. Time: Priority Development Directions of Ukraine and Poland, Vol. 3, pp. 106–109, Wolomin, Republic of Poland (2018). 19. Mangal, S. K., Mangal, Sh.: Learning and Teaching. PHI Learning Pvt. Ltd. (2019).