=Paper=
{{Paper
|id=Vol-2501/paper3
|storemode=property
|title=Sensing Team Interaction to Enhance Learning and Performance during Adaptive Instruction
|pdfUrl=https://ceur-ws.org/Vol-2501/paper3.pdf
|volume=Vol-2501
|authors=Robert Sottilare,Ross Hoehn
|dblpUrl=https://dblp.org/rec/conf/aied/SottilareH19
}}
==Sensing Team Interaction to Enhance Learning and Performance during Adaptive Instruction==
https://ceur-ws.org/Vol-2501/paper3.pdf
Sensing Team Interaction to Enhance Learning and
Performance during Adaptive Instruction
Robert Sottilare & Ross Hoehn
Soar Technology, Inc.
{bob.sottilare, ross.hoehn}@soartech.com
Abstract. Adaptive instruction is any individual or collective learning experience
guided by artificially-intelligent, computer-based systems that tailor instruction
and recommendations based on their goals, needs and preferences. Research into
adaptive instructional methods has gained prominence over the last five years
with a greater understanding of the benefits that tailored training and educational
experiences provide to learner. While the application of adaptive instruction to
task domains for individual learners has been prevalent, there is a growing desire
to realize the same benefits for team instruction. As with any instruction, there
are a set of measures that determine progress toward a set of learning objectives.
In this paper, we discuss the importance of measures related to the interaction of
team members, how this data might be captured, and how it might be interpreted
to identify trends, provide recommendations, and select optimal instructional ac-
tions (e.g., feedback, support, direction) for teams.
Keywords: Adaptive Instruction, Sensors, Team Interaction, Team Learning,
Team Performance
1 Introduction
Building upon work by Burke [1], we define teams and contributions to their success
as follows:
A team is set of two or more individuals, interacting interdependently and adaptively
towards a common valued goal or set of goals.
Team members generally have defined roles and responsibilities, but their roles may
overlap and in some cases be redundant.
Teams must master both taskwork and teamwork skills to be optimally effective
Team effectiveness is also influenced by the level of effort provided by team mem-
bers, their performance strategies, and the individual and collective knowledge and
skills in the task domain(s) they operate within.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).
16
Individual and team knowledge, skills and abilities are greatly influenced by the
amount and effectiveness of deliberate practice (training).
Adaptive instructional systems (AISs) should tailor training to the capabilities of the
team and its members, and provide relevant content and effective strategies in pur-
suing the goal of optimal team learning and performance.
AISs are artificially-intelligent, computer-based systems that guide learning experi-
ences by tailoring instruction and recommendations based on the goals, needs, and pref-
erences of each individual learner or team in the context of domain learning objectives
[2]. The goal of adaptive instruction is to provide computer-guided, self-regulated ex-
periences for individuals and groups that are equivalent to or better than instruction
provided by an expert human tutor. AISs support technology-enhanced learning (TEL)
which “aims to design, develop and test sociotechnical innovations that will support
and enhance learning practices” [3]. AIS learning technologies include intelligent tu-
toring systems (ITSs), recommender systems, and other intelligent media that model
the learner and tailor instruction based on that learner model (Figure 1).
Fig. 1. Categories of Adaptive Instructional Systems
ITSs are computer learning environments that help learners master knowledge and
skills using intelligent algorithms that tailor to learner idiosyncrasies at a fine-grained
level and that instantiate complex principles of learning [4-5]. ITSs normally work with
one learner at a time, but emerging capabilities are targeted to support automated in-
struction for groups of collaborative learners or teams of learners. AIS architectures
such as the Generalized Intelligent Framework for Tutoring (GIFT) [4, 6-7] are not
specifically ITSs, but they do provide the building blocks (components, tools, and pro-
cesses) needed to generate ITSs and instantiate the design principles that govern the
delivery of automated instruction by their ITSs [6-7].
17
Recommender systems provide strategies or plans for the AIS’s next action based
upon the learner’s state(s) or suggestions about what the learner should do next. Rec-
ommendations can include suggestions about where to find novel domain resources,
identification of other learners with similar interests or optimal learning paths through
the learning resources [3]. We have created the category of “intelligent media” as a
catch-all for AISs that are not ITSs or recommender systems.
Automatically instructing individual learners is primarily focused on a process of
acquiring data about a specific learner’s behaviors and physiology, and then using that
data to classify current states and predict future states. Learner states (e.g., performance,
emotional) are then used by an AIS to determine readiness to learn and gaps between
the individual’s knowledge and skills and the instructional objectives. While this is dif-
ficult, automating the acquisition of individual learner behaviors and classifying their
states to determine their progress toward learning objectives has received substantial
attention and resulted in significant progress in both government and academic research
during the past 10 years. This includes research on evidence-centered design [8], self-
regulation [9], and stealth assessment [10]. Now the automated assessment of teams is
gaining strong support.
One critical need recently identified by the US Army’s Synthetic Training Environ-
ment (STE) program is the capability to automatically facilitate the training and educa-
tion of teams. Whether it is a fire team training to learn building clearing tactics, a squad
working to collaboratively to solve problems in the field, or a staff working to collec-
tively develop recommended courses of action for their commander, the members of a
team act, but also interact (e.g., communication, coaching, cooperating) with each
other, and this makes the modeling of teams considerably more complex than the mod-
eling of individual learners.
Team instruction includes the need to understand not only the progress toward learn-
ing objectives, but also interactions (e.g., teamwork) that have second-order effects on
learning. We contend that a lack of interaction data makes it difficult to detect, classify,
and predict interactions between team members, and we propose that there exists a need
to adapt or develop sensors that are capable of acquiring this interaction data to identify
team behavioral markers as recommended by Sottilare et al. [11].
Most solutions to the team interaction process have been manual with human ob-
servers identifying interaction occurrences and determining their meaning. Many times,
this observation process is time consuming and expensive, but also occasionally inac-
curate. Ideally, we seek to automate this observation process, but technology (a tool or
method) is required to capture data associated with these events in real-time to support
efficient and effective training. This paper targets the automated specific team interac-
tions and provides recommendations for sensors to support the detection, acquisition,
and classification of team interactions.
18
2 Challenges
According to Sottilare et al. [11], a group interacting while under instruction may be
categorized into one of three areas: team taskwork (a group learning to do a task to-
gether), teamwork (the interactions of group members working toward a shared goal),
and collaborative learning (a group of learners with a shared learning goal or problem
to solve) [12]. Taskwork is a subset of team training that is focused on developing pro-
ficiency in task domains required for a specific duty of one’s job [12]. Teamwork is
the “coordination, cooperation, and communication among individuals to achieve a
shared goal” [13]. Collaborative learning (also referred to as cooperative learning) is “a
situation in which two or more people learn or attempt to learn something together”
[14]. The interaction between team members is the key to understanding group instruc-
tion and the acquisition of interaction data is the key to identify teamwork behaviors.
The three major challenges are:
1. Unobtrusive data acquisition - identifying or adapting existing sensors or creating
new sensors to unobtrusively acquire data about team member interaction behaviors
(primarily communication).
2. Teamwork state classification - applying appropriate machine learning methods to
accurately classify teamwork states: communication, cooperation, coordination,
cognition, coaching, conflict, and instructional conditions as identified by Salas,
Dickinson, Converse, and Tannenbaum [15].
3. Selecting optimal plans and actions - given an accurate picture of teamwork, select-
ing appropriate instructional strategies and tactics to optimize learning and perfor-
mance.
3 Related Research
Examining the three challenges identified above, we now discuss related research for
both individual learners and teams. We refer to the learning effect model as a basis for
understanding AIS processes for both individuals and team assessments [11].
3.1 Assessing Individual Learner States
Previous work in this area is related to our first challenge of acquiring learner data but
is limited to low cost sensors for assessing individual learner states rather than team
states. Carroll et al. [16] conducted a survey of low cost behavioral and physiological
sensors including EEGs, heart rate monitors, breathing straps, pressure sensors, and
low-cost eye trackers. Kokini et al. [17] evaluated the data acquired by low cost sensors
to classify learner states including workload, attention, engagement, and emotions that
mediate learning (e.g., frustration, anxiety, and boredom).
19
3.2 Assessing Teamwork States
Johnson, Sottilare, Sinatra and Burke [18] integrated several sources of research related
to assessing teamwork states within intelligent team tutoring systems (ITTSs). Sottilare
et al. [11] identified several team behavioral markers indicative of a variety of team-
work behaviors but did not specifically identify how that data would be acquired and
assessed. So, what are the next steps needed to move us forward? Since the bulk of
interaction data is verbal and non-verbal communication, we expect to investigate
methods to acquire and interpret communication data.
DeCostanza, Gamble, Estrada and Orvis [19] identified unobtrusive assessment
methods while suggesting sources of psychophysiological data (e.g., heart rate varia-
bility, eye tracking, neural responses) to support both automated team performance
(taskwork) assessment and teamwork assessment. Taskwork represents the objectives,
roles, responsibilities, and actions of the team and its members [20]. Teamwork de-
scribes the interaction behaviors of team members as they communicate, coordinate,
cooperate, lead, and support each other [20].
Freeman and Zachary [21] identified challenges associated with the design of ITTSs
including the processing of communications data and the lack of automated and gener-
alizable measures of teamwork. They also identified several essential features including
“the use of team training objectives, teamwork models, measures of teamwork, diag-
nostic capability, instructional strategies, and adaptation of training to team needs” [21].
Sinatra and Sottilare [22] also identified ITTS design features and challenges, in-
cluding the ability of the ITTS to process and respond in near-real time. Currently,
many of the behavioral markers (e.g., communications) needed for assessment and in-
structional management “rely heavily on human intervention, interpretation, and cod-
ing” [22]. A significant number of communication behavioral markers (over 100) have
been identified [11], but this communication data must be processed to assess teamwork
states and select optimal instructional strategies and tactics. In the next section, we
begin to evaluate processes to automatically analyze communications with the goal of
determining teamwork states.
3.3 Automated Analysis of Communications to Assess Teamwork
States
Communication, both verbal and non-verbal, is a common element of all team-based
activities and is the single most influential process related to teamwork assessments.
LeCouteur [23] highlighted the importance of maintaining high frequencies of commu-
nication between players during team sports, but the communication is important to
successful performance in any coordinated team activity (e.g., military maneuvers).
Empirically, communications account for 28% of the variance in team learning, and
13% of variance in team performance.
In this section, we examine methods to automatically analyze team communications
as a basis for assessing teamwork states. The reason we want to automatically conduct
this analysis is due to the expense of manual analysis. Emmert and Barker [24] identi-
fied a study in which manual communication analysis required 28 hours of transcription
20
and encoding for every hour of communication. This analysis could be reduced to one
hour with automated, real-time transcription and encoding processes. This section pro-
vides a review of some of the approaches that might facilitate automated analysis of
team communications.
Foltz and Martin [25] describe two approaches to automated analysis of team com-
munications to assess team performance: 1) theory-based and 2) model-based. In the
theory-based approach to learning analytics, the researcher uses a cognitive, social or
communication theory to identify key factors and then tests these factors to see how
well the model accounts for the key factors. In the model-based approach to learning
analytics, the researcher uses human-derived (identify by a subject matter expert) or
objective team performance measures and evaluates the relationship between these
measures and team performance. Patterns of communication provide information about
the type and duration of interactions between team members. Latent semantic analysis
(LSA) is used to analyze the content of communications by measuring and comparing
semantic information in verbal interactions.
Epistemic Network Analysis (ENA) is a theory-based approach to learning analytics
and is used to model and compare the structure of connections between elements in
coded data [26]. For example, a set of 8 verbal response modes (VRMs) represents a
generalized set of communication behaviors within a team [26]:
Disclosure - reveals thoughts, feelings, perceptions, intentions.
Advisement - attempts to guide behavior, suggestions, commands.
Edification - states objective information.
Confirmation - agreement, disagreement, shared experience or belief.
Question - requests information or guidance.
Interpretation - explains or labels the other, judgments or evaluations of behavior.
Reflection - repetition, restatements, puts other’s experience into words.
Acknowledgment - conveys receipt of communication.
ENA, uses communication data to construct models of learning that are visualized
as network graphs (Figure 2) that are mathematical representations of patterns of con-
nections [27]. When employing ENA, “it is essential to consider the semantic and
conceptual content of what gets said during social interactions in addition to tracing
the patterns of who talks to whom in a social network” [28].
21
Fig. 2. ENA comparing high and low performing teams based on VRMs [26].
Ryan et al. [29] examined automated methods to assess the verbal skills of clinicians
and identified several metrics indicative of good discourse, written or verbal commu-
nication. We adapted these metrics for team instruction:
Speaker ratio - the equitable distribution of talking time among team members indi-
cates a willingness to listen.
Turn-taking - similar to speaker ratio, turn-taking indicates a wiliness to listen
Overlapping talk - interruption or simultaneous talk may indicate a lack of respect
for the contributions of others.
Pauses - number of pauses longer than 2 seconds invites team member contributions
and indicates a willingness to listen to others.
Speed of speech - the pace of speech can influence comprehension and indicates a
desire to be understood when it the speaker moderates their communications to allow
the receiver(s) to fully understand intent.
Energy (pitch and tone) - influences receivers’ perceptions with respect to the en-
gagement and empathy of the speaker.
Plain language – speakers should evaluate word choices, sentence length, and struc-
ture to be appropriate with the receivers’ capabilities.
22
Clinical jargon - the speaker’s choice of terminology and effort to explain technical
words indicate a willingness to coach/mentor and be understood by other team mem-
bers.
Shared decision making - effort to inform, elicit, and integrate preferences of others
into decision-making processes.
Sottilare et al. [11] identified specific behavioral markers that could be identified by
LSA. Below is a small sample of teamwork states, their definitions, and a subset of
communication (verbal and non-verbal) behavioral markers that influence them:
Trust - the willingness of a team member to be vulnerable to the actions of another
team member based on the expectation that the other team member will perform a
particular action important to the trustor, irrespective of the ability to monitor or
control that other team member [30].
─ Opinion-Seeking (verbal) - occurrences of team members actively seeking out the
opinions of other team members regarding work tasks.
─ Information Sharing (verbal) - amount of task information share with fellow team
members.
Collective efficacy - a shared belief in a group’s capabilities to organize and execute
a course of action [31].
─ Help-seeking (verbal) - team members actively request backup when needed.
─ Acknowledgment/Recognition (verbal) - team members acknowledge input from
other teammates during taskwork, and incorporate their suggestions.
Conflict – the process resulting from tension between team members that is due to
real or perceived differences [32].
─ Frustration (non-verbal) – team members furrowed their brow, get red faced, or
physically agitated.
─ Loudness (verbal) - team members raise their voice when talking with each other
or otherwise communicate frustration with the team.
─ Withdrawal (non-verbal) - unwillingness to continue working with someone or
on a task.
4 Assessing Teamwork States in GIFT based on Team
Communications
In this section, we review related work and methods for assessing team interactions
within the GIFT architecture [6-7]. As noted in section 3.2, communications data is the
key to teamwork assessment and also the most challenging to interpret. We begin the
quest to assess teamwork within GIFT tutors by examining the evolution of GIFT team
models: single and multi-level.
23
4.1 Single-level Modeling in GIFT
Recently, the US Army examined a simplified approach to modeling teams by using
the existing GIFT architecture as a team model. Figure 3 shows the four primary ITS
architectural components in green: learner, pedagogical (instructional), domain, and in-
terface modules. The yellow elements define data/information passed between archi-
tectural components and to/from the individual learner.
Fig. 3. The GIFT architecture for individual learner training experiences
To use the GIFT architecture for team training, none of the architectural component nor
the principles that govern them have been altered, but the data, shown in orange, is
modified to reflect a team training scenario (Figure 4). The learner module remains
intact but is relabeled as a team model. The team scenario includes assessment of team
states based on initial team model and team data. Individual learner performance be-
comes an assessment of team performance based on team objectives, and the input of
individual team members comprises team inputs to the tutor-user interface. GIFT prin-
ciples driving strategy recommendations remain the same and are based upon best train-
ing practices found in the instructional literature. Tactic selection and presentation re-
main based on the context defined by the domain module.
24
Fig. 4. The GIFT architecture for team training experiences
While this team model works surprising well for team taskwork, the lack of individual
learner models for team members means the tutor has no knowledge of the interactions
between members. In other words, the GIFT ITS only has knowledge of the team’s
objectives, measures associated with those objectives, and progress toward the defined
objectives. The goals associated with this team scenario allow for adaptation of content
(primarily difficulty level), feedback, and support associated with the task, but not ad-
aptation based upon the state of teamwork (e.g., communication, collaboration, coach-
ing).
4.2 Multi-level Modeling in GIFT
Gilbert et al. [33] specifically modified the GIFT architecture to support both individual
and team models (Figure 5), but simplified teamwork measures to register whether
communication occurred/did not occur to enable a largely automated approach to team
tutoring. No effort was directed at interpreting any of the communication behaviors and
this concept largely focused on taskwork measures and assessments with the develop-
ment of team domain knowledge file (DKF).
25
Fig. 5. Individual and team taskwork models in a modified GIFT architecture [33].
5 Recommended Practices for Assessing Team Interaction
To fully understand progress toward objectives defined by the task and how well the
team is working together toward those objectives, we highly recommend a model of the
team that includes assessment of the team communications. Just as team performance
is assessed using measures related to taskwork objectives, we recommend team perfor-
mance is also assessed using measures related to teamwork objectives (e.g., timely res-
olution of conflict). Critical teamwork assessments should include measures (e.g., be-
havioral markers) from team interactions in order to assess their impact on teamwork
states.
If we think about the learning effect model (LEM) [11] for individuals and data flow
in that model is approximated in Figure 6, then the we might adapt Sottilare’s LEM for
teams and represent it as shown in Figure 7. In this updated model of teamwork, GIFT
is able to capture interactions within the team under assessment and use their interac-
tions to assess teamwork states.
26
Fig. 6. Learning Effect Model (LEM) for Individuals [11].
Fig. 7. Learning Effect Model (LEM) for Teamwork [11].
If we breakdown the process for identifying team interactions and other behaviors,
and examine using them as indicators of teamwork states, we need to address methods
to acquire the data, process the data (classifying teamwork states) and formulating
courses of action ranked by reward (as shown in Figure 8).
27
Fig. 8. Recommended Process for Identifying Teamwork States and Selecting Appropriate AIS
Interventions.
The following provides a set of recommendations for future research and development:
Continue investigating methods to unobtrusively acquire team interaction data.
Improve the accuracy of classification methods to determine discrete teamwork
states (e.g., low conflict or moderate workload conditions).
Develop instructional strategies rooted in best practices for teamwork, assess their
effectiveness, and adapt strategies as needed.
Continue investigating machine learning methods to select AIS actions to optimize
learning, performance, retention, and especially transfer of training per Baldwin and
Ford [34].
Finally, in the spirit of moving forward with the teamwork assessment process, we in-
troduce a concept, perceptual machine learning, which is the use of multiple sensors to
acquire data (visual, aural, olfactory, haptic/tactile) about the team and their environ-
ment. This might seem to be much the same way that human observers capture infor-
mation from the environment to classify or predict team states. However, we advocate
the use of use this data to evaluate conditions in the environment through both separate
sensory channels and through a data fusion process where each sensory channel rein-
forces the classification/prediction of team states from other sensory channels.
28
References
1. Burke, S. Using the Science of Teams to Inform GIFT Development. Presentation to the
Army Research Laboratory Expert Workshop on Team Tutoring. University of Central Flor-
ida. (2017).
2. Sottilare, R., Brawner, K. Component Interaction within the Generalized Intelligent Frame-
work for Tutoring (GIFT) as a Model for Adaptive Instructional System Standards. In the
Adaptive Instructional System (AIS) Standards Workshop of the 14th International Intelli-
gent Tutoring Systems (ITS) Conference, Montreal, Quebec, Canada. (2018).
3. Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., Koper, R. Recommender systems
in technology enhanced learning. In Recommender systems handbook,pp. 387-415.
Springer, Boston, MA. (2011).
4. Graesser, A. C., Hu, X., Nye, B., Sottilare, R. Intelligent tutoring systems, serious games,
and the Generalized Intelligent Framework for Tutoring (GIFT). In H. F. O’Neil, E. L.
Baker, and R. S. Perez (Eds.), Using games and simulation for teaching and assessment, pp.
58–79. Abingdon, UK: Routledge. (2016).
5. Graesser, A. C., Hu, X., Sottilare, R. Intelligent tutoring systems. In International handbook
of the learning sciences, pp. 246-255. Routledge. (2018).
6. Sottilare, R.A., Brawner, K.W., Goldberg, B.S. Holden, H.K. The Generalized Intelligent
Framework for Tutoring (GIFT). Concept paper released as part of GIFT software docu-
mentation. Orlando, FL: US Army Research Laboratory – Human Research & Engineering
Directorate (ARL-HRED). (2012). Retrieved from: https://gifttutoring.org/attach-
ments/152/GIFTDescription_0.pdf
7. Sottilare, R., Brawner, K., Sinatra, A. Johnston, J. An Updated Concept for a Generalized
Intelligent Framework for Tutoring (GIFT). Orlando, FL: US Army Research Laboratory.
(2017). DOI: 10.13140/RG.2.2.12941.54244.
8. Rupp, A. A., Gushta, M., Mislevy, R. J., Shaffer, D. W. Evidence-centered design of epis-
temic games: Measurement principles for complex learning environments. The Journal of
Technology, Learning and Assessment, 8(4). (2010).
9. Schunk, D. H., Zimmerman, B. (Eds.). Handbook of self-regulation of learning and perfor-
mance. Taylor & Francis. (2011).
10. Shute, V. J. Stealth assessment in computer-based games to support learning. Computer
games and instruction, 55(2), pp. 503-524. (2011).
11. Sottilare, R.A., Burke, C.S., Salas, E., Sinatra, A.M., Johnston, J.H. Gilbert, S.B. Designing
Adaptive Instruction for Teams: A Meta-Analysis. International Journal of Artificial Intelli-
gence in Education. (2017). DOI: 10.1007/s40593-017-0146-z.
12. Cannon-Bowers, J. A., Bowers, C. Team development and functioning. In S. Zedeck (Ed.),
APA handbook of industrial and organizational psychology, Vol 1: Building and developing
the organization, pp. 597–650. Washington, DC: American Psychological Association.
(2011).
13. Salas, E. Team training essentials: A research-based guide. London: Routledge. (2015).
14. Dillenbourg, P. What do you mean by collaborative learning? Collaborative-learning: Cog-
nitive and Computational Approaches, 1, pp. 1–15. (1999).
15. Salas, E., Dickinson, T. L., Converse, S. A., Tannenbaum, S. I.Toward an understanding of
team performance and training. In R. W. Swezey, E. Salas, R. W. Swezey, & E. Salas (Eds.),
Teams: Their training and performance, pp. 3–29. Westport: Ablex Publishing. (1992).
29
16. Carroll, M. Kokini, C., Champney, R., Fuchs, S., Sottilare, R., Goldberg, B. Modeling
Trainee Affective and Cognitive State Using Low Cost Sensors. In Proceedings of the In-
terservice/Industry Training Simulation & Education Conference, Orlando, Florida, Decem-
ber 2011. (2011).
17. Kokini, C., Carroll, M., Ramirez-Padron, R., Wang, X., Hale, K., Sottilare, R., Goldberg, B.
Quantification of Trainee Affective and Cognitive State in Real-time. In Proceedings of the
Interservice/Industry Training Simulation & Education Conference, Orlando, Florida, De-
cember 2012. (2012).
18. Johnston, J., Sottilare, R., Sinatra, A. M., Burke, C. S. (Eds.). Building intelligent tutoring
systems for teams: What matters. Emerald Publishing Limited. (2018).
19. DeCostanza, A. H., Gamble, K. R., Estrada, A. X., Orvis, K. L. Team measurement: Unob-
trusive strategies for intelligent tutoring systems. In Building Intelligent Tutoring Systems
for Teams: What Matters, pp. 101-130. Emerald Publishing Limited. (2018).
20. Marks, M. A., Mathieu, J. E., Zaccaro, S. J.A temporally based framework and taxonomy of
team processes. Academy of management review, 26(3), pp. 356-376. (2001).
21. Freeman, J., Zachary, W. Intelligent tutoring for team training: Lessons learned from US
military research. In Building Intelligent Tutoring Systems for Teams: What Matters, pp.
215-245. Emerald Publishing Limited. (2018).
22. Sinatra, A. M., Sottilare, R. Considerations in the Design of a Team Tutor. In Building In-
telligent Tutoring Systems for Teams: What Matters, pp. 301-312. Emerald Publishing Lim-
ited. (2018).
23. LeCouteur, A., Feo, R. Real-time communication during play: Analysis of team-mates’ talk
and interaction. Psychology of sport and exercise, 12(2), pp. 124-134. (2011).
24. Emmert, P., Barker, L. L. Measurement of communication behavior. Addison-Wesley
Longman Ltd. (1989).
25. Foltz, P. W., Martin, M. J.Automated communication analysis of teams. Team Effectiveness
in Complex Organizations. (2008).
26. Sullivan, S., Warner-Hillard, C., Eagan, B., Thompson, R. J., Ruis, A. R., Haines, K., ...
Jung, H. S. Using epistemic network analysis to identify targets for educational interventions
in trauma team communication. Surgery, 163(4), pp. 938-943. (2018).
27. Shaffer, D., Ruis, A. Epistemic network analysis: A worked example of theory-based learn-
ing analytics. Handbook of learning analytics. (2017).
28. Ruis, A.R., Hampton, A.J., Goldberg, B.S., Shaffer, D.W.Modeling Processes of Encultura-
tion in Team Training. In Sottilare, R., Graesser, A., Hu, X., & Sinatra, A. (Eds.). Design
Recommendations for Intelligent Tutoring Systems: Volume 6 – Team Tutoring. Army Re-
search Laboratory, Orlando, Florida. (2018). ISBN: 978-0-9977257-4-2.
29. Ryan, P., Luz, S., Albert, P., Vogel, C., Normand, C., Elwyn, G. Using artificial intelligence
to assess clinicians’ communication skills. Bmj, 364, pp. l161. (2019).
30. Mayer, R. C., Davis, J. H., Schoorman, F. D. (1995). An integrative model of organizational
trust. Academy of Management Review, 20, pp. 709–734. (1995).
31. Bandura, A. Self-efficacy: The exercise of control. London: Macmillan Publishers. (1997).
32. De Dreu, C. K., Weingart, L. R. (2003). Task versus relationship conflict, team performance,
and team member satisfaction: a meta-analysis. Journal of Applied Psychology, 88(4), pp.
741–749. (2003).
33. Gilbert, S. B., Slavina, A., Dorneich, M. C., Sinatra, A. M., Bonner, D., Johnston, J., ...
Winer, E. Creating a team tutor using GIFT. International Journal of Artificial Intelligence
in Education, 28(2), pp. 286-313. (2018).
34. Baldwin, T.T., Ford, J.K. Transfer of training: A review and directions for future research.
Personnel psychology, 41(1), pp.63-105. (1988).