This paper introduces QTMM2012c+, a resource which links relations between propositions (inference, conflict and rephrase) to dialogue act sequences. QTMM2012c+ builds on the MM2012c annotated corpus of BBC Moral Maze debates, extending it with new annotations - for speaker roles (chair, panellists and witnesses), speaker stances (neutral, pro and con) and locution chronological ordering - and making the information available in a queryable format. We show how the new resource allows for: i) automatic extraction of empirically-grounded dialogue rules which describe choice and frequency of dialogue acts with specific argumentative functions given the dialogue history, and ii) extraction of generation templates that reflect naturally-occurring argumentative locutions in empirically-grounded dialogue. QTMM2012c+ facilitates automatic analysis of argument transitions between speakers, extending previous manual analysis of the MM2012c corpus, enabling empirical tests of theories of argumentative dialogue. Dialogue based on argumentation, Argumentation in agent and multi-agent systems, Argument-based machine learning, Strategies in argumentation, Argumentation schemes, Dialogue rules, MM2012c 5th Workshop on Advances In Argumentation In Artificial Intelligence ( 3 2021)
Most recent work on dialogue modelling and systems is empirical in nature, aiming to model
natural task-oriented dialogue (e.g. [
Hunter [11] argues that this strictly normative orientation is too restrictive for work on computational modelling of persuasive argumentative dialogue. Our interest is not so much in purely persuasive dialogue, but rather in dialogue that explores, via argument, diferent points of view. Nevertheless the point remains that a strictly normative perspective is inadequate and empirical modelling of natural argumentation in dialogue is called for, if the aim is to eventually build systems that meet human standards of naturalness and coherence.
A key challenge for this strand of work is to uncover how relations between propositions (e.g. inference, conflict and rephrase) can be translated into moves in argumentative dialogue. In this paper, we introduce QTMM2012c+, a resource that describes such translations based on a real argumentative conversation.
After an introduction and a first analysis of QTMM2012c+ (Section 3 and Section 4), we will present a description of two use cases scenario (Section 5). The examples will shows that QTMM2012c+ can be used to define argumentation strategies or as a training resource for multi-agent dialogue systems (for example statistical systems).
Our work is based on the Moral Maze (MM2012) dataset [12, 13, 14]. This dataset has been used to define rule-based models for: i) automatic extraction of argument from dialogues [ 15, 16], and ii) automatic detection of discourse units and illocutionary structure from dialogues [17, 18]. QTMM2012c+, which specifically extends MM2012c, is in the form of a Queryable Table (hence the QT prefix) and has been enhanced with new annotations (indicated with the + post-fix).
Yaskorska-Shah [19] uses the Moral Maze dataset to define transition schemes that map propositional relations to dialogue acts, which in turn can be used to formulate dialogue rules. As a use case, we show how QTMM2012c+ allows us to automatically extract dialogue rules. As we will show in Section 5.1, our work extends that of Yaskorska-Shah in terms of (1) method (by using automatic rather than manual harvesting of schemes), (2) scale (significantly more data covered) and (3) content (we introduce new annotations that allow us to ground our schemes in dialogue speaker roles and speaker stances).
Stoyanchev and Piwek [20] developed a mapping between text segments and dialogue acts on a smaller corpus of human-authored dialogue, however they focused on discourse rather than propositional relations in text.
For an in-depth description (including reliability study and dataset statistics) of the MM2012c dataset, which our work builds on, see [14].
MM2012c is composed of five episodes, all from 2012, of the Moral Maze BBC Radio 4 show. The Moral Maze format involves a chair/moderator, four panellists and four witnesses. They discuss the moral implications of current topics. In the discussion, two panellists and two witnesses take a position in favour of the topic under discussion and the others take positions against it. An episode proceeds in ‘rounds’ where two of the panellists interrogate a witness (whose opinion is opposite to that of the panellists).
The MM2012c dataset is annotated following the Inference Anchoring Theory (IAT) [21]. An example of an annotated fragment of dialogue is shown in Figure 1. On the right-hand side, we can see two locutions, ‘MT : Don’t you think there’s a streak that says, it was a pretty balanced account, there’s nothing wrong with colonialism.’ (with speaker MT) and ‘AM : I don’t think people do think that’ (with speaker AM). They are linked by a transition box, which signifies that the second locution is a reply or response to its predecessor.
Each locution is anchored to a proposition (shown in the two left-most blue boxes) via an Illocutionary connection (IC) (in the middle yellow boxes). These represent the propositional content and Illocutionary force from Speech Act theory [22].1 In this case, the first locution’s Illocutionary connection is ‘Assertive Questioning’ and the second ‘Asserting’. Annotators were instructed to express the propositions as complete declarative sentences, removing ellipsis and reconstructing anaphoric references.
Finally, each transition between locutions is anchored, via an Illocutionary connection, to a Propositional relation. In this paper we refer to the Illocutionary connections associated with the transitions as ICTA. In this instance, Transition is anchored via the ‘Disagreeing’ Illocutionary connection to the Conflict Propositional relation. Apart from the Conflict relation between propositions, IAT singles out Inference (when one proposition is used to provide a reason to accept the other) and Rephrase (when one proposition is more or less a paraphrase of the other).2
This illustrates how IAT captures: 1) dialogue internal relations (via Transitions), 2) relations between the propositions expressed in the dialogue (via Propositional relations) and 3) the links between the two (i.e. between dialogue and propositions) via Illocutionary connections. The QTMM2012c+ Queryable Table3 was constructed from the MM2012c dataset in two steps, during which all 1747 Transitions in MM2012c were processed. (1) We started with a semi-automatic step. We developed an algorithm that extracted the transitions one map at a time following the dialogue flow (note that MM2012c is divided into maps which each covering part of the dialogue in an episode). We then manually checked 1The illocutionary force expresses the speaker’s intention in producing an utterance. 2IAT has further subdivisions of these, but we ignore them for the purpose of this paper.
3QTMM2012c+ can be downloaded from https://github.com/jacopoamidei/ Supplementary-material-A-Queryable-Empirically-grounded-Resource-of-Dialogue-with-Argumentation. the correctness of the transitions the algorithm extracted against the original data and where needed corrected any errors introduced by the algorithm. (2) In the second step we enriched the MM2012c with new/implicit information. (2a) Following Yaskorska-Shah [19], we enriched the MM2012c data set by adding “Yes” or “No” for those locutions not linked to any proposition and which disagreed or agreed with other locutions in the same transition. More precisely, in the MM2012c dataset, locutions that express disagreement or agreement elliptically (e.g., with locutions such as “That’s true”, “Yes”, “I disagree”, etc.) are not linked to propositional content. They are only indirectly linked to the proposition they respond to. To make these easier to process (without changing the actual information), we link them to “Yes” or “No”. For example, Figure 2 shows a case where the second locution has been associated with the IC “Yes” (dotted lines signal our addition to the original representation). Similarly, for cases such as the one in Figure 3, the locution which is not linked to any proposition (‘CL: Is that not true?’), has been linked to the illocutionary connection of its Transition.4 (2b) We labelled the Chair as Neutral and the non-chair speakers as either Pro or Con, depending on whether they were for or against the main claim under discussion. The three authors of this paper independently annotated for Pro/Con and agreed 100% except for the episode on Banking System which had more than one claim under discussion. For this episode two claims can be identified: A) We need more rules to make the banks trustworthy and B) The banks’ behaviour is immoral. After discussing which of the claims to designate as the main claim, resulting in selection of Claim A, we reached perfect agreement on Pro/Con. (2c) MM2012c only has a partial record of the order in which locutions occurred; we used the Moral Maze Transcripts [23] for associating with each locution a number (ID dialogue flow) representing its dialogue position. (2d) Using Moral Maze episode descriptions5, we labelled each speaker with one of the following roles: Chair, Panellist or Witness.
4Note that we did not remove the illocutionary connection linked to the transition itself. We just add that connection also to the locution. Thus, there is no loss of information.
5See https://www.bbc.co.uk/programmes/b006qk11/episodes/player. 6The columns name are listed in Appendix B. 7Reported Speech is equivalent to the Attribution discourse relation in Rhetorical Structure Theory (RST) [24]. 8All transitions involve either one (Same) or two (Diferent) speakers.
The columns Li (for 1 ≤ ≤ 7 ) store up to 7 locutions from the transition (unused columns are populated with _ ). For each locution Li, QTMM2012c+ has a column about the locution IC (Li IC), the role of the speaker that uttered that locution (Li Role) – the possible values are: Chair, Witness, Panellist –, the stance of the speaker that uttered that locution (Li Stance) – the possible values are: Neutral, Pro, Con –, the order in which the locutions were uttered in the dialogue (Li ID dialogue flow ) and the unique ID associated with that locution (Li ID).
The columns LiReported Speech (for 1 ≤ ≤ 4 ) store the reported speech used in the transition. If reported speech is not used in the transition, the column value is 0. For each reported speech LiReported Speech, QTMM2012c+ has a column about the reported speech IC (LiReported Speech IC), the role of the speaker that uttered that reported speech (LiReported Speech Role) – the possible values are: Chair, Witness, Panellist –, the stance of the speaker who uttered that reported speech (LiReported Speech Stance) – the possible values are: Neutral, Pro, Con –, and the unique ID associate to that reported speech (LiReported Speech ID).
The columns Pi (for 1 ≤ ≤ 7 ) store the propositions used in the transition. If a proposition is not used in the transition the column value is 0. For each proposition Pi, QTMM2012c+ has a column with a unique ID for that proposition (Li ID).
The columns PLiReported Speech (for 1 ≤ ≤ 4 ) store the proposition of the reported speech used in the transition. If a proposition of the reported speech is not used in the transition the column value is 0. For each reported speech PLiReported Speech QTMM2012c+ has a column for the unique ID associated with the proposition of the reported speech (PLiReported Speech ID).
The columns from “L1Reported Speech to L2Reported Speech” to “TA is nonanchoring” store information about the argument flow in the dataset. The argument flow describes the direction of the argumentation. For example the argument flow of Figure 1 goes from proposition 2 (‘people do not think that’) to proposition 1 (‘there’s a streak that says, it was a pretty balanced account, there’s nothing wrong with colonialism’). In this case, QTMM2012c+ associates the value 1 with the column P2 to P1 and the value 0 to all the columns representing alternative argument flows.
The value _ means the information required for the column is not applicable to the transition. For example, in a transition of degree 2, all the columns Li (for 3 ≤ ≤ 7 ) get the value _ .
Inference is the most used propositional relation in QTMM2012c+. It is used in 49% of transitions. Conflict is used in 12% of transitions, and Rephrase is used in 9% of transitions. In the remaining transitions (30%), the propositional relation is missing (see for example Figure 5). Arguing is the ICTA most frequently associated with Inference (99%), Disagreeing is the only ICTA associated with Conflict , and Rephrase is mostly associated with the ICTA Default Illocuting (95%). When a transition has no propositional relation, in the majority of the cases, the transition is not annotated with any illocutionary connection (74%). In some cases, the illocutionary connection of the transition is directly linked to a proposition (rather than via a propositional relation).9 When this happens, 80% of the time the ICTA used is Agreeing. For more detail, see Appendix A Table 1.
9Figure 3 is an example of these cases.
In QTMM2012c+ most transitions have a low degree, i.e. 90% are degree 2 (that is, transitions that link together two locutions) and 8% are degree 3. Conflict is 94% of the time with degree 2 transitions, and Rephrase 92% of the time. Instances of Inference, although used mainly with transitions of degree 2, can also be found with higher degrees (17%). More detail can be found in Appendix A Table 2.
We found that the transitions with a single same speaker are used more than those with multiple diferent speakers: same speaker transitions make up 71.6 % of the transitions (i.e. 1250 transitions), whereas the diferent speaker (multi-speaker) transitions make up 28.4 % (496 transitions). More detail can be found in Appendix A Table 3.
Reported speech is used in 12% of the transitions.
Checking the number of ICTA per degree, we found that Arguing is the most used ICTA (49%) and it is the only one used with degrees 5, 6 and 7. Considering only the transitions that have an ICTA, the majority of ICTA are used in transitions of degree 2 (87%). More detail can be found in Appendix A Table 4. Regarding the use of Illocutionary connections (IC) that links locution to proposition, we found that Asserting is the most used IC (78% of the times). The other most used IC are questions (13% consisting of: 3% Pure Questioning, 6% Assertive Questioning and 4% Rhetorical Questioning). For more detail, see Appendix A Table 5.
Looking at the relation between IC and speaker roles, we found that the majority of the dialogue plays out between the Panellists and the Witness, with only a minor role for the Chair.
The Chair mainly poses questions. Indeed, the Chair uses IC of question type 50% of the time. The number of IC used between the Panellists and the Witness is balanced: the Panellists use the 47% of the annotated IC, whereas the Witness use the 48% of the annotated IC. Nevertheless, an interesting discrepancy between the use of IC among the Panellists and the Witness can be identified. Although both the Panellists and the Witness use mainly Asserting (71% for Panellists and 88% for Witness), the Panellists use more questions and challenging than the Witness. In particular, the Panellists use 22% question type ICs and 2% challenging type ICs, whereas the Witness use 3% question type ICs and 0.3% challenging type ICs. This is consistent with Panellists playing the role of inquisitive speakers who challenge the Witness. For more detail, see Appendix A Table 6.
Finally we checked the number of IC per speakers’ stance. Although the Pro stance uses more IC than Con (Pro 51% of the time and Con 44% of the time), the types of IC used between the Pro and Con is reasonably balanced (for details, see Appendix A Table 7).
In order to show the flexibility and utility of QTMM2012c+, in this section we present two use cases. As a first example, we show how QTMM2012c+ can be used to define a set of dialogue rules and we compare our rules with those reported in [19]. As a second example, we show how QTMM2012c+ can be used for extracting dialogue generation templates.
QTMM2012c+ lends itself to extracting empirically-grounded rules for selecting the next dialogue act/proposition based on the dialogue history and propositional relation.
A dialogue rule is essentially an abstraction which collects together transitions that share certain properties.
We can count how many transitions fit a specific rule – below, we report this rule frequency in brackets at the end of each line. (For the sake of simplicity, the examples are extracted from transitions of degree 2 with no reported speech, and → stands for an (IAT) Inference from to .)
Let us suppose we are interested in a rule that captures what follows after pure questioning for the case of same-speaker transition. We can query QTMM2021c+ to obtain the following rule:
R1.2: Rephrase, via Asserting , where ( → ) (3 times); R1.3: Rephrase, via Assertive Questioning , where ( → ) (2 times); R1.4: Inference, via Asserting , where ( → ) (1 times); R1.5: Introducing another statement via Pure Questioning (11 times) R1.6: Introducing another statement via Assertive Questioning (1 times) R1.7: Introducing another statement via Asserting (1 times)
Yaskorska-Shah [19] defines a similar rule, but their rule for Pure Questioning only covers our R1.5 and R1.6. Also, as shown, from QTMM2012c+ we obtain rules together with their frequency. This is useful, for instance, when constructing a dialogue system, since it provides the raw material for choosing between dialogue rules using a statistical model.
Let us see a couple of further examples that illustrate the flexibility of QTMM2012c+. Suppose this time we are interested in a rule that involves the Chair, more specifically, a rule that describes the possible behaviour of a participant after the Chair performs a Pure Questioning.10
R2.2: Disagree, via Asserting Not- (3 times); R2.3: Rephrase, via Asserting , where ( → ) (14 times); R2.4: Rephrase, via Pure Questioning , where ( → ) (1 times); R2.5: Inference, via Popular Conceding , where ( → ) (1 times); R2.6: Introducing a second Pure Questioning (5 times).
With Rule 2, we capture the possible moves after the Chair performs a Pure Questioning. In this case, the rule does not specify properties of the participant who makes the response move. However, QTMM2012c+ allows for the extraction of more fine-grained rules: Rule 3 describes a Witness’s behaviour after the Chair performs a Pure Question.
10For this example, this participant can be the Chair themselves, a Witness or a Panellist.
R3.1: Rephrase, via Asserting , where ( → ) (7 times); R3.2: Inference, via Popular Conceding , where ( → ) (1 times); R3.3: Agree, via Asserting Yes (2 times); R3.4: Disagree, via Asserting Not- (2 times);
These examples illustrate the flexibility of QTMM2012c+: rules can be extracted based on a combination of roles and illocutionary connection, or a combination of stance and illocutionary connection or a combination of roles, stance and illocutionary connection. But they can be also more general, and be extracted using the illocutionary connection only.
Furthermore, frequency information can be used to build statistical models, for example, a model that aims to predict the next illocutionary connection. This kind of model can be then applied in a multi-party argumentative dialogue system.
Finally, because the rules are directly grounded in the rows that make up the Queryable Table QTMM2012c+ and which represent transitions, each rule can be traced back to the transitions that gave rise to it. For example, Figure 5 shows the transition instance underlying R1.6.11
Figure 5 shows a rule which was only observed a single time in the corpus (frequency equal to 1). This does not have to be interpreted as an aberration or an illegitimate response representing a breakdown in the dialogue. The corpus consists of real debates, where some patterns may occur very infrequently (e.g. because their felicity depends on a very specific dialogue context). Far from being considered as an error, they have to be considered as a possibly legitimate move in this kind of dialogue. That said, researchers that will use rules extracted from QTMM2012c+ can decide to use only the rules with a high frequency.
11As an illustrative example we have released the code for extracting Rule 1 at https://github.com/jacopoamidei/ Supplementary-material-A-Queryable-Empirically-grounded-Resource-of-Dialogue-with-Argumentation. This code also prints the transition instances from which the rule arose.
QTMM2012c+ can also be also used to extract generation templates at a sentence level [25].
As an example, suppose we are interested in defining templates for the Chair when (s)he performs a Pure Questioning. By querying QTMM2012c+, it is possible to extract all the locutions that where uttered by the Chair for which the Illocutionary connection is Pure Questioning.
In the dataset there are 28 locutions that satisfy this query. These locutions can be used to create generation templates. For example, among these locutions we can find the following: • MB : What do you think? • MB : Do you think we have got a lot to apologise for? • Michael : do you think Cameron has a point? • Michael : Or do you think it’s all just demonising the poor? All these examples have something in common, they query someone by using the formula: “(What) do you think”.12 Based on this observation we can define three templates: I) What do you think?, II) Do you think [statement] ? and, III) Do you think [participant name] has a point? These templates can be used in a multi-party argumentative dialogue system when the speaker in the turn is the Chair and the illocutionary connection to be used is Pure Questioning.
As the example shows, QTMM2012c+ can be used to isolate a set of locutions/propositions we are interested in (for example, locutions uttered by the Chair that are Pure Questioning). Once this set of naturally-occurring utterances from the multi-party argumentative dialogue is isolated, several strategies can be used to extract templates. For example, if the set of utterances is small, the template can be manually extracted. If the set of utterances is large, more complex models, for example statistical models, can be used.
In this paper we introduced QTMM2012c+, a queryable resource of annotated real-life debates which captures the links between propositional relations and dialogue act types, speaker roles and speaker stance. We present corpus analyses using the proposed resource and two use cases for the queryable resource. In particular, we suggest a way to use QTMM2012c+ to: i) define rules for multi-party argumentative dialogue and, ii) extract generation templates at a sentence level. The use cases are not fully developed here (this is ongoing work) and primarily aimed at providing two concrete instances of how the corpus can be used in the wider context of AI and argumentation research.
We are sharing QTMM2012c+ with the research community in the hope that its flexibility will allow others to explore natural argumentation in dialogue along the examples illustrated in this paper and beyond.
This work was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) grant EP/T024666/1 ‘Opening Up Minds: Engaging Dialogue Generated from Argument Maps’.
12Both MB and Michael refer to Michael Buerk who is Chair of all Moral Maze episodes in the MM2012c dataset. We would like to acknowledge discussions with and feedback from our project partners at Cambridge (Andreas Vlachos and Youmna Farag) and Shefield (Tom Staford and Lotty Brand) that informed the work reported in this paper. [11] A. Hunter, Towards a framework for computational persuasion with applications in behaviour change, Argument & Computation 9 (2018) 15–40. [12] J. Lawrence, C. Reed, AIFdb corpora., in: Proceedings of the Computational Models of
Argument (COMMA) Conference, 2014, pp. 465–466. [13] J. Lawrence, M. Janier, C. Reed, Working with open argument corpora, in: European
Conference on Argumentation (ECA), 2015. [14] M. Janier, Dialogical dynamics and argumentative structures in dispute mediation discourse,
Ph.D. thesis, University of Dundee, 2017. [15] K. Budzynska, M. Janier, J. Kang, C. Reed, P. Saint-Dizier, M. Stede, O. Yaskorska, Towards argument mining from dialogue., in: Proceedings of the Computational Models of Argument (COMMA) Conference, 2014, pp. 185–196. [16] K. Budzynska, M. Janier, J. Kang, B. Konat, C. Reed, P. Saint-Dizier, M. Stede, O. Yaskorska, Automatically identifying transitions between locutions in dialogue, in: Proceedings of 1st European Conference on Argumentation (ECA 2015), 2015. [17] K. Budzynska, M. Janier, C. Reed, P. Saint-Dizier, M. Stede, O. Yaskorska, A model for processing illocutionary structures and argumentation in debates, in: LREC 2014: Ninth International Conference on Language Resources and Evaluation, European Language Resources Association, 2014, pp. 917–924. [18] K. Budzynska, M. Janier, C. Reed, P. Saint-Dizier, Theoretical foundations for illocutionary structure parsing 1, Argument & Computation 7 (2016) 91–108. [19] O. Yaskorska-Shah, Managing the complexity of dialogues in context: A data-driven discovery method for dialectical reply structures, Argumentation (2021) 1–30. [20] S. Stoyanchev, P. Piwek, Harvesting re-usable high-level rules for expository dialogue generation, in: Proceedings of the 6th International Natural Language Generation Conference, Association for Computational Linguistics, Trim, Co. Meath, Ireland, 2010. URL: https://www.aclweb.org/anthology/W10-4215. [21] C. Reed, K. Budzynska, How dialogues create arguments, in: Proceedings of the 7th conference on argumentation of the International Society for the Study of Argumentation, 2011, pp. 1633–1645. [22] J. R. Searle, Speech acts: An essay in the philosophy of language, Cambridge University
Press, 1969. [23] S. Wells, Moral maze transcripts. figshare. dataset., 2013. URL: https://doi.org/10.6084/m9.
figshare.741242.v1. [24] W. C. Mann, S. A. Thompson, Rhetorical structure theory: Toward a functional theory of text organization, Text - Interdisciplinary Journal for the Study of Discourse 8 (1988) 243–281. URL: https://doi.org/10.1515/text.1.1988.8.3.243. [25] K. v. Deemter, M. Theune, E. Krahmer, Real versus template-based natural language generation: A false opposition?, Computational linguistics 31 (2005) 15–24.
’ID’, ’ID map’, ’Dataset’, ’Reported Speech’, ’Speakers’, ’Degree’, ’ICTA’, ’PropRel’, ’L1’, ’L1 IC’, ’L1 Role’, ’L1 Stance’, ’L1 ID dialogue flow’, ’L1 ID’, ’L1Reported Speech’, ’L1Reported Speech IC’, ’L1Reported Speech Role’, ’L1Reported Speech Stance’, ’L1Reported Speech ID’, ’L2’, ’L2 IC’,