Towards Automatic Minuting of Meetings Anna Nedoluzhko and Ondřej Bojar Charles University, Institute of Formal and Applied Linguistics {nedoluzko, bojar}@ufal.mff.cuni.cz Abstract: Many meetings of different kinds will poten- as orientation in methods which can be used for meeting tially benefit from technological support like automatic summarization. creation of meeting minutes. To prepare a reasonable au- In this paper, we prepare the theoretical and descrip- tomation, we need to have a detailed understanding of tive basis for automatic creation of minutes. Our main common types of meetings, of the linguistic properties objective is to suggest a reasonable classification of meet- and commonalities in the structure of meeting minutes, ings (Section 2.1), meeting minutes (Section 2.2), avail- as well as of methods for their automation. In this pa- able meeting datasets (Section 3) and methods that can be per, we summarize the quality criteria and linguistic prop- used for meeting dialogue summarization (Section 4). In erties of meeting minutes, describe the available meeting Section 5, we summarize the obtained knowledge with re- corpora and meeting datasets and propose a classification spect to our goal of designing automatic minuting and in of meetings and minutes types. Furthermore, we analyze Section 6 we present our first steps in this direction. the methods and tools for automatic minuting with respect to their use with existing types of datasets. We summarize the obtained knowledge with respect to our goal of design- 2 Meetings and Minutes Description ing automatic minuting and present our first steps in this 2.1 Types of Meetings direction. There are many different kinds of meetings carried out for different purposes. Every meeting is unique, but there 1 Introduction are some common types of meetings, which can be dis- tinguished according to aspects such as primary meeting Meeting minutes keep a record of what was discussed at goals, key participant roles or common challenges. the meeting. Meeting minutes is a written document used Different studies of meetings have been conducted by to inform participants and non-participants of what hap- anthropologists, psychologists, sociologists, political sci- pened during the meeting. The problem with meeting min- entists or business administrators (see e.g. [43, 10, 22, 14] utes is that it takes much time to write them down properly. and plenty of others). The handbooks about meeting or- Considering different kinds of meetings and minutes, we ganization begin with Robert’s Rules of Order [41], which can observe that most of meetings will potentially bene- were first published in 1876. In the last edition [42], meet- fit from technological support like automatic creation of ings are classified according to the timing and regularity meeting minutes. (into regular, special, adjourned and annual) and accord- We suggest to develop automatic minuting, i.e. the sum- ing to confidentiality (into executive and public sessions). marization of dialogue transcripts into a compact struc- Moreover, electronic meetings are distinguished as a spe- tured form. cial type. Taking into account the wide variety of real meetings, The classification may be also based on other aspects. we believe that the most effective way is to structure min- For example, authors in [6] speak about formal and infor- utes according to a meeting agenda, which is generally mal meetings, each type having specific linguistic features, prepared manually by the organizers before the meeting. such as different pronominal choice or modes of personal The audio recordings of the meetings can be transcribed address (choosing between ‘I’ and ‘we’, ‘you’ and ‘the using speech recognition techniques. Then the foreseen people’ and so on). Attorney [1] also distinguishes so- minuting software will automatically recognize and ex- called paper meetings with minutes, when participants in- tract important information from meeting transcripts and formally agree on specific corporate actions and minutes classify it to the pre-defined “slots” in the agenda (such as, are prepared as though the decision were approved at a for example, “annotation strategy”, “conference in Paris”, real meeting. The language aspects of meetings, search “next meeting timing”). This is a very complex task that for coherence and sense-making are analysed in [6], where requires a thorough understanding of meetings structure a cross-linguistic and cross-cultural comparison of Ital- with respect to what kinds of minutes they have, as well ian and British meetings has been provided. The authors point out the specificity of non-native multi-party meet- Copyright c 2019 for this paper by its authors. Use permitted un- ings, where substantial parts of communication may be der Creative Commons License Attribution 4.0 International (CC BY devoted to metalanguage details. The form of meetings— 4.0). and minutes respectively—is related to cultural concepts. Moreover, they present a cross-linguistic investigation of One of the main issues about the meeting minutes is that such pragmatic phenomena as for example, the pronomi- they should contain mainly a record of what was done at nal choice or modes of personal reference. the meeting, not what was said by the members [42]. Also, The number of meeting types in online resources1 varies the minutes are not the right place for future action items from four to sixteen. Most lists include five meeting or to-do lists. types and choose within business meetings, decision mak- Structure of minutes. Regarding the organization in- ing meetings, information sharing meetings, status update formation in the minutes, Rigley [40] distinguishes notes meetings, planning meetings, innovation meetings, prob- of meeting by which each action proposed or reported is lem solving meetings, team-building meetings, workshops laid down by a numbering or bullet system, more detailed or conferences. What happens in reality, however, is that narrative minutes written as text, resolution minutes where meetings can actually fit into several of these categories. only decisions are recorded and action minutes which dis- The analysis of existing classifications and real meet- tribute responsibilities between participants and are usu- ings which are available to the authors reveals many fac- ally written in two or three columns. tors which may affect how the meeting is organized, and, Minutes can be also categorized as agenda-based and in- subsequently, which kind of agendas and minutes it needs. formal meeting minutes, which summarize decisions taken Such factors are, e.g., the intention of the meeting, its for- and follow-up actions and responsibilities, but do not nec- mat, the size of the group, regularity, information density, essarily contain all kinds of information prescribed for the content and context of the meeting, participation styles, official minutes. Another possibility of minutes classifica- the expected audience, etc. However, it appears that such tion is structuring into three categories: expressed ideas, factors as meeting content, location, face-to-face vs. re- achieved conclusions and next steps. mote or even the group size do not really affect the core Linguistic features of minutes. As far as we have ob- goals and format of the meetings. served, there are no precise linguistic restrictions to meet- ing minutes. However, minutes are supposed to ensure brevity and clarity, so that they are easy to read. There- 2.2 Meeting Minutes fore, the corresponding websites recommend to write min- Meeting minutes are recorded in many different ways. The utes in so-called basic plain English. The relevant lan- formats can vary according to the personal style of the guage characteristics for the meeting minutes creation are, minutes writer, national, group or domain preferences, ac- writing in the same tense throughout the minutes, using cording to the degree of meetings’ formality, regularity, the simplest words that are appropriate, and avoiding jar- length and so on. Moreover, the format, style and content gon and legalese, using verbs rather than abstract nouns requirements for meeting minutes may vary depending on like “consideration”, “approval” or “clarification”, writing the meeting intention. It means that, for instance, an in- in active rather than passive phrases, not using acronyms, terdepartmental decision making meeting would look very keeping sentences short and using bullet points and num- differently from an informal idea generation meeting of bered lists where appropriate. Furthermore, especially for close colleagues. Some note takers use standard templates the minutes, it is recommended to (1) avoid inflammatory for recording minutes, which are offered by websites2 or or personal observations, (2) use as few adjectives or ad- suggested in [1, 2], etc. In this section, we present the verbs as possible and (3) avoid using people’s names. description and a rough classification of meeting minutes. What minutes definitely include. According to a va- 3 Available Datasets for Automatic riety of handbooks like [1, 2] or [42], the minutes should always include: (i) the name of the organization (ii) date, Minuting time and location of the meeting, (iii) a list of the atten- To create reliable automatic minuting, we need to have dees. The official meetings should also contain the sig- some training and test data of meetings and minutes. How- nature of the chairperson. Robert’s Rules of Order [42] ever, there is a significant disproportion between the num- also suggest some other official requirements such as a ber and domain variety of real meetings and available open statement confirming that the organization’s regular pre- datasets which can be used for this purpose. Meetings siding officer and secretary are present and mentioning of are being held all over the world thousands times a day, whether the previous meeting’s minutes were read and ap- but we can hardly use them, because the transcripts and proved. The informative part of the minutes depends on minutes are mostly publicly unavailable. The exceptions the content of the meeting. Generally, it contains an indi- are mostly in the political domain, because politicians are cation of the content under discussion, what needs to be obliged to make their meetings open to the public. For this done as a result of the meeting, decisions made during the reason, our non-political datasets will be relatively small. meeting and voting results. The brief description of them is given below. The AMI Meeting corpus3 [28] contains 100 hours 1 See, for example, https://blog.firstagenda.com/ of meeting discussions, two thirds of which are scenario the-4-most-important-types-of-meetings. 2 https://lessmeeting.com/ 3 http://groups.inf.ed.ac.uk/ami/corpus/ Table 1: Summary of English meeting corpora. Dataset Length Transcripts Minutes AMI Corpus (scenario and non-scenario) 100h (70h+30h) manual partly ICSI Corpus 70 h manual no NIST Meeting Corpus 20 h manual no ISL Corpus 10 h ASR no meetings which had been played (acted out) for creating about the scenario meetings in the AMI corpus (ca. 70 the corpus. The AMI corpus contains both audio and video hours9 ). The participants of these meetings play the roles signals and text transcripts. It also contains a wide range of employees in an electronics company developing a new of annotations such as dialogue acts and topic segmenta- type of television remote control. These are regular face- tion, named entities, extractive and abstractive summaries to-face decision making meetings which have clear goals and text minutes, which are extremely helpful for our pur- and the organized structure. The meetings follow the pre- poses. prepared agenda and the manual hand-written minutes are The meetings contained in the ICSI corpus4 [23] are created after the meeting. Among other collected meetings for the most part regular meetings of computer science (including the corpus which is being created by the authors working teams. The corpus contains 70 hours of record- of this paper) business and decision making meetings sig- ings in English (for 75 meetings collected in Berkeley dur- nificantly prevail. The meetings however differ in the for- ing the years 2000-2002). The speech files range in length mality (political meetings are much more formal than the from 17 to 103 minutes and involve from 3 to 10 partici- sessions of small business teams). Some of the meetings pants. Interestingly, the corpus contains a significant pro- can be classified as information sharing or status update portion of non-native English speakers, varying in fluency meetings, the minority can be considered as planning or from nearly-native to challenging-to-transcribe. All audio problem solving ones. The collected data does not include files are manually transcribed. workshops or conferences, nor any team-building meet- The NIST Meeting Room Corpus [30] contains 20 ings. Almost all meetings are multi-party, and non-native hours of meeting recordings (in English) in a room English speakers prevail in the data. equipped with five cameras. Its analysis in [24] sug- gests that detected peaks (conversation overlaps, and other changes at different temporal scales) can be useful in sum- 4 Methods for Meeting Summarization marization and indexing of meetings. The ISL Corpus5 [9] contains 10 hours of recordings The methodologies that are applied to solve meeting sum- (19 meetings) linked to transcripts from meetings con- marization problems are numerous and can be seen from ducted in a special conference room in the Language Tech- different perspectives. In what follows, we observe them nologies Institute at Carnegie Mellon University. from three aspects: focused – unfocused (Section 4.1), ex- Finally, there is a large number of Parliament and other tractive – abstractive (Section 4.2) and supervised – unsu- available political meetings in the official meeting and pervised (Section 4.3). minutes of European and UK parliaments, Agriculture Di- alogue Groups6 etc. With some data processing, they can 4.1 Decision-Focused Meeting Summarization be transformed into valuable datasets. There are also data available in other languages, for example, for Czech, there One aspect of meeting or dialogue summaries is their fo- are more than one thousand hours of meetings with avail- cus. If there is no focus or specific objective the system able transcripts and minutes from the Czech Parliament7 will try to collect the most relevant utterances to create and Prague City Hall meetings.8 the summary. Otherwise, if there is a special interest in A summary of English meeting corpora can be found in certain parts like proposed ideas, supporting arguments or Table 1. decisions, the system will try to first identify those parts of The typology of the meetings which are present in the text that form types of utterance and then create the sum- referred corpora is not addressed in their descriptions very mary. thoroughly. The most detailed information can be found Literature observations reveal that the interest of most of the focused dialogue summarization works is in the de- 4 http://groups.inf.ed.ac.uk/ami/icsi/ cisions. This emphasizes the fact that decisions are the 5 https://ca.talkbank.org/access/ISL.html 6 https://ec.europa.eu/agriculture/ most essential outputs of meetings, which has indeed been civil-dialogue-groups_en outlined in several works like [5] or [25]. Authors in [16] 7 http://public.psp.cz/en/sqw/hp.sqw 8 http://www.praha.eu 9 http://groups.inf.ed.ac.uk/ami/corpus/ model dialogue structure to automatically detect decisions Authors in [34] focus their entire efforts in the in multi-party meetings. They label each utterance accord- term weighting part which is essential for some of the ing to the role it plays in decision-making by using SVM most important extractive summarization schemes like (Support Vector Machines) classifier in a hierarchical way. MMR (Maximal Marginal Relevance). They report that First, sub-classifiers are trained and used to detect the their novel weighting metric (SU · IDF) outperforms class (e.g., issue, resolution or agreement) of each DDA T F · IDF. (Decision Dialogue Act). Later, a super-classifier is uti- In [35] we find another extractive approach that tries to lized to identify decision sub-dialogues. In [15] (follow-up overcome speech recognition errors in meeting transcripts. work of the above authors) they parse decision-related di- They try MMR and LSA contrasting them with supervised alogue utterances using Gemeini, an open-domain parser feature-based approaches that use lexical and prosodic fea- described in [13]. The candidate phrases that are gener- tures. They conclude that the feature-based approaches ated are further analysed using SVM which filters those perform worse because of the difficulty to find the best that would likely fit in the summary of the discussion. feature collections. Authors of [8] solve the problem in two steps. First, A different extractive approach is the one in [21] where they distinguish between dialogue acts that describe the the semantic similarity measures of utterances and the issue and those that describe its resolution. For this, they whole dialogues are compared to find out which of the make use of DGMs (Directed Graphical Models) which utterances carries important and relevant content for the according to their experiments outperform hierarchical summary. WordNet is used as a knowledge base for the SVMs when non-lexical features are used. In the sec- semantic computations. ond step, they extract words/phrases from the issue and An extractive and fully feature-based approach is the resolution utterances by analysing the later with a seman- work in [32] where authors present a set of generic conver- tic parser. The several candidates the parser produces are sational features for locating the most relevant sentences later processed with a SVM regressor which selects the for the summary. They use linear SVM as classifier and best. The SVM regression model is enriched with a pow- report that their approach is portable in various conversa- erful semantic-similarity feature computed using WordNet tional texts like meetings, emails, etc. as knowledge source. Interactive systems with user feedback such as [31] have In [48] we find an unsupervised framework that consid- also been proposed. The summarizer of this system is con- ers meeting dialogue summarization as an information ex- ceived as an agent that learns to better identify which rele- traction task. The authors adapt the relation learner of [11] vant utterances to extract by interacting with the user. The with new features and use it to identify relations between advantages it offers are adaptability in different domains decision cues and decision content of dialogue acts. The and the possibility to work even when a small initial source content output is a set of indicator-argument decision re- of data is available. lations that form the basis of the decision summary. They There are also studies that utilize both extractive and ab- show that this approach outperforms unsupervised extrac- stractive or neither extractive nor abstractive summariza- tive summarization methods and is highly promising. tion. In [47] we find a complex framework that starts by In [49] authors propose a domain-independent sum- clustering all decision-related dialogue acts (DAs). This mary generation framework. They first perform content creates certain clusters for each decision that was made. selection using a classifier which identifies potential sum- They later perform DA-level summarization by selecting mary phrases. Next, they employ an overgenerate-and- the most important DAs from each cluster and join them rank strategy to produce and rank candidate summary sen- to form a preliminary summary. SVM and LDA are used tences. The redundancy reduction process outputs the full to further compress at a token-level. Finally, they add dis- meeting summary. Their evaluation reveals that the pro- course context by augmenting the DA clusters of each de- posed system outperforms the state-of-the-art supervised cision with non-decision related DAs from the dialogue. extraction-based methods. This way the summary is more abstractive, which makes it concise and readable. Another work that combines extractive and abstractive 4.2 Extractive and Abstractive Summarization approaches for better meeting summaries is [36]. The Strategies authors start from human annotated extractive summaries and apply sentence compression to improve the readabil- The utilized text summarization strategy is another way of ity. Different sentence compression methods like integer looking at meeting summarization research works. Sum- programming [12] or a filler phase detection module and marization methods are generally either extractive or ab- the lexicalized Markov grammar-based approach [19] are stractive. Extractive methods only select suitable parts explored. Their results indicate that sentence compression (sentences, words or phrases) from the document or the is promising for producing abstractive summaries. Sim- transcript, while abstractive methods can produce an ar- ilarly, authors in [18] start by finding the most valuable bitrary text as the summary. Pure extractive approaches features for identifying and extracting the most informa- seem very common in the literature. tive and relevant DAs. In the second step, they try to in- crease the abstraction degree of the extractive summaries over human-annotated extractive ones. by including “meta” DAs in which the speakers refer to the meeting itself. They conclude that the “meta” DAs are 4.3 Supervised and Unsupervised Summarization indeed very helpful and create more coherent and informa- Methods tive meeting summaries. Authors in [33] compare extractive and abstractive di- Data-driven machine learning models (supervised, unsu- alogue summaries from a user (reader) perspective and pervised, both, etc.) are widespread today, even in stud- argue that abstractive and concise summaries are usually ies about summarization of meeting dialogues. The type favored over extractive ones. According to them, a weak- of machine learning approach they utilize is another way ness of extractive summaries is that the user does not un- to look at these studies. It is typical to find unsupervised derstand why the extracted phrases are important. They methods (clustering) in the initial step of a pipeline or build a summarizer which first maps sentences to a con- complex system. Typical examples of this category are versation ontology of decisions, action items, sentiments [39], [27], and [47]. Other frequent forms of unsupervised etc. It later identifies message patterns that abstract over approaches are MMR and LSA which are based on sim- several sentences and aggregate them to produce the sum- ilarity scores or the dependency graph of [3]. It is also mary. Authors conduct a user survey which reveals that interesting to find recent works that are unsupervised (no their automatic summaries are better than the pure extrac- need for labeled data), but still produce gramatically cor- tive ones. rect summaries. One such example is [46] where they Going in this direction (from extractive to abstractive combine the strengths of various graph-based methods like summaries) some researchers have created fully abstrac- the neural network sentence compression of [17], graph tive systems. They were mostly inspired by similar de- path reranking of [7], graph entailment of [29], etc. Au- velopments in close tasks such as text summarization of thors evaluate on both AMI and ICSI datasets and report news articles where the power the encoder-decoder frame- state-of-the-art results. work based on RNNs is utilized [44, 37, 45]. In [4] they Supervised learning as a part of the system is even more first split meeting dialogues into several topic segments. common. SVM is clearly the most frequent algorithm fol- The most important phrases in each segment are identi- lowed by Naïve Bayes and maximum entropy classifier. fied using a classifier and merged to form a one-sentence There are even studies like [26] and [16] that perform hi- summary. The dependency parses of each segment are erarchical classification, with sub-classifiers that identify combined to form a directed graph. ILP (Integer Lin- categories of different utterances and a super-classifier that ear Programming) is used to select the most informative produces the final summary. There are also studies like sub-graph and produce the one-sentence summary of each [8] where both unsupervised (directed graph and seman- topic segment, reaching to the summary of the entire meet- tic similarity measures) and supervised (SVM) are com- ing. bined together. Finally, among the most recent super- An even more complete pipeline is presented in [29] vised approaches based on neural networks, we can men- where they cluster the sentences and create an entailment tion [38] which fused verbal and non-verbal information graph which selects the most relevant sentences in each to predict the importance of each uterance. Authors utilize cluster. They further build a word-graph model by ex- MATRICS multimodal discussion corpus dataset which tending that of [17] and use a ranking strategy to select contains group discussions with various annotations and the best paths in it, compressing and aggregating the se- features (speech spectrogram, head motion spectrogram, lected sentences. Authors report that their approach is able head pose, and more). At the end, they use a multi-channel to generate long sentences with little loss in grammatical- neural network architecture based on CNNs and dense lay- ity. In [20] we find another attempt to improve abstractive ers to fuse together all types of features and predict the summarization of dialogues, this time by integrating inter- importance of the utterances. active parts into the summary. They propose a sentence- gated mechanism which models the relationships between the dialogue acts and the summary. Their benchmarks 5 Discussion with AMI meeting corpus reveal that the system outper- forms the other models. Let us now summarize the knowledge obtained by the sur- A different approach for improving abstractive meeting vey and describe our first steps towards the creation of au- summaries is the one in [39] where templates are learned tomatic minuting. from human-authored summaries. A clustering sentence The types of existing meetings, minutes and datasets fusion algorithm and WordNet semantic similarities be- show a significant disproportion between the real meetings tween words are used to generate templates. The meet- and the datasets which are available for the research. The ing transcripts are segmented based on topics and the best available data are mostly in the political domain, whereas, templates for each topic are selected using the relation- in reality, business meetings prevail and these are also the ship between the human summaries and their sources. The very meetings for which the automation of minutes would evaluation shows that their system summaries are favored bring the most benefit. For these reasons, we decided to go beyond the political domain and focus on other types of which have been played by the actors. This fact may have meetings as well, first of all on international online meet- significant effect on what the people say and how they act ings. Our goal is thus to arrange the meeting types in a in the conversation. Therefore, we decided to extend it way which explains the types of agendas and minutes ap- with our own data. plicable to them. From this point of view, we consider the Within our project ELITR11 , we started collecting meet- meeting intention as the most appropriate scale in the mul- ings of our computer science working teams. The data in- tidimensional space of meeting types. Thus, business and cludes the audio recordings, ASR transcripts, pre-prepared decision making meetings are most structured and they re- agendas and meeting minutes created by the meeting orga- quire the most clear, structured and detailed agenda. The nizers or a secretary after the meeting. For the time being, minutes are supposed to contain a list of decisions. For in- we obtained ca. 40 hours of meetings in English (of mostly formation sharing and status update meetings, the agenda non-native speakers). The corpus is under development. is also extremely important. In this case, the minutes will be a refinement of ideas given in the agenda. The situ- ation is slightly different for planning and problem solv- 7 Conclusion ing meetings, as a number of new ideas may arise during the meeting. Innovation and idea generation meetings are In this paper, we laid out foundations for research into creative and can include a lot of irrelevant brainstorming. automatic minuting of meetings. Our main goal was to During the note-taking it may not be clear what will be prepare the floor for automatic minuting by analyzing the important in the result. The minutes do not follow the sources which help to make this idea realizable. By com- agenda consistently. Team-building meetings and other paring a variety of meetings and their descriptions, we social events follow different rules, and minutes (if any) tried to get a reasonable typology of meetings, summa- have rather different functions. Naturally, we focus on the rized the types of possible minutes, described the meetings meetings which demand for agenda and minutes. For this datasets and made a survey of methods of meeting summa- reason, for example, team-building meetings will not be rization. We also drafted our first steps to the creation of included in our research. the corpus of meetings and minutes which will be further Concerning the form of automatic minutes, we incline used for developing automatic minuting. to create structured automatic notes of meetings by which the actions are fixed by the bullet system rather than de- Acknowledgement tailed narrative minutes. According to the survey of meet- ing minutes presented in Section 2.2, for some meetings, This work has been in part supported by the project no. 19- special fields for actions or resolutions may be applied. As 26934X (NEUREM3) of the Czech Science Foundation for linguistic form of the minutes, it will be defined by the and ELITR (H2020-ICT-2018-2-825460) of the EU. meetings themselves, as we will primarily use extractive We are grateful to Erion Cano for his help with this ar- summarization methods. ticle. As for the datasets, we are primarily interested in the meeting corpora which include both meeting transcripts and minutes (or other types of summarizations). Among References the datasets described in Section 3, this is the AMI cor- pus, which will be used in our experiments as first. Other [1] A.M. Attorney. The Corporate Records Handbook: Meet- datasets with the minutes are parlamental texts in the po- ings, Minutes and Resolutions. dElta printing solutions, litical domain. inc., 2007. As most international online meetings, which we avail- [2] A.M. Attorney. Nonprofit meetings, minutes and records. able to us, are held in English, we choose English as Delta printing solutions, inc., 2008. the main language for creating automatic minuting exper- [3] S. Banerjee, P. Mitra, and K. Sugiyama. Abstractive meet- iments. However, we plan to include other languages as ing summarization using dependency graph fusion. In Pro- well.10 ceedings of the 24th International Conference on World Wide Web, WWW ’15 Companion, pages 5–6, New York, NY, USA, 2015. ACM. 6 First Steps Towards Automatic Minuting [4] S. Banerjee, P. Mitra, and K. Sugiyama. Generating ab- stractive summaries from meeting transcripts. In Proceed- The available meeting corpora such as those described in ings of the 2015 ACM Symposium on Document Engineer- Section 3 can serve as a good starting point for making ex- ing, DocEng ’15, pages 51–60, New York, NY, USA, 2015. ACM. periments in automatic minuting, but they do not suffice. For example, the AMI corpus (which is the most appropri- [5] S. Banerjee, C. Rose, and A.I. Rudnicky. The necessity of a meeting recording and playback system, and the benefit ate for us) is relatively small and mostly includes meetings of topic–level annotations to meeting browsing. In M.F. 10 For example, we began to collect the meeting corpus for Czech, but, for the time being, its size is insignificant. 11 http://elitr.eu/ Costabile and F. Paternò, editors, Human-Computer Inter- Language Processing, pages 1133–1141. Association for action - INTERACT 2005, pages 643–656, Berlin, Heidel- Computational Linguistics, 2009. berg, 2005. Springer Berlin Heidelberg. [19] M. Galley and K. McKeown. Lexicalized Markov gram- [6] F. Bargiela and S.J. Harris. Managing Language: The dis- mars for sentence compression. In Human Language Tech- course of corporate meetings. John Benjamins Publishing nologies 2007: The Conference of the North American Company, 1997. Chapter of the Association for Computational Linguis- [7] F. Boudin and E. Morin. Keyphrase extraction for n-best tics; Proceedings of the Main Conference, pages 180–187, reranking in multi-sentence compression. In Proceedings Rochester, New York, April 2007. Association for Compu- of the 2013 Conference of the North American Chapter tational Linguistics. of the Association for Computational Linguistics: Human [20] Ch. Goo and Y. Chen. Abstractive dialogue summarization Language Technologies, pages 298–305, Atlanta, Georgia, with sentence-gated modeling optimized by dialogue acts. June 2013. Association for Computational Linguistics. 2018 IEEE Spoken Language Technology Workshop (SLT), [8] T. Bui, M. Frampton, J. Dowding, and S. Peters. Extracting pages 735–742, 2018. decisions from multi-party dialogue using directed graphi- [21] I. Gurevych and M. Strube. Semantic similarity applied cal models and semantic similarity. In Proceedings of the to spoken dialogue summarization. In Proceedings of the SIGDIAL 2009 Conference, pages 235–243. Association 20th International Conference on Computational Linguis- for Computational Linguistics, 2009. tics, COLING ’04, Stroudsburg, PA, USA, 2004. Associa- [9] Susanne Burger, Victoria MacLaren, and Hua Yu. The isl tion for Computational Linguistics. meeting corpus: The impact of meeting type on speech [22] K.M. Hoffman. Meeting design: For Managers, Makers style. 01 2002. and Everyone. Louis Rosenfeld, 2018. [10] S.E. Butler. Mission Critical Meetings: 81 Practical Facil- [23] A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, itation Techniques. Wheatmark, 2014. N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, [11] H. Chen, E. Benson, T. Naseem, and R. Barzilay. In- and C. Wooters. The icsi meeting corpus. pages 364–367, domain relation discovery with meta-constraints via pos- 2003. terior regularization. In Proceedings of the 49th Annual [24] R. Krishnan and S. Sarkar. Detecting Group Turn Patterns Meeting of the Association for Computational Linguistics: in Conversations Using Audio-Video Change Scale-Space. Human Language Technologies, pages 530–540, Portland, In 20th International Conference on Pattern Recognition, Oregon, USA, jun 2011. Association for Computational ICPR 2010, Istanbul, Turkey, 23-26 August 2010, pages Linguistics. 137–140, 2010. [12] J. Clarke. Global inference for sentence compression: An [25] A. Lisowska, A. Popescu-Belis, and S. Armstrong. User integer linear programming approach. J. Artif. Intell. Res., Query Analysis for the Specification and Evaluation of a 31:399–429, 2008. Dialogue Processing and Retrieval System, pages 993–996. [13] J. Dowding, J.M. Gawron, D. Appelt, J. Bear, L. Cherny, LREC 2004 (Fourth International Conference on Language R. Moore, and D. Moran. Gemini: A natural language sys- Resources and Evaluation). ELRA - European Language tem for spoken-language understanding. In HUMAN LAN- Ressources Association, 2004. ID: unige:2264. GUAGE TECHNOLOGY: Proceedings of a Workshop Held [26] F. Liu and Y. Liu. From extractive to abstractive meeting at Plainsboro, New Jersey, March 21-24, 1993, 1993. summaries: Can it be done by sentence compression? In [14] L. Espy. Bad Meetings Happen to Good People: How to Proceedings of the ACL-IJCNLP 2009 Conference Short Run Meetings That Are Effective, Focused, and Produce Papers, ACLShort ’09, pages 261–264, Stroudsburg, PA, Results. Blue Room Press, Memphis, Tenessee, 2017. USA, 2009. Association for Computational Linguistics. [15] R. Fernández, M. Frampton, J. Dowding, A. Adukuzhiyil, [27] Y. Liu, S. Xie, and F. Liu. Using n-best recognition out- P. Ehlen, and S. Peters. Identifying relevant phrases to sum- put for extractive summarization and keyword extraction in marize decisions in spoken meetings. In Ninth Annual Con- meeting speech. In 2010 IEEE International Conference ference of the International Speech Communication Asso- on Acoustics, Speech and Signal Processing, pages 5310– ciation, 2008. 5313, March 2010. [16] R. Fernández, M. Frampton, P. Ehlen, M. Purver, and S. Pe- [28] I. Mccowan, G. Lathoud, M. Lincoln, A. Lisowska, ters. Modelling and detecting decisions in multi-party W. Post, D. Reidsma, and P. Wellner. The ami meeting dialogue. In Proceedings of the 9th SIGdial Workshop corpus. In In: Proceedings Measuring Behavior 2005, on Discourse and Dialogue, SIGdial ’08, pages 156–163, 5th International Conference on Methods and Techniques Stroudsburg, PA, USA, 2008. Association for Computa- in Behavioral Research. L.P.J.J. Noldus, F. Grieco, L.W.S. tional Linguistics. Loijens and P.H. Zimmerman (Eds.), Wageningen: Noldus [17] K. Filippova. Multi-sentence compression: Finding short- Information Technology, 2005. est paths in word graphs. In Proceedings of the 23rd Inter- [29] Y. Mehdad, G. Carenini, F. Tompa, and Raymond T. NG. national Conference on Computational Linguistics, COL- Abstractive meeting summarization with entailment and fu- ING ’10, pages 322–330, Stroudsburg, PA, USA, 2010. As- sion. In Proceedings of the 14th European Workshop on sociation for Computational Linguistics. Natural Language Generation, pages 136–146. Associa- [18] M. Frampton, J. Huang, T. Bui, and S. Peters. Real-time tion for Computational Linguistics, 2013. decision detection in multi-party dialogue. In Proceedings [30] M. Michel, J. Ajot, and J.G. Fiscus. The NIST Meeting of the 2009 Conference on Empirical Methods in Natural Room Corpus 2 Phase 1. In Machine Learning for Mul- timodal Interaction, Third International Workshop, MLMI guistics. 2006, Bethesda, MD, USA, May 1-4, 2006, Revised Se- [45] A. See, P.J. Liu, and C. D. Manning. Get to the point: lected Papers, pages 13–23, 2006. Summarization with pointer-generator networks. CoRR, [31] M. Mieskes, Ch. Müller, and M. Strube. Improving extrac- abs/1704.04368, 2017. tive dialogue summarization by utilizing human feedback. [46] G. Shang, W. Ding, Z. Zhang, A. Tixier, P. Meladi- In Artificial Intelligence and Applications, 2007. anos, M. Vazirgiannis, and J.-P. Lorré. Unsupervised ab- [32] G. Murray and G. Carenini. Summarizing spoken and writ- stractive meeting summarization with multi-sentence com- ten conversations. In Proceedings of the Conference on Em- pression and budgeted submodular maximization. CoRR, pirical Methods in Natural Language Processing, EMNLP abs/1805.05271, 2018. ’08, pages 773–782, Stroudsburg, PA, USA, 2008. Associ- [47] L. Wang and C. Cardie. Summarizing decisions in spo- ation for Computational Linguistics. ken meetings. In Proceedings of the Workshop on Au- [33] G. Murray, G. Carenini, and R. Ng. Generating and tomatic Summarization for Different Genres, Media, and validating abstracts of meeting conversations: A user Languages, WASDGML ’11, pages 16–24, Stroudsburg, study. In Proceedings of the 6th International Natural Lan- PA, USA, 2011. Association for Computational Linguis- guage Generation Conference, INLG ’10, pages 105–113, tics. Stroudsburg, PA, USA, 2010. Association for Computa- [48] L. Wang and C. Cardie. Focused meeting summarization tional Linguistics. via unsupervised relation extraction. In Proceedings of [34] G. Murray and S. Renals. Term-weighting for summariza- the 13th Annual Meeting of the Special Interest Group on tion of multi-party spoken dialogues. In Proceedings of Discourse and Dialogue, SIGDIAL ’12, pages 304–313, the 4th International Conference on Machine Learning for Stroudsburg, PA, USA, 2012. Association for Computa- Multimodal Interaction, MLMI’07, pages 156–167, Berlin, tional Linguistics. Heidelberg, 2008. Springer-Verlag. [49] L. Wang and C. Cardie. Domain-independent abstract gen- [35] G. Murray, S. Renals, and J. Carletta. Extractive summa- eration for focused meeting summarization. In Proceedings rization of meeting recordings. In INTERSPEECH 2005 - of the 51st Annual Meeting of the Association for Compu- Eurospeech, 9th European Conference on Speech Commu- tational Linguistics (Volume 1: Long Papers), pages 1395– nication and Technology, Lisbon, Portugal, September 4-8, 1405. Association for Computational Linguistics, 2013. 2005, pages 593–596, 2005. [36] G. Murray and Steve Renals. Meta comments for sum- marizing meeting speech. In Andrei Popescu-Belis and Rainer Stiefelhagen, editors, Machine Learning for Mul- timodal Interaction, pages 236–247, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. [37] R. Nallapati, B. Xiang, and B. Zhou. Sequence-to-sequence rnns for text summarization. CoRR, abs/1602.06023, 2016. [38] F. Nihei, Y.I. Nakano, and Y. Takase. Fusing verbal and nonverbal information for extractive meeting summariza- tion. In Proceedings of the Group Interaction Frontiers in Technology, GIFT’18, pages 9:1–9:9, New York, NY, USA, 2018. ACM. [39] T. Oya, Y. Mehdad, G. Carenini, and R. Ng. A template- based abstractive meeting summarization: Leveraging summary and source text relationships. In Proceedings of the 8th International Natural Language Generation Con- ference (INLG), pages 45–53, 2014. [40] T. Rigley. The techniques of writing minutes. Singapore : Federal Publications, 1996. [41] H.M. Robert. Robert’s Rules of Order: The Original Edi- tion. S. C. Griggs and Company, 1876. [42] H.M. Robert, S.C. Robert, H.M. Robert III, W.J. Evans, D.H. Honemann, and T.J. Balch. Robert’s Rules of Order Newly Revised. Da Capo Press, A Member of the Perseus Books Group, 2011. [43] S. Rogelberg. The Surprising Science of Meetings. Oxford University Press, 2019. [44] A. M. Rush, S. Chopra, and J. Weston. A neural attention model for abstractive sentence summarization. In Proceed- ings of the 2015 Conference on Empirical Methods in Nat- ural Language Processing, pages 379–389, Lisbon, Portu- gal, September 2015. Association for Computational Lin-