As the need for efective tools in Corpus Linguistics continues to grow, particularly for under-resourced languages and nonstandard annotation tasks, specialized software has become essential for processing and analyzing large and complex datasets. This paper introduces CorpusCompass, a new open source tool for data extraction and dataset creation, which ofers a number of functionalities for researchers interested in analyzing corpora. The tool can derive structured datasets from text annotated with custom annotation schemes, while also checking for errors and consistency. By defining custom variables of interest and annotation rules, researchers can tailor the tool to their specific needs, making it particularly valuable for unique linguistic research domains. When used in conjunction with statistical analysis or visualization tools, CorpusCompass helps researchers to gain insights into the factors that are afecting language use. In this paper, we introduce the tool and give a real-world example in the field of language variation.
and connections within the data. Lastly, the final step,
(vi), involves publishing and sharing the corpus with the
In recent years, there has been a growing interest in wider research community, promoting collaboration and
studying language variation in under-resourced languages. further research based on the spoken language data.
Mair [
In this context, we identify six main steps for this In this paper, we provide a detailed overview of
Corprocess, each presenting its own set of practical and tech- pusCompass1, including its design, implementation, and
nical challenges, see Figure 1. Step (i) entails sourcing functionalities. The tool is based on Jupyter Notebook and
and recording data, along with the associated metadata, is designed to help Corpus Linguistics researchers
focuswhich provides essential contextual information about ing on language variation to create a structured dataset
the recordings. Following this, step (ii) involves tran- from their previously annotated corpus/corpora and a
scribing the spoken data to convert it into a text-based list of variables of interest (see Section 3.1). The tool is
format, allowing for more straightforward analysis. The coded in Python and can be run in an interactive manner
third step (iii) involves annotating the data with linguistic using Google Colab. Once run, it generates a structured
features, such as phonological or morphological informa- dataset, i.e. a systematically organized collection of data,
tion. Subsequently, step (iv) requires data preprocessing to that includes linguistic variables and potentially relevant
clean and organize the data, preparing it for the fifth step, metadata (see Section 3.2).
(v), which is data analysis. This stage allows researchers On the one hand, the dataset enables corpus
exploto derive insights from the corpus by examining patterns ration, assisting in discovering patterns that can inform
the creation of new hypotheses or the dismissal of initial
assumptions (see Al-Wer et al. [
website https://www.corpuscompass.com/. Please note that the code for CorpusCompass is constantly evolving and, in this paper, we refer to version 1.0.0.
[
To exemplify the practical application of CorpusCom- involving various phonological, morphological and lexipass within a research context, Section 4 explores a case cal variables as well as diferent speakers, as detailed and study on inter-generational linguistic variation among structured extraction is necessary. Sociolinguistic studIraqi speakers living in Germany, and highlights various ies, in particular, require flexibility in handling multiple potential uses of the tool. speakers and their background information. With Cor
For linguists without any or limited programming pusCompass, we aim to address this gap in functionality. skills, CorpusCompass has the potential of saving time WordSmith and Monoconc Pro share these limitations (by automating repetitive tasks) and improving the ac- with AntConc, but they also have an additional drawback: curacy of their research. It is intended to bridge the gap they are not freely available. This lack of open access can between researchers and advanced statistical analysis be a significant barrier for corpus linguistics practitioners by facilitating the connection between them, as well as who may not have the financial resources to purchase addressing research questions that require the use of these tools. manually annotated data. Moreover, CorpusCompass was Other useful modern tools are typically focused on the developed by researchers in Linguistics and Computer annotation process rather than the analysis. For example, Science to advance Corpus Linguistics tools and promote INCEpTION2 [12] is a cloud-based platform that enables interdisciplinary collaboration. researchers to create and share linguistic annotations in a collaborative environment, for various languages.
Another example is FLAT [13], a web-based linguistic 2. Related Work annotation tool that revolves around the FoLiA format, a customizable XML-based format for linguistic annotation.
However, for state-of-the-art analysis and error-checking of annotations produced by such tools, we would prefer to not rely on their built-in capabilities, but instead make use of common data science methodology such as statistical analysis in R or visualization with Gephi. This requires exporting the annotations from the typically XML-like formats that these tools use to a tabular data format used in data science such as CSV or TSV files.
This is a core functionality of CorpusCompass, addressing a gap in the existing tools.
Biber et al. [14], Gries [15], and Weisser [16] suggest that learning programming and developing one’s own analytical tools can overcome limitations in existing corpus tools.
Biber et al. [14] suggest that this would allow corpus linguists to perform faster and more accurate analyses, and the ability to tailor the output to suit the particular research requirements. Furthermore, according to Gries [15], utilising a pre-existing tool lets the researcher become dependent of the company or individual developing them, whereas programming allows them to have control over their research needs. Therefore, corpus lin
ily in the context of Artificial Intelligence and Natural Language Processing. However, such tools typically do not address specific needs, focusing more on the annotation of informational content rather than linguistic properties of text (phonological, grammatical, lexical, etc.). For this reason, linguists typically work with tools that have been developed specifically for the purpose of annotating linguistic corpora. In the following, we will briefly outline the importance of these tools and demonstrate how CorpusCompass complements and extends their functionality.
There are several tools available for researchers in Corpus Linguistics (see Neves and Ševa [6], Berberich and Kleiber [7]). AntConc [8], Monoconc Pro [9], and WordSmith [10] remain popular tools due to their wide range of functions, including KWIC (key word in context) concordancers, collocates, word frequencies, keywords and other corpus analysis features.
AntConc is a powerful corpus analysis tool, but it has certain drawbacks. One major limitation is the lack of a feature to create structured datasets from the corpus or various corpora, which is essential for statistical analysis pipelines as well as to share data. This is particularly guists have clear benefits from learning a programming language, both in terms of the flexibility to develop tools for specialized tasks, as well as providing them with an understanding of the issues faced by tool developers creating general purpose tools.
As seen in the overview above, each tool addresses unique needs, thus implementing specific functionalities. In comparison, CorpusCompass tackles a complementary set of challenges. It is implemented as a Jupyter Notebook and can be run in an interactive manner using Google Colab, which not only makes it more accessible to users but also gives beginners the possibility to get familiar with programming.
CorpusCompass, as it allows researchers to tailor the tool to what they aim to investigate. In the field of Corpus Linguistics, variables are often used to study language variation and how it is afected by various factors such as speaker demographics (e.g. age, sex, education), linguistic context (e.g. dialect, register), social context (e.g. audience, situation), or properties of a construction (e.g. morphemes, idioms). By defining both independent and dependent variables in their structured dataset, researchers have maximum freedom in the exploration of their corpora and the creation of their unique datasets.
tives, researchers may use automated annotation tools The modular code structure of CorpusCompass includes or choose to manually annotate data in more complex a file handling module for managing JSON and CSV linguistic situations (as described in Section 4). This leads ifles, an annotation parsing module that extracts anno- to a broad range of annotation rules. To accommodate tations using regular expressions (refer to Section 3.1), this variety, CorpusCompass employs regular expressions a dataset construction module for creating a structured (regex) [17] for accurate extraction of annotations from CSV dataset, and a logging module that displays rele- the corpus. Regex allows the user to define text patterns, vant information such as program status and annotation and is useful for tasks such as input validation, text moddetails. CorpusCompass provides helpful functions for ifications, and data extraction. string manipulation, data cleaning, and error handling, Figure 2 shows four paragraphs taken from our corsimplifying the data analysis process. These functions pus3, where annotations are highlighted in green. For enhance dataset accuracy by mitigating errors and in- the sake of simplicity, we kept only the dependent variconsistencies during the data preparation phase. In the able that are at the basis of our analysis in Section 4.2. following, we describe the pipeline of CorpusCompass and demonstrate how it simplifies the process of extracting valuable insights from spoken language data.
on pre-selected specific features, as adhering to it for the extensive 22-hour audio recordings would have been time-consuming.
After the variables and regex rules have been defined, researchers can run CorpusCompass to generate a structured dataset. The dataset construction module automatically performs several steps, including cleaning and preprocessing the extracted annotations, grouping them by speaker and file, and writing them to a CSV file. Six output files are created, including five CSVs (dataset, annotation_info, missed_annotations, unk_variables, and binary_dataset) and one JSON file (corpus_stats).
The dataset file is a structured dataset based on the defined variables, with each row corresponding to a token in the annotated corpus and each column representing a variable category. This organized representation of annotations facilitates the analysis. The annotation_info ifle contains information about the annotations included in the dataset lfie, such as the token itself, the number of times it appears in the dataset, and the number of times it appears for each speaker. This information can be useful for identifying patterns in the data, such as the most common tokens or the distribution of annotations across speakers.
The missed_annotations file tracks tokens that were previously annotated but not consistently annotated in subsequent instances4. The file’s purpose is to identify tokens that were once deemed important but not annotated consistently. Furthermore, in projects with multiple annotators, inter-annotator disagreement is a known challenge [18]. During the annotation process, it is possible for researchers to come up with new variables that were not previously specified in the JSON. However, researchers may forget to add these variables to the file, leading to inconsistencies in the dataset. The unk_variables file contains a list of variables that were not specified in the JSON. Finally, the corpus_stats file provides an overview of the corpus by reporting key statistics. Access to these statistics enables researchers to better understand the size and structure of their corpus, and can also provide valuable information for reproducibility purposes.
For a comprehensive description and additional information regarding the CSV files, please see Appendix B.
Using CorpusCompass Following the annotation of the data with linguistic features, we used CorpusCompass for preprocessing in order to prepare the data for the analysis process. This transition from data preprocessing to analysis was facilitated by the integration of the tool into our workflow, significantly enhancing the eficiency and efectiveness of our exploration process.
In our case study, we examine Arabic-speaking communities, specifically Iraqis and Syrians, residing in Germany since 2014, following standard sociolinguistic variationist research practices. The participants are Iraqi and Syrian Arabic native speakers. We select phonological, morphological, and lexical variables for statistical analysis, with age as a key independent variable influencing linguistic variation. The study aims to examine inter-generational diferences within the two groups and explore the extent to which a koiné (common variety) results from dialect and language contact in the migration context between the Syrian and Iraqi participants.
illustrated in Figure 2, represents only half of our entire dataset and has been phonetically transcribed5 using the International Phonetic Alphabet (IPA) and annotated by a single person in Notepad++. While having one annotator can be a common case in the field, mostly due to resource limitation, it can also be prone to errors and inconsistencies. The analyzed corpus is comprised of 2,101 paragraphs and encompasses 114,550 words and 654,431 characters. It features 24 speakers in total, 14 of whom are Iraqis, considered speakers of interest for our analysis. The dataset contains 35 variables, with 25 being dependent variables and 10 being independent variables. These variables are represented by 69 distinct values, 53 of which correspond to dependent variable values, and 16 to independent variable values.
In total, the corpus contains 3,366 unique annotations, with 13,641 annotated tokens. Given the substantial size of the dataset and the numerous variables involved, organizing the data in a structured manner is crucial for eficient analysis.
For comprehensive details regarding the corpus collection process, transcription methodology, annotation procedures, and the specific tool employed, please refer to Appendix C.
does not diferentiate between diferent meanings of the same token.
data.
we have selected two categories of dependent variables to investigate: (i) religious expressions6, represented by the label RELIG; (ii) the influence of German language, represented by multiple labels such as G-DL for daily life, G-EDU for education, and G-JOB for working contexts. Since we are interested in the general use of German words, we generalize the labels to GERM.
sen for analysis in the corpus is age, which is an important factor to consider in language variation. The speakers are divided into two categories, young (21-26 years) and old (46-55 years).
This section discusses two types of analysis: error checking and data analysis. Error checking is the process of identifying and fixing errors or inconsistencies in the data, while data analysis involves using statistical and visualization techniques to extract insights and draw conclusions from the data. frequency) of dependent and independent variables. Table 1 presents these frequencies along with the total number of words spoken by young and old speakers. By normalizing the frequencies and estimating the proportions (old vs. young), we observe the following: = = 1.89 ·
= = 0.40 ·
Correspondingly, old speakers use 40% of the amount of German borrowings compared to young speakers. To assess the significance of our findings, we conducted standard t-test analyses with DATAtab [22]. When comparing the proportion of older speakers using religious expressions to that of younger speakers using the same expressions, the result was the following: (10695) = − 7.91, < .001 Research Question In the following section, we use Table 1 CorpusCompass to answer two research questions focus- Frequency of dependent variables (GERM and RELIG) across ing on Iraqi speakers: (i) does age influence the usage age groups (Old and Young) along with the total number of of religious expressions? (ii) are young speakers more words spoken by each group. subject to German borrowings while speaking Arabic? Age Group GERM RELIG Words These questions will guide our exploration of potential correlations within the dataset, with the understanding Old 221 357 42,483 that the current analysis serves as a simplified demon- Young 505 175 39,406 stration. However, it should be noted that the dataset is well-suited for rigorous statistical analysis, including techniques such as regression analysis.
Error Checking CorpusCompass can identify any er- older and younger populations yielded: rors or inconsistencies during the annotation process and generate separate CSV files that provide information (10695) = 10.59, < .001 on annotated and non-annotated tokens. Thanks to the generated file, we were able to find circa 400 (3% of all the annotations) ill-formatted annotations (e.g. “[$GJOB.biriif["), 1, 305 missed annotations of which 205 where considered correctly identified and more than 9 unknown variables, i.e. dependent variables that are present in the corpus but not specified beforehand by the user.
The p-value suggests that the dependent variables (German borrowings and religious expressions) play a role in the language variation exhibited by young and old Iraqi migrants residing in Germany and requires further investigation. Ultimately, these findings validate our initial research question and demonstrate the value of structured data in facilitating robust statistical analyses.
Data Analysis By importing the binary_dataset in Excel [21], we determined the cross-tabulation (pair-wise
as Inshallaah (God willing), alhamdulillaah (Praise be to God), Allah ysallimak (may God protect you) etc. as “Allah expressions”. They include an explicit or implicit reference to Allah, which is literally translated as “the God”.
without its limitations. In this section, we outline some of the primary constraints of the tool, alongside potential future developments.
The user interface, especially the integration of regex existing analysis tools to answer unique research quesand Jupyter Notebook might pose a challenge to linguists, tions. With CorpusCompass, we aim to contribute to the particularly those hesitant to engage with programming. development of tools for spoken language corpora. The This could be solved by developing the tool into a more existence of this tool and its accessibility to researchers user-friendly application. Furthermore, we showcased without a background in programming will lead to more the functionality of CorpusCompass in addressing a spe- quantitative studies that analyse such corpora. The tool cific research question. While the tool has already been exemplifies interdisciplinary collaboration and emphaapplied to address other research questions [23] and on sizes the importance of linguistics researchers working another corpus7, the extend of its usability remains a with experts from computer science and engineering. point of investigation. It is essential to assess its perfor- This collaboration results in the development of fleximance on multiple corpora to enhance its robustness and ble corpus tools applicable to a wide range of research confirm its applicability for diverse research contexts. studies.
Additionally, while the tool identifies errors, as delineated in Section 4.2, the manual correction process can be Sharing is Caring It is essential to highlight that structedious and time-consuming. Looking forward, future it- tured datasets are crucial for sharing data in linguistic erations of CorpusCompass might integrate an automatic research, particularly in connection to research data manerror correction feature that suggests possible correc- agement practices and platforms such as Figshare [24]. tions, allowing users to either accept or decline them. Organized data facilitates sharing and reusability among Another area for consideration is the tool’s reliance on researchers, enabling more extensive collaborations and the CSV format, which might present compatibility is- the creation of larger datasets. Furthermore, structured sues with other linguistic tools. Transitioning to more datasets allow researchers to replicate and verify research standardized formats, such as XML, upcoming versions findings, promoting transparency and accountability in could address this limitation. the scientific community. Therefore, creating a struc
In summary, there are numerous opportunities to re- tured dataset is not only essential for internal analysis ifne and enhance CorpusCompass. By addressing its cur- but also for the advancement of the field and the dissemrent constraints, introducing new functionalities, and ination of knowledge. emphasizing user-centric enhancements, this tool has the potential to become an even more invaluable asset in Corpus Linguistics. Acknowledgments
Creating and analyzing a corpus is a complex task that requires a range of technical and practical skills. In this paper, we have explored the challenges involved in these steps and introduced CorpusCompass as an innovative solution. The tool’s aim is to simplify data extraction and dataset generation, facilitating the identification of significant features and syntactic errors in the annotations. This contributes to advancing the overall replicability of studies within the field of Corpus Linguistics. As CorpusCompass is implemented as a Jupyter Notebook, it also serves as an accessible introduction to programming for researchers who wish to expand their skill set and gain more control over their analytical processes.
Additionally, we have presented a real-world example of how CorpusCompass can be applied in the field of language variation by using a subset of our corpus of Arabic varieties spoken by migrants in Germany, representing an under-resourced language. The example shows how the generated dataset can be used in conjunction with
provided by Prof. Dr. Jonathan Owens.
(University of Amsterdam) and Prof. Dr. Jonathan Owens (University of Bayreuth) for their invaluable feedback.
The development of CorpusCompass was undertaken as part of the project Modernity, Migration, and Minorities: Three Case Studies of Arabic in Contact at the University of Bayreuth and funded by the Deutsche Forschungsgemeinschaft (DFG), No. 429257272. [4] H. Norman, C. H. Hull, SPSS: Statistical package for terns in strings, in: J. VAN LEEUWEN the social sciences, McGraw-Hill Book Company, (Ed.), Algorithms and Complexity, Hand1975. book of Theoretical Computer Science, El[5] R Core Team, R: A Language and Environment for sevier, Amsterdam, 1990, pp. 255–300. URL: Statistical Computing, R Foundation for Statistical https://www.sciencedirect.com/science/article/pii/ Computing, Vienna, Austria, 2021. URL: https:// B9780444880710500102. doi:https://doi.org/ www.R-project.org/. 10.1016/B978-0-444-88071-0.50010-2. [6] M. Neves, J. Ševa, An extensive review of tools [18] Y. Oortwijn, T. Ossenkoppele, A. Betti, Interrater for manual annotation of documents, Briefings disagreement resolution: A systematic procedure in Bioinformatics 22 (2019) 146–163. URL: https: to reach consensus in annotation tasks, in: Pro//doi.org/10.1093/bib/bbz130. doi:10.1093/bib/ ceedings of the Workshop on Human Evaluation of bbz130. NLP Systems (HumEval), Association for Computa[7] K. Berberich, I. Kleiber, Tools for corpus linguistics, tional Linguistics, Online, 2021, pp. 131–141. URL: https://corpus-analysis.com/ (Mar. 2023), https://aclanthology.org/2021.humeval-1.15. 2020. [19] A. A. Jaradat, The linguistic variants of allah ex[8] L. Anthony, Antconc: A learner and classroom pressions in jordanian arabic, Cross-Cultural Comfriendly, multi-platform corpus analysis toolkit, munication 10 (2014) 61–68.
proceedings of IWLeL (2004) 7–13. [20] M. Piamenta, The Muslim conception of [9] A. Svedkauskaite, Monoconc pro 2.0 and the corpus God and human welfare: As reflected in of spoken professional american english: Resources everyday Arabic speech, Brill Archive, from athelstan, Style 38 (2004) 127–133. Leiden, The Netherlands, 1983. URL: [10] S. Mike, Wordsmith tools version 8, 2023. https://brill.com/view/title/1464. doi:https: [11] R. E. De Castilho, E. Mújdricza-Maydt, S. M. Yimam, //doi.org/10.1163/9789004661820.
S. Hartmann, I. Gurevych, A. Frank, C. Biemann, [21] C. Microsoft, Microsoft excel, A web-based tool for the integrated annotation of https://office.microsoft.com/excel semantic and syntactic structures, in: Proceedings (Apr. 2023), 2023. URL: https://ofice.microsoft.com/ of the workshop on language technology resources excel. and tools for digital humanities (LT4DH), 2016, pp. [22] T. DATAtab, Datatab: Online statistics calculator, 76–84. https://datatab.net/ (Apr. 2023), 2023. URL: [12] J.-C. Klie, M. Bugert, B. Boullosa, R. E. de Castilho, https://datatab.net/.
I. Gurevych, The inception platform: Machine- [23] M. Adnan, J. Owens, Imperfect Verbal Prefixes as assisted and knowledge-oriented interactive anno- Discourse Markers, volume XXXV, Perspectives tation, in: Proceedings of the 27th International on Arabic Linguistic, Benjamins, Amsterdam, To Conference on Computational Linguistics: System Appear.
Demonstrations, Association for Computational [24] T. Figshare, Figshare: the open research repository Linguistics, 2018, pp. 5–9. URL: http://tubiblio.ulb. platform, 2023. URL: https://figshare.com/. tu-darmstadt.de/106270/, event Title: The 27th In- [25] L. Milroy, M. Gordon, Sociolinguistics: ternational Conference on Computational Linguis- Method and interpretation, Wiley, 2003. tics (COLING 2018). URL: https://doi.org/10.1002/9780470758359. [13] M. van Gompel, M. Reynaert, Folia: A practical xml doi:10.1002/9780470758359. format for linguistic annotation–a descriptive and [26] S. A. Tagliamonte, Analysing sociolinguistic variacomparative study, Computational Linguistics in tion, Cambridge University Press, 2006. the Netherlands Journal 3 (2013) 63–81. [27] P. Boersma, D. Weenink, Praat: Doing phonet[14] D. Biber, S. Conrad, R. Reppen, Corpus linguis- ics by computer, Ear and Hearing 32 (2011). tics: Investigating language structure and use, Cam- URL: https://journals.lww.com/ear-hearing/ bridge University Press, Shaftesbury Road ,Cam- Fulltext/2011/03000/Praat__Doing_Phonetics_by_ bridge, CB2 8BS , United Kingdom, 1998. Computer.12.aspx. [15] S. T. Gries, What is corpus linguistics?, Language
and linguistics compass 3 (2009) 1225–1241. [16] M. Weisser, Essential programming for linguistics,
Edinburgh University Press, The Tun - Holyrood Road. 12 (2f) Jackson’s Entry. Edinburgh EH8 8PJ.
UK., 2009. [17] A. V. Aho, Algorithms for finding pat
missed annotations and unknown variables are presented for the entire corpus, which serves as the foundation for The study presented in Section 4.1 revolves around the the analysis in Section 4.2. This distinction is necessary variables and speaker information detailed in Table 3. as the subset corpus exhibits no errors or missed annotaThe employed syntax adheres to the JSON specification, tions. providing considerable flexibility in the examination of linguistic variables.
There are two main types of variables: independent and dependent variables. Independent variables, also known as input variables, are the factors that the researcher manipulates or controls in a study. In contrast, dependent variables, also known as output variables, are the outcomes or responses being measured. The dependent variables are afected by the independent variables.
The speakers involved in the example are represented by four anonymized aliases. For each individual, two attributes (i.e. independent variables), namely age and gender, are taken into consideration. These attributes encompass the categories of male and female, as well as old and young. Finally, the corpus is annotated with two dependent variables: german context, which can be categorized as either G-SCHOOL or G-JOB, and religious phrases, identified by the annotation RELIG.
that reflects the defined variables, where each row corresponds to a token within the annotated corpus, and each column represents a distinct variable category (see Figure 4). Additionally, a binary_dataset file contains a one-hot encoded version of the dataset, specifically designed for seamless integration with statistical models and machine learning pipelines, without necessitating additional preprocessing steps.
Within the provided CSV example, Figure 4, token refers to the individual tokens in the corpus, German Context indicates the context of the text (either G-JOB or G-SCHOOL), Religious Phrases denotes the presence of religious phrases (annotated with RELIG), age and gender specify the respective attributes of the speaker, speaker identifies the speaker’s anonymized alias, interlocutor/s denotes the interlocutor(s) in the conversation, file points to the file path where the token was spotted (truncated in the example), and context provides additional contextual information.
For instance, the token kindarbifleega has G-JOB as the German context, no religious phrases, a young female speaker (age and gender), identified as A, and an anonymized speaker-interlocutor combination of BSH, S, SUH. The corresponding context is A uw huwwa yixtis.ir kindarbifleega... .
Additional details regarding the files generated by CorpusCompass are provided in the following. In order to maintain simplicity and avoid overwhelming the reader with excessive information, we conducted calculations on a subset corpus, as illustrated in Figure 2. This facilitates a visual association between the output of CorpusCompass and the input data. Conversely, the files containing
Annotation Information The annotation_info file contains information about the annotations included in the dataset file, such as the token itself, the number of times it appears in the dataset, and the number of times it appears for each speaker, Figure 5.
Missed Annotations The missed_annotations file tracks tokens that were previously annotated but not consistently annotated in subsequent instances. It contains the token and its context, determined by a user-defined ngram size. Figure 6 reports an example taken from the from corpus in Section 4.1.
Unknown Variables The unk_variables file is designed The individuals in each group come from the same point to a list of variables that were not specified in the dialect area and almost all come from the same circle of JSON file. This file includes information about the speak- friends/family. In order to minimize possible influences ers, the context, and the file it was taken from, making it on the interview conversations, the interviews were coneasy for researchers to identify and correct any inconsis- ducted by two assistants who are native speakers of the tencies in the dataset, Figure 7. respective varieties.
Since the sociolinguistic interview is used as a basic Descriptive Statistics Knowing basic descriptive statis- tool in the study of sociolinguistic variation and it is the tics is fundamental in language research. The corpus_stats most common method for collecting sociolinguistic data JSON file provides an overview of the corpus by reporting [25], the research data were collected using this method. key statistics. The file contains four types of informa- The goal was to move from general and impersonal question: (i) word-related information such as the number of tions to more specific and personal questions. Questions paragraphs8, words, and characters; (ii) variable informa- on selected topics encouraged respondents to narratively tion, including the number of dependent and independent talk about their personal experiences (e.g., life in Gervariables and their values; (iii) speaker-wise information, many/home country, refugee experience, friends/family, such as the total number of speakers, speakers of interest, fears and concerns). Thus, the speaker’s natural language and words spoken per speaker; and (iv) annotation-wise could be elicited [26]. After data collection, phonetic traninformation, such as the number of unique annotations scription of the recordings was performed using the tranand annotated tokens, see Table 2. scription program Praat [27]. Demographic information, as well as details about the respondents’ backgrounds and environments, were also collected. In addition, questionC. Corpus naires were employed to gather data on the interviewees’ language contact behavior with speakers of other languages and language varieties.
views (circa 60 minutes each) conducted in Bayreuth and Nuremberg, located in Bavaria, Germany. Additionally, two group conversations were recorded with the same speakers (90 minutes per interview), where four Iraqi and Syrian speakers were paired together.
"paragraphs": 4, "speakers_of_interest": 4, "all_speakers": 4, "dependent_variables": 2, "independent_variables": 6, "variables": 8, "variables_values": 18, "dependent_variable_values": 10, "independent_variable_values": 8,