Sharing datasets has many benefits, such as enabling study replication, and supporting secondary analysis. However, many of the publicly available datasets in computing education lack comprehensive documentation or omit key contextual information. For example, missing classroom contextual information (such classroom demographics) or instructional interventions used makes datasets dificult to interpret, and therefore, inhibits the usefulness of shared data. While there is work in standardizing data formats for Computing Ed research (e.g. ProgSnap2 format), there is no standard for describing contextual metadata which is vital to provide high-quality scientific insights. In this paper, we propose a documentation practice to support CS Ed researchers in creating a comprehensive, consistent dataset documentation. We also call for collaboration among researchers to adopt the proposed documentation approach to allow convenient, and reliable use of educational datasets.
CEUR
ceur-ws.org ‘datasheet’ design that explains the dataset motivation, usage, and collection process [4]. Pushkarna et al. presented ‘Data Cards’ which is a proposed structured summary of essential information in ML datasets that could not be inferred directly from data (such as data training methods, and intended use cases) [5]. While the ‘datasheet’ and ‘Data Cards’ ideas are relevant to our proposed work, they do not align with the specific documentation needs of CS Ed datasets (such as description of instructional strategies and classroom context). In this preliminary work, we propose a documentation design embedded within a single, guided file, that prompts researchers with targeted questions to describe key elements of their dataset (e.g., classroom setting, assessment structure, table summaries). Such documentation design can reduce cognitive load by eliminating the need to decide where or how to begin the documentation process.
For space limitations, we provide in Appendix A a template for our proposed documentation design
where future users can use. This design has been revised by several CS Ed researchers. For each element
in documentation design, we provide a question designed to illustrate the information needed (discussed
in detail below). We use the following design principles to create the proposed documentation design:
(
The proposed documentation workflow is implemented as a structured set of questions (see Appendix A), where researchers can complete it by simply answering each question, resulting in a standardized file that serves as a documentation for their dataset. The documentation design begins with a ‘Documentation Objective’ which acts as a motivating introduction highlighting the value of a clear and complete documentation. This is followed by a section on ‘Author Contact Information’, which is essential when future users have questions or need access to other resources. The documentation is then organized into four categories of dataset information: ‘Dataset Overview’: This section collects general information about the dataset, such as its name, and relevant published papers to this dataset. The ’Dataset Log Data Attributes’ section collects details about the dataset structure and content, such as files format, tables and their attributes. The ’Dataset Contextual Questions’ section collects metadata information which is any information about the context of the classroom or lab study such as programming environments used, participants’ demographics, or any relevant contextual variables that helps in interpreting data accurately. ’Dataset Resources’ section documents any instructional or assessment materials that would help in understanding any evidence resulting from secondary data analysis. For example, dataset resources include the problems used, the assessment questions, assessment policies, final exams, …, etc. These resources can be directly added to the documentation, or just add a link for this data resource. In case any of the above information cannot be shared publicly, the Author Contact information sections can enable users to request access permission to this data.
this documentation design where some fields can be autocompleted, such as the tables’ names, or column names that could be automatically extracted from the uploaded data. Additionally, we want to call the CER community to incentivize datasets’ documentation to encourage dataset owners to document their data, which is crucial to enable reuse and secondary analysis on their datasets.
Please answer as many questions as you can. If a question has no answer, write ’N/A’.
1. Name:
This section provides an overview of the dataset.
This section includes all information about the dataset tables. A.4.1. Dataset Format A.4.2. Dataset Tables
1. Name:
values of this column] 1. Name:
values of this column] A.4.3. Dataset Properties
This section contains questions designed to provide any relevant details that help in understanding and interpreting the data.
A.5.1. Classroom/Lab Settings As applicable, please provide:
A.5.2. Participants’ Demographics As applicable, please provide a paper that includes the student/participants demographics, or type the population’s:
2. Age:
…)? [answer]
This section collects additional file resources to give more context to the log data. For each question, you can either type your answer or upload a file. If not applicable, write N/A.