Incorporating REDCap for Data Collection of K-12 CS Education Research Monica M. McGill CSEdResearch.org Knox College monica@csedresearch.org ABSTRACT HIPAA guidelines and that could be used in single- or multi- Over the course of the last two years, the resource cen- site research projects [3]. Since 2004, the project has grown ter for the computer science educational community at cse- to be used across more than 4,000 institutions with over 1 dresearch.org has been collecting evaluation instruments for million users, having evolved from its original clinical roots measuring the impact of CS education primarily in primary into a general data collection platform. Institutions that and secondary schools. The CSEdResearch.org Resource use REDCap include Harvard affiliated institutions (e.g., Center provides over 100 surveys and protocols for use in re- Boston Children’s Hospital, Harvard T.H. Chan School of searching and evaluating computer science education. Work Public Health, and Joslin Diabetes Center) [4], Weill Cornell has begun to expand the capability of the site so that re- Medicine Clinical and Translational Science Center [14], the searchers and evaluators can distribute the surveys directly University of Minnesota Clinical and Translational Science on the site. This enhancement is a packaged plug-in called Center [12], and even the prestigious Mayo Clinic [6]. Closer Research Electronic Data Capture (REDCap), a product of to education and social sciences, REDCap is used in insti- Vanderbilt University built under support from the National tutions such as the PennState Social Science Research Insti- Institutes for Health (NIH) [2][3]. In this extended abstract, tute [11], Deakin University [1], and University of Nebraska- we present a summary of REDCap, the benefits it can pro- Lincoln Methodology and Evaluation Research Core [13]. vide to our community, and how we intend to vet the tool before releasing it to the public. Key features of the tool includes the capability to use ex- isting surveys, easily managed contact lists, scheduling of surveys, and creating multiple arms of surveys in longitu- Keywords dinal studies. Users can download their data sets to Ex- Educational research, resources, evaluation, assessment, con- cel, SAS, Stata, R, or SPSS for data analysis. It can also structs, K-12, primary, secondary, REDCap, data collection, means important data security and privacy protocols, such open science as removing identifiers from datasets prior to exporting and logging and audit trails on all data interactions. 1. INTRODUCTION The CSEdResearch.org Resource Center is designed to sup- port the computer science (CS) education community, with 2. DATA COLLECTION USING REDCAP a particular focus on K-12 researchers, evaluators, teachers, REDCap is maintained and supported by consortium part- administrators, curriculum designers, and policymakers [8, ners’ local hosts. Prior to releasing REDCap, three educa- 9]. The site currently houses data curated from over 600 tional researchers created an evaluation plan that consists primary and secondary CS K-12 education articles across of four primary categories: Usability, Functionality, Data ten venues(2012-2019) and currently houses over over 110 Integrity, and Security and Privacy. We are in the process computing education evaluation instruments and protocols. of evaluating the tool against the criteria (summer 2020). Initially, three undergraduate students and the primary re- We are currently in the process of vetting a data collection searcher will use the evaluation plan to vet the tool. To per- tool, the Resource Electronic Data Capture (REDCap) tool, form this evaluation, each tester will have a copy of the test- for integration [2][3]. REDCap (https://projectredcap. ing template to complete. We anticipate that going through org) is a well-established data collection tool that was orig- each of the criteria in the evaluation plan will take about 40- inally built for clinical researchers to collect data that met 60 hours for each team member to properly learn the tool, compare it to at least one other tool (e.g., Survey Monkey, Qualtrics), and gain maximum understanding of its capa- bilities. For each item to be evaluated, each team member will rate the tool on a scale of Pass (Excellent), Pass (Very Good), Pass (Good), Pass (Fair), and Fail. Each team mem- ber will also provide notes about their examination process and how they made a decision for each item. Copyright c 2020 for this paper by its authors. Use permitted under Cre- Once the team has completed their evaluation, we will re- ative Commons License Attribution 4.0 International (CC BY 4.0). view the results for a better understanding of the tool’s pros and cons. We will then decide if REDCap should be aban- [1] Deakin University Faculty of Arts and Education. doned as potential platform or if alpha testing should begin. Staff news. https://blogs.deakin.edu.au/artsed/ For alpha testing, we will identify 2-4 users in the computer 2020/04/28/online-sessions-for-redcap-and- science education research community to test the tool us- qualtrics-on-24th-april-and-4th-may/, 2020. ing real surveys. Alpha testers will be interviewed to ensure [2] P. A. Harris, R. Taylor, B. L. Minor, V. Elliott, that their needs are met. We will then review the feedback M. Fernandez, L. O’Neal, L. McLeod, G. Delacqua, from the alpha testers and, in conjunction with them, will F. Delacqua, J. Kirby, et al. The redcap consortium: again decide if REDCap should be abandoned as a potential Building an international community of software platform or if beta testing should begin. platform partners. Journal of biomedical informatics, 95:103208, 2019. For beta testing, we will recruit 15-20 potential users. Sim- [3] P. A. Harris, R. Taylor, R. Thielke, J. Payne, ilar to alpha, we will train the beta users and ask them to N. Gonzalez, J. G. Conde, et al. A metadata-driven use the tool for surveying for at least one project. We will methodology and workflow process for providing ask them for their feedback to determine if this tool would translational research informatics support. J Biomed provide the necessary support for their data collection needs Inform, 42(2):377–81, 2009. in their projects. If beta testing is successful, then we will [4] Harvard Catalyst. Redcap. release REDCap to the public. https://catalyst.harvard.edu/services/redcap/, 2020. 3. DATASET [5] F. G. Hillary and J. D. Medaglia. What the The data for REDCap is stored in a MySQL database that replication crisis means for intervention science, 2019. is self-contained across 132 tables, including permissions, [6] Mayo Clinic Center for Clinical and Translational folder and project structures, and other tool settings. For Science (CCATS). Workforce development redcap individual users, data can be exported for analysis in Excel, software. https://www.mayo.edu/research/centers- R, SPSS, Python, STATA, XML, and SAS. programs/center-clinical-translational- science/education/workforce- Discovering best practices in computer science education development/classroom-courses/redcap-software, cannot be achieved by one researcher alone. Large-scale col- 2020. laborative science can provide a way to ”...understand which [7] M. M. McGill. Discovering empirically-based best instructional practices work for whom and under what con- practices in computing education through replication, dition....” [5, abstract]. To conduct large-scale collaboration reproducibility, and meta-analysis studies. In across computer science education requires tools to gather Proceedings of the 19th Koli Calling International this data in one place or provide another means of inte- Conference on Computing Education Research, pages grating the data for meta-analysis [7]. Meta-analysis can 1–5, 2019. then be performed across projects with heterogeneous prop- [8] M. M. McGill and A. Decker. csedresearch.org, 2017. erties selected by the user. Building this data lends itself [9] M. M. McGill and A. Decker. Defining requirements to the empirical evaluation of heterogeneous research stud- for a repository to meet the needs of k-12 computer ies through quantitative meta-analysis [7, 10]. Additionally, science educators, researchers, and evaluators. In 2018 an open science approach to data makes it possible for data IEEE Frontiers in Education Conference (FIE), pages analysis to be reproduced easily and systematically. 1–9. IEEE, 2018. [10] M. M. McGill, T. McKlin, and E. Kaylor. Defining This type of data collection and analysis is already hap- what empirically works best: Dynamic generation of pening within tools designed to teach computer science and meta-analysis for computer science education. In computational thinking. By extending this practice to in- Proceedings of the 2019 ACM Conference on clude multiple studies across multiple sites, we can start to International Computing Education Research, pages synthesize the data and potentially accelerate the process of 199–207, 2019. determining promising practices for various learners. [11] PennState Social Science Research Institute. Electronic management of data and regulatory 4. CONCLUSION documentation. https://ssri.psu.edu/electronic- With our work to build a platform for the community to use management-data-and-regulatory-documentation, as a central data collection site, the power of the underlying 2020. data becomes open for further analysis. As the tool’s us- [12] University of Minnesota Clinical & Translational age grows, there becomes the potential for comparing what Science Center. Redcap. may work better for some groups of learners than for oth- https://www.ctsi.umn.edu/researcher- ers, providing information to feedback to researchers and resources/tools-and-software/redcap, 2020. practitioners. [13] University of Nebraska-Lincoln Methodology and Evaluation Research Core. Redcap. 5. ACKNOWLEDGEMENTS https://merc.unl.edu/redcap, 2020. This material is based upon work supported by the U.S. [14] Weill Cornell Medicine Clinical & Translational National Science Foundation under Grant Nos. 1745199 and Science Center. Redcap. 1757402. https://ctscweb.weill.cornell.edu/research- resources/clinical-translational-research- 6. REFERENCES informatics/redcap, 2020.