Approaching First-Year Composition (FYC) as a \big data" phenomenon, we have prototyped software to study revision in a large corpus of student papers and thus to address a question central to Composition and Rhetoric scholarship: \What role does revision play in students' writing processes?" After running our program on a corpus of student writing, we see that our computational analysis challenges past research on revision and extends the methodological reach of Composition and Rhetoric to include \big data" analytics.
As valuable as this research has been in helping us
understand and respond to student revision, it is limited in two
important respects, limitations that Faigley and Witte
acknowledge in their own and prior studies and that still seem
applicable today. First, owing to the \complexity of the
analysis" involved, researchers have restricted their studies
to only a \small number of subjects" [3, p. 411]. Faigley
and Witte, for instance, include only 18 subjects in their
study while Sommers [
Both these limitations involve problems of scale: too few subjects and too few variables considered. Towards overcoming these limitations as well as answering the question with which this essay begins (\How can we know what happens between all of these rst and nal drafts?"), we approached revision, and FYC more generally, as a \big data" phenomenon. More speci cally, we built a corpus of rst and nal drafts from our students' portfolios and developed software to process them. This software allows us to examine revisions in student papers, to explore correlations between these revisions and the situational variables that may in uence them, and to perform both of these operations at scale. What we found di ers considerably from past research: unlike students in other studies, ours rarely focused on minor edits and surface corrections; instead, when they did revise, their changes primarily involved deleting and, more frequently, inserting complete sentences. What this suggests more generally is that our students see revision not as a \rewording activity," but as a sentence deletion and insertion activity, treating their original drafts as xed structures into which they plug or unplug not words, but sentences. In the rest of this paper, we describe our data set, the program we developed to analyze it, and the results it produced. We conclude by outlining future directions for our project and how \big data" analytics informs that work.
FYC at the University of South Carolina is taught in about 150 (fall semester) and 120 (spring semester) sections, each with about 20-24 students who each write three or four draft and nal papers, for a rough total of about 10,000 pairs of papers each semester. These are submitted to a content management system from which we download the papers. Earlier downloads have been manual; we have devised a system for a more automatic script for download. Most of these are submitted as dot doc or dot docx les, which can be turned into ASCII text with a Python program. Scripts and programs convert these to a standard le naming and clean the ASCII les of the various Unicode or nonstandard characters that would complicate later processing (smart quotes, em dashes, en dashes, ellipses, and so forth).
We do lose some data along the way. A small fraction of the papers are submitted in formats other than dot doc or dot docx, and at present we do not process these. Subsequent versions of our code may be able to make use of pdf, or Pages, or odt les, for example. We have not done that yet, though. We are at present drawing rather coarse conclusions from a corpus that is already large, and we would not expect students submitting pdf les, for example, to be statistically di erent as writers from students submitting doc les. We remark that each paper averages a little less than 10,000 characters, so that 10,000 pairs of papers is only about 200 megabytes of data each semester. This is substantial enough to require some management and organization but is by no means problematic; the quantity of data is less a management problem than is separating the les into class sections, keeping track of which papers come from which standard assignment, etc.
Similarly, we do admit that our \cleaning" process could introduce corruptions in ways that might make some detailed analysis di cult or impossible. Again, however, we do not imagine that a few such character changes, if done consistently to draft and nal, would change the overall analysis currently being done.
To analyze the data set, we used Python programs (only
about 2500 total lines of code) together with the Natural
Language ToolKit (NLTK) [
When we ran our program on a test corpus, the results surprised us because they di ered from what the scholarship on revisions says we should be seeing. In other words, unlike other studies of revision which found that students typically focus on minor changes in diction, punctuation, and grammar, we found that when our students revised, the majority of their changes involved deleting and, especially, adding sentences. Consider Figure 1. This stacked bar chart shows the percentages of unchanged sentences (light blue), lightly edited (red) sentences, sentences deleted from the draft or inserted into the nal (green), and heavily edited (purple) sentences. By far, the largest portion of sentences fall into the unchanged category. That is, the bulk of student writing survives unaltered from rst to nal draft. When we consider text that students actually changed, the bulk of those changes involve deleted and inserted sentences, followed by heavily edited and then lightly edited sentences. So while students do edit their text to some extent, their primary revision strategy involves treating their drafts as relatively xed structures into which the plug or unplug, not words, but complete sentences.
We remark that almost no great shifting of text occurs in our student papers. Our alignment algorithm is somewhat nave in that it anchors the initial alignment to unchanged sentences and then continues with that alignment. Clearly, if entire paragraphs were moved, our algorithm would work poorly and we would see anomalous results for those papers. In fact, we see this happening in only a very small fraction of the papers.
Our next steps involve explaining the revision practices we are observing. In other words, having gained a better sense of what happens between all those rst and nal drafts, we now plan to explore why it happens. Toward that end, our work will continue to be informed by big-data analytics. What do we mean by this? The phrase \big data" refers to a large data set and to a collection of computational techniques for analyzing it. Both meanings apply to our project. Our corpus will eventually consist of tens of thousands of papers, a size much too large for humans to analyze in detail, so we will use natural language processing to capture and quantify features that, taken together, o er a linguistic prole of each paper. Once those features are quanti ed, we will employ other computational techniques (e.g., linear regression and cluster analysis) to search for correlations (and other patterns) among the papers in the corpus. The program we have already developed supplies us with a relatively nely tuned computational model for revision. Equipped with this model, we have multiple paths forward, and, in the spirit of big data, we will explore as many of them as we can|including, but not limited to, the following: Turn each draft- nal pair into a four-dimensional vector (i.e., the frequency of unchanged, inserted, deleted, and edited sentences) and use cluster analysis to see if those pairs fall into any groupings. If they do, then look within and across those clusters to see if other written features or situational variables correlate with those groupings.
Compare the aggregate of deleted sentences (which students presumably thought were bad) with the aggregate of inserted ones (which they presumably thought were better).
Measure sentence complexity trends in our corpus against
those found in other genres, using a distinction
between clausal complexity (a characteristic of spoken
discourse) and phrasal complexity (a characteristic of
academic writing) ([
Collaborate with other institutions that have assembled similar corpora of student writing and run their data through our program. By seeing results produced by other institutions, we will gain a better sense of whether the sentence deletion and insertion practice we observed in our corpus is a more general trend or a phenomenon peculiar to our FYC program and its curriculum. Either result would be of interest: if the data from other institutions looks like the USC data, then perhaps we have identi ed a broad characteristic of student writing. If that data is di erent, then we will have new questions to ask to determine why one group of students revises di erently from the other.
Thus far, our project addresses one of the limitations Faigley and Witte point out in revision research: that is, rather than restricting our research to \a small number of subjects," we are able to examine revision patterns in tens of thousands of student papers at one go. In doing so, we have unearthed trends in student writing that past studies of revision fail to predict|namely, the prevalence of the sentence deletion and insertion trend. As we move forward with our project, we will address Faigley and Witte's second limitation: too few \situational variables" considered. Having quanti ed revision, we can now explore correlations between it and dozens of these variables, including grades, student major, teacher feedback, gender, and a host of features in the co-text of student revisions (e.g., sentence complexity, lexical sophistication, metadiscourse, etc.). As we do so, we will continue to enrich our understanding of what happens between all of those rst and nal drafts.
We acknowledge and o er thanks for the nancial support for student assistants from the Center for Digital Humanities at the University of South Carolina, and the work of graduate student Gerald Jackson and undergraduate students Brian Flick, Chelsea Reeser, Sam Watson, and Ming Wong at various stages of this research.