Generative AI models are capable of creating high-fidelity outputs, sometimes indistinguishable from what could be produced by human efort. However, some domains possess an objective bar of quality, and the probabilistic nature of generative models suggests that there may be imperfections or flaws in their output. In software engineering, for example, code produced by a generative model may not compile, or it may contain bugs or logical errors. Various models of human-AI interaction, such as mixed-initiative user interfaces, suggest that human efort ought to be applied to a generative model's outputs in order to improve its quality. We report results from a controlled experiment in which data scientists used multiple models including a GNN-based generative model - to generate and subsequently edit documentation for data science code within Jupyter notebooks. In analyzing their edit-patterns, we discovered various ways that humans made improvements to the generated documentation, and speculate that such edit data could be used to train generative models to not only identify which parts of their output might require human attention, but also how those parts could be improved.
split between human and computer. Typical approaches
asked, in efect, “who goes first?” and many models went
no further than a single cycle of
human-initiates-andAI-responds or AI-initiates-and-human-responds (e.g.,
[
More recent work has deconstructed the older concept
of unitary “initiative” into a flexible and collaborative
framework in which increased initiative by one party
(e.g. human) does not imply a decrease of initiative by
the other (e.g. AI) [
In this paper, we examine how humans interact with
a generative AI model in the context of writing data
science documentation. We specifically aim to extend
the human-initiates-and-AI-responds interaction pattern reflected in code-centric use cases: Brockschmidt et al.
to include a step in which a human may make subsequent [39] proposed the use of generative models for source
edits to the outputs of the model. Prior work by our team code, and Tufano et al. [40] used generative models to
has explored how data scientists use various kinds of fix bugs. In this paper, we conduct a deeper analysis of
models – including a GNN-based generative model – to participant interactions with Wang et al. [
We found that 85% of participants’ edit-patterns fully and Software Engineering accepted the algorithmically-generated text, or built upon In a recent study, Weisz et al. [41] examined the use of an the generated text; and only 15% of the instances involved unsupervised neural machine translation (NMT) model a complete rewrite. At first glance, these results suggest in addressing a task in application modernization, specifthat the generated text was well-accepted. However, par- ically regarding translating code from a legacy language ticipants modified the generated text in 41% of the cases. to a modern one. They found that software engineers Thus, (1) a human requests the generation of text; (2) the would be tolerant of errors or mistakes in the output of an AI provides that text; and then (3) the human makes a NMT model, as the code produced by that model would decision to use - or not to use - that text, and (4) may go be subject to the same review and testing procedures as on to edit the generated text. In the future, these three- code produced by everyone else on their team. In adto-four dialogic ”moves” could become the basis for an dition, a code highlighting feature that indicated where extended conversation between human AI. in the code human attention might be needed, based on
In this paper, we develop a taxonomy of edit-patterns, an aggregation of the model’s token-level confidences, discovering that some edits added missing details while was very desirable. Although a subsequent analysis by other edits explained the function of the code. A third Agarwal et al. [42] demonstrated how current metrics category of edits was primarly concerned with modifying of model confidence may not necessarily correlate with the formatting or style of the documentation. external truth of code quality (approximated by lint errors), the ability for a generative model to be able to “ask 2. Background for help” via code highlights was nonetheless highly valued by software engineers. In this work, we push even further by seeking to understand whether a generative model can also specify what kind of it help it needs from its human user.
We discuss recent work in the area of AI and machine learning applied to data science and software engineering, as well as the application of generative models to this domain. We also discuss recent studies on human interactions with generative models in software engineering.
2.1. AI and Machine Learning in In order to understand the co-creation process of data
Software Engineering science documentation, we examined Themisto [
A 3.1. Themisto: A System for Automatic pant was randomly given one of the two notebooks and Documentation Generation asked to document the notebook within 12 minutes; two participants failed to complete the task within that timeWe implemented the automatic documentation gener- period, and were excluded from further analysis. Before ation system as an extension to JupyterLab (Figure 1). the study, we provided a quick demo on the functionality The extension generates three types of documentation of the extension. for a given code snippet. The first type of documentation is generated using a Graph Neural Network based approach [45] which is commonly used in code summa- 3.3. Data Collection rization tasks. The second type of documentation is gen- We collected the completed final notebooks (N=24) after erated by retrieving relevant external API documentation participants finished the task. All study sessions were for a code function (e.g., functions defined in Pandas 2, conducted remotely using a teleconferencing tool and Numpy3, and Scikit-learn4). Lastly, the extension also we recorded participants’ screens with their permissions. provides a prompt-based approach where users are given After the session, we conducted a retrospective interview a short prompt to manually create the documentation. to ask about their experience and feedback. For example, if a code cell contains a graphic output, We wanted to understand how participants used the the extension would generate the prompt to ask users to algorithmically-generated documentation. For orientainterpret the output. tion, we review that 13 participants made documentation choices and optional modifications in each of nine 3.2. User Study Setup markdown cells in each of two notebooks ("Covid" and "house") - i.e., 13 participants for each notebook. In the We are interested in how users make revisions on the data from each notebook, we discovered two participants suggested explanations. Thus, we recruited 26 data sci- (per notebook) who did not complete the task, or who entists to add documentation to a given draft notebook created extra cells. We could not be certain how to "map" using the prototype. We prepared two draft notebooks these extra cells to the structure that was common across with the same length (9 code cells) and similar levels of the other 11 participants. Because we wanted to compare dificulty, but for two diferent problems. Each partici- participants’ responses in a disciplined way, we treated each of these people (who created extra cells) as outliers, and excluded them from analysis. This exclusion left us with 11 participants per notebook who had worked with
the algorithmically-generated documentation. this workshop paper, we present new analyses to examine the edit-patterns in the 41% of the cells with participant
There were few statistically significant diferences beTo prepare the documentation for analysis, we grouped tween participant data from the two notebooks. We all of the texts for each cell together (separately for each briefly report those analyses here, before the qualitative notebook). We then used a bag-of-words method to analyses that are the core of this paper. Because we did identify words (tokens) that were not included in the not find diferences between the two notebooks, we will algorithmically-generated documentation. In the quo- then perform content analyses of participants’ text on tations in this paper, the participant-introduced words the combined data from the two notebooks in Sections 5 appear in bracketed [blue-ink].5 Table 1 provides an illus- and 6. trative example. This was a slightly conservative method for identifying new text, because we might fail to detect that a participant had typed "data" (for example) in their 4.1. Starting Points for Documentation own usage, rather than including the word "data" from We provided three diferent Sources of documentation: the algorithmically-generated text. However, we used AI, Query, and Prompt. A chi-square analysis found no this method only to orient ourselves to the texts. significant diferences in the proportions of Sources cho
We next read each text by each participant. After read- sen by participants in each notebook. We also looked at ings all of the texts, two researchers agreed upon a code- combinations of Sources - i.e., no discernable source vs. book of edit-patterns. One researcher than applied that a single source vs. multiple sources. Again, a chi-square codebook rigorously to all of the texts. analysis showed no significant diferences between the notebooks. 3.3.2. Reference Notation We identify each text in terms of the participant number (1-26, with two non-completers and two outliers excluded), the notebook ("covid" or "house"), and the cell in the notebook (1-9). For example, "p21-house+4" refers to participant 21, in the "house" notebook, in the 4th markdown cell.
Participants accepted the algorithmically-generated documentation unchanged in 45% of the cells, and they edited the algorithmically-generated documentation in 41% of the cells. The remaining 9% of the cells were left blank. In
https://doccenter.freedomscientific.com/doccenter/doccenter/ rs25c51746a0cc/2012-06-20_TextFormatting/02_TextFormatting. htm for information on how to access font-attributes through JAWS.
We describe distinct Edit-Patterns below in Sections 5 and 6. Here, we briefly state that we used chi-square tests to examine whether participants used diferent editpatterns between the two notebooks. Only one of editpatterns showed a significant diference, and that pattern was concerned with the format (not the content) of the documentation (i.e., levels of headers in the markdown cell).
Similarly to the "combinations" analysis of Section 4.1, we found no significant diferences across notebooks for cells with zero edit-patterns, a single edit-pattern, or multiple edit-patterns.
• check for [missing/null] values for [some countries/regions there is no province/state data this is probably correct and not a flaw in] the [data] (p01-covid+5) 6
gle, stylistic diference in participants’ work with the two notebooks. We therefore combine our qualitative 5.1.2. This-step analyses of edit-patterns across the two notebooks. This-step edit-patterns occurred in many distinct subcat
We manually coded the edit-patterns in each partici- egories. The first subcategory is the addition of a few pant’s text in each markdown cell, according to our code- words to clarify the current step: book (Section 3.3.1). For the 22 participants in nine markdown cells, we thus coded 99 texts in each notebook, for • ### importing libraries ### importing [the necessary] a total of 198 coded markdown cells. We applied an infor- libraries (p20-covid+1) mal version of thematic analysis [46], noting Braun’s and While P20-covid’s addition might be only a matter of Clarke’s advice that there are multiple ways of conduct- emphasis, other simple additions provided much more ing a thematic analysis [47]. Previous grounded theory specific information about what was being done in the and thematic analysis studies have involved from 6 to step. P08-covid changed the meaning of the generated 74 participants [48, 49], and so our sample of 22 partic- documentation by adding specificity about what value ipants is within that conventional range. Within this was being computed: sample, we used the saturation practices of Guest et al. [50] (recommended by Ando et al. [46]) and Majid et al. • ### check [number of] the missing values (p08-covid+5) [51], defining saturation by a code that appeared from at least two participants. We made no restrictions on the P03–house provided a diferent kind of specificity about number of codes that we identified in a single text. Thus the types of datasets being used: a text might have zero codes if the participant simply • ### read the [training and test datasets] (p03-house+2) accepted the algorithmically-generated documentation, or it might have as many as three or four diferent codes P13-covid engaged in the same type of edit-pattern (in in complex cases. the other notebook), but included much greater detail:
• ### create the target and the test data [re-create train- pants may solve the same problem, in the same notebooking] and test [datasets based on] the [size of] the [orig- cell, in diferent ways. We found many examples of difinal training dataset] (p03-house+7) ferent strategies and/or diferent conceptions of what the intended reader would need to know, such as this conBy contrast, P210-covid focused on the treatment of miss- trast between P13-covid’s rather minimalist description, ing values vs. P01-covid’s much more extensive description: • ### check for [any] missing values [note] that [province/ • ### replace a specified phrase [(_)] with another specistate have quite a few] missing (p021-covid+5) ifed phrase [( ) then transform] the [datatype to int] (p13-covid+4) and P01-covid provided an even-more-detailed account of the same issue, with commentary on what they had observed:
command, "###". We will have more to say about this kind of stylistic edit-pattern, below.
by adding details. There were three subcategories of details: Contextual information, information about Thisstep (the current step), and information about Subsequent steps. 5.1.1. Contextual Details
clarified how the prior steps had produced materials that were used in the current step: • ### read a comma-separated values (csv) file into dataframe [of training] data [and test] data return the first 5 rows [of] the [training] data (p13-covid+2) We contrast P08-covid, P03-house, and P13-covid, who were adding information about what was being calculated or input, vs. P07-house and P26-house, who described how the operations were done: • ### create [train] and test data [by splitting
dataframe] (p07-house+7) • ### create the target and the test data and [use slicing] (p026-house+7) • ### data [preparation in] the [training] data [set] re- 5.2. Explanation Edit-Patterns place the [dashes] with [spaces for] the [date column and] convert the data [type to] integer (P01-covid+4)
ing a more extended Explanation. P02-house and P15
In some cases, participants added specific algorithmic house provided brief examples, in which they added opdetails that none of the generated texts had included. A erational explanations of how to perform the activity in repeated example was to mention (and sometimes dis- the cell: cuss) root mean square error and its importance: • ### return the first 5 rows [(defvalue=5)] (p02-house+3) • ### evaluate a score by cross-validation [uses rmse as an evaluation metric] with [5-fold] cross-validation (p03- • ### [separate train] and test [subsets post feature enhouse+9) gineering set] the target [as saleprice] (p15-house+7) • ### evaluate a score by [5-fold] cross-validation [using
rmse] (p15-house+9) 5.1.3. Next-Step
• ### read [and sanity check] the data (p04-covid+2) However, in other cases, the participant provided a much richer description of the next steps, as in this recitation by P05-covid: • ### Model A random forest is a meta estimator that fits a number of decision tree classifiers on various subsamples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting the [first line below initiates] a model [instance] and the [second line] fits the model on the [training data] (p05covid+8) P02-covid and P14-house went further, documenting the nature of source data files and their formats, and also the functional significance of additional modules: • read the [data: from] the [two files: ‘traincsv‘ and ‘testcsv‘ they contain] data [in csv format now ‘train‘ contains] the [train] data [and ‘test‘ contains] the [test] data [start on] the [train] data first (p02-covid+2) • ### importing libraries - pandas for [dataframes (like excel spreadsheet)] - numpy for [fast vector operations - sklearn] for [simple] data analysis [(in] this [case linear model)] (p14-house+1)
of information, including operational aspects and extended explanatory material about data files and programmatic resources. As we noted above, with more data, we may discover that Explanations may need to be combined with Details. Another possibility is that Explanations may turn out to be a subset of Tutorial edit-patterns, which we describe in the next subsection.
5.1.4. Details Patterns Summary We have shown three edit-patterns in which participants have provided more detail than was available in the generated texts. These patterns might be considered as spanning an imagined audience’s reading experience. In some • ### convert [training] data [remove dashes (‘-‘) in] the cases, participants wrote Contextual information into [dates this is done by applying] the [‘replace‘ function the generated texts. This contextual information was ‘astype‘ sets] the [‘date‘ column to integer type] (p05generally retrospective - i.e., what should the reader have covid+4) known in order to understand the code? In contrasting cases, participants focused less on context, and more on Similarly, P03-house gave instructions about how to work content within the current cell (This-step). Finally, in a with several datasets, including which columns (factors) few cases, participants wrote to anticipate the next cell or were involved and how to process those columns: cells. In the Discussion, we will think further about this • [## dataset preparation] the [next few cells prepare] kind of participants’ mental model of their audience’s the [train and test datasets] ### concatenate the [train experiences. and test datasets] with a [subset of columns (mssub
We also wish to acknowledge that some of our category class to salecondition)] is [format(a b)] (p03-house+4) boundaries are fuzzy. The next category of edit-patterns poses the question - what is the diference between a De- In a diferent cell, P05-covid explained the meaning of a tails edit-pattern and an Explanation edit-pattern? With function call and gave further instruction about how to further research, we may need to redraw this boundary. use the results of the function:
teaching the reader how to do the analysis. For example, P05-covid provide detailed explanations about how to carry-out a series of operations in python: • ### check the missing values detect the [number of] missing values for [each column ‘isnull()‘ returns] an [array of indicators of whether each value in a column is] missing [and ‘sum()‘ calculates] the [total number of] missing values [along] that [column] (p05-covid+5) P11-house taught how a conceptual operation worked and also gave advice about the naming of the statistical action: • [#####] this code cell is for [handling] missing values [which are replaced with] the [mean value] for [that feature] this is [also known as column-wise meanimputation] (p11-house+6) Finally, we note that P24-covid took a somewhat diferent tutorial strategy. They left the original generated text intact as provided by the algorithm, and then added a link to more information about the python code-structures that were used in the cell that the algorithm had described: • p24-covid/expt ### replace a specified phrase with another specified phrase [[for more information about lambda](https://realpythoncom/python-lambda/)] (p24covid+4)
Tutorial edit-patterns went much further than Explanation edit-patterns (which themselves had gone further than Details edit-patterns). Tutorial edit-patterns provide not only how-to information, but also interpretations of the meaning or purpose of actions, and in one case a link to further information.
In the preceding subsection, we began to describe a dimension of increasing complexity in the information that participants provided, and also (we infer) increasing effort on the part of the informants to think "beyond" what was given in the generated text. The last edit-pattern in this series involves even more "beyond" work: beyond previous transformations of the generated text, and probably beyond previous levels of efort. We call this editpattern "Rewriting," because it involves nearly complete replacement of the generated text.
It may be that the generated text in certain cells led to more instances of Rewriting. For example, three diferent participants rewrote the generated text of cell 4 of the House notebook:
Covid notebook: • ### [make predictions] (p17-covid+9) • ### [test] to [see how] the [model performs] (p20covid+9)
• ### [run] the [model] to [generate predictions] on the [test data] and [store them as a ‘dataframe‘] (p04covid+9) • use the [trained model] to [predict] the [target] on the [test data] (p05-covid+9)
In the Rewriting edit-pattern, we see diverse strategies, ranging from brief summaries to extensive new text, as well as high-level abstractions. Significantly, in multiple cases, participants made distinctly diferent Rewritings of the same generated text (i.e., in the same cell of the notebook). Thus, while the category of the edit-pattern is the same, the individual strategies can be quite diferent. We recall that we saw analogous patterns in the This-step Details edit-patterns of subsesction 5.1.2. While there may be agreement among participants that the generated text in certain cells requires a certain type of change, participants clearly adopt diferent strategies about how to make those changes.
In this part of Results, we have examined how participants changed the contents of the generated texts. Figure 2 summarizes a dimension that runs from simple Details, to more complex Explanations, to instructive Tutorials, • ### [join features from train and test into one df] (p15- and finally to complete Rewritings of the generated text. house+4) Collectively, participants have an extensive repertoire of edit-patterns that they apply to particular problems in • ### [transform and clean] the [data] (p22-house+4) particular documentation cells. We next examine more stylistic changes that participants applied to generated • ### [concat train and test col salecondition] (p02- text.
house+4)
• ### model [creating] (p21-covid+8) • ### model [training] (p24-covid+8) Sometimes in combination with other edit-patterns, par- However, participants also engaged in more complex ticipants modified the markdown formatting from the ways of completing a sentence. For example, P01-covid generated texts. Initially, all markdown texts were pro- both added a verb and changed the object of that verb:7 vided at the same hierarchical level (###). In multiple • [generate] the [predictions] (p01-covid+9) cases, participants modified those levels, placing texts in super-order / sub-order relation to one another: • [#####] this code cell is for [handling] missing values [which are replaced with] the [mean value] for [that feature] this is [also known as column-wise mean- • ### [fit regression] model (p02-house+8) imputation] (p11-house+6)
when combining the contents of two diferent forms of generated text: • ### [create] a [classifier ####] random forest [classifier] (p17-covid+8)
Some of the changes to content appeared to clarify what was being done in the code. The primary subcategory of these changes was to add a verb to a noun-phrase: • ### [fit] the model (p01-covid+8) • ### [train the] model (p10-house+8) • ### [fit] a lasso linear model [to the training data] (p07-house+8) • ### [train] lasso [cv] linear model (p15-house+8) There were also even more complex cases, in which it is not clear if the participant’s purpose was to complete a sentence Here, we repeat one example of P17-covid from the previous subsection, which illustrates our point about the ambiguity of complex cases: • ### [create] a [classifier ####] random forest [classifier] (p17-covid+8) • ### [leverage] the random forest model and [fit] the model [with training] dataset [(a] random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control [over-fitting)] (p13-covid+8)
in the sentence. In these two examples, we see P21-covid 7We use ”verb” and ”object” in the technical senses of Englishand P24-covid modifying the same generated text, but language grammar (e.g., [52]). A ”verb” performs an action. An with diferent versbs: ”object” receives the efect of that action.
peared to make the generated text more conversational. To avoid asserting our own judgments of what ”conversational” might mean, we show only examples in which the participant added a personal pronoun - typically ”we” or ”you”. For ease in reading, we have bolded those pronouns in the following examples: • importing libraries: [in] this code [segment you import the python] libraries [first that include ‘numpy‘] and [‘panda‘ you also import] a [class from sklean if you need to display some warning import the warnings] library [as shown] (p21-covid+1) • ### [define] and [configure] the model a random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting [ we also train] the model [with ‘fit()‘] (p04-covid+8)
We acknowledge that the distinction between content and style is far from clear (e.g., [53]). Therefore, we consider that our current categorization of Content-Related edit-patterns and Stylistic edit-patterns may require revision. With larger datasets, we may for example conclude that Conversational Tone is more related to Tutorial changes, and less related to ”style.” The same may occur with Completing a Sentence. We will also need to understand better the relationship of header-styles to content in brief documentation text snippets.
in which each of the above Content edit-pattern appeared.
Details edit-patterns were the most frequent. This may be unsurprising, because these kinds of edit-patterns took relatively little efort (Figure 2). In general, edit-patterns that were most costly of efort (Tutorial, Rewritten) had lower frequencies of occurrence, with Rewritten editpatterns occurring in fewer than 15% of the edited cells.
These results have implications for the design of algorithmic documentation systems. Taken together with our prior work on TransCoder, we can also see emergent • [##### here we] evaluate the [square-root of] the [5- ideas about how people can understand and make use fold cross-validated mean-squared-error of] the [trained] of AI outcomes, even in the absence of formal explanamodel with the [training set ‘(x_train y)‘ (p11-house+9) tory systems (e.g., Explainable AI, or XAI). Finally, these two projects point us toward important questions in the • ### [we now show] the [predicted values] (p24-covid+9) design of future human-AI collaborative systems. Note: A single cell could contain multiple edit-patterns.
Therefore, sums of percentages may not be meaningful.
lead us to think about a few implications to further improve the automatic documentation approach. First, the Details edit-patterns, Explanation edit-patterns, and Tutorial edit-patterns are relevant to the purpose of the notebook and the target audience. We believe that a future version of the generative approach should tailor the automatic documentation based on the usage scenario. Data scientists can benefit from more candidate documentation where the level of details varied.
With a larger dataset, we could associate these editpatterns to particular patterns in the algorithmicallygenerated texts. Based on those associations, we could modify the algorithms to anticipate the kinds of edits that humans have previously made (e.g., [54]). For example, if we can remove the need for Details-related and Conversational-Tone edits, then humans can focus on higher-value editing, such as Tutorials. We may then see emergent categories of even more task-specific and/or domain-oriented edit-patterns, when humans no longer need to put work into less significant edits.
One way to do this, is to include a reinforcement
learning component into the algorithm that could learn from
users’ modifications to the proposed texts. Our current
GNN model relies on the size of the training dataset to en- If the human edits the target code in the TransCoder
sure the quality of the results. However, data science code project, then a secondary AI (e.g., [
Useful Outcomes (e.g., [57]).
In our earlier generative documentation paper [
This outcome is consistent with our previous study of confidence scores alongside its output, and Weisz et al.
code translation, in which engineers reported that they [41] show the utility such scores can have in steering
preferred an imperfect translation to no translation at all. human attention toward reviewing portions of the output
While we hope to improve our generative algorithms in in which the model has low confidence. In this work, we
both research programs, we also envision future studies demonstrate how having an understanding of the nature
in which we will calibrate the quality of the outcomes, to of human edit-patterns to a generative model’s output
determine the threshold of ”poor quality” below which an can enable a generative model to not only identify where
algorithm should not be deployed. We could then perhaps human attention is needed, but also how human efort
provide a more ”skeletal” outcome, such as an outline of can be used to improve the quality of its output.
documentation rather than full-text. We could also treat One of the interesting questions will be exactly how
”poor-quality” instances as higher-priority opportunities to choose among those alternatives - i.e., when is
typefor algorithmic improvement. ahead useful, and how should a watchful but respectful
AI intervene with advice, and what dialogic or other
com8.3. Deepening Human-AI Collaborations munication structures should be involved in an on-going
AI-human consultation [58, 59, 60])? Our experiences
We now consider each of our research programs (transla- with the NMT algorithm in the TransCoder experiment
tion and documentation) in terms of published patterns showed that, with a suficiently broad beam-search [ 61],
of human-AI collaboration [
These patterns remain consistent with simple initiative The GNN in the Documentation project might also be
models (e.g., [
logical validity. We asked participants to document someone else’s notebook. By contrast, the canonical case in Jupyter notebooks is to document one’s own code. A future goal should be a more naturalistic practice of documenting my notebook.
Paradoxically, for precision of evaluation, we may also need to perform an even more controlled study, in which each person receives only one algorithm’s text at a time. This approach could help us to assess each algorithmic approach more independently than in the preliminary experiment in this workshop paper.
We also note that our analytic method could be strengthened in future research. Our bag-of-words approach was insensitive to word-order, and we looked only at patterns of added words. Future work should also examine patterns of deleted words.
Finally, we note the obvious sampling weaknesses. We conducted a relatively small study in a single institution. We hope to examine similar practices in other settings, and with more participants.
In this workshop paper, we have addressed topics in human-AI collaboration in data science and software engineering. We reported text analytic results from a study of generative documentation, showing that participants accepted generated text with or without modification in the majority of instances. These results are consistent with our earlier work, in which engineers were enthusiastic about using imperfect NMT-generated translations of software code. Similarly, participants in this study were also quite ready to accept or to work with imperfect GNN-generated texts. We also analyzed the edit-patterns in the generated text, developing categories that suggest future work directions. Finally, going beyond early unidirectional models of ”initiative,” we sketched promising directions for longer-term, on-going human-AI collaborations. [35] M. White, M. Tufano, C. Vendome, D. Poshyvanyk, man Factors in Computing Systems, 2019, pp. 1–13.
Deep learning code fragments for code clone detec- [49] A. Pradhan, B. Jelen, K. A. Siek, J. Chan, A. Lazar, tion, in: 2016 31st IEEE/ACM International Con- Understanding older adults’ participation in design ference on Automated Software Engineering (ASE), workshops, in: Proceedings of the 2020 CHI ConIEEE, 2016, pp. 87–98. ference on Human Factors in Computing Systems, [36] M. Allamanis, E. T. Barr, P. Devanbu, C. Sutton, A 2020, pp. 1–15.
survey of machine learning for big code and natu- [50] G. Guest, A. Bunce, L. Johnson, How many interralness, ACM Computing Surveys (CSUR) 51 (2018) views are enough? an experiment with data sat1–37. uration and variability, Field methods 18 (2006) [37] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, 59–82.
I. Sutskever, Language models are unsupervised [51] M. A. A. Majid, M. Othman, S. F. Mohamad, S. A. H. multitask learners, OpenAI blog 1 (2019) 9. Lim, Achieving data saturation: evidence from a [38] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Ka- qualitative study of job satisfaction, Social and plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- Management Research Journal 15 (2018) 66–77. try, A. Askell, et al., Language models are few-shot [52] B. S. Azar, D. A. Azar, R. S. Koch, Understanding learners, arXiv preprint arXiv:2005.14165 (2020). and Using English Grammar: Workbook, Longman, [39] M. Brockschmidt, M. Allamanis, A. L. Gaunt, 2000.
O. Polozov, Generative code modeling with graphs, [53] G. Lakof, A figure of thought, Metaphor and arXiv preprint arXiv:1805.08490 (2018). symbol 1 (1986) 215–225. [40] M. Tufano, C. Watson, G. Bavota, M. D. Penta, [54] K.-H. Zeng, M. Shoeybi, M.-Y. Liu, Style exampleM. White, D. Poshyvanyk, An empirical study on guided text generation using generative adversarlearning bug-fixing patches in the wild via neural ial transformers, arXiv preprint arXiv:2003.00674 machine translation, ACM Transactions on Soft- (2020). ware Engineering and Methodology (TOSEM) 28 [55] P. Almasan, J. Suárez-Varela, A. Badia-Sampera, (2019) 1–29. K. Rusek, P. Barlet-Ros, A. Cabellos-Aparicio, Deep [41] J. D. Weisz, M. Muller, S. Houde, J. Richards, S. L. reinforcement learning meets graph neural netRoss, F. Martinez, M. Agarwal, K. Talamadupula, works: exploring a routing optimization use case, Perfection not required? human-ai partnerships in arXiv (2019) arXiv–1910.
code translation, in: Proceedings of IUI 2021, 2021. [56] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, [42] M. Agarwal, K. Talamadupula, S. Houde, F. Mar- M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the tinez, M. Muller, J. Richards, S. Ross, J. D. Weisz, limits of transfer learning with a unified text-toQuality estimation & interpretability for code trans- text transformer, arXiv preprint arXiv:1910.10683 lation, arXiv preprint arXiv:2012.07581 (2020). (2019). [43] M. B. Kery, M. Radensky, M. Arya, B. E. John, B. A. [57] R. Kazman, G. Abowd, L. Bass, P. Clements, Myers, The story in the notebook: Exploratory Scenario-based analysis of software architecture, data science using a literate programming tool, in: IEEE software 13 (1996) 47–55.
Proceedings of the 2018 CHI Conference on Human [58] E. Horvitz, Uncertainty, action, and interaction: In Factors in Computing Systems, 2018, pp. 1–11. pursuit of mixed-initiative computing, Intelligent [44] J. M. Perkel, Why jupyter is data scientists’ com- Systems (1999) 17–20.
putational notebook of choice., Nature 563 (2018) [59] S. Ross, E. Brownholtz, R. Armes, Voice user in145–147. terface principles for a conversational agent, in: [45] A. LeClair, S. Haque, L. Wu, C. McMillan, Improved Proceedings of the 9th International Conference on code summarization via a graph neural network, Intelligent User Interfaces, 2004, pp. 364–365. arXiv preprint arXiv:2004.02843 (2020). [60] S. Ross, E. Brownholtz, R. Armes, A multiple[46] H. Ando, R. Cousins, C. Young, Achieving satura- application conversational agent, in: Proceedings tion in thematic analysis: Development and refine- of the 9th International Conference on Intelligent ment of a codebook, Comprehensive Psychology 3 User Interfaces, 2004, pp. 319–321. (2014) 03–CP. [61] C. Wilt, J. Thayer, W. Ruml, A comparison of greedy [47] V. Braun, V. Clarke, Using thematic analysis in search algorithms (2010).
psychology, Qualitative research in psychology 3 [62] E. Myers, An o(nd) diference algorithm and its (2006) 77–101. variations, Algorithmica 1 (1986) 251–266. [48] C. M. Baker, L. R. Milne, R. E. Ladner, Understanding the impact of tvis on technology use and selection by children with visual impairments, in: Proceedings of the 2019 CHI Conference on Hu