Crowdsourcing for API documentation: A Preliminary Investigation Allahbaksh M. Asadullah Shilpi Jain Infosys Labs, Infosys Ltd. FORE School of Management Bangalore, India New Delhi, India Email: allahbaksh.asadullah@infosys.com Email: shilpijain@fsm.ac.in Abstract— Developers and researchers have been using Besides source code, several documents accompany a crowdsourcing in a variety of fields related to software software system, because there is a possibility that the source development and software engineering. Crowd based code perhaps is not sufficient to convey the objective of the documentation is another yield of crowdsourcing where the coder project. Hence, moderate to large sized development projects, community or workers document the software. In the present irrespective of application, generate a large amount of work, we have analyzed how crowdsourcing can be used for an API documentation. The study is based on the fact that good associated documentation such as documentation of code, programmers write descriptive variables and method names and algorithms, application programming interface (API), UML continue to do so for future references. A variety of tools such as diagrams, sequence diagrams, class diagrams, design Amazon Mechanical Turk, ETurk and DocIt were evaluated for documents etc. The set of these components are popularly the purpose. Among these, DocIt and ETurk were built in-house. known as software artifacts or technical documents. The evaluation of the documentation was performed by Usually, the process of documentation is elaborate and experienced coders. This is a preliminary experiment which was requires a significant amount of effort. A substantiate performed in a controlled environment. Results were encouraging documentation reduces the maintenance work and further and help us to determine that in future crowd based improves the productivity and reusability of the code. documentation might help to reduce time to market and improve software quality. Many developers believe that the documentation doesn’t require a high intelligent quotient and it is a waste of time and Keywords: Crowdsourcing, API Documentation, Amazon effort. They consider that the code written by them is sufficient Mechanical, Turk, DocIt, E-Turk and self-explanatory. On the other hand, another set of I. INTRODUCTION programmers appreciate its importance but tend to avoid due to paucity of time or limited resources. Segal [1] observed that Development and maintenance of large software systems1 professional developers do not volunteer to produce code remains a difficult and daunting problem for any project team. documents. If necessary, they will write a page or two as a Based on the studies, the percentage of effort goes in formality. Brief and inappropriate documentation is a matter of requirements phase is 15-20%, analysis and design is 15-20%, concern [2]. To overcome this limitation, new programmers development effort is 25-30%, system testing is 15-20%, and can leverage the abundant repository of unofficial API maintenance effort across the software development life cycle documents generated by API users on community portals. This is typically 5-10%. Until early 90s, in a conventional software unofficial documentation is popularly known as crowd development, the modification (adding or deleting a module or documentation because it is generated from crowdsourcing [3]. functionality) of the code had been a great challenge. This These documents have sufficient coverage for practical usages. paradigm shifted with the emergence of object oriented Blog post and ‘stack overflow’ are two types of crowd programming and open source software development that documentation that have highest coverage ratios [4] . Latoza et supports modular programming and library reuse. By breaking al. [5] describe the advantages of crowd based development in down the problem into multiple tasks, different developers 2 can their work. Crowdsourcing introduces agility in the component work in parallel. Modular programming allows distributed development of large-scale industrial projects, especially when development that shorten the development and documentation dealing with changes in API. time. Moreover, individual modules are easier to design, Besides having several benefits of API documentation there implement, and test. are not many tools available in the market which can perform 1 2 At Infosys Ltd., any software project’s code that exceeds 50k Please note that in the current study, words like developer, line of code (LOC) falls in the category of large software system (project). coder, programmer, and worker are used as synonyms. Workshop on Alternate Workforces for Software Engineering (WAWSE 2015) 43 this task efficiently. Hence for the purpose, we have developed A variety of web platforms are available for software two prototypes. In the current study we propose to evaluate and crowdsourcing such as Amazon Mechanical Turk (also known compare the performance of Amazon Mechanical Turk (also as MTurk). MTurk is one of the sites of Amazon Web Services. known as MTurk) with two in-house developed tools (ETurk It is a crowdsourcing Internet marketplace that enables and DocIt). The other objective is to test whether they can individuals and businesses to collaborate and perform complex produce consumable APIs. tasks that computers are unable to conclude. A user of Mechanical Turk can be either a "Worker" (employee) or a II. BACKGROUND LITERATURE "Requester" (employer). MTurk is one of the sites of Amazon Web Services. Employers post jobs known as HITs (Human API documentation or Programmers documentation, is a Intelligence Tasks), such as choosing the best among several deliverable of technical writing in which a technical writer photographs of a storefront, writing product descriptions, or develops instructions about how to effectively use a software identifying performers on music CDs. Workers can then API, hardware (SCPIs) or web-API [6]. In addition to browse among existing jobs and complete them for a monetary established coders, it is even useful for new coders as it helps payment set by the employer. On the basis of jobs performed, them to understand and learn best practices and Turk creates qualification profile for the workers. implementation details. Among many, API documentation is a MTurk encounter certain limitations. In MTurk, it was subset of software documentation. It is often embedded within difficult for coders to highlight the syntax and due to which the source code like Javadoc comments in Java. API code comprehension was a challenge. Second limitation, the documentation is written in plain language which requires a tool is limited to open source applications and doesn’t support thorough understanding of the API, its arguments, its return proprietary software. Therefore, for internal enterprise type and the languages and interface it supports. The text is applications, we developed a similar tool named as E-Turk often supported by images or hyperlink to other elements of the with enhanced features. The tool was tested with a team of Java source code. API usability is important because of the spread coders and it was observed that many developers were still of APIs in almost every application domain [7]. Among the finding it difficult to associate the source code with required factors that affect API usability, is the lack of diverse API class file (which might me parent class, implementations etc.). documents [8]. They suggested that it would be helpful to give information The extant literature noticed that unlike open source projects, about methods and classes that are doing similar nature of API documents are rarely updated in industrial setting. In open work. Hence we added a feature that can develop a connection source development, API documentation for deprecated code between similar classes & methods semantically. The new is taken seriously and this task is mandatory before they release version was renamed as DociT. To confirm the utility of DociT candidate of the library. A set of specialized tools are available and E-Turk over MTurk we performed an experiment based for source code documentation, but rarely used [5]. Typically, study in industrial setting. software engineers decide on their own what kind of documentation is worthwhile to produce and maintain, and III. RESEARCH DESIGN adopt selective tools which help them for the purpose [9] [10] [11]. This has been reported that software engineers tend to The research was conducted in two main phases: ignore complex and time-consuming documentation [11]. Literature shows that scientific documentation does not follow Phase1: Preliminary investigation using surveys with recommended standards proposed by SEBoK software programmers. (www.sebokwiki.org) Previous studies have shown that a majority of Java Phase 2: Experiment study where software developers wrote developers prefer to use Javadoc [2] [1] instead of APIs. In API documentation on MTurk and customized prototypes. another research, Latoza et al. [5] explained how a complex Limited time, dynamic requirements, confined research task can be decomposed into set of smaller tasks in crowd group, and small user base are some of the reasons cited for the development. However, they did not discuss or refer any absence of documentation [10] [2] [1]. Several researchers specific case studies for the same. Kittur et al. [12] showcased raised this concern but how to achieve the goal remain how an article writing can be achieved by crowdsourcing. Jiau unanswered. To estimate the root cause, in phase I of the and Yang [13] studied the API documentation of few open research, we conducted a preliminary survey with developer’s source project like GWT, SWT and Swing. They found that the community in the Silicon Valley of India, i.e. Bangalore, India. quality of documentation produced in open source forums was The objective was to understand how frequently and precisely of better quality. To the best of our knowledge, we could not our coders document the code, which is the preferred tool for find any research study that specifically talks about the the purpose. The survey was designed by experts which had 15 development of API documents through crowdsourcing. questions to capture the responses. The survey was sent to Workshop on Alternate Workforces for Software Engineering (WAWSE 2015) 44 approximately 127 Java coders with a minimum coding Enterprise Turk (E-Turk): E-Turk was designed by experience of 3 years and maximum of 5 years. Out of 127, mimicking the MTurk and has several modules such as user only 95 responded, 2 responses were dropped because of registration, user modules etc. The user interface of E-Turk is incompleteness and finally 93 responses (63 males and 30 easy to use and almost similar to Mechanical Turk with an females) constituted the final sample size. exception. Unlike MTurk, in E-Turk, the users were allowed to view all HITs, the source code was rendered with highlighted A. Phase I Findings syntax which makes it easier to comprehend, and submission The results indicated that 46.4% of respondents are in habit of the task is easy in comparison to Amazon Mechanical Turk. to include comments before defining any function, method, In MTurk we faced difficulties in posting of certain tasks class or a variable, while 34.7% do it only when they feel it is (HITs) where the code snippet had special characters. needed (when writing a complex algorithm or function), and DocIt: This tool was superset of E-Turk. The tool can rest i.e. Approximately 19% do it rarely as they consider that browse, search and navigate the source code online. Also the their code is self-explanatory and documenting it is a waste of tool featured semantic search and gives information on related time. In response to another question, 86% developers reported method and classes. This resembled Eclipse in many aspects. that they follow proper naming conventions and give The UI for DocIt was borrowed from another API explorer meaningful names. These findings led the foundation of our tool. When a developer clicks on any previously documented research as our research is based on the fact that programmers method it gives the latest document which the developer could write descriptive method and variable names. Further, 83% of edit to improve. If there is no documentation for the method developers deliberated that appropriately documented source then the developer may add the same. code facilitate in understanding of the code and leads to The experiment study with the second platform gave us more reduction in code comprehension time, 33% think insights into the problems faced by the crowd while documentation writing is a dull & boring job, and 16% of documenting the source code. For examples, developer could developers think that API documentation is not needed. Since frequently observe the related classes / interfaces. They projects have stricter deadlines, their focus remains on adding experienced different patterns in the source code to find out features and deliver. Almost every coder reported that how an object could have created (in case of Factory, Abstract documentation is a time consuming process and in most of the Factory or Prototype Design pattern). These features are fairly cases hard deadlines doesn’t provide any room for it. Another common and available in other IDEs. Hence, DocIt was culprit is ‘frequent changes in requirements’ which completely designed to address those key issues. shifts the focus of a programmer from documentation. Prior to the initiation of experiment, certain preparations B. Experiment Study were made. We followed systems approach and developed a Mechanical Turk: Mechanical Turk (also known as MTurk) process flow diagram (see Figure 1). is a decent platform but generic in nature. It lack certain special features. In MTurk, the task submitted as a single HIT is difficult to comprehend because of the code dependency. The HIT is an independent work unit. The main advantage of MTurk is that it offers insights into how one should design the platform, what are the key problem that needs to be addressed which a developer may face. A CSV file from an open source project, Apache Drill3 was selected to upload in MTurk. The worker had to pass the Java programming test before getting assigned to any live project. Once passed, they attempt to HIT. Every HIT included a fully qualified name of the class, a method and to be documented, and class body. The workers were expected to write the API Fig. 1. System Diagram documentation (java doc) of the method in the text area provided. We have included these three fields so that the Version Control Plugin (VCP) – We designed a VCP that workers can understand the package hierarchy from the class extracts latest source code from the version control system. name, document the method from the method body provided Since Team Foundation Server (TFS) is widely used in the and use the class body as a reference to identify the relationship organizations, we designed VCP for the TFS. It is important to of the method from other methods present in the class. Each note here that one can design or extend VCP for any other time the HIT is answered by a specified number of users, the version control system by using appropriate libraries such as HIT list is updated dynamically with the pool of pending hits. JHG, JGit etc. This plugin pulls the source code from the HITs are exhibited to users in random fashion. Workshop on Alternate Workforces for Software Engineering (WAWSE 2015) 45 Version Control System which is then be acted by different They were free to choose their tool from the available 3 components of the system. options. We advertised about these tools through several Source Code Parser – The source code which is pulled from internal media channels. Nevertheless, MTurk wasn’t the Version Control System is input to the Source Code Parser. The preferred choice for documentation. Merely 30% of the HITs source code parser can parse the elements of the source code. were resolved from the given list, and only 23 out of 237 For example, in Java it can tell about the various methods in (approx.10%) coders participated in the exercise. These the class, field variables and their initial values etc. The source findings were quite surprising hence, we asked the rest for the code parser analyzes the source code which is a .java file and reasons through a semi structured questionnaire. Their create an Abstract Syntax Tree (AST). The AST provides responses are summarized in Table 1. detailed view of the source code. The necessary information from AST is then extracted and is saved as Comma Separated TABLE I. PHASE I RESPONSES Percentage of Responses File (CSV file) and an index file. 1 Lack of experience with MTurk 65% Each .java file created more than 1 row in the CSV file. A 2 Tool Usability (issues with UI and 72% row in a CSV file contains details of method. The column framework). attributes are method body, class body, class full name 3 Complicated HITs 57% 4 Non-challenging HITs / Ambiguous 77% (Package Name +Class Name), which means that for each HITs method (in .java file), we will have a row entry in the CSV file. The CSV file and the index is input to the systems like In case of Mechanical Turk, the workers had no other Mechanical Turk, E-Turk and DocIt. Each row in the CSV file alternative to visualize additional methods which they can use correspond to one unit of work which is called as HIT (in to document the same class. This means that the person Amazon Mechanical Turk). documenting the method will be given a random method mij Each HIT is independent of the other HIT and the tasks can from a class Ci where mij in Ci. Hence the worker has to read be completed without any further detailed knowledge. There is the new class and understand it which was time consuming. no navigation facility available in Amazon Mechanical Turk The HITs were independent of each other, therefore, there are and E-Turk. Based on our knowledge and other results, it was chances that for every HIT worker may probably get a new realized that navigation plays important role in source code class, which leads to waste of time of the worker writing the comprehension. Consequently, this limitation was addressed in documentation. HITs are released in certain batches, hence DocIt. Besides navigability, DocIt has additional functionality. there is no guarantee that all the methods of a single class will It has rich interface that can browse and explore the source be released one at a time. About 60% of coders reported that code effectively. DocIt uses two inputs: CSV file and Index. the HITs did not interest them or motivate to respond. The index feature facilitates navigation. In E-Turk, the results were comparatively better that MTurk. Below is step by Step process This could be because of two reasons: a. the coder community, tool, and application were internal resources (i.e. available 1. We first extracted the source code elements and related within the organization setup), b. Since these applications were artifacts from version control system (VCS). developed for the client, they were well written and followed 2. The source code is then parsed and a CSV File and industry standards of coding and formatting. All developers Index file is prepared. were experienced Java coders. 3. These artifacts are provided to the crowd developers In case of DocIt, a cross reference to the classes and methods via Mechanical Turk, ETurk, and DocIt. was available. This helped the developers in accessing more 4. Based on the available source code and other information about the methods and the classes. They could information, developers wrote API documents. navigate easily to other classes and methods resulting in 5. The API documents were evaluated by a handful of improved documentation. DocIt was used by 15 developers. experienced system architects. We observed that the developers did navigate the source code 6. Once the API document for particular method is and read related classes and methods before documenting the approved, the source code was updated and committed method. Many of the developers looked at the other artifacts to the version control. which were presented in the tool based on the class the IV. FINDINGS developer was browsing in the tool. For example, for a class called HousingLoan, the developers paid attention towards the In case of E-Turk and DocIt we did not conduct any test to related documents and look over the threaded discussions on check the knowledge of the developer, instead we handpicked AutomobileLoan. Almost 45% of the developers resorted to a team of coders (Java) from Infosys. These programmers had reading these descriptions and found them useful in API scored more than 65% of marks in an internal Java exam. An document preparations. From the logs of the tool, we found that Enterprise Java application was given for the documentation. developers spent a considerable amount of time towards Workshop on Alternate Workforces for Software Engineering (WAWSE 2015) 46 understanding the classes like ILoan (interface), IInterest etc. are some of the metrics which we picked to check how much which were related to the Loan class before writing the API time and effort it takes to create API documentation. document. V. CONCLUSION AND FUTURE WORK A. Validation of API Documents Post experiment, a team of system architects and authors of Our preliminary investigation showed that software API the class files reviewed API documents. The documents were documentation can be achieved by crowdsourcing. A variety of checked for completeness, context and precision. On an developers review API documentation and prepare the final average coders wrote 6 lines per method in the API document output through code review system. Prior research confirms and reviewers’ added 2.5 lines to the submitted API that the crowd documentation is dependent upon the paradigm documentation. Three out of four reviewers did spend time to of the programming language. The person writing the check whether the documentation conveyed intended meaning documentation should have the knowledge of programming and relevant to the context. The average time per method spent paradigm and the domain of the project. Comparatively, for review was 7 to 8 minutes. These metrics are decent as developers spent lesser amount of time and effort in successful open source projects follow the same metrics before documenting a code where modules are interacting using APIs. their source code is committed to any VCS (Version Control We conclude by making an observation that the development System). of API documentation by crowd sourcing saves time and effort. Our study was based on the fact that the available tools are It further helps the software industry and academia to evolve insufficient to perform accurate API documentation. For and generate new software systems and innovate rapidly. We validation, two other source code documentation tools (e.g., are further exploring additional artifacts that could be helpful TwinText and Doc-O-Matic) were used which comprehend the in documentation. During the time of writing this paper, we are source code and generates the API documentation. These tools experimenting with unit test cases and version commit doesn’t require any human intervention. messages. These studies are still in progress. TwinText3 uses code comments with source code analysis to generate the API documentation. One can define the style of REFERENCES API documentation in TwinText. This tool has a limitation, API documentation produced here embed originator’s comments. Some of these comments were generic and written [1] J. Segal, "Models of scientific software development," in 1st International merely for understanding purposes (e.g. the code comments Workshop on Software Engineering for Computational Science and written by the developer merely to understand the internal code Engineering, 2008. structure) which might not be worthwhile for code reader or [2] R. Sanders and D. Kelly, "Dealing with Risk in Software Development," Software-IEEE, vol. 25, no. 4, pp. 21-28, 2008. developer who might use these API in future. The tool does not [3] J. Howe, "The Rise of Crowdsourcing. Wired," The Wired, vol. 14, no. 6, 2006. provide any consumable APIs. Doc-O-Matic 4 uses the source code as primary artifact and [4] C. Parnin and C. Treude, "Measuring API Documentation on the Web. , pages 25–30, May 2011.," in In Proceeding of the 2nd International Workshop on add additional information based on the domain through Web 2.0 for Software Engineering, 2011. external inputs. This tools uses Java language semantics to [5] T. D. LaToza, B. W. Towne, A. V. D. Hoek and J. D. Herbsleb, "Crowd identify the package name, member variables, and method Development," in 6th International Workshop on Cooperative and Human names and then generates the API documentation. We used Aspects of Software Engineering (CHASE), 2013. Doc-O-Matic on the Apache Drill project, and the output (API [6] G. Singh, "What is API Documentation?," 27 March 2012. [Online]. Available: documentation) was inappropriate. The descriptions had http://technicalwritingtoolbox.com/2012/03/27/what_is_API_documentation, [Accessed 22 February 2015]. random words and control characters. Our guess is that [7] J. M. Daughtry, U. Farooq, J. Stylos and B. A. Myers, "API Usability," in In probably their NLP (Natural language Processor) was unable Proceedings of the 27th International Conference on Human Factors in to generate meaningful sentences. Computing Systems: CHI’2009 Special Interest Group Meeting, 2009. The objective of current study was to evaluate and compare [8] M. P. Robillard, "What Makes APIs Hard to Learn? Answers from the performance of DocIt with other similar tools available in Developers," IEEE Software, vol. 26, no. 6, pp. 27-34, 2009. the market and whether they can produce consumable APIs. [9] T. C. Lethbridge, J. Singer and A. Forward, "How Software Engineers use We found that none of the available options could solve the Documentation: The State of Practice," Software, IEEE, vol. 20, no. 6, pp. 35- purpose. Manually generated API documents were better and 39, 2003. meaningful over automatic API documentation tools. Below [10] L. Nguyen-Hoan, S. Flint and R. Sankaranarayana, "A Survey of Scientific Software Development," in International Symposium on Empirical Software Engineering and Measurement, 2010. 3http://www.ptlogica.com/TwinText/ 4 http://www.doc-o-matic.com/ Workshop on Alternate Workforces for Software Engineering (WAWSE 2015) 47 [11] A. Pawlik, J. Segal and M. Petre, "Documentation Practices in Scientific [15] A. Forward and T. C. Lethbridge, "The relevance of software documentation, Software Development," in 5th International Workshop on Cooperative and tools and technologies: A Survey," in ACM symposium on Document Human Aspects of Software Engineering (CHASE), 2012. Engineering, 2002. [12] A. Kittur, B. Smus, S. Khamkar and R. E. C. Kraut, "Crowdforge: [16] D. Kramer, "API Documentation from Source Code Comments" A Case Study Crowdsourcing Complex Work.," in 24th Annual ACM Symposium on User of Javadoc.," in 17th Annual International Conference on Computer Interface Software and Technology, 2011. Documentation, 1999. [13] H. C. Jiau and F. P. Yang, "Facing up to the inequality of crowdsourced API [17] K. T. Stolee and S. Elbaum, "Exploring the Use of Crowdsourcing to Support documentation," in Sigsoft Software Enfgineering Notes, 2012. Empirical Studies in Software Engineering," in International Symposium on [14] B. Dagenias and M. P. Robillard, "Creating and evolving developer Empirical Software Engineering and Measurement, 2010. documentation: Understanding the decisions of open source contributors," in 18th ACM Sigsoft International Symposium on Foundations of Software Engineering, 2010. Workshop on Alternate Workforces for Software Engineering (WAWSE 2015) 48