Test cases can be generated from different sources of information [BP06, BHM + 15]: requirements specifications (black-box and model-based testing), source code (white-box testing), possible faults (fault-based testing), former versions and similar code (regression and metamorphic testing). When generating test cases by exploiting requirements specifications, the goal is to identify a finite set of test inputs that properly sample the execution space (partition testing), and a set of assertions (oracles) that check the results of testing the software system. Most of the approaches for automatically generating test cases proposed so far focus on generating test inputs from formal and semi-formal specifications [PY07]. Relatively little work takes advantage of natural language requirements, and it uses simplistic techniques, such as pattern matching, to determine conditions related to nullness of parameters (Tan et al.'s @tComment [TMTL12]), part-of-speech tagging and pattern-matching to generates simple pre-and post-conditions (Pandita et al.'s ALICS [PXZ + 12]). Some approaches take advantage of the simplifications induced by the structure of semi-formal specifications to generate test inputs. We have investigated more powerful and effective approaches. As illustrated in the following simple example, the Javadoc tags indicate the scope of the specifications (i.e. whether it is a pre-condition on a parameter, or a post-condition on the method execution result), and this simplifies the task of processing the information for testing. However, Javadoc tags may predicate on program elements that are partially implicit, for instance implicit subjects referring to parameters, and often use developers' jargon, for instance not null. Such features pose a challenge for traditional natural language processing: 1 / * * 2 * Merges the arrays in input 3 * 4 * @param x the first array, not null 5 * @param y the second array, not null 6 * @return an array which is the result of the merge, empty if both arrays are empty 7 * @throws IllegalArgumentEsxception if either array is null

Listing 1: Sample Javadoc specification of a method @param tags indicate the preconditions on the method input parameters, the @return tag (at most one) and @throws tags (one for each exception that the method may rise) indicate the postconditions of the method execution. The information expressed with Javadoc tags is useful to determine the correctness of the results of the test executions, but it is necessary to translate it into executable code assertions to act as test oracles.

For example, the translation of the specification expressed in the @param tags in Listing 1 is the following executable assertion, which automatically acts as testing oracle:

x != null && y!=null the @throws tag translation is:

and the @return tag translation is:

We designed and developed Toradocu [GGEP16] later extended to Jdoctor [BGK + 18], a technique that automatically infers executable assertions from comments in Javadoc tags expressed in natural language, as shown in the example. The early Toradocu approach performs simple translations of exceptional postconditions. Jdoctor extends Toradocu to all Javadoc tags and greatly improves the translation abilities of Toradocu, supporting also semantic similarity for interpreting synonyms [KSKW15].

Toradocu and Jdoctor use the Stanford Parser to produce a semantic graph for each sentence. First, they preprocess the text in natural language to deal with the peculiarities of Javadoc comments, which are rarely complete and grammatically sound English sentences. For example, most Javadoc specifications lack punctuation, many have implicit subjects and verbs, and often intermix mathematical or code notation with English. Also, different types of tags need different preprocesses, and Toradocu and Jdoctor take this into account.

The core idea of Toradocu and Jdoctor is to exploit information already present in the source code to produce ready-to-use executable assertions. This approach does not require any other external intervention or effort from developers. The last experimental results obtained by executing Jdoctor on 6 popular open source Java projects are encouraging: the tool achieves 92% recall and 83% precision on 829 translations [BGK + 18]. Also, Jdoctor assertions are officially integrated with Randoop [PLEB07], and in our evaluation they produce test cases that raise fewer false alarms and reveal more defects.

The results of our past work confirm the research hypothesis of our long term research plan: automatically generating test inputs and oracles from requirements specifications given in natural language with NLP is feasible and effective.

In the short term, we plan to analyze free, unstructured text in Javadoc, beside the specific Javadoc tags that we already support. Moreover, we aim to extract information beyond functional properties. We plan to focus on temporal, security, performance and other non-functional properties. As an example, Figure 1 shows some temporal properties about call protocols1 . Such information is very useful for reducing the amount false alarms that affect existing testing approaches. We will combine different approaches to interpret various properties expressed in natural language. We will resort to Open Information Extraction to infer information from unstructured text [DCG13, FSE11, SBS + 12], and natural language parsing, pattern matching, semantic similarities and machine translation techniques to match documentation with code elements.

Our mid-term plan aims to extend Jdoctor to deal with information coming from other artifacts in natural language, such as wikis, issue trackers, and community forums, which are commonly available for popular applications. Even if these artifacts do not have a narrow scope as Javadoc comments do -i.e., Javadoc refers to a specific method or a specific class -they are still often partially-structured, and thus we believe that our techniques, if properly extended, can deal with them. The software engineering research community already produced some techniques that derive test artifacts from system requirements such as use-case requirements [WPG + 15, MPGB18].

Our long term plan is to define and develop a set of techniques to automatically test human-centric software systems. Such systems have key features that make them different from traditional software systems: First and foremost, the user is an integral part of the system. Secondly, they often integrate different sensors and physical devices. Lastly, they often rely on machine learning components that drive the decisions of the system based on the observed inputs from sensors. In a nutshell, human-centric software systems can be seen as an evolution of ultra large software systems also called systems of systems [MPS08].

To deal with human-centric software systems we need to radically change the considered scenario and widen the set of techniques that we plan to use, mostly because the expected behavior of the system may be hard to predict, and it is seldom specified in the requirements. So far we studied the problem of automatically generating test inputs and oracles for functional properties of program units (classes and methods) with deterministic behaviors.

To address the problem of properly testing human-centric software systems, we need to move from functional properties of software components with deterministic behavior, to properties of subsystems with non deterministic behavior. Non determinism may derive from concurrency, and may be due to machine learning components that act differently depending on the underlying model they use. Moreover, external physical sensors and users involved in the system may increase the uncertainty of the expected behavior of the system.

Despite these challenges, we still plan to focus our analysis on natural language artifacts, and aim to infer the missing information to test such systems. No matter how complex such systems may be, their requirements still have to be expressed in some form: It could be, for example, classical user stories. We will investigate what kind of artifacts are mostly used to document such type of systems. We will study different ways of contextualizing the fragmented and incomplete information expressed in natural language to solve ambiguities and incompleteness, and we will properly exploit the inferred specification to test these complex systems.