Many of the enterprise software applications today are developed using a service-oriented architecture (SOA) where web services, especially microservices play a fundamental role. A RESTful web service is a lightweight and scalable web service that provide data over Internet via an API using the HTTP protocol. Testing of a RESTful API is challenging because the user inputs are sequences of HTTP requests to a remote server and the server responses are also in HTTP format. In this paper, we aim to analyse ten diferent tools for testing RESTful API web services and provide tool comparisons first based on the request support and second based on multiple functionalities - implementation, platform, languages, testing techniques and supported imports. Taking into account this analysis and tools possibilities, we perform testing on RESTful API service for solving multidimensional time-independent Schrödinger equation using Hermite DVR approach. The findings show that not all tests could be run on all testing tools since some of them do not ofer ready-made assertions for validating the response time or HTTP body content or there is no option for validating the HTTP body.
information over the HyperText Transfer Protocol (HTTP) between a client and a server [
REpresentational State Transfer (REST) architecture was introduced in 2000 by Roy Thomas
Fielding and it relies on Uniform Resource Identifiers (URIs) for detection and interaction with
resources [
The goal of testing APIs is to validate the ways an API is expected to behave in terms of
functionality, reliability, performance and security [
Even though to the best of our knowledge there are not many papers that have examined
the diferences between several RESTful API testing tools, there are a few that describe the
process of developing RESTful API testing tools and automated testing approach. In [
The aim of this paper is to explore diferent RESTful API testing tools and examine the diferences between the functionalities they ofer. We will be discussing ten diferent RESTful API testing tools and we will examine them by testing a RESTful web service for solving multidimensional time–independent Schrödinger equation using Hermite DVR approach.
This paper is organized as follows: in Section 2 we provide an overview on the ten selected RESTful API testing tools that we are going to examine and draw a comparison table to see the diferences between them; in Section 4 we describe our case study, testing approach and examine the results; and finally in Section 5 we draw conclusions.
In this Section we give a short overview of several RESTful API Testing Tools and we provide tool comparison based on diferent criteria.
2.1.1. SoapUI
SoapUI [
Katalon Studio [
Apache JMeter [15] is an open source software that allows load testing, functional testing and measuring performance. Although it was originally designed to test Web Applications, it has since expanded and now ofers a wider range of functionalities. All tests in Apache JMeter must have a test plan, a thread group and at least one sampler. A test plan is the basis of any test and it describes a sequence of steps that JMeter will execute once it is run. The thread group ofers control over the number of threads/users that will be used when executing the tests, the ramp–up period (in what time frame should all threads be executed) and the number of times the test should be executed. Samplers simply send requests to a server and wait for a response. This tool ofers various assertions among which the most popular are Response and JSON assertions. The HTTP methods that Apache JMeter supports are: GET, POST, HEAD, PUT, OPTIONS, TRACE, DELETE, PATCH, PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK, REPORT, MKCALENDAR and SEARCH. For load testing, there is a CLI Mode which allows loading tests from any Java compatible OS. As mentioned earlier, it supports multithreading and it ofers CI support as well.
Rest–Assured [16] is a Java library that is used for testing XML–based and JSON–based Web Services. It supports POST, GET, PUT, DELETE, OPTIONS, PATCH and HEAD requests. To use the Rest–Assured library, it needs to be imported in the Maven or Gradle file in a Java project. It supports BDD given/when/then syntax and it also allows data–driven testing.
Assertible [17] is a tool for testing and monitoring APIs and its focus is on automation. It supports the following HTTP requests: GET, PUT, POST, PATCH, DELETE, HEAD, OPTIONS. Web services can be manually created or imported with a Swagger file, a Postman collection file or by a curl command. Assertible allows monitoring uptime and performance and continuous integration as well. This tool utilizes the power of jq (command line JSON processor) to modify and manipulate variables before they are used for a test run. Assertible has a feature called encrypted variables which is a safe way to store sensitive data during testing. 2.1.7. Karate Karate [18] is an open–source testing tool that allows automation, mocks, performance–testing and UI–automation. The most common usage is in a Java IDE, but it can also be used as a stand– alone executable. Karate supports the following requests: GET, POST, PUT, DELETE, PATCH, OPTIONS, HEAD, CONNECT and TRACE. Furthermore, Karate supports multi–threaded parallel execution. It has native support for YAML and CSV files which can be used for data–driven testing. It can easily be integrated in CI/CD.
Swagger Inspector [19] is an open source tool built around the OpenAPI Specification for testing and documenting APIs. It supports importing Swagger/OAS 2, OAS 3 and WSDL files. It is a web–based editor (full support for Firefox and Google Chrome) and it requires installation of an extension in order to work properly. The history of requests is visible in the sidebar and you can pin them and add them to collections. OpenAPI 3.0 supports GET, POST, PUT, PATCH, DELETE, HEAD and OPTIONS requests.
Insomnia [20] is a cross–platform REST API client. It provides an intuitive approach to managing APIs with sending requests, defining environment variables, authentication helpers, etc. It supports GET, POST, DELETE, PUT, PATCH, OPTIONS, HEAD and additionally a custom HTTP method (common examples being LINK, UNLINK, FIND, PURGE). Insomnia supports environment variables, which are most commonly used when the user is using identical values across several diferent requests.
HttpMaster [21] is a software development and testing tool for web applications and RESTful web services. HttpMaster ofers support for the most common rest methods like GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, PATCH, but also a custom method can be specified. There is a functionality available for global and multi–valued parameters. It also utilizes Command Line Interface to automate execution of HttpMaster projects and it allows OpenAPI (Swagger) imports. Additionally, it supports load testing.
Below, we present two comparison tables that show how each tool difers from the others. Table 1 shows the range of requests each tool supports (Fields marked with * represent that the methods are covered through a custom HTTP request), while in Table 2 there is a detailed tools comparison based on the following characteristics: 1. Implementation – how the tool is used is presented to the tester for usage: • IDE – Integrated Development Editor; • Framework; • Library; • CLI – Command Line Interface; 2. Platform – technologies/a framework of services the software relies on for standard operations; 3. Languages – languages used for writing test cases; 4. Testing techniques – the types of testing the tool supports; 5. Supported imports – the types of files that can be used to load already created tests. Groovy, Javascript Groovy Javascript Java Java
Java x x x Windows, macOS, Linux Windows, macOS, Linux Windows, macOS, Linux JVM JVM SwaggerHub / Windows, macOS, / Linux Windows x * * x x x * * x x x * * x * * x x x x * x Testing techniques functional, performance, interoperability, regression, mocking, automation, data–driven, security, load validation, functional, UI, security, load, runtime error detection, penetration, fuzz exploratory, automation performance, functional, load automation, data–driven automation, mock, performance, UI automation manual manual x * * * * x Supported imports WADL, URI Open API, WSDL, Postman RAML, WADL, Open API , GraphQL Schema, Runscope file / Open API, Postman collection Open API, WSDL Framework, CLI
Java DSL Swagger Inspector
Web IDE
Web–based
UI–based commands automation UI–based commands load, automation
Open API
The tools that have their own IDE are similar in the way that they are constructed. More or less, working on each of them is a similar experience. However, there is a diference in the fact that in most of the tools that have an IDE, aside from Postman, don’t require any programming knowledge to construct basic test cases. For more advanced testing, Katalon, SoapUI and Apache JMeter ofer the option to use the languages they support for writing more specific test cases. Postman uses Javascript for constructing test cases.
One advantage Apache JMeter, SoapUI and Karate have over the other testing tools is the functionality of using concurrent threads – which allows load testing. This allows choosing how many threads/virtual users will make a request to the server.
When it comes to testing response duration, some tools fell short – they did not ofer the option to validate the response time. Katalon Studio, SoapUI, Swagger Inspector, Insomnia and HttpMaster do not ofer ready–made assertions for the response time. However, Katalon Studio and SoapUI still have the option available with manually scripting assertions.
The libraries that were used as testing tools – Rest–Assured and Karate, proved to be suficient in conducting tests for our use case. It is important to note that the biggest advantage Karate has over Rest–Assured is that for validating payloads it supports assertions such as “contains”, which isn’t directly available with Rest–Assured. In order to use such assertion in Rest–Assured you need to use external libraries such as Hamcrest.
The free version of HttpMaster ofers obscure validation options: only whether the response status code is 1xx, 2xx, 3xx for a successful call and 4xx or 5xx for an unsuccessful call. The tool did not allow us to run the assertions if the HTTP body of the response contains an opening bracket and the response time for the call.
Two of the tools – Insomnia and Swagger Inspector did not ofer options for writing test cases. The only testing that can be done with them is manual – the tester themselves have to manually validate the response of the API call. 3. Use case – testing RESTful API for solving multidimensional time–independent Schrödinger equation using Hermite DVR approach
To test the abilities of the aforementioned tools, we conducted testing on the Schrödinger API [22] – a RESTful web service for solving multidimensional time–independent Schrödinger equation using Hermite DVR approach. This RESTful web service provides a method for solution of one–dimensional, two–dimensional and three–dimensional time–independent Schrödinger equation based on the Gauss–Hermite Discrete Variable Representation (DVR) approach. Web service source code is based on a python module that solves one–dimensional potentials using a DVR method [23].
The solution of one–dimensional Schrödinger equation is illustrated in the case of following model potentials: Morse potential; Simple Harmonic Oscillator (SHO) potential; Sombrero potential (Mexican hat); Woods–Saxon potential.
Solutions of two–dimensional and three–dimensional Schrödinger equations are illustrated for the following two model potentials: multidimensional Morse potential and multidimensional SHO potential.
The test scenarios that we will consider for testing this RESTful API are: • Check the response on default values for parameters (happy path); • Check the response on valid values for parameters (happy path); • Check the response on invalid values for parameters (negative testing with invalid input); • Check the response on large values for parameters (destructive testing). When validating the response we consider the following response elements: • HTTP status code – Validate the response status code (expected 200 for all scenarios, except when invalid values are entered, in which case we expect a response code of 400); • response payload – Validate whether the body of the response contains an opening bracket "[" – all valid responses contain an opening and a closing bracket; • basic performance sanity – Validate the response time of the request being less than 1000 ms.
The values that were used for each scenario are shown in Table 3.
The necessity for system interoperabilty and reusing imposes the need for creation of web services. RESTful web services came into play in the early 2000s. Testing of such web services
Test Scenario Default values
Valid values
Invalid values difered from testing of SOAP services because REST approach did not follow any unified description standards, but it required inputs criteria related to the requests and output criteria related to the API responses. To explore the current testing options for RESTful APIs, we have analyzed the existing testing tools and identified their main characteristics and diferences in terms of HTTP request support and functionalities such as implementation, platform, languages, testing techniques and supported imports. As a use case we considered the REST API for solving multidimensional time–independent Schrödinger equation. We defined diferent testing scenarios and success criteria. Based on the results, the majority of the tests that failed were defined as a part of the destructive testing scenario. Also, some tools did not provide ready– made assertions for validating the execution time. This research contributes to the theory by analyzing the options for testing RESTful APIs. It also helped us to think about the possible code optimization in order to satisfy the desired execution time requirements.
This work was supported in part by the European Union’s Horizon 2020 research and innovation programme, project National Initiatives for Open Science – Europe, NI4OS–Europe, [857645] and by the Faculty of Computer Science and Engineering, Skopje, North Macedonia. [14] Postman HQ, Postman | the collaboration platform for api development, https://www.
postman.com/, 2012. (Accessed on 05/22/2020). [15] Apache Software Foundation, Apache jmeter - apache jmeter™, https://jmeter.apache.org/, 1998. (Accessed on 05/22/2020). [16] J. Haleby, Rest assured, http://rest-assured.io/, 2010. (Accessed on 05/22/2020). [17] Assertible, The easiest way to test and monitor your web services : Assertible, https: //assertible.com/, 2015. (Accessed on 05/22/2020). [18] P. Thomas, Karate, https://intuit.github.io/karate/, 2017. (Accessed on 05/22/2020). [19] SmartBear Software,, Openapi design & documentation tools | swagger, https://swagger.
io/tools/swagger-inspector/, 2010. (Accessed on 05/22/2020). [20] Kong Inc, Insomnia | api design platform and rest client, https://insomnia.rest/, 2019.
(Accessed on 05/22/2020). [21] Borvid, Httpmaster | master http testing and debugging, https://www.httpmaster.net/, 2013. (Accessed on 05/22/2020). [22] Koteska et al, SchrödingerAPI - Jupyter Notebook, https://schrodinger.chem-api.finki.ukim.
mk/, 2021. (Accessed on 06/02/2022). [23] A. Richie-Halford, richford/dvr_py: Python module that solves one-dimensional potentials using a discrete variable representation method., https://github.com/richford/dvr_py/, 2017. (Accessed on 05/25/2020).