Online testing of RESTful APIs: promises and challenges

@article{MartinLopez2022OnlineTO,
  title={Online testing of RESTful APIs: promises and challenges},
  author={Alberto Martin-Lopez and Sergio Segura and Antonio Ruiz-Cort{\'e}s},
  journal={Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
  year={2022},
  url={https://api.semanticscholar.org/CorpusID:253421826}
}
The results of an empirical study on the use of automated test case generation methods for online testing of RESTful APIs used the RESTest framework to automatically generate and execute test cases in 13 industrial APIs for 15 days non-stop, resulting in over one million test cases.
View on ACM
doi.org
25 Citations
Highly Influential Citations
4
Background Citations
4
Methods Citations
3

Figures and Tables from this paper

25 Citations

AGORA: Automated Generation of Test Oracles for REST APIs

AGORA, an approach for the automated generation of test oracles for REST APIs through the detection of invariants—properties of the output that should always hold, can be seamlessly integrated into existing API testing tools.

ESEC/FSE: G: Automated Generation of Test Oracles for REST APIs

AGORA, an approach for the automated generation of test oracles for REST APIs through the detection of invariants —properties of the output that should always hold, can be seamlessly integrated into existing API testing tools.

SATORI: Static Test Oracle Generation for REST APIs

The findings show that static and dynamic oracle inference methods are complementary: together, SATORI and AGORA+ found 90% of the oracles in the authors' annotated ground-truth dataset.

KAT: Dependency-Aware Automated API Testing with Large Language Models

KAT (Katalon API Testing) is presented, a novel AI -driven approach that leverages the large language model GPT in conjunction with advanced prompting techniques to autonomously generate test cases to validate RESTful APIs and indicates the effectiveness of using the large language model for generating test scripts and data for API testing.

Exploring behaviours of RESTful APIs in an industrial setting

This paper proposes a set of behavioural properties, common to REST APIs, which are used to generate examples of behaviours that these APIs exhibit and shows that basing test generation on behavioural properties provides tests that are less dependent on the state of the system, while at the same time yielding a similar code coverage as state-of-the-art methods in REST API fuzzing in a given time limit.

Test Oracle Generation for REST APIs

The number and complexity of test case generation tools for REST APIs have significantly increased in recent years. These tools excel in automating input generation but are limited by their test…

A Public Benchmark of REST APIs

This paper provides a comprehensive and Public REST API Benchmark (PRAB), to be utilized by researchers in their evaluations of REST API studies, and provides their documentation in a publicly available GitHub repository.

Minimizing Test cases in Rest API Fuzzing

This paper takes an OpenAPI Specification and automatically starts generating test cases based on the endpoints and starts to test the given REST API endpoint and infers dependencies between the endpoints hence it is stateful.

Fuzzing frameworks for server-side web applications: a survey

This study reviews the state-of-the-art fuzzing frameworks for testing web applications through web API, identifies open challenges, and gives potential future research, including fuzzing for web client programming.

An Intelligent Agent for Automated Test Generation from OpenAPI Specifications

An intelligent agent based on LLMs to interpret OpenAPI documents and automatically generate test collections is proposed, demonstrating feasibility and reduced manual effort, while indicating the need for further validation and refinement for CI/CD use.

81 References

RESTest: automated black-box testing of RESTful web APIs

RESTest is presented, an open source black-box testing framework for RESTful web APIs that supports the generation of test cases using different testing techniques such as fuzzing and constraint-based testing, among others.

Empirical Comparison of Black-box Test Case Generation Tools for RESTful APIs

Among the considered tools, RESTler appears to be the most solid, able to successfully test all case studies (the other tools experienced crashes), and test cases generated by RestTestGen scored the highest coverage, suggesting that its testing strategy is the most effective in testing REST APIs.

RESTful API Automated Test Case Generation with EvoMaster

This article proposes a fully automated white-box testing approach for RESTful APIs, where test cases are automatically generated using an evolutionary algorithm, and shows that this novel technique automatically found 80 real bugs in those applications.

ARTE: Automated Generation of Realistic Test Inputs for Web APIs

ARTE leverages the specification of the API parameters to automatically search for realistic test inputs using natural language processing, search-based, and knowledge extraction techniques, outperforming the state-of-the-art approach SAIGEN.

Restats: A Test Coverage Tool for RESTful APIs

This paper presents Restats, a test coverage tool for REST APIs that supports eight state-of-the-art test coverage metrics with a black-box perspective, i.e., only relying on the OpenAPI interface specification of the REST API under test.

Test coverage criteria for RESTful web APIs

A set of ten coverage criteria that allow to determine the degree to which a test suite exercises the different inputs and outputs of a RESTful API are presented, which enables the automated assessment and comparison of testing techniques according to the overall coverage and TCL achieved by their generated test suites.

QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs

A method to explore the behaviour of a RESTful API by using automatic property-based tests produced from OpenAPI documents that describe the REST API under test, which supports building additional knowledge about the system under test by automatically exposing misalignment of specification and implementation.

Automatic Generation of Test Cases for REST APIs: A Specification-Based Approach

This paper proposes an approach to generate specification-based test cases for REST APIs to make sure that such APIs meet the requirements defined in their specifications, and particularly the OpenAPI one.

Automated black‐box testing of nominal and error scenarios in RESTful APIs

RestTestGen is presented, a novel black‐box approach to automatically generate test cases for REST APIs, based on their interface definition (an OpenAPI specification), highlighting the effectiveness of the approach in revealing actual faults in already deployed services.

Combinatorial Testing of RESTful APIs

This paper presents RestCT, a systematic and fully automatic approach that adopts Combinatorial Testing (CT) to test RESTful APIs, which can find eight new bugs, where only one of them can be triggered by the state-of-the-art testing tool of RESTful API.
...