API testing is a critical stage in the QA process that involves analyzing application program interfaces (APIs) to evaluate their functionality, security, performance, and reliability.
In particular, for complex applications, the API may consist of dozens of endpoints and triggers that affect each other. To prevent regression issues from arising in production, it is highly beneficial to have a set of end-to-end automated tests. However, before implementing these tests, automation QA (AQA) engineers and management need to determine the framework on which these tests will be based.
In this article, we will explore and compare the two most popular API testing frameworks in the market: Karate and REST-Assured. Both frameworks are Java-based. The provided comparison will assist you in deciding which framework best suits your specific needs. Let’s get started!
Karate overview
Karate is a relatively new open-source framework that enables testers, even without a programming background, to conduct API, web service, and microservice testing. You might assume it originated from Japan, right? Actually, this framework was developed by Peter Thomas in 2017 and has since been adopted by enterprises worldwide.
Let’s delve into the details of the Karate tool. In terms of installation, Karate is available as a single, executable JARJAR file. It can also be added as a Maven dependency to the pom.xml file. If you prefer not to integrate it with a testing framework, you can opt for the core version:
<!-- https://mvnrepository.com/artifact/com.intuit.karate/karate-core -->
<dependency>
<groupId>com.intuit.karate</groupId>
<artifactId>karate-core</artifactId>
<version>1.2.0</version>
</dependency>Otherwise, there are versions that are compatible with the TestNG and JUnit testing frameworks.
<!-- https://mvnrepository.com/artifact/com.intuit.karate/karate-junit5 -->
<dependency>
<groupId>com.intuit.karate</groupId>
<artifactId>karate-junit5</artifactId>
<version>1.2.0 </version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.intuit.karate/karate-testng -->
<dependency>
<groupId>com.intuit.karate</groupId>
<artifactId>karate-testng</artifactId>
<version>0.8.0.1</version>
</dependency>Karate utilizes its own test scripting language called “Karate-Script”. If you have experience with behavior-driven development (BDD) frameworks like Cucumber, you’ll find that Karate’s code structure closely resembles it:
Feature: sample karate test script
for help, see https://github.com/karatelabs/karate/wiki/IDE-support
Background:
* url 'https://jsonplaceholder.typicode.com/'
Scenario: get all users and then get the first user by id
Given path 'users'
When method get
Then status 200
* def first = response[0]
Given path 'users', first.id
When method get
Then status 200
And assert response.name == "Leanne Graham"
And assert response.email == "[email protected]"Indeed, Karate shares many similarities with Cucumber:
- keywords like “Feature”, “Scenario”, “Given”, “When”, “Then”, “And”
- storing tests in .feature files
However, the main differentiating factor of Karate is that “all the step definitions are already written for us”. This is particularly advantageous for API testing. Unlike UI tests, API testing typically follows a uniform schema:
- Background
- Providing request parameters
- Validation of response status
So it makes a lot of sense to use pre-prepared step definitions instead of creating your own. Let’s now see how Karate handles all three of the mentioned steps.
Background
Background usually includes common properties that are shared among all scenarios in the .feature file. Some of these properties are already pre-defined, such as the URL:
Background:
* url 'https://jsonplaceholder.typicode.com'And you can directly define variables in the feature file using the “def” keyword:
* def user =
"""
{
"name": "Test User",
"username": "testuser",
"email": "[email protected]",
"address": {
"street": "Has No Name",
"suite": "Apt. 123",
"city": "Electri",
"zipcode": "54321-6789"
}
}
"""
Providing request parameters
Firstly, we need to provide a path which can also include query parameters separated by commas.
Given path 'users', 1234Next, we can include the request payload if needed, which supports both JSON and XML formats.
JSON:
And request
{
"name":"Vidhya",
"age":29,"locale":"en",
"twitter":"VidhyaJava",
"email":"[email protected]"
}XML:
* def payload =
"""
<note>
<to>Tove</to>
<from>Jani</from>
<heading>Reminder</heading>
<body>Don't forget me this weekend!</body>
</note>
"""
And request payloadFinally, we specify the HTTP method for the request, which can be one of the following: get, post, delete, patch, or put.
When method getValidation of response
Firstly, we validate the response code for both positive (200, 201) and negative (4xx) test cases.
Then status 201
Then status 404Next, we proceed with the validation of the response schema, where we can verify the response payload in both JSON and XML formats. In the case of JSON:
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "[email protected]",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered client-server neural-net",
"bs": "harness real-time e-markets"
}
}
If we want to validate specific properties, the assertions may include:
And assert response.name == "Leanne Graham"
And assert response.email == "[email protected]"However, it is important to note that Karate provides a built-in mechanism to validate the entire JSON response, unlike REST-Assured:
And match response == {
"id":1,"name":"LeanneGraham","username":"Bret",
"email":"[email protected]",
"address":{"street":"Kulas Light","suite":"Apt. 556",
"city":"Gwenborough","zipcode":"92998-3874",
"geo":{"lat":"-37.3159","lng":"81.1496"}},
"phone":"1-770-736-8031x56442",
"website":"hildegard.org",
"company":{
"name":"Romaguera-Crona",
"catchPhrase":"Multi-layered client-server neural-net",
"bs":"harness real-time e-markets"
}
}Karate includes a built-in reporting tool that generates reports in the target\karate-reports directory. The main report file is the karate-summary.html:
REST-Assured overview
REST-Assured is a Java-based library created by JayWay Company to streamline the testing and validation of Restful Web Services. It serves as an efficient catalyst for automating the testing process of REST APIs.
REST-Assured can be easily installed by adding the Maven dependency:
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>rest-assured</artifactId>
<version>5.1.1</version>
<scope>test</scope>
</dependency>
A testing framework like JUnit or TestNG is required as a test runner. The code example below demonstrates the usage of JUnit. Now, let’s take a look at a test case based on REST-Assured:
public class UsersTest {
@Before
public void setup() {
RestAssured.baseURI = "https://jsonplaceholder.typicode.com/";
}
@Test
public void getAllUsersAndThenGetFirstUserById() {
Response responseAllUsers =
given()
.when()
.get("/users")
.then()
.statusCode(200)
.extract()
.response();
int firstUserId = responseAllUsers.jsonPath().getInt("id[0]");
Response responseFirstUser =
given()
.pathParam("id", firstUserId)
.when()
.get("/users/{id}")
.then()
.statusCode(200)
.extract()
.response();
Assert.assertEquals(responseFirstUser.jsonPath().getString("name"),
"Leanne Graham");
Assert.assertEquals(responseFirstUser.jsonPath().getString("email"),
"[email protected]");
}
}We can see that tests based on REST-Assured are written in a BDD-style format, following the Given-When-Then structure. However, unlike Karate, these tests are embedded within the Java code. In other words, to view the implemented test cases, one needs to dig down into the API tests and examine the code. By default, REST-Assured does not utilize .feature files, although it can be integrated with tools like Cucumber or other BDD frameworks.
REST-Assured handles the stages of API tests schema in the following way (using the example of JUnit testing framework):
Background
We can simply use the @Before annotation:
@Before
public void setup() {
RestAssured.baseURI = "https://jsonplaceholder.typicode.com";
}
Providing request parameters
This stage is divided into 2 parts by the given() and when() keywords.
- With given(), we can provide the content-type and request body, if required. It is possible to pass JSON or XML payloads to a String object:
public void createUser() {
String body = "{\"name\": \"Test User\"," +
"\"username\": \"testuser\", " +
"\"email\": \"[email protected]\"," +
"\"address\": " +
"{ \"street\": \"Has No Name\"," +
"\"suite\": \"Apt. 123\"," +
"\"city\": \"Electri\"," +
"\"zipcode\": \"54321-6789\"}}";
Response response =
given()
.contentType(ContentType.JSON)
.body(body)
A more accurate approach is to utilize Object Mapping when working with REST-Assured. Define the request payload schema in the appropriate class:
public static class User {
private String name;
private String username;
public User(String name, String username) {
this.name = name;
this.username = username;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getUsername() {
return username;
}
public void setUsername(String job) {
this.username = username;
}
}
Then, you can simply pass a new instance of the class into the request payload:
@Test
public void createUser() {
User user = new User("Test User", "testuser");
Response response =
given()
.contentType(ContentType.JSON)
.body(user)
Alternatively, we can import the payload directly from a file:
@Test
public void createUserXML() {
File xmlDataInFile = new File("src/test/resources/user.xml");
Response response =
given()
.contentType(ContentType.XML)
.body(xmlDataInFile)
- With when() we need to specify path and method:
.when()
.get("/users")The path in REST-Assured also accepts variables:
int firstUserId = 123;
*********************
.pathParam("id", firstUserId)
.when()
.get("/users/{id}")
Validation of response
Validation of the response is done using the then() keyword. Firstly, we typically validate the response status code:
.then()
.statusCode(200)
Then, we can proceed to extract the response object:
.extract()
.response();
and validate some of its properties:
Assert.assertEquals(responseFirstUser.jsonPath().getString("name"), "Leanne Graham");
Assert.assertEquals(responseFirstUser.jsonPath().getString("email"), "[email protected]");In terms of reporting, REST-Assured does not provide a built-in reporting tool. Therefore, it is necessary to use a suitable external reporting library such as Allure Report.
Karate vs REST-Assured
Well, now it’s time to compare two given tools and decide which one to choose for our specific needs. Let’s compare their performance for the simple test scenario mentioned above:
As we can see, Karate takes much less time to execute than REST-Assured in the case of a single scenario. However, when considering multiple scenarios (i.e., the entire test scope), we can expect that the difference will not be as significant.
The comparison of the two frameworks based on other main criteria is presented in the table below:
BDD Syntax
Yes
Yes
Test-Scripting Language
Karate-Script (Gherkin)
Java
Validate whole JSON response
Yes
No (extra Java library required)
Test runner
Optional (JUnit, TestNG)
Required (JUnit, TestNG)
Retry ability
Yes
No (extra Java library required)
Reporting tool
Built-in
External tool required (Allure Report, etc.)
Read file
Yes
No (extra Java library required)
Performance testing
Yes (possible to re-use Karate tests Gatling tests)
No
Both Karate and REST-Assured testing tools have their strengths and weaknesses. In general, Karate has a larger number of built-in tools compared to REST-Assured. This means that you can start writing tests without spending additional time on configuring external libraries. Moreover, Karate uses a simple and straightforward test-scripting language, making it suitable for teams with limited Java experience. If you are satisfied with Karate’s standard reports and don’t want to invest time in configuring custom reports, it may be a better choice.
On the other hand, REST-Assured requires a strong Java background from the QA Automation team and the use of extra libraries. However, in the case of complex and highly-customized APIs, REST-Assured might be a better choice. Writing tests directly in Java may be easier in such cases.
REST-Assured also allows the use of libraries like Hamcrest matchers, providing more options for additional validation in specific scenarios. If you prefer to use BDD and Karate’s pre-prepared step definitions don’t meet your requirements, REST-Assured can be used in conjunction with an external BDD tool such as Cucumber.
Final Thoughts
In conclusion, it is worth noting that both Karate and REST-Assured tools are actively developed and well-documented. Ultimately, the choice between Karate and REST-Assured should be based on factors such as the complexity of your API, the level of Java expertise within your team, the need for specific validation or customization options, and your preferred testing approach (e.g. BDD).
With the information provided above, you can make an informed decision about which tool best suits your specific needs. Good luck!
See how we helped Union54
future-proof Zambia #1 card-issuing
API with test automation
