Postman Continue Test Suite After Failure

Importance of API testing

In today's software development, API becomes a center of data transformation and transmission between different systems. API testing consists of a request sent to an API endpoint and a set of assertions that validates the response. Traditional testing focuses on the UI, validating look and feel, buttons and fields of the UI interface, while API testing focuses more on business logic, data response, performance and security. The most common protocol used in API is HTTP with REST.

Why API testing and what is the benefit of API testing?

API testing creates a more reliable code since it works at a lower level. It can be done by both developers and testers before development is completed with help of API contracts. Following are several benefits of API testing.

Faster and less time consuming during test run compared to UI design testing.
Testing earlier. Testers can start API testing before development is completed with the help of API contracts.
Bigger test coverage. Because of its efficiency and faster run time compared to UI testing, testers can create extraordinary inputs and conditions and push applications to its limits.
Easy integration. No user interface required while performing API testing makes it easier to integrate.
Independent. API interchange data using JSON on top of HTTP protocol, validate its request and response regardless of the underlying technologies.

There are number of variant types of API Testing that we use at Ice House in our methodology as follow:

Unit testing
A level of test that focuses on individuals of software components, also known as white-box testing.
Integration Testing
A level of test that focuses on interactions between components or systems.
Functional testing
A test that is performed to evaluate if a component or system satisfies functional requirements.
Performance testing
A test to determine the performance of a component or system.
Security testing
A test to determine the security of the software product.

This article will be focusing more on Functional testing, for both Progression and Regression testing.

Progression testing or incremental testing focuses on the new functionality to ensure it works according to the requirements. This type of testing is performed during active sprints.

Regression testing is a test to assure the current functions of the application are not broken from the addition of the new code or changes introduced by bug fixes.

Why Postman and TestRail

There are many tools available for API testing. Some of the examples are:

Postman
SoapUI
JMeter
Swagger

In this article, we will focus our discussion on Postman as the selected API tool and it's integration to TestRail as the TestCase Management System.

Postman is a collaboration platform for API development, simplifying each step of building an API and streamline collaboration to create better APIs. Postman provides the following benefits.

Ease of test cases creation
Easier to collaborate with
Support CI/CD integration
Cost saving
Free and Easy to Start
Wide support for all APIs and Schemas
Extensible
Support and Community

TestRail helps to manage and track software testing efforts and organize the QA department. Its intuitive web-based user interface makes it easy to create test cases, manage test runs and coordinate the entire testing process. TestRail provides the following features.

Modern Test Management
Track Test Results
Reporting & Metrics
Seamless Integration
Team & Productivity
Customizable & Scalable

Effective API testing

In today's Agile development, due to limited time, API test scenarios don't always get tested completely. The following should help testers and developers testing the API more effectively during the development phase.

API contracts are completed and prioritized.
Start creating API tests soon after API contracts are available.
Leverage API mock to validate request and response of API test scripts while waiting for development to complete.
Have a good communication between developers, business analysts, testers and clients.
Automate API testing for continuous testing with CI/CD tools such as Jenkins.

For API testing the following items are mandatory to be validated.

HTTP response code, e.g HTTP 200 OK.
HTTP header Content-Type value if any, e.g for REST API testing, Content-Type value should be application/json.
Response time, acceptable response time e.g 1000ms.
Response body, for REST API validation.
Response body schema, is a valid JSON Schema as per API contract.
Validation for each attribute name/value/data type.

API testing with Postman and TestRail

Requirement

Postman
NodeJS
Newman command line
newman-reporter-htmlextra plugin
newman-reporter-testrail plugin
TestRail account
TestRail API key
Environment Variable for TestRail integration

Installation Steps

Postman installation:
Download and install Postman here!

NodeJS installation:
Download and install NodeJS here! After successful installation, open terminal and run node -v to verify, it should show the node version.

Postman plugin installation:
To install newman, newman-reporter-htmlextra and newman-reporter-testrail, open terminal and run the following command:

          npm install -g newman newman-reporter-htmlextra newman-reporter-testrail

Docker Installation:
Required if you want to run Newman inside docker container. To install docker please follow docker installation for your platform.

Jenkins installation:
Required if you want to run a Postman test with Jenkins CI. Follow Jenkins documentation to install Jenkins and create Jenkins jobs.

TestRail Preparation

Create TestRail Test Cases:
Create API test cases in TestRail by adding test description and expected outcome.

Create TestRail Test Run:
After test cases creation completed, then create Test Run and add test cases to the test run. When Postman tests get integrated with TestRail, Postman will automatically add test results for each test case inside the test run.

Generate TestRail API Key:
To enable Postman and TestRail integration, create a TestRail API key. Login to your TestRail and create your TestRail API key.

Add Environment Variable:
Add the following environment variable to your favourite shell (i.e bash shell). Run the following command from terminal.

          export TESTRAIL_DOMAIN=<TESTRAIL_URL> export TESTRAIL_USERNAME=<TESTRAIL_USERNAME> export TESTRAIL_APIKEY=<TESTRAIL_API_KEY> export TESTRAIL_PROJECTID=<PROJECT_ID_WITHOUT_PREFIX_P> export TESTRAIL_RUNID=<RUN_ID_WITHOUT_PREFIX_R> export TESTRAIL_SUITEID=<SUITE_ID_WITHOUT_PREFIX_S> export TESTRAIL_TITLE=<OPTIONAL>

Or add to your .bash_profile or .zshrc and run source ~/.bash_profile or source ~/.zshrc to reload the environment variable.
Run echo $VAR_NAME to verify environment variable value e.g echo $TESTRAIL_APIKEY

Create API Project on Postman

Open Postman application → Collections tab → click New Collection to create a new Postman collection.
See how to use the API builder in Postman UI.

To add integration between Postman test and TestRail, add TestRail test case id to your Postman test scripts. See below example on how to add TestRail test case id.

Create tc_id variable on the first line of the test script and assign TestRail test case id.
On each pm.test add tc_id + " your test name", function() {});

Run requests from Postman UI to verify the tests scripts work and all pm.test are passed before run your Postman test from Newman command line. Here is an example of a test results report which includes test case id from TestRail test case.

Postman API Test Results — Postman API Test results

Run Postman with Newman
TestRail integration required Postman test execution from the command line with help of newman command line and newman testrail plugin. Newman is a command-line collection runner for Postman. It allows you to effortlessly run and test a Postman collection directly from the command line. Newman command line allows you to execute Postman tests from a collection file or from a collection URL.

Run from collection file:
To run the test from the collection file use the following command line syntax

          newman run <COLLECTION_FILE_NAME> --environment <ENVIRONMENT_FILE_NAME>  --reporters <REPORTER_NAME>.

E.g the following command will run collection file project-template.postman_collection.json with environment fileproject-template.postman_environment.json and reporter cli

          newman run "project-template.postman_collection.json"  --environment "project-template.postman_environment.json"  --reporters "cli"

Run from URL:
To run the test from the Postman URL use the following command line syntax

          newman run <COLLECTION_URL> --environment <ENVIRONMENT_FILE_NAME> --reporters <REPORTER_NAME>

E.g the following command will run Postman collection from URL

          newman run "https://www.getpostman.com/collections/12b32fffr6bda96f5dfc"  --environment "project-template.postman_environment.json"  --reporters "cli"

Newman with newman-reporter-testrail
After adding TestRail test case id to Postman tests script (see add TestRail test cases id and set environment variable for TestRail), add testrail reporters in --reporters parameters to enable test execution with testrail. E.g

          newman run "project-template.postman_collection.json"  --environment "project-template.postman_environment.json"  --reporters "cli,testrail"

You should see TestRail URL and test run ID at the end of the test execution. Following is an example of the output.

After test execution is completed, go to the TestRail testrun section and check the test execution result. TestRail plugin automatically updates test result, adds API request and response as a result comment. See below for the example of the test result updates in TestRail.

Newman with newman-reporter-htmlextra
We can add multiple reporters as Newman reporter parameters. To run with newman-reporter-htmlextra, cli and testrail for example, use the following command line (by default reporter will generate html report under the collection file directory or current directory)

          newman run "project-template.postman_collection.json"  --environment "project-template.postman_environment.json"  --reporters "cli,htmlextra,testrail"

To override default directory for html report add flag

          --reporter-htmlextra-export <REPORT_PATH>  e.g --reporter-htmlextra-export /home/newman/report

Below is the example of running Postman test with Newman, enable TestRail integration and generate html extra report under /home/newman/report

          newman run "project-template.postman_collection.json"  --environment "project-template.postman_environment.json"  --reporters "cli,htmlextra" --reporter-htmlextra-export /home/newman/report

To open the html report file go to report directory and open html file with browser, example of the html extra report.

Newman Tips
Override environment variable from command line:To override the Postman environment variable value from the Newman command line, use --env-var flag. E.g to override ___service_endpoint___ environment variable value

          newman run "project-template.postman_collection.json"  --environment "project-template.postman_environment.json"  --reporters "cli,htmlextra,testrail"  --env-var "___service_endpoint___=https://localhost"

To override global variable use --global-var

Disable Newman SSL validation:If you use self-signed SSL, then you need to disable SSL verification checks and allow self-signed SSL certificates using --insecure flag to be able to run Postman tests from the Newman command line. E.g

          newman run "project-template.postman_collection.json"  --environment "project-template.postman_environment.json"  --reporters "cli,htmlextra,testrail"  --env-var "___service_endpoint___=https://localhost" --insecure

See complete list of Newman command line options.

Run Newman with Docker

If you don't want to install all required dependencies on your host, you can run a Newman with docker. Let's say we create a postman_test directory containing Postman collection json and Postman environment json. Now create a simple Dockerfile with the following contents

          FROM postman/newman:5-alpine  LABEL maintainer="<YOUR_EMAIL_ADDRESS>"  WORKDIR /home/newman  RUN npm install -g newman newman-reporter-testrail newman-reporter-htmlextra

Create another file to export environment variables for TestRail, we will use --env-file flag to load environment variable to docker container, create new file e.g testrail_env under postman_test directory with contents

          TESTRAIL_DOMAIN=<TESTRAIL_URL> TESTRAIL_USERNAME=<TESTRAIL_USERNAME> TESTRAIL_APIKEY=<TESTRAIL_API_KEY> TESTRAIL_PROJECTID=<PROJECT_ID_WITHOUT_PREFIX_P> TESTRAIL_RUNID=<RUN_ID_WITHOUT_PREFIX_R> TESTRAIL_SUITEID=<SUITE_ID_WITHOUT_PREFIX_S> TESTRAIL_TITLE=<OPTIONAL>

Now build docker image called newman_testrail with tag 1.0 from previous Dockerfile using the following command (ensure that postman_test is your current working directory)

          docker image build -t newman_testrail:1.0 .

After docker image created successfully, then run

          docker container run --env-file="testrail_env" -v `pwd`:/home/newman newman_testrail:1.0 run "project-template.postman_collection.json" --environment "project-template.postman_environment.json" --reporters "cli,htmlextra,testrail"

A better solution to run your Postman with docker is using docker-compose.

Run with Jenkins for CI

To integrate with Jenkins for CI, create Jenkins jobs and use Newman command line from section run Postman with Newman

Integration Issue Troubleshooting

TestRail plugin connects to 127.0.0.1 when you run Newman with docker. This issue might happen when the docker container is unable to load the environment variable from host. Run below command to list all available environment variable inside a container

          docker container run --env-file="testrail_env" -it --entrypoint /bin/sh -v `pwd`:/home/newman newman_testrail:1.0 -c 'env'

TestRail plugin returns error code 401 Authentication failed.
Ensure TESTRAIL_APIKEY contains correct TestRail key.

Troubleshoot 401 error during Testrail integration

TestRail plugin returns error code 400, the run id is not a valid test run.
Ensure TESTRAIL_RUNID has the correct TestRail run id.

Troubleshoot 400 error during Testrail integration

Conclusion

Postman, Newman and TestRail can be used for progression and regression testing to speed up API test execution, and can be integrated with CI/CD tools such as Jenkins for automation, scheduler, etc.

Reference

ISTQB glossary
Postman documentation
TestRail documentation
Docker documentation
Newman TestRail reporter
Newman HTML Extra reporter

stoutmorts1960.blogspot.com

Source: https://icehousecorp.com/leveraging-postman-and-testrail-for-progression-and-regression-api-testing/