Open Credo

March 23, 2017 | Data Engineering, Machine Learning

Automating Your Security Acceptance Tests

On previous blog posts we have provided examples of different types of acceptance tests coverage, UI, API and Performance. One area where automation is often lacking is around validating the security of the application under test. This has been discussed in the post on non functional testing You Are Ignoring Non-functional Testing. With this post we will enhance the automation framework to quickly check for some common security flaws.


Carlos Lindo

Carlos Lindo

Automating Your Security Acceptance Tests

This post is part of a series which introduces key concepts in successful test automation. Each post contains sample code using the test-automation-quickstart project, a sample Java test automation framework available from Github. For a full list of concepts introduced so far, see the parent post Introducing the test automation quick start project.

Increase test coverage with security tests

There are a few free tools out there that can give you a reasonable feedback on security issues, for example Vega, Wapiti or W3af. I have found that the Java client API for OWASP ZAP is really easy to integrate into our Test Automation Quickstart project.


To run the security tests we need to have OWASP ZAP running on our local machine. You can download OWASP ZAP from here. The security tests use OWASP ZAP version 2.5.0.

Getting started

Our test automation framework is available on Github. Follow the setup and requirements carefully.
The Maven build will first install dependencies, and then run the tests. There are a few assumptions that I made and you can change if you like. One is that the Zed Attack Proxy will run on your localhost using port 8888 (if this port is already used you can use any other free port), and the other, to keep it simple, is that the Java client API will not be using any API key. You can change these, please read OWASP ZAP documentation and make sure you change the security tests code accordingly.


The security tests can be found on a new module inside the project called security-acceptance-tests. The examples in this module closely match those in the UI module, with the addition of verification for security requirements. The security requirements that we will be verifying are business risks, you can read more about what these are on OWASP Risk Rating Methodology document.

Feature: Demonstrate the Security test framework

 Scenario: Simple security test
   Given I am on the Google search page
   When I search for "OpenCredo"
   Then the site "" should be present in the results
   And the number of risks per category should not be greater than
     | low | medium | high |
     | 30  | 10     | 0    |

To run the tests you first need to start OWASP ZAP on your local machine, this can be done using the following command:

./ -daemon -port 8888 -config api.disablekey=true.

If your tests are running on a CI/CD tool you may want to configure your job to start the OWASP ZAP before your tests run. If you are using Jenkins there is a ZAP plugin that can handle the proxy start and shutdown procedure within a job.

Once ZAP is running you just have to go to test-automation-quickstart directory and run:

mvn clean install -Psecurity-acceptance-tests


Security acceptance tests are configured to generate two HTML reports. The security related report can be found in test-automation-quickstart/security-acceptance-tests/security-reports/security-report.html. It will show how many risks were identified, their severity and a long description about each one of them. There is also a Cucumber tests report for each feature that was run in test-automation-quickstart/security-acceptance-tests/target/cucumber-parallel/.

Adding more coverage

With this example as a starting point, you can create other BDD scenarios that focus on validation of a specific security problem, or security risk. You can also try a different approach such as the OWASP spider attack.

For further reading about the topic, read the ZAP client api documentation. You could also try the other tools that I mentioned before: Vega, Wapiti or W3af.


This blog is written exclusively by the OpenCredo team. We do not accept external contributions.



Twitter LinkedIn Facebook Email