Open Credo

March 14, 2016 | Software Consultancy

Test Automation Concepts – Parallel test execution

Test automation provides fast feedback on regressions. In order to achieve this tests need to execute quickly, something which becomes more of a problem as test suites grow. This is especially true of tests which exercise a user interface where the interaction with the system is slower.

A good way to address this is to have your tests execute in parallel rather than consecutively. Given sufficient resources this allows your execution time to remain low almost indefinitely as more scenarios are added to the suite.

WRITTEN BY

Tristan Mccarthy

Tristan Mccarthy

Test Automation Concepts – Parallel test execution

This post is part of a series which introduce key concepts in successful test automation. Each post contains sample code using the test-automation-quickstart project, a sample Java test automation framework available from Github. For a full list of concepts introduced so far, see the parent post “Introducing the test automation quick start project”.

Reducing test feedback time with parallel test runs

Test automation provides fast feedback on regressions. In order to achieve this tests need to execute quickly, something which becomes more of a problem as test suites grow. This is especially true of tests which exercise a user interface where the interaction with the system is slower.

A good way to address this is to have your tests execute in parallel rather than consecutively. Given sufficient resources this allows your execution time to remain low almost indefinitely as more scenarios are added to the suite.

In this post we will demonstrate the implementation of parallel test execution with the test-automation-quickstart project. We will modify the project to run Cucumber-JVM features in parallel and spin up multiple browser windows to interact with a web based system. The approach used here was introduced in a previous post on this blog, but the process has been simplified thanks to the release of the Cucumber-JVM parallel plugin released by temyers on Github.

Getting Started

To begin with download the test-automation-quickstart project from the OpenCredo GitHub account. Once done, follow the instructions on the ReadMe and ensure that you have the required dependencies installed (Maven, FireFox). For those who want to dive straight in at this point, you can run the code with the following console command from the project root directory:

This will result in some simple example tests running, including several browser integration tests running concurrently.

Before we get into running tests consecutively we’ll spend some time on the basics of how the test execution is set up. If you are familiar with Cucumber-JVM and Selenium Webdriver you can skip straight to Implementing Parallel Execution.

Defining A Test

The first step is to ensure that you have described the behaviour that you are verifying. In our case, we are describing functionality in natural language using Cucumber. For our start point we’ll use a simple test which searches on Google and checks that the result contains an expected result. This tests lives in a feature file in our test resources.

Implementing Test Automation

Next up is to begin implementing the automation for the described test case. In the Cucumber world this consists of creating a step definition class which contains “glue code” to link the natural language statements to a Java method. For each statement (Step) in the test case, we have a matching method which is linked via a regular expression match.

Interacting With The System

The glue code above doesn’t interact directly with the system, but instead calls methods on a GoogleSearchPage class. This is an example of the Page Object pattern widely used in automation frameworks. The page object is a representation of a webpage where methods for common user interaction with the page can be kept. While not necessary this structure makes for far more maintainable test code. Our page object class contains a couple of simple methods for initiating a search and subsequently checking the search result. The interaction is via Selenium Webdriver using some common methods.

As a side note, the GoogleSearchPage object is included in our step definition class via Spring dependency injection – something which is out of the scope of this post.

Implementing Parallel Execution

Now that we’ve gone through the basics of how the test is running we can move on to the project configuration which is specific to running multiple tests consecutively.

The first part is the configuration of the Surefire plugin in the Maven POM. This plugin runs our test classes, and with a small piece of configuration can be told to start up multiple JVM’s to run the tests. The parameter “acceptance.test.parallel.count” determines the number of parallel threads to run, and is defaulted to 2 in the parent POM.

In the above configuration you can see the “include” parameter which contains the property “**/*IT.class”. This means that any classes with a name which ends in “IT” will be run as a test. We have not yet defined any of these test runner classes, and this is where the Cucumber-JVM parallel plugin mentioned earlier comes into play.

This method of running Cucumber-JVM tests in parallel was first introduced several years ago in a post on this blog, at which point we manually created a test runner class for every feature that we had defined. Several users of the approach commented that this became unwieldy and difficult to maintain for larger test suites as the number of test runners grew. Given that these test runners each contained only a small amount of boilerplate code it was an unfortunate limitation. Fortunately Github user temyers has since addressed this problem by releasing a maven plugin which automatically generates a test runner class for each feature file in your test suite at run time, removing the requirement that these are manually maintained. This is added and configured in the Maven POM along with the Surefire plugin.

The above configuration results in a runner class being created for every feature file. Any tests marked with the tag @ignore (more on tags later) will not be run. At run time, these runner classes will be picked up by the Surefire plugin and used to execute the features in parallel (within the limitations of the configured thread count).

The final piece in the demonstration of parallel test runs is to create another feature file which can run at the same time as the one we have already defined. In this case we have simply duplicated our SimpleUiTest.feature file as ParallelUiTest.feature.

You can now execute the tests via Maven and observe that multiple browser windows are created and interacted with in parallel. Bear in mind that with this simple example it can sometimes be difficult to see the benefit of parallel execution due to the overhead of starting up a browser session. Using this approach in a real codebase will very quickly demonstrate the improvement in execution time.

Running A Subset Of Tests

We briefly touched on the idea of tags earlier, a basic concept in Cucumber’s Gherkin language. Simply put, a tag is a text string with an @ symbol, which can put put either at the top of a feature file to apply to all scenarios or an individual scenario. When running Cucumber tests it is then possible to pass through tags to specifically include or exclude, in our case this is handled via the cucumber-jvm-parallel plugin configuration. Any given feature or scenario can have multiple tags.

In the above example, our feature is tagged as @ui-demo-simple and @ignore. The cucumber-jvm-parallel-plugin configuration includes the entry “~@ignore” in the “tags” parameter, where ~ indicates a negative, i.e. do not run scenarios with the tag @ignore.

There are a few situations where you might want to be able to run a subset of tests, and this can be done with the tagging functionality. A common example is a set of fast running smoke tests which can be run ahead of the main set in order to ensure that the system under test is not fundamentally broken. Adding the tag @smoke to the appropriate tests, you can update the cucumber-jvm-parallel-plugin configuration:

This will now run tests tagged as @smoke which aren’t also tagged as @ignore. Given that this is a maven property, the values can easily be passed in at run time with command line arguments.

Reporting

UI acceptance tests result in a HTML report for each feature in test-automation-quickstart/ui-acceptance-tests/target/cucumber-parallel/. With parallel execution you no longer get a single unified report. Instead a report is generated for each individual feature and will need to be combined for reporting on the health of an entire application.

If your tests run as part of a continuous integration pipeline, these individual reports can be automatically joined using plugins such as the Jenkins Cucumber-JVM-Reports plugin.

Without access to a plugin which handles this you could write a simple piece of code that merges the report files together – this would work better if using the JSON output (change “format” parameter in the cucumber-jvm-parallel plugin to “json” instead of “html”).

RETURN TO BLOG

SHARE

Twitter LinkedIn Facebook Email

SIMILAR POSTS

Blog