Addressing slow test runs when using Cucumber JVM with web automation tools
UPDATE March 2016: While the content of this post still accurately describes an approach to running cucumber scenarios in parallel, the associated sample code is by now somewhat out of date. This topic has been revisited in a new blog post which incorporates new tools and up to date versions. The sample code for the new post is part of the quick-start test framework and as such is being actively maintained.
If your team is using continuous integration this becomes especially noticeable, forcing teams to either wait for acceptance tests to complete before deploying or having to ignore the bulk of the tests.
The sample code and description in this post will show you how to convert your suite to running tests in parallel, something which has historically been problematic with unanswered questions and outstanding Cucumber-JVM bugs on the subject. Tying the described approach in with Selenium Grid2 will allow you distribute your testing across several machines if your suite is especially large or slow.
The full source code for this example can be found on Github.
Once you have retrieved the source, you can run by navigating into the Cucumber-JVM-Parallel directory and issuing the command:
mvn clean install
This will run the example project and start two browser windows concurrently. Each browser window corresponds to a Cucumber feature file in the example. Once the run is complete, a report will have been generated at /target/cucumber-report/index.html.
The rest of this post will go into further detail on the structure of the example project, though it is assumed that you have some prior experience of Cucumber. If not, the Cucumber-JVM readme is a great place to start.
The first thing you need is your feature files to describe the behaviour you expect. In this example, we have two separate features, though you can also run scenarios within a single feature in parallel.
The way we can do is is by using Cucumber tags, which can be applied either to all scenarios in a feature or to individual scenarios.
Above, you can see that we have two feature files. These reside in the ‘src/test/resources’ folder. Each feature file is tagged (@autocorrect and @search), and contains a single scenario.
Now that we have our scenarios, we need to add some glue code to tie each step into our underlying test framework
These are referred to as step definitions, and can be found in ‘src/java/cucumber.jvm.parallel/cucumber/stepdefs’.
In the above snippet, you can see that we use an instance of ShareDriver to communicate directly with a browser window. This is based on one of the cucumber examples for sharing a single browser session between all tests (using dependency injection) to remove the need for starting up a browser instance per test and thus speed up execution. In our case, this results in one browser session per thread. The ShareDriver class can be found in ‘/src/test/java/cucumber.jvm.parallel/cucumber’.
The snippet also shows that we use an instance of ‘SearchPageObject’, which is simply a class which represents the Google search page, found in ‘/src/test/java/cucumber.jvm.parallel/pageobjects’.
This is not required, but it’s good practice to use the page object pattern for ease of maintainability on a larger project.
Above, you can see that the page object contains identifiers for elements on the page as well as methods specific to that page.
The next stage in the process is adding the test runners. We are using JUnit as opposed to the CLI, and this is where we need to start structuring things specifically to handle parallel running of tests.
In the above snippet from ‘SearchAT.class’ you can see that we are specifying the location of the feature files. We are also specifying a tag (@search) which relates to one of our cucumber feature file tags and a html report destination for test results.
What this says is “run all tests tagged as @search and write results to /search folder”.
We then have another class, ‘AutoCorrectAT’ which does the same for all tests tagged ‘@autocorrect’. Both of these classes can be found under ‘/src/test/java/cucumber.jvm.parallel/cucumber’.
Adding another thread is simply a case of adding a new runner class with a different tag.
Parallel Test Runs
Up to this point, the instructions are identical to creating a relatively simple non-parallel set of Cucumber-JVM tests using WebDriver to interact with a website.
We now need to go to the Maven POM file to see how we are making the tests run in parallel.
In the above snippet, you can see that the maven-surefire-plugin is used to run our acceptance tests – any classes that end in *AT will be run as a JUnit test class. Thanks to JUnit, making the tests run in parallel is now a simple case of setting the forkCount configuration option. In the example project, this is set to 5, meaning that we can run up to 5 threads (ie, 5 runner classes) at a time.
Running the test suite at this point would result in our two feature files being run in parallel, with their results being written out to separate reporting directories. For ease of reporting, our final step is to merge these reports into a single file. To do this, we have some code in ‘/src/main/java/cucumber.jvm.parallel/ReportMerger’. This is executed by the exec-maven-plugin in the Maven POM.
The report merger code consists of the following steps:
For a given report directory, go through each sub directory:
Rename all embedded images to have a unique name (without this there would be clashes between different reports being merged).
Update the report.js file to reflect the new embedded image file names.
Copy renamed images to root report directory.
Merge report.js into file in root report directory.
The result of this is a single index.html file in the root report directory which consists of all subreports. The subreports are left in place, but these could easily be deleted at this point if you are archiving test results on your CI server and wish to save space.