This post is part of a series which introduces key concepts in successful test automation. Each post contains sample code using the test-automation-quickstart project, a sample Java test automation framework available from Github. For a full list of concepts introduced so far, see the parent post “Introducing the test automation quick start project”.
Raise your test coverage with automated email testing
Acceptance test suites generally are used for UI and API testing, and we have covered both these approaches in our Test Automation Quickstart project. However, an application may, for example, send registration or expiration warning emails. Often, tests related to this are left to manual testing, instead of putting them into an automated test suite.
However, there’s no need to check emails manually: it suffers from all the same problems as other manual testing. It’s slow, expensive, and inconsistent. There are many libraries available to interact with email through code – this post will focus on how to use them within an automated test suite.
Our test automation framework is available from Github. Follow the setup and requirements carefully.
The tests are automatically configured to use Gmail, with a fake testing account we have set up for display purposes. Feel free to change the following details to your own testing accounts:
./test-automation-quickstart/acceptance-tests-common /src/main/java/com/opencredo/test/emailAdaptor.java private static final String IMAP_HOST = "imap.gmail.com"; private static final String IMAP_PROTOCOL = "imap"; private static final String SMTP_HOST = "smtp.gmail.com"; private static final String SMTP_PORT = "587"; private static final String EMAIL_ADDRESS = "firstname.lastname@example.org"; private static final String PASSWORD = "Testing123!"; private static final int MAX_RECENT_MESSAGES_TO_SEARCH = 1000; private static final String INBOX_FOLDER = "INBOX";
The details belong to the email account expected to receive emails from the system under test.
If using Gmail for your test email account, you may face problems with authentication and need to allow less secure apps to access your account – see this article for further information.
Examining the tests
On the high level, there is a feature file within the API tests which exercises some simple email functions:
./test-automation-quickstart/api-acceptance-tests /src/test/resources/cucumber/emailApiTest.feature Feature: Demonstrate the email API acceptance testing framework Scenario: Simple email interaction Given test emails have been deleted from my account When I send a test email Then there should be '1' test email in my inbox
It is important, as with all automated tests, to make the test repeatable. In this simple test, the teardown (deleting previous emails) happens right before the test starts. This isn’t always an ideal solution, and you may want to:
- Generate a random email and refer to it by an alias throughout your test. This means that individual emails can be located and referred to by a human readable name. The random generation ensures that the tests will be idempotent – so we do not need to delete any. My colleague Tristan has covered how to do aliasing in his post Test Automation Concepts – Test Data and Aliases.
- Automatically removing the test emails after the suite completes – this can be done in cucumber by using an after hook.
In our quickstart project, we have used JavaMail. Most modern languages should have an email library that you can use in your test framework.
Our dependencies are handled by Maven, so we add this to our pom file:
./test-automation-quickstart/pom.xml <dependency> <groupId>javax.mail</groupId> <artifactId>mail</artifactId> <version>1.4.4</version> </dependency>
Most of the useful abilities lie within the email Adaptor class, where the email configuration details are. Within here, you have examples of how to:
- Send emails
- Find and read emails
- Delete emails
Keeping your inbox under control
When first implementing email testing on one project, I came across a test email inbox with more than 150k messages, mostly automatically created and now useless. This slows down the test code a lot, as it searches through large numbers of emails. The test in the feature file above shouldn’t suffer from this, as it cleans up as part of each run, but if you wish to use aliasing, or you use the inbox for more than just test emails, then you will need to make sure your inbox doesn’t get too full.
Here are some ways to deal with this:
- Using a paid for service like Mailinator, create and access a random different email address per test (under your custom domain), which is then automatically wiped each night for you
- As part of your test suite teardown, delete old emails over a certain amount – say, keep the latest thousand or so email messages (or whichever you prefer). Exercise caution when doing this method – it should always be on a test email account where you don’t mind losing old data.
- Instead of a blunt approach like above, specifically, delete the test data you’ve created as part of the test suite teardown (tearing it down as part of the test steps is discouraged – if your test fails, it will not delete the email).