Cucumber Android vs Appium with cucumber jvm

Android applications

Before starting to write any Android test, you will need to have the code for the application that you want to test. When you create a test project, you will be asked to provide the target application that you want to write your tests for and you will have to select the target project.

I developed my test project using a simple Android application that you can find here, under the name of TestApp. You can download and use this application as the target project and develop your own tests using it.

To understand more about the terminology used in this example, you should know that an Android application consists of multiple screens which users can interact with, such as send an email, view new emails, take a photo or view a map. Each screen that a user can interact with is called an activity. Each activity is given a window in which its user interface is presented.

An application usually consists of multiple activities that are loosely bound to each other. Typically, one activity in an application is specified as the “main” activity, which is presented to the user when launching the application for the first time. Each activity can then start another activity in order to perform different actions.

Setting up the test environment

Before starting to write the tests, you should review the application UI. For this, you need to set up your test development environment. The easiest way is to use Eclipse (with the ADT plugin). However, for your own test development, you are free to use the IDE of your choice or the command-line. If you would like to use another IDE please refer to this documentation.

The first step is to download the Android SDK. This includes a version of the Eclipse IDE with built-in ADT (Android Developer Tools). If you want to use a different Eclipse, you will need to download Eclipse separately and add ADT plugin for Eclipse. To import the Android application project that you want to test against, in Eclipse, go to: File -> Import -> Android – Existing Android code and browse the application project.

Run the application

To see what the application does, you will need to run it. Before being able to do so, you need to configure a device to run the application on. This post will demonstrate running an application on an emulator, though you can also use a physical device. If you would like to read more about how to run the application on a real device, please refer to this documentation.

You will need to configure an Android Virtual Device (AVD). In Eclipse, go to Window -> Android Virtual Manager. In the Android Virtual Device Manager panel, click New… and fill in the details requested (name, platform target, SD card size). After creating the emulator, select it from Android Virtual Device Manager panel and click Start… . Be patient, as it will take a while for the emulator to start. After it starts, unlock the screen. The applications won’t run if the screen is locked.

Now you can go back to the project that you imported previously, right click on it, and select Run As – Android application. The application should run now on the emulator with the main application screen displayed. You can enter text and press the Send button, the text that you entered will be shown in the next screen (activity).

At this point the application code is compiled to a .apk file and deployed to the emulator. The file is created in the /bin folder on your project directory. If you navigate to the apps on your emulator you will find the TestApp application. You can open it and use it.

You can see that the application has a main activity that is presented first, displaying a text field and a button. Pressing the button will trigger the second activity. If you look for a moment in the code, you can see that under src/com/example/testapp package there are two classes that describe the two activities mentioned (MainActivity and DisplayMessageActivity classes).

Create your tests

Now that you got a bit more familiar with the application and the terminology, you can create your test project. The following diagram summarizes how your tests interact with the emulator :

If you want to go quicker through the post, you can download the test project TetsingMyAndroidAppTest found here, import it in Eclipse in the same way you did with the application and read the below information having the code in front of you to fully understand it. The test automates the scenario where a user enters a text, presses Send button and verifies that the text appears correctly on the next activity.

If you want to create a new test project, you need to go to: File -> New -> Other -> Android – Android test project. When asked about the test target, the application project that you previously imported will appear in the list. Select it and click Finish.

At this point you should have your test project created. The test project has automatically created an AndroidManifest.xml file, visible in the main directory, which presents essential information about the application to the Android system, information the system must have before it can run any of the application’s code. Among other things, the manifest does the following:

• It names the Java package for your test application. The package name serves as a unique identifier for the application. By convention, your test package name should follow the same name as the application package, suffixed with “.test”.

  package="com.example.testapp.test"

• It lists the Instrumentation classes. Instrumentation classes run the tests against an Android package (application). The target package of the application project also appears declared in this section.

  android:name="android.test.InstrumentationTestRunner"
  android:targetPackage="com.example.testapp"

• It declares the minimum level of the Android API that the application requires. You can also add the target level that you wish your application to be tested against, which will be probably the latest Android API available.

Now you can move on and create your first test. You need to create a test class under src/com/ example/testapp/test package. To create a test class, right click on the package name of your test project and create a new Java class. Give it a name. By convention, the test classes should be named Test (for example, for testing MainActivity, the test class will be named MainActivityTest). In the superclass field, browse and look for ActivityInstrumentationTestCase2, where is the name for the activity you are testing. If you want to test another activity, specify its name. The ActivityInstrumentationTestCase2 class is designed to do functional testing of one or more Activities in an application. Click finish. This is what your class should look like at this point:

public class MainActivityTest extends ActivityInstrumentationTestCase2{

For this class, you will need to add a constructor with no parameters, that just calls the constructor of the base class, specifying the activity you are testing:

public MainActivityTest() {
super(MainActivity.class);
}

When creating your first test method, name it test.

public void testAndroidApp() {  …  }

Android tests are based on JUnit. You can use JUnit assertions to verify the output of your tests. You will need to call the setUp method for the superclass, as requested by JUnit and you will need to start an activity,

super.setUp();
mainActivity = getActivity();

identify the elements that you want to interact with (for example text fields, buttons),

textView = (TextView) mainActivity.findViewById(R.id.edit_message);
button = (Button) mainActivity.findViewById(R.id.button);

interact with those elements

//enter a text in the text field
mainActivity.runOnUiThread(new Runnable() {
@Override
public void run() {
textView.requestFocus();
textView.setText(textToEnter);
}
});
//click Send button
button.callOnClick();

and verify that the output is the expected one.

assertEquals("The displayed text is not correct", textToEnter, textView.getText().toString());

You will have to declare the objects you are using:

private MainActivity mainActivity;
private TextView textView;
private Button button;
private String textToEnter = "test text";

One of the most important aspects of testing is identifying the elements that you want to interact with. In this example, the elements are identified by their ids that are used when creating the main application. If you look at gen.com.example.testapp.R class from the main application code you will see the ids for the textview and the button, both which are being used in this example. Using Eclipse you will have available the autocomplete, that will suggest you the available options as you create your test.

@SuppressLint(“NewApi”) is an annotation used by the Android Lint tool. Android Lint is a new tool introduced in ADT 16 (and Tools 16) which scans Android project sources for potential bugs. Lint will tell you whenever something in your code isn’t optimal or may crash. By passing NewApi, you’re suppressing all warnings that would tell you if you’re using any API introduced after your minSdkVersion

Running your test

To run your test from Eclipse you need to right click on the test class, select Run As and then choose Android JUnit test. You will see the tests running on the emulator. Remember to have the screen of the emulator unlocked before running the tests. Eclipse will show the results of the test in the JUnit view, next to the Package Explorer tab.

If you encounter any problem with your test or if you would like to read more details about Android test projects, please refer to Android documentation.

Adding BDD

BDD stands for Behaviour Driven Development. It is a popular agile testing concept, related to TDD (Test Driven Development). It adds a natural language layer on top of test automation. It is used for capturing system requirements clearly, so that the development team and the business stakeholders have a better understanding of the scenarios that will be tested.

Cucumber is a BDD tool. It uses terminology like: features, scenarios and steps. The term “feature” is largely interchangeable with an agile “story”. A feature normally consists of a small piece of behaviour which extends the system in some way. A scenario is a test case. A feature consists of a number of scenarios which, when grouped together, express all of the behaviour that the feature is intended to cover. Each scenario reflects a use case or “what if” situation. A step is an action within the scenario. Each scenario will consist of multiple steps which form one of three types – Given, When or Then, which represent the preconditions, user actions and the expected output.

You can find a sample code for an Android Cucumber test in my repository.

The scenarios are linked to the execution code with the help of java classes that use regex expressions for matching. For a step that says : “Given I am launching the application”, you will have a class with the following method:

import cucumber.api.java.en.Given;
@Given("^I am launching the application$")
public void I_am_launching_the_application() throws Exception {  …  }

You will find the same rules for any step of the scenario.

Once you have the test described above passing you can enhance it with Cucumber. Adding Cucumber on top of the existing android test using Cucumber-Android is a fairly new topic so the documentation is limited. I will try to give as many details as possible in this post so you end up achieving your goal of adding Cucumber on top of your tests.

Below you will find all the details necessary for using Cucumber.

In your Android test project, you will need to do the following:

1. Start by opening AndroidManifest.xml file from your test project. Inside tag, change android:name from:

into:

2. Create a “libs” folder in your projects directory, containing all the cucumber dependencies. The following dependencies need to be added:

• cucumber-core

• cucumber-java

• cucumber-android

• cucumber-html

• cucumber-junit

• cucumber-jvm-deps

• picocontainer

• gherkin

The cucumber reporting is available only for 1.1.6 version of cucumber-jvm, which at this date is not released yet. To be able to see cucumber reports, you will need to add 1.1.6.SNAPSHOT versions to the above libraries, except cucumber-html, cucumber-jvm-deps, gherkin and picocontainer. To do so, go to cucumber-jvm project, clone it on your computer and build it with maven (maven clean install). This will download the SNAPSHOT jars that you need to add to your “libs” folder.

To download the latest jar for the libraries that do not need the SNAPSHOT version, you can go here and search for the latest jars and download it using this url.

After adding the libs folder to your project, go to Eclipse, right click on the project name and click refresh. Libs folder should be displayed now.

3. Using BDD (Behaviour Driven Development) means that you have feature files, where you describe the scenarios that you want to tests. The feature files should to be added under: assets/ features. Create the ‘features’ folder under the existing ‘assets’ folder and then add your feature files. For example, myFeatureFile.feature will have the following path: assets/features/myFeatureFile.feature. For my test I used the following feature file:

Feature: enter a text and verify the output
Scenario: Verify that the text entered is displayed correctly
   Given I am launching the application
   When I submit a text
   Then I receive the same text on the next page

4. You will need now to add the class that interprets the scenario steps. You need to create a new java class in your test project, under src.com.example.testapp.test package, that extends ActivityInstrumentationTestCase2. The class will look like this:

public class MainActivityStepDefs extends ActivityInstrumentationTestCase2{

@Given("^I am launching the application$")
public void I_am_launching_the_application() throws Exception { … }

@When("^I submit a text$")
public void I_submit_a_text() throws Exception { … }

@Then("^I receive the same text on the next page$")
public void I_receive_the_same_text_on_the_next_page() throws Exception { ... }
}

Inside the methods, you will use the same code that you used previously, but split into the above methods. Copy the code used in the above example and and paste it inside this methods.

5. You will need to add a runner class, that will run your new Cucumber project. Add a new Java class in the same package that you added the class described above. The class name which will have to end with “Test”. The class will look like below:

import org.junit.runner.RunWith;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
@RunWith (Cucumber.class)
@CucumberOptions(format = {"pretty", "html:/data/data/com.example.testapp/report"}, features = "features")
public class RunTest {
}

In the first part of @CucumberOptions you define the test report location. In this case, they will be stored on the emulator, on /data/data/com.example.testapp/report path. The second part mention the name of the folder where the features are stored. By default, it looks in the “assets” folder, so you just need to specify the “features” folder.

If you want to run your tests from the command line, then you can use @CucumberOptions(features = “features”) annotation, and specify the location of the reports in the command line.

6. Because Android tests are based on JUnit, you will have to add the JUnit library to your both projects. To do so, right click on the project names -> Properties -> Java build path -> Libraries tab -> Add library -> JUnit -> Next -> Finish.  Go to Order and Export tab and check the box for JUnit 4. Make sure you do the same both for the main application and for the test project.

Run the cucumber test

Your test will be run as an Android JUnit test. In order to run your cucumber tests you need to right click on the runner class, go to Run Configurations and select cucumber.api.android.CucumberInstrumentation from the dropdown list that says “Instrumentation runner”. Otherwise the tests will try to run with the default runner, and you will probably see the following error:

“ does not specify a android.test.InstrumentationTestRunner instrumentation or does not declare uses-library android.test.runner in its AndroidManifest.xml”

After running the tests, you will want to see the cucumber reports. This are now stored on the device. To be able to extract them on your computer, open a terminal window, go to /platform-tools and type the following command:

./adb pull /data/data/com.example.testapp/report cucumber-report-android

This will extract the reports from your emulator into cucumber-report-android folder, under /platform-tools.

Conclusion

Android framework

To write the tests using the Android framework you need the code of the application you want to test. It is easy to setup the environment and to write the tests, by identifying the elements through their ids, although the name of some methods is not intuitive (for example, to click on an elements, you need to use the callOnClick() method instead of a usual click() method).

Cucumber-Android

Using Cucumber with the Android framework is a relatively new topic and so the documentation is limited. When looking for solutions for errors, sometimes there are no documented answers. At this point, the best chance to get answers is to post your errors in forums. The forums are active, and you will get an answer quickly, though responses tend to vary in quality.

A big downside for using Cucumber with the Android framework is that the documentation is poor and the test reporting is not available yet. The version of the library that supports cucumber reports is not released yet. In this example I used a SNAPSHOT version to get the reports, and not a stable released version.

Appium with cucumber-jvm

With Appium is easy to write tests both for Android and iOS. The same test code is used. When switching from testing one platform to the other you only need to change a few lines of configuration code. Cucumber reports are generated, which represent a big advantage of using Appium with cucumber-jvm, as they represent one of the most important reasons for using cucumber.

However, it can be inconsistent sometimes. The same tests pass or fail at different run times without having the code changed.

Recommendation

At this point, I would recommend using Appium with cucumber-jvm, because it uses the same code as testing web pages, it has support both for Android and iOS, it has recording possibilities, cucumber reports are being generated correctly. The community is active and there is enough documentation available. For me personally it was easier to write Appium tests, having the web testing background. Writing Cucumber-Android tests required more time to learn as there were different terminologies to assimilate and less documentation available. Finding a solution for error messages often came with no result, finding myself in a position of starting all over again or trying possible solutions that I could think of. Also, the biggest disadvantage is represented by the cucumber-reports that are not supported in the released version, and using a SNAPSHOT version is not stable enough for using it in your tests.

This blog is written exclusively by the OpenCredo team. We do not accept external contributions.