Android Testing Tutorial: Unit Testing like a True Green Droid

As experienced app developers, as the applications we develop mature, we get a gut feeling that it is the time to start testing. Business rules often imply that the system has to provide stability throughout different releases. We also ideally want to automate the build process and publish the application automatically. For this, we need Adnroid testing tools in place to guarantee that the build is working as expected.

Tests can provide the extra level of confidence about things we build. It is difficult (if not impossible) to build a perfect, bug-free product. Therefore, our goal will be to improve our odds of succeeding on the market by setting up a test suite that will quickly spot newly introduced bugs in our application.
When it comes to Android, and the various mobile platforms in general, app testing can be a challenge. Implementing unit tests and following principles of test-driven development or similar can often feel unintuitive, at the least. Nonetheless, testing is important, and shouldn?t be taken for granted or ignored. David, Kent and Martin have discussed the benefits and pitfalls of testing in a series of conversations between themselves compiled in an article titled ?Is TDD dead??. You can also find the actual video conversations there and get more insight if testing fits your development process and to which extent could you incorporate it, starting now.

In this Android testing tutorial I will walk you through unit and acceptance, regression testing on Android. We will focus on the abstraction of the unit of tests on Android, followed by examples of acceptance testing, with the focus on making the process as fast and simple as possible to shorten developer-QA feedback cycles.

Should I Read It?

This tutorial will explore the different possibilities when it comes to testing Android applications. Developers or project managers who want to better understand the current testing possibilities of the Android platform can decide using this tutorial if they want to take any of the approaches mentioned in this article. However, this is no silver bullet, as the discussion involved in such a topic inherently varies from product to product along with deadlines, codebase quality of code, level of coupling of the system, developer?s preference in architecture design, projected lifespan of the feature to test, etc.

Thinking in Units: Android Testing

Ideally, we want to test one logical unit/component of an architecture independently. This way we can guarantee that our component works properly for the set of inputs that we expect. The dependencies can be mocked, which will enable us to write tests that execute fast. Furthermore, we will be able to simulate different system states based on the supplied input to the test, covering exotic cases in the process.

The goal of Android unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits. ?Wikipedia

Robolectric

Robolectric is an Android unit testing framework that allows you to run tests inside the JVM on your development workstation. Robolectric rewrites Android SDK classes as they?re being loaded and makes it possible for them to run on a regular JVM, resulting in fast test times. Furthermore, it handles inflation of views, resource loading, and more stuff that?s implemented in native C code on Android devices, making the need for emulators and physical devices to run automated tests obsolete.

Mockito

Mockito is a mocking framework that enables us to write clean tests in java. It simplifies the process of creating test doubles (mocks), which are used to replace the original dependencies of a component/module used in production. A StackOverflow answer discusses about the differences between mocks and stubs in fairly simple terms that you can read to learn more.

// you can mock concrete classes, not only interfaces
 LinkedList mockedList = mock(LinkedList.class);

// stubbing appears before the actual execution
when(mockedList.get(0)).thenReturn("first");

// the following prints "first"
System.out.println(mockedList.get(0));

// the following prints "null" because get(999) was not stubbed
System.out.println(mockedList.get(999));

Additionally, with Mockito we can verify if a method has been called:

// mock creation
List mockedList = mock(List.class);

// using mock object - it does not throw any "unexpected interaction" exception
mockedList.add("one");
mockedList.clear();

// selective, explicit, highly readable verification
verify(mockedList).add("one");
verify(mockedList).clear();

Now, we know that we can specify action-reaction pairs that define what happens once we execute a specific action on the mocked object/component. Therefore, we can mock entire modules of our application, and for each test case make the mocked module react in a different way. The different ways will reflect the possible states of the tested component and mocked component pair.

Unit Testing

In this section, we will assume MVP (Model View Presenter) architecture. The activities and fragments are the views, models being the repository layer for calls to the database or remote services, and presenter being the ?brain? that binds all these together implementing specific logic to control views, models, and the flow of data through the application.

Abstracting Components

Mocking views and Models

In this Android testing example, we will mock views, models, and repository components, and we will unit test the presenter. This is one of the smallest tests, targeting a single component in the architecture. Furthermore, we will use method stubbing for setting up a proper, testable chain of reactions:

@RunWith(RobolectricTestRunner.class)
@Config(manifest = "app/src/main/AndroidManifest.xml", emulateSdk = 18)
public class FitnessListPresenterTest {

 private Calendar cal = Calendar.getInstance();

 @Mock
 private IFitnessListModel model;

 @Mock
 private IFitnessListView view;

 private IFitnessListPresenter presenter;

 @Before
 public void setup() {
  MockitoAnnotations.initMocks(this);

  final FitnessEntry entryMock = mock(FitnessEntry.class);

  presenter = new FitnessListPresenter(view, model);
  /*
   Define the desired behaviour.

   Queuing the action in "doAnswer" for "when" is executed.
   Clear and synchronous way of setting reactions for actions (stubbing).
   */
  doAnswer((new Answer<Object>() {
   @Override
   public Object answer(InvocationOnMock invocation) throws Throwable {
    ArrayList<FitnessEntry> items = new ArrayList<>();
    items.add(entryMock);

    ((IFitnessListPresenterCallback) presenter).onFetchAllSuccess(items);
    return null;
   }
  })).when(model).fetchAllItems((IFitnessListPresenterCallback) presenter);
 }

 /**
  Verify if model.fetchItems was called once.
  Verify if view.onFetchSuccess is called once with the specified list of type FitnessEntry

  The concrete implementation of ((IFitnessListPresenterCallback) presenter).onFetchAllSuccess(items); 
  calls the view.onFetchSuccess(...) method. This is why we verify that view.onFetchSuccess is called once.
 */
 @Test
 public void testFetchAll() {
  presenter.fetchAllItems(false);
  // verify can be called only on mock objects
  verify(model, times(1)).fetchAllItems((IFitnessListPresenterCallback) presenter);
  verify(view, times(1)).onFetchSuccess(new ArrayList<>(anyListOf(FitnessEntry.class)));
 }
}

Mocking the Global Networking Layer with MockWebServer

It is often convenient to be able to mock the global networking layer. MockWebServer allows us to queue responses for specific requests that we execute in our tests. This gives us the chance to simulate obscure responses that we expect from the server, but are not straightforward to reproduce. It allows us to ensure full coverage while writing little additional code.
MockWebServer?s code repository provides a neat example that you can refer to for a better understanding of this library.

Custom Test Doubles

You can write your own model or respoistory component and inject it into the test by providing a different module to the object graph using Dagger (http://square.github.io/dagger/). We have the option to check if the view state was updated properly based on the data supplied by the mocked model component:

/**
 Custom mock model class 
*/
public class FitnessListErrorTestModel extends FitnessListModel {

 // ...

 @Override
 public void fetchAllItems(IFitnessListPresenterCallback callback) {
  callback.onError();
 }

 @Override
 public void fetchItemsInRange(final IFitnessListPresenterCallback callback, DateFilter filter) {
  callback.onError();
 }

}

@RunWith(RobolectricTestRunner.class)
@Config(manifest = "app/src/main/AndroidManifest.xml", emulateSdk = 18)
public class FitnessListPresenterDaggerTest {

    private FitnessActivity activity;
    private FitnessListFragment fitnessListFragment;

    @Before
    public void setup() {
        /*
            setupActivity runs the Activity lifecycle methods on the specified class
        */
        activity = Robolectric.setupActivity(FitnessActivity.class);
        fitnessListFragment = activity.getFitnessListFragment();
        
        /*
            Create the objectGraph with the TestModule
        */
        ObjectGraph localGraph = ObjectGraph.create(TestModule.newInstance(fitnessListFragment));
        /*
            Injection
        */
        localGraph.inject(fitnessListFragment);
        localGraph.inject(fitnessListFragment.getPresenter());
    }

    @Test
    public void testInteractorError() {
        fitnessListFragment.getPresenter().fetchAllItems(false);

        /*
            suppose that our view shows a Toast message with the specified text below when an error is reported, so we check for it.
        */
        assertEquals(ShadowToast.getTextOfLatestToast(), "Something went wrong!");
    }

    @Module(
            injects = {
                    FitnessListFragment.class,
                    FitnessListPresenter.class
            }, overrides = true,
            library = true
    )
    static class TestModule {
        private IFitnessListView view;

        private TestModule(IFitnessListView view){
            this.view = view;
        }

        public static TestModule newInstance(IFitnessListView view){
            return new TestModule(view);
        }

        @Provides
        public IFitnessListInteractor provideFitnessListInteractor(){
            return new FitnessListErrorTestModel();
        }

        @Provides public IFitnessListPresenter provideFitnessPresenter(){
            return new FitnessListPresenter(view);
        }
    }

}

Running Tests

Android Studio

You can easily right click a test class, method, or whole test package and run the tests from the options dialog in the IDE.

Terminal

Running Android app tests from the terminal creates reports for the tested classes in the ?build? folder of the target module. Even more, if you plan to setup an automated build process, you will use the terminal approach. With Gradle, you can run all debug flavored tests by executing the following:

gradle testDebug

Accessing Source Set ?test? from Android Studio Version

Version 1.1 of Android Studio and the Android Gradle plugin brings support for unit testing your code. You can learn more by reading their excellent documentation on it. The feature is experimental, but also a great inclusion since you can now easily switch between your unit tests and instrumentation test source sets from the IDE. It behaves in the same way as if you would switch flavours in the IDE.

Easing the Process

Writing Android app tests may not be as fun as developing the original application. Hence, some tips on how to ease the process of writing tests and avoiding common problems while setting up the project will help a long way.

AssertJ Android

AssertJ Android, as you may have guessed from the name, is a set of helper functions that is built with Android in mind. It is an extension to the popular library AssertJ. Functionality provided by AssertJ Android ranges from simple assertions, such as ?assertThat(view).isGone()?, to things as complex as:

assertThat(layout).isVisible()
    .isVertical()
    .hasChildCount(4)
    .hasShowDividers(SHOW_DIVIDERS_MIDDLE)

With AssertJ Android and its extensibility, you are guaranteed a simple, good starting point for writing tests for Android applications.

Robolectric and Manifest Path

While using Robolectric, you may notice that you have to specify the manifest location, and that the SDK version is set to 18. You can do this by including a ?Config? annotation.

@Config(manifest = "app/src/main/AndroidManifest.xml", emulateSdk = 18)

Running tests that require Robolectric from the terminal can introduce new challenges. For example, you may see exceptions like ?Theme not set?. If the tests are executing properly from the IDE, but not from the terminal, you may be trying to run it from a path in the terminal where the specified manifest path can not be resolved. The hard-coded config value for the manifest path may not be pointing to the right location from the point of execution of the command. This can be solved through the use of custom runners:

public class RobolectricGradleTestRunner extends RobolectricTestRunner {
 public RobolectricGradleTestRunner(Class<?> testClass) throws InitializationError {
  super(testClass);
 }

 @Override
 protected AndroidManifest getAppManifest(Config config) {
  String appRoot = "../app/src/main/";
  String manifestPath = appRoot + "AndroidManifest.xml";
  String resDir = appRoot + "res";
  String assetsDir = appRoot + "assets";
  AndroidManifest manifest = createAppManifest(Fs.fileFromPath(manifestPath),
      Fs.fileFromPath(resDir),
      Fs.fileFromPath(assetsDir));
  return manifest;
 }
}

Gradle Configuration

You can use the following to configure Gradle for unit testing. You may need to modify the dependency names and versions required based on your project needs.

// Robolectric
testCompile 'junit:junit:4.12'
testCompile 'org.mockito:mockito-core:1.9.5'
testCompile 'com.squareup.dagger:dagger:1.2.2'
testProvided 'com.squareup.dagger:dagger-compiler:1.2.2'

testCompile 'com.android.support:support-v4:21.0.+'
testCompile 'com.android.support:appcompat-v7:21.0.3'

testCompile('org.robolectric:robolectric:2.4') {
 exclude module: 'classworlds'
 exclude module: 'commons-logging'
 exclude module: 'httpclient'
 exclude module: 'maven-artifact'
 exclude module: 'maven-artifact-manager'
 exclude module: 'maven-error-diagnostics'
 exclude module: 'maven-model'
 exclude module: 'maven-project'
 exclude module: 'maven-settings'
 exclude module: 'plexus-container-default'
 exclude module: 'plexus-interpolation'
 exclude module: 'plexus-utils'
 exclude module: 'wagon-file'
 exclude module: 'wagon-http-lightweight'
 exclude module: 'wagon-provider-api'
}

Robolectric and Play Services

If you are using Google Play Services, you will have to create your own integer constant for the Play Services version in order for Robolectric to work properly in this application configuration.

<meta-data
 android:name="com.google.android.gms.version"
 android:value="@integer/gms_version"
 tools:replace="android:value" />

Robolectric Dependencies to Support Libraries

Another interesting testing problem is that Robolectric isn?t able to reference support libraries properly. The solution is to add a ?project.properties? file to the module where the tests are. For example, for the Support-v4 and AppCompat libraries the file should contain:

android.library.reference.1=../../build/intermediates/exploded-aar/com.android.support/support-v4/21.0.3
android.library.reference.2=../../build/intermediates/exploded-aar/com.android.support/appcompat-v7/21.0.3

Acceptance/Regression Testing

Acceptance/Regression testing automates part of the final step of testing on a real, 100% percent Android environment. We do not use mocked Android OS classes at this level - the tests are run on real devices and emulators.

These circumstances make the process much more unstable due to the variety of physical devices, emulator configurations, device states, and feature sets of each device. Furthermore, it is highly dependent on the version of the operating system and the screen size of the phone to decide how the content will be displayed.
It is a bit complex to create the right test that passes on a wide range of devices, but as always you should dream big yet start small. Creation of tests with Robotium is an iterative process. With a few tricks, it can be simplified a lot.

Robotium

Robotium is an open source Android test automation framework that has been in existence since January 2010. It is worth mentioning that Robotium is a paid solution, but comes with a fair free trial.
To speed up the process of writing Robotium tests, we will move away from manual test writing to test recording. The trade-off is between quality of code and speed. If you are making heavy changes to your user interface, you will benefit a lot from the test recording approach and being able to record new tests quickly.
Testdroid Recorder is a free test recorder that creates Robotium tests as it records the clicks you perform on the user interface. Installing the tool is super easy, as described in their documentations accompanied with a step-by-step video.
Since Testdroid Recorder is an Eclipse plugin and we are referring to Android Studio throughout this article, it would ideally be a reason of concern. However, in this case it is not a problem, as you can use the plugin directly with an APK and record the tests against it.
Once you create the tests, you can copy and paste them in Android Studio, together with any dependency that Testdroid recorder requires, and you are set to go. The recorded test would look something like the class below:

public class LoginTest extends ActivityInstrumentationTestCase2<Activity> {

private static final String LAUNCHER_ACTIVITY_CLASSNAME = "com.toptal.fitnesstracker.view.activity.SplashActivity";
private static Class<?> launchActivityClass;
static {
try {
 launchActivityClass = Class.forName(LAUNCHER_ACTIVITY_CLASSNAME);
  } catch (ClassNotFoundException e) {
   throw new RuntimeException(e);
  }
 }
 private ExtSolo solo;

 @SuppressWarnings("unchecked")
 public LoginTest() {
  super((Class<Activity>) launchActivityClass);
 }

 // executed before every test method
 @Override
 public void setUp() throws Exception {
  super.setUp();
  solo = new ExtSolo(getInstrumentation(), getActivity(), this.getClass()
    .getCanonicalName(), getName());
 }
 
 // executed after every test method
 @Override
 public void tearDown() throws Exception {
  solo.finishOpenedActivities();
  solo.tearDown();
  super.tearDown();
 }

 public void testRecorded() throws Exception {
  try {
   assertTrue(
     "Wait for edit text (id: com.toptal.fitnesstracker.R.id.login_username_input) failed.",
     solo.waitForEditTextById(
       "com.toptal.fitnesstracker.R.id.login_username_input",
       20000));
   solo.enterText(
     (EditText) solo
       .findViewById("com.toptal.fitnesstracker.R.id.login_username_input"),
     "[email protected]");
   solo.sendKey(ExtSolo.ENTER);
   solo.sleep(500);
   assertTrue(
     "Wait for edit text (id: com.toptal.fitnesstracker.R.id.login_password_input) failed.",
     solo.waitForEditTextById(
       "com.toptal.fitnesstracker.R.id.login_password_input",
       20000));
   solo.enterText(
     (EditText) solo
       .findViewById("com.toptal.fitnesstracker.R.id.login_password_input"),
     "123456");
   solo.sendKey(ExtSolo.ENTER);
   solo.sleep(500);
   assertTrue(
     "Wait for button (id: com.toptal.fitnesstracker.R.id.parse_login_button) failed.",
     solo.waitForButtonById(
       "com.toptal.fitnesstracker.R.id.parse_login_button",
       20000));
   solo.clickOnButton((Button) solo
                    .findViewById("com.toptal.fitnesstracker.R.id.parse_login_button"));
            assertTrue("Wait for text fitness list activity.",
                    solo.waitForActivity(FitnessActivity.class));
   assertTrue("Wait for text KM.",
     solo.waitForText("KM", 20000));

   /*
    Custom class that enables proper clicking of ActionBar action items
   */
            TestUtils.customClickOnView(solo, R.id.action_logout);

            solo.waitForDialogToOpen();
            solo.waitForText("OK");
            solo.clickOnText("OK");

            assertTrue("waiting for ParseLoginActivity after logout", solo.waitForActivity(ParseLoginActivity.class));
            assertTrue(
                    "Wait for button (id: com.toptal.fitnesstracker.R.id.parse_login_button) failed.",
                    solo.waitForButtonById(
                            "com.toptal.fitnesstracker.R.id.parse_login_button",
                            20000));
  } catch (AssertionFailedError e) {
   solo.fail(
     "com.example.android.apis.test.Test.testRecorded_scr_fail",
     e);
   throw e;
  } catch (Exception e) {
   solo.fail(
     "com.example.android.apis.test.Test.testRecorded_scr_fail",
     e);
   throw e;
  }
 }
}

If you look closely, you will notice how much of the code is rather straight-forward.
When recording tests, don?t go scarce on ?wait? statements. Wait for dialogs to appear, activities to appear, texts to appear. This will guarantee that the activity and the view hierarchy are ready to be interacted with when you perform the action on the current screen. At the same time, take screenshots. Automated tests are usually unattended, and screenshots are one of the ways you get to see what actually happened during those tests.
Whether tests pass or fail, reports are your best friend. You can find them under the build directory ?module/build/outputs/reports?:

In theory, the QA team could record tests and optimize them. By putting effort in a standardized model for optimizing test cases, it could be done. When you normally record tests, you always have to tweak a couple of things to make it work flawlessly.
Finally, to run these tests from Android Studio, you can select them and run as you would run unit tests. From the terminal, it is a one-liner:

gradle connectedAndroidTest

Performance of Testing

Android unit testing with Robolectric is extremely fast, because it runs directly within the JVM on your machine. Compared to that, acceptance testing on emulators and physical devices is a lot slower. Depending on the size of flows that you are testing, it can take anywhere from a few seconds to a few minutes per test case. The acceptance test phase should be used as part of an automated build process on a continuous integration server.
Speed can be improved by parallelization on multiple devices. Check out this great tool from Jake Wharton and the guys at Square http://square.github.io/spoon/. It has some nice reporting, too.

The Takeaway

There are a variety of Android testing tools available, and as the ecosystem matures, the process of setting up a testable environment and writing tests will become easier. There are still more challenges to tackle, and with a wide community of developers working on daily problems, there is a lot of room for constructive discussions and fast feedback.
Use the approaches described in this Android testing tutorial to guide you in tackling the challenges ahead of you. If and when you run into problems, check back with this article or the references linked within for solutions to known problems.
In a future post, we will discuss parallelization, build automation, continuous integration, Github/BitBucket hooks, artifact versioning, and the best practices for managing massive mobile application projects in greater depth.