Thursday, December 22, 2011

Android Continuous Integration Guide

Android Application Testing Guide features a whole chapter about Continuous Integration (Chapter 8), however some latest changes and additions to the tools available may require a more in-depth coverage of the subject.
Consequently, I'm preparing an Android Continuous Integration Guide to compile all the information laying around on the subject and to provide concise, working examples that you could use to base your own projects on.
In the creation of these examples using Jenkins and EMMA for code coverage one of the most annoying things is the R class affecting the results of the coverage report, as most probably you are not creating tests for such auto-generated class.

This screenshot show how the coverage report for classes reaches 100% once we filter out the R class.

The problem is that in current Android SDK Tools (Rev 16), there's no way to filter classes from EMMA coverage unless you modify the file /tools/ant/build.xml, changing the emma target to include the filters as showed in this code snippet:



            <!-- It only instruments class files, not any external libs -->
            <emma enabled="true">
               <instr verbosity="${verbosity}"
                               mode="overwrite"
                               instrpath="${out.absolute.dir}/classes"
                               outdir="${out.absolute.dir}/classes">
                    <!-- DTM: 2011-12-23: added filter for R -->
                     <filter excludes="*.R" />
                     <filter excludes="*.R$*" />
                </instr>
                <!-- TODO: exclusion filters on R*.class and allowing custom exclusion from
                             user defined file -->
            </emma>

I hope this help you getting started and stay tuned, Android Continuous Integration Guide is scheduled to be released by the end of January 2012.
Any comments, suggestions and requests are welcome and can be entered using the Google+ pages. 


Tuesday, November 15, 2011

Obtaining code coverage of a running Android application

How can we obtain the code coverage of a running application, not just its tests ?
I have been asked this question many times. Recently, Jonas posted a similar question as comment to Eclipse, Android and EMMA code coverage. So we will elaborate the solution to this problem.
But firstly, let's do a brief introduction of the concepts.

EMMA: a free Java code coverage tool
EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other tools by going after a unique feature combination: support for large-scale enterprise software development while keeping individual developer's work fast and iterative.

Android includes EMMA v2.0, build 5312, which includes some minor changes introduced by Android to adapt it for the platform specifics.

Android Instrumentation
The instrumentation framework is the foundation of the testing framework. Instrumentation controls the application under test and permits the injection of mock components required by the application to run.
Usually, an InstrumentationTestRunner, a special class the extends Instrumentation, is used to run various types of TestCases, against an android application.
Typically, this Instrumentation is declared in the test project's AndroidManifest.xml and then run from Eclipse or from the command line using am instrument.
Also, to generate EMMA code coverage -e coverage true option is added to the command line.
Basically, we have all the components but in different places because we want to obtain the code coverage from the running application not from its tests.

EmmaInstrumentation
The first thing we need to do is to create a new Instrumentation that starts the Activity Under Test using EMMA instrumentation and when this Activity is finished the coverage data is saved to a file.
To be notified of this Activity finish we need a listener that we can set extending the AUT because one of our objectives is to keep it unchanged.

To illustrate this technique we will be using the Temperature Converter application that we have used many times in other posts. The source code is as usual available through github.


package com.example.instrumentation;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

import com.example.i2at.tc.TemperatureConverterActivity;
//import com.vladium.emma.rt.RT;

import android.app.Activity;
import android.app.Instrumentation;
import android.content.Intent;
import android.os.Bundle;
import android.os.Looper;
import android.util.Log;

public class EmmaInstrumentation extends Instrumentation implements FinishListener {

    private static final String TAG = "EmmaInstrumentation";

    private static final boolean LOGD = true;

    private static final String DEFAULT_COVERAGE_FILE_PATH = "/mnt/sdcard/coverage.ec";

    private final Bundle mResults = new Bundle();

    private Intent mIntent;

    private boolean mCoverage = true;

    private String mCoverageFilePath;

    /**
     * Extends the AUT to provide the necessary behavior to invoke the
     * {@link FinishListener} that may have been provided using
     * {@link #setFinishListener(FinishListener)}.
     * 
     * It's important to note that the original Activity has not been modified.
     * Also, the Activity must be declared in the
     * <code>AndroidManifest.xml</code> because it is started by an intent in
     * {@link EmmaInstrumentation#onStart()}. This turns more difficult to use
     * other methods like using template classes. This latter method could be
     * viable, but all Activity methods should be re-written to invoke the
     * template parameter class corresponding methods.
     * 
     * @author diego
     * 
     */
    public static class InstrumentedActivity extends
    TemperatureConverterActivity {
        private FinishListener mListener;

        public void setFinishListener(FinishListener listener) {
            mListener = listener;
        }

        @Override
        public void finish() {
            if (LOGD)
                Log.d(TAG + ".InstrumentedActivity", "finish()");
            super.finish();
            if (mListener != null) {
                mListener.onActivityFinished();
            }
        }

    }

    /**
     * Constructor
     */
    public EmmaInstrumentation() {

    }

    @Override
    public void onCreate(Bundle arguments) {
        if (LOGD)
            Log.d(TAG, "onCreate(" + arguments + ")");
        super.onCreate(arguments);

        if (arguments != null) {
            mCoverage = getBooleanArgument(arguments, "coverage");
            mCoverageFilePath = arguments.getString("coverageFile");
        }

        mIntent = new Intent(getTargetContext(), InstrumentedActivity.class);
        mIntent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
        start();
    }

    @Override
    public void onStart() {
        if (LOGD)
            Log.d(TAG, "onStart()");
        super.onStart();

        Looper.prepare();
        InstrumentedActivity activity = (InstrumentedActivity) startActivitySync(mIntent);
        activity.setFinishListener(this);
    }

    private boolean getBooleanArgument(Bundle arguments, String tag) {
        String tagString = arguments.getString(tag);
        return tagString != null && Boolean.parseBoolean(tagString);
    }

    private void generateCoverageReport() {
        if (LOGD)
            Log.d(TAG, "generateCoverageReport()");

        java.io.File coverageFile = new java.io.File(getCoverageFilePath());

        // We may use this if we want to avoid refecltion and we include
        // emma.jar
        // RT.dumpCoverageData(coverageFile, false, false);

        // Use reflection to call emma dump coverage method, to avoid
        // always statically compiling against emma jar
        try {
            Class<?> emmaRTClass = Class.forName("com.vladium.emma.rt.RT");
            Method dumpCoverageMethod = emmaRTClass.getMethod(
                    "dumpCoverageData", coverageFile.getClass(), boolean.class,
                    boolean.class);
            dumpCoverageMethod.invoke(null, coverageFile, false, false);
        } catch (ClassNotFoundException e) {
            reportEmmaError("Is emma jar on classpath?", e);
        } catch (SecurityException e) {
            reportEmmaError(e);
        } catch (NoSuchMethodException e) {
            reportEmmaError(e);
        } catch (IllegalArgumentException e) {
            reportEmmaError(e);
        } catch (IllegalAccessException e) {
            reportEmmaError(e);
        } catch (InvocationTargetException e) {
            reportEmmaError(e);
        }
    }

    private String getCoverageFilePath() {
        if (mCoverageFilePath == null) {
            return DEFAULT_COVERAGE_FILE_PATH;
        } else {
            return mCoverageFilePath;
        }
    }

    private void reportEmmaError(Exception e) {
        reportEmmaError("", e);
    }

    private void reportEmmaError(String hint, Exception e) {
        String msg = "Failed to generate emma coverage. " + hint;
        Log.e(TAG, msg, e);
        mResults.putString(Instrumentation.REPORT_KEY_STREAMRESULT, "\nError: "
                + msg);
    }

    /* (non-Javadoc)
     * @see com.example.instrumentation.FinishListener#onActivityFinished()
     */
    @Override
    public void onActivityFinished() {
        if (LOGD)
            Log.d(TAG, "onActivityFinished()");
        if (mCoverage) {
            generateCoverageReport();
        }
        finish(Activity.RESULT_OK, mResults);
    }

}

We are also implementing the FinishListener interface, which is defined as


package com.example.instrumentation;

/**
 * Listen for an Activity to finish and invokes {@link #onActivityFinished()} when this happens.
 * 
 * @author diego
 *
 */
public interface FinishListener {

        /**
         * Invoked when the Activity finishes.
         */
        void onActivityFinished();

}

Running the instrumented application
Once we have the EmmaInstrumentation class in place we need a few more adjustments to be able to get the coverage report of the running application.
Firstly, we need to add the new Activity to the manifest. Secondly, we should allow our application to write to the sdcard if this is where we decided to generate the coverage report. To do it you should grant the android.permission.WRITE_EXTERNAL_STORAGE permission.
Then, it's time to build and install the instrumented apk:

$ ant clean
$ ant instrument
$ ant installi

Everything is ready to start the instrumented application

$ adb shell am instrument -e coverage true \
     -w com.example.i2at.tc/\
        com.example.instrumentation.EmmaInstrumentation

If everything went well, the Temperature Converter application will be running and we can use it for a while


when we exit by pressing the BACK button we can see that the coverage data was written to the file and reflected in the logcat

I/System.out(2453): EMMA: runtime coverage data written to [/mnt/sdcard/coverage.ec] {in 975 ms}

this file can then be moved to the host computer using adb pull.

Hope this helps you obtaining the code coverage for your application to help you understand its usage patterns. As always, comments and questions are always welcome.

Thursday, November 10, 2011

Android: Using monkey from Java


The latest version of the Android SDK and tools include chimpchat, a library that facilitates the use of monkey from Java. This is equivalent to monkeyrunner, which is the bridge between monkey and the Python scripting language.
While Python is an incredibly powerful and expressive scripting language and will permit you creating tests with just a few statements, there are some occasions when you don't want to introduce a new language to the project leaving your Java confort zone or you prefer to leverage the use of previously created libraries instead of writing new ones.
In such cases, you can now have the same access to monkey running on the device with the help of chimpchat, as we are going to demonstrate.


Creating a Java project
Our first step will be to create a new Java project and we will add the required libraries to the Java Build Path as External Jars.
We are naming the project JavaMonkey, for obvious reasons.




We are adding these libraries from Android SDK, which are used directly or indirectly by our project, to the Java Build Path:

  • chimpchat.jar
  • ddmlib.jar
  • guavalib.jar
  • sdklib.jar


JavaMonkey.java
Our intention is to create a simple class, serving the purpose of a simple example to get as started. We will be simply:

  1. Creating a JavaMonkey object
  2. initializing it, this implies creating the connection with any emulator or device found or throwing an exception is not connection was made before the timeout expires
  3. listing all the properties in the device or emulator
  4. shutting down the connection

Following, is the JavaMonkey class: 



/**
 * Copyright (C) 2011  Diego Torres Milano
 */
package com.example.javamonkey;

import java.util.TreeMap;

import com.android.chimpchat.ChimpChat;
import com.android.chimpchat.core.IChimpDevice;

/**
 * @author diego
 *
 */
public class JavaMonkey {

        private static final String ADB = "/Users/diego/opt/android-sdk/platform-tools/adb";
        private static final long TIMEOUT = 5000;
        private ChimpChat mChimpchat;
        private IChimpDevice mDevice;

        /**
         * Constructor
         */
        public JavaMonkey() {
                super();
        TreeMap<String, String> options = new TreeMap<String, String>();
        options.put("backend", "adb");
        options.put("adbLocation", ADB);
        mChimpchat = ChimpChat.getInstance(options);
        }

        /**
         * Initializes the JavaMonkey.
         */
        private void init() {
                mDevice = mChimpchat.waitForConnection(TIMEOUT, ".*");
                if ( mDevice == null ) {
                        throw new RuntimeException("Couldn't connect.");
                }
                mDevice.wake();
        }

        /**
         * List all properties.
         */
        private void listProperties() {
                if ( mDevice == null ) {
                        throw new IllegalStateException("init() must be called first.");
                }
                for (String prop: mDevice.getPropertyList()) {
                        System.out.println(prop + ": " + mDevice.getProperty(prop));
                }
        }

        /**
         * Terminates this JavaMonkey.
         */
        private void shutdown() {
                mChimpchat.shutdown();
                mDevice = null;
        }

        /**
         * @param args
         */
        public static void main(String[] args) {
                final JavaMonkey javaMonkey = new JavaMonkey();
                javaMonkey.init();
                javaMonkey.listProperties();
                javaMonkey.shutdown();
        }

}


Configuration
One of the important things you have to adapt to your environment is the location of the adb command. Otherwise if you don't set it you will receive:

E/adb: Failed to get the adb version: Cannot run program "adb": error=2, No such file or directory


Hope this helps you get started with chimpchat. As always, comments and questions are always welcome.

Thursday, November 03, 2011

Android Application Testing Guide: Q&A

Q:  Hi Diego, I wanted to ask that can i write a monkey runner script which controls a web based apk ?
Eg; I install youtube.apk which is nothing but a browser with hardcoded youtube url.
Now my monkeyrunner script shall install this apk and then pass events such as a search string etc on this web based application.
All this i want to do and control externally through the monkey runner script. Is this possible? If yes, then could you please guide me by some pseudo code? 


Comment on Using Android monkeyrunner from Eclipse


Posted by latha



A:  This is an interesting question and a good monkeyrunner example, so here we go. monkeyrunner has the ability of installing APKs after obtaining the connection with the device. Then we start Youtube main activity, sleep for a bit to let things settle down.
Once we have the activity running is time to start our search. To do it, we touch the Search icon, enter the desired search string, 'android' in this particular case and the we touch the Search button again to actually start the action.
Following, is the script that translates our plan to monkeyrunner:


#! /usr/bin/env monkeyrunner

import sys
import os
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice

YOUTUBE = 'com.google.android.youtube-2.1.6.apk'
prog = os.path.basename(sys.argv[0])

def usage():
        print >>sys.stderr, "usage: %s" % prog
        sys.exit(1)

def main():
        if len(sys.argv) != 1:
                usage()

        print "waiting for connection..."
        device = MonkeyRunner.waitForConnection()

        print "installing youtube"
        device.installPackage(YOUTUBE)

        device.startActivity(component="com.google.android.youtube/.HomeActivity")
        MonkeyRunner.sleep(3)
        # search
        device.touch(450, 80, MonkeyDevice.DOWN_AND_UP)
        MonkeyRunner.sleep(5)
        device.type('android')
        # done
        device.touch(450, 740, MonkeyDevice.DOWN_AND_UP)



if __name__ == '__main__':
    main()


This script covers the case described in the question but it could be easily adapted for other cases and application.

I hope this is the answer you were looking for.

Saturday, October 01, 2011

Android Application Testing Guide: Q&A

Q: i am working on android on automation i have setup all the environment required for it like(eclipse,jdk, Android SDk) and i am having monkeyrunner installed.

Now i want to automate the process to do this task--The monkeyrunner API can apply one or more test
suites across multiple devices or emulators. You can physically attach all the
devices or start up all the emulators (or both) at once, connect to each one in
turn programmatically, and then run one or more tests

i thought of doing this through monkeyrunner extending plugins concept, this concept doesn't help me becoz communicaton between two or more plugins creating a new problem. so ,please help me how to get my task done through other approach



Posted by vijju



A: Probably the best solution to your problem is based on the divide and conquer philosophy. Create your test scripts in a way that they receive a serial number argument to determine which device to connect to. The following is a very simple example of this concept. We will call it getprop.mr


#! /usr/bin/env monkeyrunner

import sys, os
from com.android.monkeyrunner import MonkeyRunner

prog = os.path.basename(sys.argv[0])

def usage():
        print >>sys.stderr, "usage: %s serial-no" % prog
        sys.exit(1)

def main():
        if len(sys.argv) != 2:
                usage()

        serialno = sys.argv[1]
        print "waiting for connection to %s..." % serialno
        device = MonkeyRunner.waitForConnection(30, serialno)

        s = ""
        for p in ['build.manufacturer', 'build.device', 'build.model']:
                s += " " + device.getProperty(p)
        print s

if __name__ == '__main__':
    main()

This script expects the first argument to be the serial number of the emulator or device where the specific tests will be run. In this simple case we are just obtaining some properties to identify it.


Once we have our test script we need a driver to run it in every device we specify. In this example we will be using a python script but a bash script would be good enough.


#! /usr/bin/env python

import os

devices = [ 'XXX00000001', 'emulator-5554' ]
cmd = 'getprop.mr'

for d in devices:
        os.system(cmd + " " + d)

This script will run the test, getprop.mr in this case, for each of the devices which serial numbers are in the list.
I hope this is the answer you were looking for.

Thursday, September 08, 2011

Android Testing: Running tests from code


I was answering some questions at StackOverflow today and one caught my attention. It was asking, well, not directly but I understood this was the intention, how to run the tests not from another computer using adb or Eclipse but from an android application itself. Then, here we are presenting a solution to run instrumentation from code.



private void runTests() {
   final String packageName = getPackageName();
   final List<InstrumentationInfo> list = 
         getPackageManager().queryInstrumentation(packageName, 0);
   if ( list.isEmpty() ) {
      Toast.makeText(this, "Cannot find instrumentation for " + packageName,
         Toast.LENGTH_SHORT).show();
      return;
   }
   final InstrumentationInfo instrumentationInfo = list.get(0);
   final ComponentName componentName = 
         new ComponentName(instrumentationInfo.packageName,
         instrumentationInfo.name);
   if ( !startInstrumentation(componentName, null, null) ) {
      Toast.makeText(this, "Cannot run instrumentation for " + packageName,
         Toast.LENGTH_SHORT).show();
   }
}


You need a valid context to call startInstrumentation() so this is probably added to an Activity.

I can imagine only a valid use case for this, which is running the tests when you don't have the device connected to a computer.
Do you have another use case ?
Speak out.

Hope this helps.

Friday, August 26, 2011

Android Application Testing Guide: Q&A


Q:I followed your example. I set up the test project in a similar way.
But try to write a TestCase for the utility class from the original
project.
Eclipse says "Class under test does not exist in the current project."

It's kind of reasonable to me, since that class is indeed in another
project.
I only have experienced using JUnit to test normal Java Project where
the test directory is inside the project.

Also, I checked out the two example projects and found there is a
build.properties and build.xml. Is that the reason that you can import
the original class:
"import com.example.i2at.tc.TemperatureConverter;"?

Thanks

Best wishes,
Ryan
Posted by Ryan Huang.

A:If you have imported both projects (main & test) into Eclipse you should have no problems because the versions available at github have the required properties set.
However, if for some reason they were not set properly, this is what you should verify in your test project's Java Build Path -> Libraries




The other files you mentioned are used when you build with ant.

Thursday, August 18, 2011

LinuxCon 2011 North America: Introduction to Android Testing

It was terrific having such a great audience showing a big interest in the topic of my tutorial, asking questions and starting discussions that made the presentation more enjoyable. I really want to thank you all.

For those who missed it, here are the slides that should be available at the Linux Foundation web site any time soon too.

Friday, August 05, 2011

Android Application Testing Guide: Q&A


Q:Diego, if we were to test something that is asynchronous, like in my case I'm wanting to test if a webview loads an URL how would I go about waiting for the webpage to finish loading in my webview ?
Posted by Pedro Veloso.

A:Actually this is a very interesting question as there are many ways and you should be cautious about WebView semantics and how some of the WebViewClient methods are called.

For example onPageFinished may be invoked wether the page was successfully loaded or there was an error. So, you may need a different approach if your intention is to test if an url was successfully loaded.

In this code snippet I'm using a MockWebViewClient to detect error conditions and simply waiting some time for the page to load. We could also iterate over a period of time checking if the value has changed instead of just waiting but we are keeping this as simple as possible. This is also assuming you have an Activity holding the WebView and it has the required getters.


/**
 * 
 */
package com.example.aatg.webview.test;

import android.test.ActivityInstrumentationTestCase2;
import android.test.suitebuilder.annotation.Suppress;
import android.webkit.WebView;
import android.webkit.WebViewClient;

import com.example.aatg.webview.AndroidHelloWebViewActivity;

/**
 * @author diego
 *
 */
public class AndroidHelloWebViewActivityTests extends
        ActivityInstrumentationTestCase2<AndroidHelloWebViewActivity> {

    private static final String VALID_URL = "http://developer.android.com";
    private static final String INVALID_URL = "http://invalid.url.doesnotexist987.com";

    private static final long TIMEOUT = 5000;

    private AndroidHelloWebViewActivity mActivity;
    private WebView mWebView;
    private MockWebViewClient mMockWebViewClient;

    /**
     * @param name
     */
    public AndroidHelloWebViewActivityTests() {
        super(AndroidHelloWebViewActivity.class);
    }

    /* (non-Javadoc)
     * @see android.test.ActivityInstrumentationTestCase2#setUp()
     */
    protected void setUp() throws Exception {
        super.setUp();
        mActivity = getActivity();
        mWebView = mActivity.getWebView();
        mMockWebViewClient = new MockWebViewClient();
        mWebView.setWebViewClient(mMockWebViewClient);
    }

    /* (non-Javadoc)
     * @see android.test.ActivityInstrumentationTestCase2#tearDown()
     */
    protected void tearDown() throws Exception {
        super.tearDown();
    }

    public final void testLoadValidUrl() {
        assertLoadUrl(VALID_URL);
        assertFalse(mMockWebViewClient.mError);
    }

    public final void testLoadInvalidUrl() {
        assertLoadUrl(INVALID_URL);
        assertTrue(mMockWebViewClient.mError);
    }

    private void assertLoadUrl(String url) {
        mWebView.loadUrl(url);
        sleep();
        assertTrue(!(mWebView.getProgress() < 100));
    }

    private void sleep() {
        try {
            Thread.sleep(TIMEOUT);
        } catch (InterruptedException e) {
            fail("Unexpected timeout");
        }
    }

    private class MockWebViewClient extends WebViewClient {
        boolean mError;

        @Override
        public void onReceivedError(WebView view, int errorCode,
                String description, String failingUrl) {
            mError = true;
        }
    }
}


Hope this helps.

Tuesday, August 02, 2011

Android Application Testing Guide: Q&A

Lately I've been receiving some comments or questions about some of the book's subjects in my email. To help the community the best I think that is preferable that you share your questions here, as comments to this post if you cannot find a post dealing with the same topic. If possible I will answer also here for the benefit of all.

Friday, July 08, 2011

Eclipse, Android and EMMA code coverage

On of the examples I'm including in the tutorial I will be presenting at LinuxCon North America 2011 (link to presentation) is a step by step use of EMMA code coverage from Eclipse to help you navigate through the Android application source code while applying Test Driven Development techniques.


This screenshot, taken from the tutorial examples, shows:

  1. The code coverage results directly summarized and highlighted in the Activity source code inside Eclipse editor
  2. The coverage view lists coverage summaries for the Android project, allowing drill-down to method level
If you are seriously developing Android applications you shouldn't miss this tutorial.
Hope to see you there.

Friday, June 24, 2011

Using Android monkeyrunner to automate test steps

It is not unlikely that in writing your tests you discover that some simple steps are required by several tests.

 monkeyrunner is a great tool in this respect because it offers a tremendously powerful and complete language like python.
You can write your tests using python but also you can build libraries containing primitives belonging to the test domain.
This example shows how you can lock and unlock the device screen. Usually this methods should be in a different class or module but for the sake of simplicity we are including lockDevice() and unlockDevice() here in the main script.


#! /usr/bin/env monkeyrunner
'''
Created on Jun 22, 2011

@author: diego
'''

from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice


def lockDevice(device):
    return device.press("POWER", MonkeyDevice.DOWN_AND_UP)


def unlockDevice(device):
    device.wake()
    device.drag((130, 620), (350, 620), 1.0, 120)

def main():
    device = MonkeyRunner.waitForConnection()
    if device:
        lockDevice(device)
        MonkeyRunner.sleep(5.0)
        unlockDevice(device)

if __name__ == '__main__':
    main()

Now, a demonstration of this script running on a Nexus One, locking, waking up, and unlocking the screen.



Hope this helps you start creating you monkeyrunner libraries.
Comments are welcome.

Sunday, June 19, 2011

Save the date: Android Testing at LinuxCon North America 2011

Mark your calendars. On Wednesday August 17th, 2011, 10:30 (schedule) I will be presenting the tutorial Introduction to Android Testing. It will give you an overview of current methodologies and tools available on Android. This tutorial will also introduce Test Driven Development, Behaviour Driven Development and Continuous Integration, techniques that every serious development project should at least consider.


More information at LinuxCon North America 2011.
Hope to see you there.

Friday, June 03, 2011

Android Application Testing Guide

The wait is almost over and after a year of hard work the book is finished and is expected to be published by PACKT this month (June 2011).

You can Pre-order now !

Approach
Adroid Application Testing Guide is a highly detailed book which gives step-by-step examples for a great variety of real-world cases, providing professional guidelines and recommendations that will be extremely valuable for optimizing your development time and resources. In the chapters you will find an introduction to specific testing techniques, and tools for specific situations.

Overview of Android Application Testing Guide
  • The first and only book that focuses on testing Android applications
  • Step-by-step approach clearly explaining the most efficient testing methodologies
  • Real world examples with practical test cases that you can reuse
  • eBook available as PDF and ePub downloads and also on PacktLib
More detailed information can be obtained from its web page at PACKT.

    Saturday, April 23, 2011

    Running instrumentation from monkeyrunner

    Sometimes you may want to run the instrumentation for a package from monkeyrunner, and the following script does just this. It has the advantage of using the package manager to find the correct instrumentation for the specified package simplifying the parameters you must provide.

    #! /usr/bin/env monkeyrunner
    
    import sys
    import os
    from com.android.monkeyrunner import MonkeyRunner
    
    PLI = 'pm list instrumentation'
    prog = os.path.basename(sys.argv[0])
    
    def usage():
       print >>sys.stderr, \
           "usage: %s target-package-name" % prog
       sys.exit(1)
    
    def main():
       if len(sys.argv) != 2:
          usage()
    
       pkg = sys.argv[1]
    
       print "waiting for connection..."
       device = MonkeyRunner.waitForConnection()
    
       print "running istrumentation for %s" % pkg
       for (i, t) in map(lambda l: l.split(), device.shell(PLI).splitlines()):
          if t == '(target=%s)' % pkg:
             print device.instrument(i.split(':')[1], { 'wait':True })['stream']
             return
       
       print >>sys.stderr, "ERROR: instrumentation for %s not found" % pkg
    
    
    if __name__ == '__main__':
        main()

    You can invoke this script as in the following command line:

       $ instrumentation.mr com.example.package

    compare this against the command line you may need to invoke the same instrumentation using plain am instrument.
    Once you invoke the script with the correct package name, the instrumentation for that package is run and the results are presented, like in this example:


    waiting for connection...
    running istrumentation for com.example.package


    Test results for InstrumentationTestRunner=..................................
    Time: 37.844


    OK (34 tests)

    Sunday, April 10, 2011

    monkeyrunner: visual image comparison

    We have discussed taking screenshots with monkeyrunner in previous posts, but now we are taking a step further by comparing the screenshots obtained during the test run against some reference image.

    Firstly, we need these reference images that you can obtain by running specific cases of the tests specially designed to do it. It's not recommended to use DDMS to take the reference image as in some cases the image format and compression may slightly differ from the ones taken by monkeyrunner leading to false positive identification of the dissimilarity.

    As a fairly simple example we will be creating a test to verify the correct drag and drop to a new position of a screen widget  , in this case the Home screen tips. We will be sending touch events to move it from its original position in row 2:


    to row 4:


    This second image, showing the Home screen tips widget in row 4 will be our reference image or the final state we want to verify in our test.
    There are some items we can anticipate will be different, like the time in the status bar, the battery level,  the connectivity, etc. Thus why we expect some degree of flexibility in our test not to fail under these circumstances.

    We also use compare from the great ImageMagick package that you should have installed for this script to work.
    In Debian/Ubuntu and derivatives
       $ sudo apt-get install imagemagick

    Having this in mind, our monkeyrunner test would look something like this.

    #! /usr/bin/env monkeyrunner import sys import subprocess from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice, MonkeyImage TIMEOUT = 30 SERIALNO = 'emulator-5554' REF = 'reference.png' SCR = 'screenshot.png' CMP = 'comparison.png' ACCEPTANCE = 0.9 device = None def testDropWidgetIntoFourthRow():     reference = MonkeyImage.loadFromFile(REF)     print "moving widget..."     device.drag((230, 300), (230, 600), 2.0, 125)     MonkeyRunner.sleep(3)     screenshot = device.takeSnapshot()     screenshot.writeToFile(SCR)     if not screenshot.sameAs(reference, ACCEPTANCE):        print "comparison failed, getting visual comparison..."        subprocess.call(["/usr/bin/compare", REF, SCR, CMP]) def main():     global device     print "waiting for connection..."     device = MonkeyRunner.waitForConnection(TIMEOUT, SERIALNO)     if device:        testDropWidgetIntoFourthRow() if __name__ == '__main__':      main()

    We are using MonkeyImage.loadFromFile() to load the reference image from a file.

    This method was not available in monkeyrunner so I implemented it myself and I decided to do it in MonkeyImage, but  it is now included in latest verions of monkeyrunner but as MonkeyRunner.loadImageFromFile().

    In case you don't have it in your version you can build it from source (being sure that this patch is included).
    monkeyrunner is a SDK component and thus is included in this project that can be downloaded from android SDK source.
    Anyway, if you can't do it you may just avoid this step and replace the condition in the if by True.


    A brief explanation of the script is:

    1. in the main method we obtain the connection with the device using the serial number specified in SERIALNO. A more sophisticated script should use parameters.
    2. if the connection was successful we run the test
    3. in the test method we load the reference image from the specified file
    4. we send a simulated drag event to move the widget from the second row to the fourth row, using screen coordinates
    5. sleep for a while
    6. get the screenshot
    7. if the images are not the same, considering 90% acceptance value, we use the visual comparison to get a clue of the difference
    If you run the script you can verify that the screenshot is taken and the test succeed because they are the same. They only have minor differences in the time and battery level but they lie above the acceptance value.
    Now, if we run the test again but this time using (230, 400) instead of (230, 600) in the drag command the widget will be dropped in the third line instead of the fourth and we expect this difference be detected as our reference image for the final state contains the widget in the fourth row, and thus precisely what happens.
    This is the comparison image highlighting in red the differences:


    Check that as we mentioned before the battery level and time in the status bar are also detected as differences but they are usually under the threshold.

    This is not a real repeatable test as we are not leaving the system in the same state that was found but the idea is to present the subject in the simplest possible way to avoid deviating from the main goal which is demonstrating how you can add visual comparison to your tests using monkeyrunner and ImageMagick.
    I'm sure this will give you lots of ideas to implement you own tests.
    As always, comments are gladly welcome.

    Saturday, March 12, 2011

    Using Android monkeyrunner from Eclipse

    Amazingly, this post is at the top of the stats meaning that still a lot of people are still using monkeyrunner and haven't discovered yet AndroidViewClient/culebra. Come on! It's time to evolve.



    You may want to edit scripts and run them from Eclipse.
    To be able to do this you must first install PyDev from Eclipse Marketplace.

    Latest versions of PyDev don't work with monkeyrunner because it is not detected as a valid interpreter. I'm sure this will be finxed in the future. In the meantime you should stick to PyDev 1.6.5.


    Then you have to define a new python interpreter


    but before you can do it you need to do some changes to the Android installation. We need to replace the monkeyrunner interpreter because Eclipse invokes it using the -u command line option,  unbuffer stdin, stdout and stderr, which is not supported by monkeyrunner. To solve it we should rename the original interpreter to monkeyrunner-original and then use this script as a replacement for  monkeyrunner.

    # /bin/bash
    if [ "$1" = '-u' ]
    then
     shift
    fi
    
    exec /opt/android-sdk-linux_86/tools/monkeyrunner-original "$@"
    
    if you are using Microsoft Windows, you should use something like this (script contributed by tagmaster)

    REM
    @echo off
    if("%1")==("-u") shift
    "C:\Program Files\Android\android-sdk\tools"/monkeyrunner-original %1 %2 %3 %4 %5 %6 %7 %8 
    Once the new interpreter is defined and assigned to a PyDev project you will be able to edit and run monkeyrunner scripts from Eclipse



    Don't forget to set the previously defined interpreter to the project properties.




    Update: July 2011
    This patch was added to monkeyrunner to ignore the -u option sent by PyDev:
    http://android.git.kernel.org/?p=platform/sdk.git;a=commitdiff;h=f07d8c2026606633f1de7e1ab9d985c74ad8b340
    However, the usage string does not include the -u option yet. BTW, credit would be nice.
    So, the wrapper is not needed any more.

    Update: September 2011
    monkeyrunner included in Android SDK Tools revision 12 supports -u option, however as it was mentioned before it is not described in the help text.
    Come on, we are only one step behind...

    Update: March 2012
    A working Eclipse configuration that works with monkeyrunner has been detailed in a new post: Eclipse: working monkeyrunner configuration