This tutorial guides you through creating a "hello world" Trade Federation (TF) test configuration and gives you a hands-on introduction to the TF framework. Starting from a development environment, you will create a simple configuration and add features.
The tutorial presents the test development process as a set of exercises, each consisting of several steps, that demonstrate how to build and gradually refine your configuration. All sample code you need to complete the test configuration is provided, and the title of each exercise is annotated with a letter describing the roles involved in that step:
After completing the tutorial, you will have a functioning TF configuration and understand many important concepts in the TF framework.
For details on seting up the TF development environment, see Machine Setup. The rest of this tutorial assumes you have a shell open that has been initialized to the TF environment.
For simplicity, this tutorial illustrates adding a configuration and its classes to the TF framework core library. This can be extended to developing modules outside the source tree by compiling the tradefed JAR, then compiling your modules against that JAR.
Lets create a hello world test that just dumps a message to stdout. A tradefed test generally implements the IRemoteTest interface. Here's an implementation for the HelloWorldTest:
package com.android.tradefed.example;
import com.android.tradefed.device.DeviceNotAvailableException;
import com.android.tradefed.result.ITestInvocationListener;
import com.android.tradefed.testtype.IRemoteTest;
public class HelloWorldTest implements IRemoteTest {
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World!");
}
}
Save this sample code to
<tree>/tools/tradefederation/core/prod-tests/src/com/android/tradefed/example/HelloWorldTest.java
and rebuild tradefed from your shell:
m -jNNote that System.out in the example above may not actually
direct output to the console. While this is acceptable for this test example,
you should establish logging in Trade Federation as described in Logging (D, I, R).
If the build does not succeed, consult Machine Setup to ensure you didn't miss a step.
Trade Federation tests are made executable by creating a Configuration, an XML file that instructs tradefed on which test (or tests) to run, as well as which other modules to execute and in what order.
Lets create a new Configuration for our HelloWorldTest (note the full class name of the HelloWorldTest):
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
</configuration>Save this data to a helloworld.xml file anywhere on your local
filesystem (e.g. /tmp/helloworld.xml). TF will parse the
Configuration XML file (aka config), load the specified class using
reflection, instantiate it, cast it to a IRemoteTest, and call its
run method.
From your shell, launch the tradefed console:
$ tradefed.sh
Ensure a device is connected to the host machine and is visible to tradefed:
tf >list devices
Serial State Product Variant Build Battery
004ad9880810a548 Available mako mako JDQ39 100
Configurations can be executed using the run <config>
console command. Try:
tf> run /tmp/helloworld.xml
05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!
You should see "Hello, TF World!" output on the terminal.
For convenience of deployment, you can also bundle configs into the tradefed JARs themselves. Tradefed automatically recognizes all configurations placed in config folders on the classpath.
To illustrate, move the helloworld.xml file into the tradefed
core library
(<tree>/tools/tradefederation/core/prod-tests/res/config/example/helloworld.xml).
Rebuild tradefed, restart the tradefed console, then ask tradefed to display the
list of configurations from the classpath:
tf> list configs
[…]
example/helloworld: Runs the hello world test
You can now run the helloworld config using:
tf >run example/helloworld
05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!
So far, our HelloWorldTest isn't doing anything interesting. Tradefed's specialty is running tests using Android devices, so lets add an Android device to the test.
Tests can get a reference to an Android device by implementing the IDeviceTest interface. Here's a sample implementation of what this looks like:
public class HelloWorldTest implements IRemoteTest, IDeviceTest {
private ITestDevice mDevice;
@Override
public void setDevice(ITestDevice device) {
mDevice = device;
}
@Override
public ITestDevice getDevice() {
return mDevice;
}
…
}
The Trade Federation framework will inject the ITestDevice
reference into your test via the IDeviceTest#setDevice method,
before the IRemoteTest#run method is called.
Let's modify the HelloWorldTest print message to display the serial number of the device:
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber());
}
Now rebuild tradefed and check the list of devices:
$ tradefed.sh
tf >list devices
Serial State Product Variant Build Battery
004ad9880810a548 Available mako mako JDQ39 100
Take note of the serial number listed as Available; that is the device that should be allocated to HelloWorld:
tf >run example/helloworld
05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
You should see the new print message displaying the serial number of the device.
IRemoteTest reports results by calling methods on the
ITestInvocationListener
instance provided to the #run method. The TF framework itself is
responsible for reporting the start (via
ITestInvocationListener#invocationStarted)
and end (via
ITestInvocationListener#invocationEnded)
of each Invocation.
A test run is a logical collection of tests. To report test results,
IRemoteTest is responsible for reporting the start of a test run,
the start and end of each test, and the end of the test run.
Here's what the HelloWorldTest implementation might look like with a single failed test result.
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber());
TestIdentifier testId = new TestIdentifier("com.example.TestClassName", "sampleTest");
listener.testRunStarted("helloworldrun", 1);
listener.testStarted(testId);
listener.testFailed(testId, "oh noes, test failed");
listener.testEnded(testId, Collections.emptyMap());
listener.testRunEnded(0, Collections.emptyMap());
}TF includes several IRemoteTest implementations you can reuse
instead of writing your own from scratch. For example,
InstrumentationTest
can run an Android application's tests remotely on an Android device, parse the
results, and forward those results to the ITestInvocationListener).
For details, see
Test
Types.
The default test listener implementation for a TF config is TextResultReporter, which dumps the results of an invocation to stdout. To illustrate, run the HelloWorldTest config from the previous section:
$ ./tradefed.sh
tf >run example/helloworld
05-16 20:03:15 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
05-16 20:03:15 I/InvocationToJUnitResultForwarder: run helloworldrun started: 1 tests
Test FAILURE: com.example.TestClassName#sampleTest
stack: oh noes, test failed
05-16 20:03:15 I/InvocationToJUnitResultForwarder: run ended 0 ms
To store the results of an invocation elsewhere, such as in a file, specify a
custom ITestInvocationListener implementation using the
result_reporter tag in your configuration.
TF also includes the
XmlResultReporter
listener, which writes test results to an XML file in a format similar to that
used by the ant JUnit XML writer. To specify the result_reporter in the
configuration, edit the …/res/config/example/helloworld.xml
config:
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
<result_reporter class="com.android.tradefed.result.XmlResultReporter" />
</configuration>
Now rebuild tradefed and re-run the hello world sample:
tf >run example/helloworld
05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt
05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_2991649128735283633/host_log_6307746032218561704.txt
05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /tmp/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0
Notice the log message stating that an XML file has been generated; the generated file should look like this:
<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost">
<properties />
<testcase name="sampleTest" classname="com.example.TestClassName" time="0">
<failure>oh noes, test failed
</failure>
</testcase>
</testsuite>
You can also write your own custom invocation listeners—they simply need to implement the ITestInvocationListener interface.
Tradefed supports multiple invocation listeners, so you can send test results
to multiple independent destinations. To do this, just specify multiple
<result_reporter> tags in your config.
TF's logging facilities include the ability to:
The TF framework automatically captures the logcat from the allocated device
and sends it to the invocation listener for processing.
XmlResultReporter then saves the captured device logcat as a file.
TF host logs are reported using the
CLog wrapper
for the ddmlib Log class. Let's convert the
previous System.out.println call in HelloWorldTest to a
CLog call:
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
CLog.i("Hello, TF World! I have device %s", getDevice().getSerialNumber());
CLog handles string interpolation directly, similar to
String.format. When you rebuild and rerun TF, you should see the
log message on stdout:
tf> run example/helloworld
…
05-16 21:30:46 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
…
By default, tradefed
outputs host log
messages to stdout. TF also includes a log implementation that writes
messages to a file:
FileLogger.
To add file logging, add a logger tag to the config, specifying the
full class name of FileLogger:
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
<result_reporter class="com.android.tradefed.result.XmlResultReporter" />
<logger class="com.android.tradefed.log.FileLogger" />
</configuration>
Now, rebuild and run the helloworld example again:
tf >run example/helloworld
…
05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt
05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
…
The log message indicates the path of the host log, which, when viewed, should contain your HelloWorldTest log message:
$ more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
…
05-16 21:38:21 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
Objects loaded from a TF Configuration (aka Configuration objects)
can also receive data from command line arguments through the use of the
@Option annotation.
To participate, a Configuration object class applies the @Option
annotation to a member field and provides it a unique name. This enables that
member field value to be populated via a command line option (and also
automatically adds that option to the configuration help system).
Note: Not all field types are supported. For a description of supported types, see OptionSetter.
Let's add an @Option to HelloWorldTest:
@Option(name="my_option",
shortName='m',
description="this is the option's help text",
// always display this option in the default help text
importance=Importance.ALWAYS)
private String mMyOption = "thisisthedefault";
Next, let's add a log message to display the value of the option in HelloWorldTest so we can demonstrate it was received correctly:
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
…
CLog.logAndDisplay(LogLevel.INFO, "I received option '%s'", mMyOption);
Finally, rebuild TF and run helloworld; you should see a log message with the
my_option default value:
tf> run example/helloworld
…
05-24 18:30:05 I/HelloWorldTest: I received option 'thisisthedefault'
Pass in a value for my_option; you should see
my_option populated with that value:
tf> run example/helloworld --my_option foo
…
05-24 18:33:44 I/HelloWorldTest: I received option 'foo'
TF configurations also include a help system, which automatically displays
help text for @Option fields. Try it now, and you should see the
help text for my_option:
tf> run example/helloworld --help
Printing help for only the important options. To see help for all options, use the --help-all flag
cmd_options options:
--[no-]help display the help text for the most important/critical options. Default: false.
--[no-]help-all display the full help text for all options. Default: false.
--[no-]loop keep running continuously. Default: false.
test options:
-m, --my_option this is the option's help text Default: thisisthedefault.
'file' logger options:
--log-level-display the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error.
Note the message about "printing only the important options." To reduce
option help clutter, TF uses the Option#importance attribute to
determine whether to show a particular @Option field help text when
--help is specified. --help-all always shows help for
all @Option fields, regardless of importance. For details, see
Option.Importance.
You can also specify an Option value within the config by adding a
<option name="" value=""> element. Test it using
helloworld.xml:
<test class="com.android.tradefed.example.HelloWorldTest" >
<option name="my_option" value="fromxml" />
</test>
Re-building and running helloworld should now produce this output:
05-24 20:38:25 I/HelloWorldTest: I received option 'fromxml'
The configuration help should also update to indicate the default value of
my_option:
tf> run example/helloworld --help
test options:
-m, --my_option this is the option's help text Default: fromxml.
Other configuration objects included in the helloworld config, such as
FileLogger, also accept options. The option
--log-level-display is interesting because it filters the logs that
show up on stdout. Earlier in the tutorial, you may have noticed the "Hello, TF
World! I have device …' log message stopped being displayed on stdout after we
switched to using FileLogger. You can increase the verbosity of
logging to stdout by passing in the --log-level-display arg.
Try this now, and you should see the 'I have device' log message reappear on stdout, in addition to being logged to a file:
tf >run example/helloworld --log-level-display info
…
05-24 18:53:50 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
As a reminder, if you're stuck on something, the Trade Federation source code has a lot of useful information that isn't exposed in the documentation. If all else fails, try asking on the android-platform Google Group, with "Trade Federation" in the message subject.