Automating the tests

Deqp test modules can be integrated to automated test systems in multiple ways. The best approach depends on the existing test infrastructure and target environment.

The primary output from a test run is always the test log file, i.e. the file with a .qpa postfix. Full test results can be parsed from the test log. Console output is debug information only and may not be available on all platforms.

Test binaries can be invoked directly from a test automation system. The test binary can be launched for a specific case, for a test set, or for all available tests. If a fatal error occurs during execution (such as certain API errors or a crash), the test execution will abort. For regression testing, the best approach is to invoke the test binaries for individual cases or small test sets separately, in order to have partial results available even in the event of hard failure.

The deqp comes with command line test execution tools that can be used in combination with the execution service to achieve a more robust integration. The executor detects test process termination and will resume test execution on the next available case. A single log file is produced from the full test session. This setup is ideal for lightweight test systems that don’t provide crash recovery facilities.

Command line test execution tools

The current command line tool set includes a remote test execution tool, a test log comparison generator for regression analysis, a test-log-to-CSV converter, a test-log-to-XML converter, and a testlog-to-JUnit converter.

The source code for these tools is in the executor directory, and the binaries are built into the <builddir>/executor directory.

Command line Test Executor

The command line Test Executor is a portable C++ tool for launching a test run on a device and collecting the resulting logs from it over TCP/IP. The Executor communicates with the execution service (execserver) on the target device. Together they provide functionality such as recovery from test process crashes. The following examples demonstrate how to use the command line Test Executor (use --help for more details):

Example 1: Run GLES2 functional tests on an Android device:

executor --connect=127.0.0.1 --port=50016 --binaryname=
com.drawelements.deqp/android.app.NativeActivity
--caselistdir=caselists
--testset=dEQP-GLES2.* --out=BatchResult.qpa
--cmdline="--deqp-crashhandler=enable --deqp-watchdog=enable
--deqp-gl-config-name=rgba8888d24s8"

Example 2: Continue a partial OpenGL ES 2 test run locally:

executor --start-server=execserver/execserver --port=50016
--binaryname=deqp-gles2 --workdir=modules/opengl
--caselistdir=caselists
--testset=dEQP-GLES2.* --exclude=dEQP-GLES2.performance.* --in=BatchResult.qpa
--out=BatchResult.qpa

Test log CSV export and compare

The deqp has a tool for converting test logs (.qpa files) into CSV files. The CSV output contains a list of test cases and their results. The tool can also compare two or more batch results and list only the test cases that have different status codes in the input batch results. The comparison will also print the number of matching cases.

The output in CSV format is very practical for further processing with standard command line utilities or with a spreadsheet editor. An additional, human-readable, plain-text format can be selected using the following command line argument: --format=text

Example 1: Export test log in CSV format

testlog-to-csv --value=code BatchResult.qpa > Result_statuscodes.csv
testlog-to-csv --value=details BatchResult.qpa > Result_statusdetails.csv

Example 2: List differences of test results between two test logs

testlog-to-csv --mode=diff --format=text Device_v1.qpa Device_v2.qpa

Note: The argument --value=code outputs the test result code, such as "Pass" or "Fail". The argument --value=details selects the further explanation of the result or numerical value produced by a performance, capability, or accuracy test.

Test log XML export

Test log files can be converted to valid XML documents using the testlog-to-xml utility. Two output modes are supported:

Separate documents mode, where each test case and the caselist.xml summary document are written to a destination directory
Single file mode, where all results in the .qpa file are written to single XML document.

Exported test log files can be viewed in a browser using an XML style sheet. Sample style sheet documents (testlog.xsl and testlog.css) are provided in the doc/testlog-stylesheet directory. To render the log files in a browser, copy the two style sheet files into the same directory where the exported XML documents are located.

If you are using Google Chrome, the files must be accessed over HTTP as Chrome limits local file access for security reasons. The standard Python installation includes a basic HTTP server that can be launched to serve the current directory with the python –m SimpleHTTPServer 8000 command. After launching the server, just point the Chrome browser to http://localhost:8000 to view the test log.

Conversion to a JUnit test log

Many test automation systems can generate test run result reports from JUnit output. The deqp test log files can be converted to the JUnit output format using the testlog-to-junit tool.

The tool currently supports translating the test case verdict only. As JUnit supports only "pass" and "fail" results, a passing result of the deqp is mapped to "JUnit pass" and other results are considered failures. The original deqp result code is available in the JUnit output. Other data, such as log messages and result images, are not preserved in the conversion.