Description

The robottestingframework-testrunner is a command line tool to load and run multiple tests which has been developed as Robot Testing Framework plug-ins. The test case plug-ins are built as dynamic loadable libraries and depending on the target platform can be found as *.so, *.dll or *.dylib or using Lua, Python, Ruby and other scripting languages (e.g., .lua, .py, .rb). The robottestingframework-testrunner can be used to run a single test case, multiple test cases from a given path, a Test suite or multiple test suites from a given path.

Running a single test case

To run a single test case plug-in, it is enough to give the name of the corresponding plug-ing file to the robottestingframework-testrunner using –test option. For example on a Linux machine:

 $ robottestingframework-testrunner --verbose --test libmytest.so

The robottestingframework-testrunner follows the platform policy to look for the plug-in file. On a Linux machine, the default folders for the shared libraries and the paths indicated in the LD_LIBRARY_PATH are searched for the corresponding plug-in. Alternatively, the complete path to the plug-in file can be given to load the corresponding plug-in:

 $ robottestingframework-testrunner --verbose --test ~/my-plugins/libmytest.so

In the similar way you can run a test case which has been developed using Lua, Python, Ruby or other languages:

 $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.lua
 $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.py
 $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.rb

The --verbose enables the verbose mode which allows that the result of the tests to be also written to the standard out put. By default the result of the test are stored in the result.txt which can be changed by using --output option.

Running all test cases from a given path

Multiple test cases can be easily loaded and run using --tests option. The expected value for this option is a directory where test case plug-ins are stored.

 $ robottestingframework-testrunner --verbose --tests ~/my-plugins

If the --recursive switch is given with the --tests option, the sub-folders are also searched for plug-ins.

Test suite

By definition, a test suite is a set of test cases which share the same test fixture. In Robot Testing Framework a set of test cases (plug-ins) can be grouped in a test suite using a XML file. Here is an example of test suite:

<?xml version="1.0" encoding="UTF-8"?>

<suite name="my suite">
    <description> a simple example </description>
    <environment> --robot simulator </environment>
    <fixture param="--from myfixture.ini"> libfixture.so </fixture>
    <fixture param="--from myfixture2.ini"> libfixture.so </fixture>

    <test param="--from myconf.ini"> libtest1 </test>
    <test type="dll"> libtest1 </test>
    <test param=""> ~/mytest/libtest2.so </test>
    <test repetition="5"> ~/mytest/test3.lua </test>
    <test param=""> ~/mytest/test4.py </test>
    <test param=""> ~/mytest/test5.rb </test>
    ...
</suite>

The example demonstrates a test suite which includes two test cases (libmytest1.so and libmytest2.so). An optional string parameter can be also specified for each test using param attribute in the test tag. The parameter string is parsed into (argc/argv) format and will be passed to the test case using robottestingframework::TestCase::setup(int argc, char** argv). Alternatively, the raw value of the parameter can be access in the test plug-in implementation using robottestingframework::TestCase::getParam method.

The environment can be used in the implementation of the test cases to know in which setup (e.g., simulation, real world environment) the test case is employed. The environment value can also be accessed in the test plug-in implementation using robottestingframework::TestCase::getEnvironment method.

If a fixture manager plug-in is specified using <fixture> tag, it will be loaded by the robottestingframework-testrunner and is set for the current test suite. When suite is executed, the fixture manager will be called automatically to setup and monitor the test fixture which is implemented in the fixture plug-in. (Please see the examples/fixture-plugin folder for an example of implementing fixture plug-ins.)

Notice that, for libtest2.so, a complete path to the plug-in file is given. Thus, the robottestingframework-testrunner does not look into system default paths (e.g., LD_LIBRARY_PATH) for the plug-in file and it expects to find the plug-in file using the given path. However, this is not the case for libtest1.so.

The robottestingframework-testrunner usually checks for the extension of the plug-in file name to recognize the type of the plug-in. There is also possibility to explicitly indicate the type of the plug-in using the type attribute. For example if type="dll", then the plug-in test should be given without the extension. Depending on which platform (Windows, Linux or Mac) the test is running, the robottestingframework-testrunner looks for the correct (shared library) plug-in name (.dll, .so or .dylib).

Running a single test suite

A single test suite (XML file) can be run using --suite option. For example:

 $ robottestingframework-testrunner --verbose --suite ~/my-suites/suite.xml

Running multiple test suites

Multiple test suites can be loaded and rune using --suites option. The expected value for this option is a directory where test suites (XML files) are stored.

 $ robottestingframework-testrunner --verbose --suites ~/my-suites

If the --recursive switch is given with the --suites option, the sub-folders are also searched for test suites.

Monitoring tests from your browser

The robottestingframework-testrunner uses robottestingframework::WebProgressListener to provide a simple web server which allows you to monitor the test's running progress from a browser. To do that, you need to run a test/suite by the robottestingframework-testrunner using --web-reporter option.

 $ robottestingframework-testrunner --web-reporter --verbose --suites ~/my-suites

Then open a web browser and type http://127.0.0.1:8080 or the remote address where the robottestingframework-testrunner has been launched. You can also change the robottestingframework-testrunner default port by using --web-port <port number> option.

Notice that, the web reporting functionality may not be available by default and you need to enable it when building the Robot Testing Framework by turning on the ENABLE_WEB_LISTENER from the cmake.

Using Robot Testing Framework for stress testing

For stress testing, it is possible to repeat a test case for desired iterations. This can be easily done by providing the number of the test runs as the parameter (e.g., --repetition 5) for the robottestingframework-testrunner or setting as the attribute of the test case within a test suite XML file. The robottestingframework-testrunner repeats running the test for the desired iterations if the test does not fail. Notice that, only the robottestingframework::TestCase::run() will be called repeatedly.

 $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.lua --repetition 5

The robottestingframework-testrunner options

Here is the full list of robottestingframework-testrunner options:

usage: robottestingframework-testrunner [options] ...
options:
  -t, --test            Runs a single test from the given plugin file. (string [=])
      --tests           Runs multiple tests from the given folder which contains plugins. (string [=])
  -s, --suite           Runs a single test suite from the given XML file. (string [=])
      --suites          Runs multiple test suites from the given folder which contains XML files. (string [=])
  -o, --output          The output file to save the result. Default is result.txt (string [=])
      --output-type     The output file type (text, junit) (string [=text])
  -p, --param           Sets the test case parameters. (Can be used only with --test option.) (string [=])
  -e, --environment     Sets the test case environment. (Can be used only with --test option.) (string [=])
      --repetition      Sets the test run repetition. (Can be used only with --test option.) (j [=0])
  -w, --web-reporter    Enables web reporter
      --web-port        Sets the web reporter server port. (The default port number is 8080.) (i [=8080])
  -r, --recursive       Searches into subfolders for plugins or XML files. (Can be used with --tests or --suites options.)
  -d, --detail          Enables verbose mode of test assertions.
  -v, --verbose         Enables verbose mode.
      --version         Shows version information.
  -?, --help            print this message

Author: Ali Paikan

Contents