Using plug-in to develop test cases

Author

Ali Paikan

Description

Test cases can be written as dynamic loadable libraries (e.g., .so, .dll) to be loaded and executed as separate plug-ins. This allows to have a simple test runner which loads and run the tests at run time without any needs to recompile the test runner. Using a simple example, we show how this can be done.

Note

The source code of the tutorial can be also found in the 'examples/plugin' folder within the Robot Testing Framework project directory. To compile it, you just need to enable the BUILD_EXAMPLES=ON flag of the CMake (if it is not already enabled) and build the Robot Testing Framework.

Requirements

You do not need any external library to use the plug-ins. All you need is to simply enable the plug-in system in the Robot Testing Framework and recompile it if it is not already enabled (it should be enabled by default).

   $ cd robot-testing-framework/build
   $ cmake ../ -DENABLE_PLUGIN=ON

Writing the plug-in

Create a class which inherited from the TestCase class. Override the run() method and/or setup(int argc, char** argv) , tearDown() if needed. The parameters set for the test case using robottestingframework::TestCase::setParam are parsed into standard (int argc, char**argv) format and are passed to the setup method. The original string parameter is also accessible using robottestingframework::TestCase::getParam. Here is an example:

/*
 * Robot Testing Framework
 *
 * Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2.1 of the License, or (at your option) any later version.
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this library; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
 */


#ifndef ROBOTTESTINGFRAMEWORK_MYTEST_H
#define ROBOTTESTINGFRAMEWORK_MYTEST_H

#include <robottestingframework/TestCase.h>

class MyTest : public robottestingframework::TestCase
{
public:
    MyTest();

    bool setup(int argc, char** argv) override;

    void tearDown() override;

    void run() override;
};

#endif // ROBOTTESTINGFRAMEWORK_MYTEST_H

Implement the test body in the run() method and add use the 'PREPARE_PLUGIN(...)' macro to prepare the plug-in. The macro implements the necessary functionalities to make your test class be accessible from a shared library.

/*
 * Robot Testing Framework
 *
 * Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2.1 of the License, or (at your option) any later version.
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this library; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
 */


#include "MyTest.h"
#include <robottestingframework/TestAssert.h>
#include <robottestingframework/dll/Plugin.h>

using namespace robottestingframework;

ROBOTTESTINGFRAMEWORK_PREPARE_PLUGIN(MyTest)

MyTest::MyTest() :
        TestCase("MyTest")
{
}

bool MyTest::setup(int argc, char** argv)
{
    ROBOTTESTINGFRAMEWORK_TEST_REPORT("running MyTest::setup...");
    return true;
}

void MyTest::tearDown()
{
    ROBOTTESTINGFRAMEWORK_TEST_REPORT("running MyTest::teardown...");
    // assert an arbitray error for example.
    ROBOTTESTINGFRAMEWORK_ASSERT_ERROR("this is just for example!");
}

void MyTest::run()
{
    ROBOTTESTINGFRAMEWORK_TEST_REPORT("testing integers");
    ROBOTTESTINGFRAMEWORK_TEST_FAIL_IF_FALSE(2 < 3, "is not smaller");
    int a = 5;
    int b = 3;
    ROBOTTESTINGFRAMEWORK_TEST_FAIL_IF_FALSE(a < b, Asserter::format("%d is not smaller than %d.", a, b));
}

Writing the test runner

Note

Basically you do not need to develop a test runner. There is the robottestingframework-testrunner utility (see Running test case plug-ins using robottestingframework-testrunner) already implemented in Robot Testing Framework which does the job for you. However, the following example is just for your reference and to understand better how a test plug-ing can be loaded and executed from the code.

Your plug-in is ready and can be compiled and built. Before doing, that we show how to use robottestingframework::DllPluginLoader to develop a simple test runner for loading the plug-in and running the test. It is quite simple. First create an instance of robottestingframework::plugin::DllPluginLoader class and call its open() method with the plug-in library (.dll, .so, ...) filename as its paramter. The robottestingframework::plugin::DllPluginLoader loads the library and return a pointer to the robottestingframework::TestCase implemented into the plug-in. If the plug-in cannot be loaded or it is not an Robot Testing Framework plug-in file, the robottestingframework::plugin::DllPluginLoader::open() returns a null pointer and the error message can be gotten using robottestingframework::plugin::DllPluginLoader::getLastError(). When you have an instance of your test case, it can be used to run the test as usual (see Some examples if you are not familiar with running a test case). In the following example we use a robottestingframework::TestRunner to execute our test:

/*
 * Robot Testing Framework
 *
 * Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2.1 of the License, or (at your option) any later version.
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this library; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
 */


#include <robottestingframework/ConsoleListener.h>
#include <robottestingframework/TestCase.h>
#include <robottestingframework/TestResult.h>
#include <robottestingframework/TestRunner.h>
#include <robottestingframework/dll/DllPluginLoader.h>

#include <stdio.h>


using namespace robottestingframework;
using namespace robottestingframework::plugin;

int main(int argc, char* argv[])
{

    if (argc < 2) {
        printf("Usage: %s <plugin file name>\n", argv[0]);
        printf("for example: %s libmytest.so\n", argv[0]);
        return 0;
    }

    // load the test case plugin
    printf("Loading the plugin... \n");
    DllPluginLoader loader;
    TestCase* test = loader.open(argv[1]);
    if (test == NULL) {
        printf("%s\n", loader.getLastError().c_str());
        return 0;
    }

    // create a test listener to collect the result
    ConsoleListener listener(false);

    // create a test result and add the listeners
    TestResult result;
    result.addListener(&listener);

    // create a test runner and run the test case
    TestRunner runner;
    runner.addTest(test);
    runner.run(result);

    return 0;
}

Build and run the test

Now you can compile and build the plug-in and the runner. There is a CMake file in the examples/plugin folder which helps you to compile and build your test plugin and the simple test runner. Make sure that RobotTestingFramework_DIR environment variable is correctly set to point your Robot Testing Framework build or installed directory so that CMake can find the required modules to configure the project.

# Robot Testing Framework
#
# Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA


cmake_minimum_required(VERSION 3.5)

find_package(RobotTestingFramework COMPONENTS DLL)

add_library(mytest MODULE MyTest.cpp MyTest.h)
target_link_libraries(mytest RobotTestingFramework::RTF
                             RobotTestingFramework::RTF_dll)

add_executable(simple_run run.cpp)
target_link_libraries(simple_run RobotTestingFramework::RTF
                                 RobotTestingFramework::RTF_dll)

Now you can build and run the test as follows:

$ cd examples/plugin; mkdir build
$ cd build; cmake ../; make;
$ ./simple_run libmytest.so
 Loading the plugin...
 Starting test runner.
 Test case MyTest started...
 [INFO]  (MyTest) reports: running MyTest::setup...
 [INFO]  (MyTest) reports: testing integers
 [FAIL]  (MyTest) checking (a<b): 5 is not smaller than 3.
 [INFO]  (MyTest) reports: running MyTest::teardown...
 [ERROR] (MyTest) asserts error with exception: this is just for example!
 Test case MyTest failed!
 Ending test runner.