Approaches to testing Tango devices

Overview

The follow sections detail different approaches that can be used when automating tests. This includes starting the real devices as normal in a Tango facility, using the DeviceTestContext for a more lightweight test, a hybrid approach mixing DeviceTestContext and real Tango devices in a Tango facility, and starting multiple devices with the DeviceTestContext and MultiDeviceTestContext.

Testing a single device without DeviceTestContext

Note

This approach is not recommended for unit testing.

Testing without a DeviceTestContext requires a complete Tango environment to be running (this environment is orchestrated by Makefiles and Docker containers in our Tango Example repo). That is, the following four components/processes should be present and configured:

• DSConfig tool

• Tango Databaseds Server

• MySQL/MariaDB

• Tango Device Server (with Tango device under test inside it)

In order to successfully constitute a working Tango environment, the following sequence of operations is required:

1. A running MySQL/MariaDB service.

2. The Tango Databaseds Server configured to connect to the database.

3. The DSConfig tool can be run to bootstrap the database configuration of the Tango Device based on configuration from a file.

4. The Tango Device Server that has been initialised and running the Tango Device.

5. In the test, you can instantiate a PyTango DeviceProxy object to interact with the Tango device under test.

This is a lot of infrastructure and complicated to orchestrate - it is not conducive to lightweight, fast running unit tests. Thus it is not recommended.

Figure 1. A schematic diagram showing the agents involved when testing a Tango device using the real Tango database and their interactions.

Examples:

• test_2_test_server_using_client.py

• test_3_test_motor_server.py

• test_4_test_event_receiver_server.py

Testing a single device with DeviceTestContext

A utility class is provided by PyTango that aids in testing Tango Devices. It automates a lot of the operations required to start up a Tango runtime environment.:

from tango.test_context import DeviceTestContext

The DeviceTestContext accepts a Tango Device Python class, as an argument, that will be under test (PowerSupply). It also accepts some additional arguments such as properties - see the method signature of DeviceTestContext constructor. It will then do the following: Generate stubbed data file that has the minimum configuration details for a Tango Device Server to initialise the Tango Device under test (PowerSupply). It will start the Tango Device Server that contains the Tango Device (in a separate thread by default, but optionally in a subprocess). DServer is a “meta” Tango Device that provides an administrative interface to control all the devices in the Tango Device Server process. The DeviceProxy object can be retrieved from the DeviceContext and can be invoked to interact with Tango Device under test. A DeviceProxy object will expose all the attributes and commands specified for the Tango Device as Python objects, but invoking them will communicate with the real device via CORBA. If events are used, these are transported via ZeroMQ.

Figure 2. A schematic diagram showing the agents involved when testing a Tango device using the DeviceTestContext and their interactions.

You may now proceed to exercise the Tango Device’s interface by invoking the appropriate methods/properties on the proxy:

Example Code Snippet	Tango Concept	Description
powersupply_proxy.turn_on()	Tango Command	An action that the Tango Device performs.
powersupply_proxy.voltage	Tango Attribute	A value that the Tango Device exposes.

Example:

• test_1_server_in_devicetestcontext.py

Testing a single device with DeviceTestContext combined with a real device(s) using the Tango database

This use case first requires the whole test infrastructure described in use case 1 above to be up before the tests can be run against the device (DishLeafNode) in the DeviceTestContext. The following sequence of events occur to run rests against the device (DishLeafNode):

• Set up the test infrastructure for the real device - DishMaster (all the steps defined for use case 1 above apply).

• Set up the test infrastructure for the device (DishLeafNode) in the DeviceTestContext (all steps in use case 2 above apply).

• Create a proxy (dish_proxy) which exposes the attributes and commands of the real device to be tested.

  • There’s a proxy in the provisioned DeviceTestContext which knows about the real device but cannot expose its attributes and commands in that context, hence the need for the dish_proxy.

Figure 3. A schematic diagram showing the agents involved when testing multiple Tango devices using the DeviceTestContext together with the real Tango database and their interactions.

Examples:

• DishLeafNode/conftest.py

Testing with multiple DeviceTestContexts

Warning

This approach is not recommended - rather use MultiDeviceTestContext.

The testing scenario depicted in Figure 3 can be implemented without using the real Tango database. In this use case, the underlying device (DishMaster) is provisioned using the DeviceTestContext. Just like in the use case above, another proxy (dish_proxy) is created to expose the commands and attributes of the DishMaster Device. The sequence of events which take place to provision each of these DeviceTestContexts are exactly the same as described in use case 1. This is not recommended because it can be done more easily using the MultiDeviceTestContext, as shown in the next section.

From PyTango 9.5.0, this will require setting process=True on the nested DeviceTestContext instances.

Figure 4. A schematic diagram showing the agents involved when testing multiple Tango devices using the DeviceTestContext and their interactions.

Examples:

• Tango forum post

Testing with MultiDeviceTestContext

There is another testing class available in PyTango: MultiDeviceTestContext, which helps to simplify testing of multiple devices. In this case the multiple devices are all launched in a single device server.:

from tango.test_context import MultiDeviceTestContext

The testing scenario depicted in Figure 4 can be implemented with just a single MultiDeviceTestContext instead of two DeviceTestContext instances (and still without using the real Tango database). In this use case, both devices (DishMaster and DishLeafNode) are provisioned using the MultiDeviceTestContext. Just like in the use case above, another proxy (dish_proxy) is created to expose the commands and attributes of the DishMaster Device to the test runner. The sequence of events which take place to provision this MultiDeviceTestContexts is similar that use case 1. The main difference is the devices_info the must be specified beforehand. Here we can define the devices that must be started, their names, and initial properties.

Figure 5. A schematic diagram showing the agents involved when testing multiple Tango devices using the MultiDeviceTestContext and their interactions.

Examples:

• MultiDeviceTestContext with fixtures

Issues

1. A single process that attempts to use a DeviceTestContext multiple times in threaded mode (so kwarg process=False, or unspecified), will get a segmentation fault on the second usage. The segfault can be avoided using the pytest-forked plugin to run tests in separate processes (Linux and macOS, but not Windows). Either mark individual tests with the @pytest.forked decorator, or use the pytest --forked command line option to run every test in a new process. Running each test in a new process slows things down, so consider if all tests or just some tests need this.

2. Another way to avoid the segfault with multiple uses of DeviceTestContext is by setting the kwarg process=True. In this case we don’t need the forked execution, but consider the disadvantages in the previous section.

3. Forwarded attributes do not work.

4. There is no way to unit test (in the strict definition), since the Tango device objects cannot be directly instantiated.

5. The DeviceTestContext is quite a heavyweight utility class in terms of the dependent components it needs to orchestrate so that testing can be done. It requires the Tango runtime, including ZeroMQ for events, and a Database stub file as a minimum.

Note

The same issues apply to MultiDeviceTestContext.

The process kwarg: thread vs. subprocess modes

When using DeviceTestContext or (MultiDeviceTestContext) with the kwarg process=False (default), the Tango Device server runs in the same operating system process as the code that starts the DeviceTestContext (normally the test runner). With process=True, a new subprocess is created and the device server runs in that subprocess. In other words, a different operating system process to the test runner. In both cases, the test runner can communicate with the device is via a client like DeviceProxy or AttributeProxy.

There is a subtle detail to note when using the process=True option with a nested device class. Consider the following example:

from tango import DevState
from tango.server import Device
from tango.test_context import DeviceTestContext


def test_empty_device_in_state_unknown():
    class TestDevice(Device):
        pass

    with DeviceTestContext(TestDevice, process=False) as proxy:
        assert proxy.state() == DevState.UNKNOWN

This will work fine, using thread-based mode. However, if we use the subproces mode, i.e., process=True, we will get an error about serialisation of the class using pickle, like: AttributeError: Can't pickle local object 'test_empty_device_in_state_unknown.<locals>.TestDevice'. It fails when Python creates a subprocess using multiprocessing - the nested class definition cannot be passed to the subprocess.

One solution is to move the test class out of the function:

from tango import DevState
from tango.server import Device
from tango.test_context import DeviceTestContext


class TestEmptyDevice(Device):
    pass


def test_empty_device_in_state_unknown():
    with DeviceTestContext(TestEmptyDevice, process=True) as proxy:
        assert proxy.state() == DevState.UNKNOWN

The next detail to consider is that the memory address space for two processes is independent. If we use process=False the device under test is running in the same process as the test runner, so we can access variables inside the class being tested (or even the device instance, if we keep a reference to it). With process=True the test runner cannot access the device server’s memory.

Example of accessing class variables and device internals (only possible with process=False):

from weakref import WeakValueDictionary

from tango.server import Device
from tango.test_context import DeviceTestContext


class TestDeviceInternals(Device):
    class_variable = 0
    instances = WeakValueDictionary()

    def init_device(self):
        super().init_device()
        TestDeviceInternals.class_variable = 123
        TestDeviceInternals.instances[self.get_name()] = self
        self._instance_variable = 456


def test_class_and_device_internals_accessible_with_process_false():
    with DeviceTestContext(TestDeviceInternals, process=True) as proxy:
        assert TestDeviceInternals.class_variable == 123

        device_instance = TestDeviceInternals.instances[proxy.dev_name()]
        assert device_instance._instance_variable == 456

The weakref.WeakValueDictionary isn’t critical to this test (it could have been a standard dict), but it is shown as a way to avoid reference cycles in the instances dict. For example, if a device server were creating and deleting device instances at runtime. The reference cycles would prevent the cleanup of device instances by Python’s garbage collector.

Acknowledgement

Initial content for this page contributed by the Square Kilometre Array.