Migration from boost.python to pybind11

Migration from boost.python to pybind11#

PyTango 10.1 was moved from Boost.Python to PyBind11 for the C++ extension layer. The vast majority of code using PyTango will work as before, however these are some API changes, which are described here.

Rationale#

First of all, why did we do it? Boost.Python was the first successful library which offered the possibility to bind two almost completely opposite languages in one project: dynamically-typed, interpreted with fully hidden from user memory management Python, and strictly typed, compiled, with direct memory control C++. However, even though Boost.Python is still around, there are several successors, which took the best from Boost.Python, but made it easier, better and more modern. In PyTango, we decided to move our project to pybind11, the de facto standard for C++/Python bindings, and there were several reasons:

1. Simplicity & Cleaner Code#

Pybind11 provides a modern, intuitive API for binding C++ to Python with minimal boilerplate code. Unlike Boost.Python, which requires extensive macro usage and setup, Pybind11 enables developers to write clean and maintainable code. We also used the opportunity simplify, remove duplication and improve the code layout.

2. Header-Only Library (No Linking Hassles) with Reduced Dependencies#

Pybind11 is a header-only library, meaning it does not require additional linking steps. Using Boost.Python means bringing in the entire Boost library, which is large and mostly unused. It was especially complicated to build for Windows.

3. Better C++11/14/17/20 Support#

Pybind11 is designed with modern C++ standards in mind, making it easier to work with smart pointers, lambdas, move semantics, and other advanced C++ features.

4. Better Python Version Compatibility#

Boost.Python often lags in supporting newer Python versions and core dependencies, leading to maintenance challenges. E.g., the official Boost release to support NumPy 2.0 was about 6 months late. We also had to re-build the library for each new version of Python, on each supported platform.

5. Better Documentation#

To be honest, PyBind11 documentation cannot be really called perfect, but compared to Boost.Python it is much better.

6. Easier Debugging & Maintenance#

Pybind11 is easier to debug compared to Boost.Python. Just look to the typical frame stack of Boost function call vs. PyBind11.

7. Bonus#

As a result of some rigorous testing, a number of old bugs were discovered and fixed. We also improved our test coverage along the way.

API changes#

And now what changed from PyTango user side:

Version constant#

Obviously, BOOST_VERSION constant and Compile class member was replaced with PYBIND11_VERSION

Pipes removed#

The pipes bindings were not re-written, since pipes in general are scheduled to be removed from Tango, and we decided not to spend time to their adaptation. Due to this, the following methods/classes were removed:

tango.pipe module with PipeConfig class.
tango.pipe_data module with PipeData class.
Pipe, PipeEventData, PipeWriteType, PipeInfoList, PipeInfo, DevicePipe, UserDefaultPipeProp, CmdArgType.DevPipeBlob, EventType.PIPE_EVENT classes.
DeviceClass: pipe_list member, and get_pipe_list, get_pipe_by_name methods.
DeviceImpl: push_pipe_event method. 5 high-level API Device: pipe class from tango.server used to define pipe as class member or by method decorator.
DeviceProxy: high-level DeviceProxy does not list pipes as class members, high-level write and read to/from pipes is not possible.
DeviceProxy: get_pipe_config, set_pipe_config, read_pipe, write_pipe, get_pipe_list methods.

Enums#

Pybind11 has a different mechanism to export Enums (pybind11 typically creates enum as close to Python’s enums as possible, while Boost does something very different by creating a singleton). Due to this the object identity (e.g., device.State() is DevState.ON) does not work any more. You must use equality of value: device.State() == DevState.ON.

Also __repr__() of Enums has changed: with Boost if you did repr(DevState.ON) you got "tango._tango.DevState.ON", while now you will get "<DevState.ON: 0>". So if you do string analysis of Enum representation in you code, please adapt it. The __str__() method for Enums has not changed: str(DevState.ON) is still "ON".

As the pybind11 Enums are more like Python Enums, the integer value is retrieved with .value, instead of .real. E.g., DevState.ON.value. Or better, just cast to an integer, like int(DevState.ON). More generally, the new enums don’t inherit from Python’s int class, so the related attributes are no longer present:

Fields removed: real, imag, numerator, denominator.
Methods removed: as_integer_ratio, bit_count, bit_length, conjugate, from_bytes, to_bytes, is_integer.

If enums are used to parameterize pytest test cases, the test names will use integers instead of the name of the enum instance. This can be fixed by providing a callable for the IDs: @pytest.mark.parametrize(..., ids=str).

Type coercion for commands with boolean return type#

A command that returns a boolean, dtype_out=bool, will no longer convert a None value into False. The caller will get a DevFailed exception. Either return a value of the correct type, or change the return type to None (i.e., DevVoid).

from tango.server import Device, command


class MyDevice(Device):

    @command(dtype_out=bool)
    def bad_boolean_return_command(self):
        pass  # don't do this and don't return None

    @command(dtype_out=bool)
    def good_boolean_return_command(self):
        return False

    @command()
    def good_void_command(self):
        pass

    @command(dtype_out=None)
    def another_good_void_command(self):
        pass

Attribute and WAttribute: dim_x and dim_y#

The tango.Attribute.set_value(), tango.Attribute.set_value_date_quality(), and tango.WAttribute.set_write_value() methods no longer support dim_x and dim_y parameters (long deprecated).

Pushing events: dim_x and dim_y#

The various methods for pushing events no longer support the dim_x and dim_y arguments. The dimensions are determined automatically from the data. Methods affected:

`tango.server.Device`	`tango.LatestDeviceImpl`
`push_alarm_event()`	`push_alarm_event()`
`push_archive_event()`	`push_archive_event()`
`push_change_event()`	`push_change_event()`
`push_event()`	`push_event()`

Keyword args for set_change_event, etc.#

The various methods for declaring that a device pushes its own events, e.g., set_change_event, can now be used with keyword arguments. This can make code more readable. Usage is optional.

from tango.server import Device

class MyDevice(Device):
    def init_device(self):
        # instead of:
        self.set_change_event("State", True, False)
        # rather use:
        self.set_change_event("State", implemented=True, detect=False)

This kind of change applies to:

`tango.server.Device`	`tango.LatestDeviceImpl`
`set_alarm_event()`	`set_alarm_event()`
`set_archive_event()`	`set_archive_event()`
`set_change_event()`	`set_change_event()`
`set_data_ready_event()`	`set_data_ready_event()`

`tango.Attribute`	`tango.Attr`
`set_alarm_event()`	`set_alarm_event()`
`set_archive_event()`	`set_archive_event()`
`set_change_event()`	`set_change_event()`

Asynch attribute read/command inout#

The callback result from command_inout_asynch() is still a CmdDoneEvent object. However, the argout field behaves differently in case the command failed (exception on server side, or timeout on client side). Accessing argout will now raise a DevFailed exception, instead of returning None. Check the err field before trying to access argout.

if not result.err:
    print(result.argout)

The callback result from read_attribute_asynch() and read_attributes_asynch() is still a AttrReadEvent object. However, if there was a timeout on the client side, the argout field is an empty list, [], instead of None, for consistency with a successful read. The err field is unchanged, and will be True in case of a timeout.

Std vectors#

Vectors such as StdStringVector, StdLongVector, StdDoubleVector are now implicitly convertible to Python lists, so there is no need to convert methods arguments to them. Similarly, methods return values changed to Python list[str], list[int] and list[float], respectively
StdGroupAttrReplyVector, StdGroupCmdReplyVector, StdGroupReplyVector aren’t exported any more (due to bad implementation on cpptango side). Instead, user receives list[GroupAttrReply], list[GroupCmdReply], list[GroupReply], respectively

Docstring#

Due to implementation details, almost all docstrings for classes, methods and enums in pybind11 aren’t mutable, so if you need to change it in your code - now you must create your own class, method, enum inheriting/calling parent method and put your docstring there

Attribute configuration structs interface frozen#

Many structs related to attributes and attribute configuration used generate a PytangoUserWarning if you assigned to a field that didn’t exist in the structure.

E.g., tango.ChangeEventProp().invalid_field = "123" would generate a warning.

Now, the interfaces of these structs are frozen, so attempting to write to a non-existent field will raise an AttributeError.

The list of affected structs is:

ArchiveEventProp
AttributeAlarm
AttributeConfig
AttributeConfig_2
AttributeConfig_3
AttributeConfig_5
ChangeEventProp
EventProperties
MultiAttrProp
PeriodicEventProp

Modules removed#

The following top-level modules can no longer be imported. They were for defining docstrings, and didn’t have classes or functions for public use: tango.api_util, tango.callback, tango.device_data, tango.exception.

And this should be all notable API changes!