--- domain: rfc.tango-controls.org shortname: 12/PubSub name: Publisher-Subscriber protocol status: draft editor: Thomas Juerges (thomas.juerges@skao.int) --- # 12/Publisher-Subscriber protocol ## Preamble Copyright (c) 2019 Tango Controls Community. This Specification is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This Specification is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, see . x This Specification is a [free and open standard](https://web.archive.org/web/20161002091934/http://www.digistan.org/open-standard:definition) and is governed by the Digital Standards Organization's [Consensus-Oriented Specification System](https://web.archive.org/web/20161002092144/http://www.digistan.org/spec:1/coss). The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC-2119](https://tools.ietf.org/html/rfc2119). ## Publisher-Subscriber protocol Specification It is a relationship between Tango client (Client) and Tango server (Server). Based on criteria negotiated as part of a subscription, Tango client will receive updates. Beyond a set of basic requirements, various refinements are addressed. These refinements include: periodicity of updates, filtering out updates based on Tango server configuration and fault tolerance. ## Basic Concepts This document refers to other Tango Controls RFCs, corresponding link is always given in this case. Here is given a brief description of the main concepts. This document targets arbitrary software developer(s) who are to implement *Tango Client Library* and *Tango Server Library*. *Tango Server* (Server) - a process that runs on a machine. *Tango Device* (Device) - for more details see [2/Device](../2/Device.md). *Tango Admin Device* (Admin) - for more details see [8/Server](../8/Server.md). *Tango Client* (Client) - a process that instantiates communication with *Tango Server* using Request-Reply protocol [10/RequestReply](../10/RequestReply.md). *Tango Client Library* (client kernel library) - library that encapsulates communication with *Tango Server* and provides *Tango Client API* to a user. *Tango Client API* (client API) - an API that user uses to develop applications based on communication with *Tango Server*s. *Tango Server Library* (server kernel library) - library that encapsulates *Tango Server* features and provides *Tango Server API* to a user. *Tango Server API* (server API) - an API that user uses to develop *Tango Device* functionality. *Upstream Tango Server/Device* (upstream server/device) - a *Tango Device* to which updates *client* subscribes for. *Polling thread* (polling thread) - a thread within *Tango Server* that polls periodically defined *Tango Device*'s attributes/sometimes commands. *Tango Publisher-Subscriber protocol* (pub-sub) - is a sub-set of *Tango Request-Reply protocol* [10/RequestReply](../10/RequestReply.md) that defines a relationship between *client* and *server* established via a negotiation phase and lasts till explicitly broken. During this relationship client receives updates published by *upstream server*. Updates are sourced from server's *polling thread* or directly through *server API*. Updates are filtered according to *device*'s configuration. *Tango Event* (event) - an update sent by *server*. *Tango Event Callback* (callback) - a piece of code that *client library* executes when an update is received. *Channel* - virtual client-server point-to-point connection for transmitting events ## Definitions This section offers a number of ABNF formal definitions for entities repeatedly used below in this document. > **NOTE**: in the below ABNF some basic definitions are used as rules, spaces are replaced with '_' > **NOTE**: in the below ABNF data types are sometimes embedded into the rule e.g. bool:isExcept is equivalent to isExcept = true / false ```ABNF endpoint = URL ; usually with port SUBSCRIBER_INFO = upstream_device attribute_name action EVENT_TYPE client_idl_ver SUBSCRIPTION_INFO = server_library_release_ver upstream_device_idl_ver zmq_sub_event_hwm rate ivl zmq_release_ver heartbeat_endpoint event_endpoint 1*alternate_heartbeat_endpoint 1*alternate_event_endpoint HEARTBEAT_CHANNEL EVENT_CHANNEL EVENT_INFO = EVENT_DATA/1*EVENT_ERROR bool:isExcept ; see below EVENT_DATA = ATT_CONF_DATA/PIPE_DATA/DATA_READY_DATA/INTERFACE_CHANGE_DATA/ATTRIBUTE_DATA ATT_CONF_DATA = ATTRIBUTE_PROPERTIES ; Attribute properties from [4/Attribute](../4/Attribute.md) PIPE_DATA = int:size name timeVal PIPE_BLOB ; pipeBlob must be defined in [7/Pipe](../7/Pipe.md) DATA_READY_DATA = attribute_name data_type int:counter INTERFACE_CHANGE_DATA = *ATTRIBUTE_PROPERTIES *COMMAND_META_INFO bool:deviceStarted ATTRIBUTE_DATA = ATTRIBUTE_DEFINITION ; Attribute definition from [4/Attribute](../4/Attribute.md) EVENT_ERROR = reason severity desc origin EVENT_TYPE = "INTERFACE_CHANGE"/"PIPE_EVENT"/"ATTR_CONF_EVENT"/"CHANGE_EVENT"/"PERIODIC_EVENT"/"ARCHIVE_EVENT"/"ALARM_EVENT"/"USER_EVENT"/"DATA_READY_EVENT" EVENT_CHANNEL = channel HEARTBEAT_CHANNEL = channel channel = string ; implementation specific ``` > **NOTE**: In Tango V9 EVENT_DATA MAY include source idl version, event type ## Runtime requirements Client and server are up and running. Server is reachable from client i.e. may communicate using Request-Reply protocol [10/RequestReply](../10/RequestReply.md). ## Publisher-Subscriber protocol ![Subscription_1.png](images/Subscription_1.png) The main goal of this protocol implementation is to allow a Client receive events from a Server: * Server MUST publish events of EVENT_TYPE * Server library MUST provide an API to publish events of EVENT_TYPE * Client SHOULD process events * Client library SHOULD provide user an API to register user defined event callbacks ### Negotiation ![Subscription_2.png](images/Subscription_2.png) Negotiation phase implies that client and server exchange required data using Tango commands [3/Command](../3/Command.md): * Client MUST notify server providing SUBSCRIBER_INFO to establish Publisher-Subscriber relationship > in Tango 9 the above is done by executing EventSubscriptionChange admin command > Tango 9 forces client to explicitly execute ZmqEventSubscriptionChange to use ZMQ based implementation. * Server MUST respond to the client providing SUBSCRIPTION_INFO * Client MUST connect to upstream server using provided SUBSCRIPTION_INFO * Client SHOULD postpone event subscription process in case of connection failure and try again later Tango Event System uses concept of channels for delivering events. Channel is established between client and admin. There are two channels per subscription: heartbeat and event. * Client MUST establish channel or multicast connection to heartbeat/event endpoints * Server MAY send and client MAY iterate over alternative heartbeat/event endpoints to establish connection * Client MAY NOT use provided heartbeat/event channels to identify event source ### Subscription * Server MUST send events via corresponding EVENT_CHANNEL * Server MUST send HEARTBEAT via corresponding HEARTBEAT_CHANNEL event every 9 seconds * Client SHOULD indicate its interest in subscription not later than every 600s ### Cancel event subscription * Client MAY cancel its subscription * Server SHOULD NOT send events to client whose subscription is canceled ### Fault tolerance Tango Pub-Sub protocol implies a number of fault tolerance techniques: - server heartbeat - transparent client reconnection - client subscription confirmation - events client/server buffer * Client SHOULD try to re-subscribe once HEARTBEAT event is missing * Client MAY try to re-subscribe indefinitely * Server MAY cancel client's subscription if it does not confirm its interest * Client/Server SHOULD buffer the (in/out)coming events to prevent overload and overflow * Clients SHOULD be notified in case of events overflow * Client SHOULD be notified in case of missing events > e.g.: In the current C++ implementation at least, the client callback is executed with an API_MissedEvents error event when the client library detected that some events got lost. There is a mechanism with some counters associated with each event to help detecting this kind of problems