Ohara Lab

Distributed Models for Device Control

There are several ways to model a control system. Each has advantages and disadvantages depending on the circumstances; the models discussed later are not necessarily superior. It is important to choose an appropriate model when developing software to control equipment. These are schematic representations, and details may differ from actual implementations.

I am not a specialist in information technology or control engineering, so some of my terminology may be imprecise. I know that expressions such as “server-to-server” and “server–client” can sound rather naive when viewed from the perspective of socket communication patterns, but they are easier for non-specialists to understand.

Representative Schematics of Distributed Control Models

The diagram below shows a common development setup in an experimental environment. A detector (Device A) operates on a dedicated control PC, while hardware for moving stages (Device B) is also connected to that PC. Running a control program on the PC allows it to operate both Device A and Device B. At first glance, this seems sufficient. However:

In practice, this simple model is often unsuitable beyond testing or small demonstration setups.

Centralized Device Control

In practice, many systems described as centralized are better understood as collections of several centralized systems, as discussed later. Thus, the following is perhaps little more than a schematic model. Nevertheless, understanding the distinction between centralized ↔ distributed architectures is important for sound control-system design.

Device Control Using a Full-Fledged Distributed Model

Full-fledged distributed systems such as MADOCA/DARUMA and TANGO use technologies such as ZMQ and CORBA to implement distributed control architectures.

Server-to-server communication connects the nodes, allowing users to operate related nodes through access to a single node. Omitting the details here, MADOCA can place message servers at a finer granularity than one server per device. This allows it to maintain connections across more complex combinations of networks and devices. For a single beamline or individual experiment, of course, this may be more functionality than is strictly necessary.

In any case, as long as users access individual device servers transparently through device proxies or similar interfaces, the user-side code looks much like an ordinary, single Python program. The underlying distributed structure need not be a major concern for the user.

A Practical Configuration (Full-Fledged Distributed Model)

This approach makes active use of server-to-server communication. In a typical distributed system, it becomes relatively straightforward to:

In a sense, the organization of an ordinary class library can be extended to the server level. A full-fledged distributed protocol such as MADOCA can support this sort of architecture within a unified framework.

Spanning Multiple Network Segments

Whether it is worthwhile to use a unified communication protocol and routing scheme throughout an entire network is a separate question. Nevertheless, some projects must operate across multiple network segments.

In other words, the control system must span several network segments.

TANGO is also a distributed-control framework, but its design ties message-serving functions more closely to each device server. This gives coarser control over component granularity, and protocol-level routing across network segments is not supported in the same way as in MADOCA. Its design is less focused on controlling everything through one uniform routing protocol, whether or not that is actually needed.

The distinction may ultimately be one of where responsibilities are placed: should the protocol and framework handle more complexity, providing a unified way to manage systems of very different sizes? Or should the protocol and framework remain as compact as possible, with the operational system handling the remaining complexity?

Centralized and Distributed Architectures

A rough summary of the discussion so far:

Distributed Use of a Centralized Architecture

Deploying Multiple Centralized Systems

Systems such as BL774 do not use a full-fledged distributed protocol. Instead, they focus on operating experimental equipment within a compact network. They deliberately omit protocol-level server-to-server communication. Device servers communicate individually with clients rather than communicating directly with one another as an intrinsic framework function. From a protocol perspective, the basic architecture is therefore centralized.

In this configuration, users must connect to each device server (node) they need, with separate communication to each. The servers are almost completely independent, making the system very simple, even though device control remains separated into individual device-server components.

If the device servers offer transparent interfaces, user-side Python code need not pay special attention to the distribution. It can work with them much as it would with ordinary Python classes.

Management

As the number of devices increases, managing device servers becomes essential in distributed control systems. Susanoo does not yet provide such management functionality, and may never provide it.

TANGO has a Starter device server for launching other device servers. Its database, which forms the foundation of the TANGO network by providing device-server name resolution and object information, also runs as a device server. Thus, there is a mechanism for starting and managing components in sequence through Starter device servers. I found the design remarkably consistent and thorough in its philosophy, and genuinely elegant and satisfying. Admittedly, the implementation and operations could feel somewhat cluttered. Still, being able to centralize startup-level control through a management tool was convenient in an experimental environment.

Of course, TANGO also works without these management facilities: one can start device servers individually and connect to them directly. In a small network, there is no need to launch device servers through the management system.

MADOCA has no direct equivalent of TANGO’s Starter device server. Because MADOCA was designed for large, complex network environments, it placed less emphasis on having device servers manage other device servers through a unified mechanism. Message servers and EMs (roughly analogous to device servers) could be controlled to some extent, but there was no Starter device server in the TANGO sense. Crucially, there was no mechanism for bootstrapping the message servers themselves. As I recall, EMs could be stopped but not started. Even where control was possible, it worked only when a query was made; otherwise the management traffic would have been enormous.

Designing around centralized, network-distributed management of a huge experimental system introduces various constraints. My recollection is imperfect, but the conclusion was that having device servers start other device servers did not fit the architecture. The discussion eventually came down to whether there was much point in creating a process merely to start another process. After all, even TANGO’s Starter device server must itself be bootstrapped by something, typically systemd.

Either way, if one is going to build a dedicated device-server management mechanism, it may be more sensible to use existing process-management facilities such as systemd or Windows Services, or simply a lightweight launcher such as the MADOCA launcher. In that sense, Astor is also a launcher for TANGO device servers.

Terms such as TANGO, MADOCA, and BL774 describe frameworks more than individual protocols. At the lower communication layer, TANGO uses CORBA + ZMQ, MADOCA2 uses ZMQ, and BL774 uses REST APIs. This section attempts to put these approaches into perspective. Some of my research and information may be dated or biased, but I hope to outline developments in protocols and frameworks at large synchrotron facilities—and touch on robotics as well.

For example, TANGO at the ESRF still uses CORBA for command/attribute request–reply communication and ZMQ for event distribution. Discussions about reducing dependence on CORBA continue, but TANGO v10 did not completely migrate to ZMQ.

ROS, widely used in European robotics, primarily used its own transport protocols such as TCPROS and UDPROS in ROS 1. ROS 2 abstracts the communication middleware and uses DDS as its standard foundation. DENSO WAVE’s ORiN2 can be extended to connect to ROS 1, but it does not support ROS 2 at all and is consequently becoming a relic of an earlier era.

Author