Ohara Lab

Fundamentals of Communication Between Distributed Nodes

Rather than treating server–client communication and server–server communication as separate subjects, it is easier to understand both under the general heading of inter-node communication, using a common model of communication between nodes. At a sufficiently low level, communication is ultimately just socket communication. At that level, the side that binds and the side that connects form a server–client relationship. Put differently, every connection looks like a server–client relationship when viewed locally. Classifying socket patterns makes the overall communication architecture easier to understand.

(*) I am not a specialist in information technology or control engineering, so there may be misunderstandings or errors here, including in the discussion of intended use cases.

Classification of Socket Patterns

With systems such as MADOCA/DARUMA and TANGO, one sometimes writes code close to the socket level. Starting with ZMQ can therefore make these ideas feel relatively natural. ZMQ is a lightweight, compact, high-performance communication library: a lightweight messaging middleware that adds message queues and routing capabilities on top of socket communication. It is both a useful entry point into modern communication systems and a library that can remain useful in more advanced applications. Learning communication patterns through ZMQ is consequently quite common.

Broadly speaking, socket communication patterns can be divided into REQ/REP, ROUTER/DEALER, and PUB/SUB. Web communication typically follows a REQ/REP-like pattern; ROUTER/DEALER is fundamental in many ZMQ-based designs; and modern systems such as MQTT and DDS make extensive use of PUB/SUB. What people call a “distributed protocol” often involves ROUTER/DEALER or PUB/SUB patterns.

Separate from the choice of communication pattern, it is important to distinguish synchronous from asynchronous operation. HTTP uses a request/response pattern, but HTTP itself does not determine whether an implementation is synchronous or asynchronous. A client can wait synchronously for a response, or the implementation can use asynchronous I/O and concurrent processing. Here, communication patterns and implementation-level synchrony are treated separately.

For example, in synchrotron facilities, MADOCA2 uses ZMQ as its communication foundation. TANGO currently uses CORBA for command/attribute request–reply communication and ZMQ for event distribution. TANGO’s client API also provides both synchronous and asynchronous request models.

Below, all communication patterns are described using ZMQ terminology.

One-to-One Communication: REQ / REP (The Minimal Pattern)

First, it is important to understand the basic pattern of two-way messaging. Familiar web communication such as HTTP (also used by BL774) corresponds, at the socket-pattern level, to a REQ (client) / REP (server) model.

The sequence is:

REQ (Client Side)

REP (Server Side)

An important point is that web communication (HTTP) has an asymmetric structure: the client initiates the connection and the server responds. Thus, the two nodes are not equal peers in a one-to-one exchange. In this sense, it is a typical server–client architecture.

Separating the side that listens (bind) from the side that initiates a connection (connect), and distinguishing that relationship from REQ/REP, helps prevent confusion between server roles and the direction of message exchange.

Once this distinction is understood, communication architectures become easier to follow. An interesting—and sometimes confusing—feature of ZMQ is its flexibility:

In other words, connection direction and data-flow direction are independent.

With this distinction in mind, consider server-to-server communication. HTTP is a request/response protocol; it is not, by itself, a distributed communication protocol designed to interconnect servers directly. Of course, a server or service can act as a client when communicating with another server. For example, an action received by a web server can trigger a service API that then accesses another server.

An even simpler approach is to run separate servers for multiple devices and have the client connect directly to each device server. This is the basic model used by BL774 and Susanoo.

TANGO’s device-server model also does not assume a multistage routing network of the kind used by MADOCA. Abstractly, its communication architecture can be understood as a client accessing several device servers directly—in other words, as a collection of REQ/REP-like exchanges.

Whether a collection of REQ/REP-based server–client connections should be called “distributed communication” is debatable, and some people would likely object to that description. Similarly, nginx can act as a reverse proxy that forwards web requests to other servers, but that alone is clearly not what is meant here by distributed communication.

Communication between devices can, of course, be achieved simply by exchanging REQ/REP roles; there is no need to modify the user’s control client every time. This makes the model reasonably practical. However, because the protocol is not fundamentally designed around autonomous inter-node communication, enabling device servers to communicate independently can complicate server design. In particular, the system must account for the possibility of intervening requests from a client while servers are communicating with one another.

Ultimately, the choice becomes: Should we make the protocol more sophisticated? The implementation? The operational procedures? Or should we instead deliberately restrict operations to keep them simple?

The key idea behind the BL774 approach is to avoid the high cost of developing and maintaining a sophisticated protocol, and to avoid a full-fledged distributed protocol that may be excessive for a beamline. At the same time, using mainly web technologies substantially reduces development and operating costs. The significance of the approach lies in reconsidering the development and operational structure itself. (This is my personal interpretation.)

The following discussion may seem like a step backward in time, but to understand the current situation properly, I will look at what happens when the protocol itself is designed for distributed communication.

Multiplexed One-to-One Communication: ROUTER / DEALER

REQ/REP is the most basic communication pattern. However, it imposes a fairly strict sequence: one request followed by one reply. As a result, it is not always well suited to tasks such as:

REQ/REP is easy to understand as a minimal communication model, but it lacks some capabilities needed for distributed inter-node communication. To relax these restrictions, ZMQ provides the ROUTER/DEALER pattern and corresponding socket types.

Like REQ/REP, ROUTER/DEALER can be used for back-and-forth messaging, but unlike REQ/REP it can:

Consequently, ROUTER/DEALER is often used for routing messages between nodes, rather than for simple client–server exchanges. This makes it central to many distributed communication architectures. Consider a concrete example with three connections from three nodes:

TCP connection A ──┐
TCP connection B ──┼── ROUTER socket
TCP connection C ──┘

This is a three-to-one connection topology. It does not mean that multiple messages are transmitted simultaneously through a single socket operation: messages are handled as a sequence, while the individual communication flows can proceed asynchronously.

(node A) ─ DEALER ─┐
(node B) ─ DEALER ─┼── ROUTER socket
(node C) ─ DEALER ─┘

A ROUTER socket maintains a routing ID for each connected DEALER peer. At the underlying connection level, each peer has its own connection to the ROUTER socket.

Physically, multiple peers connect to one port. Logically, however, attaching a routing-ID “envelope” to messages allows many one-to-one conversations to be carried over that shared endpoint: multiplexing.

As its name suggests, a ROUTER can route messages to other nodes, including other DEALER peers.

(node A) ─ DEALER ─┐           ┌─ DEALER ─ (node D)
(node B) ─ DEALER ─┼── ROUTER ─┼─ DEALER ─ (node E)
(node C) ─ DEALER ─┘           └─ DEALER ─ (node F)

This is a centralized use of DEALER/ROUTER, with one ROUTER at the center deciding which DEALER should receive each message.

By contrast, SPring-8’s MADOCA is designed to route messages across nodes through successive hops, without a single central ROUTER. Conceptually:

(node A) DEALER ─ (node B) ROUTER ─ (node C) ROUTER ─ (node E) DEALER

Each message server has routing functionality, and the server process determines the next destination as messages hop across the network. This is a distributed, multi-hop message-routing architecture. Combining ROUTER and DEALER sockets allows messages to traverse network boundaries.

Devices (DEALERs) connect to ROUTERs, and ROUTERs connect to one another. Messages can therefore hop across the network to reach devices on remote segments. These connections operate asynchronously.

In MADOCA, the device-side component is called an EM (Equipment Manager), corresponding roughly to the device-side DEALER in this ZMQ-based explanation. It is distinguished from the MS (Message Server), which performs the ROUTER role.

In TANGO terminology, the corresponding server functionality is associated with individual devices, and the whole component is called a device server. MADOCA does not traditionally use that term: in MADOCA, “server” generally means MS, while EM is not called a server. After working extensively with TANGO, however, I tend to call EM a device server too—it looks like a server process to someone learning the system.

A later MADOCA deployment approach moved closer to TANGO for convenience in beamline operations, pairing each EM with an MS. For better or worse, this configuration met with considerable opposition. Placing MS components independently permits finer-grained nodes and flexible placement of EM and MS responsibilities; that flexibility is characteristic of MADOCA’s approach to networked equipment control. Those most familiar with MADOCA were therefore often the strongest opponents of the paired configuration. This created a number of troublesome disagreements over design philosophy and operational practice.

One-to-Many Communication: PUB/SUB (Events)

REQ/REP and ROUTER/DEALER are fundamentally models for two-way message exchanges: send a message to someone and receive a response. Actual distributed systems also need a different kind of communication, for example:

In these situations, publishing information when it arises is more natural than asking someone for it. PUB/SUB is a common pattern for this purpose. One could stubbornly implement such behavior using ROUTER/DEALER alone, but that may make the architecture unnecessarily convoluted.

Broker-Based (MQTT)

A central broker distributes messages by topic. This is easy to manage, but the broker can become a single point of failure: if the center goes down, the system can stop functioning. It can also introduce additional latency.

Brokerless (ZMQ, DDS)

This approach can provide low latency without a central single point of failure. However, it does not inherently provide a mechanism for keeping track of where every participant is located; a separate discovery or management mechanism may be needed.

Author