Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison-Wesley Signature Series (Fowler))

by Hohpe Gregor

Selective Consumers make a single channel act like multiple Datatype Channels (111). Different types of messages can have different selection values so that a consumer that is specialized for a particular type will only receive messages of that type. This can facilitate sending a large number of types using a small number of channels. This approach can also conserve channels in an enterprise that requires more channels than a messaging system can support.

Whereas a messaging system can ensure only authorized applications successfully receive messages from a channel, they usually do not authorize a consumer’s selection criteria, so a malicious consumer authorized to access the channel can gain access to unauthorized messages by changing its criteria. Separate datatype channels are needed to securely lock out applications.

none of the Selective Consumers on a channel match the message’s selector value, the message will be ignored as if the channel has no receivers. A similar problem in procedural programming is a case statement where none of the cases match the value being tested. Thus, a case statement can have a default case that matches values that aren’t matched by any other case. Applying this approach to messaging, it may seem tempting to create some sort of default consumer for messages that otherwise have no matching consumer. Yet, such a default consumer will not work as desired because it would require an expression that matches all selector values, so it would compete with all other consumers. Instead, to implement a default consumer, use a Message Dispatcher (508) that implements a case statement with a default option for unhandled cases that uses the default consumer.

A Durable Subscriber is simply a subscriber on a Publish-Subscribe Channel (106). However, when the subscriber disconnects from the messaging system, it becomes inactive and the messaging system will save any messages published on its channel until it becomes active again. Meanwhile, other subscribers to the same channel may not be durable; they’re nondurable subscribers.

One challenge with durable subscriptions is differentiating between an old subscriber that is reconnecting and a completely new subscriber.

Once an application has a durable subscription on a topic, it will have the opportunity to receive all messages published to that topic, even if the subscriber closes its connection (or if it crashes and the messaging system closes the subscriber’s connection for it). To stop the messaging system from queuing messages for this inactive subscriber, the application must explicitly unsubscribe its durable subscription.

The channel patterns in Chapter 3, “Messaging Systems,” discuss how to make messaging channels reliable by using Guaranteed Delivery (122). However, even some reliable messaging implementations can produce duplicate messages. In other scenarios, Guaranteed Delivery (122) may not be available because the communication relies on inherently unreliable protocols.

In order to detect and eliminate duplicate messages based on the message identifier, the message recipient has to keep a list of already received message identifiers. One of the key design decisions is how long to keep this history of messages and whether to persist the history to permanent storage such as disk. This decision depends primarily on the contract between the sender and the receiver. In the simplest case, the sender sends one message at a time, awaiting the receiver’s acknowledgment after every message.

Eliminating duplicate messages is another example where we can learn quite a bit by having a closer look at the low-level TCP/IP protocol. When IP network packets are routed across the network, duplicate packets can be generated. The TCP/IP protocol ensures elimination of duplicate packets by attaching a unique identifier to each packet. Sender and receiver negotiate a “window size” that the recipient allocates in order to detect duplicates. For a thorough discussion of how TCP/IP implements this mechanism, see [Stevens].

In some cases, it may be tempting to use a business key as the message identifier and let the persistence layer handle the de-duping. For example, let’s assume that an application persists incoming orders into a database. If each order contains a unique order number, and we configure the database to use a unique key on the order number field, the insert operation into the database would fail if a duplicate order message is received. This solution appears elegant because we delegated the checking of duplicates to the database systems, which is very efficient at detecting duplicate keys. But we have to be cautious because we associated dual semantics to a single field. Specifically, we tied infrastructure-related semantics (a duplicate message) to a business field (order number).

Imagine that the business requirements change so that customers can amend existing orders by sending another message with the same order number (this is quite common). We would now have to make changes to our message structure, since we tied the unique message identifier to a business field. ...

An alternative approach to achieve idempotency is to define the semantics of a message such that resending the message does not impact the system. For example, rather than defining a message as “Add $10 to account 12345,” we could change the message to “Set the balance of account 12345 to $110.” Both messages achieve the same result if the current account balance is $100. The second message is idempotent because receiving it twice will not have any effect.

While developing a messaging solution is no easy task, operating such a solution in production is equally challenging: A message-based integration solution may produce, route, and transform thousands or even millions of messages in a day. We have to deal with exceptions, performance bottlenecks, and changes in the participating systems. To make things ever more challenging, components are distributed across many platforms and machines that can reside at multiple locations.

the architectural benefits of loose coupling actually make testing and debugging a system harder.

“architect’s dream, developer’s nightmare” symptom: Architectural principles of loose coupling and indirection reduce the assumptions systems make about each other and therefore provide flexibility. However, testing a system where a message producer is not aware of who the consumers of a message are can be challenging. Add to that the asynchronous and temporal aspects of messaging and things get even more complicated.

Naturally, enterprise integration systems are distributed. In fact, one of the defining qualities of an enterprise messaging system is to enable communication between disparate systems. Messaging systems allow information to be routed and transformed so that data can be exchanged between these systems.

Besides just knowing whether a system or a component is up and running, you also need to monitor the dynamic behavior of the system. What is the message throughput? Are there any unusual delays? Are channels filling up? Some of this information requires tracking of message travel times between components or through components. This requires the collection and combination of information from more than one machine.

Often, you need to make adjustments or change configuration settings while the system is running. For example, you may need to turn logging features on or off while the system is running. Many applications use property files and error logs to read configuration information and report error conditions.

Likewise, some control messages are of a lower priority than application messages. If we have components publish periodic status messages, delaying or losing an “I am alive” control message may be a lot less troublesome than delaying or losing the “Order for $1 million” message.

Each component in the system is now connected to two messaging subsystems: 1. The Application Message Flow 2. The Control Bus The application message flow transports all application-related messages. The components subscribe and publish to these channels just as they would in an unmanaged scenario. In addition, each component also sends and receives messages from the channels that make up the Control Bus. These channels connect to a central management component. The Control Bus is well suited to carry the following types of messages: 1. Configuration—Each component involved in the message flow should have configurable parameters that can be changed as required. These parameters include channel addresses, message data formats, timeouts, and so on. Components use the Control Bus rather than property files to retrieve this information from a central repository, allowing a central point of configuration and the reconfiguration of the integration solution at runtime. For example, the routing table inside a Content-Based Router (230) may need to be updated dynamically based on system conditions, such as overload or component failure. 2. Heartbeat—Each component may send a periodic heartbeat message on the Control Bus at specified intervals so that a central console application can verify that the component is functioning properly. This heartbeat may also include metrics about the component, such as number of messages processed and the amount of available memory on the machine. 3...

Many of the functions that a Control Bus supports resemble traditional network management functions that are used to monitor and maintain any networked solution. A Control Bus allows us to implement equivalent management functions at the messaging system level—essentially elevating them from the low-level IP network level to the richer messaging level. Providing management functionality is as vital to the successful operation of a messaging infrastructure as it is for a network infrastructure.

When we design message processing components, we architect the core processor around three interfaces (see figure). The inbound data interface receives incoming messages from the message channel. The outbound data interface sends processed messages to the outbound channel. The control interface sends and receives control messages from and to the Control Bus.

We can compare these validations to assert statements in source code that are executed in the debug configuration but not in the release configuration of the executable.

Likewise, during troubleshooting, it may be useful to route messages through additional steps for logging or monitoring purposes. Being able to turn these logging steps on and off allows us to maximize message throughput under normal circumstances.

Many messaging systems provide a peek method that allows a component to inspect messages inside a Point-to-Point Channel (103) without consuming the message. This approach has one important limitation though: Once the intended consumer consumes the message, the peek method can no longer see the message. Therefore, this approach does not allow us to analyze messages after they have been consumed.

The Wire Tap (also known as tee) is a fixed Recipient List (249) with two output channels. It consumes messages off the input channel and publishes the unmodified message to both output channels. To insert the Wire Tap into a channel, you need to create an additional channel and change the destination receiver to consume the second channel. Because the analysis is performed by a separate component, we can insert a generic Wire Tap into any channel without any danger of unintentionally modifying the primary channel behavior. This improves reuse and reduces the risk of undesirable side effects when instrumenting an existing solution.

we can combine multiple Wire Taps to send copies of messages to a central component for analysis.

One of key benefits of a message-based system is the loose coupling between participants; the message sender and recipient make no (or few) assumptions about each other’s identity. If a message recipient retrieves a message from a message channel, it generally does not know nor care which application put the message on the channel. The message is by definition self-contained and is not associated with a specific sender. This is one of the architectural strengths of message-based systems.

The Message History is most useful if a series of messages flows through a number of different filters that together perform a specific business function or process. If it is important to manage the path that a message takes, a Process Manager (312) can be useful. The Process Manager (312) creates one process instance for each incoming trigger message. The flow of the message through various components is now managed centrally, alleviating the need to tag each message with its history.

Assume we implement a system that propagates address changes to multiple systems via Publish-Subscribe Channels (106). Each address change is broadcast to all interested systems so that they may update their records. This approach is very flexible toward the addition of new systems—the new system will automatically receive the broadcast message without requiring any changes to the existing messaging system.

Each change to the database fields causes a message to be triggered to notify all systems of the change. By nature of the publish-subscribe paradigm, all systems subscribing to the address-changed channel will receive the event.

How can we report against message information without disturbing the loosely coupled and transient nature of a messaging system?

The very properties that make messaging powerful can also make it difficult to manage. Asynchronous messaging guarantees delivery but does not guarantee when the message will be delivered. For many practical applications, though, the response time of a system may be critical. Also, while asynchronous messaging treats each message individually, information that spans multiple messages—for example, the number of messages passing through the system within a certain time interval—can be very useful.

A pair of Wire Taps (547) can be used to track messages that flow through a component, but this approach assumes that the component publishes messages to a fixed output channel. However, many service-style components publish reply messages to the channel specified by the Return Address (159) included in the request message.

Use a Smart Proxy to store the Return Address supplied by the original requestor and replace it with the address of the Smart Proxy. When the service sends the reply message, route it to the original Return Address.

The Control Bus (540) queue is connected to a simple file writer that writes each incoming message to a file.

This class configures an inbound channel and starts an Event-Driven Consumer (498) to receive messages.

Also, we may not want multiple services to load balance. For example, we may have a volume agreement with the primary service provider that gives us substantial discounts if we meet certain usage quotas. Splitting the traffic across two providers will likely cost us more. Alternatively, we may be using a low-cost provider as our primary service provider and want to switch over to a premium provider only when the low-cost provider fails. (For an excellent discussion of architectural decisions driven by licensing considerations, see [Hohmann].)

Connecting the individual components inside the management console through delegates and events results in a loosely coupled architecture. This architecture allows us to reuse the individual components and recompose them into different constellations similar to the Pipes and Filters (70) architectural style introduced at the beginning of the book. Essentially, passing messages arriving on the Control Bus (540) by using delegates resembles creating an application-internal Publish-Subscribe Channel (106). Because the control bus events arrive on a Point-to-Point Channel (103), we have to use a single consumer, which then publishes the event to any interested subscriber inside the application.

First, market data comes into the system. Market data is data regarding the price and other properties of the bond, representing what people are willing to buy and sell the bond for on the free market. The market data is immediately sent to the analytics engine that alters the data. Analytics refers to mathematical functions for financial applications that alter the prices and other attributes of bonds. These are generic functions that use input variables to tailor the results of the function to a particular bond. The client application that will run on each trader desktop will configure the analytics engine on a per-trader basis, controlling the specifics of the analytics for each bond the trader is pricing. Once the analytics are applied to the market data, the modified data is sent out to various trading venues where traders from other firms can buy or sell the bonds.

Therefore, all communication between the gateways and the clients will be accomplished through messaging.

This is effectively creating a Message Bus (137) or an architecture that will make it possible for future systems to integrate with the current system with little or no changes to the messaging infrastructure. This way, the business functionality of the application can be easily used by other applications

A pair of channel adapters that connects two messaging systems is a Messaging Bridge

A key to working with patterns is knowing not only when to use which pattern, but also knowing how to most effectively use it. Each pattern implementation has to take into account specifics of the technology platform as well as other design criteria.

We could create a system that publishes on a few channels, and subscribers could listen only for prices they are interested in. This would require subscribers to use a Message Filter (237) or Selective Consumer (515) to filter the entire data flow for interesting bond prices, deciding whether each message should be processed as it is received. Given that the market data is published on bond-dedicated channels, subscribers can register for updates on a series of bonds. This effectively allows subscribers to “filter” by selectively subscribing to channels and only receiving updates of interest rather than deciding after the message is received. It is important to note that using multiple channels to avoid filtering is a nonstandard use of messaging channels.

Problem Solving with Patterns Patterns are tools, and collections of patterns are toolboxes. They help solve problems. Some think that patterns are only useful during design. Following the toolbox analogy, this is like saying that tools are only useful when you build a house, not when you fix it. The fact is that patterns are a useful tool throughout a project when applied well.

The other pattern of interest is the Aggregator (268). The Aggregator (268) is used to manage the reconciliation of multiple, related messages into a single message, potentially reducing the message flow.

The Message Dispatcher (508) describes an approach whereby you add several performers to a pool. Each performer can run its own execution thread. One main consumer listens to the Message Channel (60) and delegates the message on to an unoccupied performer in the pool, then immediately returns to listening on the channel. This achieves the parallel processing benefit of Competing Consumers (502) but works on Publish-Subscribe Channels (106).

We encountered a performance problem based on a design flaw that prevented the client from processing messages in parallel. Applying patterns greatly reduced the problem but did not completely fix it, because the real problem was the client becoming a bottleneck. This couldn’t be fixed with a thousand patterns. We later addressed this problem by refactoring the message flow architecture to route messages directly from the pricing gateway to the contribution gateway. So, patterns can help design and maintain a system, but they don’t necessarily make up for poor upfront design.

As data flows across system and domain boundaries through messaging conduits, and as developers and architects become more proficient in the patterns that govern messaging systems, new standards and products will emerge to extend the tactical reach of those patterns. Over time, patterns tend to strengthen but otherwise change little, if at all; but their implementation strategies often evolve rapidly to allow developers to apply them to much broader scales of sophistication.