Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison-Wesley Signature Series (Fowler))

by Hohpe Gregor

While a Correlation Identifier is used to match a reply with its request, the request may also have a Return Address (159) that states what channel to put the reply on. Whereas a correlation identifier is used to matching a reply message with its request, a Message Sequence’s (170) identifiers are used to specify a message’s position within a series of messages from the same sender.

My application needs to send a huge amount of data to another process, more than may fit in a single message. Or, my application has made a request whose reply contains too much data for a single message.

It’s nice to think that messages can be arbitrarily large, but there are practical limits to how much data a single message can hold. Some messaging implementations place an absolute limit on how big a message can be. Other implementations allow messages to get quite big, but large messages nevertheless hurt performance. Even if the messaging implementation allows large messages, the message producer or consumer may place a limit on the amount of data it can process at once. For example, many COBOL-based and mainframe-based systems will consume or produce data only in 32 Kb chunks.

So, how do you get around this? One approach is to limit your application so it never needs to transfer more data than the messaging layer can store in a single message. This is an arbitrary limit, though, which can prevent your application from producing the desired functionality. If the large amount of data is the result of a request, the caller could issue multiple requests, one for each result chunk, but that increases network traffic and assumes the caller even knows how many result chunks will be needed. The receiver could listen for data chunks until there are no more (but how does it know there aren’t any more?) and then try to figure out how to reassemble the chunks into the original, large piece of data, but that would be error-prone.

Whenever a large set of data needs to be broken into message-size chunks, send the data as a Message Sequence and mark each message with sequence identification fields.

Hey there, Transmission Control Protocol!

The three Message Sequence identification fields are as follows. 1. Sequence identifier—Distinguishes this cluster of messages from others. 2. Position identifier—Uniquely identifies and sequentially orders each message in a sequence. 3. Size or End indicator—Specifies the number of messages in the cluster or marks the last message in the cluster (whose position identifier then specifies the size of the cluster).

The sequences are typically designed so that each message in a sequence indicates the total size of the sequence—that is, the number of messages in that sequence. As an alternative, you can design the sequences so that each message indicates whether it is the final message in that sequence.

If a receiver gets some of the messages in a sequence but doesn’t get all of them, it should reroute the ones it did receive to the Invalid Message Channel (60).

The sender can send all of the messages in a sequence using a single transaction. This way, none of the messages will be delivered until all of them have been sent. Likewise, a receiver may wish to use a single transaction to receive the messages so that it does not truly consume any of the messages until it receives all of them. If any of the messages in the sequence are missing, the receiver can choose to roll back the transaction so that the messages can be consumed later. In many messaging system implementations, if a sequence of messages is sent in one transaction, the messages will be received in the order they are sent, which simplifies the receiver’s job of putting the data back together.

An alternative to Message Sequence is to use a Claim Check (346). Rather than transmitting a large document between two applications, if the applications both have access to a common database or file system, store the document and just transmit a key to the document in a single message.

How can a sender indicate when a message should be considered stale and thus shouldn’t be processed?

Often, a message’s contents have a practical limit for how long they’re useful. A caller issuing a stock quote request probably loses interest if it does not receive an answer within a minute or so. That means the request should not take more than a minute to transmit but also that the answer had better transmit back very quickly. A stock quote reply more than a minute or two old is probably too old and therefore irrelevant.

Time-to-live is a relative setting specifying how long after the message is sent it should expire.

This is a simple but powerful example, transmitting a request and transmitting back a reply. It consists of two main classes: • Requestor—The object that sends the request message and expects to receive the reply message. • Replier—The object that receives the request message and sends a reply message in response.

These two simple classes sending simple messages illustrate a number of the patterns: • Message Channel (60) and Point-to-Point Channel (103)—One channel for transmitting the requests, another for transmitting the replies. • Document Message (147)—The default type of message, used as both the request and the reply. • Request-Reply (154)—A pair of messages sent over a pair of channels, allowing the two applications to have a two-way conversation. • Return Address (159)—The channel to send the response on. • Correlation Identifier (163)—The ID of the request that caused this response. • Datatype Channel (111)—All of the messages on each channel should be of the same type. • Invalid Message Channel (115)—What happens to messages that aren’t of the right type. The example code also demonstrates a couple of patterns from Chapter 10, “Messaging Endpoints”: • Polling Consumer (494)—How the requestor consumes reply messages. • Event-Driven Consumer (498)—How the replier consumes request messages.

An application needs only one connection to a messaging system, but each component in the application that wishes to send and receive messages independently needs its own session. Two threads cannot share a single session; they should each use a different session so that the sessions will work properly.

It uses the queue names to look up the queues, which are Destinations.

It creates a MessageProducer for sending messages on the request queue, a MessageConsumer for receiving messages from the reply queue, and another producer for moving messages to the invalid message queue.

A replier is what an application might use to receive a request and send a reply. The application provides its requestor a connection to the messaging system, as well as the JNDI names of the request and invalid message queues. (It does not need to specify the name of the reply queue because, as we’ll see, that will be provided by the message’s Return Address [159].) This is the information the requestor needs to initialize itself.

The reply message uses a Correlation Identifier (163) to specify which request this is a reply for.

The requestor implements a Polling Consumer (494) to receive replies, whereas the replier implements an Event-Driven Consumer (498) to receive requests.

The request and reply queues are Datatype Channels (111); when a consumer receives a message that is not of the right type, it reroutes the messa...

1. Requestor—A Message Endpoint (95) that sends a request message and waits to receive a reply message as a response. 2. Replier—A Message Endpoint (95) that waits to receive the request message; when it does, it responds by sending the reply message.

An application that wants to send requests and receive replies could use a requestor to do so. The application specifies the pathnames of two queues: the request queue and the reply queue. This is the information the requestor needs to initialize itself.

a Remote Procedure Invocation (50) only works when the source of the call, the target, and the network connecting them are all working properly. If a subject announces a change and a remote observer is not ready to process the notification or is disconnected from the network, the observer loses the notification.

Distribution also favors the push model over the pull model. As discussed earlier, push requires a single, one-way communication, whereas pull requires three. When the distribution is implemented via RPCs (Remote Procedure Calls), push requires one call (Update()), whereas pull requires at least two calls (Update() and GetState()). RPCs have more overhead than nondistributed method invocations, so the extra calls required by the push approach can quickly hurt performance.

A Publish-Subscribe Channel (106) implements the Observer pattern, making the pattern much easier to use among distributed applications.

ObserverGateway is another Messaging Gateway (468), this time between the observer (not shown) and the messaging system. The observer creates the gateway, then uses attach() to start the connection (which is analogous to calling the Attach(Observer) method in the Observer pattern). The gateway is an Event-Driven Consumer (498), so it implements the MessageListener interface, which requires the onMessage method. In this way, when an update is received, the gateway processes the message to get the new state and calls its own update(String) method, which then calls the corresponding message in the observer.

For distributed notification between applications, the publish-subscribe (e.g., messaging) approach has several advantages over the traditional, synchronous (e.g., RPC) approach of implementing Observer.

Simplifies notification.

Simplifies attach/detach.

Simplifies concurrent threading.

Simplifies remote access.

Increases reliability.

Another potential downside of the publish-subscribe approach is that the pull model is more complex than the push model. As discussed earlier, the pull model requires more back-and-forth discussion than the push model. When the discussion is among distributed applications, the extra communication can significantly hurt performance.

One channel, the get-state-request channel, goes from an observer to the subject; an observer sends the state request on that channel. The other channel, the get-state-reply channel, goes from the subject back to the observer; the subject sends the state reply on that channel. All of the observers can share the same request channel, but they will probably each need their own reply channel. Each observer needs to receive not just any response but the particular response for its specific request, and the easiest way to ensure this is for each observer to have its own reply channel.

use Correlation Identifiers [163] to figure out which reply goes to which observer, but a separate channel per observer is a lot easier to implement.)

Real enterprise applications are much more complex. An application can contain lots of subjects that need to announce changes. Each subject often contains several different pieces of state, called aspects, that can change independently. A single observer might be interested in several different aspects in several different subjects, where the subjects are not only multiple instances of the same class but may well be instances of different classes.

Also, the SASE (Self-Addresses Stamped Envelope) pattern describes a combination of the Observer and Command patterns whereby an observer specifies the command a subject should send it when a certain change occurs

let’s consider the implications for messaging, namely: How many channels will we need?

An enterprise may have several different applications responsible for storing a customer’s contact information, such as a mailing address.

When a customer’s address is updated in one of these applications, the application should notify other applications that may need this new information as well. Meanwhile, there may be several applications that need to know when an address changes, so they would like to register to receive notification.

The enterprise may also have applications that need to announce when they are out of a product and others that need to receive these notifications so that can reorder the product. This is just a different example of the last problem, and it is solved the same way by using a Publish-Subscribe Channel (106) to make out-of-product announcements.

applications interested in address changes are probably not the same ones interested in product updates, so if the messages use the same channel, an application will frequently receive notifications it’s not interested in. Thus, it makes sense to have two separate address change and product update channels.

applications can use Selective Consumers (515) to get only the messages of interest. If selective consumers prove to be complicated and a messaging system can easily support more channels, then perhaps separate channels would be better after all.

As with many issues in enterprise architecture and design, there are no simple answers and lots of trade-offs. With Publish-Subscribe Channel (106), as with any message channel, the goal is to help ensure that the observers receive only the notifications they need, without an explosion of separate channels and without taxing the typical observer with lots of threads running lots of consumers monitoring lots of channels.

The push model is simpler and often more efficient than the pull model, especially for distributed notification and with messaging. Yet, the pull model can also be implemented using messaging. In complex applications where lots of data can change, it may be tempting to create a channel for every different thing that can change, but it’s often more practical to use the same channel to transmit similar notifications going to the same observers. Even if your applications don’t need messaging for anything else, if they need to notify each other of changes, it may well be worth using Messaging (53) just so you can take advantage of Publish-Subscribe Channels (106).

The Content-Based Router (230) inspects the content of a message and routes it to another channel based on the content of the message. Using such a router enables the message producer to send messages to a single channel and leave it to the Content-Based Router (230) to route them to the proper destination.

A Message Filter (237) is a special form of a Content-Based Router (230). It examines the message content and passes the message to another channel only if the message content matches certain criteria.

The Composed Message Processor (294) splits a single message into multiple parts, whereas the Scatter-Gather (297) sends a copy of the same message to multiple recipients.