Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison-Wesley Signature Series (Fowler))

by Hohpe Gregor

When discussing Pipes and Filters architectures, we need to be cautious with the term filter. We later define two additional patterns, the Message Filter (237) and the Content Filter (342). While both of these are special cases of a generic filter, so are many other patterns in this pattern language. In other words, a pattern does not have to involve a filtering function (e.g., eliminating fields or messages) in order to be a filter in the sense of Pipes and Filters. We could have avoided this confusion by renaming the Pipes and Filters architectural style. However, we felt that Pipes and Filters is such an important and widely discussed concept that it would be even more confusing if we gave it a new name. We are trying to use the word filter cautiously throughout these patterns and trying to be clear about whether we are talking about a generic filter as in Pipes and Filters or a Message Filter (237)/Content Filter (342) that filters messages. If we thought there might still be confusion, we called the generic filter a component, which is a generic enough (and often abused enough) term that it should not get us into any trouble.

The basic mechanism underlying CSPs is the synchronization of two processes via input-output (I/O). I/O occurs when process A indicates that it is ready to output to process B, and process B states that it is ready to input from process A. If one of these happens without the other being true, the process is put on a wait queue until the other process is ready. CSPs are different from integration solutions in that they are not as loosely coupled, nor do the “pipes” provide any queuing mechanisms.

The base implementation simply prints the body of the received message and sends it to the output port. A more interesting filter would subclass the Processor class and override the ProcessMessage method to perform additional actions on the message—that is, transform the message content or route it to different output channels.

The Pipes and Filters (70) architectural style connects filters directly to each other with fixed pipes. This makes sense because many applications of the Pipes and Filters (70) pattern (e.g., [POSA]) are based on a large set of data items, each of which undergoes the same sequential processing steps.

a compiler will always execute the lexical analysis first, the syntactic analysis second, and the semantic analysis last.

You could create a separate Message Channel (60) for each message type (a concept explained in more detail later as a Datatype Channel [111]) and connect each channel to the required processing steps for that message type. However, this would require the message originators to be aware of the selection criteria for different processing steps in order to publish the message to the correct channel. It could also lead to an explosion of the number of Message Channels (60). Furthermore, the decision on which steps the message undergoes may not just depend on the origin of the message.

Message Channels (60) provide a very basic form of routing capabilities. An application publishes a Message (66) to a Message Channel (60) and has no further knowledge of that Message’s (66) destination. Therefore, the path of the Message (66) can change depending on which component subscribes to the Message Channel (60). However, this type of “routing” does not take into account the properties of individual messages. Once a component subscribes to a Message Channel (60), it will by default consume all messages from that channel regardless of the individual message’s specific properties. This behavior is similar to the use of the pipe symbol in UNIX to process text files. It allows you to compose processes into a Pipes and Filters (70) chain, but for the lifetime of the chain, all lines of text undergo the same steps.

Some messaging systems allow receivers to inspect message properties without removing the message from the channel so that it can decide whether to consume the message.

One advantage of Pipes and Filters (70) is the composability of the individual components. This property enables us to insert additional steps into the filter chain without having to change existing components. This opens up the option of decoupling two filters by inserting between them another filter that determines what step to execute next.

a special filter, a Message Router, which consumes a Message from one Message Channel and republishes it to a different Message Channel, depending on a set of conditions.

However, thanks to the Pipes and Filters (70) architecture, the components surrounding the Message Router are completely unaware of the existence of a Message Router. They simply consume messages off one channel and publish them to another. A defining property of the Message Router is that it does not modify the message contents; it concerns itself only with the destination of the message.

The key benefit of using a Message Router is that the decision criteria for the destination of a message are maintained in a single location. If new message types are defined, new processing components are added, or routing rules change, we need to change only the Message Router logic, while all other components remain unaffected. Also, since all messages pass through a single Message Router, incoming messages are guaranteed to be processed one by one in the correct order.

The Message Router component must have knowledge of all possible destination channels in order to send the message to the correct channel. If the list of possible destinations changes frequently, the Message Router can turn into a maintenance bottleneck. In those cases, it would be better to let the individual recipients decide which messages they are interested in. You can accomplish this by using a Publish-Subscribe Channel (106) and an array of Message Filters (237). We contrast these two alternatives by calling them predictive routing and reactive filtering

Loosely coupled systems can make it difficult to understand the “big picture” of the solution—the overall flow of messages through the system.

If everything is loosely coupled to everything else, it becomes impossible to understand in which direction messages actually flow. This can complicate testing, debugging, and maintenance. A number of tools can help alleviate this problem. First, we can use the Message History (551) to inspect messages at runtime and see which components they traversed. Alternatively, we can compile a list of all channels to which each component in the system subscribes or publishes. With this knowledge we can draw a graph of all possible message flows across components. Many EAI packages maintain channel subscription information in a central repository, making this type of static analysis easier.

Other Message Routers decide the message’s destination based on environment conditions. We call these routers context-based routers. Such routers are commonly used to perform load-balancing, test, or failover functionality. For example, if a processing component fails, the context-based router can reroute messages to another processing component and thus provide failover capability. Other routers split the flow of messages evenly across multiple channels to achieve parallel processing similar to a load balancer.

a Message Router can have additional built-in intelligence to route the messages as opposed to a simple round-robin implemented by the channel.

Many Message Routers are stateless—in other words, they look at only one message at a time to make the routing decision. Other routers take the content of previous messages into account when making a routing decision.

Most Message Routers contain hard-coded logic for the routing decision. However, some variants connect to a Control Bus (540) so that the middleware solution can change the decision criteria without having to make any code changes or interrupting the flow of messages.

The Dynamic Router (243) configures itself dynamically based on control messages from each potential recipient.

The application’s underlying data model usually drives the design of the physical database schema, an interface file format, or an application programming interface (API)—those entities with which an integration solution must interface. As a result, each application typically expects to receive messages that mimic the application’s internal data format.

In many cases, the integration solution needs to be able to communicate with external parties using the “official” data formats, even though the internal systems are based on proprietary formats.

Or, all data elements may be represented in XML format but use different tag names. To summarize the different kinds of translation, we can divide it into multiple layers (loosely borrowing from the OSI Reference Model).

The Transport layer at the bottom of the “stack” provides data transfer between the different systems. It is responsible for complete and reliable data transfer across different network segments and deals with lost data packets and other network errors. Some EAI vendors provide their own transport protocols (e.g., TIBCO RendezVous), whereas other integration technologies leverage TCP/IP protocols (e.g., SOAP). Translation between different transport layers can be provided by the Channel Adapter (127) pattern.

The Data Representation layer is also referred to as the syntax layer. This layer defines the representation of data that is transported.

The Data Types layer defines the application data types on which the application (domain) model is based. Here we deal with such decisions as whether date fields are represented as strings or as native date structures, whether dates carry a time-of-day component, which time zone they are based on, and so on.

The issues related to data types go beyond whether a field is of type string or integer. Consider sales data that is organized by region. The application used by one department may divide the country into four regions: West, Central, South, and East, identified by the letters W, C, S, and E. Another department may differentiate the Pacific region from the mountain region and distinguish the Northeast from the Southeast. Each region is identified by a two-digit number. What number does the letter E correspond to?

Many of the design trade-offs in integration are driven by the need to decouple components or applications. Decoupling is an essential tool to enable the management of change.

Message Channels (60) decouple applications from having to know each other’s location. A Message Router (78) can even decouple applications from having to agree on a common Message Channel (60). However, this form of decoupling achieves only limited independence between applications if they still depend on each other’s data formats. A Message Translator (85) can remove this additional level of dependency.

The beauty of a layered model is that you can treat one layer without worrying about the lower layers and therefore can focus on one level of abstraction at a time

An Envelope Wrapper (330) wraps the message data inside an envelope so that it can be transported across a messaging system.

A Content Enricher (336) augments the information inside a message, whereas the Content Filter (342) removes information.

The Claim Check (346) removes information but stores it f...

The Normalizer (352) can convert a number of different message formats in...

the Canonical Data Model (355) shows how to leverage multiple Message Translators to achi...

Being able to drag and drop transformations shortens the learning curve for developing a Message Translator dramatically. As so often though, visual tools can also become a liability when it comes to debugging or when you need to create complex solutions. Therefore, many tools let you switch back and forth between XSL and the visual representation.

Message Endpoint Applications are communicating by sending Messages (66) to each other via Message Channels (60).

The application and the messaging system are two separate sets of software. The application provides functionally for some type of user, whereas the messaging system manages messaging channels for transmitting messages for communication. Even if the messaging system is incorporated as a fundamental part of the application, it is still a separate, specialized provider of functionality, much like a database management system or a Web server.

A messaging system is a type of server, capable of taking requests and responding to them. Like a database accepting and retrieving data, a messaging server accepts and delivers messages. A messaging system is a messaging server.

The messaging server, like a database server, has a client API that the application can use to interact with the server. The API is not application-specific but is domain-specific, where the domain is messaging. The application must contain a set of code that connects and unites the messaging domain with the application to allow the application to perform messaging.

Message Endpoint code is custom to both the application and the messaging system’s client API. The rest of the application knows little about message formats, messaging channels, or any of the other details of communicating with other applications via messaging.

It is the endpoint that receives a message, extracts the contents, and gives them to the application in a meaningful way.

A Message Endpoint is a specialized Channel Adapter (127) one that has been custom developed for and integrated into its application.

A Message Endpoint should be designed as a Messaging Gateway (468) to encapsulate the messaging code and hide the message system from the rest of the application.

It can employ a Messaging Mapper (477) to transfer data between domain...

It can be structured as a Service Activator (532) to provide asynchronous message access to a synchronous service or function call. An endpoint can explicitly control transactions with th...

Sending messages is pretty easy, so many endpoint patterns concern different approaches for receiving messages. A message receiver can be a Polling Consum...

Multiple consumers can receive messages from the same channel either as Competing Consumers (502) or via a Message Dispatcher (508). A receiver can decide which messages to con...

the consumer can be an Idempotent Receiver (528) that correctly detects and handles duplicate messages.

When two applications want to exchange data, they do so by sending the data through a channel that connects the two. The application sending the data may not know which application will receive the data. However, by selecting a particular channel on which to send the data, the sender knows that the receiver will be one that is looking for that sort of data by looking for it on that channel. In this way, the applications that produce shared data have a way to communicate with those that wish to consume it.