Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison-Wesley Signature Series (Fowler))

by Hohpe Gregor

Both the shipping and the billing systems store addresses in a relational database, so we use a database Channel Adapter (127) to update the data in each system.

imagine an interface that exposes a separate method to change each address field. This approach would be efficient if the communication happens inside a single application—you update only those fields that changed. In an integration scenario, sending six or seven messages to update an address would be a significant overhead, plus we would have to deal with synchronizing the individual messages. Fine-grained interfaces also lead to tight coupling. If we change the address format by adding a new field, we have to define new message formats and change all other applications to send an additional message.

A dynamic Recipient List (249) is the combination of two Message Routing (78) patterns. A Recipient List (249) is a router that propagates a single message to a set of recipients. The main difference between the Recipient List (249) and a Publish-Subscribe Channel (106) is that the Recipient List (249) addresses each recipient specifically and therefore has tight control over who receives messages. A Dynamic Router (243) is a router whose routing algorithm can change based on control messages. These control messages can take the form of subscription preferences issued by the subscribers. A dynamic Recipient List (249) is the result of combining these two patterns.

Each recipient channel is then identified by an e-mail address. Likewise, if customers prefer to receive announcements via a Web services interface, each recipient channel is implemented by a SOAP request, and the channel address is the URI of the Web service. This example illustrates that the patterns we use to describe the solution design are independent of a specific transport technology.

Monitoring the correct execution of messages is a critical operations and support function. The Message Store (555) can provide us with some important business metrics, such as the average time to fulfill an order. However, we may need more detailed information for the successful operation of an integration solution.

the external service can process more than one request at a time, so we need to be able to match up request and reply messages.

since we treat the external service as a shared service inside our enterprise, we want to allow the service consumer to specify a Return Address (159), the channel where the service should send the reply message. Not knowing to which channel the reply is being sent can make it difficult to match request and reply messages.

We insert the Smart Proxy (558) between any service consumer and the external service. The Smart Proxy (558) intercepts each request to the service and replaces the Return Address (159) specified by the service consumer with a fixed reply channel. This causes the service to send all reply messages to the channel specified by the Smart Proxy (558). The proxy stores the original Return Address (159) so that it can forward the reply message to the channel originally specified by the consumer.

The Smart Proxy (558) also measures the time elapsed between request and reply messages from the external service. The Smart Proxy (558) publishes this data to the Control Bus (540). The Control Bus (540) connects to a management console that collects metrics from many different components.

Enterprise integration is the task of making disparate applications work together to produce a unified set of functionality. These applications can be custom developed in house or purchased from third-party vendors. They likely run on multiple computers, which may represent multiple platforms, and may be geographically dispersed. Some of the applications may be run outside of the enterprise by business partners or customers. Other applications might not have been designed with integration in mind and are difficult to change. These issues and others like them make application integration complicated.

Application coupling—Integrated applications should minimize their dependencies on each other so that each can evolve without causing problems to the others.

the interfaces for integrating applications should be specific enough to implement useful functionality but general enough to allow the implementation to change as needed.

Intrusiveness—When integrating an application into an enterprise, developers should strive to minimize both changes to the application and the amount of integration code needed. Yet, changes and new code are often necessary to provide good integration functionality, and the approaches with the least impact on the application may not provide the best integration into the enterprise.

Technology selection—Different integration techniques require varying amounts of specialized software and hardware. Such tools can be expensive, can lead to vendor lock-in, and can increase the learning curve for developers. On the other hand, creating an integration solution from scratch usually res...

Data format—Integrated applications must agree on the format of the data they exchange. Changing existing applications to use a unified data format may be difficult or impossible. Alternatively, an intermediate translator can unify applications that insist on different data formats. A related issue is data format evolution and extensibil...

Data timeliness—Integration should minimize the length of time between when one application decides to share some data and other applications have that data. This can be accomplished by exchanging data frequently and in small chunks. However, chunking a large set of data into small pieces may introduce inefficiencies. Latency in data sharing must be factored into the integration design. Ideally, receiver applications should be informed as soon as shared data is ready for consumption. The lo...

Data or functionality—Many integration solutions allow applications to share not only data but functionality as well, because sharing of functionality can provider better abstraction between the applications. Even though invoking functionality in a remote application may seem the same as invoking local functionality, it work...

Remote Communication—Computer processing is typically synchronous—that is, a procedure waits while its subprocedure executes. However, calling a remote subprocedure is much slower than a local one so that a procedure may not want to wait for the subprocedure to complete; instead, it may want to invoke the subprocedure asynchronously, that is, starting the subprocedure but continuing with its own processing simultaneously. Asynchronicity can make for a much more efficient solution, but such a solution is also more complex to design, develop, and debug.

Reliability—Remote connections are not only slow, but they are much less reliable than a local function call. When a procedure calls a subprocedure inside a single application, it’s a given that the subprocedure is available. This is not necessarily true when communicating remotely; the remote application may not even be running or the network may be temporarily unavailable. Reliable, asynchronous communication enables the source application to go on to other work, confident that the remote application will act sometime later.

The various approaches can be summed up in four main integration styles. File Transfer (43)—Have each application produce files of shared data for others to consume and consume files that others have produced. Shared Database (47)—Have the applications store the data they wish to share in a common database. Remote Procedure Invocation (50)—Have each application expose some of its procedures so that they can be invoked remotely, and have applications invoke those to initiate behavior and exchange data. Messaging (53)—Have each application connect to a common messaging system, and exchange data and invoke behavior using messages.

The four patterns share the same problem statement—the need to integrate applications—and very similar contexts. What differentiates them are the forces searching for a more elegant solution. Each pattern builds on the last, looking for a more sophisticated approach to address the shortcomings of its predecessors. Thus, the pattern order reflects an increasing order of sophistication, but also increasing complexity.

The patterns in the remainder of this book expand on the Messaging (53) integration style. We focus on messaging because we believe that it provides a good balance between the integration criteria but is also the most difficult style to work with. As a result, messaging is still the least well understood of the integration styles and a technology ripe with patterns that quickly explain how to use it best.

Develop each application as a large-scale object or component with encapsulated data. Provide an interface to allow other applications to interact with the running application.

Remote Procedure Invocation applies the principle of encapsulation to integrating applications. If an application needs some information that is owned by another application, it asks that application directly. If one application needs to modify the data of another, it does so by making a call to the other application. This allows each application to maintain the integrity of the data it owns.

Asynchronous messaging is fundamentally a pragmatic reaction to the problems of distributed systems. Sending a message does not require both systems to be up and ready at the same time. Furthermore, thinking about the communication in an asynchronous manner forces developers to recognize that working with a remote application is slower, which encourages design of components with high cohesion (lots of work locally) and low adhesion (selective work remotely).

An application that wishes to use messaging will implement Message Endpoints (95) to perform the actual sending and receiving.

Channels—Messaging applications transmit data through a Message Channel (60), a virtual pipe that connects a sender to a receiver. A newly installed messaging system typically doesn’t contain any channels; you must determine how your applications need to communicate and then create the channels to facilitate it.

Messages—A Message (66) is an atomic packet of data that can be transmitted on a channel. Thus, to transmit data, an application must break the data into one or more packets, wrap each packet as a message, and then send the message on a channel. Likewise, a receiver application receives a message and must extract the data from the message to process it. The message system will try repeatedly to deliver the message (e.g., transmit it from the sender to the receiver) until it succeeds.

Pipes and Filters—In the simplest case, the messaging system delivers a message directly from the sender’s computer to the receiver’s computer. However, certain actions often need to be performed on the message after it is sent by its original sender but before it is received by its final receiver. For example, the message may have to be validated or transformed because the receiver expects a message format different from the sender’s. The ...

Routing—In a large enterprise with numerous applications and channels to connect them, a message may have to go through several channels to reach its final destination. The route a message must follow may be so complex that the original sender does not know what channel will get the message to the final receiver. Instead, the original sender sends the message to a Message Router (78), an application component that takes the place of a filter in the Pipes and Filters (70) architecture. The router t...

Transformation—Various applications may not agree on the format for the same conceptual data; the sender formats the message one way, but the receiver expects it to be formatted another way. To reconcile this, the message must go through an intermediate filter, a Message...

Endpoints—Most applications do not have any built-in capability to interface with a messaging system. Rather, they must contain a layer of code that knows both how the application works and how the messaging system works, bridging the two so that they work together. This bridge code is a set of coordinated...

Rather, the application sending out the information knows what sort of information it is, and the applications that would like to receive information aren’t looking for just any information but for particular types of information they can use. So the messaging system isn’t a big bucket that applications throw information into and pull information out of. It’s a set of connections that enables applications to communicate by transmitting information in predetermined, predictable ways.

The application sending information doesn’t necessarily know what particular application will end up retrieving it, but it can be assured that the application that retrieves the information is interested in the information. This is because the messaging system has different Message Channels for different types of information the applications want to communicate. When an application sends information, it doesn’t randomly add the information to any channel available; it adds it to a channel whose specific purpose is to communicate that sort of information.

Likewise, an application that wants to receive particular information doesn’t pull info off some random channel; it selects what channel to get information from based on what type of information it wants.

Perhaps every Message Endpoint (95) has a direct connection to every other endpoint, or perhaps they’re all connected through a central hub.

Sometimes we say that an application listens on a channel to which another application talks. In the world of Web services, we generally talk about a requester and a provider. These terms usually imply that the requester sends a message to the provider and receives a response back. In the olden days we called these client and server (the terms are equivalent, but saying “client” and “server” is not cool).

An application that sends or receives messages may be called a client of the messaging system; a more specific term is endpoint or message endpoint.

Even if an enterprise system had unlimited memory and disk space, any messaging system implementation usually imposes some hard or practical limit to how many channels it can service consistently. So plan on creating new channels as your application needs them, but if it needs thousands of channels or needs to scale in ways that may require thousands of channels, you’ll need to choose a highly scalable messaging system implementation and test that scalability to make sure it meets your needs.

Many messaging systems support a hierarchical channel-naming scheme, which enables you to organize channels in a way that is similar to a file system with folders and subfolders. For example, MyCorp/Prod/OrderProcessing/NewOrders would indicate a channel that is used in a production application at MyCorp and contains new orders.

Applications that wish to use Messaging (53) but do not have access to a messaging client can still connect to the messaging system using Channel Adapters (127). A well-designed set of channels forms a Message Bus (137) that acts like a messaging API for a whole group of applications.

In many enterprise integration scenarios, a single event triggers a sequence of processing steps, each performing a specific function.

Use the Pipes and Filters architectural style to divide a larger processing task into a sequence of smaller, independent processing steps (filters) that are connected by channels (pipes).

Each filter exposes a very simple interface: It receives messages on the inbound pipe, processes the message, and publishes the results to the outbound pipe. The pipe connects one filter to the next, sending output messages from one filter to the next. Because all components use the same external interface, they can be composed into different solutions by connecting the components to different pipes. We can add new filters, omit existing ones, or rearrange them into a new sequence—all without having to change the filters themselves. The connection between filter and pipe is sometimes called a port. In the basic form, each filter component has one input port and one output port.

Pipes and Filters describes a fundamental architectural style for messaging systems: Individual processing steps (filters) are chained together through the messaging channels (pipes). Many patterns in this and the following sections, such as routing and transformation patterns, are based on this Pipes and Filters architectural style. This lets you easily combine individual patterns into larger solutions.

However, a Message Channel (60) provided by a messaging infrastructure can be quite heavyweight if all components can in fact reside on the same machine. Using a simple in-memory queue to implement the pipes would be much more efficient. Therefore, it is useful to design the components so that they communicate with an abstract pipe interface. The implementation of that interface can then be swapped out to use a Message Channel (60) or an alternative implementation such as an in-memory queue. The Messaging Gateway (468) describes how to design components for this flexibility.

Connecting components with asynchronous Message Channels (60) allows each unit in the chain to operate in its own thread or its own process. When a unit has completed processing one message, it can send the message to the output channel and immediately start processing another message. It does not have to wait for the subsequent components to read and process the message. This allows multiple messages to be processed concurrently as they pass through the individual stages.

We call such a configuration a processing pipeline because messages flow through the filters like liquid flows through a pipe. When compared to strictly sequential processing, a processing pipeline can significantly increase system throughput.

Parallelizing filters works best if each filter is stateless—that is, it returns to the previous state after a message has been processed. This means that we cannot easily run multiple parallel de-dupe components because the component maintains a history of all messages that it already received and is therefore not stateless.

[Kahn] described Kahn Process Networks in 1974 as a set of parallel processes that are connected by unbounded FIFO (First-In, First-Out) channels. [Garlan] contains a good chapter on different architectural styles, including Pipes and Filters. [Monroe] gives a detailed treatment of the relationships between architectural styles and design patterns.