Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison-Wesley Signature Series (Fowler))

by Hohpe Gregor

Learning the technology, however, is just the first step—the real goal is to learn how to effectively apply the technology. The nice thing about platform technologies is that they constrain you to performing certain tasks. But, as far as the technology is concerned, you can do whatever you want and quite often get into trouble if you don’t do things appropriately.

“Patterns seem to be the sweet spot of design.”

Not only is messaging key to integration, but it will most likely be the predominant focus in Web services for years to come. There is so much noise today in the Web services world, it’s a delicate and complex endeavor just to identify the specifications and technologies to focus on. The goal remains the same, however—software helps you solve a problem.

Integration is important because applications cannot live isolated from each other. We need techniques that allow us to take applications that were never designed to interoperate and break down the stovepipes so we can gain a greater benefit than the individual applications can offer us.

Various technologies have been around that promise to solve the integration puzzle. We all concluded that messaging is the technology that carries the greatest promise. The challenge we faced was to convey how to do messaging effectively. The biggest challenge in this is that messages are by their nature asynchronous, and there are significant differences in the design approaches that you use in an asynchronous world.

Enterprise integration goes beyond creating a single application with a distributed n-tier architecture, which enables a single application to be distributed across several computers. Whereas one tier in a distributed application cannot run by itself, integrated applications are independent programs that can each run by themselves, yet that function by coordinating with each other in a loosely coupled way.

Messaging enables multiple applications to exchange data or commands across the network using a “send and forget” approach. This allows the caller to send the information and immediately go on to other work while the information is transmitted by the messaging system. Optionally, the caller can later be notified of the result through a callback. Asynchronous calls and callbacks can make a design more complex than a synchronous approach, but an asynchronous call can be retried until it succeeds, which makes the communication much more reliable.

The advantages and limitations of asynchronous messaging as compared to other integration techniques. • How to determine the message channels your applications will need, how to control whether multiple consumers can receive the same message, and how to handle invalid messages. • When to send a message, what it should contain, and how to use special message properties. • How to route a message to its ultimate destination even when the sender does not know where that is. • How to convert messages when the sender and receiver do not agree on a common format. • How to design the code that connects an application to the messaging system. • How to manage and monitor a messaging system once it’s in use as part of the enterprise.

security, complex data mapping, workflow, rule engines, scalability and robustness, and distributed transaction processing

Patterns are a proven way to capture experts’ knowledge in fields where there are no simple “one size fits all” answers, such as application architecture, object-oriented design, or integration solutions based on asynchronous messaging architectures.

Each pattern poses a specific design problem, discusses the considerations surrounding the problem, and presents an elegant solution that balances the various forces or drivers. In most cases, the solution is not the first approach that comes to mind, but one that has evolved through actual use over time. As a result, each pattern incorporates the experience base that senior integration developers and architects have gained by repeatedly building solutions and learning from their mistakes.

the pattern names give you a common vocabulary to efficiently discuss integration design alternatives

Interesting applications rarely live in isolation. Whether your sales application must interface with your inventory application, your procurement application must connect to an auction site, or your PDA’s calendar must synchronize with the corporate calendar server, it seems that any application can be made better by integrating it with other applications.

Networks are unreliable. Integration solutions have to transport data from one computer to another across networks. Compared to a process running on a single computer, distributed computing has to be prepared to deal with a much larger set of possible problems. Often, two systems to be integrated are separated by continents, and data between them has to travel through phone lines, LAN segments, routers, switches, public networks, and satellite links. Each step can cause delays or interruptions.

Networks are slow. Sending data across a network is multiple orders of magnitude slower than making a local method call. Designing a widely distributed solution the same way you would approach a single application could have disastrous performance implications.

File Transfer (43)—One application writes a file that another later reads. The applications need to agree on the filename and location, the format of the file, the timing of when it will be written and read, and who will delete the file.

Shared Database (47)—Multiple applications share the same database schema, located in a single physical database. Because there is no duplicate data storage, no data has to be transferred from one application to the other.

Remote Procedure Invocation (50)—One application exposes some of its functionality so that it can be accessed remotely by other applications as a remote procedure. The communication occurs in real time and synchronously.

Messaging (53)—One application publishes a message to a common message channel. Other applications can read the message from the channel at a later time. The applications must agree on a channel as well as on the format of the message. The communication is asynchronous.

While all four approaches solve essentially the same problem, each style has its distinct advantages and disadvantages. In fact, applications may integrate using multiple styles such that each point of integration takes advantage of the style that suits it best.

Messaging is a technology that enables high-speed, asynchronous, program-to-program communication with reliable delivery.

Channels, also known as queues, are logical pathways that connect the programs and convey messages. A channel behaves like a collection or array of messages, but one that is magically shared across multiple computers and can be used concurrently by multiple applications. A sender or producer is a program that sends a message by writing the message to a channel. A receiver or consumer is a program that receives a message by reading (and deleting) it from a channel.

A messaging system manages messaging the way a database system manages data persistence. Just as an administrator must populate the database with the schema for an application’s data, an administrator must configure the messaging system with the channels that define the paths of communication between the applications. The messaging system then coordinates and manages the sending and receiving of messages. The primary purpose of a database system is to make sure each data record is safely persisted, and likewise the main task of a messaging system is to move messages from the sender’s computer to the receiver’s computer in a reliable fashion.

A messaging system is needed to move messages from one computer to another because computers and the networks that connect them are inherently unreliable. Just because one application is ready to send data does not mean that the other application is ready to receive it. Even if both applications are ready, the network may not be working or may fail to transmit the data properly. A messaging system overcomes these limitations by repeatedly trying to transmit the message until it succeeds. Under ideal circumstances, the message is transmitted successfully on the first try, but circumstances are often not ideal.

two important messaging concepts: 1. Send and forget—In step 2, the sending application sends the message to the message channel. Once that send is complete, the sender can go on to other work while the messaging system transmits the message in the background. The sender can be confident that the receiver will eventually receive the message and does not have to wait until that happens. 2. Store and forward—In step 2, when the sending application sends the message to the message channel, the messaging system stores the message on the sender’s computer, either in memory or on disk. In step 3, the messaging system delivers the message by forwarding it from the sender’s computer to the receiver’s computer, and then stores the message once again on the receiver’s computer. This store-and-forward process may be repeated many times as the message is moved from one computer to another until it reaches the receiver’s computer.

The create, send, receive, and process steps may seem like unnecessary overhead. Why not simply deliver the data to the receiver? By wrapping the data as a message and storing it in the messaging system, the applications delegate to the messaging system the responsibility of delivering the data. Because the data is wrapped as an atomic message, delivery can be retried until it succeeds, and the receiver can be assured of reliably receiving exactly one copy of the data.

messaging is more immediate than File Transfer (43), better encapsulated than Shared Database (47), and more reliable than Remote Procedure Invocation (50).

Specific benefits of messaging include:

Remote Communication. Messaging enables separate applications to communicate and transfer data. Two objects that reside in the same process can simply share the same data in memory. Sending data to another computer is a lot more complicated and requires data to be copied from one computer to another. This means that objects have to be “serializable”—that is, they can be converted into a simple byte stream that can be sent across the network. Messaging takes care of this conversion so that the applications do not have to worry about it.

Platform/Language Integration. When connecting multiple computer systems via remote communication, these systems likely use different languages, technologies, and platforms, perhaps because they were developed over time by independent teams. Integrating such divergent applications can require a neutral zone of middleware to negotiate between the applications, often using the lowest common denominator—such as flat data files with obscure formats. In these circumstances, a messaging system can be a universal translator between the applications that works with each one’s language and platform on its own...

Asynchronous Communication. Messaging enables a send-and-forget approach to communication. The sender does not have to wait for the receiver to receive and process the message; it does not even have to wait for the messaging system to deliver the message. The sender only needs to wait for the message to be sent, that is, for the message to be successfully stored in the channel by the messaging system. Once the mes...

Variable Timing. With synchronous communication, the caller must wait for the receiver to finish processing the call before the caller can receive the result and continue. In this way, the caller can make calls only as fast as the receiver can perform them. Asynchronous communication allows the sender to submit requests to the receiver at its own pace and the receiver to consume the requests at its own different pace. This allows both applications to run at ma...

Throttling. A problem with remote procedure calls (RPCs) is that too many of them on a single receiver at the same time can overload the receiver. This can cause performance degradation and even cause the receiver to crash. Because the messaging system queues up requests until the receiver is ready to process them, the receiver can control the rate at which it consumes requests so as not to become overloaded by too many simultaneous requests. The callers are unaffe...

Reliable Communication. Messaging provides reliable delivery that an RPC cannot. The reason messaging is more reliable than RPC is that messaging uses a store-and-forward approach to transmitting messages. The data is packaged as messages, which are atomic, independent units. When the sender sends a message, the messaging system stores the message. It then delivers the message by forwarding it to the receiver’s computer, where it is stored again. Storing the message on the sender’s computer and the receiver’s computer is assumed to be reliable. (To make it even more reliable, the messages can be stored to disk instead of memory; see Guaranteed Delivery [122].) What is unreliable is forwarding (moving) the message from the sender’s computer to the receiver’s computer, because the receiver or the network may not be running properly. The messaging system...

Disconnected Operation. Some applications are specifically designed to run disconnected from the network, yet to synchronize with servers when a network connection is available. Such applications are deployed on platforms like laptop computers and PDAs. Messaging is ideal for enabling these applications to synchronize—data to be synchronize...

Mediation. The messaging system acts as a mediator—as in the Mediator pattern [GoF]—between all of the programs that can send and receive messages. An application can use it as a directory of other applications or services available to integrate with. If an application becomes disconnected from the others, it need only reconnect to the messaging system, not to all of the other messaging applications. The messaging system can employ redundant resources to provide...

Thread Management. Asynchronous communication means that one application does not have to block while waiting for another application to perform a task, unless it wants to. Rather than blocking to wait for a reply, the caller can use a callback that will alert the caller when the reply arrives. (See the Request-Reply [154] pattern.) A large number of blocked threads or threads blocked for a long time can leave the application with too few available threads to perform real work. Also, if an application with a dynamic number of blocked threads crashes, reestablishing those threads will be difficult when the application restarts and recovers its former state. With callbacks, the only threads that block are a small, known...

Asynchronous messaging is not the panacea of integration. It resolves many of the challenges of integrating disparate systems in an elegant way, but it also introduces new challenges.

Complex programming model. Asynchronous messaging requires developers to work with an event-driven programming model. Application logic can no longer be coded in a single method that invokes other methods, but instead the logic is now split up into a number of event handlers that respond to incoming messages. Such a system is more complex and harder to develop and debug. For example, the equivalent of a simple method call can require a request message and a request channel, a reply message and a reply channel, a correlation identifier and an invalid message queue (as described in Request-Reply [154]).

Sequence issues. Message channels guarantee message delivery, but they do not guarantee when the message will be delivered. This can cause messages that are sent in sequence to get out of sequence. In situations where messages depend on each other, special care has to be taken to reestablish the message sequence (see Resequencer [283]).

For such a bulk data replication step, ETL (extract, transform, and load) tools are much more efficient than messaging. Messaging is best suited to keeping the systems in sync after the initial data replication.

Vendor lock-in. Many messaging system implementations rely on proprietary protocols. Even common messaging specifications such as JMS do not control the physical implementation of the solution. As a result, different messaging systems usually do not connect to one another. This can leave you with a whole new integration challenge: integrating multiple integration solutions! (See the Messaging Bridge [133] pattern.)

In contrast, integrated applications are independent applications that can each run by themselves but that coordinate with each other in a loosely coupled way.

This enables each application to focus on one comprehensive set of functionality and yet delegate to other applications for related functionality.

Many messaging vendors have incorporated some of this book’s patterns as features of their products, which simplifies applying the patterns and accelerates solution development.

Christopher Alexander pioneered the concept of patterns and pattern languages in his books A Pattern Language and A Timeless Way of Building. Each pattern represents a decision that must be made and the considerations that go into that decision. A pattern language is a web of related patterns where each pattern leads to others, guiding you through the decision-making process. This approach is a powerful technique for documenting an expert’s knowledge so that it can be readily understood and applied by others.

A pattern language teaches you how to solve a limitless variety of problems within a bounded problem space. Because the overall problem that is being solved is different every time, the path through the patterns and how they’re applied is also unique.

It is not enough to simply say, “When you face this problem, apply this solution.” For you to truly learn from a pattern, the pattern has to document why the problem is difficult to solve, consider possible solutions that in fact don’t work well, and explain why the solution offered is the best available. Likewise, the patterns need to connect to each other so as to walk you from one problem to the next. In this way, the pattern form can be used to teach not just what solutions to apply but also how to solve problems the authors could not have predicted.

Patterns should be prescriptive, meaning that they should tell you what to do. They don’t just describe a problem, and they don’t just describe how to solve it—they tell you what to do to solve it.

The point of the patterns and the pattern language is to help you make decisions that lead to a good solution for your specific problem, even if the authors didn’t have that specific problem in mind and even if you don’t have the knowledge and experience to develop that solution on your own.