Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison-Wesley Signature Series (Fowler))

by Hohpe Gregor

Use a specialized transformer, a Content Enricher, to access an external data source in order to augment a message with missing information.

The Content Enricher uses information inside the incoming message (e.g., key fields) to retrieve data from an external source. After the Content Enricher retrieves the required data from the resource, it appends the data to the message. The original information from the incoming message may be carried over into the resulting message or may no longer be needed, depending on the specific needs of the receiving application. The additional information injected by the Content Enricher has to be available somewhere in the system.

The Content Enricher may be able to compute the missing information. In this case, the algorithm incorporates the additional information.

The Content Enricher may be able to retrieve the additional data from the operating environment. The most common example is a time stamp. For example, the receiving system may require each message to carry a time stamp. If the sending system does not include this field, the Content Enricher can get the current time from the operating system and add it to the message.

The Content Enricher has to retrieve the missing data from another system. This data resource can take on a number of forms, including a database, a file, an LDAP directory, an application, or a user who manually enters missing data.

the communication between the Content Enricher and the resource can occur via Messaging (53) or via any other communication mechanism

the interaction between the Content Enricher and the data source is by definition synchronous (the Content Enricher cannot send the enriched message until the data source returns the requested data), a synchronous protocol (e.g., HTTP or an ODBC connection to a database) may result in better performance than using asynchronous messaging. The Content Enricher and the data source are inherently tightly coupled, so achieving loose coupling through Message Channels (60) is not as important.

we often choose to pass simple references to objects rather than pass a complete object with all data elements. These references usually take the form of keys or unique IDs. When the message needs to be processed by a system, we need to retrieve the required data items based on the object references included in the original message. We use a Content Enricher to perform this task.

Use a Content Filter to remove unimportant data items from a message, leaving only important items.

the process engine now serves as the datastore, storing our message data.

In a business-to-business (B2B) integration scenario, it is quite common for an enterprise to receive messages from different business partners. These messages may have the same meaning but follow different formats, depending on the partners’ internal systems and preferences.

How can you minimize dependencies when integrating applications that use different data formats?

Independently developed applications tend to use different data formats because each format was designed with just that application in mind. When an application is designed to send messages to or receive messages from some unknown application, the application will naturally use the message format that is most convenient for it. Likewise, commercial adapters used to integrate packaged applications typically publish and consume messages in a data format that resembles the application’s internal data structure.

However, a solution consisting of six applications requires 30 (!) Message Translators (85) without a Canonical Data Model and only 12 Message Translators (85) when using a Canonical Data Model.

How do you make applications conform to the common format?

Change the applications’ internal data format. This may be possible in theory, but it is unlikely in a complex, real-life scenario.

Implement a Messaging Mapper (477) inside the application. Custom applications can use a mapper to generate the desired data format.

Use an external Message Translator (85). You can use an external Message Translator (85) to translate from the application-specific message format into the format specified by the Canonical Data Model.

Unfortunately, the S in SOAP is no longer valid. We have jokingly postulated that SOAP be renamed to Complex Remote Access Protocol—you figure out the acronym. Seriously, designing a robust Web services interface requires us to dive into a number of terminologies and associated design trade-offs.

SOAP was developed with messaging in mind, the vast majority of the Web service applications use HTTP as the transport protocol. HTTP is a natural choice because it is the most commonly used protocol on the Web and can sneak through firewalls. However, HTTP is the HyperText Transfer Protocol—it was designed to allow users with Web browsers to retrieve documents over the Internet, not for applications to communicate with each other. HTTP is inherently unreliable and designed for synchronous document retrieval—the client application uses the same connection for sending the request to the server and receiving the response.

A typical process diagram includes a series of integration tasks. These include control tasks such as forks, sync bars or decision points, signal tasks to send and receive messages, and execution tasks such as data translation, routing, and system integration.

Changing a metadata definition can easily break another part of your system. Without proper testing and development practices, message-based systems can easily become “add-only” systems, meaning metadata is only added to the system and never changed for fear of breaking something.

One of the big advantages of modeling processes using a visual process modeling tool is that the running “code” looks very similar to the UML activity diagram that we used to design the solution.

Visual workflows can be easy to view and understand, but just as in code, this relative simplicity can turn into a maze of confusion as implementations become more complex.

the Aggregator (268) for the replies doesn’t need to know the number of expected replies. The potential downside of this flexibility is that it can hide errors resulting from incorrect subject naming, developer error, or dropped messages.

The visual, GUI-driven implementation with TIBCO provides a corollary approach to the code-level implementations. While the sheer power and flexibility of code cannot be argued against, a more graphical environment can greatly simplify very complex tasks.

Configuration is a very large part of integration, and graphical environments work well for this. At times, it may seem that developers have to choose between writing code and using a vendor developm...

Process management tools can help hide the complexities of messaging, allowing developers to focus on integration tasks without worrying about what happens behind the scenes. However, ignorance of messaging infrastructure has led to the death of more than one project.

Polling provides the best throttling because if the server is busy, it won’t ask for more messages, so the messages will queue up. Consumers that are event-driven tend to process messages as fast as they arrive, which could overload the server; but each consumer can only process one message at a time, so limiting the number of consumers effectively throttles the consumption rate.

Whatever other patterns an endpoint implements, it should also be a Messaging Gateway (468). So, don’t think of what one pattern to use—think of the combinations. That’s the beauty of solving the problems with patterns.

Use a Messaging Gateway, a class that wraps messaging-specific method calls and exposes domain-specific methods to the application.

A blocking Messaging Gateway encapsulates the asynchronous nature of the messaging interaction, exposing a regular synchronous method to the application logic.

An event-driven Messaging Gateway exposes the asynchronous nature of the messaging layer to the application. When the application makes the domain-specific request to the Messaging Gateway, it provides a domain-specific callback for the reply. Control returns immediately to the application. When the reply message arrives, the Messaging Gateway processes it and then invokes the callback

One of the challenges of using an event-driven Messaging Gateway is that the Messaging Gateway requires the application to maintain state between the request method and the callback event (the call stack takes care of this in the blocking case). When the Messaging Gateway invokes the callback event into the application logic, the application must be able to correlate the reply with the request it made earlier so that it can continue processing the correct thread of execution.

It can be beneficial to create more than one layer of Messaging Gateways. The “lower-level” Messaging Gateway can simply abstract the syntax of the messaging system but maintain generic messaging semantics, for example, SendMessage. This Messaging Gateway can help shield the rest of the application when the enterprise changes messaging technologies, for example, from MSMQ to Web services.

We often see people write “abstraction layers” that wrap the messaging infrastructure API, effectively making the code that deals with messaging independent from the messaging API. Such a layer provides a level of indirection because it separates the messaging interface from the messaging implementation. Therefore, we can reuse the messaging-related code even if we have to switch to another vendor’s messaging layer.

It has been said that computer science is the area where every problem can be solved by adding just one more level of indirection, so does this rule hold true here? The additional indirection gives us the ability to compensate for changes in the canonical model inside the messaging layer without having to touch application code. It also allows us to simplify the mapping logic inside the application by leaving tedious field mappings and datatype changes (e.g., a numeric ZIP_Code field to an alphanumeric Postal_Code field) to the mapping tools of messaging layer that are optimized for this kind of work.

A messaging system, by necessity, uses transactional behavior internally. It may be useful for an external client to be able to control the scope of the transactions that impact its behavior.

To manage all of this, messaging systems internally use transactions to make sure a message gets added or doesn’t get added to the channel, and gets read or doesn’t get read from the channel. Messaging systems also have to employ transactions—preferably two-phase, distributed transactions—to copy a message from the sender’s computer to the receiver’s computer, such that at any given time, the message is “really” only on one computer or the other.

Transactions are often described as being ACID: atomic, consistent, isolated, and durable. Only transactions for Guaranteed Messaging (53) are durable, and a message by definition is atomic. But all messaging transactions have to be consistent and isolated.

Message/Database Coordination 1. What to do: Start a transaction, receive a message, update the database, and then commit. Or, update the database and send a message to report the update to others, and then commit. (This behavior is often implemented by a Channel Adapter [127].) 2. What this does: The message will not be removed unless the database is updated (or the database change will not stick if the message cannot be sent). 3. Transaction type: Since the messaging system and the database each has its own transaction manager, the transaction to coordinate them will be a distributed one.

Message/Workflow Coordination 1. What to do: Use a pair of Request-Reply (154) messages to perform a work item. Start a transaction, acquire the work item, send the request message, and then commit. Or, start another transaction, receive the reply message, complete or abort the work item, then commit. 2. What this does: The work item will not be committed unless the request is sent; the reply will not be removed unless the work item is updated. 3. Transaction type: Since the messaging system and the workflow engine each has its own transaction manager, the transaction to coordinate them will be a distributed one.

So, an event-driven consumer tends to work the same whether or not its client is transactional.

But how does the consumer know when a new message is available? The easiest approach is for the consumer to repeatedly check the channel to see if a message is available. When a message is available, it consumes the message and then goes back to checking for the next one. This process is called polling.

The code for an Event-Driven Consumer consists of two parts: 1. Initialization—The application creates an application-specific consumer and associates it with a particular Message Channel (60). After this code is run once, the consumer is ready to receive a series of messages. 2. Consumption—The consumer receives a message and passes it to the application for processing. This code is run once per message being consumed.

A messaging system that allows multiple consumers to attempt consuming the same message can make a Transactional Client (484) very inefficient. The client thinks it has a message, consumes it, spends effort processing the message, then tries to commit and cannot (because the message has already been consumed by a competitor). Frequently performing work just to roll it back hurts throughput, whereas the point of this solution is to increase throughput. Thus, the performance of competing transactional consumers should be measured carefully; it could vary significantly on different messaging system implementations and configurations.

The ability to have multiple applications running on multiple computers using multiple threads to consume messages provides virtually unlimited message processing capacity, where the only limit is the messaging system’s ability to deliver messages from the channel to the consumers.

The coordination of competing consumers depends on each messaging system’s implementation. If the client wants to implement this coordination itself, it should use a Message Dispatcher (508). Competing Consumers can be Polling Consumers (494), Event-Driven Consumers (498), or a combination thereof. Competing Transactional Clients (484) can waste significant effort processing messages whose receive operations do not commit successfully and have to be rolled back.

Multiple consumers on a single Point-to-Point Channel (103) act as Competing Consumers (502). That’s fine when the consumers are interchangeable, but it does not allow for specializing the consumers so that certain consumers are better able to consume certain messages.

A Mediator coordinates a group of objects so that they don’t need to know how to coordinate with each other. What we need for messaging is a mediator that coordinates the consumers for a channel. Then, each consumer could focus on processing a particular kind of message, and the coordinator could make sure the right message gets to the right consumer.