More on this book
Kindle Notes & Highlights
by
Hohpe Gregor
at its core, IP traffic is asynchronous and unreliable and has to deal with many of the same issues enterprise integration solutions do.
both the DelayProcessor and the Resequencer have a few things in common: They both read messages from an input queue and publish them to an output queue. The only difference is in what happens in between—the actual processing of the message. Therefore, we created a common base class that encapsulates the basic functionality of this generic filter
the Content-Based Router (230) and Splitter (259) patterns processes an incoming order consisting of individual line items. Each line item requires an inventory check with the respective inventory system. After all items have been verified, we want to pass the validated order message to the next processing step.
A Splitter (259) can split a single message into multiple parts. A Content-Based Router (230) could then route individual submessages through the correct processing steps based on message content or type. The Pipes and Filters (70) architectural style allows us to chain together these two patterns so that we can route each item in the composed message to the appropriate processing steps:
each order item is routed to the proper inventory system to be verified. The inventory systems are decoupled from each other, and each system receives only items that can be processed by it.
The shortcoming of the setup so far is that we cannot find out whether all items that have been ordered are actually in stock and can be shipped. We also need to retrieve the prices for all items (factoring volume discounts) and assemble them into a single invoice. This requires us to continue processing as if the order is still a single message even though we just chopped it up into many submessages.
Each suborder is treated as an independent process. In some instances, lack of control over the downstream process may make this approach the only available solution. For example, Amazon follows this approach for a large portion of the goods it sells. Orders are routed to different fulfillment houses and managed from there.
It makes sense to combine the routing logic, the recipients, and the postprocessing of the individual messages into one logical component.
We can combine the Scatter-Gather with the Composed Message Processor (294) to process each incoming order, sequence it into individual items, pass each item up for a bid, aggregate the bids for each item into a combined bid response, and then aggregate all bid responses into a complete quote. This is a very real example of how multiple integration patterns can be combined into a complete solution. The composition of individual patterns into larger patterns allows us to discuss the solution at a higher level of abstraction. It also allows us to modify details of the implementation without
...more
The Pipes and Filters (70) architectural style gives us an elegant approach to represent a sequence of processing steps as independent filters connected by pipes (channels). In its default configuration, the filters are connected by fixed pipes. If we want to allow messages to be routed to different filters dynamically, we can use special filters that act as Message Routers (78). The routers dynamically determine the next filter to route the message to.
In the proposed solution, we assume that this routing logic is built into the processing component itself. If we look back at Option A, we remember that we dismissed this approach partly because we had to hard-code some logic into each component. How is the Routing Slip better? The key difference is that the router used in the Routing Slip is generic and does not have to change with changes in the routing logic. The routing logic incorporated into each component is similar to a Return Address (159), where the return address is selected from a list of addresses.
The Routing Slip assumes that we have the ability to augment the individual components with the router logic. If we are dealing with legacy applications or packaged applications, we may not be able to influence the functionality of the component itself. Rather, we need to use an external router that communicates with the component via messaging. This inevitably increases the number of channels and components in use. However, the Routing Slip still provides the best trade-off between our goals of efficiency, flexibility, and maintainability.
Each step is a stateless transformation. For example, let’s assume that we receive orders from a variety of business partners. All orders arrive on a common channel but in different formats depending on the partner. As a result, each message may require different transformation steps. Messages from some partners may require decryption; others may not. Some may require transformation or enrichment; others may not. Keeping a Routing Slip for each partner gives us an easy way to reconfigure the steps for each partner in a central location.
Each step gathers data, but makes no decisions (see Content Enricher [336]). In some cases, we receive a message that contains reference identifiers to other data.
We need to go to external sources to determine the customer’s name, the central office servicing the line, the distance from the central office, and so on. Once we have a complete message with all relevant data, we can decide what package to offer to the customer. In this scenario, the decision is postponed until the end, so we can use a Routing Slip to collect the necessary information. We need to assess, though, whether we really require the flexibility of the Routing Slip. Otherwise, a simple hard-wired chain of Pipes and Filters (70) may be sufficient.
The Chain of Responsibility allows each component to accept a message or route it to the next component in the list. The Routing Slip is a static list of all participants. This still implies that a central component has to have knowledge of all possible recipients. However, the component does not need to know which messages each component consumes.
Using a Routing Slip avoids the risk of duplicate message processing. Likewise, it is easy to determine if a message was not processed by any component. The main trade-off is the slower processing and increased network traffic.
We can reduce this number if we can arrange the systems in such a way that the first systems to receive the message have a higher chance of handling the message, but the number of messages will likely remain higher than with a predictive Content-Based Router (230).
When creating a service-oriented architecture, a single logical function is often composed of multiple independent steps. This situation occurs commonly for two primary reasons. First, packaged applications tend to expose fine-grained interfaces based on their internal APIs.
The Routing Slip (301) demonstrates how a message can be routed through a dynamic series of processing steps. The solution of the Routing Slip (301) is based on two key assumptions: The sequence of processing steps has to be determined up front, and the sequence is linear. In many cases, these assumptions may not be fulfilled. For example, routing decisions might have to be made based on intermediate results. Or, the processing steps may not be sequential, but multiple steps might be executed in parallel.
How do we route a message through multiple processing steps when the required steps may not be known at design time and may not be sequential?
One of the primary advantages of a Pipes and Filters (70) architectural style is the composability of individual processing units (“filters”) into a sequence by connecting them with channels (“pipes”). Each message is then routed through the sequence of processing units (or components). If we need to be able to change the sequence for each message, we can use multiple Content-Based Routers (230). This solution provides the maximum flexibility but has the disadvantage that the routing logic is spread across many routing components. The Routing Slip (301) provides a central point of control by
...more
First of all, let’s clarify that the design and configuration of a Process Manager is a pretty extensive topic. We could probably fill a whole book (Volume 2, maybe?) with patterns related to the design of workflow or business process management. Therefore, this pattern is intended primarily to “round off” the topic of routing patterns and to provide a pointer into the direction of workflow and process modeling. By no means is it a comprehensive treatment of business process design.
An incoming message initializes the Process Manager. We call this message the trigger message. Based on the rules inside the Process Manager, it sends a message (1) to the first processing step, implemented by Processing Unit A. After unit A completes its task, it sends a reply message back to the Process Manager. The Process Manager determines the next step to be executed and sends a message (2) to the next processing unit. As a result, all message traffic runs through this central “hub,” hence the term hub-and-spoke. The downside of this central control element is the danger of turning the
...more
the Process Manager needs to maintain the current position in the process execution.
the Process Manager can store this information without burdening the subsequent processing units with passing this data back and forth. This allows the individual processing steps to be independent of each other because they do not have to worry about data produced or consumed by other units.
Because the process execution can span many steps and can therefore take a long time, the Process Manager needs to be prepared to receive new trigger messages while another process is still executing. In order to manage multiple parallel executions, the Process Manager creates a new process instance for each incoming trigger message. The process instance stores the state associated with the execution of the process initiated by the trigger message. The state includes the current execution step of the process and any associated data. Each process instance is identified by a unique process
...more
The process definition is a design construct that defines the sequence of steps to be executed, comparable to a class in object-oriented languages.
The process instance is an active execution of a specific template, comparable to an object instance in an OO language.
Because multiple process instances may be executing simultaneously, the Process Manager needs to be able to associate an incoming message with the correct instance.
The Correlation Identifier (163) allows a component to associate an incoming reply message with the original request by storing a unique identifier in the reply message that correlates it to the request message.
In many cases, it makes most sense to use multiple Process Manager components, each of which houses a particular aspect of a larger process. The Process Manager components can then communicate with each other through a Pipes and Filters (70) architecture.
Using a central Process Manager makes it easy to retrieve the current state of a process and the associated data. Debugging a fully distributed architecture can be a lot more challenging and is almost impossible without the assistance of such mechanisms as the Message History (551) or Message Store (555).
Most commercial EAI implementations include a Process Manager component combined with visual tools to model the process definition. Most visual tools use a notation that resembles UML activity diagrams because the semantics of a Process Manager and those of an activity diagram are fairly similar.
The semantics of a process definition can be described in rather simple terms. The basic building block is an activity (sometimes called task or action). Usually, an activity can send a message to another component, wait for an incoming message, or execute a specific function internally
Activities can be connected in serial fashion, or multiple activities can be executed in parallel using a fork and join construct. A fork allows multiple activities to execute at the same time. It is semantically equivalent to a Publish-Subscribe Channel (106) in a hard-wired Pipes and Filters (70) architecture. A join synchronizes multiple parallel threads of execution back into a single thread. Execution after a join can continue only if all parallel threads have completed their respective activities. In the Pipes and Filters (70) style, an Aggregator (268) often serves this purpose. The
...more
A central point of control and state management can also mean a central point of failure or a performance bottleneck. For this reason, most Process Manager implementations allow persistent storage of process instance state in a file or in a database.
If the process engine persists all state information in a shared database, the system can become robust enough to survive the failure of a process engine—another engine can simply pick up where the previous one left off. The downside of this approach is that the state of each process instance has to be persisted in a central database after each processing step. This could easily turn the database into a new performance bottleneck. As so often happens, the architect has to find the correct balance between performance, robustness, cost, and maintainability.
Many patterns in this chapter present ways to route messages to the proper destination without the originating application being aware of the ultimate destination of the message. Most of the patterns focus on specific types of routing logic. However, in aggregate, these patterns solve a bigger problem. How can you decouple the destination of a message from the sender and maintain central control over the flow of messages?
All but the most trivial messaging solutions connect a number of different applications. If we created individual message channels to connect each application to each other application, the channels in the system would quickly explode into an unmanageable number, resulting in integration spaghetti
To avoid making the applications responsible for determining a message’s ultimate destination, the middleware should include a Message Router (78) that can route messages to the appropriate destination.
Also, implementing the logic inside the middleware layer allows us to make the logic “smarter” than would be practical inside of the application.
using a dynamic Recipient List (249) can avoid coding changes when new systems are added to the integration solution.
One excellent architectural style that uses a combination of Message Brokers is a Message Broker hierarchy (see figure). This configuration resembles a network configuration composed out of individual subnets. If a message has to travel only between two applications inside a subnet, the local Message Broker can manage the routing of the message. If the message is destined for another subnet, the local Message Broker can pass the message to the central Message Broker, which then determines the ultimate destination. The central Message Broker performs the same functions as a local Message
...more
A Content Enricher (336) is needed if the target system requires data fields that the originating system cannot supply. It has the ability to look up missing information or compute it from the available data. The Content Filter (342) does the opposite—it removes unwanted data from a message.
The Normalizer (352) translates messages arriving in many different formats into a common format.
Remote Procedure Invocation (50) has to make requests in the data format specified by the service that is to be called even if the application’s internal format is different. This typically requires the calling application to perform a transformation of data. Some of the most sophisticated transformation engines are incorporated into ETL (extract, transform, load) tools, such as Informatica or DataMirror. These tools typically transform a large set of data at once instead of transforming individual messages.
Some large enterprises use more than one messaging infrastructure. Thus, a message may have to be routed across messaging systems using a Messaging Bridge (133). Each messaging system is likely to have different requirements for the format of the message body as well as the header. This scenario is another case where we can learn by looking at existing TCP/IP-based network protocols. In many cases, connectivity to another system is restricted to a specific protocol, such as Telnet or Secure Shell (ssh). In order to enable communication using another protocol (for example, FTP), that protocol
...more
application data is wrapped into a TCP envelope first, which is then wrapped into an IP envelope, which is then wrapped into an Ethernet envelope. Since networks are stream-oriented, an envelope can consist of both a header and a trailer, marking the beginning and the end of the data stream.
In a loosely coupled system, we do not want one system to instruct the next one on what to do. We instead send an Event Message (151) and let the other systems decide what to do.
