In Pursuit of AgilityIn Pursuit of Agility
As integration requirements rise, many organizations seek a technology platform that provides looser coupling and simpler information exchange. Is enterprise service bus (ESB) the most intelligent approach for your organization?
Ever-faster business change means that organizations must be more agile than ever to survive, much less capture emerging competitive advantages. An agile enterprise must be able to change the way it does business by adjusting processes and organizational structure without also having to undertake costly, lengthy, and risky alterations to its information systems.
Can an enterprise service bus (ESB) help bring greater agility? ESB has become the industry buzz for the next generation of enterprise application integration (EAI). In this article, I will describe the business needs for such a facility, define technical requirements that are critical for an ESB to meet such needs, and offer some observations about the current state and future trends of ESB adoption.
Agility Equals Survival
Envision an enterprise environment where you can make significant changes to the operation of the business by adjusting the business organization and its processes — and where the need to accomplish these changes by overhauling computer applications is the exception, not the rule. This idea is the essence of an agile enterprise, which must be prepared to adapt quickly to global marketplace changes, the development of new products and services, more extended enterprise relationships, as well as major technology advances.
Most enterprises are entangled in large, complex information systems with embedded business processes and formidable nests of links between applications. Changes to the business often require information system changes that could take months or years and involve expensive, high-risk operations. Adding to the complexity and risk is a myriad of products, standards, and protocols for integrating applications and exchanging information with business partners.
ESB is a marketing term; it doesn't have a precise definition. Generally — as opposed to traditional, hub-and-spoke EAI solutions — ESB describes a decentralized operation that leverages Web services and XML technology and supports a service-oriented architecture (SOA). Several vendors, including BEA, IBM, KnowNow, Sonic Software, Tibco, and WebMethods are developing and promoting solutions that might be characterized as ESB.
I define an ESB as a flexible integration infrastructure capability. An ESB enables systems to interact over a network in a loosely coupled manner by exchanging messages. The ESB provides any-to-any connectivity, equivalent to the Internet, and incorporates supporting services to provide reliable, secure access to events and delivery of messages. A robust ESB incorporates products from multiple vendors, each providing best-in-class solutions to particular needs.
An ESB enhances agility through supporting SOA. With SOA, enterprises will evolve to systems of smaller applications orchestrated by automated business processes. The business processes will be more flexible, and the applications will be more easily managed and replaced.
Agile Enterprise Requirements
The fundamental business need for an ESB is to enable exchange of messages among systems, both within the enterprise and with other enterprises over the public Internet. This basic need gives rise to a number of business and IT requirements:
EAI. While the ESB may be the future of EAI, many enterprises already have facilities in place using established EAI products. The ESB should interoperate with the existing EAI facilities so that there's no need for an immediate conversion. Applications can then migrate to the ESB over time.
SOA. The primary requirement here is open access to services, without the implementation of special links or protocols. The environment should support request-response access to services as well as collaboration among autonomous processes as they perform services.
Security. There must be authentication and authorization control over the acceptance of messages. ESB-based systems must support digital signatures for validation and nonrepudiation. It will be necessary to encrypt some messages, depending on the level of confidentiality required and the level of exposure of the communications facilities.
Events. The ESB should support a publish-and-subscribe capability for events. It should be possible for new applications or ad hoc monitoring applications to tap into events without modification of source applications.
Business process integration. Typically, different products in different parts of the enterprise implement automated business processes. The ESB must support the integration of these business processes across different internal implementations and outside, with those of business partner enterprises.
Message compatibility. Different applications will exchange data in different forms. The ESB must support message transformation to achieve compatibility. In general, the transformation should be to a common format understood by different applications at the same time — and as applications change over time.
Application abstraction. The ESB should provide a standard abstraction of message exchange for business applications. The application developer must not require a deep understanding of the technology and standards involved in the message exchanges.
Replaceable components. The ESB should enable the replacement of specific components, independent of vendor, as technology and standards evolve. ESB evolution should be orderly and incremental, not marked by major service disruptions.
Adaptive interface. At any one time, exchanges occurring within an enterprise and with other enterprises may employ different protocols within an application and even for the same business transaction. The ESB must adapt to multiple, evolving protocol requirements.
Monitoring and control. It must be possible to monitor and control ESB activity from one or more control points to manage resources, respond to outages, and address potential security threats.
Legacy adapters. And finally, ESB support for EAI will surely require legacy system adapters.
ESB in Operation
Figure 1: An example of an ESB environment.
As you can see in Figure 1, an ESB is formed by a collection of standards-based middleware and services, rather than a single product. Central to the ESB is a network that's typically based on TCP/IP but could involve other protocols. ESB middleware links most of the components to the network; various vendors offer this middleware for different platforms.
Here, I'd like to describe the operation of the ESB by describing potential message exchanges through the network. In Figure 1, the business function is driven by a business process that requires interaction with a service. The business function sends a message to the services registry to find a desired service. The registry responds with specifications for a service or list of alternative services that satisfy the request.
The business function selects a service and uses the service specification from the services registry to set up for the appropriate messaging protocols and message exchange sequence. The business function must use a business process compatible with the choreography (message exchange sequence) specified for the service, as well as the technical protocols for message exchange.
The message formats used by the business function may not be in the desired standard format. In that case, the ESB system will route the message to the target service through the transformation service. Now in a standard format, the ESB system delivers the message through the network to the target service, which could be another business function within the same or another enterprise.
If the target service requires messages in a non-standard format, then the message will be routed again through the transformation service. The service will respond with an appropriate message back to the requesting business function, passing through the transformation services as necessary.
The target service may restrict access to authorized users. Consequently, the service will utilize the authentication and authorization. This service may also provide public keys for signatures as well as encryption and decryption of documents being exchanged.
For accountability purposes, your organization may have to capture the documents being exchanged in the archive. If the documents are signed, then the signatures must be captured with the documents to ensure nonrepudiation. If a signed document is transformed, it must be captured in its original form before any transformation. The transformation service will capture the signed document before transformation and forward the transformed document. For nonrepudiation, the transformation service may be required to sign the transformed document to certify its validity.
The event registry supports another mode of messaging: publish and subscribe. The business function may require notice of the occurrence of certain events. It sends a subscription request to the event registry. Potential sources of events (that is, other business functions) have already registered their events with the event registry, which accepts a request for an event notice and forwards the request to the potential sources of the event type. Each event source then sends notices directly to all subscribers. If a subscription is terminated, the event registry will send the termination notice to all sources to which subscriptions were sent.
Figure 1 shows two links to other networks. An Internet gateway provides firewall protection and may only allow receipt of messages over the public Internet from authorized sources. The information broker bridge links the ESB network to a conventional EAI network, enabling the exchange of messages through an established information broker, using its associated transformation and adapter facilities.
The ESB management function monitors and controls the ESB network's activity. Events for monitoring activity are received from the ESB middleware and the associated services; commands are sent from the ESB management function to the middleware and services to control their operation.
Middleware Requirements
Figure 2: The ESB middleware architecture. the boxes down the center represent protocol stack levels; the dashed line represents the middleware boundry.
The ESB supports decentralized, loosely coupled messaging. From the ESB perspective, communications are from any to any. As a result, each sender must be able to communicate with each receiver in a way that the receiver can understand. Figure 2 depicts the ESB middleware architecture. The dashed line represents the middleware boundary.
ESB middleware is the interface between each business application and the network that links it to other business applications and services. The application interfaces to the ESB middleware should be industry standard so the applications aren't locked into specific vendor products.
The Protocol Stack
ESB middleware provides an adaptive interface. The boxes down the center of Figure 2 represent protocol stack levels. Each can be set to perform different protocols at its level. When the system wants to establish communications with a particular service or application, the middleware binds each of these levels according to the protocol requirements of that particular relationship. Different protocols may be used concurrently for different business transactions and relationships.
The typical message transport layer binding will be TCP/IP using HTTP; however, there may be alternatives to HTTP, and a new version of HTTP may require a different binding. The binding should enable using alternative transport facilities transparent to the applications.
The reliable messaging level ensures that a message is delivered once and only once. The binding capability should enable transparent use of alternative message-oriented middleware (for example, an EAI message broker) for the exchange of messages with legacy integration systems or to achieve improved performance.
Message packaging involves embedding the message payload in an appropriate envelope. This is typically a Simple Object Access Protocol (SOAP) envelope. However, variations in SOAP specifications will require alternative bindings.
The orchestration level implements the appropriate process for the sequence of exchange of messages to comply with the choreography specification. It's also the point of control for interaction with legacy applications where information must be submitted or retrieved to support the intended business transaction.
Transformation
A transformation function embedded within the local ESB middleware will achieve better performance with messages than a separate, loosely coupled service. In addition, the ESB middleware has access to both the original message and the transformed message so that it receives signed messages in their original form and can sign and send outgoing messages after they're transformed. However, a loosely coupled transformation service may provide greater flexibility in the management of transformations.
Routing and Subscriptions
The routing and subscriptions component determines the network destination addresses of messages. It's desirable for the business application to direct messages to logical destinations to be converted to the specific Uniform Resource Identifier (URI) by the routing component. The system may also route messages through the transformation service, depending on the source and target destination.
The routing and subscriptions component also determines the addresses of event messages. A subscriber subscribes to a topic, and the routing and subscriptions function will receive relevant subscriptions (and cancellations) from the event registry service. The routing function will direct an event message to all the subscribers for the associated message topic.
Rate of Adoption
The ESB architecture can provide flexibility for ad hoc exchanges and event-driven operations. However, for several reasons, adoption is still in the early stages. First, potential customers already have substantial investments in EAI technology, and conversion calls for another substantial investment. Second, organizations still use Web services primarily for exchanges within relatively small, closed communities where they can control protocols and coordinate changes carefully. Third, Web services protocols are evolving, so that without adaptive interface capabilities, applications built on these protocols will require continued revision by skilled people to keep up with the technology. Finally, for the past several years, most corporate IT organizations have been focused on cost reductions and gaining rapid return on investment. ESB implementation requires a long-term investment with speculative business value.
What will change these conditions? I see several developments to watch:
As the economy improves, competition will shift concerns from cost control to higher performance and agility.
Enterprises will be transitioning from batch processing to event-driven processing; a flexible infrastructure supports this transition.
The outsourcing of business functions that aren't key to competitive advantage will increase, driving a need for a more adaptive infrastructure with integration extending outside the enterprise.
Realization of the full capabilities of an ESB infrastructure will require development of additional industry standards. A user of this technology shouldn't be locked into a single vendor, and it's unlikely that a single vendor will be able to respond quickly to all changes in protocols and adapt effectively to all legacy technologies.
In particular, standards are needed for the following:
Application programming interfaces for the ESB middleware and protocol components.
Event registry service
ESB management protocols.
With an adaptive Web services interface in place, the continued evolution of Web services standards won't be a barrier to expansion. Standards efforts will then shift to the definition of various business transaction protocols — particularly the messages exchanged and the choreographies for particular types of business transactions.
Until then, the potential for competitive advantage is limited. Most organizations will continue to implement Web services interfaces on a case-by-case basis and adapt their EAI infrastructures to immediate integration needs.
Fred A. Cummins, EDS Fellow, is a systems architect and consultant with EDS. He is chair of the Business Enterprise Integration Domain Task Force for the OMG, and has more than 30 years of experience in management consulting and information systems development.
Resources
About the Author
You May Also Like