Rethinking the ESB: What is SOA Architecture with APIs?

Some managers consider APIs a buzzword related to integrations, a problem that they have already addressed with SOA + ESB. Nevertheless, APIs address new demands, which are not well managed by SOA, while SOA focuses on problems that APIs do not have.

‍With that in mind, how to know if:

• Having an ESB for integrations is enough?
• API Gateway and ESB are complementary?
• I should replace my ESB with APIs + Microservices?

Understanding SOA / ESB gaps and the role of the API Management Platform

The new wave of digital transformation is not only a matter of digitizing internal processes and integrations, but also of exploring the potential of the partner ecosystem, using legacy systems alongside new technologies (IoT, Cloud, AI + Analytics, Blockchain, etc.), having real-time monitoring and reacting quickly, providing experiences through different channels and interfaces, involving customers and collaborators, and overcoming scalability and security challenges, among other elements.

Main concerns regarding ESB:

• Complex scalability: ESBs are monolithic and scale vertically, with a greater risk of unavailability.
• Affected performance when recording logs: they may reduce performance in transaction processing by up to 30%.
• Vendor lock-in: expensive consulting and the need to acquire many products (BAM, BPEL, and Service Registry).
• Focus only on integrations: little concern for the experience of frontend partners and developers.
• Few resources for Governance, Monitoring, and Monetization: lack of real-time visibility and complete logs of transactions, documentation, versioning, utilization and billing plans, etc.
• Limited support for security: OAuth 2.0, OpenID, JWT, etc.

Lack of modern standards and lighter protocols: ESBs have been or tend to be discontinued.

Why use APIs for internal and external integrations?

It is a simple, agile and scalable way to integrate systems and create decoupled applications in the shortest possible time. When using an API Management platform, modules are obtained to support security, authentication, scalability, dev commitment, governance, monitoring, monetization, and the test environment, among other resources.

How API Management and ESB supplement the capabilities of the integration layer:

API Management and the ESB can exist alone, depending on the scenario and focus. In most scenarios, companies would benefit from both. In these cases, the architecture is even more robust, as long as we manage to enhance the strengths of each of them:

Focus 1: API Management + ESB Coexistence

To reach a high degree of coverage in all the capacities of an integration layer, a possible approach would be to opt for the coexistence of the API Management platform with ESB. This approach enables the utilization of resources of the API Management platform (scalability, security, governance, dev commitment, Monetization, Analytics, etc.) with the capabilities of the ESB (variety of connectors, complex transaction processing, etc.).

There are, however, some limitations in terms of scalability, lock-in, and performance, while the rule engine can be obtained by means of a specific solution.

Approach 2: Replacing the ESB using API Management + API front (Microservices + Connectors)

To cover the capabilities that the API management platform does not focus on, without the limitations of ESB, it is possible to create a layer referred to as API Front.

This layer is composed of APIs, microservices and connectors, thereby reducing the difficulties of scalability, decoupling, reuse, lock-in, and performance, as well as specific platforms for batch processing and the rules engine.

Benefits of API Management + Microservices in its integration architecture

• More flexible integrations, facilitating faster deliveries and reducing costs

• Standardized interfaces, enabling multi-channel and agility in the integration with partners

• Establishment of governance and real-time monitoring of integrations

• Security and control over the use of APIs

• Mechanisms to optimize the performance of applications

• Separation of responsibilities

• Scalability

Detailed comparison of integration layer capabilities that may be supplied by an API Management Platform or by an ESB:

CapabilitiesPatternsAPI ManagementESBAlternative API ManagementAlternative ESB

‍

‍Communication asynchronousWebSocketYesPartial – some ESBs supportN/AMessaging (eg AMQP)Synchronous communicationREST / JSONYesPartial – some ESBs supportN/AWS-SOAPGraphQLYesNoN/AN/ASecurityOAuth 2.0YesPartial – some ESBs supportN/AWS-Security and Basic AuthDDoSYesNoN/AUse of third-party solutionsWAF (Injections, XSS, CSRF)YesNoN/AUse of third-party solutionsSpike ArrestYesNoN/AUse of third-party solutionsTechnical monitoringReal Time AlertsYesNoN/AN/ATechnical monitoringDashboards & ReportsYesPartial – requires add-onN/ABAM (add-on)Business MonitoringReal Time AlertsYesNoN/AN/ADashboards & ReportsYesPartial – requires add-onN/ABAM (add-on)Dev SupportDeveloper PortalYesNoN/Acustom portalComposition & EnrichmentService CalloutYesYesN/AN/AXSLTYesYesN/AN/Amessage ManipulationYesYesN/AN/AContract and Documentation StandardsOpenAPI SpecYesNoN/AWSDLContract and Documentation StandardsIterative documentationYesNoN/Acustom portalGeneration SDKYesNoN/ASDK manual generationJSON Schema ValidationYesNoN/AXSDGovernanceUsage quotasYesNoN/AUse of third-party solutionsVisibility ControlYesPartial – requires add-onN/AService Registry & Repository (add-on)API / Service CatalogYesPartial – requiresOn dd,N/AService Registry & Repository (add-on)LifecycleYesPartial – requires add-onN/AService Registry & Repository (add-on)Dependency VisibilityYesPartial – requires add-onN/AService Registry & Repository (add-on)TroubleshootingTraceYesYesN/AN/AAsynchronous LoggingYesNoN/AN/ACustom ReportsYesNoN/AUse of third-party solutions to process logsMessage TranslationXSLTYesYesN/AN/AMessage Mapping WizardNoYesCustom Mapping Code SnippetN/AAutomatic Translation (eg XML to JSON)YesYesN/AN/AMessage ManipulationYesYesN/AN/AText ManipulationYesYesN/AN/AProtocol TranslationSOAP to RESTYesPartial – some ESBs supportN/AN/ATo HTTP-based RESTYesPartial – some ESBs supportN/Acustom translationDatabase ConnectorYesYesN/AN/ASAP ConnectorNo – some support as addonYesCustom ConnectorN/AFTP ConnectorNoYesCustom ConnectorN/AOrchestrationProcess ChoreographyNoYescustom orchestration with MicroservicesN/ACompensation FlowNoYescustom orchestration with MicroservicesN/ARetry QueuesNoYescustom orchestration with MicroserviçosN/ABatch processingParallel Processing(High Data Volumes)NoYesUse third-party solutions – Data Integration PlatformsN/AJob SchedulingNoYesUse third-party solutions – Data Integration PlatformsN/AWorkload AutomationNoPartial – requires add-onUse third-party solutions – Data Integration PlatformsN/ARoutingStatic RoutingYesYesN/AN/AContent-based RoutingYesYesN/AN/ARules-based RoutingYesYesN/AN/APolicy-based RoutingYesYesN/AN/ALogging & Traceabilitystack TraceYesYesN/AN/ATransaction IdentifierYesNotN/AN/ATransaction ManagementDatabase Transaction HandlingNoYesCustom with MicroservicesN/AXA Transactions (2 Phase Commit)NoYesCustom with MicroservicesN/AWS-TransactionNoYesCustom with MicroservicesN/ARules EngineBRMSNoPartially – requires add-onUse third-party solutionsN/A