Integrations and leaky abstractions

How we deal with common integration challenges in our FTTX BSS suite to provide a better developer experience by building flexible, testable and observable BPMN workflows.

SwitchedOn Fibre™ is our modular FTTX BSS suite that automates processes across wholesale, retail and enterprise services for fibre communication service providers (CSP). For the uninitiated, the Business Support System (BSS) layer manages the interactions between the customers and the underlying telecoms network, which itself is managed by the Operations Support System (OSS).

As is typical with enterprise software solutions in other domains, BSS solutions need to integrate with numerous third-party systems inside and outside CSP organisations.

The difficulties in integration arise because although business processes managed by the BSS layer are much the same for different operators, at a high level, they do differ in their implementations in some fundamental ways: both in how the business process is organised, but also in what OSS elements they have to interact with. To shorten delivery timelines for our clients and to help us scale as an organisation, we work to avoid rebuilding core services to suit the needs of individual customers. To help us achieve this goal with our BSS, we’ve devolved parts of this work to a configurable process orchestration engine.

Parts? Why not all of it? Well, that’s what I hope to explain in this post.

Our preferred integration approach is to push integration to the edges of the architecture where:

• We have core services, with a fixed set of APIs, that interact with each other, to provide a coherent and well-defined set of business features to external clients.
• The core services implement the high-level BSS features that are common to all providers, while still allowing the flexibility required for bespoke changes.
• When these core services need to interact with external services in an ad-hoc or bespoke manner they do so via a configurable process orchestration (workflow) engine.
• The integration service provides a standard, simple, and consistent API to allow these core services to make safe requests to the workflow integrations.
• Using a standard API means that the integration engineers can focus on the implementation of their specific integration workflows, without needing to know how their work fits into the overall business process.
• The API also improves the developer experience by abstracting out all of the orchestration and even signalling required for complex, long-running integrations. This allows a developer to focus just on the work of interacting with the external service, and not on how to interact with SwitchedOn Fibre™ itself.

By the end of this post, we’ll hopefully have shown how this approach retains the flexibility that the workflow engines provide, while avoiding the additional complexity that they often introduce to the development, QA, operational and support phases of a deployment.

Integration at the edges?

Low-code development platforms promise to put business process modelling within the reach of business analysts and other non-developers. Our experience is that this approach becomes less tenable as workflows become more complex. By way of example, consider the following.

Credit (Formal Analysis of BPMN Via a Translation into COWS)

Two points stand out:

1. The mindset for implementing such a complex workflow, with loops and variables, is essentially a developer’s mindset. The same technical skill is required whether writing code, or using a graphical low-code tool.
2. Low-code tools generally do not have the types of troubleshooting features generally found in traditional developer tooling - capabilities like version control, debugging, unit and integration testing.

That said, low-code platforms do provide a significant degree of flexibility. They are applications that often have a wide degree of consumer and developer acceptance, with lots of publicly available training and documentation.

For SwitchedOn Fibre™, we chose to get the best of both worlds:

• Programmed core services communicate through a shared interface There is no single configurable workflow to manage the entire business process. Rather, there are a set of Spring microservices. These implement FTTX BSS fibre-specific features with well-defined, though flexible, interfaces - for example, appointment booking. All communication is via asynchronous events raised through a common event bus. This has the fortunate side-effect of making the entire application, including all integrations, completely observable.

• Separate low-code processes for individual business activities To fully implement these features, these core services must integrate with external services, which would likely be from different vendors running in external environments. To standardise how these integrations are implemented, we developed a System Integration Service that exposes a type-safe, event-based interface. Using this approach, each integration is related to a single business, or technical process and implements a well-defined interface. As such, it’s more readily testable and verifiable.

• Complex processes chain together multiple smaller workflows Single requests, for example “Install a new connection at this place”, are broken down into many smaller steps, with each requiring an integration of its own. These integrations can be anything from a simple once-off REST call, to a long-running event-based interaction (e.g. book an appointment with an external system and wait for a few weeks until it’s completed).

  

  Any attempt to implement this in a single process would result in a workflow so complex it would be difficult to prove that it worked for every possible case.

To solve this problem, we implemented an Order fulfilment Service that can process a well-formed standard request, and decompose into smaller, coherent workflows, for the System Integration Service to handle.

These smaller coherent tasks make the end-to-end process more readily observable. In addition, each element of the process can be validated separately and, in production environments, localising process failures and identifying root causes are less labour intensive.

• Simpler integration development process A benefit of this approach is that:

  • The business owner defining the overall process still needs a high degree of business knowledge to do so, but doesn’t require any programming skill.
  • The System Integration and Order fulfilment Services abstract away the need for the system integrator to understand either the overall business process, or even the inner workings of the SwitchedOn Fibre™ platform.

Finally, the approach outlined eases the onboarding path for new developers on our BSS teams.

What does an integration look like?

What follows is a brief overview of the work a developer needs to carry out for a new system integration. The goal is to demonstrate the benefits of this approach over implementing a single complex centralised workflow.

Implementing complex, coherent integrations

The System Integration Service wraps the Camunda platform in a type-safe yet generic interface. This abstraction removes the need for system integrators to deal with any inter-service communication and instead focus on implementing the requirements of a strict type-safe interface.

These processes can be simple integrations, such as doing a REST call and translating the results. Or it could be more complex; for example, send an asynchronous request to an external service, and listen on a webhook for updates that indicate process completion.

The only assumption is that they are coherent tasks related to a single action on some entity. That single action may map to a single technical action (e.g. a REST call) or to a long running interaction (e.g. send a request that could take days, and poll the external system until there’s a result). By way of example, defining an entity in the network inventory and deleting that entity would be classed as two separate workflows.

The integration process has been simplified so:

• The business owner will define a specific set of parameters to be expected, and a set of required results. They may also specify the need to raise events out to other systems.

• To send results, the integration developer only needs to save them as BPMN process variables using the correct names and formats. The System Integration Service then maps these to the standard internal format and raises the appropriate events that can then be consumed by interested core services.

• The integration is then implemented using all the freedom BPMN provides, but with much of complexity delegated to the System Integration Service where it handles:

  • Raising bespoke events out of the integration process

  • Listening for external events and translating them into BPMN messages to interrupt the process, where the interruption could be anything from a change of state, a request to halt the integration.

  • Deploy Java-based plugins that follow a type-safe interface. These allow for families of plugins to be defined using a standard Java interface, which can be implemented to provide the same functionality in different ways. For example, the SwitchedOn Fibre™ platform has an appointment service, with a standard type-safe java interface for finding timeslots.

    

    When we need to implement the work required to translate our internal API to the remote API of the workforce management service, we simply create a new implementation of our “find timeslots” interface, and bundle this as a plugin for our integrators to use in a workflow.

Organising complex business processes

The Order fulfilment Service provides a standard interface for chaining sets of these complex integrations into a single business request.

For example, a request to install a new FTTX connection at some address may need to be broken down into a chain of processes like the following.

A fulfilment plan represents the entire body of work for a business process, with each item being implemented a discrete integration, with its own BPMN workflow.

The Order fulfilment Service takes this fulfilment plan and manages all of the messaging required to complete it, such as:

• Mapping the plan items to BPMN workflows in the System Integration Service
• Processing the tasks in order, so a task won’t start until all the tasks it depends on are complete
• Raising events when the overall plan or individual item changes state
• Passing results from one task to another, later in the fulfilment plan. As with the system integration service, this abstraction means that the integration developer doesn’t need to concern themselves with how this happens in the context of the SwitchedOn Fibre™ platform. They only need to set the expected results as BPMN process variables in their System Integration Service workflow implementation.

Reducing complexity in the high-level plans

An important philosophical position that we took here was that the Order fulfilment Service wouldn’t attempt to correct or rollback a failed fulfilment plan, should there be a failure in a given plan item.

Rather, each item in the fulfilment plan would have a different implementation related to different actions - i.e. there are distinct BPMN workflow implementations for, say, activating a line, and deactivating a line - and subsequent “correction” plans would be submitted to reset any network or business elements. In practice this means:

• After a customer order is placed to, say, activate a new connection, a complete fulfilment plan is derived, with individual plan items mapping to workflows that create or configure the required network elements.
• If, halfway through the plan execution, the order is cancelled, the order fulfilment service will not attempt to roll back any of the completed plan items, it will just halt, and raise events to indicate which plan items were not halted in time.
• A second “correction” plan is then derived for these completed plan items, mapping workflows that delete or disable the network elements (or in some cases, the same “configuration” workflows, passing in the old configuration).

The benefits of this approach are:

• Since the process of correcting partial orders is standardised at the plan level, the overall plan is easier to follow and implement. There are no hidden cases in the middle of a complex plan that could change the entire outcome.
• The individual plan item workflows are easier to implement, and it’s easier to prove they were implemented correctly.
• The individual plan items also allow for a greater flexibility when handling errors on specific deployments.

Flexibility without the complexity

Low-code platforms are great for integrating with external systems while still providing the flexibility to make bespoke changes as required, and the Camunda platform is flexible, and easy to use. However, large, real-world, complex processes that interact with multiple external systems and manage multiple entities are challenging to test and maintain.

This post describes our philosophy of giving business analysts the freedom to specify ad-hoc external service integrations while retaining a great developer experience by making the integrations easy to implement, test, evolve, and maintain long-term.

If you like to learn more about our integration approach or about our SwitchedOn Fibre™ platform, schedule a demo today.