Thinking Outside the Microservice Box: End-to-End Solutions (Part 3)

You might already do some kind of process orchestration. It’s inevitable if you’re looking to improve efficiency or digitalization. This post will help you understand what it means to embrace not only process orchestration but specifically end-to-end process orchestration.

Both concepts share the need to make repeatable, predictable, and efficient processes. The difference is the scope of the project and the return on investment, which for true end-to-end processes can be quite remarkable.

End-to-end processes should directly orchestrate every system, service, and user involved in the process from the very beginning to the very end, regardless of the technical and departmental boundaries the process may cross. Creating these processes also requires communication between stakeholders from both business and technical teams. While this might seem like an insurmountable project, orchestration engines like Camunda are designed for this specific purpose. The benefits are very much worth the integration effort. 

In this post I’m going to discuss what a true end-to-end solution looks like and compare it a bit to a classic microservice architecture. Finally, I’ll highlight the benefits for both software architects looking for something sustainable and fast and decision-makers who want a clear understanding of their business. 

Building an end-to-end solution?

When thinking about an end-to-end solution, you need to think of the absolute beginning and the absolute end of your process.

There’s no better way to define that than opening a BPMN modeler and creating a start event and end event and naming them accordingly. 

Congratulations! You’ve just defined the scope of your process. It might seem like a trivial step, but this is fundamental to success. Defining the state you’re expecting at the beginning and end of the process helps you clarify both what’s in scope and what’s out of scope.

For instance, this example uses book has been sold as an end event. This should make it very clear that returning a book after it’s sold is out of scope for this process. 

Once you understand all of the tasks, actions, events, and decisions that need to take place, you can take a look at the variety of systems that are involved. This will help you understand what needs to be built or accessed in the process to make it work. It also shows you the individual responsibilities of each service.

Take for example the Check Delivery Options task. It’s responsible for letting you know if delivery is possible, and if so how. But it’s not responsible for actively communicating to the user if there isn’t an option. It’s also not responsible for when and how to send the information to the logistics company. This becomes the responsibility of the orchestrator itself.

Making these kinds of decisions visible at design time is incredibly helpful, but it’s more important to have this visualized after production. 

Simply put, an end-to-end process aims to incorporate all departments, people, and technologies without limitation on the natural business scope of the process. People follow this approach for both technical and business reasons.

Technical benefits

A typical microservice project has uniform communication and data structures. Because of this, their scope is limited to systems that can follow those requirements. Microservice projects also tend to embed the logic of process routing in the services themselves, because there isn’t really another way to do it (more about that here).

Adding a process orchestration component to the architecture immediately removes these two limitations (limited scope and logic embedded in the services) by:

1.  Giving you access to services and systems that would normally be a pain to integrate into a microservice project.
2. Adding a component designed to keep ownership of all routing logic, so you no longer need that code in your services.

In fact, you now have a central location where updates can be made to the routing of your process independently of the processes being executed. This will also work as centralized error handling and give you the ability to manipulate running instances in flight in ways that would be impossible normally. 

Process instance modification—you can move or cancel an instance of the state.

This is just the tip of the iceberg. Externalizing the routing logic also gives you version control over it, meaning faster and easier changes can be reverted if necessary, all with minimal impact on the systems, services, and people involved in the orchestration. 

Business benefits 

Improving business outcomes has two stages—getting answers about your processes and implementing change.

Getting answers to questions about your business processes

“How long does it take from a customer order to a delivery of the product?” is a basic question I would hope most companies have some kind of answer to. However, it’s also a question that isn’t going to prompt any specific action except maybe further investigation.

You need questions that are going to help you understand the state of the process and give you ideas about where to find improvements, like: 

• “What are the correlations between orders that end up being late?”
• “Which elements of the process have the longest delay in terms of KPIs?” 
• “Which parts of the process should we be scaling up in order to increase speed of delivery?”

If you work with processes distributed across unconnected functions and hidden in log files or stack traces, you’re in luck. The entire industry of process mining exists to try to help discover the answers to questions like these.

It’s quite obvious as well that if you build a process using an orchestration engine and implement it end to end, you wont need to buy additional tooling to find out what’s already there. But that’s not even the main benefit of the One Model approach.

Report showing the most common path through the process

Report showing the average duration that elements take to complete. 

Implementing the changes that have been identified

The second stage of improving business outcomes is implementing the changes that you identified in stage one. This is truly where the business value cannot be matched. The speed at which new and improved processes can be built and deployed can be measured in days.

These changes can even be implemented in parallel with the older processes, where you can A-B test versions to ensure the desired outcome . It would be a monumental task to accomplish this with traditional software architecture. 

Conclusion

When you’ve embedded decision-making inside your services and you’ve had to limit your architecture to exclude diverse endpoints and systems, you simply don’t have a clear overview of what your end-to-end process looks like.

You might know which services are executed, but if you were to pick out a single execution, it would be really hard to know how it got to where it is and where it’s going to go next. Impossible in fact, unless you dive into the code. That’s a problem because this kind of information is vital to process improvement, and process improvement is fundamental to achieving business goals.

Basic but essential questions like, “Do our processes run efficiently?” or “What causes some processes to take too long or even fail?” would require someone to really dig into the system to answer accurately. Even if successful, making suggested changes would require code changes across multiple services. Measuring whether those changes made a positive impact is another round of research. 

Starting with an end-to-end process orchestration approach frees up services from having to do the routing, gives you a visual clarity on what’s happening, and lets you implement changes far faster than any alternative.