In this short series (see part 1 here) I’ll be doing a deep dive into some of the limitations of modern event-based microservices architectures. I’ll then be showing how looking at that architecture through the lens of process orchestration can solve these issues and even expand a lot on what you thought is possible for distributed systems. I’ve previously discussed the role of decision making, and in this post I want to talk about the consequences of diverse endpoints to an architecture.
In a perfect world services would use the same communication protocols and the same data structures. Sadly, as the old saying goes, variety is the spice of life and bane of software developers who just want an easy life.
Sure, you can decide that all your services will send and receive Kafka messages, but what happens when you’ve got to integrate a legacy system? Or an external service that only has a GraphQL endpoint? Well let’s discuss those decisions and how embracing a diversity of endpoints can provide a more true-to-life solution.
Diverse Endpoints
A business process that runs through your microservices is easier to manage if the services follow the 12 rules, as laid out by Adam Wiggins. These rules give new developers a good baseline for building new features. It also ensures the scope of services remains relatively similar, avoiding the creation of a “god service” within the architecture. But an often unconsidered value in connecting lots of independent services is being able to streamline your systems in a way that makes it easier to scale throughput, expand complexity and integrate new services. The vast majority of processes today require integration with technologies outside the scope of a typical microservice.
A recent study we commissioned revealed that 60% of new IT projects integrate 26 or more different systems. They need to connect to RPA bots, SaaS endpoints, and front end applications, many of which will not be able to abide by your rules. So what do you do? Most people consider something like a front end application simply out of scope for a microservice architecture, and it might be. It’s not out of scope for the process though. The process doesn’t care that in the middle of your beautifully compliant microservices architecture, there needs to sit a weird little front end that breaks all the rules. If you need to call out to a collection of strange unique endpoints for your process to complete you can’t say that they are out of scope without also admitting the inability of the solution to integrate those services. Meaning the architecture can no longer fulfill the requirements process.
Faced with this issue most people would attempt to build some kind of facade between the services, which creates technical debt without really solving the problem. To avoid this, designers simply redefine the microservice section of the architecture as being limited to completing “most of the process.” This seems like a perfectly fine compromise, but in time what’s happening outside of a collection of uniform endpoints and services will grow. Making your collection of microservices a smaller part of the process as a whole—because of the inability to integrate what’s needed no matter how old, how new or how strange—will either hamper what your business can achieve or totally limit the impact of your software solution.
Integrating diverse endpoints
There is of course a very good reason why we tend to avoid diverse endpoints. In a point-to-point microservices architecture knowing that each service uses the same data structures and protocols makes everything easier to manage and maintain. So the thinking is that introducing diverse endpoints will be harder to manage and maintain. That’s not strictly true.
Replacing point-to-point communication with a dedicated process orchestrator means there’s a space between two services. This space can be used to process or transform data structures from one service so it can be used in a protocol required in another service. To be clear this doesn’t replace the dumb pipes paradigm Martin Fowler has spoken about. In fact an orchestrator component works by keeping the pipes dumb and ensuring any logic around connectivity and protocol can be uniform within those pipes.
The example above demonstrates how It might look when integrating some service calls, message events and even front end interfaces.
There are two really big benefits to being able to easily integrate very different systems and technologies. The first (and likely you’ve guessed this) is that more systems are made available to you for your solution. Makes for fewer compromises in your overall solution.
The other big benefit is that you can now increase the scope of your project in order to implement the complete end-to-end process. Earlier I spoke about how limiting it is to compose your system within the framework of a microservice architecture, and this is how you break out of that limitation. It’s not that the microservice approach is now redundant or obsolete, but rather it can be expanded to better reflect a full and complete solution. The scope of the project no longer ends within the parameters of a technical limitation. Your project ends when you can see an end-to-end solution that is complete. A solution that incorporates any and all necessary systems, services, people, processes and stakeholders. As the diagram below demonstrates.
Conclusion
At the beginning of this post I mentioned three goals of any good distributed system.
- Easy to scale throughput
- Easy to expand complexity
- Easy to integrate new services
To accomplish these goals, limitations are imposed on most systems because the wisdom of time suggests that without them, the goals cannot be accomplished. So, the compromise set out by these limitations becomes accepted practice. But now orchestrators allow for these goals to be reached without the same limitations, and in fact give more freedom to better solve more complex problems. It’s clear to me that for someone who is trying to build a system for a critical business process an orchestrator is fundamental to avoiding unwanted compromises that will in the end limit the effectiveness of the overall solution.
Start the discussion at forum.camunda.io